raphael
0355c73211
generate_messages.py → gen_messages.py and several gem/ headers moved under gem/store/ (carrier_store.hpp → store/carriers.hpp, etc.); e84.hpp split into e84_state.hpp. The guided-tour chapters still pointed at the old paths — relink them so the deep-link footnotes resolve. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
256 lines
14 KiB
Markdown
256 lines
14 KiB
Markdown
# The secs-gem guided tour
|
||
|
||
A tutorial series that teaches **SECS/GEM the protocol** and
|
||
**secs-gem the codebase** at the same time. Starts from zero — no
|
||
prior knowledge of semiconductor manufacturing or factory automation
|
||
required — and ends with a working mental model of every namespace,
|
||
state machine, and YAML file in this repository.
|
||
|
||
Each chapter does two things in parallel:
|
||
|
||
1. **Explains a SEMI concept** in plain language, with diagrams and a
|
||
concrete example.
|
||
2. **Shows where it lives in this codebase**, with file paths and
|
||
line references you can click straight into.
|
||
|
||
By the end you'll be able to read any commit, audit any YAML, or
|
||
extend any subsystem without having to ask "what does that even
|
||
mean?"
|
||
|
||
---
|
||
|
||
## Who this is for
|
||
|
||
- **Software engineers new to fab automation** who need to make a
|
||
semiconductor tool talk to a Manufacturing Execution System (MES).
|
||
- **Vendor integrators** who own the C++ side of an equipment
|
||
deployment and need to wire `secs-gem` into a real recipe engine,
|
||
PLC, or sensor stack.
|
||
- **Fab IT / MES owners** who need to understand what their
|
||
equipment is sending them and why.
|
||
- **Auditors and reviewers** who want a structured walk through the
|
||
codebase before signing off on compliance claims.
|
||
|
||
If you already know SECS/GEM and just want the codebase, skip to
|
||
[Part 3](#part-3--the-codebase). If you know neither, start at
|
||
[Part 1, Chapter 01](01_what_is_secs_gem.md) and read straight
|
||
through.
|
||
|
||
---
|
||
|
||
## How the standards stack fits together
|
||
|
||
Before we dive in, here's the one-screen mental model. Every chapter
|
||
in Part 2 fills in one of these boxes:
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────┐
|
||
│ GEM 300 — fab-floor behaviour (one rule book per concern) │
|
||
│ │
|
||
│ E40 process jobs E94 control jobs E87 carriers │
|
||
│ E90 substrates E157 modules E116 perf time │
|
||
│ E84 AMHS handoff E120 common equip E148 time sync │
|
||
│ E42 formatted PP E39 object services E5 §13 wafer maps │
|
||
└──────────────────────────────────────────────────────────────┘
|
||
▲ uses messages defined by
|
||
│
|
||
┌──────────────────────────────────────────────────────────────┐
|
||
│ E30 — GEM: communication state + control state + scenarios │
|
||
└──────────────────────────────────────────────────────────────┘
|
||
▲ uses messages encoded by
|
||
│
|
||
┌──────────────────────────────────────────────────────────────┐
|
||
│ E5 — SECS-II: items, lists, format codes, message bodies │
|
||
└──────────────────────────────────────────────────────────────┘
|
||
▲ carried over
|
||
│
|
||
┌──────────────────────────────────────────────────────────────┐
|
||
│ E37 — HSMS (TCP) │ E4 — SECS-I (RS-232 / RS-422) │
|
||
└──────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
Read top-to-bottom: a GEM 300 chapter (say, E40 process jobs)
|
||
*reasons* about lifecycle states and emits SECS-II messages defined
|
||
by E5, which travel over an HSMS connection defined by E37. The
|
||
codebase mirrors that layering: `secsgem::gem` (top) sits on
|
||
`secsgem::secs2` (codec) which is moved by `secsgem::hsms` or
|
||
`secsgem::secsi` (bottom).
|
||
|
||
---
|
||
|
||
## The series
|
||
|
||
Twenty-four chapters in five parts. Read linearly the first time;
|
||
on later visits, jump to whatever section answers your question.
|
||
|
||
### Part 1 — Foundations
|
||
|
||
You can read these without ever opening a code file.
|
||
|
||
| # | Title | What you'll know after |
|
||
|---|-------|------------------------|
|
||
| [01](01_what_is_secs_gem.md) | What is SECS/GEM? | Why a fab needs a protocol at all; the SEMI standards body; the one-paragraph history from SECS-I to GEM 300. |
|
||
| [02](02_the_cast.md) | The cast of characters | Equipment vs. host vs. MES vs. scheduler vs. AMHS — who talks to whom and why. |
|
||
| [03](03_vocabulary_and_a_wafers_journey.md) | Vocabulary + a wafer's journey | Every acronym you'll meet (SVID, CEID, ALID, PPID, ALCD, HCACK, …) traced through one wafer moving end-to-end through a fab. |
|
||
|
||
### Part 2 — The standards in detail
|
||
|
||
Each chapter takes one SEMI standard (or a tight family), explains
|
||
what problem it solves, walks through the on-the-wire messages, and
|
||
shows where the spec is implemented in this codebase. Hexdumps and
|
||
state diagrams included.
|
||
|
||
| # | Title | Covers |
|
||
|---|-------|--------|
|
||
| [10](10_e5_secs_ii_data_items.md) | **E5** — SECS-II data items | The 14 format codes, length-byte arithmetic, lists, the variant `Item` type, encode/decode. |
|
||
| [11](11_e37_hsms.md) | **E37** — HSMS transport | TCP framing, the SELECT handshake, T1–T8 timers, HSMS-SS vs. HSMS-GS, S9 error replies. |
|
||
| [12](12_e4_secs_i.md) | **E4** — SECS-I serial | The original RS-232 transport; ENQ/EOT/ACK/NAK, blocking, T1/T2/T3/T4, why it still matters. |
|
||
| [13](13_e30_gem.md) | **E30** — GEM behaviour | Communication state, control state, GEM Fundamentals + Additionals, scenarios, host commands. |
|
||
| [14](14_e40_e94_jobs.md) | **E40 + E94** — process and control jobs | The PJ and CJ lifecycles, S16/S14 messages, the start-stop dance, cascading state. |
|
||
| [15](15_e87_carriers.md) | **E87** — carriers and load ports | FOUPs, load-port states, slot maps, carrier transfer, ProceedWithCarrier, CancelCarrier. |
|
||
| [16](16_e90_e157_substrates_modules.md) | **E90 + E157** — substrate and module tracking | Per-wafer state, process-module state, the relationship between PJ ↔ substrate ↔ module. |
|
||
| [17](17_e116_e120_e39_objects.md) | **E116 + E120 + E39** — performance, CEM, objects | Equipment Performance Tracking time-buckets, Common Equipment Model, object-services GetAttr/SetAttr. |
|
||
| [18](18_e84_parallel_io.md) | **E84** — parallel I/O handoff | The 8-line AMHS handshake, TA1/TA2/TA3 timers, why a robot doesn't drop a $20k FOUP. |
|
||
| [19](19_e42_e148_s9_misc.md) | **E42 + E148 + S9** — enhanced PPs, time sync, exceptions | Formatted process programs, distributed clock, S5F9–F18 exception recovery, the auto-S9 paths. |
|
||
|
||
### Part 3 — The codebase
|
||
|
||
Now we open the source. Every chapter is a guided walk through a
|
||
specific namespace, with the call graphs, ownership rules, and
|
||
extension points spelled out.
|
||
|
||
| # | Title | Covers |
|
||
|---|-------|--------|
|
||
| [30](30_repository_tour.md) | Repository tour | Directory layout, build system, the eight apps, the test suite. |
|
||
| [31](31_spec_as_data_and_codegen.md) | Spec-as-data + codegen | The five YAML files, how `tools/gen_messages.py` turns `messages.yaml` into typed C++. |
|
||
| [32](32_stores_and_the_data_model.md) | Stores + the data model | `EquipmentDataModel`, every per-domain store (SVIDs, alarms, carriers, substrates, …) and how they compose. |
|
||
| [33](33_transport.md) | Transport | `hsms::Connection` (asio TCP) and `secsi::Protocol` (FSM-only); the strand-threading contract; T-timer wiring. |
|
||
| [34](34_codec_and_sml.md) | Codec + SML | `secs2::Item` variant, `encode`/`decode`, the SML parser and printer, the identifier-wildcard rule. |
|
||
| [35](35_state_machines_and_dispatch.md) | State machines + dispatch | Control / PJ / CJ / EPT / E84 FSMs, the YAML-driven `ControlTransitionTable`, `gem::Router`. |
|
||
| [36](36_persistence_validation_metrics.md) | Persistence + validation + metrics | Per-store journals, multi-version reads, `--validate-config`, the Prometheus exporter. |
|
||
|
||
### Part 4 — Operations
|
||
|
||
Reading the code teaches you what it does; this section teaches you
|
||
how to run it.
|
||
|
||
| # | Title | Covers |
|
||
|---|-------|--------|
|
||
| [40](40_building_running_demo.md) | Building, running, the demo | Docker setup, the two-container demo, every transaction it walks through. |
|
||
| [41](41_integration_hardware_mes_production.md) | Integration | Wiring sensors and recipes, talking to a real MES, HSMS-GS for multi-MES, persistence layout, monitoring, security hardening, performance envelope. |
|
||
|
||
### Part 5 — Reference
|
||
|
||
Look-up material rather than narrative.
|
||
|
||
| # | Title | Covers |
|
||
|---|-------|--------|
|
||
| [50](50_api_messages_yaml_reference.md) | API + message catalog + YAML schemas | Every namespace, every message in `data/messages.yaml`, every YAML key the config loader recognises. |
|
||
| [51](51_extending_the_codebase.md) | Extending the codebase | How to add a new SVID, host command, state, message, store, FSM, or persistence backend — the actual mechanical steps. |
|
||
|
||
---
|
||
|
||
## How to read this guide
|
||
|
||
Pick a path based on what you're trying to do.
|
||
|
||
**"I'm new to SECS/GEM and to this codebase."**
|
||
Read Parts 1, 2, 3 in order. Skim Part 4 to know what's there. Use
|
||
Part 5 as reference. Budget: a long afternoon.
|
||
|
||
**"I know SECS/GEM, I just need to learn this codebase."**
|
||
Skim Part 1.03 for vocabulary, skip Part 2, read Part 3 in full,
|
||
then Part 4. Budget: 2 hours.
|
||
|
||
**"I'm new to SECS/GEM but only need to consume what this tool
|
||
emits."**
|
||
Read Parts 1, 2. Skip Parts 3, 4, 5. Budget: 3 hours.
|
||
|
||
**"I'm integrating a real tool right now and need answers fast."**
|
||
Read Part 4 chapter 41; cross-reference Part 5 chapter 51 for each
|
||
new behaviour you have to add. Budget: as long as the integration
|
||
takes.
|
||
|
||
**"I'm auditing for compliance / signing off on a deployment."**
|
||
Read [`COMPLIANCE.md`](COMPLIANCE.md) first. Then read each Part 2
|
||
chapter for the standards in scope. Cross-check the code references
|
||
against [`PROOFS.md`](PROOFS.md).
|
||
|
||
---
|
||
|
||
## Conventions used throughout
|
||
|
||
**File references** look like `src/secs2/codec.cpp:123` — a path
|
||
relative to the repo root, optional line number after a colon. When
|
||
a function or symbol is the target, the form is `namespace::Symbol`
|
||
followed by the file where it lives.
|
||
|
||
**Wire dumps** are shown in two forms — annotated SML (the
|
||
human-readable SECS-II) on the left, raw hex on the right:
|
||
|
||
```
|
||
S1F1 W │ 00 00 00 0A length prefix
|
||
. │ 00 00 session_id
|
||
│ 81 01 S=1, W=1, F=1
|
||
│ 00 00 PType/SType (data)
|
||
│ 00 00 00 01 system_bytes
|
||
```
|
||
|
||
**Diagrams** use the box-drawing characters above. No Mermaid — the
|
||
repo's render targets (Gitea, GitHub, plain text) all handle the
|
||
box-drawing characters uniformly.
|
||
|
||
**Cross-references**: chapter X.YZ refers to Part X, chapter YZ.
|
||
E.g. "see 3.32 §3" means Part 3, chapter 32, section 3.
|
||
|
||
**Spec citations** look like `E30 §6.5` — SEMI standard E30,
|
||
section 6.5. The standards themselves are paywalled and *not*
|
||
included in this repo. This guide is written to be readable without
|
||
them; the section numbers are there so a reader who *does* have
|
||
access can cross-check.
|
||
|
||
---
|
||
|
||
## What this guide is not
|
||
|
||
- **Not a substitute for the SEMI standards** if you're certifying
|
||
for production. We aim for accuracy, but if you're shipping into
|
||
a fab, buy the official PDFs.
|
||
- **Not a GEM RTS run.** [`COMPLIANCE.md`](COMPLIANCE.md) §8
|
||
explains the difference between "spec-implementing codebase" and
|
||
"third-party-certified compliant equipment."
|
||
- **Not a replacement for `../PROOFS.md`.** The proof table is the
|
||
empirical claim; this guide is the explanatory text.
|
||
|
||
---
|
||
|
||
## Where the rest of the docs live
|
||
|
||
The other docs in this directory are reference / audit artifacts;
|
||
this guide is the *tutorial path* that ties them together.
|
||
|
||
| Doc | What it is | When to read |
|
||
|--------------------------------------------------|-------------------------------------------------------------|--------------|
|
||
| [`../README.md`](../README.md) | One-page project summary + quick start | First contact |
|
||
| [`PROOFS.md`](PROOFS.md) | The eight commands that prove feature-completeness | Verifying claims |
|
||
| [`COMPLIANCE.md`](COMPLIANCE.md) | Per-capability audit against every SEMI standard | Compliance review |
|
||
| [`ARCHITECTURE.md`](ARCHITECTURE.md) | One-page architecture overview | Quick mental model |
|
||
| [`INTEGRATION.md`](INTEGRATION.md) | Vendor-side integration tutorial | Going to production |
|
||
| [`VERIFICATION.md`](VERIFICATION.md) | External validator test plan | Verification deep dive |
|
||
| [`BENCHMARKS.md`](BENCHMARKS.md) | Performance envelope | Capacity planning |
|
||
| [`MES_INTEROP.md`](MES_INTEROP.md) | Day-1 punch list for commercial MES integration | Pre-flight before MES connect |
|
||
| [`SECURITY.md`](SECURITY.md) | Concrete nftables / stunnel / minisign / SIEM configs | Production hardening |
|
||
| [`GLOSSARY.md`](GLOSSARY.md) | SEMI vocabulary cheat sheet | Quick term lookup |
|
||
| [`FAQ.md`](FAQ.md) | Common questions, canonical answers | Stuck? Check here first |
|
||
| [`../examples/pvd_tool/`](../examples/pvd_tool/) | A complete fictional PVD tool — YAML + main.cpp | Concrete reference deployment |
|
||
|
||
---
|
||
|
||
## Status of this guide
|
||
|
||
**All 24 chapters published.** Read linearly from
|
||
[01](01_what_is_secs_gem.md) or jump in at whichever part fits your
|
||
goal (see "How to read this guide" above).
|
||
|
||
Next chapter: [→ 01 What is SECS/GEM?](01_what_is_secs_gem.md)
|