secs-gem/docs/00_index.md

# The secs-gem guided tour

A tutorial series that teaches **SECS/GEM the protocol** and
**secs-gem the codebase** at the same time.  Starts from zero — no
prior knowledge of semiconductor manufacturing or factory automation
required — and ends with a working mental model of every namespace,
state machine, and YAML file in this repository.

Each chapter does two things in parallel:

1. **Explains a SEMI concept** in plain language, with diagrams and a
   concrete example.
2. **Shows where it lives in this codebase**, with file paths and
   line references you can click straight into.

By the end you'll be able to read any commit, audit any YAML, or
extend any subsystem without having to ask "what does that even
mean?"

---

## Who this is for

- **Software engineers new to fab automation** who need to make a
  semiconductor tool talk to a Manufacturing Execution System (MES).
- **Vendor integrators** who own the C++ side of an equipment
  deployment and need to wire `secs-gem` into a real recipe engine,
  PLC, or sensor stack.
- **Fab IT / MES owners** who need to understand what their
  equipment is sending them and why.
- **Auditors and reviewers** who want a structured walk through the
  codebase before signing off on compliance claims.

If you already know SECS/GEM and just want the codebase, skip to
[Part 3](#part-3--the-codebase).  If you know neither, start at
[Part 1, Chapter 01](01_what_is_secs_gem.md) and read straight
through.

---

## How the standards stack fits together

Before we dive in, here's the one-screen mental model.  Every chapter
in Part 2 fills in one of these boxes:

```
┌──────────────────────────────────────────────────────────────┐
│  GEM 300 — fab-floor behaviour (one rule book per concern)   │
│                                                              │
│   E40 process jobs     E94 control jobs    E87 carriers      │
│   E90 substrates       E157 modules        E116 perf time    │
│   E84 AMHS handoff     E120 common equip   E148 time sync    │
│   E42 formatted PP     E39 object services E5 §13 wafer maps │
└──────────────────────────────────────────────────────────────┘
              ▲  uses messages defined by
              │
┌──────────────────────────────────────────────────────────────┐
│  E30 — GEM: communication state + control state + scenarios  │
└──────────────────────────────────────────────────────────────┘
              ▲  uses messages encoded by
              │
┌──────────────────────────────────────────────────────────────┐
│  E5 — SECS-II: items, lists, format codes, message bodies    │
└──────────────────────────────────────────────────────────────┘
              ▲  carried over
              │
┌──────────────────────────────────────────────────────────────┐
│  E37 — HSMS (TCP)        │   E4 — SECS-I (RS-232 / RS-422)   │
└──────────────────────────────────────────────────────────────┘
```

Read top-to-bottom: a GEM 300 chapter (say, E40 process jobs)
*reasons* about lifecycle states and emits SECS-II messages defined
by E5, which travel over an HSMS connection defined by E37.  The
codebase mirrors that layering: `secsgem::gem` (top) sits on
`secsgem::secs2` (codec) which is moved by `secsgem::hsms` or
`secsgem::secsi` (bottom).

---

## The series

Twenty-four chapters in five parts.  Read linearly the first time;
on later visits, jump to whatever section answers your question.

### Part 1 — Foundations

You can read these without ever opening a code file.

| # | Title | What you'll know after |
|---|-------|------------------------|
| [01](01_what_is_secs_gem.md) | What is SECS/GEM? | Why a fab needs a protocol at all; the SEMI standards body; the one-paragraph history from SECS-I to GEM 300. |
| [02](02_the_cast.md) | The cast of characters | Equipment vs. host vs. MES vs. scheduler vs. AMHS — who talks to whom and why. |
| [03](03_vocabulary_and_a_wafers_journey.md) | Vocabulary + a wafer's journey | Every acronym you'll meet (SVID, CEID, ALID, PPID, ALCD, HCACK, …) traced through one wafer moving end-to-end through a fab. |

### Part 2 — The standards in detail

Each chapter takes one SEMI standard (or a tight family), explains
what problem it solves, walks through the on-the-wire messages, and
shows where the spec is implemented in this codebase.  Hexdumps and
state diagrams included.

| # | Title | Covers |
|---|-------|--------|
| [10](10_e5_secs_ii_data_items.md) | **E5** — SECS-II data items | The 14 format codes, length-byte arithmetic, lists, the variant `Item` type, encode/decode. |
| [11](11_e37_hsms.md) | **E37** — HSMS transport | TCP framing, the SELECT handshake, T1–T8 timers, HSMS-SS vs. HSMS-GS, S9 error replies. |
| [12](12_e4_secs_i.md) | **E4** — SECS-I serial | The original RS-232 transport; ENQ/EOT/ACK/NAK, blocking, T1/T2/T3/T4, why it still matters. |
| [13](13_e30_gem.md) | **E30** — GEM behaviour | Communication state, control state, GEM Fundamentals + Additionals, scenarios, host commands. |
| [14](14_e40_e94_jobs.md) | **E40 + E94** — process and control jobs | The PJ and CJ lifecycles, S16/S14 messages, the start-stop dance, cascading state. |
| [15](15_e87_carriers.md) | **E87** — carriers and load ports | FOUPs, load-port states, slot maps, carrier transfer, ProceedWithCarrier, CancelCarrier. |
| [16](16_e90_e157_substrates_modules.md) | **E90 + E157** — substrate and module tracking | Per-wafer state, process-module state, the relationship between PJ ↔ substrate ↔ module. |
| [17](17_e116_e120_e39_objects.md) | **E116 + E120 + E39** — performance, CEM, objects | Equipment Performance Tracking time-buckets, Common Equipment Model, object-services GetAttr/SetAttr. |
| [18](18_e84_parallel_io.md) | **E84** — parallel I/O handoff | The 8-line AMHS handshake, TA1/TA2/TA3 timers, why a robot doesn't drop a $20k FOUP. |
| [19](19_e42_e148_s9_misc.md) | **E42 + E148 + S9** — enhanced PPs, time sync, exceptions | Formatted process programs, distributed clock, S5F9–F18 exception recovery, the auto-S9 paths. |

### Part 3 — The codebase

Now we open the source.  Every chapter is a guided walk through a
specific namespace, with the call graphs, ownership rules, and
extension points spelled out.

| # | Title | Covers |
|---|-------|--------|
| [30](30_repository_tour.md) | Repository tour | Directory layout, build system, the eight apps, the test suite. |
| [31](31_spec_as_data_and_codegen.md) | Spec-as-data + codegen | The five YAML files, how `tools/generate_messages.py` turns `messages.yaml` into typed C++. |
| [32](32_stores_and_the_data_model.md) | Stores + the data model | `EquipmentDataModel`, every per-domain store (SVIDs, alarms, carriers, substrates, …) and how they compose. |
| [33](33_transport.md) | Transport | `hsms::Connection` (asio TCP) and `secsi::Protocol` (FSM-only); the strand-threading contract; T-timer wiring. |
| [34](34_codec_and_sml.md) | Codec + SML | `secs2::Item` variant, `encode`/`decode`, the SML parser and printer, the identifier-wildcard rule. |
| [35](35_state_machines_and_dispatch.md) | State machines + dispatch | Control / PJ / CJ / EPT / E84 FSMs, the YAML-driven `ControlTransitionTable`, `gem::Router`. |
| [36](36_persistence_validation_metrics.md) | Persistence + validation + metrics | Per-store journals, multi-version reads, `--validate-config`, the Prometheus exporter. |

### Part 4 — Operations

Reading the code teaches you what it does; this section teaches you
how to run it.

| # | Title | Covers |
|---|-------|--------|
| [40](40_building_running_demo.md) | Building, running, the demo | Docker setup, the two-container demo, every transaction it walks through. |
| [41](41_integration_hardware_mes_production.md) | Integration | Wiring sensors and recipes, talking to a real MES, HSMS-GS for multi-MES, persistence layout, monitoring, security hardening, performance envelope. |

### Part 5 — Reference

Look-up material rather than narrative.

| # | Title | Covers |
|---|-------|--------|
| [50](50_api_messages_yaml_reference.md) | API + message catalog + YAML schemas | Every namespace, every message in `data/messages.yaml`, every YAML key the config loader recognises. |
| [51](51_extending_the_codebase.md) | Extending the codebase | How to add a new SVID, host command, state, message, store, FSM, or persistence backend — the actual mechanical steps. |

---

## How to read this guide

Pick a path based on what you're trying to do.

**"I'm new to SECS/GEM and to this codebase."**
Read Parts 1, 2, 3 in order.  Skim Part 4 to know what's there.  Use
Part 5 as reference.  Budget: a long afternoon.

**"I know SECS/GEM, I just need to learn this codebase."**
Skim Part 1.03 for vocabulary, skip Part 2, read Part 3 in full,
then Part 4.  Budget: 2 hours.

**"I'm new to SECS/GEM but only need to consume what this tool
emits."**
Read Parts 1, 2.  Skip Parts 3, 4, 5.  Budget: 3 hours.

**"I'm integrating a real tool right now and need answers fast."**
Read Part 4 chapter 41; cross-reference Part 5 chapter 51 for each
new behaviour you have to add.  Budget: as long as the integration
takes.

**"I'm auditing for compliance / signing off on a deployment."**
Read [`../COMPLIANCE.md`](../COMPLIANCE.md) first.  Then read each
Part 2 chapter for the standards in scope.  Cross-check the code
references against [`../PROOFS.md`](../PROOFS.md).

---

## Conventions used throughout

**File references** look like `src/secs2/codec.cpp:123` — a path
relative to the repo root, optional line number after a colon.  When
a function or symbol is the target, the form is `namespace::Symbol`
followed by the file where it lives.

**Wire dumps** are shown in two forms — annotated SML (the
human-readable SECS-II) on the left, raw hex on the right:

```
S1F1 W                       │ 00 00 00 0A   length prefix
.                            │ 00 00         session_id
                             │ 81 01         S=1, W=1, F=1
                             │ 00 00         PType/SType (data)
                             │ 00 00 00 01   system_bytes
```

**Diagrams** use the box-drawing characters above.  No Mermaid — the
repo's render targets (Gitea, GitHub, plain text) all handle the
box-drawing characters uniformly.

**Cross-references**: chapter X.YZ refers to Part X, chapter YZ.
E.g. "see 3.32 §3" means Part 3, chapter 32, section 3.

**Spec citations** look like `E30 §6.5` — SEMI standard E30,
section 6.5.  The standards themselves are paywalled and *not*
included in this repo.  This guide is written to be readable without
them; the section numbers are there so a reader who *does* have
access can cross-check.

---

## What this guide is not

- **Not a substitute for the SEMI standards** if you're certifying
  for production.  We aim for accuracy, but if you're shipping into
  a fab, buy the official PDFs.
- **Not a GEM RTS run.**  [`../COMPLIANCE.md`](../COMPLIANCE.md) §8
  explains the difference between "spec-implementing codebase" and
  "third-party-certified compliant equipment."
- **Not a replacement for `../PROOFS.md`.**  The proof table is the
  empirical claim; this guide is the explanatory text.

---

## Where the rest of the docs live

Existing root-level docs are reference / audit artifacts.  This guide
is the *tutorial path* that ties them together.

| Root doc                                | What it is                                                  | When to read |
|-----------------------------------------|-------------------------------------------------------------|--------------|
| [`../README.md`](../README.md)          | One-page project summary + quick start                       | First contact |
| [`../PROOFS.md`](../PROOFS.md)          | The eight commands that prove feature-completeness           | Verifying claims |
| [`../COMPLIANCE.md`](../COMPLIANCE.md)  | Per-capability audit against every SEMI standard             | Compliance review |
| [`../ARCHITECTURE.md`](../ARCHITECTURE.md) | One-page architecture overview                              | Quick mental model |
| [`../INTEGRATION.md`](../INTEGRATION.md) | Vendor-side integration tutorial                            | Going to production |
| [`../VERIFICATION.md`](../VERIFICATION.md) | External validator test plan                                | Verification deep dive |
| [`../BENCHMARKS.md`](../BENCHMARKS.md)  | Performance envelope                                         | Capacity planning |
| [`../MES_INTEROP.md`](../MES_INTEROP.md) | Day-1 punch list for commercial MES integration             | Pre-flight before MES connect |
| [`../SECURITY.md`](../SECURITY.md)      | Concrete nftables / stunnel / minisign / SIEM configs        | Production hardening |
| [`../GLOSSARY.md`](../GLOSSARY.md)      | SEMI vocabulary cheat sheet                                  | Quick term lookup |
| [`../FAQ.md`](../FAQ.md)                | Common questions, canonical answers                          | Stuck? Check here first |
| [`../examples/pvd_tool/`](../examples/pvd_tool/) | A complete fictional PVD tool — YAML + main.cpp     | Concrete reference deployment |

---

## Status of this guide

Chapters publish as they're written.  The list above is the table of
contents; individual files exist once the chapter has been written.
A chapter without a working link is on the to-write list.

**Currently published:** Chapter 00 (this index).

**In progress:** Chapter 01 — *What is SECS/GEM?*

Next chapter: [→ 01 What is SECS/GEM?](01_what_is_secs_gem.md)