secs-gem/docs/00_index.md

The secs-gem guided tour

A tutorial series that teaches SECS/GEM the protocol and secs-gem the codebase at the same time. Starts from zero — no prior knowledge of semiconductor manufacturing or factory automation required — and ends with a working mental model of every namespace, state machine, and YAML file in this repository.

Each chapter does two things in parallel:

1. Explains a SEMI concept in plain language, with diagrams and a concrete example.
2. Shows where it lives in this codebase, with file paths and line references you can click straight into.

By the end you'll be able to read any commit, audit any YAML, or extend any subsystem without having to ask "what does that even mean?"

---

Who this is for

• Software engineers new to fab automation who need to make a semiconductor tool talk to a Manufacturing Execution System (MES).
• Vendor integrators who own the C++ side of an equipment deployment and need to wire secs-gem into a real recipe engine, PLC, or sensor stack.
• Fab IT / MES owners who need to understand what their equipment is sending them and why.
• Auditors and reviewers who want a structured walk through the codebase before signing off on compliance claims.

If you already know SECS/GEM and just want the codebase, skip to Part 3. If you know neither, start at Part 1, Chapter 01 and read straight through.

---

How the standards stack fits together

Before we dive in, here's the one-screen mental model. Every chapter in Part 2 fills in one of these boxes:

┌──────────────────────────────────────────────────────────────┐
│  GEM 300 — fab-floor behaviour (one rule book per concern)   │
│                                                              │
│   E40 process jobs     E94 control jobs    E87 carriers      │
│   E90 substrates       E157 modules        E116 perf time    │
│   E84 AMHS handoff     E120 common equip   E148 time sync    │
│   E42 formatted PP     E39 object services E5 §13 wafer maps │
└──────────────────────────────────────────────────────────────┘
              ▲  uses messages defined by
              │
┌──────────────────────────────────────────────────────────────┐
│  E30 — GEM: communication state + control state + scenarios  │
└──────────────────────────────────────────────────────────────┘
              ▲  uses messages encoded by
              │
┌──────────────────────────────────────────────────────────────┐
│  E5 — SECS-II: items, lists, format codes, message bodies    │
└──────────────────────────────────────────────────────────────┘
              ▲  carried over
              │
┌──────────────────────────────────────────────────────────────┐
│  E37 — HSMS (TCP)        │   E4 — SECS-I (RS-232 / RS-422)   │
└──────────────────────────────────────────────────────────────┘

Read top-to-bottom: a GEM 300 chapter (say, E40 process jobs) reasons about lifecycle states and emits SECS-II messages defined by E5, which travel over an HSMS connection defined by E37. The codebase mirrors that layering: secsgem::gem (top) sits on secsgem::secs2 (codec) which is moved by secsgem::hsms or secsgem::secsi (bottom).

---

The series

Twenty-four chapters in five parts. Read linearly the first time; on later visits, jump to whatever section answers your question.

Part 1 — Foundations

You can read these without ever opening a code file.

#	Title	What you'll know after
01	What is SECS/GEM?	Why a fab needs a protocol at all; the SEMI standards body; the one-paragraph history from SECS-I to GEM 300.
02	The cast of characters	Equipment vs. host vs. MES vs. scheduler vs. AMHS — who talks to whom and why.
03	Vocabulary + a wafer's journey	Every acronym you'll meet (SVID, CEID, ALID, PPID, ALCD, HCACK, …) traced through one wafer moving end-to-end through a fab.

Part 2 — The standards in detail

Each chapter takes one SEMI standard (or a tight family), explains what problem it solves, walks through the on-the-wire messages, and shows where the spec is implemented in this codebase. Hexdumps and state diagrams included.

#	Title	Covers
10	E5 — SECS-II data items	The 14 format codes, length-byte arithmetic, lists, the variant Item type, encode/decode.
11	E37 — HSMS transport	TCP framing, the SELECT handshake, T1–T8 timers, HSMS-SS vs. HSMS-GS, S9 error replies.
12	E4 — SECS-I serial	The original RS-232 transport; ENQ/EOT/ACK/NAK, blocking, T1/T2/T3/T4, why it still matters.
13	E30 — GEM behaviour	Communication state, control state, GEM Fundamentals + Additionals, scenarios, host commands.
14	E40 + E94 — process and control jobs	The PJ and CJ lifecycles, S16/S14 messages, the start-stop dance, cascading state.
15	E87 — carriers and load ports	FOUPs, load-port states, slot maps, carrier transfer, ProceedWithCarrier, CancelCarrier.
16	E90 + E157 — substrate and module tracking	Per-wafer state, process-module state, the relationship between PJ ↔ substrate ↔ module.
17	E116 + E120 + E39 — performance, CEM, objects	Equipment Performance Tracking time-buckets, Common Equipment Model, object-services GetAttr/SetAttr.
18	E84 — parallel I/O handoff	The 8-line AMHS handshake, TA1/TA2/TA3 timers, why a robot doesn't drop a $20k FOUP.
19	E42 + E148 + S9 — enhanced PPs, time sync, exceptions	Formatted process programs, distributed clock, S5F9–F18 exception recovery, the auto-S9 paths.

Part 3 — The codebase

Now we open the source. Every chapter is a guided walk through a specific namespace, with the call graphs, ownership rules, and extension points spelled out.

#	Title	Covers
30	Repository tour	Directory layout, build system, the eight apps, the test suite.
31	Spec-as-data + codegen	The five YAML files, how tools/generate_messages.py turns messages.yaml into typed C++.
32	Stores + the data model	EquipmentDataModel, every per-domain store (SVIDs, alarms, carriers, substrates, …) and how they compose.
33	Transport	hsms::Connection (asio TCP) and secsi::Protocol (FSM-only); the strand-threading contract; T-timer wiring.
34	Codec + SML	secs2::Item variant, encode/decode, the SML parser and printer, the identifier-wildcard rule.
35	State machines + dispatch	Control / PJ / CJ / EPT / E84 FSMs, the YAML-driven ControlTransitionTable, gem::Router.
36	Persistence + validation + metrics	Per-store journals, multi-version reads, --validate-config, the Prometheus exporter.

Part 4 — Operations

Reading the code teaches you what it does; this section teaches you how to run it.

#	Title	Covers
40	Building, running, the demo	Docker setup, the two-container demo, every transaction it walks through.
41	Integration	Wiring sensors and recipes, talking to a real MES, HSMS-GS for multi-MES, persistence layout, monitoring, security hardening, performance envelope.

Part 5 — Reference

Look-up material rather than narrative.

#	Title	Covers
50	API + message catalog + YAML schemas	Every namespace, every message in data/messages.yaml, every YAML key the config loader recognises.
51	Extending the codebase	How to add a new SVID, host command, state, message, store, FSM, or persistence backend — the actual mechanical steps.

---

How to read this guide

Pick a path based on what you're trying to do.

"I'm new to SECS/GEM and to this codebase." Read Parts 1, 2, 3 in order. Skim Part 4 to know what's there. Use Part 5 as reference. Budget: a long afternoon.

"I know SECS/GEM, I just need to learn this codebase." Skim Part 1.03 for vocabulary, skip Part 2, read Part 3 in full, then Part 4. Budget: 2 hours.

"I'm new to SECS/GEM but only need to consume what this tool emits." Read Parts 1, 2. Skip Parts 3, 4, 5. Budget: 3 hours.

"I'm integrating a real tool right now and need answers fast." Read Part 4 chapter 41; cross-reference Part 5 chapter 51 for each new behaviour you have to add. Budget: as long as the integration takes.

"I'm auditing for compliance / signing off on a deployment." Read ../COMPLIANCE.md first. Then read each Part 2 chapter for the standards in scope. Cross-check the code references against ../PROOFS.md.

---

Conventions used throughout

File references look like src/secs2/codec.cpp:123 — a path relative to the repo root, optional line number after a colon. When a function or symbol is the target, the form is namespace::Symbol followed by the file where it lives.

Wire dumps are shown in two forms — annotated SML (the human-readable SECS-II) on the left, raw hex on the right:

S1F1 W                       │ 00 00 00 0A   length prefix
.                            │ 00 00         session_id
                             │ 81 01         S=1, W=1, F=1
                             │ 00 00         PType/SType (data)
                             │ 00 00 00 01   system_bytes

Diagrams use the box-drawing characters above. No Mermaid — the repo's render targets (Gitea, GitHub, plain text) all handle the box-drawing characters uniformly.

Cross-references: chapter X.YZ refers to Part X, chapter YZ. E.g. "see 3.32 §3" means Part 3, chapter 32, section 3.

Spec citations look like E30 §6.5 — SEMI standard E30, section 6.5. The standards themselves are paywalled and not included in this repo. This guide is written to be readable without them; the section numbers are there so a reader who does have access can cross-check.

---

What this guide is not

• Not a substitute for the SEMI standards if you're certifying for production. We aim for accuracy, but if you're shipping into a fab, buy the official PDFs.
• Not a GEM RTS run. ../COMPLIANCE.md §8 explains the difference between "spec-implementing codebase" and "third-party-certified compliant equipment."
• Not a replacement for ../PROOFS.md. The proof table is the empirical claim; this guide is the explanatory text.

---

Where the rest of the docs live

Existing root-level docs are reference / audit artifacts. This guide is the tutorial path that ties them together.

Root doc	What it is	When to read
../README.md	One-page project summary + quick start	First contact
../PROOFS.md	The eight commands that prove feature-completeness	Verifying claims
../COMPLIANCE.md	Per-capability audit against every SEMI standard	Compliance review
../ARCHITECTURE.md	One-page architecture overview	Quick mental model
../INTEGRATION.md	Vendor-side integration tutorial	Going to production
../VERIFICATION.md	External validator test plan	Verification deep dive
../BENCHMARKS.md	Performance envelope	Capacity planning
../MES_INTEROP.md	Day-1 punch list for commercial MES integration	Pre-flight before MES connect
../SECURITY.md	Concrete nftables / stunnel / minisign / SIEM configs	Production hardening
../GLOSSARY.md	SEMI vocabulary cheat sheet	Quick term lookup
../FAQ.md	Common questions, canonical answers	Stuck? Check here first
../examples/pvd_tool/	A complete fictional PVD tool — YAML + main.cpp	Concrete reference deployment

---

Status of this guide

Chapters publish as they're written. The list above is the table of contents; individual files exist once the chapter has been written. A chapter without a working link is on the to-write list.

Currently published: Chapter 00 (this index).

In progress: Chapter 01 — What is SECS/GEM?

Next chapter: → 01 What is SECS/GEM?