docs: chapters 30–36 — the codebase (Part 3 complete)

Seven chapters walking the implementation top-to-bottom. 30 — Repository tour. Top-level layout, directory by directory. The eight built binaries. The dependency graph from TCP socket up through EquipmentDataModel. CMake's role. Test layout. 31 — Spec-as-data and codegen. Why the design choice fits SECS/ GEM specifically. The five YAML files: messages catalog, control/PJ/CJ transition tables, equipment dictionary. How tools/gen_messages.py turns messages.yaml into typed C++ at build time. The --validate-config multi-error validator. How to add a new SVID / CEID / host command / state / message without C++. 32 — Stores and the data model. What a store IS (records + API + change handler + optional persistence). Every store in the codebase mapped to the SEMI standard it serves (table of 21). EquipmentDataModel as plain composition + cross-store convenience methods (vid_value, compose_reports_for). The no-locks single- threaded contract. How to add a new store. 33 — Transport. hsms::Connection read path (length+payload async chain), write path (queue + one outstanding write), timer model (5 steady_timers + per-request T3). The asio executor / strand model and why it's the right shape. secsi::Protocol as the IO- free FSM with Action / Event variants; secsi::TcpTransport as the asio adapter. Pattern repeats for E84 + GEM comm-state. 34 — Codec and SML. The four files (170 + 30 + 52 + 32 lines of header, 229 + 220 lines of impl). Item variant storage layout (11 alternatives, 16 formats, shared storage where E5 permits). encode_into recursion; decode_at with bounds checks throwing CodecError. Message wrapper. SML printer + try_parse_sml + why SML round-trips Items but not necessarily bytes. 35 — State machines and dispatch. gem::Router as a typed (stream, function) dispatch table. How an S2F41 round-trip walks through parser → store dispatch → side-effect → CEID emission → S6F11 build → spool-aware deliver. The 11 FSMs all sharing the same three-property shape (pure data table + pure FSM + observer pattern). CEID cascading from FSM transitions to wire bytes. 36 — Persistence, validation, metrics. Which 7 stores have file journals + why the others don't. Per-record file pattern (atomic rename, partial-write safe). Schema versioning + multi-version read. Multi-error YAML validator (--validate-config) + cross-file reference checks. Prometheus registry + HTTP exporter + worked metric patterns from the PVD example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-09 20:23:05 +02:00
parent 40df3067a4
commit cae98d9a7d
7 changed files with 2127 additions and 0 deletions
@@ -0,0 +1,395 @@
+# 31 — Spec-as-data and codegen
+
+← [30 Repository tour](30_repository_tour.md) | [Back to index](00_index.md) | Next: [32 Stores and the data model](32_stores_and_the_data_model.md) →
+
+The single design choice that keeps this codebase small is
+**spec-as-data**: every SEMI behavioural rule, state-transition
+table, and message body shape lives in YAML.  The C++ is the engine
+that reads them.
+
+This chapter explains:
+
+- Why spec-as-data is the right choice for SECS/GEM specifically.
+- The five YAML files that drive everything.
+- How the message catalog gets codegen'd into typed C++.
+- How transition tables and equipment dictionaries load at runtime.
+- How to add a new SVID / state / message / host command without
+  touching C++.
+
+---
+
+## Why spec-as-data here
+
+Three properties of SECS/GEM standards push toward data-driven
+implementation:
+
+1. **Per-tool variation is enormous.**  Every fab tool has its own
+   list of SVIDs, ECIDs, CEIDs, alarms.  Hardcoding even one of
+   them in C++ would mean recompiling per deployment.
+2. **The standards themselves are largely declarative.**  E30 §6.2
+   is a 5×8 transition table, not an algorithm.  E5 §9 is a
+   format-byte arithmetic.  E40 §6 is a state graph.  These map to
+   YAML cleanly.
+3. **Customers need to audit the rules.**  A fab QA team can read
+   `data/control_state.yaml` and see the transitions; they can't
+   read 600 lines of `if/else` and trust it the same way.
+
+So the rule is: **anything a vendor or customer might want to
+change without recompiling lives in YAML**.  The C++ is the
+runtime that reads it.
+
+---
+
+## The five YAML files
+
+All under [`data/`](../data/).
+
+### `messages.yaml` — the SECS-II message catalog
+
+164 entries.  Each one names a SECS-II message (SxFy) and describes
+its body's typed shape.  Used at **build time** by codegen.
+
+Excerpt:
+
+```yaml
+- id: S1F1
+  stream: 1
+  function: 1
+  w: true
+  builder: s1f1
+  parser: parse_s1f1
+  body: none
+
+- id: S1F2
+  stream: 1
+  function: 2
+  w: false
+  builder: s1f2
+  parser: parse_s1f2
+  body:
+    kind: list
+    struct_name: OnlineIdentification
+    fields:
+      - {name: mdln,    shape: {kind: scalar, item_type: ASCII}}
+      - {name: softrev, shape: {kind: scalar, item_type: ASCII}}
+```
+
+The `body` field is the codegen's input.  It supports:
+
+- **`none`** — header-only, no body.
+- **`scalar`** — one Item; the codegen picks an appropriate C++
+  parameter type from `item_type` (`ASCII`, `U1`–`U8`, etc.).
+- **`list`** — fixed-arity `<L,k>` with named fields.  Optionally
+  generates a `struct StructName { … }`.
+- **`list_of`** — variable-arity `<L,n>` with a uniform element
+  shape.
+
+The grammar is documented at the top of
+[`tools/gen_messages.py`](../tools/gen_messages.py) and at the top
+of [`data/messages.yaml`](../data/messages.yaml).
+
+### `control_state.yaml` — the E30 §6.2 control state transition table
+
+```yaml
+transitions:
+  - {from: EquipmentOffline, on: operator_switch_online, to: AttemptOnline, then: OnlineRemote}
+  - {from: OnlineRemote,     on: host_request_offline,   to: HostOffline, ack: Accept}
+  - {from: OnlineLocal,      on: host_request_remote,    ack: NotAccept}
+```
+
+Loaded at **runtime** by `config::load_control_state_table`.  The
+default table — used by tests when no YAML is given —
+mirrors this file exactly (in
+`ControlTransitionTable::default_table()`).
+
+### `process_job_state.yaml` — the E40 PJ transition table
+
+Same shape as control_state.yaml but for PJs.  Drives
+`ProcessJobStateMachine`.
+
+### `control_job_state.yaml` — the E94 CJ transition table
+
+Same shape for CJs.  Drives `ControlJobStateMachine`.
+
+### `equipment.yaml` — the demo equipment data dictionary
+
+Excerpt:
+
+```yaml
+device:
+  mdln: "SECS-GEM Demo Equipment"
+  softrev: "1.0.0"
+  capabilities: [Establish, OnLine, …]
+
+svids:
+  - {id: 1, name: ControlState, units: "", type: ASCII, value: ""}
+  - {id: 2, name: Clock, units: "", type: ASCII, value: ""}
+  - {id: 3, name: WaferCounter, units: "wafers", type: U4, value: 0}
+
+ecids:
+  - {id: 100, name: T3, units: "s", type: U4, value: 45, min: 1, max: 600}
+
+ceids:
+  - {id: 100, name: ControlStateChange}
+  - {id: 300, name: ProcessStarted}
+
+alarms:
+  - {id: 1, alcd: 0x84, text: "Chamber pressure above threshold"}
+
+host_commands:
+  - {name: START, ack: Accept, emit_ceid: 300}
+  - {name: FAULT, ack: Accept, set_alarm: 1}
+```
+
+Loaded at startup by `config::load_equipment`.  Every key under
+this YAML maps to a typed struct in `config::EquipmentDescriptor`.
+
+`examples/pvd_tool/equipment.yaml` is a more realistic version
+with 29 SVIDs, 7 ECIDs, 21 CEIDs, 12 alarms.
+
+---
+
+## The codegen pass
+
+`messages.yaml` is too large and too repetitive to write by hand —
+164 messages × (builder + parser + struct + tests) would be ~5 k
+lines of boilerplate.  Instead, `tools/gen_messages.py` reads the
+YAML at build time and emits one inline header:
+**`build/generated/secsgem/gem/messages.hpp`**.
+
+### What gets generated
+
+Per message, the codegen emits:
+
+```cpp
+namespace secsgem::gem::messages {
+
+// Optional struct if body has `struct_name`.
+struct OnlineIdentification {
+  std::string mdln;
+  std::string softrev;
+  bool operator==(const OnlineIdentification&) const = default;
+};
+
+// Builder: takes typed params, returns a secs2::Message.
+inline secs2::Message s1f1();
+inline secs2::Message s1f2(const std::string& mdln, const std::string& softrev);
+
+// Parser: takes a Message body, returns std::optional<Struct> (or the
+// primitive type for scalar bodies).
+inline std::optional<OnlineIdentification> parse_s1f2(const secs2::Item& body);
+
+}  // namespace
+```
+
+For ~160 named messages, the generated header is ~3 500 lines, all
+inline.  Tests in
+[`tests/test_messages.cpp`](../tests/test_messages.cpp) (82 cases)
+exercise every builder + parser round-trip.
+
+### How CMake invokes it
+
+CMakeLists.txt has a custom command:
+
+```cmake
+add_custom_command(
+  OUTPUT  ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp
+  COMMAND ${Python3_EXECUTABLE}
+          ${CMAKE_SOURCE_DIR}/tools/gen_messages.py
+          ${CMAKE_SOURCE_DIR}/data/messages.yaml
+          ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp
+  DEPENDS ${CMAKE_SOURCE_DIR}/data/messages.yaml
+          ${CMAKE_SOURCE_DIR}/tools/gen_messages.py
+)
+```
+
+Re-runs on `data/messages.yaml` edits *or* on
+`tools/gen_messages.py` edits.  Generated header goes into a
+sibling include directory so the library can include it as
+`#include "secsgem/gem/messages.hpp"`.
+
+### Why Python rather than templates / constexpr
+
+Three reasons:
+
+1. **YAML parsing** — full grammar matters and `PyYAML` is more
+   reliable than yaml-cpp at parse-time gymnastics.
+2. **Code shape control** — the generated C++ is easier to read
+   when generated by a textual templater than by C++ metaprogramming.
+3. **Debuggability** — a customer who wants to see "what code is
+   actually being run for S2F33" can `grep` the generated header.
+   No mystery types, no instantiation chains.
+
+The codegen is ~388 lines of Python; the input grammar is
+documented at its top.
+
+---
+
+## Runtime loading
+
+The other four YAMLs (`control_state`, `process_job_state`,
+`control_job_state`, `equipment`) load at runtime, not build time.
+The same loader handles all of them:
+
+```cpp
+// include/secsgem/config/loader.hpp
+namespace secsgem::config {
+  EquipmentDescriptor    load_equipment(const std::string& path);
+  ControlStateConfig     load_control_state_table(const std::string& path);
+  ProcessJobStateConfig  load_process_job_state(const std::string& path);
+  ControlJobStateConfig  load_control_job_state(const std::string& path);
+}
+```
+
+Each `load_*` returns a typed config struct on success or throws on
+malformed YAML.  Throwing is OK because YAML loading happens once
+at startup — before binding the port — so a malformed file fails
+the process up front.
+
+---
+
+## The `--validate-config` pass
+
+YAML loaders that throw on first error are unfriendly: customers
+often have multiple typos in a new equipment.yaml.  The codebase
+ships a multi-error validator:
+
+```cpp
+// include/secsgem/config/validate.hpp
+class ConfigValidator {
+public:
+  void validate_equipment(const std::string& path);
+  void validate_control_state(const std::string& path);
+  // ...
+
+  bool has_errors() const;
+  void format_issues_to(std::ostream&, …) const;
+};
+```
+
+It tries to load each file, accumulates *every* issue it can find,
+and prints them all.  Then exits 0 or 1.
+
+Invoke via:
+
+```bash
+secs_server --validate-config \
+    --config data/equipment.yaml \
+    --state-table data/control_state.yaml \
+    --pj-state-table data/process_job_state.yaml \
+    --cj-state-table data/control_job_state.yaml
+```
+
+This is proof #5 in [PROOFS.md](PROOFS.md) — runs in CI to
+guarantee every shipped YAML is structurally + referentially
+sound.
+
+Tests:
+[`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp)
+(8 cases — every category of validation issue).
+
+---
+
+## How to add a capability without C++
+
+The point of spec-as-data is that **adding behaviour almost never
+requires a C++ change**.
+
+### New SVID
+
+```yaml
+# data/equipment.yaml
+svids:
+  - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}
+```
+
+Restart.  Done.  Host can now read SVID 4 via S1F3.
+
+### New CEID with linked report
+
+```yaml
+# data/equipment.yaml
+ceids:
+  - {id: 350, name: ChamberTempHigh}
+
+events:
+  default_reports:
+    - {ceid: 350, vids: [4]}
+```
+
+Restart.  Done.  When the EAP fires CEID 350, the report carries
+SVID 4 automatically.
+
+### New host command
+
+```yaml
+host_commands:
+  - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}
+```
+
+Restart.  Done.  Host sends `S2F41(RCMD=VENT)` → ACK=Accept,
+CEID 400 fires, ALID 2 set.
+
+### New control-state transition
+
+```yaml
+# data/control_state.yaml
+transitions:
+  - {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept}
+```
+
+Restart.  Done.
+
+### New SECS-II message
+
+```yaml
+# data/messages.yaml
+- id: S6F30
+  stream: 6
+  function: 30
+  w: true
+  builder: s6f30_request
+  parser: parse_s6f30
+  body:
+    kind: list
+    struct_name: TempQuery
+    fields:
+      - {name: vid, shape: {kind: scalar, item_type: U4}}
+```
+
+`docker compose run --rm builder` regenerates `messages.hpp`.  A
+new `s6f30_request(uint32_t vid)` builder and a `parse_s6f30(item)
+→ std::optional<TempQuery>` parser appear.  Now the *handler* is
+still C++ — `gem::Router::on(6, 30, ...)` — because the side-effect
+of "host asked for the temperature" needs application logic.
+
+---
+
+## When spec-as-data isn't the right fit
+
+Three categories that *do* need C++:
+
+1. **Application logic** — what an alarm threshold actually is,
+   how a recipe step gets executed.  No YAML schema can express
+   "vent the chamber if pressure > 1 Torr."
+2. **State-machine actions** — when a CJ transitions to Executing,
+   *which* PJ to select next isn't a table entry; it's an
+   algorithm.
+3. **External integrations** — talking to a PLC, reading a sensor,
+   driving a robot.  Hardware bindings are vendor-specific code.
+
+The codebase draws the line **at the message catalog and the
+transition tables**.  Everything below (codec, transport) is fixed
+C++.  Everything above (application wiring) is per-EAP C++.
+Everything between (data dictionary + state model) is YAML.
+
+---
+
+## Where to go next
+
+You now know how the YAML drives the runtime.  The next chapter
+gets concrete about the **stores** — the per-domain bundles
+(SVIDs, CEIDs, alarms, carriers, …) that the YAML populates and
+the Router handlers operate over.
+
+Next: [→ 32 Stores and the data model](32_stores_and_the_data_model.md)