# 31 — Spec-as-data and codegen ← [30 Repository tour](30_repository_tour.md) | [Back to index](00_index.md) | Next: [32 Stores and the data model](32_stores_and_the_data_model.md) → The single design choice that keeps this codebase small is **spec-as-data**: every SEMI behavioural rule, state-transition table, and message body shape lives in YAML. The C++ is the engine that reads them. This chapter explains: - Why spec-as-data is the right choice for SECS/GEM specifically. - The five YAML files that drive everything. - How the message catalog gets codegen'd into typed C++. - How transition tables and equipment dictionaries load at runtime. - How to add a new SVID / state / message / host command without touching C++. --- ## Why spec-as-data here Three properties of SECS/GEM standards push toward data-driven implementation: 1. **Per-tool variation is enormous.** Every fab tool has its own list of SVIDs, ECIDs, CEIDs, alarms. Hardcoding even one of them in C++ would mean recompiling per deployment. 2. **The standards themselves are largely declarative.** E30 §6.2 is a 5×8 transition table, not an algorithm. E5 §9 is a format-byte arithmetic. E40 §6 is a state graph. These map to YAML cleanly. 3. **Customers need to audit the rules.** A fab QA team can read `data/control_state.yaml` and see the transitions; they can't read 600 lines of `if/else` and trust it the same way. So the rule is: **anything a vendor or customer might want to change without recompiling lives in YAML**. The C++ is the runtime that reads it. --- ## The five YAML files All under [`data/`](../data/). ### `messages.yaml` — the SECS-II message catalog 164 entries. Each one names a SECS-II message (SxFy) and describes its body's typed shape. Used at **build time** by codegen. Excerpt: ```yaml - id: S1F1 stream: 1 function: 1 w: true builder: s1f1 parser: parse_s1f1 body: none - id: S1F2 stream: 1 function: 2 w: false builder: s1f2 parser: parse_s1f2 body: kind: list struct_name: OnlineIdentification fields: - {name: mdln, shape: {kind: scalar, item_type: ASCII}} - {name: softrev, shape: {kind: scalar, item_type: ASCII}} ``` The `body` field is the codegen's input. It supports: - **`none`** — header-only, no body. - **`scalar`** — one Item; the codegen picks an appropriate C++ parameter type from `item_type` (`ASCII`, `U1`–`U8`, etc.). - **`list`** — fixed-arity `` with named fields. Optionally generates a `struct StructName { … }`. - **`list_of`** — variable-arity `` with a uniform element shape. The grammar is documented at the top of [`tools/gen_messages.py`](../tools/gen_messages.py) and at the top of [`data/messages.yaml`](../data/messages.yaml). ### `control_state.yaml` — the E30 §6.2 control state transition table ```yaml transitions: - {from: EquipmentOffline, on: operator_switch_online, to: AttemptOnline, then: OnlineRemote} - {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept} - {from: OnlineLocal, on: host_request_remote, ack: NotAccept} ``` Loaded at **runtime** by `config::load_control_state_table`. The default table — used by tests when no YAML is given — mirrors this file exactly (in `ControlTransitionTable::default_table()`). ### `process_job_state.yaml` — the E40 PJ transition table Same shape as control_state.yaml but for PJs. Drives `ProcessJobStateMachine`. ### `control_job_state.yaml` — the E94 CJ transition table Same shape for CJs. Drives `ControlJobStateMachine`. ### `equipment.yaml` — the demo equipment data dictionary Excerpt: ```yaml device: mdln: "SECS-GEM Demo Equipment" softrev: "1.0.0" capabilities: [Establish, OnLine, …] svids: - {id: 1, name: ControlState, units: "", type: ASCII, value: ""} - {id: 2, name: Clock, units: "", type: ASCII, value: ""} - {id: 3, name: WaferCounter, units: "wafers", type: U4, value: 0} ecids: - {id: 100, name: T3, units: "s", type: U4, value: 45, min: 1, max: 600} ceids: - {id: 100, name: ControlStateChange} - {id: 300, name: ProcessStarted} alarms: - {id: 1, alcd: 0x84, text: "Chamber pressure above threshold"} host_commands: - {name: START, ack: Accept, emit_ceid: 300} - {name: FAULT, ack: Accept, set_alarm: 1} ``` Loaded at startup by `config::load_equipment`. Every key under this YAML maps to a typed struct in `config::EquipmentDescriptor`. `examples/pvd_tool/equipment.yaml` is a more realistic version with 29 SVIDs, 7 ECIDs, 21 CEIDs, 12 alarms. --- ## The codegen pass `messages.yaml` is too large and too repetitive to write by hand — 164 messages × (builder + parser + struct + tests) would be ~5 k lines of boilerplate. Instead, `tools/gen_messages.py` reads the YAML at build time and emits one inline header: **`build/generated/secsgem/gem/messages.hpp`**. ### What gets generated Per message, the codegen emits: ```cpp namespace secsgem::gem::messages { // Optional struct if body has `struct_name`. struct OnlineIdentification { std::string mdln; std::string softrev; bool operator==(const OnlineIdentification&) const = default; }; // Builder: takes typed params, returns a secs2::Message. inline secs2::Message s1f1(); inline secs2::Message s1f2(const std::string& mdln, const std::string& softrev); // Parser: takes a Message body, returns std::optional (or the // primitive type for scalar bodies). inline std::optional parse_s1f2(const secs2::Item& body); } // namespace ``` For ~160 named messages, the generated header is ~3 500 lines, all inline. Tests in [`tests/test_messages.cpp`](../tests/test_messages.cpp) (82 cases) exercise every builder + parser round-trip. ### How CMake invokes it CMakeLists.txt has a custom command: ```cmake add_custom_command( OUTPUT ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/tools/gen_messages.py ${CMAKE_SOURCE_DIR}/data/messages.yaml ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp DEPENDS ${CMAKE_SOURCE_DIR}/data/messages.yaml ${CMAKE_SOURCE_DIR}/tools/gen_messages.py ) ``` Re-runs on `data/messages.yaml` edits *or* on `tools/gen_messages.py` edits. Generated header goes into a sibling include directory so the library can include it as `#include "secsgem/gem/messages.hpp"`. ### Why Python rather than templates / constexpr Three reasons: 1. **YAML parsing** — full grammar matters and `PyYAML` is more reliable than yaml-cpp at parse-time gymnastics. 2. **Code shape control** — the generated C++ is easier to read when generated by a textual templater than by C++ metaprogramming. 3. **Debuggability** — a customer who wants to see "what code is actually being run for S2F33" can `grep` the generated header. No mystery types, no instantiation chains. The codegen is ~388 lines of Python; the input grammar is documented at its top. --- ## Runtime loading The other four YAMLs (`control_state`, `process_job_state`, `control_job_state`, `equipment`) load at runtime, not build time. The same loader handles all of them: ```cpp // include/secsgem/config/loader.hpp namespace secsgem::config { EquipmentDescriptor load_equipment(const std::string& path); ControlStateConfig load_control_state_table(const std::string& path); ProcessJobStateConfig load_process_job_state(const std::string& path); ControlJobStateConfig load_control_job_state(const std::string& path); } ``` Each `load_*` returns a typed config struct on success or throws on malformed YAML. Throwing is OK because YAML loading happens once at startup — before binding the port — so a malformed file fails the process up front. --- ## The `--validate-config` pass YAML loaders that throw on first error are unfriendly: customers often have multiple typos in a new equipment.yaml. The codebase ships a multi-error validator: ```cpp // include/secsgem/config/validate.hpp class ConfigValidator { public: void validate_equipment(const std::string& path); void validate_control_state(const std::string& path); // ... bool has_errors() const; void format_issues_to(std::ostream&, …) const; }; ``` It tries to load each file, accumulates *every* issue it can find, and prints them all. Then exits 0 or 1. Invoke via: ```bash secs_server --validate-config \ --config data/equipment.yaml \ --state-table data/control_state.yaml \ --pj-state-table data/process_job_state.yaml \ --cj-state-table data/control_job_state.yaml ``` This is proof #5 in [PROOFS.md](PROOFS.md) — runs in CI to guarantee every shipped YAML is structurally + referentially sound. Tests: [`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp) (8 cases — every category of validation issue). --- ## How to add a capability without C++ The point of spec-as-data is that **adding behaviour almost never requires a C++ change**. ### New SVID ```yaml # data/equipment.yaml svids: - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25} ``` Restart. Done. Host can now read SVID 4 via S1F3. ### New CEID with linked report ```yaml # data/equipment.yaml ceids: - {id: 350, name: ChamberTempHigh} events: default_reports: - {ceid: 350, vids: [4]} ``` Restart. Done. When the EAP fires CEID 350, the report carries SVID 4 automatically. ### New host command ```yaml host_commands: - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2} ``` Restart. Done. Host sends `S2F41(RCMD=VENT)` → ACK=Accept, CEID 400 fires, ALID 2 set. ### New control-state transition ```yaml # data/control_state.yaml transitions: - {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept} ``` Restart. Done. ### New SECS-II message ```yaml # data/messages.yaml - id: S6F30 stream: 6 function: 30 w: true builder: s6f30_request parser: parse_s6f30 body: kind: list struct_name: TempQuery fields: - {name: vid, shape: {kind: scalar, item_type: U4}} ``` `docker compose run --rm builder` regenerates `messages.hpp`. A new `s6f30_request(uint32_t vid)` builder and a `parse_s6f30(item) → std::optional` parser appear. Now the *handler* is still C++ — `gem::Router::on(6, 30, ...)` — because the side-effect of "host asked for the temperature" needs application logic. --- ## When spec-as-data isn't the right fit Three categories that *do* need C++: 1. **Application logic** — what an alarm threshold actually is, how a recipe step gets executed. No YAML schema can express "vent the chamber if pressure > 1 Torr." 2. **State-machine actions** — when a CJ transitions to Executing, *which* PJ to select next isn't a table entry; it's an algorithm. 3. **External integrations** — talking to a PLC, reading a sensor, driving a robot. Hardware bindings are vendor-specific code. The codebase draws the line **at the message catalog and the transition tables**. Everything below (codec, transport) is fixed C++. Everything above (application wiring) is per-EAP C++. Everything between (data dictionary + state model) is YAML. --- ## Where to go next You now know how the YAML drives the runtime. The next chapter gets concrete about the **stores** — the per-domain bundles (SVIDs, CEIDs, alarms, carriers, …) that the YAML populates and the Router handlers operate over. Next: [→ 32 Stores and the data model](32_stores_and_the_data_model.md)