secs-gem/docs/31_spec_as_data_and_codegen.md

31 — Spec-as-data and codegen

← 30 Repository tour | Back to index | Next: 32 Stores and the data model →

The single design choice that keeps this codebase small is spec-as-data: every SEMI behavioural rule, state-transition table, and message body shape lives in YAML. The C++ is the engine that reads them.

This chapter explains:

• Why spec-as-data is the right choice for SECS/GEM specifically.
• The five YAML files that drive everything.
• How the message catalog gets codegen'd into typed C++.
• How transition tables and equipment dictionaries load at runtime.
• How to add a new SVID / state / message / host command without touching C++.

---

Why spec-as-data here

Three properties of SECS/GEM standards push toward data-driven implementation:

1. Per-tool variation is enormous. Every fab tool has its own list of SVIDs, ECIDs, CEIDs, alarms. Hardcoding even one of them in C++ would mean recompiling per deployment.
2. The standards themselves are largely declarative. E30 §6.2 is a 5×8 transition table, not an algorithm. E5 §9 is a format-byte arithmetic. E40 §6 is a state graph. These map to YAML cleanly.
3. Customers need to audit the rules. A fab QA team can read data/control_state.yaml and see the transitions; they can't read 600 lines of if/else and trust it the same way.

So the rule is: anything a vendor or customer might want to change without recompiling lives in YAML. The C++ is the runtime that reads it.

---

The five YAML files

All under data/.

messages.yaml — the SECS-II message catalog

164 entries. Each one names a SECS-II message (SxFy) and describes its body's typed shape. Used at build time by codegen.

Excerpt:

- id: S1F1
  stream: 1
  function: 1
  w: true
  builder: s1f1
  parser: parse_s1f1
  body: none

- id: S1F2
  stream: 1
  function: 2
  w: false
  builder: s1f2
  parser: parse_s1f2
  body:
    kind: list
    struct_name: OnlineIdentification
    fields:
      - {name: mdln,    shape: {kind: scalar, item_type: ASCII}}
      - {name: softrev, shape: {kind: scalar, item_type: ASCII}}

The body field is the codegen's input. It supports:

• none — header-only, no body.
• scalar — one Item; the codegen picks an appropriate C++ parameter type from item_type (ASCII, U1–U8, etc.).
• list — fixed-arity <L,k> with named fields. Optionally generates a struct StructName { … }.
• list_of — variable-arity <L,n> with a uniform element shape.

The grammar is documented at the top of tools/gen_messages.py and at the top of data/messages.yaml.

control_state.yaml — the E30 §6.2 control state transition table

transitions:
  - {from: EquipmentOffline, on: operator_switch_online, to: AttemptOnline, then: OnlineRemote}
  - {from: OnlineRemote,     on: host_request_offline,   to: HostOffline, ack: Accept}
  - {from: OnlineLocal,      on: host_request_remote,    ack: NotAccept}

Loaded at runtime by config::load_control_state_table. The default table — used by tests when no YAML is given — mirrors this file exactly (in ControlTransitionTable::default_table()).

process_job_state.yaml — the E40 PJ transition table

Same shape as control_state.yaml but for PJs. Drives ProcessJobStateMachine.

control_job_state.yaml — the E94 CJ transition table

Same shape for CJs. Drives ControlJobStateMachine.

equipment.yaml — the demo equipment data dictionary

Excerpt:

device:
  mdln: "SECS-GEM Demo Equipment"
  softrev: "1.0.0"
  capabilities: [Establish, OnLine, …]

svids:
  - {id: 1, name: ControlState, units: "", type: ASCII, value: ""}
  - {id: 2, name: Clock, units: "", type: ASCII, value: ""}
  - {id: 3, name: WaferCounter, units: "wafers", type: U4, value: 0}

ecids:
  - {id: 100, name: T3, units: "s", type: U4, value: 45, min: 1, max: 600}

ceids:
  - {id: 100, name: ControlStateChange}
  - {id: 300, name: ProcessStarted}

alarms:
  # `category` is the lower-7 ALCD severity bitmap; bit 7 (set/clear) is
  # applied at emit time and must NOT be in YAML. `name` is an optional
  # local key for name-based APIs (the gRPC daemon / Python client).
  - {id: 1, name: pressure_high, category: 4,
     text: "Chamber pressure above threshold"}

host_commands:
  - {name: START, ack: Accept, emit_ceid: 300}
  - {name: FAULT, ack: Accept, set_alarm: 1}

# Role bindings: which of the ids above the engine's built-in behaviours
# target (control-state/clock SVID refresh, CJ state CEIDs).
roles:
  control_state_svid: 1
  clock_svid: 2

Loaded at startup by config::load_equipment. Every key under this YAML maps to a typed struct in config::EquipmentDescriptor.

examples/pvd_tool/equipment.yaml is a more realistic version with 29 SVIDs, 7 ECIDs, 21 CEIDs, 12 alarms.

---

The codegen pass

messages.yaml is too large and too repetitive to write by hand — 164 messages × (builder + parser + struct + tests) would be ~5 k lines of boilerplate. Instead, tools/gen_messages.py reads the YAML at build time and emits one inline header: build/generated/secsgem/gem/messages.hpp.

What gets generated

Per message, the codegen emits:

namespace secsgem::gem::messages {

// Optional struct if body has `struct_name`.
struct OnlineIdentification {
  std::string mdln;
  std::string softrev;
  bool operator==(const OnlineIdentification&) const = default;
};

// Builder: takes typed params, returns a secs2::Message.
inline secs2::Message s1f1();
inline secs2::Message s1f2(const std::string& mdln, const std::string& softrev);

// Parser: takes a Message body, returns std::optional<Struct> (or the
// primitive type for scalar bodies).
inline std::optional<OnlineIdentification> parse_s1f2(const secs2::Item& body);

}  // namespace

For ~160 named messages, the generated header is ~3 500 lines, all inline. Tests in tests/test_messages.cpp (82 cases) exercise every builder + parser round-trip.

How CMake invokes it

CMakeLists.txt has a custom command:

add_custom_command(
  OUTPUT  ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp
  COMMAND ${Python3_EXECUTABLE}
          ${CMAKE_SOURCE_DIR}/tools/gen_messages.py
          ${CMAKE_SOURCE_DIR}/data/messages.yaml
          ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp
  DEPENDS ${CMAKE_SOURCE_DIR}/data/messages.yaml
          ${CMAKE_SOURCE_DIR}/tools/gen_messages.py
)

Re-runs on data/messages.yaml edits or on tools/gen_messages.py edits. Generated header goes into a sibling include directory so the library can include it as #include "secsgem/gem/messages.hpp".

Why Python rather than templates / constexpr

Three reasons:

1. YAML parsing — full grammar matters and PyYAML is more reliable than yaml-cpp at parse-time gymnastics.
2. Code shape control — the generated C++ is easier to read when generated by a textual templater than by C++ metaprogramming.
3. Debuggability — a customer who wants to see "what code is actually being run for S2F33" can grep the generated header. No mystery types, no instantiation chains.

The codegen is ~388 lines of Python; the input grammar is documented at its top.

---

Runtime loading

The other four YAMLs (control_state, process_job_state, control_job_state, equipment) load at runtime, not build time. The same loader handles all of them:

// include/secsgem/config/loader.hpp
namespace secsgem::config {
  EquipmentDescriptor    load_equipment(const std::string& path);
  ControlStateConfig     load_control_state_table(const std::string& path);
  ProcessJobStateConfig  load_process_job_state(const std::string& path);
  ControlJobStateConfig  load_control_job_state(const std::string& path);
}

Each load_* returns a typed config struct on success or throws on malformed YAML. Throwing is OK because YAML loading happens once at startup — before binding the port — so a malformed file fails the process up front.

---

The --validate-config pass

YAML loaders that throw on first error are unfriendly: customers often have multiple typos in a new equipment.yaml. The codebase ships a multi-error validator:

// include/secsgem/config/validate.hpp
class ConfigValidator {
public:
  void validate_equipment(const std::string& path);
  void validate_control_state(const std::string& path);
  // ...

  bool has_errors() const;
  void format_issues_to(std::ostream&, …) const;
};

It tries to load each file, accumulates every issue it can find, and prints them all. Then exits 0 or 1.

Invoke via:

secs_server --validate-config \
    --config data/equipment.yaml \
    --state-table data/control_state.yaml \
    --pj-state-table data/process_job_state.yaml \
    --cj-state-table data/control_job_state.yaml

This is proof #5 in PROOFS.md — runs in CI to guarantee every shipped YAML is structurally + referentially sound.

Tests: tests/test_config_validate.cpp (8 cases — every category of validation issue).

---

How to add a capability without C++

The point of spec-as-data is that adding behaviour almost never requires a C++ change.

New SVID

# data/equipment.yaml
svids:
  - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}

Restart. Done. Host can now read SVID 4 via S1F3.

New CEID with linked report

# data/equipment.yaml
ceids:
  - {id: 350, name: ChamberTempHigh}

events:
  default_reports:
    - {ceid: 350, vids: [4]}

Restart. Done. When the EAP fires CEID 350, the report carries SVID 4 automatically.

New host command

host_commands:
  - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}

Restart. Done. Host sends S2F41(RCMD=VENT) → ACK=Accept, CEID 400 fires, ALID 2 set.

New control-state transition

# data/control_state.yaml
transitions:
  - {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept}

Restart. Done.

New SECS-II message

# data/messages.yaml
- id: S6F30
  stream: 6
  function: 30
  w: true
  builder: s6f30_request
  parser: parse_s6f30
  body:
    kind: list
    struct_name: TempQuery
    fields:
      - {name: vid, shape: {kind: scalar, item_type: U4}}

docker compose run --rm builder regenerates messages.hpp. A new s6f30_request(uint32_t vid) builder and a parse_s6f30(item) → std::optional<TempQuery> parser appear. Now the handler is still C++ — gem::Router::on(6, 30, ...) — because the side-effect of "host asked for the temperature" needs application logic.

---

When spec-as-data isn't the right fit

Three categories that do need C++:

1. Application logic — what an alarm threshold actually is, how a recipe step gets executed. No YAML schema can express "vent the chamber if pressure > 1 Torr."
2. State-machine actions — when a CJ transitions to Executing, which PJ to select next isn't a table entry; it's an algorithm.
3. External integrations — talking to a PLC, reading a sensor, driving a robot. Hardware bindings are vendor-specific code.

The codebase draws the line at the message catalog and the transition tables. Everything below (codec, transport) is fixed C++. Everything above (application wiring) is per-EAP C++. Everything between (data dictionary + state model) is YAML.

---

Where to go next

You now know how the YAML drives the runtime. The next chapter gets concrete about the stores — the per-domain bundles (SVIDs, CEIDs, alarms, carriers, …) that the YAML populates and the Router handlers operate over.

Next: → 32 Stores and the data model