secs-gem/docs/31_spec_as_data_and_codegen.md

# 31 — Spec-as-data and codegen

← [30 Repository tour](30_repository_tour.md) | [Back to index](00_index.md) | Next: [32 Stores and the data model](32_stores_and_the_data_model.md) →

The single design choice that keeps this codebase small is
**spec-as-data**: every SEMI behavioural rule, state-transition
table, and message body shape lives in YAML.  The C++ is the engine
that reads them.

This chapter explains:

- Why spec-as-data is the right choice for SECS/GEM specifically.
- The five YAML files that drive everything.
- How the message catalog gets codegen'd into typed C++.
- How transition tables and equipment dictionaries load at runtime.
- How to add a new SVID / state / message / host command without
  touching C++.

---

## Why spec-as-data here

Three properties of SECS/GEM standards push toward data-driven
implementation:

1. **Per-tool variation is enormous.**  Every fab tool has its own
   list of SVIDs, ECIDs, CEIDs, alarms.  Hardcoding even one of
   them in C++ would mean recompiling per deployment.
2. **The standards themselves are largely declarative.**  E30 §6.2
   is a 5×8 transition table, not an algorithm.  E5 §9 is a
   format-byte arithmetic.  E40 §6 is a state graph.  These map to
   YAML cleanly.
3. **Customers need to audit the rules.**  A fab QA team can read
   `data/control_state.yaml` and see the transitions; they can't
   read 600 lines of `if/else` and trust it the same way.

So the rule is: **anything a vendor or customer might want to
change without recompiling lives in YAML**.  The C++ is the
runtime that reads it.

---

## The five YAML files

All under [`data/`](../data/).

### `messages.yaml` — the SECS-II message catalog

164 entries.  Each one names a SECS-II message (SxFy) and describes
its body's typed shape.  Used at **build time** by codegen.

Excerpt:

```yaml
- id: S1F1
  stream: 1
  function: 1
  w: true
  builder: s1f1
  parser: parse_s1f1
  body: none

- id: S1F2
  stream: 1
  function: 2
  w: false
  builder: s1f2
  parser: parse_s1f2
  body:
    kind: list
    struct_name: OnlineIdentification
    fields:
      - {name: mdln,    shape: {kind: scalar, item_type: ASCII}}
      - {name: softrev, shape: {kind: scalar, item_type: ASCII}}
```

The `body` field is the codegen's input.  It supports:

- **`none`** — header-only, no body.
- **`scalar`** — one Item; the codegen picks an appropriate C++
  parameter type from `item_type` (`ASCII`, `U1`–`U8`, etc.).
- **`list`** — fixed-arity `<L,k>` with named fields.  Optionally
  generates a `struct StructName { … }`.
- **`list_of`** — variable-arity `<L,n>` with a uniform element
  shape.

The grammar is documented at the top of
[`tools/gen_messages.py`](../tools/gen_messages.py) and at the top
of [`data/messages.yaml`](../data/messages.yaml).

### `control_state.yaml` — the E30 §6.2 control state transition table

```yaml
transitions:
  - {from: EquipmentOffline, on: operator_switch_online, to: AttemptOnline, then: OnlineRemote}
  - {from: OnlineRemote,     on: host_request_offline,   to: HostOffline, ack: Accept}
  - {from: OnlineLocal,      on: host_request_remote,    ack: NotAccept}
```

Loaded at **runtime** by `config::load_control_state_table`.  The
default table — used by tests when no YAML is given —
mirrors this file exactly (in
`ControlTransitionTable::default_table()`).

### `process_job_state.yaml` — the E40 PJ transition table

Same shape as control_state.yaml but for PJs.  Drives
`ProcessJobStateMachine`.

### `control_job_state.yaml` — the E94 CJ transition table

Same shape for CJs.  Drives `ControlJobStateMachine`.

### `equipment.yaml` — the demo equipment data dictionary

Excerpt:

```yaml
device:
  mdln: "SECS-GEM Demo Equipment"
  softrev: "1.0.0"
  capabilities: [Establish, OnLine, …]

svids:
  - {id: 1, name: ControlState, units: "", type: ASCII, value: ""}
  - {id: 2, name: Clock, units: "", type: ASCII, value: ""}
  - {id: 3, name: WaferCounter, units: "wafers", type: U4, value: 0}

ecids:
  - {id: 100, name: T3, units: "s", type: U4, value: 45, min: 1, max: 600}

ceids:
  - {id: 100, name: ControlStateChange}
  - {id: 300, name: ProcessStarted}

alarms:
  # `category` is the lower-7 ALCD severity bitmap; bit 7 (set/clear) is
  # applied at emit time and must NOT be in YAML. `name` is an optional
  # local key for name-based APIs (the gRPC daemon / Python client).
  - {id: 1, name: pressure_high, category: 4,
     text: "Chamber pressure above threshold"}

host_commands:
  - {name: START, ack: Accept, emit_ceid: 300}
  - {name: FAULT, ack: Accept, set_alarm: 1}

# Role bindings: which of the ids above the engine's built-in behaviours
# target (control-state/clock SVID refresh, CJ state CEIDs).
roles:
  control_state_svid: 1
  clock_svid: 2
```

Loaded at startup by `config::load_equipment`.  Every key under
this YAML maps to a typed struct in `config::EquipmentDescriptor`.

`examples/pvd_tool/equipment.yaml` is a more realistic version
with 29 SVIDs, 7 ECIDs, 21 CEIDs, 12 alarms.

---

## The codegen pass

`messages.yaml` is too large and too repetitive to write by hand —
164 messages × (builder + parser + struct + tests) would be ~5 k
lines of boilerplate.  Instead, `tools/gen_messages.py` reads the
YAML at build time and emits one inline header:
**`build/generated/secsgem/gem/messages.hpp`**.

### What gets generated

Per message, the codegen emits:

```cpp
namespace secsgem::gem::messages {

// Optional struct if body has `struct_name`.
struct OnlineIdentification {
  std::string mdln;
  std::string softrev;
  bool operator==(const OnlineIdentification&) const = default;
};

// Builder: takes typed params, returns a secs2::Message.
inline secs2::Message s1f1();
inline secs2::Message s1f2(const std::string& mdln, const std::string& softrev);

// Parser: takes a Message body, returns std::optional<Struct> (or the
// primitive type for scalar bodies).
inline std::optional<OnlineIdentification> parse_s1f2(const secs2::Item& body);

}  // namespace
```

For ~160 named messages, the generated header is ~3 500 lines, all
inline.  Tests in
[`tests/test_messages.cpp`](../tests/test_messages.cpp) (82 cases)
exercise every builder + parser round-trip.

### How CMake invokes it

CMakeLists.txt has a custom command:

```cmake
add_custom_command(
  OUTPUT  ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp
  COMMAND ${Python3_EXECUTABLE}
          ${CMAKE_SOURCE_DIR}/tools/gen_messages.py
          ${CMAKE_SOURCE_DIR}/data/messages.yaml
          ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp
  DEPENDS ${CMAKE_SOURCE_DIR}/data/messages.yaml
          ${CMAKE_SOURCE_DIR}/tools/gen_messages.py
)
```

Re-runs on `data/messages.yaml` edits *or* on
`tools/gen_messages.py` edits.  Generated header goes into a
sibling include directory so the library can include it as
`#include "secsgem/gem/messages.hpp"`.

### Why Python rather than templates / constexpr

Three reasons:

1. **YAML parsing** — full grammar matters and `PyYAML` is more
   reliable than yaml-cpp at parse-time gymnastics.
2. **Code shape control** — the generated C++ is easier to read
   when generated by a textual templater than by C++ metaprogramming.
3. **Debuggability** — a customer who wants to see "what code is
   actually being run for S2F33" can `grep` the generated header.
   No mystery types, no instantiation chains.

The codegen is ~388 lines of Python; the input grammar is
documented at its top.

---

## Runtime loading

The other four YAMLs (`control_state`, `process_job_state`,
`control_job_state`, `equipment`) load at runtime, not build time.
The same loader handles all of them:

```cpp
// include/secsgem/config/loader.hpp
namespace secsgem::config {
  EquipmentDescriptor    load_equipment(const std::string& path);
  ControlStateConfig     load_control_state_table(const std::string& path);
  ProcessJobStateConfig  load_process_job_state(const std::string& path);
  ControlJobStateConfig  load_control_job_state(const std::string& path);
}
```

Each `load_*` returns a typed config struct on success or throws on
malformed YAML.  Throwing is OK because YAML loading happens once
at startup — before binding the port — so a malformed file fails
the process up front.

---

## The `--validate-config` pass

YAML loaders that throw on first error are unfriendly: customers
often have multiple typos in a new equipment.yaml.  The codebase
ships a multi-error validator:

```cpp
// include/secsgem/config/validate.hpp
class ConfigValidator {
public:
  void validate_equipment(const std::string& path);
  void validate_control_state(const std::string& path);
  // ...

  bool has_errors() const;
  void format_issues_to(std::ostream&, …) const;
};
```

It tries to load each file, accumulates *every* issue it can find,
and prints them all.  Then exits 0 or 1.

Invoke via:

```bash
secs_server --validate-config \
    --config data/equipment.yaml \
    --state-table data/control_state.yaml \
    --pj-state-table data/process_job_state.yaml \
    --cj-state-table data/control_job_state.yaml
```

This is proof #5 in [PROOFS.md](PROOFS.md) — runs in CI to
guarantee every shipped YAML is structurally + referentially
sound.

Tests:
[`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp)
(8 cases — every category of validation issue).

---

## How to add a capability without C++

The point of spec-as-data is that **adding behaviour almost never
requires a C++ change**.

### New SVID

```yaml
# data/equipment.yaml
svids:
  - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}
```

Restart.  Done.  Host can now read SVID 4 via S1F3.

### New CEID with linked report

```yaml
# data/equipment.yaml
ceids:
  - {id: 350, name: ChamberTempHigh}

events:
  default_reports:
    - {ceid: 350, vids: [4]}
```

Restart.  Done.  When the EAP fires CEID 350, the report carries
SVID 4 automatically.

### New host command

```yaml
host_commands:
  - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}
```

Restart.  Done.  Host sends `S2F41(RCMD=VENT)` → ACK=Accept,
CEID 400 fires, ALID 2 set.

### New control-state transition

```yaml
# data/control_state.yaml
transitions:
  - {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept}
```

Restart.  Done.

### New SECS-II message

```yaml
# data/messages.yaml
- id: S6F30
  stream: 6
  function: 30
  w: true
  builder: s6f30_request
  parser: parse_s6f30
  body:
    kind: list
    struct_name: TempQuery
    fields:
      - {name: vid, shape: {kind: scalar, item_type: U4}}
```

`docker compose run --rm builder` regenerates `messages.hpp`.  A
new `s6f30_request(uint32_t vid)` builder and a `parse_s6f30(item)
→ std::optional<TempQuery>` parser appear.  Now the *handler* is
still C++ — `gem::Router::on(6, 30, ...)` — because the side-effect
of "host asked for the temperature" needs application logic.

---

## When spec-as-data isn't the right fit

Three categories that *do* need C++:

1. **Application logic** — what an alarm threshold actually is,
   how a recipe step gets executed.  No YAML schema can express
   "vent the chamber if pressure > 1 Torr."
2. **State-machine actions** — when a CJ transitions to Executing,
   *which* PJ to select next isn't a table entry; it's an
   algorithm.
3. **External integrations** — talking to a PLC, reading a sensor,
   driving a robot.  Hardware bindings are vendor-specific code.

The codebase draws the line **at the message catalog and the
transition tables**.  Everything below (codec, transport) is fixed
C++.  Everything above (application wiring) is per-EAP C++.
Everything between (data dictionary + state model) is YAML.

---

## Where to go next

You now know how the YAML drives the runtime.  The next chapter
gets concrete about the **stores** — the per-domain bundles
(SVIDs, CEIDs, alarms, carriers, …) that the YAML populates and
the Router handlers operate over.

Next: [→ 32 Stores and the data model](32_stores_and_the_data_model.md)