diff --git a/docs/30_repository_tour.md b/docs/30_repository_tour.md new file mode 100644 index 0000000..6c83888 --- /dev/null +++ b/docs/30_repository_tour.md @@ -0,0 +1,231 @@ +# 30 — Repository tour + +← [19 E42 + E148 + S9 — Misc](19_e42_e148_s9_misc.md) | [Back to index](00_index.md) | Next: [31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md) → + +You've seen what every SEMI standard *does*. Now we shift to how +this **codebase** is laid out. This chapter answers: when you +`git clone` this repo, what are you looking at? + +The repo is small — about 15 k lines of C++ + tests + tooling. It +fits in your head with a little patience. By the end of this +chapter you'll know: + +- What each top-level directory contains. +- Which binaries get built. +- The dependency graph between modules. +- How the build system finds and links them. + +--- + +## Top-level layout + +``` +secs-gem/ +├── README.md One-page project summary. +├── LICENSE Proprietary terms. +├── CMakeLists.txt Build config (CMake 3.16+, single file). +├── Dockerfile Ubuntu 24.04 + g++-13 + libasio + yaml-cpp. +├── docker-compose.yml Multi-container demo wiring. +├── .gitea/workflows/ci.yml CI pipeline. +│ +├── include/secsgem/ Public headers. All API here. +│ ├── secs2/ E5 codec + SML. +│ ├── hsms/ E37 transport (TCP + framing). +│ ├── secsi/ E4 transport (FSM + TCP tunnel). +│ ├── config/ YAML loader + multi-error validator. +│ ├── metrics/ Prometheus exporter. +│ ├── endpoint.hpp asio::ip::tcp::endpoint factory. +│ └── gem/ E30 + every GEM 300 standard. +│ ├── store/ Per-domain bundles (SVIDs, alarms, …). +│ └── *.hpp State machines + composers. +│ +├── src/ Implementations. Mirrors include/. +│ ├── secs2/{codec,sml}.cpp +│ ├── hsms/{header,connection}.cpp +│ ├── secsi/{header,block,protocol,tcp_transport}.cpp +│ ├── config/... +│ ├── gem/... +│ └── endpoint.cpp +│ +├── apps/ Standalone binaries. +│ ├── secs_server.cpp Passive equipment (demo + integration target). +│ ├── secs_client.cpp Active host driving the demo flow. +│ ├── secs_conformance.cpp 47-check wire-level conformance harness. +│ ├── secs_interop_probe.cpp Probe against secsgem-py passive equip. +│ ├── secs_bench.cpp Throughput / latency / memory bench. +│ ├── fuzz_secs2_decode.cpp libFuzzer harness for secs2::decode. +│ └── fuzz_sml_parse.cpp libFuzzer harness for try_parse_sml. +│ +├── tests/ doctest unit + integration tests. +│ └── test_*.cpp 50 files, 445 cases, 2753 assertions. +│ +├── data/ YAML configs (the spec-as-data). +│ ├── messages.yaml SECS-II message catalog (164 msgs). +│ ├── control_state.yaml E30 §6.2 transition table. +│ ├── process_job_state.yaml E40 transition table. +│ ├── control_job_state.yaml E94 transition table. +│ └── equipment.yaml Demo SVIDs/ECIDs/CEIDs/alarms/recipes. +│ +├── tools/ Build-time scripts. +│ └── generate_messages.py Codegen: messages.yaml → messages.hpp. +│ +├── interop/ External-validator harnesses. +│ ├── README.md Harness-by-harness detail. +│ ├── host_vs_cpp_server.py secsgem-py active host driving us. +│ ├── passive_equipment.py secsgem-py passive equipment for us to drive. +│ ├── raw_gem300_harness.py Raw S3/S14/S16/S12 round-trip. +│ ├── tshark_validate.sh pcap + tshark HSMS dissector check. +│ ├── secs4j_validate.sh secs4java8 (Java) cross-validation. +│ └── secs4j/ Dockerfile + harness for secs4java8. +│ +├── examples/ +│ └── pvd_tool/ Worked vendor example: fictional PVD tool. +│ ├── README.md What the example shows. +│ ├── equipment.yaml Realistic SVIDs/ECIDs/CEIDs/alarms/recipes. +│ └── main.cpp Sensor sim, recipe runner, alarm monitor. +│ +└── docs/ This guide + reference docs. + ├── 00_index.md The series TOC. + ├── 01–51_*.md Tutorial chapters. + ├── ARCHITECTURE.md One-page architecture overview. + ├── COMPLIANCE.md Per-capability audit. + ├── INTEGRATION.md Vendor-side production deploy. + ├── PROOFS.md 8 commands proving feature-completeness. + ├── VERIFICATION.md External-validator test plan. + ├── BENCHMARKS.md Performance envelope. + ├── MES_INTEROP.md Commercial-MES day-1 punch list. + ├── SECURITY.md nftables / stunnel / minisign configs. + ├── GLOSSARY.md SEMI vocabulary cheat sheet. + └── FAQ.md Canonical answers. +``` + +--- + +## The dependency graph + +``` + data/*.yaml + │ + ┌─────────────┼──────────────────┐ + │ (codegen) │ (runtime load) │ + ▼ ▼ ▼ + generated/messages.hpp config::loader + │ │ + └──────────► gem::EquipmentDataModel + │ + │ used by + ▼ + gem::Router + │ + │ wraps + ▼ + secs2::Message ◄─── codec / SML + │ + │ over + ▼ + hsms::Connection / secsi::TcpTransport + │ + ▼ + TCP socket +``` + +Read it bottom-up: a TCP socket carries bytes; `hsms::Connection` +frames them into `secs2::Message`s; `gem::Router` dispatches by +`(stream, function)` to handlers; handlers read/write +`EquipmentDataModel`; the model composes per-domain stores; the +stores were built from the YAML at startup. + +No layer ever calls *up* the graph. `secs2::Item` has no idea +HSMS exists. `hsms::Connection` doesn't know about CEIDs. +`gem::Router` doesn't know whether the bytes came over HSMS or +SECS-I. Strict layering is what keeps the codebase small. + +--- + +## The binaries + +Built by [`CMakeLists.txt`](../CMakeLists.txt) (one file, ~250 +lines). Each binary lives in `build/` after `cmake --build`. + +| Binary | Source | What it does | +|----------------------|-----------------------------------------------------------------|-------------------------------------------------------| +| `secs_server` | [`apps/secs_server.cpp`](../apps/secs_server.cpp) | Passive equipment. Listens on TCP, dispatches via Router. | +| `secs_client` | [`apps/secs_client.cpp`](../apps/secs_client.cpp) | Active host. Drives ~24 transactions in the demo. | +| `secs_conformance` | [`apps/secs_conformance.cpp`](../apps/secs_conformance.cpp) | 47 wire-level conformance checks against a live server. | +| `secs_interop_probe` | [`apps/secs_interop_probe.cpp`](../apps/secs_interop_probe.cpp) | Active host probing a secsgem-py passive equipment. | +| `secs_bench` | [`apps/secs_bench.cpp`](../apps/secs_bench.cpp) | Throughput / latency / memory harness. | +| `secsgem_tests` | All `tests/*.cpp` | The 445-case doctest binary. | +| `fuzz_secs2_decode` | [`apps/fuzz_secs2_decode.cpp`](../apps/fuzz_secs2_decode.cpp) | libFuzzer (clang only, opt-in `-DSECSGEM_FUZZ=ON`). | +| `fuzz_sml_parse` | [`apps/fuzz_sml_parse.cpp`](../apps/fuzz_sml_parse.cpp) | libFuzzer for the SML parser. | + +A worked example binary `pvd_tool` (from `examples/pvd_tool/`) is +also built by the same `CMakeLists.txt` when the example is +included. + +--- + +## How the build system finds everything + +`CMakeLists.txt` does five things in order: + +1. **Pull in dependencies** — `find_package(Threads)`, + `find_package(yaml-cpp)`, `FetchContent` for doctest. Standalone + Asio is header-only (no link step). +2. **Run codegen** — invokes `tools/generate_messages.py` to turn + `data/messages.yaml` into `build/generated/secsgem/gem/messages.hpp`. + Listed as a custom command so it re-runs when `messages.yaml` + changes. +3. **Build the library** — `add_library(secsgem ...)` with every + source under `src/` plus the generated header. +4. **Build the apps** — one `add_executable` per `apps/*.cpp`, + each linking against `secsgem`. +5. **Build the tests** — `add_executable(secsgem_tests ...)` with + every `tests/*.cpp`, linked against doctest + `secsgem`. + +Build flags: + +- **`-DSECSGEM_TSAN=ON`** — adds `-fsanitize=thread` to a + separate build dir. CI runs this lane. +- **`-DSECSGEM_FUZZ=ON`** — requires clang; adds libFuzzer + ASan + + UBSan; builds the two fuzz harnesses. + +Everything else (Release / Debug, parallelism, output dirs) is +standard CMake. + +--- + +## Test layout + +50 test files; 445 test cases; 2 753 assertions. One file per +concern. Naming is `test_.cpp` consistently: + +- `test_secs2.cpp`, `test_e5_kat.cpp`, `test_sml.cpp`, + `test_messages.cpp` — codec. +- `test_hsms*.cpp` (5 files), `test_secsi*.cpp` (3 files) — transport. +- `test_control_state.cpp`, `test_communication_state.cpp`, + `test_data_model.cpp`, `test_host_handler.cpp`, `test_loader.cpp`, + `test_config_validate.cpp` — E30. +- `test_process_jobs.cpp`, `test_control_jobs.cpp`, + `test_carriers.cpp`, `test_substrates.cpp`, `test_ept.cpp`, + `test_modules.cpp`, `test_cem_objects.cpp`, `test_e84*.cpp`, + `test_e42_formatted_pp.cpp` — GEM 300. +- `test_*_persistence.cpp` (4) — file-backed journal. +- `test_robustness_fuzz.cpp` — randomized property test. +- `test_thread_safety.cpp` — TSan-validated single-threaded contract. +- `test_metrics_prometheus.cpp` — Prometheus exporter. +- `test_wire_ceid_emission.cpp` — CEID firings observed on a real socket. +- `test_live_gem300.cpp`, `test_gem300_scenario.cpp` — multi-FSM cascades. + +Full per-standard breakdown: +[`docs/PROOFS.md`](PROOFS.md) "Per-standard test coverage" table. + +--- + +## Where to go next + +Now that you know what's where, the next chapter explains the +*philosophy* that makes the codebase this small: the **spec-as-data** +principle, and how the YAML files + codegen + runtime loader work +together so adding a new SVID / state / message rarely requires C++. + +Next: [→ 31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md) diff --git a/docs/31_spec_as_data_and_codegen.md b/docs/31_spec_as_data_and_codegen.md new file mode 100644 index 0000000..d55a15e --- /dev/null +++ b/docs/31_spec_as_data_and_codegen.md @@ -0,0 +1,395 @@ +# 31 — Spec-as-data and codegen + +← [30 Repository tour](30_repository_tour.md) | [Back to index](00_index.md) | Next: [32 Stores and the data model](32_stores_and_the_data_model.md) → + +The single design choice that keeps this codebase small is +**spec-as-data**: every SEMI behavioural rule, state-transition +table, and message body shape lives in YAML. The C++ is the engine +that reads them. + +This chapter explains: + +- Why spec-as-data is the right choice for SECS/GEM specifically. +- The five YAML files that drive everything. +- How the message catalog gets codegen'd into typed C++. +- How transition tables and equipment dictionaries load at runtime. +- How to add a new SVID / state / message / host command without + touching C++. + +--- + +## Why spec-as-data here + +Three properties of SECS/GEM standards push toward data-driven +implementation: + +1. **Per-tool variation is enormous.** Every fab tool has its own + list of SVIDs, ECIDs, CEIDs, alarms. Hardcoding even one of + them in C++ would mean recompiling per deployment. +2. **The standards themselves are largely declarative.** E30 §6.2 + is a 5×8 transition table, not an algorithm. E5 §9 is a + format-byte arithmetic. E40 §6 is a state graph. These map to + YAML cleanly. +3. **Customers need to audit the rules.** A fab QA team can read + `data/control_state.yaml` and see the transitions; they can't + read 600 lines of `if/else` and trust it the same way. + +So the rule is: **anything a vendor or customer might want to +change without recompiling lives in YAML**. The C++ is the +runtime that reads it. + +--- + +## The five YAML files + +All under [`data/`](../data/). + +### `messages.yaml` — the SECS-II message catalog + +164 entries. Each one names a SECS-II message (SxFy) and describes +its body's typed shape. Used at **build time** by codegen. + +Excerpt: + +```yaml +- id: S1F1 + stream: 1 + function: 1 + w: true + builder: s1f1 + parser: parse_s1f1 + body: none + +- id: S1F2 + stream: 1 + function: 2 + w: false + builder: s1f2 + parser: parse_s1f2 + body: + kind: list + struct_name: OnlineIdentification + fields: + - {name: mdln, shape: {kind: scalar, item_type: ASCII}} + - {name: softrev, shape: {kind: scalar, item_type: ASCII}} +``` + +The `body` field is the codegen's input. It supports: + +- **`none`** — header-only, no body. +- **`scalar`** — one Item; the codegen picks an appropriate C++ + parameter type from `item_type` (`ASCII`, `U1`–`U8`, etc.). +- **`list`** — fixed-arity `` with named fields. Optionally + generates a `struct StructName { … }`. +- **`list_of`** — variable-arity `` with a uniform element + shape. + +The grammar is documented at the top of +[`tools/gen_messages.py`](../tools/gen_messages.py) and at the top +of [`data/messages.yaml`](../data/messages.yaml). + +### `control_state.yaml` — the E30 §6.2 control state transition table + +```yaml +transitions: + - {from: EquipmentOffline, on: operator_switch_online, to: AttemptOnline, then: OnlineRemote} + - {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept} + - {from: OnlineLocal, on: host_request_remote, ack: NotAccept} +``` + +Loaded at **runtime** by `config::load_control_state_table`. The +default table — used by tests when no YAML is given — +mirrors this file exactly (in +`ControlTransitionTable::default_table()`). + +### `process_job_state.yaml` — the E40 PJ transition table + +Same shape as control_state.yaml but for PJs. Drives +`ProcessJobStateMachine`. + +### `control_job_state.yaml` — the E94 CJ transition table + +Same shape for CJs. Drives `ControlJobStateMachine`. + +### `equipment.yaml` — the demo equipment data dictionary + +Excerpt: + +```yaml +device: + mdln: "SECS-GEM Demo Equipment" + softrev: "1.0.0" + capabilities: [Establish, OnLine, …] + +svids: + - {id: 1, name: ControlState, units: "", type: ASCII, value: ""} + - {id: 2, name: Clock, units: "", type: ASCII, value: ""} + - {id: 3, name: WaferCounter, units: "wafers", type: U4, value: 0} + +ecids: + - {id: 100, name: T3, units: "s", type: U4, value: 45, min: 1, max: 600} + +ceids: + - {id: 100, name: ControlStateChange} + - {id: 300, name: ProcessStarted} + +alarms: + - {id: 1, alcd: 0x84, text: "Chamber pressure above threshold"} + +host_commands: + - {name: START, ack: Accept, emit_ceid: 300} + - {name: FAULT, ack: Accept, set_alarm: 1} +``` + +Loaded at startup by `config::load_equipment`. Every key under +this YAML maps to a typed struct in `config::EquipmentDescriptor`. + +`examples/pvd_tool/equipment.yaml` is a more realistic version +with 29 SVIDs, 7 ECIDs, 21 CEIDs, 12 alarms. + +--- + +## The codegen pass + +`messages.yaml` is too large and too repetitive to write by hand — +164 messages × (builder + parser + struct + tests) would be ~5 k +lines of boilerplate. Instead, `tools/gen_messages.py` reads the +YAML at build time and emits one inline header: +**`build/generated/secsgem/gem/messages.hpp`**. + +### What gets generated + +Per message, the codegen emits: + +```cpp +namespace secsgem::gem::messages { + +// Optional struct if body has `struct_name`. +struct OnlineIdentification { + std::string mdln; + std::string softrev; + bool operator==(const OnlineIdentification&) const = default; +}; + +// Builder: takes typed params, returns a secs2::Message. +inline secs2::Message s1f1(); +inline secs2::Message s1f2(const std::string& mdln, const std::string& softrev); + +// Parser: takes a Message body, returns std::optional (or the +// primitive type for scalar bodies). +inline std::optional parse_s1f2(const secs2::Item& body); + +} // namespace +``` + +For ~160 named messages, the generated header is ~3 500 lines, all +inline. Tests in +[`tests/test_messages.cpp`](../tests/test_messages.cpp) (82 cases) +exercise every builder + parser round-trip. + +### How CMake invokes it + +CMakeLists.txt has a custom command: + +```cmake +add_custom_command( + OUTPUT ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp + COMMAND ${Python3_EXECUTABLE} + ${CMAKE_SOURCE_DIR}/tools/gen_messages.py + ${CMAKE_SOURCE_DIR}/data/messages.yaml + ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp + DEPENDS ${CMAKE_SOURCE_DIR}/data/messages.yaml + ${CMAKE_SOURCE_DIR}/tools/gen_messages.py +) +``` + +Re-runs on `data/messages.yaml` edits *or* on +`tools/gen_messages.py` edits. Generated header goes into a +sibling include directory so the library can include it as +`#include "secsgem/gem/messages.hpp"`. + +### Why Python rather than templates / constexpr + +Three reasons: + +1. **YAML parsing** — full grammar matters and `PyYAML` is more + reliable than yaml-cpp at parse-time gymnastics. +2. **Code shape control** — the generated C++ is easier to read + when generated by a textual templater than by C++ metaprogramming. +3. **Debuggability** — a customer who wants to see "what code is + actually being run for S2F33" can `grep` the generated header. + No mystery types, no instantiation chains. + +The codegen is ~388 lines of Python; the input grammar is +documented at its top. + +--- + +## Runtime loading + +The other four YAMLs (`control_state`, `process_job_state`, +`control_job_state`, `equipment`) load at runtime, not build time. +The same loader handles all of them: + +```cpp +// include/secsgem/config/loader.hpp +namespace secsgem::config { + EquipmentDescriptor load_equipment(const std::string& path); + ControlStateConfig load_control_state_table(const std::string& path); + ProcessJobStateConfig load_process_job_state(const std::string& path); + ControlJobStateConfig load_control_job_state(const std::string& path); +} +``` + +Each `load_*` returns a typed config struct on success or throws on +malformed YAML. Throwing is OK because YAML loading happens once +at startup — before binding the port — so a malformed file fails +the process up front. + +--- + +## The `--validate-config` pass + +YAML loaders that throw on first error are unfriendly: customers +often have multiple typos in a new equipment.yaml. The codebase +ships a multi-error validator: + +```cpp +// include/secsgem/config/validate.hpp +class ConfigValidator { +public: + void validate_equipment(const std::string& path); + void validate_control_state(const std::string& path); + // ... + + bool has_errors() const; + void format_issues_to(std::ostream&, …) const; +}; +``` + +It tries to load each file, accumulates *every* issue it can find, +and prints them all. Then exits 0 or 1. + +Invoke via: + +```bash +secs_server --validate-config \ + --config data/equipment.yaml \ + --state-table data/control_state.yaml \ + --pj-state-table data/process_job_state.yaml \ + --cj-state-table data/control_job_state.yaml +``` + +This is proof #5 in [PROOFS.md](PROOFS.md) — runs in CI to +guarantee every shipped YAML is structurally + referentially +sound. + +Tests: +[`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp) +(8 cases — every category of validation issue). + +--- + +## How to add a capability without C++ + +The point of spec-as-data is that **adding behaviour almost never +requires a C++ change**. + +### New SVID + +```yaml +# data/equipment.yaml +svids: + - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25} +``` + +Restart. Done. Host can now read SVID 4 via S1F3. + +### New CEID with linked report + +```yaml +# data/equipment.yaml +ceids: + - {id: 350, name: ChamberTempHigh} + +events: + default_reports: + - {ceid: 350, vids: [4]} +``` + +Restart. Done. When the EAP fires CEID 350, the report carries +SVID 4 automatically. + +### New host command + +```yaml +host_commands: + - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2} +``` + +Restart. Done. Host sends `S2F41(RCMD=VENT)` → ACK=Accept, +CEID 400 fires, ALID 2 set. + +### New control-state transition + +```yaml +# data/control_state.yaml +transitions: + - {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept} +``` + +Restart. Done. + +### New SECS-II message + +```yaml +# data/messages.yaml +- id: S6F30 + stream: 6 + function: 30 + w: true + builder: s6f30_request + parser: parse_s6f30 + body: + kind: list + struct_name: TempQuery + fields: + - {name: vid, shape: {kind: scalar, item_type: U4}} +``` + +`docker compose run --rm builder` regenerates `messages.hpp`. A +new `s6f30_request(uint32_t vid)` builder and a `parse_s6f30(item) +→ std::optional` parser appear. Now the *handler* is +still C++ — `gem::Router::on(6, 30, ...)` — because the side-effect +of "host asked for the temperature" needs application logic. + +--- + +## When spec-as-data isn't the right fit + +Three categories that *do* need C++: + +1. **Application logic** — what an alarm threshold actually is, + how a recipe step gets executed. No YAML schema can express + "vent the chamber if pressure > 1 Torr." +2. **State-machine actions** — when a CJ transitions to Executing, + *which* PJ to select next isn't a table entry; it's an + algorithm. +3. **External integrations** — talking to a PLC, reading a sensor, + driving a robot. Hardware bindings are vendor-specific code. + +The codebase draws the line **at the message catalog and the +transition tables**. Everything below (codec, transport) is fixed +C++. Everything above (application wiring) is per-EAP C++. +Everything between (data dictionary + state model) is YAML. + +--- + +## Where to go next + +You now know how the YAML drives the runtime. The next chapter +gets concrete about the **stores** — the per-domain bundles +(SVIDs, CEIDs, alarms, carriers, …) that the YAML populates and +the Router handlers operate over. + +Next: [→ 32 Stores and the data model](32_stores_and_the_data_model.md) diff --git a/docs/32_stores_and_the_data_model.md b/docs/32_stores_and_the_data_model.md new file mode 100644 index 0000000..395d99d --- /dev/null +++ b/docs/32_stores_and_the_data_model.md @@ -0,0 +1,308 @@ +# 32 — Stores and the data model + +← [31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md) | [Back to index](00_index.md) | Next: [33 Transport](33_transport.md) → + +The previous chapter showed how YAML drives behaviour. This +chapter shows the runtime data structures that the YAML populates +and that the Router handlers operate on: **stores** and the +**`EquipmentDataModel`** that composes them. + +By the end you'll know: + +- What a **store** is (and what it isn't). +- Every store in the codebase, one sentence each. +- How `EquipmentDataModel` composes them. +- The "no locks; single-threaded" contract. +- How to add a new store. + +--- + +## What a store is + +A **store** is a per-domain bundle of: + +- A few typed records (`std::map`, `std::vector`, …). +- A small API for reading + mutating them. +- A `change_handler` that emits events on transitions. +- Optional file-backed persistence. + +The naming is consistent: `AlarmRegistry` for active alarms, +`CarrierStore` for carriers, `ProcessJobStore` for PJs. +Headers live in +[`include/secsgem/gem/store/`](../include/secsgem/gem/store/); +implementations are typically inline in the header (these are small). + +Each store maps onto **one concern from one SEMI standard**: + +| Store | Concern | Standard | +|-----------------------------|--------------------------------------------------|----------------| +| `StatusVariableStore` | SVIDs + values | E30 §6.13 | +| `DataVariableStore` | DVIDs + values | E30 §6.11 | +| `EquipmentConstantStore` | ECIDs + values + min/max bounds | E30 §6.16 | +| `EventReportSubscriptions` | RPTID definitions + CEID linkings + enables | E30 §6.6 | +| `AlarmRegistry` | ALIDs + ALCD/ALTX + enable bits + active set | E30 §6.14 | +| `RecipeStore` | PPIDs + PPBODY (unformatted) + formatted bodies | E30 §6.17 + E42| +| `Clock` | Wall-clock + drift + quality | E30 §6.20 + E148 | +| `HostCommandRegistry` | RCMD names + per-command ack + side effects | E30 §6.15 | +| `SpoolStore` | Per-stream whitelist + queue + persistent journal| E30 §6.22 | +| `LimitMonitorStore` | LIMITIDs + upper/lower bounds + active state | E30 §6.21 | +| `TraceStore` | TRIDs + active sampling config | E30 §6.12 | +| `ProcessJobStore` | PJs + state + material list + persistent | E40 | +| `ControlJobStore` | CJs + state + PJ refs + persistent | E94 | +| `ExceptionStore` | EXIDs + recovery state + persistent | E5 §13 | +| `CarrierStore` | Carrier IDs + state machines + persistent | E87 | +| `LoadPortStore` | LP IDs + transfer/reservation/association FSMs | E87 | +| `SubstrateStore` | Substrate IDs + 3 FSMs + location + persistent | E90 | +| `EptStateMachine` | EPT state + time buckets | E116 | +| `CemObjectStore` | E120 typed object hierarchy | E120 | +| `ModuleStore` | Module IDs + state | E157 | +| `E84PortStore` | Per-LP E84 FSM + signals + timers | E84 | + +Each is a header. Each is independently testable: you can +`#include "secsgem/gem/store/alarms.hpp"` and exercise +`AlarmRegistry` without pulling in the rest. This is the same +shape as the per-standard tests in +[`tests/`](../tests/) — one test file per store. + +--- + +## EquipmentDataModel — the composite + +[`include/secsgem/gem/data_model.hpp`](../include/secsgem/gem/data_model.hpp) +defines: + +```cpp +struct EquipmentDataModel { + StatusVariableStore svids; + DataVariableStore dvids; + EquipmentConstantStore ecids; + EventReportSubscriptions events; + AlarmRegistry alarms; + RecipeStore recipes; + Clock clock; + HostCommandRegistry commands; + SpoolStore spool; + LimitMonitorStore limits; + TraceStore traces; + ProcessJobStore process_jobs; + ControlJobStore control_jobs; + ExceptionStore exceptions; + CarrierStore carriers; + LoadPortStore load_ports; + SubstrateStore substrates; + EptStateMachine ept; + CemObjectStore cem; + ModuleStore modules; + E84PortStore e84_ports; + // ... convenience methods spanning stores +}; +``` + +That's it. No locks, no smart pointers, no interfaces, no DI +container. Each store is a value member; ownership is the +`EquipmentDataModel` itself. + +The application typically holds one `shared_ptr` +and passes it to every Router handler. Handlers operate on the +stores directly: + +```cpp +router.on(1, 3, [model](const secs2::Message& m) { + // S1F3 — host requests SVID values + auto svids = parse_s1f3(m.body()); + return build_s1f4(model->svids.values(svids)); +}); +``` + +### Convenience methods + +`EquipmentDataModel` adds a few cross-store helpers (`data_model.hpp:54`): + +```cpp +std::optional vid_value(uint32_t vid) const { + // Look up VID in svids first, then dvids. +} + +std::vector compose_reports_for(uint32_t ceid) const { + // Walk events store -> reports store -> svids/dvids, + // assemble the S6F11 report payload for one CEID firing. +} +``` + +`compose_reports_for` is the *heart* of event notification — it +walks three stores to assemble the body for one S6F11 frame. See +chapter [13](13_e30_gem.md) for the wire flow. + +--- + +## The single-threaded contract + +**Every store mutation runs on the io_context strand.** No locks, +no atomics, no condition variables. This is documented in +[`docs/INTEGRATION.md`](INTEGRATION.md) §3 and enforced under +ThreadSanitizer. + +Why? Two reasons: + +1. **Performance.** Locking a `std::map` for every SVID read is a + waste in a hot path that processes thousands of messages a + second. The asio strand model gives the same correctness + guarantee for free. +2. **Simplicity.** Every method on every store is the obvious + non-locking implementation. Reading the code, you don't have + to track which lock protects what. + +The cost: **callers from other threads must `asio::post` onto +the executor**. + +```cpp +// From a sensor thread: +asio::post(io_context, [model, vid, value] { + model->svids.set_value(vid, secs2::Item::f4(value)); +}); +``` + +Tested by +[`tests/test_thread_safety.cpp`](../tests/test_thread_safety.cpp) +under TSan: N producer threads `asio::post` updates; TSan reports +zero races. Chapter [33](33_transport.md) covers the strand model +in more detail. + +--- + +## How a store's API looks (a small one) + +Pick `AlarmRegistry` — one of the smallest: + +```cpp +class AlarmRegistry { + public: + // Register an alarm definition. + void register_alarm(uint32_t alid, uint8_t alcd, const std::string& altx); + + // Set / clear an active alarm. Fires the change handler. + void set(uint32_t alid); + void clear(uint32_t alid); + + // Enable / disable host notification (S5F3). + void set_enabled(uint32_t alid, bool enabled); + bool is_enabled(uint32_t alid) const; + + // List active / all alarms. + std::vector all() const; + std::vector active() const; + + // Observer: change handler signature. + using ChangeHandler = std::function; + void set_change_handler(ChangeHandler); +}; +``` + +Every store follows that same shape: mutator + reader + observer. +The Router handler for `S5F1` doesn't fire `S5F1` itself — it +mutates the store; the change handler (registered at startup by +the EAP) fires `S5F1` via the connection. + +--- + +## How a store's API looks (a bigger one) + +[`ProcessJobStore`](../include/secsgem/gem/store/process_jobs.hpp) +adds: + +- Submit a PJ (record entry + fire `Created` event). +- Get / set state of any PJ. +- Apply a host-driven event (PJSTART / PJPAUSE / …) and route to + the FSM. +- Iterate active PJs (for serializing on restart). +- Persistent journal: `enable_persistence(dir)`. + +The FSM logic isn't *inside* the store — `ProcessJobStateMachine` +in [`process_job_state.hpp`](../include/secsgem/gem/process_job_state.hpp) +owns transitions. The store holds one `ProcessJobStateMachine` +per PJ and dispatches. + +This separation — *store* (records) vs *state machine* (transitions) — +keeps each layer testable in isolation. + +--- + +## Persistence + +Six stores have file-backed persistence: spool, process_jobs, +control_jobs, exceptions, carriers, load_ports, substrates. + +Each opts in via `enable_persistence(dir)`: + +```cpp +model->process_jobs.enable_persistence("/var/lib/secsgem/pj"); +``` + +That: + +1. Creates the directory if needed. +2. **Replays** every record file found there back into in-memory + state. +3. Sets up the on-disk journal: every mutation writes (or rewrites, + or deletes) one file per record, named by ID. + +Per-record-per-file means the journal is **partial-write safe**: +if the equipment power-cycles mid-write of one record, the others +are untouched; the partial file is detected and dropped at the +next startup. + +Chapter [36](36_persistence_validation_metrics.md) walks the +mechanism, the multi-version reads, and the test patterns. + +--- + +## How to add a new store + +Two cases: + +### Case 1: Standard already implemented, new sub-area + +E.g., add a "Reticle" store to track lithography reticles +distinctly from substrates. + +1. Create `include/secsgem/gem/store/reticles.hpp` with a class + `ReticleStore` exposing the standard + register / set-state / get / change-handler shape. +2. Add a member to `EquipmentDataModel`: + ```cpp + ReticleStore reticles; + ``` +3. Write `tests/test_reticles.cpp` mirroring the pattern from any + other store's test. +4. Wire Router handlers in `apps/secs_server.cpp` (or the EAP) for + whatever S/F messages drive it. + +### Case 2: Brand new SEMI standard + +E.g., implement E170 (a new GEM standard). + +Same as case 1, plus: + +5. Update [`data/messages.yaml`](../data/messages.yaml) with any + new S/F messages. `docker compose run --rm builder` regens + `messages.hpp`. +6. If E170 has its own transition table, create + `data/e170_state.yaml` and a `load_e170_state(...)` loader in + `config::`. +7. Update [`docs/COMPLIANCE.md`](COMPLIANCE.md) with the new + capability row. + +The architecture is **specifically designed** to add new standards +without disturbing existing ones. + +--- + +## Where to go next + +You've now seen how every per-domain data record is shaped and +how `EquipmentDataModel` composes them. Next, we drop back down +to transport: how `hsms::Connection` and `secsi::Protocol` actually +move bytes, and the asio strand model that makes the +single-threaded contract work. + +Next: [→ 33 Transport](33_transport.md) diff --git a/docs/33_transport.md b/docs/33_transport.md new file mode 100644 index 0000000..47476b2 --- /dev/null +++ b/docs/33_transport.md @@ -0,0 +1,292 @@ +# 33 — Transport + +← [32 Stores and the data model](32_stores_and_the_data_model.md) | [Back to index](00_index.md) | Next: [34 Codec and SML](34_codec_and_sml.md) → + +We covered the standards-level view of HSMS and SECS-I in +chapters 11 and 12. This chapter drops down into the +implementation: how `hsms::Connection` actually moves bytes, +the asio executor model, the single-threaded strand contract, +and why the transport layer doesn't need locks. + +--- + +## The two transport modules + +``` +include/secsgem/hsms/ +├── header.hpp — Frame format primitives (length prefix, header, SType). +└── connection.hpp — One-socket session manager + T-timers + S9 emission. + +include/secsgem/secsi/ +├── header.hpp — 10-byte SECS-I block header. +├── block.hpp — Block split / assemble (multi-block messages). +├── protocol.hpp — IO-free line-turnaround FSM. +└── tcp_transport.hpp — asio TCP wrapper around the FSM (tunnel for testing). +``` + +Each module owns one TCP endpoint (or in the SECS-I case, a tunnel +endpoint). Both are **single-threaded by design**. + +--- + +## hsms::Connection — top to bottom + +### Lifecycle + +```cpp +// apps/secs_server.cpp — passive-equipment startup +asio::io_context io; + +asio::ip::tcp::acceptor acc(io, asio::ip::tcp::endpoint{asio::ip::tcp::v4(), port}); + +acc.async_accept([&](std::error_code ec, asio::ip::tcp::socket sock) { + auto conn = std::make_shared( + std::move(sock), Mode::Passive, /*device_id=*/0, timers); + conn->set_message_handler(...); + conn->set_closed_handler(...); + conn->start(); +}); + +io.run(); // blocks until all work is done +``` + +`Connection::start()` either: + +- **Passive** — arms T7 (waiting for Select.req) and starts the + read loop. +- **Active** — initiates the Select.req exchange, then starts the + read loop. + +### Read path + +Three async steps repeated forever: + +``` +async_read(socket, 4 bytes) → on_length() +async_read(socket, length bytes) → on_payload() +handle_frame(decoded Frame) → dispatch +``` + +In code, [`src/hsms/connection.cpp`](../src/hsms/connection.cpp): + +```cpp +void Connection::read_length() { + asio::async_read(socket_, asio::buffer(len_buf_, 4), + [self = shared_from_this()](std::error_code ec, std::size_t n) { + self->on_length(ec, n); + }); +} + +void Connection::on_length(std::error_code ec, std::size_t n) { + if (ec) return close("read_length"); + uint32_t len = decode_be32(len_buf_); + payload_.resize(len); + asio::async_read(socket_, asio::buffer(payload_), + [self = shared_from_this()](std::error_code ec, std::size_t n) { + self->on_payload(ec, n); + }); +} +``` + +Each callback is on the socket's executor. No locks because +nothing else can be touching the read state — by construction. + +### Write path + +A send queue + one outstanding `async_write`: + +```cpp +void Connection::send_frame(Frame frame) { + send_queue_.push_back(std::move(frame)); + if (send_queue_.size() == 1) write_next(); +} + +void Connection::write_next() { + auto& frame = send_queue_.front(); + send_buf_ = frame.encode(); + asio::async_write(socket_, asio::buffer(send_buf_), + [self = shared_from_this()](std::error_code ec, std::size_t) { + self->send_queue_.pop_front(); + if (ec) return self->close("write"); + if (!self->send_queue_.empty()) self->write_next(); + }); +} +``` + +Same single-threaded discipline — `send_queue_` is only touched +on the executor. Callers from other threads must `asio::post`. + +### Timers + +Five `asio::steady_timer`s, one per HSMS T-timer: + +```cpp +// src/hsms/connection.cpp:30 +Connection::Connection(...) + : socket_(std::move(sock)), + t3_timer_(socket_.get_executor()), + t6_timer_(socket_.get_executor()), + t7_timer_(socket_.get_executor()), + t8_timer_(socket_.get_executor()), + linktest_timer_(socket_.get_executor()), + timers_(timers) { } +``` + +All five share the socket's executor. When a timer fires, its +handler runs on the same executor as the read/write loop — so +again no locks for timer-vs-IO interaction. + +T3 is special: there's one T3 timer per in-flight W=1 message +(correlated by `system_bytes`). These are short-lived +`steady_timer`s allocated when the request is sent and destroyed +when the reply arrives. See `src/hsms/connection.cpp:447`: + +```cpp +auto t3 = std::make_shared(socket_.get_executor()); +t3->expires_after(timers_.t3); +in_flight_.insert({system_bytes, RequestState{std::move(cb), t3, ...}}); +t3->async_wait([self, system_bytes](std::error_code ec) { + if (ec) return; // cancelled (reply arrived) + self->on_t3_expire(system_bytes); +}); +``` + +--- + +## The asio executor / strand model + +All `Connection` state lives on **one executor**. In simple cases +that's just `io_context.get_executor()` — a single-threaded loop. +In production, an EAP may run multiple `io_context::run()` threads +*per connection* by wrapping work in an `asio::strand`. + +### What a strand is + +A **strand** is an executor that guarantees mutual exclusion +between handlers it dispatches. Multiple threads can call +`io.run()`; the strand picks one of them at a time to run the +next handler in its queue. + +For HSMS purposes, `socket_.get_executor()` already gives strand +semantics if the underlying `io_context` is single-threaded. For +multi-threaded `io_context`, the application wraps with +`asio::make_strand`: + +```cpp +auto strand = asio::make_strand(io); +asio::ip::tcp::socket sock(strand); +auto conn = std::make_shared(std::move(sock), ...); +``` + +Now `conn` has all its IO running on `strand`, while the +`io_context` can use 8 threads to handle 100 different connections. + +### What this means for the caller + +From any thread that isn't the strand's currently-running handler, +the caller MUST marshal onto the strand: + +```cpp +// From a sensor-thread callback: +asio::post(conn->executor(), [conn, msg = std::move(msg)] { + conn->send_data(std::move(msg)); +}); +``` + +Calling `conn->send_data` directly from another thread is **a +race**. Same for any store mutation. TSan catches this and the +test suite enforces it. + +The contract is documented in detail in +[`docs/INTEGRATION.md`](INTEGRATION.md) §3. + +--- + +## secsi::Protocol — the IO-free FSM + +SECS-I's protocol layer is structured differently: the FSM has +**no IO at all**. It takes events (bytes received, application +asked to send, timer fired) and produces a list of `Action`s +(transmit these bytes, arm a timer, deliver a block to the +application). + +```cpp +// include/secsgem/secsi/protocol.hpp +struct ActionTransmit { std::vector bytes; }; +struct ActionStartTimer { Timer which; }; +struct ActionCancelTimer { Timer which; }; +struct ActionDeliverBlock { Block block; }; +struct ActionRaiseError { std::string reason; }; +``` + +The wrapper (`secsi::TcpTransport`) drives the FSM: + +```cpp +void TcpTransport::on_byte(uint8_t b) { + auto actions = protocol_.handle(EventByte{b}); + for (const auto& a : actions) execute(a); +} + +void TcpTransport::execute(const Action& a) { + std::visit([this](auto&& v) { + using T = std::decay_t; + if constexpr (std::is_same_v) write_bytes(v.bytes); + else if constexpr (std::is_same_v) arm_timer(v.which); + else if constexpr (std::is_same_v) cancel_timer(v.which); + else if constexpr (std::is_same_v) deliver(v.block); + else if constexpr (std::is_same_v) raise(v.reason); + }, a); +} +``` + +This design makes the **whole FSM** unit-testable. No sockets, +no timers, just `Event` in → `Action` out. Tests: + +- [`tests/test_secsi.cpp`](../tests/test_secsi.cpp) — basic FSM + state walks. +- [`tests/test_secsi_timers.cpp`](../tests/test_secsi_timers.cpp) + — every timer scenario via synthetic `EventTimeout` injection. +- [`tests/test_secsi_tcp.cpp`](../tests/test_secsi_tcp.cpp) — + end-to-end via `TcpTransport`. + +Same pattern repeats for E84 (chapter 18) and the GEM +communication-state FSM (chapter 13): IO-free FSM + asio adapter ++ separate test suites for each layer. + +--- + +## Why this is the right shape + +### Pros of single-threaded + IO-free + +- **No mutexes.** Anywhere. +- **Trivial reasoning.** When you read `Connection::send_frame`, + you can be sure nothing else is mutating the queue. +- **Fast.** No lock contention, no atomic round-trips. +- **Testable.** IO-free FSM lets you exercise every transition + without IO. + +### Cons + +- **Callers must know about the strand.** Multi-threaded + applications need `asio::post` boilerplate. +- **One slow handler blocks the rest.** A 100 ms handler delays + every other message until it returns. Fix: don't write slow + handlers; if you must, dispatch the slow work to another + executor and return immediately. + +For a SECS/GEM equipment runtime — where the natural shape is "one +TCP socket, one event loop" — the pros far outweigh the cons. + +--- + +## Where to go next + +You've now seen the bottom layer in detail: how bytes move, +how state machines drive transitions without IO, how the single- +threaded contract makes everything safe. Next chapter goes back +up one level: **the codec** — the encoder/decoder that turns +the Item type from chapter 10 into wire bytes, plus the SML +human-readable form. + +Next: [→ 34 Codec and SML](34_codec_and_sml.md) diff --git a/docs/34_codec_and_sml.md b/docs/34_codec_and_sml.md new file mode 100644 index 0000000..431eb28 --- /dev/null +++ b/docs/34_codec_and_sml.md @@ -0,0 +1,303 @@ +# 34 — Codec and SML + +← [33 Transport](33_transport.md) | [Back to index](00_index.md) | Next: [35 State machines and dispatch](35_state_machines_and_dispatch.md) → + +We covered the SECS-II encoding rules in chapter 10. This +chapter is the **implementation walk** — the four files that make +up `secsgem::secs2`, how the encoder/decoder are structured, why +the variant-based `Item` works, and how the SML printer/parser +fits in. + +Four files, 733 lines total. The codec is the most-tested layer +in the codebase. + +--- + +## The four files + +``` +include/secsgem/secs2/ +├── item.hpp (170 lines) Item variant + Format enum + factories. +├── codec.hpp ( 30 lines) encode / decode declarations. +├── message.hpp ( 52 lines) Message wrapper (header fields + body Item). +└── sml.hpp ( 32 lines) to_sml / try_parse_sml declarations. + +src/secs2/ +├── codec.cpp (229 lines) encode_into / decode_at implementations. +└── sml.cpp (220 lines) SML printer + parser. +``` + +`item.hpp` and `message.hpp` are header-only. `codec.cpp` and +`sml.cpp` carry the heavy lifting. + +--- + +## The `Item` variant + +Already covered in chapter 10; quick recap of the storage: + +```cpp +// include/secsgem/secs2/item.hpp:85 +class Item { + public: + using List = std::vector; + using Storage = std::variant< + List, // List + std::string, // ASCII, JIS-8 + std::vector, // Binary, Boolean, U1 + std::vector, // I1 + std::vector, // I2 + ... + std::vector, // F4 + std::vector>; // F8 + + private: + Format format_; + Storage data_; +}; +``` + +Eleven variant alternatives serving 16 SECS-II formats — some +formats share storage (Binary/Boolean/U1 all use +`std::vector`, ASCII/JIS-8 share `std::string`, U2/C2 +share `std::vector`). Disambiguation is via `format_`. + +### Factories + +The intended way to build an `Item` is the named factories: + +```cpp +Item::list({Item::ascii("Hi"), Item::u4(42)}); +Item::ascii("Hello, world"); +Item::u4(std::vector{1, 2, 3}); +Item::u4(42); // scalar convenience overload +Item::f4(1.0f); +``` + +Each takes ownership of the storage (or constructs from a scalar +overload). No exceptions; no validity checks; trusts the caller. + +--- + +## `encode_into` — the recursive encoder + +```cpp +void encode_into(const Item& item, std::vector& out); +``` + +[`src/secs2/codec.cpp:71`](../src/secs2/codec.cpp). Two paths — +List and not-List: + +```cpp +void encode_into(const Item& item, std::vector& out) { + const Format fmt = item.format(); + + if (fmt == Format::List) { + const auto& children = item.as_list(); + write_header(out, fmt, children.size()); + for (const auto& child : children) encode_into(child, out); + return; + } + + // Scalar/array: write_header(byte count), then bytes. + switch (fmt) { + case Format::ASCII: { + const auto& s = item.as_ascii(); + write_header(out, fmt, s.size()); + out.insert(out.end(), s.begin(), s.end()); + return; + } + case Format::U4: { + const auto& v = std::get>(item.storage()); + write_header(out, fmt, v.size() * 4); + for (auto x : v) put_scalar_be(out, x); + return; + } + // ... one case per format + } +} +``` + +`write_header` picks the smallest length-byte-count and emits the +format byte + length bytes. `put_scalar_be` is the +templated big-endian writer using `std::bit_cast` for floats and +`std::make_unsigned_t` for integers (chapter 10). + +`encode(item)` is a thin wrapper: + +```cpp +std::vector encode(const Item& item) { + std::vector out; + encode_into(item, out); + return out; +} +``` + +--- + +## `decode_at` — the recursive decoder + +```cpp +Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos); +``` + +Mirror image: + +```cpp +Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos) { + // 1. Format byte + length bytes. + if (pos >= len) throw CodecError("truncated"); + const uint8_t fb = data[pos++]; + const Format fmt = static_cast(fb >> 2); + const int nlen = fb & 0x03; + if (pos + nlen > len) throw CodecError("truncated length bytes"); + std::size_t length = 0; + for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++]; + + // 2. List recursion. + if (fmt == Format::List) { + Item::List children; + children.reserve(length); + for (std::size_t i = 0; i < length; ++i) + children.push_back(decode_at(data, len, pos)); + return Item::list(std::move(children)); + } + + // 3. Scalar/array: dispatch on element size + signedness/floatness. + if (pos + length > len) throw CodecError("truncated body"); + const uint8_t* body = data + pos; + pos += length; + switch (fmt) { + case Format::ASCII: return Item::ascii(std::string((const char*)body, length)); + case Format::U4: return Item::u4(read_array(body, length)); + // ... one case per format + } + throw CodecError("unknown format code"); +} + +Item decode(const std::vector& bytes) { + std::size_t pos = 0; + Item it = decode_at(bytes.data(), bytes.size(), pos); + if (pos != bytes.size()) throw CodecError("trailing bytes"); + return it; +} +``` + +The `_at` variant is useful when an outer protocol carries a SECS-II +item *embedded* in a larger frame — the caller passes the buffer +and a position, and gets back the item plus the new position. + +Bounds checks throw `CodecError` at every step — a CodecError on +the receive side closes the connection (chapter 11's S9F7 path). + +--- + +## The Message wrapper + +```cpp +// include/secsgem/secs2/message.hpp +class Message { + public: + uint8_t stream() const; + uint8_t function() const; + bool w_bit() const; + uint32_t system_bytes() const; + const Item& body() const; + std::vector body_bytes() const; // encoded body +}; +``` + +A `Message` is just a small struct: stream + function + W-bit + +system_bytes + body Item. No encoder lives here — encoding is +done by `secs2::encode(message.body())` when the transport layer +serializes a frame. The Message exists so the Router can dispatch +on `(stream, function)` without re-decoding bytes. + +--- + +## SML — the human-readable form + +`to_sml(item)` walks the Item recursively and emits SML: + +```cpp +// src/secs2/sml.cpp — sketch +std::string to_sml(const Item& item) { + switch (item.format()) { + case Format::List: { + std::string s = ">(item.storage()); + std::string s = "U4"; + if (v.size() > 1) s += "[" + std::to_string(v.size()) + "]"; + for (auto x : v) s += " " + std::to_string(x); + return s; + } + // ... per format + } +} +``` + +`try_parse_sml(text)` is the inverse — a hand-written recursive- +descent parser that returns `std::optional`. Returns +`nullopt` on any parse error (no exceptions; this is what +libFuzzer feeds garbage into and expects it not to crash). + +Tests: +[`tests/test_sml.cpp`](../tests/test_sml.cpp) (10 cases — every +format round-trips through `to_sml` → `try_parse_sml` → identical +Item). + +### Why SML doesn't round-trip *bytes* + +A subtle point: `decode(encode(item))` round-trips exactly, but +`try_parse_sml(to_sml(item))` *also* round-trips the Item — except +encoding the round-tripped Item may produce **different bytes** +than the original. Why? + +- The original might use a 2-byte length encoding; the + round-tripped Item is a fresh `Item` and the encoder will pick + the smallest length encoding (1 byte). +- SML doesn't preserve "which list-length encoding the encoder + chose." + +If you need bit-exact round-trip of *bytes*, use `decode(encode)`. +For semantic round-trip of *values*, use SML. + +--- + +## Testing — every layer in isolation + +| Layer | Test file | Cases | Focus | +|--------------|--------------------------------------|------:|--------------------------------------------------------| +| Item factories | tests/test_secs2.cpp | 14 | Construction, equality, format dispatch. | +| Codec | tests/test_e5_kat.cpp | 19 | Known-answer tests — bit-exact bytes per SEMI E5 §9. | +| Codec | tests/test_secs2.cpp | (overlap) | encode/decode round-trip + truncation rejection. | +| Identifier wildcards | tests/test_identifier_wildcards.cpp | 6 | U1/U2/U4/U8 leniency for ID fields. | +| SML | tests/test_sml.cpp | 10 | to_sml + try_parse_sml round-trip. | +| Catalog | tests/test_messages.cpp | 82 | Every named SxFy builder + parser round-trip. | +| Random/structural | tests/test_fuzz.cpp | 8 | Random bytes, truncation, oversize lengths, nested. | +| libFuzzer | apps/fuzz_secs2_decode.cpp | (CI) | 200 k+ random inputs per minute, ASan + UBSan clean. | +| libFuzzer | apps/fuzz_sml_parse.cpp | (CI) | 1.4 M+ random SML strings per minute, ASan + UBSan. | + +The codec alone has **139 test cases / 196+ assertions for E5 +KAT**. This is intentional: every other layer trusts the codec is +correct. If it isn't, nothing above works. + +--- + +## Where to go next + +You've now seen the codec and SML implementation in detail. Next +chapter covers the **dispatch** layer that sits between the +transport (which delivers raw `Message`s) and the stores (which +hold state): `gem::Router`, the state-machine wiring, and the +generated builder/parser glue from the message catalog. + +Next: [→ 35 State machines and dispatch](35_state_machines_and_dispatch.md) diff --git a/docs/35_state_machines_and_dispatch.md b/docs/35_state_machines_and_dispatch.md new file mode 100644 index 0000000..a70e03d --- /dev/null +++ b/docs/35_state_machines_and_dispatch.md @@ -0,0 +1,305 @@ +# 35 — State machines and dispatch + +← [34 Codec and SML](34_codec_and_sml.md) | [Back to index](00_index.md) | Next: [36 Persistence, validation, metrics](36_persistence_validation_metrics.md) → + +We have transport (chapter 33) delivering `secs2::Message`s +(chapter 34) into the application. We have stores (chapter 32) +holding the data. This chapter is the **glue**: how the Router +dispatches by `(stream, function)`, how state machines compose with +stores, and how a Router handler typically looks. + +--- + +## `gem::Router` — the dispatch table + +[`include/secsgem/gem/router.hpp`](../include/secsgem/gem/router.hpp): + +```cpp +class Router { + public: + using Handler = std::function(const s2::Message&)>; + + Router& on(uint8_t stream, uint8_t function, Handler h); + Router& fallback(Handler h); + + std::optional dispatch(const s2::Message& msg) const; +}; +``` + +A `std::map<{stream, function}, Handler>`. Each handler: + +- Takes a `const s2::Message&` (the inbound primary). +- Returns a `std::optional` (the reply, or `nullopt` + for fire-and-forget primaries). + +That's it. No middleware, no decorator chain, no pre/post hooks. +Just a typed dispatch. + +### Registering handlers + +```cpp +// apps/secs_server.cpp — sketch +auto router = std::make_shared(); + +router->on(1, 1, [model](const auto& m) { + // S1F1 Are You There — reply with model name + softrev. + return messages::s1f2(model->device.mdln, model->device.softrev); +}); + +router->on(1, 3, [model](const auto& m) { + // S1F3 — read SVID values. + auto svid_list = messages::parse_s1f3(m.body()); + return messages::s1f4(model->svids.values(svid_list.value_or({}))); +}); + +router->on(2, 41, [model](const auto& m) { + // S2F41 Host Command. + auto cmd = messages::parse_s2f41(m.body()); + auto ack = model->commands.dispatch(cmd->rcmd, cmd->params); + return messages::s2f42(ack); +}); + +// ...one per S/F pair. apps/secs_server.cpp registers ~50. +``` + +The `examples/pvd_tool/main.cpp` §6 register 51 handlers in ~460 +lines. Each handler is a few lines: parse the body, mutate or read +a store, build the reply. + +### What happens for unhandled primaries + +```cpp +// router.hpp dispatch() +auto it = handlers_.find({msg.stream, msg.function}); +if (it != handlers_.end()) return it->second(msg); +if (fallback_) return fallback_(msg); +if (msg.reply_expected) return s2::Message(msg.stream, 0, false); // SxF0 Abort +return std::nullopt; +``` + +Three cases: + +1. **Registered**: handler runs. +2. **Fallback installed**: fallback runs. +3. **Neither**: if the message expects a reply, send `SxF0` (Abort) + per E5 convention. Otherwise silently drop. + +### S9 wiring + +The transport layer (chapter 11) emits S9F3 / S9F5 for unhandled +primaries. Router exposes the introspection: + +```cpp +bool has_handler(uint8_t stream, uint8_t function) const; +bool has_handler_for_stream(uint8_t stream) const; +``` + +And the wrapper that combines them with S9 emission: + +```cpp +template +std::optional dispatch_with_s9( + EmitFn emit_s9, HeaderProvider header_provider, + const s2::Message& msg) const { + if (!has_handler(msg.stream, msg.function)) { + if (auto mhead = header_provider()) { + const uint8_t f = has_handler_for_stream(msg.stream) ? 5 : 3; + emit_s9(f, *mhead); + } + } + return dispatch(msg); +} +``` + +Used by `Connection::on_data_message` → `Router::dispatch_with_s9`, +which calls back into `Connection::emit_s9` for the actual S9 +emission. Tested by +[`tests/test_s9_fallback.cpp`](../tests/test_s9_fallback.cpp) (2 +cases — unknown stream → S9F3, unknown function in known stream +→ S9F5). + +--- + +## How a typical handler looks end-to-end + +`S2F41` Host Command is the most-touched message in production — +worth tracing in full: + +### 1. The codegen'd parser/builder + +```cpp +// build/generated/secsgem/gem/messages.hpp (auto-generated) +struct RemoteCommand { + std::string rcmd; + std::vector> params; +}; + +inline std::optional parse_s2f41(const secs2::Item& body) { + // ...auto-generated body walker... +} + +inline secs2::Message s2f42(uint8_t hcack, /* per-param acks */) { + // ...auto-generated builder... +} +``` + +### 2. The Router registration + +```cpp +// apps/secs_server.cpp +router->on(2, 41, [model](const secs2::Message& m) { + auto cmd = messages::parse_s2f41(m.body()); + if (!cmd) { + // Body didn't parse — reply S2F42 with HCACK = 1 (invalid). + return messages::s2f42(1, {}); + } + auto outcome = model->commands.dispatch(cmd->rcmd, cmd->params); + return messages::s2f42(static_cast(outcome.ack), outcome.cpacks); +}); +``` + +### 3. The store dispatch + +```cpp +// include/secsgem/gem/store/host_commands.hpp +class HostCommandRegistry { + public: + CommandOutcome dispatch(const std::string& rcmd, const ParamList& params) { + auto it = commands_.find(rcmd); + if (it == commands_.end()) return {HostCmdAck::InvalidCommand, ...}; + const auto& cmd = it->second; + // Apply configured side effects: emit_ceid, set_alarm, … + for (auto ceid : cmd.emit_ceids) on_emit_ceid_(ceid); + for (auto alid : cmd.set_alarms) alarm_registry_->set(alid); + return {cmd.default_ack, ...}; + } +}; +``` + +### 4. The side-effect dispatcher + +Steps in `dispatch` like `on_emit_ceid_(ceid)` call back into +the EAP: + +```cpp +// Set up at startup: +model->commands.set_emit_ceid_handler([conn, model](uint32_t ceid) { + if (!model->is_event_enabled(ceid)) return; + auto reports = model->compose_reports_for(ceid); + auto msg = build_s6f11(ceid, reports); + conn->send_data(std::move(msg)); +}); +``` + +### 5. The wire + +`conn->send_data(s6f11)` walks through `secs2::encode` → +`hsms::Frame::encode` → `async_write` to the socket. Host sees +the unsolicited S6F11. + +--- + +## The state-machine pattern + +Every state machine in the codebase follows the same shape. Pick +`ControlStateMachine`: + +```cpp +// include/secsgem/gem/control_state.hpp +class ControlStateMachine { + public: + ControlStateMachine(ControlTransitionTable table); + + ControlState state() const; + + // Apply an event; returns the transition row (if any). + const ControlTransition* on_event(ControlEvent e); + + // Observer for transitions. + using StateChangeHandler = + std::function; + void set_state_change_handler(StateChangeHandler h); +}; +``` + +Three properties: + +1. **Pure data table** (`ControlTransitionTable`) decides what + transitions exist. +2. **Pure FSM** (`ControlStateMachine`) applies events against the + table, updates state, emits the change. +3. **Observer pattern** — the EAP registers a change handler that + does the wire-level work (fire CEID, emit S6F11, log to + metrics). + +This pattern repeats for: + +- `ProcessJobStateMachine` (E40) +- `ControlJobStateMachine` (E94) +- `EptStateMachine` (E116) +- `ExceptionStateMachine` (E5 §13) +- `CarrierStateMachine` (E87 — actually composes 3 sub-FSMs) +- `LoadPortStateMachine` (E87 — same) +- `SubstrateStateMachine` (E90 — same) +- `ModuleStateMachine` (E157) +- `E84StateMachine` (E84) +- `CommunicationStateMachine` (E30 §6.5) + +Eleven FSMs. All follow the same shape. All testable in +isolation without IO. + +--- + +## How FSM transitions cascade into CEIDs + +A common need: "when PJ-1 transitions to Processing, fire +CEID=ProcessStarted, which fires an S6F11 with linked reports." + +The wiring is set up at startup in the EAP: + +```cpp +// apps/secs_server.cpp — sketch +model->process_jobs.set_state_change_handler( + [conn, model](const std::string& pjid, + ProcessJobState from, ProcessJobState to, + ProcessJobEvent ev) { + // Configured per-state CEID from data/equipment.yaml. + auto ceid = ceid_for_pj_state(to); + if (!ceid || !model->is_event_enabled(*ceid)) return; + auto reports = model->compose_reports_for(*ceid); + auto msg = build_s6f11(*ceid, reports); + deliver_or_spool(*conn, *model, std::move(msg)); + }); +``` + +`deliver_or_spool` is the spool-aware send: if the connection isn't +SELECTED (or the spool is in transmit-disabled mode), the message +queues into `SpoolStore`; otherwise it goes straight to the wire. + +So the chain for "PJ-1 starts processing": + +``` +ProcessJobStore.apply(pjid, Start) + → ProcessJobStateMachine.on_event(Start) + → State: WaitingForStart → Processing + → on_change handler fires + → looks up CEID ProcessStarted + → composes reports + → builds S6F11 message + → deliver_or_spool → connection or SpoolStore + → on the wire (eventually) +``` + +Every step is independently testable. Tests at the wire level: +[`tests/test_wire_ceid_emission.cpp`](../tests/test_wire_ceid_emission.cpp) +(6 cases — every cascade from store mutation to socket bytes). + +--- + +## Where to go next + +You've now seen everything that makes the runtime *work*. One +chapter left in Part 3: the **operational** concerns — persistence, +config validation, and metrics. + +Next: [→ 36 Persistence, validation, metrics](36_persistence_validation_metrics.md) diff --git a/docs/36_persistence_validation_metrics.md b/docs/36_persistence_validation_metrics.md new file mode 100644 index 0000000..f4b8257 --- /dev/null +++ b/docs/36_persistence_validation_metrics.md @@ -0,0 +1,293 @@ +# 36 — Persistence, validation, metrics + +← [35 State machines and dispatch](35_state_machines_and_dispatch.md) | [Back to index](00_index.md) | Next: [40 Building, running, the demo](40_building_running_demo.md) → + +Three operational concerns wrap up Part 3: + +- **Persistence** — file-backed journals for the seven stores that + survive equipment restarts. +- **Validation** — the multi-error YAML validator behind + `--validate-config`. +- **Metrics** — the Prometheus exporter. + +Each is a small slice of the codebase but load-bearing for +production deployments. + +--- + +## Persistence + +### Which stores persist + +Seven of the 21 stores have file-backed journals: + +| Store | Survives equipment restart | +|--------------------|------------------------------------------------------------| +| `SpoolStore` | Queued messages waiting for host comm to come back. | +| `ProcessJobStore` | All in-progress PJs and their state machines. | +| `ControlJobStore` | All in-progress CJs. | +| `ExceptionStore` | Posted exceptions and their recovery state. | +| `CarrierStore` | Docked carriers + slot maps + access state. | +| `LoadPortStore` | Per-port association + reservation. | +| `SubstrateStore` | Per-substrate location + STS / SPS / ID status. | + +The remaining 14 stores (SVIDs, ECIDs, CEIDs, alarm registry, …) +don't persist — their state is reconstructed from the YAML or +from real-time signals on restart. An ECID that the host had +changed *would* be lost on restart unless the EAP writes it back +to the YAML (E40-style `S2F15` is rare in production for +exactly this reason). + +### The per-record file pattern + +Every persistent store uses the same shape: + +``` +/var/lib/secsgem// +├── PJ-001 # one file per record +├── PJ-002 +├── PJ-003 +└── ... +``` + +One file per record, named by ID. When the store is mutated, the +file is rewritten atomically (write to `.tmp` + `rename`). When +the record is removed, the file is `unlink`'d. + +**This is partial-write safe.** If the equipment power-cycles +mid-write of one record, the others are untouched. At startup, +the store iterates the directory, reads each file, and replays +into in-memory state. A file that fails to parse (corrupted or +unfinished) is dropped with a log line. + +### How a store enables persistence + +```cpp +// apps/secs_server.cpp — startup +auto model = std::make_shared(); + +if (!spool_dir.empty()) { + model->spool.enable_persistence(spool_dir); +} +if (!pj_dir.empty()) { + model->process_jobs.enable_persistence(pj_dir); +} +// ... etc per store +``` + +`enable_persistence(dir)`: + +1. Creates `dir` if needed. +2. Iterates files in `dir`. +3. For each file, reads + parses + adds the record to the store. +4. Sets up the on-disk journal for subsequent mutations. + +The persistence is **opt-in per store**, configured via CLI flag in +`apps/secs_server.cpp`. Some deployments want spool persistence +but not job persistence (e.g., test rigs); the per-store toggle +makes that easy. + +### File format and versioning + +Each record file is a small binary blob: + +``` +magic: 4 bytes "SGv1" (store-specific magic; v1 = version 1) +version: 4 bytes (uint32_t, big-endian) — schema version +length: 4 bytes (uint32_t, big-endian) — payload length +payload: N bytes — store-specific record encoding +checksum: 4 bytes (CRC-32C over header + payload) +``` + +**Schema versioning** is built in. Every store has a `kVersion` +constant. When the store reads a file: + +```cpp +if (file_version > kVersion) + drop the file (newer than us; can't read) +if (file_version < kVersion) + apply the upgrade path (v1 → v2 → v3 reader chain) +if (file_version == kVersion) + read directly +``` + +Multi-version reads let a new equipment release process old +on-disk records without manual migration. Tested by +[`tests/test_persistence_upgrade.cpp`](../tests/test_persistence_upgrade.cpp) +(7 cases — every store with persistence, write v1, restart at +v2, verify replay). + +### Tests + +| Store | Test file | Cases | +|---------------------|----------------------------------------------------|------:| +| Spool | bundled into `tests/test_data_model.cpp` | — | +| Process Jobs | `tests/test_job_persistence.cpp` (PJ + CJ together)| 7 | +| Control Jobs | same | — | +| Exception | `tests/test_exception_persistence.cpp` | 5 | +| Carrier | `tests/test_carrier_persistence.cpp` | 6 | +| Substrate | `tests/test_substrate_persistence.cpp` | 7 | +| Upgrade path | `tests/test_persistence_upgrade.cpp` | 7 | + +Each persistence test covers: write a record, restart, verify +replayed; partial-write recovery (truncated file dropped); remove +deletes the file; corrupted file is dropped without throwing. + +--- + +## Validation + +### Why a separate validator + +YAML loaders throw on first error. That's the right behaviour +at process startup — fail fast — but it's frustrating for an +operator with a fresh equipment.yaml that has three typos. + +`--validate-config` is a separate CLI flag that: + +1. Doesn't bind the port. +2. Tries to load every YAML. +3. Accumulates *every* issue (across files). +4. Prints them all. +5. Exits 0 or 1. + +```bash +secs_server --validate-config \ + --config data/equipment.yaml \ + --state-table data/control_state.yaml \ + --pj-state-table data/process_job_state.yaml \ + --cj-state-table data/control_job_state.yaml +``` + +Typical output: + +``` +data/equipment.yaml:42: SVID 5 references undefined enum 'ChamberStateEnum' +data/equipment.yaml:78: alarm 3 has ALCD bit-7 cleared but alarm is declared 'active' +data/control_state.yaml:11: transition from OnlineRemote on host_request_remote has no `to` or `ack` field +data/equipment.yaml:104: host_command VENT references unknown CEID 999 + +4 error(s), 0 warning(s) across 4 files +``` + +Then exit 1. + +### How it's implemented + +[`include/secsgem/config/validate.hpp`](../include/secsgem/config/validate.hpp): + +```cpp +class ConfigValidator { + public: + void validate_equipment(const std::string& path); + void validate_control_state(const std::string& path); + void validate_process_job_state(const std::string& path); + void validate_control_job_state(const std::string& path); + + std::size_t error_count() const; + std::size_t warning_count() const; + bool has_errors() const; + + const std::vector& issues() const; + void format_issues_to(std::ostream&, FormatOptions = {}) const; +}; +``` + +Each `validate_*` method: + +1. Loads the YAML (catching parse errors as one issue). +2. Walks every record, applying structural + referential checks. +3. Adds each problem as an `Issue{path, line, severity, message}`. + +Tests: +[`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp) +(8 cases — every category of issue: missing required field, +typed mismatch, dangling reference, duplicate ID, …). + +### Reference checks across files + +Cross-file references are validated last (after all files are +parsed). Examples: + +- `host_commands[].emit_ceid` must reference a CEID defined in + `equipment.yaml::ceids`. +- `events.default_reports[].vids` must reference SVIDs or DVIDs + defined elsewhere. +- `control_state.yaml::transitions` `from`/`to` must reference + states declared by the schema (the 5 standard control states). + +This catches "I deleted the CEID but forgot to update the +host_command" before runtime. + +--- + +## Metrics + +### What gets exported + +The codebase ships a Prometheus exporter +([`include/secsgem/metrics/prometheus.hpp`](../include/secsgem/metrics/prometheus.hpp)) +with two parts: + +- **Registry** — accumulates `Counter` and `Gauge` series with + labels. +- **Server** — exposes them on a configurable HTTP port at + `/metrics`. + +Typical wiring: + +```cpp +auto registry = std::make_shared(); +registry->register_metric("secsgem_ceid_emits_total", metrics::MetricType::Counter); +registry->register_metric("secsgem_spool_depth", metrics::MetricType::Gauge); +registry->register_metric("secsgem_pj_state", metrics::MetricType::Gauge); + +// ...later, in the CEID-emit handler: +registry->counter("secsgem_ceid_emits_total", {{"ceid", std::to_string(ceid)}}).inc(); + +// ...periodically: +registry->gauge("secsgem_spool_depth").set(model->spool.size()); + +// Start the HTTP server: +auto exporter = std::make_shared(io, /*port=*/9090, registry); +``` + +The exporter is wire-compatible with Prometheus scrape (text +format). Tested by +[`tests/test_metrics_prometheus.cpp`](../tests/test_metrics_prometheus.cpp) +(3 cases — counter increment, gauge set, HTTP scrape format). + +### What to expose + +Common patterns from +[`examples/pvd_tool/main.cpp`](../examples/pvd_tool/main.cpp) §7: + +- Per-CEID counters (`secsgem_ceid_emits_total{ceid="300"}`). +- Per-alarm counters (`secsgem_alarm_set_total{alid="42"}`). +- Spool depth gauge (alarm in operations if it climbs). +- Per-state EPT durations (sample of E116 buckets). +- T3 timeout counter (alarm in operations if non-zero). + +The exporter doesn't dictate which metrics to expose — the EAP +decides. See +[`docs/INTEGRATION.md`](INTEGRATION.md) §6.4 for the production +patterns. + +--- + +## End of Part 3 + +You now know every layer of the runtime: + +- The repository layout (chapter 30). +- The spec-as-data philosophy + codegen (chapter 31). +- The stores + data model (chapter 32). +- The transport implementation (chapter 33). +- The codec + SML (chapter 34). +- Router + state machines + dispatch (chapter 35). +- Persistence + validation + metrics (this chapter). + +Part 4 turns to operations — how a customer actually builds, runs, +deploys, and integrates this codebase into a real fab tool. + +Next: [→ 40 Building, running, the demo](40_building_running_demo.md)