docs: chapters 30–36 — the codebase (Part 3 complete)
Seven chapters walking the implementation top-to-bottom. 30 — Repository tour. Top-level layout, directory by directory. The eight built binaries. The dependency graph from TCP socket up through EquipmentDataModel. CMake's role. Test layout. 31 — Spec-as-data and codegen. Why the design choice fits SECS/ GEM specifically. The five YAML files: messages catalog, control/PJ/CJ transition tables, equipment dictionary. How tools/gen_messages.py turns messages.yaml into typed C++ at build time. The --validate-config multi-error validator. How to add a new SVID / CEID / host command / state / message without C++. 32 — Stores and the data model. What a store IS (records + API + change handler + optional persistence). Every store in the codebase mapped to the SEMI standard it serves (table of 21). EquipmentDataModel as plain composition + cross-store convenience methods (vid_value, compose_reports_for). The no-locks single- threaded contract. How to add a new store. 33 — Transport. hsms::Connection read path (length+payload async chain), write path (queue + one outstanding write), timer model (5 steady_timers + per-request T3). The asio executor / strand model and why it's the right shape. secsi::Protocol as the IO- free FSM with Action / Event variants; secsi::TcpTransport as the asio adapter. Pattern repeats for E84 + GEM comm-state. 34 — Codec and SML. The four files (170 + 30 + 52 + 32 lines of header, 229 + 220 lines of impl). Item variant storage layout (11 alternatives, 16 formats, shared storage where E5 permits). encode_into recursion; decode_at with bounds checks throwing CodecError. Message wrapper. SML printer + try_parse_sml + why SML round-trips Items but not necessarily bytes. 35 — State machines and dispatch. gem::Router as a typed (stream, function) dispatch table. How an S2F41 round-trip walks through parser → store dispatch → side-effect → CEID emission → S6F11 build → spool-aware deliver. The 11 FSMs all sharing the same three-property shape (pure data table + pure FSM + observer pattern). CEID cascading from FSM transitions to wire bytes. 36 — Persistence, validation, metrics. Which 7 stores have file journals + why the others don't. Per-record file pattern (atomic rename, partial-write safe). Schema versioning + multi-version read. Multi-error YAML validator (--validate-config) + cross-file reference checks. Prometheus registry + HTTP exporter + worked metric patterns from the PVD example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,231 @@
|
|||||||
|
# 30 — Repository tour
|
||||||
|
|
||||||
|
← [19 E42 + E148 + S9 — Misc](19_e42_e148_s9_misc.md) | [Back to index](00_index.md) | Next: [31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md) →
|
||||||
|
|
||||||
|
You've seen what every SEMI standard *does*. Now we shift to how
|
||||||
|
this **codebase** is laid out. This chapter answers: when you
|
||||||
|
`git clone` this repo, what are you looking at?
|
||||||
|
|
||||||
|
The repo is small — about 15 k lines of C++ + tests + tooling. It
|
||||||
|
fits in your head with a little patience. By the end of this
|
||||||
|
chapter you'll know:
|
||||||
|
|
||||||
|
- What each top-level directory contains.
|
||||||
|
- Which binaries get built.
|
||||||
|
- The dependency graph between modules.
|
||||||
|
- How the build system finds and links them.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Top-level layout
|
||||||
|
|
||||||
|
```
|
||||||
|
secs-gem/
|
||||||
|
├── README.md One-page project summary.
|
||||||
|
├── LICENSE Proprietary terms.
|
||||||
|
├── CMakeLists.txt Build config (CMake 3.16+, single file).
|
||||||
|
├── Dockerfile Ubuntu 24.04 + g++-13 + libasio + yaml-cpp.
|
||||||
|
├── docker-compose.yml Multi-container demo wiring.
|
||||||
|
├── .gitea/workflows/ci.yml CI pipeline.
|
||||||
|
│
|
||||||
|
├── include/secsgem/ Public headers. All API here.
|
||||||
|
│ ├── secs2/ E5 codec + SML.
|
||||||
|
│ ├── hsms/ E37 transport (TCP + framing).
|
||||||
|
│ ├── secsi/ E4 transport (FSM + TCP tunnel).
|
||||||
|
│ ├── config/ YAML loader + multi-error validator.
|
||||||
|
│ ├── metrics/ Prometheus exporter.
|
||||||
|
│ ├── endpoint.hpp asio::ip::tcp::endpoint factory.
|
||||||
|
│ └── gem/ E30 + every GEM 300 standard.
|
||||||
|
│ ├── store/ Per-domain bundles (SVIDs, alarms, …).
|
||||||
|
│ └── *.hpp State machines + composers.
|
||||||
|
│
|
||||||
|
├── src/ Implementations. Mirrors include/.
|
||||||
|
│ ├── secs2/{codec,sml}.cpp
|
||||||
|
│ ├── hsms/{header,connection}.cpp
|
||||||
|
│ ├── secsi/{header,block,protocol,tcp_transport}.cpp
|
||||||
|
│ ├── config/...
|
||||||
|
│ ├── gem/...
|
||||||
|
│ └── endpoint.cpp
|
||||||
|
│
|
||||||
|
├── apps/ Standalone binaries.
|
||||||
|
│ ├── secs_server.cpp Passive equipment (demo + integration target).
|
||||||
|
│ ├── secs_client.cpp Active host driving the demo flow.
|
||||||
|
│ ├── secs_conformance.cpp 47-check wire-level conformance harness.
|
||||||
|
│ ├── secs_interop_probe.cpp Probe against secsgem-py passive equip.
|
||||||
|
│ ├── secs_bench.cpp Throughput / latency / memory bench.
|
||||||
|
│ ├── fuzz_secs2_decode.cpp libFuzzer harness for secs2::decode.
|
||||||
|
│ └── fuzz_sml_parse.cpp libFuzzer harness for try_parse_sml.
|
||||||
|
│
|
||||||
|
├── tests/ doctest unit + integration tests.
|
||||||
|
│ └── test_*.cpp 50 files, 445 cases, 2753 assertions.
|
||||||
|
│
|
||||||
|
├── data/ YAML configs (the spec-as-data).
|
||||||
|
│ ├── messages.yaml SECS-II message catalog (164 msgs).
|
||||||
|
│ ├── control_state.yaml E30 §6.2 transition table.
|
||||||
|
│ ├── process_job_state.yaml E40 transition table.
|
||||||
|
│ ├── control_job_state.yaml E94 transition table.
|
||||||
|
│ └── equipment.yaml Demo SVIDs/ECIDs/CEIDs/alarms/recipes.
|
||||||
|
│
|
||||||
|
├── tools/ Build-time scripts.
|
||||||
|
│ └── generate_messages.py Codegen: messages.yaml → messages.hpp.
|
||||||
|
│
|
||||||
|
├── interop/ External-validator harnesses.
|
||||||
|
│ ├── README.md Harness-by-harness detail.
|
||||||
|
│ ├── host_vs_cpp_server.py secsgem-py active host driving us.
|
||||||
|
│ ├── passive_equipment.py secsgem-py passive equipment for us to drive.
|
||||||
|
│ ├── raw_gem300_harness.py Raw S3/S14/S16/S12 round-trip.
|
||||||
|
│ ├── tshark_validate.sh pcap + tshark HSMS dissector check.
|
||||||
|
│ ├── secs4j_validate.sh secs4java8 (Java) cross-validation.
|
||||||
|
│ └── secs4j/ Dockerfile + harness for secs4java8.
|
||||||
|
│
|
||||||
|
├── examples/
|
||||||
|
│ └── pvd_tool/ Worked vendor example: fictional PVD tool.
|
||||||
|
│ ├── README.md What the example shows.
|
||||||
|
│ ├── equipment.yaml Realistic SVIDs/ECIDs/CEIDs/alarms/recipes.
|
||||||
|
│ └── main.cpp Sensor sim, recipe runner, alarm monitor.
|
||||||
|
│
|
||||||
|
└── docs/ This guide + reference docs.
|
||||||
|
├── 00_index.md The series TOC.
|
||||||
|
├── 01–51_*.md Tutorial chapters.
|
||||||
|
├── ARCHITECTURE.md One-page architecture overview.
|
||||||
|
├── COMPLIANCE.md Per-capability audit.
|
||||||
|
├── INTEGRATION.md Vendor-side production deploy.
|
||||||
|
├── PROOFS.md 8 commands proving feature-completeness.
|
||||||
|
├── VERIFICATION.md External-validator test plan.
|
||||||
|
├── BENCHMARKS.md Performance envelope.
|
||||||
|
├── MES_INTEROP.md Commercial-MES day-1 punch list.
|
||||||
|
├── SECURITY.md nftables / stunnel / minisign configs.
|
||||||
|
├── GLOSSARY.md SEMI vocabulary cheat sheet.
|
||||||
|
└── FAQ.md Canonical answers.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The dependency graph
|
||||||
|
|
||||||
|
```
|
||||||
|
data/*.yaml
|
||||||
|
│
|
||||||
|
┌─────────────┼──────────────────┐
|
||||||
|
│ (codegen) │ (runtime load) │
|
||||||
|
▼ ▼ ▼
|
||||||
|
generated/messages.hpp config::loader
|
||||||
|
│ │
|
||||||
|
└──────────► gem::EquipmentDataModel
|
||||||
|
│
|
||||||
|
│ used by
|
||||||
|
▼
|
||||||
|
gem::Router
|
||||||
|
│
|
||||||
|
│ wraps
|
||||||
|
▼
|
||||||
|
secs2::Message ◄─── codec / SML
|
||||||
|
│
|
||||||
|
│ over
|
||||||
|
▼
|
||||||
|
hsms::Connection / secsi::TcpTransport
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
TCP socket
|
||||||
|
```
|
||||||
|
|
||||||
|
Read it bottom-up: a TCP socket carries bytes; `hsms::Connection`
|
||||||
|
frames them into `secs2::Message`s; `gem::Router` dispatches by
|
||||||
|
`(stream, function)` to handlers; handlers read/write
|
||||||
|
`EquipmentDataModel`; the model composes per-domain stores; the
|
||||||
|
stores were built from the YAML at startup.
|
||||||
|
|
||||||
|
No layer ever calls *up* the graph. `secs2::Item` has no idea
|
||||||
|
HSMS exists. `hsms::Connection` doesn't know about CEIDs.
|
||||||
|
`gem::Router` doesn't know whether the bytes came over HSMS or
|
||||||
|
SECS-I. Strict layering is what keeps the codebase small.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The binaries
|
||||||
|
|
||||||
|
Built by [`CMakeLists.txt`](../CMakeLists.txt) (one file, ~250
|
||||||
|
lines). Each binary lives in `build/` after `cmake --build`.
|
||||||
|
|
||||||
|
| Binary | Source | What it does |
|
||||||
|
|----------------------|-----------------------------------------------------------------|-------------------------------------------------------|
|
||||||
|
| `secs_server` | [`apps/secs_server.cpp`](../apps/secs_server.cpp) | Passive equipment. Listens on TCP, dispatches via Router. |
|
||||||
|
| `secs_client` | [`apps/secs_client.cpp`](../apps/secs_client.cpp) | Active host. Drives ~24 transactions in the demo. |
|
||||||
|
| `secs_conformance` | [`apps/secs_conformance.cpp`](../apps/secs_conformance.cpp) | 47 wire-level conformance checks against a live server. |
|
||||||
|
| `secs_interop_probe` | [`apps/secs_interop_probe.cpp`](../apps/secs_interop_probe.cpp) | Active host probing a secsgem-py passive equipment. |
|
||||||
|
| `secs_bench` | [`apps/secs_bench.cpp`](../apps/secs_bench.cpp) | Throughput / latency / memory harness. |
|
||||||
|
| `secsgem_tests` | All `tests/*.cpp` | The 445-case doctest binary. |
|
||||||
|
| `fuzz_secs2_decode` | [`apps/fuzz_secs2_decode.cpp`](../apps/fuzz_secs2_decode.cpp) | libFuzzer (clang only, opt-in `-DSECSGEM_FUZZ=ON`). |
|
||||||
|
| `fuzz_sml_parse` | [`apps/fuzz_sml_parse.cpp`](../apps/fuzz_sml_parse.cpp) | libFuzzer for the SML parser. |
|
||||||
|
|
||||||
|
A worked example binary `pvd_tool` (from `examples/pvd_tool/`) is
|
||||||
|
also built by the same `CMakeLists.txt` when the example is
|
||||||
|
included.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How the build system finds everything
|
||||||
|
|
||||||
|
`CMakeLists.txt` does five things in order:
|
||||||
|
|
||||||
|
1. **Pull in dependencies** — `find_package(Threads)`,
|
||||||
|
`find_package(yaml-cpp)`, `FetchContent` for doctest. Standalone
|
||||||
|
Asio is header-only (no link step).
|
||||||
|
2. **Run codegen** — invokes `tools/generate_messages.py` to turn
|
||||||
|
`data/messages.yaml` into `build/generated/secsgem/gem/messages.hpp`.
|
||||||
|
Listed as a custom command so it re-runs when `messages.yaml`
|
||||||
|
changes.
|
||||||
|
3. **Build the library** — `add_library(secsgem ...)` with every
|
||||||
|
source under `src/` plus the generated header.
|
||||||
|
4. **Build the apps** — one `add_executable` per `apps/*.cpp`,
|
||||||
|
each linking against `secsgem`.
|
||||||
|
5. **Build the tests** — `add_executable(secsgem_tests ...)` with
|
||||||
|
every `tests/*.cpp`, linked against doctest + `secsgem`.
|
||||||
|
|
||||||
|
Build flags:
|
||||||
|
|
||||||
|
- **`-DSECSGEM_TSAN=ON`** — adds `-fsanitize=thread` to a
|
||||||
|
separate build dir. CI runs this lane.
|
||||||
|
- **`-DSECSGEM_FUZZ=ON`** — requires clang; adds libFuzzer + ASan +
|
||||||
|
UBSan; builds the two fuzz harnesses.
|
||||||
|
|
||||||
|
Everything else (Release / Debug, parallelism, output dirs) is
|
||||||
|
standard CMake.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Test layout
|
||||||
|
|
||||||
|
50 test files; 445 test cases; 2 753 assertions. One file per
|
||||||
|
concern. Naming is `test_<thing>.cpp` consistently:
|
||||||
|
|
||||||
|
- `test_secs2.cpp`, `test_e5_kat.cpp`, `test_sml.cpp`,
|
||||||
|
`test_messages.cpp` — codec.
|
||||||
|
- `test_hsms*.cpp` (5 files), `test_secsi*.cpp` (3 files) — transport.
|
||||||
|
- `test_control_state.cpp`, `test_communication_state.cpp`,
|
||||||
|
`test_data_model.cpp`, `test_host_handler.cpp`, `test_loader.cpp`,
|
||||||
|
`test_config_validate.cpp` — E30.
|
||||||
|
- `test_process_jobs.cpp`, `test_control_jobs.cpp`,
|
||||||
|
`test_carriers.cpp`, `test_substrates.cpp`, `test_ept.cpp`,
|
||||||
|
`test_modules.cpp`, `test_cem_objects.cpp`, `test_e84*.cpp`,
|
||||||
|
`test_e42_formatted_pp.cpp` — GEM 300.
|
||||||
|
- `test_*_persistence.cpp` (4) — file-backed journal.
|
||||||
|
- `test_robustness_fuzz.cpp` — randomized property test.
|
||||||
|
- `test_thread_safety.cpp` — TSan-validated single-threaded contract.
|
||||||
|
- `test_metrics_prometheus.cpp` — Prometheus exporter.
|
||||||
|
- `test_wire_ceid_emission.cpp` — CEID firings observed on a real socket.
|
||||||
|
- `test_live_gem300.cpp`, `test_gem300_scenario.cpp` — multi-FSM cascades.
|
||||||
|
|
||||||
|
Full per-standard breakdown:
|
||||||
|
[`docs/PROOFS.md`](PROOFS.md) "Per-standard test coverage" table.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Where to go next
|
||||||
|
|
||||||
|
Now that you know what's where, the next chapter explains the
|
||||||
|
*philosophy* that makes the codebase this small: the **spec-as-data**
|
||||||
|
principle, and how the YAML files + codegen + runtime loader work
|
||||||
|
together so adding a new SVID / state / message rarely requires C++.
|
||||||
|
|
||||||
|
Next: [→ 31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md)
|
||||||
@@ -0,0 +1,395 @@
|
|||||||
|
# 31 — Spec-as-data and codegen
|
||||||
|
|
||||||
|
← [30 Repository tour](30_repository_tour.md) | [Back to index](00_index.md) | Next: [32 Stores and the data model](32_stores_and_the_data_model.md) →
|
||||||
|
|
||||||
|
The single design choice that keeps this codebase small is
|
||||||
|
**spec-as-data**: every SEMI behavioural rule, state-transition
|
||||||
|
table, and message body shape lives in YAML. The C++ is the engine
|
||||||
|
that reads them.
|
||||||
|
|
||||||
|
This chapter explains:
|
||||||
|
|
||||||
|
- Why spec-as-data is the right choice for SECS/GEM specifically.
|
||||||
|
- The five YAML files that drive everything.
|
||||||
|
- How the message catalog gets codegen'd into typed C++.
|
||||||
|
- How transition tables and equipment dictionaries load at runtime.
|
||||||
|
- How to add a new SVID / state / message / host command without
|
||||||
|
touching C++.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Why spec-as-data here
|
||||||
|
|
||||||
|
Three properties of SECS/GEM standards push toward data-driven
|
||||||
|
implementation:
|
||||||
|
|
||||||
|
1. **Per-tool variation is enormous.** Every fab tool has its own
|
||||||
|
list of SVIDs, ECIDs, CEIDs, alarms. Hardcoding even one of
|
||||||
|
them in C++ would mean recompiling per deployment.
|
||||||
|
2. **The standards themselves are largely declarative.** E30 §6.2
|
||||||
|
is a 5×8 transition table, not an algorithm. E5 §9 is a
|
||||||
|
format-byte arithmetic. E40 §6 is a state graph. These map to
|
||||||
|
YAML cleanly.
|
||||||
|
3. **Customers need to audit the rules.** A fab QA team can read
|
||||||
|
`data/control_state.yaml` and see the transitions; they can't
|
||||||
|
read 600 lines of `if/else` and trust it the same way.
|
||||||
|
|
||||||
|
So the rule is: **anything a vendor or customer might want to
|
||||||
|
change without recompiling lives in YAML**. The C++ is the
|
||||||
|
runtime that reads it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The five YAML files
|
||||||
|
|
||||||
|
All under [`data/`](../data/).
|
||||||
|
|
||||||
|
### `messages.yaml` — the SECS-II message catalog
|
||||||
|
|
||||||
|
164 entries. Each one names a SECS-II message (SxFy) and describes
|
||||||
|
its body's typed shape. Used at **build time** by codegen.
|
||||||
|
|
||||||
|
Excerpt:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- id: S1F1
|
||||||
|
stream: 1
|
||||||
|
function: 1
|
||||||
|
w: true
|
||||||
|
builder: s1f1
|
||||||
|
parser: parse_s1f1
|
||||||
|
body: none
|
||||||
|
|
||||||
|
- id: S1F2
|
||||||
|
stream: 1
|
||||||
|
function: 2
|
||||||
|
w: false
|
||||||
|
builder: s1f2
|
||||||
|
parser: parse_s1f2
|
||||||
|
body:
|
||||||
|
kind: list
|
||||||
|
struct_name: OnlineIdentification
|
||||||
|
fields:
|
||||||
|
- {name: mdln, shape: {kind: scalar, item_type: ASCII}}
|
||||||
|
- {name: softrev, shape: {kind: scalar, item_type: ASCII}}
|
||||||
|
```
|
||||||
|
|
||||||
|
The `body` field is the codegen's input. It supports:
|
||||||
|
|
||||||
|
- **`none`** — header-only, no body.
|
||||||
|
- **`scalar`** — one Item; the codegen picks an appropriate C++
|
||||||
|
parameter type from `item_type` (`ASCII`, `U1`–`U8`, etc.).
|
||||||
|
- **`list`** — fixed-arity `<L,k>` with named fields. Optionally
|
||||||
|
generates a `struct StructName { … }`.
|
||||||
|
- **`list_of`** — variable-arity `<L,n>` with a uniform element
|
||||||
|
shape.
|
||||||
|
|
||||||
|
The grammar is documented at the top of
|
||||||
|
[`tools/gen_messages.py`](../tools/gen_messages.py) and at the top
|
||||||
|
of [`data/messages.yaml`](../data/messages.yaml).
|
||||||
|
|
||||||
|
### `control_state.yaml` — the E30 §6.2 control state transition table
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
transitions:
|
||||||
|
- {from: EquipmentOffline, on: operator_switch_online, to: AttemptOnline, then: OnlineRemote}
|
||||||
|
- {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept}
|
||||||
|
- {from: OnlineLocal, on: host_request_remote, ack: NotAccept}
|
||||||
|
```
|
||||||
|
|
||||||
|
Loaded at **runtime** by `config::load_control_state_table`. The
|
||||||
|
default table — used by tests when no YAML is given —
|
||||||
|
mirrors this file exactly (in
|
||||||
|
`ControlTransitionTable::default_table()`).
|
||||||
|
|
||||||
|
### `process_job_state.yaml` — the E40 PJ transition table
|
||||||
|
|
||||||
|
Same shape as control_state.yaml but for PJs. Drives
|
||||||
|
`ProcessJobStateMachine`.
|
||||||
|
|
||||||
|
### `control_job_state.yaml` — the E94 CJ transition table
|
||||||
|
|
||||||
|
Same shape for CJs. Drives `ControlJobStateMachine`.
|
||||||
|
|
||||||
|
### `equipment.yaml` — the demo equipment data dictionary
|
||||||
|
|
||||||
|
Excerpt:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
device:
|
||||||
|
mdln: "SECS-GEM Demo Equipment"
|
||||||
|
softrev: "1.0.0"
|
||||||
|
capabilities: [Establish, OnLine, …]
|
||||||
|
|
||||||
|
svids:
|
||||||
|
- {id: 1, name: ControlState, units: "", type: ASCII, value: ""}
|
||||||
|
- {id: 2, name: Clock, units: "", type: ASCII, value: ""}
|
||||||
|
- {id: 3, name: WaferCounter, units: "wafers", type: U4, value: 0}
|
||||||
|
|
||||||
|
ecids:
|
||||||
|
- {id: 100, name: T3, units: "s", type: U4, value: 45, min: 1, max: 600}
|
||||||
|
|
||||||
|
ceids:
|
||||||
|
- {id: 100, name: ControlStateChange}
|
||||||
|
- {id: 300, name: ProcessStarted}
|
||||||
|
|
||||||
|
alarms:
|
||||||
|
- {id: 1, alcd: 0x84, text: "Chamber pressure above threshold"}
|
||||||
|
|
||||||
|
host_commands:
|
||||||
|
- {name: START, ack: Accept, emit_ceid: 300}
|
||||||
|
- {name: FAULT, ack: Accept, set_alarm: 1}
|
||||||
|
```
|
||||||
|
|
||||||
|
Loaded at startup by `config::load_equipment`. Every key under
|
||||||
|
this YAML maps to a typed struct in `config::EquipmentDescriptor`.
|
||||||
|
|
||||||
|
`examples/pvd_tool/equipment.yaml` is a more realistic version
|
||||||
|
with 29 SVIDs, 7 ECIDs, 21 CEIDs, 12 alarms.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The codegen pass
|
||||||
|
|
||||||
|
`messages.yaml` is too large and too repetitive to write by hand —
|
||||||
|
164 messages × (builder + parser + struct + tests) would be ~5 k
|
||||||
|
lines of boilerplate. Instead, `tools/gen_messages.py` reads the
|
||||||
|
YAML at build time and emits one inline header:
|
||||||
|
**`build/generated/secsgem/gem/messages.hpp`**.
|
||||||
|
|
||||||
|
### What gets generated
|
||||||
|
|
||||||
|
Per message, the codegen emits:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
namespace secsgem::gem::messages {
|
||||||
|
|
||||||
|
// Optional struct if body has `struct_name`.
|
||||||
|
struct OnlineIdentification {
|
||||||
|
std::string mdln;
|
||||||
|
std::string softrev;
|
||||||
|
bool operator==(const OnlineIdentification&) const = default;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Builder: takes typed params, returns a secs2::Message.
|
||||||
|
inline secs2::Message s1f1();
|
||||||
|
inline secs2::Message s1f2(const std::string& mdln, const std::string& softrev);
|
||||||
|
|
||||||
|
// Parser: takes a Message body, returns std::optional<Struct> (or the
|
||||||
|
// primitive type for scalar bodies).
|
||||||
|
inline std::optional<OnlineIdentification> parse_s1f2(const secs2::Item& body);
|
||||||
|
|
||||||
|
} // namespace
|
||||||
|
```
|
||||||
|
|
||||||
|
For ~160 named messages, the generated header is ~3 500 lines, all
|
||||||
|
inline. Tests in
|
||||||
|
[`tests/test_messages.cpp`](../tests/test_messages.cpp) (82 cases)
|
||||||
|
exercise every builder + parser round-trip.
|
||||||
|
|
||||||
|
### How CMake invokes it
|
||||||
|
|
||||||
|
CMakeLists.txt has a custom command:
|
||||||
|
|
||||||
|
```cmake
|
||||||
|
add_custom_command(
|
||||||
|
OUTPUT ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp
|
||||||
|
COMMAND ${Python3_EXECUTABLE}
|
||||||
|
${CMAKE_SOURCE_DIR}/tools/gen_messages.py
|
||||||
|
${CMAKE_SOURCE_DIR}/data/messages.yaml
|
||||||
|
${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp
|
||||||
|
DEPENDS ${CMAKE_SOURCE_DIR}/data/messages.yaml
|
||||||
|
${CMAKE_SOURCE_DIR}/tools/gen_messages.py
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-runs on `data/messages.yaml` edits *or* on
|
||||||
|
`tools/gen_messages.py` edits. Generated header goes into a
|
||||||
|
sibling include directory so the library can include it as
|
||||||
|
`#include "secsgem/gem/messages.hpp"`.
|
||||||
|
|
||||||
|
### Why Python rather than templates / constexpr
|
||||||
|
|
||||||
|
Three reasons:
|
||||||
|
|
||||||
|
1. **YAML parsing** — full grammar matters and `PyYAML` is more
|
||||||
|
reliable than yaml-cpp at parse-time gymnastics.
|
||||||
|
2. **Code shape control** — the generated C++ is easier to read
|
||||||
|
when generated by a textual templater than by C++ metaprogramming.
|
||||||
|
3. **Debuggability** — a customer who wants to see "what code is
|
||||||
|
actually being run for S2F33" can `grep` the generated header.
|
||||||
|
No mystery types, no instantiation chains.
|
||||||
|
|
||||||
|
The codegen is ~388 lines of Python; the input grammar is
|
||||||
|
documented at its top.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Runtime loading
|
||||||
|
|
||||||
|
The other four YAMLs (`control_state`, `process_job_state`,
|
||||||
|
`control_job_state`, `equipment`) load at runtime, not build time.
|
||||||
|
The same loader handles all of them:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// include/secsgem/config/loader.hpp
|
||||||
|
namespace secsgem::config {
|
||||||
|
EquipmentDescriptor load_equipment(const std::string& path);
|
||||||
|
ControlStateConfig load_control_state_table(const std::string& path);
|
||||||
|
ProcessJobStateConfig load_process_job_state(const std::string& path);
|
||||||
|
ControlJobStateConfig load_control_job_state(const std::string& path);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Each `load_*` returns a typed config struct on success or throws on
|
||||||
|
malformed YAML. Throwing is OK because YAML loading happens once
|
||||||
|
at startup — before binding the port — so a malformed file fails
|
||||||
|
the process up front.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The `--validate-config` pass
|
||||||
|
|
||||||
|
YAML loaders that throw on first error are unfriendly: customers
|
||||||
|
often have multiple typos in a new equipment.yaml. The codebase
|
||||||
|
ships a multi-error validator:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// include/secsgem/config/validate.hpp
|
||||||
|
class ConfigValidator {
|
||||||
|
public:
|
||||||
|
void validate_equipment(const std::string& path);
|
||||||
|
void validate_control_state(const std::string& path);
|
||||||
|
// ...
|
||||||
|
|
||||||
|
bool has_errors() const;
|
||||||
|
void format_issues_to(std::ostream&, …) const;
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
It tries to load each file, accumulates *every* issue it can find,
|
||||||
|
and prints them all. Then exits 0 or 1.
|
||||||
|
|
||||||
|
Invoke via:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
secs_server --validate-config \
|
||||||
|
--config data/equipment.yaml \
|
||||||
|
--state-table data/control_state.yaml \
|
||||||
|
--pj-state-table data/process_job_state.yaml \
|
||||||
|
--cj-state-table data/control_job_state.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
This is proof #5 in [PROOFS.md](PROOFS.md) — runs in CI to
|
||||||
|
guarantee every shipped YAML is structurally + referentially
|
||||||
|
sound.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
[`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp)
|
||||||
|
(8 cases — every category of validation issue).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How to add a capability without C++
|
||||||
|
|
||||||
|
The point of spec-as-data is that **adding behaviour almost never
|
||||||
|
requires a C++ change**.
|
||||||
|
|
||||||
|
### New SVID
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# data/equipment.yaml
|
||||||
|
svids:
|
||||||
|
- {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}
|
||||||
|
```
|
||||||
|
|
||||||
|
Restart. Done. Host can now read SVID 4 via S1F3.
|
||||||
|
|
||||||
|
### New CEID with linked report
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# data/equipment.yaml
|
||||||
|
ceids:
|
||||||
|
- {id: 350, name: ChamberTempHigh}
|
||||||
|
|
||||||
|
events:
|
||||||
|
default_reports:
|
||||||
|
- {ceid: 350, vids: [4]}
|
||||||
|
```
|
||||||
|
|
||||||
|
Restart. Done. When the EAP fires CEID 350, the report carries
|
||||||
|
SVID 4 automatically.
|
||||||
|
|
||||||
|
### New host command
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
host_commands:
|
||||||
|
- {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}
|
||||||
|
```
|
||||||
|
|
||||||
|
Restart. Done. Host sends `S2F41(RCMD=VENT)` → ACK=Accept,
|
||||||
|
CEID 400 fires, ALID 2 set.
|
||||||
|
|
||||||
|
### New control-state transition
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# data/control_state.yaml
|
||||||
|
transitions:
|
||||||
|
- {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept}
|
||||||
|
```
|
||||||
|
|
||||||
|
Restart. Done.
|
||||||
|
|
||||||
|
### New SECS-II message
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# data/messages.yaml
|
||||||
|
- id: S6F30
|
||||||
|
stream: 6
|
||||||
|
function: 30
|
||||||
|
w: true
|
||||||
|
builder: s6f30_request
|
||||||
|
parser: parse_s6f30
|
||||||
|
body:
|
||||||
|
kind: list
|
||||||
|
struct_name: TempQuery
|
||||||
|
fields:
|
||||||
|
- {name: vid, shape: {kind: scalar, item_type: U4}}
|
||||||
|
```
|
||||||
|
|
||||||
|
`docker compose run --rm builder` regenerates `messages.hpp`. A
|
||||||
|
new `s6f30_request(uint32_t vid)` builder and a `parse_s6f30(item)
|
||||||
|
→ std::optional<TempQuery>` parser appear. Now the *handler* is
|
||||||
|
still C++ — `gem::Router::on(6, 30, ...)` — because the side-effect
|
||||||
|
of "host asked for the temperature" needs application logic.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## When spec-as-data isn't the right fit
|
||||||
|
|
||||||
|
Three categories that *do* need C++:
|
||||||
|
|
||||||
|
1. **Application logic** — what an alarm threshold actually is,
|
||||||
|
how a recipe step gets executed. No YAML schema can express
|
||||||
|
"vent the chamber if pressure > 1 Torr."
|
||||||
|
2. **State-machine actions** — when a CJ transitions to Executing,
|
||||||
|
*which* PJ to select next isn't a table entry; it's an
|
||||||
|
algorithm.
|
||||||
|
3. **External integrations** — talking to a PLC, reading a sensor,
|
||||||
|
driving a robot. Hardware bindings are vendor-specific code.
|
||||||
|
|
||||||
|
The codebase draws the line **at the message catalog and the
|
||||||
|
transition tables**. Everything below (codec, transport) is fixed
|
||||||
|
C++. Everything above (application wiring) is per-EAP C++.
|
||||||
|
Everything between (data dictionary + state model) is YAML.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Where to go next
|
||||||
|
|
||||||
|
You now know how the YAML drives the runtime. The next chapter
|
||||||
|
gets concrete about the **stores** — the per-domain bundles
|
||||||
|
(SVIDs, CEIDs, alarms, carriers, …) that the YAML populates and
|
||||||
|
the Router handlers operate over.
|
||||||
|
|
||||||
|
Next: [→ 32 Stores and the data model](32_stores_and_the_data_model.md)
|
||||||
@@ -0,0 +1,308 @@
|
|||||||
|
# 32 — Stores and the data model
|
||||||
|
|
||||||
|
← [31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md) | [Back to index](00_index.md) | Next: [33 Transport](33_transport.md) →
|
||||||
|
|
||||||
|
The previous chapter showed how YAML drives behaviour. This
|
||||||
|
chapter shows the runtime data structures that the YAML populates
|
||||||
|
and that the Router handlers operate on: **stores** and the
|
||||||
|
**`EquipmentDataModel`** that composes them.
|
||||||
|
|
||||||
|
By the end you'll know:
|
||||||
|
|
||||||
|
- What a **store** is (and what it isn't).
|
||||||
|
- Every store in the codebase, one sentence each.
|
||||||
|
- How `EquipmentDataModel` composes them.
|
||||||
|
- The "no locks; single-threaded" contract.
|
||||||
|
- How to add a new store.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What a store is
|
||||||
|
|
||||||
|
A **store** is a per-domain bundle of:
|
||||||
|
|
||||||
|
- A few typed records (`std::map`, `std::vector`, …).
|
||||||
|
- A small API for reading + mutating them.
|
||||||
|
- A `change_handler` that emits events on transitions.
|
||||||
|
- Optional file-backed persistence.
|
||||||
|
|
||||||
|
The naming is consistent: `AlarmRegistry` for active alarms,
|
||||||
|
`CarrierStore` for carriers, `ProcessJobStore` for PJs.
|
||||||
|
Headers live in
|
||||||
|
[`include/secsgem/gem/store/`](../include/secsgem/gem/store/);
|
||||||
|
implementations are typically inline in the header (these are small).
|
||||||
|
|
||||||
|
Each store maps onto **one concern from one SEMI standard**:
|
||||||
|
|
||||||
|
| Store | Concern | Standard |
|
||||||
|
|-----------------------------|--------------------------------------------------|----------------|
|
||||||
|
| `StatusVariableStore` | SVIDs + values | E30 §6.13 |
|
||||||
|
| `DataVariableStore` | DVIDs + values | E30 §6.11 |
|
||||||
|
| `EquipmentConstantStore` | ECIDs + values + min/max bounds | E30 §6.16 |
|
||||||
|
| `EventReportSubscriptions` | RPTID definitions + CEID linkings + enables | E30 §6.6 |
|
||||||
|
| `AlarmRegistry` | ALIDs + ALCD/ALTX + enable bits + active set | E30 §6.14 |
|
||||||
|
| `RecipeStore` | PPIDs + PPBODY (unformatted) + formatted bodies | E30 §6.17 + E42|
|
||||||
|
| `Clock` | Wall-clock + drift + quality | E30 §6.20 + E148 |
|
||||||
|
| `HostCommandRegistry` | RCMD names + per-command ack + side effects | E30 §6.15 |
|
||||||
|
| `SpoolStore` | Per-stream whitelist + queue + persistent journal| E30 §6.22 |
|
||||||
|
| `LimitMonitorStore` | LIMITIDs + upper/lower bounds + active state | E30 §6.21 |
|
||||||
|
| `TraceStore` | TRIDs + active sampling config | E30 §6.12 |
|
||||||
|
| `ProcessJobStore` | PJs + state + material list + persistent | E40 |
|
||||||
|
| `ControlJobStore` | CJs + state + PJ refs + persistent | E94 |
|
||||||
|
| `ExceptionStore` | EXIDs + recovery state + persistent | E5 §13 |
|
||||||
|
| `CarrierStore` | Carrier IDs + state machines + persistent | E87 |
|
||||||
|
| `LoadPortStore` | LP IDs + transfer/reservation/association FSMs | E87 |
|
||||||
|
| `SubstrateStore` | Substrate IDs + 3 FSMs + location + persistent | E90 |
|
||||||
|
| `EptStateMachine` | EPT state + time buckets | E116 |
|
||||||
|
| `CemObjectStore` | E120 typed object hierarchy | E120 |
|
||||||
|
| `ModuleStore` | Module IDs + state | E157 |
|
||||||
|
| `E84PortStore` | Per-LP E84 FSM + signals + timers | E84 |
|
||||||
|
|
||||||
|
Each is a header. Each is independently testable: you can
|
||||||
|
`#include "secsgem/gem/store/alarms.hpp"` and exercise
|
||||||
|
`AlarmRegistry` without pulling in the rest. This is the same
|
||||||
|
shape as the per-standard tests in
|
||||||
|
[`tests/`](../tests/) — one test file per store.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## EquipmentDataModel — the composite
|
||||||
|
|
||||||
|
[`include/secsgem/gem/data_model.hpp`](../include/secsgem/gem/data_model.hpp)
|
||||||
|
defines:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
struct EquipmentDataModel {
|
||||||
|
StatusVariableStore svids;
|
||||||
|
DataVariableStore dvids;
|
||||||
|
EquipmentConstantStore ecids;
|
||||||
|
EventReportSubscriptions events;
|
||||||
|
AlarmRegistry alarms;
|
||||||
|
RecipeStore recipes;
|
||||||
|
Clock clock;
|
||||||
|
HostCommandRegistry commands;
|
||||||
|
SpoolStore spool;
|
||||||
|
LimitMonitorStore limits;
|
||||||
|
TraceStore traces;
|
||||||
|
ProcessJobStore process_jobs;
|
||||||
|
ControlJobStore control_jobs;
|
||||||
|
ExceptionStore exceptions;
|
||||||
|
CarrierStore carriers;
|
||||||
|
LoadPortStore load_ports;
|
||||||
|
SubstrateStore substrates;
|
||||||
|
EptStateMachine ept;
|
||||||
|
CemObjectStore cem;
|
||||||
|
ModuleStore modules;
|
||||||
|
E84PortStore e84_ports;
|
||||||
|
// ... convenience methods spanning stores
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
That's it. No locks, no smart pointers, no interfaces, no DI
|
||||||
|
container. Each store is a value member; ownership is the
|
||||||
|
`EquipmentDataModel` itself.
|
||||||
|
|
||||||
|
The application typically holds one `shared_ptr<EquipmentDataModel>`
|
||||||
|
and passes it to every Router handler. Handlers operate on the
|
||||||
|
stores directly:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
router.on(1, 3, [model](const secs2::Message& m) {
|
||||||
|
// S1F3 — host requests SVID values
|
||||||
|
auto svids = parse_s1f3(m.body());
|
||||||
|
return build_s1f4(model->svids.values(svids));
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Convenience methods
|
||||||
|
|
||||||
|
`EquipmentDataModel` adds a few cross-store helpers (`data_model.hpp:54`):
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
std::optional<s2::Item> vid_value(uint32_t vid) const {
|
||||||
|
// Look up VID in svids first, then dvids.
|
||||||
|
}
|
||||||
|
|
||||||
|
std::vector<ReportData> compose_reports_for(uint32_t ceid) const {
|
||||||
|
// Walk events store -> reports store -> svids/dvids,
|
||||||
|
// assemble the S6F11 report payload for one CEID firing.
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`compose_reports_for` is the *heart* of event notification — it
|
||||||
|
walks three stores to assemble the body for one S6F11 frame. See
|
||||||
|
chapter [13](13_e30_gem.md) for the wire flow.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The single-threaded contract
|
||||||
|
|
||||||
|
**Every store mutation runs on the io_context strand.** No locks,
|
||||||
|
no atomics, no condition variables. This is documented in
|
||||||
|
[`docs/INTEGRATION.md`](INTEGRATION.md) §3 and enforced under
|
||||||
|
ThreadSanitizer.
|
||||||
|
|
||||||
|
Why? Two reasons:
|
||||||
|
|
||||||
|
1. **Performance.** Locking a `std::map` for every SVID read is a
|
||||||
|
waste in a hot path that processes thousands of messages a
|
||||||
|
second. The asio strand model gives the same correctness
|
||||||
|
guarantee for free.
|
||||||
|
2. **Simplicity.** Every method on every store is the obvious
|
||||||
|
non-locking implementation. Reading the code, you don't have
|
||||||
|
to track which lock protects what.
|
||||||
|
|
||||||
|
The cost: **callers from other threads must `asio::post` onto
|
||||||
|
the executor**.
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// From a sensor thread:
|
||||||
|
asio::post(io_context, [model, vid, value] {
|
||||||
|
model->svids.set_value(vid, secs2::Item::f4(value));
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
Tested by
|
||||||
|
[`tests/test_thread_safety.cpp`](../tests/test_thread_safety.cpp)
|
||||||
|
under TSan: N producer threads `asio::post` updates; TSan reports
|
||||||
|
zero races. Chapter [33](33_transport.md) covers the strand model
|
||||||
|
in more detail.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How a store's API looks (a small one)
|
||||||
|
|
||||||
|
Pick `AlarmRegistry` — one of the smallest:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
class AlarmRegistry {
|
||||||
|
public:
|
||||||
|
// Register an alarm definition.
|
||||||
|
void register_alarm(uint32_t alid, uint8_t alcd, const std::string& altx);
|
||||||
|
|
||||||
|
// Set / clear an active alarm. Fires the change handler.
|
||||||
|
void set(uint32_t alid);
|
||||||
|
void clear(uint32_t alid);
|
||||||
|
|
||||||
|
// Enable / disable host notification (S5F3).
|
||||||
|
void set_enabled(uint32_t alid, bool enabled);
|
||||||
|
bool is_enabled(uint32_t alid) const;
|
||||||
|
|
||||||
|
// List active / all alarms.
|
||||||
|
std::vector<AlarmDefinition> all() const;
|
||||||
|
std::vector<AlarmDefinition> active() const;
|
||||||
|
|
||||||
|
// Observer: change handler signature.
|
||||||
|
using ChangeHandler = std::function<void(uint32_t alid, bool set)>;
|
||||||
|
void set_change_handler(ChangeHandler);
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Every store follows that same shape: mutator + reader + observer.
|
||||||
|
The Router handler for `S5F1` doesn't fire `S5F1` itself — it
|
||||||
|
mutates the store; the change handler (registered at startup by
|
||||||
|
the EAP) fires `S5F1` via the connection.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How a store's API looks (a bigger one)
|
||||||
|
|
||||||
|
[`ProcessJobStore`](../include/secsgem/gem/store/process_jobs.hpp)
|
||||||
|
adds:
|
||||||
|
|
||||||
|
- Submit a PJ (record entry + fire `Created` event).
|
||||||
|
- Get / set state of any PJ.
|
||||||
|
- Apply a host-driven event (PJSTART / PJPAUSE / …) and route to
|
||||||
|
the FSM.
|
||||||
|
- Iterate active PJs (for serializing on restart).
|
||||||
|
- Persistent journal: `enable_persistence(dir)`.
|
||||||
|
|
||||||
|
The FSM logic isn't *inside* the store — `ProcessJobStateMachine`
|
||||||
|
in [`process_job_state.hpp`](../include/secsgem/gem/process_job_state.hpp)
|
||||||
|
owns transitions. The store holds one `ProcessJobStateMachine`
|
||||||
|
per PJ and dispatches.
|
||||||
|
|
||||||
|
This separation — *store* (records) vs *state machine* (transitions) —
|
||||||
|
keeps each layer testable in isolation.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Persistence
|
||||||
|
|
||||||
|
Six stores have file-backed persistence: spool, process_jobs,
|
||||||
|
control_jobs, exceptions, carriers, load_ports, substrates.
|
||||||
|
|
||||||
|
Each opts in via `enable_persistence(dir)`:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
model->process_jobs.enable_persistence("/var/lib/secsgem/pj");
|
||||||
|
```
|
||||||
|
|
||||||
|
That:
|
||||||
|
|
||||||
|
1. Creates the directory if needed.
|
||||||
|
2. **Replays** every record file found there back into in-memory
|
||||||
|
state.
|
||||||
|
3. Sets up the on-disk journal: every mutation writes (or rewrites,
|
||||||
|
or deletes) one file per record, named by ID.
|
||||||
|
|
||||||
|
Per-record-per-file means the journal is **partial-write safe**:
|
||||||
|
if the equipment power-cycles mid-write of one record, the others
|
||||||
|
are untouched; the partial file is detected and dropped at the
|
||||||
|
next startup.
|
||||||
|
|
||||||
|
Chapter [36](36_persistence_validation_metrics.md) walks the
|
||||||
|
mechanism, the multi-version reads, and the test patterns.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How to add a new store
|
||||||
|
|
||||||
|
Two cases:
|
||||||
|
|
||||||
|
### Case 1: Standard already implemented, new sub-area
|
||||||
|
|
||||||
|
E.g., add a "Reticle" store to track lithography reticles
|
||||||
|
distinctly from substrates.
|
||||||
|
|
||||||
|
1. Create `include/secsgem/gem/store/reticles.hpp` with a class
|
||||||
|
`ReticleStore` exposing the standard
|
||||||
|
register / set-state / get / change-handler shape.
|
||||||
|
2. Add a member to `EquipmentDataModel`:
|
||||||
|
```cpp
|
||||||
|
ReticleStore reticles;
|
||||||
|
```
|
||||||
|
3. Write `tests/test_reticles.cpp` mirroring the pattern from any
|
||||||
|
other store's test.
|
||||||
|
4. Wire Router handlers in `apps/secs_server.cpp` (or the EAP) for
|
||||||
|
whatever S/F messages drive it.
|
||||||
|
|
||||||
|
### Case 2: Brand new SEMI standard
|
||||||
|
|
||||||
|
E.g., implement E170 (a new GEM standard).
|
||||||
|
|
||||||
|
Same as case 1, plus:
|
||||||
|
|
||||||
|
5. Update [`data/messages.yaml`](../data/messages.yaml) with any
|
||||||
|
new S/F messages. `docker compose run --rm builder` regens
|
||||||
|
`messages.hpp`.
|
||||||
|
6. If E170 has its own transition table, create
|
||||||
|
`data/e170_state.yaml` and a `load_e170_state(...)` loader in
|
||||||
|
`config::`.
|
||||||
|
7. Update [`docs/COMPLIANCE.md`](COMPLIANCE.md) with the new
|
||||||
|
capability row.
|
||||||
|
|
||||||
|
The architecture is **specifically designed** to add new standards
|
||||||
|
without disturbing existing ones.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Where to go next
|
||||||
|
|
||||||
|
You've now seen how every per-domain data record is shaped and
|
||||||
|
how `EquipmentDataModel` composes them. Next, we drop back down
|
||||||
|
to transport: how `hsms::Connection` and `secsi::Protocol` actually
|
||||||
|
move bytes, and the asio strand model that makes the
|
||||||
|
single-threaded contract work.
|
||||||
|
|
||||||
|
Next: [→ 33 Transport](33_transport.md)
|
||||||
@@ -0,0 +1,292 @@
|
|||||||
|
# 33 — Transport
|
||||||
|
|
||||||
|
← [32 Stores and the data model](32_stores_and_the_data_model.md) | [Back to index](00_index.md) | Next: [34 Codec and SML](34_codec_and_sml.md) →
|
||||||
|
|
||||||
|
We covered the standards-level view of HSMS and SECS-I in
|
||||||
|
chapters 11 and 12. This chapter drops down into the
|
||||||
|
implementation: how `hsms::Connection` actually moves bytes,
|
||||||
|
the asio executor model, the single-threaded strand contract,
|
||||||
|
and why the transport layer doesn't need locks.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The two transport modules
|
||||||
|
|
||||||
|
```
|
||||||
|
include/secsgem/hsms/
|
||||||
|
├── header.hpp — Frame format primitives (length prefix, header, SType).
|
||||||
|
└── connection.hpp — One-socket session manager + T-timers + S9 emission.
|
||||||
|
|
||||||
|
include/secsgem/secsi/
|
||||||
|
├── header.hpp — 10-byte SECS-I block header.
|
||||||
|
├── block.hpp — Block split / assemble (multi-block messages).
|
||||||
|
├── protocol.hpp — IO-free line-turnaround FSM.
|
||||||
|
└── tcp_transport.hpp — asio TCP wrapper around the FSM (tunnel for testing).
|
||||||
|
```
|
||||||
|
|
||||||
|
Each module owns one TCP endpoint (or in the SECS-I case, a tunnel
|
||||||
|
endpoint). Both are **single-threaded by design**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## hsms::Connection — top to bottom
|
||||||
|
|
||||||
|
### Lifecycle
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// apps/secs_server.cpp — passive-equipment startup
|
||||||
|
asio::io_context io;
|
||||||
|
|
||||||
|
asio::ip::tcp::acceptor acc(io, asio::ip::tcp::endpoint{asio::ip::tcp::v4(), port});
|
||||||
|
|
||||||
|
acc.async_accept([&](std::error_code ec, asio::ip::tcp::socket sock) {
|
||||||
|
auto conn = std::make_shared<hsms::Connection>(
|
||||||
|
std::move(sock), Mode::Passive, /*device_id=*/0, timers);
|
||||||
|
conn->set_message_handler(...);
|
||||||
|
conn->set_closed_handler(...);
|
||||||
|
conn->start();
|
||||||
|
});
|
||||||
|
|
||||||
|
io.run(); // blocks until all work is done
|
||||||
|
```
|
||||||
|
|
||||||
|
`Connection::start()` either:
|
||||||
|
|
||||||
|
- **Passive** — arms T7 (waiting for Select.req) and starts the
|
||||||
|
read loop.
|
||||||
|
- **Active** — initiates the Select.req exchange, then starts the
|
||||||
|
read loop.
|
||||||
|
|
||||||
|
### Read path
|
||||||
|
|
||||||
|
Three async steps repeated forever:
|
||||||
|
|
||||||
|
```
|
||||||
|
async_read(socket, 4 bytes) → on_length()
|
||||||
|
async_read(socket, length bytes) → on_payload()
|
||||||
|
handle_frame(decoded Frame) → dispatch
|
||||||
|
```
|
||||||
|
|
||||||
|
In code, [`src/hsms/connection.cpp`](../src/hsms/connection.cpp):
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
void Connection::read_length() {
|
||||||
|
asio::async_read(socket_, asio::buffer(len_buf_, 4),
|
||||||
|
[self = shared_from_this()](std::error_code ec, std::size_t n) {
|
||||||
|
self->on_length(ec, n);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
void Connection::on_length(std::error_code ec, std::size_t n) {
|
||||||
|
if (ec) return close("read_length");
|
||||||
|
uint32_t len = decode_be32(len_buf_);
|
||||||
|
payload_.resize(len);
|
||||||
|
asio::async_read(socket_, asio::buffer(payload_),
|
||||||
|
[self = shared_from_this()](std::error_code ec, std::size_t n) {
|
||||||
|
self->on_payload(ec, n);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Each callback is on the socket's executor. No locks because
|
||||||
|
nothing else can be touching the read state — by construction.
|
||||||
|
|
||||||
|
### Write path
|
||||||
|
|
||||||
|
A send queue + one outstanding `async_write`:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
void Connection::send_frame(Frame frame) {
|
||||||
|
send_queue_.push_back(std::move(frame));
|
||||||
|
if (send_queue_.size() == 1) write_next();
|
||||||
|
}
|
||||||
|
|
||||||
|
void Connection::write_next() {
|
||||||
|
auto& frame = send_queue_.front();
|
||||||
|
send_buf_ = frame.encode();
|
||||||
|
asio::async_write(socket_, asio::buffer(send_buf_),
|
||||||
|
[self = shared_from_this()](std::error_code ec, std::size_t) {
|
||||||
|
self->send_queue_.pop_front();
|
||||||
|
if (ec) return self->close("write");
|
||||||
|
if (!self->send_queue_.empty()) self->write_next();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Same single-threaded discipline — `send_queue_` is only touched
|
||||||
|
on the executor. Callers from other threads must `asio::post`.
|
||||||
|
|
||||||
|
### Timers
|
||||||
|
|
||||||
|
Five `asio::steady_timer`s, one per HSMS T-timer:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// src/hsms/connection.cpp:30
|
||||||
|
Connection::Connection(...)
|
||||||
|
: socket_(std::move(sock)),
|
||||||
|
t3_timer_(socket_.get_executor()),
|
||||||
|
t6_timer_(socket_.get_executor()),
|
||||||
|
t7_timer_(socket_.get_executor()),
|
||||||
|
t8_timer_(socket_.get_executor()),
|
||||||
|
linktest_timer_(socket_.get_executor()),
|
||||||
|
timers_(timers) { }
|
||||||
|
```
|
||||||
|
|
||||||
|
All five share the socket's executor. When a timer fires, its
|
||||||
|
handler runs on the same executor as the read/write loop — so
|
||||||
|
again no locks for timer-vs-IO interaction.
|
||||||
|
|
||||||
|
T3 is special: there's one T3 timer per in-flight W=1 message
|
||||||
|
(correlated by `system_bytes`). These are short-lived
|
||||||
|
`steady_timer`s allocated when the request is sent and destroyed
|
||||||
|
when the reply arrives. See `src/hsms/connection.cpp:447`:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
auto t3 = std::make_shared<asio::steady_timer>(socket_.get_executor());
|
||||||
|
t3->expires_after(timers_.t3);
|
||||||
|
in_flight_.insert({system_bytes, RequestState{std::move(cb), t3, ...}});
|
||||||
|
t3->async_wait([self, system_bytes](std::error_code ec) {
|
||||||
|
if (ec) return; // cancelled (reply arrived)
|
||||||
|
self->on_t3_expire(system_bytes);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The asio executor / strand model
|
||||||
|
|
||||||
|
All `Connection` state lives on **one executor**. In simple cases
|
||||||
|
that's just `io_context.get_executor()` — a single-threaded loop.
|
||||||
|
In production, an EAP may run multiple `io_context::run()` threads
|
||||||
|
*per connection* by wrapping work in an `asio::strand`.
|
||||||
|
|
||||||
|
### What a strand is
|
||||||
|
|
||||||
|
A **strand** is an executor that guarantees mutual exclusion
|
||||||
|
between handlers it dispatches. Multiple threads can call
|
||||||
|
`io.run()`; the strand picks one of them at a time to run the
|
||||||
|
next handler in its queue.
|
||||||
|
|
||||||
|
For HSMS purposes, `socket_.get_executor()` already gives strand
|
||||||
|
semantics if the underlying `io_context` is single-threaded. For
|
||||||
|
multi-threaded `io_context`, the application wraps with
|
||||||
|
`asio::make_strand`:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
auto strand = asio::make_strand(io);
|
||||||
|
asio::ip::tcp::socket sock(strand);
|
||||||
|
auto conn = std::make_shared<hsms::Connection>(std::move(sock), ...);
|
||||||
|
```
|
||||||
|
|
||||||
|
Now `conn` has all its IO running on `strand`, while the
|
||||||
|
`io_context` can use 8 threads to handle 100 different connections.
|
||||||
|
|
||||||
|
### What this means for the caller
|
||||||
|
|
||||||
|
From any thread that isn't the strand's currently-running handler,
|
||||||
|
the caller MUST marshal onto the strand:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// From a sensor-thread callback:
|
||||||
|
asio::post(conn->executor(), [conn, msg = std::move(msg)] {
|
||||||
|
conn->send_data(std::move(msg));
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
Calling `conn->send_data` directly from another thread is **a
|
||||||
|
race**. Same for any store mutation. TSan catches this and the
|
||||||
|
test suite enforces it.
|
||||||
|
|
||||||
|
The contract is documented in detail in
|
||||||
|
[`docs/INTEGRATION.md`](INTEGRATION.md) §3.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## secsi::Protocol — the IO-free FSM
|
||||||
|
|
||||||
|
SECS-I's protocol layer is structured differently: the FSM has
|
||||||
|
**no IO at all**. It takes events (bytes received, application
|
||||||
|
asked to send, timer fired) and produces a list of `Action`s
|
||||||
|
(transmit these bytes, arm a timer, deliver a block to the
|
||||||
|
application).
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// include/secsgem/secsi/protocol.hpp
|
||||||
|
struct ActionTransmit { std::vector<uint8_t> bytes; };
|
||||||
|
struct ActionStartTimer { Timer which; };
|
||||||
|
struct ActionCancelTimer { Timer which; };
|
||||||
|
struct ActionDeliverBlock { Block block; };
|
||||||
|
struct ActionRaiseError { std::string reason; };
|
||||||
|
```
|
||||||
|
|
||||||
|
The wrapper (`secsi::TcpTransport`) drives the FSM:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
void TcpTransport::on_byte(uint8_t b) {
|
||||||
|
auto actions = protocol_.handle(EventByte{b});
|
||||||
|
for (const auto& a : actions) execute(a);
|
||||||
|
}
|
||||||
|
|
||||||
|
void TcpTransport::execute(const Action& a) {
|
||||||
|
std::visit([this](auto&& v) {
|
||||||
|
using T = std::decay_t<decltype(v)>;
|
||||||
|
if constexpr (std::is_same_v<T, ActionTransmit>) write_bytes(v.bytes);
|
||||||
|
else if constexpr (std::is_same_v<T, ActionStartTimer>) arm_timer(v.which);
|
||||||
|
else if constexpr (std::is_same_v<T, ActionCancelTimer>) cancel_timer(v.which);
|
||||||
|
else if constexpr (std::is_same_v<T, ActionDeliverBlock>) deliver(v.block);
|
||||||
|
else if constexpr (std::is_same_v<T, ActionRaiseError>) raise(v.reason);
|
||||||
|
}, a);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This design makes the **whole FSM** unit-testable. No sockets,
|
||||||
|
no timers, just `Event` in → `Action` out. Tests:
|
||||||
|
|
||||||
|
- [`tests/test_secsi.cpp`](../tests/test_secsi.cpp) — basic FSM
|
||||||
|
state walks.
|
||||||
|
- [`tests/test_secsi_timers.cpp`](../tests/test_secsi_timers.cpp)
|
||||||
|
— every timer scenario via synthetic `EventTimeout` injection.
|
||||||
|
- [`tests/test_secsi_tcp.cpp`](../tests/test_secsi_tcp.cpp) —
|
||||||
|
end-to-end via `TcpTransport`.
|
||||||
|
|
||||||
|
Same pattern repeats for E84 (chapter 18) and the GEM
|
||||||
|
communication-state FSM (chapter 13): IO-free FSM + asio adapter
|
||||||
|
+ separate test suites for each layer.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Why this is the right shape
|
||||||
|
|
||||||
|
### Pros of single-threaded + IO-free
|
||||||
|
|
||||||
|
- **No mutexes.** Anywhere.
|
||||||
|
- **Trivial reasoning.** When you read `Connection::send_frame`,
|
||||||
|
you can be sure nothing else is mutating the queue.
|
||||||
|
- **Fast.** No lock contention, no atomic round-trips.
|
||||||
|
- **Testable.** IO-free FSM lets you exercise every transition
|
||||||
|
without IO.
|
||||||
|
|
||||||
|
### Cons
|
||||||
|
|
||||||
|
- **Callers must know about the strand.** Multi-threaded
|
||||||
|
applications need `asio::post` boilerplate.
|
||||||
|
- **One slow handler blocks the rest.** A 100 ms handler delays
|
||||||
|
every other message until it returns. Fix: don't write slow
|
||||||
|
handlers; if you must, dispatch the slow work to another
|
||||||
|
executor and return immediately.
|
||||||
|
|
||||||
|
For a SECS/GEM equipment runtime — where the natural shape is "one
|
||||||
|
TCP socket, one event loop" — the pros far outweigh the cons.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Where to go next
|
||||||
|
|
||||||
|
You've now seen the bottom layer in detail: how bytes move,
|
||||||
|
how state machines drive transitions without IO, how the single-
|
||||||
|
threaded contract makes everything safe. Next chapter goes back
|
||||||
|
up one level: **the codec** — the encoder/decoder that turns
|
||||||
|
the Item type from chapter 10 into wire bytes, plus the SML
|
||||||
|
human-readable form.
|
||||||
|
|
||||||
|
Next: [→ 34 Codec and SML](34_codec_and_sml.md)
|
||||||
@@ -0,0 +1,303 @@
|
|||||||
|
# 34 — Codec and SML
|
||||||
|
|
||||||
|
← [33 Transport](33_transport.md) | [Back to index](00_index.md) | Next: [35 State machines and dispatch](35_state_machines_and_dispatch.md) →
|
||||||
|
|
||||||
|
We covered the SECS-II encoding rules in chapter 10. This
|
||||||
|
chapter is the **implementation walk** — the four files that make
|
||||||
|
up `secsgem::secs2`, how the encoder/decoder are structured, why
|
||||||
|
the variant-based `Item` works, and how the SML printer/parser
|
||||||
|
fits in.
|
||||||
|
|
||||||
|
Four files, 733 lines total. The codec is the most-tested layer
|
||||||
|
in the codebase.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The four files
|
||||||
|
|
||||||
|
```
|
||||||
|
include/secsgem/secs2/
|
||||||
|
├── item.hpp (170 lines) Item variant + Format enum + factories.
|
||||||
|
├── codec.hpp ( 30 lines) encode / decode declarations.
|
||||||
|
├── message.hpp ( 52 lines) Message wrapper (header fields + body Item).
|
||||||
|
└── sml.hpp ( 32 lines) to_sml / try_parse_sml declarations.
|
||||||
|
|
||||||
|
src/secs2/
|
||||||
|
├── codec.cpp (229 lines) encode_into / decode_at implementations.
|
||||||
|
└── sml.cpp (220 lines) SML printer + parser.
|
||||||
|
```
|
||||||
|
|
||||||
|
`item.hpp` and `message.hpp` are header-only. `codec.cpp` and
|
||||||
|
`sml.cpp` carry the heavy lifting.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The `Item` variant
|
||||||
|
|
||||||
|
Already covered in chapter 10; quick recap of the storage:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// include/secsgem/secs2/item.hpp:85
|
||||||
|
class Item {
|
||||||
|
public:
|
||||||
|
using List = std::vector<Item>;
|
||||||
|
using Storage = std::variant<
|
||||||
|
List, // List
|
||||||
|
std::string, // ASCII, JIS-8
|
||||||
|
std::vector<uint8_t>, // Binary, Boolean, U1
|
||||||
|
std::vector<int8_t>, // I1
|
||||||
|
std::vector<int16_t>, // I2
|
||||||
|
...
|
||||||
|
std::vector<float>, // F4
|
||||||
|
std::vector<double>>; // F8
|
||||||
|
|
||||||
|
private:
|
||||||
|
Format format_;
|
||||||
|
Storage data_;
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Eleven variant alternatives serving 16 SECS-II formats — some
|
||||||
|
formats share storage (Binary/Boolean/U1 all use
|
||||||
|
`std::vector<uint8_t>`, ASCII/JIS-8 share `std::string`, U2/C2
|
||||||
|
share `std::vector<uint16_t>`). Disambiguation is via `format_`.
|
||||||
|
|
||||||
|
### Factories
|
||||||
|
|
||||||
|
The intended way to build an `Item` is the named factories:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
Item::list({Item::ascii("Hi"), Item::u4(42)});
|
||||||
|
Item::ascii("Hello, world");
|
||||||
|
Item::u4(std::vector<uint32_t>{1, 2, 3});
|
||||||
|
Item::u4(42); // scalar convenience overload
|
||||||
|
Item::f4(1.0f);
|
||||||
|
```
|
||||||
|
|
||||||
|
Each takes ownership of the storage (or constructs from a scalar
|
||||||
|
overload). No exceptions; no validity checks; trusts the caller.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `encode_into` — the recursive encoder
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
void encode_into(const Item& item, std::vector<uint8_t>& out);
|
||||||
|
```
|
||||||
|
|
||||||
|
[`src/secs2/codec.cpp:71`](../src/secs2/codec.cpp). Two paths —
|
||||||
|
List and not-List:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
void encode_into(const Item& item, std::vector<uint8_t>& out) {
|
||||||
|
const Format fmt = item.format();
|
||||||
|
|
||||||
|
if (fmt == Format::List) {
|
||||||
|
const auto& children = item.as_list();
|
||||||
|
write_header(out, fmt, children.size());
|
||||||
|
for (const auto& child : children) encode_into(child, out);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scalar/array: write_header(byte count), then bytes.
|
||||||
|
switch (fmt) {
|
||||||
|
case Format::ASCII: {
|
||||||
|
const auto& s = item.as_ascii();
|
||||||
|
write_header(out, fmt, s.size());
|
||||||
|
out.insert(out.end(), s.begin(), s.end());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
case Format::U4: {
|
||||||
|
const auto& v = std::get<std::vector<uint32_t>>(item.storage());
|
||||||
|
write_header(out, fmt, v.size() * 4);
|
||||||
|
for (auto x : v) put_scalar_be(out, x);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// ... one case per format
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`write_header` picks the smallest length-byte-count and emits the
|
||||||
|
format byte + length bytes. `put_scalar_be` is the
|
||||||
|
templated big-endian writer using `std::bit_cast` for floats and
|
||||||
|
`std::make_unsigned_t` for integers (chapter 10).
|
||||||
|
|
||||||
|
`encode(item)` is a thin wrapper:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
std::vector<uint8_t> encode(const Item& item) {
|
||||||
|
std::vector<uint8_t> out;
|
||||||
|
encode_into(item, out);
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decode_at` — the recursive decoder
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos);
|
||||||
|
```
|
||||||
|
|
||||||
|
Mirror image:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos) {
|
||||||
|
// 1. Format byte + length bytes.
|
||||||
|
if (pos >= len) throw CodecError("truncated");
|
||||||
|
const uint8_t fb = data[pos++];
|
||||||
|
const Format fmt = static_cast<Format>(fb >> 2);
|
||||||
|
const int nlen = fb & 0x03;
|
||||||
|
if (pos + nlen > len) throw CodecError("truncated length bytes");
|
||||||
|
std::size_t length = 0;
|
||||||
|
for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++];
|
||||||
|
|
||||||
|
// 2. List recursion.
|
||||||
|
if (fmt == Format::List) {
|
||||||
|
Item::List children;
|
||||||
|
children.reserve(length);
|
||||||
|
for (std::size_t i = 0; i < length; ++i)
|
||||||
|
children.push_back(decode_at(data, len, pos));
|
||||||
|
return Item::list(std::move(children));
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3. Scalar/array: dispatch on element size + signedness/floatness.
|
||||||
|
if (pos + length > len) throw CodecError("truncated body");
|
||||||
|
const uint8_t* body = data + pos;
|
||||||
|
pos += length;
|
||||||
|
switch (fmt) {
|
||||||
|
case Format::ASCII: return Item::ascii(std::string((const char*)body, length));
|
||||||
|
case Format::U4: return Item::u4(read_array<uint32_t>(body, length));
|
||||||
|
// ... one case per format
|
||||||
|
}
|
||||||
|
throw CodecError("unknown format code");
|
||||||
|
}
|
||||||
|
|
||||||
|
Item decode(const std::vector<uint8_t>& bytes) {
|
||||||
|
std::size_t pos = 0;
|
||||||
|
Item it = decode_at(bytes.data(), bytes.size(), pos);
|
||||||
|
if (pos != bytes.size()) throw CodecError("trailing bytes");
|
||||||
|
return it;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The `_at` variant is useful when an outer protocol carries a SECS-II
|
||||||
|
item *embedded* in a larger frame — the caller passes the buffer
|
||||||
|
and a position, and gets back the item plus the new position.
|
||||||
|
|
||||||
|
Bounds checks throw `CodecError` at every step — a CodecError on
|
||||||
|
the receive side closes the connection (chapter 11's S9F7 path).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The Message wrapper
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// include/secsgem/secs2/message.hpp
|
||||||
|
class Message {
|
||||||
|
public:
|
||||||
|
uint8_t stream() const;
|
||||||
|
uint8_t function() const;
|
||||||
|
bool w_bit() const;
|
||||||
|
uint32_t system_bytes() const;
|
||||||
|
const Item& body() const;
|
||||||
|
std::vector<uint8_t> body_bytes() const; // encoded body
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
A `Message` is just a small struct: stream + function + W-bit +
|
||||||
|
system_bytes + body Item. No encoder lives here — encoding is
|
||||||
|
done by `secs2::encode(message.body())` when the transport layer
|
||||||
|
serializes a frame. The Message exists so the Router can dispatch
|
||||||
|
on `(stream, function)` without re-decoding bytes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SML — the human-readable form
|
||||||
|
|
||||||
|
`to_sml(item)` walks the Item recursively and emits SML:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// src/secs2/sml.cpp — sketch
|
||||||
|
std::string to_sml(const Item& item) {
|
||||||
|
switch (item.format()) {
|
||||||
|
case Format::List: {
|
||||||
|
std::string s = "<L[" + std::to_string(item.size()) + "]";
|
||||||
|
for (const auto& child : item.as_list()) {
|
||||||
|
s += ' ' + to_sml(child);
|
||||||
|
}
|
||||||
|
s += '>';
|
||||||
|
return s;
|
||||||
|
}
|
||||||
|
case Format::ASCII: return "A \"" + escape(item.as_ascii()) + "\"";
|
||||||
|
case Format::U4: {
|
||||||
|
const auto& v = std::get<std::vector<uint32_t>>(item.storage());
|
||||||
|
std::string s = "U4";
|
||||||
|
if (v.size() > 1) s += "[" + std::to_string(v.size()) + "]";
|
||||||
|
for (auto x : v) s += " " + std::to_string(x);
|
||||||
|
return s;
|
||||||
|
}
|
||||||
|
// ... per format
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`try_parse_sml(text)` is the inverse — a hand-written recursive-
|
||||||
|
descent parser that returns `std::optional<Item>`. Returns
|
||||||
|
`nullopt` on any parse error (no exceptions; this is what
|
||||||
|
libFuzzer feeds garbage into and expects it not to crash).
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
[`tests/test_sml.cpp`](../tests/test_sml.cpp) (10 cases — every
|
||||||
|
format round-trips through `to_sml` → `try_parse_sml` → identical
|
||||||
|
Item).
|
||||||
|
|
||||||
|
### Why SML doesn't round-trip *bytes*
|
||||||
|
|
||||||
|
A subtle point: `decode(encode(item))` round-trips exactly, but
|
||||||
|
`try_parse_sml(to_sml(item))` *also* round-trips the Item — except
|
||||||
|
encoding the round-tripped Item may produce **different bytes**
|
||||||
|
than the original. Why?
|
||||||
|
|
||||||
|
- The original might use a 2-byte length encoding; the
|
||||||
|
round-tripped Item is a fresh `Item` and the encoder will pick
|
||||||
|
the smallest length encoding (1 byte).
|
||||||
|
- SML doesn't preserve "which list-length encoding the encoder
|
||||||
|
chose."
|
||||||
|
|
||||||
|
If you need bit-exact round-trip of *bytes*, use `decode(encode)`.
|
||||||
|
For semantic round-trip of *values*, use SML.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Testing — every layer in isolation
|
||||||
|
|
||||||
|
| Layer | Test file | Cases | Focus |
|
||||||
|
|--------------|--------------------------------------|------:|--------------------------------------------------------|
|
||||||
|
| Item factories | tests/test_secs2.cpp | 14 | Construction, equality, format dispatch. |
|
||||||
|
| Codec | tests/test_e5_kat.cpp | 19 | Known-answer tests — bit-exact bytes per SEMI E5 §9. |
|
||||||
|
| Codec | tests/test_secs2.cpp | (overlap) | encode/decode round-trip + truncation rejection. |
|
||||||
|
| Identifier wildcards | tests/test_identifier_wildcards.cpp | 6 | U1/U2/U4/U8 leniency for ID fields. |
|
||||||
|
| SML | tests/test_sml.cpp | 10 | to_sml + try_parse_sml round-trip. |
|
||||||
|
| Catalog | tests/test_messages.cpp | 82 | Every named SxFy builder + parser round-trip. |
|
||||||
|
| Random/structural | tests/test_fuzz.cpp | 8 | Random bytes, truncation, oversize lengths, nested. |
|
||||||
|
| libFuzzer | apps/fuzz_secs2_decode.cpp | (CI) | 200 k+ random inputs per minute, ASan + UBSan clean. |
|
||||||
|
| libFuzzer | apps/fuzz_sml_parse.cpp | (CI) | 1.4 M+ random SML strings per minute, ASan + UBSan. |
|
||||||
|
|
||||||
|
The codec alone has **139 test cases / 196+ assertions for E5
|
||||||
|
KAT**. This is intentional: every other layer trusts the codec is
|
||||||
|
correct. If it isn't, nothing above works.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Where to go next
|
||||||
|
|
||||||
|
You've now seen the codec and SML implementation in detail. Next
|
||||||
|
chapter covers the **dispatch** layer that sits between the
|
||||||
|
transport (which delivers raw `Message`s) and the stores (which
|
||||||
|
hold state): `gem::Router`, the state-machine wiring, and the
|
||||||
|
generated builder/parser glue from the message catalog.
|
||||||
|
|
||||||
|
Next: [→ 35 State machines and dispatch](35_state_machines_and_dispatch.md)
|
||||||
@@ -0,0 +1,305 @@
|
|||||||
|
# 35 — State machines and dispatch
|
||||||
|
|
||||||
|
← [34 Codec and SML](34_codec_and_sml.md) | [Back to index](00_index.md) | Next: [36 Persistence, validation, metrics](36_persistence_validation_metrics.md) →
|
||||||
|
|
||||||
|
We have transport (chapter 33) delivering `secs2::Message`s
|
||||||
|
(chapter 34) into the application. We have stores (chapter 32)
|
||||||
|
holding the data. This chapter is the **glue**: how the Router
|
||||||
|
dispatches by `(stream, function)`, how state machines compose with
|
||||||
|
stores, and how a Router handler typically looks.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `gem::Router` — the dispatch table
|
||||||
|
|
||||||
|
[`include/secsgem/gem/router.hpp`](../include/secsgem/gem/router.hpp):
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
class Router {
|
||||||
|
public:
|
||||||
|
using Handler = std::function<std::optional<s2::Message>(const s2::Message&)>;
|
||||||
|
|
||||||
|
Router& on(uint8_t stream, uint8_t function, Handler h);
|
||||||
|
Router& fallback(Handler h);
|
||||||
|
|
||||||
|
std::optional<s2::Message> dispatch(const s2::Message& msg) const;
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
A `std::map<{stream, function}, Handler>`. Each handler:
|
||||||
|
|
||||||
|
- Takes a `const s2::Message&` (the inbound primary).
|
||||||
|
- Returns a `std::optional<s2::Message>` (the reply, or `nullopt`
|
||||||
|
for fire-and-forget primaries).
|
||||||
|
|
||||||
|
That's it. No middleware, no decorator chain, no pre/post hooks.
|
||||||
|
Just a typed dispatch.
|
||||||
|
|
||||||
|
### Registering handlers
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// apps/secs_server.cpp — sketch
|
||||||
|
auto router = std::make_shared<gem::Router>();
|
||||||
|
|
||||||
|
router->on(1, 1, [model](const auto& m) {
|
||||||
|
// S1F1 Are You There — reply with model name + softrev.
|
||||||
|
return messages::s1f2(model->device.mdln, model->device.softrev);
|
||||||
|
});
|
||||||
|
|
||||||
|
router->on(1, 3, [model](const auto& m) {
|
||||||
|
// S1F3 — read SVID values.
|
||||||
|
auto svid_list = messages::parse_s1f3(m.body());
|
||||||
|
return messages::s1f4(model->svids.values(svid_list.value_or({})));
|
||||||
|
});
|
||||||
|
|
||||||
|
router->on(2, 41, [model](const auto& m) {
|
||||||
|
// S2F41 Host Command.
|
||||||
|
auto cmd = messages::parse_s2f41(m.body());
|
||||||
|
auto ack = model->commands.dispatch(cmd->rcmd, cmd->params);
|
||||||
|
return messages::s2f42(ack);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ...one per S/F pair. apps/secs_server.cpp registers ~50.
|
||||||
|
```
|
||||||
|
|
||||||
|
The `examples/pvd_tool/main.cpp` §6 register 51 handlers in ~460
|
||||||
|
lines. Each handler is a few lines: parse the body, mutate or read
|
||||||
|
a store, build the reply.
|
||||||
|
|
||||||
|
### What happens for unhandled primaries
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// router.hpp dispatch()
|
||||||
|
auto it = handlers_.find({msg.stream, msg.function});
|
||||||
|
if (it != handlers_.end()) return it->second(msg);
|
||||||
|
if (fallback_) return fallback_(msg);
|
||||||
|
if (msg.reply_expected) return s2::Message(msg.stream, 0, false); // SxF0 Abort
|
||||||
|
return std::nullopt;
|
||||||
|
```
|
||||||
|
|
||||||
|
Three cases:
|
||||||
|
|
||||||
|
1. **Registered**: handler runs.
|
||||||
|
2. **Fallback installed**: fallback runs.
|
||||||
|
3. **Neither**: if the message expects a reply, send `SxF0` (Abort)
|
||||||
|
per E5 convention. Otherwise silently drop.
|
||||||
|
|
||||||
|
### S9 wiring
|
||||||
|
|
||||||
|
The transport layer (chapter 11) emits S9F3 / S9F5 for unhandled
|
||||||
|
primaries. Router exposes the introspection:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
bool has_handler(uint8_t stream, uint8_t function) const;
|
||||||
|
bool has_handler_for_stream(uint8_t stream) const;
|
||||||
|
```
|
||||||
|
|
||||||
|
And the wrapper that combines them with S9 emission:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
template <typename EmitFn, typename HeaderProvider>
|
||||||
|
std::optional<s2::Message> dispatch_with_s9(
|
||||||
|
EmitFn emit_s9, HeaderProvider header_provider,
|
||||||
|
const s2::Message& msg) const {
|
||||||
|
if (!has_handler(msg.stream, msg.function)) {
|
||||||
|
if (auto mhead = header_provider()) {
|
||||||
|
const uint8_t f = has_handler_for_stream(msg.stream) ? 5 : 3;
|
||||||
|
emit_s9(f, *mhead);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return dispatch(msg);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Used by `Connection::on_data_message` → `Router::dispatch_with_s9`,
|
||||||
|
which calls back into `Connection::emit_s9` for the actual S9
|
||||||
|
emission. Tested by
|
||||||
|
[`tests/test_s9_fallback.cpp`](../tests/test_s9_fallback.cpp) (2
|
||||||
|
cases — unknown stream → S9F3, unknown function in known stream
|
||||||
|
→ S9F5).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How a typical handler looks end-to-end
|
||||||
|
|
||||||
|
`S2F41` Host Command is the most-touched message in production —
|
||||||
|
worth tracing in full:
|
||||||
|
|
||||||
|
### 1. The codegen'd parser/builder
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// build/generated/secsgem/gem/messages.hpp (auto-generated)
|
||||||
|
struct RemoteCommand {
|
||||||
|
std::string rcmd;
|
||||||
|
std::vector<std::pair<std::string, secs2::Item>> params;
|
||||||
|
};
|
||||||
|
|
||||||
|
inline std::optional<RemoteCommand> parse_s2f41(const secs2::Item& body) {
|
||||||
|
// ...auto-generated body walker...
|
||||||
|
}
|
||||||
|
|
||||||
|
inline secs2::Message s2f42(uint8_t hcack, /* per-param acks */) {
|
||||||
|
// ...auto-generated builder...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. The Router registration
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// apps/secs_server.cpp
|
||||||
|
router->on(2, 41, [model](const secs2::Message& m) {
|
||||||
|
auto cmd = messages::parse_s2f41(m.body());
|
||||||
|
if (!cmd) {
|
||||||
|
// Body didn't parse — reply S2F42 with HCACK = 1 (invalid).
|
||||||
|
return messages::s2f42(1, {});
|
||||||
|
}
|
||||||
|
auto outcome = model->commands.dispatch(cmd->rcmd, cmd->params);
|
||||||
|
return messages::s2f42(static_cast<uint8_t>(outcome.ack), outcome.cpacks);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. The store dispatch
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// include/secsgem/gem/store/host_commands.hpp
|
||||||
|
class HostCommandRegistry {
|
||||||
|
public:
|
||||||
|
CommandOutcome dispatch(const std::string& rcmd, const ParamList& params) {
|
||||||
|
auto it = commands_.find(rcmd);
|
||||||
|
if (it == commands_.end()) return {HostCmdAck::InvalidCommand, ...};
|
||||||
|
const auto& cmd = it->second;
|
||||||
|
// Apply configured side effects: emit_ceid, set_alarm, …
|
||||||
|
for (auto ceid : cmd.emit_ceids) on_emit_ceid_(ceid);
|
||||||
|
for (auto alid : cmd.set_alarms) alarm_registry_->set(alid);
|
||||||
|
return {cmd.default_ack, ...};
|
||||||
|
}
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. The side-effect dispatcher
|
||||||
|
|
||||||
|
Steps in `dispatch` like `on_emit_ceid_(ceid)` call back into
|
||||||
|
the EAP:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// Set up at startup:
|
||||||
|
model->commands.set_emit_ceid_handler([conn, model](uint32_t ceid) {
|
||||||
|
if (!model->is_event_enabled(ceid)) return;
|
||||||
|
auto reports = model->compose_reports_for(ceid);
|
||||||
|
auto msg = build_s6f11(ceid, reports);
|
||||||
|
conn->send_data(std::move(msg));
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. The wire
|
||||||
|
|
||||||
|
`conn->send_data(s6f11)` walks through `secs2::encode` →
|
||||||
|
`hsms::Frame::encode` → `async_write` to the socket. Host sees
|
||||||
|
the unsolicited S6F11.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The state-machine pattern
|
||||||
|
|
||||||
|
Every state machine in the codebase follows the same shape. Pick
|
||||||
|
`ControlStateMachine`:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// include/secsgem/gem/control_state.hpp
|
||||||
|
class ControlStateMachine {
|
||||||
|
public:
|
||||||
|
ControlStateMachine(ControlTransitionTable table);
|
||||||
|
|
||||||
|
ControlState state() const;
|
||||||
|
|
||||||
|
// Apply an event; returns the transition row (if any).
|
||||||
|
const ControlTransition* on_event(ControlEvent e);
|
||||||
|
|
||||||
|
// Observer for transitions.
|
||||||
|
using StateChangeHandler =
|
||||||
|
std::function<void(ControlState from, ControlState to, ControlEvent)>;
|
||||||
|
void set_state_change_handler(StateChangeHandler h);
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Three properties:
|
||||||
|
|
||||||
|
1. **Pure data table** (`ControlTransitionTable`) decides what
|
||||||
|
transitions exist.
|
||||||
|
2. **Pure FSM** (`ControlStateMachine`) applies events against the
|
||||||
|
table, updates state, emits the change.
|
||||||
|
3. **Observer pattern** — the EAP registers a change handler that
|
||||||
|
does the wire-level work (fire CEID, emit S6F11, log to
|
||||||
|
metrics).
|
||||||
|
|
||||||
|
This pattern repeats for:
|
||||||
|
|
||||||
|
- `ProcessJobStateMachine` (E40)
|
||||||
|
- `ControlJobStateMachine` (E94)
|
||||||
|
- `EptStateMachine` (E116)
|
||||||
|
- `ExceptionStateMachine` (E5 §13)
|
||||||
|
- `CarrierStateMachine` (E87 — actually composes 3 sub-FSMs)
|
||||||
|
- `LoadPortStateMachine` (E87 — same)
|
||||||
|
- `SubstrateStateMachine` (E90 — same)
|
||||||
|
- `ModuleStateMachine` (E157)
|
||||||
|
- `E84StateMachine` (E84)
|
||||||
|
- `CommunicationStateMachine` (E30 §6.5)
|
||||||
|
|
||||||
|
Eleven FSMs. All follow the same shape. All testable in
|
||||||
|
isolation without IO.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How FSM transitions cascade into CEIDs
|
||||||
|
|
||||||
|
A common need: "when PJ-1 transitions to Processing, fire
|
||||||
|
CEID=ProcessStarted, which fires an S6F11 with linked reports."
|
||||||
|
|
||||||
|
The wiring is set up at startup in the EAP:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// apps/secs_server.cpp — sketch
|
||||||
|
model->process_jobs.set_state_change_handler(
|
||||||
|
[conn, model](const std::string& pjid,
|
||||||
|
ProcessJobState from, ProcessJobState to,
|
||||||
|
ProcessJobEvent ev) {
|
||||||
|
// Configured per-state CEID from data/equipment.yaml.
|
||||||
|
auto ceid = ceid_for_pj_state(to);
|
||||||
|
if (!ceid || !model->is_event_enabled(*ceid)) return;
|
||||||
|
auto reports = model->compose_reports_for(*ceid);
|
||||||
|
auto msg = build_s6f11(*ceid, reports);
|
||||||
|
deliver_or_spool(*conn, *model, std::move(msg));
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
`deliver_or_spool` is the spool-aware send: if the connection isn't
|
||||||
|
SELECTED (or the spool is in transmit-disabled mode), the message
|
||||||
|
queues into `SpoolStore`; otherwise it goes straight to the wire.
|
||||||
|
|
||||||
|
So the chain for "PJ-1 starts processing":
|
||||||
|
|
||||||
|
```
|
||||||
|
ProcessJobStore.apply(pjid, Start)
|
||||||
|
→ ProcessJobStateMachine.on_event(Start)
|
||||||
|
→ State: WaitingForStart → Processing
|
||||||
|
→ on_change handler fires
|
||||||
|
→ looks up CEID ProcessStarted
|
||||||
|
→ composes reports
|
||||||
|
→ builds S6F11 message
|
||||||
|
→ deliver_or_spool → connection or SpoolStore
|
||||||
|
→ on the wire (eventually)
|
||||||
|
```
|
||||||
|
|
||||||
|
Every step is independently testable. Tests at the wire level:
|
||||||
|
[`tests/test_wire_ceid_emission.cpp`](../tests/test_wire_ceid_emission.cpp)
|
||||||
|
(6 cases — every cascade from store mutation to socket bytes).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Where to go next
|
||||||
|
|
||||||
|
You've now seen everything that makes the runtime *work*. One
|
||||||
|
chapter left in Part 3: the **operational** concerns — persistence,
|
||||||
|
config validation, and metrics.
|
||||||
|
|
||||||
|
Next: [→ 36 Persistence, validation, metrics](36_persistence_validation_metrics.md)
|
||||||
@@ -0,0 +1,293 @@
|
|||||||
|
# 36 — Persistence, validation, metrics
|
||||||
|
|
||||||
|
← [35 State machines and dispatch](35_state_machines_and_dispatch.md) | [Back to index](00_index.md) | Next: [40 Building, running, the demo](40_building_running_demo.md) →
|
||||||
|
|
||||||
|
Three operational concerns wrap up Part 3:
|
||||||
|
|
||||||
|
- **Persistence** — file-backed journals for the seven stores that
|
||||||
|
survive equipment restarts.
|
||||||
|
- **Validation** — the multi-error YAML validator behind
|
||||||
|
`--validate-config`.
|
||||||
|
- **Metrics** — the Prometheus exporter.
|
||||||
|
|
||||||
|
Each is a small slice of the codebase but load-bearing for
|
||||||
|
production deployments.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Persistence
|
||||||
|
|
||||||
|
### Which stores persist
|
||||||
|
|
||||||
|
Seven of the 21 stores have file-backed journals:
|
||||||
|
|
||||||
|
| Store | Survives equipment restart |
|
||||||
|
|--------------------|------------------------------------------------------------|
|
||||||
|
| `SpoolStore` | Queued messages waiting for host comm to come back. |
|
||||||
|
| `ProcessJobStore` | All in-progress PJs and their state machines. |
|
||||||
|
| `ControlJobStore` | All in-progress CJs. |
|
||||||
|
| `ExceptionStore` | Posted exceptions and their recovery state. |
|
||||||
|
| `CarrierStore` | Docked carriers + slot maps + access state. |
|
||||||
|
| `LoadPortStore` | Per-port association + reservation. |
|
||||||
|
| `SubstrateStore` | Per-substrate location + STS / SPS / ID status. |
|
||||||
|
|
||||||
|
The remaining 14 stores (SVIDs, ECIDs, CEIDs, alarm registry, …)
|
||||||
|
don't persist — their state is reconstructed from the YAML or
|
||||||
|
from real-time signals on restart. An ECID that the host had
|
||||||
|
changed *would* be lost on restart unless the EAP writes it back
|
||||||
|
to the YAML (E40-style `S2F15` is rare in production for
|
||||||
|
exactly this reason).
|
||||||
|
|
||||||
|
### The per-record file pattern
|
||||||
|
|
||||||
|
Every persistent store uses the same shape:
|
||||||
|
|
||||||
|
```
|
||||||
|
/var/lib/secsgem/<store>/
|
||||||
|
├── PJ-001 # one file per record
|
||||||
|
├── PJ-002
|
||||||
|
├── PJ-003
|
||||||
|
└── ...
|
||||||
|
```
|
||||||
|
|
||||||
|
One file per record, named by ID. When the store is mutated, the
|
||||||
|
file is rewritten atomically (write to `.tmp` + `rename`). When
|
||||||
|
the record is removed, the file is `unlink`'d.
|
||||||
|
|
||||||
|
**This is partial-write safe.** If the equipment power-cycles
|
||||||
|
mid-write of one record, the others are untouched. At startup,
|
||||||
|
the store iterates the directory, reads each file, and replays
|
||||||
|
into in-memory state. A file that fails to parse (corrupted or
|
||||||
|
unfinished) is dropped with a log line.
|
||||||
|
|
||||||
|
### How a store enables persistence
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// apps/secs_server.cpp — startup
|
||||||
|
auto model = std::make_shared<gem::EquipmentDataModel>();
|
||||||
|
|
||||||
|
if (!spool_dir.empty()) {
|
||||||
|
model->spool.enable_persistence(spool_dir);
|
||||||
|
}
|
||||||
|
if (!pj_dir.empty()) {
|
||||||
|
model->process_jobs.enable_persistence(pj_dir);
|
||||||
|
}
|
||||||
|
// ... etc per store
|
||||||
|
```
|
||||||
|
|
||||||
|
`enable_persistence(dir)`:
|
||||||
|
|
||||||
|
1. Creates `dir` if needed.
|
||||||
|
2. Iterates files in `dir`.
|
||||||
|
3. For each file, reads + parses + adds the record to the store.
|
||||||
|
4. Sets up the on-disk journal for subsequent mutations.
|
||||||
|
|
||||||
|
The persistence is **opt-in per store**, configured via CLI flag in
|
||||||
|
`apps/secs_server.cpp`. Some deployments want spool persistence
|
||||||
|
but not job persistence (e.g., test rigs); the per-store toggle
|
||||||
|
makes that easy.
|
||||||
|
|
||||||
|
### File format and versioning
|
||||||
|
|
||||||
|
Each record file is a small binary blob:
|
||||||
|
|
||||||
|
```
|
||||||
|
magic: 4 bytes "SGv1" (store-specific magic; v1 = version 1)
|
||||||
|
version: 4 bytes (uint32_t, big-endian) — schema version
|
||||||
|
length: 4 bytes (uint32_t, big-endian) — payload length
|
||||||
|
payload: N bytes — store-specific record encoding
|
||||||
|
checksum: 4 bytes (CRC-32C over header + payload)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Schema versioning** is built in. Every store has a `kVersion`
|
||||||
|
constant. When the store reads a file:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
if (file_version > kVersion)
|
||||||
|
drop the file (newer than us; can't read)
|
||||||
|
if (file_version < kVersion)
|
||||||
|
apply the upgrade path (v1 → v2 → v3 reader chain)
|
||||||
|
if (file_version == kVersion)
|
||||||
|
read directly
|
||||||
|
```
|
||||||
|
|
||||||
|
Multi-version reads let a new equipment release process old
|
||||||
|
on-disk records without manual migration. Tested by
|
||||||
|
[`tests/test_persistence_upgrade.cpp`](../tests/test_persistence_upgrade.cpp)
|
||||||
|
(7 cases — every store with persistence, write v1, restart at
|
||||||
|
v2, verify replay).
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
|
||||||
|
| Store | Test file | Cases |
|
||||||
|
|---------------------|----------------------------------------------------|------:|
|
||||||
|
| Spool | bundled into `tests/test_data_model.cpp` | — |
|
||||||
|
| Process Jobs | `tests/test_job_persistence.cpp` (PJ + CJ together)| 7 |
|
||||||
|
| Control Jobs | same | — |
|
||||||
|
| Exception | `tests/test_exception_persistence.cpp` | 5 |
|
||||||
|
| Carrier | `tests/test_carrier_persistence.cpp` | 6 |
|
||||||
|
| Substrate | `tests/test_substrate_persistence.cpp` | 7 |
|
||||||
|
| Upgrade path | `tests/test_persistence_upgrade.cpp` | 7 |
|
||||||
|
|
||||||
|
Each persistence test covers: write a record, restart, verify
|
||||||
|
replayed; partial-write recovery (truncated file dropped); remove
|
||||||
|
deletes the file; corrupted file is dropped without throwing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
### Why a separate validator
|
||||||
|
|
||||||
|
YAML loaders throw on first error. That's the right behaviour
|
||||||
|
at process startup — fail fast — but it's frustrating for an
|
||||||
|
operator with a fresh equipment.yaml that has three typos.
|
||||||
|
|
||||||
|
`--validate-config` is a separate CLI flag that:
|
||||||
|
|
||||||
|
1. Doesn't bind the port.
|
||||||
|
2. Tries to load every YAML.
|
||||||
|
3. Accumulates *every* issue (across files).
|
||||||
|
4. Prints them all.
|
||||||
|
5. Exits 0 or 1.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
secs_server --validate-config \
|
||||||
|
--config data/equipment.yaml \
|
||||||
|
--state-table data/control_state.yaml \
|
||||||
|
--pj-state-table data/process_job_state.yaml \
|
||||||
|
--cj-state-table data/control_job_state.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Typical output:
|
||||||
|
|
||||||
|
```
|
||||||
|
data/equipment.yaml:42: SVID 5 references undefined enum 'ChamberStateEnum'
|
||||||
|
data/equipment.yaml:78: alarm 3 has ALCD bit-7 cleared but alarm is declared 'active'
|
||||||
|
data/control_state.yaml:11: transition from OnlineRemote on host_request_remote has no `to` or `ack` field
|
||||||
|
data/equipment.yaml:104: host_command VENT references unknown CEID 999
|
||||||
|
|
||||||
|
4 error(s), 0 warning(s) across 4 files
|
||||||
|
```
|
||||||
|
|
||||||
|
Then exit 1.
|
||||||
|
|
||||||
|
### How it's implemented
|
||||||
|
|
||||||
|
[`include/secsgem/config/validate.hpp`](../include/secsgem/config/validate.hpp):
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
class ConfigValidator {
|
||||||
|
public:
|
||||||
|
void validate_equipment(const std::string& path);
|
||||||
|
void validate_control_state(const std::string& path);
|
||||||
|
void validate_process_job_state(const std::string& path);
|
||||||
|
void validate_control_job_state(const std::string& path);
|
||||||
|
|
||||||
|
std::size_t error_count() const;
|
||||||
|
std::size_t warning_count() const;
|
||||||
|
bool has_errors() const;
|
||||||
|
|
||||||
|
const std::vector<Issue>& issues() const;
|
||||||
|
void format_issues_to(std::ostream&, FormatOptions = {}) const;
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Each `validate_*` method:
|
||||||
|
|
||||||
|
1. Loads the YAML (catching parse errors as one issue).
|
||||||
|
2. Walks every record, applying structural + referential checks.
|
||||||
|
3. Adds each problem as an `Issue{path, line, severity, message}`.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
[`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp)
|
||||||
|
(8 cases — every category of issue: missing required field,
|
||||||
|
typed mismatch, dangling reference, duplicate ID, …).
|
||||||
|
|
||||||
|
### Reference checks across files
|
||||||
|
|
||||||
|
Cross-file references are validated last (after all files are
|
||||||
|
parsed). Examples:
|
||||||
|
|
||||||
|
- `host_commands[].emit_ceid` must reference a CEID defined in
|
||||||
|
`equipment.yaml::ceids`.
|
||||||
|
- `events.default_reports[].vids` must reference SVIDs or DVIDs
|
||||||
|
defined elsewhere.
|
||||||
|
- `control_state.yaml::transitions` `from`/`to` must reference
|
||||||
|
states declared by the schema (the 5 standard control states).
|
||||||
|
|
||||||
|
This catches "I deleted the CEID but forgot to update the
|
||||||
|
host_command" before runtime.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Metrics
|
||||||
|
|
||||||
|
### What gets exported
|
||||||
|
|
||||||
|
The codebase ships a Prometheus exporter
|
||||||
|
([`include/secsgem/metrics/prometheus.hpp`](../include/secsgem/metrics/prometheus.hpp))
|
||||||
|
with two parts:
|
||||||
|
|
||||||
|
- **Registry** — accumulates `Counter` and `Gauge` series with
|
||||||
|
labels.
|
||||||
|
- **Server** — exposes them on a configurable HTTP port at
|
||||||
|
`/metrics`.
|
||||||
|
|
||||||
|
Typical wiring:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
auto registry = std::make_shared<metrics::Registry>();
|
||||||
|
registry->register_metric("secsgem_ceid_emits_total", metrics::MetricType::Counter);
|
||||||
|
registry->register_metric("secsgem_spool_depth", metrics::MetricType::Gauge);
|
||||||
|
registry->register_metric("secsgem_pj_state", metrics::MetricType::Gauge);
|
||||||
|
|
||||||
|
// ...later, in the CEID-emit handler:
|
||||||
|
registry->counter("secsgem_ceid_emits_total", {{"ceid", std::to_string(ceid)}}).inc();
|
||||||
|
|
||||||
|
// ...periodically:
|
||||||
|
registry->gauge("secsgem_spool_depth").set(model->spool.size());
|
||||||
|
|
||||||
|
// Start the HTTP server:
|
||||||
|
auto exporter = std::make_shared<metrics::PrometheusServer>(io, /*port=*/9090, registry);
|
||||||
|
```
|
||||||
|
|
||||||
|
The exporter is wire-compatible with Prometheus scrape (text
|
||||||
|
format). Tested by
|
||||||
|
[`tests/test_metrics_prometheus.cpp`](../tests/test_metrics_prometheus.cpp)
|
||||||
|
(3 cases — counter increment, gauge set, HTTP scrape format).
|
||||||
|
|
||||||
|
### What to expose
|
||||||
|
|
||||||
|
Common patterns from
|
||||||
|
[`examples/pvd_tool/main.cpp`](../examples/pvd_tool/main.cpp) §7:
|
||||||
|
|
||||||
|
- Per-CEID counters (`secsgem_ceid_emits_total{ceid="300"}`).
|
||||||
|
- Per-alarm counters (`secsgem_alarm_set_total{alid="42"}`).
|
||||||
|
- Spool depth gauge (alarm in operations if it climbs).
|
||||||
|
- Per-state EPT durations (sample of E116 buckets).
|
||||||
|
- T3 timeout counter (alarm in operations if non-zero).
|
||||||
|
|
||||||
|
The exporter doesn't dictate which metrics to expose — the EAP
|
||||||
|
decides. See
|
||||||
|
[`docs/INTEGRATION.md`](INTEGRATION.md) §6.4 for the production
|
||||||
|
patterns.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## End of Part 3
|
||||||
|
|
||||||
|
You now know every layer of the runtime:
|
||||||
|
|
||||||
|
- The repository layout (chapter 30).
|
||||||
|
- The spec-as-data philosophy + codegen (chapter 31).
|
||||||
|
- The stores + data model (chapter 32).
|
||||||
|
- The transport implementation (chapter 33).
|
||||||
|
- The codec + SML (chapter 34).
|
||||||
|
- Router + state machines + dispatch (chapter 35).
|
||||||
|
- Persistence + validation + metrics (this chapter).
|
||||||
|
|
||||||
|
Part 4 turns to operations — how a customer actually builds, runs,
|
||||||
|
deploys, and integrates this codebase into a real fab tool.
|
||||||
|
|
||||||
|
Next: [→ 40 Building, running, the demo](40_building_running_demo.md)
|
||||||
Reference in New Issue
Block a user