docs: chapters 30–36 — the codebase (Part 3 complete)
Seven chapters walking the implementation top-to-bottom. 30 — Repository tour. Top-level layout, directory by directory. The eight built binaries. The dependency graph from TCP socket up through EquipmentDataModel. CMake's role. Test layout. 31 — Spec-as-data and codegen. Why the design choice fits SECS/ GEM specifically. The five YAML files: messages catalog, control/PJ/CJ transition tables, equipment dictionary. How tools/gen_messages.py turns messages.yaml into typed C++ at build time. The --validate-config multi-error validator. How to add a new SVID / CEID / host command / state / message without C++. 32 — Stores and the data model. What a store IS (records + API + change handler + optional persistence). Every store in the codebase mapped to the SEMI standard it serves (table of 21). EquipmentDataModel as plain composition + cross-store convenience methods (vid_value, compose_reports_for). The no-locks single- threaded contract. How to add a new store. 33 — Transport. hsms::Connection read path (length+payload async chain), write path (queue + one outstanding write), timer model (5 steady_timers + per-request T3). The asio executor / strand model and why it's the right shape. secsi::Protocol as the IO- free FSM with Action / Event variants; secsi::TcpTransport as the asio adapter. Pattern repeats for E84 + GEM comm-state. 34 — Codec and SML. The four files (170 + 30 + 52 + 32 lines of header, 229 + 220 lines of impl). Item variant storage layout (11 alternatives, 16 formats, shared storage where E5 permits). encode_into recursion; decode_at with bounds checks throwing CodecError. Message wrapper. SML printer + try_parse_sml + why SML round-trips Items but not necessarily bytes. 35 — State machines and dispatch. gem::Router as a typed (stream, function) dispatch table. How an S2F41 round-trip walks through parser → store dispatch → side-effect → CEID emission → S6F11 build → spool-aware deliver. The 11 FSMs all sharing the same three-property shape (pure data table + pure FSM + observer pattern). CEID cascading from FSM transitions to wire bytes. 36 — Persistence, validation, metrics. Which 7 stores have file journals + why the others don't. Per-record file pattern (atomic rename, partial-write safe). Schema versioning + multi-version read. Multi-error YAML validator (--validate-config) + cross-file reference checks. Prometheus registry + HTTP exporter + worked metric patterns from the PVD example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,231 @@
|
||||
# 30 — Repository tour
|
||||
|
||||
← [19 E42 + E148 + S9 — Misc](19_e42_e148_s9_misc.md) | [Back to index](00_index.md) | Next: [31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md) →
|
||||
|
||||
You've seen what every SEMI standard *does*. Now we shift to how
|
||||
this **codebase** is laid out. This chapter answers: when you
|
||||
`git clone` this repo, what are you looking at?
|
||||
|
||||
The repo is small — about 15 k lines of C++ + tests + tooling. It
|
||||
fits in your head with a little patience. By the end of this
|
||||
chapter you'll know:
|
||||
|
||||
- What each top-level directory contains.
|
||||
- Which binaries get built.
|
||||
- The dependency graph between modules.
|
||||
- How the build system finds and links them.
|
||||
|
||||
---
|
||||
|
||||
## Top-level layout
|
||||
|
||||
```
|
||||
secs-gem/
|
||||
├── README.md One-page project summary.
|
||||
├── LICENSE Proprietary terms.
|
||||
├── CMakeLists.txt Build config (CMake 3.16+, single file).
|
||||
├── Dockerfile Ubuntu 24.04 + g++-13 + libasio + yaml-cpp.
|
||||
├── docker-compose.yml Multi-container demo wiring.
|
||||
├── .gitea/workflows/ci.yml CI pipeline.
|
||||
│
|
||||
├── include/secsgem/ Public headers. All API here.
|
||||
│ ├── secs2/ E5 codec + SML.
|
||||
│ ├── hsms/ E37 transport (TCP + framing).
|
||||
│ ├── secsi/ E4 transport (FSM + TCP tunnel).
|
||||
│ ├── config/ YAML loader + multi-error validator.
|
||||
│ ├── metrics/ Prometheus exporter.
|
||||
│ ├── endpoint.hpp asio::ip::tcp::endpoint factory.
|
||||
│ └── gem/ E30 + every GEM 300 standard.
|
||||
│ ├── store/ Per-domain bundles (SVIDs, alarms, …).
|
||||
│ └── *.hpp State machines + composers.
|
||||
│
|
||||
├── src/ Implementations. Mirrors include/.
|
||||
│ ├── secs2/{codec,sml}.cpp
|
||||
│ ├── hsms/{header,connection}.cpp
|
||||
│ ├── secsi/{header,block,protocol,tcp_transport}.cpp
|
||||
│ ├── config/...
|
||||
│ ├── gem/...
|
||||
│ └── endpoint.cpp
|
||||
│
|
||||
├── apps/ Standalone binaries.
|
||||
│ ├── secs_server.cpp Passive equipment (demo + integration target).
|
||||
│ ├── secs_client.cpp Active host driving the demo flow.
|
||||
│ ├── secs_conformance.cpp 47-check wire-level conformance harness.
|
||||
│ ├── secs_interop_probe.cpp Probe against secsgem-py passive equip.
|
||||
│ ├── secs_bench.cpp Throughput / latency / memory bench.
|
||||
│ ├── fuzz_secs2_decode.cpp libFuzzer harness for secs2::decode.
|
||||
│ └── fuzz_sml_parse.cpp libFuzzer harness for try_parse_sml.
|
||||
│
|
||||
├── tests/ doctest unit + integration tests.
|
||||
│ └── test_*.cpp 50 files, 445 cases, 2753 assertions.
|
||||
│
|
||||
├── data/ YAML configs (the spec-as-data).
|
||||
│ ├── messages.yaml SECS-II message catalog (164 msgs).
|
||||
│ ├── control_state.yaml E30 §6.2 transition table.
|
||||
│ ├── process_job_state.yaml E40 transition table.
|
||||
│ ├── control_job_state.yaml E94 transition table.
|
||||
│ └── equipment.yaml Demo SVIDs/ECIDs/CEIDs/alarms/recipes.
|
||||
│
|
||||
├── tools/ Build-time scripts.
|
||||
│ └── generate_messages.py Codegen: messages.yaml → messages.hpp.
|
||||
│
|
||||
├── interop/ External-validator harnesses.
|
||||
│ ├── README.md Harness-by-harness detail.
|
||||
│ ├── host_vs_cpp_server.py secsgem-py active host driving us.
|
||||
│ ├── passive_equipment.py secsgem-py passive equipment for us to drive.
|
||||
│ ├── raw_gem300_harness.py Raw S3/S14/S16/S12 round-trip.
|
||||
│ ├── tshark_validate.sh pcap + tshark HSMS dissector check.
|
||||
│ ├── secs4j_validate.sh secs4java8 (Java) cross-validation.
|
||||
│ └── secs4j/ Dockerfile + harness for secs4java8.
|
||||
│
|
||||
├── examples/
|
||||
│ └── pvd_tool/ Worked vendor example: fictional PVD tool.
|
||||
│ ├── README.md What the example shows.
|
||||
│ ├── equipment.yaml Realistic SVIDs/ECIDs/CEIDs/alarms/recipes.
|
||||
│ └── main.cpp Sensor sim, recipe runner, alarm monitor.
|
||||
│
|
||||
└── docs/ This guide + reference docs.
|
||||
├── 00_index.md The series TOC.
|
||||
├── 01–51_*.md Tutorial chapters.
|
||||
├── ARCHITECTURE.md One-page architecture overview.
|
||||
├── COMPLIANCE.md Per-capability audit.
|
||||
├── INTEGRATION.md Vendor-side production deploy.
|
||||
├── PROOFS.md 8 commands proving feature-completeness.
|
||||
├── VERIFICATION.md External-validator test plan.
|
||||
├── BENCHMARKS.md Performance envelope.
|
||||
├── MES_INTEROP.md Commercial-MES day-1 punch list.
|
||||
├── SECURITY.md nftables / stunnel / minisign configs.
|
||||
├── GLOSSARY.md SEMI vocabulary cheat sheet.
|
||||
└── FAQ.md Canonical answers.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The dependency graph
|
||||
|
||||
```
|
||||
data/*.yaml
|
||||
│
|
||||
┌─────────────┼──────────────────┐
|
||||
│ (codegen) │ (runtime load) │
|
||||
▼ ▼ ▼
|
||||
generated/messages.hpp config::loader
|
||||
│ │
|
||||
└──────────► gem::EquipmentDataModel
|
||||
│
|
||||
│ used by
|
||||
▼
|
||||
gem::Router
|
||||
│
|
||||
│ wraps
|
||||
▼
|
||||
secs2::Message ◄─── codec / SML
|
||||
│
|
||||
│ over
|
||||
▼
|
||||
hsms::Connection / secsi::TcpTransport
|
||||
│
|
||||
▼
|
||||
TCP socket
|
||||
```
|
||||
|
||||
Read it bottom-up: a TCP socket carries bytes; `hsms::Connection`
|
||||
frames them into `secs2::Message`s; `gem::Router` dispatches by
|
||||
`(stream, function)` to handlers; handlers read/write
|
||||
`EquipmentDataModel`; the model composes per-domain stores; the
|
||||
stores were built from the YAML at startup.
|
||||
|
||||
No layer ever calls *up* the graph. `secs2::Item` has no idea
|
||||
HSMS exists. `hsms::Connection` doesn't know about CEIDs.
|
||||
`gem::Router` doesn't know whether the bytes came over HSMS or
|
||||
SECS-I. Strict layering is what keeps the codebase small.
|
||||
|
||||
---
|
||||
|
||||
## The binaries
|
||||
|
||||
Built by [`CMakeLists.txt`](../CMakeLists.txt) (one file, ~250
|
||||
lines). Each binary lives in `build/` after `cmake --build`.
|
||||
|
||||
| Binary | Source | What it does |
|
||||
|----------------------|-----------------------------------------------------------------|-------------------------------------------------------|
|
||||
| `secs_server` | [`apps/secs_server.cpp`](../apps/secs_server.cpp) | Passive equipment. Listens on TCP, dispatches via Router. |
|
||||
| `secs_client` | [`apps/secs_client.cpp`](../apps/secs_client.cpp) | Active host. Drives ~24 transactions in the demo. |
|
||||
| `secs_conformance` | [`apps/secs_conformance.cpp`](../apps/secs_conformance.cpp) | 47 wire-level conformance checks against a live server. |
|
||||
| `secs_interop_probe` | [`apps/secs_interop_probe.cpp`](../apps/secs_interop_probe.cpp) | Active host probing a secsgem-py passive equipment. |
|
||||
| `secs_bench` | [`apps/secs_bench.cpp`](../apps/secs_bench.cpp) | Throughput / latency / memory harness. |
|
||||
| `secsgem_tests` | All `tests/*.cpp` | The 445-case doctest binary. |
|
||||
| `fuzz_secs2_decode` | [`apps/fuzz_secs2_decode.cpp`](../apps/fuzz_secs2_decode.cpp) | libFuzzer (clang only, opt-in `-DSECSGEM_FUZZ=ON`). |
|
||||
| `fuzz_sml_parse` | [`apps/fuzz_sml_parse.cpp`](../apps/fuzz_sml_parse.cpp) | libFuzzer for the SML parser. |
|
||||
|
||||
A worked example binary `pvd_tool` (from `examples/pvd_tool/`) is
|
||||
also built by the same `CMakeLists.txt` when the example is
|
||||
included.
|
||||
|
||||
---
|
||||
|
||||
## How the build system finds everything
|
||||
|
||||
`CMakeLists.txt` does five things in order:
|
||||
|
||||
1. **Pull in dependencies** — `find_package(Threads)`,
|
||||
`find_package(yaml-cpp)`, `FetchContent` for doctest. Standalone
|
||||
Asio is header-only (no link step).
|
||||
2. **Run codegen** — invokes `tools/generate_messages.py` to turn
|
||||
`data/messages.yaml` into `build/generated/secsgem/gem/messages.hpp`.
|
||||
Listed as a custom command so it re-runs when `messages.yaml`
|
||||
changes.
|
||||
3. **Build the library** — `add_library(secsgem ...)` with every
|
||||
source under `src/` plus the generated header.
|
||||
4. **Build the apps** — one `add_executable` per `apps/*.cpp`,
|
||||
each linking against `secsgem`.
|
||||
5. **Build the tests** — `add_executable(secsgem_tests ...)` with
|
||||
every `tests/*.cpp`, linked against doctest + `secsgem`.
|
||||
|
||||
Build flags:
|
||||
|
||||
- **`-DSECSGEM_TSAN=ON`** — adds `-fsanitize=thread` to a
|
||||
separate build dir. CI runs this lane.
|
||||
- **`-DSECSGEM_FUZZ=ON`** — requires clang; adds libFuzzer + ASan +
|
||||
UBSan; builds the two fuzz harnesses.
|
||||
|
||||
Everything else (Release / Debug, parallelism, output dirs) is
|
||||
standard CMake.
|
||||
|
||||
---
|
||||
|
||||
## Test layout
|
||||
|
||||
50 test files; 445 test cases; 2 753 assertions. One file per
|
||||
concern. Naming is `test_<thing>.cpp` consistently:
|
||||
|
||||
- `test_secs2.cpp`, `test_e5_kat.cpp`, `test_sml.cpp`,
|
||||
`test_messages.cpp` — codec.
|
||||
- `test_hsms*.cpp` (5 files), `test_secsi*.cpp` (3 files) — transport.
|
||||
- `test_control_state.cpp`, `test_communication_state.cpp`,
|
||||
`test_data_model.cpp`, `test_host_handler.cpp`, `test_loader.cpp`,
|
||||
`test_config_validate.cpp` — E30.
|
||||
- `test_process_jobs.cpp`, `test_control_jobs.cpp`,
|
||||
`test_carriers.cpp`, `test_substrates.cpp`, `test_ept.cpp`,
|
||||
`test_modules.cpp`, `test_cem_objects.cpp`, `test_e84*.cpp`,
|
||||
`test_e42_formatted_pp.cpp` — GEM 300.
|
||||
- `test_*_persistence.cpp` (4) — file-backed journal.
|
||||
- `test_robustness_fuzz.cpp` — randomized property test.
|
||||
- `test_thread_safety.cpp` — TSan-validated single-threaded contract.
|
||||
- `test_metrics_prometheus.cpp` — Prometheus exporter.
|
||||
- `test_wire_ceid_emission.cpp` — CEID firings observed on a real socket.
|
||||
- `test_live_gem300.cpp`, `test_gem300_scenario.cpp` — multi-FSM cascades.
|
||||
|
||||
Full per-standard breakdown:
|
||||
[`docs/PROOFS.md`](PROOFS.md) "Per-standard test coverage" table.
|
||||
|
||||
---
|
||||
|
||||
## Where to go next
|
||||
|
||||
Now that you know what's where, the next chapter explains the
|
||||
*philosophy* that makes the codebase this small: the **spec-as-data**
|
||||
principle, and how the YAML files + codegen + runtime loader work
|
||||
together so adding a new SVID / state / message rarely requires C++.
|
||||
|
||||
Next: [→ 31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md)
|
||||
@@ -0,0 +1,395 @@
|
||||
# 31 — Spec-as-data and codegen
|
||||
|
||||
← [30 Repository tour](30_repository_tour.md) | [Back to index](00_index.md) | Next: [32 Stores and the data model](32_stores_and_the_data_model.md) →
|
||||
|
||||
The single design choice that keeps this codebase small is
|
||||
**spec-as-data**: every SEMI behavioural rule, state-transition
|
||||
table, and message body shape lives in YAML. The C++ is the engine
|
||||
that reads them.
|
||||
|
||||
This chapter explains:
|
||||
|
||||
- Why spec-as-data is the right choice for SECS/GEM specifically.
|
||||
- The five YAML files that drive everything.
|
||||
- How the message catalog gets codegen'd into typed C++.
|
||||
- How transition tables and equipment dictionaries load at runtime.
|
||||
- How to add a new SVID / state / message / host command without
|
||||
touching C++.
|
||||
|
||||
---
|
||||
|
||||
## Why spec-as-data here
|
||||
|
||||
Three properties of SECS/GEM standards push toward data-driven
|
||||
implementation:
|
||||
|
||||
1. **Per-tool variation is enormous.** Every fab tool has its own
|
||||
list of SVIDs, ECIDs, CEIDs, alarms. Hardcoding even one of
|
||||
them in C++ would mean recompiling per deployment.
|
||||
2. **The standards themselves are largely declarative.** E30 §6.2
|
||||
is a 5×8 transition table, not an algorithm. E5 §9 is a
|
||||
format-byte arithmetic. E40 §6 is a state graph. These map to
|
||||
YAML cleanly.
|
||||
3. **Customers need to audit the rules.** A fab QA team can read
|
||||
`data/control_state.yaml` and see the transitions; they can't
|
||||
read 600 lines of `if/else` and trust it the same way.
|
||||
|
||||
So the rule is: **anything a vendor or customer might want to
|
||||
change without recompiling lives in YAML**. The C++ is the
|
||||
runtime that reads it.
|
||||
|
||||
---
|
||||
|
||||
## The five YAML files
|
||||
|
||||
All under [`data/`](../data/).
|
||||
|
||||
### `messages.yaml` — the SECS-II message catalog
|
||||
|
||||
164 entries. Each one names a SECS-II message (SxFy) and describes
|
||||
its body's typed shape. Used at **build time** by codegen.
|
||||
|
||||
Excerpt:
|
||||
|
||||
```yaml
|
||||
- id: S1F1
|
||||
stream: 1
|
||||
function: 1
|
||||
w: true
|
||||
builder: s1f1
|
||||
parser: parse_s1f1
|
||||
body: none
|
||||
|
||||
- id: S1F2
|
||||
stream: 1
|
||||
function: 2
|
||||
w: false
|
||||
builder: s1f2
|
||||
parser: parse_s1f2
|
||||
body:
|
||||
kind: list
|
||||
struct_name: OnlineIdentification
|
||||
fields:
|
||||
- {name: mdln, shape: {kind: scalar, item_type: ASCII}}
|
||||
- {name: softrev, shape: {kind: scalar, item_type: ASCII}}
|
||||
```
|
||||
|
||||
The `body` field is the codegen's input. It supports:
|
||||
|
||||
- **`none`** — header-only, no body.
|
||||
- **`scalar`** — one Item; the codegen picks an appropriate C++
|
||||
parameter type from `item_type` (`ASCII`, `U1`–`U8`, etc.).
|
||||
- **`list`** — fixed-arity `<L,k>` with named fields. Optionally
|
||||
generates a `struct StructName { … }`.
|
||||
- **`list_of`** — variable-arity `<L,n>` with a uniform element
|
||||
shape.
|
||||
|
||||
The grammar is documented at the top of
|
||||
[`tools/gen_messages.py`](../tools/gen_messages.py) and at the top
|
||||
of [`data/messages.yaml`](../data/messages.yaml).
|
||||
|
||||
### `control_state.yaml` — the E30 §6.2 control state transition table
|
||||
|
||||
```yaml
|
||||
transitions:
|
||||
- {from: EquipmentOffline, on: operator_switch_online, to: AttemptOnline, then: OnlineRemote}
|
||||
- {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept}
|
||||
- {from: OnlineLocal, on: host_request_remote, ack: NotAccept}
|
||||
```
|
||||
|
||||
Loaded at **runtime** by `config::load_control_state_table`. The
|
||||
default table — used by tests when no YAML is given —
|
||||
mirrors this file exactly (in
|
||||
`ControlTransitionTable::default_table()`).
|
||||
|
||||
### `process_job_state.yaml` — the E40 PJ transition table
|
||||
|
||||
Same shape as control_state.yaml but for PJs. Drives
|
||||
`ProcessJobStateMachine`.
|
||||
|
||||
### `control_job_state.yaml` — the E94 CJ transition table
|
||||
|
||||
Same shape for CJs. Drives `ControlJobStateMachine`.
|
||||
|
||||
### `equipment.yaml` — the demo equipment data dictionary
|
||||
|
||||
Excerpt:
|
||||
|
||||
```yaml
|
||||
device:
|
||||
mdln: "SECS-GEM Demo Equipment"
|
||||
softrev: "1.0.0"
|
||||
capabilities: [Establish, OnLine, …]
|
||||
|
||||
svids:
|
||||
- {id: 1, name: ControlState, units: "", type: ASCII, value: ""}
|
||||
- {id: 2, name: Clock, units: "", type: ASCII, value: ""}
|
||||
- {id: 3, name: WaferCounter, units: "wafers", type: U4, value: 0}
|
||||
|
||||
ecids:
|
||||
- {id: 100, name: T3, units: "s", type: U4, value: 45, min: 1, max: 600}
|
||||
|
||||
ceids:
|
||||
- {id: 100, name: ControlStateChange}
|
||||
- {id: 300, name: ProcessStarted}
|
||||
|
||||
alarms:
|
||||
- {id: 1, alcd: 0x84, text: "Chamber pressure above threshold"}
|
||||
|
||||
host_commands:
|
||||
- {name: START, ack: Accept, emit_ceid: 300}
|
||||
- {name: FAULT, ack: Accept, set_alarm: 1}
|
||||
```
|
||||
|
||||
Loaded at startup by `config::load_equipment`. Every key under
|
||||
this YAML maps to a typed struct in `config::EquipmentDescriptor`.
|
||||
|
||||
`examples/pvd_tool/equipment.yaml` is a more realistic version
|
||||
with 29 SVIDs, 7 ECIDs, 21 CEIDs, 12 alarms.
|
||||
|
||||
---
|
||||
|
||||
## The codegen pass
|
||||
|
||||
`messages.yaml` is too large and too repetitive to write by hand —
|
||||
164 messages × (builder + parser + struct + tests) would be ~5 k
|
||||
lines of boilerplate. Instead, `tools/gen_messages.py` reads the
|
||||
YAML at build time and emits one inline header:
|
||||
**`build/generated/secsgem/gem/messages.hpp`**.
|
||||
|
||||
### What gets generated
|
||||
|
||||
Per message, the codegen emits:
|
||||
|
||||
```cpp
|
||||
namespace secsgem::gem::messages {
|
||||
|
||||
// Optional struct if body has `struct_name`.
|
||||
struct OnlineIdentification {
|
||||
std::string mdln;
|
||||
std::string softrev;
|
||||
bool operator==(const OnlineIdentification&) const = default;
|
||||
};
|
||||
|
||||
// Builder: takes typed params, returns a secs2::Message.
|
||||
inline secs2::Message s1f1();
|
||||
inline secs2::Message s1f2(const std::string& mdln, const std::string& softrev);
|
||||
|
||||
// Parser: takes a Message body, returns std::optional<Struct> (or the
|
||||
// primitive type for scalar bodies).
|
||||
inline std::optional<OnlineIdentification> parse_s1f2(const secs2::Item& body);
|
||||
|
||||
} // namespace
|
||||
```
|
||||
|
||||
For ~160 named messages, the generated header is ~3 500 lines, all
|
||||
inline. Tests in
|
||||
[`tests/test_messages.cpp`](../tests/test_messages.cpp) (82 cases)
|
||||
exercise every builder + parser round-trip.
|
||||
|
||||
### How CMake invokes it
|
||||
|
||||
CMakeLists.txt has a custom command:
|
||||
|
||||
```cmake
|
||||
add_custom_command(
|
||||
OUTPUT ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp
|
||||
COMMAND ${Python3_EXECUTABLE}
|
||||
${CMAKE_SOURCE_DIR}/tools/gen_messages.py
|
||||
${CMAKE_SOURCE_DIR}/data/messages.yaml
|
||||
${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp
|
||||
DEPENDS ${CMAKE_SOURCE_DIR}/data/messages.yaml
|
||||
${CMAKE_SOURCE_DIR}/tools/gen_messages.py
|
||||
)
|
||||
```
|
||||
|
||||
Re-runs on `data/messages.yaml` edits *or* on
|
||||
`tools/gen_messages.py` edits. Generated header goes into a
|
||||
sibling include directory so the library can include it as
|
||||
`#include "secsgem/gem/messages.hpp"`.
|
||||
|
||||
### Why Python rather than templates / constexpr
|
||||
|
||||
Three reasons:
|
||||
|
||||
1. **YAML parsing** — full grammar matters and `PyYAML` is more
|
||||
reliable than yaml-cpp at parse-time gymnastics.
|
||||
2. **Code shape control** — the generated C++ is easier to read
|
||||
when generated by a textual templater than by C++ metaprogramming.
|
||||
3. **Debuggability** — a customer who wants to see "what code is
|
||||
actually being run for S2F33" can `grep` the generated header.
|
||||
No mystery types, no instantiation chains.
|
||||
|
||||
The codegen is ~388 lines of Python; the input grammar is
|
||||
documented at its top.
|
||||
|
||||
---
|
||||
|
||||
## Runtime loading
|
||||
|
||||
The other four YAMLs (`control_state`, `process_job_state`,
|
||||
`control_job_state`, `equipment`) load at runtime, not build time.
|
||||
The same loader handles all of them:
|
||||
|
||||
```cpp
|
||||
// include/secsgem/config/loader.hpp
|
||||
namespace secsgem::config {
|
||||
EquipmentDescriptor load_equipment(const std::string& path);
|
||||
ControlStateConfig load_control_state_table(const std::string& path);
|
||||
ProcessJobStateConfig load_process_job_state(const std::string& path);
|
||||
ControlJobStateConfig load_control_job_state(const std::string& path);
|
||||
}
|
||||
```
|
||||
|
||||
Each `load_*` returns a typed config struct on success or throws on
|
||||
malformed YAML. Throwing is OK because YAML loading happens once
|
||||
at startup — before binding the port — so a malformed file fails
|
||||
the process up front.
|
||||
|
||||
---
|
||||
|
||||
## The `--validate-config` pass
|
||||
|
||||
YAML loaders that throw on first error are unfriendly: customers
|
||||
often have multiple typos in a new equipment.yaml. The codebase
|
||||
ships a multi-error validator:
|
||||
|
||||
```cpp
|
||||
// include/secsgem/config/validate.hpp
|
||||
class ConfigValidator {
|
||||
public:
|
||||
void validate_equipment(const std::string& path);
|
||||
void validate_control_state(const std::string& path);
|
||||
// ...
|
||||
|
||||
bool has_errors() const;
|
||||
void format_issues_to(std::ostream&, …) const;
|
||||
};
|
||||
```
|
||||
|
||||
It tries to load each file, accumulates *every* issue it can find,
|
||||
and prints them all. Then exits 0 or 1.
|
||||
|
||||
Invoke via:
|
||||
|
||||
```bash
|
||||
secs_server --validate-config \
|
||||
--config data/equipment.yaml \
|
||||
--state-table data/control_state.yaml \
|
||||
--pj-state-table data/process_job_state.yaml \
|
||||
--cj-state-table data/control_job_state.yaml
|
||||
```
|
||||
|
||||
This is proof #5 in [PROOFS.md](PROOFS.md) — runs in CI to
|
||||
guarantee every shipped YAML is structurally + referentially
|
||||
sound.
|
||||
|
||||
Tests:
|
||||
[`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp)
|
||||
(8 cases — every category of validation issue).
|
||||
|
||||
---
|
||||
|
||||
## How to add a capability without C++
|
||||
|
||||
The point of spec-as-data is that **adding behaviour almost never
|
||||
requires a C++ change**.
|
||||
|
||||
### New SVID
|
||||
|
||||
```yaml
|
||||
# data/equipment.yaml
|
||||
svids:
|
||||
- {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}
|
||||
```
|
||||
|
||||
Restart. Done. Host can now read SVID 4 via S1F3.
|
||||
|
||||
### New CEID with linked report
|
||||
|
||||
```yaml
|
||||
# data/equipment.yaml
|
||||
ceids:
|
||||
- {id: 350, name: ChamberTempHigh}
|
||||
|
||||
events:
|
||||
default_reports:
|
||||
- {ceid: 350, vids: [4]}
|
||||
```
|
||||
|
||||
Restart. Done. When the EAP fires CEID 350, the report carries
|
||||
SVID 4 automatically.
|
||||
|
||||
### New host command
|
||||
|
||||
```yaml
|
||||
host_commands:
|
||||
- {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}
|
||||
```
|
||||
|
||||
Restart. Done. Host sends `S2F41(RCMD=VENT)` → ACK=Accept,
|
||||
CEID 400 fires, ALID 2 set.
|
||||
|
||||
### New control-state transition
|
||||
|
||||
```yaml
|
||||
# data/control_state.yaml
|
||||
transitions:
|
||||
- {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept}
|
||||
```
|
||||
|
||||
Restart. Done.
|
||||
|
||||
### New SECS-II message
|
||||
|
||||
```yaml
|
||||
# data/messages.yaml
|
||||
- id: S6F30
|
||||
stream: 6
|
||||
function: 30
|
||||
w: true
|
||||
builder: s6f30_request
|
||||
parser: parse_s6f30
|
||||
body:
|
||||
kind: list
|
||||
struct_name: TempQuery
|
||||
fields:
|
||||
- {name: vid, shape: {kind: scalar, item_type: U4}}
|
||||
```
|
||||
|
||||
`docker compose run --rm builder` regenerates `messages.hpp`. A
|
||||
new `s6f30_request(uint32_t vid)` builder and a `parse_s6f30(item)
|
||||
→ std::optional<TempQuery>` parser appear. Now the *handler* is
|
||||
still C++ — `gem::Router::on(6, 30, ...)` — because the side-effect
|
||||
of "host asked for the temperature" needs application logic.
|
||||
|
||||
---
|
||||
|
||||
## When spec-as-data isn't the right fit
|
||||
|
||||
Three categories that *do* need C++:
|
||||
|
||||
1. **Application logic** — what an alarm threshold actually is,
|
||||
how a recipe step gets executed. No YAML schema can express
|
||||
"vent the chamber if pressure > 1 Torr."
|
||||
2. **State-machine actions** — when a CJ transitions to Executing,
|
||||
*which* PJ to select next isn't a table entry; it's an
|
||||
algorithm.
|
||||
3. **External integrations** — talking to a PLC, reading a sensor,
|
||||
driving a robot. Hardware bindings are vendor-specific code.
|
||||
|
||||
The codebase draws the line **at the message catalog and the
|
||||
transition tables**. Everything below (codec, transport) is fixed
|
||||
C++. Everything above (application wiring) is per-EAP C++.
|
||||
Everything between (data dictionary + state model) is YAML.
|
||||
|
||||
---
|
||||
|
||||
## Where to go next
|
||||
|
||||
You now know how the YAML drives the runtime. The next chapter
|
||||
gets concrete about the **stores** — the per-domain bundles
|
||||
(SVIDs, CEIDs, alarms, carriers, …) that the YAML populates and
|
||||
the Router handlers operate over.
|
||||
|
||||
Next: [→ 32 Stores and the data model](32_stores_and_the_data_model.md)
|
||||
@@ -0,0 +1,308 @@
|
||||
# 32 — Stores and the data model
|
||||
|
||||
← [31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md) | [Back to index](00_index.md) | Next: [33 Transport](33_transport.md) →
|
||||
|
||||
The previous chapter showed how YAML drives behaviour. This
|
||||
chapter shows the runtime data structures that the YAML populates
|
||||
and that the Router handlers operate on: **stores** and the
|
||||
**`EquipmentDataModel`** that composes them.
|
||||
|
||||
By the end you'll know:
|
||||
|
||||
- What a **store** is (and what it isn't).
|
||||
- Every store in the codebase, one sentence each.
|
||||
- How `EquipmentDataModel` composes them.
|
||||
- The "no locks; single-threaded" contract.
|
||||
- How to add a new store.
|
||||
|
||||
---
|
||||
|
||||
## What a store is
|
||||
|
||||
A **store** is a per-domain bundle of:
|
||||
|
||||
- A few typed records (`std::map`, `std::vector`, …).
|
||||
- A small API for reading + mutating them.
|
||||
- A `change_handler` that emits events on transitions.
|
||||
- Optional file-backed persistence.
|
||||
|
||||
The naming is consistent: `AlarmRegistry` for active alarms,
|
||||
`CarrierStore` for carriers, `ProcessJobStore` for PJs.
|
||||
Headers live in
|
||||
[`include/secsgem/gem/store/`](../include/secsgem/gem/store/);
|
||||
implementations are typically inline in the header (these are small).
|
||||
|
||||
Each store maps onto **one concern from one SEMI standard**:
|
||||
|
||||
| Store | Concern | Standard |
|
||||
|-----------------------------|--------------------------------------------------|----------------|
|
||||
| `StatusVariableStore` | SVIDs + values | E30 §6.13 |
|
||||
| `DataVariableStore` | DVIDs + values | E30 §6.11 |
|
||||
| `EquipmentConstantStore` | ECIDs + values + min/max bounds | E30 §6.16 |
|
||||
| `EventReportSubscriptions` | RPTID definitions + CEID linkings + enables | E30 §6.6 |
|
||||
| `AlarmRegistry` | ALIDs + ALCD/ALTX + enable bits + active set | E30 §6.14 |
|
||||
| `RecipeStore` | PPIDs + PPBODY (unformatted) + formatted bodies | E30 §6.17 + E42|
|
||||
| `Clock` | Wall-clock + drift + quality | E30 §6.20 + E148 |
|
||||
| `HostCommandRegistry` | RCMD names + per-command ack + side effects | E30 §6.15 |
|
||||
| `SpoolStore` | Per-stream whitelist + queue + persistent journal| E30 §6.22 |
|
||||
| `LimitMonitorStore` | LIMITIDs + upper/lower bounds + active state | E30 §6.21 |
|
||||
| `TraceStore` | TRIDs + active sampling config | E30 §6.12 |
|
||||
| `ProcessJobStore` | PJs + state + material list + persistent | E40 |
|
||||
| `ControlJobStore` | CJs + state + PJ refs + persistent | E94 |
|
||||
| `ExceptionStore` | EXIDs + recovery state + persistent | E5 §13 |
|
||||
| `CarrierStore` | Carrier IDs + state machines + persistent | E87 |
|
||||
| `LoadPortStore` | LP IDs + transfer/reservation/association FSMs | E87 |
|
||||
| `SubstrateStore` | Substrate IDs + 3 FSMs + location + persistent | E90 |
|
||||
| `EptStateMachine` | EPT state + time buckets | E116 |
|
||||
| `CemObjectStore` | E120 typed object hierarchy | E120 |
|
||||
| `ModuleStore` | Module IDs + state | E157 |
|
||||
| `E84PortStore` | Per-LP E84 FSM + signals + timers | E84 |
|
||||
|
||||
Each is a header. Each is independently testable: you can
|
||||
`#include "secsgem/gem/store/alarms.hpp"` and exercise
|
||||
`AlarmRegistry` without pulling in the rest. This is the same
|
||||
shape as the per-standard tests in
|
||||
[`tests/`](../tests/) — one test file per store.
|
||||
|
||||
---
|
||||
|
||||
## EquipmentDataModel — the composite
|
||||
|
||||
[`include/secsgem/gem/data_model.hpp`](../include/secsgem/gem/data_model.hpp)
|
||||
defines:
|
||||
|
||||
```cpp
|
||||
struct EquipmentDataModel {
|
||||
StatusVariableStore svids;
|
||||
DataVariableStore dvids;
|
||||
EquipmentConstantStore ecids;
|
||||
EventReportSubscriptions events;
|
||||
AlarmRegistry alarms;
|
||||
RecipeStore recipes;
|
||||
Clock clock;
|
||||
HostCommandRegistry commands;
|
||||
SpoolStore spool;
|
||||
LimitMonitorStore limits;
|
||||
TraceStore traces;
|
||||
ProcessJobStore process_jobs;
|
||||
ControlJobStore control_jobs;
|
||||
ExceptionStore exceptions;
|
||||
CarrierStore carriers;
|
||||
LoadPortStore load_ports;
|
||||
SubstrateStore substrates;
|
||||
EptStateMachine ept;
|
||||
CemObjectStore cem;
|
||||
ModuleStore modules;
|
||||
E84PortStore e84_ports;
|
||||
// ... convenience methods spanning stores
|
||||
};
|
||||
```
|
||||
|
||||
That's it. No locks, no smart pointers, no interfaces, no DI
|
||||
container. Each store is a value member; ownership is the
|
||||
`EquipmentDataModel` itself.
|
||||
|
||||
The application typically holds one `shared_ptr<EquipmentDataModel>`
|
||||
and passes it to every Router handler. Handlers operate on the
|
||||
stores directly:
|
||||
|
||||
```cpp
|
||||
router.on(1, 3, [model](const secs2::Message& m) {
|
||||
// S1F3 — host requests SVID values
|
||||
auto svids = parse_s1f3(m.body());
|
||||
return build_s1f4(model->svids.values(svids));
|
||||
});
|
||||
```
|
||||
|
||||
### Convenience methods
|
||||
|
||||
`EquipmentDataModel` adds a few cross-store helpers (`data_model.hpp:54`):
|
||||
|
||||
```cpp
|
||||
std::optional<s2::Item> vid_value(uint32_t vid) const {
|
||||
// Look up VID in svids first, then dvids.
|
||||
}
|
||||
|
||||
std::vector<ReportData> compose_reports_for(uint32_t ceid) const {
|
||||
// Walk events store -> reports store -> svids/dvids,
|
||||
// assemble the S6F11 report payload for one CEID firing.
|
||||
}
|
||||
```
|
||||
|
||||
`compose_reports_for` is the *heart* of event notification — it
|
||||
walks three stores to assemble the body for one S6F11 frame. See
|
||||
chapter [13](13_e30_gem.md) for the wire flow.
|
||||
|
||||
---
|
||||
|
||||
## The single-threaded contract
|
||||
|
||||
**Every store mutation runs on the io_context strand.** No locks,
|
||||
no atomics, no condition variables. This is documented in
|
||||
[`docs/INTEGRATION.md`](INTEGRATION.md) §3 and enforced under
|
||||
ThreadSanitizer.
|
||||
|
||||
Why? Two reasons:
|
||||
|
||||
1. **Performance.** Locking a `std::map` for every SVID read is a
|
||||
waste in a hot path that processes thousands of messages a
|
||||
second. The asio strand model gives the same correctness
|
||||
guarantee for free.
|
||||
2. **Simplicity.** Every method on every store is the obvious
|
||||
non-locking implementation. Reading the code, you don't have
|
||||
to track which lock protects what.
|
||||
|
||||
The cost: **callers from other threads must `asio::post` onto
|
||||
the executor**.
|
||||
|
||||
```cpp
|
||||
// From a sensor thread:
|
||||
asio::post(io_context, [model, vid, value] {
|
||||
model->svids.set_value(vid, secs2::Item::f4(value));
|
||||
});
|
||||
```
|
||||
|
||||
Tested by
|
||||
[`tests/test_thread_safety.cpp`](../tests/test_thread_safety.cpp)
|
||||
under TSan: N producer threads `asio::post` updates; TSan reports
|
||||
zero races. Chapter [33](33_transport.md) covers the strand model
|
||||
in more detail.
|
||||
|
||||
---
|
||||
|
||||
## How a store's API looks (a small one)
|
||||
|
||||
Pick `AlarmRegistry` — one of the smallest:
|
||||
|
||||
```cpp
|
||||
class AlarmRegistry {
|
||||
public:
|
||||
// Register an alarm definition.
|
||||
void register_alarm(uint32_t alid, uint8_t alcd, const std::string& altx);
|
||||
|
||||
// Set / clear an active alarm. Fires the change handler.
|
||||
void set(uint32_t alid);
|
||||
void clear(uint32_t alid);
|
||||
|
||||
// Enable / disable host notification (S5F3).
|
||||
void set_enabled(uint32_t alid, bool enabled);
|
||||
bool is_enabled(uint32_t alid) const;
|
||||
|
||||
// List active / all alarms.
|
||||
std::vector<AlarmDefinition> all() const;
|
||||
std::vector<AlarmDefinition> active() const;
|
||||
|
||||
// Observer: change handler signature.
|
||||
using ChangeHandler = std::function<void(uint32_t alid, bool set)>;
|
||||
void set_change_handler(ChangeHandler);
|
||||
};
|
||||
```
|
||||
|
||||
Every store follows that same shape: mutator + reader + observer.
|
||||
The Router handler for `S5F1` doesn't fire `S5F1` itself — it
|
||||
mutates the store; the change handler (registered at startup by
|
||||
the EAP) fires `S5F1` via the connection.
|
||||
|
||||
---
|
||||
|
||||
## How a store's API looks (a bigger one)
|
||||
|
||||
[`ProcessJobStore`](../include/secsgem/gem/store/process_jobs.hpp)
|
||||
adds:
|
||||
|
||||
- Submit a PJ (record entry + fire `Created` event).
|
||||
- Get / set state of any PJ.
|
||||
- Apply a host-driven event (PJSTART / PJPAUSE / …) and route to
|
||||
the FSM.
|
||||
- Iterate active PJs (for serializing on restart).
|
||||
- Persistent journal: `enable_persistence(dir)`.
|
||||
|
||||
The FSM logic isn't *inside* the store — `ProcessJobStateMachine`
|
||||
in [`process_job_state.hpp`](../include/secsgem/gem/process_job_state.hpp)
|
||||
owns transitions. The store holds one `ProcessJobStateMachine`
|
||||
per PJ and dispatches.
|
||||
|
||||
This separation — *store* (records) vs *state machine* (transitions) —
|
||||
keeps each layer testable in isolation.
|
||||
|
||||
---
|
||||
|
||||
## Persistence
|
||||
|
||||
Six stores have file-backed persistence: spool, process_jobs,
|
||||
control_jobs, exceptions, carriers, load_ports, substrates.
|
||||
|
||||
Each opts in via `enable_persistence(dir)`:
|
||||
|
||||
```cpp
|
||||
model->process_jobs.enable_persistence("/var/lib/secsgem/pj");
|
||||
```
|
||||
|
||||
That:
|
||||
|
||||
1. Creates the directory if needed.
|
||||
2. **Replays** every record file found there back into in-memory
|
||||
state.
|
||||
3. Sets up the on-disk journal: every mutation writes (or rewrites,
|
||||
or deletes) one file per record, named by ID.
|
||||
|
||||
Per-record-per-file means the journal is **partial-write safe**:
|
||||
if the equipment power-cycles mid-write of one record, the others
|
||||
are untouched; the partial file is detected and dropped at the
|
||||
next startup.
|
||||
|
||||
Chapter [36](36_persistence_validation_metrics.md) walks the
|
||||
mechanism, the multi-version reads, and the test patterns.
|
||||
|
||||
---
|
||||
|
||||
## How to add a new store
|
||||
|
||||
Two cases:
|
||||
|
||||
### Case 1: Standard already implemented, new sub-area
|
||||
|
||||
E.g., add a "Reticle" store to track lithography reticles
|
||||
distinctly from substrates.
|
||||
|
||||
1. Create `include/secsgem/gem/store/reticles.hpp` with a class
|
||||
`ReticleStore` exposing the standard
|
||||
register / set-state / get / change-handler shape.
|
||||
2. Add a member to `EquipmentDataModel`:
|
||||
```cpp
|
||||
ReticleStore reticles;
|
||||
```
|
||||
3. Write `tests/test_reticles.cpp` mirroring the pattern from any
|
||||
other store's test.
|
||||
4. Wire Router handlers in `apps/secs_server.cpp` (or the EAP) for
|
||||
whatever S/F messages drive it.
|
||||
|
||||
### Case 2: Brand new SEMI standard
|
||||
|
||||
E.g., implement E170 (a new GEM standard).
|
||||
|
||||
Same as case 1, plus:
|
||||
|
||||
5. Update [`data/messages.yaml`](../data/messages.yaml) with any
|
||||
new S/F messages. `docker compose run --rm builder` regens
|
||||
`messages.hpp`.
|
||||
6. If E170 has its own transition table, create
|
||||
`data/e170_state.yaml` and a `load_e170_state(...)` loader in
|
||||
`config::`.
|
||||
7. Update [`docs/COMPLIANCE.md`](COMPLIANCE.md) with the new
|
||||
capability row.
|
||||
|
||||
The architecture is **specifically designed** to add new standards
|
||||
without disturbing existing ones.
|
||||
|
||||
---
|
||||
|
||||
## Where to go next
|
||||
|
||||
You've now seen how every per-domain data record is shaped and
|
||||
how `EquipmentDataModel` composes them. Next, we drop back down
|
||||
to transport: how `hsms::Connection` and `secsi::Protocol` actually
|
||||
move bytes, and the asio strand model that makes the
|
||||
single-threaded contract work.
|
||||
|
||||
Next: [→ 33 Transport](33_transport.md)
|
||||
@@ -0,0 +1,292 @@
|
||||
# 33 — Transport
|
||||
|
||||
← [32 Stores and the data model](32_stores_and_the_data_model.md) | [Back to index](00_index.md) | Next: [34 Codec and SML](34_codec_and_sml.md) →
|
||||
|
||||
We covered the standards-level view of HSMS and SECS-I in
|
||||
chapters 11 and 12. This chapter drops down into the
|
||||
implementation: how `hsms::Connection` actually moves bytes,
|
||||
the asio executor model, the single-threaded strand contract,
|
||||
and why the transport layer doesn't need locks.
|
||||
|
||||
---
|
||||
|
||||
## The two transport modules
|
||||
|
||||
```
|
||||
include/secsgem/hsms/
|
||||
├── header.hpp — Frame format primitives (length prefix, header, SType).
|
||||
└── connection.hpp — One-socket session manager + T-timers + S9 emission.
|
||||
|
||||
include/secsgem/secsi/
|
||||
├── header.hpp — 10-byte SECS-I block header.
|
||||
├── block.hpp — Block split / assemble (multi-block messages).
|
||||
├── protocol.hpp — IO-free line-turnaround FSM.
|
||||
└── tcp_transport.hpp — asio TCP wrapper around the FSM (tunnel for testing).
|
||||
```
|
||||
|
||||
Each module owns one TCP endpoint (or in the SECS-I case, a tunnel
|
||||
endpoint). Both are **single-threaded by design**.
|
||||
|
||||
---
|
||||
|
||||
## hsms::Connection — top to bottom
|
||||
|
||||
### Lifecycle
|
||||
|
||||
```cpp
|
||||
// apps/secs_server.cpp — passive-equipment startup
|
||||
asio::io_context io;
|
||||
|
||||
asio::ip::tcp::acceptor acc(io, asio::ip::tcp::endpoint{asio::ip::tcp::v4(), port});
|
||||
|
||||
acc.async_accept([&](std::error_code ec, asio::ip::tcp::socket sock) {
|
||||
auto conn = std::make_shared<hsms::Connection>(
|
||||
std::move(sock), Mode::Passive, /*device_id=*/0, timers);
|
||||
conn->set_message_handler(...);
|
||||
conn->set_closed_handler(...);
|
||||
conn->start();
|
||||
});
|
||||
|
||||
io.run(); // blocks until all work is done
|
||||
```
|
||||
|
||||
`Connection::start()` either:
|
||||
|
||||
- **Passive** — arms T7 (waiting for Select.req) and starts the
|
||||
read loop.
|
||||
- **Active** — initiates the Select.req exchange, then starts the
|
||||
read loop.
|
||||
|
||||
### Read path
|
||||
|
||||
Three async steps repeated forever:
|
||||
|
||||
```
|
||||
async_read(socket, 4 bytes) → on_length()
|
||||
async_read(socket, length bytes) → on_payload()
|
||||
handle_frame(decoded Frame) → dispatch
|
||||
```
|
||||
|
||||
In code, [`src/hsms/connection.cpp`](../src/hsms/connection.cpp):
|
||||
|
||||
```cpp
|
||||
void Connection::read_length() {
|
||||
asio::async_read(socket_, asio::buffer(len_buf_, 4),
|
||||
[self = shared_from_this()](std::error_code ec, std::size_t n) {
|
||||
self->on_length(ec, n);
|
||||
});
|
||||
}
|
||||
|
||||
void Connection::on_length(std::error_code ec, std::size_t n) {
|
||||
if (ec) return close("read_length");
|
||||
uint32_t len = decode_be32(len_buf_);
|
||||
payload_.resize(len);
|
||||
asio::async_read(socket_, asio::buffer(payload_),
|
||||
[self = shared_from_this()](std::error_code ec, std::size_t n) {
|
||||
self->on_payload(ec, n);
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
Each callback is on the socket's executor. No locks because
|
||||
nothing else can be touching the read state — by construction.
|
||||
|
||||
### Write path
|
||||
|
||||
A send queue + one outstanding `async_write`:
|
||||
|
||||
```cpp
|
||||
void Connection::send_frame(Frame frame) {
|
||||
send_queue_.push_back(std::move(frame));
|
||||
if (send_queue_.size() == 1) write_next();
|
||||
}
|
||||
|
||||
void Connection::write_next() {
|
||||
auto& frame = send_queue_.front();
|
||||
send_buf_ = frame.encode();
|
||||
asio::async_write(socket_, asio::buffer(send_buf_),
|
||||
[self = shared_from_this()](std::error_code ec, std::size_t) {
|
||||
self->send_queue_.pop_front();
|
||||
if (ec) return self->close("write");
|
||||
if (!self->send_queue_.empty()) self->write_next();
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
Same single-threaded discipline — `send_queue_` is only touched
|
||||
on the executor. Callers from other threads must `asio::post`.
|
||||
|
||||
### Timers
|
||||
|
||||
Five `asio::steady_timer`s, one per HSMS T-timer:
|
||||
|
||||
```cpp
|
||||
// src/hsms/connection.cpp:30
|
||||
Connection::Connection(...)
|
||||
: socket_(std::move(sock)),
|
||||
t3_timer_(socket_.get_executor()),
|
||||
t6_timer_(socket_.get_executor()),
|
||||
t7_timer_(socket_.get_executor()),
|
||||
t8_timer_(socket_.get_executor()),
|
||||
linktest_timer_(socket_.get_executor()),
|
||||
timers_(timers) { }
|
||||
```
|
||||
|
||||
All five share the socket's executor. When a timer fires, its
|
||||
handler runs on the same executor as the read/write loop — so
|
||||
again no locks for timer-vs-IO interaction.
|
||||
|
||||
T3 is special: there's one T3 timer per in-flight W=1 message
|
||||
(correlated by `system_bytes`). These are short-lived
|
||||
`steady_timer`s allocated when the request is sent and destroyed
|
||||
when the reply arrives. See `src/hsms/connection.cpp:447`:
|
||||
|
||||
```cpp
|
||||
auto t3 = std::make_shared<asio::steady_timer>(socket_.get_executor());
|
||||
t3->expires_after(timers_.t3);
|
||||
in_flight_.insert({system_bytes, RequestState{std::move(cb), t3, ...}});
|
||||
t3->async_wait([self, system_bytes](std::error_code ec) {
|
||||
if (ec) return; // cancelled (reply arrived)
|
||||
self->on_t3_expire(system_bytes);
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The asio executor / strand model
|
||||
|
||||
All `Connection` state lives on **one executor**. In simple cases
|
||||
that's just `io_context.get_executor()` — a single-threaded loop.
|
||||
In production, an EAP may run multiple `io_context::run()` threads
|
||||
*per connection* by wrapping work in an `asio::strand`.
|
||||
|
||||
### What a strand is
|
||||
|
||||
A **strand** is an executor that guarantees mutual exclusion
|
||||
between handlers it dispatches. Multiple threads can call
|
||||
`io.run()`; the strand picks one of them at a time to run the
|
||||
next handler in its queue.
|
||||
|
||||
For HSMS purposes, `socket_.get_executor()` already gives strand
|
||||
semantics if the underlying `io_context` is single-threaded. For
|
||||
multi-threaded `io_context`, the application wraps with
|
||||
`asio::make_strand`:
|
||||
|
||||
```cpp
|
||||
auto strand = asio::make_strand(io);
|
||||
asio::ip::tcp::socket sock(strand);
|
||||
auto conn = std::make_shared<hsms::Connection>(std::move(sock), ...);
|
||||
```
|
||||
|
||||
Now `conn` has all its IO running on `strand`, while the
|
||||
`io_context` can use 8 threads to handle 100 different connections.
|
||||
|
||||
### What this means for the caller
|
||||
|
||||
From any thread that isn't the strand's currently-running handler,
|
||||
the caller MUST marshal onto the strand:
|
||||
|
||||
```cpp
|
||||
// From a sensor-thread callback:
|
||||
asio::post(conn->executor(), [conn, msg = std::move(msg)] {
|
||||
conn->send_data(std::move(msg));
|
||||
});
|
||||
```
|
||||
|
||||
Calling `conn->send_data` directly from another thread is **a
|
||||
race**. Same for any store mutation. TSan catches this and the
|
||||
test suite enforces it.
|
||||
|
||||
The contract is documented in detail in
|
||||
[`docs/INTEGRATION.md`](INTEGRATION.md) §3.
|
||||
|
||||
---
|
||||
|
||||
## secsi::Protocol — the IO-free FSM
|
||||
|
||||
SECS-I's protocol layer is structured differently: the FSM has
|
||||
**no IO at all**. It takes events (bytes received, application
|
||||
asked to send, timer fired) and produces a list of `Action`s
|
||||
(transmit these bytes, arm a timer, deliver a block to the
|
||||
application).
|
||||
|
||||
```cpp
|
||||
// include/secsgem/secsi/protocol.hpp
|
||||
struct ActionTransmit { std::vector<uint8_t> bytes; };
|
||||
struct ActionStartTimer { Timer which; };
|
||||
struct ActionCancelTimer { Timer which; };
|
||||
struct ActionDeliverBlock { Block block; };
|
||||
struct ActionRaiseError { std::string reason; };
|
||||
```
|
||||
|
||||
The wrapper (`secsi::TcpTransport`) drives the FSM:
|
||||
|
||||
```cpp
|
||||
void TcpTransport::on_byte(uint8_t b) {
|
||||
auto actions = protocol_.handle(EventByte{b});
|
||||
for (const auto& a : actions) execute(a);
|
||||
}
|
||||
|
||||
void TcpTransport::execute(const Action& a) {
|
||||
std::visit([this](auto&& v) {
|
||||
using T = std::decay_t<decltype(v)>;
|
||||
if constexpr (std::is_same_v<T, ActionTransmit>) write_bytes(v.bytes);
|
||||
else if constexpr (std::is_same_v<T, ActionStartTimer>) arm_timer(v.which);
|
||||
else if constexpr (std::is_same_v<T, ActionCancelTimer>) cancel_timer(v.which);
|
||||
else if constexpr (std::is_same_v<T, ActionDeliverBlock>) deliver(v.block);
|
||||
else if constexpr (std::is_same_v<T, ActionRaiseError>) raise(v.reason);
|
||||
}, a);
|
||||
}
|
||||
```
|
||||
|
||||
This design makes the **whole FSM** unit-testable. No sockets,
|
||||
no timers, just `Event` in → `Action` out. Tests:
|
||||
|
||||
- [`tests/test_secsi.cpp`](../tests/test_secsi.cpp) — basic FSM
|
||||
state walks.
|
||||
- [`tests/test_secsi_timers.cpp`](../tests/test_secsi_timers.cpp)
|
||||
— every timer scenario via synthetic `EventTimeout` injection.
|
||||
- [`tests/test_secsi_tcp.cpp`](../tests/test_secsi_tcp.cpp) —
|
||||
end-to-end via `TcpTransport`.
|
||||
|
||||
Same pattern repeats for E84 (chapter 18) and the GEM
|
||||
communication-state FSM (chapter 13): IO-free FSM + asio adapter
|
||||
+ separate test suites for each layer.
|
||||
|
||||
---
|
||||
|
||||
## Why this is the right shape
|
||||
|
||||
### Pros of single-threaded + IO-free
|
||||
|
||||
- **No mutexes.** Anywhere.
|
||||
- **Trivial reasoning.** When you read `Connection::send_frame`,
|
||||
you can be sure nothing else is mutating the queue.
|
||||
- **Fast.** No lock contention, no atomic round-trips.
|
||||
- **Testable.** IO-free FSM lets you exercise every transition
|
||||
without IO.
|
||||
|
||||
### Cons
|
||||
|
||||
- **Callers must know about the strand.** Multi-threaded
|
||||
applications need `asio::post` boilerplate.
|
||||
- **One slow handler blocks the rest.** A 100 ms handler delays
|
||||
every other message until it returns. Fix: don't write slow
|
||||
handlers; if you must, dispatch the slow work to another
|
||||
executor and return immediately.
|
||||
|
||||
For a SECS/GEM equipment runtime — where the natural shape is "one
|
||||
TCP socket, one event loop" — the pros far outweigh the cons.
|
||||
|
||||
---
|
||||
|
||||
## Where to go next
|
||||
|
||||
You've now seen the bottom layer in detail: how bytes move,
|
||||
how state machines drive transitions without IO, how the single-
|
||||
threaded contract makes everything safe. Next chapter goes back
|
||||
up one level: **the codec** — the encoder/decoder that turns
|
||||
the Item type from chapter 10 into wire bytes, plus the SML
|
||||
human-readable form.
|
||||
|
||||
Next: [→ 34 Codec and SML](34_codec_and_sml.md)
|
||||
@@ -0,0 +1,303 @@
|
||||
# 34 — Codec and SML
|
||||
|
||||
← [33 Transport](33_transport.md) | [Back to index](00_index.md) | Next: [35 State machines and dispatch](35_state_machines_and_dispatch.md) →
|
||||
|
||||
We covered the SECS-II encoding rules in chapter 10. This
|
||||
chapter is the **implementation walk** — the four files that make
|
||||
up `secsgem::secs2`, how the encoder/decoder are structured, why
|
||||
the variant-based `Item` works, and how the SML printer/parser
|
||||
fits in.
|
||||
|
||||
Four files, 733 lines total. The codec is the most-tested layer
|
||||
in the codebase.
|
||||
|
||||
---
|
||||
|
||||
## The four files
|
||||
|
||||
```
|
||||
include/secsgem/secs2/
|
||||
├── item.hpp (170 lines) Item variant + Format enum + factories.
|
||||
├── codec.hpp ( 30 lines) encode / decode declarations.
|
||||
├── message.hpp ( 52 lines) Message wrapper (header fields + body Item).
|
||||
└── sml.hpp ( 32 lines) to_sml / try_parse_sml declarations.
|
||||
|
||||
src/secs2/
|
||||
├── codec.cpp (229 lines) encode_into / decode_at implementations.
|
||||
└── sml.cpp (220 lines) SML printer + parser.
|
||||
```
|
||||
|
||||
`item.hpp` and `message.hpp` are header-only. `codec.cpp` and
|
||||
`sml.cpp` carry the heavy lifting.
|
||||
|
||||
---
|
||||
|
||||
## The `Item` variant
|
||||
|
||||
Already covered in chapter 10; quick recap of the storage:
|
||||
|
||||
```cpp
|
||||
// include/secsgem/secs2/item.hpp:85
|
||||
class Item {
|
||||
public:
|
||||
using List = std::vector<Item>;
|
||||
using Storage = std::variant<
|
||||
List, // List
|
||||
std::string, // ASCII, JIS-8
|
||||
std::vector<uint8_t>, // Binary, Boolean, U1
|
||||
std::vector<int8_t>, // I1
|
||||
std::vector<int16_t>, // I2
|
||||
...
|
||||
std::vector<float>, // F4
|
||||
std::vector<double>>; // F8
|
||||
|
||||
private:
|
||||
Format format_;
|
||||
Storage data_;
|
||||
};
|
||||
```
|
||||
|
||||
Eleven variant alternatives serving 16 SECS-II formats — some
|
||||
formats share storage (Binary/Boolean/U1 all use
|
||||
`std::vector<uint8_t>`, ASCII/JIS-8 share `std::string`, U2/C2
|
||||
share `std::vector<uint16_t>`). Disambiguation is via `format_`.
|
||||
|
||||
### Factories
|
||||
|
||||
The intended way to build an `Item` is the named factories:
|
||||
|
||||
```cpp
|
||||
Item::list({Item::ascii("Hi"), Item::u4(42)});
|
||||
Item::ascii("Hello, world");
|
||||
Item::u4(std::vector<uint32_t>{1, 2, 3});
|
||||
Item::u4(42); // scalar convenience overload
|
||||
Item::f4(1.0f);
|
||||
```
|
||||
|
||||
Each takes ownership of the storage (or constructs from a scalar
|
||||
overload). No exceptions; no validity checks; trusts the caller.
|
||||
|
||||
---
|
||||
|
||||
## `encode_into` — the recursive encoder
|
||||
|
||||
```cpp
|
||||
void encode_into(const Item& item, std::vector<uint8_t>& out);
|
||||
```
|
||||
|
||||
[`src/secs2/codec.cpp:71`](../src/secs2/codec.cpp). Two paths —
|
||||
List and not-List:
|
||||
|
||||
```cpp
|
||||
void encode_into(const Item& item, std::vector<uint8_t>& out) {
|
||||
const Format fmt = item.format();
|
||||
|
||||
if (fmt == Format::List) {
|
||||
const auto& children = item.as_list();
|
||||
write_header(out, fmt, children.size());
|
||||
for (const auto& child : children) encode_into(child, out);
|
||||
return;
|
||||
}
|
||||
|
||||
// Scalar/array: write_header(byte count), then bytes.
|
||||
switch (fmt) {
|
||||
case Format::ASCII: {
|
||||
const auto& s = item.as_ascii();
|
||||
write_header(out, fmt, s.size());
|
||||
out.insert(out.end(), s.begin(), s.end());
|
||||
return;
|
||||
}
|
||||
case Format::U4: {
|
||||
const auto& v = std::get<std::vector<uint32_t>>(item.storage());
|
||||
write_header(out, fmt, v.size() * 4);
|
||||
for (auto x : v) put_scalar_be(out, x);
|
||||
return;
|
||||
}
|
||||
// ... one case per format
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`write_header` picks the smallest length-byte-count and emits the
|
||||
format byte + length bytes. `put_scalar_be` is the
|
||||
templated big-endian writer using `std::bit_cast` for floats and
|
||||
`std::make_unsigned_t` for integers (chapter 10).
|
||||
|
||||
`encode(item)` is a thin wrapper:
|
||||
|
||||
```cpp
|
||||
std::vector<uint8_t> encode(const Item& item) {
|
||||
std::vector<uint8_t> out;
|
||||
encode_into(item, out);
|
||||
return out;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `decode_at` — the recursive decoder
|
||||
|
||||
```cpp
|
||||
Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos);
|
||||
```
|
||||
|
||||
Mirror image:
|
||||
|
||||
```cpp
|
||||
Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos) {
|
||||
// 1. Format byte + length bytes.
|
||||
if (pos >= len) throw CodecError("truncated");
|
||||
const uint8_t fb = data[pos++];
|
||||
const Format fmt = static_cast<Format>(fb >> 2);
|
||||
const int nlen = fb & 0x03;
|
||||
if (pos + nlen > len) throw CodecError("truncated length bytes");
|
||||
std::size_t length = 0;
|
||||
for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++];
|
||||
|
||||
// 2. List recursion.
|
||||
if (fmt == Format::List) {
|
||||
Item::List children;
|
||||
children.reserve(length);
|
||||
for (std::size_t i = 0; i < length; ++i)
|
||||
children.push_back(decode_at(data, len, pos));
|
||||
return Item::list(std::move(children));
|
||||
}
|
||||
|
||||
// 3. Scalar/array: dispatch on element size + signedness/floatness.
|
||||
if (pos + length > len) throw CodecError("truncated body");
|
||||
const uint8_t* body = data + pos;
|
||||
pos += length;
|
||||
switch (fmt) {
|
||||
case Format::ASCII: return Item::ascii(std::string((const char*)body, length));
|
||||
case Format::U4: return Item::u4(read_array<uint32_t>(body, length));
|
||||
// ... one case per format
|
||||
}
|
||||
throw CodecError("unknown format code");
|
||||
}
|
||||
|
||||
Item decode(const std::vector<uint8_t>& bytes) {
|
||||
std::size_t pos = 0;
|
||||
Item it = decode_at(bytes.data(), bytes.size(), pos);
|
||||
if (pos != bytes.size()) throw CodecError("trailing bytes");
|
||||
return it;
|
||||
}
|
||||
```
|
||||
|
||||
The `_at` variant is useful when an outer protocol carries a SECS-II
|
||||
item *embedded* in a larger frame — the caller passes the buffer
|
||||
and a position, and gets back the item plus the new position.
|
||||
|
||||
Bounds checks throw `CodecError` at every step — a CodecError on
|
||||
the receive side closes the connection (chapter 11's S9F7 path).
|
||||
|
||||
---
|
||||
|
||||
## The Message wrapper
|
||||
|
||||
```cpp
|
||||
// include/secsgem/secs2/message.hpp
|
||||
class Message {
|
||||
public:
|
||||
uint8_t stream() const;
|
||||
uint8_t function() const;
|
||||
bool w_bit() const;
|
||||
uint32_t system_bytes() const;
|
||||
const Item& body() const;
|
||||
std::vector<uint8_t> body_bytes() const; // encoded body
|
||||
};
|
||||
```
|
||||
|
||||
A `Message` is just a small struct: stream + function + W-bit +
|
||||
system_bytes + body Item. No encoder lives here — encoding is
|
||||
done by `secs2::encode(message.body())` when the transport layer
|
||||
serializes a frame. The Message exists so the Router can dispatch
|
||||
on `(stream, function)` without re-decoding bytes.
|
||||
|
||||
---
|
||||
|
||||
## SML — the human-readable form
|
||||
|
||||
`to_sml(item)` walks the Item recursively and emits SML:
|
||||
|
||||
```cpp
|
||||
// src/secs2/sml.cpp — sketch
|
||||
std::string to_sml(const Item& item) {
|
||||
switch (item.format()) {
|
||||
case Format::List: {
|
||||
std::string s = "<L[" + std::to_string(item.size()) + "]";
|
||||
for (const auto& child : item.as_list()) {
|
||||
s += ' ' + to_sml(child);
|
||||
}
|
||||
s += '>';
|
||||
return s;
|
||||
}
|
||||
case Format::ASCII: return "A \"" + escape(item.as_ascii()) + "\"";
|
||||
case Format::U4: {
|
||||
const auto& v = std::get<std::vector<uint32_t>>(item.storage());
|
||||
std::string s = "U4";
|
||||
if (v.size() > 1) s += "[" + std::to_string(v.size()) + "]";
|
||||
for (auto x : v) s += " " + std::to_string(x);
|
||||
return s;
|
||||
}
|
||||
// ... per format
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`try_parse_sml(text)` is the inverse — a hand-written recursive-
|
||||
descent parser that returns `std::optional<Item>`. Returns
|
||||
`nullopt` on any parse error (no exceptions; this is what
|
||||
libFuzzer feeds garbage into and expects it not to crash).
|
||||
|
||||
Tests:
|
||||
[`tests/test_sml.cpp`](../tests/test_sml.cpp) (10 cases — every
|
||||
format round-trips through `to_sml` → `try_parse_sml` → identical
|
||||
Item).
|
||||
|
||||
### Why SML doesn't round-trip *bytes*
|
||||
|
||||
A subtle point: `decode(encode(item))` round-trips exactly, but
|
||||
`try_parse_sml(to_sml(item))` *also* round-trips the Item — except
|
||||
encoding the round-tripped Item may produce **different bytes**
|
||||
than the original. Why?
|
||||
|
||||
- The original might use a 2-byte length encoding; the
|
||||
round-tripped Item is a fresh `Item` and the encoder will pick
|
||||
the smallest length encoding (1 byte).
|
||||
- SML doesn't preserve "which list-length encoding the encoder
|
||||
chose."
|
||||
|
||||
If you need bit-exact round-trip of *bytes*, use `decode(encode)`.
|
||||
For semantic round-trip of *values*, use SML.
|
||||
|
||||
---
|
||||
|
||||
## Testing — every layer in isolation
|
||||
|
||||
| Layer | Test file | Cases | Focus |
|
||||
|--------------|--------------------------------------|------:|--------------------------------------------------------|
|
||||
| Item factories | tests/test_secs2.cpp | 14 | Construction, equality, format dispatch. |
|
||||
| Codec | tests/test_e5_kat.cpp | 19 | Known-answer tests — bit-exact bytes per SEMI E5 §9. |
|
||||
| Codec | tests/test_secs2.cpp | (overlap) | encode/decode round-trip + truncation rejection. |
|
||||
| Identifier wildcards | tests/test_identifier_wildcards.cpp | 6 | U1/U2/U4/U8 leniency for ID fields. |
|
||||
| SML | tests/test_sml.cpp | 10 | to_sml + try_parse_sml round-trip. |
|
||||
| Catalog | tests/test_messages.cpp | 82 | Every named SxFy builder + parser round-trip. |
|
||||
| Random/structural | tests/test_fuzz.cpp | 8 | Random bytes, truncation, oversize lengths, nested. |
|
||||
| libFuzzer | apps/fuzz_secs2_decode.cpp | (CI) | 200 k+ random inputs per minute, ASan + UBSan clean. |
|
||||
| libFuzzer | apps/fuzz_sml_parse.cpp | (CI) | 1.4 M+ random SML strings per minute, ASan + UBSan. |
|
||||
|
||||
The codec alone has **139 test cases / 196+ assertions for E5
|
||||
KAT**. This is intentional: every other layer trusts the codec is
|
||||
correct. If it isn't, nothing above works.
|
||||
|
||||
---
|
||||
|
||||
## Where to go next
|
||||
|
||||
You've now seen the codec and SML implementation in detail. Next
|
||||
chapter covers the **dispatch** layer that sits between the
|
||||
transport (which delivers raw `Message`s) and the stores (which
|
||||
hold state): `gem::Router`, the state-machine wiring, and the
|
||||
generated builder/parser glue from the message catalog.
|
||||
|
||||
Next: [→ 35 State machines and dispatch](35_state_machines_and_dispatch.md)
|
||||
@@ -0,0 +1,305 @@
|
||||
# 35 — State machines and dispatch
|
||||
|
||||
← [34 Codec and SML](34_codec_and_sml.md) | [Back to index](00_index.md) | Next: [36 Persistence, validation, metrics](36_persistence_validation_metrics.md) →
|
||||
|
||||
We have transport (chapter 33) delivering `secs2::Message`s
|
||||
(chapter 34) into the application. We have stores (chapter 32)
|
||||
holding the data. This chapter is the **glue**: how the Router
|
||||
dispatches by `(stream, function)`, how state machines compose with
|
||||
stores, and how a Router handler typically looks.
|
||||
|
||||
---
|
||||
|
||||
## `gem::Router` — the dispatch table
|
||||
|
||||
[`include/secsgem/gem/router.hpp`](../include/secsgem/gem/router.hpp):
|
||||
|
||||
```cpp
|
||||
class Router {
|
||||
public:
|
||||
using Handler = std::function<std::optional<s2::Message>(const s2::Message&)>;
|
||||
|
||||
Router& on(uint8_t stream, uint8_t function, Handler h);
|
||||
Router& fallback(Handler h);
|
||||
|
||||
std::optional<s2::Message> dispatch(const s2::Message& msg) const;
|
||||
};
|
||||
```
|
||||
|
||||
A `std::map<{stream, function}, Handler>`. Each handler:
|
||||
|
||||
- Takes a `const s2::Message&` (the inbound primary).
|
||||
- Returns a `std::optional<s2::Message>` (the reply, or `nullopt`
|
||||
for fire-and-forget primaries).
|
||||
|
||||
That's it. No middleware, no decorator chain, no pre/post hooks.
|
||||
Just a typed dispatch.
|
||||
|
||||
### Registering handlers
|
||||
|
||||
```cpp
|
||||
// apps/secs_server.cpp — sketch
|
||||
auto router = std::make_shared<gem::Router>();
|
||||
|
||||
router->on(1, 1, [model](const auto& m) {
|
||||
// S1F1 Are You There — reply with model name + softrev.
|
||||
return messages::s1f2(model->device.mdln, model->device.softrev);
|
||||
});
|
||||
|
||||
router->on(1, 3, [model](const auto& m) {
|
||||
// S1F3 — read SVID values.
|
||||
auto svid_list = messages::parse_s1f3(m.body());
|
||||
return messages::s1f4(model->svids.values(svid_list.value_or({})));
|
||||
});
|
||||
|
||||
router->on(2, 41, [model](const auto& m) {
|
||||
// S2F41 Host Command.
|
||||
auto cmd = messages::parse_s2f41(m.body());
|
||||
auto ack = model->commands.dispatch(cmd->rcmd, cmd->params);
|
||||
return messages::s2f42(ack);
|
||||
});
|
||||
|
||||
// ...one per S/F pair. apps/secs_server.cpp registers ~50.
|
||||
```
|
||||
|
||||
The `examples/pvd_tool/main.cpp` §6 register 51 handlers in ~460
|
||||
lines. Each handler is a few lines: parse the body, mutate or read
|
||||
a store, build the reply.
|
||||
|
||||
### What happens for unhandled primaries
|
||||
|
||||
```cpp
|
||||
// router.hpp dispatch()
|
||||
auto it = handlers_.find({msg.stream, msg.function});
|
||||
if (it != handlers_.end()) return it->second(msg);
|
||||
if (fallback_) return fallback_(msg);
|
||||
if (msg.reply_expected) return s2::Message(msg.stream, 0, false); // SxF0 Abort
|
||||
return std::nullopt;
|
||||
```
|
||||
|
||||
Three cases:
|
||||
|
||||
1. **Registered**: handler runs.
|
||||
2. **Fallback installed**: fallback runs.
|
||||
3. **Neither**: if the message expects a reply, send `SxF0` (Abort)
|
||||
per E5 convention. Otherwise silently drop.
|
||||
|
||||
### S9 wiring
|
||||
|
||||
The transport layer (chapter 11) emits S9F3 / S9F5 for unhandled
|
||||
primaries. Router exposes the introspection:
|
||||
|
||||
```cpp
|
||||
bool has_handler(uint8_t stream, uint8_t function) const;
|
||||
bool has_handler_for_stream(uint8_t stream) const;
|
||||
```
|
||||
|
||||
And the wrapper that combines them with S9 emission:
|
||||
|
||||
```cpp
|
||||
template <typename EmitFn, typename HeaderProvider>
|
||||
std::optional<s2::Message> dispatch_with_s9(
|
||||
EmitFn emit_s9, HeaderProvider header_provider,
|
||||
const s2::Message& msg) const {
|
||||
if (!has_handler(msg.stream, msg.function)) {
|
||||
if (auto mhead = header_provider()) {
|
||||
const uint8_t f = has_handler_for_stream(msg.stream) ? 5 : 3;
|
||||
emit_s9(f, *mhead);
|
||||
}
|
||||
}
|
||||
return dispatch(msg);
|
||||
}
|
||||
```
|
||||
|
||||
Used by `Connection::on_data_message` → `Router::dispatch_with_s9`,
|
||||
which calls back into `Connection::emit_s9` for the actual S9
|
||||
emission. Tested by
|
||||
[`tests/test_s9_fallback.cpp`](../tests/test_s9_fallback.cpp) (2
|
||||
cases — unknown stream → S9F3, unknown function in known stream
|
||||
→ S9F5).
|
||||
|
||||
---
|
||||
|
||||
## How a typical handler looks end-to-end
|
||||
|
||||
`S2F41` Host Command is the most-touched message in production —
|
||||
worth tracing in full:
|
||||
|
||||
### 1. The codegen'd parser/builder
|
||||
|
||||
```cpp
|
||||
// build/generated/secsgem/gem/messages.hpp (auto-generated)
|
||||
struct RemoteCommand {
|
||||
std::string rcmd;
|
||||
std::vector<std::pair<std::string, secs2::Item>> params;
|
||||
};
|
||||
|
||||
inline std::optional<RemoteCommand> parse_s2f41(const secs2::Item& body) {
|
||||
// ...auto-generated body walker...
|
||||
}
|
||||
|
||||
inline secs2::Message s2f42(uint8_t hcack, /* per-param acks */) {
|
||||
// ...auto-generated builder...
|
||||
}
|
||||
```
|
||||
|
||||
### 2. The Router registration
|
||||
|
||||
```cpp
|
||||
// apps/secs_server.cpp
|
||||
router->on(2, 41, [model](const secs2::Message& m) {
|
||||
auto cmd = messages::parse_s2f41(m.body());
|
||||
if (!cmd) {
|
||||
// Body didn't parse — reply S2F42 with HCACK = 1 (invalid).
|
||||
return messages::s2f42(1, {});
|
||||
}
|
||||
auto outcome = model->commands.dispatch(cmd->rcmd, cmd->params);
|
||||
return messages::s2f42(static_cast<uint8_t>(outcome.ack), outcome.cpacks);
|
||||
});
|
||||
```
|
||||
|
||||
### 3. The store dispatch
|
||||
|
||||
```cpp
|
||||
// include/secsgem/gem/store/host_commands.hpp
|
||||
class HostCommandRegistry {
|
||||
public:
|
||||
CommandOutcome dispatch(const std::string& rcmd, const ParamList& params) {
|
||||
auto it = commands_.find(rcmd);
|
||||
if (it == commands_.end()) return {HostCmdAck::InvalidCommand, ...};
|
||||
const auto& cmd = it->second;
|
||||
// Apply configured side effects: emit_ceid, set_alarm, …
|
||||
for (auto ceid : cmd.emit_ceids) on_emit_ceid_(ceid);
|
||||
for (auto alid : cmd.set_alarms) alarm_registry_->set(alid);
|
||||
return {cmd.default_ack, ...};
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### 4. The side-effect dispatcher
|
||||
|
||||
Steps in `dispatch` like `on_emit_ceid_(ceid)` call back into
|
||||
the EAP:
|
||||
|
||||
```cpp
|
||||
// Set up at startup:
|
||||
model->commands.set_emit_ceid_handler([conn, model](uint32_t ceid) {
|
||||
if (!model->is_event_enabled(ceid)) return;
|
||||
auto reports = model->compose_reports_for(ceid);
|
||||
auto msg = build_s6f11(ceid, reports);
|
||||
conn->send_data(std::move(msg));
|
||||
});
|
||||
```
|
||||
|
||||
### 5. The wire
|
||||
|
||||
`conn->send_data(s6f11)` walks through `secs2::encode` →
|
||||
`hsms::Frame::encode` → `async_write` to the socket. Host sees
|
||||
the unsolicited S6F11.
|
||||
|
||||
---
|
||||
|
||||
## The state-machine pattern
|
||||
|
||||
Every state machine in the codebase follows the same shape. Pick
|
||||
`ControlStateMachine`:
|
||||
|
||||
```cpp
|
||||
// include/secsgem/gem/control_state.hpp
|
||||
class ControlStateMachine {
|
||||
public:
|
||||
ControlStateMachine(ControlTransitionTable table);
|
||||
|
||||
ControlState state() const;
|
||||
|
||||
// Apply an event; returns the transition row (if any).
|
||||
const ControlTransition* on_event(ControlEvent e);
|
||||
|
||||
// Observer for transitions.
|
||||
using StateChangeHandler =
|
||||
std::function<void(ControlState from, ControlState to, ControlEvent)>;
|
||||
void set_state_change_handler(StateChangeHandler h);
|
||||
};
|
||||
```
|
||||
|
||||
Three properties:
|
||||
|
||||
1. **Pure data table** (`ControlTransitionTable`) decides what
|
||||
transitions exist.
|
||||
2. **Pure FSM** (`ControlStateMachine`) applies events against the
|
||||
table, updates state, emits the change.
|
||||
3. **Observer pattern** — the EAP registers a change handler that
|
||||
does the wire-level work (fire CEID, emit S6F11, log to
|
||||
metrics).
|
||||
|
||||
This pattern repeats for:
|
||||
|
||||
- `ProcessJobStateMachine` (E40)
|
||||
- `ControlJobStateMachine` (E94)
|
||||
- `EptStateMachine` (E116)
|
||||
- `ExceptionStateMachine` (E5 §13)
|
||||
- `CarrierStateMachine` (E87 — actually composes 3 sub-FSMs)
|
||||
- `LoadPortStateMachine` (E87 — same)
|
||||
- `SubstrateStateMachine` (E90 — same)
|
||||
- `ModuleStateMachine` (E157)
|
||||
- `E84StateMachine` (E84)
|
||||
- `CommunicationStateMachine` (E30 §6.5)
|
||||
|
||||
Eleven FSMs. All follow the same shape. All testable in
|
||||
isolation without IO.
|
||||
|
||||
---
|
||||
|
||||
## How FSM transitions cascade into CEIDs
|
||||
|
||||
A common need: "when PJ-1 transitions to Processing, fire
|
||||
CEID=ProcessStarted, which fires an S6F11 with linked reports."
|
||||
|
||||
The wiring is set up at startup in the EAP:
|
||||
|
||||
```cpp
|
||||
// apps/secs_server.cpp — sketch
|
||||
model->process_jobs.set_state_change_handler(
|
||||
[conn, model](const std::string& pjid,
|
||||
ProcessJobState from, ProcessJobState to,
|
||||
ProcessJobEvent ev) {
|
||||
// Configured per-state CEID from data/equipment.yaml.
|
||||
auto ceid = ceid_for_pj_state(to);
|
||||
if (!ceid || !model->is_event_enabled(*ceid)) return;
|
||||
auto reports = model->compose_reports_for(*ceid);
|
||||
auto msg = build_s6f11(*ceid, reports);
|
||||
deliver_or_spool(*conn, *model, std::move(msg));
|
||||
});
|
||||
```
|
||||
|
||||
`deliver_or_spool` is the spool-aware send: if the connection isn't
|
||||
SELECTED (or the spool is in transmit-disabled mode), the message
|
||||
queues into `SpoolStore`; otherwise it goes straight to the wire.
|
||||
|
||||
So the chain for "PJ-1 starts processing":
|
||||
|
||||
```
|
||||
ProcessJobStore.apply(pjid, Start)
|
||||
→ ProcessJobStateMachine.on_event(Start)
|
||||
→ State: WaitingForStart → Processing
|
||||
→ on_change handler fires
|
||||
→ looks up CEID ProcessStarted
|
||||
→ composes reports
|
||||
→ builds S6F11 message
|
||||
→ deliver_or_spool → connection or SpoolStore
|
||||
→ on the wire (eventually)
|
||||
```
|
||||
|
||||
Every step is independently testable. Tests at the wire level:
|
||||
[`tests/test_wire_ceid_emission.cpp`](../tests/test_wire_ceid_emission.cpp)
|
||||
(6 cases — every cascade from store mutation to socket bytes).
|
||||
|
||||
---
|
||||
|
||||
## Where to go next
|
||||
|
||||
You've now seen everything that makes the runtime *work*. One
|
||||
chapter left in Part 3: the **operational** concerns — persistence,
|
||||
config validation, and metrics.
|
||||
|
||||
Next: [→ 36 Persistence, validation, metrics](36_persistence_validation_metrics.md)
|
||||
@@ -0,0 +1,293 @@
|
||||
# 36 — Persistence, validation, metrics
|
||||
|
||||
← [35 State machines and dispatch](35_state_machines_and_dispatch.md) | [Back to index](00_index.md) | Next: [40 Building, running, the demo](40_building_running_demo.md) →
|
||||
|
||||
Three operational concerns wrap up Part 3:
|
||||
|
||||
- **Persistence** — file-backed journals for the seven stores that
|
||||
survive equipment restarts.
|
||||
- **Validation** — the multi-error YAML validator behind
|
||||
`--validate-config`.
|
||||
- **Metrics** — the Prometheus exporter.
|
||||
|
||||
Each is a small slice of the codebase but load-bearing for
|
||||
production deployments.
|
||||
|
||||
---
|
||||
|
||||
## Persistence
|
||||
|
||||
### Which stores persist
|
||||
|
||||
Seven of the 21 stores have file-backed journals:
|
||||
|
||||
| Store | Survives equipment restart |
|
||||
|--------------------|------------------------------------------------------------|
|
||||
| `SpoolStore` | Queued messages waiting for host comm to come back. |
|
||||
| `ProcessJobStore` | All in-progress PJs and their state machines. |
|
||||
| `ControlJobStore` | All in-progress CJs. |
|
||||
| `ExceptionStore` | Posted exceptions and their recovery state. |
|
||||
| `CarrierStore` | Docked carriers + slot maps + access state. |
|
||||
| `LoadPortStore` | Per-port association + reservation. |
|
||||
| `SubstrateStore` | Per-substrate location + STS / SPS / ID status. |
|
||||
|
||||
The remaining 14 stores (SVIDs, ECIDs, CEIDs, alarm registry, …)
|
||||
don't persist — their state is reconstructed from the YAML or
|
||||
from real-time signals on restart. An ECID that the host had
|
||||
changed *would* be lost on restart unless the EAP writes it back
|
||||
to the YAML (E40-style `S2F15` is rare in production for
|
||||
exactly this reason).
|
||||
|
||||
### The per-record file pattern
|
||||
|
||||
Every persistent store uses the same shape:
|
||||
|
||||
```
|
||||
/var/lib/secsgem/<store>/
|
||||
├── PJ-001 # one file per record
|
||||
├── PJ-002
|
||||
├── PJ-003
|
||||
└── ...
|
||||
```
|
||||
|
||||
One file per record, named by ID. When the store is mutated, the
|
||||
file is rewritten atomically (write to `.tmp` + `rename`). When
|
||||
the record is removed, the file is `unlink`'d.
|
||||
|
||||
**This is partial-write safe.** If the equipment power-cycles
|
||||
mid-write of one record, the others are untouched. At startup,
|
||||
the store iterates the directory, reads each file, and replays
|
||||
into in-memory state. A file that fails to parse (corrupted or
|
||||
unfinished) is dropped with a log line.
|
||||
|
||||
### How a store enables persistence
|
||||
|
||||
```cpp
|
||||
// apps/secs_server.cpp — startup
|
||||
auto model = std::make_shared<gem::EquipmentDataModel>();
|
||||
|
||||
if (!spool_dir.empty()) {
|
||||
model->spool.enable_persistence(spool_dir);
|
||||
}
|
||||
if (!pj_dir.empty()) {
|
||||
model->process_jobs.enable_persistence(pj_dir);
|
||||
}
|
||||
// ... etc per store
|
||||
```
|
||||
|
||||
`enable_persistence(dir)`:
|
||||
|
||||
1. Creates `dir` if needed.
|
||||
2. Iterates files in `dir`.
|
||||
3. For each file, reads + parses + adds the record to the store.
|
||||
4. Sets up the on-disk journal for subsequent mutations.
|
||||
|
||||
The persistence is **opt-in per store**, configured via CLI flag in
|
||||
`apps/secs_server.cpp`. Some deployments want spool persistence
|
||||
but not job persistence (e.g., test rigs); the per-store toggle
|
||||
makes that easy.
|
||||
|
||||
### File format and versioning
|
||||
|
||||
Each record file is a small binary blob:
|
||||
|
||||
```
|
||||
magic: 4 bytes "SGv1" (store-specific magic; v1 = version 1)
|
||||
version: 4 bytes (uint32_t, big-endian) — schema version
|
||||
length: 4 bytes (uint32_t, big-endian) — payload length
|
||||
payload: N bytes — store-specific record encoding
|
||||
checksum: 4 bytes (CRC-32C over header + payload)
|
||||
```
|
||||
|
||||
**Schema versioning** is built in. Every store has a `kVersion`
|
||||
constant. When the store reads a file:
|
||||
|
||||
```cpp
|
||||
if (file_version > kVersion)
|
||||
drop the file (newer than us; can't read)
|
||||
if (file_version < kVersion)
|
||||
apply the upgrade path (v1 → v2 → v3 reader chain)
|
||||
if (file_version == kVersion)
|
||||
read directly
|
||||
```
|
||||
|
||||
Multi-version reads let a new equipment release process old
|
||||
on-disk records without manual migration. Tested by
|
||||
[`tests/test_persistence_upgrade.cpp`](../tests/test_persistence_upgrade.cpp)
|
||||
(7 cases — every store with persistence, write v1, restart at
|
||||
v2, verify replay).
|
||||
|
||||
### Tests
|
||||
|
||||
| Store | Test file | Cases |
|
||||
|---------------------|----------------------------------------------------|------:|
|
||||
| Spool | bundled into `tests/test_data_model.cpp` | — |
|
||||
| Process Jobs | `tests/test_job_persistence.cpp` (PJ + CJ together)| 7 |
|
||||
| Control Jobs | same | — |
|
||||
| Exception | `tests/test_exception_persistence.cpp` | 5 |
|
||||
| Carrier | `tests/test_carrier_persistence.cpp` | 6 |
|
||||
| Substrate | `tests/test_substrate_persistence.cpp` | 7 |
|
||||
| Upgrade path | `tests/test_persistence_upgrade.cpp` | 7 |
|
||||
|
||||
Each persistence test covers: write a record, restart, verify
|
||||
replayed; partial-write recovery (truncated file dropped); remove
|
||||
deletes the file; corrupted file is dropped without throwing.
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
### Why a separate validator
|
||||
|
||||
YAML loaders throw on first error. That's the right behaviour
|
||||
at process startup — fail fast — but it's frustrating for an
|
||||
operator with a fresh equipment.yaml that has three typos.
|
||||
|
||||
`--validate-config` is a separate CLI flag that:
|
||||
|
||||
1. Doesn't bind the port.
|
||||
2. Tries to load every YAML.
|
||||
3. Accumulates *every* issue (across files).
|
||||
4. Prints them all.
|
||||
5. Exits 0 or 1.
|
||||
|
||||
```bash
|
||||
secs_server --validate-config \
|
||||
--config data/equipment.yaml \
|
||||
--state-table data/control_state.yaml \
|
||||
--pj-state-table data/process_job_state.yaml \
|
||||
--cj-state-table data/control_job_state.yaml
|
||||
```
|
||||
|
||||
Typical output:
|
||||
|
||||
```
|
||||
data/equipment.yaml:42: SVID 5 references undefined enum 'ChamberStateEnum'
|
||||
data/equipment.yaml:78: alarm 3 has ALCD bit-7 cleared but alarm is declared 'active'
|
||||
data/control_state.yaml:11: transition from OnlineRemote on host_request_remote has no `to` or `ack` field
|
||||
data/equipment.yaml:104: host_command VENT references unknown CEID 999
|
||||
|
||||
4 error(s), 0 warning(s) across 4 files
|
||||
```
|
||||
|
||||
Then exit 1.
|
||||
|
||||
### How it's implemented
|
||||
|
||||
[`include/secsgem/config/validate.hpp`](../include/secsgem/config/validate.hpp):
|
||||
|
||||
```cpp
|
||||
class ConfigValidator {
|
||||
public:
|
||||
void validate_equipment(const std::string& path);
|
||||
void validate_control_state(const std::string& path);
|
||||
void validate_process_job_state(const std::string& path);
|
||||
void validate_control_job_state(const std::string& path);
|
||||
|
||||
std::size_t error_count() const;
|
||||
std::size_t warning_count() const;
|
||||
bool has_errors() const;
|
||||
|
||||
const std::vector<Issue>& issues() const;
|
||||
void format_issues_to(std::ostream&, FormatOptions = {}) const;
|
||||
};
|
||||
```
|
||||
|
||||
Each `validate_*` method:
|
||||
|
||||
1. Loads the YAML (catching parse errors as one issue).
|
||||
2. Walks every record, applying structural + referential checks.
|
||||
3. Adds each problem as an `Issue{path, line, severity, message}`.
|
||||
|
||||
Tests:
|
||||
[`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp)
|
||||
(8 cases — every category of issue: missing required field,
|
||||
typed mismatch, dangling reference, duplicate ID, …).
|
||||
|
||||
### Reference checks across files
|
||||
|
||||
Cross-file references are validated last (after all files are
|
||||
parsed). Examples:
|
||||
|
||||
- `host_commands[].emit_ceid` must reference a CEID defined in
|
||||
`equipment.yaml::ceids`.
|
||||
- `events.default_reports[].vids` must reference SVIDs or DVIDs
|
||||
defined elsewhere.
|
||||
- `control_state.yaml::transitions` `from`/`to` must reference
|
||||
states declared by the schema (the 5 standard control states).
|
||||
|
||||
This catches "I deleted the CEID but forgot to update the
|
||||
host_command" before runtime.
|
||||
|
||||
---
|
||||
|
||||
## Metrics
|
||||
|
||||
### What gets exported
|
||||
|
||||
The codebase ships a Prometheus exporter
|
||||
([`include/secsgem/metrics/prometheus.hpp`](../include/secsgem/metrics/prometheus.hpp))
|
||||
with two parts:
|
||||
|
||||
- **Registry** — accumulates `Counter` and `Gauge` series with
|
||||
labels.
|
||||
- **Server** — exposes them on a configurable HTTP port at
|
||||
`/metrics`.
|
||||
|
||||
Typical wiring:
|
||||
|
||||
```cpp
|
||||
auto registry = std::make_shared<metrics::Registry>();
|
||||
registry->register_metric("secsgem_ceid_emits_total", metrics::MetricType::Counter);
|
||||
registry->register_metric("secsgem_spool_depth", metrics::MetricType::Gauge);
|
||||
registry->register_metric("secsgem_pj_state", metrics::MetricType::Gauge);
|
||||
|
||||
// ...later, in the CEID-emit handler:
|
||||
registry->counter("secsgem_ceid_emits_total", {{"ceid", std::to_string(ceid)}}).inc();
|
||||
|
||||
// ...periodically:
|
||||
registry->gauge("secsgem_spool_depth").set(model->spool.size());
|
||||
|
||||
// Start the HTTP server:
|
||||
auto exporter = std::make_shared<metrics::PrometheusServer>(io, /*port=*/9090, registry);
|
||||
```
|
||||
|
||||
The exporter is wire-compatible with Prometheus scrape (text
|
||||
format). Tested by
|
||||
[`tests/test_metrics_prometheus.cpp`](../tests/test_metrics_prometheus.cpp)
|
||||
(3 cases — counter increment, gauge set, HTTP scrape format).
|
||||
|
||||
### What to expose
|
||||
|
||||
Common patterns from
|
||||
[`examples/pvd_tool/main.cpp`](../examples/pvd_tool/main.cpp) §7:
|
||||
|
||||
- Per-CEID counters (`secsgem_ceid_emits_total{ceid="300"}`).
|
||||
- Per-alarm counters (`secsgem_alarm_set_total{alid="42"}`).
|
||||
- Spool depth gauge (alarm in operations if it climbs).
|
||||
- Per-state EPT durations (sample of E116 buckets).
|
||||
- T3 timeout counter (alarm in operations if non-zero).
|
||||
|
||||
The exporter doesn't dictate which metrics to expose — the EAP
|
||||
decides. See
|
||||
[`docs/INTEGRATION.md`](INTEGRATION.md) §6.4 for the production
|
||||
patterns.
|
||||
|
||||
---
|
||||
|
||||
## End of Part 3
|
||||
|
||||
You now know every layer of the runtime:
|
||||
|
||||
- The repository layout (chapter 30).
|
||||
- The spec-as-data philosophy + codegen (chapter 31).
|
||||
- The stores + data model (chapter 32).
|
||||
- The transport implementation (chapter 33).
|
||||
- The codec + SML (chapter 34).
|
||||
- Router + state machines + dispatch (chapter 35).
|
||||
- Persistence + validation + metrics (this chapter).
|
||||
|
||||
Part 4 turns to operations — how a customer actually builds, runs,
|
||||
deploys, and integrates this codebase into a real fab tool.
|
||||
|
||||
Next: [→ 40 Building, running, the demo](40_building_running_demo.md)
|
||||
Reference in New Issue
Block a user