From 91eec92b73bb6afcbe2570f68991d0e4fc97dca5 Mon Sep 17 00:00:00 2001 From: Raphael Maenle Date: Tue, 9 Jun 2026 16:46:07 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20ARCHITECTURE.md=20=E2=80=94=20how=20the?= =?UTF-8?q?=20codebase=20fits=20+=20how=20to=20extend?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Customers who want to extend the library had two paths: read the 1200-line apps/secs_server.cpp and guess at conventions, or read every store header and infer the shape. Neither is reasonable. ARCHITECTURE.md walks the five layers (apps → Router+Model → stores → FSMs → transport+codec) with a worked extension recipe per layer: - New SECS-II message (YAML edit + Router handler — no core code) - New state machine (lift from ept_state.hpp or process_job_state.hpp) - New store (paste-and-adapt from alarms.hpp or process_jobs.hpp) - New persistence backend (mirror enable_persistence pattern) - New transport (mirror Connection's contract) Explains the design choices that look unusual: - Spec-as-data — every behavioural rule in YAML, C++ is the engine - I/O-free FSMs — transport classes own asio, everything else is pure - Single-threaded by design + no locks anywhere - No DI framework, no singletons, no shared_ptr-everywhere - Exceptions only for programmer-error / corrupt-input paths Documents the persistence magic-byte registry (0xC4-0xC9 + 0xE5) so the next contributor doesn't collide; documents the codegen pipeline (messages.yaml → gen_messages.py → messages.hpp); maps "you want to understand X" → "read these files in order" for the twelve most common entry points. Doc map in README already points at ARCHITECTURE.md from the prior commit. Co-Authored-By: Claude Opus 4.7 --- ARCHITECTURE.md | 502 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 502 insertions(+) create mode 100644 ARCHITECTURE.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..02216ce --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,502 @@ +# Architecture + +How the codebase is put together, and how to extend it. Read after +[INTEGRATION.md](INTEGRATION.md) — that doc tells you what to do; +this one tells you *why*, and where to plug in new behaviour. + +--- + +## 1. Design principle: spec-as-data + +The SEMI standards describe behaviour as **tables** — state machines, +message catalogues, transition rules. C++ is the wrong language to +write those tables in directly: every spec edit becomes a recompile, +and reviewers can't audit "does the implementation match E40 §6.3" +without reading code. + +So the rule across the project is: **anything the SEMI spec encodes +as a table lives in YAML.** The C++ is the engine that reads them. + +``` + data/messages.yaml → tools/gen_messages.py → messages.hpp + data/control_state.yaml → config::load_control_state() + data/process_job_state.yaml → config::load_process_job_state() + data/control_job_state.yaml → config::load_control_job_state() + data/equipment.yaml → config::load_equipment() +``` + +Two consequences worth absorbing: + +- **Adding a new SECS-II message rarely requires C++.** Edit + `data/messages.yaml`, rebuild, register a handler with the Router. +- **Adding a new state transition rarely requires C++.** Edit the + relevant state YAML; the loader hot-loads on next start. + +Things that do require C++: new *kinds* of behaviour (new FSM, new +store, new persistence backend) — and that's what the rest of this +doc covers. + +--- + +## 2. The five layers + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ apps/ (your main.cpp lives here) │ +│ secs_server, secs_client, secs_conformance, secs_bench, │ +│ fuzz_*, secs_interop_probe │ +├─────────────────────────────────────────────────────────────────┤ +│ gem::Router + gem::EquipmentDataModel │ +│ ───────────────────────────────────────── │ +│ Router: (stream, function) → handler dispatch table │ +│ Model: composes every store + every FSM into one object │ +├─────────────────────────────────────────────────────────────────┤ +│ Per-domain stores (include/secsgem/gem/store/) │ +│ alarms, carriers, ceid+reports, exceptions, host_commands, │ +│ limits, modules, process_jobs, control_jobs, recipes, spool, │ +│ substrates, svid+dvid, trace, cem_objects, e84_ports, clock │ +├─────────────────────────────────────────────────────────────────┤ +│ Per-standard state machines │ +│ E30 control_state, E30 communication_state, E40 PJ, │ +│ E94 CJ, E87 carriers + load_ports, E90 substrates, │ +│ E116 EPT, E157 modules, E5 exceptions, E84 handshake │ +├─────────────────────────────────────────────────────────────────┤ +│ hsms::Connection (Asio) + secsi::Protocol + secs2 codec │ +│ ─────────────────────────────────────────────────────── │ +│ Transport: HSMS-SS, HSMS-GS, SECS-I (FSM only) │ +│ Codec: Item ⇄ bytes, Item ⇄ SML text │ +└─────────────────────────────────────────────────────────────────┘ +``` + +Each layer is replaceable. The codec doesn't know about the FSMs; +the FSMs don't know about the codec; the Router doesn't know about +persistence. The model composes them but doesn't own their logic. + +--- + +## 3. The codec (`secs2/`) + +`secs2::Item` is a tagged variant over the SEMI E5 §9 formats: List, +Binary, Boolean, ASCII, JIS-8, C2, U1-U8, I1-I8, F4, F8. Storage is +a `std::variant` matching each format's natural C++ type. + +``` +secs2::encode(item) → vector // bytes for the wire +secs2::decode(bytes) → Item // wire → object +secs2::to_sml(item) → string // human-readable +secs2::from_sml(text) → Item // and back +``` + +The encoder emits the format-byte arithmetic described in +[GLOSSARY.md → SEMI E5 §9](GLOSSARY.md). The decoder is strict +about format codes but lenient about U-widths in identifier fields +(per `messages_helpers::any_unsigned_first`) — that's how secsgem-py +interop works without breaking spec-correctness. + +The codec is the most-tested layer in the codebase: 196 SEMI E5 KAT +assertions, 120+ unit tests, plus libFuzzer with 70 000+ random +inputs per minute. Touch it carefully; it's the foundation +everything else stands on. + +## 4. Transport (`hsms/`, `secsi/`) + +`hsms::Connection` owns one TCP socket and one (SS) or many (GS) +session-state objects. Frames have a 4-byte length prefix + 10-byte +header (session_id, byte2, byte3, PType, SType, system_bytes) + +optional SECS-II body. + +State transitions: NOT-CONNECTED → NOT-SELECTED (T7 armed) → SELECTED. +Either side can initiate Select.req; both modes (Active / Passive) +are first-class. + +The connection class is **I/O-aware**: it owns the asio socket, arms +the T-timers, drives the read loop. Everything above it is I/O-free +and reachable through callbacks: + +```cpp +conn->set_message_handler([&router](const secs2::Message& m) { + return router.dispatch_with_s9(/*emit=*/..., /*mhead=*/..., m); +}); +``` + +SECS-I (`secsi::Protocol`) is an FSM-only port of the same idea — +serial-line framing, T1/T2/T3/T4 timers as callbacks. No asio +inside the FSM; the application drives the clock. The E84 timers +follow the same pattern (`E84AsioTimers` is the asio adapter; the +FSM stays pure). + +## 5. The model (`gem/`) + +`gem::EquipmentDataModel` (data_model.hpp) is a struct composing +every store: + +```cpp +struct EquipmentDataModel { + StatusVariableStore svids; + DataVariableStore dvids; + EquipmentConstantStore ecids; + EventReportSubscriptions events; + AlarmRegistry alarms; + RecipeStore recipes; + Clock clock; + HostCommandRegistry commands; + SpoolStore spool; + LimitMonitorStore limits; + TraceStore traces; + ProcessJobStore process_jobs; + ControlJobStore control_jobs; + ExceptionStore exceptions; + CarrierStore carriers; + LoadPortStore load_ports; + SubstrateStore substrates; + EptStateMachine ept; + CemObjectStore cem; + ModuleStore modules; + E84PortStore e84_ports; +}; +``` + +No locks. Single-threaded contract documented in INTEGRATION.md §3. +All mutation runs on the io_context strand. + +Each store is **independently usable** — you can `#include +"secsgem/gem/store/alarms.hpp"` and use `AlarmRegistry` without +pulling in any of the others. The composite is for convenience. + +### Per-store pattern + +Every store follows the same shape: + +```cpp +class FooStore { + public: + // CRUD + bool create(...); + Foo* get(id); // mutable pointer, nullable + const Foo* get(id) const; // const-mutable pointer, nullable + bool has(id) const; + bool remove(id); + std::size_t size() const; + std::vector all() const; + + // Domain operations + void fire_internal(id, FooEvent event); // application-driven + Ack on_host_command(id, FooEvent event); // host-driven + + // Observers + void set_state_change_handler(StateChangeHandler); + + // Persistence + void enable_persistence(std::filesystem::path dir); +}; +``` + +The store owns the FSM instance, the persistence file path, the +in-memory state. The FSM owns the legal-transition table. The +table comes from a YAML file (loaded into `factory_()` at +construction). + +## 6. The Router + +`gem::Router` (router.hpp) is a tiny dispatch table: + +```cpp +Router r; +r.on(1, 13, [&](const secs2::Message&) { + return gem::s1f14_establish_comms_ack(...); +}); +r.on(2, 41, [&](const secs2::Message& msg) { + auto cmd = gem::parse_s2f41(msg); + // ... handle command ... + return gem::s2f42_host_command_ack(...); +}); + +auto reply = r.dispatch(incoming_message); +``` + +Handlers are `std::function(const Message&)>`. +Return nullopt for one-way (W=0) primaries. + +`dispatch_with_s9` wraps `dispatch` to auto-emit `S9F3` (unrecognized +stream) or `S9F5` (unrecognized function) when no handler is +registered — the spec-mandated response. + +The Router is **stateless** — it just looks up handlers in a +`std::map, Handler>`. All state lives +in the model the handlers close over. + +## 7. Persistence + +Every persistable store ships a `.tmp + atomic rename` writer + a +versioned record format: + +``` +[u8 magic] +[u8 version] // 1..kVersion accepted on load +[u8 state] +... domain-specific fields ... +``` + +`enable_persistence(dir)` scans the dir on startup, replays records +into in-memory state via `install_()`, and from there writes on every +mutation. See README "Schema migrations" for the version-bump +discipline. + +The seven persistable stores (PJ, CJ, Carrier, LoadPort, Substrate, +Exception, Spool) all follow the same pattern. Adding persistence to +a new store is a paste-and-adapt: copy `control_jobs.hpp`'s +`write_record_` + `load_record_` + `enable_persistence`, change the +magic byte + the fields. + +Magic bytes claimed so far (don't reuse): + +| Magic | Store | +|-------|--------------------| +| 0xC4 | CarrierStore | +| 0xC5 | LoadPortStore | +| 0xC6 | SubstrateStore | +| 0xC7 | ProcessJobStore | +| 0xC8 | ControlJobStore | +| 0xC9 | ExceptionStore | +| 0xE5 | SpoolStore | + +--- + +## 8. Codegen pipeline + +`tools/gen_messages.py` reads `data/messages.yaml` and emits +`build/generated/secsgem/gem/messages.hpp`. The pipeline: + +``` +messages.yaml + │ + │ (CMake add_custom_command, runs on rebuild if YAML newer) + ▼ +tools/gen_messages.py + │ + │ (Python reads YAML, emits typed C++ structs + builders + parsers) + ▼ +build/generated/secsgem/gem/messages.hpp + │ + │ (#included by apps/, src/, tests/) + ▼ +secs_server.cpp / secs_client.cpp / your main.cpp +``` + +For each message in the catalog the codegen emits: + +- An optional `struct Name { ... }` (for list bodies) +- A `inline secs2::Message builder_name(args...)` that returns a + ready-to-send Message +- A `inline std::optional<...> parse_name(const secs2::Message&)` + that returns the parsed body or nullopt + +The YAML shape is documented in the file header of `messages.yaml`. +Every supported body kind (`scalar`, `list`, `list_of`) maps to a +straightforward C++ shape. + +--- + +## 9. Extending the library + +### 9.1. New SECS-II message + +Edit `data/messages.yaml`: + +```yaml +- id: S6F30 + stream: 6 + function: 30 + w: true + builder: s6f30_my_request + parser: parse_s6f30 + body: + kind: list + struct_name: MyRequest + fields: + - {name: dataid, shape: {kind: scalar, item_type: U4}} + - {name: payload, shape: {kind: scalar, item_type: ASCII}} +``` + +Rebuild — `messages.hpp` regenerates. Register a handler: + +```cpp +router.on(6, 30, [&](const secs2::Message& m) { + auto req = gem::parse_s6f30(m); + if (!req) return std::optional{secs2::Message(6, 0, false)}; // bad body + // ... + return std::optional{secs2::Message(6, 0, false)}; // W=0 reply +}); +``` + +That's the entire diff. No core code change. + +### 9.2. New state machine + +If your tool has a domain not covered by the existing stores +(say, an in-chamber gas-flow FSM): + +1. Define the states + events: + ```cpp + // include/secsgem/gem/gas_flow.hpp + enum class GasFlowState : uint8_t { Idle, Purging, Stable, Faulted }; + enum class GasFlowEvent : uint8_t { StartPurge, FlowStable, Fault, Reset }; + ``` +2. Define the transition table — pure data: + ```cpp + struct GasFlowTransition { GasFlowState from; GasFlowEvent on; std::optional to; }; + class GasFlowTransitionTable { /* mirrors ProcessJobTransitionTable */ }; + ``` +3. Define the FSM: + ```cpp + class GasFlowStateMachine { + public: + bool fire(GasFlowEvent ev); // returns whether a transition happened + GasFlowState state() const; + void set_state_change_handler(StateChangeHandler); + }; + ``` +4. (Optional) Define a store if there can be many instances: + `class GasFlowStore { /* mirrors ProcessJobStore */ }` with + create/get/has/all + state-change relay. +5. (Optional) YAML-load the transitions following `config::load_*` patterns. +6. (Optional) Persistence: copy a store's `enable_persistence` + `write_record_` + `load_record_`. + +Reference patterns to lift from: `ept_state.hpp` (single global FSM), +`process_job_state.hpp` (per-instance FSM in a store). + +### 9.3. New store + +Stores follow the consistent API shape in §5. Copy +`include/secsgem/gem/store/alarms.hpp` (smallest example) or +`include/secsgem/gem/store/process_jobs.hpp` (richest example, +includes persistence). + +Wire into `EquipmentDataModel` if it should be globally accessible +from `model->...`: + +```cpp +// data_model.hpp +#include "secsgem/gem/store/gas_flows.hpp" + +struct EquipmentDataModel { + // ... existing fields ... + GasFlowStore gas_flows; +}; +``` + +### 9.4. New persistence backend + +The seven existing stores all journal to files. If you want +database-backed persistence (SQLite, Postgres, etcd), the cleanest +pattern is to subclass-or-replace the `enable_persistence(path)` +method: + +```cpp +// Or: a sibling enable_db_persistence(connection_string) +void enable_db_persistence(std::string conn) { + db_conn_ = std::move(conn); + /* on each create / mutation, write the record to the DB */ +} +``` + +The contract is consistent with file persistence: load at startup, +write on mutation, atomic-rename equivalent (a transaction). See +`spool.hpp::enable_persistence` for the cleanest single-file +example to mirror. + +### 9.5. New transport + +`hsms::Connection` and `secsi::Protocol` are the two we ship. A +third (e.g. HSMS-over-TLS as a first-class thing, or HSMS over a +sidecar IPC) follows the same contract: + +1. Accept a transport socket / endpoint. +2. Expose `set_message_handler(...)`, `send_request(...)`, + `send_data(...)`, `set_selected_handler(...)`, + `set_closed_handler(...)`. +3. Drive the SECS-II codec via `secs2::encode` / `secs2::decode`. + +The Router and the model don't care which transport produced the +message. Both wire into the same `set_message_handler` callback +shape. + +--- + +## 10. Threading model + +Single-threaded by design. The entire model — every store, every +FSM, the Router, the Connection — is reachable only from the +io_context that drives the HSMS connection. No locks anywhere. + +This is documented as a contract in INTEGRATION.md §3 and exercised +by: + +- `test_thread_safety.cpp` — N producer threads asio::post updates + onto the worker io +- `test_concurrency.cpp` — in-flight transaction interleaving +- The ThreadSanitizer CI lane — every test under + `-fsanitize=thread` + +If you're adding work that lives on another thread (sensor poll +loop, separate metrics scraper, signal handler), marshal back to +the io_context with `asio::post(io.get_executor(), ...)`. Don't +add locks; they'll diverge from the contract and the next +contributor will be confused. + +--- + +## 11. Why C++20 + +- `std::variant` for `Item` storage — exhaustive `std::visit` + catches new format codes at compile time. +- `std::optional` everywhere — the codec, the parsers, the + store accessors all use it as the "missing value" idiom. +- Designated initializers in tests — readability. +- Concepts in template helpers (`messages_helpers.hpp`). +- `` — persistence wouldn't be a header-only feature + without it. + +`g++-13` and `clang-18` both build the codebase clean at +`-Wall -Wextra -Wpedantic`. + +--- + +## 12. Where to look in the source + +| You want to understand… | Read these in order | +|-------------------------------------|----------------------------------------------------------------| +| The wire byte layout | `secs2/item.hpp`, `secs2/codec.cpp`, `tests/test_e5_kat.cpp` | +| How a typed message is built | `data/messages.yaml`, `tools/gen_messages.py`, the generated header | +| How HSMS handshakes | `hsms/connection.hpp/.cpp`, `tests/test_hsms_*.cpp` | +| How the Router dispatches | `gem/router.hpp` | +| How a store implements persistence | `gem/store/spool.hpp` (smallest), `gem/store/process_jobs.hpp` (richest) | +| How an FSM is structured | `gem/process_job_state.hpp`, `src/gem/process_job_state.cpp` | +| How the application wires it all | `apps/secs_server.cpp` (the canonical example, ~1200 lines) | +| How a customer would write main() | `examples/pvd_tool/main.cpp` (the worked vendor example) | +| How thread-safety works | `tests/test_thread_safety.cpp`, INTEGRATION.md §3 | +| How E84 timers integrate with asio | `gem/e84_asio_timers.hpp` (the canonical I/O-adapter pattern) | +| How the property fuzz drives state | `tests/test_robustness_fuzz.cpp` | + +--- + +## 13. What we deliberately don't do + +- **No DI framework, no service locator.** Stores are owned by the + model; the model is owned by your application; everything else is + passed in by reference. C++20 has no language-level DI, and adding + one to a codebase this size is overhead with no payoff. +- **No singleton state.** The model is a value, not a global. +- **No std::shared_ptr-everywhere.** asio handlers extend the + lifetimes that need extending; the rest is owned by-value. Read + `Connection`'s lifetime contract in `hsms/connection.hpp` if you're + ever in doubt. +- **No exceptions across the API boundary** — the codec throws + `secs2::CodecError` internally, but every public accessor returns + `std::optional` or returns a bool. Exceptions are reserved for + programmer-error / corrupt-input paths. + +Every one of those constraints came from real review pressure on +prior iterations. Pushing back on them is welcome but please read +the existing tests first; the codebase's architecture is what makes +the property fuzz and the TSan lane feasible.