docs: ARCHITECTURE.md — how the codebase fits + how to extend

Customers who want to extend the library had two paths: read the 1200-line apps/secs_server.cpp and guess at conventions, or read every store header and infer the shape. Neither is reasonable. ARCHITECTURE.md walks the five layers (apps → Router+Model → stores → FSMs → transport+codec) with a worked extension recipe per layer: - New SECS-II message (YAML edit + Router handler — no core code) - New state machine (lift from ept_state.hpp or process_job_state.hpp) - New store (paste-and-adapt from alarms.hpp or process_jobs.hpp) - New persistence backend (mirror enable_persistence pattern) - New transport (mirror Connection's contract) Explains the design choices that look unusual: - Spec-as-data — every behavioural rule in YAML, C++ is the engine - I/O-free FSMs — transport classes own asio, everything else is pure - Single-threaded by design + no locks anywhere - No DI framework, no singletons, no shared_ptr-everywhere - Exceptions only for programmer-error / corrupt-input paths Documents the persistence magic-byte registry (0xC4-0xC9 + 0xE5) so the next contributor doesn't collide; documents the codegen pipeline (messages.yaml → gen_messages.py → messages.hpp); maps "you want to understand X" → "read these files in order" for the twelve most common entry points. Doc map in README already points at ARCHITECTURE.md from the prior commit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-09 16:46:07 +02:00
parent 195ecc689f
commit 91eec92b73
1 changed files with 502 additions and 0 deletions
@@ -0,0 +1,502 @@
+# Architecture
+
+How the codebase is put together, and how to extend it.  Read after
+[INTEGRATION.md](INTEGRATION.md) — that doc tells you what to do;
+this one tells you *why*, and where to plug in new behaviour.
+
+---
+
+## 1.  Design principle: spec-as-data
+
+The SEMI standards describe behaviour as **tables** — state machines,
+message catalogues, transition rules.  C++ is the wrong language to
+write those tables in directly: every spec edit becomes a recompile,
+and reviewers can't audit "does the implementation match E40 §6.3"
+without reading code.
+
+So the rule across the project is: **anything the SEMI spec encodes
+as a table lives in YAML.**  The C++ is the engine that reads them.
+
+```
+   data/messages.yaml          → tools/gen_messages.py → messages.hpp
+   data/control_state.yaml     → config::load_control_state()
+   data/process_job_state.yaml → config::load_process_job_state()
+   data/control_job_state.yaml → config::load_control_job_state()
+   data/equipment.yaml         → config::load_equipment()
+```
+
+Two consequences worth absorbing:
+
+- **Adding a new SECS-II message rarely requires C++.**  Edit
+  `data/messages.yaml`, rebuild, register a handler with the Router.
+- **Adding a new state transition rarely requires C++.**  Edit the
+  relevant state YAML; the loader hot-loads on next start.
+
+Things that do require C++: new *kinds* of behaviour (new FSM, new
+store, new persistence backend) — and that's what the rest of this
+doc covers.
+
+---
+
+## 2.  The five layers
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│  apps/  (your main.cpp lives here)                              │
+│  secs_server, secs_client, secs_conformance, secs_bench,        │
+│  fuzz_*, secs_interop_probe                                     │
+├─────────────────────────────────────────────────────────────────┤
+│  gem::Router    +    gem::EquipmentDataModel                    │
+│  ─────────────────────────────────────────                      │
+│  Router: (stream, function) → handler dispatch table             │
+│  Model:  composes every store + every FSM into one object       │
+├─────────────────────────────────────────────────────────────────┤
+│  Per-domain stores (include/secsgem/gem/store/)                 │
+│  alarms, carriers, ceid+reports, exceptions, host_commands,     │
+│  limits, modules, process_jobs, control_jobs, recipes, spool,   │
+│  substrates, svid+dvid, trace, cem_objects, e84_ports, clock    │
+├─────────────────────────────────────────────────────────────────┤
+│  Per-standard state machines                                    │
+│  E30 control_state, E30 communication_state, E40 PJ,            │
+│  E94 CJ, E87 carriers + load_ports, E90 substrates,             │
+│  E116 EPT, E157 modules, E5 exceptions, E84 handshake           │
+├─────────────────────────────────────────────────────────────────┤
+│  hsms::Connection (Asio)  +  secsi::Protocol  +  secs2 codec    │
+│  ───────────────────────────────────────────────────────        │
+│  Transport: HSMS-SS, HSMS-GS, SECS-I (FSM only)                 │
+│  Codec:     Item ⇄ bytes, Item ⇄ SML text                       │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+Each layer is replaceable.  The codec doesn't know about the FSMs;
+the FSMs don't know about the codec; the Router doesn't know about
+persistence.  The model composes them but doesn't own their logic.
+
+---
+
+## 3.  The codec (`secs2/`)
+
+`secs2::Item` is a tagged variant over the SEMI E5 §9 formats: List,
+Binary, Boolean, ASCII, JIS-8, C2, U1-U8, I1-I8, F4, F8.  Storage is
+a `std::variant` matching each format's natural C++ type.
+
+```
+secs2::encode(item) → vector<uint8_t>          // bytes for the wire
+secs2::decode(bytes) → Item                    // wire → object
+secs2::to_sml(item) → string                   // human-readable
+secs2::from_sml(text) → Item                   // and back
+```
+
+The encoder emits the format-byte arithmetic described in
+[GLOSSARY.md → SEMI E5 §9](GLOSSARY.md).  The decoder is strict
+about format codes but lenient about U-widths in identifier fields
+(per `messages_helpers::any_unsigned_first`) — that's how secsgem-py
+interop works without breaking spec-correctness.
+
+The codec is the most-tested layer in the codebase: 196 SEMI E5 KAT
+assertions, 120+ unit tests, plus libFuzzer with 70 000+ random
+inputs per minute.  Touch it carefully; it's the foundation
+everything else stands on.
+
+## 4.  Transport (`hsms/`, `secsi/`)
+
+`hsms::Connection` owns one TCP socket and one (SS) or many (GS)
+session-state objects.  Frames have a 4-byte length prefix + 10-byte
+header (session_id, byte2, byte3, PType, SType, system_bytes) +
+optional SECS-II body.
+
+State transitions: NOT-CONNECTED → NOT-SELECTED (T7 armed) → SELECTED.
+Either side can initiate Select.req; both modes (Active / Passive)
+are first-class.
+
+The connection class is **I/O-aware**: it owns the asio socket, arms
+the T-timers, drives the read loop.  Everything above it is I/O-free
+and reachable through callbacks:
+
+```cpp
+conn->set_message_handler([&router](const secs2::Message& m) {
+  return router.dispatch_with_s9(/*emit=*/..., /*mhead=*/..., m);
+});
+```
+
+SECS-I (`secsi::Protocol`) is an FSM-only port of the same idea —
+serial-line framing, T1/T2/T3/T4 timers as callbacks.  No asio
+inside the FSM; the application drives the clock.  The E84 timers
+follow the same pattern (`E84AsioTimers` is the asio adapter; the
+FSM stays pure).
+
+## 5.  The model (`gem/`)
+
+`gem::EquipmentDataModel` (data_model.hpp) is a struct composing
+every store:
+
+```cpp
+struct EquipmentDataModel {
+  StatusVariableStore       svids;
+  DataVariableStore         dvids;
+  EquipmentConstantStore    ecids;
+  EventReportSubscriptions  events;
+  AlarmRegistry             alarms;
+  RecipeStore               recipes;
+  Clock                     clock;
+  HostCommandRegistry       commands;
+  SpoolStore                spool;
+  LimitMonitorStore         limits;
+  TraceStore                traces;
+  ProcessJobStore           process_jobs;
+  ControlJobStore           control_jobs;
+  ExceptionStore            exceptions;
+  CarrierStore              carriers;
+  LoadPortStore             load_ports;
+  SubstrateStore            substrates;
+  EptStateMachine           ept;
+  CemObjectStore            cem;
+  ModuleStore               modules;
+  E84PortStore              e84_ports;
+};
+```
+
+No locks.  Single-threaded contract documented in INTEGRATION.md §3.
+All mutation runs on the io_context strand.
+
+Each store is **independently usable** — you can `#include
+"secsgem/gem/store/alarms.hpp"` and use `AlarmRegistry` without
+pulling in any of the others.  The composite is for convenience.
+
+### Per-store pattern
+
+Every store follows the same shape:
+
+```cpp
+class FooStore {
+ public:
+  // CRUD
+  bool create(...);
+  Foo* get(id);              // mutable pointer, nullable
+  const Foo* get(id) const;  // const-mutable pointer, nullable
+  bool has(id) const;
+  bool remove(id);
+  std::size_t size() const;
+  std::vector<Foo> all() const;
+
+  // Domain operations
+  void fire_internal(id, FooEvent event);   // application-driven
+  Ack on_host_command(id, FooEvent event);  // host-driven
+
+  // Observers
+  void set_state_change_handler(StateChangeHandler);
+
+  // Persistence
+  void enable_persistence(std::filesystem::path dir);
+};
+```
+
+The store owns the FSM instance, the persistence file path, the
+in-memory state.  The FSM owns the legal-transition table.  The
+table comes from a YAML file (loaded into `factory_()` at
+construction).
+
+## 6.  The Router
+
+`gem::Router` (router.hpp) is a tiny dispatch table:
+
+```cpp
+Router r;
+r.on(1, 13, [&](const secs2::Message&) {
+  return gem::s1f14_establish_comms_ack(...);
+});
+r.on(2, 41, [&](const secs2::Message& msg) {
+  auto cmd = gem::parse_s2f41(msg);
+  // ... handle command ...
+  return gem::s2f42_host_command_ack(...);
+});
+
+auto reply = r.dispatch(incoming_message);
+```
+
+Handlers are `std::function<std::optional<Message>(const Message&)>`.
+Return nullopt for one-way (W=0) primaries.
+
+`dispatch_with_s9` wraps `dispatch` to auto-emit `S9F3` (unrecognized
+stream) or `S9F5` (unrecognized function) when no handler is
+registered — the spec-mandated response.
+
+The Router is **stateless** — it just looks up handlers in a
+`std::map<std::pair<uint8_t, uint8_t>, Handler>`.  All state lives
+in the model the handlers close over.
+
+## 7.  Persistence
+
+Every persistable store ships a `.tmp + atomic rename` writer + a
+versioned record format:
+
+```
+[u8 magic]
+[u8 version]       // 1..kVersion accepted on load
+[u8 state]
+... domain-specific fields ...
+```
+
+`enable_persistence(dir)` scans the dir on startup, replays records
+into in-memory state via `install_()`, and from there writes on every
+mutation.  See README "Schema migrations" for the version-bump
+discipline.
+
+The seven persistable stores (PJ, CJ, Carrier, LoadPort, Substrate,
+Exception, Spool) all follow the same pattern.  Adding persistence to
+a new store is a paste-and-adapt: copy `control_jobs.hpp`'s
+`write_record_` + `load_record_` + `enable_persistence`, change the
+magic byte + the fields.
+
+Magic bytes claimed so far (don't reuse):
+
+| Magic | Store              |
+|-------|--------------------|
+| 0xC4  | CarrierStore       |
+| 0xC5  | LoadPortStore      |
+| 0xC6  | SubstrateStore     |
+| 0xC7  | ProcessJobStore    |
+| 0xC8  | ControlJobStore    |
+| 0xC9  | ExceptionStore     |
+| 0xE5  | SpoolStore         |
+
+---
+
+## 8.  Codegen pipeline
+
+`tools/gen_messages.py` reads `data/messages.yaml` and emits
+`build/generated/secsgem/gem/messages.hpp`.  The pipeline:
+
+```
+messages.yaml
+    │
+    │  (CMake add_custom_command, runs on rebuild if YAML newer)
+    ▼
+tools/gen_messages.py
+    │
+    │  (Python reads YAML, emits typed C++ structs + builders + parsers)
+    ▼
+build/generated/secsgem/gem/messages.hpp
+    │
+    │  (#included by apps/, src/, tests/)
+    ▼
+secs_server.cpp / secs_client.cpp / your main.cpp
+```
+
+For each message in the catalog the codegen emits:
+
+- An optional `struct Name { ... }` (for list bodies)
+- A `inline secs2::Message builder_name(args...)` that returns a
+  ready-to-send Message
+- A `inline std::optional<...> parse_name(const secs2::Message&)`
+  that returns the parsed body or nullopt
+
+The YAML shape is documented in the file header of `messages.yaml`.
+Every supported body kind (`scalar`, `list`, `list_of`) maps to a
+straightforward C++ shape.
+
+---
+
+## 9.  Extending the library
+
+### 9.1.  New SECS-II message
+
+Edit `data/messages.yaml`:
+
+```yaml
+- id: S6F30
+  stream: 6
+  function: 30
+  w: true
+  builder: s6f30_my_request
+  parser: parse_s6f30
+  body:
+    kind: list
+    struct_name: MyRequest
+    fields:
+      - {name: dataid, shape: {kind: scalar, item_type: U4}}
+      - {name: payload, shape: {kind: scalar, item_type: ASCII}}
+```
+
+Rebuild — `messages.hpp` regenerates.  Register a handler:
+
+```cpp
+router.on(6, 30, [&](const secs2::Message& m) {
+  auto req = gem::parse_s6f30(m);
+  if (!req) return std::optional{secs2::Message(6, 0, false)};  // bad body
+  // ...
+  return std::optional{secs2::Message(6, 0, false)};            // W=0 reply
+});
+```
+
+That's the entire diff.  No core code change.
+
+### 9.2.  New state machine
+
+If your tool has a domain not covered by the existing stores
+(say, an in-chamber gas-flow FSM):
+
+1. Define the states + events:
+   ```cpp
+   // include/secsgem/gem/gas_flow.hpp
+   enum class GasFlowState  : uint8_t { Idle, Purging, Stable, Faulted };
+   enum class GasFlowEvent  : uint8_t { StartPurge, FlowStable, Fault, Reset };
+   ```
+2. Define the transition table — pure data:
+   ```cpp
+   struct GasFlowTransition { GasFlowState from; GasFlowEvent on; std::optional<GasFlowState> to; };
+   class GasFlowTransitionTable { /* mirrors ProcessJobTransitionTable */ };
+   ```
+3. Define the FSM:
+   ```cpp
+   class GasFlowStateMachine {
+    public:
+     bool fire(GasFlowEvent ev);  // returns whether a transition happened
+     GasFlowState state() const;
+     void set_state_change_handler(StateChangeHandler);
+   };
+   ```
+4. (Optional) Define a store if there can be many instances:
+   `class GasFlowStore { /* mirrors ProcessJobStore */ }` with
+   create/get/has/all + state-change relay.
+5. (Optional) YAML-load the transitions following `config::load_*` patterns.
+6. (Optional) Persistence: copy a store's `enable_persistence` + `write_record_` + `load_record_`.
+
+Reference patterns to lift from: `ept_state.hpp` (single global FSM),
+`process_job_state.hpp` (per-instance FSM in a store).
+
+### 9.3.  New store
+
+Stores follow the consistent API shape in §5.  Copy
+`include/secsgem/gem/store/alarms.hpp` (smallest example) or
+`include/secsgem/gem/store/process_jobs.hpp` (richest example,
+includes persistence).
+
+Wire into `EquipmentDataModel` if it should be globally accessible
+from `model->...`:
+
+```cpp
+// data_model.hpp
+#include "secsgem/gem/store/gas_flows.hpp"
+
+struct EquipmentDataModel {
+  // ... existing fields ...
+  GasFlowStore gas_flows;
+};
+```
+
+### 9.4.  New persistence backend
+
+The seven existing stores all journal to files.  If you want
+database-backed persistence (SQLite, Postgres, etcd), the cleanest
+pattern is to subclass-or-replace the `enable_persistence(path)`
+method:
+
+```cpp
+// Or: a sibling enable_db_persistence(connection_string)
+void enable_db_persistence(std::string conn) {
+  db_conn_ = std::move(conn);
+  /* on each create / mutation, write the record to the DB */
+}
+```
+
+The contract is consistent with file persistence: load at startup,
+write on mutation, atomic-rename equivalent (a transaction).  See
+`spool.hpp::enable_persistence` for the cleanest single-file
+example to mirror.
+
+### 9.5.  New transport
+
+`hsms::Connection` and `secsi::Protocol` are the two we ship.  A
+third (e.g. HSMS-over-TLS as a first-class thing, or HSMS over a
+sidecar IPC) follows the same contract:
+
+1. Accept a transport socket / endpoint.
+2. Expose `set_message_handler(...)`, `send_request(...)`,
+   `send_data(...)`, `set_selected_handler(...)`,
+   `set_closed_handler(...)`.
+3. Drive the SECS-II codec via `secs2::encode` / `secs2::decode`.
+
+The Router and the model don't care which transport produced the
+message.  Both wire into the same `set_message_handler` callback
+shape.
+
+---
+
+## 10.  Threading model
+
+Single-threaded by design.  The entire model — every store, every
+FSM, the Router, the Connection — is reachable only from the
+io_context that drives the HSMS connection.  No locks anywhere.
+
+This is documented as a contract in INTEGRATION.md §3 and exercised
+by:
+
+- `test_thread_safety.cpp` — N producer threads asio::post updates
+  onto the worker io
+- `test_concurrency.cpp` — in-flight transaction interleaving
+- The ThreadSanitizer CI lane — every test under
+  `-fsanitize=thread`
+
+If you're adding work that lives on another thread (sensor poll
+loop, separate metrics scraper, signal handler), marshal back to
+the io_context with `asio::post(io.get_executor(), ...)`.  Don't
+add locks; they'll diverge from the contract and the next
+contributor will be confused.
+
+---
+
+## 11.  Why C++20
+
+- `std::variant` for `Item` storage — exhaustive `std::visit`
+  catches new format codes at compile time.
+- `std::optional` everywhere — the codec, the parsers, the
+  store accessors all use it as the "missing value" idiom.
+- Designated initializers in tests — readability.
+- Concepts in template helpers (`messages_helpers.hpp`).
+- `<filesystem>` — persistence wouldn't be a header-only feature
+  without it.
+
+`g++-13` and `clang-18` both build the codebase clean at
+`-Wall -Wextra -Wpedantic`.
+
+---
+
+## 12.  Where to look in the source
+
+| You want to understand…             | Read these in order                                            |
+|-------------------------------------|----------------------------------------------------------------|
+| The wire byte layout                | `secs2/item.hpp`, `secs2/codec.cpp`, `tests/test_e5_kat.cpp`   |
+| How a typed message is built        | `data/messages.yaml`, `tools/gen_messages.py`, the generated header |
+| How HSMS handshakes                 | `hsms/connection.hpp/.cpp`, `tests/test_hsms_*.cpp`            |
+| How the Router dispatches           | `gem/router.hpp`                                               |
+| How a store implements persistence  | `gem/store/spool.hpp` (smallest), `gem/store/process_jobs.hpp` (richest) |
+| How an FSM is structured            | `gem/process_job_state.hpp`, `src/gem/process_job_state.cpp`   |
+| How the application wires it all    | `apps/secs_server.cpp` (the canonical example, ~1200 lines)    |
+| How a customer would write main()   | `examples/pvd_tool/main.cpp` (the worked vendor example)       |
+| How thread-safety works             | `tests/test_thread_safety.cpp`, INTEGRATION.md §3              |
+| How E84 timers integrate with asio  | `gem/e84_asio_timers.hpp` (the canonical I/O-adapter pattern)  |
+| How the property fuzz drives state  | `tests/test_robustness_fuzz.cpp`                               |
+
+---
+
+## 13.  What we deliberately don't do
+
+- **No DI framework, no service locator.**  Stores are owned by the
+  model; the model is owned by your application; everything else is
+  passed in by reference.  C++20 has no language-level DI, and adding
+  one to a codebase this size is overhead with no payoff.
+- **No singleton state.**  The model is a value, not a global.
+- **No std::shared_ptr-everywhere.**  asio handlers extend the
+  lifetimes that need extending; the rest is owned by-value.  Read
+  `Connection`'s lifetime contract in `hsms/connection.hpp` if you're
+  ever in doubt.
+- **No exceptions across the API boundary** — the codec throws
+  `secs2::CodecError` internally, but every public accessor returns
+  `std::optional` or returns a bool.  Exceptions are reserved for
+  programmer-error / corrupt-input paths.
+
+Every one of those constraints came from real review pressure on
+prior iterations.  Pushing back on them is welcome but please read
+the existing tests first; the codebase's architecture is what makes
+the property fuzz and the TSan lane feasible.