secs-gem/docs/ARCHITECTURE.md

# Architecture

How the codebase is put together, and how to extend it.  Read after
[INTEGRATION.md](INTEGRATION.md) — that doc tells you what to do;
this one tells you *why*, and where to plug in new behaviour.

---

## 1.  Design principle: spec-as-data

The SEMI standards describe behaviour as **tables** — state machines,
message catalogues, transition rules.  C++ is the wrong language to
write those tables in directly: every spec edit becomes a recompile,
and reviewers can't audit "does the implementation match E40 §6.3"
without reading code.

So the rule across the project is: **anything the SEMI spec encodes
as a table lives in YAML.**  The C++ is the engine that reads them.

```
   data/messages.yaml          → tools/gen_messages.py → messages.hpp
   data/control_state.yaml     → config::load_control_state()
   data/process_job_state.yaml → config::load_process_job_state()
   data/control_job_state.yaml → config::load_control_job_state()
   data/equipment.yaml         → config::load_equipment()
```

Two consequences worth absorbing:

- **Adding a new SECS-II message rarely requires C++.**  Edit
  `data/messages.yaml`, rebuild, register a handler with the Router.
- **Adding a new state transition rarely requires C++.**  Edit the
  relevant state YAML; the loader hot-loads on next start.

Things that do require C++: new *kinds* of behaviour (new FSM, new
store, new persistence backend) — and that's what the rest of this
doc covers.

---

## 2.  The layers

```
┌─────────────────────────────────────────────────────────────────┐
│  vendor surfaces (out-of-process)                               │
│  secs_gemd gRPC daemon (proto/secsgem/v1) ← clients/python      │
│  secsgem-client and any-language gRPC stubs                     │
├─────────────────────────────────────────────────────────────────┤
│  apps/  (your main.cpp lives here)                              │
│  secs_server, secs_client, secs_gemd, secs_conformance,         │
│  secs_bench, fuzz_*, secs_interop_probe                         │
├─────────────────────────────────────────────────────────────────┤
│  gem::EquipmentRuntime  +  register_* capability functions      │
│  ─────────────────────────────────────────────────────          │
│  Runtime: owns io_context + Server + model + control FSM +      │
│  Router; thread-safe set/emit/alarm API, on_command hook,       │
│  read_sync, control-state mirror, link/state observers          │
│  default_handlers: the 56 GEM handlers as 15 per-capability     │
│  register_* fns (ids bound via the config's roles: block)       │
├─────────────────────────────────────────────────────────────────┤
│  gem::Router    +    gem::EquipmentDataModel                    │
│  ─────────────────────────────────────────                      │
│  Router: (stream, function) → handler dispatch table             │
│  Model:  composes every store + every FSM into one object       │
├─────────────────────────────────────────────────────────────────┤
│  Per-domain stores (include/secsgem/gem/store/)                 │
│  alarms, carriers, ceid+reports, exceptions, host_commands,     │
│  limits, modules, process_jobs, control_jobs, recipes, spool,   │
│  substrates, svid+dvid, trace, cem_objects, e84_ports, clock    │
├─────────────────────────────────────────────────────────────────┤
│  Per-standard state machines                                    │
│  E30 control_state, E30 communication_state, E40 PJ,            │
│  E94 CJ, E87 carriers + load_ports, E90 substrates,             │
│  E116 EPT, E157 modules, E5 exceptions, E84 handshake           │
├─────────────────────────────────────────────────────────────────┤
│  hsms::Connection (Asio)  +  secsi::Protocol  +  secs2 codec    │
│  ───────────────────────────────────────────────────────        │
│  Transport: HSMS-SS, HSMS-GS, SECS-I (FSM only)                 │
│  Codec:     Item ⇄ bytes, Item ⇄ SML text                       │
└─────────────────────────────────────────────────────────────────┘
```

Each layer is replaceable.  The codec doesn't know about the FSMs;
the FSMs don't know about the codec; the Router doesn't know about
persistence.  The model composes them but doesn't own their logic.

---

## 3.  The codec (`secs2/`)

`secs2::Item` is a tagged variant over the SEMI E5 §9 formats: List,
Binary, Boolean, ASCII, JIS-8, C2, U1-U8, I1-I8, F4, F8.  Storage is
a `std::variant` matching each format's natural C++ type.

```
secs2::encode(item) → vector<uint8_t>          // bytes for the wire
secs2::decode(bytes) → Item                    // wire → object
secs2::to_sml(item) → string                   // human-readable
secs2::from_sml(text) → Item                   // and back
```

The encoder emits the format-byte arithmetic described in
[GLOSSARY.md → SEMI E5 §9](GLOSSARY.md).  The decoder is strict
about format codes but lenient about U-widths in identifier fields
(per `messages_helpers::any_unsigned_first`) — that's how secsgem-py
interop works without breaking spec-correctness.

Coverage on the codec: 196 SEMI E5 KAT assertions, 120+ unit tests,
libFuzzer with 70 000+ random inputs per minute under ASan + UBSan.

## 4.  Transport (`hsms/`, `secsi/`)

`hsms::Connection` owns one TCP socket and one (SS) or many (GS)
session-state objects.  Frames have a 4-byte length prefix + 10-byte
header (session_id, byte2, byte3, PType, SType, system_bytes) +
optional SECS-II body.

State transitions: NOT-CONNECTED → NOT-SELECTED (T7 armed) → SELECTED.
Either side can initiate Select.req; both modes (Active / Passive)
are first-class.

The connection class is **I/O-aware**: it owns the asio socket, arms
the T-timers, drives the read loop.  Everything above it is I/O-free
and reachable through callbacks:

```cpp
conn->set_message_handler([&router](const secs2::Message& m) {
  return router.dispatch_with_s9(/*emit=*/..., /*mhead=*/..., m);
});
```

SECS-I (`secsi::Protocol`) is an FSM-only port of the same idea —
serial-line framing, T1/T2/T3/T4 timers as callbacks.  No asio
inside the FSM; the application drives the clock.  The E84 timers
follow the same pattern (`E84AsioTimers` is the asio adapter; the
FSM stays pure).

## 5.  The model (`gem/`)

`gem::EquipmentDataModel` (data_model.hpp) is a struct composing
every store:

```cpp
struct EquipmentDataModel {
  StatusVariableStore       svids;
  DataVariableStore         dvids;
  EquipmentConstantStore    ecids;
  EventReportSubscriptions  events;
  AlarmRegistry             alarms;
  RecipeStore               recipes;
  Clock                     clock;
  HostCommandRegistry       commands;
  SpoolStore                spool;
  LimitMonitorStore         limits;
  TraceStore                traces;
  ProcessJobStore           process_jobs;
  ControlJobStore           control_jobs;
  ExceptionStore            exceptions;
  CarrierStore              carriers;
  LoadPortStore             load_ports;
  SubstrateStore            substrates;
  EptStateMachine           ept;
  CemObjectStore            cem;
  ModuleStore               modules;
  E84PortStore              e84_ports;
};
```

No locks.  Single-threaded contract documented in INTEGRATION.md §3.
All mutation runs on the io_context strand.

Each store is **independently usable** — you can `#include
"secsgem/gem/store/alarms.hpp"` and use `AlarmRegistry` without
pulling in any of the others.  The composite is for convenience.

### Per-store pattern

Every store follows the same shape:

```cpp
class FooStore {
 public:
  // CRUD
  bool create(...);
  Foo* get(id);              // mutable pointer, nullable
  const Foo* get(id) const;  // const-mutable pointer, nullable
  bool has(id) const;
  bool remove(id);
  std::size_t size() const;
  std::vector<Foo> all() const;

  // Domain operations
  void fire_internal(id, FooEvent event);   // application-driven
  Ack on_host_command(id, FooEvent event);  // host-driven

  // Observers
  void set_state_change_handler(StateChangeHandler);

  // Persistence
  void enable_persistence(std::filesystem::path dir);
};
```

The store owns the FSM instance, the persistence file path, the
in-memory state.  The FSM owns the legal-transition table.  The
table comes from a YAML file (loaded into `factory_()` at
construction).

## 6.  The Router

`gem::Router` (router.hpp) is a tiny dispatch table:

```cpp
Router r;
r.on(1, 13, [&](const secs2::Message&) {
  return gem::s1f14_establish_comms_ack(...);
});
r.on(2, 41, [&](const secs2::Message& msg) {
  auto cmd = gem::parse_s2f41(msg);
  // ... handle command ...
  return gem::s2f42_host_command_ack(...);
});

auto reply = r.dispatch(incoming_message);
```

Handlers are `std::function<std::optional<Message>(const Message&)>`.
Return nullopt for one-way (W=0) primaries.

`dispatch_with_s9` wraps `dispatch` to auto-emit `S9F3` (unrecognized
stream) or `S9F5` (unrecognized function) when no handler is
registered — the spec-mandated response.

The Router is **stateless** — it just looks up handlers in a
`std::map<std::pair<uint8_t, uint8_t>, Handler>`.  All state lives
in the model the handlers close over.

## 7.  Persistence

Every persistable store ships a `.tmp + atomic rename` writer + a
versioned record format:

```
[u8 magic]
[u8 version]       // 1..kVersion accepted on load
[u8 state]
... domain-specific fields ...
```

`enable_persistence(dir)` scans the dir on startup, replays records
into in-memory state via `install_()`, and from there writes on every
mutation.  See README "Schema migrations" for the version-bump
discipline.

The seven persistable stores (PJ, CJ, Carrier, LoadPort, Substrate,
Exception, Spool) all follow the same pattern.  Adding persistence to
a new store is a paste-and-adapt: copy `control_jobs.hpp`'s
`write_record_` + `load_record_` + `enable_persistence`, change the
magic byte + the fields.

Magic bytes claimed so far (don't reuse):

| Magic | Store              |
|-------|--------------------|
| 0xC4  | CarrierStore       |
| 0xC5  | LoadPortStore      |
| 0xC6  | SubstrateStore     |
| 0xC7  | ProcessJobStore    |
| 0xC8  | ControlJobStore    |
| 0xC9  | ExceptionStore     |
| 0xE5  | SpoolStore         |

---

## 8.  Codegen pipeline

`tools/gen_messages.py` reads `data/messages.yaml` and emits
`build/generated/secsgem/gem/messages.hpp`.  The pipeline:

```
messages.yaml
    │
    │  (CMake add_custom_command, runs on rebuild if YAML newer)
    ▼
tools/gen_messages.py
    │
    │  (Python reads YAML, emits typed C++ structs + builders + parsers)
    ▼
build/generated/secsgem/gem/messages.hpp
    │
    │  (#included by apps/, src/, tests/)
    ▼
secs_server.cpp / secs_client.cpp / your main.cpp
```

For each message in the catalog the codegen emits:

- An optional `struct Name { ... }` (for list bodies)
- A `inline secs2::Message builder_name(args...)` that returns a
  ready-to-send Message
- A `inline std::optional<...> parse_name(const secs2::Message&)`
  that returns the parsed body or nullopt

The YAML shape is documented in the file header of `messages.yaml`.
Every supported body kind (`scalar`, `list`, `list_of`) maps to a
straightforward C++ shape.

---

## 9.  Extending the library

### 9.1.  New SECS-II message

Edit `data/messages.yaml`:

```yaml
- id: S6F30
  stream: 6
  function: 30
  w: true
  builder: s6f30_my_request
  parser: parse_s6f30
  body:
    kind: list
    struct_name: MyRequest
    fields:
      - {name: dataid, shape: {kind: scalar, item_type: U4}}
      - {name: payload, shape: {kind: scalar, item_type: ASCII}}
```

Rebuild — `messages.hpp` regenerates.  Register a handler:

```cpp
router.on(6, 30, [&](const secs2::Message& m) {
  auto req = gem::parse_s6f30(m);
  if (!req) return std::optional{secs2::Message(6, 0, false)};  // bad body
  // ...
  return std::optional{secs2::Message(6, 0, false)};            // W=0 reply
});
```

That's the entire diff.  No core code change.

### 9.2.  New state machine

If your tool has a domain not covered by the existing stores
(say, an in-chamber gas-flow FSM):

1. Define the states + events:
   ```cpp
   // include/secsgem/gem/gas_flow.hpp
   enum class GasFlowState  : uint8_t { Idle, Purging, Stable, Faulted };
   enum class GasFlowEvent  : uint8_t { StartPurge, FlowStable, Fault, Reset };
   ```
2. Define the transition table — pure data:
   ```cpp
   struct GasFlowTransition { GasFlowState from; GasFlowEvent on; std::optional<GasFlowState> to; };
   class GasFlowTransitionTable { /* mirrors ProcessJobTransitionTable */ };
   ```
3. Define the FSM:
   ```cpp
   class GasFlowStateMachine {
    public:
     bool fire(GasFlowEvent ev);  // returns whether a transition happened
     GasFlowState state() const;
     void set_state_change_handler(StateChangeHandler);
   };
   ```
4. (Optional) Define a store if there can be many instances:
   `class GasFlowStore { /* mirrors ProcessJobStore */ }` with
   create/get/has/all + state-change relay.
5. (Optional) YAML-load the transitions following `config::load_*` patterns.
6. (Optional) Persistence: copy a store's `enable_persistence` + `write_record_` + `load_record_`.

Reference patterns to lift from: `ept_state.hpp` (single global FSM),
`process_job_state.hpp` (per-instance FSM in a store).

### 9.3.  New store

Stores follow the consistent API shape in §5.  Copy
`include/secsgem/gem/store/alarms.hpp` (smallest example) or
`include/secsgem/gem/store/process_jobs.hpp` (richest example,
includes persistence).

Wire into `EquipmentDataModel` if it should be globally accessible
from `model->...`:

```cpp
// data_model.hpp
#include "secsgem/gem/store/gas_flows.hpp"

struct EquipmentDataModel {
  // ... existing fields ...
  GasFlowStore gas_flows;
};
```

### 9.4.  New persistence backend

The seven existing stores all journal to files.  If you want
database-backed persistence (SQLite, Postgres, etcd), the cleanest
pattern is to subclass-or-replace the `enable_persistence(path)`
method:

```cpp
// Or: a sibling enable_db_persistence(connection_string)
void enable_db_persistence(std::string conn) {
  db_conn_ = std::move(conn);
  /* on each create / mutation, write the record to the DB */
}
```

The contract is consistent with file persistence: load at startup,
write on mutation, atomic-rename equivalent (a transaction).  See
`spool.hpp::enable_persistence` for the cleanest single-file
example to mirror.

### 9.5.  New transport

`hsms::Connection` and `secsi::Protocol` are the two shipped
transports.  A third (e.g. HSMS-over-TLS as a first-class thing, or
HSMS over a sidecar IPC) follows the same contract:

1. Accept a transport socket / endpoint.
2. Expose `set_message_handler(...)`, `send_request(...)`,
   `send_data(...)`, `set_selected_handler(...)`,
   `set_closed_handler(...)`.
3. Drive the SECS-II codec via `secs2::encode` / `secs2::decode`.

The Router and the model don't care which transport produced the
message.  Both wire into the same `set_message_handler` callback
shape.

---

## 10.  Threading model

Single-threaded by design.  The entire model — every store, every
FSM, the Router, the Connection — is reachable only from the
io_context that drives the HSMS connection.  No locks anywhere.

This is documented as a contract in INTEGRATION.md §3 and exercised
by:

- `test_thread_safety.cpp` — N producer threads asio::post updates
  onto the worker io
- `test_concurrency.cpp` — in-flight transaction interleaving
- The ThreadSanitizer CI lane — every test under
  `-fsanitize=thread`

Work that lives on another thread (sensor poll loop, separate
metrics scraper, signal handler) must marshal back to the
io_context via `asio::post(io.get_executor(), ...)`.  The model
contract has no locks; adding any would diverge from it.

---

## 11.  Why C++20

- `std::variant` for `Item` storage — exhaustive `std::visit`
  catches new format codes at compile time.
- `std::optional` everywhere — the codec, the parsers, the
  store accessors all use it as the "missing value" idiom.
- Designated initializers in tests — readability.
- Concepts in template helpers (`messages_helpers.hpp`).
- `<filesystem>` — persistence wouldn't be a header-only feature
  without it.

`g++-13` and `clang-18` both build the codebase clean at
`-Wall -Wextra -Wpedantic`.

---

## 12.  Where to look in the source

| You want to understand…             | Read these in order                                            |
|-------------------------------------|----------------------------------------------------------------|
| The wire byte layout                | `secs2/item.hpp`, `secs2/codec.cpp`, `tests/test_e5_kat.cpp`   |
| How a typed message is built        | `data/messages.yaml`, `tools/gen_messages.py`, the generated header |
| How HSMS handshakes                 | `hsms/connection.hpp/.cpp`, `tests/test_hsms_*.cpp`            |
| How the Router dispatches           | `gem/router.hpp`                                               |
| How a store implements persistence  | `gem/store/spool.hpp` (smallest), `gem/store/process_jobs.hpp` (richest) |
| How an FSM is structured            | `gem/process_job_state.hpp`, `src/gem/process_job_state.cpp`   |
| How the application wires it all    | `gem::EquipmentRuntime` + `register_default_handlers` (apps/secs_server.cpp is now a ~110-line thin main over them) |
| How a customer would write main()   | `examples/pvd_tool/main.cpp` (the worked vendor example)       |
| How thread-safety works             | `tests/test_thread_safety.cpp`, INTEGRATION.md §3              |
| How E84 timers integrate with asio  | `gem/e84_asio_timers.hpp` (the canonical I/O-adapter pattern)  |
| How the property fuzz drives state  | `tests/test_robustness_fuzz.cpp`                               |

---

## 13.  Deliberate non-features

- **No DI framework, no service locator.**  Stores are owned by the
  model; the model is owned by the application; everything else
  passes by reference.
- **No singleton state.**  The model is a value, not a global.
- **No std::shared_ptr-everywhere.**  asio handlers extend the
  lifetimes that need extending; the rest is owned by-value.
  `Connection`'s lifetime contract is documented in
  `hsms/connection.hpp`.
- **No exceptions across the API boundary.**  The codec throws
  `secs2::CodecError` internally, but every public accessor returns
  `std::optional` or a bool.  Exceptions are reserved for
  programmer-error / corrupt-input paths.

The TSan lane and the property fuzz depend on these constraints.