secs-gem/docs/ARCHITECTURE.md

Architecture

How the codebase is put together, and how to extend it. Read after INTEGRATION.md — that doc tells you what to do; this one tells you why, and where to plug in new behaviour.

---

1. Design principle: spec-as-data

The SEMI standards describe behaviour as tables — state machines, message catalogues, transition rules. C++ is the wrong language to write those tables in directly: every spec edit becomes a recompile, and reviewers can't audit "does the implementation match E40 §6.3" without reading code.

So the rule across the project is: anything the SEMI spec encodes as a table lives in YAML. The C++ is the engine that reads them.

   data/messages.yaml          → tools/gen_messages.py → messages.hpp
   data/control_state.yaml     → config::load_control_state()
   data/process_job_state.yaml → config::load_process_job_state()
   data/control_job_state.yaml → config::load_control_job_state()
   data/equipment.yaml         → config::load_equipment()

Two consequences worth absorbing:

• Adding a new SECS-II message rarely requires C++. Edit data/messages.yaml, rebuild, register a handler with the Router.
• Adding a new state transition rarely requires C++. Edit the relevant state YAML; the loader hot-loads on next start.

Things that do require C++: new kinds of behaviour (new FSM, new store, new persistence backend) — and that's what the rest of this doc covers.

---

2. The layers

┌─────────────────────────────────────────────────────────────────┐
│  vendor surfaces (out-of-process)                               │
│  secs_gemd gRPC daemon (proto/secsgem/v1) ← clients/python      │
│  secsgem-client and any-language gRPC stubs                     │
├─────────────────────────────────────────────────────────────────┤
│  apps/  (your main.cpp lives here)                              │
│  secs_server, secs_client, secs_gemd, secs_conformance,         │
│  secs_bench, fuzz_*, secs_interop_probe                         │
├─────────────────────────────────────────────────────────────────┤
│  gem::EquipmentRuntime  +  register_* capability functions      │
│  ─────────────────────────────────────────────────────          │
│  Runtime: owns io_context + Server + model + control FSM +      │
│  Router; thread-safe set/emit/alarm API, on_command hook,       │
│  read_sync, control-state mirror, link/state observers          │
│  default_handlers: the 56 GEM handlers as 15 per-capability     │
│  register_* fns (ids bound via the config's roles: block)       │
├─────────────────────────────────────────────────────────────────┤
│  gem::Router    +    gem::EquipmentDataModel                    │
│  ─────────────────────────────────────────                      │
│  Router: (stream, function) → handler dispatch table             │
│  Model:  composes every store + every FSM into one object       │
├─────────────────────────────────────────────────────────────────┤
│  Per-domain stores (include/secsgem/gem/store/)                 │
│  alarms, carriers, ceid+reports, exceptions, host_commands,     │
│  limits, modules, process_jobs, control_jobs, recipes, spool,   │
│  substrates, svid+dvid, trace, cem_objects, e84_ports, clock    │
├─────────────────────────────────────────────────────────────────┤
│  Per-standard state machines                                    │
│  E30 control_state, E30 communication_state, E40 PJ,            │
│  E94 CJ, E87 carriers + load_ports, E90 substrates,             │
│  E116 EPT, E157 modules, E5 exceptions, E84 handshake           │
├─────────────────────────────────────────────────────────────────┤
│  hsms::Connection (Asio)  +  secsi::Protocol  +  secs2 codec    │
│  ───────────────────────────────────────────────────────        │
│  Transport: HSMS-SS, HSMS-GS, SECS-I (FSM only)                 │
│  Codec:     Item ⇄ bytes, Item ⇄ SML text                       │
└─────────────────────────────────────────────────────────────────┘

Each layer is replaceable. The codec doesn't know about the FSMs; the FSMs don't know about the codec; the Router doesn't know about persistence. The model composes them but doesn't own their logic.

---

3. The codec (secs2/)

secs2::Item is a tagged variant over the SEMI E5 §9 formats: List, Binary, Boolean, ASCII, JIS-8, C2, U1-U8, I1-I8, F4, F8. Storage is a std::variant matching each format's natural C++ type.

secs2::encode(item) → vector<uint8_t>          // bytes for the wire
secs2::decode(bytes) → Item                    // wire → object
secs2::to_sml(item) → string                   // human-readable
secs2::from_sml(text) → Item                   // and back

The encoder emits the format-byte arithmetic described in GLOSSARY.md → SEMI E5 §9. The decoder is strict about format codes but lenient about U-widths in identifier fields (per messages_helpers::any_unsigned_first) — that's how secsgem-py interop works without breaking spec-correctness.

Coverage on the codec: 196 SEMI E5 KAT assertions, 120+ unit tests, libFuzzer with 70 000+ random inputs per minute under ASan + UBSan.

4. Transport (hsms/, secsi/)

hsms::Connection owns one TCP socket and one (SS) or many (GS) session-state objects. Frames have a 4-byte length prefix + 10-byte header (session_id, byte2, byte3, PType, SType, system_bytes) + optional SECS-II body.

State transitions: NOT-CONNECTED → NOT-SELECTED (T7 armed) → SELECTED. Either side can initiate Select.req; both modes (Active / Passive) are first-class.

The connection class is I/O-aware: it owns the asio socket, arms the T-timers, drives the read loop. Everything above it is I/O-free and reachable through callbacks:

conn->set_message_handler([&router](const secs2::Message& m) {
  return router.dispatch_with_s9(/*emit=*/..., /*mhead=*/..., m);
});

SECS-I (secsi::Protocol) is an FSM-only port of the same idea — serial-line framing, T1/T2/T3/T4 timers as callbacks. No asio inside the FSM; the application drives the clock. The E84 timers follow the same pattern (E84AsioTimers is the asio adapter; the FSM stays pure).

5. The model (gem/)

gem::EquipmentDataModel (data_model.hpp) is a struct composing every store:

struct EquipmentDataModel {
  StatusVariableStore       svids;
  DataVariableStore         dvids;
  EquipmentConstantStore    ecids;
  EventReportSubscriptions  events;
  AlarmRegistry             alarms;
  RecipeStore               recipes;
  Clock                     clock;
  HostCommandRegistry       commands;
  SpoolStore                spool;
  LimitMonitorStore         limits;
  TraceStore                traces;
  ProcessJobStore           process_jobs;
  ControlJobStore           control_jobs;
  ExceptionStore            exceptions;
  CarrierStore              carriers;
  LoadPortStore             load_ports;
  SubstrateStore            substrates;
  EptStateMachine           ept;
  CemObjectStore            cem;
  ModuleStore               modules;
  E84PortStore              e84_ports;
};

No locks. Single-threaded contract documented in INTEGRATION.md §3. All mutation runs on the io_context strand.

Each store is independently usable — you can #include "secsgem/gem/store/alarms.hpp" and use AlarmRegistry without pulling in any of the others. The composite is for convenience.

Per-store pattern

Every store follows the same shape:

class FooStore {
 public:
  // CRUD
  bool create(...);
  Foo* get(id);              // mutable pointer, nullable
  const Foo* get(id) const;  // const-mutable pointer, nullable
  bool has(id) const;
  bool remove(id);
  std::size_t size() const;
  std::vector<Foo> all() const;

  // Domain operations
  void fire_internal(id, FooEvent event);   // application-driven
  Ack on_host_command(id, FooEvent event);  // host-driven

  // Observers
  void set_state_change_handler(StateChangeHandler);

  // Persistence
  void enable_persistence(std::filesystem::path dir);
};

The store owns the FSM instance, the persistence file path, the in-memory state. The FSM owns the legal-transition table. The table comes from a YAML file (loaded into factory_() at construction).

6. The Router

gem::Router (router.hpp) is a tiny dispatch table:

Router r;
r.on(1, 13, [&](const secs2::Message&) {
  return gem::s1f14_establish_comms_ack(...);
});
r.on(2, 41, [&](const secs2::Message& msg) {
  auto cmd = gem::parse_s2f41(msg);
  // ... handle command ...
  return gem::s2f42_host_command_ack(...);
});

auto reply = r.dispatch(incoming_message);

Handlers are std::function<std::optional<Message>(const Message&)>. Return nullopt for one-way (W=0) primaries.

dispatch_with_s9 wraps dispatch to auto-emit S9F3 (unrecognized stream) or S9F5 (unrecognized function) when no handler is registered — the spec-mandated response.

The Router is stateless — it just looks up handlers in a std::map<std::pair<uint8_t, uint8_t>, Handler>. All state lives in the model the handlers close over.

7. Persistence

Every persistable store ships a .tmp + atomic rename writer + a versioned record format:

[u8 magic]
[u8 version]       // 1..kVersion accepted on load
[u8 state]
... domain-specific fields ...

enable_persistence(dir) scans the dir on startup, replays records into in-memory state via install_(), and from there writes on every mutation. See README "Schema migrations" for the version-bump discipline.

The seven persistable stores (PJ, CJ, Carrier, LoadPort, Substrate, Exception, Spool) all follow the same pattern. Adding persistence to a new store is a paste-and-adapt: copy control_jobs.hpp's write_record_ + load_record_ + enable_persistence, change the magic byte + the fields.

Magic bytes claimed so far (don't reuse):

Magic	Store
0xC4	CarrierStore
0xC5	LoadPortStore
0xC6	SubstrateStore
0xC7	ProcessJobStore
0xC8	ControlJobStore
0xC9	ExceptionStore
0xE5	SpoolStore

---

8. Codegen pipeline

tools/gen_messages.py reads data/messages.yaml and emits build/generated/secsgem/gem/messages.hpp. The pipeline:

messages.yaml
    │
    │  (CMake add_custom_command, runs on rebuild if YAML newer)
    ▼
tools/gen_messages.py
    │
    │  (Python reads YAML, emits typed C++ structs + builders + parsers)
    ▼
build/generated/secsgem/gem/messages.hpp
    │
    │  (#included by apps/, src/, tests/)
    ▼
secs_server.cpp / secs_client.cpp / your main.cpp

For each message in the catalog the codegen emits:

• An optional struct Name { ... } (for list bodies)
• A inline secs2::Message builder_name(args...) that returns a ready-to-send Message
• A inline std::optional<...> parse_name(const secs2::Message&) that returns the parsed body or nullopt

The YAML shape is documented in the file header of messages.yaml. Every supported body kind (scalar, list, list_of) maps to a straightforward C++ shape.

---

9. Extending the library

9.1. New SECS-II message

Edit data/messages.yaml:

- id: S6F30
  stream: 6
  function: 30
  w: true
  builder: s6f30_my_request
  parser: parse_s6f30
  body:
    kind: list
    struct_name: MyRequest
    fields:
      - {name: dataid, shape: {kind: scalar, item_type: U4}}
      - {name: payload, shape: {kind: scalar, item_type: ASCII}}

Rebuild — messages.hpp regenerates. Register a handler:

router.on(6, 30, [&](const secs2::Message& m) {
  auto req = gem::parse_s6f30(m);
  if (!req) return std::optional{secs2::Message(6, 0, false)};  // bad body
  // ...
  return std::optional{secs2::Message(6, 0, false)};            // W=0 reply
});

That's the entire diff. No core code change.

9.2. New state machine

If your tool has a domain not covered by the existing stores (say, an in-chamber gas-flow FSM):

1. Define the states + events:

   // include/secsgem/gem/gas_flow.hpp
   enum class GasFlowState  : uint8_t { Idle, Purging, Stable, Faulted };
   enum class GasFlowEvent  : uint8_t { StartPurge, FlowStable, Fault, Reset };
   

2. Define the transition table — pure data:

   struct GasFlowTransition { GasFlowState from; GasFlowEvent on; std::optional<GasFlowState> to; };
   class GasFlowTransitionTable { /* mirrors ProcessJobTransitionTable */ };
   

3. Define the FSM:

   class GasFlowStateMachine {
    public:
     bool fire(GasFlowEvent ev);  // returns whether a transition happened
     GasFlowState state() const;
     void set_state_change_handler(StateChangeHandler);
   };
   

4. (Optional) Define a store if there can be many instances: class GasFlowStore { /* mirrors ProcessJobStore */ } with create/get/has/all + state-change relay.
5. (Optional) YAML-load the transitions following config::load_* patterns.
6. (Optional) Persistence: copy a store's enable_persistence + write_record_ + load_record_.

Reference patterns to lift from: ept_state.hpp (single global FSM), process_job_state.hpp (per-instance FSM in a store).

9.3. New store

Stores follow the consistent API shape in §5. Copy include/secsgem/gem/store/alarms.hpp (smallest example) or include/secsgem/gem/store/process_jobs.hpp (richest example, includes persistence).

Wire into EquipmentDataModel if it should be globally accessible from model->...:

// data_model.hpp
#include "secsgem/gem/store/gas_flows.hpp"

struct EquipmentDataModel {
  // ... existing fields ...
  GasFlowStore gas_flows;
};

9.4. New persistence backend

The seven existing stores all journal to files. If you want database-backed persistence (SQLite, Postgres, etcd), the cleanest pattern is to subclass-or-replace the enable_persistence(path) method:

// Or: a sibling enable_db_persistence(connection_string)
void enable_db_persistence(std::string conn) {
  db_conn_ = std::move(conn);
  /* on each create / mutation, write the record to the DB */
}

The contract is consistent with file persistence: load at startup, write on mutation, atomic-rename equivalent (a transaction). See spool.hpp::enable_persistence for the cleanest single-file example to mirror.

9.5. New transport

hsms::Connection and secsi::Protocol are the two shipped transports. A third (e.g. HSMS-over-TLS as a first-class thing, or HSMS over a sidecar IPC) follows the same contract:

1. Accept a transport socket / endpoint.
2. Expose set_message_handler(...), send_request(...), send_data(...), set_selected_handler(...), set_closed_handler(...).
3. Drive the SECS-II codec via secs2::encode / secs2::decode.

The Router and the model don't care which transport produced the message. Both wire into the same set_message_handler callback shape.

---

10. Threading model

Single-threaded by design. The entire model — every store, every FSM, the Router, the Connection — is reachable only from the io_context that drives the HSMS connection. No locks anywhere.

This is documented as a contract in INTEGRATION.md §3 and exercised by:

• test_thread_safety.cpp — N producer threads asio::post updates onto the worker io
• test_concurrency.cpp — in-flight transaction interleaving
• The ThreadSanitizer CI lane — every test under -fsanitize=thread

Work that lives on another thread (sensor poll loop, separate metrics scraper, signal handler) must marshal back to the io_context via asio::post(io.get_executor(), ...). The model contract has no locks; adding any would diverge from it.

---

11. Why C++20

• std::variant for Item storage — exhaustive std::visit catches new format codes at compile time.
• std::optional everywhere — the codec, the parsers, the store accessors all use it as the "missing value" idiom.
• Designated initializers in tests — readability.
• Concepts in template helpers (messages_helpers.hpp).
• <filesystem> — persistence wouldn't be a header-only feature without it.

g++-13 and clang-18 both build the codebase clean at -Wall -Wextra -Wpedantic.

---

12. Where to look in the source

You want to understand…	Read these in order
The wire byte layout	secs2/item.hpp, secs2/codec.cpp, tests/test_e5_kat.cpp
How a typed message is built	data/messages.yaml, tools/gen_messages.py, the generated header
How HSMS handshakes	hsms/connection.hpp/.cpp, tests/test_hsms_*.cpp
How the Router dispatches	gem/router.hpp
How a store implements persistence	gem/store/spool.hpp (smallest), gem/store/process_jobs.hpp (richest)
How an FSM is structured	gem/process_job_state.hpp, src/gem/process_job_state.cpp
How the application wires it all	gem::EquipmentRuntime + register_default_handlers (apps/secs_server.cpp is now a ~110-line thin main over them)
How a customer would write main()	examples/pvd_tool/main.cpp (the worked vendor example)
How thread-safety works	tests/test_thread_safety.cpp, INTEGRATION.md §3
How E84 timers integrate with asio	gem/e84_asio_timers.hpp (the canonical I/O-adapter pattern)
How the property fuzz drives state	tests/test_robustness_fuzz.cpp

---

13. Deliberate non-features

• No DI framework, no service locator. Stores are owned by the model; the model is owned by the application; everything else passes by reference.
• No singleton state. The model is a value, not a global.
• No std::shared_ptr-everywhere. asio handlers extend the lifetimes that need extending; the rest is owned by-value. Connection's lifetime contract is documented in hsms/connection.hpp.
• No exceptions across the API boundary. The codec throws secs2::CodecError internally, but every public accessor returns std::optional or a bool. Exceptions are reserved for programmer-error / corrupt-input paths.

The TSan lane and the property fuzz depend on these constraints.