# 32 — Stores and the data model ← [31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md) | [Back to index](00_index.md) | Next: [33 Transport](33_transport.md) → The previous chapter showed how YAML drives behaviour. This chapter shows the runtime data structures that the YAML populates and that the Router handlers operate on: **stores** and the **`EquipmentDataModel`** that composes them. By the end you'll know: - What a **store** is (and what it isn't). - Every store in the codebase, one sentence each. - How `EquipmentDataModel` composes them. - The "no locks; single-threaded" contract. - How to add a new store. --- ## What a store is A **store** is a per-domain bundle of: - A few typed records (`std::map`, `std::vector`, …). - A small API for reading + mutating them. - A `change_handler` that emits events on transitions. - Optional file-backed persistence. The naming is consistent: `AlarmRegistry` for active alarms, `CarrierStore` for carriers, `ProcessJobStore` for PJs. Headers live in [`include/secsgem/gem/store/`](../include/secsgem/gem/store/); implementations are typically inline in the header (these are small). Each store maps onto **one concern from one SEMI standard**: | Store | Concern | Standard | |-----------------------------|--------------------------------------------------|----------------| | `StatusVariableStore` | SVIDs + values | E30 §6.13 | | `DataVariableStore` | DVIDs + values | E30 §6.11 | | `EquipmentConstantStore` | ECIDs + values + min/max bounds | E30 §6.16 | | `EventReportSubscriptions` | RPTID definitions + CEID linkings + enables | E30 §6.6 | | `AlarmRegistry` | ALIDs + ALCD/ALTX + enable bits + active set | E30 §6.14 | | `RecipeStore` | PPIDs + PPBODY (unformatted) + formatted bodies | E30 §6.17 + E42| | `Clock` | Wall-clock + drift + quality | E30 §6.20 + E148 | | `HostCommandRegistry` | RCMD names + per-command ack + side effects | E30 §6.15 | | `SpoolStore` | Per-stream whitelist + queue + persistent journal| E30 §6.22 | | `LimitMonitorStore` | LIMITIDs + upper/lower bounds + active state | E30 §6.21 | | `TraceStore` | TRIDs + active sampling config | E30 §6.12 | | `ProcessJobStore` | PJs + state + material list + persistent | E40 | | `ControlJobStore` | CJs + state + PJ refs + persistent | E94 | | `ExceptionStore` | EXIDs + recovery state + persistent | E5 §13 | | `CarrierStore` | Carrier IDs + state machines + persistent | E87 | | `LoadPortStore` | LP IDs + transfer/reservation/association FSMs | E87 | | `SubstrateStore` | Substrate IDs + 3 FSMs + location + persistent | E90 | | `EptStateMachine` | EPT state + time buckets | E116 | | `CemObjectStore` | E120 typed object hierarchy | E120 | | `ModuleStore` | Module IDs + state | E157 | | `E84PortStore` | Per-LP E84 FSM + signals + timers | E84 | Each is a header. Each is independently testable: you can `#include "secsgem/gem/store/alarms.hpp"` and exercise `AlarmRegistry` without pulling in the rest. This is the same shape as the per-standard tests in [`tests/`](../tests/) — one test file per store. --- ## EquipmentDataModel — the composite [`include/secsgem/gem/data_model.hpp`](../include/secsgem/gem/data_model.hpp) defines: ```cpp struct EquipmentDataModel { StatusVariableStore svids; DataVariableStore dvids; EquipmentConstantStore ecids; EventReportSubscriptions events; AlarmRegistry alarms; RecipeStore recipes; Clock clock; HostCommandRegistry commands; SpoolStore spool; LimitMonitorStore limits; TraceStore traces; ProcessJobStore process_jobs; ControlJobStore control_jobs; ExceptionStore exceptions; CarrierStore carriers; LoadPortStore load_ports; SubstrateStore substrates; EptStateMachine ept; CemObjectStore cem; ModuleStore modules; E84PortStore e84_ports; // ... convenience methods spanning stores }; ``` That's it. No locks, no smart pointers, no interfaces, no DI container. Each store is a value member; ownership is the `EquipmentDataModel` itself. The application typically holds one `shared_ptr` and passes it to every Router handler. Handlers operate on the stores directly: ```cpp router.on(1, 3, [model](const secs2::Message& m) { // S1F3 — host requests SVID values auto svids = parse_s1f3(m.body()); return build_s1f4(model->svids.values(svids)); }); ``` ### Convenience methods `EquipmentDataModel` adds a few cross-store helpers (`data_model.hpp:54`): ```cpp std::optional vid_value(uint32_t vid) const { // Look up VID in svids first, then dvids. } std::vector compose_reports_for(uint32_t ceid) const { // Walk events store -> reports store -> svids/dvids, // assemble the S6F11 report payload for one CEID firing. } ``` `compose_reports_for` is the *heart* of event notification — it walks three stores to assemble the body for one S6F11 frame. See chapter [13](13_e30_gem.md) for the wire flow. --- ## The single-threaded contract **Every store mutation runs on the io_context strand.** No locks, no atomics, no condition variables. This is documented in [`docs/INTEGRATION.md`](INTEGRATION.md) §3 and enforced under ThreadSanitizer. Why? Two reasons: 1. **Performance.** Locking a `std::map` for every SVID read is a waste in a hot path that processes thousands of messages a second. The asio strand model gives the same correctness guarantee for free. 2. **Simplicity.** Every method on every store is the obvious non-locking implementation. Reading the code, you don't have to track which lock protects what. The cost: **callers from other threads must `asio::post` onto the executor**. ```cpp // From a sensor thread: asio::post(io_context, [model, vid, value] { model->svids.set_value(vid, secs2::Item::f4(value)); }); ``` Tested by [`tests/test_thread_safety.cpp`](../tests/test_thread_safety.cpp) under TSan: N producer threads `asio::post` updates; TSan reports zero races. Chapter [33](33_transport.md) covers the strand model in more detail. --- ## How a store's API looks (a small one) Pick `AlarmRegistry` — one of the smallest: ```cpp class AlarmRegistry { public: // Register an alarm definition. void register_alarm(uint32_t alid, uint8_t alcd, const std::string& altx); // Set / clear an active alarm. Fires the change handler. void set(uint32_t alid); void clear(uint32_t alid); // Enable / disable host notification (S5F3). void set_enabled(uint32_t alid, bool enabled); bool is_enabled(uint32_t alid) const; // List active / all alarms. std::vector all() const; std::vector active() const; // Observer: change handler signature. using ChangeHandler = std::function; void set_change_handler(ChangeHandler); }; ``` Every store follows that same shape: mutator + reader + observer. The Router handler for `S5F1` doesn't fire `S5F1` itself — it mutates the store; the change handler (registered at startup by the EAP) fires `S5F1` via the connection. --- ## How a store's API looks (a bigger one) [`ProcessJobStore`](../include/secsgem/gem/store/process_jobs.hpp) adds: - Submit a PJ (record entry + fire `Created` event). - Get / set state of any PJ. - Apply a host-driven event (PJSTART / PJPAUSE / …) and route to the FSM. - Iterate active PJs (for serializing on restart). - Persistent journal: `enable_persistence(dir)`. The FSM logic isn't *inside* the store — `ProcessJobStateMachine` in [`process_job_state.hpp`](../include/secsgem/gem/process_job_state.hpp) owns transitions. The store holds one `ProcessJobStateMachine` per PJ and dispatches. This separation — *store* (records) vs *state machine* (transitions) — keeps each layer testable in isolation. --- ## Persistence Six stores have file-backed persistence: spool, process_jobs, control_jobs, exceptions, carriers, load_ports, substrates. Each opts in via `enable_persistence(dir)`: ```cpp model->process_jobs.enable_persistence("/var/lib/secsgem/pj"); ``` That: 1. Creates the directory if needed. 2. **Replays** every record file found there back into in-memory state. 3. Sets up the on-disk journal: every mutation writes (or rewrites, or deletes) one file per record, named by ID. Per-record-per-file means the journal is **partial-write safe**: if the equipment power-cycles mid-write of one record, the others are untouched; the partial file is detected and dropped at the next startup. Chapter [36](36_persistence_validation_metrics.md) walks the mechanism, the multi-version reads, and the test patterns. --- ## How to add a new store Two cases: ### Case 1: Standard already implemented, new sub-area E.g., add a "Reticle" store to track lithography reticles distinctly from substrates. 1. Create `include/secsgem/gem/store/reticles.hpp` with a class `ReticleStore` exposing the standard register / set-state / get / change-handler shape. 2. Add a member to `EquipmentDataModel`: ```cpp ReticleStore reticles; ``` 3. Write `tests/test_reticles.cpp` mirroring the pattern from any other store's test. 4. Wire Router handlers in `apps/secs_server.cpp` (or the EAP) for whatever S/F messages drive it. ### Case 2: Brand new SEMI standard E.g., implement E170 (a new GEM standard). Same as case 1, plus: 5. Update [`data/messages.yaml`](../data/messages.yaml) with any new S/F messages. `docker compose run --rm builder` regens `messages.hpp`. 6. If E170 has its own transition table, create `data/e170_state.yaml` and a `load_e170_state(...)` loader in `config::`. 7. Update [`docs/COMPLIANCE.md`](COMPLIANCE.md) with the new capability row. The architecture is **specifically designed** to add new standards without disturbing existing ones. --- ## Where to go next You've now seen how every per-domain data record is shaped and how `EquipmentDataModel` composes them. Next, we drop back down to transport: how `hsms::Connection` and `secsi::Protocol` actually move bytes, and the asio strand model that makes the single-threaded contract work. Next: [→ 33 Transport](33_transport.md)