secs-gem/docs/32_stores_and_the_data_model.md

32 — Stores and the data model

← 31 Spec-as-data + codegen | Back to index | Next: 33 Transport →

The previous chapter showed how YAML drives behaviour. This chapter shows the runtime data structures that the YAML populates and that the Router handlers operate on: stores and the EquipmentDataModel that composes them.

By the end you'll know:

• What a store is (and what it isn't).
• Every store in the codebase, one sentence each.
• How EquipmentDataModel composes them.
• The "no locks; single-threaded" contract.
• How to add a new store.

---

What a store is

A store is a per-domain bundle of:

• A few typed records (std::map, std::vector, …).
• A small API for reading + mutating them.
• A change_handler that emits events on transitions.
• Optional file-backed persistence.

The naming is consistent: AlarmRegistry for active alarms, CarrierStore for carriers, ProcessJobStore for PJs. Headers live in include/secsgem/gem/store/; implementations are typically inline in the header (these are small).

Each store maps onto one concern from one SEMI standard:

Store	Concern	Standard
StatusVariableStore	SVIDs + values	E30 §6.13
DataVariableStore	DVIDs + values	E30 §6.11
EquipmentConstantStore	ECIDs + values + min/max bounds	E30 §6.16
EventReportSubscriptions	RPTID definitions + CEID linkings + enables	E30 §6.6
AlarmRegistry	ALIDs + ALCD/ALTX + enable bits + active set	E30 §6.14
RecipeStore	PPIDs + PPBODY (unformatted) + formatted bodies	E30 §6.17 + E42
Clock	Wall-clock + drift + quality	E30 §6.20 + E148
HostCommandRegistry	RCMD names + per-command ack + side effects	E30 §6.15
SpoolStore	Per-stream whitelist + queue + persistent journal	E30 §6.22
LimitMonitorStore	LIMITIDs + upper/lower bounds + active state	E30 §6.21
TraceStore	TRIDs + active sampling config	E30 §6.12
ProcessJobStore	PJs + state + material list + persistent	E40
ControlJobStore	CJs + state + PJ refs + persistent	E94
ExceptionStore	EXIDs + recovery state + persistent	E5 §13
CarrierStore	Carrier IDs + state machines + persistent	E87
LoadPortStore	LP IDs + transfer/reservation/association FSMs	E87
SubstrateStore	Substrate IDs + 3 FSMs + location + persistent	E90
EptStateMachine	EPT state + time buckets	E116
CemObjectStore	E120 typed object hierarchy	E120
ModuleStore	Module IDs + state	E157
E84PortStore	Per-LP E84 FSM + signals + timers	E84

Each is a header. Each is independently testable: you can #include "secsgem/gem/store/alarms.hpp" and exercise AlarmRegistry without pulling in the rest. This is the same shape as the per-standard tests in tests/ — one test file per store.

---

EquipmentDataModel — the composite

include/secsgem/gem/data_model.hpp defines:

struct EquipmentDataModel {
  StatusVariableStore       svids;
  DataVariableStore         dvids;
  EquipmentConstantStore    ecids;
  EventReportSubscriptions  events;
  AlarmRegistry             alarms;
  RecipeStore               recipes;
  Clock                     clock;
  HostCommandRegistry       commands;
  SpoolStore                spool;
  LimitMonitorStore         limits;
  TraceStore                traces;
  ProcessJobStore           process_jobs;
  ControlJobStore           control_jobs;
  ExceptionStore            exceptions;
  CarrierStore              carriers;
  LoadPortStore             load_ports;
  SubstrateStore            substrates;
  EptStateMachine           ept;
  CemObjectStore            cem;
  ModuleStore               modules;
  E84PortStore              e84_ports;
  // ... convenience methods spanning stores
};

That's it. No locks, no smart pointers, no interfaces, no DI container. Each store is a value member; ownership is the EquipmentDataModel itself.

The application typically holds one shared_ptr<EquipmentDataModel> and passes it to every Router handler. Handlers operate on the stores directly:

router.on(1, 3, [model](const secs2::Message& m) {
  // S1F3 — host requests SVID values
  auto svids = parse_s1f3(m.body());
  return build_s1f4(model->svids.values(svids));
});

Convenience methods

EquipmentDataModel adds a few cross-store helpers (data_model.hpp:54):

std::optional<s2::Item> vid_value(uint32_t vid) const {
  // Look up VID in svids first, then dvids.
}

std::vector<ReportData> compose_reports_for(uint32_t ceid) const {
  // Walk events store -> reports store -> svids/dvids,
  // assemble the S6F11 report payload for one CEID firing.
}

compose_reports_for is the heart of event notification — it walks three stores to assemble the body for one S6F11 frame. See chapter 13 for the wire flow.

---

The single-threaded contract

Every store mutation runs on the io_context strand. No locks, no atomics, no condition variables. This is documented in docs/INTEGRATION.md §3 and enforced under ThreadSanitizer.

Why? Two reasons:

1. Performance. Locking a std::map for every SVID read is a waste in a hot path that processes thousands of messages a second. The asio strand model gives the same correctness guarantee for free.
2. Simplicity. Every method on every store is the obvious non-locking implementation. Reading the code, you don't have to track which lock protects what.

The cost: callers from other threads must asio::post onto the executor.

// From a sensor thread:
asio::post(io_context, [model, vid, value] {
  model->svids.set_value(vid, secs2::Item::f4(value));
});

Tested by tests/test_thread_safety.cpp under TSan: N producer threads asio::post updates; TSan reports zero races. Chapter 33 covers the strand model in more detail.

---

How a store's API looks (a small one)

Pick AlarmRegistry — one of the smallest:

class AlarmRegistry {
 public:
  // Register an alarm definition.
  void register_alarm(uint32_t alid, uint8_t alcd, const std::string& altx);

  // Set / clear an active alarm.  Fires the change handler.
  void set(uint32_t alid);
  void clear(uint32_t alid);

  // Enable / disable host notification (S5F3).
  void set_enabled(uint32_t alid, bool enabled);
  bool is_enabled(uint32_t alid) const;

  // List active / all alarms.
  std::vector<AlarmDefinition> all() const;
  std::vector<AlarmDefinition> active() const;

  // Observer: change handler signature.
  using ChangeHandler = std::function<void(uint32_t alid, bool set)>;
  void set_change_handler(ChangeHandler);
};

Every store follows that same shape: mutator + reader + observer. The Router handler for S5F1 doesn't fire S5F1 itself — it mutates the store; the change handler (registered at startup by the EAP) fires S5F1 via the connection.

---

How a store's API looks (a bigger one)

ProcessJobStore adds:

• Submit a PJ (record entry + fire Created event).
• Get / set state of any PJ.
• Apply a host-driven event (PJSTART / PJPAUSE / …) and route to the FSM.
• Iterate active PJs (for serializing on restart).
• Persistent journal: enable_persistence(dir).

The FSM logic isn't inside the store — ProcessJobStateMachine in process_job_state.hpp owns transitions. The store holds one ProcessJobStateMachine per PJ and dispatches.

This separation — store (records) vs state machine (transitions) — keeps each layer testable in isolation.

---

Persistence

Six stores have file-backed persistence: spool, process_jobs, control_jobs, exceptions, carriers, load_ports, substrates.

Each opts in via enable_persistence(dir):

model->process_jobs.enable_persistence("/var/lib/secsgem/pj");

That:

1. Creates the directory if needed.
2. Replays every record file found there back into in-memory state.
3. Sets up the on-disk journal: every mutation writes (or rewrites, or deletes) one file per record, named by ID.

Per-record-per-file means the journal is partial-write safe: if the equipment power-cycles mid-write of one record, the others are untouched; the partial file is detected and dropped at the next startup.

Chapter 36 walks the mechanism, the multi-version reads, and the test patterns.

---

How to add a new store

Two cases:

Case 1: Standard already implemented, new sub-area

E.g., add a "Reticle" store to track lithography reticles distinctly from substrates.

1. Create include/secsgem/gem/store/reticles.hpp with a class ReticleStore exposing the standard register / set-state / get / change-handler shape.
2. Add a member to EquipmentDataModel:

   ReticleStore reticles;
   

3. Write tests/test_reticles.cpp mirroring the pattern from any other store's test.
4. Wire Router handlers in apps/secs_server.cpp (or the EAP) for whatever S/F messages drive it.

Case 2: Brand new SEMI standard

E.g., implement E170 (a new GEM standard).

Same as case 1, plus:

5. Update data/messages.yaml with any new S/F messages. docker compose run --rm builder regens messages.hpp.
6. If E170 has its own transition table, create data/e170_state.yaml and a load_e170_state(...) loader in config::.
7. Update docs/COMPLIANCE.md with the new capability row.

The architecture is specifically designed to add new standards without disturbing existing ones.

---

Where to go next

You've now seen how every per-domain data record is shaped and how EquipmentDataModel composes them. Next, we drop back down to transport: how hsms::Connection and secsi::Protocol actually move bytes, and the asio strand model that makes the single-threaded contract work.

Next: → 33 Transport