docs: chapters 30–36 — the codebase (Part 3 complete)
Seven chapters walking the implementation top-to-bottom. 30 — Repository tour. Top-level layout, directory by directory. The eight built binaries. The dependency graph from TCP socket up through EquipmentDataModel. CMake's role. Test layout. 31 — Spec-as-data and codegen. Why the design choice fits SECS/ GEM specifically. The five YAML files: messages catalog, control/PJ/CJ transition tables, equipment dictionary. How tools/gen_messages.py turns messages.yaml into typed C++ at build time. The --validate-config multi-error validator. How to add a new SVID / CEID / host command / state / message without C++. 32 — Stores and the data model. What a store IS (records + API + change handler + optional persistence). Every store in the codebase mapped to the SEMI standard it serves (table of 21). EquipmentDataModel as plain composition + cross-store convenience methods (vid_value, compose_reports_for). The no-locks single- threaded contract. How to add a new store. 33 — Transport. hsms::Connection read path (length+payload async chain), write path (queue + one outstanding write), timer model (5 steady_timers + per-request T3). The asio executor / strand model and why it's the right shape. secsi::Protocol as the IO- free FSM with Action / Event variants; secsi::TcpTransport as the asio adapter. Pattern repeats for E84 + GEM comm-state. 34 — Codec and SML. The four files (170 + 30 + 52 + 32 lines of header, 229 + 220 lines of impl). Item variant storage layout (11 alternatives, 16 formats, shared storage where E5 permits). encode_into recursion; decode_at with bounds checks throwing CodecError. Message wrapper. SML printer + try_parse_sml + why SML round-trips Items but not necessarily bytes. 35 — State machines and dispatch. gem::Router as a typed (stream, function) dispatch table. How an S2F41 round-trip walks through parser → store dispatch → side-effect → CEID emission → S6F11 build → spool-aware deliver. The 11 FSMs all sharing the same three-property shape (pure data table + pure FSM + observer pattern). CEID cascading from FSM transitions to wire bytes. 36 — Persistence, validation, metrics. Which 7 stores have file journals + why the others don't. Per-record file pattern (atomic rename, partial-write safe). Schema versioning + multi-version read. Multi-error YAML validator (--validate-config) + cross-file reference checks. Prometheus registry + HTTP exporter + worked metric patterns from the PVD example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,308 @@
|
||||
# 32 — Stores and the data model
|
||||
|
||||
← [31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md) | [Back to index](00_index.md) | Next: [33 Transport](33_transport.md) →
|
||||
|
||||
The previous chapter showed how YAML drives behaviour. This
|
||||
chapter shows the runtime data structures that the YAML populates
|
||||
and that the Router handlers operate on: **stores** and the
|
||||
**`EquipmentDataModel`** that composes them.
|
||||
|
||||
By the end you'll know:
|
||||
|
||||
- What a **store** is (and what it isn't).
|
||||
- Every store in the codebase, one sentence each.
|
||||
- How `EquipmentDataModel` composes them.
|
||||
- The "no locks; single-threaded" contract.
|
||||
- How to add a new store.
|
||||
|
||||
---
|
||||
|
||||
## What a store is
|
||||
|
||||
A **store** is a per-domain bundle of:
|
||||
|
||||
- A few typed records (`std::map`, `std::vector`, …).
|
||||
- A small API for reading + mutating them.
|
||||
- A `change_handler` that emits events on transitions.
|
||||
- Optional file-backed persistence.
|
||||
|
||||
The naming is consistent: `AlarmRegistry` for active alarms,
|
||||
`CarrierStore` for carriers, `ProcessJobStore` for PJs.
|
||||
Headers live in
|
||||
[`include/secsgem/gem/store/`](../include/secsgem/gem/store/);
|
||||
implementations are typically inline in the header (these are small).
|
||||
|
||||
Each store maps onto **one concern from one SEMI standard**:
|
||||
|
||||
| Store | Concern | Standard |
|
||||
|-----------------------------|--------------------------------------------------|----------------|
|
||||
| `StatusVariableStore` | SVIDs + values | E30 §6.13 |
|
||||
| `DataVariableStore` | DVIDs + values | E30 §6.11 |
|
||||
| `EquipmentConstantStore` | ECIDs + values + min/max bounds | E30 §6.16 |
|
||||
| `EventReportSubscriptions` | RPTID definitions + CEID linkings + enables | E30 §6.6 |
|
||||
| `AlarmRegistry` | ALIDs + ALCD/ALTX + enable bits + active set | E30 §6.14 |
|
||||
| `RecipeStore` | PPIDs + PPBODY (unformatted) + formatted bodies | E30 §6.17 + E42|
|
||||
| `Clock` | Wall-clock + drift + quality | E30 §6.20 + E148 |
|
||||
| `HostCommandRegistry` | RCMD names + per-command ack + side effects | E30 §6.15 |
|
||||
| `SpoolStore` | Per-stream whitelist + queue + persistent journal| E30 §6.22 |
|
||||
| `LimitMonitorStore` | LIMITIDs + upper/lower bounds + active state | E30 §6.21 |
|
||||
| `TraceStore` | TRIDs + active sampling config | E30 §6.12 |
|
||||
| `ProcessJobStore` | PJs + state + material list + persistent | E40 |
|
||||
| `ControlJobStore` | CJs + state + PJ refs + persistent | E94 |
|
||||
| `ExceptionStore` | EXIDs + recovery state + persistent | E5 §13 |
|
||||
| `CarrierStore` | Carrier IDs + state machines + persistent | E87 |
|
||||
| `LoadPortStore` | LP IDs + transfer/reservation/association FSMs | E87 |
|
||||
| `SubstrateStore` | Substrate IDs + 3 FSMs + location + persistent | E90 |
|
||||
| `EptStateMachine` | EPT state + time buckets | E116 |
|
||||
| `CemObjectStore` | E120 typed object hierarchy | E120 |
|
||||
| `ModuleStore` | Module IDs + state | E157 |
|
||||
| `E84PortStore` | Per-LP E84 FSM + signals + timers | E84 |
|
||||
|
||||
Each is a header. Each is independently testable: you can
|
||||
`#include "secsgem/gem/store/alarms.hpp"` and exercise
|
||||
`AlarmRegistry` without pulling in the rest. This is the same
|
||||
shape as the per-standard tests in
|
||||
[`tests/`](../tests/) — one test file per store.
|
||||
|
||||
---
|
||||
|
||||
## EquipmentDataModel — the composite
|
||||
|
||||
[`include/secsgem/gem/data_model.hpp`](../include/secsgem/gem/data_model.hpp)
|
||||
defines:
|
||||
|
||||
```cpp
|
||||
struct EquipmentDataModel {
|
||||
StatusVariableStore svids;
|
||||
DataVariableStore dvids;
|
||||
EquipmentConstantStore ecids;
|
||||
EventReportSubscriptions events;
|
||||
AlarmRegistry alarms;
|
||||
RecipeStore recipes;
|
||||
Clock clock;
|
||||
HostCommandRegistry commands;
|
||||
SpoolStore spool;
|
||||
LimitMonitorStore limits;
|
||||
TraceStore traces;
|
||||
ProcessJobStore process_jobs;
|
||||
ControlJobStore control_jobs;
|
||||
ExceptionStore exceptions;
|
||||
CarrierStore carriers;
|
||||
LoadPortStore load_ports;
|
||||
SubstrateStore substrates;
|
||||
EptStateMachine ept;
|
||||
CemObjectStore cem;
|
||||
ModuleStore modules;
|
||||
E84PortStore e84_ports;
|
||||
// ... convenience methods spanning stores
|
||||
};
|
||||
```
|
||||
|
||||
That's it. No locks, no smart pointers, no interfaces, no DI
|
||||
container. Each store is a value member; ownership is the
|
||||
`EquipmentDataModel` itself.
|
||||
|
||||
The application typically holds one `shared_ptr<EquipmentDataModel>`
|
||||
and passes it to every Router handler. Handlers operate on the
|
||||
stores directly:
|
||||
|
||||
```cpp
|
||||
router.on(1, 3, [model](const secs2::Message& m) {
|
||||
// S1F3 — host requests SVID values
|
||||
auto svids = parse_s1f3(m.body());
|
||||
return build_s1f4(model->svids.values(svids));
|
||||
});
|
||||
```
|
||||
|
||||
### Convenience methods
|
||||
|
||||
`EquipmentDataModel` adds a few cross-store helpers (`data_model.hpp:54`):
|
||||
|
||||
```cpp
|
||||
std::optional<s2::Item> vid_value(uint32_t vid) const {
|
||||
// Look up VID in svids first, then dvids.
|
||||
}
|
||||
|
||||
std::vector<ReportData> compose_reports_for(uint32_t ceid) const {
|
||||
// Walk events store -> reports store -> svids/dvids,
|
||||
// assemble the S6F11 report payload for one CEID firing.
|
||||
}
|
||||
```
|
||||
|
||||
`compose_reports_for` is the *heart* of event notification — it
|
||||
walks three stores to assemble the body for one S6F11 frame. See
|
||||
chapter [13](13_e30_gem.md) for the wire flow.
|
||||
|
||||
---
|
||||
|
||||
## The single-threaded contract
|
||||
|
||||
**Every store mutation runs on the io_context strand.** No locks,
|
||||
no atomics, no condition variables. This is documented in
|
||||
[`docs/INTEGRATION.md`](INTEGRATION.md) §3 and enforced under
|
||||
ThreadSanitizer.
|
||||
|
||||
Why? Two reasons:
|
||||
|
||||
1. **Performance.** Locking a `std::map` for every SVID read is a
|
||||
waste in a hot path that processes thousands of messages a
|
||||
second. The asio strand model gives the same correctness
|
||||
guarantee for free.
|
||||
2. **Simplicity.** Every method on every store is the obvious
|
||||
non-locking implementation. Reading the code, you don't have
|
||||
to track which lock protects what.
|
||||
|
||||
The cost: **callers from other threads must `asio::post` onto
|
||||
the executor**.
|
||||
|
||||
```cpp
|
||||
// From a sensor thread:
|
||||
asio::post(io_context, [model, vid, value] {
|
||||
model->svids.set_value(vid, secs2::Item::f4(value));
|
||||
});
|
||||
```
|
||||
|
||||
Tested by
|
||||
[`tests/test_thread_safety.cpp`](../tests/test_thread_safety.cpp)
|
||||
under TSan: N producer threads `asio::post` updates; TSan reports
|
||||
zero races. Chapter [33](33_transport.md) covers the strand model
|
||||
in more detail.
|
||||
|
||||
---
|
||||
|
||||
## How a store's API looks (a small one)
|
||||
|
||||
Pick `AlarmRegistry` — one of the smallest:
|
||||
|
||||
```cpp
|
||||
class AlarmRegistry {
|
||||
public:
|
||||
// Register an alarm definition.
|
||||
void register_alarm(uint32_t alid, uint8_t alcd, const std::string& altx);
|
||||
|
||||
// Set / clear an active alarm. Fires the change handler.
|
||||
void set(uint32_t alid);
|
||||
void clear(uint32_t alid);
|
||||
|
||||
// Enable / disable host notification (S5F3).
|
||||
void set_enabled(uint32_t alid, bool enabled);
|
||||
bool is_enabled(uint32_t alid) const;
|
||||
|
||||
// List active / all alarms.
|
||||
std::vector<AlarmDefinition> all() const;
|
||||
std::vector<AlarmDefinition> active() const;
|
||||
|
||||
// Observer: change handler signature.
|
||||
using ChangeHandler = std::function<void(uint32_t alid, bool set)>;
|
||||
void set_change_handler(ChangeHandler);
|
||||
};
|
||||
```
|
||||
|
||||
Every store follows that same shape: mutator + reader + observer.
|
||||
The Router handler for `S5F1` doesn't fire `S5F1` itself — it
|
||||
mutates the store; the change handler (registered at startup by
|
||||
the EAP) fires `S5F1` via the connection.
|
||||
|
||||
---
|
||||
|
||||
## How a store's API looks (a bigger one)
|
||||
|
||||
[`ProcessJobStore`](../include/secsgem/gem/store/process_jobs.hpp)
|
||||
adds:
|
||||
|
||||
- Submit a PJ (record entry + fire `Created` event).
|
||||
- Get / set state of any PJ.
|
||||
- Apply a host-driven event (PJSTART / PJPAUSE / …) and route to
|
||||
the FSM.
|
||||
- Iterate active PJs (for serializing on restart).
|
||||
- Persistent journal: `enable_persistence(dir)`.
|
||||
|
||||
The FSM logic isn't *inside* the store — `ProcessJobStateMachine`
|
||||
in [`process_job_state.hpp`](../include/secsgem/gem/process_job_state.hpp)
|
||||
owns transitions. The store holds one `ProcessJobStateMachine`
|
||||
per PJ and dispatches.
|
||||
|
||||
This separation — *store* (records) vs *state machine* (transitions) —
|
||||
keeps each layer testable in isolation.
|
||||
|
||||
---
|
||||
|
||||
## Persistence
|
||||
|
||||
Six stores have file-backed persistence: spool, process_jobs,
|
||||
control_jobs, exceptions, carriers, load_ports, substrates.
|
||||
|
||||
Each opts in via `enable_persistence(dir)`:
|
||||
|
||||
```cpp
|
||||
model->process_jobs.enable_persistence("/var/lib/secsgem/pj");
|
||||
```
|
||||
|
||||
That:
|
||||
|
||||
1. Creates the directory if needed.
|
||||
2. **Replays** every record file found there back into in-memory
|
||||
state.
|
||||
3. Sets up the on-disk journal: every mutation writes (or rewrites,
|
||||
or deletes) one file per record, named by ID.
|
||||
|
||||
Per-record-per-file means the journal is **partial-write safe**:
|
||||
if the equipment power-cycles mid-write of one record, the others
|
||||
are untouched; the partial file is detected and dropped at the
|
||||
next startup.
|
||||
|
||||
Chapter [36](36_persistence_validation_metrics.md) walks the
|
||||
mechanism, the multi-version reads, and the test patterns.
|
||||
|
||||
---
|
||||
|
||||
## How to add a new store
|
||||
|
||||
Two cases:
|
||||
|
||||
### Case 1: Standard already implemented, new sub-area
|
||||
|
||||
E.g., add a "Reticle" store to track lithography reticles
|
||||
distinctly from substrates.
|
||||
|
||||
1. Create `include/secsgem/gem/store/reticles.hpp` with a class
|
||||
`ReticleStore` exposing the standard
|
||||
register / set-state / get / change-handler shape.
|
||||
2. Add a member to `EquipmentDataModel`:
|
||||
```cpp
|
||||
ReticleStore reticles;
|
||||
```
|
||||
3. Write `tests/test_reticles.cpp` mirroring the pattern from any
|
||||
other store's test.
|
||||
4. Wire Router handlers in `apps/secs_server.cpp` (or the EAP) for
|
||||
whatever S/F messages drive it.
|
||||
|
||||
### Case 2: Brand new SEMI standard
|
||||
|
||||
E.g., implement E170 (a new GEM standard).
|
||||
|
||||
Same as case 1, plus:
|
||||
|
||||
5. Update [`data/messages.yaml`](../data/messages.yaml) with any
|
||||
new S/F messages. `docker compose run --rm builder` regens
|
||||
`messages.hpp`.
|
||||
6. If E170 has its own transition table, create
|
||||
`data/e170_state.yaml` and a `load_e170_state(...)` loader in
|
||||
`config::`.
|
||||
7. Update [`docs/COMPLIANCE.md`](COMPLIANCE.md) with the new
|
||||
capability row.
|
||||
|
||||
The architecture is **specifically designed** to add new standards
|
||||
without disturbing existing ones.
|
||||
|
||||
---
|
||||
|
||||
## Where to go next
|
||||
|
||||
You've now seen how every per-domain data record is shaped and
|
||||
how `EquipmentDataModel` composes them. Next, we drop back down
|
||||
to transport: how `hsms::Connection` and `secsi::Protocol` actually
|
||||
move bytes, and the asio strand model that makes the
|
||||
single-threaded contract work.
|
||||
|
||||
Next: [→ 33 Transport](33_transport.md)
|
||||
Reference in New Issue
Block a user