docs: chapters 30–36 — the codebase (Part 3 complete)
Seven chapters walking the implementation top-to-bottom. 30 — Repository tour. Top-level layout, directory by directory. The eight built binaries. The dependency graph from TCP socket up through EquipmentDataModel. CMake's role. Test layout. 31 — Spec-as-data and codegen. Why the design choice fits SECS/ GEM specifically. The five YAML files: messages catalog, control/PJ/CJ transition tables, equipment dictionary. How tools/gen_messages.py turns messages.yaml into typed C++ at build time. The --validate-config multi-error validator. How to add a new SVID / CEID / host command / state / message without C++. 32 — Stores and the data model. What a store IS (records + API + change handler + optional persistence). Every store in the codebase mapped to the SEMI standard it serves (table of 21). EquipmentDataModel as plain composition + cross-store convenience methods (vid_value, compose_reports_for). The no-locks single- threaded contract. How to add a new store. 33 — Transport. hsms::Connection read path (length+payload async chain), write path (queue + one outstanding write), timer model (5 steady_timers + per-request T3). The asio executor / strand model and why it's the right shape. secsi::Protocol as the IO- free FSM with Action / Event variants; secsi::TcpTransport as the asio adapter. Pattern repeats for E84 + GEM comm-state. 34 — Codec and SML. The four files (170 + 30 + 52 + 32 lines of header, 229 + 220 lines of impl). Item variant storage layout (11 alternatives, 16 formats, shared storage where E5 permits). encode_into recursion; decode_at with bounds checks throwing CodecError. Message wrapper. SML printer + try_parse_sml + why SML round-trips Items but not necessarily bytes. 35 — State machines and dispatch. gem::Router as a typed (stream, function) dispatch table. How an S2F41 round-trip walks through parser → store dispatch → side-effect → CEID emission → S6F11 build → spool-aware deliver. The 11 FSMs all sharing the same three-property shape (pure data table + pure FSM + observer pattern). CEID cascading from FSM transitions to wire bytes. 36 — Persistence, validation, metrics. Which 7 stores have file journals + why the others don't. Per-record file pattern (atomic rename, partial-write safe). Schema versioning + multi-version read. Multi-error YAML validator (--validate-config) + cross-file reference checks. Prometheus registry + HTTP exporter + worked metric patterns from the PVD example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,293 @@
|
||||
# 36 — Persistence, validation, metrics
|
||||
|
||||
← [35 State machines and dispatch](35_state_machines_and_dispatch.md) | [Back to index](00_index.md) | Next: [40 Building, running, the demo](40_building_running_demo.md) →
|
||||
|
||||
Three operational concerns wrap up Part 3:
|
||||
|
||||
- **Persistence** — file-backed journals for the seven stores that
|
||||
survive equipment restarts.
|
||||
- **Validation** — the multi-error YAML validator behind
|
||||
`--validate-config`.
|
||||
- **Metrics** — the Prometheus exporter.
|
||||
|
||||
Each is a small slice of the codebase but load-bearing for
|
||||
production deployments.
|
||||
|
||||
---
|
||||
|
||||
## Persistence
|
||||
|
||||
### Which stores persist
|
||||
|
||||
Seven of the 21 stores have file-backed journals:
|
||||
|
||||
| Store | Survives equipment restart |
|
||||
|--------------------|------------------------------------------------------------|
|
||||
| `SpoolStore` | Queued messages waiting for host comm to come back. |
|
||||
| `ProcessJobStore` | All in-progress PJs and their state machines. |
|
||||
| `ControlJobStore` | All in-progress CJs. |
|
||||
| `ExceptionStore` | Posted exceptions and their recovery state. |
|
||||
| `CarrierStore` | Docked carriers + slot maps + access state. |
|
||||
| `LoadPortStore` | Per-port association + reservation. |
|
||||
| `SubstrateStore` | Per-substrate location + STS / SPS / ID status. |
|
||||
|
||||
The remaining 14 stores (SVIDs, ECIDs, CEIDs, alarm registry, …)
|
||||
don't persist — their state is reconstructed from the YAML or
|
||||
from real-time signals on restart. An ECID that the host had
|
||||
changed *would* be lost on restart unless the EAP writes it back
|
||||
to the YAML (E40-style `S2F15` is rare in production for
|
||||
exactly this reason).
|
||||
|
||||
### The per-record file pattern
|
||||
|
||||
Every persistent store uses the same shape:
|
||||
|
||||
```
|
||||
/var/lib/secsgem/<store>/
|
||||
├── PJ-001 # one file per record
|
||||
├── PJ-002
|
||||
├── PJ-003
|
||||
└── ...
|
||||
```
|
||||
|
||||
One file per record, named by ID. When the store is mutated, the
|
||||
file is rewritten atomically (write to `.tmp` + `rename`). When
|
||||
the record is removed, the file is `unlink`'d.
|
||||
|
||||
**This is partial-write safe.** If the equipment power-cycles
|
||||
mid-write of one record, the others are untouched. At startup,
|
||||
the store iterates the directory, reads each file, and replays
|
||||
into in-memory state. A file that fails to parse (corrupted or
|
||||
unfinished) is dropped with a log line.
|
||||
|
||||
### How a store enables persistence
|
||||
|
||||
```cpp
|
||||
// apps/secs_server.cpp — startup
|
||||
auto model = std::make_shared<gem::EquipmentDataModel>();
|
||||
|
||||
if (!spool_dir.empty()) {
|
||||
model->spool.enable_persistence(spool_dir);
|
||||
}
|
||||
if (!pj_dir.empty()) {
|
||||
model->process_jobs.enable_persistence(pj_dir);
|
||||
}
|
||||
// ... etc per store
|
||||
```
|
||||
|
||||
`enable_persistence(dir)`:
|
||||
|
||||
1. Creates `dir` if needed.
|
||||
2. Iterates files in `dir`.
|
||||
3. For each file, reads + parses + adds the record to the store.
|
||||
4. Sets up the on-disk journal for subsequent mutations.
|
||||
|
||||
The persistence is **opt-in per store**, configured via CLI flag in
|
||||
`apps/secs_server.cpp`. Some deployments want spool persistence
|
||||
but not job persistence (e.g., test rigs); the per-store toggle
|
||||
makes that easy.
|
||||
|
||||
### File format and versioning
|
||||
|
||||
Each record file is a small binary blob:
|
||||
|
||||
```
|
||||
magic: 4 bytes "SGv1" (store-specific magic; v1 = version 1)
|
||||
version: 4 bytes (uint32_t, big-endian) — schema version
|
||||
length: 4 bytes (uint32_t, big-endian) — payload length
|
||||
payload: N bytes — store-specific record encoding
|
||||
checksum: 4 bytes (CRC-32C over header + payload)
|
||||
```
|
||||
|
||||
**Schema versioning** is built in. Every store has a `kVersion`
|
||||
constant. When the store reads a file:
|
||||
|
||||
```cpp
|
||||
if (file_version > kVersion)
|
||||
drop the file (newer than us; can't read)
|
||||
if (file_version < kVersion)
|
||||
apply the upgrade path (v1 → v2 → v3 reader chain)
|
||||
if (file_version == kVersion)
|
||||
read directly
|
||||
```
|
||||
|
||||
Multi-version reads let a new equipment release process old
|
||||
on-disk records without manual migration. Tested by
|
||||
[`tests/test_persistence_upgrade.cpp`](../tests/test_persistence_upgrade.cpp)
|
||||
(7 cases — every store with persistence, write v1, restart at
|
||||
v2, verify replay).
|
||||
|
||||
### Tests
|
||||
|
||||
| Store | Test file | Cases |
|
||||
|---------------------|----------------------------------------------------|------:|
|
||||
| Spool | bundled into `tests/test_data_model.cpp` | — |
|
||||
| Process Jobs | `tests/test_job_persistence.cpp` (PJ + CJ together)| 7 |
|
||||
| Control Jobs | same | — |
|
||||
| Exception | `tests/test_exception_persistence.cpp` | 5 |
|
||||
| Carrier | `tests/test_carrier_persistence.cpp` | 6 |
|
||||
| Substrate | `tests/test_substrate_persistence.cpp` | 7 |
|
||||
| Upgrade path | `tests/test_persistence_upgrade.cpp` | 7 |
|
||||
|
||||
Each persistence test covers: write a record, restart, verify
|
||||
replayed; partial-write recovery (truncated file dropped); remove
|
||||
deletes the file; corrupted file is dropped without throwing.
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
### Why a separate validator
|
||||
|
||||
YAML loaders throw on first error. That's the right behaviour
|
||||
at process startup — fail fast — but it's frustrating for an
|
||||
operator with a fresh equipment.yaml that has three typos.
|
||||
|
||||
`--validate-config` is a separate CLI flag that:
|
||||
|
||||
1. Doesn't bind the port.
|
||||
2. Tries to load every YAML.
|
||||
3. Accumulates *every* issue (across files).
|
||||
4. Prints them all.
|
||||
5. Exits 0 or 1.
|
||||
|
||||
```bash
|
||||
secs_server --validate-config \
|
||||
--config data/equipment.yaml \
|
||||
--state-table data/control_state.yaml \
|
||||
--pj-state-table data/process_job_state.yaml \
|
||||
--cj-state-table data/control_job_state.yaml
|
||||
```
|
||||
|
||||
Typical output:
|
||||
|
||||
```
|
||||
data/equipment.yaml:42: SVID 5 references undefined enum 'ChamberStateEnum'
|
||||
data/equipment.yaml:78: alarm 3 has ALCD bit-7 cleared but alarm is declared 'active'
|
||||
data/control_state.yaml:11: transition from OnlineRemote on host_request_remote has no `to` or `ack` field
|
||||
data/equipment.yaml:104: host_command VENT references unknown CEID 999
|
||||
|
||||
4 error(s), 0 warning(s) across 4 files
|
||||
```
|
||||
|
||||
Then exit 1.
|
||||
|
||||
### How it's implemented
|
||||
|
||||
[`include/secsgem/config/validate.hpp`](../include/secsgem/config/validate.hpp):
|
||||
|
||||
```cpp
|
||||
class ConfigValidator {
|
||||
public:
|
||||
void validate_equipment(const std::string& path);
|
||||
void validate_control_state(const std::string& path);
|
||||
void validate_process_job_state(const std::string& path);
|
||||
void validate_control_job_state(const std::string& path);
|
||||
|
||||
std::size_t error_count() const;
|
||||
std::size_t warning_count() const;
|
||||
bool has_errors() const;
|
||||
|
||||
const std::vector<Issue>& issues() const;
|
||||
void format_issues_to(std::ostream&, FormatOptions = {}) const;
|
||||
};
|
||||
```
|
||||
|
||||
Each `validate_*` method:
|
||||
|
||||
1. Loads the YAML (catching parse errors as one issue).
|
||||
2. Walks every record, applying structural + referential checks.
|
||||
3. Adds each problem as an `Issue{path, line, severity, message}`.
|
||||
|
||||
Tests:
|
||||
[`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp)
|
||||
(8 cases — every category of issue: missing required field,
|
||||
typed mismatch, dangling reference, duplicate ID, …).
|
||||
|
||||
### Reference checks across files
|
||||
|
||||
Cross-file references are validated last (after all files are
|
||||
parsed). Examples:
|
||||
|
||||
- `host_commands[].emit_ceid` must reference a CEID defined in
|
||||
`equipment.yaml::ceids`.
|
||||
- `events.default_reports[].vids` must reference SVIDs or DVIDs
|
||||
defined elsewhere.
|
||||
- `control_state.yaml::transitions` `from`/`to` must reference
|
||||
states declared by the schema (the 5 standard control states).
|
||||
|
||||
This catches "I deleted the CEID but forgot to update the
|
||||
host_command" before runtime.
|
||||
|
||||
---
|
||||
|
||||
## Metrics
|
||||
|
||||
### What gets exported
|
||||
|
||||
The codebase ships a Prometheus exporter
|
||||
([`include/secsgem/metrics/prometheus.hpp`](../include/secsgem/metrics/prometheus.hpp))
|
||||
with two parts:
|
||||
|
||||
- **Registry** — accumulates `Counter` and `Gauge` series with
|
||||
labels.
|
||||
- **Server** — exposes them on a configurable HTTP port at
|
||||
`/metrics`.
|
||||
|
||||
Typical wiring:
|
||||
|
||||
```cpp
|
||||
auto registry = std::make_shared<metrics::Registry>();
|
||||
registry->register_metric("secsgem_ceid_emits_total", metrics::MetricType::Counter);
|
||||
registry->register_metric("secsgem_spool_depth", metrics::MetricType::Gauge);
|
||||
registry->register_metric("secsgem_pj_state", metrics::MetricType::Gauge);
|
||||
|
||||
// ...later, in the CEID-emit handler:
|
||||
registry->counter("secsgem_ceid_emits_total", {{"ceid", std::to_string(ceid)}}).inc();
|
||||
|
||||
// ...periodically:
|
||||
registry->gauge("secsgem_spool_depth").set(model->spool.size());
|
||||
|
||||
// Start the HTTP server:
|
||||
auto exporter = std::make_shared<metrics::PrometheusServer>(io, /*port=*/9090, registry);
|
||||
```
|
||||
|
||||
The exporter is wire-compatible with Prometheus scrape (text
|
||||
format). Tested by
|
||||
[`tests/test_metrics_prometheus.cpp`](../tests/test_metrics_prometheus.cpp)
|
||||
(3 cases — counter increment, gauge set, HTTP scrape format).
|
||||
|
||||
### What to expose
|
||||
|
||||
Common patterns from
|
||||
[`examples/pvd_tool/main.cpp`](../examples/pvd_tool/main.cpp) §7:
|
||||
|
||||
- Per-CEID counters (`secsgem_ceid_emits_total{ceid="300"}`).
|
||||
- Per-alarm counters (`secsgem_alarm_set_total{alid="42"}`).
|
||||
- Spool depth gauge (alarm in operations if it climbs).
|
||||
- Per-state EPT durations (sample of E116 buckets).
|
||||
- T3 timeout counter (alarm in operations if non-zero).
|
||||
|
||||
The exporter doesn't dictate which metrics to expose — the EAP
|
||||
decides. See
|
||||
[`docs/INTEGRATION.md`](INTEGRATION.md) §6.4 for the production
|
||||
patterns.
|
||||
|
||||
---
|
||||
|
||||
## End of Part 3
|
||||
|
||||
You now know every layer of the runtime:
|
||||
|
||||
- The repository layout (chapter 30).
|
||||
- The spec-as-data philosophy + codegen (chapter 31).
|
||||
- The stores + data model (chapter 32).
|
||||
- The transport implementation (chapter 33).
|
||||
- The codec + SML (chapter 34).
|
||||
- Router + state machines + dispatch (chapter 35).
|
||||
- Persistence + validation + metrics (this chapter).
|
||||
|
||||
Part 4 turns to operations — how a customer actually builds, runs,
|
||||
deploys, and integrates this codebase into a real fab tool.
|
||||
|
||||
Next: [→ 40 Building, running, the demo](40_building_running_demo.md)
|
||||
Reference in New Issue
Block a user