secs-gem/docs/36_persistence_validation_metrics.md

36 — Persistence, validation, metrics

← 35 State machines and dispatch | Back to index | Next: 40 Building, running, the demo →

Three operational concerns wrap up Part 3:

• Persistence — file-backed journals for the seven stores that survive equipment restarts.
• Validation — the multi-error YAML validator behind --validate-config.
• Metrics — the Prometheus exporter.

Each is a small slice of the codebase but load-bearing for production deployments.

---

Persistence

Which stores persist

Seven of the 21 stores have file-backed journals:

Store	Survives equipment restart
SpoolStore	Queued messages waiting for host comm to come back.
ProcessJobStore	All in-progress PJs and their state machines.
ControlJobStore	All in-progress CJs.
ExceptionStore	Posted exceptions and their recovery state.
CarrierStore	Docked carriers + slot maps + access state.
LoadPortStore	Per-port association + reservation.
SubstrateStore	Per-substrate location + STS / SPS / ID status.

The remaining 14 stores (SVIDs, ECIDs, CEIDs, alarm registry, …) don't persist — their state is reconstructed from the YAML or from real-time signals on restart. An ECID that the host had changed would be lost on restart unless the EAP writes it back to the YAML (E40-style S2F15 is rare in production for exactly this reason).

The per-record file pattern

Every persistent store uses the same shape:

/var/lib/secsgem/<store>/
├── PJ-001          # one file per record
├── PJ-002
├── PJ-003
└── ...

One file per record, named by ID. When the store is mutated, the file is rewritten atomically (write to .tmp + rename). When the record is removed, the file is unlink'd.

This is partial-write safe. If the equipment power-cycles mid-write of one record, the others are untouched. At startup, the store iterates the directory, reads each file, and replays into in-memory state. A file that fails to parse (corrupted or unfinished) is dropped with a log line.

How a store enables persistence

// apps/secs_server.cpp — startup
auto model = std::make_shared<gem::EquipmentDataModel>();

if (!spool_dir.empty()) {
  model->spool.enable_persistence(spool_dir);
}
if (!pj_dir.empty()) {
  model->process_jobs.enable_persistence(pj_dir);
}
// ... etc per store

enable_persistence(dir):

1. Creates dir if needed.
2. Iterates files in dir.
3. For each file, reads + parses + adds the record to the store.
4. Sets up the on-disk journal for subsequent mutations.

The persistence is opt-in per store, configured via CLI flag in apps/secs_server.cpp. Some deployments want spool persistence but not job persistence (e.g., test rigs); the per-store toggle makes that easy.

File format and versioning

Each record file is a small binary blob:

magic:    4 bytes "SGv1" (store-specific magic; v1 = version 1)
version:  4 bytes (uint32_t, big-endian) — schema version
length:   4 bytes (uint32_t, big-endian) — payload length
payload:  N bytes — store-specific record encoding
checksum: 4 bytes (CRC-32C over header + payload)

Schema versioning is built in. Every store has a kVersion constant. When the store reads a file:

if (file_version > kVersion)
  drop the file (newer than us; can't read)
if (file_version < kVersion)
  apply the upgrade path (v1 → v2 → v3 reader chain)
if (file_version == kVersion)
  read directly

Multi-version reads let a new equipment release process old on-disk records without manual migration. Tested by tests/test_persistence_upgrade.cpp (7 cases — every store with persistence, write v1, restart at v2, verify replay).

Tests

Store	Test file	Cases
Spool	bundled into tests/test_data_model.cpp	—
Process Jobs	tests/test_job_persistence.cpp (PJ + CJ together)	7
Control Jobs	same	—
Exception	tests/test_exception_persistence.cpp	5
Carrier	tests/test_carrier_persistence.cpp	6
Substrate	tests/test_substrate_persistence.cpp	7
Upgrade path	tests/test_persistence_upgrade.cpp	7

Each persistence test covers: write a record, restart, verify replayed; partial-write recovery (truncated file dropped); remove deletes the file; corrupted file is dropped without throwing.

---

Validation

Why a separate validator

YAML loaders throw on first error. That's the right behaviour at process startup — fail fast — but it's frustrating for an operator with a fresh equipment.yaml that has three typos.

--validate-config is a separate CLI flag that:

1. Doesn't bind the port.
2. Tries to load every YAML.
3. Accumulates every issue (across files).
4. Prints them all.
5. Exits 0 or 1.

secs_server --validate-config \
    --config data/equipment.yaml \
    --state-table data/control_state.yaml \
    --pj-state-table data/process_job_state.yaml \
    --cj-state-table data/control_job_state.yaml

Typical output:

data/equipment.yaml:42: SVID 5 references undefined enum 'ChamberStateEnum'
data/equipment.yaml:78: alarm 3 has ALCD bit-7 cleared but alarm is declared 'active'
data/control_state.yaml:11: transition from OnlineRemote on host_request_remote has no `to` or `ack` field
data/equipment.yaml:104: host_command VENT references unknown CEID 999

4 error(s), 0 warning(s) across 4 files

Then exit 1.

How it's implemented

include/secsgem/config/validate.hpp:

class ConfigValidator {
 public:
  void validate_equipment(const std::string& path);
  void validate_control_state(const std::string& path);
  void validate_process_job_state(const std::string& path);
  void validate_control_job_state(const std::string& path);

  std::size_t error_count() const;
  std::size_t warning_count() const;
  bool has_errors() const;

  const std::vector<Issue>& issues() const;
  void format_issues_to(std::ostream&, FormatOptions = {}) const;
};

Each validate_* method:

1. Loads the YAML (catching parse errors as one issue).
2. Walks every record, applying structural + referential checks.
3. Adds each problem as an Issue{path, line, severity, message}.

Tests: tests/test_config_validate.cpp (8 cases — every category of issue: missing required field, typed mismatch, dangling reference, duplicate ID, …).

Reference checks across files

Cross-file references are validated last (after all files are parsed). Examples:

• host_commands[].emit_ceid must reference a CEID defined in equipment.yaml::ceids.
• events.default_reports[].vids must reference SVIDs or DVIDs defined elsewhere.
• control_state.yaml::transitions from/to must reference states declared by the schema (the 5 standard control states).

This catches "I deleted the CEID but forgot to update the host_command" before runtime.

---

Metrics

What gets exported

The codebase ships a Prometheus exporter (include/secsgem/metrics/prometheus.hpp) with two parts:

• Registry — accumulates Counter and Gauge series with labels.
• Server — exposes them on a configurable HTTP port at /metrics.

Typical wiring:

auto registry = std::make_shared<metrics::Registry>();
registry->register_metric("secsgem_ceid_emits_total", metrics::MetricType::Counter);
registry->register_metric("secsgem_spool_depth",      metrics::MetricType::Gauge);
registry->register_metric("secsgem_pj_state",         metrics::MetricType::Gauge);

// ...later, in the CEID-emit handler:
registry->counter("secsgem_ceid_emits_total", {{"ceid", std::to_string(ceid)}}).inc();

// ...periodically:
registry->gauge("secsgem_spool_depth").set(model->spool.size());

// Start the HTTP server:
auto exporter = std::make_shared<metrics::PrometheusServer>(io, /*port=*/9090, registry);

The exporter is wire-compatible with Prometheus scrape (text format). Tested by tests/test_metrics_prometheus.cpp (3 cases — counter increment, gauge set, HTTP scrape format).

What to expose

Common patterns from examples/pvd_tool/main.cpp §7:

• Per-CEID counters (secsgem_ceid_emits_total{ceid="300"}).
• Per-alarm counters (secsgem_alarm_set_total{alid="42"}).
• Spool depth gauge (alarm in operations if it climbs).
• Per-state EPT durations (sample of E116 buckets).
• T3 timeout counter (alarm in operations if non-zero).

The exporter doesn't dictate which metrics to expose — the EAP decides. See docs/INTEGRATION.md §6.4 for the production patterns.

---

End of Part 3

You now know every layer of the runtime:

• The repository layout (chapter 30).
• The spec-as-data philosophy + codegen (chapter 31).
• The stores + data model (chapter 32).
• The transport implementation (chapter 33).
• The codec + SML (chapter 34).
• Router + state machines + dispatch (chapter 35).
• Persistence + validation + metrics (this chapter).

Part 4 turns to operations — how a customer actually builds, runs, deploys, and integrates this codebase into a real fab tool.

Next: → 40 Building, running, the demo