secs-gem/docs/36_persistence_validation_metrics.md

# 36 — Persistence, validation, metrics

← [35 State machines and dispatch](35_state_machines_and_dispatch.md) | [Back to index](00_index.md) | Next: [40 Building, running, the demo](40_building_running_demo.md) →

Three operational concerns wrap up Part 3:

- **Persistence** — file-backed journals for the seven stores that
  survive equipment restarts.
- **Validation** — the multi-error YAML validator behind
  `--validate-config`.
- **Metrics** — the Prometheus exporter.

Each is a small slice of the codebase but load-bearing for
production deployments.

---

## Persistence

### Which stores persist

Seven of the 21 stores have file-backed journals:

| Store              | Survives equipment restart                                 |
|--------------------|------------------------------------------------------------|
| `SpoolStore`       | Queued messages waiting for host comm to come back.        |
| `ProcessJobStore`  | All in-progress PJs and their state machines.              |
| `ControlJobStore`  | All in-progress CJs.                                       |
| `ExceptionStore`   | Posted exceptions and their recovery state.                |
| `CarrierStore`     | Docked carriers + slot maps + access state.                |
| `LoadPortStore`    | Per-port association + reservation.                        |
| `SubstrateStore`   | Per-substrate location + STS / SPS / ID status.            |

The remaining 14 stores (SVIDs, ECIDs, CEIDs, alarm registry, …)
don't persist — their state is reconstructed from the YAML or
from real-time signals on restart.  An ECID that the host had
changed *would* be lost on restart unless the EAP writes it back
to the YAML (E40-style `S2F15` is rare in production for
exactly this reason).

### The per-record file pattern

Every persistent store uses the same shape:

```
/var/lib/secsgem/<store>/
├── PJ-001          # one file per record
├── PJ-002
├── PJ-003
└── ...
```

One file per record, named by ID.  When the store is mutated, the
file is rewritten atomically (write to `.tmp` + `rename`).  When
the record is removed, the file is `unlink`'d.

**This is partial-write safe.**  If the equipment power-cycles
mid-write of one record, the others are untouched.  At startup,
the store iterates the directory, reads each file, and replays
into in-memory state.  A file that fails to parse (corrupted or
unfinished) is dropped with a log line.

### How a store enables persistence

```cpp
// apps/secs_server.cpp — startup
auto model = std::make_shared<gem::EquipmentDataModel>();

if (!spool_dir.empty()) {
  model->spool.enable_persistence(spool_dir);
}
if (!pj_dir.empty()) {
  model->process_jobs.enable_persistence(pj_dir);
}
// ... etc per store
```

`enable_persistence(dir)`:

1. Creates `dir` if needed.
2. Iterates files in `dir`.
3. For each file, reads + parses + adds the record to the store.
4. Sets up the on-disk journal for subsequent mutations.

The persistence is **opt-in per store**, configured via CLI flag in
`apps/secs_server.cpp`.  Some deployments want spool persistence
but not job persistence (e.g., test rigs); the per-store toggle
makes that easy.

### File format and versioning

Each record file is a small binary blob:

```
magic:    4 bytes "SGv1" (store-specific magic; v1 = version 1)
version:  4 bytes (uint32_t, big-endian) — schema version
length:   4 bytes (uint32_t, big-endian) — payload length
payload:  N bytes — store-specific record encoding
checksum: 4 bytes (CRC-32C over header + payload)
```

**Schema versioning** is built in.  Every store has a `kVersion`
constant.  When the store reads a file:

```cpp
if (file_version > kVersion)
  drop the file (newer than us; can't read)
if (file_version < kVersion)
  apply the upgrade path (v1 → v2 → v3 reader chain)
if (file_version == kVersion)
  read directly
```

Multi-version reads let a new equipment release process old
on-disk records without manual migration.  Tested by
[`tests/test_persistence_upgrade.cpp`](../tests/test_persistence_upgrade.cpp)
(7 cases — every store with persistence, write v1, restart at
v2, verify replay).

### Tests

| Store               | Test file                                          | Cases |
|---------------------|----------------------------------------------------|------:|
| Spool               | bundled into `tests/test_data_model.cpp`           | —     |
| Process Jobs        | `tests/test_job_persistence.cpp` (PJ + CJ together)|  7    |
| Control Jobs        | same                                               | —     |
| Exception           | `tests/test_exception_persistence.cpp`             |  5    |
| Carrier             | `tests/test_carrier_persistence.cpp`               |  6    |
| Substrate           | `tests/test_substrate_persistence.cpp`             |  7    |
| Upgrade path        | `tests/test_persistence_upgrade.cpp`               |  7    |

Each persistence test covers: write a record, restart, verify
replayed; partial-write recovery (truncated file dropped); remove
deletes the file; corrupted file is dropped without throwing.

---

## Validation

### Why a separate validator

YAML loaders throw on first error.  That's the right behaviour
at process startup — fail fast — but it's frustrating for an
operator with a fresh equipment.yaml that has three typos.

`--validate-config` is a separate CLI flag that:

1. Doesn't bind the port.
2. Tries to load every YAML.
3. Accumulates *every* issue (across files).
4. Prints them all.
5. Exits 0 or 1.

```bash
secs_server --validate-config \
    --config data/equipment.yaml \
    --state-table data/control_state.yaml \
    --pj-state-table data/process_job_state.yaml \
    --cj-state-table data/control_job_state.yaml
```

Typical output:

```
data/equipment.yaml:42: SVID 5 references undefined enum 'ChamberStateEnum'
data/equipment.yaml:78: alarm 3 has ALCD bit-7 cleared but alarm is declared 'active'
data/control_state.yaml:11: transition from OnlineRemote on host_request_remote has no `to` or `ack` field
data/equipment.yaml:104: host_command VENT references unknown CEID 999

4 error(s), 0 warning(s) across 4 files
```

Then exit 1.

### How it's implemented

[`include/secsgem/config/validate.hpp`](../include/secsgem/config/validate.hpp):

```cpp
class ConfigValidator {
 public:
  void validate_equipment(const std::string& path);
  void validate_control_state(const std::string& path);
  void validate_process_job_state(const std::string& path);
  void validate_control_job_state(const std::string& path);

  std::size_t error_count() const;
  std::size_t warning_count() const;
  bool has_errors() const;

  const std::vector<Issue>& issues() const;
  void format_issues_to(std::ostream&, FormatOptions = {}) const;
};
```

Each `validate_*` method:

1. Loads the YAML (catching parse errors as one issue).
2. Walks every record, applying structural + referential checks.
3. Adds each problem as an `Issue{path, line, severity, message}`.

Tests:
[`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp)
(8 cases — every category of issue: missing required field,
typed mismatch, dangling reference, duplicate ID, …).

### Reference checks across files

Cross-file references are validated last (after all files are
parsed).  Examples:

- `host_commands[].emit_ceid` must reference a CEID defined in
  `equipment.yaml::ceids`.
- `events.default_reports[].vids` must reference SVIDs or DVIDs
  defined elsewhere.
- `control_state.yaml::transitions` `from`/`to` must reference
  states declared by the schema (the 5 standard control states).

This catches "I deleted the CEID but forgot to update the
host_command" before runtime.

---

## Metrics

### What gets exported

The codebase ships a Prometheus exporter
([`include/secsgem/metrics/prometheus.hpp`](../include/secsgem/metrics/prometheus.hpp))
with two parts:

- **Registry** — accumulates `Counter` and `Gauge` series with
  labels.
- **Server** — exposes them on a configurable HTTP port at
  `/metrics`.

Typical wiring:

```cpp
auto registry = std::make_shared<metrics::Registry>();
registry->register_metric("secsgem_ceid_emits_total", metrics::MetricType::Counter);
registry->register_metric("secsgem_spool_depth",      metrics::MetricType::Gauge);
registry->register_metric("secsgem_pj_state",         metrics::MetricType::Gauge);

// ...later, in the CEID-emit handler:
registry->counter("secsgem_ceid_emits_total", {{"ceid", std::to_string(ceid)}}).inc();

// ...periodically:
registry->gauge("secsgem_spool_depth").set(model->spool.size());

// Start the HTTP server:
auto exporter = std::make_shared<metrics::PrometheusServer>(io, /*port=*/9090, registry);
```

The exporter is wire-compatible with Prometheus scrape (text
format).  Tested by
[`tests/test_metrics_prometheus.cpp`](../tests/test_metrics_prometheus.cpp)
(3 cases — counter increment, gauge set, HTTP scrape format).

### What to expose

Common patterns from
[`examples/pvd_tool/main.cpp`](../examples/pvd_tool/main.cpp) §7:

- Per-CEID counters (`secsgem_ceid_emits_total{ceid="300"}`).
- Per-alarm counters (`secsgem_alarm_set_total{alid="42"}`).
- Spool depth gauge (alarm in operations if it climbs).
- Per-state EPT durations (sample of E116 buckets).
- T3 timeout counter (alarm in operations if non-zero).

The exporter doesn't dictate which metrics to expose — the EAP
decides.  See
[`docs/INTEGRATION.md`](INTEGRATION.md) §6.4 for the production
patterns.

---

## End of Part 3

You now know every layer of the runtime:

- The repository layout (chapter 30).
- The spec-as-data philosophy + codegen (chapter 31).
- The stores + data model (chapter 32).
- The transport implementation (chapter 33).
- The codec + SML (chapter 34).
- Router + state machines + dispatch (chapter 35).
- Persistence + validation + metrics (this chapter).

Part 4 turns to operations — how a customer actually builds, runs,
deploys, and integrates this codebase into a real fab tool.

Next: [→ 40 Building, running, the demo](40_building_running_demo.md)