From 31f908e1bfa21902f216027aae2169183a98306e Mon Sep 17 00:00:00 2001 From: Raphael Maenle Date: Tue, 9 Jun 2026 20:28:21 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20chapters=2040,=2041,=2050,=2051=20?= =?UTF-8?q?=E2=80=94=20Operations=20+=20Reference=20(series=20complete)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Last four chapters of the guided tour. 40 — Building, running, the demo. Docker prerequisites, the build flow, what each binary is for, running the 24-transaction demo flow annotated step by step. Running the 4 external-validator sweeps + the libFuzzer pass. Inspecting the demo with tcpdump and tshark. Reading source while running as the recommended learning workflow. 41 — Integration: hardware, MES, production. Four-phase tour: wiring sensors / recipe engine / alarms / E84 GPIO; talking to a real MES with the day-1 punch list + commercial-MES quirks (Wonderware S2F21, Camstar Linktest cadence, etc.); production hardening (nftables / stunnel / minisign / persistence layout / monitoring / runbook); performance envelope + memory footprint + capacity planning. Pointers to the long-form INTEGRATION.md / MES_INTEROP.md / SECURITY.md / BENCHMARKS.md. 50 — API + message catalog + YAML schemas reference. Namespace-by- namespace table of public symbols (secs2, hsms, secsi, gem, config, metrics) with brief descriptions. Stream-by-stream message catalog reference (S1, S2, S3, S5, S6, S7, S9, S10, S12, S14, S16). YAML schema reference for messages.yaml + the three state-table files + equipment.yaml. 51 — Extending the codebase. Seven recipes ordered from no-code to substantial: new SVID/DVID/ECID (YAML only), new CEID with reports (YAML only), new host command (YAML + optional handler), new control- state transition (YAML only), new SECS-II message (YAML + handler), new store (header + tests), new persistence backend (drop-in vs pluggable trade-off). Each recipe has the actual mechanical steps, the test pattern, and pointers to the chapter that explains why it works. Index updated to mark all 24 chapters published. Co-Authored-By: Claude Opus 4.7 --- docs/00_index.md | 10 +- docs/40_building_running_demo.md | 376 +++++++++++++++ .../41_integration_hardware_mes_production.md | 350 ++++++++++++++ docs/50_api_messages_yaml_reference.md | 345 ++++++++++++++ docs/51_extending_the_codebase.md | 439 ++++++++++++++++++ 5 files changed, 1513 insertions(+), 7 deletions(-) create mode 100644 docs/40_building_running_demo.md create mode 100644 docs/41_integration_hardware_mes_production.md create mode 100644 docs/50_api_messages_yaml_reference.md create mode 100644 docs/51_extending_the_codebase.md diff --git a/docs/00_index.md b/docs/00_index.md index c5a2f60..9822917 100644 --- a/docs/00_index.md +++ b/docs/00_index.md @@ -248,12 +248,8 @@ this guide is the *tutorial path* that ties them together. ## Status of this guide -Chapters publish as they're written. The list above is the table of -contents; individual files exist once the chapter has been written. -A chapter without a working link is on the to-write list. - -**Currently published:** Chapter 00 (this index). - -**In progress:** Chapter 01 — *What is SECS/GEM?* +**All 24 chapters published.** Read linearly from +[01](01_what_is_secs_gem.md) or jump in at whichever part fits your +goal (see "How to read this guide" above). Next chapter: [→ 01 What is SECS/GEM?](01_what_is_secs_gem.md) diff --git a/docs/40_building_running_demo.md b/docs/40_building_running_demo.md new file mode 100644 index 0000000..3746241 --- /dev/null +++ b/docs/40_building_running_demo.md @@ -0,0 +1,376 @@ +# 40 — Building, running, the demo + +← [36 Persistence, validation, metrics](36_persistence_validation_metrics.md) | [Back to index](00_index.md) | Next: [41 Integration: hardware, MES, production](41_integration_hardware_mes_production.md) → + +You've read about every layer of the codebase. Now we run it. + +This chapter is operational: build the project, start the demo, +walk what each transaction in the two-container flow actually does +and where it lives. By the end you'll have the demo running on +your laptop and you'll know what every log line means. + +--- + +## Prerequisites + +Just **Docker**. No host C++ toolchain, no Python deps, nothing +to apt-install. The toolchain image (`Dockerfile`) bundles Ubuntu +24.04 + g++-13 + CMake + Ninja + asio + yaml-cpp + Python 3 + +tshark + tcpdump + clang. + +```bash +docker --version +docker compose version +``` + +If both work, you're set. + +--- + +## Building + +```bash +docker compose run --rm builder +``` + +That: + +1. Pulls / builds the toolchain image (first time only, ~3 + minutes). +2. Runs `cmake -S /app -B /app/build -G Ninja -DCMAKE_BUILD_TYPE=Release`. +3. Runs `cmake --build /app/build`. +4. Produces every binary under `/app/build/` inside a named Docker + volume. + +Subsequent builds are incremental and take ~10–30 s. + +### What got built + +``` +build/ +├── secs_server passive equipment (the demo target) +├── secs_client active host (drives the demo) +├── secs_conformance 47-check conformance harness +├── secs_interop_probe active host probing secsgem-py equipment +├── secs_bench throughput/latency bench +├── secsgem_tests the 445-case doctest binary +└── pvd_tool worked PVD-tool example +``` + +Plus the generated `build/generated/secsgem/gem/messages.hpp` +(~3 500 lines, auto-derived from `data/messages.yaml`). + +--- + +## Running the tests + +```bash +docker compose run --rm tests +``` + +Runs `secsgem_tests` end-to-end. Expected output: + +``` +[doctest] doctest version is "2.4.11" +[doctest] run with "--help" for options +=============================================================================== +[doctest] test cases: 445 | 445 passed | 0 failed | 0 skipped +[doctest] assertions: 2753 | 2753 passed | 0 failed | +[doctest] Status: SUCCESS! +``` + +On a 2024 M-series Mac under Docker Desktop, this takes ~3.5 s. + +--- + +## The two-container demo + +```bash +docker compose up --no-deps server client +``` + +That starts: + +- A **`server`** container running `secs_server` on port 5000. +- A **`client`** container running `secs_client` against `server:5000`. + +The client drives ~24 SECS transactions through the data model. +Each transaction logs on both sides. + +### What each transaction does + +Annotated walk through the log output: + +#### Communication establishment + +``` +[host] connecting to server:5000 +[equip] accepted connection +[host] sending Select.req +[equip] Select.req received → SELECTED +[host] Select.rsp(Ok) received → SELECTED +``` + +HSMS SELECT handshake. Both sides now in SELECTED state. + +``` +[host] sending S1F13 Establish Communications +[equip] S1F13 received +[equip] sending S1F14(COMMACK=Accept, [MDLN, SOFTREV]) +[host] S1F14 received → COMMUNICATING +``` + +E30 §6.5 communication-state transition. Now GEM-level +communication is up. + +#### Identification + +``` +[host] S1F1 Are You There +[equip] S1F2 ["SECS-GEM Demo Equipment", "1.0.0"] +[host] S1F19 GEM Compliance Request +[equip] S1F20 [list of capabilities] +[host] S1F11 SVID Namelist (all) +[equip] S1F12 [SVID 1 "ControlState", SVID 2 "Clock", ...] +[host] S1F21 DVID Namelist (all) +[equip] S1F22 [DVID list] +[host] S1F23 CEID Namelist (all) +[equip] S1F24 [CEID → VID mapping] +``` + +Host walks the data dictionary. + +#### Dynamic event report setup + +``` +[host] S2F33 DefineReport(RPTID=1, VIDs=[SVID 2]) +[equip] S2F34(DRACK=0) +[host] S2F35 LinkEvent(CEID=300 → [RPTID=1]) +[equip] S2F36(LRACK=0) +[host] S2F37 EnableEvent(CEED=true, CEIDs=[300]) +[equip] S2F38(ERACK=0) +``` + +The three-message report wiring. CEID 300 now triggers an S6F11 +when it fires. + +#### Control state + remote command + +``` +[host] S2F41 RCMD=START +[equip] S2F42(HCACK=Accept) +[equip] HostCommandRegistry dispatched START +[equip] → emit CEID 300 +[equip] → compose_reports_for(300) → RPTID 1 = [Clock SV2] +[equip] → fire S6F11 +[equip] S6F11(CEID=300, [RPTID=1, [Clock]]) +[host] S6F12(ACKC6=0) +``` + +Host command dispatch + event report emission + acknowledgement. +This is the canonical GEM transaction. + +#### Alarms + +``` +[host] S5F5 List all alarms +[equip] S5F6 [ALID list with ALCD + ALTX] +[host] S5F3 EnableAlarm(ALID=1) +[equip] S5F4(ACKC5=0) +[host] S2F41 RCMD=FAULT +[equip] S2F42(HCACK=Accept) +[equip] → set ALID 1 +[equip] → fire S5F1(ALCD=0x84, ALID=1) +[equip] S5F1(...) +[host] S5F2(ACKC5=0) +``` + +#### Recipes + +``` +[host] S7F1 PP Load Inquire(PPID="NEW-RECIPE", LENGTH=64) +[equip] S7F2(PPGNT=0=Permit) +[host] S7F3 PP Send(PPID="NEW-RECIPE", PPBODY=) +[equip] S7F4(ACKC7=0) +[host] S7F5 PP Request(PPID="NEW-RECIPE") +[equip] S7F6 [PPID, PPBODY] +[host] S7F17 PP Delete(PPIDs=["NEW-RECIPE"]) +[equip] S7F18(ACKC7=0) +``` + +#### Terminal display + +``` +[host] S10F3 Terminal Display Multi (TID=0, TEXT="hello\nfrom host") +[equip] S10F4(ACKC10=0) +``` + +#### Clean shutdown + +``` +[host] S1F15 Request Offline +[equip] S1F16(OFLACK=Accept) +[host] sending Separate.req +[equip] Separate.req received → close +``` + +Total: 24 transactions exercising S1, S2, S5, S6, S7, S10. + +--- + +## Running the conformance harness + +```bash +docker compose up -d server +docker compose run --rm builder /app/build/secs_conformance --host server --port 5000 +docker compose down +``` + +Runs the 47-check conformance harness against the demo server. +Each check covers one E30 / GEM 300 wire-level behaviour: + +``` +[PASS] E37 §7.2 SELECT handshake +[PASS] E30 §6.5 S1F13/F14 Establish Comms +[PASS] E30 §6.7 S1F1/F2 Are You There +... (43 more) +[PASS] E30 §6.10 S1F19/F20 GEM Compliance + + 47 / 47 checks passed +``` + +This is proof #2 in [`docs/PROOFS.md`](PROOFS.md). + +--- + +## Running the interop sweeps + +### secsgem-py + +```bash +docker compose up -d server +docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py --host server +docker compose down +``` + +The Python `secsgem-py` 0.3.0 host drives our equipment. 31 checks +across S1/S2/S5/S6/S7/S10 + unsolicited S6F11 / S5F1. + +### secs4java8 + +```bash +bash interop/secs4j_validate.sh +``` + +The Java secs4java8 host drives our equipment via a separate +container. 55 checks covering S1/S2/S3/S5/S6/S7/S10/S14/S16 +including the GEM 300 streams that secsgem-py couldn't easily +drive. + +### tshark dissector + +```bash +docker compose run --rm builder bash /app/interop/tshark_validate.sh +``` + +Captures a pcap of the demo flow, dissects with Wireshark's HSMS +dissector, asserts no malformed packets. 69 frames, 0 errors. + +### libFuzzer (60 s, requires clang) + +```bash +docker compose run --rm builder bash -c " + cmake -S /app -B /app/build-fuzz -G Ninja -DSECSGEM_FUZZ=ON \ + -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ + cmake --build /app/build-fuzz + /app/build-fuzz/fuzz_secs2_decode -max_total_time=60 + /app/build-fuzz/fuzz_sml_parse -max_total_time=60 +" +``` + +200 k+ inputs through `secs2::decode`, 1.4 M+ through +`try_parse_sml`, ASan + UBSan clean, 0 crashes. + +All five sweeps are wired into CI; see +[`.gitea/workflows/ci.yml`](../.gitea/workflows/ci.yml). + +--- + +## Inspecting the demo from outside + +While the demo is running, you can: + +### Watch the wire + +```bash +# In another shell: +docker compose exec server tcpdump -i any -A -s 0 'tcp port 5000' +``` + +### Inspect with tshark + HSMS dissector + +```bash +docker compose run --rm builder tshark -i any -d "tcp.port==5000,hsms" -V \ + | grep -A 2 "Header" +``` + +### Watch the metrics + +`pvd_tool` example exposes a Prometheus endpoint: + +```bash +docker run --rm -p 9090:9090 pvd_tool /app/examples/pvd_tool/equipment.yaml \ + /app/data/control_state.yaml 5000 9090 +``` + +Then `curl localhost:9090/metrics`. + +--- + +## Running the bench + +```bash +docker compose run --rm builder /app/build/secs_bench \ + --requests 50000 --concurrency 32 --svid-count 32 +``` + +Outputs a markdown table of throughput + p50/p95/p99 latencies for: + +- S1F1/F2 (header-only round-trip). +- S1F3/F4 with 32 SVIDs. +- S6F11 push (W=0, fire-and-forget). +- PJ + CJ memory footprint. + +See [`docs/BENCHMARKS.md`](BENCHMARKS.md) for the baseline numbers +and capacity-planning notes. + +--- + +## Reading the source while it runs + +A common workflow when you're learning: + +1. `docker compose up --no-deps server client` in one shell. +2. Source viewer open in another (your IDE on the host — + the source isn't bind-mounted in the container, but it is + on your host). +3. Find a log line that confuses you (e.g. `[equip] S6F11 fired`). +4. Grep the source for it. Most log strings are unique enough to + land in the right file in one search. +5. Read the function around it. +6. Cross-reference back to the chapter that covers the standard. + +This is the most efficient way to internalise the codebase. The +demo runs forever (until you `Ctrl-C` — the client loops); you +can read the source at your own pace. + +--- + +## Where to go next + +You now have the demo running and you can drive any of the five +external validators. The next chapter is the **integration** +chapter — wiring the runtime to real hardware, talking to a real +MES, production deployment, security, performance tuning. + +Next: [→ 41 Integration: hardware, MES, production](41_integration_hardware_mes_production.md) diff --git a/docs/41_integration_hardware_mes_production.md b/docs/41_integration_hardware_mes_production.md new file mode 100644 index 0000000..c11d6c1 --- /dev/null +++ b/docs/41_integration_hardware_mes_production.md @@ -0,0 +1,350 @@ +# 41 — Integration: hardware, MES, production + +← [40 Building, running, the demo](40_building_running_demo.md) | [Back to index](00_index.md) | Next: [50 API + messages + YAML reference](50_api_messages_yaml_reference.md) → + +You have the demo running. Now you need to make it talk to a +**real tool**, against a **real MES**, in a **real fab**. + +This chapter walks the four phases of that journey: + +1. **Wire the EAP to physical hardware.** +2. **Integrate with a commercial MES.** +3. **Production hardening** — security, monitoring, persistence. +4. **Operational concerns** — performance, capacity, incidents. + +This is a compressed view of the long-form +[`docs/INTEGRATION.md`](INTEGRATION.md); cross-references are +inline. The long-form has more code and more configuration; this +chapter explains the *shape* of each phase. + +--- + +## Phase 1 — wiring to hardware + +### What "the EAP" actually does + +The EAP (Equipment Automation Program) sits between the SECS/GEM +runtime and the **physical tool**. It does four things: + +1. **Reads sensors** at the right cadence and updates SVIDs. +2. **Drives the recipe engine** when a host command arrives. +3. **Listens for alarms** from PLCs / hardware fault lines. +4. **Wires FSM transitions** to CEID emissions. + +[`examples/pvd_tool/main.cpp`](../examples/pvd_tool/main.cpp) is +the worked reference. Section by section: + +| Section in main.cpp | What it shows | +|-------------------------------|------------------------------------------------------------------------| +| §1 Helpers + constants | The `kSvidX / kCeidX` constants worth pinning at file scope | +| §2 Sensor simulator | Multi-cadence sensor poll loops with `asio::post` strand-marshal | +| §3 Recipe runner | PJ → SettingUp → Processing → ProcessComplete walk; per-step CEID emit | +| §4 Alarm threshold monitor | Continuous threshold evaluation against ECID setpoints | +| §5 EPT cycling | E116 transitions driven by PJ state + safety alarms | +| §6 Router handlers | 51 handlers in ~460 lines — every S/F a host might send to a PVD tool | +| §7 main() | YAML load → validate → compose → run | + +A real tool fork: + +```bash +cp -r examples/pvd_tool/ src/my_tool/ +# edit src/my_tool/equipment.yaml — your tool's SVIDs/CEIDs/alarms +# edit src/my_tool/main.cpp — replace pvd::Simulator with PLC bindings +``` + +### Sensor wiring + +The PVD example uses a random-walk simulator (§2). A real EAP +replaces this with calls into the tool's sensor stack: + +```cpp +// Original (simulated): +upd_f4(kSvidChamberPressure, target_pressure.load(), 1e-8f, 1e-7f); + +// Real: +asio::post(io, [model](){ + float pressure = plc_read_register(0x4001); // from your PLC API + model->svids.set_value(kSvidChamberPressure, secs2::Item::f4(pressure)); +}); +``` + +The `asio::post` is non-negotiable — the store mutation runs on +the io_context strand (chapter 33). + +### Recipe runner + +The PVD example's recipe runner (§3) parses the recipe body and +walks PJ states. A real tool replaces the simulator with calls +into the tool's recipe engine: + +```cpp +void start_processing(const std::string& pjid, const std::string& ppid) { + auto recipe = recipes_->find(ppid); + if (!recipe) { + model->process_jobs.apply(pjid, ProcessJobEvent::Abort); + return; + } + // Hand the recipe to the tool's actual recipe engine. + hardware_recipe_engine_->start(*recipe, [model, pjid](bool ok) { + asio::post(io, [model, pjid, ok] { + model->process_jobs.apply(pjid, + ok ? ProcessJobEvent::ProcessComplete + : ProcessJobEvent::AbortComplete); + }); + }); +} +``` + +### Alarm sources + +Real alarms come from: + +- **PLC fault lines** — interrupt callbacks. +- **Watchdog timers** — periodic checks (cooling water flow, + vacuum pressure). +- **Sensor thresholds** — continuous evaluation against ECIDs. +- **Hardware safety interlocks** — SafetyController callbacks. + +Each translates to one `model->alarms.set(alid)` call. The alarm +dispatcher takes care of S5F1 emission, host enable filtering, and +alarm persistence. + +### E84 wiring + +E84 needs a GPIO driver: + +```cpp +// On signal change from the GPIO driver: +void on_gpio_change(uint8_t port, E84Signal sig, bool value) { + asio::post(io, [model, port, sig, value]() { + model->e84_ports.at(port).fsm.on_signal_change(sig, value); + }); +} + +// When the FSM wants to assert a signal: +model->e84_ports.at(port).fsm.set_emit_handler( + [port](E84Signal sig, bool value) { + gpio_driver_write(port, sig, value); + }); +``` + +The TA1/TA2/TA3 timers are wall-clock; use the +[`E84AsioTimers`](../include/secsgem/gem/e84_asio_timers.hpp) +adapter so they fire on the same io_context. + +--- + +## Phase 2 — talking to a real MES + +### The day-1 punch list + +Before you connect to a production MES, run +[`docs/MES_INTEROP.md`](MES_INTEROP.md) against the **staging** +MES. 59 test IDs across: + +- Transport (T-01 to T-09): SELECT, Linktest, T3, T7, oversized + frames. +- Establishment (E-01 to E-08): S1F13, S1F1, S1F11, S1F19, … +- Reports (R-01 to R-07): the S2F33/F35/F37 dance. +- Alarms (A-01 to A-06). +- Commands (C-01 to C-04): S2F41 + S2F21 (legacy) + S2F49 + (enhanced). +- Recipes (P-01 to P-05). +- Terminal services (TS-01 to TS-03). +- Jobs (J-01 to J-06). +- Spool (SP-01 to SP-05). +- Clock (K-01 to K-06). + +This is the **friction-killer document**. Pass every test ID in +staging and your production cutover is much less likely to +surprise you. + +### HSMS-GS for multi-MES + +Some fabs run multiple MES against one equipment. E37 §11 +HSMS-GS multiplexes over one TCP socket: + +```cpp +auto conn = std::make_shared( + std::move(sock), Mode::Passive, /*primary device_id=*/0, timers); + +// Production MES on session 100. +conn->add_session(100); +conn->set_session_message_handler(100, production_router_handler); + +// Maintenance MES on session 200. +conn->add_session(200); +conn->set_session_message_handler(200, maintenance_router_handler); +``` + +[`docs/INTEGRATION.md`](INTEGRATION.md) §7 has the full worked +example with HA pattern. Tests: +[`tests/test_hsms_gs.cpp`](../tests/test_hsms_gs.cpp) (5 wire-level) +and +[`tests/test_hsms_gs_integration.cpp`](../tests/test_hsms_gs_integration.cpp) +(1 end-to-end three-session scenario). + +### Things commercial MES get wrong + +Real MES exhibit common deviations: + +- **Wonderware uses S2F21 (legacy) exclusively** — no S2F41. The + codebase's HostCommandRegistry handles both forms. +- **Some MES leave EQPTYP in S1F20 confused with MDLN** — the + codebase accepts either; documented in + [`docs/MES_INTEROP.md`](MES_INTEROP.md) E-02 caveat. +- **MES with old PPBODY handling reject binary recipes** — the + codebase ships both as-bytes and as-ASCII PPBODY. +- **Camstar uses Linktest at 30 s**, others at 60 s — configure + `Timers::linktest` to match the host's cadence. + +[`docs/MES_INTEROP.md`](MES_INTEROP.md) "Caveats" column lists more. + +--- + +## Phase 3 — production hardening + +### Security: SECURITY.md walk-through + +[`docs/SECURITY.md`](SECURITY.md) ships concrete configs for: + +- **nftables** — restrict the SECS port to the MES host's IP only. +- **stunnel** — wrap the HSMS port in TLS so the wire is + encrypted. +- **minisign** — sign every recipe (PPBODY) and verify on receive. +- **SIEM audit log schema** — what every store mutation emits as + JSON for log aggregation. + +Configure all four before promoting to production. No exceptions. + +### Persistence layout + +Production deployments enable persistence on every store that +needs it. Per [`docs/INTEGRATION.md`](INTEGRATION.md) §5: + +``` +/var/lib/secsgem/ +├── spool/ # SpoolStore +├── pj/ # ProcessJobStore +├── cj/ # ControlJobStore +├── exceptions/ # ExceptionStore +├── carriers/ # CarrierStore +├── load_ports/ # LoadPortStore +└── substrates/ # SubstrateStore +``` + +On an SSD with `fsync` enabled per file rewrite, this comfortably +handles a few hundred mutations / sec. On rotational media you'll +want to batch or relax durability. + +### Monitoring + +Production EAPs typically export: + +- **Per-CEID emission counter** — burst detection. +- **Spool depth gauge** — alarms when growing (MES connectivity + problem). +- **T3 timeout counter** — non-zero means the MES is slow or your + T3 is too short. +- **Per-alarm set count** — pages on certain ALIDs. +- **Equipment EPT state gauge** — fab-wide dashboard input. + +[`docs/INTEGRATION.md`](INTEGRATION.md) §6.4 covers the Grafana +panel patterns; the PVD example wires the Prometheus exporter at +§7. + +### Operational runbook + +[`README.md`](../README.md) ships a starter runbook: + +| Incident | First check | Mitigation | +|-------------------------------------|--------------------------------------|-------------------------------------------| +| HSMS connection flapping | T7 / T6 timer fires in logs | check MES reachability, network MTU | +| Spool depth growing | host MES connectivity / ACK rate | force-drain via S6F23, escalate to MES | +| State machine "stuck" | last state-change handler log line | host-issued offline + re-establish | +| Alarm storm | `AlarmRegistry::all()` snapshot | check upstream sensor; quench via S5F3 | +| Persistence dir growing unbounded | `du -s` + file count | sweep terminal-state records | +| Cross-tool inconsistency | `secsgem_tests` on canary tool | compare wire trace vs validator | + +--- + +## Phase 4 — performance + +### The envelope + +Per [`docs/BENCHMARKS.md`](BENCHMARKS.md), on a 2024 M-series Mac +under Docker Desktop: + +| Scenario | Ops/sec | p50 µs | p95 µs | p99 µs | +|-----------------------------------|--------:|--------:|-------:|-------:| +| S1F1/F2 (header-only) | ~140 k | 74 | 103 | 161 | +| S1F3/F4 (32 SVIDs) | ~79 k | 165 | 186 | 260 | +| S6F11 push (W=0) | ~572 k | n/a | n/a | n/a | + +A real fab tool sees **tens to a few hundred** events / s +sustained. We're three orders of magnitude above the push path, +two orders above the round-trip path. + +**Throughput is not the bottleneck**; tail latency under +contention is. Tune by: + +- Running on a quiet host. +- Bumping `linktest` interval up (default 0 = disabled is fine for + most deployments). +- Pinning the io_context thread to a dedicated CPU. + +### Memory footprint + +| Entity | Approx bytes / instance | +|---------------------|------------------------:| +| PJ + CJ pair | ~450 | +| Carrier (no slots) | ~80 | +| Carrier slot | ~24 | +| Substrate | ~120 | +| Spool entry | ~40 + encoded body size | + +A busy 300 mm tool with 50 carriers × 25 slots + 200 substrates + +20 active PJ+CJ pairs is under **1 MiB** of model state. RSS is +dominated by the binary itself + asio's buffers (~10–20 MiB). + +### Capacity planning + +For sizing purposes: + +- One io_context thread per `Connection` is plenty for any single + tool. +- Multiple tools share the same process if you want — one + io_context, multiple `Connection`s, each on its own strand. +- Persistence cost is one `rename(2)` per mutation; SSD-bound + fabs comfortably handle a few hundred / sec. + +--- + +## When to read the long-form + +This chapter compressed phases 1–4 into a tour. Each phase has +substantially more material in the long-form docs: + +- [`docs/INTEGRATION.md`](INTEGRATION.md) — full vendor-side + tutorial including wiring sensors, plugging FSMs, persistence + layout, monitoring, HSMS-GS HA. +- [`docs/MES_INTEROP.md`](MES_INTEROP.md) — the 59 test IDs in + full. +- [`docs/SECURITY.md`](SECURITY.md) — concrete nftables / stunnel / + minisign configs. +- [`docs/BENCHMARKS.md`](BENCHMARKS.md) — perf envelope + how to + re-run. + +When you actually start an integration, work from those. This +chapter is the map. + +--- + +## End of Part 4 + +You can now build the codebase, run the demo, drive every +external validator, and you know the shape of how a real +integration would land. Part 5 is reference material — API +namespaces, message catalog reference, the extension recipes. + +Next: [→ 50 API + messages + YAML reference](50_api_messages_yaml_reference.md) diff --git a/docs/50_api_messages_yaml_reference.md b/docs/50_api_messages_yaml_reference.md new file mode 100644 index 0000000..67e2437 --- /dev/null +++ b/docs/50_api_messages_yaml_reference.md @@ -0,0 +1,345 @@ +# 50 — API + message catalog + YAML schemas + +← [41 Integration: hardware, MES, production](41_integration_hardware_mes_production.md) | [Back to index](00_index.md) | Next: [51 Extending the codebase](51_extending_the_codebase.md) → + +This chapter is **reference**, not tutorial. Look up the namespace +or YAML key you need; cross-reference the code. + +The whole codebase is small enough that "go read the header" is +often the right answer — this chapter helps you find which header. + +--- + +## Namespace reference + +### `secsgem::secs2` — codec (chapter 34) + +```cpp +#include "secsgem/secs2/item.hpp" +#include "secsgem/secs2/codec.hpp" +#include "secsgem/secs2/message.hpp" +#include "secsgem/secs2/sml.hpp" +``` + +| Symbol | What it is | +|-------------------------------------------------|----------------------------------------------------| +| `enum class Format` | 16 SECS-II format codes. | +| `class Item` | Variant-based value type. | +| `class Message` | Stream + function + W-bit + system_bytes + body. | +| `class CodecError` | Thrown on malformed input. | +| `std::vector encode(const Item&)` | Serialize an Item to bytes. | +| `void encode_into(const Item&, std::vector&)` | Append-encode into existing buffer. | +| `Item decode(const std::vector&)` | Decode one Item from a complete buffer. | +| `Item decode_at(const uint8_t*, size_t, size_t&)`| Decode one Item from a position; advances cursor. | +| `std::string to_sml(const Item&)` | Render as SML. | +| `std::optional try_parse_sml(const std::string&)` | Parse SML; returns nullopt on error. | + +### `secsgem::hsms` — TCP transport (chapter 11, 33) + +```cpp +#include "secsgem/hsms/header.hpp" +#include "secsgem/hsms/connection.hpp" +``` + +| Symbol | What it is | +|-------------------------------------------------|----------------------------------------------------| +| `enum class SType` | 9 session types. | +| `enum class SelectStatus / DeselectStatus / RejectReason` | Reply codes. | +| `struct Header` | 10-byte HSMS header. | +| `struct Frame` | Header + body, length-prefixed on wire. | +| `class FrameError` | Thrown on framing errors. | +| `struct Timers` | T3/T5/T6/T7/T8 + linktest cadence. | +| `class Connection` | One-socket session manager. | +| `Connection::Mode { Active, Passive }` | TCP role. | +| `Connection::State { NotSelected, Selected }` | Transport state. | + +### `secsgem::secsi` — SECS-I transport (chapter 12, 33) + +```cpp +#include "secsgem/secsi/header.hpp" +#include "secsgem/secsi/block.hpp" +#include "secsgem/secsi/protocol.hpp" +#include "secsgem/secsi/tcp_transport.hpp" +``` + +| Symbol | What it is | +|-------------------------------------------------|----------------------------------------------------| +| `struct Header` | 10-byte SECS-I block header (R/W/E + system bytes).| +| `class Block` | One block (header + body + checksum). | +| `split_message(msg)` / `assemble_message(blocks)`| Multi-block split / assemble. | +| `class Protocol` | IO-free FSM. | +| `enum class Timer { T1, T2, T3, T4 }` | Timer IDs (raised via `EventTimeout`). | +| `Action / Event` variants | FSM IO. | +| `class TcpTransport` | asio adapter for testing tunnels. | + +### `secsgem::gem` — behavioural layer (chapters 13–19, 32, 35) + +```cpp +#include "secsgem/gem/data_model.hpp" // composite +#include "secsgem/gem/router.hpp" // dispatch table +#include "secsgem/gem/control_state.hpp" // E30 control FSM +#include "secsgem/gem/communication_state.hpp" // E30 comm FSM +#include "secsgem/gem/process_job_state.hpp" // E40 +#include "secsgem/gem/control_job_state.hpp" // E94 +#include "secsgem/gem/carrier_state.hpp" // E87 +#include "secsgem/gem/load_port_state.hpp" // E87 +#include "secsgem/gem/substrate_state.hpp" // E90 +#include "secsgem/gem/module_state.hpp" // E157 +#include "secsgem/gem/ept_state.hpp" // E116 +#include "secsgem/gem/exception_state.hpp" // E5 §13 +#include "secsgem/gem/e84_state.hpp" // E84 FSM +#include "secsgem/gem/e84_timers.hpp" // E84 TA1/TA2/TA3 +#include "secsgem/gem/e84_asio_timers.hpp" // asio wrapper +#include "secsgem/gem/host_handler.hpp" // host-side analogue +#include "secsgem/gem/messages_helpers.hpp" // identifier wildcards +// Plus build/generated/secsgem/gem/messages.hpp (codegen). +``` + +`include/secsgem/gem/store/` — 18 per-domain stores. See +chapter [32](32_stores_and_the_data_model.md) for the full table. + +### `secsgem::config` — YAML loader + validator (chapter 31, 36) + +```cpp +#include "secsgem/config/loader.hpp" +#include "secsgem/config/validate.hpp" +``` + +| Symbol | What it loads | +|-------------------------------------------------|----------------------------------------------------| +| `load_equipment(path)` | `data/equipment.yaml` → `EquipmentDescriptor`. | +| `load_control_state_table(path)` | `data/control_state.yaml` → `ControlStateConfig`. | +| `load_process_job_state(path)` | `data/process_job_state.yaml`. | +| `load_control_job_state(path)` | `data/control_job_state.yaml`. | +| `class ConfigValidator` | Multi-error YAML validator. | + +### `secsgem::metrics` — Prometheus exporter (chapter 36) + +```cpp +#include "secsgem/metrics/prometheus.hpp" +``` + +| Symbol | What it is | +|-------------------------------------------------|----------------------------------------------------| +| `class Registry` | Holds Counter + Gauge series with labels. | +| `enum class MetricType { Counter, Gauge }` | | +| `class PrometheusServer` | HTTP server on a configurable port. | + +--- + +## Message catalog reference + +164 entries in [`data/messages.yaml`](../data/messages.yaml). +Grouped by stream: + +### S1 — Identification + status + +| S/F | W | Name | Body | +|-------|---|-------------------------------|--------------------------------------------| +| S1F1 | W | Are You There | none | +| S1F2 | | On-Line Data | ` [MDLN, SOFTREV]` | +| S1F3 | W | Selected Equipment Status Req | ` [SVID, SVID, ...]` | +| S1F4 | | Selected Equipment Status Data| ` [SV, SV, ...]` | +| S1F11 | W | Status Variable Namelist Req | ` [SVID, ...]` | +| S1F12 | | Status Variable Namelist | ` [ [SVID, SVNAME, UNITS]]` | +| S1F13 | W | Establish Communications Req | ` [MDLN, SOFTREV]` | +| S1F14 | | Establish Communications Ack | ` [COMMACK, [MDLN, SOFTREV]]` | +| S1F15 | W | Request Offline | none | +| S1F16 | | OFLACK | `OFLACK` | +| S1F17 | W | Request Online | none | +| S1F18 | | ONLACK | `ONLACK` | +| S1F19 | W | Compliance Request | none | +| S1F20 | | Compliance Data | ` [CCODE, ...]` | +| S1F21 | W | DVID Namelist Request | ` [DVID, ...]` | +| S1F22 | | DVID Namelist | ` [ [DVID, DVNAME, UNITS]]` | +| S1F23 | W | CEID Namelist Request | ` [CEID, ...]` | +| S1F24 | | CEID Namelist | ` [ [CEID, [VID, VID, ...]]]` | + +### S2 — Equipment constants, clock, events, commands, spool + +S2F13/F14 (EC values), S2F15/F16 (set EC), S2F17/F18 (clock read), +S2F21/F22 (legacy RCMD), S2F23/F24 (trace init), S2F29/F30 (EC +namelist), S2F31/F32 (set clock), S2F33–F38 (report wiring), +S2F41/F42 (modern RCMD), S2F43/F44 (set spool streams), +S2F45–F48 (limits), S2F49/F50 (enhanced RCMD). + +### S3 — Carrier management (E87) + +S3F17/F18 (CarrierAction), S3F19/F20 (slot map verify), +S3F25/F26 (carrier transfer), S3F27/F28 (cancel carrier). + +### S5 — Alarms + exception recovery + +S5F1/F2 (alarm set/clear), S5F3/F4 (enable/disable alarm), +S5F5/F6 (list all alarms), S5F7/F8 (list enabled alarms), +S5F9–F18 (exception recovery, chapter 19). + +### S6 — Data collection + +S6F1/F2 (trace data), S6F11/F12 (event report), S6F15/F16 (event +report request), S6F19/F20 (individual report request), +S6F21/F22 (annotated individual report), S6F23/F24 (spool data +transmit/purge), S6F25/F26 (spool notification). + +### S7 — Process program management + +S7F1/F2 (PP load inquire), S7F3/F4 (PP send unformatted), +S7F5/F6 (PP request), S7F17/F18 (PP delete), S7F19/F20 (PP +namelist), S7F23/F24 (formatted PP send, E42), S7F25/F26 +(formatted PP request). + +### S9 — Protocol-error reports + +S9F1 (unrecognized device ID), S9F3 (unrecognized stream), +S9F5 (unrecognized function), S9F7 (illegal data), S9F9 (T3 +timeout), S9F11 (data too long), S9F13 (conversation timer +timeout). Auto-emitted; see chapter 11. + +### S10 — Terminal services + +S10F1/F2 (terminal display single, equipment→host), +S10F3/F4 (terminal display single, host→equipment), +S10F5/F6 (terminal display multi, host→equipment). + +### S12 — Wafer maps + +S12F* — Per E5 §13. Round-tripped through `raw_gem300_harness.py`. + +### S14 — Object services (E39) + control jobs (E94) + +S14F1/F2 (GetAttr), S14F3/F4 (SetAttr), S14F9/F10 (CreateCJ), +S14F11/F12 (DeleteCJ). + +### S16 — Process jobs (E40) + +S16F5/F6 (PRJobCommand), S16F7/F8 (PRJobMonitor), S16F9 (PRJobAlert +— unsolicited), S16F11/F12 (PRJobCreate), S16F13/F14 (PRJobDequeue), +S16F27/F28 (CJCommand). + +For per-message body shapes, look up the YAML entry in +`data/messages.yaml`. + +--- + +## YAML schema reference + +### `data/messages.yaml` + +```yaml +messages: + - id: SF # required, must match (stream, function) + stream: + function: + w: # reply expected? + builder: # C++ builder function name + parser: # C++ parser function name + body: # see chapter 31 for the grammar +``` + +Body shapes: + +```yaml +body: none + +body: + kind: scalar + item_type: ASCII | BINARY_BYTE | BOOLEAN | U1..U8 | I1..I8 | F4 | F8 | ITEM + enum: # optional + param: # optional, default 'value' + +body: + kind: list + struct_name: # optional; if set, parser returns struct + fields: + - {name: , shape: } + - ... + +body: + kind: list_of + element: + name: # parameter name, default 'values' +``` + +### `data/control_state.yaml` + +```yaml +transitions: + - {from: , on: , to: , then: , ack: } +``` + +`from`: one of `EquipmentOffline | AttemptOnline | HostOffline | OnlineLocal | OnlineRemote`. +`on`: one of `operator_switch_online | operator_switch_offline | operator_switch_local | operator_switch_remote | attempt_complete | attempt_failed | host_request_online | host_request_offline`. +`to`: optional new state. +`then`: optional chained state (for AttemptOnline pass-through). +`ack`: optional ACK code (`Accept`, `NotAccept`, `AlreadyOnline`). + +### `data/process_job_state.yaml` + +```yaml +transitions: + - {from: , on: , to: } +``` + +`from` / `to`: `Queued | SettingUp | WaitingForStart | Processing | ProcessComplete | Paused | Stopping | Aborting`. +`on`: `select | setup_complete | start | pause | resume | stop | abort | process_complete | abort_complete`. + +### `data/control_job_state.yaml` + +Same shape, different state/event names — see +[`include/secsgem/gem/control_job_state.hpp`](../include/secsgem/gem/control_job_state.hpp). + +### `data/equipment.yaml` + +```yaml +device: + mdln: + softrev: + capabilities: [] + +svids: + - {id: , name: , units: , type: , value: } + +dvids: + - {id: , name: , units: , type: } + +ecids: + - {id: , name: , units: , type: , value: , min: , max: } + +ceids: + - {id: , name: } + +alarms: + - {id: , alcd: , text: } + +recipes: + - {id: , body: } + +host_commands: + - {name: , ack: , emit_ceid: , set_alarm: } + +events: + default_reports: + - {ceid: , vids: [, ...]} + +spool: + whitelist: [, ...] + persistent_dir: # optional +``` + +Type strings: `ASCII`, `BINARY`, `BOOLEAN`, `U1`–`U8`, `I1`–`I8`, +`F4`, `F8`. Same vocabulary as `data/messages.yaml` body shapes. + +For required vs optional fields per record, see the validator +checks in +[`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp). + +--- + +## Where to go next + +The last chapter is the practical companion to this one: the +**recipes** for extending the codebase — adding a new SVID, host +command, state, message, store, or persistence backend. Code +patches you can copy. + +Next: [→ 51 Extending the codebase](51_extending_the_codebase.md) diff --git a/docs/51_extending_the_codebase.md b/docs/51_extending_the_codebase.md new file mode 100644 index 0000000..627e711 --- /dev/null +++ b/docs/51_extending_the_codebase.md @@ -0,0 +1,439 @@ +# 51 — Extending the codebase + +← [50 API + messages + YAML reference](50_api_messages_yaml_reference.md) | [Back to index](00_index.md) | End of series. + +Last chapter. Practical recipes for the seven most common +extensions, each with the actual mechanical steps. Roughly +ordered from "no C++ at all" to "the most C++ you'll write." + +| Recipe | C++ needed? | +|-----------------------------------------------|---------------------| +| 1. New SVID / DVID / ECID | None | +| 2. New CEID with linked reports | None | +| 3. New host command | None | +| 4. New control-state transition | None | +| 5. New SECS-II message | Handler only | +| 6. New store | New header + tests | +| 7. New persistence backend | Substantial | + +For each one: the YAML change (if any), the C++ change (if any), +the test to add, and where to look up details. + +--- + +## 1. New SVID / DVID / ECID + +The simplest extension. Add one line to `data/equipment.yaml`: + +```yaml +svids: + # ... existing entries ... + - {id: 50, name: ChamberTemp, units: "C", type: F4, value: 25.0} +``` + +Restart. Done. + +Host can now read SVID 50 via: + +- `S1F11 [50]` → returns its name and units. +- `S1F3 [50]` → returns its current value. + +The EAP can update it at any time: + +```cpp +model->svids.set_value(50, secs2::Item::f4(new_temperature)); +``` + +Same pattern for DVIDs and ECIDs. For ECIDs add `min` and `max` +for range validation. + +**Test**: not required for new SVIDs alone, but +[`tests/test_data_model.cpp`](../tests/test_data_model.cpp) shows +the pattern. + +**Reference**: chapter 31 §New SVID; +[`docs/COMPLIANCE.md`](COMPLIANCE.md) §4 (Variable / Status / +Constant rows). + +--- + +## 2. New CEID with linked reports + +Two-step YAML edit: + +```yaml +# data/equipment.yaml + +ceids: + - {id: 500, name: ChamberTempHigh} + +events: + default_reports: + - {ceid: 500, vids: [50]} # link to the SVID we just added +``` + +Restart. Done. When the EAP fires CEID 500: + +```cpp +on_temp_threshold_exceeded(float temp) { + asio::post(io, [model, temp] { + if (!model->is_event_enabled(500)) return; + auto reports = model->compose_reports_for(500); + auto msg = build_s6f11(500, reports); + deliver_or_spool(*conn, *model, std::move(msg)); + }); +} +``` + +S6F11 lands at the host with `[RPTID=..., V=[chamber_temp]]`. + +The host can re-link reports dynamically via S2F33/F35/F37 — the +`default_reports` YAML entry is just the initial state. + +**Test**: pattern in +[`tests/test_wire_ceid_emission.cpp`](../tests/test_wire_ceid_emission.cpp). + +--- + +## 3. New host command + +Add to `data/equipment.yaml`: + +```yaml +host_commands: + - {name: VENT, + ack: Accept, + emit_ceid: 400, + set_alarm: 2} +``` + +Restart. Done. Host sends `S2F41(RCMD="VENT")`: + +- `HCACK = 0` (Accept). +- CEID 400 fires → S6F11. +- ALID 2 set → S5F1. + +For commands with **application logic** beyond emit-CEID + +set-alarm, register a custom handler: + +```cpp +// At startup: +model->commands.register_handler("VENT", + [model](const ParamList& params) -> CommandOutcome { + // Actually vent the chamber here. + if (!vacuum_safe_to_vent()) { + return {HostCmdAck::CannotPerformNow, {}}; + } + hardware_vent_chamber(); + return {HostCmdAck::Accept, {}}; + }); +``` + +The registered handler overrides the YAML-defined default. + +**Reference**: chapter 31 §New host command; +[`include/secsgem/gem/store/host_commands.hpp`](../include/secsgem/gem/store/host_commands.hpp). + +--- + +## 4. New control-state transition + +Edit `data/control_state.yaml`: + +```yaml +transitions: + # ... existing rows ... + - {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept} +``` + +Restart. The transition is now active. No code changes. + +For transitions chaining through `AttemptOnline`, use `then`: + +```yaml +- {from: EquipmentOffline, on: operator_switch_online, + to: AttemptOnline, then: OnlineRemote, ack: Accept} +``` + +Same pattern for `process_job_state.yaml` and +`control_job_state.yaml`. + +**Test**: pattern in +[`tests/test_control_state.cpp`](../tests/test_control_state.cpp). + +--- + +## 5. New SECS-II message + +Two-part: YAML for the wire shape, C++ for the handler. + +### 5a. Add the message to the catalog + +```yaml +# data/messages.yaml +- id: S6F30 + stream: 6 + function: 30 + w: true + builder: s6f30_query + parser: parse_s6f30 + body: + kind: list + struct_name: TemperatureQuery + fields: + - {name: vid, shape: {kind: scalar, item_type: U4}} + - {name: threshold, shape: {kind: scalar, item_type: F4}} + +- id: S6F31 + stream: 6 + function: 31 + w: false + builder: s6f31_query_reply + parser: parse_s6f31 + body: + kind: scalar + item_type: BOOLEAN + param: above_threshold +``` + +`docker compose run --rm builder` regenerates `messages.hpp`. + +The codegen produces: + +```cpp +struct TemperatureQuery { + uint32_t vid; + float threshold; +}; + +inline secs2::Message s6f30_query(uint32_t vid, float threshold); +inline std::optional parse_s6f30(const secs2::Item&); + +inline secs2::Message s6f31_query_reply(bool above_threshold); +inline std::optional parse_s6f31(const secs2::Item&); +``` + +### 5b. Register a handler + +```cpp +router->on(6, 30, [model](const secs2::Message& m) { + auto query = messages::parse_s6f30(m.body()); + if (!query) return messages::s6f31_query_reply(false); + auto val = model->svids.value(query->vid); + if (!val) return messages::s6f31_query_reply(false); + float current = std::get>(val->storage())[0]; + return messages::s6f31_query_reply(current > query->threshold); +}); +``` + +That's it. The new S/F is on the wire. + +**Reference**: chapter 31 §New SECS-II message; +[`tests/test_messages.cpp`](../tests/test_messages.cpp) for the +testing pattern. + +--- + +## 6. New store + +When you need a record type that doesn't map onto an existing +store. E.g., add a `ReticleStore` for lithography reticles +distinct from substrates. + +### Create the header + +```cpp +// include/secsgem/gem/store/reticles.hpp +#pragma once + +#include +#include +#include + +namespace secsgem::gem { + +enum class ReticleState : uint8_t { + Loaded = 0, + Aligned = 1, + InUse = 2, + Unloaded = 3, +}; + +struct ReticleRecord { + std::string id; + ReticleState state; + int usage_count; +}; + +class ReticleStore { + public: + using ChangeHandler = + std::function; + + void register_reticle(std::string id); + void set_state(const std::string& id, ReticleState s); + std::optional get(const std::string& id) const; + std::vector all() const; + + void set_change_handler(ChangeHandler h) { on_change_ = std::move(h); } + + private: + std::map records_; + ChangeHandler on_change_; +}; + +} // namespace secsgem::gem +``` + +### Add to EquipmentDataModel + +```cpp +// include/secsgem/gem/data_model.hpp +struct EquipmentDataModel { + // ... existing members ... + ReticleStore reticles; +}; +``` + +### Write tests + +```cpp +// tests/test_reticles.cpp +#include "secsgem/gem/store/reticles.hpp" +#include + +using secsgem::gem::ReticleStore; +using secsgem::gem::ReticleState; + +TEST_CASE("ReticleStore: register and look up") { + ReticleStore s; + s.register_reticle("R-001"); + auto r = s.get("R-001"); + REQUIRE(r.has_value()); + CHECK(r->id == "R-001"); +} + +TEST_CASE("ReticleStore: state change fires handler") { + ReticleStore s; + s.register_reticle("R-002"); + ReticleState observed_from{}, observed_to{}; + s.set_change_handler([&](auto& id, auto from, auto to) { + observed_from = from; + observed_to = to; + }); + s.set_state("R-002", ReticleState::Aligned); + CHECK(observed_from == ReticleState::Loaded); + CHECK(observed_to == ReticleState::Aligned); +} +``` + +CMake picks up new tests automatically (glob over `tests/*.cpp`). + +### Wire Router handlers if needed + +If reticles need wire access (e.g., a custom S6FX request), add the +message to `data/messages.yaml` (recipe 5) and register handlers. + +**Reference**: chapter 32 §How to add a new store. + +--- + +## 7. New persistence backend + +The codebase ships file-backed persistence with per-record files +(chapter 36). Some deployments want different backends — SQLite, +LMDB, a key-value cache. + +The persistence is wired *inside each store* rather than through +an abstraction, so changing the backend means changing each +store's `enable_persistence` implementation. Two approaches: + +### 7a. Drop-in replacement + +Replace the file IO inside each store's `journal_write` / +`journal_remove` / `journal_replay` methods with calls to your +backend. + +Pros: no API change, no test churn. +Cons: changes 7 stores; you have to update each one. + +### 7b. Pluggable backend + +Introduce an interface: + +```cpp +class JournalBackend { + public: + virtual ~JournalBackend() = default; + virtual void write(std::string_view key, const std::vector&) = 0; + virtual std::optional> read(std::string_view key) = 0; + virtual void remove(std::string_view key) = 0; + virtual std::vector list_keys() = 0; +}; +``` + +Each store accepts a `std::shared_ptr`. The +default implementation is `FileJournalBackend` (current behaviour); +alternatives can be `SqliteJournalBackend`, `LmdbJournalBackend`, +etc. + +Pros: clean separation, multiple backends coexist. +Cons: substantial refactor across 7 stores + their tests. + +For most deployments option 7a is the right call — the file +backend is fast enough that swap-outs are rare. + +**Reference**: chapter 36 §The per-record file pattern; +[`tests/test_persistence_upgrade.cpp`](../tests/test_persistence_upgrade.cpp) +for the test patterns. + +--- + +## What to do when something doesn't fit any recipe + +Some extensions don't map onto these seven. Examples: + +- A new SEMI standard the codebase doesn't implement. +- A transport that isn't HSMS or SECS-I. +- A different codec (highly unusual). +- A different YAML schema (e.g., a third-party format). + +For these, the right move is to: + +1. **Open an issue / RFC** describing what you want. +2. **Sketch the API change** before writing code. +3. **Add tests first** — at the integration layer. +4. **Reach into the right namespace** based on the chapter map at + the top of this guide. + +The codebase is small and the layering is clean; major extensions +usually fit naturally into one of the 21 stores or one of the +existing namespaces. Resist the temptation to add a new abstraction +layer; almost everything that looks like it needs one actually +fits as a new store + a few handlers. + +--- + +## End of the guide + +You've reached the end. You should now be able to: + +- Read any SECS/GEM standard and recognise its shape. +- Read any commit in this codebase and place it in the + architecture. +- Read any wire trace and trace it back to a Router handler. +- Add new SVIDs / CEIDs / commands / states / messages without + recompiling. +- Add new stores or wire to new SECS standards with confidence. +- Stand up the demo, drive every external validator, and reason + about deployment + monitoring + security. + +If anything in the codebase still surprises you, the chapter map +at [`docs/00_index.md`](00_index.md) is your starting point for +finding the relevant section. + +The proofs in [`docs/PROOFS.md`](PROOFS.md) are the **claim**; +this guide was the **explanation**. Treat them as paired +documents. + +← [Back to index](00_index.md)