raphael/secs-gem
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: chapters 40, 41, 50, 51 — Operations + Reference (series complete)
tests / build-and-test (push) Successful in 2m7s
Details
tests / thread-sanitizer (push) Successful in 2m34s
Details
tests / tshark-dissector (push) Successful in 2m22s
Details
tests / secs4j-interop (push) Failing after 5s
Details
tests / libfuzzer (push) Successful in 3m7s
Details

Browse Source 
Last four chapters of the guided tour.

40 — Building, running, the demo.  Docker prerequisites, the build
flow, what each binary is for, running the 24-transaction demo
flow annotated step by step.  Running the 4 external-validator
sweeps + the libFuzzer pass.  Inspecting the demo with tcpdump and
tshark.  Reading source while running as the recommended learning
workflow.

41 — Integration: hardware, MES, production.  Four-phase tour:
wiring sensors / recipe engine / alarms / E84 GPIO; talking to a
real MES with the day-1 punch list + commercial-MES quirks (Wonderware
S2F21, Camstar Linktest cadence, etc.); production hardening
(nftables / stunnel / minisign / persistence layout / monitoring /
runbook); performance envelope + memory footprint + capacity
planning.  Pointers to the long-form INTEGRATION.md / MES_INTEROP.md /
SECURITY.md / BENCHMARKS.md.

50 — API + message catalog + YAML schemas reference.  Namespace-by-
namespace table of public symbols (secs2, hsms, secsi, gem, config,
metrics) with brief descriptions.  Stream-by-stream message catalog
reference (S1, S2, S3, S5, S6, S7, S9, S10, S12, S14, S16).  YAML
schema reference for messages.yaml + the three state-table files +
equipment.yaml.

51 — Extending the codebase.  Seven recipes ordered from no-code to
substantial: new SVID/DVID/ECID (YAML only), new CEID with reports
(YAML only), new host command (YAML + optional handler), new control-
state transition (YAML only), new SECS-II message (YAML + handler),
new store (header + tests), new persistence backend (drop-in vs
pluggable trade-off).  Each recipe has the actual mechanical steps,
the test pattern, and pointers to the chapter that explains why it
works.

Index updated to mark all 24 chapters published.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Raphael Maenle
2026-06-09 20:28:21 +02:00
parent cae98d9a7d
commit 31f908e1bf
5 changed files with 1513 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/00_index.md
+3 -7
View File
@@ -248,12 +248,8 @@ this guide is the *tutorial path* that ties them together.
## Status of this guide ## Status of this guide
Chapters publish as they're written. The list above is the table of **All 24 chapters published.** Read linearly from
contents; individual files exist once the chapter has been written. [01](01_what_is_secs_gem.md) or jump in at whichever part fits your
A chapter without a working link is on the to-write list. goal (see "How to read this guide" above).
**Currently published:** Chapter 00 (this index).
**In progress:** Chapter 01 — *What is SECS/GEM?*
Next chapter: [→ 01 What is SECS/GEM?](01_what_is_secs_gem.md) Next chapter: [→ 01 What is SECS/GEM?](01_what_is_secs_gem.md)
docs/40_building_running_demo.md
+376
View File
@@ -0,0 +1,376 @@
# 40 — Building, running, the demo
← [36 Persistence, validation, metrics](36_persistence_validation_metrics.md) | [Back to index](00_index.md) | Next: [41 Integration: hardware, MES, production](41_integration_hardware_mes_production.md) →
You've read about every layer of the codebase. Now we run it.
This chapter is operational: build the project, start the demo,
walk what each transaction in the two-container flow actually does
and where it lives. By the end you'll have the demo running on
your laptop and you'll know what every log line means.
---
## Prerequisites
Just **Docker**. No host C++ toolchain, no Python deps, nothing
to apt-install. The toolchain image (`Dockerfile`) bundles Ubuntu
24.04 + g++-13 + CMake + Ninja + asio + yaml-cpp + Python 3 +
tshark + tcpdump + clang.
```bash
docker --version
docker compose version
```
If both work, you're set.
---
## Building
```bash
docker compose run --rm builder
```
That:
1. Pulls / builds the toolchain image (first time only, ~3
minutes).
2. Runs `cmake -S /app -B /app/build -G Ninja -DCMAKE_BUILD_TYPE=Release`.
3. Runs `cmake --build /app/build`.
4. Produces every binary under `/app/build/` inside a named Docker
volume.
Subsequent builds are incremental and take ~10–30 s.
### What got built
```
build/
├── secs_server passive equipment (the demo target)
├── secs_client active host (drives the demo)
├── secs_conformance 47-check conformance harness
├── secs_interop_probe active host probing secsgem-py equipment
├── secs_bench throughput/latency bench
├── secsgem_tests the 445-case doctest binary
└── pvd_tool worked PVD-tool example
```
Plus the generated `build/generated/secsgem/gem/messages.hpp`
(~3 500 lines, auto-derived from `data/messages.yaml`).
---
## Running the tests
```bash
docker compose run --rm tests
```
Runs `secsgem_tests` end-to-end. Expected output:
```
[doctest] doctest version is "2.4.11"
[doctest] run with "--help" for options
===============================================================================
[doctest] test cases: 445 | 445 passed | 0 failed | 0 skipped
[doctest] assertions: 2753 | 2753 passed | 0 failed |
[doctest] Status: SUCCESS!
```
On a 2024 M-series Mac under Docker Desktop, this takes ~3.5 s.
---
## The two-container demo
```bash
docker compose up --no-deps server client
```
That starts:
- A **`server`** container running `secs_server` on port 5000.
- A **`client`** container running `secs_client` against `server:5000`.
The client drives ~24 SECS transactions through the data model.
Each transaction logs on both sides.
### What each transaction does
Annotated walk through the log output:
#### Communication establishment
```
[host] connecting to server:5000
[equip] accepted connection
[host] sending Select.req
[equip] Select.req received → SELECTED
[host] Select.rsp(Ok) received → SELECTED
```
HSMS SELECT handshake. Both sides now in SELECTED state.
```
[host] sending S1F13 Establish Communications
[equip] S1F13 received
[equip] sending S1F14(COMMACK=Accept, [MDLN, SOFTREV])
[host] S1F14 received → COMMUNICATING
```
E30 §6.5 communication-state transition. Now GEM-level
communication is up.
#### Identification
```
[host] S1F1 Are You There
[equip] S1F2 ["SECS-GEM Demo Equipment", "1.0.0"]
[host] S1F19 GEM Compliance Request
[equip] S1F20 [list of capabilities]
[host] S1F11 SVID Namelist (all)
[equip] S1F12 [SVID 1 "ControlState", SVID 2 "Clock", ...]
[host] S1F21 DVID Namelist (all)
[equip] S1F22 [DVID list]
[host] S1F23 CEID Namelist (all)
[equip] S1F24 [CEID → VID mapping]
```
Host walks the data dictionary.
#### Dynamic event report setup
```
[host] S2F33 DefineReport(RPTID=1, VIDs=[SVID 2])
[equip] S2F34(DRACK=0)
[host] S2F35 LinkEvent(CEID=300 → [RPTID=1])
[equip] S2F36(LRACK=0)
[host] S2F37 EnableEvent(CEED=true, CEIDs=[300])
[equip] S2F38(ERACK=0)
```
The three-message report wiring. CEID 300 now triggers an S6F11
when it fires.
#### Control state + remote command
```
[host] S2F41 RCMD=START
[equip] S2F42(HCACK=Accept)
[equip] HostCommandRegistry dispatched START
[equip] → emit CEID 300
[equip] → compose_reports_for(300) → RPTID 1 = [Clock SV2]
[equip] → fire S6F11
[equip] S6F11(CEID=300, [RPTID=1, [Clock]])
[host] S6F12(ACKC6=0)
```
Host command dispatch + event report emission + acknowledgement.
This is the canonical GEM transaction.
#### Alarms
```
[host] S5F5 List all alarms
[equip] S5F6 [ALID list with ALCD + ALTX]
[host] S5F3 EnableAlarm(ALID=1)
[equip] S5F4(ACKC5=0)
[host] S2F41 RCMD=FAULT
[equip] S2F42(HCACK=Accept)
[equip] → set ALID 1
[equip] → fire S5F1(ALCD=0x84, ALID=1)
[equip] S5F1(...)
[host] S5F2(ACKC5=0)
```
#### Recipes
```
[host] S7F1 PP Load Inquire(PPID="NEW-RECIPE", LENGTH=64)
[equip] S7F2(PPGNT=0=Permit)
[host] S7F3 PP Send(PPID="NEW-RECIPE", PPBODY=<bytes>)
[equip] S7F4(ACKC7=0)
[host] S7F5 PP Request(PPID="NEW-RECIPE")
[equip] S7F6 [PPID, PPBODY]
[host] S7F17 PP Delete(PPIDs=["NEW-RECIPE"])
[equip] S7F18(ACKC7=0)
```
#### Terminal display
```
[host] S10F3 Terminal Display Multi (TID=0, TEXT="hello\nfrom host")
[equip] S10F4(ACKC10=0)
```
#### Clean shutdown
```
[host] S1F15 Request Offline
[equip] S1F16(OFLACK=Accept)
[host] sending Separate.req
[equip] Separate.req received → close
```
Total: 24 transactions exercising S1, S2, S5, S6, S7, S10.
---
## Running the conformance harness
```bash
docker compose up -d server
docker compose run --rm builder /app/build/secs_conformance --host server --port 5000
docker compose down
```
Runs the 47-check conformance harness against the demo server.
Each check covers one E30 / GEM 300 wire-level behaviour:
```
[PASS] E37 §7.2 SELECT handshake
[PASS] E30 §6.5 S1F13/F14 Establish Comms
[PASS] E30 §6.7 S1F1/F2 Are You There
... (43 more)
[PASS] E30 §6.10 S1F19/F20 GEM Compliance
47 / 47 checks passed
```
This is proof #2 in [`docs/PROOFS.md`](PROOFS.md).
---
## Running the interop sweeps
### secsgem-py
```bash
docker compose up -d server
docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py --host server
docker compose down
```
The Python `secsgem-py` 0.3.0 host drives our equipment. 31 checks
across S1/S2/S5/S6/S7/S10 + unsolicited S6F11 / S5F1.
### secs4java8
```bash
bash interop/secs4j_validate.sh
```
The Java secs4java8 host drives our equipment via a separate
container. 55 checks covering S1/S2/S3/S5/S6/S7/S10/S14/S16
including the GEM 300 streams that secsgem-py couldn't easily
drive.
### tshark dissector
```bash
docker compose run --rm builder bash /app/interop/tshark_validate.sh
```
Captures a pcap of the demo flow, dissects with Wireshark's HSMS
dissector, asserts no malformed packets. 69 frames, 0 errors.
### libFuzzer (60 s, requires clang)
```bash
docker compose run --rm builder bash -c "
cmake -S /app -B /app/build-fuzz -G Ninja -DSECSGEM_FUZZ=ON \
-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
cmake --build /app/build-fuzz
/app/build-fuzz/fuzz_secs2_decode -max_total_time=60
/app/build-fuzz/fuzz_sml_parse -max_total_time=60
"
```
200 k+ inputs through `secs2::decode`, 1.4 M+ through
`try_parse_sml`, ASan + UBSan clean, 0 crashes.
All five sweeps are wired into CI; see
[`.gitea/workflows/ci.yml`](../.gitea/workflows/ci.yml).
---
## Inspecting the demo from outside
While the demo is running, you can:
### Watch the wire
```bash
# In another shell:
docker compose exec server tcpdump -i any -A -s 0 'tcp port 5000'
```
### Inspect with tshark + HSMS dissector
```bash
docker compose run --rm builder tshark -i any -d "tcp.port==5000,hsms" -V \
| grep -A 2 "Header"
```
### Watch the metrics
`pvd_tool` example exposes a Prometheus endpoint:
```bash
docker run --rm -p 9090:9090 pvd_tool /app/examples/pvd_tool/equipment.yaml \
/app/data/control_state.yaml 5000 9090
```
Then `curl localhost:9090/metrics`.
---
## Running the bench
```bash
docker compose run --rm builder /app/build/secs_bench \
--requests 50000 --concurrency 32 --svid-count 32
```
Outputs a markdown table of throughput + p50/p95/p99 latencies for:
- S1F1/F2 (header-only round-trip).
- S1F3/F4 with 32 SVIDs.
- S6F11 push (W=0, fire-and-forget).
- PJ + CJ memory footprint.
See [`docs/BENCHMARKS.md`](BENCHMARKS.md) for the baseline numbers
and capacity-planning notes.
---
## Reading the source while it runs
A common workflow when you're learning:
1. `docker compose up --no-deps server client` in one shell.
2. Source viewer open in another (your IDE on the host —
the source isn't bind-mounted in the container, but it is
on your host).
3. Find a log line that confuses you (e.g. `[equip] S6F11 fired`).
4. Grep the source for it. Most log strings are unique enough to
land in the right file in one search.
5. Read the function around it.
6. Cross-reference back to the chapter that covers the standard.
This is the most efficient way to internalise the codebase. The
demo runs forever (until you `Ctrl-C` — the client loops); you
can read the source at your own pace.
---
## Where to go next
You now have the demo running and you can drive any of the five
external validators. The next chapter is the **integration**
chapter — wiring the runtime to real hardware, talking to a real
MES, production deployment, security, performance tuning.
Next: [→ 41 Integration: hardware, MES, production](41_integration_hardware_mes_production.md)
docs/41_integration_hardware_mes_production.md
+350
View File
@@ -0,0 +1,350 @@
# 41 — Integration: hardware, MES, production
← [40 Building, running, the demo](40_building_running_demo.md) | [Back to index](00_index.md) | Next: [50 API + messages + YAML reference](50_api_messages_yaml_reference.md) →
You have the demo running. Now you need to make it talk to a
**real tool**, against a **real MES**, in a **real fab**.
This chapter walks the four phases of that journey:
1. **Wire the EAP to physical hardware.**
2. **Integrate with a commercial MES.**
3. **Production hardening** — security, monitoring, persistence.
4. **Operational concerns** — performance, capacity, incidents.
This is a compressed view of the long-form
[`docs/INTEGRATION.md`](INTEGRATION.md); cross-references are
inline. The long-form has more code and more configuration; this
chapter explains the *shape* of each phase.
---
## Phase 1 — wiring to hardware
### What "the EAP" actually does
The EAP (Equipment Automation Program) sits between the SECS/GEM
runtime and the **physical tool**. It does four things:
1. **Reads sensors** at the right cadence and updates SVIDs.
2. **Drives the recipe engine** when a host command arrives.
3. **Listens for alarms** from PLCs / hardware fault lines.
4. **Wires FSM transitions** to CEID emissions.
[`examples/pvd_tool/main.cpp`](../examples/pvd_tool/main.cpp) is
the worked reference. Section by section:
| Section in main.cpp | What it shows |
|-------------------------------|------------------------------------------------------------------------|
| §1 Helpers + constants | The `kSvidX / kCeidX` constants worth pinning at file scope |
| §2 Sensor simulator | Multi-cadence sensor poll loops with `asio::post` strand-marshal |
| §3 Recipe runner | PJ → SettingUp → Processing → ProcessComplete walk; per-step CEID emit |
| §4 Alarm threshold monitor | Continuous threshold evaluation against ECID setpoints |
| §5 EPT cycling | E116 transitions driven by PJ state + safety alarms |
| §6 Router handlers | 51 handlers in ~460 lines — every S/F a host might send to a PVD tool |
| §7 main() | YAML load → validate → compose → run |
A real tool fork:
```bash
cp -r examples/pvd_tool/ src/my_tool/
# edit src/my_tool/equipment.yaml — your tool's SVIDs/CEIDs/alarms
# edit src/my_tool/main.cpp — replace pvd::Simulator with PLC bindings
```
### Sensor wiring
The PVD example uses a random-walk simulator (§2). A real EAP
replaces this with calls into the tool's sensor stack:
```cpp
// Original (simulated):
upd_f4(kSvidChamberPressure, target_pressure.load(), 1e-8f, 1e-7f);
// Real:
asio::post(io, [model](){
float pressure = plc_read_register(0x4001); // from your PLC API
model->svids.set_value(kSvidChamberPressure, secs2::Item::f4(pressure));
});
```
The `asio::post` is non-negotiable — the store mutation runs on
the io_context strand (chapter 33).
### Recipe runner
The PVD example's recipe runner (§3) parses the recipe body and
walks PJ states. A real tool replaces the simulator with calls
into the tool's recipe engine:
```cpp
void start_processing(const std::string& pjid, const std::string& ppid) {
auto recipe = recipes_->find(ppid);
if (!recipe) {
model->process_jobs.apply(pjid, ProcessJobEvent::Abort);
return;
}
// Hand the recipe to the tool's actual recipe engine.
hardware_recipe_engine_->start(*recipe, [model, pjid](bool ok) {
asio::post(io, [model, pjid, ok] {
model->process_jobs.apply(pjid,
ok ? ProcessJobEvent::ProcessComplete
: ProcessJobEvent::AbortComplete);
});
});
}
```
### Alarm sources
Real alarms come from:
- **PLC fault lines** — interrupt callbacks.
- **Watchdog timers** — periodic checks (cooling water flow,
vacuum pressure).
- **Sensor thresholds** — continuous evaluation against ECIDs.
- **Hardware safety interlocks** — SafetyController callbacks.
Each translates to one `model->alarms.set(alid)` call. The alarm
dispatcher takes care of S5F1 emission, host enable filtering, and
alarm persistence.
### E84 wiring
E84 needs a GPIO driver:
```cpp
// On signal change from the GPIO driver:
void on_gpio_change(uint8_t port, E84Signal sig, bool value) {
asio::post(io, [model, port, sig, value]() {
model->e84_ports.at(port).fsm.on_signal_change(sig, value);
});
}
// When the FSM wants to assert a signal:
model->e84_ports.at(port).fsm.set_emit_handler(
[port](E84Signal sig, bool value) {
gpio_driver_write(port, sig, value);
});
```
The TA1/TA2/TA3 timers are wall-clock; use the
[`E84AsioTimers`](../include/secsgem/gem/e84_asio_timers.hpp)
adapter so they fire on the same io_context.
---
## Phase 2 — talking to a real MES
### The day-1 punch list
Before you connect to a production MES, run
[`docs/MES_INTEROP.md`](MES_INTEROP.md) against the **staging**
MES. 59 test IDs across:
- Transport (T-01 to T-09): SELECT, Linktest, T3, T7, oversized
frames.
- Establishment (E-01 to E-08): S1F13, S1F1, S1F11, S1F19, …
- Reports (R-01 to R-07): the S2F33/F35/F37 dance.
- Alarms (A-01 to A-06).
- Commands (C-01 to C-04): S2F41 + S2F21 (legacy) + S2F49
(enhanced).
- Recipes (P-01 to P-05).
- Terminal services (TS-01 to TS-03).
- Jobs (J-01 to J-06).
- Spool (SP-01 to SP-05).
- Clock (K-01 to K-06).
This is the **friction-killer document**. Pass every test ID in
staging and your production cutover is much less likely to
surprise you.
### HSMS-GS for multi-MES
Some fabs run multiple MES against one equipment. E37 §11
HSMS-GS multiplexes over one TCP socket:
```cpp
auto conn = std::make_shared<hsms::Connection>(
std::move(sock), Mode::Passive, /*primary device_id=*/0, timers);
// Production MES on session 100.
conn->add_session(100);
conn->set_session_message_handler(100, production_router_handler);
// Maintenance MES on session 200.
conn->add_session(200);
conn->set_session_message_handler(200, maintenance_router_handler);
```
[`docs/INTEGRATION.md`](INTEGRATION.md) §7 has the full worked
example with HA pattern. Tests:
[`tests/test_hsms_gs.cpp`](../tests/test_hsms_gs.cpp) (5 wire-level)
and
[`tests/test_hsms_gs_integration.cpp`](../tests/test_hsms_gs_integration.cpp)
(1 end-to-end three-session scenario).
### Things commercial MES get wrong
Real MES exhibit common deviations:
- **Wonderware uses S2F21 (legacy) exclusively** — no S2F41. The
codebase's HostCommandRegistry handles both forms.
- **Some MES leave EQPTYP in S1F20 confused with MDLN** — the
codebase accepts either; documented in
[`docs/MES_INTEROP.md`](MES_INTEROP.md) E-02 caveat.
- **MES with old PPBODY handling reject binary recipes** — the
codebase ships both as-bytes and as-ASCII PPBODY.
- **Camstar uses Linktest at 30 s**, others at 60 s — configure
`Timers::linktest` to match the host's cadence.
[`docs/MES_INTEROP.md`](MES_INTEROP.md) "Caveats" column lists more.
---
## Phase 3 — production hardening
### Security: SECURITY.md walk-through
[`docs/SECURITY.md`](SECURITY.md) ships concrete configs for:
- **nftables** — restrict the SECS port to the MES host's IP only.
- **stunnel** — wrap the HSMS port in TLS so the wire is
encrypted.
- **minisign** — sign every recipe (PPBODY) and verify on receive.
- **SIEM audit log schema** — what every store mutation emits as
JSON for log aggregation.
Configure all four before promoting to production. No exceptions.
### Persistence layout
Production deployments enable persistence on every store that
needs it. Per [`docs/INTEGRATION.md`](INTEGRATION.md) §5:
```
/var/lib/secsgem/
├── spool/ # SpoolStore
├── pj/ # ProcessJobStore
├── cj/ # ControlJobStore
├── exceptions/ # ExceptionStore
├── carriers/ # CarrierStore
├── load_ports/ # LoadPortStore
└── substrates/ # SubstrateStore
```
On an SSD with `fsync` enabled per file rewrite, this comfortably
handles a few hundred mutations / sec. On rotational media you'll
want to batch or relax durability.
### Monitoring
Production EAPs typically export:
- **Per-CEID emission counter** — burst detection.
- **Spool depth gauge** — alarms when growing (MES connectivity
problem).
- **T3 timeout counter** — non-zero means the MES is slow or your
T3 is too short.
- **Per-alarm set count** — pages on certain ALIDs.
- **Equipment EPT state gauge** — fab-wide dashboard input.
[`docs/INTEGRATION.md`](INTEGRATION.md) §6.4 covers the Grafana
panel patterns; the PVD example wires the Prometheus exporter at
§7.
### Operational runbook
[`README.md`](../README.md) ships a starter runbook:
| Incident | First check | Mitigation |
|-------------------------------------|--------------------------------------|-------------------------------------------|
| HSMS connection flapping | T7 / T6 timer fires in logs | check MES reachability, network MTU |
| Spool depth growing | host MES connectivity / ACK rate | force-drain via S6F23, escalate to MES |
| State machine "stuck" | last state-change handler log line | host-issued offline + re-establish |
| Alarm storm | `AlarmRegistry::all()` snapshot | check upstream sensor; quench via S5F3 |
| Persistence dir growing unbounded | `du -s` + file count | sweep terminal-state records |
| Cross-tool inconsistency | `secsgem_tests` on canary tool | compare wire trace vs validator |
---
## Phase 4 — performance
### The envelope
Per [`docs/BENCHMARKS.md`](BENCHMARKS.md), on a 2024 M-series Mac
under Docker Desktop:
| Scenario | Ops/sec | p50 µs | p95 µs | p99 µs |
|-----------------------------------|--------:|--------:|-------:|-------:|
| S1F1/F2 (header-only) | ~140 k | 74 | 103 | 161 |
| S1F3/F4 (32 SVIDs) | ~79 k | 165 | 186 | 260 |
| S6F11 push (W=0) | ~572 k | n/a | n/a | n/a |
A real fab tool sees **tens to a few hundred** events / s
sustained. We're three orders of magnitude above the push path,
two orders above the round-trip path.
**Throughput is not the bottleneck**; tail latency under
contention is. Tune by:
- Running on a quiet host.
- Bumping `linktest` interval up (default 0 = disabled is fine for
most deployments).
- Pinning the io_context thread to a dedicated CPU.
### Memory footprint
| Entity | Approx bytes / instance |
|---------------------|------------------------:|
| PJ + CJ pair | ~450 |
| Carrier (no slots) | ~80 |
| Carrier slot | ~24 |
| Substrate | ~120 |
| Spool entry | ~40 + encoded body size |
A busy 300 mm tool with 50 carriers × 25 slots + 200 substrates +
20 active PJ+CJ pairs is under **1 MiB** of model state. RSS is
dominated by the binary itself + asio's buffers (~10–20 MiB).
### Capacity planning
For sizing purposes:
- One io_context thread per `Connection` is plenty for any single
tool.
- Multiple tools share the same process if you want — one
io_context, multiple `Connection`s, each on its own strand.
- Persistence cost is one `rename(2)` per mutation; SSD-bound
fabs comfortably handle a few hundred / sec.
---
## When to read the long-form
This chapter compressed phases 1–4 into a tour. Each phase has
substantially more material in the long-form docs:
- [`docs/INTEGRATION.md`](INTEGRATION.md) — full vendor-side
tutorial including wiring sensors, plugging FSMs, persistence
layout, monitoring, HSMS-GS HA.
- [`docs/MES_INTEROP.md`](MES_INTEROP.md) — the 59 test IDs in
full.
- [`docs/SECURITY.md`](SECURITY.md) — concrete nftables / stunnel /
minisign configs.
- [`docs/BENCHMARKS.md`](BENCHMARKS.md) — perf envelope + how to
re-run.
When you actually start an integration, work from those. This
chapter is the map.
---
## End of Part 4
You can now build the codebase, run the demo, drive every
external validator, and you know the shape of how a real
integration would land. Part 5 is reference material — API
namespaces, message catalog reference, the extension recipes.
Next: [→ 50 API + messages + YAML reference](50_api_messages_yaml_reference.md)
docs/50_api_messages_yaml_reference.md
+345
View File
@@ -0,0 +1,345 @@
# 50 — API + message catalog + YAML schemas
← [41 Integration: hardware, MES, production](41_integration_hardware_mes_production.md) | [Back to index](00_index.md) | Next: [51 Extending the codebase](51_extending_the_codebase.md) →
This chapter is **reference**, not tutorial. Look up the namespace
or YAML key you need; cross-reference the code.
The whole codebase is small enough that "go read the header" is
often the right answer — this chapter helps you find which header.
---
## Namespace reference
### `secsgem::secs2` — codec (chapter 34)
```cpp
#include "secsgem/secs2/item.hpp"
#include "secsgem/secs2/codec.hpp"
#include "secsgem/secs2/message.hpp"
#include "secsgem/secs2/sml.hpp"
```
| Symbol | What it is |
|-------------------------------------------------|----------------------------------------------------|
| `enum class Format` | 16 SECS-II format codes. |
| `class Item` | Variant-based value type. |
| `class Message` | Stream + function + W-bit + system_bytes + body. |
| `class CodecError` | Thrown on malformed input. |
| `std::vector<uint8_t> encode(const Item&)` | Serialize an Item to bytes. |
| `void encode_into(const Item&, std::vector<uint8_t>&)` | Append-encode into existing buffer. |
| `Item decode(const std::vector<uint8_t>&)` | Decode one Item from a complete buffer. |
| `Item decode_at(const uint8_t*, size_t, size_t&)`| Decode one Item from a position; advances cursor. |
| `std::string to_sml(const Item&)` | Render as SML. |
| `std::optional<Item> try_parse_sml(const std::string&)` | Parse SML; returns nullopt on error. |
### `secsgem::hsms` — TCP transport (chapter 11, 33)
```cpp
#include "secsgem/hsms/header.hpp"
#include "secsgem/hsms/connection.hpp"
```
| Symbol | What it is |
|-------------------------------------------------|----------------------------------------------------|
| `enum class SType` | 9 session types. |
| `enum class SelectStatus / DeselectStatus / RejectReason` | Reply codes. |
| `struct Header` | 10-byte HSMS header. |
| `struct Frame` | Header + body, length-prefixed on wire. |
| `class FrameError` | Thrown on framing errors. |
| `struct Timers` | T3/T5/T6/T7/T8 + linktest cadence. |
| `class Connection` | One-socket session manager. |
| `Connection::Mode { Active, Passive }` | TCP role. |
| `Connection::State { NotSelected, Selected }` | Transport state. |
### `secsgem::secsi` — SECS-I transport (chapter 12, 33)
```cpp
#include "secsgem/secsi/header.hpp"
#include "secsgem/secsi/block.hpp"
#include "secsgem/secsi/protocol.hpp"
#include "secsgem/secsi/tcp_transport.hpp"
```
| Symbol | What it is |
|-------------------------------------------------|----------------------------------------------------|
| `struct Header` | 10-byte SECS-I block header (R/W/E + system bytes).|
| `class Block` | One block (header + body + checksum). |
| `split_message(msg)` / `assemble_message(blocks)`| Multi-block split / assemble. |
| `class Protocol` | IO-free FSM. |
| `enum class Timer { T1, T2, T3, T4 }` | Timer IDs (raised via `EventTimeout`). |
| `Action / Event` variants | FSM IO. |
| `class TcpTransport` | asio adapter for testing tunnels. |
### `secsgem::gem` — behavioural layer (chapters 13–19, 32, 35)
```cpp
#include "secsgem/gem/data_model.hpp" // composite
#include "secsgem/gem/router.hpp" // dispatch table
#include "secsgem/gem/control_state.hpp" // E30 control FSM
#include "secsgem/gem/communication_state.hpp" // E30 comm FSM
#include "secsgem/gem/process_job_state.hpp" // E40
#include "secsgem/gem/control_job_state.hpp" // E94
#include "secsgem/gem/carrier_state.hpp" // E87
#include "secsgem/gem/load_port_state.hpp" // E87
#include "secsgem/gem/substrate_state.hpp" // E90
#include "secsgem/gem/module_state.hpp" // E157
#include "secsgem/gem/ept_state.hpp" // E116
#include "secsgem/gem/exception_state.hpp" // E5 §13
#include "secsgem/gem/e84_state.hpp" // E84 FSM
#include "secsgem/gem/e84_timers.hpp" // E84 TA1/TA2/TA3
#include "secsgem/gem/e84_asio_timers.hpp" // asio wrapper
#include "secsgem/gem/host_handler.hpp" // host-side analogue
#include "secsgem/gem/messages_helpers.hpp" // identifier wildcards
// Plus build/generated/secsgem/gem/messages.hpp (codegen).
```
`include/secsgem/gem/store/` — 18 per-domain stores. See
chapter [32](32_stores_and_the_data_model.md) for the full table.
### `secsgem::config` — YAML loader + validator (chapter 31, 36)
```cpp
#include "secsgem/config/loader.hpp"
#include "secsgem/config/validate.hpp"
```
| Symbol | What it loads |
|-------------------------------------------------|----------------------------------------------------|
| `load_equipment(path)` | `data/equipment.yaml` → `EquipmentDescriptor`. |
| `load_control_state_table(path)` | `data/control_state.yaml` → `ControlStateConfig`. |
| `load_process_job_state(path)` | `data/process_job_state.yaml`. |
| `load_control_job_state(path)` | `data/control_job_state.yaml`. |
| `class ConfigValidator` | Multi-error YAML validator. |
### `secsgem::metrics` — Prometheus exporter (chapter 36)
```cpp
#include "secsgem/metrics/prometheus.hpp"
```
| Symbol | What it is |
|-------------------------------------------------|----------------------------------------------------|
| `class Registry` | Holds Counter + Gauge series with labels. |
| `enum class MetricType { Counter, Gauge }` | |
| `class PrometheusServer` | HTTP server on a configurable port. |
---
## Message catalog reference
164 entries in [`data/messages.yaml`](../data/messages.yaml).
Grouped by stream:
### S1 — Identification + status
| S/F | W | Name | Body |
|-------|---|-------------------------------|--------------------------------------------|
| S1F1 | W | Are You There | none |
| S1F2 | | On-Line Data | `<L,2> [MDLN, SOFTREV]` |
| S1F3 | W | Selected Equipment Status Req | `<L,n> [SVID, SVID, ...]` |
| S1F4 | | Selected Equipment Status Data| `<L,n> [SV, SV, ...]` |
| S1F11 | W | Status Variable Namelist Req | `<L,n> [SVID, ...]` |
| S1F12 | | Status Variable Namelist | `<L,n> [<L,3> [SVID, SVNAME, UNITS]]` |
| S1F13 | W | Establish Communications Req | `<L,2> [MDLN, SOFTREV]` |
| S1F14 | | Establish Communications Ack | `<L,2> [COMMACK, <L,2> [MDLN, SOFTREV]]` |
| S1F15 | W | Request Offline | none |
| S1F16 | | OFLACK | `OFLACK` |
| S1F17 | W | Request Online | none |
| S1F18 | | ONLACK | `ONLACK` |
| S1F19 | W | Compliance Request | none |
| S1F20 | | Compliance Data | `<L,n> [CCODE, ...]` |
| S1F21 | W | DVID Namelist Request | `<L,n> [DVID, ...]` |
| S1F22 | | DVID Namelist | `<L,n> [<L,3> [DVID, DVNAME, UNITS]]` |
| S1F23 | W | CEID Namelist Request | `<L,n> [CEID, ...]` |
| S1F24 | | CEID Namelist | `<L,n> [<L,2> [CEID, [VID, VID, ...]]]` |
### S2 — Equipment constants, clock, events, commands, spool
S2F13/F14 (EC values), S2F15/F16 (set EC), S2F17/F18 (clock read),
S2F21/F22 (legacy RCMD), S2F23/F24 (trace init), S2F29/F30 (EC
namelist), S2F31/F32 (set clock), S2F33–F38 (report wiring),
S2F41/F42 (modern RCMD), S2F43/F44 (set spool streams),
S2F45–F48 (limits), S2F49/F50 (enhanced RCMD).
### S3 — Carrier management (E87)
S3F17/F18 (CarrierAction), S3F19/F20 (slot map verify),
S3F25/F26 (carrier transfer), S3F27/F28 (cancel carrier).
### S5 — Alarms + exception recovery
S5F1/F2 (alarm set/clear), S5F3/F4 (enable/disable alarm),
S5F5/F6 (list all alarms), S5F7/F8 (list enabled alarms),
S5F9–F18 (exception recovery, chapter 19).
### S6 — Data collection
S6F1/F2 (trace data), S6F11/F12 (event report), S6F15/F16 (event
report request), S6F19/F20 (individual report request),
S6F21/F22 (annotated individual report), S6F23/F24 (spool data
transmit/purge), S6F25/F26 (spool notification).
### S7 — Process program management
S7F1/F2 (PP load inquire), S7F3/F4 (PP send unformatted),
S7F5/F6 (PP request), S7F17/F18 (PP delete), S7F19/F20 (PP
namelist), S7F23/F24 (formatted PP send, E42), S7F25/F26
(formatted PP request).
### S9 — Protocol-error reports
S9F1 (unrecognized device ID), S9F3 (unrecognized stream),
S9F5 (unrecognized function), S9F7 (illegal data), S9F9 (T3
timeout), S9F11 (data too long), S9F13 (conversation timer
timeout). Auto-emitted; see chapter 11.
### S10 — Terminal services
S10F1/F2 (terminal display single, equipment→host),
S10F3/F4 (terminal display single, host→equipment),
S10F5/F6 (terminal display multi, host→equipment).
### S12 — Wafer maps
S12F* — Per E5 §13. Round-tripped through `raw_gem300_harness.py`.
### S14 — Object services (E39) + control jobs (E94)
S14F1/F2 (GetAttr), S14F3/F4 (SetAttr), S14F9/F10 (CreateCJ),
S14F11/F12 (DeleteCJ).
### S16 — Process jobs (E40)
S16F5/F6 (PRJobCommand), S16F7/F8 (PRJobMonitor), S16F9 (PRJobAlert
— unsolicited), S16F11/F12 (PRJobCreate), S16F13/F14 (PRJobDequeue),
S16F27/F28 (CJCommand).
For per-message body shapes, look up the YAML entry in
`data/messages.yaml`.
---
## YAML schema reference
### `data/messages.yaml`
```yaml
messages:
- id: S<X>F<Y> # required, must match (stream, function)
stream: <int 1-127>
function: <int 0-255>
w: <bool> # reply expected?
builder: <ident> # C++ builder function name
parser: <ident> # C++ parser function name
body: <body-shape> # see chapter 31 for the grammar
```
Body shapes:
```yaml
body: none
body:
kind: scalar
item_type: ASCII | BINARY_BYTE | BOOLEAN | U1..U8 | I1..I8 | F4 | F8 | ITEM
enum: <C++ enum type> # optional
param: <name> # optional, default 'value'
body:
kind: list
struct_name: <C++ type> # optional; if set, parser returns struct
fields:
- {name: <field>, shape: <body-shape>}
- ...
body:
kind: list_of
element: <body-shape>
name: <name> # parameter name, default 'values'
```
### `data/control_state.yaml`
```yaml
transitions:
- {from: <state>, on: <event>, to: <state>, then: <state>, ack: <code>}
```
`from`: one of `EquipmentOffline | AttemptOnline | HostOffline | OnlineLocal | OnlineRemote`.
`on`: one of `operator_switch_online | operator_switch_offline | operator_switch_local | operator_switch_remote | attempt_complete | attempt_failed | host_request_online | host_request_offline`.
`to`: optional new state.
`then`: optional chained state (for AttemptOnline pass-through).
`ack`: optional ACK code (`Accept`, `NotAccept`, `AlreadyOnline`).
### `data/process_job_state.yaml`
```yaml
transitions:
- {from: <state>, on: <event>, to: <state>}
```
`from` / `to`: `Queued | SettingUp | WaitingForStart | Processing | ProcessComplete | Paused | Stopping | Aborting`.
`on`: `select | setup_complete | start | pause | resume | stop | abort | process_complete | abort_complete`.
### `data/control_job_state.yaml`
Same shape, different state/event names — see
[`include/secsgem/gem/control_job_state.hpp`](../include/secsgem/gem/control_job_state.hpp).
### `data/equipment.yaml`
```yaml
device:
mdln: <ASCII>
softrev: <ASCII>
capabilities: [<list of GEM capability strings>]
svids:
- {id: <U4>, name: <ASCII>, units: <ASCII>, type: <type>, value: <default>}
dvids:
- {id: <U4>, name: <ASCII>, units: <ASCII>, type: <type>}
ecids:
- {id: <U4>, name: <ASCII>, units: <ASCII>, type: <type>, value: <default>, min: <num>, max: <num>}
ceids:
- {id: <U4>, name: <ASCII>}
alarms:
- {id: <U4>, alcd: <byte>, text: <ASCII>}
recipes:
- {id: <ASCII>, body: <bytes-or-string>}
host_commands:
- {name: <ASCII>, ack: <ACK enum>, emit_ceid: <U4>, set_alarm: <U4>}
events:
default_reports:
- {ceid: <U4>, vids: [<U4>, ...]}
spool:
whitelist: [<stream>, ...]
persistent_dir: <path> # optional
```
Type strings: `ASCII`, `BINARY`, `BOOLEAN`, `U1`–`U8`, `I1`–`I8`,
`F4`, `F8`. Same vocabulary as `data/messages.yaml` body shapes.
For required vs optional fields per record, see the validator
checks in
[`tests/test_config_validate.cpp`](../tests/test_config_validate.cpp).
---
## Where to go next
The last chapter is the practical companion to this one: the
**recipes** for extending the codebase — adding a new SVID, host
command, state, message, store, or persistence backend. Code
patches you can copy.
Next: [→ 51 Extending the codebase](51_extending_the_codebase.md)
docs/51_extending_the_codebase.md
+439
View File
@@ -0,0 +1,439 @@
# 51 — Extending the codebase
← [50 API + messages + YAML reference](50_api_messages_yaml_reference.md) | [Back to index](00_index.md) | End of series.
Last chapter. Practical recipes for the seven most common
extensions, each with the actual mechanical steps. Roughly
ordered from "no C++ at all" to "the most C++ you'll write."
| Recipe | C++ needed? |
|-----------------------------------------------|---------------------|
| 1. New SVID / DVID / ECID | None |
| 2. New CEID with linked reports | None |
| 3. New host command | None |
| 4. New control-state transition | None |
| 5. New SECS-II message | Handler only |
| 6. New store | New header + tests |
| 7. New persistence backend | Substantial |
For each one: the YAML change (if any), the C++ change (if any),
the test to add, and where to look up details.
---
## 1. New SVID / DVID / ECID
The simplest extension. Add one line to `data/equipment.yaml`:
```yaml
svids:
# ... existing entries ...
- {id: 50, name: ChamberTemp, units: "C", type: F4, value: 25.0}
```
Restart. Done.
Host can now read SVID 50 via:
- `S1F11 [50]` → returns its name and units.
- `S1F3 [50]` → returns its current value.
The EAP can update it at any time:
```cpp
model->svids.set_value(50, secs2::Item::f4(new_temperature));
```
Same pattern for DVIDs and ECIDs. For ECIDs add `min` and `max`
for range validation.
**Test**: not required for new SVIDs alone, but
[`tests/test_data_model.cpp`](../tests/test_data_model.cpp) shows
the pattern.
**Reference**: chapter 31 §New SVID;
[`docs/COMPLIANCE.md`](COMPLIANCE.md) §4 (Variable / Status /
Constant rows).
---
## 2. New CEID with linked reports
Two-step YAML edit:
```yaml
# data/equipment.yaml
ceids:
- {id: 500, name: ChamberTempHigh}
events:
default_reports:
- {ceid: 500, vids: [50]} # link to the SVID we just added
```
Restart. Done. When the EAP fires CEID 500:
```cpp
on_temp_threshold_exceeded(float temp) {
asio::post(io, [model, temp] {
if (!model->is_event_enabled(500)) return;
auto reports = model->compose_reports_for(500);
auto msg = build_s6f11(500, reports);
deliver_or_spool(*conn, *model, std::move(msg));
});
}
```
S6F11 lands at the host with `[RPTID=..., V=[chamber_temp]]`.
The host can re-link reports dynamically via S2F33/F35/F37 — the
`default_reports` YAML entry is just the initial state.
**Test**: pattern in
[`tests/test_wire_ceid_emission.cpp`](../tests/test_wire_ceid_emission.cpp).
---
## 3. New host command
Add to `data/equipment.yaml`:
```yaml
host_commands:
- {name: VENT,
ack: Accept,
emit_ceid: 400,
set_alarm: 2}
```
Restart. Done. Host sends `S2F41(RCMD="VENT")`:
- `HCACK = 0` (Accept).
- CEID 400 fires → S6F11.
- ALID 2 set → S5F1.
For commands with **application logic** beyond emit-CEID +
set-alarm, register a custom handler:
```cpp
// At startup:
model->commands.register_handler("VENT",
[model](const ParamList& params) -> CommandOutcome {
// Actually vent the chamber here.
if (!vacuum_safe_to_vent()) {
return {HostCmdAck::CannotPerformNow, {}};
}
hardware_vent_chamber();
return {HostCmdAck::Accept, {}};
});
```
The registered handler overrides the YAML-defined default.
**Reference**: chapter 31 §New host command;
[`include/secsgem/gem/store/host_commands.hpp`](../include/secsgem/gem/store/host_commands.hpp).
---
## 4. New control-state transition
Edit `data/control_state.yaml`:
```yaml
transitions:
# ... existing rows ...
- {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept}
```
Restart. The transition is now active. No code changes.
For transitions chaining through `AttemptOnline`, use `then`:
```yaml
- {from: EquipmentOffline, on: operator_switch_online,
to: AttemptOnline, then: OnlineRemote, ack: Accept}
```
Same pattern for `process_job_state.yaml` and
`control_job_state.yaml`.
**Test**: pattern in
[`tests/test_control_state.cpp`](../tests/test_control_state.cpp).
---
## 5. New SECS-II message
Two-part: YAML for the wire shape, C++ for the handler.
### 5a. Add the message to the catalog
```yaml
# data/messages.yaml
- id: S6F30
stream: 6
function: 30
w: true
builder: s6f30_query
parser: parse_s6f30
body:
kind: list
struct_name: TemperatureQuery
fields:
- {name: vid, shape: {kind: scalar, item_type: U4}}
- {name: threshold, shape: {kind: scalar, item_type: F4}}
- id: S6F31
stream: 6
function: 31
w: false
builder: s6f31_query_reply
parser: parse_s6f31
body:
kind: scalar
item_type: BOOLEAN
param: above_threshold
```
`docker compose run --rm builder` regenerates `messages.hpp`.
The codegen produces:
```cpp
struct TemperatureQuery {
uint32_t vid;
float threshold;
};
inline secs2::Message s6f30_query(uint32_t vid, float threshold);
inline std::optional<TemperatureQuery> parse_s6f30(const secs2::Item&);
inline secs2::Message s6f31_query_reply(bool above_threshold);
inline std::optional<bool> parse_s6f31(const secs2::Item&);
```
### 5b. Register a handler
```cpp
router->on(6, 30, [model](const secs2::Message& m) {
auto query = messages::parse_s6f30(m.body());
if (!query) return messages::s6f31_query_reply(false);
auto val = model->svids.value(query->vid);
if (!val) return messages::s6f31_query_reply(false);
float current = std::get<std::vector<float>>(val->storage())[0];
return messages::s6f31_query_reply(current > query->threshold);
});
```
That's it. The new S/F is on the wire.
**Reference**: chapter 31 §New SECS-II message;
[`tests/test_messages.cpp`](../tests/test_messages.cpp) for the
testing pattern.
---
## 6. New store
When you need a record type that doesn't map onto an existing
store. E.g., add a `ReticleStore` for lithography reticles
distinct from substrates.
### Create the header
```cpp
// include/secsgem/gem/store/reticles.hpp
#pragma once
#include <map>
#include <string>
#include <vector>
namespace secsgem::gem {
enum class ReticleState : uint8_t {
Loaded = 0,
Aligned = 1,
InUse = 2,
Unloaded = 3,
};
struct ReticleRecord {
std::string id;
ReticleState state;
int usage_count;
};
class ReticleStore {
public:
using ChangeHandler =
std::function<void(const std::string&, ReticleState from, ReticleState to)>;
void register_reticle(std::string id);
void set_state(const std::string& id, ReticleState s);
std::optional<ReticleRecord> get(const std::string& id) const;
std::vector<ReticleRecord> all() const;
void set_change_handler(ChangeHandler h) { on_change_ = std::move(h); }
private:
std::map<std::string, ReticleRecord> records_;
ChangeHandler on_change_;
};
} // namespace secsgem::gem
```
### Add to EquipmentDataModel
```cpp
// include/secsgem/gem/data_model.hpp
struct EquipmentDataModel {
// ... existing members ...
ReticleStore reticles;
};
```
### Write tests
```cpp
// tests/test_reticles.cpp
#include "secsgem/gem/store/reticles.hpp"
#include <doctest/doctest.h>
using secsgem::gem::ReticleStore;
using secsgem::gem::ReticleState;
TEST_CASE("ReticleStore: register and look up") {
ReticleStore s;
s.register_reticle("R-001");
auto r = s.get("R-001");
REQUIRE(r.has_value());
CHECK(r->id == "R-001");
}
TEST_CASE("ReticleStore: state change fires handler") {
ReticleStore s;
s.register_reticle("R-002");
ReticleState observed_from{}, observed_to{};
s.set_change_handler([&](auto& id, auto from, auto to) {
observed_from = from;
observed_to = to;
});
s.set_state("R-002", ReticleState::Aligned);
CHECK(observed_from == ReticleState::Loaded);
CHECK(observed_to == ReticleState::Aligned);
}
```
CMake picks up new tests automatically (glob over `tests/*.cpp`).
### Wire Router handlers if needed
If reticles need wire access (e.g., a custom S6FX request), add the
message to `data/messages.yaml` (recipe 5) and register handlers.
**Reference**: chapter 32 §How to add a new store.
---
## 7. New persistence backend
The codebase ships file-backed persistence with per-record files
(chapter 36). Some deployments want different backends — SQLite,
LMDB, a key-value cache.
The persistence is wired *inside each store* rather than through
an abstraction, so changing the backend means changing each
store's `enable_persistence` implementation. Two approaches:
### 7a. Drop-in replacement
Replace the file IO inside each store's `journal_write` /
`journal_remove` / `journal_replay` methods with calls to your
backend.
Pros: no API change, no test churn.
Cons: changes 7 stores; you have to update each one.
### 7b. Pluggable backend
Introduce an interface:
```cpp
class JournalBackend {
public:
virtual ~JournalBackend() = default;
virtual void write(std::string_view key, const std::vector<uint8_t>&) = 0;
virtual std::optional<std::vector<uint8_t>> read(std::string_view key) = 0;
virtual void remove(std::string_view key) = 0;
virtual std::vector<std::string> list_keys() = 0;
};
```
Each store accepts a `std::shared_ptr<JournalBackend>`. The
default implementation is `FileJournalBackend` (current behaviour);
alternatives can be `SqliteJournalBackend`, `LmdbJournalBackend`,
etc.
Pros: clean separation, multiple backends coexist.
Cons: substantial refactor across 7 stores + their tests.
For most deployments option 7a is the right call — the file
backend is fast enough that swap-outs are rare.
**Reference**: chapter 36 §The per-record file pattern;
[`tests/test_persistence_upgrade.cpp`](../tests/test_persistence_upgrade.cpp)
for the test patterns.
---
## What to do when something doesn't fit any recipe
Some extensions don't map onto these seven. Examples:
- A new SEMI standard the codebase doesn't implement.
- A transport that isn't HSMS or SECS-I.
- A different codec (highly unusual).
- A different YAML schema (e.g., a third-party format).
For these, the right move is to:
1. **Open an issue / RFC** describing what you want.
2. **Sketch the API change** before writing code.
3. **Add tests first** — at the integration layer.
4. **Reach into the right namespace** based on the chapter map at
the top of this guide.
The codebase is small and the layering is clean; major extensions
usually fit naturally into one of the 21 stores or one of the
existing namespaces. Resist the temptation to add a new abstraction
layer; almost everything that looks like it needs one actually
fits as a new store + a few handlers.
---
## End of the guide
You've reached the end. You should now be able to:
- Read any SECS/GEM standard and recognise its shape.
- Read any commit in this codebase and place it in the
architecture.
- Read any wire trace and trace it back to a Router handler.
- Add new SVIDs / CEIDs / commands / states / messages without
recompiling.
- Add new stores or wire to new SECS standards with confidence.
- Stand up the demo, drive every external validator, and reason
about deployment + monitoring + security.
If anything in the codebase still surprises you, the chapter map
at [`docs/00_index.md`](00_index.md) is your starting point for
finding the relevant section.
The proofs in [`docs/PROOFS.md`](PROOFS.md) are the **claim**;
this guide was the **explanation**. Treat them as paired
documents.
← [Back to index](00_index.md)