docs: customer integration tutorial (INTEGRATION.md)

End-to-end guide for an equipment vendor integrating the library
into a real semiconductor tool:

  1. Architecture: what the runtime provides vs what the application
     contributes — three boundary classes (EquipmentDataModel,
     Router, hsms::Connection).
  2. 30-minute first connection: YAML + minimal main() + run.
  3. Wiring real sensors to SVIDs.
  4. Plugging the FSMs into the tool: EPT, carriers, substrates,
     E40 PJ / E94 CJ, alarms, recoverable exceptions.
  5. Persistence: enable_persistence(dir) per store, storage budget,
     replay semantics, current caveats.
  6. Monitoring + observability: connection lifecycle hooks,
     state-change handlers, S9 protocol errors.
  7. Recommended deployment layout (/opt/acme-secsgem/...).
  8. Integration testing checklist.
  9. When to extend the runtime.
 10. The honest gap between "this stack runs" and "this is a
     certified GEM tool".

Cross-referenced from COMPLIANCE.md §9 distinction (stack vs tool).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

INTEGRATION.md

@@ -0,0 +1,473 @@
+# Integration tutorial
+
+How a semiconductor equipment vendor takes this library and turns it
+into a SECS/GEM-compliant interface on a real tool.
+
+The library gives you **the runtime stack** — wire codecs, the HSMS
+connection state machine, every GEM 300 sub-state-machine, persistent
+stores, the message catalog, and a dispatcher.  What you bring is
+**the application**: knowledge of your tool's real sensors, recipes,
+alarms, processing states, and chamber I/O.  This guide walks through
+how those two halves meet.
+
+> **Audience.** Firmware / controls engineers integrating
+> SECS/GEM on a tool for the first time.  Familiarity with SEMI
+> E5/E30/E37 helps but isn't required — every spec reference is
+> pinned in `COMPLIANCE.md`.
+
+---
+
+## 1.  What you get vs. what you build
+
+```
+┌───────────────────────────────────────────────────────────┐
+│         your equipment application (you write)            │
+│  recipe runner • sensor polling • alarm sources • UI hooks│
+├───────────────────────────────────────────────────────────┤
+│            secs-gem runtime stack (this library)          │
+│  data model • FSMs • SECS-II codec • HSMS connection      │
+│  message catalog • routers • persistence • spool          │
+├───────────────────────────────────────────────────────────┤
+│      OS + Asio (provided) + your serial/Ethernet driver   │
+└───────────────────────────────────────────────────────────┘
+```
+
+The boundary lives at three classes:
+
+- `gem::EquipmentDataModel` — the data dictionary (SVIDs, ECIDs,
+  CEIDs, alarms, recipes, jobs, carriers, substrates …).  Your
+  application reads/writes it; the dispatcher serves it on the wire.
+- `gem::Router` — maps `(stream, function) → handler`.  Wire it once
+  at startup; messages flow through it.
+- `hsms::Connection` (or `secsi::TcpTransport` for SECS-I) — the
+  byte-level transport.  You feed it a TCP socket and a router; it
+  runs.
+
+You don't subclass the FSMs.  You don't write parsers.  You don't
+patch the dispatcher.  Your code lives in two places: **YAML**
+(static configuration) and **callbacks** (dynamic glue).
+
+---
+
+## 2.  The 30-minute first connection
+
+The shortest path from `git clone` to "a host can talk to my tool":
+
+### 2.1.  Describe your tool in YAML
+
+Copy `data/equipment.yaml`, rename to your tool, and edit:
+
+```yaml
+device:
+  id: 1                            # E37 SESSION-ID
+  model_name: "ACME-PVD-3000"
+  software_rev: "1.4.2"
+  equipment_type: "PVD"            # S1F20 EQPTYP
+
+svids:                              # status variables (read-only)
+  - {id: 1, name: ControlState,  units: "",      type: ASCII,   value: ""}
+  - {id: 100, name: ChamberPressureTorr, units: "Torr", type: F4, value: 0.0}
+  - {id: 101, name: WaferCounter, units: "wafer", type: U4,      value: 0}
+
+ecids:                              # equipment constants (host can set)
+  - {id: 10, name: ChamberSetpointTorr, units: "Torr", type: F4,
+     value: 1.0e-6, min: "1.0e-9", max: "1.0"}
+
+ceids:                              # collection events
+  - {id: 300, name: ProcessStarted}
+  - {id: 301, name: ProcessCompleted}
+
+alarms:
+  - {id: 1, text: "Chamber pressure out of range", category: 4}
+
+recipes:
+  - {id: "RECIPE-A", body: "STEP HEAT 350C 60s\nEND"}
+
+host_commands:
+  - {name: START, ack: Accept, emit_ceid: 300}
+  - {name: STOP,  ack: Accept}
+```
+
+That's the GEM data dictionary.  The library will serve every
+S1F3/F11, S2F13/F29, S2F33-F38, S5F5, S7F19, S2F41, etc. against
+this YAML without any C++ changes.
+
+### 2.2.  Stand it up
+
+A minimal `main()` looks like `apps/secs_server.cpp`.  In your code:
+
+```cpp
+auto model = std::make_shared<gem::EquipmentDataModel>();
+auto desc  = config::load_equipment("/etc/acme/equipment.yaml", *model);
+auto sm    = std::make_shared<gem::ControlStateMachine>(
+                 gem::ControlStateMachine::default_table(),
+                 gem::ControlState::HostOffline);
+
+asio::io_context io;
+Server server(io, {/*port=*/5000, desc.device_id, {}});
+
+server.on_accept([&](std::shared_ptr<hsms::Connection> conn) {
+    auto router = std::make_shared<gem::Router>();
+    register_default_handlers(*router, model, sm, conn);  // your function
+    conn->set_message_handler([router, conn](const secs2::Message& m) {
+        return router->dispatch_with_s9(
+            [&](uint8_t f, const std::array<uint8_t, 10>& mhead) {
+                conn->emit_s9(f, mhead);
+            },
+            [&]() -> std::optional<std::array<uint8_t, 10>> {
+                auto* h = conn->current_header();
+                return h ? std::optional{h->encode()} : std::nullopt;
+            }, m);
+    });
+});
+
+server.start();
+io.run();
+```
+
+`register_default_handlers` is the only piece of glue you write at
+the start.  The repo's `apps/secs_server.cpp` is a complete worked
+example — copy it verbatim, then customize the YAML to your tool.
+
+### 2.3.  Run it
+
+```sh
+docker compose up server                # or your own deployment
+# host fires up secsgem-py / wonderware / equipment manager:
+#   selects, S1F13, S1F1, S1F3 → you're talking GEM.
+```
+
+That's the floor.  From here, every section below adds capability.
+
+---
+
+## 3.  Wiring real sensors to SVIDs
+
+The YAML's `value:` field is the *initial* value.  Your application
+updates the live value as the tool runs:
+
+```cpp
+// In your sensor-poll thread (running on a separate executor):
+double torr = read_baratron();
+model->svids.set_value(/*ChamberPressure=*/100, secs2::Item::f4(float(torr)));
+```
+
+That's it — the next S1F3 from the host returns the fresh value.
+
+Two patterns scale well:
+
+1. **One updater per sensor, fixed cadence.**  Each sensor's driver
+   owns the (vid, set_value) pair.
+2. **A single refresh tick.**  A periodic timer dumps all polled
+   values at once (`refresh()` in `apps/secs_server.cpp` does this
+   for two virtual SVIDs).
+
+The SECS-II Item shape must match the YAML's `type:`.  If the YAML
+says `F4` and you call `set_value(100, secs2::Item::ascii("..."))`,
+the host will get the string back — the library doesn't enforce a
+runtime check.  Treat the YAML type as a contract you maintain.
+
+---
+
+## 4.  Plugging the FSMs into your tool
+
+Every GEM 300 sub-state-machine in the library is a behavior model.
+You decide *when* state transitions happen by firing events:
+
+### 4.1.  Equipment processing (E116 EPT)
+
+```cpp
+// At startup or whenever the operator clicks "Run":
+model->ept.on_event(gem::EptEvent::EnterStandby);
+model->ept.on_event(gem::EptEvent::EnterProductive);
+
+// Auto-emit S6F11(ControlEvent_*) on every transition:
+model->ept.set_state_change_handler(
+    [&](gem::EptState, gem::EptState to, gem::EptEvent,
+        std::chrono::milliseconds /*dwell*/) {
+        uint32_t ceid = ept_state_to_ceid(to);  // your switch/case
+        if (!ceid || !model->is_event_enabled(ceid)) return;
+        conn->send_data(gem::s6f11_event_report(
+            next_dataid++, ceid, model->compose_reports_for(ceid)));
+    });
+```
+
+Helpers:
+- `model->ept.accumulated(state)` — total milliseconds spent in
+  `state` since startup.  Use it to populate E116 SVIDs.
+- `model->ept.reset_history()` — call at shift boundary.
+
+### 4.2.  Carriers + load ports (E87)
+
+When AMHS delivers a carrier:
+
+```cpp
+model->carriers.create("CAR-A1B2", /*port=*/1, /*capacity=*/25);
+model->carriers.fire_id_event("CAR-A1B2", gem::CarrierIDEvent::Read);
+// ... host sends S3F17(ProceedWithCarrier), the registered handler
+// in the Router calls fire_id_event(..., ProceedWithCarrier) and
+// CIDS moves NotConfirmed → Confirmed.
+```
+
+When your slot-map scanner finishes:
+
+```cpp
+auto* c = model->carriers.get("CAR-A1B2");
+for (std::size_t i = 0; i < scan_result.size(); ++i)
+    c->slots[i].state = scan_result[i];   // 0 empty, 1 occupied
+model->carriers.fire_slot_map_event("CAR-A1B2", gem::SlotMapEvent::Read);
+```
+
+The S3F19/F20 verify handler will compare against this map.
+
+### 4.3.  Substrates (E90)
+
+For each wafer you start tracking:
+
+```cpp
+model->substrates.create("W-2024-001", "CAR-A1B2", /*slot=*/1);
+model->substrates.fire_location_event(
+    "W-2024-001", gem::SubstrateEvent::Acquire, /*location=*/"ChamberA");
+model->substrates.fire_processing_event(
+    "W-2024-001", gem::SubstrateProcessingEvent::StartProcessing);
+// ... when done:
+model->substrates.fire_processing_event(
+    "W-2024-001", gem::SubstrateProcessingEvent::EndProcessing);
+model->substrates.fire_location_event(
+    "W-2024-001", gem::SubstrateEvent::Release, /*location=*/"OutCarrier");
+```
+
+History is tracked per-substrate (`model->substrates.history(id)`)
+and can power your downtime / yield reports.
+
+### 4.4.  Process jobs + control jobs (E40 / E94)
+
+The host creates these via S16F11 / S14F9.  Your application drives
+their internal transitions as the recipe engine progresses:
+
+```cpp
+// Recipe runner reports setup done:
+model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::SetupComplete);
+// Operator hits Start (or autorun is on):
+model->process_jobs.on_host_command("PJ-1", gem::ProcessJobEvent::Start);
+// Recipe completed normally:
+model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::ProcessComplete);
+```
+
+CJ state cascades the same way (E94).
+
+### 4.5.  Alarms (E5 §13)
+
+```cpp
+// Sensor crosses threshold:
+model->alarms.set(/*alid=*/1, /*set=*/true);  // emits S5F1(ALCD=0x84)
+// Later it clears:
+model->alarms.set(1, false);                  // emits S5F1(ALCD=0x04)
+```
+
+The dispatcher takes care of the wire frame — you just toggle.
+
+### 4.6.  Recoverable exceptions (E5 §9, S5F9–F18)
+
+For faults where you want a host/equipment recovery dialogue:
+
+```cpp
+model->exceptions.post(/*exid=*/42, "VACUUM",
+                       "lost vacuum in chamber A",
+                       {"PURGE", "RECOVER", "ABORT"});  // emits S5F9
+// Host picks PURGE via S5F13; the registered handler calls
+// model->exceptions.on_recover(42, "PURGE"), state moves to Recovering.
+// Your purge routine completes:
+model->exceptions.fire_internal(42, gem::ExceptionEvent::RecoveryComplete);
+// state → Cleared; S5F11 fires; entry removed.
+```
+
+---
+
+## 5.  Persistence
+
+GEM equipment that loses power mid-job can recover gracefully
+because every store the library ships supports an opt-in file-backed
+journal.  Enable per store, at startup, BEFORE the connection comes up:
+
+```cpp
+auto base = std::filesystem::path("/var/lib/acme/secsgem");
+model->spool.enable_persistence(base / "spool");
+model->carriers.enable_persistence(base / "carriers");
+model->load_ports.enable_persistence(base / "loadports");
+model->substrates.enable_persistence(base / "substrates");
+model->process_jobs.enable_persistence(base / "pjobs");
+model->control_jobs.enable_persistence(base / "cjobs");
+model->exceptions.enable_persistence(base / "exceptions");
+```
+
+On enable, the store scans the directory, replays records into
+in-memory state, and from there keeps the directory in sync on
+every create / state-change / remove.  Writes use a
+`.tmp + rename` pattern so a power loss mid-write can lose at most
+the in-flight record (older records remain coherent).
+
+Storage budget per store, roughly:
+- spool: one file per spooled S6F11 (typically tens of bytes each)
+- carriers: one file per carrier (~50 bytes + slot count)
+- load_ports: one file per LP (~30 bytes)
+- substrates: one file per wafer (~80 bytes)
+- pjobs: one file per active PJ (~100 bytes), plus `order.idx`
+- cjobs: one file per active CJ (~80 bytes)
+- exceptions: one file per Posted/Recovering exception
+
+Even a busy fab tool tops out at a few hundred files in each
+directory — well within filesystem caps.  Sweep terminal-state
+entries (completed PJs, cleared exceptions) periodically if you
+care about directory size.
+
+Caveats currently captured in the persistence tests:
+- Substrate **history** is intentionally NOT journaled — only the
+  *current* state of each axis.  Replay starts with an empty
+  history vector.
+- PJ `rcpvars` / `prprocessparams` (the optional E40 `secs2::Item`
+  trailers) are not journaled in v1; call `set_e40_extras` again on
+  the application side after restart if you need them.
+
+---
+
+## 6.  Monitoring + observability
+
+### 6.1.  Connection lifecycle
+
+```cpp
+conn->set_log_handler([](const std::string& m) {
+    syslog(LOG_INFO, "hsms: %s", m.c_str());
+});
+conn->set_selected_handler([] { metrics.inc("hsms.selected"); });
+conn->set_closed_handler([](const std::string& r) {
+    metrics.inc("hsms.closed", {{"reason", r}});
+});
+```
+
+### 6.2.  State change observers
+
+Every store / FSM exposes a `set_*_change_handler`.  Hook them up
+to your metrics / log pipeline:
+
+```cpp
+model->control_jobs.set_state_change_handler(
+    [](const std::string& cj, gem::ControlJobState f, gem::ControlJobState t,
+       gem::ControlJobEvent) {
+        log("CJ " + cj + " " + gem::control_job_state_name(f) +
+            " → " + gem::control_job_state_name(t));
+    });
+```
+
+### 6.3.  Self-emitted protocol errors
+
+Look for `S9F*` traffic in your logs.  S9F3 / F5 mean the host
+asked for something your router doesn't handle; S9F7 means a bad
+body arrived; S9F9 means a reply didn't arrive in T3; S9F11 means
+a frame exceeded the 16 MiB cap.  None of these are normal — they're
+real diagnostic events.
+
+---
+
+## 7.  Recommended layout for a vendor application
+
+```
+/opt/acme-secsgem/
+├─ bin/
+│  └─ secsgem-equipment          # your built binary
+├─ etc/
+│  ├─ equipment.yaml             # your tool's dictionary
+│  └─ control_state.yaml         # your tool-specific state model
+├─ var/
+│  ├─ spool/                     # populated at runtime
+│  ├─ carriers/
+│  ├─ substrates/
+│  ├─ pjobs/
+│  ├─ cjobs/
+│  └─ exceptions/
+└─ share/
+   └─ doc/                       # COMPLIANCE.md, INTEGRATION.md
+```
+
+Your application reads `etc/`, writes to `var/`, and never touches
+`share/`.  YAML edits don't require a rebuild — restart the
+process.
+
+The control-state YAML is your tool's *processing* state machine —
+E30 deliberately leaves the concrete states (IDLE / SETUP / READY /
+EXECUTING / PAUSE / …) up to the tool builder.  Copy
+`data/control_state.yaml` as a starting point.
+
+---
+
+## 8.  Test the integration
+
+Don't ship without:
+
+1. **Round-trip every host-facing message you serve.**  The library's
+   own test suite covers the codecs; you should also drive your
+   YAML's specific SVIDs / CEIDs / alarms / recipes against the
+   built-in passive server using the `interop/host_vs_cpp_server.py`
+   harness as a template.
+
+2. **Power-loss simulation.**  Kill -9 the process mid-job, restart,
+   confirm the stores replay the correct state.  The persistence
+   tests give you a template; copy and parameterize for your store
+   directories.
+
+3. **Multi-hour soak.**  Spool fills up if persistence is enabled and
+   the host link is down — make sure your fab's MES side ack-rate
+   keeps up.  Run a 24h test with the host periodically disconnecting
+   and watch the journal directory.
+
+4. **The two-container demo** in this repo gives you a starting
+   harness — the host emulator (`apps/secs_client.cpp`) drives
+   ~20 transactions through your server.  Adapt it to your message
+   set.
+
+---
+
+## 9.  When to extend the runtime
+
+The library is open to extension.  Common reasons to add code:
+
+- **A new SECS-II message** the catalog doesn't cover.  Edit
+  `data/messages.yaml`, run the codegen (built into the CMake
+  pipeline), add a Router handler.  No core code change.
+- **A new state machine** specific to your tool (e.g. an in-chamber
+  cooling cycle FSM).  Lift the pattern from `ept_state.hpp`:
+  define your states + events + transition table; let your
+  application drive it.
+- **An additional persistence backend** (DB instead of files).
+  Mirror the spool `.enable_persistence` pattern — it's about 100
+  lines per store.
+
+If your change is broadly useful, it's worth landing in the library
+itself.  See `COMPLIANCE.md` for the standards still on the
+"explicitly out of scope" list — anything there is a possible
+contribution.
+
+---
+
+## 10.  Going from "stack" to "certified GEM tool"
+
+This codebase passes its own conformance harness and cross-validates
+against `secsgem-py`, but a real *certified* GEM tool needs more:
+
+- **Independent third-party validator**.  Run a GEM RTS (Reference
+  Test System) or equivalent against your integration.  The library
+  serves the messages; the validator decides whether your data is
+  consistent.
+
+- **Vendor application code**.  The runtime cannot, by design, know
+  what your SVID values *should* be at any given moment.  That's
+  your domain knowledge plugged into the data model and FSMs.
+
+- **Documentation**.  E30 §6.10 (Documentation capability) requires
+  you to publish what you implement.  `COMPLIANCE.md` in this repo
+  is the template — fork it, prune to your actual coverage, ship
+  it with your tool.
+
+- **Operations**: monitoring dashboards, alarm escalation, log
+  retention — the standard SRE concerns, no different from any
+  other piece of fab software.