docs: chapters 40, 41, 50, 51 — Operations + Reference (series complete)

Last four chapters of the guided tour. 40 — Building, running, the demo. Docker prerequisites, the build flow, what each binary is for, running the 24-transaction demo flow annotated step by step. Running the 4 external-validator sweeps + the libFuzzer pass. Inspecting the demo with tcpdump and tshark. Reading source while running as the recommended learning workflow. 41 — Integration: hardware, MES, production. Four-phase tour: wiring sensors / recipe engine / alarms / E84 GPIO; talking to a real MES with the day-1 punch list + commercial-MES quirks (Wonderware S2F21, Camstar Linktest cadence, etc.); production hardening (nftables / stunnel / minisign / persistence layout / monitoring / runbook); performance envelope + memory footprint + capacity planning. Pointers to the long-form INTEGRATION.md / MES_INTEROP.md / SECURITY.md / BENCHMARKS.md. 50 — API + message catalog + YAML schemas reference. Namespace-by- namespace table of public symbols (secs2, hsms, secsi, gem, config, metrics) with brief descriptions. Stream-by-stream message catalog reference (S1, S2, S3, S5, S6, S7, S9, S10, S12, S14, S16). YAML schema reference for messages.yaml + the three state-table files + equipment.yaml. 51 — Extending the codebase. Seven recipes ordered from no-code to substantial: new SVID/DVID/ECID (YAML only), new CEID with reports (YAML only), new host command (YAML + optional handler), new control- state transition (YAML only), new SECS-II message (YAML + handler), new store (header + tests), new persistence backend (drop-in vs pluggable trade-off). Each recipe has the actual mechanical steps, the test pattern, and pointers to the chapter that explains why it works. Index updated to mark all 24 chapters published. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-09 20:28:21 +02:00
parent cae98d9a7d
commit 31f908e1bf
5 changed files with 1513 additions and 7 deletions
@@ -0,0 +1,350 @@
+# 41 — Integration: hardware, MES, production
+
+← [40 Building, running, the demo](40_building_running_demo.md) | [Back to index](00_index.md) | Next: [50 API + messages + YAML reference](50_api_messages_yaml_reference.md) →
+
+You have the demo running.  Now you need to make it talk to a
+**real tool**, against a **real MES**, in a **real fab**.
+
+This chapter walks the four phases of that journey:
+
+1. **Wire the EAP to physical hardware.**
+2. **Integrate with a commercial MES.**
+3. **Production hardening** — security, monitoring, persistence.
+4. **Operational concerns** — performance, capacity, incidents.
+
+This is a compressed view of the long-form
+[`docs/INTEGRATION.md`](INTEGRATION.md); cross-references are
+inline.  The long-form has more code and more configuration; this
+chapter explains the *shape* of each phase.
+
+---
+
+## Phase 1 — wiring to hardware
+
+### What "the EAP" actually does
+
+The EAP (Equipment Automation Program) sits between the SECS/GEM
+runtime and the **physical tool**.  It does four things:
+
+1. **Reads sensors** at the right cadence and updates SVIDs.
+2. **Drives the recipe engine** when a host command arrives.
+3. **Listens for alarms** from PLCs / hardware fault lines.
+4. **Wires FSM transitions** to CEID emissions.
+
+[`examples/pvd_tool/main.cpp`](../examples/pvd_tool/main.cpp) is
+the worked reference.  Section by section:
+
+| Section in main.cpp           | What it shows                                                          |
+|-------------------------------|------------------------------------------------------------------------|
+| §1 Helpers + constants        | The `kSvidX / kCeidX` constants worth pinning at file scope            |
+| §2 Sensor simulator           | Multi-cadence sensor poll loops with `asio::post` strand-marshal       |
+| §3 Recipe runner              | PJ → SettingUp → Processing → ProcessComplete walk; per-step CEID emit |
+| §4 Alarm threshold monitor    | Continuous threshold evaluation against ECID setpoints                 |
+| §5 EPT cycling                | E116 transitions driven by PJ state + safety alarms                    |
+| §6 Router handlers            | 51 handlers in ~460 lines — every S/F a host might send to a PVD tool  |
+| §7 main()                     | YAML load → validate → compose → run                                   |
+
+A real tool fork:
+
+```bash
+cp -r examples/pvd_tool/ src/my_tool/
+# edit src/my_tool/equipment.yaml — your tool's SVIDs/CEIDs/alarms
+# edit src/my_tool/main.cpp — replace pvd::Simulator with PLC bindings
+```
+
+### Sensor wiring
+
+The PVD example uses a random-walk simulator (§2).  A real EAP
+replaces this with calls into the tool's sensor stack:
+
+```cpp
+// Original (simulated):
+upd_f4(kSvidChamberPressure, target_pressure.load(), 1e-8f, 1e-7f);
+
+// Real:
+asio::post(io, [model](){
+  float pressure = plc_read_register(0x4001);  // from your PLC API
+  model->svids.set_value(kSvidChamberPressure, secs2::Item::f4(pressure));
+});
+```
+
+The `asio::post` is non-negotiable — the store mutation runs on
+the io_context strand (chapter 33).
+
+### Recipe runner
+
+The PVD example's recipe runner (§3) parses the recipe body and
+walks PJ states.  A real tool replaces the simulator with calls
+into the tool's recipe engine:
+
+```cpp
+void start_processing(const std::string& pjid, const std::string& ppid) {
+  auto recipe = recipes_->find(ppid);
+  if (!recipe) {
+    model->process_jobs.apply(pjid, ProcessJobEvent::Abort);
+    return;
+  }
+  // Hand the recipe to the tool's actual recipe engine.
+  hardware_recipe_engine_->start(*recipe, [model, pjid](bool ok) {
+    asio::post(io, [model, pjid, ok] {
+      model->process_jobs.apply(pjid,
+          ok ? ProcessJobEvent::ProcessComplete
+             : ProcessJobEvent::AbortComplete);
+    });
+  });
+}
+```
+
+### Alarm sources
+
+Real alarms come from:
+
+- **PLC fault lines** — interrupt callbacks.
+- **Watchdog timers** — periodic checks (cooling water flow,
+  vacuum pressure).
+- **Sensor thresholds** — continuous evaluation against ECIDs.
+- **Hardware safety interlocks** — SafetyController callbacks.
+
+Each translates to one `model->alarms.set(alid)` call.  The alarm
+dispatcher takes care of S5F1 emission, host enable filtering, and
+alarm persistence.
+
+### E84 wiring
+
+E84 needs a GPIO driver:
+
+```cpp
+// On signal change from the GPIO driver:
+void on_gpio_change(uint8_t port, E84Signal sig, bool value) {
+  asio::post(io, [model, port, sig, value]() {
+    model->e84_ports.at(port).fsm.on_signal_change(sig, value);
+  });
+}
+
+// When the FSM wants to assert a signal:
+model->e84_ports.at(port).fsm.set_emit_handler(
+    [port](E84Signal sig, bool value) {
+      gpio_driver_write(port, sig, value);
+    });
+```
+
+The TA1/TA2/TA3 timers are wall-clock; use the
+[`E84AsioTimers`](../include/secsgem/gem/e84_asio_timers.hpp)
+adapter so they fire on the same io_context.
+
+---
+
+## Phase 2 — talking to a real MES
+
+### The day-1 punch list
+
+Before you connect to a production MES, run
+[`docs/MES_INTEROP.md`](MES_INTEROP.md) against the **staging**
+MES.  59 test IDs across:
+
+- Transport (T-01 to T-09): SELECT, Linktest, T3, T7, oversized
+  frames.
+- Establishment (E-01 to E-08): S1F13, S1F1, S1F11, S1F19, …
+- Reports (R-01 to R-07): the S2F33/F35/F37 dance.
+- Alarms (A-01 to A-06).
+- Commands (C-01 to C-04): S2F41 + S2F21 (legacy) + S2F49
+  (enhanced).
+- Recipes (P-01 to P-05).
+- Terminal services (TS-01 to TS-03).
+- Jobs (J-01 to J-06).
+- Spool (SP-01 to SP-05).
+- Clock (K-01 to K-06).
+
+This is the **friction-killer document**.  Pass every test ID in
+staging and your production cutover is much less likely to
+surprise you.
+
+### HSMS-GS for multi-MES
+
+Some fabs run multiple MES against one equipment.  E37 §11
+HSMS-GS multiplexes over one TCP socket:
+
+```cpp
+auto conn = std::make_shared<hsms::Connection>(
+    std::move(sock), Mode::Passive, /*primary device_id=*/0, timers);
+
+// Production MES on session 100.
+conn->add_session(100);
+conn->set_session_message_handler(100, production_router_handler);
+
+// Maintenance MES on session 200.
+conn->add_session(200);
+conn->set_session_message_handler(200, maintenance_router_handler);
+```
+
+[`docs/INTEGRATION.md`](INTEGRATION.md) §7 has the full worked
+example with HA pattern.  Tests:
+[`tests/test_hsms_gs.cpp`](../tests/test_hsms_gs.cpp) (5 wire-level)
+and
+[`tests/test_hsms_gs_integration.cpp`](../tests/test_hsms_gs_integration.cpp)
+(1 end-to-end three-session scenario).
+
+### Things commercial MES get wrong
+
+Real MES exhibit common deviations:
+
+- **Wonderware uses S2F21 (legacy) exclusively** — no S2F41.  The
+  codebase's HostCommandRegistry handles both forms.
+- **Some MES leave EQPTYP in S1F20 confused with MDLN** — the
+  codebase accepts either; documented in
+  [`docs/MES_INTEROP.md`](MES_INTEROP.md) E-02 caveat.
+- **MES with old PPBODY handling reject binary recipes** — the
+  codebase ships both as-bytes and as-ASCII PPBODY.
+- **Camstar uses Linktest at 30 s**, others at 60 s — configure
+  `Timers::linktest` to match the host's cadence.
+
+[`docs/MES_INTEROP.md`](MES_INTEROP.md) "Caveats" column lists more.
+
+---
+
+## Phase 3 — production hardening
+
+### Security: SECURITY.md walk-through
+
+[`docs/SECURITY.md`](SECURITY.md) ships concrete configs for:
+
+- **nftables** — restrict the SECS port to the MES host's IP only.
+- **stunnel** — wrap the HSMS port in TLS so the wire is
+  encrypted.
+- **minisign** — sign every recipe (PPBODY) and verify on receive.
+- **SIEM audit log schema** — what every store mutation emits as
+  JSON for log aggregation.
+
+Configure all four before promoting to production.  No exceptions.
+
+### Persistence layout
+
+Production deployments enable persistence on every store that
+needs it.  Per [`docs/INTEGRATION.md`](INTEGRATION.md) §5:
+
+```
+/var/lib/secsgem/
+├── spool/         # SpoolStore
+├── pj/            # ProcessJobStore
+├── cj/            # ControlJobStore
+├── exceptions/    # ExceptionStore
+├── carriers/      # CarrierStore
+├── load_ports/    # LoadPortStore
+└── substrates/    # SubstrateStore
+```
+
+On an SSD with `fsync` enabled per file rewrite, this comfortably
+handles a few hundred mutations / sec.  On rotational media you'll
+want to batch or relax durability.
+
+### Monitoring
+
+Production EAPs typically export:
+
+- **Per-CEID emission counter** — burst detection.
+- **Spool depth gauge** — alarms when growing (MES connectivity
+  problem).
+- **T3 timeout counter** — non-zero means the MES is slow or your
+  T3 is too short.
+- **Per-alarm set count** — pages on certain ALIDs.
+- **Equipment EPT state gauge** — fab-wide dashboard input.
+
+[`docs/INTEGRATION.md`](INTEGRATION.md) §6.4 covers the Grafana
+panel patterns; the PVD example wires the Prometheus exporter at
+§7.
+
+### Operational runbook
+
+[`README.md`](../README.md) ships a starter runbook:
+
+| Incident                            | First check                          | Mitigation                                |
+|-------------------------------------|--------------------------------------|-------------------------------------------|
+| HSMS connection flapping            | T7 / T6 timer fires in logs          | check MES reachability, network MTU       |
+| Spool depth growing                 | host MES connectivity / ACK rate     | force-drain via S6F23, escalate to MES    |
+| State machine "stuck"               | last state-change handler log line   | host-issued offline + re-establish        |
+| Alarm storm                         | `AlarmRegistry::all()` snapshot      | check upstream sensor; quench via S5F3    |
+| Persistence dir growing unbounded   | `du -s` + file count                 | sweep terminal-state records              |
+| Cross-tool inconsistency            | `secsgem_tests` on canary tool       | compare wire trace vs validator           |
+
+---
+
+## Phase 4 — performance
+
+### The envelope
+
+Per [`docs/BENCHMARKS.md`](BENCHMARKS.md), on a 2024 M-series Mac
+under Docker Desktop:
+
+| Scenario                          | Ops/sec |  p50 µs | p95 µs | p99 µs |
+|-----------------------------------|--------:|--------:|-------:|-------:|
+| S1F1/F2 (header-only)             |  ~140 k |     74  |    103 |    161 |
+| S1F3/F4 (32 SVIDs)                |   ~79 k |    165  |    186 |    260 |
+| S6F11 push (W=0)                  |  ~572 k |    n/a  |    n/a |    n/a |
+
+A real fab tool sees **tens to a few hundred** events / s
+sustained.  We're three orders of magnitude above the push path,
+two orders above the round-trip path.
+
+**Throughput is not the bottleneck**; tail latency under
+contention is.  Tune by:
+
+- Running on a quiet host.
+- Bumping `linktest` interval up (default 0 = disabled is fine for
+  most deployments).
+- Pinning the io_context thread to a dedicated CPU.
+
+### Memory footprint
+
+| Entity              | Approx bytes / instance |
+|---------------------|------------------------:|
+| PJ + CJ pair        |                    ~450 |
+| Carrier (no slots)  |                     ~80 |
+| Carrier slot        |                     ~24 |
+| Substrate           |                    ~120 |
+| Spool entry         | ~40 + encoded body size |
+
+A busy 300 mm tool with 50 carriers × 25 slots + 200 substrates +
+20 active PJ+CJ pairs is under **1 MiB** of model state.  RSS is
+dominated by the binary itself + asio's buffers (~10–20 MiB).
+
+### Capacity planning
+
+For sizing purposes:
+
+- One io_context thread per `Connection` is plenty for any single
+  tool.
+- Multiple tools share the same process if you want — one
+  io_context, multiple `Connection`s, each on its own strand.
+- Persistence cost is one `rename(2)` per mutation; SSD-bound
+  fabs comfortably handle a few hundred / sec.
+
+---
+
+## When to read the long-form
+
+This chapter compressed phases 1–4 into a tour.  Each phase has
+substantially more material in the long-form docs:
+
+- [`docs/INTEGRATION.md`](INTEGRATION.md) — full vendor-side
+  tutorial including wiring sensors, plugging FSMs, persistence
+  layout, monitoring, HSMS-GS HA.
+- [`docs/MES_INTEROP.md`](MES_INTEROP.md) — the 59 test IDs in
+  full.
+- [`docs/SECURITY.md`](SECURITY.md) — concrete nftables / stunnel /
+  minisign configs.
+- [`docs/BENCHMARKS.md`](BENCHMARKS.md) — perf envelope + how to
+  re-run.
+
+When you actually start an integration, work from those.  This
+chapter is the map.
+
+---
+
+## End of Part 4
+
+You can now build the codebase, run the demo, drive every
+external validator, and you know the shape of how a real
+integration would land.  Part 5 is reference material — API
+namespaces, message catalog reference, the extension recipes.
+
+Next: [→ 50 API + messages + YAML reference](50_api_messages_yaml_reference.md)