From 195ecc689f83ac84754af3995933cc56303e9ff1 Mon Sep 17 00:00:00 2001 From: Raphael Maenle Date: Tue, 9 Jun 2026 16:43:45 +0200 Subject: [PATCH] docs: GLOSSARY + FAQ + interop README refresh + doc-map fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fills four documentation gaps surfaced by the doc audit: 1. README "Documentation map" was missing VERIFICATION.md (the file that backs the proof-of-feature-completeness claims) and is now pointing at the new files added in this commit too — ARCHITECTURE, GLOSSARY, FAQ, examples/pvd_tool/ (the last two land next). 2. interop/README.md only documented secsgem-py. Three of the five external validators (tshark, secs4j, libFuzzer) plus the E5 KAT were invisible from the directory's own README. Rewritten as a complete index — what's external, what each catches, how to run, what bugs they've already surfaced, when to add a new validator. 3. GLOSSARY.md is new. Every SEMI acronym used in the codebase or the docs gets one row: SVID, DVID, CEID, RPTID, ALID, ECID, PPID, MID, CARRIERID, PRJOBID, CTLJOBID, SUBSTID, OBJSPEC, OBJTYPE, MDLN, SOFTREV, EQPTYP, DATAID + every ACK code (COMMACK, ONLACK, OFLACK, HCACK, CMDA, ACKC5-7-10, DRACK, LRACK, ERACK, EAC, TIACK, GRANT, ALCD, OBJACK) + stream/function shorthand + HSMS terms + T-timers + E84 signals + the standards lineup + codebase shortcuts ("the model", "the router", "the proof", etc.). Cuts week-1 onboarding time. 4. FAQ.md is new. Canonical answers to the questions that come up once per integration: why HSMS unencrypted, SVID vs DVID, PJ vs CJ, who fires FSM transitions, what runs on which thread, how to add a new SECS-II message, ASCII vs Binary, common MES quirks, how spool works, robustness fuzz vs libFuzzer, conformance vs interop, what's not implemented. Co-Authored-By: Claude Opus 4.7 --- FAQ.md | 225 ++++++++++++++++++++++++++++++++++++++++++++++ GLOSSARY.md | 163 +++++++++++++++++++++++++++++++++ README.md | 5 ++ interop/README.md | 152 +++++++++++++++++++++++-------- 4 files changed, 506 insertions(+), 39 deletions(-) create mode 100644 FAQ.md create mode 100644 GLOSSARY.md diff --git a/FAQ.md b/FAQ.md new file mode 100644 index 0000000..2ee4e25 --- /dev/null +++ b/FAQ.md @@ -0,0 +1,225 @@ +# FAQ + +Questions we hear once per integration. Skim before you ask. If +your question isn't here and isn't obvious from the other docs, +ask once — your question probably belongs in this file and we'll +add it. + +## Why is HSMS unencrypted? + +Because SEMI E37 says so. HSMS is plain TCP with a 14-byte +framing header — no TLS, no auth, no nonces. Every commercial MES +on the market speaks exactly that wire, and changing it would make +us incompatible with all of them. Encryption and authentication +belong at the network layer: see [SECURITY.md](SECURITY.md) for +the stunnel.conf + nftables setup that wraps the unencrypted TCP +in mTLS without modifying the wire protocol. + +## What's the difference between SVID and DVID? + +**SVID** is a *status* variable — equipment state the host queries +(chamber pressure, current control state, wafer counter). +**DVID** is a *data* variable — intermediate values, typically +computed or sensor-derived, that aren't part of the equipment's +state model. + +In practice fab tools blur the line. The library treats them +identically except for which message reports them: `S1F3 / S1F11` +for SVIDs, `S1F21 / S1F22` for DVIDs. Variable lookups by VID +span both (`EquipmentDataModel::vid_value`). + +## Do I really need all four YAML files? + +Yes for production; no for a quick "does it compile": + +- `equipment.yaml` — your tool's data dictionary. Required. +- `control_state.yaml` — the E30 control state machine (HostOffline, + AttemptOnline, OnlineRemote, …). The default in `data/` works as + a starting point; you may customize transitions. +- `process_job_state.yaml` — the E40 PJ FSM. Default is spec-typical; + customize only if your tool has unusual recipe semantics. +- `control_job_state.yaml` — the E94 CJ FSM. Same. + +`secs_server --validate-config` checks all four in one pass and +exits 0 / 1. Run it in CI on every config change. + +## PJ vs CJ — what's the difference? + +A **PJ** (E40 Process Job) is "process this batch of material with +this recipe." One PJ = one recipe run = one set of wafers. It +has its own FSM (Queued → SettingUp → Processing → ProcessComplete). + +A **CJ** (E94 Control Job) is "execute these PJs in order, as a +unit, with start/pause/abort semantics." A CJ owns an ordered list +of PRJOBIDs. When the host issues `CJSTART`, the CJ promotes its +PJs through their lifecycles. + +You typically need both: the MES creates a CJ containing N PJs, +then starts the CJ. PJs without a CJ are legal — they just sit in +Queued waiting for someone to select them — but most MES drives +batches through CJs. + +## Who fires FSM transitions — the library or my code? + +**Your code.** The library implements the FSMs (legal transitions, +validation, persistence) but it doesn't know when a wafer was +actually loaded or when a recipe step finished — those signals come +from your tool. The pattern across every store is: + +```cpp +// You fire the event; the FSM validates + transitions + emits. +model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::SetupComplete); +model->carriers.fire_id_event("CAR-A1B2", gem::CarrierIDEvent::Read); +``` + +Host commands (`S2F41` RCMD=START, `S16F5` PRJSTART, `S16F27` CJSTART) +arrive via the wire and get dispatched into your registered +handlers; the handler typically calls `fire_internal` or +`on_host_command` on the relevant store. + +See INTEGRATION.md §4 for the worked patterns. + +## What runs on which thread? + +**Everything that touches the data model runs on the io_context +thread.** There are no locks in `EquipmentDataModel`. + +- The Router dispatch (incoming wire messages) — on the io_context. +- All `set_*_change_handler` callbacks — on the io_context. +- Periodic timers you register via asio — on the io_context. + +If your code lives on another thread (typical for sensor polling), +marshal updates via `asio::post`: + +```cpp +asio::post(io.get_executor(), [model, value] { + model->svids.set_value(100, secs2::Item::f4(value)); +}); +``` + +INTEGRATION.md §3 has the full thread-safety contract. + +## How do I add a new SECS-II message? + +Edit `data/messages.yaml`, add a row, rebuild. The codegen +(`tools/gen_messages.py`) emits a typed builder + parser into +`messages.hpp`. Then register a Router handler in your `main.cpp` +for the new `(stream, function)` pair. See README "Adding a +capability" or ARCHITECTURE.md for the full walkthrough. + +## What's the difference between `Item::ascii("X")` and `Item::binary({'X'})`? + +The wire format byte differs — `0x41 01 58` for ASCII vs +`0x21 01 58` for Binary. Some peers (notably secsgem-py) default +PPBODY to ASCII; others use Binary. Our codec accepts either via +the `BINARY_OR_ASCII` codegen type for fields the spec lists as +`ASCII | Binary | List` (the PPBODY case in S7F3/F6). + +For most fields it doesn't matter — pick the format that matches +your data semantically. + +## My MES sends a message that worked in `interop` but fails in production. What's going on? + +Three usual suspects: + +1. **U-width.** Your MES is sending `DATAID` as U1 but our handler + was strict for U4. We're lenient now via `any_unsigned_first`, + but if you have custom handlers in your code, use that helper + rather than `as_u4_scalar` for identifier fields. +2. **PPBODY direction.** Some MES send PPBODY as ASCII even when + the spec says it can be Binary. Use `as_text_or_binary` not + `as_binary`. +3. **Trailing fields.** Some MES add proprietary trailing fields + to S2F41 / S16F11 / S3F17 bodies that aren't in the standard. + Our parsers are tolerant of extras; check your handler's + assumptions. + +See MES_INTEROP.md §13 for the per-MES quirk register. + +## What if the spec is ambiguous on some detail? + +Cross-check against the secsgem-py and secs4j wire output: +[VERIFICATION.md](VERIFICATION.md). If both peers agree on a +shape, that's the working interpretation regardless of how you read +the spec text. If they disagree, the secsgem-py output usually +wins (it's the de-facto Python reference and most MES vendors test +against it), but file the question — we may need a new test. + +## Can I run this without Docker? + +In principle yes — you need g++-13 (or any C++20 compiler), CMake, +Ninja, libasio-dev, libyaml-cpp-dev, python3. But every doc, every +CI lane, every test command in the repo assumes Docker. Going +off-piste means re-deriving the build on your host. We don't +support it; we don't actively break it. + +## How does persistence survive a crash mid-write? + +Every store uses a `.tmp + atomic rename` pattern: writes go to +`.tmp`, then `rename(2)`s into place. POSIX guarantees the +rename is atomic on the same filesystem. A crash mid-write loses +the `.tmp` (corrupt-drop on next replay) but leaves the prior +record intact. + +Every store's loader accepts versions in `[1, kVersion]` so future +schema bumps don't nuke old records — see README §Production +deployment "Schema migrations." + +## What does the "spool" actually do? + +When the host MES disconnects, the equipment can't deliver +unsolicited S5F1 alarms / S6F11 events. Without spool they'd be +lost. + +With spool enabled (`SpoolStore::set_spoolable_streams({5, 6})`), +those frames queue to in-memory FIFO (and persistent disk if +`enable_persistence` is set). On the host's next SELECT, the +equipment emits `S6F25 SpoolDataReady(count)`; the host issues +`S6F23(Transmit)` to drain, or `S6F23(Purge)` to discard. + +It's the GEM equivalent of an outbox. See E30 §6.22 and our +SpoolStore source. + +## How is "robustness fuzz" different from "libFuzzer"? + +- **Robustness fuzz** (`tests/test_robustness_fuzz.cpp`) is a + *model-level* property test. It picks random tool operations + (PJ create, alarm set, substrate move, …) respecting FSM + legality, and checks invariants after each. +- **libFuzzer** (`apps/fuzz_*.cpp`) is a *byte-level* coverage- + guided fuzzer. It feeds arbitrary bytes to the codec and + asserts no crash / UB. + +They cover different concerns: robustness fuzz catches *semantic* +bugs (lost data, wrong state); libFuzzer catches *parser* bugs +(crashes, UB, buffer overruns). + +## What's "conformance" vs "interop"? + +- **Conformance** (`build/secs_conformance`) is *us* driving *us* + through every claimed E30 capability and asserting the spec- + mandated reply S/F. Catches our regressions against our own + understanding of the spec. +- **Interop** (`interop/*.py`, `interop/secs4j/*.java`, + `interop/tshark_validate.sh`) is third-party tools agreeing on + the wire bytes our equipment produces. Catches "we got the + spec wrong" — which conformance can't. + +Both are necessary; neither replaces the other. See VERIFICATION.md. + +## How do I bring this to a customer site? + +Run through the five external proofs in +[README's proof table](README.md#proof-of-feature-completeness) at +the customer's network. Then walk MES_INTEROP.md against their +actual MES. Then deploy per [SECURITY.md](SECURITY.md) for the +nftables / stunnel / signing setup. Then page on the metrics from +INTEGRATION.md §6.4. + +## What's not implemented? + +See [COMPLIANCE.md](COMPLIANCE.md) §8 ("Explicitly out of scope") +for the honest list. The short version: multi-block SECS-I +transfers (irrelevant on HSMS), tool-specific Equipment Processing +States (the engine is there, vendors plug in their states), and +GEM RTS certification (paid third-party gate, not a code feature). diff --git a/GLOSSARY.md b/GLOSSARY.md new file mode 100644 index 0000000..d4cacd4 --- /dev/null +++ b/GLOSSARY.md @@ -0,0 +1,163 @@ +# Glossary + +SECS/GEM has roughly thirty acronyms that the spec uses without +introduction. This is the one-page decoder. Each entry has the +expansion, a one-line definition, and (where useful) the place in +this codebase where it shows up. + +## Identifiers and data + +| Term | Stands for | Meaning | +|------------|----------------------------------|------------------------------------------------------------------------------------------| +| **SVID** | Status Variable Identifier | A read-only equipment-side value the host queries via `S1F3` (e.g. ChamberPressureTorr). | +| **DVID** | Data Variable Identifier | Same shape as SVID but conceptually a *data variable* (intermediate, not status). Reported via `S1F21/F22`. | +| **ECID** | Equipment Constant Identifier | A host-settable equipment parameter (e.g. T-timers, thresholds). Set via `S2F15`, read via `S2F13`. | +| **CEID** | Collection Event Identifier | An event the equipment can emit (e.g. `ProcessStarted`). Bound to reports via `S2F35`, fires via `S6F11`. | +| **RPTID** | Report Identifier | A named bundle of VIDs (`S2F33`). CEIDs link to RPTIDs; the report carries the VID values. | +| **ALID** | Alarm Identifier | An equipment alarm (e.g. `ChamberPressureHigh`). `S5F5/F1/F3`. | +| **EXID** | Exception Identifier | A recoverable exception condition. `S5F9 → S5F13 → S5F11/F15` lifecycle. | +| **PPID** | Process Program Identifier | A recipe name. `S7F19` lists, `S7F5` requests, `S7F3` sends. | +| **MID** | Material Identifier | A wafer / substrate id (used in E40 PJ `mtrloutspec`). | +| **CARRIERID** | Carrier Identifier | A FOUP / cassette id (E87). | +| **PRJOBID**| Process Job Identifier | E40 PJ id; one PJ = one recipe-run for one batch of material. | +| **CTLJOBID** | Control Job Identifier | E94 CJ id; a CJ owns an ordered list of PRJOBIDs. | +| **SUBSTID**| Substrate Identifier | E90 wafer id, distinct from MID (which can be looser). | +| **OBJSPEC** | Object Specifier | E39 generic object reference (e.g. an instance of an E120 CEM object). | +| **OBJTYPE** | Object Type | Companion to OBJSPEC — the class. | +| **MDLN** | Model Number | Equipment model identifier (e.g. `ACME-PVD-3000`). Sent in `S1F2` and `S1F14`. | +| **SOFTREV**| Software Revision | Equipment software version string. Sent in `S1F2` and `S1F14`. | +| **EQPTYP** | Equipment Type | A category string (e.g. `PVD`) sent in `S1F20`. | +| **DATAID** | Data Identifier | A correlation id within multi-step setups (e.g. tying `S2F33`+`S2F35`+`S2F37` together). | + +## Acknowledgement codes + +Every host-issued request that mutates state gets back a 1-byte +acknowledgement code. The mnemonic tells you which spec section +the enum lives in. + +| Code | Used in | Values | +|-----------|-------------------|---------------------------------------------------------------------------------------------| +| **COMMACK**| `S1F14` | 0 = Accept, 1 = Denied (equipment not ready). | +| **ONLACK** | `S1F18` | 0 = Accept, 1 = NotAccept, 2 = AlreadyOnline. | +| **OFLACK** | `S1F16` | Only 0 = Accept defined. | +| **HCACK** | `S2F42` / `S16F6` / `S16F12` etc. | 0 = Accept, 1 = InvalidCommand, 2 = CannotDoNow, 3 = ParameterInvalid, 4 = AcceptedWillFinishLater, 5 = Rejected, 6 = InvalidObject. | +| **CMDA** | `S2F22` | Same enum as HCACK; spelled differently in the spec. | +| **ACKC5** | `S5F4` / `S5F2` | 0 = Accept, 1 = Error. | +| **ACKC6** | `S6F12` / `S6F2` etc. | 0 = Accept, 1 = Error. | +| **ACKC7** | `S7F4` / `S7F18` | 0 = Accept, 1-6 = various PP-management errors. | +| **ACKC10** | `S10F2` / `S10F4` / `S10F6` | 0 = Accept, 1-3 = TerminalDisplay errors. | +| **DRACK** | `S2F34` | Define Report Ack: 0 = Accept, 1-4 = various definition errors. | +| **LRACK** | `S2F36` | Link Event Ack: 0 = Accept, 1-5. | +| **ERACK** | `S2F38` | Enable Event Ack: 0 = Accept, 1 = UnknownCEID. | +| **EAC** | `S2F16` | Equipment Constant ack: 0 = Accept, 1 = Denied_OutOfRange, 2 = BusyOrUnknown, 3 = MajorOOR.| +| **TIACK** | `S2F32` | Time Ack: 0 = Accept, 1 = Error, 2 = NotDone. | +| **GRANT** | `S2F40` / `S6F6` | 0 = Grant, 1-3 = denials. | +| **ALCD** | `S5F1` | **Alarm Code**: bit-7 = set/clear flag; lower 7 bits = severity bitmap (E5 §10.3). | +| **OBJACK** | `S14F2` / `S14F10` etc. | E39 object service ack: 0 = Success, plus per-call denial codes. | + +## Streams and functions + +SECS-II messages are named **`SsFf`** — Stream *s*, Function *f*. +Odd functions are **primary** (initiating a transaction); even +functions are the **reply** to function *f − 1*. A primary with the +**W-bit** set expects a reply. + +| Stream | Domain | Example exchange | +|-------:|----------------------------------------------|------------------------------------------------------------------| +| S1 | Equipment status | S1F1/F2 "Are You There", S1F13/F14 Establish Comms | +| S2 | Equipment control + configuration | S2F33 define report, S2F41 host command | +| S3 | E87 carrier management | S3F17 CarrierAction, S3F19 SlotMapVerify | +| S5 | Exception reporting | S5F1 alarm send, S5F9-F18 exception recovery | +| S6 | Data collection | S6F11 unsolicited event report, S6F15 event report request | +| S7 | Process program management | S7F5 PP request, S7F19 PP list, S7F23 E42 formatted PP | +| S9 | System errors | S9F1, F3, F5, F7, F9, F11 — protocol-error notifications | +| S10 | Terminal services | S10F3 host→equipment display, S10F1 equipment→host | +| S12 | Wafer maps (E5 §13) | S12F1 setup, S12F7/F9/F11 send (3 formats) | +| S14 | E39 / E94 object services | S14F1 GetAttr, S14F9 CreateControlJob | +| S16 | E40 / E94 jobs | S16F11 PRJobCreate, S16F27 CJobCommand | + +## HSMS terminology + +| Term | Stands for | Meaning | +|------------|----------------------------------|------------------------------------------------------------------------------------------| +| **HSMS** | High-Speed Message Service | The TCP-based SECS transport defined by SEMI E37. | +| **HSMS-SS**| Single-Session | The common case: one session per TCP connection. | +| **HSMS-GS**| General-Session | Multi-session: multiple session IDs share one TCP connection. | +| **PType** | Presentation Type | 1-byte header field; 0 = SECS-II body. | +| **SType** | Session Type | 1-byte header field identifying the message class (data, Select.req, Linktest, etc.). | +| **MHEAD** | Message Header (10 bytes) | The HSMS framing header; appears unchanged in `S9F3/F5/F7/F11` payloads. | +| **NOT-SELECTED** / **SELECTED** | HSMS connection state | Reached via Select.req/rsp; required before data messages can flow. | + +## T-timers (E37 §10) + +| Timer | Purpose | Typical default | Where enforced | +|------:|----------------------------------------------------|-----------------|----------------------| +| **T3**| Reply timeout for a W=1 primary | 45 s | per-transaction asio timer | +| **T5**| Connect-separation: how long before retrying after a connection failure | 10 s | client retry loop | +| **T6**| Control transaction timeout (Select / Linktest) | 5 s | one concurrent control transaction | +| **T7**| Not-selected timeout — passive side, after TCP up | 10 s | armed on accept | +| **T8**| Intercharacter timeout — bounds the payload read after the 4-byte length prefix | 6 s | data-read loop | + +For SECS-I (E4), T1/T2/T3/T4 are the analogous serial-line timers +covering inter-character, protocol, reply, and inter-block respectively. + +## E84 signals (parallel I/O, AMHS handshake) + +| Signal | Direction | Meaning | +|--------|-------------------------|-----------------------------------------------| +| CS_0 | AMHS → equipment | Carrier-stage select 0 (multi-port equipment) | +| CS_1 | AMHS → equipment | Carrier-stage select 1 | +| VALID | AMHS → equipment | Handshake start | +| TR_REQ | AMHS → equipment | Transfer request | +| BUSY | AMHS → equipment | Transfer in progress | +| COMPT | AMHS → equipment | Transfer complete | +| L_REQ | equipment → AMHS | Load request — port ready to receive | +| U_REQ | equipment → AMHS | Unload request — port ready to release | +| READY | equipment → AMHS | Ready | +| ES | either | Emergency stop | + +Handshake timers TA1 (VALID→L_REQ), TA2 (Load/UnloadReady→BUSY), +TA3 (BUSY duration) live alongside the signals — see +`include/secsgem/gem/e84_state.hpp` and INTEGRATION.md §4.6. + +## Standards lineup + +| Spec | Topic | +|---------|-------------------------------------------------| +| **E4** | SECS-I serial transport (block protocol) | +| **E5** | SECS-II message structure + encoding rules | +| **E30** | GEM — generic equipment model + capabilities | +| **E37** | HSMS — TCP transport for SECS-II | +| **E39** | Generic object services (`S14F1/F3` GetAttr/SetAttr) | +| **E40** | Process job management (`S16F5/F11/F13`) | +| **E42** | Formatted process programs (`S7F23-F26`) | +| **E84** | Parallel I/O AMHS handshake | +| **E87** | Carrier management (`S3F17/F19/F25/F27`) | +| **E90** | Substrate tracking | +| **E94** | Control job management (`S14F9/F11`, `S16F27`) | +| **E116**| Equipment Performance Tracking | +| **E120**| Common Equipment Model | +| **E148**| Time synchronization | +| **E157**| Module process tracking | + +## Codebase shortcuts + +| Term | What it refers to in this repo | +|-----------------|-----------------------------------------------------------------------| +| **The model** | `gem::EquipmentDataModel` — the composed bundle of every store. | +| **A store** | One of the per-domain bundles under `include/secsgem/gem/store/` (alarms, carriers, spool, substrates, …). | +| **The router** | `gem::Router` — `(stream, function) → handler` dispatch table. | +| **The codec** | `secs2::encode` / `secs2::decode` for the wire bytes. | +| **The catalog** | `data/messages.yaml` — every SECS-II message we ship, codegen'd to `messages.hpp`. | +| **The proof** | The 8 commands in [README.md](README.md#proof-of-feature-completeness). | +| **The bench** | `apps/secs_bench.cpp` — single-threaded throughput / latency / memory harness. | +| **The fuzz** | `tests/test_robustness_fuzz.cpp` — randomized property test of the model. | + +## See also + +- [INTEGRATION.md](INTEGRATION.md) — when you've grasped the + vocabulary, this is how you put it together. +- [COMPLIANCE.md](COMPLIANCE.md) — every term above has a + spec-anchored implementation; the audit cross-references both. +- [FAQ.md](FAQ.md) — "OK, but *why*…" answers for the most common + next questions. diff --git a/README.md b/README.md index 6a5c2dd..087bd72 100644 --- a/README.md +++ b/README.md @@ -88,9 +88,14 @@ through the data model. Watch the logs interleave. |-----------------------------------------------|-------------------------------------------------------------------------| | [COMPLIANCE.md](COMPLIANCE.md) | Per-capability audit against every SEMI standard implemented | | [INTEGRATION.md](INTEGRATION.md) | Vendor-side tutorial: YAML → callbacks → production deploy | +| [ARCHITECTURE.md](ARCHITECTURE.md) | How the pieces fit + how to extend (new store / FSM / message) | +| [VERIFICATION.md](VERIFICATION.md) | Test plan for the external validators behind the proof table | | [BENCHMARKS.md](BENCHMARKS.md) | Performance envelope (throughput, latency, memory) + how to re-run | | [MES_INTEROP.md](MES_INTEROP.md) | Day-1 punch list to run against your commercial MES (60+ test IDs) | | [SECURITY.md](SECURITY.md) | Concrete configs: nftables, stunnel, minisign, SIEM audit-log schema | +| [GLOSSARY.md](GLOSSARY.md) | SEMI vocabulary: SVID, CEID, PPID, ALCD, HCACK, T-timers, … | +| [FAQ.md](FAQ.md) | Common questions and their canonical answers | +| [examples/pvd_tool/](examples/pvd_tool/) | Worked example: a realistic fictional PVD tool, YAML + C++ wiring | | [LICENSE](LICENSE) | Proprietary license terms | --- diff --git a/interop/README.md b/interop/README.md index ba3cdf2..ccbaa12 100644 --- a/interop/README.md +++ b/interop/README.md @@ -1,59 +1,133 @@ -# secsgem-py interop harness +# External cross-validation harnesses -Cross-validates our C++ SECS-II / HSMS / GEM implementation against -[secsgem-py](https://pypi.org/project/secsgem/) 0.3.0, the de-facto -Python reference. Everything runs in Docker — no Python or secsgem-py -on the host. +Every harness in this directory exists so a reviewer doesn't have to +take our word for it. Each one validates our C++ codec / framing / +dispatch against an **independent third-party implementation** that +read the SEMI standards without talking to us. -## What it tests +See [`../VERIFICATION.md`](../VERIFICATION.md) for the full test plan +and the honest accounting of which proofs are external vs internal. -| Driver | Peer | Coverage | -| ------------------------------ | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `host_vs_cpp_server.py` | C++ `secs_server` (passive) | HSMS select/separate, S1F1/F3/F11/F17/F23, S2F13/F17/F29/F33/F35/F37/F41, S5F3/F5/F7, S5F1 unsolicited, S6F11 unsolicited, S7F3/F5/F19, S10F1/F3, S1F15 | -| `secs_interop_probe` (C++) | `passive_equipment.py` (secsgem-py GemEquipmentHandler) | HSMS select, S1F13/F14, S1F1/F2, S1F3/F4, clean separate | -| `raw_gem300_harness.py` | C++ `secs_server` (passive) | GEM 300 streams secsgem-py upstream doesn't ship: S3F17/F18 (E87 carrier action), S16F5/F6 (E40 PRJobCommand), S16F27/F28 (E94 CJobCommand) — built with custom `SecsStreamFunction` subclasses + registered custom `DataItem`s | +## What's here -24 named checks on the C++-server side; 4 explicit checks on the -C++-host side; 4 GEM-300 raw-frame checks. Implicit HSMS state-machine -and wire-level framing validation everywhere. +| Validator | Independence | Coverage | +|--------------------------------------------|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------| +| `host_vs_cpp_server.py` + `passive_equipment.py` | secsgem-py 0.3.0 — Python reference impl | ~24 + 4 checks: S1, S2, S5, S6, S7, S10 happy paths | +| `raw_gem300_harness.py` | secsgem-py with hand-crafted SecsStreamFunctions | 3 checks: S3F17, S16F5, S16F27 (limited by SFDL grammar) | +| `secs4j/Secs4jHostHarness.java` | secs4java8 — Apache 2.0 Java impl by kenta-shimizu | **55 checks** across S1/S2/S3/S5/S6/S7/S10/S14/S16, including the full E40 body that defeated secsgem-py and unsolicited S6F11/S5F1 observation | +| `tshark_validate.sh` | Wireshark's built-in HSMS dissector | 69 captured frames dissected with no malformed-packet warnings | +| `spool_persistence_test.py` | secsgem-py + a docker-restart loop | Restart-survives-spool integrity | +| ⚙️ `../tests/test_e5_kat.cpp` | SEMI E5 §9 encoding rules | 196 known-answer byte assertions across every format code | +| ⚙️ `../apps/fuzz_secs2_decode.cpp` + `fuzz_sml_parse.cpp` | libFuzzer + ASan + UBSan | ~70 000 + ~285 000 random inputs per minute, 0 crashes | -## Running +The ⚙️ entries aren't in `interop/` directly because they don't +involve a network peer — they're either pure codec round-trips +(KAT) or coverage-guided fuzzing. Listed here so the full external +proof inventory lives in one place. -```bash -# Start C++ passive server, then drive it with secsgem-py host: +## Running each validator + +### secsgem-py — secsgem-py active host → C++ server + +```sh docker compose up -d server docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py \ --host server --port 5000 --session-id 0 +``` -# Start Python passive equipment, then probe it with the C++ host: +### secsgem-py — C++ host → secsgem-py equipment + +```sh docker compose up -d equipment_py docker compose run --rm builder /app/build/secs_interop_probe \ --host equipment_py --port 5000 --device 0 ``` -Both exit 0 on success. +### secsgem-py — raw GEM 300 frames -## What this caught +```sh +docker compose up -d server +docker compose run --rm interop python3 /app/interop/raw_gem300_harness.py \ + --host server --port 5000 --session-id 0 +``` -Real bugs surfaced by interop (now fixed): +### secs4j — independent Java host → C++ server -1. **Strict U4 parsing rejected U1-encoded identifiers.** SEMI E5 - declares DATAID, RPTID, VID, CEID, ALID, EXID, etc. as - `U1 | U2 | U4 | U8`; secsgem-py picks the smallest width that fits. - Our `as_u4_scalar`, `as_u2_scalar`, etc. were strict. Now lenient - with range-checked downcasts (`messages_helpers.hpp::any_unsigned_first`). -2. **PPBODY rejected when sent as ASCII.** SEMI lets PPBODY be +```sh +bash interop/secs4j_validate.sh +``` + +Builds an `eclipse-temurin:21-jdk` sidecar with secs4java8 cloned + +compiled at image build, then drives 55 checks against +`compose up server`. See `secs4j/Secs4jHostHarness.java` for the +list and `secs4j/Dockerfile` for the build. + +### tshark — Wireshark HSMS dissector + +```sh +docker compose run --rm builder bash /app/interop/tshark_validate.sh +``` + +Captures a pcap of the demo flow, runs `tshark -V` with the HSMS +dissector forced for the test port, asserts no malformed packets + +that all expected control/data frames parse. + +### spool persistence — restart-survives test + +```sh +bash interop/spool_persistence_test.py +``` + +Drops the host link mid-flight, kills the server, restarts it, and +asserts the spooled S5F1 / S6F11 frames drain to the host on +reconnect. + +## What these harnesses caught + +Real bugs surfaced during interop development (now fixed): + +1. **Strict U-width parsing rejected U1-encoded identifiers.** SEMI + E5 declares DATAID, RPTID, VID, CEID, ALID, EXID etc. as + `U1 | U2 | U4 | U8`; secsgem-py picks the smallest width that + fits. Our scalar accessors were strict. Now lenient with + range-checked downcasts (`messages_helpers.hpp::any_unsigned_first`). + +2. **PPBODY rejected when sent as ASCII.** SEMI allows PPBODY to be `ASCII | Binary | List`; secsgem-py defaults to ASCII. Added the - `BINARY_OR_ASCII` codegen item type plus a permissive - `as_text_or_binary` accessor, used for S7F3/F6. -3. **Missing S1F23 / S1F24 (Collection Event Namelist).** Added the - wire schema in `data/messages.yaml`, a `vids_for(ceid)` accessor on - the event-report store, and the dispatch handler in `secs_server.cpp`. -4. **Missing S10F3 handler (Terminal Display Single, host→equipment).** - Our server only registered S10F1; per SEMI E5, S10F1 is - equipment→host and S10F3 is the host→equipment counterpart. Added - the missing dispatch. + `BINARY_OR_ASCII` codegen type and the `as_text_or_binary` + accessor. -The C++ test suite still passes (278 cases / 1436 assertions) after -each of these changes — the fixes are purely permissive widenings, no -existing behaviour was broken. +3. **Missing S1F23 / S1F24 (Collection Event Namelist).** Added the + wire schema, the `vids_for(ceid)` accessor, and the dispatch + handler. + +4. **Missing S10F3 handler (host→equipment Terminal Display).** Our + server only registered S10F1; per SEMI E5 §13 those are opposite + directions. Added the missing dispatch. + +5. **TSan use-after-free in `act_exception_complete`** (test code, + not library): held a pointer across `fire_internal(RecoveryComplete)` + which deletes the entry. Found by the ThreadSanitizer lane on + first run. + +The C++ test suite stayed green through every one of these fixes — +the changes were purely permissive widenings or additive features, +no existing behaviour broke. + +## When to add a new validator + +A new third-party SECS implementation, or a new dissector, or a new +fuzzer target — anything that exercises our wire surface from an +angle the existing five don't cover — is worth adding. The pattern +is consistent: + +1. New script / harness lives here (or a sidecar Docker context for + non-Python validators). +2. Wired into `.gitea/workflows/ci.yml` as a separate job. +3. Listed in this README's table + in `../VERIFICATION.md`. +4. Surfaced in the README's proof-of-feature-completeness table if + it adds a meaningful new dimension. + +Bug reports from a new validator → file at +`raphael@maenle.net` with the wire trace, the validator's output, +and the equipment YAML so we can reproduce.