raphael/secs-gem
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: GLOSSARY + FAQ + interop README refresh + doc-map fixes

Browse Source 
Fills four documentation gaps surfaced by the doc audit:

1. README "Documentation map" was missing VERIFICATION.md (the file
   that backs the proof-of-feature-completeness claims) and is now
   pointing at the new files added in this commit too — ARCHITECTURE,
   GLOSSARY, FAQ, examples/pvd_tool/ (the last two land next).

2. interop/README.md only documented secsgem-py.  Three of the five
   external validators (tshark, secs4j, libFuzzer) plus the E5 KAT
   were invisible from the directory's own README.  Rewritten as a
   complete index — what's external, what each catches, how to run,
   what bugs they've already surfaced, when to add a new validator.

3. GLOSSARY.md is new.  Every SEMI acronym used in the codebase or
   the docs gets one row: SVID, DVID, CEID, RPTID, ALID, ECID, PPID,
   MID, CARRIERID, PRJOBID, CTLJOBID, SUBSTID, OBJSPEC, OBJTYPE,
   MDLN, SOFTREV, EQPTYP, DATAID + every ACK code (COMMACK, ONLACK,
   OFLACK, HCACK, CMDA, ACKC5-7-10, DRACK, LRACK, ERACK, EAC, TIACK,
   GRANT, ALCD, OBJACK) + stream/function shorthand + HSMS terms +
   T-timers + E84 signals + the standards lineup + codebase shortcuts
   ("the model", "the router", "the proof", etc.).  Cuts week-1
   onboarding time.

4. FAQ.md is new.  Canonical answers to the questions that come up
   once per integration: why HSMS unencrypted, SVID vs DVID, PJ vs
   CJ, who fires FSM transitions, what runs on which thread, how to
   add a new SECS-II message, ASCII vs Binary, common MES quirks,
   how spool works, robustness fuzz vs libFuzzer, conformance vs
   interop, what's not implemented.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Raphael Maenle
2026-06-09 16:43:45 +02:00
parent db90a21e1d
commit 195ecc689f
4 changed files with 506 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
FAQ.md
+225
View File
@@ -0,0 +1,225 @@
# FAQ
Questions we hear once per integration. Skim before you ask. If
your question isn't here and isn't obvious from the other docs,
ask once — your question probably belongs in this file and we'll
add it.
## Why is HSMS unencrypted?
Because SEMI E37 says so. HSMS is plain TCP with a 14-byte
framing header — no TLS, no auth, no nonces. Every commercial MES
on the market speaks exactly that wire, and changing it would make
us incompatible with all of them. Encryption and authentication
belong at the network layer: see [SECURITY.md](SECURITY.md) for
the stunnel.conf + nftables setup that wraps the unencrypted TCP
in mTLS without modifying the wire protocol.
## What's the difference between SVID and DVID?
**SVID** is a *status* variable — equipment state the host queries
(chamber pressure, current control state, wafer counter).
**DVID** is a *data* variable — intermediate values, typically
computed or sensor-derived, that aren't part of the equipment's
state model.
In practice fab tools blur the line. The library treats them
identically except for which message reports them: `S1F3 / S1F11`
for SVIDs, `S1F21 / S1F22` for DVIDs. Variable lookups by VID
span both (`EquipmentDataModel::vid_value`).
## Do I really need all four YAML files?
Yes for production; no for a quick "does it compile":
- `equipment.yaml` — your tool's data dictionary. Required.
- `control_state.yaml` — the E30 control state machine (HostOffline,
AttemptOnline, OnlineRemote, …). The default in `data/` works as
a starting point; you may customize transitions.
- `process_job_state.yaml` — the E40 PJ FSM. Default is spec-typical;
customize only if your tool has unusual recipe semantics.
- `control_job_state.yaml` — the E94 CJ FSM. Same.
`secs_server --validate-config` checks all four in one pass and
exits 0 / 1. Run it in CI on every config change.
## PJ vs CJ — what's the difference?
A **PJ** (E40 Process Job) is "process this batch of material with
this recipe." One PJ = one recipe run = one set of wafers. It
has its own FSM (Queued → SettingUp → Processing → ProcessComplete).
A **CJ** (E94 Control Job) is "execute these PJs in order, as a
unit, with start/pause/abort semantics." A CJ owns an ordered list
of PRJOBIDs. When the host issues `CJSTART`, the CJ promotes its
PJs through their lifecycles.
You typically need both: the MES creates a CJ containing N PJs,
then starts the CJ. PJs without a CJ are legal — they just sit in
Queued waiting for someone to select them — but most MES drives
batches through CJs.
## Who fires FSM transitions — the library or my code?
**Your code.** The library implements the FSMs (legal transitions,
validation, persistence) but it doesn't know when a wafer was
actually loaded or when a recipe step finished — those signals come
from your tool. The pattern across every store is:
```cpp
// You fire the event; the FSM validates + transitions + emits.
model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::SetupComplete);
model->carriers.fire_id_event("CAR-A1B2", gem::CarrierIDEvent::Read);
```
Host commands (`S2F41` RCMD=START, `S16F5` PRJSTART, `S16F27` CJSTART)
arrive via the wire and get dispatched into your registered
handlers; the handler typically calls `fire_internal` or
`on_host_command` on the relevant store.
See INTEGRATION.md §4 for the worked patterns.
## What runs on which thread?
**Everything that touches the data model runs on the io_context
thread.** There are no locks in `EquipmentDataModel`.
- The Router dispatch (incoming wire messages) — on the io_context.
- All `set_*_change_handler` callbacks — on the io_context.
- Periodic timers you register via asio — on the io_context.
If your code lives on another thread (typical for sensor polling),
marshal updates via `asio::post`:
```cpp
asio::post(io.get_executor(), [model, value] {
model->svids.set_value(100, secs2::Item::f4(value));
});
```
INTEGRATION.md §3 has the full thread-safety contract.
## How do I add a new SECS-II message?
Edit `data/messages.yaml`, add a row, rebuild. The codegen
(`tools/gen_messages.py`) emits a typed builder + parser into
`messages.hpp`. Then register a Router handler in your `main.cpp`
for the new `(stream, function)` pair. See README "Adding a
capability" or ARCHITECTURE.md for the full walkthrough.
## What's the difference between `Item::ascii("X")` and `Item::binary({'X'})`?
The wire format byte differs — `0x41 01 58` for ASCII vs
`0x21 01 58` for Binary. Some peers (notably secsgem-py) default
PPBODY to ASCII; others use Binary. Our codec accepts either via
the `BINARY_OR_ASCII` codegen type for fields the spec lists as
`ASCII | Binary | List` (the PPBODY case in S7F3/F6).
For most fields it doesn't matter — pick the format that matches
your data semantically.
## My MES sends a message that worked in `interop` but fails in production. What's going on?
Three usual suspects:
1. **U-width.** Your MES is sending `DATAID` as U1 but our handler
was strict for U4. We're lenient now via `any_unsigned_first`,
but if you have custom handlers in your code, use that helper
rather than `as_u4_scalar` for identifier fields.
2. **PPBODY direction.** Some MES send PPBODY as ASCII even when
the spec says it can be Binary. Use `as_text_or_binary` not
`as_binary`.
3. **Trailing fields.** Some MES add proprietary trailing fields
to S2F41 / S16F11 / S3F17 bodies that aren't in the standard.
Our parsers are tolerant of extras; check your handler's
assumptions.
See MES_INTEROP.md §13 for the per-MES quirk register.
## What if the spec is ambiguous on some detail?
Cross-check against the secsgem-py and secs4j wire output:
[VERIFICATION.md](VERIFICATION.md). If both peers agree on a
shape, that's the working interpretation regardless of how you read
the spec text. If they disagree, the secsgem-py output usually
wins (it's the de-facto Python reference and most MES vendors test
against it), but file the question — we may need a new test.
## Can I run this without Docker?
In principle yes — you need g++-13 (or any C++20 compiler), CMake,
Ninja, libasio-dev, libyaml-cpp-dev, python3. But every doc, every
CI lane, every test command in the repo assumes Docker. Going
off-piste means re-deriving the build on your host. We don't
support it; we don't actively break it.
## How does persistence survive a crash mid-write?
Every store uses a `.tmp + atomic rename` pattern: writes go to
`<file>.tmp`, then `rename(2)`s into place. POSIX guarantees the
rename is atomic on the same filesystem. A crash mid-write loses
the `.tmp` (corrupt-drop on next replay) but leaves the prior
record intact.
Every store's loader accepts versions in `[1, kVersion]` so future
schema bumps don't nuke old records — see README §Production
deployment "Schema migrations."
## What does the "spool" actually do?
When the host MES disconnects, the equipment can't deliver
unsolicited S5F1 alarms / S6F11 events. Without spool they'd be
lost.
With spool enabled (`SpoolStore::set_spoolable_streams({5, 6})`),
those frames queue to in-memory FIFO (and persistent disk if
`enable_persistence` is set). On the host's next SELECT, the
equipment emits `S6F25 SpoolDataReady(count)`; the host issues
`S6F23(Transmit)` to drain, or `S6F23(Purge)` to discard.
It's the GEM equivalent of an outbox. See E30 §6.22 and our
SpoolStore source.
## How is "robustness fuzz" different from "libFuzzer"?
- **Robustness fuzz** (`tests/test_robustness_fuzz.cpp`) is a
*model-level* property test. It picks random tool operations
(PJ create, alarm set, substrate move, …) respecting FSM
legality, and checks invariants after each.
- **libFuzzer** (`apps/fuzz_*.cpp`) is a *byte-level* coverage-
guided fuzzer. It feeds arbitrary bytes to the codec and
asserts no crash / UB.
They cover different concerns: robustness fuzz catches *semantic*
bugs (lost data, wrong state); libFuzzer catches *parser* bugs
(crashes, UB, buffer overruns).
## What's "conformance" vs "interop"?
- **Conformance** (`build/secs_conformance`) is *us* driving *us*
through every claimed E30 capability and asserting the spec-
mandated reply S/F. Catches our regressions against our own
understanding of the spec.
- **Interop** (`interop/*.py`, `interop/secs4j/*.java`,
`interop/tshark_validate.sh`) is third-party tools agreeing on
the wire bytes our equipment produces. Catches "we got the
spec wrong" — which conformance can't.
Both are necessary; neither replaces the other. See VERIFICATION.md.
## How do I bring this to a customer site?
Run through the five external proofs in
[README's proof table](README.md#proof-of-feature-completeness) at
the customer's network. Then walk MES_INTEROP.md against their
actual MES. Then deploy per [SECURITY.md](SECURITY.md) for the
nftables / stunnel / signing setup. Then page on the metrics from
INTEGRATION.md §6.4.
## What's not implemented?
See [COMPLIANCE.md](COMPLIANCE.md) §8 ("Explicitly out of scope")
for the honest list. The short version: multi-block SECS-I
transfers (irrelevant on HSMS), tool-specific Equipment Processing
States (the engine is there, vendors plug in their states), and
GEM RTS certification (paid third-party gate, not a code feature).
GLOSSARY.md
+163
View File
@@ -0,0 +1,163 @@
# Glossary
SECS/GEM has roughly thirty acronyms that the spec uses without
introduction. This is the one-page decoder. Each entry has the
expansion, a one-line definition, and (where useful) the place in
this codebase where it shows up.
## Identifiers and data
| Term | Stands for | Meaning |
|------------|----------------------------------|------------------------------------------------------------------------------------------|
| **SVID** | Status Variable Identifier | A read-only equipment-side value the host queries via `S1F3` (e.g. ChamberPressureTorr). |
| **DVID** | Data Variable Identifier | Same shape as SVID but conceptually a *data variable* (intermediate, not status). Reported via `S1F21/F22`. |
| **ECID** | Equipment Constant Identifier | A host-settable equipment parameter (e.g. T-timers, thresholds). Set via `S2F15`, read via `S2F13`. |
| **CEID** | Collection Event Identifier | An event the equipment can emit (e.g. `ProcessStarted`). Bound to reports via `S2F35`, fires via `S6F11`. |
| **RPTID** | Report Identifier | A named bundle of VIDs (`S2F33`). CEIDs link to RPTIDs; the report carries the VID values. |
| **ALID** | Alarm Identifier | An equipment alarm (e.g. `ChamberPressureHigh`). `S5F5/F1/F3`. |
| **EXID** | Exception Identifier | A recoverable exception condition. `S5F9 → S5F13 → S5F11/F15` lifecycle. |
| **PPID** | Process Program Identifier | A recipe name. `S7F19` lists, `S7F5` requests, `S7F3` sends. |
| **MID** | Material Identifier | A wafer / substrate id (used in E40 PJ `mtrloutspec`). |
| **CARRIERID** | Carrier Identifier | A FOUP / cassette id (E87). |
| **PRJOBID**| Process Job Identifier | E40 PJ id; one PJ = one recipe-run for one batch of material. |
| **CTLJOBID** | Control Job Identifier | E94 CJ id; a CJ owns an ordered list of PRJOBIDs. |
| **SUBSTID**| Substrate Identifier | E90 wafer id, distinct from MID (which can be looser). |
| **OBJSPEC** | Object Specifier | E39 generic object reference (e.g. an instance of an E120 CEM object). |
| **OBJTYPE** | Object Type | Companion to OBJSPEC — the class. |
| **MDLN** | Model Number | Equipment model identifier (e.g. `ACME-PVD-3000`). Sent in `S1F2` and `S1F14`. |
| **SOFTREV**| Software Revision | Equipment software version string. Sent in `S1F2` and `S1F14`. |
| **EQPTYP** | Equipment Type | A category string (e.g. `PVD`) sent in `S1F20`. |
| **DATAID** | Data Identifier | A correlation id within multi-step setups (e.g. tying `S2F33`+`S2F35`+`S2F37` together). |
## Acknowledgement codes
Every host-issued request that mutates state gets back a 1-byte
acknowledgement code. The mnemonic tells you which spec section
the enum lives in.
| Code | Used in | Values |
|-----------|-------------------|---------------------------------------------------------------------------------------------|
| **COMMACK**| `S1F14` | 0 = Accept, 1 = Denied (equipment not ready). |
| **ONLACK** | `S1F18` | 0 = Accept, 1 = NotAccept, 2 = AlreadyOnline. |
| **OFLACK** | `S1F16` | Only 0 = Accept defined. |
| **HCACK** | `S2F42` / `S16F6` / `S16F12` etc. | 0 = Accept, 1 = InvalidCommand, 2 = CannotDoNow, 3 = ParameterInvalid, 4 = AcceptedWillFinishLater, 5 = Rejected, 6 = InvalidObject. |
| **CMDA** | `S2F22` | Same enum as HCACK; spelled differently in the spec. |
| **ACKC5** | `S5F4` / `S5F2` | 0 = Accept, 1 = Error. |
| **ACKC6** | `S6F12` / `S6F2` etc. | 0 = Accept, 1 = Error. |
| **ACKC7** | `S7F4` / `S7F18` | 0 = Accept, 1-6 = various PP-management errors. |
| **ACKC10** | `S10F2` / `S10F4` / `S10F6` | 0 = Accept, 1-3 = TerminalDisplay errors. |
| **DRACK** | `S2F34` | Define Report Ack: 0 = Accept, 1-4 = various definition errors. |
| **LRACK** | `S2F36` | Link Event Ack: 0 = Accept, 1-5. |
| **ERACK** | `S2F38` | Enable Event Ack: 0 = Accept, 1 = UnknownCEID. |
| **EAC** | `S2F16` | Equipment Constant ack: 0 = Accept, 1 = Denied_OutOfRange, 2 = BusyOrUnknown, 3 = MajorOOR.|
| **TIACK** | `S2F32` | Time Ack: 0 = Accept, 1 = Error, 2 = NotDone. |
| **GRANT** | `S2F40` / `S6F6` | 0 = Grant, 1-3 = denials. |
| **ALCD** | `S5F1` | **Alarm Code**: bit-7 = set/clear flag; lower 7 bits = severity bitmap (E5 §10.3). |
| **OBJACK** | `S14F2` / `S14F10` etc. | E39 object service ack: 0 = Success, plus per-call denial codes. |
## Streams and functions
SECS-II messages are named **`SsFf`** — Stream *s*, Function *f*.
Odd functions are **primary** (initiating a transaction); even
functions are the **reply** to function *f − 1*. A primary with the
**W-bit** set expects a reply.
| Stream | Domain | Example exchange |
|-------:|----------------------------------------------|------------------------------------------------------------------|
| S1 | Equipment status | S1F1/F2 "Are You There", S1F13/F14 Establish Comms |
| S2 | Equipment control + configuration | S2F33 define report, S2F41 host command |
| S3 | E87 carrier management | S3F17 CarrierAction, S3F19 SlotMapVerify |
| S5 | Exception reporting | S5F1 alarm send, S5F9-F18 exception recovery |
| S6 | Data collection | S6F11 unsolicited event report, S6F15 event report request |
| S7 | Process program management | S7F5 PP request, S7F19 PP list, S7F23 E42 formatted PP |
| S9 | System errors | S9F1, F3, F5, F7, F9, F11 — protocol-error notifications |
| S10 | Terminal services | S10F3 host→equipment display, S10F1 equipment→host |
| S12 | Wafer maps (E5 §13) | S12F1 setup, S12F7/F9/F11 send (3 formats) |
| S14 | E39 / E94 object services | S14F1 GetAttr, S14F9 CreateControlJob |
| S16 | E40 / E94 jobs | S16F11 PRJobCreate, S16F27 CJobCommand |
## HSMS terminology
| Term | Stands for | Meaning |
|------------|----------------------------------|------------------------------------------------------------------------------------------|
| **HSMS** | High-Speed Message Service | The TCP-based SECS transport defined by SEMI E37. |
| **HSMS-SS**| Single-Session | The common case: one session per TCP connection. |
| **HSMS-GS**| General-Session | Multi-session: multiple session IDs share one TCP connection. |
| **PType** | Presentation Type | 1-byte header field; 0 = SECS-II body. |
| **SType** | Session Type | 1-byte header field identifying the message class (data, Select.req, Linktest, etc.). |
| **MHEAD** | Message Header (10 bytes) | The HSMS framing header; appears unchanged in `S9F3/F5/F7/F11` payloads. |
| **NOT-SELECTED** / **SELECTED** | HSMS connection state | Reached via Select.req/rsp; required before data messages can flow. |
## T-timers (E37 §10)
| Timer | Purpose | Typical default | Where enforced |
|------:|----------------------------------------------------|-----------------|----------------------|
| **T3**| Reply timeout for a W=1 primary | 45 s | per-transaction asio timer |
| **T5**| Connect-separation: how long before retrying after a connection failure | 10 s | client retry loop |
| **T6**| Control transaction timeout (Select / Linktest) | 5 s | one concurrent control transaction |
| **T7**| Not-selected timeout — passive side, after TCP up | 10 s | armed on accept |
| **T8**| Intercharacter timeout — bounds the payload read after the 4-byte length prefix | 6 s | data-read loop |
For SECS-I (E4), T1/T2/T3/T4 are the analogous serial-line timers
covering inter-character, protocol, reply, and inter-block respectively.
## E84 signals (parallel I/O, AMHS handshake)
| Signal | Direction | Meaning |
|--------|-------------------------|-----------------------------------------------|
| CS_0 | AMHS → equipment | Carrier-stage select 0 (multi-port equipment) |
| CS_1 | AMHS → equipment | Carrier-stage select 1 |
| VALID | AMHS → equipment | Handshake start |
| TR_REQ | AMHS → equipment | Transfer request |
| BUSY | AMHS → equipment | Transfer in progress |
| COMPT | AMHS → equipment | Transfer complete |
| L_REQ | equipment → AMHS | Load request — port ready to receive |
| U_REQ | equipment → AMHS | Unload request — port ready to release |
| READY | equipment → AMHS | Ready |
| ES | either | Emergency stop |
Handshake timers TA1 (VALID→L_REQ), TA2 (Load/UnloadReady→BUSY),
TA3 (BUSY duration) live alongside the signals — see
`include/secsgem/gem/e84_state.hpp` and INTEGRATION.md §4.6.
## Standards lineup
| Spec | Topic |
|---------|-------------------------------------------------|
| **E4** | SECS-I serial transport (block protocol) |
| **E5** | SECS-II message structure + encoding rules |
| **E30** | GEM — generic equipment model + capabilities |
| **E37** | HSMS — TCP transport for SECS-II |
| **E39** | Generic object services (`S14F1/F3` GetAttr/SetAttr) |
| **E40** | Process job management (`S16F5/F11/F13`) |
| **E42** | Formatted process programs (`S7F23-F26`) |
| **E84** | Parallel I/O AMHS handshake |
| **E87** | Carrier management (`S3F17/F19/F25/F27`) |
| **E90** | Substrate tracking |
| **E94** | Control job management (`S14F9/F11`, `S16F27`) |
| **E116**| Equipment Performance Tracking |
| **E120**| Common Equipment Model |
| **E148**| Time synchronization |
| **E157**| Module process tracking |
## Codebase shortcuts
| Term | What it refers to in this repo |
|-----------------|-----------------------------------------------------------------------|
| **The model** | `gem::EquipmentDataModel` — the composed bundle of every store. |
| **A store** | One of the per-domain bundles under `include/secsgem/gem/store/` (alarms, carriers, spool, substrates, …). |
| **The router** | `gem::Router` — `(stream, function) → handler` dispatch table. |
| **The codec** | `secs2::encode` / `secs2::decode` for the wire bytes. |
| **The catalog** | `data/messages.yaml` — every SECS-II message we ship, codegen'd to `messages.hpp`. |
| **The proof** | The 8 commands in [README.md](README.md#proof-of-feature-completeness). |
| **The bench** | `apps/secs_bench.cpp` — single-threaded throughput / latency / memory harness. |
| **The fuzz** | `tests/test_robustness_fuzz.cpp` — randomized property test of the model. |
## See also
- [INTEGRATION.md](INTEGRATION.md) — when you've grasped the
vocabulary, this is how you put it together.
- [COMPLIANCE.md](COMPLIANCE.md) — every term above has a
spec-anchored implementation; the audit cross-references both.
- [FAQ.md](FAQ.md) — "OK, but *why*…" answers for the most common
next questions.
README.md
+5
View File
@@ -88,9 +88,14 @@ through the data model. Watch the logs interleave.
|-----------------------------------------------|-------------------------------------------------------------------------| |-----------------------------------------------|-------------------------------------------------------------------------|
| [COMPLIANCE.md](COMPLIANCE.md) | Per-capability audit against every SEMI standard implemented | | [COMPLIANCE.md](COMPLIANCE.md) | Per-capability audit against every SEMI standard implemented |
| [INTEGRATION.md](INTEGRATION.md) | Vendor-side tutorial: YAML → callbacks → production deploy | | [INTEGRATION.md](INTEGRATION.md) | Vendor-side tutorial: YAML → callbacks → production deploy |
| [ARCHITECTURE.md](ARCHITECTURE.md) | How the pieces fit + how to extend (new store / FSM / message) |
| [VERIFICATION.md](VERIFICATION.md) | Test plan for the external validators behind the proof table |
| [BENCHMARKS.md](BENCHMARKS.md) | Performance envelope (throughput, latency, memory) + how to re-run | | [BENCHMARKS.md](BENCHMARKS.md) | Performance envelope (throughput, latency, memory) + how to re-run |
| [MES_INTEROP.md](MES_INTEROP.md) | Day-1 punch list to run against your commercial MES (60+ test IDs) | | [MES_INTEROP.md](MES_INTEROP.md) | Day-1 punch list to run against your commercial MES (60+ test IDs) |
| [SECURITY.md](SECURITY.md) | Concrete configs: nftables, stunnel, minisign, SIEM audit-log schema | | [SECURITY.md](SECURITY.md) | Concrete configs: nftables, stunnel, minisign, SIEM audit-log schema |
| [GLOSSARY.md](GLOSSARY.md) | SEMI vocabulary: SVID, CEID, PPID, ALCD, HCACK, T-timers, … |
| [FAQ.md](FAQ.md) | Common questions and their canonical answers |
| [examples/pvd_tool/](examples/pvd_tool/) | Worked example: a realistic fictional PVD tool, YAML + C++ wiring |
| [LICENSE](LICENSE) | Proprietary license terms | | [LICENSE](LICENSE) | Proprietary license terms |
--- ---
interop/README.md
+113 -39
View File
@@ -1,59 +1,133 @@
# secsgem-py interop harness # External cross-validation harnesses
Cross-validates our C++ SECS-II / HSMS / GEM implementation against Every harness in this directory exists so a reviewer doesn't have to
[secsgem-py](https://pypi.org/project/secsgem/) 0.3.0, the de-facto take our word for it. Each one validates our C++ codec / framing /
Python reference. Everything runs in Docker — no Python or secsgem-py dispatch against an **independent third-party implementation** that
on the host. read the SEMI standards without talking to us.
## What it tests See [`../VERIFICATION.md`](../VERIFICATION.md) for the full test plan
and the honest accounting of which proofs are external vs internal.
| Driver | Peer | Coverage | ## What's here
| ------------------------------ | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `host_vs_cpp_server.py` | C++ `secs_server` (passive) | HSMS select/separate, S1F1/F3/F11/F17/F23, S2F13/F17/F29/F33/F35/F37/F41, S5F3/F5/F7, S5F1 unsolicited, S6F11 unsolicited, S7F3/F5/F19, S10F1/F3, S1F15 |
| `secs_interop_probe` (C++) | `passive_equipment.py` (secsgem-py GemEquipmentHandler) | HSMS select, S1F13/F14, S1F1/F2, S1F3/F4, clean separate |
| `raw_gem300_harness.py` | C++ `secs_server` (passive) | GEM 300 streams secsgem-py upstream doesn't ship: S3F17/F18 (E87 carrier action), S16F5/F6 (E40 PRJobCommand), S16F27/F28 (E94 CJobCommand) — built with custom `SecsStreamFunction` subclasses + registered custom `DataItem`s |
24 named checks on the C++-server side; 4 explicit checks on the | Validator | Independence | Coverage |
C++-host side; 4 GEM-300 raw-frame checks. Implicit HSMS state-machine |--------------------------------------------|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
and wire-level framing validation everywhere. | `host_vs_cpp_server.py` + `passive_equipment.py` | secsgem-py 0.3.0 — Python reference impl | ~24 + 4 checks: S1, S2, S5, S6, S7, S10 happy paths |
| `raw_gem300_harness.py` | secsgem-py with hand-crafted SecsStreamFunctions | 3 checks: S3F17, S16F5, S16F27 (limited by SFDL grammar) |
| `secs4j/Secs4jHostHarness.java` | secs4java8 — Apache 2.0 Java impl by kenta-shimizu | **55 checks** across S1/S2/S3/S5/S6/S7/S10/S14/S16, including the full E40 body that defeated secsgem-py and unsolicited S6F11/S5F1 observation |
| `tshark_validate.sh` | Wireshark's built-in HSMS dissector | 69 captured frames dissected with no malformed-packet warnings |
| `spool_persistence_test.py` | secsgem-py + a docker-restart loop | Restart-survives-spool integrity |
| ⚙️ `../tests/test_e5_kat.cpp` | SEMI E5 §9 encoding rules | 196 known-answer byte assertions across every format code |
| ⚙️ `../apps/fuzz_secs2_decode.cpp` + `fuzz_sml_parse.cpp` | libFuzzer + ASan + UBSan | ~70 000 + ~285 000 random inputs per minute, 0 crashes |
## Running The ⚙️ entries aren't in `interop/` directly because they don't
involve a network peer — they're either pure codec round-trips
(KAT) or coverage-guided fuzzing. Listed here so the full external
proof inventory lives in one place.
```bash ## Running each validator
# Start C++ passive server, then drive it with secsgem-py host:
### secsgem-py — secsgem-py active host → C++ server
```sh
docker compose up -d server docker compose up -d server
docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py \ docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py \
--host server --port 5000 --session-id 0 --host server --port 5000 --session-id 0
```
# Start Python passive equipment, then probe it with the C++ host: ### secsgem-py — C++ host → secsgem-py equipment
```sh
docker compose up -d equipment_py docker compose up -d equipment_py
docker compose run --rm builder /app/build/secs_interop_probe \ docker compose run --rm builder /app/build/secs_interop_probe \
--host equipment_py --port 5000 --device 0 --host equipment_py --port 5000 --device 0
``` ```
Both exit 0 on success. ### secsgem-py — raw GEM 300 frames
## What this caught ```sh
docker compose up -d server
docker compose run --rm interop python3 /app/interop/raw_gem300_harness.py \
--host server --port 5000 --session-id 0
```
Real bugs surfaced by interop (now fixed): ### secs4j — independent Java host → C++ server
1. **Strict U4 parsing rejected U1-encoded identifiers.** SEMI E5 ```sh
declares DATAID, RPTID, VID, CEID, ALID, EXID, etc. as bash interop/secs4j_validate.sh
`U1 | U2 | U4 | U8`; secsgem-py picks the smallest width that fits. ```
Our `as_u4_scalar`, `as_u2_scalar`, etc. were strict. Now lenient
with range-checked downcasts (`messages_helpers.hpp::any_unsigned_first`). Builds an `eclipse-temurin:21-jdk` sidecar with secs4java8 cloned +
2. **PPBODY rejected when sent as ASCII.** SEMI lets PPBODY be compiled at image build, then drives 55 checks against
`compose up server`. See `secs4j/Secs4jHostHarness.java` for the
list and `secs4j/Dockerfile` for the build.
### tshark — Wireshark HSMS dissector
```sh
docker compose run --rm builder bash /app/interop/tshark_validate.sh
```
Captures a pcap of the demo flow, runs `tshark -V` with the HSMS
dissector forced for the test port, asserts no malformed packets +
that all expected control/data frames parse.
### spool persistence — restart-survives test
```sh
bash interop/spool_persistence_test.py
```
Drops the host link mid-flight, kills the server, restarts it, and
asserts the spooled S5F1 / S6F11 frames drain to the host on
reconnect.
## What these harnesses caught
Real bugs surfaced during interop development (now fixed):
1. **Strict U-width parsing rejected U1-encoded identifiers.** SEMI
E5 declares DATAID, RPTID, VID, CEID, ALID, EXID etc. as
`U1 | U2 | U4 | U8`; secsgem-py picks the smallest width that
fits. Our scalar accessors were strict. Now lenient with
range-checked downcasts (`messages_helpers.hpp::any_unsigned_first`).
2. **PPBODY rejected when sent as ASCII.** SEMI allows PPBODY to be
`ASCII | Binary | List`; secsgem-py defaults to ASCII. Added the `ASCII | Binary | List`; secsgem-py defaults to ASCII. Added the
`BINARY_OR_ASCII` codegen item type plus a permissive `BINARY_OR_ASCII` codegen type and the `as_text_or_binary`
`as_text_or_binary` accessor, used for S7F3/F6. accessor.
3. **Missing S1F23 / S1F24 (Collection Event Namelist).** Added the
wire schema in `data/messages.yaml`, a `vids_for(ceid)` accessor on
the event-report store, and the dispatch handler in `secs_server.cpp`.
4. **Missing S10F3 handler (Terminal Display Single, host→equipment).**
Our server only registered S10F1; per SEMI E5, S10F1 is
equipment→host and S10F3 is the host→equipment counterpart. Added
the missing dispatch.
The C++ test suite still passes (278 cases / 1436 assertions) after 3. **Missing S1F23 / S1F24 (Collection Event Namelist).** Added the
each of these changes — the fixes are purely permissive widenings, no wire schema, the `vids_for(ceid)` accessor, and the dispatch
existing behaviour was broken. handler.
4. **Missing S10F3 handler (host→equipment Terminal Display).** Our
server only registered S10F1; per SEMI E5 §13 those are opposite
directions. Added the missing dispatch.
5. **TSan use-after-free in `act_exception_complete`** (test code,
not library): held a pointer across `fire_internal(RecoveryComplete)`
which deletes the entry. Found by the ThreadSanitizer lane on
first run.
The C++ test suite stayed green through every one of these fixes —
the changes were purely permissive widenings or additive features,
no existing behaviour broke.
## When to add a new validator
A new third-party SECS implementation, or a new dissector, or a new
fuzzer target — anything that exercises our wire surface from an
angle the existing five don't cover — is worth adding. The pattern
is consistent:
1. New script / harness lives here (or a sidecar Docker context for
non-Python validators).
2. Wired into `.gitea/workflows/ci.yml` as a separate job.
3. Listed in this README's table + in `../VERIFICATION.md`.
4. Surfaced in the README's proof-of-feature-completeness table if
it adds a meaningful new dimension.
Bug reports from a new validator → file at
`raphael@maenle.net` with the wire trace, the validator's output,
and the equipment YAML so we can reproduce.