raphael/secs-gem
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: GLOSSARY + FAQ + interop README refresh + doc-map fixes

Browse Source 
Fills four documentation gaps surfaced by the doc audit:

1. README "Documentation map" was missing VERIFICATION.md (the file
   that backs the proof-of-feature-completeness claims) and is now
   pointing at the new files added in this commit too — ARCHITECTURE,
   GLOSSARY, FAQ, examples/pvd_tool/ (the last two land next).

2. interop/README.md only documented secsgem-py.  Three of the five
   external validators (tshark, secs4j, libFuzzer) plus the E5 KAT
   were invisible from the directory's own README.  Rewritten as a
   complete index — what's external, what each catches, how to run,
   what bugs they've already surfaced, when to add a new validator.

3. GLOSSARY.md is new.  Every SEMI acronym used in the codebase or
   the docs gets one row: SVID, DVID, CEID, RPTID, ALID, ECID, PPID,
   MID, CARRIERID, PRJOBID, CTLJOBID, SUBSTID, OBJSPEC, OBJTYPE,
   MDLN, SOFTREV, EQPTYP, DATAID + every ACK code (COMMACK, ONLACK,
   OFLACK, HCACK, CMDA, ACKC5-7-10, DRACK, LRACK, ERACK, EAC, TIACK,
   GRANT, ALCD, OBJACK) + stream/function shorthand + HSMS terms +
   T-timers + E84 signals + the standards lineup + codebase shortcuts
   ("the model", "the router", "the proof", etc.).  Cuts week-1
   onboarding time.

4. FAQ.md is new.  Canonical answers to the questions that come up
   once per integration: why HSMS unencrypted, SVID vs DVID, PJ vs
   CJ, who fires FSM transitions, what runs on which thread, how to
   add a new SECS-II message, ASCII vs Binary, common MES quirks,
   how spool works, robustness fuzz vs libFuzzer, conformance vs
   interop, what's not implemented.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Raphael Maenle
2026-06-09 16:43:45 +02:00
parent db90a21e1d
commit 195ecc689f
4 changed files with 506 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
interop/README.md
+113 -39
View File
@@ -1,59 +1,133 @@
# secsgem-py interop harness
# External cross-validation harnesses
Cross-validates our C++ SECS-II / HSMS / GEM implementation against
[secsgem-py](https://pypi.org/project/secsgem/) 0.3.0, the de-facto
Python reference. Everything runs in Docker — no Python or secsgem-py
on the host.
Every harness in this directory exists so a reviewer doesn't have to
take our word for it. Each one validates our C++ codec / framing /
dispatch against an **independent third-party implementation** that
read the SEMI standards without talking to us.
## What it tests
See [`../VERIFICATION.md`](../VERIFICATION.md) for the full test plan
and the honest accounting of which proofs are external vs internal.
| Driver | Peer | Coverage |
| ------------------------------ | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `host_vs_cpp_server.py` | C++ `secs_server` (passive) | HSMS select/separate, S1F1/F3/F11/F17/F23, S2F13/F17/F29/F33/F35/F37/F41, S5F3/F5/F7, S5F1 unsolicited, S6F11 unsolicited, S7F3/F5/F19, S10F1/F3, S1F15 |
| `secs_interop_probe` (C++) | `passive_equipment.py` (secsgem-py GemEquipmentHandler) | HSMS select, S1F13/F14, S1F1/F2, S1F3/F4, clean separate |
| `raw_gem300_harness.py` | C++ `secs_server` (passive) | GEM 300 streams secsgem-py upstream doesn't ship: S3F17/F18 (E87 carrier action), S16F5/F6 (E40 PRJobCommand), S16F27/F28 (E94 CJobCommand) — built with custom `SecsStreamFunction` subclasses + registered custom `DataItem`s |
## What's here
24 named checks on the C++-server side; 4 explicit checks on the
C++-host side; 4 GEM-300 raw-frame checks. Implicit HSMS state-machine
and wire-level framing validation everywhere.
| Validator | Independence | Coverage |
|--------------------------------------------|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| `host_vs_cpp_server.py` + `passive_equipment.py` | secsgem-py 0.3.0 — Python reference impl | ~24 + 4 checks: S1, S2, S5, S6, S7, S10 happy paths |
| `raw_gem300_harness.py` | secsgem-py with hand-crafted SecsStreamFunctions | 3 checks: S3F17, S16F5, S16F27 (limited by SFDL grammar) |
| `secs4j/Secs4jHostHarness.java` | secs4java8 — Apache 2.0 Java impl by kenta-shimizu | **55 checks** across S1/S2/S3/S5/S6/S7/S10/S14/S16, including the full E40 body that defeated secsgem-py and unsolicited S6F11/S5F1 observation |
| `tshark_validate.sh` | Wireshark's built-in HSMS dissector | 69 captured frames dissected with no malformed-packet warnings |
| `spool_persistence_test.py` | secsgem-py + a docker-restart loop | Restart-survives-spool integrity |
| ⚙️ `../tests/test_e5_kat.cpp` | SEMI E5 §9 encoding rules | 196 known-answer byte assertions across every format code |
| ⚙️ `../apps/fuzz_secs2_decode.cpp` + `fuzz_sml_parse.cpp` | libFuzzer + ASan + UBSan | ~70 000 + ~285 000 random inputs per minute, 0 crashes |
## Running
The ⚙️ entries aren't in `interop/` directly because they don't
involve a network peer — they're either pure codec round-trips
(KAT) or coverage-guided fuzzing. Listed here so the full external
proof inventory lives in one place.
```bash
# Start C++ passive server, then drive it with secsgem-py host:
## Running each validator
### secsgem-py — secsgem-py active host → C++ server
```sh
docker compose up -d server
docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py \
--host server --port 5000 --session-id 0
```
# Start Python passive equipment, then probe it with the C++ host:
### secsgem-py — C++ host → secsgem-py equipment
```sh
docker compose up -d equipment_py
docker compose run --rm builder /app/build/secs_interop_probe \
--host equipment_py --port 5000 --device 0
```
Both exit 0 on success.
### secsgem-py — raw GEM 300 frames
## What this caught
```sh
docker compose up -d server
docker compose run --rm interop python3 /app/interop/raw_gem300_harness.py \
--host server --port 5000 --session-id 0
```
Real bugs surfaced by interop (now fixed):
### secs4j — independent Java host → C++ server
1. **Strict U4 parsing rejected U1-encoded identifiers.** SEMI E5
declares DATAID, RPTID, VID, CEID, ALID, EXID, etc. as
`U1 | U2 | U4 | U8`; secsgem-py picks the smallest width that fits.
Our `as_u4_scalar`, `as_u2_scalar`, etc. were strict. Now lenient
with range-checked downcasts (`messages_helpers.hpp::any_unsigned_first`).
2. **PPBODY rejected when sent as ASCII.** SEMI lets PPBODY be
```sh
bash interop/secs4j_validate.sh
```
Builds an `eclipse-temurin:21-jdk` sidecar with secs4java8 cloned +
compiled at image build, then drives 55 checks against
`compose up server`. See `secs4j/Secs4jHostHarness.java` for the
list and `secs4j/Dockerfile` for the build.
### tshark — Wireshark HSMS dissector
```sh
docker compose run --rm builder bash /app/interop/tshark_validate.sh
```
Captures a pcap of the demo flow, runs `tshark -V` with the HSMS
dissector forced for the test port, asserts no malformed packets +
that all expected control/data frames parse.
### spool persistence — restart-survives test
```sh
bash interop/spool_persistence_test.py
```
Drops the host link mid-flight, kills the server, restarts it, and
asserts the spooled S5F1 / S6F11 frames drain to the host on
reconnect.
## What these harnesses caught
Real bugs surfaced during interop development (now fixed):
1. **Strict U-width parsing rejected U1-encoded identifiers.** SEMI
E5 declares DATAID, RPTID, VID, CEID, ALID, EXID etc. as
`U1 | U2 | U4 | U8`; secsgem-py picks the smallest width that
fits. Our scalar accessors were strict. Now lenient with
range-checked downcasts (`messages_helpers.hpp::any_unsigned_first`).
2. **PPBODY rejected when sent as ASCII.** SEMI allows PPBODY to be
`ASCII | Binary | List`; secsgem-py defaults to ASCII. Added the
`BINARY_OR_ASCII` codegen item type plus a permissive
`as_text_or_binary` accessor, used for S7F3/F6.
3. **Missing S1F23 / S1F24 (Collection Event Namelist).** Added the
wire schema in `data/messages.yaml`, a `vids_for(ceid)` accessor on
the event-report store, and the dispatch handler in `secs_server.cpp`.
4. **Missing S10F3 handler (Terminal Display Single, host→equipment).**
Our server only registered S10F1; per SEMI E5, S10F1 is
equipment→host and S10F3 is the host→equipment counterpart. Added
the missing dispatch.
`BINARY_OR_ASCII` codegen type and the `as_text_or_binary`
accessor.
The C++ test suite still passes (278 cases / 1436 assertions) after
each of these changes — the fixes are purely permissive widenings, no
existing behaviour was broken.
3. **Missing S1F23 / S1F24 (Collection Event Namelist).** Added the
wire schema, the `vids_for(ceid)` accessor, and the dispatch
handler.
4. **Missing S10F3 handler (host→equipment Terminal Display).** Our
server only registered S10F1; per SEMI E5 §13 those are opposite
directions. Added the missing dispatch.
5. **TSan use-after-free in `act_exception_complete`** (test code,
not library): held a pointer across `fire_internal(RecoveryComplete)`
which deletes the entry. Found by the ThreadSanitizer lane on
first run.
The C++ test suite stayed green through every one of these fixes —
the changes were purely permissive widenings or additive features,
no existing behaviour broke.
## When to add a new validator
A new third-party SECS implementation, or a new dissector, or a new
fuzzer target — anything that exercises our wire surface from an
angle the existing five don't cover — is worth adding. The pattern
is consistent:
1. New script / harness lives here (or a sidecar Docker context for
non-Python validators).
2. Wired into `.gitea/workflows/ci.yml` as a separate job.
3. Listed in this README's table + in `../VERIFICATION.md`.
4. Surfaced in the README's proof-of-feature-completeness table if
it adds a meaningful new dimension.
Bug reports from a new validator → file at
`raphael@maenle.net` with the wire trace, the validator's output,
and the equipment YAML so we can reproduce.