Audit pass over the public-facing surface so a customer can read it
end-to-end without tripping on stale numbers or self-contradictions.
README + docs accuracy:
- Test counts 426 → 445, assertions 2 557 → 2 753 (verified via
doctest run); E5 row was missing test_e5_kat (19 cases)
- Interop checks 24 → 31, COMPLIANCE.md message count 149 → 164,
COMPLIANCE.md "291 cases / 1515 assertions" → 445 / 2 753
- README "60+ test IDs" for MES_INTEROP → actual 59
- PVD example counts: 32 SVIDs/17 CEIDs → 29/21, "~40 handlers
in ~200 lines" → 51 in ~460, "~700 lines" → ~1,100; main.cpp
header table-of-contents resynced with the actual 7 sections
Out-of-scope honesty (COMPLIANCE.md §8 + FAQ.md):
- Removed HSMS-GS (was both ✅ implemented in §1 and "out of scope"
in §8; INTEGRATION.md §7 documents using it)
- Removed multi-block SECS-I (split_message/assemble_message exist
with 4 dedicated tests)
- Added serial-port wiring as the genuine open ⬜ item — FSM is
tested end-to-end over TCP; only the asio serial_port glue is
deferred
- COMPLIANCE.md intro now lists E42 and notes "E37 (SS + GS)"
README restructure:
- Moved the 8-command proof table and per-standard test-coverage
table to a new PROOFS.md (72 lines)
- README now leads with what / Quick start / Documentation map,
then a one-paragraph "How it's proved" linking to PROOFS.md
- Updated cross-refs in FAQ.md, GLOSSARY.md, VERIFICATION.md, and
interop/README.md to point at PROOFS.md
CI fix — tshark-dissector job:
- interop/tshark_validate.sh hardcoded /app/build/secs_server etc.
which only works inside the docker image. Now derives ROOT from
the script's own location and accepts BUILD/SERVER/CLIENT/DATA
env overrides, so CI can run it from the workspace dir
- Verified still passes in docker (69 frames, 0 malformed)
.gitignore:
- Added build-fuzz/ and build-tsan/ (were showing as untracked)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fills four documentation gaps surfaced by the doc audit:
1. README "Documentation map" was missing VERIFICATION.md (the file
that backs the proof-of-feature-completeness claims) and is now
pointing at the new files added in this commit too — ARCHITECTURE,
GLOSSARY, FAQ, examples/pvd_tool/ (the last two land next).
2. interop/README.md only documented secsgem-py. Three of the five
external validators (tshark, secs4j, libFuzzer) plus the E5 KAT
were invisible from the directory's own README. Rewritten as a
complete index — what's external, what each catches, how to run,
what bugs they've already surfaced, when to add a new validator.
3. GLOSSARY.md is new. Every SEMI acronym used in the codebase or
the docs gets one row: SVID, DVID, CEID, RPTID, ALID, ECID, PPID,
MID, CARRIERID, PRJOBID, CTLJOBID, SUBSTID, OBJSPEC, OBJTYPE,
MDLN, SOFTREV, EQPTYP, DATAID + every ACK code (COMMACK, ONLACK,
OFLACK, HCACK, CMDA, ACKC5-7-10, DRACK, LRACK, ERACK, EAC, TIACK,
GRANT, ALCD, OBJACK) + stream/function shorthand + HSMS terms +
T-timers + E84 signals + the standards lineup + codebase shortcuts
("the model", "the router", "the proof", etc.). Cuts week-1
onboarding time.
4. FAQ.md is new. Canonical answers to the questions that come up
once per integration: why HSMS unencrypted, SVID vs DVID, PJ vs
CJ, who fires FSM transitions, what runs on which thread, how to
add a new SECS-II message, ASCII vs Binary, common MES quirks,
how spool works, robustness fuzz vs libFuzzer, conformance vs
interop, what's not implemented.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Every check the user could ask for now lands. secs4j's
comm.send(stream, function, w, body) takes arbitrary S/F + arbitrary
Secs2 body, so coverage was never coverage-limited by the Java side
— the original 20 was just the minimum to fill the gaps secsgem-py
couldn't reach.
Adds:
- Status data: S1F3, S1F11
- EC management: S2F13, S2F15 (set TimeFormat), S2F29
- Event reports: S2F33, S2F35, S2F37 (full define-link-enable
sequence), S6F15, S6F19, S6F21
- Remote control: S2F41 (modern RCMD=START + observed S6F11),
S2F21 (legacy RCMD=STOP),
S2F41 RCMD=FAULT + observed S5F1
- Alarms: S5F3, S5F5, S5F7
- Spool: S2F43, S6F23
- PP management: S7F1, S7F3, S7F5, S7F17, S7F19
- Terminal: S10F3 (single), S10F5 (multi-line)
- E40 PJ: S16F11 (full E40 body — MF + PRRECIPEMETHOD +
RecipeSpec + mtrloutspec + processparams),
S16F7 (monitor), S16F13 (dequeue)
- Limits: S2F45, S2F47
- Trace: S2F23 (5-field body)
- E39: S14F1 (GetAttr)
Plus a SecsMessageReceiveListener that captures every equipment-
initiated primary into a ConcurrentLinkedQueue and replies to S5F1
(ACKC5=0), S6F11 (ACKC6=0), S16F9 (W=0 no reply) so the
equipment's T3 doesn't fire on our watch. Two checks now assert
the unsolicited path:
- After RCMD=START, an S6F11 with the linked report must arrive
within 400ms
- After RCMD=FAULT, an S5F1 with the alarm must arrive within
400ms
Both observed against the demo equipment.
Result: 55/55 PASS. Two independent implementations
(secsgem-py + secs4java8) now corroborate the wire surface in
overlapping but distinct slices. Full E40 body — the one that
defeated secsgem-py's SFDL grammar — round-trips cleanly through
secs4j.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
20 cross-validation checks PASS against [secs4java8] (Apache 2.0,
kenta-shimizu) — an independent SECS/HSMS implementation in Java by
a different author from a different language ecosystem. Distinct
implementer = independent spec interpretation. Two libraries
agreeing on wire bytes is much stronger evidence of spec-correctness
than either alone.
Coverage targets the gap the secsgem-py interop deliberately skipped
(secsgem-py's SFDL grammar couldn't easily express GEM 300 bodies
with variable lists of named scalars):
- S1F1/F13/F17/F19/F21/F23 — establish comms + namelists
- S2F17 — clock
- S2F23 — trace init (5-field body)
- S2F49 — enhanced remote command (DATAID + OBJSPEC + RCMD + params)
- S3F17/F19/F25/F27 — full E87 carrier surface (action, slot map
verify, transfer with port pair, cancel)
- S5F13/F17 — exception recovery (EXID + EXRECVRA)
- S14F9/F11 — E94 CJ create with prjobids list, CJ delete
- S16F5/F27 — E40 PJ command, E94 CJ command
- S1F15 — offline cleanup
20/20 PASS against the demo equipment. Reply S/F matches the spec
for every transaction; specific ACK values vary by equipment state
(CarrierIDUnknown for an unknown carrier is just as valid as Accept
for a known one) so we assert on the wire shape, not the result.
Ship layout:
interop/secs4j/Dockerfile — eclipse-temurin:21-jdk + clone
+ build of secs4java8 → Export.jar
interop/secs4j/Secs4jHostHarness.java
— 20 round_trip assertions; uses
Secs2.list/uint4/ascii to build
full GEM 300 bodies; comm.send()
for arbitrary S/F pairs
interop/secs4j_validate.sh — orchestrator: builds image,
compiles harness, starts compose
server, runs Java container on
the secs network against it
.gitea/workflows/ci.yml — secs4j-interop job in CI
README.md — proof table grows to 7 commands
.gitignore — *.class
After this commit our proof chain has:
- SEMI E5 KAT (standards body's own arithmetic)
- tshark dissector (Wireshark's HSMS impl)
- secsgem-py interop (Python reference impl)
- **secs4j interop** (independent Java impl)
+ 426 unit tests, 47 conformance harness checks, 100k random ops,
YAML validation
Four independent external proofs, three of them on overlapping wire
surface from independent angles.
Plan: VERIFICATION.md §3.
[secs4java8]: https://github.com/kenta-shimizu/secs4java8
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wireshark's built-in HSMS dissector — written by network-protocol
authors who don't know us, didn't talk to us, and don't share
implementation details with secsgem-py — is a third independent codec
for our framing. If they parse our pcap without warnings, our HSMS
framing is wire-correct independently of both our internal tests and
the secsgem-py interop path.
interop/tshark_validate.sh:
- Boots secs_server on 127.0.0.1:5099 (away from the demo port)
- Captures the loopback wire traffic with tcpdump
- Runs secs_client through ~24 transactions plus Separate.req +
TCP FIN
- Parses the pcap with tshark -V using the HSMS dissector
- Asserts: no "Malformed Packet", no "Dissector bug", at least one
HSMS frame, expected tokens present (Select.req/rsp, Separate.req,
Data message), reports histogram (count by control type + distinct
S/F pairs)
Result against the demo: 69 HSMS frames dissected, 49 distinct
S/F pairs (S01F01..S16F28), all clean.
Dockerfile gains tshark + tcpdump. .gitea/workflows/ci.yml gains a
`tshark-dissector` job that runs this validator as part of every
push to main. README proof table grows to 6 commands.
VERIFICATION.md §1a documents a follow-up: round-trip the KAT
fixtures through secsgem-py to corroborate that the format codes
we used match an independent implementation. Strengthens the KAT
proof from "internally consistent" to "confirmed by a second
implementer who read the spec without talking to us."
Plan: VERIFICATION.md §2.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds round-trip checks for the SECS-II messages added in the AA
catalog-growth commit but never cross-validated against secsgem-py:
* S2F21/F22 — legacy remote command (no params). secsgem-py's
stock S2F21 sends with W=0; we register a W=1 override so the
transaction awaits our S2F22 reply. Also widens CMDA's allowed
types to include Binary (secsgem-py 0.3.0 declares CMDA as
Dynamic[U1, I1] only; SEMI E5 §10.18 says Binary, and our server
emits it that way).
* S6F15/F16 — event-report request by CEID.
* S6F19/F20 — individual report request by RPTID.
* S6F21/F22 — annotated individual report request.
* S7F1/F2 — PP load inquire.
* S7F17/F18 — PP delete.
Suite is now 32 named host-vs-server checks — all green in three
consecutive runs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a docker-compose service `server-spool` that runs secs_server
with --spool-dir pointed at a named volume. Two-phase Python
harness (interop/spool_persistence_test.py):
1. Enqueue phase: force-spool one S6F11(CEID=300) via the
SPOOL_ON / START / SPOOL_OFF RCMD trio, then disconnect.
2. Driver runs `docker compose restart server-spool` between
the phases — the named volume preserves the journal files.
3. Drain phase: reconnect, send S6F23(Transmit), verify the
replayed S6F11 carries CEID 300.
Surfaces a real interop bug along the way: secsgem-py 0.3.0 encodes
RSDC (and other "single-byte status" fields) as <U1>, while SEMI E5
spells them as <B>. Our `as_binary_first` was strict on Binary; now
accepts either (the byte semantics are identical, and the leniency is
symmetric with the U-type widening from the first interop commit).
Result: enqueue → docker restart → drain returns CEID 300 cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cross-validates the GEM 300 streams secsgem-py 0.3.0 doesn't ship
(S3 carriers, S14 control jobs, S16 process jobs) by minting custom
`SecsStreamFunction` subclasses on the fly and registering the
matching `DataItem` definitions (CARRIERID, CTLJOBID, PRJOBID, PRCMD,
CTLJOBCMD, MF, …) with `secsgem.secs.data_items`.
Drives the C++ passive server through:
* S3F17/F18 (E87 carrier action) — server replies CarrierIDUnknown
for the unregistered carrier.
* S16F5/F6 (E40 PRJobCommand) — server returns InvalidObject
for the nonexistent PJ.
* S16F27/F28 (E94 CJobCommand) — server cascades CJSTART.
Scope cut: S16F11 full-body and S14F9 (both have variable-length
nested lists with named scalar elements) hit a quirk of secsgem-py's
SFDL tokenizer where `< L name > <SCALAR> >` parses as a fixed-1
list, not a variable-length list of SCALARs. The full-body S16F11
is already round-tripped by the C++ unit tests (and via secsgem-py's
host driver in `host_vs_cpp_server.py`), so the raw harness focuses
on the no-variable-list messages where the SFDL grammar cooperates.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a Docker-based interop harness that drives the C++ server with
secsgem-py 0.3.0 as the active host and probes a secsgem-py-passive
equipment from a minimal C++ active client. Surfaces and fixes four
interoperability bugs uncovered by cross-testing:
* SEMI E5 identifier formatcodes are a U1|U2|U4|U8 wildcard;
secsgem-py picks the narrowest fitting width while our parsers
only accepted U4. `as_uN_scalar` / `as_iN_scalar` now accept
any unsigned/signed width and range-check the downcast.
* PPBODY (S7F3/F6) is "ASCII | Binary | List" per the spec;
secsgem-py defaults to ASCII. Added BINARY_OR_ASCII codegen
item type with `as_text_or_binary` accessor.
* S1F23/F24 Collection Event Namelist was unimplemented; added
schema + `vids_for(ceid)` accessor on EventReportSubscriptions
plus the dispatch handler.
* S10F1 was registered as a host->equipment handler, but per
SEMI E5 §12 S10F1 is equipment->host; S10F3 is the actual
host->equipment Terminal Display Single. Added an S10F3
handler alongside (we keep S10F1 too for backward compat).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
raphael