The HCACK-4 contract, implemented end to end. For every YAML-declared
command the service registers a forwarding handler (new HostCommandRegistry
names()/spec() accessors): with a subscribed tool client the command is
queued onto the Subscribe stream (id + name + params via from_item) and the
host is answered S2F42 HCACK=4 immediately — never blocking the io thread or
the T3 window; with NO subscriber the command takes its declarative YAML ack
(the honest pre-daemon behaviour). Settled + documented in the proto: v1 is
a firehose with no buffering/replay. CompleteCommand correlates the pending
id (audit; unknown id => PARAMETER_INVALID). Side effects stay suppressed on
HCACK-4 (router applies them only on Accept), so the completion event the
TOOL fires is the host's real signal — exactly E30's intent.
Tests (daemon suite 101 -> 124 assertions): a real S2F41 dispatched through
the full default-handler router ON the io thread under run_async — HCACK 4
with subscriber + params on the stream, declarative Accept without,
CompleteCommand known/unknown, fallback restored after unsubscribe.
Interop (now 20 checks, all green): the complete conformant loop against
the secsgem-py reference host — S2F41 START -> S2F42 HCACK=4 -> tool
receives Command(name=START, id=1) -> CompleteCommand -> FireEvent -> host
receives S6F11.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A2 — alarms: optional 'name:' on alarm config (a LOCAL key — SEMI only
defines numeric ALID + freetext ALTX; field appended last so existing
{id, text, category} brace-inits compile unchanged), parsed by the loader,
checked by the validator, shipped in equipment.yaml. SetAlarm/ClearAlarm
RPCs resolve config name OR stringified ALID via a constructor snapshot.
A3 — control state + health: RequestControlState fires operator events on
the io thread (read_sync) and reports what the E30 table actually did —
ACCEPT iff the equipment landed in the requested state, CANNOT_DO_NOW naming
the actual state otherwise (the shipped table has no operator path to
EquipmentOffline; the test pins that honesty). ATTEMPT_ONLINE is rejected as
transient. WatchHealth streams an immediate snapshot then pushes on link/
control-state changes via service observers (add_link_observer +
add_control_state_observer — the HandlerSlot work paying off), spool depth
sampled at the 500ms poll; ends on cancel or engine stop.
Tests: daemon suite 61 -> 101 assertions (alarm lifecycle by name/id/unknown,
WatchHealth initial + change push, all four RequestControlState semantics);
loader test for the alarm name (present + absent fallback); core 467/3055.
Interop now 15 checks incl. gRPC SetAlarm -> host receives S5F1 ALCD=0x84
ALID=1, and RequestControlState(HOST_OFFLINE) -> GetControlState confirms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EquipmentRuntime::read_sync establishes THE pattern for reading mutable
engine state from gRPC/binding threads (Phase 0 item 6): post the read onto
the io thread (the model's single owner), wait on a future with a deadline,
nullopt => UNAVAILABLE at the RPC edge. Always truthful, no cache to
invalidate; milliseconds are irrelevant at SECS rates.
GetVariables: name resolution against the service snapshot (empty query =
all; unknown name => INVALID_ARGUMENT naming the offender), values read via
read_sync, converted by the new from_item reverse conversion (single-element
numeric arrays => scalars, multi-element => List; Boolean/Binary/text per
format; C2-as-integer and U8>2^63 wrap documented as TODOs).
Tests run the engine in run_async — the daemon's PRODUCTION threading mode,
previously untested — and round-trip through both conversions: SetVariables
(declared-format write) then GetVariables (read) over a real in-process
channel. Daemon suite 41 -> 61 assertions. daemon_interop.py gains a live
GetVariables round-trip check vs the running daemon (verified green).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
daemon_interop.py drives a running secs_gemd through BOTH faces at once: a
gRPC tool client and a secsgem-py active host. Proves the gRPC<->HSMS bridge
against a reference GEM implementation, not just in-process:
- gRPC GetControlState agrees with the HSMS-driven control state
- gRPC SetVariables(ChamberPressure=2.5) + FireEvent(ProcessStarted) makes
the host receive S6F11 CEID 300 carrying 2.5 (value flowed gRPC -> engine
-> HSMS -> host)
- unknown variable/event names rejected at the gRPC edge
Mirrors the existing host_vs_cpp_server.py pattern. New 'gemd' compose service
(HSMS :5000 + gRPC :50051); interop image gains grpcio/grpcio-tools (proto
stubs generated at runtime, flat to avoid the secsgem package-name clash).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tone pass across the non-tutorial markdown — README, PROOFS,
ARCHITECTURE, BENCHMARKS, COMPLIANCE, FAQ, MES_INTEROP, SECURITY,
and interop/README. Three patterns came out:
- Bug-history war stories ("Past interop sweeps surfaced…",
"What these harnesses caught: 1. Strict U-width parsing…").
- Chat-with-reader framing ("Don't skip TLS unless…", "Treat as a
punch list", "If you're running in a pod…", "Misconfiguration
incidents drop dramatically").
- Self-referential narration ("we ship", "our codec", "the
codebase's most-tested layer", "three orders of magnitude above
fab load", "the gift that keeps giving").
README also drops the standalone ThreadSanitizer subsection under
Build details (now a single line under the new Testing section).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The harness previously bound the source tree into a compose service
and built inside it. That breaks under docker-in-docker (gitea-act,
GitHub Actions runners with /var/run/docker.sock mounted) because
bind-mount sources resolve against the *host* daemon's filesystem,
not the runner container's. Now Dockerfile.server bakes a Release
secs_server into its own image, and secs4j_validate.sh wires server
and harness together on a dedicated bridge — no volumes needed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Second secs4j-interop CI failure:
ensuring secs4j-interop image is built...
compiling Secs4jHostHarness.java...
error: file not found: Secs4jHostHarness.java
FAIL: javac
The script bind-mounted $PWD/interop/secs4j into /work inside the
container so it could javac the harness at runtime. That works
locally where docker daemon and script share a filesystem, but
fails in CI: the act runner runs the workflow inside a container,
the docker socket is mounted from the host, and the daemon
interprets bind-mount paths against the host filesystem — where
$PWD/interop/secs4j doesn't exist. Result: empty /work, javac
errors, job fails.
Fix: COPY Secs4jHostHarness.java into the image and javac it at
image build time. The script just runs the container — no bind
mount, no docker-in-docker mount path translation, works in CI and
locally.
Verified locally with a fresh image rebuild: 55/55 checks pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Picks up the file renames that landed alongside the previous commit
and fixes everything that pointed at the old root locations:
- README.md doc-map updated: every entry now points at docs/X.md,
with a new "docs/" lead entry pointing at the guided-tour index.
- README inline cross-refs (ARCHITECTURE / INTEGRATION / SECURITY /
BENCHMARKS / MES_INTEROP / PROOFS) repointed to docs/.
- README "Interop" section rewritten — used to mention only
secsgem-py; now covers all four external validators (secsgem-py
31 / secs4java8 55 / tshark 69 frames / libFuzzer 200 k+ runs)
with a one-line summary each, plus pointers to interop/README.md
and docs/VERIFICATION.md.
- README "Deferred follow-ups" cleaned: dropped the explanatory
"Listed here so reviewers don't go looking for them in
COMPLIANCE.md and find an 'out of scope' entry that sounds
defensive" sentence — the section header speaks for itself.
- docs/00_index.md "Where the rest of the docs live" table: dropped
every `../` prefix since the docs are now siblings.
- docs/01_what_is_secs_gem.md PROOFS reference updated to sibling.
- docs/02_the_cast.md INTEGRATION + MES_INTEROP refs updated to
siblings; dropped the stale "at the repo root" wording.
- interop/README.md: VERIFICATION + PROOFS refs updated to
../docs/X.md; stale "~24 + 4 checks" updated to 31 (matches
PROOFS.md and README).
- examples/pvd_tool/README.md: every doc cross-ref now points at
../../docs/X.md.
- Source / data / CI comments mentioning doc names (e.g.
"INTEGRATION.md §3", "COMPLIANCE.md gap") rewritten to
"docs/INTEGRATION.md §3" etc. — affects 9 files across
include/, apps/, tests/, data/, examples/, .gitea/workflows/.
Verified: full build under docker passes, 445/445 test cases pass,
2 753/2 753 assertions pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Audit pass over the public-facing surface so a customer can read it
end-to-end without tripping on stale numbers or self-contradictions.
README + docs accuracy:
- Test counts 426 → 445, assertions 2 557 → 2 753 (verified via
doctest run); E5 row was missing test_e5_kat (19 cases)
- Interop checks 24 → 31, COMPLIANCE.md message count 149 → 164,
COMPLIANCE.md "291 cases / 1515 assertions" → 445 / 2 753
- README "60+ test IDs" for MES_INTEROP → actual 59
- PVD example counts: 32 SVIDs/17 CEIDs → 29/21, "~40 handlers
in ~200 lines" → 51 in ~460, "~700 lines" → ~1,100; main.cpp
header table-of-contents resynced with the actual 7 sections
Out-of-scope honesty (COMPLIANCE.md §8 + FAQ.md):
- Removed HSMS-GS (was both ✅ implemented in §1 and "out of scope"
in §8; INTEGRATION.md §7 documents using it)
- Removed multi-block SECS-I (split_message/assemble_message exist
with 4 dedicated tests)
- Added serial-port wiring as the genuine open ⬜ item — FSM is
tested end-to-end over TCP; only the asio serial_port glue is
deferred
- COMPLIANCE.md intro now lists E42 and notes "E37 (SS + GS)"
README restructure:
- Moved the 8-command proof table and per-standard test-coverage
table to a new PROOFS.md (72 lines)
- README now leads with what / Quick start / Documentation map,
then a one-paragraph "How it's proved" linking to PROOFS.md
- Updated cross-refs in FAQ.md, GLOSSARY.md, VERIFICATION.md, and
interop/README.md to point at PROOFS.md
CI fix — tshark-dissector job:
- interop/tshark_validate.sh hardcoded /app/build/secs_server etc.
which only works inside the docker image. Now derives ROOT from
the script's own location and accepts BUILD/SERVER/CLIENT/DATA
env overrides, so CI can run it from the workspace dir
- Verified still passes in docker (69 frames, 0 malformed)
.gitignore:
- Added build-fuzz/ and build-tsan/ (were showing as untracked)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fills four documentation gaps surfaced by the doc audit:
1. README "Documentation map" was missing VERIFICATION.md (the file
that backs the proof-of-feature-completeness claims) and is now
pointing at the new files added in this commit too — ARCHITECTURE,
GLOSSARY, FAQ, examples/pvd_tool/ (the last two land next).
2. interop/README.md only documented secsgem-py. Three of the five
external validators (tshark, secs4j, libFuzzer) plus the E5 KAT
were invisible from the directory's own README. Rewritten as a
complete index — what's external, what each catches, how to run,
what bugs they've already surfaced, when to add a new validator.
3. GLOSSARY.md is new. Every SEMI acronym used in the codebase or
the docs gets one row: SVID, DVID, CEID, RPTID, ALID, ECID, PPID,
MID, CARRIERID, PRJOBID, CTLJOBID, SUBSTID, OBJSPEC, OBJTYPE,
MDLN, SOFTREV, EQPTYP, DATAID + every ACK code (COMMACK, ONLACK,
OFLACK, HCACK, CMDA, ACKC5-7-10, DRACK, LRACK, ERACK, EAC, TIACK,
GRANT, ALCD, OBJACK) + stream/function shorthand + HSMS terms +
T-timers + E84 signals + the standards lineup + codebase shortcuts
("the model", "the router", "the proof", etc.). Cuts week-1
onboarding time.
4. FAQ.md is new. Canonical answers to the questions that come up
once per integration: why HSMS unencrypted, SVID vs DVID, PJ vs
CJ, who fires FSM transitions, what runs on which thread, how to
add a new SECS-II message, ASCII vs Binary, common MES quirks,
how spool works, robustness fuzz vs libFuzzer, conformance vs
interop, what's not implemented.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Every check the user could ask for now lands. secs4j's
comm.send(stream, function, w, body) takes arbitrary S/F + arbitrary
Secs2 body, so coverage was never coverage-limited by the Java side
— the original 20 was just the minimum to fill the gaps secsgem-py
couldn't reach.
Adds:
- Status data: S1F3, S1F11
- EC management: S2F13, S2F15 (set TimeFormat), S2F29
- Event reports: S2F33, S2F35, S2F37 (full define-link-enable
sequence), S6F15, S6F19, S6F21
- Remote control: S2F41 (modern RCMD=START + observed S6F11),
S2F21 (legacy RCMD=STOP),
S2F41 RCMD=FAULT + observed S5F1
- Alarms: S5F3, S5F5, S5F7
- Spool: S2F43, S6F23
- PP management: S7F1, S7F3, S7F5, S7F17, S7F19
- Terminal: S10F3 (single), S10F5 (multi-line)
- E40 PJ: S16F11 (full E40 body — MF + PRRECIPEMETHOD +
RecipeSpec + mtrloutspec + processparams),
S16F7 (monitor), S16F13 (dequeue)
- Limits: S2F45, S2F47
- Trace: S2F23 (5-field body)
- E39: S14F1 (GetAttr)
Plus a SecsMessageReceiveListener that captures every equipment-
initiated primary into a ConcurrentLinkedQueue and replies to S5F1
(ACKC5=0), S6F11 (ACKC6=0), S16F9 (W=0 no reply) so the
equipment's T3 doesn't fire on our watch. Two checks now assert
the unsolicited path:
- After RCMD=START, an S6F11 with the linked report must arrive
within 400ms
- After RCMD=FAULT, an S5F1 with the alarm must arrive within
400ms
Both observed against the demo equipment.
Result: 55/55 PASS. Two independent implementations
(secsgem-py + secs4java8) now corroborate the wire surface in
overlapping but distinct slices. Full E40 body — the one that
defeated secsgem-py's SFDL grammar — round-trips cleanly through
secs4j.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
20 cross-validation checks PASS against [secs4java8] (Apache 2.0,
kenta-shimizu) — an independent SECS/HSMS implementation in Java by
a different author from a different language ecosystem. Distinct
implementer = independent spec interpretation. Two libraries
agreeing on wire bytes is much stronger evidence of spec-correctness
than either alone.
Coverage targets the gap the secsgem-py interop deliberately skipped
(secsgem-py's SFDL grammar couldn't easily express GEM 300 bodies
with variable lists of named scalars):
- S1F1/F13/F17/F19/F21/F23 — establish comms + namelists
- S2F17 — clock
- S2F23 — trace init (5-field body)
- S2F49 — enhanced remote command (DATAID + OBJSPEC + RCMD + params)
- S3F17/F19/F25/F27 — full E87 carrier surface (action, slot map
verify, transfer with port pair, cancel)
- S5F13/F17 — exception recovery (EXID + EXRECVRA)
- S14F9/F11 — E94 CJ create with prjobids list, CJ delete
- S16F5/F27 — E40 PJ command, E94 CJ command
- S1F15 — offline cleanup
20/20 PASS against the demo equipment. Reply S/F matches the spec
for every transaction; specific ACK values vary by equipment state
(CarrierIDUnknown for an unknown carrier is just as valid as Accept
for a known one) so we assert on the wire shape, not the result.
Ship layout:
interop/secs4j/Dockerfile — eclipse-temurin:21-jdk + clone
+ build of secs4java8 → Export.jar
interop/secs4j/Secs4jHostHarness.java
— 20 round_trip assertions; uses
Secs2.list/uint4/ascii to build
full GEM 300 bodies; comm.send()
for arbitrary S/F pairs
interop/secs4j_validate.sh — orchestrator: builds image,
compiles harness, starts compose
server, runs Java container on
the secs network against it
.gitea/workflows/ci.yml — secs4j-interop job in CI
README.md — proof table grows to 7 commands
.gitignore — *.class
After this commit our proof chain has:
- SEMI E5 KAT (standards body's own arithmetic)
- tshark dissector (Wireshark's HSMS impl)
- secsgem-py interop (Python reference impl)
- **secs4j interop** (independent Java impl)
+ 426 unit tests, 47 conformance harness checks, 100k random ops,
YAML validation
Four independent external proofs, three of them on overlapping wire
surface from independent angles.
Plan: VERIFICATION.md §3.
[secs4java8]: https://github.com/kenta-shimizu/secs4java8
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wireshark's built-in HSMS dissector — written by network-protocol
authors who don't know us, didn't talk to us, and don't share
implementation details with secsgem-py — is a third independent codec
for our framing. If they parse our pcap without warnings, our HSMS
framing is wire-correct independently of both our internal tests and
the secsgem-py interop path.
interop/tshark_validate.sh:
- Boots secs_server on 127.0.0.1:5099 (away from the demo port)
- Captures the loopback wire traffic with tcpdump
- Runs secs_client through ~24 transactions plus Separate.req +
TCP FIN
- Parses the pcap with tshark -V using the HSMS dissector
- Asserts: no "Malformed Packet", no "Dissector bug", at least one
HSMS frame, expected tokens present (Select.req/rsp, Separate.req,
Data message), reports histogram (count by control type + distinct
S/F pairs)
Result against the demo: 69 HSMS frames dissected, 49 distinct
S/F pairs (S01F01..S16F28), all clean.
Dockerfile gains tshark + tcpdump. .gitea/workflows/ci.yml gains a
`tshark-dissector` job that runs this validator as part of every
push to main. README proof table grows to 6 commands.
VERIFICATION.md §1a documents a follow-up: round-trip the KAT
fixtures through secsgem-py to corroborate that the format codes
we used match an independent implementation. Strengthens the KAT
proof from "internally consistent" to "confirmed by a second
implementer who read the spec without talking to us."
Plan: VERIFICATION.md §2.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds round-trip checks for the SECS-II messages added in the AA
catalog-growth commit but never cross-validated against secsgem-py:
* S2F21/F22 — legacy remote command (no params). secsgem-py's
stock S2F21 sends with W=0; we register a W=1 override so the
transaction awaits our S2F22 reply. Also widens CMDA's allowed
types to include Binary (secsgem-py 0.3.0 declares CMDA as
Dynamic[U1, I1] only; SEMI E5 §10.18 says Binary, and our server
emits it that way).
* S6F15/F16 — event-report request by CEID.
* S6F19/F20 — individual report request by RPTID.
* S6F21/F22 — annotated individual report request.
* S7F1/F2 — PP load inquire.
* S7F17/F18 — PP delete.
Suite is now 32 named host-vs-server checks — all green in three
consecutive runs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a docker-compose service `server-spool` that runs secs_server
with --spool-dir pointed at a named volume. Two-phase Python
harness (interop/spool_persistence_test.py):
1. Enqueue phase: force-spool one S6F11(CEID=300) via the
SPOOL_ON / START / SPOOL_OFF RCMD trio, then disconnect.
2. Driver runs `docker compose restart server-spool` between
the phases — the named volume preserves the journal files.
3. Drain phase: reconnect, send S6F23(Transmit), verify the
replayed S6F11 carries CEID 300.
Surfaces a real interop bug along the way: secsgem-py 0.3.0 encodes
RSDC (and other "single-byte status" fields) as <U1>, while SEMI E5
spells them as <B>. Our `as_binary_first` was strict on Binary; now
accepts either (the byte semantics are identical, and the leniency is
symmetric with the U-type widening from the first interop commit).
Result: enqueue → docker restart → drain returns CEID 300 cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cross-validates the GEM 300 streams secsgem-py 0.3.0 doesn't ship
(S3 carriers, S14 control jobs, S16 process jobs) by minting custom
`SecsStreamFunction` subclasses on the fly and registering the
matching `DataItem` definitions (CARRIERID, CTLJOBID, PRJOBID, PRCMD,
CTLJOBCMD, MF, …) with `secsgem.secs.data_items`.
Drives the C++ passive server through:
* S3F17/F18 (E87 carrier action) — server replies CarrierIDUnknown
for the unregistered carrier.
* S16F5/F6 (E40 PRJobCommand) — server returns InvalidObject
for the nonexistent PJ.
* S16F27/F28 (E94 CJobCommand) — server cascades CJSTART.
Scope cut: S16F11 full-body and S14F9 (both have variable-length
nested lists with named scalar elements) hit a quirk of secsgem-py's
SFDL tokenizer where `< L name > <SCALAR> >` parses as a fixed-1
list, not a variable-length list of SCALARs. The full-body S16F11
is already round-tripped by the C++ unit tests (and via secsgem-py's
host driver in `host_vs_cpp_server.py`), so the raw harness focuses
on the no-variable-list messages where the SFDL grammar cooperates.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a Docker-based interop harness that drives the C++ server with
secsgem-py 0.3.0 as the active host and probes a secsgem-py-passive
equipment from a minimal C++ active client. Surfaces and fixes four
interoperability bugs uncovered by cross-testing:
* SEMI E5 identifier formatcodes are a U1|U2|U4|U8 wildcard;
secsgem-py picks the narrowest fitting width while our parsers
only accepted U4. `as_uN_scalar` / `as_iN_scalar` now accept
any unsigned/signed width and range-check the downcast.
* PPBODY (S7F3/F6) is "ASCII | Binary | List" per the spec;
secsgem-py defaults to ASCII. Added BINARY_OR_ASCII codegen
item type with `as_text_or_binary` accessor.
* S1F23/F24 Collection Event Namelist was unimplemented; added
schema + `vids_for(ceid)` accessor on EventReportSubscriptions
plus the dispatch handler.
* S10F1 was registered as a host->equipment handler, but per
SEMI E5 §12 S10F1 is equipment->host; S10F3 is the actual
host->equipment Terminal Display Single. Added an S10F3
handler alongside (we keep S10F1 too for backward compat).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
raphael