raphael
b031f057af
Audit pass over the public-facing surface so a customer can read it end-to-end without tripping on stale numbers or self-contradictions. README + docs accuracy: - Test counts 426 → 445, assertions 2 557 → 2 753 (verified via doctest run); E5 row was missing test_e5_kat (19 cases) - Interop checks 24 → 31, COMPLIANCE.md message count 149 → 164, COMPLIANCE.md "291 cases / 1515 assertions" → 445 / 2 753 - README "60+ test IDs" for MES_INTEROP → actual 59 - PVD example counts: 32 SVIDs/17 CEIDs → 29/21, "~40 handlers in ~200 lines" → 51 in ~460, "~700 lines" → ~1,100; main.cpp header table-of-contents resynced with the actual 7 sections Out-of-scope honesty (COMPLIANCE.md §8 + FAQ.md): - Removed HSMS-GS (was both ✅ implemented in §1 and "out of scope" in §8; INTEGRATION.md §7 documents using it) - Removed multi-block SECS-I (split_message/assemble_message exist with 4 dedicated tests) - Added serial-port wiring as the genuine open ⬜ item — FSM is tested end-to-end over TCP; only the asio serial_port glue is deferred - COMPLIANCE.md intro now lists E42 and notes "E37 (SS + GS)" README restructure: - Moved the 8-command proof table and per-standard test-coverage table to a new PROOFS.md (72 lines) - README now leads with what / Quick start / Documentation map, then a one-paragraph "How it's proved" linking to PROOFS.md - Updated cross-refs in FAQ.md, GLOSSARY.md, VERIFICATION.md, and interop/README.md to point at PROOFS.md CI fix — tshark-dissector job: - interop/tshark_validate.sh hardcoded /app/build/secs_server etc. which only works inside the docker image. Now derives ROOT from the script's own location and accepts BUILD/SERVER/CLIENT/DATA env overrides, so CI can run it from the workspace dir - Verified still passes in docker (69 frames, 0 malformed) .gitignore: - Added build-fuzz/ and build-tsan/ (were showing as untracked) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
134 lines
6.1 KiB
Markdown
134 lines
6.1 KiB
Markdown
# External cross-validation harnesses
|
|
|
|
Every harness in this directory exists so a reviewer doesn't have to
|
|
take our word for it. Each one validates our C++ codec / framing /
|
|
dispatch against an **independent third-party implementation** that
|
|
read the SEMI standards without talking to us.
|
|
|
|
See [`../VERIFICATION.md`](../VERIFICATION.md) for the full test plan
|
|
and the honest accounting of which proofs are external vs internal.
|
|
|
|
## What's here
|
|
|
|
| Validator | Independence | Coverage |
|
|
|--------------------------------------------|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
|
|
| `host_vs_cpp_server.py` + `passive_equipment.py` | secsgem-py 0.3.0 — Python reference impl | ~24 + 4 checks: S1, S2, S5, S6, S7, S10 happy paths |
|
|
| `raw_gem300_harness.py` | secsgem-py with hand-crafted SecsStreamFunctions | 3 checks: S3F17, S16F5, S16F27 (limited by SFDL grammar) |
|
|
| `secs4j/Secs4jHostHarness.java` | secs4java8 — Apache 2.0 Java impl by kenta-shimizu | **55 checks** across S1/S2/S3/S5/S6/S7/S10/S14/S16, including the full E40 body that defeated secsgem-py and unsolicited S6F11/S5F1 observation |
|
|
| `tshark_validate.sh` | Wireshark's built-in HSMS dissector | 69 captured frames dissected with no malformed-packet warnings |
|
|
| `spool_persistence_test.py` | secsgem-py + a docker-restart loop | Restart-survives-spool integrity |
|
|
| ⚙️ `../tests/test_e5_kat.cpp` | SEMI E5 §9 encoding rules | 196 known-answer byte assertions across every format code |
|
|
| ⚙️ `../apps/fuzz_secs2_decode.cpp` + `fuzz_sml_parse.cpp` | libFuzzer + ASan + UBSan | ~70 000 + ~285 000 random inputs per minute, 0 crashes |
|
|
|
|
The ⚙️ entries aren't in `interop/` directly because they don't
|
|
involve a network peer — they're either pure codec round-trips
|
|
(KAT) or coverage-guided fuzzing. Listed here so the full external
|
|
proof inventory lives in one place.
|
|
|
|
## Running each validator
|
|
|
|
### secsgem-py — secsgem-py active host → C++ server
|
|
|
|
```sh
|
|
docker compose up -d server
|
|
docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py \
|
|
--host server --port 5000 --session-id 0
|
|
```
|
|
|
|
### secsgem-py — C++ host → secsgem-py equipment
|
|
|
|
```sh
|
|
docker compose up -d equipment_py
|
|
docker compose run --rm builder /app/build/secs_interop_probe \
|
|
--host equipment_py --port 5000 --device 0
|
|
```
|
|
|
|
### secsgem-py — raw GEM 300 frames
|
|
|
|
```sh
|
|
docker compose up -d server
|
|
docker compose run --rm interop python3 /app/interop/raw_gem300_harness.py \
|
|
--host server --port 5000 --session-id 0
|
|
```
|
|
|
|
### secs4j — independent Java host → C++ server
|
|
|
|
```sh
|
|
bash interop/secs4j_validate.sh
|
|
```
|
|
|
|
Builds an `eclipse-temurin:21-jdk` sidecar with secs4java8 cloned +
|
|
compiled at image build, then drives 55 checks against
|
|
`compose up server`. See `secs4j/Secs4jHostHarness.java` for the
|
|
list and `secs4j/Dockerfile` for the build.
|
|
|
|
### tshark — Wireshark HSMS dissector
|
|
|
|
```sh
|
|
docker compose run --rm builder bash /app/interop/tshark_validate.sh
|
|
```
|
|
|
|
Captures a pcap of the demo flow, runs `tshark -V` with the HSMS
|
|
dissector forced for the test port, asserts no malformed packets +
|
|
that all expected control/data frames parse.
|
|
|
|
### spool persistence — restart-survives test
|
|
|
|
```sh
|
|
bash interop/spool_persistence_test.py
|
|
```
|
|
|
|
Drops the host link mid-flight, kills the server, restarts it, and
|
|
asserts the spooled S5F1 / S6F11 frames drain to the host on
|
|
reconnect.
|
|
|
|
## What these harnesses caught
|
|
|
|
Real bugs surfaced during interop development (now fixed):
|
|
|
|
1. **Strict U-width parsing rejected U1-encoded identifiers.** SEMI
|
|
E5 declares DATAID, RPTID, VID, CEID, ALID, EXID etc. as
|
|
`U1 | U2 | U4 | U8`; secsgem-py picks the smallest width that
|
|
fits. Our scalar accessors were strict. Now lenient with
|
|
range-checked downcasts (`messages_helpers.hpp::any_unsigned_first`).
|
|
|
|
2. **PPBODY rejected when sent as ASCII.** SEMI allows PPBODY to be
|
|
`ASCII | Binary | List`; secsgem-py defaults to ASCII. Added the
|
|
`BINARY_OR_ASCII` codegen type and the `as_text_or_binary`
|
|
accessor.
|
|
|
|
3. **Missing S1F23 / S1F24 (Collection Event Namelist).** Added the
|
|
wire schema, the `vids_for(ceid)` accessor, and the dispatch
|
|
handler.
|
|
|
|
4. **Missing S10F3 handler (host→equipment Terminal Display).** Our
|
|
server only registered S10F1; per SEMI E5 §13 those are opposite
|
|
directions. Added the missing dispatch.
|
|
|
|
5. **TSan use-after-free in `act_exception_complete`** (test code,
|
|
not library): held a pointer across `fire_internal(RecoveryComplete)`
|
|
which deletes the entry. Found by the ThreadSanitizer lane on
|
|
first run.
|
|
|
|
The C++ test suite stayed green through every one of these fixes —
|
|
the changes were purely permissive widenings or additive features,
|
|
no existing behaviour broke.
|
|
|
|
## When to add a new validator
|
|
|
|
A new third-party SECS implementation, or a new dissector, or a new
|
|
fuzzer target — anything that exercises our wire surface from an
|
|
angle the existing five don't cover — is worth adding. The pattern
|
|
is consistent:
|
|
|
|
1. New script / harness lives here (or a sidecar Docker context for
|
|
non-Python validators).
|
|
2. Wired into `.gitea/workflows/ci.yml` as a separate job.
|
|
3. Listed in this README's table + in `../VERIFICATION.md`.
|
|
4. Surfaced in [`../PROOFS.md`](../PROOFS.md) if it adds a
|
|
meaningful new dimension.
|
|
|
|
Bug reports from a new validator → file at
|
|
`raphael@maenle.net` with the wire trace, the validator's output,
|
|
and the equipment YAML so we can reproduce.
|