raphael
d63c92166d
Previously written as a forward-looking plan ("Plan: (1) KAT → (2)
tshark → (3) secs4j → (4) libFuzzer", "Effort: ~3 hours", "Survey
step (do this first)"). All four validators have shipped —
test_e5_kat.cpp, interop/secs4j/Secs4jHostHarness.java,
interop/tshark_validate.sh, apps/fuzz_*.cpp. Rewritten as
documentation of what's there: file paths, CI job names, actual
result numbers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
6.0 KiB
External verification
The unit suite is internal regression coverage. Four external
validators run alongside it: SEMI E5 known-answer tests, Wireshark's
HSMS dissector on a captured pcap, secs4java8 cross-validation, and
libFuzzer over the decoder + SML parser. Each runs in CI on every
push to main.
| Channel | Source of independence |
|---|---|
| 445 unit/integration tests | Internal |
| 47 conformance harness checks | Internal |
| SEMI E5 KAT | External — standards body's encoding rules |
| Wireshark HSMS dissector | External — independent network-protocol authors |
| secs4java8 interop (55) | External — second independent SECS implementation |
| secsgem-py interop (31) | External — Python reference impl |
| libFuzzer (ASan + UBSan) | External — coverage-guided structural search |
| 100 k random tool ops | Internal — property test |
| YAML validation | Internal |
1. SEMI E5 known-answer tests
tests/test_e5_kat.cpp pins the encoder and decoder to the byte
patterns SEMI E5 requires. Each fixture is a (canonical_hex, expected_item) pair; encode(expected_item) must produce
canonical_hex and decode(canonical_hex) must round-trip back.
| Format | Code | Fixtures |
|---|---|---|
| List | 0x00 |
empty, nested, mixed-type |
| Binary | 0x20 |
empty, 1-byte, 256-byte (2-byte length), 65 536-byte (3-byte length) |
| Boolean | 0x24 |
TRUE, FALSE, multi-element |
| ASCII | 0x40 |
empty, single char, 255-byte, 256-byte |
| JIS-8 | 0x44 |
empty, non-ASCII bytes |
| U1/U2/U4/U8 | 0xA4 / 0xA8 / 0xAC / 0xA0 |
0, mid, max, multi-element |
| I1/I2/I4/I8 | 0x64 / 0x68 / 0x6C / 0x60 |
0, ±1, INT_MIN, INT_MAX |
| F4/F8 | 0x84 / 0x80 |
0.0, ±1.0, NaN, ±Inf, subnormal |
Length-byte counts of 1, 2, and 3 are exercised explicitly.
Caveat on authority. SEMI does not publish official test vectors for E5 (unlike NIST for crypto). The bytes are derived from the encoding rules in the spec, so KAT alone proves the codec is internally consistent with that reading. Independent corroboration of every format code arrives through secs4java8 and Wireshark, both with their own decoders.
2. Wireshark / tshark HSMS dissector
interop/tshark_validate.sh starts the C++ server, captures a pcap
of the two-container demo with tcpdump, then dissects every frame
with Wireshark's HSMS dissector. The script fails if tshark reports
any Malformed Packet, Dissector bug, or Unknown PType/SType, and
asserts that Select.req, Linktest.req, S1F13, and S6F11 each
appear at least once.
Wireshark's dissector is written by network-protocol authors with no shared code with this repository or with secsgem-py. Clean dissection of the pcap is an independent check on HSMS framing.
Coverage. HSMS framing (4-byte length prefix + 10-byte header) and control-message shapes (Select / Deselect / Linktest / Separate / Reject). Wireshark renders SECS-II bodies as hex blobs and doesn't decode S/F semantics — KAT and secs4j cover that.
Result. 69 HSMS frames per run, 0 malformed. Wired into CI as
the tshark-dissector job.
3. secs4java8 cross-validation
interop/secs4j/ is a Docker harness wrapping
secs4java8 (Apache 2.0).
Secs4jHostHarness.java connects as an active HSMS host to the
passive C++ server and drives 55 cross-validation checks across S1,
S2, S3, S5, S6, S7, S10, S14, and S16.
The harness covers the full-body GEM 300 shapes secsgem-py cannot
easily drive: E40 process-job creation bodies, E94 control-job
create, E87 carrier actions with slot maps, S2F49 enhanced commands,
S5F13–F18 exception recovery, and S12 wafer maps.
interop/secs4j_validate.sh orchestrates the harness against the
server image; wired into CI as the secs4j-interop job.
secsgem-py (Python) and secs4java8 (Java) are independent implementations of the same standards. Agreement on every frame across both peers is wire correctness from two independent angles.
4. libFuzzer over codec + SML parser
apps/fuzz_secs2_decode.cpp and apps/fuzz_sml_parse.cpp are
libFuzzer entry points built with -DSECSGEM_FUZZ=ON
(-fsanitize=fuzzer,address,undefined). The CI lane runs each for
60 seconds — roughly 200 000 inputs through secs2::decode and 1.4 M
through try_parse_sml.
The corpus is seeded from the SECS-II hex fixtures shared with the rest of the suite, so the fuzzer starts from a known-good baseline and mutates outward.
Coverage. Crashes and undefined behaviour on adversarial input — length-byte overflow, malformed format codes, recursive list bombs, truncated frames. A decoder that returns the wrong value silently is invisible to libFuzzer; KAT and the interop harnesses cover that.
Result. 0 crashes, 0 ASan reports, 0 UBSan flags across both targets.
What this does NOT replace
- A GEM RTS run. Still required for certification; still costs money and needs hardware. See MES_INTEROP.md §10.
- Per-MES interop sweeps against the customer's actual MES (Camstar, FactoryWorks, etc.). Still required for any production deployment. See MES_INTEROP.md.
- Real-fab wire traces. No public corpus exists; fabs treat their captures as IP.
Those remain customer-side work.