raphael
b031f057af
Audit pass over the public-facing surface so a customer can read it end-to-end without tripping on stale numbers or self-contradictions. README + docs accuracy: - Test counts 426 → 445, assertions 2 557 → 2 753 (verified via doctest run); E5 row was missing test_e5_kat (19 cases) - Interop checks 24 → 31, COMPLIANCE.md message count 149 → 164, COMPLIANCE.md "291 cases / 1515 assertions" → 445 / 2 753 - README "60+ test IDs" for MES_INTEROP → actual 59 - PVD example counts: 32 SVIDs/17 CEIDs → 29/21, "~40 handlers in ~200 lines" → 51 in ~460, "~700 lines" → ~1,100; main.cpp header table-of-contents resynced with the actual 7 sections Out-of-scope honesty (COMPLIANCE.md §8 + FAQ.md): - Removed HSMS-GS (was both ✅ implemented in §1 and "out of scope" in §8; INTEGRATION.md §7 documents using it) - Removed multi-block SECS-I (split_message/assemble_message exist with 4 dedicated tests) - Added serial-port wiring as the genuine open ⬜ item — FSM is tested end-to-end over TCP; only the asio serial_port glue is deferred - COMPLIANCE.md intro now lists E42 and notes "E37 (SS + GS)" README restructure: - Moved the 8-command proof table and per-standard test-coverage table to a new PROOFS.md (72 lines) - README now leads with what / Quick start / Documentation map, then a one-paragraph "How it's proved" linking to PROOFS.md - Updated cross-refs in FAQ.md, GLOSSARY.md, VERIFICATION.md, and interop/README.md to point at PROOFS.md CI fix — tshark-dissector job: - interop/tshark_validate.sh hardcoded /app/build/secs_server etc. which only works inside the docker image. Now derives ROOT from the script's own location and accepts BUILD/SERVER/CLIENT/DATA env overrides, so CI can run it from the workspace dir - Verified still passes in docker (69 frames, 0 malformed) .gitignore: - Added build-fuzz/ and build-tsan/ (were showing as untracked) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
6.9 KiB
6.9 KiB
Proof of feature-completeness
"Feature-complete" is a claim that the code must prove, not the README. These eight commands are the proof. If they all exit zero on a fresh clone, the codebase implements what COMPLIANCE.md claims.
| # | Command | What it proves |
|---|---|---|
| 1 | docker compose run --rm tests |
445 test cases / 2 753 assertions pass: every store, FSM, codec, parser, persistence path |
| 2 | docker compose run --rm builder /app/build/secs_conformance --host server --port 5000 |
47 wire-level conformance checks PASS against a live passive equipment |
| 3 | docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py --host server |
31 interop checks PASS against secsgem-py 0.3.0 (the Python reference impl) |
| 4 | SECSGEM_ROBUSTNESS_SOAK=1 docker compose run --rm builder /app/build/secsgem_tests -tc='*soak*' |
100 000 random tool operations execute with all invariants and persistence round-trips holding |
| 5 | docker compose run --rm builder /app/build/secs_server --validate-config --config /app/data/equipment.yaml --state-table /app/data/control_state.yaml --pj-state-table /app/data/process_job_state.yaml --cj-state-table /app/data/control_job_state.yaml |
Every shipped YAML config passes structural + referential validation |
| 6 | docker compose run --rm builder bash /app/interop/tshark_validate.sh |
69 HSMS frames dissected by Wireshark's HSMS dissector (independent third codec) with no malformed packets |
| 7 | bash interop/secs4j_validate.sh |
55 cross-validation checks PASS against secs4java8 (independent Java implementation), covering S1/S2/S3/S5/S6/S7/S10/S14/S16, the full E40 PJ body, dynamic event reports + unsolicited S6F11 / S5F1 observation, alarm management, spool, PP management, terminal services, limits, trace, E39, and the GEM 300 streams secsgem-py couldn't easily drive |
| 8 | cmake -B build-fuzz -DSECSGEM_FUZZ=ON && build-fuzz/fuzz_secs2_decode -max_total_time=60 |
200 000+ random inputs through secs2::decode, 1.4 M+ through try_parse_sml per 60 s lane, ASan + UBSan + libFuzzer coverage, 0 crashes |
Plus, on every push to main, Gitea Actions
runs both a Release build + full test suite and a separate
ThreadSanitizer lane that builds with -fsanitize=thread and
fails on any race. All 445 cases / 2 753 assertions pass under TSan
clean.
Per-standard test coverage
Every claimed standard has dedicated tests. Counts are
grep -c TEST_CASE; cross-cutting tests (e.g. test_robustness_fuzz,
test_gem300_scenario) exercise multiple standards in concert.
| Standard | Test files | Cases |
|---|---|---|
| E5 — SECS-II encoding | test_secs2, test_sml, test_messages, test_e5_kat, test_identifier_wildcards, test_fuzz |
139 |
| E5 §13 — exceptions | test_exceptions, test_exception_persistence |
16 |
| E4 — SECS-I transport | test_secsi, test_secsi_timers, test_secsi_tcp |
27 |
| E37 — HSMS (SS + GS) | test_hsms, test_hsms_connection, test_hsms_timers, test_hsms_s9, test_hsms_gs, test_hsms_gs_integration, test_s9_fallback, test_concurrency |
34 |
| E30 — GEM core | test_control_state, test_communication_state, test_host_handler, test_data_model, test_loader, test_config_validate |
71 |
| E40 — process jobs | test_process_jobs |
21 |
| E94 — control jobs | test_control_jobs |
9 |
| E42 — formatted PP | test_e42_formatted_pp |
6 |
| E87 — carriers + load ports | test_carriers, test_carrier_state, test_carrier_persistence, test_e87_wire_scenarios |
27 |
| E90 — substrate tracking | test_substrates, test_substrate_persistence |
21 |
| E116 — EPT | test_ept |
7 |
| E120 / E39 — common equip / object service | test_cem_objects |
3 |
| E157 — module process tracking | test_modules |
5 |
| E84 — parallel I/O + timers | test_e84, test_e84_ports, test_e84_timers, test_e84_asio_timers |
27 |
| Persistence + cross-cutting | test_job_persistence, test_persistence_upgrade, test_wire_ceid_emission, test_gem300_scenario, test_live_gem300, test_thread_safety, test_metrics_prometheus, test_robustness_fuzz |
32 |
| Total | 445 |
A single command to see this live: docker compose run --rm builder /app/build/secsgem_tests --list-test-cases | wc -l (currently 445).
What each proof actually demonstrates
The eight commands above split into four kinds of evidence:
- Internal (#1, #2, #4, #5) — our code testing our code: unit suite, conformance harness, soak property test, config validator. Necessary but not independent.
- External, second implementation (#3, #7) — round-trip against secsgem-py 0.3.0 (Python) and secs4java8 (Java). Two independent SECS implementations must agree with us on every frame.
- External, third codec (#6) — Wireshark's HSMS dissector, written by network-protocol authors who don't share code with either of us. Catches framing bugs the implementations might both share.
- External, structural search (#8) — libFuzzer + ASan + UBSan exploring the decoder and SML parser surface for crashes, memory errors, and UB.
See VERIFICATION.md for the full test plan and the rationale for each external validator.