raphael
dae6bfd747
Tone pass across the non-tutorial markdown — README, PROOFS,
ARCHITECTURE, BENCHMARKS, COMPLIANCE, FAQ, MES_INTEROP, SECURITY,
and interop/README. Three patterns came out:
- Bug-history war stories ("Past interop sweeps surfaced…",
"What these harnesses caught: 1. Strict U-width parsing…").
- Chat-with-reader framing ("Don't skip TLS unless…", "Treat as a
punch list", "If you're running in a pod…", "Misconfiguration
incidents drop dramatically").
- Self-referential narration ("we ship", "our codec", "the
codebase's most-tested layer", "three orders of magnitude above
fab load", "the gift that keeps giving").
README also drops the standalone ThreadSanitizer subsection under
Build details (now a single line under the new Testing section).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
67 lines
6.5 KiB
Markdown
67 lines
6.5 KiB
Markdown
# Proofs
|
|
|
|
Eight commands. If they all exit zero on a fresh clone, the codebase
|
|
implements what [COMPLIANCE.md](COMPLIANCE.md) claims.
|
|
|
|
| # | Command | What it proves |
|
|
|---|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
|
|
| 1 | `docker compose run --rm tests` | **445 test cases / 2 753 assertions** pass: every store, FSM, codec, parser, persistence path |
|
|
| 2 | `docker compose run --rm builder /app/build/secs_conformance --host server --port 5000` | **47 wire-level conformance checks** PASS against a live passive equipment |
|
|
| 3 | `docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py --host server` | **31 interop checks** PASS against secsgem-py 0.3.0 (the Python reference impl) |
|
|
| 4 | `SECSGEM_ROBUSTNESS_SOAK=1 docker compose run --rm builder /app/build/secsgem_tests -tc='*soak*'` | **100 000 random tool operations** execute with all invariants and persistence round-trips holding |
|
|
| 5 | `docker compose run --rm builder /app/build/secs_server --validate-config --config /app/data/equipment.yaml --state-table /app/data/control_state.yaml --pj-state-table /app/data/process_job_state.yaml --cj-state-table /app/data/control_job_state.yaml` | Every shipped YAML config passes structural + referential validation |
|
|
| 6 | `docker compose run --rm builder bash /app/interop/tshark_validate.sh` | **69 HSMS frames** dissected by Wireshark's HSMS dissector (independent third codec) with no malformed packets |
|
|
| 7 | `bash interop/secs4j_validate.sh` | **55 cross-validation checks** PASS against [secs4java8](https://github.com/kenta-shimizu/secs4java8) (independent Java implementation), covering S1/S2/S3/S5/S6/S7/S10/S14/S16, the full E40 PJ body, dynamic event reports + unsolicited S6F11 / S5F1 observation, alarm management, spool, PP management, terminal services, limits, trace, E39, and the GEM 300 streams secsgem-py couldn't easily drive |
|
|
| 8 | `cmake -B build-fuzz -DSECSGEM_FUZZ=ON && build-fuzz/fuzz_secs2_decode -max_total_time=60` | **200 000+ random inputs** through `secs2::decode`, **1.4 M+** through `try_parse_sml` per 60 s lane, ASan + UBSan + libFuzzer coverage, **0 crashes** |
|
|
|
|
CI ([Gitea Actions](.gitea/workflows/ci.yml)) runs a Release build +
|
|
full suite and a separate `-fsanitize=thread` lane on every push to
|
|
`main`. All 445 cases / 2 753 assertions pass under TSan clean.
|
|
|
|
## Per-standard test coverage
|
|
|
|
Every claimed standard has dedicated tests. Counts are
|
|
`grep -c TEST_CASE`; cross-cutting tests (e.g. `test_robustness_fuzz`,
|
|
`test_gem300_scenario`) exercise multiple standards in concert.
|
|
|
|
| Standard | Test files | Cases |
|
|
|-----------------------------------|-------------------------------------------------------------------------------------------|------:|
|
|
| **E5** — SECS-II encoding | `test_secs2`, `test_sml`, `test_messages`, `test_e5_kat`, `test_identifier_wildcards`, `test_fuzz` | 139 |
|
|
| **E5 §13** — exceptions | `test_exceptions`, `test_exception_persistence` | 16 |
|
|
| **E4** — SECS-I transport | `test_secsi`, `test_secsi_timers`, `test_secsi_tcp` | 27 |
|
|
| **E37** — HSMS (SS + GS) | `test_hsms`, `test_hsms_connection`, `test_hsms_timers`, `test_hsms_s9`, `test_hsms_gs`, `test_hsms_gs_integration`, `test_s9_fallback`, `test_concurrency` | 34 |
|
|
| **E30** — GEM core | `test_control_state`, `test_communication_state`, `test_host_handler`, `test_data_model`, `test_loader`, `test_config_validate` | 71 |
|
|
| **E40** — process jobs | `test_process_jobs` | 21 |
|
|
| **E94** — control jobs | `test_control_jobs` | 9 |
|
|
| **E42** — formatted PP | `test_e42_formatted_pp` | 6 |
|
|
| **E87** — carriers + load ports | `test_carriers`, `test_carrier_state`, `test_carrier_persistence`, `test_e87_wire_scenarios` | 27 |
|
|
| **E90** — substrate tracking | `test_substrates`, `test_substrate_persistence` | 21 |
|
|
| **E116** — EPT | `test_ept` | 7 |
|
|
| **E120 / E39** — common equip / object service | `test_cem_objects` | 3 |
|
|
| **E157** — module process tracking | `test_modules` | 5 |
|
|
| **E84** — parallel I/O + timers | `test_e84`, `test_e84_ports`, `test_e84_timers`, `test_e84_asio_timers` | 27 |
|
|
| Persistence + cross-cutting | `test_job_persistence`, `test_persistence_upgrade`, `test_wire_ceid_emission`, `test_gem300_scenario`, `test_live_gem300`, `test_thread_safety`, `test_metrics_prometheus`, `test_robustness_fuzz` | 32 |
|
|
| **Total** | | **445** |
|
|
|
|
`docker compose run --rm builder /app/build/secsgem_tests --list-test-cases | wc -l`
|
|
currently reports 445.
|
|
|
|
## Categories of evidence
|
|
|
|
The eight commands split into four kinds:
|
|
|
|
- **Internal** (#1, #2, #4, #5) — unit suite, conformance harness,
|
|
soak property test, config validator. Necessary but not
|
|
independent.
|
|
- **External, second implementation** (#3, #7) — secsgem-py (Python)
|
|
and secs4java8 (Java). Two independent SECS implementations agree
|
|
on every frame.
|
|
- **External, third codec** (#6) — Wireshark's HSMS dissector,
|
|
written by network-protocol authors with no shared code. Catches
|
|
framing bugs the implementations might share.
|
|
- **External, structural search** (#8) — libFuzzer + ASan + UBSan
|
|
over the decoder and SML parser surface.
|
|
|
|
[VERIFICATION.md](VERIFICATION.md) covers each external validator in
|
|
detail.
|