raphael/secs-gem
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b01dedfaa5d6796e20b39c477b7c642ad02bc012
secs-gem/PROOFS.md
T
raphael b031f057af docs: customer-ready sweep + README restructure + tshark CI fix 
Audit pass over the public-facing surface so a customer can read it
end-to-end without tripping on stale numbers or self-contradictions.

README + docs accuracy:
- Test counts 426 → 445, assertions 2 557 → 2 753 (verified via
  doctest run); E5 row was missing test_e5_kat (19 cases)
- Interop checks 24 → 31, COMPLIANCE.md message count 149 → 164,
  COMPLIANCE.md "291 cases / 1515 assertions" → 445 / 2 753
- README "60+ test IDs" for MES_INTEROP → actual 59
- PVD example counts: 32 SVIDs/17 CEIDs → 29/21, "~40 handlers
  in ~200 lines" → 51 in ~460, "~700 lines" → ~1,100; main.cpp
  header table-of-contents resynced with the actual 7 sections

Out-of-scope honesty (COMPLIANCE.md §8 + FAQ.md):
- Removed HSMS-GS (was both ✅ implemented in §1 and "out of scope"
  in §8; INTEGRATION.md §7 documents using it)
- Removed multi-block SECS-I (split_message/assemble_message exist
  with 4 dedicated tests)
- Added serial-port wiring as the genuine open ⬜ item — FSM is
  tested end-to-end over TCP; only the asio serial_port glue is
  deferred
- COMPLIANCE.md intro now lists E42 and notes "E37 (SS + GS)"

README restructure:
- Moved the 8-command proof table and per-standard test-coverage
  table to a new PROOFS.md (72 lines)
- README now leads with what / Quick start / Documentation map,
  then a one-paragraph "How it's proved" linking to PROOFS.md
- Updated cross-refs in FAQ.md, GLOSSARY.md, VERIFICATION.md, and
  interop/README.md to point at PROOFS.md

CI fix — tshark-dissector job:
- interop/tshark_validate.sh hardcoded /app/build/secs_server etc.
  which only works inside the docker image.  Now derives ROOT from
  the script's own location and accepts BUILD/SERVER/CLIENT/DATA
  env overrides, so CI can run it from the workspace dir
- Verified still passes in docker (69 frames, 0 malformed)

.gitignore:
- Added build-fuzz/ and build-tsan/ (were showing as untracked)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-09 18:59:17 +02:00

73 lines
6.9 KiB
Markdown
Raw Blame History

# Proof of feature-completeness
"Feature-complete" is a claim that the code must prove, not the
README. These eight commands are the proof. If they all exit zero
on a fresh clone, the codebase implements what
[COMPLIANCE.md](COMPLIANCE.md) claims.
| # | Command | What it proves |
|---|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
| 1 | `docker compose run --rm tests` | **445 test cases / 2 753 assertions** pass: every store, FSM, codec, parser, persistence path |
| 2 | `docker compose run --rm builder /app/build/secs_conformance --host server --port 5000` | **47 wire-level conformance checks** PASS against a live passive equipment |
| 3 | `docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py --host server` | **31 interop checks** PASS against secsgem-py 0.3.0 (the Python reference impl) |
| 4 | `SECSGEM_ROBUSTNESS_SOAK=1 docker compose run --rm builder /app/build/secsgem_tests -tc='*soak*'` | **100 000 random tool operations** execute with all invariants and persistence round-trips holding |
| 5 | `docker compose run --rm builder /app/build/secs_server --validate-config --config /app/data/equipment.yaml --state-table /app/data/control_state.yaml --pj-state-table /app/data/process_job_state.yaml --cj-state-table /app/data/control_job_state.yaml` | Every shipped YAML config passes structural + referential validation |
| 6 | `docker compose run --rm builder bash /app/interop/tshark_validate.sh` | **69 HSMS frames** dissected by Wireshark's HSMS dissector (independent third codec) with no malformed packets |
| 7 | `bash interop/secs4j_validate.sh` | **55 cross-validation checks** PASS against [secs4java8](https://github.com/kenta-shimizu/secs4java8) (independent Java implementation), covering S1/S2/S3/S5/S6/S7/S10/S14/S16, the full E40 PJ body, dynamic event reports + unsolicited S6F11 / S5F1 observation, alarm management, spool, PP management, terminal services, limits, trace, E39, and the GEM 300 streams secsgem-py couldn't easily drive |
| 8 | `cmake -B build-fuzz -DSECSGEM_FUZZ=ON && build-fuzz/fuzz_secs2_decode -max_total_time=60` | **200 000+ random inputs** through `secs2::decode`, **1.4 M+** through `try_parse_sml` per 60 s lane, ASan + UBSan + libFuzzer coverage, **0 crashes** |
Plus, on every push to `main`, [Gitea Actions](.gitea/workflows/ci.yml)
runs both a **Release build + full test suite** and a separate
**ThreadSanitizer lane** that builds with `-fsanitize=thread` and
fails on any race. All 445 cases / 2 753 assertions pass under TSan
clean.
## Per-standard test coverage
Every claimed standard has dedicated tests. Counts are
`grep -c TEST_CASE`; cross-cutting tests (e.g. `test_robustness_fuzz`,
`test_gem300_scenario`) exercise multiple standards in concert.
| Standard | Test files | Cases |
|-----------------------------------|-------------------------------------------------------------------------------------------|------:|
| **E5** — SECS-II encoding | `test_secs2`, `test_sml`, `test_messages`, `test_e5_kat`, `test_identifier_wildcards`, `test_fuzz` | 139 |
| **E5 §13** — exceptions | `test_exceptions`, `test_exception_persistence` | 16 |
| **E4** — SECS-I transport | `test_secsi`, `test_secsi_timers`, `test_secsi_tcp` | 27 |
| **E37** — HSMS (SS + GS) | `test_hsms`, `test_hsms_connection`, `test_hsms_timers`, `test_hsms_s9`, `test_hsms_gs`, `test_hsms_gs_integration`, `test_s9_fallback`, `test_concurrency` | 34 |
| **E30** — GEM core | `test_control_state`, `test_communication_state`, `test_host_handler`, `test_data_model`, `test_loader`, `test_config_validate` | 71 |
| **E40** — process jobs | `test_process_jobs` | 21 |
| **E94** — control jobs | `test_control_jobs` | 9 |
| **E42** — formatted PP | `test_e42_formatted_pp` | 6 |
| **E87** — carriers + load ports | `test_carriers`, `test_carrier_state`, `test_carrier_persistence`, `test_e87_wire_scenarios` | 27 |
| **E90** — substrate tracking | `test_substrates`, `test_substrate_persistence` | 21 |
| **E116** — EPT | `test_ept` | 7 |
| **E120 / E39** — common equip / object service | `test_cem_objects` | 3 |
| **E157** — module process tracking | `test_modules` | 5 |
| **E84** — parallel I/O + timers | `test_e84`, `test_e84_ports`, `test_e84_timers`, `test_e84_asio_timers` | 27 |
| Persistence + cross-cutting | `test_job_persistence`, `test_persistence_upgrade`, `test_wire_ceid_emission`, `test_gem300_scenario`, `test_live_gem300`, `test_thread_safety`, `test_metrics_prometheus`, `test_robustness_fuzz` | 32 |
| **Total** | | **445** |
A single command to see this live: `docker compose run --rm builder
/app/build/secsgem_tests --list-test-cases | wc -l` (currently 445).
## What each proof actually demonstrates
The eight commands above split into four kinds of evidence:
- **Internal** (#1, #2, #4, #5) — our code testing our code: unit
suite, conformance harness, soak property test, config validator.
Necessary but not independent.
- **External, second implementation** (#3, #7) — round-trip against
secsgem-py 0.3.0 (Python) and secs4java8 (Java). Two independent
SECS implementations must agree with us on every frame.
- **External, third codec** (#6) — Wireshark's HSMS dissector,
written by network-protocol authors who don't share code with
either of us. Catches framing bugs the implementations might both
share.
- **External, structural search** (#8) — libFuzzer + ASan + UBSan
exploring the decoder and SML parser surface for crashes, memory
errors, and UB.
See [VERIFICATION.md](VERIFICATION.md) for the full test plan and
the rationale for each external validator.
Reference in New Issue View Git Blame Copy Permalink