b1772cfefd11824a7b3b1c8a5244e49b7e53cdfb
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 raphael
|
b30443089f |
feat(clients)+test(interop): C++ client + Java validation of the daemon (B7)
tests / build-and-test (push) Successful in 2m54s
tests / thread-sanitizer (push) Successful in 3m48s
tests / tshark-dissector (push) Successful in 2m24s
tests / secs4j-interop (push) Successful in 1m44s
tests / python-interop (push) Successful in 3m10s
tests / libfuzzer (push) Successful in 3m38s
B7 — the daemon's HSMS face under the Java reference: Dockerfile.server now
bakes secs_gemd alongside secs_server (grpc deps in both stages), and
secs4j_validate.sh gains TARGET=gemd to point the 55-check secs4java8 suite
at the daemon instead. Result: 55/55 green. With secsgem-py already
validating both faces, byte-identical GEM between secs_server and secs_gemd
is now proven by both reference implementations, not inferred from shared
code. CI runs the daemon target as an extra step (image layers shared).
Second client — clients/cpp: a header-only C++ twin of the Python client
over the same proto. eq.set("ChamberPressure", 2.5) with bare literals
(integral/floating dispatch avoids variant ambiguity), get/fire/alarm/
clear, control_state/request_control_state/health, on("START", fn) +
listen()/listen_async()/stop() with auto-CompleteCommand, SecsGemError
carrying the daemon's message. cpp_mini_tool (~30 lines) mirrors the
Python mini_tool. Tested end-to-end over real loopback TCP against the
service inside secs_gemd_tests — now 4 cases / 141 assertions — including
set/get round-trips, error text, alarm-by-name into the model, health,
and the full HCACK-4 command loop with parameters.
(Build note: two grpc-heavy TUs at -O3 OOM even at -j2 on Docker Desktop;
built -j1. Known environment limitation, roadmap-documented.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
raphael
|
cf230d4119 |
chore(phase0): name validation, golden frames, daemon into library tree, TSan daemon lane
Item 8a — ConfigValidator warns on non-identifier variable/event/alarm/ command names ([A-Za-z_][A-Za-z0-9_]*): language bindings expose names as kwargs/attributes, so 'Chamber Pressure' would be unusable in the planned Python client. Warning not error — the wire doesn't care. Tested (4 warning sites + good-name negative). Item 4 tail — golden frames for S5F1 (Binary ALCD / U4 ALID / ASCII ALTX) and a composed S6F11 (the production-critical report shape), bytes hand- computed from E5 encoding rules: external pins on message composition. Item 7 — equipment_service.hpp moved to include/secsgem/daemon/ (apps/ include-path hack removed) and a TSan daemon lane added locally + in CI. tools/tsan.supp suppresses races whose accesses sit entirely inside the UNinstrumented system libgrpc/libgpr/libabsl (epoll wakeups, absl Mutex GraphCycles bookkeeping); our frames stay fully checked. The lane earned its keep on first run: it caught a REAL threading-contract violation — a daemon test reading model stores from the test thread while the io thread serviced posted writes — fixed to use read_sync, exactly per the documented contract. Now TSan-clean under halt_on_error=1 in the full production threading shape. Suites: core 470/3068, daemon Release+TSan 125/125 each. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
42044e92e2 |
ci(interop): one-command external-validation suite + CI lanes for the daemon
tests / build-and-test (push) Successful in 2m42s
tests / thread-sanitizer (push) Successful in 2m50s
tests / tshark-dissector (push) Successful in 2m24s
tests / secs4j-interop (push) Successful in 37s
tests / python-interop (push) Successful in 2m56s
tests / libfuzzer (push) Successful in 3m44s
tools/run_interop.sh runs ALL nine validation steps with a PASS/FAIL summary: build, unit (464), daemon-unit (41), secsgem-py host vs server (31 checks), secs_conformance (47), gRPC+secsgem-py daemon bridge, spool persistence across restart, tshark HSMS dissector, secs4java8 (55 checks). Verified green end-to-end. The unit suite is partly self-referential (our parsers validate our builders); these external validators are the real oracle — now they run with one command instead of by hand. Two bugs found by running it: unbounded ninja at -O3 OOM-kills cc1plus in memory-constrained Docker VMs (build with -j 2) and bash-3.2 lacks negative array subscripts. CI: grpc deps added to the build job so secs_gemd + secs_gemd_tests build and RUN in CI (previously the daemon silently dropped out — now fails loudly if missing), plus a python-interop lane running py-host/conformance/daemon harnesses against localhost in one container (no docker-in-docker). Service hardening while in there: reject proto Values with no kind set at the RPC edge (previously silently became ASCII ""), TODO markers for list element formats and daemon graceful shutdown. New tests: unset-Value guard + a property test iterating ALL configured variables via gRPC asserting each keeps its declared SECS-II format (daemon tests 16 -> 41 assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
fc3422a4a9 |
docs: move root .md files into docs/ + update every reference
Picks up the file renames that landed alongside the previous commit and fixes everything that pointed at the old root locations: - README.md doc-map updated: every entry now points at docs/X.md, with a new "docs/" lead entry pointing at the guided-tour index. - README inline cross-refs (ARCHITECTURE / INTEGRATION / SECURITY / BENCHMARKS / MES_INTEROP / PROOFS) repointed to docs/. - README "Interop" section rewritten — used to mention only secsgem-py; now covers all four external validators (secsgem-py 31 / secs4java8 55 / tshark 69 frames / libFuzzer 200 k+ runs) with a one-line summary each, plus pointers to interop/README.md and docs/VERIFICATION.md. - README "Deferred follow-ups" cleaned: dropped the explanatory "Listed here so reviewers don't go looking for them in COMPLIANCE.md and find an 'out of scope' entry that sounds defensive" sentence — the section header speaks for itself. - docs/00_index.md "Where the rest of the docs live" table: dropped every `../` prefix since the docs are now siblings. - docs/01_what_is_secs_gem.md PROOFS reference updated to sibling. - docs/02_the_cast.md INTEGRATION + MES_INTEROP refs updated to siblings; dropped the stale "at the repo root" wording. - interop/README.md: VERIFICATION + PROOFS refs updated to ../docs/X.md; stale "~24 + 4 checks" updated to 31 (matches PROOFS.md and README). - examples/pvd_tool/README.md: every doc cross-ref now points at ../../docs/X.md. - Source / data / CI comments mentioning doc names (e.g. "INTEGRATION.md §3", "COMPLIANCE.md gap") rewritten to "docs/INTEGRATION.md §3" etc. — affects 9 files across include/, apps/, tests/, data/, examples/, .gitea/workflows/. Verified: full build under docker passes, 445/445 test cases pass, 2 753/2 753 assertions pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
bc54de7711 |
ci: secs4j-interop bootstrap resilient to runner image variant
CI log showed: Run export DEBIAN_FRONTEND=noninteractive apt-get: command not found Failure - Main Bootstrap (node + git) exit status 127 The secs4j-interop job runs on the bare runner (not inside a `container:`) because it needs the host's docker socket to run `docker compose up -d server`. The runner image isn't fixed across deployments — catthehacker/ubuntu has apt-get, but a minimal node image doesn't. The old script hard-coded `apt-get` and exit 127'd on anything else. New bootstrap: - Checks what's already on PATH (git, node, docker). If all three are present, exits 0 — most act-runner images come pre-loaded. - Otherwise picks the right package manager (apt-get or apk) and installs only the missing pieces. - Errors out with a useful message if neither package manager exists, instead of failing on a missing command. Also updates the inline comment that still said "20 checks" — actual is 55 (matches the count in README / PROOFS.md / COMPLIANCE.md). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
4ddf8e0f48 |
verify: libFuzzer harness for secs2::decode + try_parse_sml
Coverage-guided structural search for crashes and undefined behaviour on arbitrary input to our two parsers. What's wired: - -DSECSGEM_FUZZ=ON CMake option, clang-only. Adds -fsanitize=fuzzer-no-link,address,undefined to all targets + -fsanitize=fuzzer to the two fuzz executables. - apps/fuzz_secs2_decode.cpp — feeds raw bytes to secs2::decode. Catches secs2::CodecError (expected) but traps on anything else leaking (would be a hardening bug). - apps/fuzz_sml_parse.cpp — feeds string to try_parse_sml, which is contractually nothrow-equivalent; traps on any exception. - .gitea/workflows/ci.yml — `libfuzzer` job builds with clang and runs each fuzzer for 60s in CI. Any crash / ASan / UBSan flag fails the job. - Dockerfile gains clang + libclang-rt-18-dev so devs can run locally with the same toolchain. Result on a fresh 30-second local run: fuzz_secs2_decode: 70 727 random inputs, 0 crashes fuzz_sml_parse: 284 950 random inputs, 0 crashes The coverage-guided search found and synthesized inputs that exercise: zero-byte, single-byte format tags, all length-byte counts (1/2/3), nested lists, format bytes with reserved bits, the "BOOLEAN" SML token, malformed quoted strings, etc. libFuzzer's recommended dictionary at the end of each run shows what bytes / substrings the coverage feedback discovered as discriminating — useful signals if we ever want a hand-curated corpus. README proof table grows to 8 commands. After this: - 426 unit tests (internal) - 47 conformance harness checks (internal) - 24 secsgem-py interop checks (external — Python ref impl) - 20 secs4j interop checks (external — independent Java impl) - 69 frames dissected by Wireshark HSMS dissector (external) - 196 SEMI E5 KAT assertions (standards body's encoding rules) - **~70k + ~285k random inputs, 0 crashes (external)** - 100k random tool ops with all invariants holding (internal) - YAML validation (internal) - TSan clean on 2 557 assertions (internal correctness aid) Five distinct external proofs now, each covering a different angle. Plan: VERIFICATION.md §4. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
2fce2fad0c |
verify: secs4j cross-validation (independent Java implementation)
20 cross-validation checks PASS against [secs4java8] (Apache 2.0,
kenta-shimizu) — an independent SECS/HSMS implementation in Java by
a different author from a different language ecosystem. Distinct
implementer = independent spec interpretation. Two libraries
agreeing on wire bytes is much stronger evidence of spec-correctness
than either alone.
Coverage targets the gap the secsgem-py interop deliberately skipped
(secsgem-py's SFDL grammar couldn't easily express GEM 300 bodies
with variable lists of named scalars):
- S1F1/F13/F17/F19/F21/F23 — establish comms + namelists
- S2F17 — clock
- S2F23 — trace init (5-field body)
- S2F49 — enhanced remote command (DATAID + OBJSPEC + RCMD + params)
- S3F17/F19/F25/F27 — full E87 carrier surface (action, slot map
verify, transfer with port pair, cancel)
- S5F13/F17 — exception recovery (EXID + EXRECVRA)
- S14F9/F11 — E94 CJ create with prjobids list, CJ delete
- S16F5/F27 — E40 PJ command, E94 CJ command
- S1F15 — offline cleanup
20/20 PASS against the demo equipment. Reply S/F matches the spec
for every transaction; specific ACK values vary by equipment state
(CarrierIDUnknown for an unknown carrier is just as valid as Accept
for a known one) so we assert on the wire shape, not the result.
Ship layout:
interop/secs4j/Dockerfile — eclipse-temurin:21-jdk + clone
+ build of secs4java8 → Export.jar
interop/secs4j/Secs4jHostHarness.java
— 20 round_trip assertions; uses
Secs2.list/uint4/ascii to build
full GEM 300 bodies; comm.send()
for arbitrary S/F pairs
interop/secs4j_validate.sh — orchestrator: builds image,
compiles harness, starts compose
server, runs Java container on
the secs network against it
.gitea/workflows/ci.yml — secs4j-interop job in CI
README.md — proof table grows to 7 commands
.gitignore — *.class
After this commit our proof chain has:
- SEMI E5 KAT (standards body's own arithmetic)
- tshark dissector (Wireshark's HSMS impl)
- secsgem-py interop (Python reference impl)
- **secs4j interop** (independent Java impl)
+ 426 unit tests, 47 conformance harness checks, 100k random ops,
YAML validation
Four independent external proofs, three of them on overlapping wire
surface from independent angles.
Plan: VERIFICATION.md §3.
[secs4java8]: https://github.com/kenta-shimizu/secs4java8
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
5baf3f4dc7 |
verify: tshark HSMS dissector validation (independent third codec)
Wireshark's built-in HSMS dissector — written by network-protocol authors who don't know us, didn't talk to us, and don't share implementation details with secsgem-py — is a third independent codec for our framing. If they parse our pcap without warnings, our HSMS framing is wire-correct independently of both our internal tests and the secsgem-py interop path. interop/tshark_validate.sh: - Boots secs_server on 127.0.0.1:5099 (away from the demo port) - Captures the loopback wire traffic with tcpdump - Runs secs_client through ~24 transactions plus Separate.req + TCP FIN - Parses the pcap with tshark -V using the HSMS dissector - Asserts: no "Malformed Packet", no "Dissector bug", at least one HSMS frame, expected tokens present (Select.req/rsp, Separate.req, Data message), reports histogram (count by control type + distinct S/F pairs) Result against the demo: 69 HSMS frames dissected, 49 distinct S/F pairs (S01F01..S16F28), all clean. Dockerfile gains tshark + tcpdump. .gitea/workflows/ci.yml gains a `tshark-dissector` job that runs this validator as part of every push to main. README proof table grows to 6 commands. VERIFICATION.md §1a documents a follow-up: round-trip the KAT fixtures through secsgem-py to corroborate that the format codes we used match an independent implementation. Strengthens the KAT proof from "internally consistent" to "confirmed by a second implementer who read the spec without talking to us." Plan: VERIFICATION.md §2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
943f3bbcd5 |
ci: ThreadSanitizer lane + fix use-after-free TSan flagged
Adds a -DSECSGEM_TSAN=ON CMake option that builds every target with -fsanitize=thread + debug symbols + -O1 + frame pointers. Wires a dedicated thread-sanitizer job into .gitea/workflows/ci.yml that builds and runs the full test suite under TSan with TSAN_OPTIONS=halt_on_error=1 (any flagged race fails the job, not just warns). Result against the full 426-case / 2557-assertion suite: 0 warnings, all green. That converts the existing test_thread_safety.cpp (which exercised the asio::post-onto-strand pattern) and test_concurrency (in-flight transaction interleaving) and test_robustness_fuzz (28 random action types × thousands of ticks) from "pattern smoke-tests" into actual race detection. The first TSan run caught a real bug in test_robustness_fuzz's act_exception_complete: it held a pointer to an ExceptionStore entry across fire_internal(RecoveryComplete), which deletes the entry. The subsequent state() read was a use-after-free. TSan flagged it 8 times (4 reads × 2 stack-frame variants). Fix is scoped lookup + re-check via has() after the mutation; matches the contract any reasonable caller would follow. The asio std_fenced_block atomic_thread_fence path generates TSan "not supported" warnings during compile — those are asio's, not ours, and don't affect runtime detection. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
db426cbeed |
ci: bootstrap node before actions/checkout on Gitea runners
`actions/checkout@v4` is a JavaScript action — it expects `node` on PATH in the runner image. Gitea Actions (and local `act`) running against `ubuntu:24.04` had neither node nor git pre-installed, so checkout failed with: ❌ Failure - Main actions/checkout@v4 exitcode '127': command not found The pre-step now installs nodejs + git + ca-certificates from apt before checkout runs. The rest of the C++ toolchain installs in a second step after the source tree is on disk. Doesn't affect GitHub-hosted runners (their images already have node); doesn't change build behaviour either. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
16b734e946 |
#1 CI: run unit tests on push to main + on PRs
tests / build-and-test (push) Failing after 35s
Gitea Actions workflow at .gitea/workflows/ci.yml. Spins up an ubuntu:24.04 container, installs the same apt deps the Dockerfile uses (build-essential, cmake, ninja-build, libasio-dev, libyaml-cpp-dev, python3, python3-yaml), checks out, runs cmake + ninja, and executes the doctest binary. Runs the same toolchain as the local Docker setup; no docker-in-docker required. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |