13 Commits

Author	SHA1	Message	Date
raphael	9876dd9b5a	feat(daemon): D10 carriers + E16 ops RPCs + stress test + virtual fab ... Completes the daemon's GEM300 surface and adds two new test tiers. D10 — E87 carriers: CarrierStore gains the HandlerSlot observer pattern (add_id/slot_map/access_handler). The daemon's id-observer forwards host S3F17 decisions onto the Subscribe stream as CarrierAction (PROCEED on a Confirmed transition, CANCEL on CancelCarrier); ReportCarrier drives the flow tool-side: WAITING creates the carrier + records the slot map, IN_ACCESS/COMPLETE advance the access FSM (INVALID_OBJECT on unknown, CANNOT_DO_NOW on an illegal transition). E16 — operations RPCs: Describe (full name inventory: variables/events/ alarms/commands/constants + device header), FlushSpool (purge or drain), SendTerminalMessage (S10F1 tool->host, honest CANNOT_DO_NOW when no host and stream 10 isn't spoolable). Stream responsiveness: Subscribe/WatchHealth poll at 100ms (was 500ms) so a cancelled stream frees its sync-server worker thread promptly — this was found by the new stress test, which hung under Subscribe churn at 500ms. Tests: - A randomized concurrent RPC stress case: 4 threads x 250 seeded ops (set/get/fire/alarm/control-state/describe + Subscribe churn), asserts no failed RPC and a still-responsive engine afterward; prints its seed; a strong TSan target. - A virtual fab (interop/virtual_fab.py + the `fab` compose service / tools/spawn_fab.sh): N daemons, each with a secsgem-py host AND a secsgem_client tool, driven by seeded random traffic with end-to-end invariant checks (set/get round-trips, event->S6F11 and alarm->S5F1 delivery, command->tool->completion). Verified green at N=3 (~150 ops/eq, all commands round-tripped, 0 violations). Wired into run_interop.sh (now 13 steps). Also fixes the CI break from the previous commit: the Python-client lane's test_values.py step lacked PYTHONPATH=clients/python (now step-level env). Two bugs found and fixed while building this, both mine from this batch: 1. carrier test hung on a CancelCarrier of a still-NotConfirmed carrier — a self-transition the FSM doesn't signal, so the observer never fired and the stream Read blocked forever. Fixed to cancel a Confirmed carrier; the NotConfirmed edge is documented as a known E87 limitation. 2. the 500ms stream poll above. Daemon suite 7 cases / 214 assertions; core 475 / 3097; virtual fab green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-26 15:05:13 +02:00
raphael	54626ceb6a	feat(daemon): Phase E — production hardening, complete ... Exposure: --grpc default flipped from 0.0.0.0 to 127.0.0.1 (the API is unauthenticated by design; auth belongs to the transport), Unix-domain- socket support (--grpc unix:///run/secs_gemd/api.sock = zero network surface), SECURITY.md documents the contract and ch42 gained a "Running it in production" section (which also documents the HSMS-SS single-session assumption). Graceful shutdown: SIGTERM/SIGINT land on an asio::signal_set on the io thread, which nudges grpc Shutdown with a 2s deadline (cancels open Subscribe/WatchHealth streams); Wait() returns on the MAIN thread, which stops the engine (rt->stop() joins the io thread, so it must not run on it). Exit 0, journal-safe, the in-code TODO is gone. --spool-dir added so host-bound events survive daemon restarts. Observability: --metrics serves Prometheus gauges secsgem_link_selected / secsgem_control_state / secsgem_spool_depth, wired via the Phase-0 add_link_observer/add_control_state_observer hooks + io-thread sampling. Deployment: deploy/secs_gemd.service — hardened systemd unit (DynamicUser, ProtectSystem=strict, StateDirectory for the spool, UDS for the API, TimeoutStopSec aligned with the graceful-shutdown window). Enforcement: tools/check_daemon_ops.sh proves all three operational claims (unix-socket gRPC accepts, all gauges present on /metrics, SIGTERM -> exit 0 + clean-stop log) — green; wired into tools/run_interop.sh (now 11 steps) and CI. CI python-interop lane also gained the pyclient and spool-restart steps, so every harness now runs in CI. TODO sweep: the shutdown TODO is fixed; the four remaining TODOs (nested list formats, C2-as-text, U8>2^63, CONNECTED link state) are deliberate deferred edge cases, each marked in code with context. Daemon suite re-verified green (175 assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-11 00:07:37 +02:00
raphael	b30443089f	feat(clients)+test(interop): C++ client + Java validation of the daemon (B7) ... B7 — the daemon's HSMS face under the Java reference: Dockerfile.server now bakes secs_gemd alongside secs_server (grpc deps in both stages), and secs4j_validate.sh gains TARGET=gemd to point the 55-check secs4java8 suite at the daemon instead. Result: 55/55 green. With secsgem-py already validating both faces, byte-identical GEM between secs_server and secs_gemd is now proven by both reference implementations, not inferred from shared code. CI runs the daemon target as an extra step (image layers shared). Second client — clients/cpp: a header-only C++ twin of the Python client over the same proto. eq.set("ChamberPressure", 2.5) with bare literals (integral/floating dispatch avoids variant ambiguity), get/fire/alarm/ clear, control_state/request_control_state/health, on("START", fn) + listen()/listen_async()/stop() with auto-CompleteCommand, SecsGemError carrying the daemon's message. cpp_mini_tool (~30 lines) mirrors the Python mini_tool. Tested end-to-end over real loopback TCP against the service inside secs_gemd_tests — now 4 cases / 141 assertions — including set/get round-trips, error text, alarm-by-name into the model, health, and the full HCACK-4 command loop with parameters. (Build note: two grpc-heavy TUs at -O3 OOM even at -j2 on Docker Desktop; built -j1. Known environment limitation, roadmap-documented.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 23:37:14 +02:00
raphael	cf230d4119	chore(phase0): name validation, golden frames, daemon into library tree, TSan daemon lane ... Item 8a — ConfigValidator warns on non-identifier variable/event/alarm/ command names ([A-Za-z_][A-Za-z0-9_]*): language bindings expose names as kwargs/attributes, so 'Chamber Pressure' would be unusable in the planned Python client. Warning not error — the wire doesn't care. Tested (4 warning sites + good-name negative). Item 4 tail — golden frames for S5F1 (Binary ALCD / U4 ALID / ASCII ALTX) and a composed S6F11 (the production-critical report shape), bytes hand- computed from E5 encoding rules: external pins on message composition. Item 7 — equipment_service.hpp moved to include/secsgem/daemon/ (apps/ include-path hack removed) and a TSan daemon lane added locally + in CI. tools/tsan.supp suppresses races whose accesses sit entirely inside the UNinstrumented system libgrpc/libgpr/libabsl (epoll wakeups, absl Mutex GraphCycles bookkeeping); our frames stay fully checked. The lane earned its keep on first run: it caught a REAL threading-contract violation — a daemon test reading model stores from the test thread while the io thread serviced posted writes — fixed to use read_sync, exactly per the documented contract. Now TSan-clean under halt_on_error=1 in the full production threading shape. Suites: core 470/3068, daemon Release+TSan 125/125 each. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 22:28:33 +02:00
raphael	42044e92e2	ci(interop): one-command external-validation suite + CI lanes for the daemon ... tools/run_interop.sh runs ALL nine validation steps with a PASS/FAIL summary: build, unit (464), daemon-unit (41), secsgem-py host vs server (31 checks), secs_conformance (47), gRPC+secsgem-py daemon bridge, spool persistence across restart, tshark HSMS dissector, secs4java8 (55 checks). Verified green end-to-end. The unit suite is partly self-referential (our parsers validate our builders); these external validators are the real oracle — now they run with one command instead of by hand. Two bugs found by running it: unbounded ninja at -O3 OOM-kills cc1plus in memory-constrained Docker VMs (build with -j 2) and bash-3.2 lacks negative array subscripts. CI: grpc deps added to the build job so secs_gemd + secs_gemd_tests build and RUN in CI (previously the daemon silently dropped out — now fails loudly if missing), plus a python-interop lane running py-host/conformance/daemon harnesses against localhost in one container (no docker-in-docker). Service hardening while in there: reject proto Values with no kind set at the RPC edge (previously silently became ASCII ""), TODO markers for list element formats and daemon graceful shutdown. New tests: unset-Value guard + a property test iterating ALL configured variables via gRPC asserting each keeps its declared SECS-II format (daemon tests 16 -> 41 assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 19:08:37 +02:00
raphael	fc3422a4a9	docs: move root .md files into docs/ + update every reference ... Picks up the file renames that landed alongside the previous commit and fixes everything that pointed at the old root locations: - README.md doc-map updated: every entry now points at docs/X.md, with a new "docs/" lead entry pointing at the guided-tour index. - README inline cross-refs (ARCHITECTURE / INTEGRATION / SECURITY / BENCHMARKS / MES_INTEROP / PROOFS) repointed to docs/. - README "Interop" section rewritten — used to mention only secsgem-py; now covers all four external validators (secsgem-py 31 / secs4java8 55 / tshark 69 frames / libFuzzer 200 k+ runs) with a one-line summary each, plus pointers to interop/README.md and docs/VERIFICATION.md. - README "Deferred follow-ups" cleaned: dropped the explanatory "Listed here so reviewers don't go looking for them in COMPLIANCE.md and find an 'out of scope' entry that sounds defensive" sentence — the section header speaks for itself. - docs/00_index.md "Where the rest of the docs live" table: dropped every `../` prefix since the docs are now siblings. - docs/01_what_is_secs_gem.md PROOFS reference updated to sibling. - docs/02_the_cast.md INTEGRATION + MES_INTEROP refs updated to siblings; dropped the stale "at the repo root" wording. - interop/README.md: VERIFICATION + PROOFS refs updated to ../docs/X.md; stale "~24 + 4 checks" updated to 31 (matches PROOFS.md and README). - examples/pvd_tool/README.md: every doc cross-ref now points at ../../docs/X.md. - Source / data / CI comments mentioning doc names (e.g. "INTEGRATION.md §3", "COMPLIANCE.md gap") rewritten to "docs/INTEGRATION.md §3" etc. — affects 9 files across include/, apps/, tests/, data/, examples/, .gitea/workflows/. Verified: full build under docker passes, 445/445 test cases pass, 2 753/2 753 assertions pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 19:36:27 +02:00
raphael	bc54de7711	ci: secs4j-interop bootstrap resilient to runner image variant ... CI log showed: Run export DEBIAN_FRONTEND=noninteractive apt-get: command not found Failure - Main Bootstrap (node + git) exit status 127 The secs4j-interop job runs on the bare runner (not inside a `container:`) because it needs the host's docker socket to run `docker compose up -d server`. The runner image isn't fixed across deployments — catthehacker/ubuntu has apt-get, but a minimal node image doesn't. The old script hard-coded `apt-get` and exit 127'd on anything else. New bootstrap: - Checks what's already on PATH (git, node, docker). If all three are present, exits 0 — most act-runner images come pre-loaded. - Otherwise picks the right package manager (apt-get or apk) and installs only the missing pieces. - Errors out with a useful message if neither package manager exists, instead of failing on a missing command. Also updates the inline comment that still said "20 checks" — actual is 55 (matches the count in README / PROOFS.md / COMPLIANCE.md). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 19:17:18 +02:00
raphael	4ddf8e0f48	verify: libFuzzer harness for secs2::decode + try_parse_sml ... Coverage-guided structural search for crashes and undefined behaviour on arbitrary input to our two parsers. What's wired: - -DSECSGEM_FUZZ=ON CMake option, clang-only. Adds -fsanitize=fuzzer-no-link,address,undefined to all targets + -fsanitize=fuzzer to the two fuzz executables. - apps/fuzz_secs2_decode.cpp — feeds raw bytes to secs2::decode. Catches secs2::CodecError (expected) but traps on anything else leaking (would be a hardening bug). - apps/fuzz_sml_parse.cpp — feeds string to try_parse_sml, which is contractually nothrow-equivalent; traps on any exception. - .gitea/workflows/ci.yml — `libfuzzer` job builds with clang and runs each fuzzer for 60s in CI. Any crash / ASan / UBSan flag fails the job. - Dockerfile gains clang + libclang-rt-18-dev so devs can run locally with the same toolchain. Result on a fresh 30-second local run: fuzz_secs2_decode: 70 727 random inputs, 0 crashes fuzz_sml_parse: 284 950 random inputs, 0 crashes The coverage-guided search found and synthesized inputs that exercise: zero-byte, single-byte format tags, all length-byte counts (1/2/3), nested lists, format bytes with reserved bits, the "BOOLEAN" SML token, malformed quoted strings, etc. libFuzzer's recommended dictionary at the end of each run shows what bytes / substrings the coverage feedback discovered as discriminating — useful signals if we ever want a hand-curated corpus. README proof table grows to 8 commands. After this: - 426 unit tests (internal) - 47 conformance harness checks (internal) - 24 secsgem-py interop checks (external — Python ref impl) - 20 secs4j interop checks (external — independent Java impl) - 69 frames dissected by Wireshark HSMS dissector (external) - 196 SEMI E5 KAT assertions (standards body's encoding rules) - **~70k + ~285k random inputs, 0 crashes (external)** - 100k random tool ops with all invariants holding (internal) - YAML validation (internal) - TSan clean on 2 557 assertions (internal correctness aid) Five distinct external proofs now, each covering a different angle. Plan: VERIFICATION.md §4. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 16:27:36 +02:00
raphael	2fce2fad0c	verify: secs4j cross-validation (independent Java implementation) ... 20 cross-validation checks PASS against [secs4java8] (Apache 2.0, kenta-shimizu) — an independent SECS/HSMS implementation in Java by a different author from a different language ecosystem. Distinct implementer = independent spec interpretation. Two libraries agreeing on wire bytes is much stronger evidence of spec-correctness than either alone. Coverage targets the gap the secsgem-py interop deliberately skipped (secsgem-py's SFDL grammar couldn't easily express GEM 300 bodies with variable lists of named scalars): - S1F1/F13/F17/F19/F21/F23 — establish comms + namelists - S2F17 — clock - S2F23 — trace init (5-field body) - S2F49 — enhanced remote command (DATAID + OBJSPEC + RCMD + params) - S3F17/F19/F25/F27 — full E87 carrier surface (action, slot map verify, transfer with port pair, cancel) - S5F13/F17 — exception recovery (EXID + EXRECVRA) - S14F9/F11 — E94 CJ create with prjobids list, CJ delete - S16F5/F27 — E40 PJ command, E94 CJ command - S1F15 — offline cleanup 20/20 PASS against the demo equipment. Reply S/F matches the spec for every transaction; specific ACK values vary by equipment state (CarrierIDUnknown for an unknown carrier is just as valid as Accept for a known one) so we assert on the wire shape, not the result. Ship layout: interop/secs4j/Dockerfile — eclipse-temurin:21-jdk + clone + build of secs4java8 → Export.jar interop/secs4j/Secs4jHostHarness.java — 20 round_trip assertions; uses Secs2.list/uint4/ascii to build full GEM 300 bodies; comm.send() for arbitrary S/F pairs interop/secs4j_validate.sh — orchestrator: builds image, compiles harness, starts compose server, runs Java container on the secs network against it .gitea/workflows/ci.yml — secs4j-interop job in CI README.md — proof table grows to 7 commands .gitignore — *.class After this commit our proof chain has: - SEMI E5 KAT (standards body's own arithmetic) - tshark dissector (Wireshark's HSMS impl) - secsgem-py interop (Python reference impl) - **secs4j interop** (independent Java impl) + 426 unit tests, 47 conformance harness checks, 100k random ops, YAML validation Four independent external proofs, three of them on overlapping wire surface from independent angles. Plan: VERIFICATION.md §3. [secs4java8]: https://github.com/kenta-shimizu/secs4java8 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 16:12:44 +02:00
raphael	5baf3f4dc7	verify: tshark HSMS dissector validation (independent third codec) ... Wireshark's built-in HSMS dissector — written by network-protocol authors who don't know us, didn't talk to us, and don't share implementation details with secsgem-py — is a third independent codec for our framing. If they parse our pcap without warnings, our HSMS framing is wire-correct independently of both our internal tests and the secsgem-py interop path. interop/tshark_validate.sh: - Boots secs_server on 127.0.0.1:5099 (away from the demo port) - Captures the loopback wire traffic with tcpdump - Runs secs_client through ~24 transactions plus Separate.req + TCP FIN - Parses the pcap with tshark -V using the HSMS dissector - Asserts: no "Malformed Packet", no "Dissector bug", at least one HSMS frame, expected tokens present (Select.req/rsp, Separate.req, Data message), reports histogram (count by control type + distinct S/F pairs) Result against the demo: 69 HSMS frames dissected, 49 distinct S/F pairs (S01F01..S16F28), all clean. Dockerfile gains tshark + tcpdump. .gitea/workflows/ci.yml gains a `tshark-dissector` job that runs this validator as part of every push to main. README proof table grows to 6 commands. VERIFICATION.md §1a documents a follow-up: round-trip the KAT fixtures through secsgem-py to corroborate that the format codes we used match an independent implementation. Strengthens the KAT proof from "internally consistent" to "confirmed by a second implementer who read the spec without talking to us." Plan: VERIFICATION.md §2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 16:02:38 +02:00
raphael	943f3bbcd5	ci: ThreadSanitizer lane + fix use-after-free TSan flagged ... Adds a -DSECSGEM_TSAN=ON CMake option that builds every target with -fsanitize=thread + debug symbols + -O1 + frame pointers. Wires a dedicated thread-sanitizer job into .gitea/workflows/ci.yml that builds and runs the full test suite under TSan with TSAN_OPTIONS=halt_on_error=1 (any flagged race fails the job, not just warns). Result against the full 426-case / 2557-assertion suite: 0 warnings, all green. That converts the existing test_thread_safety.cpp (which exercised the asio::post-onto-strand pattern) and test_concurrency (in-flight transaction interleaving) and test_robustness_fuzz (28 random action types × thousands of ticks) from "pattern smoke-tests" into actual race detection. The first TSan run caught a real bug in test_robustness_fuzz's act_exception_complete: it held a pointer to an ExceptionStore entry across fire_internal(RecoveryComplete), which deletes the entry. The subsequent state() read was a use-after-free. TSan flagged it 8 times (4 reads × 2 stack-frame variants). Fix is scoped lookup + re-check via has() after the mutation; matches the contract any reasonable caller would follow. The asio std_fenced_block atomic_thread_fence path generates TSan "not supported" warnings during compile — those are asio's, not ours, and don't affect runtime detection. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 15:32:02 +02:00
raphael	db426cbeed	ci: bootstrap node before actions/checkout on Gitea runners ... `actions/checkout@v4` is a JavaScript action — it expects `node` on PATH in the runner image. Gitea Actions (and local `act`) running against `ubuntu:24.04` had neither node nor git pre-installed, so checkout failed with: ❌ Failure - Main actions/checkout@v4 exitcode '127': command not found The pre-step now installs nodejs + git + ca-certificates from apt before checkout runs. The rest of the C++ toolchain installs in a second step after the source tree is on disk. Doesn't affect GitHub-hosted runners (their images already have node); doesn't change build behaviour either. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 14:40:44 +02:00
raphael	16b734e946	#1 CI: run unit tests on push to main + on PRs ... Gitea Actions workflow at .gitea/workflows/ci.yml. Spins up an ubuntu:24.04 container, installs the same apt deps the Dockerfile uses (build-essential, cmake, ninja-build, libasio-dev, libyaml-cpp-dev, python3, python3-yaml), checks out, runs cmake + ninja, and executes the doctest binary. Runs the same toolchain as the local Docker setup; no docker-in-docker required. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-02 09:54:23 +02:00