4 Commits

Author	SHA1	Message	Date
raphael	9876dd9b5a	feat(daemon): D10 carriers + E16 ops RPCs + stress test + virtual fab ... Completes the daemon's GEM300 surface and adds two new test tiers. D10 — E87 carriers: CarrierStore gains the HandlerSlot observer pattern (add_id/slot_map/access_handler). The daemon's id-observer forwards host S3F17 decisions onto the Subscribe stream as CarrierAction (PROCEED on a Confirmed transition, CANCEL on CancelCarrier); ReportCarrier drives the flow tool-side: WAITING creates the carrier + records the slot map, IN_ACCESS/COMPLETE advance the access FSM (INVALID_OBJECT on unknown, CANNOT_DO_NOW on an illegal transition). E16 — operations RPCs: Describe (full name inventory: variables/events/ alarms/commands/constants + device header), FlushSpool (purge or drain), SendTerminalMessage (S10F1 tool->host, honest CANNOT_DO_NOW when no host and stream 10 isn't spoolable). Stream responsiveness: Subscribe/WatchHealth poll at 100ms (was 500ms) so a cancelled stream frees its sync-server worker thread promptly — this was found by the new stress test, which hung under Subscribe churn at 500ms. Tests: - A randomized concurrent RPC stress case: 4 threads x 250 seeded ops (set/get/fire/alarm/control-state/describe + Subscribe churn), asserts no failed RPC and a still-responsive engine afterward; prints its seed; a strong TSan target. - A virtual fab (interop/virtual_fab.py + the `fab` compose service / tools/spawn_fab.sh): N daemons, each with a secsgem-py host AND a secsgem_client tool, driven by seeded random traffic with end-to-end invariant checks (set/get round-trips, event->S6F11 and alarm->S5F1 delivery, command->tool->completion). Verified green at N=3 (~150 ops/eq, all commands round-tripped, 0 violations). Wired into run_interop.sh (now 13 steps). Also fixes the CI break from the previous commit: the Python-client lane's test_values.py step lacked PYTHONPATH=clients/python (now step-level env). Two bugs found and fixed while building this, both mine from this batch: 1. carrier test hung on a CancelCarrier of a still-NotConfirmed carrier — a self-transition the FSM doesn't signal, so the observer never fired and the stream Read blocked forever. Fixed to cancel a Confirmed carrier; the NotConfirmed edge is documented as a known E87 limitation. 2. the 500ms stream poll above. Daemon suite 7 cases / 214 assertions; core 475 / 3097; virtual fab green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-26 15:05:13 +02:00
raphael	54626ceb6a	feat(daemon): Phase E — production hardening, complete ... Exposure: --grpc default flipped from 0.0.0.0 to 127.0.0.1 (the API is unauthenticated by design; auth belongs to the transport), Unix-domain- socket support (--grpc unix:///run/secs_gemd/api.sock = zero network surface), SECURITY.md documents the contract and ch42 gained a "Running it in production" section (which also documents the HSMS-SS single-session assumption). Graceful shutdown: SIGTERM/SIGINT land on an asio::signal_set on the io thread, which nudges grpc Shutdown with a 2s deadline (cancels open Subscribe/WatchHealth streams); Wait() returns on the MAIN thread, which stops the engine (rt->stop() joins the io thread, so it must not run on it). Exit 0, journal-safe, the in-code TODO is gone. --spool-dir added so host-bound events survive daemon restarts. Observability: --metrics serves Prometheus gauges secsgem_link_selected / secsgem_control_state / secsgem_spool_depth, wired via the Phase-0 add_link_observer/add_control_state_observer hooks + io-thread sampling. Deployment: deploy/secs_gemd.service — hardened systemd unit (DynamicUser, ProtectSystem=strict, StateDirectory for the spool, UDS for the API, TimeoutStopSec aligned with the graceful-shutdown window). Enforcement: tools/check_daemon_ops.sh proves all three operational claims (unix-socket gRPC accepts, all gauges present on /metrics, SIGTERM -> exit 0 + clean-stop log) — green; wired into tools/run_interop.sh (now 11 steps) and CI. CI python-interop lane also gained the pyclient and spool-restart steps, so every harness now runs in CI. TODO sweep: the shutdown TODO is fixed; the four remaining TODOs (nested list formats, C2-as-text, U8>2^63, CONNECTED link state) are deliberate deferred edge cases, each marked in code with context. Daemon suite re-verified green (175 assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-11 00:07:37 +02:00
raphael	8686654b15	feat(client): the Python client — a GEM tool in plain Python (Phase C) ... clients/python: pip-installable "secsgem-client", pure Python (stubs pre-generated from equipment.proto, import made package-relative; no compiled extension, no SEMI knowledge, no C++ toolchain). The API the whole effort aimed at: eq = Equipment("localhost:50051") eq.set(ChamberPressure=2.5); eq["WaferCounter"] = 7 eq.fire("ProcessStarted", ChamberPressure=2.75) eq.alarm("chiller_temp_high"); eq.clear("chiller_temp_high") @eq.on("START") def start(cmd): ... # auto-CompleteCommand after return eq.listen(background=True) eq.control_state; eq.request_control_state("HOST_OFFLINE"); eq.health() Errors raise SecsGemError carrying the daemon's message ("no variable named ..."). bool checked before int in conversion (isinstance(True, int)). examples/mini_tool.py is a complete GEM tool in ~25 lines. PROOF — interop/pyclient_interop.py drives the PUBLISHED package (not raw stubs) against a live secs_gemd with secsgem-py as the fab host: 13 checks all green on first run — set/get round-trips, item syntax, SecsGemError on unknown names, control state, health, fire->S6F11 on the host's wire, alarm/clear->S5F1 with correct set bit, the full command loop (host S2F41 -> HCACK=4 -> @eq.on handler -> completion event back at the host), operator offline. Conversion layer unit-tested standalone; both wired into tools/run_interop.sh as the pyclient step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 22:57:55 +02:00
raphael	42044e92e2	ci(interop): one-command external-validation suite + CI lanes for the daemon ... tools/run_interop.sh runs ALL nine validation steps with a PASS/FAIL summary: build, unit (464), daemon-unit (41), secsgem-py host vs server (31 checks), secs_conformance (47), gRPC+secsgem-py daemon bridge, spool persistence across restart, tshark HSMS dissector, secs4java8 (55 checks). Verified green end-to-end. The unit suite is partly self-referential (our parsers validate our builders); these external validators are the real oracle — now they run with one command instead of by hand. Two bugs found by running it: unbounded ninja at -O3 OOM-kills cc1plus in memory-constrained Docker VMs (build with -j 2) and bash-3.2 lacks negative array subscripts. CI: grpc deps added to the build job so secs_gemd + secs_gemd_tests build and RUN in CI (previously the daemon silently dropped out — now fails loudly if missing), plus a python-interop lane running py-host/conformance/daemon harnesses against localhost in one container (no docker-in-docker). Service hardening while in there: reject proto Values with no kind set at the RPC edge (previously silently became ASCII ""), TODO markers for list element formats and daemon graceful shutdown. New tests: unset-Value guard + a property test iterating ALL configured variables via gRPC asserting each keeps its declared SECS-II format (daemon tests 16 -> 41 assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 19:08:37 +02:00