6 Commits

Author	SHA1	Message	Date
raphael	54626ceb6a	feat(daemon): Phase E — production hardening, complete ... Exposure: --grpc default flipped from 0.0.0.0 to 127.0.0.1 (the API is unauthenticated by design; auth belongs to the transport), Unix-domain- socket support (--grpc unix:///run/secs_gemd/api.sock = zero network surface), SECURITY.md documents the contract and ch42 gained a "Running it in production" section (which also documents the HSMS-SS single-session assumption). Graceful shutdown: SIGTERM/SIGINT land on an asio::signal_set on the io thread, which nudges grpc Shutdown with a 2s deadline (cancels open Subscribe/WatchHealth streams); Wait() returns on the MAIN thread, which stops the engine (rt->stop() joins the io thread, so it must not run on it). Exit 0, journal-safe, the in-code TODO is gone. --spool-dir added so host-bound events survive daemon restarts. Observability: --metrics serves Prometheus gauges secsgem_link_selected / secsgem_control_state / secsgem_spool_depth, wired via the Phase-0 add_link_observer/add_control_state_observer hooks + io-thread sampling. Deployment: deploy/secs_gemd.service — hardened systemd unit (DynamicUser, ProtectSystem=strict, StateDirectory for the spool, UDS for the API, TimeoutStopSec aligned with the graceful-shutdown window). Enforcement: tools/check_daemon_ops.sh proves all three operational claims (unix-socket gRPC accepts, all gauges present on /metrics, SIGTERM -> exit 0 + clean-stop log) — green; wired into tools/run_interop.sh (now 11 steps) and CI. CI python-interop lane also gained the pyclient and spool-restart steps, so every harness now runs in CI. TODO sweep: the shutdown TODO is fixed; the four remaining TODOs (nested list formats, C2-as-text, U8>2^63, CONNECTED link state) are deliberate deferred edge cases, each marked in code with context. Daemon suite re-verified green (175 assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-11 00:07:37 +02:00
raphael	8686654b15	feat(client): the Python client — a GEM tool in plain Python (Phase C) ... clients/python: pip-installable "secsgem-client", pure Python (stubs pre-generated from equipment.proto, import made package-relative; no compiled extension, no SEMI knowledge, no C++ toolchain). The API the whole effort aimed at: eq = Equipment("localhost:50051") eq.set(ChamberPressure=2.5); eq["WaferCounter"] = 7 eq.fire("ProcessStarted", ChamberPressure=2.75) eq.alarm("chiller_temp_high"); eq.clear("chiller_temp_high") @eq.on("START") def start(cmd): ... # auto-CompleteCommand after return eq.listen(background=True) eq.control_state; eq.request_control_state("HOST_OFFLINE"); eq.health() Errors raise SecsGemError carrying the daemon's message ("no variable named ..."). bool checked before int in conversion (isinstance(True, int)). examples/mini_tool.py is a complete GEM tool in ~25 lines. PROOF — interop/pyclient_interop.py drives the PUBLISHED package (not raw stubs) against a live secs_gemd with secsgem-py as the fab host: 13 checks all green on first run — set/get round-trips, item syntax, SecsGemError on unknown names, control state, health, fire->S6F11 on the host's wire, alarm/clear->S5F1 with correct set bit, the full command loop (host S2F41 -> HCACK=4 -> @eq.on handler -> completion event back at the host), operator offline. Conversion layer unit-tested standalone; both wired into tools/run_interop.sh as the pyclient step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 22:57:55 +02:00
raphael	cf230d4119	chore(phase0): name validation, golden frames, daemon into library tree, TSan daemon lane ... Item 8a — ConfigValidator warns on non-identifier variable/event/alarm/ command names ([A-Za-z_][A-Za-z0-9_]*): language bindings expose names as kwargs/attributes, so 'Chamber Pressure' would be unusable in the planned Python client. Warning not error — the wire doesn't care. Tested (4 warning sites + good-name negative). Item 4 tail — golden frames for S5F1 (Binary ALCD / U4 ALID / ASCII ALTX) and a composed S6F11 (the production-critical report shape), bytes hand- computed from E5 encoding rules: external pins on message composition. Item 7 — equipment_service.hpp moved to include/secsgem/daemon/ (apps/ include-path hack removed) and a TSan daemon lane added locally + in CI. tools/tsan.supp suppresses races whose accesses sit entirely inside the UNinstrumented system libgrpc/libgpr/libabsl (epoll wakeups, absl Mutex GraphCycles bookkeeping); our frames stay fully checked. The lane earned its keep on first run: it caught a REAL threading-contract violation — a daemon test reading model stores from the test thread while the io thread serviced posted writes — fixed to use read_sync, exactly per the documented contract. Now TSan-clean under halt_on_error=1 in the full production threading shape. Suites: core 470/3068, daemon Release+TSan 125/125 each. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 22:28:33 +02:00
raphael	42044e92e2	ci(interop): one-command external-validation suite + CI lanes for the daemon ... tools/run_interop.sh runs ALL nine validation steps with a PASS/FAIL summary: build, unit (464), daemon-unit (41), secsgem-py host vs server (31 checks), secs_conformance (47), gRPC+secsgem-py daemon bridge, spool persistence across restart, tshark HSMS dissector, secs4java8 (55 checks). Verified green end-to-end. The unit suite is partly self-referential (our parsers validate our builders); these external validators are the real oracle — now they run with one command instead of by hand. Two bugs found by running it: unbounded ninja at -O3 OOM-kills cc1plus in memory-constrained Docker VMs (build with -j 2) and bash-3.2 lacks negative array subscripts. CI: grpc deps added to the build job so secs_gemd + secs_gemd_tests build and RUN in CI (previously the daemon silently dropped out — now fails loudly if missing), plus a python-interop lane running py-host/conformance/daemon harnesses against localhost in one container (no docker-in-docker). Service hardening while in there: reject proto Values with no kind set at the RPC edge (previously silently became ASCII ""), TODO markers for list element formats and daemon graceful shutdown. New tests: unset-Value guard + a property test iterating ALL configured variables via gRPC asserting each keeps its declared SECS-II format (daemon tests 16 -> 41 assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 19:08:37 +02:00
raphael	2d60571a9c	interop: secsgem-py cross-validation harness + lenient identifier parsing ... Adds a Docker-based interop harness that drives the C++ server with secsgem-py 0.3.0 as the active host and probes a secsgem-py-passive equipment from a minimal C++ active client. Surfaces and fixes four interoperability bugs uncovered by cross-testing: * SEMI E5 identifier formatcodes are a U1|U2|U4|U8 wildcard; secsgem-py picks the narrowest fitting width while our parsers only accepted U4. `as_uN_scalar` / `as_iN_scalar` now accept any unsigned/signed width and range-check the downcast. * PPBODY (S7F3/F6) is "ASCII | Binary | List" per the spec; secsgem-py defaults to ASCII. Added BINARY_OR_ASCII codegen item type with `as_text_or_binary` accessor. * S1F23/F24 Collection Event Namelist was unimplemented; added schema + `vids_for(ceid)` accessor on EventReportSubscriptions plus the dispatch handler. * S10F1 was registered as a host->equipment handler, but per SEMI E5 §12 S10F1 is equipment->host; S10F3 is the actual host->equipment Terminal Display Single. Added an S10F3 handler alongside (we keep S10F1 too for backward compat). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-08 23:17:18 +02:00
raphael	29db1caedb	#6 SxFy codegen from YAML message catalog ... The bulk of the per-SxFy boilerplate — ~90 hand-written builders and parsers across 30+ message pairs — is now generated at build time from a single YAML catalog. Adding a new SECS-II message becomes a YAML edit; the C++ code is generated, not maintained. What changed ------------ data/messages.yaml The catalog. Describes every SxFy currently supported: stream, function, W-bit, builder name, optional parser name, and a recursive body shape grammar (scalar / list / list_of). Shapes carry SECS-II item types (ASCII, BINARY_BYTE, U4, F8, ITEM, ...) and optional C++ enum types for typed ack codes. Inner-most fields can be marked external_struct: true so structs already defined elsewhere (ReportData, CommandParameter) are referenced rather than redefined. tools/gen_messages.py Python codegen. Reads the catalog and emits one inline header. Handles nested shapes via depth-unique variable names in the generated IIFEs, so S6F11's three-level nesting compiles without lambda capture conflicts. Post-order traversal ensures inner structs are emitted before outer ones that reference them. Generates positional and (where applicable) struct builder overloads, plus struct-returning parsers for messages with a `parser:` entry. CMakeLists.txt Custom command runs gen_messages.py at configure/build time and emits ${CMAKE_BINARY_DIR}/generated/secsgem/gem/messages.hpp. Added to the secsgem target's include path so `#include "secsgem/gem/messages.hpp"` resolves to the generated file. Depends on the YAML + the script, so edits trigger regen automatically. Dockerfile Added python3 + python3-yaml to the toolchain image. include/secsgem/gem/messages_helpers.hpp (new) The small set of hand-written helpers the generated header relies on: scalar accessors (as_ascii / as_u4_scalar / ...), parse_u4_list_body, u4_list_item, ack_byte, ALED byte constants, and the two special-case messages whose shape doesn't fit the codegen schema (S1F4 needs per-row std::optional<Item> semantics; S5F6 needs a per-row ALCD callback). include/secsgem/gem/messages.hpp (deleted) The hand-written builder/parser file is gone. Its content now flows through the catalog + codegen. include/secsgem/gem/data_model.hpp Moved CommandParameter to namespace scope so it can be shared between the data model and the messages.yaml's external_struct entry. Added `using CommandParam = CommandParameter` for back-compat. apps/secs_server.cpp + apps/secs_client.cpp Updated the call sites that the codegen renamed or restructured: - parse_terminal_display() split into parse_s10f1 / parse_s10f3. - s1f14_establish_comms_ack now takes a McAck struct for the nested identity (mdln, softrev) — call site uses brace init. - S2F33/S2F35 parsers return strongly-typed entries (DefineReportEntry, LinkEventEntry); the server adapts these to the model's pair-based API at the call site. - S2F15 parser returns vector<EcSet>; iterate by .ecid/.value. - S5F3 parser returns EnableAlarmRequest{aled, alid}; bool comes from (aled & 0x80) != 0. - AlarmReport's is_set()/category() methods removed; callers use the raw alcd byte with bit math (alcd & 0x80, alcd & 0x7F). - s2f42_host_command_ack and s2f41_host_command always take their second list argument explicitly (no defaulted arg from codegen). tests/test_messages.cpp Updated to construct the generated typed structs (EcSet, StatusName, EnableAlarmRequest, CommandParameter, CommandParameterAck) and to read the new field names (.ecid/.value, .rptid/.vids, .ceid/.rptids, .name/.code). Coverage -------- Generated by codegen (44 SxFy in catalog): S1F1, S1F2, S1F3, S1F11, S1F12, S1F13, S1F14, S1F15, S1F16, S1F17, S1F18 S2F13, S2F14, S2F15, S2F16, S2F17, S2F18, S2F29, S2F30, S2F31, S2F32 S2F33, S2F34, S2F35, S2F36, S2F37, S2F38, S2F41, S2F42 S5F1, S5F2, S5F3, S5F4, S5F5 S6F11, S6F12 S7F3, S7F4, S7F5, S7F6, S7F19, S7F20 S10F1, S10F2, S10F3, S10F4 Hand-written (in messages_helpers.hpp): S1F4 list-of-optional-items shape (nullopt -> <L,0>) S5F6 per-row ALCD via callback Adding a new SxFy ----------------- Append a single entry to data/messages.yaml describing the body shape. The builder + parser appear in messages.hpp after the next build. The host command above for S2F41 (or any other added SxFy) requires no C++ changes if the body fits the recursive scalar/list/list_of grammar. Tests: 67 cases / 384 assertions still passing. Demo: byte-for-byte identical behaviour (Select, Establish, Online, S1F11/F3 namelist+values, S2F29 EC namelist, S2F33/F35/F37 dynamic event subscription, S2F41 START -> S6F11 emission, S5F5/F3 alarm directory + enable, S2F41 FAULT -> S5F1 alarm + S6F11, S7F19/F5 recipe ops, S10F1 terminal, S1F15 offline, Separate). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-02 09:43:36 +02:00