160 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
raphael	b2e22c3af0	chore: remove daemon roadmap — project is complete ... All phases shipped: daemon, gRPC API, Python client, GEM300 support, operations hardening. The roadmap served its purpose; replace it with final state documented in README + docs/*.md chapters. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>	2026-06-26 23:51:25 +02:00
raphael	c76ac1023e	chore: stop tracking Python .pyc artifacts; ignore them repo-wide ... The package's __pycache__/*.pyc were committed (regenerated on every build, churning diffs and risking stale bytecode). Untrack all 7, add Python cache patterns to the root .gitignore, and drop the now-redundant interop/.gitignore (the root rule covers it). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-26 22:11:59 +02:00
raphael	8a55137e57	feat(client): typo-safe protocol enums + context manager; add wafer_tool example ... Interface cleanup so the report_* family matches the typo-safe ethos of eq.names instead of leaking raw protobuf errors on a misspelled value. - Milestone / ModuleState / JobState: importable str-enums (member == its wire name, so plain strings still work) — autocomplete + a typo-checked happy path. The clean rule: equipment-specific *names* live on eq.names; fixed protocol *value-sets* are enums. - _enum_value(): resolves an enum-or-string arg client-side and, on a bad value, raises ValueError with a close-match hint *before* the wire. Wired into report_job / report_substrate / report_module / request_control_state (all previously raised a raw protobuf ValueError). - Equipment is now a context manager (with Equipment(...) as eq: ...). - examples/wafer_tool.py: a cluster tool tracking one wafer through one module end-to-end (E90 + E157), showing the enums + context manager. - tests/test_enums.py: asserts the enums stay in lockstep with the proto and that the typo path is helpful. Wired into run_interop.sh (pyclient step). - Interop drives both the enum and string forms on the wire + the ValueError typo path. Docs (ch16/ch42) updated; names-vs-enums rule documented. All Python unit tests + 25 pyclient interop checks pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-26 22:09:48 +02:00
raphael	2218b854ce	fix(daemon)+test: accurate duplicate-ARRIVED message; broaden E90/E157 + names coverage ... The duplicate-ARRIVED fix from the previous commit returned INVALID_OBJECT with the message "no substrate 'X'" — a lie, since the substrate exists. Rewrite ReportSubstrate so ARRIVED has its own ack mapping: a duplicate is CANNOT_DO_NOW with "substrate 'X' already exists" (a state conflict, not a missing object), and we never silently re-create over live FSM state. Coverage gaps closed: - C++: ARRIVED records carrier_id/slot (now asserted); module NOT_EXECUTING reset transition; duplicate-ARRIVED expects CANNOT_DO_NOW. - Interop: @eq.command now drives the real host S2F41 path (was @eq.on, so the headline decorator had zero wire coverage); @eq.command NameError on unknown name; eq.names var/alarm + dir() + typo-suggestion; replaced the two `check(..., True)` tautologies with full E90 journey + AT_DESTINATION and real error paths (ghost wafer raises, illegal module jump raises). All 8 daemon test cases (248 assertions) and 24 pyclient interop checks pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-26 22:01:20 +02:00
raphael	d22bbc4ab2	fix(daemon)+fix(client): close four fool-proofing gaps ... C++ (equipment_service.hpp): - ReportSubstrate ARRIVED: check CreateResult and return INVALID_OBJECT when the substrate ID already exists, instead of silently doing nothing - ReportSubstrate/ReportModule default switch branches: return false (→ CANNOT_DO_NOW) for unknown enum values instead of silently accepting Python (_client.py): - @eq.command: raise NameError (client-side name validation) instead of SecsGemError (which means "daemon declined a request") — wrong type - Module docstring: update example to show @eq.command / eq.names API Test (test_daemon_service.cpp): - Add duplicate-ARRIVED check (expects INVALID_OBJECT) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-26 21:48:29 +02:00
raphael	a2ebbf7c65	feat(client)+feat(daemon): eq.names, @eq.command, E90/E157 RPCs ... Python client: - eq.names.event.* / .alarm.* / .command.* / .var.* / .constant.* — autocomplete-able, typo-safe name lookup backed by the Describe RPC (lazy, cached; AttributeError on bad name with close-match hints) - @eq.command decorator — binds a handler by function name, validated against the equipment's real command set at decoration time - eq.report_substrate() — E90 wafer milestone reporting - eq.report_module() — E157 module state reporting (auto-create) Daemon (C++ service): - ReportSubstrate RPC — drives E90 location + processing FSMs - ReportModule RPC — drives E157 module FSM (auto-create on first report) - ack_from_outcome() helper — consistent Ack mapping for read_sync results Proto: SubstrateReport, ModuleReport, EquipmentDescription, SpoolFlushRequest, TerminalMessage; Describe, FlushSpool, SendTerminalMessage RPCs Tests: C++ FSM test (journey + ghost rejection + E157 illegal jump); interop coverage for names API and E90/E157 round-trip Docs: ch42 RPC table + Python example updated; ch16 daemon-path section added Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-26 21:43:07 +02:00
raphael	9876dd9b5a	feat(daemon): D10 carriers + E16 ops RPCs + stress test + virtual fab ... Completes the daemon's GEM300 surface and adds two new test tiers. D10 — E87 carriers: CarrierStore gains the HandlerSlot observer pattern (add_id/slot_map/access_handler). The daemon's id-observer forwards host S3F17 decisions onto the Subscribe stream as CarrierAction (PROCEED on a Confirmed transition, CANCEL on CancelCarrier); ReportCarrier drives the flow tool-side: WAITING creates the carrier + records the slot map, IN_ACCESS/COMPLETE advance the access FSM (INVALID_OBJECT on unknown, CANNOT_DO_NOW on an illegal transition). E16 — operations RPCs: Describe (full name inventory: variables/events/ alarms/commands/constants + device header), FlushSpool (purge or drain), SendTerminalMessage (S10F1 tool->host, honest CANNOT_DO_NOW when no host and stream 10 isn't spoolable). Stream responsiveness: Subscribe/WatchHealth poll at 100ms (was 500ms) so a cancelled stream frees its sync-server worker thread promptly — this was found by the new stress test, which hung under Subscribe churn at 500ms. Tests: - A randomized concurrent RPC stress case: 4 threads x 250 seeded ops (set/get/fire/alarm/control-state/describe + Subscribe churn), asserts no failed RPC and a still-responsive engine afterward; prints its seed; a strong TSan target. - A virtual fab (interop/virtual_fab.py + the `fab` compose service / tools/spawn_fab.sh): N daemons, each with a secsgem-py host AND a secsgem_client tool, driven by seeded random traffic with end-to-end invariant checks (set/get round-trips, event->S6F11 and alarm->S5F1 delivery, command->tool->completion). Verified green at N=3 (~150 ops/eq, all commands round-tripped, 0 violations). Wired into run_interop.sh (now 13 steps). Also fixes the CI break from the previous commit: the Python-client lane's test_values.py step lacked PYTHONPATH=clients/python (now step-level env). Two bugs found and fixed while building this, both mine from this batch: 1. carrier test hung on a CancelCarrier of a still-NotConfirmed carrier — a self-transition the FSM doesn't signal, so the observer never fired and the stream Read blocked forever. Fixed to cancel a Confirmed carrier; the NotConfirmed edge is documented as a known E87 limitation. 2. the 500ms stream poll above. Daemon suite 7 cases / 214 assertions; core 475 / 3097; virtual fab green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-26 15:05:13 +02:00
raphael	54626ceb6a	feat(daemon): Phase E — production hardening, complete ... Exposure: --grpc default flipped from 0.0.0.0 to 127.0.0.1 (the API is unauthenticated by design; auth belongs to the transport), Unix-domain- socket support (--grpc unix:///run/secs_gemd/api.sock = zero network surface), SECURITY.md documents the contract and ch42 gained a "Running it in production" section (which also documents the HSMS-SS single-session assumption). Graceful shutdown: SIGTERM/SIGINT land on an asio::signal_set on the io thread, which nudges grpc Shutdown with a 2s deadline (cancels open Subscribe/WatchHealth streams); Wait() returns on the MAIN thread, which stops the engine (rt->stop() joins the io thread, so it must not run on it). Exit 0, journal-safe, the in-code TODO is gone. --spool-dir added so host-bound events survive daemon restarts. Observability: --metrics serves Prometheus gauges secsgem_link_selected / secsgem_control_state / secsgem_spool_depth, wired via the Phase-0 add_link_observer/add_control_state_observer hooks + io-thread sampling. Deployment: deploy/secs_gemd.service — hardened systemd unit (DynamicUser, ProtectSystem=strict, StateDirectory for the spool, UDS for the API, TimeoutStopSec aligned with the graceful-shutdown window). Enforcement: tools/check_daemon_ops.sh proves all three operational claims (unix-socket gRPC accepts, all gauges present on /metrics, SIGTERM -> exit 0 + clean-stop log) — green; wired into tools/run_interop.sh (now 11 steps) and CI. CI python-interop lane also gained the pyclient and spool-restart steps, so every harness now runs in CI. TODO sweep: the shutdown TODO is fixed; the four remaining TODOs (nested list formats, C2-as-text, U8>2^63, CONNECTED link state) are deliberate deferred edge cases, each marked in code with context. Daemon suite re-verified green (175 assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-11 00:07:37 +02:00
raphael	b1772cfefd	feat(daemon): Phase D — GEM300 in-the-loop (jobs, recipes, EC changes) ... Semantics settled and documented: v1 is observe-and-report. The engine keeps acking S16/S3/S7/S2F15 from its FSM tables — exactly the behaviour both reference implementations validated — while the tool observes lifecycle events on the Subscribe stream and reports physical progress back. Gating stays the documented v2 deferred-reply item. Engine: two new store observers (HandlerSlot pattern) — RecipeStore fires (ppid, body) after an add (S7F3 downloads), EquipmentConstantStore fires (id, value) on ACCEPTED S2F15 writes only. Unit-tested. Daemon: the service registers PJ/recipe/EC observers (io thread; add_ observers coexist with register_default_handlers' primaries) and fans the new HostRequest variants out via push_request (fire-and-forget, no- buffering contract). ProcessJob carries action (Start->START, Resume-> RESUME, Paused->PAUSE, Stopping->STOP, Aborting->ABORT) + recipe + material bindings read store-side on the io thread. ReportProcessJob maps SETTING_UP ->SetupComplete, COMPLETE->ProcessComplete, ABORTED->AbortComplete via read_sync; PROCESSING is informational; unknown job => INVALID_OBJECT, table-rejected transition => CANNOT_DO_NOW. Carriers deferred (CarrierStore has no observer machinery; ReportCarrier stays UNIMPLEMENTED) — roadmap. Python client: on_process_job / on_recipe / on_constant_change decorators + report_job(job_id, state); ProcessJob dataclass exported. Tests: daemon suite 141 -> 175 assertions — the full in-process loop (S16F11 create -> tool setup -> S16F5 PJSTART -> stream ProcessJob with recipe+carriers -> ReportProcessJob(COMPLETE) -> FSM at ProcessComplete), rejection paths, S7F3 -> ProcessProgram, S2F15 -> ConstantChange with the configured name. Core 475/3097 (observer units). Live regression: daemon interop 20 checks + pyclient 13 checks still green against the running daemon. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 23:53:45 +02:00
raphael	b30443089f	feat(clients)+test(interop): C++ client + Java validation of the daemon (B7) ... B7 — the daemon's HSMS face under the Java reference: Dockerfile.server now bakes secs_gemd alongside secs_server (grpc deps in both stages), and secs4j_validate.sh gains TARGET=gemd to point the 55-check secs4java8 suite at the daemon instead. Result: 55/55 green. With secsgem-py already validating both faces, byte-identical GEM between secs_server and secs_gemd is now proven by both reference implementations, not inferred from shared code. CI runs the daemon target as an extra step (image layers shared). Second client — clients/cpp: a header-only C++ twin of the Python client over the same proto. eq.set("ChamberPressure", 2.5) with bare literals (integral/floating dispatch avoids variant ambiguity), get/fire/alarm/ clear, control_state/request_control_state/health, on("START", fn) + listen()/listen_async()/stop() with auto-CompleteCommand, SecsGemError carrying the daemon's message. cpp_mini_tool (~30 lines) mirrors the Python mini_tool. Tested end-to-end over real loopback TCP against the service inside secs_gemd_tests — now 4 cases / 141 assertions — including set/get round-trips, error text, alarm-by-name into the model, health, and the full HCACK-4 command loop with parameters. (Build note: two grpc-heavy TUs at -O3 OOM even at -j2 on Docker Desktop; built -j1. Known environment limitation, roadmap-documented.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 23:37:14 +02:00
raphael	4f3031aeb9	feat(example)+docs: pvd_tool on the modern stack; chapter 42 teaches the daemon path ... C9 — the flagship vendor example now demonstrates the intended integration shape. examples/pvd_tool/main.cpp: 1093 -> 570 lines. The 466-line hand-registered handler section and the hand-wired Server/Router/emit plumbing are gone, replaced by EquipmentRuntime + register_default_handlers (the example now serves all 56 handlers, up from its hand-picked 51) + commands.set_handler for the START-runs-the-recipe behaviour (was a hard-coded S2F41 router override). All domain logic — sensor simulator, recipe runner, alarm threshold monitor, EPT cycler, Prometheus gauges — unchanged. pvd's SVIDs 1/2 and CEIDs 400/401 match the roles: defaults, so the built-ins bind with no config change. Verified: builds clean, boots ("registered 56 handlers", config loaded, EPT cycling), HSMS :5000 accepts, metrics :9090 answers HTTP 200. logfn flushes per line so docker/CI logs are visible immediately. Writing project — new tutorial chapter docs/42_vendor_daemon_and_clients.md: why a daemon (the host-timer argument), the proto contract and the HCACK-4 command semantics, the Python client walkthrough, EquipmentRuntime + capability registration + roles:, the threading contract (posting API / read_sync / hooks-on-io-thread) and primary-vs-observer slots, and a which-tier-do-I-pick table. Indexed in 00_index Part 4. Refreshed the three spots that still described pvd_tool's old "51 handlers in ~460 lines" shape (ch35, ch41, pvd README) — drift killed in the same commit that made it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 23:31:22 +02:00
raphael	af1a159c59	docs: bring the documentation up to the daemon/client era ... A large gap had opened between the docs and the code: the README and INTEGRATION guide did not mention the gRPC daemon or the Python client at all (the entire vendor surface), ARCHITECTURE still described secs_server as the ~1200-line canonical wiring example (it is a ~110-line thin main over EquipmentRuntime), and test counts across six files were stale (445/2753 -> 473/3087 core + the separate 125-assertion daemon suite). - README: new "Integrating your tool (pick a tier)" section — Python client / any-language gRPC / embedded C++ — plus daemon tests and tools/run_interop.sh in the Testing section. - ARCHITECTURE: layer diagram gains the vendor-surface and EquipmentRuntime/default_handlers tiers; stale wiring row fixed. - INTEGRATION: three-tier chooser up front (this guide = the C++ tier). - ch30 tour: secs_gemd + secs_gemd_tests in the binaries table. - ch31: example alarm used a nonexistent `alcd:` field with bit 7 set (which the validator forbids) -> real `category:`/`name:` fields, and the roles: block documented. - ch35: handler-location note now points at default_handlers.cpp's 15 per-capability register_* functions. - ch40: built-artifacts list + sample output counts. - ch50: secsgem::gem runtime/default_handlers/handler_slot/name_index includes + new secsgem::daemon namespace section. - PROOFS: test-count table gains the runtime/handlers/daemon row so the tally adds up; daemon suite noted. VERIFICATION/COMPLIANCE counts. - interop/README: the one-command runner + the two daemon-track harnesses (daemon_interop, pyclient_interop). Audited via a docs-vs-code sweep (the audit itself under-reported: it validated counts textually; reality was 473/3087). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 23:18:31 +02:00
raphael	8686654b15	feat(client): the Python client — a GEM tool in plain Python (Phase C) ... clients/python: pip-installable "secsgem-client", pure Python (stubs pre-generated from equipment.proto, import made package-relative; no compiled extension, no SEMI knowledge, no C++ toolchain). The API the whole effort aimed at: eq = Equipment("localhost:50051") eq.set(ChamberPressure=2.5); eq["WaferCounter"] = 7 eq.fire("ProcessStarted", ChamberPressure=2.75) eq.alarm("chiller_temp_high"); eq.clear("chiller_temp_high") @eq.on("START") def start(cmd): ... # auto-CompleteCommand after return eq.listen(background=True) eq.control_state; eq.request_control_state("HOST_OFFLINE"); eq.health() Errors raise SecsGemError carrying the daemon's message ("no variable named ..."). bool checked before int in conversion (isinstance(True, int)). examples/mini_tool.py is a complete GEM tool in ~25 lines. PROOF — interop/pyclient_interop.py drives the PUBLISHED package (not raw stubs) against a live secs_gemd with secsgem-py as the fab host: 13 checks all green on first run — set/get round-trips, item syntax, SecsGemError on unknown names, control state, health, fire->S6F11 on the host's wire, alarm/clear->S5F1 with correct set bit, the full command loop (host S2F41 -> HCACK=4 -> @eq.on handler -> completion event back at the host), operator offline. Conversion layer unit-tested standalone; both wired into tools/run_interop.sh as the pyclient step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 22:57:55 +02:00
raphael	912304966f	refactor(gem): decompose default handlers per GEM capability + YAML role bindings ... register_default_handlers was a relocated app main(): one 1086-line function, all-or-nothing. It is now 15 per-capability registration functions along the lines GEM itself defines (S1F19): identification, equipment constants, clock, event reports, remote commands, trace/limits, spooling, alarms, exceptions, material tracking (E90/E116/E157), carriers (E87), recipes, object services (E39), jobs (E40/E94), terminal services. A sensor-class tool registers three functions instead of carrying carrier/job handlers it doesn't have; register_default_handlers composes all 15. Each function derives exactly the runtime aliases its handlers use (generated programmatically from the moved bodies with boundary/substitution guards — zero hand-retyping). Magic constants are gone: the control-state/clock SVIDs (were hardcoded 1/2) and the CJ Executing/Completed CEIDs (were 400/401) now come from a "roles:" block in equipment.yaml via EquipmentDescriptor, with historical defaults when absent, loader parsing, and validation (CEID roles must name declared events). The coupling is now visible in ONE file instead of silently split between YAML and C++ — the exact drift class this repo's spec-as-data philosophy exists to kill. Tests: capability subsetting, role-driven SVID refresh via S1F3, roles loader (shipped/custom/absent). Battery: core 473/3087 incl. the 53-handler conformance sweep, daemon 125/125, live GEM300 demo (client exit 0), daemon interop 20/20 vs secsgem-py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 22:44:04 +02:00
raphael	cf230d4119	chore(phase0): name validation, golden frames, daemon into library tree, TSan daemon lane ... Item 8a — ConfigValidator warns on non-identifier variable/event/alarm/ command names ([A-Za-z_][A-Za-z0-9_]*): language bindings expose names as kwargs/attributes, so 'Chamber Pressure' would be unusable in the planned Python client. Warning not error — the wire doesn't care. Tested (4 warning sites + good-name negative). Item 4 tail — golden frames for S5F1 (Binary ALCD / U4 ALID / ASCII ALTX) and a composed S6F11 (the production-critical report shape), bytes hand- computed from E5 encoding rules: external pins on message composition. Item 7 — equipment_service.hpp moved to include/secsgem/daemon/ (apps/ include-path hack removed) and a TSan daemon lane added locally + in CI. tools/tsan.supp suppresses races whose accesses sit entirely inside the UNinstrumented system libgrpc/libgpr/libabsl (epoll wakeups, absl Mutex GraphCycles bookkeeping); our frames stay fully checked. The lane earned its keep on first run: it caught a REAL threading-contract violation — a daemon test reading model stores from the test thread while the io thread serviced posted writes — fixed to use read_sync, exactly per the documented contract. Now TSan-clean under halt_on_error=1 in the full production threading shape. Suites: core 470/3068, daemon Release+TSan 125/125 each. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 22:28:33 +02:00
raphael	e6ee927900	feat(daemon): Subscribe command stream + CompleteCommand — the vendor loop closes ... The HCACK-4 contract, implemented end to end. For every YAML-declared command the service registers a forwarding handler (new HostCommandRegistry names()/spec() accessors): with a subscribed tool client the command is queued onto the Subscribe stream (id + name + params via from_item) and the host is answered S2F42 HCACK=4 immediately — never blocking the io thread or the T3 window; with NO subscriber the command takes its declarative YAML ack (the honest pre-daemon behaviour). Settled + documented in the proto: v1 is a firehose with no buffering/replay. CompleteCommand correlates the pending id (audit; unknown id => PARAMETER_INVALID). Side effects stay suppressed on HCACK-4 (router applies them only on Accept), so the completion event the TOOL fires is the host's real signal — exactly E30's intent. Tests (daemon suite 101 -> 124 assertions): a real S2F41 dispatched through the full default-handler router ON the io thread under run_async — HCACK 4 with subscriber + params on the stream, declarative Accept without, CompleteCommand known/unknown, fallback restored after unsubscribe. Interop (now 20 checks, all green): the complete conformant loop against the secsgem-py reference host — S2F41 START -> S2F42 HCACK=4 -> tool receives Command(name=START, id=1) -> CompleteCommand -> FireEvent -> host receives S6F11. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 20:27:18 +02:00
raphael	1da56f973f	feat(daemon): alarms by name + RequestControlState + WatchHealth (Phase A complete) ... A2 — alarms: optional 'name:' on alarm config (a LOCAL key — SEMI only defines numeric ALID + freetext ALTX; field appended last so existing {id, text, category} brace-inits compile unchanged), parsed by the loader, checked by the validator, shipped in equipment.yaml. SetAlarm/ClearAlarm RPCs resolve config name OR stringified ALID via a constructor snapshot. A3 — control state + health: RequestControlState fires operator events on the io thread (read_sync) and reports what the E30 table actually did — ACCEPT iff the equipment landed in the requested state, CANNOT_DO_NOW naming the actual state otherwise (the shipped table has no operator path to EquipmentOffline; the test pins that honesty). ATTEMPT_ONLINE is rejected as transient. WatchHealth streams an immediate snapshot then pushes on link/ control-state changes via service observers (add_link_observer + add_control_state_observer — the HandlerSlot work paying off), spool depth sampled at the 500ms poll; ends on cancel or engine stop. Tests: daemon suite 61 -> 101 assertions (alarm lifecycle by name/id/unknown, WatchHealth initial + change push, all four RequestControlState semantics); loader test for the alarm name (present + absent fallback); core 467/3055. Interop now 15 checks incl. gRPC SetAlarm -> host receives S5F1 ALCD=0x84 ALID=1, and RequestControlState(HOST_OFFLINE) -> GetControlState confirms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 19:47:31 +02:00
raphael	83593bb508	docs: refresh stale roadmap status rows (GetVariables shipped, harnesses automated) ... Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 19:34:13 +02:00
raphael	1daf120431	feat(daemon): GetVariables + read_sync — the standard mutable-read pattern ... EquipmentRuntime::read_sync establishes THE pattern for reading mutable engine state from gRPC/binding threads (Phase 0 item 6): post the read onto the io thread (the model's single owner), wait on a future with a deadline, nullopt => UNAVAILABLE at the RPC edge. Always truthful, no cache to invalidate; milliseconds are irrelevant at SECS rates. GetVariables: name resolution against the service snapshot (empty query = all; unknown name => INVALID_ARGUMENT naming the offender), values read via read_sync, converted by the new from_item reverse conversion (single-element numeric arrays => scalars, multi-element => List; Boolean/Binary/text per format; C2-as-integer and U8>2^63 wrap documented as TODOs). Tests run the engine in run_async — the daemon's PRODUCTION threading mode, previously untested — and round-trip through both conversions: SetVariables (declared-format write) then GetVariables (read) over a real in-process channel. Daemon suite 41 -> 61 assertions. daemon_interop.py gains a live GetVariables round-trip check vs the running daemon (verified green). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 19:33:50 +02:00
raphael	b0a4c331cf	test(gem): table-driven conformance sweep over the default handler set ... One ordered in-process scenario drives 53 of the 56 registered handlers through Router::dispatch — S1 identification/comms/control, S2 ECs/clock/ event-config/commands/trace/limits/spool, S5 alarms+exceptions, S6 reports, S7 recipes, S10 terminal, S14/S16 E39+E40/E94 jobs, S3 carriers — asserting every reply is the paired (stream, function+1) with a body, plus targeted state checks (OnlineRemote after S1F17, PJ exists after S16F11, HostOffline after S1F15) and the Router's SxF0 abort fallback for unregistered W=1 primaries. Same flow secs_conformance runs over a live socket, but cheap enough for every build; closes the '56 handlers, 4 direct tests' gap from the design review. Also seeds message-level golden frames: S1F13's body pinned to bytes hand-computed from the E5 encoding rules — an external check on message composition, not our codec validating itself (TODO: S5F1, composed S6F11). Suite: 466 cases / 3052 assertions (+236), all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 19:26:28 +02:00
raphael	42044e92e2	ci(interop): one-command external-validation suite + CI lanes for the daemon ... tools/run_interop.sh runs ALL nine validation steps with a PASS/FAIL summary: build, unit (464), daemon-unit (41), secsgem-py host vs server (31 checks), secs_conformance (47), gRPC+secsgem-py daemon bridge, spool persistence across restart, tshark HSMS dissector, secs4java8 (55 checks). Verified green end-to-end. The unit suite is partly self-referential (our parsers validate our builders); these external validators are the real oracle — now they run with one command instead of by hand. Two bugs found by running it: unbounded ninja at -O3 OOM-kills cc1plus in memory-constrained Docker VMs (build with -j 2) and bash-3.2 lacks negative array subscripts. CI: grpc deps added to the build job so secs_gemd + secs_gemd_tests build and RUN in CI (previously the daemon silently dropped out — now fails loudly if missing), plus a python-interop lane running py-host/conformance/daemon harnesses against localhost in one container (no docker-in-docker). Service hardening while in there: reject proto Values with no kind set at the RPC edge (previously silently became ASCII ""), TODO markers for list element formats and daemon graceful shutdown. New tests: unset-Value guard + a property test iterating ALL configured variables via gRPC asserting each keeps its declared SECS-II format (daemon tests 16 -> 41 assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 19:08:37 +02:00
raphael	941f9ef458	docs: add Phase 0 (structural debts from design review); fix CompleteCommand contract comment ... Phase 0 captures the 2026-06-10 review: multi-observer callbacks (done for the critical three), CI for the interop/conformance harnesses (the unit suite is partly self-referential; the external validators are the real oracle), table-driven handler conformance + message-level golden frames, register_default_handlers decomposition per GEM capability + YAML role bindings for today's magic constants, the post+future mutable-read pattern, service relocation + TSan run_async daemon test, identifier-safe name validation. CompleteCommand's proto comment described the rejected blocking model; it now states the settled HCACK-4 contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 18:57:53 +02:00
raphael	8a48ffeed4	feat(gem): multi-observer state-change handlers via HandlerSlot ... The single-slot set_*_handler pattern was a structural blocker, hit twice: the daemon could not observe control-state changes because register_default_handlers owns the slot, forcing GetControlState to read the FSM cross-thread (a data race), and blocking WatchHealth and the Subscribe stream's ControlStateChange variant. HandlerSlot<Args...> keeps a primary slot with exact legacy semantics (set_ replaces — one existing test depends on replacement) plus an append-only observer list (add_) that survives set_ calls. Fire sites are textually unchanged (operator bool / operator() / assign-from-function). Applied to ControlStateMachine + ProcessJobStore + ControlJobStore (the roadmap-critical three; the remaining single-slot classes follow the same 3-line pattern as needed). EquipmentRuntime gains an atomic control-state mirror registered as an observer — control_state() is now safe from any thread, retiring the GetControlState race — plus add_control_state_observer and add_link_observer (selected/closed fan-out), the hooks WatchHealth and Subscribe need. Tests: observer ordering, set-replaces-primary-but-observers-survive, observers-without-primary, PJ-store coexistence, and the runtime scenario that was previously impossible (mirror + observer + default-handlers set_). Core 464/464 (2816 assertions), daemon 16/16, live GEM300 demo passes with single-fire control-state transitions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 18:57:53 +02:00
raphael	b067a76b80	docs: rewrite daemon roadmap as ordered plan with known-issues audit ... Status table brought current (format-aware daemon, secsgem-py interop), the stale Layer-0 section replaced, and the path to an excellent GEM300 repo laid out as ordered phases A–F: finish universal RPCs, the Subscribe command stream (HCACK-4 design written down as the implementation contract), the Python client package, GEM300 job/carrier in-the-loop, hardening/CI, and the fab-acceptance track. Known-issues section records what the audit found (GetControlState enum race + why the state-change-handler slot can't be reused, missing alarm name key, pvd_tool predating set_handler, manual interop harnesses, TSan gap). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 18:35:53 +02:00
raphael	99bfa794fc	fix(daemon): honour declared SECS-II formats + make service thread-safe ... Audit fixes for two real bugs in the gRPC service: 1. Format compliance: to_item() wrote F8/I8 regardless of the variable's declared wire format, so values contradicted the S1F11/S1F21 namelists (ChamberPressure is F4, WaferCounter U4; the interop trace showed <F8 2.5> on the wire). Conversion now targets the declared format — verified end-to-end: secsgem-py now receives <F4 2.5> in S6F11. 2. Thread safety: gRPC handler threads called resolve_variable/resolve_event, copying live store entries (including Item values) while the io thread mutates them. The service now snapshots the immutable name->id/format maps at construction (before run_async, per the documented ordering); all writes already post to the io thread. Remaining known narrow race (GetControlState enum read) documented in DAEMON_ROADMAP. Also: drop a stale tools/run_interop.sh reference from docker-compose.yml. Tests: daemon in-process 16/16 (new F4/U4 format assertions), core 459/459, secsgem-py interop green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 18:35:53 +02:00
raphael	92afbd2a37	docs: record secsgem-py daemon interop in roadmap ... Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 18:24:03 +02:00
raphael	3d72e50b65	test(interop): daemon end-to-end vs secsgem-py reference host ... daemon_interop.py drives a running secs_gemd through BOTH faces at once: a gRPC tool client and a secsgem-py active host. Proves the gRPC<->HSMS bridge against a reference GEM implementation, not just in-process: - gRPC GetControlState agrees with the HSMS-driven control state - gRPC SetVariables(ChamberPressure=2.5) + FireEvent(ProcessStarted) makes the host receive S6F11 CEID 300 carrying 2.5 (value flowed gRPC -> engine -> HSMS -> host) - unknown variable/event names rejected at the gRPC edge Mirrors the existing host_vs_cpp_server.py pattern. New 'gemd' compose service (HSMS :5000 + gRPC :50051); interop image gains grpcio/grpcio-tools (proto stubs generated at runtime, flat to avoid the secsgem package-name clash). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 18:23:47 +02:00
raphael	dd288eb2ac	docs: update daemon roadmap — gRPC toolchain done, secs_gemd serving ... Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 18:07:38 +02:00
raphael	cb85199f49	feat(daemon): FireEvent + event name resolution + in-process gRPC tests ... - name_index: add resolve_event(name) -> CEID (unit-tested). - equipment_service.hpp: extract the gRPC service + value/state conversion into a shared header; add FireEvent (optional per-fire variable values, then trigger the collection event by name). secs_gemd slims to main(). - test_daemon_service: real in-process gRPC integration test (client stub -> service -> EquipmentRuntime) proving SetVariables lands in the model, GetControlState reports the state, FireEvent and unknown-name paths behave. Separate secs_gemd_tests target (links grpc++/proto), gated on the daemon. Core suite 459/459 (2799 assertions); daemon gRPC tests 15/15. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 18:07:12 +02:00
raphael	fc898f8410	feat: EquipmentRuntime engine owner + secs_gemd gRPC daemon ... Extract the SECS/GEM engine wiring out of the secs_server app into a reusable class, and stand up a language-agnostic gRPC daemon on top so a tool's software (any language) can drive the equipment without linking C++ or knowing SEMI. Foundation for replacing a vendor's SECS/GEM server. Engine reuse: - EquipmentRuntime (include/secsgem/gem/runtime.hpp, src/gem/runtime.cpp): owns io_context, passive Server, model, control-state machine, Router; thread-safe outbound API (set_variable/emit_event/set_alarm/clear_alarm), on_command hook, deliver_or_spool, run()/run_async()/poll()/stop(). - register_default_handlers (src/gem/default_handlers.cpp): the 56 GEM handlers + domain emitters, relocated from secs_server so the app and the daemon speak byte-identical GEM. secs_server.cpp reduced ~1270 -> 113 lines. - name_index.hpp: resolve_variable(name) -> VID (the name->id binding layer). Daemon (apps/secs_gemd.cpp, proto/secsgem/v1/equipment.proto): - runs the engine + HSMS link on a background thread; serves the gRPC Equipment service. Increment 1: SetVariables (name-resolved, plain value->Item) and GetControlState. proto carries the full v1 surface (universal + carrier/recipe/job tiers); remaining RPCs + the Subscribe command stream are next (docs/DAEMON_ROADMAP.md). - CMake: opt-in SECSGEM_DAEMON, protoc/grpc_cpp_plugin codegen, gracefully skipped where protobuf/grpc++ are absent. Dockerfile gains the grpc deps. Tests (proof): test_runtime, test_default_handlers (S1F1->S1F2, S2F41->hook), test_name_index. Full suite 458/458, 2795 assertions; live server<->client GEM300 demo still passes on the refactored server. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 18:01:16 +02:00
raphael	4b4b2ac690	docs: correct drifted and fabricated APIs in chapters 13/17/35/51 ... An audit of doc code blocks against the real headers found APIs that do not exist in the codebase, presented as authoritative walkthroughs: - ch35 (dispatch): an entirely fabricated callback architecture — HostCommandRegistry::set_emit_ceid_handler, CommandOutcome, emit_ceids. Rewritten to the real Spec/Result/dispatch + the new set_handler hook. - ch13 (E30): wrong store names — EventStore/ReportStore -> EventReportSubscriptions, SvidStore -> StatusVariableStore, AlarmStore/AlarmDispatcher -> AlarmRegistry, ClockStore -> Clock, TerminalServiceStore -> (no store), in both the capability tables and the worked S2F33 example. - ch17 (E116): EptStore/seconds/bucket_ -> EptStateMachine/milliseconds/buckets_. - ch51 (extending): stale host-command handler -> the real set_handler signature. Verified clean by grep: no fabricated symbols remain in docs/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 18:00:58 +02:00
raphael	0090791968	feat(gem): add host-command behaviour hook to HostCommandRegistry ... Host commands were declarative-only: dispatch() returned the YAML-defined HCACK plus side effects, and ignored the command parameters entirely (the param list was a commented-out argument). Equipment could acknowledge a command but never run anything in response — the pvd_tool example worked around this by hard-coding behaviour in a C++ router handler. Add set_handler(rcmd, fn): a registered handler receives the live CPNAME/ CPVAL parameters and returns the HCACK, overriding the declarative default. Live on S2F41/F21/F49 via the shared dispatch(). No handler => byte-for-byte the previous declarative behaviour. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 18:00:44 +02:00
raphael	dae6bfd747	docs: streamline tone across reference docs ... Tone pass across the non-tutorial markdown — README, PROOFS, ARCHITECTURE, BENCHMARKS, COMPLIANCE, FAQ, MES_INTEROP, SECURITY, and interop/README. Three patterns came out: - Bug-history war stories ("Past interop sweeps surfaced…", "What these harnesses caught: 1. Strict U-width parsing…"). - Chat-with-reader framing ("Don't skip TLS unless…", "Treat as a punch list", "If you're running in a pod…", "Misconfiguration incidents drop dramatically"). - Self-referential narration ("we ship", "our codec", "the codebase's most-tested layer", "three orders of magnitude above fab load", "the gift that keeps giving"). README also drops the standalone ThreadSanitizer subsection under Build details (now a single line under the new Testing section). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-10 00:00:06 +02:00
raphael	d63c92166d	docs: rewrite VERIFICATION.md to describe shipped validators ... Previously written as a forward-looking plan ("Plan: (1) KAT → (2) tshark → (3) secs4j → (4) libFuzzer", "Effort: ~3 hours", "Survey step (do this first)"). All four validators have shipped — test_e5_kat.cpp, interop/secs4j/Secs4jHostHarness.java, interop/tshark_validate.sh, apps/fuzz_*.cpp. Rewritten as documentation of what's there: file paths, CI job names, actual result numbers. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 23:59:54 +02:00
raphael	0355c73211	docs: refresh stale file paths after store/ reorg + gen_messages rename ... generate_messages.py → gen_messages.py and several gem/ headers moved under gem/store/ (carrier_store.hpp → store/carriers.hpp, etc.); e84.hpp split into e84_state.hpp. The guided-tour chapters still pointed at the old paths — relink them so the deep-link footnotes resolve. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 23:23:42 +02:00
raphael	fee82d88c9	ci: self-contained secs_server image for secs4j interop ... The harness previously bound the source tree into a compose service and built inside it. That breaks under docker-in-docker (gitea-act, GitHub Actions runners with /var/run/docker.sock mounted) because bind-mount sources resolve against the *host* daemon's filesystem, not the runner container's. Now Dockerfile.server bakes a Release secs_server into its own image, and secs4j_validate.sh wires server and harness together on a dedicated bridge — no volumes needed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 23:23:34 +02:00
raphael	31f908e1bf	docs: chapters 40, 41, 50, 51 — Operations + Reference (series complete) ... Last four chapters of the guided tour. 40 — Building, running, the demo. Docker prerequisites, the build flow, what each binary is for, running the 24-transaction demo flow annotated step by step. Running the 4 external-validator sweeps + the libFuzzer pass. Inspecting the demo with tcpdump and tshark. Reading source while running as the recommended learning workflow. 41 — Integration: hardware, MES, production. Four-phase tour: wiring sensors / recipe engine / alarms / E84 GPIO; talking to a real MES with the day-1 punch list + commercial-MES quirks (Wonderware S2F21, Camstar Linktest cadence, etc.); production hardening (nftables / stunnel / minisign / persistence layout / monitoring / runbook); performance envelope + memory footprint + capacity planning. Pointers to the long-form INTEGRATION.md / MES_INTEROP.md / SECURITY.md / BENCHMARKS.md. 50 — API + message catalog + YAML schemas reference. Namespace-by- namespace table of public symbols (secs2, hsms, secsi, gem, config, metrics) with brief descriptions. Stream-by-stream message catalog reference (S1, S2, S3, S5, S6, S7, S9, S10, S12, S14, S16). YAML schema reference for messages.yaml + the three state-table files + equipment.yaml. 51 — Extending the codebase. Seven recipes ordered from no-code to substantial: new SVID/DVID/ECID (YAML only), new CEID with reports (YAML only), new host command (YAML + optional handler), new control- state transition (YAML only), new SECS-II message (YAML + handler), new store (header + tests), new persistence backend (drop-in vs pluggable trade-off). Each recipe has the actual mechanical steps, the test pattern, and pointers to the chapter that explains why it works. Index updated to mark all 24 chapters published. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 20:28:21 +02:00
raphael	cae98d9a7d	docs: chapters 30–36 — the codebase (Part 3 complete) ... Seven chapters walking the implementation top-to-bottom. 30 — Repository tour. Top-level layout, directory by directory. The eight built binaries. The dependency graph from TCP socket up through EquipmentDataModel. CMake's role. Test layout. 31 — Spec-as-data and codegen. Why the design choice fits SECS/ GEM specifically. The five YAML files: messages catalog, control/PJ/CJ transition tables, equipment dictionary. How tools/gen_messages.py turns messages.yaml into typed C++ at build time. The --validate-config multi-error validator. How to add a new SVID / CEID / host command / state / message without C++. 32 — Stores and the data model. What a store IS (records + API + change handler + optional persistence). Every store in the codebase mapped to the SEMI standard it serves (table of 21). EquipmentDataModel as plain composition + cross-store convenience methods (vid_value, compose_reports_for). The no-locks single- threaded contract. How to add a new store. 33 — Transport. hsms::Connection read path (length+payload async chain), write path (queue + one outstanding write), timer model (5 steady_timers + per-request T3). The asio executor / strand model and why it's the right shape. secsi::Protocol as the IO- free FSM with Action / Event variants; secsi::TcpTransport as the asio adapter. Pattern repeats for E84 + GEM comm-state. 34 — Codec and SML. The four files (170 + 30 + 52 + 32 lines of header, 229 + 220 lines of impl). Item variant storage layout (11 alternatives, 16 formats, shared storage where E5 permits). encode_into recursion; decode_at with bounds checks throwing CodecError. Message wrapper. SML printer + try_parse_sml + why SML round-trips Items but not necessarily bytes. 35 — State machines and dispatch. gem::Router as a typed (stream, function) dispatch table. How an S2F41 round-trip walks through parser → store dispatch → side-effect → CEID emission → S6F11 build → spool-aware deliver. The 11 FSMs all sharing the same three-property shape (pure data table + pure FSM + observer pattern). CEID cascading from FSM transitions to wire bytes. 36 — Persistence, validation, metrics. Which 7 stores have file journals + why the others don't. Per-record file pattern (atomic rename, partial-write safe). Schema versioning + multi-version read. Multi-error YAML validator (--validate-config) + cross-file reference checks. Prometheus registry + HTTP exporter + worked metric patterns from the PVD example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 20:23:05 +02:00
raphael	40df3067a4	docs: chapters 14–19 — GEM 300 standards (Part 2 complete) ... Six more chapters finishing Part 2. Together with chapters 10–13 they document every SEMI standard this codebase implements. 14 — E40 + E94: process jobs (8-state lifecycle, S16F11/F5/F7/F9 on the wire) and control jobs (CJ wraps PJs with batch policy, S14F9/S16F27 messages). Worked cascade showing how CJSTART propagates through the PJ FSM and triggers S6F11 CEIDs at each transition. 15 — E87 carriers: three orthogonal sub-machines (CarrierID, SlotMap, CarrierAccess) per carrier and three more (Transfer, Reservation, Association) per load port. S3F17 CarrierAction strings + CAACK codes, S3F19 SlotMap verify, the 5-state slot encoding, multi-port concurrency. 16 — E90 + E157: substrate tracking via three orthogonal axes (STS / SPS / SubstrateIDStatus) and module process tracking (NotExecuting / GeneralExecuting / StepExecuting / StepCompleted). End-to-end PVD example showing E40 + E157 + E90 transitions cascading into CEIDs. 17 — E116 + E120 + E39: equipment performance time-buckets across six states, common equipment model object hierarchy, S14F1/F3 GetAttr/SetAttr as the uniform wire access for any object type across multiple standards. 18 — E84 parallel I/O: ten signal lines, the 9-state handshake FSM, the three TA1/TA2/TA3 timing-critical timers, why a physical handshake gets modeled in software (testability, timer enforcement, CEID emission, multi-port concurrency), the pure-FSM + asio-adapter split. 19 — E42 + E148 + S5F9–F18: formatted recipes (S7F23/F25 typed PPBODY), time synchronization with 16-char + 14-char accepted on set, exception recovery as a persistent multi-step host-supervised FSM (Posted → Recovering → Cleared with abort/retry). Revisits the auto-S9 family and contrasts S9 (transport) vs S5F9 (application). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 20:14:42 +02:00
raphael	858ca22975	docs: chapters 11–13 — HSMS, SECS-I, GEM ... Three more chapters of Part 2: 11 — E37 HSMS. 4-byte length prefix + 10-byte header (R-bit + session id + W-bit + stream + function + PType + SType + system_bytes), the 9 SType control messages, the NOT-SELECTED → SELECTED state machine, T3/T5/T6/T7/T8 with what each one bounds, the auto-S9 paths (S9F1/F3/F5/F7/F9/F11), HSMS-SS vs HSMS-GS, the asio single-threaded contract. 12 — E4 SECS-I. Half-duplex line turnaround (ENQ/EOT/ACK/NAK), the 10-byte block header bit-packing (R-bit / W-bit / E-bit / system bytes), the 244-byte block cap and multi-block split/assemble, the event-driven IO-free FSM with its Action / Event variants, T1/T2/T3/T4 with semantics + defaults, master/slave contention. Notes the deferred asio serial_port adapter; explains why this chapter matters even for HSMS-only readers. 13 — E30 GEM. Disambiguates the three state machines (HSMS transport vs GEM communication vs GEM control), walks the comm-state FSM (DISABLED → WAIT-CRA → COMMUNICATING with T_CRA / T_DELAY) and the control-state FSM (5 states + the YAML transition table). Lists every Fundamental and Additional capability with its messages, code locations, and store assignments. One worked Event-Notification scenario tracing seven on-wire steps to their EquipmentDataModel internals. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 20:07:31 +02:00
raphael	338d0b974d	docs: chapter 10 — E5 SECS-II data items and encoding ... Opens Part 2 (the standards in detail). Walks the entire SECS-II encoding from first principles: the mental model (every value is one Item; a List is a recursive Item), the format-byte arithmetic (6-bit format code, 2-bit length-byte-count), the 14 format codes, length bytes 1/2/3 (with the 16 MiB cap), big-endian everywhere, the difference between byte-count (scalars) and child-count (lists). Then walks every format with worked hexdumps: empty list, nested list, ASCII with length-byte boundary crossing, Binary vs Boolean, U1/U2/U4/U8, signed integers with two's-complement edges, F4 / F8 with NaN / ±Inf / −0.0, JIS-8, C2 Unicode. Then the codebase mapping: Format enum, Item variant storage layout, encode_into / decode_at recursion, SML printer/parser, the identifier-wildcard rule (SEMI allows U1/U2/U4/U8 interchangeably for ID fields) with the messages_helpers::any_unsigned_first<Out> helper that closes the leniency contract. Closes with the well-defined CodecError conditions, what the codec deliberately doesn't reject (unknown format codes), and pointers to chapter 31 (codegen) and chapter 11 (HSMS) as the next dependencies above the codec. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 19:57:18 +02:00
raphael	5fec47ad02	ci: bake secs4j harness into image instead of bind-mounting ... Second secs4j-interop CI failure: ensuring secs4j-interop image is built... compiling Secs4jHostHarness.java... error: file not found: Secs4jHostHarness.java FAIL: javac The script bind-mounted $PWD/interop/secs4j into /work inside the container so it could javac the harness at runtime. That works locally where docker daemon and script share a filesystem, but fails in CI: the act runner runs the workflow inside a container, the docker socket is mounted from the host, and the daemon interprets bind-mount paths against the host filesystem — where $PWD/interop/secs4j doesn't exist. Result: empty /work, javac errors, job fails. Fix: COPY Secs4jHostHarness.java into the image and javac it at image build time. The script just runs the container — no bind mount, no docker-in-docker mount path translation, works in CI and locally. Verified locally with a fresh image rebuild: 55/55 checks pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 19:54:32 +02:00
raphael	fc3422a4a9	docs: move root .md files into docs/ + update every reference ... Picks up the file renames that landed alongside the previous commit and fixes everything that pointed at the old root locations: - README.md doc-map updated: every entry now points at docs/X.md, with a new "docs/" lead entry pointing at the guided-tour index. - README inline cross-refs (ARCHITECTURE / INTEGRATION / SECURITY / BENCHMARKS / MES_INTEROP / PROOFS) repointed to docs/. - README "Interop" section rewritten — used to mention only secsgem-py; now covers all four external validators (secsgem-py 31 / secs4java8 55 / tshark 69 frames / libFuzzer 200 k+ runs) with a one-line summary each, plus pointers to interop/README.md and docs/VERIFICATION.md. - README "Deferred follow-ups" cleaned: dropped the explanatory "Listed here so reviewers don't go looking for them in COMPLIANCE.md and find an 'out of scope' entry that sounds defensive" sentence — the section header speaks for itself. - docs/00_index.md "Where the rest of the docs live" table: dropped every `../` prefix since the docs are now siblings. - docs/01_what_is_secs_gem.md PROOFS reference updated to sibling. - docs/02_the_cast.md INTEGRATION + MES_INTEROP refs updated to siblings; dropped the stale "at the repo root" wording. - interop/README.md: VERIFICATION + PROOFS refs updated to ../docs/X.md; stale "~24 + 4 checks" updated to 31 (matches PROOFS.md and README). - examples/pvd_tool/README.md: every doc cross-ref now points at ../../docs/X.md. - Source / data / CI comments mentioning doc names (e.g. "INTEGRATION.md §3", "COMPLIANCE.md gap") rewritten to "docs/INTEGRATION.md §3" etc. — affects 9 files across include/, apps/, tests/, data/, examples/, .gitea/workflows/. Verified: full build under docker passes, 445/445 test cases pass, 2 753/2 753 assertions pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 19:36:27 +02:00
raphael	60fa164626	docs: chapters 02 + 03 of the guided tour (Part 1 complete) ... 02 — The cast of characters: equipment, EAP, MES, fab planner, AMHS, operator. Who initiates which conversation, why the equipment is the passive side of HSMS by convention, how the AMHS handshake is out-of-band relative to SECS. Cross-references the relevant namespace and test files for each actor. 03 — Vocabulary + a wafer's journey: follows one 300 mm wafer end-to-end through a fab and labels every SECS message and acronym that fires. Introduces SVID / DVID / ECID / CEID / RPTID / ALID / PPID / MDLN / SOFTREV / HCACK / ALCD / OFLACK / CAACK / SMACK / etc. in context rather than as a list. Includes one-screen reference tables for the remaining acknowledge codes, T-timers in all four contexts (HSMS / SECS-I / E84 / E30 communication state), and a stream-by-stream summary. Part 1 (Foundations) of the guided tour is now complete — a reader who reads chapters 01–03 can describe the protocol stack, identify the actors, and recognise every acronym they'll meet in Part 2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 19:35:43 +02:00
raphael	bc54de7711	ci: secs4j-interop bootstrap resilient to runner image variant ... CI log showed: Run export DEBIAN_FRONTEND=noninteractive apt-get: command not found Failure - Main Bootstrap (node + git) exit status 127 The secs4j-interop job runs on the bare runner (not inside a `container:`) because it needs the host's docker socket to run `docker compose up -d server`. The runner image isn't fixed across deployments — catthehacker/ubuntu has apt-get, but a minimal node image doesn't. The old script hard-coded `apt-get` and exit 127'd on anything else. New bootstrap: - Checks what's already on PATH (git, node, docker). If all three are present, exits 0 — most act-runner images come pre-loaded. - Otherwise picks the right package manager (apt-get or apk) and installs only the missing pieces. - Errors out with a useful message if neither package manager exists, instead of failing on a missing command. Also updates the inline comment that still said "20 checks" — actual is 55 (matches the count in README / PROOFS.md / COMPLIANCE.md). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 19:17:18 +02:00
raphael	01acac97d4	docs: start guided-tour tutorial series under docs/ ... A linear teach-from-zero tutorial that walks both SECS/GEM as a protocol family and this codebase as an implementation. Each chapter explains a SEMI concept and shows where it lives in code, so a reader builds a mental model of the standards and the repository simultaneously. Structure (24 chapters across 5 parts): - Part 1 (3 ch) — Foundations: what SECS/GEM is, the cast of characters, vocabulary + a wafer's end-to-end journey - Part 2 (10 ch) — Standards in detail: E5, E37, E4, E30, E40+E94, E87, E90+E157, E116+E120+E39, E84, E42+E148+S9 - Part 3 (7 ch) — Codebase: repository tour, spec-as-data + codegen, stores, transport, codec, state machines, persistence - Part 4 (2 ch) — Operations: build/run/demo, integration - Part 5 (2 ch) — Reference: API + messages + YAML, extension guide Published in this commit: - 00_index.md — guide layout, audience map, reading paths, conventions, status table - 01_what_is_secs_gem.md — the N×M integration problem, what SECS vs. HSMS vs. GEM each actually refer to, the GEM 300 suite, the transport→message→behaviour layering, where each layer lives in this codebase, an end-to-end S2F17/F18 example Chapters publish iteratively from here. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 19:16:35 +02:00
raphael	b01dedfaa5	docs: drop COMPLIANCE §8 "out of scope" and broaden §7 to all 4 validators ... §8 was carrying two items that neither read as "deliberately out of scope" nor matched the framing of the section: - Equipment Processing States — E30 §6.3 explicitly leaves concrete states tool-defined. The framework ships the ControlTransitionTable engine and YAML loader; vendors supply IDLE/SETUP/READY/EXECUTING. That's a design choice, not a gap. §3 line 94 already documents it. - Serial-port wiring for SECS-I — the FSM is implemented and tested end-to-end over TCP; only the asio serial_port adapter is missing. That's deferred, not out of scope. §1a line 64 already lists it with status ⬜. So §8 is dropped, §9 renumbers to §8, and the deferred follow-up gets its own short section in the README so customers know it's tracked without sounding defensive. §7 used to be titled "Interoperability with secsgem-py 0.3.0" and mentioned only that one external implementation. We now have four external validators (secsgem-py + secs4java8 + tshark dissector + libFuzzer), so the section is renamed "Interoperability with external implementations" and broadened to cover all of them with their actual check counts. Stale "24 named checks" updated to the current 31; "three consecutive clean runs" line dropped as audit-language no longer earning its keep now that it's a CI step. FAQ's "What's not implemented?" answer rewritten to point at the README "Deferred follow-ups" section and COMPLIANCE §8 (new numbering), with a brief note explaining that Equipment Processing States are spec-by-design tool-defined. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 19:16:21 +02:00
raphael	c8e8e80735	secs_server: relative-path defaults so the binary runs outside docker ... Previously --config / --state-table / --pj-state-table / --cj-state-table defaulted to /app/data/..., which only resolves inside the docker image. A host build run from the repo root errored out unless every flag was passed explicitly. Switch to data/equipment.yaml (and siblings) relative to CWD — docker still works because WORKDIR=/app puts /app/data/... at the same relative location, and host builds run from the repo root resolve to <repo>/data/.... Existing callers that pass explicit paths (the proof commands, tshark_validate.sh, secs4j_validate.sh, docker compose) are unaffected. Verified --validate-config under docker still finds all four YAMLs and the tshark proof still passes (69 frames, 0 malformed). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 19:00:45 +02:00
raphael	b031f057af	docs: customer-ready sweep + README restructure + tshark CI fix ... Audit pass over the public-facing surface so a customer can read it end-to-end without tripping on stale numbers or self-contradictions. README + docs accuracy: - Test counts 426 → 445, assertions 2 557 → 2 753 (verified via doctest run); E5 row was missing test_e5_kat (19 cases) - Interop checks 24 → 31, COMPLIANCE.md message count 149 → 164, COMPLIANCE.md "291 cases / 1515 assertions" → 445 / 2 753 - README "60+ test IDs" for MES_INTEROP → actual 59 - PVD example counts: 32 SVIDs/17 CEIDs → 29/21, "~40 handlers in ~200 lines" → 51 in ~460, "~700 lines" → ~1,100; main.cpp header table-of-contents resynced with the actual 7 sections Out-of-scope honesty (COMPLIANCE.md §8 + FAQ.md): - Removed HSMS-GS (was both ✅ implemented in §1 and "out of scope" in §8; INTEGRATION.md §7 documents using it) - Removed multi-block SECS-I (split_message/assemble_message exist with 4 dedicated tests) - Added serial-port wiring as the genuine open ⬜ item — FSM is tested end-to-end over TCP; only the asio serial_port glue is deferred - COMPLIANCE.md intro now lists E42 and notes "E37 (SS + GS)" README restructure: - Moved the 8-command proof table and per-standard test-coverage table to a new PROOFS.md (72 lines) - README now leads with what / Quick start / Documentation map, then a one-paragraph "How it's proved" linking to PROOFS.md - Updated cross-refs in FAQ.md, GLOSSARY.md, VERIFICATION.md, and interop/README.md to point at PROOFS.md CI fix — tshark-dissector job: - interop/tshark_validate.sh hardcoded /app/build/secs_server etc. which only works inside the docker image. Now derives ROOT from the script's own location and accepts BUILD/SERVER/CLIENT/DATA env overrides, so CI can run it from the workspace dir - Verified still passes in docker (69 frames, 0 malformed) .gitignore: - Added build-fuzz/ and build-tsan/ (were showing as untracked) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 18:59:17 +02:00
raphael	6aa4427186	docs: worked PVD-tool vendor example ... A fictional Physical Vapor Deposition tool wired end-to-end. examples/pvd_tool/ is the template a real customer should fork. Files: - equipment.yaml: 32 SVIDs (chamber pressure, temperature, source power, gas flows, cooling water, wafer counters, recipe step state, EPT name, 4 load ports), 5 DVIDs, 7 ECIDs (setpoints + T_CRA/T_DELAY + cleaning interval + retry count), 17 CEIDs (control state, alarms, process lifecycle, material movement, EPT), 12 alarms with realistic categories (safety, error, warning, attention), 3 multi-step recipes (Al / Ti / Cu), 9 host commands. - main.cpp (~860 lines): the vendor-side application: §1 helpers + constants §2 sensor simulator — 4 sensors at 10 Hz + 1 Hz cadences, random-walk around step-targeted setpoints, asio::post-on-strand thread-safety pattern §3 recipe runner — parses recipe body (STEP NAME duration=120s power=2500W gas=Argon flow=50sccm), walks each step at 1s per declared-second, fires step-started/completed CEIDs, drives PJ FSM through ProcessComplete §4 alarm threshold monitor — chamber-pressure-over-setpoint and cleaning-interval logic, continuous evaluation, set/clear emission gated on alarm-enable §5 EPT cycler — Standby ↔ Productive ↔ UnscheduledDowntime based on PJ activity + safety alarms §6 Prometheus exporter on :9090 (pvd_messages_total, pvd_chamber_pressure_torr, pvd_spool_depth, pvd_events_total, pvd_alarm_set_total) §7 Router handlers — full E30 set (~40 handlers) so a host can do real work §8 main() — YAML validation, model construction, server wiring, periodic gauge updates - README.md: section-by-section walkthrough, what's the same as apps/secs_server.cpp, what this adds (simulator + recipe runner + alarm monitor + EPT cycler + metrics), what's not here (persistence + E84 + real I/O), and what to change for your tool. Verification: 47/47 conformance harness checks PASS against the PVD tool — same as the demo server. CMakeLists.txt adds the pvd_tool target. README's documentation map points at examples/pvd_tool/. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-06-09 16:57:10 +02:00