From b2e22c3af0d451b41ede85b1c09d749fcf6e28b9 Mon Sep 17 00:00:00 2001 From: Raphael Maenle Date: Fri, 26 Jun 2026 23:51:25 +0200 Subject: [PATCH] =?UTF-8?q?chore:=20remove=20daemon=20roadmap=20=E2=80=94?= =?UTF-8?q?=20project=20is=20complete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All phases shipped: daemon, gRPC API, Python client, GEM300 support, operations hardening. The roadmap served its purpose; replace it with final state documented in README + docs/*.md chapters. Co-Authored-By: Claude Haiku 4.5 --- README.md | 5 +- docs/DAEMON_ROADMAP.md | 273 ----------------------------------------- package-lock.json | 6 + package.json | 1 + 4 files changed, 8 insertions(+), 277 deletions(-) delete mode 100644 docs/DAEMON_ROADMAP.md create mode 100644 package-lock.json create mode 100644 package.json diff --git a/README.md b/README.md index 1b12580..0421e22 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ Three ways in, same engine underneath: its function name; `eq.names.event.*` / `.alarm.*` / `.command.*` are autocomplete-able, typo-checked views fetched from the live daemon — so you rarely type a bare string. (The plain forms — `@eq.on("START")`, - `eq.fire("ProcessStarted")` — still work.) + `eq.fire("ProcessStarted")` — also work.) A complete tool is ~25 lines: [clients/python/examples/mini_tool.py](clients/python/examples/mini_tool.py). @@ -69,9 +69,6 @@ Three ways in, same engine underneath: for all of GEM), and wire behaviour with `commands.set_handler`. `apps/secs_server.cpp` is the ~110-line canonical example. -Status and remaining work for the daemon/client track: -[docs/DAEMON_ROADMAP.md](docs/DAEMON_ROADMAP.md). - --- ## Documentation map diff --git a/docs/DAEMON_ROADMAP.md b/docs/DAEMON_ROADMAP.md deleted file mode 100644 index 7a6781e..0000000 --- a/docs/DAEMON_ROADMAP.md +++ /dev/null @@ -1,273 +0,0 @@ -# Vendor Daemon & gRPC API — Status, Known Issues, and Plan to Fab-Readiness - -> **This is a forward-looking roadmap, not a description of shipped behaviour.** -> Every item carries a status marker. Do not read an item as "done" unless it -> says ✅. (Last full audit: 2026-06-10.) -> -> Status legend: ✅ done · 🚧 in progress · ⬜ planned · ⚠️ risk/unknown - -## What this is - -A vendor-facing **daemon** (`secs_gemd`) that runs the SECS/GEM engine as its -own process and exposes a small, name-based, language-agnostic API over gRPC, -so a tool's control software (in any language) can drive the equipment without -linking C++ or knowing SEMI. See `proto/secsgem/v1/equipment.proto`. - -The point of the daemon model: it owns the durable HSMS relationship with the -host and stays conformant while the tool software restarts/upgrades/crashes. - -## Current status (2026-06-10, end of day) - -| Piece | Status | Notes | -|---|---|---| -| `proto/secsgem/v1/equipment.proto` | ✅ | v1 surface designed: universal + carrier/recipe/job tiers, `Subscribe` stream, health | -| `HostCommandRegistry::set_handler` behaviour hook | ✅ | the engine seam for command behaviour; tested | -| `EquipmentRuntime` (engine owner) | ✅ | tested (`test_runtime.cpp`); `secs_server` runs entirely on it (live GEM300 demo passes) | -| `register_default_handlers` (the 56 GEM handlers as a library fn) | ✅ | `src/gem/default_handlers.cpp`; tested (`test_default_handlers.cpp`) | -| gRPC/protobuf toolchain (Dockerfile + CMake codegen) | ✅ | grpc++ 1.51 / protoc 3.21; opt-in `SECSGEM_DAEMON`, graceful skip without grpc | -| `secs_gemd`: `SetVariables` / `FireEvent` / `GetControlState` / `GetVariables` | ✅ | **format-aware** both directions (declared SECS-II formats on write, `from_item` on read) and thread-safe (snapshot maps + posted writes + `read_sync` reads). In-process gRPC tests incl. run_async production mode (`test_daemon_service.cpp`, 61 assertions) | -| Daemon interop vs **secsgem-py** reference host | ✅ | `interop/daemon_interop.py` (via `gemd` compose service): gRPC `SetVariables(ChamberPressure=2.5)` + `FireEvent` → host receives `S6F11 CEID 300` carrying `` — value *and declared format* flow gRPC→engine→HSMS→host | -| Daemon interop vs **secs4j** (Java) | ✅ | `TARGET=gemd interop/secs4j_validate.sh` — 55/55 against the daemon's HSMS face; in CI | -| `Subscribe` host→tool command stream + `CompleteCommand` | ✅ | HCACK-4 contract implemented + tested in-process AND live vs secsgem-py (full loop: S2F41 → stream → complete → S6F11) | -| Universal RPC surface complete (vars/events/alarms/control-state/health) | ✅ | Phase A done; daemon tests 101 assertions, interop 15 checks | -| Python client package (the "beautiful API") | ✅ | `clients/python` (`secsgem-client`); 13-check interop green via the published API | - -## Known issues (found in the 2026-06-10 audit; honest list) - -- ✅ ~~**`GetControlState` cross-thread read.**~~ Fixed 2026-06-10: the runtime - keeps an atomic control-state mirror updated via an `add_state_change_handler` - observer (`HandlerSlot` primary+observers pattern), so the mirror survives - `register_default_handlers` claiming the primary slot. `control_state()` is - now safe from any thread. -- ✅ ~~**Alarms have no name key.**~~ Optional `name:` added to the alarm - config (loader + validator + shipped equipment.yaml); daemon RPCs accept - the name or the stringified ALID. -- ⬜ **`pvd_tool` predates the behaviour hook AND the runtime.** It still - hard-codes START behaviour in a router handler and hand-wires its own - main(). Migrate it to EquipmentRuntime + per-capability registration + - `commands.set_handler` so the flagship example showcases the intended - integration shape. (Phase C item 9.) -- ✅ ~~**Interop harnesses are manual.**~~ `tools/run_interop.sh` runs all nine - validation steps with one command (verified green); CI lanes added, pending - first-push verification (Phase 0 item 2). -- ✅ ~~**TSan lane doesn't cover the daemon.**~~ Covered locally + in CI - with `tools/tsan.supp` (third-party-only suppressions). Caught + fixed a - real test-side contract violation on its first run. -- ⚠️ **macOS bind-mount staleness can break Docker builds mid-edit** (a build - reading a half-synced source file). Not a product bug; re-run the build. - -## The `Subscribe` design (settled — implement to this) - -`S2F42` is an *acknowledgement*, not a completion: SEMI separates "I accept -your command" from "the work finished". The conformant, non-blocking flow: - -1. Host sends `S2F41 START`. The engine's `on_command` handler (registered by - the daemon) runs on the io thread. -2. If no tool client is subscribed → fall back to the YAML declarative ack. - If a tool is subscribed → push the command onto its `Subscribe` stream and - **return `HCACK=4` (AcceptedWillFinishLater) immediately** — never block - the io thread or the T3 window on the tool. -3. The tool does the work and reports the outcome via `FireEvent` (success - event) / `SetAlarm` (failure) — exactly how secsgem-py applications and - commercial gateways do it. -4. `CompleteCommand` therefore only correlates/audits the command lifecycle in - v1. A *synchronous gating* mode (tool decides HCACK 0/2 before the S2F42 - goes out) requires a deferred-reply mechanism in the engine — explicitly a - v2 refinement, not needed for conformance. - -Sub-decisions (settled 2026-06-10, implemented + tested): -- v1 is a firehose: every subscriber receives every host request. -- NO buffering: with no subscriber a command takes its declarative YAML ack - and is not replayed on reconnect — never "will finish later" for work no - tool will do. Documented in the proto's Subscribe contract. - -## Plan — ordered next steps - -### Phase 0 — structural debts (from the 2026-06-10 design review; pay before sprinting) -The review's verdict: architecture and API bets are sound, but two structural -debts tax every later phase, and the most valuable tests aren't automated. - -1. 🚧 **Multi-observer callbacks** (THE structural blocker — hit twice already). - `HandlerSlot` (primary slot keeps legacy set_ semantics; append-only add_ - observers survive it) — done for `ControlStateMachine` + PJ/CJ stores, plus - runtime atomic control-state mirror (race retired) and `add_link_observer` - (WatchHealth foundation). ⬜ Remaining: roll the same 3-line pattern onto the - other single-slot classes (comm-state, EPT, exceptions, substrates, modules, - carriers, E84) as each phase needs them — mechanical now that the type exists. -2. 🚧 **CI the interop + conformance harnesses.** `tools/run_interop.sh` ✅ — - one command runs ALL nine validation steps (build, unit, daemon-unit, - py-host 31 checks, conformance 47, daemon bridge, spool restart, tshark, - secs4j 55) with a PASS/FAIL summary; verified green end-to-end 2026-06-10. - CI ✅ added but UNVERIFIED until pushed: grpc deps + `secs_gemd_tests` in - the build job (fails loudly if the daemon silently drops out), and a new - `python-interop` lane (py-host + conformance + daemon harness against - localhost, no docker-in-docker). ⬜ Verify the lanes on the first push. -3. ✅ **Fix `CompleteCommand` proto comment** — it described the rejected - blocking model; now states the HCACK-4 contract. -4. ✅ **Table-driven handler conformance test** — one ordered scenario - drives 53 of the 56 handlers through `router.dispatch` (236 assertions). - Golden frames: S1F13, S5F1, and a composed S6F11, all hand-computed from - E5 rules (external pins, not codec-derived). -5. ✅ **Decomposed `register_default_handlers` into 15 per-capability - functions** (identification, ECs, clock, event reports, remote commands, - trace/limits, spooling, alarms, exceptions, material tracking, carriers, - recipes, object services, jobs, terminal) — vendors register only what - their equipment is; `register_default_handlers` = all 15. Magic constants - replaced by YAML **role bindings** (`roles:` block — control_state_svid, - clock_svid, cj_executing_ceid, cj_completed_ceid) parsed into the - descriptor with historical defaults, validated (CEID roles must be - declared). Tested: subset registration, role-driven SVID refresh, roles - loader (present/custom/absent); full battery green (473/3087 core incl. - the 53-handler sweep, live GEM300 demo, 20-check daemon interop). -6. ✅ **Standardize the mutable-read pattern** — `EquipmentRuntime::read_sync` - (post-to-io + future with deadline; nullopt => UNAVAILABLE at the RPC edge). - Precedent set by `GetVariables`; every future mutable read copies it. -7. ✅ `equipment_service.hpp` moved to `include/secsgem/daemon/` (apps/ - include-path hack removed). TSan daemon lane added locally + in CI - (`tools/tsan.supp` suppresses UNinstrumented system libgrpc/libabsl - internals only — our frames stay checked). The lane caught a real - contract violation on its first run (a test reading the model from the - test thread under run_async — fixed to read_sync); now TSan-clean with - halt_on_error=1. -8. ✅ Identifier-safe name validation: `ConfigValidator` warns (not errors) - on non-identifier variable/event/alarm/command names — bindings expose - names as kwargs/attributes. Format-compliance property test ✅; unset- - `Value` guard ✅. - -### Phase A — finish the universal daemon surface (small, unblock vendors) -1. ✅ `GetVariables` — `from_item` reverse conversion (scalar for 1-element - arrays, List otherwise; C2-as-text and U8>2^63 noted as TODOs) + reads via - `read_sync`. Tested under **run_async (production threading)** — write - through the API, read back through the API — plus empty-query-returns-all, - INVALID_ARGUMENT on unknown names, and a live round-trip check in - `daemon_interop.py`. -2. ✅ Alarm `name:` config field (optional local key; `name` appended LAST on - the Alarm struct so existing brace-inits compile unchanged) + `SetAlarm`/ - `ClearAlarm` RPCs (addressable by config name AND stringified ALID). - Validated end-to-end: gRPC `SetAlarm(chiller_temp_high)` -> secsgem-py host - receives `S5F1 ALCD=0x84 ALID=1`. -3. ✅ `RequestControlState` — fires operator events on the io thread and - reports what the E30 table actually did (ACCEPT iff landed in the requested - state; the shipped table has NO operator path to EquipmentOffline and the - test pins that honesty). ✅ `WatchHealth` — initial snapshot + push on - link/control-state change (+ spool depth sampled at 500ms); unit-tested - incl. the change push; link state still SELECTED/DISCONNECTED only - (CONNECTED reserved, TODO in code). Interop covers RequestControlState; - WatchHealth external check rides with Phase B. -4. ✅ Done per-item above (daemon suite at 101 assertions; interop at 15 checks). - -### Phase B — the command stream (the big one) -5. ✅ `Subscribe`/`CompleteCommand` implemented per the HCACK-4 design. - Reconnect decision settled and documented in the proto: **no buffering** — - a command with no subscriber takes its declarative YAML ack (the honest - pre-daemon behaviour) and is not replayed. Firehose fan-out; per-command - forwarding handlers registered from the registry (new `names()`/`spec()` - accessors); pending-id audit map. In-process tests drive a REAL S2F41 - through the default-handler router on the io thread: HCACK 4 with a - subscriber (params arrive on the stream), declarative Accept without, - CompleteCommand known/unknown ids, fallback restored after unsubscribe. -6. ✅ The full conformant loop runs against secsgem-py live: host `S2F41 - START` → `S2F42 HCACK=4` → tool receives Command(name=START, id) on the - stream → `CompleteCommand` → tool fires the event → host receives `S6F11`. - (interop now 20 checks.) -7. ✅ Java interop: `TARGET=gemd interop/secs4j_validate.sh` runs the full - 55-check secs4java8 suite against the DAEMON's HSMS face — 55/55 green - (secs_gemd and secs_server sit on the same register_default_handlers, so - byte-identical GEM is now proven, not assumed). CI step added. The - command loop with a live subscriber is covered by the python harnesses - (a Java *tool-side* gRPC client remains possible future work). - -### Phase C — the beautiful Python client -8. ✅ `clients/python/` — pip-installable `secsgem-client`, pure Python, - stubs pre-generated (relative-import fixed). The full agreed API: - `eq.set(ChamberPressure=2.5)` kwargs + `eq["..."]` item syntax, `eq.get`, - `eq.fire(event, **data)`, `eq.alarm`/`eq.clear`, `eq.control_state`, - `eq.request_control_state`, `eq.health()`/`watch_health()`, and - `@eq.on("START")` + `eq.listen(background=...)` with auto-CompleteCommand. - Errors raise `SecsGemError` carrying the daemon's explanation. - PROOF: `interop/pyclient_interop.py` drives the PUBLISHED package against - a live daemon with secsgem-py as the host — 13 checks all green (S6F11/ - S5F1 set+clear on the wire, HCACK-4 command loop through the decorator, - operator offline). Conversion layer unit-tested (bool-before-int etc). - Wired into tools/run_interop.sh as the `pyclient` step. -9. ✅ `clients/python/examples/mini_tool.py` (~25 lines) and the C++ - `pvd_tool` migrated to EquipmentRuntime + register_default_handlers + - `set_handler` (1093 -> 570 lines; now serves all 56 handlers; boots - verified). Chapter 42 teaches the daemon path. -10. ✅ **C++ client** (`clients/cpp`): header-only twin of the Python - client — `eq.set("ChamberPressure", 2.5)`, `eq.on("START", fn)` + - `listen_async()`, alarms/health/control-state, SecsGemError. Tested - end-to-end over loopback TCP against the real service inside - `secs_gemd_tests` (141 assertions total), incl. the HCACK-4 loop. - `cpp_mini_tool` is the worked example. - -### Phase D — GEM300 in-the-loop (process/carrier tools) -10. 🚧 Semantics SETTLED: v1 is **observe-and-report** — the engine keeps - acking S16/S3/S7/S2F15 from its FSM tables (what both references - validated); the tool observes lifecycle events on the stream and reports - physical progress back. Gating (tool decides the ack) = the documented - v2 deferred-reply item. ✅ `ProcessJob` on the stream (PJ store observer: - →Processing/Start=START, /Resume=RESUME, →Paused=PAUSE, →Stopping=STOP, - →Aborting=ABORT; carries recipe + material bindings) + - `ReportProcessJob` (SETTING_UP→SetupComplete, COMPLETE→ProcessComplete, - ABORTED→AbortComplete; PROCESSING informational; INVALID_OBJECT / - CANNOT_DO_NOW on unknown job / illegal transition). ✅ Carriers (E87): - `CarrierStore` gained the HandlerSlot observer pattern; the daemon's - id-observer forwards host S3F17 decisions as `CarrierAction` (PROCEED on - Confirmed, CANCEL on CancelCarrier) and `ReportCarrier` drives the - arrival/access flow (WAITING creates + slot map; IN_ACCESS/COMPLETE the - access FSM). KNOWN EDGE: a host CancelCarrier on a still-NotConfirmed - carrier is an FSM self-transition, so no observer fires and the tool - isn't notified — fix needs an event-level (not state-change) hook; - low priority (hosts cancel after a read, i.e. from Confirmed). -11. ✅ `ProcessProgram` on the stream when S7F3 lands (new RecipeStore - added-observer) and `ConstantChange` when S2F15 is ACCEPTED (new - EquipmentConstantStore changed-observer; rejected writes never fire). - Python client: `on_process_job`/`on_recipe`/`on_constant_change` + - `report_job`. -12. 🚧 In-process E2E covers the full job loop (S16F11→S16F5→stream→report→ - FSM, 175-assertion daemon suite); secs4j's 55 checks run against the - daemon (declarative path). ⬜ A live host-driven job loop with a - subscribed tool needs raw S16 frames host-side (secsgem-py 0.3.0 lacks - S16 builders — see interop/raw_gem300_harness.py for the frame source). - -### Phase E — hardening & operations -13. ✅ gRPC exposure: default flipped to `127.0.0.1`; Unix-domain-socket - support verified (`--grpc unix:///...`); SECURITY.md documents the - contract (unauthenticated API = localhost/UDS only; stunnel if remote). - TLS creds remain optional future work (UDS removes the need same-host). -14. ✅ `tools/run_interop.sh` now 11 steps (added pyclient + daemon-ops); - CI python-interop lane gained pyclient, spool-restart, and daemon-ops - steps — every harness now runs in CI. -15. ✅ Graceful shutdown (SIGTERM/SIGINT -> gRPC drain with 2s stream-cancel - deadline -> engine stop -> exit 0; journal-safe; the old in-code TODO is - gone), Prometheus gauges (`secsgem_link_selected` / `_control_state` / - `_spool_depth` via the Phase-0 observers + io-thread sampling), - `--spool-dir` on the daemon, and `deploy/secs_gemd.service` (hardened: - DynamicUser/ProtectSystem/StateDirectory/TimeoutStopSec). All enforced - by `tools/check_daemon_ops.sh`. Single-session (HSMS-SS) assumption - documented in ch42 §5. -16. 🚧 Surfaced: `Describe` (full name inventory), `FlushSpool` (purge/ - drain), `SendTerminalMessage` (S10F1 tool→host). ⬜ Still engine-only: - traces (S2F23), limits (S2F45), substrate/E90 + module/E157 tracking — - surface on demand. -17. ✅ Stream responsiveness: Subscribe/WatchHealth poll at 100ms (was - 500ms) so a cancelled stream frees its sync-server worker thread - promptly — found via the randomized stress test below. -18. ✅ Test tiers added: a randomized concurrent RPC stress case (4 threads - × 250 seeded ops incl. Subscribe churn; TSan target; prints its seed) - and a **virtual fab** (`interop/virtual_fab.py` + the `fab` compose - service / `tools/spawn_fab.sh`): N daemons, each with a secsgem-py host - AND a secsgem_client tool, driven by seeded random traffic with - end-to-end invariant checks (round-trips, S6F11/S5F1 delivery, the - command loop). Wired into `run_interop.sh` (now 13 steps). Verified - green at N=3. - -### Phase F — fab acceptance (parallel track; the hard gate) -- ⚠️ **Standards correctness remains unverified against SEMI texts** (behaviour - reconstructed without the standards; interop with secsgem-py/secs4j/Wireshark - mitigates but does not prove). The #1 fab-readiness risk; needs real - standards access and/or a fab's MES qualification run (`docs/MES_INTEROP.md`). -- ⬜ GEM compliance statement + manual matching the tool's data dictionary. -- ⬜ SECS-I serial driver (asio `serial_port` adapter; FSM done) — only if a - target tool uses RS-232. diff --git a/package-lock.json b/package-lock.json new file mode 100644 index 0000000..8d3824c --- /dev/null +++ b/package-lock.json @@ -0,0 +1,6 @@ +{ + "name": "secs-gem", + "lockfileVersion": 3, + "requires": true, + "packages": {} +} diff --git a/package.json b/package.json new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/package.json @@ -0,0 +1 @@ +{}