secs-gem/docs/DAEMON_ROADMAP.md

Vendor Daemon & gRPC API — Status, Known Issues, and Plan to Fab-Readiness

This is a forward-looking roadmap, not a description of shipped behaviour. Every item carries a status marker. Do not read an item as "done" unless it says ✅. (Last full audit: 2026-06-10.)

Status legend: ✅ done · 🚧 in progress · ⬜ planned · ⚠️ risk/unknown

What this is

A vendor-facing daemon (secs_gemd) that runs the SECS/GEM engine as its own process and exposes a small, name-based, language-agnostic API over gRPC, so a tool's control software (in any language) can drive the equipment without linking C++ or knowing SEMI. See proto/secsgem/v1/equipment.proto.

The point of the daemon model: it owns the durable HSMS relationship with the host and stays conformant while the tool software restarts/upgrades/crashes.

Current status (2026-06-10, end of day)

Piece	Status	Notes
proto/secsgem/v1/equipment.proto	✅	v1 surface designed: universal + carrier/recipe/job tiers, Subscribe stream, health
HostCommandRegistry::set_handler behaviour hook	✅	the engine seam for command behaviour; tested
EquipmentRuntime (engine owner)	✅	tested (test_runtime.cpp); secs_server runs entirely on it (live GEM300 demo passes)
register_default_handlers (the 56 GEM handlers as a library fn)	✅	src/gem/default_handlers.cpp; tested (test_default_handlers.cpp)
gRPC/protobuf toolchain (Dockerfile + CMake codegen)	✅	grpc++ 1.51 / protoc 3.21; opt-in SECSGEM_DAEMON, graceful skip without grpc
secs_gemd: SetVariables / FireEvent / GetControlState / GetVariables	✅	format-aware both directions (declared SECS-II formats on write, from_item on read) and thread-safe (snapshot maps + posted writes + read_sync reads). In-process gRPC tests incl. run_async production mode (test_daemon_service.cpp, 61 assertions)
Daemon interop vs secsgem-py reference host	✅	interop/daemon_interop.py (via gemd compose service): gRPC SetVariables(ChamberPressure=2.5) + FireEvent → host receives S6F11 CEID 300 carrying <F4 2.5> — value and declared format flow gRPC→engine→HSMS→host
Daemon interop vs secs4j (Java)	✅	TARGET=gemd interop/secs4j_validate.sh — 55/55 against the daemon's HSMS face; in CI
Subscribe host→tool command stream + CompleteCommand	✅	HCACK-4 contract implemented + tested in-process AND live vs secsgem-py (full loop: S2F41 → stream → complete → S6F11)
Universal RPC surface complete (vars/events/alarms/control-state/health)	✅	Phase A done; daemon tests 101 assertions, interop 15 checks
Python client package (the "beautiful API")	✅	clients/python (secsgem-client); 13-check interop green via the published API

Known issues (found in the 2026-06-10 audit; honest list)

• ✅ GetControlState cross-thread read. Fixed 2026-06-10: the runtime keeps an atomic control-state mirror updated via an add_state_change_handler observer (HandlerSlot primary+observers pattern), so the mirror survives register_default_handlers claiming the primary slot. control_state() is now safe from any thread.
• ✅ Alarms have no name key. Optional name: added to the alarm config (loader + validator + shipped equipment.yaml); daemon RPCs accept the name or the stringified ALID.
• ⬜ pvd_tool predates the behaviour hook AND the runtime. It still hard-codes START behaviour in a router handler and hand-wires its own main(). Migrate it to EquipmentRuntime + per-capability registration + commands.set_handler so the flagship example showcases the intended integration shape. (Phase C item 9.)
• ✅ Interop harnesses are manual. tools/run_interop.sh runs all nine validation steps with one command (verified green); CI lanes added, pending first-push verification (Phase 0 item 2).
• ✅ TSan lane doesn't cover the daemon. Covered locally + in CI with tools/tsan.supp (third-party-only suppressions). Caught + fixed a real test-side contract violation on its first run.
• ⚠️ macOS bind-mount staleness can break Docker builds mid-edit (a build reading a half-synced source file). Not a product bug; re-run the build.

The Subscribe design (settled — implement to this)

S2F42 is an acknowledgement, not a completion: SEMI separates "I accept your command" from "the work finished". The conformant, non-blocking flow:

1. Host sends S2F41 START. The engine's on_command handler (registered by the daemon) runs on the io thread.
2. If no tool client is subscribed → fall back to the YAML declarative ack. If a tool is subscribed → push the command onto its Subscribe stream and return HCACK=4 (AcceptedWillFinishLater) immediately — never block the io thread or the T3 window on the tool.
3. The tool does the work and reports the outcome via FireEvent (success event) / SetAlarm (failure) — exactly how secsgem-py applications and commercial gateways do it.
4. CompleteCommand therefore only correlates/audits the command lifecycle in v1. A synchronous gating mode (tool decides HCACK 0/2 before the S2F42 goes out) requires a deferred-reply mechanism in the engine — explicitly a v2 refinement, not needed for conformance.

Sub-decisions (settled 2026-06-10, implemented + tested):

• v1 is a firehose: every subscriber receives every host request.
• NO buffering: with no subscriber a command takes its declarative YAML ack and is not replayed on reconnect — never "will finish later" for work no tool will do. Documented in the proto's Subscribe contract.

Plan — ordered next steps

Phase 0 — structural debts (from the 2026-06-10 design review; pay before sprinting)

The review's verdict: architecture and API bets are sound, but two structural debts tax every later phase, and the most valuable tests aren't automated.

1. 🚧 Multi-observer callbacks (THE structural blocker — hit twice already). HandlerSlot (primary slot keeps legacy set_ semantics; append-only add_ observers survive it) — done for ControlStateMachine + PJ/CJ stores, plus runtime atomic control-state mirror (race retired) and add_link_observer (WatchHealth foundation). ⬜ Remaining: roll the same 3-line pattern onto the other single-slot classes (comm-state, EPT, exceptions, substrates, modules, carriers, E84) as each phase needs them — mechanical now that the type exists.
2. 🚧 CI the interop + conformance harnesses. tools/run_interop.sh ✅ — one command runs ALL nine validation steps (build, unit, daemon-unit, py-host 31 checks, conformance 47, daemon bridge, spool restart, tshark, secs4j 55) with a PASS/FAIL summary; verified green end-to-end 2026-06-10. CI ✅ added but UNVERIFIED until pushed: grpc deps + secs_gemd_tests in the build job (fails loudly if the daemon silently drops out), and a new python-interop lane (py-host + conformance + daemon harness against localhost, no docker-in-docker). ⬜ Verify the lanes on the first push.
3. ✅ Fix CompleteCommand proto comment — it described the rejected blocking model; now states the HCACK-4 contract.
4. ✅ Table-driven handler conformance test — one ordered scenario drives 53 of the 56 handlers through router.dispatch (236 assertions). Golden frames: S1F13, S5F1, and a composed S6F11, all hand-computed from E5 rules (external pins, not codec-derived).
5. ✅ Decomposed register_default_handlers into 15 per-capability functions (identification, ECs, clock, event reports, remote commands, trace/limits, spooling, alarms, exceptions, material tracking, carriers, recipes, object services, jobs, terminal) — vendors register only what their equipment is; register_default_handlers = all 15. Magic constants replaced by YAML role bindings (roles: block — control_state_svid, clock_svid, cj_executing_ceid, cj_completed_ceid) parsed into the descriptor with historical defaults, validated (CEID roles must be declared). Tested: subset registration, role-driven SVID refresh, roles loader (present/custom/absent); full battery green (473/3087 core incl. the 53-handler sweep, live GEM300 demo, 20-check daemon interop).
6. ✅ Standardize the mutable-read pattern — EquipmentRuntime::read_sync (post-to-io + future with deadline; nullopt => UNAVAILABLE at the RPC edge). Precedent set by GetVariables; every future mutable read copies it.
7. ✅ equipment_service.hpp moved to include/secsgem/daemon/ (apps/ include-path hack removed). TSan daemon lane added locally + in CI (tools/tsan.supp suppresses UNinstrumented system libgrpc/libabsl internals only — our frames stay checked). The lane caught a real contract violation on its first run (a test reading the model from the test thread under run_async — fixed to read_sync); now TSan-clean with halt_on_error=1.
8. ✅ Identifier-safe name validation: ConfigValidator warns (not errors) on non-identifier variable/event/alarm/command names — bindings expose names as kwargs/attributes. Format-compliance property test ✅; unset- Value guard ✅.

Phase A — finish the universal daemon surface (small, unblock vendors)

1. ✅ GetVariables — from_item reverse conversion (scalar for 1-element arrays, List otherwise; C2-as-text and U8>2^63 noted as TODOs) + reads via read_sync. Tested under run_async (production threading) — write through the API, read back through the API — plus empty-query-returns-all, INVALID_ARGUMENT on unknown names, and a live round-trip check in daemon_interop.py.
2. ✅ Alarm name: config field (optional local key; name appended LAST on the Alarm struct so existing brace-inits compile unchanged) + SetAlarm/ ClearAlarm RPCs (addressable by config name AND stringified ALID). Validated end-to-end: gRPC SetAlarm(chiller_temp_high) -> secsgem-py host receives S5F1 ALCD=0x84 ALID=1.
3. ✅ RequestControlState — fires operator events on the io thread and reports what the E30 table actually did (ACCEPT iff landed in the requested state; the shipped table has NO operator path to EquipmentOffline and the test pins that honesty). ✅ WatchHealth — initial snapshot + push on link/control-state change (+ spool depth sampled at 500ms); unit-tested incl. the change push; link state still SELECTED/DISCONNECTED only (CONNECTED reserved, TODO in code). Interop covers RequestControlState; WatchHealth external check rides with Phase B.
4. ✅ Done per-item above (daemon suite at 101 assertions; interop at 15 checks).

Phase B — the command stream (the big one)

5. ✅ Subscribe/CompleteCommand implemented per the HCACK-4 design. Reconnect decision settled and documented in the proto: no buffering — a command with no subscriber takes its declarative YAML ack (the honest pre-daemon behaviour) and is not replayed. Firehose fan-out; per-command forwarding handlers registered from the registry (new names()/spec() accessors); pending-id audit map. In-process tests drive a REAL S2F41 through the default-handler router on the io thread: HCACK 4 with a subscriber (params arrive on the stream), declarative Accept without, CompleteCommand known/unknown ids, fallback restored after unsubscribe.
6. ✅ The full conformant loop runs against secsgem-py live: host S2F41 START → S2F42 HCACK=4 → tool receives Command(name=START, id) on the stream → CompleteCommand → tool fires the event → host receives S6F11. (interop now 20 checks.)
7. ✅ Java interop: TARGET=gemd interop/secs4j_validate.sh runs the full 55-check secs4java8 suite against the DAEMON's HSMS face — 55/55 green (secs_gemd and secs_server sit on the same register_default_handlers, so byte-identical GEM is now proven, not assumed). CI step added. The command loop with a live subscriber is covered by the python harnesses (a Java tool-side gRPC client remains possible future work).

Phase C — the beautiful Python client

8. ✅ clients/python/ — pip-installable secsgem-client, pure Python, stubs pre-generated (relative-import fixed). The full agreed API: eq.set(ChamberPressure=2.5) kwargs + eq["..."] item syntax, eq.get, eq.fire(event, **data), eq.alarm/eq.clear, eq.control_state, eq.request_control_state, eq.health()/watch_health(), and @eq.on("START") + eq.listen(background=...) with auto-CompleteCommand. Errors raise SecsGemError carrying the daemon's explanation. PROOF: interop/pyclient_interop.py drives the PUBLISHED package against a live daemon with secsgem-py as the host — 13 checks all green (S6F11/ S5F1 set+clear on the wire, HCACK-4 command loop through the decorator, operator offline). Conversion layer unit-tested (bool-before-int etc). Wired into tools/run_interop.sh as the pyclient step.
9. ✅ clients/python/examples/mini_tool.py (~25 lines) and the C++ pvd_tool migrated to EquipmentRuntime + register_default_handlers + set_handler (1093 -> 570 lines; now serves all 56 handlers; boots verified). Chapter 42 teaches the daemon path.
10. ✅ C++ client (clients/cpp): header-only twin of the Python client — eq.set("ChamberPressure", 2.5), eq.on("START", fn) + listen_async(), alarms/health/control-state, SecsGemError. Tested end-to-end over loopback TCP against the real service inside secs_gemd_tests (141 assertions total), incl. the HCACK-4 loop. cpp_mini_tool is the worked example.

Phase D — GEM300 in-the-loop (process/carrier tools)

10. 🚧 Semantics SETTLED: v1 is observe-and-report — the engine keeps acking S16/S3/S7/S2F15 from its FSM tables (what both references validated); the tool observes lifecycle events on the stream and reports physical progress back. Gating (tool decides the ack) = the documented v2 deferred-reply item. ✅ ProcessJob on the stream (PJ store observer: →Processing/Start=START, /Resume=RESUME, →Paused=PAUSE, →Stopping=STOP, →Aborting=ABORT; carries recipe + material bindings) + ReportProcessJob (SETTING_UP→SetupComplete, COMPLETE→ProcessComplete, ABORTED→AbortComplete; PROCESSING informational; INVALID_OBJECT / CANNOT_DO_NOW on unknown job / illegal transition). ⬜ Carriers deferred: CarrierStore has no observer machinery yet — apply the HandlerSlot pattern, then mirror the job wiring; ReportCarrier stays gRPC UNIMPLEMENTED (the stub default) until then.
11. ✅ ProcessProgram on the stream when S7F3 lands (new RecipeStore added-observer) and ConstantChange when S2F15 is ACCEPTED (new EquipmentConstantStore changed-observer; rejected writes never fire). Python client: on_process_job/on_recipe/on_constant_change + report_job.
12. 🚧 In-process E2E covers the full job loop (S16F11→S16F5→stream→report→ FSM, 175-assertion daemon suite); secs4j's 55 checks run against the daemon (declarative path). ⬜ A live host-driven job loop with a subscribed tool needs raw S16 frames host-side (secsgem-py 0.3.0 lacks S16 builders — see interop/raw_gem300_harness.py for the frame source).

Phase E — hardening & operations

13. ✅ gRPC exposure: default flipped to 127.0.0.1; Unix-domain-socket support verified (--grpc unix:///...); SECURITY.md documents the contract (unauthenticated API = localhost/UDS only; stunnel if remote). TLS creds remain optional future work (UDS removes the need same-host).
14. ✅ tools/run_interop.sh now 11 steps (added pyclient + daemon-ops); CI python-interop lane gained pyclient, spool-restart, and daemon-ops steps — every harness now runs in CI.
15. ✅ Graceful shutdown (SIGTERM/SIGINT -> gRPC drain with 2s stream-cancel deadline -> engine stop -> exit 0; journal-safe; the old in-code TODO is gone), Prometheus gauges (secsgem_link_selected / _control_state / _spool_depth via the Phase-0 observers + io-thread sampling), --spool-dir on the daemon, and deploy/secs_gemd.service (hardened: DynamicUser/ProtectSystem/StateDirectory/TimeoutStopSec). All enforced by tools/check_daemon_ops.sh. Single-session (HSMS-SS) assumption documented in ch42 §5.
16. ⬜ Remaining Layer-1 API: traces, limits, substrates/modules, terminal services, spool flush RPC, Describe RPC. (Engine-side all exists; surface on demand.)

Phase F — fab acceptance (parallel track; the hard gate)

• ⚠️ Standards correctness remains unverified against SEMI texts (behaviour reconstructed without the standards; interop with secsgem-py/secs4j/Wireshark mitigates but does not prove). The #1 fab-readiness risk; needs real standards access and/or a fab's MES qualification run (docs/MES_INTEROP.md).
• ⬜ GEM compliance statement + manual matching the tool's data dictionary.
• ⬜ SECS-I serial driver (asio serial_port adapter; FSM done) — only if a target tool uses RS-232.