secs-gem/docs/DAEMON_ROADMAP.md

Vendor Daemon & gRPC API — Status, Known Issues, and Plan to Fab-Readiness

This is a forward-looking roadmap, not a description of shipped behaviour. Every item carries a status marker. Do not read an item as "done" unless it says ✅. (Last full audit: 2026-06-10.)

Status legend: ✅ done · 🚧 in progress · ⬜ planned · ⚠️ risk/unknown

What this is

A vendor-facing daemon (secs_gemd) that runs the SECS/GEM engine as its own process and exposes a small, name-based, language-agnostic API over gRPC, so a tool's control software (in any language) can drive the equipment without linking C++ or knowing SEMI. See proto/secsgem/v1/equipment.proto.

The point of the daemon model: it owns the durable HSMS relationship with the host and stays conformant while the tool software restarts/upgrades/crashes.

Current status (2026-06-10, end of day)

Piece	Status	Notes
proto/secsgem/v1/equipment.proto	✅	v1 surface designed: universal + carrier/recipe/job tiers, Subscribe stream, health
HostCommandRegistry::set_handler behaviour hook	✅	the engine seam for command behaviour; tested
EquipmentRuntime (engine owner)	✅	tested (test_runtime.cpp); secs_server runs entirely on it (live GEM300 demo passes)
register_default_handlers (the 56 GEM handlers as a library fn)	✅	src/gem/default_handlers.cpp; tested (test_default_handlers.cpp)
gRPC/protobuf toolchain (Dockerfile + CMake codegen)	✅	grpc++ 1.51 / protoc 3.21; opt-in SECSGEM_DAEMON, graceful skip without grpc
secs_gemd: SetVariables / FireEvent / GetControlState	✅	format-aware (converts to each variable's declared SECS-II format) and thread-safe (name/format maps snapshotted at construction; all writes post to the io thread). In-process gRPC tests (test_daemon_service.cpp, 16 assertions)
Daemon interop vs secsgem-py reference host	✅	interop/daemon_interop.py (via gemd compose service): gRPC SetVariables(ChamberPressure=2.5) + FireEvent → host receives S6F11 CEID 300 carrying <F4 2.5> — value and declared format flow gRPC→engine→HSMS→host
Daemon interop vs secs4j (Java)	⬜	mirror the secsgem-py harness against interop/secs4j
Subscribe host→tool command stream	⬜	design settled (HCACK-4, see below); not implemented
Remaining universal RPCs (GetVariables, alarms, RequestControlState, WatchHealth)	⬜	see plan
Python client package (the "beautiful API")	⬜	thin wrapper over generated stubs

Known issues (found in the 2026-06-10 audit; honest list)

• ✅ GetControlState cross-thread read. Fixed 2026-06-10: the runtime keeps an atomic control-state mirror updated via an add_state_change_handler observer (HandlerSlot primary+observers pattern), so the mirror survives register_default_handlers claiming the primary slot. control_state() is now safe from any thread.
• ⬜ Alarms have no name key. equipment.yaml alarms carry only numeric id + freetext text (matches SEMI: ALID/ALTX; there is no standard short name). The name-based SetAlarm/ClearAlarm RPCs need an optional local name: field in the alarm config (fallback: stringified id).
• ⬜ pvd_tool predates the behaviour hook. It still hard-codes if (rcmd=="START") recipe->start(...) in a router handler. Migrate it to commands.set_handler so the flagship example showcases the intended seam.
• ⬜ Interop harnesses are manual. daemon_interop.py (and the older host/server harnesses) run via ad-hoc compose invocations; there is no tools/run_interop.sh or CI lane that runs them. Add one script + CI job.
• ⬜ TSan lane doesn't cover the daemon. secs_gemd_tests should also be built/run under -DSECSGEM_TSAN=ON once the control-state mirror lands.
• ⚠️ macOS bind-mount staleness can break Docker builds mid-edit (a build reading a half-synced source file). Not a product bug; re-run the build.

The Subscribe design (settled — implement to this)

S2F42 is an acknowledgement, not a completion: SEMI separates "I accept your command" from "the work finished". The conformant, non-blocking flow:

1. Host sends S2F41 START. The engine's on_command handler (registered by the daemon) runs on the io thread.
2. If no tool client is subscribed → fall back to the YAML declarative ack. If a tool is subscribed → push the command onto its Subscribe stream and return HCACK=4 (AcceptedWillFinishLater) immediately — never block the io thread or the T3 window on the tool.
3. The tool does the work and reports the outcome via FireEvent (success event) / SetAlarm (failure) — exactly how secsgem-py applications and commercial gateways do it.
4. CompleteCommand therefore only correlates/audits the command lifecycle in v1. A synchronous gating mode (tool decides HCACK 0/2 before the S2F42 goes out) requires a deferred-reply mechanism in the engine — explicitly a v2 refinement, not needed for conformance.

Open sub-decisions to settle while implementing:

• Per-command routing (subscribe to specific RCMDs?) or one firehose? (v1: firehose.)
• Reconnect semantics: buffer commands while no subscriber (bounded queue + declarative fallback after timeout) or reject with HCACK 2? Must be decided and TESTED before calling the stream production-ready.

Plan — ordered next steps

Phase 0 — structural debts (from the 2026-06-10 design review; pay before sprinting)

The review's verdict: architecture and API bets are sound, but two structural debts tax every later phase, and the most valuable tests aren't automated.

1. 🚧 Multi-observer callbacks (THE structural blocker — hit twice already). HandlerSlot (primary slot keeps legacy set_ semantics; append-only add_ observers survive it) — done for ControlStateMachine + PJ/CJ stores, plus runtime atomic control-state mirror (race retired) and add_link_observer (WatchHealth foundation). ⬜ Remaining: roll the same 3-line pattern onto the other single-slot classes (comm-state, EPT, exceptions, substrates, modules, carriers, E84) as each phase needs them — mechanical now that the type exists.
2. 🚧 CI the interop + conformance harnesses. tools/run_interop.sh ✅ — one command runs ALL nine validation steps (build, unit, daemon-unit, py-host 31 checks, conformance 47, daemon bridge, spool restart, tshark, secs4j 55) with a PASS/FAIL summary; verified green end-to-end 2026-06-10. CI ✅ added but UNVERIFIED until pushed: grpc deps + secs_gemd_tests in the build job (fails loudly if the daemon silently drops out), and a new python-interop lane (py-host + conformance + daemon harness against localhost, no docker-in-docker). ⬜ Verify the lanes on the first push.
3. ✅ Fix CompleteCommand proto comment — it described the rejected blocking model; now states the HCACK-4 contract.
4. ⬜ Table-driven handler conformance test ((request, expected-reply-shape) pairs through router.dispatch for broad coverage of the 56 handlers) + message-level golden wire frames (codec KATs exist; message-level don't).
5. ⬜ Decompose register_default_handlers per GEM capability (it is a relocated main(), not a designed component) and replace magic constants (SVIDs 1/2 refresh(), CEIDs 400/401) with YAML role bindings (control_state_svid:, cj_executing_ceid: …). Gradual; aligns with the capability structure GEM itself defines (S1F19) and enables vendor subsetting.
6. ⬜ Standardize the mutable-read pattern for daemon RPCs: post-to-io + future with deadline (always truthful; latency irrelevant at SECS rates). First consumer: GetVariables (Phase A1) — set the precedent there.
7. ⬜ Move apps/equipment_service.hpp into the library tree (include/secsgem/daemon/) once Phase B grows it; add a TSan-built run_async + concurrent-RPC daemon test (today's daemon tests only poll()).
8. 🚧 Validate names are identifier-safe in ConfigValidator (the Python client's kwargs API depends on it) — ⬜. Generalized format-compliance property test (iterates ALL configured variables via gRPC, asserts each keeps its declared wire format) — ✅, plus an unset-Value guard at the RPC edge (was silently writing ASCII "").

Phase A — finish the universal daemon surface (small, unblock vendors)

1. ⬜ GetVariables — needs the reverse Item → proto Value conversion (read via post-to-io + future, or serve from a daemon-side cache of last set values; decide and document).
2. ⬜ Alarm name: config field + SetAlarm/ClearAlarm RPCs + tests.
3. ⬜ RequestControlState (operator online/offline) + control-state atomic mirror (fixes the known race) + WatchHealth stream (link state from the selected/closed handlers, spool depth, control state).
4. ⬜ Extend test_daemon_service.cpp + daemon_interop.py for all of the above.

Phase B — the command stream (the big one)

5. ⬜ Implement Subscribe/CompleteCommand per the design above, including the no-subscriber fallback and bounded buffering. In-process gRPC tests: command arrives on stream; HCACK 4 on the wire; declarative fallback when unsubscribed.
6. ⬜ Extend daemon_interop.py: secsgem-py host sends S2F41 START → gRPC tool receives it on the stream → tool fires completion event → host sees S6F11. (The full conformant loop against the reference implementation.)
7. ⬜ Java interop: secs4j host variant of the same scenario.

Phase C — the beautiful Python client

8. ⬜ clients/python/ package (pip install secsgem-client): wraps generated stubs in the agreed API — eq.set(chamber_pressure=2.5), eq.fire("wafer_complete", thickness=1.2), eq.alarm("pressure_high"), @eq.on("START") consuming the stream, eq.health(). Pure Python (no compiled ext). Ship stubs pre-generated.
9. ⬜ Example: rewrite a minimal pvd_tool-equivalent in ~40 lines of Python against the daemon; also migrate the C++ pvd_tool to set_handler.

Phase D — GEM300 in-the-loop (process/carrier tools)

10. ⬜ Settle job/carrier semantics (who acks S16F5/S3F17, gate vs observe — see proto comments), then wire ProcessJob/CarrierAction onto the stream + ReportProcessJob/ReportCarrier into the PJ/CJ/carrier stores.
11. ⬜ Recipe download (ProcessProgram on the stream when S7F3 lands) and EC-change notification (ConstantChange when S2F15 lands).
12. ⬜ Interop scenarios for jobs/carriers vs secsgem-py + secs4j.

Phase E — hardening & operations

13. ⬜ gRPC exposure: default to localhost + document UDS; optional TLS creds.
14. ⬜ tools/run_interop.sh + CI lanes: all interop harnesses + TSan daemon lane.
15. ⬜ Daemon Prometheus metrics + supervised deployment recipe (systemd unit).
16. ⬜ Remaining Layer-1 API: traces, limits, substrates/modules, terminal services, spool depth/flush, Describe RPC.

Phase F — fab acceptance (parallel track; the hard gate)

• ⚠️ Standards correctness remains unverified against SEMI texts (behaviour reconstructed without the standards; interop with secsgem-py/secs4j/Wireshark mitigates but does not prove). The #1 fab-readiness risk; needs real standards access and/or a fab's MES qualification run (docs/MES_INTEROP.md).
• ⬜ GEM compliance statement + manual matching the tool's data dictionary.
• ⬜ SECS-I serial driver (asio serial_port adapter; FSM done) — only if a target tool uses RS-232.