secs-gem/docs/DAEMON_ROADMAP.md

Vendor Daemon & gRPC API — Status and Roadmap to Fab-Readiness

This is a forward-looking roadmap, not a description of shipped behaviour. Every item carries a status marker. Do not read an item as "done" unless it says ✅. (Written 2026-06-10.)

Status legend: ✅ done · 🚧 in progress · ⬜ planned · ⚠️ risk/unknown

What this is

A vendor-facing daemon that runs the SECS/GEM engine as its own process and exposes a small, name-based, language-agnostic API over gRPC, so a tool's control software (in any language) can drive the equipment without linking C++ or knowing SEMI. See proto/secsgem/v1/equipment.proto for the API.

The point of the daemon model: it owns the durable HSMS relationship with the host and stays conformant while the tool software restarts/upgrades/crashes.

Current status (2026-06-10)

Piece	Status	Notes
proto/secsgem/v1/equipment.proto	✅	v1 API surface designed (universal + carrier/recipe/job tiers)
HostCommandRegistry::set_handler behaviour hook	✅	the engine seam the daemon binds to; tested
EquipmentRuntime (engine owner)	✅	infra + outbound API built & tested (tests/test_runtime.cpp); secs_server now runs entirely on it (verified by the live server↔client GEM300 demo — full job/spool/control-state flow, client exit 0).
register_default_handlers in the library (so the daemon reuses the 56 handlers)	✅	relocated into src/gem/default_handlers.cpp (programmatic move, zero retype); secs_server reduced to ~113 lines and calls it. Tested (tests/test_default_handlers.cpp: S1F1→S1F2, S2F41→on_command hook) + live GEM300 demo still passes.
gRPC/protobuf in toolchain (Dockerfile + CMake)	✅	image rebuilt (grpc++ 1.51, protoc 3.21); CMake proto codegen wired (opt-in SECSGEM_DAEMON, graceful skip without grpc).
secs_gemd daemon	🚧	runs — engine + HSMS on a background thread, serves gRPC. RPCs done + tested over a real in-process channel: SetVariables (name-resolved, value→Item, lands in the model), FireEvent (name→CEID), GetControlState. tests/test_daemon_service.cpp (15 assertions). Still TODO: GetVariables (needs Item→Value), SetAlarm/ClearAlarm (needs an alarm name config field), RequestControlState, WatchHealth, and the host→tool Subscribe command stream + CompleteCommand (the hard part).
secs_gemd daemon implementing the service	⬜	translate RPCs ↔ runtime; stream host requests
Reference client library (Python)	⬜	thin wrapper over generated stubs
Daemon interop test vs secsgem-py	✅	interop/daemon_interop.py — a gRPC tool + a secsgem-py active host both drive a live secs_gemd; proves a gRPC SetVariables+FireEvent reaches the reference host as S6F11 over HSMS. Run via the gemd compose service. Java (secs4j) equivalent: ⬜.

Nothing in the proto is wired to the engine yet. The engine itself is broad (56 wire handlers across S1/2/3/5/6/7/10/14/16; all GEM300 stores) — the daemon is about exposing that, not building it.

Gaps to fab-readiness

Layer 0 — Make it run at all (blocks everything)

• ⬜ Extract EquipmentRuntime from apps/secs_server.cpp (io_context, Server, model, router, emit lambdas, set_handler). Reduce secs_server to a thin main() over it. Verify against the existing test suite.
• ⬜ Add gRPC/protobuf to the Dockerfile + CMake codegen for the proto.
• ⬜ Implement secs_gemd: construct the runtime, io.run() on a background thread, map each RPC to a runtime call, route host requests onto the Subscribe stream via set_handler + the FSM change handlers.
• ⬜ One reference client (Python) proving the end-to-end loop.

Layer 1 — API completeness (engine supports these; surface them)

• ⬜ Job/carrier in-the-loop semantics. The proto has ProcessJob/ CarrierAction + report RPCs, but the exact contract is unspecified: who acks the host's S16F5/S3Fxx, whether the tool gates a job start or only observes, and timing vs. T3. Design this before implementing the daemon for process tools.
• ⬜ Trace data collection (engine: TraceStore, S2F23/S6F1).
• ⬜ Limits monitoring (engine: LimitMonitorStore, S2F45).
• ⬜ Substrate/E90 + module/E157 tracking (engine: SubstrateStore, ModuleStore).
• ⬜ Terminal services / operator messages (engine: S10F1–F6) — host↔tool HMI text.
• ⬜ Spool depth + force-flush API (engine: SpoolStore).
• ⬜ Describe RPC: enumerate configured variables/events/alarms/commands at runtime (diagnostics & tooling).

Layer 2 — Production hardening

• ⬜ gRPC auth / exposure. No auth today. Bind to a Unix domain socket or localhost-only, or add credentials. Never expose the API on the equipment LAN unauthenticated.
• ⬜ Subscribe reconnect/replay semantics. Define what happens to host requests (commands, jobs) if the tool client disconnects and reconnects: are they buffered/replayed, or dropped? Required for a 24/7 tool. (Correctness gap.)
• ⬜ Supervised deployment (systemd unit / container), auto-restart; rely on the existing spool persistence so queued host events survive a daemon restart.
• ⬜ Expose the existing Prometheus metrics + structured logs from the daemon.
• ⬜ Decide multi-host (HSMS-GS) story — engine supports it; v1 assumes one equipment/session. Probably fine; document the assumption.

Layer 3 — Actual fab acceptance (the hard gate)

• ⚠️ Standards correctness is unverified. The SECS/GEM behaviour in this repo was substantially reconstructed without access to the SEMI standard texts. Interop tests (secsgem-py, secs4java8, Wireshark) mitigate but do not prove conformance. Subtle wire/state-machine deviations could fail a real host. This is the #1 fab-readiness risk and it is verification, not features.
• ⬜ Pass a specific fab's MES qualification suite against their real host (see docs/MES_INTEROP.md for the punch-list). Fab acceptance is empirical and per-fab.
• ⬜ Produce the GEM compliance statement (S1F19/F20) + written GEM manual matching the tool's actual data dictionary.
• ⬜ Finish the SECS-I serial driver (FSM done; asio serial_port adapter missing) — only if a target tool uses RS-232 rather than HSMS/TCP.
• ⬜ Per-tool equipment.yaml authored to match the tool's real SVIDs/CEIDs/ ECIDs/alarms/recipes and the fab's spec (vendor work; the config validator helps).

Sequencing recommendation

Layer 0 in order (runtime → deps → daemon → client), then Layer 1's job/carrier semantics, then Layer 2 hardening. Layer 3 runs in parallel and is gated by access to real standards and a real host — treat it as the thing that decides whether any of this is truly fab-ready.