# Vendor Daemon & gRPC API — Status, Known Issues, and Plan to Fab-Readiness

> **This is a forward-looking roadmap, not a description of shipped behaviour.**
> Every item carries a status marker. Do not read an item as "done" unless it
> says ✅. (Last full audit: 2026-06-10.)
>
> Status legend: ✅ done · 🚧 in progress · ⬜ planned · ⚠️ risk/unknown

## What this is

A vendor-facing **daemon** (`secs_gemd`) that runs the SECS/GEM engine as its
own process and exposes a small, name-based, language-agnostic API over gRPC,
so a tool's control software (in any language) can drive the equipment without
linking C++ or knowing SEMI. See `proto/secsgem/v1/equipment.proto`.

The point of the daemon model: it owns the durable HSMS relationship with the
host and stays conformant while the tool software restarts/upgrades/crashes.

## Current status (2026-06-10, end of day)

| Piece | Status | Notes |
|---|---|---|
| `proto/secsgem/v1/equipment.proto` | ✅ | v1 surface designed: universal + carrier/recipe/job tiers, `Subscribe` stream, health |
| `HostCommandRegistry::set_handler` behaviour hook | ✅ | the engine seam for command behaviour; tested |
| `EquipmentRuntime` (engine owner) | ✅ | tested (`test_runtime.cpp`); `secs_server` runs entirely on it (live GEM300 demo passes) |
| `register_default_handlers` (the 56 GEM handlers as a library fn) | ✅ | `src/gem/default_handlers.cpp`; tested (`test_default_handlers.cpp`) |
| gRPC/protobuf toolchain (Dockerfile + CMake codegen) | ✅ | grpc++ 1.51 / protoc 3.21; opt-in `SECSGEM_DAEMON`, graceful skip without grpc |
| `secs_gemd`: `SetVariables` / `FireEvent` / `GetControlState` | ✅ | **format-aware** (converts to each variable's declared SECS-II format) and thread-safe (name/format maps snapshotted at construction; all writes post to the io thread). In-process gRPC tests (`test_daemon_service.cpp`, 16 assertions) |
| Daemon interop vs **secsgem-py** reference host | ✅ | `interop/daemon_interop.py` (via `gemd` compose service): gRPC `SetVariables(ChamberPressure=2.5)` + `FireEvent` → host receives `S6F11 CEID 300` carrying `<F4 2.5>` — value *and declared format* flow gRPC→engine→HSMS→host |
| Daemon interop vs **secs4j** (Java) | ⬜ | mirror the secsgem-py harness against `interop/secs4j` |
| `Subscribe` host→tool command stream | ⬜ | design settled (HCACK-4, see below); not implemented |
| Remaining universal RPCs (`GetVariables`, alarms, `RequestControlState`, `WatchHealth`) | ⬜ | see plan |
| Python client package (the "beautiful API") | ⬜ | thin wrapper over generated stubs |

## Known issues (found in the 2026-06-10 audit; honest list)

- 🚧 **`GetControlState` cross-thread read.** The gRPC handler reads the control
  FSM's state enum while the io thread may transition it — a narrow data race
  (single enum read; benign on every real ABI, but TSan-visible and sloppy).
  Fix: an atomic state mirror in `EquipmentRuntime`. NOTE: cannot piggyback on
  `ControlStateMachine::set_state_change_handler` — that is a single slot
  already owned by `register_default_handlers`; either add multi-handler
  support or update the mirror inside the runtime's own wiring.
- ⬜ **Alarms have no name key.** `equipment.yaml` alarms carry only numeric
  `id` + freetext `text` (matches SEMI: ALID/ALTX; there is no standard short
  name). The name-based `SetAlarm`/`ClearAlarm` RPCs need an optional local
  `name:` field in the alarm config (fallback: stringified id).
- ⬜ **`pvd_tool` predates the behaviour hook.** It still hard-codes
  `if (rcmd=="START") recipe->start(...)` in a router handler. Migrate it to
  `commands.set_handler` so the flagship example showcases the intended seam.
- ⬜ **Interop harnesses are manual.** `daemon_interop.py` (and the older
  host/server harnesses) run via ad-hoc compose invocations; there is no
  `tools/run_interop.sh` or CI lane that runs them. Add one script + CI job.
- ⬜ **TSan lane doesn't cover the daemon.** `secs_gemd_tests` should also be
  built/run under `-DSECSGEM_TSAN=ON` once the control-state mirror lands.
- ⚠️ **macOS bind-mount staleness can break Docker builds mid-edit** (a build
  reading a half-synced source file). Not a product bug; re-run the build.

## The `Subscribe` design (settled — implement to this)

`S2F42` is an *acknowledgement*, not a completion: SEMI separates "I accept
your command" from "the work finished". The conformant, non-blocking flow:

1. Host sends `S2F41 START`. The engine's `on_command` handler (registered by
   the daemon) runs on the io thread.
2. If no tool client is subscribed → fall back to the YAML declarative ack.
   If a tool is subscribed → push the command onto its `Subscribe` stream and
   **return `HCACK=4` (AcceptedWillFinishLater) immediately** — never block
   the io thread or the T3 window on the tool.
3. The tool does the work and reports the outcome via `FireEvent` (success
   event) / `SetAlarm` (failure) — exactly how secsgem-py applications and
   commercial gateways do it.
4. `CompleteCommand` therefore only correlates/audits the command lifecycle in
   v1. A *synchronous gating* mode (tool decides HCACK 0/2 before the S2F42
   goes out) requires a deferred-reply mechanism in the engine — explicitly a
   v2 refinement, not needed for conformance.

Open sub-decisions to settle while implementing:
- Per-command routing (subscribe to specific RCMDs?) or one firehose? (v1: firehose.)
- Reconnect semantics: buffer commands while no subscriber (bounded queue +
  declarative fallback after timeout) or reject with HCACK 2? Must be decided
  and TESTED before calling the stream production-ready.

## Plan — ordered next steps

### Phase A — finish the universal daemon surface (small, unblock vendors)
1. ⬜ `GetVariables` — needs the reverse `Item → proto Value` conversion
   (read via post-to-io + future, or serve from a daemon-side cache of last
   set values; decide and document).
2. ⬜ Alarm `name:` config field + `SetAlarm`/`ClearAlarm` RPCs + tests.
3. ⬜ `RequestControlState` (operator online/offline) + control-state atomic
   mirror (fixes the known race) + `WatchHealth` stream (link state from the
   selected/closed handlers, spool depth, control state).
4. ⬜ Extend `test_daemon_service.cpp` + `daemon_interop.py` for all of the above.

### Phase B — the command stream (the big one)
5. ⬜ Implement `Subscribe`/`CompleteCommand` per the design above, including
   the no-subscriber fallback and bounded buffering. In-process gRPC tests:
   command arrives on stream; HCACK 4 on the wire; declarative fallback when
   unsubscribed.
6. ⬜ Extend `daemon_interop.py`: secsgem-py host sends `S2F41 START` → gRPC
   tool receives it on the stream → tool fires completion event → host sees
   `S6F11`. (The full conformant loop against the reference implementation.)
7. ⬜ Java interop: `secs4j` host variant of the same scenario.

### Phase C — the beautiful Python client
8. ⬜ `clients/python/` package (`pip install secsgem-client`): wraps generated
   stubs in the agreed API — `eq.set(chamber_pressure=2.5)`, `eq.fire("wafer_complete", thickness=1.2)`,
   `eq.alarm("pressure_high")`, `@eq.on("START")` consuming the stream,
   `eq.health()`. Pure Python (no compiled ext). Ship stubs pre-generated.
9. ⬜ Example: rewrite a minimal `pvd_tool`-equivalent in ~40 lines of Python
   against the daemon; also migrate the C++ `pvd_tool` to `set_handler`.

### Phase D — GEM300 in-the-loop (process/carrier tools)
10. ⬜ Settle job/carrier semantics (who acks S16F5/S3F17, gate vs observe —
    see proto comments), then wire `ProcessJob`/`CarrierAction` onto the
    stream + `ReportProcessJob`/`ReportCarrier` into the PJ/CJ/carrier stores.
11. ⬜ Recipe download (`ProcessProgram` on the stream when S7F3 lands) and
    EC-change notification (`ConstantChange` when S2F15 lands).
12. ⬜ Interop scenarios for jobs/carriers vs secsgem-py + secs4j.

### Phase E — hardening & operations
13. ⬜ gRPC exposure: default to localhost + document UDS; optional TLS creds.
14. ⬜ `tools/run_interop.sh` + CI lanes: all interop harnesses + TSan daemon lane.
15. ⬜ Daemon Prometheus metrics + supervised deployment recipe (systemd unit).
16. ⬜ Remaining Layer-1 API: traces, limits, substrates/modules, terminal
    services, spool depth/flush, `Describe` RPC.

### Phase F — fab acceptance (parallel track; the hard gate)
- ⚠️ **Standards correctness remains unverified against SEMI texts** (behaviour
  reconstructed without the standards; interop with secsgem-py/secs4j/Wireshark
  mitigates but does not prove). The #1 fab-readiness risk; needs real
  standards access and/or a fab's MES qualification run (`docs/MES_INTEROP.md`).
- ⬜ GEM compliance statement + manual matching the tool's data dictionary.
- ⬜ SECS-I serial driver (asio `serial_port` adapter; FSM done) — only if a
  target tool uses RS-232.