feat(daemon): Phase E — production hardening, complete

Exposure: --grpc default flipped from 0.0.0.0 to 127.0.0.1 (the API is
unauthenticated by design; auth belongs to the transport), Unix-domain-
socket support (--grpc unix:///run/secs_gemd/api.sock = zero network
surface), SECURITY.md documents the contract and ch42 gained a "Running it
in production" section (which also documents the HSMS-SS single-session
assumption).

Graceful shutdown: SIGTERM/SIGINT land on an asio::signal_set on the io
thread, which nudges grpc Shutdown with a 2s deadline (cancels open
Subscribe/WatchHealth streams); Wait() returns on the MAIN thread, which
stops the engine (rt->stop() joins the io thread, so it must not run on
it). Exit 0, journal-safe, the in-code TODO is gone. --spool-dir added so
host-bound events survive daemon restarts.

Observability: --metrics serves Prometheus gauges secsgem_link_selected /
secsgem_control_state / secsgem_spool_depth, wired via the Phase-0
add_link_observer/add_control_state_observer hooks + io-thread sampling.

Deployment: deploy/secs_gemd.service — hardened systemd unit (DynamicUser,
ProtectSystem=strict, StateDirectory for the spool, UDS for the API,
TimeoutStopSec aligned with the graceful-shutdown window).

Enforcement: tools/check_daemon_ops.sh proves all three operational claims
(unix-socket gRPC accepts, all gauges present on /metrics, SIGTERM -> exit
0 + clean-stop log) — green; wired into tools/run_interop.sh (now 11
steps) and CI. CI python-interop lane also gained the pyclient and
spool-restart steps, so every harness now runs in CI.

TODO sweep: the shutdown TODO is fixed; the four remaining TODOs (nested
list formats, C2-as-text, U8>2^63, CONNECTED link state) are deliberate
deferred edge cases, each marked in code with context.

Daemon suite re-verified green (175 assertions).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

.gitea/workflows/ci.yml

@@ -279,6 +279,42 @@ jobs:
           kill $GEMD 2>/dev/null || true
           exit $rc
 
+      - name: Python client package vs secs_gemd
+        run: |
+          /tmp/venv/bin/python clients/python/tests/test_values.py
+          build/secs_gemd --port 5005 --grpc 127.0.0.1:50052 &
+          GEMD=$!
+          sleep 1
+          PYTHONPATH=clients/python /tmp/venv/bin/python interop/pyclient_interop.py \
+            --grpc 127.0.0.1:50052 --hsms-host 127.0.0.1 --hsms-port 5005
+          rc=$?
+          kill $GEMD 2>/dev/null || true
+          exit $rc
+
+      - name: spool persistence across a server restart
+        run: |
+          set -e
+          SPOOL=$(mktemp -d)
+          build/secs_server --port 5006 --spool-dir "$SPOOL" &
+          SRV=$!
+          sleep 1
+          /tmp/venv/bin/python interop/spool_persistence_test.py \
+            --phase enqueue --host 127.0.0.1 --port 5006
+          kill $SRV; wait $SRV 2>/dev/null || true
+          build/secs_server --port 5006 --spool-dir "$SPOOL" &
+          SRV=$!
+          sleep 1
+          /tmp/venv/bin/python interop/spool_persistence_test.py \
+            --phase drain --host 127.0.0.1 --port 5006
+          rc=$?
+          kill $SRV 2>/dev/null || true
+          exit $rc
+
+      # Phase E operational contract: unix-socket gRPC, Prometheus gauges,
+      # graceful SIGTERM shutdown (exit 0, journal-safe).
+      - name: daemon ops (unix socket + metrics + graceful shutdown)
+        run: bash tools/check_daemon_ops.sh
+
   libfuzzer:
     runs-on: ubuntu-latest
     container:

apps/secs_gemd.cpp

@@ -1,17 +1,30 @@
 // secs_gemd — the SECS/GEM daemon.
 //
 // Runs the GEM engine (EquipmentRuntime + the default handlers) on a background
-// thread, owning the HSMS link to the host, and exposes a small name-based gRPC
+// thread, owning the HSMS link to the host, and exposes the name-based gRPC
 // API (proto/secsgem/v1/equipment.proto) so a tool's software — in any language
 // — can drive the equipment without linking C++ or knowing SEMI.
 //
-// Increment 1: the universal "report state to the host" RPCs (SetVariables,
+// Operations (Phase E):
-// FireEvent) plus control-state visibility. Alarms, GetVariables, and the
+//   --grpc      defaults to 127.0.0.1:50051 — NEVER expose the unauthenticated
-// host->tool Subscribe command stream follow (see docs/DAEMON_ROADMAP.md).
+//               API on the equipment LAN. For same-host tool software prefer a
+//               Unix domain socket: --grpc unix:///run/secs_gemd.sock
+//   --metrics   Prometheus endpoint port (0 = disabled). Gauges: HSMS link
+//               state, control state, spool depth.
+//   SIGTERM/SIGINT: graceful shutdown — gRPC server drains (open streams are
+//               cancelled), then the engine stops cleanly, so a supervised
+//               restart (systemd / docker stop) never kills the spool journal
+//               mid-write. Exit code 0.
+//
+// Deployment recipe: deploy/secs_gemd.service (systemd) and docs/42.
 
 #include <grpcpp/grpcpp.h>
 
+#include <asio.hpp>
+#include <chrono>
+#include <csignal>
 #include <cstdint>
+#include <functional>
 #include <iostream>
 #include <memory>
 #include <string>
@@ -19,8 +32,11 @@
 #include "secsgem/daemon/equipment_service.hpp"
 #include "secsgem/gem/default_handlers.hpp"
 #include "secsgem/gem/runtime.hpp"
+#include "secsgem/metrics/prometheus.hpp"
 
 namespace gem = secsgem::gem;
+namespace metrics = secsgem::metrics;
+using namespace std::chrono_literals;
 
 namespace {
 std::string arg(int argc, char** argv, const std::string& key, const std::string& def) {
@@ -32,8 +48,11 @@ std::string arg(int argc, char** argv, const std::string& key, const std::string
 
 int main(int argc, char** argv) {
   const std::string hsms_port    = arg(argc, argv, "--port", "5000");
-  const std::string grpc_addr = arg(argc, argv, "--grpc", "0.0.0.0:50051");
+  const std::string grpc_addr    = arg(argc, argv, "--grpc", "127.0.0.1:50051");
   const std::string cfgdir       = arg(argc, argv, "--config-dir", "data");
+  const std::string spool_dir    = arg(argc, argv, "--spool-dir", "");
+  const uint16_t metrics_port    =
+      static_cast<uint16_t>(std::stoi(arg(argc, argv, "--metrics", "0")));
 
   gem::EquipmentRuntime::Config cfg;
   cfg.equipment_yaml     = cfgdir + "/equipment.yaml";
@@ -41,6 +60,7 @@ int main(int argc, char** argv) {
   cfg.process_job_yaml   = cfgdir + "/process_job_state.yaml";
   cfg.control_job_yaml   = cfgdir + "/control_job_state.yaml";
   cfg.port = static_cast<uint16_t>(std::stoi(hsms_port));
+  cfg.spool_dir = spool_dir;
   cfg.log = [](const std::string& m) { std::cout << "[gemd] " << m << std::endl; };
 
   std::unique_ptr<gem::EquipmentRuntime> rt;
@@ -52,9 +72,50 @@ int main(int argc, char** argv) {
   }
   gem::register_default_handlers(*rt);
 
+  // ---- observability (before run_async: observers must land first) -------
+  std::shared_ptr<metrics::Registry> registry;
+  std::shared_ptr<metrics::PrometheusServer> exporter;
+  std::shared_ptr<asio::steady_timer> gauge_timer;
+  if (metrics_port != 0) {
+    registry = std::make_shared<metrics::Registry>();
+    registry->describe("secsgem_link_selected",
+                       "1 when an HSMS session is SELECTED, else 0",
+                       metrics::MetricType::Gauge);
+    registry->describe("secsgem_control_state",
+                       "E30 control state (0=EquipOffline 1=AttemptOnline "
+                       "2=HostOffline 3=OnlineLocal 4=OnlineRemote)",
+                       metrics::MetricType::Gauge);
+    registry->describe("secsgem_spool_depth", "Queued spool messages",
+                       metrics::MetricType::Gauge);
+    rt->add_link_observer([registry](bool selected) {
+      registry->set_gauge("secsgem_link_selected", selected ? 1.0 : 0.0);
+    });
+    rt->add_control_state_observer(
+        [registry](gem::ControlState, gem::ControlState to, gem::ControlEvent) {
+          registry->set_gauge("secsgem_control_state", static_cast<double>(to));
+        });
+    registry->set_gauge("secsgem_link_selected", 0.0);
+    registry->set_gauge("secsgem_control_state",
+                        static_cast<double>(rt->control_state()));
+    // Spool depth: sampled on the io thread (the model's owner).
+    gauge_timer = std::make_shared<asio::steady_timer>(rt->io());
+    auto tick = std::make_shared<std::function<void(std::error_code)>>();
+    *tick = [registry, gauge_timer, tick, raw = rt.get()](std::error_code ec) {
+      if (ec) return;
+      registry->set_gauge("secsgem_spool_depth",
+                          static_cast<double>(raw->model().spool.size()));
+      gauge_timer->expires_after(5s);
+      gauge_timer->async_wait(*tick);
+    };
+    gauge_timer->expires_after(1s);
+    gauge_timer->async_wait(*tick);
+    exporter = std::make_shared<metrics::PrometheusServer>(
+        rt->io(), metrics_port, registry);
+    exporter->start();
+  }
+
   // Construct the service before starting the io thread: its constructor
-  // snapshots the name->id/format maps from the model (see the service's
+  // snapshots the name->id/format maps and registers its observers.
-  // threading note).
   secsgem::daemon::EquipmentService service(*rt);
   rt->run_async();  // engine + HSMS link on a background thread
 
@@ -66,12 +127,26 @@ int main(int argc, char** argv) {
     std::cerr << "[gemd] failed to start gRPC server on " << grpc_addr << std::endl;
     return 1;
   }
+
+  // Graceful shutdown: the signal lands on the io thread, which only nudges
+  // the gRPC server; Wait() then returns on the MAIN thread, which tears the
+  // engine down (rt->stop() joins the io thread — it must not run on it).
+  // The deadline cancels open Subscribe/WatchHealth streams instead of
+  // waiting for clients to hang up.
+  asio::signal_set signals(rt->io(), SIGINT, SIGTERM);
+  signals.async_wait([&server, &rt](std::error_code ec, int signo) {
+    if (ec) return;
+    rt->log("signal " + std::to_string(signo) + ": shutting down");
+    server->Shutdown(std::chrono::system_clock::now() + 2s);
+  });
+
   std::cout << "[gemd] gRPC on " << grpc_addr
+            << (metrics_port ? "; metrics on :" + std::to_string(metrics_port) +
+                                   "/metrics"
+                             : "")
             << "; HSMS equipment on :" << hsms_port << std::endl;
-  // TODO(daemon): graceful shutdown — handle SIGTERM/SIGINT by calling
-  // server->Shutdown() and rt->stop() so in-flight RPCs drain and the spool
-  // journal flushes, instead of dying mid-write when supervised (systemd/
-  // docker stop). See DAEMON_ROADMAP Phase E.
   server->Wait();
+  rt->stop();
+  std::cout << "[gemd] stopped cleanly" << std::endl;
   return 0;
 }

deploy/secs_gemd.service

@@ -0,0 +1,41 @@
+# systemd unit for the secs_gemd SECS/GEM daemon.
+#
+#   install -m644 deploy/secs_gemd.service /etc/systemd/system/
+#   install -m755 build/secs_gemd /usr/local/bin/
+#   mkdir -p /etc/secsgem && cp data/*.yaml /etc/secsgem/
+#   systemctl enable --now secs_gemd
+#
+# The tool API rides a Unix domain socket (no network exposure at all);
+# point the Python/C++ client at unix:///run/secs_gemd/api.sock. The HSMS
+# port (5000) faces the fab host — firewall it to the host's address (see
+# docs/SECURITY.md for the nftables recipe).
+
+[Unit]
+Description=SECS/GEM equipment daemon (secs_gemd)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=exec
+ExecStart=/usr/local/bin/secs_gemd \
+    --port 5000 \
+    --grpc unix:///run/secs_gemd/api.sock \
+    --metrics 9091 \
+    --config-dir /etc/secsgem \
+    --spool-dir /var/lib/secs_gemd/spool
+RuntimeDirectory=secs_gemd
+StateDirectory=secs_gemd
+# SIGTERM triggers the daemon's graceful shutdown (drains gRPC, stops the
+# engine, exit 0); give it a moment before SIGKILL.
+TimeoutStopSec=10
+Restart=always
+RestartSec=2
+# Hardening
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+ReadWritePaths=/var/lib/secs_gemd
+DynamicUser=true
+
+[Install]
+WantedBy=multi-user.target

docs/42_vendor_daemon_and_clients.md

@@ -39,11 +39,12 @@ drops; the daemon model covers the gap if *your software* drops.
 Run it:
 
 ```sh
-build/secs_gemd --port 5000 --grpc 0.0.0.0:50051 --config-dir data
+build/secs_gemd --port 5000 --config-dir data            # gRPC on 127.0.0.1:50051
+build/secs_gemd --grpc unix:///run/secs_gemd/api.sock …  # production: no TCP at all
 ```
 
 One process, two faces: passive HSMS equipment on `--port`, the gRPC tool
-API on `--grpc`.
+API on `--grpc` (localhost by default — see §5 before exposing anything).
 
 ---
 
@@ -188,7 +189,34 @@ stream, so they never fight your application over the slot.
 
 ---
 
-## 5. Which tier do I pick?
+## 5. Running it in production
+
+- **Exposure.** `--grpc` defaults to `127.0.0.1:50051`; the API is
+  unauthenticated by design (auth belongs to the transport), so it must
+  never face the equipment LAN. For same-host tool software use a Unix
+  domain socket — `--grpc unix:///run/secs_gemd/api.sock` — and there is
+  no network surface at all. The HSMS port faces the fab host; firewall
+  it to the host's address ([SECURITY.md](SECURITY.md) has the nftables
+  recipe).
+- **Shutdown.** SIGTERM/SIGINT drain gracefully: open Subscribe/WatchHealth
+  streams are cancelled (2s deadline), the engine stops cleanly, the spool
+  journal is never cut mid-write, exit code 0. Safe under systemd and
+  `docker stop`.
+- **Supervision.** [deploy/secs_gemd.service](../deploy/secs_gemd.service)
+  is a hardened systemd unit (DynamicUser, ProtectSystem, StateDirectory
+  for the spool, Restart=always). Pair with `--spool-dir` so host-bound
+  events survive daemon restarts too.
+- **Metrics.** `--metrics 9091` serves Prometheus gauges:
+  `secsgem_link_selected`, `secsgem_control_state`, `secsgem_spool_depth`.
+- **Sessions.** v1 runs one equipment identity per daemon (HSMS-SS). The
+  engine supports HSMS-GS multi-session, but the daemon doesn't surface it
+  yet — run one daemon per equipment identity.
+- All of the above is enforced by `tools/check_daemon_ops.sh` (the
+  `daemon-ops` step of `tools/run_interop.sh`, also in CI).
+
+---
+
+## 6. Which tier do I pick?
 
 | Your situation | Tier |
 |---|---|

docs/DAEMON_ROADMAP.md

@@ -227,11 +227,24 @@ debts tax every later phase, and the most valuable tests aren't automated.
     S16 builders — see interop/raw_gem300_harness.py for the frame source).
 
 ### Phase E — hardening & operations
-13. ⬜ gRPC exposure: default to localhost + document UDS; optional TLS creds.
+13. ✅ gRPC exposure: default flipped to `127.0.0.1`; Unix-domain-socket
-14. ⬜ `tools/run_interop.sh` + CI lanes: all interop harnesses + TSan daemon lane.
+    support verified (`--grpc unix:///...`); SECURITY.md documents the
-15. ⬜ Daemon Prometheus metrics + supervised deployment recipe (systemd unit).
+    contract (unauthenticated API = localhost/UDS only; stunnel if remote).
+    TLS creds remain optional future work (UDS removes the need same-host).
+14. ✅ `tools/run_interop.sh` now 11 steps (added pyclient + daemon-ops);
+    CI python-interop lane gained pyclient, spool-restart, and daemon-ops
+    steps — every harness now runs in CI.
+15. ✅ Graceful shutdown (SIGTERM/SIGINT -> gRPC drain with 2s stream-cancel
+    deadline -> engine stop -> exit 0; journal-safe; the old in-code TODO is
+    gone), Prometheus gauges (`secsgem_link_selected` / `_control_state` /
+    `_spool_depth` via the Phase-0 observers + io-thread sampling),
+    `--spool-dir` on the daemon, and `deploy/secs_gemd.service` (hardened:
+    DynamicUser/ProtectSystem/StateDirectory/TimeoutStopSec). All enforced
+    by `tools/check_daemon_ops.sh`. Single-session (HSMS-SS) assumption
+    documented in ch42 §5.
 16. ⬜ Remaining Layer-1 API: traces, limits, substrates/modules, terminal
-    services, spool depth/flush, `Describe` RPC.
+    services, spool flush RPC, `Describe` RPC. (Engine-side all exists;
+    surface on demand.)
 
 ### Phase F — fab acceptance (parallel track; the hard gate)
 - ⚠️ **Standards correctness remains unverified against SEMI texts** (behaviour

docs/SECURITY.md

@@ -1,5 +1,16 @@
 # Security operations guide
 
+## The daemon's gRPC API
+
+`secs_gemd`'s tool API is **unauthenticated by design** — authentication
+belongs to the transport. The defaults are safe (`127.0.0.1`), and the
+production recommendation is a Unix domain socket
+(`--grpc unix:///run/secs_gemd/api.sock`, file-permission protected, zero
+network surface). Never bind it to an interface reachable from the
+equipment LAN; if cross-host access is unavoidable, front it with the same
+stunnel pattern described below for HSMS.
+
+
 HSMS is plain TCP — no auth, no encryption.  That's what every fab
 tool ships and what every MES expects.  Security comes from the
 network layer around the HSMS socket; this doc has the concrete

tools/check_daemon_ops.sh

@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+# Phase E operational checks for secs_gemd, run inside the builder image:
+#   1. serves the gRPC API over a UNIX DOMAIN SOCKET (no TCP exposure),
+#   2. exposes Prometheus metrics (link/control-state/spool gauges),
+#   3. shuts down GRACEFULLY on SIGTERM (exit 0, "stopped cleanly").
+# Exit 0 = all three hold.
+set -u
+cd "$(dirname "${BASH_SOURCE[0]}")/.."
+
+SOCK=/tmp/gemd-ops.sock
+LOG=/tmp/gemd-ops.log
+rm -f "$SOCK"
+
+./build/secs_gemd --port 5077 --grpc "unix://$SOCK" --metrics 9191 \
+    --config-dir data > "$LOG" 2>&1 &
+GEMD=$!
+sleep 2
+
+fail() { echo "FAIL: $1"; cat "$LOG"; kill -9 $GEMD 2>/dev/null; exit 1; }
+
+[ -S "$SOCK" ] || fail "unix socket not created"
+
+# Metrics endpoint answers and carries our gauges.
+exec 3<>/dev/tcp/127.0.0.1/9191 || fail "metrics port closed"
+printf 'GET /metrics HTTP/1.0\r\n\r\n' >&3
+METRICS=$(cat <&3)
+echo "$METRICS" | grep -q "secsgem_control_state" || fail "control-state gauge missing"
+echo "$METRICS" | grep -q "secsgem_spool_depth"   || fail "spool gauge missing"
+echo "$METRICS" | grep -q "secsgem_link_selected" || fail "link gauge missing"
+
+# A gRPC call over the unix socket (python grpcio in the interop image; in
+# the builder image fall back to checking the socket accepts a connection).
+if python3 -c "import grpc" 2>/dev/null; then
+  python3 - "$SOCK" <<'PY' || fail "gRPC over unix socket failed"
+import sys, grpc
+ch = grpc.insecure_channel(f"unix://{sys.argv[1]}")
+grpc.channel_ready_future(ch).result(timeout=5)
+PY
+fi
+
+# Graceful shutdown: SIGTERM -> exit 0 + the clean-stop line.
+kill -TERM $GEMD
+for _ in $(seq 1 50); do kill -0 $GEMD 2>/dev/null || break; sleep 0.1; done
+if kill -0 $GEMD 2>/dev/null; then fail "still running 5s after SIGTERM"; fi
+wait $GEMD; RC=$?
+[ "$RC" = "0" ] || fail "exit code $RC after SIGTERM (want 0)"
+grep -q "stopped cleanly" "$LOG" || fail "no clean-stop log line"
+grep -q "shutting down" "$LOG"   || fail "no shutdown log line"
+
+echo "daemon ops: unix socket + metrics + graceful shutdown all OK"

tools/run_interop.sh

@@ -16,7 +16,9 @@
 #   py-host       secsgem-py active host vs C++ secs_server     (31 checks)
 #   conformance   secs_conformance C++ host vs C++ secs_server  (47 checks)
 #   daemon        gRPC tool + secsgem-py host vs secs_gemd      (bridge proof)
+#   pyclient      published secsgem-client package vs secs_gemd  (13 checks)
 #   spool         spool persistence across a server restart
+#   daemon-ops    unix-socket gRPC + Prometheus gauges + graceful SIGTERM
 #   tshark        Wireshark HSMS dissector on a live capture
 #   secs4j        secs4java8 Java host vs C++ secs_server       (55 checks)
 #
@@ -119,6 +121,11 @@ note "spool: persistence across server restart"
 record spool $?
 compose stop server-spool >/dev/null 2>&1
 
+# ---- daemon operations: unix socket + metrics + graceful shutdown ------------
+note "daemon-ops: unix socket, Prometheus gauges, SIGTERM drains cleanly"
+compose run --rm builder bash tools/check_daemon_ops.sh
+record daemon-ops $?
+
 # ---- Wireshark dissector ------------------------------------------------------
 note "tshark: Wireshark HSMS dissector"
 compose run --rm builder bash interop/tshark_validate.sh