B7 — the daemon's HSMS face under the Java reference: Dockerfile.server now
bakes secs_gemd alongside secs_server (grpc deps in both stages), and
secs4j_validate.sh gains TARGET=gemd to point the 55-check secs4java8 suite
at the daemon instead. Result: 55/55 green. With secsgem-py already
validating both faces, byte-identical GEM between secs_server and secs_gemd
is now proven by both reference implementations, not inferred from shared
code. CI runs the daemon target as an extra step (image layers shared).
Second client — clients/cpp: a header-only C++ twin of the Python client
over the same proto. eq.set("ChamberPressure", 2.5) with bare literals
(integral/floating dispatch avoids variant ambiguity), get/fire/alarm/
clear, control_state/request_control_state/health, on("START", fn) +
listen()/listen_async()/stop() with auto-CompleteCommand, SecsGemError
carrying the daemon's message. cpp_mini_tool (~30 lines) mirrors the
Python mini_tool. Tested end-to-end over real loopback TCP against the
service inside secs_gemd_tests — now 4 cases / 141 assertions — including
set/get round-trips, error text, alarm-by-name into the model, health,
and the full HCACK-4 command loop with parameters.
(Build note: two grpc-heavy TUs at -O3 OOM even at -j2 on Docker Desktop;
built -j1. Known environment limitation, roadmap-documented.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
clients/python: pip-installable "secsgem-client", pure Python (stubs
pre-generated from equipment.proto, import made package-relative; no
compiled extension, no SEMI knowledge, no C++ toolchain). The API the whole
effort aimed at:
eq = Equipment("localhost:50051")
eq.set(ChamberPressure=2.5); eq["WaferCounter"] = 7
eq.fire("ProcessStarted", ChamberPressure=2.75)
eq.alarm("chiller_temp_high"); eq.clear("chiller_temp_high")
@eq.on("START")
def start(cmd): ... # auto-CompleteCommand after return
eq.listen(background=True)
eq.control_state; eq.request_control_state("HOST_OFFLINE"); eq.health()
Errors raise SecsGemError carrying the daemon's message ("no variable named
..."). bool checked before int in conversion (isinstance(True, int)).
examples/mini_tool.py is a complete GEM tool in ~25 lines.
PROOF — interop/pyclient_interop.py drives the PUBLISHED package (not raw
stubs) against a live secs_gemd with secsgem-py as the fab host: 13 checks
all green on first run — set/get round-trips, item syntax, SecsGemError on
unknown names, control state, health, fire->S6F11 on the host's wire,
alarm/clear->S5F1 with correct set bit, the full command loop (host S2F41 ->
HCACK=4 -> @eq.on handler -> completion event back at the host), operator
offline. Conversion layer unit-tested standalone; both wired into
tools/run_interop.sh as the pyclient step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
register_default_handlers was a relocated app main(): one 1086-line function,
all-or-nothing. It is now 15 per-capability registration functions along the
lines GEM itself defines (S1F19): identification, equipment constants, clock,
event reports, remote commands, trace/limits, spooling, alarms, exceptions,
material tracking (E90/E116/E157), carriers (E87), recipes, object services
(E39), jobs (E40/E94), terminal services. A sensor-class tool registers three
functions instead of carrying carrier/job handlers it doesn't have;
register_default_handlers composes all 15. Each function derives exactly the
runtime aliases its handlers use (generated programmatically from the moved
bodies with boundary/substitution guards — zero hand-retyping).
Magic constants are gone: the control-state/clock SVIDs (were hardcoded 1/2)
and the CJ Executing/Completed CEIDs (were 400/401) now come from a "roles:"
block in equipment.yaml via EquipmentDescriptor, with historical defaults
when absent, loader parsing, and validation (CEID roles must name declared
events). The coupling is now visible in ONE file instead of silently split
between YAML and C++ — the exact drift class this repo's spec-as-data
philosophy exists to kill.
Tests: capability subsetting, role-driven SVID refresh via S1F3, roles
loader (shipped/custom/absent). Battery: core 473/3087 incl. the 53-handler
conformance sweep, daemon 125/125, live GEM300 demo (client exit 0), daemon
interop 20/20 vs secsgem-py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Item 8a — ConfigValidator warns on non-identifier variable/event/alarm/
command names ([A-Za-z_][A-Za-z0-9_]*): language bindings expose names as
kwargs/attributes, so 'Chamber Pressure' would be unusable in the planned
Python client. Warning not error — the wire doesn't care. Tested (4 warning
sites + good-name negative).
Item 4 tail — golden frames for S5F1 (Binary ALCD / U4 ALID / ASCII ALTX)
and a composed S6F11 (the production-critical report shape), bytes hand-
computed from E5 encoding rules: external pins on message composition.
Item 7 — equipment_service.hpp moved to include/secsgem/daemon/ (apps/
include-path hack removed) and a TSan daemon lane added locally + in CI.
tools/tsan.supp suppresses races whose accesses sit entirely inside the
UNinstrumented system libgrpc/libgpr/libabsl (epoll wakeups, absl Mutex
GraphCycles bookkeeping); our frames stay fully checked. The lane earned its
keep on first run: it caught a REAL threading-contract violation — a daemon
test reading model stores from the test thread while the io thread serviced
posted writes — fixed to use read_sync, exactly per the documented contract.
Now TSan-clean under halt_on_error=1 in the full production threading shape.
Suites: core 470/3068, daemon Release+TSan 125/125 each.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The HCACK-4 contract, implemented end to end. For every YAML-declared
command the service registers a forwarding handler (new HostCommandRegistry
names()/spec() accessors): with a subscribed tool client the command is
queued onto the Subscribe stream (id + name + params via from_item) and the
host is answered S2F42 HCACK=4 immediately — never blocking the io thread or
the T3 window; with NO subscriber the command takes its declarative YAML ack
(the honest pre-daemon behaviour). Settled + documented in the proto: v1 is
a firehose with no buffering/replay. CompleteCommand correlates the pending
id (audit; unknown id => PARAMETER_INVALID). Side effects stay suppressed on
HCACK-4 (router applies them only on Accept), so the completion event the
TOOL fires is the host's real signal — exactly E30's intent.
Tests (daemon suite 101 -> 124 assertions): a real S2F41 dispatched through
the full default-handler router ON the io thread under run_async — HCACK 4
with subscriber + params on the stream, declarative Accept without,
CompleteCommand known/unknown, fallback restored after unsubscribe.
Interop (now 20 checks, all green): the complete conformant loop against
the secsgem-py reference host — S2F41 START -> S2F42 HCACK=4 -> tool
receives Command(name=START, id=1) -> CompleteCommand -> FireEvent -> host
receives S6F11.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A2 — alarms: optional 'name:' on alarm config (a LOCAL key — SEMI only
defines numeric ALID + freetext ALTX; field appended last so existing
{id, text, category} brace-inits compile unchanged), parsed by the loader,
checked by the validator, shipped in equipment.yaml. SetAlarm/ClearAlarm
RPCs resolve config name OR stringified ALID via a constructor snapshot.
A3 — control state + health: RequestControlState fires operator events on
the io thread (read_sync) and reports what the E30 table actually did —
ACCEPT iff the equipment landed in the requested state, CANNOT_DO_NOW naming
the actual state otherwise (the shipped table has no operator path to
EquipmentOffline; the test pins that honesty). ATTEMPT_ONLINE is rejected as
transient. WatchHealth streams an immediate snapshot then pushes on link/
control-state changes via service observers (add_link_observer +
add_control_state_observer — the HandlerSlot work paying off), spool depth
sampled at the 500ms poll; ends on cancel or engine stop.
Tests: daemon suite 61 -> 101 assertions (alarm lifecycle by name/id/unknown,
WatchHealth initial + change push, all four RequestControlState semantics);
loader test for the alarm name (present + absent fallback); core 467/3055.
Interop now 15 checks incl. gRPC SetAlarm -> host receives S5F1 ALCD=0x84
ALID=1, and RequestControlState(HOST_OFFLINE) -> GetControlState confirms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EquipmentRuntime::read_sync establishes THE pattern for reading mutable
engine state from gRPC/binding threads (Phase 0 item 6): post the read onto
the io thread (the model's single owner), wait on a future with a deadline,
nullopt => UNAVAILABLE at the RPC edge. Always truthful, no cache to
invalidate; milliseconds are irrelevant at SECS rates.
GetVariables: name resolution against the service snapshot (empty query =
all; unknown name => INVALID_ARGUMENT naming the offender), values read via
read_sync, converted by the new from_item reverse conversion (single-element
numeric arrays => scalars, multi-element => List; Boolean/Binary/text per
format; C2-as-integer and U8>2^63 wrap documented as TODOs).
Tests run the engine in run_async — the daemon's PRODUCTION threading mode,
previously untested — and round-trip through both conversions: SetVariables
(declared-format write) then GetVariables (read) over a real in-process
channel. Daemon suite 41 -> 61 assertions. daemon_interop.py gains a live
GetVariables round-trip check vs the running daemon (verified green).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One ordered in-process scenario drives 53 of the 56 registered handlers
through Router::dispatch — S1 identification/comms/control, S2 ECs/clock/
event-config/commands/trace/limits/spool, S5 alarms+exceptions, S6 reports,
S7 recipes, S10 terminal, S14/S16 E39+E40/E94 jobs, S3 carriers — asserting
every reply is the paired (stream, function+1) with a body, plus targeted
state checks (OnlineRemote after S1F17, PJ exists after S16F11, HostOffline
after S1F15) and the Router's SxF0 abort fallback for unregistered W=1
primaries. Same flow secs_conformance runs over a live socket, but cheap
enough for every build; closes the '56 handlers, 4 direct tests' gap from
the design review.
Also seeds message-level golden frames: S1F13's body pinned to bytes
hand-computed from the E5 encoding rules — an external check on message
composition, not our codec validating itself (TODO: S5F1, composed S6F11).
Suite: 466 cases / 3052 assertions (+236), all green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tools/run_interop.sh runs ALL nine validation steps with a PASS/FAIL summary:
build, unit (464), daemon-unit (41), secsgem-py host vs server (31 checks),
secs_conformance (47), gRPC+secsgem-py daemon bridge, spool persistence
across restart, tshark HSMS dissector, secs4java8 (55 checks). Verified green
end-to-end. The unit suite is partly self-referential (our parsers validate
our builders); these external validators are the real oracle — now they run
with one command instead of by hand. Two bugs found by running it: unbounded
ninja at -O3 OOM-kills cc1plus in memory-constrained Docker VMs (build with
-j 2) and bash-3.2 lacks negative array subscripts.
CI: grpc deps added to the build job so secs_gemd + secs_gemd_tests build and
RUN in CI (previously the daemon silently dropped out — now fails loudly if
missing), plus a python-interop lane running py-host/conformance/daemon
harnesses against localhost in one container (no docker-in-docker).
Service hardening while in there: reject proto Values with no kind set at
the RPC edge (previously silently became ASCII ""), TODO markers for list
element formats and daemon graceful shutdown. New tests: unset-Value guard
+ a property test iterating ALL configured variables via gRPC asserting each
keeps its declared SECS-II format (daemon tests 16 -> 41 assertions).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 0 captures the 2026-06-10 review: multi-observer callbacks (done for
the critical three), CI for the interop/conformance harnesses (the unit
suite is partly self-referential; the external validators are the real
oracle), table-driven handler conformance + message-level golden frames,
register_default_handlers decomposition per GEM capability + YAML role
bindings for today's magic constants, the post+future mutable-read pattern,
service relocation + TSan run_async daemon test, identifier-safe name
validation. CompleteCommand's proto comment described the rejected blocking
model; it now states the settled HCACK-4 contract.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Status table brought current (format-aware daemon, secsgem-py interop), the
stale Layer-0 section replaced, and the path to an excellent GEM300 repo laid
out as ordered phases A–F: finish universal RPCs, the Subscribe command
stream (HCACK-4 design written down as the implementation contract), the
Python client package, GEM300 job/carrier in-the-loop, hardening/CI, and the
fab-acceptance track. Known-issues section records what the audit found
(GetControlState enum race + why the state-change-handler slot can't be
reused, missing alarm name key, pvd_tool predating set_handler, manual
interop harnesses, TSan gap).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extract the SECS/GEM engine wiring out of the secs_server app into a
reusable class, and stand up a language-agnostic gRPC daemon on top so a
tool's software (any language) can drive the equipment without linking C++
or knowing SEMI. Foundation for replacing a vendor's SECS/GEM server.
Engine reuse:
- EquipmentRuntime (include/secsgem/gem/runtime.hpp, src/gem/runtime.cpp):
owns io_context, passive Server, model, control-state machine, Router;
thread-safe outbound API (set_variable/emit_event/set_alarm/clear_alarm),
on_command hook, deliver_or_spool, run()/run_async()/poll()/stop().
- register_default_handlers (src/gem/default_handlers.cpp): the 56 GEM
handlers + domain emitters, relocated from secs_server so the app and the
daemon speak byte-identical GEM. secs_server.cpp reduced ~1270 -> 113 lines.
- name_index.hpp: resolve_variable(name) -> VID (the name->id binding layer).
Daemon (apps/secs_gemd.cpp, proto/secsgem/v1/equipment.proto):
- runs the engine + HSMS link on a background thread; serves the gRPC
Equipment service. Increment 1: SetVariables (name-resolved, plain
value->Item) and GetControlState. proto carries the full v1 surface
(universal + carrier/recipe/job tiers); remaining RPCs + the Subscribe
command stream are next (docs/DAEMON_ROADMAP.md).
- CMake: opt-in SECSGEM_DAEMON, protoc/grpc_cpp_plugin codegen, gracefully
skipped where protobuf/grpc++ are absent. Dockerfile gains the grpc deps.
Tests (proof): test_runtime, test_default_handlers (S1F1->S1F2, S2F41->hook),
test_name_index. Full suite 458/458, 2795 assertions; live server<->client
GEM300 demo still passes on the refactored server.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
raphael