2218b854ce490609dd89178043a0e017590ca8dc
157 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 raphael
|
2218b854ce |
fix(daemon)+test: accurate duplicate-ARRIVED message; broaden E90/E157 + names coverage
The duplicate-ARRIVED fix from the previous commit returned INVALID_OBJECT with the message "no substrate 'X'" — a lie, since the substrate exists. Rewrite ReportSubstrate so ARRIVED has its own ack mapping: a duplicate is CANNOT_DO_NOW with "substrate 'X' already exists" (a state conflict, not a missing object), and we never silently re-create over live FSM state. Coverage gaps closed: - C++: ARRIVED records carrier_id/slot (now asserted); module NOT_EXECUTING reset transition; duplicate-ARRIVED expects CANNOT_DO_NOW. - Interop: @eq.command now drives the real host S2F41 path (was @eq.on, so the headline decorator had zero wire coverage); @eq.command NameError on unknown name; eq.names var/alarm + dir() + typo-suggestion; replaced the two `check(..., True)` tautologies with full E90 journey + AT_DESTINATION and real error paths (ghost wafer raises, illegal module jump raises). All 8 daemon test cases (248 assertions) and 24 pyclient interop checks pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
raphael
|
d22bbc4ab2 |
fix(daemon)+fix(client): close four fool-proofing gaps
C++ (equipment_service.hpp): - ReportSubstrate ARRIVED: check CreateResult and return INVALID_OBJECT when the substrate ID already exists, instead of silently doing nothing - ReportSubstrate/ReportModule default switch branches: return false (→ CANNOT_DO_NOW) for unknown enum values instead of silently accepting Python (_client.py): - @eq.command: raise NameError (client-side name validation) instead of SecsGemError (which means "daemon declined a request") — wrong type - Module docstring: update example to show @eq.command / eq.names API Test (test_daemon_service.cpp): - Add duplicate-ARRIVED check (expects INVALID_OBJECT) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
raphael
|
a2ebbf7c65 |
feat(client)+feat(daemon): eq.names, @eq.command, E90/E157 RPCs
Python client:
- eq.names.event.* / .alarm.* / .command.* / .var.* / .constant.* —
autocomplete-able, typo-safe name lookup backed by the Describe RPC
(lazy, cached; AttributeError on bad name with close-match hints)
- @eq.command decorator — binds a handler by function name, validated
against the equipment's real command set at decoration time
- eq.report_substrate() — E90 wafer milestone reporting
- eq.report_module() — E157 module state reporting (auto-create)
Daemon (C++ service):
- ReportSubstrate RPC — drives E90 location + processing FSMs
- ReportModule RPC — drives E157 module FSM (auto-create on first report)
- ack_from_outcome() helper — consistent Ack mapping for read_sync results
Proto: SubstrateReport, ModuleReport, EquipmentDescription,
SpoolFlushRequest, TerminalMessage; Describe, FlushSpool,
SendTerminalMessage RPCs
Tests: C++ FSM test (journey + ghost rejection + E157 illegal jump);
interop coverage for names API and E90/E157 round-trip
Docs: ch42 RPC table + Python example updated; ch16 daemon-path section added
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
raphael
|
9876dd9b5a |
feat(daemon): D10 carriers + E16 ops RPCs + stress test + virtual fab
tests / build-and-test (push) Successful in 2m59s
tests / thread-sanitizer (push) Successful in 3m36s
tests / tshark-dissector (push) Successful in 2m25s
tests / secs4j-interop (push) Successful in 59s
tests / python-interop (push) Successful in 3m20s
tests / libfuzzer (push) Successful in 3m40s
Completes the daemon's GEM300 surface and adds two new test tiers. D10 — E87 carriers: CarrierStore gains the HandlerSlot observer pattern (add_id/slot_map/access_handler). The daemon's id-observer forwards host S3F17 decisions onto the Subscribe stream as CarrierAction (PROCEED on a Confirmed transition, CANCEL on CancelCarrier); ReportCarrier drives the flow tool-side: WAITING creates the carrier + records the slot map, IN_ACCESS/COMPLETE advance the access FSM (INVALID_OBJECT on unknown, CANNOT_DO_NOW on an illegal transition). E16 — operations RPCs: Describe (full name inventory: variables/events/ alarms/commands/constants + device header), FlushSpool (purge or drain), SendTerminalMessage (S10F1 tool->host, honest CANNOT_DO_NOW when no host and stream 10 isn't spoolable). Stream responsiveness: Subscribe/WatchHealth poll at 100ms (was 500ms) so a cancelled stream frees its sync-server worker thread promptly — this was found by the new stress test, which hung under Subscribe churn at 500ms. Tests: - A randomized concurrent RPC stress case: 4 threads x 250 seeded ops (set/get/fire/alarm/control-state/describe + Subscribe churn), asserts no failed RPC and a still-responsive engine afterward; prints its seed; a strong TSan target. - A virtual fab (interop/virtual_fab.py + the `fab` compose service / tools/spawn_fab.sh): N daemons, each with a secsgem-py host AND a secsgem_client tool, driven by seeded random traffic with end-to-end invariant checks (set/get round-trips, event->S6F11 and alarm->S5F1 delivery, command->tool->completion). Verified green at N=3 (~150 ops/eq, all commands round-tripped, 0 violations). Wired into run_interop.sh (now 13 steps). Also fixes the CI break from the previous commit: the Python-client lane's test_values.py step lacked PYTHONPATH=clients/python (now step-level env). Two bugs found and fixed while building this, both mine from this batch: 1. carrier test hung on a CancelCarrier of a still-NotConfirmed carrier — a self-transition the FSM doesn't signal, so the observer never fired and the stream Read blocked forever. Fixed to cancel a Confirmed carrier; the NotConfirmed edge is documented as a known E87 limitation. 2. the 500ms stream poll above. Daemon suite 7 cases / 214 assertions; core 475 / 3097; virtual fab green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
54626ceb6a |
feat(daemon): Phase E — production hardening, complete
tests / build-and-test (push) Successful in 2m59s
tests / thread-sanitizer (push) Successful in 3m28s
tests / tshark-dissector (push) Successful in 2m22s
tests / secs4j-interop (push) Successful in 2m6s
tests / python-interop (push) Failing after 3m8s
tests / libfuzzer (push) Successful in 3m44s
Exposure: --grpc default flipped from 0.0.0.0 to 127.0.0.1 (the API is unauthenticated by design; auth belongs to the transport), Unix-domain- socket support (--grpc unix:///run/secs_gemd/api.sock = zero network surface), SECURITY.md documents the contract and ch42 gained a "Running it in production" section (which also documents the HSMS-SS single-session assumption). Graceful shutdown: SIGTERM/SIGINT land on an asio::signal_set on the io thread, which nudges grpc Shutdown with a 2s deadline (cancels open Subscribe/WatchHealth streams); Wait() returns on the MAIN thread, which stops the engine (rt->stop() joins the io thread, so it must not run on it). Exit 0, journal-safe, the in-code TODO is gone. --spool-dir added so host-bound events survive daemon restarts. Observability: --metrics serves Prometheus gauges secsgem_link_selected / secsgem_control_state / secsgem_spool_depth, wired via the Phase-0 add_link_observer/add_control_state_observer hooks + io-thread sampling. Deployment: deploy/secs_gemd.service — hardened systemd unit (DynamicUser, ProtectSystem=strict, StateDirectory for the spool, UDS for the API, TimeoutStopSec aligned with the graceful-shutdown window). Enforcement: tools/check_daemon_ops.sh proves all three operational claims (unix-socket gRPC accepts, all gauges present on /metrics, SIGTERM -> exit 0 + clean-stop log) — green; wired into tools/run_interop.sh (now 11 steps) and CI. CI python-interop lane also gained the pyclient and spool-restart steps, so every harness now runs in CI. TODO sweep: the shutdown TODO is fixed; the four remaining TODOs (nested list formats, C2-as-text, U8>2^63, CONNECTED link state) are deliberate deferred edge cases, each marked in code with context. Daemon suite re-verified green (175 assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
b1772cfefd |
feat(daemon): Phase D — GEM300 in-the-loop (jobs, recipes, EC changes)
Semantics settled and documented: v1 is observe-and-report. The engine keeps acking S16/S3/S7/S2F15 from its FSM tables — exactly the behaviour both reference implementations validated — while the tool observes lifecycle events on the Subscribe stream and reports physical progress back. Gating stays the documented v2 deferred-reply item. Engine: two new store observers (HandlerSlot pattern) — RecipeStore fires (ppid, body) after an add (S7F3 downloads), EquipmentConstantStore fires (id, value) on ACCEPTED S2F15 writes only. Unit-tested. Daemon: the service registers PJ/recipe/EC observers (io thread; add_ observers coexist with register_default_handlers' primaries) and fans the new HostRequest variants out via push_request (fire-and-forget, no- buffering contract). ProcessJob carries action (Start->START, Resume-> RESUME, Paused->PAUSE, Stopping->STOP, Aborting->ABORT) + recipe + material bindings read store-side on the io thread. ReportProcessJob maps SETTING_UP ->SetupComplete, COMPLETE->ProcessComplete, ABORTED->AbortComplete via read_sync; PROCESSING is informational; unknown job => INVALID_OBJECT, table-rejected transition => CANNOT_DO_NOW. Carriers deferred (CarrierStore has no observer machinery; ReportCarrier stays UNIMPLEMENTED) — roadmap. Python client: on_process_job / on_recipe / on_constant_change decorators + report_job(job_id, state); ProcessJob dataclass exported. Tests: daemon suite 141 -> 175 assertions — the full in-process loop (S16F11 create -> tool setup -> S16F5 PJSTART -> stream ProcessJob with recipe+carriers -> ReportProcessJob(COMPLETE) -> FSM at ProcessComplete), rejection paths, S7F3 -> ProcessProgram, S2F15 -> ConstantChange with the configured name. Core 475/3097 (observer units). Live regression: daemon interop 20 checks + pyclient 13 checks still green against the running daemon. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
b30443089f |
feat(clients)+test(interop): C++ client + Java validation of the daemon (B7)
tests / build-and-test (push) Successful in 2m54s
tests / thread-sanitizer (push) Successful in 3m48s
tests / tshark-dissector (push) Successful in 2m24s
tests / secs4j-interop (push) Successful in 1m44s
tests / python-interop (push) Successful in 3m10s
tests / libfuzzer (push) Successful in 3m38s
B7 — the daemon's HSMS face under the Java reference: Dockerfile.server now
bakes secs_gemd alongside secs_server (grpc deps in both stages), and
secs4j_validate.sh gains TARGET=gemd to point the 55-check secs4java8 suite
at the daemon instead. Result: 55/55 green. With secsgem-py already
validating both faces, byte-identical GEM between secs_server and secs_gemd
is now proven by both reference implementations, not inferred from shared
code. CI runs the daemon target as an extra step (image layers shared).
Second client — clients/cpp: a header-only C++ twin of the Python client
over the same proto. eq.set("ChamberPressure", 2.5) with bare literals
(integral/floating dispatch avoids variant ambiguity), get/fire/alarm/
clear, control_state/request_control_state/health, on("START", fn) +
listen()/listen_async()/stop() with auto-CompleteCommand, SecsGemError
carrying the daemon's message. cpp_mini_tool (~30 lines) mirrors the
Python mini_tool. Tested end-to-end over real loopback TCP against the
service inside secs_gemd_tests — now 4 cases / 141 assertions — including
set/get round-trips, error text, alarm-by-name into the model, health,
and the full HCACK-4 command loop with parameters.
(Build note: two grpc-heavy TUs at -O3 OOM even at -j2 on Docker Desktop;
built -j1. Known environment limitation, roadmap-documented.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
raphael
|
4f3031aeb9 |
feat(example)+docs: pvd_tool on the modern stack; chapter 42 teaches the daemon path
C9 — the flagship vendor example now demonstrates the intended integration
shape. examples/pvd_tool/main.cpp: 1093 -> 570 lines. The 466-line
hand-registered handler section and the hand-wired Server/Router/emit
plumbing are gone, replaced by EquipmentRuntime + register_default_handlers
(the example now serves all 56 handlers, up from its hand-picked 51) +
commands.set_handler for the START-runs-the-recipe behaviour (was a
hard-coded S2F41 router override). All domain logic — sensor simulator,
recipe runner, alarm threshold monitor, EPT cycler, Prometheus gauges —
unchanged. pvd's SVIDs 1/2 and CEIDs 400/401 match the roles: defaults, so
the built-ins bind with no config change. Verified: builds clean, boots
("registered 56 handlers", config loaded, EPT cycling), HSMS :5000 accepts,
metrics :9090 answers HTTP 200. logfn flushes per line so docker/CI logs
are visible immediately.
Writing project — new tutorial chapter docs/42_vendor_daemon_and_clients.md:
why a daemon (the host-timer argument), the proto contract and the HCACK-4
command semantics, the Python client walkthrough, EquipmentRuntime +
capability registration + roles:, the threading contract (posting API /
read_sync / hooks-on-io-thread) and primary-vs-observer slots, and a
which-tier-do-I-pick table. Indexed in 00_index Part 4. Refreshed the three
spots that still described pvd_tool's old "51 handlers in ~460 lines" shape
(ch35, ch41, pvd README) — drift killed in the same commit that made it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
raphael
|
af1a159c59 |
docs: bring the documentation up to the daemon/client era
A large gap had opened between the docs and the code: the README and INTEGRATION guide did not mention the gRPC daemon or the Python client at all (the entire vendor surface), ARCHITECTURE still described secs_server as the ~1200-line canonical wiring example (it is a ~110-line thin main over EquipmentRuntime), and test counts across six files were stale (445/2753 -> 473/3087 core + the separate 125-assertion daemon suite). - README: new "Integrating your tool (pick a tier)" section — Python client / any-language gRPC / embedded C++ — plus daemon tests and tools/run_interop.sh in the Testing section. - ARCHITECTURE: layer diagram gains the vendor-surface and EquipmentRuntime/default_handlers tiers; stale wiring row fixed. - INTEGRATION: three-tier chooser up front (this guide = the C++ tier). - ch30 tour: secs_gemd + secs_gemd_tests in the binaries table. - ch31: example alarm used a nonexistent `alcd:` field with bit 7 set (which the validator forbids) -> real `category:`/`name:` fields, and the roles: block documented. - ch35: handler-location note now points at default_handlers.cpp's 15 per-capability register_* functions. - ch40: built-artifacts list + sample output counts. - ch50: secsgem::gem runtime/default_handlers/handler_slot/name_index includes + new secsgem::daemon namespace section. - PROOFS: test-count table gains the runtime/handlers/daemon row so the tally adds up; daemon suite noted. VERIFICATION/COMPLIANCE counts. - interop/README: the one-command runner + the two daemon-track harnesses (daemon_interop, pyclient_interop). Audited via a docs-vs-code sweep (the audit itself under-reported: it validated counts textually; reality was 473/3087). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
8686654b15 |
feat(client): the Python client — a GEM tool in plain Python (Phase C)
clients/python: pip-installable "secsgem-client", pure Python (stubs
pre-generated from equipment.proto, import made package-relative; no
compiled extension, no SEMI knowledge, no C++ toolchain). The API the whole
effort aimed at:
eq = Equipment("localhost:50051")
eq.set(ChamberPressure=2.5); eq["WaferCounter"] = 7
eq.fire("ProcessStarted", ChamberPressure=2.75)
eq.alarm("chiller_temp_high"); eq.clear("chiller_temp_high")
@eq.on("START")
def start(cmd): ... # auto-CompleteCommand after return
eq.listen(background=True)
eq.control_state; eq.request_control_state("HOST_OFFLINE"); eq.health()
Errors raise SecsGemError carrying the daemon's message ("no variable named
..."). bool checked before int in conversion (isinstance(True, int)).
examples/mini_tool.py is a complete GEM tool in ~25 lines.
PROOF — interop/pyclient_interop.py drives the PUBLISHED package (not raw
stubs) against a live secs_gemd with secsgem-py as the fab host: 13 checks
all green on first run — set/get round-trips, item syntax, SecsGemError on
unknown names, control state, health, fire->S6F11 on the host's wire,
alarm/clear->S5F1 with correct set bit, the full command loop (host S2F41 ->
HCACK=4 -> @eq.on handler -> completion event back at the host), operator
offline. Conversion layer unit-tested standalone; both wired into
tools/run_interop.sh as the pyclient step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
raphael
|
912304966f |
refactor(gem): decompose default handlers per GEM capability + YAML role bindings
register_default_handlers was a relocated app main(): one 1086-line function, all-or-nothing. It is now 15 per-capability registration functions along the lines GEM itself defines (S1F19): identification, equipment constants, clock, event reports, remote commands, trace/limits, spooling, alarms, exceptions, material tracking (E90/E116/E157), carriers (E87), recipes, object services (E39), jobs (E40/E94), terminal services. A sensor-class tool registers three functions instead of carrying carrier/job handlers it doesn't have; register_default_handlers composes all 15. Each function derives exactly the runtime aliases its handlers use (generated programmatically from the moved bodies with boundary/substitution guards — zero hand-retyping). Magic constants are gone: the control-state/clock SVIDs (were hardcoded 1/2) and the CJ Executing/Completed CEIDs (were 400/401) now come from a "roles:" block in equipment.yaml via EquipmentDescriptor, with historical defaults when absent, loader parsing, and validation (CEID roles must name declared events). The coupling is now visible in ONE file instead of silently split between YAML and C++ — the exact drift class this repo's spec-as-data philosophy exists to kill. Tests: capability subsetting, role-driven SVID refresh via S1F3, roles loader (shipped/custom/absent). Battery: core 473/3087 incl. the 53-handler conformance sweep, daemon 125/125, live GEM300 demo (client exit 0), daemon interop 20/20 vs secsgem-py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
cf230d4119 |
chore(phase0): name validation, golden frames, daemon into library tree, TSan daemon lane
Item 8a — ConfigValidator warns on non-identifier variable/event/alarm/ command names ([A-Za-z_][A-Za-z0-9_]*): language bindings expose names as kwargs/attributes, so 'Chamber Pressure' would be unusable in the planned Python client. Warning not error — the wire doesn't care. Tested (4 warning sites + good-name negative). Item 4 tail — golden frames for S5F1 (Binary ALCD / U4 ALID / ASCII ALTX) and a composed S6F11 (the production-critical report shape), bytes hand- computed from E5 encoding rules: external pins on message composition. Item 7 — equipment_service.hpp moved to include/secsgem/daemon/ (apps/ include-path hack removed) and a TSan daemon lane added locally + in CI. tools/tsan.supp suppresses races whose accesses sit entirely inside the UNinstrumented system libgrpc/libgpr/libabsl (epoll wakeups, absl Mutex GraphCycles bookkeeping); our frames stay fully checked. The lane earned its keep on first run: it caught a REAL threading-contract violation — a daemon test reading model stores from the test thread while the io thread serviced posted writes — fixed to use read_sync, exactly per the documented contract. Now TSan-clean under halt_on_error=1 in the full production threading shape. Suites: core 470/3068, daemon Release+TSan 125/125 each. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
e6ee927900 |
feat(daemon): Subscribe command stream + CompleteCommand — the vendor loop closes
The HCACK-4 contract, implemented end to end. For every YAML-declared command the service registers a forwarding handler (new HostCommandRegistry names()/spec() accessors): with a subscribed tool client the command is queued onto the Subscribe stream (id + name + params via from_item) and the host is answered S2F42 HCACK=4 immediately — never blocking the io thread or the T3 window; with NO subscriber the command takes its declarative YAML ack (the honest pre-daemon behaviour). Settled + documented in the proto: v1 is a firehose with no buffering/replay. CompleteCommand correlates the pending id (audit; unknown id => PARAMETER_INVALID). Side effects stay suppressed on HCACK-4 (router applies them only on Accept), so the completion event the TOOL fires is the host's real signal — exactly E30's intent. Tests (daemon suite 101 -> 124 assertions): a real S2F41 dispatched through the full default-handler router ON the io thread under run_async — HCACK 4 with subscriber + params on the stream, declarative Accept without, CompleteCommand known/unknown, fallback restored after unsubscribe. Interop (now 20 checks, all green): the complete conformant loop against the secsgem-py reference host — S2F41 START -> S2F42 HCACK=4 -> tool receives Command(name=START, id=1) -> CompleteCommand -> FireEvent -> host receives S6F11. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
1da56f973f |
feat(daemon): alarms by name + RequestControlState + WatchHealth (Phase A complete)
A2 — alarms: optional 'name:' on alarm config (a LOCAL key — SEMI only
defines numeric ALID + freetext ALTX; field appended last so existing
{id, text, category} brace-inits compile unchanged), parsed by the loader,
checked by the validator, shipped in equipment.yaml. SetAlarm/ClearAlarm
RPCs resolve config name OR stringified ALID via a constructor snapshot.
A3 — control state + health: RequestControlState fires operator events on
the io thread (read_sync) and reports what the E30 table actually did —
ACCEPT iff the equipment landed in the requested state, CANNOT_DO_NOW naming
the actual state otherwise (the shipped table has no operator path to
EquipmentOffline; the test pins that honesty). ATTEMPT_ONLINE is rejected as
transient. WatchHealth streams an immediate snapshot then pushes on link/
control-state changes via service observers (add_link_observer +
add_control_state_observer — the HandlerSlot work paying off), spool depth
sampled at the 500ms poll; ends on cancel or engine stop.
Tests: daemon suite 61 -> 101 assertions (alarm lifecycle by name/id/unknown,
WatchHealth initial + change push, all four RequestControlState semantics);
loader test for the alarm name (present + absent fallback); core 467/3055.
Interop now 15 checks incl. gRPC SetAlarm -> host receives S5F1 ALCD=0x84
ALID=1, and RequestControlState(HOST_OFFLINE) -> GetControlState confirms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
raphael
|
83593bb508 |
docs: refresh stale roadmap status rows (GetVariables shipped, harnesses automated)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
1daf120431 |
feat(daemon): GetVariables + read_sync — the standard mutable-read pattern
EquipmentRuntime::read_sync establishes THE pattern for reading mutable engine state from gRPC/binding threads (Phase 0 item 6): post the read onto the io thread (the model's single owner), wait on a future with a deadline, nullopt => UNAVAILABLE at the RPC edge. Always truthful, no cache to invalidate; milliseconds are irrelevant at SECS rates. GetVariables: name resolution against the service snapshot (empty query = all; unknown name => INVALID_ARGUMENT naming the offender), values read via read_sync, converted by the new from_item reverse conversion (single-element numeric arrays => scalars, multi-element => List; Boolean/Binary/text per format; C2-as-integer and U8>2^63 wrap documented as TODOs). Tests run the engine in run_async — the daemon's PRODUCTION threading mode, previously untested — and round-trip through both conversions: SetVariables (declared-format write) then GetVariables (read) over a real in-process channel. Daemon suite 41 -> 61 assertions. daemon_interop.py gains a live GetVariables round-trip check vs the running daemon (verified green). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
b0a4c331cf |
test(gem): table-driven conformance sweep over the default handler set
One ordered in-process scenario drives 53 of the 56 registered handlers through Router::dispatch — S1 identification/comms/control, S2 ECs/clock/ event-config/commands/trace/limits/spool, S5 alarms+exceptions, S6 reports, S7 recipes, S10 terminal, S14/S16 E39+E40/E94 jobs, S3 carriers — asserting every reply is the paired (stream, function+1) with a body, plus targeted state checks (OnlineRemote after S1F17, PJ exists after S16F11, HostOffline after S1F15) and the Router's SxF0 abort fallback for unregistered W=1 primaries. Same flow secs_conformance runs over a live socket, but cheap enough for every build; closes the '56 handlers, 4 direct tests' gap from the design review. Also seeds message-level golden frames: S1F13's body pinned to bytes hand-computed from the E5 encoding rules — an external check on message composition, not our codec validating itself (TODO: S5F1, composed S6F11). Suite: 466 cases / 3052 assertions (+236), all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
42044e92e2 |
ci(interop): one-command external-validation suite + CI lanes for the daemon
tests / build-and-test (push) Successful in 2m42s
tests / thread-sanitizer (push) Successful in 2m50s
tests / tshark-dissector (push) Successful in 2m24s
tests / secs4j-interop (push) Successful in 37s
tests / python-interop (push) Successful in 2m56s
tests / libfuzzer (push) Successful in 3m44s
tools/run_interop.sh runs ALL nine validation steps with a PASS/FAIL summary: build, unit (464), daemon-unit (41), secsgem-py host vs server (31 checks), secs_conformance (47), gRPC+secsgem-py daemon bridge, spool persistence across restart, tshark HSMS dissector, secs4java8 (55 checks). Verified green end-to-end. The unit suite is partly self-referential (our parsers validate our builders); these external validators are the real oracle — now they run with one command instead of by hand. Two bugs found by running it: unbounded ninja at -O3 OOM-kills cc1plus in memory-constrained Docker VMs (build with -j 2) and bash-3.2 lacks negative array subscripts. CI: grpc deps added to the build job so secs_gemd + secs_gemd_tests build and RUN in CI (previously the daemon silently dropped out — now fails loudly if missing), plus a python-interop lane running py-host/conformance/daemon harnesses against localhost in one container (no docker-in-docker). Service hardening while in there: reject proto Values with no kind set at the RPC edge (previously silently became ASCII ""), TODO markers for list element formats and daemon graceful shutdown. New tests: unset-Value guard + a property test iterating ALL configured variables via gRPC asserting each keeps its declared SECS-II format (daemon tests 16 -> 41 assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
941f9ef458 |
docs: add Phase 0 (structural debts from design review); fix CompleteCommand contract comment
Phase 0 captures the 2026-06-10 review: multi-observer callbacks (done for the critical three), CI for the interop/conformance harnesses (the unit suite is partly self-referential; the external validators are the real oracle), table-driven handler conformance + message-level golden frames, register_default_handlers decomposition per GEM capability + YAML role bindings for today's magic constants, the post+future mutable-read pattern, service relocation + TSan run_async daemon test, identifier-safe name validation. CompleteCommand's proto comment described the rejected blocking model; it now states the settled HCACK-4 contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
8a48ffeed4 |
feat(gem): multi-observer state-change handlers via HandlerSlot
The single-slot set_*_handler pattern was a structural blocker, hit twice: the daemon could not observe control-state changes because register_default_handlers owns the slot, forcing GetControlState to read the FSM cross-thread (a data race), and blocking WatchHealth and the Subscribe stream's ControlStateChange variant. HandlerSlot<Args...> keeps a primary slot with exact legacy semantics (set_ replaces — one existing test depends on replacement) plus an append-only observer list (add_) that survives set_ calls. Fire sites are textually unchanged (operator bool / operator() / assign-from-function). Applied to ControlStateMachine + ProcessJobStore + ControlJobStore (the roadmap-critical three; the remaining single-slot classes follow the same 3-line pattern as needed). EquipmentRuntime gains an atomic control-state mirror registered as an observer — control_state() is now safe from any thread, retiring the GetControlState race — plus add_control_state_observer and add_link_observer (selected/closed fan-out), the hooks WatchHealth and Subscribe need. Tests: observer ordering, set-replaces-primary-but-observers-survive, observers-without-primary, PJ-store coexistence, and the runtime scenario that was previously impossible (mirror + observer + default-handlers set_). Core 464/464 (2816 assertions), daemon 16/16, live GEM300 demo passes with single-fire control-state transitions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
b067a76b80 |
docs: rewrite daemon roadmap as ordered plan with known-issues audit
Status table brought current (format-aware daemon, secsgem-py interop), the stale Layer-0 section replaced, and the path to an excellent GEM300 repo laid out as ordered phases A–F: finish universal RPCs, the Subscribe command stream (HCACK-4 design written down as the implementation contract), the Python client package, GEM300 job/carrier in-the-loop, hardening/CI, and the fab-acceptance track. Known-issues section records what the audit found (GetControlState enum race + why the state-change-handler slot can't be reused, missing alarm name key, pvd_tool predating set_handler, manual interop harnesses, TSan gap). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
99bfa794fc |
fix(daemon): honour declared SECS-II formats + make service thread-safe
Audit fixes for two real bugs in the gRPC service: 1. Format compliance: to_item() wrote F8/I8 regardless of the variable's declared wire format, so values contradicted the S1F11/S1F21 namelists (ChamberPressure is F4, WaferCounter U4; the interop trace showed <F8 2.5> on the wire). Conversion now targets the declared format — verified end-to-end: secsgem-py now receives <F4 2.5> in S6F11. 2. Thread safety: gRPC handler threads called resolve_variable/resolve_event, copying live store entries (including Item values) while the io thread mutates them. The service now snapshots the immutable name->id/format maps at construction (before run_async, per the documented ordering); all writes already post to the io thread. Remaining known narrow race (GetControlState enum read) documented in DAEMON_ROADMAP. Also: drop a stale tools/run_interop.sh reference from docker-compose.yml. Tests: daemon in-process 16/16 (new F4/U4 format assertions), core 459/459, secsgem-py interop green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
92afbd2a37 |
docs: record secsgem-py daemon interop in roadmap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
3d72e50b65 |
test(interop): daemon end-to-end vs secsgem-py reference host
daemon_interop.py drives a running secs_gemd through BOTH faces at once: a
gRPC tool client and a secsgem-py active host. Proves the gRPC<->HSMS bridge
against a reference GEM implementation, not just in-process:
- gRPC GetControlState agrees with the HSMS-driven control state
- gRPC SetVariables(ChamberPressure=2.5) + FireEvent(ProcessStarted) makes
the host receive S6F11 CEID 300 carrying 2.5 (value flowed gRPC -> engine
-> HSMS -> host)
- unknown variable/event names rejected at the gRPC edge
Mirrors the existing host_vs_cpp_server.py pattern. New 'gemd' compose service
(HSMS :5000 + gRPC :50051); interop image gains grpcio/grpcio-tools (proto
stubs generated at runtime, flat to avoid the secsgem package-name clash).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
raphael
|
dd288eb2ac |
docs: update daemon roadmap — gRPC toolchain done, secs_gemd serving
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
cb85199f49 |
feat(daemon): FireEvent + event name resolution + in-process gRPC tests
- name_index: add resolve_event(name) -> CEID (unit-tested). - equipment_service.hpp: extract the gRPC service + value/state conversion into a shared header; add FireEvent (optional per-fire variable values, then trigger the collection event by name). secs_gemd slims to main(). - test_daemon_service: real in-process gRPC integration test (client stub -> service -> EquipmentRuntime) proving SetVariables lands in the model, GetControlState reports the state, FireEvent and unknown-name paths behave. Separate secs_gemd_tests target (links grpc++/proto), gated on the daemon. Core suite 459/459 (2799 assertions); daemon gRPC tests 15/15. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
fc898f8410 |
feat: EquipmentRuntime engine owner + secs_gemd gRPC daemon
Extract the SECS/GEM engine wiring out of the secs_server app into a reusable class, and stand up a language-agnostic gRPC daemon on top so a tool's software (any language) can drive the equipment without linking C++ or knowing SEMI. Foundation for replacing a vendor's SECS/GEM server. Engine reuse: - EquipmentRuntime (include/secsgem/gem/runtime.hpp, src/gem/runtime.cpp): owns io_context, passive Server, model, control-state machine, Router; thread-safe outbound API (set_variable/emit_event/set_alarm/clear_alarm), on_command hook, deliver_or_spool, run()/run_async()/poll()/stop(). - register_default_handlers (src/gem/default_handlers.cpp): the 56 GEM handlers + domain emitters, relocated from secs_server so the app and the daemon speak byte-identical GEM. secs_server.cpp reduced ~1270 -> 113 lines. - name_index.hpp: resolve_variable(name) -> VID (the name->id binding layer). Daemon (apps/secs_gemd.cpp, proto/secsgem/v1/equipment.proto): - runs the engine + HSMS link on a background thread; serves the gRPC Equipment service. Increment 1: SetVariables (name-resolved, plain value->Item) and GetControlState. proto carries the full v1 surface (universal + carrier/recipe/job tiers); remaining RPCs + the Subscribe command stream are next (docs/DAEMON_ROADMAP.md). - CMake: opt-in SECSGEM_DAEMON, protoc/grpc_cpp_plugin codegen, gracefully skipped where protobuf/grpc++ are absent. Dockerfile gains the grpc deps. Tests (proof): test_runtime, test_default_handlers (S1F1->S1F2, S2F41->hook), test_name_index. Full suite 458/458, 2795 assertions; live server<->client GEM300 demo still passes on the refactored server. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
4b4b2ac690 |
docs: correct drifted and fabricated APIs in chapters 13/17/35/51
An audit of doc code blocks against the real headers found APIs that do not exist in the codebase, presented as authoritative walkthroughs: - ch35 (dispatch): an entirely fabricated callback architecture — HostCommandRegistry::set_emit_ceid_handler, CommandOutcome, emit_ceids. Rewritten to the real Spec/Result/dispatch + the new set_handler hook. - ch13 (E30): wrong store names — EventStore/ReportStore -> EventReportSubscriptions, SvidStore -> StatusVariableStore, AlarmStore/AlarmDispatcher -> AlarmRegistry, ClockStore -> Clock, TerminalServiceStore -> (no store), in both the capability tables and the worked S2F33 example. - ch17 (E116): EptStore/seconds/bucket_ -> EptStateMachine/milliseconds/buckets_. - ch51 (extending): stale host-command handler -> the real set_handler signature. Verified clean by grep: no fabricated symbols remain in docs/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
0090791968 |
feat(gem): add host-command behaviour hook to HostCommandRegistry
Host commands were declarative-only: dispatch() returned the YAML-defined HCACK plus side effects, and ignored the command parameters entirely (the param list was a commented-out argument). Equipment could acknowledge a command but never run anything in response — the pvd_tool example worked around this by hard-coding behaviour in a C++ router handler. Add set_handler(rcmd, fn): a registered handler receives the live CPNAME/ CPVAL parameters and returns the HCACK, overriding the declarative default. Live on S2F41/F21/F49 via the shared dispatch(). No handler => byte-for-byte the previous declarative behaviour. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
dae6bfd747 |
docs: streamline tone across reference docs
Tone pass across the non-tutorial markdown — README, PROOFS,
ARCHITECTURE, BENCHMARKS, COMPLIANCE, FAQ, MES_INTEROP, SECURITY,
and interop/README. Three patterns came out:
- Bug-history war stories ("Past interop sweeps surfaced…",
"What these harnesses caught: 1. Strict U-width parsing…").
- Chat-with-reader framing ("Don't skip TLS unless…", "Treat as a
punch list", "If you're running in a pod…", "Misconfiguration
incidents drop dramatically").
- Self-referential narration ("we ship", "our codec", "the
codebase's most-tested layer", "three orders of magnitude above
fab load", "the gift that keeps giving").
README also drops the standalone ThreadSanitizer subsection under
Build details (now a single line under the new Testing section).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
d63c92166d |
docs: rewrite VERIFICATION.md to describe shipped validators
Previously written as a forward-looking plan ("Plan: (1) KAT → (2)
tshark → (3) secs4j → (4) libFuzzer", "Effort: ~3 hours", "Survey
step (do this first)"). All four validators have shipped —
test_e5_kat.cpp, interop/secs4j/Secs4jHostHarness.java,
interop/tshark_validate.sh, apps/fuzz_*.cpp. Rewritten as
documentation of what's there: file paths, CI job names, actual
result numbers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
0355c73211 |
docs: refresh stale file paths after store/ reorg + gen_messages rename
generate_messages.py → gen_messages.py and several gem/ headers moved under gem/store/ (carrier_store.hpp → store/carriers.hpp, etc.); e84.hpp split into e84_state.hpp. The guided-tour chapters still pointed at the old paths — relink them so the deep-link footnotes resolve. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
fee82d88c9 |
ci: self-contained secs_server image for secs4j interop
The harness previously bound the source tree into a compose service and built inside it. That breaks under docker-in-docker (gitea-act, GitHub Actions runners with /var/run/docker.sock mounted) because bind-mount sources resolve against the *host* daemon's filesystem, not the runner container's. Now Dockerfile.server bakes a Release secs_server into its own image, and secs4j_validate.sh wires server and harness together on a dedicated bridge — no volumes needed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
31f908e1bf |
docs: chapters 40, 41, 50, 51 — Operations + Reference (series complete)
Last four chapters of the guided tour. 40 — Building, running, the demo. Docker prerequisites, the build flow, what each binary is for, running the 24-transaction demo flow annotated step by step. Running the 4 external-validator sweeps + the libFuzzer pass. Inspecting the demo with tcpdump and tshark. Reading source while running as the recommended learning workflow. 41 — Integration: hardware, MES, production. Four-phase tour: wiring sensors / recipe engine / alarms / E84 GPIO; talking to a real MES with the day-1 punch list + commercial-MES quirks (Wonderware S2F21, Camstar Linktest cadence, etc.); production hardening (nftables / stunnel / minisign / persistence layout / monitoring / runbook); performance envelope + memory footprint + capacity planning. Pointers to the long-form INTEGRATION.md / MES_INTEROP.md / SECURITY.md / BENCHMARKS.md. 50 — API + message catalog + YAML schemas reference. Namespace-by- namespace table of public symbols (secs2, hsms, secsi, gem, config, metrics) with brief descriptions. Stream-by-stream message catalog reference (S1, S2, S3, S5, S6, S7, S9, S10, S12, S14, S16). YAML schema reference for messages.yaml + the three state-table files + equipment.yaml. 51 — Extending the codebase. Seven recipes ordered from no-code to substantial: new SVID/DVID/ECID (YAML only), new CEID with reports (YAML only), new host command (YAML + optional handler), new control- state transition (YAML only), new SECS-II message (YAML + handler), new store (header + tests), new persistence backend (drop-in vs pluggable trade-off). Each recipe has the actual mechanical steps, the test pattern, and pointers to the chapter that explains why it works. Index updated to mark all 24 chapters published. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
cae98d9a7d |
docs: chapters 30–36 — the codebase (Part 3 complete)
Seven chapters walking the implementation top-to-bottom. 30 — Repository tour. Top-level layout, directory by directory. The eight built binaries. The dependency graph from TCP socket up through EquipmentDataModel. CMake's role. Test layout. 31 — Spec-as-data and codegen. Why the design choice fits SECS/ GEM specifically. The five YAML files: messages catalog, control/PJ/CJ transition tables, equipment dictionary. How tools/gen_messages.py turns messages.yaml into typed C++ at build time. The --validate-config multi-error validator. How to add a new SVID / CEID / host command / state / message without C++. 32 — Stores and the data model. What a store IS (records + API + change handler + optional persistence). Every store in the codebase mapped to the SEMI standard it serves (table of 21). EquipmentDataModel as plain composition + cross-store convenience methods (vid_value, compose_reports_for). The no-locks single- threaded contract. How to add a new store. 33 — Transport. hsms::Connection read path (length+payload async chain), write path (queue + one outstanding write), timer model (5 steady_timers + per-request T3). The asio executor / strand model and why it's the right shape. secsi::Protocol as the IO- free FSM with Action / Event variants; secsi::TcpTransport as the asio adapter. Pattern repeats for E84 + GEM comm-state. 34 — Codec and SML. The four files (170 + 30 + 52 + 32 lines of header, 229 + 220 lines of impl). Item variant storage layout (11 alternatives, 16 formats, shared storage where E5 permits). encode_into recursion; decode_at with bounds checks throwing CodecError. Message wrapper. SML printer + try_parse_sml + why SML round-trips Items but not necessarily bytes. 35 — State machines and dispatch. gem::Router as a typed (stream, function) dispatch table. How an S2F41 round-trip walks through parser → store dispatch → side-effect → CEID emission → S6F11 build → spool-aware deliver. The 11 FSMs all sharing the same three-property shape (pure data table + pure FSM + observer pattern). CEID cascading from FSM transitions to wire bytes. 36 — Persistence, validation, metrics. Which 7 stores have file journals + why the others don't. Per-record file pattern (atomic rename, partial-write safe). Schema versioning + multi-version read. Multi-error YAML validator (--validate-config) + cross-file reference checks. Prometheus registry + HTTP exporter + worked metric patterns from the PVD example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
40df3067a4 |
docs: chapters 14–19 — GEM 300 standards (Part 2 complete)
Six more chapters finishing Part 2. Together with chapters 10–13 they document every SEMI standard this codebase implements. 14 — E40 + E94: process jobs (8-state lifecycle, S16F11/F5/F7/F9 on the wire) and control jobs (CJ wraps PJs with batch policy, S14F9/S16F27 messages). Worked cascade showing how CJSTART propagates through the PJ FSM and triggers S6F11 CEIDs at each transition. 15 — E87 carriers: three orthogonal sub-machines (CarrierID, SlotMap, CarrierAccess) per carrier and three more (Transfer, Reservation, Association) per load port. S3F17 CarrierAction strings + CAACK codes, S3F19 SlotMap verify, the 5-state slot encoding, multi-port concurrency. 16 — E90 + E157: substrate tracking via three orthogonal axes (STS / SPS / SubstrateIDStatus) and module process tracking (NotExecuting / GeneralExecuting / StepExecuting / StepCompleted). End-to-end PVD example showing E40 + E157 + E90 transitions cascading into CEIDs. 17 — E116 + E120 + E39: equipment performance time-buckets across six states, common equipment model object hierarchy, S14F1/F3 GetAttr/SetAttr as the uniform wire access for any object type across multiple standards. 18 — E84 parallel I/O: ten signal lines, the 9-state handshake FSM, the three TA1/TA2/TA3 timing-critical timers, why a physical handshake gets modeled in software (testability, timer enforcement, CEID emission, multi-port concurrency), the pure-FSM + asio-adapter split. 19 — E42 + E148 + S5F9–F18: formatted recipes (S7F23/F25 typed PPBODY), time synchronization with 16-char + 14-char accepted on set, exception recovery as a persistent multi-step host-supervised FSM (Posted → Recovering → Cleared with abort/retry). Revisits the auto-S9 family and contrasts S9 (transport) vs S5F9 (application). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
858ca22975 |
docs: chapters 11–13 — HSMS, SECS-I, GEM
Three more chapters of Part 2: 11 — E37 HSMS. 4-byte length prefix + 10-byte header (R-bit + session id + W-bit + stream + function + PType + SType + system_bytes), the 9 SType control messages, the NOT-SELECTED → SELECTED state machine, T3/T5/T6/T7/T8 with what each one bounds, the auto-S9 paths (S9F1/F3/F5/F7/F9/F11), HSMS-SS vs HSMS-GS, the asio single-threaded contract. 12 — E4 SECS-I. Half-duplex line turnaround (ENQ/EOT/ACK/NAK), the 10-byte block header bit-packing (R-bit / W-bit / E-bit / system bytes), the 244-byte block cap and multi-block split/assemble, the event-driven IO-free FSM with its Action / Event variants, T1/T2/T3/T4 with semantics + defaults, master/slave contention. Notes the deferred asio serial_port adapter; explains why this chapter matters even for HSMS-only readers. 13 — E30 GEM. Disambiguates the three state machines (HSMS transport vs GEM communication vs GEM control), walks the comm-state FSM (DISABLED → WAIT-CRA → COMMUNICATING with T_CRA / T_DELAY) and the control-state FSM (5 states + the YAML transition table). Lists every Fundamental and Additional capability with its messages, code locations, and store assignments. One worked Event-Notification scenario tracing seven on-wire steps to their EquipmentDataModel internals. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
338d0b974d |
docs: chapter 10 — E5 SECS-II data items and encoding
Opens Part 2 (the standards in detail). Walks the entire SECS-II encoding from first principles: the mental model (every value is one Item; a List is a recursive Item), the format-byte arithmetic (6-bit format code, 2-bit length-byte-count), the 14 format codes, length bytes 1/2/3 (with the 16 MiB cap), big-endian everywhere, the difference between byte-count (scalars) and child-count (lists). Then walks every format with worked hexdumps: empty list, nested list, ASCII with length-byte boundary crossing, Binary vs Boolean, U1/U2/U4/U8, signed integers with two's-complement edges, F4 / F8 with NaN / ±Inf / −0.0, JIS-8, C2 Unicode. Then the codebase mapping: Format enum, Item variant storage layout, encode_into / decode_at recursion, SML printer/parser, the identifier-wildcard rule (SEMI allows U1/U2/U4/U8 interchangeably for ID fields) with the messages_helpers::any_unsigned_first<Out> helper that closes the leniency contract. Closes with the well-defined CodecError conditions, what the codec deliberately doesn't reject (unknown format codes), and pointers to chapter 31 (codegen) and chapter 11 (HSMS) as the next dependencies above the codec. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
5fec47ad02 |
ci: bake secs4j harness into image instead of bind-mounting
Second secs4j-interop CI failure: ensuring secs4j-interop image is built... compiling Secs4jHostHarness.java... error: file not found: Secs4jHostHarness.java FAIL: javac The script bind-mounted $PWD/interop/secs4j into /work inside the container so it could javac the harness at runtime. That works locally where docker daemon and script share a filesystem, but fails in CI: the act runner runs the workflow inside a container, the docker socket is mounted from the host, and the daemon interprets bind-mount paths against the host filesystem — where $PWD/interop/secs4j doesn't exist. Result: empty /work, javac errors, job fails. Fix: COPY Secs4jHostHarness.java into the image and javac it at image build time. The script just runs the container — no bind mount, no docker-in-docker mount path translation, works in CI and locally. Verified locally with a fresh image rebuild: 55/55 checks pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
fc3422a4a9 |
docs: move root .md files into docs/ + update every reference
Picks up the file renames that landed alongside the previous commit and fixes everything that pointed at the old root locations: - README.md doc-map updated: every entry now points at docs/X.md, with a new "docs/" lead entry pointing at the guided-tour index. - README inline cross-refs (ARCHITECTURE / INTEGRATION / SECURITY / BENCHMARKS / MES_INTEROP / PROOFS) repointed to docs/. - README "Interop" section rewritten — used to mention only secsgem-py; now covers all four external validators (secsgem-py 31 / secs4java8 55 / tshark 69 frames / libFuzzer 200 k+ runs) with a one-line summary each, plus pointers to interop/README.md and docs/VERIFICATION.md. - README "Deferred follow-ups" cleaned: dropped the explanatory "Listed here so reviewers don't go looking for them in COMPLIANCE.md and find an 'out of scope' entry that sounds defensive" sentence — the section header speaks for itself. - docs/00_index.md "Where the rest of the docs live" table: dropped every `../` prefix since the docs are now siblings. - docs/01_what_is_secs_gem.md PROOFS reference updated to sibling. - docs/02_the_cast.md INTEGRATION + MES_INTEROP refs updated to siblings; dropped the stale "at the repo root" wording. - interop/README.md: VERIFICATION + PROOFS refs updated to ../docs/X.md; stale "~24 + 4 checks" updated to 31 (matches PROOFS.md and README). - examples/pvd_tool/README.md: every doc cross-ref now points at ../../docs/X.md. - Source / data / CI comments mentioning doc names (e.g. "INTEGRATION.md §3", "COMPLIANCE.md gap") rewritten to "docs/INTEGRATION.md §3" etc. — affects 9 files across include/, apps/, tests/, data/, examples/, .gitea/workflows/. Verified: full build under docker passes, 445/445 test cases pass, 2 753/2 753 assertions pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
60fa164626 |
docs: chapters 02 + 03 of the guided tour (Part 1 complete)
02 — The cast of characters: equipment, EAP, MES, fab planner, AMHS, operator. Who initiates which conversation, why the equipment is the passive side of HSMS by convention, how the AMHS handshake is out-of-band relative to SECS. Cross-references the relevant namespace and test files for each actor. 03 — Vocabulary + a wafer's journey: follows one 300 mm wafer end-to-end through a fab and labels every SECS message and acronym that fires. Introduces SVID / DVID / ECID / CEID / RPTID / ALID / PPID / MDLN / SOFTREV / HCACK / ALCD / OFLACK / CAACK / SMACK / etc. in context rather than as a list. Includes one-screen reference tables for the remaining acknowledge codes, T-timers in all four contexts (HSMS / SECS-I / E84 / E30 communication state), and a stream-by-stream summary. Part 1 (Foundations) of the guided tour is now complete — a reader who reads chapters 01–03 can describe the protocol stack, identify the actors, and recognise every acronym they'll meet in Part 2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
bc54de7711 |
ci: secs4j-interop bootstrap resilient to runner image variant
CI log showed: Run export DEBIAN_FRONTEND=noninteractive apt-get: command not found Failure - Main Bootstrap (node + git) exit status 127 The secs4j-interop job runs on the bare runner (not inside a `container:`) because it needs the host's docker socket to run `docker compose up -d server`. The runner image isn't fixed across deployments — catthehacker/ubuntu has apt-get, but a minimal node image doesn't. The old script hard-coded `apt-get` and exit 127'd on anything else. New bootstrap: - Checks what's already on PATH (git, node, docker). If all three are present, exits 0 — most act-runner images come pre-loaded. - Otherwise picks the right package manager (apt-get or apk) and installs only the missing pieces. - Errors out with a useful message if neither package manager exists, instead of failing on a missing command. Also updates the inline comment that still said "20 checks" — actual is 55 (matches the count in README / PROOFS.md / COMPLIANCE.md). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
01acac97d4 |
docs: start guided-tour tutorial series under docs/
A linear teach-from-zero tutorial that walks both SECS/GEM as a protocol family and this codebase as an implementation. Each chapter explains a SEMI concept and shows where it lives in code, so a reader builds a mental model of the standards and the repository simultaneously. Structure (24 chapters across 5 parts): - Part 1 (3 ch) — Foundations: what SECS/GEM is, the cast of characters, vocabulary + a wafer's end-to-end journey - Part 2 (10 ch) — Standards in detail: E5, E37, E4, E30, E40+E94, E87, E90+E157, E116+E120+E39, E84, E42+E148+S9 - Part 3 (7 ch) — Codebase: repository tour, spec-as-data + codegen, stores, transport, codec, state machines, persistence - Part 4 (2 ch) — Operations: build/run/demo, integration - Part 5 (2 ch) — Reference: API + messages + YAML, extension guide Published in this commit: - 00_index.md — guide layout, audience map, reading paths, conventions, status table - 01_what_is_secs_gem.md — the N×M integration problem, what SECS vs. HSMS vs. GEM each actually refer to, the GEM 300 suite, the transport→message→behaviour layering, where each layer lives in this codebase, an end-to-end S2F17/F18 example Chapters publish iteratively from here. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
b01dedfaa5 |
docs: drop COMPLIANCE §8 "out of scope" and broaden §7 to all 4 validators
§8 was carrying two items that neither read as "deliberately out of scope" nor matched the framing of the section: - Equipment Processing States — E30 §6.3 explicitly leaves concrete states tool-defined. The framework ships the ControlTransitionTable engine and YAML loader; vendors supply IDLE/SETUP/READY/EXECUTING. That's a design choice, not a gap. §3 line 94 already documents it. - Serial-port wiring for SECS-I — the FSM is implemented and tested end-to-end over TCP; only the asio serial_port adapter is missing. That's deferred, not out of scope. §1a line 64 already lists it with status ⬜. So §8 is dropped, §9 renumbers to §8, and the deferred follow-up gets its own short section in the README so customers know it's tracked without sounding defensive. §7 used to be titled "Interoperability with secsgem-py 0.3.0" and mentioned only that one external implementation. We now have four external validators (secsgem-py + secs4java8 + tshark dissector + libFuzzer), so the section is renamed "Interoperability with external implementations" and broadened to cover all of them with their actual check counts. Stale "24 named checks" updated to the current 31; "three consecutive clean runs" line dropped as audit-language no longer earning its keep now that it's a CI step. FAQ's "What's not implemented?" answer rewritten to point at the README "Deferred follow-ups" section and COMPLIANCE §8 (new numbering), with a brief note explaining that Equipment Processing States are spec-by-design tool-defined. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
c8e8e80735 |
secs_server: relative-path defaults so the binary runs outside docker
Previously --config / --state-table / --pj-state-table / --cj-state-table defaulted to /app/data/..., which only resolves inside the docker image. A host build run from the repo root errored out unless every flag was passed explicitly. Switch to data/equipment.yaml (and siblings) relative to CWD — docker still works because WORKDIR=/app puts /app/data/... at the same relative location, and host builds run from the repo root resolve to <repo>/data/.... Existing callers that pass explicit paths (the proof commands, tshark_validate.sh, secs4j_validate.sh, docker compose) are unaffected. Verified --validate-config under docker still finds all four YAMLs and the tshark proof still passes (69 frames, 0 malformed). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
b031f057af |
docs: customer-ready sweep + README restructure + tshark CI fix
Audit pass over the public-facing surface so a customer can read it end-to-end without tripping on stale numbers or self-contradictions. README + docs accuracy: - Test counts 426 → 445, assertions 2 557 → 2 753 (verified via doctest run); E5 row was missing test_e5_kat (19 cases) - Interop checks 24 → 31, COMPLIANCE.md message count 149 → 164, COMPLIANCE.md "291 cases / 1515 assertions" → 445 / 2 753 - README "60+ test IDs" for MES_INTEROP → actual 59 - PVD example counts: 32 SVIDs/17 CEIDs → 29/21, "~40 handlers in ~200 lines" → 51 in ~460, "~700 lines" → ~1,100; main.cpp header table-of-contents resynced with the actual 7 sections Out-of-scope honesty (COMPLIANCE.md §8 + FAQ.md): - Removed HSMS-GS (was both ✅ implemented in §1 and "out of scope" in §8; INTEGRATION.md §7 documents using it) - Removed multi-block SECS-I (split_message/assemble_message exist with 4 dedicated tests) - Added serial-port wiring as the genuine open ⬜ item — FSM is tested end-to-end over TCP; only the asio serial_port glue is deferred - COMPLIANCE.md intro now lists E42 and notes "E37 (SS + GS)" README restructure: - Moved the 8-command proof table and per-standard test-coverage table to a new PROOFS.md (72 lines) - README now leads with what / Quick start / Documentation map, then a one-paragraph "How it's proved" linking to PROOFS.md - Updated cross-refs in FAQ.md, GLOSSARY.md, VERIFICATION.md, and interop/README.md to point at PROOFS.md CI fix — tshark-dissector job: - interop/tshark_validate.sh hardcoded /app/build/secs_server etc. which only works inside the docker image. Now derives ROOT from the script's own location and accepts BUILD/SERVER/CLIENT/DATA env overrides, so CI can run it from the workspace dir - Verified still passes in docker (69 frames, 0 malformed) .gitignore: - Added build-fuzz/ and build-tsan/ (were showing as untracked) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
6aa4427186 |
docs: worked PVD-tool vendor example
A fictional Physical Vapor Deposition tool wired end-to-end.
examples/pvd_tool/ is the template a real customer should fork.
Files:
- equipment.yaml: 32 SVIDs (chamber pressure, temperature, source
power, gas flows, cooling water, wafer counters, recipe step
state, EPT name, 4 load ports), 5 DVIDs, 7 ECIDs (setpoints
+ T_CRA/T_DELAY + cleaning interval + retry count), 17 CEIDs
(control state, alarms, process lifecycle, material movement,
EPT), 12 alarms with realistic categories (safety, error,
warning, attention), 3 multi-step recipes (Al / Ti / Cu),
9 host commands.
- main.cpp (~860 lines): the vendor-side application:
§1 helpers + constants
§2 sensor simulator — 4 sensors at 10 Hz + 1 Hz cadences,
random-walk around step-targeted setpoints, asio::post-on-strand
thread-safety pattern
§3 recipe runner — parses recipe body (STEP NAME duration=120s
power=2500W gas=Argon flow=50sccm), walks each step at 1s
per declared-second, fires step-started/completed CEIDs,
drives PJ FSM through ProcessComplete
§4 alarm threshold monitor — chamber-pressure-over-setpoint and
cleaning-interval logic, continuous evaluation, set/clear
emission gated on alarm-enable
§5 EPT cycler — Standby ↔ Productive ↔ UnscheduledDowntime
based on PJ activity + safety alarms
§6 Prometheus exporter on :9090 (pvd_messages_total,
pvd_chamber_pressure_torr, pvd_spool_depth, pvd_events_total,
pvd_alarm_set_total)
§7 Router handlers — full E30 set (~40 handlers) so a host can
do real work
§8 main() — YAML validation, model construction, server wiring,
periodic gauge updates
- README.md: section-by-section walkthrough, what's the same as
apps/secs_server.cpp, what this adds (simulator + recipe runner
+ alarm monitor + EPT cycler + metrics), what's not here
(persistence + E84 + real I/O), and what to change for your tool.
Verification: 47/47 conformance harness checks PASS against the
PVD tool — same as the demo server.
CMakeLists.txt adds the pvd_tool target.
README's documentation map points at examples/pvd_tool/.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
91eec92b73 |
docs: ARCHITECTURE.md — how the codebase fits + how to extend
Customers who want to extend the library had two paths: read the 1200-line apps/secs_server.cpp and guess at conventions, or read every store header and infer the shape. Neither is reasonable. ARCHITECTURE.md walks the five layers (apps → Router+Model → stores → FSMs → transport+codec) with a worked extension recipe per layer: - New SECS-II message (YAML edit + Router handler — no core code) - New state machine (lift from ept_state.hpp or process_job_state.hpp) - New store (paste-and-adapt from alarms.hpp or process_jobs.hpp) - New persistence backend (mirror enable_persistence pattern) - New transport (mirror Connection's contract) Explains the design choices that look unusual: - Spec-as-data — every behavioural rule in YAML, C++ is the engine - I/O-free FSMs — transport classes own asio, everything else is pure - Single-threaded by design + no locks anywhere - No DI framework, no singletons, no shared_ptr-everywhere - Exceptions only for programmer-error / corrupt-input paths Documents the persistence magic-byte registry (0xC4-0xC9 + 0xE5) so the next contributor doesn't collide; documents the codegen pipeline (messages.yaml → gen_messages.py → messages.hpp); maps "you want to understand X" → "read these files in order" for the twelve most common entry points. Doc map in README already points at ARCHITECTURE.md from the prior commit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
195ecc689f |
docs: GLOSSARY + FAQ + interop README refresh + doc-map fixes
Fills four documentation gaps surfaced by the doc audit:
1. README "Documentation map" was missing VERIFICATION.md (the file
that backs the proof-of-feature-completeness claims) and is now
pointing at the new files added in this commit too — ARCHITECTURE,
GLOSSARY, FAQ, examples/pvd_tool/ (the last two land next).
2. interop/README.md only documented secsgem-py. Three of the five
external validators (tshark, secs4j, libFuzzer) plus the E5 KAT
were invisible from the directory's own README. Rewritten as a
complete index — what's external, what each catches, how to run,
what bugs they've already surfaced, when to add a new validator.
3. GLOSSARY.md is new. Every SEMI acronym used in the codebase or
the docs gets one row: SVID, DVID, CEID, RPTID, ALID, ECID, PPID,
MID, CARRIERID, PRJOBID, CTLJOBID, SUBSTID, OBJSPEC, OBJTYPE,
MDLN, SOFTREV, EQPTYP, DATAID + every ACK code (COMMACK, ONLACK,
OFLACK, HCACK, CMDA, ACKC5-7-10, DRACK, LRACK, ERACK, EAC, TIACK,
GRANT, ALCD, OBJACK) + stream/function shorthand + HSMS terms +
T-timers + E84 signals + the standards lineup + codebase shortcuts
("the model", "the router", "the proof", etc.). Cuts week-1
onboarding time.
4. FAQ.md is new. Canonical answers to the questions that come up
once per integration: why HSMS unencrypted, SVID vs DVID, PJ vs
CJ, who fires FSM transitions, what runs on which thread, how to
add a new SECS-II message, ASCII vs Binary, common MES quirks,
how spool works, robustness fuzz vs libFuzzer, conformance vs
interop, what's not implemented.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
db90a21e1d |
verify: expand secs4j harness 20 → 55 checks
Every check the user could ask for now lands. secs4j's
comm.send(stream, function, w, body) takes arbitrary S/F + arbitrary
Secs2 body, so coverage was never coverage-limited by the Java side
— the original 20 was just the minimum to fill the gaps secsgem-py
couldn't reach.
Adds:
- Status data: S1F3, S1F11
- EC management: S2F13, S2F15 (set TimeFormat), S2F29
- Event reports: S2F33, S2F35, S2F37 (full define-link-enable
sequence), S6F15, S6F19, S6F21
- Remote control: S2F41 (modern RCMD=START + observed S6F11),
S2F21 (legacy RCMD=STOP),
S2F41 RCMD=FAULT + observed S5F1
- Alarms: S5F3, S5F5, S5F7
- Spool: S2F43, S6F23
- PP management: S7F1, S7F3, S7F5, S7F17, S7F19
- Terminal: S10F3 (single), S10F5 (multi-line)
- E40 PJ: S16F11 (full E40 body — MF + PRRECIPEMETHOD +
RecipeSpec + mtrloutspec + processparams),
S16F7 (monitor), S16F13 (dequeue)
- Limits: S2F45, S2F47
- Trace: S2F23 (5-field body)
- E39: S14F1 (GetAttr)
Plus a SecsMessageReceiveListener that captures every equipment-
initiated primary into a ConcurrentLinkedQueue and replies to S5F1
(ACKC5=0), S6F11 (ACKC6=0), S16F9 (W=0 no reply) so the
equipment's T3 doesn't fire on our watch. Two checks now assert
the unsolicited path:
- After RCMD=START, an S6F11 with the linked report must arrive
within 400ms
- After RCMD=FAULT, an S5F1 with the alarm must arrive within
400ms
Both observed against the demo equipment.
Result: 55/55 PASS. Two independent implementations
(secsgem-py + secs4java8) now corroborate the wire surface in
overlapping but distinct slices. Full E40 body — the one that
defeated secsgem-py's SFDL grammar — round-trips cleanly through
secs4j.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|