cf230d4119ac9ecbf07eefc1ddb1a6e92cc6caeb
146 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 raphael
|
cf230d4119 |
chore(phase0): name validation, golden frames, daemon into library tree, TSan daemon lane
Item 8a — ConfigValidator warns on non-identifier variable/event/alarm/ command names ([A-Za-z_][A-Za-z0-9_]*): language bindings expose names as kwargs/attributes, so 'Chamber Pressure' would be unusable in the planned Python client. Warning not error — the wire doesn't care. Tested (4 warning sites + good-name negative). Item 4 tail — golden frames for S5F1 (Binary ALCD / U4 ALID / ASCII ALTX) and a composed S6F11 (the production-critical report shape), bytes hand- computed from E5 encoding rules: external pins on message composition. Item 7 — equipment_service.hpp moved to include/secsgem/daemon/ (apps/ include-path hack removed) and a TSan daemon lane added locally + in CI. tools/tsan.supp suppresses races whose accesses sit entirely inside the UNinstrumented system libgrpc/libgpr/libabsl (epoll wakeups, absl Mutex GraphCycles bookkeeping); our frames stay fully checked. The lane earned its keep on first run: it caught a REAL threading-contract violation — a daemon test reading model stores from the test thread while the io thread serviced posted writes — fixed to use read_sync, exactly per the documented contract. Now TSan-clean under halt_on_error=1 in the full production threading shape. Suites: core 470/3068, daemon Release+TSan 125/125 each. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
e6ee927900 |
feat(daemon): Subscribe command stream + CompleteCommand — the vendor loop closes
The HCACK-4 contract, implemented end to end. For every YAML-declared command the service registers a forwarding handler (new HostCommandRegistry names()/spec() accessors): with a subscribed tool client the command is queued onto the Subscribe stream (id + name + params via from_item) and the host is answered S2F42 HCACK=4 immediately — never blocking the io thread or the T3 window; with NO subscriber the command takes its declarative YAML ack (the honest pre-daemon behaviour). Settled + documented in the proto: v1 is a firehose with no buffering/replay. CompleteCommand correlates the pending id (audit; unknown id => PARAMETER_INVALID). Side effects stay suppressed on HCACK-4 (router applies them only on Accept), so the completion event the TOOL fires is the host's real signal — exactly E30's intent. Tests (daemon suite 101 -> 124 assertions): a real S2F41 dispatched through the full default-handler router ON the io thread under run_async — HCACK 4 with subscriber + params on the stream, declarative Accept without, CompleteCommand known/unknown, fallback restored after unsubscribe. Interop (now 20 checks, all green): the complete conformant loop against the secsgem-py reference host — S2F41 START -> S2F42 HCACK=4 -> tool receives Command(name=START, id=1) -> CompleteCommand -> FireEvent -> host receives S6F11. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
1da56f973f |
feat(daemon): alarms by name + RequestControlState + WatchHealth (Phase A complete)
A2 — alarms: optional 'name:' on alarm config (a LOCAL key — SEMI only
defines numeric ALID + freetext ALTX; field appended last so existing
{id, text, category} brace-inits compile unchanged), parsed by the loader,
checked by the validator, shipped in equipment.yaml. SetAlarm/ClearAlarm
RPCs resolve config name OR stringified ALID via a constructor snapshot.
A3 — control state + health: RequestControlState fires operator events on
the io thread (read_sync) and reports what the E30 table actually did —
ACCEPT iff the equipment landed in the requested state, CANNOT_DO_NOW naming
the actual state otherwise (the shipped table has no operator path to
EquipmentOffline; the test pins that honesty). ATTEMPT_ONLINE is rejected as
transient. WatchHealth streams an immediate snapshot then pushes on link/
control-state changes via service observers (add_link_observer +
add_control_state_observer — the HandlerSlot work paying off), spool depth
sampled at the 500ms poll; ends on cancel or engine stop.
Tests: daemon suite 61 -> 101 assertions (alarm lifecycle by name/id/unknown,
WatchHealth initial + change push, all four RequestControlState semantics);
loader test for the alarm name (present + absent fallback); core 467/3055.
Interop now 15 checks incl. gRPC SetAlarm -> host receives S5F1 ALCD=0x84
ALID=1, and RequestControlState(HOST_OFFLINE) -> GetControlState confirms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
raphael
|
83593bb508 |
docs: refresh stale roadmap status rows (GetVariables shipped, harnesses automated)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
1daf120431 |
feat(daemon): GetVariables + read_sync — the standard mutable-read pattern
EquipmentRuntime::read_sync establishes THE pattern for reading mutable engine state from gRPC/binding threads (Phase 0 item 6): post the read onto the io thread (the model's single owner), wait on a future with a deadline, nullopt => UNAVAILABLE at the RPC edge. Always truthful, no cache to invalidate; milliseconds are irrelevant at SECS rates. GetVariables: name resolution against the service snapshot (empty query = all; unknown name => INVALID_ARGUMENT naming the offender), values read via read_sync, converted by the new from_item reverse conversion (single-element numeric arrays => scalars, multi-element => List; Boolean/Binary/text per format; C2-as-integer and U8>2^63 wrap documented as TODOs). Tests run the engine in run_async — the daemon's PRODUCTION threading mode, previously untested — and round-trip through both conversions: SetVariables (declared-format write) then GetVariables (read) over a real in-process channel. Daemon suite 41 -> 61 assertions. daemon_interop.py gains a live GetVariables round-trip check vs the running daemon (verified green). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
b0a4c331cf |
test(gem): table-driven conformance sweep over the default handler set
One ordered in-process scenario drives 53 of the 56 registered handlers through Router::dispatch — S1 identification/comms/control, S2 ECs/clock/ event-config/commands/trace/limits/spool, S5 alarms+exceptions, S6 reports, S7 recipes, S10 terminal, S14/S16 E39+E40/E94 jobs, S3 carriers — asserting every reply is the paired (stream, function+1) with a body, plus targeted state checks (OnlineRemote after S1F17, PJ exists after S16F11, HostOffline after S1F15) and the Router's SxF0 abort fallback for unregistered W=1 primaries. Same flow secs_conformance runs over a live socket, but cheap enough for every build; closes the '56 handlers, 4 direct tests' gap from the design review. Also seeds message-level golden frames: S1F13's body pinned to bytes hand-computed from the E5 encoding rules — an external check on message composition, not our codec validating itself (TODO: S5F1, composed S6F11). Suite: 466 cases / 3052 assertions (+236), all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
42044e92e2 |
ci(interop): one-command external-validation suite + CI lanes for the daemon
tests / build-and-test (push) Successful in 2m42s
tests / thread-sanitizer (push) Successful in 2m50s
tests / tshark-dissector (push) Successful in 2m24s
tests / secs4j-interop (push) Successful in 37s
tests / python-interop (push) Successful in 2m56s
tests / libfuzzer (push) Successful in 3m44s
tools/run_interop.sh runs ALL nine validation steps with a PASS/FAIL summary: build, unit (464), daemon-unit (41), secsgem-py host vs server (31 checks), secs_conformance (47), gRPC+secsgem-py daemon bridge, spool persistence across restart, tshark HSMS dissector, secs4java8 (55 checks). Verified green end-to-end. The unit suite is partly self-referential (our parsers validate our builders); these external validators are the real oracle — now they run with one command instead of by hand. Two bugs found by running it: unbounded ninja at -O3 OOM-kills cc1plus in memory-constrained Docker VMs (build with -j 2) and bash-3.2 lacks negative array subscripts. CI: grpc deps added to the build job so secs_gemd + secs_gemd_tests build and RUN in CI (previously the daemon silently dropped out — now fails loudly if missing), plus a python-interop lane running py-host/conformance/daemon harnesses against localhost in one container (no docker-in-docker). Service hardening while in there: reject proto Values with no kind set at the RPC edge (previously silently became ASCII ""), TODO markers for list element formats and daemon graceful shutdown. New tests: unset-Value guard + a property test iterating ALL configured variables via gRPC asserting each keeps its declared SECS-II format (daemon tests 16 -> 41 assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
941f9ef458 |
docs: add Phase 0 (structural debts from design review); fix CompleteCommand contract comment
Phase 0 captures the 2026-06-10 review: multi-observer callbacks (done for the critical three), CI for the interop/conformance harnesses (the unit suite is partly self-referential; the external validators are the real oracle), table-driven handler conformance + message-level golden frames, register_default_handlers decomposition per GEM capability + YAML role bindings for today's magic constants, the post+future mutable-read pattern, service relocation + TSan run_async daemon test, identifier-safe name validation. CompleteCommand's proto comment described the rejected blocking model; it now states the settled HCACK-4 contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
8a48ffeed4 |
feat(gem): multi-observer state-change handlers via HandlerSlot
The single-slot set_*_handler pattern was a structural blocker, hit twice: the daemon could not observe control-state changes because register_default_handlers owns the slot, forcing GetControlState to read the FSM cross-thread (a data race), and blocking WatchHealth and the Subscribe stream's ControlStateChange variant. HandlerSlot<Args...> keeps a primary slot with exact legacy semantics (set_ replaces — one existing test depends on replacement) plus an append-only observer list (add_) that survives set_ calls. Fire sites are textually unchanged (operator bool / operator() / assign-from-function). Applied to ControlStateMachine + ProcessJobStore + ControlJobStore (the roadmap-critical three; the remaining single-slot classes follow the same 3-line pattern as needed). EquipmentRuntime gains an atomic control-state mirror registered as an observer — control_state() is now safe from any thread, retiring the GetControlState race — plus add_control_state_observer and add_link_observer (selected/closed fan-out), the hooks WatchHealth and Subscribe need. Tests: observer ordering, set-replaces-primary-but-observers-survive, observers-without-primary, PJ-store coexistence, and the runtime scenario that was previously impossible (mirror + observer + default-handlers set_). Core 464/464 (2816 assertions), daemon 16/16, live GEM300 demo passes with single-fire control-state transitions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
b067a76b80 |
docs: rewrite daemon roadmap as ordered plan with known-issues audit
Status table brought current (format-aware daemon, secsgem-py interop), the stale Layer-0 section replaced, and the path to an excellent GEM300 repo laid out as ordered phases A–F: finish universal RPCs, the Subscribe command stream (HCACK-4 design written down as the implementation contract), the Python client package, GEM300 job/carrier in-the-loop, hardening/CI, and the fab-acceptance track. Known-issues section records what the audit found (GetControlState enum race + why the state-change-handler slot can't be reused, missing alarm name key, pvd_tool predating set_handler, manual interop harnesses, TSan gap). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
99bfa794fc |
fix(daemon): honour declared SECS-II formats + make service thread-safe
Audit fixes for two real bugs in the gRPC service: 1. Format compliance: to_item() wrote F8/I8 regardless of the variable's declared wire format, so values contradicted the S1F11/S1F21 namelists (ChamberPressure is F4, WaferCounter U4; the interop trace showed <F8 2.5> on the wire). Conversion now targets the declared format — verified end-to-end: secsgem-py now receives <F4 2.5> in S6F11. 2. Thread safety: gRPC handler threads called resolve_variable/resolve_event, copying live store entries (including Item values) while the io thread mutates them. The service now snapshots the immutable name->id/format maps at construction (before run_async, per the documented ordering); all writes already post to the io thread. Remaining known narrow race (GetControlState enum read) documented in DAEMON_ROADMAP. Also: drop a stale tools/run_interop.sh reference from docker-compose.yml. Tests: daemon in-process 16/16 (new F4/U4 format assertions), core 459/459, secsgem-py interop green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
92afbd2a37 |
docs: record secsgem-py daemon interop in roadmap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
3d72e50b65 |
test(interop): daemon end-to-end vs secsgem-py reference host
daemon_interop.py drives a running secs_gemd through BOTH faces at once: a
gRPC tool client and a secsgem-py active host. Proves the gRPC<->HSMS bridge
against a reference GEM implementation, not just in-process:
- gRPC GetControlState agrees with the HSMS-driven control state
- gRPC SetVariables(ChamberPressure=2.5) + FireEvent(ProcessStarted) makes
the host receive S6F11 CEID 300 carrying 2.5 (value flowed gRPC -> engine
-> HSMS -> host)
- unknown variable/event names rejected at the gRPC edge
Mirrors the existing host_vs_cpp_server.py pattern. New 'gemd' compose service
(HSMS :5000 + gRPC :50051); interop image gains grpcio/grpcio-tools (proto
stubs generated at runtime, flat to avoid the secsgem package-name clash).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
raphael
|
dd288eb2ac |
docs: update daemon roadmap — gRPC toolchain done, secs_gemd serving
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
cb85199f49 |
feat(daemon): FireEvent + event name resolution + in-process gRPC tests
- name_index: add resolve_event(name) -> CEID (unit-tested). - equipment_service.hpp: extract the gRPC service + value/state conversion into a shared header; add FireEvent (optional per-fire variable values, then trigger the collection event by name). secs_gemd slims to main(). - test_daemon_service: real in-process gRPC integration test (client stub -> service -> EquipmentRuntime) proving SetVariables lands in the model, GetControlState reports the state, FireEvent and unknown-name paths behave. Separate secs_gemd_tests target (links grpc++/proto), gated on the daemon. Core suite 459/459 (2799 assertions); daemon gRPC tests 15/15. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
fc898f8410 |
feat: EquipmentRuntime engine owner + secs_gemd gRPC daemon
Extract the SECS/GEM engine wiring out of the secs_server app into a reusable class, and stand up a language-agnostic gRPC daemon on top so a tool's software (any language) can drive the equipment without linking C++ or knowing SEMI. Foundation for replacing a vendor's SECS/GEM server. Engine reuse: - EquipmentRuntime (include/secsgem/gem/runtime.hpp, src/gem/runtime.cpp): owns io_context, passive Server, model, control-state machine, Router; thread-safe outbound API (set_variable/emit_event/set_alarm/clear_alarm), on_command hook, deliver_or_spool, run()/run_async()/poll()/stop(). - register_default_handlers (src/gem/default_handlers.cpp): the 56 GEM handlers + domain emitters, relocated from secs_server so the app and the daemon speak byte-identical GEM. secs_server.cpp reduced ~1270 -> 113 lines. - name_index.hpp: resolve_variable(name) -> VID (the name->id binding layer). Daemon (apps/secs_gemd.cpp, proto/secsgem/v1/equipment.proto): - runs the engine + HSMS link on a background thread; serves the gRPC Equipment service. Increment 1: SetVariables (name-resolved, plain value->Item) and GetControlState. proto carries the full v1 surface (universal + carrier/recipe/job tiers); remaining RPCs + the Subscribe command stream are next (docs/DAEMON_ROADMAP.md). - CMake: opt-in SECSGEM_DAEMON, protoc/grpc_cpp_plugin codegen, gracefully skipped where protobuf/grpc++ are absent. Dockerfile gains the grpc deps. Tests (proof): test_runtime, test_default_handlers (S1F1->S1F2, S2F41->hook), test_name_index. Full suite 458/458, 2795 assertions; live server<->client GEM300 demo still passes on the refactored server. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
4b4b2ac690 |
docs: correct drifted and fabricated APIs in chapters 13/17/35/51
An audit of doc code blocks against the real headers found APIs that do not exist in the codebase, presented as authoritative walkthroughs: - ch35 (dispatch): an entirely fabricated callback architecture — HostCommandRegistry::set_emit_ceid_handler, CommandOutcome, emit_ceids. Rewritten to the real Spec/Result/dispatch + the new set_handler hook. - ch13 (E30): wrong store names — EventStore/ReportStore -> EventReportSubscriptions, SvidStore -> StatusVariableStore, AlarmStore/AlarmDispatcher -> AlarmRegistry, ClockStore -> Clock, TerminalServiceStore -> (no store), in both the capability tables and the worked S2F33 example. - ch17 (E116): EptStore/seconds/bucket_ -> EptStateMachine/milliseconds/buckets_. - ch51 (extending): stale host-command handler -> the real set_handler signature. Verified clean by grep: no fabricated symbols remain in docs/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
0090791968 |
feat(gem): add host-command behaviour hook to HostCommandRegistry
Host commands were declarative-only: dispatch() returned the YAML-defined HCACK plus side effects, and ignored the command parameters entirely (the param list was a commented-out argument). Equipment could acknowledge a command but never run anything in response — the pvd_tool example worked around this by hard-coding behaviour in a C++ router handler. Add set_handler(rcmd, fn): a registered handler receives the live CPNAME/ CPVAL parameters and returns the HCACK, overriding the declarative default. Live on S2F41/F21/F49 via the shared dispatch(). No handler => byte-for-byte the previous declarative behaviour. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
raphael
|
dae6bfd747 |
docs: streamline tone across reference docs
Tone pass across the non-tutorial markdown — README, PROOFS,
ARCHITECTURE, BENCHMARKS, COMPLIANCE, FAQ, MES_INTEROP, SECURITY,
and interop/README. Three patterns came out:
- Bug-history war stories ("Past interop sweeps surfaced…",
"What these harnesses caught: 1. Strict U-width parsing…").
- Chat-with-reader framing ("Don't skip TLS unless…", "Treat as a
punch list", "If you're running in a pod…", "Misconfiguration
incidents drop dramatically").
- Self-referential narration ("we ship", "our codec", "the
codebase's most-tested layer", "three orders of magnitude above
fab load", "the gift that keeps giving").
README also drops the standalone ThreadSanitizer subsection under
Build details (now a single line under the new Testing section).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
d63c92166d |
docs: rewrite VERIFICATION.md to describe shipped validators
Previously written as a forward-looking plan ("Plan: (1) KAT → (2)
tshark → (3) secs4j → (4) libFuzzer", "Effort: ~3 hours", "Survey
step (do this first)"). All four validators have shipped —
test_e5_kat.cpp, interop/secs4j/Secs4jHostHarness.java,
interop/tshark_validate.sh, apps/fuzz_*.cpp. Rewritten as
documentation of what's there: file paths, CI job names, actual
result numbers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
0355c73211 |
docs: refresh stale file paths after store/ reorg + gen_messages rename
generate_messages.py → gen_messages.py and several gem/ headers moved under gem/store/ (carrier_store.hpp → store/carriers.hpp, etc.); e84.hpp split into e84_state.hpp. The guided-tour chapters still pointed at the old paths — relink them so the deep-link footnotes resolve. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
fee82d88c9 |
ci: self-contained secs_server image for secs4j interop
The harness previously bound the source tree into a compose service and built inside it. That breaks under docker-in-docker (gitea-act, GitHub Actions runners with /var/run/docker.sock mounted) because bind-mount sources resolve against the *host* daemon's filesystem, not the runner container's. Now Dockerfile.server bakes a Release secs_server into its own image, and secs4j_validate.sh wires server and harness together on a dedicated bridge — no volumes needed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
31f908e1bf |
docs: chapters 40, 41, 50, 51 — Operations + Reference (series complete)
Last four chapters of the guided tour. 40 — Building, running, the demo. Docker prerequisites, the build flow, what each binary is for, running the 24-transaction demo flow annotated step by step. Running the 4 external-validator sweeps + the libFuzzer pass. Inspecting the demo with tcpdump and tshark. Reading source while running as the recommended learning workflow. 41 — Integration: hardware, MES, production. Four-phase tour: wiring sensors / recipe engine / alarms / E84 GPIO; talking to a real MES with the day-1 punch list + commercial-MES quirks (Wonderware S2F21, Camstar Linktest cadence, etc.); production hardening (nftables / stunnel / minisign / persistence layout / monitoring / runbook); performance envelope + memory footprint + capacity planning. Pointers to the long-form INTEGRATION.md / MES_INTEROP.md / SECURITY.md / BENCHMARKS.md. 50 — API + message catalog + YAML schemas reference. Namespace-by- namespace table of public symbols (secs2, hsms, secsi, gem, config, metrics) with brief descriptions. Stream-by-stream message catalog reference (S1, S2, S3, S5, S6, S7, S9, S10, S12, S14, S16). YAML schema reference for messages.yaml + the three state-table files + equipment.yaml. 51 — Extending the codebase. Seven recipes ordered from no-code to substantial: new SVID/DVID/ECID (YAML only), new CEID with reports (YAML only), new host command (YAML + optional handler), new control- state transition (YAML only), new SECS-II message (YAML + handler), new store (header + tests), new persistence backend (drop-in vs pluggable trade-off). Each recipe has the actual mechanical steps, the test pattern, and pointers to the chapter that explains why it works. Index updated to mark all 24 chapters published. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
cae98d9a7d |
docs: chapters 30–36 — the codebase (Part 3 complete)
Seven chapters walking the implementation top-to-bottom. 30 — Repository tour. Top-level layout, directory by directory. The eight built binaries. The dependency graph from TCP socket up through EquipmentDataModel. CMake's role. Test layout. 31 — Spec-as-data and codegen. Why the design choice fits SECS/ GEM specifically. The five YAML files: messages catalog, control/PJ/CJ transition tables, equipment dictionary. How tools/gen_messages.py turns messages.yaml into typed C++ at build time. The --validate-config multi-error validator. How to add a new SVID / CEID / host command / state / message without C++. 32 — Stores and the data model. What a store IS (records + API + change handler + optional persistence). Every store in the codebase mapped to the SEMI standard it serves (table of 21). EquipmentDataModel as plain composition + cross-store convenience methods (vid_value, compose_reports_for). The no-locks single- threaded contract. How to add a new store. 33 — Transport. hsms::Connection read path (length+payload async chain), write path (queue + one outstanding write), timer model (5 steady_timers + per-request T3). The asio executor / strand model and why it's the right shape. secsi::Protocol as the IO- free FSM with Action / Event variants; secsi::TcpTransport as the asio adapter. Pattern repeats for E84 + GEM comm-state. 34 — Codec and SML. The four files (170 + 30 + 52 + 32 lines of header, 229 + 220 lines of impl). Item variant storage layout (11 alternatives, 16 formats, shared storage where E5 permits). encode_into recursion; decode_at with bounds checks throwing CodecError. Message wrapper. SML printer + try_parse_sml + why SML round-trips Items but not necessarily bytes. 35 — State machines and dispatch. gem::Router as a typed (stream, function) dispatch table. How an S2F41 round-trip walks through parser → store dispatch → side-effect → CEID emission → S6F11 build → spool-aware deliver. The 11 FSMs all sharing the same three-property shape (pure data table + pure FSM + observer pattern). CEID cascading from FSM transitions to wire bytes. 36 — Persistence, validation, metrics. Which 7 stores have file journals + why the others don't. Per-record file pattern (atomic rename, partial-write safe). Schema versioning + multi-version read. Multi-error YAML validator (--validate-config) + cross-file reference checks. Prometheus registry + HTTP exporter + worked metric patterns from the PVD example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
40df3067a4 |
docs: chapters 14–19 — GEM 300 standards (Part 2 complete)
Six more chapters finishing Part 2. Together with chapters 10–13 they document every SEMI standard this codebase implements. 14 — E40 + E94: process jobs (8-state lifecycle, S16F11/F5/F7/F9 on the wire) and control jobs (CJ wraps PJs with batch policy, S14F9/S16F27 messages). Worked cascade showing how CJSTART propagates through the PJ FSM and triggers S6F11 CEIDs at each transition. 15 — E87 carriers: three orthogonal sub-machines (CarrierID, SlotMap, CarrierAccess) per carrier and three more (Transfer, Reservation, Association) per load port. S3F17 CarrierAction strings + CAACK codes, S3F19 SlotMap verify, the 5-state slot encoding, multi-port concurrency. 16 — E90 + E157: substrate tracking via three orthogonal axes (STS / SPS / SubstrateIDStatus) and module process tracking (NotExecuting / GeneralExecuting / StepExecuting / StepCompleted). End-to-end PVD example showing E40 + E157 + E90 transitions cascading into CEIDs. 17 — E116 + E120 + E39: equipment performance time-buckets across six states, common equipment model object hierarchy, S14F1/F3 GetAttr/SetAttr as the uniform wire access for any object type across multiple standards. 18 — E84 parallel I/O: ten signal lines, the 9-state handshake FSM, the three TA1/TA2/TA3 timing-critical timers, why a physical handshake gets modeled in software (testability, timer enforcement, CEID emission, multi-port concurrency), the pure-FSM + asio-adapter split. 19 — E42 + E148 + S5F9–F18: formatted recipes (S7F23/F25 typed PPBODY), time synchronization with 16-char + 14-char accepted on set, exception recovery as a persistent multi-step host-supervised FSM (Posted → Recovering → Cleared with abort/retry). Revisits the auto-S9 family and contrasts S9 (transport) vs S5F9 (application). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
858ca22975 |
docs: chapters 11–13 — HSMS, SECS-I, GEM
Three more chapters of Part 2: 11 — E37 HSMS. 4-byte length prefix + 10-byte header (R-bit + session id + W-bit + stream + function + PType + SType + system_bytes), the 9 SType control messages, the NOT-SELECTED → SELECTED state machine, T3/T5/T6/T7/T8 with what each one bounds, the auto-S9 paths (S9F1/F3/F5/F7/F9/F11), HSMS-SS vs HSMS-GS, the asio single-threaded contract. 12 — E4 SECS-I. Half-duplex line turnaround (ENQ/EOT/ACK/NAK), the 10-byte block header bit-packing (R-bit / W-bit / E-bit / system bytes), the 244-byte block cap and multi-block split/assemble, the event-driven IO-free FSM with its Action / Event variants, T1/T2/T3/T4 with semantics + defaults, master/slave contention. Notes the deferred asio serial_port adapter; explains why this chapter matters even for HSMS-only readers. 13 — E30 GEM. Disambiguates the three state machines (HSMS transport vs GEM communication vs GEM control), walks the comm-state FSM (DISABLED → WAIT-CRA → COMMUNICATING with T_CRA / T_DELAY) and the control-state FSM (5 states + the YAML transition table). Lists every Fundamental and Additional capability with its messages, code locations, and store assignments. One worked Event-Notification scenario tracing seven on-wire steps to their EquipmentDataModel internals. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
338d0b974d |
docs: chapter 10 — E5 SECS-II data items and encoding
Opens Part 2 (the standards in detail). Walks the entire SECS-II encoding from first principles: the mental model (every value is one Item; a List is a recursive Item), the format-byte arithmetic (6-bit format code, 2-bit length-byte-count), the 14 format codes, length bytes 1/2/3 (with the 16 MiB cap), big-endian everywhere, the difference between byte-count (scalars) and child-count (lists). Then walks every format with worked hexdumps: empty list, nested list, ASCII with length-byte boundary crossing, Binary vs Boolean, U1/U2/U4/U8, signed integers with two's-complement edges, F4 / F8 with NaN / ±Inf / −0.0, JIS-8, C2 Unicode. Then the codebase mapping: Format enum, Item variant storage layout, encode_into / decode_at recursion, SML printer/parser, the identifier-wildcard rule (SEMI allows U1/U2/U4/U8 interchangeably for ID fields) with the messages_helpers::any_unsigned_first<Out> helper that closes the leniency contract. Closes with the well-defined CodecError conditions, what the codec deliberately doesn't reject (unknown format codes), and pointers to chapter 31 (codegen) and chapter 11 (HSMS) as the next dependencies above the codec. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
5fec47ad02 |
ci: bake secs4j harness into image instead of bind-mounting
Second secs4j-interop CI failure: ensuring secs4j-interop image is built... compiling Secs4jHostHarness.java... error: file not found: Secs4jHostHarness.java FAIL: javac The script bind-mounted $PWD/interop/secs4j into /work inside the container so it could javac the harness at runtime. That works locally where docker daemon and script share a filesystem, but fails in CI: the act runner runs the workflow inside a container, the docker socket is mounted from the host, and the daemon interprets bind-mount paths against the host filesystem — where $PWD/interop/secs4j doesn't exist. Result: empty /work, javac errors, job fails. Fix: COPY Secs4jHostHarness.java into the image and javac it at image build time. The script just runs the container — no bind mount, no docker-in-docker mount path translation, works in CI and locally. Verified locally with a fresh image rebuild: 55/55 checks pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
fc3422a4a9 |
docs: move root .md files into docs/ + update every reference
Picks up the file renames that landed alongside the previous commit and fixes everything that pointed at the old root locations: - README.md doc-map updated: every entry now points at docs/X.md, with a new "docs/" lead entry pointing at the guided-tour index. - README inline cross-refs (ARCHITECTURE / INTEGRATION / SECURITY / BENCHMARKS / MES_INTEROP / PROOFS) repointed to docs/. - README "Interop" section rewritten — used to mention only secsgem-py; now covers all four external validators (secsgem-py 31 / secs4java8 55 / tshark 69 frames / libFuzzer 200 k+ runs) with a one-line summary each, plus pointers to interop/README.md and docs/VERIFICATION.md. - README "Deferred follow-ups" cleaned: dropped the explanatory "Listed here so reviewers don't go looking for them in COMPLIANCE.md and find an 'out of scope' entry that sounds defensive" sentence — the section header speaks for itself. - docs/00_index.md "Where the rest of the docs live" table: dropped every `../` prefix since the docs are now siblings. - docs/01_what_is_secs_gem.md PROOFS reference updated to sibling. - docs/02_the_cast.md INTEGRATION + MES_INTEROP refs updated to siblings; dropped the stale "at the repo root" wording. - interop/README.md: VERIFICATION + PROOFS refs updated to ../docs/X.md; stale "~24 + 4 checks" updated to 31 (matches PROOFS.md and README). - examples/pvd_tool/README.md: every doc cross-ref now points at ../../docs/X.md. - Source / data / CI comments mentioning doc names (e.g. "INTEGRATION.md §3", "COMPLIANCE.md gap") rewritten to "docs/INTEGRATION.md §3" etc. — affects 9 files across include/, apps/, tests/, data/, examples/, .gitea/workflows/. Verified: full build under docker passes, 445/445 test cases pass, 2 753/2 753 assertions pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
60fa164626 |
docs: chapters 02 + 03 of the guided tour (Part 1 complete)
02 — The cast of characters: equipment, EAP, MES, fab planner, AMHS, operator. Who initiates which conversation, why the equipment is the passive side of HSMS by convention, how the AMHS handshake is out-of-band relative to SECS. Cross-references the relevant namespace and test files for each actor. 03 — Vocabulary + a wafer's journey: follows one 300 mm wafer end-to-end through a fab and labels every SECS message and acronym that fires. Introduces SVID / DVID / ECID / CEID / RPTID / ALID / PPID / MDLN / SOFTREV / HCACK / ALCD / OFLACK / CAACK / SMACK / etc. in context rather than as a list. Includes one-screen reference tables for the remaining acknowledge codes, T-timers in all four contexts (HSMS / SECS-I / E84 / E30 communication state), and a stream-by-stream summary. Part 1 (Foundations) of the guided tour is now complete — a reader who reads chapters 01–03 can describe the protocol stack, identify the actors, and recognise every acronym they'll meet in Part 2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
bc54de7711 |
ci: secs4j-interop bootstrap resilient to runner image variant
CI log showed: Run export DEBIAN_FRONTEND=noninteractive apt-get: command not found Failure - Main Bootstrap (node + git) exit status 127 The secs4j-interop job runs on the bare runner (not inside a `container:`) because it needs the host's docker socket to run `docker compose up -d server`. The runner image isn't fixed across deployments — catthehacker/ubuntu has apt-get, but a minimal node image doesn't. The old script hard-coded `apt-get` and exit 127'd on anything else. New bootstrap: - Checks what's already on PATH (git, node, docker). If all three are present, exits 0 — most act-runner images come pre-loaded. - Otherwise picks the right package manager (apt-get or apk) and installs only the missing pieces. - Errors out with a useful message if neither package manager exists, instead of failing on a missing command. Also updates the inline comment that still said "20 checks" — actual is 55 (matches the count in README / PROOFS.md / COMPLIANCE.md). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
01acac97d4 |
docs: start guided-tour tutorial series under docs/
A linear teach-from-zero tutorial that walks both SECS/GEM as a protocol family and this codebase as an implementation. Each chapter explains a SEMI concept and shows where it lives in code, so a reader builds a mental model of the standards and the repository simultaneously. Structure (24 chapters across 5 parts): - Part 1 (3 ch) — Foundations: what SECS/GEM is, the cast of characters, vocabulary + a wafer's end-to-end journey - Part 2 (10 ch) — Standards in detail: E5, E37, E4, E30, E40+E94, E87, E90+E157, E116+E120+E39, E84, E42+E148+S9 - Part 3 (7 ch) — Codebase: repository tour, spec-as-data + codegen, stores, transport, codec, state machines, persistence - Part 4 (2 ch) — Operations: build/run/demo, integration - Part 5 (2 ch) — Reference: API + messages + YAML, extension guide Published in this commit: - 00_index.md — guide layout, audience map, reading paths, conventions, status table - 01_what_is_secs_gem.md — the N×M integration problem, what SECS vs. HSMS vs. GEM each actually refer to, the GEM 300 suite, the transport→message→behaviour layering, where each layer lives in this codebase, an end-to-end S2F17/F18 example Chapters publish iteratively from here. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
b01dedfaa5 |
docs: drop COMPLIANCE §8 "out of scope" and broaden §7 to all 4 validators
§8 was carrying two items that neither read as "deliberately out of scope" nor matched the framing of the section: - Equipment Processing States — E30 §6.3 explicitly leaves concrete states tool-defined. The framework ships the ControlTransitionTable engine and YAML loader; vendors supply IDLE/SETUP/READY/EXECUTING. That's a design choice, not a gap. §3 line 94 already documents it. - Serial-port wiring for SECS-I — the FSM is implemented and tested end-to-end over TCP; only the asio serial_port adapter is missing. That's deferred, not out of scope. §1a line 64 already lists it with status ⬜. So §8 is dropped, §9 renumbers to §8, and the deferred follow-up gets its own short section in the README so customers know it's tracked without sounding defensive. §7 used to be titled "Interoperability with secsgem-py 0.3.0" and mentioned only that one external implementation. We now have four external validators (secsgem-py + secs4java8 + tshark dissector + libFuzzer), so the section is renamed "Interoperability with external implementations" and broadened to cover all of them with their actual check counts. Stale "24 named checks" updated to the current 31; "three consecutive clean runs" line dropped as audit-language no longer earning its keep now that it's a CI step. FAQ's "What's not implemented?" answer rewritten to point at the README "Deferred follow-ups" section and COMPLIANCE §8 (new numbering), with a brief note explaining that Equipment Processing States are spec-by-design tool-defined. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
c8e8e80735 |
secs_server: relative-path defaults so the binary runs outside docker
Previously --config / --state-table / --pj-state-table / --cj-state-table defaulted to /app/data/..., which only resolves inside the docker image. A host build run from the repo root errored out unless every flag was passed explicitly. Switch to data/equipment.yaml (and siblings) relative to CWD — docker still works because WORKDIR=/app puts /app/data/... at the same relative location, and host builds run from the repo root resolve to <repo>/data/.... Existing callers that pass explicit paths (the proof commands, tshark_validate.sh, secs4j_validate.sh, docker compose) are unaffected. Verified --validate-config under docker still finds all four YAMLs and the tshark proof still passes (69 frames, 0 malformed). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
b031f057af |
docs: customer-ready sweep + README restructure + tshark CI fix
Audit pass over the public-facing surface so a customer can read it end-to-end without tripping on stale numbers or self-contradictions. README + docs accuracy: - Test counts 426 → 445, assertions 2 557 → 2 753 (verified via doctest run); E5 row was missing test_e5_kat (19 cases) - Interop checks 24 → 31, COMPLIANCE.md message count 149 → 164, COMPLIANCE.md "291 cases / 1515 assertions" → 445 / 2 753 - README "60+ test IDs" for MES_INTEROP → actual 59 - PVD example counts: 32 SVIDs/17 CEIDs → 29/21, "~40 handlers in ~200 lines" → 51 in ~460, "~700 lines" → ~1,100; main.cpp header table-of-contents resynced with the actual 7 sections Out-of-scope honesty (COMPLIANCE.md §8 + FAQ.md): - Removed HSMS-GS (was both ✅ implemented in §1 and "out of scope" in §8; INTEGRATION.md §7 documents using it) - Removed multi-block SECS-I (split_message/assemble_message exist with 4 dedicated tests) - Added serial-port wiring as the genuine open ⬜ item — FSM is tested end-to-end over TCP; only the asio serial_port glue is deferred - COMPLIANCE.md intro now lists E42 and notes "E37 (SS + GS)" README restructure: - Moved the 8-command proof table and per-standard test-coverage table to a new PROOFS.md (72 lines) - README now leads with what / Quick start / Documentation map, then a one-paragraph "How it's proved" linking to PROOFS.md - Updated cross-refs in FAQ.md, GLOSSARY.md, VERIFICATION.md, and interop/README.md to point at PROOFS.md CI fix — tshark-dissector job: - interop/tshark_validate.sh hardcoded /app/build/secs_server etc. which only works inside the docker image. Now derives ROOT from the script's own location and accepts BUILD/SERVER/CLIENT/DATA env overrides, so CI can run it from the workspace dir - Verified still passes in docker (69 frames, 0 malformed) .gitignore: - Added build-fuzz/ and build-tsan/ (were showing as untracked) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
6aa4427186 |
docs: worked PVD-tool vendor example
A fictional Physical Vapor Deposition tool wired end-to-end.
examples/pvd_tool/ is the template a real customer should fork.
Files:
- equipment.yaml: 32 SVIDs (chamber pressure, temperature, source
power, gas flows, cooling water, wafer counters, recipe step
state, EPT name, 4 load ports), 5 DVIDs, 7 ECIDs (setpoints
+ T_CRA/T_DELAY + cleaning interval + retry count), 17 CEIDs
(control state, alarms, process lifecycle, material movement,
EPT), 12 alarms with realistic categories (safety, error,
warning, attention), 3 multi-step recipes (Al / Ti / Cu),
9 host commands.
- main.cpp (~860 lines): the vendor-side application:
§1 helpers + constants
§2 sensor simulator — 4 sensors at 10 Hz + 1 Hz cadences,
random-walk around step-targeted setpoints, asio::post-on-strand
thread-safety pattern
§3 recipe runner — parses recipe body (STEP NAME duration=120s
power=2500W gas=Argon flow=50sccm), walks each step at 1s
per declared-second, fires step-started/completed CEIDs,
drives PJ FSM through ProcessComplete
§4 alarm threshold monitor — chamber-pressure-over-setpoint and
cleaning-interval logic, continuous evaluation, set/clear
emission gated on alarm-enable
§5 EPT cycler — Standby ↔ Productive ↔ UnscheduledDowntime
based on PJ activity + safety alarms
§6 Prometheus exporter on :9090 (pvd_messages_total,
pvd_chamber_pressure_torr, pvd_spool_depth, pvd_events_total,
pvd_alarm_set_total)
§7 Router handlers — full E30 set (~40 handlers) so a host can
do real work
§8 main() — YAML validation, model construction, server wiring,
periodic gauge updates
- README.md: section-by-section walkthrough, what's the same as
apps/secs_server.cpp, what this adds (simulator + recipe runner
+ alarm monitor + EPT cycler + metrics), what's not here
(persistence + E84 + real I/O), and what to change for your tool.
Verification: 47/47 conformance harness checks PASS against the
PVD tool — same as the demo server.
CMakeLists.txt adds the pvd_tool target.
README's documentation map points at examples/pvd_tool/.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
91eec92b73 |
docs: ARCHITECTURE.md — how the codebase fits + how to extend
Customers who want to extend the library had two paths: read the 1200-line apps/secs_server.cpp and guess at conventions, or read every store header and infer the shape. Neither is reasonable. ARCHITECTURE.md walks the five layers (apps → Router+Model → stores → FSMs → transport+codec) with a worked extension recipe per layer: - New SECS-II message (YAML edit + Router handler — no core code) - New state machine (lift from ept_state.hpp or process_job_state.hpp) - New store (paste-and-adapt from alarms.hpp or process_jobs.hpp) - New persistence backend (mirror enable_persistence pattern) - New transport (mirror Connection's contract) Explains the design choices that look unusual: - Spec-as-data — every behavioural rule in YAML, C++ is the engine - I/O-free FSMs — transport classes own asio, everything else is pure - Single-threaded by design + no locks anywhere - No DI framework, no singletons, no shared_ptr-everywhere - Exceptions only for programmer-error / corrupt-input paths Documents the persistence magic-byte registry (0xC4-0xC9 + 0xE5) so the next contributor doesn't collide; documents the codegen pipeline (messages.yaml → gen_messages.py → messages.hpp); maps "you want to understand X" → "read these files in order" for the twelve most common entry points. Doc map in README already points at ARCHITECTURE.md from the prior commit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
195ecc689f |
docs: GLOSSARY + FAQ + interop README refresh + doc-map fixes
Fills four documentation gaps surfaced by the doc audit:
1. README "Documentation map" was missing VERIFICATION.md (the file
that backs the proof-of-feature-completeness claims) and is now
pointing at the new files added in this commit too — ARCHITECTURE,
GLOSSARY, FAQ, examples/pvd_tool/ (the last two land next).
2. interop/README.md only documented secsgem-py. Three of the five
external validators (tshark, secs4j, libFuzzer) plus the E5 KAT
were invisible from the directory's own README. Rewritten as a
complete index — what's external, what each catches, how to run,
what bugs they've already surfaced, when to add a new validator.
3. GLOSSARY.md is new. Every SEMI acronym used in the codebase or
the docs gets one row: SVID, DVID, CEID, RPTID, ALID, ECID, PPID,
MID, CARRIERID, PRJOBID, CTLJOBID, SUBSTID, OBJSPEC, OBJTYPE,
MDLN, SOFTREV, EQPTYP, DATAID + every ACK code (COMMACK, ONLACK,
OFLACK, HCACK, CMDA, ACKC5-7-10, DRACK, LRACK, ERACK, EAC, TIACK,
GRANT, ALCD, OBJACK) + stream/function shorthand + HSMS terms +
T-timers + E84 signals + the standards lineup + codebase shortcuts
("the model", "the router", "the proof", etc.). Cuts week-1
onboarding time.
4. FAQ.md is new. Canonical answers to the questions that come up
once per integration: why HSMS unencrypted, SVID vs DVID, PJ vs
CJ, who fires FSM transitions, what runs on which thread, how to
add a new SECS-II message, ASCII vs Binary, common MES quirks,
how spool works, robustness fuzz vs libFuzzer, conformance vs
interop, what's not implemented.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
db90a21e1d |
verify: expand secs4j harness 20 → 55 checks
Every check the user could ask for now lands. secs4j's
comm.send(stream, function, w, body) takes arbitrary S/F + arbitrary
Secs2 body, so coverage was never coverage-limited by the Java side
— the original 20 was just the minimum to fill the gaps secsgem-py
couldn't reach.
Adds:
- Status data: S1F3, S1F11
- EC management: S2F13, S2F15 (set TimeFormat), S2F29
- Event reports: S2F33, S2F35, S2F37 (full define-link-enable
sequence), S6F15, S6F19, S6F21
- Remote control: S2F41 (modern RCMD=START + observed S6F11),
S2F21 (legacy RCMD=STOP),
S2F41 RCMD=FAULT + observed S5F1
- Alarms: S5F3, S5F5, S5F7
- Spool: S2F43, S6F23
- PP management: S7F1, S7F3, S7F5, S7F17, S7F19
- Terminal: S10F3 (single), S10F5 (multi-line)
- E40 PJ: S16F11 (full E40 body — MF + PRRECIPEMETHOD +
RecipeSpec + mtrloutspec + processparams),
S16F7 (monitor), S16F13 (dequeue)
- Limits: S2F45, S2F47
- Trace: S2F23 (5-field body)
- E39: S14F1 (GetAttr)
Plus a SecsMessageReceiveListener that captures every equipment-
initiated primary into a ConcurrentLinkedQueue and replies to S5F1
(ACKC5=0), S6F11 (ACKC6=0), S16F9 (W=0 no reply) so the
equipment's T3 doesn't fire on our watch. Two checks now assert
the unsolicited path:
- After RCMD=START, an S6F11 with the linked report must arrive
within 400ms
- After RCMD=FAULT, an S5F1 with the alarm must arrive within
400ms
Both observed against the demo equipment.
Result: 55/55 PASS. Two independent implementations
(secsgem-py + secs4java8) now corroborate the wire surface in
overlapping but distinct slices. Full E40 body — the one that
defeated secsgem-py's SFDL grammar — round-trips cleanly through
secs4j.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
4ddf8e0f48 |
verify: libFuzzer harness for secs2::decode + try_parse_sml
Coverage-guided structural search for crashes and undefined behaviour on arbitrary input to our two parsers. What's wired: - -DSECSGEM_FUZZ=ON CMake option, clang-only. Adds -fsanitize=fuzzer-no-link,address,undefined to all targets + -fsanitize=fuzzer to the two fuzz executables. - apps/fuzz_secs2_decode.cpp — feeds raw bytes to secs2::decode. Catches secs2::CodecError (expected) but traps on anything else leaking (would be a hardening bug). - apps/fuzz_sml_parse.cpp — feeds string to try_parse_sml, which is contractually nothrow-equivalent; traps on any exception. - .gitea/workflows/ci.yml — `libfuzzer` job builds with clang and runs each fuzzer for 60s in CI. Any crash / ASan / UBSan flag fails the job. - Dockerfile gains clang + libclang-rt-18-dev so devs can run locally with the same toolchain. Result on a fresh 30-second local run: fuzz_secs2_decode: 70 727 random inputs, 0 crashes fuzz_sml_parse: 284 950 random inputs, 0 crashes The coverage-guided search found and synthesized inputs that exercise: zero-byte, single-byte format tags, all length-byte counts (1/2/3), nested lists, format bytes with reserved bits, the "BOOLEAN" SML token, malformed quoted strings, etc. libFuzzer's recommended dictionary at the end of each run shows what bytes / substrings the coverage feedback discovered as discriminating — useful signals if we ever want a hand-curated corpus. README proof table grows to 8 commands. After this: - 426 unit tests (internal) - 47 conformance harness checks (internal) - 24 secsgem-py interop checks (external — Python ref impl) - 20 secs4j interop checks (external — independent Java impl) - 69 frames dissected by Wireshark HSMS dissector (external) - 196 SEMI E5 KAT assertions (standards body's encoding rules) - **~70k + ~285k random inputs, 0 crashes (external)** - 100k random tool ops with all invariants holding (internal) - YAML validation (internal) - TSan clean on 2 557 assertions (internal correctness aid) Five distinct external proofs now, each covering a different angle. Plan: VERIFICATION.md §4. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
2fce2fad0c |
verify: secs4j cross-validation (independent Java implementation)
20 cross-validation checks PASS against [secs4java8] (Apache 2.0,
kenta-shimizu) — an independent SECS/HSMS implementation in Java by
a different author from a different language ecosystem. Distinct
implementer = independent spec interpretation. Two libraries
agreeing on wire bytes is much stronger evidence of spec-correctness
than either alone.
Coverage targets the gap the secsgem-py interop deliberately skipped
(secsgem-py's SFDL grammar couldn't easily express GEM 300 bodies
with variable lists of named scalars):
- S1F1/F13/F17/F19/F21/F23 — establish comms + namelists
- S2F17 — clock
- S2F23 — trace init (5-field body)
- S2F49 — enhanced remote command (DATAID + OBJSPEC + RCMD + params)
- S3F17/F19/F25/F27 — full E87 carrier surface (action, slot map
verify, transfer with port pair, cancel)
- S5F13/F17 — exception recovery (EXID + EXRECVRA)
- S14F9/F11 — E94 CJ create with prjobids list, CJ delete
- S16F5/F27 — E40 PJ command, E94 CJ command
- S1F15 — offline cleanup
20/20 PASS against the demo equipment. Reply S/F matches the spec
for every transaction; specific ACK values vary by equipment state
(CarrierIDUnknown for an unknown carrier is just as valid as Accept
for a known one) so we assert on the wire shape, not the result.
Ship layout:
interop/secs4j/Dockerfile — eclipse-temurin:21-jdk + clone
+ build of secs4java8 → Export.jar
interop/secs4j/Secs4jHostHarness.java
— 20 round_trip assertions; uses
Secs2.list/uint4/ascii to build
full GEM 300 bodies; comm.send()
for arbitrary S/F pairs
interop/secs4j_validate.sh — orchestrator: builds image,
compiles harness, starts compose
server, runs Java container on
the secs network against it
.gitea/workflows/ci.yml — secs4j-interop job in CI
README.md — proof table grows to 7 commands
.gitignore — *.class
After this commit our proof chain has:
- SEMI E5 KAT (standards body's own arithmetic)
- tshark dissector (Wireshark's HSMS impl)
- secsgem-py interop (Python reference impl)
- **secs4j interop** (independent Java impl)
+ 426 unit tests, 47 conformance harness checks, 100k random ops,
YAML validation
Four independent external proofs, three of them on overlapping wire
surface from independent angles.
Plan: VERIFICATION.md §3.
[secs4java8]: https://github.com/kenta-shimizu/secs4java8
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
5baf3f4dc7 |
verify: tshark HSMS dissector validation (independent third codec)
Wireshark's built-in HSMS dissector — written by network-protocol authors who don't know us, didn't talk to us, and don't share implementation details with secsgem-py — is a third independent codec for our framing. If they parse our pcap without warnings, our HSMS framing is wire-correct independently of both our internal tests and the secsgem-py interop path. interop/tshark_validate.sh: - Boots secs_server on 127.0.0.1:5099 (away from the demo port) - Captures the loopback wire traffic with tcpdump - Runs secs_client through ~24 transactions plus Separate.req + TCP FIN - Parses the pcap with tshark -V using the HSMS dissector - Asserts: no "Malformed Packet", no "Dissector bug", at least one HSMS frame, expected tokens present (Select.req/rsp, Separate.req, Data message), reports histogram (count by control type + distinct S/F pairs) Result against the demo: 69 HSMS frames dissected, 49 distinct S/F pairs (S01F01..S16F28), all clean. Dockerfile gains tshark + tcpdump. .gitea/workflows/ci.yml gains a `tshark-dissector` job that runs this validator as part of every push to main. README proof table grows to 6 commands. VERIFICATION.md §1a documents a follow-up: round-trip the KAT fixtures through secsgem-py to corroborate that the format codes we used match an independent implementation. Strengthens the KAT proof from "internally consistent" to "confirmed by a second implementer who read the spec without talking to us." Plan: VERIFICATION.md §2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
a79973ed4c |
test: SEMI E5 known-answer tests for SECS-II encoding
Hex-string fixtures constructed directly from the SEMI E5 §9
format-byte encoding rules:
format_byte = (format_code << 2) | length_byte_count
length_byte_count ∈ {1, 2, 3}
Coverage:
- Every format code (L, B, BOOLEAN, A, J, C, U1-U8, I1-I8, F4, F8)
- Every length-byte-count variant (1, 2, 3 bytes — exercises the
255 → 256 → 65 536 transitions)
- Numeric edges: 0, ±1, MIN, MAX, ±Inf, NaN, -0.0, multi-element vectors
- Empty and single-element variants
- Nested lists
- A "format byte layout per format code" regression tripwire that
pins every code → byte mapping
19 test cases, 196 assertions. Every fixture round-trips
byte-identical against the codec.
Why this is the strongest single codec test: every other validator
(secsgem-py interop, conformance harness, in-house unit tests) is
one implementer's interpretation. KAT is the standard's own
arithmetic. If our encoder matches these canonical bytes and our
decoder reverses them to the same Item, our SECS-II layer is wire-
compatible with anything else that obeys E5 §9.
NaN / signed-zero / Inf use a bit-pattern compare (IEEE NaN != NaN
breaks the default Item == path) — decode the canonical, re-encode
the decoded, assert byte-identical.
The 3-byte-length fixture (ASCII 65 536 × 'X') generates a ~200 KB
expected-bytes string in the test — slow to write but trivial to
check and forces the 3-byte length-prefix path that 99 % of real
traffic doesn't exercise.
Plan: VERIFICATION.md §1.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
257a148d34 |
docs: VERIFICATION.md — external validation test plan
Honest accounting of what's currently external vs internal in the
five proofs:
- 4 of 5 proofs are us-testing-us (unit tests, conformance
harness, robustness fuzz, YAML validation)
- Only secsgem-py interop is external, and it covers ~15-20 %
of the claimed wire surface (skips most of GEM 300, HSMS-GS,
exception recovery, wafer maps, enhanced commands, every
wire-level edge case that isn't message-shaped)
Plan documents four additional external validators with goals,
methods, success criteria, scope limits, and effort estimates:
1. SEMI E5 known-answer tests — hex fixtures from the spec's
own encoding rules; the strongest single codec test
2. tshark/Wireshark HSMS dissector — independent third codec
parsing our pcap captures
3. secs4j cross-validation — Apache-2.0 Java implementation
by a different author; catches "we both got it wrong the
same way" relative to secsgem-py
4. libFuzzer over secs2::decode + secs2::from_sml — coverage-
guided structural search for crashes and UB
After all four: 5 external proofs (KAT + tshark + secsgem-py +
secs4j + libFuzzer), three of them on overlapping wire surface
from independent angles.
Plan also explicitly lists what these validators do NOT replace:
GEM RTS certification, per-MES interop sweeps, real-fab wire
trace corroboration. Those remain customer-side work.
Order of execution: KAT → tshark → secs4j → libFuzzer. KAT
first because it produces fixtures the others can reuse;
libFuzzer last because it benefits from the KAT corpus.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
e82f67ecad |
docs: README restructure + proof-of-feature-completeness section
The old README mixed intro, quickstart, architecture, and 10 sections
of production deployment over 419 lines, with significant overlap
with INTEGRATION.md. It claimed "implements every standard" without
making the claim concrete.
Restructured to ~250 lines with the proof front and center.
New top-of-README "Proof of feature-completeness" section: five
commands that, when they all exit zero on a fresh clone, prove the
COMPLIANCE.md claims. Each command verified end-to-end before
landing in this commit:
1. docker compose run --rm tests
→ 426 cases / 2557 assertions PASS
2. secs_conformance --host server --port 5000
→ 47 / 47 wire-level checks PASS
3. host_vs_cpp_server.py --host server
→ 24 secsgem-py interop checks PASS
4. SECSGEM_ROBUSTNESS_SOAK=1 secsgem_tests -tc='*soak*'
→ 100 000 random tool operations, all invariants hold
5. secs_server --validate-config <all four YAMLs>
→ 0 errors, 0 warnings across the shipped configs
Plus a per-standard test-coverage table mapping every claimed SEMI
standard (E5, E5 §13, E4, E37, E30, E40, E94, E42, E87, E90, E116,
E120/E39, E157, E84) to its test files and case count, summing to
426 to match the doctest totals. Counts verified by
`grep -c TEST_CASE` per file.
CI also runs the TSan lane (separate job in
.gitea/workflows/ci.yml); README documents it under Build details.
Content moved out of README into specialized docs (eliminates
duplication):
- Security configs → SECURITY.md (was 14-line bullet list; now a
365-line file with nftables, stunnel, minisign, SIEM schema)
- Persistence layout + monitoring + HA + deployment patterns +
upgrade discipline + fab-stack integration → INTEGRATION.md
- Performance envelope → BENCHMARKS.md
- MES interop punch list → MES_INTEROP.md
README now reads top-to-bottom: what this is → license → proof →
quickstart → doc map → architecture → adding capabilities →
production (1-line pointers to the deep docs) → build details →
interop.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
0df229905d |
docs: SECURITY.md with concrete configs
README §2 used to list security categories ("network isolation",
"TLS tunnel", "authentication", "audit logging", "YAML signing")
without configs. Customers deploying to a real fab can't act on
bullet points — they need files to drop in and paths to verify.
SECURITY.md replaces the bullets with:
- nftables ruleset locking the HSMS + Prometheus + SSH ports to
known source IPs (with the test command to lint before reload)
- Kubernetes NetworkPolicy equivalent for pod deployments
- stunnel.conf for equipment side (terminator) AND MES side
(initiator), with mTLS, TLS 1.3 minimum, and bind-127.0.0.1
pattern so the cleartext socket never sees the network
- minisign-based YAML config signing: keygen, sign-at-deploy,
systemd ExecStartPre verification. Refuses to start on bad sig.
- Audit logging JSON schema for SIEM ingest, with one-line example
per frame and the structured-dispatch wrapper to emit it
- SIEM alert thresholds: S9F rate, distinct source IPs, TLS
handshake failures, signature-verify failures, spool depth,
T-timer expiry counter
- Secrets handling: stunnel keys + minisign signing key custody
- Incident response capture protocol (tcpdump, journal snapshot,
no-restart-until-captured) + reporting-back format
Every section has a runnable example. Nothing here is invented
under pressure during an incident.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
943f3bbcd5 |
ci: ThreadSanitizer lane + fix use-after-free TSan flagged
Adds a -DSECSGEM_TSAN=ON CMake option that builds every target with -fsanitize=thread + debug symbols + -O1 + frame pointers. Wires a dedicated thread-sanitizer job into .gitea/workflows/ci.yml that builds and runs the full test suite under TSan with TSAN_OPTIONS=halt_on_error=1 (any flagged race fails the job, not just warns). Result against the full 426-case / 2557-assertion suite: 0 warnings, all green. That converts the existing test_thread_safety.cpp (which exercised the asio::post-onto-strand pattern) and test_concurrency (in-flight transaction interleaving) and test_robustness_fuzz (28 random action types × thousands of ticks) from "pattern smoke-tests" into actual race detection. The first TSan run caught a real bug in test_robustness_fuzz's act_exception_complete: it held a pointer to an ExceptionStore entry across fire_internal(RecoveryComplete), which deletes the entry. The subsequent state() read was a use-after-free. TSan flagged it 8 times (4 reads × 2 stack-frame variants). Fix is scoped lookup + re-check via has() after the mutation; matches the contract any reasonable caller would follow. The asio std_fenced_block atomic_thread_fence path generates TSan "not supported" warnings during compile — those are asio's, not ours, and don't affect runtime detection. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
ca3559ef57 |
test: randomized robustness fuzz (4 seeds × 2k ops + 100k soak)
tests / build-and-test (push) Successful in 2m9s
Property-based robustness test that drives long sequences of random tool operations against EquipmentDataModel and verifies invariants + persistence round-trip after every action. Replaces hand-written state-pinning tests with a generative approach that explores combinations no human author would think to write. Action menu (28 weighted actions covering the full standard surface): - PJ create / event / dequeue (E40) - CJ create / event / delete (E94) - Carrier create / id / slot (E87) - Substrate create / location / proc (E90) - Alarm set / clear / enable toggle (E5 §13) - SVID updates (E30 §6.13) - Define-report / link-event / enable (E30 §6.6) - Exception post / recover / complete (E5 §9, S5F9-F18) - Module event (E157) - EPT event (E116) - Spool enqueue / drain / force-toggle (E30 §6.22) Every action is "adjusted": it picks a verb at random, then checks state-machine legality before applying. A Pause is only fired on a Processing PJ; a Recover only on a Posted exception; pj_dequeue skips PJs bound to active CJs (mirrors E94's "can't dequeue CJ-bound PJ" rule the fuzz itself discovered when the first run flagged a CJ→missing-PJ reference). Invariants checked every 64 ticks: - Every tracked PJ exists in the store (size matches) - Every CJ's prjobids all exist in PJ store - No FSM in NoState sentinel - EPT bucket total monotonically non-decreasing - Defined reports' VIDs all exist - Substrate / carrier counts match enumeration Persistence round-trip every 500 ticks: - Fresh shadow EquipmentDataModel loads from the same journal dir - Diffs PJ + CJ states one-by-one + carrier/substrate/exception counts against the live model - Catches any "mutation didn't reach disk" or "replay didn't reconstruct state correctly" bugs Reproducibility: - Each TEST_CASE uses a fixed seed (0x1, 0xdeadbeef, 0xfeedface, 0xc0ffee — 8000 ops total in the fast suite) - World keeps a rolling 20-action trace, printed on invariant violation so the failing sequence can be pasted into a targeted regression test - SECSGEM_ROBUSTNESS_SOAK=1 enables a 100k-tick soak case (~3-5 minutes in Docker; not run by default) The very first run found a real edge case: act_pj_dequeue removed PJs that were bound to active CJs, leaving dangling refs. Fixed the fuzz to filter; the underlying behavior is intentional (store trusts the application to gate), but the fuzz now mirrors the correct E94 contract. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
b99d84f956 |
hsms-gs: worked integration example + INTEGRATION.md §7
The codebase has supported HSMS-GS since the original landing
(test_hsms_gs.cpp covers the wire-level Select.req-per-session
walk-list, the per-session Reject(EntityNotSelected) behaviour,
and session-routed data dispatch). But the documentation said
exactly one line about it ("Connection::add_session(device_id)
registers extra sessions on one TCP socket") and there was no
end-to-end test using the Server/Client API customers actually
build against.
INTEGRATION.md §7 is a new section showing the realistic pattern:
- Server-side: register the primary session via Server::Config,
then `add_session` for the second MES in the on_connection
callback. Per-session message handler + selected handler so
each MES gets its own router (or its own per-session data view
over a shared EquipmentDataModel).
- Active-mode: same `add_session` on the host-side Connection
for multi-tool fleet controllers.
- Equipment-initiated push: pick the session_id when sending
unsolicited primaries (S5F1, S6F11, S10F1).
- Pointer to the wire tests + the new integration test for
customers who want to see the failure modes.
tests/test_hsms_gs_integration.cpp drives two MES sessions
(device_id 1 + 2) through the Server/Client API end to end:
- Both sessions complete Select.req independently
- S1F1 sent on each session returns a distinct MDLN
("EQUIP-SESS-1" vs "EQUIP-SESS-2"), proving per-session
dispatch routes correctly
- Per-session router fires exactly once per session, no
cross-talk
Pre-existing §§8-10 in INTEGRATION.md got bumped to §§9-11 to
make room.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
e3765a5176 |
persistence: multi-version reads across every store
ProcessJobStore and SubstrateStore already implemented the loader-accepts-any-version-in-[1, kVersion] pattern. The other five stores (ControlJobStore, CarrierStore, LoadPortStore, ExceptionStore, SpoolStore) used strict `header[1] != kVersion` rejection, meaning a future kVersion bump there would silently nuke every persisted record on first replay. That's a footgun the test_persistence_upgrade test already flagged as a tripwire. This commit flips the strict checks to `< 1 || > kVersion`, mirroring PJ + Substrate. No format change (kVersion stays at 1 across the five stores), but: - Future v2 of any store now Just Works: add fields at the end of write_record_, bump kVersion to 2, gate the new reads behind `if (version >= 2)`. Old v1 records on disk continue to replay with the new fields defaulted. - Future versions beyond kVersion still get rejected (downgrade protection — older code can't try to decode trailers it doesn't understand). Comment blocks on each kVersion declaration now describe the upgrade discipline so the next contributor doesn't reinvent it. Test additions: - Positive test that v1 ControlJob records load on current code (will continue to pass when kVersion bumps to 2, proving v1 is still readable) - ExceptionStore rejects a v9 (future) record, matching CJ + Carrier - The existing tripwire tests get retitled from "rejects unknown version" to "rejects a future version" to reflect the new contract README §6 gets honest: every store is now multi-version-aware, not just PJ + Substrate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |