raphael/secs-gem
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
160 Commits 2 Branches 0 Tags
main
Commit Graph

6 Commits

Author SHA1 Message Date
raphael 9876dd9b5a feat(daemon): D10 carriers + E16 ops RPCs + stress test + virtual fab
tests / build-and-test (push) Successful in 2m59s
Details
tests / thread-sanitizer (push) Successful in 3m36s
Details
tests / tshark-dissector (push) Successful in 2m25s
Details
tests / secs4j-interop (push) Successful in 59s
Details
tests / python-interop (push) Successful in 3m20s
Details
tests / libfuzzer (push) Successful in 3m40s
Details 
Completes the daemon's GEM300 surface and adds two new test tiers.

D10 — E87 carriers: CarrierStore gains the HandlerSlot observer pattern
(add_id/slot_map/access_handler). The daemon's id-observer forwards host
S3F17 decisions onto the Subscribe stream as CarrierAction (PROCEED on a
Confirmed transition, CANCEL on CancelCarrier); ReportCarrier drives the
flow tool-side: WAITING creates the carrier + records the slot map,
IN_ACCESS/COMPLETE advance the access FSM (INVALID_OBJECT on unknown,
CANNOT_DO_NOW on an illegal transition).

E16 — operations RPCs: Describe (full name inventory: variables/events/
alarms/commands/constants + device header), FlushSpool (purge or drain),
SendTerminalMessage (S10F1 tool->host, honest CANNOT_DO_NOW when no host
and stream 10 isn't spoolable).

Stream responsiveness: Subscribe/WatchHealth poll at 100ms (was 500ms) so a
cancelled stream frees its sync-server worker thread promptly — this was
found by the new stress test, which hung under Subscribe churn at 500ms.

Tests:
- A randomized concurrent RPC stress case: 4 threads x 250 seeded ops
  (set/get/fire/alarm/control-state/describe + Subscribe churn), asserts no
  failed RPC and a still-responsive engine afterward; prints its seed; a
  strong TSan target.
- A virtual fab (interop/virtual_fab.py + the `fab` compose service /
  tools/spawn_fab.sh): N daemons, each with a secsgem-py host AND a
  secsgem_client tool, driven by seeded random traffic with end-to-end
  invariant checks (set/get round-trips, event->S6F11 and alarm->S5F1
  delivery, command->tool->completion). Verified green at N=3 (~150 ops/eq,
  all commands round-tripped, 0 violations). Wired into run_interop.sh
  (now 13 steps).

Also fixes the CI break from the previous commit: the Python-client lane's
test_values.py step lacked PYTHONPATH=clients/python (now step-level env).

Two bugs found and fixed while building this, both mine from this batch:
1. carrier test hung on a CancelCarrier of a still-NotConfirmed carrier — a
   self-transition the FSM doesn't signal, so the observer never fired and
   the stream Read blocked forever. Fixed to cancel a Confirmed carrier;
   the NotConfirmed edge is documented as a known E87 limitation.
2. the 500ms stream poll above.

Daemon suite 7 cases / 214 assertions; core 475 / 3097; virtual fab green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-26 15:05:13 +02:00
raphael 99bfa794fc fix(daemon): honour declared SECS-II formats + make service thread-safe 
Audit fixes for two real bugs in the gRPC service:

1. Format compliance: to_item() wrote F8/I8 regardless of the variable's
   declared wire format, so values contradicted the S1F11/S1F21 namelists
   (ChamberPressure is F4, WaferCounter U4; the interop trace showed <F8 2.5>
   on the wire). Conversion now targets the declared format — verified
   end-to-end: secsgem-py now receives <F4 2.5> in S6F11.

2. Thread safety: gRPC handler threads called resolve_variable/resolve_event,
   copying live store entries (including Item values) while the io thread
   mutates them. The service now snapshots the immutable name->id/format maps
   at construction (before run_async, per the documented ordering); all writes
   already post to the io thread. Remaining known narrow race (GetControlState
   enum read) documented in DAEMON_ROADMAP.

Also: drop a stale tools/run_interop.sh reference from docker-compose.yml.
Tests: daemon in-process 16/16 (new F4/U4 format assertions), core 459/459,
secsgem-py interop green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-06-10 18:35:53 +02:00
raphael 3d72e50b65 test(interop): daemon end-to-end vs secsgem-py reference host 
daemon_interop.py drives a running secs_gemd through BOTH faces at once: a
gRPC tool client and a secsgem-py active host. Proves the gRPC<->HSMS bridge
against a reference GEM implementation, not just in-process:

  - gRPC GetControlState agrees with the HSMS-driven control state
  - gRPC SetVariables(ChamberPressure=2.5) + FireEvent(ProcessStarted) makes
    the host receive S6F11 CEID 300 carrying 2.5 (value flowed gRPC -> engine
    -> HSMS -> host)
  - unknown variable/event names rejected at the gRPC edge

Mirrors the existing host_vs_cpp_server.py pattern. New 'gemd' compose service
(HSMS :5000 + gRPC :50051); interop image gains grpcio/grpcio-tools (proto
stubs generated at runtime, flat to avoid the secsgem package-name clash).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-10 18:23:47 +02:00
raphael a1dc7937d4 test: live persistent-spool restart end-to-end 
Adds a docker-compose service `server-spool` that runs secs_server
with --spool-dir pointed at a named volume.  Two-phase Python
harness (interop/spool_persistence_test.py):

  1. Enqueue phase: force-spool one S6F11(CEID=300) via the
     SPOOL_ON / START / SPOOL_OFF RCMD trio, then disconnect.
  2. Driver runs `docker compose restart server-spool` between
     the phases — the named volume preserves the journal files.
  3. Drain phase: reconnect, send S6F23(Transmit), verify the
     replayed S6F11 carries CEID 300.

Surfaces a real interop bug along the way: secsgem-py 0.3.0 encodes
RSDC (and other "single-byte status" fields) as <U1>, while SEMI E5
spells them as <B>.  Our `as_binary_first` was strict on Binary; now
accepts either (the byte semantics are identical, and the leniency is
symmetric with the U-type widening from the first interop commit).

Result: enqueue → docker restart → drain returns CEID 300 cleanly.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-09 01:04:49 +02:00
raphael 2d60571a9c interop: secsgem-py cross-validation harness + lenient identifier parsing 
Adds a Docker-based interop harness that drives the C++ server with
secsgem-py 0.3.0 as the active host and probes a secsgem-py-passive
equipment from a minimal C++ active client.  Surfaces and fixes four
interoperability bugs uncovered by cross-testing:

  * SEMI E5 identifier formatcodes are a U1|U2|U4|U8 wildcard;
    secsgem-py picks the narrowest fitting width while our parsers
    only accepted U4.  `as_uN_scalar` / `as_iN_scalar` now accept
    any unsigned/signed width and range-check the downcast.
  * PPBODY (S7F3/F6) is "ASCII | Binary | List" per the spec;
    secsgem-py defaults to ASCII.  Added BINARY_OR_ASCII codegen
    item type with `as_text_or_binary` accessor.
  * S1F23/F24 Collection Event Namelist was unimplemented; added
    schema + `vids_for(ceid)` accessor on EventReportSubscriptions
    plus the dispatch handler.
  * S10F1 was registered as a host->equipment handler, but per
    SEMI E5 §12 S10F1 is equipment->host; S10F3 is the actual
    host->equipment Terminal Display Single.  Added an S10F3
    handler alongside (we keep S10F1 too for backward compat).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-08 23:17:18 +02:00
raphael 96b02f8b50 Initial commit: C++20 SECS-II / HSMS / GEM client + server 
A fully containerised SECS/GEM toolchain. Single docker compose project,
no host build tools. 63 unit-test cases / 278 assertions, two demo
executables, end-to-end two-container demo exercising every implemented
capability.

Architecture (bottom-up):

  secs2/   E5 SECS-II codec
    Item        variant over L/A/B/BOOLEAN/I1-8/U1-8/F4/F8
    encode/decode  big-endian, 1/2/3-byte length encoding
    Message     SxFy + W-bit + optional root item
    to_sml      human-readable text rendering

  hsms/    E37 HSMS transport (TCP)
    Header      10-byte header + SType enum (Data/Select/Deselect/
                Linktest/Reject/Separate)
    Frame       4-byte length prefix + payload encode/decode
    Connection  async Asio TCP, NOT-SELECTED -> SELECTED state machine,
                T3/T5/T6/T7/T8 timers, system-bytes reply correlation,
                graceful close-after-flush separation

  endpoint  active Client (connect with T5 retry) and passive Server
            (accept loop) wrappers over Connection

  gem/     E30 GEM logic
    ControlStateMachine  5-state E30 control model with operator
                         actions, host requests, SEMI-mandated ack
                         codes (OnlineAck, OfflineAck, CommAck), and
                         a state-change handler
    EquipmentDataModel   in-memory dictionary: SVIDs, DVIDs, ECIDs
                         (with EAC), CEIDs, report defs, CEID->report
                         links, enabled-events set, alarm table
                         (ALCD, enabled, active), process programs,
                         host command registry, clock (16-char
                         YYYYMMDDhhmmsscc with offset)
    messages.hpp         builders + parsers for every SxFy below

GEM message coverage (full list):

  S1F1/F2    Are You There / On Line Data
  S1F3/F4    Selected Equipment Status Request / Data
  S1F11/F12  Status Variable Namelist Request / Data
  S1F13/F14  Establish Communications (+ CommAck)
  S1F15/F16  Request OFFLINE (+ OfflineAck)
  S1F17/F18  Request ONLINE (+ OnlineAck)
  S2F13/F14  Equipment Constant Request / Data
  S2F15/F16  EC Send + EquipmentAck (Accept/UnknownEcid/Busy/OutOfRange)
  S2F17/F18  Date and Time Request / Data
  S2F29/F30  Equipment Constant Namelist Request / Data
  S2F31/F32  Date and Time Set Request / TimeAck
  S2F33/F34  Define Report + DefineReportAck (5 enum values)
  S2F35/F36  Link Event Report + LinkEventAck
  S2F37/F38  Enable / Disable Event Report + EnableEventAck
  S2F41/F42  Host Command + HostCmdAck (7 values) + per-param CPACKs
  S5F1/F2    Alarm Report Send + AlarmAck (ALCD bit-7 set/cleared
             + lower-7 category)
  S5F3/F4    Enable/Disable Alarm Send + AlarmAck
  S5F5/F6    List Alarms Request / Data (active alarms tagged in ALCD)
  S6F11/F12  Event Report Send (equipment-initiated CEID emission
             with full report data) + EventReportAck
  S7F3/F4    Process Program Send + ProcessProgramAck (7 values)
  S7F5/F6    Process Program Request / Data
  S7F19/F20  Current EPPD List Request / Data
  S10F1/F2   Terminal Display Single (host->equipment) + TerminalAck
  S10F3/F4   Terminal Display Single (equipment->host)

Demo apps:

  apps/secs_server.cpp   passive equipment. Populates the data model
                         with 3 SVIDs (ControlState, Clock,
                         EventsEnabled), 2 ECIDs, 3 CEIDs
                         (ControlStateChanged, AlarmSetEvent,
                         ProcessStarted), 2 alarms (Chiller Temp High
                         cat 4, Door Open cat 1), 2 recipes
                         (RECIPE-A, RECIPE-B), and 4 host commands
                         (START, STOP, PAUSE, FAULT). Emits S6F11 on
                         every control state transition + on START;
                         emits S5F1 + the AlarmSetEvent CEID on FAULT.
                         Pushes an S10F3 welcome message when the host
                         comes online.

  apps/secs_client.cpp   active host. Walks 17 steps: Establish ->
                         Online -> S1F11 SVID namelist -> S1F3 read ->
                         S2F29 EC namelist -> S2F13 read ->
                         S2F17 clock -> S2F33/S2F35/S2F37 dynamic
                         event subscription -> S2F41 START
                         (-> receives S6F11) -> S5F5 alarm list ->
                         S5F3 enable alarm 1 -> S2F41 FAULT
                         (-> receives S5F1 + S6F11) -> S7F19/S7F5
                         recipe list + body -> S10F1 terminal ->
                         S1F15 Offline -> Separate. Handles inbound
                         S6F11, S5F1, S10F3 primaries.

Testing:

  tests/test_secs2.cpp         codec round-trip for every format,
                               byte-layout assertions for known values,
                               truncation/trailing-byte rejection,
                               nested list round-trip, SML rendering
  tests/test_hsms.cpp          header byte layout, data + control
                               header round-trip, full frame round-
                               trip with length prefix, short-payload
                               rejection
  tests/test_control_state.cpp every (state, event) pair in the E30
                               control state machine, including
                               AlreadyOnline / NotAccept rejections
                               and idempotent offline-while-offline
  tests/test_data_model.cpp    SVID/ECID/Alarm/Recipe CRUD, clock
                               format + parse, host command registry,
                               full event-report pipeline (define ->
                               link -> enable -> compose) with
                               every error path (InvalidVid,
                               UnknownCeid, UnknownRptid), alarm
                               set/clear with ALCD bit-7 semantics
  tests/test_messages.cpp      round-trip + byte-layout for every
                               builder/parser pair, including S6F11
                               event reports with mixed item types

Toolchain:

  Dockerfile          Ubuntu 24.04, g++-13, CMake, Ninja, libasio-dev
  docker-compose.yml  builder / tests / server / client services,
                      source bind-mounted, build artifacts in a
                      named volume so the host tree stays clean
  CMakeLists.txt      C++20, -Wall -Wextra -Wpedantic, standalone
                      Asio (ASIO_STANDALONE), doctest via FetchContent

Documentation:

  README.md           architecture, quick start, demo log
  COMPLIANCE.md       honest per-capability E5/E30/E37 audit with
                      spec section refs. Calls out what's implemented,
                      what's partial (Reject.req, Alarms missing F7/F8,
                      EC range validation, PP without verify, terminal
                      single-line only), and what's intentionally not
                      yet implemented (spooling, S9 error stream,
                      Documentation S1F19/F20+F21/F22, limits monitoring,
                      trace data collection, multi-block, material
                      movement). Does NOT claim "100% GEM-compliant" and
                      lists the work required to honestly make that claim.

This is Layer 0 + the start of Layer 1 from implementation_plan.md.
The transition-table-driven "spec-as-data" architecture (Layer 1
proper) is not yet implemented; the current code uses imperative
state machines that are structurally ready to be refactored onto
tables.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-02 00:21:10 +02:00