raphael/secs-gem
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0721db954241d0b6dd60bc2480e1eae2fb2579b6
secs-gem/COMPLIANCE.md
T
raphael 0721db9542
tests / build-and-test (push) Failing after 42s
Details
Close COMPLIANCE.md gap: spooling (E30 §6.22) 
Implements the largest functional gap from the compliance audit. The
equipment now queues events the host can't immediately receive (either
because there's no SELECTED session or because the demo's force-spool
flag is on) and transmits the queue on host request.

What's new

  include/secsgem/gem/store/spool.hpp
    SpoolStore: a deque queue with a configurable per-stream whitelist
    (so only streams 5+6 spool by default), a max_size cap with FIFO
    eviction on overflow, and a `force_spool` test flag.  Enqueue
    returns one of Queued / Dropped_NotSpoolable / Dropped_Full so the
    caller can fall back to live delivery when appropriate.  Drain
    pops the entire queue in FIFO order.  Two new ack enums:
    ResetSpoolAck (S2F44 RSPACK) and SpoolRequestAck (S6F24 RSDA), plus
    SpoolRequestCode (S6F23 RSDC, Transmit/Purge).

  data/messages.yaml + auto-regenerated messages.hpp
    S2F43 W   <L,n <B stream>>            Reset Spooling
    S2F44     <L,2 <B RSPACK> <L,a ...>>  Reset Spooling Ack
    S6F23 W   <B RSDC>                    Request Spooled Data
    S6F24     <B RSDA>                    Request Spooled Data Ack

  data/equipment.yaml
    `spool:` section: max_size + spoolable_streams list.  Two new host
    commands SPOOL_ON / SPOOL_OFF that flip the force-spool flag (these
    stand in for "host link down" in the demo without dropping TCP).

  include/secsgem/gem/store/host_commands.hpp
    Spec/Result gain an optional<bool> force_spool field.  S2F41
    dispatch returns the result, the server applies it after S2F42 is
    queued.

  src/config/loader.cpp
    Reads `spool:` from equipment.yaml; reads `force_spool` from each
    host_commands entry; populates SpoolStore + CommandSpec.

  apps/secs_server.cpp
    New `deliver_or_spool(msg, what)` helper.  emit_event and
    emit_alarm_set funnel through it: if force_spool is on (or there's
    no active session), msg.stream is checked against the spoolable
    list and the message is enqueued; otherwise it's sent live.
    Two new handlers:
      S2F43  parses the stream list, updates SpoolStore, replies S2F44
      S6F23  RSDC=Transmit drains and re-sends each as a fresh primary
             (posted on the executor so the S6F24 ack flushes first);
             RSDC=Purge clears the queue and acks.
    The S2F41 handler now also propagates result.force_spool into the
    SpoolStore.

  apps/secs_client.cpp
    Demo extended with 4 new steps after the FAULT branch:
      SPOOL_ON  -> S2F42 Accept
      START     -> S2F42 Accept; CEID 300 emission spooled (no live S6F11)
      SPOOL_OFF -> S2F42 Accept; queue still has the message
      S6F23(Transmit) -> S6F24 Accept; spooled S6F11 arrives next
    Then the existing S7F19/S7F5/S10F1/S1F15/Separate flow continues.

  tests/test_data_model.cpp
    Four new TEST_CASEs for SpoolStore (whitelist, FIFO eviction at
    max_size, drain ordering, force flag).

  tests/test_loader.cpp
    Confirms equipment.yaml's `spool:` section populates the store and
    `force_spool: true/false` flows through to dispatch results.

  COMPLIANCE.md
    Spooling moves from ⬜ to 🟡.  Adds S2F43/F44 + S6F23/F24 as ✅ in
    the message coverage matrix; calls out what's still missing
    (S6F25/F26 notification, automatic activation on HSMS NOT-SELECTED,
    persistent on-disk spool).

Verified

  - Tests: 73 cases / 383 assertions pass (+4 spool cases).
  - Demo (docker compose up server client) walks the full happy path
    and the spool path, observed in the server log as:
        spool: force_spool=true (depth=0)
        spool: S6F11 CEID=300 queued (depth=1)
        spool: force_spool=false (depth=1)
        S6F23 transmit: draining 1 messages
    and on the host side as the queued S6F11 arriving in the correct
    order after S6F24.

Known limitations (logged in COMPLIANCE.md)

  - Spool activation is manual via SPOOL_ON/OFF rather than
    automatically triggered by HSMS NOT-SELECTED.
  - No S6F25/F26 spooled-data-ready notification on re-SELECT.
  - In-memory only; an equipment restart loses queued events.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-05 22:06:55 +02:00

14 KiB
Raw Blame History

SECS/GEM Compliance Audit

An honest, per-capability accounting of which parts of SEMI E5 (SECS-II), E30 (GEM), and E37 (HSMS) this codebase implements, which parts it implements partially, and which parts it does not implement at all.

TL;DR. This is not a 100%-conformant GEM equipment. It implements every GEM Fundamental capability except formal Documentation (S1F19–F22) and the S9 error stream, plus most of the high-value GEM Additional capabilities (Dynamic Event Reporting, Alarms, EC, Clock, Remote Control, Process Programs, Terminal). Spooling, Limits Monitoring, Trace Data Collection, Multi-block, and Material Movement are intentionally not yet implemented. See the per-capability tables below for spec section refs.

Legend:

  • ✅ Full — message/feature implemented; round-tripped in tests.
  • 🟡 Partial — implemented in the demo path but with documented limitations.
  • ⬜ Not implemented — intentionally out of scope for this iteration.

1. E37 — HSMS transport

Item Status Spec ref Notes
TCP transport ✅ E37 §6 hsms::Connection (Asio).
4-byte length prefix + 10-byte header ✅ E37 §8.2 hsms::Frame::encode/decode.
Session ID, byte2, byte3, PType, SType, system-bytes ✅ E37 §8.3 hsms::Header.
Select.req / .rsp ✅ E37 §7.2 SType 1/2; SelectStatus enum (0–3).
Deselect.req / .rsp ✅ E37 §7.4 SType 3/4; DeselectStatus enum (0–2).
Linktest.req / .rsp ✅ E37 §7.5 SType 5/6; periodic interval configurable.
Separate.req ✅ E37 §7.6 SType 9; graceful close after flush.
Reject.req 🟡 E37 §7.7 Emitted on data-while-NOT-SELECTED with reason 4; we don't yet reject on PType/SType-not-supported.
Connection state machine NOT-CONNECTED → NOT-SELECTED → SELECTED ✅ E37 §6.3 Both Active and Passive modes.
T3 reply timeout ✅ E37 §10 per-transaction steady_timer.
T5 connect separation timeout ✅ E37 §10 Client::schedule_retry.
T6 control transaction timeout ✅ E37 §10 single slot (no concurrent control transactions).
T7 not-selected timeout (passive) ✅ E37 §10 armed on connect / on Deselect.req.
T8 intercharacter timeout ✅ E37 §10 bounds the payload read after length prefix.
Multi-host / multi-session ⬜ E37 §6 single SELECTED session at a time.
HSMS-SS vs HSMS-GS ⬜ E37 §11 implemented HSMS-SS only.

2. E5 — SECS-II encoding

Item Status Spec ref Notes
Format byte + 1/2/3 length bytes ✅ E5 §9 secs2::encode_into.
List (L) ✅ E5 §9.3 recursive.
ASCII (A) ✅ E5 §9.5
Binary (B) ✅ E5 §9.5
Boolean (BOOLEAN) ✅ E5 §9.5
U1, U2, U4, U8 (big-endian) ✅ E5 §9.5
I1, I2, I4, I8 (big-endian, two's complement) ✅ E5 §9.5
F4, F8 (IEEE 754 big-endian) ✅ E5 §9.5 bit-cast round-trip.
JIS-8, C2 (Unicode) ⬜ E5 §9.5 rarely used in modern fabs.
SML text rendering ✅ E5 Annex secs2::to_sml.

3. E30 — GEM Fundamental capabilities (§5.2)

Fundamental Capability Status Spec ref Messages Notes
State models ✅ E30 §6.2 — Control state machine (5 states) + HSMS comm state. Equipment-processing state is left to the equipment app.
Equipment Processing States ⬜ E30 §6.3 — Standard says equipment may model these; tool-specific. Plug-in point not yet exposed.
Host-Initiated S1F13/F14 scenario ✅ E30 §6.5 S1F13/F14
Event Notification ✅ E30 §6.6 S6F11/F12 Equipment-initiated, host-acknowledged.
On-Line Identification ✅ E30 §6.7 S1F1/F2 MDLN + SOFTREV.
Error Messages 🟡 E30 §6.9 S9F* HSMS Reject.req covers the transport-level case. No S9 stream (S9F1–F13) for "Unrecognized Device ID", "Unrecognized Stream Type", T3/T6/T7/T8 violations, etc.
Documentation ⬜ E30 §6.10 S1F19/F20 (GEM compliance), S1F21/F22 (data variable namelist) Not implemented.
Control (Operator-Initiated) ✅ E30 §6.2 — ControlStateMachine::operator_online/offline/local/remote.

4. E30 — GEM Additional capabilities (§5.3)

Additional Capability Status Spec ref Messages Notes
Establish Communications ✅ E30 §6.5 S1F13/F14 Both directions modeled; COMMACK enum.
Dynamic Event Report Configuration ✅ E30 §6.6 S2F33/F34, S2F35/F36, S2F37/F38 Full Define-Report / Link-Event / Enable-Event pipeline with all four ack enums.
Variable Data Collection ✅ E30 §6.11 — DVID table; values resolvable via vid_value.
Trace Data Collection ⬜ E30 §6.12 S2F23/F24, S6F1/F2 Not implemented.
Status Data Collection ✅ E30 §6.13 S1F3/F4, S1F11/F12
Alarm Management 🟡 E30 §6.14 S5F1/F2, S5F3/F4, S5F5/F6 F1–F6 implemented; S5F7/F8 list-enabled-alarms not implemented. ALCD bit-7 set/cleared, lower-7 category supported.
Remote Control ✅ E30 §6.15 S2F41/F42 Full HCACK 7-value enum + per-parameter CPACKs.
Equipment Constants ✅ E30 §6.16 S2F13/F14, S2F15/F16, S2F29/F30 EAC range validation against min/max is NOT performed — set always accepts a known ECID; this would reject out-of-range with EAC=4 in a conformant equipment.
Process Program Management 🟡 E30 §6.17 S7F3/F4, S7F5/F6, S7F19/F20 Unformatted PP send/request/list. No E42 enhanced PP, no S7F23/F24/F25/F26 PP verify, no PPID validation. PPBODY treated as raw bytes (Binary item).
Material Movement ⬜ E30 §6.18 S3F*, E40 Tied to E40 carrier handling; separate standard.
Equipment Terminal Services 🟡 E30 §6.19 S10F1/F2, S10F3/F4 Single-line only. S10F5/F6 multi-block, S10F7 broadcast not implemented.
Clock ✅ E30 §6.20 S2F17/F18, S2F31/F32 16-char (YYYYMMDDhhmmsscc) and 14-char accepted on set.
Limits Monitoring ⬜ E30 §6.21 S2F45/F46, S2F47/F48, S6F45/F46 Not implemented.
Spooling 🟡 E30 §6.22 S2F43/F44, S6F23/F24 Spoolable-streams config (S2F43/F44) + spool request transmit/purge (S6F23/F24) implemented; SpoolStore queues equipment-initiated primaries when the host is unreachable or force-spool is on, drains FIFO on S6F23 transmit, purges on S6F23 purge. Not implemented: S6F25/F26 spooled-data-ready notification, automatic spool activation tied to HSMS NOT-SELECTED, persistent (on-disk) spool.
Control ✅ E30 §6.2 — See Fundamental.

5. Message coverage matrix

Pair Direction Status Implemented in Tested
S1F1 / S1F2 H↔E ✅ messages.hpp ✅ round-trip
S1F3 / S1F4 H→E ✅ messages.hpp ✅ round-trip
S1F11 / S1F12 H→E ✅ messages.hpp ✅ round-trip
S1F13 / S1F14 H↔E ✅ messages.hpp ✅ in demo
S1F15 / S1F16 H→E ✅ messages.hpp ✅ in demo
S1F17 / S1F18 H→E ✅ messages.hpp ✅ in demo
S1F19 / S1F20 H→E ⬜ — —
S1F21 / S1F22 H→E ⬜ — —
S2F13 / S2F14 H→E ✅ messages.hpp ✅ in demo
S2F15 / S2F16 H→E ✅ messages.hpp ✅ round-trip
S2F17 / S2F18 H→E ✅ messages.hpp ✅ round-trip
S2F23 / S2F24 H→E ⬜ — —
S2F29 / S2F30 H→E ✅ messages.hpp ✅ in demo
S2F31 / S2F32 H→E ✅ messages.hpp ✅ in demo
S2F33 / S2F34 H→E ✅ messages.hpp ✅ round-trip
S2F35 / S2F36 H→E ✅ messages.hpp ✅ round-trip
S2F37 / S2F38 H→E ✅ messages.hpp ✅ round-trip
S2F41 / S2F42 H→E ✅ messages.hpp ✅ round-trip
S2F45–F48 H→E ⬜ — —
S5F1 / S5F2 E→H ✅ messages.hpp ✅ round-trip
S5F3 / S5F4 H→E ✅ messages.hpp ✅ round-trip
S5F5 / S5F6 H→E ✅ messages.hpp ✅ in demo
S5F7 / S5F8 H→E ⬜ — —
S6F1 / S6F2 E→H ⬜ — —
S6F5 / S6F6 H↔E ⬜ — multi-block
S6F7 / S6F8 H↔E ⬜ — multi-block
S6F11 / S6F12 E→H ✅ messages.hpp ✅ round-trip + demo
S6F15 / S6F16 H→E ⬜ — event report request
S2F43 / S2F44 H→E ✅ catalog ✅ round-trip + demo
S6F23 / S6F24 H→E ✅ catalog ✅ round-trip + demo
S6F25 / S6F26 spool ⬜ — spool-data-ready notification
S7F3 / S7F4 H→E ✅ messages.hpp ✅ round-trip
S7F5 / S7F6 H→E ✅ messages.hpp ✅ in demo
S7F19 / S7F20 H→E ✅ messages.hpp ✅ round-trip + demo
S7F23–F26 H↔E ⬜ — enhanced PP
S9F* E→H ⬜ — error stream
S10F1 / S10F2 H→E ✅ messages.hpp ✅ in demo
S10F3 / S10F4 E→H ✅ messages.hpp ✅ round-trip + demo
S10F5 / S10F6 H→E ⬜ — multi-line

6. Demo evidence

The two-container demo (docker compose up --no-deps server client) exercises this concrete sequence end-to-end:

  1. TCP connect → Select.req → Select.rsp(Ok) → SELECTED on both sides.
  2. S1F13/S1F14 Establish Comms.
  3. S1F17/S1F18 Request Online; control state transitions HostOffline → AttemptOnline → OnlineRemote.
  4. Server pushes S10F3 welcome → host acks with S10F4.
  5. S1F11/S1F12 SVID namelist discovery → S1F3/S1F4 SVID read.
  6. S2F29/S2F30 EC namelist → S2F13/S2F14 EC read.
  7. S2F17/S2F18 clock read.
  8. S2F33/S2F34 Define Report 1000 over the 3 SVIDs (DRACK=0).
  9. S2F35/S2F36 Link CEIDs 200 and 300 to Report 1000 (LRACK=0).
  10. S2F37/S2F38 Enable CEIDs 200, 300 (ERACK=0).
  11. S2F41/S2F42 host command START (HCACK=0) → server emits S6F11(CEID=300) carrying the linked Report 1000 → host acks S6F12.
  12. S5F5/S5F6 list alarm directory.
  13. S5F3/S5F4 enable alarm 1 (ACKC5=0).
  14. S2F41/S2F42 host command FAULT (HCACK=0) → server emits S5F1 (ALCD=0x84, set + cat 4) → host acks S5F2; server also emits S6F11(CEID=200).
  15. S7F19/S7F20 recipe list, S7F5/S7F6 fetch RECIPE-A.
  16. S10F1/S10F2 host → equipment terminal display.
  17. S1F15/S1F16 Request Offline; control state goes back to HostOffline. (CEID 100 ControlStateChanged emission is correctly suppressed because the host never enabled CEID 100 — this is the correct GEM behavior.)
  18. Separate.req → clean close on both sides.

Unit tests: 63 cases / 278 assertions pass (docker compose run --rm tests).

7. What it would take to claim "100% GEM-compliant"

The honest list, in priority order:

  1. Finish spooling: S6F25/F26 spooled-data-ready notification, plus automatic activation when HSMS goes NOT-SELECTED (and automatic notification on re-SELECT) so the host doesn't have to manually flip the test-only force_spool flag. Optional: persistent on-disk spool so equipment restarts don't lose queued events.
  2. Implement the S9 error stream (S9F1, F3, F5, F7, F9, F11, F13) for the documented transport/protocol error cases.
  3. Implement Documentation messages S1F19/F20 (GEM-compliance) and S1F21/F22 (data variable namelist) — needed for E30 conformance.
  4. Implement EC range validation in set_equipment_constant_value so out-of-range sets return EAC=4 instead of being silently accepted.
  5. Implement Limits Monitoring (S2F45–F48, S6F45/F46) if the target equipment publishes monitored variables.
  6. Implement Trace Data Collection (S2F23/F24, S6F1/F2).
  7. Implement S5F7/F8 list-enabled-alarms.
  8. Implement multi-block transfers (S6F5/F6, S6F7/F8).
  9. Implement equipment processing state model with operator hooks (E30 §6.3) — the abstract model is in E30 but the concrete states are equipment-specific.
  10. Run the implementation against a real conformance test generator (Layer 4 of the implementation plan) on a representative tool.

After all of the above, a GEM Reference Test specification (RTS) review would still be needed before any "GEM compliant" marketing claim could be honestly made.

Reference in New Issue View Git Blame Copy Permalink