raphael
0721db9542
tests / build-and-test (push) Failing after 42s
Implements the largest functional gap from the compliance audit. The
equipment now queues events the host can't immediately receive (either
because there's no SELECTED session or because the demo's force-spool
flag is on) and transmits the queue on host request.
What's new
include/secsgem/gem/store/spool.hpp
SpoolStore: a deque queue with a configurable per-stream whitelist
(so only streams 5+6 spool by default), a max_size cap with FIFO
eviction on overflow, and a `force_spool` test flag. Enqueue
returns one of Queued / Dropped_NotSpoolable / Dropped_Full so the
caller can fall back to live delivery when appropriate. Drain
pops the entire queue in FIFO order. Two new ack enums:
ResetSpoolAck (S2F44 RSPACK) and SpoolRequestAck (S6F24 RSDA), plus
SpoolRequestCode (S6F23 RSDC, Transmit/Purge).
data/messages.yaml + auto-regenerated messages.hpp
S2F43 W <L,n <B stream>> Reset Spooling
S2F44 <L,2 <B RSPACK> <L,a ...>> Reset Spooling Ack
S6F23 W <B RSDC> Request Spooled Data
S6F24 <B RSDA> Request Spooled Data Ack
data/equipment.yaml
`spool:` section: max_size + spoolable_streams list. Two new host
commands SPOOL_ON / SPOOL_OFF that flip the force-spool flag (these
stand in for "host link down" in the demo without dropping TCP).
include/secsgem/gem/store/host_commands.hpp
Spec/Result gain an optional<bool> force_spool field. S2F41
dispatch returns the result, the server applies it after S2F42 is
queued.
src/config/loader.cpp
Reads `spool:` from equipment.yaml; reads `force_spool` from each
host_commands entry; populates SpoolStore + CommandSpec.
apps/secs_server.cpp
New `deliver_or_spool(msg, what)` helper. emit_event and
emit_alarm_set funnel through it: if force_spool is on (or there's
no active session), msg.stream is checked against the spoolable
list and the message is enqueued; otherwise it's sent live.
Two new handlers:
S2F43 parses the stream list, updates SpoolStore, replies S2F44
S6F23 RSDC=Transmit drains and re-sends each as a fresh primary
(posted on the executor so the S6F24 ack flushes first);
RSDC=Purge clears the queue and acks.
The S2F41 handler now also propagates result.force_spool into the
SpoolStore.
apps/secs_client.cpp
Demo extended with 4 new steps after the FAULT branch:
SPOOL_ON -> S2F42 Accept
START -> S2F42 Accept; CEID 300 emission spooled (no live S6F11)
SPOOL_OFF -> S2F42 Accept; queue still has the message
S6F23(Transmit) -> S6F24 Accept; spooled S6F11 arrives next
Then the existing S7F19/S7F5/S10F1/S1F15/Separate flow continues.
tests/test_data_model.cpp
Four new TEST_CASEs for SpoolStore (whitelist, FIFO eviction at
max_size, drain ordering, force flag).
tests/test_loader.cpp
Confirms equipment.yaml's `spool:` section populates the store and
`force_spool: true/false` flows through to dispatch results.
COMPLIANCE.md
Spooling moves from ⬜ to 🟡. Adds S2F43/F44 + S6F23/F24 as ✅ in
the message coverage matrix; calls out what's still missing
(S6F25/F26 notification, automatic activation on HSMS NOT-SELECTED,
persistent on-disk spool).
Verified
- Tests: 73 cases / 383 assertions pass (+4 spool cases).
- Demo (docker compose up server client) walks the full happy path
and the spool path, observed in the server log as:
spool: force_spool=true (depth=0)
spool: S6F11 CEID=300 queued (depth=1)
spool: force_spool=false (depth=1)
S6F23 transmit: draining 1 messages
and on the host side as the queued S6F11 arriving in the correct
order after S6F24.
Known limitations (logged in COMPLIANCE.md)
- Spool activation is manual via SPOOL_ON/OFF rather than
automatically triggered by HSMS NOT-SELECTED.
- No S6F25/F26 spooled-data-ready notification on re-SELECT.
- In-memory only; an equipment restart loses queued events.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
210 lines
14 KiB
Markdown
210 lines
14 KiB
Markdown
# SECS/GEM Compliance Audit
|
||
|
||
An honest, per-capability accounting of which parts of SEMI **E5 (SECS-II)**,
|
||
**E30 (GEM)**, and **E37 (HSMS)** this codebase implements, which parts it
|
||
implements partially, and which parts it does not implement at all.
|
||
|
||
> **TL;DR.** This is *not* a 100%-conformant GEM equipment. It implements
|
||
> every **GEM Fundamental capability** except formal Documentation (S1F19–F22)
|
||
> and the S9 error stream, plus most of the high-value **GEM Additional
|
||
> capabilities** (Dynamic Event Reporting, Alarms, EC, Clock, Remote Control,
|
||
> Process Programs, Terminal). Spooling, Limits Monitoring, Trace Data
|
||
> Collection, Multi-block, and Material Movement are intentionally not yet
|
||
> implemented. See the per-capability tables below for spec section refs.
|
||
|
||
Legend:
|
||
|
||
- ✅ **Full** — message/feature implemented; round-tripped in tests.
|
||
- 🟡 **Partial** — implemented in the demo path but with documented limitations.
|
||
- ⬜ **Not implemented** — intentionally out of scope for this iteration.
|
||
|
||
---
|
||
|
||
## 1. E37 — HSMS transport
|
||
|
||
| Item | Status | Spec ref | Notes |
|
||
|---------------------------------------|--------|----------|-------|
|
||
| TCP transport | ✅ | E37 §6 | `hsms::Connection` (Asio). |
|
||
| 4-byte length prefix + 10-byte header | ✅ | E37 §8.2 | `hsms::Frame::encode/decode`. |
|
||
| Session ID, byte2, byte3, PType, SType, system-bytes | ✅ | E37 §8.3 | `hsms::Header`. |
|
||
| `Select.req / .rsp` | ✅ | E37 §7.2 | `SType` 1/2; SelectStatus enum (0–3). |
|
||
| `Deselect.req / .rsp` | ✅ | E37 §7.4 | `SType` 3/4; DeselectStatus enum (0–2). |
|
||
| `Linktest.req / .rsp` | ✅ | E37 §7.5 | `SType` 5/6; periodic interval configurable. |
|
||
| `Separate.req` | ✅ | E37 §7.6 | `SType` 9; graceful close after flush. |
|
||
| `Reject.req` | 🟡 | E37 §7.7 | Emitted on data-while-NOT-SELECTED with reason 4; we don't yet reject on PType/SType-not-supported. |
|
||
| Connection state machine NOT-CONNECTED → NOT-SELECTED → SELECTED | ✅ | E37 §6.3 | Both Active and Passive modes. |
|
||
| T3 reply timeout | ✅ | E37 §10 | per-transaction `steady_timer`. |
|
||
| T5 connect separation timeout | ✅ | E37 §10 | `Client::schedule_retry`. |
|
||
| T6 control transaction timeout | ✅ | E37 §10 | single slot (no concurrent control transactions). |
|
||
| T7 not-selected timeout (passive) | ✅ | E37 §10 | armed on connect / on Deselect.req. |
|
||
| T8 intercharacter timeout | ✅ | E37 §10 | bounds the payload read after length prefix. |
|
||
| Multi-host / multi-session | ⬜ | E37 §6 | single SELECTED session at a time. |
|
||
| HSMS-SS vs HSMS-GS | ⬜ | E37 §11 | implemented HSMS-SS only. |
|
||
|
||
---
|
||
|
||
## 2. E5 — SECS-II encoding
|
||
|
||
| Item | Status | Spec ref | Notes |
|
||
|---------------------------------------|--------|----------|-------|
|
||
| Format byte + 1/2/3 length bytes | ✅ | E5 §9 | `secs2::encode_into`. |
|
||
| List (`L`) | ✅ | E5 §9.3 | recursive. |
|
||
| ASCII (`A`) | ✅ | E5 §9.5 | |
|
||
| Binary (`B`) | ✅ | E5 §9.5 | |
|
||
| Boolean (`BOOLEAN`) | ✅ | E5 §9.5 | |
|
||
| `U1, U2, U4, U8` (big-endian) | ✅ | E5 §9.5 | |
|
||
| `I1, I2, I4, I8` (big-endian, two's complement) | ✅ | E5 §9.5 | |
|
||
| `F4, F8` (IEEE 754 big-endian) | ✅ | E5 §9.5 | bit-cast round-trip. |
|
||
| JIS-8, C2 (Unicode) | ⬜ | E5 §9.5 | rarely used in modern fabs. |
|
||
| SML text rendering | ✅ | E5 Annex | `secs2::to_sml`. |
|
||
|
||
---
|
||
|
||
## 3. E30 — GEM Fundamental capabilities (§5.2)
|
||
|
||
| Fundamental Capability | Status | Spec ref | Messages | Notes |
|
||
|---------------------------------------|--------|----------|----------|-------|
|
||
| State models | ✅ | E30 §6.2 | — | Control state machine (5 states) + HSMS comm state. Equipment-processing state is left to the equipment app. |
|
||
| Equipment Processing States | ⬜ | E30 §6.3 | — | Standard says equipment **may** model these; tool-specific. Plug-in point not yet exposed. |
|
||
| Host-Initiated S1F13/F14 scenario | ✅ | E30 §6.5 | S1F13/F14 | |
|
||
| Event Notification | ✅ | E30 §6.6 | S6F11/F12 | Equipment-initiated, host-acknowledged. |
|
||
| On-Line Identification | ✅ | E30 §6.7 | S1F1/F2 | MDLN + SOFTREV. |
|
||
| Error Messages | 🟡 | E30 §6.9 | S9F* | HSMS Reject.req covers the transport-level case. **No S9 stream (S9F1–F13)** for "Unrecognized Device ID", "Unrecognized Stream Type", T3/T6/T7/T8 violations, etc. |
|
||
| Documentation | ⬜ | E30 §6.10| S1F19/F20 (GEM compliance), S1F21/F22 (data variable namelist) | Not implemented. |
|
||
| Control (Operator-Initiated) | ✅ | E30 §6.2 | — | `ControlStateMachine::operator_online/offline/local/remote`. |
|
||
|
||
---
|
||
|
||
## 4. E30 — GEM Additional capabilities (§5.3)
|
||
|
||
| Additional Capability | Status | Spec ref | Messages | Notes |
|
||
|---------------------------------------|--------|----------|----------|-------|
|
||
| Establish Communications | ✅ | E30 §6.5 | S1F13/F14 | Both directions modeled; COMMACK enum. |
|
||
| Dynamic Event Report Configuration | ✅ | E30 §6.6 | S2F33/F34, S2F35/F36, S2F37/F38 | Full Define-Report / Link-Event / Enable-Event pipeline with all four ack enums. |
|
||
| Variable Data Collection | ✅ | E30 §6.11| — | `DVID` table; values resolvable via `vid_value`. |
|
||
| Trace Data Collection | ⬜ | E30 §6.12| S2F23/F24, S6F1/F2 | Not implemented. |
|
||
| Status Data Collection | ✅ | E30 §6.13| S1F3/F4, S1F11/F12 | |
|
||
| Alarm Management | 🟡 | E30 §6.14| S5F1/F2, S5F3/F4, S5F5/F6 | F1–F6 implemented; **S5F7/F8 list-enabled-alarms not implemented**. ALCD bit-7 set/cleared, lower-7 category supported. |
|
||
| Remote Control | ✅ | E30 §6.15| S2F41/F42 | Full HCACK 7-value enum + per-parameter CPACKs. |
|
||
| Equipment Constants | ✅ | E30 §6.16| S2F13/F14, S2F15/F16, S2F29/F30 | **EAC range validation against min/max is NOT performed** — set always accepts a known ECID; this would reject out-of-range with EAC=4 in a conformant equipment. |
|
||
| Process Program Management | 🟡 | E30 §6.17| S7F3/F4, S7F5/F6, S7F19/F20 | Unformatted PP send/request/list. **No E42 enhanced PP**, **no S7F23/F24/F25/F26 PP verify**, **no PPID validation**. PPBODY treated as raw bytes (Binary item). |
|
||
| Material Movement | ⬜ | E30 §6.18| S3F*, E40 | Tied to E40 carrier handling; separate standard. |
|
||
| Equipment Terminal Services | 🟡 | E30 §6.19| S10F1/F2, S10F3/F4 | Single-line only. **S10F5/F6 multi-block, S10F7 broadcast not implemented.** |
|
||
| Clock | ✅ | E30 §6.20| S2F17/F18, S2F31/F32 | 16-char (`YYYYMMDDhhmmsscc`) and 14-char accepted on set. |
|
||
| Limits Monitoring | ⬜ | E30 §6.21| S2F45/F46, S2F47/F48, S6F45/F46 | Not implemented. |
|
||
| Spooling | 🟡 | E30 §6.22| S2F43/F44, S6F23/F24 | Spoolable-streams config (S2F43/F44) + spool request transmit/purge (S6F23/F24) implemented; SpoolStore queues equipment-initiated primaries when the host is unreachable or force-spool is on, drains FIFO on S6F23 transmit, purges on S6F23 purge. **Not implemented**: S6F25/F26 spooled-data-ready notification, automatic spool activation tied to HSMS NOT-SELECTED, persistent (on-disk) spool. |
|
||
| Control | ✅ | E30 §6.2 | — | See Fundamental. |
|
||
|
||
---
|
||
|
||
## 5. Message coverage matrix
|
||
|
||
| Pair | Direction | Status | Implemented in | Tested |
|
||
|------------------|-----------|--------|----------------|--------|
|
||
| S1F1 / S1F2 | H↔E | ✅ | `messages.hpp` | ✅ round-trip |
|
||
| S1F3 / S1F4 | H→E | ✅ | `messages.hpp` | ✅ round-trip |
|
||
| S1F11 / S1F12 | H→E | ✅ | `messages.hpp` | ✅ round-trip |
|
||
| S1F13 / S1F14 | H↔E | ✅ | `messages.hpp` | ✅ in demo |
|
||
| S1F15 / S1F16 | H→E | ✅ | `messages.hpp` | ✅ in demo |
|
||
| S1F17 / S1F18 | H→E | ✅ | `messages.hpp` | ✅ in demo |
|
||
| S1F19 / S1F20 | H→E | ⬜ | — | — |
|
||
| S1F21 / S1F22 | H→E | ⬜ | — | — |
|
||
| S2F13 / S2F14 | H→E | ✅ | `messages.hpp` | ✅ in demo |
|
||
| S2F15 / S2F16 | H→E | ✅ | `messages.hpp` | ✅ round-trip |
|
||
| S2F17 / S2F18 | H→E | ✅ | `messages.hpp` | ✅ round-trip |
|
||
| S2F23 / S2F24 | H→E | ⬜ | — | — |
|
||
| S2F29 / S2F30 | H→E | ✅ | `messages.hpp` | ✅ in demo |
|
||
| S2F31 / S2F32 | H→E | ✅ | `messages.hpp` | ✅ in demo |
|
||
| S2F33 / S2F34 | H→E | ✅ | `messages.hpp` | ✅ round-trip |
|
||
| S2F35 / S2F36 | H→E | ✅ | `messages.hpp` | ✅ round-trip |
|
||
| S2F37 / S2F38 | H→E | ✅ | `messages.hpp` | ✅ round-trip |
|
||
| S2F41 / S2F42 | H→E | ✅ | `messages.hpp` | ✅ round-trip |
|
||
| S2F45–F48 | H→E | ⬜ | — | — |
|
||
| S5F1 / S5F2 | E→H | ✅ | `messages.hpp` | ✅ round-trip |
|
||
| S5F3 / S5F4 | H→E | ✅ | `messages.hpp` | ✅ round-trip |
|
||
| S5F5 / S5F6 | H→E | ✅ | `messages.hpp` | ✅ in demo |
|
||
| S5F7 / S5F8 | H→E | ⬜ | — | — |
|
||
| S6F1 / S6F2 | E→H | ⬜ | — | — |
|
||
| S6F5 / S6F6 | H↔E | ⬜ | — | multi-block |
|
||
| S6F7 / S6F8 | H↔E | ⬜ | — | multi-block |
|
||
| S6F11 / S6F12 | E→H | ✅ | `messages.hpp` | ✅ round-trip + demo |
|
||
| S6F15 / S6F16 | H→E | ⬜ | — | event report request |
|
||
| S2F43 / S2F44 | H→E | ✅ | catalog | ✅ round-trip + demo |
|
||
| S6F23 / S6F24 | H→E | ✅ | catalog | ✅ round-trip + demo |
|
||
| S6F25 / S6F26 | spool | ⬜ | — | spool-data-ready notification |
|
||
| S7F3 / S7F4 | H→E | ✅ | `messages.hpp` | ✅ round-trip |
|
||
| S7F5 / S7F6 | H→E | ✅ | `messages.hpp` | ✅ in demo |
|
||
| S7F19 / S7F20 | H→E | ✅ | `messages.hpp` | ✅ round-trip + demo |
|
||
| S7F23–F26 | H↔E | ⬜ | — | enhanced PP |
|
||
| S9F* | E→H | ⬜ | — | error stream |
|
||
| S10F1 / S10F2 | H→E | ✅ | `messages.hpp` | ✅ in demo |
|
||
| S10F3 / S10F4 | E→H | ✅ | `messages.hpp` | ✅ round-trip + demo |
|
||
| S10F5 / S10F6 | H→E | ⬜ | — | multi-line |
|
||
|
||
---
|
||
|
||
## 6. Demo evidence
|
||
|
||
The two-container demo (`docker compose up --no-deps server client`)
|
||
exercises this concrete sequence end-to-end:
|
||
|
||
1. TCP connect → `Select.req` → `Select.rsp(Ok)` → SELECTED on both sides.
|
||
2. `S1F13`/`S1F14` Establish Comms.
|
||
3. `S1F17`/`S1F18` Request Online; control state transitions
|
||
`HostOffline → AttemptOnline → OnlineRemote`.
|
||
4. Server pushes `S10F3` welcome → host acks with `S10F4`.
|
||
5. `S1F11`/`S1F12` SVID namelist discovery → `S1F3`/`S1F4` SVID read.
|
||
6. `S2F29`/`S2F30` EC namelist → `S2F13`/`S2F14` EC read.
|
||
7. `S2F17`/`S2F18` clock read.
|
||
8. `S2F33`/`S2F34` Define Report 1000 over the 3 SVIDs (DRACK=0).
|
||
9. `S2F35`/`S2F36` Link CEIDs 200 and 300 to Report 1000 (LRACK=0).
|
||
10. `S2F37`/`S2F38` Enable CEIDs 200, 300 (ERACK=0).
|
||
11. `S2F41`/`S2F42` host command **START** (HCACK=0) → server emits
|
||
`S6F11(CEID=300)` carrying the linked Report 1000 → host acks `S6F12`.
|
||
12. `S5F5`/`S5F6` list alarm directory.
|
||
13. `S5F3`/`S5F4` enable alarm 1 (ACKC5=0).
|
||
14. `S2F41`/`S2F42` host command **FAULT** (HCACK=0) → server emits
|
||
`S5F1` (ALCD=0x84, set + cat 4) → host acks `S5F2`; server also
|
||
emits `S6F11(CEID=200)`.
|
||
15. `S7F19`/`S7F20` recipe list, `S7F5`/`S7F6` fetch RECIPE-A.
|
||
16. `S10F1`/`S10F2` host → equipment terminal display.
|
||
17. `S1F15`/`S1F16` Request Offline; control state goes back to
|
||
HostOffline. (CEID 100 ControlStateChanged emission is correctly
|
||
*suppressed* because the host never enabled CEID 100 — this is the
|
||
correct GEM behavior.)
|
||
18. `Separate.req` → clean close on both sides.
|
||
|
||
Unit tests: **63 cases / 278 assertions pass** (`docker compose run --rm tests`).
|
||
|
||
---
|
||
|
||
## 7. What it would take to claim "100% GEM-compliant"
|
||
|
||
The honest list, in priority order:
|
||
|
||
1. **Finish spooling**: S6F25/F26 spooled-data-ready notification, plus
|
||
automatic activation when HSMS goes NOT-SELECTED (and automatic
|
||
notification on re-SELECT) so the host doesn't have to manually flip
|
||
the test-only `force_spool` flag. Optional: persistent on-disk spool
|
||
so equipment restarts don't lose queued events.
|
||
2. **Implement the S9 error stream** (S9F1, F3, F5, F7, F9, F11, F13)
|
||
for the documented transport/protocol error cases.
|
||
3. **Implement Documentation messages** S1F19/F20 (GEM-compliance) and
|
||
S1F21/F22 (data variable namelist) — needed for E30 conformance.
|
||
4. **Implement EC range validation** in `set_equipment_constant_value`
|
||
so out-of-range sets return EAC=4 instead of being silently accepted.
|
||
5. **Implement Limits Monitoring** (S2F45–F48, S6F45/F46) if the
|
||
target equipment publishes monitored variables.
|
||
6. **Implement Trace Data Collection** (S2F23/F24, S6F1/F2).
|
||
7. **Implement S5F7/F8** list-enabled-alarms.
|
||
8. **Implement multi-block transfers** (S6F5/F6, S6F7/F8).
|
||
9. **Implement equipment processing state model** with operator hooks
|
||
(E30 §6.3) — the abstract model is in E30 but the concrete states
|
||
are equipment-specific.
|
||
10. **Run the implementation against a real conformance test generator**
|
||
(Layer 4 of the implementation plan) on a representative tool.
|
||
|
||
After all of the above, a GEM Reference Test specification (RTS) review
|
||
would still be needed before any "GEM compliant" marketing claim could
|
||
be honestly made.
|