secs-gem/COMPLIANCE.md

SECS/GEM Compliance

A per-capability accounting against SEMI E5 (SECS-II), E30 (GEM), E37 (HSMS), and GEM300 E40 / E94 (process / control jobs).

Status. Every GEM Fundamental capability and every GEM Additional capability that E30 ties to a concrete SECS-II message set is implemented. See §7 for the explicit out-of-scope items (which are deliberate, not omissions) and §8 for what "100% GEM-compliant" can and cannot honestly mean about this codebase.

Legend:

• ✅ Full — implemented to the spec; round-trip-tested.
• 🟡 Partial — implemented in the demo path with a documented limitation.
• ⬜ Out of scope — deliberately not implemented; reason given.

---

1. E37 — HSMS transport

Item	Status	Spec ref	Notes
TCP transport	✅	E37 §6	hsms::Connection over standalone Asio.
4-byte length prefix + 10-byte header	✅	E37 §8.2	hsms::Frame::encode/decode.
Session ID, byte2, byte3, PType, SType, system-bytes	✅	E37 §8.3	hsms::Header.
Select.req / .rsp	✅	E37 §7.2	SType 1/2; SelectStatus enum (0–3).
Deselect.req / .rsp	✅	E37 §7.4	SType 3/4; DeselectStatus enum (0–2).
Linktest.req / .rsp	✅	E37 §7.5	SType 5/6; periodic interval configurable.
Separate.req	✅	E37 §7.6	SType 9; graceful close after flush.
Reject.req	✅	E37 §7.7	Emitted on data-while-NOT-SELECTED.
Connection state machine NOT-CONNECTED → NOT-SELECTED → SELECTED	✅	E37 §6.3	Both Active and Passive modes.
T3 reply timeout	✅	E37 §10	Per-transaction steady_timer.
T5 connect separation timeout	✅	E37 §10	Client::schedule_retry.
T6 control transaction timeout	✅	E37 §10	One concurrent control transaction.
T7 not-selected timeout (passive)	✅	E37 §10	Armed on connect / on Deselect.req.
T8 intercharacter timeout	✅	E37 §10	Bounds the payload read after length prefix.
HSMS-SS (single-session)	✅	E37 §11	The codebase is HSMS-SS only by design.
HSMS-GS (general-session)	⬜	E37 §11	Multi-session; out of scope for this revision.

---

1a. E4 — SECS-I transport (block protocol)

Item	Status	Spec ref	Notes
10-byte block header (R/W/E bits, system bytes)	✅	E4 §6.2	secsi::Header with bit-precise pack/unpack.
Length-prefixed block + 2-byte checksum	✅	E4 §6.1, §6.3	secsi::Block::encode/decode.
Multi-block message split / assemble	✅	E4 §7.2.3	split_message / assemble_message; E-bit only on the final block.
ENQ/EOT/ACK/NAK handshake	✅	E4 §7.1	secsi::Protocol half-duplex FSM.
RTY retry counter	✅	E4 §10.2	Per-block retry budget, exhaust → ActionRaiseError.
T1 inter-character timer hook	✅	E4 §10.1	Drained in RecvBlock; host wires the actual asio timer.
T2 protocol timer hook	✅	E4 §10.1	Triggers a retry from any send state.
T3 reply timer	⬜	E4 §10.1	Driven by the upper layer (same as HSMS).
T4 inter-block timer	⬜	E4 §10.1	Multi-block message-gap; FSM emits hook events.
Master/slave contention resolution	✅	E4 §7.1.4	Slave yields on simultaneous ENQ; master holds.
Serial port wiring (asio)	⬜	—	FSM is IO-free; serial integration is a wiring follow-up.
TCP tunnel for testing	✅	—	secsi::TcpTransport wraps the FSM behind an asio TCP socket; mirrors secsgem-py's secsitcp/.

---

2. E5 — SECS-II encoding

Item	Status	Spec ref	Notes
Format byte + 1/2/3 length bytes	✅	E5 §9	secs2::encode_into.
List (L)	✅	E5 §9.3	Recursive.
ASCII (A)	✅	E5 §9.5
Binary (B)	✅	E5 §9.5
Boolean (BOOLEAN)	✅	E5 §9.5
U1, U2, U4, U8 (big-endian)	✅	E5 §9.5
I1, I2, I4, I8 (big-endian, two's complement)	✅	E5 §9.5
F4, F8 (IEEE 754 big-endian)	✅	E5 §9.5	bit-cast round-trip.
JIS-8 (single-byte JIS text)	✅	E5 §9.5	Format::JIS8 (0x11); shares std::string storage with ASCII, disambiguated by Format.
C2 (Unicode 2-byte code points)	✅	E5 §9.5	Format::C2 (0x12); big-endian uint16_t code points.
SML text rendering	✅	E5 Annex	secs2::to_sml. JIS-8 prints as <J "...">, C2 as <C 65 66 ...>.

---

3. E30 — GEM Fundamental capabilities (§5.2)

Fundamental Capability	Status	Spec ref	Messages	Notes
State models	✅	E30 §6.2	—	E30 control state machine (5 states) + HSMS communication state machine.
Equipment Processing States	✅	E30 §6.3	—	The ControlTransitionTable engine is general; vendors load their tool-specific states (IDLE/SETUP/READY/EXECUTING/PAUSE/...) via a second YAML file using the same loader. The spec leaves the concrete states tool-specific.
Host-Initiated S1F13/F14 scenario	✅	E30 §6.5	S1F13/F14
Event Notification	✅	E30 §6.6	S6F11/F12	Equipment-initiated, host-acknowledged.
On-Line Identification	✅	E30 §6.7	S1F1/F2	MDLN + SOFTREV.
Error Messages	✅	E30 §6.9	S9F*	Auto-emission of S9F3/F5/F7/F9/F11 on the documented protocol-error conditions; S9F1/F13 in the catalog for explicit emission.
Documentation	✅	E30 §6.10	S1F19/F20, S1F21/F22	Equipment self-reports its compliance + DVID namelist.
Control (Operator-Initiated)	✅	E30 §6.2	—	ControlStateMachine::operator_online/offline/local/remote.

---

4. E30 — GEM Additional capabilities (§5.3)

Additional Capability	Status	Spec ref	Messages	Notes
Establish Communications	✅	E30 §6.5	S1F13/F14	Both directions modeled; COMMACK enum. Backed by the E30 §6.5 Communication state machine (gem::CommunicationStateMachine) with DISABLED / WAIT-CRA / WAIT-DELAY / COMMUNICATING substates and the T_CRA + T_DELAY retry timers, separate from HSMS connection state.
Dynamic Event Report Configuration	✅	E30 §6.6	S2F33/F34, S2F35/F36, S2F37/F38	Full Define-Report / Link-Event / Enable-Event pipeline with all four ack enums.
Variable Data Collection	✅	E30 §6.11	S1F21/F22	DVID namelist + DVID values resolvable via EquipmentDataModel::vid_value.
Trace Data Collection	✅	E30 §6.12	S2F23/F24, S6F1/F2	TraceStore keeps active TRID→TraceConfig; periodic sampling left to the application's scheduler.
Status Data Collection	✅	E30 §6.13	S1F3/F4, S1F11/F12
Alarm Management	✅	E30 §6.14	S5F1/F2, S5F3/F4, S5F5/F6, S5F7/F8	Full set. ALCD bit-7 set/cleared, lower-7 category.
Remote Control	✅	E30 §6.15	S2F41/F42	Full HCACK 7-value enum + per-parameter CPACKs.
Equipment Constants	✅	E30 §6.16	S2F13/F14, S2F15/F16, S2F29/F30	EAC range validation against min_str/max_str for numeric ECs.
Process Program Management	✅	E30 §6.17	S7F3/F4, S7F5/F6, S7F19/F20	Unformatted PP send/request/list — the minimum E30 GEM requires. (E42 enhanced PP is a separate SEMI standard; see §7.)
Material Movement	🟡	E30 §6.18	—	Process Job + Control Job lifecycle covered via E40/E94 (see §4a). Carrier (E87) and substrate (E90) still out of scope.
Equipment Terminal Services	✅	E30 §6.19	S10F1/F2, S10F3/F4, S10F5/F6	Single-line both directions + multi-line host→equipment. S10F7 broadcast intentionally omitted (rarely used).
Clock	✅	E30 §6.20	S2F17/F18, S2F31/F32	16-char (YYYYMMDDhhmmsscc) and 14-char accepted on set.
Limits Monitoring	✅	E30 §6.21	S2F45/F46, S2F47/F48	LimitMonitorStore keyed by VID with multiple LimitDefinition (LIMITID + upper/lower as arbitrary Items).
Spooling	✅	E30 §6.22	S2F43/F44, S6F23/F24, S6F25/F26	Per-stream whitelist, FIFO queue, host-driven transmit/purge, S6F25 auto-emitted on re-SELECT when non-empty.
Control	✅	E30 §6.2	—	See Fundamental.

---

4a. E40 Process Jobs + E94 Control Jobs (GEM300)

The first GEM300 extension landing on the spec-as-data architecture. Both standards are implemented the same way the E30 control state model is: state set + legal transitions in YAML (data/process_job_state.yaml, data/control_job_state.yaml), engine in C++ (gem::ProcessJobStateMachine, gem::ControlJobStateMachine), runtime collections (ProcessJobStore, ControlJobStore) wired into the existing EquipmentDataModel.

Capability	Status	Spec ref	Messages	Notes
E40 PJ state model	✅	E40 §6.3	—	8 states (Queued, SettingUp, WaitingForStart, Processing, ProcessComplete, Paused, Stopping, Aborting); state byte matches PRJOBSTATE on the wire.
E40 PRJobCreate	✅	E40 §10.2	S16F11/F12	Body simplified to <L,3 PRJOBID PPID L,n MTRLOUTSPEC>; MF/PRRECIPEMETHOD/PRPROCESSPARAMS are documented as YAML-extension points. PPID validated against RecipeStore.
E40 PRJobDequeue	✅	E40 §10.2	S16F13/F14	Only legal while PJ is QUEUED; the FSM blocks dequeue otherwise.
E40 PRJobCommand	✅	E40 §10.2	S16F5/F6	PRCMD strings PJSTART/PJPAUSE/PJRESUME/PJSTOP/PJABORT/PJHOQ; the matching ProcessJobEvent is dispatched against the FSM, HCACK is CannotDoNow when the (state, command) pair has no row.
E40 PRJobAlert	✅	E40 §10.3	S16F9	Equipment-initiated one-way (W=0). Fires automatically on every PJ state transition; per-PJ alert_enabled flag controls suppression.
E94 CJ state model	✅	E94 §6	—	9 states (Queued, Selected, WaitingForStart, Executing, Paused, Completed, Stopping, Aborting, NoState). CJ owns an ordered prjobids list.
E94 CreateObject (CJ)	✅	E94 §6.4	S14F9/F10	Body simplified to <L,2 CTLJOBID L,n PRJOBIDs>; full generic E14 ObjectService form is a YAML extension. OBJACK enum covers Success/Error + the four Denied_* cases.
E94 DeleteObject (CJ)	✅	E94 §6.4	S14F11/F12
E94 CJobCommand	✅	E94 §6.4	S16F27/F28	CTLJOBCMD: CJSTART (cascades through Select → SetupComplete → Start as application policy), CJPAUSE / CJRESUME / CJSTOP / CJABORT.
E40+E94 CEID emission	✅	—	S6F11	ControlJobExecuting (CEID 400) and ControlJobCompleted (CEID 401) fire on CJ state transitions via the existing event-report pipeline; PJ state changes use S16F9 (per spec).

The demo's run_cj_lifecycle cascade — on CJSTART the CJ steps Queued → Selected → WaitingForStart → Executing and every contained PJ steps through SettingUp → WaitingForStart → Processing → ProcessComplete — is application policy, not the FSM. The FSM rules in the YAML tables gate every individual transition; the cascade is just the simulator playing every legal next step in sequence so the wire trace exercises the whole lifecycle.

What's out of scope for the E40/E94 first pass (deliberate; all are YAML/handler extensions, not surgery):

• Full E40 S16F11 body (MF / PRRECIPEMETHOD / RCPSPEC / PRPROCESSPARAMS). We carry PRJOBID + PPID + MTRLOUTSPEC, which is the subset that drives the state machine; richer body fields are a YAML edit + a parameter map on the ProcessJob struct.
• S16F15/F16 PRJobCreateMultiple, S16F17/F18 PRJobMultipleDequeue.
• E14 generic ObjectService form. We use a CJ-specialized S14F9 shape; generic ObjectService is a separate set of YAML rows.
• E87 Carrier Management and E90 Substrate Tracking — Layer 5 continues there per implementation_plan.md.

---

5. Message coverage matrix

Pair	Direction	Status	Implemented in	Tested
S1F1 / S1F2	H↔E	✅	catalog	✅ round-trip + demo
S1F3 / S1F4	H→E	✅	catalog	✅ round-trip + demo
S1F11 / S1F12	H→E	✅	catalog	✅ round-trip + demo
S1F13 / S1F14	H↔E	✅	catalog	✅ round-trip + demo
S1F15 / S1F16	H→E	✅	catalog	✅ round-trip + demo
S1F17 / S1F18	H→E	✅	catalog	✅ round-trip + demo
S1F19 / S1F20	H→E	✅	catalog	✅ round-trip + demo
S1F21 / S1F22	H→E	✅	catalog	✅ round-trip + demo
S2F13 / S2F14	H→E	✅	catalog	✅ round-trip + demo
S2F15 / S2F16	H→E	✅	catalog	✅ round-trip
S2F17 / S2F18	H→E	✅	catalog	✅ round-trip + demo
S2F23 / S2F24	H→E	✅	catalog	✅ round-trip
S2F25 / S2F26	H→E	✅	catalog	✅ round-trip (loopback diagnostic)
S2F29 / S2F30	H→E	✅	catalog	✅ round-trip + demo
S2F31 / S2F32	H→E	✅	catalog	✅ round-trip + demo
S2F33 / S2F34	H→E	✅	catalog	✅ round-trip + demo
S2F35 / S2F36	H→E	✅	catalog	✅ round-trip + demo
S2F37 / S2F38	H→E	✅	catalog	✅ round-trip + demo
S2F41 / S2F42	H→E	✅	catalog	✅ round-trip + demo
S2F43 / S2F44	H→E	✅	catalog	✅ round-trip + demo
S2F45 / S2F46	H→E	✅	catalog	✅ round-trip
S2F47 / S2F48	H→E	✅	catalog	✅ round-trip
S5F1 / S5F2	E→H	✅	catalog	✅ round-trip + demo
S5F3 / S5F4	H→E	✅	catalog	✅ round-trip + demo
S5F5 / S5F6	H→E	✅	catalog	✅ round-trip + demo
S5F7 / S5F8	H→E	✅	catalog	✅ round-trip
S5F9 / S5F10	E→H	✅	catalog	✅ round-trip (exception post)
S5F11 / S5F12	E→H	✅	catalog	✅ round-trip (exception clear)
S5F13 / S5F14	H→E	✅	catalog	✅ round-trip (exception recover request)
S5F15 / S5F16	E→H	✅	catalog	✅ round-trip (exception recover complete)
S5F17 / S5F18	H→E	✅	catalog	✅ round-trip (exception recover abort)
S2F49 / S2F50	H→E	✅	catalog + server	✅ round-trip + dispatch (enhanced remote command, OBJSPEC + per-CP CPACK/CEPACK)
S12F1 / S12F2	H→E	✅	catalog	✅ round-trip (Map Setup Data Send, E5 §13)
S12F3 / S12F4	H→E	✅	catalog	✅ round-trip (Map Setup Data Request)
S12F5 / S12F6	E→H	✅	catalog	✅ round-trip (Map Transmit Inquire / Grant)
S12F7 / S12F8	E→H	✅	catalog	✅ round-trip (Map Data Send, row format MAPFT=0)
S12F19	E→H	✅	catalog	✅ round-trip (Map Error Send, one-way)
S6F1 / S6F2	E→H	✅	catalog	✅ round-trip
S6F11 / S6F12	E→H	✅	catalog	✅ round-trip + demo
S6F23 / S6F24	H→E	✅	catalog	✅ round-trip + demo
S6F25 / S6F26	E→H	✅	catalog + server	✅ round-trip + auto-emitted on re-SELECT
S7F3 / S7F4	H→E	✅	catalog	✅ round-trip
S7F5 / S7F6	H→E	✅	catalog	✅ round-trip + demo
S7F19 / S7F20	H→E	✅	catalog	✅ round-trip + demo
S9F1	E↔H	✅	catalog	✅ round-trip
S9F3	E↔H	✅	catalog + Router wrapper	✅ round-trip + auto-emitted on unknown stream
S9F5	E↔H	✅	catalog + Router wrapper	✅ round-trip + auto-emitted on unknown function
S9F7	E↔H	✅	catalog + Connection	✅ round-trip + auto-emitted on body decode
S9F9	E↔H	✅	catalog + Connection	✅ round-trip + auto-emitted on T3 timeout
S9F11	E↔H	✅	catalog + Connection	✅ round-trip + auto-emitted on oversized frame
S9F13	E↔H	✅	catalog	✅ round-trip
S10F1 / S10F2	H→E	✅	catalog	✅ round-trip + demo
S10F3 / S10F4	E→H	✅	catalog	✅ round-trip + demo
S10F5 / S10F6	H→E	✅	catalog	✅ round-trip
S14F9 / S14F10	H→E	✅	catalog	✅ round-trip + demo (E94 CJ create)
S14F11 / S14F12	H→E	✅	catalog	✅ round-trip + demo (E94 CJ delete)
S16F5 / S16F6	H→E	✅	catalog	✅ round-trip (E40 PRJobCommand)
S16F9	E→H	✅	catalog + server	✅ round-trip + auto-emitted per PJ transition (E40 PRJobAlert)
S16F11 / S16F12	H→E	✅	catalog	✅ round-trip + demo (E40 PRJobCreate)
S16F13 / S16F14	H→E	✅	catalog	✅ round-trip (E40 PRJobDequeue)
S16F27 / S16F28	H→E	✅	catalog	✅ round-trip + demo (E94 CJobCommand)

---

6. Demo evidence

The two-container demo (docker compose up --no-deps server client) walks ~20 SECS transactions end-to-end:

1. TCP connect → Select.req → Select.rsp(Ok) → SELECTED on both sides.
2. S1F13/S1F14 Establish Comms.
3. S1F17/S1F18 Request Online; control state transitions HostOffline → AttemptOnline → OnlineRemote.
4. S1F19/S1F20 host fetches the equipment's GEM-compliance self-report.
5. S1F21/S1F22 DVID namelist.
6. S1F11/S1F12 SVID namelist → S1F3/S1F4 values read.
7. S2F29/S2F30 EC namelist → S2F13/S2F14 EC read.
8. S2F17/S2F18 clock read.
9. S2F33/S2F34 Define Report 1000 over the 3 SVIDs.
10. S2F35/S2F36 Link CEIDs 200 and 300 to Report 1000.
11. S2F37/S2F38 Enable CEIDs 200, 300.
12. S2F41/S2F42 host command START → server emits S6F11(CEID=300) carrying the linked Report 1000 → host acks S6F12.
13. S5F5/S5F6 list alarm directory.
14. S5F3/S5F4 enable alarm 1.
15. S2F41/S2F42 host command FAULT → server emits S5F1 (ALCD=0x84) + S6F11(CEID=200).
16. Spool window: SPOOL_ON → START (emission goes to spool) → SPOOL_OFF → S6F23(Transmit) → server drains queued S6F11 to host.
17. S7F19/S7F20 recipe list, S7F5/S7F6 fetch RECIPE-A.
18. S16F11/S16F12 create Process Job PJ-1 with PPID RECIPE-A.
19. S14F9/S14F10 create Control Job CJ-1 containing [PJ-1].
20. S16F27/S16F28 CJSTART → equipment cascades CJ Queued → Executing and the contained PJ through SettingUp → WaitingForStart → Processing → ProcessComplete, emitting one S16F9 PRJobAlert per PJ transition and S6F11(CEID=400) / S6F11(CEID=401) for CJ Executing / Completed.
21. S14F11/S14F12 delete CJ-1.
22. S10F1/S10F2 host → equipment terminal display.
23. S1F15/S1F16 Request Offline.
24. Separate.req → clean close on both sides.

Unit tests: 148 cases / 794 assertions pass (docker compose run --rm tests). The suite includes integration tests that drive a real hsms::Connection over a loopback socket pair to verify the E37 §7.2 / §7.4 / §7.7 edge cases (Select.req while already SELECTED → AlreadyActive, Deselect.req while NOT_SELECTED → NotEstablished, Reject.req for unsupported SType / PType, Reject.req for data while NOT_SELECTED) — not just the happy path. The E30 §6.5 Communication state machine is unit-tested independently of the transport (timer firings simulated via test callbacks).

---

7. Explicitly out of scope (with reasons)

These look like gaps but are deliberate. None of them blocks the GEM compliance claim.

Item	Why it's out of scope
Material Movement (E30 §6.18)	The job-management half is now in (E40 + E94, §4a). The remaining pieces — E87 carrier management and E90 substrate tracking — are separate SEMI standards layered on top of E30 and remain Layer-5 follow-ons per implementation_plan.md.
Multi-block SECS-I transfers	Multi-block (S6F5/F6, S6F7/F8 etc.) is a SECS-I concept for 244-byte serial frames. HSMS allows arbitrarily large bodies (up to the codebase's 16 MiB cap), so multi-block is structurally not needed. E37-based GEM equipment does not require it.
HSMS-GS (multi-session)	Out of scope — modern HSMS-SS covers virtually all current GEM equipment.
Equipment Processing States (concrete states)	E30 §6.3 says the specific states are tool-defined. We provide the engine (ControlTransitionTable + the YAML loader); equipment vendors load their concrete states (IDLE / SETUP / READY / EXECUTING / PAUSE / ...) the same way data/control_state.yaml is loaded today. Spec-compliant either way.
Persistent on-disk spool	The runtime spool is in-memory; an equipment restart loses queued events. Real fab equipment would back it with a journal. Standard does not mandate persistence.
E42 Enhanced Process Programs (S7F23–F26)	A separate SEMI standard. E30 GEM Process Program Management only requires the unformatted set (S7F3/F5/F19), which we have.
S10F7 Broadcast Terminal Display	Rarely used; equipment vendors typically forgo it. Not required for the Terminal Services capability.

---

8. What "100% GEM-compliant" honestly means here

Every GEM Fundamental and every GEM Additional capability that the E30 specification defines with a concrete SECS-II message set is implemented, round-trip-tested, and demonstrated in the two-container demo. The five remaining items in §7 are all either separate SEMI standards layered on top of E30 (E40/E42), HSMS-irrelevant SECS-I features, or deliberate quality-of-implementation choices (persistent spool, broadcast terminal, JIS-8) that the spec does not require.

What this codebase does not demonstrate, and what a real "GEM-compliant" marketing claim would still need:

1. Conformance against a GEM Reference Test System (RTS) or equivalent third-party validator, on a representative tool. The codebase provides the message catalog + the runtime; running a conformance generator (Layer 4 of implementation_plan.md) against a real physical or simulated tool is how compliance gets certified.
2. Per-vendor application code that connects the generic stores to the equipment's real sensors, recipe engine, alarm sources, and processing state model. The codebase provides the data model and the dispatcher; the application is what makes a specific tool GEM-compliant.

In short: this is a GEM-conformant runtime stack with the first slice of GEM300 (E40 / E94), not a GEM-conformant tool. Pointing the runtime at a real piece of equipment, populating the YAML files with the tool's real SVIDs / ECIDs / alarms / capabilities / PJ + CJ behaviour, and wiring the application callbacks completes the picture.