docs: chapters 02 + 03 of the guided tour (Part 1 complete)
02 — The cast of characters: equipment, EAP, MES, fab planner, AMHS, operator. Who initiates which conversation, why the equipment is the passive side of HSMS by convention, how the AMHS handshake is out-of-band relative to SECS. Cross-references the relevant namespace and test files for each actor. 03 — Vocabulary + a wafer's journey: follows one 300 mm wafer end-to-end through a fab and labels every SECS message and acronym that fires. Introduces SVID / DVID / ECID / CEID / RPTID / ALID / PPID / MDLN / SOFTREV / HCACK / ALCD / OFLACK / CAACK / SMACK / etc. in context rather than as a list. Includes one-screen reference tables for the remaining acknowledge codes, T-timers in all four contexts (HSMS / SECS-I / E84 / E30 communication state), and a stream-by-stream summary. Part 1 (Foundations) of the guided tour is now complete — a reader who reads chapters 01–03 can describe the protocol stack, identify the actors, and recognise every acronym they'll meet in Part 2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,417 @@
|
||||
# 02 — The cast of characters
|
||||
|
||||
← [01 What is SECS/GEM?](01_what_is_secs_gem.md) | [Back to index](00_index.md) | Next: [03 Vocabulary + a wafer's journey](03_vocabulary_and_a_wafers_journey.md) →
|
||||
|
||||
Chapter [01](01_what_is_secs_gem.md) explained that SECS/GEM is the
|
||||
protocol a fab uses to make ~100 tools talk to a central MES. That
|
||||
description hid a lot of structure. In reality, there are at least
|
||||
**six distinct actors** in a typical fab automation stack — six
|
||||
roles, each implemented by different software (often by different
|
||||
vendors), each with its own concerns.
|
||||
|
||||
This chapter introduces them all, draws who-talks-to-whom, and
|
||||
locates each one in this codebase. After this you'll be able to read
|
||||
any SECS conversation and know which actor is initiating, which is
|
||||
responding, and why.
|
||||
|
||||
---
|
||||
|
||||
## The six actors
|
||||
|
||||
```
|
||||
┌──────────────────┐
|
||||
│ Fab planner │ "make 100 wafers
|
||||
│ (MES upper) │ of recipe R by
|
||||
└────────┬─────────┘ Friday"
|
||||
│
|
||||
recipes, lot │ yields, KPIs,
|
||||
assignments, │ alarms, status
|
||||
process programs │
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ MES (the host) │ per-step orchestration
|
||||
└────────┬─────────┘
|
||||
│
|
||||
SECS/GEM │ SECS/GEM
|
||||
S2F41 RCMD, │ S6F11 events,
|
||||
S7F3 PP send, │ S5F1 alarms,
|
||||
S2F33 reports, │ S1F4 status
|
||||
… │ …
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ EAP / equipment automation program │
|
||||
│ (vendor application layer) │
|
||||
├──────────────────────────────────────────┤
|
||||
│ THIS CODEBASE — the SECS/GEM runtime │
|
||||
│ secsgem::gem / secsgem::hsms / … │
|
||||
└────────┬─────────────────────────────────┘
|
||||
│
|
||||
PLC / sensor / recipe-engine APIs (tool-specific)
|
||||
│
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ Equipment │ the physical tool:
|
||||
│ (the tool) │ chambers, robots,
|
||||
└────────┬─────────┘ sensors, recipes
|
||||
│
|
||||
E84 8-line │ (carrier moves, no SECS bytes)
|
||||
parallel I/O │
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ AMHS │ robot rails / OHT
|
||||
│ (the carriers) │ that move FOUPs
|
||||
└──────────────────┘
|
||||
|
||||
←───── Operator ──────→ panel buttons,
|
||||
recipe overrides,
|
||||
Online/Offline/Local/Remote
|
||||
```
|
||||
|
||||
Read the diagram top-down: a fab planner schedules work, the MES
|
||||
dispatches it tool by tool, each tool's EAP receives commands over
|
||||
SECS/GEM, the EAP drives the actual hardware, the AMHS robots feed
|
||||
carriers in and out. An operator can intervene at any layer.
|
||||
|
||||
Each actor has a section below.
|
||||
|
||||
---
|
||||
|
||||
## 1. Equipment — the tool itself
|
||||
|
||||
**What it is.** A physical processing tool — a chemical-vapor
|
||||
deposition (CVD) chamber, a plasma etcher, a wafer prober, a
|
||||
photolithography stepper, an ion implanter, an inspection
|
||||
microscope. Anywhere from one chamber the size of a microwave to a
|
||||
full lithography cluster the size of a small bus.
|
||||
|
||||
**What it does in SECS/GEM terms.**
|
||||
|
||||
- **Reports state** — its current control state (Equipment Offline,
|
||||
Online Remote, …), its processing state (IDLE, EXECUTING, …), its
|
||||
carrier slots, its current recipe.
|
||||
- **Emits events** — when something happens worth recording
|
||||
(processing started, wafer processed, alarm raised, recipe
|
||||
completed), it fires an `S6F11` to the host.
|
||||
- **Accepts commands** — START, STOP, ABORT, PAUSE, CHANGE-RECIPE,
|
||||
CARRIER-PROCEED, etc., delivered as `S2F41` Host Commands.
|
||||
- **Stores its data dictionary** — every Status Variable (SVID),
|
||||
Equipment Constant (ECID), Data Variable (DVID), Collection Event
|
||||
(CEID), Alarm (ALID), and Process Program (PPID) it supports.
|
||||
- **Manages its own physical safety** — it can refuse a host command
|
||||
if the requested action would damage hardware, and it can raise
|
||||
alarms autonomously.
|
||||
|
||||
**In SECS/GEM, the equipment is almost always the "passive" side of
|
||||
the connection** — it binds a TCP port and waits for the host to
|
||||
connect, rather than the other way around. This codebase reflects
|
||||
that: `apps/secs_server.cpp` is the equipment, and it listens.
|
||||
|
||||
**Where it lives in this codebase.**
|
||||
|
||||
- The equipment role's main binary: [`apps/secs_server.cpp`](../apps/secs_server.cpp).
|
||||
- The data dictionary: [`include/secsgem/gem/data_model.hpp`](../include/secsgem/gem/data_model.hpp)
|
||||
defines `EquipmentDataModel`, which composes every per-domain
|
||||
store (SVIDs, ECIDs, CEIDs, alarms, carriers, substrates, recipes,
|
||||
spool, …).
|
||||
- A worked example with sensor simulation, recipe runner, and alarm
|
||||
monitoring: [`examples/pvd_tool/main.cpp`](../examples/pvd_tool/main.cpp).
|
||||
- Tests covering equipment behaviour: [`tests/test_data_model.cpp`](../tests/test_data_model.cpp),
|
||||
[`tests/test_control_state.cpp`](../tests/test_control_state.cpp),
|
||||
[`tests/test_host_handler.cpp`](../tests/test_host_handler.cpp).
|
||||
|
||||
---
|
||||
|
||||
## 2. EAP — the equipment automation program
|
||||
|
||||
**What it is.** The vendor-written software layer that sits on top
|
||||
of the SECS/GEM runtime and *makes the tool actually do things*.
|
||||
The EAP is the glue between:
|
||||
|
||||
- The SECS/GEM library (this codebase),
|
||||
- The tool's PLCs / sensors / recipe engine / robot controllers,
|
||||
- The tool vendor's domain logic.
|
||||
|
||||
Every tool vendor ships their own EAP. Two CVD tools from different
|
||||
vendors both speak GEM, but their EAPs are entirely different
|
||||
codebases doing entirely different things internally.
|
||||
|
||||
**Why it's a separate role.** The SECS/GEM standards spell out
|
||||
*what* messages mean — "S2F41 with RCMD=START must initiate
|
||||
processing on the currently loaded recipe." They don't spell out
|
||||
*how* a specific CVD tool initiates processing on its specific
|
||||
hardware. The EAP is the layer that resolves that.
|
||||
|
||||
In particular:
|
||||
|
||||
- When `S2F41 RCMD=START` arrives, the EAP decides whether the tool
|
||||
is in a state to start (chamber pressure low enough? robot at
|
||||
home position? recipe loaded?), and if so, calls the tool's
|
||||
proprietary recipe engine to begin the cycle.
|
||||
- When a sensor reads a temperature change, the EAP decides whether
|
||||
to update an SVID, fire a CEID, or raise an alarm — and the
|
||||
per-tool rules for that aren't in any SEMI standard.
|
||||
- When a `S7F3` arrives with a new recipe payload, the EAP decides
|
||||
how to validate the recipe against the tool's actual hardware
|
||||
capabilities.
|
||||
|
||||
**Where it lives in this codebase.**
|
||||
|
||||
This codebase provides the SECS/GEM runtime; the EAP is what a
|
||||
customer writes on top of it. We ship two reference EAPs:
|
||||
|
||||
- [`apps/secs_server.cpp`](../apps/secs_server.cpp) — the demo
|
||||
server. Wires every Router handler the demo flow needs; uses
|
||||
static YAML data and doesn't simulate any sensors. Useful as a
|
||||
starting fork.
|
||||
- [`examples/pvd_tool/main.cpp`](../examples/pvd_tool/main.cpp) — a
|
||||
fictional PVD tool that adds a sensor simulator, a recipe runner,
|
||||
an alarm threshold monitor, EPT state cycling, and Prometheus
|
||||
metrics. This is the closest thing to "what a real EAP looks
|
||||
like" that we ship. See [`examples/pvd_tool/README.md`](../examples/pvd_tool/README.md)
|
||||
for the section-by-section walk.
|
||||
|
||||
The integration tutorial — how to *write* an EAP for a real tool —
|
||||
is [`INTEGRATION.md`](INTEGRATION.md). Chapter
|
||||
[41](41_integration_hardware_mes_production.md) in this series covers
|
||||
the same material with cross-references back to the standards.
|
||||
|
||||
---
|
||||
|
||||
## 3. MES — the host
|
||||
|
||||
**What it is.** The **Manufacturing Execution System**. A
|
||||
fab-wide server (or cluster) that orchestrates production across
|
||||
every tool, manages lots and recipes, collects yield and statistical
|
||||
process control (SPC) data, and provides the operator UI for the
|
||||
production floor.
|
||||
|
||||
Commercial MES vendors you'll meet: Applied Materials **E3**, Camstar
|
||||
**InSite**, Wonderware **MES**, Aegis **FactoryWorks**, Inficon
|
||||
**FabGuard**, Critical Manufacturing **MES**, and many in-house
|
||||
custom builds especially at the largest fabs.
|
||||
|
||||
**What it does in SECS/GEM terms.**
|
||||
|
||||
- **Connects** to each tool's equipment process. In SECS/GEM
|
||||
language, the MES is the **active** side of the HSMS connection
|
||||
(it initiates the TCP connect and sends `Select.req`).
|
||||
- **Establishes communications** — sends `S1F13` to which the
|
||||
equipment replies `S1F14(COMMACK=Accept)`.
|
||||
- **Identifies the tool** — sends `S1F1` (Are You There), reads
|
||||
back the `MDLN` (model name) and `SOFTREV` (software revision)
|
||||
in `S1F2`.
|
||||
- **Reads the data dictionary** — `S1F11` for the SVID namelist,
|
||||
`S2F29` for the ECID namelist, `S1F23` for the CEID namelist,
|
||||
`S5F5` for the alarm directory, `S7F19` for the recipe list.
|
||||
- **Configures event reports** — `S2F33` defines a report,
|
||||
`S2F35` links it to a Collection Event, `S2F37` enables it. This
|
||||
is how the MES tells the tool "when CEID 300 fires, send me the
|
||||
values of SVIDs 1 and 2 along with it."
|
||||
- **Issues remote commands** — `S2F41 RCMD=START`, `S2F41
|
||||
RCMD=PAUSE`, `S2F41 RCMD=ABORT`, etc.
|
||||
- **Manages recipes** — `S7F3` to send a recipe, `S7F19` to list,
|
||||
`S7F17` to delete, `S7F5` to read one back.
|
||||
- **Orchestrates process and control jobs** — `S16F11` to create
|
||||
a Process Job, `S14F9` to wrap it in a Control Job, `S16F27`
|
||||
CJSTART to begin execution.
|
||||
- **Receives alarms and events** — `S5F1` for alarm set/clear,
|
||||
`S6F11` for collection events. Acknowledges with `S5F2` and
|
||||
`S6F12` respectively.
|
||||
- **Sets and reads the equipment's clock** — `S2F17`/`S2F18` to
|
||||
read, `S2F31`/`S2F32` to set.
|
||||
|
||||
**Where it lives in this codebase.**
|
||||
|
||||
We don't *implement* an MES — that's a separate, much larger product
|
||||
category. We implement the host *side* of SECS/GEM so the codebase
|
||||
can drive equipment too, mainly for testing.
|
||||
|
||||
- [`apps/secs_client.cpp`](../apps/secs_client.cpp) — the active
|
||||
host that drives the demo server through ~24 transactions.
|
||||
- [`apps/secs_conformance.cpp`](../apps/secs_conformance.cpp) — the
|
||||
host-driven conformance harness that runs the 47 wire-level checks.
|
||||
- [`include/secsgem/gem/host_handler.hpp`](../include/secsgem/gem/host_handler.hpp)
|
||||
+ [`src/gem/host_handler.cpp`](../src/gem/host_handler.cpp) —
|
||||
symmetric handler module so the host side can decode equipment
|
||||
replies and act on equipment-initiated S5F1 / S6F11.
|
||||
- [`interop/host_vs_cpp_server.py`](../interop/host_vs_cpp_server.py)
|
||||
— the secsgem-py active host driving our C++ passive server.
|
||||
|
||||
For integrating against a **commercial** MES,
|
||||
[`MES_INTEROP.md`](MES_INTEROP.md) is the day-1 punch list.
|
||||
|
||||
---
|
||||
|
||||
## 4. Fab planner / MES upper layer
|
||||
|
||||
**What it is.** The layer *above* the MES. Goes by many names:
|
||||
**Advanced Planning and Scheduling (APS)**, **Fab scheduler**,
|
||||
**Dispatcher**, **MES upper**. Big fabs separate this from the
|
||||
operational MES; smaller ones bundle it in.
|
||||
|
||||
**What it does.** Decides which lot runs on which tool, in what
|
||||
order, against what recipe, by what deadline. This is fab-wide
|
||||
optimisation across hundreds of in-flight lots and dozens of routes.
|
||||
|
||||
**SECS/GEM contact:** none directly. The planner talks to the MES
|
||||
via REST / SQL / a message queue / a proprietary API. The MES
|
||||
translates planner decisions into SECS commands.
|
||||
|
||||
**Where it lives in this codebase.** Not implemented; out of scope.
|
||||
Mentioned here so the reader knows where the recipes and lot
|
||||
assignments ultimately come from, but no codebase artifact
|
||||
corresponds to this layer.
|
||||
|
||||
---
|
||||
|
||||
## 5. AMHS — Automated Material Handling System
|
||||
|
||||
**What it is.** The robot-rail network and overhead hoist transport
|
||||
(**OHT**) system that physically moves carriers (FOUPs holding ~25
|
||||
wafers each) between tools. In a modern 300 mm fab the AMHS is
|
||||
*always* moving carriers between tools 24/7; humans never touch a
|
||||
substrate.
|
||||
|
||||
**What it does in SECS/GEM terms.**
|
||||
|
||||
- The AMHS itself **doesn't speak SECS/GEM** — it has its own
|
||||
control plane talking to a Material Control System (MCS) which is
|
||||
conceptually peer to the MES.
|
||||
- But every time a carrier *arrives at* or *departs from* an
|
||||
equipment's load port, the AMHS-side robot and the equipment-side
|
||||
load port **handshake over 8 parallel I/O lines** defined by
|
||||
**E84**. This is a physical-layer handshake (CMOS-level voltages
|
||||
on real wires) with strict timing — TA1, TA2, TA3 timers — to
|
||||
make sure a $20 000 FOUP doesn't get dropped because both sides
|
||||
thought the other one was holding it.
|
||||
- Once the carrier is physically docked, the equipment fires a
|
||||
`S6F11(CarrierArrived)` event to the MES and the MES sends back a
|
||||
`S3F17(CarrierAction=ProceedWithCarrier)` to authorise processing.
|
||||
|
||||
**Where it lives in this codebase.**
|
||||
|
||||
- The E84 handshake state machine: [`include/secsgem/gem/e84.hpp`](../include/secsgem/gem/e84.hpp)
|
||||
+ [`src/gem/e84.cpp`](../src/gem/e84.cpp).
|
||||
- The TA1/TA2/TA3 timer wiring: [`include/secsgem/gem/e84_timers.hpp`](../include/secsgem/gem/e84_timers.hpp),
|
||||
[`include/secsgem/gem/e84_asio_timers.hpp`](../include/secsgem/gem/e84_asio_timers.hpp).
|
||||
- The per-port store: `e84_ports.hpp` (see [`include/secsgem/gem/e84_ports.hpp`](../include/secsgem/gem/e84_ports.hpp)).
|
||||
- Tests covering the timing rules: [`tests/test_e84.cpp`](../tests/test_e84.cpp),
|
||||
[`tests/test_e84_timers.cpp`](../tests/test_e84_timers.cpp),
|
||||
[`tests/test_e84_asio_timers.cpp`](../tests/test_e84_asio_timers.cpp),
|
||||
[`tests/test_e84_ports.cpp`](../tests/test_e84_ports.cpp).
|
||||
|
||||
Chapter [18](18_e84_parallel_io.md) covers E84 in full.
|
||||
|
||||
---
|
||||
|
||||
## 6. Operator — the human
|
||||
|
||||
**What it is.** The fab technician at the tool's local panel. Their
|
||||
job is to handle anything the automation can't: load a non-AMHS
|
||||
carrier, clear a jammed wafer, run a maintenance recipe, respond to
|
||||
an alarm the MES can't auto-clear.
|
||||
|
||||
**What they do in SECS/GEM terms.**
|
||||
|
||||
- **Mode switch.** The operator can push the equipment between
|
||||
control states: `EquipmentOffline`, `OnlineLocal` (commands
|
||||
accepted only from the local panel), `OnlineRemote` (commands
|
||||
accepted from the MES). This is E30 §6.2.
|
||||
- **Override.** An operator can override an MES command (refuse to
|
||||
start, force-clear an alarm, manually unload a carrier). In
|
||||
SECS/GEM terms this is reflected by control-state transitions:
|
||||
`OnlineRemote` → `OnlineLocal` means "operator has taken control."
|
||||
- **Local alarm acknowledgement.** Some alarms can be cleared at
|
||||
the panel without the MES being involved; the equipment then
|
||||
emits an `S5F1` with the cleared bit so the MES catches up.
|
||||
|
||||
**Where it lives in this codebase.**
|
||||
|
||||
- The control state machine: [`include/secsgem/gem/control_state.hpp`](../include/secsgem/gem/control_state.hpp)
|
||||
+ [`src/gem/control_state.cpp`](../src/gem/control_state.cpp).
|
||||
- The transition table loaded from YAML: [`data/control_state.yaml`](../data/control_state.yaml).
|
||||
- The operator-initiated transition handlers:
|
||||
`ControlStateMachine::operator_online`, `::operator_offline`,
|
||||
`::operator_local`, `::operator_remote` in the same header.
|
||||
- Tests: [`tests/test_control_state.cpp`](../tests/test_control_state.cpp).
|
||||
|
||||
Chapter [13](13_e30_gem.md) walks through control state in detail.
|
||||
|
||||
---
|
||||
|
||||
## Who talks to whom
|
||||
|
||||
A short reference table. "Init." marks who initiates the
|
||||
conversation; "Channel" marks the protocol layer.
|
||||
|
||||
| Pair | Init. | Channel | Examples |
|
||||
|----------------------------|------------|---------------------------------------------|-----------------------------------------------------------|
|
||||
| Planner ↔ MES | Planner | REST / SQL / queue (out of scope) | "run lot L on tool T with recipe R" |
|
||||
| MES ↔ EAP | MES | HSMS-SS (one TCP socket, equipment passive) | `S1F1`, `S2F41`, `S6F11`, `S5F1`, … |
|
||||
| MES ↔ EAP (multi-MES) | MES | HSMS-GS (one TCP socket, multiple sessions) | Same messages, demuxed by session_id |
|
||||
| EAP ↔ Equipment | Either | PLC / sensor APIs / recipe engine (tool-specific) | Out of scope of SECS/GEM |
|
||||
| AMHS ↔ Load port | AMHS | E84 8-line parallel I/O | VALID/CS_0/CS_1/TR_REQ/READY/BUSY/COMPT/CONT/L_REQ/U_REQ/ES |
|
||||
| MES ↔ EAP (carrier flow) | Equipment | HSMS | `S6F11(CarrierArrived)`, `S3F17(ProceedWithCarrier)` |
|
||||
| Operator ↔ Equipment | Operator | Local panel | Online/Offline buttons, alarm acks |
|
||||
|
||||
The four interesting things in this table:
|
||||
|
||||
1. **The MES is the active side, the equipment is the passive side.**
|
||||
Always. Equipment binds the port; MES connects to it. Some MES
|
||||
want this reversed and will negotiate, but the GEM default is
|
||||
equipment-passive.
|
||||
2. **One TCP socket per (MES, equipment) pair.** HSMS-SS doesn't
|
||||
multiplex; one connection serves one conversation. HSMS-GS adds
|
||||
session multiplexing on top.
|
||||
3. **Equipment-initiated traffic exists.** `S6F11` events and
|
||||
`S5F1` alarms fire from equipment to MES *autonomously*, not
|
||||
in reply to a host command. An EAP that never emits unsolicited
|
||||
traffic is broken.
|
||||
4. **The AMHS handshake is out-of-band relative to SECS/GEM.**
|
||||
E84's 8 parallel I/O lines are real wires with real voltages;
|
||||
the SECS messages that follow (`S6F11`, `S3F17`) are just the
|
||||
*bookkeeping* around a handoff that already happened in
|
||||
hardware.
|
||||
|
||||
---
|
||||
|
||||
## A small mental check
|
||||
|
||||
If you've internalised the chapter, you should be able to answer:
|
||||
|
||||
1. When the MES sends `S1F13`, who initiates the TCP connection?
|
||||
2. When a chamber pressure sensor reads out of range, who decides
|
||||
whether to fire `S5F1`?
|
||||
3. What language does the AMHS speak to the equipment to coordinate
|
||||
a FOUP handoff?
|
||||
4. Is the operator at the local panel a SECS/GEM actor?
|
||||
5. Is the fab planner a SECS/GEM actor?
|
||||
|
||||
Answers, in order:
|
||||
|
||||
1. The MES. The equipment is passive; it binds and waits. TCP
|
||||
connect is the MES's first move.
|
||||
2. The EAP (the vendor's application code on top of this library).
|
||||
The SECS/GEM library doesn't know what's a "normal" pressure;
|
||||
the EAP does. Once the EAP decides "this is alarm-worthy," it
|
||||
calls into the alarm store and the library emits `S5F1`.
|
||||
3. E84 8-line parallel I/O — physical wires, not SECS. After the
|
||||
handoff, the *bookkeeping* SECS messages (`S6F11`, `S3F17`) flow
|
||||
between equipment and MES.
|
||||
4. Yes — through E30 §6.2 control-state transitions. Not a SECS
|
||||
message *sender*, but a state-transition source the equipment
|
||||
has to report on.
|
||||
5. No. The planner talks to the MES; the MES talks to the EAP.
|
||||
The planner is invisible from the SECS wire.
|
||||
|
||||
---
|
||||
|
||||
## What's next
|
||||
|
||||
You now know who's in the room and who's talking to whom. The next
|
||||
chapter introduces the **vocabulary** — every SEMI acronym you'll
|
||||
read in a debug log (SVID, CEID, ALID, PPID, ALCD, HCACK, T-timers,
|
||||
…) — by tracing **one wafer's journey** end-to-end through a fab and
|
||||
labelling every SECS message that fires along the way.
|
||||
|
||||
Next: [→ 03 Vocabulary + a wafer's journey](03_vocabulary_and_a_wafers_journey.md)
|
||||
@@ -0,0 +1,592 @@
|
||||
# 03 — Vocabulary + a wafer's journey
|
||||
|
||||
← [02 The cast of characters](02_the_cast.md) | [Back to index](00_index.md) | Next: [10 E5 — SECS-II data items](10_e5_secs_ii_data_items.md) →
|
||||
|
||||
The SEMI standards bury everything in acronyms. Three-letter, four-
|
||||
letter, sometimes the same letter pattern (`ACKC5`, `ACKC6`, `ACKC7`,
|
||||
`ACKC10`) but with completely different semantics depending on which
|
||||
stream you're in. Most readers learn them by absorbing them over
|
||||
years of integration work.
|
||||
|
||||
This chapter accelerates that. We follow **one 300 mm wafer** from
|
||||
the moment it enters the fab to the moment it leaves as a finished
|
||||
die, and at every step we name every acronym that fires, what it
|
||||
means, and where it lives in this codebase. By the end you'll have
|
||||
seen `SVID`, `CEID`, `ALID`, `PPID`, `HCACK`, `ALCD`, `RPTID`,
|
||||
`OFLACK`, `MDLN`, `SOFTREV`, `CAACK`, `SMACK`, and the rest in
|
||||
*context* — not as a vocabulary list.
|
||||
|
||||
If you're confident with the vocabulary already, skip to Part 2,
|
||||
Chapter [10](10_e5_secs_ii_data_items.md) (SECS-II encoding).
|
||||
|
||||
---
|
||||
|
||||
## The setup
|
||||
|
||||
- **The wafer**: an unpatterned 300 mm silicon disc, 775 µm thick,
|
||||
with a serial number `W-2026-06-09-A47` etched on the bevel.
|
||||
- **The carrier**: a Front-Opening Universal Pod (**FOUP**) that
|
||||
holds 25 wafers in vertical slots. Our wafer is in slot 14. The
|
||||
FOUP's bar code reads `C-31415`.
|
||||
- **The tools**:
|
||||
- **PVD-1** (physical vapour deposition — deposits a metal layer)
|
||||
- **LITHO-3** (photolithography — patterns the metal layer)
|
||||
- **ETCH-7** (plasma etch — removes uncovered metal)
|
||||
- **The host**: a fab-wide MES called `meta-fab.example`.
|
||||
- **The recipe**: `RECIPE-Cu-A` for PVD-1, `RECIPE-193nm-X` for
|
||||
LITHO-3, `RECIPE-CL2-B` for ETCH-7.
|
||||
|
||||
For brevity we'll only show the wafer's first pass through PVD-1.
|
||||
The same pattern repeats for LITHO-3 and ETCH-7.
|
||||
|
||||
---
|
||||
|
||||
## Stage 1 — Establishing communications (already done)
|
||||
|
||||
Before any wafer arrives, **PVD-1 and meta-fab.example have already
|
||||
HSMS-SELECTed each other**. That's the once-per-power-on dance:
|
||||
|
||||
```
|
||||
host (active) equipment (passive)
|
||||
───────────── ───────────────────
|
||||
TCP SYN ─────────────────────────► (bind on :5000)
|
||||
◄──── TCP SYN-ACK
|
||||
HSMS Select.req (sessionID=0) ───►
|
||||
◄──── HSMS Select.rsp (SELECT_STATUS=0=accept)
|
||||
[transport state: SELECTED]
|
||||
S1F13 Establish Communications ──►
|
||||
◄──── S1F14 (COMMACK=0=accepted, [MDLN, SOFTREV])
|
||||
[GEM communication state: COMMUNICATING]
|
||||
```
|
||||
|
||||
This introduces three acronyms:
|
||||
|
||||
- **`MDLN`** — Model Name. An ASCII string up to 20 chars
|
||||
identifying the equipment model. PVD-1 returns `"ACME-PVD-3000"`.
|
||||
- **`SOFTREV`** — Software Revision. ASCII string up to 20 chars
|
||||
identifying the firmware / EAP version. PVD-1 returns `"1.4.2"`.
|
||||
- **`COMMACK`** — Communication Acknowledge. One byte; 0 = accepted,
|
||||
1 = denied. Defined in E30 §6.5.
|
||||
|
||||
**Where:** see `equipment.yaml` device block; emission flows through
|
||||
[`gem::Router`](../include/secsgem/gem/router.hpp) →
|
||||
[`secsgem::secs2::Message`](../include/secsgem/secs2/message.hpp).
|
||||
|
||||
---
|
||||
|
||||
## Stage 2 — The carrier arrives at PVD-1's load port
|
||||
|
||||
The AMHS overhead hoist swings FOUP `C-31415` onto load port 1.
|
||||
*Before* anything SECS happens, the **E84 handshake** runs on the
|
||||
physical I/O lines:
|
||||
|
||||
```
|
||||
AMHS robot load port 1
|
||||
────────── ───────────
|
||||
CS_0 asserted ───────────────────► (carrier select bit 0)
|
||||
CS_1 asserted ───────────────────► (carrier select bit 1)
|
||||
VALID asserted ──────────────────► (lines above are stable)
|
||||
◄──── L_REQ asserted (LOAD allowed)
|
||||
TR_REQ asserted ─────────────────► (transfer requested)
|
||||
◄──── READY asserted (kinematic interlocks ok)
|
||||
BUSY asserted ───────────────────► (placement in progress)
|
||||
… mechanical placement happens (~5 seconds) …
|
||||
BUSY de-asserted ────────────────► (placement done)
|
||||
◄──── COMPT asserted (complete)
|
||||
CONT asserted ───────────────────► (carrier connected to load port)
|
||||
```
|
||||
|
||||
This introduces the E84 line-name acronyms (`VALID`, `CS_0`, `CS_1`,
|
||||
`TR_REQ`, `READY`, `BUSY`, `COMPT`, `CONT`, `L_REQ`, `U_REQ`, `ES`)
|
||||
and three timer names:
|
||||
|
||||
- **`TA1`** — armed when `VALID` asserts; the load port must respond
|
||||
with `L_REQ` within `TA1`. Default ~2 seconds.
|
||||
- **`TA2`** — armed when `L_REQ` asserts; `TR_REQ` must follow
|
||||
within `TA2`. Default ~2 seconds.
|
||||
- **`TA3`** — armed when `BUSY` asserts (transfer in progress); the
|
||||
whole transfer must finish within `TA3`. Default ~60 seconds.
|
||||
|
||||
Any of these timing out → both sides go to `HandoffFault` and the
|
||||
operator gets paged. No FOUP gets dropped because the protocol
|
||||
guarantees both sides agreed on every step.
|
||||
|
||||
**Where:** [`include/secsgem/gem/e84.hpp`](../include/secsgem/gem/e84.hpp)
|
||||
defines the FSM; [`include/secsgem/gem/e84_timers.hpp`](../include/secsgem/gem/e84_timers.hpp)
|
||||
defines the timer enforcement; chapter [18](18_e84_parallel_io.md)
|
||||
walks the whole handshake.
|
||||
|
||||
---
|
||||
|
||||
## Stage 3 — The carrier is on the load port; PVD-1 tells the MES
|
||||
|
||||
The E84 handshake gave PVD-1 a docked carrier. Now SECS messages
|
||||
flow:
|
||||
|
||||
```
|
||||
PVD-1 (equipment) meta-fab (host)
|
||||
───────────────── ───────────────
|
||||
S6F11 CarrierArrived ────────────►
|
||||
CEID = 10001
|
||||
DATAID = 1
|
||||
[ {RPTID=100, V=[CarrierID="C-31415", PortID=1]} ]
|
||||
◄──── S6F12 (ACKC6=0=accepted)
|
||||
|
||||
S3F19 Slot Map Verify ───────────►
|
||||
CARRIERID = "C-31415"
|
||||
[ slot_state[1..25] ]
|
||||
◄──── S3F20 (SMACK=0=match)
|
||||
```
|
||||
|
||||
New acronyms in this stage:
|
||||
|
||||
- **`CEID`** — Collection Event ID. An identifier (any unsigned
|
||||
width; we'll use `U4`) for a noteworthy thing that happened. CEIDs
|
||||
are *defined in the equipment's YAML* and the MES learns them via
|
||||
`S1F23/F24`. CEID 10001 = `CarrierArrived` per E87.
|
||||
- **`RPTID`** — Report ID. A bundle of variables. When CEID 10001
|
||||
fires, the MES gets back the values of every variable linked to
|
||||
every report linked to CEID 10001. Reports are *defined by the
|
||||
host* via `S2F33` and linked to CEIDs via `S2F35`.
|
||||
- **`DATAID`** — Data ID, a per-host transaction counter. Lets the
|
||||
host correlate report data to a specific request.
|
||||
- **`ACKC6`** — Acknowledge Code 6. S6F12 reply byte. 0 =
|
||||
accepted, anything else = MES couldn't process the event.
|
||||
- **`CARRIERID`** — Carrier ID, an ASCII string. Matches what the
|
||||
AMHS told us via E84.
|
||||
- **`SMACK`** — Slot Map Acknowledge. S3F20 reply byte. 0 =
|
||||
matches what the MES expected, 1 = mismatch. Defined in E87.
|
||||
- **`CAACK`** — Carrier Action Acknowledge (we'll see this one
|
||||
shortly). S3F18 reply byte for carrier-action commands.
|
||||
|
||||
Note the pattern: every primary message ends in an odd function
|
||||
(F11, F19), every reply ends in the next even function (F12, F20).
|
||||
This is invariant across SECS-II. See chapter [10](10_e5_secs_ii_data_items.md)
|
||||
for the encoding details.
|
||||
|
||||
**Where:** `gem::CarrierStore` in [`include/secsgem/gem/carrier_store.hpp`](../include/secsgem/gem/carrier_store.hpp);
|
||||
the E87 wire tests in [`tests/test_e87_wire_scenarios.cpp`](../tests/test_e87_wire_scenarios.cpp).
|
||||
|
||||
---
|
||||
|
||||
## Stage 4 — The host authorises processing
|
||||
|
||||
```
|
||||
meta-fab (host) PVD-1 (equipment)
|
||||
─────────────── ─────────────────
|
||||
S3F17 CarrierAction ─────────────►
|
||||
CARRIERACTION = "ProceedWithCarrier"
|
||||
CARRIERID = "C-31415"
|
||||
◄──── S3F18 (CAACK=0=accepted)
|
||||
```
|
||||
|
||||
- **`CARRIERACTION`** — an ASCII string from a fixed E87 set:
|
||||
`ProceedWithCarrier`, `CancelCarrier`, `CarrierOut`, …
|
||||
- **`CAACK`** — Carrier Action Acknowledge. S3F18 reply byte. 0 =
|
||||
accepted, 1 = unknown carrier, 2 = invalid action, 3 = invalid
|
||||
state, 4 = mismatch, 5 = unknown.
|
||||
|
||||
The host could have sent `CancelCarrier` here instead and PVD-1
|
||||
would have rejected the FOUP without processing. That decision
|
||||
lives entirely on the host side.
|
||||
|
||||
---
|
||||
|
||||
## Stage 5 — The host queues a process job
|
||||
|
||||
```
|
||||
meta-fab (host) PVD-1 (equipment)
|
||||
─────────────── ─────────────────
|
||||
S16F11 PRJobCreate ──────────────►
|
||||
PRJobID = "PJ-2026-06-09-001"
|
||||
MF = "Substrate"
|
||||
PRMtlOutSpec = []
|
||||
PRRecipeMethod = "RecipeOnly"
|
||||
RCPSpec = "RECIPE-Cu-A"
|
||||
PRProcessStart = false (we'll start it explicitly later)
|
||||
PRMtlnameList = ["W-2026-06-09-A47"]
|
||||
◄──── S16F12 (PRJobAck=0)
|
||||
|
||||
S14F9 CreateControlJob ─────────►
|
||||
CJobID = "CJ-2026-06-09-001"
|
||||
PRJobIDList = ["PJ-2026-06-09-001"]
|
||||
◄──── S14F10 (OBJACK=0)
|
||||
```
|
||||
|
||||
New acronyms:
|
||||
|
||||
- **`PRJobID`** — Process Job ID, an ASCII string the host invents
|
||||
for tracking. Sometimes called `PJID`.
|
||||
- **`MF`** — Material Format. ASCII; `Substrate`, `Carrier`,
|
||||
`SubstrateLocation`. Tells the equipment what scale the job is
|
||||
about.
|
||||
- **`RCPSpec`** — Recipe specification. References a recipe by ID
|
||||
(`PPID`, below).
|
||||
- **`PPID`** — Process Program ID. The recipe's identifier. In our
|
||||
case `"RECIPE-Cu-A"`.
|
||||
- **`PRJobAck`** — S16F12 reply byte. 0 = accepted, non-zero
|
||||
values for each failure mode.
|
||||
- **`CJobID`** — Control Job ID. A control job wraps one or more
|
||||
process jobs and adds scheduling semantics (start order, abort
|
||||
policy, dependency on other CJs).
|
||||
- **`OBJACK`** — Object Acknowledge. S14F10 reply byte. Generic
|
||||
E39 object-services ack: 0 = accepted, 1 = error.
|
||||
|
||||
E40 governs process jobs; E94 governs control jobs above them.
|
||||
|
||||
**Where:** [`include/secsgem/gem/process_jobs.hpp`](../include/secsgem/gem/process_jobs.hpp),
|
||||
[`include/secsgem/gem/control_jobs.hpp`](../include/secsgem/gem/control_jobs.hpp).
|
||||
The state machines are loaded from
|
||||
[`data/process_job_state.yaml`](../data/process_job_state.yaml) and
|
||||
[`data/control_job_state.yaml`](../data/control_job_state.yaml).
|
||||
See chapter [14](14_e40_e94_jobs.md) for the lifecycle in full.
|
||||
|
||||
---
|
||||
|
||||
## Stage 6 — The host configures event reports
|
||||
|
||||
Before processing starts, the MES wants to subscribe to specific
|
||||
events. This is a three-message dance:
|
||||
|
||||
```
|
||||
meta-fab (host) PVD-1 (equipment)
|
||||
─────────────── ─────────────────
|
||||
S2F33 DefineReport ──────────────►
|
||||
DATAID = 2
|
||||
[ { RPTID=200, VID=[1, 2, 5] } ] (link RPTID 200 to SVIDs 1,2,5)
|
||||
◄──── S2F34 (DRACK=0=accepted)
|
||||
|
||||
S2F35 LinkEvent ─────────────────►
|
||||
DATAID = 3
|
||||
[ { CEID=300, RPTID=[200] } ] (when CEID 300 fires, send RPTID 200)
|
||||
◄──── S2F36 (LRACK=0=accepted)
|
||||
|
||||
S2F37 EnableEvent ───────────────►
|
||||
CEED = true
|
||||
CEID = [300] (enable CEID 300)
|
||||
◄──── S2F38 (ERACK=0=accepted)
|
||||
```
|
||||
|
||||
New acronyms:
|
||||
|
||||
- **`SVID`** — Status Variable ID. An identifier (`U4` typical) for
|
||||
a *long-lived* value the host can read at any time — current
|
||||
control state, chamber pressure, wafer counter, recipe in progress,
|
||||
clock. Roughly: instance variables that survive across events.
|
||||
- **`DVID`** — Data Variable ID. Same shape, but only meaningful
|
||||
*at the moment an event fires*. E.g. the temperature at the time
|
||||
the `ProcessStarted` event was emitted. Not readable independently
|
||||
via `S1F3`; only delivered as part of a report.
|
||||
- **`ECID`** — Equipment Constant ID. Same shape, but the host can
|
||||
*set* it (within declared `min`/`max` bounds) via `S2F15`.
|
||||
Settings that survive power-cycle: nominal chamber pressure, T3
|
||||
timeout, T7 timeout, etc.
|
||||
- **`VID`** — Variable ID. A generic SVID-or-DVID, used in report
|
||||
definitions.
|
||||
- **`CEED`** — Collection Event Enable Disable. Boolean. `true` =
|
||||
enable the listed CEIDs, `false` = disable them.
|
||||
- **`DRACK`** — Define Report Acknowledge. S2F34 reply.
|
||||
- **`LRACK`** — Link Report Acknowledge. S2F36 reply.
|
||||
- **`ERACK`** — Enable Report Acknowledge. S2F38 reply.
|
||||
|
||||
Acknowledge bytes are *distinct enums per stream/function*. `DRACK
|
||||
= 0` = accepted; `LRACK = 0` = accepted; `ERACK = 0` = accepted —
|
||||
same value, but each one has its own enumeration of failure codes
|
||||
(`3 = at least one CEID does not exist`, etc.). Don't reuse one
|
||||
stream's enum for another's.
|
||||
|
||||
**Where:** [`include/secsgem/gem/report_store.hpp`](../include/secsgem/gem/report_store.hpp),
|
||||
[`include/secsgem/gem/event_store.hpp`](../include/secsgem/gem/event_store.hpp).
|
||||
The configuration flow is the heart of E30 §6.6 Dynamic Event Report
|
||||
Configuration; see chapter [13](13_e30_gem.md).
|
||||
|
||||
---
|
||||
|
||||
## Stage 7 — Processing begins
|
||||
|
||||
```
|
||||
meta-fab (host) PVD-1 (equipment)
|
||||
─────────────── ─────────────────
|
||||
S2F41 RemoteCommand ─────────────►
|
||||
RCMD = "START"
|
||||
CPNAME[] / CPVAL[] = []
|
||||
◄──── S2F42 (HCACK=0=accepted)
|
||||
|
||||
S6F11 ProcessStarted ──────────►
|
||||
CEID = 300
|
||||
DATAID = 4
|
||||
[ {RPTID=200,
|
||||
V=[ControlState="OnlineRemote",
|
||||
Clock="20260609173000",
|
||||
WaferCount=147]} ]
|
||||
◄──── S6F12 (ACKC6=0)
|
||||
```
|
||||
|
||||
New acronyms:
|
||||
|
||||
- **`RCMD`** — Remote Command name. ASCII. `"START"`, `"STOP"`,
|
||||
`"PAUSE"`, `"ABORT"`, `"VENT"`, etc. Equipment-vendor-defined.
|
||||
- **`CPNAME` / `CPVAL`** — Command Parameter name / value pairs.
|
||||
Empty list here; some commands take parameters (e.g.
|
||||
`RCMD="CHANGE-RECIPE", CPNAME="PPID", CPVAL="RECIPE-Cu-B"`).
|
||||
- **`HCACK`** — Host Command Acknowledge. S2F42 reply. 0 =
|
||||
accepted, 1 = invalid command, 2 = cannot perform now, 3 = at
|
||||
least one parameter is invalid, 4 = accepted-and-will-finish-later,
|
||||
5 = rejected, 6 = invalid object.
|
||||
|
||||
The `S6F11(CEID=300)` that fires next is the event report
|
||||
**defined three messages earlier**. The MES correlated this by:
|
||||
|
||||
1. Earlier sent `S2F33` → equipment now knows that "RPTID 200 =
|
||||
[SVID 1, SVID 2, SVID 5]."
|
||||
2. Earlier sent `S2F35` → equipment now knows that "when CEID 300
|
||||
fires, the report payload should include RPTID 200."
|
||||
3. Earlier sent `S2F37` → CEID 300 is enabled, so when the
|
||||
processing logic fires it, an `S6F11` actually leaves the wire.
|
||||
(If CEID 300 had been left *disabled*, the processing logic
|
||||
would still fire it but the wire would stay quiet.)
|
||||
|
||||
**Where:** [`include/secsgem/gem/host_command_registry.hpp`](../include/secsgem/gem/host_command_registry.hpp)
|
||||
maps `RCMD` strings to handlers; the report-emission machinery lives
|
||||
in `EquipmentDataModel` ([`include/secsgem/gem/data_model.hpp`](../include/secsgem/gem/data_model.hpp))
|
||||
via `compose_reports_for(ceid)`. Wire-level tests:
|
||||
[`tests/test_wire_ceid_emission.cpp`](../tests/test_wire_ceid_emission.cpp).
|
||||
|
||||
---
|
||||
|
||||
## Stage 8 — An alarm fires
|
||||
|
||||
Mid-processing, the chamber pressure sensor reads above its
|
||||
configured `ECID="ChamberPressureMax"` threshold. The EAP's alarm
|
||||
monitor decides this is alarm-worthy and calls
|
||||
`alarms.set(ALID=42)`:
|
||||
|
||||
```
|
||||
PVD-1 (equipment) meta-fab (host)
|
||||
───────────────── ───────────────
|
||||
S5F1 AlarmReport ────────────────►
|
||||
ALCD = 0x84 (bit 7 set + category 4 = process)
|
||||
ALID = 42
|
||||
ALTX = "Chamber pressure above max threshold"
|
||||
◄──── S5F2 (ACKC5=0)
|
||||
```
|
||||
|
||||
- **`ALID`** — Alarm ID. An identifier (`U4` typical) for one
|
||||
named alarm in the equipment's alarm directory.
|
||||
- **`ALCD`** — Alarm Code. One byte. Bit 7 = "set" (1) or "clear"
|
||||
(0). Lower 7 bits = category (1 = personal safety, 2 = equipment
|
||||
safety, 3 = parameter control warning, 4 = parameter control
|
||||
error, 5 = irrecoverable error, 6 = equipment status warning, 7 =
|
||||
attention flag, 8 = data integrity, others reserved). E5 §13.
|
||||
- **`ALTX`** — Alarm Text. ASCII description, up to 120 chars per
|
||||
E5 §13.
|
||||
- **`ACKC5`** — Acknowledge Code 5. S5F2 reply. 0 = accepted.
|
||||
|
||||
If the host had previously *disabled* ALID 42 (via `S5F3
|
||||
ALED=0x00`), this `S5F1` wouldn't have left the wire — the equipment
|
||||
would still note the alarm internally (so `S5F5` would list it), but
|
||||
the host wouldn't get pinged.
|
||||
|
||||
When the pressure returns to range, a second `S5F1` fires with
|
||||
`ALCD=0x04` (bit 7 cleared) and the same ALID, signalling "alarm
|
||||
cleared."
|
||||
|
||||
**Where:** [`include/secsgem/gem/alarm_store.hpp`](../include/secsgem/gem/alarm_store.hpp);
|
||||
the dispatcher gates emission on the enable list in
|
||||
[`include/secsgem/gem/alarm_dispatcher.hpp`](../include/secsgem/gem/alarm_dispatcher.hpp).
|
||||
|
||||
---
|
||||
|
||||
## Stage 9 — Processing completes
|
||||
|
||||
```
|
||||
PVD-1 (equipment) meta-fab (host)
|
||||
───────────────── ───────────────
|
||||
S6F11 ProcessCompleted ──────────►
|
||||
CEID = 301
|
||||
DATAID = 5
|
||||
[ {RPTID=200, V=[…]} ]
|
||||
◄──── S6F12 (ACKC6=0)
|
||||
```
|
||||
|
||||
Same pattern as `ProcessStarted` — just a different CEID for a
|
||||
different lifecycle moment.
|
||||
|
||||
The MES sees `CEID=301` and updates its tracking: process job
|
||||
`PJ-2026-06-09-001` is now `ProcessComplete` per E40. It clears its
|
||||
"in progress" counter and updates wafer `W-2026-06-09-A47`'s
|
||||
location in E90 substrate tracking.
|
||||
|
||||
---
|
||||
|
||||
## Stage 10 — Carrier transfers out
|
||||
|
||||
```
|
||||
meta-fab (host) PVD-1 (equipment)
|
||||
─────────────── ─────────────────
|
||||
S3F25 CarrierTransfer ───────────►
|
||||
CARRIERID = "C-31415"
|
||||
PortID = 2 (transfer from LP1 to LP2 = outbound)
|
||||
◄──── S3F26 (CAACK=0)
|
||||
|
||||
PVD-1 (equipment) meta-fab (host)
|
||||
───────────────── ───────────────
|
||||
S6F11 CarrierTransfered ─────────►
|
||||
CEID = 10002
|
||||
[ … ]
|
||||
◄──── S6F12 (ACKC6=0)
|
||||
```
|
||||
|
||||
Then E84 runs in reverse: the AMHS robot couples to the load port,
|
||||
the parallel I/O lines hand control back, the OHT hoist lifts the
|
||||
FOUP, and `C-31415` heads to LITHO-3 for the next process step.
|
||||
|
||||
---
|
||||
|
||||
## Wait, what other acronyms exist?
|
||||
|
||||
The journey above covered the most common acronyms but skipped a
|
||||
handful that show up in other contexts. A reference list for the
|
||||
rest:
|
||||
|
||||
### Acknowledge codes you haven't met yet
|
||||
|
||||
| Code | Stream | Where | 0 = accepted, then… |
|
||||
|----------|--------|--------------------------------|----------------------------------------------|
|
||||
| `OFLACK` | 1 | S1F16 reply to "Request Offline" | 0=accept, 1=already offline |
|
||||
| `ONLACK` | 1 | S1F18 reply to "Request Online" | 0=accept, 1=not allowed, 2=already online |
|
||||
| `ACKC7` | 7 | S7F4 / S7F18 (recipe send/delete) | 0=accept, 1=permission denied, 2=length err,3=matrix err,4=PPID not found,5=mode unsupported,6=other |
|
||||
| `ACKC10` | 10 | S10F2 / F4 / F6 (terminal services)| 0=accept, 1=not displayed, 2=no terminal |
|
||||
| `CMDA` | 2 | S2F22 reply to legacy `S2F21` | 0=ok, 1=invalid command, 2=cannot do now,3=invalid arg |
|
||||
| `TIACK` | 2 | S2F32 reply to "Set Clock" | 0=accept, 1=err not done |
|
||||
| `EAC` | 2 | S2F16 reply to "Set EC values" | 0=accept, 1=≥1 constant out of range, 2=busy, 3=≥1 constant unknown |
|
||||
| `RSPACK` | 2 | S2F44 reply to "Set Spool Streams" | 0=accept, 1=spool not supported, 2=≥1 stream unknown |
|
||||
| `RSDA` | 6 | S6F24 reply to "Spool Data Send" | 0=ok, 1=denied |
|
||||
| `PPGNT` | 7 | S7F2 reply to "PP Load Inquire"| 0=permit, 1=already have, 2=no room, 3=invalid PPID, 4=mode unsupported, 5=PP non-existent, 6=other |
|
||||
|
||||
### Control codes you haven't met
|
||||
|
||||
| Code | Stream | Where | What it means |
|
||||
|----------|--------|-----------------------------|----------------------------------------------------------------|
|
||||
| `RSDC` | 6 | S6F23 host command to spool | 0=transmit spooled, 1=purge spool |
|
||||
| `ALED` | 5 | S5F3 host enable/disable alarm | bit 7 set = enable, bit 7 cleared = disable |
|
||||
| `TID` | 10 | S10F1/F3/F5 terminal display | Which terminal screen to address (0 = main) |
|
||||
| `TEXT` | 10 | S10F1/F3/F5 terminal display | The ASCII text payload |
|
||||
|
||||
### Object-services codes (E39)
|
||||
|
||||
| Code | Stream | Where | What it means |
|
||||
|-----------|--------|-----------------------------|--------------------------------------------------------------|
|
||||
| `OBJSPEC` | various | S2F49, S14F1 | An "object specifier" — a typed path identifying a target object |
|
||||
| `OBJACK` | 14 | S14F2 / F10 / F12 reply | 0=ok, 1=command failed |
|
||||
| `CPACK` | 2 | S2F42 reply (modern) | Per-parameter ack for each `CPNAME/CPVAL` in the command |
|
||||
| `CEPACK` | 2 | S2F50 reply (enhanced) | Per-parameter ack for enhanced remote commands |
|
||||
|
||||
---
|
||||
|
||||
## All the T-timers in one place
|
||||
|
||||
You'll meet timer acronyms in three different contexts. They use
|
||||
the same letters with different meanings — pin them down:
|
||||
|
||||
### HSMS T-timers (E37 §10)
|
||||
|
||||
These bound the *network* behaviour. Only fire when something is
|
||||
slow or stuck.
|
||||
|
||||
| Name | Default | What it bounds |
|
||||
|------|-------------|------------------------------------------------------|
|
||||
| `T3` | 45 s | Reply timeout for a W=1 primary message |
|
||||
| `T5` | 10 s | How long active side waits between connect attempts |
|
||||
| `T6` | 5 s | Control-transaction (Select / Linktest) reply timeout |
|
||||
| `T7` | 10 s | Passive side: max time without Select.req after TCP |
|
||||
| `T8` | 5 s | Max time between bytes of a single frame |
|
||||
|
||||
Chapter [11](11_e37_hsms.md) covers each one in detail.
|
||||
|
||||
### SECS-I T-timers (E4 §10)
|
||||
|
||||
These bound the *serial-block* behaviour. Distinct from HSMS
|
||||
T-timers despite the name overlap.
|
||||
|
||||
| Name | Default | What it bounds |
|
||||
|------|-------------|------------------------------------------------------|
|
||||
| `T1` | 500 ms | Inter-character timeout within one block |
|
||||
| `T2` | 10 s | Protocol timer (handshake state) |
|
||||
| `T3` | 45 s | Reply timeout for a W=1 primary message |
|
||||
| `T4` | 45 s | Inter-block timeout in a multi-block message |
|
||||
|
||||
`T3` exists in both HSMS and SECS-I with the same semantics — it's
|
||||
load-bearing in both transports.
|
||||
|
||||
### E84 timers (E84 §6)
|
||||
|
||||
These bound the *physical handoff* timing. Distinct again.
|
||||
|
||||
| Name | Default | What it bounds |
|
||||
|-------|---------|-------------------------------------------------|
|
||||
| `TA1` | ~2 s | `VALID` → `L_REQ` |
|
||||
| `TA2` | ~2 s | `L_REQ` → `TR_REQ` |
|
||||
| `TA3` | ~60 s | `BUSY` → transfer complete |
|
||||
|
||||
### E30 communication-state timers (E30 §6.5)
|
||||
|
||||
These bound the *application-level* establish-communications loop:
|
||||
|
||||
| Name | Default | What it bounds |
|
||||
|-----------|---------|-----------------------------------------------------------------|
|
||||
| `T_CRA` | 45 s | Wait for `S1F14` (Comm Request Acknowledge) reply after `S1F13` |
|
||||
| `T_DELAY` | 10 s | Retry interval after a failed `S1F13` round-trip |
|
||||
|
||||
Defined in [`include/secsgem/gem/communication_state.hpp`](../include/secsgem/gem/communication_state.hpp);
|
||||
tested in [`tests/test_communication_state.cpp`](../tests/test_communication_state.cpp).
|
||||
|
||||
---
|
||||
|
||||
## Stream-by-stream summary
|
||||
|
||||
The streams you'll meet most often, with one sentence each:
|
||||
|
||||
| Stream | What it's for | Most-used messages |
|
||||
|--------|----------------------------------------------------------|---------------------------------------------------|
|
||||
| S1 | Identification, status, control | `S1F1/F2`, `S1F3/F4`, `S1F11/F12`, `S1F13/F14`, `S1F15-F18`, `S1F19/F20`, `S1F21-F24` |
|
||||
| S2 | Equipment constants, clock, events, commands | `S2F13-F18`, `S2F29-F38`, `S2F41/F42`, `S2F43-F50` |
|
||||
| S3 | Carrier management (E87) | `S3F17/F18`, `S3F19/F20`, `S3F25-F28` |
|
||||
| S5 | Alarms, exception recovery | `S5F1-F8`, `S5F9-F18` |
|
||||
| S6 | Data collection, event reports, spool | `S6F11/F12`, `S6F15/F16`, `S6F19-F22`, `S6F23-F26` |
|
||||
| S7 | Recipe / process program management | `S7F1-F6`, `S7F17-F20`, `S7F23-F26` |
|
||||
| S9 | Protocol-error reports (auto-emitted by equipment) | `S9F1`, `S9F3`, `S9F5`, `S9F7`, `S9F9`, `S9F11`, `S9F13` |
|
||||
| S10 | Terminal services | `S10F1-F6` |
|
||||
| S12 | Wafer maps | (per-stream — chapter [10](10_e5_secs_ii_data_items.md) §6) |
|
||||
| S14 | Generic object services (E39), control jobs (E94) | `S14F1/F2`, `S14F9-F12` |
|
||||
| S16 | Process jobs (E40) | `S16F5-F8`, `S16F9`, `S16F11-F14`, `S16F27/F28` |
|
||||
|
||||
Where every named message lives in code: [`build/generated/secsgem/gem/messages.hpp`](../build/generated/secsgem/gem/messages.hpp)
|
||||
(after a build) — generated from
|
||||
[`data/messages.yaml`](../data/messages.yaml) by
|
||||
[`tools/generate_messages.py`](../tools/generate_messages.py).
|
||||
Chapter [31](31_spec_as_data_and_codegen.md) walks the codegen.
|
||||
|
||||
---
|
||||
|
||||
## You've made it through Part 1
|
||||
|
||||
You can now:
|
||||
|
||||
- Explain why SECS/GEM exists, and what each of "SECS", "HSMS", and
|
||||
"GEM" actually refers to.
|
||||
- Name every actor in the fab automation stack and describe who
|
||||
talks to whom.
|
||||
- Recognise every common acronym (SVID, ECID, DVID, CEID, RPTID,
|
||||
ALID, PPID, MDLN, SOFTREV, HCACK, ALCD, OFLACK, …) and the
|
||||
acknowledge bytes that go with each stream.
|
||||
- List the T-timers in four different contexts (HSMS, SECS-I, E84,
|
||||
E30 communication state) and not confuse them.
|
||||
|
||||
Part 2 of this guide takes one standard at a time, from the
|
||||
ground up: byte-level encoding, wire diagrams, every message, every
|
||||
ack value, the FSMs, and the code that implements each. We start
|
||||
with the foundation everything else stands on — **E5 SECS-II**, the
|
||||
data-item encoding.
|
||||
|
||||
Next: [→ 10 E5 — SECS-II data items](10_e5_secs_ii_data_items.md)
|
||||
@@ -0,0 +1,502 @@
|
||||
# Architecture
|
||||
|
||||
How the codebase is put together, and how to extend it. Read after
|
||||
[INTEGRATION.md](INTEGRATION.md) — that doc tells you what to do;
|
||||
this one tells you *why*, and where to plug in new behaviour.
|
||||
|
||||
---
|
||||
|
||||
## 1. Design principle: spec-as-data
|
||||
|
||||
The SEMI standards describe behaviour as **tables** — state machines,
|
||||
message catalogues, transition rules. C++ is the wrong language to
|
||||
write those tables in directly: every spec edit becomes a recompile,
|
||||
and reviewers can't audit "does the implementation match E40 §6.3"
|
||||
without reading code.
|
||||
|
||||
So the rule across the project is: **anything the SEMI spec encodes
|
||||
as a table lives in YAML.** The C++ is the engine that reads them.
|
||||
|
||||
```
|
||||
data/messages.yaml → tools/gen_messages.py → messages.hpp
|
||||
data/control_state.yaml → config::load_control_state()
|
||||
data/process_job_state.yaml → config::load_process_job_state()
|
||||
data/control_job_state.yaml → config::load_control_job_state()
|
||||
data/equipment.yaml → config::load_equipment()
|
||||
```
|
||||
|
||||
Two consequences worth absorbing:
|
||||
|
||||
- **Adding a new SECS-II message rarely requires C++.** Edit
|
||||
`data/messages.yaml`, rebuild, register a handler with the Router.
|
||||
- **Adding a new state transition rarely requires C++.** Edit the
|
||||
relevant state YAML; the loader hot-loads on next start.
|
||||
|
||||
Things that do require C++: new *kinds* of behaviour (new FSM, new
|
||||
store, new persistence backend) — and that's what the rest of this
|
||||
doc covers.
|
||||
|
||||
---
|
||||
|
||||
## 2. The five layers
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ apps/ (your main.cpp lives here) │
|
||||
│ secs_server, secs_client, secs_conformance, secs_bench, │
|
||||
│ fuzz_*, secs_interop_probe │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ gem::Router + gem::EquipmentDataModel │
|
||||
│ ───────────────────────────────────────── │
|
||||
│ Router: (stream, function) → handler dispatch table │
|
||||
│ Model: composes every store + every FSM into one object │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ Per-domain stores (include/secsgem/gem/store/) │
|
||||
│ alarms, carriers, ceid+reports, exceptions, host_commands, │
|
||||
│ limits, modules, process_jobs, control_jobs, recipes, spool, │
|
||||
│ substrates, svid+dvid, trace, cem_objects, e84_ports, clock │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ Per-standard state machines │
|
||||
│ E30 control_state, E30 communication_state, E40 PJ, │
|
||||
│ E94 CJ, E87 carriers + load_ports, E90 substrates, │
|
||||
│ E116 EPT, E157 modules, E5 exceptions, E84 handshake │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ hsms::Connection (Asio) + secsi::Protocol + secs2 codec │
|
||||
│ ─────────────────────────────────────────────────────── │
|
||||
│ Transport: HSMS-SS, HSMS-GS, SECS-I (FSM only) │
|
||||
│ Codec: Item ⇄ bytes, Item ⇄ SML text │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Each layer is replaceable. The codec doesn't know about the FSMs;
|
||||
the FSMs don't know about the codec; the Router doesn't know about
|
||||
persistence. The model composes them but doesn't own their logic.
|
||||
|
||||
---
|
||||
|
||||
## 3. The codec (`secs2/`)
|
||||
|
||||
`secs2::Item` is a tagged variant over the SEMI E5 §9 formats: List,
|
||||
Binary, Boolean, ASCII, JIS-8, C2, U1-U8, I1-I8, F4, F8. Storage is
|
||||
a `std::variant` matching each format's natural C++ type.
|
||||
|
||||
```
|
||||
secs2::encode(item) → vector<uint8_t> // bytes for the wire
|
||||
secs2::decode(bytes) → Item // wire → object
|
||||
secs2::to_sml(item) → string // human-readable
|
||||
secs2::from_sml(text) → Item // and back
|
||||
```
|
||||
|
||||
The encoder emits the format-byte arithmetic described in
|
||||
[GLOSSARY.md → SEMI E5 §9](GLOSSARY.md). The decoder is strict
|
||||
about format codes but lenient about U-widths in identifier fields
|
||||
(per `messages_helpers::any_unsigned_first`) — that's how secsgem-py
|
||||
interop works without breaking spec-correctness.
|
||||
|
||||
The codec is the most-tested layer in the codebase: 196 SEMI E5 KAT
|
||||
assertions, 120+ unit tests, plus libFuzzer with 70 000+ random
|
||||
inputs per minute. Touch it carefully; it's the foundation
|
||||
everything else stands on.
|
||||
|
||||
## 4. Transport (`hsms/`, `secsi/`)
|
||||
|
||||
`hsms::Connection` owns one TCP socket and one (SS) or many (GS)
|
||||
session-state objects. Frames have a 4-byte length prefix + 10-byte
|
||||
header (session_id, byte2, byte3, PType, SType, system_bytes) +
|
||||
optional SECS-II body.
|
||||
|
||||
State transitions: NOT-CONNECTED → NOT-SELECTED (T7 armed) → SELECTED.
|
||||
Either side can initiate Select.req; both modes (Active / Passive)
|
||||
are first-class.
|
||||
|
||||
The connection class is **I/O-aware**: it owns the asio socket, arms
|
||||
the T-timers, drives the read loop. Everything above it is I/O-free
|
||||
and reachable through callbacks:
|
||||
|
||||
```cpp
|
||||
conn->set_message_handler([&router](const secs2::Message& m) {
|
||||
return router.dispatch_with_s9(/*emit=*/..., /*mhead=*/..., m);
|
||||
});
|
||||
```
|
||||
|
||||
SECS-I (`secsi::Protocol`) is an FSM-only port of the same idea —
|
||||
serial-line framing, T1/T2/T3/T4 timers as callbacks. No asio
|
||||
inside the FSM; the application drives the clock. The E84 timers
|
||||
follow the same pattern (`E84AsioTimers` is the asio adapter; the
|
||||
FSM stays pure).
|
||||
|
||||
## 5. The model (`gem/`)
|
||||
|
||||
`gem::EquipmentDataModel` (data_model.hpp) is a struct composing
|
||||
every store:
|
||||
|
||||
```cpp
|
||||
struct EquipmentDataModel {
|
||||
StatusVariableStore svids;
|
||||
DataVariableStore dvids;
|
||||
EquipmentConstantStore ecids;
|
||||
EventReportSubscriptions events;
|
||||
AlarmRegistry alarms;
|
||||
RecipeStore recipes;
|
||||
Clock clock;
|
||||
HostCommandRegistry commands;
|
||||
SpoolStore spool;
|
||||
LimitMonitorStore limits;
|
||||
TraceStore traces;
|
||||
ProcessJobStore process_jobs;
|
||||
ControlJobStore control_jobs;
|
||||
ExceptionStore exceptions;
|
||||
CarrierStore carriers;
|
||||
LoadPortStore load_ports;
|
||||
SubstrateStore substrates;
|
||||
EptStateMachine ept;
|
||||
CemObjectStore cem;
|
||||
ModuleStore modules;
|
||||
E84PortStore e84_ports;
|
||||
};
|
||||
```
|
||||
|
||||
No locks. Single-threaded contract documented in INTEGRATION.md §3.
|
||||
All mutation runs on the io_context strand.
|
||||
|
||||
Each store is **independently usable** — you can `#include
|
||||
"secsgem/gem/store/alarms.hpp"` and use `AlarmRegistry` without
|
||||
pulling in any of the others. The composite is for convenience.
|
||||
|
||||
### Per-store pattern
|
||||
|
||||
Every store follows the same shape:
|
||||
|
||||
```cpp
|
||||
class FooStore {
|
||||
public:
|
||||
// CRUD
|
||||
bool create(...);
|
||||
Foo* get(id); // mutable pointer, nullable
|
||||
const Foo* get(id) const; // const-mutable pointer, nullable
|
||||
bool has(id) const;
|
||||
bool remove(id);
|
||||
std::size_t size() const;
|
||||
std::vector<Foo> all() const;
|
||||
|
||||
// Domain operations
|
||||
void fire_internal(id, FooEvent event); // application-driven
|
||||
Ack on_host_command(id, FooEvent event); // host-driven
|
||||
|
||||
// Observers
|
||||
void set_state_change_handler(StateChangeHandler);
|
||||
|
||||
// Persistence
|
||||
void enable_persistence(std::filesystem::path dir);
|
||||
};
|
||||
```
|
||||
|
||||
The store owns the FSM instance, the persistence file path, the
|
||||
in-memory state. The FSM owns the legal-transition table. The
|
||||
table comes from a YAML file (loaded into `factory_()` at
|
||||
construction).
|
||||
|
||||
## 6. The Router
|
||||
|
||||
`gem::Router` (router.hpp) is a tiny dispatch table:
|
||||
|
||||
```cpp
|
||||
Router r;
|
||||
r.on(1, 13, [&](const secs2::Message&) {
|
||||
return gem::s1f14_establish_comms_ack(...);
|
||||
});
|
||||
r.on(2, 41, [&](const secs2::Message& msg) {
|
||||
auto cmd = gem::parse_s2f41(msg);
|
||||
// ... handle command ...
|
||||
return gem::s2f42_host_command_ack(...);
|
||||
});
|
||||
|
||||
auto reply = r.dispatch(incoming_message);
|
||||
```
|
||||
|
||||
Handlers are `std::function<std::optional<Message>(const Message&)>`.
|
||||
Return nullopt for one-way (W=0) primaries.
|
||||
|
||||
`dispatch_with_s9` wraps `dispatch` to auto-emit `S9F3` (unrecognized
|
||||
stream) or `S9F5` (unrecognized function) when no handler is
|
||||
registered — the spec-mandated response.
|
||||
|
||||
The Router is **stateless** — it just looks up handlers in a
|
||||
`std::map<std::pair<uint8_t, uint8_t>, Handler>`. All state lives
|
||||
in the model the handlers close over.
|
||||
|
||||
## 7. Persistence
|
||||
|
||||
Every persistable store ships a `.tmp + atomic rename` writer + a
|
||||
versioned record format:
|
||||
|
||||
```
|
||||
[u8 magic]
|
||||
[u8 version] // 1..kVersion accepted on load
|
||||
[u8 state]
|
||||
... domain-specific fields ...
|
||||
```
|
||||
|
||||
`enable_persistence(dir)` scans the dir on startup, replays records
|
||||
into in-memory state via `install_()`, and from there writes on every
|
||||
mutation. See README "Schema migrations" for the version-bump
|
||||
discipline.
|
||||
|
||||
The seven persistable stores (PJ, CJ, Carrier, LoadPort, Substrate,
|
||||
Exception, Spool) all follow the same pattern. Adding persistence to
|
||||
a new store is a paste-and-adapt: copy `control_jobs.hpp`'s
|
||||
`write_record_` + `load_record_` + `enable_persistence`, change the
|
||||
magic byte + the fields.
|
||||
|
||||
Magic bytes claimed so far (don't reuse):
|
||||
|
||||
| Magic | Store |
|
||||
|-------|--------------------|
|
||||
| 0xC4 | CarrierStore |
|
||||
| 0xC5 | LoadPortStore |
|
||||
| 0xC6 | SubstrateStore |
|
||||
| 0xC7 | ProcessJobStore |
|
||||
| 0xC8 | ControlJobStore |
|
||||
| 0xC9 | ExceptionStore |
|
||||
| 0xE5 | SpoolStore |
|
||||
|
||||
---
|
||||
|
||||
## 8. Codegen pipeline
|
||||
|
||||
`tools/gen_messages.py` reads `data/messages.yaml` and emits
|
||||
`build/generated/secsgem/gem/messages.hpp`. The pipeline:
|
||||
|
||||
```
|
||||
messages.yaml
|
||||
│
|
||||
│ (CMake add_custom_command, runs on rebuild if YAML newer)
|
||||
▼
|
||||
tools/gen_messages.py
|
||||
│
|
||||
│ (Python reads YAML, emits typed C++ structs + builders + parsers)
|
||||
▼
|
||||
build/generated/secsgem/gem/messages.hpp
|
||||
│
|
||||
│ (#included by apps/, src/, tests/)
|
||||
▼
|
||||
secs_server.cpp / secs_client.cpp / your main.cpp
|
||||
```
|
||||
|
||||
For each message in the catalog the codegen emits:
|
||||
|
||||
- An optional `struct Name { ... }` (for list bodies)
|
||||
- A `inline secs2::Message builder_name(args...)` that returns a
|
||||
ready-to-send Message
|
||||
- A `inline std::optional<...> parse_name(const secs2::Message&)`
|
||||
that returns the parsed body or nullopt
|
||||
|
||||
The YAML shape is documented in the file header of `messages.yaml`.
|
||||
Every supported body kind (`scalar`, `list`, `list_of`) maps to a
|
||||
straightforward C++ shape.
|
||||
|
||||
---
|
||||
|
||||
## 9. Extending the library
|
||||
|
||||
### 9.1. New SECS-II message
|
||||
|
||||
Edit `data/messages.yaml`:
|
||||
|
||||
```yaml
|
||||
- id: S6F30
|
||||
stream: 6
|
||||
function: 30
|
||||
w: true
|
||||
builder: s6f30_my_request
|
||||
parser: parse_s6f30
|
||||
body:
|
||||
kind: list
|
||||
struct_name: MyRequest
|
||||
fields:
|
||||
- {name: dataid, shape: {kind: scalar, item_type: U4}}
|
||||
- {name: payload, shape: {kind: scalar, item_type: ASCII}}
|
||||
```
|
||||
|
||||
Rebuild — `messages.hpp` regenerates. Register a handler:
|
||||
|
||||
```cpp
|
||||
router.on(6, 30, [&](const secs2::Message& m) {
|
||||
auto req = gem::parse_s6f30(m);
|
||||
if (!req) return std::optional{secs2::Message(6, 0, false)}; // bad body
|
||||
// ...
|
||||
return std::optional{secs2::Message(6, 0, false)}; // W=0 reply
|
||||
});
|
||||
```
|
||||
|
||||
That's the entire diff. No core code change.
|
||||
|
||||
### 9.2. New state machine
|
||||
|
||||
If your tool has a domain not covered by the existing stores
|
||||
(say, an in-chamber gas-flow FSM):
|
||||
|
||||
1. Define the states + events:
|
||||
```cpp
|
||||
// include/secsgem/gem/gas_flow.hpp
|
||||
enum class GasFlowState : uint8_t { Idle, Purging, Stable, Faulted };
|
||||
enum class GasFlowEvent : uint8_t { StartPurge, FlowStable, Fault, Reset };
|
||||
```
|
||||
2. Define the transition table — pure data:
|
||||
```cpp
|
||||
struct GasFlowTransition { GasFlowState from; GasFlowEvent on; std::optional<GasFlowState> to; };
|
||||
class GasFlowTransitionTable { /* mirrors ProcessJobTransitionTable */ };
|
||||
```
|
||||
3. Define the FSM:
|
||||
```cpp
|
||||
class GasFlowStateMachine {
|
||||
public:
|
||||
bool fire(GasFlowEvent ev); // returns whether a transition happened
|
||||
GasFlowState state() const;
|
||||
void set_state_change_handler(StateChangeHandler);
|
||||
};
|
||||
```
|
||||
4. (Optional) Define a store if there can be many instances:
|
||||
`class GasFlowStore { /* mirrors ProcessJobStore */ }` with
|
||||
create/get/has/all + state-change relay.
|
||||
5. (Optional) YAML-load the transitions following `config::load_*` patterns.
|
||||
6. (Optional) Persistence: copy a store's `enable_persistence` + `write_record_` + `load_record_`.
|
||||
|
||||
Reference patterns to lift from: `ept_state.hpp` (single global FSM),
|
||||
`process_job_state.hpp` (per-instance FSM in a store).
|
||||
|
||||
### 9.3. New store
|
||||
|
||||
Stores follow the consistent API shape in §5. Copy
|
||||
`include/secsgem/gem/store/alarms.hpp` (smallest example) or
|
||||
`include/secsgem/gem/store/process_jobs.hpp` (richest example,
|
||||
includes persistence).
|
||||
|
||||
Wire into `EquipmentDataModel` if it should be globally accessible
|
||||
from `model->...`:
|
||||
|
||||
```cpp
|
||||
// data_model.hpp
|
||||
#include "secsgem/gem/store/gas_flows.hpp"
|
||||
|
||||
struct EquipmentDataModel {
|
||||
// ... existing fields ...
|
||||
GasFlowStore gas_flows;
|
||||
};
|
||||
```
|
||||
|
||||
### 9.4. New persistence backend
|
||||
|
||||
The seven existing stores all journal to files. If you want
|
||||
database-backed persistence (SQLite, Postgres, etcd), the cleanest
|
||||
pattern is to subclass-or-replace the `enable_persistence(path)`
|
||||
method:
|
||||
|
||||
```cpp
|
||||
// Or: a sibling enable_db_persistence(connection_string)
|
||||
void enable_db_persistence(std::string conn) {
|
||||
db_conn_ = std::move(conn);
|
||||
/* on each create / mutation, write the record to the DB */
|
||||
}
|
||||
```
|
||||
|
||||
The contract is consistent with file persistence: load at startup,
|
||||
write on mutation, atomic-rename equivalent (a transaction). See
|
||||
`spool.hpp::enable_persistence` for the cleanest single-file
|
||||
example to mirror.
|
||||
|
||||
### 9.5. New transport
|
||||
|
||||
`hsms::Connection` and `secsi::Protocol` are the two we ship. A
|
||||
third (e.g. HSMS-over-TLS as a first-class thing, or HSMS over a
|
||||
sidecar IPC) follows the same contract:
|
||||
|
||||
1. Accept a transport socket / endpoint.
|
||||
2. Expose `set_message_handler(...)`, `send_request(...)`,
|
||||
`send_data(...)`, `set_selected_handler(...)`,
|
||||
`set_closed_handler(...)`.
|
||||
3. Drive the SECS-II codec via `secs2::encode` / `secs2::decode`.
|
||||
|
||||
The Router and the model don't care which transport produced the
|
||||
message. Both wire into the same `set_message_handler` callback
|
||||
shape.
|
||||
|
||||
---
|
||||
|
||||
## 10. Threading model
|
||||
|
||||
Single-threaded by design. The entire model — every store, every
|
||||
FSM, the Router, the Connection — is reachable only from the
|
||||
io_context that drives the HSMS connection. No locks anywhere.
|
||||
|
||||
This is documented as a contract in INTEGRATION.md §3 and exercised
|
||||
by:
|
||||
|
||||
- `test_thread_safety.cpp` — N producer threads asio::post updates
|
||||
onto the worker io
|
||||
- `test_concurrency.cpp` — in-flight transaction interleaving
|
||||
- The ThreadSanitizer CI lane — every test under
|
||||
`-fsanitize=thread`
|
||||
|
||||
If you're adding work that lives on another thread (sensor poll
|
||||
loop, separate metrics scraper, signal handler), marshal back to
|
||||
the io_context with `asio::post(io.get_executor(), ...)`. Don't
|
||||
add locks; they'll diverge from the contract and the next
|
||||
contributor will be confused.
|
||||
|
||||
---
|
||||
|
||||
## 11. Why C++20
|
||||
|
||||
- `std::variant` for `Item` storage — exhaustive `std::visit`
|
||||
catches new format codes at compile time.
|
||||
- `std::optional` everywhere — the codec, the parsers, the
|
||||
store accessors all use it as the "missing value" idiom.
|
||||
- Designated initializers in tests — readability.
|
||||
- Concepts in template helpers (`messages_helpers.hpp`).
|
||||
- `<filesystem>` — persistence wouldn't be a header-only feature
|
||||
without it.
|
||||
|
||||
`g++-13` and `clang-18` both build the codebase clean at
|
||||
`-Wall -Wextra -Wpedantic`.
|
||||
|
||||
---
|
||||
|
||||
## 12. Where to look in the source
|
||||
|
||||
| You want to understand… | Read these in order |
|
||||
|-------------------------------------|----------------------------------------------------------------|
|
||||
| The wire byte layout | `secs2/item.hpp`, `secs2/codec.cpp`, `tests/test_e5_kat.cpp` |
|
||||
| How a typed message is built | `data/messages.yaml`, `tools/gen_messages.py`, the generated header |
|
||||
| How HSMS handshakes | `hsms/connection.hpp/.cpp`, `tests/test_hsms_*.cpp` |
|
||||
| How the Router dispatches | `gem/router.hpp` |
|
||||
| How a store implements persistence | `gem/store/spool.hpp` (smallest), `gem/store/process_jobs.hpp` (richest) |
|
||||
| How an FSM is structured | `gem/process_job_state.hpp`, `src/gem/process_job_state.cpp` |
|
||||
| How the application wires it all | `apps/secs_server.cpp` (the canonical example, ~1200 lines) |
|
||||
| How a customer would write main() | `examples/pvd_tool/main.cpp` (the worked vendor example) |
|
||||
| How thread-safety works | `tests/test_thread_safety.cpp`, INTEGRATION.md §3 |
|
||||
| How E84 timers integrate with asio | `gem/e84_asio_timers.hpp` (the canonical I/O-adapter pattern) |
|
||||
| How the property fuzz drives state | `tests/test_robustness_fuzz.cpp` |
|
||||
|
||||
---
|
||||
|
||||
## 13. What we deliberately don't do
|
||||
|
||||
- **No DI framework, no service locator.** Stores are owned by the
|
||||
model; the model is owned by your application; everything else is
|
||||
passed in by reference. C++20 has no language-level DI, and adding
|
||||
one to a codebase this size is overhead with no payoff.
|
||||
- **No singleton state.** The model is a value, not a global.
|
||||
- **No std::shared_ptr-everywhere.** asio handlers extend the
|
||||
lifetimes that need extending; the rest is owned by-value. Read
|
||||
`Connection`'s lifetime contract in `hsms/connection.hpp` if you're
|
||||
ever in doubt.
|
||||
- **No exceptions across the API boundary** — the codec throws
|
||||
`secs2::CodecError` internally, but every public accessor returns
|
||||
`std::optional` or returns a bool. Exceptions are reserved for
|
||||
programmer-error / corrupt-input paths.
|
||||
|
||||
Every one of those constraints came from real review pressure on
|
||||
prior iterations. Pushing back on them is welcome but please read
|
||||
the existing tests first; the codebase's architecture is what makes
|
||||
the property fuzz and the TSan lane feasible.
|
||||
@@ -0,0 +1,69 @@
|
||||
# Performance baseline
|
||||
|
||||
Numbers from `build/secs_bench --requests 20000 --concurrency 16` on
|
||||
Docker / Ubuntu 24.04 inside Docker Desktop on macOS (M-series), single
|
||||
io_context thread. Treat as **rough envelope for capacity planning**,
|
||||
not lab-grade benchmarks; re-run on your target hardware before
|
||||
sizing pods or VMs.
|
||||
|
||||
## Round-trip throughput / latency
|
||||
|
||||
| Scenario | Ops | Elapsed | Ops/sec | p50 µs | p95 µs | p99 µs |
|
||||
|----------------------------------|--------:|--------:|-----------:|--------:|--------:|--------:|
|
||||
| S1F1/F2 (header-only) | 20000 | 0.14 | ~140000 | 74 | 103 | 161 |
|
||||
| S1F3/F4 (32 SVIDs) | 20000 | 0.25 | ~79000 | 165 | 186 | 260 |
|
||||
| S6F11 push (W=0) | 20000 | 0.03 | ~572000 | n/a | n/a | n/a |
|
||||
|
||||
**Read the table this way.** A real fab tool needs to handle tens to a
|
||||
few hundred S6F11 events/second sustained. We're three orders of
|
||||
magnitude above that on the push path, two orders above on synchronous
|
||||
round-trips. Throughput is not the bottleneck; latency tail under
|
||||
contention is.
|
||||
|
||||
## Memory footprint
|
||||
|
||||
A `ProcessJob` + `ControlJob` pair (no persistence enabled) is around
|
||||
**~450 bytes** of heap (1000 pairs ≈ 0.45 MiB, measured on a fresh
|
||||
process). With persistence enabled add ~200 bytes of in-memory journal
|
||||
path tracking per record.
|
||||
|
||||
| Active entity | Approx bytes / instance |
|
||||
|----------------------|------------------------:|
|
||||
| PJ + CJ pair | ~450 |
|
||||
| Carrier (no slots) | ~80 |
|
||||
| Carrier slot | ~24 |
|
||||
| Substrate | ~120 |
|
||||
| Spool entry | ~40 + encoded body size |
|
||||
|
||||
A busy fab tool tracking 50 carriers × 25 slots, 200 substrates, 20
|
||||
active PJ+CJ pairs comes in well under 1 MiB of model state. RSS will
|
||||
be dominated by the binary itself + asio's buffers (~10-20 MiB),
|
||||
not the model.
|
||||
|
||||
## How to re-run
|
||||
|
||||
```sh
|
||||
docker compose run --rm builder /app/build/secs_bench \
|
||||
--requests 50000 \
|
||||
--concurrency 32 \
|
||||
--svid-count 32 \
|
||||
--store-pairs 10000
|
||||
```
|
||||
|
||||
Output is markdown — pipe to a file and commit it to your CI so
|
||||
regressions show up as diffs.
|
||||
|
||||
## What this does NOT measure
|
||||
|
||||
- **Real network**. Loopback TCP has no MTU fragmentation, no
|
||||
retransmits, no jitter. Production HSMS over a fab control LAN will
|
||||
see higher tail latency.
|
||||
- **Persistence write amplification**. The bench runs with persistence
|
||||
disabled. Each store mutation with persistence enabled is one
|
||||
atomic-rename to disk; on rotational media that limits you to a few
|
||||
hundred mutations/sec. SSD-backed deployments are fine.
|
||||
- **Concurrent S6F11 enable filtering**. Real CEID emission gates on
|
||||
the host's enable/disable list — this bench fires raw S6F11s.
|
||||
- **Multi-session HSMS-GS** dispatch overhead — single-session only.
|
||||
- **TLS-tunneled sockets** (via stunnel/sidecar) — these add ~50 µs
|
||||
per round-trip on modern hardware.
|
||||
@@ -0,0 +1,456 @@
|
||||
# SECS/GEM Compliance
|
||||
|
||||
A per-capability accounting against the foundational SEMI standards
|
||||
**E5 (SECS-II)**, **E30 (GEM)**, **E37 (HSMS, SS + GS)**, **E4 (SECS-I)**,
|
||||
plus the full GEM 300 stack: **E40** (process jobs), **E42** (formatted
|
||||
process programs), **E94** (control jobs), **E87** (carriers), **E90**
|
||||
(substrates), **E116** (equipment performance tracking), **E120** (common
|
||||
equipment model), **E157** (module process tracking), **E84** (parallel
|
||||
I/O), **E148** (time synchronization), **E39** (object services), plus
|
||||
**E5 §13** wafer maps.
|
||||
|
||||
> **Status.** Every GEM Fundamental and every GEM Additional capability
|
||||
> that E30 binds to a concrete SECS-II message set is implemented, and
|
||||
> every GEM 300 standard the project sets out to cover is implemented
|
||||
> end-to-end (state machines + stores + wire messages + dispatch). See
|
||||
> §8 for what "100% GEM-compliant" can and cannot honestly mean about a
|
||||
> codebase, and the README "Deferred follow-ups" section for the
|
||||
> non-shipped pieces that aren't behavioural gaps.
|
||||
|
||||
Legend:
|
||||
|
||||
- ✅ **Full** — implemented to the spec; round-trip-tested.
|
||||
- 🟡 **Partial** — implemented in the demo path with a documented limitation.
|
||||
- ⬜ **Out of scope** — deliberately not implemented; reason given.
|
||||
|
||||
---
|
||||
|
||||
## 1. E37 — HSMS transport
|
||||
|
||||
| Item | Status | Spec ref | Notes |
|
||||
|---------------------------------------|--------|----------|-------|
|
||||
| TCP transport | ✅ | E37 §6 | `hsms::Connection` over standalone Asio. |
|
||||
| 4-byte length prefix + 10-byte header | ✅ | E37 §8.2 | `hsms::Frame::encode/decode`. |
|
||||
| Session ID, byte2, byte3, PType, SType, system-bytes | ✅ | E37 §8.3 | `hsms::Header`. |
|
||||
| `Select.req / .rsp` | ✅ | E37 §7.2 | `SType` 1/2; SelectStatus enum (0–3). |
|
||||
| `Deselect.req / .rsp` | ✅ | E37 §7.4 | `SType` 3/4; DeselectStatus enum (0–2). |
|
||||
| `Linktest.req / .rsp` | ✅ | E37 §7.5 | `SType` 5/6; periodic interval configurable. |
|
||||
| `Separate.req` | ✅ | E37 §7.6 | `SType` 9; graceful close after flush. |
|
||||
| `Reject.req` | ✅ | E37 §7.7 | Emitted on data-while-NOT-SELECTED. |
|
||||
| Connection state machine NOT-CONNECTED → NOT-SELECTED → SELECTED | ✅ | E37 §6.3 | Both Active and Passive modes. |
|
||||
| T3 reply timeout | ✅ | E37 §10 | Per-transaction `steady_timer`. |
|
||||
| T5 connect separation timeout | ✅ | E37 §10 | `Client::schedule_retry`. |
|
||||
| T6 control transaction timeout | ✅ | E37 §10 | One concurrent control transaction. |
|
||||
| T7 not-selected timeout (passive) | ✅ | E37 §10 | Armed on connect / on Deselect.req. |
|
||||
| T8 intercharacter timeout | ✅ | E37 §10 | Bounds the payload read after length prefix. |
|
||||
| HSMS-SS (single-session) | ✅ | E37 §11 | Default mode: the constructor registers a single session. |
|
||||
| HSMS-GS (general-session) | ✅ | E37 §11 | `Connection::add_session(device_id)` registers extra sessions; per-session SELECTED state + message handlers; Select.req carries session_id=device_id in GS mode. |
|
||||
|
||||
---
|
||||
|
||||
## 1a. E4 — SECS-I transport (block protocol)
|
||||
|
||||
| Item | Status | Spec ref | Notes |
|
||||
|---------------------------------------|--------|----------|-------|
|
||||
| 10-byte block header (R/W/E bits, system bytes) | ✅ | E4 §6.2 | `secsi::Header` with bit-precise pack/unpack. |
|
||||
| Length-prefixed block + 2-byte checksum | ✅ | E4 §6.1, §6.3 | `secsi::Block::encode/decode`. |
|
||||
| Multi-block message split / assemble | ✅ | E4 §7.2.3 | `split_message` / `assemble_message`; E-bit only on the final block. |
|
||||
| ENQ/EOT/ACK/NAK handshake | ✅ | E4 §7.1 | `secsi::Protocol` half-duplex FSM. |
|
||||
| RTY retry counter | ✅ | E4 §10.2 | Per-block retry budget, exhaust → ActionRaiseError. |
|
||||
| T1 inter-character timer hook | ✅ | E4 §10.1 | Drained in `RecvBlock`; host wires the actual asio timer. |
|
||||
| T2 protocol timer hook | ✅ | E4 §10.1 | Triggers a retry from any send state. |
|
||||
| T3 reply timer | ✅ | E4 §10.1 | FSM tracks system_bytes of outstanding W=1 primaries; arms T3 on send-complete, cancels on matching reply, aborts on expiry. |
|
||||
| T4 inter-block timer | ✅ | E4 §10.1 | FSM arms T4 when a block delivers with end_block=false; cancels when the next block arrives, aborts on expiry. |
|
||||
| Master/slave contention resolution | ✅ | E4 §7.1.4 | Slave yields on simultaneous ENQ; master holds. |
|
||||
| Serial port wiring (asio) | ⬜ | — | FSM is IO-free; serial integration is a wiring follow-up. |
|
||||
| TCP tunnel for testing | ✅ | — | `secsi::TcpTransport` wraps the FSM behind an asio TCP socket; mirrors secsgem-py's `secsitcp/`. |
|
||||
|
||||
---
|
||||
|
||||
## 2. E5 — SECS-II encoding
|
||||
|
||||
| Item | Status | Spec ref | Notes |
|
||||
|---------------------------------------|--------|----------|-------|
|
||||
| Format byte + 1/2/3 length bytes | ✅ | E5 §9 | `secs2::encode_into`. |
|
||||
| List (`L`) | ✅ | E5 §9.3 | Recursive. |
|
||||
| ASCII (`A`) | ✅ | E5 §9.5 | |
|
||||
| Binary (`B`) | ✅ | E5 §9.5 | |
|
||||
| Boolean (`BOOLEAN`) | ✅ | E5 §9.5 | |
|
||||
| `U1, U2, U4, U8` (big-endian) | ✅ | E5 §9.5 | Identifier parsers accept any width per the SEMI wildcard rule. |
|
||||
| `I1, I2, I4, I8` (big-endian, two's complement) | ✅ | E5 §9.5 | Same lenient-width policy. |
|
||||
| `F4, F8` (IEEE 754 big-endian) | ✅ | E5 §9.5 | bit-cast round-trip. |
|
||||
| JIS-8 (single-byte JIS text) | ✅ | E5 §9.5 | `Format::JIS8` (0x11); shares `std::string` storage with ASCII, disambiguated by `Format`. |
|
||||
| C2 (Unicode 2-byte code points) | ✅ | E5 §9.5 | `Format::C2` (0x12); big-endian uint16_t code points. |
|
||||
| SML text rendering | ✅ | E5 Annex | `secs2::to_sml`. JIS-8 prints as `<J "...">`, C2 as `<C 65 66 ...>`. |
|
||||
| SML parser (inverse of `to_sml`) | ✅ | — | `secs2::from_sml`; round-trips every format. |
|
||||
| `ASCII | Binary` wildcard fields | ✅ | E5 | `BINARY_OR_ASCII` schema type for PPBODY etc.; accepted by `as_text_or_binary`. |
|
||||
|
||||
---
|
||||
|
||||
## 3. E30 — GEM Fundamental capabilities (§5.2)
|
||||
|
||||
| Fundamental Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| State models | ✅ | E30 §6.2 | — | E30 control state machine (5 states) + HSMS communication state machine. |
|
||||
| Equipment Processing States | ✅ | E30 §6.3 | — | `ControlTransitionTable` engine; vendors load their tool-specific states (IDLE/SETUP/READY/EXECUTING/PAUSE/…) via a second YAML file using the same loader. The spec leaves the concrete states tool-specific. |
|
||||
| Host-Initiated S1F13/F14 scenario | ✅ | E30 §6.5 | S1F13/F14 | |
|
||||
| Event Notification | ✅ | E30 §6.6 | S6F11/F12 | Equipment-initiated, host-acknowledged. |
|
||||
| On-Line Identification | ✅ | E30 §6.7 | S1F1/F2 | MDLN + SOFTREV. |
|
||||
| Error Messages | ✅ | E30 §6.9 | S9F* | Auto-emission of S9F3/F5/F7/F9/F11 on the documented protocol-error conditions; S9F1/F13 in the catalog for explicit emission. |
|
||||
| Documentation | ✅ | E30 §6.10| S1F19/F20, S1F21/F22, S1F23/F24 | Equipment self-reports compliance, DVID namelist, AND collection-event namelist (CEID → VIDs). |
|
||||
| Control (Operator-Initiated) | ✅ | E30 §6.2 | — | `ControlStateMachine::operator_online/offline/local/remote`. |
|
||||
|
||||
---
|
||||
|
||||
## 4. E30 — GEM Additional capabilities (§5.3)
|
||||
|
||||
| Additional Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| Establish Communications | ✅ | E30 §6.5 | S1F13/F14 | Both directions modeled; COMMACK enum. Backed by the E30 §6.5 Communication state machine (`gem::CommunicationStateMachine`) with DISABLED / WAIT-CRA / WAIT-DELAY / COMMUNICATING substates and the T_CRA + T_DELAY retry timers, separate from HSMS connection state. |
|
||||
| Dynamic Event Report Configuration | ✅ | E30 §6.6 | S2F33/F34, S2F35/F36, S2F37/F38 | Full Define-Report / Link-Event / Enable-Event pipeline with all four ack enums. Host-initiated readback via S6F15/F16, S6F19/F20, S6F21/F22. |
|
||||
| Variable Data Collection | ✅ | E30 §6.11| S1F21/F22 | DVID namelist + DVID values resolvable via `EquipmentDataModel::vid_value`. |
|
||||
| Trace Data Collection | ✅ | E30 §6.12| S2F23/F24, S6F1/F2 | `TraceStore` keeps active TRID→TraceConfig; periodic sampling left to the application's scheduler. |
|
||||
| Status Data Collection | ✅ | E30 §6.13| S1F3/F4, S1F11/F12 | |
|
||||
| Alarm Management | ✅ | E30 §6.14| S5F1/F2, S5F3/F4, S5F5/F6, S5F7/F8 | Full set. ALCD bit-7 set/cleared, lower-7 category. |
|
||||
| Remote Control | ✅ | E30 §6.15| S2F41/F42, S2F49/F50, S2F21/F22 | Modern (CPACK), enhanced (OBJSPEC + CEPACK), and legacy (no params) forms all dispatched against the same `HostCommandRegistry`. |
|
||||
| Equipment Constants | ✅ | E30 §6.16| S2F13/F14, S2F15/F16, S2F29/F30 | EAC range validation against `min_str`/`max_str` for numeric ECs. |
|
||||
| Process Program Management | ✅ | E30 §6.17| S7F1/F2, S7F3/F4, S7F5/F6, S7F17/F18, S7F19/F20, S7F23/F24, S7F25/F26 | Unformatted PP load-inquire/send/request/delete/list + E42 enhanced (formatted) PP send/request via S7F23-F26. RecipeStore carries both views simultaneously. |
|
||||
| Material Movement | ✅ | E30 §6.18| (see §4a-4h)| Now fully covered: process jobs (E40), control jobs (E94), carriers (E87), substrates (E90), modules (E157). |
|
||||
| Equipment Terminal Services | ✅ | E30 §6.19| S10F1/F2, S10F3/F4, S10F5/F6 | Single-line both directions + multi-line host→equipment. S10F7 broadcast intentionally omitted (rarely used). |
|
||||
| Clock | ✅ | E30 §6.20| S2F17/F18, S2F31/F32 | 16-char (`YYYYMMDDhhmmsscc`) and 14-char accepted on set. Drift tracking + quality via E148 (§4g). |
|
||||
| Limits Monitoring | ✅ | E30 §6.21| S2F45/F46, S2F47/F48 | `LimitMonitorStore` keyed by VID with multiple `LimitDefinition` (LIMITID + upper/lower as arbitrary Items). |
|
||||
| Spooling | ✅ | E30 §6.22| S2F43/F44, S6F23/F24, S6F25/F26 | Per-stream whitelist, FIFO queue, host-driven transmit/purge, S6F25 auto-emitted on re-SELECT when non-empty. **Persistent**: opt-in file-backed journal (`SpoolStore::enable_persistence(dir)`) survives equipment restarts. |
|
||||
| Control | ✅ | E30 §6.2 | — | See Fundamental. |
|
||||
|
||||
---
|
||||
|
||||
## 4a. E40 Process Jobs
|
||||
|
||||
| Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| PJ state model | ✅ | E40 §6.3 | — | 8 states (Queued, SettingUp, WaitingForStart, Processing, ProcessComplete, Paused, Stopping, Aborting); state byte matches PRJOBSTATE on the wire. |
|
||||
| PRJobCreate (full body) | ✅ | E40 §10.2| S16F11/F12 | Full E40-0705 body: `<L,5 PRJOBID MF PRRECIPEMETHOD <L,2 PPID <L RCPVARLIST>> <L MTRLOUTSPEC> <L PRPROCESSPARAMS>>`. PPID validated against `RecipeStore`. |
|
||||
| PRJobCreateMultiple | ✅ | E40 §10 | S16F15/F16 | Bulk variant; per-job ACK list. |
|
||||
| PRJobDequeue | ✅ | E40 §10.2| S16F13/F14 | Only legal while PJ is QUEUED. |
|
||||
| PRJobMonitor | ✅ | E40 §10 | S16F7/F8 | Per-PJ alert enable/disable. |
|
||||
| PRJobCommand | ✅ | E40 §10.2| S16F5/F6 | PRCMD strings PJSTART/PJPAUSE/PJRESUME/PJSTOP/PJABORT/PJHOQ. |
|
||||
| PRJobAlert | ✅ | E40 §10.3| S16F9 | Equipment-initiated one-way (W=0). Fires automatically on every PJ state transition. |
|
||||
|
||||
## 4b. E94 Control Jobs
|
||||
|
||||
| Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| CJ state model | ✅ | E94 §6 | — | 9 states; CJ owns an ordered `prjobids` list. |
|
||||
| CreateObject (CJ) | ✅ | E94 §6.4 | S14F9/F10 | Body `<L,2 CTLJOBID L,n PRJOBIDs>`. |
|
||||
| DeleteObject (CJ) | ✅ | E94 §6.4 | S14F11/F12 | |
|
||||
| CJobCommand | ✅ | E94 §6.4 | S16F27/F28 | CTLJOBCMD: CJSTART, CJPAUSE, CJRESUME, CJSTOP, CJABORT. |
|
||||
| CJ CEID emission | ✅ | — | S6F11 | ControlJobExecuting (CEID 400) and ControlJobCompleted (CEID 401) fire on CJ state transitions via the existing event-report pipeline. |
|
||||
|
||||
## 4c. E87 Carrier Management
|
||||
|
||||
| Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| Carrier state model | ✅ | E87 §6 | — | CarrierStateMachine + LoadPortStateMachine; CIDS, CSMS, CASS, LPRS, LPTS axes. |
|
||||
| CarrierAction | ✅ | E87 §10.2| S3F17/F18 | ProceedWithCarrier, CancelCarrier, BindCarrierID. |
|
||||
| Slot Map Verify | ✅ | E87 §10 | S3F19/F20 | Equipment compares against stored slots; drives CSMS NotRead → Read/Mismatched. |
|
||||
| Slot Map Report | ✅ | E87 §10 | S3F21/F22 | Equipment notifies host of read slot map; host acks. |
|
||||
| Port Group Change | ✅ | E87 §10.4| S3F23/F24 | Host modifies load-port grouping; PortGroupAck. |
|
||||
| Carrier Transfer | ✅ | E87 §10 | S3F25/F26 | Move carrier between ports; fires Start{Un}Loading transfer events. |
|
||||
| Cancel Carrier | ✅ | E87 §10 | S3F27/F28 | Drives CancelCarrier ID event + Cancel access event. |
|
||||
|
||||
## 4d. E90 Substrate Tracking
|
||||
|
||||
| Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| Substrate state model | ✅ | E90 §6 | — | Three axes: location (AtSource/AtWork/AtDestination), processing (NeedsProcessing/InProcess/Processed/Aborted/Stopped/Rejected/Lost/Skipped), id-status (Confirmed/Unconfirmed/Lost). |
|
||||
| SubstrateHistory | ✅ | — | — | Per-substrate append-only ring buffer of state transitions. |
|
||||
| Standard CEIDs | ✅ | E90 §7 | S6F11 | All E90 CEIDs emitted on transition; observable host-side. |
|
||||
|
||||
## 4e. E116 Equipment Performance Tracking
|
||||
|
||||
| Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| EPT state machine | ✅ | E116 §6 | — | NonScheduledTime / ScheduledDowntime / UnscheduledDowntime / Engineering / Standby / Productive. |
|
||||
| Time-bucket accounting | ✅ | E116 | — | Cumulative ms per state; resettable. |
|
||||
| EPT CEIDs | ✅ | E116 | S6F11 | One CEID per state. |
|
||||
|
||||
## 4f. E120 Common Equipment Model
|
||||
|
||||
| Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| Generic ObjectService (E39) | ✅ | E39 §5 | S14F1/F2, S14F3/F4 | GetAttr / SetAttr against `CemObjectStore`; OBJTYPE validation. |
|
||||
| CemObjectStore | ✅ | E120 | — | Typed objects keyed by OBJSPEC + ObjType. |
|
||||
|
||||
## 4g. E148 Time Synchronization
|
||||
|
||||
| Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| Time-sync drift tracking | ✅ | E148 | S2F31/F32 | Drift metric maintained on every set; quality score for hosts. |
|
||||
|
||||
## 4h. E157 Module Process Tracking
|
||||
|
||||
| Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| Module state machine | ✅ | E157 §6 | — | NotExecuting / GeneralExecuting / StepExecuting / StepCompleted. |
|
||||
| Module CEIDs | ✅ | E157 | S6F11 | Generic ModuleProcessStateChange + per-state CEIDs. |
|
||||
|
||||
## 4i. E84 Parallel I/O Handoff
|
||||
|
||||
| Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| Handoff state machine | ✅ | E84 | — | Full LOAD / UNLOAD signal vocabulary (CS_0/CS_1, VALID, TR_REQ, BUSY, COMPT, AM_AVBL, ES). Per-port via `E84PortStore` keyed by `port_id`; independent FSMs run in parallel per load port. |
|
||||
| Handshake timers TA1 / TA2 / TA3 | ✅ | E84 §6 | — | `E84StateMachine::set_timeouts({ta1, ta2, ta3})` + `set_timer_handlers(arm, cancel)`. TA1 armed in ValidAsserted, TA2 in Load/UnloadReady, TA3 in Transferring; cancelled on the matching transition out. Expiry transitions to `HandoffFault` (latched until `reset()`). FSM stays I/O-free — application drives the real clock (asio::steady_timer in the reference server). |
|
||||
|
||||
## 4j. E5 §13 Wafer Maps (S12)
|
||||
|
||||
| Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| Map Setup Data | ✅ | E5 §13 | S12F1/F2, S12F3/F4 | |
|
||||
| Map Transmit Inquire / Grant | ✅ | E5 §13 | S12F5/F6 | |
|
||||
| Map Data Send — row format (MAPFT=0) | ✅ | E5 §13 | S12F7/F8 | |
|
||||
| Map Data Send — array format (MAPFT=1)| ✅ | E5 §13 | S12F9/F10 | STRP + BINLT. |
|
||||
| Map Data Send — coord format (MAPFT=2)| ✅ | E5 §13 | S12F11/F12 | XYPOS + per-die BIN. |
|
||||
| Map Data Request — row | ✅ | E5 §13 | S12F13/F14 | |
|
||||
| Map Data Request — array | ✅ | E5 §13 | S12F15/F16 | |
|
||||
| Map Data Request — coord | ✅ | E5 §13 | S12F17/F18 | |
|
||||
| Map Error Send | ✅ | E5 §13 | S12F19 | One-way error report. |
|
||||
|
||||
## 4k. Exception Recovery (beyond E5 base alarms)
|
||||
|
||||
| Capability | Status | Spec ref | Messages | Notes |
|
||||
|---------------------------------------|--------|----------|----------|-------|
|
||||
| ExceptionStateMachine FSM | ✅ | — | — | Per-EXID Posted / Recovering / RecoverFailed / Cleared lifecycle. Not in upstream secsgem-py. |
|
||||
| Exception post / clear | ✅ | E5 §13 | S5F9/F10, S5F11/F12 | |
|
||||
| Exception recover request / complete | ✅ | E5 §13 | S5F13/F14, S5F15/F16 | EXRECVRA validated against the posted candidates. |
|
||||
| Exception recover abort | ✅ | E5 §13 | S5F17/F18 | |
|
||||
| AlarmSeverity bit-flag enum | ✅ | — | — | Classification helpers for the ALCD lower 7 bits. |
|
||||
|
||||
---
|
||||
|
||||
## 5. Message coverage matrix
|
||||
|
||||
164 SECS-II messages in the catalog, spanning streams 1, 2, 3, 5, 6, 7, 9, 10, 12, 14, 16.
|
||||
|
||||
| Pair | Direction | Status | Notes |
|
||||
|------------------|-----------|--------|-------|
|
||||
| S1F1 / S1F2 | H↔E | ✅ | round-trip + demo |
|
||||
| S1F3 / S1F4 | H→E | ✅ | round-trip + demo |
|
||||
| S1F11 / S1F12 | H→E | ✅ | round-trip + demo |
|
||||
| S1F13 / S1F14 | H↔E | ✅ | round-trip + demo |
|
||||
| S1F15 / S1F16 | H→E | ✅ | round-trip + demo |
|
||||
| S1F17 / S1F18 | H→E | ✅ | round-trip + demo |
|
||||
| S1F19 / S1F20 | H→E | ✅ | round-trip + demo (compliance self-report) |
|
||||
| S1F21 / S1F22 | H→E | ✅ | round-trip + demo |
|
||||
| S1F23 / S1F24 | H→E | ✅ | collection event namelist (CEID → VID) |
|
||||
| S2F13 / S2F14 | H→E | ✅ | EC values |
|
||||
| S2F15 / S2F16 | H→E | ✅ | EC set |
|
||||
| S2F17 / S2F18 | H→E | ✅ | clock |
|
||||
| S2F21 / S2F22 | H→E | ✅ | legacy remote command (no params) |
|
||||
| S2F23 / S2F24 | H→E | ✅ | trace initialize |
|
||||
| S2F25 / S2F26 | H→E | ✅ | loopback diagnostic |
|
||||
| S2F29 / S2F30 | H→E | ✅ | EC namelist |
|
||||
| S2F31 / S2F32 | H→E | ✅ | set time |
|
||||
| S2F33 / S2F34 | H→E | ✅ | define report |
|
||||
| S2F35 / S2F36 | H→E | ✅ | link event report |
|
||||
| S2F37 / S2F38 | H→E | ✅ | enable event |
|
||||
| S2F41 / S2F42 | H→E | ✅ | host command (modern) |
|
||||
| S2F43 / S2F44 | H→E | ✅ | reset spooling |
|
||||
| S2F45 / S2F46 | H→E | ✅ | define variable limits |
|
||||
| S2F47 / S2F48 | H→E | ✅ | request limit attrs |
|
||||
| S2F49 / S2F50 | H→E | ✅ | enhanced remote command (OBJSPEC + CPACK/CEPACK) |
|
||||
| S3F17 / S3F18 | H→E | ✅ | E87 carrier action |
|
||||
| S3F19 / S3F20 | H→E | ✅ | E87 slot map verify |
|
||||
| S3F21 / S3F22 | E→H | ✅ | E87 slot map report |
|
||||
| S3F23 / S3F24 | H→E | ✅ | E87 port group change |
|
||||
| S3F25 / S3F26 | H→E | ✅ | E87 carrier transfer |
|
||||
| S3F27 / S3F28 | H→E | ✅ | E87 cancel carrier |
|
||||
| S5F1 / S5F2 | E→H | ✅ | alarm send |
|
||||
| S5F3 / S5F4 | H→E | ✅ | enable alarm |
|
||||
| S5F5 / S5F6 | H→E | ✅ | list alarms |
|
||||
| S5F7 / S5F8 | H→E | ✅ | list enabled alarms |
|
||||
| S5F9 / S5F10 | E→H | ✅ | exception post |
|
||||
| S5F11 / S5F12 | E→H | ✅ | exception clear |
|
||||
| S5F13 / S5F14 | H→E | ✅ | exception recover request |
|
||||
| S5F15 / S5F16 | E→H | ✅ | exception recover complete |
|
||||
| S5F17 / S5F18 | H→E | ✅ | exception recover abort |
|
||||
| S6F1 / S6F2 | E→H | ✅ | trace data |
|
||||
| S6F5 / S6F6 | E→H | ✅ | multi-block inquire / grant |
|
||||
| S6F7 / S6F8 | H→E | ✅ | data transfer request / send |
|
||||
| S6F11 / S6F12 | E→H | ✅ | event report (unsolicited) |
|
||||
| S6F15 / S6F16 | H→E | ✅ | event report request (host-initiated) |
|
||||
| S6F19 / S6F20 | H→E | ✅ | individual report request |
|
||||
| S6F21 / S6F22 | H→E | ✅ | annotated individual report request |
|
||||
| S6F23 / S6F24 | H→E | ✅ | request spooled data |
|
||||
| S6F25 / S6F26 | E→H | ✅ | spool data ready (auto on re-SELECT) |
|
||||
| S7F1 / S7F2 | H→E | ✅ | PP load inquire / grant |
|
||||
| S7F3 / S7F4 | H→E | ✅ | PP send |
|
||||
| S7F5 / S7F6 | H→E | ✅ | PP request |
|
||||
| S7F17 / S7F18 | H→E | ✅ | PP delete |
|
||||
| S7F19 / S7F20 | H→E | ✅ | PP list |
|
||||
| S7F23 / S7F24 | H→E | ✅ | E42 formatted PP send |
|
||||
| S7F25 / S7F26 | H→E | ✅ | E42 formatted PP request |
|
||||
| S9F1, F3, F5, F7, F9, F11, F13 | E↔H | ✅ | protocol errors; auto-emitted on the documented conditions |
|
||||
| S10F1 / S10F2 | E→H | ✅ | terminal request (equipment originated) |
|
||||
| S10F3 / S10F4 | H→E | ✅ | terminal display single |
|
||||
| S10F5 / S10F6 | H→E | ✅ | terminal display multi |
|
||||
| S10F7 | H→E | ✅ | terminal display broadcast (W=0, no reply) |
|
||||
| S12F1–F19 | H↔E | ✅ | wafer maps — row, array, coord; setup, request, send, error |
|
||||
| S14F1 / S14F2 | H→E | ✅ | E39 GetAttr |
|
||||
| S14F3 / S14F4 | H→E | ✅ | E39 SetAttr |
|
||||
| S14F9 / S14F10 | H→E | ✅ | E94 CJ create |
|
||||
| S14F11 / S14F12 | H→E | ✅ | E94 CJ delete |
|
||||
| S16F5 / S16F6 | H→E | ✅ | E40 PRJobCommand |
|
||||
| S16F7 / S16F8 | H→E | ✅ | E40 PRJobMonitor |
|
||||
| S16F9 | E→H | ✅ | E40 PRJobAlert (auto on transition) |
|
||||
| S16F11 / S16F12 | H→E | ✅ | E40 PRJobCreate (full body) |
|
||||
| S16F13 / S16F14 | H→E | ✅ | E40 PRJobDequeue |
|
||||
| S16F15 / S16F16 | H→E | ✅ | E40 PRJobCreateMultiple |
|
||||
| S16F27 / S16F28 | H→E | ✅ | E94 CJobCommand |
|
||||
|
||||
---
|
||||
|
||||
## 6. Demo evidence
|
||||
|
||||
The two-container demo (`docker compose up --no-deps server client`)
|
||||
walks ~20 SECS transactions end-to-end:
|
||||
|
||||
1. TCP connect → `Select.req` → `Select.rsp(Ok)` → SELECTED on both sides.
|
||||
2. `S1F13`/`S1F14` Establish Comms.
|
||||
3. `S1F17`/`S1F18` Request Online; control state transitions
|
||||
`HostOffline → AttemptOnline → OnlineRemote`.
|
||||
4. `S1F19`/`S1F20` host fetches the equipment's GEM-compliance self-report.
|
||||
5. `S1F21`/`S1F22` DVID namelist.
|
||||
6. `S1F11`/`S1F12` SVID namelist → `S1F3`/`S1F4` values read.
|
||||
7. `S2F29`/`S2F30` EC namelist → `S2F13`/`S2F14` EC read.
|
||||
8. `S2F17`/`S2F18` clock read.
|
||||
9. `S2F33`/`S2F34` Define Report 1000 over the 3 SVIDs.
|
||||
10. `S2F35`/`S2F36` Link CEIDs 200 and 300 to Report 1000.
|
||||
11. `S2F37`/`S2F38` Enable CEIDs 200, 300.
|
||||
12. `S2F41`/`S2F42` host command **START** → server emits
|
||||
`S6F11(CEID=300)` carrying the linked Report 1000 → host acks `S6F12`.
|
||||
13. `S5F5`/`S5F6` list alarm directory.
|
||||
14. `S5F3`/`S5F4` enable alarm 1.
|
||||
15. `S2F41`/`S2F42` host command **FAULT** → server emits
|
||||
`S5F1` (ALCD=0x84) + `S6F11(CEID=200)`.
|
||||
16. Spool window: `SPOOL_ON` → `START` (emission goes to spool) →
|
||||
`SPOOL_OFF` → `S6F23(Transmit)` → server drains queued S6F11 to host.
|
||||
17. `S7F19`/`S7F20` recipe list, `S7F5`/`S7F6` fetch RECIPE-A.
|
||||
18. `S16F11`/`S16F12` create Process Job `PJ-1` with PPID `RECIPE-A`.
|
||||
19. `S14F9`/`S14F10` create Control Job `CJ-1` containing `[PJ-1]`.
|
||||
20. `S16F27`/`S16F28` CJSTART → equipment cascades CJ Queued → Executing
|
||||
and the contained PJ through SettingUp → WaitingForStart →
|
||||
Processing → ProcessComplete, emitting one `S16F9 PRJobAlert` per PJ
|
||||
transition and `S6F11(CEID=400)` / `S6F11(CEID=401)` for CJ Executing
|
||||
/ Completed.
|
||||
21. `S14F11`/`S14F12` delete `CJ-1`.
|
||||
22. `S10F1`/`S10F2` host → equipment terminal display.
|
||||
23. `S1F15`/`S1F16` Request Offline.
|
||||
24. `Separate.req` → clean close on both sides.
|
||||
|
||||
Unit tests: **445 cases / 2753 assertions pass** (`docker compose run --rm tests`).
|
||||
The suite includes integration tests that drive a real `hsms::Connection`
|
||||
over a loopback socket pair to verify the E37 §7.2 / §7.4 / §7.7
|
||||
edge cases — not just the happy path.
|
||||
The E30 §6.5 Communication state machine is unit-tested independently of
|
||||
the transport (timer firings simulated via test callbacks).
|
||||
|
||||
Live conformance harness: **`build/secs_conformance --host <ip> --port <p>`**
|
||||
walks 47 host-driven checks against a passive equipment, covering every
|
||||
E30 fundamental + additional capability that COMPLIANCE.md ✅ — establish
|
||||
comms, identification, status/DVID/CEID/EC namelists + values, dynamic
|
||||
event reports (define/link/enable + S6F15/F19/F21 readbacks), unsolicited
|
||||
S6F11 observation after RCMD, all three remote-command forms
|
||||
(S2F41/F21/F49), trace init, limits, spool reset + transmit, alarms
|
||||
(list/list-enabled/enable), exception recover/abort (S5F13/F17), PP
|
||||
load-inquire/list/request, both terminal-display directions
|
||||
(S10F3/F5), E40 PJ create/monitor/command/dequeue, E94 CJ
|
||||
create/command/delete, E87 carrier action/slot-map/transfer/cancel,
|
||||
and E39 GetAttr. Exits 0 with PASS/FAIL summary; intended to be run
|
||||
against vendor equipment as the first-line conformance probe.
|
||||
|
||||
---
|
||||
|
||||
## 7. Interoperability with external implementations
|
||||
|
||||
Four independent external validators cross-check the codebase. None
|
||||
of them shares code with us; three of them are not even C++. Full
|
||||
test plan in [VERIFICATION.md](VERIFICATION.md); proof commands in
|
||||
[PROOFS.md](PROOFS.md).
|
||||
|
||||
**secsgem-py 0.3.0** (Python reference implementation, Apache 2.0).
|
||||
Three harnesses under `interop/`:
|
||||
|
||||
- **secsgem-py active host → C++ passive server**: 31 named checks
|
||||
across S1/S2/S5/S6/S7/S10 plus unsolicited S5F1/S6F11.
|
||||
- **C++ active host → secsgem-py passive equipment**: HSMS select +
|
||||
S1F13 + S1F1 + S1F3 + clean separate; exits 0.
|
||||
- **C++ active host → raw GEM 300 streams** (`raw_gem300_harness.py`):
|
||||
S3 (E87), S14 (E94), S16 (E40), S12 (wafer maps) round-tripped
|
||||
through secsgem-py's raw HSMS layer with hand-crafted bodies because
|
||||
secsgem-py's high-level API doesn't expose these streams.
|
||||
|
||||
**secs4java8** (independent Java SECS implementation by Kenta
|
||||
Shimizu, Apache 2.0). 55 cross-validation checks under
|
||||
`interop/secs4j/` covering S1/S2/S3/S5/S6/S7/S10/S14/S16, the full
|
||||
E40 PJ body, dynamic event reports + unsolicited S6F11/S5F1
|
||||
observation, alarm management, spool, PP management, terminal
|
||||
services, limits, trace, E39, and the GEM 300 streams secsgem-py
|
||||
couldn't easily drive. This is the only validator that exercises
|
||||
S2F49 (enhanced remote command) and S5F13–F18 (exception recovery)
|
||||
end-to-end against a second SECS implementation.
|
||||
|
||||
**Wireshark / tshark HSMS dissector** (independent network-protocol
|
||||
authors). `interop/tshark_validate.sh` captures a pcap of the demo
|
||||
run, dissects with tshark's built-in HSMS dissector, asserts no
|
||||
malformed-packet warnings and that every expected control + data
|
||||
frame parses. 69 HSMS frames dissected cleanly. This catches
|
||||
framing bugs that two SECS implementations might *both* share but
|
||||
that a third party reading the bytes would flag.
|
||||
|
||||
**libFuzzer + ASan + UBSan** (coverage-guided structural search).
|
||||
`apps/fuzz_secs2_decode.cpp` and `apps/fuzz_sml_parse.cpp` feed
|
||||
random inputs to the decoder and SML parser under
|
||||
AddressSanitizer + UndefinedBehaviorSanitizer. 60-second CI lanes
|
||||
typically explore 200 000+ inputs through `secs2::decode` and
|
||||
1 400 000+ through `try_parse_sml`; 0 crashes, 0 ASan/UBSan reports.
|
||||
|
||||
Bugs surfaced and fixed across the four channels include: strict
|
||||
per-width parsing rejected U1-encoded identifiers (SEMI E5 allows
|
||||
`U1|U2|U4|U8`); PPBODY-as-ASCII was rejected; S1F23/F24 wasn't
|
||||
implemented; S10F3 (host→equipment Terminal Display Single) wasn't
|
||||
wired; one HSMS framing edge case caught by the tshark dissector;
|
||||
several SML edge cases caught by libFuzzer.
|
||||
|
||||
---
|
||||
|
||||
## 8. What "100% GEM-compliant" honestly means here
|
||||
|
||||
Every GEM Fundamental and every GEM Additional capability that the E30
|
||||
specification defines with a concrete SECS-II message set is implemented,
|
||||
round-trip-tested, demonstrated in the two-container demo, AND
|
||||
cross-validated against secsgem-py 0.3.0 on the overlap. Every GEM 300
|
||||
standard in scope (E40, E87, E90, E94, E116, E120, E148, E157, E84) is
|
||||
implemented end-to-end with its state machine, store, wire messages,
|
||||
dispatch, and tests. Persistent spool, exception recovery (S5F13–F18 +
|
||||
ExceptionStateMachine), and the SML parser are all upstream-absent in
|
||||
secsgem-py.
|
||||
|
||||
What this codebase does **not** demonstrate, and what a real
|
||||
"GEM-compliant" marketing claim would still need:
|
||||
|
||||
1. **Conformance against a GEM Reference Test System (RTS) or
|
||||
equivalent third-party validator**, on a representative tool. The
|
||||
codebase provides the message catalog + the runtime; running an
|
||||
external validator against a real physical or simulated tool is
|
||||
how compliance gets *certified*.
|
||||
2. **Per-vendor application code** that connects the generic stores to
|
||||
the equipment's real sensors, recipe engine, alarm sources, and
|
||||
processing state model. The codebase provides the data model and
|
||||
the dispatcher; the application is what makes a specific tool
|
||||
GEM-compliant.
|
||||
|
||||
In short: this is a **GEM-conformant runtime stack with the full GEM
|
||||
300 suite**, not a GEM-conformant *tool*. Pointing the runtime at a
|
||||
real piece of equipment, populating the YAML files with the tool's
|
||||
real SVIDs / ECIDs / alarms / capabilities / job behaviour, and wiring
|
||||
the application callbacks completes the picture.
|
||||
+238
@@ -0,0 +1,238 @@
|
||||
# FAQ
|
||||
|
||||
Questions we hear once per integration. Skim before you ask. If
|
||||
your question isn't here and isn't obvious from the other docs,
|
||||
ask once — your question probably belongs in this file and we'll
|
||||
add it.
|
||||
|
||||
## Why is HSMS unencrypted?
|
||||
|
||||
Because SEMI E37 says so. HSMS is plain TCP with a 14-byte
|
||||
framing header — no TLS, no auth, no nonces. Every commercial MES
|
||||
on the market speaks exactly that wire, and changing it would make
|
||||
us incompatible with all of them. Encryption and authentication
|
||||
belong at the network layer: see [SECURITY.md](SECURITY.md) for
|
||||
the stunnel.conf + nftables setup that wraps the unencrypted TCP
|
||||
in mTLS without modifying the wire protocol.
|
||||
|
||||
## What's the difference between SVID and DVID?
|
||||
|
||||
**SVID** is a *status* variable — equipment state the host queries
|
||||
(chamber pressure, current control state, wafer counter).
|
||||
**DVID** is a *data* variable — intermediate values, typically
|
||||
computed or sensor-derived, that aren't part of the equipment's
|
||||
state model.
|
||||
|
||||
In practice fab tools blur the line. The library treats them
|
||||
identically except for which message reports them: `S1F3 / S1F11`
|
||||
for SVIDs, `S1F21 / S1F22` for DVIDs. Variable lookups by VID
|
||||
span both (`EquipmentDataModel::vid_value`).
|
||||
|
||||
## Do I really need all four YAML files?
|
||||
|
||||
Yes for production; no for a quick "does it compile":
|
||||
|
||||
- `equipment.yaml` — your tool's data dictionary. Required.
|
||||
- `control_state.yaml` — the E30 control state machine (HostOffline,
|
||||
AttemptOnline, OnlineRemote, …). The default in `data/` works as
|
||||
a starting point; you may customize transitions.
|
||||
- `process_job_state.yaml` — the E40 PJ FSM. Default is spec-typical;
|
||||
customize only if your tool has unusual recipe semantics.
|
||||
- `control_job_state.yaml` — the E94 CJ FSM. Same.
|
||||
|
||||
`secs_server --validate-config` checks all four in one pass and
|
||||
exits 0 / 1. Run it in CI on every config change.
|
||||
|
||||
## PJ vs CJ — what's the difference?
|
||||
|
||||
A **PJ** (E40 Process Job) is "process this batch of material with
|
||||
this recipe." One PJ = one recipe run = one set of wafers. It
|
||||
has its own FSM (Queued → SettingUp → Processing → ProcessComplete).
|
||||
|
||||
A **CJ** (E94 Control Job) is "execute these PJs in order, as a
|
||||
unit, with start/pause/abort semantics." A CJ owns an ordered list
|
||||
of PRJOBIDs. When the host issues `CJSTART`, the CJ promotes its
|
||||
PJs through their lifecycles.
|
||||
|
||||
You typically need both: the MES creates a CJ containing N PJs,
|
||||
then starts the CJ. PJs without a CJ are legal — they just sit in
|
||||
Queued waiting for someone to select them — but most MES drives
|
||||
batches through CJs.
|
||||
|
||||
## Who fires FSM transitions — the library or my code?
|
||||
|
||||
**Your code.** The library implements the FSMs (legal transitions,
|
||||
validation, persistence) but it doesn't know when a wafer was
|
||||
actually loaded or when a recipe step finished — those signals come
|
||||
from your tool. The pattern across every store is:
|
||||
|
||||
```cpp
|
||||
// You fire the event; the FSM validates + transitions + emits.
|
||||
model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::SetupComplete);
|
||||
model->carriers.fire_id_event("CAR-A1B2", gem::CarrierIDEvent::Read);
|
||||
```
|
||||
|
||||
Host commands (`S2F41` RCMD=START, `S16F5` PRJSTART, `S16F27` CJSTART)
|
||||
arrive via the wire and get dispatched into your registered
|
||||
handlers; the handler typically calls `fire_internal` or
|
||||
`on_host_command` on the relevant store.
|
||||
|
||||
See INTEGRATION.md §4 for the worked patterns.
|
||||
|
||||
## What runs on which thread?
|
||||
|
||||
**Everything that touches the data model runs on the io_context
|
||||
thread.** There are no locks in `EquipmentDataModel`.
|
||||
|
||||
- The Router dispatch (incoming wire messages) — on the io_context.
|
||||
- All `set_*_change_handler` callbacks — on the io_context.
|
||||
- Periodic timers you register via asio — on the io_context.
|
||||
|
||||
If your code lives on another thread (typical for sensor polling),
|
||||
marshal updates via `asio::post`:
|
||||
|
||||
```cpp
|
||||
asio::post(io.get_executor(), [model, value] {
|
||||
model->svids.set_value(100, secs2::Item::f4(value));
|
||||
});
|
||||
```
|
||||
|
||||
INTEGRATION.md §3 has the full thread-safety contract.
|
||||
|
||||
## How do I add a new SECS-II message?
|
||||
|
||||
Edit `data/messages.yaml`, add a row, rebuild. The codegen
|
||||
(`tools/gen_messages.py`) emits a typed builder + parser into
|
||||
`messages.hpp`. Then register a Router handler in your `main.cpp`
|
||||
for the new `(stream, function)` pair. See README "Adding a
|
||||
capability" or ARCHITECTURE.md for the full walkthrough.
|
||||
|
||||
## What's the difference between `Item::ascii("X")` and `Item::binary({'X'})`?
|
||||
|
||||
The wire format byte differs — `0x41 01 58` for ASCII vs
|
||||
`0x21 01 58` for Binary. Some peers (notably secsgem-py) default
|
||||
PPBODY to ASCII; others use Binary. Our codec accepts either via
|
||||
the `BINARY_OR_ASCII` codegen type for fields the spec lists as
|
||||
`ASCII | Binary | List` (the PPBODY case in S7F3/F6).
|
||||
|
||||
For most fields it doesn't matter — pick the format that matches
|
||||
your data semantically.
|
||||
|
||||
## My MES sends a message that worked in `interop` but fails in production. What's going on?
|
||||
|
||||
Three usual suspects:
|
||||
|
||||
1. **U-width.** Your MES is sending `DATAID` as U1 but our handler
|
||||
was strict for U4. We're lenient now via `any_unsigned_first`,
|
||||
but if you have custom handlers in your code, use that helper
|
||||
rather than `as_u4_scalar` for identifier fields.
|
||||
2. **PPBODY direction.** Some MES send PPBODY as ASCII even when
|
||||
the spec says it can be Binary. Use `as_text_or_binary` not
|
||||
`as_binary`.
|
||||
3. **Trailing fields.** Some MES add proprietary trailing fields
|
||||
to S2F41 / S16F11 / S3F17 bodies that aren't in the standard.
|
||||
Our parsers are tolerant of extras; check your handler's
|
||||
assumptions.
|
||||
|
||||
See MES_INTEROP.md §13 for the per-MES quirk register.
|
||||
|
||||
## What if the spec is ambiguous on some detail?
|
||||
|
||||
Cross-check against the secsgem-py and secs4j wire output:
|
||||
[VERIFICATION.md](VERIFICATION.md). If both peers agree on a
|
||||
shape, that's the working interpretation regardless of how you read
|
||||
the spec text. If they disagree, the secsgem-py output usually
|
||||
wins (it's the de-facto Python reference and most MES vendors test
|
||||
against it), but file the question — we may need a new test.
|
||||
|
||||
## Can I run this without Docker?
|
||||
|
||||
In principle yes — you need g++-13 (or any C++20 compiler), CMake,
|
||||
Ninja, libasio-dev, libyaml-cpp-dev, python3. But every doc, every
|
||||
CI lane, every test command in the repo assumes Docker. Going
|
||||
off-piste means re-deriving the build on your host. We don't
|
||||
support it; we don't actively break it.
|
||||
|
||||
## How does persistence survive a crash mid-write?
|
||||
|
||||
Every store uses a `.tmp + atomic rename` pattern: writes go to
|
||||
`<file>.tmp`, then `rename(2)`s into place. POSIX guarantees the
|
||||
rename is atomic on the same filesystem. A crash mid-write loses
|
||||
the `.tmp` (corrupt-drop on next replay) but leaves the prior
|
||||
record intact.
|
||||
|
||||
Every store's loader accepts versions in `[1, kVersion]` so future
|
||||
schema bumps don't nuke old records — see README §Production
|
||||
deployment "Schema migrations."
|
||||
|
||||
## What does the "spool" actually do?
|
||||
|
||||
When the host MES disconnects, the equipment can't deliver
|
||||
unsolicited S5F1 alarms / S6F11 events. Without spool they'd be
|
||||
lost.
|
||||
|
||||
With spool enabled (`SpoolStore::set_spoolable_streams({5, 6})`),
|
||||
those frames queue to in-memory FIFO (and persistent disk if
|
||||
`enable_persistence` is set). On the host's next SELECT, the
|
||||
equipment emits `S6F25 SpoolDataReady(count)`; the host issues
|
||||
`S6F23(Transmit)` to drain, or `S6F23(Purge)` to discard.
|
||||
|
||||
It's the GEM equivalent of an outbox. See E30 §6.22 and our
|
||||
SpoolStore source.
|
||||
|
||||
## How is "robustness fuzz" different from "libFuzzer"?
|
||||
|
||||
- **Robustness fuzz** (`tests/test_robustness_fuzz.cpp`) is a
|
||||
*model-level* property test. It picks random tool operations
|
||||
(PJ create, alarm set, substrate move, …) respecting FSM
|
||||
legality, and checks invariants after each.
|
||||
- **libFuzzer** (`apps/fuzz_*.cpp`) is a *byte-level* coverage-
|
||||
guided fuzzer. It feeds arbitrary bytes to the codec and
|
||||
asserts no crash / UB.
|
||||
|
||||
They cover different concerns: robustness fuzz catches *semantic*
|
||||
bugs (lost data, wrong state); libFuzzer catches *parser* bugs
|
||||
(crashes, UB, buffer overruns).
|
||||
|
||||
## What's "conformance" vs "interop"?
|
||||
|
||||
- **Conformance** (`build/secs_conformance`) is *us* driving *us*
|
||||
through every claimed E30 capability and asserting the spec-
|
||||
mandated reply S/F. Catches our regressions against our own
|
||||
understanding of the spec.
|
||||
- **Interop** (`interop/*.py`, `interop/secs4j/*.java`,
|
||||
`interop/tshark_validate.sh`) is third-party tools agreeing on
|
||||
the wire bytes our equipment produces. Catches "we got the
|
||||
spec wrong" — which conformance can't.
|
||||
|
||||
Both are necessary; neither replaces the other. See VERIFICATION.md.
|
||||
|
||||
## How do I bring this to a customer site?
|
||||
|
||||
Run through the five external proofs in
|
||||
[the eight commands in PROOFS.md](PROOFS.md) at
|
||||
the customer's network. Then walk MES_INTEROP.md against their
|
||||
actual MES. Then deploy per [SECURITY.md](SECURITY.md) for the
|
||||
nftables / stunnel / signing setup. Then page on the metrics from
|
||||
INTEGRATION.md §6.4.
|
||||
|
||||
## What's not implemented?
|
||||
|
||||
Every E30 Fundamental + Additional capability and every GEM 300
|
||||
standard in scope is shipped. The two non-shipped pieces are:
|
||||
|
||||
1. **The asio `serial_port` adapter for SECS-I** — the FSM is
|
||||
implemented and tested end-to-end over TCP
|
||||
([`secsi::TcpTransport`](include/secsgem/secsi/tcp_transport.hpp));
|
||||
the serial-port driver is a deferred follow-up (most modern GEM
|
||||
equipment runs HSMS). Listed under "Deferred follow-ups" in
|
||||
[README.md](README.md).
|
||||
2. **A GEM Reference Test System (RTS) run** — paid third-party
|
||||
certification gate, not a code feature. See
|
||||
[COMPLIANCE.md](COMPLIANCE.md) §8 for what "100% GEM-compliant"
|
||||
honestly means about a codebase vs. a certified tool.
|
||||
|
||||
Note: Equipment Processing States are tool-defined per E30 §6.3 — the
|
||||
engine ships, and vendors load their concrete states (IDLE / SETUP /
|
||||
READY / EXECUTING / …) the same way `data/control_state.yaml` is
|
||||
loaded. That isn't a gap, it's how the spec is designed.
|
||||
@@ -0,0 +1,163 @@
|
||||
# Glossary
|
||||
|
||||
SECS/GEM has roughly thirty acronyms that the spec uses without
|
||||
introduction. This is the one-page decoder. Each entry has the
|
||||
expansion, a one-line definition, and (where useful) the place in
|
||||
this codebase where it shows up.
|
||||
|
||||
## Identifiers and data
|
||||
|
||||
| Term | Stands for | Meaning |
|
||||
|------------|----------------------------------|------------------------------------------------------------------------------------------|
|
||||
| **SVID** | Status Variable Identifier | A read-only equipment-side value the host queries via `S1F3` (e.g. ChamberPressureTorr). |
|
||||
| **DVID** | Data Variable Identifier | Same shape as SVID but conceptually a *data variable* (intermediate, not status). Reported via `S1F21/F22`. |
|
||||
| **ECID** | Equipment Constant Identifier | A host-settable equipment parameter (e.g. T-timers, thresholds). Set via `S2F15`, read via `S2F13`. |
|
||||
| **CEID** | Collection Event Identifier | An event the equipment can emit (e.g. `ProcessStarted`). Bound to reports via `S2F35`, fires via `S6F11`. |
|
||||
| **RPTID** | Report Identifier | A named bundle of VIDs (`S2F33`). CEIDs link to RPTIDs; the report carries the VID values. |
|
||||
| **ALID** | Alarm Identifier | An equipment alarm (e.g. `ChamberPressureHigh`). `S5F5/F1/F3`. |
|
||||
| **EXID** | Exception Identifier | A recoverable exception condition. `S5F9 → S5F13 → S5F11/F15` lifecycle. |
|
||||
| **PPID** | Process Program Identifier | A recipe name. `S7F19` lists, `S7F5` requests, `S7F3` sends. |
|
||||
| **MID** | Material Identifier | A wafer / substrate id (used in E40 PJ `mtrloutspec`). |
|
||||
| **CARRIERID** | Carrier Identifier | A FOUP / cassette id (E87). |
|
||||
| **PRJOBID**| Process Job Identifier | E40 PJ id; one PJ = one recipe-run for one batch of material. |
|
||||
| **CTLJOBID** | Control Job Identifier | E94 CJ id; a CJ owns an ordered list of PRJOBIDs. |
|
||||
| **SUBSTID**| Substrate Identifier | E90 wafer id, distinct from MID (which can be looser). |
|
||||
| **OBJSPEC** | Object Specifier | E39 generic object reference (e.g. an instance of an E120 CEM object). |
|
||||
| **OBJTYPE** | Object Type | Companion to OBJSPEC — the class. |
|
||||
| **MDLN** | Model Number | Equipment model identifier (e.g. `ACME-PVD-3000`). Sent in `S1F2` and `S1F14`. |
|
||||
| **SOFTREV**| Software Revision | Equipment software version string. Sent in `S1F2` and `S1F14`. |
|
||||
| **EQPTYP** | Equipment Type | A category string (e.g. `PVD`) sent in `S1F20`. |
|
||||
| **DATAID** | Data Identifier | A correlation id within multi-step setups (e.g. tying `S2F33`+`S2F35`+`S2F37` together). |
|
||||
|
||||
## Acknowledgement codes
|
||||
|
||||
Every host-issued request that mutates state gets back a 1-byte
|
||||
acknowledgement code. The mnemonic tells you which spec section
|
||||
the enum lives in.
|
||||
|
||||
| Code | Used in | Values |
|
||||
|-----------|-------------------|---------------------------------------------------------------------------------------------|
|
||||
| **COMMACK**| `S1F14` | 0 = Accept, 1 = Denied (equipment not ready). |
|
||||
| **ONLACK** | `S1F18` | 0 = Accept, 1 = NotAccept, 2 = AlreadyOnline. |
|
||||
| **OFLACK** | `S1F16` | Only 0 = Accept defined. |
|
||||
| **HCACK** | `S2F42` / `S16F6` / `S16F12` etc. | 0 = Accept, 1 = InvalidCommand, 2 = CannotDoNow, 3 = ParameterInvalid, 4 = AcceptedWillFinishLater, 5 = Rejected, 6 = InvalidObject. |
|
||||
| **CMDA** | `S2F22` | Same enum as HCACK; spelled differently in the spec. |
|
||||
| **ACKC5** | `S5F4` / `S5F2` | 0 = Accept, 1 = Error. |
|
||||
| **ACKC6** | `S6F12` / `S6F2` etc. | 0 = Accept, 1 = Error. |
|
||||
| **ACKC7** | `S7F4` / `S7F18` | 0 = Accept, 1-6 = various PP-management errors. |
|
||||
| **ACKC10** | `S10F2` / `S10F4` / `S10F6` | 0 = Accept, 1-3 = TerminalDisplay errors. |
|
||||
| **DRACK** | `S2F34` | Define Report Ack: 0 = Accept, 1-4 = various definition errors. |
|
||||
| **LRACK** | `S2F36` | Link Event Ack: 0 = Accept, 1-5. |
|
||||
| **ERACK** | `S2F38` | Enable Event Ack: 0 = Accept, 1 = UnknownCEID. |
|
||||
| **EAC** | `S2F16` | Equipment Constant ack: 0 = Accept, 1 = Denied_OutOfRange, 2 = BusyOrUnknown, 3 = MajorOOR.|
|
||||
| **TIACK** | `S2F32` | Time Ack: 0 = Accept, 1 = Error, 2 = NotDone. |
|
||||
| **GRANT** | `S2F40` / `S6F6` | 0 = Grant, 1-3 = denials. |
|
||||
| **ALCD** | `S5F1` | **Alarm Code**: bit-7 = set/clear flag; lower 7 bits = severity bitmap (E5 §10.3). |
|
||||
| **OBJACK** | `S14F2` / `S14F10` etc. | E39 object service ack: 0 = Success, plus per-call denial codes. |
|
||||
|
||||
## Streams and functions
|
||||
|
||||
SECS-II messages are named **`SsFf`** — Stream *s*, Function *f*.
|
||||
Odd functions are **primary** (initiating a transaction); even
|
||||
functions are the **reply** to function *f − 1*. A primary with the
|
||||
**W-bit** set expects a reply.
|
||||
|
||||
| Stream | Domain | Example exchange |
|
||||
|-------:|----------------------------------------------|------------------------------------------------------------------|
|
||||
| S1 | Equipment status | S1F1/F2 "Are You There", S1F13/F14 Establish Comms |
|
||||
| S2 | Equipment control + configuration | S2F33 define report, S2F41 host command |
|
||||
| S3 | E87 carrier management | S3F17 CarrierAction, S3F19 SlotMapVerify |
|
||||
| S5 | Exception reporting | S5F1 alarm send, S5F9-F18 exception recovery |
|
||||
| S6 | Data collection | S6F11 unsolicited event report, S6F15 event report request |
|
||||
| S7 | Process program management | S7F5 PP request, S7F19 PP list, S7F23 E42 formatted PP |
|
||||
| S9 | System errors | S9F1, F3, F5, F7, F9, F11 — protocol-error notifications |
|
||||
| S10 | Terminal services | S10F3 host→equipment display, S10F1 equipment→host |
|
||||
| S12 | Wafer maps (E5 §13) | S12F1 setup, S12F7/F9/F11 send (3 formats) |
|
||||
| S14 | E39 / E94 object services | S14F1 GetAttr, S14F9 CreateControlJob |
|
||||
| S16 | E40 / E94 jobs | S16F11 PRJobCreate, S16F27 CJobCommand |
|
||||
|
||||
## HSMS terminology
|
||||
|
||||
| Term | Stands for | Meaning |
|
||||
|------------|----------------------------------|------------------------------------------------------------------------------------------|
|
||||
| **HSMS** | High-Speed Message Service | The TCP-based SECS transport defined by SEMI E37. |
|
||||
| **HSMS-SS**| Single-Session | The common case: one session per TCP connection. |
|
||||
| **HSMS-GS**| General-Session | Multi-session: multiple session IDs share one TCP connection. |
|
||||
| **PType** | Presentation Type | 1-byte header field; 0 = SECS-II body. |
|
||||
| **SType** | Session Type | 1-byte header field identifying the message class (data, Select.req, Linktest, etc.). |
|
||||
| **MHEAD** | Message Header (10 bytes) | The HSMS framing header; appears unchanged in `S9F3/F5/F7/F11` payloads. |
|
||||
| **NOT-SELECTED** / **SELECTED** | HSMS connection state | Reached via Select.req/rsp; required before data messages can flow. |
|
||||
|
||||
## T-timers (E37 §10)
|
||||
|
||||
| Timer | Purpose | Typical default | Where enforced |
|
||||
|------:|----------------------------------------------------|-----------------|----------------------|
|
||||
| **T3**| Reply timeout for a W=1 primary | 45 s | per-transaction asio timer |
|
||||
| **T5**| Connect-separation: how long before retrying after a connection failure | 10 s | client retry loop |
|
||||
| **T6**| Control transaction timeout (Select / Linktest) | 5 s | one concurrent control transaction |
|
||||
| **T7**| Not-selected timeout — passive side, after TCP up | 10 s | armed on accept |
|
||||
| **T8**| Intercharacter timeout — bounds the payload read after the 4-byte length prefix | 6 s | data-read loop |
|
||||
|
||||
For SECS-I (E4), T1/T2/T3/T4 are the analogous serial-line timers
|
||||
covering inter-character, protocol, reply, and inter-block respectively.
|
||||
|
||||
## E84 signals (parallel I/O, AMHS handshake)
|
||||
|
||||
| Signal | Direction | Meaning |
|
||||
|--------|-------------------------|-----------------------------------------------|
|
||||
| CS_0 | AMHS → equipment | Carrier-stage select 0 (multi-port equipment) |
|
||||
| CS_1 | AMHS → equipment | Carrier-stage select 1 |
|
||||
| VALID | AMHS → equipment | Handshake start |
|
||||
| TR_REQ | AMHS → equipment | Transfer request |
|
||||
| BUSY | AMHS → equipment | Transfer in progress |
|
||||
| COMPT | AMHS → equipment | Transfer complete |
|
||||
| L_REQ | equipment → AMHS | Load request — port ready to receive |
|
||||
| U_REQ | equipment → AMHS | Unload request — port ready to release |
|
||||
| READY | equipment → AMHS | Ready |
|
||||
| ES | either | Emergency stop |
|
||||
|
||||
Handshake timers TA1 (VALID→L_REQ), TA2 (Load/UnloadReady→BUSY),
|
||||
TA3 (BUSY duration) live alongside the signals — see
|
||||
`include/secsgem/gem/e84_state.hpp` and INTEGRATION.md §4.6.
|
||||
|
||||
## Standards lineup
|
||||
|
||||
| Spec | Topic |
|
||||
|---------|-------------------------------------------------|
|
||||
| **E4** | SECS-I serial transport (block protocol) |
|
||||
| **E5** | SECS-II message structure + encoding rules |
|
||||
| **E30** | GEM — generic equipment model + capabilities |
|
||||
| **E37** | HSMS — TCP transport for SECS-II |
|
||||
| **E39** | Generic object services (`S14F1/F3` GetAttr/SetAttr) |
|
||||
| **E40** | Process job management (`S16F5/F11/F13`) |
|
||||
| **E42** | Formatted process programs (`S7F23-F26`) |
|
||||
| **E84** | Parallel I/O AMHS handshake |
|
||||
| **E87** | Carrier management (`S3F17/F19/F25/F27`) |
|
||||
| **E90** | Substrate tracking |
|
||||
| **E94** | Control job management (`S14F9/F11`, `S16F27`) |
|
||||
| **E116**| Equipment Performance Tracking |
|
||||
| **E120**| Common Equipment Model |
|
||||
| **E148**| Time synchronization |
|
||||
| **E157**| Module process tracking |
|
||||
|
||||
## Codebase shortcuts
|
||||
|
||||
| Term | What it refers to in this repo |
|
||||
|-----------------|-----------------------------------------------------------------------|
|
||||
| **The model** | `gem::EquipmentDataModel` — the composed bundle of every store. |
|
||||
| **A store** | One of the per-domain bundles under `include/secsgem/gem/store/` (alarms, carriers, spool, substrates, …). |
|
||||
| **The router** | `gem::Router` — `(stream, function) → handler` dispatch table. |
|
||||
| **The codec** | `secs2::encode` / `secs2::decode` for the wire bytes. |
|
||||
| **The catalog** | `data/messages.yaml` — every SECS-II message we ship, codegen'd to `messages.hpp`. |
|
||||
| **The proof** | The 8 commands in [PROOFS.md](PROOFS.md). |
|
||||
| **The bench** | `apps/secs_bench.cpp` — single-threaded throughput / latency / memory harness. |
|
||||
| **The fuzz** | `tests/test_robustness_fuzz.cpp` — randomized property test of the model. |
|
||||
|
||||
## See also
|
||||
|
||||
- [INTEGRATION.md](INTEGRATION.md) — when you've grasped the
|
||||
vocabulary, this is how you put it together.
|
||||
- [COMPLIANCE.md](COMPLIANCE.md) — every term above has a
|
||||
spec-anchored implementation; the audit cross-references both.
|
||||
- [FAQ.md](FAQ.md) — "OK, but *why*…" answers for the most common
|
||||
next questions.
|
||||
@@ -0,0 +1,715 @@
|
||||
# Integration tutorial
|
||||
|
||||
How a semiconductor equipment vendor takes this library and turns it
|
||||
into a SECS/GEM-compliant interface on a real tool.
|
||||
|
||||
The library gives you **the runtime stack** — wire codecs, the HSMS
|
||||
connection state machine, every GEM 300 sub-state-machine, persistent
|
||||
stores, the message catalog, and a dispatcher. What you bring is
|
||||
**the application**: knowledge of your tool's real sensors, recipes,
|
||||
alarms, processing states, and chamber I/O. This guide walks through
|
||||
how those two halves meet.
|
||||
|
||||
> **Audience.** Firmware / controls engineers integrating
|
||||
> SECS/GEM on a tool for the first time. Familiarity with SEMI
|
||||
> E5/E30/E37 helps but isn't required — every spec reference is
|
||||
> pinned in `COMPLIANCE.md`.
|
||||
|
||||
---
|
||||
|
||||
## 1. What you get vs. what you build
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────────────────────────┐
|
||||
│ your equipment application (you write) │
|
||||
│ recipe runner • sensor polling • alarm sources • UI hooks│
|
||||
├───────────────────────────────────────────────────────────┤
|
||||
│ secs-gem runtime stack (this library) │
|
||||
│ data model • FSMs • SECS-II codec • HSMS connection │
|
||||
│ message catalog • routers • persistence • spool │
|
||||
├───────────────────────────────────────────────────────────┤
|
||||
│ OS + Asio (provided) + your serial/Ethernet driver │
|
||||
└───────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The boundary lives at three classes:
|
||||
|
||||
- `gem::EquipmentDataModel` — the data dictionary (SVIDs, ECIDs,
|
||||
CEIDs, alarms, recipes, jobs, carriers, substrates …). Your
|
||||
application reads/writes it; the dispatcher serves it on the wire.
|
||||
- `gem::Router` — maps `(stream, function) → handler`. Wire it once
|
||||
at startup; messages flow through it.
|
||||
- `hsms::Connection` (or `secsi::TcpTransport` for SECS-I) — the
|
||||
byte-level transport. You feed it a TCP socket and a router; it
|
||||
runs.
|
||||
|
||||
You don't subclass the FSMs. You don't write parsers. You don't
|
||||
patch the dispatcher. Your code lives in two places: **YAML**
|
||||
(static configuration) and **callbacks** (dynamic glue).
|
||||
|
||||
---
|
||||
|
||||
## 2. The 30-minute first connection
|
||||
|
||||
The shortest path from `git clone` to "a host can talk to my tool":
|
||||
|
||||
### 2.1. Describe your tool in YAML
|
||||
|
||||
Copy `data/equipment.yaml`, rename to your tool, and edit:
|
||||
|
||||
```yaml
|
||||
device:
|
||||
id: 1 # E37 SESSION-ID
|
||||
model_name: "ACME-PVD-3000"
|
||||
software_rev: "1.4.2"
|
||||
equipment_type: "PVD" # S1F20 EQPTYP
|
||||
|
||||
svids: # status variables (read-only)
|
||||
- {id: 1, name: ControlState, units: "", type: ASCII, value: ""}
|
||||
- {id: 100, name: ChamberPressureTorr, units: "Torr", type: F4, value: 0.0}
|
||||
- {id: 101, name: WaferCounter, units: "wafer", type: U4, value: 0}
|
||||
|
||||
ecids: # equipment constants (host can set)
|
||||
- {id: 10, name: ChamberSetpointTorr, units: "Torr", type: F4,
|
||||
value: 1.0e-6, min: "1.0e-9", max: "1.0"}
|
||||
|
||||
ceids: # collection events
|
||||
- {id: 300, name: ProcessStarted}
|
||||
- {id: 301, name: ProcessCompleted}
|
||||
|
||||
alarms:
|
||||
- {id: 1, text: "Chamber pressure out of range", category: 4}
|
||||
|
||||
recipes:
|
||||
- {id: "RECIPE-A", body: "STEP HEAT 350C 60s\nEND"}
|
||||
|
||||
host_commands:
|
||||
- {name: START, ack: Accept, emit_ceid: 300}
|
||||
- {name: STOP, ack: Accept}
|
||||
```
|
||||
|
||||
That's the GEM data dictionary. The library will serve every
|
||||
S1F3/F11, S2F13/F29, S2F33-F38, S5F5, S7F19, S2F41, etc. against
|
||||
this YAML without any C++ changes.
|
||||
|
||||
### 2.2. Stand it up
|
||||
|
||||
A minimal `main()` looks like `apps/secs_server.cpp`. In your code:
|
||||
|
||||
```cpp
|
||||
auto model = std::make_shared<gem::EquipmentDataModel>();
|
||||
auto desc = config::load_equipment("/etc/acme/equipment.yaml", *model);
|
||||
auto sm = std::make_shared<gem::ControlStateMachine>(
|
||||
gem::ControlStateMachine::default_table(),
|
||||
gem::ControlState::HostOffline);
|
||||
|
||||
asio::io_context io;
|
||||
Server server(io, {/*port=*/5000, desc.device_id, {}});
|
||||
|
||||
server.on_accept([&](std::shared_ptr<hsms::Connection> conn) {
|
||||
auto router = std::make_shared<gem::Router>();
|
||||
register_default_handlers(*router, model, sm, conn); // your function
|
||||
conn->set_message_handler([router, conn](const secs2::Message& m) {
|
||||
return router->dispatch_with_s9(
|
||||
[&](uint8_t f, const std::array<uint8_t, 10>& mhead) {
|
||||
conn->emit_s9(f, mhead);
|
||||
},
|
||||
[&]() -> std::optional<std::array<uint8_t, 10>> {
|
||||
auto* h = conn->current_header();
|
||||
return h ? std::optional{h->encode()} : std::nullopt;
|
||||
}, m);
|
||||
});
|
||||
});
|
||||
|
||||
server.start();
|
||||
io.run();
|
||||
```
|
||||
|
||||
`register_default_handlers` is the only piece of glue you write at
|
||||
the start. The repo's `apps/secs_server.cpp` is a complete worked
|
||||
example — copy it verbatim, then customize the YAML to your tool.
|
||||
|
||||
### 2.3. Validate before you run
|
||||
|
||||
YAML edits are easy to get wrong: an unknown SECS-II type, a
|
||||
duplicate ID, a `host_command` referencing a CEID you forgot to
|
||||
declare. The server has a `--validate-config` mode that reads every
|
||||
YAML, accumulates *every* problem it can find, prints them with file
|
||||
and line number, and exits 0 / 1 without binding the port:
|
||||
|
||||
```sh
|
||||
secs_server --validate-config \
|
||||
--config /etc/acme/equipment.yaml \
|
||||
--state-table /etc/acme/control_state.yaml \
|
||||
--pj-state-table /etc/acme/process_job_state.yaml \
|
||||
--cj-state-table /etc/acme/control_job_state.yaml
|
||||
# [error] equipment.yaml:5 svids[0].type — unknown SECS-II type `WTF`
|
||||
# [error] equipment.yaml:7 alarms[0].category — value 200 out of range [0, 127]
|
||||
# [error] equipment.yaml:9 host_commands[0].emit_ceid — CEID 999 not declared in `ceids` section
|
||||
# 3 error(s), 0 warning(s) across 4 files
|
||||
```
|
||||
|
||||
Run this in CI on every config change and you skip the slow
|
||||
load-fail-edit-restart loop the first deployment otherwise becomes.
|
||||
|
||||
### 2.4. Run it
|
||||
|
||||
```sh
|
||||
docker compose up server # or your own deployment
|
||||
# host fires up secsgem-py / wonderware / equipment manager:
|
||||
# selects, S1F13, S1F1, S1F3 → you're talking GEM.
|
||||
```
|
||||
|
||||
That's the floor. From here, every section below adds capability.
|
||||
|
||||
---
|
||||
|
||||
## 3. Wiring real sensors to SVIDs
|
||||
|
||||
The YAML's `value:` field is the *initial* value. Your application
|
||||
updates the live value as the tool runs.
|
||||
|
||||
> **Thread-safety contract.** Every store in `EquipmentDataModel` is
|
||||
> single-threaded by design: there are no locks. All access — reads
|
||||
> from the dispatcher, writes from your application — must run on the
|
||||
> io_context that drives the HSMS connection. If your sensor polls
|
||||
> live on a different thread (typical), marshal the update via
|
||||
> `asio::post`:
|
||||
>
|
||||
> ```cpp
|
||||
> // Sensor-poll thread (separate from the io_context thread):
|
||||
> double torr = read_baratron();
|
||||
> asio::post(io.get_executor(), [model, torr] {
|
||||
> model->svids.set_value(/*ChamberPressure=*/100,
|
||||
> secs2::Item::f4(float(torr)));
|
||||
> });
|
||||
> ```
|
||||
>
|
||||
> Calling `set_value(...)` directly from the sensor thread is a data
|
||||
> race against the dispatcher reading the same SVID for an inbound
|
||||
> S1F3 — the library has no mutex to defend you. This is also true
|
||||
> for every `set_*_change_handler` callback you register: those fire
|
||||
> on the io_context thread, and any state observers (metrics
|
||||
> exporters, log shippers) must be thread-safe themselves or must
|
||||
> hand the work off.
|
||||
|
||||
Two patterns scale well:
|
||||
|
||||
1. **One updater per sensor, fixed cadence.** Each sensor's driver
|
||||
owns the (vid, set_value) pair and `asio::post`s into the io_context.
|
||||
2. **A single refresh tick.** A periodic timer dumps all polled
|
||||
values at once (`refresh()` in `apps/secs_server.cpp` does this
|
||||
for two virtual SVIDs). Because the periodic timer runs *on* the
|
||||
io_context, no posting is needed.
|
||||
|
||||
The SECS-II Item shape must match the YAML's `type:`. If the YAML
|
||||
says `F4` and you call `set_value(100, secs2::Item::ascii("..."))`,
|
||||
the host will get the string back — the library doesn't enforce a
|
||||
runtime check. Treat the YAML type as a contract you maintain.
|
||||
|
||||
---
|
||||
|
||||
## 4. Plugging the FSMs into your tool
|
||||
|
||||
Every GEM 300 sub-state-machine in the library is a behavior model.
|
||||
You decide *when* state transitions happen by firing events:
|
||||
|
||||
### 4.1. Equipment processing (E116 EPT)
|
||||
|
||||
```cpp
|
||||
// At startup or whenever the operator clicks "Run":
|
||||
model->ept.on_event(gem::EptEvent::EnterStandby);
|
||||
model->ept.on_event(gem::EptEvent::EnterProductive);
|
||||
|
||||
// Auto-emit S6F11(ControlEvent_*) on every transition:
|
||||
model->ept.set_state_change_handler(
|
||||
[&](gem::EptState, gem::EptState to, gem::EptEvent,
|
||||
std::chrono::milliseconds /*dwell*/) {
|
||||
uint32_t ceid = ept_state_to_ceid(to); // your switch/case
|
||||
if (!ceid || !model->is_event_enabled(ceid)) return;
|
||||
conn->send_data(gem::s6f11_event_report(
|
||||
next_dataid++, ceid, model->compose_reports_for(ceid)));
|
||||
});
|
||||
```
|
||||
|
||||
Helpers:
|
||||
- `model->ept.accumulated(state)` — total milliseconds spent in
|
||||
`state` since startup. Use it to populate E116 SVIDs.
|
||||
- `model->ept.reset_history()` — call at shift boundary.
|
||||
|
||||
### 4.2. Carriers + load ports (E87)
|
||||
|
||||
When AMHS delivers a carrier:
|
||||
|
||||
```cpp
|
||||
model->carriers.create("CAR-A1B2", /*port=*/1, /*capacity=*/25);
|
||||
model->carriers.fire_id_event("CAR-A1B2", gem::CarrierIDEvent::Read);
|
||||
// ... host sends S3F17(ProceedWithCarrier), the registered handler
|
||||
// in the Router calls fire_id_event(..., ProceedWithCarrier) and
|
||||
// CIDS moves NotConfirmed → Confirmed.
|
||||
```
|
||||
|
||||
When your slot-map scanner finishes:
|
||||
|
||||
```cpp
|
||||
auto* c = model->carriers.get("CAR-A1B2");
|
||||
for (std::size_t i = 0; i < scan_result.size(); ++i)
|
||||
c->slots[i].state = scan_result[i]; // 0 empty, 1 occupied
|
||||
model->carriers.fire_slot_map_event("CAR-A1B2", gem::SlotMapEvent::Read);
|
||||
```
|
||||
|
||||
The S3F19/F20 verify handler will compare against this map.
|
||||
|
||||
### 4.3. Substrates (E90)
|
||||
|
||||
For each wafer you start tracking:
|
||||
|
||||
```cpp
|
||||
model->substrates.create("W-2024-001", "CAR-A1B2", /*slot=*/1);
|
||||
model->substrates.fire_location_event(
|
||||
"W-2024-001", gem::SubstrateEvent::Acquire, /*location=*/"ChamberA");
|
||||
model->substrates.fire_processing_event(
|
||||
"W-2024-001", gem::SubstrateProcessingEvent::StartProcessing);
|
||||
// ... when done:
|
||||
model->substrates.fire_processing_event(
|
||||
"W-2024-001", gem::SubstrateProcessingEvent::EndProcessing);
|
||||
model->substrates.fire_location_event(
|
||||
"W-2024-001", gem::SubstrateEvent::Release, /*location=*/"OutCarrier");
|
||||
```
|
||||
|
||||
History is tracked per-substrate (`model->substrates.history(id)`)
|
||||
and can power your downtime / yield reports.
|
||||
|
||||
### 4.4. Process jobs + control jobs (E40 / E94)
|
||||
|
||||
The host creates these via S16F11 / S14F9. Your application drives
|
||||
their internal transitions as the recipe engine progresses:
|
||||
|
||||
```cpp
|
||||
// Recipe runner reports setup done:
|
||||
model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::SetupComplete);
|
||||
// Operator hits Start (or autorun is on):
|
||||
model->process_jobs.on_host_command("PJ-1", gem::ProcessJobEvent::Start);
|
||||
// Recipe completed normally:
|
||||
model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::ProcessComplete);
|
||||
```
|
||||
|
||||
CJ state cascades the same way (E94).
|
||||
|
||||
### 4.5. Alarms (E5 §13)
|
||||
|
||||
```cpp
|
||||
// Sensor crosses threshold:
|
||||
model->alarms.set(/*alid=*/1, /*set=*/true); // emits S5F1(ALCD=0x84)
|
||||
// Later it clears:
|
||||
model->alarms.set(1, false); // emits S5F1(ALCD=0x04)
|
||||
```
|
||||
|
||||
The dispatcher takes care of the wire frame — you just toggle.
|
||||
|
||||
### 4.6. E84 parallel I/O handoff (AMHS)
|
||||
|
||||
For each load port that talks to the AMHS robot:
|
||||
|
||||
```cpp
|
||||
#include "secsgem/gem/e84_asio_timers.hpp"
|
||||
|
||||
auto* fsm = model->e84_ports.get(/*port_id=*/1);
|
||||
if (!fsm) { model->e84_ports.create(1); fsm = model->e84_ports.get(1); }
|
||||
|
||||
// SEMI E84 §6 handshake timers. Defaults below are spec-typical; tune
|
||||
// per port. TA1=AMHS waits for L_REQ/U_REQ after VALID; TA2=equipment
|
||||
// waits for BUSY after port is ready; TA3=BUSY phase budget.
|
||||
fsm->set_timeouts({std::chrono::seconds(2),
|
||||
std::chrono::seconds(2),
|
||||
std::chrono::seconds(60)});
|
||||
|
||||
// Wire arm/cancel into asio so the FSM polices the real wall clock.
|
||||
auto driver = std::make_shared<gem::E84AsioTimers>(io.get_executor(), *fsm);
|
||||
driver->attach();
|
||||
// Keep `driver` alive for the lifetime of the FSM (e.g. as a member
|
||||
// of your per-port object).
|
||||
|
||||
// Optional: log handoff faults.
|
||||
fsm->set_fault_handler([port_id = 1](gem::E84Fault reason) {
|
||||
log("E84 port " + std::to_string(port_id) + " fault: " +
|
||||
gem::e84_fault_name(reason));
|
||||
});
|
||||
|
||||
// Now feed signal changes from your I/O bridge. On a real AMHS the
|
||||
// bridge polls or interrupts on the parallel-I/O lines:
|
||||
model->e84_ports.on_signal_change(1, gem::E84Signal::CS_0, true);
|
||||
model->e84_ports.on_signal_change(1, gem::E84Signal::VALID, true);
|
||||
// equipment side asserts when port is physically ready:
|
||||
model->e84_ports.on_signal_change(1, gem::E84Signal::L_REQ, true);
|
||||
// ... AMHS continues with BUSY / COMPT.
|
||||
```
|
||||
|
||||
If TA1, TA2, or TA3 expires the FSM transitions to `HandoffFault` and
|
||||
the fault handler fires with the precise `E84Fault` reason. Your
|
||||
application is then responsible for whatever the tool's fault policy is
|
||||
(typically: assert your local ES line and raise an alarm).
|
||||
|
||||
### 4.7. Recoverable exceptions (E5 §9, S5F9–F18)
|
||||
|
||||
For faults where you want a host/equipment recovery dialogue:
|
||||
|
||||
```cpp
|
||||
model->exceptions.post(/*exid=*/42, "VACUUM",
|
||||
"lost vacuum in chamber A",
|
||||
{"PURGE", "RECOVER", "ABORT"}); // emits S5F9
|
||||
// Host picks PURGE via S5F13; the registered handler calls
|
||||
// model->exceptions.on_recover(42, "PURGE"), state moves to Recovering.
|
||||
// Your purge routine completes:
|
||||
model->exceptions.fire_internal(42, gem::ExceptionEvent::RecoveryComplete);
|
||||
// state → Cleared; S5F11 fires; entry removed.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Persistence
|
||||
|
||||
GEM equipment that loses power mid-job can recover gracefully
|
||||
because every store the library ships supports an opt-in file-backed
|
||||
journal. Enable per store, at startup, BEFORE the connection comes up:
|
||||
|
||||
```cpp
|
||||
auto base = std::filesystem::path("/var/lib/acme/secsgem");
|
||||
model->spool.enable_persistence(base / "spool");
|
||||
model->carriers.enable_persistence(base / "carriers");
|
||||
model->load_ports.enable_persistence(base / "loadports");
|
||||
model->substrates.enable_persistence(base / "substrates");
|
||||
model->process_jobs.enable_persistence(base / "pjobs");
|
||||
model->control_jobs.enable_persistence(base / "cjobs");
|
||||
model->exceptions.enable_persistence(base / "exceptions");
|
||||
```
|
||||
|
||||
On enable, the store scans the directory, replays records into
|
||||
in-memory state, and from there keeps the directory in sync on
|
||||
every create / state-change / remove. Writes use a
|
||||
`.tmp + rename` pattern so a power loss mid-write can lose at most
|
||||
the in-flight record (older records remain coherent).
|
||||
|
||||
Storage budget per store, roughly:
|
||||
- spool: one file per spooled S6F11 (typically tens of bytes each)
|
||||
- carriers: one file per carrier (~50 bytes + slot count)
|
||||
- load_ports: one file per LP (~30 bytes)
|
||||
- substrates: one file per wafer (~80 bytes)
|
||||
- pjobs: one file per active PJ (~100 bytes), plus `order.idx`
|
||||
- cjobs: one file per active CJ (~80 bytes)
|
||||
- exceptions: one file per Posted/Recovering exception
|
||||
|
||||
Even a busy fab tool tops out at a few hundred files in each
|
||||
directory — well within filesystem caps. Sweep terminal-state
|
||||
entries (completed PJs, cleared exceptions) periodically if you
|
||||
care about directory size.
|
||||
|
||||
Caveats currently captured in the persistence tests:
|
||||
- Substrate **history** is intentionally NOT journaled — only the
|
||||
*current* state of each axis. Replay starts with an empty
|
||||
history vector.
|
||||
- PJ `rcpvars` / `prprocessparams` (the optional E40 `secs2::Item`
|
||||
trailers) are not journaled in v1; call `set_e40_extras` again on
|
||||
the application side after restart if you need them.
|
||||
|
||||
---
|
||||
|
||||
## 6. Monitoring + observability
|
||||
|
||||
### 6.1. Connection lifecycle
|
||||
|
||||
```cpp
|
||||
conn->set_log_handler([](const std::string& m) {
|
||||
syslog(LOG_INFO, "hsms: %s", m.c_str());
|
||||
});
|
||||
conn->set_selected_handler([] { metrics.inc("hsms.selected"); });
|
||||
conn->set_closed_handler([](const std::string& r) {
|
||||
metrics.inc("hsms.closed", {{"reason", r}});
|
||||
});
|
||||
```
|
||||
|
||||
### 6.2. State change observers
|
||||
|
||||
Every store / FSM exposes a `set_*_change_handler`. Hook them up
|
||||
to your metrics / log pipeline:
|
||||
|
||||
```cpp
|
||||
model->control_jobs.set_state_change_handler(
|
||||
[](const std::string& cj, gem::ControlJobState f, gem::ControlJobState t,
|
||||
gem::ControlJobEvent) {
|
||||
log("CJ " + cj + " " + gem::control_job_state_name(f) +
|
||||
" → " + gem::control_job_state_name(t));
|
||||
});
|
||||
```
|
||||
|
||||
### 6.3. Self-emitted protocol errors
|
||||
|
||||
Look for `S9F*` traffic in your logs. S9F3 / F5 mean the host
|
||||
asked for something your router doesn't handle; S9F7 means a bad
|
||||
body arrived; S9F9 means a reply didn't arrive in T3; S9F11 means
|
||||
a frame exceeded the 16 MiB cap. None of these are normal — they're
|
||||
real diagnostic events.
|
||||
|
||||
### 6.4. Prometheus exporter (worked example)
|
||||
|
||||
`include/secsgem/metrics/prometheus.hpp` ships a minimal Registry +
|
||||
asio-backed HTTP server. Drop it next to your equipment and scrape
|
||||
from your fab's Prometheus.
|
||||
|
||||
```cpp
|
||||
#include "secsgem/metrics/prometheus.hpp"
|
||||
namespace metrics = secsgem::metrics;
|
||||
|
||||
auto reg = std::make_shared<metrics::Registry>();
|
||||
reg->describe("secsgem_messages_total", "messages dispatched",
|
||||
metrics::MetricType::Counter);
|
||||
reg->describe("secsgem_alarms_active", "currently-active alarms",
|
||||
metrics::MetricType::Gauge);
|
||||
reg->describe("secsgem_spool_depth", "queued spool messages",
|
||||
metrics::MetricType::Gauge);
|
||||
reg->describe("secsgem_t_timer_total", "T-timer expiry by id",
|
||||
metrics::MetricType::Counter);
|
||||
|
||||
// HTTP /metrics on :9090. Same io_context as the HSMS connection —
|
||||
// scraping runs on the strand, so updates and reads serialize for free.
|
||||
auto exporter = std::make_shared<metrics::PrometheusServer>(io, 9090, reg);
|
||||
exporter->start();
|
||||
|
||||
// Wire counters into the connection + model hooks you already set up
|
||||
// in §6.1 / §6.2. These all fire on the io_context thread.
|
||||
conn->set_selected_handler([reg, conn] {
|
||||
reg->set_gauge("secsgem_hsms_selected", 1);
|
||||
});
|
||||
conn->set_closed_handler([reg](const std::string& reason) {
|
||||
reg->set_gauge("secsgem_hsms_selected", 0);
|
||||
// T-timer expirations surface here as `reason` starting with "T".
|
||||
if (!reason.empty() && reason[0] == 'T')
|
||||
reg->inc("secsgem_t_timer_total", {{"timer", reason.substr(0, 2)}});
|
||||
});
|
||||
|
||||
// Per-message dispatch — wrap your existing router.dispatch() call.
|
||||
auto orig_handler = conn->message_handler(); // (or whatever you set)
|
||||
conn->set_message_handler([reg, orig_handler](const secs2::Message& m) {
|
||||
reg->inc("secsgem_messages_total",
|
||||
{{"dir", "rx"},
|
||||
{"stream", std::to_string(m.stream)},
|
||||
{"function", std::to_string(m.function)}});
|
||||
return orig_handler(m);
|
||||
});
|
||||
|
||||
// Push gauge snapshots from a periodic timer on the same io.
|
||||
auto poll = std::make_shared<asio::steady_timer>(io);
|
||||
std::function<void(std::error_code)> tick = [reg, model, poll, &tick](std::error_code ec) {
|
||||
if (ec) return;
|
||||
reg->set_gauge("secsgem_spool_depth",
|
||||
static_cast<double>(model->spool.size()));
|
||||
std::size_t active = 0;
|
||||
for (auto& a : model->alarms.all())
|
||||
if (model->alarms.active(a.id)) ++active;
|
||||
reg->set_gauge("secsgem_alarms_active", static_cast<double>(active));
|
||||
poll->expires_after(std::chrono::seconds(5));
|
||||
poll->async_wait(tick);
|
||||
};
|
||||
poll->expires_after(std::chrono::seconds(5));
|
||||
poll->async_wait(tick);
|
||||
```
|
||||
|
||||
What lands at `/metrics`:
|
||||
|
||||
```
|
||||
# HELP secsgem_messages_total messages dispatched
|
||||
# TYPE secsgem_messages_total counter
|
||||
secsgem_messages_total{dir="rx",function="13",stream="1"} 42
|
||||
# TYPE secsgem_spool_depth gauge
|
||||
secsgem_spool_depth 7
|
||||
# TYPE secsgem_hsms_selected gauge
|
||||
secsgem_hsms_selected 1
|
||||
```
|
||||
|
||||
Wire this into your fab's Prometheus + Grafana and you've got the
|
||||
starter dashboard the README §3 table describes. The exporter has
|
||||
**no auth and no TLS** — drop nginx or Caddy in front with mTLS for
|
||||
production.
|
||||
|
||||
---
|
||||
|
||||
## 7. HSMS-GS: one tool, multiple MES
|
||||
|
||||
Most fab tools talk to one MES. Some — particularly tools shared by
|
||||
multiple production lines or sites — need to serve two or more MES
|
||||
schedulers simultaneously over a single HSMS-GS connection. E37 §11
|
||||
calls these "general sessions": one TCP socket, multiple session
|
||||
identifiers, independent SELECTED state per session.
|
||||
|
||||
The library models this as additional sessions on the same
|
||||
`hsms::Connection`:
|
||||
|
||||
```cpp
|
||||
server.on_connection([](std::shared_ptr<Connection> conn) {
|
||||
// Primary session (device_id=1) was registered by Server::Config;
|
||||
// add a second session for the second MES.
|
||||
conn->add_session(/*device_id=*/2);
|
||||
|
||||
// Per-session message routing — each MES gets a distinct dispatcher,
|
||||
// distinct SVID views, distinct alarm enable state, distinct
|
||||
// recipe namespace if you want. Or share state via a common
|
||||
// EquipmentDataModel and just route messages here.
|
||||
conn->set_session_message_handler(1, [model_1](const secs2::Message& m) {
|
||||
return router_1.dispatch(m);
|
||||
});
|
||||
conn->set_session_message_handler(2, [model_2](const secs2::Message& m) {
|
||||
return router_2.dispatch(m);
|
||||
});
|
||||
|
||||
// Per-session SELECT state observers. These fire when each MES
|
||||
// completes its Select.req handshake; independent of each other.
|
||||
conn->set_session_selected_handler(1, [] {
|
||||
log("MES-1 selected");
|
||||
});
|
||||
conn->set_session_selected_handler(2, [] {
|
||||
log("MES-2 selected");
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
When the equipment emits an unsolicited primary (S5F1, S6F11,
|
||||
S10F1), choose the session explicitly:
|
||||
|
||||
```cpp
|
||||
// Alarm goes to MES-1 only.
|
||||
conn->send_data(/*session_id=*/1, gem::s5f1_alarm_report(0x84, 1, "high"));
|
||||
|
||||
// Event report goes to both.
|
||||
auto event = gem::s6f11_event_report(0, 300, reports);
|
||||
conn->send_data(1, event);
|
||||
conn->send_data(2, event);
|
||||
```
|
||||
|
||||
### Active-mode (host side) GS
|
||||
|
||||
The host (active) connection initiates Select.req for each registered
|
||||
session serially — session 1 first, then once 1 reaches SELECTED,
|
||||
session 2. Customers building a multi-tool fleet controller use the
|
||||
same `add_session` API on the `Client`-derived `Connection`:
|
||||
|
||||
```cpp
|
||||
client.on_connection([](std::shared_ptr<Connection> conn) {
|
||||
conn->add_session(2); // a second tool's session
|
||||
conn->set_session_selected_handler(1, [] { /* tool A ready */ });
|
||||
conn->set_session_selected_handler(2, [] { /* tool B ready */ });
|
||||
});
|
||||
```
|
||||
|
||||
### Rejection semantics
|
||||
|
||||
A data frame whose `session_id` field doesn't match any registered
|
||||
session gets a Reject(EntityNotSelected) response, per E37 §7.7 — the
|
||||
peer's MES will see this and know to back off. See
|
||||
`tests/test_hsms_gs.cpp` for the wire-level coverage and
|
||||
`tests/test_hsms_gs_integration.cpp` for the end-to-end Server/Client
|
||||
pattern.
|
||||
|
||||
---
|
||||
|
||||
## 8. Recommended layout for a vendor application
|
||||
|
||||
```
|
||||
/opt/acme-secsgem/
|
||||
├─ bin/
|
||||
│ └─ secsgem-equipment # your built binary
|
||||
├─ etc/
|
||||
│ ├─ equipment.yaml # your tool's dictionary
|
||||
│ └─ control_state.yaml # your tool-specific state model
|
||||
├─ var/
|
||||
│ ├─ spool/ # populated at runtime
|
||||
│ ├─ carriers/
|
||||
│ ├─ substrates/
|
||||
│ ├─ pjobs/
|
||||
│ ├─ cjobs/
|
||||
│ └─ exceptions/
|
||||
└─ share/
|
||||
└─ doc/ # COMPLIANCE.md, INTEGRATION.md
|
||||
```
|
||||
|
||||
Your application reads `etc/`, writes to `var/`, and never touches
|
||||
`share/`. YAML edits don't require a rebuild — restart the
|
||||
process.
|
||||
|
||||
The control-state YAML is your tool's *processing* state machine —
|
||||
E30 deliberately leaves the concrete states (IDLE / SETUP / READY /
|
||||
EXECUTING / PAUSE / …) up to the tool builder. Copy
|
||||
`data/control_state.yaml` as a starting point.
|
||||
|
||||
---
|
||||
|
||||
## 9. Test the integration
|
||||
|
||||
Don't ship without:
|
||||
|
||||
1. **Round-trip every host-facing message you serve.** The library's
|
||||
own test suite covers the codecs; you should also drive your
|
||||
YAML's specific SVIDs / CEIDs / alarms / recipes against the
|
||||
built-in passive server using the `interop/host_vs_cpp_server.py`
|
||||
harness as a template.
|
||||
|
||||
2. **Power-loss simulation.** Kill -9 the process mid-job, restart,
|
||||
confirm the stores replay the correct state. The persistence
|
||||
tests give you a template; copy and parameterize for your store
|
||||
directories.
|
||||
|
||||
3. **Multi-hour soak.** Spool fills up if persistence is enabled and
|
||||
the host link is down — make sure your fab's MES side ack-rate
|
||||
keeps up. Run a 24h test with the host periodically disconnecting
|
||||
and watch the journal directory.
|
||||
|
||||
4. **The two-container demo** in this repo gives you a starting
|
||||
harness — the host emulator (`apps/secs_client.cpp`) drives
|
||||
~20 transactions through your server. Adapt it to your message
|
||||
set.
|
||||
|
||||
---
|
||||
|
||||
## 10. When to extend the runtime
|
||||
|
||||
The library is open to extension. Common reasons to add code:
|
||||
|
||||
- **A new SECS-II message** the catalog doesn't cover. Edit
|
||||
`data/messages.yaml`, run the codegen (built into the CMake
|
||||
pipeline), add a Router handler. No core code change.
|
||||
- **A new state machine** specific to your tool (e.g. an in-chamber
|
||||
cooling cycle FSM). Lift the pattern from `ept_state.hpp`:
|
||||
define your states + events + transition table; let your
|
||||
application drive it.
|
||||
- **An additional persistence backend** (DB instead of files).
|
||||
Mirror the spool `.enable_persistence` pattern — it's about 100
|
||||
lines per store.
|
||||
|
||||
If your change is broadly useful, it's worth landing in the library
|
||||
itself. See `COMPLIANCE.md` for the standards still on the
|
||||
"explicitly out of scope" list — anything there is a possible
|
||||
contribution.
|
||||
|
||||
---
|
||||
|
||||
## 11. Going from "stack" to "certified GEM tool"
|
||||
|
||||
This codebase passes its own conformance harness and cross-validates
|
||||
against `secsgem-py`, but a real *certified* GEM tool needs more:
|
||||
|
||||
- **Independent third-party validator**. Run a GEM RTS (Reference
|
||||
Test System) or equivalent against your integration. The library
|
||||
serves the messages; the validator decides whether your data is
|
||||
consistent.
|
||||
|
||||
- **Vendor application code**. The runtime cannot, by design, know
|
||||
what your SVID values *should* be at any given moment. That's
|
||||
your domain knowledge plugged into the data model and FSMs.
|
||||
|
||||
- **Documentation**. E30 §6.10 (Documentation capability) requires
|
||||
you to publish what you implement. `COMPLIANCE.md` in this repo
|
||||
is the template — fork it, prune to your actual coverage, ship
|
||||
it with your tool.
|
||||
|
||||
- **Operations**: monitoring dashboards, alarm escalation, log
|
||||
retention — the standard SRE concerns, no different from any
|
||||
other piece of fab software.
|
||||
@@ -0,0 +1,224 @@
|
||||
# Real-MES interop test plan
|
||||
|
||||
The codebase cross-validates against [secsgem-py](https://pypi.org/project/secsgem/)
|
||||
0.3.0 in CI. That's the Python reference implementation. It is **not
|
||||
what a fab actually runs.** Real MES stacks include:
|
||||
|
||||
- **Camstar** (Siemens / Opcenter Execution Semiconductor)
|
||||
- **FactoryWorks** (IBM, now various OEM ports)
|
||||
- **Inficon FabGuard**
|
||||
- **Wonderware MES** (AVEVA)
|
||||
- **Mozaic / Mozaic Suite**
|
||||
- **Critical Manufacturing CMNavigo**
|
||||
- **Eyelit MES**
|
||||
|
||||
Each one ships its own SECS/GEM stack with its own quirks. This doc
|
||||
is the structured day-1 protocol your integration team runs against
|
||||
**your** MES before connecting a real tool. Treat it as a punch
|
||||
list you tick off; capture wire traces from every step.
|
||||
|
||||
> **You can't skip this.** The in-repo `secs_conformance` harness +
|
||||
> the `interop/` secsgem-py cross-validation prove the codebase is
|
||||
> spec-conformant. They cannot prove the *combination* of (this
|
||||
> codebase, your YAML config, your MES's choice of optional
|
||||
> behaviours) works. Every gap surfaced in prior interop sweeps
|
||||
> (S1F23/F24 missing, S10F3 direction confusion, lenient U-width
|
||||
> parsing) was a real bug masked by passing internal tests.
|
||||
|
||||
## 0. Prerequisites
|
||||
|
||||
- A staging copy of the MES configured to talk to a single tool over
|
||||
HSMS-SS or HSMS-GS. Get the MES team to point it at your test
|
||||
equipment endpoint (port 5000 by default).
|
||||
- A wire-trace collector running on the equipment side:
|
||||
```sh
|
||||
tcpdump -i any -w mes-day1-$(date +%Y%m%d).pcap 'tcp port 5000'
|
||||
```
|
||||
Keep this running across every test below. Wireshark with the
|
||||
HSMS dissector (or the secsgem-py decoder in `interop/`) parses
|
||||
these traces after the fact.
|
||||
- The standard logger / metrics exporter wired up (see
|
||||
INTEGRATION.md §6.4) — you want to see the `S9F*` self-emissions
|
||||
and the FSM transitions land in your dashboard.
|
||||
- The `--validate-config` flag run clean on your equipment YAMLs.
|
||||
|
||||
Record each test's outcome in a single spreadsheet — test ID,
|
||||
date, MES build, equipment build, PASS/FAIL/PARTIAL, notes,
|
||||
wire-trace timestamp. This is your audit trail for the integration.
|
||||
|
||||
## 1. HSMS transport plumbing (E37)
|
||||
|
||||
| ID | Test | Expected wire behaviour | Common quirks |
|
||||
|-------|-------------------------------------------------------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| T-01 | MES connects (active) → Select.req | Equipment replies Select.rsp(0=Ok); SELECTED on both sides | Some MES default session ID ≠ 0; override in our YAML |
|
||||
| T-02 | Idle Linktest cycle (5 min observation) | Linktest.req every N seconds from MES; equipment Linktest.rsp | Camstar uses 30s default; FactoryWorks 60s |
|
||||
| T-03 | MES sends Separate.req → graceful close | Equipment closes socket within 1s; no FIN_WAIT2 leak | Some MES expect equipment to close first |
|
||||
| T-04 | Equipment-initiated Linktest (set `Timers::linktest` to 10s) | MES replies Linktest.rsp | Mozaic ignores equipment-initiated; not a bug |
|
||||
| T-05 | MES disconnects TCP without Separate | Equipment detects closed socket, fires `closed_handler("eof")` | Watch for spool starting to fill |
|
||||
| T-06 | MES reconnects after T-05 → S6F25 if spool has content | Equipment auto-emits S6F25 with queued count | MES must support S6F25/F26 — older Wonderware doesn't |
|
||||
| T-07 | T3 violation: MES sends primary but never replies | Equipment fires T3 → auto-emits S9F9 | Confirm S9F9 appears in trace, not just our logs |
|
||||
| T-08 | T7 violation: MES connects but never sends Select.req | Equipment closes after T7 with reason "T7" | |
|
||||
| T-09 | Oversized frame: MES sends 17 MiB body | Equipment auto-emits S9F11 + closes | If MES doesn't generate this, skip |
|
||||
|
||||
## 2. Establish comms + identification (E30 Fundamental)
|
||||
|
||||
| ID | Test | Expected wire behaviour | Common quirks |
|
||||
|-------|-------------------------------------------------------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| E-01 | MES → S1F13 | Equipment → S1F14(COMMACK=Accept, [MDLN, SOFTREV]) | MES that send empty MDLN list — check YAML matches |
|
||||
| E-02 | MES → S1F1 | Equipment → S1F2(MDLN, SOFTREV) | EQPTYP in S1F20 is sometimes confused with MDLN |
|
||||
| E-03 | MES → S1F15 (request offline) | Equipment → S1F16(OFLACK=Accept), control state → HostOffline | |
|
||||
| E-04 | MES → S1F17 (request online) | Equipment → S1F18(ONLACK=Accept), control state → OnlineRemote | |
|
||||
| E-05 | MES → S1F19 (compliance request) | Equipment → S1F20 with full CCODE list matching `capabilities:` YAML | |
|
||||
| E-06 | MES → S1F11 (SVID namelist), then S1F3 (values) | Namelist matches `svids:` YAML; values within type-declared ranges | MES may expect SVID 1 = ControlState string |
|
||||
| E-07 | MES → S1F21 (DVID namelist), then S1F3 (values for DVID set) | Equipment returns DVIDs separately from SVIDs | Some MES treat S1F3 as union; spec says SVID-only |
|
||||
| E-08 | MES → S1F23 (CEID namelist) | Equipment returns CEID→VIDs map matching `events.link_event_reports` | |
|
||||
|
||||
## 3. Dynamic event reports (E30 §6.6)
|
||||
|
||||
This is where most integrations break. MES often defines reports
|
||||
in a different order than spec, expects different ACK enum
|
||||
positions, or uses non-standard RPTID/DATAID widths.
|
||||
|
||||
| ID | Test | Expected | Common quirks |
|
||||
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| R-01 | MES → S2F33 (define report 1000 over 3 SVIDs) | S2F34(DRACK=0=Accept) | MES may use U1 for RPTID; our parser accepts widths |
|
||||
| R-02 | MES → S2F35 (link CEID 300 ↔ RPTID 1000) | S2F36(LRACK=0=Accept) | |
|
||||
| R-03 | MES → S2F37 (enable CEID 300) | S2F38(ERACK=0=Accept) | Some MES send empty CEID list = enable-all |
|
||||
| R-04 | Equipment fires CEID 300 (driven by `model->compose_reports_for(300)`) | MES → S6F12(ACKC6=0) | MES may take >T6 to reply — extend our T6 if needed |
|
||||
| R-05 | MES → S6F15 (event report request) | Equipment → S6F16 with current values | |
|
||||
| R-06 | MES → S6F19/F21 (individual / annotated report request) | Equipment → S6F20 / S6F22 with current values | |
|
||||
| R-07 | MES → S2F33 with DATAID=0 (clear all reports) | S2F34(DRACK=0); all link bindings flushed | Older MES use S2F33 + empty body; check both forms |
|
||||
|
||||
## 4. Alarms (E30 §6.14, E5 §13)
|
||||
|
||||
| ID | Test | Expected | Common quirks |
|
||||
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| A-01 | MES → S5F5 (list alarms) | S5F6 with full alarm directory matching `alarms:` YAML | |
|
||||
| A-02 | MES → S5F3 (enable ALID 1) | S5F4(ACKC5=0) | |
|
||||
| A-03 | MES → S5F7 (list enabled) | S5F8 contains ALID 1 | |
|
||||
| A-04 | Equipment sets alarm 1 active | S5F1(ALCD=0x80 \| category) sent to MES → MES S5F2 ack | ALCD bit-7 must be SET, not cleared |
|
||||
| A-05 | Equipment clears alarm 1 | S5F1(ALCD=0x00 \| category) → S5F2 ack | |
|
||||
| A-06 | Equipment fires an alarm while alarm is *disabled* (S5F5 says no) | NO S5F1 wire frame — alarm registry tracks active, dispatcher gates | Easy to get wrong; covered by our tests |
|
||||
|
||||
## 5. Remote control (E30 §6.15)
|
||||
|
||||
| ID | Test | Expected | Common quirks |
|
||||
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| C-01 | MES → S2F41 (RCMD=START, params) | S2F42(HCACK=0=Accept) + emit configured CEID | |
|
||||
| C-02 | MES → S2F21 (legacy: RCMD only, no params) | S2F22(CMDA=0) | Old Wonderware uses F21 exclusively |
|
||||
| C-03 | MES → S2F49 (enhanced: OBJSPEC + CPACK) | S2F50(HCACK=0) | |
|
||||
| C-04 | MES → S2F41 RCMD=UNKNOWN | S2F42(HCACK=1=InvalidCommand) | |
|
||||
|
||||
## 6. Process programs (E30 §6.17)
|
||||
|
||||
| ID | Test | Expected | Common quirks |
|
||||
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| P-01 | MES → S7F19 (list) | S7F20 with all PPIDs from `recipes:` YAML | |
|
||||
| P-02 | MES → S7F5(PPID=RECIPE-A) | S7F6 with body (ASCII default; binary if recipe is binary) | PPBODY direction was our biggest interop bug |
|
||||
| P-03 | MES → S7F1 → S7F2(Accept) → S7F3 (PP send, new PPID) | S7F4(ACKC7=0); recipe in store | |
|
||||
| P-04 | MES → S7F17 (delete) | S7F18(ACKC7=0); recipe removed | |
|
||||
| P-05 | MES → S7F23 (E42 formatted PP send) | S7F24(ACKC7=0) | Many MES don't speak E42; OK to skip if unused |
|
||||
|
||||
## 7. Terminal services (E30 §6.19)
|
||||
|
||||
| ID | Test | Expected | Common quirks |
|
||||
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| TS-01 | MES → S10F3 (terminal display single, host→equipment) | S10F4(ACKC10=0) | OUR codebase's bug fix — confirm direction is right |
|
||||
| TS-02 | MES → S10F5 (terminal display multi) | S10F6(ACKC10=0) | |
|
||||
| TS-03 | Equipment → S10F1 (operator request, equipment→host) | MES → S10F2(ACKC10=0) | Some MES don't accept S10F1 at all; document policy |
|
||||
|
||||
## 8. GEM 300 streams (E40 / E87 / E94)
|
||||
|
||||
| ID | Test | Expected | Common quirks |
|
||||
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| J-01 | MES → S16F11 PRJobCreate (full body: MF / PPID / mtrloutspec / params) | S16F12(HCACK=0) | MES often uses MF=Substrate; check |
|
||||
| J-02 | MES → S14F9 CreateControlJob with [PJ-1] | S14F10(OBJACK=0) | |
|
||||
| J-03 | MES → S16F27 CJSTART | S16F28(HCACK=0); cascade through PJ states; S16F9 alerts on each | This is the heaviest dance — capture full trace |
|
||||
| J-04 | Equipment fires S6F11(CEID=ControlJobExecuting=400) | MES → S6F12(ACKC6=0) | CEID 400/401 must be in `ceids:` YAML |
|
||||
| J-05 | MES → S3F17 (CarrierAction=ProceedWithCarrier) | S3F18(CAACK=0) | |
|
||||
| J-06 | MES → S3F19 (slot map verify) against carrier with stored map | S3F20(SMACK=0 if match, =1 if mismatch) | MES may not store maps; expect SMACK=NotRead |
|
||||
|
||||
## 9. Spool (E30 §6.22)
|
||||
|
||||
| ID | Test | Expected | Common quirks |
|
||||
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| SP-01 | MES → S2F43 (set spoolable streams to {5, 6}) | S2F44(RSPACK=0) | |
|
||||
| SP-02 | Disconnect MES, equipment fires S5F1+S6F11 | Both queued in spool (size grows) | |
|
||||
| SP-03 | Reconnect MES | Equipment auto-emits S6F25(queued count) | |
|
||||
| SP-04 | MES → S6F23(RSDC=0=Transmit) | S6F24(RSDA=0=Accept), then spooled messages drain in order | |
|
||||
| SP-05 | MES → S6F23(RSDC=1=Purge) | S6F24(RSDA=0); spool size → 0 | |
|
||||
|
||||
## 10. Clock + ECs (E30 §6.16, §6.20)
|
||||
|
||||
| ID | Test | Expected | Common quirks |
|
||||
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| K-01 | MES → S2F17 (clock read) | S2F18(YYYYMMDDhhmmsscc) — 16 chars | Some MES expect 14-char; we accept both |
|
||||
| K-02 | MES → S2F31 (clock set) | S2F32(TIACK=0=Accept) | Do this in a maintenance window only |
|
||||
| K-03 | MES → S2F29 (EC namelist) | S2F30 with all ECIDs | |
|
||||
| K-04 | MES → S2F13 (EC values) | S2F14 with current values | |
|
||||
| K-05 | MES → S2F15 (EC set within range) | S2F16(EAC=0=Accept); value reflected on next S2F13 | |
|
||||
| K-06 | MES → S2F15 (EC set OUT of range) | S2F16(EAC=1=Denied_OutOfRange) | Triggers our `min_str`/`max_str` range check |
|
||||
|
||||
## 11. Soak (4-24 hours)
|
||||
|
||||
Once the punch list above passes, leave the connection up overnight
|
||||
with synthetic transactions every 60 s and a real recipe running
|
||||
periodically. Watch for:
|
||||
|
||||
- Memory growth in equipment process RSS (should be flat)
|
||||
- Spool growing unbounded (something is filling it without draining)
|
||||
- T-timer expirations in equipment logs (network-layer trouble)
|
||||
- MES-side "stale tool" alarms (linktest reply lagging)
|
||||
|
||||
## 12. Pre-cutover checklist
|
||||
|
||||
Before promoting from staging to a real tool on the production fab
|
||||
floor:
|
||||
|
||||
- [ ] Every test ID above marked PASS or "N/A - documented reason"
|
||||
- [ ] Wire traces archived for every test (90-day retention minimum)
|
||||
- [ ] `secs_conformance --host <staging-tool>` exits 0
|
||||
- [ ] 4-hour soak with no abnormal spool growth or T-timer expiries
|
||||
- [ ] Dashboard panels for: spool depth, T-timer counter, alarm
|
||||
active count, FSM state gauges
|
||||
- [ ] Runbook entries for every incident in README §10
|
||||
- [ ] License agreement with the copyright holder signed (see
|
||||
[LICENSE](LICENSE))
|
||||
|
||||
## 13. Known MES quirks worth pre-empting
|
||||
|
||||
Compiled from prior fab integrations. Not exhaustive; treat as
|
||||
search-priors when something doesn't behave the way you expect.
|
||||
|
||||
- **Camstar Opcenter**: ALCD bit-7 sometimes inverted in their
|
||||
internal model; double-check the alarm wire trace. Linktest at
|
||||
30 s default.
|
||||
- **FactoryWorks**: Uses S2F21 (legacy) almost exclusively over
|
||||
S2F41. Make sure your `host_commands` registry routes both.
|
||||
- **Wonderware MES**: Doesn't reliably support S6F25; spool-drain
|
||||
flow may need explicit S6F23 from MES. Older versions don't
|
||||
speak E42 (S7F23 family) at all.
|
||||
- **Mozaic**: Sometimes sends S2F33 with DATAID encoded as U1
|
||||
where the spec allows U1-U8 — make sure our lenient parser is on
|
||||
(it is by default since the secsgem-py interop work).
|
||||
- **Inficon FabGuard**: Strict on S1F1 — expects MDLN and SOFTREV
|
||||
in ASCII even if the YAML uses U-types for MDLN elsewhere.
|
||||
- **CMNavigo**: Expects equipment to initiate S1F13 within 5 s of
|
||||
SELECT, before MES sends its own. Configure `Timers::linktest`
|
||||
+ a startup S1F13 emitter.
|
||||
|
||||
## 14. Reporting back
|
||||
|
||||
If you find an MES-specific bug that this codebase needs to handle,
|
||||
file it via `raphael@maenle.net` with:
|
||||
|
||||
1. MES name + build version
|
||||
2. The test ID from this doc
|
||||
3. Wire trace excerpt (pcap clip is fine)
|
||||
4. Expected vs actual behaviour
|
||||
5. Your equipment YAML + secs-gem commit SHA
|
||||
|
||||
Bugs surfaced through this process are how we got
|
||||
[S1F23/F24](interop/README.md), [S10F3 direction
|
||||
fix](interop/README.md), and the lenient identifier-width parser.
|
||||
The interop sweep is the gift that keeps giving.
|
||||
@@ -0,0 +1,72 @@
|
||||
# Proof of feature-completeness
|
||||
|
||||
"Feature-complete" is a claim that the code must prove, not the
|
||||
README. These eight commands are the proof. If they all exit zero
|
||||
on a fresh clone, the codebase implements what
|
||||
[COMPLIANCE.md](COMPLIANCE.md) claims.
|
||||
|
||||
| # | Command | What it proves |
|
||||
|---|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
|
||||
| 1 | `docker compose run --rm tests` | **445 test cases / 2 753 assertions** pass: every store, FSM, codec, parser, persistence path |
|
||||
| 2 | `docker compose run --rm builder /app/build/secs_conformance --host server --port 5000` | **47 wire-level conformance checks** PASS against a live passive equipment |
|
||||
| 3 | `docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py --host server` | **31 interop checks** PASS against secsgem-py 0.3.0 (the Python reference impl) |
|
||||
| 4 | `SECSGEM_ROBUSTNESS_SOAK=1 docker compose run --rm builder /app/build/secsgem_tests -tc='*soak*'` | **100 000 random tool operations** execute with all invariants and persistence round-trips holding |
|
||||
| 5 | `docker compose run --rm builder /app/build/secs_server --validate-config --config /app/data/equipment.yaml --state-table /app/data/control_state.yaml --pj-state-table /app/data/process_job_state.yaml --cj-state-table /app/data/control_job_state.yaml` | Every shipped YAML config passes structural + referential validation |
|
||||
| 6 | `docker compose run --rm builder bash /app/interop/tshark_validate.sh` | **69 HSMS frames** dissected by Wireshark's HSMS dissector (independent third codec) with no malformed packets |
|
||||
| 7 | `bash interop/secs4j_validate.sh` | **55 cross-validation checks** PASS against [secs4java8](https://github.com/kenta-shimizu/secs4java8) (independent Java implementation), covering S1/S2/S3/S5/S6/S7/S10/S14/S16, the full E40 PJ body, dynamic event reports + unsolicited S6F11 / S5F1 observation, alarm management, spool, PP management, terminal services, limits, trace, E39, and the GEM 300 streams secsgem-py couldn't easily drive |
|
||||
| 8 | `cmake -B build-fuzz -DSECSGEM_FUZZ=ON && build-fuzz/fuzz_secs2_decode -max_total_time=60` | **200 000+ random inputs** through `secs2::decode`, **1.4 M+** through `try_parse_sml` per 60 s lane, ASan + UBSan + libFuzzer coverage, **0 crashes** |
|
||||
|
||||
Plus, on every push to `main`, [Gitea Actions](.gitea/workflows/ci.yml)
|
||||
runs both a **Release build + full test suite** and a separate
|
||||
**ThreadSanitizer lane** that builds with `-fsanitize=thread` and
|
||||
fails on any race. All 445 cases / 2 753 assertions pass under TSan
|
||||
clean.
|
||||
|
||||
## Per-standard test coverage
|
||||
|
||||
Every claimed standard has dedicated tests. Counts are
|
||||
`grep -c TEST_CASE`; cross-cutting tests (e.g. `test_robustness_fuzz`,
|
||||
`test_gem300_scenario`) exercise multiple standards in concert.
|
||||
|
||||
| Standard | Test files | Cases |
|
||||
|-----------------------------------|-------------------------------------------------------------------------------------------|------:|
|
||||
| **E5** — SECS-II encoding | `test_secs2`, `test_sml`, `test_messages`, `test_e5_kat`, `test_identifier_wildcards`, `test_fuzz` | 139 |
|
||||
| **E5 §13** — exceptions | `test_exceptions`, `test_exception_persistence` | 16 |
|
||||
| **E4** — SECS-I transport | `test_secsi`, `test_secsi_timers`, `test_secsi_tcp` | 27 |
|
||||
| **E37** — HSMS (SS + GS) | `test_hsms`, `test_hsms_connection`, `test_hsms_timers`, `test_hsms_s9`, `test_hsms_gs`, `test_hsms_gs_integration`, `test_s9_fallback`, `test_concurrency` | 34 |
|
||||
| **E30** — GEM core | `test_control_state`, `test_communication_state`, `test_host_handler`, `test_data_model`, `test_loader`, `test_config_validate` | 71 |
|
||||
| **E40** — process jobs | `test_process_jobs` | 21 |
|
||||
| **E94** — control jobs | `test_control_jobs` | 9 |
|
||||
| **E42** — formatted PP | `test_e42_formatted_pp` | 6 |
|
||||
| **E87** — carriers + load ports | `test_carriers`, `test_carrier_state`, `test_carrier_persistence`, `test_e87_wire_scenarios` | 27 |
|
||||
| **E90** — substrate tracking | `test_substrates`, `test_substrate_persistence` | 21 |
|
||||
| **E116** — EPT | `test_ept` | 7 |
|
||||
| **E120 / E39** — common equip / object service | `test_cem_objects` | 3 |
|
||||
| **E157** — module process tracking | `test_modules` | 5 |
|
||||
| **E84** — parallel I/O + timers | `test_e84`, `test_e84_ports`, `test_e84_timers`, `test_e84_asio_timers` | 27 |
|
||||
| Persistence + cross-cutting | `test_job_persistence`, `test_persistence_upgrade`, `test_wire_ceid_emission`, `test_gem300_scenario`, `test_live_gem300`, `test_thread_safety`, `test_metrics_prometheus`, `test_robustness_fuzz` | 32 |
|
||||
| **Total** | | **445** |
|
||||
|
||||
A single command to see this live: `docker compose run --rm builder
|
||||
/app/build/secsgem_tests --list-test-cases | wc -l` (currently 445).
|
||||
|
||||
## What each proof actually demonstrates
|
||||
|
||||
The eight commands above split into four kinds of evidence:
|
||||
|
||||
- **Internal** (#1, #2, #4, #5) — our code testing our code: unit
|
||||
suite, conformance harness, soak property test, config validator.
|
||||
Necessary but not independent.
|
||||
- **External, second implementation** (#3, #7) — round-trip against
|
||||
secsgem-py 0.3.0 (Python) and secs4java8 (Java). Two independent
|
||||
SECS implementations must agree with us on every frame.
|
||||
- **External, third codec** (#6) — Wireshark's HSMS dissector,
|
||||
written by network-protocol authors who don't share code with
|
||||
either of us. Catches framing bugs the implementations might both
|
||||
share.
|
||||
- **External, structural search** (#8) — libFuzzer + ASan + UBSan
|
||||
exploring the decoder and SML parser surface for crashes, memory
|
||||
errors, and UB.
|
||||
|
||||
See [VERIFICATION.md](VERIFICATION.md) for the full test plan and
|
||||
the rationale for each external validator.
|
||||
@@ -0,0 +1,365 @@
|
||||
# Security operations guide
|
||||
|
||||
HSMS is the spec's wire protocol: plain TCP, no auth, no encryption.
|
||||
That's what every fab tool ships and what every MES expects, and we
|
||||
don't change it. Security comes from the network layer around the
|
||||
HSMS socket. This doc has the concrete configs you'll need; no
|
||||
hand-waving.
|
||||
|
||||
> If you're shipping to a production fab, treat every section here
|
||||
> as mandatory unless your fab security architect signs off on a
|
||||
> deviation in writing. HSMS on an exposed network with no controls
|
||||
> is how an unauthenticated MES impersonation incident becomes a
|
||||
> wafer-loss event.
|
||||
|
||||
## 1. Network isolation
|
||||
|
||||
### 1.1 Subnet placement
|
||||
|
||||
HSMS must run on a **control LAN** — physically or VLAN-separated
|
||||
from corporate / engineering networks. The MES host's IP is the
|
||||
only thing that should be able to reach the equipment's HSMS port.
|
||||
|
||||
### 1.2 Host firewall (nftables example)
|
||||
|
||||
Drop in `/etc/nftables.d/50-secsgem.nft`, then `systemctl reload
|
||||
nftables`:
|
||||
|
||||
```nftables
|
||||
table inet filter {
|
||||
set mes_hosts {
|
||||
type ipv4_addr
|
||||
flags interval
|
||||
elements = {
|
||||
10.40.1.10, # camstar-primary.fab.example
|
||||
10.40.1.11, # camstar-standby.fab.example
|
||||
}
|
||||
}
|
||||
|
||||
chain input {
|
||||
type filter hook input priority filter; policy drop;
|
||||
|
||||
# Allow established + loopback unconditionally.
|
||||
ct state established,related accept
|
||||
iifname "lo" accept
|
||||
|
||||
# HSMS port: only from known MES hosts.
|
||||
tcp dport 5000 ip saddr @mes_hosts accept
|
||||
|
||||
# Prometheus exporter on :9090: only from monitoring subnet.
|
||||
tcp dport 9090 ip saddr 10.40.99.0/24 accept
|
||||
|
||||
# SSH for ops: only from the bastion.
|
||||
tcp dport 22 ip saddr 10.40.99.1 accept
|
||||
|
||||
# Anything else is dropped (policy default).
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Test the ruleset against a known-bad source before reloading:
|
||||
|
||||
```sh
|
||||
nft -c -f /etc/nftables.d/50-secsgem.nft # syntax check
|
||||
nft list set inet filter mes_hosts # confirm the set is loaded
|
||||
```
|
||||
|
||||
### 1.3 Pod-network policy (Kubernetes / K3s deployments)
|
||||
|
||||
If you're running the equipment in a pod, use a `NetworkPolicy`:
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: secsgem-equipment-ingress
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: secsgem-equipment
|
||||
policyTypes: [Ingress]
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
tier: mes
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: camstar-host
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5000
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
tier: monitoring
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9090
|
||||
```
|
||||
|
||||
Calico, Cilium, or whatever your CNI is all enforce the same.
|
||||
|
||||
## 2. TLS tunnel for cross-site HSMS
|
||||
|
||||
For most fabs the control LAN is good enough. Cross-site HSMS (rare
|
||||
but real for shared-MES architectures) needs encryption. **Do not
|
||||
modify the HSMS wire protocol** — wrap the TCP socket in stunnel
|
||||
or a sidecar TLS proxy.
|
||||
|
||||
### 2.1 stunnel.conf — equipment side (terminator)
|
||||
|
||||
```ini
|
||||
; /etc/stunnel/secsgem-equipment.conf
|
||||
foreground = no
|
||||
pid = /run/stunnel/secsgem-equipment.pid
|
||||
|
||||
setuid = stunnel
|
||||
setgid = stunnel
|
||||
|
||||
debug = 5
|
||||
syslog = yes
|
||||
|
||||
[secsgem-hsms]
|
||||
accept = 0.0.0.0:5443 ; TLS port the MES connects to
|
||||
connect = 127.0.0.1:5000 ; equipment HSMS listener (localhost)
|
||||
|
||||
cert = /etc/stunnel/certs/equipment.fab.example.crt
|
||||
key = /etc/stunnel/certs/equipment.fab.example.key
|
||||
|
||||
CAfile = /etc/stunnel/certs/mes-ca-bundle.crt
|
||||
verifyChain = yes
|
||||
verifyPeer = yes
|
||||
checkHost = camstar-primary.fab.example
|
||||
|
||||
sslVersionMin = TLSv1.3
|
||||
ciphers = TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
|
||||
```
|
||||
|
||||
Bind the C++ server to `127.0.0.1` only (so the cleartext socket isn't
|
||||
reachable from the network):
|
||||
|
||||
```sh
|
||||
secs_server --port 5000 --bind 127.0.0.1 \
|
||||
--config /etc/acme-secsgem/equipment.yaml ...
|
||||
```
|
||||
|
||||
(The `--bind` flag is a small addition you'll need to add to
|
||||
`apps/secs_server.cpp` if you adopt this pattern — the demo binary
|
||||
binds INADDR_ANY today. Filed as a follow-up.)
|
||||
|
||||
### 2.2 stunnel.conf — MES side (initiator)
|
||||
|
||||
```ini
|
||||
; /etc/stunnel/secsgem-host.conf
|
||||
[secsgem-hsms]
|
||||
client = yes
|
||||
accept = 127.0.0.1:5000 ; MES connects here as if it were the equipment
|
||||
connect = equipment.fab.example:5443
|
||||
|
||||
CAfile = /etc/stunnel/certs/equipment-ca-bundle.crt
|
||||
verifyChain = yes
|
||||
verifyPeer = yes
|
||||
|
||||
; mTLS — present a client cert the equipment-side CA trusts.
|
||||
cert = /etc/stunnel/certs/camstar-primary.fab.example.crt
|
||||
key = /etc/stunnel/certs/camstar-primary.fab.example.key
|
||||
|
||||
sslVersionMin = TLSv1.3
|
||||
```
|
||||
|
||||
### 2.3 Performance impact
|
||||
|
||||
TLS adds ~50 µs per round-trip on modern hardware (measured via
|
||||
`secs_bench` with stunnel in the loop vs. direct connection). At a
|
||||
few hundred S6F11 events/sec sustained that's invisible. Don't skip
|
||||
TLS for performance reasons unless your latency budget is genuinely
|
||||
sub-millisecond.
|
||||
|
||||
## 3. Authentication
|
||||
|
||||
HSMS itself has no peer auth — Select.req sends a session ID and
|
||||
that's it. Two production-grade defenses:
|
||||
|
||||
1. **mTLS via the sidecar above** — the MES has to present a client
|
||||
cert signed by your fab's CA. Without it, the TLS handshake fails
|
||||
before HSMS is touched.
|
||||
|
||||
2. **Per-tool firewall ACLs** — even with mTLS, restrict the source
|
||||
IPs (§1.2 / §1.3). Defense in depth.
|
||||
|
||||
Do not try to add auth at the HSMS layer. No commercial MES would
|
||||
accept the protocol change, and the wire spec is what makes the
|
||||
codebase auditable.
|
||||
|
||||
## 4. YAML config integrity
|
||||
|
||||
`equipment.yaml`, `control_state.yaml`, the two job tables, and
|
||||
`messages.yaml` together define the equipment's behaviour. An
|
||||
attacker who can rewrite any of them owns the SECS/GEM surface.
|
||||
|
||||
### 4.1 Signing with minisign
|
||||
|
||||
[`minisign`](https://jedisct1.github.io/minisign/) is the smallest
|
||||
viable signing tool — single binary, single keypair file, Ed25519
|
||||
under the hood, used by Wasmer / OpenBSD / others. Two-line install:
|
||||
|
||||
```sh
|
||||
apt-get install -y minisign # Ubuntu 24.04
|
||||
minisign -G -p /etc/acme-secsgem/keys/acme.pub \
|
||||
-s ~/.minisign/acme.sec
|
||||
```
|
||||
|
||||
Sign every config bundle at deployment time:
|
||||
|
||||
```sh
|
||||
cd /etc/acme-secsgem
|
||||
minisign -S -s ~/.minisign/acme.sec equipment.yaml
|
||||
minisign -S -s ~/.minisign/acme.sec control_state.yaml
|
||||
minisign -S -s ~/.minisign/acme.sec process_job_state.yaml
|
||||
minisign -S -s ~/.minisign/acme.sec control_job_state.yaml
|
||||
# .minisig files appear next to each.
|
||||
```
|
||||
|
||||
Verify on the tool before the server starts (systemd ExecStartPre):
|
||||
|
||||
```sh
|
||||
#!/usr/bin/env bash
|
||||
# /usr/local/libexec/secsgem-verify-configs.sh
|
||||
set -euo pipefail
|
||||
ETC=/etc/acme-secsgem
|
||||
PUB=${ETC}/keys/acme.pub
|
||||
for f in equipment.yaml control_state.yaml \
|
||||
process_job_state.yaml control_job_state.yaml; do
|
||||
minisign -V -p "$PUB" -m "${ETC}/$f"
|
||||
done
|
||||
```
|
||||
|
||||
Wire into systemd:
|
||||
|
||||
```ini
|
||||
[Service]
|
||||
ExecStartPre=/usr/local/libexec/secsgem-verify-configs.sh
|
||||
ExecStart=/usr/local/bin/secs_server --config /etc/acme-secsgem/equipment.yaml ...
|
||||
```
|
||||
|
||||
If any signature fails, the unit refuses to start. Misconfiguration
|
||||
incidents drop dramatically when this is in place.
|
||||
|
||||
### 4.2 Validate before signing
|
||||
|
||||
Always run `secs_server --validate-config` against the YAML before
|
||||
signing it. Signing a broken config just transmits the breakage
|
||||
cryptographically:
|
||||
|
||||
```sh
|
||||
secs_server --validate-config \
|
||||
--config equipment.yaml \
|
||||
--state-table control_state.yaml \
|
||||
--pj-state-table process_job_state.yaml \
|
||||
--cj-state-table control_job_state.yaml \
|
||||
|| { echo "config invalid; refusing to sign"; exit 1; }
|
||||
minisign -S -s ~/.minisign/acme.sec equipment.yaml
|
||||
```
|
||||
|
||||
## 5. Audit logging for SIEM
|
||||
|
||||
Every wire frame should be retrievable for a configurable retention
|
||||
window (90 days is the common ask). The library exposes a log hook
|
||||
on `hsms::Connection`; ship JSON-line records to your SIEM.
|
||||
|
||||
### 5.1 Recommended JSON schema
|
||||
|
||||
```json
|
||||
{
|
||||
"@timestamp": "2026-06-09T14:23:55.412Z",
|
||||
"host": "tool-acme-pvd-3000-01",
|
||||
"session_id": 0,
|
||||
"direction": "rx",
|
||||
"stream": 2,
|
||||
"function": 41,
|
||||
"system_bytes": 1234567890,
|
||||
"reply_expected": true,
|
||||
"body_sml": "<L [2] <A 'START'> <L [0]>>",
|
||||
"body_bytes": 36,
|
||||
"elapsed_ms_since_select": 84210
|
||||
}
|
||||
```
|
||||
|
||||
One line per frame. Stream → splunk-forwarder / vector.dev / fluent-bit
|
||||
→ your fab's SIEM.
|
||||
|
||||
### 5.2 Wiring it up
|
||||
|
||||
```cpp
|
||||
conn->set_log_handler([&](const std::string& msg) {
|
||||
// The connection's built-in log_handler gets a free-text line.
|
||||
// For structured logging, intercept at the message_handler level:
|
||||
// wrap router.dispatch and emit JSON for each frame in/out.
|
||||
syslog(LOG_LOCAL0 | LOG_INFO, "secsgem: %s", msg.c_str());
|
||||
});
|
||||
|
||||
// Structured frame log via a wrapped dispatcher:
|
||||
conn->set_message_handler([&](const secs2::Message& m) {
|
||||
emit_audit_json("rx", m);
|
||||
auto reply = router.dispatch(m);
|
||||
if (reply) emit_audit_json("tx", *reply);
|
||||
return reply;
|
||||
});
|
||||
```
|
||||
|
||||
Where `emit_audit_json` writes a single line in the schema above to
|
||||
a file `vector.dev` is tailing, or to systemd-journal with `sd_journal_send`.
|
||||
|
||||
### 5.3 What to alert on
|
||||
|
||||
Threshold rules in the SIEM that should page on-call:
|
||||
|
||||
| Signal | Threshold | Why |
|
||||
|-----------------------------------------|------------------------|----------------------------------|
|
||||
| S9F* emission rate | > 1 / minute sustained | malformed peer or schema drift |
|
||||
| Distinct source IPs on HSMS port | > expected MES count | spoofed connection attempts |
|
||||
| TLS handshake failures (stunnel log) | > 5 / minute | bad client cert or rogue scanner |
|
||||
| Failed signature verification (start) | any | tampered YAML |
|
||||
| HSMS connection-flap rate | > 1 / minute | MES instability or net event |
|
||||
| Spool depth | > 1000 sustained | MES backpressure or outage |
|
||||
| T-timer expiry counter | rising | network-layer trouble |
|
||||
|
||||
## 6. Secrets handling
|
||||
|
||||
### 6.1 Stunnel keys
|
||||
|
||||
- Store at `/etc/stunnel/certs/`, mode `0600`, owner `stunnel`.
|
||||
- Rotate annually. Ed25519 keys never expire cryptographically but
|
||||
fab policy usually mandates rotation regardless.
|
||||
- Don't commit private keys to git. Don't share them across tools.
|
||||
|
||||
### 6.2 Minisign signing key
|
||||
|
||||
- Live on a hardened build host, not on the tools themselves.
|
||||
- The public key (`acme.pub`) is what ships to every tool.
|
||||
- Sign in CI from a passphrase-protected key stored as a CI secret;
|
||||
never echo the passphrase, never log it.
|
||||
|
||||
## 7. Incident response
|
||||
|
||||
When something goes wrong:
|
||||
|
||||
1. **Capture the wire trace immediately** — `tcpdump -w` on the
|
||||
equipment's HSMS interface. Retain for 24h minimum even if no
|
||||
incident is suspected.
|
||||
2. **Don't restart the equipment** until the wire trace and the
|
||||
journal directory (`/var/lib/acme-secsgem/`) are snapshotted.
|
||||
Restarting wipes in-memory state the incident analysis may need.
|
||||
3. **Pull recent audit logs from the SIEM** for the affected session
|
||||
ID and host.
|
||||
4. **Cross-check against the runbook** in README §10 — common
|
||||
incidents have documented mitigation paths.
|
||||
|
||||
Filing an incident with us (`raphael@maenle.net`):
|
||||
- Wire trace (pcap, scrubbed of any production-sensitive payloads)
|
||||
- Equipment logs covering the incident window
|
||||
- Journal directory `tar.gz`
|
||||
- Equipment build SHA + YAML SHAs
|
||||
- MES vendor + build
|
||||
- What you tried that didn't work
|
||||
@@ -0,0 +1,305 @@
|
||||
# External verification plan
|
||||
|
||||
The proofs in [PROOFS.md](PROOFS.md) are mostly **us testing us**:
|
||||
|
||||
| Proof | Independence |
|
||||
|--------------------------------|--------------------------------------------------------------|
|
||||
| 445 unit/integration tests | Internal — our code testing our code |
|
||||
| 47 conformance harness checks | Internal — our host driving our server |
|
||||
| 31 secsgem-py interop checks | **External**, but covers ~15–20 % of the claimed wire surface |
|
||||
| 100 k random tool ops | Internal — property test of our model |
|
||||
| YAML validation | Internal — our validator on our YAML |
|
||||
|
||||
Only the secsgem-py row is external, and it's thin: it skips most of
|
||||
GEM 300 (E40 multi-create, E94 CJ-create, E87 slot map / transfer /
|
||||
cancel, E116, E120, E148, E157), HSMS-GS, S5F9–F18 exception
|
||||
recovery, S12 wafer maps, S2F49 enhanced commands, and every
|
||||
wire-level edge case that isn't message-shaped (frame framing, T-timer
|
||||
expiry behaviours, auto-S9F path). That's an enormous footprint to
|
||||
leave on "we both interpret the spec the same way" trust.
|
||||
|
||||
This document plans the work to plug that gap with **four independent
|
||||
external validators**. None of them is a GEM RTS (that costs money
|
||||
and needs hardware); none replaces a real-MES integration sweep
|
||||
([MES_INTEROP.md](MES_INTEROP.md)). But together they convert the
|
||||
proof-of-completeness from "trust the unit-test count" to "four
|
||||
independent codecs, two independent implementations, the standards
|
||||
body's own bytes, and one fuzzer all agree."
|
||||
|
||||
---
|
||||
|
||||
## 1. SEMI E5 known-answer tests (KAT)
|
||||
|
||||
**Goal.** Assert our encoder produces the exact bytes the SEMI E5
|
||||
encoding rules require, and our decoder reverses any spec-conformant
|
||||
byte stream to the original Item. Hex-string fixtures, no peer
|
||||
implementation involved.
|
||||
|
||||
**Why it's the strongest single test.** Every other validator is one
|
||||
implementer's interpretation of the spec. KAT is the *spec's own
|
||||
arithmetic*. If our codec matches the format-byte construction rules
|
||||
(§9.2-§9.5), it is wire-compatible with anything else that obeys
|
||||
those rules.
|
||||
|
||||
**Method.** A new `tests/test_e5_kat.cpp` with hex-string fixtures
|
||||
covering every format code:
|
||||
|
||||
| Format | Code | KAT fixture content |
|
||||
|--------|--------|---------------------------------------------------------------|
|
||||
| List | `0x00` | empty list `<L[0]>`, nested list, list with mixed-type items |
|
||||
| Binary | `0x20` | empty, 1-byte, 256-byte (length-byte count = 2), 65 536-byte (length-byte count = 3) |
|
||||
| Boolean| `0x24` | TRUE, FALSE, multi-element vector |
|
||||
| ASCII | `0x40` | empty, single char, "Hello", 255-byte string, 256-byte string |
|
||||
| JIS-8 | `0x44` | empty, non-ASCII bytes |
|
||||
| C2 | `0x48` | empty, ASCII subset, BMP code points |
|
||||
| U1 | `0xA4` | 0, 1, 0x7F, 0xFF, multi-element |
|
||||
| U2 | `0xA8` | 0, 0x0102 big-endian, 0xFFFF, multi-element |
|
||||
| U4 | `0xAC` | 0, 0x01020304, 0xFFFFFFFF, multi-element |
|
||||
| U8 | `0xA0` | 0, 0x0102030405060708, multi-element |
|
||||
| I1 | `0x64` | 0, 1, -1 (0xFF), -128 (0x80), 127 (0x7F) |
|
||||
| I2 | `0x68` | 0, 1, -1, INT16_MIN, INT16_MAX |
|
||||
| I4 | `0x6C` | 0, 1, -1, INT32_MIN, INT32_MAX |
|
||||
| I8 | `0x60` | 0, 1, -1, INT64_MIN, INT64_MAX |
|
||||
| F4 | `0x84` | 0.0, 1.0, -1.0, NaN, +Inf, -Inf, subnormal |
|
||||
| F8 | `0x80` | 0.0, 1.0, -1.0, NaN, +Inf, -Inf |
|
||||
|
||||
Plus the format-byte length-count cases:
|
||||
|
||||
- `length_bytes = 1` (body ≤ 255 bytes)
|
||||
- `length_bytes = 2` (body 256 – 65 535 bytes)
|
||||
- `length_bytes = 3` (body 65 536 – 16 777 215 bytes)
|
||||
|
||||
Each row in the test is a `(canonical_hex, expected_item)` pair.
|
||||
`encode(expected_item)` must produce `canonical_hex`; `decode(canonical_hex)`
|
||||
must produce a value equal to `expected_item`.
|
||||
|
||||
**Success criterion.** Every fixture round-trips byte-identical.
|
||||
Failure on any single one is a spec-deviation bug — fix the codec,
|
||||
not the fixture.
|
||||
|
||||
**Effort.** ~3 hours. Most of it is constructing the byte sequences
|
||||
correctly the first time (a one-byte error in a fixture invalidates
|
||||
the proof).
|
||||
|
||||
**Scope limits.** KAT proves byte-level encoding only. It does not
|
||||
prove higher-level message structure (S1F3 body has these fields in
|
||||
this order) — that's covered by `test_messages.cpp`.
|
||||
|
||||
**Honest disclosure about authority.** SEMI does NOT publish official
|
||||
test vectors for E5 (unlike NIST, which ships `.rsp` files for every
|
||||
crypto standard). The hex bytes in `test_e5_kat.cpp` are constructed
|
||||
by us from the encoding rules described in the spec. They prove our
|
||||
encoder is internally consistent with *our reading* of the rules — if
|
||||
we somehow got a format code wrong, the KAT would happily match our
|
||||
buggy codec. The mitigation is the secsgem-py interop and the
|
||||
secs4j cross-validation in §3: those use independent decoders, so
|
||||
disagreement on a format code surfaces there. KAT + interop combined
|
||||
is a strong proof; KAT alone is a regression test.
|
||||
|
||||
### 1a. KAT corroboration via secsgem-py
|
||||
|
||||
To close the "we might have gotten the format codes wrong" loophole,
|
||||
a follow-up step is to round-trip every KAT fixture through
|
||||
secsgem-py's decoder and assert it returns the same value. Concrete
|
||||
plan:
|
||||
|
||||
1. Export the KAT fixtures to a JSON file
|
||||
(`tests/data/e5_kat.json`) listing each `(name, canonical_hex,
|
||||
sml_repr)`.
|
||||
2. Add `interop/kat_corroborate.py` that reads the JSON, feeds each
|
||||
canonical hex to `secsgem.secs.functions.SecsStreamFunction`'s
|
||||
decoder, and asserts the parsed structure matches the `sml_repr`.
|
||||
3. Wire into CI as a separate job after the C++ tests pass.
|
||||
|
||||
Effort: ~2 hours. Lifts the KATs from "our format codes are
|
||||
internally consistent" to "our format codes are confirmed by an
|
||||
independent Python implementation that read the spec without
|
||||
talking to us."
|
||||
|
||||
---
|
||||
|
||||
## 2. tshark / Wireshark HSMS dissector
|
||||
|
||||
**Goal.** Validate our HSMS framing against an independent third
|
||||
codec — Wireshark's built-in HSMS dissector (in tree since ~2017).
|
||||
|
||||
**Why.** Wireshark's dissector is written by network-protocol
|
||||
authors who don't read our code, didn't talk to us, and don't share
|
||||
implementation details with secsgem-py. If they parse our pcap
|
||||
without warnings, our HSMS framing is wire-correct independently of
|
||||
both our internal tests and the secsgem-py path.
|
||||
|
||||
**Method.** A new script `interop/tshark_dissector_check.sh` that:
|
||||
|
||||
1. Starts the C++ server.
|
||||
2. Captures a pcap of the demo flow via `tcpdump -i any -w trace.pcap 'tcp port 5000'`.
|
||||
3. Runs the two-container demo client to generate ~24 transactions.
|
||||
4. Stops the server.
|
||||
5. Parses `trace.pcap` with `tshark -V -r trace.pcap -d tcp.port==5000,hsms`.
|
||||
6. Greps the parsed output for `Malformed Packet`, `Dissector bug`,
|
||||
or `Unknown PType/SType` and asserts none appear.
|
||||
7. Greps for known good frames (`Select.req`, `Linktest.req`,
|
||||
`S1F13`, `S6F11`) and asserts they appear at least once each.
|
||||
|
||||
Wired into `.gitea/workflows/ci.yml` as an additional CI job
|
||||
(installs `tshark` from apt, runs the script, fails on grep
|
||||
mismatches).
|
||||
|
||||
**Success criterion.** tshark dissects every captured HSMS frame
|
||||
without errors or warnings.
|
||||
|
||||
**Effort.** ~3 hours including CI wiring.
|
||||
|
||||
**Scope limits.** Validates HSMS *framing* (4-byte length prefix +
|
||||
10-byte header) and *control message* shapes (Select / Deselect /
|
||||
Linktest / Separate / Reject). Does NOT validate SECS-II body
|
||||
structure beyond the dissector's depth (which is shallow — Wireshark
|
||||
displays bodies as hex blobs, doesn't decode S/F semantics). That's
|
||||
where KAT and secs4j pick up.
|
||||
|
||||
---
|
||||
|
||||
## 3. secs4j cross-validation
|
||||
|
||||
**Goal.** Add a second independent SECS implementation as a peer:
|
||||
[`secs4j`](https://github.com/kenta-shimizu/secs4j), Apache-2.0 Java.
|
||||
|
||||
**Why.** secsgem-py and secs4j were written by different authors,
|
||||
from different language ecosystems, against the same SEMI standards.
|
||||
Disagreements between them mark spec ambiguities; agreement marks
|
||||
genuine wire-correctness. Our secsgem-py interop is *one* peer; this
|
||||
adds a second. Most likely to surface GEM 300 issues — secs4j
|
||||
historically covers E40/E94/E87/E116 more thoroughly than secsgem-py.
|
||||
|
||||
**Method.**
|
||||
|
||||
1. Add a Docker sidecar `interop/secs4j/` with `eclipse-temurin:21-jdk`,
|
||||
maven, and a copy of secs4j cloned + built.
|
||||
2. Write a `Secs4jHostHarness.java` that:
|
||||
- Connects as active HSMS host to our C++ server.
|
||||
- Runs the same ~24 checks as `host_vs_cpp_server.py` (S1, S2, S5,
|
||||
S6, S7, S10) so we have a like-for-like comparison.
|
||||
- Plus the GEM 300 streams secs4j covers natively (S3 carrier
|
||||
actions, S14 CJ create, S16 PJ create/command including the full
|
||||
variable-list bodies that defeated secsgem-py's SFDL).
|
||||
- Asserts each transaction's response code is in the spec-defined
|
||||
range. Exits 0 on success.
|
||||
3. Cron the harness into `interop/run-secs4j.sh` and add a CI job
|
||||
that runs it.
|
||||
|
||||
**Survey step (do this first).** Before committing, build secs4j and
|
||||
catalog which streams/functions it actually supports. If it covers
|
||||
strictly less than secsgem-py, the value drops. Estimated 30 min to
|
||||
clone + build + list functions.
|
||||
|
||||
**Success criterion.** Every check the harness defines exits PASS
|
||||
against the C++ server, AND secs4j's output for at least 3 streams
|
||||
secsgem-py couldn't drive (S14, S16 full bodies, S3 slot map) lands
|
||||
clean.
|
||||
|
||||
**Effort.** ~6 hours, with risk:
|
||||
- Build / dependency problems (Java, maven, secs4j build)
|
||||
- Coverage gaps (secs4j may not cover what we hoped)
|
||||
- API differences requiring a different harness structure
|
||||
|
||||
If at the survey step secs4j proves unhelpful, we'll write up what we
|
||||
learned and skip the rest.
|
||||
|
||||
**Scope limits.** Same as secsgem-py — peer-implementation comparison
|
||||
catches "we both got it wrong the same way" but not "we both got it
|
||||
wrong differently." Three peers (KAT, secsgem-py, secs4j) covering
|
||||
overlapping subsets together approach the asymptote.
|
||||
|
||||
---
|
||||
|
||||
## 4. libFuzzer over codec + SML parser
|
||||
|
||||
**Goal.** Catch crashes, out-of-bounds reads, integer overflows, and
|
||||
infinite loops on arbitrary input to `secs2::decode` and
|
||||
`secs2::from_sml`.
|
||||
|
||||
**Why.** The 26-line `test_fuzz.cpp` exists but is one-shot — it
|
||||
runs a fixed handful of malformed inputs. Real fuzzing runs millions
|
||||
of inputs guided by coverage feedback. Catches the class of bug
|
||||
where a malicious or buggy peer sends a frame designed to crash us.
|
||||
|
||||
**Method.**
|
||||
|
||||
1. Add a `SECSGEM_FUZZ=ON` CMake option that:
|
||||
- Sets compiler to clang (libFuzzer is a clang feature).
|
||||
- Adds `-fsanitize=fuzzer,address,undefined` to the fuzzer
|
||||
targets.
|
||||
- Adds two new executables, `fuzz_secs2_decode` and
|
||||
`fuzz_sml_parse`, each with a `LLVMFuzzerTestOneInput` entry
|
||||
point that calls the respective decoder on the input bytes.
|
||||
2. Wire a CI job that builds with `SECSGEM_FUZZ=ON` and runs each
|
||||
fuzzer for **5 minutes** (long enough to cover the easy bugs;
|
||||
short enough to fit a PR cycle). Stores any crashing inputs as
|
||||
CI artifacts.
|
||||
3. Seed corpus with the SEMI E5 KAT fixtures from (1) plus the
|
||||
wire payloads our `interop/` runs produce. Coverage-guided
|
||||
fuzzing starts from a known-good baseline and explores edges.
|
||||
|
||||
**Success criterion.** 5-minute run finds no crashes; coverage map
|
||||
shows growth over time (proving the fuzzer is actually exploring,
|
||||
not stuck).
|
||||
|
||||
**Effort.** ~4 hours including the corpus seed + CI wiring.
|
||||
|
||||
**Scope limits.** Catches *crashes and UB*, not *semantic
|
||||
mismatches*. A decoder that returns the wrong value silently is
|
||||
invisible to libFuzzer; KAT and interop catch that. Combined, they
|
||||
cover both axes.
|
||||
|
||||
---
|
||||
|
||||
## Order of execution
|
||||
|
||||
Plan: **(1) KAT → (2) tshark → (3) secs4j → (4) libFuzzer.**
|
||||
|
||||
Rationale:
|
||||
- KAT first because it's the highest-leverage individual test (the
|
||||
standard's own arithmetic), is cheap, and produces fixtures that
|
||||
later seed libFuzzer's corpus.
|
||||
- tshark second because it's cheap and gives us a third independent
|
||||
framing codec.
|
||||
- secs4j third because it has the largest variance — could be huge
|
||||
win, could be a dud. Worth de-risking with the survey step.
|
||||
- libFuzzer last because it benefits from the KAT corpus and its
|
||||
CI wiring is mostly orthogonal to everything else.
|
||||
|
||||
After all four:
|
||||
|
||||
| Proof channel | Independence |
|
||||
|--------------------------------|------------------------------------------------------|
|
||||
| 445 unit/integration tests | Internal |
|
||||
| 47 conformance harness checks | Internal |
|
||||
| **SEMI E5 KAT** | **External — standards body's own encoding rules** |
|
||||
| **tshark dissector** | **External — independent network-protocol authors** |
|
||||
| **secs4j interop** | **External — second independent SECS implementation**|
|
||||
| secsgem-py interop | External — Python reference impl |
|
||||
| **libFuzzer 5-min run** | **External — coverage-guided structural search** |
|
||||
| 100 k random tool ops | Internal — property test |
|
||||
| YAML validation | Internal |
|
||||
|
||||
That's **four external proofs**, three of them validating overlapping
|
||||
slices of the same surface from independent angles. An adversarial
|
||||
review can no longer say "you wrote the tests, of course they pass."
|
||||
|
||||
---
|
||||
|
||||
## What this plan does NOT replace
|
||||
|
||||
- **A GEM RTS run.** Still required for certification; still costs
|
||||
money + needs hardware. Documented in
|
||||
[MES_INTEROP.md](MES_INTEROP.md) §10.
|
||||
- **Per-MES interop sweeps** against the customer's actual MES
|
||||
(Camstar, FactoryWorks, etc.). Still required for any production
|
||||
deployment. See [MES_INTEROP.md](MES_INTEROP.md).
|
||||
- **Real-fab wire traces.** No public corpus exists; fabs treat
|
||||
their captures as IP.
|
||||
|
||||
Those three remain customer-side work. But the validators in this
|
||||
plan move "we claim feature completeness" from one external proof
|
||||
(thin secsgem-py interop) to five (KAT + tshark + secsgem-py +
|
||||
secs4j + libFuzzer), and that's worth doing.
|
||||
Reference in New Issue
Block a user