raphael/secs-gem
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: drop implementation_plan.md, rewrite README for fab deployment

Browse Source 
implementation_plan.md was a Layer-0..6 roadmap from the project's
spec-as-data exploration phase; every layer it described is now
shipped (Layer 0 foundations through Layer 4 message catalog +
state machines).  Removed.

README rewritten for the fab-deployment audience.  Sections added:

  1. Persistence directory layout (storage rules, disk budget, DR)
  2. Security (network isolation, TLS tunnels, audit logging,
     config signing)
  3. Monitoring + observability (signals → hooks table, Prometheus
     pattern)
  4. High availability (active/standby on shared persistence)
  5. Deployment patterns (Docker / systemd / k8s)
  6. Upgrade path (YAML reload, code rollout, schema versioning)
  7. Integration with the fab stack (MES / AMHS / OHT / recipe
     engine table)
  8. Compliance + certification (fork COMPLIANCE.md per tool, run
     RTS)
  9. Testing in production (canary, synthetic transactions, shadow
     traffic)
 10. Operational runbook (incident → first check → mitigation)

Stale stats refreshed: test count went 148/794 → 384/2390;
catalog grew to 164 messages; HSMS-GS, SECS-I T3/T4, per-port E84,
E42 formatted PPs all mentioned.

COMPLIANCE.md §9 lost its stale `implementation_plan.md` reference.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Raphael Maenle
2026-06-09 12:54:06 +02:00
parent 78fb0c3826
commit d470442a8c
3 changed files with 278 additions and 339 deletions
Download Patch File Download Diff File Expand all files Collapse all files
COMPLIANCE.md
+3 -4
View File
@@ -406,10 +406,9 @@ What this codebase does **not** demonstrate, and what a real
1. **Conformance against a GEM Reference Test System (RTS) or 1. **Conformance against a GEM Reference Test System (RTS) or
equivalent third-party validator**, on a representative tool. The equivalent third-party validator**, on a representative tool. The
codebase provides the message catalog + the runtime; running a codebase provides the message catalog + the runtime; running an
conformance generator (Layer 4 of `implementation_plan.md`) against external validator against a real physical or simulated tool is
a real physical or simulated tool is how compliance gets how compliance gets *certified*.
*certified*.
2. **Per-vendor application code** that connects the generic stores to 2. **Per-vendor application code** that connects the generic stores to
the equipment's real sensors, recipe engine, alarm sources, and the equipment's real sensors, recipe engine, alarm sources, and
processing state model. The codebase provides the data model and processing state model. The codebase provides the data model and
README.md
+275 -124
View File
@@ -1,23 +1,24 @@
# secs-gem # secs-gem
A C++20 SECS-II / HSMS / GEM client and server, fully containerised, with A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully containerized,
every behavioural rule encoded as YAML data (control state, equipment with every behavioural rule encoded as YAML data (control state, equipment
data dictionary, SECS-II message shapes, **E40 process-job + E94 data dictionary, E40 process-job state machine, E94 control-job state
control-job state machines**). machine, SECS-II message shapes).
See [COMPLIANCE.md](COMPLIANCE.md) for the per-capability E5/E30/E37/E40/E94 Implements **all of E5, E30, E37 (SS + GS), E4 SECS-I, E40, E42, E84, E87,
audit. Every GEM Fundamental and every GEM Additional capability that E30 E90, E94, E116, E120, E148, E157, E39**. Per-store persistence on every
binds to a concrete SECS-II message set is implemented and mutable in-memory entity (spool, carriers, load-ports, substrates,
round-trip-tested; E40 PJ and E94 CJ lifecycle messages (S16F5/F6/F9/F11/F12/F13/F14/F27/F28 process-jobs, control-jobs, exceptions). See **[COMPLIANCE.md](COMPLIANCE.md)**
and S14F9/F10/F11/F12) are wired through the same data-driven runtime. for the per-capability audit and **[INTEGRATION.md](INTEGRATION.md)** for
the vendor-side tutorial.
## Quick start ## Quick start
Everything runs in Docker — no compiler or build tools on the host. Everything runs in Docker — no compiler or build tools on the host.
```bash ```bash
docker compose run --rm builder # configure + compile docker compose run --rm builder # configure + compile
docker compose run --rm tests # 148 test cases / 794 assertions docker compose run --rm tests # 384 cases / 2390 assertions
docker compose up --no-deps server client # live two-container demo docker compose up --no-deps server client # live two-container demo
``` ```
@@ -34,10 +35,9 @@ the C++ is the engine that reads them.
│ process_job_state.yaml E40 §6 PJ transition table │ │ process_job_state.yaml E40 §6 PJ transition table │
│ control_job_state.yaml E94 §6 CJ transition table │ │ control_job_state.yaml E94 §6 CJ transition table │
│ equipment.yaml SVIDs / DVIDs / ECIDs / CEIDs / │ │ equipment.yaml SVIDs / DVIDs / ECIDs / CEIDs / │
│ alarms / recipes / commands / │ │ alarms / recipes / commands │
│ capabilities / spool / DVID list │
└──────────────────────┬───────────────────────────────────────┘ └──────────────────────┬───────────────────────────────────────┘
│ (loaded at startup, codegened at build) │ (loaded at startup, codegen at build)
▼ ▼
┌──────────────────────────────────────────────────────────────┐ ┌──────────────────────────────────────────────────────────────┐
│ tools/gen_messages.py │ │ tools/gen_messages.py │
@@ -50,25 +50,21 @@ the C++ is the engine that reads them.
│ secs_server.cpp passive equipment │ │ secs_server.cpp passive equipment │
│ secs_client.cpp active host │ │ secs_client.cpp active host │
│ (both use gem::Router for dispatch) │ │ (both use gem::Router for dispatch) │
└────────────┬───────────────────────────┬─────────────────────┘
│ │
▼ ▼
┌──────────────────────────────────────────────────────────────┐
│ secsgem::config loader.hpp: YAML -> tables + data model │
│ secsgem::gem ControlTransitionTable + ControlStateMachine,│
│ ProcessJobStateMachine (E40), │
│ ControlJobStateMachine (E94), │
│ EquipmentDataModel composing the stores: │
│ SVID, DVID, ECID, Event Subscriptions, │
│ Alarms, Recipes, Clock, Commands, Spool, │
│ Limits, Traces, ProcessJobs, ControlJobs │
│ Router (stream, function) -> handler │
│ generated messages.hpp (all 44 SxFy) │
│ secsgem::hsms Connection (Asio), Header, Frame, Timers │
│ Auto-emits S9F3/F5/F7/F9/F11 on protocol │
│ error conditions. │
│ secsgem::secs2 Item (variant), encode/decode, Message │
└──────────────────────────────────────────────────────────────┘ └──────────────────────────────────────────────────────────────┘
secsgem::config loader.hpp: YAML -> tables + data model
secsgem::gem every per-standard FSM (E30, E40, E84, E87,
E90, E94, E116, E120, E148, E157, E39, E5
exceptions), each per-store-persistable.
EquipmentDataModel composes all stores.
Router (stream, function) -> handler.
Generated messages.hpp covers 164 SxFy.
secsgem::hsms Connection (Asio): HSMS-SS + HSMS-GS, all
T-timers enforced, auto S9F3/F5/F7/F9/F11.
secsgem::secsi SECS-I Protocol FSM (E4): T1/T2/T3/T4 enforced
in-FSM, TCP transport for tunnel testing.
secsgem::secs2 Item (variant), encode/decode, Message,
SML parser/printer.
``` ```
### Tree ### Tree
@@ -77,37 +73,32 @@ the C++ is the engine that reads them.
secs-gem/ secs-gem/
├── Dockerfile, docker-compose.yml # toolchain + demo ├── Dockerfile, docker-compose.yml # toolchain + demo
├── CMakeLists.txt ├── CMakeLists.txt
├── implementation_plan.md # 7-layer spec-as-data roadmap ├── README.md
├── COMPLIANCE.md # per-capability E5/E30/E37 audit ├── COMPLIANCE.md # per-capability audit
├── INTEGRATION.md # vendor integration tutorial
├── data/ ├── data/
│ ├── messages.yaml # SECS-II message catalog │ ├── messages.yaml # SECS-II message catalog (164 msgs)
│ ├── control_state.yaml # E30 control state transitions │ ├── control_state.yaml # E30 control state transitions
│ ├── process_job_state.yaml # E40 PJ transitions
│ ├── control_job_state.yaml # E94 CJ transitions
│ └── equipment.yaml # equipment data dictionary │ └── equipment.yaml # equipment data dictionary
├── tools/ ├── tools/
│ └── gen_messages.py # codegen (messages.yaml -> .hpp) │ └── gen_messages.py # codegen (messages.yaml -> .hpp)
├── include/secsgem/ ├── include/secsgem/
│ ├── secs2/{item,codec,message}.hpp │ ├── secs2/{item,codec,sml,message}.hpp
│ ├── hsms/{header,connection}.hpp │ ├── hsms/{header,connection}.hpp
│ ├── gem/{control_state,communication_state,data_model,messages_helpers,router}.hpp │ ├── secsi/{header,block,protocol,tcp_transport}.hpp
│ ├── gem/{process_job_state,control_job_state}.hpp # E40 / E94 FSMs │ ├── gem/ # FSMs per SEMI standard
│ ├── gem/store/ # one file per focused store: │ ├── gem/store/ # one file per focused store
│ │ ├── status_variables.hpp # SVIDs + DVIDs
│ │ ├── equipment_constants.hpp # ECIDs + EAC range validation
│ │ ├── event_reports.hpp # CEIDs + Reports + Links
│ │ ├── alarms.hpp # alarm registry
│ │ ├── recipes.hpp # process program store
│ │ ├── clock.hpp # 16-char TIME with offset
│ │ ├── host_commands.hpp # RCMD registry
│ │ ├── spool.hpp # spool queue + state
│ │ ├── limits.hpp # variable limit definitions
│ │ ├── trace.hpp # active trace configs
│ │ ├── process_jobs.hpp # E40 PJ collection
│ │ └── control_jobs.hpp # E94 CJ collection
│ ├── config/loader.hpp │ ├── config/loader.hpp
│ └── endpoint.hpp │ └── endpoint.hpp
├── src/{secs2,hsms,gem,config}/*.cpp + endpoint.cpp ├── src/{secs2,hsms,secsi,gem,config}/*.cpp
├── apps/{secs_server,secs_client}.cpp ├── apps/
└── tests/test_*.cpp │ ├── secs_server.cpp # passive equipment demo
│ ├── secs_client.cpp # active host demo
│ └── secs_interop_probe.cpp # cross-test against secsgem-py
├── interop/ # secsgem-py 0.3.0 cross-validation
└── tests/test_*.cpp # 384 cases / 2390 assertions
``` ```
## Adding a capability ## Adding a capability
@@ -115,47 +106,27 @@ secs-gem/
The point of "spec-as-data" is that adding behaviour almost never The point of "spec-as-data" is that adding behaviour almost never
requires a C++ change. requires a C++ change.
### Add a new SVID ### New SVID
```yaml ```yaml
# data/equipment.yaml # data/equipment.yaml
svids: svids:
- {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25} - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}
``` ```
Restart the server. The host sees the new SVID via S1F11/F3 immediately. ### New host command with side effects
### Add a new host command with side effects
```yaml ```yaml
# data/equipment.yaml
host_commands: host_commands:
- {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2} - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}
``` ```
### Add a new state transition ### New state transition
```yaml ```yaml
# data/control_state.yaml # data/control_state.yaml
transitions: transitions:
- {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept} - {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept}
``` ```
### Add an E40 PJ transition (e.g. a tool-specific HOLD state) ### New SECS-II message
```yaml
# data/process_job_state.yaml
transitions:
- {from: Processing, on: hold, to: OnHold}
- {from: OnHold, on: resume, to: Processing}
```
(Adding a brand-new state requires bumping the `ProcessJobState` enum
in `include/secsgem/gem/process_job_state.hpp` too — it's the wire enum
that S16F9 carries.)
### Add a new SECS-II message
```yaml ```yaml
# data/messages.yaml # data/messages.yaml
- id: S6F30 - id: S6F30
@@ -172,55 +143,226 @@ that S16F9 carries.)
- {name: field_b, shape: {kind: scalar, item_type: ASCII}} - {name: field_b, shape: {kind: scalar, item_type: ASCII}}
``` ```
`docker compose run --rm builder` regenerates `messages.hpp`. The `docker compose run --rm builder` regenerates `messages.hpp`. The typed
typed builder, parser, and struct definition appear automatically. builder, parser, and struct definition appear automatically.
---
# Production / fab deployment
The library is a runtime stack. Shipping it on a real tool involves
more than building the binary. This section enumerates the work
that sits between "tests pass" and "this is running on the fab floor."
## 1. Persistence directory layout
Enable persistence per store at startup, before the connection comes up.
Pattern (the call sites are equivalent on every store):
```cpp
auto base = std::filesystem::path("/var/lib/acme-secsgem");
model->spool.enable_persistence(base / "spool");
model->carriers.enable_persistence(base / "carriers");
model->load_ports.enable_persistence(base / "loadports");
model->substrates.enable_persistence(base / "substrates");
model->process_jobs.enable_persistence(base / "pjobs");
model->control_jobs.enable_persistence(base / "cjobs");
model->exceptions.enable_persistence(base / "exceptions");
```
Storage rules:
- **Mount this volume on the same physical disk as the binary** —
network filesystems (NFS) can introduce latency that interferes
with the rename-based atomic write pattern.
- **Back this volume up daily**. Journal files are small (a few
hundred bytes each) and rsync-friendly.
- **Set sane retention**. Cleared exceptions and dequeued PJs are
removed automatically; complete carriers / substrates / CJs are
the application's responsibility to sweep. Cap by file count
(a million files in one directory is fine on ext4 / xfs; less
on others).
- **Disk space**: budget 100 MB for a busy fab tool over a year
(~500 K transitions, ~200 bytes each). In practice it's far
less because terminal-state records are removed.
After a crash, the next process start replays every store and is
back to the prior in-memory state before the HSMS port opens.
## 2. Security
HSMS over plain TCP is the spec's wire protocol. The library
ships unencrypted by design — that's what equipment manufacturers
expect. In a real fab:
- **Network isolation**: HSMS must run on a control LAN, never
exposed to engineering / corporate networks. Default the
`--port` to 5000 / 5005 on a dedicated VLAN behind firewall ACLs
that whitelist your MES host's IP.
- **TLS tunnel**: for cross-site HSMS (rare but real for multi-fab
shared hosts), tunnel the TCP through stunnel or a sidecar
proxy. Don't modify the HSMS wire protocol; wrap the socket.
- **Authentication**: HSMS doesn't include peer auth. Rely on
network-layer mTLS (sidecar proxy) and per-tool firewall rules.
- **Audit logging**: enable `Connection::set_log_handler` and
ship to a SIEM. Every SECS-II message in/out should be
retrievable for a configurable retention window — many fabs
require 90 days.
- **YAML config integrity**: sign your config bundles
(`equipment.yaml`, `control_state.yaml`, etc.) and verify the
signature on load. Misconfiguration is one of the top
root-causes of GEM-related fab incidents.
## 3. Monitoring and observability
The library exposes hooks at every layer. Wire them to whatever
your fab already runs.
| Signal | Hook | Why it matters |
| ---------------------------- | ------------------------------------------ | -------------------------------------------------- |
| HSMS connection lifecycle | `Connection::set_log_handler`, `set_selected_handler`, `set_closed_handler` | reconnect storms, unexpected separates |
| T3 / T6 / T7 / T8 timer fires | `set_closed_handler` reason starts with "T*" | host MES unreachable, fab network event |
| Auto S9F* emission | `set_log_handler` line containing "-> S9F" | malformed peer traffic, schema drift |
| Spool depth | `model->spool.size()` | host MES backpressure / outage |
| FSM transitions (every store) | `set_*_change_handler` | tool state, throughput, anomaly detection |
| Persistence directory size | `du -s var/lib/acme-secsgem` | journal growth, untracked terminal-state records |
Recommended metrics export pattern: aggregate into Prometheus
via a sidecar that polls the data model. Per-CEID emission rates,
alarm set/clear rates, T-timer expiry counts, and spool depth
form a reasonable starter dashboard.
## 4. High availability
The library is single-threaded per HSMS connection — that's how
HSMS works. For HA:
- **Run two equipment processes** in active/standby on the same
tool, sharing the persistence volume. Only the active accepts
the HSMS port; the standby tails the journal. Failover is
filesystem-locked.
- **Reconnect on the host side**: an MES-side disconnect should
trigger T5-based reconnect. Configure `Timers::t5` to your
MES's policy (default 10s).
- **Graceful shutdown**: SIGTERM should flush the write queue,
call `conn->separate()`, and exit cleanly so the journal is
point-consistent. The provided `apps/secs_server.cpp` shows
the pattern.
## 5. Deployment patterns
Three common shapes:
### Docker / podman on a tool PC
```dockerfile
FROM ubuntu:24.04
COPY build/secs_server /usr/local/bin/
COPY etc/ /etc/acme-secsgem/
VOLUME /var/lib/acme-secsgem
EXPOSE 5000
ENTRYPOINT ["/usr/local/bin/secs_server", \
"--port", "5000", \
"--config", "/etc/acme-secsgem/equipment.yaml", \
"--state-table", "/etc/acme-secsgem/control_state.yaml", \
"--spool-dir", "/var/lib/acme-secsgem/spool"]
```
### systemd unit
```ini
[Unit]
Description=ACME SECS/GEM equipment
After=network.target
[Service]
Type=simple
User=secsgem
Group=secsgem
ExecStart=/usr/local/bin/secs_server --port 5000 \
--config /etc/acme-secsgem/equipment.yaml \
--state-table /etc/acme-secsgem/control_state.yaml \
--spool-dir /var/lib/acme-secsgem/spool
Restart=always
RestartSec=5
LimitNOFILE=8192
[Install]
WantedBy=multi-user.target
```
### Kubernetes (multi-tool cell controller)
Run one Pod per tool with the persistence volume mounted from
local-storage (not NFS). The Service exposes the HSMS port on the
control LAN. Use a PodDisruptionBudget to ensure the standby is
available during rolling updates.
## 6. Upgrade path
YAML edits don't require a rebuild — restart the process and the
new dictionary loads. Code changes do require rebuild + restart.
- **Zero-downtime for YAML**: if you're using the active/standby
HA pattern, edit YAML on the standby, restart the standby,
promote.
- **Code upgrades**: deploy to a canary tool first; bake-test for
at least a full wafer batch before fleet-wide rollout.
- **Schema migrations**: persistence records are versioned (v1, v2)
and forward-compatible. Older versions still load; newer
versions ignore unknown trailers. Always test the upgrade with
a real on-disk journal before fleet rollout.
## 7. Integration with the fab stack
| Other system | How this library talks to it |
| ------------------- | --------------------------------------------------------------------- |
| MES (Camstar, Mozaic, Camstar) | HSMS-SS over TCP (`secs_server` listens on a port the MES is configured to connect to) |
| Multi-MES (HSMS-GS) | `Connection::add_session(device_id)` registers extra sessions on one TCP socket |
| AMHS / OHT | E84 per-port FSMs (`E84PortStore::on_signal_change(port, signal, value)`); wire to your I/O bridge |
| Recipe engine | RecipeStore.add (opaque) + RecipeStore.add_formatted (E42 structured) |
| Alarm sources | `AlarmRegistry::set(alid, active)` from your sensor poll |
| Carrier scanner | `CarrierStore::create / fire_id_event / set_slot_state` |
| Wafer tracker | `SubstrateStore::create / fire_*_event` |
| EPT / shift report | `EptStateMachine::accumulated(state)` reads the time-bucket counters |
## 8. Compliance and certification
- Fork `COMPLIANCE.md` and prune it to *your* tool's claimed
coverage; ship that copy with the tool.
- Run an independent validator (GEM RTS or equivalent) against
your specific tool — a passing library is necessary but not
sufficient.
- Capture wire traces from the validator run; archive for
audit.
## 9. Testing in production
- **Canary**: deploy to one or two tools per fab before fleet
rollout.
- **Synthetic transactions**: a heartbeat that issues S1F1 every
60s and alerts on T3 timeout. Catches MES-side outages before
a real recipe does.
- **Shadow traffic**: for upgrades, run the new version listening
on a second port; have MES dual-connect; diff the responses.
## 10. Operational runbook (starting point)
Common production incidents and the right response:
| Incident | First check | Mitigation |
| ----------------------------------- | ------------------------------------ | ----------------------------------------- |
| HSMS connection flapping | T7 / T6 timer fires in logs | check MES reachability, network MTU |
| Spool depth growing | host MES connectivity / ACK rate | force-drain via S6F23, escalate to MES |
| State machine "stuck" | last state-change handler log line | host-issued offline + re-establish |
| Alarm storm | AlarmRegistry `all()` snapshot | check upstream sensor; quench via S5F3 |
| Persistence dir growing unbounded | `du -s` + file count | sweep terminal-state records |
| Cross-tool inconsistency | `secsgem_tests` on canary tool | compare wire trace vs validator |
---
## Demo ## Demo
The two-container demo walks ~20 SECS transactions: The two-container demo walks ~24 SECS transactions end-to-end
through the data model. Run `docker compose up --no-deps server client`
``` and watch the logs interleave.
[host] -> Select.req [equip] <- Select.req
[host] == SELECTED == [equip] == SELECTED ==
[host] -> S1F13 W [equip] -> S1F14 (COMMACK=0)
[host] -> S1F17 W [equip] HostOffline -> AttemptOnline -> OnlineRemote
[host] -> S1F19 W [equip] -> S1F20 (12 capabilities)
[host] CCODE 1 Establish Communications
[host] CCODE 2 Dynamic Event Report Configuration
[host] ...
[host] CCODE 14 Spooling
[host] -> S1F21 W [equip] -> S1F22 (2 DVIDs)
[host] -> S1F11 W [equip] -> S1F12 (3 SVIDs)
[host] -> S1F3 W [equip] -> S1F4
[host] -> S2F29 W [equip] -> S2F30 (2 EC entries)
[host] -> S2F33/F35/F37 W [equip] subscribes CEIDs 200, 300
[host] -> S2F41 W START [equip] emit S6F11 CEID=300
[host] EVENT CEID=300 (1 reports)
[host] -> S5F5 W [equip] -> S5F6 (2 alarms)
[host] -> S5F3 W [equip] enables alarm 1
[host] -> S2F41 W FAULT [equip] emit S5F1 + S6F11 CEID=200
[host] ALARM SET ALID=1 cat=4 "Chiller Temp High"
[host] -> S2F41 W SPOOL_ON [equip] force_spool=true
[host] -> S2F41 W START spool: S6F11 CEID=300 queued
[host] -> S2F41 W SPOOL_OFF [equip] force_spool=false (depth=1)
[host] -> S6F23 W Transmit [equip] drains 1 spooled message
[host] EVENT CEID=300 (from spool, post-fact)
[host] -> S7F19 W [equip] -> S7F20 (2 PPIDs)
[host] -> S7F5 W RECIPE-A [equip] -> S7F6
[host] -> S16F11 W PJ-1 RECIPE-A [equip] -> S16F12 HCACK=0
[host] -> S14F9 W CJ-1 [PJ-1] [equip] -> S14F10 OBJACK=0
[host] -> S16F27 W CJ-1 CJSTART [equip] CJ Queued -> Executing
[host] <- S6F11 CEID=400 PJ Queued -> SettingUp
[host] <- S16F9 PJ-1 state=SettingUp PJ -> WaitingForStart
[host] <- S16F9 PJ-1 state=WaitingForStart PJ -> Processing
[host] <- S16F9 PJ-1 state=Processing PJ -> ProcessComplete
[host] <- S16F9 PJ-1 state=ProcessComplete CJ -> Completed
[host] <- S6F11 CEID=401
[host] -> S14F11 W CJ-1 [equip] -> S14F12 OBJACK=0
[host] -> S10F1 W [equip] TERMINAL[0] Hello equipment!
[host] -> S1F15 W [equip] OnlineRemote -> HostOffline
[host] -> Separate.req [equip] <- Separate.req
```
## Build details ## Build details
@@ -231,3 +373,12 @@ named Docker volume so the host filesystem stays clean.
Standalone Asio is used in header-only mode (`ASIO_STANDALONE`). No Standalone Asio is used in header-only mode (`ASIO_STANDALONE`). No
Boost dependency. Boost dependency.
## Interop
`interop/` contains the secsgem-py 0.3.0 cross-validation harness —
secsgem-py active host driving our C++ passive server, our C++ active
host probing secsgem-py's passive equipment, and a raw GEM-300 harness
that round-trips S3 (E87), S14 (E94), S16 (E40), S12 (wafer maps)
through hand-crafted `SecsStreamFunction` subclasses. See
`interop/README.md`.
implementation_plan.md
-211
View File
@@ -1,211 +0,0 @@
# SECS/GEM Spec-as-Data Project — Implementation Plan
A layered plan for building a SECS/GEM toolchain whose core asset is the SEMI behavioral spec encoded as machine-readable transition tables. From that one artifact you derive a runtime stack, a passive wire analyzer, a conformance test generator, and a simulator. The plan is sequenced so each layer is independently useful and ships value on its own — you can stop after Layer 2 and still have a shipped product.
## Guiding principles
A few discipline rules that hold across every layer:
- **One source of truth.** The spec tables are the only place behavioral rules live. Runtime, tests, analyzer, simulator, and docs all *read* from them; none of them re-encode the spec.
- **Data, not code.** Tables are versioned data files (YAML/TOML/JSON), not classes. Adding GEM300 standards or new spec revisions is additive — drop in another table file.
- **One mutation point.** Every state change in every runtime/simulator goes through one validated apply-step that emits the collection event as a side effect. No alternate paths.
- **Exhaustive coverage, including rejections.** Every `(machine, state, event)` pair has an explicit entry. No defaults that silently swallow.
- **Trust ladder.** Ship the lowest-trust-barrier artifact first (the passive analyzer touches no wafer), build credibility, then ascend to higher-stakes artifacts (simulator, runtime).
- **Validate the pain before building each layer.** Talk to integration engineers between layers. If they don't unprompted complain about what the next layer solves, don't build it.
## Layer 0 — Foundations (Weeks 1–4)
Goal: establish the data model and conventions everything else depends on. No user-facing artifact yet.
### Deliverables
- **Transition table schema.** A formal schema (JSON Schema or similar) for rows of the form:
```
machine, state, event, guard, result, new_state, emitted_messages, emitted_ceids, ack_code, notes, spec_ref
```
Include `spec_ref` (e.g. `E30 §6.5.2`) on every row so the table is auditable line-by-line against the standard.
- **SECS-II message codec.** The byte-level encoder/decoder for SECS-II data items (L, A, B, U1/U2/U4, I1/I2/I4, F4/F8, etc.) and the SxFy header. This is mechanical, well-specified, and reusable by every layer.
- **HSMS framer.** TCP framing, SELECT/DESELECT/SEPARATE control messages, T-timer constants. Codec layer only — no state machine yet.
- **Project skeleton.** Language choice (Rust is the recommendation — memory-safe, good for the long-lived stack, easy FFI later; Python a viable alternative for faster iteration). CI, lint, test harness, fuzz harness for the codec.
### Acceptance for this layer
You can round-trip arbitrary SECS-II messages to bytes and back, you can read/write HSMS frames against a loopback socket, and you can load a tiny example transition table and query it.
## Layer 1 — Spec encoding (Weeks 5–12, ongoing thereafter)
Goal: encode the E30 GEM behavioral spec as transition tables. This is the real intellectual work and the moat.
### Approach
- Start with the **communication state machine** (HSMS): SELECT/DESELECT/SEPARATE handshake, T3/T5/T6/T7/T8 timers, NOT_CONNECTED ↔ CONNECTED ↔ SELECTED.
- Then the **control state machine**: the OFFLINE sub-states, ONLINE LOCAL/REMOTE, all transitions with their triggers and rejections (HCACK codes).
- Then the **processing/equipment state machine**: IDLE → SETUP → READY → EXECUTING → PAUSE.
- Then the **event/report configuration**: S2F33 (define report), S2F35 (link to CEID), S2F37 (enable), and the S6F11 emission rule.
- Then **alarm management** (S5F1/S5F3) and **remote commands** (S2F41/S2F42 with the full HCACK enumeration).
- Then **spooling**: the SPOOL state machine, queue policy, transmit-on-reconnect ordering.
For each subsection: write rows for every legal transition *and* every illegal one (with the spec-mandated rejection/ACK code). Cite the spec section in `spec_ref`. Add a `notes` field whenever the spec is ambiguous, so the choice is recorded.
### Tooling for this layer
- A **table linter**: every `(machine, state, event)` pair must have either a transition row or an explicit "ignore"/"reject" row. The linter fails CI on missing pairs. This is what guarantees exhaustive coverage.
- A **table-to-Markdown renderer**: generate human-readable state tables and diagrams from the data, so reviewing the encoding against the SEMI document is tractable.
### Acceptance
The base E30 tables are complete, lint-clean, and human-reviewable against the SEMI document. You haven't built any runtime yet, but you have the asset everything else generates from.
## Layer 2 — Passive wire analyzer (Weeks 9–16, overlaps Layer 1)
Goal: ship the first user-facing artifact. The "Wireshark for SECS, but understands the state machines." Zero trust barrier, vendor-neutral, immediately useful.
### Architecture
```
pcap file / live capture / log file
│
▼
HSMS framer → SECS-II decoder → message stream
│
▼
┌──────────────────────────────────┐
│ passive state reconstructor │
│ (runs the transition tables │
│ on observed messages, in │
│ "observer" mode — never │
│ sends anything) │
└──────────┬───────────────────────┘
▼
┌──────────────────────────────────────────┐
│ diagnostic engine │
│ - reports protocol violations │
│ - explains rejections ("S2F41 rejected │
│ because control was LOCAL") │
│ - explains silences ("CEID 12 fired │
│ but report 5 was never enabled") │
│ - flags timeout misses │
└──────────┬───────────────────────────────┘
▼
UI (web or TUI):
- timeline of messages
- live state-machine view
- time-travel scrubbing
- golden-trace diff
```
### Deliverables
- **Passive reconstructor.** Consumes a SECS message stream and runs both endpoints' state machines in parallel (one for host, one for equipment), inferring state from observed messages.
- **Diagnostic engine.** Knows every legal transition (from the tables) and flags violations with human-readable explanations citing the spec reference.
- **UI.** Web UI is the right call (cross-platform, easy to share, screenshot-able for support tickets). Timeline + state panels + scrubber + filterable message inspector.
- **Capture options:** read pcap, attach to a live HSMS connection in tap mode (e.g. port mirror or local proxy that forwards and tees), import vendor log files (start with the secsgem Python format and add formats as users request).
### Acceptance
An integration engineer can drop a captured session in, see exactly why the bring-up failed, and point a senior at the rendered explanation rather than the raw log. This is the credibility-building artifact — open-source it.
### Suggested name worth squatting
Something like `secscope` or `gemtrace`. Pick early; matters for adoption.
## Layer 3 — Simulator (Weeks 16–24)
Goal: an active GEM-compliant simulator usable as (a) a virtual equipment for host developers, (b) a virtual host for equipment developers, and (c) the engine that drives Layer 4's test generator.
### Approach
The simulator is the transition tables, run in *active* mode rather than observer mode. The same dispatcher/event-queue architecture, but now it originates messages instead of just observing them. Key additions over the analyzer:
- **Scriptable scenarios.** A small DSL or just YAML for "carrier arrives at LP2, host issues START, after 30s a wafer-complete event fires." Lets users script reproducible test situations.
- **Equipment-specific data dictionary.** SVID/ECID/CEID/DVID definitions loaded from a config file per simulated tool. Default to a generic minimal tool; allow users to load richer ones.
- **Fault injection.** Drop the link mid-transaction, delay a reply past T3, return malformed messages, send out-of-spec ACK codes. This is what makes the simulator valuable for hardening host implementations.
- **Replay mode.** Take a captured session and replay it as either side. Enables "develop your host against last week's tool session."
### Acceptance
A host developer can run the simulator locally and develop against it without owning a physical tool, and an equipment developer can point their tool at it as a fake host for bring-up testing.
## Layer 4 — Conformance test generator (Weeks 20–28)
Goal: from the tables, mechanically generate the exhaustive conformance test suite (especially the negative cases) and run it against any implementation.
### How it works
For each row in the transition table, the generator emits a test case consisting of:
1. **Setup:** the sequence of messages needed to drive the system into `state`. Computed by graph-search over the table — find a shortest path from the initial state to the target state, using only known transitions.
2. **Stimulus:** the `event` (a message to send, or an internal trigger the simulator can fake).
3. **Assertions:** the expected `result`, `emitted_messages`, `ack_code`, and resulting state — all read directly from the row.
The generator runs as a host (or equipment) using the Layer 3 simulator engine, connected over HSMS to the system under test, and produces a structured report: pass/fail per row, with the spec reference cited on every failure.
### Negative-case coverage
The win is that the table contains every illegal `(state, event)` pair with its mandated rejection. The generator emits a test for each. This is the coverage humans skip out of tedium — sending S2F41 in every non-REMOTE state and asserting the correct HCACK comes back, for instance.
### Equipment-specific discovery
For the parts of the test suite that need tool-specific knowledge (SVID list, CEID list, recipe names), the generator first runs a discovery phase: S1F11 for SVID names, S1F13 for capabilities, S7F19 for recipe names, and so on. The discovered dictionary is merged with the generic tables to produce the full test plan.
### Acceptance
Point the generator at an implementation; get a pass/fail report against the full E30 (and later GEM300) behavioral spec, including all negative cases, with spec references on every failure.
## Layer 5 — GEM300 extensions (Months 6–12)
Goal: add E87 (carrier management), E90 (substrate tracking), E40 (process jobs), E94 (control jobs), E116 (EPT) as additional table files.
Because the architecture is additive — each standard is another set of transition rows plus possibly another machine in the dispatcher — adding GEM300 is wiring, not surgery. Sequence:
1. **E90 first** (substrate tracking) — applies to almost every tool, smallest dependency surface.
2. **E87** (carrier management) — only for tools that handle carriers, but the most asked-for after E90.
3. **E40/E94** (jobs) — adds the largest new state machines; do them as a pair since E94 references E40.
4. **E116** (EPT) — comparatively simple state model, valuable for fab metrics customers.
Each addition automatically gets analyzer support (new states render in the UI), simulator support (new scenarios scriptable), and conformance tests (new rows → new tests).
## Layer 6 — Spec revision diff and impact reporter (Month 9+)
Goal: when a new SEMI revision lands, produce a machine-generated report of exactly what changes for any given implementation.
### Mechanism
- Maintain one table file per spec revision (`e30-0307.yaml`, `e30-0712.yaml`, etc.).
- A **table diff tool** computes structural diffs: which rows added/removed/modified, which ACK codes changed, which transitions newly legal/illegal.
- An **impact reporter** re-runs the Layer 4 conformance generator against a customer's implementation using the new tables and produces a focused report: "these N behaviors must change to remain compliant under revision X; here are the spec references and the exact transitions affected."
### Honest framing
This is not "live" in the sense of auto-detecting that SEMI published something — a human still encodes each new revision into the table. The value is converting "your implementation drifted out of compliance, found out at a fab acceptance test" into "here is a precise diff the day you load the new table." That conversion is the product.
## Layer 7 (optional) — Runtime stack for equipment makers
Goal: a production-grade SECS/GEM runtime that equipment vendors embed in their tools. This is the high-trust, slow-cycle, sales-heavy artifact — only worth attempting after Layers 2–4 have built credibility and identified the right customer segment.
The runtime is the same dispatcher/event-queue executing the same tables, but in production mode: persistent spool, real timers, OS integration, supported APIs (Rust/C/C++/Python/.NET bindings via FFI). Skip this layer entirely if the project stays a side project; it's where commercial competition is hardest.
## Sequencing summary
| Months | Focus | User-visible artifact |
|---|---|---|
| 1 | Foundations (codec, framer, table schema) | none |
| 2–3 | Encode E30 base tables | none (internal asset) |
| 3–4 | Passive analyzer MVP | **Open-source release: SECS analyzer** |
| 5–6 | Simulator | simulator alongside analyzer |
| 6–7 | Conformance test generator | conformance reports |
| 7–9 | GEM300 (E90, E87) | analyzer + tests cover GEM300 |
| 9–12 | E40/E94/E116 + spec-revision diff | full GEM300 coverage + impact reports |
| 12+ | Optional: commercial runtime | (if pursued) |
## Risks and how each layer mitigates them
- **Wrong-table risk** (you encode the spec wrong, confidently emit wrong tests): mitigated by the `spec_ref` requirement and table-to-Markdown review, plus community review once open-sourced.
- **Adoption risk** (nobody uses it): mitigated by leading with the analyzer — low trust barrier, immediately useful, no commitment.
- **Scope risk** (GEM300 is huge): mitigated by additive architecture and only adding standards customers actually request.
- **Incumbent response risk** (a Cimetrix/PEER builds the analyzer themselves): mitigated by being open-source and vendor-neutral, which they structurally won't match.
- **Solo-bandwidth risk**: every layer is independently shippable. If life happens after Layer 2, the analyzer is still a real contribution.
## First concrete next step
Before any code: pick three integration engineers at equipment vendors (your IMS network has adjacencies) and ask them, unprompted, *"what's the worst part of bringing up a SECS/GEM interface?"* If "blind debugging" and "compliance test maintenance" come back without you naming them, the wedge is validated. Then start Layer 0.