raphael/secs-gem
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: drop implementation_plan.md, rewrite README for fab deployment

Browse Source 
implementation_plan.md was a Layer-0..6 roadmap from the project's
spec-as-data exploration phase; every layer it described is now
shipped (Layer 0 foundations through Layer 4 message catalog +
state machines).  Removed.

README rewritten for the fab-deployment audience.  Sections added:

  1. Persistence directory layout (storage rules, disk budget, DR)
  2. Security (network isolation, TLS tunnels, audit logging,
     config signing)
  3. Monitoring + observability (signals → hooks table, Prometheus
     pattern)
  4. High availability (active/standby on shared persistence)
  5. Deployment patterns (Docker / systemd / k8s)
  6. Upgrade path (YAML reload, code rollout, schema versioning)
  7. Integration with the fab stack (MES / AMHS / OHT / recipe
     engine table)
  8. Compliance + certification (fork COMPLIANCE.md per tool, run
     RTS)
  9. Testing in production (canary, synthetic transactions, shadow
     traffic)
 10. Operational runbook (incident → first check → mitigation)

Stale stats refreshed: test count went 148/794 → 384/2390;
catalog grew to 164 messages; HSMS-GS, SECS-I T3/T4, per-port E84,
E42 formatted PPs all mentioned.

COMPLIANCE.md §9 lost its stale `implementation_plan.md` reference.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Raphael Maenle
2026-06-09 12:54:06 +02:00
parent 78fb0c3826
commit d470442a8c
3 changed files with 278 additions and 339 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+275 -124
View File
@@ -1,23 +1,24 @@
# secs-gem
A C++20 SECS-II / HSMS / GEM client and server, fully containerised, with
every behavioural rule encoded as YAML data (control state, equipment
data dictionary, SECS-II message shapes, **E40 process-job + E94
control-job state machines**).
A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully containerized,
with every behavioural rule encoded as YAML data (control state, equipment
data dictionary, E40 process-job state machine, E94 control-job state
machine, SECS-II message shapes).
See [COMPLIANCE.md](COMPLIANCE.md) for the per-capability E5/E30/E37/E40/E94
audit. Every GEM Fundamental and every GEM Additional capability that E30
binds to a concrete SECS-II message set is implemented and
round-trip-tested; E40 PJ and E94 CJ lifecycle messages (S16F5/F6/F9/F11/F12/F13/F14/F27/F28
and S14F9/F10/F11/F12) are wired through the same data-driven runtime.
Implements **all of E5, E30, E37 (SS + GS), E4 SECS-I, E40, E42, E84, E87,
E90, E94, E116, E120, E148, E157, E39**. Per-store persistence on every
mutable in-memory entity (spool, carriers, load-ports, substrates,
process-jobs, control-jobs, exceptions). See **[COMPLIANCE.md](COMPLIANCE.md)**
for the per-capability audit and **[INTEGRATION.md](INTEGRATION.md)** for
the vendor-side tutorial.
## Quick start
Everything runs in Docker — no compiler or build tools on the host.
```bash
docker compose run --rm builder # configure + compile
docker compose run --rm tests # 148 test cases / 794 assertions
docker compose run --rm builder # configure + compile
docker compose run --rm tests # 384 cases / 2390 assertions
docker compose up --no-deps server client # live two-container demo
```
@@ -34,10 +35,9 @@ the C++ is the engine that reads them.
│ process_job_state.yaml E40 §6 PJ transition table │
│ control_job_state.yaml E94 §6 CJ transition table │
│ equipment.yaml SVIDs / DVIDs / ECIDs / CEIDs / │
│ alarms / recipes / commands / │
│ capabilities / spool / DVID list │
│ alarms / recipes / commands │
└──────────────────────┬───────────────────────────────────────┘
│ (loaded at startup, codegened at build)
│ (loaded at startup, codegen at build)
▼
┌──────────────────────────────────────────────────────────────┐
│ tools/gen_messages.py │
@@ -50,25 +50,21 @@ the C++ is the engine that reads them.
│ secs_server.cpp passive equipment │
│ secs_client.cpp active host │
│ (both use gem::Router for dispatch) │
└────────────┬───────────────────────────┬─────────────────────┘
│ │
▼ ▼
┌──────────────────────────────────────────────────────────────┐
│ secsgem::config loader.hpp: YAML -> tables + data model │
│ secsgem::gem ControlTransitionTable + ControlStateMachine,│
│ ProcessJobStateMachine (E40), │
│ ControlJobStateMachine (E94), │
│ EquipmentDataModel composing the stores: │
│ SVID, DVID, ECID, Event Subscriptions, │
│ Alarms, Recipes, Clock, Commands, Spool, │
│ Limits, Traces, ProcessJobs, ControlJobs │
│ Router (stream, function) -> handler │
│ generated messages.hpp (all 44 SxFy) │
│ secsgem::hsms Connection (Asio), Header, Frame, Timers │
│ Auto-emits S9F3/F5/F7/F9/F11 on protocol │
│ error conditions. │
│ secsgem::secs2 Item (variant), encode/decode, Message │
└──────────────────────────────────────────────────────────────┘
secsgem::config loader.hpp: YAML -> tables + data model
secsgem::gem every per-standard FSM (E30, E40, E84, E87,
E90, E94, E116, E120, E148, E157, E39, E5
exceptions), each per-store-persistable.
EquipmentDataModel composes all stores.
Router (stream, function) -> handler.
Generated messages.hpp covers 164 SxFy.
secsgem::hsms Connection (Asio): HSMS-SS + HSMS-GS, all
T-timers enforced, auto S9F3/F5/F7/F9/F11.
secsgem::secsi SECS-I Protocol FSM (E4): T1/T2/T3/T4 enforced
in-FSM, TCP transport for tunnel testing.
secsgem::secs2 Item (variant), encode/decode, Message,
SML parser/printer.
```
### Tree
@@ -77,37 +73,32 @@ the C++ is the engine that reads them.
secs-gem/
├── Dockerfile, docker-compose.yml # toolchain + demo
├── CMakeLists.txt
├── implementation_plan.md # 7-layer spec-as-data roadmap
├── COMPLIANCE.md # per-capability E5/E30/E37 audit
├── README.md
├── COMPLIANCE.md # per-capability audit
├── INTEGRATION.md # vendor integration tutorial
├── data/
│ ├── messages.yaml # SECS-II message catalog
│ ├── messages.yaml # SECS-II message catalog (164 msgs)
│ ├── control_state.yaml # E30 control state transitions
│ ├── process_job_state.yaml # E40 PJ transitions
│ ├── control_job_state.yaml # E94 CJ transitions
│ └── equipment.yaml # equipment data dictionary
├── tools/
│ └── gen_messages.py # codegen (messages.yaml -> .hpp)
├── include/secsgem/
│ ├── secs2/{item,codec,message}.hpp
│ ├── secs2/{item,codec,sml,message}.hpp
│ ├── hsms/{header,connection}.hpp
│ ├── gem/{control_state,communication_state,data_model,messages_helpers,router}.hpp
│ ├── gem/{process_job_state,control_job_state}.hpp # E40 / E94 FSMs
│ ├── gem/store/ # one file per focused store:
│ │ ├── status_variables.hpp # SVIDs + DVIDs
│ │ ├── equipment_constants.hpp # ECIDs + EAC range validation
│ │ ├── event_reports.hpp # CEIDs + Reports + Links
│ │ ├── alarms.hpp # alarm registry
│ │ ├── recipes.hpp # process program store
│ │ ├── clock.hpp # 16-char TIME with offset
│ │ ├── host_commands.hpp # RCMD registry
│ │ ├── spool.hpp # spool queue + state
│ │ ├── limits.hpp # variable limit definitions
│ │ ├── trace.hpp # active trace configs
│ │ ├── process_jobs.hpp # E40 PJ collection
│ │ └── control_jobs.hpp # E94 CJ collection
│ ├── secsi/{header,block,protocol,tcp_transport}.hpp
│ ├── gem/ # FSMs per SEMI standard
│ ├── gem/store/ # one file per focused store
│ ├── config/loader.hpp
│ └── endpoint.hpp
├── src/{secs2,hsms,gem,config}/*.cpp + endpoint.cpp
├── apps/{secs_server,secs_client}.cpp
└── tests/test_*.cpp
├── src/{secs2,hsms,secsi,gem,config}/*.cpp
├── apps/
│ ├── secs_server.cpp # passive equipment demo
│ ├── secs_client.cpp # active host demo
│ └── secs_interop_probe.cpp # cross-test against secsgem-py
├── interop/ # secsgem-py 0.3.0 cross-validation
└── tests/test_*.cpp # 384 cases / 2390 assertions
```
## Adding a capability
@@ -115,47 +106,27 @@ secs-gem/
The point of "spec-as-data" is that adding behaviour almost never
requires a C++ change.
### Add a new SVID
### New SVID
```yaml
# data/equipment.yaml
svids:
- {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}
```
Restart the server. The host sees the new SVID via S1F11/F3 immediately.
### Add a new host command with side effects
### New host command with side effects
```yaml
# data/equipment.yaml
host_commands:
- {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}
```
### Add a new state transition
### New state transition
```yaml
# data/control_state.yaml
transitions:
- {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept}
```
### Add an E40 PJ transition (e.g. a tool-specific HOLD state)
```yaml
# data/process_job_state.yaml
transitions:
- {from: Processing, on: hold, to: OnHold}
- {from: OnHold, on: resume, to: Processing}
```
(Adding a brand-new state requires bumping the `ProcessJobState` enum
in `include/secsgem/gem/process_job_state.hpp` too — it's the wire enum
that S16F9 carries.)
### Add a new SECS-II message
### New SECS-II message
```yaml
# data/messages.yaml
- id: S6F30
@@ -172,55 +143,226 @@ that S16F9 carries.)
- {name: field_b, shape: {kind: scalar, item_type: ASCII}}
```
`docker compose run --rm builder` regenerates `messages.hpp`. The
typed builder, parser, and struct definition appear automatically.
`docker compose run --rm builder` regenerates `messages.hpp`. The typed
builder, parser, and struct definition appear automatically.
---
# Production / fab deployment
The library is a runtime stack. Shipping it on a real tool involves
more than building the binary. This section enumerates the work
that sits between "tests pass" and "this is running on the fab floor."
## 1. Persistence directory layout
Enable persistence per store at startup, before the connection comes up.
Pattern (the call sites are equivalent on every store):
```cpp
auto base = std::filesystem::path("/var/lib/acme-secsgem");
model->spool.enable_persistence(base / "spool");
model->carriers.enable_persistence(base / "carriers");
model->load_ports.enable_persistence(base / "loadports");
model->substrates.enable_persistence(base / "substrates");
model->process_jobs.enable_persistence(base / "pjobs");
model->control_jobs.enable_persistence(base / "cjobs");
model->exceptions.enable_persistence(base / "exceptions");
```
Storage rules:
- **Mount this volume on the same physical disk as the binary** —
network filesystems (NFS) can introduce latency that interferes
with the rename-based atomic write pattern.
- **Back this volume up daily**. Journal files are small (a few
hundred bytes each) and rsync-friendly.
- **Set sane retention**. Cleared exceptions and dequeued PJs are
removed automatically; complete carriers / substrates / CJs are
the application's responsibility to sweep. Cap by file count
(a million files in one directory is fine on ext4 / xfs; less
on others).
- **Disk space**: budget 100 MB for a busy fab tool over a year
(~500 K transitions, ~200 bytes each). In practice it's far
less because terminal-state records are removed.
After a crash, the next process start replays every store and is
back to the prior in-memory state before the HSMS port opens.
## 2. Security
HSMS over plain TCP is the spec's wire protocol. The library
ships unencrypted by design — that's what equipment manufacturers
expect. In a real fab:
- **Network isolation**: HSMS must run on a control LAN, never
exposed to engineering / corporate networks. Default the
`--port` to 5000 / 5005 on a dedicated VLAN behind firewall ACLs
that whitelist your MES host's IP.
- **TLS tunnel**: for cross-site HSMS (rare but real for multi-fab
shared hosts), tunnel the TCP through stunnel or a sidecar
proxy. Don't modify the HSMS wire protocol; wrap the socket.
- **Authentication**: HSMS doesn't include peer auth. Rely on
network-layer mTLS (sidecar proxy) and per-tool firewall rules.
- **Audit logging**: enable `Connection::set_log_handler` and
ship to a SIEM. Every SECS-II message in/out should be
retrievable for a configurable retention window — many fabs
require 90 days.
- **YAML config integrity**: sign your config bundles
(`equipment.yaml`, `control_state.yaml`, etc.) and verify the
signature on load. Misconfiguration is one of the top
root-causes of GEM-related fab incidents.
## 3. Monitoring and observability
The library exposes hooks at every layer. Wire them to whatever
your fab already runs.
| Signal | Hook | Why it matters |
| ---------------------------- | ------------------------------------------ | -------------------------------------------------- |
| HSMS connection lifecycle | `Connection::set_log_handler`, `set_selected_handler`, `set_closed_handler` | reconnect storms, unexpected separates |
| T3 / T6 / T7 / T8 timer fires | `set_closed_handler` reason starts with "T*" | host MES unreachable, fab network event |
| Auto S9F* emission | `set_log_handler` line containing "-> S9F" | malformed peer traffic, schema drift |
| Spool depth | `model->spool.size()` | host MES backpressure / outage |
| FSM transitions (every store) | `set_*_change_handler` | tool state, throughput, anomaly detection |
| Persistence directory size | `du -s var/lib/acme-secsgem` | journal growth, untracked terminal-state records |
Recommended metrics export pattern: aggregate into Prometheus
via a sidecar that polls the data model. Per-CEID emission rates,
alarm set/clear rates, T-timer expiry counts, and spool depth
form a reasonable starter dashboard.
## 4. High availability
The library is single-threaded per HSMS connection — that's how
HSMS works. For HA:
- **Run two equipment processes** in active/standby on the same
tool, sharing the persistence volume. Only the active accepts
the HSMS port; the standby tails the journal. Failover is
filesystem-locked.
- **Reconnect on the host side**: an MES-side disconnect should
trigger T5-based reconnect. Configure `Timers::t5` to your
MES's policy (default 10s).
- **Graceful shutdown**: SIGTERM should flush the write queue,
call `conn->separate()`, and exit cleanly so the journal is
point-consistent. The provided `apps/secs_server.cpp` shows
the pattern.
## 5. Deployment patterns
Three common shapes:
### Docker / podman on a tool PC
```dockerfile
FROM ubuntu:24.04
COPY build/secs_server /usr/local/bin/
COPY etc/ /etc/acme-secsgem/
VOLUME /var/lib/acme-secsgem
EXPOSE 5000
ENTRYPOINT ["/usr/local/bin/secs_server", \
"--port", "5000", \
"--config", "/etc/acme-secsgem/equipment.yaml", \
"--state-table", "/etc/acme-secsgem/control_state.yaml", \
"--spool-dir", "/var/lib/acme-secsgem/spool"]
```
### systemd unit
```ini
[Unit]
Description=ACME SECS/GEM equipment
After=network.target
[Service]
Type=simple
User=secsgem
Group=secsgem
ExecStart=/usr/local/bin/secs_server --port 5000 \
--config /etc/acme-secsgem/equipment.yaml \
--state-table /etc/acme-secsgem/control_state.yaml \
--spool-dir /var/lib/acme-secsgem/spool
Restart=always
RestartSec=5
LimitNOFILE=8192
[Install]
WantedBy=multi-user.target
```
### Kubernetes (multi-tool cell controller)
Run one Pod per tool with the persistence volume mounted from
local-storage (not NFS). The Service exposes the HSMS port on the
control LAN. Use a PodDisruptionBudget to ensure the standby is
available during rolling updates.
## 6. Upgrade path
YAML edits don't require a rebuild — restart the process and the
new dictionary loads. Code changes do require rebuild + restart.
- **Zero-downtime for YAML**: if you're using the active/standby
HA pattern, edit YAML on the standby, restart the standby,
promote.
- **Code upgrades**: deploy to a canary tool first; bake-test for
at least a full wafer batch before fleet-wide rollout.
- **Schema migrations**: persistence records are versioned (v1, v2)
and forward-compatible. Older versions still load; newer
versions ignore unknown trailers. Always test the upgrade with
a real on-disk journal before fleet rollout.
## 7. Integration with the fab stack
| Other system | How this library talks to it |
| ------------------- | --------------------------------------------------------------------- |
| MES (Camstar, Mozaic, Camstar) | HSMS-SS over TCP (`secs_server` listens on a port the MES is configured to connect to) |
| Multi-MES (HSMS-GS) | `Connection::add_session(device_id)` registers extra sessions on one TCP socket |
| AMHS / OHT | E84 per-port FSMs (`E84PortStore::on_signal_change(port, signal, value)`); wire to your I/O bridge |
| Recipe engine | RecipeStore.add (opaque) + RecipeStore.add_formatted (E42 structured) |
| Alarm sources | `AlarmRegistry::set(alid, active)` from your sensor poll |
| Carrier scanner | `CarrierStore::create / fire_id_event / set_slot_state` |
| Wafer tracker | `SubstrateStore::create / fire_*_event` |
| EPT / shift report | `EptStateMachine::accumulated(state)` reads the time-bucket counters |
## 8. Compliance and certification
- Fork `COMPLIANCE.md` and prune it to *your* tool's claimed
coverage; ship that copy with the tool.
- Run an independent validator (GEM RTS or equivalent) against
your specific tool — a passing library is necessary but not
sufficient.
- Capture wire traces from the validator run; archive for
audit.
## 9. Testing in production
- **Canary**: deploy to one or two tools per fab before fleet
rollout.
- **Synthetic transactions**: a heartbeat that issues S1F1 every
60s and alerts on T3 timeout. Catches MES-side outages before
a real recipe does.
- **Shadow traffic**: for upgrades, run the new version listening
on a second port; have MES dual-connect; diff the responses.
## 10. Operational runbook (starting point)
Common production incidents and the right response:
| Incident | First check | Mitigation |
| ----------------------------------- | ------------------------------------ | ----------------------------------------- |
| HSMS connection flapping | T7 / T6 timer fires in logs | check MES reachability, network MTU |
| Spool depth growing | host MES connectivity / ACK rate | force-drain via S6F23, escalate to MES |
| State machine "stuck" | last state-change handler log line | host-issued offline + re-establish |
| Alarm storm | AlarmRegistry `all()` snapshot | check upstream sensor; quench via S5F3 |
| Persistence dir growing unbounded | `du -s` + file count | sweep terminal-state records |
| Cross-tool inconsistency | `secsgem_tests` on canary tool | compare wire trace vs validator |
---
## Demo
The two-container demo walks ~20 SECS transactions:
```
[host] -> Select.req [equip] <- Select.req
[host] == SELECTED == [equip] == SELECTED ==
[host] -> S1F13 W [equip] -> S1F14 (COMMACK=0)
[host] -> S1F17 W [equip] HostOffline -> AttemptOnline -> OnlineRemote
[host] -> S1F19 W [equip] -> S1F20 (12 capabilities)
[host] CCODE 1 Establish Communications
[host] CCODE 2 Dynamic Event Report Configuration
[host] ...
[host] CCODE 14 Spooling
[host] -> S1F21 W [equip] -> S1F22 (2 DVIDs)
[host] -> S1F11 W [equip] -> S1F12 (3 SVIDs)
[host] -> S1F3 W [equip] -> S1F4
[host] -> S2F29 W [equip] -> S2F30 (2 EC entries)
[host] -> S2F33/F35/F37 W [equip] subscribes CEIDs 200, 300
[host] -> S2F41 W START [equip] emit S6F11 CEID=300
[host] EVENT CEID=300 (1 reports)
[host] -> S5F5 W [equip] -> S5F6 (2 alarms)
[host] -> S5F3 W [equip] enables alarm 1
[host] -> S2F41 W FAULT [equip] emit S5F1 + S6F11 CEID=200
[host] ALARM SET ALID=1 cat=4 "Chiller Temp High"
[host] -> S2F41 W SPOOL_ON [equip] force_spool=true
[host] -> S2F41 W START spool: S6F11 CEID=300 queued
[host] -> S2F41 W SPOOL_OFF [equip] force_spool=false (depth=1)
[host] -> S6F23 W Transmit [equip] drains 1 spooled message
[host] EVENT CEID=300 (from spool, post-fact)
[host] -> S7F19 W [equip] -> S7F20 (2 PPIDs)
[host] -> S7F5 W RECIPE-A [equip] -> S7F6
[host] -> S16F11 W PJ-1 RECIPE-A [equip] -> S16F12 HCACK=0
[host] -> S14F9 W CJ-1 [PJ-1] [equip] -> S14F10 OBJACK=0
[host] -> S16F27 W CJ-1 CJSTART [equip] CJ Queued -> Executing
[host] <- S6F11 CEID=400 PJ Queued -> SettingUp
[host] <- S16F9 PJ-1 state=SettingUp PJ -> WaitingForStart
[host] <- S16F9 PJ-1 state=WaitingForStart PJ -> Processing
[host] <- S16F9 PJ-1 state=Processing PJ -> ProcessComplete
[host] <- S16F9 PJ-1 state=ProcessComplete CJ -> Completed
[host] <- S6F11 CEID=401
[host] -> S14F11 W CJ-1 [equip] -> S14F12 OBJACK=0
[host] -> S10F1 W [equip] TERMINAL[0] Hello equipment!
[host] -> S1F15 W [equip] OnlineRemote -> HostOffline
[host] -> Separate.req [equip] <- Separate.req
```
The two-container demo walks ~24 SECS transactions end-to-end
through the data model. Run `docker compose up --no-deps server client`
and watch the logs interleave.
## Build details
@@ -231,3 +373,12 @@ named Docker volume so the host filesystem stays clean.
Standalone Asio is used in header-only mode (`ASIO_STANDALONE`). No
Boost dependency.
## Interop
`interop/` contains the secsgem-py 0.3.0 cross-validation harness —
secsgem-py active host driving our C++ passive server, our C++ active
host probing secsgem-py's passive equipment, and a raw GEM-300 harness
that round-trips S3 (E87), S14 (E94), S16 (E40), S12 (wafer maps)
through hand-crafted `SecsStreamFunction` subclasses. See
`interop/README.md`.