docs: drop implementation_plan.md, rewrite README for fab deployment
implementation_plan.md was a Layer-0..6 roadmap from the project's
spec-as-data exploration phase; every layer it described is now
shipped (Layer 0 foundations through Layer 4 message catalog +
state machines). Removed.
README rewritten for the fab-deployment audience. Sections added:
1. Persistence directory layout (storage rules, disk budget, DR)
2. Security (network isolation, TLS tunnels, audit logging,
config signing)
3. Monitoring + observability (signals → hooks table, Prometheus
pattern)
4. High availability (active/standby on shared persistence)
5. Deployment patterns (Docker / systemd / k8s)
6. Upgrade path (YAML reload, code rollout, schema versioning)
7. Integration with the fab stack (MES / AMHS / OHT / recipe
engine table)
8. Compliance + certification (fork COMPLIANCE.md per tool, run
RTS)
9. Testing in production (canary, synthetic transactions, shadow
traffic)
10. Operational runbook (incident → first check → mitigation)
Stale stats refreshed: test count went 148/794 → 384/2390;
catalog grew to 164 messages; HSMS-GS, SECS-I T3/T4, per-port E84,
E42 formatted PPs all mentioned.
COMPLIANCE.md §9 lost its stale `implementation_plan.md` reference.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
+3
-4
@@ -406,10 +406,9 @@ What this codebase does **not** demonstrate, and what a real
|
||||
|
||||
1. **Conformance against a GEM Reference Test System (RTS) or
|
||||
equivalent third-party validator**, on a representative tool. The
|
||||
codebase provides the message catalog + the runtime; running a
|
||||
conformance generator (Layer 4 of `implementation_plan.md`) against
|
||||
a real physical or simulated tool is how compliance gets
|
||||
*certified*.
|
||||
codebase provides the message catalog + the runtime; running an
|
||||
external validator against a real physical or simulated tool is
|
||||
how compliance gets *certified*.
|
||||
2. **Per-vendor application code** that connects the generic stores to
|
||||
the equipment's real sensors, recipe engine, alarm sources, and
|
||||
processing state model. The codebase provides the data model and
|
||||
|
||||
@@ -1,23 +1,24 @@
|
||||
# secs-gem
|
||||
|
||||
A C++20 SECS-II / HSMS / GEM client and server, fully containerised, with
|
||||
every behavioural rule encoded as YAML data (control state, equipment
|
||||
data dictionary, SECS-II message shapes, **E40 process-job + E94
|
||||
control-job state machines**).
|
||||
A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully containerized,
|
||||
with every behavioural rule encoded as YAML data (control state, equipment
|
||||
data dictionary, E40 process-job state machine, E94 control-job state
|
||||
machine, SECS-II message shapes).
|
||||
|
||||
See [COMPLIANCE.md](COMPLIANCE.md) for the per-capability E5/E30/E37/E40/E94
|
||||
audit. Every GEM Fundamental and every GEM Additional capability that E30
|
||||
binds to a concrete SECS-II message set is implemented and
|
||||
round-trip-tested; E40 PJ and E94 CJ lifecycle messages (S16F5/F6/F9/F11/F12/F13/F14/F27/F28
|
||||
and S14F9/F10/F11/F12) are wired through the same data-driven runtime.
|
||||
Implements **all of E5, E30, E37 (SS + GS), E4 SECS-I, E40, E42, E84, E87,
|
||||
E90, E94, E116, E120, E148, E157, E39**. Per-store persistence on every
|
||||
mutable in-memory entity (spool, carriers, load-ports, substrates,
|
||||
process-jobs, control-jobs, exceptions). See **[COMPLIANCE.md](COMPLIANCE.md)**
|
||||
for the per-capability audit and **[INTEGRATION.md](INTEGRATION.md)** for
|
||||
the vendor-side tutorial.
|
||||
|
||||
## Quick start
|
||||
|
||||
Everything runs in Docker — no compiler or build tools on the host.
|
||||
|
||||
```bash
|
||||
docker compose run --rm builder # configure + compile
|
||||
docker compose run --rm tests # 148 test cases / 794 assertions
|
||||
docker compose run --rm builder # configure + compile
|
||||
docker compose run --rm tests # 384 cases / 2390 assertions
|
||||
docker compose up --no-deps server client # live two-container demo
|
||||
```
|
||||
|
||||
@@ -34,10 +35,9 @@ the C++ is the engine that reads them.
|
||||
│ process_job_state.yaml E40 §6 PJ transition table │
|
||||
│ control_job_state.yaml E94 §6 CJ transition table │
|
||||
│ equipment.yaml SVIDs / DVIDs / ECIDs / CEIDs / │
|
||||
│ alarms / recipes / commands / │
|
||||
│ capabilities / spool / DVID list │
|
||||
│ alarms / recipes / commands │
|
||||
└──────────────────────┬───────────────────────────────────────┘
|
||||
│ (loaded at startup, codegened at build)
|
||||
│ (loaded at startup, codegen at build)
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ tools/gen_messages.py │
|
||||
@@ -50,25 +50,21 @@ the C++ is the engine that reads them.
|
||||
│ secs_server.cpp passive equipment │
|
||||
│ secs_client.cpp active host │
|
||||
│ (both use gem::Router for dispatch) │
|
||||
└────────────┬───────────────────────────┬─────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ secsgem::config loader.hpp: YAML -> tables + data model │
|
||||
│ secsgem::gem ControlTransitionTable + ControlStateMachine,│
|
||||
│ ProcessJobStateMachine (E40), │
|
||||
│ ControlJobStateMachine (E94), │
|
||||
│ EquipmentDataModel composing the stores: │
|
||||
│ SVID, DVID, ECID, Event Subscriptions, │
|
||||
│ Alarms, Recipes, Clock, Commands, Spool, │
|
||||
│ Limits, Traces, ProcessJobs, ControlJobs │
|
||||
│ Router (stream, function) -> handler │
|
||||
│ generated messages.hpp (all 44 SxFy) │
|
||||
│ secsgem::hsms Connection (Asio), Header, Frame, Timers │
|
||||
│ Auto-emits S9F3/F5/F7/F9/F11 on protocol │
|
||||
│ error conditions. │
|
||||
│ secsgem::secs2 Item (variant), encode/decode, Message │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
|
||||
secsgem::config loader.hpp: YAML -> tables + data model
|
||||
secsgem::gem every per-standard FSM (E30, E40, E84, E87,
|
||||
E90, E94, E116, E120, E148, E157, E39, E5
|
||||
exceptions), each per-store-persistable.
|
||||
EquipmentDataModel composes all stores.
|
||||
Router (stream, function) -> handler.
|
||||
Generated messages.hpp covers 164 SxFy.
|
||||
secsgem::hsms Connection (Asio): HSMS-SS + HSMS-GS, all
|
||||
T-timers enforced, auto S9F3/F5/F7/F9/F11.
|
||||
secsgem::secsi SECS-I Protocol FSM (E4): T1/T2/T3/T4 enforced
|
||||
in-FSM, TCP transport for tunnel testing.
|
||||
secsgem::secs2 Item (variant), encode/decode, Message,
|
||||
SML parser/printer.
|
||||
```
|
||||
|
||||
### Tree
|
||||
@@ -77,37 +73,32 @@ the C++ is the engine that reads them.
|
||||
secs-gem/
|
||||
├── Dockerfile, docker-compose.yml # toolchain + demo
|
||||
├── CMakeLists.txt
|
||||
├── implementation_plan.md # 7-layer spec-as-data roadmap
|
||||
├── COMPLIANCE.md # per-capability E5/E30/E37 audit
|
||||
├── README.md
|
||||
├── COMPLIANCE.md # per-capability audit
|
||||
├── INTEGRATION.md # vendor integration tutorial
|
||||
├── data/
|
||||
│ ├── messages.yaml # SECS-II message catalog
|
||||
│ ├── messages.yaml # SECS-II message catalog (164 msgs)
|
||||
│ ├── control_state.yaml # E30 control state transitions
|
||||
│ ├── process_job_state.yaml # E40 PJ transitions
|
||||
│ ├── control_job_state.yaml # E94 CJ transitions
|
||||
│ └── equipment.yaml # equipment data dictionary
|
||||
├── tools/
|
||||
│ └── gen_messages.py # codegen (messages.yaml -> .hpp)
|
||||
├── include/secsgem/
|
||||
│ ├── secs2/{item,codec,message}.hpp
|
||||
│ ├── secs2/{item,codec,sml,message}.hpp
|
||||
│ ├── hsms/{header,connection}.hpp
|
||||
│ ├── gem/{control_state,communication_state,data_model,messages_helpers,router}.hpp
|
||||
│ ├── gem/{process_job_state,control_job_state}.hpp # E40 / E94 FSMs
|
||||
│ ├── gem/store/ # one file per focused store:
|
||||
│ │ ├── status_variables.hpp # SVIDs + DVIDs
|
||||
│ │ ├── equipment_constants.hpp # ECIDs + EAC range validation
|
||||
│ │ ├── event_reports.hpp # CEIDs + Reports + Links
|
||||
│ │ ├── alarms.hpp # alarm registry
|
||||
│ │ ├── recipes.hpp # process program store
|
||||
│ │ ├── clock.hpp # 16-char TIME with offset
|
||||
│ │ ├── host_commands.hpp # RCMD registry
|
||||
│ │ ├── spool.hpp # spool queue + state
|
||||
│ │ ├── limits.hpp # variable limit definitions
|
||||
│ │ ├── trace.hpp # active trace configs
|
||||
│ │ ├── process_jobs.hpp # E40 PJ collection
|
||||
│ │ └── control_jobs.hpp # E94 CJ collection
|
||||
│ ├── secsi/{header,block,protocol,tcp_transport}.hpp
|
||||
│ ├── gem/ # FSMs per SEMI standard
|
||||
│ ├── gem/store/ # one file per focused store
|
||||
│ ├── config/loader.hpp
|
||||
│ └── endpoint.hpp
|
||||
├── src/{secs2,hsms,gem,config}/*.cpp + endpoint.cpp
|
||||
├── apps/{secs_server,secs_client}.cpp
|
||||
└── tests/test_*.cpp
|
||||
├── src/{secs2,hsms,secsi,gem,config}/*.cpp
|
||||
├── apps/
|
||||
│ ├── secs_server.cpp # passive equipment demo
|
||||
│ ├── secs_client.cpp # active host demo
|
||||
│ └── secs_interop_probe.cpp # cross-test against secsgem-py
|
||||
├── interop/ # secsgem-py 0.3.0 cross-validation
|
||||
└── tests/test_*.cpp # 384 cases / 2390 assertions
|
||||
```
|
||||
|
||||
## Adding a capability
|
||||
@@ -115,47 +106,27 @@ secs-gem/
|
||||
The point of "spec-as-data" is that adding behaviour almost never
|
||||
requires a C++ change.
|
||||
|
||||
### Add a new SVID
|
||||
|
||||
### New SVID
|
||||
```yaml
|
||||
# data/equipment.yaml
|
||||
svids:
|
||||
- {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}
|
||||
```
|
||||
|
||||
Restart the server. The host sees the new SVID via S1F11/F3 immediately.
|
||||
|
||||
### Add a new host command with side effects
|
||||
|
||||
### New host command with side effects
|
||||
```yaml
|
||||
# data/equipment.yaml
|
||||
host_commands:
|
||||
- {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}
|
||||
```
|
||||
|
||||
### Add a new state transition
|
||||
|
||||
### New state transition
|
||||
```yaml
|
||||
# data/control_state.yaml
|
||||
transitions:
|
||||
- {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept}
|
||||
```
|
||||
|
||||
### Add an E40 PJ transition (e.g. a tool-specific HOLD state)
|
||||
|
||||
```yaml
|
||||
# data/process_job_state.yaml
|
||||
transitions:
|
||||
- {from: Processing, on: hold, to: OnHold}
|
||||
- {from: OnHold, on: resume, to: Processing}
|
||||
```
|
||||
|
||||
(Adding a brand-new state requires bumping the `ProcessJobState` enum
|
||||
in `include/secsgem/gem/process_job_state.hpp` too — it's the wire enum
|
||||
that S16F9 carries.)
|
||||
|
||||
### Add a new SECS-II message
|
||||
|
||||
### New SECS-II message
|
||||
```yaml
|
||||
# data/messages.yaml
|
||||
- id: S6F30
|
||||
@@ -172,55 +143,226 @@ that S16F9 carries.)
|
||||
- {name: field_b, shape: {kind: scalar, item_type: ASCII}}
|
||||
```
|
||||
|
||||
`docker compose run --rm builder` regenerates `messages.hpp`. The
|
||||
typed builder, parser, and struct definition appear automatically.
|
||||
`docker compose run --rm builder` regenerates `messages.hpp`. The typed
|
||||
builder, parser, and struct definition appear automatically.
|
||||
|
||||
---
|
||||
|
||||
# Production / fab deployment
|
||||
|
||||
The library is a runtime stack. Shipping it on a real tool involves
|
||||
more than building the binary. This section enumerates the work
|
||||
that sits between "tests pass" and "this is running on the fab floor."
|
||||
|
||||
## 1. Persistence directory layout
|
||||
|
||||
Enable persistence per store at startup, before the connection comes up.
|
||||
Pattern (the call sites are equivalent on every store):
|
||||
|
||||
```cpp
|
||||
auto base = std::filesystem::path("/var/lib/acme-secsgem");
|
||||
model->spool.enable_persistence(base / "spool");
|
||||
model->carriers.enable_persistence(base / "carriers");
|
||||
model->load_ports.enable_persistence(base / "loadports");
|
||||
model->substrates.enable_persistence(base / "substrates");
|
||||
model->process_jobs.enable_persistence(base / "pjobs");
|
||||
model->control_jobs.enable_persistence(base / "cjobs");
|
||||
model->exceptions.enable_persistence(base / "exceptions");
|
||||
```
|
||||
|
||||
Storage rules:
|
||||
|
||||
- **Mount this volume on the same physical disk as the binary** —
|
||||
network filesystems (NFS) can introduce latency that interferes
|
||||
with the rename-based atomic write pattern.
|
||||
- **Back this volume up daily**. Journal files are small (a few
|
||||
hundred bytes each) and rsync-friendly.
|
||||
- **Set sane retention**. Cleared exceptions and dequeued PJs are
|
||||
removed automatically; complete carriers / substrates / CJs are
|
||||
the application's responsibility to sweep. Cap by file count
|
||||
(a million files in one directory is fine on ext4 / xfs; less
|
||||
on others).
|
||||
- **Disk space**: budget 100 MB for a busy fab tool over a year
|
||||
(~500 K transitions, ~200 bytes each). In practice it's far
|
||||
less because terminal-state records are removed.
|
||||
|
||||
After a crash, the next process start replays every store and is
|
||||
back to the prior in-memory state before the HSMS port opens.
|
||||
|
||||
## 2. Security
|
||||
|
||||
HSMS over plain TCP is the spec's wire protocol. The library
|
||||
ships unencrypted by design — that's what equipment manufacturers
|
||||
expect. In a real fab:
|
||||
|
||||
- **Network isolation**: HSMS must run on a control LAN, never
|
||||
exposed to engineering / corporate networks. Default the
|
||||
`--port` to 5000 / 5005 on a dedicated VLAN behind firewall ACLs
|
||||
that whitelist your MES host's IP.
|
||||
- **TLS tunnel**: for cross-site HSMS (rare but real for multi-fab
|
||||
shared hosts), tunnel the TCP through stunnel or a sidecar
|
||||
proxy. Don't modify the HSMS wire protocol; wrap the socket.
|
||||
- **Authentication**: HSMS doesn't include peer auth. Rely on
|
||||
network-layer mTLS (sidecar proxy) and per-tool firewall rules.
|
||||
- **Audit logging**: enable `Connection::set_log_handler` and
|
||||
ship to a SIEM. Every SECS-II message in/out should be
|
||||
retrievable for a configurable retention window — many fabs
|
||||
require 90 days.
|
||||
- **YAML config integrity**: sign your config bundles
|
||||
(`equipment.yaml`, `control_state.yaml`, etc.) and verify the
|
||||
signature on load. Misconfiguration is one of the top
|
||||
root-causes of GEM-related fab incidents.
|
||||
|
||||
## 3. Monitoring and observability
|
||||
|
||||
The library exposes hooks at every layer. Wire them to whatever
|
||||
your fab already runs.
|
||||
|
||||
| Signal | Hook | Why it matters |
|
||||
| ---------------------------- | ------------------------------------------ | -------------------------------------------------- |
|
||||
| HSMS connection lifecycle | `Connection::set_log_handler`, `set_selected_handler`, `set_closed_handler` | reconnect storms, unexpected separates |
|
||||
| T3 / T6 / T7 / T8 timer fires | `set_closed_handler` reason starts with "T*" | host MES unreachable, fab network event |
|
||||
| Auto S9F* emission | `set_log_handler` line containing "-> S9F" | malformed peer traffic, schema drift |
|
||||
| Spool depth | `model->spool.size()` | host MES backpressure / outage |
|
||||
| FSM transitions (every store) | `set_*_change_handler` | tool state, throughput, anomaly detection |
|
||||
| Persistence directory size | `du -s var/lib/acme-secsgem` | journal growth, untracked terminal-state records |
|
||||
|
||||
Recommended metrics export pattern: aggregate into Prometheus
|
||||
via a sidecar that polls the data model. Per-CEID emission rates,
|
||||
alarm set/clear rates, T-timer expiry counts, and spool depth
|
||||
form a reasonable starter dashboard.
|
||||
|
||||
## 4. High availability
|
||||
|
||||
The library is single-threaded per HSMS connection — that's how
|
||||
HSMS works. For HA:
|
||||
|
||||
- **Run two equipment processes** in active/standby on the same
|
||||
tool, sharing the persistence volume. Only the active accepts
|
||||
the HSMS port; the standby tails the journal. Failover is
|
||||
filesystem-locked.
|
||||
- **Reconnect on the host side**: an MES-side disconnect should
|
||||
trigger T5-based reconnect. Configure `Timers::t5` to your
|
||||
MES's policy (default 10s).
|
||||
- **Graceful shutdown**: SIGTERM should flush the write queue,
|
||||
call `conn->separate()`, and exit cleanly so the journal is
|
||||
point-consistent. The provided `apps/secs_server.cpp` shows
|
||||
the pattern.
|
||||
|
||||
## 5. Deployment patterns
|
||||
|
||||
Three common shapes:
|
||||
|
||||
### Docker / podman on a tool PC
|
||||
```dockerfile
|
||||
FROM ubuntu:24.04
|
||||
COPY build/secs_server /usr/local/bin/
|
||||
COPY etc/ /etc/acme-secsgem/
|
||||
VOLUME /var/lib/acme-secsgem
|
||||
EXPOSE 5000
|
||||
ENTRYPOINT ["/usr/local/bin/secs_server", \
|
||||
"--port", "5000", \
|
||||
"--config", "/etc/acme-secsgem/equipment.yaml", \
|
||||
"--state-table", "/etc/acme-secsgem/control_state.yaml", \
|
||||
"--spool-dir", "/var/lib/acme-secsgem/spool"]
|
||||
```
|
||||
|
||||
### systemd unit
|
||||
```ini
|
||||
[Unit]
|
||||
Description=ACME SECS/GEM equipment
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=secsgem
|
||||
Group=secsgem
|
||||
ExecStart=/usr/local/bin/secs_server --port 5000 \
|
||||
--config /etc/acme-secsgem/equipment.yaml \
|
||||
--state-table /etc/acme-secsgem/control_state.yaml \
|
||||
--spool-dir /var/lib/acme-secsgem/spool
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
LimitNOFILE=8192
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
### Kubernetes (multi-tool cell controller)
|
||||
Run one Pod per tool with the persistence volume mounted from
|
||||
local-storage (not NFS). The Service exposes the HSMS port on the
|
||||
control LAN. Use a PodDisruptionBudget to ensure the standby is
|
||||
available during rolling updates.
|
||||
|
||||
## 6. Upgrade path
|
||||
|
||||
YAML edits don't require a rebuild — restart the process and the
|
||||
new dictionary loads. Code changes do require rebuild + restart.
|
||||
|
||||
- **Zero-downtime for YAML**: if you're using the active/standby
|
||||
HA pattern, edit YAML on the standby, restart the standby,
|
||||
promote.
|
||||
- **Code upgrades**: deploy to a canary tool first; bake-test for
|
||||
at least a full wafer batch before fleet-wide rollout.
|
||||
- **Schema migrations**: persistence records are versioned (v1, v2)
|
||||
and forward-compatible. Older versions still load; newer
|
||||
versions ignore unknown trailers. Always test the upgrade with
|
||||
a real on-disk journal before fleet rollout.
|
||||
|
||||
## 7. Integration with the fab stack
|
||||
|
||||
| Other system | How this library talks to it |
|
||||
| ------------------- | --------------------------------------------------------------------- |
|
||||
| MES (Camstar, Mozaic, Camstar) | HSMS-SS over TCP (`secs_server` listens on a port the MES is configured to connect to) |
|
||||
| Multi-MES (HSMS-GS) | `Connection::add_session(device_id)` registers extra sessions on one TCP socket |
|
||||
| AMHS / OHT | E84 per-port FSMs (`E84PortStore::on_signal_change(port, signal, value)`); wire to your I/O bridge |
|
||||
| Recipe engine | RecipeStore.add (opaque) + RecipeStore.add_formatted (E42 structured) |
|
||||
| Alarm sources | `AlarmRegistry::set(alid, active)` from your sensor poll |
|
||||
| Carrier scanner | `CarrierStore::create / fire_id_event / set_slot_state` |
|
||||
| Wafer tracker | `SubstrateStore::create / fire_*_event` |
|
||||
| EPT / shift report | `EptStateMachine::accumulated(state)` reads the time-bucket counters |
|
||||
|
||||
## 8. Compliance and certification
|
||||
|
||||
- Fork `COMPLIANCE.md` and prune it to *your* tool's claimed
|
||||
coverage; ship that copy with the tool.
|
||||
- Run an independent validator (GEM RTS or equivalent) against
|
||||
your specific tool — a passing library is necessary but not
|
||||
sufficient.
|
||||
- Capture wire traces from the validator run; archive for
|
||||
audit.
|
||||
|
||||
## 9. Testing in production
|
||||
|
||||
- **Canary**: deploy to one or two tools per fab before fleet
|
||||
rollout.
|
||||
- **Synthetic transactions**: a heartbeat that issues S1F1 every
|
||||
60s and alerts on T3 timeout. Catches MES-side outages before
|
||||
a real recipe does.
|
||||
- **Shadow traffic**: for upgrades, run the new version listening
|
||||
on a second port; have MES dual-connect; diff the responses.
|
||||
|
||||
## 10. Operational runbook (starting point)
|
||||
|
||||
Common production incidents and the right response:
|
||||
|
||||
| Incident | First check | Mitigation |
|
||||
| ----------------------------------- | ------------------------------------ | ----------------------------------------- |
|
||||
| HSMS connection flapping | T7 / T6 timer fires in logs | check MES reachability, network MTU |
|
||||
| Spool depth growing | host MES connectivity / ACK rate | force-drain via S6F23, escalate to MES |
|
||||
| State machine "stuck" | last state-change handler log line | host-issued offline + re-establish |
|
||||
| Alarm storm | AlarmRegistry `all()` snapshot | check upstream sensor; quench via S5F3 |
|
||||
| Persistence dir growing unbounded | `du -s` + file count | sweep terminal-state records |
|
||||
| Cross-tool inconsistency | `secsgem_tests` on canary tool | compare wire trace vs validator |
|
||||
|
||||
---
|
||||
|
||||
## Demo
|
||||
|
||||
The two-container demo walks ~20 SECS transactions:
|
||||
|
||||
```
|
||||
[host] -> Select.req [equip] <- Select.req
|
||||
[host] == SELECTED == [equip] == SELECTED ==
|
||||
[host] -> S1F13 W [equip] -> S1F14 (COMMACK=0)
|
||||
[host] -> S1F17 W [equip] HostOffline -> AttemptOnline -> OnlineRemote
|
||||
[host] -> S1F19 W [equip] -> S1F20 (12 capabilities)
|
||||
[host] CCODE 1 Establish Communications
|
||||
[host] CCODE 2 Dynamic Event Report Configuration
|
||||
[host] ...
|
||||
[host] CCODE 14 Spooling
|
||||
[host] -> S1F21 W [equip] -> S1F22 (2 DVIDs)
|
||||
[host] -> S1F11 W [equip] -> S1F12 (3 SVIDs)
|
||||
[host] -> S1F3 W [equip] -> S1F4
|
||||
[host] -> S2F29 W [equip] -> S2F30 (2 EC entries)
|
||||
[host] -> S2F33/F35/F37 W [equip] subscribes CEIDs 200, 300
|
||||
[host] -> S2F41 W START [equip] emit S6F11 CEID=300
|
||||
[host] EVENT CEID=300 (1 reports)
|
||||
[host] -> S5F5 W [equip] -> S5F6 (2 alarms)
|
||||
[host] -> S5F3 W [equip] enables alarm 1
|
||||
[host] -> S2F41 W FAULT [equip] emit S5F1 + S6F11 CEID=200
|
||||
[host] ALARM SET ALID=1 cat=4 "Chiller Temp High"
|
||||
[host] -> S2F41 W SPOOL_ON [equip] force_spool=true
|
||||
[host] -> S2F41 W START spool: S6F11 CEID=300 queued
|
||||
[host] -> S2F41 W SPOOL_OFF [equip] force_spool=false (depth=1)
|
||||
[host] -> S6F23 W Transmit [equip] drains 1 spooled message
|
||||
[host] EVENT CEID=300 (from spool, post-fact)
|
||||
[host] -> S7F19 W [equip] -> S7F20 (2 PPIDs)
|
||||
[host] -> S7F5 W RECIPE-A [equip] -> S7F6
|
||||
[host] -> S16F11 W PJ-1 RECIPE-A [equip] -> S16F12 HCACK=0
|
||||
[host] -> S14F9 W CJ-1 [PJ-1] [equip] -> S14F10 OBJACK=0
|
||||
[host] -> S16F27 W CJ-1 CJSTART [equip] CJ Queued -> Executing
|
||||
[host] <- S6F11 CEID=400 PJ Queued -> SettingUp
|
||||
[host] <- S16F9 PJ-1 state=SettingUp PJ -> WaitingForStart
|
||||
[host] <- S16F9 PJ-1 state=WaitingForStart PJ -> Processing
|
||||
[host] <- S16F9 PJ-1 state=Processing PJ -> ProcessComplete
|
||||
[host] <- S16F9 PJ-1 state=ProcessComplete CJ -> Completed
|
||||
[host] <- S6F11 CEID=401
|
||||
[host] -> S14F11 W CJ-1 [equip] -> S14F12 OBJACK=0
|
||||
[host] -> S10F1 W [equip] TERMINAL[0] Hello equipment!
|
||||
[host] -> S1F15 W [equip] OnlineRemote -> HostOffline
|
||||
[host] -> Separate.req [equip] <- Separate.req
|
||||
```
|
||||
The two-container demo walks ~24 SECS transactions end-to-end
|
||||
through the data model. Run `docker compose up --no-deps server client`
|
||||
and watch the logs interleave.
|
||||
|
||||
## Build details
|
||||
|
||||
@@ -231,3 +373,12 @@ named Docker volume so the host filesystem stays clean.
|
||||
|
||||
Standalone Asio is used in header-only mode (`ASIO_STANDALONE`). No
|
||||
Boost dependency.
|
||||
|
||||
## Interop
|
||||
|
||||
`interop/` contains the secsgem-py 0.3.0 cross-validation harness —
|
||||
secsgem-py active host driving our C++ passive server, our C++ active
|
||||
host probing secsgem-py's passive equipment, and a raw GEM-300 harness
|
||||
that round-trips S3 (E87), S14 (E94), S16 (E40), S12 (wafer maps)
|
||||
through hand-crafted `SecsStreamFunction` subclasses. See
|
||||
`interop/README.md`.
|
||||
|
||||
@@ -1,211 +0,0 @@
|
||||
# SECS/GEM Spec-as-Data Project — Implementation Plan
|
||||
|
||||
A layered plan for building a SECS/GEM toolchain whose core asset is the SEMI behavioral spec encoded as machine-readable transition tables. From that one artifact you derive a runtime stack, a passive wire analyzer, a conformance test generator, and a simulator. The plan is sequenced so each layer is independently useful and ships value on its own — you can stop after Layer 2 and still have a shipped product.
|
||||
|
||||
## Guiding principles
|
||||
|
||||
A few discipline rules that hold across every layer:
|
||||
|
||||
- **One source of truth.** The spec tables are the only place behavioral rules live. Runtime, tests, analyzer, simulator, and docs all *read* from them; none of them re-encode the spec.
|
||||
- **Data, not code.** Tables are versioned data files (YAML/TOML/JSON), not classes. Adding GEM300 standards or new spec revisions is additive — drop in another table file.
|
||||
- **One mutation point.** Every state change in every runtime/simulator goes through one validated apply-step that emits the collection event as a side effect. No alternate paths.
|
||||
- **Exhaustive coverage, including rejections.** Every `(machine, state, event)` pair has an explicit entry. No defaults that silently swallow.
|
||||
- **Trust ladder.** Ship the lowest-trust-barrier artifact first (the passive analyzer touches no wafer), build credibility, then ascend to higher-stakes artifacts (simulator, runtime).
|
||||
- **Validate the pain before building each layer.** Talk to integration engineers between layers. If they don't unprompted complain about what the next layer solves, don't build it.
|
||||
|
||||
## Layer 0 — Foundations (Weeks 1–4)
|
||||
|
||||
Goal: establish the data model and conventions everything else depends on. No user-facing artifact yet.
|
||||
|
||||
### Deliverables
|
||||
|
||||
- **Transition table schema.** A formal schema (JSON Schema or similar) for rows of the form:
|
||||
```
|
||||
machine, state, event, guard, result, new_state, emitted_messages, emitted_ceids, ack_code, notes, spec_ref
|
||||
```
|
||||
Include `spec_ref` (e.g. `E30 §6.5.2`) on every row so the table is auditable line-by-line against the standard.
|
||||
- **SECS-II message codec.** The byte-level encoder/decoder for SECS-II data items (L, A, B, U1/U2/U4, I1/I2/I4, F4/F8, etc.) and the SxFy header. This is mechanical, well-specified, and reusable by every layer.
|
||||
- **HSMS framer.** TCP framing, SELECT/DESELECT/SEPARATE control messages, T-timer constants. Codec layer only — no state machine yet.
|
||||
- **Project skeleton.** Language choice (Rust is the recommendation — memory-safe, good for the long-lived stack, easy FFI later; Python a viable alternative for faster iteration). CI, lint, test harness, fuzz harness for the codec.
|
||||
|
||||
### Acceptance for this layer
|
||||
|
||||
You can round-trip arbitrary SECS-II messages to bytes and back, you can read/write HSMS frames against a loopback socket, and you can load a tiny example transition table and query it.
|
||||
|
||||
## Layer 1 — Spec encoding (Weeks 5–12, ongoing thereafter)
|
||||
|
||||
Goal: encode the E30 GEM behavioral spec as transition tables. This is the real intellectual work and the moat.
|
||||
|
||||
### Approach
|
||||
|
||||
- Start with the **communication state machine** (HSMS): SELECT/DESELECT/SEPARATE handshake, T3/T5/T6/T7/T8 timers, NOT_CONNECTED ↔ CONNECTED ↔ SELECTED.
|
||||
- Then the **control state machine**: the OFFLINE sub-states, ONLINE LOCAL/REMOTE, all transitions with their triggers and rejections (HCACK codes).
|
||||
- Then the **processing/equipment state machine**: IDLE → SETUP → READY → EXECUTING → PAUSE.
|
||||
- Then the **event/report configuration**: S2F33 (define report), S2F35 (link to CEID), S2F37 (enable), and the S6F11 emission rule.
|
||||
- Then **alarm management** (S5F1/S5F3) and **remote commands** (S2F41/S2F42 with the full HCACK enumeration).
|
||||
- Then **spooling**: the SPOOL state machine, queue policy, transmit-on-reconnect ordering.
|
||||
|
||||
For each subsection: write rows for every legal transition *and* every illegal one (with the spec-mandated rejection/ACK code). Cite the spec section in `spec_ref`. Add a `notes` field whenever the spec is ambiguous, so the choice is recorded.
|
||||
|
||||
### Tooling for this layer
|
||||
|
||||
- A **table linter**: every `(machine, state, event)` pair must have either a transition row or an explicit "ignore"/"reject" row. The linter fails CI on missing pairs. This is what guarantees exhaustive coverage.
|
||||
- A **table-to-Markdown renderer**: generate human-readable state tables and diagrams from the data, so reviewing the encoding against the SEMI document is tractable.
|
||||
|
||||
### Acceptance
|
||||
|
||||
The base E30 tables are complete, lint-clean, and human-reviewable against the SEMI document. You haven't built any runtime yet, but you have the asset everything else generates from.
|
||||
|
||||
## Layer 2 — Passive wire analyzer (Weeks 9–16, overlaps Layer 1)
|
||||
|
||||
Goal: ship the first user-facing artifact. The "Wireshark for SECS, but understands the state machines." Zero trust barrier, vendor-neutral, immediately useful.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
pcap file / live capture / log file
|
||||
│
|
||||
▼
|
||||
HSMS framer → SECS-II decoder → message stream
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ passive state reconstructor │
|
||||
│ (runs the transition tables │
|
||||
│ on observed messages, in │
|
||||
│ "observer" mode — never │
|
||||
│ sends anything) │
|
||||
└──────────┬───────────────────────┘
|
||||
▼
|
||||
┌──────────────────────────────────────────┐
|
||||
│ diagnostic engine │
|
||||
│ - reports protocol violations │
|
||||
│ - explains rejections ("S2F41 rejected │
|
||||
│ because control was LOCAL") │
|
||||
│ - explains silences ("CEID 12 fired │
|
||||
│ but report 5 was never enabled") │
|
||||
│ - flags timeout misses │
|
||||
└──────────┬───────────────────────────────┘
|
||||
▼
|
||||
UI (web or TUI):
|
||||
- timeline of messages
|
||||
- live state-machine view
|
||||
- time-travel scrubbing
|
||||
- golden-trace diff
|
||||
```
|
||||
|
||||
### Deliverables
|
||||
|
||||
- **Passive reconstructor.** Consumes a SECS message stream and runs both endpoints' state machines in parallel (one for host, one for equipment), inferring state from observed messages.
|
||||
- **Diagnostic engine.** Knows every legal transition (from the tables) and flags violations with human-readable explanations citing the spec reference.
|
||||
- **UI.** Web UI is the right call (cross-platform, easy to share, screenshot-able for support tickets). Timeline + state panels + scrubber + filterable message inspector.
|
||||
- **Capture options:** read pcap, attach to a live HSMS connection in tap mode (e.g. port mirror or local proxy that forwards and tees), import vendor log files (start with the secsgem Python format and add formats as users request).
|
||||
|
||||
### Acceptance
|
||||
|
||||
An integration engineer can drop a captured session in, see exactly why the bring-up failed, and point a senior at the rendered explanation rather than the raw log. This is the credibility-building artifact — open-source it.
|
||||
|
||||
### Suggested name worth squatting
|
||||
|
||||
Something like `secscope` or `gemtrace`. Pick early; matters for adoption.
|
||||
|
||||
## Layer 3 — Simulator (Weeks 16–24)
|
||||
|
||||
Goal: an active GEM-compliant simulator usable as (a) a virtual equipment for host developers, (b) a virtual host for equipment developers, and (c) the engine that drives Layer 4's test generator.
|
||||
|
||||
### Approach
|
||||
|
||||
The simulator is the transition tables, run in *active* mode rather than observer mode. The same dispatcher/event-queue architecture, but now it originates messages instead of just observing them. Key additions over the analyzer:
|
||||
|
||||
- **Scriptable scenarios.** A small DSL or just YAML for "carrier arrives at LP2, host issues START, after 30s a wafer-complete event fires." Lets users script reproducible test situations.
|
||||
- **Equipment-specific data dictionary.** SVID/ECID/CEID/DVID definitions loaded from a config file per simulated tool. Default to a generic minimal tool; allow users to load richer ones.
|
||||
- **Fault injection.** Drop the link mid-transaction, delay a reply past T3, return malformed messages, send out-of-spec ACK codes. This is what makes the simulator valuable for hardening host implementations.
|
||||
- **Replay mode.** Take a captured session and replay it as either side. Enables "develop your host against last week's tool session."
|
||||
|
||||
### Acceptance
|
||||
|
||||
A host developer can run the simulator locally and develop against it without owning a physical tool, and an equipment developer can point their tool at it as a fake host for bring-up testing.
|
||||
|
||||
## Layer 4 — Conformance test generator (Weeks 20–28)
|
||||
|
||||
Goal: from the tables, mechanically generate the exhaustive conformance test suite (especially the negative cases) and run it against any implementation.
|
||||
|
||||
### How it works
|
||||
|
||||
For each row in the transition table, the generator emits a test case consisting of:
|
||||
|
||||
1. **Setup:** the sequence of messages needed to drive the system into `state`. Computed by graph-search over the table — find a shortest path from the initial state to the target state, using only known transitions.
|
||||
2. **Stimulus:** the `event` (a message to send, or an internal trigger the simulator can fake).
|
||||
3. **Assertions:** the expected `result`, `emitted_messages`, `ack_code`, and resulting state — all read directly from the row.
|
||||
|
||||
The generator runs as a host (or equipment) using the Layer 3 simulator engine, connected over HSMS to the system under test, and produces a structured report: pass/fail per row, with the spec reference cited on every failure.
|
||||
|
||||
### Negative-case coverage
|
||||
|
||||
The win is that the table contains every illegal `(state, event)` pair with its mandated rejection. The generator emits a test for each. This is the coverage humans skip out of tedium — sending S2F41 in every non-REMOTE state and asserting the correct HCACK comes back, for instance.
|
||||
|
||||
### Equipment-specific discovery
|
||||
|
||||
For the parts of the test suite that need tool-specific knowledge (SVID list, CEID list, recipe names), the generator first runs a discovery phase: S1F11 for SVID names, S1F13 for capabilities, S7F19 for recipe names, and so on. The discovered dictionary is merged with the generic tables to produce the full test plan.
|
||||
|
||||
### Acceptance
|
||||
|
||||
Point the generator at an implementation; get a pass/fail report against the full E30 (and later GEM300) behavioral spec, including all negative cases, with spec references on every failure.
|
||||
|
||||
## Layer 5 — GEM300 extensions (Months 6–12)
|
||||
|
||||
Goal: add E87 (carrier management), E90 (substrate tracking), E40 (process jobs), E94 (control jobs), E116 (EPT) as additional table files.
|
||||
|
||||
Because the architecture is additive — each standard is another set of transition rows plus possibly another machine in the dispatcher — adding GEM300 is wiring, not surgery. Sequence:
|
||||
|
||||
1. **E90 first** (substrate tracking) — applies to almost every tool, smallest dependency surface.
|
||||
2. **E87** (carrier management) — only for tools that handle carriers, but the most asked-for after E90.
|
||||
3. **E40/E94** (jobs) — adds the largest new state machines; do them as a pair since E94 references E40.
|
||||
4. **E116** (EPT) — comparatively simple state model, valuable for fab metrics customers.
|
||||
|
||||
Each addition automatically gets analyzer support (new states render in the UI), simulator support (new scenarios scriptable), and conformance tests (new rows → new tests).
|
||||
|
||||
## Layer 6 — Spec revision diff and impact reporter (Month 9+)
|
||||
|
||||
Goal: when a new SEMI revision lands, produce a machine-generated report of exactly what changes for any given implementation.
|
||||
|
||||
### Mechanism
|
||||
|
||||
- Maintain one table file per spec revision (`e30-0307.yaml`, `e30-0712.yaml`, etc.).
|
||||
- A **table diff tool** computes structural diffs: which rows added/removed/modified, which ACK codes changed, which transitions newly legal/illegal.
|
||||
- An **impact reporter** re-runs the Layer 4 conformance generator against a customer's implementation using the new tables and produces a focused report: "these N behaviors must change to remain compliant under revision X; here are the spec references and the exact transitions affected."
|
||||
|
||||
### Honest framing
|
||||
|
||||
This is not "live" in the sense of auto-detecting that SEMI published something — a human still encodes each new revision into the table. The value is converting "your implementation drifted out of compliance, found out at a fab acceptance test" into "here is a precise diff the day you load the new table." That conversion is the product.
|
||||
|
||||
## Layer 7 (optional) — Runtime stack for equipment makers
|
||||
|
||||
Goal: a production-grade SECS/GEM runtime that equipment vendors embed in their tools. This is the high-trust, slow-cycle, sales-heavy artifact — only worth attempting after Layers 2–4 have built credibility and identified the right customer segment.
|
||||
|
||||
The runtime is the same dispatcher/event-queue executing the same tables, but in production mode: persistent spool, real timers, OS integration, supported APIs (Rust/C/C++/Python/.NET bindings via FFI). Skip this layer entirely if the project stays a side project; it's where commercial competition is hardest.
|
||||
|
||||
## Sequencing summary
|
||||
|
||||
| Months | Focus | User-visible artifact |
|
||||
|---|---|---|
|
||||
| 1 | Foundations (codec, framer, table schema) | none |
|
||||
| 2–3 | Encode E30 base tables | none (internal asset) |
|
||||
| 3–4 | Passive analyzer MVP | **Open-source release: SECS analyzer** |
|
||||
| 5–6 | Simulator | simulator alongside analyzer |
|
||||
| 6–7 | Conformance test generator | conformance reports |
|
||||
| 7–9 | GEM300 (E90, E87) | analyzer + tests cover GEM300 |
|
||||
| 9–12 | E40/E94/E116 + spec-revision diff | full GEM300 coverage + impact reports |
|
||||
| 12+ | Optional: commercial runtime | (if pursued) |
|
||||
|
||||
## Risks and how each layer mitigates them
|
||||
|
||||
- **Wrong-table risk** (you encode the spec wrong, confidently emit wrong tests): mitigated by the `spec_ref` requirement and table-to-Markdown review, plus community review once open-sourced.
|
||||
- **Adoption risk** (nobody uses it): mitigated by leading with the analyzer — low trust barrier, immediately useful, no commitment.
|
||||
- **Scope risk** (GEM300 is huge): mitigated by additive architecture and only adding standards customers actually request.
|
||||
- **Incumbent response risk** (a Cimetrix/PEER builds the analyzer themselves): mitigated by being open-source and vendor-neutral, which they structurally won't match.
|
||||
- **Solo-bandwidth risk**: every layer is independently shippable. If life happens after Layer 2, the analyzer is still a real contribution.
|
||||
|
||||
## First concrete next step
|
||||
|
||||
Before any code: pick three integration engineers at equipment vendors (your IMS network has adjacencies) and ask them, unprompted, *"what's the worst part of bringing up a SECS/GEM interface?"* If "blind debugging" and "compliance test maintenance" come back without you naming them, the wedge is validated. Then start Layer 0.
|
||||
Reference in New Issue
Block a user