# SECS/GEM Spec-as-Data Project — Implementation Plan A layered plan for building a SECS/GEM toolchain whose core asset is the SEMI behavioral spec encoded as machine-readable transition tables. From that one artifact you derive a runtime stack, a passive wire analyzer, a conformance test generator, and a simulator. The plan is sequenced so each layer is independently useful and ships value on its own — you can stop after Layer 2 and still have a shipped product. ## Guiding principles A few discipline rules that hold across every layer: - **One source of truth.** The spec tables are the only place behavioral rules live. Runtime, tests, analyzer, simulator, and docs all *read* from them; none of them re-encode the spec. - **Data, not code.** Tables are versioned data files (YAML/TOML/JSON), not classes. Adding GEM300 standards or new spec revisions is additive — drop in another table file. - **One mutation point.** Every state change in every runtime/simulator goes through one validated apply-step that emits the collection event as a side effect. No alternate paths. - **Exhaustive coverage, including rejections.** Every `(machine, state, event)` pair has an explicit entry. No defaults that silently swallow. - **Trust ladder.** Ship the lowest-trust-barrier artifact first (the passive analyzer touches no wafer), build credibility, then ascend to higher-stakes artifacts (simulator, runtime). - **Validate the pain before building each layer.** Talk to integration engineers between layers. If they don't unprompted complain about what the next layer solves, don't build it. ## Layer 0 — Foundations (Weeks 1–4) Goal: establish the data model and conventions everything else depends on. No user-facing artifact yet. ### Deliverables - **Transition table schema.** A formal schema (JSON Schema or similar) for rows of the form: ``` machine, state, event, guard, result, new_state, emitted_messages, emitted_ceids, ack_code, notes, spec_ref ``` Include `spec_ref` (e.g. `E30 §6.5.2`) on every row so the table is auditable line-by-line against the standard. - **SECS-II message codec.** The byte-level encoder/decoder for SECS-II data items (L, A, B, U1/U2/U4, I1/I2/I4, F4/F8, etc.) and the SxFy header. This is mechanical, well-specified, and reusable by every layer. - **HSMS framer.** TCP framing, SELECT/DESELECT/SEPARATE control messages, T-timer constants. Codec layer only — no state machine yet. - **Project skeleton.** Language choice (Rust is the recommendation — memory-safe, good for the long-lived stack, easy FFI later; Python a viable alternative for faster iteration). CI, lint, test harness, fuzz harness for the codec. ### Acceptance for this layer You can round-trip arbitrary SECS-II messages to bytes and back, you can read/write HSMS frames against a loopback socket, and you can load a tiny example transition table and query it. ## Layer 1 — Spec encoding (Weeks 5–12, ongoing thereafter) Goal: encode the E30 GEM behavioral spec as transition tables. This is the real intellectual work and the moat. ### Approach - Start with the **communication state machine** (HSMS): SELECT/DESELECT/SEPARATE handshake, T3/T5/T6/T7/T8 timers, NOT_CONNECTED ↔ CONNECTED ↔ SELECTED. - Then the **control state machine**: the OFFLINE sub-states, ONLINE LOCAL/REMOTE, all transitions with their triggers and rejections (HCACK codes). - Then the **processing/equipment state machine**: IDLE → SETUP → READY → EXECUTING → PAUSE. - Then the **event/report configuration**: S2F33 (define report), S2F35 (link to CEID), S2F37 (enable), and the S6F11 emission rule. - Then **alarm management** (S5F1/S5F3) and **remote commands** (S2F41/S2F42 with the full HCACK enumeration). - Then **spooling**: the SPOOL state machine, queue policy, transmit-on-reconnect ordering. For each subsection: write rows for every legal transition *and* every illegal one (with the spec-mandated rejection/ACK code). Cite the spec section in `spec_ref`. Add a `notes` field whenever the spec is ambiguous, so the choice is recorded. ### Tooling for this layer - A **table linter**: every `(machine, state, event)` pair must have either a transition row or an explicit "ignore"/"reject" row. The linter fails CI on missing pairs. This is what guarantees exhaustive coverage. - A **table-to-Markdown renderer**: generate human-readable state tables and diagrams from the data, so reviewing the encoding against the SEMI document is tractable. ### Acceptance The base E30 tables are complete, lint-clean, and human-reviewable against the SEMI document. You haven't built any runtime yet, but you have the asset everything else generates from. ## Layer 2 — Passive wire analyzer (Weeks 9–16, overlaps Layer 1) Goal: ship the first user-facing artifact. The "Wireshark for SECS, but understands the state machines." Zero trust barrier, vendor-neutral, immediately useful. ### Architecture ``` pcap file / live capture / log file │ ▼ HSMS framer → SECS-II decoder → message stream │ ▼ ┌──────────────────────────────────┐ │ passive state reconstructor │ │ (runs the transition tables │ │ on observed messages, in │ │ "observer" mode — never │ │ sends anything) │ └──────────┬───────────────────────┘ ▼ ┌──────────────────────────────────────────┐ │ diagnostic engine │ │ - reports protocol violations │ │ - explains rejections ("S2F41 rejected │ │ because control was LOCAL") │ │ - explains silences ("CEID 12 fired │ │ but report 5 was never enabled") │ │ - flags timeout misses │ └──────────┬───────────────────────────────┘ ▼ UI (web or TUI): - timeline of messages - live state-machine view - time-travel scrubbing - golden-trace diff ``` ### Deliverables - **Passive reconstructor.** Consumes a SECS message stream and runs both endpoints' state machines in parallel (one for host, one for equipment), inferring state from observed messages. - **Diagnostic engine.** Knows every legal transition (from the tables) and flags violations with human-readable explanations citing the spec reference. - **UI.** Web UI is the right call (cross-platform, easy to share, screenshot-able for support tickets). Timeline + state panels + scrubber + filterable message inspector. - **Capture options:** read pcap, attach to a live HSMS connection in tap mode (e.g. port mirror or local proxy that forwards and tees), import vendor log files (start with the secsgem Python format and add formats as users request). ### Acceptance An integration engineer can drop a captured session in, see exactly why the bring-up failed, and point a senior at the rendered explanation rather than the raw log. This is the credibility-building artifact — open-source it. ### Suggested name worth squatting Something like `secscope` or `gemtrace`. Pick early; matters for adoption. ## Layer 3 — Simulator (Weeks 16–24) Goal: an active GEM-compliant simulator usable as (a) a virtual equipment for host developers, (b) a virtual host for equipment developers, and (c) the engine that drives Layer 4's test generator. ### Approach The simulator is the transition tables, run in *active* mode rather than observer mode. The same dispatcher/event-queue architecture, but now it originates messages instead of just observing them. Key additions over the analyzer: - **Scriptable scenarios.** A small DSL or just YAML for "carrier arrives at LP2, host issues START, after 30s a wafer-complete event fires." Lets users script reproducible test situations. - **Equipment-specific data dictionary.** SVID/ECID/CEID/DVID definitions loaded from a config file per simulated tool. Default to a generic minimal tool; allow users to load richer ones. - **Fault injection.** Drop the link mid-transaction, delay a reply past T3, return malformed messages, send out-of-spec ACK codes. This is what makes the simulator valuable for hardening host implementations. - **Replay mode.** Take a captured session and replay it as either side. Enables "develop your host against last week's tool session." ### Acceptance A host developer can run the simulator locally and develop against it without owning a physical tool, and an equipment developer can point their tool at it as a fake host for bring-up testing. ## Layer 4 — Conformance test generator (Weeks 20–28) Goal: from the tables, mechanically generate the exhaustive conformance test suite (especially the negative cases) and run it against any implementation. ### How it works For each row in the transition table, the generator emits a test case consisting of: 1. **Setup:** the sequence of messages needed to drive the system into `state`. Computed by graph-search over the table — find a shortest path from the initial state to the target state, using only known transitions. 2. **Stimulus:** the `event` (a message to send, or an internal trigger the simulator can fake). 3. **Assertions:** the expected `result`, `emitted_messages`, `ack_code`, and resulting state — all read directly from the row. The generator runs as a host (or equipment) using the Layer 3 simulator engine, connected over HSMS to the system under test, and produces a structured report: pass/fail per row, with the spec reference cited on every failure. ### Negative-case coverage The win is that the table contains every illegal `(state, event)` pair with its mandated rejection. The generator emits a test for each. This is the coverage humans skip out of tedium — sending S2F41 in every non-REMOTE state and asserting the correct HCACK comes back, for instance. ### Equipment-specific discovery For the parts of the test suite that need tool-specific knowledge (SVID list, CEID list, recipe names), the generator first runs a discovery phase: S1F11 for SVID names, S1F13 for capabilities, S7F19 for recipe names, and so on. The discovered dictionary is merged with the generic tables to produce the full test plan. ### Acceptance Point the generator at an implementation; get a pass/fail report against the full E30 (and later GEM300) behavioral spec, including all negative cases, with spec references on every failure. ## Layer 5 — GEM300 extensions (Months 6–12) Goal: add E87 (carrier management), E90 (substrate tracking), E40 (process jobs), E94 (control jobs), E116 (EPT) as additional table files. Because the architecture is additive — each standard is another set of transition rows plus possibly another machine in the dispatcher — adding GEM300 is wiring, not surgery. Sequence: 1. **E90 first** (substrate tracking) — applies to almost every tool, smallest dependency surface. 2. **E87** (carrier management) — only for tools that handle carriers, but the most asked-for after E90. 3. **E40/E94** (jobs) — adds the largest new state machines; do them as a pair since E94 references E40. 4. **E116** (EPT) — comparatively simple state model, valuable for fab metrics customers. Each addition automatically gets analyzer support (new states render in the UI), simulator support (new scenarios scriptable), and conformance tests (new rows → new tests). ## Layer 6 — Spec revision diff and impact reporter (Month 9+) Goal: when a new SEMI revision lands, produce a machine-generated report of exactly what changes for any given implementation. ### Mechanism - Maintain one table file per spec revision (`e30-0307.yaml`, `e30-0712.yaml`, etc.). - A **table diff tool** computes structural diffs: which rows added/removed/modified, which ACK codes changed, which transitions newly legal/illegal. - An **impact reporter** re-runs the Layer 4 conformance generator against a customer's implementation using the new tables and produces a focused report: "these N behaviors must change to remain compliant under revision X; here are the spec references and the exact transitions affected." ### Honest framing This is not "live" in the sense of auto-detecting that SEMI published something — a human still encodes each new revision into the table. The value is converting "your implementation drifted out of compliance, found out at a fab acceptance test" into "here is a precise diff the day you load the new table." That conversion is the product. ## Layer 7 (optional) — Runtime stack for equipment makers Goal: a production-grade SECS/GEM runtime that equipment vendors embed in their tools. This is the high-trust, slow-cycle, sales-heavy artifact — only worth attempting after Layers 2–4 have built credibility and identified the right customer segment. The runtime is the same dispatcher/event-queue executing the same tables, but in production mode: persistent spool, real timers, OS integration, supported APIs (Rust/C/C++/Python/.NET bindings via FFI). Skip this layer entirely if the project stays a side project; it's where commercial competition is hardest. ## Sequencing summary | Months | Focus | User-visible artifact | |---|---|---| | 1 | Foundations (codec, framer, table schema) | none | | 2–3 | Encode E30 base tables | none (internal asset) | | 3–4 | Passive analyzer MVP | **Open-source release: SECS analyzer** | | 5–6 | Simulator | simulator alongside analyzer | | 6–7 | Conformance test generator | conformance reports | | 7–9 | GEM300 (E90, E87) | analyzer + tests cover GEM300 | | 9–12 | E40/E94/E116 + spec-revision diff | full GEM300 coverage + impact reports | | 12+ | Optional: commercial runtime | (if pursued) | ## Risks and how each layer mitigates them - **Wrong-table risk** (you encode the spec wrong, confidently emit wrong tests): mitigated by the `spec_ref` requirement and table-to-Markdown review, plus community review once open-sourced. - **Adoption risk** (nobody uses it): mitigated by leading with the analyzer — low trust barrier, immediately useful, no commitment. - **Scope risk** (GEM300 is huge): mitigated by additive architecture and only adding standards customers actually request. - **Incumbent response risk** (a Cimetrix/PEER builds the analyzer themselves): mitigated by being open-source and vendor-neutral, which they structurally won't match. - **Solo-bandwidth risk**: every layer is independently shippable. If life happens after Layer 2, the analyzer is still a real contribution. ## First concrete next step Before any code: pick three integration engineers at equipment vendors (your IMS network has adjacencies) and ask them, unprompted, *"what's the worst part of bringing up a SECS/GEM interface?"* If "blind debugging" and "compliance test maintenance" come back without you naming them, the wedge is validated. Then start Layer 0.