From f206df763e25e79e074303d4b52483bf26678954 Mon Sep 17 00:00:00 2001 From: Raphael Maenle Date: Tue, 9 Jun 2026 10:58:42 +0200 Subject: [PATCH] docs: customer integration tutorial (INTEGRATION.md) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit End-to-end guide for an equipment vendor integrating the library into a real semiconductor tool: 1. Architecture: what the runtime provides vs what the application contributes — three boundary classes (EquipmentDataModel, Router, hsms::Connection). 2. 30-minute first connection: YAML + minimal main() + run. 3. Wiring real sensors to SVIDs. 4. Plugging the FSMs into the tool: EPT, carriers, substrates, E40 PJ / E94 CJ, alarms, recoverable exceptions. 5. Persistence: enable_persistence(dir) per store, storage budget, replay semantics, current caveats. 6. Monitoring + observability: connection lifecycle hooks, state-change handlers, S9 protocol errors. 7. Recommended deployment layout (/opt/acme-secsgem/...). 8. Integration testing checklist. 9. When to extend the runtime. 10. The honest gap between "this stack runs" and "this is a certified GEM tool". Cross-referenced from COMPLIANCE.md §9 distinction (stack vs tool). Co-Authored-By: Claude Opus 4.7 --- INTEGRATION.md | 473 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 473 insertions(+) create mode 100644 INTEGRATION.md diff --git a/INTEGRATION.md b/INTEGRATION.md new file mode 100644 index 0000000..4b00de3 --- /dev/null +++ b/INTEGRATION.md @@ -0,0 +1,473 @@ +# Integration tutorial + +How a semiconductor equipment vendor takes this library and turns it +into a SECS/GEM-compliant interface on a real tool. + +The library gives you **the runtime stack** — wire codecs, the HSMS +connection state machine, every GEM 300 sub-state-machine, persistent +stores, the message catalog, and a dispatcher. What you bring is +**the application**: knowledge of your tool's real sensors, recipes, +alarms, processing states, and chamber I/O. This guide walks through +how those two halves meet. + +> **Audience.** Firmware / controls engineers integrating +> SECS/GEM on a tool for the first time. Familiarity with SEMI +> E5/E30/E37 helps but isn't required — every spec reference is +> pinned in `COMPLIANCE.md`. + +--- + +## 1. What you get vs. what you build + +``` +┌───────────────────────────────────────────────────────────┐ +│ your equipment application (you write) │ +│ recipe runner • sensor polling • alarm sources • UI hooks│ +├───────────────────────────────────────────────────────────┤ +│ secs-gem runtime stack (this library) │ +│ data model • FSMs • SECS-II codec • HSMS connection │ +│ message catalog • routers • persistence • spool │ +├───────────────────────────────────────────────────────────┤ +│ OS + Asio (provided) + your serial/Ethernet driver │ +└───────────────────────────────────────────────────────────┘ +``` + +The boundary lives at three classes: + +- `gem::EquipmentDataModel` — the data dictionary (SVIDs, ECIDs, + CEIDs, alarms, recipes, jobs, carriers, substrates …). Your + application reads/writes it; the dispatcher serves it on the wire. +- `gem::Router` — maps `(stream, function) → handler`. Wire it once + at startup; messages flow through it. +- `hsms::Connection` (or `secsi::TcpTransport` for SECS-I) — the + byte-level transport. You feed it a TCP socket and a router; it + runs. + +You don't subclass the FSMs. You don't write parsers. You don't +patch the dispatcher. Your code lives in two places: **YAML** +(static configuration) and **callbacks** (dynamic glue). + +--- + +## 2. The 30-minute first connection + +The shortest path from `git clone` to "a host can talk to my tool": + +### 2.1. Describe your tool in YAML + +Copy `data/equipment.yaml`, rename to your tool, and edit: + +```yaml +device: + id: 1 # E37 SESSION-ID + model_name: "ACME-PVD-3000" + software_rev: "1.4.2" + equipment_type: "PVD" # S1F20 EQPTYP + +svids: # status variables (read-only) + - {id: 1, name: ControlState, units: "", type: ASCII, value: ""} + - {id: 100, name: ChamberPressureTorr, units: "Torr", type: F4, value: 0.0} + - {id: 101, name: WaferCounter, units: "wafer", type: U4, value: 0} + +ecids: # equipment constants (host can set) + - {id: 10, name: ChamberSetpointTorr, units: "Torr", type: F4, + value: 1.0e-6, min: "1.0e-9", max: "1.0"} + +ceids: # collection events + - {id: 300, name: ProcessStarted} + - {id: 301, name: ProcessCompleted} + +alarms: + - {id: 1, text: "Chamber pressure out of range", category: 4} + +recipes: + - {id: "RECIPE-A", body: "STEP HEAT 350C 60s\nEND"} + +host_commands: + - {name: START, ack: Accept, emit_ceid: 300} + - {name: STOP, ack: Accept} +``` + +That's the GEM data dictionary. The library will serve every +S1F3/F11, S2F13/F29, S2F33-F38, S5F5, S7F19, S2F41, etc. against +this YAML without any C++ changes. + +### 2.2. Stand it up + +A minimal `main()` looks like `apps/secs_server.cpp`. In your code: + +```cpp +auto model = std::make_shared(); +auto desc = config::load_equipment("/etc/acme/equipment.yaml", *model); +auto sm = std::make_shared( + gem::ControlStateMachine::default_table(), + gem::ControlState::HostOffline); + +asio::io_context io; +Server server(io, {/*port=*/5000, desc.device_id, {}}); + +server.on_accept([&](std::shared_ptr conn) { + auto router = std::make_shared(); + register_default_handlers(*router, model, sm, conn); // your function + conn->set_message_handler([router, conn](const secs2::Message& m) { + return router->dispatch_with_s9( + [&](uint8_t f, const std::array& mhead) { + conn->emit_s9(f, mhead); + }, + [&]() -> std::optional> { + auto* h = conn->current_header(); + return h ? std::optional{h->encode()} : std::nullopt; + }, m); + }); +}); + +server.start(); +io.run(); +``` + +`register_default_handlers` is the only piece of glue you write at +the start. The repo's `apps/secs_server.cpp` is a complete worked +example — copy it verbatim, then customize the YAML to your tool. + +### 2.3. Run it + +```sh +docker compose up server # or your own deployment +# host fires up secsgem-py / wonderware / equipment manager: +# selects, S1F13, S1F1, S1F3 → you're talking GEM. +``` + +That's the floor. From here, every section below adds capability. + +--- + +## 3. Wiring real sensors to SVIDs + +The YAML's `value:` field is the *initial* value. Your application +updates the live value as the tool runs: + +```cpp +// In your sensor-poll thread (running on a separate executor): +double torr = read_baratron(); +model->svids.set_value(/*ChamberPressure=*/100, secs2::Item::f4(float(torr))); +``` + +That's it — the next S1F3 from the host returns the fresh value. + +Two patterns scale well: + +1. **One updater per sensor, fixed cadence.** Each sensor's driver + owns the (vid, set_value) pair. +2. **A single refresh tick.** A periodic timer dumps all polled + values at once (`refresh()` in `apps/secs_server.cpp` does this + for two virtual SVIDs). + +The SECS-II Item shape must match the YAML's `type:`. If the YAML +says `F4` and you call `set_value(100, secs2::Item::ascii("..."))`, +the host will get the string back — the library doesn't enforce a +runtime check. Treat the YAML type as a contract you maintain. + +--- + +## 4. Plugging the FSMs into your tool + +Every GEM 300 sub-state-machine in the library is a behavior model. +You decide *when* state transitions happen by firing events: + +### 4.1. Equipment processing (E116 EPT) + +```cpp +// At startup or whenever the operator clicks "Run": +model->ept.on_event(gem::EptEvent::EnterStandby); +model->ept.on_event(gem::EptEvent::EnterProductive); + +// Auto-emit S6F11(ControlEvent_*) on every transition: +model->ept.set_state_change_handler( + [&](gem::EptState, gem::EptState to, gem::EptEvent, + std::chrono::milliseconds /*dwell*/) { + uint32_t ceid = ept_state_to_ceid(to); // your switch/case + if (!ceid || !model->is_event_enabled(ceid)) return; + conn->send_data(gem::s6f11_event_report( + next_dataid++, ceid, model->compose_reports_for(ceid))); + }); +``` + +Helpers: +- `model->ept.accumulated(state)` — total milliseconds spent in + `state` since startup. Use it to populate E116 SVIDs. +- `model->ept.reset_history()` — call at shift boundary. + +### 4.2. Carriers + load ports (E87) + +When AMHS delivers a carrier: + +```cpp +model->carriers.create("CAR-A1B2", /*port=*/1, /*capacity=*/25); +model->carriers.fire_id_event("CAR-A1B2", gem::CarrierIDEvent::Read); +// ... host sends S3F17(ProceedWithCarrier), the registered handler +// in the Router calls fire_id_event(..., ProceedWithCarrier) and +// CIDS moves NotConfirmed → Confirmed. +``` + +When your slot-map scanner finishes: + +```cpp +auto* c = model->carriers.get("CAR-A1B2"); +for (std::size_t i = 0; i < scan_result.size(); ++i) + c->slots[i].state = scan_result[i]; // 0 empty, 1 occupied +model->carriers.fire_slot_map_event("CAR-A1B2", gem::SlotMapEvent::Read); +``` + +The S3F19/F20 verify handler will compare against this map. + +### 4.3. Substrates (E90) + +For each wafer you start tracking: + +```cpp +model->substrates.create("W-2024-001", "CAR-A1B2", /*slot=*/1); +model->substrates.fire_location_event( + "W-2024-001", gem::SubstrateEvent::Acquire, /*location=*/"ChamberA"); +model->substrates.fire_processing_event( + "W-2024-001", gem::SubstrateProcessingEvent::StartProcessing); +// ... when done: +model->substrates.fire_processing_event( + "W-2024-001", gem::SubstrateProcessingEvent::EndProcessing); +model->substrates.fire_location_event( + "W-2024-001", gem::SubstrateEvent::Release, /*location=*/"OutCarrier"); +``` + +History is tracked per-substrate (`model->substrates.history(id)`) +and can power your downtime / yield reports. + +### 4.4. Process jobs + control jobs (E40 / E94) + +The host creates these via S16F11 / S14F9. Your application drives +their internal transitions as the recipe engine progresses: + +```cpp +// Recipe runner reports setup done: +model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::SetupComplete); +// Operator hits Start (or autorun is on): +model->process_jobs.on_host_command("PJ-1", gem::ProcessJobEvent::Start); +// Recipe completed normally: +model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::ProcessComplete); +``` + +CJ state cascades the same way (E94). + +### 4.5. Alarms (E5 §13) + +```cpp +// Sensor crosses threshold: +model->alarms.set(/*alid=*/1, /*set=*/true); // emits S5F1(ALCD=0x84) +// Later it clears: +model->alarms.set(1, false); // emits S5F1(ALCD=0x04) +``` + +The dispatcher takes care of the wire frame — you just toggle. + +### 4.6. Recoverable exceptions (E5 §9, S5F9–F18) + +For faults where you want a host/equipment recovery dialogue: + +```cpp +model->exceptions.post(/*exid=*/42, "VACUUM", + "lost vacuum in chamber A", + {"PURGE", "RECOVER", "ABORT"}); // emits S5F9 +// Host picks PURGE via S5F13; the registered handler calls +// model->exceptions.on_recover(42, "PURGE"), state moves to Recovering. +// Your purge routine completes: +model->exceptions.fire_internal(42, gem::ExceptionEvent::RecoveryComplete); +// state → Cleared; S5F11 fires; entry removed. +``` + +--- + +## 5. Persistence + +GEM equipment that loses power mid-job can recover gracefully +because every store the library ships supports an opt-in file-backed +journal. Enable per store, at startup, BEFORE the connection comes up: + +```cpp +auto base = std::filesystem::path("/var/lib/acme/secsgem"); +model->spool.enable_persistence(base / "spool"); +model->carriers.enable_persistence(base / "carriers"); +model->load_ports.enable_persistence(base / "loadports"); +model->substrates.enable_persistence(base / "substrates"); +model->process_jobs.enable_persistence(base / "pjobs"); +model->control_jobs.enable_persistence(base / "cjobs"); +model->exceptions.enable_persistence(base / "exceptions"); +``` + +On enable, the store scans the directory, replays records into +in-memory state, and from there keeps the directory in sync on +every create / state-change / remove. Writes use a +`.tmp + rename` pattern so a power loss mid-write can lose at most +the in-flight record (older records remain coherent). + +Storage budget per store, roughly: +- spool: one file per spooled S6F11 (typically tens of bytes each) +- carriers: one file per carrier (~50 bytes + slot count) +- load_ports: one file per LP (~30 bytes) +- substrates: one file per wafer (~80 bytes) +- pjobs: one file per active PJ (~100 bytes), plus `order.idx` +- cjobs: one file per active CJ (~80 bytes) +- exceptions: one file per Posted/Recovering exception + +Even a busy fab tool tops out at a few hundred files in each +directory — well within filesystem caps. Sweep terminal-state +entries (completed PJs, cleared exceptions) periodically if you +care about directory size. + +Caveats currently captured in the persistence tests: +- Substrate **history** is intentionally NOT journaled — only the + *current* state of each axis. Replay starts with an empty + history vector. +- PJ `rcpvars` / `prprocessparams` (the optional E40 `secs2::Item` + trailers) are not journaled in v1; call `set_e40_extras` again on + the application side after restart if you need them. + +--- + +## 6. Monitoring + observability + +### 6.1. Connection lifecycle + +```cpp +conn->set_log_handler([](const std::string& m) { + syslog(LOG_INFO, "hsms: %s", m.c_str()); +}); +conn->set_selected_handler([] { metrics.inc("hsms.selected"); }); +conn->set_closed_handler([](const std::string& r) { + metrics.inc("hsms.closed", {{"reason", r}}); +}); +``` + +### 6.2. State change observers + +Every store / FSM exposes a `set_*_change_handler`. Hook them up +to your metrics / log pipeline: + +```cpp +model->control_jobs.set_state_change_handler( + [](const std::string& cj, gem::ControlJobState f, gem::ControlJobState t, + gem::ControlJobEvent) { + log("CJ " + cj + " " + gem::control_job_state_name(f) + + " → " + gem::control_job_state_name(t)); + }); +``` + +### 6.3. Self-emitted protocol errors + +Look for `S9F*` traffic in your logs. S9F3 / F5 mean the host +asked for something your router doesn't handle; S9F7 means a bad +body arrived; S9F9 means a reply didn't arrive in T3; S9F11 means +a frame exceeded the 16 MiB cap. None of these are normal — they're +real diagnostic events. + +--- + +## 7. Recommended layout for a vendor application + +``` +/opt/acme-secsgem/ +├─ bin/ +│ └─ secsgem-equipment # your built binary +├─ etc/ +│ ├─ equipment.yaml # your tool's dictionary +│ └─ control_state.yaml # your tool-specific state model +├─ var/ +│ ├─ spool/ # populated at runtime +│ ├─ carriers/ +│ ├─ substrates/ +│ ├─ pjobs/ +│ ├─ cjobs/ +│ └─ exceptions/ +└─ share/ + └─ doc/ # COMPLIANCE.md, INTEGRATION.md +``` + +Your application reads `etc/`, writes to `var/`, and never touches +`share/`. YAML edits don't require a rebuild — restart the +process. + +The control-state YAML is your tool's *processing* state machine — +E30 deliberately leaves the concrete states (IDLE / SETUP / READY / +EXECUTING / PAUSE / …) up to the tool builder. Copy +`data/control_state.yaml` as a starting point. + +--- + +## 8. Test the integration + +Don't ship without: + +1. **Round-trip every host-facing message you serve.** The library's + own test suite covers the codecs; you should also drive your + YAML's specific SVIDs / CEIDs / alarms / recipes against the + built-in passive server using the `interop/host_vs_cpp_server.py` + harness as a template. + +2. **Power-loss simulation.** Kill -9 the process mid-job, restart, + confirm the stores replay the correct state. The persistence + tests give you a template; copy and parameterize for your store + directories. + +3. **Multi-hour soak.** Spool fills up if persistence is enabled and + the host link is down — make sure your fab's MES side ack-rate + keeps up. Run a 24h test with the host periodically disconnecting + and watch the journal directory. + +4. **The two-container demo** in this repo gives you a starting + harness — the host emulator (`apps/secs_client.cpp`) drives + ~20 transactions through your server. Adapt it to your message + set. + +--- + +## 9. When to extend the runtime + +The library is open to extension. Common reasons to add code: + +- **A new SECS-II message** the catalog doesn't cover. Edit + `data/messages.yaml`, run the codegen (built into the CMake + pipeline), add a Router handler. No core code change. +- **A new state machine** specific to your tool (e.g. an in-chamber + cooling cycle FSM). Lift the pattern from `ept_state.hpp`: + define your states + events + transition table; let your + application drive it. +- **An additional persistence backend** (DB instead of files). + Mirror the spool `.enable_persistence` pattern — it's about 100 + lines per store. + +If your change is broadly useful, it's worth landing in the library +itself. See `COMPLIANCE.md` for the standards still on the +"explicitly out of scope" list — anything there is a possible +contribution. + +--- + +## 10. Going from "stack" to "certified GEM tool" + +This codebase passes its own conformance harness and cross-validates +against `secsgem-py`, but a real *certified* GEM tool needs more: + +- **Independent third-party validator**. Run a GEM RTS (Reference + Test System) or equivalent against your integration. The library + serves the messages; the validator decides whether your data is + consistent. + +- **Vendor application code**. The runtime cannot, by design, know + what your SVID values *should* be at any given moment. That's + your domain knowledge plugged into the data model and FSMs. + +- **Documentation**. E30 §6.10 (Documentation capability) requires + you to publish what you implement. `COMPLIANCE.md` in this repo + is the template — fork it, prune to your actual coverage, ship + it with your tool. + +- **Operations**: monitoring dashboards, alarm escalation, log + retention — the standard SRE concerns, no different from any + other piece of fab software.