# Integration tutorial How a semiconductor equipment vendor takes this library and turns it into a SECS/GEM-compliant interface on a real tool. The library gives you **the runtime stack** — wire codecs, the HSMS connection state machine, every GEM 300 sub-state-machine, persistent stores, the message catalog, and a dispatcher. What you bring is **the application**: knowledge of your tool's real sensors, recipes, alarms, processing states, and chamber I/O. This guide walks through how those two halves meet. > **Audience.** Firmware / controls engineers integrating > SECS/GEM on a tool for the first time. Familiarity with SEMI > E5/E30/E37 helps but isn't required — every spec reference is > pinned in `COMPLIANCE.md`. --- ## 1. What you get vs. what you build ``` ┌───────────────────────────────────────────────────────────┐ │ your equipment application (you write) │ │ recipe runner • sensor polling • alarm sources • UI hooks│ ├───────────────────────────────────────────────────────────┤ │ secs-gem runtime stack (this library) │ │ data model • FSMs • SECS-II codec • HSMS connection │ │ message catalog • routers • persistence • spool │ ├───────────────────────────────────────────────────────────┤ │ OS + Asio (provided) + your serial/Ethernet driver │ └───────────────────────────────────────────────────────────┘ ``` The boundary lives at three classes: - `gem::EquipmentDataModel` — the data dictionary (SVIDs, ECIDs, CEIDs, alarms, recipes, jobs, carriers, substrates …). Your application reads/writes it; the dispatcher serves it on the wire. - `gem::Router` — maps `(stream, function) → handler`. Wire it once at startup; messages flow through it. - `hsms::Connection` (or `secsi::TcpTransport` for SECS-I) — the byte-level transport. You feed it a TCP socket and a router; it runs. You don't subclass the FSMs. You don't write parsers. You don't patch the dispatcher. Your code lives in two places: **YAML** (static configuration) and **callbacks** (dynamic glue). --- ## 2. The 30-minute first connection The shortest path from `git clone` to "a host can talk to my tool": ### 2.1. Describe your tool in YAML Copy `data/equipment.yaml`, rename to your tool, and edit: ```yaml device: id: 1 # E37 SESSION-ID model_name: "ACME-PVD-3000" software_rev: "1.4.2" equipment_type: "PVD" # S1F20 EQPTYP svids: # status variables (read-only) - {id: 1, name: ControlState, units: "", type: ASCII, value: ""} - {id: 100, name: ChamberPressureTorr, units: "Torr", type: F4, value: 0.0} - {id: 101, name: WaferCounter, units: "wafer", type: U4, value: 0} ecids: # equipment constants (host can set) - {id: 10, name: ChamberSetpointTorr, units: "Torr", type: F4, value: 1.0e-6, min: "1.0e-9", max: "1.0"} ceids: # collection events - {id: 300, name: ProcessStarted} - {id: 301, name: ProcessCompleted} alarms: - {id: 1, text: "Chamber pressure out of range", category: 4} recipes: - {id: "RECIPE-A", body: "STEP HEAT 350C 60s\nEND"} host_commands: - {name: START, ack: Accept, emit_ceid: 300} - {name: STOP, ack: Accept} ``` That's the GEM data dictionary. The library will serve every S1F3/F11, S2F13/F29, S2F33-F38, S5F5, S7F19, S2F41, etc. against this YAML without any C++ changes. ### 2.2. Stand it up A minimal `main()` looks like `apps/secs_server.cpp`. In your code: ```cpp auto model = std::make_shared(); auto desc = config::load_equipment("/etc/acme/equipment.yaml", *model); auto sm = std::make_shared( gem::ControlStateMachine::default_table(), gem::ControlState::HostOffline); asio::io_context io; Server server(io, {/*port=*/5000, desc.device_id, {}}); server.on_accept([&](std::shared_ptr conn) { auto router = std::make_shared(); register_default_handlers(*router, model, sm, conn); // your function conn->set_message_handler([router, conn](const secs2::Message& m) { return router->dispatch_with_s9( [&](uint8_t f, const std::array& mhead) { conn->emit_s9(f, mhead); }, [&]() -> std::optional> { auto* h = conn->current_header(); return h ? std::optional{h->encode()} : std::nullopt; }, m); }); }); server.start(); io.run(); ``` `register_default_handlers` is the only piece of glue you write at the start. The repo's `apps/secs_server.cpp` is a complete worked example — copy it verbatim, then customize the YAML to your tool. ### 2.3. Run it ```sh docker compose up server # or your own deployment # host fires up secsgem-py / wonderware / equipment manager: # selects, S1F13, S1F1, S1F3 → you're talking GEM. ``` That's the floor. From here, every section below adds capability. --- ## 3. Wiring real sensors to SVIDs The YAML's `value:` field is the *initial* value. Your application updates the live value as the tool runs: ```cpp // In your sensor-poll thread (running on a separate executor): double torr = read_baratron(); model->svids.set_value(/*ChamberPressure=*/100, secs2::Item::f4(float(torr))); ``` That's it — the next S1F3 from the host returns the fresh value. Two patterns scale well: 1. **One updater per sensor, fixed cadence.** Each sensor's driver owns the (vid, set_value) pair. 2. **A single refresh tick.** A periodic timer dumps all polled values at once (`refresh()` in `apps/secs_server.cpp` does this for two virtual SVIDs). The SECS-II Item shape must match the YAML's `type:`. If the YAML says `F4` and you call `set_value(100, secs2::Item::ascii("..."))`, the host will get the string back — the library doesn't enforce a runtime check. Treat the YAML type as a contract you maintain. --- ## 4. Plugging the FSMs into your tool Every GEM 300 sub-state-machine in the library is a behavior model. You decide *when* state transitions happen by firing events: ### 4.1. Equipment processing (E116 EPT) ```cpp // At startup or whenever the operator clicks "Run": model->ept.on_event(gem::EptEvent::EnterStandby); model->ept.on_event(gem::EptEvent::EnterProductive); // Auto-emit S6F11(ControlEvent_*) on every transition: model->ept.set_state_change_handler( [&](gem::EptState, gem::EptState to, gem::EptEvent, std::chrono::milliseconds /*dwell*/) { uint32_t ceid = ept_state_to_ceid(to); // your switch/case if (!ceid || !model->is_event_enabled(ceid)) return; conn->send_data(gem::s6f11_event_report( next_dataid++, ceid, model->compose_reports_for(ceid))); }); ``` Helpers: - `model->ept.accumulated(state)` — total milliseconds spent in `state` since startup. Use it to populate E116 SVIDs. - `model->ept.reset_history()` — call at shift boundary. ### 4.2. Carriers + load ports (E87) When AMHS delivers a carrier: ```cpp model->carriers.create("CAR-A1B2", /*port=*/1, /*capacity=*/25); model->carriers.fire_id_event("CAR-A1B2", gem::CarrierIDEvent::Read); // ... host sends S3F17(ProceedWithCarrier), the registered handler // in the Router calls fire_id_event(..., ProceedWithCarrier) and // CIDS moves NotConfirmed → Confirmed. ``` When your slot-map scanner finishes: ```cpp auto* c = model->carriers.get("CAR-A1B2"); for (std::size_t i = 0; i < scan_result.size(); ++i) c->slots[i].state = scan_result[i]; // 0 empty, 1 occupied model->carriers.fire_slot_map_event("CAR-A1B2", gem::SlotMapEvent::Read); ``` The S3F19/F20 verify handler will compare against this map. ### 4.3. Substrates (E90) For each wafer you start tracking: ```cpp model->substrates.create("W-2024-001", "CAR-A1B2", /*slot=*/1); model->substrates.fire_location_event( "W-2024-001", gem::SubstrateEvent::Acquire, /*location=*/"ChamberA"); model->substrates.fire_processing_event( "W-2024-001", gem::SubstrateProcessingEvent::StartProcessing); // ... when done: model->substrates.fire_processing_event( "W-2024-001", gem::SubstrateProcessingEvent::EndProcessing); model->substrates.fire_location_event( "W-2024-001", gem::SubstrateEvent::Release, /*location=*/"OutCarrier"); ``` History is tracked per-substrate (`model->substrates.history(id)`) and can power your downtime / yield reports. ### 4.4. Process jobs + control jobs (E40 / E94) The host creates these via S16F11 / S14F9. Your application drives their internal transitions as the recipe engine progresses: ```cpp // Recipe runner reports setup done: model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::SetupComplete); // Operator hits Start (or autorun is on): model->process_jobs.on_host_command("PJ-1", gem::ProcessJobEvent::Start); // Recipe completed normally: model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::ProcessComplete); ``` CJ state cascades the same way (E94). ### 4.5. Alarms (E5 §13) ```cpp // Sensor crosses threshold: model->alarms.set(/*alid=*/1, /*set=*/true); // emits S5F1(ALCD=0x84) // Later it clears: model->alarms.set(1, false); // emits S5F1(ALCD=0x04) ``` The dispatcher takes care of the wire frame — you just toggle. ### 4.6. E84 parallel I/O handoff (AMHS) For each load port that talks to the AMHS robot: ```cpp #include "secsgem/gem/e84_asio_timers.hpp" auto* fsm = model->e84_ports.get(/*port_id=*/1); if (!fsm) { model->e84_ports.create(1); fsm = model->e84_ports.get(1); } // SEMI E84 §6 handshake timers. Defaults below are spec-typical; tune // per port. TA1=AMHS waits for L_REQ/U_REQ after VALID; TA2=equipment // waits for BUSY after port is ready; TA3=BUSY phase budget. fsm->set_timeouts({std::chrono::seconds(2), std::chrono::seconds(2), std::chrono::seconds(60)}); // Wire arm/cancel into asio so the FSM polices the real wall clock. auto driver = std::make_shared(io.get_executor(), *fsm); driver->attach(); // Keep `driver` alive for the lifetime of the FSM (e.g. as a member // of your per-port object). // Optional: log handoff faults. fsm->set_fault_handler([port_id = 1](gem::E84Fault reason) { log("E84 port " + std::to_string(port_id) + " fault: " + gem::e84_fault_name(reason)); }); // Now feed signal changes from your I/O bridge. On a real AMHS the // bridge polls or interrupts on the parallel-I/O lines: model->e84_ports.on_signal_change(1, gem::E84Signal::CS_0, true); model->e84_ports.on_signal_change(1, gem::E84Signal::VALID, true); // equipment side asserts when port is physically ready: model->e84_ports.on_signal_change(1, gem::E84Signal::L_REQ, true); // ... AMHS continues with BUSY / COMPT. ``` If TA1, TA2, or TA3 expires the FSM transitions to `HandoffFault` and the fault handler fires with the precise `E84Fault` reason. Your application is then responsible for whatever the tool's fault policy is (typically: assert your local ES line and raise an alarm). ### 4.7. Recoverable exceptions (E5 §9, S5F9–F18) For faults where you want a host/equipment recovery dialogue: ```cpp model->exceptions.post(/*exid=*/42, "VACUUM", "lost vacuum in chamber A", {"PURGE", "RECOVER", "ABORT"}); // emits S5F9 // Host picks PURGE via S5F13; the registered handler calls // model->exceptions.on_recover(42, "PURGE"), state moves to Recovering. // Your purge routine completes: model->exceptions.fire_internal(42, gem::ExceptionEvent::RecoveryComplete); // state → Cleared; S5F11 fires; entry removed. ``` --- ## 5. Persistence GEM equipment that loses power mid-job can recover gracefully because every store the library ships supports an opt-in file-backed journal. Enable per store, at startup, BEFORE the connection comes up: ```cpp auto base = std::filesystem::path("/var/lib/acme/secsgem"); model->spool.enable_persistence(base / "spool"); model->carriers.enable_persistence(base / "carriers"); model->load_ports.enable_persistence(base / "loadports"); model->substrates.enable_persistence(base / "substrates"); model->process_jobs.enable_persistence(base / "pjobs"); model->control_jobs.enable_persistence(base / "cjobs"); model->exceptions.enable_persistence(base / "exceptions"); ``` On enable, the store scans the directory, replays records into in-memory state, and from there keeps the directory in sync on every create / state-change / remove. Writes use a `.tmp + rename` pattern so a power loss mid-write can lose at most the in-flight record (older records remain coherent). Storage budget per store, roughly: - spool: one file per spooled S6F11 (typically tens of bytes each) - carriers: one file per carrier (~50 bytes + slot count) - load_ports: one file per LP (~30 bytes) - substrates: one file per wafer (~80 bytes) - pjobs: one file per active PJ (~100 bytes), plus `order.idx` - cjobs: one file per active CJ (~80 bytes) - exceptions: one file per Posted/Recovering exception Even a busy fab tool tops out at a few hundred files in each directory — well within filesystem caps. Sweep terminal-state entries (completed PJs, cleared exceptions) periodically if you care about directory size. Caveats currently captured in the persistence tests: - Substrate **history** is intentionally NOT journaled — only the *current* state of each axis. Replay starts with an empty history vector. - PJ `rcpvars` / `prprocessparams` (the optional E40 `secs2::Item` trailers) are not journaled in v1; call `set_e40_extras` again on the application side after restart if you need them. --- ## 6. Monitoring + observability ### 6.1. Connection lifecycle ```cpp conn->set_log_handler([](const std::string& m) { syslog(LOG_INFO, "hsms: %s", m.c_str()); }); conn->set_selected_handler([] { metrics.inc("hsms.selected"); }); conn->set_closed_handler([](const std::string& r) { metrics.inc("hsms.closed", {{"reason", r}}); }); ``` ### 6.2. State change observers Every store / FSM exposes a `set_*_change_handler`. Hook them up to your metrics / log pipeline: ```cpp model->control_jobs.set_state_change_handler( [](const std::string& cj, gem::ControlJobState f, gem::ControlJobState t, gem::ControlJobEvent) { log("CJ " + cj + " " + gem::control_job_state_name(f) + " → " + gem::control_job_state_name(t)); }); ``` ### 6.3. Self-emitted protocol errors Look for `S9F*` traffic in your logs. S9F3 / F5 mean the host asked for something your router doesn't handle; S9F7 means a bad body arrived; S9F9 means a reply didn't arrive in T3; S9F11 means a frame exceeded the 16 MiB cap. None of these are normal — they're real diagnostic events. --- ## 7. Recommended layout for a vendor application ``` /opt/acme-secsgem/ ├─ bin/ │ └─ secsgem-equipment # your built binary ├─ etc/ │ ├─ equipment.yaml # your tool's dictionary │ └─ control_state.yaml # your tool-specific state model ├─ var/ │ ├─ spool/ # populated at runtime │ ├─ carriers/ │ ├─ substrates/ │ ├─ pjobs/ │ ├─ cjobs/ │ └─ exceptions/ └─ share/ └─ doc/ # COMPLIANCE.md, INTEGRATION.md ``` Your application reads `etc/`, writes to `var/`, and never touches `share/`. YAML edits don't require a rebuild — restart the process. The control-state YAML is your tool's *processing* state machine — E30 deliberately leaves the concrete states (IDLE / SETUP / READY / EXECUTING / PAUSE / …) up to the tool builder. Copy `data/control_state.yaml` as a starting point. --- ## 8. Test the integration Don't ship without: 1. **Round-trip every host-facing message you serve.** The library's own test suite covers the codecs; you should also drive your YAML's specific SVIDs / CEIDs / alarms / recipes against the built-in passive server using the `interop/host_vs_cpp_server.py` harness as a template. 2. **Power-loss simulation.** Kill -9 the process mid-job, restart, confirm the stores replay the correct state. The persistence tests give you a template; copy and parameterize for your store directories. 3. **Multi-hour soak.** Spool fills up if persistence is enabled and the host link is down — make sure your fab's MES side ack-rate keeps up. Run a 24h test with the host periodically disconnecting and watch the journal directory. 4. **The two-container demo** in this repo gives you a starting harness — the host emulator (`apps/secs_client.cpp`) drives ~20 transactions through your server. Adapt it to your message set. --- ## 9. When to extend the runtime The library is open to extension. Common reasons to add code: - **A new SECS-II message** the catalog doesn't cover. Edit `data/messages.yaml`, run the codegen (built into the CMake pipeline), add a Router handler. No core code change. - **A new state machine** specific to your tool (e.g. an in-chamber cooling cycle FSM). Lift the pattern from `ept_state.hpp`: define your states + events + transition table; let your application drive it. - **An additional persistence backend** (DB instead of files). Mirror the spool `.enable_persistence` pattern — it's about 100 lines per store. If your change is broadly useful, it's worth landing in the library itself. See `COMPLIANCE.md` for the standards still on the "explicitly out of scope" list — anything there is a possible contribution. --- ## 10. Going from "stack" to "certified GEM tool" This codebase passes its own conformance harness and cross-validates against `secsgem-py`, but a real *certified* GEM tool needs more: - **Independent third-party validator**. Run a GEM RTS (Reference Test System) or equivalent against your integration. The library serves the messages; the validator decides whether your data is consistent. - **Vendor application code**. The runtime cannot, by design, know what your SVID values *should* be at any given moment. That's your domain knowledge plugged into the data model and FSMs. - **Documentation**. E30 §6.10 (Documentation capability) requires you to publish what you implement. `COMPLIANCE.md` in this repo is the template — fork it, prune to your actual coverage, ship it with your tool. - **Operations**: monitoring dashboards, alarm escalation, log retention — the standard SRE concerns, no different from any other piece of fab software.