secs-gem/INTEGRATION.md

Integration tutorial

How a semiconductor equipment vendor takes this library and turns it into a SECS/GEM-compliant interface on a real tool.

The library gives you the runtime stack — wire codecs, the HSMS connection state machine, every GEM 300 sub-state-machine, persistent stores, the message catalog, and a dispatcher. What you bring is the application: knowledge of your tool's real sensors, recipes, alarms, processing states, and chamber I/O. This guide walks through how those two halves meet.

Audience. Firmware / controls engineers integrating SECS/GEM on a tool for the first time. Familiarity with SEMI E5/E30/E37 helps but isn't required — every spec reference is pinned in COMPLIANCE.md.

---

1. What you get vs. what you build

┌───────────────────────────────────────────────────────────┐
│         your equipment application (you write)            │
│  recipe runner • sensor polling • alarm sources • UI hooks│
├───────────────────────────────────────────────────────────┤
│            secs-gem runtime stack (this library)          │
│  data model • FSMs • SECS-II codec • HSMS connection      │
│  message catalog • routers • persistence • spool          │
├───────────────────────────────────────────────────────────┤
│      OS + Asio (provided) + your serial/Ethernet driver   │
└───────────────────────────────────────────────────────────┘

The boundary lives at three classes:

• gem::EquipmentDataModel — the data dictionary (SVIDs, ECIDs, CEIDs, alarms, recipes, jobs, carriers, substrates …). Your application reads/writes it; the dispatcher serves it on the wire.
• gem::Router — maps (stream, function) → handler. Wire it once at startup; messages flow through it.
• hsms::Connection (or secsi::TcpTransport for SECS-I) — the byte-level transport. You feed it a TCP socket and a router; it runs.

You don't subclass the FSMs. You don't write parsers. You don't patch the dispatcher. Your code lives in two places: YAML (static configuration) and callbacks (dynamic glue).

---

2. The 30-minute first connection

The shortest path from git clone to "a host can talk to my tool":

2.1. Describe your tool in YAML

Copy data/equipment.yaml, rename to your tool, and edit:

device:
  id: 1                            # E37 SESSION-ID
  model_name: "ACME-PVD-3000"
  software_rev: "1.4.2"
  equipment_type: "PVD"            # S1F20 EQPTYP

svids:                              # status variables (read-only)
  - {id: 1, name: ControlState,  units: "",      type: ASCII,   value: ""}
  - {id: 100, name: ChamberPressureTorr, units: "Torr", type: F4, value: 0.0}
  - {id: 101, name: WaferCounter, units: "wafer", type: U4,      value: 0}

ecids:                              # equipment constants (host can set)
  - {id: 10, name: ChamberSetpointTorr, units: "Torr", type: F4,
     value: 1.0e-6, min: "1.0e-9", max: "1.0"}

ceids:                              # collection events
  - {id: 300, name: ProcessStarted}
  - {id: 301, name: ProcessCompleted}

alarms:
  - {id: 1, text: "Chamber pressure out of range", category: 4}

recipes:
  - {id: "RECIPE-A", body: "STEP HEAT 350C 60s\nEND"}

host_commands:
  - {name: START, ack: Accept, emit_ceid: 300}
  - {name: STOP,  ack: Accept}

That's the GEM data dictionary. The library will serve every S1F3/F11, S2F13/F29, S2F33-F38, S5F5, S7F19, S2F41, etc. against this YAML without any C++ changes.

2.2. Stand it up

A minimal main() looks like apps/secs_server.cpp. In your code:

auto model = std::make_shared<gem::EquipmentDataModel>();
auto desc  = config::load_equipment("/etc/acme/equipment.yaml", *model);
auto sm    = std::make_shared<gem::ControlStateMachine>(
                 gem::ControlStateMachine::default_table(),
                 gem::ControlState::HostOffline);

asio::io_context io;
Server server(io, {/*port=*/5000, desc.device_id, {}});

server.on_accept([&](std::shared_ptr<hsms::Connection> conn) {
    auto router = std::make_shared<gem::Router>();
    register_default_handlers(*router, model, sm, conn);  // your function
    conn->set_message_handler([router, conn](const secs2::Message& m) {
        return router->dispatch_with_s9(
            [&](uint8_t f, const std::array<uint8_t, 10>& mhead) {
                conn->emit_s9(f, mhead);
            },
            [&]() -> std::optional<std::array<uint8_t, 10>> {
                auto* h = conn->current_header();
                return h ? std::optional{h->encode()} : std::nullopt;
            }, m);
    });
});

server.start();
io.run();

register_default_handlers is the only piece of glue you write at the start. The repo's apps/secs_server.cpp is a complete worked example — copy it verbatim, then customize the YAML to your tool.

2.3. Run it

docker compose up server                # or your own deployment
# host fires up secsgem-py / wonderware / equipment manager:
#   selects, S1F13, S1F1, S1F3 → you're talking GEM.

That's the floor. From here, every section below adds capability.

---

3. Wiring real sensors to SVIDs

The YAML's value: field is the initial value. Your application updates the live value as the tool runs:

// In your sensor-poll thread (running on a separate executor):
double torr = read_baratron();
model->svids.set_value(/*ChamberPressure=*/100, secs2::Item::f4(float(torr)));

That's it — the next S1F3 from the host returns the fresh value.

Two patterns scale well:

1. One updater per sensor, fixed cadence. Each sensor's driver owns the (vid, set_value) pair.
2. A single refresh tick. A periodic timer dumps all polled values at once (refresh() in apps/secs_server.cpp does this for two virtual SVIDs).

The SECS-II Item shape must match the YAML's type:. If the YAML says F4 and you call set_value(100, secs2::Item::ascii("...")), the host will get the string back — the library doesn't enforce a runtime check. Treat the YAML type as a contract you maintain.

---

4. Plugging the FSMs into your tool

Every GEM 300 sub-state-machine in the library is a behavior model. You decide when state transitions happen by firing events:

4.1. Equipment processing (E116 EPT)

// At startup or whenever the operator clicks "Run":
model->ept.on_event(gem::EptEvent::EnterStandby);
model->ept.on_event(gem::EptEvent::EnterProductive);

// Auto-emit S6F11(ControlEvent_*) on every transition:
model->ept.set_state_change_handler(
    [&](gem::EptState, gem::EptState to, gem::EptEvent,
        std::chrono::milliseconds /*dwell*/) {
        uint32_t ceid = ept_state_to_ceid(to);  // your switch/case
        if (!ceid || !model->is_event_enabled(ceid)) return;
        conn->send_data(gem::s6f11_event_report(
            next_dataid++, ceid, model->compose_reports_for(ceid)));
    });

Helpers:

• model->ept.accumulated(state) — total milliseconds spent in state since startup. Use it to populate E116 SVIDs.
• model->ept.reset_history() — call at shift boundary.

4.2. Carriers + load ports (E87)

When AMHS delivers a carrier:

model->carriers.create("CAR-A1B2", /*port=*/1, /*capacity=*/25);
model->carriers.fire_id_event("CAR-A1B2", gem::CarrierIDEvent::Read);
// ... host sends S3F17(ProceedWithCarrier), the registered handler
// in the Router calls fire_id_event(..., ProceedWithCarrier) and
// CIDS moves NotConfirmed → Confirmed.

When your slot-map scanner finishes:

auto* c = model->carriers.get("CAR-A1B2");
for (std::size_t i = 0; i < scan_result.size(); ++i)
    c->slots[i].state = scan_result[i];   // 0 empty, 1 occupied
model->carriers.fire_slot_map_event("CAR-A1B2", gem::SlotMapEvent::Read);

The S3F19/F20 verify handler will compare against this map.

4.3. Substrates (E90)

For each wafer you start tracking:

model->substrates.create("W-2024-001", "CAR-A1B2", /*slot=*/1);
model->substrates.fire_location_event(
    "W-2024-001", gem::SubstrateEvent::Acquire, /*location=*/"ChamberA");
model->substrates.fire_processing_event(
    "W-2024-001", gem::SubstrateProcessingEvent::StartProcessing);
// ... when done:
model->substrates.fire_processing_event(
    "W-2024-001", gem::SubstrateProcessingEvent::EndProcessing);
model->substrates.fire_location_event(
    "W-2024-001", gem::SubstrateEvent::Release, /*location=*/"OutCarrier");

History is tracked per-substrate (model->substrates.history(id)) and can power your downtime / yield reports.

4.4. Process jobs + control jobs (E40 / E94)

The host creates these via S16F11 / S14F9. Your application drives their internal transitions as the recipe engine progresses:

// Recipe runner reports setup done:
model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::SetupComplete);
// Operator hits Start (or autorun is on):
model->process_jobs.on_host_command("PJ-1", gem::ProcessJobEvent::Start);
// Recipe completed normally:
model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::ProcessComplete);

CJ state cascades the same way (E94).

4.5. Alarms (E5 §13)

// Sensor crosses threshold:
model->alarms.set(/*alid=*/1, /*set=*/true);  // emits S5F1(ALCD=0x84)
// Later it clears:
model->alarms.set(1, false);                  // emits S5F1(ALCD=0x04)

The dispatcher takes care of the wire frame — you just toggle.

4.6. Recoverable exceptions (E5 §9, S5F9–F18)

For faults where you want a host/equipment recovery dialogue:

model->exceptions.post(/*exid=*/42, "VACUUM",
                       "lost vacuum in chamber A",
                       {"PURGE", "RECOVER", "ABORT"});  // emits S5F9
// Host picks PURGE via S5F13; the registered handler calls
// model->exceptions.on_recover(42, "PURGE"), state moves to Recovering.
// Your purge routine completes:
model->exceptions.fire_internal(42, gem::ExceptionEvent::RecoveryComplete);
// state → Cleared; S5F11 fires; entry removed.

---

5. Persistence

GEM equipment that loses power mid-job can recover gracefully because every store the library ships supports an opt-in file-backed journal. Enable per store, at startup, BEFORE the connection comes up:

auto base = std::filesystem::path("/var/lib/acme/secsgem");
model->spool.enable_persistence(base / "spool");
model->carriers.enable_persistence(base / "carriers");
model->load_ports.enable_persistence(base / "loadports");
model->substrates.enable_persistence(base / "substrates");
model->process_jobs.enable_persistence(base / "pjobs");
model->control_jobs.enable_persistence(base / "cjobs");
model->exceptions.enable_persistence(base / "exceptions");

On enable, the store scans the directory, replays records into in-memory state, and from there keeps the directory in sync on every create / state-change / remove. Writes use a .tmp + rename pattern so a power loss mid-write can lose at most the in-flight record (older records remain coherent).

Storage budget per store, roughly:

• spool: one file per spooled S6F11 (typically tens of bytes each)
• carriers: one file per carrier (~50 bytes + slot count)
• load_ports: one file per LP (~30 bytes)
• substrates: one file per wafer (~80 bytes)
• pjobs: one file per active PJ (~100 bytes), plus order.idx
• cjobs: one file per active CJ (~80 bytes)
• exceptions: one file per Posted/Recovering exception

Even a busy fab tool tops out at a few hundred files in each directory — well within filesystem caps. Sweep terminal-state entries (completed PJs, cleared exceptions) periodically if you care about directory size.

Caveats currently captured in the persistence tests:

• Substrate history is intentionally NOT journaled — only the current state of each axis. Replay starts with an empty history vector.
• PJ rcpvars / prprocessparams (the optional E40 secs2::Item trailers) are not journaled in v1; call set_e40_extras again on the application side after restart if you need them.

---

6. Monitoring + observability

6.1. Connection lifecycle

conn->set_log_handler([](const std::string& m) {
    syslog(LOG_INFO, "hsms: %s", m.c_str());
});
conn->set_selected_handler([] { metrics.inc("hsms.selected"); });
conn->set_closed_handler([](const std::string& r) {
    metrics.inc("hsms.closed", {{"reason", r}});
});

6.2. State change observers

Every store / FSM exposes a set_*_change_handler. Hook them up to your metrics / log pipeline:

model->control_jobs.set_state_change_handler(
    [](const std::string& cj, gem::ControlJobState f, gem::ControlJobState t,
       gem::ControlJobEvent) {
        log("CJ " + cj + " " + gem::control_job_state_name(f) +
            " → " + gem::control_job_state_name(t));
    });

6.3. Self-emitted protocol errors

Look for S9F* traffic in your logs. S9F3 / F5 mean the host asked for something your router doesn't handle; S9F7 means a bad body arrived; S9F9 means a reply didn't arrive in T3; S9F11 means a frame exceeded the 16 MiB cap. None of these are normal — they're real diagnostic events.

---

7. Recommended layout for a vendor application

/opt/acme-secsgem/
├─ bin/
│  └─ secsgem-equipment          # your built binary
├─ etc/
│  ├─ equipment.yaml             # your tool's dictionary
│  └─ control_state.yaml         # your tool-specific state model
├─ var/
│  ├─ spool/                     # populated at runtime
│  ├─ carriers/
│  ├─ substrates/
│  ├─ pjobs/
│  ├─ cjobs/
│  └─ exceptions/
└─ share/
   └─ doc/                       # COMPLIANCE.md, INTEGRATION.md

Your application reads etc/, writes to var/, and never touches share/. YAML edits don't require a rebuild — restart the process.

The control-state YAML is your tool's processing state machine — E30 deliberately leaves the concrete states (IDLE / SETUP / READY / EXECUTING / PAUSE / …) up to the tool builder. Copy data/control_state.yaml as a starting point.

---

8. Test the integration

Don't ship without:

1. Round-trip every host-facing message you serve. The library's own test suite covers the codecs; you should also drive your YAML's specific SVIDs / CEIDs / alarms / recipes against the built-in passive server using the interop/host_vs_cpp_server.py harness as a template.

2. Power-loss simulation. Kill -9 the process mid-job, restart, confirm the stores replay the correct state. The persistence tests give you a template; copy and parameterize for your store directories.

3. Multi-hour soak. Spool fills up if persistence is enabled and the host link is down — make sure your fab's MES side ack-rate keeps up. Run a 24h test with the host periodically disconnecting and watch the journal directory.

4. The two-container demo in this repo gives you a starting harness — the host emulator (apps/secs_client.cpp) drives ~20 transactions through your server. Adapt it to your message set.

---

9. When to extend the runtime

The library is open to extension. Common reasons to add code:

• A new SECS-II message the catalog doesn't cover. Edit data/messages.yaml, run the codegen (built into the CMake pipeline), add a Router handler. No core code change.
• A new state machine specific to your tool (e.g. an in-chamber cooling cycle FSM). Lift the pattern from ept_state.hpp: define your states + events + transition table; let your application drive it.
• An additional persistence backend (DB instead of files). Mirror the spool .enable_persistence pattern — it's about 100 lines per store.

If your change is broadly useful, it's worth landing in the library itself. See COMPLIANCE.md for the standards still on the "explicitly out of scope" list — anything there is a possible contribution.

---

10. Going from "stack" to "certified GEM tool"

This codebase passes its own conformance harness and cross-validates against secsgem-py, but a real certified GEM tool needs more:

• Independent third-party validator. Run a GEM RTS (Reference Test System) or equivalent against your integration. The library serves the messages; the validator decides whether your data is consistent.

• Vendor application code. The runtime cannot, by design, know what your SVID values should be at any given moment. That's your domain knowledge plugged into the data model and FSMs.

• Documentation. E30 §6.10 (Documentation capability) requires you to publish what you implement. COMPLIANCE.md in this repo is the template — fork it, prune to your actual coverage, ship it with your tool.

• Operations: monitoring dashboards, alarm escalation, log retention — the standard SRE concerns, no different from any other piece of fab software.