# 41 — Integration: hardware, MES, production

← [40 Building, running, the demo](40_building_running_demo.md) | [Back to index](00_index.md) | Next: [50 API + messages + YAML reference](50_api_messages_yaml_reference.md) →

You have the demo running.  Now you need to make it talk to a
**real tool**, against a **real MES**, in a **real fab**.

This chapter walks the four phases of that journey:

1. **Wire the EAP to physical hardware.**
2. **Integrate with a commercial MES.**
3. **Production hardening** — security, monitoring, persistence.
4. **Operational concerns** — performance, capacity, incidents.

This is a compressed view of the long-form
[`docs/INTEGRATION.md`](INTEGRATION.md); cross-references are
inline.  The long-form has more code and more configuration; this
chapter explains the *shape* of each phase.

---

## Phase 1 — wiring to hardware

### What "the EAP" actually does

The EAP (Equipment Automation Program) sits between the SECS/GEM
runtime and the **physical tool**.  It does four things:

1. **Reads sensors** at the right cadence and updates SVIDs.
2. **Drives the recipe engine** when a host command arrives.
3. **Listens for alarms** from PLCs / hardware fault lines.
4. **Wires FSM transitions** to CEID emissions.

[`examples/pvd_tool/main.cpp`](../examples/pvd_tool/main.cpp) is
the worked reference.  Section by section:

| Section in main.cpp           | What it shows                                                          |
|-------------------------------|------------------------------------------------------------------------|
| §1 Helpers + constants        | The `kSvidX / kCeidX` constants worth pinning at file scope            |
| §2 Sensor simulator           | Multi-cadence sensor poll loops with `asio::post` strand-marshal       |
| §3 Recipe runner              | PJ → SettingUp → Processing → ProcessComplete walk; per-step CEID emit |
| §4 Alarm threshold monitor    | Continuous threshold evaluation against ECID setpoints                 |
| §5 EPT cycling                | E116 transitions driven by PJ state + safety alarms                    |
| §6 Router handlers            | 51 handlers in ~460 lines — every S/F a host might send to a PVD tool  |
| §7 main()                     | YAML load → validate → compose → run                                   |

A real tool fork:

```bash
cp -r examples/pvd_tool/ src/my_tool/
# edit src/my_tool/equipment.yaml — your tool's SVIDs/CEIDs/alarms
# edit src/my_tool/main.cpp — replace pvd::Simulator with PLC bindings
```

### Sensor wiring

The PVD example uses a random-walk simulator (§2).  A real EAP
replaces this with calls into the tool's sensor stack:

```cpp
// Original (simulated):
upd_f4(kSvidChamberPressure, target_pressure.load(), 1e-8f, 1e-7f);

// Real:
asio::post(io, [model](){
  float pressure = plc_read_register(0x4001);  // from your PLC API
  model->svids.set_value(kSvidChamberPressure, secs2::Item::f4(pressure));
});
```

The `asio::post` is non-negotiable — the store mutation runs on
the io_context strand (chapter 33).

### Recipe runner

The PVD example's recipe runner (§3) parses the recipe body and
walks PJ states.  A real tool replaces the simulator with calls
into the tool's recipe engine:

```cpp
void start_processing(const std::string& pjid, const std::string& ppid) {
  auto recipe = recipes_->find(ppid);
  if (!recipe) {
    model->process_jobs.apply(pjid, ProcessJobEvent::Abort);
    return;
  }
  // Hand the recipe to the tool's actual recipe engine.
  hardware_recipe_engine_->start(*recipe, [model, pjid](bool ok) {
    asio::post(io, [model, pjid, ok] {
      model->process_jobs.apply(pjid,
          ok ? ProcessJobEvent::ProcessComplete
             : ProcessJobEvent::AbortComplete);
    });
  });
}
```

### Alarm sources

Real alarms come from:

- **PLC fault lines** — interrupt callbacks.
- **Watchdog timers** — periodic checks (cooling water flow,
  vacuum pressure).
- **Sensor thresholds** — continuous evaluation against ECIDs.
- **Hardware safety interlocks** — SafetyController callbacks.

Each translates to one `model->alarms.set(alid)` call.  The alarm
dispatcher takes care of S5F1 emission, host enable filtering, and
alarm persistence.

### E84 wiring

E84 needs a GPIO driver:

```cpp
// On signal change from the GPIO driver:
void on_gpio_change(uint8_t port, E84Signal sig, bool value) {
  asio::post(io, [model, port, sig, value]() {
    model->e84_ports.at(port).fsm.on_signal_change(sig, value);
  });
}

// When the FSM wants to assert a signal:
model->e84_ports.at(port).fsm.set_emit_handler(
    [port](E84Signal sig, bool value) {
      gpio_driver_write(port, sig, value);
    });
```

The TA1/TA2/TA3 timers are wall-clock; use the
[`E84AsioTimers`](../include/secsgem/gem/e84_asio_timers.hpp)
adapter so they fire on the same io_context.

---

## Phase 2 — talking to a real MES

### The day-1 punch list

Before you connect to a production MES, run
[`docs/MES_INTEROP.md`](MES_INTEROP.md) against the **staging**
MES.  59 test IDs across:

- Transport (T-01 to T-09): SELECT, Linktest, T3, T7, oversized
  frames.
- Establishment (E-01 to E-08): S1F13, S1F1, S1F11, S1F19, …
- Reports (R-01 to R-07): the S2F33/F35/F37 dance.
- Alarms (A-01 to A-06).
- Commands (C-01 to C-04): S2F41 + S2F21 (legacy) + S2F49
  (enhanced).
- Recipes (P-01 to P-05).
- Terminal services (TS-01 to TS-03).
- Jobs (J-01 to J-06).
- Spool (SP-01 to SP-05).
- Clock (K-01 to K-06).

This is the **friction-killer document**.  Pass every test ID in
staging and your production cutover is much less likely to
surprise you.

### HSMS-GS for multi-MES

Some fabs run multiple MES against one equipment.  E37 §11
HSMS-GS multiplexes over one TCP socket:

```cpp
auto conn = std::make_shared<hsms::Connection>(
    std::move(sock), Mode::Passive, /*primary device_id=*/0, timers);

// Production MES on session 100.
conn->add_session(100);
conn->set_session_message_handler(100, production_router_handler);

// Maintenance MES on session 200.
conn->add_session(200);
conn->set_session_message_handler(200, maintenance_router_handler);
```

[`docs/INTEGRATION.md`](INTEGRATION.md) §7 has the full worked
example with HA pattern.  Tests:
[`tests/test_hsms_gs.cpp`](../tests/test_hsms_gs.cpp) (5 wire-level)
and
[`tests/test_hsms_gs_integration.cpp`](../tests/test_hsms_gs_integration.cpp)
(1 end-to-end three-session scenario).

### Things commercial MES get wrong

Real MES exhibit common deviations:

- **Wonderware uses S2F21 (legacy) exclusively** — no S2F41.  The
  codebase's HostCommandRegistry handles both forms.
- **Some MES leave EQPTYP in S1F20 confused with MDLN** — the
  codebase accepts either; documented in
  [`docs/MES_INTEROP.md`](MES_INTEROP.md) E-02 caveat.
- **MES with old PPBODY handling reject binary recipes** — the
  codebase ships both as-bytes and as-ASCII PPBODY.
- **Camstar uses Linktest at 30 s**, others at 60 s — configure
  `Timers::linktest` to match the host's cadence.

[`docs/MES_INTEROP.md`](MES_INTEROP.md) "Caveats" column lists more.

---

## Phase 3 — production hardening

### Security: SECURITY.md walk-through

[`docs/SECURITY.md`](SECURITY.md) ships concrete configs for:

- **nftables** — restrict the SECS port to the MES host's IP only.
- **stunnel** — wrap the HSMS port in TLS so the wire is
  encrypted.
- **minisign** — sign every recipe (PPBODY) and verify on receive.
- **SIEM audit log schema** — what every store mutation emits as
  JSON for log aggregation.

Configure all four before promoting to production.  No exceptions.

### Persistence layout

Production deployments enable persistence on every store that
needs it.  Per [`docs/INTEGRATION.md`](INTEGRATION.md) §5:

```
/var/lib/secsgem/
├── spool/         # SpoolStore
├── pj/            # ProcessJobStore
├── cj/            # ControlJobStore
├── exceptions/    # ExceptionStore
├── carriers/      # CarrierStore
├── load_ports/    # LoadPortStore
└── substrates/    # SubstrateStore
```

On an SSD with `fsync` enabled per file rewrite, this comfortably
handles a few hundred mutations / sec.  On rotational media you'll
want to batch or relax durability.

### Monitoring

Production EAPs typically export:

- **Per-CEID emission counter** — burst detection.
- **Spool depth gauge** — alarms when growing (MES connectivity
  problem).
- **T3 timeout counter** — non-zero means the MES is slow or your
  T3 is too short.
- **Per-alarm set count** — pages on certain ALIDs.
- **Equipment EPT state gauge** — fab-wide dashboard input.

[`docs/INTEGRATION.md`](INTEGRATION.md) §6.4 covers the Grafana
panel patterns; the PVD example wires the Prometheus exporter at
§7.

### Operational runbook

[`README.md`](../README.md) ships a starter runbook:

| Incident                            | First check                          | Mitigation                                |
|-------------------------------------|--------------------------------------|-------------------------------------------|
| HSMS connection flapping            | T7 / T6 timer fires in logs          | check MES reachability, network MTU       |
| Spool depth growing                 | host MES connectivity / ACK rate     | force-drain via S6F23, escalate to MES    |
| State machine "stuck"               | last state-change handler log line   | host-issued offline + re-establish        |
| Alarm storm                         | `AlarmRegistry::all()` snapshot      | check upstream sensor; quench via S5F3    |
| Persistence dir growing unbounded   | `du -s` + file count                 | sweep terminal-state records              |
| Cross-tool inconsistency            | `secsgem_tests` on canary tool       | compare wire trace vs validator           |

---

## Phase 4 — performance

### The envelope

Per [`docs/BENCHMARKS.md`](BENCHMARKS.md), on a 2024 M-series Mac
under Docker Desktop:

| Scenario                          | Ops/sec |  p50 µs | p95 µs | p99 µs |
|-----------------------------------|--------:|--------:|-------:|-------:|
| S1F1/F2 (header-only)             |  ~140 k |     74  |    103 |    161 |
| S1F3/F4 (32 SVIDs)                |   ~79 k |    165  |    186 |    260 |
| S6F11 push (W=0)                  |  ~572 k |    n/a  |    n/a |    n/a |

A real fab tool sees **tens to a few hundred** events / s
sustained.  We're three orders of magnitude above the push path,
two orders above the round-trip path.

**Throughput is not the bottleneck**; tail latency under
contention is.  Tune by:

- Running on a quiet host.
- Bumping `linktest` interval up (default 0 = disabled is fine for
  most deployments).
- Pinning the io_context thread to a dedicated CPU.

### Memory footprint

| Entity              | Approx bytes / instance |
|---------------------|------------------------:|
| PJ + CJ pair        |                    ~450 |
| Carrier (no slots)  |                     ~80 |
| Carrier slot        |                     ~24 |
| Substrate           |                    ~120 |
| Spool entry         | ~40 + encoded body size |

A busy 300 mm tool with 50 carriers × 25 slots + 200 substrates +
20 active PJ+CJ pairs is under **1 MiB** of model state.  RSS is
dominated by the binary itself + asio's buffers (~10–20 MiB).

### Capacity planning

For sizing purposes:

- One io_context thread per `Connection` is plenty for any single
  tool.
- Multiple tools share the same process if you want — one
  io_context, multiple `Connection`s, each on its own strand.
- Persistence cost is one `rename(2)` per mutation; SSD-bound
  fabs comfortably handle a few hundred / sec.

---

## When to read the long-form

This chapter compressed phases 1–4 into a tour.  Each phase has
substantially more material in the long-form docs:

- [`docs/INTEGRATION.md`](INTEGRATION.md) — full vendor-side
  tutorial including wiring sensors, plugging FSMs, persistence
  layout, monitoring, HSMS-GS HA.
- [`docs/MES_INTEROP.md`](MES_INTEROP.md) — the 59 test IDs in
  full.
- [`docs/SECURITY.md`](SECURITY.md) — concrete nftables / stunnel /
  minisign configs.
- [`docs/BENCHMARKS.md`](BENCHMARKS.md) — perf envelope + how to
  re-run.

When you actually start an integration, work from those.  This
chapter is the map.

---

## End of Part 4

You can now build the codebase, run the demo, drive every
external validator, and you know the shape of how a real
integration would land.  Part 5 is reference material — API
namespaces, message catalog reference, the extension recipes.

Next: [→ 50 API + messages + YAML reference](50_api_messages_yaml_reference.md)