secs-gem/docs/41_integration_hardware_mes_production.md

41 — Integration: hardware, MES, production

← 40 Building, running, the demo | Back to index | Next: 50 API + messages + YAML reference →

You have the demo running. Now you need to make it talk to a real tool, against a real MES, in a real fab.

This chapter walks the four phases of that journey:

1. Wire the EAP to physical hardware.
2. Integrate with a commercial MES.
3. Production hardening — security, monitoring, persistence.
4. Operational concerns — performance, capacity, incidents.

This is a compressed view of the long-form docs/INTEGRATION.md; cross-references are inline. The long-form has more code and more configuration; this chapter explains the shape of each phase.

---

Phase 1 — wiring to hardware

What "the EAP" actually does

The EAP (Equipment Automation Program) sits between the SECS/GEM runtime and the physical tool. It does four things:

1. Reads sensors at the right cadence and updates SVIDs.
2. Drives the recipe engine when a host command arrives.
3. Listens for alarms from PLCs / hardware fault lines.
4. Wires FSM transitions to CEID emissions.

examples/pvd_tool/main.cpp is the worked reference. Section by section:

Section in main.cpp	What it shows
§1 Helpers + constants	The kSvidX / kCeidX constants worth pinning at file scope
§2 Sensor simulator	Multi-cadence sensor poll loops with asio::post strand-marshal
§3 Recipe runner	PJ → SettingUp → Processing → ProcessComplete walk; per-step CEID emit
§4 Alarm threshold monitor	Continuous threshold evaluation against ECID setpoints
§5 EPT cycling	E116 transitions driven by PJ state + safety alarms
§6 main()	EquipmentRuntime + register_default_handlers (all 56) + set_handler for START — was 51 hand-written handlers before the runtime
§7 main()	YAML load → validate → compose → run

A real tool fork:

cp -r examples/pvd_tool/ src/my_tool/
# edit src/my_tool/equipment.yaml — your tool's SVIDs/CEIDs/alarms
# edit src/my_tool/main.cpp — replace pvd::Simulator with PLC bindings

Sensor wiring

The PVD example uses a random-walk simulator (§2). A real EAP replaces this with calls into the tool's sensor stack:

// Original (simulated):
upd_f4(kSvidChamberPressure, target_pressure.load(), 1e-8f, 1e-7f);

// Real:
asio::post(io, [model](){
  float pressure = plc_read_register(0x4001);  // from your PLC API
  model->svids.set_value(kSvidChamberPressure, secs2::Item::f4(pressure));
});

The asio::post is non-negotiable — the store mutation runs on the io_context strand (chapter 33).

Recipe runner

The PVD example's recipe runner (§3) parses the recipe body and walks PJ states. A real tool replaces the simulator with calls into the tool's recipe engine:

void start_processing(const std::string& pjid, const std::string& ppid) {
  auto recipe = recipes_->find(ppid);
  if (!recipe) {
    model->process_jobs.apply(pjid, ProcessJobEvent::Abort);
    return;
  }
  // Hand the recipe to the tool's actual recipe engine.
  hardware_recipe_engine_->start(*recipe, [model, pjid](bool ok) {
    asio::post(io, [model, pjid, ok] {
      model->process_jobs.apply(pjid,
          ok ? ProcessJobEvent::ProcessComplete
             : ProcessJobEvent::AbortComplete);
    });
  });
}

Alarm sources

Real alarms come from:

• PLC fault lines — interrupt callbacks.
• Watchdog timers — periodic checks (cooling water flow, vacuum pressure).
• Sensor thresholds — continuous evaluation against ECIDs.
• Hardware safety interlocks — SafetyController callbacks.

Each translates to one model->alarms.set(alid) call. The alarm dispatcher takes care of S5F1 emission, host enable filtering, and alarm persistence.

E84 wiring

E84 needs a GPIO driver:

// On signal change from the GPIO driver:
void on_gpio_change(uint8_t port, E84Signal sig, bool value) {
  asio::post(io, [model, port, sig, value]() {
    model->e84_ports.at(port).fsm.on_signal_change(sig, value);
  });
}

// When the FSM wants to assert a signal:
model->e84_ports.at(port).fsm.set_emit_handler(
    [port](E84Signal sig, bool value) {
      gpio_driver_write(port, sig, value);
    });

The TA1/TA2/TA3 timers are wall-clock; use the E84AsioTimers adapter so they fire on the same io_context.

---

Phase 2 — talking to a real MES

The day-1 punch list

Before you connect to a production MES, run docs/MES_INTEROP.md against the staging MES. 59 test IDs across:

• Transport (T-01 to T-09): SELECT, Linktest, T3, T7, oversized frames.
• Establishment (E-01 to E-08): S1F13, S1F1, S1F11, S1F19, …
• Reports (R-01 to R-07): the S2F33/F35/F37 dance.
• Alarms (A-01 to A-06).
• Commands (C-01 to C-04): S2F41 + S2F21 (legacy) + S2F49 (enhanced).
• Recipes (P-01 to P-05).
• Terminal services (TS-01 to TS-03).
• Jobs (J-01 to J-06).
• Spool (SP-01 to SP-05).
• Clock (K-01 to K-06).

This is the friction-killer document. Pass every test ID in staging and your production cutover is much less likely to surprise you.

HSMS-GS for multi-MES

Some fabs run multiple MES against one equipment. E37 §11 HSMS-GS multiplexes over one TCP socket:

auto conn = std::make_shared<hsms::Connection>(
    std::move(sock), Mode::Passive, /*primary device_id=*/0, timers);

// Production MES on session 100.
conn->add_session(100);
conn->set_session_message_handler(100, production_router_handler);

// Maintenance MES on session 200.
conn->add_session(200);
conn->set_session_message_handler(200, maintenance_router_handler);

docs/INTEGRATION.md §7 has the full worked example with HA pattern. Tests: tests/test_hsms_gs.cpp (5 wire-level) and tests/test_hsms_gs_integration.cpp (1 end-to-end three-session scenario).

Things commercial MES get wrong

Real MES exhibit common deviations:

• Wonderware uses S2F21 (legacy) exclusively — no S2F41. The codebase's HostCommandRegistry handles both forms.
• Some MES leave EQPTYP in S1F20 confused with MDLN — the codebase accepts either; documented in docs/MES_INTEROP.md E-02 caveat.
• MES with old PPBODY handling reject binary recipes — the codebase ships both as-bytes and as-ASCII PPBODY.
• Camstar uses Linktest at 30 s, others at 60 s — configure Timers::linktest to match the host's cadence.

docs/MES_INTEROP.md "Caveats" column lists more.

---

Phase 3 — production hardening

Security: SECURITY.md walk-through

docs/SECURITY.md ships concrete configs for:

• nftables — restrict the SECS port to the MES host's IP only.
• stunnel — wrap the HSMS port in TLS so the wire is encrypted.
• minisign — sign every recipe (PPBODY) and verify on receive.
• SIEM audit log schema — what every store mutation emits as JSON for log aggregation.

Configure all four before promoting to production. No exceptions.

Persistence layout

Production deployments enable persistence on every store that needs it. Per docs/INTEGRATION.md §5:

/var/lib/secsgem/
├── spool/         # SpoolStore
├── pj/            # ProcessJobStore
├── cj/            # ControlJobStore
├── exceptions/    # ExceptionStore
├── carriers/      # CarrierStore
├── load_ports/    # LoadPortStore
└── substrates/    # SubstrateStore

On an SSD with fsync enabled per file rewrite, this comfortably handles a few hundred mutations / sec. On rotational media you'll want to batch or relax durability.

Monitoring

Production EAPs typically export:

• Per-CEID emission counter — burst detection.
• Spool depth gauge — alarms when growing (MES connectivity problem).
• T3 timeout counter — non-zero means the MES is slow or your T3 is too short.
• Per-alarm set count — pages on certain ALIDs.
• Equipment EPT state gauge — fab-wide dashboard input.

docs/INTEGRATION.md §6.4 covers the Grafana panel patterns; the PVD example wires the Prometheus exporter at §7.

Operational runbook

README.md ships a starter runbook:

Incident	First check	Mitigation
HSMS connection flapping	T7 / T6 timer fires in logs	check MES reachability, network MTU
Spool depth growing	host MES connectivity / ACK rate	force-drain via S6F23, escalate to MES
State machine "stuck"	last state-change handler log line	host-issued offline + re-establish
Alarm storm	AlarmRegistry::all() snapshot	check upstream sensor; quench via S5F3
Persistence dir growing unbounded	du -s + file count	sweep terminal-state records
Cross-tool inconsistency	secsgem_tests on canary tool	compare wire trace vs validator

---

Phase 4 — performance

The envelope

Per docs/BENCHMARKS.md, on a 2024 M-series Mac under Docker Desktop:

Scenario	Ops/sec	p50 µs	p95 µs	p99 µs
S1F1/F2 (header-only)	~140 k	74	103	161
S1F3/F4 (32 SVIDs)	~79 k	165	186	260
S6F11 push (W=0)	~572 k	n/a	n/a	n/a

A real fab tool sees tens to a few hundred events / s sustained. We're three orders of magnitude above the push path, two orders above the round-trip path.

Throughput is not the bottleneck; tail latency under contention is. Tune by:

• Running on a quiet host.
• Bumping linktest interval up (default 0 = disabled is fine for most deployments).
• Pinning the io_context thread to a dedicated CPU.

Memory footprint

Entity	Approx bytes / instance
PJ + CJ pair	~450
Carrier (no slots)	~80
Carrier slot	~24
Substrate	~120
Spool entry	~40 + encoded body size

A busy 300 mm tool with 50 carriers × 25 slots + 200 substrates + 20 active PJ+CJ pairs is under 1 MiB of model state. RSS is dominated by the binary itself + asio's buffers (~10–20 MiB).

Capacity planning

For sizing purposes:

• One io_context thread per Connection is plenty for any single tool.
• Multiple tools share the same process if you want — one io_context, multiple Connections, each on its own strand.
• Persistence cost is one rename(2) per mutation; SSD-bound fabs comfortably handle a few hundred / sec.

---

When to read the long-form

This chapter compressed phases 1–4 into a tour. Each phase has substantially more material in the long-form docs:

• docs/INTEGRATION.md — full vendor-side tutorial including wiring sensors, plugging FSMs, persistence layout, monitoring, HSMS-GS HA.
• docs/MES_INTEROP.md — the 59 test IDs in full.
• docs/SECURITY.md — concrete nftables / stunnel / minisign configs.
• docs/BENCHMARKS.md — perf envelope + how to re-run.

When you actually start an integration, work from those. This chapter is the map.

---

End of Part 4

You can now build the codebase, run the demo, drive every external validator, and you know the shape of how a real integration would land. Part 5 is reference material — API namespaces, message catalog reference, the extension recipes.

Next: → 50 API + messages + YAML reference