secs-gem/README.md

secs-gem

A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully containerized. Every behavioural rule lives in YAML; the C++ is the engine that reads them. Implements all of E4, E5, E30, E37 (SS + GS), E39, E40, E42, E84, E87, E90, E94, E116, E120, E148, E157.

License: proprietary — see LICENSE. No use, copy, compile, evaluate, benchmark, or deploy without a written license from the copyright holder. Contact raphael@maenle.net for commercial licensing, evaluation terms, or fab deployment.

---

Quick start

Everything runs in Docker — no compiler or build tools on the host.

docker compose run --rm builder         # configure + compile
docker compose run --rm tests           # 473 cases / 3 087 assertions
docker compose up --no-deps server client   # live two-container demo

The two-container demo walks ~24 SECS transactions end-to-end through the data model. Watch the logs interleave.

---

Integrating your tool (pick a tier)

Three ways in, same engine underneath:

1. Python, no SEMI knowledge — run the secs_gemd daemon and pip install the pure-Python client in clients/python:

   from secsgem_client import Equipment
   
   eq = Equipment("localhost:50051")
   eq.set(ChamberPressure=2.5)          # variables: kwargs, not strings
   
   @eq.command                          # the function name IS the command,
   def START(cmd):                      # validated against the real equipment
       run_recipe(cmd.params.get("PPID"))   # — so a typo fails at startup
       eq.fire(eq.names.event.ProcessStarted)   # autocomplete + typo-safe
   
   eq.listen()
   

   Names come from your equipment.yaml. @eq.command binds a handler by its function name; eq.names.event.* / .alarm.* / .command.* are autocomplete-able, typo-checked views fetched from the live daemon — so you rarely type a bare string. (The plain forms — @eq.on("START"), eq.fire("ProcessStarted") — also work.)

   A complete tool is ~25 lines: clients/python/examples/mini_tool.py.

2. Any language over gRPC — secs_gemd exposes the name-based API in proto/secsgem/v1/equipment.proto (variables, events, alarms, control state, health stream, and the host-command stream with the SEMI-conformant HCACK-4 contract). The daemon owns the durable HSMS link: your tool software can restart without the fab host ever noticing.

3. Embedded C++ — construct a gem::EquipmentRuntime, call the per-capability register_* functions (or register_default_handlers for all of GEM), and wire behaviour with commands.set_handler. apps/secs_server.cpp is the ~110-line canonical example.

---

Documentation map

File	What it covers
docs/	Guided-tour tutorial series — teach-from-zero across the protocol and the codebase
docs/COMPLIANCE.md	Per-capability audit against every SEMI standard implemented
docs/INTEGRATION.md	Vendor-side tutorial: YAML → callbacks → production deploy
docs/ARCHITECTURE.md	How the pieces fit + how to extend (new store / FSM / message)
docs/PROOFS.md	The eight commands that prove the feature-completeness claim
docs/VERIFICATION.md	Test plan for the external validators behind the proof table
docs/BENCHMARKS.md	Performance envelope (throughput, latency, memory) + how to re-run
docs/MES_INTEROP.md	Day-1 punch list to run against your commercial MES (59 test IDs)
docs/SECURITY.md	Concrete configs: nftables, stunnel, minisign, SIEM audit-log schema
docs/GLOSSARY.md	SEMI vocabulary: SVID, CEID, PPID, ALCD, HCACK, T-timers, …
docs/FAQ.md	Common questions and their canonical answers
examples/pvd_tool/	Worked example: a realistic fictional PVD tool, YAML + C++ wiring
LICENSE	Proprietary license terms

---

Testing

• Unit + integration — docker compose run --rm tests runs 473 cases / 3 087 assertions across every store, FSM, codec, parser, and persistence path.
• Live conformance harness — 47 wire-level checks against the passive server.
• Interop — round-trips against secsgem-py (31 checks), secs4java8 (55 checks), and Wireshark's HSMS dissector (69 frames, 0 malformed).
• Soak + fuzz — 100 000-op property test; libFuzzer with ASan + UBSan over secs2::decode and the SML parser, 0 crashes.
• Daemon — secs_gemd_tests exercises the gRPC service over real in-process channels (125 assertions), in Release and under ThreadSanitizer; interop/daemon_interop.py and interop/pyclient_interop.py prove the gRPC↔HSMS bridge and the published Python client against a live daemon with secsgem-py as host.
• One command for all of it — tools/run_interop.sh runs every validation step (build, both unit suites, secsgem-py host, C++ conformance, Python client, daemon bridge, spool restart, tshark, secs4java8) with a PASS/FAIL summary.
• Config validation — secs_server --validate-config rejects malformed YAML before startup.
• CI — Gitea Actions runs the full suite plus a -fsanitize=thread lane on every push to main; all 473 cases pass clean under TSan.

Exact commands, exit codes, and per-standard test counts are in docs/PROOFS.md; the rationale behind the external validators is in docs/VERIFICATION.md.

---

Architecture

The project is spec-as-data: the SEMI behavioural rules live in YAML; the C++ is the engine that reads them.

   ┌──────────────────────────────────────────────────────────────┐
   │ data/                                                        │
   │   messages.yaml          SECS-II message catalog (164 msgs)  │
   │   control_state.yaml     E30 §6.2 control transition table   │
   │   process_job_state.yaml E40 §6 PJ transition table          │
   │   control_job_state.yaml E94 §6 CJ transition table          │
   │   equipment.yaml         SVIDs / DVIDs / ECIDs / CEIDs /     │
   │                          alarms / recipes / commands         │
   └──────────────────────┬───────────────────────────────────────┘
                          │  (codegen at build, YAML loaded at startup)
                          ▼
   ┌──────────────────────────────────────────────────────────────┐
   │ apps/                                                        │
   │   secs_server   passive equipment      secs_bench   perf     │
   │   secs_client   active host            secs_conformance      │
   │   secs_interop_probe                                         │
   └──────────────────────────────────────────────────────────────┘

   secsgem::config   loader.hpp + validate.hpp:
                     YAML -> data model, with multi-error validator
                     surfacing every issue at once (`--validate-config`)
   secsgem::gem      per-standard FSM + per-store persistence
                     (every store accepts v ∈ [1, kVersion] for
                     forward-compatible schema migrations).
                     EquipmentDataModel composes all stores.
                     Router (stream, function) -> handler.
                     Generated messages.hpp covers 164 SxFy.
   secsgem::hsms     Connection (Asio): HSMS-SS + HSMS-GS, all
                     T-timers enforced, auto S9F3/F5/F7/F9/F11.
   secsgem::secsi    SECS-I Protocol FSM (E4): T1/T2/T3/T4 enforced
                     in-FSM, TCP transport for tunnel testing.
   secsgem::secs2    Item (variant), encode/decode, Message,
                     SML parser/printer.
   secsgem::metrics  Prometheus exporter (Registry + HTTP server).

See docs/ARCHITECTURE.md for how to extend it (new store / FSM / message).

---

Adding a capability

The point of "spec-as-data" is that adding behaviour almost never requires a C++ change.

New SVID

# data/equipment.yaml
svids:
  - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}

New host command with side effects

host_commands:
  - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}

New state transition

# data/control_state.yaml
transitions:
  - {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept}

New SECS-II message

# data/messages.yaml
- id: S6F30
  stream: 6
  function: 30
  w: true
  builder: s6f30_something
  parser: parse_s6f30
  body:
    kind: list
    struct_name: Something
    fields:
      - {name: field_a, shape: {kind: scalar, item_type: U4}}
      - {name: field_b, shape: {kind: scalar, item_type: ASCII}}

docker compose run --rm builder regenerates messages.hpp. The typed builder, parser, and struct definition appear automatically. Run --validate-config after every YAML edit.

---

Production deployment

See docs/INTEGRATION.md for the full vendor-side tutorial — wiring sensors, plugging FSMs into the tool, persistence layout, monitoring/observability, HSMS-GS multi-MES setup.

See docs/SECURITY.md for concrete nftables / stunnel / minisign / SIEM configs.

See docs/BENCHMARKS.md for the performance envelope — roughly 140 k req/s S1F1, 79 k req/s S1F3 (32 SVIDs), 572 k S6F11/s push, ~450 bytes per PJ+CJ pair. Three orders of magnitude above typical fab tool load.

See docs/MES_INTEROP.md for the day-1 punch list to run against your commercial MES before promoting from staging to a real tool.

Operational runbook (starter)

Incident	First check	Mitigation
HSMS connection flapping	T7 / T6 timer fires in logs	check MES reachability, network MTU
Spool depth growing	host MES connectivity / ACK rate	force-drain via S6F23, escalate to MES
State machine "stuck"	last state-change handler log line	host-issued offline + re-establish
Alarm storm	AlarmRegistry::all() snapshot	check upstream sensor; quench via S5F3
Persistence dir growing unbounded	du -s + file count	sweep terminal-state records
Cross-tool inconsistency	secsgem_tests on canary tool	compare wire trace vs validator

---

Deferred follow-ups

• asio serial_port adapter for SECS-I. secsi::Protocol is tested end-to-end over secsi::TcpTransport; the matching serial driver isn't written yet. Mirror TcpTransport to add it.

---

Build details

The toolchain image (Dockerfile) is Ubuntu 24.04 with g++-13, CMake, Ninja, libasio-dev, libyaml-cpp-dev, and Python 3 for the codegen. doctest is fetched via CMake FetchContent. Build artifacts live in a named Docker volume so the host filesystem stays clean.

Standalone Asio is used in header-only mode (ASIO_STANDALONE). No Boost dependency.

---

Interop

Four independent external validators cross-check the codebase:

• secsgem-py 0.3.0 (Python reference impl) — three harnesses under interop/: secsgem-py active host driving the C++ passive server (31 checks), C++ active host probing secsgem-py's passive equipment, and a raw GEM 300 harness round-tripping S3 / S14 / S16 / S12 through hand-crafted SecsStreamFunction subclasses.
• secs4java8 (independent Java SECS implementation) — 55 cross-validation checks covering S1/S2/S3/S5/S6/S7/S10/S14/S16, full-body GEM 300 shapes, S2F49 enhanced commands, S5F13–F18 exception recovery.
• Wireshark / tshark HSMS dissector (independent network-protocol authors) — 69 HSMS frames dissected on a recorded pcap, no malformed-packet warnings.
• libFuzzer + ASan + UBSan — 200 000+ inputs through secs2::decode and 1.4 M+ through try_parse_sml per 60 s lane, 0 crashes.

See interop/README.md for harness-by-harness detail and docs/VERIFICATION.md for the test plan rationale.