# secs-gem A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully containerized. Every behavioural rule lives in YAML; the C++ is the engine that reads them. Implements **all of E4, E5, E30, E37 (SS + GS), E39, E40, E42, E84, E87, E90, E94, E116, E120, E148, E157**. > **License: proprietary — see [LICENSE](LICENSE).** No use, copy, > compile, evaluate, benchmark, or deploy without a written license > from the copyright holder. Contact `raphael@maenle.net` for > commercial licensing, evaluation terms, or fab deployment. --- ## Quick start Everything runs in Docker — no compiler or build tools on the host. ```bash docker compose run --rm builder # configure + compile docker compose run --rm tests # 473 cases / 3 087 assertions docker compose up --no-deps server client # live two-container demo ``` The two-container demo walks ~24 SECS transactions end-to-end through the data model. Watch the logs interleave. --- ## Integrating your tool (pick a tier) Three ways in, same engine underneath: 1. **Python, no SEMI knowledge** — run the `secs_gemd` daemon and `pip install` the pure-Python client in [clients/python](clients/python): ```python from secsgem_client import Equipment eq = Equipment("localhost:50051") eq.set(ChamberPressure=2.5) # variables: kwargs, not strings @eq.command # the function name IS the command, def START(cmd): # validated against the real equipment run_recipe(cmd.params.get("PPID")) # — so a typo fails at startup eq.fire(eq.names.event.ProcessStarted) # autocomplete + typo-safe eq.listen() ``` Names come from *your* `equipment.yaml`. `@eq.command` binds a handler by its function name; `eq.names.event.*` / `.alarm.*` / `.command.*` are autocomplete-able, typo-checked views fetched from the live daemon — so you rarely type a bare string. (The plain forms — `@eq.on("START")`, `eq.fire("ProcessStarted")` — still work.) A complete tool is ~25 lines: [clients/python/examples/mini_tool.py](clients/python/examples/mini_tool.py). 2. **Any language over gRPC** — `secs_gemd` exposes the name-based API in [proto/secsgem/v1/equipment.proto](proto/secsgem/v1/equipment.proto) (variables, events, alarms, control state, health stream, and the host-command stream with the SEMI-conformant HCACK-4 contract). The daemon owns the durable HSMS link: your tool software can restart without the fab host ever noticing. 3. **Embedded C++** — construct a `gem::EquipmentRuntime`, call the per-capability `register_*` functions (or `register_default_handlers` for all of GEM), and wire behaviour with `commands.set_handler`. `apps/secs_server.cpp` is the ~110-line canonical example. Status and remaining work for the daemon/client track: [docs/DAEMON_ROADMAP.md](docs/DAEMON_ROADMAP.md). --- ## Documentation map | File | What it covers | |------------------------------------------------------------|-------------------------------------------------------------------------| | [docs/](docs/00_index.md) | Guided-tour tutorial series — teach-from-zero across the protocol and the codebase | | [docs/COMPLIANCE.md](docs/COMPLIANCE.md) | Per-capability audit against every SEMI standard implemented | | [docs/INTEGRATION.md](docs/INTEGRATION.md) | Vendor-side tutorial: YAML → callbacks → production deploy | | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | How the pieces fit + how to extend (new store / FSM / message) | | [docs/PROOFS.md](docs/PROOFS.md) | The eight commands that prove the feature-completeness claim | | [docs/VERIFICATION.md](docs/VERIFICATION.md) | Test plan for the external validators behind the proof table | | [docs/BENCHMARKS.md](docs/BENCHMARKS.md) | Performance envelope (throughput, latency, memory) + how to re-run | | [docs/MES_INTEROP.md](docs/MES_INTEROP.md) | Day-1 punch list to run against your commercial MES (59 test IDs) | | [docs/SECURITY.md](docs/SECURITY.md) | Concrete configs: nftables, stunnel, minisign, SIEM audit-log schema | | [docs/GLOSSARY.md](docs/GLOSSARY.md) | SEMI vocabulary: SVID, CEID, PPID, ALCD, HCACK, T-timers, … | | [docs/FAQ.md](docs/FAQ.md) | Common questions and their canonical answers | | [examples/pvd_tool/](examples/pvd_tool/) | Worked example: a realistic fictional PVD tool, YAML + C++ wiring | | [LICENSE](LICENSE) | Proprietary license terms | --- ## Testing - **Unit + integration** — `docker compose run --rm tests` runs 473 cases / 3 087 assertions across every store, FSM, codec, parser, and persistence path. - **Live conformance harness** — 47 wire-level checks against the passive server. - **Interop** — round-trips against secsgem-py (31 checks), secs4java8 (55 checks), and Wireshark's HSMS dissector (69 frames, 0 malformed). - **Soak + fuzz** — 100 000-op property test; libFuzzer with ASan + UBSan over `secs2::decode` and the SML parser, 0 crashes. - **Daemon** — `secs_gemd_tests` exercises the gRPC service over real in-process channels (125 assertions), in Release and under ThreadSanitizer; `interop/daemon_interop.py` and `interop/pyclient_interop.py` prove the gRPC↔HSMS bridge and the published Python client against a live daemon with secsgem-py as host. - **One command for all of it** — `tools/run_interop.sh` runs every validation step (build, both unit suites, secsgem-py host, C++ conformance, Python client, daemon bridge, spool restart, tshark, secs4java8) with a PASS/FAIL summary. - **Config validation** — `secs_server --validate-config` rejects malformed YAML before startup. - **CI** — [Gitea Actions](.gitea/workflows/ci.yml) runs the full suite plus a `-fsanitize=thread` lane on every push to `main`; all 473 cases pass clean under TSan. Exact commands, exit codes, and per-standard test counts are in [docs/PROOFS.md](docs/PROOFS.md); the rationale behind the external validators is in [docs/VERIFICATION.md](docs/VERIFICATION.md). --- ## Architecture The project is **spec-as-data**: the SEMI behavioural rules live in YAML; the C++ is the engine that reads them. ``` ┌──────────────────────────────────────────────────────────────┐ │ data/ │ │ messages.yaml SECS-II message catalog (164 msgs) │ │ control_state.yaml E30 §6.2 control transition table │ │ process_job_state.yaml E40 §6 PJ transition table │ │ control_job_state.yaml E94 §6 CJ transition table │ │ equipment.yaml SVIDs / DVIDs / ECIDs / CEIDs / │ │ alarms / recipes / commands │ └──────────────────────┬───────────────────────────────────────┘ │ (codegen at build, YAML loaded at startup) ▼ ┌──────────────────────────────────────────────────────────────┐ │ apps/ │ │ secs_server passive equipment secs_bench perf │ │ secs_client active host secs_conformance │ │ secs_interop_probe │ └──────────────────────────────────────────────────────────────┘ secsgem::config loader.hpp + validate.hpp: YAML -> data model, with multi-error validator surfacing every issue at once (`--validate-config`) secsgem::gem per-standard FSM + per-store persistence (every store accepts v ∈ [1, kVersion] for forward-compatible schema migrations). EquipmentDataModel composes all stores. Router (stream, function) -> handler. Generated messages.hpp covers 164 SxFy. secsgem::hsms Connection (Asio): HSMS-SS + HSMS-GS, all T-timers enforced, auto S9F3/F5/F7/F9/F11. secsgem::secsi SECS-I Protocol FSM (E4): T1/T2/T3/T4 enforced in-FSM, TCP transport for tunnel testing. secsgem::secs2 Item (variant), encode/decode, Message, SML parser/printer. secsgem::metrics Prometheus exporter (Registry + HTTP server). ``` See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for how to extend it (new store / FSM / message). --- ## Adding a capability The point of "spec-as-data" is that adding behaviour almost never requires a C++ change. ### New SVID ```yaml # data/equipment.yaml svids: - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25} ``` ### New host command with side effects ```yaml host_commands: - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2} ``` ### New state transition ```yaml # data/control_state.yaml transitions: - {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept} ``` ### New SECS-II message ```yaml # data/messages.yaml - id: S6F30 stream: 6 function: 30 w: true builder: s6f30_something parser: parse_s6f30 body: kind: list struct_name: Something fields: - {name: field_a, shape: {kind: scalar, item_type: U4}} - {name: field_b, shape: {kind: scalar, item_type: ASCII}} ``` `docker compose run --rm builder` regenerates `messages.hpp`. The typed builder, parser, and struct definition appear automatically. Run `--validate-config` after every YAML edit. --- ## Production deployment See [docs/INTEGRATION.md](docs/INTEGRATION.md) for the full vendor-side tutorial — wiring sensors, plugging FSMs into the tool, persistence layout, monitoring/observability, HSMS-GS multi-MES setup. See [docs/SECURITY.md](docs/SECURITY.md) for concrete nftables / stunnel / minisign / SIEM configs. See [docs/BENCHMARKS.md](docs/BENCHMARKS.md) for the performance envelope — roughly **140 k req/s S1F1**, **79 k req/s S1F3 (32 SVIDs)**, **572 k S6F11/s push**, **~450 bytes per PJ+CJ pair**. Three orders of magnitude above typical fab tool load. See [docs/MES_INTEROP.md](docs/MES_INTEROP.md) for the day-1 punch list to run against your commercial MES before promoting from staging to a real tool. ### Operational runbook (starter) | Incident | First check | Mitigation | |-------------------------------------|--------------------------------------|-------------------------------------------| | HSMS connection flapping | T7 / T6 timer fires in logs | check MES reachability, network MTU | | Spool depth growing | host MES connectivity / ACK rate | force-drain via S6F23, escalate to MES | | State machine "stuck" | last state-change handler log line | host-issued offline + re-establish | | Alarm storm | `AlarmRegistry::all()` snapshot | check upstream sensor; quench via S5F3 | | Persistence dir growing unbounded | `du -s` + file count | sweep terminal-state records | | Cross-tool inconsistency | `secsgem_tests` on canary tool | compare wire trace vs validator | --- ## Deferred follow-ups - **asio `serial_port` adapter for SECS-I.** `secsi::Protocol` is tested end-to-end over `secsi::TcpTransport`; the matching serial driver isn't written yet. Mirror `TcpTransport` to add it. --- ## Build details The toolchain image (`Dockerfile`) is Ubuntu 24.04 with `g++-13`, CMake, Ninja, `libasio-dev`, `libyaml-cpp-dev`, and Python 3 for the codegen. doctest is fetched via CMake FetchContent. Build artifacts live in a named Docker volume so the host filesystem stays clean. Standalone Asio is used in header-only mode (`ASIO_STANDALONE`). No Boost dependency. --- ## Interop Four independent external validators cross-check the codebase: - **secsgem-py 0.3.0** (Python reference impl) — three harnesses under `interop/`: secsgem-py active host driving the C++ passive server (31 checks), C++ active host probing secsgem-py's passive equipment, and a raw GEM 300 harness round-tripping S3 / S14 / S16 / S12 through hand-crafted `SecsStreamFunction` subclasses. - **secs4java8** (independent Java SECS implementation) — 55 cross-validation checks covering S1/S2/S3/S5/S6/S7/S10/S14/S16, full-body GEM 300 shapes, S2F49 enhanced commands, S5F13–F18 exception recovery. - **Wireshark / tshark HSMS dissector** (independent network-protocol authors) — 69 HSMS frames dissected on a recorded pcap, no malformed-packet warnings. - **libFuzzer + ASan + UBSan** — 200 000+ inputs through `secs2::decode` and 1.4 M+ through `try_parse_sml` per 60 s lane, 0 crashes. See [`interop/README.md`](interop/README.md) for harness-by-harness detail and [docs/VERIFICATION.md](docs/VERIFICATION.md) for the test plan rationale.