# secs-gem A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully containerized, with every behavioural rule encoded as YAML data (control state, equipment data dictionary, E40 process-job state machine, E94 control-job state machine, SECS-II message shapes). Implements **all of E5, E30, E37 (SS + GS), E4 SECS-I, E40, E42, E84, E87, E90, E94, E116, E120, E148, E157, E39**. Per-store persistence on every mutable in-memory entity (spool, carriers, load-ports, substrates, process-jobs, control-jobs, exceptions). See **[COMPLIANCE.md](COMPLIANCE.md)** for the per-capability audit, **[INTEGRATION.md](INTEGRATION.md)** for the vendor-side tutorial, **[BENCHMARKS.md](BENCHMARKS.md)** for the performance envelope, and **[MES_INTEROP.md](MES_INTEROP.md)** for the day-1 punch list to run against your commercial MES. > **License: proprietary — see [LICENSE](LICENSE).** No use, copy, > compile, evaluate, benchmark, or deploy without a written license > from the copyright holder. Contact `raphael@maenle.net` for > commercial licensing, evaluation terms, or fab deployment. ## Quick start Everything runs in Docker — no compiler or build tools on the host. ```bash docker compose run --rm builder # configure + compile docker compose run --rm tests # 384 cases / 2390 assertions docker compose up --no-deps server client # live two-container demo ``` ## Architecture The project is "spec-as-data": the SEMI behavioural rules live in YAML; the C++ is the engine that reads them. ``` ┌──────────────────────────────────────────────────────────────┐ │ data/ │ │ messages.yaml SECS-II message catalog │ │ control_state.yaml E30 §6.2 control transition table │ │ process_job_state.yaml E40 §6 PJ transition table │ │ control_job_state.yaml E94 §6 CJ transition table │ │ equipment.yaml SVIDs / DVIDs / ECIDs / CEIDs / │ │ alarms / recipes / commands │ └──────────────────────┬───────────────────────────────────────┘ │ (loaded at startup, codegen at build) ▼ ┌──────────────────────────────────────────────────────────────┐ │ tools/gen_messages.py │ │ reads messages.yaml -> emits generated/secsgem/gem/messages.hpp └──────────────────────┬───────────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────────┐ │ apps/ │ │ secs_server.cpp passive equipment │ │ secs_client.cpp active host │ │ (both use gem::Router for dispatch) │ └──────────────────────────────────────────────────────────────┘ secsgem::config loader.hpp: YAML -> tables + data model secsgem::gem every per-standard FSM (E30, E40, E84, E87, E90, E94, E116, E120, E148, E157, E39, E5 exceptions), each per-store-persistable. EquipmentDataModel composes all stores. Router (stream, function) -> handler. Generated messages.hpp covers 164 SxFy. secsgem::hsms Connection (Asio): HSMS-SS + HSMS-GS, all T-timers enforced, auto S9F3/F5/F7/F9/F11. secsgem::secsi SECS-I Protocol FSM (E4): T1/T2/T3/T4 enforced in-FSM, TCP transport for tunnel testing. secsgem::secs2 Item (variant), encode/decode, Message, SML parser/printer. ``` ### Tree ``` secs-gem/ ├── Dockerfile, docker-compose.yml # toolchain + demo ├── CMakeLists.txt ├── README.md ├── COMPLIANCE.md # per-capability audit ├── INTEGRATION.md # vendor integration tutorial ├── data/ │ ├── messages.yaml # SECS-II message catalog (164 msgs) │ ├── control_state.yaml # E30 control state transitions │ ├── process_job_state.yaml # E40 PJ transitions │ ├── control_job_state.yaml # E94 CJ transitions │ └── equipment.yaml # equipment data dictionary ├── tools/ │ └── gen_messages.py # codegen (messages.yaml -> .hpp) ├── include/secsgem/ │ ├── secs2/{item,codec,sml,message}.hpp │ ├── hsms/{header,connection}.hpp │ ├── secsi/{header,block,protocol,tcp_transport}.hpp │ ├── gem/ # FSMs per SEMI standard │ ├── gem/store/ # one file per focused store │ ├── config/loader.hpp │ └── endpoint.hpp ├── src/{secs2,hsms,secsi,gem,config}/*.cpp ├── apps/ │ ├── secs_server.cpp # passive equipment demo │ ├── secs_client.cpp # active host demo │ └── secs_interop_probe.cpp # cross-test against secsgem-py ├── interop/ # secsgem-py 0.3.0 cross-validation └── tests/test_*.cpp # 384 cases / 2390 assertions ``` ## Adding a capability The point of "spec-as-data" is that adding behaviour almost never requires a C++ change. ### New SVID ```yaml # data/equipment.yaml svids: - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25} ``` ### New host command with side effects ```yaml host_commands: - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2} ``` ### New state transition ```yaml # data/control_state.yaml transitions: - {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept} ``` ### New SECS-II message ```yaml # data/messages.yaml - id: S6F30 stream: 6 function: 30 w: true builder: s6f30_something parser: parse_s6f30 body: kind: list struct_name: Something fields: - {name: field_a, shape: {kind: scalar, item_type: U4}} - {name: field_b, shape: {kind: scalar, item_type: ASCII}} ``` `docker compose run --rm builder` regenerates `messages.hpp`. The typed builder, parser, and struct definition appear automatically. --- # Production / fab deployment The library is a runtime stack. Shipping it on a real tool involves more than building the binary. This section enumerates the work that sits between "tests pass" and "this is running on the fab floor." ## 1. Persistence directory layout Enable persistence per store at startup, before the connection comes up. Pattern (the call sites are equivalent on every store): ```cpp auto base = std::filesystem::path("/var/lib/acme-secsgem"); model->spool.enable_persistence(base / "spool"); model->carriers.enable_persistence(base / "carriers"); model->load_ports.enable_persistence(base / "loadports"); model->substrates.enable_persistence(base / "substrates"); model->process_jobs.enable_persistence(base / "pjobs"); model->control_jobs.enable_persistence(base / "cjobs"); model->exceptions.enable_persistence(base / "exceptions"); ``` Storage rules: - **Mount this volume on the same physical disk as the binary** — network filesystems (NFS) can introduce latency that interferes with the rename-based atomic write pattern. - **Back this volume up daily**. Journal files are small (a few hundred bytes each) and rsync-friendly. - **Set sane retention**. Cleared exceptions and dequeued PJs are removed automatically; complete carriers / substrates / CJs are the application's responsibility to sweep. Cap by file count (a million files in one directory is fine on ext4 / xfs; less on others). - **Disk space**: budget 100 MB for a busy fab tool over a year (~500 K transitions, ~200 bytes each). In practice it's far less because terminal-state records are removed. After a crash, the next process start replays every store and is back to the prior in-memory state before the HSMS port opens. ## 2. Security HSMS over plain TCP is the spec's wire protocol. The library ships unencrypted by design — that's what equipment manufacturers expect. In a real fab: - **Network isolation**: HSMS must run on a control LAN, never exposed to engineering / corporate networks. Default the `--port` to 5000 / 5005 on a dedicated VLAN behind firewall ACLs that whitelist your MES host's IP. - **TLS tunnel**: for cross-site HSMS (rare but real for multi-fab shared hosts), tunnel the TCP through stunnel or a sidecar proxy. Don't modify the HSMS wire protocol; wrap the socket. - **Authentication**: HSMS doesn't include peer auth. Rely on network-layer mTLS (sidecar proxy) and per-tool firewall rules. - **Audit logging**: enable `Connection::set_log_handler` and ship to a SIEM. Every SECS-II message in/out should be retrievable for a configurable retention window — many fabs require 90 days. - **YAML config integrity**: sign your config bundles (`equipment.yaml`, `control_state.yaml`, etc.) and verify the signature on load. Misconfiguration is one of the top root-causes of GEM-related fab incidents. ## 3. Monitoring and observability The library exposes hooks at every layer. Wire them to whatever your fab already runs. | Signal | Hook | Why it matters | | ---------------------------- | ------------------------------------------ | -------------------------------------------------- | | HSMS connection lifecycle | `Connection::set_log_handler`, `set_selected_handler`, `set_closed_handler` | reconnect storms, unexpected separates | | T3 / T6 / T7 / T8 timer fires | `set_closed_handler` reason starts with "T*" | host MES unreachable, fab network event | | Auto S9F* emission | `set_log_handler` line containing "-> S9F" | malformed peer traffic, schema drift | | Spool depth | `model->spool.size()` | host MES backpressure / outage | | FSM transitions (every store) | `set_*_change_handler` | tool state, throughput, anomaly detection | | Persistence directory size | `du -s var/lib/acme-secsgem` | journal growth, untracked terminal-state records | Recommended metrics export pattern: aggregate into Prometheus via a sidecar that polls the data model. Per-CEID emission rates, alarm set/clear rates, T-timer expiry counts, and spool depth form a reasonable starter dashboard. **Hooks fire on the io_context thread.** Every `set_*_change_handler` callback the library invokes runs on the connection's io_context (there are no locks anywhere in `EquipmentDataModel`). Metrics exporters and log shippers wired into those callbacks must either be thread-safe themselves or hand the work off (a lock-free queue, a separate exporter thread polling published counters, `asio::post` onto another executor). Doing blocking I/O from inside a handler stalls the dispatcher — keep handlers cheap. See INTEGRATION.md §3 for the cross-thread update pattern. ## 4. High availability The library is single-threaded per HSMS connection — that's how HSMS works. For HA: - **Run two equipment processes** in active/standby on the same tool, sharing the persistence volume. Only the active accepts the HSMS port; the standby tails the journal. Failover is filesystem-locked. - **Reconnect on the host side**: an MES-side disconnect should trigger T5-based reconnect. Configure `Timers::t5` to your MES's policy (default 10s). - **Graceful shutdown**: SIGTERM should flush the write queue, call `conn->separate()`, and exit cleanly so the journal is point-consistent. The provided `apps/secs_server.cpp` shows the pattern. ## 5. Deployment patterns Three common shapes: ### Docker / podman on a tool PC ```dockerfile FROM ubuntu:24.04 COPY build/secs_server /usr/local/bin/ COPY etc/ /etc/acme-secsgem/ VOLUME /var/lib/acme-secsgem EXPOSE 5000 ENTRYPOINT ["/usr/local/bin/secs_server", \ "--port", "5000", \ "--config", "/etc/acme-secsgem/equipment.yaml", \ "--state-table", "/etc/acme-secsgem/control_state.yaml", \ "--spool-dir", "/var/lib/acme-secsgem/spool"] ``` ### systemd unit ```ini [Unit] Description=ACME SECS/GEM equipment After=network.target [Service] Type=simple User=secsgem Group=secsgem ExecStart=/usr/local/bin/secs_server --port 5000 \ --config /etc/acme-secsgem/equipment.yaml \ --state-table /etc/acme-secsgem/control_state.yaml \ --spool-dir /var/lib/acme-secsgem/spool Restart=always RestartSec=5 LimitNOFILE=8192 [Install] WantedBy=multi-user.target ``` ### Kubernetes (multi-tool cell controller) Run one Pod per tool with the persistence volume mounted from local-storage (not NFS). The Service exposes the HSMS port on the control LAN. Use a PodDisruptionBudget to ensure the standby is available during rolling updates. ## 6. Upgrade path YAML edits don't require a rebuild — restart the process and the new dictionary loads. Code changes do require rebuild + restart. - **Zero-downtime for YAML**: if you're using the active/standby HA pattern, edit YAML on the standby, restart the standby, promote. - **Code upgrades**: deploy to a canary tool first; bake-test for at least a full wafer batch before fleet-wide rollout. - **Schema migrations**: persistence records carry a 1-byte version stamp after the magic byte. `ProcessJobStore` and `SubstrateStore` currently implement multi-version reads: code at kVersion=2 still loads v1 records (the v2 trailer fields default to empty). The remaining stores (`ControlJobStore`, `CarrierStore`, `LoadPortStore`, `ExceptionStore`, `SpoolStore`) use strict version equality — a future kVersion bump there requires adding a parser for the prior version at the same time, otherwise replay will reject old records. Tests in `tests/test_persistence_upgrade.cpp` lock down both contracts and act as a tripwire if a kVersion bumps silently. Always test the upgrade with a real on-disk journal before fleet rollout. ## 7. Integration with the fab stack | Other system | How this library talks to it | | ------------------- | --------------------------------------------------------------------- | | MES (Camstar, Mozaic, Camstar) | HSMS-SS over TCP (`secs_server` listens on a port the MES is configured to connect to) | | Multi-MES (HSMS-GS) | `Connection::add_session(device_id)` registers extra sessions on one TCP socket | | AMHS / OHT | E84 per-port FSMs (`E84PortStore::on_signal_change(port, signal, value)`); wire to your I/O bridge | | Recipe engine | RecipeStore.add (opaque) + RecipeStore.add_formatted (E42 structured) | | Alarm sources | `AlarmRegistry::set(alid, active)` from your sensor poll | | Carrier scanner | `CarrierStore::create / fire_id_event / set_slot_state` | | Wafer tracker | `SubstrateStore::create / fire_*_event` | | EPT / shift report | `EptStateMachine::accumulated(state)` reads the time-bucket counters | ## 8. Compliance and certification - Fork `COMPLIANCE.md` and prune it to *your* tool's claimed coverage; ship that copy with the tool. - Run the in-repo conformance harness against your tool: ``` build/secs_conformance --host --port 5000 --device 0 ``` Exits 0 with a per-check PASS / FAIL summary covering every E30 fundamental capability (establish comms, on-line ID, status data, equipment constants, clock, alarms, PP management, documentation). Adapt `apps/secs_conformance.cpp` to add your tool's capability-specific checks. - Run an independent third-party validator (GEM RTS or equivalent) against your specific tool — a passing library + in-repo harness is necessary but not sufficient for certification. - Capture wire traces from every validator run; archive for audit. ## 9. Testing in production - **Canary**: deploy to one or two tools per fab before fleet rollout. - **Synthetic transactions**: a heartbeat that issues S1F1 every 60s and alerts on T3 timeout. Catches MES-side outages before a real recipe does. - **Shadow traffic**: for upgrades, run the new version listening on a second port; have MES dual-connect; diff the responses. ## 10. Operational runbook (starting point) Common production incidents and the right response: | Incident | First check | Mitigation | | ----------------------------------- | ------------------------------------ | ----------------------------------------- | | HSMS connection flapping | T7 / T6 timer fires in logs | check MES reachability, network MTU | | Spool depth growing | host MES connectivity / ACK rate | force-drain via S6F23, escalate to MES | | State machine "stuck" | last state-change handler log line | host-issued offline + re-establish | | Alarm storm | AlarmRegistry `all()` snapshot | check upstream sensor; quench via S5F3 | | Persistence dir growing unbounded | `du -s` + file count | sweep terminal-state records | | Cross-tool inconsistency | `secsgem_tests` on canary tool | compare wire trace vs validator | --- ## Demo The two-container demo walks ~24 SECS transactions end-to-end through the data model. Run `docker compose up --no-deps server client` and watch the logs interleave. ## Build details The toolchain image (`Dockerfile`) is Ubuntu 24.04 with `g++-13`, CMake, Ninja, `libasio-dev`, `libyaml-cpp-dev`, and Python 3 for the codegen. doctest is fetched via CMake FetchContent. Build artifacts live in a named Docker volume so the host filesystem stays clean. Standalone Asio is used in header-only mode (`ASIO_STANDALONE`). No Boost dependency. ## Interop `interop/` contains the secsgem-py 0.3.0 cross-validation harness — secsgem-py active host driving our C++ passive server, our C++ active host probing secsgem-py's passive equipment, and a raw GEM-300 harness that round-trips S3 (E87), S14 (E94), S16 (E40), S12 (wafer maps) through hand-crafted `SecsStreamFunction` subclasses. See `interop/README.md`.