secs-gem/README.md

secs-gem

A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully containerized. Every behavioural rule lives in YAML; the C++ is the engine that reads them. Implements all of E4, E5, E30, E37 (SS + GS), E39, E40, E42, E84, E87, E90, E94, E116, E120, E148, E157.

License: proprietary — see LICENSE. No use, copy, compile, evaluate, benchmark, or deploy without a written license from the copyright holder. Contact raphael@maenle.net for commercial licensing, evaluation terms, or fab deployment.

---

Proof of feature-completeness

"Feature-complete" is a claim that the code must prove, not the README. These five commands are the proof. If they all exit zero on a fresh clone, the codebase implements what COMPLIANCE.md claims.

#	Command	What it proves
1	docker compose run --rm tests	426 test cases / 2 557 assertions pass: every store, FSM, codec, parser, persistence path
2	docker compose run --rm builder /app/build/secs_conformance --host server --port 5000	47 wire-level conformance checks PASS against a live passive equipment
3	docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py --host server	24 interop checks PASS against secsgem-py 0.3.0 (the Python reference impl)
4	SECSGEM_ROBUSTNESS_SOAK=1 docker compose run --rm builder /app/build/secsgem_tests -tc='*soak*'	100 000 random tool operations execute with all invariants and persistence round-trips holding
5	docker compose run --rm builder /app/build/secs_server --validate-config --config /app/data/equipment.yaml --state-table /app/data/control_state.yaml --pj-state-table /app/data/process_job_state.yaml --cj-state-table /app/data/control_job_state.yaml	Every shipped YAML config passes structural + referential validation
6	docker compose run --rm builder bash /app/interop/tshark_validate.sh	69 HSMS frames dissected by Wireshark's HSMS dissector (independent third codec) with no malformed packets
7	bash interop/secs4j_validate.sh	55 cross-validation checks PASS against secs4java8 (independent Java implementation), covering S1/S2/S3/S5/S6/S7/S10/S14/S16, the full E40 PJ body, dynamic event reports + unsolicited S6F11 / S5F1 observation, alarm management, spool, PP management, terminal services, limits, trace, E39, and the GEM 300 streams secsgem-py couldn't easily drive
8	cmake -B build-fuzz -DSECSGEM_FUZZ=ON && build-fuzz/fuzz_secs2_decode -max_total_time=60	~70 000 random inputs through secs2::decode, ~285 000 through try_parse_sml, ASan + UBSan + libFuzzer coverage, 0 crashes

Plus, on every push to main, Gitea Actions runs both a Release build + full test suite and a separate ThreadSanitizer lane that builds with -fsanitize=thread and fails on any race. All 426 cases / 2 557 assertions pass under TSan clean.

Per-standard test coverage

Every claimed standard has dedicated tests. Counts are grep -c TEST_CASE; cross-cutting tests (e.g. test_robustness_fuzz, test_gem300_scenario) exercise multiple standards in concert.

Standard	Test files	Cases
E5 — SECS-II encoding	test_secs2, test_sml, test_messages, test_identifier_wildcards, test_fuzz	120
E5 §13 — exceptions	test_exceptions, test_exception_persistence	16
E4 — SECS-I transport	test_secsi, test_secsi_timers, test_secsi_tcp	27
E37 — HSMS (SS + GS)	test_hsms, test_hsms_connection, test_hsms_timers, test_hsms_s9, test_hsms_gs, test_hsms_gs_integration, test_s9_fallback, test_concurrency	34
E30 — GEM core	test_control_state, test_communication_state, test_host_handler, test_data_model, test_loader, test_config_validate	71
E40 — process jobs	test_process_jobs	21
E94 — control jobs	test_control_jobs	9
E42 — formatted PP	test_e42_formatted_pp	6
E87 — carriers + load ports	test_carriers, test_carrier_state, test_carrier_persistence, test_e87_wire_scenarios	27
E90 — substrate tracking	test_substrates, test_substrate_persistence	21
E116 — EPT	test_ept	7
E120 / E39 — common equip / object service	test_cem_objects	3
E157 — module process tracking	test_modules	5
E84 — parallel I/O + timers	test_e84, test_e84_ports, test_e84_timers, test_e84_asio_timers	27
Persistence + cross-cutting	test_job_persistence, test_persistence_upgrade, test_wire_ceid_emission, test_gem300_scenario, test_live_gem300, test_thread_safety, test_metrics_prometheus, test_robustness_fuzz	32
Total		426

A single command to see this live: docker compose run --rm builder /app/build/secsgem_tests --list-test-cases | wc -l (currently 426).

---

Quick start

Everything runs in Docker — no compiler or build tools on the host.

docker compose run --rm builder         # configure + compile
docker compose run --rm tests           # 426 cases / 2 557 assertions
docker compose up --no-deps server client   # live two-container demo

The two-container demo walks ~24 SECS transactions end-to-end through the data model. Watch the logs interleave.

---

Documentation map

File	What it covers
COMPLIANCE.md	Per-capability audit against every SEMI standard implemented
INTEGRATION.md	Vendor-side tutorial: YAML → callbacks → production deploy
ARCHITECTURE.md	How the pieces fit + how to extend (new store / FSM / message)
VERIFICATION.md	Test plan for the external validators behind the proof table
BENCHMARKS.md	Performance envelope (throughput, latency, memory) + how to re-run
MES_INTEROP.md	Day-1 punch list to run against your commercial MES (60+ test IDs)
SECURITY.md	Concrete configs: nftables, stunnel, minisign, SIEM audit-log schema
GLOSSARY.md	SEMI vocabulary: SVID, CEID, PPID, ALCD, HCACK, T-timers, …
FAQ.md	Common questions and their canonical answers
examples/pvd_tool/	Worked example: a realistic fictional PVD tool, YAML + C++ wiring
LICENSE	Proprietary license terms

---

Architecture

The project is spec-as-data: the SEMI behavioural rules live in YAML; the C++ is the engine that reads them.

   ┌──────────────────────────────────────────────────────────────┐
   │ data/                                                        │
   │   messages.yaml          SECS-II message catalog (164 msgs)  │
   │   control_state.yaml     E30 §6.2 control transition table   │
   │   process_job_state.yaml E40 §6 PJ transition table          │
   │   control_job_state.yaml E94 §6 CJ transition table          │
   │   equipment.yaml         SVIDs / DVIDs / ECIDs / CEIDs /     │
   │                          alarms / recipes / commands         │
   └──────────────────────┬───────────────────────────────────────┘
                          │  (codegen at build, YAML loaded at startup)
                          ▼
   ┌──────────────────────────────────────────────────────────────┐
   │ apps/                                                        │
   │   secs_server   passive equipment      secs_bench   perf     │
   │   secs_client   active host            secs_conformance      │
   │   secs_interop_probe                                         │
   └──────────────────────────────────────────────────────────────┘

   secsgem::config   loader.hpp + validate.hpp:
                     YAML -> data model, with multi-error validator
                     surfacing every issue at once (`--validate-config`)
   secsgem::gem      per-standard FSM + per-store persistence
                     (every store accepts v ∈ [1, kVersion] for
                     forward-compatible schema migrations).
                     EquipmentDataModel composes all stores.
                     Router (stream, function) -> handler.
                     Generated messages.hpp covers 164 SxFy.
   secsgem::hsms     Connection (Asio): HSMS-SS + HSMS-GS, all
                     T-timers enforced, auto S9F3/F5/F7/F9/F11.
   secsgem::secsi    SECS-I Protocol FSM (E4): T1/T2/T3/T4 enforced
                     in-FSM, TCP transport for tunnel testing.
   secsgem::secs2    Item (variant), encode/decode, Message,
                     SML parser/printer.
   secsgem::metrics  Prometheus exporter (Registry + HTTP server).

---

Adding a capability

The point of "spec-as-data" is that adding behaviour almost never requires a C++ change.

New SVID

# data/equipment.yaml
svids:
  - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}

New host command with side effects

host_commands:
  - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}

New state transition

# data/control_state.yaml
transitions:
  - {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept}

New SECS-II message

# data/messages.yaml
- id: S6F30
  stream: 6
  function: 30
  w: true
  builder: s6f30_something
  parser: parse_s6f30
  body:
    kind: list
    struct_name: Something
    fields:
      - {name: field_a, shape: {kind: scalar, item_type: U4}}
      - {name: field_b, shape: {kind: scalar, item_type: ASCII}}

docker compose run --rm builder regenerates messages.hpp. The typed builder, parser, and struct definition appear automatically. Run --validate-config after every YAML edit.

---

Production deployment

See INTEGRATION.md for the full vendor-side tutorial — wiring sensors, plugging FSMs into the tool, persistence layout, monitoring/observability, HSMS-GS multi-MES setup.

See SECURITY.md for concrete nftables / stunnel / minisign / SIEM configs.

See BENCHMARKS.md for the performance envelope — roughly 140 k req/s S1F1, 79 k req/s S1F3 (32 SVIDs), 572 k S6F11/s push, ~450 bytes per PJ+CJ pair. Three orders of magnitude above typical fab tool load.

See MES_INTEROP.md for the day-1 punch list to run against your commercial MES before promoting from staging to a real tool.

Operational runbook (starter)

Incident	First check	Mitigation
HSMS connection flapping	T7 / T6 timer fires in logs	check MES reachability, network MTU
Spool depth growing	host MES connectivity / ACK rate	force-drain via S6F23, escalate to MES
State machine "stuck"	last state-change handler log line	host-issued offline + re-establish
Alarm storm	AlarmRegistry::all() snapshot	check upstream sensor; quench via S5F3
Persistence dir growing unbounded	du -s + file count	sweep terminal-state records
Cross-tool inconsistency	secsgem_tests on canary tool	compare wire trace vs validator

---

Build details

The toolchain image (Dockerfile) is Ubuntu 24.04 with g++-13, CMake, Ninja, libasio-dev, libyaml-cpp-dev, and Python 3 for the codegen. doctest is fetched via CMake FetchContent. Build artifacts live in a named Docker volume so the host filesystem stays clean.

Standalone Asio is used in header-only mode (ASIO_STANDALONE). No Boost dependency.

ThreadSanitizer

cmake -S . -B build-tsan -G Ninja -DCMAKE_BUILD_TYPE=Debug -DSECSGEM_TSAN=ON
cmake --build build-tsan
TSAN_OPTIONS=halt_on_error=1 build-tsan/secsgem_tests

Runs as a separate lane in CI. Catches data races in the io_context strand contract documented in INTEGRATION.md §3.

---

Interop

interop/ contains the secsgem-py 0.3.0 cross-validation harness — secsgem-py active host driving our C++ passive server, our C++ active host probing secsgem-py's passive equipment, and a raw GEM-300 harness that round-trips S3 (E87), S14 (E94), S16 (E40), S12 (wafer maps) through hand-crafted SecsStreamFunction subclasses. See interop/README.md.