diff --git a/README.md b/README.md index ed2f896..bcfd136 100644 --- a/README.md +++ b/README.md @@ -1,68 +1,127 @@ # secs-gem -A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully containerized, -with every behavioural rule encoded as YAML data (control state, equipment -data dictionary, E40 process-job state machine, E94 control-job state -machine, SECS-II message shapes). - -Implements **all of E5, E30, E37 (SS + GS), E4 SECS-I, E40, E42, E84, E87, -E90, E94, E116, E120, E148, E157, E39**. Per-store persistence on every -mutable in-memory entity (spool, carriers, load-ports, substrates, -process-jobs, control-jobs, exceptions). See **[COMPLIANCE.md](COMPLIANCE.md)** -for the per-capability audit, **[INTEGRATION.md](INTEGRATION.md)** for -the vendor-side tutorial, **[BENCHMARKS.md](BENCHMARKS.md)** for the -performance envelope, and **[MES_INTEROP.md](MES_INTEROP.md)** for the -day-1 punch list to run against your commercial MES. +A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully +containerized. Every behavioural rule lives in YAML; the C++ is the +engine that reads them. Implements **all of E4, E5, E30, E37 +(SS + GS), E39, E40, E42, E84, E87, E90, E94, E116, E120, E148, +E157**. > **License: proprietary — see [LICENSE](LICENSE).** No use, copy, > compile, evaluate, benchmark, or deploy without a written license > from the copyright holder. Contact `raphael@maenle.net` for > commercial licensing, evaluation terms, or fab deployment. +--- + +## Proof of feature-completeness + +"Feature-complete" is a claim that the code must prove, not the +README. These five commands are the proof. If they all exit zero on +a fresh clone, the codebase implements what +[COMPLIANCE.md](COMPLIANCE.md) claims. + +| # | Command | What it proves | +|---|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------| +| 1 | `docker compose run --rm tests` | **426 test cases / 2 557 assertions** pass: every store, FSM, codec, parser, persistence path | +| 2 | `docker compose run --rm builder /app/build/secs_conformance --host server --port 5000` | **47 wire-level conformance checks** PASS against a live passive equipment | +| 3 | `docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py --host server` | **24 interop checks** PASS against secsgem-py 0.3.0 (the Python reference impl) | +| 4 | `SECSGEM_ROBUSTNESS_SOAK=1 docker compose run --rm builder /app/build/secsgem_tests -tc='*soak*'` | **100 000 random tool operations** execute with all invariants and persistence round-trips holding | +| 5 | `docker compose run --rm builder /app/build/secs_server --validate-config --config /app/data/equipment.yaml --state-table /app/data/control_state.yaml --pj-state-table /app/data/process_job_state.yaml --cj-state-table /app/data/control_job_state.yaml` | Every shipped YAML config passes structural + referential validation | + +Plus, on every push to `main`, [Gitea Actions](.gitea/workflows/ci.yml) +runs both a **Release build + full test suite** and a separate +**ThreadSanitizer lane** that builds with `-fsanitize=thread` and +fails on any race. All 426 cases / 2 557 assertions pass under TSan +clean. + +### Per-standard test coverage + +Every claimed standard has dedicated tests. Counts are +`grep -c TEST_CASE`; cross-cutting tests (e.g. `test_robustness_fuzz`, +`test_gem300_scenario`) exercise multiple standards in concert. + +| Standard | Test files | Cases | +|-----------------------------------|-------------------------------------------------------------------------------------------|------:| +| **E5** — SECS-II encoding | `test_secs2`, `test_sml`, `test_messages`, `test_identifier_wildcards`, `test_fuzz` | 120 | +| **E5 §13** — exceptions | `test_exceptions`, `test_exception_persistence` | 16 | +| **E4** — SECS-I transport | `test_secsi`, `test_secsi_timers`, `test_secsi_tcp` | 27 | +| **E37** — HSMS (SS + GS) | `test_hsms`, `test_hsms_connection`, `test_hsms_timers`, `test_hsms_s9`, `test_hsms_gs`, `test_hsms_gs_integration`, `test_s9_fallback`, `test_concurrency` | 34 | +| **E30** — GEM core | `test_control_state`, `test_communication_state`, `test_host_handler`, `test_data_model`, `test_loader`, `test_config_validate` | 71 | +| **E40** — process jobs | `test_process_jobs` | 21 | +| **E94** — control jobs | `test_control_jobs` | 9 | +| **E42** — formatted PP | `test_e42_formatted_pp` | 6 | +| **E87** — carriers + load ports | `test_carriers`, `test_carrier_state`, `test_carrier_persistence`, `test_e87_wire_scenarios` | 27 | +| **E90** — substrate tracking | `test_substrates`, `test_substrate_persistence` | 21 | +| **E116** — EPT | `test_ept` | 7 | +| **E120 / E39** — common equip / object service | `test_cem_objects` | 3 | +| **E157** — module process tracking | `test_modules` | 5 | +| **E84** — parallel I/O + timers | `test_e84`, `test_e84_ports`, `test_e84_timers`, `test_e84_asio_timers` | 27 | +| Persistence + cross-cutting | `test_job_persistence`, `test_persistence_upgrade`, `test_wire_ceid_emission`, `test_gem300_scenario`, `test_live_gem300`, `test_thread_safety`, `test_metrics_prometheus`, `test_robustness_fuzz` | 32 | +| **Total** | | **426** | + +A single command to see this live: `docker compose run --rm builder +/app/build/secsgem_tests --list-test-cases | wc -l` (currently 426). + +--- + ## Quick start Everything runs in Docker — no compiler or build tools on the host. ```bash docker compose run --rm builder # configure + compile -docker compose run --rm tests # 384 cases / 2390 assertions +docker compose run --rm tests # 426 cases / 2 557 assertions docker compose up --no-deps server client # live two-container demo ``` +The two-container demo walks ~24 SECS transactions end-to-end +through the data model. Watch the logs interleave. + +--- + +## Documentation map + +| File | What it covers | +|-----------------------------------------------|-------------------------------------------------------------------------| +| [COMPLIANCE.md](COMPLIANCE.md) | Per-capability audit against every SEMI standard implemented | +| [INTEGRATION.md](INTEGRATION.md) | Vendor-side tutorial: YAML → callbacks → production deploy | +| [BENCHMARKS.md](BENCHMARKS.md) | Performance envelope (throughput, latency, memory) + how to re-run | +| [MES_INTEROP.md](MES_INTEROP.md) | Day-1 punch list to run against your commercial MES (60+ test IDs) | +| [SECURITY.md](SECURITY.md) | Concrete configs: nftables, stunnel, minisign, SIEM audit-log schema | +| [LICENSE](LICENSE) | Proprietary license terms | + +--- + ## Architecture -The project is "spec-as-data": the SEMI behavioural rules live in YAML; -the C++ is the engine that reads them. +The project is **spec-as-data**: the SEMI behavioural rules live in +YAML; the C++ is the engine that reads them. ``` ┌──────────────────────────────────────────────────────────────┐ │ data/ │ - │ messages.yaml SECS-II message catalog │ + │ messages.yaml SECS-II message catalog (164 msgs) │ │ control_state.yaml E30 §6.2 control transition table │ │ process_job_state.yaml E40 §6 PJ transition table │ │ control_job_state.yaml E94 §6 CJ transition table │ │ equipment.yaml SVIDs / DVIDs / ECIDs / CEIDs / │ │ alarms / recipes / commands │ └──────────────────────┬───────────────────────────────────────┘ - │ (loaded at startup, codegen at build) - ▼ - ┌──────────────────────────────────────────────────────────────┐ - │ tools/gen_messages.py │ - │ reads messages.yaml -> emits generated/secsgem/gem/messages.hpp - └──────────────────────┬───────────────────────────────────────┘ - │ + │ (codegen at build, YAML loaded at startup) ▼ ┌──────────────────────────────────────────────────────────────┐ │ apps/ │ - │ secs_server.cpp passive equipment │ - │ secs_client.cpp active host │ - │ (both use gem::Router for dispatch) │ + │ secs_server passive equipment secs_bench perf │ + │ secs_client active host secs_conformance │ + │ secs_interop_probe │ └──────────────────────────────────────────────────────────────┘ - secsgem::config loader.hpp: YAML -> tables + data model - secsgem::gem every per-standard FSM (E30, E40, E84, E87, - E90, E94, E116, E120, E148, E157, E39, E5 - exceptions), each per-store-persistable. + secsgem::config loader.hpp + validate.hpp: + YAML -> data model, with multi-error validator + surfacing every issue at once (`--validate-config`) + secsgem::gem per-standard FSM + per-store persistence + (every store accepts v ∈ [1, kVersion] for + forward-compatible schema migrations). EquipmentDataModel composes all stores. Router (stream, function) -> handler. Generated messages.hpp covers 164 SxFy. @@ -72,41 +131,10 @@ the C++ is the engine that reads them. in-FSM, TCP transport for tunnel testing. secsgem::secs2 Item (variant), encode/decode, Message, SML parser/printer. + secsgem::metrics Prometheus exporter (Registry + HTTP server). ``` -### Tree - -``` -secs-gem/ -├── Dockerfile, docker-compose.yml # toolchain + demo -├── CMakeLists.txt -├── README.md -├── COMPLIANCE.md # per-capability audit -├── INTEGRATION.md # vendor integration tutorial -├── data/ -│ ├── messages.yaml # SECS-II message catalog (164 msgs) -│ ├── control_state.yaml # E30 control state transitions -│ ├── process_job_state.yaml # E40 PJ transitions -│ ├── control_job_state.yaml # E94 CJ transitions -│ └── equipment.yaml # equipment data dictionary -├── tools/ -│ └── gen_messages.py # codegen (messages.yaml -> .hpp) -├── include/secsgem/ -│ ├── secs2/{item,codec,sml,message}.hpp -│ ├── hsms/{header,connection}.hpp -│ ├── secsi/{header,block,protocol,tcp_transport}.hpp -│ ├── gem/ # FSMs per SEMI standard -│ ├── gem/store/ # one file per focused store -│ ├── config/loader.hpp -│ └── endpoint.hpp -├── src/{secs2,hsms,secsi,gem,config}/*.cpp -├── apps/ -│ ├── secs_server.cpp # passive equipment demo -│ ├── secs_client.cpp # active host demo -│ └── secs_interop_probe.cpp # cross-test against secsgem-py -├── interop/ # secsgem-py 0.3.0 cross-validation -└── tests/test_*.cpp # 384 cases / 2390 assertions -``` +--- ## Adding a capability @@ -150,269 +178,71 @@ transitions: - {name: field_b, shape: {kind: scalar, item_type: ASCII}} ``` -`docker compose run --rm builder` regenerates `messages.hpp`. The typed -builder, parser, and struct definition appear automatically. +`docker compose run --rm builder` regenerates `messages.hpp`. The +typed builder, parser, and struct definition appear automatically. +Run `--validate-config` after every YAML edit. --- -# Production / fab deployment +## Production deployment -The library is a runtime stack. Shipping it on a real tool involves -more than building the binary. This section enumerates the work -that sits between "tests pass" and "this is running on the fab floor." +See [INTEGRATION.md](INTEGRATION.md) for the full vendor-side +tutorial — wiring sensors, plugging FSMs into the tool, persistence +layout, monitoring/observability, HSMS-GS multi-MES setup. -## 1. Persistence directory layout +See [SECURITY.md](SECURITY.md) for concrete nftables / stunnel / +minisign / SIEM configs. -Enable persistence per store at startup, before the connection comes up. -Pattern (the call sites are equivalent on every store): +See [BENCHMARKS.md](BENCHMARKS.md) for the performance envelope — +roughly **140 k req/s S1F1**, **79 k req/s S1F3 (32 SVIDs)**, **572 +k S6F11/s push**, **~450 bytes per PJ+CJ pair**. Three orders of +magnitude above typical fab tool load. -```cpp -auto base = std::filesystem::path("/var/lib/acme-secsgem"); -model->spool.enable_persistence(base / "spool"); -model->carriers.enable_persistence(base / "carriers"); -model->load_ports.enable_persistence(base / "loadports"); -model->substrates.enable_persistence(base / "substrates"); -model->process_jobs.enable_persistence(base / "pjobs"); -model->control_jobs.enable_persistence(base / "cjobs"); -model->exceptions.enable_persistence(base / "exceptions"); -``` +See [MES_INTEROP.md](MES_INTEROP.md) for the day-1 punch list to run +against your commercial MES before promoting from staging to a real +tool. -Storage rules: - -- **Mount this volume on the same physical disk as the binary** — - network filesystems (NFS) can introduce latency that interferes - with the rename-based atomic write pattern. -- **Back this volume up daily**. Journal files are small (a few - hundred bytes each) and rsync-friendly. -- **Set sane retention**. Cleared exceptions and dequeued PJs are - removed automatically; complete carriers / substrates / CJs are - the application's responsibility to sweep. Cap by file count - (a million files in one directory is fine on ext4 / xfs; less - on others). -- **Disk space**: budget 100 MB for a busy fab tool over a year - (~500 K transitions, ~200 bytes each). In practice it's far - less because terminal-state records are removed. - -After a crash, the next process start replays every store and is -back to the prior in-memory state before the HSMS port opens. - -## 2. Security - -HSMS over plain TCP is the spec's wire protocol. The library -ships unencrypted by design — that's what equipment manufacturers -expect. In a real fab: - -- **Network isolation**: HSMS must run on a control LAN, never - exposed to engineering / corporate networks. Default the - `--port` to 5000 / 5005 on a dedicated VLAN behind firewall ACLs - that whitelist your MES host's IP. -- **TLS tunnel**: for cross-site HSMS (rare but real for multi-fab - shared hosts), tunnel the TCP through stunnel or a sidecar - proxy. Don't modify the HSMS wire protocol; wrap the socket. -- **Authentication**: HSMS doesn't include peer auth. Rely on - network-layer mTLS (sidecar proxy) and per-tool firewall rules. -- **Audit logging**: enable `Connection::set_log_handler` and - ship to a SIEM. Every SECS-II message in/out should be - retrievable for a configurable retention window — many fabs - require 90 days. -- **YAML config integrity**: sign your config bundles - (`equipment.yaml`, `control_state.yaml`, etc.) and verify the - signature on load. Misconfiguration is one of the top - root-causes of GEM-related fab incidents. - -## 3. Monitoring and observability - -The library exposes hooks at every layer. Wire them to whatever -your fab already runs. - -| Signal | Hook | Why it matters | -| ---------------------------- | ------------------------------------------ | -------------------------------------------------- | -| HSMS connection lifecycle | `Connection::set_log_handler`, `set_selected_handler`, `set_closed_handler` | reconnect storms, unexpected separates | -| T3 / T6 / T7 / T8 timer fires | `set_closed_handler` reason starts with "T*" | host MES unreachable, fab network event | -| Auto S9F* emission | `set_log_handler` line containing "-> S9F" | malformed peer traffic, schema drift | -| Spool depth | `model->spool.size()` | host MES backpressure / outage | -| FSM transitions (every store) | `set_*_change_handler` | tool state, throughput, anomaly detection | -| Persistence directory size | `du -s var/lib/acme-secsgem` | journal growth, untracked terminal-state records | - -Recommended metrics export pattern: aggregate into Prometheus -via a sidecar that polls the data model. Per-CEID emission rates, -alarm set/clear rates, T-timer expiry counts, and spool depth -form a reasonable starter dashboard. - -**Hooks fire on the io_context thread.** Every `set_*_change_handler` -callback the library invokes runs on the connection's io_context -(there are no locks anywhere in `EquipmentDataModel`). Metrics -exporters and log shippers wired into those callbacks must either be -thread-safe themselves or hand the work off (a lock-free queue, a -separate exporter thread polling published counters, `asio::post` -onto another executor). Doing blocking I/O from inside a handler -stalls the dispatcher — keep handlers cheap. See INTEGRATION.md §3 -for the cross-thread update pattern. - -## 4. High availability - -The library is single-threaded per HSMS connection — that's how -HSMS works. For HA: - -- **Run two equipment processes** in active/standby on the same - tool, sharing the persistence volume. Only the active accepts - the HSMS port; the standby tails the journal. Failover is - filesystem-locked. -- **Reconnect on the host side**: an MES-side disconnect should - trigger T5-based reconnect. Configure `Timers::t5` to your - MES's policy (default 10s). -- **Graceful shutdown**: SIGTERM should flush the write queue, - call `conn->separate()`, and exit cleanly so the journal is - point-consistent. The provided `apps/secs_server.cpp` shows - the pattern. - -## 5. Deployment patterns - -Three common shapes: - -### Docker / podman on a tool PC -```dockerfile -FROM ubuntu:24.04 -COPY build/secs_server /usr/local/bin/ -COPY etc/ /etc/acme-secsgem/ -VOLUME /var/lib/acme-secsgem -EXPOSE 5000 -ENTRYPOINT ["/usr/local/bin/secs_server", \ - "--port", "5000", \ - "--config", "/etc/acme-secsgem/equipment.yaml", \ - "--state-table", "/etc/acme-secsgem/control_state.yaml", \ - "--spool-dir", "/var/lib/acme-secsgem/spool"] -``` - -### systemd unit -```ini -[Unit] -Description=ACME SECS/GEM equipment -After=network.target - -[Service] -Type=simple -User=secsgem -Group=secsgem -ExecStart=/usr/local/bin/secs_server --port 5000 \ - --config /etc/acme-secsgem/equipment.yaml \ - --state-table /etc/acme-secsgem/control_state.yaml \ - --spool-dir /var/lib/acme-secsgem/spool -Restart=always -RestartSec=5 -LimitNOFILE=8192 - -[Install] -WantedBy=multi-user.target -``` - -### Kubernetes (multi-tool cell controller) -Run one Pod per tool with the persistence volume mounted from -local-storage (not NFS). The Service exposes the HSMS port on the -control LAN. Use a PodDisruptionBudget to ensure the standby is -available during rolling updates. - -## 6. Upgrade path - -YAML edits don't require a rebuild — restart the process and the -new dictionary loads. Code changes do require rebuild + restart. - -- **Zero-downtime for YAML**: if you're using the active/standby - HA pattern, edit YAML on the standby, restart the standby, - promote. -- **Code upgrades**: deploy to a canary tool first; bake-test for - at least a full wafer batch before fleet-wide rollout. -- **Schema migrations**: persistence records carry a 1-byte version - stamp after the magic byte. Every store (`ProcessJobStore`, - `SubstrateStore`, `ControlJobStore`, `CarrierStore`, `LoadPortStore`, - `ExceptionStore`, `SpoolStore`) accepts any version in - `[1, kVersion]`: code at kVersion=2 loads both v1 and v2 records - (v1 trailer fields default to empty). Future versions beyond - `kVersion` are rejected so a downgrade can't silently corrupt - data. Upgrade discipline: when adding fields, bump `kVersion` and - gate the new trailer behind `if (version >= N)` in the loader. - Tests in `tests/test_persistence_upgrade.cpp` lock down the - contract and act as a tripwire if a writer bumps `kVersion` - without teaching the loader to handle prior versions. Always - test the upgrade with a real on-disk journal before fleet rollout. - -## 7. Integration with the fab stack - -| Other system | How this library talks to it | -| ------------------- | --------------------------------------------------------------------- | -| MES (Camstar, Mozaic, Camstar) | HSMS-SS over TCP (`secs_server` listens on a port the MES is configured to connect to) | -| Multi-MES (HSMS-GS) | `Connection::add_session(device_id)` registers extra sessions on one TCP socket | -| AMHS / OHT | E84 per-port FSMs (`E84PortStore::on_signal_change(port, signal, value)`); wire to your I/O bridge | -| Recipe engine | RecipeStore.add (opaque) + RecipeStore.add_formatted (E42 structured) | -| Alarm sources | `AlarmRegistry::set(alid, active)` from your sensor poll | -| Carrier scanner | `CarrierStore::create / fire_id_event / set_slot_state` | -| Wafer tracker | `SubstrateStore::create / fire_*_event` | -| EPT / shift report | `EptStateMachine::accumulated(state)` reads the time-bucket counters | - -## 8. Compliance and certification - -- Fork `COMPLIANCE.md` and prune it to *your* tool's claimed - coverage; ship that copy with the tool. -- Run the in-repo conformance harness against your tool: - ``` - build/secs_conformance --host --port 5000 --device 0 - ``` - Exits 0 with a per-check PASS / FAIL summary covering every E30 - fundamental capability (establish comms, on-line ID, status data, - equipment constants, clock, alarms, PP management, documentation). - Adapt `apps/secs_conformance.cpp` to add your tool's - capability-specific checks. -- Run an independent third-party validator (GEM RTS or equivalent) - against your specific tool — a passing library + in-repo harness - is necessary but not sufficient for certification. -- Capture wire traces from every validator run; archive for audit. - -## 9. Testing in production - -- **Canary**: deploy to one or two tools per fab before fleet - rollout. -- **Synthetic transactions**: a heartbeat that issues S1F1 every - 60s and alerts on T3 timeout. Catches MES-side outages before - a real recipe does. -- **Shadow traffic**: for upgrades, run the new version listening - on a second port; have MES dual-connect; diff the responses. - -## 10. Operational runbook (starting point) - -Common production incidents and the right response: +### Operational runbook (starter) | Incident | First check | Mitigation | -| ----------------------------------- | ------------------------------------ | ----------------------------------------- | +|-------------------------------------|--------------------------------------|-------------------------------------------| | HSMS connection flapping | T7 / T6 timer fires in logs | check MES reachability, network MTU | | Spool depth growing | host MES connectivity / ACK rate | force-drain via S6F23, escalate to MES | | State machine "stuck" | last state-change handler log line | host-issued offline + re-establish | -| Alarm storm | AlarmRegistry `all()` snapshot | check upstream sensor; quench via S5F3 | +| Alarm storm | `AlarmRegistry::all()` snapshot | check upstream sensor; quench via S5F3 | | Persistence dir growing unbounded | `du -s` + file count | sweep terminal-state records | | Cross-tool inconsistency | `secsgem_tests` on canary tool | compare wire trace vs validator | --- -## Demo - -The two-container demo walks ~24 SECS transactions end-to-end -through the data model. Run `docker compose up --no-deps server client` -and watch the logs interleave. - ## Build details -The toolchain image (`Dockerfile`) is Ubuntu 24.04 with `g++-13`, CMake, -Ninja, `libasio-dev`, `libyaml-cpp-dev`, and Python 3 for the codegen. -doctest is fetched via CMake FetchContent. Build artifacts live in a -named Docker volume so the host filesystem stays clean. +The toolchain image (`Dockerfile`) is Ubuntu 24.04 with `g++-13`, +CMake, Ninja, `libasio-dev`, `libyaml-cpp-dev`, and Python 3 for the +codegen. doctest is fetched via CMake FetchContent. Build artifacts +live in a named Docker volume so the host filesystem stays clean. Standalone Asio is used in header-only mode (`ASIO_STANDALONE`). No Boost dependency. +### ThreadSanitizer + +```bash +cmake -S . -B build-tsan -G Ninja -DCMAKE_BUILD_TYPE=Debug -DSECSGEM_TSAN=ON +cmake --build build-tsan +TSAN_OPTIONS=halt_on_error=1 build-tsan/secsgem_tests +``` + +Runs as a separate lane in CI. Catches data races in the io_context +strand contract documented in INTEGRATION.md §3. + +--- + ## Interop `interop/` contains the secsgem-py 0.3.0 cross-validation harness — -secsgem-py active host driving our C++ passive server, our C++ active -host probing secsgem-py's passive equipment, and a raw GEM-300 harness -that round-trips S3 (E87), S14 (E94), S16 (E40), S12 (wafer maps) -through hand-crafted `SecsStreamFunction` subclasses. See -`interop/README.md`. +secsgem-py active host driving our C++ passive server, our C++ +active host probing secsgem-py's passive equipment, and a raw GEM-300 +harness that round-trips S3 (E87), S14 (E94), S16 (E40), S12 (wafer +maps) through hand-crafted `SecsStreamFunction` subclasses. See +[`interop/README.md`](interop/README.md).