secs-gem/README.md

# secs-gem

A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully
containerized.  Every behavioural rule lives in YAML; the C++ is the
engine that reads them.  Implements **all of E4, E5, E30, E37
(SS + GS), E39, E40, E42, E84, E87, E90, E94, E116, E120, E148,
E157**.

> **License: proprietary — see [LICENSE](LICENSE).**  No use, copy,
> compile, evaluate, benchmark, or deploy without a written license
> from the copyright holder.  Contact `raphael@maenle.net` for
> commercial licensing, evaluation terms, or fab deployment.

---

## Quick start

Everything runs in Docker — no compiler or build tools on the host.

```bash
docker compose run --rm builder         # configure + compile
docker compose run --rm tests           # 445 cases / 2 753 assertions
docker compose up --no-deps server client   # live two-container demo
```

The two-container demo walks ~24 SECS transactions end-to-end
through the data model.  Watch the logs interleave.

---

## Documentation map

| File                                          | What it covers                                                          |
|-----------------------------------------------|-------------------------------------------------------------------------|
| [COMPLIANCE.md](COMPLIANCE.md)                | Per-capability audit against every SEMI standard implemented           |
| [INTEGRATION.md](INTEGRATION.md)              | Vendor-side tutorial: YAML → callbacks → production deploy             |
| [ARCHITECTURE.md](ARCHITECTURE.md)            | How the pieces fit + how to extend (new store / FSM / message)         |
| [PROOFS.md](PROOFS.md)                        | The eight commands that prove the feature-completeness claim           |
| [VERIFICATION.md](VERIFICATION.md)            | Test plan for the external validators behind the proof table           |
| [BENCHMARKS.md](BENCHMARKS.md)                | Performance envelope (throughput, latency, memory) + how to re-run     |
| [MES_INTEROP.md](MES_INTEROP.md)              | Day-1 punch list to run against your commercial MES (59 test IDs)      |
| [SECURITY.md](SECURITY.md)                    | Concrete configs: nftables, stunnel, minisign, SIEM audit-log schema   |
| [GLOSSARY.md](GLOSSARY.md)                    | SEMI vocabulary: SVID, CEID, PPID, ALCD, HCACK, T-timers, …            |
| [FAQ.md](FAQ.md)                              | Common questions and their canonical answers                            |
| [examples/pvd_tool/](examples/pvd_tool/)      | Worked example: a realistic fictional PVD tool, YAML + C++ wiring      |
| [LICENSE](LICENSE)                            | Proprietary license terms                                              |

---

## How it's proved

"Feature-complete" is a claim that the code must prove, not the
README.  On a fresh clone, **eight commands** demonstrate it:
unit + integration suite (445 cases / 2 753 assertions), a live
conformance harness (47 wire-level checks), interop against
secsgem-py (31 checks) and secs4java8 (55 checks), a 100 000-op
soak property test, YAML config validation, Wireshark's HSMS
dissector on a recorded pcap (69 frames, 0 malformed), and libFuzzer
(ASan + UBSan, 0 crashes).  Each command, exit code, and the
external-vs-internal breakdown live in [PROOFS.md](PROOFS.md).

CI runs the full suite plus a separate ThreadSanitizer lane on
every push to `main` ([Gitea Actions](.gitea/workflows/ci.yml));
all 445 cases pass clean under `-fsanitize=thread`.

---

## Architecture

The project is **spec-as-data**: the SEMI behavioural rules live in
YAML; the C++ is the engine that reads them.

```
   ┌──────────────────────────────────────────────────────────────┐
   │ data/                                                        │
   │   messages.yaml          SECS-II message catalog (164 msgs)  │
   │   control_state.yaml     E30 §6.2 control transition table   │
   │   process_job_state.yaml E40 §6 PJ transition table          │
   │   control_job_state.yaml E94 §6 CJ transition table          │
   │   equipment.yaml         SVIDs / DVIDs / ECIDs / CEIDs /     │
   │                          alarms / recipes / commands         │
   └──────────────────────┬───────────────────────────────────────┘
                          │  (codegen at build, YAML loaded at startup)
                          ▼
   ┌──────────────────────────────────────────────────────────────┐
   │ apps/                                                        │
   │   secs_server   passive equipment      secs_bench   perf     │
   │   secs_client   active host            secs_conformance      │
   │   secs_interop_probe                                         │
   └──────────────────────────────────────────────────────────────┘

   secsgem::config   loader.hpp + validate.hpp:
                     YAML -> data model, with multi-error validator
                     surfacing every issue at once (`--validate-config`)
   secsgem::gem      per-standard FSM + per-store persistence
                     (every store accepts v ∈ [1, kVersion] for
                     forward-compatible schema migrations).
                     EquipmentDataModel composes all stores.
                     Router (stream, function) -> handler.
                     Generated messages.hpp covers 164 SxFy.
   secsgem::hsms     Connection (Asio): HSMS-SS + HSMS-GS, all
                     T-timers enforced, auto S9F3/F5/F7/F9/F11.
   secsgem::secsi    SECS-I Protocol FSM (E4): T1/T2/T3/T4 enforced
                     in-FSM, TCP transport for tunnel testing.
   secsgem::secs2    Item (variant), encode/decode, Message,
                     SML parser/printer.
   secsgem::metrics  Prometheus exporter (Registry + HTTP server).
```

See [ARCHITECTURE.md](ARCHITECTURE.md) for how to extend it (new
store / FSM / message).

---

## Adding a capability

The point of "spec-as-data" is that adding behaviour almost never
requires a C++ change.

### New SVID
```yaml
# data/equipment.yaml
svids:
  - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}
```

### New host command with side effects
```yaml
host_commands:
  - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}
```

### New state transition
```yaml
# data/control_state.yaml
transitions:
  - {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept}
```

### New SECS-II message
```yaml
# data/messages.yaml
- id: S6F30
  stream: 6
  function: 30
  w: true
  builder: s6f30_something
  parser: parse_s6f30
  body:
    kind: list
    struct_name: Something
    fields:
      - {name: field_a, shape: {kind: scalar, item_type: U4}}
      - {name: field_b, shape: {kind: scalar, item_type: ASCII}}
```

`docker compose run --rm builder` regenerates `messages.hpp`.  The
typed builder, parser, and struct definition appear automatically.
Run `--validate-config` after every YAML edit.

---

## Production deployment

See [INTEGRATION.md](INTEGRATION.md) for the full vendor-side
tutorial — wiring sensors, plugging FSMs into the tool, persistence
layout, monitoring/observability, HSMS-GS multi-MES setup.

See [SECURITY.md](SECURITY.md) for concrete nftables / stunnel /
minisign / SIEM configs.

See [BENCHMARKS.md](BENCHMARKS.md) for the performance envelope —
roughly **140 k req/s S1F1**, **79 k req/s S1F3 (32 SVIDs)**, **572
k S6F11/s push**, **~450 bytes per PJ+CJ pair**.  Three orders of
magnitude above typical fab tool load.

See [MES_INTEROP.md](MES_INTEROP.md) for the day-1 punch list to run
against your commercial MES before promoting from staging to a real
tool.

### Operational runbook (starter)

| Incident                            | First check                          | Mitigation                                |
|-------------------------------------|--------------------------------------|-------------------------------------------|
| HSMS connection flapping            | T7 / T6 timer fires in logs          | check MES reachability, network MTU       |
| Spool depth growing                 | host MES connectivity / ACK rate     | force-drain via S6F23, escalate to MES    |
| State machine "stuck"               | last state-change handler log line   | host-issued offline + re-establish        |
| Alarm storm                         | `AlarmRegistry::all()` snapshot      | check upstream sensor; quench via S5F3    |
| Persistence dir growing unbounded   | `du -s` + file count                 | sweep terminal-state records              |
| Cross-tool inconsistency            | `secsgem_tests` on canary tool       | compare wire trace vs validator           |

---

## Build details

The toolchain image (`Dockerfile`) is Ubuntu 24.04 with `g++-13`,
CMake, Ninja, `libasio-dev`, `libyaml-cpp-dev`, and Python 3 for the
codegen.  doctest is fetched via CMake FetchContent.  Build artifacts
live in a named Docker volume so the host filesystem stays clean.

Standalone Asio is used in header-only mode (`ASIO_STANDALONE`).  No
Boost dependency.

### ThreadSanitizer

```bash
cmake -S . -B build-tsan -G Ninja -DCMAKE_BUILD_TYPE=Debug -DSECSGEM_TSAN=ON
cmake --build build-tsan
TSAN_OPTIONS=halt_on_error=1 build-tsan/secsgem_tests
```

Runs as a separate lane in CI.  Catches data races in the io_context
strand contract documented in INTEGRATION.md §3.

---

## Interop

`interop/` contains the secsgem-py 0.3.0 cross-validation harness —
secsgem-py active host driving our C++ passive server, our C++
active host probing secsgem-py's passive equipment, and a raw GEM-300
harness that round-trips S3 (E87), S14 (E94), S16 (E40), S12 (wafer
maps) through hand-crafted `SecsStreamFunction` subclasses.  See
[`interop/README.md`](interop/README.md).