secs-gem/README.md

# secs-gem

A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully
containerized.  Every behavioural rule lives in YAML; the C++ is the
engine that reads them.  Implements **all of E4, E5, E30, E37
(SS + GS), E39, E40, E42, E84, E87, E90, E94, E116, E120, E148,
E157**.

> **License: proprietary — see [LICENSE](LICENSE).**  No use, copy,
> compile, evaluate, benchmark, or deploy without a written license
> from the copyright holder.  Contact `raphael@maenle.net` for
> commercial licensing, evaluation terms, or fab deployment.

---

## Quick start

Everything runs in Docker — no compiler or build tools on the host.

```bash
docker compose run --rm builder         # configure + compile
docker compose run --rm tests           # 473 cases / 3 087 assertions
docker compose up --no-deps server client   # live two-container demo
```

The two-container demo walks ~24 SECS transactions end-to-end
through the data model.  Watch the logs interleave.

---

## Integrating your tool (pick a tier)

Three ways in, same engine underneath:

1. **Python, no SEMI knowledge** — run the `secs_gemd` daemon and
   `pip install` the pure-Python client in [clients/python](clients/python):

   ```python
   from secsgem_client import Equipment

   eq = Equipment("localhost:50051")
   eq.set(ChamberPressure=2.5)          # variables: kwargs, not strings

   @eq.command                          # the function name IS the command,
   def START(cmd):                      # validated against the real equipment
       run_recipe(cmd.params.get("PPID"))   # — so a typo fails at startup
       eq.fire(eq.names.event.ProcessStarted)   # autocomplete + typo-safe

   eq.listen()
   ```

   Names come from *your* `equipment.yaml`. `@eq.command` binds a handler by
   its function name; `eq.names.event.*` / `.alarm.*` / `.command.*` are
   autocomplete-able, typo-checked views fetched from the live daemon — so
   you rarely type a bare string. (The plain forms — `@eq.on("START")`,
   `eq.fire("ProcessStarted")` — also work.)

   A complete tool is ~25 lines: [clients/python/examples/mini_tool.py](clients/python/examples/mini_tool.py).

2. **Any language over gRPC** — `secs_gemd` exposes the name-based API in
   [proto/secsgem/v1/equipment.proto](proto/secsgem/v1/equipment.proto)
   (variables, events, alarms, control state, health stream, and the
   host-command stream with the SEMI-conformant HCACK-4 contract). The
   daemon owns the durable HSMS link: your tool software can restart
   without the fab host ever noticing.

3. **Embedded C++** — construct a `gem::EquipmentRuntime`, call the
   per-capability `register_*` functions (or `register_default_handlers`
   for all of GEM), and wire behaviour with `commands.set_handler`.
   `apps/secs_server.cpp` is the ~110-line canonical example.

---

## Documentation map

| File                                                       | What it covers                                                          |
|------------------------------------------------------------|-------------------------------------------------------------------------|
| [docs/](docs/00_index.md)                                  | Guided-tour tutorial series — teach-from-zero across the protocol and the codebase |
| [docs/COMPLIANCE.md](docs/COMPLIANCE.md)                   | Per-capability audit against every SEMI standard implemented           |
| [docs/INTEGRATION.md](docs/INTEGRATION.md)                 | Vendor-side tutorial: YAML → callbacks → production deploy             |
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)               | How the pieces fit + how to extend (new store / FSM / message)         |
| [docs/PROOFS.md](docs/PROOFS.md)                           | The eight commands that prove the feature-completeness claim           |
| [docs/VERIFICATION.md](docs/VERIFICATION.md)               | Test plan for the external validators behind the proof table           |
| [docs/BENCHMARKS.md](docs/BENCHMARKS.md)                   | Performance envelope (throughput, latency, memory) + how to re-run     |
| [docs/MES_INTEROP.md](docs/MES_INTEROP.md)                 | Day-1 punch list to run against your commercial MES (59 test IDs)      |
| [docs/SECURITY.md](docs/SECURITY.md)                       | Concrete configs: nftables, stunnel, minisign, SIEM audit-log schema   |
| [docs/GLOSSARY.md](docs/GLOSSARY.md)                       | SEMI vocabulary: SVID, CEID, PPID, ALCD, HCACK, T-timers, …            |
| [docs/FAQ.md](docs/FAQ.md)                                 | Common questions and their canonical answers                            |
| [examples/pvd_tool/](examples/pvd_tool/)                   | Worked example: a realistic fictional PVD tool, YAML + C++ wiring      |
| [LICENSE](LICENSE)                                         | Proprietary license terms                                              |

---

## Testing

- **Unit + integration** — `docker compose run --rm tests` runs 473
  cases / 3 087 assertions across every store, FSM, codec, parser, and
  persistence path.
- **Live conformance harness** — 47 wire-level checks against the
  passive server.
- **Interop** — round-trips against secsgem-py (31 checks), secs4java8
  (55 checks), and Wireshark's HSMS dissector (69 frames, 0 malformed).
- **Soak + fuzz** — 100 000-op property test; libFuzzer with ASan +
  UBSan over `secs2::decode` and the SML parser, 0 crashes.
- **Daemon** — `secs_gemd_tests` exercises the gRPC service over real
  in-process channels (125 assertions), in Release and under
  ThreadSanitizer; `interop/daemon_interop.py` and
  `interop/pyclient_interop.py` prove the gRPC↔HSMS bridge and the
  published Python client against a live daemon with secsgem-py as host.
- **One command for all of it** — `tools/run_interop.sh` runs every
  validation step (build, both unit suites, secsgem-py host, C++
  conformance, Python client, daemon bridge, spool restart, tshark,
  secs4java8) with a PASS/FAIL summary.
- **Config validation** — `secs_server --validate-config` rejects
  malformed YAML before startup.
- **CI** — [Gitea Actions](.gitea/workflows/ci.yml) runs the full
  suite plus a `-fsanitize=thread` lane on every push to `main`; all
  473 cases pass clean under TSan.

Exact commands, exit codes, and per-standard test counts are in
[docs/PROOFS.md](docs/PROOFS.md); the rationale behind the external
validators is in [docs/VERIFICATION.md](docs/VERIFICATION.md).

---

## Architecture

The project is **spec-as-data**: the SEMI behavioural rules live in
YAML; the C++ is the engine that reads them.

```
   ┌──────────────────────────────────────────────────────────────┐
   │ data/                                                        │
   │   messages.yaml          SECS-II message catalog (164 msgs)  │
   │   control_state.yaml     E30 §6.2 control transition table   │
   │   process_job_state.yaml E40 §6 PJ transition table          │
   │   control_job_state.yaml E94 §6 CJ transition table          │
   │   equipment.yaml         SVIDs / DVIDs / ECIDs / CEIDs /     │
   │                          alarms / recipes / commands         │
   └──────────────────────┬───────────────────────────────────────┘
                          │  (codegen at build, YAML loaded at startup)
                          ▼
   ┌──────────────────────────────────────────────────────────────┐
   │ apps/                                                        │
   │   secs_server   passive equipment      secs_bench   perf     │
   │   secs_client   active host            secs_conformance      │
   │   secs_interop_probe                                         │
   └──────────────────────────────────────────────────────────────┘

   secsgem::config   loader.hpp + validate.hpp:
                     YAML -> data model, with multi-error validator
                     surfacing every issue at once (`--validate-config`)
   secsgem::gem      per-standard FSM + per-store persistence
                     (every store accepts v ∈ [1, kVersion] for
                     forward-compatible schema migrations).
                     EquipmentDataModel composes all stores.
                     Router (stream, function) -> handler.
                     Generated messages.hpp covers 164 SxFy.
   secsgem::hsms     Connection (Asio): HSMS-SS + HSMS-GS, all
                     T-timers enforced, auto S9F3/F5/F7/F9/F11.
   secsgem::secsi    SECS-I Protocol FSM (E4): T1/T2/T3/T4 enforced
                     in-FSM, TCP transport for tunnel testing.
   secsgem::secs2    Item (variant), encode/decode, Message,
                     SML parser/printer.
   secsgem::metrics  Prometheus exporter (Registry + HTTP server).
```

See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for how to extend
it (new store / FSM / message).

---

## Adding a capability

The point of "spec-as-data" is that adding behaviour almost never
requires a C++ change.

### New SVID
```yaml
# data/equipment.yaml
svids:
  - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}
```

### New host command with side effects
```yaml
host_commands:
  - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}
```

### New state transition
```yaml
# data/control_state.yaml
transitions:
  - {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept}
```

### New SECS-II message
```yaml
# data/messages.yaml
- id: S6F30
  stream: 6
  function: 30
  w: true
  builder: s6f30_something
  parser: parse_s6f30
  body:
    kind: list
    struct_name: Something
    fields:
      - {name: field_a, shape: {kind: scalar, item_type: U4}}
      - {name: field_b, shape: {kind: scalar, item_type: ASCII}}
```

`docker compose run --rm builder` regenerates `messages.hpp`.  The
typed builder, parser, and struct definition appear automatically.
Run `--validate-config` after every YAML edit.

---

## Production deployment

See [docs/INTEGRATION.md](docs/INTEGRATION.md) for the full
vendor-side tutorial — wiring sensors, plugging FSMs into the tool,
persistence layout, monitoring/observability, HSMS-GS multi-MES
setup.

See [docs/SECURITY.md](docs/SECURITY.md) for concrete nftables /
stunnel / minisign / SIEM configs.

See [docs/BENCHMARKS.md](docs/BENCHMARKS.md) for the performance
envelope — roughly **140 k req/s S1F1**, **79 k req/s S1F3 (32
SVIDs)**, **572 k S6F11/s push**, **~450 bytes per PJ+CJ pair**.
Three orders of magnitude above typical fab tool load.

See [docs/MES_INTEROP.md](docs/MES_INTEROP.md) for the day-1 punch
list to run against your commercial MES before promoting from
staging to a real tool.

### Operational runbook (starter)

| Incident                            | First check                          | Mitigation                                |
|-------------------------------------|--------------------------------------|-------------------------------------------|
| HSMS connection flapping            | T7 / T6 timer fires in logs          | check MES reachability, network MTU       |
| Spool depth growing                 | host MES connectivity / ACK rate     | force-drain via S6F23, escalate to MES    |
| State machine "stuck"               | last state-change handler log line   | host-issued offline + re-establish        |
| Alarm storm                         | `AlarmRegistry::all()` snapshot      | check upstream sensor; quench via S5F3    |
| Persistence dir growing unbounded   | `du -s` + file count                 | sweep terminal-state records              |
| Cross-tool inconsistency            | `secsgem_tests` on canary tool       | compare wire trace vs validator           |

---

## Deferred follow-ups

- **asio `serial_port` adapter for SECS-I.** `secsi::Protocol` is
  tested end-to-end over `secsi::TcpTransport`; the matching serial
  driver isn't written yet.  Mirror `TcpTransport` to add it.

---

## Build details

The toolchain image (`Dockerfile`) is Ubuntu 24.04 with `g++-13`,
CMake, Ninja, `libasio-dev`, `libyaml-cpp-dev`, and Python 3 for the
codegen.  doctest is fetched via CMake FetchContent.  Build artifacts
live in a named Docker volume so the host filesystem stays clean.

Standalone Asio is used in header-only mode (`ASIO_STANDALONE`).  No
Boost dependency.

---

## Interop

Four independent external validators cross-check the codebase:

- **secsgem-py 0.3.0** (Python reference impl) — three harnesses
  under `interop/`: secsgem-py active host driving the C++ passive
  server (31 checks), C++ active host probing secsgem-py's passive
  equipment, and a raw GEM 300 harness round-tripping S3 / S14 /
  S16 / S12 through hand-crafted `SecsStreamFunction` subclasses.
- **secs4java8** (independent Java SECS implementation) — 55
  cross-validation checks covering S1/S2/S3/S5/S6/S7/S10/S14/S16,
  full-body GEM 300 shapes, S2F49 enhanced commands, S5F13–F18
  exception recovery.
- **Wireshark / tshark HSMS dissector** (independent network-protocol
  authors) — 69 HSMS frames dissected on a recorded pcap, no
  malformed-packet warnings.
- **libFuzzer + ASan + UBSan** — 200 000+ inputs through
  `secs2::decode` and 1.4 M+ through `try_parse_sml` per 60 s lane,
  0 crashes.

See [`interop/README.md`](interop/README.md) for harness-by-harness
detail and [docs/VERIFICATION.md](docs/VERIFICATION.md) for the test
plan rationale.