raphael/secs-gem
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a79973ed4c8fdb49100478abece201a50fa9cf30
secs-gem/README.md
T
raphael e82f67ecad docs: README restructure + proof-of-feature-completeness section 
The old README mixed intro, quickstart, architecture, and 10 sections
of production deployment over 419 lines, with significant overlap
with INTEGRATION.md.  It claimed "implements every standard" without
making the claim concrete.

Restructured to ~250 lines with the proof front and center.

New top-of-README "Proof of feature-completeness" section: five
commands that, when they all exit zero on a fresh clone, prove the
COMPLIANCE.md claims.  Each command verified end-to-end before
landing in this commit:

  1. docker compose run --rm tests
       → 426 cases / 2557 assertions PASS
  2. secs_conformance --host server --port 5000
       → 47 / 47 wire-level checks PASS
  3. host_vs_cpp_server.py --host server
       → 24 secsgem-py interop checks PASS
  4. SECSGEM_ROBUSTNESS_SOAK=1 secsgem_tests -tc='*soak*'
       → 100 000 random tool operations, all invariants hold
  5. secs_server --validate-config <all four YAMLs>
       → 0 errors, 0 warnings across the shipped configs

Plus a per-standard test-coverage table mapping every claimed SEMI
standard (E5, E5 §13, E4, E37, E30, E40, E94, E42, E87, E90, E116,
E120/E39, E157, E84) to its test files and case count, summing to
426 to match the doctest totals.  Counts verified by
`grep -c TEST_CASE` per file.

CI also runs the TSan lane (separate job in
.gitea/workflows/ci.yml); README documents it under Build details.

Content moved out of README into specialized docs (eliminates
duplication):

- Security configs → SECURITY.md (was 14-line bullet list; now a
  365-line file with nftables, stunnel, minisign, SIEM schema)
- Persistence layout + monitoring + HA + deployment patterns +
  upgrade discipline + fab-stack integration → INTEGRATION.md
- Performance envelope → BENCHMARKS.md
- MES interop punch list → MES_INTEROP.md

README now reads top-to-bottom: what this is → license → proof →
quickstart → doc map → architecture → adding capabilities →
production (1-line pointers to the deep docs) → build details →
interop.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-09 15:37:33 +02:00

249 lines
14 KiB
Markdown
Raw Blame History

# secs-gem
A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully
containerized. Every behavioural rule lives in YAML; the C++ is the
engine that reads them. Implements **all of E4, E5, E30, E37
(SS + GS), E39, E40, E42, E84, E87, E90, E94, E116, E120, E148,
E157**.
> **License: proprietary — see [LICENSE](LICENSE).** No use, copy,
> compile, evaluate, benchmark, or deploy without a written license
> from the copyright holder. Contact `raphael@maenle.net` for
> commercial licensing, evaluation terms, or fab deployment.
---
## Proof of feature-completeness
"Feature-complete" is a claim that the code must prove, not the
README. These five commands are the proof. If they all exit zero on
a fresh clone, the codebase implements what
[COMPLIANCE.md](COMPLIANCE.md) claims.
| # | Command | What it proves |
|---|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
| 1 | `docker compose run --rm tests` | **426 test cases / 2 557 assertions** pass: every store, FSM, codec, parser, persistence path |
| 2 | `docker compose run --rm builder /app/build/secs_conformance --host server --port 5000` | **47 wire-level conformance checks** PASS against a live passive equipment |
| 3 | `docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py --host server` | **24 interop checks** PASS against secsgem-py 0.3.0 (the Python reference impl) |
| 4 | `SECSGEM_ROBUSTNESS_SOAK=1 docker compose run --rm builder /app/build/secsgem_tests -tc='*soak*'` | **100 000 random tool operations** execute with all invariants and persistence round-trips holding |
| 5 | `docker compose run --rm builder /app/build/secs_server --validate-config --config /app/data/equipment.yaml --state-table /app/data/control_state.yaml --pj-state-table /app/data/process_job_state.yaml --cj-state-table /app/data/control_job_state.yaml` | Every shipped YAML config passes structural + referential validation |
Plus, on every push to `main`, [Gitea Actions](.gitea/workflows/ci.yml)
runs both a **Release build + full test suite** and a separate
**ThreadSanitizer lane** that builds with `-fsanitize=thread` and
fails on any race. All 426 cases / 2 557 assertions pass under TSan
clean.
### Per-standard test coverage
Every claimed standard has dedicated tests. Counts are
`grep -c TEST_CASE`; cross-cutting tests (e.g. `test_robustness_fuzz`,
`test_gem300_scenario`) exercise multiple standards in concert.
| Standard | Test files | Cases |
|-----------------------------------|-------------------------------------------------------------------------------------------|------:|
| **E5** — SECS-II encoding | `test_secs2`, `test_sml`, `test_messages`, `test_identifier_wildcards`, `test_fuzz` | 120 |
| **E5 §13** — exceptions | `test_exceptions`, `test_exception_persistence` | 16 |
| **E4** — SECS-I transport | `test_secsi`, `test_secsi_timers`, `test_secsi_tcp` | 27 |
| **E37** — HSMS (SS + GS) | `test_hsms`, `test_hsms_connection`, `test_hsms_timers`, `test_hsms_s9`, `test_hsms_gs`, `test_hsms_gs_integration`, `test_s9_fallback`, `test_concurrency` | 34 |
| **E30** — GEM core | `test_control_state`, `test_communication_state`, `test_host_handler`, `test_data_model`, `test_loader`, `test_config_validate` | 71 |
| **E40** — process jobs | `test_process_jobs` | 21 |
| **E94** — control jobs | `test_control_jobs` | 9 |
| **E42** — formatted PP | `test_e42_formatted_pp` | 6 |
| **E87** — carriers + load ports | `test_carriers`, `test_carrier_state`, `test_carrier_persistence`, `test_e87_wire_scenarios` | 27 |
| **E90** — substrate tracking | `test_substrates`, `test_substrate_persistence` | 21 |
| **E116** — EPT | `test_ept` | 7 |
| **E120 / E39** — common equip / object service | `test_cem_objects` | 3 |
| **E157** — module process tracking | `test_modules` | 5 |
| **E84** — parallel I/O + timers | `test_e84`, `test_e84_ports`, `test_e84_timers`, `test_e84_asio_timers` | 27 |
| Persistence + cross-cutting | `test_job_persistence`, `test_persistence_upgrade`, `test_wire_ceid_emission`, `test_gem300_scenario`, `test_live_gem300`, `test_thread_safety`, `test_metrics_prometheus`, `test_robustness_fuzz` | 32 |
| **Total** | | **426** |
A single command to see this live: `docker compose run --rm builder
/app/build/secsgem_tests --list-test-cases | wc -l` (currently 426).
---
## Quick start
Everything runs in Docker — no compiler or build tools on the host.
```bash
docker compose run --rm builder # configure + compile
docker compose run --rm tests # 426 cases / 2 557 assertions
docker compose up --no-deps server client # live two-container demo
```
The two-container demo walks ~24 SECS transactions end-to-end
through the data model. Watch the logs interleave.
---
## Documentation map
| File | What it covers |
|-----------------------------------------------|-------------------------------------------------------------------------|
| [COMPLIANCE.md](COMPLIANCE.md) | Per-capability audit against every SEMI standard implemented |
| [INTEGRATION.md](INTEGRATION.md) | Vendor-side tutorial: YAML → callbacks → production deploy |
| [BENCHMARKS.md](BENCHMARKS.md) | Performance envelope (throughput, latency, memory) + how to re-run |
| [MES_INTEROP.md](MES_INTEROP.md) | Day-1 punch list to run against your commercial MES (60+ test IDs) |
| [SECURITY.md](SECURITY.md) | Concrete configs: nftables, stunnel, minisign, SIEM audit-log schema |
| [LICENSE](LICENSE) | Proprietary license terms |
---
## Architecture
The project is **spec-as-data**: the SEMI behavioural rules live in
YAML; the C++ is the engine that reads them.
```
┌──────────────────────────────────────────────────────────────┐
│ data/ │
│ messages.yaml SECS-II message catalog (164 msgs) │
│ control_state.yaml E30 §6.2 control transition table │
│ process_job_state.yaml E40 §6 PJ transition table │
│ control_job_state.yaml E94 §6 CJ transition table │
│ equipment.yaml SVIDs / DVIDs / ECIDs / CEIDs / │
│ alarms / recipes / commands │
└──────────────────────┬───────────────────────────────────────┘
│ (codegen at build, YAML loaded at startup)
▼
┌──────────────────────────────────────────────────────────────┐
│ apps/ │
│ secs_server passive equipment secs_bench perf │
│ secs_client active host secs_conformance │
│ secs_interop_probe │
└──────────────────────────────────────────────────────────────┘
secsgem::config loader.hpp + validate.hpp:
YAML -> data model, with multi-error validator
surfacing every issue at once (`--validate-config`)
secsgem::gem per-standard FSM + per-store persistence
(every store accepts v ∈ [1, kVersion] for
forward-compatible schema migrations).
EquipmentDataModel composes all stores.
Router (stream, function) -> handler.
Generated messages.hpp covers 164 SxFy.
secsgem::hsms Connection (Asio): HSMS-SS + HSMS-GS, all
T-timers enforced, auto S9F3/F5/F7/F9/F11.
secsgem::secsi SECS-I Protocol FSM (E4): T1/T2/T3/T4 enforced
in-FSM, TCP transport for tunnel testing.
secsgem::secs2 Item (variant), encode/decode, Message,
SML parser/printer.
secsgem::metrics Prometheus exporter (Registry + HTTP server).
```
---
## Adding a capability
The point of "spec-as-data" is that adding behaviour almost never
requires a C++ change.
### New SVID
```yaml
# data/equipment.yaml
svids:
- {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}
```
### New host command with side effects
```yaml
host_commands:
- {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}
```
### New state transition
```yaml
# data/control_state.yaml
transitions:
- {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept}
```
### New SECS-II message
```yaml
# data/messages.yaml
- id: S6F30
stream: 6
function: 30
w: true
builder: s6f30_something
parser: parse_s6f30
body:
kind: list
struct_name: Something
fields:
- {name: field_a, shape: {kind: scalar, item_type: U4}}
- {name: field_b, shape: {kind: scalar, item_type: ASCII}}
```
`docker compose run --rm builder` regenerates `messages.hpp`. The
typed builder, parser, and struct definition appear automatically.
Run `--validate-config` after every YAML edit.
---
## Production deployment
See [INTEGRATION.md](INTEGRATION.md) for the full vendor-side
tutorial — wiring sensors, plugging FSMs into the tool, persistence
layout, monitoring/observability, HSMS-GS multi-MES setup.
See [SECURITY.md](SECURITY.md) for concrete nftables / stunnel /
minisign / SIEM configs.
See [BENCHMARKS.md](BENCHMARKS.md) for the performance envelope —
roughly **140 k req/s S1F1**, **79 k req/s S1F3 (32 SVIDs)**, **572
k S6F11/s push**, **~450 bytes per PJ+CJ pair**. Three orders of
magnitude above typical fab tool load.
See [MES_INTEROP.md](MES_INTEROP.md) for the day-1 punch list to run
against your commercial MES before promoting from staging to a real
tool.
### Operational runbook (starter)
| Incident | First check | Mitigation |
|-------------------------------------|--------------------------------------|-------------------------------------------|
| HSMS connection flapping | T7 / T6 timer fires in logs | check MES reachability, network MTU |
| Spool depth growing | host MES connectivity / ACK rate | force-drain via S6F23, escalate to MES |
| State machine "stuck" | last state-change handler log line | host-issued offline + re-establish |
| Alarm storm | `AlarmRegistry::all()` snapshot | check upstream sensor; quench via S5F3 |
| Persistence dir growing unbounded | `du -s` + file count | sweep terminal-state records |
| Cross-tool inconsistency | `secsgem_tests` on canary tool | compare wire trace vs validator |
---
## Build details
The toolchain image (`Dockerfile`) is Ubuntu 24.04 with `g++-13`,
CMake, Ninja, `libasio-dev`, `libyaml-cpp-dev`, and Python 3 for the
codegen. doctest is fetched via CMake FetchContent. Build artifacts
live in a named Docker volume so the host filesystem stays clean.
Standalone Asio is used in header-only mode (`ASIO_STANDALONE`). No
Boost dependency.
### ThreadSanitizer
```bash
cmake -S . -B build-tsan -G Ninja -DCMAKE_BUILD_TYPE=Debug -DSECSGEM_TSAN=ON
cmake --build build-tsan
TSAN_OPTIONS=halt_on_error=1 build-tsan/secsgem_tests
```
Runs as a separate lane in CI. Catches data races in the io_context
strand contract documented in INTEGRATION.md §3.
---
## Interop
`interop/` contains the secsgem-py 0.3.0 cross-validation harness —
secsgem-py active host driving our C++ passive server, our C++
active host probing secsgem-py's passive equipment, and a raw GEM-300
harness that round-trips S3 (E87), S14 (E94), S16 (E40), S12 (wafer
maps) through hand-crafted `SecsStreamFunction` subclasses. See
[`interop/README.md`](interop/README.md).
Reference in New Issue View Git Blame Copy Permalink