raphael/secs-gem
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
91eec92b73bb6afcbe2570f68991d0e4fc97dca5
secs-gem/README.md
T
raphael 195ecc689f docs: GLOSSARY + FAQ + interop README refresh + doc-map fixes 
Fills four documentation gaps surfaced by the doc audit:

1. README "Documentation map" was missing VERIFICATION.md (the file
   that backs the proof-of-feature-completeness claims) and is now
   pointing at the new files added in this commit too — ARCHITECTURE,
   GLOSSARY, FAQ, examples/pvd_tool/ (the last two land next).

2. interop/README.md only documented secsgem-py.  Three of the five
   external validators (tshark, secs4j, libFuzzer) plus the E5 KAT
   were invisible from the directory's own README.  Rewritten as a
   complete index — what's external, what each catches, how to run,
   what bugs they've already surfaced, when to add a new validator.

3. GLOSSARY.md is new.  Every SEMI acronym used in the codebase or
   the docs gets one row: SVID, DVID, CEID, RPTID, ALID, ECID, PPID,
   MID, CARRIERID, PRJOBID, CTLJOBID, SUBSTID, OBJSPEC, OBJTYPE,
   MDLN, SOFTREV, EQPTYP, DATAID + every ACK code (COMMACK, ONLACK,
   OFLACK, HCACK, CMDA, ACKC5-7-10, DRACK, LRACK, ERACK, EAC, TIACK,
   GRANT, ALCD, OBJACK) + stream/function shorthand + HSMS terms +
   T-timers + E84 signals + the standards lineup + codebase shortcuts
   ("the model", "the router", "the proof", etc.).  Cuts week-1
   onboarding time.

4. FAQ.md is new.  Canonical answers to the questions that come up
   once per integration: why HSMS unencrypted, SVID vs DVID, PJ vs
   CJ, who fires FSM transitions, what runs on which thread, how to
   add a new SECS-II message, ASCII vs Binary, common MES quirks,
   how spool works, robustness fuzz vs libFuzzer, conformance vs
   interop, what's not implemented.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-09 16:43:45 +02:00

257 lines
15 KiB
Markdown
Raw Blame History

# secs-gem
A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully
containerized. Every behavioural rule lives in YAML; the C++ is the
engine that reads them. Implements **all of E4, E5, E30, E37
(SS + GS), E39, E40, E42, E84, E87, E90, E94, E116, E120, E148,
E157**.
> **License: proprietary — see [LICENSE](LICENSE).** No use, copy,
> compile, evaluate, benchmark, or deploy without a written license
> from the copyright holder. Contact `raphael@maenle.net` for
> commercial licensing, evaluation terms, or fab deployment.
---
## Proof of feature-completeness
"Feature-complete" is a claim that the code must prove, not the
README. These five commands are the proof. If they all exit zero on
a fresh clone, the codebase implements what
[COMPLIANCE.md](COMPLIANCE.md) claims.
| # | Command | What it proves |
|---|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
| 1 | `docker compose run --rm tests` | **426 test cases / 2 557 assertions** pass: every store, FSM, codec, parser, persistence path |
| 2 | `docker compose run --rm builder /app/build/secs_conformance --host server --port 5000` | **47 wire-level conformance checks** PASS against a live passive equipment |
| 3 | `docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py --host server` | **24 interop checks** PASS against secsgem-py 0.3.0 (the Python reference impl) |
| 4 | `SECSGEM_ROBUSTNESS_SOAK=1 docker compose run --rm builder /app/build/secsgem_tests -tc='*soak*'` | **100 000 random tool operations** execute with all invariants and persistence round-trips holding |
| 5 | `docker compose run --rm builder /app/build/secs_server --validate-config --config /app/data/equipment.yaml --state-table /app/data/control_state.yaml --pj-state-table /app/data/process_job_state.yaml --cj-state-table /app/data/control_job_state.yaml` | Every shipped YAML config passes structural + referential validation |
| 6 | `docker compose run --rm builder bash /app/interop/tshark_validate.sh` | **69 HSMS frames** dissected by Wireshark's HSMS dissector (independent third codec) with no malformed packets |
| 7 | `bash interop/secs4j_validate.sh` | **55 cross-validation checks** PASS against [secs4java8](https://github.com/kenta-shimizu/secs4java8) (independent Java implementation), covering S1/S2/S3/S5/S6/S7/S10/S14/S16, the full E40 PJ body, dynamic event reports + unsolicited S6F11 / S5F1 observation, alarm management, spool, PP management, terminal services, limits, trace, E39, and the GEM 300 streams secsgem-py couldn't easily drive |
| 8 | `cmake -B build-fuzz -DSECSGEM_FUZZ=ON && build-fuzz/fuzz_secs2_decode -max_total_time=60` | **~70 000 random inputs** through `secs2::decode`, **~285 000** through `try_parse_sml`, ASan + UBSan + libFuzzer coverage, **0 crashes** |
Plus, on every push to `main`, [Gitea Actions](.gitea/workflows/ci.yml)
runs both a **Release build + full test suite** and a separate
**ThreadSanitizer lane** that builds with `-fsanitize=thread` and
fails on any race. All 426 cases / 2 557 assertions pass under TSan
clean.
### Per-standard test coverage
Every claimed standard has dedicated tests. Counts are
`grep -c TEST_CASE`; cross-cutting tests (e.g. `test_robustness_fuzz`,
`test_gem300_scenario`) exercise multiple standards in concert.
| Standard | Test files | Cases |
|-----------------------------------|-------------------------------------------------------------------------------------------|------:|
| **E5** — SECS-II encoding | `test_secs2`, `test_sml`, `test_messages`, `test_identifier_wildcards`, `test_fuzz` | 120 |
| **E5 §13** — exceptions | `test_exceptions`, `test_exception_persistence` | 16 |
| **E4** — SECS-I transport | `test_secsi`, `test_secsi_timers`, `test_secsi_tcp` | 27 |
| **E37** — HSMS (SS + GS) | `test_hsms`, `test_hsms_connection`, `test_hsms_timers`, `test_hsms_s9`, `test_hsms_gs`, `test_hsms_gs_integration`, `test_s9_fallback`, `test_concurrency` | 34 |
| **E30** — GEM core | `test_control_state`, `test_communication_state`, `test_host_handler`, `test_data_model`, `test_loader`, `test_config_validate` | 71 |
| **E40** — process jobs | `test_process_jobs` | 21 |
| **E94** — control jobs | `test_control_jobs` | 9 |
| **E42** — formatted PP | `test_e42_formatted_pp` | 6 |
| **E87** — carriers + load ports | `test_carriers`, `test_carrier_state`, `test_carrier_persistence`, `test_e87_wire_scenarios` | 27 |
| **E90** — substrate tracking | `test_substrates`, `test_substrate_persistence` | 21 |
| **E116** — EPT | `test_ept` | 7 |
| **E120 / E39** — common equip / object service | `test_cem_objects` | 3 |
| **E157** — module process tracking | `test_modules` | 5 |
| **E84** — parallel I/O + timers | `test_e84`, `test_e84_ports`, `test_e84_timers`, `test_e84_asio_timers` | 27 |
| Persistence + cross-cutting | `test_job_persistence`, `test_persistence_upgrade`, `test_wire_ceid_emission`, `test_gem300_scenario`, `test_live_gem300`, `test_thread_safety`, `test_metrics_prometheus`, `test_robustness_fuzz` | 32 |
| **Total** | | **426** |
A single command to see this live: `docker compose run --rm builder
/app/build/secsgem_tests --list-test-cases | wc -l` (currently 426).
---
## Quick start
Everything runs in Docker — no compiler or build tools on the host.
```bash
docker compose run --rm builder # configure + compile
docker compose run --rm tests # 426 cases / 2 557 assertions
docker compose up --no-deps server client # live two-container demo
```
The two-container demo walks ~24 SECS transactions end-to-end
through the data model. Watch the logs interleave.
---
## Documentation map
| File | What it covers |
|-----------------------------------------------|-------------------------------------------------------------------------|
| [COMPLIANCE.md](COMPLIANCE.md) | Per-capability audit against every SEMI standard implemented |
| [INTEGRATION.md](INTEGRATION.md) | Vendor-side tutorial: YAML → callbacks → production deploy |
| [ARCHITECTURE.md](ARCHITECTURE.md) | How the pieces fit + how to extend (new store / FSM / message) |
| [VERIFICATION.md](VERIFICATION.md) | Test plan for the external validators behind the proof table |
| [BENCHMARKS.md](BENCHMARKS.md) | Performance envelope (throughput, latency, memory) + how to re-run |
| [MES_INTEROP.md](MES_INTEROP.md) | Day-1 punch list to run against your commercial MES (60+ test IDs) |
| [SECURITY.md](SECURITY.md) | Concrete configs: nftables, stunnel, minisign, SIEM audit-log schema |
| [GLOSSARY.md](GLOSSARY.md) | SEMI vocabulary: SVID, CEID, PPID, ALCD, HCACK, T-timers, … |
| [FAQ.md](FAQ.md) | Common questions and their canonical answers |
| [examples/pvd_tool/](examples/pvd_tool/) | Worked example: a realistic fictional PVD tool, YAML + C++ wiring |
| [LICENSE](LICENSE) | Proprietary license terms |
---
## Architecture
The project is **spec-as-data**: the SEMI behavioural rules live in
YAML; the C++ is the engine that reads them.
```
┌──────────────────────────────────────────────────────────────┐
│ data/ │
│ messages.yaml SECS-II message catalog (164 msgs) │
│ control_state.yaml E30 §6.2 control transition table │
│ process_job_state.yaml E40 §6 PJ transition table │
│ control_job_state.yaml E94 §6 CJ transition table │
│ equipment.yaml SVIDs / DVIDs / ECIDs / CEIDs / │
│ alarms / recipes / commands │
└──────────────────────┬───────────────────────────────────────┘
│ (codegen at build, YAML loaded at startup)
▼
┌──────────────────────────────────────────────────────────────┐
│ apps/ │
│ secs_server passive equipment secs_bench perf │
│ secs_client active host secs_conformance │
│ secs_interop_probe │
└──────────────────────────────────────────────────────────────┘
secsgem::config loader.hpp + validate.hpp:
YAML -> data model, with multi-error validator
surfacing every issue at once (`--validate-config`)
secsgem::gem per-standard FSM + per-store persistence
(every store accepts v ∈ [1, kVersion] for
forward-compatible schema migrations).
EquipmentDataModel composes all stores.
Router (stream, function) -> handler.
Generated messages.hpp covers 164 SxFy.
secsgem::hsms Connection (Asio): HSMS-SS + HSMS-GS, all
T-timers enforced, auto S9F3/F5/F7/F9/F11.
secsgem::secsi SECS-I Protocol FSM (E4): T1/T2/T3/T4 enforced
in-FSM, TCP transport for tunnel testing.
secsgem::secs2 Item (variant), encode/decode, Message,
SML parser/printer.
secsgem::metrics Prometheus exporter (Registry + HTTP server).
```
---
## Adding a capability
The point of "spec-as-data" is that adding behaviour almost never
requires a C++ change.
### New SVID
```yaml
# data/equipment.yaml
svids:
- {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}
```
### New host command with side effects
```yaml
host_commands:
- {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}
```
### New state transition
```yaml
# data/control_state.yaml
transitions:
- {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept}
```
### New SECS-II message
```yaml
# data/messages.yaml
- id: S6F30
stream: 6
function: 30
w: true
builder: s6f30_something
parser: parse_s6f30
body:
kind: list
struct_name: Something
fields:
- {name: field_a, shape: {kind: scalar, item_type: U4}}
- {name: field_b, shape: {kind: scalar, item_type: ASCII}}
```
`docker compose run --rm builder` regenerates `messages.hpp`. The
typed builder, parser, and struct definition appear automatically.
Run `--validate-config` after every YAML edit.
---
## Production deployment
See [INTEGRATION.md](INTEGRATION.md) for the full vendor-side
tutorial — wiring sensors, plugging FSMs into the tool, persistence
layout, monitoring/observability, HSMS-GS multi-MES setup.
See [SECURITY.md](SECURITY.md) for concrete nftables / stunnel /
minisign / SIEM configs.
See [BENCHMARKS.md](BENCHMARKS.md) for the performance envelope —
roughly **140 k req/s S1F1**, **79 k req/s S1F3 (32 SVIDs)**, **572
k S6F11/s push**, **~450 bytes per PJ+CJ pair**. Three orders of
magnitude above typical fab tool load.
See [MES_INTEROP.md](MES_INTEROP.md) for the day-1 punch list to run
against your commercial MES before promoting from staging to a real
tool.
### Operational runbook (starter)
| Incident | First check | Mitigation |
|-------------------------------------|--------------------------------------|-------------------------------------------|
| HSMS connection flapping | T7 / T6 timer fires in logs | check MES reachability, network MTU |
| Spool depth growing | host MES connectivity / ACK rate | force-drain via S6F23, escalate to MES |
| State machine "stuck" | last state-change handler log line | host-issued offline + re-establish |
| Alarm storm | `AlarmRegistry::all()` snapshot | check upstream sensor; quench via S5F3 |
| Persistence dir growing unbounded | `du -s` + file count | sweep terminal-state records |
| Cross-tool inconsistency | `secsgem_tests` on canary tool | compare wire trace vs validator |
---
## Build details
The toolchain image (`Dockerfile`) is Ubuntu 24.04 with `g++-13`,
CMake, Ninja, `libasio-dev`, `libyaml-cpp-dev`, and Python 3 for the
codegen. doctest is fetched via CMake FetchContent. Build artifacts
live in a named Docker volume so the host filesystem stays clean.
Standalone Asio is used in header-only mode (`ASIO_STANDALONE`). No
Boost dependency.
### ThreadSanitizer
```bash
cmake -S . -B build-tsan -G Ninja -DCMAKE_BUILD_TYPE=Debug -DSECSGEM_TSAN=ON
cmake --build build-tsan
TSAN_OPTIONS=halt_on_error=1 build-tsan/secsgem_tests
```
Runs as a separate lane in CI. Catches data races in the io_context
strand contract documented in INTEGRATION.md §3.
---
## Interop
`interop/` contains the secsgem-py 0.3.0 cross-validation harness —
secsgem-py active host driving our C++ passive server, our C++
active host probing secsgem-py's passive equipment, and a raw GEM-300
harness that round-trips S3 (E87), S14 (E94), S16 (E40), S12 (wafer
maps) through hand-crafted `SecsStreamFunction` subclasses. See
[`interop/README.md`](interop/README.md).
Reference in New Issue View Git Blame Copy Permalink