diff --git a/README.md b/README.md index bb15a9f..506e259 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Everything runs in Docker — no compiler or build tools on the host. ```bash docker compose run --rm builder # configure + compile -docker compose run --rm tests # 445 cases / 2 753 assertions +docker compose run --rm tests # 473 cases / 3 087 assertions docker compose up --no-deps server client # live two-container demo ``` @@ -28,6 +28,46 @@ through the data model. Watch the logs interleave. --- +## Integrating your tool (pick a tier) + +Three ways in, same engine underneath: + +1. **Python, no SEMI knowledge** — run the `secs_gemd` daemon and + `pip install` the pure-Python client in [clients/python](clients/python): + + ```python + from secsgem_client import Equipment + + eq = Equipment("localhost:50051") + eq.set(ChamberPressure=2.5) # host sees it on its next poll + eq.fire("ProcessStarted") # S6F11, report auto-assembled + + @eq.on("START") # host remote commands -> your function + def start(cmd): + run_recipe(cmd.params.get("PPID")) + + eq.listen() + ``` + + A complete tool is ~25 lines: [clients/python/examples/mini_tool.py](clients/python/examples/mini_tool.py). + +2. **Any language over gRPC** — `secs_gemd` exposes the name-based API in + [proto/secsgem/v1/equipment.proto](proto/secsgem/v1/equipment.proto) + (variables, events, alarms, control state, health stream, and the + host-command stream with the SEMI-conformant HCACK-4 contract). The + daemon owns the durable HSMS link: your tool software can restart + without the fab host ever noticing. + +3. **Embedded C++** — construct a `gem::EquipmentRuntime`, call the + per-capability `register_*` functions (or `register_default_handlers` + for all of GEM), and wire behaviour with `commands.set_handler`. + `apps/secs_server.cpp` is the ~110-line canonical example. + +Status and remaining work for the daemon/client track: +[docs/DAEMON_ROADMAP.md](docs/DAEMON_ROADMAP.md). + +--- + ## Documentation map | File | What it covers | @@ -50,8 +90,8 @@ through the data model. Watch the logs interleave. ## Testing -- **Unit + integration** — `docker compose run --rm tests` runs 445 - cases / 2 753 assertions across every store, FSM, codec, parser, and +- **Unit + integration** — `docker compose run --rm tests` runs 473 + cases / 3 087 assertions across every store, FSM, codec, parser, and persistence path. - **Live conformance harness** — 47 wire-level checks against the passive server. @@ -59,11 +99,20 @@ through the data model. Watch the logs interleave. (55 checks), and Wireshark's HSMS dissector (69 frames, 0 malformed). - **Soak + fuzz** — 100 000-op property test; libFuzzer with ASan + UBSan over `secs2::decode` and the SML parser, 0 crashes. +- **Daemon** — `secs_gemd_tests` exercises the gRPC service over real + in-process channels (125 assertions), in Release and under + ThreadSanitizer; `interop/daemon_interop.py` and + `interop/pyclient_interop.py` prove the gRPC↔HSMS bridge and the + published Python client against a live daemon with secsgem-py as host. +- **One command for all of it** — `tools/run_interop.sh` runs every + validation step (build, both unit suites, secsgem-py host, C++ + conformance, Python client, daemon bridge, spool restart, tshark, + secs4java8) with a PASS/FAIL summary. - **Config validation** — `secs_server --validate-config` rejects malformed YAML before startup. - **CI** — [Gitea Actions](.gitea/workflows/ci.yml) runs the full suite plus a `-fsanitize=thread` lane on every push to `main`; all - 445 cases pass clean under TSan. + 473 cases pass clean under TSan. Exact commands, exit codes, and per-standard test counts are in [docs/PROOFS.md](docs/PROOFS.md); the rationale behind the external diff --git a/docs/30_repository_tour.md b/docs/30_repository_tour.md index 26f8404..c07e76c 100644 --- a/docs/30_repository_tour.md +++ b/docs/30_repository_tour.md @@ -57,7 +57,7 @@ secs-gem/ │ └── fuzz_sml_parse.cpp libFuzzer harness for try_parse_sml. │ ├── tests/ doctest unit + integration tests. -│ └── test_*.cpp 50 files, 445 cases, 2753 assertions. +│ └── test_*.cpp 55 files, 473 cases, 3087 assertions. │ ├── data/ YAML configs (the spec-as-data). │ ├── messages.yaml SECS-II message catalog (164 msgs). @@ -154,7 +154,9 @@ lines). Each binary lives in `build/` after `cmake --build`. | `secs_conformance` | [`apps/secs_conformance.cpp`](../apps/secs_conformance.cpp) | 47 wire-level conformance checks against a live server. | | `secs_interop_probe` | [`apps/secs_interop_probe.cpp`](../apps/secs_interop_probe.cpp) | Active host probing a secsgem-py passive equipment. | | `secs_bench` | [`apps/secs_bench.cpp`](../apps/secs_bench.cpp) | Throughput / latency / memory harness. | -| `secsgem_tests` | All `tests/*.cpp` | The 445-case doctest binary. | +| `secsgem_tests` | All `tests/*.cpp` | The 473-case doctest binary. | +| `secs_gemd` | `apps/secs_gemd.cpp` + `proto/secsgem/v1` | The gRPC daemon: HSMS equipment + name-based tool API. | +| `secs_gemd_tests` | `tests/test_daemon_service.cpp` | In-process gRPC service tests (built when grpc++ is). | | `fuzz_secs2_decode` | [`apps/fuzz_secs2_decode.cpp`](../apps/fuzz_secs2_decode.cpp) | libFuzzer (clang only, opt-in `-DSECSGEM_FUZZ=ON`). | | `fuzz_sml_parse` | [`apps/fuzz_sml_parse.cpp`](../apps/fuzz_sml_parse.cpp) | libFuzzer for the SML parser. | @@ -196,7 +198,7 @@ standard CMake. ## Test layout -50 test files; 445 test cases; 2 753 assertions. One file per +50 test files; 473 test cases; 3 087 assertions. One file per concern. Naming is `test_.cpp` consistently: - `test_secs2.cpp`, `test_e5_kat.cpp`, `test_sml.cpp`, diff --git a/docs/31_spec_as_data_and_codegen.md b/docs/31_spec_as_data_and_codegen.md index d55a15e..3edbea7 100644 --- a/docs/31_spec_as_data_and_codegen.md +++ b/docs/31_spec_as_data_and_codegen.md @@ -134,11 +134,21 @@ ceids: - {id: 300, name: ProcessStarted} alarms: - - {id: 1, alcd: 0x84, text: "Chamber pressure above threshold"} + # `category` is the lower-7 ALCD severity bitmap; bit 7 (set/clear) is + # applied at emit time and must NOT be in YAML. `name` is an optional + # local key for name-based APIs (the gRPC daemon / Python client). + - {id: 1, name: pressure_high, category: 4, + text: "Chamber pressure above threshold"} host_commands: - {name: START, ack: Accept, emit_ceid: 300} - {name: FAULT, ack: Accept, set_alarm: 1} + +# Role bindings: which of the ids above the engine's built-in behaviours +# target (control-state/clock SVID refresh, CJ state CEIDs). +roles: + control_state_svid: 1 + clock_svid: 2 ``` Loaded at startup by `config::load_equipment`. Every key under diff --git a/docs/35_state_machines_and_dispatch.md b/docs/35_state_machines_and_dispatch.md index 4ce6dec..ddfd479 100644 --- a/docs/35_state_machines_and_dispatch.md +++ b/docs/35_state_machines_and_dispatch.md @@ -59,7 +59,9 @@ router->on(2, 41, [model](const auto& m) { return messages::s2f42(ack); }); -// ...one per S/F pair. apps/secs_server.cpp registers ~50. +// ...one per S/F pair. The default GEM set (56) lives in +// src/gem/default_handlers.cpp, decomposed into 15 per-capability +// register_* functions; register_default_handlers(runtime) wires them all. ``` The `examples/pvd_tool/main.cpp` §6 register 51 handlers in ~460 diff --git a/docs/40_building_running_demo.md b/docs/40_building_running_demo.md index b508a8e..8977d04 100644 --- a/docs/40_building_running_demo.md +++ b/docs/40_building_running_demo.md @@ -53,7 +53,9 @@ build/ ├── secs_conformance 47-check conformance harness ├── secs_interop_probe active host probing secsgem-py equipment ├── secs_bench throughput/latency bench -├── secsgem_tests the 445-case doctest binary +├── secsgem_tests the 473-case doctest binary +├── secs_gemd gRPC daemon: HSMS equipment + name-based tool API +├── secs_gemd_tests in-process gRPC service tests (when grpc++ present) └── pvd_tool worked PVD-tool example ``` @@ -74,8 +76,8 @@ Runs `secsgem_tests` end-to-end. Expected output: [doctest] doctest version is "2.4.11" [doctest] run with "--help" for options =============================================================================== -[doctest] test cases: 445 | 445 passed | 0 failed | 0 skipped -[doctest] assertions: 2753 | 2753 passed | 0 failed | +[doctest] test cases: 473 | 473 passed | 0 failed | 0 skipped +[doctest] assertions: 3087 | 3087 passed | 0 failed | [doctest] Status: SUCCESS! ``` diff --git a/docs/50_api_messages_yaml_reference.md b/docs/50_api_messages_yaml_reference.md index 67e2437..bb9823c 100644 --- a/docs/50_api_messages_yaml_reference.md +++ b/docs/50_api_messages_yaml_reference.md @@ -98,6 +98,32 @@ often the right answer — this chapter helps you find which header. `include/secsgem/gem/store/` — 18 per-domain stores. See chapter [32](32_stores_and_the_data_model.md) for the full table. +Engine-owner and default-behaviour entry points (added with the daemon +track; see [DAEMON_ROADMAP.md](DAEMON_ROADMAP.md)): + +```cpp +#include "secsgem/gem/runtime.hpp" // EquipmentRuntime: owns + // io_context + Server + model + control FSM + Router; thread-safe + // set_variable/emit_event/set_alarm/clear_alarm, on_command hook, + // read_sync (the standard cross-thread read), control-state mirror, + // add_control_state_observer / add_link_observer. +#include "secsgem/gem/default_handlers.hpp" // the 56 GEM handlers as 15 + // per-capability register_* functions + register_default_handlers. +#include "secsgem/gem/handler_slot.hpp" // primary + observer handler slots +#include "secsgem/gem/name_index.hpp" // name -> VID/CEID resolution +``` + +### `secsgem::daemon` — the gRPC vendor surface + +```cpp +#include "secsgem/daemon/equipment_service.hpp" // EquipmentService: the + // proto/secsgem/v1 Equipment service over an EquipmentRuntime + // (SetVariables/GetVariables/FireEvent/SetAlarm/ClearAlarm/ + // GetControlState/RequestControlState/WatchHealth/Subscribe/ + // CompleteCommand). Built into apps/secs_gemd.cpp; the Python + // client in clients/python wraps the same proto. +``` + ### `secsgem::config` — YAML loader + validator (chapter 31, 36) ```cpp diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 9a9f184..bbf5b67 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -38,13 +38,25 @@ doc covers. --- -## 2. The five layers +## 2. The layers ``` ┌─────────────────────────────────────────────────────────────────┐ +│ vendor surfaces (out-of-process) │ +│ secs_gemd gRPC daemon (proto/secsgem/v1) ← clients/python │ +│ secsgem-client and any-language gRPC stubs │ +├─────────────────────────────────────────────────────────────────┤ │ apps/ (your main.cpp lives here) │ -│ secs_server, secs_client, secs_conformance, secs_bench, │ -│ fuzz_*, secs_interop_probe │ +│ secs_server, secs_client, secs_gemd, secs_conformance, │ +│ secs_bench, fuzz_*, secs_interop_probe │ +├─────────────────────────────────────────────────────────────────┤ +│ gem::EquipmentRuntime + register_* capability functions │ +│ ───────────────────────────────────────────────────── │ +│ Runtime: owns io_context + Server + model + control FSM + │ +│ Router; thread-safe set/emit/alarm API, on_command hook, │ +│ read_sync, control-state mirror, link/state observers │ +│ default_handlers: the 56 GEM handlers as 15 per-capability │ +│ register_* fns (ids bound via the config's roles: block) │ ├─────────────────────────────────────────────────────────────────┤ │ gem::Router + gem::EquipmentDataModel │ │ ───────────────────────────────────────── │ @@ -469,7 +481,7 @@ contract has no locks; adding any would diverge from it. | How the Router dispatches | `gem/router.hpp` | | How a store implements persistence | `gem/store/spool.hpp` (smallest), `gem/store/process_jobs.hpp` (richest) | | How an FSM is structured | `gem/process_job_state.hpp`, `src/gem/process_job_state.cpp` | -| How the application wires it all | `apps/secs_server.cpp` (the canonical example, ~1200 lines) | +| How the application wires it all | `gem::EquipmentRuntime` + `register_default_handlers` (apps/secs_server.cpp is now a ~110-line thin main over them) | | How a customer would write main() | `examples/pvd_tool/main.cpp` (the worked vendor example) | | How thread-safety works | `tests/test_thread_safety.cpp`, INTEGRATION.md §3 | | How E84 timers integrate with asio | `gem/e84_asio_timers.hpp` (the canonical I/O-adapter pattern) | diff --git a/docs/COMPLIANCE.md b/docs/COMPLIANCE.md index cbe97f8..23b83a9 100644 --- a/docs/COMPLIANCE.md +++ b/docs/COMPLIANCE.md @@ -346,7 +346,7 @@ walks ~20 SECS transactions end-to-end: 23. `S1F15`/`S1F16` Request Offline. 24. `Separate.req` → clean close on both sides. -Unit tests: **445 cases / 2753 assertions pass** (`docker compose run --rm tests`). +Unit tests: **473 cases / 3087 assertions pass** (`docker compose run --rm tests`). The suite includes integration tests that drive a real `hsms::Connection` over a loopback socket pair to verify the E37 §7.2 / §7.4 / §7.7 edge cases — not just the happy path. diff --git a/docs/INTEGRATION.md b/docs/INTEGRATION.md index 1532a28..8f55a89 100644 --- a/docs/INTEGRATION.md +++ b/docs/INTEGRATION.md @@ -17,6 +17,23 @@ how those two halves meet. --- +## 0. Three ways to integrate — pick your tier + +This guide covers the **embedded C++** path. Since the daemon track +landed there are two higher-level options that need no C++ at all: + +| Tier | You write | Best for | +|---|---|---| +| **Python client** ([clients/python](../clients/python)) | ~25 lines of Python against `secs_gemd` | new tools, lab/R&D, fastest start | +| **gRPC, any language** ([proto/secsgem/v1](../proto/secsgem/v1/equipment.proto)) | a thin client in Go/C#/Java/… | existing controllers, multi-process tools | +| **Embedded C++** (this guide) | a main() over `gem::EquipmentRuntime` | in-process integration, custom transports | + +In the daemon tiers `secs_gemd` owns the durable HSMS link — your tool +software can crash/upgrade/restart without the fab host noticing. See +[DAEMON_ROADMAP.md](DAEMON_ROADMAP.md) for status. + +--- + ## 1. What you get vs. what you build ``` diff --git a/docs/PROOFS.md b/docs/PROOFS.md index 66aa560..5a0058c 100644 --- a/docs/PROOFS.md +++ b/docs/PROOFS.md @@ -5,7 +5,7 @@ implements what [COMPLIANCE.md](COMPLIANCE.md) claims. | # | Command | What it proves | |---|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------| -| 1 | `docker compose run --rm tests` | **445 test cases / 2 753 assertions** pass: every store, FSM, codec, parser, persistence path | +| 1 | `docker compose run --rm tests` | **473 test cases / 3 087 assertions** pass: every store, FSM, codec, parser, persistence path | | 2 | `docker compose run --rm builder /app/build/secs_conformance --host server --port 5000` | **47 wire-level conformance checks** PASS against a live passive equipment | | 3 | `docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py --host server` | **31 interop checks** PASS against secsgem-py 0.3.0 (the Python reference impl) | | 4 | `SECSGEM_ROBUSTNESS_SOAK=1 docker compose run --rm builder /app/build/secsgem_tests -tc='*soak*'` | **100 000 random tool operations** execute with all invariants and persistence round-trips holding | @@ -16,7 +16,7 @@ implements what [COMPLIANCE.md](COMPLIANCE.md) claims. CI ([Gitea Actions](.gitea/workflows/ci.yml)) runs a Release build + full suite and a separate `-fsanitize=thread` lane on every push to -`main`. All 445 cases / 2 753 assertions pass under TSan clean. +`main`. All 473 cases / 3 087 assertions pass under TSan clean. ## Per-standard test coverage @@ -41,10 +41,13 @@ Every claimed standard has dedicated tests. Counts are | **E157** — module process tracking | `test_modules` | 5 | | **E84** — parallel I/O + timers | `test_e84`, `test_e84_ports`, `test_e84_timers`, `test_e84_asio_timers` | 27 | | Persistence + cross-cutting | `test_job_persistence`, `test_persistence_upgrade`, `test_wire_ceid_emission`, `test_gem300_scenario`, `test_live_gem300`, `test_thread_safety`, `test_metrics_prometheus`, `test_robustness_fuzz` | 32 | -| **Total** | | **445** | +| Runtime / handlers / daemon glue | `test_runtime`, `test_default_handlers`, `test_handler_conformance` (53-handler sweep + golden frames), `test_name_index`, loader/validator additions | 28 | +| **Total** | | **473** | `docker compose run --rm builder /app/build/secsgem_tests --list-test-cases | wc -l` -currently reports 445. +currently reports 473. The gRPC daemon has its own binary on top: +`secs_gemd_tests` (in-process gRPC service tests, 125 assertions, also run +under ThreadSanitizer in CI). ## Categories of evidence diff --git a/docs/VERIFICATION.md b/docs/VERIFICATION.md index c9c9549..3417863 100644 --- a/docs/VERIFICATION.md +++ b/docs/VERIFICATION.md @@ -8,7 +8,7 @@ push to `main`. | Channel | Source of independence | |----------------------------------|-------------------------------------------------------| -| 445 unit/integration tests | Internal | +| 473 unit/integration tests | Internal | | 47 conformance harness checks | Internal | | **SEMI E5 KAT** | **External — standards body's encoding rules** | | **Wireshark HSMS dissector** | **External — independent network-protocol authors** | diff --git a/interop/README.md b/interop/README.md index a173247..0c7de3f 100644 --- a/interop/README.md +++ b/interop/README.md @@ -24,6 +24,12 @@ involve a network peer — they're either pure codec round-trips (KAT) or coverage-guided fuzzing. Listed here so the full external proof inventory lives in one place. +## One command for everything + +`tools/run_interop.sh` (from the repo root) runs every validation step — +build, both unit suites, all the harnesses below, tshark, and secs4java8 — +with a PASS/FAIL summary. `SKIP_SECS4J=1` skips the Java image build. + ## Running each validator ### secsgem-py — secsgem-py active host → C++ server @@ -34,6 +40,29 @@ docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py \ --host server --port 5000 --session-id 0 ``` +### daemon bridge — gRPC tool + secsgem-py host → secs_gemd + +```sh +docker compose up -d --no-deps gemd +docker compose run --rm --no-deps interop python3 daemon_interop.py \ + --grpc gemd:50051 --hsms-host gemd +``` + +Both faces of the daemon at once: 20 checks proving gRPC SetVariables/ +FireEvent/SetAlarm reach the reference host as S6F11/S5F1 over HSMS, and +the HCACK-4 command loop (host S2F41 → tool stream → completion event). + +### Python client — published secsgem-client package → secs_gemd + +```sh +docker compose up -d --no-deps gemd +docker compose run --rm --no-deps -e PYTHONPATH=/app/clients/python interop \ + python3 pyclient_interop.py --grpc gemd:50051 --hsms-host gemd +``` + +13 checks driving the PUBLISHED Python API (eq.set / eq.fire / eq.alarm / +@eq.on) against a live daemon, with secsgem-py judging the wire. + ### secsgem-py — C++ host → secsgem-py equipment ```sh