docs: streamline tone across reference docs
tests / build-and-test (push) Successful in 2m7s
tests / thread-sanitizer (push) Successful in 2m35s
tests / tshark-dissector (push) Successful in 2m19s
tests / secs4j-interop (push) Successful in 36s
tests / libfuzzer (push) Successful in 3m8s

Tone pass across the non-tutorial markdown — README, PROOFS,
ARCHITECTURE, BENCHMARKS, COMPLIANCE, FAQ, MES_INTEROP, SECURITY,
and interop/README.  Three patterns came out:

- Bug-history war stories ("Past interop sweeps surfaced…",
  "What these harnesses caught: 1. Strict U-width parsing…").
- Chat-with-reader framing ("Don't skip TLS unless…", "Treat as a
  punch list", "If you're running in a pod…", "Misconfiguration
  incidents drop dramatically").
- Self-referential narration ("we ship", "our codec", "the
  codebase's most-tested layer", "three orders of magnitude above
  fab load", "the gift that keeps giving").

README also drops the standalone ThreadSanitizer subsection under
Build details (now a single line under the new Testing section).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-06-10 00:00:06 +02:00
parent d63c92166d
commit dae6bfd747
9 changed files with 130 additions and 219 deletions
+22 -30
View File
@@ -48,21 +48,26 @@ through the data model. Watch the logs interleave.
--- ---
## How it's proved ## Testing
"Feature-complete" is a claim that the code must prove, not the - **Unit + integration** — `docker compose run --rm tests` runs 445
README. On a fresh clone, **eight commands** demonstrate it: cases / 2 753 assertions across every store, FSM, codec, parser, and
unit + integration suite (445 cases / 2 753 assertions), a live persistence path.
conformance harness (47 wire-level checks), interop against - **Live conformance harness** — 47 wire-level checks against the
secsgem-py (31 checks) and secs4java8 (55 checks), a 100 000-op passive server.
soak property test, YAML config validation, Wireshark's HSMS - **Interop** — round-trips against secsgem-py (31 checks), secs4java8
dissector on a recorded pcap (69 frames, 0 malformed), and libFuzzer (55 checks), and Wireshark's HSMS dissector (69 frames, 0 malformed).
(ASan + UBSan, 0 crashes). Each command, exit code, and the - **Soak + fuzz** — 100 000-op property test; libFuzzer with ASan +
external-vs-internal breakdown live in [docs/PROOFS.md](docs/PROOFS.md). UBSan over `secs2::decode` and the SML parser, 0 crashes.
- **Config validation** — `secs_server --validate-config` rejects
malformed YAML before startup.
- **CI** — [Gitea Actions](.gitea/workflows/ci.yml) runs the full
suite plus a `-fsanitize=thread` lane on every push to `main`; all
445 cases pass clean under TSan.
CI runs the full suite plus a separate ThreadSanitizer lane on Exact commands, exit codes, and per-standard test counts are in
every push to `main` ([Gitea Actions](.gitea/workflows/ci.yml)); [docs/PROOFS.md](docs/PROOFS.md); the rationale behind the external
all 445 cases pass clean under `-fsanitize=thread`. validators is in [docs/VERIFICATION.md](docs/VERIFICATION.md).
--- ---
@@ -195,11 +200,9 @@ staging to a real tool.
## Deferred follow-ups ## Deferred follow-ups
- **asio `serial_port` adapter for SECS-I.** The SECS-I FSM - **asio `serial_port` adapter for SECS-I.** `secsi::Protocol` is
(`secsi::Protocol`) is implemented and tested end-to-end over the tested end-to-end over `secsi::TcpTransport`; the matching serial
asio TCP transport (`secsi::TcpTransport`). The serial driver — driver isn't written yet. Mirror `TcpTransport` to add it.
a `serial_port` mirror of `TcpTransport`, a few hundred lines —
hasn't been written. Mirror `TcpTransport` to add it.
--- ---
@@ -213,17 +216,6 @@ live in a named Docker volume so the host filesystem stays clean.
Standalone Asio is used in header-only mode (`ASIO_STANDALONE`). No Standalone Asio is used in header-only mode (`ASIO_STANDALONE`). No
Boost dependency. Boost dependency.
### ThreadSanitizer
```bash
cmake -S . -B build-tsan -G Ninja -DCMAKE_BUILD_TYPE=Debug -DSECSGEM_TSAN=ON
cmake --build build-tsan
TSAN_OPTIONS=halt_on_error=1 build-tsan/secsgem_tests
```
Runs as a separate lane in CI. Catches data races in the io_context
strand contract documented in [docs/INTEGRATION.md](docs/INTEGRATION.md) §3.
--- ---
## Interop ## Interop
@@ -231,7 +223,7 @@ strand contract documented in [docs/INTEGRATION.md](docs/INTEGRATION.md) §3.
Four independent external validators cross-check the codebase: Four independent external validators cross-check the codebase:
- **secsgem-py 0.3.0** (Python reference impl) — three harnesses - **secsgem-py 0.3.0** (Python reference impl) — three harnesses
under `interop/`: secsgem-py active host driving our C++ passive under `interop/`: secsgem-py active host driving the C++ passive
server (31 checks), C++ active host probing secsgem-py's passive server (31 checks), C++ active host probing secsgem-py's passive
equipment, and a raw GEM 300 harness round-tripping S3 / S14 / equipment, and a raw GEM 300 harness round-tripping S3 / S14 /
S16 / S12 through hand-crafted `SecsStreamFunction` subclasses. S16 / S12 through hand-crafted `SecsStreamFunction` subclasses.
+18 -25
View File
@@ -93,10 +93,8 @@ about format codes but lenient about U-widths in identifier fields
(per `messages_helpers::any_unsigned_first`) — that's how secsgem-py (per `messages_helpers::any_unsigned_first`) — that's how secsgem-py
interop works without breaking spec-correctness. interop works without breaking spec-correctness.
The codec is the most-tested layer in the codebase: 196 SEMI E5 KAT Coverage on the codec: 196 SEMI E5 KAT assertions, 120+ unit tests,
assertions, 120+ unit tests, plus libFuzzer with 70 000+ random libFuzzer with 70 000+ random inputs per minute under ASan + UBSan.
inputs per minute. Touch it carefully; it's the foundation
everything else stands on.
## 4. Transport (`hsms/`, `secsi/`) ## 4. Transport (`hsms/`, `secsi/`)
@@ -407,9 +405,9 @@ example to mirror.
### 9.5. New transport ### 9.5. New transport
`hsms::Connection` and `secsi::Protocol` are the two we ship. A `hsms::Connection` and `secsi::Protocol` are the two shipped
third (e.g. HSMS-over-TLS as a first-class thing, or HSMS over a transports. A third (e.g. HSMS-over-TLS as a first-class thing, or
sidecar IPC) follows the same contract: HSMS over a sidecar IPC) follows the same contract:
1. Accept a transport socket / endpoint. 1. Accept a transport socket / endpoint.
2. Expose `set_message_handler(...)`, `send_request(...)`, 2. Expose `set_message_handler(...)`, `send_request(...)`,
@@ -438,11 +436,10 @@ by:
- The ThreadSanitizer CI lane — every test under - The ThreadSanitizer CI lane — every test under
`-fsanitize=thread` `-fsanitize=thread`
If you're adding work that lives on another thread (sensor poll Work that lives on another thread (sensor poll loop, separate
loop, separate metrics scraper, signal handler), marshal back to metrics scraper, signal handler) must marshal back to the
the io_context with `asio::post(io.get_executor(), ...)`. Don't io_context via `asio::post(io.get_executor(), ...)`. The model
add locks; they'll diverge from the contract and the next contract has no locks; adding any would diverge from it.
contributor will be confused.
--- ---
@@ -480,23 +477,19 @@ contributor will be confused.
--- ---
## 13. What we deliberately don't do ## 13. Deliberate non-features
- **No DI framework, no service locator.** Stores are owned by the - **No DI framework, no service locator.** Stores are owned by the
model; the model is owned by your application; everything else is model; the model is owned by the application; everything else
passed in by reference. C++20 has no language-level DI, and adding passes by reference.
one to a codebase this size is overhead with no payoff.
- **No singleton state.** The model is a value, not a global. - **No singleton state.** The model is a value, not a global.
- **No std::shared_ptr-everywhere.** asio handlers extend the - **No std::shared_ptr-everywhere.** asio handlers extend the
lifetimes that need extending; the rest is owned by-value. Read lifetimes that need extending; the rest is owned by-value.
`Connection`'s lifetime contract in `hsms/connection.hpp` if you're `Connection`'s lifetime contract is documented in
ever in doubt. `hsms/connection.hpp`.
- **No exceptions across the API boundary** — the codec throws - **No exceptions across the API boundary.** The codec throws
`secs2::CodecError` internally, but every public accessor returns `secs2::CodecError` internally, but every public accessor returns
`std::optional` or returns a bool. Exceptions are reserved for `std::optional` or a bool. Exceptions are reserved for
programmer-error / corrupt-input paths. programmer-error / corrupt-input paths.
Every one of those constraints came from real review pressure on The TSan lane and the property fuzz depend on these constraints.
prior iterations. Pushing back on them is welcome but please read
the existing tests first; the codebase's architecture is what makes
the property fuzz and the TSan lane feasible.
+5 -9
View File
@@ -2,9 +2,8 @@
Numbers from `build/secs_bench --requests 20000 --concurrency 16` on Numbers from `build/secs_bench --requests 20000 --concurrency 16` on
Docker / Ubuntu 24.04 inside Docker Desktop on macOS (M-series), single Docker / Ubuntu 24.04 inside Docker Desktop on macOS (M-series), single
io_context thread. Treat as **rough envelope for capacity planning**, io_context thread. A rough capacity-planning envelope, not a
not lab-grade benchmarks; re-run on your target hardware before lab-grade benchmark re-run on target hardware before sizing.
sizing pods or VMs.
## Round-trip throughput / latency ## Round-trip throughput / latency
@@ -14,10 +13,8 @@ sizing pods or VMs.
| S1F3/F4 (32 SVIDs) | 20000 | 0.25 | ~79000 | 165 | 186 | 260 | | S1F3/F4 (32 SVIDs) | 20000 | 0.25 | ~79000 | 165 | 186 | 260 |
| S6F11 push (W=0) | 20000 | 0.03 | ~572000 | n/a | n/a | n/a | | S6F11 push (W=0) | 20000 | 0.03 | ~572000 | n/a | n/a | n/a |
**Read the table this way.** A real fab tool needs to handle tens to a A real fab tool sustains tens to a few hundred S6F11 events/second.
few hundred S6F11 events/second sustained. We're three orders of Throughput is not the bottleneck on this stack; latency tail under
magnitude above that on the push path, two orders above on synchronous
round-trips. Throughput is not the bottleneck; latency tail under
contention is. contention is.
## Memory footprint ## Memory footprint
@@ -50,8 +47,7 @@ docker compose run --rm builder /app/build/secs_bench \
--store-pairs 10000 --store-pairs 10000
``` ```
Output is markdown — pipe to a file and commit it to your CI so Output is markdown.
regressions show up as diffs.
## What this does NOT measure ## What this does NOT measure
+17 -27
View File
@@ -9,13 +9,12 @@ equipment model), **E157** (module process tracking), **E84** (parallel
I/O), **E148** (time synchronization), **E39** (object services), plus I/O), **E148** (time synchronization), **E39** (object services), plus
**E5 §13** wafer maps. **E5 §13** wafer maps.
> **Status.** Every GEM Fundamental and every GEM Additional capability > **Status.** Every GEM Fundamental and Additional capability that
> that E30 binds to a concrete SECS-II message set is implemented, and > E30 binds to a SECS-II message set is implemented end-to-end
> every GEM 300 standard the project sets out to cover is implemented > (state machine + store + wire messages + dispatch), and so is
> end-to-end (state machines + stores + wire messages + dispatch). See > every in-scope GEM 300 standard. §8 covers what "100%
> §8 for what "100% GEM-compliant" can and cannot honestly mean about a > GEM-compliant" can and cannot mean about a codebase; the README
> codebase, and the README "Deferred follow-ups" section for the > "Deferred follow-ups" section lists the non-shipped pieces.
> non-shipped pieces that aren't behavioural gaps.
Legend: Legend:
@@ -372,10 +371,9 @@ against vendor equipment as the first-line conformance probe.
## 7. Interoperability with external implementations ## 7. Interoperability with external implementations
Four independent external validators cross-check the codebase. None Four external validators cross-check the codebase. Three of them
of them shares code with us; three of them are not even C++. Full are not C++. Full description in [VERIFICATION.md](VERIFICATION.md);
test plan in [VERIFICATION.md](VERIFICATION.md); proof commands in proof commands in [PROOFS.md](PROOFS.md).
[PROOFS.md](PROOFS.md).
**secsgem-py 0.3.0** (Python reference implementation, Apache 2.0). **secsgem-py 0.3.0** (Python reference implementation, Apache 2.0).
Three harnesses under `interop/`: Three harnesses under `interop/`:
@@ -401,10 +399,9 @@ end-to-end against a second SECS implementation.
**Wireshark / tshark HSMS dissector** (independent network-protocol **Wireshark / tshark HSMS dissector** (independent network-protocol
authors). `interop/tshark_validate.sh` captures a pcap of the demo authors). `interop/tshark_validate.sh` captures a pcap of the demo
run, dissects with tshark's built-in HSMS dissector, asserts no run, dissects with tshark's built-in HSMS dissector, and asserts no
malformed-packet warnings and that every expected control + data malformed-packet warnings. 69 HSMS frames dissected cleanly per
frame parses. 69 HSMS frames dissected cleanly. This catches run. Catches framing bugs two SECS implementations might share but
framing bugs that two SECS implementations might *both* share but
that a third party reading the bytes would flag. that a third party reading the bytes would flag.
**libFuzzer + ASan + UBSan** (coverage-guided structural search). **libFuzzer + ASan + UBSan** (coverage-guided structural search).
@@ -414,16 +411,9 @@ AddressSanitizer + UndefinedBehaviorSanitizer. 60-second CI lanes
typically explore 200 000+ inputs through `secs2::decode` and typically explore 200 000+ inputs through `secs2::decode` and
1 400 000+ through `try_parse_sml`; 0 crashes, 0 ASan/UBSan reports. 1 400 000+ through `try_parse_sml`; 0 crashes, 0 ASan/UBSan reports.
Bugs surfaced and fixed across the four channels include: strict
per-width parsing rejected U1-encoded identifiers (SEMI E5 allows
`U1|U2|U4|U8`); PPBODY-as-ASCII was rejected; S1F23/F24 wasn't
implemented; S10F3 (host→equipment Terminal Display Single) wasn't
wired; one HSMS framing edge case caught by the tshark dissector;
several SML edge cases caught by libFuzzer.
--- ---
## 8. What "100% GEM-compliant" honestly means here ## 8. What "100% GEM-compliant" means here
Every GEM Fundamental and every GEM Additional capability that the E30 Every GEM Fundamental and every GEM Additional capability that the E30
specification defines with a concrete SECS-II message set is implemented, specification defines with a concrete SECS-II message set is implemented,
@@ -450,7 +440,7 @@ What this codebase does **not** demonstrate, and what a real
GEM-compliant. GEM-compliant.
In short: this is a **GEM-conformant runtime stack with the full GEM In short: this is a **GEM-conformant runtime stack with the full GEM
300 suite**, not a GEM-conformant *tool*. Pointing the runtime at a 300 suite**, not a GEM-conformant *tool*. A GEM-conformant tool is
real piece of equipment, populating the YAML files with the tool's this runtime plus an application populating the YAML with real
real SVIDs / ECIDs / alarms / capabilities / job behaviour, and wiring SVIDs / ECIDs / alarms / capabilities / job behaviour and wiring the
the application callbacks completes the picture. callbacks.
+1 -4
View File
@@ -1,9 +1,6 @@
# FAQ # FAQ
Questions we hear once per integration. Skim before you ask. If Questions we hear once per integration.
your question isn't here and isn't obvious from the other docs,
ask once — your question probably belongs in this file and we'll
add it.
## Why is HSMS unencrypted? ## Why is HSMS unencrypted?
+19 -28
View File
@@ -13,17 +13,14 @@ what a fab actually runs.** Real MES stacks include:
- **Eyelit MES** - **Eyelit MES**
Each one ships its own SECS/GEM stack with its own quirks. This doc Each one ships its own SECS/GEM stack with its own quirks. This doc
is the structured day-1 protocol your integration team runs against is the day-1 punch list your integration team runs against **your**
**your** MES before connecting a real tool. Treat it as a punch MES before connecting a real tool. Capture wire traces from every
list you tick off; capture wire traces from every step. step.
> **You can't skip this.** The in-repo `secs_conformance` harness + The in-repo `secs_conformance` and secsgem-py cross-validation prove
> the `interop/` secsgem-py cross-validation prove the codebase is the codebase is spec-conformant; they cannot prove the combination
> spec-conformant. They cannot prove the *combination* of (this of (this codebase, your YAML, your MES's choice of optional
> codebase, your YAML config, your MES's choice of optional behaviours) works.
> behaviours) works. Every gap surfaced in prior interop sweeps
> (S1F23/F24 missing, S10F3 direction confusion, lenient U-width
> parsing) was a real bug masked by passing internal tests.
## 0. Prerequisites ## 0. Prerequisites
@@ -50,13 +47,13 @@ wire-trace timestamp. This is your audit trail for the integration.
| ID | Test | Expected wire behaviour | Common quirks | | ID | Test | Expected wire behaviour | Common quirks |
|-------|-------------------------------------------------------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------| |-------|-------------------------------------------------------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------|
| T-01 | MES connects (active) → Select.req | Equipment replies Select.rsp(0=Ok); SELECTED on both sides | Some MES default session ID ≠ 0; override in our YAML | | T-01 | MES connects (active) → Select.req | Equipment replies Select.rsp(0=Ok); SELECTED on both sides | Some MES default session ID ≠ 0; override in equipment.yaml |
| T-02 | Idle Linktest cycle (5 min observation) | Linktest.req every N seconds from MES; equipment Linktest.rsp | Camstar uses 30s default; FactoryWorks 60s | | T-02 | Idle Linktest cycle (5 min observation) | Linktest.req every N seconds from MES; equipment Linktest.rsp | Camstar uses 30s default; FactoryWorks 60s |
| T-03 | MES sends Separate.req → graceful close | Equipment closes socket within 1s; no FIN_WAIT2 leak | Some MES expect equipment to close first | | T-03 | MES sends Separate.req → graceful close | Equipment closes socket within 1s; no FIN_WAIT2 leak | Some MES expect equipment to close first |
| T-04 | Equipment-initiated Linktest (set `Timers::linktest` to 10s) | MES replies Linktest.rsp | Mozaic ignores equipment-initiated; not a bug | | T-04 | Equipment-initiated Linktest (set `Timers::linktest` to 10s) | MES replies Linktest.rsp | Mozaic ignores equipment-initiated; not a bug |
| T-05 | MES disconnects TCP without Separate | Equipment detects closed socket, fires `closed_handler("eof")` | Watch for spool starting to fill | | T-05 | MES disconnects TCP without Separate | Equipment detects closed socket, fires `closed_handler("eof")` | Watch for spool starting to fill |
| T-06 | MES reconnects after T-05 → S6F25 if spool has content | Equipment auto-emits S6F25 with queued count | MES must support S6F25/F26 — older Wonderware doesn't | | T-06 | MES reconnects after T-05 → S6F25 if spool has content | Equipment auto-emits S6F25 with queued count | MES must support S6F25/F26 — older Wonderware doesn't |
| T-07 | T3 violation: MES sends primary but never replies | Equipment fires T3 → auto-emits S9F9 | Confirm S9F9 appears in trace, not just our logs | | T-07 | T3 violation: MES sends primary but never replies | Equipment fires T3 → auto-emits S9F9 | Confirm S9F9 appears in the wire trace, not just logs |
| T-08 | T7 violation: MES connects but never sends Select.req | Equipment closes after T7 with reason "T7" | | | T-08 | T7 violation: MES connects but never sends Select.req | Equipment closes after T7 with reason "T7" | |
| T-09 | Oversized frame: MES sends 17 MiB body | Equipment auto-emits S9F11 + closes | If MES doesn't generate this, skip | | T-09 | Oversized frame: MES sends 17 MiB body | Equipment auto-emits S9F11 + closes | If MES doesn't generate this, skip |
@@ -81,10 +78,10 @@ positions, or uses non-standard RPTID/DATAID widths.
| ID | Test | Expected | Common quirks | | ID | Test | Expected | Common quirks |
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| |-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
| R-01 | MES → S2F33 (define report 1000 over 3 SVIDs) | S2F34(DRACK=0=Accept) | MES may use U1 for RPTID; our parser accepts widths | | R-01 | MES → S2F33 (define report 1000 over 3 SVIDs) | S2F34(DRACK=0=Accept) | MES may use U1 for RPTID; the parser accepts any width|
| R-02 | MES → S2F35 (link CEID 300 ↔ RPTID 1000) | S2F36(LRACK=0=Accept) | | | R-02 | MES → S2F35 (link CEID 300 ↔ RPTID 1000) | S2F36(LRACK=0=Accept) | |
| R-03 | MES → S2F37 (enable CEID 300) | S2F38(ERACK=0=Accept) | Some MES send empty CEID list = enable-all | | R-03 | MES → S2F37 (enable CEID 300) | S2F38(ERACK=0=Accept) | Some MES send empty CEID list = enable-all |
| R-04 | Equipment fires CEID 300 (driven by `model->compose_reports_for(300)`) | MES → S6F12(ACKC6=0) | MES may take >T6 to reply — extend our T6 if needed | | R-04 | Equipment fires CEID 300 (driven by `model->compose_reports_for(300)`) | MES → S6F12(ACKC6=0) | MES may take >T6 to reply — extend T6 if needed |
| R-05 | MES → S6F15 (event report request) | Equipment → S6F16 with current values | | | R-05 | MES → S6F15 (event report request) | Equipment → S6F16 with current values | |
| R-06 | MES → S6F19/F21 (individual / annotated report request) | Equipment → S6F20 / S6F22 with current values | | | R-06 | MES → S6F19/F21 (individual / annotated report request) | Equipment → S6F20 / S6F22 with current values | |
| R-07 | MES → S2F33 with DATAID=0 (clear all reports) | S2F34(DRACK=0); all link bindings flushed | Older MES use S2F33 + empty body; check both forms | | R-07 | MES → S2F33 with DATAID=0 (clear all reports) | S2F34(DRACK=0); all link bindings flushed | Older MES use S2F33 + empty body; check both forms |
@@ -98,7 +95,7 @@ positions, or uses non-standard RPTID/DATAID widths.
| A-03 | MES → S5F7 (list enabled) | S5F8 contains ALID 1 | | | A-03 | MES → S5F7 (list enabled) | S5F8 contains ALID 1 | |
| A-04 | Equipment sets alarm 1 active | S5F1(ALCD=0x80 \| category) sent to MES → MES S5F2 ack | ALCD bit-7 must be SET, not cleared | | A-04 | Equipment sets alarm 1 active | S5F1(ALCD=0x80 \| category) sent to MES → MES S5F2 ack | ALCD bit-7 must be SET, not cleared |
| A-05 | Equipment clears alarm 1 | S5F1(ALCD=0x00 \| category) → S5F2 ack | | | A-05 | Equipment clears alarm 1 | S5F1(ALCD=0x00 \| category) → S5F2 ack | |
| A-06 | Equipment fires an alarm while alarm is *disabled* (S5F5 says no) | NO S5F1 wire frame — alarm registry tracks active, dispatcher gates | Easy to get wrong; covered by our tests | | A-06 | Equipment fires an alarm while alarm is *disabled* (S5F5 says no) | NO S5F1 wire frame — alarm registry tracks active, dispatcher gates | |
## 5. Remote control (E30 §6.15) ## 5. Remote control (E30 §6.15)
@@ -114,7 +111,7 @@ positions, or uses non-standard RPTID/DATAID widths.
| ID | Test | Expected | Common quirks | | ID | Test | Expected | Common quirks |
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| |-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
| P-01 | MES → S7F19 (list) | S7F20 with all PPIDs from `recipes:` YAML | | | P-01 | MES → S7F19 (list) | S7F20 with all PPIDs from `recipes:` YAML | |
| P-02 | MES → S7F5(PPID=RECIPE-A) | S7F6 with body (ASCII default; binary if recipe is binary) | PPBODY direction was our biggest interop bug | | P-02 | MES → S7F5(PPID=RECIPE-A) | S7F6 with body (ASCII default; binary if recipe is binary) | Check whether your MES expects ASCII or Binary PPBODY |
| P-03 | MES → S7F1 → S7F2(Accept) → S7F3 (PP send, new PPID) | S7F4(ACKC7=0); recipe in store | | | P-03 | MES → S7F1 → S7F2(Accept) → S7F3 (PP send, new PPID) | S7F4(ACKC7=0); recipe in store | |
| P-04 | MES → S7F17 (delete) | S7F18(ACKC7=0); recipe removed | | | P-04 | MES → S7F17 (delete) | S7F18(ACKC7=0); recipe removed | |
| P-05 | MES → S7F23 (E42 formatted PP send) | S7F24(ACKC7=0) | Many MES don't speak E42; OK to skip if unused | | P-05 | MES → S7F23 (E42 formatted PP send) | S7F24(ACKC7=0) | Many MES don't speak E42; OK to skip if unused |
@@ -123,7 +120,7 @@ positions, or uses non-standard RPTID/DATAID widths.
| ID | Test | Expected | Common quirks | | ID | Test | Expected | Common quirks |
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| |-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
| TS-01 | MES → S10F3 (terminal display single, host→equipment) | S10F4(ACKC10=0) | OUR codebase's bug fix — confirm direction is right | | TS-01 | MES → S10F3 (terminal display single, host→equipment) | S10F4(ACKC10=0) | Confirm direction matches S10F3 host→equipment |
| TS-02 | MES → S10F5 (terminal display multi) | S10F6(ACKC10=0) | | | TS-02 | MES → S10F5 (terminal display multi) | S10F6(ACKC10=0) | |
| TS-03 | Equipment → S10F1 (operator request, equipment→host) | MES → S10F2(ACKC10=0) | Some MES don't accept S10F1 at all; document policy | | TS-03 | Equipment → S10F1 (operator request, equipment→host) | MES → S10F2(ACKC10=0) | Some MES don't accept S10F1 at all; document policy |
@@ -152,12 +149,12 @@ positions, or uses non-standard RPTID/DATAID widths.
| ID | Test | Expected | Common quirks | | ID | Test | Expected | Common quirks |
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| |-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
| K-01 | MES → S2F17 (clock read) | S2F18(YYYYMMDDhhmmsscc) — 16 chars | Some MES expect 14-char; we accept both | | K-01 | MES → S2F17 (clock read) | S2F18(YYYYMMDDhhmmsscc) — 16 chars | Some MES expect 14-char; the library accepts both |
| K-02 | MES → S2F31 (clock set) | S2F32(TIACK=0=Accept) | Do this in a maintenance window only | | K-02 | MES → S2F31 (clock set) | S2F32(TIACK=0=Accept) | Do this in a maintenance window only |
| K-03 | MES → S2F29 (EC namelist) | S2F30 with all ECIDs | | | K-03 | MES → S2F29 (EC namelist) | S2F30 with all ECIDs | |
| K-04 | MES → S2F13 (EC values) | S2F14 with current values | | | K-04 | MES → S2F13 (EC values) | S2F14 with current values | |
| K-05 | MES → S2F15 (EC set within range) | S2F16(EAC=0=Accept); value reflected on next S2F13 | | | K-05 | MES → S2F15 (EC set within range) | S2F16(EAC=0=Accept); value reflected on next S2F13 | |
| K-06 | MES → S2F15 (EC set OUT of range) | S2F16(EAC=1=Denied_OutOfRange) | Triggers our `min_str`/`max_str` range check | | K-06 | MES → S2F15 (EC set OUT of range) | S2F16(EAC=1=Denied_OutOfRange) | Range enforced via `min_str`/`max_str` in equipment.yaml |
## 11. Soak (4-24 hours) ## 11. Soak (4-24 hours)
@@ -187,8 +184,7 @@ floor:
## 13. Known MES quirks worth pre-empting ## 13. Known MES quirks worth pre-empting
Compiled from prior fab integrations. Not exhaustive; treat as Quirks observed in prior fab integrations. Not exhaustive.
search-priors when something doesn't behave the way you expect.
- **Camstar Opcenter**: ALCD bit-7 sometimes inverted in their - **Camstar Opcenter**: ALCD bit-7 sometimes inverted in their
internal model; double-check the alarm wire trace. Linktest at internal model; double-check the alarm wire trace. Linktest at
@@ -199,8 +195,8 @@ search-priors when something doesn't behave the way you expect.
flow may need explicit S6F23 from MES. Older versions don't flow may need explicit S6F23 from MES. Older versions don't
speak E42 (S7F23 family) at all. speak E42 (S7F23 family) at all.
- **Mozaic**: Sometimes sends S2F33 with DATAID encoded as U1 - **Mozaic**: Sometimes sends S2F33 with DATAID encoded as U1
where the spec allows U1-U8 — make sure our lenient parser is on where the spec allows U1-U8. The library's identifier parser
(it is by default since the secsgem-py interop work). accepts any unsigned width.
- **Inficon FabGuard**: Strict on S1F1 — expects MDLN and SOFTREV - **Inficon FabGuard**: Strict on S1F1 — expects MDLN and SOFTREV
in ASCII even if the YAML uses U-types for MDLN elsewhere. in ASCII even if the YAML uses U-types for MDLN elsewhere.
- **CMNavigo**: Expects equipment to initiate S1F13 within 5 s of - **CMNavigo**: Expects equipment to initiate S1F13 within 5 s of
@@ -217,8 +213,3 @@ file it via `raphael@maenle.net` with:
3. Wire trace excerpt (pcap clip is fine) 3. Wire trace excerpt (pcap clip is fine)
4. Expected vs actual behaviour 4. Expected vs actual behaviour
5. Your equipment YAML + secs-gem commit SHA 5. Your equipment YAML + secs-gem commit SHA
Bugs surfaced through this process are how we got
[S1F23/F24](interop/README.md), [S10F3 direction
fix](interop/README.md), and the lenient identifier-width parser.
The interop sweep is the gift that keeps giving.
+21 -27
View File
@@ -1,9 +1,7 @@
# Proof of feature-completeness # Proofs
"Feature-complete" is a claim that the code must prove, not the Eight commands. If they all exit zero on a fresh clone, the codebase
README. These eight commands are the proof. If they all exit zero implements what [COMPLIANCE.md](COMPLIANCE.md) claims.
on a fresh clone, the codebase implements what
[COMPLIANCE.md](COMPLIANCE.md) claims.
| # | Command | What it proves | | # | Command | What it proves |
|---|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------| |---|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
@@ -16,11 +14,9 @@ on a fresh clone, the codebase implements what
| 7 | `bash interop/secs4j_validate.sh` | **55 cross-validation checks** PASS against [secs4java8](https://github.com/kenta-shimizu/secs4java8) (independent Java implementation), covering S1/S2/S3/S5/S6/S7/S10/S14/S16, the full E40 PJ body, dynamic event reports + unsolicited S6F11 / S5F1 observation, alarm management, spool, PP management, terminal services, limits, trace, E39, and the GEM 300 streams secsgem-py couldn't easily drive | | 7 | `bash interop/secs4j_validate.sh` | **55 cross-validation checks** PASS against [secs4java8](https://github.com/kenta-shimizu/secs4java8) (independent Java implementation), covering S1/S2/S3/S5/S6/S7/S10/S14/S16, the full E40 PJ body, dynamic event reports + unsolicited S6F11 / S5F1 observation, alarm management, spool, PP management, terminal services, limits, trace, E39, and the GEM 300 streams secsgem-py couldn't easily drive |
| 8 | `cmake -B build-fuzz -DSECSGEM_FUZZ=ON && build-fuzz/fuzz_secs2_decode -max_total_time=60` | **200 000+ random inputs** through `secs2::decode`, **1.4 M+** through `try_parse_sml` per 60 s lane, ASan + UBSan + libFuzzer coverage, **0 crashes** | | 8 | `cmake -B build-fuzz -DSECSGEM_FUZZ=ON && build-fuzz/fuzz_secs2_decode -max_total_time=60` | **200 000+ random inputs** through `secs2::decode`, **1.4 M+** through `try_parse_sml` per 60 s lane, ASan + UBSan + libFuzzer coverage, **0 crashes** |
Plus, on every push to `main`, [Gitea Actions](.gitea/workflows/ci.yml) CI ([Gitea Actions](.gitea/workflows/ci.yml)) runs a Release build +
runs both a **Release build + full test suite** and a separate full suite and a separate `-fsanitize=thread` lane on every push to
**ThreadSanitizer lane** that builds with `-fsanitize=thread` and `main`. All 445 cases / 2 753 assertions pass under TSan clean.
fails on any race. All 445 cases / 2 753 assertions pass under TSan
clean.
## Per-standard test coverage ## Per-standard test coverage
@@ -47,26 +43,24 @@ Every claimed standard has dedicated tests. Counts are
| Persistence + cross-cutting | `test_job_persistence`, `test_persistence_upgrade`, `test_wire_ceid_emission`, `test_gem300_scenario`, `test_live_gem300`, `test_thread_safety`, `test_metrics_prometheus`, `test_robustness_fuzz` | 32 | | Persistence + cross-cutting | `test_job_persistence`, `test_persistence_upgrade`, `test_wire_ceid_emission`, `test_gem300_scenario`, `test_live_gem300`, `test_thread_safety`, `test_metrics_prometheus`, `test_robustness_fuzz` | 32 |
| **Total** | | **445** | | **Total** | | **445** |
A single command to see this live: `docker compose run --rm builder `docker compose run --rm builder /app/build/secsgem_tests --list-test-cases | wc -l`
/app/build/secsgem_tests --list-test-cases | wc -l` (currently 445). currently reports 445.
## What each proof actually demonstrates ## Categories of evidence
The eight commands above split into four kinds of evidence: The eight commands split into four kinds:
- **Internal** (#1, #2, #4, #5) — our code testing our code: unit - **Internal** (#1, #2, #4, #5) — unit suite, conformance harness,
suite, conformance harness, soak property test, config validator. soak property test, config validator. Necessary but not
Necessary but not independent. independent.
- **External, second implementation** (#3, #7) — round-trip against - **External, second implementation** (#3, #7) — secsgem-py (Python)
secsgem-py 0.3.0 (Python) and secs4java8 (Java). Two independent and secs4java8 (Java). Two independent SECS implementations agree
SECS implementations must agree with us on every frame. on every frame.
- **External, third codec** (#6) — Wireshark's HSMS dissector, - **External, third codec** (#6) — Wireshark's HSMS dissector,
written by network-protocol authors who don't share code with written by network-protocol authors with no shared code. Catches
either of us. Catches framing bugs the implementations might both framing bugs the implementations might share.
share.
- **External, structural search** (#8) — libFuzzer + ASan + UBSan - **External, structural search** (#8) — libFuzzer + ASan + UBSan
exploring the decoder and SML parser surface for crashes, memory over the decoder and SML parser surface.
errors, and UB.
See [VERIFICATION.md](VERIFICATION.md) for the full test plan and [VERIFICATION.md](VERIFICATION.md) covers each external validator in
the rationale for each external validator. detail.
+17 -23
View File
@@ -1,16 +1,14 @@
# Security operations guide # Security operations guide
HSMS is the spec's wire protocol: plain TCP, no auth, no encryption. HSMS is plain TCP no auth, no encryption. That's what every fab
That's what every fab tool ships and what every MES expects, and we tool ships and what every MES expects. Security comes from the
don't change it. Security comes from the network layer around the network layer around the HSMS socket; this doc has the concrete
HSMS socket. This doc has the concrete configs you'll need; no configs.
hand-waving.
> If you're shipping to a production fab, treat every section here For production deployments treat the sections below as mandatory
> as mandatory unless your fab security architect signs off on a unless your fab security architect signs off on a deviation. HSMS
> deviation in writing. HSMS on an exposed network with no controls on an exposed network with no controls is how MES impersonation
> is how an unauthenticated MES impersonation incident becomes a becomes a wafer-loss incident.
> wafer-loss event.
## 1. Network isolation ## 1. Network isolation
@@ -66,7 +64,7 @@ nft list set inet filter mes_hosts # confirm the set is loaded
### 1.3 Pod-network policy (Kubernetes / K3s deployments) ### 1.3 Pod-network policy (Kubernetes / K3s deployments)
If you're running the equipment in a pod, use a `NetworkPolicy`: For pod deployments, use a `NetworkPolicy`:
```yaml ```yaml
apiVersion: networking.k8s.io/v1 apiVersion: networking.k8s.io/v1
@@ -144,9 +142,7 @@ secs_server --port 5000 --bind 127.0.0.1 \
--config /etc/acme-secsgem/equipment.yaml ... --config /etc/acme-secsgem/equipment.yaml ...
``` ```
(The `--bind` flag is a small addition you'll need to add to The demo binary binds INADDR_ANY; a `--bind` flag is a follow-up.
`apps/secs_server.cpp` if you adopt this pattern — the demo binary
binds INADDR_ANY today. Filed as a follow-up.)
### 2.2 stunnel.conf — MES side (initiator) ### 2.2 stunnel.conf — MES side (initiator)
@@ -171,10 +167,9 @@ sslVersionMin = TLSv1.3
### 2.3 Performance impact ### 2.3 Performance impact
TLS adds ~50 µs per round-trip on modern hardware (measured via TLS adds ~50 µs per round-trip on modern hardware (measured via
`secs_bench` with stunnel in the loop vs. direct connection). At a `secs_bench` with stunnel in the loop vs. direct connection). At
few hundred S6F11 events/sec sustained that's invisible. Don't skip sustained rates in the few-hundred-events/sec range, the overhead
TLS for performance reasons unless your latency budget is genuinely is invisible against the fab-tool latency budget.
sub-millisecond.
## 3. Authentication ## 3. Authentication
@@ -243,14 +238,13 @@ ExecStartPre=/usr/local/libexec/secsgem-verify-configs.sh
ExecStart=/usr/local/bin/secs_server --config /etc/acme-secsgem/equipment.yaml ... ExecStart=/usr/local/bin/secs_server --config /etc/acme-secsgem/equipment.yaml ...
``` ```
If any signature fails, the unit refuses to start. Misconfiguration If any signature fails, the unit refuses to start.
incidents drop dramatically when this is in place.
### 4.2 Validate before signing ### 4.2 Validate before signing
Always run `secs_server --validate-config` against the YAML before `secs_server --validate-config` must run clean against the YAML
signing it. Signing a broken config just transmits the breakage before signing — signing a broken config only transmits the
cryptographically: breakage cryptographically:
```sh ```sh
secs_server --validate-config \ secs_server --validate-config \
+10 -46
View File
@@ -1,13 +1,11 @@
# External cross-validation harnesses # External cross-validation harnesses
Every harness in this directory exists so a reviewer doesn't have to Each harness in this directory validates the C++ codec, framing, and
take our word for it. Each one validates our C++ codec / framing / dispatch against an **independent third-party implementation** of
dispatch against an **independent third-party implementation** that the same SEMI standards.
read the SEMI standards without talking to us.
See [`../docs/VERIFICATION.md`](../docs/VERIFICATION.md) for the full See [`../docs/VERIFICATION.md`](../docs/VERIFICATION.md) for the
test plan and the honest accounting of which proofs are external vs internal-vs-external breakdown across all validators.
internal.
## What's here ## What's here
@@ -83,44 +81,11 @@ Drops the host link mid-flight, kills the server, restarts it, and
asserts the spooled S5F1 / S6F11 frames drain to the host on asserts the spooled S5F1 / S6F11 frames drain to the host on
reconnect. reconnect.
## What these harnesses caught
Real bugs surfaced during interop development (now fixed):
1. **Strict U-width parsing rejected U1-encoded identifiers.** SEMI
E5 declares DATAID, RPTID, VID, CEID, ALID, EXID etc. as
`U1 | U2 | U4 | U8`; secsgem-py picks the smallest width that
fits. Our scalar accessors were strict. Now lenient with
range-checked downcasts (`messages_helpers.hpp::any_unsigned_first`).
2. **PPBODY rejected when sent as ASCII.** SEMI allows PPBODY to be
`ASCII | Binary | List`; secsgem-py defaults to ASCII. Added the
`BINARY_OR_ASCII` codegen type and the `as_text_or_binary`
accessor.
3. **Missing S1F23 / S1F24 (Collection Event Namelist).** Added the
wire schema, the `vids_for(ceid)` accessor, and the dispatch
handler.
4. **Missing S10F3 handler (host→equipment Terminal Display).** Our
server only registered S10F1; per SEMI E5 §13 those are opposite
directions. Added the missing dispatch.
5. **TSan use-after-free in `act_exception_complete`** (test code,
not library): held a pointer across `fire_internal(RecoveryComplete)`
which deletes the entry. Found by the ThreadSanitizer lane on
first run.
The C++ test suite stayed green through every one of these fixes —
the changes were purely permissive widenings or additive features,
no existing behaviour broke.
## When to add a new validator ## When to add a new validator
A new third-party SECS implementation, or a new dissector, or a new A new third-party SECS implementation, dissector, or fuzzer target
fuzzer target — anything that exercises our wire surface from an that exercises the wire surface from an angle the existing five
angle the existing five don't cover is worth adding. The pattern don't cover is worth adding. The pattern is consistent:
is consistent:
1. New script / harness lives here (or a sidecar Docker context for 1. New script / harness lives here (or a sidecar Docker context for
non-Python validators). non-Python validators).
@@ -129,6 +94,5 @@ is consistent:
4. Surfaced in [`../docs/PROOFS.md`](../docs/PROOFS.md) if it adds a 4. Surfaced in [`../docs/PROOFS.md`](../docs/PROOFS.md) if it adds a
meaningful new dimension. meaningful new dimension.
Bug reports from a new validator → file at Bug reports from a new validator → file at `raphael@maenle.net`
`raphael@maenle.net` with the wire trace, the validator's output, with the wire trace, the validator's output, and the equipment YAML.
and the equipment YAML so we can reproduce.