40df3067a44b69929a8f4c616d4c77ddbe6ceb20
122 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 raphael
|
40df3067a4 |
docs: chapters 14–19 — GEM 300 standards (Part 2 complete)
Six more chapters finishing Part 2. Together with chapters 10–13 they document every SEMI standard this codebase implements. 14 — E40 + E94: process jobs (8-state lifecycle, S16F11/F5/F7/F9 on the wire) and control jobs (CJ wraps PJs with batch policy, S14F9/S16F27 messages). Worked cascade showing how CJSTART propagates through the PJ FSM and triggers S6F11 CEIDs at each transition. 15 — E87 carriers: three orthogonal sub-machines (CarrierID, SlotMap, CarrierAccess) per carrier and three more (Transfer, Reservation, Association) per load port. S3F17 CarrierAction strings + CAACK codes, S3F19 SlotMap verify, the 5-state slot encoding, multi-port concurrency. 16 — E90 + E157: substrate tracking via three orthogonal axes (STS / SPS / SubstrateIDStatus) and module process tracking (NotExecuting / GeneralExecuting / StepExecuting / StepCompleted). End-to-end PVD example showing E40 + E157 + E90 transitions cascading into CEIDs. 17 — E116 + E120 + E39: equipment performance time-buckets across six states, common equipment model object hierarchy, S14F1/F3 GetAttr/SetAttr as the uniform wire access for any object type across multiple standards. 18 — E84 parallel I/O: ten signal lines, the 9-state handshake FSM, the three TA1/TA2/TA3 timing-critical timers, why a physical handshake gets modeled in software (testability, timer enforcement, CEID emission, multi-port concurrency), the pure-FSM + asio-adapter split. 19 — E42 + E148 + S5F9–F18: formatted recipes (S7F23/F25 typed PPBODY), time synchronization with 16-char + 14-char accepted on set, exception recovery as a persistent multi-step host-supervised FSM (Posted → Recovering → Cleared with abort/retry). Revisits the auto-S9 family and contrasts S9 (transport) vs S5F9 (application). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
858ca22975 |
docs: chapters 11–13 — HSMS, SECS-I, GEM
Three more chapters of Part 2: 11 — E37 HSMS. 4-byte length prefix + 10-byte header (R-bit + session id + W-bit + stream + function + PType + SType + system_bytes), the 9 SType control messages, the NOT-SELECTED → SELECTED state machine, T3/T5/T6/T7/T8 with what each one bounds, the auto-S9 paths (S9F1/F3/F5/F7/F9/F11), HSMS-SS vs HSMS-GS, the asio single-threaded contract. 12 — E4 SECS-I. Half-duplex line turnaround (ENQ/EOT/ACK/NAK), the 10-byte block header bit-packing (R-bit / W-bit / E-bit / system bytes), the 244-byte block cap and multi-block split/assemble, the event-driven IO-free FSM with its Action / Event variants, T1/T2/T3/T4 with semantics + defaults, master/slave contention. Notes the deferred asio serial_port adapter; explains why this chapter matters even for HSMS-only readers. 13 — E30 GEM. Disambiguates the three state machines (HSMS transport vs GEM communication vs GEM control), walks the comm-state FSM (DISABLED → WAIT-CRA → COMMUNICATING with T_CRA / T_DELAY) and the control-state FSM (5 states + the YAML transition table). Lists every Fundamental and Additional capability with its messages, code locations, and store assignments. One worked Event-Notification scenario tracing seven on-wire steps to their EquipmentDataModel internals. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
338d0b974d |
docs: chapter 10 — E5 SECS-II data items and encoding
Opens Part 2 (the standards in detail). Walks the entire SECS-II encoding from first principles: the mental model (every value is one Item; a List is a recursive Item), the format-byte arithmetic (6-bit format code, 2-bit length-byte-count), the 14 format codes, length bytes 1/2/3 (with the 16 MiB cap), big-endian everywhere, the difference between byte-count (scalars) and child-count (lists). Then walks every format with worked hexdumps: empty list, nested list, ASCII with length-byte boundary crossing, Binary vs Boolean, U1/U2/U4/U8, signed integers with two's-complement edges, F4 / F8 with NaN / ±Inf / −0.0, JIS-8, C2 Unicode. Then the codebase mapping: Format enum, Item variant storage layout, encode_into / decode_at recursion, SML printer/parser, the identifier-wildcard rule (SEMI allows U1/U2/U4/U8 interchangeably for ID fields) with the messages_helpers::any_unsigned_first<Out> helper that closes the leniency contract. Closes with the well-defined CodecError conditions, what the codec deliberately doesn't reject (unknown format codes), and pointers to chapter 31 (codegen) and chapter 11 (HSMS) as the next dependencies above the codec. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
5fec47ad02 |
ci: bake secs4j harness into image instead of bind-mounting
Second secs4j-interop CI failure: ensuring secs4j-interop image is built... compiling Secs4jHostHarness.java... error: file not found: Secs4jHostHarness.java FAIL: javac The script bind-mounted $PWD/interop/secs4j into /work inside the container so it could javac the harness at runtime. That works locally where docker daemon and script share a filesystem, but fails in CI: the act runner runs the workflow inside a container, the docker socket is mounted from the host, and the daemon interprets bind-mount paths against the host filesystem — where $PWD/interop/secs4j doesn't exist. Result: empty /work, javac errors, job fails. Fix: COPY Secs4jHostHarness.java into the image and javac it at image build time. The script just runs the container — no bind mount, no docker-in-docker mount path translation, works in CI and locally. Verified locally with a fresh image rebuild: 55/55 checks pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
fc3422a4a9 |
docs: move root .md files into docs/ + update every reference
Picks up the file renames that landed alongside the previous commit and fixes everything that pointed at the old root locations: - README.md doc-map updated: every entry now points at docs/X.md, with a new "docs/" lead entry pointing at the guided-tour index. - README inline cross-refs (ARCHITECTURE / INTEGRATION / SECURITY / BENCHMARKS / MES_INTEROP / PROOFS) repointed to docs/. - README "Interop" section rewritten — used to mention only secsgem-py; now covers all four external validators (secsgem-py 31 / secs4java8 55 / tshark 69 frames / libFuzzer 200 k+ runs) with a one-line summary each, plus pointers to interop/README.md and docs/VERIFICATION.md. - README "Deferred follow-ups" cleaned: dropped the explanatory "Listed here so reviewers don't go looking for them in COMPLIANCE.md and find an 'out of scope' entry that sounds defensive" sentence — the section header speaks for itself. - docs/00_index.md "Where the rest of the docs live" table: dropped every `../` prefix since the docs are now siblings. - docs/01_what_is_secs_gem.md PROOFS reference updated to sibling. - docs/02_the_cast.md INTEGRATION + MES_INTEROP refs updated to siblings; dropped the stale "at the repo root" wording. - interop/README.md: VERIFICATION + PROOFS refs updated to ../docs/X.md; stale "~24 + 4 checks" updated to 31 (matches PROOFS.md and README). - examples/pvd_tool/README.md: every doc cross-ref now points at ../../docs/X.md. - Source / data / CI comments mentioning doc names (e.g. "INTEGRATION.md §3", "COMPLIANCE.md gap") rewritten to "docs/INTEGRATION.md §3" etc. — affects 9 files across include/, apps/, tests/, data/, examples/, .gitea/workflows/. Verified: full build under docker passes, 445/445 test cases pass, 2 753/2 753 assertions pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
60fa164626 |
docs: chapters 02 + 03 of the guided tour (Part 1 complete)
02 — The cast of characters: equipment, EAP, MES, fab planner, AMHS, operator. Who initiates which conversation, why the equipment is the passive side of HSMS by convention, how the AMHS handshake is out-of-band relative to SECS. Cross-references the relevant namespace and test files for each actor. 03 — Vocabulary + a wafer's journey: follows one 300 mm wafer end-to-end through a fab and labels every SECS message and acronym that fires. Introduces SVID / DVID / ECID / CEID / RPTID / ALID / PPID / MDLN / SOFTREV / HCACK / ALCD / OFLACK / CAACK / SMACK / etc. in context rather than as a list. Includes one-screen reference tables for the remaining acknowledge codes, T-timers in all four contexts (HSMS / SECS-I / E84 / E30 communication state), and a stream-by-stream summary. Part 1 (Foundations) of the guided tour is now complete — a reader who reads chapters 01–03 can describe the protocol stack, identify the actors, and recognise every acronym they'll meet in Part 2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
bc54de7711 |
ci: secs4j-interop bootstrap resilient to runner image variant
CI log showed: Run export DEBIAN_FRONTEND=noninteractive apt-get: command not found Failure - Main Bootstrap (node + git) exit status 127 The secs4j-interop job runs on the bare runner (not inside a `container:`) because it needs the host's docker socket to run `docker compose up -d server`. The runner image isn't fixed across deployments — catthehacker/ubuntu has apt-get, but a minimal node image doesn't. The old script hard-coded `apt-get` and exit 127'd on anything else. New bootstrap: - Checks what's already on PATH (git, node, docker). If all three are present, exits 0 — most act-runner images come pre-loaded. - Otherwise picks the right package manager (apt-get or apk) and installs only the missing pieces. - Errors out with a useful message if neither package manager exists, instead of failing on a missing command. Also updates the inline comment that still said "20 checks" — actual is 55 (matches the count in README / PROOFS.md / COMPLIANCE.md). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
01acac97d4 |
docs: start guided-tour tutorial series under docs/
A linear teach-from-zero tutorial that walks both SECS/GEM as a protocol family and this codebase as an implementation. Each chapter explains a SEMI concept and shows where it lives in code, so a reader builds a mental model of the standards and the repository simultaneously. Structure (24 chapters across 5 parts): - Part 1 (3 ch) — Foundations: what SECS/GEM is, the cast of characters, vocabulary + a wafer's end-to-end journey - Part 2 (10 ch) — Standards in detail: E5, E37, E4, E30, E40+E94, E87, E90+E157, E116+E120+E39, E84, E42+E148+S9 - Part 3 (7 ch) — Codebase: repository tour, spec-as-data + codegen, stores, transport, codec, state machines, persistence - Part 4 (2 ch) — Operations: build/run/demo, integration - Part 5 (2 ch) — Reference: API + messages + YAML, extension guide Published in this commit: - 00_index.md — guide layout, audience map, reading paths, conventions, status table - 01_what_is_secs_gem.md — the N×M integration problem, what SECS vs. HSMS vs. GEM each actually refer to, the GEM 300 suite, the transport→message→behaviour layering, where each layer lives in this codebase, an end-to-end S2F17/F18 example Chapters publish iteratively from here. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
b01dedfaa5 |
docs: drop COMPLIANCE §8 "out of scope" and broaden §7 to all 4 validators
§8 was carrying two items that neither read as "deliberately out of scope" nor matched the framing of the section: - Equipment Processing States — E30 §6.3 explicitly leaves concrete states tool-defined. The framework ships the ControlTransitionTable engine and YAML loader; vendors supply IDLE/SETUP/READY/EXECUTING. That's a design choice, not a gap. §3 line 94 already documents it. - Serial-port wiring for SECS-I — the FSM is implemented and tested end-to-end over TCP; only the asio serial_port adapter is missing. That's deferred, not out of scope. §1a line 64 already lists it with status ⬜. So §8 is dropped, §9 renumbers to §8, and the deferred follow-up gets its own short section in the README so customers know it's tracked without sounding defensive. §7 used to be titled "Interoperability with secsgem-py 0.3.0" and mentioned only that one external implementation. We now have four external validators (secsgem-py + secs4java8 + tshark dissector + libFuzzer), so the section is renamed "Interoperability with external implementations" and broadened to cover all of them with their actual check counts. Stale "24 named checks" updated to the current 31; "three consecutive clean runs" line dropped as audit-language no longer earning its keep now that it's a CI step. FAQ's "What's not implemented?" answer rewritten to point at the README "Deferred follow-ups" section and COMPLIANCE §8 (new numbering), with a brief note explaining that Equipment Processing States are spec-by-design tool-defined. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
c8e8e80735 |
secs_server: relative-path defaults so the binary runs outside docker
Previously --config / --state-table / --pj-state-table / --cj-state-table defaulted to /app/data/..., which only resolves inside the docker image. A host build run from the repo root errored out unless every flag was passed explicitly. Switch to data/equipment.yaml (and siblings) relative to CWD — docker still works because WORKDIR=/app puts /app/data/... at the same relative location, and host builds run from the repo root resolve to <repo>/data/.... Existing callers that pass explicit paths (the proof commands, tshark_validate.sh, secs4j_validate.sh, docker compose) are unaffected. Verified --validate-config under docker still finds all four YAMLs and the tshark proof still passes (69 frames, 0 malformed). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
b031f057af |
docs: customer-ready sweep + README restructure + tshark CI fix
Audit pass over the public-facing surface so a customer can read it end-to-end without tripping on stale numbers or self-contradictions. README + docs accuracy: - Test counts 426 → 445, assertions 2 557 → 2 753 (verified via doctest run); E5 row was missing test_e5_kat (19 cases) - Interop checks 24 → 31, COMPLIANCE.md message count 149 → 164, COMPLIANCE.md "291 cases / 1515 assertions" → 445 / 2 753 - README "60+ test IDs" for MES_INTEROP → actual 59 - PVD example counts: 32 SVIDs/17 CEIDs → 29/21, "~40 handlers in ~200 lines" → 51 in ~460, "~700 lines" → ~1,100; main.cpp header table-of-contents resynced with the actual 7 sections Out-of-scope honesty (COMPLIANCE.md §8 + FAQ.md): - Removed HSMS-GS (was both ✅ implemented in §1 and "out of scope" in §8; INTEGRATION.md §7 documents using it) - Removed multi-block SECS-I (split_message/assemble_message exist with 4 dedicated tests) - Added serial-port wiring as the genuine open ⬜ item — FSM is tested end-to-end over TCP; only the asio serial_port glue is deferred - COMPLIANCE.md intro now lists E42 and notes "E37 (SS + GS)" README restructure: - Moved the 8-command proof table and per-standard test-coverage table to a new PROOFS.md (72 lines) - README now leads with what / Quick start / Documentation map, then a one-paragraph "How it's proved" linking to PROOFS.md - Updated cross-refs in FAQ.md, GLOSSARY.md, VERIFICATION.md, and interop/README.md to point at PROOFS.md CI fix — tshark-dissector job: - interop/tshark_validate.sh hardcoded /app/build/secs_server etc. which only works inside the docker image. Now derives ROOT from the script's own location and accepts BUILD/SERVER/CLIENT/DATA env overrides, so CI can run it from the workspace dir - Verified still passes in docker (69 frames, 0 malformed) .gitignore: - Added build-fuzz/ and build-tsan/ (were showing as untracked) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
6aa4427186 |
docs: worked PVD-tool vendor example
A fictional Physical Vapor Deposition tool wired end-to-end.
examples/pvd_tool/ is the template a real customer should fork.
Files:
- equipment.yaml: 32 SVIDs (chamber pressure, temperature, source
power, gas flows, cooling water, wafer counters, recipe step
state, EPT name, 4 load ports), 5 DVIDs, 7 ECIDs (setpoints
+ T_CRA/T_DELAY + cleaning interval + retry count), 17 CEIDs
(control state, alarms, process lifecycle, material movement,
EPT), 12 alarms with realistic categories (safety, error,
warning, attention), 3 multi-step recipes (Al / Ti / Cu),
9 host commands.
- main.cpp (~860 lines): the vendor-side application:
§1 helpers + constants
§2 sensor simulator — 4 sensors at 10 Hz + 1 Hz cadences,
random-walk around step-targeted setpoints, asio::post-on-strand
thread-safety pattern
§3 recipe runner — parses recipe body (STEP NAME duration=120s
power=2500W gas=Argon flow=50sccm), walks each step at 1s
per declared-second, fires step-started/completed CEIDs,
drives PJ FSM through ProcessComplete
§4 alarm threshold monitor — chamber-pressure-over-setpoint and
cleaning-interval logic, continuous evaluation, set/clear
emission gated on alarm-enable
§5 EPT cycler — Standby ↔ Productive ↔ UnscheduledDowntime
based on PJ activity + safety alarms
§6 Prometheus exporter on :9090 (pvd_messages_total,
pvd_chamber_pressure_torr, pvd_spool_depth, pvd_events_total,
pvd_alarm_set_total)
§7 Router handlers — full E30 set (~40 handlers) so a host can
do real work
§8 main() — YAML validation, model construction, server wiring,
periodic gauge updates
- README.md: section-by-section walkthrough, what's the same as
apps/secs_server.cpp, what this adds (simulator + recipe runner
+ alarm monitor + EPT cycler + metrics), what's not here
(persistence + E84 + real I/O), and what to change for your tool.
Verification: 47/47 conformance harness checks PASS against the
PVD tool — same as the demo server.
CMakeLists.txt adds the pvd_tool target.
README's documentation map points at examples/pvd_tool/.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
91eec92b73 |
docs: ARCHITECTURE.md — how the codebase fits + how to extend
Customers who want to extend the library had two paths: read the 1200-line apps/secs_server.cpp and guess at conventions, or read every store header and infer the shape. Neither is reasonable. ARCHITECTURE.md walks the five layers (apps → Router+Model → stores → FSMs → transport+codec) with a worked extension recipe per layer: - New SECS-II message (YAML edit + Router handler — no core code) - New state machine (lift from ept_state.hpp or process_job_state.hpp) - New store (paste-and-adapt from alarms.hpp or process_jobs.hpp) - New persistence backend (mirror enable_persistence pattern) - New transport (mirror Connection's contract) Explains the design choices that look unusual: - Spec-as-data — every behavioural rule in YAML, C++ is the engine - I/O-free FSMs — transport classes own asio, everything else is pure - Single-threaded by design + no locks anywhere - No DI framework, no singletons, no shared_ptr-everywhere - Exceptions only for programmer-error / corrupt-input paths Documents the persistence magic-byte registry (0xC4-0xC9 + 0xE5) so the next contributor doesn't collide; documents the codegen pipeline (messages.yaml → gen_messages.py → messages.hpp); maps "you want to understand X" → "read these files in order" for the twelve most common entry points. Doc map in README already points at ARCHITECTURE.md from the prior commit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
195ecc689f |
docs: GLOSSARY + FAQ + interop README refresh + doc-map fixes
Fills four documentation gaps surfaced by the doc audit:
1. README "Documentation map" was missing VERIFICATION.md (the file
that backs the proof-of-feature-completeness claims) and is now
pointing at the new files added in this commit too — ARCHITECTURE,
GLOSSARY, FAQ, examples/pvd_tool/ (the last two land next).
2. interop/README.md only documented secsgem-py. Three of the five
external validators (tshark, secs4j, libFuzzer) plus the E5 KAT
were invisible from the directory's own README. Rewritten as a
complete index — what's external, what each catches, how to run,
what bugs they've already surfaced, when to add a new validator.
3. GLOSSARY.md is new. Every SEMI acronym used in the codebase or
the docs gets one row: SVID, DVID, CEID, RPTID, ALID, ECID, PPID,
MID, CARRIERID, PRJOBID, CTLJOBID, SUBSTID, OBJSPEC, OBJTYPE,
MDLN, SOFTREV, EQPTYP, DATAID + every ACK code (COMMACK, ONLACK,
OFLACK, HCACK, CMDA, ACKC5-7-10, DRACK, LRACK, ERACK, EAC, TIACK,
GRANT, ALCD, OBJACK) + stream/function shorthand + HSMS terms +
T-timers + E84 signals + the standards lineup + codebase shortcuts
("the model", "the router", "the proof", etc.). Cuts week-1
onboarding time.
4. FAQ.md is new. Canonical answers to the questions that come up
once per integration: why HSMS unencrypted, SVID vs DVID, PJ vs
CJ, who fires FSM transitions, what runs on which thread, how to
add a new SECS-II message, ASCII vs Binary, common MES quirks,
how spool works, robustness fuzz vs libFuzzer, conformance vs
interop, what's not implemented.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
db90a21e1d |
verify: expand secs4j harness 20 → 55 checks
Every check the user could ask for now lands. secs4j's
comm.send(stream, function, w, body) takes arbitrary S/F + arbitrary
Secs2 body, so coverage was never coverage-limited by the Java side
— the original 20 was just the minimum to fill the gaps secsgem-py
couldn't reach.
Adds:
- Status data: S1F3, S1F11
- EC management: S2F13, S2F15 (set TimeFormat), S2F29
- Event reports: S2F33, S2F35, S2F37 (full define-link-enable
sequence), S6F15, S6F19, S6F21
- Remote control: S2F41 (modern RCMD=START + observed S6F11),
S2F21 (legacy RCMD=STOP),
S2F41 RCMD=FAULT + observed S5F1
- Alarms: S5F3, S5F5, S5F7
- Spool: S2F43, S6F23
- PP management: S7F1, S7F3, S7F5, S7F17, S7F19
- Terminal: S10F3 (single), S10F5 (multi-line)
- E40 PJ: S16F11 (full E40 body — MF + PRRECIPEMETHOD +
RecipeSpec + mtrloutspec + processparams),
S16F7 (monitor), S16F13 (dequeue)
- Limits: S2F45, S2F47
- Trace: S2F23 (5-field body)
- E39: S14F1 (GetAttr)
Plus a SecsMessageReceiveListener that captures every equipment-
initiated primary into a ConcurrentLinkedQueue and replies to S5F1
(ACKC5=0), S6F11 (ACKC6=0), S16F9 (W=0 no reply) so the
equipment's T3 doesn't fire on our watch. Two checks now assert
the unsolicited path:
- After RCMD=START, an S6F11 with the linked report must arrive
within 400ms
- After RCMD=FAULT, an S5F1 with the alarm must arrive within
400ms
Both observed against the demo equipment.
Result: 55/55 PASS. Two independent implementations
(secsgem-py + secs4java8) now corroborate the wire surface in
overlapping but distinct slices. Full E40 body — the one that
defeated secsgem-py's SFDL grammar — round-trips cleanly through
secs4j.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
4ddf8e0f48 |
verify: libFuzzer harness for secs2::decode + try_parse_sml
Coverage-guided structural search for crashes and undefined behaviour on arbitrary input to our two parsers. What's wired: - -DSECSGEM_FUZZ=ON CMake option, clang-only. Adds -fsanitize=fuzzer-no-link,address,undefined to all targets + -fsanitize=fuzzer to the two fuzz executables. - apps/fuzz_secs2_decode.cpp — feeds raw bytes to secs2::decode. Catches secs2::CodecError (expected) but traps on anything else leaking (would be a hardening bug). - apps/fuzz_sml_parse.cpp — feeds string to try_parse_sml, which is contractually nothrow-equivalent; traps on any exception. - .gitea/workflows/ci.yml — `libfuzzer` job builds with clang and runs each fuzzer for 60s in CI. Any crash / ASan / UBSan flag fails the job. - Dockerfile gains clang + libclang-rt-18-dev so devs can run locally with the same toolchain. Result on a fresh 30-second local run: fuzz_secs2_decode: 70 727 random inputs, 0 crashes fuzz_sml_parse: 284 950 random inputs, 0 crashes The coverage-guided search found and synthesized inputs that exercise: zero-byte, single-byte format tags, all length-byte counts (1/2/3), nested lists, format bytes with reserved bits, the "BOOLEAN" SML token, malformed quoted strings, etc. libFuzzer's recommended dictionary at the end of each run shows what bytes / substrings the coverage feedback discovered as discriminating — useful signals if we ever want a hand-curated corpus. README proof table grows to 8 commands. After this: - 426 unit tests (internal) - 47 conformance harness checks (internal) - 24 secsgem-py interop checks (external — Python ref impl) - 20 secs4j interop checks (external — independent Java impl) - 69 frames dissected by Wireshark HSMS dissector (external) - 196 SEMI E5 KAT assertions (standards body's encoding rules) - **~70k + ~285k random inputs, 0 crashes (external)** - 100k random tool ops with all invariants holding (internal) - YAML validation (internal) - TSan clean on 2 557 assertions (internal correctness aid) Five distinct external proofs now, each covering a different angle. Plan: VERIFICATION.md §4. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
2fce2fad0c |
verify: secs4j cross-validation (independent Java implementation)
20 cross-validation checks PASS against [secs4java8] (Apache 2.0,
kenta-shimizu) — an independent SECS/HSMS implementation in Java by
a different author from a different language ecosystem. Distinct
implementer = independent spec interpretation. Two libraries
agreeing on wire bytes is much stronger evidence of spec-correctness
than either alone.
Coverage targets the gap the secsgem-py interop deliberately skipped
(secsgem-py's SFDL grammar couldn't easily express GEM 300 bodies
with variable lists of named scalars):
- S1F1/F13/F17/F19/F21/F23 — establish comms + namelists
- S2F17 — clock
- S2F23 — trace init (5-field body)
- S2F49 — enhanced remote command (DATAID + OBJSPEC + RCMD + params)
- S3F17/F19/F25/F27 — full E87 carrier surface (action, slot map
verify, transfer with port pair, cancel)
- S5F13/F17 — exception recovery (EXID + EXRECVRA)
- S14F9/F11 — E94 CJ create with prjobids list, CJ delete
- S16F5/F27 — E40 PJ command, E94 CJ command
- S1F15 — offline cleanup
20/20 PASS against the demo equipment. Reply S/F matches the spec
for every transaction; specific ACK values vary by equipment state
(CarrierIDUnknown for an unknown carrier is just as valid as Accept
for a known one) so we assert on the wire shape, not the result.
Ship layout:
interop/secs4j/Dockerfile — eclipse-temurin:21-jdk + clone
+ build of secs4java8 → Export.jar
interop/secs4j/Secs4jHostHarness.java
— 20 round_trip assertions; uses
Secs2.list/uint4/ascii to build
full GEM 300 bodies; comm.send()
for arbitrary S/F pairs
interop/secs4j_validate.sh — orchestrator: builds image,
compiles harness, starts compose
server, runs Java container on
the secs network against it
.gitea/workflows/ci.yml — secs4j-interop job in CI
README.md — proof table grows to 7 commands
.gitignore — *.class
After this commit our proof chain has:
- SEMI E5 KAT (standards body's own arithmetic)
- tshark dissector (Wireshark's HSMS impl)
- secsgem-py interop (Python reference impl)
- **secs4j interop** (independent Java impl)
+ 426 unit tests, 47 conformance harness checks, 100k random ops,
YAML validation
Four independent external proofs, three of them on overlapping wire
surface from independent angles.
Plan: VERIFICATION.md §3.
[secs4java8]: https://github.com/kenta-shimizu/secs4java8
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
5baf3f4dc7 |
verify: tshark HSMS dissector validation (independent third codec)
Wireshark's built-in HSMS dissector — written by network-protocol authors who don't know us, didn't talk to us, and don't share implementation details with secsgem-py — is a third independent codec for our framing. If they parse our pcap without warnings, our HSMS framing is wire-correct independently of both our internal tests and the secsgem-py interop path. interop/tshark_validate.sh: - Boots secs_server on 127.0.0.1:5099 (away from the demo port) - Captures the loopback wire traffic with tcpdump - Runs secs_client through ~24 transactions plus Separate.req + TCP FIN - Parses the pcap with tshark -V using the HSMS dissector - Asserts: no "Malformed Packet", no "Dissector bug", at least one HSMS frame, expected tokens present (Select.req/rsp, Separate.req, Data message), reports histogram (count by control type + distinct S/F pairs) Result against the demo: 69 HSMS frames dissected, 49 distinct S/F pairs (S01F01..S16F28), all clean. Dockerfile gains tshark + tcpdump. .gitea/workflows/ci.yml gains a `tshark-dissector` job that runs this validator as part of every push to main. README proof table grows to 6 commands. VERIFICATION.md §1a documents a follow-up: round-trip the KAT fixtures through secsgem-py to corroborate that the format codes we used match an independent implementation. Strengthens the KAT proof from "internally consistent" to "confirmed by a second implementer who read the spec without talking to us." Plan: VERIFICATION.md §2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
a79973ed4c |
test: SEMI E5 known-answer tests for SECS-II encoding
Hex-string fixtures constructed directly from the SEMI E5 §9
format-byte encoding rules:
format_byte = (format_code << 2) | length_byte_count
length_byte_count ∈ {1, 2, 3}
Coverage:
- Every format code (L, B, BOOLEAN, A, J, C, U1-U8, I1-I8, F4, F8)
- Every length-byte-count variant (1, 2, 3 bytes — exercises the
255 → 256 → 65 536 transitions)
- Numeric edges: 0, ±1, MIN, MAX, ±Inf, NaN, -0.0, multi-element vectors
- Empty and single-element variants
- Nested lists
- A "format byte layout per format code" regression tripwire that
pins every code → byte mapping
19 test cases, 196 assertions. Every fixture round-trips
byte-identical against the codec.
Why this is the strongest single codec test: every other validator
(secsgem-py interop, conformance harness, in-house unit tests) is
one implementer's interpretation. KAT is the standard's own
arithmetic. If our encoder matches these canonical bytes and our
decoder reverses them to the same Item, our SECS-II layer is wire-
compatible with anything else that obeys E5 §9.
NaN / signed-zero / Inf use a bit-pattern compare (IEEE NaN != NaN
breaks the default Item == path) — decode the canonical, re-encode
the decoded, assert byte-identical.
The 3-byte-length fixture (ASCII 65 536 × 'X') generates a ~200 KB
expected-bytes string in the test — slow to write but trivial to
check and forces the 3-byte length-prefix path that 99 % of real
traffic doesn't exercise.
Plan: VERIFICATION.md §1.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
257a148d34 |
docs: VERIFICATION.md — external validation test plan
Honest accounting of what's currently external vs internal in the
five proofs:
- 4 of 5 proofs are us-testing-us (unit tests, conformance
harness, robustness fuzz, YAML validation)
- Only secsgem-py interop is external, and it covers ~15-20 %
of the claimed wire surface (skips most of GEM 300, HSMS-GS,
exception recovery, wafer maps, enhanced commands, every
wire-level edge case that isn't message-shaped)
Plan documents four additional external validators with goals,
methods, success criteria, scope limits, and effort estimates:
1. SEMI E5 known-answer tests — hex fixtures from the spec's
own encoding rules; the strongest single codec test
2. tshark/Wireshark HSMS dissector — independent third codec
parsing our pcap captures
3. secs4j cross-validation — Apache-2.0 Java implementation
by a different author; catches "we both got it wrong the
same way" relative to secsgem-py
4. libFuzzer over secs2::decode + secs2::from_sml — coverage-
guided structural search for crashes and UB
After all four: 5 external proofs (KAT + tshark + secsgem-py +
secs4j + libFuzzer), three of them on overlapping wire surface
from independent angles.
Plan also explicitly lists what these validators do NOT replace:
GEM RTS certification, per-MES interop sweeps, real-fab wire
trace corroboration. Those remain customer-side work.
Order of execution: KAT → tshark → secs4j → libFuzzer. KAT
first because it produces fixtures the others can reuse;
libFuzzer last because it benefits from the KAT corpus.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
e82f67ecad |
docs: README restructure + proof-of-feature-completeness section
The old README mixed intro, quickstart, architecture, and 10 sections
of production deployment over 419 lines, with significant overlap
with INTEGRATION.md. It claimed "implements every standard" without
making the claim concrete.
Restructured to ~250 lines with the proof front and center.
New top-of-README "Proof of feature-completeness" section: five
commands that, when they all exit zero on a fresh clone, prove the
COMPLIANCE.md claims. Each command verified end-to-end before
landing in this commit:
1. docker compose run --rm tests
→ 426 cases / 2557 assertions PASS
2. secs_conformance --host server --port 5000
→ 47 / 47 wire-level checks PASS
3. host_vs_cpp_server.py --host server
→ 24 secsgem-py interop checks PASS
4. SECSGEM_ROBUSTNESS_SOAK=1 secsgem_tests -tc='*soak*'
→ 100 000 random tool operations, all invariants hold
5. secs_server --validate-config <all four YAMLs>
→ 0 errors, 0 warnings across the shipped configs
Plus a per-standard test-coverage table mapping every claimed SEMI
standard (E5, E5 §13, E4, E37, E30, E40, E94, E42, E87, E90, E116,
E120/E39, E157, E84) to its test files and case count, summing to
426 to match the doctest totals. Counts verified by
`grep -c TEST_CASE` per file.
CI also runs the TSan lane (separate job in
.gitea/workflows/ci.yml); README documents it under Build details.
Content moved out of README into specialized docs (eliminates
duplication):
- Security configs → SECURITY.md (was 14-line bullet list; now a
365-line file with nftables, stunnel, minisign, SIEM schema)
- Persistence layout + monitoring + HA + deployment patterns +
upgrade discipline + fab-stack integration → INTEGRATION.md
- Performance envelope → BENCHMARKS.md
- MES interop punch list → MES_INTEROP.md
README now reads top-to-bottom: what this is → license → proof →
quickstart → doc map → architecture → adding capabilities →
production (1-line pointers to the deep docs) → build details →
interop.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
0df229905d |
docs: SECURITY.md with concrete configs
README §2 used to list security categories ("network isolation",
"TLS tunnel", "authentication", "audit logging", "YAML signing")
without configs. Customers deploying to a real fab can't act on
bullet points — they need files to drop in and paths to verify.
SECURITY.md replaces the bullets with:
- nftables ruleset locking the HSMS + Prometheus + SSH ports to
known source IPs (with the test command to lint before reload)
- Kubernetes NetworkPolicy equivalent for pod deployments
- stunnel.conf for equipment side (terminator) AND MES side
(initiator), with mTLS, TLS 1.3 minimum, and bind-127.0.0.1
pattern so the cleartext socket never sees the network
- minisign-based YAML config signing: keygen, sign-at-deploy,
systemd ExecStartPre verification. Refuses to start on bad sig.
- Audit logging JSON schema for SIEM ingest, with one-line example
per frame and the structured-dispatch wrapper to emit it
- SIEM alert thresholds: S9F rate, distinct source IPs, TLS
handshake failures, signature-verify failures, spool depth,
T-timer expiry counter
- Secrets handling: stunnel keys + minisign signing key custody
- Incident response capture protocol (tcpdump, journal snapshot,
no-restart-until-captured) + reporting-back format
Every section has a runnable example. Nothing here is invented
under pressure during an incident.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
943f3bbcd5 |
ci: ThreadSanitizer lane + fix use-after-free TSan flagged
Adds a -DSECSGEM_TSAN=ON CMake option that builds every target with -fsanitize=thread + debug symbols + -O1 + frame pointers. Wires a dedicated thread-sanitizer job into .gitea/workflows/ci.yml that builds and runs the full test suite under TSan with TSAN_OPTIONS=halt_on_error=1 (any flagged race fails the job, not just warns). Result against the full 426-case / 2557-assertion suite: 0 warnings, all green. That converts the existing test_thread_safety.cpp (which exercised the asio::post-onto-strand pattern) and test_concurrency (in-flight transaction interleaving) and test_robustness_fuzz (28 random action types × thousands of ticks) from "pattern smoke-tests" into actual race detection. The first TSan run caught a real bug in test_robustness_fuzz's act_exception_complete: it held a pointer to an ExceptionStore entry across fire_internal(RecoveryComplete), which deletes the entry. The subsequent state() read was a use-after-free. TSan flagged it 8 times (4 reads × 2 stack-frame variants). Fix is scoped lookup + re-check via has() after the mutation; matches the contract any reasonable caller would follow. The asio std_fenced_block atomic_thread_fence path generates TSan "not supported" warnings during compile — those are asio's, not ours, and don't affect runtime detection. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
ca3559ef57 |
test: randomized robustness fuzz (4 seeds × 2k ops + 100k soak)
tests / build-and-test (push) Successful in 2m9s
Property-based robustness test that drives long sequences of random tool operations against EquipmentDataModel and verifies invariants + persistence round-trip after every action. Replaces hand-written state-pinning tests with a generative approach that explores combinations no human author would think to write. Action menu (28 weighted actions covering the full standard surface): - PJ create / event / dequeue (E40) - CJ create / event / delete (E94) - Carrier create / id / slot (E87) - Substrate create / location / proc (E90) - Alarm set / clear / enable toggle (E5 §13) - SVID updates (E30 §6.13) - Define-report / link-event / enable (E30 §6.6) - Exception post / recover / complete (E5 §9, S5F9-F18) - Module event (E157) - EPT event (E116) - Spool enqueue / drain / force-toggle (E30 §6.22) Every action is "adjusted": it picks a verb at random, then checks state-machine legality before applying. A Pause is only fired on a Processing PJ; a Recover only on a Posted exception; pj_dequeue skips PJs bound to active CJs (mirrors E94's "can't dequeue CJ-bound PJ" rule the fuzz itself discovered when the first run flagged a CJ→missing-PJ reference). Invariants checked every 64 ticks: - Every tracked PJ exists in the store (size matches) - Every CJ's prjobids all exist in PJ store - No FSM in NoState sentinel - EPT bucket total monotonically non-decreasing - Defined reports' VIDs all exist - Substrate / carrier counts match enumeration Persistence round-trip every 500 ticks: - Fresh shadow EquipmentDataModel loads from the same journal dir - Diffs PJ + CJ states one-by-one + carrier/substrate/exception counts against the live model - Catches any "mutation didn't reach disk" or "replay didn't reconstruct state correctly" bugs Reproducibility: - Each TEST_CASE uses a fixed seed (0x1, 0xdeadbeef, 0xfeedface, 0xc0ffee — 8000 ops total in the fast suite) - World keeps a rolling 20-action trace, printed on invariant violation so the failing sequence can be pasted into a targeted regression test - SECSGEM_ROBUSTNESS_SOAK=1 enables a 100k-tick soak case (~3-5 minutes in Docker; not run by default) The very first run found a real edge case: act_pj_dequeue removed PJs that were bound to active CJs, leaving dangling refs. Fixed the fuzz to filter; the underlying behavior is intentional (store trusts the application to gate), but the fuzz now mirrors the correct E94 contract. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
b99d84f956 |
hsms-gs: worked integration example + INTEGRATION.md §7
The codebase has supported HSMS-GS since the original landing
(test_hsms_gs.cpp covers the wire-level Select.req-per-session
walk-list, the per-session Reject(EntityNotSelected) behaviour,
and session-routed data dispatch). But the documentation said
exactly one line about it ("Connection::add_session(device_id)
registers extra sessions on one TCP socket") and there was no
end-to-end test using the Server/Client API customers actually
build against.
INTEGRATION.md §7 is a new section showing the realistic pattern:
- Server-side: register the primary session via Server::Config,
then `add_session` for the second MES in the on_connection
callback. Per-session message handler + selected handler so
each MES gets its own router (or its own per-session data view
over a shared EquipmentDataModel).
- Active-mode: same `add_session` on the host-side Connection
for multi-tool fleet controllers.
- Equipment-initiated push: pick the session_id when sending
unsolicited primaries (S5F1, S6F11, S10F1).
- Pointer to the wire tests + the new integration test for
customers who want to see the failure modes.
tests/test_hsms_gs_integration.cpp drives two MES sessions
(device_id 1 + 2) through the Server/Client API end to end:
- Both sessions complete Select.req independently
- S1F1 sent on each session returns a distinct MDLN
("EQUIP-SESS-1" vs "EQUIP-SESS-2"), proving per-session
dispatch routes correctly
- Per-session router fires exactly once per session, no
cross-talk
Pre-existing §§8-10 in INTEGRATION.md got bumped to §§9-11 to
make room.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
e3765a5176 |
persistence: multi-version reads across every store
ProcessJobStore and SubstrateStore already implemented the loader-accepts-any-version-in-[1, kVersion] pattern. The other five stores (ControlJobStore, CarrierStore, LoadPortStore, ExceptionStore, SpoolStore) used strict `header[1] != kVersion` rejection, meaning a future kVersion bump there would silently nuke every persisted record on first replay. That's a footgun the test_persistence_upgrade test already flagged as a tripwire. This commit flips the strict checks to `< 1 || > kVersion`, mirroring PJ + Substrate. No format change (kVersion stays at 1 across the five stores), but: - Future v2 of any store now Just Works: add fields at the end of write_record_, bump kVersion to 2, gate the new reads behind `if (version >= 2)`. Old v1 records on disk continue to replay with the new fields defaulted. - Future versions beyond kVersion still get rejected (downgrade protection — older code can't try to decode trailers it doesn't understand). Comment blocks on each kVersion declaration now describe the upgrade discipline so the next contributor doesn't reinvent it. Test additions: - Positive test that v1 ControlJob records load on current code (will continue to pass when kVersion bumps to 2, proving v1 is still readable) - ExceptionStore rejects a v9 (future) record, matching CJ + Carrier - The existing tripwire tests get retitled from "rejects unknown version" to "rejects a future version" to reflect the new contract README §6 gets honest: every store is now multi-version-aware, not just PJ + Substrate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
ce5abb4f72 |
docs: real-MES interop test plan (day-1 punch list)
interop/ cross-validates against secsgem-py 0.3.0 — the Python reference. That's not what a fab actually runs. Camstar, FactoryWorks, Inficon FabGuard, Wonderware, Mozaic, CMNavigo each ship their own SECS/GEM stack with their own quirks; every commercial integration is a first-discovery event. MES_INTEROP.md is the structured protocol customers run against their MES *before* connecting a real tool: - 9 test sections covering HSMS plumbing, establish-comms, dynamic event reports, alarms, remote control, PP management, terminal services, GEM 300 (E40/E87/E94), spool, clock+ECs - 60+ test IDs with expected wire behaviour and known quirks per MES vendor (compiled from prior integration support) - Soak + cutover checklist (memory, spool, T-timers, dashboards) - Reporting-back protocol for MES-specific bugs that this codebase should handle Treated as a punch list with PASS/FAIL/N-A per row, captured wire trace per row, and a 90-day archive of the lot — that's the audit trail a fab's quality team will ask for. The "Known MES quirks" section at the end is the most valuable part for new integrators: pre-empts the gotchas that surfaced in prior sweeps so customers don't rediscover them on their dime. README header gets a fifth bullet pointing at the file. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
6c6dc84c22 |
metrics: Prometheus exporter sample + worked INTEGRATION example
README §3 promised a monitoring story ("aggregate into Prometheus via
a sidecar that polls the data model"). Nothing shipped. Customers
running a real fab without a metrics pipeline find out about T7
storms, spool blowups, and stalled CJs after their MES does — not
the position you want SRE in.
This commit ships:
- include/secsgem/metrics/prometheus.hpp: header-only. A Registry
(counters + gauges + HELP/TYPE descriptions, label-keyed,
mutex-guarded so updates from the io thread and scrape renders from
the same io serialize cleanly) plus a PrometheusServer (asio
acceptor, replies to any GET with the text-exposition rendering,
no auth — drop nginx in front for that).
- tests/test_metrics_prometheus.cpp: 3 cases / 19 assertions.
Render counter+gauge with labels, scrape via raw TCP and parse the
HTTP body, verify live updates land on subsequent scrapes.
- INTEGRATION.md §6.4: worked example that pairs the exporter with the
Connection + EquipmentDataModel hooks documented in §6.1/§6.2.
Shows the wrap-around-handler trick for message counters, a 5s
polling timer for gauges (spool depth, active alarms), and the
expected /metrics output.
Deliberately *not* shipped:
- A StandardMetrics helper that auto-wires everything — would force
a single hook owner per store, breaking customers who want
composable observers. Customers wire what they need; the registry
gives them counters + gauges + an HTTP endpoint, no policy.
- TLS / auth on the HTTP endpoint. Reverse-proxy territory.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
db426cbeed |
ci: bootstrap node before actions/checkout on Gitea runners
`actions/checkout@v4` is a JavaScript action — it expects `node` on PATH in the runner image. Gitea Actions (and local `act`) running against `ubuntu:24.04` had neither node nor git pre-installed, so checkout failed with: ❌ Failure - Main actions/checkout@v4 exitcode '127': command not found The pre-step now installs nodejs + git + ca-certificates from apt before checkout runs. The rest of the C++ toolchain installs in a second step after the source tree is on disk. Doesn't affect GitHub-hosted runners (their images already have node); doesn't change build behaviour either. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
9c5d67fdad |
bench: secs_bench harness + BENCHMARKS.md baseline
Customer SREs and capacity planners had nothing to point at.
INTEGRATION.md asked the right questions ("how many tx/sec?"
"how much memory per active CJ?") but had no numbers.
secs_bench spins up an in-process passive equipment + active host
on an OS-allocated port, runs three canned workloads, and emits a
markdown table customers can capture and diff across commits:
- S1F1/F2 header-only round-trip — dispatch + framing baseline
- S1F3/F4 with N SVIDs — encode + decode throughput
- S6F11 push (W=0) — one-way emission ceiling
- PJ + CJ pair memory footprint — bytes per active job
Latency reports p50/p95/p99/max via std::nth_element over the
sample vector. RSS is read from /proc/self/statm on Linux,
mach_task_basic_info on macOS.
CLI: --requests / --concurrency / --svid-count / --store-pairs.
Default 20k req @ 16 concurrent.
BENCHMARKS.md checks in a reference run (Docker on M-series
macOS): ~140k req/s S1F1, ~79k req/s S1F3 with 32-SVID list,
~572k S6F11/s push, ~450 bytes per PJ+CJ pair. Three orders of
magnitude headroom over typical fab tool load.
The doc is explicit about what the bench does NOT measure (real
network, persistence I/O, TLS tunnel overhead, multi-session GS
dispatch) — customers should re-run on their target hardware.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
a4599b3b9d |
config: multi-error YAML validator + --validate-config CLI flag
The existing loader throws ConfigError on the first problem it hits.
A customer with a tool-specific equipment.yaml that has six issues
sees one, fixes, restarts, sees the next, fixes, restarts — six
edit-restart cycles before the server even binds. Day-1 friction
is the top support ticket source in fab integrations.
This commit adds a parallel validator that does a separate read-only
pass and surfaces *every* issue at once:
$ secs_server --validate-config \
--config equipment.yaml \
--state-table control_state.yaml
[error] equipment.yaml:5 svids[0].type — unknown SECS-II type `WTF`
[error] equipment.yaml:7 alarms[0].category — value 200 out of range [0, 127]
[error] equipment.yaml:9 host_commands[0].emit_ceid — CEID 999 not declared in `ceids` section
3 error(s), 0 warning(s) across 4 files
What it catches:
- Missing required fields (device.model_name, .software_rev, …)
- Range violations (alarm category must be 0–127, spool streams 1–127,
device.id fits u16, etc.)
- Unknown enum values (SECS-II types, HCACK values, control/PJ/CJ
state and event names — using the right case + snake convention
the runtime parsers enforce)
- Duplicate IDs within svids / dvids / ecids / ceids / alarms,
duplicate PPIDs in recipes, duplicate command names in host_commands
- Referential integrity: host_commands[*].emit_ceid must exist in
ceids; host_commands[*].set_alarm must exist in alarms;
emit_on_control_change must exist in ceids
- PJ-table-specific: `NoState` sentinel rejected as `initial`,
`from`, or `to` (matches loader's existing runtime check)
- yaml-cpp Mark → 1-based line numbers when available
What it doesn't catch (out of scope this round):
- JSON Schema for editor red-squigglies (future)
- Deep semantic checks across state-table reachability
- ECID min/max value parsing (would need numeric type coupling)
Tests cover: clean file passes; multi-error YAML surfaces every issue
on a single pass; line numbers populate; control_state /
process_job_state / control_job_state casing conventions;
format_issues_to renders both severities; the shipped
data/equipment.yaml etc. validate cleanly (regression tripwire if
anyone breaks the demo configs).
INTEGRATION.md §2.3 calls out the flag and suggests CI use.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
d73906f372 |
license: switch contact email to raphael@maenle.net
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
05d58f1a0a |
license: All Rights Reserved proprietary
Hard blocker for any fab customer's procurement / legal review — without a LICENSE in the repo they couldn't even begin evaluation, because permission to read the source is itself something the copyright holder has to grant. This license grants nothing by default. Viewing the repo is the only implicit allowance; everything else (compile, evaluate, benchmark, deploy, sublicense, train ML on, reverse-engineer) requires a separate written agreement with r.maenle@gmail.com. Explicitly *not* granting the carve-outs that open-source licenses imply: no fair use, no internal evaluation, no academic research, no demo, no production deployment. Customers who want any of those need to talk to Raphael first. SPDX-License-Identifier: LicenseRef-Proprietary for tooling. README header gains a license callout pointing at the file and contact email so anyone landing on the GitHub frontpage sees the restriction before reading further. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
7871718848 |
persistence: v1->v2 upgrade test + honest README
README §6 claimed bidirectional forward-compat for journal records. Reality is narrower: - ProcessJobStore (kVersion=2) and SubstrateStore (kVersion=2) accept v1 records on replay — their loaders explicitly switch on the version byte and treat the v2 trailer fields as empty when absent. This is the actual upgrade path the README half-described. - ControlJobStore, CarrierStore, LoadPortStore, ExceptionStore, and SpoolStore use strict `header[1] != kVersion` rejection. A future kVersion bump there without a matching loader-side dispatch would silently nuke every replayed record. The README sold this as a feature; it isn't yet. This commit adds: - tests/test_persistence_upgrade.cpp: five cases that craft journal records byte-by-byte so format drift is caught (no codec round-trip hiding the field layout). PJ v1 -> v2 read; PJ v1 rewrite stamps current kVersion=2; PJ unknown future version rejected; Substrate v1 read with empty history trailer; CJ + Carrier reject unknown versions (tripwire for the strict-version stores). - README §6: replaces the rosy "newer versions ignore unknown trailers" claim with what's actually implemented — multi-version reads on PJ + Substrate, strict equality elsewhere — and points at the test as the contract anchor. When the strict-version stores grow their own v2, the rejection tests will need to flip to acceptance; the layout is right there in the test so the edit is mechanical. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
9653a54584 |
docs+test: thread-safety contract for EquipmentDataModel
INTEGRATION.md §3 used to show a sensor-poll thread calling model->svids.set_value() directly while the io_context thread reads the same SVID for an inbound S1F3. That's a data race — there are zero locks anywhere in EquipmentDataModel and there's no intention to add them. The library is single-threaded by design; the doc was just inviting trouble. This commit makes the actual contract explicit: - INTEGRATION.md §3: thread-safety callout box. All access must run on the io_context that drives the HSMS connection. Sensor updates from other threads marshal via asio::post(io.get_executor(), ...). Same applies to set_*_change_handler callbacks (they fire on the io_context thread; observers must be thread-safe or hand work off). - README.md §3 (Monitoring & observability): added a paragraph noting that hooks fire on the io_context thread, blocking I/O inside a handler stalls the dispatcher, and metrics exporters must respect the same contract. - tests/test_thread_safety.cpp: two scenarios that exercise the canonical pattern — N producer threads asio::post sensor updates onto a worker-driven io_context; reads marshal back through the io. Catches obvious regressions (e.g. someone adding a "convenience" cross-thread mutator that bypasses the strand). A passing run isn't proof of race-freedom under ThreadSanitizer — it pins down the *pattern* customers should follow. TSan integration is a separate workstream. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
54dcf6c532 |
e84: asio adapter for handshake timers + wall-clock test
The E84StateMachine timers landed last commit but stayed theoretical — arming was delivered via abstract callbacks the application had to glue to a real clock. This commit ships the canonical glue: - include/secsgem/gem/e84_asio_timers.hpp: header-only E84AsioTimers wraps three asio::steady_timers, wires set_timer_handlers on attach(), routes async_wait expiry back into fsm.on_timeout(). detach() cancels everything cleanly. - tests/test_e84_asio_timers.cpp: four scenarios exercised through a real asio::io_context with wall-clock timers — TA1 expiry, signal-driven cancel before TA1 fires, TA3 expiry from the Transferring state, and detach() halting further transitions. These cover the integration the synthetic unit tests in test_e84_timers.cpp can't reach. - INTEGRATION.md §4.6: the vendor-side recipe — create the port, set timeouts, make_shared<E84AsioTimers>(...)::attach(), feed signals from your I/O bridge. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
2ea3ab796a |
e84: SEMI §6 handshake timers TA1/TA2/TA3
E84StateMachine had the full signal-level handshake but no timer
enforcement. In a real AMHS that's a deadlock: if equipment is slow to
assert L_REQ / U_REQ, or AMHS is slow to assert BUSY / COMPT, neither
side notices — the wires just sit stuck. SEMI E84 §6 mandates three
timers that bound each leg of the dance.
TA1 — armed in ValidAsserted, cancelled in Load/UnloadReady.
AMHS bounds how long equipment takes to acknowledge VALID.
TA2 — armed in Load/UnloadReady, cancelled in Transferring.
Equipment bounds how long AMHS takes to start the transfer.
TA3 — armed in Transferring, cancelled on Complete.
Equipment bounds the BUSY-phase duration.
The FSM stays I/O-free (it's the design invariant): arm/cancel are
delivered via callbacks, the application owns the asio::steady_timer,
and the application calls `fsm.on_timeout(id)` when its real clock
fires. Stale on_timeout calls (post-cancel race) are no-ops.
On expiry, the FSM transitions to a new `HandoffFault` state, records
the `E84Fault` reason, fires the optional fault_handler, and latches
the fault until `reset()`. Signal jitter on the wires cannot silently
clear a recorded handshake timeout — once you've crossed the timer,
you stop.
Defaults are all-zero, which disables arming. This is what every
existing test relies on, and what back-to-back simulation (no
wall-clock) needs. Production tools call `set_timeouts({2s, 2s, 60s})`
or whatever their port spec dictates.
12 new test cases / 59 assertions: arming per state, cancelling per
exit, expiry-to-fault for all three timers, ES cancels everything,
stale-expiry no-op, fault latching across signal jitter, and a
full-cycle arm/cancel trace.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
a4419e15cd |
conformance: expand harness from 8 to 47 host-driven checks
The previous harness only exercised S1F1/F11/F13/F19, S2F17/F29, S5F5, S7F19 — about 15% of what COMPLIANCE.md claims as ✅. Customers running secs_conformance against their tool got near-zero conformance signal on dynamic event reports, GEM 300, alarm management, exception recovery, terminal services, spool, and PP management. This expansion covers, in one sequential run: - Establish comms + identification (S1F13/F1) - Status / DVID / CEID / EC namelists + values (S1F11/F3/F21/F23, S2F29/F13) - Dynamic event reports: define / link / enable + readback paths (S2F33/F35/F37, S6F15/F19/F21) - All three remote-command forms (S2F41/F21/F49) - Equipment-initiated S6F11 observation triggered by RCMD=START - Trace init, limits attrs, spool reset + transmit (S2F23, S2F47, S2F43, S6F23) - Alarm management: list, list-enabled, enable (S5F5/F7/F3) - Exception recovery: request + abort (S5F13/F17) - PP load-inquire / list / request (S7F1/F19/F5) - Terminal display both directions (S10F3, S10F5) - E40 PJ create / monitor / command / dequeue (S16F11/F7/F5/F13) - E94 CJ create / command / delete (S14F9, S16F27, S14F11) - E87 carrier action / slot map / transfer / cancel (S3F17/F19/F25/F27) - E39 GetAttr (S14F1) - GEM compliance self-report (S1F19) Pass criterion is the spec-mandated reply function code, not any specific ACK value — CarrierIDUnknown / Denied_UnknownObject / PpidNotFound / Error are well-formed F-coded replies and count as protocol-conformant. This lets the harness run against any equipment without preloading state. 47 / 47 PASS against the in-repo demo server. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
06f287b415 |
conformance: standalone secs_conformance harness binary
The closest thing to an in-repo "RTS" — a runnable executable that points at any HSMS-SS equipment and walks through every E30 fundamental + additional capability, reporting pass/fail per check and exiting with the right code for CI / canary use. build/secs_conformance --host <ip> --port 5000 --device 0 Each check sends a host-initiated primary and asserts the equipment replies with the expected stream/function within T3. Checks chain forward through async callbacks (each reply handler kicks off the next check) so the conformance run stays inside one io.run(). Initial check set (mirrors COMPLIANCE.md §3 fundamentals): E37 §7.2 SELECT handshake E30 §6.5 S1F13/F14 Establish Comms E30 §6.7 S1F1/F2 Are You There E30 §6.13 S1F11/F12 SVID Namelist E30 §6.16 S2F29/F30 ECID Namelist E30 §6.20 S2F17/F18 Clock E30 §6.14 S5F5/F6 List Alarms E30 §6.17 S7F19/F20 PP List E30 §6.10 S1F19/F20 GEM Compliance Validated against the demo server: 9/9 PASS. README.md §8 (Compliance + certification) updated to point at the harness as the suggested first-line conformance check. Tool vendors fork apps/secs_conformance.cpp and add their own capability-specific checks alongside. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
d470442a8c |
docs: drop implementation_plan.md, rewrite README for fab deployment
implementation_plan.md was a Layer-0..6 roadmap from the project's
spec-as-data exploration phase; every layer it described is now
shipped (Layer 0 foundations through Layer 4 message catalog +
state machines). Removed.
README rewritten for the fab-deployment audience. Sections added:
1. Persistence directory layout (storage rules, disk budget, DR)
2. Security (network isolation, TLS tunnels, audit logging,
config signing)
3. Monitoring + observability (signals → hooks table, Prometheus
pattern)
4. High availability (active/standby on shared persistence)
5. Deployment patterns (Docker / systemd / k8s)
6. Upgrade path (YAML reload, code rollout, schema versioning)
7. Integration with the fab stack (MES / AMHS / OHT / recipe
engine table)
8. Compliance + certification (fork COMPLIANCE.md per tool, run
RTS)
9. Testing in production (canary, synthetic transactions, shadow
traffic)
10. Operational runbook (incident → first check → mitigation)
Stale stats refreshed: test count went 148/794 → 384/2390;
catalog grew to 164 messages; HSMS-GS, SECS-I T3/T4, per-port E84,
E42 formatted PPs all mentioned.
COMPLIANCE.md §9 lost its stale `implementation_plan.md` reference.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
78fb0c3826 |
e42: enhanced (formatted) process programs S7F23-F26
E42 was an explicit out-of-scope item in the prior COMPLIANCE.md.
This commit closes it.
Wire messages added via the catalog:
S7F23 Formatted PP Send (H↔E, W=1)
S7F24 Formatted PP Ack (ProcessProgramAck)
S7F25 Formatted PP Request (PPID, W=1)
S7F26 Formatted PP Data (E→H, no reply)
Body shape: <L,4 PPID MDLN SOFTREV <L,n <L,2 CCODE <L,m <L,2
PNAME PVAL>>>>>. PVAL is declared ITEM so any SECS-II Item type
round-trips — proven by a test that mixes ASCII, BOOLEAN, U4, F8,
Binary, and nested List values in one step.
RecipeStore extension:
add_formatted(ppid, FormattedRecipe{mdln, softrev, steps})
get_formatted(ppid) -> optional<FormattedRecipe>
has_formatted(ppid) -> bool
Formatted + opaque views live alongside each other: a PPID can carry
both, size() counts unique PPIDs. remove() kills both views.
Six new tests cover wire round-trip per function, every
ProcessProgramAck code, ITEM passthrough, and the store's dual-view
semantics.
COMPLIANCE.md updated: E30 §6.17 row mentions S7F23-F26, S5 message
table grows two rows, §8 "out of scope" entry for E42 removed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
d4d1a411d7 |
secsi: T3 / T4 enforcement moved into the FSM
The SECS-I Protocol FSM now enforces T3 (reply timeout) and T4
(inter-block timeout) directly, instead of leaving them as
upper-layer hooks.
T3: on complete_send, if the block we just acked had W=1, record its
system_bytes in awaiting_reply_sys_ and emit ActionStartTimer{T3}.
deliver_recv cancels T3 when a block arrives whose system_bytes
match the outstanding request. EventTimeout{T3} aborts the FSM with
"T3 reply timeout".
T4: deliver_recv emits ActionStartTimer{T4} whenever the delivered
block has end_block=false. The next block's deliver_recv cancels
the timer; EventTimeout{T4} aborts with "T4 inter-block timeout".
abort() now also cancels T3/T4 and clears the tracking state.
Test changes:
- Old "T3/T4 are FSM-level no-ops" test → REPLACED by four new
tests: T3 arm+expire, T3 arm+matching-reply cancels, T4
arm+expire, T4 arm+next-block cancels.
- Two new observer accessors on Protocol (awaiting_reply,
awaiting_next_block) so the tests can assert tracking state
without poking internals.
COMPLIANCE.md §1a: T3 + T4 rows go ⬜ → ✅.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
77197b9c1e |
e84: per-port FSM via E84PortStore
E84 (Parallel I/O) is fundamentally per-load-port: each port has its own ten-wire handshake with the AMHS. Earlier revisions modeled it as a single equipment-wide FSM; this commit refactors to a per-port store, so multi-LP tools can run independent handshakes in parallel. Public API change in EquipmentDataModel: E84StateMachine e84; -> removed E84PortStore e84_ports; // create(port_id), get(port_id), ... Convenience pass-throughs: E84PortStore::on_signal_change auto-creates the port on first use (ergonomic for demos); applications should call create() explicitly with their full port set. The two existing callsites (test_gem300_scenario, test_e87_wire_scenarios) are updated. The multi-LP test now demonstrates the actual win: interleaved LP1 load + LP2 unload handshakes that reach their respective Ready states without sequencing, and an ES on LP1 that does NOT affect LP2 — exactly the failure mode the previous design couldn't catch. Five new dedicated tests in test_e84_ports.cpp for the store itself. COMPLIANCE.md §4i updated: row now reflects per-port design. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
2f0a4ba339 |
e30: S10F7 broadcast terminal display
Adds the last terminal-services message: a multi-line broadcast push to all terminals, no reply. Same TID+lines body as S10F5, W=0. Generated via the catalog: data/messages.yaml schema entry + auto-generated s10f7_terminal_display_broadcast / parse_s10f7. Test round-trips TID and a 3-line broadcast through the builder and parser, confirms W=0. COMPLIANCE.md updated: S10F7 row in §5 added; §8 "out of scope" entry removed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
d0c7fb71b6 |
hsms: HSMS-GS multi-session support (E37 §11)
Connection now supports both HSMS-SS (single session — the
constructor's behaviour, unchanged) and HSMS-GS (multi-session).
add_session(device_id) registers additional sessions; each one has
its own NotSelected/Selected state and its own message/selected
handlers. In GS mode the Select.req carries session_id=device_id;
in SS mode it stays at 0xFFFF (legacy). Linktest/Separate remain
connection-scope per spec.
Public API additions:
add_session(device_id)
set_session_message_handler(device_id, h)
set_session_selected_handler(device_id, h)
session_state(device_id) -> State
is_session_selected(device_id) -> bool
send_request(device_id, msg, cb)
send_data(device_id, msg)
Internal refactor: state_/on_message_/on_selected_ folded into a
SessionSlot map keyed by device_id; SS-style getters/setters route
through the primary session. T7 + linktest are connection-scope —
T7 fires only when no session is selected; linktest runs while at
least one is.
Five wire-level tests:
- passive: two sessions selected independently via Select.req
with their own session_id
- GS Select.req for an unregistered session id is Rejected
(EntityNotSelected)
- data routed by session_id; data on a not-selected session is
Rejected
- active: two registered sessions both end up selected via
serialized Select.req per session
- SS legacy: existing single-session API still works (session_id
0xFFFF in Select.req)
COMPLIANCE.md §1 updated: HSMS-GS row goes ⬜ → ✅.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
998e81b3d8 |
persistence: substrate history journaling in v2 record
Per-substrate transition history now survives restart. Each entry's
steady_clock timestamp is written as a system_clock-millis snapshot;
on replay the steady_clock time_point is reconstructed relative to
the current (steady_now, system_now) pair, so inter-event spacing
is preserved across restarts even if the FSM is in a different
process. Absolute wall-clock accuracy degrades by any NTP step
that happened between write and read; that's a documented caveat.
Record format goes v1 → v2. v1 (history-less) records still load,
just with empty history.
Test updates:
- the old "history is NOT journaled" test is REPLACED with one
that asserts every axis + event + label round-trips.
- hand-crafted v1 record on disk still loads (proves backwards
compat).
- 15 ms-spaced events restore with their spacing intact (±slop
for scheduler jitter).
Closes the "substrate history persistence" caveat from the post-#1-13
status writeup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
d9f23d6db8 |
persistence: PJ rcpvars + prprocessparams in v2 record format
Closes the v1 caveat: the optional E40-0705 trailers on S16F11 —
recipe variables (RcpVar) and process parameters (ProcessParam),
each carrying a secs2::Item value of arbitrary type — now survive
restart.
Record format bumps to v2:
v2 header = v1 header
+ [u16 rcpvar_count][repeat: u16 name_len, name, u32 enc_len,
secs2::encode(value)]
+ [u16 ppparam_count][...same shape]
v1 records are still accepted by load_record_ (no extras come back).
Two new tests:
- round-trip mixed F4 / ASCII / U4 / nested-list values through
rcpvars + prprocessparams
- hand-crafted v1 record on disk still loads cleanly, just with
empty extras (proves backwards compat)
Closes the "PJ rcpvars / prprocessparams persistence" caveat from
the post-#1-13 status writeup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
f206df763e |
docs: customer integration tutorial (INTEGRATION.md)
End-to-end guide for an equipment vendor integrating the library
into a real semiconductor tool:
1. Architecture: what the runtime provides vs what the application
contributes — three boundary classes (EquipmentDataModel,
Router, hsms::Connection).
2. 30-minute first connection: YAML + minimal main() + run.
3. Wiring real sensors to SVIDs.
4. Plugging the FSMs into the tool: EPT, carriers, substrates,
E40 PJ / E94 CJ, alarms, recoverable exceptions.
5. Persistence: enable_persistence(dir) per store, storage budget,
replay semantics, current caveats.
6. Monitoring + observability: connection lifecycle hooks,
state-change handlers, S9 protocol errors.
7. Recommended deployment layout (/opt/acme-secsgem/...).
8. Integration testing checklist.
9. When to extend the runtime.
10. The honest gap between "this stack runs" and "this is a
certified GEM tool".
Cross-referenced from COMPLIANCE.md §9 distinction (stack vs tool).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
7213ddfbf1 |
tests: HSMS connection concurrency / interleaved transactions
Real GEM sessions don't serialize requests — the host can have many
primaries outstanding, replies may arrive in any order, and both
peers can talk at once. Connection demuxes via system_bytes per
E37 §8.3; this commit pins the behaviour with four wire tests:
- 5 in-flight requests; equipment buffers all primaries before
replying — proves Connection holds the pending map correctly
even when no replies are coming.
- 7 pipelined primaries with synchronous in-handler replies;
every host callback fires with the correct function and stream.
- Bidirectional in-flight: host issues 3 primaries while equipment
issues 3 of its own; all 6 callbacks resolve with the right
replies.
- 100-burst sequential cycle; confirms the pending_requests_ map
doesn't leak entries (every reply delivered ⇒ map drained).
Closes #13 in the test-gap backlog.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
158ebed5c8 |
tests: identifier-width wildcard matrix
SEMI E5 allows identifier fields (DATAID, RPTID, VID, CEID, ALID,
EXID, OBJID, …) to be encoded as U1, U2, U4, or U8. Our parsers
route through any_unsigned_first<T> in messages_helpers.hpp. The
existing per-message round-trip tests prove the U4 path; this
commit adds the cross-width matrix that the interop incident with
secsgem-py demanded:
- as_u4_scalar accepts U1/U2/U4/U8 inputs for the same value
- as_u8_scalar accepts every narrower width
- as_u1_scalar accepts wider widths when the value fits
- as_u1_scalar / as_u2_scalar REJECT out-of-range values rather
than silently truncating
- codec round-trip preserves the format byte AND the value
- signed counterparts (as_i4_scalar) follow the same rule for I1/I2
If a future code-gen change hard-codes a single width on any
identifier field, the rejection case here breaks loudly.
Closes #12 in the test-gap backlog (renumbered: this is gap entry
"identifier wildcard matrix").
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|