4f3031aeb96ab6daa88f87bd3ecc029148d92efb
24 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 raphael
|
af1a159c59 |
docs: bring the documentation up to the daemon/client era
A large gap had opened between the docs and the code: the README and INTEGRATION guide did not mention the gRPC daemon or the Python client at all (the entire vendor surface), ARCHITECTURE still described secs_server as the ~1200-line canonical wiring example (it is a ~110-line thin main over EquipmentRuntime), and test counts across six files were stale (445/2753 -> 473/3087 core + the separate 125-assertion daemon suite). - README: new "Integrating your tool (pick a tier)" section — Python client / any-language gRPC / embedded C++ — plus daemon tests and tools/run_interop.sh in the Testing section. - ARCHITECTURE: layer diagram gains the vendor-surface and EquipmentRuntime/default_handlers tiers; stale wiring row fixed. - INTEGRATION: three-tier chooser up front (this guide = the C++ tier). - ch30 tour: secs_gemd + secs_gemd_tests in the binaries table. - ch31: example alarm used a nonexistent `alcd:` field with bit 7 set (which the validator forbids) -> real `category:`/`name:` fields, and the roles: block documented. - ch35: handler-location note now points at default_handlers.cpp's 15 per-capability register_* functions. - ch40: built-artifacts list + sample output counts. - ch50: secsgem::gem runtime/default_handlers/handler_slot/name_index includes + new secsgem::daemon namespace section. - PROOFS: test-count table gains the runtime/handlers/daemon row so the tally adds up; daemon suite noted. VERIFICATION/COMPLIANCE counts. - interop/README: the one-command runner + the two daemon-track harnesses (daemon_interop, pyclient_interop). Audited via a docs-vs-code sweep (the audit itself under-reported: it validated counts textually; reality was 473/3087). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
raphael
|
dae6bfd747 |
docs: streamline tone across reference docs
Tone pass across the non-tutorial markdown — README, PROOFS,
ARCHITECTURE, BENCHMARKS, COMPLIANCE, FAQ, MES_INTEROP, SECURITY,
and interop/README. Three patterns came out:
- Bug-history war stories ("Past interop sweeps surfaced…",
"What these harnesses caught: 1. Strict U-width parsing…").
- Chat-with-reader framing ("Don't skip TLS unless…", "Treat as a
punch list", "If you're running in a pod…", "Misconfiguration
incidents drop dramatically").
- Self-referential narration ("we ship", "our codec", "the
codebase's most-tested layer", "three orders of magnitude above
fab load", "the gift that keeps giving").
README also drops the standalone ThreadSanitizer subsection under
Build details (now a single line under the new Testing section).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
fc3422a4a9 |
docs: move root .md files into docs/ + update every reference
Picks up the file renames that landed alongside the previous commit and fixes everything that pointed at the old root locations: - README.md doc-map updated: every entry now points at docs/X.md, with a new "docs/" lead entry pointing at the guided-tour index. - README inline cross-refs (ARCHITECTURE / INTEGRATION / SECURITY / BENCHMARKS / MES_INTEROP / PROOFS) repointed to docs/. - README "Interop" section rewritten — used to mention only secsgem-py; now covers all four external validators (secsgem-py 31 / secs4java8 55 / tshark 69 frames / libFuzzer 200 k+ runs) with a one-line summary each, plus pointers to interop/README.md and docs/VERIFICATION.md. - README "Deferred follow-ups" cleaned: dropped the explanatory "Listed here so reviewers don't go looking for them in COMPLIANCE.md and find an 'out of scope' entry that sounds defensive" sentence — the section header speaks for itself. - docs/00_index.md "Where the rest of the docs live" table: dropped every `../` prefix since the docs are now siblings. - docs/01_what_is_secs_gem.md PROOFS reference updated to sibling. - docs/02_the_cast.md INTEGRATION + MES_INTEROP refs updated to siblings; dropped the stale "at the repo root" wording. - interop/README.md: VERIFICATION + PROOFS refs updated to ../docs/X.md; stale "~24 + 4 checks" updated to 31 (matches PROOFS.md and README). - examples/pvd_tool/README.md: every doc cross-ref now points at ../../docs/X.md. - Source / data / CI comments mentioning doc names (e.g. "INTEGRATION.md §3", "COMPLIANCE.md gap") rewritten to "docs/INTEGRATION.md §3" etc. — affects 9 files across include/, apps/, tests/, data/, examples/, .gitea/workflows/. Verified: full build under docker passes, 445/445 test cases pass, 2 753/2 753 assertions pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
b01dedfaa5 |
docs: drop COMPLIANCE §8 "out of scope" and broaden §7 to all 4 validators
§8 was carrying two items that neither read as "deliberately out of scope" nor matched the framing of the section: - Equipment Processing States — E30 §6.3 explicitly leaves concrete states tool-defined. The framework ships the ControlTransitionTable engine and YAML loader; vendors supply IDLE/SETUP/READY/EXECUTING. That's a design choice, not a gap. §3 line 94 already documents it. - Serial-port wiring for SECS-I — the FSM is implemented and tested end-to-end over TCP; only the asio serial_port adapter is missing. That's deferred, not out of scope. §1a line 64 already lists it with status ⬜. So §8 is dropped, §9 renumbers to §8, and the deferred follow-up gets its own short section in the README so customers know it's tracked without sounding defensive. §7 used to be titled "Interoperability with secsgem-py 0.3.0" and mentioned only that one external implementation. We now have four external validators (secsgem-py + secs4java8 + tshark dissector + libFuzzer), so the section is renamed "Interoperability with external implementations" and broadened to cover all of them with their actual check counts. Stale "24 named checks" updated to the current 31; "three consecutive clean runs" line dropped as audit-language no longer earning its keep now that it's a CI step. FAQ's "What's not implemented?" answer rewritten to point at the README "Deferred follow-ups" section and COMPLIANCE §8 (new numbering), with a brief note explaining that Equipment Processing States are spec-by-design tool-defined. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
b031f057af |
docs: customer-ready sweep + README restructure + tshark CI fix
Audit pass over the public-facing surface so a customer can read it end-to-end without tripping on stale numbers or self-contradictions. README + docs accuracy: - Test counts 426 → 445, assertions 2 557 → 2 753 (verified via doctest run); E5 row was missing test_e5_kat (19 cases) - Interop checks 24 → 31, COMPLIANCE.md message count 149 → 164, COMPLIANCE.md "291 cases / 1515 assertions" → 445 / 2 753 - README "60+ test IDs" for MES_INTEROP → actual 59 - PVD example counts: 32 SVIDs/17 CEIDs → 29/21, "~40 handlers in ~200 lines" → 51 in ~460, "~700 lines" → ~1,100; main.cpp header table-of-contents resynced with the actual 7 sections Out-of-scope honesty (COMPLIANCE.md §8 + FAQ.md): - Removed HSMS-GS (was both ✅ implemented in §1 and "out of scope" in §8; INTEGRATION.md §7 documents using it) - Removed multi-block SECS-I (split_message/assemble_message exist with 4 dedicated tests) - Added serial-port wiring as the genuine open ⬜ item — FSM is tested end-to-end over TCP; only the asio serial_port glue is deferred - COMPLIANCE.md intro now lists E42 and notes "E37 (SS + GS)" README restructure: - Moved the 8-command proof table and per-standard test-coverage table to a new PROOFS.md (72 lines) - README now leads with what / Quick start / Documentation map, then a one-paragraph "How it's proved" linking to PROOFS.md - Updated cross-refs in FAQ.md, GLOSSARY.md, VERIFICATION.md, and interop/README.md to point at PROOFS.md CI fix — tshark-dissector job: - interop/tshark_validate.sh hardcoded /app/build/secs_server etc. which only works inside the docker image. Now derives ROOT from the script's own location and accepts BUILD/SERVER/CLIENT/DATA env overrides, so CI can run it from the workspace dir - Verified still passes in docker (69 frames, 0 malformed) .gitignore: - Added build-fuzz/ and build-tsan/ (were showing as untracked) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
195ecc689f |
docs: GLOSSARY + FAQ + interop README refresh + doc-map fixes
Fills four documentation gaps surfaced by the doc audit:
1. README "Documentation map" was missing VERIFICATION.md (the file
that backs the proof-of-feature-completeness claims) and is now
pointing at the new files added in this commit too — ARCHITECTURE,
GLOSSARY, FAQ, examples/pvd_tool/ (the last two land next).
2. interop/README.md only documented secsgem-py. Three of the five
external validators (tshark, secs4j, libFuzzer) plus the E5 KAT
were invisible from the directory's own README. Rewritten as a
complete index — what's external, what each catches, how to run,
what bugs they've already surfaced, when to add a new validator.
3. GLOSSARY.md is new. Every SEMI acronym used in the codebase or
the docs gets one row: SVID, DVID, CEID, RPTID, ALID, ECID, PPID,
MID, CARRIERID, PRJOBID, CTLJOBID, SUBSTID, OBJSPEC, OBJTYPE,
MDLN, SOFTREV, EQPTYP, DATAID + every ACK code (COMMACK, ONLACK,
OFLACK, HCACK, CMDA, ACKC5-7-10, DRACK, LRACK, ERACK, EAC, TIACK,
GRANT, ALCD, OBJACK) + stream/function shorthand + HSMS terms +
T-timers + E84 signals + the standards lineup + codebase shortcuts
("the model", "the router", "the proof", etc.). Cuts week-1
onboarding time.
4. FAQ.md is new. Canonical answers to the questions that come up
once per integration: why HSMS unencrypted, SVID vs DVID, PJ vs
CJ, who fires FSM transitions, what runs on which thread, how to
add a new SECS-II message, ASCII vs Binary, common MES quirks,
how spool works, robustness fuzz vs libFuzzer, conformance vs
interop, what's not implemented.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
db90a21e1d |
verify: expand secs4j harness 20 → 55 checks
Every check the user could ask for now lands. secs4j's
comm.send(stream, function, w, body) takes arbitrary S/F + arbitrary
Secs2 body, so coverage was never coverage-limited by the Java side
— the original 20 was just the minimum to fill the gaps secsgem-py
couldn't reach.
Adds:
- Status data: S1F3, S1F11
- EC management: S2F13, S2F15 (set TimeFormat), S2F29
- Event reports: S2F33, S2F35, S2F37 (full define-link-enable
sequence), S6F15, S6F19, S6F21
- Remote control: S2F41 (modern RCMD=START + observed S6F11),
S2F21 (legacy RCMD=STOP),
S2F41 RCMD=FAULT + observed S5F1
- Alarms: S5F3, S5F5, S5F7
- Spool: S2F43, S6F23
- PP management: S7F1, S7F3, S7F5, S7F17, S7F19
- Terminal: S10F3 (single), S10F5 (multi-line)
- E40 PJ: S16F11 (full E40 body — MF + PRRECIPEMETHOD +
RecipeSpec + mtrloutspec + processparams),
S16F7 (monitor), S16F13 (dequeue)
- Limits: S2F45, S2F47
- Trace: S2F23 (5-field body)
- E39: S14F1 (GetAttr)
Plus a SecsMessageReceiveListener that captures every equipment-
initiated primary into a ConcurrentLinkedQueue and replies to S5F1
(ACKC5=0), S6F11 (ACKC6=0), S16F9 (W=0 no reply) so the
equipment's T3 doesn't fire on our watch. Two checks now assert
the unsolicited path:
- After RCMD=START, an S6F11 with the linked report must arrive
within 400ms
- After RCMD=FAULT, an S5F1 with the alarm must arrive within
400ms
Both observed against the demo equipment.
Result: 55/55 PASS. Two independent implementations
(secsgem-py + secs4java8) now corroborate the wire surface in
overlapping but distinct slices. Full E40 body — the one that
defeated secsgem-py's SFDL grammar — round-trips cleanly through
secs4j.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
4ddf8e0f48 |
verify: libFuzzer harness for secs2::decode + try_parse_sml
Coverage-guided structural search for crashes and undefined behaviour on arbitrary input to our two parsers. What's wired: - -DSECSGEM_FUZZ=ON CMake option, clang-only. Adds -fsanitize=fuzzer-no-link,address,undefined to all targets + -fsanitize=fuzzer to the two fuzz executables. - apps/fuzz_secs2_decode.cpp — feeds raw bytes to secs2::decode. Catches secs2::CodecError (expected) but traps on anything else leaking (would be a hardening bug). - apps/fuzz_sml_parse.cpp — feeds string to try_parse_sml, which is contractually nothrow-equivalent; traps on any exception. - .gitea/workflows/ci.yml — `libfuzzer` job builds with clang and runs each fuzzer for 60s in CI. Any crash / ASan / UBSan flag fails the job. - Dockerfile gains clang + libclang-rt-18-dev so devs can run locally with the same toolchain. Result on a fresh 30-second local run: fuzz_secs2_decode: 70 727 random inputs, 0 crashes fuzz_sml_parse: 284 950 random inputs, 0 crashes The coverage-guided search found and synthesized inputs that exercise: zero-byte, single-byte format tags, all length-byte counts (1/2/3), nested lists, format bytes with reserved bits, the "BOOLEAN" SML token, malformed quoted strings, etc. libFuzzer's recommended dictionary at the end of each run shows what bytes / substrings the coverage feedback discovered as discriminating — useful signals if we ever want a hand-curated corpus. README proof table grows to 8 commands. After this: - 426 unit tests (internal) - 47 conformance harness checks (internal) - 24 secsgem-py interop checks (external — Python ref impl) - 20 secs4j interop checks (external — independent Java impl) - 69 frames dissected by Wireshark HSMS dissector (external) - 196 SEMI E5 KAT assertions (standards body's encoding rules) - **~70k + ~285k random inputs, 0 crashes (external)** - 100k random tool ops with all invariants holding (internal) - YAML validation (internal) - TSan clean on 2 557 assertions (internal correctness aid) Five distinct external proofs now, each covering a different angle. Plan: VERIFICATION.md §4. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
2fce2fad0c |
verify: secs4j cross-validation (independent Java implementation)
20 cross-validation checks PASS against [secs4java8] (Apache 2.0,
kenta-shimizu) — an independent SECS/HSMS implementation in Java by
a different author from a different language ecosystem. Distinct
implementer = independent spec interpretation. Two libraries
agreeing on wire bytes is much stronger evidence of spec-correctness
than either alone.
Coverage targets the gap the secsgem-py interop deliberately skipped
(secsgem-py's SFDL grammar couldn't easily express GEM 300 bodies
with variable lists of named scalars):
- S1F1/F13/F17/F19/F21/F23 — establish comms + namelists
- S2F17 — clock
- S2F23 — trace init (5-field body)
- S2F49 — enhanced remote command (DATAID + OBJSPEC + RCMD + params)
- S3F17/F19/F25/F27 — full E87 carrier surface (action, slot map
verify, transfer with port pair, cancel)
- S5F13/F17 — exception recovery (EXID + EXRECVRA)
- S14F9/F11 — E94 CJ create with prjobids list, CJ delete
- S16F5/F27 — E40 PJ command, E94 CJ command
- S1F15 — offline cleanup
20/20 PASS against the demo equipment. Reply S/F matches the spec
for every transaction; specific ACK values vary by equipment state
(CarrierIDUnknown for an unknown carrier is just as valid as Accept
for a known one) so we assert on the wire shape, not the result.
Ship layout:
interop/secs4j/Dockerfile — eclipse-temurin:21-jdk + clone
+ build of secs4java8 → Export.jar
interop/secs4j/Secs4jHostHarness.java
— 20 round_trip assertions; uses
Secs2.list/uint4/ascii to build
full GEM 300 bodies; comm.send()
for arbitrary S/F pairs
interop/secs4j_validate.sh — orchestrator: builds image,
compiles harness, starts compose
server, runs Java container on
the secs network against it
.gitea/workflows/ci.yml — secs4j-interop job in CI
README.md — proof table grows to 7 commands
.gitignore — *.class
After this commit our proof chain has:
- SEMI E5 KAT (standards body's own arithmetic)
- tshark dissector (Wireshark's HSMS impl)
- secsgem-py interop (Python reference impl)
- **secs4j interop** (independent Java impl)
+ 426 unit tests, 47 conformance harness checks, 100k random ops,
YAML validation
Four independent external proofs, three of them on overlapping wire
surface from independent angles.
Plan: VERIFICATION.md §3.
[secs4java8]: https://github.com/kenta-shimizu/secs4java8
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
5baf3f4dc7 |
verify: tshark HSMS dissector validation (independent third codec)
Wireshark's built-in HSMS dissector — written by network-protocol authors who don't know us, didn't talk to us, and don't share implementation details with secsgem-py — is a third independent codec for our framing. If they parse our pcap without warnings, our HSMS framing is wire-correct independently of both our internal tests and the secsgem-py interop path. interop/tshark_validate.sh: - Boots secs_server on 127.0.0.1:5099 (away from the demo port) - Captures the loopback wire traffic with tcpdump - Runs secs_client through ~24 transactions plus Separate.req + TCP FIN - Parses the pcap with tshark -V using the HSMS dissector - Asserts: no "Malformed Packet", no "Dissector bug", at least one HSMS frame, expected tokens present (Select.req/rsp, Separate.req, Data message), reports histogram (count by control type + distinct S/F pairs) Result against the demo: 69 HSMS frames dissected, 49 distinct S/F pairs (S01F01..S16F28), all clean. Dockerfile gains tshark + tcpdump. .gitea/workflows/ci.yml gains a `tshark-dissector` job that runs this validator as part of every push to main. README proof table grows to 6 commands. VERIFICATION.md §1a documents a follow-up: round-trip the KAT fixtures through secsgem-py to corroborate that the format codes we used match an independent implementation. Strengthens the KAT proof from "internally consistent" to "confirmed by a second implementer who read the spec without talking to us." Plan: VERIFICATION.md §2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
e82f67ecad |
docs: README restructure + proof-of-feature-completeness section
The old README mixed intro, quickstart, architecture, and 10 sections
of production deployment over 419 lines, with significant overlap
with INTEGRATION.md. It claimed "implements every standard" without
making the claim concrete.
Restructured to ~250 lines with the proof front and center.
New top-of-README "Proof of feature-completeness" section: five
commands that, when they all exit zero on a fresh clone, prove the
COMPLIANCE.md claims. Each command verified end-to-end before
landing in this commit:
1. docker compose run --rm tests
→ 426 cases / 2557 assertions PASS
2. secs_conformance --host server --port 5000
→ 47 / 47 wire-level checks PASS
3. host_vs_cpp_server.py --host server
→ 24 secsgem-py interop checks PASS
4. SECSGEM_ROBUSTNESS_SOAK=1 secsgem_tests -tc='*soak*'
→ 100 000 random tool operations, all invariants hold
5. secs_server --validate-config <all four YAMLs>
→ 0 errors, 0 warnings across the shipped configs
Plus a per-standard test-coverage table mapping every claimed SEMI
standard (E5, E5 §13, E4, E37, E30, E40, E94, E42, E87, E90, E116,
E120/E39, E157, E84) to its test files and case count, summing to
426 to match the doctest totals. Counts verified by
`grep -c TEST_CASE` per file.
CI also runs the TSan lane (separate job in
.gitea/workflows/ci.yml); README documents it under Build details.
Content moved out of README into specialized docs (eliminates
duplication):
- Security configs → SECURITY.md (was 14-line bullet list; now a
365-line file with nftables, stunnel, minisign, SIEM schema)
- Persistence layout + monitoring + HA + deployment patterns +
upgrade discipline + fab-stack integration → INTEGRATION.md
- Performance envelope → BENCHMARKS.md
- MES interop punch list → MES_INTEROP.md
README now reads top-to-bottom: what this is → license → proof →
quickstart → doc map → architecture → adding capabilities →
production (1-line pointers to the deep docs) → build details →
interop.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
e3765a5176 |
persistence: multi-version reads across every store
ProcessJobStore and SubstrateStore already implemented the loader-accepts-any-version-in-[1, kVersion] pattern. The other five stores (ControlJobStore, CarrierStore, LoadPortStore, ExceptionStore, SpoolStore) used strict `header[1] != kVersion` rejection, meaning a future kVersion bump there would silently nuke every persisted record on first replay. That's a footgun the test_persistence_upgrade test already flagged as a tripwire. This commit flips the strict checks to `< 1 || > kVersion`, mirroring PJ + Substrate. No format change (kVersion stays at 1 across the five stores), but: - Future v2 of any store now Just Works: add fields at the end of write_record_, bump kVersion to 2, gate the new reads behind `if (version >= 2)`. Old v1 records on disk continue to replay with the new fields defaulted. - Future versions beyond kVersion still get rejected (downgrade protection — older code can't try to decode trailers it doesn't understand). Comment blocks on each kVersion declaration now describe the upgrade discipline so the next contributor doesn't reinvent it. Test additions: - Positive test that v1 ControlJob records load on current code (will continue to pass when kVersion bumps to 2, proving v1 is still readable) - ExceptionStore rejects a v9 (future) record, matching CJ + Carrier - The existing tripwire tests get retitled from "rejects unknown version" to "rejects a future version" to reflect the new contract README §6 gets honest: every store is now multi-version-aware, not just PJ + Substrate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
ce5abb4f72 |
docs: real-MES interop test plan (day-1 punch list)
interop/ cross-validates against secsgem-py 0.3.0 — the Python reference. That's not what a fab actually runs. Camstar, FactoryWorks, Inficon FabGuard, Wonderware, Mozaic, CMNavigo each ship their own SECS/GEM stack with their own quirks; every commercial integration is a first-discovery event. MES_INTEROP.md is the structured protocol customers run against their MES *before* connecting a real tool: - 9 test sections covering HSMS plumbing, establish-comms, dynamic event reports, alarms, remote control, PP management, terminal services, GEM 300 (E40/E87/E94), spool, clock+ECs - 60+ test IDs with expected wire behaviour and known quirks per MES vendor (compiled from prior integration support) - Soak + cutover checklist (memory, spool, T-timers, dashboards) - Reporting-back protocol for MES-specific bugs that this codebase should handle Treated as a punch list with PASS/FAIL/N-A per row, captured wire trace per row, and a 90-day archive of the lot — that's the audit trail a fab's quality team will ask for. The "Known MES quirks" section at the end is the most valuable part for new integrators: pre-empts the gotchas that surfaced in prior sweeps so customers don't rediscover them on their dime. README header gets a fifth bullet pointing at the file. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
9c5d67fdad |
bench: secs_bench harness + BENCHMARKS.md baseline
Customer SREs and capacity planners had nothing to point at.
INTEGRATION.md asked the right questions ("how many tx/sec?"
"how much memory per active CJ?") but had no numbers.
secs_bench spins up an in-process passive equipment + active host
on an OS-allocated port, runs three canned workloads, and emits a
markdown table customers can capture and diff across commits:
- S1F1/F2 header-only round-trip — dispatch + framing baseline
- S1F3/F4 with N SVIDs — encode + decode throughput
- S6F11 push (W=0) — one-way emission ceiling
- PJ + CJ pair memory footprint — bytes per active job
Latency reports p50/p95/p99/max via std::nth_element over the
sample vector. RSS is read from /proc/self/statm on Linux,
mach_task_basic_info on macOS.
CLI: --requests / --concurrency / --svid-count / --store-pairs.
Default 20k req @ 16 concurrent.
BENCHMARKS.md checks in a reference run (Docker on M-series
macOS): ~140k req/s S1F1, ~79k req/s S1F3 with 32-SVID list,
~572k S6F11/s push, ~450 bytes per PJ+CJ pair. Three orders of
magnitude headroom over typical fab tool load.
The doc is explicit about what the bench does NOT measure (real
network, persistence I/O, TLS tunnel overhead, multi-session GS
dispatch) — customers should re-run on their target hardware.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
d73906f372 |
license: switch contact email to raphael@maenle.net
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
05d58f1a0a |
license: All Rights Reserved proprietary
Hard blocker for any fab customer's procurement / legal review — without a LICENSE in the repo they couldn't even begin evaluation, because permission to read the source is itself something the copyright holder has to grant. This license grants nothing by default. Viewing the repo is the only implicit allowance; everything else (compile, evaluate, benchmark, deploy, sublicense, train ML on, reverse-engineer) requires a separate written agreement with r.maenle@gmail.com. Explicitly *not* granting the carve-outs that open-source licenses imply: no fair use, no internal evaluation, no academic research, no demo, no production deployment. Customers who want any of those need to talk to Raphael first. SPDX-License-Identifier: LicenseRef-Proprietary for tooling. README header gains a license callout pointing at the file and contact email so anyone landing on the GitHub frontpage sees the restriction before reading further. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
7871718848 |
persistence: v1->v2 upgrade test + honest README
README §6 claimed bidirectional forward-compat for journal records. Reality is narrower: - ProcessJobStore (kVersion=2) and SubstrateStore (kVersion=2) accept v1 records on replay — their loaders explicitly switch on the version byte and treat the v2 trailer fields as empty when absent. This is the actual upgrade path the README half-described. - ControlJobStore, CarrierStore, LoadPortStore, ExceptionStore, and SpoolStore use strict `header[1] != kVersion` rejection. A future kVersion bump there without a matching loader-side dispatch would silently nuke every replayed record. The README sold this as a feature; it isn't yet. This commit adds: - tests/test_persistence_upgrade.cpp: five cases that craft journal records byte-by-byte so format drift is caught (no codec round-trip hiding the field layout). PJ v1 -> v2 read; PJ v1 rewrite stamps current kVersion=2; PJ unknown future version rejected; Substrate v1 read with empty history trailer; CJ + Carrier reject unknown versions (tripwire for the strict-version stores). - README §6: replaces the rosy "newer versions ignore unknown trailers" claim with what's actually implemented — multi-version reads on PJ + Substrate, strict equality elsewhere — and points at the test as the contract anchor. When the strict-version stores grow their own v2, the rejection tests will need to flip to acceptance; the layout is right there in the test so the edit is mechanical. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
9653a54584 |
docs+test: thread-safety contract for EquipmentDataModel
INTEGRATION.md §3 used to show a sensor-poll thread calling model->svids.set_value() directly while the io_context thread reads the same SVID for an inbound S1F3. That's a data race — there are zero locks anywhere in EquipmentDataModel and there's no intention to add them. The library is single-threaded by design; the doc was just inviting trouble. This commit makes the actual contract explicit: - INTEGRATION.md §3: thread-safety callout box. All access must run on the io_context that drives the HSMS connection. Sensor updates from other threads marshal via asio::post(io.get_executor(), ...). Same applies to set_*_change_handler callbacks (they fire on the io_context thread; observers must be thread-safe or hand work off). - README.md §3 (Monitoring & observability): added a paragraph noting that hooks fire on the io_context thread, blocking I/O inside a handler stalls the dispatcher, and metrics exporters must respect the same contract. - tests/test_thread_safety.cpp: two scenarios that exercise the canonical pattern — N producer threads asio::post sensor updates onto a worker-driven io_context; reads marshal back through the io. Catches obvious regressions (e.g. someone adding a "convenience" cross-thread mutator that bypasses the strand). A passing run isn't proof of race-freedom under ThreadSanitizer — it pins down the *pattern* customers should follow. TSan integration is a separate workstream. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
06f287b415 |
conformance: standalone secs_conformance harness binary
The closest thing to an in-repo "RTS" — a runnable executable that points at any HSMS-SS equipment and walks through every E30 fundamental + additional capability, reporting pass/fail per check and exiting with the right code for CI / canary use. build/secs_conformance --host <ip> --port 5000 --device 0 Each check sends a host-initiated primary and asserts the equipment replies with the expected stream/function within T3. Checks chain forward through async callbacks (each reply handler kicks off the next check) so the conformance run stays inside one io.run(). Initial check set (mirrors COMPLIANCE.md §3 fundamentals): E37 §7.2 SELECT handshake E30 §6.5 S1F13/F14 Establish Comms E30 §6.7 S1F1/F2 Are You There E30 §6.13 S1F11/F12 SVID Namelist E30 §6.16 S2F29/F30 ECID Namelist E30 §6.20 S2F17/F18 Clock E30 §6.14 S5F5/F6 List Alarms E30 §6.17 S7F19/F20 PP List E30 §6.10 S1F19/F20 GEM Compliance Validated against the demo server: 9/9 PASS. README.md §8 (Compliance + certification) updated to point at the harness as the suggested first-line conformance check. Tool vendors fork apps/secs_conformance.cpp and add their own capability-specific checks alongside. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
d470442a8c |
docs: drop implementation_plan.md, rewrite README for fab deployment
implementation_plan.md was a Layer-0..6 roadmap from the project's
spec-as-data exploration phase; every layer it described is now
shipped (Layer 0 foundations through Layer 4 message catalog +
state machines). Removed.
README rewritten for the fab-deployment audience. Sections added:
1. Persistence directory layout (storage rules, disk budget, DR)
2. Security (network isolation, TLS tunnels, audit logging,
config signing)
3. Monitoring + observability (signals → hooks table, Prometheus
pattern)
4. High availability (active/standby on shared persistence)
5. Deployment patterns (Docker / systemd / k8s)
6. Upgrade path (YAML reload, code rollout, schema versioning)
7. Integration with the fab stack (MES / AMHS / OHT / recipe
engine table)
8. Compliance + certification (fork COMPLIANCE.md per tool, run
RTS)
9. Testing in production (canary, synthetic transactions, shadow
traffic)
10. Operational runbook (incident → first check → mitigation)
Stale stats refreshed: test count went 148/794 → 384/2390;
catalog grew to 164 messages; HSMS-GS, SECS-I T3/T4, per-port E84,
E42 formatted PPs all mentioned.
COMPLIANCE.md §9 lost its stale `implementation_plan.md` reference.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
90c177b7ce |
E40 Process Jobs + E94 Control Jobs + E30 communication state
GEM300 layer: SEMI E40-0705 Process Job and E94-0705 Control Job state machines, plus the E30 §6.1 communication-state machine that sits between HSMS SELECT and full GEM communication. Data-driven via data/process_job_state.yaml and data/control_job_state.yaml, mirroring the existing control_state.yaml pattern. Wire coverage: S14F9/F10 CreateObject (CJ) host -> equipment S14F11/F12 DeleteObject (CJ) host -> equipment S16F5/F6 PRJobCommand host -> equipment S16F9 PRJobAlert equipment -> host S16F11/F12 PRJobCreate (simplified body) host -> equipment S16F13/F14 PRJobDequeue host -> equipment S16F27/F28 CJobCommand host -> equipment Process Job FSM exposes 8 states matching PRJOBSTATE bytes (E40 §10.3.2); HOQ is reorder-aware (move-to-head against an insertion-order vector); Stop/Abort on a Queued PJ routes through ABORTING so the host observes PRJOBSTATE=7 on the wire (§6.3); alert_enabled is settable per-PJ for PRALERT control; FSM dispatches through ProcessJobStore::on_change_ dynamically so a late set_state_change_handler() reaches existing PJs. Hardening: loader rejects NoState (sentinel) as initial/from/to and rejects `on: created` rows; static_asserts pin enum values to wire bytes; ProcessJobStore is non-movable to keep the per-PJ this-capture safe. Server simulator cascades the full CJ -> PJ lifecycle on CJSTART so the wire trace exercises every legal state. CEIDs 400/401 fire on CJ state changes via the existing event-report pipeline. Tests: 60+ new assertions across test_process_jobs, test_control_jobs, test_communication_state, test_hsms_connection, plus loader and messages round-trip coverage. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
1f67aad985 |
100%/F: S10F5/F6 multi-line + honest 100% in COMPLIANCE.md + README pass
tests / build-and-test (push) Failing after 33s
The final additions: S10F5/F6 multi-line terminal display (closes the last partial Additional capability — Equipment Terminal Services flips ✅), and a thorough COMPLIANCE.md / README pass that states the 100% claim honestly. Catalog + handlers data/messages.yaml S10F5 / S10F6 added. apps/secs_server.cpp router.on(10, 5) iterates the line list, acks with S10F6. tests/test_messages.cpp Round-trips a 3-line multi-line display. COMPLIANCE.md (rewritten) Every GEM Fundamental ✅. Every GEM Additional that E30 binds to a concrete message set ✅. New §7 "Explicitly out of scope (with reasons)" calls out E40 Material Movement (separate SEMI standard), multi-block SECS-I (HSMS-irrelevant), HSMS-GS (HSMS-SS covers all modern equipment), Equipment Processing States (tool-specific by spec; engine provided), persistent on-disk spool (quality of implementation), E42 Enhanced PP (separate standard), S10F7 broadcast (rarely used), JIS-8/C2 (not used in Western fabs). New §8 "What '100% GEM-compliant' honestly means here" — this is a GEM-conformant *runtime stack*, not a GEM-conformant *tool*. Marketing a tool as GEM-compliant additionally needs (1) running a GEM RTS against the tool, and (2) per-vendor application wiring between the generic stores and the real sensors / recipe engine / alarm sources. README.md (rewritten) Architecture diagram updated to reflect the actual store list (nine stores). "Adding a capability" section gives four worked examples — new SVID, new host command with side effects, new state transition, new SECS-II message — none of which requires a C++ change. Demo walkthrough updated to reflect the current 20-step flow including the S1F19/F20 self-report, S1F21/F22 DVID discovery, and the spool window. Code clarity include/secsgem/gem/data_model.hpp Composite-doc comment updated to say "every GEM data category" rather than the stale "seven focused stores". Verified - Tests: 84 cases / 487 assertions pass. - Demo: 198 server/host log lines; exits 0 end-to-end. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
raphael
|
b871cd9da2 |
Table/YAML-driven refactor (Layer 1 start)
Move equipment capabilities and the E30 control state machine out of C++
code and into YAML data files; introduce a Router for SECS dispatch;
consolidate small files.
Behavioural changes: none. Demo identical (15 SxFy transactions +
3 equipment-initiated primaries), 67 test cases / 384 assertions still
all green. Structural changes only.
Why
---
The previous server.cpp held the equipment data dictionary (3 SVIDs,
2 ECIDs, 3 CEIDs, 2 alarms, 2 recipes, 4 host commands) as imperative
C++ in a 50-line `populate()` function, and routed inbound messages
through a 150-line if-ladder. Adding a new SVID required a recompile.
Adding a new state transition required editing two switch statements
(`operator_*` and `on_host_request_*`). The control state machine's
behavioural rules were spread across imperative code in two methods.
This is exactly what implementation_plan.md calls out as the wrong
shape: behavioural rules should live in versioned data, and every
runtime/test/analyzer should read from that data rather than re-encode
it. This commit starts that move.
What's new
----------
data/equipment.yaml
Equipment data dictionary. Declarative SVIDs / ECIDs / CEIDs /
alarms / recipes / host commands. Host commands carry their HCACK
ack code plus optional `emit_ceid` and `set_alarm` side-effects.
Adding a new SVID or command is a YAML edit, no recompile.
data/control_state.yaml
The E30 §6.2 control state transition table as data. Each row is
(from, on) -> (to [, then] [, ack]). `then` chains an auto-advance
through the transient AttemptOnline state. The previous
imperative switch is gone.
include/secsgem/config/loader.hpp + src/config/loader.cpp
yaml-cpp-backed loader. `load_control_state(path)` returns a
ControlTransitionTable + initial state; `load_equipment(path, model)`
populates the EquipmentDataModel and returns the device descriptor
(id, MDLN, SOFTREV, optional auto-emit CEID). Surfaces config
errors with file path + field name via ConfigError.
include/secsgem/gem/router.hpp (header-only)
Small (stream, function) -> handler map. Server registers all
handlers once at startup, then the Connection's message handler is
just `router.dispatch(msg)`. Unhandled primaries with W set get
SxF0 by default. Replaces the if-ladder in secs_server.cpp.
include/secsgem/gem/control_state.hpp + .cpp
ControlTransitionTable is the new pure data type. ControlStateMachine
is now a thin engine over the table: `fire(event)` looks up the row,
optionally transitions, optionally chains a `then` transition, returns
the ack code. Behaviour rules no longer live in C++ switches.
The default in-code table matches data/control_state.yaml row for row;
tests rely on it so they don't need the YAML file.
include/secsgem/gem/data_model.hpp + .cpp
`register_command(rcmd, CommandSpec)` replaces the function-handler
signature. CommandSpec = (HostCmdAck, optional emit_ceid, optional
set_alarm). `dispatch_command` returns a CommandResult so the server
can fire the side-effects after S2F42 is sent.
apps/secs_server.cpp
No populate(), no if-ladder. Loads equipment.yaml + control_state.yaml
at startup (clean error on bad config), wires the Router once,
delegates dispatch. Sm change handler reads emit_on_control_change
from the YAML. Welcome S10F3 removed for parity with config (a future
YAML rule could re-introduce it declaratively).
tests/test_loader.cpp (new)
Verifies the YAML loader produces the same shape as the in-code
default table, and that equipment.yaml populates every section
(SVIDs/ECIDs/CEIDs/alarms/recipes/commands). SECSGEM_DATA_DIR
CMake define points at ${CMAKE_SOURCE_DIR}/data so tests don't
depend on cwd.
CMakeLists.txt, Dockerfile
find_package(yaml-cpp) and link. libyaml-cpp-dev added to the
Ubuntu base image (yaml-cpp 0.8 ships the modern target name).
File consolidation
------------------
Five small files removed; their content lives in fewer headers:
- secs2/item.cpp -> inline in secs2/item.hpp
- secs2/message.cpp -> inline in secs2/message.hpp
- hsms/types.hpp -> merged into hsms/header.hpp
- hsms/frame.hpp -> merged into hsms/header.hpp
- hsms/frame.cpp -> merged into hsms/header.cpp
hsms/header.hpp is now "the HSMS wire format" in one place: SType + status
enums + Timers + Header + Frame + constants. All includers updated.
Net effect
----------
Before: equipment data dictionary lived in 50 lines of imperative
populate() inside secs_server.cpp; dispatch in a 20-branch if-ladder.
After: equipment data dictionary lives in 47 lines of YAML; dispatch
is a Router built once. Adding a new capability is now a YAML edit
in the common case.
Test count up to 67 cases / 384 assertions (+4 cases / +106 assertions)
covering the loader and the new table-driven SM paths.
What's NOT changed
------------------
The per-SxFy reply construction still lives in C++ (each message has a
unique body shape). Moving those into YAML/JSON message-shape
definitions is the next refactor step but requires a generic typed
encoder/decoder driven by shape descriptors; out of scope here.
Spooling, the S9 error stream, S1F19/F20, and the other gaps in
COMPLIANCE.md remain unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
raphael
|
96b02f8b50 |
Initial commit: C++20 SECS-II / HSMS / GEM client + server
A fully containerised SECS/GEM toolchain. Single docker compose project,
no host build tools. 63 unit-test cases / 278 assertions, two demo
executables, end-to-end two-container demo exercising every implemented
capability.
Architecture (bottom-up):
secs2/ E5 SECS-II codec
Item variant over L/A/B/BOOLEAN/I1-8/U1-8/F4/F8
encode/decode big-endian, 1/2/3-byte length encoding
Message SxFy + W-bit + optional root item
to_sml human-readable text rendering
hsms/ E37 HSMS transport (TCP)
Header 10-byte header + SType enum (Data/Select/Deselect/
Linktest/Reject/Separate)
Frame 4-byte length prefix + payload encode/decode
Connection async Asio TCP, NOT-SELECTED -> SELECTED state machine,
T3/T5/T6/T7/T8 timers, system-bytes reply correlation,
graceful close-after-flush separation
endpoint active Client (connect with T5 retry) and passive Server
(accept loop) wrappers over Connection
gem/ E30 GEM logic
ControlStateMachine 5-state E30 control model with operator
actions, host requests, SEMI-mandated ack
codes (OnlineAck, OfflineAck, CommAck), and
a state-change handler
EquipmentDataModel in-memory dictionary: SVIDs, DVIDs, ECIDs
(with EAC), CEIDs, report defs, CEID->report
links, enabled-events set, alarm table
(ALCD, enabled, active), process programs,
host command registry, clock (16-char
YYYYMMDDhhmmsscc with offset)
messages.hpp builders + parsers for every SxFy below
GEM message coverage (full list):
S1F1/F2 Are You There / On Line Data
S1F3/F4 Selected Equipment Status Request / Data
S1F11/F12 Status Variable Namelist Request / Data
S1F13/F14 Establish Communications (+ CommAck)
S1F15/F16 Request OFFLINE (+ OfflineAck)
S1F17/F18 Request ONLINE (+ OnlineAck)
S2F13/F14 Equipment Constant Request / Data
S2F15/F16 EC Send + EquipmentAck (Accept/UnknownEcid/Busy/OutOfRange)
S2F17/F18 Date and Time Request / Data
S2F29/F30 Equipment Constant Namelist Request / Data
S2F31/F32 Date and Time Set Request / TimeAck
S2F33/F34 Define Report + DefineReportAck (5 enum values)
S2F35/F36 Link Event Report + LinkEventAck
S2F37/F38 Enable / Disable Event Report + EnableEventAck
S2F41/F42 Host Command + HostCmdAck (7 values) + per-param CPACKs
S5F1/F2 Alarm Report Send + AlarmAck (ALCD bit-7 set/cleared
+ lower-7 category)
S5F3/F4 Enable/Disable Alarm Send + AlarmAck
S5F5/F6 List Alarms Request / Data (active alarms tagged in ALCD)
S6F11/F12 Event Report Send (equipment-initiated CEID emission
with full report data) + EventReportAck
S7F3/F4 Process Program Send + ProcessProgramAck (7 values)
S7F5/F6 Process Program Request / Data
S7F19/F20 Current EPPD List Request / Data
S10F1/F2 Terminal Display Single (host->equipment) + TerminalAck
S10F3/F4 Terminal Display Single (equipment->host)
Demo apps:
apps/secs_server.cpp passive equipment. Populates the data model
with 3 SVIDs (ControlState, Clock,
EventsEnabled), 2 ECIDs, 3 CEIDs
(ControlStateChanged, AlarmSetEvent,
ProcessStarted), 2 alarms (Chiller Temp High
cat 4, Door Open cat 1), 2 recipes
(RECIPE-A, RECIPE-B), and 4 host commands
(START, STOP, PAUSE, FAULT). Emits S6F11 on
every control state transition + on START;
emits S5F1 + the AlarmSetEvent CEID on FAULT.
Pushes an S10F3 welcome message when the host
comes online.
apps/secs_client.cpp active host. Walks 17 steps: Establish ->
Online -> S1F11 SVID namelist -> S1F3 read ->
S2F29 EC namelist -> S2F13 read ->
S2F17 clock -> S2F33/S2F35/S2F37 dynamic
event subscription -> S2F41 START
(-> receives S6F11) -> S5F5 alarm list ->
S5F3 enable alarm 1 -> S2F41 FAULT
(-> receives S5F1 + S6F11) -> S7F19/S7F5
recipe list + body -> S10F1 terminal ->
S1F15 Offline -> Separate. Handles inbound
S6F11, S5F1, S10F3 primaries.
Testing:
tests/test_secs2.cpp codec round-trip for every format,
byte-layout assertions for known values,
truncation/trailing-byte rejection,
nested list round-trip, SML rendering
tests/test_hsms.cpp header byte layout, data + control
header round-trip, full frame round-
trip with length prefix, short-payload
rejection
tests/test_control_state.cpp every (state, event) pair in the E30
control state machine, including
AlreadyOnline / NotAccept rejections
and idempotent offline-while-offline
tests/test_data_model.cpp SVID/ECID/Alarm/Recipe CRUD, clock
format + parse, host command registry,
full event-report pipeline (define ->
link -> enable -> compose) with
every error path (InvalidVid,
UnknownCeid, UnknownRptid), alarm
set/clear with ALCD bit-7 semantics
tests/test_messages.cpp round-trip + byte-layout for every
builder/parser pair, including S6F11
event reports with mixed item types
Toolchain:
Dockerfile Ubuntu 24.04, g++-13, CMake, Ninja, libasio-dev
docker-compose.yml builder / tests / server / client services,
source bind-mounted, build artifacts in a
named volume so the host tree stays clean
CMakeLists.txt C++20, -Wall -Wextra -Wpedantic, standalone
Asio (ASIO_STANDALONE), doctest via FetchContent
Documentation:
README.md architecture, quick start, demo log
COMPLIANCE.md honest per-capability E5/E30/E37 audit with
spec section refs. Calls out what's implemented,
what's partial (Reject.req, Alarms missing F7/F8,
EC range validation, PP without verify, terminal
single-line only), and what's intentionally not
yet implemented (spooling, S9 error stream,
Documentation S1F19/F20+F21/F22, limits monitoring,
trace data collection, multi-block, material
movement). Does NOT claim "100% GEM-compliant" and
lists the work required to honestly make that claim.
This is Layer 0 + the start of Layer 1 from implementation_plan.md.
The transition-table-driven "spec-as-data" architecture (Layer 1
proper) is not yet implemented; the current code uses imperative
state machines that are structurally ready to be refactored onto
tables.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|