Status table brought current (format-aware daemon, secsgem-py interop), the
stale Layer-0 section replaced, and the path to an excellent GEM300 repo laid
out as ordered phases A–F: finish universal RPCs, the Subscribe command
stream (HCACK-4 design written down as the implementation contract), the
Python client package, GEM300 job/carrier in-the-loop, hardening/CI, and the
fab-acceptance track. Known-issues section records what the audit found
(GetControlState enum race + why the state-change-handler slot can't be
reused, missing alarm name key, pvd_tool predating set_handler, manual
interop harnesses, TSan gap).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extract the SECS/GEM engine wiring out of the secs_server app into a
reusable class, and stand up a language-agnostic gRPC daemon on top so a
tool's software (any language) can drive the equipment without linking C++
or knowing SEMI. Foundation for replacing a vendor's SECS/GEM server.
Engine reuse:
- EquipmentRuntime (include/secsgem/gem/runtime.hpp, src/gem/runtime.cpp):
owns io_context, passive Server, model, control-state machine, Router;
thread-safe outbound API (set_variable/emit_event/set_alarm/clear_alarm),
on_command hook, deliver_or_spool, run()/run_async()/poll()/stop().
- register_default_handlers (src/gem/default_handlers.cpp): the 56 GEM
handlers + domain emitters, relocated from secs_server so the app and the
daemon speak byte-identical GEM. secs_server.cpp reduced ~1270 -> 113 lines.
- name_index.hpp: resolve_variable(name) -> VID (the name->id binding layer).
Daemon (apps/secs_gemd.cpp, proto/secsgem/v1/equipment.proto):
- runs the engine + HSMS link on a background thread; serves the gRPC
Equipment service. Increment 1: SetVariables (name-resolved, plain
value->Item) and GetControlState. proto carries the full v1 surface
(universal + carrier/recipe/job tiers); remaining RPCs + the Subscribe
command stream are next (docs/DAEMON_ROADMAP.md).
- CMake: opt-in SECSGEM_DAEMON, protoc/grpc_cpp_plugin codegen, gracefully
skipped where protobuf/grpc++ are absent. Dockerfile gains the grpc deps.
Tests (proof): test_runtime, test_default_handlers (S1F1->S1F2, S2F41->hook),
test_name_index. Full suite 458/458, 2795 assertions; live server<->client
GEM300 demo still passes on the refactored server.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An audit of doc code blocks against the real headers found APIs that do
not exist in the codebase, presented as authoritative walkthroughs:
- ch35 (dispatch): an entirely fabricated callback architecture —
HostCommandRegistry::set_emit_ceid_handler, CommandOutcome, emit_ceids.
Rewritten to the real Spec/Result/dispatch + the new set_handler hook.
- ch13 (E30): wrong store names — EventStore/ReportStore -> EventReportSubscriptions,
SvidStore -> StatusVariableStore, AlarmStore/AlarmDispatcher -> AlarmRegistry,
ClockStore -> Clock, TerminalServiceStore -> (no store), in both the
capability tables and the worked S2F33 example.
- ch17 (E116): EptStore/seconds/bucket_ -> EptStateMachine/milliseconds/buckets_.
- ch51 (extending): stale host-command handler -> the real set_handler signature.
Verified clean by grep: no fabricated symbols remain in docs/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tone pass across the non-tutorial markdown — README, PROOFS,
ARCHITECTURE, BENCHMARKS, COMPLIANCE, FAQ, MES_INTEROP, SECURITY,
and interop/README. Three patterns came out:
- Bug-history war stories ("Past interop sweeps surfaced…",
"What these harnesses caught: 1. Strict U-width parsing…").
- Chat-with-reader framing ("Don't skip TLS unless…", "Treat as a
punch list", "If you're running in a pod…", "Misconfiguration
incidents drop dramatically").
- Self-referential narration ("we ship", "our codec", "the
codebase's most-tested layer", "three orders of magnitude above
fab load", "the gift that keeps giving").
README also drops the standalone ThreadSanitizer subsection under
Build details (now a single line under the new Testing section).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously written as a forward-looking plan ("Plan: (1) KAT → (2)
tshark → (3) secs4j → (4) libFuzzer", "Effort: ~3 hours", "Survey
step (do this first)"). All four validators have shipped —
test_e5_kat.cpp, interop/secs4j/Secs4jHostHarness.java,
interop/tshark_validate.sh, apps/fuzz_*.cpp. Rewritten as
documentation of what's there: file paths, CI job names, actual
result numbers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
generate_messages.py → gen_messages.py and several gem/ headers moved
under gem/store/ (carrier_store.hpp → store/carriers.hpp, etc.);
e84.hpp split into e84_state.hpp. The guided-tour chapters still
pointed at the old paths — relink them so the deep-link footnotes
resolve.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Last four chapters of the guided tour.
40 — Building, running, the demo. Docker prerequisites, the build
flow, what each binary is for, running the 24-transaction demo
flow annotated step by step. Running the 4 external-validator
sweeps + the libFuzzer pass. Inspecting the demo with tcpdump and
tshark. Reading source while running as the recommended learning
workflow.
41 — Integration: hardware, MES, production. Four-phase tour:
wiring sensors / recipe engine / alarms / E84 GPIO; talking to a
real MES with the day-1 punch list + commercial-MES quirks (Wonderware
S2F21, Camstar Linktest cadence, etc.); production hardening
(nftables / stunnel / minisign / persistence layout / monitoring /
runbook); performance envelope + memory footprint + capacity
planning. Pointers to the long-form INTEGRATION.md / MES_INTEROP.md /
SECURITY.md / BENCHMARKS.md.
50 — API + message catalog + YAML schemas reference. Namespace-by-
namespace table of public symbols (secs2, hsms, secsi, gem, config,
metrics) with brief descriptions. Stream-by-stream message catalog
reference (S1, S2, S3, S5, S6, S7, S9, S10, S12, S14, S16). YAML
schema reference for messages.yaml + the three state-table files +
equipment.yaml.
51 — Extending the codebase. Seven recipes ordered from no-code to
substantial: new SVID/DVID/ECID (YAML only), new CEID with reports
(YAML only), new host command (YAML + optional handler), new control-
state transition (YAML only), new SECS-II message (YAML + handler),
new store (header + tests), new persistence backend (drop-in vs
pluggable trade-off). Each recipe has the actual mechanical steps,
the test pattern, and pointers to the chapter that explains why it
works.
Index updated to mark all 24 chapters published.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Seven chapters walking the implementation top-to-bottom.
30 — Repository tour. Top-level layout, directory by directory.
The eight built binaries. The dependency graph from TCP socket
up through EquipmentDataModel. CMake's role. Test layout.
31 — Spec-as-data and codegen. Why the design choice fits SECS/
GEM specifically. The five YAML files: messages catalog,
control/PJ/CJ transition tables, equipment dictionary. How
tools/gen_messages.py turns messages.yaml into typed C++ at build
time. The --validate-config multi-error validator. How to add a
new SVID / CEID / host command / state / message without C++.
32 — Stores and the data model. What a store IS (records + API +
change handler + optional persistence). Every store in the
codebase mapped to the SEMI standard it serves (table of 21).
EquipmentDataModel as plain composition + cross-store convenience
methods (vid_value, compose_reports_for). The no-locks single-
threaded contract. How to add a new store.
33 — Transport. hsms::Connection read path (length+payload async
chain), write path (queue + one outstanding write), timer model
(5 steady_timers + per-request T3). The asio executor / strand
model and why it's the right shape. secsi::Protocol as the IO-
free FSM with Action / Event variants; secsi::TcpTransport as the
asio adapter. Pattern repeats for E84 + GEM comm-state.
34 — Codec and SML. The four files (170 + 30 + 52 + 32 lines of
header, 229 + 220 lines of impl). Item variant storage layout
(11 alternatives, 16 formats, shared storage where E5 permits).
encode_into recursion; decode_at with bounds checks throwing
CodecError. Message wrapper. SML printer + try_parse_sml +
why SML round-trips Items but not necessarily bytes.
35 — State machines and dispatch. gem::Router as a typed
(stream, function) dispatch table. How an S2F41 round-trip walks
through parser → store dispatch → side-effect → CEID emission →
S6F11 build → spool-aware deliver. The 11 FSMs all sharing the
same three-property shape (pure data table + pure FSM + observer
pattern). CEID cascading from FSM transitions to wire bytes.
36 — Persistence, validation, metrics. Which 7 stores have file
journals + why the others don't. Per-record file pattern (atomic
rename, partial-write safe). Schema versioning + multi-version
read. Multi-error YAML validator (--validate-config) + cross-file
reference checks. Prometheus registry + HTTP exporter + worked
metric patterns from the PVD example.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Six more chapters finishing Part 2. Together with chapters 10–13
they document every SEMI standard this codebase implements.
14 — E40 + E94: process jobs (8-state lifecycle, S16F11/F5/F7/F9
on the wire) and control jobs (CJ wraps PJs with batch policy,
S14F9/S16F27 messages). Worked cascade showing how CJSTART
propagates through the PJ FSM and triggers S6F11 CEIDs at each
transition.
15 — E87 carriers: three orthogonal sub-machines (CarrierID,
SlotMap, CarrierAccess) per carrier and three more (Transfer,
Reservation, Association) per load port. S3F17 CarrierAction
strings + CAACK codes, S3F19 SlotMap verify, the 5-state slot
encoding, multi-port concurrency.
16 — E90 + E157: substrate tracking via three orthogonal axes
(STS / SPS / SubstrateIDStatus) and module process tracking
(NotExecuting / GeneralExecuting / StepExecuting / StepCompleted).
End-to-end PVD example showing E40 + E157 + E90 transitions
cascading into CEIDs.
17 — E116 + E120 + E39: equipment performance time-buckets across
six states, common equipment model object hierarchy, S14F1/F3
GetAttr/SetAttr as the uniform wire access for any object type
across multiple standards.
18 — E84 parallel I/O: ten signal lines, the 9-state handshake
FSM, the three TA1/TA2/TA3 timing-critical timers, why a physical
handshake gets modeled in software (testability, timer enforcement,
CEID emission, multi-port concurrency), the pure-FSM + asio-adapter
split.
19 — E42 + E148 + S5F9–F18: formatted recipes (S7F23/F25 typed
PPBODY), time synchronization with 16-char + 14-char accepted on
set, exception recovery as a persistent multi-step host-supervised
FSM (Posted → Recovering → Cleared with abort/retry). Revisits
the auto-S9 family and contrasts S9 (transport) vs S5F9
(application).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three more chapters of Part 2:
11 — E37 HSMS. 4-byte length prefix + 10-byte header (R-bit + session
id + W-bit + stream + function + PType + SType + system_bytes), the
9 SType control messages, the NOT-SELECTED → SELECTED state machine,
T3/T5/T6/T7/T8 with what each one bounds, the auto-S9 paths
(S9F1/F3/F5/F7/F9/F11), HSMS-SS vs HSMS-GS, the asio
single-threaded contract.
12 — E4 SECS-I. Half-duplex line turnaround (ENQ/EOT/ACK/NAK), the
10-byte block header bit-packing (R-bit / W-bit / E-bit / system
bytes), the 244-byte block cap and multi-block split/assemble, the
event-driven IO-free FSM with its Action / Event variants, T1/T2/T3/T4
with semantics + defaults, master/slave contention. Notes the
deferred asio serial_port adapter; explains why this chapter
matters even for HSMS-only readers.
13 — E30 GEM. Disambiguates the three state machines (HSMS transport
vs GEM communication vs GEM control), walks the comm-state FSM
(DISABLED → WAIT-CRA → COMMUNICATING with T_CRA / T_DELAY) and the
control-state FSM (5 states + the YAML transition table). Lists
every Fundamental and Additional capability with its messages, code
locations, and store assignments. One worked Event-Notification
scenario tracing seven on-wire steps to their EquipmentDataModel
internals.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Opens Part 2 (the standards in detail). Walks the entire SECS-II
encoding from first principles: the mental model (every value is one
Item; a List is a recursive Item), the format-byte arithmetic
(6-bit format code, 2-bit length-byte-count), the 14 format codes,
length bytes 1/2/3 (with the 16 MiB cap), big-endian everywhere,
the difference between byte-count (scalars) and child-count (lists).
Then walks every format with worked hexdumps: empty list, nested
list, ASCII with length-byte boundary crossing, Binary vs Boolean,
U1/U2/U4/U8, signed integers with two's-complement edges, F4 / F8
with NaN / ±Inf / −0.0, JIS-8, C2 Unicode.
Then the codebase mapping: Format enum, Item variant storage layout,
encode_into / decode_at recursion, SML printer/parser, the
identifier-wildcard rule (SEMI allows U1/U2/U4/U8 interchangeably
for ID fields) with the messages_helpers::any_unsigned_first<Out>
helper that closes the leniency contract.
Closes with the well-defined CodecError conditions, what the codec
deliberately doesn't reject (unknown format codes), and pointers to
chapter 31 (codegen) and chapter 11 (HSMS) as the next dependencies
above the codec.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Picks up the file renames that landed alongside the previous commit
and fixes everything that pointed at the old root locations:
- README.md doc-map updated: every entry now points at docs/X.md,
with a new "docs/" lead entry pointing at the guided-tour index.
- README inline cross-refs (ARCHITECTURE / INTEGRATION / SECURITY /
BENCHMARKS / MES_INTEROP / PROOFS) repointed to docs/.
- README "Interop" section rewritten — used to mention only
secsgem-py; now covers all four external validators (secsgem-py
31 / secs4java8 55 / tshark 69 frames / libFuzzer 200 k+ runs)
with a one-line summary each, plus pointers to interop/README.md
and docs/VERIFICATION.md.
- README "Deferred follow-ups" cleaned: dropped the explanatory
"Listed here so reviewers don't go looking for them in
COMPLIANCE.md and find an 'out of scope' entry that sounds
defensive" sentence — the section header speaks for itself.
- docs/00_index.md "Where the rest of the docs live" table: dropped
every `../` prefix since the docs are now siblings.
- docs/01_what_is_secs_gem.md PROOFS reference updated to sibling.
- docs/02_the_cast.md INTEGRATION + MES_INTEROP refs updated to
siblings; dropped the stale "at the repo root" wording.
- interop/README.md: VERIFICATION + PROOFS refs updated to
../docs/X.md; stale "~24 + 4 checks" updated to 31 (matches
PROOFS.md and README).
- examples/pvd_tool/README.md: every doc cross-ref now points at
../../docs/X.md.
- Source / data / CI comments mentioning doc names (e.g.
"INTEGRATION.md §3", "COMPLIANCE.md gap") rewritten to
"docs/INTEGRATION.md §3" etc. — affects 9 files across
include/, apps/, tests/, data/, examples/, .gitea/workflows/.
Verified: full build under docker passes, 445/445 test cases pass,
2 753/2 753 assertions pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
02 — The cast of characters: equipment, EAP, MES, fab planner, AMHS,
operator. Who initiates which conversation, why the equipment is
the passive side of HSMS by convention, how the AMHS handshake is
out-of-band relative to SECS. Cross-references the relevant
namespace and test files for each actor.
03 — Vocabulary + a wafer's journey: follows one 300 mm wafer
end-to-end through a fab and labels every SECS message and acronym
that fires. Introduces SVID / DVID / ECID / CEID / RPTID / ALID /
PPID / MDLN / SOFTREV / HCACK / ALCD / OFLACK / CAACK / SMACK / etc.
in context rather than as a list. Includes one-screen reference
tables for the remaining acknowledge codes, T-timers in all four
contexts (HSMS / SECS-I / E84 / E30 communication state), and a
stream-by-stream summary.
Part 1 (Foundations) of the guided tour is now complete — a reader
who reads chapters 01–03 can describe the protocol stack, identify
the actors, and recognise every acronym they'll meet in Part 2.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
A linear teach-from-zero tutorial that walks both SECS/GEM as a
protocol family and this codebase as an implementation. Each
chapter explains a SEMI concept and shows where it lives in code,
so a reader builds a mental model of the standards and the
repository simultaneously.
Structure (24 chapters across 5 parts):
- Part 1 (3 ch) — Foundations: what SECS/GEM is, the cast of
characters, vocabulary + a wafer's end-to-end journey
- Part 2 (10 ch) — Standards in detail: E5, E37, E4, E30,
E40+E94, E87, E90+E157, E116+E120+E39, E84, E42+E148+S9
- Part 3 (7 ch) — Codebase: repository tour, spec-as-data + codegen,
stores, transport, codec, state machines, persistence
- Part 4 (2 ch) — Operations: build/run/demo, integration
- Part 5 (2 ch) — Reference: API + messages + YAML, extension guide
Published in this commit:
- 00_index.md — guide layout, audience map, reading paths,
conventions, status table
- 01_what_is_secs_gem.md — the N×M integration problem, what SECS
vs. HSMS vs. GEM each actually refer to, the GEM 300 suite, the
transport→message→behaviour layering, where each layer lives in
this codebase, an end-to-end S2F17/F18 example
Chapters publish iteratively from here.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
raphael