docs: chapters 30–36 — the codebase (Part 3 complete)
Seven chapters walking the implementation top-to-bottom. 30 — Repository tour. Top-level layout, directory by directory. The eight built binaries. The dependency graph from TCP socket up through EquipmentDataModel. CMake's role. Test layout. 31 — Spec-as-data and codegen. Why the design choice fits SECS/ GEM specifically. The five YAML files: messages catalog, control/PJ/CJ transition tables, equipment dictionary. How tools/gen_messages.py turns messages.yaml into typed C++ at build time. The --validate-config multi-error validator. How to add a new SVID / CEID / host command / state / message without C++. 32 — Stores and the data model. What a store IS (records + API + change handler + optional persistence). Every store in the codebase mapped to the SEMI standard it serves (table of 21). EquipmentDataModel as plain composition + cross-store convenience methods (vid_value, compose_reports_for). The no-locks single- threaded contract. How to add a new store. 33 — Transport. hsms::Connection read path (length+payload async chain), write path (queue + one outstanding write), timer model (5 steady_timers + per-request T3). The asio executor / strand model and why it's the right shape. secsi::Protocol as the IO- free FSM with Action / Event variants; secsi::TcpTransport as the asio adapter. Pattern repeats for E84 + GEM comm-state. 34 — Codec and SML. The four files (170 + 30 + 52 + 32 lines of header, 229 + 220 lines of impl). Item variant storage layout (11 alternatives, 16 formats, shared storage where E5 permits). encode_into recursion; decode_at with bounds checks throwing CodecError. Message wrapper. SML printer + try_parse_sml + why SML round-trips Items but not necessarily bytes. 35 — State machines and dispatch. gem::Router as a typed (stream, function) dispatch table. How an S2F41 round-trip walks through parser → store dispatch → side-effect → CEID emission → S6F11 build → spool-aware deliver. The 11 FSMs all sharing the same three-property shape (pure data table + pure FSM + observer pattern). CEID cascading from FSM transitions to wire bytes. 36 — Persistence, validation, metrics. Which 7 stores have file journals + why the others don't. Per-record file pattern (atomic rename, partial-write safe). Schema versioning + multi-version read. Multi-error YAML validator (--validate-config) + cross-file reference checks. Prometheus registry + HTTP exporter + worked metric patterns from the PVD example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,231 @@
|
||||
# 30 — Repository tour
|
||||
|
||||
← [19 E42 + E148 + S9 — Misc](19_e42_e148_s9_misc.md) | [Back to index](00_index.md) | Next: [31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md) →
|
||||
|
||||
You've seen what every SEMI standard *does*. Now we shift to how
|
||||
this **codebase** is laid out. This chapter answers: when you
|
||||
`git clone` this repo, what are you looking at?
|
||||
|
||||
The repo is small — about 15 k lines of C++ + tests + tooling. It
|
||||
fits in your head with a little patience. By the end of this
|
||||
chapter you'll know:
|
||||
|
||||
- What each top-level directory contains.
|
||||
- Which binaries get built.
|
||||
- The dependency graph between modules.
|
||||
- How the build system finds and links them.
|
||||
|
||||
---
|
||||
|
||||
## Top-level layout
|
||||
|
||||
```
|
||||
secs-gem/
|
||||
├── README.md One-page project summary.
|
||||
├── LICENSE Proprietary terms.
|
||||
├── CMakeLists.txt Build config (CMake 3.16+, single file).
|
||||
├── Dockerfile Ubuntu 24.04 + g++-13 + libasio + yaml-cpp.
|
||||
├── docker-compose.yml Multi-container demo wiring.
|
||||
├── .gitea/workflows/ci.yml CI pipeline.
|
||||
│
|
||||
├── include/secsgem/ Public headers. All API here.
|
||||
│ ├── secs2/ E5 codec + SML.
|
||||
│ ├── hsms/ E37 transport (TCP + framing).
|
||||
│ ├── secsi/ E4 transport (FSM + TCP tunnel).
|
||||
│ ├── config/ YAML loader + multi-error validator.
|
||||
│ ├── metrics/ Prometheus exporter.
|
||||
│ ├── endpoint.hpp asio::ip::tcp::endpoint factory.
|
||||
│ └── gem/ E30 + every GEM 300 standard.
|
||||
│ ├── store/ Per-domain bundles (SVIDs, alarms, …).
|
||||
│ └── *.hpp State machines + composers.
|
||||
│
|
||||
├── src/ Implementations. Mirrors include/.
|
||||
│ ├── secs2/{codec,sml}.cpp
|
||||
│ ├── hsms/{header,connection}.cpp
|
||||
│ ├── secsi/{header,block,protocol,tcp_transport}.cpp
|
||||
│ ├── config/...
|
||||
│ ├── gem/...
|
||||
│ └── endpoint.cpp
|
||||
│
|
||||
├── apps/ Standalone binaries.
|
||||
│ ├── secs_server.cpp Passive equipment (demo + integration target).
|
||||
│ ├── secs_client.cpp Active host driving the demo flow.
|
||||
│ ├── secs_conformance.cpp 47-check wire-level conformance harness.
|
||||
│ ├── secs_interop_probe.cpp Probe against secsgem-py passive equip.
|
||||
│ ├── secs_bench.cpp Throughput / latency / memory bench.
|
||||
│ ├── fuzz_secs2_decode.cpp libFuzzer harness for secs2::decode.
|
||||
│ └── fuzz_sml_parse.cpp libFuzzer harness for try_parse_sml.
|
||||
│
|
||||
├── tests/ doctest unit + integration tests.
|
||||
│ └── test_*.cpp 50 files, 445 cases, 2753 assertions.
|
||||
│
|
||||
├── data/ YAML configs (the spec-as-data).
|
||||
│ ├── messages.yaml SECS-II message catalog (164 msgs).
|
||||
│ ├── control_state.yaml E30 §6.2 transition table.
|
||||
│ ├── process_job_state.yaml E40 transition table.
|
||||
│ ├── control_job_state.yaml E94 transition table.
|
||||
│ └── equipment.yaml Demo SVIDs/ECIDs/CEIDs/alarms/recipes.
|
||||
│
|
||||
├── tools/ Build-time scripts.
|
||||
│ └── generate_messages.py Codegen: messages.yaml → messages.hpp.
|
||||
│
|
||||
├── interop/ External-validator harnesses.
|
||||
│ ├── README.md Harness-by-harness detail.
|
||||
│ ├── host_vs_cpp_server.py secsgem-py active host driving us.
|
||||
│ ├── passive_equipment.py secsgem-py passive equipment for us to drive.
|
||||
│ ├── raw_gem300_harness.py Raw S3/S14/S16/S12 round-trip.
|
||||
│ ├── tshark_validate.sh pcap + tshark HSMS dissector check.
|
||||
│ ├── secs4j_validate.sh secs4java8 (Java) cross-validation.
|
||||
│ └── secs4j/ Dockerfile + harness for secs4java8.
|
||||
│
|
||||
├── examples/
|
||||
│ └── pvd_tool/ Worked vendor example: fictional PVD tool.
|
||||
│ ├── README.md What the example shows.
|
||||
│ ├── equipment.yaml Realistic SVIDs/ECIDs/CEIDs/alarms/recipes.
|
||||
│ └── main.cpp Sensor sim, recipe runner, alarm monitor.
|
||||
│
|
||||
└── docs/ This guide + reference docs.
|
||||
├── 00_index.md The series TOC.
|
||||
├── 01–51_*.md Tutorial chapters.
|
||||
├── ARCHITECTURE.md One-page architecture overview.
|
||||
├── COMPLIANCE.md Per-capability audit.
|
||||
├── INTEGRATION.md Vendor-side production deploy.
|
||||
├── PROOFS.md 8 commands proving feature-completeness.
|
||||
├── VERIFICATION.md External-validator test plan.
|
||||
├── BENCHMARKS.md Performance envelope.
|
||||
├── MES_INTEROP.md Commercial-MES day-1 punch list.
|
||||
├── SECURITY.md nftables / stunnel / minisign configs.
|
||||
├── GLOSSARY.md SEMI vocabulary cheat sheet.
|
||||
└── FAQ.md Canonical answers.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The dependency graph
|
||||
|
||||
```
|
||||
data/*.yaml
|
||||
│
|
||||
┌─────────────┼──────────────────┐
|
||||
│ (codegen) │ (runtime load) │
|
||||
▼ ▼ ▼
|
||||
generated/messages.hpp config::loader
|
||||
│ │
|
||||
└──────────► gem::EquipmentDataModel
|
||||
│
|
||||
│ used by
|
||||
▼
|
||||
gem::Router
|
||||
│
|
||||
│ wraps
|
||||
▼
|
||||
secs2::Message ◄─── codec / SML
|
||||
│
|
||||
│ over
|
||||
▼
|
||||
hsms::Connection / secsi::TcpTransport
|
||||
│
|
||||
▼
|
||||
TCP socket
|
||||
```
|
||||
|
||||
Read it bottom-up: a TCP socket carries bytes; `hsms::Connection`
|
||||
frames them into `secs2::Message`s; `gem::Router` dispatches by
|
||||
`(stream, function)` to handlers; handlers read/write
|
||||
`EquipmentDataModel`; the model composes per-domain stores; the
|
||||
stores were built from the YAML at startup.
|
||||
|
||||
No layer ever calls *up* the graph. `secs2::Item` has no idea
|
||||
HSMS exists. `hsms::Connection` doesn't know about CEIDs.
|
||||
`gem::Router` doesn't know whether the bytes came over HSMS or
|
||||
SECS-I. Strict layering is what keeps the codebase small.
|
||||
|
||||
---
|
||||
|
||||
## The binaries
|
||||
|
||||
Built by [`CMakeLists.txt`](../CMakeLists.txt) (one file, ~250
|
||||
lines). Each binary lives in `build/` after `cmake --build`.
|
||||
|
||||
| Binary | Source | What it does |
|
||||
|----------------------|-----------------------------------------------------------------|-------------------------------------------------------|
|
||||
| `secs_server` | [`apps/secs_server.cpp`](../apps/secs_server.cpp) | Passive equipment. Listens on TCP, dispatches via Router. |
|
||||
| `secs_client` | [`apps/secs_client.cpp`](../apps/secs_client.cpp) | Active host. Drives ~24 transactions in the demo. |
|
||||
| `secs_conformance` | [`apps/secs_conformance.cpp`](../apps/secs_conformance.cpp) | 47 wire-level conformance checks against a live server. |
|
||||
| `secs_interop_probe` | [`apps/secs_interop_probe.cpp`](../apps/secs_interop_probe.cpp) | Active host probing a secsgem-py passive equipment. |
|
||||
| `secs_bench` | [`apps/secs_bench.cpp`](../apps/secs_bench.cpp) | Throughput / latency / memory harness. |
|
||||
| `secsgem_tests` | All `tests/*.cpp` | The 445-case doctest binary. |
|
||||
| `fuzz_secs2_decode` | [`apps/fuzz_secs2_decode.cpp`](../apps/fuzz_secs2_decode.cpp) | libFuzzer (clang only, opt-in `-DSECSGEM_FUZZ=ON`). |
|
||||
| `fuzz_sml_parse` | [`apps/fuzz_sml_parse.cpp`](../apps/fuzz_sml_parse.cpp) | libFuzzer for the SML parser. |
|
||||
|
||||
A worked example binary `pvd_tool` (from `examples/pvd_tool/`) is
|
||||
also built by the same `CMakeLists.txt` when the example is
|
||||
included.
|
||||
|
||||
---
|
||||
|
||||
## How the build system finds everything
|
||||
|
||||
`CMakeLists.txt` does five things in order:
|
||||
|
||||
1. **Pull in dependencies** — `find_package(Threads)`,
|
||||
`find_package(yaml-cpp)`, `FetchContent` for doctest. Standalone
|
||||
Asio is header-only (no link step).
|
||||
2. **Run codegen** — invokes `tools/generate_messages.py` to turn
|
||||
`data/messages.yaml` into `build/generated/secsgem/gem/messages.hpp`.
|
||||
Listed as a custom command so it re-runs when `messages.yaml`
|
||||
changes.
|
||||
3. **Build the library** — `add_library(secsgem ...)` with every
|
||||
source under `src/` plus the generated header.
|
||||
4. **Build the apps** — one `add_executable` per `apps/*.cpp`,
|
||||
each linking against `secsgem`.
|
||||
5. **Build the tests** — `add_executable(secsgem_tests ...)` with
|
||||
every `tests/*.cpp`, linked against doctest + `secsgem`.
|
||||
|
||||
Build flags:
|
||||
|
||||
- **`-DSECSGEM_TSAN=ON`** — adds `-fsanitize=thread` to a
|
||||
separate build dir. CI runs this lane.
|
||||
- **`-DSECSGEM_FUZZ=ON`** — requires clang; adds libFuzzer + ASan +
|
||||
UBSan; builds the two fuzz harnesses.
|
||||
|
||||
Everything else (Release / Debug, parallelism, output dirs) is
|
||||
standard CMake.
|
||||
|
||||
---
|
||||
|
||||
## Test layout
|
||||
|
||||
50 test files; 445 test cases; 2 753 assertions. One file per
|
||||
concern. Naming is `test_<thing>.cpp` consistently:
|
||||
|
||||
- `test_secs2.cpp`, `test_e5_kat.cpp`, `test_sml.cpp`,
|
||||
`test_messages.cpp` — codec.
|
||||
- `test_hsms*.cpp` (5 files), `test_secsi*.cpp` (3 files) — transport.
|
||||
- `test_control_state.cpp`, `test_communication_state.cpp`,
|
||||
`test_data_model.cpp`, `test_host_handler.cpp`, `test_loader.cpp`,
|
||||
`test_config_validate.cpp` — E30.
|
||||
- `test_process_jobs.cpp`, `test_control_jobs.cpp`,
|
||||
`test_carriers.cpp`, `test_substrates.cpp`, `test_ept.cpp`,
|
||||
`test_modules.cpp`, `test_cem_objects.cpp`, `test_e84*.cpp`,
|
||||
`test_e42_formatted_pp.cpp` — GEM 300.
|
||||
- `test_*_persistence.cpp` (4) — file-backed journal.
|
||||
- `test_robustness_fuzz.cpp` — randomized property test.
|
||||
- `test_thread_safety.cpp` — TSan-validated single-threaded contract.
|
||||
- `test_metrics_prometheus.cpp` — Prometheus exporter.
|
||||
- `test_wire_ceid_emission.cpp` — CEID firings observed on a real socket.
|
||||
- `test_live_gem300.cpp`, `test_gem300_scenario.cpp` — multi-FSM cascades.
|
||||
|
||||
Full per-standard breakdown:
|
||||
[`docs/PROOFS.md`](PROOFS.md) "Per-standard test coverage" table.
|
||||
|
||||
---
|
||||
|
||||
## Where to go next
|
||||
|
||||
Now that you know what's where, the next chapter explains the
|
||||
*philosophy* that makes the codebase this small: the **spec-as-data**
|
||||
principle, and how the YAML files + codegen + runtime loader work
|
||||
together so adding a new SVID / state / message rarely requires C++.
|
||||
|
||||
Next: [→ 31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md)
|
||||
Reference in New Issue
Block a user