raphael
cae98d9a7d
Seven chapters walking the implementation top-to-bottom. 30 — Repository tour. Top-level layout, directory by directory. The eight built binaries. The dependency graph from TCP socket up through EquipmentDataModel. CMake's role. Test layout. 31 — Spec-as-data and codegen. Why the design choice fits SECS/ GEM specifically. The five YAML files: messages catalog, control/PJ/CJ transition tables, equipment dictionary. How tools/gen_messages.py turns messages.yaml into typed C++ at build time. The --validate-config multi-error validator. How to add a new SVID / CEID / host command / state / message without C++. 32 — Stores and the data model. What a store IS (records + API + change handler + optional persistence). Every store in the codebase mapped to the SEMI standard it serves (table of 21). EquipmentDataModel as plain composition + cross-store convenience methods (vid_value, compose_reports_for). The no-locks single- threaded contract. How to add a new store. 33 — Transport. hsms::Connection read path (length+payload async chain), write path (queue + one outstanding write), timer model (5 steady_timers + per-request T3). The asio executor / strand model and why it's the right shape. secsi::Protocol as the IO- free FSM with Action / Event variants; secsi::TcpTransport as the asio adapter. Pattern repeats for E84 + GEM comm-state. 34 — Codec and SML. The four files (170 + 30 + 52 + 32 lines of header, 229 + 220 lines of impl). Item variant storage layout (11 alternatives, 16 formats, shared storage where E5 permits). encode_into recursion; decode_at with bounds checks throwing CodecError. Message wrapper. SML printer + try_parse_sml + why SML round-trips Items but not necessarily bytes. 35 — State machines and dispatch. gem::Router as a typed (stream, function) dispatch table. How an S2F41 round-trip walks through parser → store dispatch → side-effect → CEID emission → S6F11 build → spool-aware deliver. The 11 FSMs all sharing the same three-property shape (pure data table + pure FSM + observer pattern). CEID cascading from FSM transitions to wire bytes. 36 — Persistence, validation, metrics. Which 7 stores have file journals + why the others don't. Per-record file pattern (atomic rename, partial-write safe). Schema versioning + multi-version read. Multi-error YAML validator (--validate-config) + cross-file reference checks. Prometheus registry + HTTP exporter + worked metric patterns from the PVD example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
232 lines
11 KiB
Markdown
232 lines
11 KiB
Markdown
# 30 — Repository tour
|
||
|
||
← [19 E42 + E148 + S9 — Misc](19_e42_e148_s9_misc.md) | [Back to index](00_index.md) | Next: [31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md) →
|
||
|
||
You've seen what every SEMI standard *does*. Now we shift to how
|
||
this **codebase** is laid out. This chapter answers: when you
|
||
`git clone` this repo, what are you looking at?
|
||
|
||
The repo is small — about 15 k lines of C++ + tests + tooling. It
|
||
fits in your head with a little patience. By the end of this
|
||
chapter you'll know:
|
||
|
||
- What each top-level directory contains.
|
||
- Which binaries get built.
|
||
- The dependency graph between modules.
|
||
- How the build system finds and links them.
|
||
|
||
---
|
||
|
||
## Top-level layout
|
||
|
||
```
|
||
secs-gem/
|
||
├── README.md One-page project summary.
|
||
├── LICENSE Proprietary terms.
|
||
├── CMakeLists.txt Build config (CMake 3.16+, single file).
|
||
├── Dockerfile Ubuntu 24.04 + g++-13 + libasio + yaml-cpp.
|
||
├── docker-compose.yml Multi-container demo wiring.
|
||
├── .gitea/workflows/ci.yml CI pipeline.
|
||
│
|
||
├── include/secsgem/ Public headers. All API here.
|
||
│ ├── secs2/ E5 codec + SML.
|
||
│ ├── hsms/ E37 transport (TCP + framing).
|
||
│ ├── secsi/ E4 transport (FSM + TCP tunnel).
|
||
│ ├── config/ YAML loader + multi-error validator.
|
||
│ ├── metrics/ Prometheus exporter.
|
||
│ ├── endpoint.hpp asio::ip::tcp::endpoint factory.
|
||
│ └── gem/ E30 + every GEM 300 standard.
|
||
│ ├── store/ Per-domain bundles (SVIDs, alarms, …).
|
||
│ └── *.hpp State machines + composers.
|
||
│
|
||
├── src/ Implementations. Mirrors include/.
|
||
│ ├── secs2/{codec,sml}.cpp
|
||
│ ├── hsms/{header,connection}.cpp
|
||
│ ├── secsi/{header,block,protocol,tcp_transport}.cpp
|
||
│ ├── config/...
|
||
│ ├── gem/...
|
||
│ └── endpoint.cpp
|
||
│
|
||
├── apps/ Standalone binaries.
|
||
│ ├── secs_server.cpp Passive equipment (demo + integration target).
|
||
│ ├── secs_client.cpp Active host driving the demo flow.
|
||
│ ├── secs_conformance.cpp 47-check wire-level conformance harness.
|
||
│ ├── secs_interop_probe.cpp Probe against secsgem-py passive equip.
|
||
│ ├── secs_bench.cpp Throughput / latency / memory bench.
|
||
│ ├── fuzz_secs2_decode.cpp libFuzzer harness for secs2::decode.
|
||
│ └── fuzz_sml_parse.cpp libFuzzer harness for try_parse_sml.
|
||
│
|
||
├── tests/ doctest unit + integration tests.
|
||
│ └── test_*.cpp 50 files, 445 cases, 2753 assertions.
|
||
│
|
||
├── data/ YAML configs (the spec-as-data).
|
||
│ ├── messages.yaml SECS-II message catalog (164 msgs).
|
||
│ ├── control_state.yaml E30 §6.2 transition table.
|
||
│ ├── process_job_state.yaml E40 transition table.
|
||
│ ├── control_job_state.yaml E94 transition table.
|
||
│ └── equipment.yaml Demo SVIDs/ECIDs/CEIDs/alarms/recipes.
|
||
│
|
||
├── tools/ Build-time scripts.
|
||
│ └── generate_messages.py Codegen: messages.yaml → messages.hpp.
|
||
│
|
||
├── interop/ External-validator harnesses.
|
||
│ ├── README.md Harness-by-harness detail.
|
||
│ ├── host_vs_cpp_server.py secsgem-py active host driving us.
|
||
│ ├── passive_equipment.py secsgem-py passive equipment for us to drive.
|
||
│ ├── raw_gem300_harness.py Raw S3/S14/S16/S12 round-trip.
|
||
│ ├── tshark_validate.sh pcap + tshark HSMS dissector check.
|
||
│ ├── secs4j_validate.sh secs4java8 (Java) cross-validation.
|
||
│ └── secs4j/ Dockerfile + harness for secs4java8.
|
||
│
|
||
├── examples/
|
||
│ └── pvd_tool/ Worked vendor example: fictional PVD tool.
|
||
│ ├── README.md What the example shows.
|
||
│ ├── equipment.yaml Realistic SVIDs/ECIDs/CEIDs/alarms/recipes.
|
||
│ └── main.cpp Sensor sim, recipe runner, alarm monitor.
|
||
│
|
||
└── docs/ This guide + reference docs.
|
||
├── 00_index.md The series TOC.
|
||
├── 01–51_*.md Tutorial chapters.
|
||
├── ARCHITECTURE.md One-page architecture overview.
|
||
├── COMPLIANCE.md Per-capability audit.
|
||
├── INTEGRATION.md Vendor-side production deploy.
|
||
├── PROOFS.md 8 commands proving feature-completeness.
|
||
├── VERIFICATION.md External-validator test plan.
|
||
├── BENCHMARKS.md Performance envelope.
|
||
├── MES_INTEROP.md Commercial-MES day-1 punch list.
|
||
├── SECURITY.md nftables / stunnel / minisign configs.
|
||
├── GLOSSARY.md SEMI vocabulary cheat sheet.
|
||
└── FAQ.md Canonical answers.
|
||
```
|
||
|
||
---
|
||
|
||
## The dependency graph
|
||
|
||
```
|
||
data/*.yaml
|
||
│
|
||
┌─────────────┼──────────────────┐
|
||
│ (codegen) │ (runtime load) │
|
||
▼ ▼ ▼
|
||
generated/messages.hpp config::loader
|
||
│ │
|
||
└──────────► gem::EquipmentDataModel
|
||
│
|
||
│ used by
|
||
▼
|
||
gem::Router
|
||
│
|
||
│ wraps
|
||
▼
|
||
secs2::Message ◄─── codec / SML
|
||
│
|
||
│ over
|
||
▼
|
||
hsms::Connection / secsi::TcpTransport
|
||
│
|
||
▼
|
||
TCP socket
|
||
```
|
||
|
||
Read it bottom-up: a TCP socket carries bytes; `hsms::Connection`
|
||
frames them into `secs2::Message`s; `gem::Router` dispatches by
|
||
`(stream, function)` to handlers; handlers read/write
|
||
`EquipmentDataModel`; the model composes per-domain stores; the
|
||
stores were built from the YAML at startup.
|
||
|
||
No layer ever calls *up* the graph. `secs2::Item` has no idea
|
||
HSMS exists. `hsms::Connection` doesn't know about CEIDs.
|
||
`gem::Router` doesn't know whether the bytes came over HSMS or
|
||
SECS-I. Strict layering is what keeps the codebase small.
|
||
|
||
---
|
||
|
||
## The binaries
|
||
|
||
Built by [`CMakeLists.txt`](../CMakeLists.txt) (one file, ~250
|
||
lines). Each binary lives in `build/` after `cmake --build`.
|
||
|
||
| Binary | Source | What it does |
|
||
|----------------------|-----------------------------------------------------------------|-------------------------------------------------------|
|
||
| `secs_server` | [`apps/secs_server.cpp`](../apps/secs_server.cpp) | Passive equipment. Listens on TCP, dispatches via Router. |
|
||
| `secs_client` | [`apps/secs_client.cpp`](../apps/secs_client.cpp) | Active host. Drives ~24 transactions in the demo. |
|
||
| `secs_conformance` | [`apps/secs_conformance.cpp`](../apps/secs_conformance.cpp) | 47 wire-level conformance checks against a live server. |
|
||
| `secs_interop_probe` | [`apps/secs_interop_probe.cpp`](../apps/secs_interop_probe.cpp) | Active host probing a secsgem-py passive equipment. |
|
||
| `secs_bench` | [`apps/secs_bench.cpp`](../apps/secs_bench.cpp) | Throughput / latency / memory harness. |
|
||
| `secsgem_tests` | All `tests/*.cpp` | The 445-case doctest binary. |
|
||
| `fuzz_secs2_decode` | [`apps/fuzz_secs2_decode.cpp`](../apps/fuzz_secs2_decode.cpp) | libFuzzer (clang only, opt-in `-DSECSGEM_FUZZ=ON`). |
|
||
| `fuzz_sml_parse` | [`apps/fuzz_sml_parse.cpp`](../apps/fuzz_sml_parse.cpp) | libFuzzer for the SML parser. |
|
||
|
||
A worked example binary `pvd_tool` (from `examples/pvd_tool/`) is
|
||
also built by the same `CMakeLists.txt` when the example is
|
||
included.
|
||
|
||
---
|
||
|
||
## How the build system finds everything
|
||
|
||
`CMakeLists.txt` does five things in order:
|
||
|
||
1. **Pull in dependencies** — `find_package(Threads)`,
|
||
`find_package(yaml-cpp)`, `FetchContent` for doctest. Standalone
|
||
Asio is header-only (no link step).
|
||
2. **Run codegen** — invokes `tools/generate_messages.py` to turn
|
||
`data/messages.yaml` into `build/generated/secsgem/gem/messages.hpp`.
|
||
Listed as a custom command so it re-runs when `messages.yaml`
|
||
changes.
|
||
3. **Build the library** — `add_library(secsgem ...)` with every
|
||
source under `src/` plus the generated header.
|
||
4. **Build the apps** — one `add_executable` per `apps/*.cpp`,
|
||
each linking against `secsgem`.
|
||
5. **Build the tests** — `add_executable(secsgem_tests ...)` with
|
||
every `tests/*.cpp`, linked against doctest + `secsgem`.
|
||
|
||
Build flags:
|
||
|
||
- **`-DSECSGEM_TSAN=ON`** — adds `-fsanitize=thread` to a
|
||
separate build dir. CI runs this lane.
|
||
- **`-DSECSGEM_FUZZ=ON`** — requires clang; adds libFuzzer + ASan +
|
||
UBSan; builds the two fuzz harnesses.
|
||
|
||
Everything else (Release / Debug, parallelism, output dirs) is
|
||
standard CMake.
|
||
|
||
---
|
||
|
||
## Test layout
|
||
|
||
50 test files; 445 test cases; 2 753 assertions. One file per
|
||
concern. Naming is `test_<thing>.cpp` consistently:
|
||
|
||
- `test_secs2.cpp`, `test_e5_kat.cpp`, `test_sml.cpp`,
|
||
`test_messages.cpp` — codec.
|
||
- `test_hsms*.cpp` (5 files), `test_secsi*.cpp` (3 files) — transport.
|
||
- `test_control_state.cpp`, `test_communication_state.cpp`,
|
||
`test_data_model.cpp`, `test_host_handler.cpp`, `test_loader.cpp`,
|
||
`test_config_validate.cpp` — E30.
|
||
- `test_process_jobs.cpp`, `test_control_jobs.cpp`,
|
||
`test_carriers.cpp`, `test_substrates.cpp`, `test_ept.cpp`,
|
||
`test_modules.cpp`, `test_cem_objects.cpp`, `test_e84*.cpp`,
|
||
`test_e42_formatted_pp.cpp` — GEM 300.
|
||
- `test_*_persistence.cpp` (4) — file-backed journal.
|
||
- `test_robustness_fuzz.cpp` — randomized property test.
|
||
- `test_thread_safety.cpp` — TSan-validated single-threaded contract.
|
||
- `test_metrics_prometheus.cpp` — Prometheus exporter.
|
||
- `test_wire_ceid_emission.cpp` — CEID firings observed on a real socket.
|
||
- `test_live_gem300.cpp`, `test_gem300_scenario.cpp` — multi-FSM cascades.
|
||
|
||
Full per-standard breakdown:
|
||
[`docs/PROOFS.md`](PROOFS.md) "Per-standard test coverage" table.
|
||
|
||
---
|
||
|
||
## Where to go next
|
||
|
||
Now that you know what's where, the next chapter explains the
|
||
*philosophy* that makes the codebase this small: the **spec-as-data**
|
||
principle, and how the YAML files + codegen + runtime loader work
|
||
together so adding a new SVID / state / message rarely requires C++.
|
||
|
||
Next: [→ 31 Spec-as-data + codegen](31_spec_as_data_and_codegen.md)
|