docs: chapters 40, 41, 50, 51 — Operations + Reference (series complete)
Last four chapters of the guided tour. 40 — Building, running, the demo. Docker prerequisites, the build flow, what each binary is for, running the 24-transaction demo flow annotated step by step. Running the 4 external-validator sweeps + the libFuzzer pass. Inspecting the demo with tcpdump and tshark. Reading source while running as the recommended learning workflow. 41 — Integration: hardware, MES, production. Four-phase tour: wiring sensors / recipe engine / alarms / E84 GPIO; talking to a real MES with the day-1 punch list + commercial-MES quirks (Wonderware S2F21, Camstar Linktest cadence, etc.); production hardening (nftables / stunnel / minisign / persistence layout / monitoring / runbook); performance envelope + memory footprint + capacity planning. Pointers to the long-form INTEGRATION.md / MES_INTEROP.md / SECURITY.md / BENCHMARKS.md. 50 — API + message catalog + YAML schemas reference. Namespace-by- namespace table of public symbols (secs2, hsms, secsi, gem, config, metrics) with brief descriptions. Stream-by-stream message catalog reference (S1, S2, S3, S5, S6, S7, S9, S10, S12, S14, S16). YAML schema reference for messages.yaml + the three state-table files + equipment.yaml. 51 — Extending the codebase. Seven recipes ordered from no-code to substantial: new SVID/DVID/ECID (YAML only), new CEID with reports (YAML only), new host command (YAML + optional handler), new control- state transition (YAML only), new SECS-II message (YAML + handler), new store (header + tests), new persistence backend (drop-in vs pluggable trade-off). Each recipe has the actual mechanical steps, the test pattern, and pointers to the chapter that explains why it works. Index updated to mark all 24 chapters published. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,376 @@
|
||||
# 40 — Building, running, the demo
|
||||
|
||||
← [36 Persistence, validation, metrics](36_persistence_validation_metrics.md) | [Back to index](00_index.md) | Next: [41 Integration: hardware, MES, production](41_integration_hardware_mes_production.md) →
|
||||
|
||||
You've read about every layer of the codebase. Now we run it.
|
||||
|
||||
This chapter is operational: build the project, start the demo,
|
||||
walk what each transaction in the two-container flow actually does
|
||||
and where it lives. By the end you'll have the demo running on
|
||||
your laptop and you'll know what every log line means.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Just **Docker**. No host C++ toolchain, no Python deps, nothing
|
||||
to apt-install. The toolchain image (`Dockerfile`) bundles Ubuntu
|
||||
24.04 + g++-13 + CMake + Ninja + asio + yaml-cpp + Python 3 +
|
||||
tshark + tcpdump + clang.
|
||||
|
||||
```bash
|
||||
docker --version
|
||||
docker compose version
|
||||
```
|
||||
|
||||
If both work, you're set.
|
||||
|
||||
---
|
||||
|
||||
## Building
|
||||
|
||||
```bash
|
||||
docker compose run --rm builder
|
||||
```
|
||||
|
||||
That:
|
||||
|
||||
1. Pulls / builds the toolchain image (first time only, ~3
|
||||
minutes).
|
||||
2. Runs `cmake -S /app -B /app/build -G Ninja -DCMAKE_BUILD_TYPE=Release`.
|
||||
3. Runs `cmake --build /app/build`.
|
||||
4. Produces every binary under `/app/build/` inside a named Docker
|
||||
volume.
|
||||
|
||||
Subsequent builds are incremental and take ~10–30 s.
|
||||
|
||||
### What got built
|
||||
|
||||
```
|
||||
build/
|
||||
├── secs_server passive equipment (the demo target)
|
||||
├── secs_client active host (drives the demo)
|
||||
├── secs_conformance 47-check conformance harness
|
||||
├── secs_interop_probe active host probing secsgem-py equipment
|
||||
├── secs_bench throughput/latency bench
|
||||
├── secsgem_tests the 445-case doctest binary
|
||||
└── pvd_tool worked PVD-tool example
|
||||
```
|
||||
|
||||
Plus the generated `build/generated/secsgem/gem/messages.hpp`
|
||||
(~3 500 lines, auto-derived from `data/messages.yaml`).
|
||||
|
||||
---
|
||||
|
||||
## Running the tests
|
||||
|
||||
```bash
|
||||
docker compose run --rm tests
|
||||
```
|
||||
|
||||
Runs `secsgem_tests` end-to-end. Expected output:
|
||||
|
||||
```
|
||||
[doctest] doctest version is "2.4.11"
|
||||
[doctest] run with "--help" for options
|
||||
===============================================================================
|
||||
[doctest] test cases: 445 | 445 passed | 0 failed | 0 skipped
|
||||
[doctest] assertions: 2753 | 2753 passed | 0 failed |
|
||||
[doctest] Status: SUCCESS!
|
||||
```
|
||||
|
||||
On a 2024 M-series Mac under Docker Desktop, this takes ~3.5 s.
|
||||
|
||||
---
|
||||
|
||||
## The two-container demo
|
||||
|
||||
```bash
|
||||
docker compose up --no-deps server client
|
||||
```
|
||||
|
||||
That starts:
|
||||
|
||||
- A **`server`** container running `secs_server` on port 5000.
|
||||
- A **`client`** container running `secs_client` against `server:5000`.
|
||||
|
||||
The client drives ~24 SECS transactions through the data model.
|
||||
Each transaction logs on both sides.
|
||||
|
||||
### What each transaction does
|
||||
|
||||
Annotated walk through the log output:
|
||||
|
||||
#### Communication establishment
|
||||
|
||||
```
|
||||
[host] connecting to server:5000
|
||||
[equip] accepted connection
|
||||
[host] sending Select.req
|
||||
[equip] Select.req received → SELECTED
|
||||
[host] Select.rsp(Ok) received → SELECTED
|
||||
```
|
||||
|
||||
HSMS SELECT handshake. Both sides now in SELECTED state.
|
||||
|
||||
```
|
||||
[host] sending S1F13 Establish Communications
|
||||
[equip] S1F13 received
|
||||
[equip] sending S1F14(COMMACK=Accept, [MDLN, SOFTREV])
|
||||
[host] S1F14 received → COMMUNICATING
|
||||
```
|
||||
|
||||
E30 §6.5 communication-state transition. Now GEM-level
|
||||
communication is up.
|
||||
|
||||
#### Identification
|
||||
|
||||
```
|
||||
[host] S1F1 Are You There
|
||||
[equip] S1F2 ["SECS-GEM Demo Equipment", "1.0.0"]
|
||||
[host] S1F19 GEM Compliance Request
|
||||
[equip] S1F20 [list of capabilities]
|
||||
[host] S1F11 SVID Namelist (all)
|
||||
[equip] S1F12 [SVID 1 "ControlState", SVID 2 "Clock", ...]
|
||||
[host] S1F21 DVID Namelist (all)
|
||||
[equip] S1F22 [DVID list]
|
||||
[host] S1F23 CEID Namelist (all)
|
||||
[equip] S1F24 [CEID → VID mapping]
|
||||
```
|
||||
|
||||
Host walks the data dictionary.
|
||||
|
||||
#### Dynamic event report setup
|
||||
|
||||
```
|
||||
[host] S2F33 DefineReport(RPTID=1, VIDs=[SVID 2])
|
||||
[equip] S2F34(DRACK=0)
|
||||
[host] S2F35 LinkEvent(CEID=300 → [RPTID=1])
|
||||
[equip] S2F36(LRACK=0)
|
||||
[host] S2F37 EnableEvent(CEED=true, CEIDs=[300])
|
||||
[equip] S2F38(ERACK=0)
|
||||
```
|
||||
|
||||
The three-message report wiring. CEID 300 now triggers an S6F11
|
||||
when it fires.
|
||||
|
||||
#### Control state + remote command
|
||||
|
||||
```
|
||||
[host] S2F41 RCMD=START
|
||||
[equip] S2F42(HCACK=Accept)
|
||||
[equip] HostCommandRegistry dispatched START
|
||||
[equip] → emit CEID 300
|
||||
[equip] → compose_reports_for(300) → RPTID 1 = [Clock SV2]
|
||||
[equip] → fire S6F11
|
||||
[equip] S6F11(CEID=300, [RPTID=1, [Clock]])
|
||||
[host] S6F12(ACKC6=0)
|
||||
```
|
||||
|
||||
Host command dispatch + event report emission + acknowledgement.
|
||||
This is the canonical GEM transaction.
|
||||
|
||||
#### Alarms
|
||||
|
||||
```
|
||||
[host] S5F5 List all alarms
|
||||
[equip] S5F6 [ALID list with ALCD + ALTX]
|
||||
[host] S5F3 EnableAlarm(ALID=1)
|
||||
[equip] S5F4(ACKC5=0)
|
||||
[host] S2F41 RCMD=FAULT
|
||||
[equip] S2F42(HCACK=Accept)
|
||||
[equip] → set ALID 1
|
||||
[equip] → fire S5F1(ALCD=0x84, ALID=1)
|
||||
[equip] S5F1(...)
|
||||
[host] S5F2(ACKC5=0)
|
||||
```
|
||||
|
||||
#### Recipes
|
||||
|
||||
```
|
||||
[host] S7F1 PP Load Inquire(PPID="NEW-RECIPE", LENGTH=64)
|
||||
[equip] S7F2(PPGNT=0=Permit)
|
||||
[host] S7F3 PP Send(PPID="NEW-RECIPE", PPBODY=<bytes>)
|
||||
[equip] S7F4(ACKC7=0)
|
||||
[host] S7F5 PP Request(PPID="NEW-RECIPE")
|
||||
[equip] S7F6 [PPID, PPBODY]
|
||||
[host] S7F17 PP Delete(PPIDs=["NEW-RECIPE"])
|
||||
[equip] S7F18(ACKC7=0)
|
||||
```
|
||||
|
||||
#### Terminal display
|
||||
|
||||
```
|
||||
[host] S10F3 Terminal Display Multi (TID=0, TEXT="hello\nfrom host")
|
||||
[equip] S10F4(ACKC10=0)
|
||||
```
|
||||
|
||||
#### Clean shutdown
|
||||
|
||||
```
|
||||
[host] S1F15 Request Offline
|
||||
[equip] S1F16(OFLACK=Accept)
|
||||
[host] sending Separate.req
|
||||
[equip] Separate.req received → close
|
||||
```
|
||||
|
||||
Total: 24 transactions exercising S1, S2, S5, S6, S7, S10.
|
||||
|
||||
---
|
||||
|
||||
## Running the conformance harness
|
||||
|
||||
```bash
|
||||
docker compose up -d server
|
||||
docker compose run --rm builder /app/build/secs_conformance --host server --port 5000
|
||||
docker compose down
|
||||
```
|
||||
|
||||
Runs the 47-check conformance harness against the demo server.
|
||||
Each check covers one E30 / GEM 300 wire-level behaviour:
|
||||
|
||||
```
|
||||
[PASS] E37 §7.2 SELECT handshake
|
||||
[PASS] E30 §6.5 S1F13/F14 Establish Comms
|
||||
[PASS] E30 §6.7 S1F1/F2 Are You There
|
||||
... (43 more)
|
||||
[PASS] E30 §6.10 S1F19/F20 GEM Compliance
|
||||
|
||||
47 / 47 checks passed
|
||||
```
|
||||
|
||||
This is proof #2 in [`docs/PROOFS.md`](PROOFS.md).
|
||||
|
||||
---
|
||||
|
||||
## Running the interop sweeps
|
||||
|
||||
### secsgem-py
|
||||
|
||||
```bash
|
||||
docker compose up -d server
|
||||
docker compose run --rm interop python3 /app/interop/host_vs_cpp_server.py --host server
|
||||
docker compose down
|
||||
```
|
||||
|
||||
The Python `secsgem-py` 0.3.0 host drives our equipment. 31 checks
|
||||
across S1/S2/S5/S6/S7/S10 + unsolicited S6F11 / S5F1.
|
||||
|
||||
### secs4java8
|
||||
|
||||
```bash
|
||||
bash interop/secs4j_validate.sh
|
||||
```
|
||||
|
||||
The Java secs4java8 host drives our equipment via a separate
|
||||
container. 55 checks covering S1/S2/S3/S5/S6/S7/S10/S14/S16
|
||||
including the GEM 300 streams that secsgem-py couldn't easily
|
||||
drive.
|
||||
|
||||
### tshark dissector
|
||||
|
||||
```bash
|
||||
docker compose run --rm builder bash /app/interop/tshark_validate.sh
|
||||
```
|
||||
|
||||
Captures a pcap of the demo flow, dissects with Wireshark's HSMS
|
||||
dissector, asserts no malformed packets. 69 frames, 0 errors.
|
||||
|
||||
### libFuzzer (60 s, requires clang)
|
||||
|
||||
```bash
|
||||
docker compose run --rm builder bash -c "
|
||||
cmake -S /app -B /app/build-fuzz -G Ninja -DSECSGEM_FUZZ=ON \
|
||||
-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
|
||||
cmake --build /app/build-fuzz
|
||||
/app/build-fuzz/fuzz_secs2_decode -max_total_time=60
|
||||
/app/build-fuzz/fuzz_sml_parse -max_total_time=60
|
||||
"
|
||||
```
|
||||
|
||||
200 k+ inputs through `secs2::decode`, 1.4 M+ through
|
||||
`try_parse_sml`, ASan + UBSan clean, 0 crashes.
|
||||
|
||||
All five sweeps are wired into CI; see
|
||||
[`.gitea/workflows/ci.yml`](../.gitea/workflows/ci.yml).
|
||||
|
||||
---
|
||||
|
||||
## Inspecting the demo from outside
|
||||
|
||||
While the demo is running, you can:
|
||||
|
||||
### Watch the wire
|
||||
|
||||
```bash
|
||||
# In another shell:
|
||||
docker compose exec server tcpdump -i any -A -s 0 'tcp port 5000'
|
||||
```
|
||||
|
||||
### Inspect with tshark + HSMS dissector
|
||||
|
||||
```bash
|
||||
docker compose run --rm builder tshark -i any -d "tcp.port==5000,hsms" -V \
|
||||
| grep -A 2 "Header"
|
||||
```
|
||||
|
||||
### Watch the metrics
|
||||
|
||||
`pvd_tool` example exposes a Prometheus endpoint:
|
||||
|
||||
```bash
|
||||
docker run --rm -p 9090:9090 pvd_tool /app/examples/pvd_tool/equipment.yaml \
|
||||
/app/data/control_state.yaml 5000 9090
|
||||
```
|
||||
|
||||
Then `curl localhost:9090/metrics`.
|
||||
|
||||
---
|
||||
|
||||
## Running the bench
|
||||
|
||||
```bash
|
||||
docker compose run --rm builder /app/build/secs_bench \
|
||||
--requests 50000 --concurrency 32 --svid-count 32
|
||||
```
|
||||
|
||||
Outputs a markdown table of throughput + p50/p95/p99 latencies for:
|
||||
|
||||
- S1F1/F2 (header-only round-trip).
|
||||
- S1F3/F4 with 32 SVIDs.
|
||||
- S6F11 push (W=0, fire-and-forget).
|
||||
- PJ + CJ memory footprint.
|
||||
|
||||
See [`docs/BENCHMARKS.md`](BENCHMARKS.md) for the baseline numbers
|
||||
and capacity-planning notes.
|
||||
|
||||
---
|
||||
|
||||
## Reading the source while it runs
|
||||
|
||||
A common workflow when you're learning:
|
||||
|
||||
1. `docker compose up --no-deps server client` in one shell.
|
||||
2. Source viewer open in another (your IDE on the host —
|
||||
the source isn't bind-mounted in the container, but it is
|
||||
on your host).
|
||||
3. Find a log line that confuses you (e.g. `[equip] S6F11 fired`).
|
||||
4. Grep the source for it. Most log strings are unique enough to
|
||||
land in the right file in one search.
|
||||
5. Read the function around it.
|
||||
6. Cross-reference back to the chapter that covers the standard.
|
||||
|
||||
This is the most efficient way to internalise the codebase. The
|
||||
demo runs forever (until you `Ctrl-C` — the client loops); you
|
||||
can read the source at your own pace.
|
||||
|
||||
---
|
||||
|
||||
## Where to go next
|
||||
|
||||
You now have the demo running and you can drive any of the five
|
||||
external validators. The next chapter is the **integration**
|
||||
chapter — wiring the runtime to real hardware, talking to a real
|
||||
MES, production deployment, security, performance tuning.
|
||||
|
||||
Next: [→ 41 Integration: hardware, MES, production](41_integration_hardware_mes_production.md)
|
||||
Reference in New Issue
Block a user