docs: chapters 40, 41, 50, 51 — Operations + Reference (series complete)

Last four chapters of the guided tour. 40 — Building, running, the demo. Docker prerequisites, the build flow, what each binary is for, running the 24-transaction demo flow annotated step by step. Running the 4 external-validator sweeps + the libFuzzer pass. Inspecting the demo with tcpdump and tshark. Reading source while running as the recommended learning workflow. 41 — Integration: hardware, MES, production. Four-phase tour: wiring sensors / recipe engine / alarms / E84 GPIO; talking to a real MES with the day-1 punch list + commercial-MES quirks (Wonderware S2F21, Camstar Linktest cadence, etc.); production hardening (nftables / stunnel / minisign / persistence layout / monitoring / runbook); performance envelope + memory footprint + capacity planning. Pointers to the long-form INTEGRATION.md / MES_INTEROP.md / SECURITY.md / BENCHMARKS.md. 50 — API + message catalog + YAML schemas reference. Namespace-by- namespace table of public symbols (secs2, hsms, secsi, gem, config, metrics) with brief descriptions. Stream-by-stream message catalog reference (S1, S2, S3, S5, S6, S7, S9, S10, S12, S14, S16). YAML schema reference for messages.yaml + the three state-table files + equipment.yaml. 51 — Extending the codebase. Seven recipes ordered from no-code to substantial: new SVID/DVID/ECID (YAML only), new CEID with reports (YAML only), new host command (YAML + optional handler), new control- state transition (YAML only), new SECS-II message (YAML + handler), new store (header + tests), new persistence backend (drop-in vs pluggable trade-off). Each recipe has the actual mechanical steps, the test pattern, and pointers to the chapter that explains why it works. Index updated to mark all 24 chapters published. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-09 20:28:21 +02:00
parent cae98d9a7d
commit 31f908e1bf
5 changed files with 1513 additions and 7 deletions
@@ -0,0 +1,439 @@
+# 51 — Extending the codebase
+
+← [50 API + messages + YAML reference](50_api_messages_yaml_reference.md) | [Back to index](00_index.md) | End of series.
+
+Last chapter.  Practical recipes for the seven most common
+extensions, each with the actual mechanical steps.  Roughly
+ordered from "no C++ at all" to "the most C++ you'll write."
+
+| Recipe                                        | C++ needed?         |
+|-----------------------------------------------|---------------------|
+| 1. New SVID / DVID / ECID                     | None                |
+| 2. New CEID with linked reports               | None                |
+| 3. New host command                           | None                |
+| 4. New control-state transition               | None                |
+| 5. New SECS-II message                        | Handler only        |
+| 6. New store                                  | New header + tests  |
+| 7. New persistence backend                    | Substantial         |
+
+For each one: the YAML change (if any), the C++ change (if any),
+the test to add, and where to look up details.
+
+---
+
+## 1. New SVID / DVID / ECID
+
+The simplest extension.  Add one line to `data/equipment.yaml`:
+
+```yaml
+svids:
+  # ... existing entries ...
+  - {id: 50, name: ChamberTemp, units: "C", type: F4, value: 25.0}
+```
+
+Restart.  Done.
+
+Host can now read SVID 50 via:
+
+- `S1F11 [50]` → returns its name and units.
+- `S1F3  [50]` → returns its current value.
+
+The EAP can update it at any time:
+
+```cpp
+model->svids.set_value(50, secs2::Item::f4(new_temperature));
+```
+
+Same pattern for DVIDs and ECIDs.  For ECIDs add `min` and `max`
+for range validation.
+
+**Test**: not required for new SVIDs alone, but
+[`tests/test_data_model.cpp`](../tests/test_data_model.cpp) shows
+the pattern.
+
+**Reference**: chapter 31 §New SVID;
+[`docs/COMPLIANCE.md`](COMPLIANCE.md) §4 (Variable / Status /
+Constant rows).
+
+---
+
+## 2. New CEID with linked reports
+
+Two-step YAML edit:
+
+```yaml
+# data/equipment.yaml
+
+ceids:
+  - {id: 500, name: ChamberTempHigh}
+
+events:
+  default_reports:
+    - {ceid: 500, vids: [50]}    # link to the SVID we just added
+```
+
+Restart.  Done.  When the EAP fires CEID 500:
+
+```cpp
+on_temp_threshold_exceeded(float temp) {
+  asio::post(io, [model, temp] {
+    if (!model->is_event_enabled(500)) return;
+    auto reports = model->compose_reports_for(500);
+    auto msg = build_s6f11(500, reports);
+    deliver_or_spool(*conn, *model, std::move(msg));
+  });
+}
+```
+
+S6F11 lands at the host with `[RPTID=..., V=[chamber_temp]]`.
+
+The host can re-link reports dynamically via S2F33/F35/F37 — the
+`default_reports` YAML entry is just the initial state.
+
+**Test**: pattern in
+[`tests/test_wire_ceid_emission.cpp`](../tests/test_wire_ceid_emission.cpp).
+
+---
+
+## 3. New host command
+
+Add to `data/equipment.yaml`:
+
+```yaml
+host_commands:
+  - {name: VENT,
+     ack: Accept,
+     emit_ceid: 400,
+     set_alarm: 2}
+```
+
+Restart.  Done.  Host sends `S2F41(RCMD="VENT")`:
+
+- `HCACK = 0` (Accept).
+- CEID 400 fires → S6F11.
+- ALID 2 set → S5F1.
+
+For commands with **application logic** beyond emit-CEID +
+set-alarm, register a custom handler:
+
+```cpp
+// At startup:
+model->commands.register_handler("VENT",
+    [model](const ParamList& params) -> CommandOutcome {
+      // Actually vent the chamber here.
+      if (!vacuum_safe_to_vent()) {
+        return {HostCmdAck::CannotPerformNow, {}};
+      }
+      hardware_vent_chamber();
+      return {HostCmdAck::Accept, {}};
+    });
+```
+
+The registered handler overrides the YAML-defined default.
+
+**Reference**: chapter 31 §New host command;
+[`include/secsgem/gem/store/host_commands.hpp`](../include/secsgem/gem/store/host_commands.hpp).
+
+---
+
+## 4. New control-state transition
+
+Edit `data/control_state.yaml`:
+
+```yaml
+transitions:
+  # ... existing rows ...
+  - {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept}
+```
+
+Restart.  The transition is now active.  No code changes.
+
+For transitions chaining through `AttemptOnline`, use `then`:
+
+```yaml
+- {from: EquipmentOffline, on: operator_switch_online,
+   to: AttemptOnline, then: OnlineRemote, ack: Accept}
+```
+
+Same pattern for `process_job_state.yaml` and
+`control_job_state.yaml`.
+
+**Test**: pattern in
+[`tests/test_control_state.cpp`](../tests/test_control_state.cpp).
+
+---
+
+## 5. New SECS-II message
+
+Two-part: YAML for the wire shape, C++ for the handler.
+
+### 5a. Add the message to the catalog
+
+```yaml
+# data/messages.yaml
+- id: S6F30
+  stream: 6
+  function: 30
+  w: true
+  builder: s6f30_query
+  parser: parse_s6f30
+  body:
+    kind: list
+    struct_name: TemperatureQuery
+    fields:
+      - {name: vid, shape: {kind: scalar, item_type: U4}}
+      - {name: threshold, shape: {kind: scalar, item_type: F4}}
+
+- id: S6F31
+  stream: 6
+  function: 31
+  w: false
+  builder: s6f31_query_reply
+  parser: parse_s6f31
+  body:
+    kind: scalar
+    item_type: BOOLEAN
+    param: above_threshold
+```
+
+`docker compose run --rm builder` regenerates `messages.hpp`.
+
+The codegen produces:
+
+```cpp
+struct TemperatureQuery {
+  uint32_t vid;
+  float    threshold;
+};
+
+inline secs2::Message s6f30_query(uint32_t vid, float threshold);
+inline std::optional<TemperatureQuery> parse_s6f30(const secs2::Item&);
+
+inline secs2::Message s6f31_query_reply(bool above_threshold);
+inline std::optional<bool> parse_s6f31(const secs2::Item&);
+```
+
+### 5b. Register a handler
+
+```cpp
+router->on(6, 30, [model](const secs2::Message& m) {
+  auto query = messages::parse_s6f30(m.body());
+  if (!query) return messages::s6f31_query_reply(false);
+  auto val = model->svids.value(query->vid);
+  if (!val) return messages::s6f31_query_reply(false);
+  float current = std::get<std::vector<float>>(val->storage())[0];
+  return messages::s6f31_query_reply(current > query->threshold);
+});
+```
+
+That's it.  The new S/F is on the wire.
+
+**Reference**: chapter 31 §New SECS-II message;
+[`tests/test_messages.cpp`](../tests/test_messages.cpp) for the
+testing pattern.
+
+---
+
+## 6. New store
+
+When you need a record type that doesn't map onto an existing
+store.  E.g., add a `ReticleStore` for lithography reticles
+distinct from substrates.
+
+### Create the header
+
+```cpp
+// include/secsgem/gem/store/reticles.hpp
+#pragma once
+
+#include <map>
+#include <string>
+#include <vector>
+
+namespace secsgem::gem {
+
+enum class ReticleState : uint8_t {
+  Loaded   = 0,
+  Aligned  = 1,
+  InUse    = 2,
+  Unloaded = 3,
+};
+
+struct ReticleRecord {
+  std::string id;
+  ReticleState state;
+  int usage_count;
+};
+
+class ReticleStore {
+ public:
+  using ChangeHandler =
+      std::function<void(const std::string&, ReticleState from, ReticleState to)>;
+
+  void register_reticle(std::string id);
+  void set_state(const std::string& id, ReticleState s);
+  std::optional<ReticleRecord> get(const std::string& id) const;
+  std::vector<ReticleRecord> all() const;
+
+  void set_change_handler(ChangeHandler h) { on_change_ = std::move(h); }
+
+ private:
+  std::map<std::string, ReticleRecord> records_;
+  ChangeHandler on_change_;
+};
+
+}  // namespace secsgem::gem
+```
+
+### Add to EquipmentDataModel
+
+```cpp
+// include/secsgem/gem/data_model.hpp
+struct EquipmentDataModel {
+  // ... existing members ...
+  ReticleStore reticles;
+};
+```
+
+### Write tests
+
+```cpp
+// tests/test_reticles.cpp
+#include "secsgem/gem/store/reticles.hpp"
+#include <doctest/doctest.h>
+
+using secsgem::gem::ReticleStore;
+using secsgem::gem::ReticleState;
+
+TEST_CASE("ReticleStore: register and look up") {
+  ReticleStore s;
+  s.register_reticle("R-001");
+  auto r = s.get("R-001");
+  REQUIRE(r.has_value());
+  CHECK(r->id == "R-001");
+}
+
+TEST_CASE("ReticleStore: state change fires handler") {
+  ReticleStore s;
+  s.register_reticle("R-002");
+  ReticleState observed_from{}, observed_to{};
+  s.set_change_handler([&](auto& id, auto from, auto to) {
+    observed_from = from;
+    observed_to = to;
+  });
+  s.set_state("R-002", ReticleState::Aligned);
+  CHECK(observed_from == ReticleState::Loaded);
+  CHECK(observed_to == ReticleState::Aligned);
+}
+```
+
+CMake picks up new tests automatically (glob over `tests/*.cpp`).
+
+### Wire Router handlers if needed
+
+If reticles need wire access (e.g., a custom S6FX request), add the
+message to `data/messages.yaml` (recipe 5) and register handlers.
+
+**Reference**: chapter 32 §How to add a new store.
+
+---
+
+## 7. New persistence backend
+
+The codebase ships file-backed persistence with per-record files
+(chapter 36).  Some deployments want different backends — SQLite,
+LMDB, a key-value cache.
+
+The persistence is wired *inside each store* rather than through
+an abstraction, so changing the backend means changing each
+store's `enable_persistence` implementation.  Two approaches:
+
+### 7a. Drop-in replacement
+
+Replace the file IO inside each store's `journal_write` /
+`journal_remove` / `journal_replay` methods with calls to your
+backend.
+
+Pros: no API change, no test churn.
+Cons: changes 7 stores; you have to update each one.
+
+### 7b. Pluggable backend
+
+Introduce an interface:
+
+```cpp
+class JournalBackend {
+ public:
+  virtual ~JournalBackend() = default;
+  virtual void write(std::string_view key, const std::vector<uint8_t>&) = 0;
+  virtual std::optional<std::vector<uint8_t>> read(std::string_view key) = 0;
+  virtual void remove(std::string_view key) = 0;
+  virtual std::vector<std::string> list_keys() = 0;
+};
+```
+
+Each store accepts a `std::shared_ptr<JournalBackend>`.  The
+default implementation is `FileJournalBackend` (current behaviour);
+alternatives can be `SqliteJournalBackend`, `LmdbJournalBackend`,
+etc.
+
+Pros: clean separation, multiple backends coexist.
+Cons: substantial refactor across 7 stores + their tests.
+
+For most deployments option 7a is the right call — the file
+backend is fast enough that swap-outs are rare.
+
+**Reference**: chapter 36 §The per-record file pattern;
+[`tests/test_persistence_upgrade.cpp`](../tests/test_persistence_upgrade.cpp)
+for the test patterns.
+
+---
+
+## What to do when something doesn't fit any recipe
+
+Some extensions don't map onto these seven.  Examples:
+
+- A new SEMI standard the codebase doesn't implement.
+- A transport that isn't HSMS or SECS-I.
+- A different codec (highly unusual).
+- A different YAML schema (e.g., a third-party format).
+
+For these, the right move is to:
+
+1. **Open an issue / RFC** describing what you want.
+2. **Sketch the API change** before writing code.
+3. **Add tests first** — at the integration layer.
+4. **Reach into the right namespace** based on the chapter map at
+   the top of this guide.
+
+The codebase is small and the layering is clean; major extensions
+usually fit naturally into one of the 21 stores or one of the
+existing namespaces.  Resist the temptation to add a new abstraction
+layer; almost everything that looks like it needs one actually
+fits as a new store + a few handlers.
+
+---
+
+## End of the guide
+
+You've reached the end.  You should now be able to:
+
+- Read any SECS/GEM standard and recognise its shape.
+- Read any commit in this codebase and place it in the
+  architecture.
+- Read any wire trace and trace it back to a Router handler.
+- Add new SVIDs / CEIDs / commands / states / messages without
+  recompiling.
+- Add new stores or wire to new SECS standards with confidence.
+- Stand up the demo, drive every external validator, and reason
+  about deployment + monitoring + security.
+
+If anything in the codebase still surprises you, the chapter map
+at [`docs/00_index.md`](00_index.md) is your starting point for
+finding the relevant section.
+
+The proofs in [`docs/PROOFS.md`](PROOFS.md) are the **claim**;
+this guide was the **explanation**.  Treat them as paired
+documents.
+
+← [Back to index](00_index.md)