# 51 — Extending the codebase ← [50 API + messages + YAML reference](50_api_messages_yaml_reference.md) | [Back to index](00_index.md) | End of series. Last chapter. Practical recipes for the seven most common extensions, each with the actual mechanical steps. Roughly ordered from "no C++ at all" to "the most C++ you'll write." | Recipe | C++ needed? | |-----------------------------------------------|---------------------| | 1. New SVID / DVID / ECID | None | | 2. New CEID with linked reports | None | | 3. New host command | None | | 4. New control-state transition | None | | 5. New SECS-II message | Handler only | | 6. New store | New header + tests | | 7. New persistence backend | Substantial | For each one: the YAML change (if any), the C++ change (if any), the test to add, and where to look up details. --- ## 1. New SVID / DVID / ECID The simplest extension. Add one line to `data/equipment.yaml`: ```yaml svids: # ... existing entries ... - {id: 50, name: ChamberTemp, units: "C", type: F4, value: 25.0} ``` Restart. Done. Host can now read SVID 50 via: - `S1F11 [50]` → returns its name and units. - `S1F3 [50]` → returns its current value. The EAP can update it at any time: ```cpp model->svids.set_value(50, secs2::Item::f4(new_temperature)); ``` Same pattern for DVIDs and ECIDs. For ECIDs add `min` and `max` for range validation. **Test**: not required for new SVIDs alone, but [`tests/test_data_model.cpp`](../tests/test_data_model.cpp) shows the pattern. **Reference**: chapter 31 §New SVID; [`docs/COMPLIANCE.md`](COMPLIANCE.md) §4 (Variable / Status / Constant rows). --- ## 2. New CEID with linked reports Two-step YAML edit: ```yaml # data/equipment.yaml ceids: - {id: 500, name: ChamberTempHigh} events: default_reports: - {ceid: 500, vids: [50]} # link to the SVID we just added ``` Restart. Done. When the EAP fires CEID 500: ```cpp on_temp_threshold_exceeded(float temp) { asio::post(io, [model, temp] { if (!model->is_event_enabled(500)) return; auto reports = model->compose_reports_for(500); auto msg = build_s6f11(500, reports); deliver_or_spool(*conn, *model, std::move(msg)); }); } ``` S6F11 lands at the host with `[RPTID=..., V=[chamber_temp]]`. The host can re-link reports dynamically via S2F33/F35/F37 — the `default_reports` YAML entry is just the initial state. **Test**: pattern in [`tests/test_wire_ceid_emission.cpp`](../tests/test_wire_ceid_emission.cpp). --- ## 3. New host command Add to `data/equipment.yaml`: ```yaml host_commands: - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2} ``` Restart. Done. Host sends `S2F41(RCMD="VENT")`: - `HCACK = 0` (Accept). - CEID 400 fires → S6F11. - ALID 2 set → S5F1. For commands with **application logic** beyond emit-CEID + set-alarm, register a custom handler: ```cpp // At startup: model->commands.register_handler("VENT", [model](const ParamList& params) -> CommandOutcome { // Actually vent the chamber here. if (!vacuum_safe_to_vent()) { return {HostCmdAck::CannotPerformNow, {}}; } hardware_vent_chamber(); return {HostCmdAck::Accept, {}}; }); ``` The registered handler overrides the YAML-defined default. **Reference**: chapter 31 §New host command; [`include/secsgem/gem/store/host_commands.hpp`](../include/secsgem/gem/store/host_commands.hpp). --- ## 4. New control-state transition Edit `data/control_state.yaml`: ```yaml transitions: # ... existing rows ... - {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept} ``` Restart. The transition is now active. No code changes. For transitions chaining through `AttemptOnline`, use `then`: ```yaml - {from: EquipmentOffline, on: operator_switch_online, to: AttemptOnline, then: OnlineRemote, ack: Accept} ``` Same pattern for `process_job_state.yaml` and `control_job_state.yaml`. **Test**: pattern in [`tests/test_control_state.cpp`](../tests/test_control_state.cpp). --- ## 5. New SECS-II message Two-part: YAML for the wire shape, C++ for the handler. ### 5a. Add the message to the catalog ```yaml # data/messages.yaml - id: S6F30 stream: 6 function: 30 w: true builder: s6f30_query parser: parse_s6f30 body: kind: list struct_name: TemperatureQuery fields: - {name: vid, shape: {kind: scalar, item_type: U4}} - {name: threshold, shape: {kind: scalar, item_type: F4}} - id: S6F31 stream: 6 function: 31 w: false builder: s6f31_query_reply parser: parse_s6f31 body: kind: scalar item_type: BOOLEAN param: above_threshold ``` `docker compose run --rm builder` regenerates `messages.hpp`. The codegen produces: ```cpp struct TemperatureQuery { uint32_t vid; float threshold; }; inline secs2::Message s6f30_query(uint32_t vid, float threshold); inline std::optional parse_s6f30(const secs2::Item&); inline secs2::Message s6f31_query_reply(bool above_threshold); inline std::optional parse_s6f31(const secs2::Item&); ``` ### 5b. Register a handler ```cpp router->on(6, 30, [model](const secs2::Message& m) { auto query = messages::parse_s6f30(m.body()); if (!query) return messages::s6f31_query_reply(false); auto val = model->svids.value(query->vid); if (!val) return messages::s6f31_query_reply(false); float current = std::get>(val->storage())[0]; return messages::s6f31_query_reply(current > query->threshold); }); ``` That's it. The new S/F is on the wire. **Reference**: chapter 31 §New SECS-II message; [`tests/test_messages.cpp`](../tests/test_messages.cpp) for the testing pattern. --- ## 6. New store When you need a record type that doesn't map onto an existing store. E.g., add a `ReticleStore` for lithography reticles distinct from substrates. ### Create the header ```cpp // include/secsgem/gem/store/reticles.hpp #pragma once #include #include #include namespace secsgem::gem { enum class ReticleState : uint8_t { Loaded = 0, Aligned = 1, InUse = 2, Unloaded = 3, }; struct ReticleRecord { std::string id; ReticleState state; int usage_count; }; class ReticleStore { public: using ChangeHandler = std::function; void register_reticle(std::string id); void set_state(const std::string& id, ReticleState s); std::optional get(const std::string& id) const; std::vector all() const; void set_change_handler(ChangeHandler h) { on_change_ = std::move(h); } private: std::map records_; ChangeHandler on_change_; }; } // namespace secsgem::gem ``` ### Add to EquipmentDataModel ```cpp // include/secsgem/gem/data_model.hpp struct EquipmentDataModel { // ... existing members ... ReticleStore reticles; }; ``` ### Write tests ```cpp // tests/test_reticles.cpp #include "secsgem/gem/store/reticles.hpp" #include using secsgem::gem::ReticleStore; using secsgem::gem::ReticleState; TEST_CASE("ReticleStore: register and look up") { ReticleStore s; s.register_reticle("R-001"); auto r = s.get("R-001"); REQUIRE(r.has_value()); CHECK(r->id == "R-001"); } TEST_CASE("ReticleStore: state change fires handler") { ReticleStore s; s.register_reticle("R-002"); ReticleState observed_from{}, observed_to{}; s.set_change_handler([&](auto& id, auto from, auto to) { observed_from = from; observed_to = to; }); s.set_state("R-002", ReticleState::Aligned); CHECK(observed_from == ReticleState::Loaded); CHECK(observed_to == ReticleState::Aligned); } ``` CMake picks up new tests automatically (glob over `tests/*.cpp`). ### Wire Router handlers if needed If reticles need wire access (e.g., a custom S6FX request), add the message to `data/messages.yaml` (recipe 5) and register handlers. **Reference**: chapter 32 §How to add a new store. --- ## 7. New persistence backend The codebase ships file-backed persistence with per-record files (chapter 36). Some deployments want different backends — SQLite, LMDB, a key-value cache. The persistence is wired *inside each store* rather than through an abstraction, so changing the backend means changing each store's `enable_persistence` implementation. Two approaches: ### 7a. Drop-in replacement Replace the file IO inside each store's `journal_write` / `journal_remove` / `journal_replay` methods with calls to your backend. Pros: no API change, no test churn. Cons: changes 7 stores; you have to update each one. ### 7b. Pluggable backend Introduce an interface: ```cpp class JournalBackend { public: virtual ~JournalBackend() = default; virtual void write(std::string_view key, const std::vector&) = 0; virtual std::optional> read(std::string_view key) = 0; virtual void remove(std::string_view key) = 0; virtual std::vector list_keys() = 0; }; ``` Each store accepts a `std::shared_ptr`. The default implementation is `FileJournalBackend` (current behaviour); alternatives can be `SqliteJournalBackend`, `LmdbJournalBackend`, etc. Pros: clean separation, multiple backends coexist. Cons: substantial refactor across 7 stores + their tests. For most deployments option 7a is the right call — the file backend is fast enough that swap-outs are rare. **Reference**: chapter 36 §The per-record file pattern; [`tests/test_persistence_upgrade.cpp`](../tests/test_persistence_upgrade.cpp) for the test patterns. --- ## What to do when something doesn't fit any recipe Some extensions don't map onto these seven. Examples: - A new SEMI standard the codebase doesn't implement. - A transport that isn't HSMS or SECS-I. - A different codec (highly unusual). - A different YAML schema (e.g., a third-party format). For these, the right move is to: 1. **Open an issue / RFC** describing what you want. 2. **Sketch the API change** before writing code. 3. **Add tests first** — at the integration layer. 4. **Reach into the right namespace** based on the chapter map at the top of this guide. The codebase is small and the layering is clean; major extensions usually fit naturally into one of the 21 stores or one of the existing namespaces. Resist the temptation to add a new abstraction layer; almost everything that looks like it needs one actually fits as a new store + a few handlers. --- ## End of the guide You've reached the end. You should now be able to: - Read any SECS/GEM standard and recognise its shape. - Read any commit in this codebase and place it in the architecture. - Read any wire trace and trace it back to a Router handler. - Add new SVIDs / CEIDs / commands / states / messages without recompiling. - Add new stores or wire to new SECS standards with confidence. - Stand up the demo, drive every external validator, and reason about deployment + monitoring + security. If anything in the codebase still surprises you, the chapter map at [`docs/00_index.md`](00_index.md) is your starting point for finding the relevant section. The proofs in [`docs/PROOFS.md`](PROOFS.md) are the **claim**; this guide was the **explanation**. Treat them as paired documents. ← [Back to index](00_index.md)