secs-gem/docs/51_extending_the_codebase.md

51 — Extending the codebase

← 50 API + messages + YAML reference | Back to index | End of series.

Last chapter. Practical recipes for the seven most common extensions, each with the actual mechanical steps. Roughly ordered from "no C++ at all" to "the most C++ you'll write."

Recipe	C++ needed?
1. New SVID / DVID / ECID	None
2. New CEID with linked reports	None
3. New host command	None
4. New control-state transition	None
5. New SECS-II message	Handler only
6. New store	New header + tests
7. New persistence backend	Substantial

For each one: the YAML change (if any), the C++ change (if any), the test to add, and where to look up details.

---

1. New SVID / DVID / ECID

The simplest extension. Add one line to data/equipment.yaml:

svids:
  # ... existing entries ...
  - {id: 50, name: ChamberTemp, units: "C", type: F4, value: 25.0}

Restart. Done.

Host can now read SVID 50 via:

• S1F11 [50] → returns its name and units.
• S1F3 [50] → returns its current value.

The EAP can update it at any time:

model->svids.set_value(50, secs2::Item::f4(new_temperature));

Same pattern for DVIDs and ECIDs. For ECIDs add min and max for range validation.

Test: not required for new SVIDs alone, but tests/test_data_model.cpp shows the pattern.

Reference: chapter 31 §New SVID; docs/COMPLIANCE.md §4 (Variable / Status / Constant rows).

---

2. New CEID with linked reports

Two-step YAML edit:

# data/equipment.yaml

ceids:
  - {id: 500, name: ChamberTempHigh}

events:
  default_reports:
    - {ceid: 500, vids: [50]}    # link to the SVID we just added

Restart. Done. When the EAP fires CEID 500:

on_temp_threshold_exceeded(float temp) {
  asio::post(io, [model, temp] {
    if (!model->is_event_enabled(500)) return;
    auto reports = model->compose_reports_for(500);
    auto msg = build_s6f11(500, reports);
    deliver_or_spool(*conn, *model, std::move(msg));
  });
}

S6F11 lands at the host with [RPTID=..., V=[chamber_temp]].

The host can re-link reports dynamically via S2F33/F35/F37 — the default_reports YAML entry is just the initial state.

Test: pattern in tests/test_wire_ceid_emission.cpp.

---

3. New host command

Add to data/equipment.yaml:

host_commands:
  - {name: VENT,
     ack: Accept,
     emit_ceid: 400,
     set_alarm: 2}

Restart. Done. Host sends S2F41(RCMD="VENT"):

• HCACK = 0 (Accept).
• CEID 400 fires → S6F11.
• ALID 2 set → S5F1.

For commands with application logic beyond emit-CEID + set-alarm, register a custom handler:

// At startup:
model->commands.register_handler("VENT",
    [model](const ParamList& params) -> CommandOutcome {
      // Actually vent the chamber here.
      if (!vacuum_safe_to_vent()) {
        return {HostCmdAck::CannotPerformNow, {}};
      }
      hardware_vent_chamber();
      return {HostCmdAck::Accept, {}};
    });

The registered handler overrides the YAML-defined default.

Reference: chapter 31 §New host command; include/secsgem/gem/store/host_commands.hpp.

---

4. New control-state transition

Edit data/control_state.yaml:

transitions:
  # ... existing rows ...
  - {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept}

Restart. The transition is now active. No code changes.

For transitions chaining through AttemptOnline, use then:

- {from: EquipmentOffline, on: operator_switch_online,
   to: AttemptOnline, then: OnlineRemote, ack: Accept}

Same pattern for process_job_state.yaml and control_job_state.yaml.

Test: pattern in tests/test_control_state.cpp.

---

5. New SECS-II message

Two-part: YAML for the wire shape, C++ for the handler.

5a. Add the message to the catalog

# data/messages.yaml
- id: S6F30
  stream: 6
  function: 30
  w: true
  builder: s6f30_query
  parser: parse_s6f30
  body:
    kind: list
    struct_name: TemperatureQuery
    fields:
      - {name: vid, shape: {kind: scalar, item_type: U4}}
      - {name: threshold, shape: {kind: scalar, item_type: F4}}

- id: S6F31
  stream: 6
  function: 31
  w: false
  builder: s6f31_query_reply
  parser: parse_s6f31
  body:
    kind: scalar
    item_type: BOOLEAN
    param: above_threshold

docker compose run --rm builder regenerates messages.hpp.

The codegen produces:

struct TemperatureQuery {
  uint32_t vid;
  float    threshold;
};

inline secs2::Message s6f30_query(uint32_t vid, float threshold);
inline std::optional<TemperatureQuery> parse_s6f30(const secs2::Item&);

inline secs2::Message s6f31_query_reply(bool above_threshold);
inline std::optional<bool> parse_s6f31(const secs2::Item&);

5b. Register a handler

router->on(6, 30, [model](const secs2::Message& m) {
  auto query = messages::parse_s6f30(m.body());
  if (!query) return messages::s6f31_query_reply(false);
  auto val = model->svids.value(query->vid);
  if (!val) return messages::s6f31_query_reply(false);
  float current = std::get<std::vector<float>>(val->storage())[0];
  return messages::s6f31_query_reply(current > query->threshold);
});

That's it. The new S/F is on the wire.

Reference: chapter 31 §New SECS-II message; tests/test_messages.cpp for the testing pattern.

---

6. New store

When you need a record type that doesn't map onto an existing store. E.g., add a ReticleStore for lithography reticles distinct from substrates.

Create the header

// include/secsgem/gem/store/reticles.hpp
#pragma once

#include <map>
#include <string>
#include <vector>

namespace secsgem::gem {

enum class ReticleState : uint8_t {
  Loaded   = 0,
  Aligned  = 1,
  InUse    = 2,
  Unloaded = 3,
};

struct ReticleRecord {
  std::string id;
  ReticleState state;
  int usage_count;
};

class ReticleStore {
 public:
  using ChangeHandler =
      std::function<void(const std::string&, ReticleState from, ReticleState to)>;

  void register_reticle(std::string id);
  void set_state(const std::string& id, ReticleState s);
  std::optional<ReticleRecord> get(const std::string& id) const;
  std::vector<ReticleRecord> all() const;

  void set_change_handler(ChangeHandler h) { on_change_ = std::move(h); }

 private:
  std::map<std::string, ReticleRecord> records_;
  ChangeHandler on_change_;
};

}  // namespace secsgem::gem

Add to EquipmentDataModel

// include/secsgem/gem/data_model.hpp
struct EquipmentDataModel {
  // ... existing members ...
  ReticleStore reticles;
};

Write tests

// tests/test_reticles.cpp
#include "secsgem/gem/store/reticles.hpp"
#include <doctest/doctest.h>

using secsgem::gem::ReticleStore;
using secsgem::gem::ReticleState;

TEST_CASE("ReticleStore: register and look up") {
  ReticleStore s;
  s.register_reticle("R-001");
  auto r = s.get("R-001");
  REQUIRE(r.has_value());
  CHECK(r->id == "R-001");
}

TEST_CASE("ReticleStore: state change fires handler") {
  ReticleStore s;
  s.register_reticle("R-002");
  ReticleState observed_from{}, observed_to{};
  s.set_change_handler([&](auto& id, auto from, auto to) {
    observed_from = from;
    observed_to = to;
  });
  s.set_state("R-002", ReticleState::Aligned);
  CHECK(observed_from == ReticleState::Loaded);
  CHECK(observed_to == ReticleState::Aligned);
}

CMake picks up new tests automatically (glob over tests/*.cpp).

Wire Router handlers if needed

If reticles need wire access (e.g., a custom S6FX request), add the message to data/messages.yaml (recipe 5) and register handlers.

Reference: chapter 32 §How to add a new store.

---

7. New persistence backend

The codebase ships file-backed persistence with per-record files (chapter 36). Some deployments want different backends — SQLite, LMDB, a key-value cache.

The persistence is wired inside each store rather than through an abstraction, so changing the backend means changing each store's enable_persistence implementation. Two approaches:

7a. Drop-in replacement

Replace the file IO inside each store's journal_write / journal_remove / journal_replay methods with calls to your backend.

Pros: no API change, no test churn. Cons: changes 7 stores; you have to update each one.

7b. Pluggable backend

Introduce an interface:

class JournalBackend {
 public:
  virtual ~JournalBackend() = default;
  virtual void write(std::string_view key, const std::vector<uint8_t>&) = 0;
  virtual std::optional<std::vector<uint8_t>> read(std::string_view key) = 0;
  virtual void remove(std::string_view key) = 0;
  virtual std::vector<std::string> list_keys() = 0;
};

Each store accepts a std::shared_ptr<JournalBackend>. The default implementation is FileJournalBackend (current behaviour); alternatives can be SqliteJournalBackend, LmdbJournalBackend, etc.

Pros: clean separation, multiple backends coexist. Cons: substantial refactor across 7 stores + their tests.

For most deployments option 7a is the right call — the file backend is fast enough that swap-outs are rare.

Reference: chapter 36 §The per-record file pattern; tests/test_persistence_upgrade.cpp for the test patterns.

---

What to do when something doesn't fit any recipe

Some extensions don't map onto these seven. Examples:

• A new SEMI standard the codebase doesn't implement.
• A transport that isn't HSMS or SECS-I.
• A different codec (highly unusual).
• A different YAML schema (e.g., a third-party format).

For these, the right move is to:

1. Open an issue / RFC describing what you want.
2. Sketch the API change before writing code.
3. Add tests first — at the integration layer.
4. Reach into the right namespace based on the chapter map at the top of this guide.

The codebase is small and the layering is clean; major extensions usually fit naturally into one of the 21 stores or one of the existing namespaces. Resist the temptation to add a new abstraction layer; almost everything that looks like it needs one actually fits as a new store + a few handlers.

---

End of the guide

You've reached the end. You should now be able to:

• Read any SECS/GEM standard and recognise its shape.
• Read any commit in this codebase and place it in the architecture.
• Read any wire trace and trace it back to a Router handler.
• Add new SVIDs / CEIDs / commands / states / messages without recompiling.
• Add new stores or wire to new SECS standards with confidence.
• Stand up the demo, drive every external validator, and reason about deployment + monitoring + security.

If anything in the codebase still surprises you, the chapter map at docs/00_index.md is your starting point for finding the relevant section.

The proofs in docs/PROOFS.md are the claim; this guide was the explanation. Treat them as paired documents.

← Back to index