secs-gem/docs/51_extending_the_codebase.md

# 51 — Extending the codebase

← [50 API + messages + YAML reference](50_api_messages_yaml_reference.md) | [Back to index](00_index.md) | End of series.

Last chapter.  Practical recipes for the seven most common
extensions, each with the actual mechanical steps.  Roughly
ordered from "no C++ at all" to "the most C++ you'll write."

| Recipe                                        | C++ needed?         |
|-----------------------------------------------|---------------------|
| 1. New SVID / DVID / ECID                     | None                |
| 2. New CEID with linked reports               | None                |
| 3. New host command                           | None                |
| 4. New control-state transition               | None                |
| 5. New SECS-II message                        | Handler only        |
| 6. New store                                  | New header + tests  |
| 7. New persistence backend                    | Substantial         |

For each one: the YAML change (if any), the C++ change (if any),
the test to add, and where to look up details.

---

## 1. New SVID / DVID / ECID

The simplest extension.  Add one line to `data/equipment.yaml`:

```yaml
svids:
  # ... existing entries ...
  - {id: 50, name: ChamberTemp, units: "C", type: F4, value: 25.0}
```

Restart.  Done.

Host can now read SVID 50 via:

- `S1F11 [50]` → returns its name and units.
- `S1F3  [50]` → returns its current value.

The EAP can update it at any time:

```cpp
model->svids.set_value(50, secs2::Item::f4(new_temperature));
```

Same pattern for DVIDs and ECIDs.  For ECIDs add `min` and `max`
for range validation.

**Test**: not required for new SVIDs alone, but
[`tests/test_data_model.cpp`](../tests/test_data_model.cpp) shows
the pattern.

**Reference**: chapter 31 §New SVID;
[`docs/COMPLIANCE.md`](COMPLIANCE.md) §4 (Variable / Status /
Constant rows).

---

## 2. New CEID with linked reports

Two-step YAML edit:

```yaml
# data/equipment.yaml

ceids:
  - {id: 500, name: ChamberTempHigh}

events:
  default_reports:
    - {ceid: 500, vids: [50]}    # link to the SVID we just added
```

Restart.  Done.  When the EAP fires CEID 500:

```cpp
on_temp_threshold_exceeded(float temp) {
  asio::post(io, [model, temp] {
    if (!model->is_event_enabled(500)) return;
    auto reports = model->compose_reports_for(500);
    auto msg = build_s6f11(500, reports);
    deliver_or_spool(*conn, *model, std::move(msg));
  });
}
```

S6F11 lands at the host with `[RPTID=..., V=[chamber_temp]]`.

The host can re-link reports dynamically via S2F33/F35/F37 — the
`default_reports` YAML entry is just the initial state.

**Test**: pattern in
[`tests/test_wire_ceid_emission.cpp`](../tests/test_wire_ceid_emission.cpp).

---

## 3. New host command

Add to `data/equipment.yaml`:

```yaml
host_commands:
  - {name: VENT,
     ack: Accept,
     emit_ceid: 400,
     set_alarm: 2}
```

Restart.  Done.  Host sends `S2F41(RCMD="VENT")`:

- `HCACK = 0` (Accept).
- CEID 400 fires → S6F11.
- ALID 2 set → S5F1.

For commands with **application logic** beyond emit-CEID +
set-alarm, register a custom handler:

```cpp
// At startup:
model->commands.register_handler("VENT",
    [model](const ParamList& params) -> CommandOutcome {
      // Actually vent the chamber here.
      if (!vacuum_safe_to_vent()) {
        return {HostCmdAck::CannotPerformNow, {}};
      }
      hardware_vent_chamber();
      return {HostCmdAck::Accept, {}};
    });
```

The registered handler overrides the YAML-defined default.

**Reference**: chapter 31 §New host command;
[`include/secsgem/gem/store/host_commands.hpp`](../include/secsgem/gem/store/host_commands.hpp).

---

## 4. New control-state transition

Edit `data/control_state.yaml`:

```yaml
transitions:
  # ... existing rows ...
  - {from: OnlineRemote, on: host_request_offline, to: HostOffline, ack: Accept}
```

Restart.  The transition is now active.  No code changes.

For transitions chaining through `AttemptOnline`, use `then`:

```yaml
- {from: EquipmentOffline, on: operator_switch_online,
   to: AttemptOnline, then: OnlineRemote, ack: Accept}
```

Same pattern for `process_job_state.yaml` and
`control_job_state.yaml`.

**Test**: pattern in
[`tests/test_control_state.cpp`](../tests/test_control_state.cpp).

---

## 5. New SECS-II message

Two-part: YAML for the wire shape, C++ for the handler.

### 5a. Add the message to the catalog

```yaml
# data/messages.yaml
- id: S6F30
  stream: 6
  function: 30
  w: true
  builder: s6f30_query
  parser: parse_s6f30
  body:
    kind: list
    struct_name: TemperatureQuery
    fields:
      - {name: vid, shape: {kind: scalar, item_type: U4}}
      - {name: threshold, shape: {kind: scalar, item_type: F4}}

- id: S6F31
  stream: 6
  function: 31
  w: false
  builder: s6f31_query_reply
  parser: parse_s6f31
  body:
    kind: scalar
    item_type: BOOLEAN
    param: above_threshold
```

`docker compose run --rm builder` regenerates `messages.hpp`.

The codegen produces:

```cpp
struct TemperatureQuery {
  uint32_t vid;
  float    threshold;
};

inline secs2::Message s6f30_query(uint32_t vid, float threshold);
inline std::optional<TemperatureQuery> parse_s6f30(const secs2::Item&);

inline secs2::Message s6f31_query_reply(bool above_threshold);
inline std::optional<bool> parse_s6f31(const secs2::Item&);
```

### 5b. Register a handler

```cpp
router->on(6, 30, [model](const secs2::Message& m) {
  auto query = messages::parse_s6f30(m.body());
  if (!query) return messages::s6f31_query_reply(false);
  auto val = model->svids.value(query->vid);
  if (!val) return messages::s6f31_query_reply(false);
  float current = std::get<std::vector<float>>(val->storage())[0];
  return messages::s6f31_query_reply(current > query->threshold);
});
```

That's it.  The new S/F is on the wire.

**Reference**: chapter 31 §New SECS-II message;
[`tests/test_messages.cpp`](../tests/test_messages.cpp) for the
testing pattern.

---

## 6. New store

When you need a record type that doesn't map onto an existing
store.  E.g., add a `ReticleStore` for lithography reticles
distinct from substrates.

### Create the header

```cpp
// include/secsgem/gem/store/reticles.hpp
#pragma once

#include <map>
#include <string>
#include <vector>

namespace secsgem::gem {

enum class ReticleState : uint8_t {
  Loaded   = 0,
  Aligned  = 1,
  InUse    = 2,
  Unloaded = 3,
};

struct ReticleRecord {
  std::string id;
  ReticleState state;
  int usage_count;
};

class ReticleStore {
 public:
  using ChangeHandler =
      std::function<void(const std::string&, ReticleState from, ReticleState to)>;

  void register_reticle(std::string id);
  void set_state(const std::string& id, ReticleState s);
  std::optional<ReticleRecord> get(const std::string& id) const;
  std::vector<ReticleRecord> all() const;

  void set_change_handler(ChangeHandler h) { on_change_ = std::move(h); }

 private:
  std::map<std::string, ReticleRecord> records_;
  ChangeHandler on_change_;
};

}  // namespace secsgem::gem
```

### Add to EquipmentDataModel

```cpp
// include/secsgem/gem/data_model.hpp
struct EquipmentDataModel {
  // ... existing members ...
  ReticleStore reticles;
};
```

### Write tests

```cpp
// tests/test_reticles.cpp
#include "secsgem/gem/store/reticles.hpp"
#include <doctest/doctest.h>

using secsgem::gem::ReticleStore;
using secsgem::gem::ReticleState;

TEST_CASE("ReticleStore: register and look up") {
  ReticleStore s;
  s.register_reticle("R-001");
  auto r = s.get("R-001");
  REQUIRE(r.has_value());
  CHECK(r->id == "R-001");
}

TEST_CASE("ReticleStore: state change fires handler") {
  ReticleStore s;
  s.register_reticle("R-002");
  ReticleState observed_from{}, observed_to{};
  s.set_change_handler([&](auto& id, auto from, auto to) {
    observed_from = from;
    observed_to = to;
  });
  s.set_state("R-002", ReticleState::Aligned);
  CHECK(observed_from == ReticleState::Loaded);
  CHECK(observed_to == ReticleState::Aligned);
}
```

CMake picks up new tests automatically (glob over `tests/*.cpp`).

### Wire Router handlers if needed

If reticles need wire access (e.g., a custom S6FX request), add the
message to `data/messages.yaml` (recipe 5) and register handlers.

**Reference**: chapter 32 §How to add a new store.

---

## 7. New persistence backend

The codebase ships file-backed persistence with per-record files
(chapter 36).  Some deployments want different backends — SQLite,
LMDB, a key-value cache.

The persistence is wired *inside each store* rather than through
an abstraction, so changing the backend means changing each
store's `enable_persistence` implementation.  Two approaches:

### 7a. Drop-in replacement

Replace the file IO inside each store's `journal_write` /
`journal_remove` / `journal_replay` methods with calls to your
backend.

Pros: no API change, no test churn.
Cons: changes 7 stores; you have to update each one.

### 7b. Pluggable backend

Introduce an interface:

```cpp
class JournalBackend {
 public:
  virtual ~JournalBackend() = default;
  virtual void write(std::string_view key, const std::vector<uint8_t>&) = 0;
  virtual std::optional<std::vector<uint8_t>> read(std::string_view key) = 0;
  virtual void remove(std::string_view key) = 0;
  virtual std::vector<std::string> list_keys() = 0;
};
```

Each store accepts a `std::shared_ptr<JournalBackend>`.  The
default implementation is `FileJournalBackend` (current behaviour);
alternatives can be `SqliteJournalBackend`, `LmdbJournalBackend`,
etc.

Pros: clean separation, multiple backends coexist.
Cons: substantial refactor across 7 stores + their tests.

For most deployments option 7a is the right call — the file
backend is fast enough that swap-outs are rare.

**Reference**: chapter 36 §The per-record file pattern;
[`tests/test_persistence_upgrade.cpp`](../tests/test_persistence_upgrade.cpp)
for the test patterns.

---

## What to do when something doesn't fit any recipe

Some extensions don't map onto these seven.  Examples:

- A new SEMI standard the codebase doesn't implement.
- A transport that isn't HSMS or SECS-I.
- A different codec (highly unusual).
- A different YAML schema (e.g., a third-party format).

For these, the right move is to:

1. **Open an issue / RFC** describing what you want.
2. **Sketch the API change** before writing code.
3. **Add tests first** — at the integration layer.
4. **Reach into the right namespace** based on the chapter map at
   the top of this guide.

The codebase is small and the layering is clean; major extensions
usually fit naturally into one of the 21 stores or one of the
existing namespaces.  Resist the temptation to add a new abstraction
layer; almost everything that looks like it needs one actually
fits as a new store + a few handlers.

---

## End of the guide

You've reached the end.  You should now be able to:

- Read any SECS/GEM standard and recognise its shape.
- Read any commit in this codebase and place it in the
  architecture.
- Read any wire trace and trace it back to a Router handler.
- Add new SVIDs / CEIDs / commands / states / messages without
  recompiling.
- Add new stores or wire to new SECS standards with confidence.
- Stand up the demo, drive every external validator, and reason
  about deployment + monitoring + security.

If anything in the codebase still surprises you, the chapter map
at [`docs/00_index.md`](00_index.md) is your starting point for
finding the relevant section.

The proofs in [`docs/PROOFS.md`](PROOFS.md) are the **claim**;
this guide was the **explanation**.  Treat them as paired
documents.

← [Back to index](00_index.md)