docs: chapters 30–36 — the codebase (Part 3 complete)
Seven chapters walking the implementation top-to-bottom. 30 — Repository tour. Top-level layout, directory by directory. The eight built binaries. The dependency graph from TCP socket up through EquipmentDataModel. CMake's role. Test layout. 31 — Spec-as-data and codegen. Why the design choice fits SECS/ GEM specifically. The five YAML files: messages catalog, control/PJ/CJ transition tables, equipment dictionary. How tools/gen_messages.py turns messages.yaml into typed C++ at build time. The --validate-config multi-error validator. How to add a new SVID / CEID / host command / state / message without C++. 32 — Stores and the data model. What a store IS (records + API + change handler + optional persistence). Every store in the codebase mapped to the SEMI standard it serves (table of 21). EquipmentDataModel as plain composition + cross-store convenience methods (vid_value, compose_reports_for). The no-locks single- threaded contract. How to add a new store. 33 — Transport. hsms::Connection read path (length+payload async chain), write path (queue + one outstanding write), timer model (5 steady_timers + per-request T3). The asio executor / strand model and why it's the right shape. secsi::Protocol as the IO- free FSM with Action / Event variants; secsi::TcpTransport as the asio adapter. Pattern repeats for E84 + GEM comm-state. 34 — Codec and SML. The four files (170 + 30 + 52 + 32 lines of header, 229 + 220 lines of impl). Item variant storage layout (11 alternatives, 16 formats, shared storage where E5 permits). encode_into recursion; decode_at with bounds checks throwing CodecError. Message wrapper. SML printer + try_parse_sml + why SML round-trips Items but not necessarily bytes. 35 — State machines and dispatch. gem::Router as a typed (stream, function) dispatch table. How an S2F41 round-trip walks through parser → store dispatch → side-effect → CEID emission → S6F11 build → spool-aware deliver. The 11 FSMs all sharing the same three-property shape (pure data table + pure FSM + observer pattern). CEID cascading from FSM transitions to wire bytes. 36 — Persistence, validation, metrics. Which 7 stores have file journals + why the others don't. Per-record file pattern (atomic rename, partial-write safe). Schema versioning + multi-version read. Multi-error YAML validator (--validate-config) + cross-file reference checks. Prometheus registry + HTTP exporter + worked metric patterns from the PVD example. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,303 @@
|
||||
# 34 — Codec and SML
|
||||
|
||||
← [33 Transport](33_transport.md) | [Back to index](00_index.md) | Next: [35 State machines and dispatch](35_state_machines_and_dispatch.md) →
|
||||
|
||||
We covered the SECS-II encoding rules in chapter 10. This
|
||||
chapter is the **implementation walk** — the four files that make
|
||||
up `secsgem::secs2`, how the encoder/decoder are structured, why
|
||||
the variant-based `Item` works, and how the SML printer/parser
|
||||
fits in.
|
||||
|
||||
Four files, 733 lines total. The codec is the most-tested layer
|
||||
in the codebase.
|
||||
|
||||
---
|
||||
|
||||
## The four files
|
||||
|
||||
```
|
||||
include/secsgem/secs2/
|
||||
├── item.hpp (170 lines) Item variant + Format enum + factories.
|
||||
├── codec.hpp ( 30 lines) encode / decode declarations.
|
||||
├── message.hpp ( 52 lines) Message wrapper (header fields + body Item).
|
||||
└── sml.hpp ( 32 lines) to_sml / try_parse_sml declarations.
|
||||
|
||||
src/secs2/
|
||||
├── codec.cpp (229 lines) encode_into / decode_at implementations.
|
||||
└── sml.cpp (220 lines) SML printer + parser.
|
||||
```
|
||||
|
||||
`item.hpp` and `message.hpp` are header-only. `codec.cpp` and
|
||||
`sml.cpp` carry the heavy lifting.
|
||||
|
||||
---
|
||||
|
||||
## The `Item` variant
|
||||
|
||||
Already covered in chapter 10; quick recap of the storage:
|
||||
|
||||
```cpp
|
||||
// include/secsgem/secs2/item.hpp:85
|
||||
class Item {
|
||||
public:
|
||||
using List = std::vector<Item>;
|
||||
using Storage = std::variant<
|
||||
List, // List
|
||||
std::string, // ASCII, JIS-8
|
||||
std::vector<uint8_t>, // Binary, Boolean, U1
|
||||
std::vector<int8_t>, // I1
|
||||
std::vector<int16_t>, // I2
|
||||
...
|
||||
std::vector<float>, // F4
|
||||
std::vector<double>>; // F8
|
||||
|
||||
private:
|
||||
Format format_;
|
||||
Storage data_;
|
||||
};
|
||||
```
|
||||
|
||||
Eleven variant alternatives serving 16 SECS-II formats — some
|
||||
formats share storage (Binary/Boolean/U1 all use
|
||||
`std::vector<uint8_t>`, ASCII/JIS-8 share `std::string`, U2/C2
|
||||
share `std::vector<uint16_t>`). Disambiguation is via `format_`.
|
||||
|
||||
### Factories
|
||||
|
||||
The intended way to build an `Item` is the named factories:
|
||||
|
||||
```cpp
|
||||
Item::list({Item::ascii("Hi"), Item::u4(42)});
|
||||
Item::ascii("Hello, world");
|
||||
Item::u4(std::vector<uint32_t>{1, 2, 3});
|
||||
Item::u4(42); // scalar convenience overload
|
||||
Item::f4(1.0f);
|
||||
```
|
||||
|
||||
Each takes ownership of the storage (or constructs from a scalar
|
||||
overload). No exceptions; no validity checks; trusts the caller.
|
||||
|
||||
---
|
||||
|
||||
## `encode_into` — the recursive encoder
|
||||
|
||||
```cpp
|
||||
void encode_into(const Item& item, std::vector<uint8_t>& out);
|
||||
```
|
||||
|
||||
[`src/secs2/codec.cpp:71`](../src/secs2/codec.cpp). Two paths —
|
||||
List and not-List:
|
||||
|
||||
```cpp
|
||||
void encode_into(const Item& item, std::vector<uint8_t>& out) {
|
||||
const Format fmt = item.format();
|
||||
|
||||
if (fmt == Format::List) {
|
||||
const auto& children = item.as_list();
|
||||
write_header(out, fmt, children.size());
|
||||
for (const auto& child : children) encode_into(child, out);
|
||||
return;
|
||||
}
|
||||
|
||||
// Scalar/array: write_header(byte count), then bytes.
|
||||
switch (fmt) {
|
||||
case Format::ASCII: {
|
||||
const auto& s = item.as_ascii();
|
||||
write_header(out, fmt, s.size());
|
||||
out.insert(out.end(), s.begin(), s.end());
|
||||
return;
|
||||
}
|
||||
case Format::U4: {
|
||||
const auto& v = std::get<std::vector<uint32_t>>(item.storage());
|
||||
write_header(out, fmt, v.size() * 4);
|
||||
for (auto x : v) put_scalar_be(out, x);
|
||||
return;
|
||||
}
|
||||
// ... one case per format
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`write_header` picks the smallest length-byte-count and emits the
|
||||
format byte + length bytes. `put_scalar_be` is the
|
||||
templated big-endian writer using `std::bit_cast` for floats and
|
||||
`std::make_unsigned_t` for integers (chapter 10).
|
||||
|
||||
`encode(item)` is a thin wrapper:
|
||||
|
||||
```cpp
|
||||
std::vector<uint8_t> encode(const Item& item) {
|
||||
std::vector<uint8_t> out;
|
||||
encode_into(item, out);
|
||||
return out;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `decode_at` — the recursive decoder
|
||||
|
||||
```cpp
|
||||
Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos);
|
||||
```
|
||||
|
||||
Mirror image:
|
||||
|
||||
```cpp
|
||||
Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos) {
|
||||
// 1. Format byte + length bytes.
|
||||
if (pos >= len) throw CodecError("truncated");
|
||||
const uint8_t fb = data[pos++];
|
||||
const Format fmt = static_cast<Format>(fb >> 2);
|
||||
const int nlen = fb & 0x03;
|
||||
if (pos + nlen > len) throw CodecError("truncated length bytes");
|
||||
std::size_t length = 0;
|
||||
for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++];
|
||||
|
||||
// 2. List recursion.
|
||||
if (fmt == Format::List) {
|
||||
Item::List children;
|
||||
children.reserve(length);
|
||||
for (std::size_t i = 0; i < length; ++i)
|
||||
children.push_back(decode_at(data, len, pos));
|
||||
return Item::list(std::move(children));
|
||||
}
|
||||
|
||||
// 3. Scalar/array: dispatch on element size + signedness/floatness.
|
||||
if (pos + length > len) throw CodecError("truncated body");
|
||||
const uint8_t* body = data + pos;
|
||||
pos += length;
|
||||
switch (fmt) {
|
||||
case Format::ASCII: return Item::ascii(std::string((const char*)body, length));
|
||||
case Format::U4: return Item::u4(read_array<uint32_t>(body, length));
|
||||
// ... one case per format
|
||||
}
|
||||
throw CodecError("unknown format code");
|
||||
}
|
||||
|
||||
Item decode(const std::vector<uint8_t>& bytes) {
|
||||
std::size_t pos = 0;
|
||||
Item it = decode_at(bytes.data(), bytes.size(), pos);
|
||||
if (pos != bytes.size()) throw CodecError("trailing bytes");
|
||||
return it;
|
||||
}
|
||||
```
|
||||
|
||||
The `_at` variant is useful when an outer protocol carries a SECS-II
|
||||
item *embedded* in a larger frame — the caller passes the buffer
|
||||
and a position, and gets back the item plus the new position.
|
||||
|
||||
Bounds checks throw `CodecError` at every step — a CodecError on
|
||||
the receive side closes the connection (chapter 11's S9F7 path).
|
||||
|
||||
---
|
||||
|
||||
## The Message wrapper
|
||||
|
||||
```cpp
|
||||
// include/secsgem/secs2/message.hpp
|
||||
class Message {
|
||||
public:
|
||||
uint8_t stream() const;
|
||||
uint8_t function() const;
|
||||
bool w_bit() const;
|
||||
uint32_t system_bytes() const;
|
||||
const Item& body() const;
|
||||
std::vector<uint8_t> body_bytes() const; // encoded body
|
||||
};
|
||||
```
|
||||
|
||||
A `Message` is just a small struct: stream + function + W-bit +
|
||||
system_bytes + body Item. No encoder lives here — encoding is
|
||||
done by `secs2::encode(message.body())` when the transport layer
|
||||
serializes a frame. The Message exists so the Router can dispatch
|
||||
on `(stream, function)` without re-decoding bytes.
|
||||
|
||||
---
|
||||
|
||||
## SML — the human-readable form
|
||||
|
||||
`to_sml(item)` walks the Item recursively and emits SML:
|
||||
|
||||
```cpp
|
||||
// src/secs2/sml.cpp — sketch
|
||||
std::string to_sml(const Item& item) {
|
||||
switch (item.format()) {
|
||||
case Format::List: {
|
||||
std::string s = "<L[" + std::to_string(item.size()) + "]";
|
||||
for (const auto& child : item.as_list()) {
|
||||
s += ' ' + to_sml(child);
|
||||
}
|
||||
s += '>';
|
||||
return s;
|
||||
}
|
||||
case Format::ASCII: return "A \"" + escape(item.as_ascii()) + "\"";
|
||||
case Format::U4: {
|
||||
const auto& v = std::get<std::vector<uint32_t>>(item.storage());
|
||||
std::string s = "U4";
|
||||
if (v.size() > 1) s += "[" + std::to_string(v.size()) + "]";
|
||||
for (auto x : v) s += " " + std::to_string(x);
|
||||
return s;
|
||||
}
|
||||
// ... per format
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`try_parse_sml(text)` is the inverse — a hand-written recursive-
|
||||
descent parser that returns `std::optional<Item>`. Returns
|
||||
`nullopt` on any parse error (no exceptions; this is what
|
||||
libFuzzer feeds garbage into and expects it not to crash).
|
||||
|
||||
Tests:
|
||||
[`tests/test_sml.cpp`](../tests/test_sml.cpp) (10 cases — every
|
||||
format round-trips through `to_sml` → `try_parse_sml` → identical
|
||||
Item).
|
||||
|
||||
### Why SML doesn't round-trip *bytes*
|
||||
|
||||
A subtle point: `decode(encode(item))` round-trips exactly, but
|
||||
`try_parse_sml(to_sml(item))` *also* round-trips the Item — except
|
||||
encoding the round-tripped Item may produce **different bytes**
|
||||
than the original. Why?
|
||||
|
||||
- The original might use a 2-byte length encoding; the
|
||||
round-tripped Item is a fresh `Item` and the encoder will pick
|
||||
the smallest length encoding (1 byte).
|
||||
- SML doesn't preserve "which list-length encoding the encoder
|
||||
chose."
|
||||
|
||||
If you need bit-exact round-trip of *bytes*, use `decode(encode)`.
|
||||
For semantic round-trip of *values*, use SML.
|
||||
|
||||
---
|
||||
|
||||
## Testing — every layer in isolation
|
||||
|
||||
| Layer | Test file | Cases | Focus |
|
||||
|--------------|--------------------------------------|------:|--------------------------------------------------------|
|
||||
| Item factories | tests/test_secs2.cpp | 14 | Construction, equality, format dispatch. |
|
||||
| Codec | tests/test_e5_kat.cpp | 19 | Known-answer tests — bit-exact bytes per SEMI E5 §9. |
|
||||
| Codec | tests/test_secs2.cpp | (overlap) | encode/decode round-trip + truncation rejection. |
|
||||
| Identifier wildcards | tests/test_identifier_wildcards.cpp | 6 | U1/U2/U4/U8 leniency for ID fields. |
|
||||
| SML | tests/test_sml.cpp | 10 | to_sml + try_parse_sml round-trip. |
|
||||
| Catalog | tests/test_messages.cpp | 82 | Every named SxFy builder + parser round-trip. |
|
||||
| Random/structural | tests/test_fuzz.cpp | 8 | Random bytes, truncation, oversize lengths, nested. |
|
||||
| libFuzzer | apps/fuzz_secs2_decode.cpp | (CI) | 200 k+ random inputs per minute, ASan + UBSan clean. |
|
||||
| libFuzzer | apps/fuzz_sml_parse.cpp | (CI) | 1.4 M+ random SML strings per minute, ASan + UBSan. |
|
||||
|
||||
The codec alone has **139 test cases / 196+ assertions for E5
|
||||
KAT**. This is intentional: every other layer trusts the codec is
|
||||
correct. If it isn't, nothing above works.
|
||||
|
||||
---
|
||||
|
||||
## Where to go next
|
||||
|
||||
You've now seen the codec and SML implementation in detail. Next
|
||||
chapter covers the **dispatch** layer that sits between the
|
||||
transport (which delivers raw `Message`s) and the stores (which
|
||||
hold state): `gem::Router`, the state-machine wiring, and the
|
||||
generated builder/parser glue from the message catalog.
|
||||
|
||||
Next: [→ 35 State machines and dispatch](35_state_machines_and_dispatch.md)
|
||||
Reference in New Issue
Block a user