secs-gem/docs/34_codec_and_sml.md

# 34 — Codec and SML

← [33 Transport](33_transport.md) | [Back to index](00_index.md) | Next: [35 State machines and dispatch](35_state_machines_and_dispatch.md) →

We covered the SECS-II encoding rules in chapter 10.  This
chapter is the **implementation walk** — the four files that make
up `secsgem::secs2`, how the encoder/decoder are structured, why
the variant-based `Item` works, and how the SML printer/parser
fits in.

Four files, 733 lines total.  The codec is the most-tested layer
in the codebase.

---

## The four files

```
include/secsgem/secs2/
├── item.hpp     (170 lines)  Item variant + Format enum + factories.
├── codec.hpp    ( 30 lines)  encode / decode declarations.
├── message.hpp  ( 52 lines)  Message wrapper (header fields + body Item).
└── sml.hpp      ( 32 lines)  to_sml / try_parse_sml declarations.

src/secs2/
├── codec.cpp    (229 lines)  encode_into / decode_at implementations.
└── sml.cpp      (220 lines)  SML printer + parser.
```

`item.hpp` and `message.hpp` are header-only.  `codec.cpp` and
`sml.cpp` carry the heavy lifting.

---

## The `Item` variant

Already covered in chapter 10; quick recap of the storage:

```cpp
// include/secsgem/secs2/item.hpp:85
class Item {
 public:
  using List = std::vector<Item>;
  using Storage = std::variant<
      List,                   // List
      std::string,            // ASCII, JIS-8
      std::vector<uint8_t>,   // Binary, Boolean, U1
      std::vector<int8_t>,    // I1
      std::vector<int16_t>,   // I2
      ...
      std::vector<float>,     // F4
      std::vector<double>>;   // F8

 private:
  Format format_;
  Storage data_;
};
```

Eleven variant alternatives serving 16 SECS-II formats — some
formats share storage (Binary/Boolean/U1 all use
`std::vector<uint8_t>`, ASCII/JIS-8 share `std::string`, U2/C2
share `std::vector<uint16_t>`).  Disambiguation is via `format_`.

### Factories

The intended way to build an `Item` is the named factories:

```cpp
Item::list({Item::ascii("Hi"), Item::u4(42)});
Item::ascii("Hello, world");
Item::u4(std::vector<uint32_t>{1, 2, 3});
Item::u4(42);   // scalar convenience overload
Item::f4(1.0f);
```

Each takes ownership of the storage (or constructs from a scalar
overload).  No exceptions; no validity checks; trusts the caller.

---

## `encode_into` — the recursive encoder

```cpp
void encode_into(const Item& item, std::vector<uint8_t>& out);
```

[`src/secs2/codec.cpp:71`](../src/secs2/codec.cpp).  Two paths —
List and not-List:

```cpp
void encode_into(const Item& item, std::vector<uint8_t>& out) {
  const Format fmt = item.format();

  if (fmt == Format::List) {
    const auto& children = item.as_list();
    write_header(out, fmt, children.size());
    for (const auto& child : children) encode_into(child, out);
    return;
  }

  // Scalar/array: write_header(byte count), then bytes.
  switch (fmt) {
    case Format::ASCII: {
      const auto& s = item.as_ascii();
      write_header(out, fmt, s.size());
      out.insert(out.end(), s.begin(), s.end());
      return;
    }
    case Format::U4: {
      const auto& v = std::get<std::vector<uint32_t>>(item.storage());
      write_header(out, fmt, v.size() * 4);
      for (auto x : v) put_scalar_be(out, x);
      return;
    }
    // ... one case per format
  }
}
```

`write_header` picks the smallest length-byte-count and emits the
format byte + length bytes.  `put_scalar_be` is the
templated big-endian writer using `std::bit_cast` for floats and
`std::make_unsigned_t` for integers (chapter 10).

`encode(item)` is a thin wrapper:

```cpp
std::vector<uint8_t> encode(const Item& item) {
  std::vector<uint8_t> out;
  encode_into(item, out);
  return out;
}
```

---

## `decode_at` — the recursive decoder

```cpp
Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos);
```

Mirror image:

```cpp
Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos) {
  // 1. Format byte + length bytes.
  if (pos >= len) throw CodecError("truncated");
  const uint8_t  fb   = data[pos++];
  const Format   fmt  = static_cast<Format>(fb >> 2);
  const int      nlen = fb & 0x03;
  if (pos + nlen > len) throw CodecError("truncated length bytes");
  std::size_t length = 0;
  for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++];

  // 2. List recursion.
  if (fmt == Format::List) {
    Item::List children;
    children.reserve(length);
    for (std::size_t i = 0; i < length; ++i)
      children.push_back(decode_at(data, len, pos));
    return Item::list(std::move(children));
  }

  // 3. Scalar/array: dispatch on element size + signedness/floatness.
  if (pos + length > len) throw CodecError("truncated body");
  const uint8_t* body = data + pos;
  pos += length;
  switch (fmt) {
    case Format::ASCII: return Item::ascii(std::string((const char*)body, length));
    case Format::U4:    return Item::u4(read_array<uint32_t>(body, length));
    // ... one case per format
  }
  throw CodecError("unknown format code");
}

Item decode(const std::vector<uint8_t>& bytes) {
  std::size_t pos = 0;
  Item it = decode_at(bytes.data(), bytes.size(), pos);
  if (pos != bytes.size()) throw CodecError("trailing bytes");
  return it;
}
```

The `_at` variant is useful when an outer protocol carries a SECS-II
item *embedded* in a larger frame — the caller passes the buffer
and a position, and gets back the item plus the new position.

Bounds checks throw `CodecError` at every step — a CodecError on
the receive side closes the connection (chapter 11's S9F7 path).

---

## The Message wrapper

```cpp
// include/secsgem/secs2/message.hpp
class Message {
 public:
  uint8_t  stream() const;
  uint8_t  function() const;
  bool     w_bit() const;
  uint32_t system_bytes() const;
  const Item& body() const;
  std::vector<uint8_t> body_bytes() const;  // encoded body
};
```

A `Message` is just a small struct: stream + function + W-bit +
system_bytes + body Item.  No encoder lives here — encoding is
done by `secs2::encode(message.body())` when the transport layer
serializes a frame.  The Message exists so the Router can dispatch
on `(stream, function)` without re-decoding bytes.

---

## SML — the human-readable form

`to_sml(item)` walks the Item recursively and emits SML:

```cpp
// src/secs2/sml.cpp — sketch
std::string to_sml(const Item& item) {
  switch (item.format()) {
    case Format::List: {
      std::string s = "<L[" + std::to_string(item.size()) + "]";
      for (const auto& child : item.as_list()) {
        s += ' ' + to_sml(child);
      }
      s += '>';
      return s;
    }
    case Format::ASCII: return "A \"" + escape(item.as_ascii()) + "\"";
    case Format::U4: {
      const auto& v = std::get<std::vector<uint32_t>>(item.storage());
      std::string s = "U4";
      if (v.size() > 1) s += "[" + std::to_string(v.size()) + "]";
      for (auto x : v) s += " " + std::to_string(x);
      return s;
    }
    // ... per format
  }
}
```

`try_parse_sml(text)` is the inverse — a hand-written recursive-
descent parser that returns `std::optional<Item>`.  Returns
`nullopt` on any parse error (no exceptions; this is what
libFuzzer feeds garbage into and expects it not to crash).

Tests:
[`tests/test_sml.cpp`](../tests/test_sml.cpp) (10 cases — every
format round-trips through `to_sml` → `try_parse_sml` → identical
Item).

### Why SML doesn't round-trip *bytes*

A subtle point: `decode(encode(item))` round-trips exactly, but
`try_parse_sml(to_sml(item))` *also* round-trips the Item — except
encoding the round-tripped Item may produce **different bytes**
than the original.  Why?

- The original might use a 2-byte length encoding; the
  round-tripped Item is a fresh `Item` and the encoder will pick
  the smallest length encoding (1 byte).
- SML doesn't preserve "which list-length encoding the encoder
  chose."

If you need bit-exact round-trip of *bytes*, use `decode(encode)`.
For semantic round-trip of *values*, use SML.

---

## Testing — every layer in isolation

| Layer        | Test file                            | Cases | Focus                                                  |
|--------------|--------------------------------------|------:|--------------------------------------------------------|
| Item factories | tests/test_secs2.cpp               | 14    | Construction, equality, format dispatch.               |
| Codec        | tests/test_e5_kat.cpp                | 19    | Known-answer tests — bit-exact bytes per SEMI E5 §9.   |
| Codec        | tests/test_secs2.cpp                 | (overlap) | encode/decode round-trip + truncation rejection.   |
| Identifier wildcards | tests/test_identifier_wildcards.cpp | 6 | U1/U2/U4/U8 leniency for ID fields.                |
| SML          | tests/test_sml.cpp                   | 10    | to_sml + try_parse_sml round-trip.                     |
| Catalog      | tests/test_messages.cpp              | 82    | Every named SxFy builder + parser round-trip.          |
| Random/structural | tests/test_fuzz.cpp             | 8     | Random bytes, truncation, oversize lengths, nested.    |
| libFuzzer    | apps/fuzz_secs2_decode.cpp           | (CI)  | 200 k+ random inputs per minute, ASan + UBSan clean.   |
| libFuzzer    | apps/fuzz_sml_parse.cpp              | (CI)  | 1.4 M+ random SML strings per minute, ASan + UBSan.    |

The codec alone has **139 test cases / 196+ assertions for E5
KAT**.  This is intentional: every other layer trusts the codec is
correct.  If it isn't, nothing above works.

---

## Where to go next

You've now seen the codec and SML implementation in detail.  Next
chapter covers the **dispatch** layer that sits between the
transport (which delivers raw `Message`s) and the stores (which
hold state): `gem::Router`, the state-machine wiring, and the
generated builder/parser glue from the message catalog.

Next: [→ 35 State machines and dispatch](35_state_machines_and_dispatch.md)