secs-gem/docs/34_codec_and_sml.md

34 — Codec and SML

← 33 Transport | Back to index | Next: 35 State machines and dispatch →

We covered the SECS-II encoding rules in chapter 10. This chapter is the implementation walk — the four files that make up secsgem::secs2, how the encoder/decoder are structured, why the variant-based Item works, and how the SML printer/parser fits in.

Four files, 733 lines total. The codec is the most-tested layer in the codebase.

---

The four files

include/secsgem/secs2/
├── item.hpp     (170 lines)  Item variant + Format enum + factories.
├── codec.hpp    ( 30 lines)  encode / decode declarations.
├── message.hpp  ( 52 lines)  Message wrapper (header fields + body Item).
└── sml.hpp      ( 32 lines)  to_sml / try_parse_sml declarations.

src/secs2/
├── codec.cpp    (229 lines)  encode_into / decode_at implementations.
└── sml.cpp      (220 lines)  SML printer + parser.

item.hpp and message.hpp are header-only. codec.cpp and sml.cpp carry the heavy lifting.

---

The Item variant

Already covered in chapter 10; quick recap of the storage:

// include/secsgem/secs2/item.hpp:85
class Item {
 public:
  using List = std::vector<Item>;
  using Storage = std::variant<
      List,                   // List
      std::string,            // ASCII, JIS-8
      std::vector<uint8_t>,   // Binary, Boolean, U1
      std::vector<int8_t>,    // I1
      std::vector<int16_t>,   // I2
      ...
      std::vector<float>,     // F4
      std::vector<double>>;   // F8

 private:
  Format format_;
  Storage data_;
};

Eleven variant alternatives serving 16 SECS-II formats — some formats share storage (Binary/Boolean/U1 all use std::vector<uint8_t>, ASCII/JIS-8 share std::string, U2/C2 share std::vector<uint16_t>). Disambiguation is via format_.

Factories

The intended way to build an Item is the named factories:

Item::list({Item::ascii("Hi"), Item::u4(42)});
Item::ascii("Hello, world");
Item::u4(std::vector<uint32_t>{1, 2, 3});
Item::u4(42);   // scalar convenience overload
Item::f4(1.0f);

Each takes ownership of the storage (or constructs from a scalar overload). No exceptions; no validity checks; trusts the caller.

---

encode_into — the recursive encoder

void encode_into(const Item& item, std::vector<uint8_t>& out);

src/secs2/codec.cpp:71. Two paths — List and not-List:

void encode_into(const Item& item, std::vector<uint8_t>& out) {
  const Format fmt = item.format();

  if (fmt == Format::List) {
    const auto& children = item.as_list();
    write_header(out, fmt, children.size());
    for (const auto& child : children) encode_into(child, out);
    return;
  }

  // Scalar/array: write_header(byte count), then bytes.
  switch (fmt) {
    case Format::ASCII: {
      const auto& s = item.as_ascii();
      write_header(out, fmt, s.size());
      out.insert(out.end(), s.begin(), s.end());
      return;
    }
    case Format::U4: {
      const auto& v = std::get<std::vector<uint32_t>>(item.storage());
      write_header(out, fmt, v.size() * 4);
      for (auto x : v) put_scalar_be(out, x);
      return;
    }
    // ... one case per format
  }
}

write_header picks the smallest length-byte-count and emits the format byte + length bytes. put_scalar_be is the templated big-endian writer using std::bit_cast for floats and std::make_unsigned_t for integers (chapter 10).

encode(item) is a thin wrapper:

std::vector<uint8_t> encode(const Item& item) {
  std::vector<uint8_t> out;
  encode_into(item, out);
  return out;
}

---

decode_at — the recursive decoder

Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos);

Mirror image:

Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos) {
  // 1. Format byte + length bytes.
  if (pos >= len) throw CodecError("truncated");
  const uint8_t  fb   = data[pos++];
  const Format   fmt  = static_cast<Format>(fb >> 2);
  const int      nlen = fb & 0x03;
  if (pos + nlen > len) throw CodecError("truncated length bytes");
  std::size_t length = 0;
  for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++];

  // 2. List recursion.
  if (fmt == Format::List) {
    Item::List children;
    children.reserve(length);
    for (std::size_t i = 0; i < length; ++i)
      children.push_back(decode_at(data, len, pos));
    return Item::list(std::move(children));
  }

  // 3. Scalar/array: dispatch on element size + signedness/floatness.
  if (pos + length > len) throw CodecError("truncated body");
  const uint8_t* body = data + pos;
  pos += length;
  switch (fmt) {
    case Format::ASCII: return Item::ascii(std::string((const char*)body, length));
    case Format::U4:    return Item::u4(read_array<uint32_t>(body, length));
    // ... one case per format
  }
  throw CodecError("unknown format code");
}

Item decode(const std::vector<uint8_t>& bytes) {
  std::size_t pos = 0;
  Item it = decode_at(bytes.data(), bytes.size(), pos);
  if (pos != bytes.size()) throw CodecError("trailing bytes");
  return it;
}

The _at variant is useful when an outer protocol carries a SECS-II item embedded in a larger frame — the caller passes the buffer and a position, and gets back the item plus the new position.

Bounds checks throw CodecError at every step — a CodecError on the receive side closes the connection (chapter 11's S9F7 path).

---

The Message wrapper

// include/secsgem/secs2/message.hpp
class Message {
 public:
  uint8_t  stream() const;
  uint8_t  function() const;
  bool     w_bit() const;
  uint32_t system_bytes() const;
  const Item& body() const;
  std::vector<uint8_t> body_bytes() const;  // encoded body
};

A Message is just a small struct: stream + function + W-bit + system_bytes + body Item. No encoder lives here — encoding is done by secs2::encode(message.body()) when the transport layer serializes a frame. The Message exists so the Router can dispatch on (stream, function) without re-decoding bytes.

---

SML — the human-readable form

to_sml(item) walks the Item recursively and emits SML:

// src/secs2/sml.cpp — sketch
std::string to_sml(const Item& item) {
  switch (item.format()) {
    case Format::List: {
      std::string s = "<L[" + std::to_string(item.size()) + "]";
      for (const auto& child : item.as_list()) {
        s += ' ' + to_sml(child);
      }
      s += '>';
      return s;
    }
    case Format::ASCII: return "A \"" + escape(item.as_ascii()) + "\"";
    case Format::U4: {
      const auto& v = std::get<std::vector<uint32_t>>(item.storage());
      std::string s = "U4";
      if (v.size() > 1) s += "[" + std::to_string(v.size()) + "]";
      for (auto x : v) s += " " + std::to_string(x);
      return s;
    }
    // ... per format
  }
}

try_parse_sml(text) is the inverse — a hand-written recursive- descent parser that returns std::optional<Item>. Returns nullopt on any parse error (no exceptions; this is what libFuzzer feeds garbage into and expects it not to crash).

Tests: tests/test_sml.cpp (10 cases — every format round-trips through to_sml → try_parse_sml → identical Item).

Why SML doesn't round-trip bytes

A subtle point: decode(encode(item)) round-trips exactly, but try_parse_sml(to_sml(item)) also round-trips the Item — except encoding the round-tripped Item may produce different bytes than the original. Why?

• The original might use a 2-byte length encoding; the round-tripped Item is a fresh Item and the encoder will pick the smallest length encoding (1 byte).
• SML doesn't preserve "which list-length encoding the encoder chose."

If you need bit-exact round-trip of bytes, use decode(encode). For semantic round-trip of values, use SML.

---

Testing — every layer in isolation

Layer	Test file	Cases	Focus
Item factories	tests/test_secs2.cpp	14	Construction, equality, format dispatch.
Codec	tests/test_e5_kat.cpp	19	Known-answer tests — bit-exact bytes per SEMI E5 §9.
Codec	tests/test_secs2.cpp	(overlap)	encode/decode round-trip + truncation rejection.
Identifier wildcards	tests/test_identifier_wildcards.cpp	6	U1/U2/U4/U8 leniency for ID fields.
SML	tests/test_sml.cpp	10	to_sml + try_parse_sml round-trip.
Catalog	tests/test_messages.cpp	82	Every named SxFy builder + parser round-trip.
Random/structural	tests/test_fuzz.cpp	8	Random bytes, truncation, oversize lengths, nested.
libFuzzer	apps/fuzz_secs2_decode.cpp	(CI)	200 k+ random inputs per minute, ASan + UBSan clean.
libFuzzer	apps/fuzz_sml_parse.cpp	(CI)	1.4 M+ random SML strings per minute, ASan + UBSan.

The codec alone has 139 test cases / 196+ assertions for E5 KAT. This is intentional: every other layer trusts the codec is correct. If it isn't, nothing above works.

---

Where to go next

You've now seen the codec and SML implementation in detail. Next chapter covers the dispatch layer that sits between the transport (which delivers raw Messages) and the stores (which hold state): gem::Router, the state-machine wiring, and the generated builder/parser glue from the message catalog.

Next: → 35 State machines and dispatch