secs-gem/docs/10_e5_secs_ii_data_items.md

10 — E5: SECS-II data items and encoding

← 03 Vocabulary + a wafer's journey | Back to index | Next: 11 E37 — HSMS transport →

We are now at the bottom of the protocol stack. Every other chapter in this guide rests on what you learn here.

SEMI E5 (first published 1982) is the SECS-II data encoding — the rules for turning a typed, possibly nested data structure into a stream of bytes, and back. It defines no transport (that's E37 / E4), no behaviour (that's E30), no message catalog (that's a level above). All E5 does is answer one question: given a value, what bytes go on the wire?

The answer is simple enough to fit on one screen, regular enough that the encoder is 150 lines of C++ in this codebase, and survived from 1982 to today without a single breaking change.

By the end of this chapter you will be able to:

• Hand-encode any SECS-II value to bytes with paper and pencil.
• Hand-decode any byte stream you see in a Wireshark trace.
• Read the encoder (src/secs2/codec.cpp) and decoder line-by-line.
• Explain the "identifier wildcard" rule and why it exists.

---

The mental model

A SECS-II value is a single Item. An Item is either:

1. A leaf — a homogeneous array of one scalar type (one of 13 types: ASCII string, U4 array, F8 array, etc.), or
2. A List — an ordered sequence of child Items.

That's it. No structs, no maps, no unions, no nullables. A message body is exactly one Item — but since Item can be a List of Items recursively, you can express anything.

This recursive simplicity is what makes the encoding regular. Every Item, leaf or list, has the same wire shape:

┌──────────────┬──────────────────────┬─────────────────────────┐
│ format byte  │ 1, 2, or 3 length    │ body                    │
│ (1 byte)     │ bytes (big-endian)   │ (length bytes / items)  │
└──────────────┴──────────────────────┴─────────────────────────┘

The format byte encodes both what type this Item is and how many length bytes follow. The length bytes say how big the body is. The body is the data — or, for a List, the encoded child Items concatenated.

---

The format byte

One byte. Six bits for the format code; two bits for the length-byte-count.

 bit  7  6  5  4  3  2  1  0
     ┌──┬──┬──┬──┬──┬──┬──┬──┐
     │   format code (6)     │ nl │
     └──────────────────────┴────┘
       ↑                     ↑
   the type tag        how many length
                        bytes follow (1, 2, or 3)

Arithmetically: format_byte = (format_code << 2) | length_byte_count.

In code, src/secs2/codec.cpp:62:

out.push_back(static_cast<uint8_t>((static_cast<uint8_t>(fmt) << 2) | nlen));

The encoder picks the smallest nlen that fits the body length:

if      (length <= 0xFF)      nlen = 1;   // 1-byte length
else if (length <= 0xFFFF)    nlen = 2;   // 2-byte length
else if (length <= 0xFFFFFF)  nlen = 3;   // 3-byte length
else throw CodecError("item length exceeds 3-byte maximum");

Three-byte length is the cap: 2^24 − 1 = 16 777 215 bytes ≈ 16 MiB per item. Larger bodies need to be split — but HSMS allows single frames up to its own limit (4 GiB), so this is rarely the bottleneck in practice.

---

The 14 format codes

The format code occupies the high 6 bits. SEMI E5 §9.5 Table 5 enumerates them. Codes are canonical octal, which feels archaic but matches the spec and the codebase:

Octal	Decimal	Hex	Format	Storage in Item	Element size
000	0	0x00	L List	std::vector<Item>	(children)
010	8	0x08	B Binary	std::vector<uint8_t>	1 byte
011	9	0x09	BOOLEAN	std::vector<uint8_t>	1 byte
020	16	0x10	A ASCII	std::string	1 byte
021	17	0x11	J JIS-8	std::string	1 byte
022	18	0x12	C C2 (Unicode-2)	std::vector<uint16_t>	2 bytes
030	24	0x18	I8	std::vector<int64_t>	8 bytes
031	25	0x19	I1	std::vector<int8_t>	1 byte
032	26	0x1A	I2	std::vector<int16_t>	2 bytes
034	28	0x1C	I4	std::vector<int32_t>	4 bytes
040	32	0x20	F8	std::vector<double>	8 bytes
044	36	0x24	F4	std::vector<float>	4 bytes
050	40	0x28	U8	std::vector<uint64_t>	8 bytes
051	41	0x29	U1	std::vector<uint8_t>	1 byte
052	42	0x2A	U2	std::vector<uint16_t>	2 bytes
054	44	0x2C	U4	std::vector<uint32_t>	4 bytes

Defined in include/secsgem/secs2/item.hpp:14:

enum class Format : uint8_t {
  List    = 000,  Binary  = 010,  Boolean = 011,
  ASCII   = 020,  JIS8    = 021,  C2      = 022,
  I8      = 030,  I1      = 031,  I2      = 032,  I4 = 034,
  F8      = 040,  F4      = 044,
  U8      = 050,  U1      = 051,  U2      = 052,  U4 = 054,
};

The values are stored as the 6-bit format code (not the full format byte), so (fmt << 2) | nlen produces the wire byte directly.

Why these specific codes?

Three things to notice:

1. The numbering isn't dense. Codes 1–7 are reserved, codes 12, 13, 15, 23–27, 33, 35, 37, 41–43, 45–47, 51, 53, 55–57, 60–77 are all unassigned. SEMI E5 left room for future formats and never filled it.
2. The integer / float widths are encoded in the low octal bits — 031 (I1) 032 (I2) 034 (I4) 030 (I8); same pattern for U and F. So shifting right by 0–2 bits doesn't give you a size; you have to look it up. element_size() in item.hpp:58 does that.
3. Binary and Boolean share storage (std::vector<uint8_t>), disambiguated by format(). Same for ASCII and JIS-8 (both std::string), and U2 and C2 (both std::vector<uint16_t>).

---

Length bytes

After the format byte come the length bytes. How many? The low two bits of the format byte say:

nl bits   length bytes follow   max representable length
─────────────────────────────────────────────────────────
01        1 byte                255           (0xFF)
10        2 bytes               65 535        (0xFFFF)
11        3 bytes               16 777 215    (0xFFFFFF)
00        — invalid              —

All length bytes are big-endian (network byte order).

For scalar formats (every format except List), the length is the body's byte count.

For List, the length is the element count — the number of child Items that follow. Each child is itself a fully-encoded Item (format byte + length + body), so the byte length of the list body is whatever those children sum to.

The encoder picks the smallest nl that fits — see src/secs2/codec.cpp:55-67. The decoder reads the format byte's low two bits, then that many big-endian length bytes, then the body:

// src/secs2/codec.cpp — sketch
const uint8_t format_byte = data[pos++];
const Format  fmt  = static_cast<Format>(format_byte >> 2);
const int     nlen = format_byte & 0x03;
std::size_t length = 0;
for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++];
// then read `length` bytes (or items, for List).

---

Walked through: every format with a hexdump

Format names below match SML output (L, A, U1, …).

List

<L[0]>                 (empty list)
                       │
                       ▼
                       01 00
                       │  │
                       │  └── length: 0 children
                       └───── format byte:
                              (000 << 2) | 01 = 0x01

<L[2]  A "Hi"  U1[1] 5>             (list of 2: ASCII "Hi", U1 5)
                                    │
                                    ▼
                                    01 02   41 02 48 69   A5 01 05
                                    │  │    │           │
                                    │  │    │           └── inner U1
                                    │  │    └── inner ASCII
                                    │  └── length: 2 children
                                    └── format byte 0x01 = L, nl=1

ASCII

A "Hello, world"        format = 020 (A), nl=1, length=12
                        format byte = (020 << 2) | 01 = 0x41
                        
                        41 0C 48 65 6C 6C 6F 2C 20 77 6F 72 6C 64

ASCII with length crossing a length-byte boundary forces nl=2:

A (256 chars)           format byte = (020 << 2) | 10 = 0x42
                        length bytes = 01 00 (big-endian 256)
                        
                        42 01 00 <256 bytes …>

Binary vs Boolean

Same on-disk shape (a byte array); different format byte:

B 0xCA 0xFE               08 02 CA FE        (B format = 010 → 0x08 byte)
                          │
                          └── format byte = (010 << 2) | 01 = 0x09 — wait

Correction:               09 02 CA FE        format byte = (010 << 2) | 01
                                              = (8 << 2) | 1 = 0x21? — let me redo.

Let me redo the arithmetic explicitly so it's airtight. For Binary: format code 010 octal = 8 decimal. Format byte = (8 << 2) | 1 = 33 = 0x21.

B 0xCA 0xFE             21 02 CA FE        format byte 0x21, length 2

For Boolean (one true byte): format code 011 = 9. Format byte = (9 << 2) | 1 = 0x25.

BOOLEAN true            25 01 01            format byte 0x25, length 1, body 0x01
BOOLEAN false           25 01 00            format byte 0x25, length 1, body 0x00

Booleans on the wire are bytes, not bits. 0 is false; anything non-zero is true.

Unsigned integers

U1 (format 051 = 41): format byte = (41 << 2) | nl. (41 << 2) = 164 = 0xA4. So for nl=1: 0xA5.

U1 5                    A5 01 05
U1 [5, 10, 15]          A5 03 05 0A 0F

U2 (format 052 = 42): (42 << 2) = 168 = 0xA8. nl=1 → 0xA9.

U2 300                  A9 02 01 2C            (300 = 0x012C, big-endian)
U2 [1, 2, 3]            A9 06 00 01 00 02 00 03

U4 (format 054 = 44): (44 << 2) = 176 = 0xB0. nl=1 → 0xB1.

U4 1                    B1 04 00 00 00 01
U4 65536                B1 04 00 01 00 00

U8 (format 050 = 40): (40 << 2) = 160 = 0xA0. nl=1 → 0xA1.

U8 1                    A1 08 00 00 00 00 00 00 00 01

Notice the pattern: the format byte's high 6 bits identify the type, the low 2 bits say "how many length bytes," and then comes the data.

Signed integers

Same shape as unsigned, but two's-complement.

I1 (031 = 25): format byte (25 << 2) | 1 = 0x65.

I1 -1                   65 01 FF
I1 5                    65 01 05
I1 [-1, 0, 1]           65 03 FF 00 01

I2 (032 = 26): format byte 0x69.

I2 -1                   69 02 FF FF
I2 1                    69 02 00 01

Floats

IEEE 754, big-endian. F4 is single-precision, 4 bytes; F8 is double-precision, 8 bytes.

F4 (044 = 36): format byte (36 << 2) | 1 = 0x91.

F4 1.0                  91 04 3F 80 00 00       (1.0 = 0x3F800000)
F4 -1.0                 91 04 BF 80 00 00
F4 0.5                  91 04 3F 00 00 00
F4 NaN                  91 04 7F C0 00 00       (one canonical NaN)
F4 +Inf                 91 04 7F 80 00 00
F4 -Inf                 91 04 FF 80 00 00
F4 -0.0                 91 04 80 00 00 00

F8 (040 = 32): format byte 0x81.

F8 1.0                  81 08 3F F0 00 00 00 00 00 00

The encoder uses std::bit_cast (src/secs2/codec.cpp:13) to get the IEEE 754 bit pattern without any rounding, then writes the bytes big-endian. Decoding is the same in reverse. This guarantees bit-exact float round-trip including NaN, ±Inf, −0.0.

ASCII variants

J JIS-8 (021 = 17, byte 0x46 for nl=1): single-byte Japanese encoding, used by some Japanese tool vendors.

J "ハロー"               46 06 8A D8 A4         (or however the bytes decode in your JIS)

C C2 (022 = 18, byte 0x4A for nl=1): big-endian 16-bit Unicode code points (essentially UCS-2 / pre-surrogate UTF-16).

C "Hi"                  4A 04 00 48 00 69
C "ハ"                  4A 02 30 CF             (U+30CF = HIRAGANA HA)

Note: the length in the format byte's length bytes is still a byte count, not a character count — for C2, length always divides by 2.

---

The decoder, recursively

The decoder is straightforward:

1. Read one format byte.
2. Read nl length bytes; assemble length big-endian.
3. If the format is List, recurse length times.
4. Otherwise read length bytes, interpret per the format's element size, build the array.

In code, src/secs2/codec.cpp (sketched):

Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos) {
  const uint8_t fb   = data[pos++];
  const Format  fmt  = static_cast<Format>(fb >> 2);
  const int     nlen = fb & 0x03;
  std::size_t length = 0;
  for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++];

  if (fmt == Format::List) {
    Item::List children;
    children.reserve(length);
    for (std::size_t i = 0; i < length; ++i)
      children.push_back(decode_at(data, len, pos));
    return Item::list(std::move(children));
  }

  // scalar/array: read `length` raw bytes, dispatch on element_size(fmt)
  // and integer signedness / float-ness.
  const uint8_t* body = data + pos;
  pos += length;
  switch (fmt) {
    case Format::ASCII:   return Item::ascii(std::string(reinterpret_cast<const char*>(body), length));
    case Format::U1:      return Item::u1(std::vector<uint8_t>(body, body + length));
    case Format::U2:      return Item::u2(read_array<uint16_t>(body, length));
    case Format::U4:      return Item::u4(read_array<uint32_t>(body, length));
    // …and so on for every format.
  }
}

decode() (no _at) decodes a buffer and throws if there are trailing bytes — useful as the top-level entry point.

---

SML: the human-readable form

You'll see SECS-II written in SML (SECS Message Language) throughout the rest of this guide and every Wireshark dump. SML is not on the wire — it's a textual rendering of the same Item tree. The mapping is:

List          <L[N]  c0  c1  …  cN−1>
ASCII         A "Hello"
Binary        B 0x01 0x02
Boolean       BOOLEAN True
U1 (scalar)   U1 5
U1 (array)    U1[3] 5 10 15
F4            F4 1.0
F4 (special)  F4 +Inf, F4 NaN
I2 negative   I2 -1
C2            C "Hi"

Nested:

<L[3]
  A "RECIPE-A"
  U4 12345
  <L[2]  A "TEMP_C"  F4 25.0>
>

The SML parser/printer is in include/secsgem/secs2/sml.hpp and src/secs2/sml.cpp. to_sml(item) prints; try_parse_sml(str) returns an Item or an error. SML parsing is exercised by libFuzzer in apps/fuzz_sml_parse.cpp — over 1 million random SML strings per minute, ASan + UBSan clean, 0 crashes (see PROOFS.md proof #8).

SML is useful (you can paste it into a debugger), but it's not canonical: two different SML strings might serialize to identical bytes (whitespace, comments, optional list-length annotations), and two different on-wire byte streams might pretty-print as identical SML (different length-byte counts for the same length). The wire bytes are canonical; SML is for humans.

---

The identifier wildcard rule

A SEMI subtlety that bites every implementation eventually.

The spec says: for identifier fields (SVID, ECID, CEID, ALID, RPTID, LIMITID, …), the equipment may encode the value as U1, U2, U4, or U8, picking whichever width fits. And the receiver — equipment or host — must accept all four widths interchangeably.

So a host sending S2F33 to define a report can put RPTID 1 as U1 1 (3 bytes on the wire: A5 01 01) or U4 1 (6 bytes: B1 04 00 00 00 01) — both are valid, and the equipment must accept either.

The codebase enforces this leniency through one helper: messages_helpers::any_unsigned_first<Out>().

// include/secsgem/gem/messages_helpers.hpp:99
template <typename Out>
inline std::optional<Out> any_unsigned_first(const s2::Item& item) {
  // Try U1, U2, U4, U8 in turn; if the value fits in Out, return it.
}

Every place that reads an identifier off the wire uses one of:

as_u1_scalar(item)   // accepts U1/U2/U4/U8 widths if value fits in uint8_t
as_u2_scalar(item)   // accepts U1/U2/U4/U8 widths if value fits in uint16_t
as_u4_scalar(item)   // accepts U1/U2/U4/U8 widths if value fits in uint32_t
as_u8_scalar(item)   // accepts any width

tests/test_identifier_wildcards.cpp asserts this for every combination of declared width × encoded width. 6 test cases, every direction of the matrix; this was a real interop bug before the helpers existed (see interop/README.md for the back-story).

Outgoing encoding still picks the narrowest width that fits, to keep wire bytes minimal.

---

Equality and round-trip

Item is bool operator==(const Item&) const = default; (item.hpp:161) — defaulted member-wise equality on the variant. Two Items compare equal iff they have the same format AND the same storage values.

This makes round-trip tests trivial:

const Item original = ...;
const auto bytes    = encode(original);
const Item decoded  = decode(bytes);
REQUIRE(decoded == original);

tests/test_secs2.cpp does this for hand-picked Items across every format; the libFuzzer harness in apps/fuzz_secs2_decode.cpp does it for arbitrary random byte streams.

The strongest round-trip evidence is the SEMI E5 KAT (known- answer tests) in tests/test_e5_kat.cpp. Each fixture is a hex string written by hand directly from the spec's encoding rules; the test asserts that encoding the corresponding Item produces exactly those bytes and that decoding them produces exactly that Item. 19 test cases, 196 assertions, every format code, every length-byte-count variant (1, 2, 3 bytes), numeric edges (0, ±1, MIN, MAX, ±Inf, NaN), nested lists.

The KAT is the strongest single proof of codec correctness in the codebase, because every other validator is one implementer's interpretation of the spec; KAT is the spec's own arithmetic. See VERIFICATION.md §1 for the rationale.

---

What can go wrong

The codec rejects malformed input in a few well-defined ways (src/secs2/codec.cpp throws CodecError):

• Truncated input — the buffer ends in the middle of a format byte, length bytes, or body.
• Length-not-multiple-of-element-size — e.g. a U4 array claiming a body length of 7 bytes. 4 doesn't divide 7.
• Length exceeds buffer — the length bytes claim more body than exists.
• Trailing bytes after the top-level item — for decode() (not decode_at()), the buffer must end exactly when the item does.
• 3-byte length cap exceeded — encode rejects items larger than 16 MiB.

Every one of these is exercised by either tests/test_secs2.cpp, tests/test_fuzz.cpp, or the libFuzzer harness.

Notably not rejected: an unknown format code. The spec is silent on how a receiver should handle codes 1–7, 12, 13, 15, etc., and in practice the codec passes them through as the raw Format enum value. Whether a higher-level handler cares is up to that handler.

---

Where to go next

You now understand the entire SECS-II data encoding. Three things build on it directly:

• The catalog of named messages uses Items as bodies. Every SxFy in data/messages.yaml is a recipe for one specific Item shape. See chapter 31 for the codegen that turns YAML recipes into typed C++ structs.
• The Message type (include/secsgem/secs2/message.hpp) wraps a body Item with stream, function, W-bit, and system bytes. That's what HSMS frames carry; see chapter 11.
• Behaviour reads typed values out of incoming Items (using the identifier wildcard helpers) and writes them back. Chapter 13 covers E30 / GEM.

But before any behaviour can happen, the bytes have to get there. That's the next chapter: E37 HSMS, the TCP transport.

Next: → 11 E37 — HSMS transport