# 10 — E5: SECS-II data items and encoding ← [03 Vocabulary + a wafer's journey](03_vocabulary_and_a_wafers_journey.md) | [Back to index](00_index.md) | Next: [11 E37 — HSMS transport](11_e37_hsms.md) → We are now at the bottom of the protocol stack. Every other chapter in this guide rests on what you learn here. **SEMI E5** (first published 1982) is the **SECS-II data encoding** — the rules for turning a typed, possibly nested data structure into a stream of bytes, and back. It defines no transport (that's E37 / E4), no behaviour (that's E30), no message catalog (that's a level above). All E5 does is answer one question: *given a value, what bytes go on the wire?* The answer is simple enough to fit on one screen, regular enough that the encoder is 150 lines of C++ in this codebase, and survived from 1982 to today without a single breaking change. By the end of this chapter you will be able to: - Hand-encode any SECS-II value to bytes with paper and pencil. - Hand-decode any byte stream you see in a Wireshark trace. - Read the encoder ([`src/secs2/codec.cpp`](../src/secs2/codec.cpp)) and decoder line-by-line. - Explain the "identifier wildcard" rule and why it exists. --- ## The mental model A SECS-II value is a single **Item**. An Item is either: 1. A **leaf** — a homogeneous array of one scalar type (one of 13 types: ASCII string, U4 array, F8 array, etc.), or 2. A **List** — an ordered sequence of child Items. That's it. No structs, no maps, no unions, no nullables. A message body is exactly one Item — but since Item can be a List of Items recursively, you can express anything. This recursive simplicity is what makes the encoding regular. Every Item, leaf or list, has the same wire shape: ``` ┌──────────────┬──────────────────────┬─────────────────────────┐ │ format byte │ 1, 2, or 3 length │ body │ │ (1 byte) │ bytes (big-endian) │ (length bytes / items) │ └──────────────┴──────────────────────┴─────────────────────────┘ ``` The format byte encodes both *what type this Item is* and *how many length bytes follow*. The length bytes say how big the body is. The body is the data — or, for a List, the encoded child Items concatenated. --- ## The format byte One byte. Six bits for the format code; two bits for the length-byte-count. ``` bit 7 6 5 4 3 2 1 0 ┌──┬──┬──┬──┬──┬──┬──┬──┐ │ format code (6) │ nl │ └──────────────────────┴────┘ ↑ ↑ the type tag how many length bytes follow (1, 2, or 3) ``` Arithmetically: `format_byte = (format_code << 2) | length_byte_count`. In code, [`src/secs2/codec.cpp:62`](../src/secs2/codec.cpp): ```cpp out.push_back(static_cast((static_cast(fmt) << 2) | nlen)); ``` The encoder picks the smallest `nlen` that fits the body length: ```cpp if (length <= 0xFF) nlen = 1; // 1-byte length else if (length <= 0xFFFF) nlen = 2; // 2-byte length else if (length <= 0xFFFFFF) nlen = 3; // 3-byte length else throw CodecError("item length exceeds 3-byte maximum"); ``` Three-byte length is the cap: **2^24 − 1 = 16 777 215 bytes ≈ 16 MiB per item**. Larger bodies need to be split — but HSMS allows single frames up to its own limit (4 GiB), so this is rarely the bottleneck in practice. --- ## The 14 format codes The format code occupies the high 6 bits. SEMI E5 §9.5 Table 5 enumerates them. Codes are canonical *octal*, which feels archaic but matches the spec and the codebase: | Octal | Decimal | Hex | Format | Storage in `Item` | Element size | |-------|---------|------|--------------|-------------------------|---------------| | 000 | 0 | 0x00 | **L** List | `std::vector` | (children) | | 010 | 8 | 0x08 | **B** Binary | `std::vector` | 1 byte | | 011 | 9 | 0x09 | **BOOLEAN** | `std::vector` | 1 byte | | 020 | 16 | 0x10 | **A** ASCII | `std::string` | 1 byte | | 021 | 17 | 0x11 | **J** JIS-8 | `std::string` | 1 byte | | 022 | 18 | 0x12 | **C** C2 (Unicode-2) | `std::vector` | 2 bytes | | 030 | 24 | 0x18 | **I8** | `std::vector` | 8 bytes | | 031 | 25 | 0x19 | **I1** | `std::vector` | 1 byte | | 032 | 26 | 0x1A | **I2** | `std::vector` | 2 bytes | | 034 | 28 | 0x1C | **I4** | `std::vector` | 4 bytes | | 040 | 32 | 0x20 | **F8** | `std::vector` | 8 bytes | | 044 | 36 | 0x24 | **F4** | `std::vector` | 4 bytes | | 050 | 40 | 0x28 | **U8** | `std::vector` | 8 bytes | | 051 | 41 | 0x29 | **U1** | `std::vector` | 1 byte | | 052 | 42 | 0x2A | **U2** | `std::vector` | 2 bytes | | 054 | 44 | 0x2C | **U4** | `std::vector` | 4 bytes | Defined in [`include/secsgem/secs2/item.hpp:14`](../include/secsgem/secs2/item.hpp): ```cpp enum class Format : uint8_t { List = 000, Binary = 010, Boolean = 011, ASCII = 020, JIS8 = 021, C2 = 022, I8 = 030, I1 = 031, I2 = 032, I4 = 034, F8 = 040, F4 = 044, U8 = 050, U1 = 051, U2 = 052, U4 = 054, }; ``` The values are stored as the 6-bit format code (not the full format byte), so `(fmt << 2) | nlen` produces the wire byte directly. ### Why these specific codes? Three things to notice: 1. **The numbering isn't dense.** Codes 1–7 are reserved, codes 12, 13, 15, 23–27, 33, 35, 37, 41–43, 45–47, 51, 53, 55–57, 60–77 are all unassigned. SEMI E5 left room for future formats and never filled it. 2. **The integer / float widths are encoded in the low octal bits** — `031` (I1) `032` (I2) `034` (I4) `030` (I8); same pattern for U and F. So shifting right by 0–2 bits doesn't give you a size; you have to look it up. `element_size()` in [`item.hpp:58`](../include/secsgem/secs2/item.hpp) does that. 3. **Binary and Boolean share storage** (`std::vector`), disambiguated by `format()`. Same for ASCII and JIS-8 (both `std::string`), and U2 and C2 (both `std::vector`). --- ## Length bytes After the format byte come the length bytes. How many? The low two bits of the format byte say: ``` nl bits length bytes follow max representable length ───────────────────────────────────────────────────────── 01 1 byte 255 (0xFF) 10 2 bytes 65 535 (0xFFFF) 11 3 bytes 16 777 215 (0xFFFFFF) 00 — invalid — ``` All length bytes are **big-endian** (network byte order). For **scalar formats** (every format except `List`), the length is the body's **byte count**. For **List**, the length is the **element count** — the number of child Items that follow. Each child is itself a fully-encoded Item (format byte + length + body), so the *byte* length of the list body is whatever those children sum to. The encoder picks the smallest nl that fits — see [`src/secs2/codec.cpp:55-67`](../src/secs2/codec.cpp). The decoder reads the format byte's low two bits, then that many big-endian length bytes, then the body: ```cpp // src/secs2/codec.cpp — sketch const uint8_t format_byte = data[pos++]; const Format fmt = static_cast(format_byte >> 2); const int nlen = format_byte & 0x03; std::size_t length = 0; for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++]; // then read `length` bytes (or items, for List). ``` --- ## Walked through: every format with a hexdump Format names below match SML output (`L`, `A`, `U1`, …). ### List ``` (empty list) │ ▼ 01 00 │ │ │ └── length: 0 children └───── format byte: (000 << 2) | 01 = 0x01 ``` ``` (list of 2: ASCII "Hi", U1 5) │ ▼ 01 02 41 02 48 69 A5 01 05 │ │ │ │ │ │ │ └── inner U1 │ │ └── inner ASCII │ └── length: 2 children └── format byte 0x01 = L, nl=1 ``` ### ASCII ``` A "Hello, world" format = 020 (A), nl=1, length=12 format byte = (020 << 2) | 01 = 0x41 41 0C 48 65 6C 6C 6F 2C 20 77 6F 72 6C 64 ``` ASCII with length crossing a length-byte boundary forces `nl=2`: ``` A (256 chars) format byte = (020 << 2) | 10 = 0x42 length bytes = 01 00 (big-endian 256) 42 01 00 <256 bytes …> ``` ### Binary vs Boolean Same on-disk shape (a byte array); different format byte: ``` B 0xCA 0xFE 08 02 CA FE (B format = 010 → 0x08 byte) │ └── format byte = (010 << 2) | 01 = 0x09 — wait Correction: 09 02 CA FE format byte = (010 << 2) | 01 = (8 << 2) | 1 = 0x21? — let me redo. ``` Let me redo the arithmetic explicitly so it's airtight. For **Binary**: format code `010` octal = `8` decimal. Format byte = `(8 << 2) | 1 = 33 = 0x21`. ``` B 0xCA 0xFE 21 02 CA FE format byte 0x21, length 2 ``` For **Boolean** (one true byte): format code `011` = `9`. Format byte = `(9 << 2) | 1 = 0x25`. ``` BOOLEAN true 25 01 01 format byte 0x25, length 1, body 0x01 BOOLEAN false 25 01 00 format byte 0x25, length 1, body 0x00 ``` Booleans on the wire are *bytes*, not bits. 0 is false; anything non-zero is true. ### Unsigned integers `U1` (format `051` = `41`): format byte = `(41 << 2) | nl`. `(41 << 2) = 164 = 0xA4`. So for `nl=1`: `0xA5`. ``` U1 5 A5 01 05 U1 [5, 10, 15] A5 03 05 0A 0F ``` `U2` (format `052` = `42`): `(42 << 2) = 168 = 0xA8`. `nl=1` → `0xA9`. ``` U2 300 A9 02 01 2C (300 = 0x012C, big-endian) U2 [1, 2, 3] A9 06 00 01 00 02 00 03 ``` `U4` (format `054` = `44`): `(44 << 2) = 176 = 0xB0`. `nl=1` → `0xB1`. ``` U4 1 B1 04 00 00 00 01 U4 65536 B1 04 00 01 00 00 ``` `U8` (format `050` = `40`): `(40 << 2) = 160 = 0xA0`. `nl=1` → `0xA1`. ``` U8 1 A1 08 00 00 00 00 00 00 00 01 ``` Notice the pattern: the format byte's high 6 bits identify the type, the low 2 bits say "how many length bytes," and then comes the data. ### Signed integers Same shape as unsigned, but two's-complement. `I1` (`031` = 25): format byte `(25 << 2) | 1 = 0x65`. ``` I1 -1 65 01 FF I1 5 65 01 05 I1 [-1, 0, 1] 65 03 FF 00 01 ``` `I2` (`032` = 26): format byte `0x69`. ``` I2 -1 69 02 FF FF I2 1 69 02 00 01 ``` ### Floats IEEE 754, big-endian. `F4` is single-precision, 4 bytes; `F8` is double-precision, 8 bytes. `F4` (`044` = 36): format byte `(36 << 2) | 1 = 0x91`. ``` F4 1.0 91 04 3F 80 00 00 (1.0 = 0x3F800000) F4 -1.0 91 04 BF 80 00 00 F4 0.5 91 04 3F 00 00 00 F4 NaN 91 04 7F C0 00 00 (one canonical NaN) F4 +Inf 91 04 7F 80 00 00 F4 -Inf 91 04 FF 80 00 00 F4 -0.0 91 04 80 00 00 00 ``` `F8` (`040` = 32): format byte `0x81`. ``` F8 1.0 81 08 3F F0 00 00 00 00 00 00 ``` The encoder uses `std::bit_cast` ([`src/secs2/codec.cpp:13`](../src/secs2/codec.cpp)) to get the IEEE 754 bit pattern without any rounding, then writes the bytes big-endian. Decoding is the same in reverse. This guarantees bit-exact float round-trip including NaN, ±Inf, −0.0. ### ASCII variants `J` JIS-8 (`021` = 17, byte 0x46 for nl=1): single-byte Japanese encoding, used by some Japanese tool vendors. ``` J "ハロー" 46 06 8A D8 A4 (or however the bytes decode in your JIS) ``` `C` C2 (`022` = 18, byte 0x4A for nl=1): big-endian 16-bit Unicode code points (essentially UCS-2 / pre-surrogate UTF-16). ``` C "Hi" 4A 04 00 48 00 69 C "ハ" 4A 02 30 CF (U+30CF = HIRAGANA HA) ``` Note: the *length* in the format byte's length bytes is still a **byte count**, not a character count — for C2, length always divides by 2. --- ## The decoder, recursively The decoder is straightforward: 1. Read one format byte. 2. Read `nl` length bytes; assemble length big-endian. 3. If the format is List, recurse `length` times. 4. Otherwise read `length` bytes, interpret per the format's element size, build the array. In code, [`src/secs2/codec.cpp`](../src/secs2/codec.cpp) (sketched): ```cpp Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos) { const uint8_t fb = data[pos++]; const Format fmt = static_cast(fb >> 2); const int nlen = fb & 0x03; std::size_t length = 0; for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++]; if (fmt == Format::List) { Item::List children; children.reserve(length); for (std::size_t i = 0; i < length; ++i) children.push_back(decode_at(data, len, pos)); return Item::list(std::move(children)); } // scalar/array: read `length` raw bytes, dispatch on element_size(fmt) // and integer signedness / float-ness. const uint8_t* body = data + pos; pos += length; switch (fmt) { case Format::ASCII: return Item::ascii(std::string(reinterpret_cast(body), length)); case Format::U1: return Item::u1(std::vector(body, body + length)); case Format::U2: return Item::u2(read_array(body, length)); case Format::U4: return Item::u4(read_array(body, length)); // …and so on for every format. } } ``` `decode()` (no `_at`) decodes a buffer and throws if there are trailing bytes — useful as the top-level entry point. --- ## SML: the human-readable form You'll see SECS-II written in **SML** (SECS Message Language) throughout the rest of this guide and every Wireshark dump. SML is not on the wire — it's a textual rendering of the same Item tree. The mapping is: ``` List ASCII A "Hello" Binary B 0x01 0x02 Boolean BOOLEAN True U1 (scalar) U1 5 U1 (array) U1[3] 5 10 15 F4 F4 1.0 F4 (special) F4 +Inf, F4 NaN I2 negative I2 -1 C2 C "Hi" ``` Nested: ``` > ``` The SML parser/printer is in [`include/secsgem/secs2/sml.hpp`](../include/secsgem/secs2/sml.hpp) and [`src/secs2/sml.cpp`](../src/secs2/sml.cpp). `to_sml(item)` prints; `try_parse_sml(str)` returns an `Item` or an error. SML parsing is exercised by libFuzzer in [`apps/fuzz_sml_parse.cpp`](../apps/fuzz_sml_parse.cpp) — over 1 million random SML strings per minute, ASan + UBSan clean, 0 crashes (see [PROOFS.md](PROOFS.md) proof #8). SML is *useful* (you can paste it into a debugger), but it's not canonical: two different SML strings might serialize to identical bytes (whitespace, comments, optional list-length annotations), and two different on-wire byte streams might pretty-print as identical SML (different length-byte counts for the same length). The wire bytes are canonical; SML is for humans. --- ## The identifier wildcard rule A SEMI subtlety that bites every implementation eventually. **The spec says**: for identifier fields (SVID, ECID, CEID, ALID, RPTID, LIMITID, …), the equipment may encode the value as `U1`, `U2`, `U4`, *or* `U8`, picking whichever width fits. And the receiver — equipment or host — **must accept all four widths interchangeably**. So a host sending `S2F33` to define a report can put RPTID 1 as `U1 1` (3 bytes on the wire: `A5 01 01`) or `U4 1` (6 bytes: `B1 04 00 00 00 01`) — both are valid, and the equipment must accept either. The codebase enforces this leniency through one helper: [`messages_helpers::any_unsigned_first()`](../include/secsgem/gem/messages_helpers.hpp). ```cpp // include/secsgem/gem/messages_helpers.hpp:99 template inline std::optional any_unsigned_first(const s2::Item& item) { // Try U1, U2, U4, U8 in turn; if the value fits in Out, return it. } ``` Every place that reads an identifier off the wire uses one of: ```cpp as_u1_scalar(item) // accepts U1/U2/U4/U8 widths if value fits in uint8_t as_u2_scalar(item) // accepts U1/U2/U4/U8 widths if value fits in uint16_t as_u4_scalar(item) // accepts U1/U2/U4/U8 widths if value fits in uint32_t as_u8_scalar(item) // accepts any width ``` [`tests/test_identifier_wildcards.cpp`](../tests/test_identifier_wildcards.cpp) asserts this for every combination of declared width × encoded width. 6 test cases, every direction of the matrix; this was a real interop bug before the helpers existed (see [`interop/README.md`](../interop/README.md) for the back-story). Outgoing encoding still picks the *narrowest* width that fits, to keep wire bytes minimal. --- ## Equality and round-trip `Item` is `bool operator==(const Item&) const = default;` ([`item.hpp:161`](../include/secsgem/secs2/item.hpp)) — defaulted member-wise equality on the variant. Two Items compare equal iff they have the same format AND the same storage values. This makes round-trip tests trivial: ```cpp const Item original = ...; const auto bytes = encode(original); const Item decoded = decode(bytes); REQUIRE(decoded == original); ``` [`tests/test_secs2.cpp`](../tests/test_secs2.cpp) does this for hand-picked Items across every format; the libFuzzer harness in [`apps/fuzz_secs2_decode.cpp`](../apps/fuzz_secs2_decode.cpp) does it for arbitrary random byte streams. The strongest round-trip evidence is the **SEMI E5 KAT** (known- answer tests) in [`tests/test_e5_kat.cpp`](../tests/test_e5_kat.cpp). Each fixture is a hex string written by hand directly from the spec's encoding rules; the test asserts that encoding the corresponding Item produces *exactly those bytes* and that decoding them produces exactly that Item. 19 test cases, 196 assertions, every format code, every length-byte-count variant (1, 2, 3 bytes), numeric edges (0, ±1, MIN, MAX, ±Inf, NaN), nested lists. The KAT is the strongest single proof of codec correctness in the codebase, because every other validator is *one* implementer's interpretation of the spec; KAT is the spec's own arithmetic. See [VERIFICATION.md](VERIFICATION.md) §1 for the rationale. --- ## What can go wrong The codec rejects malformed input in a few well-defined ways ([`src/secs2/codec.cpp`](../src/secs2/codec.cpp) throws `CodecError`): - **Truncated input** — the buffer ends in the middle of a format byte, length bytes, or body. - **Length-not-multiple-of-element-size** — e.g. a U4 array claiming a body length of 7 bytes. 4 doesn't divide 7. - **Length exceeds buffer** — the length bytes claim more body than exists. - **Trailing bytes after the top-level item** — for `decode()` (not `decode_at()`), the buffer must end exactly when the item does. - **3-byte length cap exceeded** — encode rejects items larger than 16 MiB. Every one of these is exercised by either [`tests/test_secs2.cpp`](../tests/test_secs2.cpp), [`tests/test_fuzz.cpp`](../tests/test_fuzz.cpp), or the libFuzzer harness. Notably *not* rejected: an unknown format code. The spec is silent on how a receiver should handle codes 1–7, 12, 13, 15, etc., and in practice the codec passes them through as the raw `Format` enum value. Whether a higher-level handler cares is up to that handler. --- ## Where to go next You now understand the entire SECS-II data encoding. Three things build on it directly: - **The catalog of named messages** uses Items as bodies. Every SxFy in [`data/messages.yaml`](../data/messages.yaml) is a recipe for one specific Item shape. See chapter [31](31_spec_as_data_and_codegen.md) for the codegen that turns YAML recipes into typed C++ structs. - **The Message type** ([`include/secsgem/secs2/message.hpp`](../include/secsgem/secs2/message.hpp)) wraps a body Item with stream, function, W-bit, and system bytes. That's what HSMS frames carry; see chapter [11](11_e37_hsms.md). - **Behaviour** reads typed values out of incoming Items (using the identifier wildcard helpers) and writes them back. Chapter [13](13_e30_gem.md) covers E30 / GEM. But before any behaviour can happen, the bytes have to *get there*. That's the next chapter: **E37 HSMS**, the TCP transport. Next: [→ 11 E37 — HSMS transport](11_e37_hsms.md)