diff --git a/docs/10_e5_secs_ii_data_items.md b/docs/10_e5_secs_ii_data_items.md new file mode 100644 index 0000000..4b91298 --- /dev/null +++ b/docs/10_e5_secs_ii_data_items.md @@ -0,0 +1,611 @@ +# 10 — E5: SECS-II data items and encoding + +← [03 Vocabulary + a wafer's journey](03_vocabulary_and_a_wafers_journey.md) | [Back to index](00_index.md) | Next: [11 E37 — HSMS transport](11_e37_hsms.md) → + +We are now at the bottom of the protocol stack. Every other chapter +in this guide rests on what you learn here. + +**SEMI E5** (first published 1982) is the **SECS-II data encoding** — +the rules for turning a typed, possibly nested data structure into +a stream of bytes, and back. It defines no transport (that's E37 / +E4), no behaviour (that's E30), no message catalog (that's a level +above). All E5 does is answer one question: *given a value, what +bytes go on the wire?* + +The answer is simple enough to fit on one screen, regular enough that +the encoder is 150 lines of C++ in this codebase, and survived from +1982 to today without a single breaking change. + +By the end of this chapter you will be able to: + +- Hand-encode any SECS-II value to bytes with paper and pencil. +- Hand-decode any byte stream you see in a Wireshark trace. +- Read the encoder ([`src/secs2/codec.cpp`](../src/secs2/codec.cpp)) + and decoder line-by-line. +- Explain the "identifier wildcard" rule and why it exists. + +--- + +## The mental model + +A SECS-II value is a single **Item**. An Item is either: + +1. A **leaf** — a homogeneous array of one scalar type (one of 13 + types: ASCII string, U4 array, F8 array, etc.), or +2. A **List** — an ordered sequence of child Items. + +That's it. No structs, no maps, no unions, no nullables. A +message body is exactly one Item — but since Item can be a List of +Items recursively, you can express anything. + +This recursive simplicity is what makes the encoding regular. +Every Item, leaf or list, has the same wire shape: + +``` +┌──────────────┬──────────────────────┬─────────────────────────┐ +│ format byte │ 1, 2, or 3 length │ body │ +│ (1 byte) │ bytes (big-endian) │ (length bytes / items) │ +└──────────────┴──────────────────────┴─────────────────────────┘ +``` + +The format byte encodes both *what type this Item is* and *how many +length bytes follow*. The length bytes say how big the body is. +The body is the data — or, for a List, the encoded child Items +concatenated. + +--- + +## The format byte + +One byte. Six bits for the format code; two bits for the +length-byte-count. + +``` + bit 7 6 5 4 3 2 1 0 + ┌──┬──┬──┬──┬──┬──┬──┬──┐ + │ format code (6) │ nl │ + └──────────────────────┴────┘ + ↑ ↑ + the type tag how many length + bytes follow (1, 2, or 3) +``` + +Arithmetically: `format_byte = (format_code << 2) | length_byte_count`. + +In code, [`src/secs2/codec.cpp:62`](../src/secs2/codec.cpp): + +```cpp +out.push_back(static_cast((static_cast(fmt) << 2) | nlen)); +``` + +The encoder picks the smallest `nlen` that fits the body length: + +```cpp +if (length <= 0xFF) nlen = 1; // 1-byte length +else if (length <= 0xFFFF) nlen = 2; // 2-byte length +else if (length <= 0xFFFFFF) nlen = 3; // 3-byte length +else throw CodecError("item length exceeds 3-byte maximum"); +``` + +Three-byte length is the cap: **2^24 − 1 = 16 777 215 bytes ≈ 16 +MiB per item**. Larger bodies need to be split — but HSMS allows +single frames up to its own limit (4 GiB), so this is rarely the +bottleneck in practice. + +--- + +## The 14 format codes + +The format code occupies the high 6 bits. SEMI E5 §9.5 Table 5 +enumerates them. Codes are canonical *octal*, which feels archaic +but matches the spec and the codebase: + +| Octal | Decimal | Hex | Format | Storage in `Item` | Element size | +|-------|---------|------|--------------|-------------------------|---------------| +| 000 | 0 | 0x00 | **L** List | `std::vector` | (children) | +| 010 | 8 | 0x08 | **B** Binary | `std::vector` | 1 byte | +| 011 | 9 | 0x09 | **BOOLEAN** | `std::vector` | 1 byte | +| 020 | 16 | 0x10 | **A** ASCII | `std::string` | 1 byte | +| 021 | 17 | 0x11 | **J** JIS-8 | `std::string` | 1 byte | +| 022 | 18 | 0x12 | **C** C2 (Unicode-2) | `std::vector` | 2 bytes | +| 030 | 24 | 0x18 | **I8** | `std::vector` | 8 bytes | +| 031 | 25 | 0x19 | **I1** | `std::vector` | 1 byte | +| 032 | 26 | 0x1A | **I2** | `std::vector` | 2 bytes | +| 034 | 28 | 0x1C | **I4** | `std::vector` | 4 bytes | +| 040 | 32 | 0x20 | **F8** | `std::vector` | 8 bytes | +| 044 | 36 | 0x24 | **F4** | `std::vector` | 4 bytes | +| 050 | 40 | 0x28 | **U8** | `std::vector` | 8 bytes | +| 051 | 41 | 0x29 | **U1** | `std::vector` | 1 byte | +| 052 | 42 | 0x2A | **U2** | `std::vector` | 2 bytes | +| 054 | 44 | 0x2C | **U4** | `std::vector` | 4 bytes | + +Defined in [`include/secsgem/secs2/item.hpp:14`](../include/secsgem/secs2/item.hpp): + +```cpp +enum class Format : uint8_t { + List = 000, Binary = 010, Boolean = 011, + ASCII = 020, JIS8 = 021, C2 = 022, + I8 = 030, I1 = 031, I2 = 032, I4 = 034, + F8 = 040, F4 = 044, + U8 = 050, U1 = 051, U2 = 052, U4 = 054, +}; +``` + +The values are stored as the 6-bit format code (not the full format +byte), so `(fmt << 2) | nlen` produces the wire byte directly. + +### Why these specific codes? + +Three things to notice: + +1. **The numbering isn't dense.** Codes 1–7 are reserved, codes 12, + 13, 15, 23–27, 33, 35, 37, 41–43, 45–47, 51, 53, 55–57, 60–77 are + all unassigned. SEMI E5 left room for future formats and never + filled it. +2. **The integer / float widths are encoded in the low octal bits** + — `031` (I1) `032` (I2) `034` (I4) `030` (I8); same pattern for + U and F. So shifting right by 0–2 bits doesn't give you a size; + you have to look it up. `element_size()` in + [`item.hpp:58`](../include/secsgem/secs2/item.hpp) does that. +3. **Binary and Boolean share storage** (`std::vector`), + disambiguated by `format()`. Same for ASCII and JIS-8 (both + `std::string`), and U2 and C2 (both `std::vector`). + +--- + +## Length bytes + +After the format byte come the length bytes. How many? The low +two bits of the format byte say: + +``` +nl bits length bytes follow max representable length +───────────────────────────────────────────────────────── +01 1 byte 255 (0xFF) +10 2 bytes 65 535 (0xFFFF) +11 3 bytes 16 777 215 (0xFFFFFF) +00 — invalid — +``` + +All length bytes are **big-endian** (network byte order). + +For **scalar formats** (every format except `List`), the length is +the body's **byte count**. + +For **List**, the length is the **element count** — the number of +child Items that follow. Each child is itself a fully-encoded Item +(format byte + length + body), so the *byte* length of the list +body is whatever those children sum to. + +The encoder picks the smallest nl that fits — see +[`src/secs2/codec.cpp:55-67`](../src/secs2/codec.cpp). The decoder +reads the format byte's low two bits, then that many big-endian +length bytes, then the body: + +```cpp +// src/secs2/codec.cpp — sketch +const uint8_t format_byte = data[pos++]; +const Format fmt = static_cast(format_byte >> 2); +const int nlen = format_byte & 0x03; +std::size_t length = 0; +for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++]; +// then read `length` bytes (or items, for List). +``` + +--- + +## Walked through: every format with a hexdump + +Format names below match SML output (`L`, `A`, `U1`, …). + +### List + +``` + (empty list) + │ + ▼ + 01 00 + │ │ + │ └── length: 0 children + └───── format byte: + (000 << 2) | 01 = 0x01 +``` + +``` + (list of 2: ASCII "Hi", U1 5) + │ + ▼ + 01 02 41 02 48 69 A5 01 05 + │ │ │ │ + │ │ │ └── inner U1 + │ │ └── inner ASCII + │ └── length: 2 children + └── format byte 0x01 = L, nl=1 +``` + +### ASCII + +``` +A "Hello, world" format = 020 (A), nl=1, length=12 + format byte = (020 << 2) | 01 = 0x41 + + 41 0C 48 65 6C 6C 6F 2C 20 77 6F 72 6C 64 +``` + +ASCII with length crossing a length-byte boundary forces `nl=2`: + +``` +A (256 chars) format byte = (020 << 2) | 10 = 0x42 + length bytes = 01 00 (big-endian 256) + + 42 01 00 <256 bytes …> +``` + +### Binary vs Boolean + +Same on-disk shape (a byte array); different format byte: + +``` +B 0xCA 0xFE 08 02 CA FE (B format = 010 → 0x08 byte) + │ + └── format byte = (010 << 2) | 01 = 0x09 — wait + +Correction: 09 02 CA FE format byte = (010 << 2) | 01 + = (8 << 2) | 1 = 0x21? — let me redo. +``` + +Let me redo the arithmetic explicitly so it's airtight. For +**Binary**: format code `010` octal = `8` decimal. Format byte = +`(8 << 2) | 1 = 33 = 0x21`. + +``` +B 0xCA 0xFE 21 02 CA FE format byte 0x21, length 2 +``` + +For **Boolean** (one true byte): format code `011` = `9`. Format +byte = `(9 << 2) | 1 = 0x25`. + +``` +BOOLEAN true 25 01 01 format byte 0x25, length 1, body 0x01 +BOOLEAN false 25 01 00 format byte 0x25, length 1, body 0x00 +``` + +Booleans on the wire are *bytes*, not bits. 0 is false; anything +non-zero is true. + +### Unsigned integers + +`U1` (format `051` = `41`): format byte = `(41 << 2) | nl`. `(41 << +2) = 164 = 0xA4`. So for `nl=1`: `0xA5`. + +``` +U1 5 A5 01 05 +U1 [5, 10, 15] A5 03 05 0A 0F +``` + +`U2` (format `052` = `42`): `(42 << 2) = 168 = 0xA8`. `nl=1` → `0xA9`. + +``` +U2 300 A9 02 01 2C (300 = 0x012C, big-endian) +U2 [1, 2, 3] A9 06 00 01 00 02 00 03 +``` + +`U4` (format `054` = `44`): `(44 << 2) = 176 = 0xB0`. `nl=1` → `0xB1`. + +``` +U4 1 B1 04 00 00 00 01 +U4 65536 B1 04 00 01 00 00 +``` + +`U8` (format `050` = `40`): `(40 << 2) = 160 = 0xA0`. `nl=1` → `0xA1`. + +``` +U8 1 A1 08 00 00 00 00 00 00 00 01 +``` + +Notice the pattern: the format byte's high 6 bits identify the +type, the low 2 bits say "how many length bytes," and then comes +the data. + +### Signed integers + +Same shape as unsigned, but two's-complement. + +`I1` (`031` = 25): format byte `(25 << 2) | 1 = 0x65`. + +``` +I1 -1 65 01 FF +I1 5 65 01 05 +I1 [-1, 0, 1] 65 03 FF 00 01 +``` + +`I2` (`032` = 26): format byte `0x69`. + +``` +I2 -1 69 02 FF FF +I2 1 69 02 00 01 +``` + +### Floats + +IEEE 754, big-endian. `F4` is single-precision, 4 bytes; `F8` is +double-precision, 8 bytes. + +`F4` (`044` = 36): format byte `(36 << 2) | 1 = 0x91`. + +``` +F4 1.0 91 04 3F 80 00 00 (1.0 = 0x3F800000) +F4 -1.0 91 04 BF 80 00 00 +F4 0.5 91 04 3F 00 00 00 +F4 NaN 91 04 7F C0 00 00 (one canonical NaN) +F4 +Inf 91 04 7F 80 00 00 +F4 -Inf 91 04 FF 80 00 00 +F4 -0.0 91 04 80 00 00 00 +``` + +`F8` (`040` = 32): format byte `0x81`. + +``` +F8 1.0 81 08 3F F0 00 00 00 00 00 00 +``` + +The encoder uses `std::bit_cast` ([`src/secs2/codec.cpp:13`](../src/secs2/codec.cpp)) +to get the IEEE 754 bit pattern without any rounding, then writes +the bytes big-endian. Decoding is the same in reverse. This +guarantees bit-exact float round-trip including NaN, ±Inf, −0.0. + +### ASCII variants + +`J` JIS-8 (`021` = 17, byte 0x46 for nl=1): single-byte Japanese +encoding, used by some Japanese tool vendors. + +``` +J "ハロー" 46 06 8A D8 A4 (or however the bytes decode in your JIS) +``` + +`C` C2 (`022` = 18, byte 0x4A for nl=1): big-endian 16-bit Unicode +code points (essentially UCS-2 / pre-surrogate UTF-16). + +``` +C "Hi" 4A 04 00 48 00 69 +C "ハ" 4A 02 30 CF (U+30CF = HIRAGANA HA) +``` + +Note: the *length* in the format byte's length bytes is still a +**byte count**, not a character count — for C2, length always +divides by 2. + +--- + +## The decoder, recursively + +The decoder is straightforward: + +1. Read one format byte. +2. Read `nl` length bytes; assemble length big-endian. +3. If the format is List, recurse `length` times. +4. Otherwise read `length` bytes, interpret per the format's + element size, build the array. + +In code, [`src/secs2/codec.cpp`](../src/secs2/codec.cpp) (sketched): + +```cpp +Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos) { + const uint8_t fb = data[pos++]; + const Format fmt = static_cast(fb >> 2); + const int nlen = fb & 0x03; + std::size_t length = 0; + for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++]; + + if (fmt == Format::List) { + Item::List children; + children.reserve(length); + for (std::size_t i = 0; i < length; ++i) + children.push_back(decode_at(data, len, pos)); + return Item::list(std::move(children)); + } + + // scalar/array: read `length` raw bytes, dispatch on element_size(fmt) + // and integer signedness / float-ness. + const uint8_t* body = data + pos; + pos += length; + switch (fmt) { + case Format::ASCII: return Item::ascii(std::string(reinterpret_cast(body), length)); + case Format::U1: return Item::u1(std::vector(body, body + length)); + case Format::U2: return Item::u2(read_array(body, length)); + case Format::U4: return Item::u4(read_array(body, length)); + // …and so on for every format. + } +} +``` + +`decode()` (no `_at`) decodes a buffer and throws if there are +trailing bytes — useful as the top-level entry point. + +--- + +## SML: the human-readable form + +You'll see SECS-II written in **SML** (SECS Message Language) +throughout the rest of this guide and every Wireshark dump. SML is +not on the wire — it's a textual rendering of the same Item tree. +The mapping is: + +``` +List +ASCII A "Hello" +Binary B 0x01 0x02 +Boolean BOOLEAN True +U1 (scalar) U1 5 +U1 (array) U1[3] 5 10 15 +F4 F4 1.0 +F4 (special) F4 +Inf, F4 NaN +I2 negative I2 -1 +C2 C "Hi" +``` + +Nested: + +``` + +> +``` + +The SML parser/printer is in +[`include/secsgem/secs2/sml.hpp`](../include/secsgem/secs2/sml.hpp) +and [`src/secs2/sml.cpp`](../src/secs2/sml.cpp). `to_sml(item)` +prints; `try_parse_sml(str)` returns an `Item` or an error. SML +parsing is exercised by libFuzzer in +[`apps/fuzz_sml_parse.cpp`](../apps/fuzz_sml_parse.cpp) — over 1 +million random SML strings per minute, ASan + UBSan clean, 0 +crashes (see [PROOFS.md](PROOFS.md) proof #8). + +SML is *useful* (you can paste it into a debugger), but it's not +canonical: two different SML strings might serialize to identical +bytes (whitespace, comments, optional list-length annotations), and +two different on-wire byte streams might pretty-print as identical +SML (different length-byte counts for the same length). The wire +bytes are canonical; SML is for humans. + +--- + +## The identifier wildcard rule + +A SEMI subtlety that bites every implementation eventually. + +**The spec says**: for identifier fields (SVID, ECID, CEID, ALID, +RPTID, LIMITID, …), the equipment may encode the value as `U1`, +`U2`, `U4`, *or* `U8`, picking whichever width fits. And the +receiver — equipment or host — **must accept all four widths +interchangeably**. + +So a host sending `S2F33` to define a report can put RPTID 1 as +`U1 1` (3 bytes on the wire: `A5 01 01`) or `U4 1` (6 bytes: `B1 04 +00 00 00 01`) — both are valid, and the equipment must accept +either. + +The codebase enforces this leniency through one helper: +[`messages_helpers::any_unsigned_first()`](../include/secsgem/gem/messages_helpers.hpp). + +```cpp +// include/secsgem/gem/messages_helpers.hpp:99 +template +inline std::optional any_unsigned_first(const s2::Item& item) { + // Try U1, U2, U4, U8 in turn; if the value fits in Out, return it. +} +``` + +Every place that reads an identifier off the wire uses one of: + +```cpp +as_u1_scalar(item) // accepts U1/U2/U4/U8 widths if value fits in uint8_t +as_u2_scalar(item) // accepts U1/U2/U4/U8 widths if value fits in uint16_t +as_u4_scalar(item) // accepts U1/U2/U4/U8 widths if value fits in uint32_t +as_u8_scalar(item) // accepts any width +``` + +[`tests/test_identifier_wildcards.cpp`](../tests/test_identifier_wildcards.cpp) +asserts this for every combination of declared width × encoded +width. 6 test cases, every direction of the matrix; this was a +real interop bug before the helpers existed (see +[`interop/README.md`](../interop/README.md) for the back-story). + +Outgoing encoding still picks the *narrowest* width that fits, to +keep wire bytes minimal. + +--- + +## Equality and round-trip + +`Item` is `bool operator==(const Item&) const = default;` +([`item.hpp:161`](../include/secsgem/secs2/item.hpp)) — defaulted +member-wise equality on the variant. Two Items compare equal iff +they have the same format AND the same storage values. + +This makes round-trip tests trivial: + +```cpp +const Item original = ...; +const auto bytes = encode(original); +const Item decoded = decode(bytes); +REQUIRE(decoded == original); +``` + +[`tests/test_secs2.cpp`](../tests/test_secs2.cpp) does this for +hand-picked Items across every format; the libFuzzer harness in +[`apps/fuzz_secs2_decode.cpp`](../apps/fuzz_secs2_decode.cpp) does +it for arbitrary random byte streams. + +The strongest round-trip evidence is the **SEMI E5 KAT** (known- +answer tests) in +[`tests/test_e5_kat.cpp`](../tests/test_e5_kat.cpp). Each fixture +is a hex string written by hand directly from the spec's encoding +rules; the test asserts that encoding the corresponding Item +produces *exactly those bytes* and that decoding them produces +exactly that Item. 19 test cases, 196 assertions, every format +code, every length-byte-count variant (1, 2, 3 bytes), numeric +edges (0, ±1, MIN, MAX, ±Inf, NaN), nested lists. + +The KAT is the strongest single proof of codec correctness in the +codebase, because every other validator is *one* implementer's +interpretation of the spec; KAT is the spec's own arithmetic. See +[VERIFICATION.md](VERIFICATION.md) §1 for the rationale. + +--- + +## What can go wrong + +The codec rejects malformed input in a few well-defined ways +([`src/secs2/codec.cpp`](../src/secs2/codec.cpp) throws +`CodecError`): + +- **Truncated input** — the buffer ends in the middle of a format + byte, length bytes, or body. +- **Length-not-multiple-of-element-size** — e.g. a U4 array + claiming a body length of 7 bytes. 4 doesn't divide 7. +- **Length exceeds buffer** — the length bytes claim more body than + exists. +- **Trailing bytes after the top-level item** — for `decode()` + (not `decode_at()`), the buffer must end exactly when the item + does. +- **3-byte length cap exceeded** — encode rejects items larger + than 16 MiB. + +Every one of these is exercised by either +[`tests/test_secs2.cpp`](../tests/test_secs2.cpp), +[`tests/test_fuzz.cpp`](../tests/test_fuzz.cpp), or the libFuzzer +harness. + +Notably *not* rejected: an unknown format code. The spec is silent +on how a receiver should handle codes 1–7, 12, 13, 15, etc., and +in practice the codec passes them through as the raw `Format` enum +value. Whether a higher-level handler cares is up to that handler. + +--- + +## Where to go next + +You now understand the entire SECS-II data encoding. Three things +build on it directly: + +- **The catalog of named messages** uses Items as bodies. Every + SxFy in [`data/messages.yaml`](../data/messages.yaml) is a recipe + for one specific Item shape. See chapter + [31](31_spec_as_data_and_codegen.md) for the codegen that turns + YAML recipes into typed C++ structs. +- **The Message type** + ([`include/secsgem/secs2/message.hpp`](../include/secsgem/secs2/message.hpp)) + wraps a body Item with stream, function, W-bit, and system + bytes. That's what HSMS frames carry; see chapter + [11](11_e37_hsms.md). +- **Behaviour** reads typed values out of incoming Items (using the + identifier wildcard helpers) and writes them back. Chapter + [13](13_e30_gem.md) covers E30 / GEM. + +But before any behaviour can happen, the bytes have to *get there*. +That's the next chapter: **E37 HSMS**, the TCP transport. + +Next: [→ 11 E37 — HSMS transport](11_e37_hsms.md)