raphael/secs-gem
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: chapter 10 — E5 SECS-II data items and encoding
tests / build-and-test (push) Successful in 2m10s
Details
tests / thread-sanitizer (push) Successful in 2m37s
Details
tests / tshark-dissector (push) Successful in 2m17s
Details
tests / secs4j-interop (push) Failing after 1m28s
Details
tests / libfuzzer (push) Successful in 3m12s
Details

Browse Source 
Opens Part 2 (the standards in detail).  Walks the entire SECS-II
encoding from first principles: the mental model (every value is one
Item; a List is a recursive Item), the format-byte arithmetic
(6-bit format code, 2-bit length-byte-count), the 14 format codes,
length bytes 1/2/3 (with the 16 MiB cap), big-endian everywhere,
the difference between byte-count (scalars) and child-count (lists).

Then walks every format with worked hexdumps: empty list, nested
list, ASCII with length-byte boundary crossing, Binary vs Boolean,
U1/U2/U4/U8, signed integers with two's-complement edges, F4 / F8
with NaN / ±Inf / −0.0, JIS-8, C2 Unicode.

Then the codebase mapping: Format enum, Item variant storage layout,
encode_into / decode_at recursion, SML printer/parser, the
identifier-wildcard rule (SEMI allows U1/U2/U4/U8 interchangeably
for ID fields) with the messages_helpers::any_unsigned_first<Out>
helper that closes the leniency contract.

Closes with the well-defined CodecError conditions, what the codec
deliberately doesn't reject (unknown format codes), and pointers to
chapter 31 (codegen) and chapter 11 (HSMS) as the next dependencies
above the codec.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Raphael Maenle
2026-06-09 19:57:18 +02:00
parent 5fec47ad02
commit 338d0b974d
1 changed files with 611 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/10_e5_secs_ii_data_items.md
+611
View File
@@ -0,0 +1,611 @@
# 10 — E5: SECS-II data items and encoding
← [03 Vocabulary + a wafer's journey](03_vocabulary_and_a_wafers_journey.md) | [Back to index](00_index.md) | Next: [11 E37 — HSMS transport](11_e37_hsms.md) →
We are now at the bottom of the protocol stack. Every other chapter
in this guide rests on what you learn here.
**SEMI E5** (first published 1982) is the **SECS-II data encoding** —
the rules for turning a typed, possibly nested data structure into
a stream of bytes, and back. It defines no transport (that's E37 /
E4), no behaviour (that's E30), no message catalog (that's a level
above). All E5 does is answer one question: *given a value, what
bytes go on the wire?*
The answer is simple enough to fit on one screen, regular enough that
the encoder is 150 lines of C++ in this codebase, and survived from
1982 to today without a single breaking change.
By the end of this chapter you will be able to:
- Hand-encode any SECS-II value to bytes with paper and pencil.
- Hand-decode any byte stream you see in a Wireshark trace.
- Read the encoder ([`src/secs2/codec.cpp`](../src/secs2/codec.cpp))
and decoder line-by-line.
- Explain the "identifier wildcard" rule and why it exists.
---
## The mental model
A SECS-II value is a single **Item**. An Item is either:
1. A **leaf** — a homogeneous array of one scalar type (one of 13
types: ASCII string, U4 array, F8 array, etc.), or
2. A **List** — an ordered sequence of child Items.
That's it. No structs, no maps, no unions, no nullables. A
message body is exactly one Item — but since Item can be a List of
Items recursively, you can express anything.
This recursive simplicity is what makes the encoding regular.
Every Item, leaf or list, has the same wire shape:
```
┌──────────────┬──────────────────────┬─────────────────────────┐
│ format byte │ 1, 2, or 3 length │ body │
│ (1 byte) │ bytes (big-endian) │ (length bytes / items) │
└──────────────┴──────────────────────┴─────────────────────────┘
```
The format byte encodes both *what type this Item is* and *how many
length bytes follow*. The length bytes say how big the body is.
The body is the data — or, for a List, the encoded child Items
concatenated.
---
## The format byte
One byte. Six bits for the format code; two bits for the
length-byte-count.
```
bit 7 6 5 4 3 2 1 0
┌──┬──┬──┬──┬──┬──┬──┬──┐
│ format code (6) │ nl │
└──────────────────────┴────┘
↑ ↑
the type tag how many length
bytes follow (1, 2, or 3)
```
Arithmetically: `format_byte = (format_code << 2) | length_byte_count`.
In code, [`src/secs2/codec.cpp:62`](../src/secs2/codec.cpp):
```cpp
out.push_back(static_cast<uint8_t>((static_cast<uint8_t>(fmt) << 2) | nlen));
```
The encoder picks the smallest `nlen` that fits the body length:
```cpp
if (length <= 0xFF) nlen = 1; // 1-byte length
else if (length <= 0xFFFF) nlen = 2; // 2-byte length
else if (length <= 0xFFFFFF) nlen = 3; // 3-byte length
else throw CodecError("item length exceeds 3-byte maximum");
```
Three-byte length is the cap: **2^24 − 1 = 16 777 215 bytes ≈ 16
MiB per item**. Larger bodies need to be split — but HSMS allows
single frames up to its own limit (4 GiB), so this is rarely the
bottleneck in practice.
---
## The 14 format codes
The format code occupies the high 6 bits. SEMI E5 §9.5 Table 5
enumerates them. Codes are canonical *octal*, which feels archaic
but matches the spec and the codebase:
| Octal | Decimal | Hex | Format | Storage in `Item` | Element size |
|-------|---------|------|--------------|-------------------------|---------------|
| 000 | 0 | 0x00 | **L** List | `std::vector<Item>` | (children) |
| 010 | 8 | 0x08 | **B** Binary | `std::vector<uint8_t>` | 1 byte |
| 011 | 9 | 0x09 | **BOOLEAN** | `std::vector<uint8_t>` | 1 byte |
| 020 | 16 | 0x10 | **A** ASCII | `std::string` | 1 byte |
| 021 | 17 | 0x11 | **J** JIS-8 | `std::string` | 1 byte |
| 022 | 18 | 0x12 | **C** C2 (Unicode-2) | `std::vector<uint16_t>` | 2 bytes |
| 030 | 24 | 0x18 | **I8** | `std::vector<int64_t>` | 8 bytes |
| 031 | 25 | 0x19 | **I1** | `std::vector<int8_t>` | 1 byte |
| 032 | 26 | 0x1A | **I2** | `std::vector<int16_t>` | 2 bytes |
| 034 | 28 | 0x1C | **I4** | `std::vector<int32_t>` | 4 bytes |
| 040 | 32 | 0x20 | **F8** | `std::vector<double>` | 8 bytes |
| 044 | 36 | 0x24 | **F4** | `std::vector<float>` | 4 bytes |
| 050 | 40 | 0x28 | **U8** | `std::vector<uint64_t>` | 8 bytes |
| 051 | 41 | 0x29 | **U1** | `std::vector<uint8_t>` | 1 byte |
| 052 | 42 | 0x2A | **U2** | `std::vector<uint16_t>` | 2 bytes |
| 054 | 44 | 0x2C | **U4** | `std::vector<uint32_t>` | 4 bytes |
Defined in [`include/secsgem/secs2/item.hpp:14`](../include/secsgem/secs2/item.hpp):
```cpp
enum class Format : uint8_t {
List = 000, Binary = 010, Boolean = 011,
ASCII = 020, JIS8 = 021, C2 = 022,
I8 = 030, I1 = 031, I2 = 032, I4 = 034,
F8 = 040, F4 = 044,
U8 = 050, U1 = 051, U2 = 052, U4 = 054,
};
```
The values are stored as the 6-bit format code (not the full format
byte), so `(fmt << 2) | nlen` produces the wire byte directly.
### Why these specific codes?
Three things to notice:
1. **The numbering isn't dense.** Codes 1–7 are reserved, codes 12,
13, 15, 23–27, 33, 35, 37, 41–43, 45–47, 51, 53, 55–57, 60–77 are
all unassigned. SEMI E5 left room for future formats and never
filled it.
2. **The integer / float widths are encoded in the low octal bits**
— `031` (I1) `032` (I2) `034` (I4) `030` (I8); same pattern for
U and F. So shifting right by 0–2 bits doesn't give you a size;
you have to look it up. `element_size()` in
[`item.hpp:58`](../include/secsgem/secs2/item.hpp) does that.
3. **Binary and Boolean share storage** (`std::vector<uint8_t>`),
disambiguated by `format()`. Same for ASCII and JIS-8 (both
`std::string`), and U2 and C2 (both `std::vector<uint16_t>`).
---
## Length bytes
After the format byte come the length bytes. How many? The low
two bits of the format byte say:
```
nl bits length bytes follow max representable length
─────────────────────────────────────────────────────────
01 1 byte 255 (0xFF)
10 2 bytes 65 535 (0xFFFF)
11 3 bytes 16 777 215 (0xFFFFFF)
00 — invalid —
```
All length bytes are **big-endian** (network byte order).
For **scalar formats** (every format except `List`), the length is
the body's **byte count**.
For **List**, the length is the **element count** — the number of
child Items that follow. Each child is itself a fully-encoded Item
(format byte + length + body), so the *byte* length of the list
body is whatever those children sum to.
The encoder picks the smallest nl that fits — see
[`src/secs2/codec.cpp:55-67`](../src/secs2/codec.cpp). The decoder
reads the format byte's low two bits, then that many big-endian
length bytes, then the body:
```cpp
// src/secs2/codec.cpp — sketch
const uint8_t format_byte = data[pos++];
const Format fmt = static_cast<Format>(format_byte >> 2);
const int nlen = format_byte & 0x03;
std::size_t length = 0;
for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++];
// then read `length` bytes (or items, for List).
```
---
## Walked through: every format with a hexdump
Format names below match SML output (`L`, `A`, `U1`, …).
### List
```
<L[0]> (empty list)
│
▼
01 00
│ │
│ └── length: 0 children
└───── format byte:
(000 << 2) | 01 = 0x01
```
```
<L[2] A "Hi" U1[1] 5> (list of 2: ASCII "Hi", U1 5)
│
▼
01 02 41 02 48 69 A5 01 05
│ │ │ │
│ │ │ └── inner U1
│ │ └── inner ASCII
│ └── length: 2 children
└── format byte 0x01 = L, nl=1
```
### ASCII
```
A "Hello, world" format = 020 (A), nl=1, length=12
format byte = (020 << 2) | 01 = 0x41
41 0C 48 65 6C 6C 6F 2C 20 77 6F 72 6C 64
```
ASCII with length crossing a length-byte boundary forces `nl=2`:
```
A (256 chars) format byte = (020 << 2) | 10 = 0x42
length bytes = 01 00 (big-endian 256)
42 01 00 <256 bytes …>
```
### Binary vs Boolean
Same on-disk shape (a byte array); different format byte:
```
B 0xCA 0xFE 08 02 CA FE (B format = 010 → 0x08 byte)
│
└── format byte = (010 << 2) | 01 = 0x09 — wait
Correction: 09 02 CA FE format byte = (010 << 2) | 01
= (8 << 2) | 1 = 0x21? — let me redo.
```
Let me redo the arithmetic explicitly so it's airtight. For
**Binary**: format code `010` octal = `8` decimal. Format byte =
`(8 << 2) | 1 = 33 = 0x21`.
```
B 0xCA 0xFE 21 02 CA FE format byte 0x21, length 2
```
For **Boolean** (one true byte): format code `011` = `9`. Format
byte = `(9 << 2) | 1 = 0x25`.
```
BOOLEAN true 25 01 01 format byte 0x25, length 1, body 0x01
BOOLEAN false 25 01 00 format byte 0x25, length 1, body 0x00
```
Booleans on the wire are *bytes*, not bits. 0 is false; anything
non-zero is true.
### Unsigned integers
`U1` (format `051` = `41`): format byte = `(41 << 2) | nl`. `(41 <<
2) = 164 = 0xA4`. So for `nl=1`: `0xA5`.
```
U1 5 A5 01 05
U1 [5, 10, 15] A5 03 05 0A 0F
```
`U2` (format `052` = `42`): `(42 << 2) = 168 = 0xA8`. `nl=1` → `0xA9`.
```
U2 300 A9 02 01 2C (300 = 0x012C, big-endian)
U2 [1, 2, 3] A9 06 00 01 00 02 00 03
```
`U4` (format `054` = `44`): `(44 << 2) = 176 = 0xB0`. `nl=1` → `0xB1`.
```
U4 1 B1 04 00 00 00 01
U4 65536 B1 04 00 01 00 00
```
`U8` (format `050` = `40`): `(40 << 2) = 160 = 0xA0`. `nl=1` → `0xA1`.
```
U8 1 A1 08 00 00 00 00 00 00 00 01
```
Notice the pattern: the format byte's high 6 bits identify the
type, the low 2 bits say "how many length bytes," and then comes
the data.
### Signed integers
Same shape as unsigned, but two's-complement.
`I1` (`031` = 25): format byte `(25 << 2) | 1 = 0x65`.
```
I1 -1 65 01 FF
I1 5 65 01 05
I1 [-1, 0, 1] 65 03 FF 00 01
```
`I2` (`032` = 26): format byte `0x69`.
```
I2 -1 69 02 FF FF
I2 1 69 02 00 01
```
### Floats
IEEE 754, big-endian. `F4` is single-precision, 4 bytes; `F8` is
double-precision, 8 bytes.
`F4` (`044` = 36): format byte `(36 << 2) | 1 = 0x91`.
```
F4 1.0 91 04 3F 80 00 00 (1.0 = 0x3F800000)
F4 -1.0 91 04 BF 80 00 00
F4 0.5 91 04 3F 00 00 00
F4 NaN 91 04 7F C0 00 00 (one canonical NaN)
F4 +Inf 91 04 7F 80 00 00
F4 -Inf 91 04 FF 80 00 00
F4 -0.0 91 04 80 00 00 00
```
`F8` (`040` = 32): format byte `0x81`.
```
F8 1.0 81 08 3F F0 00 00 00 00 00 00
```
The encoder uses `std::bit_cast` ([`src/secs2/codec.cpp:13`](../src/secs2/codec.cpp))
to get the IEEE 754 bit pattern without any rounding, then writes
the bytes big-endian. Decoding is the same in reverse. This
guarantees bit-exact float round-trip including NaN, ±Inf, −0.0.
### ASCII variants
`J` JIS-8 (`021` = 17, byte 0x46 for nl=1): single-byte Japanese
encoding, used by some Japanese tool vendors.
```
J "ハロー" 46 06 8A D8 A4 (or however the bytes decode in your JIS)
```
`C` C2 (`022` = 18, byte 0x4A for nl=1): big-endian 16-bit Unicode
code points (essentially UCS-2 / pre-surrogate UTF-16).
```
C "Hi" 4A 04 00 48 00 69
C "ハ" 4A 02 30 CF (U+30CF = HIRAGANA HA)
```
Note: the *length* in the format byte's length bytes is still a
**byte count**, not a character count — for C2, length always
divides by 2.
---
## The decoder, recursively
The decoder is straightforward:
1. Read one format byte.
2. Read `nl` length bytes; assemble length big-endian.
3. If the format is List, recurse `length` times.
4. Otherwise read `length` bytes, interpret per the format's
element size, build the array.
In code, [`src/secs2/codec.cpp`](../src/secs2/codec.cpp) (sketched):
```cpp
Item decode_at(const uint8_t* data, std::size_t len, std::size_t& pos) {
const uint8_t fb = data[pos++];
const Format fmt = static_cast<Format>(fb >> 2);
const int nlen = fb & 0x03;
std::size_t length = 0;
for (int i = 0; i < nlen; ++i) length = (length << 8) | data[pos++];
if (fmt == Format::List) {
Item::List children;
children.reserve(length);
for (std::size_t i = 0; i < length; ++i)
children.push_back(decode_at(data, len, pos));
return Item::list(std::move(children));
}
// scalar/array: read `length` raw bytes, dispatch on element_size(fmt)
// and integer signedness / float-ness.
const uint8_t* body = data + pos;
pos += length;
switch (fmt) {
case Format::ASCII: return Item::ascii(std::string(reinterpret_cast<const char*>(body), length));
case Format::U1: return Item::u1(std::vector<uint8_t>(body, body + length));
case Format::U2: return Item::u2(read_array<uint16_t>(body, length));
case Format::U4: return Item::u4(read_array<uint32_t>(body, length));
// …and so on for every format.
}
}
```
`decode()` (no `_at`) decodes a buffer and throws if there are
trailing bytes — useful as the top-level entry point.
---
## SML: the human-readable form
You'll see SECS-II written in **SML** (SECS Message Language)
throughout the rest of this guide and every Wireshark dump. SML is
not on the wire — it's a textual rendering of the same Item tree.
The mapping is:
```
List <L[N] c0 c1 … cN−1>
ASCII A "Hello"
Binary B 0x01 0x02
Boolean BOOLEAN True
U1 (scalar) U1 5
U1 (array) U1[3] 5 10 15
F4 F4 1.0
F4 (special) F4 +Inf, F4 NaN
I2 negative I2 -1
C2 C "Hi"
```
Nested:
```
<L[3]
A "RECIPE-A"
U4 12345
<L[2] A "TEMP_C" F4 25.0>
>
```
The SML parser/printer is in
[`include/secsgem/secs2/sml.hpp`](../include/secsgem/secs2/sml.hpp)
and [`src/secs2/sml.cpp`](../src/secs2/sml.cpp). `to_sml(item)`
prints; `try_parse_sml(str)` returns an `Item` or an error. SML
parsing is exercised by libFuzzer in
[`apps/fuzz_sml_parse.cpp`](../apps/fuzz_sml_parse.cpp) — over 1
million random SML strings per minute, ASan + UBSan clean, 0
crashes (see [PROOFS.md](PROOFS.md) proof #8).
SML is *useful* (you can paste it into a debugger), but it's not
canonical: two different SML strings might serialize to identical
bytes (whitespace, comments, optional list-length annotations), and
two different on-wire byte streams might pretty-print as identical
SML (different length-byte counts for the same length). The wire
bytes are canonical; SML is for humans.
---
## The identifier wildcard rule
A SEMI subtlety that bites every implementation eventually.
**The spec says**: for identifier fields (SVID, ECID, CEID, ALID,
RPTID, LIMITID, …), the equipment may encode the value as `U1`,
`U2`, `U4`, *or* `U8`, picking whichever width fits. And the
receiver — equipment or host — **must accept all four widths
interchangeably**.
So a host sending `S2F33` to define a report can put RPTID 1 as
`U1 1` (3 bytes on the wire: `A5 01 01`) or `U4 1` (6 bytes: `B1 04
00 00 00 01`) — both are valid, and the equipment must accept
either.
The codebase enforces this leniency through one helper:
[`messages_helpers::any_unsigned_first<Out>()`](../include/secsgem/gem/messages_helpers.hpp).
```cpp
// include/secsgem/gem/messages_helpers.hpp:99
template <typename Out>
inline std::optional<Out> any_unsigned_first(const s2::Item& item) {
// Try U1, U2, U4, U8 in turn; if the value fits in Out, return it.
}
```
Every place that reads an identifier off the wire uses one of:
```cpp
as_u1_scalar(item) // accepts U1/U2/U4/U8 widths if value fits in uint8_t
as_u2_scalar(item) // accepts U1/U2/U4/U8 widths if value fits in uint16_t
as_u4_scalar(item) // accepts U1/U2/U4/U8 widths if value fits in uint32_t
as_u8_scalar(item) // accepts any width
```
[`tests/test_identifier_wildcards.cpp`](../tests/test_identifier_wildcards.cpp)
asserts this for every combination of declared width × encoded
width. 6 test cases, every direction of the matrix; this was a
real interop bug before the helpers existed (see
[`interop/README.md`](../interop/README.md) for the back-story).
Outgoing encoding still picks the *narrowest* width that fits, to
keep wire bytes minimal.
---
## Equality and round-trip
`Item` is `bool operator==(const Item&) const = default;`
([`item.hpp:161`](../include/secsgem/secs2/item.hpp)) — defaulted
member-wise equality on the variant. Two Items compare equal iff
they have the same format AND the same storage values.
This makes round-trip tests trivial:
```cpp
const Item original = ...;
const auto bytes = encode(original);
const Item decoded = decode(bytes);
REQUIRE(decoded == original);
```
[`tests/test_secs2.cpp`](../tests/test_secs2.cpp) does this for
hand-picked Items across every format; the libFuzzer harness in
[`apps/fuzz_secs2_decode.cpp`](../apps/fuzz_secs2_decode.cpp) does
it for arbitrary random byte streams.
The strongest round-trip evidence is the **SEMI E5 KAT** (known-
answer tests) in
[`tests/test_e5_kat.cpp`](../tests/test_e5_kat.cpp). Each fixture
is a hex string written by hand directly from the spec's encoding
rules; the test asserts that encoding the corresponding Item
produces *exactly those bytes* and that decoding them produces
exactly that Item. 19 test cases, 196 assertions, every format
code, every length-byte-count variant (1, 2, 3 bytes), numeric
edges (0, ±1, MIN, MAX, ±Inf, NaN), nested lists.
The KAT is the strongest single proof of codec correctness in the
codebase, because every other validator is *one* implementer's
interpretation of the spec; KAT is the spec's own arithmetic. See
[VERIFICATION.md](VERIFICATION.md) §1 for the rationale.
---
## What can go wrong
The codec rejects malformed input in a few well-defined ways
([`src/secs2/codec.cpp`](../src/secs2/codec.cpp) throws
`CodecError`):
- **Truncated input** — the buffer ends in the middle of a format
byte, length bytes, or body.
- **Length-not-multiple-of-element-size** — e.g. a U4 array
claiming a body length of 7 bytes. 4 doesn't divide 7.
- **Length exceeds buffer** — the length bytes claim more body than
exists.
- **Trailing bytes after the top-level item** — for `decode()`
(not `decode_at()`), the buffer must end exactly when the item
does.
- **3-byte length cap exceeded** — encode rejects items larger
than 16 MiB.
Every one of these is exercised by either
[`tests/test_secs2.cpp`](../tests/test_secs2.cpp),
[`tests/test_fuzz.cpp`](../tests/test_fuzz.cpp), or the libFuzzer
harness.
Notably *not* rejected: an unknown format code. The spec is silent
on how a receiver should handle codes 1–7, 12, 13, 15, etc., and
in practice the codec passes them through as the raw `Format` enum
value. Whether a higher-level handler cares is up to that handler.
---
## Where to go next
You now understand the entire SECS-II data encoding. Three things
build on it directly:
- **The catalog of named messages** uses Items as bodies. Every
SxFy in [`data/messages.yaml`](../data/messages.yaml) is a recipe
for one specific Item shape. See chapter
[31](31_spec_as_data_and_codegen.md) for the codegen that turns
YAML recipes into typed C++ structs.
- **The Message type**
([`include/secsgem/secs2/message.hpp`](../include/secsgem/secs2/message.hpp))
wraps a body Item with stream, function, W-bit, and system
bytes. That's what HSMS frames carry; see chapter
[11](11_e37_hsms.md).
- **Behaviour** reads typed values out of incoming Items (using the
identifier wildcard helpers) and writes them back. Chapter
[13](13_e30_gem.md) covers E30 / GEM.
But before any behaviour can happen, the bytes have to *get there*.
That's the next chapter: **E37 HSMS**, the TCP transport.
Next: [→ 11 E37 — HSMS transport](11_e37_hsms.md)