raphael
dae6bfd747
Tone pass across the non-tutorial markdown — README, PROOFS,
ARCHITECTURE, BENCHMARKS, COMPLIANCE, FAQ, MES_INTEROP, SECURITY,
and interop/README. Three patterns came out:
- Bug-history war stories ("Past interop sweeps surfaced…",
"What these harnesses caught: 1. Strict U-width parsing…").
- Chat-with-reader framing ("Don't skip TLS unless…", "Treat as a
punch list", "If you're running in a pod…", "Misconfiguration
incidents drop dramatically").
- Self-referential narration ("we ship", "our codec", "the
codebase's most-tested layer", "three orders of magnitude above
fab load", "the gift that keeps giving").
README also drops the standalone ThreadSanitizer subsection under
Build details (now a single line under the new Testing section).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
22 KiB
22 KiB
Real-MES interop test plan
The codebase cross-validates against secsgem-py 0.3.0 in CI. That's the Python reference implementation. It is not what a fab actually runs. Real MES stacks include:
- Camstar (Siemens / Opcenter Execution Semiconductor)
- FactoryWorks (IBM, now various OEM ports)
- Inficon FabGuard
- Wonderware MES (AVEVA)
- Mozaic / Mozaic Suite
- Critical Manufacturing CMNavigo
- Eyelit MES
Each one ships its own SECS/GEM stack with its own quirks. This doc is the day-1 punch list your integration team runs against your MES before connecting a real tool. Capture wire traces from every step.
The in-repo secs_conformance and secsgem-py cross-validation prove
the codebase is spec-conformant; they cannot prove the combination
of (this codebase, your YAML, your MES's choice of optional
behaviours) works.
0. Prerequisites
- A staging copy of the MES configured to talk to a single tool over HSMS-SS or HSMS-GS. Get the MES team to point it at your test equipment endpoint (port 5000 by default).
- A wire-trace collector running on the equipment side:
Keep this running across every test below. Wireshark with the HSMS dissector (or the secsgem-py decoder in
tcpdump -i any -w mes-day1-$(date +%Y%m%d).pcap 'tcp port 5000'interop/) parses these traces after the fact. - The standard logger / metrics exporter wired up (see
INTEGRATION.md §6.4) — you want to see the
S9F*self-emissions and the FSM transitions land in your dashboard. - The
--validate-configflag run clean on your equipment YAMLs.
Record each test's outcome in a single spreadsheet — test ID, date, MES build, equipment build, PASS/FAIL/PARTIAL, notes, wire-trace timestamp. This is your audit trail for the integration.
1. HSMS transport plumbing (E37)
| ID | Test | Expected wire behaviour | Common quirks |
|---|---|---|---|
| T-01 | MES connects (active) → Select.req | Equipment replies Select.rsp(0=Ok); SELECTED on both sides | Some MES default session ID ≠ 0; override in equipment.yaml |
| T-02 | Idle Linktest cycle (5 min observation) | Linktest.req every N seconds from MES; equipment Linktest.rsp | Camstar uses 30s default; FactoryWorks 60s |
| T-03 | MES sends Separate.req → graceful close | Equipment closes socket within 1s; no FIN_WAIT2 leak | Some MES expect equipment to close first |
| T-04 | Equipment-initiated Linktest (set Timers::linktest to 10s) |
MES replies Linktest.rsp | Mozaic ignores equipment-initiated; not a bug |
| T-05 | MES disconnects TCP without Separate | Equipment detects closed socket, fires closed_handler("eof") |
Watch for spool starting to fill |
| T-06 | MES reconnects after T-05 → S6F25 if spool has content | Equipment auto-emits S6F25 with queued count | MES must support S6F25/F26 — older Wonderware doesn't |
| T-07 | T3 violation: MES sends primary but never replies | Equipment fires T3 → auto-emits S9F9 | Confirm S9F9 appears in the wire trace, not just logs |
| T-08 | T7 violation: MES connects but never sends Select.req | Equipment closes after T7 with reason "T7" | |
| T-09 | Oversized frame: MES sends 17 MiB body | Equipment auto-emits S9F11 + closes | If MES doesn't generate this, skip |
2. Establish comms + identification (E30 Fundamental)
| ID | Test | Expected wire behaviour | Common quirks |
|---|---|---|---|
| E-01 | MES → S1F13 | Equipment → S1F14(COMMACK=Accept, [MDLN, SOFTREV]) | MES that send empty MDLN list — check YAML matches |
| E-02 | MES → S1F1 | Equipment → S1F2(MDLN, SOFTREV) | EQPTYP in S1F20 is sometimes confused with MDLN |
| E-03 | MES → S1F15 (request offline) | Equipment → S1F16(OFLACK=Accept), control state → HostOffline | |
| E-04 | MES → S1F17 (request online) | Equipment → S1F18(ONLACK=Accept), control state → OnlineRemote | |
| E-05 | MES → S1F19 (compliance request) | Equipment → S1F20 with full CCODE list matching capabilities: YAML |
|
| E-06 | MES → S1F11 (SVID namelist), then S1F3 (values) | Namelist matches svids: YAML; values within type-declared ranges |
MES may expect SVID 1 = ControlState string |
| E-07 | MES → S1F21 (DVID namelist), then S1F3 (values for DVID set) | Equipment returns DVIDs separately from SVIDs | Some MES treat S1F3 as union; spec says SVID-only |
| E-08 | MES → S1F23 (CEID namelist) | Equipment returns CEID→VIDs map matching events.link_event_reports |
3. Dynamic event reports (E30 §6.6)
This is where most integrations break. MES often defines reports in a different order than spec, expects different ACK enum positions, or uses non-standard RPTID/DATAID widths.
| ID | Test | Expected | Common quirks |
|---|---|---|---|
| R-01 | MES → S2F33 (define report 1000 over 3 SVIDs) | S2F34(DRACK=0=Accept) | MES may use U1 for RPTID; the parser accepts any width |
| R-02 | MES → S2F35 (link CEID 300 ↔ RPTID 1000) | S2F36(LRACK=0=Accept) | |
| R-03 | MES → S2F37 (enable CEID 300) | S2F38(ERACK=0=Accept) | Some MES send empty CEID list = enable-all |
| R-04 | Equipment fires CEID 300 (driven by model->compose_reports_for(300)) |
MES → S6F12(ACKC6=0) | MES may take >T6 to reply — extend T6 if needed |
| R-05 | MES → S6F15 (event report request) | Equipment → S6F16 with current values | |
| R-06 | MES → S6F19/F21 (individual / annotated report request) | Equipment → S6F20 / S6F22 with current values | |
| R-07 | MES → S2F33 with DATAID=0 (clear all reports) | S2F34(DRACK=0); all link bindings flushed | Older MES use S2F33 + empty body; check both forms |
4. Alarms (E30 §6.14, E5 §13)
| ID | Test | Expected | Common quirks |
|---|---|---|---|
| A-01 | MES → S5F5 (list alarms) | S5F6 with full alarm directory matching alarms: YAML |
|
| A-02 | MES → S5F3 (enable ALID 1) | S5F4(ACKC5=0) | |
| A-03 | MES → S5F7 (list enabled) | S5F8 contains ALID 1 | |
| A-04 | Equipment sets alarm 1 active | S5F1(ALCD=0x80 | category) sent to MES → MES S5F2 ack | ALCD bit-7 must be SET, not cleared |
| A-05 | Equipment clears alarm 1 | S5F1(ALCD=0x00 | category) → S5F2 ack | |
| A-06 | Equipment fires an alarm while alarm is disabled (S5F5 says no) | NO S5F1 wire frame — alarm registry tracks active, dispatcher gates |
5. Remote control (E30 §6.15)
| ID | Test | Expected | Common quirks |
|---|---|---|---|
| C-01 | MES → S2F41 (RCMD=START, params) | S2F42(HCACK=0=Accept) + emit configured CEID | |
| C-02 | MES → S2F21 (legacy: RCMD only, no params) | S2F22(CMDA=0) | Old Wonderware uses F21 exclusively |
| C-03 | MES → S2F49 (enhanced: OBJSPEC + CPACK) | S2F50(HCACK=0) | |
| C-04 | MES → S2F41 RCMD=UNKNOWN | S2F42(HCACK=1=InvalidCommand) |
6. Process programs (E30 §6.17)
| ID | Test | Expected | Common quirks |
|---|---|---|---|
| P-01 | MES → S7F19 (list) | S7F20 with all PPIDs from recipes: YAML |
|
| P-02 | MES → S7F5(PPID=RECIPE-A) | S7F6 with body (ASCII default; binary if recipe is binary) | Check whether your MES expects ASCII or Binary PPBODY |
| P-03 | MES → S7F1 → S7F2(Accept) → S7F3 (PP send, new PPID) | S7F4(ACKC7=0); recipe in store | |
| P-04 | MES → S7F17 (delete) | S7F18(ACKC7=0); recipe removed | |
| P-05 | MES → S7F23 (E42 formatted PP send) | S7F24(ACKC7=0) | Many MES don't speak E42; OK to skip if unused |
7. Terminal services (E30 §6.19)
| ID | Test | Expected | Common quirks |
|---|---|---|---|
| TS-01 | MES → S10F3 (terminal display single, host→equipment) | S10F4(ACKC10=0) | Confirm direction matches S10F3 host→equipment |
| TS-02 | MES → S10F5 (terminal display multi) | S10F6(ACKC10=0) | |
| TS-03 | Equipment → S10F1 (operator request, equipment→host) | MES → S10F2(ACKC10=0) | Some MES don't accept S10F1 at all; document policy |
8. GEM 300 streams (E40 / E87 / E94)
| ID | Test | Expected | Common quirks |
|---|---|---|---|
| J-01 | MES → S16F11 PRJobCreate (full body: MF / PPID / mtrloutspec / params) | S16F12(HCACK=0) | MES often uses MF=Substrate; check |
| J-02 | MES → S14F9 CreateControlJob with [PJ-1] | S14F10(OBJACK=0) | |
| J-03 | MES → S16F27 CJSTART | S16F28(HCACK=0); cascade through PJ states; S16F9 alerts on each | This is the heaviest dance — capture full trace |
| J-04 | Equipment fires S6F11(CEID=ControlJobExecuting=400) | MES → S6F12(ACKC6=0) | CEID 400/401 must be in ceids: YAML |
| J-05 | MES → S3F17 (CarrierAction=ProceedWithCarrier) | S3F18(CAACK=0) | |
| J-06 | MES → S3F19 (slot map verify) against carrier with stored map | S3F20(SMACK=0 if match, =1 if mismatch) | MES may not store maps; expect SMACK=NotRead |
9. Spool (E30 §6.22)
| ID | Test | Expected | Common quirks |
|---|---|---|---|
| SP-01 | MES → S2F43 (set spoolable streams to {5, 6}) | S2F44(RSPACK=0) | |
| SP-02 | Disconnect MES, equipment fires S5F1+S6F11 | Both queued in spool (size grows) | |
| SP-03 | Reconnect MES | Equipment auto-emits S6F25(queued count) | |
| SP-04 | MES → S6F23(RSDC=0=Transmit) | S6F24(RSDA=0=Accept), then spooled messages drain in order | |
| SP-05 | MES → S6F23(RSDC=1=Purge) | S6F24(RSDA=0); spool size → 0 |
10. Clock + ECs (E30 §6.16, §6.20)
| ID | Test | Expected | Common quirks |
|---|---|---|---|
| K-01 | MES → S2F17 (clock read) | S2F18(YYYYMMDDhhmmsscc) — 16 chars | Some MES expect 14-char; the library accepts both |
| K-02 | MES → S2F31 (clock set) | S2F32(TIACK=0=Accept) | Do this in a maintenance window only |
| K-03 | MES → S2F29 (EC namelist) | S2F30 with all ECIDs | |
| K-04 | MES → S2F13 (EC values) | S2F14 with current values | |
| K-05 | MES → S2F15 (EC set within range) | S2F16(EAC=0=Accept); value reflected on next S2F13 | |
| K-06 | MES → S2F15 (EC set OUT of range) | S2F16(EAC=1=Denied_OutOfRange) | Range enforced via min_str/max_str in equipment.yaml |
11. Soak (4-24 hours)
Once the punch list above passes, leave the connection up overnight with synthetic transactions every 60 s and a real recipe running periodically. Watch for:
- Memory growth in equipment process RSS (should be flat)
- Spool growing unbounded (something is filling it without draining)
- T-timer expirations in equipment logs (network-layer trouble)
- MES-side "stale tool" alarms (linktest reply lagging)
12. Pre-cutover checklist
Before promoting from staging to a real tool on the production fab floor:
- Every test ID above marked PASS or "N/A - documented reason"
- Wire traces archived for every test (90-day retention minimum)
secs_conformance --host <staging-tool>exits 0- 4-hour soak with no abnormal spool growth or T-timer expiries
- Dashboard panels for: spool depth, T-timer counter, alarm active count, FSM state gauges
- Runbook entries for every incident in README §10
- License agreement with the copyright holder signed (see LICENSE)
13. Known MES quirks worth pre-empting
Quirks observed in prior fab integrations. Not exhaustive.
- Camstar Opcenter: ALCD bit-7 sometimes inverted in their internal model; double-check the alarm wire trace. Linktest at 30 s default.
- FactoryWorks: Uses S2F21 (legacy) almost exclusively over
S2F41. Make sure your
host_commandsregistry routes both. - Wonderware MES: Doesn't reliably support S6F25; spool-drain flow may need explicit S6F23 from MES. Older versions don't speak E42 (S7F23 family) at all.
- Mozaic: Sometimes sends S2F33 with DATAID encoded as U1 where the spec allows U1-U8. The library's identifier parser accepts any unsigned width.
- Inficon FabGuard: Strict on S1F1 — expects MDLN and SOFTREV in ASCII even if the YAML uses U-types for MDLN elsewhere.
- CMNavigo: Expects equipment to initiate S1F13 within 5 s of
SELECT, before MES sends its own. Configure
Timers::linktest- a startup S1F13 emitter.
14. Reporting back
If you find an MES-specific bug that this codebase needs to handle,
file it via raphael@maenle.net with:
- MES name + build version
- The test ID from this doc
- Wire trace excerpt (pcap clip is fine)
- Expected vs actual behaviour
- Your equipment YAML + secs-gem commit SHA