# Real-MES interop test plan The codebase cross-validates against [secsgem-py](https://pypi.org/project/secsgem/) 0.3.0 in CI. That's the Python reference implementation. It is **not what a fab actually runs.** Real MES stacks include: - **Camstar** (Siemens / Opcenter Execution Semiconductor) - **FactoryWorks** (IBM, now various OEM ports) - **Inficon FabGuard** - **Wonderware MES** (AVEVA) - **Mozaic / Mozaic Suite** - **Critical Manufacturing CMNavigo** - **Eyelit MES** Each one ships its own SECS/GEM stack with its own quirks. This doc is the day-1 punch list your integration team runs against **your** MES before connecting a real tool. Capture wire traces from every step. The in-repo `secs_conformance` and secsgem-py cross-validation prove the codebase is spec-conformant; they cannot prove the combination of (this codebase, your YAML, your MES's choice of optional behaviours) works. ## 0. Prerequisites - A staging copy of the MES configured to talk to a single tool over HSMS-SS or HSMS-GS. Get the MES team to point it at your test equipment endpoint (port 5000 by default). - A wire-trace collector running on the equipment side: ```sh tcpdump -i any -w mes-day1-$(date +%Y%m%d).pcap 'tcp port 5000' ``` Keep this running across every test below. Wireshark with the HSMS dissector (or the secsgem-py decoder in `interop/`) parses these traces after the fact. - The standard logger / metrics exporter wired up (see INTEGRATION.md §6.4) — you want to see the `S9F*` self-emissions and the FSM transitions land in your dashboard. - The `--validate-config` flag run clean on your equipment YAMLs. Record each test's outcome in a single spreadsheet — test ID, date, MES build, equipment build, PASS/FAIL/PARTIAL, notes, wire-trace timestamp. This is your audit trail for the integration. ## 1. HSMS transport plumbing (E37) | ID | Test | Expected wire behaviour | Common quirks | |-------|-------------------------------------------------------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------| | T-01 | MES connects (active) → Select.req | Equipment replies Select.rsp(0=Ok); SELECTED on both sides | Some MES default session ID ≠ 0; override in equipment.yaml | | T-02 | Idle Linktest cycle (5 min observation) | Linktest.req every N seconds from MES; equipment Linktest.rsp | Camstar uses 30s default; FactoryWorks 60s | | T-03 | MES sends Separate.req → graceful close | Equipment closes socket within 1s; no FIN_WAIT2 leak | Some MES expect equipment to close first | | T-04 | Equipment-initiated Linktest (set `Timers::linktest` to 10s) | MES replies Linktest.rsp | Mozaic ignores equipment-initiated; not a bug | | T-05 | MES disconnects TCP without Separate | Equipment detects closed socket, fires `closed_handler("eof")` | Watch for spool starting to fill | | T-06 | MES reconnects after T-05 → S6F25 if spool has content | Equipment auto-emits S6F25 with queued count | MES must support S6F25/F26 — older Wonderware doesn't | | T-07 | T3 violation: MES sends primary but never replies | Equipment fires T3 → auto-emits S9F9 | Confirm S9F9 appears in the wire trace, not just logs | | T-08 | T7 violation: MES connects but never sends Select.req | Equipment closes after T7 with reason "T7" | | | T-09 | Oversized frame: MES sends 17 MiB body | Equipment auto-emits S9F11 + closes | If MES doesn't generate this, skip | ## 2. Establish comms + identification (E30 Fundamental) | ID | Test | Expected wire behaviour | Common quirks | |-------|-------------------------------------------------------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------| | E-01 | MES → S1F13 | Equipment → S1F14(COMMACK=Accept, [MDLN, SOFTREV]) | MES that send empty MDLN list — check YAML matches | | E-02 | MES → S1F1 | Equipment → S1F2(MDLN, SOFTREV) | EQPTYP in S1F20 is sometimes confused with MDLN | | E-03 | MES → S1F15 (request offline) | Equipment → S1F16(OFLACK=Accept), control state → HostOffline | | | E-04 | MES → S1F17 (request online) | Equipment → S1F18(ONLACK=Accept), control state → OnlineRemote | | | E-05 | MES → S1F19 (compliance request) | Equipment → S1F20 with full CCODE list matching `capabilities:` YAML | | | E-06 | MES → S1F11 (SVID namelist), then S1F3 (values) | Namelist matches `svids:` YAML; values within type-declared ranges | MES may expect SVID 1 = ControlState string | | E-07 | MES → S1F21 (DVID namelist), then S1F3 (values for DVID set) | Equipment returns DVIDs separately from SVIDs | Some MES treat S1F3 as union; spec says SVID-only | | E-08 | MES → S1F23 (CEID namelist) | Equipment returns CEID→VIDs map matching `events.link_event_reports` | | ## 3. Dynamic event reports (E30 §6.6) This is where most integrations break. MES often defines reports in a different order than spec, expects different ACK enum positions, or uses non-standard RPTID/DATAID widths. | ID | Test | Expected | Common quirks | |-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| | R-01 | MES → S2F33 (define report 1000 over 3 SVIDs) | S2F34(DRACK=0=Accept) | MES may use U1 for RPTID; the parser accepts any width| | R-02 | MES → S2F35 (link CEID 300 ↔ RPTID 1000) | S2F36(LRACK=0=Accept) | | | R-03 | MES → S2F37 (enable CEID 300) | S2F38(ERACK=0=Accept) | Some MES send empty CEID list = enable-all | | R-04 | Equipment fires CEID 300 (driven by `model->compose_reports_for(300)`) | MES → S6F12(ACKC6=0) | MES may take >T6 to reply — extend T6 if needed | | R-05 | MES → S6F15 (event report request) | Equipment → S6F16 with current values | | | R-06 | MES → S6F19/F21 (individual / annotated report request) | Equipment → S6F20 / S6F22 with current values | | | R-07 | MES → S2F33 with DATAID=0 (clear all reports) | S2F34(DRACK=0); all link bindings flushed | Older MES use S2F33 + empty body; check both forms | ## 4. Alarms (E30 §6.14, E5 §13) | ID | Test | Expected | Common quirks | |-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| | A-01 | MES → S5F5 (list alarms) | S5F6 with full alarm directory matching `alarms:` YAML | | | A-02 | MES → S5F3 (enable ALID 1) | S5F4(ACKC5=0) | | | A-03 | MES → S5F7 (list enabled) | S5F8 contains ALID 1 | | | A-04 | Equipment sets alarm 1 active | S5F1(ALCD=0x80 \| category) sent to MES → MES S5F2 ack | ALCD bit-7 must be SET, not cleared | | A-05 | Equipment clears alarm 1 | S5F1(ALCD=0x00 \| category) → S5F2 ack | | | A-06 | Equipment fires an alarm while alarm is *disabled* (S5F5 says no) | NO S5F1 wire frame — alarm registry tracks active, dispatcher gates | | ## 5. Remote control (E30 §6.15) | ID | Test | Expected | Common quirks | |-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| | C-01 | MES → S2F41 (RCMD=START, params) | S2F42(HCACK=0=Accept) + emit configured CEID | | | C-02 | MES → S2F21 (legacy: RCMD only, no params) | S2F22(CMDA=0) | Old Wonderware uses F21 exclusively | | C-03 | MES → S2F49 (enhanced: OBJSPEC + CPACK) | S2F50(HCACK=0) | | | C-04 | MES → S2F41 RCMD=UNKNOWN | S2F42(HCACK=1=InvalidCommand) | | ## 6. Process programs (E30 §6.17) | ID | Test | Expected | Common quirks | |-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| | P-01 | MES → S7F19 (list) | S7F20 with all PPIDs from `recipes:` YAML | | | P-02 | MES → S7F5(PPID=RECIPE-A) | S7F6 with body (ASCII default; binary if recipe is binary) | Check whether your MES expects ASCII or Binary PPBODY | | P-03 | MES → S7F1 → S7F2(Accept) → S7F3 (PP send, new PPID) | S7F4(ACKC7=0); recipe in store | | | P-04 | MES → S7F17 (delete) | S7F18(ACKC7=0); recipe removed | | | P-05 | MES → S7F23 (E42 formatted PP send) | S7F24(ACKC7=0) | Many MES don't speak E42; OK to skip if unused | ## 7. Terminal services (E30 §6.19) | ID | Test | Expected | Common quirks | |-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| | TS-01 | MES → S10F3 (terminal display single, host→equipment) | S10F4(ACKC10=0) | Confirm direction matches S10F3 host→equipment | | TS-02 | MES → S10F5 (terminal display multi) | S10F6(ACKC10=0) | | | TS-03 | Equipment → S10F1 (operator request, equipment→host) | MES → S10F2(ACKC10=0) | Some MES don't accept S10F1 at all; document policy | ## 8. GEM 300 streams (E40 / E87 / E94) | ID | Test | Expected | Common quirks | |-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| | J-01 | MES → S16F11 PRJobCreate (full body: MF / PPID / mtrloutspec / params) | S16F12(HCACK=0) | MES often uses MF=Substrate; check | | J-02 | MES → S14F9 CreateControlJob with [PJ-1] | S14F10(OBJACK=0) | | | J-03 | MES → S16F27 CJSTART | S16F28(HCACK=0); cascade through PJ states; S16F9 alerts on each | This is the heaviest dance — capture full trace | | J-04 | Equipment fires S6F11(CEID=ControlJobExecuting=400) | MES → S6F12(ACKC6=0) | CEID 400/401 must be in `ceids:` YAML | | J-05 | MES → S3F17 (CarrierAction=ProceedWithCarrier) | S3F18(CAACK=0) | | | J-06 | MES → S3F19 (slot map verify) against carrier with stored map | S3F20(SMACK=0 if match, =1 if mismatch) | MES may not store maps; expect SMACK=NotRead | ## 9. Spool (E30 §6.22) | ID | Test | Expected | Common quirks | |-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| | SP-01 | MES → S2F43 (set spoolable streams to {5, 6}) | S2F44(RSPACK=0) | | | SP-02 | Disconnect MES, equipment fires S5F1+S6F11 | Both queued in spool (size grows) | | | SP-03 | Reconnect MES | Equipment auto-emits S6F25(queued count) | | | SP-04 | MES → S6F23(RSDC=0=Transmit) | S6F24(RSDA=0=Accept), then spooled messages drain in order | | | SP-05 | MES → S6F23(RSDC=1=Purge) | S6F24(RSDA=0); spool size → 0 | | ## 10. Clock + ECs (E30 §6.16, §6.20) | ID | Test | Expected | Common quirks | |-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| | K-01 | MES → S2F17 (clock read) | S2F18(YYYYMMDDhhmmsscc) — 16 chars | Some MES expect 14-char; the library accepts both | | K-02 | MES → S2F31 (clock set) | S2F32(TIACK=0=Accept) | Do this in a maintenance window only | | K-03 | MES → S2F29 (EC namelist) | S2F30 with all ECIDs | | | K-04 | MES → S2F13 (EC values) | S2F14 with current values | | | K-05 | MES → S2F15 (EC set within range) | S2F16(EAC=0=Accept); value reflected on next S2F13 | | | K-06 | MES → S2F15 (EC set OUT of range) | S2F16(EAC=1=Denied_OutOfRange) | Range enforced via `min_str`/`max_str` in equipment.yaml | ## 11. Soak (4-24 hours) Once the punch list above passes, leave the connection up overnight with synthetic transactions every 60 s and a real recipe running periodically. Watch for: - Memory growth in equipment process RSS (should be flat) - Spool growing unbounded (something is filling it without draining) - T-timer expirations in equipment logs (network-layer trouble) - MES-side "stale tool" alarms (linktest reply lagging) ## 12. Pre-cutover checklist Before promoting from staging to a real tool on the production fab floor: - [ ] Every test ID above marked PASS or "N/A - documented reason" - [ ] Wire traces archived for every test (90-day retention minimum) - [ ] `secs_conformance --host ` exits 0 - [ ] 4-hour soak with no abnormal spool growth or T-timer expiries - [ ] Dashboard panels for: spool depth, T-timer counter, alarm active count, FSM state gauges - [ ] Runbook entries for every incident in README §10 - [ ] License agreement with the copyright holder signed (see [LICENSE](LICENSE)) ## 13. Known MES quirks worth pre-empting Quirks observed in prior fab integrations. Not exhaustive. - **Camstar Opcenter**: ALCD bit-7 sometimes inverted in their internal model; double-check the alarm wire trace. Linktest at 30 s default. - **FactoryWorks**: Uses S2F21 (legacy) almost exclusively over S2F41. Make sure your `host_commands` registry routes both. - **Wonderware MES**: Doesn't reliably support S6F25; spool-drain flow may need explicit S6F23 from MES. Older versions don't speak E42 (S7F23 family) at all. - **Mozaic**: Sometimes sends S2F33 with DATAID encoded as U1 where the spec allows U1-U8. The library's identifier parser accepts any unsigned width. - **Inficon FabGuard**: Strict on S1F1 — expects MDLN and SOFTREV in ASCII even if the YAML uses U-types for MDLN elsewhere. - **CMNavigo**: Expects equipment to initiate S1F13 within 5 s of SELECT, before MES sends its own. Configure `Timers::linktest` + a startup S1F13 emitter. ## 14. Reporting back If you find an MES-specific bug that this codebase needs to handle, file it via `raphael@maenle.net` with: 1. MES name + build version 2. The test ID from this doc 3. Wire trace excerpt (pcap clip is fine) 4. Expected vs actual behaviour 5. Your equipment YAML + secs-gem commit SHA