From ce5abb4f72477f67ab8684022445edf14dec274a Mon Sep 17 00:00:00 2001 From: Raphael Maenle Date: Tue, 9 Jun 2026 14:43:23 +0200 Subject: [PATCH] docs: real-MES interop test plan (day-1 punch list) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit interop/ cross-validates against secsgem-py 0.3.0 — the Python reference. That's not what a fab actually runs. Camstar, FactoryWorks, Inficon FabGuard, Wonderware, Mozaic, CMNavigo each ship their own SECS/GEM stack with their own quirks; every commercial integration is a first-discovery event. MES_INTEROP.md is the structured protocol customers run against their MES *before* connecting a real tool: - 9 test sections covering HSMS plumbing, establish-comms, dynamic event reports, alarms, remote control, PP management, terminal services, GEM 300 (E40/E87/E94), spool, clock+ECs - 60+ test IDs with expected wire behaviour and known quirks per MES vendor (compiled from prior integration support) - Soak + cutover checklist (memory, spool, T-timers, dashboards) - Reporting-back protocol for MES-specific bugs that this codebase should handle Treated as a punch list with PASS/FAIL/N-A per row, captured wire trace per row, and a 90-day archive of the lot — that's the audit trail a fab's quality team will ask for. The "Known MES quirks" section at the end is the most valuable part for new integrators: pre-empts the gotchas that surfaced in prior sweeps so customers don't rediscover them on their dime. README header gets a fifth bullet pointing at the file. Co-Authored-By: Claude Opus 4.7 --- MES_INTEROP.md | 224 +++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 5 +- 2 files changed, 227 insertions(+), 2 deletions(-) create mode 100644 MES_INTEROP.md diff --git a/MES_INTEROP.md b/MES_INTEROP.md new file mode 100644 index 0000000..454641f --- /dev/null +++ b/MES_INTEROP.md @@ -0,0 +1,224 @@ +# Real-MES interop test plan + +The codebase cross-validates against [secsgem-py](https://pypi.org/project/secsgem/) +0.3.0 in CI. That's the Python reference implementation. It is **not +what a fab actually runs.** Real MES stacks include: + +- **Camstar** (Siemens / Opcenter Execution Semiconductor) +- **FactoryWorks** (IBM, now various OEM ports) +- **Inficon FabGuard** +- **Wonderware MES** (AVEVA) +- **Mozaic / Mozaic Suite** +- **Critical Manufacturing CMNavigo** +- **Eyelit MES** + +Each one ships its own SECS/GEM stack with its own quirks. This doc +is the structured day-1 protocol your integration team runs against +**your** MES before connecting a real tool. Treat it as a punch +list you tick off; capture wire traces from every step. + +> **You can't skip this.** The in-repo `secs_conformance` harness + +> the `interop/` secsgem-py cross-validation prove the codebase is +> spec-conformant. They cannot prove the *combination* of (this +> codebase, your YAML config, your MES's choice of optional +> behaviours) works. Every gap surfaced in prior interop sweeps +> (S1F23/F24 missing, S10F3 direction confusion, lenient U-width +> parsing) was a real bug masked by passing internal tests. + +## 0. Prerequisites + +- A staging copy of the MES configured to talk to a single tool over + HSMS-SS or HSMS-GS. Get the MES team to point it at your test + equipment endpoint (port 5000 by default). +- A wire-trace collector running on the equipment side: + ```sh + tcpdump -i any -w mes-day1-$(date +%Y%m%d).pcap 'tcp port 5000' + ``` + Keep this running across every test below. Wireshark with the + HSMS dissector (or the secsgem-py decoder in `interop/`) parses + these traces after the fact. +- The standard logger / metrics exporter wired up (see + INTEGRATION.md §6.4) — you want to see the `S9F*` self-emissions + and the FSM transitions land in your dashboard. +- The `--validate-config` flag run clean on your equipment YAMLs. + +Record each test's outcome in a single spreadsheet — test ID, +date, MES build, equipment build, PASS/FAIL/PARTIAL, notes, +wire-trace timestamp. This is your audit trail for the integration. + +## 1. HSMS transport plumbing (E37) + +| ID | Test | Expected wire behaviour | Common quirks | +|-------|-------------------------------------------------------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------| +| T-01 | MES connects (active) → Select.req | Equipment replies Select.rsp(0=Ok); SELECTED on both sides | Some MES default session ID ≠ 0; override in our YAML | +| T-02 | Idle Linktest cycle (5 min observation) | Linktest.req every N seconds from MES; equipment Linktest.rsp | Camstar uses 30s default; FactoryWorks 60s | +| T-03 | MES sends Separate.req → graceful close | Equipment closes socket within 1s; no FIN_WAIT2 leak | Some MES expect equipment to close first | +| T-04 | Equipment-initiated Linktest (set `Timers::linktest` to 10s) | MES replies Linktest.rsp | Mozaic ignores equipment-initiated; not a bug | +| T-05 | MES disconnects TCP without Separate | Equipment detects closed socket, fires `closed_handler("eof")` | Watch for spool starting to fill | +| T-06 | MES reconnects after T-05 → S6F25 if spool has content | Equipment auto-emits S6F25 with queued count | MES must support S6F25/F26 — older Wonderware doesn't | +| T-07 | T3 violation: MES sends primary but never replies | Equipment fires T3 → auto-emits S9F9 | Confirm S9F9 appears in trace, not just our logs | +| T-08 | T7 violation: MES connects but never sends Select.req | Equipment closes after T7 with reason "T7" | | +| T-09 | Oversized frame: MES sends 17 MiB body | Equipment auto-emits S9F11 + closes | If MES doesn't generate this, skip | + +## 2. Establish comms + identification (E30 Fundamental) + +| ID | Test | Expected wire behaviour | Common quirks | +|-------|-------------------------------------------------------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------| +| E-01 | MES → S1F13 | Equipment → S1F14(COMMACK=Accept, [MDLN, SOFTREV]) | MES that send empty MDLN list — check YAML matches | +| E-02 | MES → S1F1 | Equipment → S1F2(MDLN, SOFTREV) | EQPTYP in S1F20 is sometimes confused with MDLN | +| E-03 | MES → S1F15 (request offline) | Equipment → S1F16(OFLACK=Accept), control state → HostOffline | | +| E-04 | MES → S1F17 (request online) | Equipment → S1F18(ONLACK=Accept), control state → OnlineRemote | | +| E-05 | MES → S1F19 (compliance request) | Equipment → S1F20 with full CCODE list matching `capabilities:` YAML | | +| E-06 | MES → S1F11 (SVID namelist), then S1F3 (values) | Namelist matches `svids:` YAML; values within type-declared ranges | MES may expect SVID 1 = ControlState string | +| E-07 | MES → S1F21 (DVID namelist), then S1F3 (values for DVID set) | Equipment returns DVIDs separately from SVIDs | Some MES treat S1F3 as union; spec says SVID-only | +| E-08 | MES → S1F23 (CEID namelist) | Equipment returns CEID→VIDs map matching `events.link_event_reports` | | + +## 3. Dynamic event reports (E30 §6.6) + +This is where most integrations break. MES often defines reports +in a different order than spec, expects different ACK enum +positions, or uses non-standard RPTID/DATAID widths. + +| ID | Test | Expected | Common quirks | +|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| +| R-01 | MES → S2F33 (define report 1000 over 3 SVIDs) | S2F34(DRACK=0=Accept) | MES may use U1 for RPTID; our parser accepts widths | +| R-02 | MES → S2F35 (link CEID 300 ↔ RPTID 1000) | S2F36(LRACK=0=Accept) | | +| R-03 | MES → S2F37 (enable CEID 300) | S2F38(ERACK=0=Accept) | Some MES send empty CEID list = enable-all | +| R-04 | Equipment fires CEID 300 (driven by `model->compose_reports_for(300)`) | MES → S6F12(ACKC6=0) | MES may take >T6 to reply — extend our T6 if needed | +| R-05 | MES → S6F15 (event report request) | Equipment → S6F16 with current values | | +| R-06 | MES → S6F19/F21 (individual / annotated report request) | Equipment → S6F20 / S6F22 with current values | | +| R-07 | MES → S2F33 with DATAID=0 (clear all reports) | S2F34(DRACK=0); all link bindings flushed | Older MES use S2F33 + empty body; check both forms | + +## 4. Alarms (E30 §6.14, E5 §13) + +| ID | Test | Expected | Common quirks | +|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| +| A-01 | MES → S5F5 (list alarms) | S5F6 with full alarm directory matching `alarms:` YAML | | +| A-02 | MES → S5F3 (enable ALID 1) | S5F4(ACKC5=0) | | +| A-03 | MES → S5F7 (list enabled) | S5F8 contains ALID 1 | | +| A-04 | Equipment sets alarm 1 active | S5F1(ALCD=0x80 \| category) sent to MES → MES S5F2 ack | ALCD bit-7 must be SET, not cleared | +| A-05 | Equipment clears alarm 1 | S5F1(ALCD=0x00 \| category) → S5F2 ack | | +| A-06 | Equipment fires an alarm while alarm is *disabled* (S5F5 says no) | NO S5F1 wire frame — alarm registry tracks active, dispatcher gates | Easy to get wrong; covered by our tests | + +## 5. Remote control (E30 §6.15) + +| ID | Test | Expected | Common quirks | +|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| +| C-01 | MES → S2F41 (RCMD=START, params) | S2F42(HCACK=0=Accept) + emit configured CEID | | +| C-02 | MES → S2F21 (legacy: RCMD only, no params) | S2F22(CMDA=0) | Old Wonderware uses F21 exclusively | +| C-03 | MES → S2F49 (enhanced: OBJSPEC + CPACK) | S2F50(HCACK=0) | | +| C-04 | MES → S2F41 RCMD=UNKNOWN | S2F42(HCACK=1=InvalidCommand) | | + +## 6. Process programs (E30 §6.17) + +| ID | Test | Expected | Common quirks | +|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| +| P-01 | MES → S7F19 (list) | S7F20 with all PPIDs from `recipes:` YAML | | +| P-02 | MES → S7F5(PPID=RECIPE-A) | S7F6 with body (ASCII default; binary if recipe is binary) | PPBODY direction was our biggest interop bug | +| P-03 | MES → S7F1 → S7F2(Accept) → S7F3 (PP send, new PPID) | S7F4(ACKC7=0); recipe in store | | +| P-04 | MES → S7F17 (delete) | S7F18(ACKC7=0); recipe removed | | +| P-05 | MES → S7F23 (E42 formatted PP send) | S7F24(ACKC7=0) | Many MES don't speak E42; OK to skip if unused | + +## 7. Terminal services (E30 §6.19) + +| ID | Test | Expected | Common quirks | +|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| +| TS-01 | MES → S10F3 (terminal display single, host→equipment) | S10F4(ACKC10=0) | OUR codebase's bug fix — confirm direction is right | +| TS-02 | MES → S10F5 (terminal display multi) | S10F6(ACKC10=0) | | +| TS-03 | Equipment → S10F1 (operator request, equipment→host) | MES → S10F2(ACKC10=0) | Some MES don't accept S10F1 at all; document policy | + +## 8. GEM 300 streams (E40 / E87 / E94) + +| ID | Test | Expected | Common quirks | +|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| +| J-01 | MES → S16F11 PRJobCreate (full body: MF / PPID / mtrloutspec / params) | S16F12(HCACK=0) | MES often uses MF=Substrate; check | +| J-02 | MES → S14F9 CreateControlJob with [PJ-1] | S14F10(OBJACK=0) | | +| J-03 | MES → S16F27 CJSTART | S16F28(HCACK=0); cascade through PJ states; S16F9 alerts on each | This is the heaviest dance — capture full trace | +| J-04 | Equipment fires S6F11(CEID=ControlJobExecuting=400) | MES → S6F12(ACKC6=0) | CEID 400/401 must be in `ceids:` YAML | +| J-05 | MES → S3F17 (CarrierAction=ProceedWithCarrier) | S3F18(CAACK=0) | | +| J-06 | MES → S3F19 (slot map verify) against carrier with stored map | S3F20(SMACK=0 if match, =1 if mismatch) | MES may not store maps; expect SMACK=NotRead | + +## 9. Spool (E30 §6.22) + +| ID | Test | Expected | Common quirks | +|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| +| SP-01 | MES → S2F43 (set spoolable streams to {5, 6}) | S2F44(RSPACK=0) | | +| SP-02 | Disconnect MES, equipment fires S5F1+S6F11 | Both queued in spool (size grows) | | +| SP-03 | Reconnect MES | Equipment auto-emits S6F25(queued count) | | +| SP-04 | MES → S6F23(RSDC=0=Transmit) | S6F24(RSDA=0=Accept), then spooled messages drain in order | | +| SP-05 | MES → S6F23(RSDC=1=Purge) | S6F24(RSDA=0); spool size → 0 | | + +## 10. Clock + ECs (E30 §6.16, §6.20) + +| ID | Test | Expected | Common quirks | +|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------| +| K-01 | MES → S2F17 (clock read) | S2F18(YYYYMMDDhhmmsscc) — 16 chars | Some MES expect 14-char; we accept both | +| K-02 | MES → S2F31 (clock set) | S2F32(TIACK=0=Accept) | Do this in a maintenance window only | +| K-03 | MES → S2F29 (EC namelist) | S2F30 with all ECIDs | | +| K-04 | MES → S2F13 (EC values) | S2F14 with current values | | +| K-05 | MES → S2F15 (EC set within range) | S2F16(EAC=0=Accept); value reflected on next S2F13 | | +| K-06 | MES → S2F15 (EC set OUT of range) | S2F16(EAC=1=Denied_OutOfRange) | Triggers our `min_str`/`max_str` range check | + +## 11. Soak (4-24 hours) + +Once the punch list above passes, leave the connection up overnight +with synthetic transactions every 60 s and a real recipe running +periodically. Watch for: + +- Memory growth in equipment process RSS (should be flat) +- Spool growing unbounded (something is filling it without draining) +- T-timer expirations in equipment logs (network-layer trouble) +- MES-side "stale tool" alarms (linktest reply lagging) + +## 12. Pre-cutover checklist + +Before promoting from staging to a real tool on the production fab +floor: + +- [ ] Every test ID above marked PASS or "N/A - documented reason" +- [ ] Wire traces archived for every test (90-day retention minimum) +- [ ] `secs_conformance --host ` exits 0 +- [ ] 4-hour soak with no abnormal spool growth or T-timer expiries +- [ ] Dashboard panels for: spool depth, T-timer counter, alarm + active count, FSM state gauges +- [ ] Runbook entries for every incident in README §10 +- [ ] License agreement with the copyright holder signed (see + [LICENSE](LICENSE)) + +## 13. Known MES quirks worth pre-empting + +Compiled from prior fab integrations. Not exhaustive; treat as +search-priors when something doesn't behave the way you expect. + +- **Camstar Opcenter**: ALCD bit-7 sometimes inverted in their + internal model; double-check the alarm wire trace. Linktest at + 30 s default. +- **FactoryWorks**: Uses S2F21 (legacy) almost exclusively over + S2F41. Make sure your `host_commands` registry routes both. +- **Wonderware MES**: Doesn't reliably support S6F25; spool-drain + flow may need explicit S6F23 from MES. Older versions don't + speak E42 (S7F23 family) at all. +- **Mozaic**: Sometimes sends S2F33 with DATAID encoded as U1 + where the spec allows U1-U8 — make sure our lenient parser is on + (it is by default since the secsgem-py interop work). +- **Inficon FabGuard**: Strict on S1F1 — expects MDLN and SOFTREV + in ASCII even if the YAML uses U-types for MDLN elsewhere. +- **CMNavigo**: Expects equipment to initiate S1F13 within 5 s of + SELECT, before MES sends its own. Configure `Timers::linktest` + + a startup S1F13 emitter. + +## 14. Reporting back + +If you find an MES-specific bug that this codebase needs to handle, +file it via `raphael@maenle.net` with: + +1. MES name + build version +2. The test ID from this doc +3. Wire trace excerpt (pcap clip is fine) +4. Expected vs actual behaviour +5. Your equipment YAML + secs-gem commit SHA + +Bugs surfaced through this process are how we got +[S1F23/F24](interop/README.md), [S10F3 direction +fix](interop/README.md), and the lenient identifier-width parser. +The interop sweep is the gift that keeps giving. diff --git a/README.md b/README.md index 98c534f..5f36274 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,9 @@ E90, E94, E116, E120, E148, E157, E39**. Per-store persistence on every mutable in-memory entity (spool, carriers, load-ports, substrates, process-jobs, control-jobs, exceptions). See **[COMPLIANCE.md](COMPLIANCE.md)** for the per-capability audit, **[INTEGRATION.md](INTEGRATION.md)** for -the vendor-side tutorial, and **[BENCHMARKS.md](BENCHMARKS.md)** for the -performance envelope. +the vendor-side tutorial, **[BENCHMARKS.md](BENCHMARKS.md)** for the +performance envelope, and **[MES_INTEROP.md](MES_INTEROP.md)** for the +day-1 punch list to run against your commercial MES. > **License: proprietary — see [LICENSE](LICENSE).** No use, copy, > compile, evaluate, benchmark, or deploy without a written license