raphael
60fa164626
02 — The cast of characters: equipment, EAP, MES, fab planner, AMHS, operator. Who initiates which conversation, why the equipment is the passive side of HSMS by convention, how the AMHS handshake is out-of-band relative to SECS. Cross-references the relevant namespace and test files for each actor. 03 — Vocabulary + a wafer's journey: follows one 300 mm wafer end-to-end through a fab and labels every SECS message and acronym that fires. Introduces SVID / DVID / ECID / CEID / RPTID / ALID / PPID / MDLN / SOFTREV / HCACK / ALCD / OFLACK / CAACK / SMACK / etc. in context rather than as a list. Includes one-screen reference tables for the remaining acknowledge codes, T-timers in all four contexts (HSMS / SECS-I / E84 / E30 communication state), and a stream-by-stream summary. Part 1 (Foundations) of the guided tour is now complete — a reader who reads chapters 01–03 can describe the protocol stack, identify the actors, and recognise every acronym they'll meet in Part 2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
225 lines
23 KiB
Markdown
225 lines
23 KiB
Markdown
# Real-MES interop test plan
|
|
|
|
The codebase cross-validates against [secsgem-py](https://pypi.org/project/secsgem/)
|
|
0.3.0 in CI. That's the Python reference implementation. It is **not
|
|
what a fab actually runs.** Real MES stacks include:
|
|
|
|
- **Camstar** (Siemens / Opcenter Execution Semiconductor)
|
|
- **FactoryWorks** (IBM, now various OEM ports)
|
|
- **Inficon FabGuard**
|
|
- **Wonderware MES** (AVEVA)
|
|
- **Mozaic / Mozaic Suite**
|
|
- **Critical Manufacturing CMNavigo**
|
|
- **Eyelit MES**
|
|
|
|
Each one ships its own SECS/GEM stack with its own quirks. This doc
|
|
is the structured day-1 protocol your integration team runs against
|
|
**your** MES before connecting a real tool. Treat it as a punch
|
|
list you tick off; capture wire traces from every step.
|
|
|
|
> **You can't skip this.** The in-repo `secs_conformance` harness +
|
|
> the `interop/` secsgem-py cross-validation prove the codebase is
|
|
> spec-conformant. They cannot prove the *combination* of (this
|
|
> codebase, your YAML config, your MES's choice of optional
|
|
> behaviours) works. Every gap surfaced in prior interop sweeps
|
|
> (S1F23/F24 missing, S10F3 direction confusion, lenient U-width
|
|
> parsing) was a real bug masked by passing internal tests.
|
|
|
|
## 0. Prerequisites
|
|
|
|
- A staging copy of the MES configured to talk to a single tool over
|
|
HSMS-SS or HSMS-GS. Get the MES team to point it at your test
|
|
equipment endpoint (port 5000 by default).
|
|
- A wire-trace collector running on the equipment side:
|
|
```sh
|
|
tcpdump -i any -w mes-day1-$(date +%Y%m%d).pcap 'tcp port 5000'
|
|
```
|
|
Keep this running across every test below. Wireshark with the
|
|
HSMS dissector (or the secsgem-py decoder in `interop/`) parses
|
|
these traces after the fact.
|
|
- The standard logger / metrics exporter wired up (see
|
|
INTEGRATION.md §6.4) — you want to see the `S9F*` self-emissions
|
|
and the FSM transitions land in your dashboard.
|
|
- The `--validate-config` flag run clean on your equipment YAMLs.
|
|
|
|
Record each test's outcome in a single spreadsheet — test ID,
|
|
date, MES build, equipment build, PASS/FAIL/PARTIAL, notes,
|
|
wire-trace timestamp. This is your audit trail for the integration.
|
|
|
|
## 1. HSMS transport plumbing (E37)
|
|
|
|
| ID | Test | Expected wire behaviour | Common quirks |
|
|
|-------|-------------------------------------------------------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------|
|
|
| T-01 | MES connects (active) → Select.req | Equipment replies Select.rsp(0=Ok); SELECTED on both sides | Some MES default session ID ≠ 0; override in our YAML |
|
|
| T-02 | Idle Linktest cycle (5 min observation) | Linktest.req every N seconds from MES; equipment Linktest.rsp | Camstar uses 30s default; FactoryWorks 60s |
|
|
| T-03 | MES sends Separate.req → graceful close | Equipment closes socket within 1s; no FIN_WAIT2 leak | Some MES expect equipment to close first |
|
|
| T-04 | Equipment-initiated Linktest (set `Timers::linktest` to 10s) | MES replies Linktest.rsp | Mozaic ignores equipment-initiated; not a bug |
|
|
| T-05 | MES disconnects TCP without Separate | Equipment detects closed socket, fires `closed_handler("eof")` | Watch for spool starting to fill |
|
|
| T-06 | MES reconnects after T-05 → S6F25 if spool has content | Equipment auto-emits S6F25 with queued count | MES must support S6F25/F26 — older Wonderware doesn't |
|
|
| T-07 | T3 violation: MES sends primary but never replies | Equipment fires T3 → auto-emits S9F9 | Confirm S9F9 appears in trace, not just our logs |
|
|
| T-08 | T7 violation: MES connects but never sends Select.req | Equipment closes after T7 with reason "T7" | |
|
|
| T-09 | Oversized frame: MES sends 17 MiB body | Equipment auto-emits S9F11 + closes | If MES doesn't generate this, skip |
|
|
|
|
## 2. Establish comms + identification (E30 Fundamental)
|
|
|
|
| ID | Test | Expected wire behaviour | Common quirks |
|
|
|-------|-------------------------------------------------------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------|
|
|
| E-01 | MES → S1F13 | Equipment → S1F14(COMMACK=Accept, [MDLN, SOFTREV]) | MES that send empty MDLN list — check YAML matches |
|
|
| E-02 | MES → S1F1 | Equipment → S1F2(MDLN, SOFTREV) | EQPTYP in S1F20 is sometimes confused with MDLN |
|
|
| E-03 | MES → S1F15 (request offline) | Equipment → S1F16(OFLACK=Accept), control state → HostOffline | |
|
|
| E-04 | MES → S1F17 (request online) | Equipment → S1F18(ONLACK=Accept), control state → OnlineRemote | |
|
|
| E-05 | MES → S1F19 (compliance request) | Equipment → S1F20 with full CCODE list matching `capabilities:` YAML | |
|
|
| E-06 | MES → S1F11 (SVID namelist), then S1F3 (values) | Namelist matches `svids:` YAML; values within type-declared ranges | MES may expect SVID 1 = ControlState string |
|
|
| E-07 | MES → S1F21 (DVID namelist), then S1F3 (values for DVID set) | Equipment returns DVIDs separately from SVIDs | Some MES treat S1F3 as union; spec says SVID-only |
|
|
| E-08 | MES → S1F23 (CEID namelist) | Equipment returns CEID→VIDs map matching `events.link_event_reports` | |
|
|
|
|
## 3. Dynamic event reports (E30 §6.6)
|
|
|
|
This is where most integrations break. MES often defines reports
|
|
in a different order than spec, expects different ACK enum
|
|
positions, or uses non-standard RPTID/DATAID widths.
|
|
|
|
| ID | Test | Expected | Common quirks |
|
|
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
|
| R-01 | MES → S2F33 (define report 1000 over 3 SVIDs) | S2F34(DRACK=0=Accept) | MES may use U1 for RPTID; our parser accepts widths |
|
|
| R-02 | MES → S2F35 (link CEID 300 ↔ RPTID 1000) | S2F36(LRACK=0=Accept) | |
|
|
| R-03 | MES → S2F37 (enable CEID 300) | S2F38(ERACK=0=Accept) | Some MES send empty CEID list = enable-all |
|
|
| R-04 | Equipment fires CEID 300 (driven by `model->compose_reports_for(300)`) | MES → S6F12(ACKC6=0) | MES may take >T6 to reply — extend our T6 if needed |
|
|
| R-05 | MES → S6F15 (event report request) | Equipment → S6F16 with current values | |
|
|
| R-06 | MES → S6F19/F21 (individual / annotated report request) | Equipment → S6F20 / S6F22 with current values | |
|
|
| R-07 | MES → S2F33 with DATAID=0 (clear all reports) | S2F34(DRACK=0); all link bindings flushed | Older MES use S2F33 + empty body; check both forms |
|
|
|
|
## 4. Alarms (E30 §6.14, E5 §13)
|
|
|
|
| ID | Test | Expected | Common quirks |
|
|
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
|
| A-01 | MES → S5F5 (list alarms) | S5F6 with full alarm directory matching `alarms:` YAML | |
|
|
| A-02 | MES → S5F3 (enable ALID 1) | S5F4(ACKC5=0) | |
|
|
| A-03 | MES → S5F7 (list enabled) | S5F8 contains ALID 1 | |
|
|
| A-04 | Equipment sets alarm 1 active | S5F1(ALCD=0x80 \| category) sent to MES → MES S5F2 ack | ALCD bit-7 must be SET, not cleared |
|
|
| A-05 | Equipment clears alarm 1 | S5F1(ALCD=0x00 \| category) → S5F2 ack | |
|
|
| A-06 | Equipment fires an alarm while alarm is *disabled* (S5F5 says no) | NO S5F1 wire frame — alarm registry tracks active, dispatcher gates | Easy to get wrong; covered by our tests |
|
|
|
|
## 5. Remote control (E30 §6.15)
|
|
|
|
| ID | Test | Expected | Common quirks |
|
|
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
|
| C-01 | MES → S2F41 (RCMD=START, params) | S2F42(HCACK=0=Accept) + emit configured CEID | |
|
|
| C-02 | MES → S2F21 (legacy: RCMD only, no params) | S2F22(CMDA=0) | Old Wonderware uses F21 exclusively |
|
|
| C-03 | MES → S2F49 (enhanced: OBJSPEC + CPACK) | S2F50(HCACK=0) | |
|
|
| C-04 | MES → S2F41 RCMD=UNKNOWN | S2F42(HCACK=1=InvalidCommand) | |
|
|
|
|
## 6. Process programs (E30 §6.17)
|
|
|
|
| ID | Test | Expected | Common quirks |
|
|
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
|
| P-01 | MES → S7F19 (list) | S7F20 with all PPIDs from `recipes:` YAML | |
|
|
| P-02 | MES → S7F5(PPID=RECIPE-A) | S7F6 with body (ASCII default; binary if recipe is binary) | PPBODY direction was our biggest interop bug |
|
|
| P-03 | MES → S7F1 → S7F2(Accept) → S7F3 (PP send, new PPID) | S7F4(ACKC7=0); recipe in store | |
|
|
| P-04 | MES → S7F17 (delete) | S7F18(ACKC7=0); recipe removed | |
|
|
| P-05 | MES → S7F23 (E42 formatted PP send) | S7F24(ACKC7=0) | Many MES don't speak E42; OK to skip if unused |
|
|
|
|
## 7. Terminal services (E30 §6.19)
|
|
|
|
| ID | Test | Expected | Common quirks |
|
|
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
|
| TS-01 | MES → S10F3 (terminal display single, host→equipment) | S10F4(ACKC10=0) | OUR codebase's bug fix — confirm direction is right |
|
|
| TS-02 | MES → S10F5 (terminal display multi) | S10F6(ACKC10=0) | |
|
|
| TS-03 | Equipment → S10F1 (operator request, equipment→host) | MES → S10F2(ACKC10=0) | Some MES don't accept S10F1 at all; document policy |
|
|
|
|
## 8. GEM 300 streams (E40 / E87 / E94)
|
|
|
|
| ID | Test | Expected | Common quirks |
|
|
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
|
| J-01 | MES → S16F11 PRJobCreate (full body: MF / PPID / mtrloutspec / params) | S16F12(HCACK=0) | MES often uses MF=Substrate; check |
|
|
| J-02 | MES → S14F9 CreateControlJob with [PJ-1] | S14F10(OBJACK=0) | |
|
|
| J-03 | MES → S16F27 CJSTART | S16F28(HCACK=0); cascade through PJ states; S16F9 alerts on each | This is the heaviest dance — capture full trace |
|
|
| J-04 | Equipment fires S6F11(CEID=ControlJobExecuting=400) | MES → S6F12(ACKC6=0) | CEID 400/401 must be in `ceids:` YAML |
|
|
| J-05 | MES → S3F17 (CarrierAction=ProceedWithCarrier) | S3F18(CAACK=0) | |
|
|
| J-06 | MES → S3F19 (slot map verify) against carrier with stored map | S3F20(SMACK=0 if match, =1 if mismatch) | MES may not store maps; expect SMACK=NotRead |
|
|
|
|
## 9. Spool (E30 §6.22)
|
|
|
|
| ID | Test | Expected | Common quirks |
|
|
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
|
| SP-01 | MES → S2F43 (set spoolable streams to {5, 6}) | S2F44(RSPACK=0) | |
|
|
| SP-02 | Disconnect MES, equipment fires S5F1+S6F11 | Both queued in spool (size grows) | |
|
|
| SP-03 | Reconnect MES | Equipment auto-emits S6F25(queued count) | |
|
|
| SP-04 | MES → S6F23(RSDC=0=Transmit) | S6F24(RSDA=0=Accept), then spooled messages drain in order | |
|
|
| SP-05 | MES → S6F23(RSDC=1=Purge) | S6F24(RSDA=0); spool size → 0 | |
|
|
|
|
## 10. Clock + ECs (E30 §6.16, §6.20)
|
|
|
|
| ID | Test | Expected | Common quirks |
|
|
|-------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------------------------------------|
|
|
| K-01 | MES → S2F17 (clock read) | S2F18(YYYYMMDDhhmmsscc) — 16 chars | Some MES expect 14-char; we accept both |
|
|
| K-02 | MES → S2F31 (clock set) | S2F32(TIACK=0=Accept) | Do this in a maintenance window only |
|
|
| K-03 | MES → S2F29 (EC namelist) | S2F30 with all ECIDs | |
|
|
| K-04 | MES → S2F13 (EC values) | S2F14 with current values | |
|
|
| K-05 | MES → S2F15 (EC set within range) | S2F16(EAC=0=Accept); value reflected on next S2F13 | |
|
|
| K-06 | MES → S2F15 (EC set OUT of range) | S2F16(EAC=1=Denied_OutOfRange) | Triggers our `min_str`/`max_str` range check |
|
|
|
|
## 11. Soak (4-24 hours)
|
|
|
|
Once the punch list above passes, leave the connection up overnight
|
|
with synthetic transactions every 60 s and a real recipe running
|
|
periodically. Watch for:
|
|
|
|
- Memory growth in equipment process RSS (should be flat)
|
|
- Spool growing unbounded (something is filling it without draining)
|
|
- T-timer expirations in equipment logs (network-layer trouble)
|
|
- MES-side "stale tool" alarms (linktest reply lagging)
|
|
|
|
## 12. Pre-cutover checklist
|
|
|
|
Before promoting from staging to a real tool on the production fab
|
|
floor:
|
|
|
|
- [ ] Every test ID above marked PASS or "N/A - documented reason"
|
|
- [ ] Wire traces archived for every test (90-day retention minimum)
|
|
- [ ] `secs_conformance --host <staging-tool>` exits 0
|
|
- [ ] 4-hour soak with no abnormal spool growth or T-timer expiries
|
|
- [ ] Dashboard panels for: spool depth, T-timer counter, alarm
|
|
active count, FSM state gauges
|
|
- [ ] Runbook entries for every incident in README §10
|
|
- [ ] License agreement with the copyright holder signed (see
|
|
[LICENSE](LICENSE))
|
|
|
|
## 13. Known MES quirks worth pre-empting
|
|
|
|
Compiled from prior fab integrations. Not exhaustive; treat as
|
|
search-priors when something doesn't behave the way you expect.
|
|
|
|
- **Camstar Opcenter**: ALCD bit-7 sometimes inverted in their
|
|
internal model; double-check the alarm wire trace. Linktest at
|
|
30 s default.
|
|
- **FactoryWorks**: Uses S2F21 (legacy) almost exclusively over
|
|
S2F41. Make sure your `host_commands` registry routes both.
|
|
- **Wonderware MES**: Doesn't reliably support S6F25; spool-drain
|
|
flow may need explicit S6F23 from MES. Older versions don't
|
|
speak E42 (S7F23 family) at all.
|
|
- **Mozaic**: Sometimes sends S2F33 with DATAID encoded as U1
|
|
where the spec allows U1-U8 — make sure our lenient parser is on
|
|
(it is by default since the secsgem-py interop work).
|
|
- **Inficon FabGuard**: Strict on S1F1 — expects MDLN and SOFTREV
|
|
in ASCII even if the YAML uses U-types for MDLN elsewhere.
|
|
- **CMNavigo**: Expects equipment to initiate S1F13 within 5 s of
|
|
SELECT, before MES sends its own. Configure `Timers::linktest`
|
|
+ a startup S1F13 emitter.
|
|
|
|
## 14. Reporting back
|
|
|
|
If you find an MES-specific bug that this codebase needs to handle,
|
|
file it via `raphael@maenle.net` with:
|
|
|
|
1. MES name + build version
|
|
2. The test ID from this doc
|
|
3. Wire trace excerpt (pcap clip is fine)
|
|
4. Expected vs actual behaviour
|
|
5. Your equipment YAML + secs-gem commit SHA
|
|
|
|
Bugs surfaced through this process are how we got
|
|
[S1F23/F24](interop/README.md), [S10F3 direction
|
|
fix](interop/README.md), and the lenient identifier-width parser.
|
|
The interop sweep is the gift that keeps giving.
|