secs-gem/docs/FAQ.md

# FAQ

Questions we hear once per integration.

## Why is HSMS unencrypted?

Because SEMI E37 says so.  HSMS is plain TCP with a 14-byte
framing header — no TLS, no auth, no nonces.  Every commercial MES
on the market speaks exactly that wire, and changing it would make
us incompatible with all of them.  Encryption and authentication
belong at the network layer: see [SECURITY.md](SECURITY.md) for
the stunnel.conf + nftables setup that wraps the unencrypted TCP
in mTLS without modifying the wire protocol.

## What's the difference between SVID and DVID?

**SVID** is a *status* variable — equipment state the host queries
(chamber pressure, current control state, wafer counter).
**DVID** is a *data* variable — intermediate values, typically
computed or sensor-derived, that aren't part of the equipment's
state model.

In practice fab tools blur the line.  The library treats them
identically except for which message reports them: `S1F3 / S1F11`
for SVIDs, `S1F21 / S1F22` for DVIDs.  Variable lookups by VID
span both (`EquipmentDataModel::vid_value`).

## Do I really need all four YAML files?

Yes for production; no for a quick "does it compile":

- `equipment.yaml` — your tool's data dictionary.  Required.
- `control_state.yaml` — the E30 control state machine (HostOffline,
  AttemptOnline, OnlineRemote, …).  The default in `data/` works as
  a starting point; you may customize transitions.
- `process_job_state.yaml` — the E40 PJ FSM.  Default is spec-typical;
  customize only if your tool has unusual recipe semantics.
- `control_job_state.yaml` — the E94 CJ FSM.  Same.

`secs_server --validate-config` checks all four in one pass and
exits 0 / 1.  Run it in CI on every config change.

## PJ vs CJ — what's the difference?

A **PJ** (E40 Process Job) is "process this batch of material with
this recipe."  One PJ = one recipe run = one set of wafers.  It
has its own FSM (Queued → SettingUp → Processing → ProcessComplete).

A **CJ** (E94 Control Job) is "execute these PJs in order, as a
unit, with start/pause/abort semantics."  A CJ owns an ordered list
of PRJOBIDs.  When the host issues `CJSTART`, the CJ promotes its
PJs through their lifecycles.

You typically need both: the MES creates a CJ containing N PJs,
then starts the CJ.  PJs without a CJ are legal — they just sit in
Queued waiting for someone to select them — but most MES drives
batches through CJs.

## Who fires FSM transitions — the library or my code?

**Your code.**  The library implements the FSMs (legal transitions,
validation, persistence) but it doesn't know when a wafer was
actually loaded or when a recipe step finished — those signals come
from your tool.  The pattern across every store is:

```cpp
// You fire the event; the FSM validates + transitions + emits.
model->process_jobs.fire_internal("PJ-1", gem::ProcessJobEvent::SetupComplete);
model->carriers.fire_id_event("CAR-A1B2", gem::CarrierIDEvent::Read);
```

Host commands (`S2F41` RCMD=START, `S16F5` PRJSTART, `S16F27` CJSTART)
arrive via the wire and get dispatched into your registered
handlers; the handler typically calls `fire_internal` or
`on_host_command` on the relevant store.

See INTEGRATION.md §4 for the worked patterns.

## What runs on which thread?

**Everything that touches the data model runs on the io_context
thread.**  There are no locks in `EquipmentDataModel`.

- The Router dispatch (incoming wire messages) — on the io_context.
- All `set_*_change_handler` callbacks — on the io_context.
- Periodic timers you register via asio — on the io_context.

If your code lives on another thread (typical for sensor polling),
marshal updates via `asio::post`:

```cpp
asio::post(io.get_executor(), [model, value] {
  model->svids.set_value(100, secs2::Item::f4(value));
});
```

INTEGRATION.md §3 has the full thread-safety contract.

## How do I add a new SECS-II message?

Edit `data/messages.yaml`, add a row, rebuild.  The codegen
(`tools/gen_messages.py`) emits a typed builder + parser into
`messages.hpp`.  Then register a Router handler in your `main.cpp`
for the new `(stream, function)` pair.  See README "Adding a
capability" or ARCHITECTURE.md for the full walkthrough.

## What's the difference between `Item::ascii("X")` and `Item::binary({'X'})`?

The wire format byte differs — `0x41 01 58` for ASCII vs
`0x21 01 58` for Binary.  Some peers (notably secsgem-py) default
PPBODY to ASCII; others use Binary.  Our codec accepts either via
the `BINARY_OR_ASCII` codegen type for fields the spec lists as
`ASCII | Binary | List` (the PPBODY case in S7F3/F6).

For most fields it doesn't matter — pick the format that matches
your data semantically.

## My MES sends a message that worked in `interop` but fails in production. What's going on?

Three usual suspects:

1. **U-width.**  Your MES is sending `DATAID` as U1 but our handler
   was strict for U4.  We're lenient now via `any_unsigned_first`,
   but if you have custom handlers in your code, use that helper
   rather than `as_u4_scalar` for identifier fields.
2. **PPBODY direction.**  Some MES send PPBODY as ASCII even when
   the spec says it can be Binary.  Use `as_text_or_binary` not
   `as_binary`.
3. **Trailing fields.**  Some MES add proprietary trailing fields
   to S2F41 / S16F11 / S3F17 bodies that aren't in the standard.
   Our parsers are tolerant of extras; check your handler's
   assumptions.

See MES_INTEROP.md §13 for the per-MES quirk register.

## What if the spec is ambiguous on some detail?

Cross-check against the secsgem-py and secs4j wire output:
[VERIFICATION.md](VERIFICATION.md).  If both peers agree on a
shape, that's the working interpretation regardless of how you read
the spec text.  If they disagree, the secsgem-py output usually
wins (it's the de-facto Python reference and most MES vendors test
against it), but file the question — we may need a new test.

## Can I run this without Docker?

In principle yes — you need g++-13 (or any C++20 compiler), CMake,
Ninja, libasio-dev, libyaml-cpp-dev, python3.  But every doc, every
CI lane, every test command in the repo assumes Docker.  Going
off-piste means re-deriving the build on your host.  We don't
support it; we don't actively break it.

## How does persistence survive a crash mid-write?

Every store uses a `.tmp + atomic rename` pattern: writes go to
`<file>.tmp`, then `rename(2)`s into place.  POSIX guarantees the
rename is atomic on the same filesystem.  A crash mid-write loses
the `.tmp` (corrupt-drop on next replay) but leaves the prior
record intact.

Every store's loader accepts versions in `[1, kVersion]` so future
schema bumps don't nuke old records — see README §Production
deployment "Schema migrations."

## What does the "spool" actually do?

When the host MES disconnects, the equipment can't deliver
unsolicited S5F1 alarms / S6F11 events.  Without spool they'd be
lost.

With spool enabled (`SpoolStore::set_spoolable_streams({5, 6})`),
those frames queue to in-memory FIFO (and persistent disk if
`enable_persistence` is set).  On the host's next SELECT, the
equipment emits `S6F25 SpoolDataReady(count)`; the host issues
`S6F23(Transmit)` to drain, or `S6F23(Purge)` to discard.

It's the GEM equivalent of an outbox.  See E30 §6.22 and our
SpoolStore source.

## How is "robustness fuzz" different from "libFuzzer"?

- **Robustness fuzz** (`tests/test_robustness_fuzz.cpp`) is a
  *model-level* property test.  It picks random tool operations
  (PJ create, alarm set, substrate move, …) respecting FSM
  legality, and checks invariants after each.
- **libFuzzer** (`apps/fuzz_*.cpp`) is a *byte-level* coverage-
  guided fuzzer.  It feeds arbitrary bytes to the codec and
  asserts no crash / UB.

They cover different concerns: robustness fuzz catches *semantic*
bugs (lost data, wrong state); libFuzzer catches *parser* bugs
(crashes, UB, buffer overruns).

## What's "conformance" vs "interop"?

- **Conformance** (`build/secs_conformance`) is *us* driving *us*
  through every claimed E30 capability and asserting the spec-
  mandated reply S/F.  Catches our regressions against our own
  understanding of the spec.
- **Interop** (`interop/*.py`, `interop/secs4j/*.java`,
  `interop/tshark_validate.sh`) is third-party tools agreeing on
  the wire bytes our equipment produces.  Catches "we got the
  spec wrong" — which conformance can't.

Both are necessary; neither replaces the other.  See VERIFICATION.md.

## How do I bring this to a customer site?

Run through the five external proofs in
[the eight commands in PROOFS.md](PROOFS.md) at
the customer's network.  Then walk MES_INTEROP.md against their
actual MES.  Then deploy per [SECURITY.md](SECURITY.md) for the
nftables / stunnel / signing setup.  Then page on the metrics from
INTEGRATION.md §6.4.

## What's not implemented?

Every E30 Fundamental + Additional capability and every GEM 300
standard in scope is shipped.  The two non-shipped pieces are:

1. **The asio `serial_port` adapter for SECS-I** — the FSM is
   implemented and tested end-to-end over TCP
   ([`secsi::TcpTransport`](include/secsgem/secsi/tcp_transport.hpp));
   the serial-port driver is a deferred follow-up (most modern GEM
   equipment runs HSMS).  Listed under "Deferred follow-ups" in
   [README.md](README.md).
2. **A GEM Reference Test System (RTS) run** — paid third-party
   certification gate, not a code feature.  See
   [COMPLIANCE.md](COMPLIANCE.md) §8 for what "100% GEM-compliant"
   honestly means about a codebase vs. a certified tool.

Note: Equipment Processing States are tool-defined per E30 §6.3 — the
engine ships, and vendors load their concrete states (IDLE / SETUP /
READY / EXECUTING / …) the same way `data/control_state.yaml` is
loaded.  That isn't a gap, it's how the spec is designed.