secs-gem/README.md

secs-gem

A C++20 SECS-II / HSMS / SECS-I / GEM / GEM 300 runtime, fully containerized, with every behavioural rule encoded as YAML data (control state, equipment data dictionary, E40 process-job state machine, E94 control-job state machine, SECS-II message shapes).

Implements all of E5, E30, E37 (SS + GS), E4 SECS-I, E40, E42, E84, E87, E90, E94, E116, E120, E148, E157, E39. Per-store persistence on every mutable in-memory entity (spool, carriers, load-ports, substrates, process-jobs, control-jobs, exceptions). See COMPLIANCE.md for the per-capability audit and INTEGRATION.md for the vendor-side tutorial.

Quick start

Everything runs in Docker — no compiler or build tools on the host.

docker compose run --rm builder         # configure + compile
docker compose run --rm tests           # 384 cases / 2390 assertions
docker compose up --no-deps server client   # live two-container demo

Architecture

The project is "spec-as-data": the SEMI behavioural rules live in YAML; the C++ is the engine that reads them.

   ┌──────────────────────────────────────────────────────────────┐
   │ data/                                                        │
   │   messages.yaml          SECS-II message catalog             │
   │   control_state.yaml     E30 §6.2 control transition table   │
   │   process_job_state.yaml E40 §6 PJ transition table          │
   │   control_job_state.yaml E94 §6 CJ transition table          │
   │   equipment.yaml         SVIDs / DVIDs / ECIDs / CEIDs /     │
   │                          alarms / recipes / commands         │
   └──────────────────────┬───────────────────────────────────────┘
                          │  (loaded at startup, codegen at build)
                          ▼
   ┌──────────────────────────────────────────────────────────────┐
   │ tools/gen_messages.py                                        │
   │   reads messages.yaml -> emits generated/secsgem/gem/messages.hpp
   └──────────────────────┬───────────────────────────────────────┘
                          │
                          ▼
   ┌──────────────────────────────────────────────────────────────┐
   │ apps/                                                        │
   │   secs_server.cpp     passive equipment                      │
   │   secs_client.cpp     active host                            │
   │                       (both use gem::Router for dispatch)    │
   └──────────────────────────────────────────────────────────────┘

   secsgem::config   loader.hpp: YAML -> tables + data model
   secsgem::gem      every per-standard FSM (E30, E40, E84, E87,
                     E90, E94, E116, E120, E148, E157, E39, E5
                     exceptions), each per-store-persistable.
                     EquipmentDataModel composes all stores.
                     Router (stream, function) -> handler.
                     Generated messages.hpp covers 164 SxFy.
   secsgem::hsms     Connection (Asio): HSMS-SS + HSMS-GS, all
                     T-timers enforced, auto S9F3/F5/F7/F9/F11.
   secsgem::secsi    SECS-I Protocol FSM (E4): T1/T2/T3/T4 enforced
                     in-FSM, TCP transport for tunnel testing.
   secsgem::secs2    Item (variant), encode/decode, Message,
                     SML parser/printer.

Tree

secs-gem/
├── Dockerfile, docker-compose.yml      # toolchain + demo
├── CMakeLists.txt
├── README.md
├── COMPLIANCE.md                       # per-capability audit
├── INTEGRATION.md                      # vendor integration tutorial
├── data/
│   ├── messages.yaml                   # SECS-II message catalog (164 msgs)
│   ├── control_state.yaml              # E30 control state transitions
│   ├── process_job_state.yaml          # E40 PJ transitions
│   ├── control_job_state.yaml          # E94 CJ transitions
│   └── equipment.yaml                  # equipment data dictionary
├── tools/
│   └── gen_messages.py                 # codegen (messages.yaml -> .hpp)
├── include/secsgem/
│   ├── secs2/{item,codec,sml,message}.hpp
│   ├── hsms/{header,connection}.hpp
│   ├── secsi/{header,block,protocol,tcp_transport}.hpp
│   ├── gem/                            # FSMs per SEMI standard
│   ├── gem/store/                      # one file per focused store
│   ├── config/loader.hpp
│   └── endpoint.hpp
├── src/{secs2,hsms,secsi,gem,config}/*.cpp
├── apps/
│   ├── secs_server.cpp                 # passive equipment demo
│   ├── secs_client.cpp                 # active host demo
│   └── secs_interop_probe.cpp          # cross-test against secsgem-py
├── interop/                            # secsgem-py 0.3.0 cross-validation
└── tests/test_*.cpp                    # 384 cases / 2390 assertions

Adding a capability

The point of "spec-as-data" is that adding behaviour almost never requires a C++ change.

New SVID

# data/equipment.yaml
svids:
  - {id: 4, name: ChamberTemp, units: "C", type: U4, value: 25}

New host command with side effects

host_commands:
  - {name: VENT, ack: Accept, emit_ceid: 400, set_alarm: 2}

New state transition

# data/control_state.yaml
transitions:
  - {from: OnlineRemote, on: host_request_offline, to: EquipmentOffline, ack: Accept}

New SECS-II message

# data/messages.yaml
- id: S6F30
  stream: 6
  function: 30
  w: true
  builder: s6f30_something
  parser: parse_s6f30
  body:
    kind: list
    struct_name: Something
    fields:
      - {name: field_a, shape: {kind: scalar, item_type: U4}}
      - {name: field_b, shape: {kind: scalar, item_type: ASCII}}

docker compose run --rm builder regenerates messages.hpp. The typed builder, parser, and struct definition appear automatically.

---

Production / fab deployment

The library is a runtime stack. Shipping it on a real tool involves more than building the binary. This section enumerates the work that sits between "tests pass" and "this is running on the fab floor."

1. Persistence directory layout

Enable persistence per store at startup, before the connection comes up. Pattern (the call sites are equivalent on every store):

auto base = std::filesystem::path("/var/lib/acme-secsgem");
model->spool.enable_persistence(base / "spool");
model->carriers.enable_persistence(base / "carriers");
model->load_ports.enable_persistence(base / "loadports");
model->substrates.enable_persistence(base / "substrates");
model->process_jobs.enable_persistence(base / "pjobs");
model->control_jobs.enable_persistence(base / "cjobs");
model->exceptions.enable_persistence(base / "exceptions");

Storage rules:

• Mount this volume on the same physical disk as the binary — network filesystems (NFS) can introduce latency that interferes with the rename-based atomic write pattern.
• Back this volume up daily. Journal files are small (a few hundred bytes each) and rsync-friendly.
• Set sane retention. Cleared exceptions and dequeued PJs are removed automatically; complete carriers / substrates / CJs are the application's responsibility to sweep. Cap by file count (a million files in one directory is fine on ext4 / xfs; less on others).
• Disk space: budget 100 MB for a busy fab tool over a year (~500 K transitions, ~200 bytes each). In practice it's far less because terminal-state records are removed.

After a crash, the next process start replays every store and is back to the prior in-memory state before the HSMS port opens.

2. Security

HSMS over plain TCP is the spec's wire protocol. The library ships unencrypted by design — that's what equipment manufacturers expect. In a real fab:

• Network isolation: HSMS must run on a control LAN, never exposed to engineering / corporate networks. Default the --port to 5000 / 5005 on a dedicated VLAN behind firewall ACLs that whitelist your MES host's IP.
• TLS tunnel: for cross-site HSMS (rare but real for multi-fab shared hosts), tunnel the TCP through stunnel or a sidecar proxy. Don't modify the HSMS wire protocol; wrap the socket.
• Authentication: HSMS doesn't include peer auth. Rely on network-layer mTLS (sidecar proxy) and per-tool firewall rules.
• Audit logging: enable Connection::set_log_handler and ship to a SIEM. Every SECS-II message in/out should be retrievable for a configurable retention window — many fabs require 90 days.
• YAML config integrity: sign your config bundles (equipment.yaml, control_state.yaml, etc.) and verify the signature on load. Misconfiguration is one of the top root-causes of GEM-related fab incidents.

3. Monitoring and observability

The library exposes hooks at every layer. Wire them to whatever your fab already runs.

Signal	Hook	Why it matters
HSMS connection lifecycle	Connection::set_log_handler, set_selected_handler, set_closed_handler	reconnect storms, unexpected separates
T3 / T6 / T7 / T8 timer fires	set_closed_handler reason starts with "T*"	host MES unreachable, fab network event
Auto S9F* emission	set_log_handler line containing "-> S9F"	malformed peer traffic, schema drift
Spool depth	model->spool.size()	host MES backpressure / outage
FSM transitions (every store)	set_*_change_handler	tool state, throughput, anomaly detection
Persistence directory size	du -s var/lib/acme-secsgem	journal growth, untracked terminal-state records

Recommended metrics export pattern: aggregate into Prometheus via a sidecar that polls the data model. Per-CEID emission rates, alarm set/clear rates, T-timer expiry counts, and spool depth form a reasonable starter dashboard.

4. High availability

The library is single-threaded per HSMS connection — that's how HSMS works. For HA:

• Run two equipment processes in active/standby on the same tool, sharing the persistence volume. Only the active accepts the HSMS port; the standby tails the journal. Failover is filesystem-locked.
• Reconnect on the host side: an MES-side disconnect should trigger T5-based reconnect. Configure Timers::t5 to your MES's policy (default 10s).
• Graceful shutdown: SIGTERM should flush the write queue, call conn->separate(), and exit cleanly so the journal is point-consistent. The provided apps/secs_server.cpp shows the pattern.

5. Deployment patterns

Three common shapes:

Docker / podman on a tool PC

FROM ubuntu:24.04
COPY build/secs_server /usr/local/bin/
COPY etc/ /etc/acme-secsgem/
VOLUME /var/lib/acme-secsgem
EXPOSE 5000
ENTRYPOINT ["/usr/local/bin/secs_server", \
            "--port", "5000", \
            "--config", "/etc/acme-secsgem/equipment.yaml", \
            "--state-table", "/etc/acme-secsgem/control_state.yaml", \
            "--spool-dir", "/var/lib/acme-secsgem/spool"]

systemd unit

[Unit]
Description=ACME SECS/GEM equipment
After=network.target

[Service]
Type=simple
User=secsgem
Group=secsgem
ExecStart=/usr/local/bin/secs_server --port 5000 \
          --config /etc/acme-secsgem/equipment.yaml \
          --state-table /etc/acme-secsgem/control_state.yaml \
          --spool-dir /var/lib/acme-secsgem/spool
Restart=always
RestartSec=5
LimitNOFILE=8192

[Install]
WantedBy=multi-user.target

Kubernetes (multi-tool cell controller)

Run one Pod per tool with the persistence volume mounted from local-storage (not NFS). The Service exposes the HSMS port on the control LAN. Use a PodDisruptionBudget to ensure the standby is available during rolling updates.

6. Upgrade path

YAML edits don't require a rebuild — restart the process and the new dictionary loads. Code changes do require rebuild + restart.

• Zero-downtime for YAML: if you're using the active/standby HA pattern, edit YAML on the standby, restart the standby, promote.
• Code upgrades: deploy to a canary tool first; bake-test for at least a full wafer batch before fleet-wide rollout.
• Schema migrations: persistence records are versioned (v1, v2) and forward-compatible. Older versions still load; newer versions ignore unknown trailers. Always test the upgrade with a real on-disk journal before fleet rollout.

7. Integration with the fab stack

Other system	How this library talks to it
MES (Camstar, Mozaic, Camstar)	HSMS-SS over TCP (secs_server listens on a port the MES is configured to connect to)
Multi-MES (HSMS-GS)	Connection::add_session(device_id) registers extra sessions on one TCP socket
AMHS / OHT	E84 per-port FSMs (E84PortStore::on_signal_change(port, signal, value)); wire to your I/O bridge
Recipe engine	RecipeStore.add (opaque) + RecipeStore.add_formatted (E42 structured)
Alarm sources	AlarmRegistry::set(alid, active) from your sensor poll
Carrier scanner	CarrierStore::create / fire_id_event / set_slot_state
Wafer tracker	SubstrateStore::create / fire_*_event
EPT / shift report	EptStateMachine::accumulated(state) reads the time-bucket counters

8. Compliance and certification

• Fork COMPLIANCE.md and prune it to your tool's claimed coverage; ship that copy with the tool.
• Run an independent validator (GEM RTS or equivalent) against your specific tool — a passing library is necessary but not sufficient.
• Capture wire traces from the validator run; archive for audit.

9. Testing in production

• Canary: deploy to one or two tools per fab before fleet rollout.
• Synthetic transactions: a heartbeat that issues S1F1 every 60s and alerts on T3 timeout. Catches MES-side outages before a real recipe does.
• Shadow traffic: for upgrades, run the new version listening on a second port; have MES dual-connect; diff the responses.

10. Operational runbook (starting point)

Common production incidents and the right response:

Incident	First check	Mitigation
HSMS connection flapping	T7 / T6 timer fires in logs	check MES reachability, network MTU
Spool depth growing	host MES connectivity / ACK rate	force-drain via S6F23, escalate to MES
State machine "stuck"	last state-change handler log line	host-issued offline + re-establish
Alarm storm	AlarmRegistry all() snapshot	check upstream sensor; quench via S5F3
Persistence dir growing unbounded	du -s + file count	sweep terminal-state records
Cross-tool inconsistency	secsgem_tests on canary tool	compare wire trace vs validator

---

Demo

The two-container demo walks ~24 SECS transactions end-to-end through the data model. Run docker compose up --no-deps server client and watch the logs interleave.

Build details

The toolchain image (Dockerfile) is Ubuntu 24.04 with g++-13, CMake, Ninja, libasio-dev, libyaml-cpp-dev, and Python 3 for the codegen. doctest is fetched via CMake FetchContent. Build artifacts live in a named Docker volume so the host filesystem stays clean.

Standalone Asio is used in header-only mode (ASIO_STANDALONE). No Boost dependency.

Interop

interop/ contains the secsgem-py 0.3.0 cross-validation harness — secsgem-py active host driving our C++ passive server, our C++ active host probing secsgem-py's passive equipment, and a raw GEM-300 harness that round-trips S3 (E87), S14 (E94), S16 (E40), S12 (wafer maps) through hand-crafted SecsStreamFunction subclasses. See interop/README.md.