# Security operations guide

HSMS is plain TCP — no auth, no encryption.  That's what every fab
tool ships and what every MES expects.  Security comes from the
network layer around the HSMS socket; this doc has the concrete
configs.

For production deployments treat the sections below as mandatory
unless your fab security architect signs off on a deviation.  HSMS
on an exposed network with no controls is how MES impersonation
becomes a wafer-loss incident.

## 1. Network isolation

### 1.1 Subnet placement

HSMS must run on a **control LAN** — physically or VLAN-separated
from corporate / engineering networks.  The MES host's IP is the
only thing that should be able to reach the equipment's HSMS port.

### 1.2 Host firewall (nftables example)

Drop in `/etc/nftables.d/50-secsgem.nft`, then `systemctl reload
nftables`:

```nftables
table inet filter {
    set mes_hosts {
        type ipv4_addr
        flags interval
        elements = {
            10.40.1.10,           # camstar-primary.fab.example
            10.40.1.11,           # camstar-standby.fab.example
        }
    }

    chain input {
        type filter hook input priority filter; policy drop;

        # Allow established + loopback unconditionally.
        ct state established,related accept
        iifname "lo" accept

        # HSMS port: only from known MES hosts.
        tcp dport 5000 ip saddr @mes_hosts accept

        # Prometheus exporter on :9090: only from monitoring subnet.
        tcp dport 9090 ip saddr 10.40.99.0/24 accept

        # SSH for ops: only from the bastion.
        tcp dport 22 ip saddr 10.40.99.1 accept

        # Anything else is dropped (policy default).
    }
}
```

Test the ruleset against a known-bad source before reloading:

```sh
nft -c -f /etc/nftables.d/50-secsgem.nft   # syntax check
nft list set inet filter mes_hosts          # confirm the set is loaded
```

### 1.3 Pod-network policy (Kubernetes / K3s deployments)

For pod deployments, use a `NetworkPolicy`:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: secsgem-equipment-ingress
spec:
  podSelector:
    matchLabels:
      app: secsgem-equipment
  policyTypes: [Ingress]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          tier: mes
      podSelector:
        matchLabels:
          app: camstar-host
    ports:
    - protocol: TCP
      port: 5000
  - from:
    - namespaceSelector:
        matchLabels:
          tier: monitoring
    ports:
    - protocol: TCP
      port: 9090
```

Calico, Cilium, or whatever your CNI is all enforce the same.

## 2. TLS tunnel for cross-site HSMS

For most fabs the control LAN is good enough.  Cross-site HSMS (rare
but real for shared-MES architectures) needs encryption.  **Do not
modify the HSMS wire protocol** — wrap the TCP socket in stunnel
or a sidecar TLS proxy.

### 2.1 stunnel.conf — equipment side (terminator)

```ini
; /etc/stunnel/secsgem-equipment.conf
foreground = no
pid = /run/stunnel/secsgem-equipment.pid

setuid = stunnel
setgid = stunnel

debug = 5
syslog = yes

[secsgem-hsms]
accept = 0.0.0.0:5443           ; TLS port the MES connects to
connect = 127.0.0.1:5000        ; equipment HSMS listener (localhost)

cert = /etc/stunnel/certs/equipment.fab.example.crt
key = /etc/stunnel/certs/equipment.fab.example.key

CAfile = /etc/stunnel/certs/mes-ca-bundle.crt
verifyChain = yes
verifyPeer = yes
checkHost = camstar-primary.fab.example

sslVersionMin = TLSv1.3
ciphers = TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
```

Bind the C++ server to `127.0.0.1` only (so the cleartext socket isn't
reachable from the network):

```sh
secs_server --port 5000 --bind 127.0.0.1 \
    --config /etc/acme-secsgem/equipment.yaml ...
```

The demo binary binds INADDR_ANY; a `--bind` flag is a follow-up.

### 2.2 stunnel.conf — MES side (initiator)

```ini
; /etc/stunnel/secsgem-host.conf
[secsgem-hsms]
client = yes
accept = 127.0.0.1:5000         ; MES connects here as if it were the equipment
connect = equipment.fab.example:5443

CAfile = /etc/stunnel/certs/equipment-ca-bundle.crt
verifyChain = yes
verifyPeer = yes

; mTLS — present a client cert the equipment-side CA trusts.
cert = /etc/stunnel/certs/camstar-primary.fab.example.crt
key = /etc/stunnel/certs/camstar-primary.fab.example.key

sslVersionMin = TLSv1.3
```

### 2.3 Performance impact

TLS adds ~50 µs per round-trip on modern hardware (measured via
`secs_bench` with stunnel in the loop vs. direct connection).  At
sustained rates in the few-hundred-events/sec range, the overhead
is invisible against the fab-tool latency budget.

## 3. Authentication

HSMS itself has no peer auth — Select.req sends a session ID and
that's it.  Two production-grade defenses:

1. **mTLS via the sidecar above** — the MES has to present a client
   cert signed by your fab's CA.  Without it, the TLS handshake fails
   before HSMS is touched.

2. **Per-tool firewall ACLs** — even with mTLS, restrict the source
   IPs (§1.2 / §1.3).  Defense in depth.

Do not try to add auth at the HSMS layer.  No commercial MES would
accept the protocol change, and the wire spec is what makes the
codebase auditable.

## 4. YAML config integrity

`equipment.yaml`, `control_state.yaml`, the two job tables, and
`messages.yaml` together define the equipment's behaviour.  An
attacker who can rewrite any of them owns the SECS/GEM surface.

### 4.1 Signing with minisign

[`minisign`](https://jedisct1.github.io/minisign/) is the smallest
viable signing tool — single binary, single keypair file, Ed25519
under the hood, used by Wasmer / OpenBSD / others.  Two-line install:

```sh
apt-get install -y minisign   # Ubuntu 24.04
minisign -G -p /etc/acme-secsgem/keys/acme.pub \
            -s ~/.minisign/acme.sec
```

Sign every config bundle at deployment time:

```sh
cd /etc/acme-secsgem
minisign -S -s ~/.minisign/acme.sec equipment.yaml
minisign -S -s ~/.minisign/acme.sec control_state.yaml
minisign -S -s ~/.minisign/acme.sec process_job_state.yaml
minisign -S -s ~/.minisign/acme.sec control_job_state.yaml
# .minisig files appear next to each.
```

Verify on the tool before the server starts (systemd ExecStartPre):

```sh
#!/usr/bin/env bash
# /usr/local/libexec/secsgem-verify-configs.sh
set -euo pipefail
ETC=/etc/acme-secsgem
PUB=${ETC}/keys/acme.pub
for f in equipment.yaml control_state.yaml \
         process_job_state.yaml control_job_state.yaml; do
    minisign -V -p "$PUB" -m "${ETC}/$f"
done
```

Wire into systemd:

```ini
[Service]
ExecStartPre=/usr/local/libexec/secsgem-verify-configs.sh
ExecStart=/usr/local/bin/secs_server --config /etc/acme-secsgem/equipment.yaml ...
```

If any signature fails, the unit refuses to start.

### 4.2 Validate before signing

`secs_server --validate-config` must run clean against the YAML
before signing — signing a broken config only transmits the
breakage cryptographically:

```sh
secs_server --validate-config \
    --config equipment.yaml \
    --state-table control_state.yaml \
    --pj-state-table process_job_state.yaml \
    --cj-state-table control_job_state.yaml \
    || { echo "config invalid; refusing to sign"; exit 1; }
minisign -S -s ~/.minisign/acme.sec equipment.yaml
```

## 5. Audit logging for SIEM

Every wire frame should be retrievable for a configurable retention
window (90 days is the common ask).  The library exposes a log hook
on `hsms::Connection`; ship JSON-line records to your SIEM.

### 5.1 Recommended JSON schema

```json
{
  "@timestamp": "2026-06-09T14:23:55.412Z",
  "host": "tool-acme-pvd-3000-01",
  "session_id": 0,
  "direction": "rx",
  "stream": 2,
  "function": 41,
  "system_bytes": 1234567890,
  "reply_expected": true,
  "body_sml": "<L [2] <A 'START'> <L [0]>>",
  "body_bytes": 36,
  "elapsed_ms_since_select": 84210
}
```

One line per frame.  Stream → splunk-forwarder / vector.dev / fluent-bit
→ your fab's SIEM.

### 5.2 Wiring it up

```cpp
conn->set_log_handler([&](const std::string& msg) {
  // The connection's built-in log_handler gets a free-text line.
  // For structured logging, intercept at the message_handler level:
  // wrap router.dispatch and emit JSON for each frame in/out.
  syslog(LOG_LOCAL0 | LOG_INFO, "secsgem: %s", msg.c_str());
});

// Structured frame log via a wrapped dispatcher:
conn->set_message_handler([&](const secs2::Message& m) {
  emit_audit_json("rx", m);
  auto reply = router.dispatch(m);
  if (reply) emit_audit_json("tx", *reply);
  return reply;
});
```

Where `emit_audit_json` writes a single line in the schema above to
a file `vector.dev` is tailing, or to systemd-journal with `sd_journal_send`.

### 5.3 What to alert on

Threshold rules in the SIEM that should page on-call:

| Signal                                  | Threshold              | Why                              |
|-----------------------------------------|------------------------|----------------------------------|
| S9F* emission rate                      | > 1 / minute sustained | malformed peer or schema drift   |
| Distinct source IPs on HSMS port        | > expected MES count   | spoofed connection attempts      |
| TLS handshake failures (stunnel log)    | > 5 / minute           | bad client cert or rogue scanner |
| Failed signature verification (start)   | any                    | tampered YAML                    |
| HSMS connection-flap rate               | > 1 / minute           | MES instability or net event     |
| Spool depth                              | > 1000 sustained       | MES backpressure or outage       |
| T-timer expiry counter                  | rising                 | network-layer trouble            |

## 6. Secrets handling

### 6.1 Stunnel keys

- Store at `/etc/stunnel/certs/`, mode `0600`, owner `stunnel`.
- Rotate annually.  Ed25519 keys never expire cryptographically but
  fab policy usually mandates rotation regardless.
- Don't commit private keys to git.  Don't share them across tools.

### 6.2 Minisign signing key

- Live on a hardened build host, not on the tools themselves.
- The public key (`acme.pub`) is what ships to every tool.
- Sign in CI from a passphrase-protected key stored as a CI secret;
  never echo the passphrase, never log it.

## 7. Incident response

When something goes wrong:

1. **Capture the wire trace immediately** — `tcpdump -w` on the
   equipment's HSMS interface.  Retain for 24h minimum even if no
   incident is suspected.
2. **Don't restart the equipment** until the wire trace and the
   journal directory (`/var/lib/acme-secsgem/`) are snapshotted.
   Restarting wipes in-memory state the incident analysis may need.
3. **Pull recent audit logs from the SIEM** for the affected session
   ID and host.
4. **Cross-check against the runbook** in README §10 — common
   incidents have documented mitigation paths.

Filing an incident with us (`raphael@maenle.net`):
- Wire trace (pcap, scrubbed of any production-sensitive payloads)
- Equipment logs covering the incident window
- Journal directory `tar.gz`
- Equipment build SHA + YAML SHAs
- MES vendor + build
- What you tried that didn't work