secs-gem/docs/SECURITY.md

Security operations guide

The daemon's gRPC API

secs_gemd's tool API is unauthenticated by design — authentication belongs to the transport. The defaults are safe (127.0.0.1), and the production recommendation is a Unix domain socket (--grpc unix:///run/secs_gemd/api.sock, file-permission protected, zero network surface). Never bind it to an interface reachable from the equipment LAN; if cross-host access is unavoidable, front it with the same stunnel pattern described below for HSMS.

HSMS is plain TCP — no auth, no encryption. That's what every fab tool ships and what every MES expects. Security comes from the network layer around the HSMS socket; this doc has the concrete configs.

For production deployments treat the sections below as mandatory unless your fab security architect signs off on a deviation. HSMS on an exposed network with no controls is how MES impersonation becomes a wafer-loss incident.

1. Network isolation

1.1 Subnet placement

HSMS must run on a control LAN — physically or VLAN-separated from corporate / engineering networks. The MES host's IP is the only thing that should be able to reach the equipment's HSMS port.

1.2 Host firewall (nftables example)

Drop in /etc/nftables.d/50-secsgem.nft, then systemctl reload nftables:

table inet filter {
    set mes_hosts {
        type ipv4_addr
        flags interval
        elements = {
            10.40.1.10,           # camstar-primary.fab.example
            10.40.1.11,           # camstar-standby.fab.example
        }
    }

    chain input {
        type filter hook input priority filter; policy drop;

        # Allow established + loopback unconditionally.
        ct state established,related accept
        iifname "lo" accept

        # HSMS port: only from known MES hosts.
        tcp dport 5000 ip saddr @mes_hosts accept

        # Prometheus exporter on :9090: only from monitoring subnet.
        tcp dport 9090 ip saddr 10.40.99.0/24 accept

        # SSH for ops: only from the bastion.
        tcp dport 22 ip saddr 10.40.99.1 accept

        # Anything else is dropped (policy default).
    }
}

Test the ruleset against a known-bad source before reloading:

nft -c -f /etc/nftables.d/50-secsgem.nft   # syntax check
nft list set inet filter mes_hosts          # confirm the set is loaded

1.3 Pod-network policy (Kubernetes / K3s deployments)

For pod deployments, use a NetworkPolicy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: secsgem-equipment-ingress
spec:
  podSelector:
    matchLabels:
      app: secsgem-equipment
  policyTypes: [Ingress]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          tier: mes
      podSelector:
        matchLabels:
          app: camstar-host
    ports:
    - protocol: TCP
      port: 5000
  - from:
    - namespaceSelector:
        matchLabels:
          tier: monitoring
    ports:
    - protocol: TCP
      port: 9090

Calico, Cilium, or whatever your CNI is all enforce the same.

2. TLS tunnel for cross-site HSMS

For most fabs the control LAN is good enough. Cross-site HSMS (rare but real for shared-MES architectures) needs encryption. Do not modify the HSMS wire protocol — wrap the TCP socket in stunnel or a sidecar TLS proxy.

2.1 stunnel.conf — equipment side (terminator)

; /etc/stunnel/secsgem-equipment.conf
foreground = no
pid = /run/stunnel/secsgem-equipment.pid

setuid = stunnel
setgid = stunnel

debug = 5
syslog = yes

[secsgem-hsms]
accept = 0.0.0.0:5443           ; TLS port the MES connects to
connect = 127.0.0.1:5000        ; equipment HSMS listener (localhost)

cert = /etc/stunnel/certs/equipment.fab.example.crt
key = /etc/stunnel/certs/equipment.fab.example.key

CAfile = /etc/stunnel/certs/mes-ca-bundle.crt
verifyChain = yes
verifyPeer = yes
checkHost = camstar-primary.fab.example

sslVersionMin = TLSv1.3
ciphers = TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

Bind the C++ server to 127.0.0.1 only (so the cleartext socket isn't reachable from the network):

secs_server --port 5000 --bind 127.0.0.1 \
    --config /etc/acme-secsgem/equipment.yaml ...

The demo binary binds INADDR_ANY; a --bind flag is a follow-up.

2.2 stunnel.conf — MES side (initiator)

; /etc/stunnel/secsgem-host.conf
[secsgem-hsms]
client = yes
accept = 127.0.0.1:5000         ; MES connects here as if it were the equipment
connect = equipment.fab.example:5443

CAfile = /etc/stunnel/certs/equipment-ca-bundle.crt
verifyChain = yes
verifyPeer = yes

; mTLS — present a client cert the equipment-side CA trusts.
cert = /etc/stunnel/certs/camstar-primary.fab.example.crt
key = /etc/stunnel/certs/camstar-primary.fab.example.key

sslVersionMin = TLSv1.3

2.3 Performance impact

TLS adds ~50 µs per round-trip on modern hardware (measured via secs_bench with stunnel in the loop vs. direct connection). At sustained rates in the few-hundred-events/sec range, the overhead is invisible against the fab-tool latency budget.

3. Authentication

HSMS itself has no peer auth — Select.req sends a session ID and that's it. Two production-grade defenses:

1. mTLS via the sidecar above — the MES has to present a client cert signed by your fab's CA. Without it, the TLS handshake fails before HSMS is touched.

2. Per-tool firewall ACLs — even with mTLS, restrict the source IPs (§1.2 / §1.3). Defense in depth.

Do not try to add auth at the HSMS layer. No commercial MES would accept the protocol change, and the wire spec is what makes the codebase auditable.

4. YAML config integrity

equipment.yaml, control_state.yaml, the two job tables, and messages.yaml together define the equipment's behaviour. An attacker who can rewrite any of them owns the SECS/GEM surface.

4.1 Signing with minisign

minisign is the smallest viable signing tool — single binary, single keypair file, Ed25519 under the hood, used by Wasmer / OpenBSD / others. Two-line install:

apt-get install -y minisign   # Ubuntu 24.04
minisign -G -p /etc/acme-secsgem/keys/acme.pub \
            -s ~/.minisign/acme.sec

Sign every config bundle at deployment time:

cd /etc/acme-secsgem
minisign -S -s ~/.minisign/acme.sec equipment.yaml
minisign -S -s ~/.minisign/acme.sec control_state.yaml
minisign -S -s ~/.minisign/acme.sec process_job_state.yaml
minisign -S -s ~/.minisign/acme.sec control_job_state.yaml
# .minisig files appear next to each.

Verify on the tool before the server starts (systemd ExecStartPre):

#!/usr/bin/env bash
# /usr/local/libexec/secsgem-verify-configs.sh
set -euo pipefail
ETC=/etc/acme-secsgem
PUB=${ETC}/keys/acme.pub
for f in equipment.yaml control_state.yaml \
         process_job_state.yaml control_job_state.yaml; do
    minisign -V -p "$PUB" -m "${ETC}/$f"
done

Wire into systemd:

[Service]
ExecStartPre=/usr/local/libexec/secsgem-verify-configs.sh
ExecStart=/usr/local/bin/secs_server --config /etc/acme-secsgem/equipment.yaml ...

If any signature fails, the unit refuses to start.

4.2 Validate before signing

secs_server --validate-config must run clean against the YAML before signing — signing a broken config only transmits the breakage cryptographically:

secs_server --validate-config \
    --config equipment.yaml \
    --state-table control_state.yaml \
    --pj-state-table process_job_state.yaml \
    --cj-state-table control_job_state.yaml \
    || { echo "config invalid; refusing to sign"; exit 1; }
minisign -S -s ~/.minisign/acme.sec equipment.yaml

5. Audit logging for SIEM

Every wire frame should be retrievable for a configurable retention window (90 days is the common ask). The library exposes a log hook on hsms::Connection; ship JSON-line records to your SIEM.

5.1 Recommended JSON schema

{
  "@timestamp": "2026-06-09T14:23:55.412Z",
  "host": "tool-acme-pvd-3000-01",
  "session_id": 0,
  "direction": "rx",
  "stream": 2,
  "function": 41,
  "system_bytes": 1234567890,
  "reply_expected": true,
  "body_sml": "<L [2] <A 'START'> <L [0]>>",
  "body_bytes": 36,
  "elapsed_ms_since_select": 84210
}

One line per frame. Stream → splunk-forwarder / vector.dev / fluent-bit → your fab's SIEM.

5.2 Wiring it up

conn->set_log_handler([&](const std::string& msg) {
  // The connection's built-in log_handler gets a free-text line.
  // For structured logging, intercept at the message_handler level:
  // wrap router.dispatch and emit JSON for each frame in/out.
  syslog(LOG_LOCAL0 | LOG_INFO, "secsgem: %s", msg.c_str());
});

// Structured frame log via a wrapped dispatcher:
conn->set_message_handler([&](const secs2::Message& m) {
  emit_audit_json("rx", m);
  auto reply = router.dispatch(m);
  if (reply) emit_audit_json("tx", *reply);
  return reply;
});

Where emit_audit_json writes a single line in the schema above to a file vector.dev is tailing, or to systemd-journal with sd_journal_send.

5.3 What to alert on

Threshold rules in the SIEM that should page on-call:

Signal	Threshold	Why
S9F* emission rate	> 1 / minute sustained	malformed peer or schema drift
Distinct source IPs on HSMS port	> expected MES count	spoofed connection attempts
TLS handshake failures (stunnel log)	> 5 / minute	bad client cert or rogue scanner
Failed signature verification (start)	any	tampered YAML
HSMS connection-flap rate	> 1 / minute	MES instability or net event
Spool depth	> 1000 sustained	MES backpressure or outage
T-timer expiry counter	rising	network-layer trouble

6. Secrets handling

6.1 Stunnel keys

• Store at /etc/stunnel/certs/, mode 0600, owner stunnel.
• Rotate annually. Ed25519 keys never expire cryptographically but fab policy usually mandates rotation regardless.
• Don't commit private keys to git. Don't share them across tools.

6.2 Minisign signing key

• Live on a hardened build host, not on the tools themselves.
• The public key (acme.pub) is what ships to every tool.
• Sign in CI from a passphrase-protected key stored as a CI secret; never echo the passphrase, never log it.

7. Incident response

When something goes wrong:

1. Capture the wire trace immediately — tcpdump -w on the equipment's HSMS interface. Retain for 24h minimum even if no incident is suspected.
2. Don't restart the equipment until the wire trace and the journal directory (/var/lib/acme-secsgem/) are snapshotted. Restarting wipes in-memory state the incident analysis may need.
3. Pull recent audit logs from the SIEM for the affected session ID and host.
4. Cross-check against the runbook in README §10 — common incidents have documented mitigation paths.

Filing an incident with us (raphael@maenle.net):

• Wire trace (pcap, scrubbed of any production-sensitive payloads)
• Equipment logs covering the incident window
• Journal directory tar.gz
• Equipment build SHA + YAML SHAs
• MES vendor + build
• What you tried that didn't work