From 2fce2fad0c8db9c859efb3863e08dc5ee8797cc0 Mon Sep 17 00:00:00 2001 From: Raphael Maenle Date: Tue, 9 Jun 2026 16:12:44 +0200 Subject: [PATCH] verify: secs4j cross-validation (independent Java implementation) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 20 cross-validation checks PASS against [secs4java8] (Apache 2.0, kenta-shimizu) — an independent SECS/HSMS implementation in Java by a different author from a different language ecosystem. Distinct implementer = independent spec interpretation. Two libraries agreeing on wire bytes is much stronger evidence of spec-correctness than either alone. Coverage targets the gap the secsgem-py interop deliberately skipped (secsgem-py's SFDL grammar couldn't easily express GEM 300 bodies with variable lists of named scalars): - S1F1/F13/F17/F19/F21/F23 — establish comms + namelists - S2F17 — clock - S2F23 — trace init (5-field body) - S2F49 — enhanced remote command (DATAID + OBJSPEC + RCMD + params) - S3F17/F19/F25/F27 — full E87 carrier surface (action, slot map verify, transfer with port pair, cancel) - S5F13/F17 — exception recovery (EXID + EXRECVRA) - S14F9/F11 — E94 CJ create with prjobids list, CJ delete - S16F5/F27 — E40 PJ command, E94 CJ command - S1F15 — offline cleanup 20/20 PASS against the demo equipment. Reply S/F matches the spec for every transaction; specific ACK values vary by equipment state (CarrierIDUnknown for an unknown carrier is just as valid as Accept for a known one) so we assert on the wire shape, not the result. Ship layout: interop/secs4j/Dockerfile — eclipse-temurin:21-jdk + clone + build of secs4java8 → Export.jar interop/secs4j/Secs4jHostHarness.java — 20 round_trip assertions; uses Secs2.list/uint4/ascii to build full GEM 300 bodies; comm.send() for arbitrary S/F pairs interop/secs4j_validate.sh — orchestrator: builds image, compiles harness, starts compose server, runs Java container on the secs network against it .gitea/workflows/ci.yml — secs4j-interop job in CI README.md — proof table grows to 7 commands .gitignore — *.class After this commit our proof chain has: - SEMI E5 KAT (standards body's own arithmetic) - tshark dissector (Wireshark's HSMS impl) - secsgem-py interop (Python reference impl) - **secs4j interop** (independent Java impl) + 426 unit tests, 47 conformance harness checks, 100k random ops, YAML validation Four independent external proofs, three of them on overlapping wire surface from independent angles. Plan: VERIFICATION.md §3. [secs4java8]: https://github.com/kenta-shimizu/secs4java8 Co-Authored-By: Claude Opus 4.7 --- .gitea/workflows/ci.yml | 21 +++ .gitignore | 1 + README.md | 1 + interop/secs4j/Dockerfile | 33 ++++ interop/secs4j/Secs4jHostHarness.java | 253 ++++++++++++++++++++++++++ interop/secs4j_validate.sh | 49 +++++ 6 files changed, 358 insertions(+) create mode 100644 interop/secs4j/Dockerfile create mode 100644 interop/secs4j/Secs4jHostHarness.java create mode 100755 interop/secs4j_validate.sh diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index 581f6d2..7c86e68 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -121,3 +121,24 @@ jobs: run: bash interop/tshark_validate.sh env: PORT: "5099" + + secs4j-interop: + runs-on: ubuntu-latest + # secs4j-interop builds its own docker image (with JDK) and starts + # secs_server via docker compose; needs the docker socket. + steps: + - name: Bootstrap (node + git) + run: | + export DEBIAN_FRONTEND=noninteractive + apt-get update + apt-get install -y --no-install-recommends \ + git ca-certificates nodejs docker.io docker-compose-plugin + + - uses: actions/checkout@v4 + + # Cross-validate against secs4java8 (Apache 2.0, kenta-shimizu) — + # an independent SECS implementation in Java. 20 checks + # covering S1/S2/S3/S5/S14/S16 with full-body GEM 300 shapes + # that secsgem-py couldn't easily drive. + - name: secs4j cross-validation + run: bash interop/secs4j_validate.sh diff --git a/.gitignore b/.gitignore index ac9297e..4191f35 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,7 @@ build/ *.obj *.a *.so +*.class .DS_Store compile_commands.json diff --git a/README.md b/README.md index 16015ca..7ec8e45 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ a fresh clone, the codebase implements what | 4 | `SECSGEM_ROBUSTNESS_SOAK=1 docker compose run --rm builder /app/build/secsgem_tests -tc='*soak*'` | **100 000 random tool operations** execute with all invariants and persistence round-trips holding | | 5 | `docker compose run --rm builder /app/build/secs_server --validate-config --config /app/data/equipment.yaml --state-table /app/data/control_state.yaml --pj-state-table /app/data/process_job_state.yaml --cj-state-table /app/data/control_job_state.yaml` | Every shipped YAML config passes structural + referential validation | | 6 | `docker compose run --rm builder bash /app/interop/tshark_validate.sh` | **69 HSMS frames** dissected by Wireshark's HSMS dissector (independent third codec) with no malformed packets | +| 7 | `bash interop/secs4j_validate.sh` | **20 cross-validation checks** PASS against [secs4java8](https://github.com/kenta-shimizu/secs4java8) (independent Java implementation), covering full GEM 300 body shapes (S3, S14, S16) plus S2F49 / S5F13 / S2F23 | Plus, on every push to `main`, [Gitea Actions](.gitea/workflows/ci.yml) runs both a **Release build + full test suite** and a separate diff --git a/interop/secs4j/Dockerfile b/interop/secs4j/Dockerfile new file mode 100644 index 0000000..0f13eea --- /dev/null +++ b/interop/secs4j/Dockerfile @@ -0,0 +1,33 @@ +FROM eclipse-temurin:21-jdk + +# secs4java8 (kenta-shimizu) is a second independent SECS/HSMS +# implementation in Java. We use it as a third-party peer to +# cross-validate our C++ server against — distinct author, distinct +# language ecosystem, distinct interpretation of the SEMI standards. +# If both libraries agree on wire bytes, that's much stronger evidence +# of spec-correctness than either alone. + +ENV DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && apt-get install -y --no-install-recommends \ + git \ + ca-certificates \ + bash \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /opt + +# Pin to a tagged release if/when one becomes available; until then, +# pin to a known-working commit via depth=1 + checkout. Replace with +# a SHA in CI for full reproducibility. +RUN git clone --depth 1 https://github.com/kenta-shimizu/secs4java8.git secs4java8 +WORKDIR /opt/secs4java8 + +# secs4java8 ships a compile.sh that builds Export.jar. Run it once +# at image build so the harness can just include the resulting jar. +RUN bash compile.sh + +# The harness source lives outside this image and is mounted at /work +# when the container runs (so editing the harness doesn't require a +# rebuild). +WORKDIR /work diff --git a/interop/secs4j/Secs4jHostHarness.java b/interop/secs4j/Secs4jHostHarness.java new file mode 100644 index 0000000..ef2cefd --- /dev/null +++ b/interop/secs4j/Secs4jHostHarness.java @@ -0,0 +1,253 @@ +// Cross-validate our C++ secs_server against secs4java8 (Apache 2.0, +// kenta-shimizu). Independent SECS/HSMS implementation by a different +// author + language ecosystem = independent spec interpretation. If +// secs4java8 and our C++ codec both parse each other's bytes cleanly, +// spec-correctness has corroboration that secsgem-py alone can't give. +// +// Targets the wire surface the secsgem-py interop deliberately skipped: +// GEM 300 (S3 E87, S14 E94, S16 E40) with full-body messages, S2F49 +// enhanced commands, S5F13 exception recovery, S2F23 trace init. +// +// Build (via Dockerfile in this directory): +// javac -cp /opt/secs4java8/Export.jar Secs4jHostHarness.java +// java -cp /opt/secs4java8/Export.jar:. Secs4jHostHarness + +import java.net.InetSocketAddress; +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import com.shimizukenta.secs.SecsMessage; +import com.shimizukenta.secs.hsms.HsmsConnectionMode; +import com.shimizukenta.secs.hsmsss.HsmsSsCommunicator; +import com.shimizukenta.secs.hsmsss.HsmsSsCommunicatorConfig; +import com.shimizukenta.secs.secs2.Secs2; + +public class Secs4jHostHarness { + + static final AtomicInteger pass = new AtomicInteger(); + static final AtomicInteger fail = new AtomicInteger(); + static final List failures = new ArrayList<>(); + + static void check(String label, boolean ok, String detail) { + String tag = ok ? "[OK ]" : "[FAIL]"; + System.out.println(tag + " " + label + (detail.isEmpty() ? "" : " — " + detail)); + if (ok) pass.incrementAndGet(); + else { fail.incrementAndGet(); failures.add(label + ": " + detail); } + } + + // Run a request and check the reply's S/F matches the expected pair. + // We deliberately don't assert specific ACK values — secs4java8 and + // our codec must agree on which S/F pair the equipment replies with, + // and that the body parses; the specific ACK ("Carrier ID Unknown" + // vs "Accept") depends on equipment state, which we don't pre-load. + static void roundtrip(HsmsSsCommunicator comm, String label, + int stream, int function, boolean w, Secs2 body, + int expectedReplyStream, int expectedReplyFunction) { + try { + Optional reply = comm.send(stream, function, w, body); + if (!reply.isPresent()) { + check(label, false, "no reply (W=" + w + ")"); + return; + } + int rs = reply.get().getStream(); + int rf = reply.get().getFunction(); + boolean ok = rs == expectedReplyStream && rf == expectedReplyFunction; + check(label, ok, + ok ? ("got S" + rs + "F" + rf) + : ("expected S" + expectedReplyStream + "F" + expectedReplyFunction + + " got S" + rs + "F" + rf)); + } catch (Exception e) { + check(label, false, "threw " + e.getClass().getSimpleName() + + ": " + e.getMessage()); + } + } + + public static void main(String[] args) throws Exception { + String host = args.length > 0 ? args[0] : "127.0.0.1"; + int port = args.length > 1 ? Integer.parseInt(args[1]) : 5099; + + HsmsSsCommunicatorConfig config = new HsmsSsCommunicatorConfig(); + config.socketAddress(new InetSocketAddress(host, port)); + config.connectionMode(HsmsConnectionMode.ACTIVE); + config.sessionId(0); + config.isEquip(false); + config.notLinktest(); + config.timeout().t3(5.0F); + config.timeout().t5(2.0F); + config.timeout().t6(5.0F); + config.timeout().t8(5.0F); + config.gem().mdln("SECS4J"); + config.gem().softrev("1.0"); + + System.out.println("connecting active to " + host + ":" + port); + + try (HsmsSsCommunicator comm = HsmsSsCommunicator.newInstance(config)) { + CountDownLatch selected = new CountDownLatch(1); + comm.addSecsCommunicatableStateChangeListener(s -> { + if (s) selected.countDown(); + }); + comm.open(); + + if (!selected.await(15, TimeUnit.SECONDS)) { + System.err.println("FAIL: never reached SELECTED"); + System.exit(2); + } + System.out.println("HSMS SELECTED."); + + // -------- Basic round-trips (overlap with secsgem-py) -------- + // S1F1 → S1F2 (Are You There) + roundtrip(comm, "S1F1 → S1F2 (Are You There)", + 1, 1, true, Secs2.empty(), 1, 2); + + // S1F13 → S1F14 (Establish Communications) + // body: > + roundtrip(comm, "S1F13 → S1F14 (Establish Comms)", + 1, 13, true, + Secs2.list(Secs2.ascii("SECS4J"), Secs2.ascii("1.0")), + 1, 14); + + // S1F17 → S1F18 (Request Online) + roundtrip(comm, "S1F17 → S1F18 (Online)", + 1, 17, true, Secs2.empty(), 1, 18); + + // S1F19 → S1F20 (GEM Compliance) — empty body, list reply + roundtrip(comm, "S1F19 → S1F20 (GEM Compliance)", + 1, 19, true, Secs2.empty(), 1, 20); + + // S1F21 → S1F22 (DVID namelist) — empty list = all DVIDs + roundtrip(comm, "S1F21 → S1F22 (DVID namelist)", + 1, 21, true, Secs2.list(), 1, 22); + + // S1F23 → S1F24 (CEID namelist) — empty list = all CEIDs + roundtrip(comm, "S1F23 → S1F24 (CEID namelist)", + 1, 23, true, Secs2.list(), 1, 24); + + // S2F17 → S2F18 (Clock) — empty body + roundtrip(comm, "S2F17 → S2F18 (Clock)", + 2, 17, true, Secs2.empty(), 2, 18); + + // -------- GEM 300 streams: the gap secsgem-py couldn't fill ---- + + // S3F17 → S3F18 (E87 carrier action): + // body: > + roundtrip(comm, "S3F17 → S3F18 (E87 carrier action, full body)", + 3, 17, true, + Secs2.list( + Secs2.uint4(1L), + Secs2.ascii("ProceedWithCarrier"), + Secs2.ascii("CAR-SECS4J-1"), + Secs2.list()), + 3, 18); + + // S3F19 → S3F20 (Slot map verify): + // body: >> + roundtrip(comm, "S3F19 → S3F20 (slot map verify)", + 3, 19, true, + Secs2.list( + Secs2.ascii("CAR-SECS4J-1"), + Secs2.list()), + 3, 20); + + // S3F25 → S3F26 (Carrier transfer): + // body: > + roundtrip(comm, "S3F25 → S3F26 (carrier transfer)", + 3, 25, true, + Secs2.list( + Secs2.ascii("CAR-SECS4J-1"), + Secs2.uint1(1), + Secs2.uint1(2)), + 3, 26); + + // S3F27 → S3F28 (Cancel carrier): body: + roundtrip(comm, "S3F27 → S3F28 (cancel carrier)", + 3, 27, true, Secs2.ascii("CAR-SECS4J-1"), 3, 28); + + // S14F9 → S14F10 (E94 ControlJob create): + // body: >> + roundtrip(comm, "S14F9 → S14F10 (E94 CJ create, full body)", + 14, 9, true, + Secs2.list( + Secs2.ascii("CJ-SECS4J-1"), + Secs2.list(Secs2.ascii("PJ-SECS4J-1"))), + 14, 10); + + // S16F5 → S16F6 (E40 PRJobCommand): + // body: > + roundtrip(comm, "S16F5 → S16F6 (E40 PJ command)", + 16, 5, true, + Secs2.list( + Secs2.ascii("PJ-NONEXISTENT"), + Secs2.ascii("PJABORT")), + 16, 6); + + // S16F27 → S16F28 (E94 CJobCommand): + // body: > + roundtrip(comm, "S16F27 → S16F28 (E94 CJ command)", + 16, 27, true, + Secs2.list( + Secs2.ascii("CJ-SECS4J-1"), + Secs2.ascii("CJSTOP")), + 16, 28); + + // S14F11 → S14F12 (E94 DeleteControlJob): body: + roundtrip(comm, "S14F11 → S14F12 (E94 CJ delete)", + 14, 11, true, Secs2.ascii("CJ-SECS4J-1"), 14, 12); + + // -------- Other streams secsgem-py interop skipped -------- + + // S2F49 → S2F50 (Enhanced remote command): + // body: > + roundtrip(comm, "S2F49 → S2F50 (enhanced remote command)", + 2, 49, true, + Secs2.list( + Secs2.uint4(1L), + Secs2.ascii("EQUIPMENT"), + Secs2.ascii("STOP"), + Secs2.list()), + 2, 50); + + // S2F23 → S2F24 (Trace init): + // body: > + roundtrip(comm, "S2F23 → S2F24 (trace init)", + 2, 23, true, + Secs2.list( + Secs2.uint4(8001L), + Secs2.ascii("000001"), + Secs2.uint4(0L), + Secs2.uint4(1L), + Secs2.list(Secs2.uint4(1L))), + 2, 24); + + // S5F13 → S5F14 (Exception recover): + // body: > + roundtrip(comm, "S5F13 → S5F14 (exception recover)", + 5, 13, true, + Secs2.list(Secs2.uint4(999L), Secs2.ascii("NOP")), + 5, 14); + + // S5F17 → S5F18 (Exception recover abort): body: + roundtrip(comm, "S5F17 → S5F18 (exception recover abort)", + 5, 17, true, Secs2.uint4(999L), 5, 18); + + // S1F15 → S1F16 (Request Offline) — cleanup + roundtrip(comm, "S1F15 → S1F16 (offline)", + 1, 15, true, Secs2.empty(), 1, 16); + + System.out.println(); + System.out.println("=== secs4j interop summary ==="); + System.out.println(" passed: " + pass.get()); + System.out.println(" failed: " + fail.get()); + if (!failures.isEmpty()) { + System.out.println(); + System.out.println("FAILURES:"); + for (String s : failures) System.out.println(" - " + s); + } + } + + System.exit(fail.get() == 0 ? 0 : 1); + } +} diff --git a/interop/secs4j_validate.sh b/interop/secs4j_validate.sh new file mode 100755 index 0000000..28baa2e --- /dev/null +++ b/interop/secs4j_validate.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# Orchestrate the secs4java8 interop harness against our C++ server. +# +# Boots secs_server on the docker bridge, then runs the Java harness +# in its own container against it. Wired into CI via a separate +# job (see .gitea/workflows/ci.yml). +# +# Usage: bash interop/secs4j_validate.sh +# Exit codes: +# 0 — every check the harness defines passed +# 1 — one or more checks failed +# 2 — script / orchestration error +set -euo pipefail + +PORT=${PORT:-5099} + +cd "$(dirname "$0")/.." + +# Ensure the secs4j-interop image is built (idempotent). +echo "ensuring secs4j-interop image is built..." +docker build -t secsgem-secs4j-interop -f interop/secs4j/Dockerfile interop/secs4j \ + >/dev/null 2>&1 + +# Ensure the harness is compiled. +echo "compiling Secs4jHostHarness.java..." +docker run --rm -v "$PWD/interop/secs4j:/work" secsgem-secs4j-interop \ + bash -c "cd /work && javac -cp /opt/secs4java8/Export.jar Secs4jHostHarness.java" \ + || { echo "FAIL: javac"; exit 2; } + +# Start the C++ server in background on the compose bridge network so +# the secs4j container can DNS-resolve it. +echo "starting secs_server (compose)..." +docker compose up -d server >/dev/null 2>&1 +trap 'docker compose stop server >/dev/null 2>&1 || true' EXIT + +# Give the server a moment to bind. +sleep 1 + +# Run the harness on the same compose network as the server, against +# the compose-defined hostname "server" on port 5000 (the demo +# server's default). +echo "running Secs4jHostHarness..." +docker run --rm \ + --network secs-gem_secs \ + -v "$PWD/interop/secs4j:/work" \ + secsgem-secs4j-interop \ + java -cp /opt/secs4java8/Export.jar:/work Secs4jHostHarness server 5000 + +# trap handles teardown.