Adds a -DSECSGEM_TSAN=ON CMake option that builds every target with
-fsanitize=thread + debug symbols + -O1 + frame pointers. Wires a
dedicated thread-sanitizer job into .gitea/workflows/ci.yml that
builds and runs the full test suite under TSan with
TSAN_OPTIONS=halt_on_error=1 (any flagged race fails the job, not
just warns).
Result against the full 426-case / 2557-assertion suite: 0 warnings,
all green. That converts the existing test_thread_safety.cpp (which
exercised the asio::post-onto-strand pattern) and test_concurrency
(in-flight transaction interleaving) and test_robustness_fuzz (28
random action types × thousands of ticks) from "pattern smoke-tests"
into actual race detection.
The first TSan run caught a real bug in test_robustness_fuzz's
act_exception_complete: it held a pointer to an ExceptionStore
entry across fire_internal(RecoveryComplete), which deletes the
entry. The subsequent state() read was a use-after-free. TSan
flagged it 8 times (4 reads × 2 stack-frame variants). Fix is
scoped lookup + re-check via has() after the mutation; matches the
contract any reasonable caller would follow.
The asio std_fenced_block atomic_thread_fence path generates TSan
"not supported" warnings during compile — those are asio's, not
ours, and don't affect runtime detection.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Property-based robustness test that drives long sequences of random
tool operations against EquipmentDataModel and verifies invariants +
persistence round-trip after every action. Replaces hand-written
state-pinning tests with a generative approach that explores
combinations no human author would think to write.
Action menu (28 weighted actions covering the full standard surface):
- PJ create / event / dequeue (E40)
- CJ create / event / delete (E94)
- Carrier create / id / slot (E87)
- Substrate create / location / proc (E90)
- Alarm set / clear / enable toggle (E5 §13)
- SVID updates (E30 §6.13)
- Define-report / link-event / enable (E30 §6.6)
- Exception post / recover / complete (E5 §9, S5F9-F18)
- Module event (E157)
- EPT event (E116)
- Spool enqueue / drain / force-toggle (E30 §6.22)
Every action is "adjusted": it picks a verb at random, then checks
state-machine legality before applying. A Pause is only fired on a
Processing PJ; a Recover only on a Posted exception; pj_dequeue
skips PJs bound to active CJs (mirrors E94's "can't dequeue
CJ-bound PJ" rule the fuzz itself discovered when the first run
flagged a CJ→missing-PJ reference).
Invariants checked every 64 ticks:
- Every tracked PJ exists in the store (size matches)
- Every CJ's prjobids all exist in PJ store
- No FSM in NoState sentinel
- EPT bucket total monotonically non-decreasing
- Defined reports' VIDs all exist
- Substrate / carrier counts match enumeration
Persistence round-trip every 500 ticks:
- Fresh shadow EquipmentDataModel loads from the same journal dir
- Diffs PJ + CJ states one-by-one + carrier/substrate/exception
counts against the live model
- Catches any "mutation didn't reach disk" or
"replay didn't reconstruct state correctly" bugs
Reproducibility:
- Each TEST_CASE uses a fixed seed (0x1, 0xdeadbeef, 0xfeedface,
0xc0ffee — 8000 ops total in the fast suite)
- World keeps a rolling 20-action trace, printed on invariant
violation so the failing sequence can be pasted into a targeted
regression test
- SECSGEM_ROBUSTNESS_SOAK=1 enables a 100k-tick soak case
(~3-5 minutes in Docker; not run by default)
The very first run found a real edge case: act_pj_dequeue removed
PJs that were bound to active CJs, leaving dangling refs. Fixed
the fuzz to filter; the underlying behavior is intentional (store
trusts the application to gate), but the fuzz now mirrors the
correct E94 contract.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
raphael