MetaCrate/docs/concurrency-hardening.md

Concurrency, cancellation, and resource audit

metacrate-concurrency-audit is the deterministic native Rust stress/soak gate for client lifetime and resource ownership. Every run executes the work; there are no ignored tests, environment-dependent skips, or success paths that merely report an unavailable service.

The default mode is deliberately offline. It does not read .env, authenticate, or contact Second Life or an OpenSim grid. Those credentials are reserved for the separately documented live-grid smoke gate. This audit instead uses injected HTTP responses, temporary files it owns, localhost UDP sockets, and the real native encrypted WebRTC/Opus loopback transport. That makes saturation, races, and resource baselines reproducible without mutating a grid account.

Scenarios and invariants

After one unmeasured warm-up, every measured cycle performs all of these steps:

• start a client-owned task and stop/join it through four concurrent, idempotent GridClient shutdown calls;
• run concurrent deduplicated downloads, independent cancellation, and a saturated 256-entry dispatcher queue whose excess requests must be rejected;
• create, mutate, and remove inventory folders/items while an actual inventory cleanup worker is alive;
• attach and release inventory, appearance, and asset subscriptions, verified with weak-reference retention probes;
• create, read, close, and delete temporary files and their owned directory;
• repeatedly open localhost UDP clients, exchange datagrams, close them, and rebind the server address to prove socket release;
• establish and shut down a real native WebRTC loopback session and require both the peer and signaling task counters to return to zero.

The focused library regressions additionally prove that asset and appearance subscriber panics cannot prevent later handlers from running, client service shutdown callbacks execute outside the registry lock, and saturated download queues drain completely when cancelled. Concurrent voice provisioning admits exactly one session without holding the signaling state lock across awaited I/O or leaking the rejected attempt.

Thresholds and evidence

ci/concurrency-thresholds.json is the reviewed policy. It requires at least four and at most 256 cycles, fixes 32 operations per cycle, permits at most 16 MiB of retained allocator growth after warm-up, and limits a run to 180 seconds. Task, dispatcher, download, inventory-worker, subscription, file, socket, voice, and signaling counts must all exactly match the zero baseline after shutdown; the memory allowance does not relax those exact resource checks.

Run the minimum deterministic stress gate with:

cargo run --locked -p metacrate-concurrency-audit -- \
  --cycles 4 --evidence /tmp/metacrate-concurrency-audit.json

Run the CI soak length with --cycles 16. Evidence contains only scenario names, counts, allocation deltas, elapsed time, thresholds, and outcome. It does not contain credentials, hostnames, capability URLs, temporary paths, or grid identities. Gitea uploads the evidence produced by the 16-cycle run.

The resource counters are portable Rust ownership counters rather than Linux-only /proc measurements, so the same runner and invariants apply on Linux, macOS, and Windows. Repository CI runs on ubuntu-latest as required.