# Grid-agent observability, audit, metrics, and replay

The grid agent records externally observable decisions and outcomes without
claiming access to private model reasoning. The stable JSON schema version is
`1`. `StructuredEvent` is the transport-neutral representation used by the
in-memory ring, optional JSONL journal, control API, and replay reader.

## Event contract

Every event has `schema_version`, monotonic `event_id`, `unix_millis`,
`severity`, `component`, `family`, `origin`, correlation IDs, optional duration,
retry count, result/reason codes, redaction flags, bounded scalar fields, and
forward-compatible extension fields. Version 1 families cover lifecycle and
behavior transitions; inbound/outbound messages; conversation creation/expiry;
inference requests/results; model action summaries; tool proposals,
policy/approval decisions and tool execution; scheduled jobs; control commands;
shutdown; and diagnostic envelopes.

One recorder assigns a total admission order. This preserves order for an
action and gives concurrent producers a deterministic recorded order, but does
not claim causal order between tasks before recorder admission. Correlation
values are stable SHA-256 pseudonyms, so an operator can join a timeline without
retaining original avatar, session, request, or action identifiers. Runtime tool
names are likewise pseudonymized. Readable state/reason/result values are
compile-time enumeration codes.

Later readers ignore unknown top-level and `fields` keys. The Rust reader
retains both across a decode/encode cycle.

```json
{"schema_version":1,"event_id":42,"unix_millis":1787025600000,"severity":"info","component":"policy","family":"tool_result","correlation":{"session_id":"id:83c1...","request_id":"id:bd10...","action_id":"id:11a0..."},"origin":"policy","duration_millis":18,"retry_count":0,"result_code":"completed","reason_code":"allowed","redaction_flags":["tool_arguments"],"fields":{"tool_calls":1}}
```

## Privacy boundary

The event builder cannot accept free-form string fields. Message/prompt bodies,
model responses, authorization headers, API keys, grid passwords, capability
and asset URLs, inventory payloads, and hidden chain-of-thought have no runtime
event representation. Debug output omits field and extension values. Control
messages retain operation, role, and bounded metadata only.

Diagnostic capture is explicit and bounded by `max_diagnostic_entries`. It
emits a warning envelope with part count, total bytes, and a digest; prompt and
response text is never retained. It carries `diagnostic_capture` and
`content_omitted` flags. Hashes and timing can still reveal equality, so enabling
capture remains a privacy decision.

## Memory, subscribers, and JSONL journal

`Observability::memory` creates a fixed ring. Eviction increments
`dropped_ring_events`. Each subscriber has a bounded queue. Resuming before
retained history reports an explicit missing interval. Slow subscribers are
disconnected and counted; producers never await them.

`Observability::journaled` adds an optional bounded background writer. Producers
use `try_send`, so slow disks cannot block chat, lifecycle, or control handling.
Queue loss increments `dropped_journal_events`. The active file is
`events-current.jsonl.tmp`; complete segments are atomically renamed to
`events-NNN.jsonl`. Segment count and combined active/archive bytes are bounded,
with oldest complete segments removed first. `sync_each_record` trades latency
for durability. `ObservabilityRuntime::shutdown` drains, flushes, and syncs.

A crash may tear only the final active record. Startup truncates that fragment
to the last newline. A single immutable reservation marker allocates the next
bounded ID range before events are admitted, so IDs observed by subscribers but
dropped by a saturated journal queue are not reused after restart. Malformed
complete records, oversized records, and exhausted ID ranges fail closed.

```rust,no_run
use metacrate_grid_agent::{JournalConfig, Observability, ObservabilityLimits};
use std::path::PathBuf;

# async fn example() -> Result<(), Box<dyn std::error::Error>> {
let (events, runtime) = Observability::journaled(
    ObservabilityLimits::default(),
    JournalConfig {
        directory: PathBuf::from("data/grid-agent/audit"),
        max_segment_bytes: 4 * 1024 * 1024,
        max_segments: 8,
        max_total_bytes: 32 * 1024 * 1024,
        sync_each_record: false,
    },
)?;
runtime.shutdown(&events).await?;
# Ok(()) }
```

## Metrics and control views

Metrics use fixed atomics with no caller-defined labels or external telemetry.
The snapshot contains readiness, reconnects, active sessions/tasks, queue use,
inference/tool latency count/sum/max, fixed inference/tool/policy outcome
buckets, rate-limit use, recorded events, and ring/subscriber/journal drop
counters.

Control protocol v1 provides read-only `metrics` and paginated
`list_observability_events` requests in addition to the policy audit. Observer
and operator roles may read them; sensitive payloads cannot enter either view.

## Deterministic replay

`replay_journal` reads retained segments and the recovered active file with
explicit event-size/count limits. It rebuilds lifecycle and behavior state,
session counts, policy/tool outcomes, and per-action observable timelines.
Redaction flags set `unavailable_private_content`, making missing context clear.

Replay is data-only: it has no grid backend, LLM client, policy gateway, or tool
executor, so it cannot resend messages or repeat mutations. Rotation means it
reconstructs the retained window, not already expired history.

## Focused verification

```sh
cargo test --locked -p metacrate-grid-agent --lib observability_tests
cargo test --locked -p metacrate-grid-agent --test dependency_policy
cargo clippy --locked -p metacrate-grid-agent --all-targets -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --locked -p metacrate-grid-agent --no-deps
```