Grid-agent observability, audit, metrics, and replay

The grid agent records externally observable decisions and outcomes without claiming access to private model reasoning. The stable JSON schema version is 1. StructuredEvent is the transport-neutral representation used by the in-memory ring, optional JSONL journal, control API, and replay reader.

Event contract

Every event has schema_version, monotonic event_id, unix_millis, severity, component, family, origin, correlation IDs, optional duration, retry count, result/reason codes, redaction flags, bounded scalar fields, and forward-compatible extension fields. Version 1 families cover lifecycle and behavior transitions; inbound/outbound messages; conversation creation/expiry; inference requests/results; model action summaries; tool proposals, policy/approval decisions and tool execution; scheduled jobs; control commands; shutdown; and diagnostic envelopes.

One recorder assigns a total admission order. This preserves order for an action and gives concurrent producers a deterministic recorded order, but does not claim causal order between tasks before recorder admission. Correlation values are stable SHA-256 pseudonyms, so an operator can join a timeline without retaining original avatar, session, request, or action identifiers. Runtime tool names are likewise pseudonymized. Readable state/reason/result values are compile-time enumeration codes.

Later readers ignore unknown top-level and fields keys. The Rust reader retains both across a decode/encode cycle.

{"schema_version":1,"event_id":42,"unix_millis":1787025600000,"severity":"info","component":"policy","family":"tool_result","correlation":{"session_id":"id:83c1...","request_id":"id:bd10...","action_id":"id:11a0..."},"origin":"policy","duration_millis":18,"retry_count":0,"result_code":"completed","reason_code":"allowed","redaction_flags":["tool_arguments"],"fields":{"tool_calls":1}}

Privacy boundary

The event builder cannot accept free-form string fields. Message/prompt bodies, model responses, authorization headers, API keys, grid passwords, capability and asset URLs, inventory payloads, and hidden chain-of-thought have no runtime event representation. Debug output omits field and extension values. Control messages retain operation, role, and bounded metadata only.

Diagnostic capture is explicit and bounded by max_diagnostic_entries. It emits a warning envelope with part count, total bytes, and a digest; prompt and response text is never retained. It carries diagnostic_capture and content_omitted flags. Hashes and timing can still reveal equality, so enabling capture remains a privacy decision.

Memory, subscribers, and JSONL journal

Observability::memory creates a fixed ring. Eviction increments dropped_ring_events. Each subscriber has a bounded queue. Resuming before retained history reports an explicit missing interval. Slow subscribers are disconnected and counted; producers never await them.

Observability::journaled adds an optional bounded background writer. Producers use try_send, so slow disks cannot block chat, lifecycle, or control handling. Queue loss increments dropped_journal_events. The active file is events-current.jsonl.tmp; complete segments are atomically renamed to events-NNN.jsonl. Segment count and combined active/archive bytes are bounded, with oldest complete segments removed first. sync_each_record trades latency for durability. ObservabilityRuntime::shutdown drains, flushes, and syncs.

A crash may tear only the final active record. Startup truncates that fragment to the last newline. A single immutable reservation marker allocates the next bounded ID range before events are admitted, so IDs observed by subscribers but dropped by a saturated journal queue are not reused after restart. Malformed complete records, oversized records, and exhausted ID ranges fail closed.

use metacrate_grid_agent::{JournalConfig, Observability, ObservabilityLimits};
use std::path::PathBuf;

# async fn example() -> Result<(), Box<dyn std::error::Error>> {
let (events, runtime) = Observability::journaled(
    ObservabilityLimits::default(),
    JournalConfig {
        directory: PathBuf::from("data/grid-agent/audit"),
        max_segment_bytes: 4 * 1024 * 1024,
        max_segments: 8,
        max_total_bytes: 32 * 1024 * 1024,
        sync_each_record: false,
    },
)?;
runtime.shutdown(&events).await?;
# Ok(()) }

Metrics and control views

Metrics use fixed atomics with no caller-defined labels or external telemetry. The snapshot contains readiness, reconnects, active sessions/tasks, queue use, inference/tool latency count/sum/max, fixed inference/tool/policy outcome buckets, rate-limit use, recorded events, and ring/subscriber/journal drop counters.

Control protocol v1 provides read-only metrics and paginated list_observability_events requests in addition to the policy audit. Observer and operator roles may read them; sensitive payloads cannot enter either view.

Deterministic replay

replay_journal reads retained segments and the recovered active file with explicit event-size/count limits. It rebuilds lifecycle and behavior state, session counts, policy/tool outcomes, and per-action observable timelines. Redaction flags set unavailable_private_content, making missing context clear.

Replay is data-only: it has no grid backend, LLM client, policy gateway, or tool executor, so it cannot resend messages or repeat mutations. Rotation means it reconstructs the retained window, not already expired history.

Focused verification

cargo test --locked -p metacrate-grid-agent --lib observability_tests
cargo test --locked -p metacrate-grid-agent --test dependency_policy
cargo clippy --locked -p metacrate-grid-agent --all-targets -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --locked -p metacrate-grid-agent --no-deps

5.8 KiB Raw Blame History