119 lines
5.8 KiB
Markdown
119 lines
5.8 KiB
Markdown
# Grid-agent observability, audit, metrics, and replay
|
|
|
|
The grid agent records externally observable decisions and outcomes without
|
|
claiming access to private model reasoning. The stable JSON schema version is
|
|
`1`. `StructuredEvent` is the transport-neutral representation used by the
|
|
in-memory ring, optional JSONL journal, control API, and replay reader.
|
|
|
|
## Event contract
|
|
|
|
Every event has `schema_version`, monotonic `event_id`, `unix_millis`,
|
|
`severity`, `component`, `family`, `origin`, correlation IDs, optional duration,
|
|
retry count, result/reason codes, redaction flags, bounded scalar fields, and
|
|
forward-compatible extension fields. Version 1 families cover lifecycle and
|
|
behavior transitions; inbound/outbound messages; conversation creation/expiry;
|
|
inference requests/results; model action summaries; tool proposals,
|
|
policy/approval decisions and tool execution; scheduled jobs; control commands;
|
|
shutdown; and diagnostic envelopes.
|
|
|
|
One recorder assigns a total admission order. This preserves order for an
|
|
action and gives concurrent producers a deterministic recorded order, but does
|
|
not claim causal order between tasks before recorder admission. Correlation
|
|
values are stable SHA-256 pseudonyms, so an operator can join a timeline without
|
|
retaining original avatar, session, request, or action identifiers. Runtime tool
|
|
names are likewise pseudonymized. Readable state/reason/result values are
|
|
compile-time enumeration codes.
|
|
|
|
Later readers ignore unknown top-level and `fields` keys. The Rust reader
|
|
retains both across a decode/encode cycle.
|
|
|
|
```json
|
|
{"schema_version":1,"event_id":42,"unix_millis":1787025600000,"severity":"info","component":"policy","family":"tool_result","correlation":{"session_id":"id:83c1...","request_id":"id:bd10...","action_id":"id:11a0..."},"origin":"policy","duration_millis":18,"retry_count":0,"result_code":"completed","reason_code":"allowed","redaction_flags":["tool_arguments"],"fields":{"tool_calls":1}}
|
|
```
|
|
|
|
## Privacy boundary
|
|
|
|
The event builder cannot accept free-form string fields. Message/prompt bodies,
|
|
model responses, authorization headers, API keys, grid passwords, capability
|
|
and asset URLs, inventory payloads, and hidden chain-of-thought have no runtime
|
|
event representation. Debug output omits field and extension values. Control
|
|
messages retain operation, role, and bounded metadata only.
|
|
|
|
Diagnostic capture is explicit and bounded by `max_diagnostic_entries`. It
|
|
emits a warning envelope with part count, total bytes, and a digest; prompt and
|
|
response text is never retained. It carries `diagnostic_capture` and
|
|
`content_omitted` flags. Hashes and timing can still reveal equality, so enabling
|
|
capture remains a privacy decision.
|
|
|
|
## Memory, subscribers, and JSONL journal
|
|
|
|
`Observability::memory` creates a fixed ring. Eviction increments
|
|
`dropped_ring_events`. Each subscriber has a bounded queue. Resuming before
|
|
retained history reports an explicit missing interval. Slow subscribers are
|
|
disconnected and counted; producers never await them.
|
|
|
|
`Observability::journaled` adds an optional bounded background writer. Producers
|
|
use `try_send`, so slow disks cannot block chat, lifecycle, or control handling.
|
|
Queue loss increments `dropped_journal_events`. The active file is
|
|
`events-current.jsonl.tmp`; complete segments are atomically renamed to
|
|
`events-NNN.jsonl`. Segment count and combined active/archive bytes are bounded,
|
|
with oldest complete segments removed first. `sync_each_record` trades latency
|
|
for durability. `ObservabilityRuntime::shutdown` drains, flushes, and syncs.
|
|
|
|
A crash may tear only the final active record. Startup truncates that fragment
|
|
to the last newline. A single immutable reservation marker allocates the next
|
|
bounded ID range before events are admitted, so IDs observed by subscribers but
|
|
dropped by a saturated journal queue are not reused after restart. Malformed
|
|
complete records, oversized records, and exhausted ID ranges fail closed.
|
|
|
|
```rust,no_run
|
|
use metacrate_grid_agent::{JournalConfig, Observability, ObservabilityLimits};
|
|
use std::path::PathBuf;
|
|
|
|
# async fn example() -> Result<(), Box<dyn std::error::Error>> {
|
|
let (events, runtime) = Observability::journaled(
|
|
ObservabilityLimits::default(),
|
|
JournalConfig {
|
|
directory: PathBuf::from("data/grid-agent/audit"),
|
|
max_segment_bytes: 4 * 1024 * 1024,
|
|
max_segments: 8,
|
|
max_total_bytes: 32 * 1024 * 1024,
|
|
sync_each_record: false,
|
|
},
|
|
)?;
|
|
runtime.shutdown(&events).await?;
|
|
# Ok(()) }
|
|
```
|
|
|
|
## Metrics and control views
|
|
|
|
Metrics use fixed atomics with no caller-defined labels or external telemetry.
|
|
The snapshot contains readiness, reconnects, active sessions/tasks, queue use,
|
|
inference/tool latency count/sum/max, fixed inference/tool/policy outcome
|
|
buckets, rate-limit use, recorded events, and ring/subscriber/journal drop
|
|
counters.
|
|
|
|
Control protocol v1 provides read-only `metrics` and paginated
|
|
`list_observability_events` requests in addition to the policy audit. Observer
|
|
and operator roles may read them; sensitive payloads cannot enter either view.
|
|
|
|
## Deterministic replay
|
|
|
|
`replay_journal` reads retained segments and the recovered active file with
|
|
explicit event-size/count limits. It rebuilds lifecycle and behavior state,
|
|
session counts, policy/tool outcomes, and per-action observable timelines.
|
|
Redaction flags set `unavailable_private_content`, making missing context clear.
|
|
|
|
Replay is data-only: it has no grid backend, LLM client, policy gateway, or tool
|
|
executor, so it cannot resend messages or repeat mutations. Rotation means it
|
|
reconstructs the retained window, not already expired history.
|
|
|
|
## Focused verification
|
|
|
|
```sh
|
|
cargo test --locked -p metacrate-grid-agent --lib observability_tests
|
|
cargo test --locked -p metacrate-grid-agent --test dependency_policy
|
|
cargo clippy --locked -p metacrate-grid-agent --all-targets -- -D warnings
|
|
RUSTDOCFLAGS="-D warnings" cargo doc --locked -p metacrate-grid-agent --no-deps
|
|
```
|