feat(grid-agent): add structured observability and replay (#127)

docs/grid-agent-control-plane.md

@@ -48,11 +48,11 @@ The JSON request envelope is stable and versioned. For example:
 {"version":1,"request_id":"health-1","request":{"method":"health"}}
 ```
 
-Observers can call `health`, `runtime`, `list_sessions`,
-`list_scheduled_jobs`, `list_pending_approvals`, `list_audit_events`, and
-`subscribe_events`. Operators can additionally call `cancel_request`,
-`pause_autonomy`, `resume_autonomy`, `cancel_action`, `decide_approval`,
-`force_reconnect`, `expire_conversation`, `set_roaming_job`,
+Observers can call `health`, `metrics`, `runtime`, `list_sessions`,
+`list_scheduled_jobs`, `list_pending_approvals`, `list_audit_events`,
+`list_observability_events`, and `subscribe_events`. Operators can additionally
+call `cancel_request`, `pause_autonomy`, `resume_autonomy`, `cancel_action`,
+`decide_approval`, `force_reconnect`, `expire_conversation`, `set_roaming_job`,
 `inject_operator_message`, and `graceful_shutdown`. Cancellation is a mutation
 and is operator-only. List requests use an opaque numeric cursor and a page
 size of 1 through 100.
@@ -62,6 +62,9 @@ region and pose fields when known, behavior mode, control-queue utilization,
 and aggregate budget use. Conversation responses contain metadata only. Audit
 and approval responses exclude arguments, prompt contents, credentials,
 authorization headers, capability URLs, model reasoning, and filesystem data.
+Unified events use pseudonymous correlation IDs and fixed-cardinality metrics;
+their schema, journal, redaction, and replay rules are documented in
+[`grid-agent-observability.md`](grid-agent-observability.md).
 `cancel_action` binds to the exact bounded action ID carried by behavior audit
 observations; it can cancel a queued or executing embodied action without
 preempting unrelated work. The built-in roaming job ID is `default-roaming`.

docs/grid-agent-observability.md

@@ -0,0 +1,118 @@
+# Grid-agent observability, audit, metrics, and replay
+
+The grid agent records externally observable decisions and outcomes without
+claiming access to private model reasoning. The stable JSON schema version is
+`1`. `StructuredEvent` is the transport-neutral representation used by the
+in-memory ring, optional JSONL journal, control API, and replay reader.
+
+## Event contract
+
+Every event has `schema_version`, monotonic `event_id`, `unix_millis`,
+`severity`, `component`, `family`, `origin`, correlation IDs, optional duration,
+retry count, result/reason codes, redaction flags, bounded scalar fields, and
+forward-compatible extension fields. Version 1 families cover lifecycle and
+behavior transitions; inbound/outbound messages; conversation creation/expiry;
+inference requests/results; model action summaries; tool proposals,
+policy/approval decisions and tool execution; scheduled jobs; control commands;
+shutdown; and diagnostic envelopes.
+
+One recorder assigns a total admission order. This preserves order for an
+action and gives concurrent producers a deterministic recorded order, but does
+not claim causal order between tasks before recorder admission. Correlation
+values are stable SHA-256 pseudonyms, so an operator can join a timeline without
+retaining original avatar, session, request, or action identifiers. Runtime tool
+names are likewise pseudonymized. Readable state/reason/result values are
+compile-time enumeration codes.
+
+Later readers ignore unknown top-level and `fields` keys. The Rust reader
+retains both across a decode/encode cycle.
+
+```json
+{"schema_version":1,"event_id":42,"unix_millis":1787025600000,"severity":"info","component":"policy","family":"tool_result","correlation":{"session_id":"id:83c1...","request_id":"id:bd10...","action_id":"id:11a0..."},"origin":"policy","duration_millis":18,"retry_count":0,"result_code":"completed","reason_code":"allowed","redaction_flags":["tool_arguments"],"fields":{"tool_calls":1}}
+```
+
+## Privacy boundary
+
+The event builder cannot accept free-form string fields. Message/prompt bodies,
+model responses, authorization headers, API keys, grid passwords, capability
+and asset URLs, inventory payloads, and hidden chain-of-thought have no runtime
+event representation. Debug output omits field and extension values. Control
+messages retain operation, role, and bounded metadata only.
+
+Diagnostic capture is explicit and bounded by `max_diagnostic_entries`. It
+emits a warning envelope with part count, total bytes, and a digest; prompt and
+response text is never retained. It carries `diagnostic_capture` and
+`content_omitted` flags. Hashes and timing can still reveal equality, so enabling
+capture remains a privacy decision.
+
+## Memory, subscribers, and JSONL journal
+
+`Observability::memory` creates a fixed ring. Eviction increments
+`dropped_ring_events`. Each subscriber has a bounded queue. Resuming before
+retained history reports an explicit missing interval. Slow subscribers are
+disconnected and counted; producers never await them.
+
+`Observability::journaled` adds an optional bounded background writer. Producers
+use `try_send`, so slow disks cannot block chat, lifecycle, or control handling.
+Queue loss increments `dropped_journal_events`. The active file is
+`events-current.jsonl.tmp`; complete segments are atomically renamed to
+`events-NNN.jsonl`. Segment count and combined active/archive bytes are bounded,
+with oldest complete segments removed first. `sync_each_record` trades latency
+for durability. `ObservabilityRuntime::shutdown` drains, flushes, and syncs.
+
+A crash may tear only the final active record. Startup truncates that fragment
+to the last newline. A single immutable reservation marker allocates the next
+bounded ID range before events are admitted, so IDs observed by subscribers but
+dropped by a saturated journal queue are not reused after restart. Malformed
+complete records, oversized records, and exhausted ID ranges fail closed.
+
+```rust,no_run
+use metacrate_grid_agent::{JournalConfig, Observability, ObservabilityLimits};
+use std::path::PathBuf;
+
+# async fn example() -> Result<(), Box<dyn std::error::Error>> {
+let (events, runtime) = Observability::journaled(
+    ObservabilityLimits::default(),
+    JournalConfig {
+        directory: PathBuf::from("data/grid-agent/audit"),
+        max_segment_bytes: 4 * 1024 * 1024,
+        max_segments: 8,
+        max_total_bytes: 32 * 1024 * 1024,
+        sync_each_record: false,
+    },
+)?;
+runtime.shutdown(&events).await?;
+# Ok(()) }
+```
+
+## Metrics and control views
+
+Metrics use fixed atomics with no caller-defined labels or external telemetry.
+The snapshot contains readiness, reconnects, active sessions/tasks, queue use,
+inference/tool latency count/sum/max, fixed inference/tool/policy outcome
+buckets, rate-limit use, recorded events, and ring/subscriber/journal drop
+counters.
+
+Control protocol v1 provides read-only `metrics` and paginated
+`list_observability_events` requests in addition to the policy audit. Observer
+and operator roles may read them; sensitive payloads cannot enter either view.
+
+## Deterministic replay
+
+`replay_journal` reads retained segments and the recovered active file with
+explicit event-size/count limits. It rebuilds lifecycle and behavior state,
+session counts, policy/tool outcomes, and per-action observable timelines.
+Redaction flags set `unavailable_private_content`, making missing context clear.
+
+Replay is data-only: it has no grid backend, LLM client, policy gateway, or tool
+executor, so it cannot resend messages or repeat mutations. Rotation means it
+reconstructs the retained window, not already expired history.
+
+## Focused verification
+
+```sh
+cargo test --locked -p metacrate-grid-agent --lib observability_tests
+cargo test --locked -p metacrate-grid-agent --test dependency_policy
+cargo clippy --locked -p metacrate-grid-agent --all-targets -- -D warnings
+RUSTDOCFLAGS="-D warnings" cargo doc --locked -p metacrate-grid-agent --no-deps
+```