[Grid agent] Record structured observability, audit, metrics, and replay data #127

New Issue

hugo · 2026-08-17T19:31:36Z

hugo commented

2026-08-17 19:31:36 +00:00

Objective

Make the agent's externally visible decision process, world actions, and failures inspectable without exposing secrets or claiming access to hidden model reasoning.

Event and storage model

Define stable structured events for lifecycle transitions, inbound/outbound messages, session creation/expiry, inference request/result metadata, model-authored action summaries, proposed tools, policy decisions, approvals, tool start/progress/result, behavior transitions, scheduled jobs, control commands, and shutdown.
Carry event ID, timestamp, severity, component, avatar/session/request/action correlation IDs, origin, duration, retry count, result/reason code, and redaction flags. Preserve ordering per action and document cross-task ordering limits.
Do not log API keys, passwords, authorization headers, capability/asset URLs, inventory payloads, full private conversations by default, or hidden chain-of-thought. Offer explicit bounded diagnostic capture of redacted prompt/response envelopes with a warning and retention limit.
Provide an in-memory ring buffer plus optional rotating JSONL journal using atomic/recoverable writes. Bound disk use, retention, field sizes, subscriber queues, and metrics cardinality; recover safely from a truncated final record.
Export a compact health/metrics snapshot through the control plane: readiness, reconnects, active sessions/tasks, queue depth, inference/tool latency, tool/policy outcomes, rate-limit use, and dropped-event counters. No always-on external telemetry.
Add a deterministic replay reader that reconstructs externally observable state transitions and tool/policy outcomes without re-contacting the grid or LLM and without re-executing mutations.

Acceptance criteria

Golden-schema tests cover each event family, redaction, rotation, recovery, event gaps, overflow, correlation, and forward-compatible unknown fields.
Secret canaries placed in all inputs never appear in logs, metrics, control events, panic output, or Debug representations.
Replay produces the same observable action timeline and clearly marks unavailable/private content.
Logging and metric collection remain bounded and do not materially block message handling under load.

Dependencies

Depends on architecture and control-plane event contracts. Every later feature must add its event coverage before completion.

## Objective Make the agent's externally visible decision process, world actions, and failures inspectable without exposing secrets or claiming access to hidden model reasoning. ## Event and storage model - Define stable structured events for lifecycle transitions, inbound/outbound messages, session creation/expiry, inference request/result metadata, model-authored action summaries, proposed tools, policy decisions, approvals, tool start/progress/result, behavior transitions, scheduled jobs, control commands, and shutdown. - Carry event ID, timestamp, severity, component, avatar/session/request/action correlation IDs, origin, duration, retry count, result/reason code, and redaction flags. Preserve ordering per action and document cross-task ordering limits. - Do not log API keys, passwords, authorization headers, capability/asset URLs, inventory payloads, full private conversations by default, or hidden chain-of-thought. Offer explicit bounded diagnostic capture of redacted prompt/response envelopes with a warning and retention limit. - Provide an in-memory ring buffer plus optional rotating JSONL journal using atomic/recoverable writes. Bound disk use, retention, field sizes, subscriber queues, and metrics cardinality; recover safely from a truncated final record. - Export a compact health/metrics snapshot through the control plane: readiness, reconnects, active sessions/tasks, queue depth, inference/tool latency, tool/policy outcomes, rate-limit use, and dropped-event counters. No always-on external telemetry. - Add a deterministic replay reader that reconstructs externally observable state transitions and tool/policy outcomes without re-contacting the grid or LLM and without re-executing mutations. ## Acceptance criteria - [x] Golden-schema tests cover each event family, redaction, rotation, recovery, event gaps, overflow, correlation, and forward-compatible unknown fields. - [x] Secret canaries placed in all inputs never appear in logs, metrics, control events, panic output, or `Debug` representations. - [x] Replay produces the same observable action timeline and clearly marks unavailable/private content. - [x] Logging and metric collection remain bounded and do not materially block message handling under load. ## Dependencies Depends on architecture and control-plane event contracts. Every later feature must add its event coverage before completion.

hugo added this to the 14 - metacrate grid agent milestone 2026-08-17 19:31:36 +00:00

hugo added the enhancement label 2026-08-17 19:31:36 +00:00

hugo referenced this issue

2026-08-17 19:32:14 +00:00

implement an AI driven agent that runs in opensim #117

hugo referenced this issue from a commit

2026-08-18 05:29:00 +00:00

feat(grid-agent): add structured observability and replay (#127)

hugo commented

2026-08-18 05:29:20 +00:00

Implemented and pushed as commit 8658c04.

This adds the versioned structured event schema and stable lifecycle/message/session/inference/model/tool/policy/approval/behavior/job/control/shutdown families; pseudonymous correlations and redacted Debug surfaces; a bounded memory ring and slow-subscriber/gap handling; an optional bounded rotating JSONL journal with torn-tail recovery, durable non-reused event IDs, hard retention limits, and nonblocking producer behavior; fixed-cardinality health, latency, outcome, rate, and drop metrics exposed through the control plane; unified policy audit mirroring; live session/interaction/perception/behavior/control wiring; content-free diagnostic envelopes; and deterministic data-only replay. Documentation and golden/redaction/rotation/recovery/overflow/correlation/replay tests are included.

Verified on this host:

cargo test --locked -j 1 -p metacrate-grid-agent --lib --no-fail-fast: 101 passed
all five grid-agent compatibility targets: 24 passed after the reviewed source-count update
cargo clippy --locked -j 1 -p metacrate-grid-agent --all-targets -- -D warnings
RUSTDOCFLAGS=-D warnings cargo doc --locked -j 1 -p metacrate-grid-agent --no-deps
cargo deny check advisories licenses bans sources --hide-inclusion-graph
real default metacrate-grid-agent binary build, check-config, and run-once
live-grid binary wiring compiles against the API-faithful libremetaverse harness

I am intentionally leaving #127 open for the larger-machine handoff. The mandatory build against the real monolithic libremetaverse crate still receives SIGKILL on this 7.6 GiB host with no swap, including single-job/debug-disabled and reduced-rustc-memory attempts. On the larger machine, rerun: CARGO_PROFILE_DEV_DEBUG=0 CARGO_INCREMENTAL=0 cargo build --locked -j 1 -p metacrate-grid-agent --features live-grid --bin metacrate-grid-agent. Close #127 only after that real build succeeds.

Implemented and pushed as commit 8658c04. This adds the versioned structured event schema and stable lifecycle/message/session/inference/model/tool/policy/approval/behavior/job/control/shutdown families; pseudonymous correlations and redacted Debug surfaces; a bounded memory ring and slow-subscriber/gap handling; an optional bounded rotating JSONL journal with torn-tail recovery, durable non-reused event IDs, hard retention limits, and nonblocking producer behavior; fixed-cardinality health, latency, outcome, rate, and drop metrics exposed through the control plane; unified policy audit mirroring; live session/interaction/perception/behavior/control wiring; content-free diagnostic envelopes; and deterministic data-only replay. Documentation and golden/redaction/rotation/recovery/overflow/correlation/replay tests are included. Verified on this host: - cargo test --locked -j 1 -p metacrate-grid-agent --lib --no-fail-fast: 101 passed - all five grid-agent compatibility targets: 24 passed after the reviewed source-count update - cargo clippy --locked -j 1 -p metacrate-grid-agent --all-targets -- -D warnings - RUSTDOCFLAGS=-D warnings cargo doc --locked -j 1 -p metacrate-grid-agent --no-deps - cargo deny check advisories licenses bans sources --hide-inclusion-graph - real default metacrate-grid-agent binary build, check-config, and run-once - live-grid binary wiring compiles against the API-faithful libremetaverse harness I am intentionally leaving #127 open for the larger-machine handoff. The mandatory build against the real monolithic libremetaverse crate still receives SIGKILL on this 7.6 GiB host with no swap, including single-job/debug-disabled and reduced-rustc-memory attempts. On the larger machine, rerun: CARGO_PROFILE_DEV_DEBUG=0 CARGO_INCREMENTAL=0 cargo build --locked -j 1 -p metacrate-grid-agent --features live-grid --bin metacrate-grid-agent. Close #127 only after that real build succeeds.

hugo commented

2026-08-18 07:57:42 +00:00

Larger-machine handoff completed on commit e99ee0f.

Closure evidence:

Normal full binary build passed: cargo build --locked --offline -p metacrate-grid-agent --features live-grid --bin metacrate-grid-agent.
The actual live-enabled agent completed one supervised login, full readiness, and clean logout cycle against the configured grid: grid agent completed one supervised login/logout cycle.
Focused observability acceptance suite: 7 passed.
Dependency-policy suite: 5 passed.
Full live-grid agent suite previously passed: 126 tests including doc tests.
Clippy passed with -D warnings for all targets with live-grid.
Rustdoc passed with -D warnings and live-grid.
The full pure-Rust workspace/all-target build passed; native OpenJPEG and native Skia remain opt-in.

The prior monolithic-build blocker is resolved, all acceptance criteria are verified, and the worktree is clean.

Larger-machine handoff completed on commit e99ee0f. Closure evidence: - Normal full binary build passed: `cargo build --locked --offline -p metacrate-grid-agent --features live-grid --bin metacrate-grid-agent`. - The actual live-enabled agent completed one supervised login, full readiness, and clean logout cycle against the configured grid: `grid agent completed one supervised login/logout cycle`. - Focused observability acceptance suite: 7 passed. - Dependency-policy suite: 5 passed. - Full live-grid agent suite previously passed: 126 tests including doc tests. - Clippy passed with `-D warnings` for all targets with `live-grid`. - Rustdoc passed with `-D warnings` and `live-grid`. - The full pure-Rust workspace/all-target build passed; native OpenJPEG and native Skia remain opt-in. The prior monolithic-build blocker is resolved, all acceptance criteria are verified, and the worktree is clean.

hugo closed this issue

2026-08-18 07:57:42 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/MetaCrate#127