[Grid agent] Expose bounded world perception and read-only knowledge tools #124

New Issue

hugo · 2026-08-17T19:31:35Z

hugo commented

2026-08-17 19:31:35 +00:00

Objective

Give the model a compact, current, inspectable understanding of the virtual world through bounded read-only tools rather than dumping raw protocol state into prompts.

Tool surface

Build a generation-tagged world snapshot from existing agent, object, avatar, parcel, terrain/environment, inventory, directory/social, and simulator managers only after their readiness conditions are met.
Provide narrowly scoped tools for current location/region, agent state, nearby avatars with distance/direction, visible/tracked objects and safe properties, parcel/environment facts, inventory search/metadata, received landmarks, and recent conversation participants.
Distinguish observed facts, stale/cached facts, unknown values, and model inference. Include observation timestamps and region IDs. Never invent data when a capability or manager is unavailable.
Apply distance, count, byte, field, and execution-time bounds; sort deterministically and paginate or summarize. Sanitize object names, descriptions, hover text, parcel text, and inventory metadata as untrusted content.
Exclude secrets, asset/capability URLs, owner-only/private data not needed for behavior, raw binary assets, and other users' conversation contents.
Cache expensive observations for a short explicit TTL and invalidate on region change/reconnect. Read tools must not trigger purchases, uploads, inventory acceptance, object selection side effects, or unrelated network floods.

Acceptance criteria

Each tool has a stable schema, policy declaration, size/time budget, freshness semantics, and representative model-facing example.
Deterministic fixtures test empty/partial worlds, large crowds/object sets, stale generations, malicious metadata, invalid coordinates, region crossing, cancellation, and missing capabilities.
Prompt payload tests prove bounded size and preserve the untrusted-data markers.
Read-only tools emit correlated timing/result summaries to observability without logging sensitive full payloads.

Dependencies

Depends on architecture, lifecycle generation fencing, LLM tools, and policy. Visual pixel rendering is a separate issue.

## Objective Give the model a compact, current, inspectable understanding of the virtual world through bounded read-only tools rather than dumping raw protocol state into prompts. ## Tool surface - Build a generation-tagged world snapshot from existing agent, object, avatar, parcel, terrain/environment, inventory, directory/social, and simulator managers only after their readiness conditions are met. - Provide narrowly scoped tools for current location/region, agent state, nearby avatars with distance/direction, visible/tracked objects and safe properties, parcel/environment facts, inventory search/metadata, received landmarks, and recent conversation participants. - Distinguish observed facts, stale/cached facts, unknown values, and model inference. Include observation timestamps and region IDs. Never invent data when a capability or manager is unavailable. - Apply distance, count, byte, field, and execution-time bounds; sort deterministically and paginate or summarize. Sanitize object names, descriptions, hover text, parcel text, and inventory metadata as untrusted content. - Exclude secrets, asset/capability URLs, owner-only/private data not needed for behavior, raw binary assets, and other users' conversation contents. - Cache expensive observations for a short explicit TTL and invalidate on region change/reconnect. Read tools must not trigger purchases, uploads, inventory acceptance, object selection side effects, or unrelated network floods. ## Acceptance criteria - [ ] Each tool has a stable schema, policy declaration, size/time budget, freshness semantics, and representative model-facing example. - [ ] Deterministic fixtures test empty/partial worlds, large crowds/object sets, stale generations, malicious metadata, invalid coordinates, region crossing, cancellation, and missing capabilities. - [ ] Prompt payload tests prove bounded size and preserve the untrusted-data markers. - [ ] Read-only tools emit correlated timing/result summaries to observability without logging sensitive full payloads. ## Dependencies Depends on architecture, lifecycle generation fencing, LLM tools, and policy. Visual pixel rendering is a separate issue.

hugo added this to the 14 - metacrate grid agent milestone 2026-08-17 19:31:35 +00:00

hugo added the enhancement label 2026-08-17 19:31:35 +00:00

hugo referenced this issue

2026-08-17 19:32:14 +00:00

implement an AI driven agent that runs in opensim #117

hugo referenced this issue from a commit

2026-08-17 23:52:46 +00:00

feat(grid-agent): add bounded world perception tools (#124)

hugo commented

2026-08-17 23:53:10 +00:00

Implemented in commit 3a0ead7 (pushed to main).

What landed:

generation/region-fenced, readiness-gated snapshots over existing native agent/simulator/avatar/object/parcel/terrain/environment/inventory caches, with reconnect and simulator-change invalidation;
eight stable read-only tools: world_location, agent_state, nearby_avatars, visible_objects, parcel_environment, inventory_search, received_landmarks, and recent_participants;
deterministic bounded selection/sort/pagination, 2s cache TTL, one end-to-end 3s deadline, cancellation, single-flight capture, 256m radius, 20-item pages, 2,048-item source caps, 32KiB results, and explicit source_truncated/freshness/provenance/unknown markers;
untrusted metadata sanitization and URL/secret redaction; attachments, owners, permissions, media/capability URLs, raw assets, and conversation bodies are absent;
policy registration at Informational/ReadOnly/idempotent/one-call cost. Owner-private inventory, landmarks, and participant metadata are limited to authorized IM, local operator, or an explicit scheduler grant;
content-free correlated perception telemetry and complete operator/model-facing schema/budget/examples in docs/grid-agent-perception.md;
live service wiring reuses the native manager graph and never starts directory searches, asset fetches, object selection, purchases, uploads, or inventory acceptance.

Verification:

70 library tests green, including 13 focused deterministic perception cases covering empty/partial state, 2,048-avatar and 2,048-object fixtures, stale generation, invalid coordinates, malicious metadata, missing facts, cache hit/expiry, region crossing, cancellation/timeout, concurrent single-flight capture, policy privacy, payload bounds/markers, and telemetry;
dependency policy 4/4, policy integration 3/3, conversation isolation 1/1, and LLM/tool-loop integration 14/14 green;
cargo clippy --all-targets -D warnings, rustdoc -D warnings, cargo fmt --check, git diff --check, and cargo deny check green;
live-grid API-faithful feature check and clippy -D warnings green.

Runner limitations recorded transparently: compiling the repository full generated libremetaverse source is SIGKILLed by this 8GiB runner before rustc reaches metacrate-grid-agent, both with normal debug settings and debug info disabled. The focused API-faithful live boundary build checks the exact manager signatures used here. The Windows target check reaches aws-lc-sys but this runner lacks x86_64-w64-mingw32-gcc; the implementation itself adds no platform-specific API.

Post-implementation review against every #124 objective/acceptance bullet found no remaining issue-scope gap.

Implemented in commit 3a0ead7 (pushed to main). What landed: - generation/region-fenced, readiness-gated snapshots over existing native agent/simulator/avatar/object/parcel/terrain/environment/inventory caches, with reconnect and simulator-change invalidation; - eight stable read-only tools: world_location, agent_state, nearby_avatars, visible_objects, parcel_environment, inventory_search, received_landmarks, and recent_participants; - deterministic bounded selection/sort/pagination, 2s cache TTL, one end-to-end 3s deadline, cancellation, single-flight capture, 256m radius, 20-item pages, 2,048-item source caps, 32KiB results, and explicit source_truncated/freshness/provenance/unknown markers; - untrusted metadata sanitization and URL/secret redaction; attachments, owners, permissions, media/capability URLs, raw assets, and conversation bodies are absent; - policy registration at Informational/ReadOnly/idempotent/one-call cost. Owner-private inventory, landmarks, and participant metadata are limited to authorized IM, local operator, or an explicit scheduler grant; - content-free correlated perception telemetry and complete operator/model-facing schema/budget/examples in docs/grid-agent-perception.md; - live service wiring reuses the native manager graph and never starts directory searches, asset fetches, object selection, purchases, uploads, or inventory acceptance. Verification: - 70 library tests green, including 13 focused deterministic perception cases covering empty/partial state, 2,048-avatar and 2,048-object fixtures, stale generation, invalid coordinates, malicious metadata, missing facts, cache hit/expiry, region crossing, cancellation/timeout, concurrent single-flight capture, policy privacy, payload bounds/markers, and telemetry; - dependency policy 4/4, policy integration 3/3, conversation isolation 1/1, and LLM/tool-loop integration 14/14 green; - cargo clippy --all-targets -D warnings, rustdoc -D warnings, cargo fmt --check, git diff --check, and cargo deny check green; - live-grid API-faithful feature check and clippy -D warnings green. Runner limitations recorded transparently: compiling the repository full generated libremetaverse source is SIGKILLed by this 8GiB runner before rustc reaches metacrate-grid-agent, both with normal debug settings and debug info disabled. The focused API-faithful live boundary build checks the exact manager signatures used here. The Windows target check reaches aws-lc-sys but this runner lacks x86_64-w64-mingw32-gcc; the implementation itself adds no platform-specific API. Post-implementation review against every #124 objective/acceptance bullet found no remaining issue-scope gap.

hugo closed this issue

2026-08-17 23:53:17 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/MetaCrate#124