[Grid agent] Render pure-Rust viewport snapshots for multimodal questions #132

New Issue

hugo · 2026-08-17T19:31:38Z

hugo commented

2026-08-17 19:31:38 +00:00

Objective

Create a bounded, deterministic viewport snapshot from the agent's current virtual camera using pure Rust rendering/codec paths, then allow the generic endpoint to answer questions that explicitly reference visual context.

Capture pipeline

Define what the synthetic viewport represents: camera pose/FOV, current simulator generation, tracked visible objects/avatars, terrain/environment, and available textures. Record timestamp and known completeness limits; never imply it is a viewer framebuffer when data is missing.
Reuse the native rendering/image crates with the audited pure-Rust backend. The grid-agent dependency graph for this feature must not enable Google Skia/C++, native codecs, GPU/vendor libraries, subprocesses, or platform-only capture APIs.
Bound resolution, objects, triangles, texture fetch count/bytes, decode dimensions, render time, memory, concurrent captures, and encoded image size. Cancel on disconnect, region change, superseding request, or operator action.
Produce a deterministic orientation and color-managed PNG (or another documented pure-Rust encoding) plus a compact textual scene summary. Mark residents according to privacy policy and do not persist snapshots by default.

LLM integration

Add image content through the generic OpenAI-compatible request schema without provider branches. If the configured endpoint rejects multimodal input, return a typed capability error and offer the textual scene summary; do not silently send repeated large requests.
Only capture for an explicit visual question or authorized/operator request, subject to rate limits. Correlate the image hash/metadata, not raw pixels, in normal audit logs.

Acceptance criteria

Golden scene fixtures verify camera transform, culling, occlusion/depth ordering, object/avatar placement, deterministic pixels/hash, missing textures, and bounded fallback behavior.
Fake endpoint tests inspect multimodal payloads, rejection fallback, cancellation, and size limits.
Dependency audit proves the enabled grid-agent vision path is pure Rust on Linux/Windows/macOS code paths.
Live capture never blocks chat/lifecycle responsiveness and exposes progress/failure through control/TUI.

Dependencies

Depends on world perception, embodied camera state, generic LLM transport, policy, and observability. This is an advanced feature and may land after the text agent is complete.

## Objective Create a bounded, deterministic viewport snapshot from the agent's current virtual camera using pure Rust rendering/codec paths, then allow the generic endpoint to answer questions that explicitly reference visual context. ## Capture pipeline - Define what the synthetic viewport represents: camera pose/FOV, current simulator generation, tracked visible objects/avatars, terrain/environment, and available textures. Record timestamp and known completeness limits; never imply it is a viewer framebuffer when data is missing. - Reuse the native rendering/image crates with the audited pure-Rust backend. The grid-agent dependency graph for this feature must not enable Google Skia/C++, native codecs, GPU/vendor libraries, subprocesses, or platform-only capture APIs. - Bound resolution, objects, triangles, texture fetch count/bytes, decode dimensions, render time, memory, concurrent captures, and encoded image size. Cancel on disconnect, region change, superseding request, or operator action. - Produce a deterministic orientation and color-managed PNG (or another documented pure-Rust encoding) plus a compact textual scene summary. Mark residents according to privacy policy and do not persist snapshots by default. ## LLM integration - Add image content through the generic OpenAI-compatible request schema without provider branches. If the configured endpoint rejects multimodal input, return a typed capability error and offer the textual scene summary; do not silently send repeated large requests. - Only capture for an explicit visual question or authorized/operator request, subject to rate limits. Correlate the image hash/metadata, not raw pixels, in normal audit logs. ## Acceptance criteria - [ ] Golden scene fixtures verify camera transform, culling, occlusion/depth ordering, object/avatar placement, deterministic pixels/hash, missing textures, and bounded fallback behavior. - [ ] Fake endpoint tests inspect multimodal payloads, rejection fallback, cancellation, and size limits. - [ ] Dependency audit proves the enabled grid-agent vision path is pure Rust on Linux/Windows/macOS code paths. - [ ] Live capture never blocks chat/lifecycle responsiveness and exposes progress/failure through control/TUI. ## Dependencies Depends on world perception, embodied camera state, generic LLM transport, policy, and observability. This is an advanced feature and may land after the text agent is complete.

hugo added this to the 14 - metacrate grid agent milestone 2026-08-17 19:31:38 +00:00

hugo added the enhancement label 2026-08-17 19:31:38 +00:00

hugo referenced this issue

2026-08-17 19:32:14 +00:00

implement an AI driven agent that runs in opensim #117

hugo referenced this issue from a commit

2026-08-18 09:57:59 +00:00

Implement pure Rust visual snapshots (#132)

hugo commented

2026-08-18 09:58:18 +00:00

Implemented in 11a8c37. Added a bounded deterministic synthetic viewport pipeline in metacrate-grid-agent: live camera/FOV and simulator-generation capture, pure-Rust faceted mesh rendering and bounded rust-j2k texture decoding, privacy-marked avatars, terrain/environment, z-buffer occlusion, deterministic color-managed sRGB PNG encoding, completeness/timestamp summary, and hash-only observations. Integrated explicit authorized visual questions with the provider-neutral OpenAI-compatible image schema, typed multimodal capability rejection with textual fallback and no large-request retry. Added rate/resource/concurrency/time limits plus cancellation for external requests, supersession, generation/region changes, and operator pause/cancel; control/TUI report progress and hashes without pixels. Namespace policy is enforced: no new libremetaverse crate or metacrate reexport from compatibility facades. Verification: vision golden/adversarial tests (6), control runtime (1), TUI (5), dependency policy (7), full related fake-LLM transport suite (15), live-grid cargo doc, all-target live-grid clippy -D warnings, cargo tree pure-Rust feature audit, and git diff checks all pass.

Implemented in 11a8c37. Added a bounded deterministic synthetic viewport pipeline in metacrate-grid-agent: live camera/FOV and simulator-generation capture, pure-Rust faceted mesh rendering and bounded rust-j2k texture decoding, privacy-marked avatars, terrain/environment, z-buffer occlusion, deterministic color-managed sRGB PNG encoding, completeness/timestamp summary, and hash-only observations. Integrated explicit authorized visual questions with the provider-neutral OpenAI-compatible image schema, typed multimodal capability rejection with textual fallback and no large-request retry. Added rate/resource/concurrency/time limits plus cancellation for external requests, supersession, generation/region changes, and operator pause/cancel; control/TUI report progress and hashes without pixels. Namespace policy is enforced: no new libremetaverse crate or metacrate reexport from compatibility facades. Verification: vision golden/adversarial tests (6), control runtime (1), TUI (5), dependency policy (7), full related fake-LLM transport suite (15), live-grid cargo doc, all-target live-grid clippy -D warnings, cargo tree pure-Rust feature audit, and git diff checks all pass.

hugo closed this issue

2026-08-18 09:58:18 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/MetaCrate#132