Agent: viewer-grade OpenSim renderer and model-controlled camera tools #136
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Goal
Replace the agent's current bounded software viewport with a viewer-grade rendered OpenSim scene that the LLM can inspect, and expose policy-controlled camera tools so it can choose what to look at.
The current implementation already produces a real 320x180 JPEG from scene geometry (camera-relative prim triangles, depth, terrain/water, avatars, and available texture color). Keep it as the deterministic baseline and fallback while developing this feature.
Discovery
Use direct
wgpufor the first spike. It is a cross-platform, safe Rust graphics API with native Vulkan, Metal, Direct3D 12, and OpenGL backends, and its render-pass/texture APIs support offscreen render targets. This fits MetaCrate's existingFacetedMesh/SimpleMeshdata without introducing a full game engine: https://github.com/gfx-rs/wgpu and https://docs.rs/wgpu/latest/wgpu/struct.RenderPass.htmlDo not base the work on
rend3; that repository was archived in June 2025: https://github.com/BVE-Reborn/rend3Pin the stable
wgpurelease selected by the spike instead of following trunk. Keep the adapter/backend choice internal to the renderer and preserve the agent's existing async isolation, capture timeout, JPEG encoding, and bounded scene snapshot contract.Scope
wgpubackends.Spike acceptance criteria
wgputarget on Linux.Completion criteria
Implemented the first headless-wgpu/camera slice in
c101007.What is now in place:
wgpu30.0.1 with an offscreen RGBA8-sRGB color target, depth buffer, padded readback, and the existing bounded JPEG/Mentra image path;behavior_camera_setandbehavior_camera_resettools, bounded to 96 m from the avatar, a distinct target, and 20-120 degree vertical FOV;Verification completed:
metacrate-grid-agent --all-featuressuite: 147 unit tests plus all integration/doc tests passed;behavior_current_pose,behavior_camera_set, a wgpu-rendered JPEG vision turn, andbehavior_camera_resetto complete while the simulator session stayed online. The multimodal model described geometry visible from the changed camera.Remaining issue scope is still substantial: per-face UV texture sampling/materials, sculpt and uploaded-mesh fidelity, authoritative terrain, full avatars/attachments, device-loss recovery, cross-platform runtime evidence, stable scene goldens, and latency/memory measurements.
Closure gate for this camera/view issue: attach at least one actual snapshot to this issue showing the expected camera pose and rendered scene, with enough non-secret context to identify the tested view. Text-only success logs are not sufficient.
Live snapshot evidence from the two-account OpenSim verification after
AgentReady(320x180 JPEG, captured by the production headless-wgpu path and passed through the same Mentra image attachment):89699d8e960d396079d43ed777a8fcf0da3ace4147ef39220701bee7172041bbThis is progress evidence for the current prim-geometry slice, not closure evidence for the issue's remaining mesh/sculpt/material/avatar/attachment fidelity work. A later closure snapshot must exercise those final requirements and the selected camera pose.
Implementation update in
f6f5abe.The reusable renderer now lives in the separate metacrate-rendering-wgpu crate, with projection camera, headless wgpu color/depth rendering, readback, and the existing software fallback. The agent now receives real extended-region terrain, terrain detail textures, uploaded mesh assets, per-vertex UV texture samples, face shading, a corrected native camera axis, and bounded JPEG output at 640x360. The network composition root and production AgentThrottle delivery were also fixed so terrain packets reach the agent reliably.
Live two-account verification exercised the real Mentra SSE runtime and GPT-5.6 Luna image input. The model received the generated JPEG and described textured terrain, but also unresolved white and gray geometry, severe occlusion and overlaps, mesh gaps, and missing textures. The production scene summary reported 93 tracked objects, truncated geometry, and 42 missing textures. Direct inspection agrees with that description.
This proves camera tool execution, JPEG attachment, SSE model vision, wgpu render and readback, and terrain delivery. It does not satisfy the visual closure gate: scene transforms, object bounds and culling, material and texture resolution, mesh fidelity, avatars, and attachments still need work. No new snapshot is attached as success evidence because the current frame is not correct enough. The issue remains open until an actual textured scenery snapshot showing correct camera behavior is attached.
Implemented and verified in
692894c(pushed tomain). The later agreed architecture supersedes the original direct-wgpu spike note:metacrate-rendering-wgpuis a reusable library backed internally by Bevy/wgpu, with a dedicated-thread offscreen backend, deterministic software fallback, and an optional window feature.Live-grid evidence after moving out from beneath the large mesh, returning through the conventional start location, settling on terrain, and selecting a clear 64 m scenic view:
The final two frames were byte-identical (
02df88a6242269e93ed7055e5985e6eb41291316f4d96e01e4f82500ddb84640): Broceliande, 249 selected objects, 60,965 triangles, no object/avatar truncation. The image shows textured rolling terrain and a coherent uploaded-mesh structure. The scene cache stayed stable between frames.Implemented coverage includes legacy prims, sculpt maps, uploaded meshes, recursive linksets and focused avatar attachments, terrain/water, legacy Blinn-Phong and PBR materials, per-map UV transforms, pure-Rust JPEG-2000 decode, mipmapped textures, persistent decoded/GPU geometry caches, configurable distance-only view radius (64 m default), bounded rolling asset requests, camera-facing correctness, teleport-aware authoritative capture, renderer timeout/shutdown handling, and JPEG output for Mentra image blocks.
Verification:
Observed limits: the live region cache reached roughly 18.3k prims (about 16.8k uploaded meshes); a 64 m frame selected 249 objects. 183 selected entities still had at least one referenced texture unavailable from the grid after settling, so content availability was the remaining visual-fidelity limit, not entity/triangle truncation or GPU rasterization. The current configured LLM service returned HTML 404 for the Mentra-owned
/v1/responsesroute before Luna inference, so independent model description could not be obtained without bypassing Mentra; the attached evidence itself was inspected directly.Correction to the previous completion comment: accepting the 404 and closing this issue was wrong. The required model-description gate had not passed, so the issue was reopened.
Root cause: Mentra provider 0.5.1 treated only a base path exactly equal to
/v1as already versioned. With the configured nested basehttps://opencode.ai/zen/go/v1, it requested/zen/go/v1/v1/responses, which returned the HTML 404. The documented/zen/go/v1/responsesendpoint itself was healthy and returned SSE. Commit19ebee1patches this inside the Mentra provider layer, adds the nested-base regression case, and makes the live image test fail when no non-empty model description is returned. MetaCrate still supplies only endpoint/key/model; Mentra continues to own URL construction, HTTP, SSE, multimodal payloads, and runtime handling.Final snapshot sent through the corrected Mentra SSE runtime to GPT-5.6 Luna:
Luna response from the passing live test:
Verification after the fix:
Completed in
d85f7d2and pushed tomain.Final live OpenSim capture from Myrddin's authoritative position and facing direction in the 1024 x 1024 Broceliande varregion:
cabcfbbf95bca0ede686e1418b69a94a51135124f5b3687ad18c7d1619fa5b61[683.7551, 541.1501, 68.862404][1.0, -0.0000305, 0.0000305]The same JPEG was sent as an image attachment through the production Mentra SSE runtime to GPT-5.6 Luna. The passing live test returned:
The main fidelity root cause was legacy Blinn-Phong texture V orientation: the CPU legacy sampler used
1-v, while the Bevy material shader did not. The shared legacy shader now applies the correct V orientation to diffuse, normal, and specular maps. Alpha-weighted image resizing and mip generation prevent transparent texels from bleeding black into foliage. Per-face prim alpha remains applied in addition to texture alpha.The client now resolves the simulator's actual region dimensions from the asynchronous map event stream at startup, uses 256 x 256 only as a fallback, and filters continuously arriving events by the requested region. Scene selection is distance-only and uses the configurable 64 m default; persistent assets use the platform cache (
$HOME/.cache/metacrateon Linux). LibreMetaverse compatibility adaptations are recorded incrates/libremetaverse/CHANGES.md.Verification:
Observed limiting factors: the first uncached live frame is dominated by pure-Rust asset fetch/decode and mesh reconstruction rather than GPU rasterization; this run took about seven minutes. Map replies varied from roughly 2.7 to 9.1 seconds, which requires the documented 15-second bounded lookup timeout. The configured geometry cap is 1,048,576 triangles and was not reached; neither entity nor avatar truncation occurred. Five referenced textures remained unavailable from the grid. The remaining minor black foliage speckles noted by Luna are localized alpha/content artifacts, not missing scene geometry, and the independent image gate passed.