Add headless wgpu vision and camera controls
Some checks failed
CI / rust-skia (Rust only) (push) Has been cancelled
CI / required (push) Has been cancelled

This commit is contained in:
2026-08-22 11:36:34 +02:00
parent 0dd2ca5824
commit c101007362
19 changed files with 1250 additions and 56 deletions

View File

@@ -56,16 +56,22 @@ facility, not a requirement for autonomous operation.
## Images
Vision captures are real software-rendered viewport images. MetaCrate renders
the current scene, encodes the final frame directly as JPEG, and attaches it as
a Mentra image content block. The default 320x180 frame, quality, entity,
triangle, texture-fetch, decoded-pixel, byte, rate, and concurrency limits are
validated before runtime. JPEG avoids the much larger PNG payloads.
Vision captures are reconstructed viewport images rather than screenshots.
The live service keeps one headless `wgpu` device, renders the current scene to
an offscreen color/depth target, reads the frame back, encodes it directly as
JPEG, and attaches it as a Mentra image content block. GPU setup and readback
run outside the simulator event path. If no compatible adapter is available or
a frame fails, the same scene is rendered by the deterministic software
rasterizer. The default 320x180 frame, quality, entity, triangle, texture-fetch,
decoded-pixel, byte, rate, and concurrency limits are validated before runtime.
JPEG avoids the much larger PNG payloads.
The current renderer handles legacy prim geometry, approximate texture colors,
simple avatars, and flat terrain/water. Viewer-grade mesh/sculpt rendering,
full UV materials, lighting, authoritative terrain, and model-controlled camera
tools are tracked separately.
simple avatars, and flat terrain/water. `behavior_camera_set` gives an
authorized model a bounded position, target, and vertical field of view;
`behavior_camera_reset` restores the avatar-facing view. Viewer-grade
mesh/sculpt rendering, full UV materials, lighting, authoritative terrain, and
avatar/attachment fidelity remain tracked in the renderer issue.
## Bounds and cancellation