Add headless wgpu vision and camera controls
Some checks failed
CI / rust-skia (Rust only) (push) Has been cancelled
CI / required (push) Has been cancelled

This commit is contained in:
2026-08-22 11:36:34 +02:00
parent 0dd2ca5824
commit c101007362
19 changed files with 1250 additions and 56 deletions

View File

@@ -113,8 +113,9 @@ hash. Do not record vendor presets or identifiers. Exercise, in order:
every created prim through the ownership-checked cleanup path.
6. Create and catalog a current-location landmark, accept a controlled landmark
offer, use a short bounded folder schedule, teleport, then disable the
schedule. Capture the synthetic scene and ask one visual question. Record the endpoint
capability fallback if image input is rejected.
schedule. Set and reset a bounded camera view, capture the reconstructed scene,
and ask one visual question. Record the endpoint capability fallback if image
input is rejected.
7. Gracefully stop. Confirm no pending approvals, scheduled jobs, inventory
offers, owned test prims, tasks, sockets, or sessions. List any unavoidable
inventory artifact and its manual recovery ID in private operator notes,

View File

@@ -119,9 +119,15 @@ avatar session -> persistent Mentra agent -> Responses SSE / compaction / memory
| |
| +-> AuthorizedToolBackend
+-> one-shot Mentra safety reviewer when required
scene snapshot -> cached headless wgpu color/depth render -> JPEG -> Mentra image
`-> deterministic software fallback
```
The package has no build script or direct native dependency. The focused
The package has no build script. Its cross-platform `wgpu` path uses the
platform graphics backend selected by wgpu and requires no window or display;
the software fallback remains available when adapter creation or rendering
fails. The focused
`dependency_policy` test rejects subprocess launch sites, unsafe/native ABI
source, build scripts, and unreviewed direct dependency names in this package.
The precise LLM compatibility and cancellation contract is documented in

View File

@@ -56,16 +56,22 @@ facility, not a requirement for autonomous operation.
## Images
Vision captures are real software-rendered viewport images. MetaCrate renders
the current scene, encodes the final frame directly as JPEG, and attaches it as
a Mentra image content block. The default 320x180 frame, quality, entity,
triangle, texture-fetch, decoded-pixel, byte, rate, and concurrency limits are
validated before runtime. JPEG avoids the much larger PNG payloads.
Vision captures are reconstructed viewport images rather than screenshots.
The live service keeps one headless `wgpu` device, renders the current scene to
an offscreen color/depth target, reads the frame back, encodes it directly as
JPEG, and attaches it as a Mentra image content block. GPU setup and readback
run outside the simulator event path. If no compatible adapter is available or
a frame fails, the same scene is rendered by the deterministic software
rasterizer. The default 320x180 frame, quality, entity, triangle, texture-fetch,
decoded-pixel, byte, rate, and concurrency limits are validated before runtime.
JPEG avoids the much larger PNG payloads.
The current renderer handles legacy prim geometry, approximate texture colors,
simple avatars, and flat terrain/water. Viewer-grade mesh/sculpt rendering,
full UV materials, lighting, authoritative terrain, and model-controlled camera
tools are tracked separately.
simple avatars, and flat terrain/water. `behavior_camera_set` gives an
authorized model a bounded position, target, and vertical field of view;
`behavior_camera_reset` restores the avatar-facing view. Viewer-grade
mesh/sculpt rendering, full UV materials, lighting, authoritative terrain, and
avatar/attachment fidelity remain tracked in the renderer issue.
## Bounds and cancellation

View File

@@ -226,8 +226,12 @@ The example records all current queue, message, conversation, tool, behavior,
reconnect, and interaction defaults. Important defaults include 256 grid events,
32 control commands, 512 observations, four concurrent inference requests,
16 tool calls, 512 active senders/sessions, a two-minute model window, and bounded
10-second shutdown. Vision defaults to a 320x180 software-rendered JPEG with bounded
entities, triangles, texture work, JPEG bytes, time, and concurrency.
10-second shutdown. Vision defaults to a 320x180 headless-wgpu JPEG with a
deterministic software fallback and bounded entities, triangles, texture work,
JPEG bytes, time, and concurrency. Authorized `behavior_camera_set` calls are
limited to 96 meters from the avatar by default, require a distinct target, and
accept a 20-120 degree vertical field of view; `behavior_camera_reset` restores
the avatar-facing 60-degree view.
Unsupported by design: arbitrary raw packets or agent-control flags, arbitrary
shell/subprocess execution, provider/model discovery outside Mentra, remote plaintext
@@ -248,7 +252,8 @@ cargo audit
Record the binary byte size (`stat -c %s` on Linux or `Get-Item ... .Length` on
PowerShell) and the `cargo tree` inventory. Reject CLR/scripting/provider SDK,
subprocess adapters, Skia/GPU, or undeclared native libraries on the agent path.
subprocess adapters, unreviewed graphics stacks, or undeclared native libraries
on the agent path.
The consolidated Gitea gate runs only on `ubuntu-latest`; Windows portability is
proved by the existing cross-target compile/static gate rather than a Windows
Gitea runner.

View File

@@ -23,7 +23,7 @@ The audited tree contains the MetaCrate agent, the original compatibility
crates it consumes, the pure-Rust J2K/meshing/rendering path, Tokio, Reqwest,
Rustls, AWS-LC, terminal/serialization/bounds utilities, and their build-time
Rust tooling. It contains no CLR/Mono/.NET runtime, Python/Node/Lua scripting
runtime, provider SDK, Skia/GPU stack, OpenJPEG feature, shell/subprocess
runtime, provider SDK, unreviewed graphics stack, shell/subprocess
adapter, or dynamically loaded undeclared application library.
Regenerate for a release candidate with the commands in