Add headless wgpu vision and camera controls
This commit is contained in:
@@ -113,8 +113,9 @@ hash. Do not record vendor presets or identifiers. Exercise, in order:
|
||||
every created prim through the ownership-checked cleanup path.
|
||||
6. Create and catalog a current-location landmark, accept a controlled landmark
|
||||
offer, use a short bounded folder schedule, teleport, then disable the
|
||||
schedule. Capture the synthetic scene and ask one visual question. Record the endpoint
|
||||
capability fallback if image input is rejected.
|
||||
schedule. Set and reset a bounded camera view, capture the reconstructed scene,
|
||||
and ask one visual question. Record the endpoint capability fallback if image
|
||||
input is rejected.
|
||||
7. Gracefully stop. Confirm no pending approvals, scheduled jobs, inventory
|
||||
offers, owned test prims, tasks, sockets, or sessions. List any unavoidable
|
||||
inventory artifact and its manual recovery ID in private operator notes,
|
||||
|
||||
@@ -119,9 +119,15 @@ avatar session -> persistent Mentra agent -> Responses SSE / compaction / memory
|
||||
| |
|
||||
| +-> AuthorizedToolBackend
|
||||
+-> one-shot Mentra safety reviewer when required
|
||||
|
||||
scene snapshot -> cached headless wgpu color/depth render -> JPEG -> Mentra image
|
||||
`-> deterministic software fallback
|
||||
```
|
||||
|
||||
The package has no build script or direct native dependency. The focused
|
||||
The package has no build script. Its cross-platform `wgpu` path uses the
|
||||
platform graphics backend selected by wgpu and requires no window or display;
|
||||
the software fallback remains available when adapter creation or rendering
|
||||
fails. The focused
|
||||
`dependency_policy` test rejects subprocess launch sites, unsafe/native ABI
|
||||
source, build scripts, and unreviewed direct dependency names in this package.
|
||||
The precise LLM compatibility and cancellation contract is documented in
|
||||
|
||||
@@ -56,16 +56,22 @@ facility, not a requirement for autonomous operation.
|
||||
|
||||
## Images
|
||||
|
||||
Vision captures are real software-rendered viewport images. MetaCrate renders
|
||||
the current scene, encodes the final frame directly as JPEG, and attaches it as
|
||||
a Mentra image content block. The default 320x180 frame, quality, entity,
|
||||
triangle, texture-fetch, decoded-pixel, byte, rate, and concurrency limits are
|
||||
validated before runtime. JPEG avoids the much larger PNG payloads.
|
||||
Vision captures are reconstructed viewport images rather than screenshots.
|
||||
The live service keeps one headless `wgpu` device, renders the current scene to
|
||||
an offscreen color/depth target, reads the frame back, encodes it directly as
|
||||
JPEG, and attaches it as a Mentra image content block. GPU setup and readback
|
||||
run outside the simulator event path. If no compatible adapter is available or
|
||||
a frame fails, the same scene is rendered by the deterministic software
|
||||
rasterizer. The default 320x180 frame, quality, entity, triangle, texture-fetch,
|
||||
decoded-pixel, byte, rate, and concurrency limits are validated before runtime.
|
||||
JPEG avoids the much larger PNG payloads.
|
||||
|
||||
The current renderer handles legacy prim geometry, approximate texture colors,
|
||||
simple avatars, and flat terrain/water. Viewer-grade mesh/sculpt rendering,
|
||||
full UV materials, lighting, authoritative terrain, and model-controlled camera
|
||||
tools are tracked separately.
|
||||
simple avatars, and flat terrain/water. `behavior_camera_set` gives an
|
||||
authorized model a bounded position, target, and vertical field of view;
|
||||
`behavior_camera_reset` restores the avatar-facing view. Viewer-grade
|
||||
mesh/sculpt rendering, full UV materials, lighting, authoritative terrain, and
|
||||
avatar/attachment fidelity remain tracked in the renderer issue.
|
||||
|
||||
## Bounds and cancellation
|
||||
|
||||
|
||||
@@ -226,8 +226,12 @@ The example records all current queue, message, conversation, tool, behavior,
|
||||
reconnect, and interaction defaults. Important defaults include 256 grid events,
|
||||
32 control commands, 512 observations, four concurrent inference requests,
|
||||
16 tool calls, 512 active senders/sessions, a two-minute model window, and bounded
|
||||
10-second shutdown. Vision defaults to a 320x180 software-rendered JPEG with bounded
|
||||
entities, triangles, texture work, JPEG bytes, time, and concurrency.
|
||||
10-second shutdown. Vision defaults to a 320x180 headless-wgpu JPEG with a
|
||||
deterministic software fallback and bounded entities, triangles, texture work,
|
||||
JPEG bytes, time, and concurrency. Authorized `behavior_camera_set` calls are
|
||||
limited to 96 meters from the avatar by default, require a distinct target, and
|
||||
accept a 20-120 degree vertical field of view; `behavior_camera_reset` restores
|
||||
the avatar-facing 60-degree view.
|
||||
|
||||
Unsupported by design: arbitrary raw packets or agent-control flags, arbitrary
|
||||
shell/subprocess execution, provider/model discovery outside Mentra, remote plaintext
|
||||
@@ -248,7 +252,8 @@ cargo audit
|
||||
|
||||
Record the binary byte size (`stat -c %s` on Linux or `Get-Item ... .Length` on
|
||||
PowerShell) and the `cargo tree` inventory. Reject CLR/scripting/provider SDK,
|
||||
subprocess adapters, Skia/GPU, or undeclared native libraries on the agent path.
|
||||
subprocess adapters, unreviewed graphics stacks, or undeclared native libraries
|
||||
on the agent path.
|
||||
The consolidated Gitea gate runs only on `ubuntu-latest`; Windows portability is
|
||||
proved by the existing cross-target compile/static gate rather than a Windows
|
||||
Gitea runner.
|
||||
|
||||
@@ -23,7 +23,7 @@ The audited tree contains the MetaCrate agent, the original compatibility
|
||||
crates it consumes, the pure-Rust J2K/meshing/rendering path, Tokio, Reqwest,
|
||||
Rustls, AWS-LC, terminal/serialization/bounds utilities, and their build-time
|
||||
Rust tooling. It contains no CLR/Mono/.NET runtime, Python/Node/Lua scripting
|
||||
runtime, provider SDK, Skia/GPU stack, OpenJPEG feature, shell/subprocess
|
||||
runtime, provider SDK, unreviewed graphics stack, shell/subprocess
|
||||
adapter, or dynamically loaded undeclared application library.
|
||||
|
||||
Regenerate for a release candidate with the commands in
|
||||
|
||||
Reference in New Issue
Block a user