6.2 KiB
Grid-agent LLM runtime
MetaCrate delegates the conversation runtime to Mentra. llm.endpoint_url is
the provider base URL, llm.api_key is its bearer credential, and llm.model
is the model ID. Mentra owns the Responses request path, HTTP/SSE streaming,
message history, tool-call rounds, provider errors, cancellation, compaction,
and persisted runtime state. MetaCrate does not implement a second HTTP client
or parse SSE itself.
For an OpenAI-compatible provider, configure the base that Mentra can extend
with v1/responses. For example, a proxy base ending in /go is valid when
its Responses endpoint is /go/v1/responses; do not include that final path in
MetaCrate configuration. Endpoint behavior, redirects, response transport, and
provider retry semantics are Mentra responsibilities.
Conversation and memory
The main system prompt is intentionally minimal:
You are in an OpenSim virtual world. Use the available tools to act there.
It contains no authorization claims. MetaCrate derives authority from the
authenticated sender UUID and channel, then gives Mentra only the permitted
tool profile. Every tool call is checked again by PolicyGateway; prompt text
cannot grant authority.
Authorized IM agents have a stable avatar-scoped Mentra identity, so history,
compaction, and memory survive IM session changes and process restarts. Public
and unprivileged conversations remain session-scoped and receive neither
durable memory tools nor privileged grid tools. Authorized agents receive
Mentra's memory_search, memory_pin, and memory_forget tools. Automatic
memory injection is disabled because Mentra 0.18.3 appends recalled memory as a
new user turn after the current request; explicit memory tools preserve durable
learning without displacing the command being handled.
Runtime records use HybridRuntimeStore: conversation/runtime state is stored
in runtime.sqlite, with the associated Mentra memory store and transcript,
task, team, and workspace paths under the configured state directory.
Tools and autonomous safety review
Mentra receives the JSON schemas registered by MetaCrate. It validates and
orchestrates model tool calls; PolicyToolExecutor is the only bridge to grid
backends. The gateway independently binds authorization to the authenticated
principal, origin, exact canonical arguments, estimated resource cost, expiry,
and single execution.
An action above its approval-free risk threshold does not wait for a human
operator. MetaCrate starts a separate one-shot Mentra agent backed by a fresh
volatile store. That reviewer has no tools, history, or memory and receives
only the proposed tool, exact arguments, and resource estimate as untrusted
data. Only an explicit ALLOW verdict grants the existing single-use bound
approval; DENY, transport failure, an invalid verdict, timeout, or
cancellation fails closed. Human control-plane decisions remain an emergency
facility, not a requirement for autonomous operation.
Images
Vision captures are reconstructed viewport images rather than screenshots.
metacrate-rendering-wgpu owns a dedicated-thread, headless Bevy/wgpu device,
renders the current scene to an offscreen color/depth target, reads the frame
back, encodes it directly as JPEG, and attaches it as a Mentra image content
block. GPU setup, scene conversion, and readback run outside the simulator event
path. If no compatible adapter is available or a frame fails, the deterministic
software rasterizer handles the same snapshot. The default 640x360 frame and
the entity, triangle, texture, JPEG, time, and concurrency safety ceilings are
validated before runtime. JPEG avoids the much larger PNG payloads.
The live source reconstructs authoritative terrain heights and detail textures,
legacy prims, sculpt maps, uploaded meshes, linksets, legacy Blinn-Phong and
PBR materials, independent UV transforms, water, and privacy-marked avatar
cards. It keeps bounded decoded-texture and geometry caches per region; Bevy
also retains unchanged GPU textures and meshes between frames. Textures are
decoded through the pure-Rust JPEG-2000 path, reduced to viewport-appropriate
resolution, and uploaded with filtered mip chains. The view radius is
configurable as vision.max_distance_meters and defaults to 64 meters. World
objects are selected by that distance; avatar attachments are rigged through
their linkset hierarchy and included only when the camera is focused on the
avatar, after world-object capacity is reserved.
Raw mesh, texture, and material assets are treated as immutable by UUID and persist in the
platform MetaCrate cache. vision.asset_cache_max_bytes bounds that cache with a 2 GiB default;
successful reads refresh its LRU order and pruning removes the least recently used assets first.
While logged in, the live scene source continuously runs its normal distance-bounded collection,
asset fetch, texture decode, material parse, and mesh reconstruction path in the background. A
snapshot therefore reuses the warm persistent, decoded, geometry, material, and GPU caches instead
of beginning asset discovery when the model asks to look.
behavior_camera_set gives an authorized model a bounded position, target, and
vertical field of view; behavior_camera_reset restores the avatar-facing
view. Full avatar body/attachment mesh fidelity and baking remain separately
tracked work.
Bounds and cancellation
MetaCrate sets generous but finite run budgets on Mentra: model rounds, tool calls, stored history, and total wall-clock time. The default interaction/model window is two minutes. Grid disconnect, session replacement, expiry, operator cancel, and shutdown bridge the generation cancellation token into Mentra. Non-idempotent ambiguous mutations stop immediately and are never retried.
The provider proxy used in live verification could not combine replayed input
with previous_response_id, so MetaCrate selects Mentra's full-history
Responses mode for that provider. Mentra still owns the transport and history.
Verification
The focused suite uses loopback Responses/SSE endpoints and covers streamed text, images, tool rounds, compaction, persisted memory across restart, and schema validation:
cargo test --locked -p metacrate-grid-agent --test llm_transport
cargo clippy --locked -p metacrate-grid-agent --all-targets --all-features -- -D warnings