5.0 KiB
Grid-agent LLM runtime
MetaCrate delegates the conversation runtime to Mentra. llm.endpoint_url is
the provider base URL, llm.api_key is its bearer credential, and llm.model
is the model ID. Mentra owns the Responses request path, HTTP/SSE streaming,
message history, tool-call rounds, provider errors, cancellation, compaction,
and persisted runtime state. MetaCrate does not implement a second HTTP client
or parse SSE itself.
For an OpenAI-compatible provider, configure the base that Mentra can extend
with v1/responses. For example, a proxy base ending in /go is valid when
its Responses endpoint is /go/v1/responses; do not include that final path in
MetaCrate configuration. Endpoint behavior, redirects, response transport, and
provider retry semantics are Mentra responsibilities.
Conversation and memory
The main system prompt is intentionally minimal:
You are in an OpenSim virtual world. Use the available tools to act there.
It contains no authorization claims. MetaCrate derives authority from the
authenticated sender UUID and channel, then gives Mentra only the permitted
tool profile. Every tool call is checked again by PolicyGateway; prompt text
cannot grant authority.
Authorized IM agents have a stable avatar-scoped Mentra identity, so history,
compaction, and memory survive IM session changes and process restarts. Public
and unprivileged conversations remain session-scoped and receive neither
durable memory tools nor privileged grid tools. Authorized agents receive
Mentra's memory_search, memory_pin, and memory_forget tools. Automatic
memory injection is disabled because Mentra 0.18.3 appends recalled memory as a
new user turn after the current request; explicit memory tools preserve durable
learning without displacing the command being handled.
Runtime records use HybridRuntimeStore: conversation/runtime state is stored
in runtime.sqlite, with the associated Mentra memory store and transcript,
task, team, and workspace paths under the configured state directory.
Tools and autonomous safety review
Mentra receives the JSON schemas registered by MetaCrate. It validates and
orchestrates model tool calls; PolicyToolExecutor is the only bridge to grid
backends. The gateway independently binds authorization to the authenticated
principal, origin, exact canonical arguments, estimated resource cost, expiry,
and single execution.
An action above its approval-free risk threshold does not wait for a human
operator. MetaCrate starts a separate one-shot Mentra agent backed by a fresh
volatile store. That reviewer has no tools, history, or memory and receives
only the proposed tool, exact arguments, and resource estimate as untrusted
data. Only an explicit ALLOW verdict grants the existing single-use bound
approval; DENY, transport failure, an invalid verdict, timeout, or
cancellation fails closed. Human control-plane decisions remain an emergency
facility, not a requirement for autonomous operation.
Images
Vision captures are reconstructed viewport images rather than screenshots.
The live service keeps one headless wgpu device, renders the current scene to
an offscreen color/depth target, reads the frame back, encodes it directly as
JPEG, and attaches it as a Mentra image content block. GPU setup and readback
run outside the simulator event path. If no compatible adapter is available or
a frame fails, the same scene is rendered by the deterministic software
rasterizer. The default 320x180 frame, quality, entity, triangle, texture-fetch,
decoded-pixel, byte, rate, and concurrency limits are validated before runtime.
JPEG avoids the much larger PNG payloads.
The current renderer handles legacy prim geometry, approximate texture colors,
simple avatars, and flat terrain/water. behavior_camera_set gives an
authorized model a bounded position, target, and vertical field of view;
behavior_camera_reset restores the avatar-facing view. Viewer-grade
mesh/sculpt rendering, full UV materials, lighting, authoritative terrain, and
avatar/attachment fidelity remain tracked in the renderer issue.
Bounds and cancellation
MetaCrate sets generous but finite run budgets on Mentra: model rounds, tool calls, stored history, and total wall-clock time. The default interaction/model window is two minutes. Grid disconnect, session replacement, expiry, operator cancel, and shutdown bridge the generation cancellation token into Mentra. Non-idempotent ambiguous mutations stop immediately and are never retried.
The provider proxy used in live verification could not combine replayed input
with previous_response_id, so MetaCrate selects Mentra's full-history
Responses mode for that provider. Mentra still owns the transport and history.
Verification
The focused suite uses loopback Responses/SSE endpoints and covers streamed text, images, tool rounds, compaction, persisted memory across restart, and schema validation:
cargo test --locked -p metacrate-grid-agent --test llm_transport
cargo clippy --locked -p metacrate-grid-agent --all-targets --all-features -- -D warnings