# Grid-agent LLM runtime

MetaCrate delegates the conversation runtime to Mentra. `llm.endpoint_url` is
the provider base URL, `llm.api_key` is its bearer credential, and `llm.model`
is the model ID. Mentra owns the Responses request path, HTTP/SSE streaming,
message history, tool-call rounds, provider errors, cancellation, compaction,
and persisted runtime state. MetaCrate does not implement a second HTTP client
or parse SSE itself.

For an OpenAI-compatible provider, configure the base that Mentra can extend
with `v1/responses`. For example, a proxy base ending in `/go` is valid when
its Responses endpoint is `/go/v1/responses`; do not include that final path in
MetaCrate configuration. Endpoint behavior, redirects, response transport, and
provider retry semantics are Mentra responsibilities.

## Conversation and memory

The main system prompt is intentionally minimal:

> You are in an OpenSim virtual world. Use the available tools to act there.

It contains no authorization claims. MetaCrate derives authority from the
authenticated sender UUID and channel, then gives Mentra only the permitted
tool profile. Every tool call is checked again by `PolicyGateway`; prompt text
cannot grant authority.

Authorized IM agents have a stable avatar-scoped Mentra identity, so history,
compaction, and memory survive IM session changes and process restarts. Public
and unprivileged conversations remain session-scoped and receive neither
durable memory tools nor privileged grid tools. Authorized agents receive
Mentra's `memory_search`, `memory_pin`, and `memory_forget` tools. Automatic
memory injection is disabled because Mentra 0.18.3 appends recalled memory as a
new user turn after the current request; explicit memory tools preserve durable
learning without displacing the command being handled.

Runtime records use `HybridRuntimeStore`: conversation/runtime state is stored
in `runtime.sqlite`, with the associated Mentra memory store and transcript,
task, team, and workspace paths under the configured state directory.

## Tools and autonomous safety review

Mentra receives the JSON schemas registered by MetaCrate. It validates and
orchestrates model tool calls; `PolicyToolExecutor` is the only bridge to grid
backends. The gateway independently binds authorization to the authenticated
principal, origin, exact canonical arguments, estimated resource cost, expiry,
and single execution.

An action above its approval-free risk threshold does not wait for a human
operator. MetaCrate starts a separate one-shot Mentra agent backed by a fresh
volatile store. That reviewer has no tools, history, or memory and receives
only the proposed tool, exact arguments, and resource estimate as untrusted
data. Only an explicit `ALLOW` verdict grants the existing single-use bound
approval; `DENY`, transport failure, an invalid verdict, timeout, or
cancellation fails closed. Human control-plane decisions remain an emergency
facility, not a requirement for autonomous operation.

## Images

Vision captures are reconstructed viewport images rather than screenshots.
`metacrate-rendering-wgpu` owns a dedicated-thread, headless Bevy/wgpu device,
renders the current scene to an offscreen color/depth target, reads the frame
back, encodes it directly as JPEG, and attaches it as a Mentra image content
block. GPU setup, scene conversion, and readback run outside the simulator event
path. If no compatible adapter is available or a frame fails, the deterministic
software rasterizer handles the same snapshot. The default 640x360 frame and
the entity, triangle, texture, JPEG, time, and concurrency safety ceilings are
validated before runtime. JPEG avoids the much larger PNG payloads.

The live source reconstructs authoritative terrain heights and detail textures,
legacy prims, sculpt maps, uploaded meshes, linksets, legacy Blinn-Phong and
PBR materials, independent UV transforms, water, and privacy-marked avatar
cards. It keeps bounded decoded-texture and geometry caches per region; Bevy
also retains unchanged GPU textures and meshes between frames. Textures are
decoded through the pure-Rust JPEG-2000 path, reduced to viewport-appropriate
resolution, and uploaded with filtered mip chains. The view radius is
configurable as `vision.max_distance_meters` and defaults to 64 meters. World
objects are selected by that distance; avatar attachments are rigged through
their linkset hierarchy and included only when the camera is focused on the
avatar, after world-object capacity is reserved.

Raw mesh, texture, and material assets are treated as immutable by UUID and persist in the
platform MetaCrate cache. `vision.asset_cache_max_bytes` bounds that cache with a 2 GiB default;
successful reads refresh its LRU order and pruning removes the least recently used assets first.
While logged in, the live scene source continuously runs its normal distance-bounded collection,
asset fetch, texture decode, material parse, and mesh reconstruction path in the background. A
snapshot therefore reuses the warm persistent, decoded, geometry, material, and GPU caches instead
of beginning asset discovery when the model asks to look.

`behavior_camera_set` gives an authorized model a bounded position, target, and
vertical field of view; `behavior_camera_reset` restores the avatar-facing
view. Full avatar body/attachment mesh fidelity and baking remain separately
tracked work.

## Bounds and cancellation

MetaCrate sets generous but finite run budgets on Mentra: model rounds, tool
calls, stored history, and total wall-clock time. The default interaction/model
window is two minutes. Grid disconnect, session replacement, expiry, operator
cancel, and shutdown bridge the generation cancellation token into Mentra.
Non-idempotent ambiguous mutations stop immediately and are never retried.

The provider proxy used in live verification could not combine replayed input
with `previous_response_id`, so MetaCrate selects Mentra's full-history
Responses mode for that provider. Mentra still owns the transport and history.

## Verification

The focused suite uses loopback Responses/SSE endpoints and covers streamed
text, images, tool rounds, compaction, persisted memory across restart, and
schema validation:

```sh
cargo test --locked -p metacrate-grid-agent --test llm_transport
cargo clippy --locked -p metacrate-grid-agent --all-targets --all-features -- -D warnings
```