119 lines
6.5 KiB
Markdown
119 lines
6.5 KiB
Markdown
# Grid-agent LLM runtime
|
|
|
|
MetaCrate delegates the conversation runtime to Mentra. `llm.endpoint_url` is
|
|
the provider base URL, `llm.api_key` is its bearer credential, and `llm.model`
|
|
is the model ID. Mentra owns the Responses request path, HTTP/SSE streaming,
|
|
message history, tool-call rounds, provider errors, cancellation, compaction,
|
|
and persisted runtime state. MetaCrate does not implement a second HTTP client
|
|
or parse SSE itself.
|
|
|
|
For an OpenAI-compatible provider, configure the base that Mentra can extend
|
|
with `v1/responses`. For example, a proxy base ending in `/go` is valid when
|
|
its Responses endpoint is `/go/v1/responses`; do not include that final path in
|
|
MetaCrate configuration. Endpoint behavior, redirects, response transport, and
|
|
provider retry semantics are Mentra responsibilities.
|
|
|
|
## Conversation and memory
|
|
|
|
The main system prompt is intentionally minimal:
|
|
|
|
> You are in an OpenSim virtual world. Use the available tools to act there.
|
|
|
|
It contains no authorization claims. MetaCrate derives authority from the
|
|
authenticated sender UUID and channel, then gives Mentra only the permitted
|
|
tool profile. Every tool call is checked again by `PolicyGateway`; prompt text
|
|
cannot grant authority.
|
|
|
|
Authorized IM agents have a stable avatar-scoped Mentra identity, so history,
|
|
compaction, and memory survive IM session changes and process restarts. Public
|
|
and unprivileged conversations remain session-scoped and receive neither
|
|
durable memory tools nor privileged grid tools. Authorized agents receive
|
|
Mentra's `memory_search`, `memory_pin`, and `memory_forget` tools. Automatic
|
|
memory injection is disabled because Mentra 0.18.3 appends recalled memory as a
|
|
new user turn after the current request; explicit memory tools preserve durable
|
|
learning without displacing the command being handled.
|
|
|
|
Runtime records use `HybridRuntimeStore`: conversation/runtime state is stored
|
|
in `runtime.sqlite`, with the associated Mentra memory store and transcript,
|
|
task, team, and workspace paths under the configured state directory.
|
|
`memory-agents.json` atomically persists the authenticated avatar, logical
|
|
agent key, and actual Mentra agent ID for operator browsing. This explicit
|
|
mapping is restored independently of generated agent names.
|
|
|
|
## Tools and autonomous safety review
|
|
|
|
Mentra receives the JSON schemas registered by MetaCrate. It validates and
|
|
orchestrates model tool calls; `PolicyToolExecutor` is the only bridge to grid
|
|
backends. The gateway independently binds authorization to the authenticated
|
|
principal, origin, exact canonical arguments, estimated resource cost, expiry,
|
|
and single execution.
|
|
|
|
An action above its approval-free risk threshold does not wait for a human
|
|
operator. MetaCrate starts a separate one-shot Mentra agent backed by a fresh
|
|
volatile store. That reviewer has no tools, history, or memory and receives
|
|
only the proposed tool, exact arguments, and resource estimate as untrusted
|
|
data. Only an explicit `ALLOW` verdict grants the existing single-use bound
|
|
approval; `DENY`, transport failure, an invalid verdict, timeout, or
|
|
cancellation fails closed. Human control-plane decisions remain an emergency
|
|
facility, not a requirement for autonomous operation.
|
|
|
|
## Images
|
|
|
|
Vision captures are reconstructed viewport images rather than screenshots.
|
|
`metacrate-rendering-wgpu` owns a dedicated-thread, headless Bevy/wgpu device,
|
|
renders the current scene to an offscreen color/depth target, reads the frame
|
|
back, encodes it directly as JPEG, and attaches it as a Mentra image content
|
|
block. GPU setup, scene conversion, and readback run outside the simulator event
|
|
path. If no compatible adapter is available or a frame fails, the deterministic
|
|
software rasterizer handles the same snapshot. The default 640x360 frame and
|
|
the entity, triangle, texture, JPEG, time, and concurrency safety ceilings are
|
|
validated before runtime. JPEG avoids the much larger PNG payloads.
|
|
|
|
The live source reconstructs authoritative terrain heights and detail textures,
|
|
legacy prims, sculpt maps, uploaded meshes, linksets, legacy Blinn-Phong and
|
|
PBR materials, independent UV transforms, water, and privacy-marked avatar
|
|
cards. It keeps bounded decoded-texture and geometry caches per region; Bevy
|
|
also retains unchanged GPU textures and meshes between frames. Textures are
|
|
decoded through the pure-Rust JPEG-2000 path, reduced to viewport-appropriate
|
|
resolution, and uploaded with filtered mip chains. The view radius is
|
|
configurable as `vision.max_distance_meters` and defaults to 64 meters. World
|
|
objects are selected by that distance; avatar attachments are rigged through
|
|
their linkset hierarchy and included only when the camera is focused on the
|
|
avatar, after world-object capacity is reserved.
|
|
|
|
Raw mesh, texture, and material assets are treated as immutable by UUID and persist in the
|
|
platform MetaCrate cache. `vision.asset_cache_max_bytes` bounds that cache with a 2 GiB default;
|
|
successful reads refresh its LRU order and pruning removes the least recently used assets first.
|
|
While logged in, the live scene source continuously runs its normal distance-bounded collection,
|
|
asset fetch, texture decode, material parse, and mesh reconstruction path in the background. A
|
|
snapshot therefore reuses the warm persistent, decoded, geometry, material, and GPU caches instead
|
|
of beginning asset discovery when the model asks to look.
|
|
|
|
`behavior_camera_set` gives an authorized model a bounded position, target, and
|
|
vertical field of view; `behavior_camera_reset` restores the avatar-facing
|
|
view. Full avatar body/attachment mesh fidelity and baking remain separately
|
|
tracked work.
|
|
|
|
## Bounds and cancellation
|
|
|
|
MetaCrate sets generous but finite run budgets on Mentra: model rounds, tool
|
|
calls, stored history, and total wall-clock time. The default interaction/model
|
|
window is two minutes. Grid disconnect, session replacement, expiry, operator
|
|
cancel, and shutdown bridge the generation cancellation token into Mentra.
|
|
Non-idempotent ambiguous mutations stop immediately and are never retried.
|
|
|
|
The provider proxy used in live verification could not combine replayed input
|
|
with `previous_response_id`, so MetaCrate selects Mentra's full-history
|
|
Responses mode for that provider. Mentra still owns the transport and history.
|
|
|
|
## Verification
|
|
|
|
The focused suite uses loopback Responses/SSE endpoints and covers streamed
|
|
text, images, tool rounds, compaction, persisted memory across restart, and
|
|
schema validation:
|
|
|
|
```sh
|
|
cargo test --locked -p metacrate-grid-agent --test llm_transport
|
|
cargo clippy --locked -p metacrate-grid-agent --all-targets --all-features -- -D warnings
|
|
```
|