MetaCrate/docs/grid-agent-operations.md

MetaCrate grid-agent operations

This is the operator runbook for the portable metacrate-grid-agent binary. The binary has no dependency on systemd, a Windows service manager, a CLR, Python, Node.js, a provider SDK, or a shell at runtime. Linux service files and install helpers are packaging conveniences around the same foreground process and versioned control protocol.

Quick start and modes

Build the credential-free binary and validate the example without contacting a grid or LLM:

cargo build --locked --release -p metacrate-grid-agent
target/release/metacrate-grid-agent --config config/grid-agent.example.yml --check-config
target/release/metacrate-grid-agent --config config/grid-agent.example.yml --run-once

--check-config parses YAML and any legacy referenced secret files, validates all bounds and TLS files, then exits before constructing the grid or LLM clients. The example is fake/offline mode (integrated=false, split=false) and all credential-looking values are placeholders.

For live foreground or an embedded TUI, build intentionally:

cargo build --locked --release -p metacrate-grid-agent --features live-grid
target/release/metacrate-grid-agent --config /path/to/config.yml
target/release/metacrate-grid-agent --config /path/to/config.yml --tui

Set integrated=true for the foreground and embedded-TUI commands. Set split=true for a headless service, then run the same binary as the TUI client:

metacrate-grid-agent --config /path/to/config.yml
metacrate-grid-agent --config /path/to/config.yml --tui-client

The TUI client and service must use the same control address and operator token. --tui-client never needs the grid password or LLM key. --run-once performs one supervised readiness/login and clean logout cycle. --print-paths prints the platform defaults without loading configuration. If --config is omitted, the default file is used only when it already exists:

Platform	Configuration	Non-secret state
Linux/Unix	$XDG_CONFIG_HOME/metacrate/config.yml, otherwise $HOME/.config/metacrate/config.yml	$XDG_DATA_HOME/metacrate/grid-agent, otherwise $HOME/.local/share/metacrate/grid-agent
macOS	$HOME/Library/Application Support/MetaCrate/config.yml	$HOME/Library/Application Support/MetaCrate/grid-agent
Windows	%APPDATA%\MetaCrate\config.yml	%LOCALAPPDATA%\MetaCrate/grid-agent

Relative paths in a YAML document resolve relative to that document where specified; service deployments should use absolute paths.

Downloaded grid assets use the platform cache directory: $XDG_CACHE_HOME/metacrate or $HOME/.cache/metacrate on Linux, $HOME/Library/Caches/metacrate on macOS, and %LOCALAPPDATA%\metacrate\cache on Windows. The cache treats UUID-addressed assets as immutable, defaults to a 2 GiB LRU limit, and is configurable with vision.asset_cache_max_bytes. The live scene source continuously prefetches and prepares assets inside the configured view radius while the avatar is logged in; prefetching is stopped before logout and shutdown.

Endpoint, grid identity, and authority

llm.endpoint_url is the OpenAI-compatible provider base URL. Mentra owns the Responses path, SSE stream, endpoint behavior, and conversation runtime. For a proxy whose Responses endpoint is /go/v1/responses, configure the base ending in /go, not the final request path. Set llm.model to the model ID and keep llm.api_key in the private config.yml.

Live modes require grid.login_url, grid.avatar_name, and grid.password. authorized_avatar_uuids contains exact grid UUIDs, never display names. Text claiming an authorized identity grants no authority. Public chat can request bounded informational work and safe public LSL delivery; movement, teleport, building, roaming changes, and administration remain policy-gated and require an authenticated authorized IM or a narrowly bound scheduler grant. Above-threshold actions receive an isolated one-shot LLM safety review; they do not depend on a continuously present human operator.

Configuration contract and migration

The YAML root uses schema_version: 1. Omitting it is accepted as legacy version 1. Any other value fails before startup with a migration message. There is no automatic in-place migration: copy the file, update the copy using the release notes/example, validate the copy, then atomically select it. Unknown fields fail closed. Installers never overwrite or migrate operator files.

Resolution order is:

1. bounded built-in defaults;
2. the platform config.yml or explicit --config document;
3. legacy secret files referenced by that document.

Process environment variables do not override connection, mode, or authority. Migrate an existing .env once with metacrate-grid-agent --import-env /path/to/.env, inspect it using --preferences, then remove the old file under the operator's retention policy. JSON remains readable only for migration.

Mode, endpoints, credentials, authorization UUIDs, TLS, storage, queue/resource limits, reconnect policy, behavior, and interaction settings are restart-only. Runtime control can pause/resume autonomy, toggle the roaming job, make an emergency decision on an existing approval, cancel an active action, expire/delete conversation state, inject an operator message, reconnect, or shut down; it does not silently rewrite the configuration. A future reloadable field must be explicitly added to the versioned control/config contract.

Secrets and privacy

Threat model: grid residents, object/avatar/inventory metadata, capability replies, model output, public chat, IM text, and persisted files are untrusted. Remote peers may inject instructions, spoof names, replay approvals/call IDs, flood queues, delay or truncate replies, and attempt secret exfiltration. The service identity, local config/secret ACLs, authenticated UUID/control role, opaque policy authorization, generation fencing, bounded queues, and audit sink are trust boundaries. Host/root compromise, a malicious binary/dependency, and an operator deliberately approving a harmful action are outside the process sandbox and require OS/supply-chain/operational controls.

Never place API keys, grid passwords, or control tokens in command arguments, unit files, wrapper XML, logs, crash-report commands, issue reports, or TUI screens. Secret wrappers redact Debug and Display, are not serializable, and normal observability stores pseudonymous correlation IDs, result codes, bounds, and hashes—not prompt bodies, credentials, visual pixels, or raw tool arguments.

Secret files must be regular, non-symlink, bounded UTF-8 files containing one line. On Unix, startup rejects group/other permission bits; use mode 0600 or stricter and ownership by the service identity. On Windows, set an NTFS ACL that grants only the service identity and Administrators, for example with icacls; Rust's portable metadata API cannot prove arbitrary Windows ACL semantics, and FAT/network filesystems may not enforce them. Treat an unverifiable filesystem as unsuitable for unattended secrets.

Rotate one credential at a time in --preferences (or an ACL-restricted copy of config.yml), save, run --check-config, then restart. Revoke the old credential only after readiness. Control observer and operator tokens must differ from each other and from grid/LLM credentials.

Linux systemd

The hardened example is ../packaging/metacrate-grid-agent/systemd/metacrate-grid-agent.service. Create the unprivileged metacrate-agent identity, copy (do not overwrite) an operator configuration to /etc/metacrate/config.yml and create /var/lib/metacrate/grid-agent. Keep the configuration owned by the service identity with mode 0600; do not put credentials in the unit environment.

Install the unit, inspect the sandbox, validate, then start:

sudo systemd-analyze security packaging/metacrate-grid-agent/systemd/metacrate-grid-agent.service
sudo -u metacrate-agent /usr/local/bin/metacrate-grid-agent --config /etc/metacrate/config.yml --check-config
sudo systemctl daemon-reload
sudo systemctl enable --now metacrate-grid-agent.service

The unit uses SIGINT, matching foreground Ctrl-C and the owned graceful shutdown path. Do not replace it with SIGKILL. ProtectSystem=strict permits writes only under the state directory.

Windows service operation

Run the same .exe, YAML, control TCP protocol, and shutdown path. Portable foreground operation in PowerShell is the baseline:

& 'C:\Program Files\MetaCrate\metacrate-grid-agent.exe' --config 'C:\ProgramData\MetaCrate\config.yml' --check-config
& 'C:\Program Files\MetaCrate\metacrate-grid-agent.exe' --config 'C:\ProgramData\MetaCrate\config.yml'

For unattended use, configure a maintained Windows service wrapper (for example WinSW) to launch exactly that command as a dedicated low-privilege account and to translate SCM Stop into Ctrl-C/console control before its timeout. Keep config.yml ACL-restricted to the service identity. Configure restart-on-failure, not unconditional rapid restart. Validate as the service identity before registration. The wrapper must not capture environment values or command output into a world-readable log. The supplied PowerShell installer replaces only the executable and never state.

Control, network, and health

Split control defaults to loopback TCP. A non-loopback bind is rejected unless an explicit Rustls certificate/private key is configured. Use host firewalls to allow only operator networks, protect token files, and prefer loopback plus an authenticated tunnel. Observer tokens are read-only. Protocol framing, role permissions, cancellation, event gaps, and limits are in the control-plane doc.

Readiness is stricter than a TCP connection: the runtime view must show transport_connected=true and agent_ready=true. The TUI Overview and observer Runtime request are health/readiness checks; process existence alone is not readiness. Alert on authentication-blocked, sustained degraded/backoff, queue saturation/dropped-event counters, audit failure, orphan builds, and a stale generation.

Retention, backup, upgrade, and rollback

Conversation snapshots, landmark state, and audit journals are non-secret but privacy-sensitive. Stop or pause mutation, take a filesystem-consistent backup of the configured data directory, and encrypt/restrict the backup. Do not back up secrets with ordinary state. Journal rotation is bounded by the configured observability policy; ship rotated files to restricted storage before deletion when retention policy requires it. Replay is diagnostic and never executes an action.

Release installation replaces only the binary. The POSIX and PowerShell helpers stage a temporary executable then move it into place; neither touches config, secrets, conversations, landmarks, or journals. For upgrade: back up state, install the new binary, validate the existing config, stop gracefully, start, and verify readiness. For rollback: stop gracefully, restore the prior binary, restore state only when the new version changed it incompatibly, validate, and start. Never run two service generations against one writable data directory.

Failure playbooks

• Authentication blocked: pause retries, verify login URL/avatar and rotate the password file; never paste it into logs. Force reconnect after correction.
• LLM unavailable/rate limited: autonomy remains bounded; verify the provider base URL, firewall/DNS, key, and model compatibility. Multimodal rejection falls back to the textual scene summary without resending the large image.
• Maintenance/disconnect: allow generation fencing and bounded backoff. Stale inference/mutation results are discarded; do not bypass reconnect controls.
• Emergency: use operator Pause first, Cancel the exact active action when appropriate, then Graceful Shutdown. Ctrl-C/SIGINT follows the same cleanup.
• Orphan build: keep the reported object IDs, inspect ownership in-world, and manually recover only those IDs. Never bulk-delete by name or proximity.
• Corrupt persistence: preserve the quarantined file for restricted diagnosis; the service recovers an older valid generation or starts fail-closed. Do not hand-edit a live journal.
• Full disk/audit backpressure: pause autonomy, free space according to retention policy, and restart only after the audit path is writable. Policy fails closed when required audit records cannot be accepted.

Resource defaults and unsupported operations

The example records all current queue, message, conversation, tool, behavior, reconnect, and interaction defaults. Important defaults include 256 grid events, 32 control commands, 512 observations, four concurrent inference requests, 16 tool calls, 512 active senders/sessions, a two-minute model window, and bounded 10-second shutdown. Vision defaults to a 1920x1080 headless Bevy/wgpu JPEG, a 64-meter configurable view radius, a bounded in-memory decoded-texture cache, a persistent platform asset cache, and a deterministic software fallback with bounded entities, triangles, asset work, JPEG bytes, time, and concurrency. Authorized behavior_camera_set calls are limited to 96 meters from the avatar by default, require a distinct target, and accept a 20-120 degree vertical field of view; behavior_camera_reset restores the avatar-facing 60-degree view.

Unsupported by design: arbitrary raw packets or agent-control flags, arbitrary shell/subprocess execution, provider/model discovery outside Mentra, remote plaintext control, unauthenticated mutation, unrestricted walking/teleport/touch/follow, automatic config migration, persistence of viewport pixels, framebuffer/screen capture, and treating untrusted grid/LLM text as instructions or authority.

Release evidence

Build the intentional graphs and retain the commands/output with the release:

cargo build --locked --release -p metacrate-grid-agent
cargo build --locked --release -p metacrate-grid-agent --features live-grid
cargo tree --locked -p metacrate-grid-agent --features live-grid -e normal,build
cargo audit

Record the binary byte size (stat -c %s on Linux or Get-Item ... .Length on PowerShell) and the cargo tree inventory. Reject CLR/scripting/provider SDK, subprocess adapters, unreviewed graphics stacks, or undeclared native libraries on the agent path. The consolidated Gitea gate runs only on ubuntu-latest; Windows portability is proved by the existing cross-target compile/static gate rather than a Windows Gitea runner.

The milestone baseline is recorded in grid-agent-release-evidence.md.

Related contracts: architecture, policy, session, control, observability, TUI, and conversation storage.