278 lines
15 KiB
Markdown
278 lines
15 KiB
Markdown
# MetaCrate grid-agent operations
|
|
|
|
This is the operator runbook for the portable `metacrate-grid-agent` binary.
|
|
The binary has no dependency on systemd, a Windows service manager, a CLR,
|
|
Python, Node.js, a provider SDK, or a shell at runtime. Linux service files and
|
|
install helpers are packaging conveniences around the same foreground process
|
|
and versioned control protocol.
|
|
|
|
## Quick start and modes
|
|
|
|
Build the credential-free binary and validate the example without contacting a
|
|
grid or LLM:
|
|
|
|
```sh
|
|
cargo build --locked --release -p metacrate-grid-agent
|
|
target/release/metacrate-grid-agent --config config/grid-agent.example.yml --check-config
|
|
target/release/metacrate-grid-agent --config config/grid-agent.example.yml --run-once
|
|
```
|
|
|
|
`--check-config` parses YAML and any legacy referenced secret files, validates all
|
|
bounds and TLS files, then exits before constructing the grid or LLM clients.
|
|
The example is fake/offline mode (`integrated=false`, `split=false`) and all
|
|
credential-looking values are placeholders.
|
|
|
|
For live foreground or an embedded TUI, build intentionally:
|
|
|
|
```sh
|
|
cargo build --locked --release -p metacrate-grid-agent --features live-grid
|
|
target/release/metacrate-grid-agent --config /path/to/config.yml
|
|
target/release/metacrate-grid-agent --config /path/to/config.yml --tui
|
|
```
|
|
|
|
Set `integrated=true` for the foreground and embedded-TUI commands. Set
|
|
`split=true` for a headless service, then run the same binary as the TUI client:
|
|
|
|
```sh
|
|
metacrate-grid-agent --config /path/to/config.yml
|
|
metacrate-grid-agent --config /path/to/config.yml --tui-client
|
|
```
|
|
|
|
The TUI client and service must use the same control address and operator token.
|
|
`--tui-client` never needs the grid password or LLM key. `--run-once` performs
|
|
one supervised readiness/login and clean logout cycle. `--print-paths` prints
|
|
the platform defaults without loading configuration. If `--config` is omitted,
|
|
the default file is used only when it already exists:
|
|
|
|
| Platform | Configuration | Non-secret state |
|
|
|---|---|---|
|
|
| Linux/Unix | `$XDG_CONFIG_HOME/metacrate/config.yml`, otherwise `$HOME/.config/metacrate/config.yml` | `$XDG_DATA_HOME/metacrate/grid-agent`, otherwise `$HOME/.local/share/metacrate/grid-agent` |
|
|
| macOS | `$HOME/Library/Application Support/MetaCrate/config.yml` | `$HOME/Library/Application Support/MetaCrate/grid-agent` |
|
|
| Windows | `%APPDATA%\MetaCrate\config.yml` | `%LOCALAPPDATA%\MetaCrate/grid-agent` |
|
|
|
|
Relative paths in a YAML document resolve relative to that document where specified; service
|
|
deployments should use absolute paths.
|
|
|
|
Downloaded grid assets use the platform cache directory: `$XDG_CACHE_HOME/metacrate`
|
|
or `$HOME/.cache/metacrate` on Linux, `$HOME/Library/Caches/metacrate` on macOS,
|
|
and `%LOCALAPPDATA%\metacrate\cache` on Windows. The cache treats UUID-addressed assets as
|
|
immutable, defaults to a 2 GiB LRU limit, and is configurable with
|
|
`vision.asset_cache_max_bytes`. The live scene source continuously prefetches and prepares assets
|
|
inside the configured view radius while the avatar is logged in; prefetching is stopped before
|
|
logout and shutdown.
|
|
|
|
## Endpoint, grid identity, and authority
|
|
|
|
`llm.endpoint_url` is the OpenAI-compatible provider base URL. Mentra owns the
|
|
Responses path, SSE stream, endpoint behavior, and conversation runtime. For a
|
|
proxy whose Responses endpoint is `/go/v1/responses`, configure the base ending
|
|
in `/go`, not the final request path. Set `llm.model` to the model ID and keep
|
|
`llm.api_key` in the private `config.yml`.
|
|
|
|
Live modes require `grid.login_url`, `grid.avatar_name`, and `grid.password`.
|
|
`authorized_avatar_uuids` contains exact
|
|
grid UUIDs, never display names. Text claiming an authorized identity grants no
|
|
authority. Public chat can request bounded informational work and safe public
|
|
LSL delivery; movement, teleport, building, roaming changes, and administration
|
|
remain policy-gated and require an authenticated authorized IM or a narrowly
|
|
bound scheduler grant. Above-threshold actions receive an isolated one-shot LLM
|
|
safety review; they do not depend on a continuously present human operator.
|
|
|
|
## Configuration contract and migration
|
|
|
|
The YAML root uses `schema_version: 1`. Omitting it is accepted as legacy
|
|
version 1. Any other value fails before startup with a migration message. There
|
|
is no automatic in-place migration: copy the file, update the copy using the
|
|
release notes/example, validate the copy, then atomically select it. Unknown
|
|
fields fail closed. Installers never overwrite or migrate operator files.
|
|
|
|
Resolution order is:
|
|
|
|
1. bounded built-in defaults;
|
|
2. the platform `config.yml` or explicit `--config` document;
|
|
3. legacy secret files referenced by that document.
|
|
|
|
Process environment variables do not override connection, mode, or authority.
|
|
Migrate an existing `.env` once with `metacrate-grid-agent --import-env
|
|
/path/to/.env`, inspect it using `--preferences`, then remove the old file under
|
|
the operator's retention policy. JSON remains readable only for migration.
|
|
|
|
Mode, endpoints, credentials, authorization UUIDs, TLS, storage, queue/resource
|
|
limits, reconnect policy, behavior, and interaction settings are restart-only.
|
|
Runtime control can pause/resume autonomy, toggle the roaming job, make an
|
|
emergency decision on an existing approval, cancel an active action, expire/delete conversation state,
|
|
inject an operator message, reconnect, or shut down; it does not silently
|
|
rewrite the configuration. A future reloadable field must be explicitly added
|
|
to the versioned control/config contract.
|
|
|
|
## Secrets and privacy
|
|
|
|
Threat model: grid residents, object/avatar/inventory metadata, capability
|
|
replies, model output, public chat, IM text, and persisted files are untrusted.
|
|
Remote peers may inject instructions, spoof names, replay approvals/call IDs,
|
|
flood queues, delay or truncate replies, and attempt secret exfiltration. The
|
|
service identity, local config/secret ACLs, authenticated UUID/control role,
|
|
opaque policy authorization, generation fencing, bounded queues, and audit sink
|
|
are trust boundaries. Host/root compromise, a malicious binary/dependency, and
|
|
an operator deliberately approving a harmful action are outside the process
|
|
sandbox and require OS/supply-chain/operational controls.
|
|
|
|
Never place API keys, grid passwords, or control tokens in command arguments,
|
|
unit files, wrapper XML, logs, crash-report commands, issue reports, or TUI
|
|
screens. Secret wrappers redact `Debug` and `Display`, are not serializable, and
|
|
normal observability stores pseudonymous correlation IDs, result codes, bounds,
|
|
and hashes—not prompt bodies, credentials, visual pixels, or raw tool arguments.
|
|
|
|
Secret files must be regular, non-symlink, bounded UTF-8 files containing one
|
|
line. On Unix, startup rejects group/other permission bits; use mode `0600` or
|
|
stricter and ownership by the service identity. On Windows, set an NTFS ACL that
|
|
grants only the service identity and Administrators, for example with `icacls`;
|
|
Rust's portable metadata API cannot prove arbitrary Windows ACL semantics, and
|
|
FAT/network filesystems may not enforce them. Treat an unverifiable filesystem
|
|
as unsuitable for unattended secrets.
|
|
|
|
Rotate one credential at a time in `--preferences` (or an ACL-restricted copy
|
|
of `config.yml`), save, run `--check-config`, then restart. Revoke the old credential only after readiness. Control observer
|
|
and operator tokens must differ from each other and from grid/LLM credentials.
|
|
|
|
## Linux systemd
|
|
|
|
The hardened example is
|
|
[`../packaging/metacrate-grid-agent/systemd/metacrate-grid-agent.service`](../packaging/metacrate-grid-agent/systemd/metacrate-grid-agent.service).
|
|
Create the unprivileged `metacrate-agent` identity, copy (do not overwrite) an
|
|
operator configuration to `/etc/metacrate/config.yml` and create
|
|
`/var/lib/metacrate/grid-agent`. Keep the configuration owned by the service
|
|
identity with mode `0600`; do not put credentials in the unit environment.
|
|
|
|
Install the unit, inspect the sandbox, validate, then start:
|
|
|
|
```sh
|
|
sudo systemd-analyze security packaging/metacrate-grid-agent/systemd/metacrate-grid-agent.service
|
|
sudo -u metacrate-agent /usr/local/bin/metacrate-grid-agent --config /etc/metacrate/config.yml --check-config
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now metacrate-grid-agent.service
|
|
```
|
|
|
|
The unit uses `SIGINT`, matching foreground Ctrl-C and the owned graceful
|
|
shutdown path. Do not replace it with `SIGKILL`. `ProtectSystem=strict` permits
|
|
writes only under the state directory.
|
|
|
|
## Windows service operation
|
|
|
|
Run the same `.exe`, YAML, control TCP protocol, and shutdown path. Portable
|
|
foreground operation in PowerShell is the baseline:
|
|
|
|
```powershell
|
|
& 'C:\Program Files\MetaCrate\metacrate-grid-agent.exe' --config 'C:\ProgramData\MetaCrate\config.yml' --check-config
|
|
& 'C:\Program Files\MetaCrate\metacrate-grid-agent.exe' --config 'C:\ProgramData\MetaCrate\config.yml'
|
|
```
|
|
|
|
For unattended use, configure a maintained Windows service wrapper (for
|
|
example WinSW) to launch exactly that command as a dedicated low-privilege
|
|
account and to translate SCM Stop into Ctrl-C/console control before its timeout.
|
|
Keep `config.yml` ACL-restricted to the service identity. Configure restart-on-failure, not unconditional rapid
|
|
restart. Validate as the service identity before registration. The wrapper must
|
|
not capture environment values or command output into a world-readable log.
|
|
The supplied PowerShell installer replaces only the executable and never state.
|
|
|
|
## Control, network, and health
|
|
|
|
Split control defaults to loopback TCP. A non-loopback bind is rejected unless
|
|
an explicit Rustls certificate/private key is configured. Use host firewalls to
|
|
allow only operator networks, protect token files, and prefer loopback plus an
|
|
authenticated tunnel. Observer tokens are read-only. Protocol framing, role
|
|
permissions, cancellation, event gaps, and limits are in the control-plane doc.
|
|
|
|
Readiness is stricter than a TCP connection: the runtime view must show
|
|
`transport_connected=true` and `agent_ready=true`. The TUI Overview and
|
|
observer Runtime request are health/readiness checks; process existence alone
|
|
is not readiness. Alert on authentication-blocked, sustained degraded/backoff,
|
|
queue saturation/dropped-event counters, audit failure, orphan builds, and a
|
|
stale generation.
|
|
|
|
## Retention, backup, upgrade, and rollback
|
|
|
|
Conversation snapshots, landmark state, and audit journals are non-secret but
|
|
privacy-sensitive. Stop or pause mutation, take a filesystem-consistent backup
|
|
of the configured data directory, and encrypt/restrict the backup. Do not back
|
|
up secrets with ordinary state. Journal rotation is bounded by the configured
|
|
observability policy; ship rotated files to restricted storage before deletion
|
|
when retention policy requires it. Replay is diagnostic and never executes an
|
|
action.
|
|
|
|
Release installation replaces only the binary. The POSIX and PowerShell helpers
|
|
stage a temporary executable then move it into place; neither touches config,
|
|
secrets, conversations, landmarks, or journals. For upgrade: back up state,
|
|
install the new binary, validate the existing config, stop gracefully, start,
|
|
and verify readiness. For rollback: stop gracefully, restore the prior binary,
|
|
restore state only when the new version changed it incompatibly, validate, and
|
|
start. Never run two service generations against one writable data directory.
|
|
|
|
## Failure playbooks
|
|
|
|
- Authentication blocked: pause retries, verify login URL/avatar and rotate the
|
|
password file; never paste it into logs. Force reconnect after correction.
|
|
- LLM unavailable/rate limited: autonomy remains bounded; verify the provider
|
|
base URL, firewall/DNS, key, and model compatibility. Multimodal rejection falls back to the textual scene
|
|
summary without resending the large image.
|
|
- Maintenance/disconnect: allow generation fencing and bounded backoff. Stale
|
|
inference/mutation results are discarded; do not bypass reconnect controls.
|
|
- Emergency: use operator Pause first, Cancel the exact active action when
|
|
appropriate, then Graceful Shutdown. Ctrl-C/SIGINT follows the same cleanup.
|
|
- Orphan build: keep the reported object IDs, inspect ownership in-world, and
|
|
manually recover only those IDs. Never bulk-delete by name or proximity.
|
|
- Corrupt persistence: preserve the quarantined file for restricted diagnosis;
|
|
the service recovers an older valid generation or starts fail-closed. Do not
|
|
hand-edit a live journal.
|
|
- Full disk/audit backpressure: pause autonomy, free space according to retention
|
|
policy, and restart only after the audit path is writable. Policy fails closed
|
|
when required audit records cannot be accepted.
|
|
|
|
## Resource defaults and unsupported operations
|
|
|
|
The example records all current queue, message, conversation, tool, behavior,
|
|
reconnect, and interaction defaults. Important defaults include 256 grid events,
|
|
32 control commands, 512 observations, four concurrent inference requests,
|
|
16 tool calls, 512 active senders/sessions, a two-minute model window, and bounded
|
|
10-second shutdown. Vision defaults to a 1920x1080 headless Bevy/wgpu JPEG, a
|
|
64-meter configurable view radius, a bounded in-memory decoded-texture cache,
|
|
a persistent platform asset cache, and a deterministic software fallback with
|
|
bounded entities, triangles, asset work, JPEG bytes, time, and concurrency.
|
|
Authorized `behavior_camera_set` calls are
|
|
limited to 96 meters from the avatar by default, require a distinct target, and
|
|
accept a 20-120 degree vertical field of view; `behavior_camera_reset` restores
|
|
the avatar-facing 60-degree view.
|
|
|
|
Unsupported by design: arbitrary raw packets or agent-control flags, arbitrary
|
|
shell/subprocess execution, provider/model discovery outside Mentra, remote plaintext
|
|
control, unauthenticated mutation, unrestricted walking/teleport/touch/follow,
|
|
automatic config migration, persistence of viewport pixels, framebuffer/screen
|
|
capture, and treating untrusted grid/LLM text as instructions or authority.
|
|
|
|
## Release evidence
|
|
|
|
Build the intentional graphs and retain the commands/output with the release:
|
|
|
|
```sh
|
|
cargo build --locked --release -p metacrate-grid-agent
|
|
cargo build --locked --release -p metacrate-grid-agent --features live-grid
|
|
cargo tree --locked -p metacrate-grid-agent --features live-grid -e normal,build
|
|
cargo audit
|
|
```
|
|
|
|
Record the binary byte size (`stat -c %s` on Linux or `Get-Item ... .Length` on
|
|
PowerShell) and the `cargo tree` inventory. Reject CLR/scripting/provider SDK,
|
|
subprocess adapters, unreviewed graphics stacks, or undeclared native libraries
|
|
on the agent path.
|
|
The consolidated Gitea gate runs only on `ubuntu-latest`; Windows portability is
|
|
proved by the existing cross-target compile/static gate rather than a Windows
|
|
Gitea runner.
|
|
|
|
The milestone baseline is recorded in
|
|
[grid-agent-release-evidence.md](grid-agent-release-evidence.md).
|
|
|
|
Related contracts: [architecture](grid-agent-architecture.md),
|
|
[policy](grid-agent-policy.md), [session](grid-agent-session.md),
|
|
[control](grid-agent-control-plane.md), [observability](grid-agent-observability.md),
|
|
[TUI](grid-agent-tui.md), and [conversation storage](grid-agent-conversation.md).
|