feat(grid-agent): supervise grid sessions (#121)
Some checks failed
CI / rust-skia (Rust only) (push) Successful in 2m47s
CI / required (push) Failing after 2m54s

This commit is contained in:
2026-08-17 21:56:15 +00:00
parent e3b9d575f9
commit 3553c83ffa
13 changed files with 2377 additions and 36 deletions

View File

@@ -10,18 +10,21 @@ platform-specific core path.
## Ownership and bounds
`AgentService::start` is the sole task-creation point. It validates the complete
configuration before allocating channels, calling a backend, or permitting
network access. `ServiceHandle` then exclusively owns cancellation, both join
handles, the control sender, and observable receiver.
`AgentService::start` owns offline task creation, while
`SessionSupervisor::start` owns the live lifecycle task. Configuration is
validated before either allocates channels, calls a backend, or permits network
access. Their handles exclusively own cancellation, join handles, controls, and
observable receivers.
| Resource | Owner | Hard/configured bound | Backpressure/termination |
| --- | --- | --- | --- |
| Grid-event queue | coordinator receives; backend sends | 8,192 / `grid_event_queue` | async send or cancellation |
| Control queue | coordinator receives; handle sends | 256 / `control_queue` | async send; closed after stop |
| Observable queue | handle receives; coordinator/backend send | 8,192 / `observable_queue` | async send or cancellation |
| Backend task | `ServiceHandle.tasks[0]` | exactly one | shared cancellation token, joined first |
| Backend task | `ServiceHandle.tasks[0]` | exactly one in offline mode | shared cancellation token, joined first |
| Coordinator task | `ServiceHandle.tasks[1]` | exactly one | shared cancellation token, joined second |
| Live session supervisor | `SessionSupervisorHandle` | one owner; one active generation/session | generation cancellation, exact-once logout, bounded join |
| Offline session work | live supervisor | 1,024 hard / `reconnect.offline_work_capacity` | read-only queue; mutations rejected while not ready |
| Body / message | typed boundary owners | 8 MiB / 64 KiB hard ceilings, with lower configured limits | rejected before enqueue |
| Conversation / tool calls | request owner | 256 messages / 64 calls, with lower configured limits | rejected before request |
| LLM request slots | shared `LlmClient` semaphore | 256 hard / configured concurrent requests | async acquire or cancellation |
@@ -36,7 +39,7 @@ type. Dynamic configuration can lower these absolute ceilings but cannot raise
them. Backend error text is not forwarded; observations publish a fixed bounded
diagnostic.
## State machine and shutdown
## State machines and shutdown
The explicit service states are `starting -> running <-> paused -> stopping ->
stopped`, with `failed` reserved for task failure. The offline backend publishes
@@ -54,6 +57,13 @@ feature-enabled live adapter drops its
`LibremetaverseClientOwner` last, invoking the existing client ownership
shutdown.
Live modes use `stopped`, `connecting`, `degraded` (transport connected but not
fully ready), `online`, `backoff`, `authentication-blocked`, `paused`, and
`shutting-down`. Every attempt rotates a generation cancellation token.
Disconnect, pause, logout, force reconnect, or shutdown invalidates it before
cleanup, fencing old events and late LLM/tool results. See
[`grid-agent-session.md`](grid-agent-session.md).
## Trust boundaries
- JSON configuration and environment text are untrusted. Unknown fields,
@@ -83,7 +93,7 @@ shutdown.
metacrate-grid-agent binary (portable Ctrl-C + config path)
|
v
AgentService -> bounded Tokio channels/tasks -> injected GridBackend
offline AgentService -> bounded Tokio channels/tasks -> injected GridBackend
| |
v v
typed config/events/policy boundaries live-grid feature boundary
@@ -104,3 +114,5 @@ The precise LLM compatibility and cancellation contract is documented in
[`grid-agent-llm.md`](grid-agent-llm.md).
The origin/capability matrix and opaque mutation boundary are documented in
[`grid-agent-policy.md`](grid-agent-policy.md).
The live lifecycle and generation contract is documented in
[`grid-agent-session.md`](grid-agent-session.md).

View File

@@ -0,0 +1,75 @@
# Grid-session supervision
Live integrated and split modes run through `SessionSupervisor`. Its injected
`GridSessionBackend` makes the lifecycle deterministic under a fake grid while
the `live-grid` adapter reuses `libremetaverse::NetworkManager` for native
login, event-queue readiness, disconnect callbacks, and logout.
The native login does not yield a session until the login response has populated
the existing inventory skeleton and a current simulator/UDP circuit exists.
The adapter then waits for that simulator's event queue before publishing full
readiness. Existing `GridClient` inventory/world managers and its single
movement-update owner remain the composition owners; the adapter never creates
duplicates. Only its generation-scoped readiness/disconnect subscriptions are
recreated, and their RAII guards are dropped before logout.
## States and reasons
| State | Transport connected | Agent ready | Exit condition |
| --- | --- | --- | --- |
| `stopped` | no | no | startup, resume, or force reconnect |
| `connecting` | no | no | login result, operator control, or cancellation |
| `degraded` | yes | no | readiness, disconnect, or operator control |
| `online` | yes | yes | readiness loss, disconnect, or operator control |
| `backoff` | no | no | cancellation-driven timer or operator control |
| `authentication-blocked` | no | no | resume/force reconnect or shutdown |
| `paused` | no | no | resume/force reconnect, logout, or shutdown |
| `shutting-down` | no | no | session logout, audit flush, and joined owner task |
Every transition carries a stable `SessionReason`; raw login responses, server
messages, credentials, capability URLs, and session tokens cannot enter the
failure or observation types. `SessionStatus` reports transport connectivity
and complete readiness separately.
Invalid credentials and invalid local login configuration stop automatic
retry. Transport failures, maintenance, kicks, simulator disconnects, and
server failures retry with exponential backoff. `reconnect.maximum_delay_seconds`
is a hard cap. A server retry hint is a minimum up to that cap. Per-instance
jitter is bounded by `jitter_basis_points`, preventing synchronized reconnects,
and a connection that remains up for `stable_reset_seconds` resets the failure
streak.
Backoff defaults are one second initially, 60 seconds maximum, 20 percent
jitter, and a 120-second stable reset window. All waits use Tokio timers inside
`select!` with cancellation/control; there are no blocking sleeps.
## Generation and work safety
Each attempt receives a monotonically changing generation and cancellation
token. Session-scoped subscriptions, readiness ownership, and workers belong to
the returned `GridSession` and are closed once by its consuming `logout` method.
Late inference/tool results are accepted only when `accepts_result` sees their
exact generation in fully ready state.
While not ready, bounded read-only work may queue. Mutation work—including
nominally idempotent mutation—is rejected, because the supervisor cannot prove
that a previous policy authorization remains valid. Work IDs are deduplicated
across reconnects with a bounded recent-ID set, so reconnect never silently
duplicates an outbound response. Queued reads are released once into the new
generation.
## Shutdown
Shutdown first rejects new work and invalidates the generation token, which
cancels inference, tools, and scheduled consumers sharing it. It then consumes
the active session for one logout attempt, waits within
`timeouts.shutdown_seconds`, calls the backend's bounded audit-flush hook, and
joins the supervisor owner task. A late task is aborted and awaited by the
handle, so no task or socket is detached.
Focused deterministic verification:
```sh
cargo test --locked -p metacrate-grid-agent --lib session_tests
cargo clippy --locked -p metacrate-grid-agent --all-targets -- -D warnings
```