feat(grid-agent): supervise grid sessions (#121)
This commit is contained in:
@@ -10,18 +10,21 @@ platform-specific core path.
|
||||
|
||||
## Ownership and bounds
|
||||
|
||||
`AgentService::start` is the sole task-creation point. It validates the complete
|
||||
configuration before allocating channels, calling a backend, or permitting
|
||||
network access. `ServiceHandle` then exclusively owns cancellation, both join
|
||||
handles, the control sender, and observable receiver.
|
||||
`AgentService::start` owns offline task creation, while
|
||||
`SessionSupervisor::start` owns the live lifecycle task. Configuration is
|
||||
validated before either allocates channels, calls a backend, or permits network
|
||||
access. Their handles exclusively own cancellation, join handles, controls, and
|
||||
observable receivers.
|
||||
|
||||
| Resource | Owner | Hard/configured bound | Backpressure/termination |
|
||||
| --- | --- | --- | --- |
|
||||
| Grid-event queue | coordinator receives; backend sends | 8,192 / `grid_event_queue` | async send or cancellation |
|
||||
| Control queue | coordinator receives; handle sends | 256 / `control_queue` | async send; closed after stop |
|
||||
| Observable queue | handle receives; coordinator/backend send | 8,192 / `observable_queue` | async send or cancellation |
|
||||
| Backend task | `ServiceHandle.tasks[0]` | exactly one | shared cancellation token, joined first |
|
||||
| Backend task | `ServiceHandle.tasks[0]` | exactly one in offline mode | shared cancellation token, joined first |
|
||||
| Coordinator task | `ServiceHandle.tasks[1]` | exactly one | shared cancellation token, joined second |
|
||||
| Live session supervisor | `SessionSupervisorHandle` | one owner; one active generation/session | generation cancellation, exact-once logout, bounded join |
|
||||
| Offline session work | live supervisor | 1,024 hard / `reconnect.offline_work_capacity` | read-only queue; mutations rejected while not ready |
|
||||
| Body / message | typed boundary owners | 8 MiB / 64 KiB hard ceilings, with lower configured limits | rejected before enqueue |
|
||||
| Conversation / tool calls | request owner | 256 messages / 64 calls, with lower configured limits | rejected before request |
|
||||
| LLM request slots | shared `LlmClient` semaphore | 256 hard / configured concurrent requests | async acquire or cancellation |
|
||||
@@ -36,7 +39,7 @@ type. Dynamic configuration can lower these absolute ceilings but cannot raise
|
||||
them. Backend error text is not forwarded; observations publish a fixed bounded
|
||||
diagnostic.
|
||||
|
||||
## State machine and shutdown
|
||||
## State machines and shutdown
|
||||
|
||||
The explicit service states are `starting -> running <-> paused -> stopping ->
|
||||
stopped`, with `failed` reserved for task failure. The offline backend publishes
|
||||
@@ -54,6 +57,13 @@ feature-enabled live adapter drops its
|
||||
`LibremetaverseClientOwner` last, invoking the existing client ownership
|
||||
shutdown.
|
||||
|
||||
Live modes use `stopped`, `connecting`, `degraded` (transport connected but not
|
||||
fully ready), `online`, `backoff`, `authentication-blocked`, `paused`, and
|
||||
`shutting-down`. Every attempt rotates a generation cancellation token.
|
||||
Disconnect, pause, logout, force reconnect, or shutdown invalidates it before
|
||||
cleanup, fencing old events and late LLM/tool results. See
|
||||
[`grid-agent-session.md`](grid-agent-session.md).
|
||||
|
||||
## Trust boundaries
|
||||
|
||||
- JSON configuration and environment text are untrusted. Unknown fields,
|
||||
@@ -83,7 +93,7 @@ shutdown.
|
||||
metacrate-grid-agent binary (portable Ctrl-C + config path)
|
||||
|
|
||||
v
|
||||
AgentService -> bounded Tokio channels/tasks -> injected GridBackend
|
||||
offline AgentService -> bounded Tokio channels/tasks -> injected GridBackend
|
||||
| |
|
||||
v v
|
||||
typed config/events/policy boundaries live-grid feature boundary
|
||||
@@ -104,3 +114,5 @@ The precise LLM compatibility and cancellation contract is documented in
|
||||
[`grid-agent-llm.md`](grid-agent-llm.md).
|
||||
The origin/capability matrix and opaque mutation boundary are documented in
|
||||
[`grid-agent-policy.md`](grid-agent-policy.md).
|
||||
The live lifecycle and generation contract is documented in
|
||||
[`grid-agent-session.md`](grid-agent-session.md).
|
||||
|
||||
75
docs/grid-agent-session.md
Normal file
75
docs/grid-agent-session.md
Normal file
@@ -0,0 +1,75 @@
|
||||
# Grid-session supervision
|
||||
|
||||
Live integrated and split modes run through `SessionSupervisor`. Its injected
|
||||
`GridSessionBackend` makes the lifecycle deterministic under a fake grid while
|
||||
the `live-grid` adapter reuses `libremetaverse::NetworkManager` for native
|
||||
login, event-queue readiness, disconnect callbacks, and logout.
|
||||
|
||||
The native login does not yield a session until the login response has populated
|
||||
the existing inventory skeleton and a current simulator/UDP circuit exists.
|
||||
The adapter then waits for that simulator's event queue before publishing full
|
||||
readiness. Existing `GridClient` inventory/world managers and its single
|
||||
movement-update owner remain the composition owners; the adapter never creates
|
||||
duplicates. Only its generation-scoped readiness/disconnect subscriptions are
|
||||
recreated, and their RAII guards are dropped before logout.
|
||||
|
||||
## States and reasons
|
||||
|
||||
| State | Transport connected | Agent ready | Exit condition |
|
||||
| --- | --- | --- | --- |
|
||||
| `stopped` | no | no | startup, resume, or force reconnect |
|
||||
| `connecting` | no | no | login result, operator control, or cancellation |
|
||||
| `degraded` | yes | no | readiness, disconnect, or operator control |
|
||||
| `online` | yes | yes | readiness loss, disconnect, or operator control |
|
||||
| `backoff` | no | no | cancellation-driven timer or operator control |
|
||||
| `authentication-blocked` | no | no | resume/force reconnect or shutdown |
|
||||
| `paused` | no | no | resume/force reconnect, logout, or shutdown |
|
||||
| `shutting-down` | no | no | session logout, audit flush, and joined owner task |
|
||||
|
||||
Every transition carries a stable `SessionReason`; raw login responses, server
|
||||
messages, credentials, capability URLs, and session tokens cannot enter the
|
||||
failure or observation types. `SessionStatus` reports transport connectivity
|
||||
and complete readiness separately.
|
||||
|
||||
Invalid credentials and invalid local login configuration stop automatic
|
||||
retry. Transport failures, maintenance, kicks, simulator disconnects, and
|
||||
server failures retry with exponential backoff. `reconnect.maximum_delay_seconds`
|
||||
is a hard cap. A server retry hint is a minimum up to that cap. Per-instance
|
||||
jitter is bounded by `jitter_basis_points`, preventing synchronized reconnects,
|
||||
and a connection that remains up for `stable_reset_seconds` resets the failure
|
||||
streak.
|
||||
|
||||
Backoff defaults are one second initially, 60 seconds maximum, 20 percent
|
||||
jitter, and a 120-second stable reset window. All waits use Tokio timers inside
|
||||
`select!` with cancellation/control; there are no blocking sleeps.
|
||||
|
||||
## Generation and work safety
|
||||
|
||||
Each attempt receives a monotonically changing generation and cancellation
|
||||
token. Session-scoped subscriptions, readiness ownership, and workers belong to
|
||||
the returned `GridSession` and are closed once by its consuming `logout` method.
|
||||
Late inference/tool results are accepted only when `accepts_result` sees their
|
||||
exact generation in fully ready state.
|
||||
|
||||
While not ready, bounded read-only work may queue. Mutation work—including
|
||||
nominally idempotent mutation—is rejected, because the supervisor cannot prove
|
||||
that a previous policy authorization remains valid. Work IDs are deduplicated
|
||||
across reconnects with a bounded recent-ID set, so reconnect never silently
|
||||
duplicates an outbound response. Queued reads are released once into the new
|
||||
generation.
|
||||
|
||||
## Shutdown
|
||||
|
||||
Shutdown first rejects new work and invalidates the generation token, which
|
||||
cancels inference, tools, and scheduled consumers sharing it. It then consumes
|
||||
the active session for one logout attempt, waits within
|
||||
`timeouts.shutdown_seconds`, calls the backend's bounded audit-flush hook, and
|
||||
joins the supervisor owner task. A late task is aborted and awaited by the
|
||||
handle, so no task or socket is detached.
|
||||
|
||||
Focused deterministic verification:
|
||||
|
||||
```sh
|
||||
cargo test --locked -p metacrate-grid-agent --lib session_tests
|
||||
cargo clippy --locked -p metacrate-grid-agent --all-targets -- -D warnings
|
||||
```
|
||||
Reference in New Issue
Block a user