[Grid agent] Supervise login, disconnect recovery, and graceful shutdown #121

New Issue

hugo · 2026-08-17T19:31:34Z

hugo commented

2026-08-17 19:31:34 +00:00

Objective

Implement a supervised grid-session state machine that logs in, detects loss of session, reconnects responsibly after maintenance or kicks, and shuts down without orphaned tasks or duplicate subscriptions.

Required behavior

Model explicit states such as stopped, connecting, online, degraded, backoff, authentication-blocked, paused, and shutting-down. Publish transitions and reason codes to observability/control consumers.
Use the existing native NetworkManager login/disconnect lifecycle. Classify authentication/configuration failures as operator-actionable and do not loop; classify transient transport, simulator, maintenance, and server failures for retry.
Retry with bounded exponential backoff, jitter, configurable cap, and a stable-session reset window. Honor server retry hints where available. Prevent tight reconnect loops and reconnect storms after many simultaneous failures.
Recreate event subscriptions, inventory/world readiness, movement update ownership, and session-scoped tasks exactly once after login. Fence old-generation events and LLM/tool results so they cannot act after reconnect.
Queue or reject inbound work predictably while offline. Mutating work must not silently replay unless it has a safe idempotency contract and remains authorized.
Support operator pause/resume, force reconnect, logout, and graceful shutdown. Shutdown cancels inference and scheduled work, stops new actions, joins workers, logs out once, flushes bounded audit state, and exits within a documented deadline.
Never expose grid credentials, login response secrets, capability URLs, or session tokens in errors or telemetry.

Acceptance criteria

Paused-time unit tests cover transient failure, invalid credentials, maintenance, kick, simulator disconnect, flapping, stable reset, manual pause/reconnect, and shutdown during each state.
Fake-grid integration proves repeated reconnects leave one set of callbacks/workers, no stale tool execution, no duplicate outbound response, and zero tasks/sockets after shutdown.
Status reports distinguish connected transport from fully ready agent state.
Reconnect waits are cancellation-driven rather than blocking sleeps.

Dependencies

Depends on the architecture issue. It may use the observability/control event interfaces once available but must remain testable without a TUI or real LLM.

## Objective Implement a supervised grid-session state machine that logs in, detects loss of session, reconnects responsibly after maintenance or kicks, and shuts down without orphaned tasks or duplicate subscriptions. ## Required behavior - Model explicit states such as stopped, connecting, online, degraded, backoff, authentication-blocked, paused, and shutting-down. Publish transitions and reason codes to observability/control consumers. - Use the existing native `NetworkManager` login/disconnect lifecycle. Classify authentication/configuration failures as operator-actionable and do not loop; classify transient transport, simulator, maintenance, and server failures for retry. - Retry with bounded exponential backoff, jitter, configurable cap, and a stable-session reset window. Honor server retry hints where available. Prevent tight reconnect loops and reconnect storms after many simultaneous failures. - Recreate event subscriptions, inventory/world readiness, movement update ownership, and session-scoped tasks exactly once after login. Fence old-generation events and LLM/tool results so they cannot act after reconnect. - Queue or reject inbound work predictably while offline. Mutating work must not silently replay unless it has a safe idempotency contract and remains authorized. - Support operator pause/resume, force reconnect, logout, and graceful shutdown. Shutdown cancels inference and scheduled work, stops new actions, joins workers, logs out once, flushes bounded audit state, and exits within a documented deadline. - Never expose grid credentials, login response secrets, capability URLs, or session tokens in errors or telemetry. ## Acceptance criteria - [ ] Paused-time unit tests cover transient failure, invalid credentials, maintenance, kick, simulator disconnect, flapping, stable reset, manual pause/reconnect, and shutdown during each state. - [ ] Fake-grid integration proves repeated reconnects leave one set of callbacks/workers, no stale tool execution, no duplicate outbound response, and zero tasks/sockets after shutdown. - [ ] Status reports distinguish connected transport from fully ready agent state. - [ ] Reconnect waits are cancellation-driven rather than blocking sleeps. ## Dependencies Depends on the architecture issue. It may use the observability/control event interfaces once available but must remain testable without a TUI or real LLM.

hugo added this to the 14 - metacrate grid agent milestone 2026-08-17 19:31:34 +00:00

hugo added the enhancement label 2026-08-17 19:31:34 +00:00

hugo referenced this issue

2026-08-17 19:32:14 +00:00

implement an AI driven agent that runs in opensim #117

hugo referenced this issue from a commit

2026-08-17 21:56:26 +00:00

feat(grid-agent): supervise grid sessions (#121)

hugo commented

2026-08-17 21:58:20 +00:00

Implemented and pushed in commit 3553c83ffa.

Implementation:

Added the explicit stopped, connecting, online, degraded, backoff, auth-blocked, paused, and shutting-down supervisor states with structured, secret-safe reason events.
Added a single-owner session actor with generation fencing, cancellation of stale inference, tool, and scheduler work, bounded offline read-only work, reconnect-safe work deduplication, and explicit pause, resume, reconnect, logout, and shutdown controls.
Added bounded exponential reconnect backoff with jitter, server retry hints, stable-ready reset, and flapping and storm protection.
Added exact session resource ownership and graceful bounded cleanup and flush behavior.
Wired the live adapter to the existing native libremetaverse NetworkManager lifecycle. Subscriptions are removed before consuming logout, and the native managers retain inventory, world, and movement ownership.
Documented the lifecycle and configuration and updated the example configuration.

Verification:

cargo fmt --all -- --check
cargo clippy -p metacrate-grid-agent --all-targets -- -D warnings
cargo test -p metacrate-grid-agent --all-targets: 58 tests passed
Focused paused-time supervisor tests: 8 passed
Focused fake-grid resource and fencing integration test: passed
Doctests and rustdoc with -D warnings: passed
Example configuration validation: passed
Explicit Linux target check: passed
cargo deny check: advisories, bans, licenses, and sources passed
cargo machete: only the pre-existing unrelated libremetaverse-types to md-5 report

Review found and corrected several edge cases before commit: work arriving during connection no longer cancels login; jitter cannot undercut a server retry hint; readiness, not mere transport connection, controls stable reset; duplicate work is fenced across reconnects; cleanup and audit flushing share the shutdown deadline; native Debug output is opaque; and generation overflow shuts down safely.

The focused acceptance tests use fake grid sessions, so no credentials from .env were read or committed. A local live-grid feature check was attempted several ways, but this environment compiler was SIGKILLed for memory while checking the pre-existing monolithic libremetaverse crate, before reaching grid-agent and without a Rust diagnostic. The pushed workflow remains the authoritative all-features check.

Implemented and pushed in commit 3553c83ffac8fd84d8fa544e1408dca355094a31. Implementation: - Added the explicit stopped, connecting, online, degraded, backoff, auth-blocked, paused, and shutting-down supervisor states with structured, secret-safe reason events. - Added a single-owner session actor with generation fencing, cancellation of stale inference, tool, and scheduler work, bounded offline read-only work, reconnect-safe work deduplication, and explicit pause, resume, reconnect, logout, and shutdown controls. - Added bounded exponential reconnect backoff with jitter, server retry hints, stable-ready reset, and flapping and storm protection. - Added exact session resource ownership and graceful bounded cleanup and flush behavior. - Wired the live adapter to the existing native libremetaverse NetworkManager lifecycle. Subscriptions are removed before consuming logout, and the native managers retain inventory, world, and movement ownership. - Documented the lifecycle and configuration and updated the example configuration. Verification: - cargo fmt --all -- --check - cargo clippy -p metacrate-grid-agent --all-targets -- -D warnings - cargo test -p metacrate-grid-agent --all-targets: 58 tests passed - Focused paused-time supervisor tests: 8 passed - Focused fake-grid resource and fencing integration test: passed - Doctests and rustdoc with -D warnings: passed - Example configuration validation: passed - Explicit Linux target check: passed - cargo deny check: advisories, bans, licenses, and sources passed - cargo machete: only the pre-existing unrelated libremetaverse-types to md-5 report Review found and corrected several edge cases before commit: work arriving during connection no longer cancels login; jitter cannot undercut a server retry hint; readiness, not mere transport connection, controls stable reset; duplicate work is fenced across reconnects; cleanup and audit flushing share the shutdown deadline; native Debug output is opaque; and generation overflow shuts down safely. The focused acceptance tests use fake grid sessions, so no credentials from .env were read or committed. A local live-grid feature check was attempted several ways, but this environment compiler was SIGKILLed for memory while checking the pre-existing monolithic libremetaverse crate, before reaching grid-agent and without a Rust diagnostic. The pushed workflow remains the authoritative all-features check.

hugo closed this issue

2026-08-17 21:59:11 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/MetaCrate#121