implement an AI driven agent that runs in opensim #117

New Issue

hugo · 2026-08-14T19:02:06Z

hugo commented

2026-08-14 19:02:06 +00:00

Implement an agent harness that connects to opensim and provides internal tools to the LLM that are based on libremetaverse. This agent should be able to log in, re-login when kicked out (due to maintenance or things like that), and actually be able to act in the world and interact with the world. starting a bilities:

login and relogin when kicked out with error handling and proper wait-times
public chat with people near them, reacting to @ notifications and using the LLM to generate answers
- public chat is ALWAYS seen as non-secure and the agent will never accept commands from public chat
- public chat is useable for querying informations
- the agent will honor users asking for LSL scripts in public chat and create a script in inverntory and then hand that over full-perm to the user asking for it
- the agent keeps track of a chat session per user. if a user is not interacting with the agent for 30 min, the agent starts a new session
instant message with avatars are handled like public chat, just on an IM channel
- instant message conversations can be elevated permissions by checking a list of allowed UUIDs
- higher permissions accept commands
- lower permissions work just like public chat
- unlike public chat, instant message sessions are kept alive for longer - if the user is not communicating for 24h, the session is closed and a new session is started
the agent can receive landmarks from authorized users and used them to teleport there when told to by authorized users
the agent can receive folders with landmarks and will accept a command to randomly teleport to a landmark from that folder every X minutesl
authorized users can tell the agent to create scripted objects, which the agent creates out of prims
the agent can take a snapshot of its viewport and use that to react to questions referencing visuals
the agent will turn to a user addressing them in public
the agent runs a TUI so that the admin can watch what it is doing
- the tui can be integrated with the agent in one process, but must support a split setup
- one part will be the actual agent process that can then be configured as a systemd service
- the other part will be the tui that connects to the service process
- the admin can check activity logs of the agent via the tui

The agent will use a configureable openai compatible endpoint for the model and should be able to work with self-hosted or cheap models (think GPT 5.6 Luna or Deepseek V4 Flash or Qwen 3.6 35B) properly, so some of the features like the build feature might be more of an advanced feature for later implementation.

this all needs to be fleshed out, this is more an idea list here, this has to be turned into a proper backlog under its own milestone. additional commands and actions might make sense, based on what the library provides.

Implement an agent harness that connects to opensim and provides internal tools to the LLM that are based on libremetaverse. This agent should be able to log in, re-login when kicked out (due to maintenance or things like that), and actually be able to act in the world and interact with the world. starting a bilities: - login and relogin when kicked out with error handling and proper wait-times - public chat with people near them, reacting to @ notifications and using the LLM to generate answers - public chat is ALWAYS seen as non-secure and the agent will never accept commands from public chat - public chat is useable for querying informations - the agent will honor users asking for LSL scripts in public chat and create a script in inverntory and then hand that over full-perm to the user asking for it - the agent keeps track of a chat session per user. if a user is not interacting with the agent for 30 min, the agent starts a new session - instant message with avatars are handled like public chat, just on an IM channel - instant message conversations can be elevated permissions by checking a list of allowed UUIDs - higher permissions accept commands - lower permissions work just like public chat - unlike public chat, instant message sessions are kept alive for longer - if the user is not communicating for 24h, the session is closed and a new session is started - the agent can receive landmarks from authorized users and used them to teleport there when told to by authorized users - the agent can receive folders with landmarks and will accept a command to randomly teleport to a landmark from that folder every X minutesl - authorized users can tell the agent to create scripted objects, which the agent creates out of prims - the agent can take a snapshot of its viewport and use that to react to questions referencing visuals - the agent will turn to a user addressing them in public - the agent runs a TUI so that the admin can watch what it is doing - the tui can be integrated with the agent in one process, but must support a split setup - one part will be the actual agent process that can then be configured as a systemd service - the other part will be the tui that connects to the service process - the admin can check activity logs of the agent via the tui The agent will use a configureable openai compatible endpoint for the model and should be able to work with self-hosted or cheap models (think GPT 5.6 Luna or Deepseek V4 Flash or Qwen 3.6 35B) properly, so some of the features like the build feature might be more of an advanced feature for later implementation. this all needs to be fleshed out, this is more an idea list here, this has to be turned into a proper backlog under its own milestone. additional commands and actions might make sense, based on what the library provides.

hugo added this to the 13 - Extensions milestone 2026-08-14 19:02:06 +00:00

hugo added the idea label 2026-08-14 19:02:06 +00:00

hugo commented

2026-08-17 19:32:14 +00:00

Backlog planning is complete. The idea has been expanded into milestone 14 - metacreate grid agent with 18 self-contained enhancement issues.

Suggested delivery sequence:

Foundation and safety: #118 architecture/config, #119 generic endpoint/tool loop, #120 policy/authorization, #121 lifecycle, #122 sessions.
Core in-world behavior: #123 messaging, #124 perception, #125 embodied presence.
External management and learning: #126 control plane, #127 observability/replay, #128 TUI.
World capabilities: #129 LSL delivery, #130 landmarks/teleport/roaming, #131 scripted prim builds, #132 pure-Rust visual snapshots.
Proof and operations: #133 deterministic/adversarial harness, #134 portable service packaging/docs, #135 final offline and opt-in live-grid gate.

The milestone fixes the architectural constraints up front: a compact pure-Rust implementation; cross-platform code and APIs; Gitea jobs on ubuntu-latest only; one exact OpenAI-compatible endpoint URL plus API key with no provider presets or model catalog; deny-by-default command authorization; bounded resource use; deterministic fake-grid/fake-LLM testing; and externally observable action summaries, policy/tool traces, and outcomes without exposing secrets or hidden chain-of-thought.

No implementation was started in this planning pass. Closing this idea issue in favor of the milestone backlog.

Backlog planning is complete. The idea has been expanded into milestone [14 - metacreate grid agent](https://git.rfc1437.de/hugo/MetaCrate/issues?milestone=24) with 18 self-contained enhancement issues. Suggested delivery sequence: - Foundation and safety: #118 architecture/config, #119 generic endpoint/tool loop, #120 policy/authorization, #121 lifecycle, #122 sessions. - Core in-world behavior: #123 messaging, #124 perception, #125 embodied presence. - External management and learning: #126 control plane, #127 observability/replay, #128 TUI. - World capabilities: #129 LSL delivery, #130 landmarks/teleport/roaming, #131 scripted prim builds, #132 pure-Rust visual snapshots. - Proof and operations: #133 deterministic/adversarial harness, #134 portable service packaging/docs, #135 final offline and opt-in live-grid gate. The milestone fixes the architectural constraints up front: a compact pure-Rust implementation; cross-platform code and APIs; Gitea jobs on `ubuntu-latest` only; one exact OpenAI-compatible endpoint URL plus API key with no provider presets or model catalog; deny-by-default command authorization; bounded resource use; deterministic fake-grid/fake-LLM testing; and externally observable action summaries, policy/tool traces, and outcomes without exposing secrets or hidden chain-of-thought. No implementation was started in this planning pass. Closing this idea issue in favor of the milestone backlog.

hugo closed this issue

2026-08-17 19:32:15 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/MetaCrate#117