Files
MetaCrate/RUSTREWRITE.md

781 lines
46 KiB
Markdown

# LibreMetaverse to Rust rewrite plan
## 1. Scope and source of truth
MetaCrate is a new native Rust implementation of the public API and observable
behavior of LibreMetaverse. LibreMetaverse is reference material, not a runtime
dependency or implementation component. The source snapshot used to create this
shell is:
- repository: <https://github.com/cinderblocks/libremetaverse>
- branch: `master`
- commit: `2aa70bb68513b39795da5d13c88f31b86e85a3ba`
- commit date: 2026-08-07
- local checkout state during inventory: clean
- license base: `libremetaverse/LICENSE.txt`, BSD 3-Clause
Generated inventories are intentionally checked in:
- `api/TYPES.tsv` maps every discovered public C# type declaration to a Rust
crate/module/type and source line.
- `api/SURFACE.tsv` records every public declaration line in the 13 library
projects, including a SHA-256 digest.
- `api/public-api.json` is the authoritative compiled-metadata catalog for all
13 library assemblies after all six source generators run. It also records
every external type referenced by a public signature.
- `api/RUST-TYPES.tsv` resolves all 3,066 public and 142 external signature
types, plus two support traits referenced by public inheritance metadata.
- `api/RUST-MAPPING.tsv` records the reviewed Rust destination and complete
signature decision for all 30,789 public members.
- `tests/upstream-tests.json` records every NUnit invocation and the SHA-256 of
its C# method body.
- `tests/PARITY.md` maps every NUnit invocation to its Rust test name.
- `programs/upstream-programs.json` records every source file and digest for the
nine sample/tool projects.
Run `python3 tools/generate_surface.py` after deliberately changing the pinned
upstream snapshot, then run `python3 tools/extract_public_api.py` to rebuild the
authoritative metadata catalog twice and require byte-identical output. The
source generator rejects a test count other than 1,295 so upstream drift cannot
silently remove tests. Update both commit constants and review all ledger diffs
in the same change.
### Current shell status
The current workspace is a structural baseline, not a working client:
- all 13 public .NET library projects have corresponding Rust crates;
- 1,705 source-discovered public C# type declarations produce compiling Rust
type/trait shims;
- 12,195 public declaration lines are retained in the API ledger;
- compiled metadata records 3,066 public types and 30,789 public members with
matching independent ECMA-335 counts, with all 142 external signature types
resolved to cross-platform Rust or project-owned boundary types;
- the completed Types, StructuredData, Imaging, PrimMesher, rendering, main
wire/data, and core runtime/networking slices expose 2,404 final Rust-facing
types and all 25,257 callable members with real fields/constants/enum values
and standardized failing bodies;
- all 1,295 NUnit `[Test]`/`[TestCase]` invocations have compiling Rust test
entries that intentionally panic with their source identity and body hash;
- all nine sample/tool projects have compiling Rust binary targets;
- the 128 TestClient command source files are retained as a command inventory.
The remaining 662 generated type shims preserve names and module placement, but
do not yet expose their 5,532 callable members. `api/SURFACE.tsv` is an inventory aid,
not proof of API coverage: it records declaration lines but does not fully parse
multiline signatures. Likewise, a generated `pending` test is a parity
placeholder, not a semantic translation. **This repository is therefore at
Stage 0 and does not yet satisfy either the full API-shim gate or the full
test-suite gate.**
No functional porting starts until both missing gates are complete across the
entire workspace, rather than one crate at a time:
1. every public C# type and member has a reviewed, callable Rust signature with
an intentionally failing body;
2. every C# test is translated semantically and compiles while calling that
Rust API;
3. only then is behavior implemented, one crate at a time, until those already
translated tests turn green.
### Full-rewrite constraint
The final project must be entirely implemented in new Rust code. The following
are prohibited in production libraries, examples, tools, tests presented as
Rust behavior, and release artifacts:
- loading or hosting the CLR/.NET runtime;
- P/Invoke, COM, C ABI, UniFFI, generated bindings, or any other foreign-function
bridge to LibreMetaverse assemblies;
- invoking `dotnet`, LibreMetaverse programs, or helper services as a subprocess
to implement Rust API behavior;
- embedding, redistributing, dynamically loading, or downloading compiled
LibreMetaverse assemblies;
- forwarding calls over IPC/RPC to a running C# implementation;
- retaining a C# fallback path for APIs not yet implemented in Rust;
- treating an interoperability comparison harness as the implementation.
During development, the pinned C# source and compiled metadata may be inspected
to inventory public signatures, understand algorithms and state transitions,
and create golden inputs/outputs. The original C# tests and programs may be run
separately as an oracle to produce or compare fixtures. Those development tools
must remain outside the Rust runtime graph, must not be required to build or use
published Rust crates, and must never satisfy a Rust compatibility test by
executing the C# implementation.
Third-party native libraries such as OpenJPEG, Skia, libopus, operating-system
APIs, and ordinary Rust crates are allowed where documented and license-
compatible; the prohibition is specifically against reusing LibreMetaverse or
another port of its implementation behind a foreign interface. All
LibreMetaverse-specific protocol logic, models, state machines, managers,
serialization rules, and public behavior must be reimplemented in Rust.
The final release gate must verify that published crate metadata and built
artifacts contain no dependency on `dotnet`, Mono, CLR hosting APIs, C#
assemblies, or LibreMetaverse bridge libraries. A clean machine with only the
documented Rust/native prerequisites must be able to build, test, document, and
run the Rust implementation.
## 2. Workspace design
| Rust crate | C# project | C# files | C# lines | Public declaration rows | Responsibility |
|---|---|---:|---:|---:|---|
| `libremetaverse-types` | `LibreMetaverse.Types` | 22 | 9,292 | 703 | UUIDs, math types, enums, caches and small collections |
| `libremetaverse-structured-data` | `LibreMetaverse.StructuredData` | 10 | 4,069 | 303 | OSD value model and LLSD encodings |
| `libremetaverse-imaging` | `LibreMetaverse.Imaging.Abstractions` | 2 | 454 | 18 | codec traits and managed image representation |
| `libremetaverse-imaging-skia` | `LibreMetaverse.Imaging.Skia` | 1 | 369 | 3 | optional Skia codec adapter |
| `libremetaverse-prim-mesher` | `LibreMetaverse.PrimMesher` | 5 | 3,477 | 222 | prim/sculpt mesh generation |
| `libremetaverse` | `LibreMetaverse` | 210 | 138,916 | 8,982 | client, protocol, managers, assets and rendering model |
| `libremetaverse-rendering-simple` | `LibreMetaverse.Rendering.Simple` | 1 | 161 | 6 | simple renderer |
| `libremetaverse-rendering-mesh-foundry` | `LibreMetaverse.Rendering.MeshFoundry` | 1 | 852 | 14 | full mesh renderer |
| `libremetaverse-lsl-tools` | `LibreMetaverse.LslTools` | 175 | 27,856 | 962 | LSL lexer/parser generator and generated grammar |
| `libremetaverse-rlv` | `LibreMetaverse.RLV` | 32 | 6,296 | 262 | RLV parsing, restrictions, locks and callbacks |
| `libremetaverse-utilities` | `LibreMetaverse.Utilities` | 1 | 289 | 12 | high-level convenience helpers |
| `libremetaverse-voice-vivox` | `LibreMetaverse.Voice.Vivox` | 12 | 4,544 | 440 | Vivox control protocol adapter |
| `libremetaverse-voice-webrtc` | `LibreMetaverse.Voice.WebRTC` | 11 | 7,023 | 268 | WebRTC voice adapter |
`libremetaverse` is the umbrella client crate. It re-exports the foundational
crates under `types`, `structured_data`, and `imaging_abstractions`. Rendering,
voice, RLV, LSL tools, and native Skia remain separate so users do not pay for
large or platform-specific dependency trees unless requested.
The workspace uses Rust 2024 with MSRV 1.85. Generated code is committed so
ordinary consumers do not need Python, .NET, Roslyn, or the upstream checkout.
## 3. Public API translation policy
Compatibility means the same concepts, wire behavior, state transitions,
errors, and event payloads. It does not mean mechanically reproducing C# syntax.
Apply these rules consistently:
| C# surface | Rust surface |
|---|---|
| `LibreMetaverse.Foo.Bar` | the owning crate plus `foo::Bar`; omit the repeated assembly namespace |
| PascalCase methods | `snake_case`; add a deprecated PascalCase forwarding method only when migration evidence justifies it |
| `Task<T>` / `Task` | `async fn -> Result<T, Error>` / `async fn -> Result<(), Error>` |
| `Async` method suffix | omit it when the Rust method is `async`; document the C# source name |
| `CancellationToken` | `tokio_util::sync::CancellationToken` at long-lived operation boundaries |
| nullable reference | `Option<T>`; never use a sentinel UUID/string unless the protocol defines one |
| exception hierarchy | small non-exhaustive `thiserror` enums per crate; retain source error and protocol context |
| public mutable property | private field plus validated getter/setter, or a public field only for inert value records |
| overloads | one method with typed parameter structs; use separate descriptive names when operations differ semantically |
| interface | object-safe trait when dynamic dispatch is required, otherwise a generic trait bound |
| delegate | `Fn`, `FnMut`, or `FnOnce` bounds; boxed callback only when callbacks are stored |
| .NET event | typed subscription returning a guard, or a `tokio::sync::broadcast` receiver for asynchronous fan-out |
| `IDisposable` | ownership and `Drop`; add explicit `close`/`shutdown().await` when failure or async work must be observed |
| `byte[]` | borrowed `&[u8]` for parsing, `bytes::Bytes` for shared network payloads, `Vec<u8>` for owned mutation |
| `IReadOnlyList<T>` | `&[T]` or `Arc<[T]>`; return iterators when callers do not need storage |
| `Dictionary` | `HashMap`; use `BTreeMap` only when deterministic ordering is observable |
| immutable collections | owned `Vec`/`HashMap` or `Arc<[T]>`; do not add a persistent-collection crate without a measured need |
| monitor/reader-writer lock | `std::sync` first; use `parking_lot` only after profiling or poisoning-policy review |
| generated flags enum | `bitflags`; preserve unknown bits with `from_bits_retain` |
| C# enum on the wire | `#[repr(...)]` plus checked conversion; include `Unknown(raw)` when the protocol is extensible |
| `UUID` | a compatibility newtype around `uuid::Uuid`, with explicit big/little-endian wire helpers |
| vectors/quaternions/matrices | compatibility structs with LibreMetaverse field/order semantics; use `glam` internally after parity tests prove conversions |
| dependency injection container | normal constructors and explicit trait parameters; add no service-locator equivalent |
The checked-in mapping ledgers make those rules concrete:
- properties become getter/setter methods; indexers become `item`/`set_item`
methods with their index parameters;
- events become typed subscriptions returning a guard; stored delegates become
`Fn` callbacks, while CLR `BeginInvoke`/`EndInvoke` rows are retained as
intentional differences;
- `in` borrows immutably, `ref` borrows mutably, and `out` is a mutable output;
optional parameters use `Option<T>` rather than Rust default arguments;
- nullable values use `Option<T>` at the exact metadata node, including nested
generic arguments;
- inheritance maps to traits and composition rather than class inheritance;
generic parameters retain their names and cataloged constraints as explicit
Rust bounds recorded in the mapping ledgers;
- operators receive stable trait-style names; overloads receive parameter-based
descriptive suffixes and a stable ID suffix only when necessary;
- `[Flags]` enums use `bitflags` with unknown bits retained; wire enums use their
exact width/discriminants and an `Unknown(raw)` form only when extensible;
- all third-party types exposed by C# signatures map through project-owned
cross-platform boundary types. No mapping may name a macOS-only API unless
equivalent Linux and Windows implementations are provided.
Run `python3 tools/generate_rust_mapping.py --check` after any catalog or mapping
rule change. It regenerates in memory and rejects duplicate source IDs or Rust
destinations, stale/missing members, unresolved referenced types, invalid
statuses, missing assembly representatives, and platform-specific targets.
Public structs and enums that come from the protocol must be `#[non_exhaustive]`
when servers may add values. Do not derive `Serialize`/`Deserialize` as a
substitute for the specified wire encoding: LLUDP, LLSD, MessagePack, asset
formats, and login XML each require their own compatibility tests.
### 3.1 Full signature-shim gate
The first handover task after this structural shell is a complete callable API
shim. Do not start real behavior while completing it.
Build the authoritative API catalog during development from compiled .NET
metadata after running the original source generators, not from the line-
oriented regex ledger alone. This extraction is an inventory tool only and must
not become a build-time or runtime dependency of the Rust crates.
Build one supported target framework into a temporary output directory and use
`System.Reflection.Metadata` or Roslyn symbols to emit machine-readable records
for every public/nested-public:
- class, struct, record, interface, enum and delegate, including generic
parameters and constraints;
- constructor and method, including overloads, ref/out/in parameters, optional
values and return type;
- property/indexer and getter/setter visibility;
- field and constant, including value and numeric width;
- event and delegate payload;
- base type, implemented interface and relevant public attribute.
`api/public-api.json` is that catalog. Its `external_types` section records each
non-LibreMetaverse type used by a public signature, its defining assembly, and
every owning LibreMetaverse documentation ID; it intentionally does not copy
the external assembly's member surface. Create `api/RUST-MAPPING.tsv` with one
row per C# member and resolve every external entry to Rust core/std, an adopted
crate, or a native MetaCrate replacement: stable C# documentation ID, C#
signature, Rust crate/path, Rust signature, mapping decision, and status. A
coverage checker must fail for missing members, duplicate Rust destinations,
unresolved external types, or stale source IDs. `api/SURFACE.tsv` remains useful
for source navigation but is not the coverage authority.
Each mapped Rust item must actually compile and be callable:
- methods, constructors and property accessors have their final reviewed Rust
names, parameters, ownership/borrowing, asyncness, and return types;
- overloads use stable descriptive Rust names or typed argument structs, with
the mapping recorded explicitly;
- enums contain their real variants/discriminants and flags contain their real
bits, because tests and downstream code must compile against them;
- event payloads, delegates and interfaces have their real fields/trait methods;
- generic bounds and thread-safety guarantees are deliberate, not erased to an
untyped placeholder.
The body is the only part allowed to be fake. Fallible Rust APIs return a shared
typed `NotImplemented { csharp_member }` error. Infallible constructors,
accessors, operators and trait methods call a shared `unimplemented_api!`
placeholder that panics with the stable C# member ID. Constants and enum values
must have their real values. Never return plausible defaults such as `false`,
zero, an empty collection, or a nil UUID: those could make translated tests pass
for the wrong reason.
The failure contract lives in `libremetaverse-types` and every API crate exposes
it as `crate::Error`:
- synchronous fallible members return `Err(NotImplemented::new(CSHARP_ID))`;
async members do the same when first polled and start no work first;
- a stream-producing operation returns the typed error before exposing a
stream; an intrinsically infallible stream factory uses `unimplemented_api!`
instead of returning an empty stream;
- callback registration and stored-trait construction fail before retaining the
callback/object; generated callbacks are never silent no-ops;
- infallible constructors, accessors, operators, and trait methods call
`unimplemented_api!(CSHARP_ID)`, whose panic marker is distinct from every
typed domain error;
- constants, enum discriminants, flags, and inert public fields use their real
catalog values/layout. Generated shims do not derive `Default` unless the
reviewed API explicitly defines that behavior.
`python3 tools/generate_rust_mapping.py --check` scans every generated shim and
rejects `Default` derives, plausible false/zero/empty/nil returns, and any
function body that lacks the standardized typed error or panic mechanism.
The signature gate is complete only when:
- the metadata catalog reports 100% public type/member mapping coverage for all
13 library assemblies, including generated APIs;
- all crates build and `cargo doc` resolves every public signature;
- a downstream compile fixture can name/call every mapped Rust item;
- there are no erased `ShimValue`, dynamic argument bags, invented variadics,
or member signatures present only as ledger text;
- implementation bodies contain only the standardized failure placeholder,
constants, enum discriminants, and inert data-layout boilerplate.
### 3.2 Full semantic test-suite gate
After the full signature shim compiles, translate the entire C# test suite while
leaving all library bodies unimplemented. Every one of the 1,295 NUnit
invocations must become a Rust test with the same inputs, setup, operation,
observations, assertions, tolerances and expected error/event behavior. Shared
C# test helpers become shared Rust test helpers; embedded fixtures and literal
payloads are copied with license/source attribution and byte hashes.
The Rust tests must call the public APIs rather than internal replacements.
Where the C# tests call internal members through friend-assembly access, record
that fact and place equivalent Rust unit tests inside the owning crate without
making the member public solely for testing.
The tests must compile even though running them is red. An expected C# exception
maps to a specific Rust `Err` variant, never to `should_panic` around a public
API call; this ensures an `unimplemented_api!` panic remains a failure rather
than accidentally satisfying the test. Only tests whose C# purpose is genuinely
to verify a thrown runtime invariant may use `catch_unwind`/`should_panic`, and
they must distinguish the intended invariant from the placeholder panic.
The test-suite gate is complete only when:
- `pending(` occurs zero times in Rust test sources;
- the parity catalog reports exactly one reviewed Rust case for every upstream
`[Test]`/`[TestCase]`, with the matching C# body hash;
- `cargo test --workspace --no-run` compiles all tests against public signatures;
- a controlled test run fails at standardized unimplemented member IDs rather
than from missing symbols, compile errors, malformed fixtures, or weakened
assertions;
- live-server/device tests are faithfully translated but `#[ignore]` by default
with documented opt-in configuration;
- no production method contains real behavior beyond what is required to make
signatures and constants compile.
## 4. Validated Rust dependency map
Versions below were queried from crates.io on 2026-08-08 with `cargo search`
and, for risky/native crates, `cargo info`. They are candidates, not blanket
dependencies. Add one to a crate only when its first implementation uses it,
then commit `Cargo.lock` for application/test reproducibility. Recheck versions,
MSRV, features, and licenses at adoption time.
| .NET/NuGet responsibility | Rust candidate and validated version | Adoption notes |
|---|---|---|
| async runtime, channels, timers | [`tokio` 1.53.1](https://crates.io/crates/tokio/1.53.1), [`tokio-util` 0.7.19](https://crates.io/crates/tokio-util/0.7.19), [`futures` 0.3.33](https://crates.io/crates/futures/0.3.33) | One Tokio runtime owned by the application; libraries never create nested runtimes. |
| `System.Net.Http` / WinHTTP | [`reqwest` 0.13.4](https://crates.io/crates/reqwest/0.13.4) | MSRV 1.85. Use rustls/default TLS deliberately, connection pooling, streaming bodies, and injected client/timeouts. |
| JSON and general data mapping | [`serde` 1.0.229](https://crates.io/crates/serde/1.0.229), [`serde_json` 1.0.151](https://crates.io/crates/serde_json/1.0.151), [`serde_bytes` 0.11.19](https://crates.io/crates/serde_bytes/0.11.19) | Use only where the C# implementation is schema-driven JSON. Hand-write LLSD encodings. |
| MessagePack 3.1.8 | [`rmp-serde` 1.3.1](https://crates.io/crates/rmp-serde/1.3.1), [`rmpv` 1.3.1](https://crates.io/crates/rmpv/1.3.1) | Golden byte vectors must prove integer widths, maps, extensions, and field ordering. |
| UUID | [`uuid` 1.24.0](https://crates.io/crates/uuid/1.24.0) | Wrap it; do not expose dependency-specific endian assumptions as protocol behavior. |
| byte buffers | [`bytes` 1.12.1](https://crates.io/crates/bytes/1.12.1), [`bytemuck` 1.25.2](https://crates.io/crates/bytemuck/1.25.2) | Prefer explicit endian reads. `bytemuck` is only for proven POD layouts, never untrusted variable packets. |
| vector/matrix math | [`glam` 0.33.3](https://crates.io/crates/glam/0.33.3) | Keep compatibility newtypes to control component order, precision, equality, and serialization. |
| flags | [`bitflags` 2.13.1](https://crates.io/crates/bitflags/2.13.1) | Retain unknown bits from newer grids. |
| errors | [`thiserror` 2.0.20](https://crates.io/crates/thiserror/2.0.20) | Application binaries may use richer reporting; public libraries return typed errors. |
| ZLogger / Microsoft logging | [`tracing` 0.1.44](https://crates.io/crates/tracing/0.1.44), [`tracing-subscriber` 0.3.23](https://crates.io/crates/tracing-subscriber/0.3.23) | Libraries emit spans/events; binaries choose formatting and filters. Never log credentials or capability tokens. |
| rate limiting | [`governor` 0.10.4](https://crates.io/crates/governor/0.10.4) | Candidate for caps categories. First reproduce burst/refill/cancellation behavior with deterministic clock tests. |
| compression and tar archives | [`flate2` 1.1.9](https://crates.io/crates/flate2/1.1.9), [`tar` 0.4.46](https://crates.io/crates/tar/0.4.46) | Enforce decompressed-size, path traversal, and entry-count limits on untrusted OAR data. |
| XML and URL | [`quick-xml` 0.41.0](https://crates.io/crates/quick-xml/0.41.0), [`url` 2.5.8](https://crates.io/crates/url/2.5.8), [`base64` 0.23.1](https://crates.io/crates/base64/0.23.1) | Streaming XML for LLSD/login; retain exact URL escaping behavior with fixtures. |
| CoreJ2K 2.3.3.91 | [`jpeg2k` 0.10.1](https://crates.io/crates/jpeg2k/0.10.1) | Defaults to OpenJPEG/native bindings. Keep behind a codec trait and feature; test channel order, alpha, dimensions, discard levels, and malformed input. |
| SkiaSharp 4.150.1 | [`skia-safe` 0.99.0](https://crates.io/crates/skia-safe/0.99.0) | MSRV 1.85 and native/binary-cache build. Optional adapter only; core image APIs must not leak Skia types. |
| Pfim 0.11.4 | [`image` 0.25.10](https://crates.io/crates/image/0.25.10), [`ddsfile` 0.6.0](https://crates.io/crates/ddsfile/0.6.0) | `image` covers TGA and common DDS decoding; `ddsfile` exposes DDS container details. Golden files decide whether both are needed. |
| OggVorbisEncoder 1.2.2 | [`vorbis_rs` 0.5.6](https://crates.io/crates/vorbis_rs/0.5.6) | BSD-3-Clause, MSRV 1.82, backed by C libraries. Feature-gate native audio encoding. |
| SIPSorcery 8.0.23 | [`webrtc` 0.20.0](https://crates.io/crates/webrtc/0.20.0), [`cpal` 0.18.1](https://crates.io/crates/cpal/0.18.1), [`opus` 0.3.1](https://crates.io/crates/opus/0.3.1) | Validate SDP, ICE, data-channel framing, audio formats, device hotplug, and native libopus deployment separately. Do not claim parity from successful compilation. |
| LSL generated parser | [`lalrpop` 0.23.1](https://crates.io/crates/lalrpop/0.23.1) | Candidate only. Preserve grammar conflicts, recovery, token positions, and diagnostics before replacing the generated parser. |
| NUnit/Moq | built-in test harness, [`mockall` 0.15.0](https://crates.io/crates/mockall/0.15.0), [`proptest` 1.11.0](https://crates.io/crates/proptest/1.11.0) | Prefer fakes and deterministic protocol fixtures; use mocks only for interaction contracts. Add properties after direct parity cases exist. |
| NUnit benchmarks | [`criterion` 0.8.2](https://crates.io/crates/criterion/0.8.2) | Port benchmark-category methods to `benches/`; never make timing thresholds correctness tests. |
| sample command lines | [`clap` 4.6.6](https://crates.io/crates/clap/4.6.6) | Add when implementing the first real program CLI; keep credentials in arguments/env only long enough to authenticate. |
`System.*`, BCL compatibility packages, immutable collections, DI abstractions,
and object pooling usually map to the Rust standard library and ownership. Do
not select a crate merely because NuGet used one. In particular, begin with
`Mutex<HashMap<...>>` rather than a concurrent-map dependency and add a pool
only after allocations appear in a profile.
## 5. Module-by-module implementation guidance
### 5.1 `libremetaverse-types`
This is the first implementation crate because nearly every protocol and test
depends on it.
- **UUID:** wrap `uuid::Uuid`; implement zero/random/parse/format, byte-array
constructors, CRC/combine helpers, ordering, hashing, and every endian path
from `UUID.cs`. Keep the `UUID` name as a public compatibility type.
- **Vector2/3/3d/4, Quaternion, Matrix4, Ray, Color4:** preserve constructors,
constants, component order, normalization of zero values, quaternion
multiplication direction, approximate-comparison helpers, and binary/string
formats. Use `glam` internally only after conversion tests pass.
- **Enums and primitive enums:** translate exact numeric discriminants and flag
widths. Use `bitflags` where C# uses `[Flags]`; retain unknown values/bits.
- **CRC32 and conversions:** port byte-for-byte with fixed golden vectors and
explicit endianness. Avoid platform-sized integers on the wire.
- **CacheDictionary, ExpiringCache, TokenBucket:** inject a clock in tests;
preserve expiry/burst semantics. Start with `HashMap` plus one lock.
- **DoubleDictionary and MultiValueDictionary:** expose iterators and borrowing
rather than cloning collections. Preserve duplicate and replacement rules.
Completion gate: every Types-related parity test is a semantic Rust test, all
format/byte fixtures match C#, and no generated shim remains for a Types API.
### 5.2 `libremetaverse-structured-data`
- **OSD value model:** implement a non-exhaustive enum for undefined, boolean,
integer, real, string, UUID, date, URI, binary, array, and map. Preserve C#
conversion/default behavior deliberately; do not let `serde_json::Value`
define the public model.
- **OSDArray/OSDMap:** wrap `Vec<OSD>` and an order-appropriate map. Verify
whether serialized map ordering is observable in each format before choosing
`HashMap` versus `BTreeMap`/insertion order.
- **Binary LLSD:** streaming parser over `&[u8]`/`Read`; checked lengths,
recursion limits, exact endian rules, and no panics on malformed input.
- **Notation LLSD:** preserve escaping, length-prefixed strings/binary, numbers,
dates, URIs, whitespace, and error positions.
- **XML LLSD:** use `quick-xml`; reject/entity-limit hostile input and retain
exact element mapping.
- **JSON LLSD:** use `serde_json` behind explicit OSD conversions so UUID/date/
URI/binary tagging remains compatible.
- **Protobuf OSD:** first identify whether this is a stable external format. If
stable, preserve tags with generated Rust types; otherwise keep it internal.
Completion gate: parse and emit C# golden bytes/text for every format, including
malformed, deep, empty, Unicode, NaN/infinity, and cross-format cases.
### 5.3 `libremetaverse-imaging`
- Translate `ManagedImage` as an owned image buffer with explicit dimensions,
channel flags/layout, stride, and checked size arithmetic.
- Translate `ITextureCodec` to an object-safe `TextureCodec` trait only if
runtime codec selection is needed. Codec methods return typed decode/encode
errors and accept limits.
- Keep public APIs independent of Skia/OpenJPEG types so pure-Rust or platform
codecs can be substituted.
Completion gate: channel interleave/deinterleave, resize, alpha/bump behavior,
and invalid-dimension tests match the original.
### 5.4 `libremetaverse-imaging-skia`
Implement `SkiaTextureCodec` as an optional adapter with `skia-safe`. Convert at
the boundary to/from `ManagedImage`; never expose `skia_safe` types in the core
trait. CI must cover macOS, Linux, and Windows or explicitly document unsupported
targets. Validate native binary provenance and packaging.
### 5.5 `libremetaverse-prim-mesher`
- Port `Coord`, `Quat`, faces, paths, profiles, viewer faces, and indexing as
value types with checked indices.
- Port linear/circular extrusion before sculpt meshing. Preserve winding,
normals, UVs, face numbering, hollow/profile cuts, twists, tapers, and shears.
- Port `SculptMap` decoding through `libremetaverse-imaging`, then `SculptMesh`.
- Make OBJ export a formatting layer over mesh data; deterministic output is a
testable contract.
Completion gate: all 48 PrimMesher rendering-test invocations and all
MeshFoundry tests are semantic ports, face indices stay in range, and C# versus
Rust mesh fixtures compare within the original tolerances.
### 5.6 `libremetaverse` protocol and client core
Implement the central crate in dependency slices, not file order.
#### Wire protocol and generated packets
- Port `BitPack`, zero coding, packet headers, ACK handling, sequence numbers,
fragmentation, throttles, and `UDPPacketBuffer` first.
- Replace the six Roslyn source generators with a Rust `codegen` tool/build step
reading the same `data/` and `linden/` inputs. Commit generated packet,
visual-parameter, skeleton, tree, genepool, and attention Rust sources.
- The generator must produce deterministic output; CI regenerates to a temporary
directory and fails on diff. Generated packet tests compare exact bytes and
IDs against C# fixtures.
- Parsers operate on borrowed slices where practical and reject truncated,
oversized, or unknown blocks without unchecked indexing.
#### Networking and capabilities
- `UDPBase`, `NetworkManager`, `Simulator`, `ProtocolManager`, and `UdpThrottle`
use Tokio UDP tasks with explicit ownership and cancellation. One task owns
socket writes; decoded events cross bounded channels to prevent unbounded
memory growth.
- `Caps`, `HttpCapsClient`, event queue, caps-to-packet, and rate limiting use an
injected `reqwest::Client`. Preserve redirects, content types, retry policy,
cancellation, progress reporting, and non-HTTP location rejection.
- Login/XML-RPC, seed capabilities, region crossing, reconnect, logout, and
timeout state transitions must be modeled explicitly and tested with local
fake servers before live-grid use.
- Capability URLs/tokens and login credentials are secrets: redact them from
`Debug` and logs.
#### Grid client and service composition
- `GridClient` owns managers and shared runtime state. Construct it with a
`GridClientBuilder` only for real optional policies/codecs; required services
remain constructor arguments.
- Replace `IGridClient` and service-collection extensions with focused traits at
test seams. Avoid a general service locator.
- `Settings/*` become typed configuration structs grouped like the C# settings,
with validated durations/sizes and defaults proven by tests.
- `Logger` maps to `tracing`; `UtilizationStatistics` exports snapshots without
making a metrics backend mandatory.
#### Agent, movement, chat, effects, teleport and money
- Port `AgentManager` partial files as modules under `agent`, but expose one
coherent `AgentManager` API. Keep camera/movement state separate from network
commands.
- Event-argument classes become event structs. Subscriptions must be removable
and must not retain the whole client after drop.
- Teleport and region crossing are explicit state machines with timeout,
cancellation, progress, failure reason, and simulator handoff tests.
- Money and permissions paths use fixed integer widths and never log sensitive
transaction descriptions without opt-in.
#### Inventory
- Port inventory enums/base records/nodes/store before `InventoryManager`.
Preserve link resolution, folder/item distinctions, ownership and permissions.
- Separate local tree/cache mutation from UDP/capability transport. This makes
`InventoryAISClient` independently testable with recorded LLSD fixtures.
- Async fetch, give, rez, task inventory, current outfit folder, and callbacks
must have cancellation and deduplication tests. Do not hold locks across
`.await`.
- Persisted cache format needs versioning, atomic replacement, corruption
handling, and bounded deserialization.
#### Assets, appearance, avatar and animesh
- Port the base `Asset` model and individual asset types as typed wrappers over
validated bytes. Decode lazily where the C# behavior permits.
- `AssetManager`, `AssetCache`, `DownloadManager`, and `TexturePipeline` share a
deduplicating request layer with bounded concurrency and cancellation.
- Port wearables, baking, current-outfit policies, texture compositing, avatar
definitions, skeleton, visual params, animations, gestures, and animesh in
that order. Copy and retain the upstream Linden assets only after reviewing
their separate `cc-by-sa-3.0.txt` obligations.
- Animation interpolation, joint transforms, skinning, attachment rigs, and
physics must use the original test tolerances and matrix multiplication order.
#### Objects, parcels, terrain, environment and rendering model
- Port `Primitive`, texture entry, media, particles, permissions, materials,
and simulator collections before `ObjectManager`.
- Object updates need an explicit decode/apply split; preserve terse/compressed/
full update semantics, stale-update behavior, and parent-child linking.
- Parcel, estate, grid, terrain, sound, environment, interest-list, directory,
friends, groups, marketplace, and experience managers each get a transport-
independent state/model layer plus network handlers.
- Terrain codecs and GLTF/material handling use golden fixtures and size limits.
GLTF JSON may use Serde, but binary buffers/accessors require checked offset
arithmetic.
#### Archives and import/export
- `OarFile`, tar reader/writer, region settings, and asset archiver use `tar` and
`flate2` with path normalization, extraction-root enforcement, and size/count
limits.
- Collada/OBJ/GLTF import/export keeps format DTOs separate from world objects.
Deterministic output and round-trip fixtures are required; live upload is a
later layer.
#### Threading and utilities
- Replace custom events/semaphores/read-write locks with Tokio or `std::sync`
primitives where semantics match. Port custom optimistic/spin locks only if a
benchmark and correctness test show they are still required.
- Replace array pools with `Vec`/`Bytes` reuse first. Add pooling only after
profiling and with a bounded pool.
- Observable/event dictionaries expose change streams without executing user
callbacks while holding internal locks.
Completion gate for the core crate: no public core shim remains; all offline
parity tests pass; malformed-input fuzz targets do not panic; fake-server login,
caps, UDP, reconnect, and cancellation scenarios pass on all supported OSes.
### 5.7 Rendering crates
`libremetaverse-rendering-simple` ports `SimpleRenderer` first as the reference
implementation of the core rendering trait. It should favor clarity and serve
as a fixture generator.
`libremetaverse-rendering-mesh-foundry` then ports the complete MeshFoundry
pipeline using types, OSD, core assets, PrimMesher, and imaging. Preserve face
grouping, materials, transforms, skin weights, normals/tangents, texture
coordinates, and output ordering. Do not merge the two crates: users should be
able to choose the small renderer without the full mesh pipeline.
### 5.8 `libremetaverse-lsl-tools`
- Treat `Tools/` as the parser-generator implementation and `YYClass/` as
generated grammar output. Do not line-by-line port all generated classes.
- First translate lexer tokens, source locations, errors, comments, precedence,
grammar productions, and recovery behavior into compatibility tests.
- Re-express the grammar in LALRPOP only if it reproduces accepted/rejected
programs and diagnostic locations. Otherwise port the existing table machine
with generated Rust tables.
- Generated tables are committed and deterministically reproducible. Keep public
token/parser entry points recognizable for migration, while Rust iterators and
`Result` replace C# enumerators/exceptions.
Completion gate: all grammar fixtures and error-recovery cases match, and the
generated code is warning-free without 150 hand-maintained class shims.
### 5.9 `libremetaverse-rlv`
- Port message/command parsing and enums first, preserving case, separators,
attachment point aliases, option handling, and invalid-command errors.
- Port blacklist/provider/callback interfaces as narrow traits. Return typed
actions from pure parsing/decision code; execute viewer actions through an
injected callback adapter.
- Port restriction state, camera settings, shared folders, inventory maps,
locked folders, attachment requests, permissions, and service orchestration.
- Use immutable snapshots or short locks for restriction reads; callbacks run
after locks are released.
Completion gate: every RLV command/query/restriction/exception test case is a
semantic Rust test and all original expected error distinctions remain visible.
### 5.10 `libremetaverse-utilities`
Port the convenience functions only after their owning lower-level APIs exist.
Prefer free functions or extension traits with explicit client references. Do
not use this crate to bypass ownership, cancellation, or error handling in the
core crate.
### 5.11 Voice crates
For `libremetaverse-voice-vivox`:
- port XML/control definitions, account/session/participant models, TCP pipe,
connector, gateway, and manager as a protocol adapter;
- keep blocking compatibility APIs as thin wrappers in the program layer, not
library methods that create runtimes;
- Vivox SDK/server availability and licensing are external prerequisites. The
crate can be protocol-complete without shipping proprietary native binaries.
For `libremetaverse-voice-webrtc`:
- port signaling messages and data-channel framing before peer/media handling;
- adapt `webrtc`, `cpal`, and `opus` behind local traits so protocol tests do not
require audio hardware;
- make device selection, sample format/rate/channel conversion, mute, reconnect,
ICE/SDP negotiation, and teardown explicit state transitions;
- CI uses virtual/fake audio; real-device and live voice validation is a manual
gate on macOS, Linux, and Windows.
Completion gate: recorded signaling/data-channel fixtures pass offline, fake
audio round-trips have bounded latency/loss behavior, teardown leaves no tasks,
and opt-in live validation succeeds without logging tokens.
## 6. Test-first migration workflow
The compatibility suite is deliberately red in the shell stage. First finish
the complete signature gate in section 3.1 across all crates. Then translate the
complete suite in section 3.2 across all crates. Only after both global gates
may implementation proceed in dependency order.
For each generated test:
1. Open the exact C# source/line and verify its method-body SHA-256 still matches
`tests/upstream-tests.json`.
2. Identify setup, action, assertions, parameterized cases, fixtures, categories,
and intended exception/event/timeout behavior.
3. Replace only that Rust `pending(...)` body with semantically equivalent Rust.
Keep the generated Rust test name or update the ledger in the same change.
4. Port literal bytes, UUIDs, timestamps, tolerances, culture assumptions, and
ordering exactly. Do not weaken an assertion merely because the shim differs.
5. If the needed API is absent, stop and treat that as a signature-coverage
defect: add the missing catalog/mapping entry and final failing signature,
then rerun the 100% API coverage gate.
6. Keep the test red at the standardized unimplemented member boundary; do not
implement behavior during the test-translation stage.
7. Mark the JSON entry with a future `status: semantic` field only after review;
enhance the generator/checker to preserve those reviewed entries rather than
overwrite them.
Test translation rules:
- NUnit `[TestCase]` invocations remain separate Rust tests so failures identify
the exact case.
- `Assert.Throws` maps to matching a specific Rust error variant, not any error.
- floating assertions retain the original tolerance and NaN behavior.
- async tests use paused/injected time where possible; no arbitrary sleeps.
- Moq interaction tests prefer a small fake recording calls; use Mockall only
when a trait contract has many independent expectations.
- `RequiresLiveServer` tests become `#[ignore]` and require explicit grid URL,
first/last name, and secret environment variables. They never run in normal CI.
- benchmark-category methods move to Criterion and are removed from correctness
pass/fail counts.
- malformed wire/asset tests become fuzz seeds after direct parity is passing.
Required gates:
```sh
cargo fmt --all -- --check
cargo check --workspace --all-targets
cargo test --workspace --no-run
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo doc --workspace --no-deps
```
During Stage 0, `cargo test --workspace` fails in generated parity placeholders.
During the full test-suite gate, `pending(` is the explicit remaining-work
count and CI prohibits increases. At the end of that gate, tests still fail,
but only because callable production signatures report standardized
not-implemented member IDs. During implementation, that failure inventory then
shrinks as behavior turns the fixed suite green.
## 7. Program and live-grid validation plan
The `programs` package contains these binary targets:
| Rust binary | C# project | Port purpose |
|---|---|---|
| `simple-bot` | `SimpleBot` | login, IM/chat events, movement and animations |
| `prim-inspector` | `PrimInspector` | object discovery/properties and transforms |
| `inventory-explorer` | `InventoryExplorer` | inventory fetch, search, stats and export |
| `packet-dump` | `PacketDump` | raw/decoded packet subscription and logging |
| `irc-gateway` | `IRCGateway` | concurrent external chat bridge |
| `test-client` | `TestClient` | full interactive client and all 128 command targets |
| `vivox-test` | `VivoxTest` | Vivox account/session/participant validation |
| `webrtc-test` | `WebRtcTest` | WebRTC signaling, devices and audio validation |
| `osd-inspector` | `OSDInspector` | offline LLSD inspect/convert/validate/primitive round trip |
Port `osd-inspector` first because it is offline and validates StructuredData.
Then port `simple-bot`, `packet-dump`, `prim-inspector`, and
`inventory-explorer` as vertical grid slices. Port TestClient commands by folder
(system/login first, then communication, inventory, objects, movement, land,
groups, friends, directory, appearance, voice, stats) while keeping
`commands::TEST_CLIENT_COMMANDS` as the completeness inventory. Voice programs
come last because they need native/device and service prerequisites.
Live validation must use a dedicated test account and an explicitly supplied
grid URI. Secrets are read from environment variables or the OS secret store,
never source/config committed to Git. A live smoke run records sanitized:
- server/grid/version and Rust commit;
- login and initial simulator/capabilities completion;
- IM/chat send/receive;
- movement/teleport/region crossing;
- inventory fetch and one reversible test-folder operation;
- object discovery/property request;
- asset/texture download and decode;
- clean logout and task/socket teardown.
Operations that spend currency, modify estate/parcel state, upload assets,
delete inventory, or affect other users require separate opt-in flags and are
not part of the default smoke test.
## 8. Ordered delivery milestones
1. **Structural inventory shell (current Stage 0):** crates, type/declaration
ledgers, failing test parity entries, program targets, license, and this plan
compile. This is not the callable API or semantic test suite.
2. **Full public API signature shim:** extract authoritative compiled metadata,
map 100% of public members to reviewed Rust signatures, and implement only
standardized failing bodies. All signatures and downstream compile fixtures
build before test translation begins.
3. **Full semantic test translation:** replace all 1,295 `pending` cases with
exact reviewed Rust equivalents using the public shim. The entire suite
compiles and fails only at standardized unimplemented member boundaries.
4. **Types implementation:** UUID, math, enums, conversions, collections,
cache/token bucket; turn only the already-existing Types tests green.
5. **StructuredData implementation:** OSD plus binary/notation/XML/JSON/protobuf
compatibility; turn only the already-existing StructuredData tests green.
6. **Imaging and PrimMesher implementation:** managed images, codec trait,
TGA/J2K adapter,
extrusion/sculpt/OBJ and rendering tests.
7. **Wire codegen:** deterministic Rust packet/visual/skeleton/tree/genepool/
attention generation and golden packet bytes.
8. **Networking/client lifecycle:** UDP, caps, login, simulators, settings,
cancellation, reconnect and logout against fake servers.
9. **World vertical slices:** agent, inventory, assets/appearance, objects,
parcels/terrain/environment, social/directory/groups, marketplace/estate.
10. **Rendering, RLV, LSL tools:** implement against their already-translated
suites and add public migration docs.
11. **Programs/live grid:** offline inspector, four core examples, TestClient,
then voice adapters and opt-in live/device validation.
12. **Hardening/release:** fuzz untrusted parsers, cross-platform native CI,
MSRV, docs/examples, semver/API audit, license/asset audit, performance
comparison against the separately run pinned C# implementation, and an
artifact/dependency audit proving there is no CLR, .NET assembly, subprocess,
RPC, or LibreMetaverse foreign-interface dependency.
Each milestone ends with zero unreviewed generated shims in its owned modules,
no weakened parity assertions, formatted/clippy-clean code, public Rust docs
that cite the corresponding C# concept, and a refreshed completeness report
from the checked-in ledgers.