Initial MetaCrate rewrite shell
This commit is contained in:
711
RUSTREWRITE.md
Normal file
711
RUSTREWRITE.md
Normal file
@@ -0,0 +1,711 @@
|
||||
# LibreMetaverse to Rust rewrite plan
|
||||
|
||||
## 1. Scope and source of truth
|
||||
|
||||
MetaCrate is a new native Rust implementation of the public API and observable
|
||||
behavior of LibreMetaverse. LibreMetaverse is reference material, not a runtime
|
||||
dependency or implementation component. The source snapshot used to create this
|
||||
shell is:
|
||||
|
||||
- repository: <https://github.com/cinderblocks/libremetaverse>
|
||||
- branch: `master`
|
||||
- commit: `2aa70bb68513b39795da5d13c88f31b86e85a3ba`
|
||||
- commit date: 2026-08-07
|
||||
- local checkout state during inventory: clean
|
||||
- license base: `libremetaverse/LICENSE.txt`, BSD 3-Clause
|
||||
|
||||
Generated inventories are intentionally checked in:
|
||||
|
||||
- `api/TYPES.tsv` maps every discovered public C# type declaration to a Rust
|
||||
crate/module/type and source line.
|
||||
- `api/SURFACE.tsv` records every public declaration line in the 13 library
|
||||
projects, including a SHA-256 digest.
|
||||
- `tests/upstream-tests.json` records every NUnit invocation and the SHA-256 of
|
||||
its C# method body.
|
||||
- `tests/PARITY.md` maps every NUnit invocation to its Rust test name.
|
||||
- `programs/upstream-programs.json` records every source file and digest for the
|
||||
nine sample/tool projects.
|
||||
|
||||
Run `python3 tools/generate_surface.py` after deliberately changing the pinned
|
||||
upstream snapshot. The generator rejects a test count other than 1,295 so an
|
||||
upstream drift cannot silently remove tests. Update the commit constant and
|
||||
review all ledger diffs in the same change.
|
||||
|
||||
### Current shell status
|
||||
|
||||
The current workspace is a structural baseline, not a working client:
|
||||
|
||||
- all 13 public .NET library projects have corresponding Rust crates;
|
||||
- 1,705 public C# type declarations produce compiling Rust type/trait shims;
|
||||
- 12,195 public declaration lines are retained in the API ledger;
|
||||
- all 1,295 NUnit `[Test]`/`[TestCase]` invocations have compiling Rust test
|
||||
entries that intentionally panic with their source identity and body hash;
|
||||
- all nine sample/tool projects have compiling Rust binary targets;
|
||||
- the 128 TestClient command source files are retained as a command inventory.
|
||||
|
||||
The generated type shims preserve names and module placement, but they do not
|
||||
yet expose callable C# methods, properties, fields, event payloads, enum values,
|
||||
or delegate signatures. `api/SURFACE.tsv` is an inventory aid, not proof of API
|
||||
coverage: it records declaration lines but does not fully parse multiline
|
||||
signatures. Likewise, a generated `pending` test is a parity placeholder, not a
|
||||
semantic translation. **This repository is therefore at Stage 0 and does not
|
||||
yet satisfy either the full API-shim gate or the full test-suite gate.**
|
||||
|
||||
No functional porting starts until both missing gates are complete across the
|
||||
entire workspace, rather than one crate at a time:
|
||||
|
||||
1. every public C# type and member has a reviewed, callable Rust signature with
|
||||
an intentionally failing body;
|
||||
2. every C# test is translated semantically and compiles while calling that
|
||||
Rust API;
|
||||
3. only then is behavior implemented, one crate at a time, until those already
|
||||
translated tests turn green.
|
||||
|
||||
### Full-rewrite constraint
|
||||
|
||||
The final project must be entirely implemented in new Rust code. The following
|
||||
are prohibited in production libraries, examples, tools, tests presented as
|
||||
Rust behavior, and release artifacts:
|
||||
|
||||
- loading or hosting the CLR/.NET runtime;
|
||||
- P/Invoke, COM, C ABI, UniFFI, generated bindings, or any other foreign-function
|
||||
bridge to LibreMetaverse assemblies;
|
||||
- invoking `dotnet`, LibreMetaverse programs, or helper services as a subprocess
|
||||
to implement Rust API behavior;
|
||||
- embedding, redistributing, dynamically loading, or downloading compiled
|
||||
LibreMetaverse assemblies;
|
||||
- forwarding calls over IPC/RPC to a running C# implementation;
|
||||
- retaining a C# fallback path for APIs not yet implemented in Rust;
|
||||
- treating an interoperability comparison harness as the implementation.
|
||||
|
||||
During development, the pinned C# source and compiled metadata may be inspected
|
||||
to inventory public signatures, understand algorithms and state transitions,
|
||||
and create golden inputs/outputs. The original C# tests and programs may be run
|
||||
separately as an oracle to produce or compare fixtures. Those development tools
|
||||
must remain outside the Rust runtime graph, must not be required to build or use
|
||||
published Rust crates, and must never satisfy a Rust compatibility test by
|
||||
executing the C# implementation.
|
||||
|
||||
Third-party native libraries such as OpenJPEG, Skia, libopus, operating-system
|
||||
APIs, and ordinary Rust crates are allowed where documented and license-
|
||||
compatible; the prohibition is specifically against reusing LibreMetaverse or
|
||||
another port of its implementation behind a foreign interface. All
|
||||
LibreMetaverse-specific protocol logic, models, state machines, managers,
|
||||
serialization rules, and public behavior must be reimplemented in Rust.
|
||||
|
||||
The final release gate must verify that published crate metadata and built
|
||||
artifacts contain no dependency on `dotnet`, Mono, CLR hosting APIs, C#
|
||||
assemblies, or LibreMetaverse bridge libraries. A clean machine with only the
|
||||
documented Rust/native prerequisites must be able to build, test, document, and
|
||||
run the Rust implementation.
|
||||
|
||||
## 2. Workspace design
|
||||
|
||||
| Rust crate | C# project | C# files | C# lines | Public declaration rows | Responsibility |
|
||||
|---|---|---:|---:|---:|---|
|
||||
| `libremetaverse-types` | `LibreMetaverse.Types` | 22 | 9,292 | 703 | UUIDs, math types, enums, caches and small collections |
|
||||
| `libremetaverse-structured-data` | `LibreMetaverse.StructuredData` | 10 | 4,069 | 303 | OSD value model and LLSD encodings |
|
||||
| `libremetaverse-imaging` | `LibreMetaverse.Imaging.Abstractions` | 2 | 454 | 18 | codec traits and managed image representation |
|
||||
| `libremetaverse-imaging-skia` | `LibreMetaverse.Imaging.Skia` | 1 | 369 | 3 | optional Skia codec adapter |
|
||||
| `libremetaverse-prim-mesher` | `LibreMetaverse.PrimMesher` | 5 | 3,477 | 222 | prim/sculpt mesh generation |
|
||||
| `libremetaverse` | `LibreMetaverse` | 210 | 138,916 | 8,982 | client, protocol, managers, assets and rendering model |
|
||||
| `libremetaverse-rendering-simple` | `LibreMetaverse.Rendering.Simple` | 1 | 161 | 6 | simple renderer |
|
||||
| `libremetaverse-rendering-mesh-foundry` | `LibreMetaverse.Rendering.MeshFoundry` | 1 | 852 | 14 | full mesh renderer |
|
||||
| `libremetaverse-lsl-tools` | `LibreMetaverse.LslTools` | 175 | 27,856 | 962 | LSL lexer/parser generator and generated grammar |
|
||||
| `libremetaverse-rlv` | `LibreMetaverse.RLV` | 32 | 6,296 | 262 | RLV parsing, restrictions, locks and callbacks |
|
||||
| `libremetaverse-utilities` | `LibreMetaverse.Utilities` | 1 | 289 | 12 | high-level convenience helpers |
|
||||
| `libremetaverse-voice-vivox` | `LibreMetaverse.Voice.Vivox` | 12 | 4,544 | 440 | Vivox control protocol adapter |
|
||||
| `libremetaverse-voice-webrtc` | `LibreMetaverse.Voice.WebRTC` | 11 | 7,023 | 268 | WebRTC voice adapter |
|
||||
|
||||
`libremetaverse` is the umbrella client crate. It re-exports the foundational
|
||||
crates under `types`, `structured_data`, and `imaging_abstractions`. Rendering,
|
||||
voice, RLV, LSL tools, and native Skia remain separate so users do not pay for
|
||||
large or platform-specific dependency trees unless requested.
|
||||
|
||||
The workspace uses Rust 2024 with MSRV 1.85. Generated code is committed so
|
||||
ordinary consumers do not need Python, .NET, Roslyn, or the upstream checkout.
|
||||
|
||||
## 3. Public API translation policy
|
||||
|
||||
Compatibility means the same concepts, wire behavior, state transitions,
|
||||
errors, and event payloads. It does not mean mechanically reproducing C# syntax.
|
||||
Apply these rules consistently:
|
||||
|
||||
| C# surface | Rust surface |
|
||||
|---|---|
|
||||
| `LibreMetaverse.Foo.Bar` | the owning crate plus `foo::Bar`; omit the repeated assembly namespace |
|
||||
| PascalCase methods | `snake_case`; add a deprecated PascalCase forwarding method only when migration evidence justifies it |
|
||||
| `Task<T>` / `Task` | `async fn -> Result<T, Error>` / `async fn -> Result<(), Error>` |
|
||||
| `Async` method suffix | omit it when the Rust method is `async`; document the C# source name |
|
||||
| `CancellationToken` | `tokio_util::sync::CancellationToken` at long-lived operation boundaries |
|
||||
| nullable reference | `Option<T>`; never use a sentinel UUID/string unless the protocol defines one |
|
||||
| exception hierarchy | small non-exhaustive `thiserror` enums per crate; retain source error and protocol context |
|
||||
| public mutable property | private field plus validated getter/setter, or a public field only for inert value records |
|
||||
| overloads | one method with typed parameter structs; use separate descriptive names when operations differ semantically |
|
||||
| interface | object-safe trait when dynamic dispatch is required, otherwise a generic trait bound |
|
||||
| delegate | `Fn`, `FnMut`, or `FnOnce` bounds; boxed callback only when callbacks are stored |
|
||||
| .NET event | typed subscription returning a guard, or a `tokio::sync::broadcast` receiver for asynchronous fan-out |
|
||||
| `IDisposable` | ownership and `Drop`; add explicit `close`/`shutdown().await` when failure or async work must be observed |
|
||||
| `byte[]` | borrowed `&[u8]` for parsing, `bytes::Bytes` for shared network payloads, `Vec<u8>` for owned mutation |
|
||||
| `IReadOnlyList<T>` | `&[T]` or `Arc<[T]>`; return iterators when callers do not need storage |
|
||||
| `Dictionary` | `HashMap`; use `BTreeMap` only when deterministic ordering is observable |
|
||||
| immutable collections | owned `Vec`/`HashMap` or `Arc<[T]>`; do not add a persistent-collection crate without a measured need |
|
||||
| monitor/reader-writer lock | `std::sync` first; use `parking_lot` only after profiling or poisoning-policy review |
|
||||
| generated flags enum | `bitflags`; preserve unknown bits with `from_bits_retain` |
|
||||
| C# enum on the wire | `#[repr(...)]` plus checked conversion; include `Unknown(raw)` when the protocol is extensible |
|
||||
| `UUID` | a compatibility newtype around `uuid::Uuid`, with explicit big/little-endian wire helpers |
|
||||
| vectors/quaternions/matrices | compatibility structs with LibreMetaverse field/order semantics; use `glam` internally after parity tests prove conversions |
|
||||
| dependency injection container | normal constructors and explicit trait parameters; add no service-locator equivalent |
|
||||
|
||||
Public structs and enums that come from the protocol must be `#[non_exhaustive]`
|
||||
when servers may add values. Do not derive `Serialize`/`Deserialize` as a
|
||||
substitute for the specified wire encoding: LLUDP, LLSD, MessagePack, asset
|
||||
formats, and login XML each require their own compatibility tests.
|
||||
|
||||
### 3.1 Full signature-shim gate
|
||||
|
||||
The first handover task after this structural shell is a complete callable API
|
||||
shim. Do not start real behavior while completing it.
|
||||
|
||||
Build the authoritative API catalog during development from compiled .NET
|
||||
metadata after running the original source generators, not from the line-
|
||||
oriented regex ledger alone. This extraction is an inventory tool only and must
|
||||
not become a build-time or runtime dependency of the Rust crates.
|
||||
Build one supported target framework into a temporary output directory and use
|
||||
`System.Reflection.Metadata` or Roslyn symbols to emit machine-readable records
|
||||
for every public/nested-public:
|
||||
|
||||
- class, struct, record, interface, enum and delegate, including generic
|
||||
parameters and constraints;
|
||||
- constructor and method, including overloads, ref/out/in parameters, optional
|
||||
values and return type;
|
||||
- property/indexer and getter/setter visibility;
|
||||
- field and constant, including value and numeric width;
|
||||
- event and delegate payload;
|
||||
- base type, implemented interface and relevant public attribute.
|
||||
|
||||
Create `api/public-api.json` as that catalog and `api/RUST-MAPPING.tsv` with one
|
||||
row per C# member: stable C# documentation ID, C# signature, Rust crate/path,
|
||||
Rust signature, mapping decision, and status. A coverage checker must fail for
|
||||
missing members, duplicate Rust destinations, unresolved types, or stale source
|
||||
IDs. `api/SURFACE.tsv` remains useful for source navigation but is not the
|
||||
coverage authority.
|
||||
|
||||
Each mapped Rust item must actually compile and be callable:
|
||||
|
||||
- methods, constructors and property accessors have their final reviewed Rust
|
||||
names, parameters, ownership/borrowing, asyncness, and return types;
|
||||
- overloads use stable descriptive Rust names or typed argument structs, with
|
||||
the mapping recorded explicitly;
|
||||
- enums contain their real variants/discriminants and flags contain their real
|
||||
bits, because tests and downstream code must compile against them;
|
||||
- event payloads, delegates and interfaces have their real fields/trait methods;
|
||||
- generic bounds and thread-safety guarantees are deliberate, not erased to an
|
||||
untyped placeholder.
|
||||
|
||||
The body is the only part allowed to be fake. Fallible Rust APIs return a shared
|
||||
typed `NotImplemented { csharp_member }` error. Infallible constructors,
|
||||
accessors, operators and trait methods call a shared `unimplemented_api!`
|
||||
placeholder that panics with the stable C# member ID. Constants and enum values
|
||||
must have their real values. Never return plausible defaults such as `false`,
|
||||
zero, an empty collection, or a nil UUID: those could make translated tests pass
|
||||
for the wrong reason.
|
||||
|
||||
The signature gate is complete only when:
|
||||
|
||||
- the metadata catalog reports 100% public type/member mapping coverage for all
|
||||
13 library assemblies, including generated APIs;
|
||||
- all crates build and `cargo doc` resolves every public signature;
|
||||
- a downstream compile fixture can name/call every mapped Rust item;
|
||||
- there are no erased `ShimValue`, dynamic argument bags, invented variadics,
|
||||
or member signatures present only as ledger text;
|
||||
- implementation bodies contain only the standardized failure placeholder,
|
||||
constants, enum discriminants, and inert data-layout boilerplate.
|
||||
|
||||
### 3.2 Full semantic test-suite gate
|
||||
|
||||
After the full signature shim compiles, translate the entire C# test suite while
|
||||
leaving all library bodies unimplemented. Every one of the 1,295 NUnit
|
||||
invocations must become a Rust test with the same inputs, setup, operation,
|
||||
observations, assertions, tolerances and expected error/event behavior. Shared
|
||||
C# test helpers become shared Rust test helpers; embedded fixtures and literal
|
||||
payloads are copied with license/source attribution and byte hashes.
|
||||
|
||||
The Rust tests must call the public APIs rather than internal replacements.
|
||||
Where the C# tests call internal members through friend-assembly access, record
|
||||
that fact and place equivalent Rust unit tests inside the owning crate without
|
||||
making the member public solely for testing.
|
||||
|
||||
The tests must compile even though running them is red. An expected C# exception
|
||||
maps to a specific Rust `Err` variant, never to `should_panic` around a public
|
||||
API call; this ensures an `unimplemented_api!` panic remains a failure rather
|
||||
than accidentally satisfying the test. Only tests whose C# purpose is genuinely
|
||||
to verify a thrown runtime invariant may use `catch_unwind`/`should_panic`, and
|
||||
they must distinguish the intended invariant from the placeholder panic.
|
||||
|
||||
The test-suite gate is complete only when:
|
||||
|
||||
- `pending(` occurs zero times in Rust test sources;
|
||||
- the parity catalog reports exactly one reviewed Rust case for every upstream
|
||||
`[Test]`/`[TestCase]`, with the matching C# body hash;
|
||||
- `cargo test --workspace --no-run` compiles all tests against public signatures;
|
||||
- a controlled test run fails at standardized unimplemented member IDs rather
|
||||
than from missing symbols, compile errors, malformed fixtures, or weakened
|
||||
assertions;
|
||||
- live-server/device tests are faithfully translated but `#[ignore]` by default
|
||||
with documented opt-in configuration;
|
||||
- no production method contains real behavior beyond what is required to make
|
||||
signatures and constants compile.
|
||||
|
||||
## 4. Validated Rust dependency map
|
||||
|
||||
Versions below were queried from crates.io on 2026-08-08 with `cargo search`
|
||||
and, for risky/native crates, `cargo info`. They are candidates, not blanket
|
||||
dependencies. Add one to a crate only when its first implementation uses it,
|
||||
then commit `Cargo.lock` for application/test reproducibility. Recheck versions,
|
||||
MSRV, features, and licenses at adoption time.
|
||||
|
||||
| .NET/NuGet responsibility | Rust candidate and validated version | Adoption notes |
|
||||
|---|---|---|
|
||||
| async runtime, channels, timers | [`tokio` 1.53.1](https://crates.io/crates/tokio/1.53.1), [`tokio-util` 0.7.19](https://crates.io/crates/tokio-util/0.7.19), [`futures` 0.3.33](https://crates.io/crates/futures/0.3.33) | One Tokio runtime owned by the application; libraries never create nested runtimes. |
|
||||
| `System.Net.Http` / WinHTTP | [`reqwest` 0.13.4](https://crates.io/crates/reqwest/0.13.4) | MSRV 1.85. Use rustls/default TLS deliberately, connection pooling, streaming bodies, and injected client/timeouts. |
|
||||
| JSON and general data mapping | [`serde` 1.0.229](https://crates.io/crates/serde/1.0.229), [`serde_json` 1.0.151](https://crates.io/crates/serde_json/1.0.151), [`serde_bytes` 0.11.19](https://crates.io/crates/serde_bytes/0.11.19) | Use only where the C# implementation is schema-driven JSON. Hand-write LLSD encodings. |
|
||||
| MessagePack 3.1.8 | [`rmp-serde` 1.3.1](https://crates.io/crates/rmp-serde/1.3.1), [`rmpv` 1.3.1](https://crates.io/crates/rmpv/1.3.1) | Golden byte vectors must prove integer widths, maps, extensions, and field ordering. |
|
||||
| UUID | [`uuid` 1.24.0](https://crates.io/crates/uuid/1.24.0) | Wrap it; do not expose dependency-specific endian assumptions as protocol behavior. |
|
||||
| byte buffers | [`bytes` 1.12.1](https://crates.io/crates/bytes/1.12.1), [`bytemuck` 1.25.2](https://crates.io/crates/bytemuck/1.25.2) | Prefer explicit endian reads. `bytemuck` is only for proven POD layouts, never untrusted variable packets. |
|
||||
| vector/matrix math | [`glam` 0.33.3](https://crates.io/crates/glam/0.33.3) | Keep compatibility newtypes to control component order, precision, equality, and serialization. |
|
||||
| flags | [`bitflags` 2.13.1](https://crates.io/crates/bitflags/2.13.1) | Retain unknown bits from newer grids. |
|
||||
| errors | [`thiserror` 2.0.20](https://crates.io/crates/thiserror/2.0.20) | Application binaries may use richer reporting; public libraries return typed errors. |
|
||||
| ZLogger / Microsoft logging | [`tracing` 0.1.44](https://crates.io/crates/tracing/0.1.44), [`tracing-subscriber` 0.3.23](https://crates.io/crates/tracing-subscriber/0.3.23) | Libraries emit spans/events; binaries choose formatting and filters. Never log credentials or capability tokens. |
|
||||
| rate limiting | [`governor` 0.10.4](https://crates.io/crates/governor/0.10.4) | Candidate for caps categories. First reproduce burst/refill/cancellation behavior with deterministic clock tests. |
|
||||
| compression and tar archives | [`flate2` 1.1.9](https://crates.io/crates/flate2/1.1.9), [`tar` 0.4.46](https://crates.io/crates/tar/0.4.46) | Enforce decompressed-size, path traversal, and entry-count limits on untrusted OAR data. |
|
||||
| XML and URL | [`quick-xml` 0.41.0](https://crates.io/crates/quick-xml/0.41.0), [`url` 2.5.8](https://crates.io/crates/url/2.5.8), [`base64` 0.23.1](https://crates.io/crates/base64/0.23.1) | Streaming XML for LLSD/login; retain exact URL escaping behavior with fixtures. |
|
||||
| CoreJ2K 2.3.3.91 | [`jpeg2k` 0.10.1](https://crates.io/crates/jpeg2k/0.10.1) | Defaults to OpenJPEG/native bindings. Keep behind a codec trait and feature; test channel order, alpha, dimensions, discard levels, and malformed input. |
|
||||
| SkiaSharp 4.150.1 | [`skia-safe` 0.99.0](https://crates.io/crates/skia-safe/0.99.0) | MSRV 1.85 and native/binary-cache build. Optional adapter only; core image APIs must not leak Skia types. |
|
||||
| Pfim 0.11.4 | [`image` 0.25.10](https://crates.io/crates/image/0.25.10), [`ddsfile` 0.6.0](https://crates.io/crates/ddsfile/0.6.0) | `image` covers TGA and common DDS decoding; `ddsfile` exposes DDS container details. Golden files decide whether both are needed. |
|
||||
| OggVorbisEncoder 1.2.2 | [`vorbis_rs` 0.5.6](https://crates.io/crates/vorbis_rs/0.5.6) | BSD-3-Clause, MSRV 1.82, backed by C libraries. Feature-gate native audio encoding. |
|
||||
| SIPSorcery 8.0.23 | [`webrtc` 0.20.0](https://crates.io/crates/webrtc/0.20.0), [`cpal` 0.18.1](https://crates.io/crates/cpal/0.18.1), [`opus` 0.3.1](https://crates.io/crates/opus/0.3.1) | Validate SDP, ICE, data-channel framing, audio formats, device hotplug, and native libopus deployment separately. Do not claim parity from successful compilation. |
|
||||
| LSL generated parser | [`lalrpop` 0.23.1](https://crates.io/crates/lalrpop/0.23.1) | Candidate only. Preserve grammar conflicts, recovery, token positions, and diagnostics before replacing the generated parser. |
|
||||
| NUnit/Moq | built-in test harness, [`mockall` 0.15.0](https://crates.io/crates/mockall/0.15.0), [`proptest` 1.11.0](https://crates.io/crates/proptest/1.11.0) | Prefer fakes and deterministic protocol fixtures; use mocks only for interaction contracts. Add properties after direct parity cases exist. |
|
||||
| NUnit benchmarks | [`criterion` 0.8.2](https://crates.io/crates/criterion/0.8.2) | Port benchmark-category methods to `benches/`; never make timing thresholds correctness tests. |
|
||||
| sample command lines | [`clap` 4.6.6](https://crates.io/crates/clap/4.6.6) | Add when implementing the first real program CLI; keep credentials in arguments/env only long enough to authenticate. |
|
||||
|
||||
`System.*`, BCL compatibility packages, immutable collections, DI abstractions,
|
||||
and object pooling usually map to the Rust standard library and ownership. Do
|
||||
not select a crate merely because NuGet used one. In particular, begin with
|
||||
`Mutex<HashMap<...>>` rather than a concurrent-map dependency and add a pool
|
||||
only after allocations appear in a profile.
|
||||
|
||||
## 5. Module-by-module implementation guidance
|
||||
|
||||
### 5.1 `libremetaverse-types`
|
||||
|
||||
This is the first implementation crate because nearly every protocol and test
|
||||
depends on it.
|
||||
|
||||
- **UUID:** wrap `uuid::Uuid`; implement zero/random/parse/format, byte-array
|
||||
constructors, CRC/combine helpers, ordering, hashing, and every endian path
|
||||
from `UUID.cs`. Keep the `UUID` name as a public compatibility type.
|
||||
- **Vector2/3/3d/4, Quaternion, Matrix4, Ray, Color4:** preserve constructors,
|
||||
constants, component order, normalization of zero values, quaternion
|
||||
multiplication direction, approximate-comparison helpers, and binary/string
|
||||
formats. Use `glam` internally only after conversion tests pass.
|
||||
- **Enums and primitive enums:** translate exact numeric discriminants and flag
|
||||
widths. Use `bitflags` where C# uses `[Flags]`; retain unknown values/bits.
|
||||
- **CRC32 and conversions:** port byte-for-byte with fixed golden vectors and
|
||||
explicit endianness. Avoid platform-sized integers on the wire.
|
||||
- **CacheDictionary, ExpiringCache, TokenBucket:** inject a clock in tests;
|
||||
preserve expiry/burst semantics. Start with `HashMap` plus one lock.
|
||||
- **DoubleDictionary and MultiValueDictionary:** expose iterators and borrowing
|
||||
rather than cloning collections. Preserve duplicate and replacement rules.
|
||||
|
||||
Completion gate: every Types-related parity test is a semantic Rust test, all
|
||||
format/byte fixtures match C#, and no generated shim remains for a Types API.
|
||||
|
||||
### 5.2 `libremetaverse-structured-data`
|
||||
|
||||
- **OSD value model:** implement a non-exhaustive enum for undefined, boolean,
|
||||
integer, real, string, UUID, date, URI, binary, array, and map. Preserve C#
|
||||
conversion/default behavior deliberately; do not let `serde_json::Value`
|
||||
define the public model.
|
||||
- **OSDArray/OSDMap:** wrap `Vec<OSD>` and an order-appropriate map. Verify
|
||||
whether serialized map ordering is observable in each format before choosing
|
||||
`HashMap` versus `BTreeMap`/insertion order.
|
||||
- **Binary LLSD:** streaming parser over `&[u8]`/`Read`; checked lengths,
|
||||
recursion limits, exact endian rules, and no panics on malformed input.
|
||||
- **Notation LLSD:** preserve escaping, length-prefixed strings/binary, numbers,
|
||||
dates, URIs, whitespace, and error positions.
|
||||
- **XML LLSD:** use `quick-xml`; reject/entity-limit hostile input and retain
|
||||
exact element mapping.
|
||||
- **JSON LLSD:** use `serde_json` behind explicit OSD conversions so UUID/date/
|
||||
URI/binary tagging remains compatible.
|
||||
- **Protobuf OSD:** first identify whether this is a stable external format. If
|
||||
stable, preserve tags with generated Rust types; otherwise keep it internal.
|
||||
|
||||
Completion gate: parse and emit C# golden bytes/text for every format, including
|
||||
malformed, deep, empty, Unicode, NaN/infinity, and cross-format cases.
|
||||
|
||||
### 5.3 `libremetaverse-imaging`
|
||||
|
||||
- Translate `ManagedImage` as an owned image buffer with explicit dimensions,
|
||||
channel flags/layout, stride, and checked size arithmetic.
|
||||
- Translate `ITextureCodec` to an object-safe `TextureCodec` trait only if
|
||||
runtime codec selection is needed. Codec methods return typed decode/encode
|
||||
errors and accept limits.
|
||||
- Keep public APIs independent of Skia/OpenJPEG types so pure-Rust or platform
|
||||
codecs can be substituted.
|
||||
|
||||
Completion gate: channel interleave/deinterleave, resize, alpha/bump behavior,
|
||||
and invalid-dimension tests match the original.
|
||||
|
||||
### 5.4 `libremetaverse-imaging-skia`
|
||||
|
||||
Implement `SkiaTextureCodec` as an optional adapter with `skia-safe`. Convert at
|
||||
the boundary to/from `ManagedImage`; never expose `skia_safe` types in the core
|
||||
trait. CI must cover macOS, Linux, and Windows or explicitly document unsupported
|
||||
targets. Validate native binary provenance and packaging.
|
||||
|
||||
### 5.5 `libremetaverse-prim-mesher`
|
||||
|
||||
- Port `Coord`, `Quat`, faces, paths, profiles, viewer faces, and indexing as
|
||||
value types with checked indices.
|
||||
- Port linear/circular extrusion before sculpt meshing. Preserve winding,
|
||||
normals, UVs, face numbering, hollow/profile cuts, twists, tapers, and shears.
|
||||
- Port `SculptMap` decoding through `libremetaverse-imaging`, then `SculptMesh`.
|
||||
- Make OBJ export a formatting layer over mesh data; deterministic output is a
|
||||
testable contract.
|
||||
|
||||
Completion gate: all 48 PrimMesher rendering-test invocations and all
|
||||
MeshFoundry tests are semantic ports, face indices stay in range, and C# versus
|
||||
Rust mesh fixtures compare within the original tolerances.
|
||||
|
||||
### 5.6 `libremetaverse` protocol and client core
|
||||
|
||||
Implement the central crate in dependency slices, not file order.
|
||||
|
||||
#### Wire protocol and generated packets
|
||||
|
||||
- Port `BitPack`, zero coding, packet headers, ACK handling, sequence numbers,
|
||||
fragmentation, throttles, and `UDPPacketBuffer` first.
|
||||
- Replace the six Roslyn source generators with a Rust `codegen` tool/build step
|
||||
reading the same `data/` and `linden/` inputs. Commit generated packet,
|
||||
visual-parameter, skeleton, tree, genepool, and attention Rust sources.
|
||||
- The generator must produce deterministic output; CI regenerates to a temporary
|
||||
directory and fails on diff. Generated packet tests compare exact bytes and
|
||||
IDs against C# fixtures.
|
||||
- Parsers operate on borrowed slices where practical and reject truncated,
|
||||
oversized, or unknown blocks without unchecked indexing.
|
||||
|
||||
#### Networking and capabilities
|
||||
|
||||
- `UDPBase`, `NetworkManager`, `Simulator`, `ProtocolManager`, and `UdpThrottle`
|
||||
use Tokio UDP tasks with explicit ownership and cancellation. One task owns
|
||||
socket writes; decoded events cross bounded channels to prevent unbounded
|
||||
memory growth.
|
||||
- `Caps`, `HttpCapsClient`, event queue, caps-to-packet, and rate limiting use an
|
||||
injected `reqwest::Client`. Preserve redirects, content types, retry policy,
|
||||
cancellation, progress reporting, and non-HTTP location rejection.
|
||||
- Login/XML-RPC, seed capabilities, region crossing, reconnect, logout, and
|
||||
timeout state transitions must be modeled explicitly and tested with local
|
||||
fake servers before live-grid use.
|
||||
- Capability URLs/tokens and login credentials are secrets: redact them from
|
||||
`Debug` and logs.
|
||||
|
||||
#### Grid client and service composition
|
||||
|
||||
- `GridClient` owns managers and shared runtime state. Construct it with a
|
||||
`GridClientBuilder` only for real optional policies/codecs; required services
|
||||
remain constructor arguments.
|
||||
- Replace `IGridClient` and service-collection extensions with focused traits at
|
||||
test seams. Avoid a general service locator.
|
||||
- `Settings/*` become typed configuration structs grouped like the C# settings,
|
||||
with validated durations/sizes and defaults proven by tests.
|
||||
- `Logger` maps to `tracing`; `UtilizationStatistics` exports snapshots without
|
||||
making a metrics backend mandatory.
|
||||
|
||||
#### Agent, movement, chat, effects, teleport and money
|
||||
|
||||
- Port `AgentManager` partial files as modules under `agent`, but expose one
|
||||
coherent `AgentManager` API. Keep camera/movement state separate from network
|
||||
commands.
|
||||
- Event-argument classes become event structs. Subscriptions must be removable
|
||||
and must not retain the whole client after drop.
|
||||
- Teleport and region crossing are explicit state machines with timeout,
|
||||
cancellation, progress, failure reason, and simulator handoff tests.
|
||||
- Money and permissions paths use fixed integer widths and never log sensitive
|
||||
transaction descriptions without opt-in.
|
||||
|
||||
#### Inventory
|
||||
|
||||
- Port inventory enums/base records/nodes/store before `InventoryManager`.
|
||||
Preserve link resolution, folder/item distinctions, ownership and permissions.
|
||||
- Separate local tree/cache mutation from UDP/capability transport. This makes
|
||||
`InventoryAISClient` independently testable with recorded LLSD fixtures.
|
||||
- Async fetch, give, rez, task inventory, current outfit folder, and callbacks
|
||||
must have cancellation and deduplication tests. Do not hold locks across
|
||||
`.await`.
|
||||
- Persisted cache format needs versioning, atomic replacement, corruption
|
||||
handling, and bounded deserialization.
|
||||
|
||||
#### Assets, appearance, avatar and animesh
|
||||
|
||||
- Port the base `Asset` model and individual asset types as typed wrappers over
|
||||
validated bytes. Decode lazily where the C# behavior permits.
|
||||
- `AssetManager`, `AssetCache`, `DownloadManager`, and `TexturePipeline` share a
|
||||
deduplicating request layer with bounded concurrency and cancellation.
|
||||
- Port wearables, baking, current-outfit policies, texture compositing, avatar
|
||||
definitions, skeleton, visual params, animations, gestures, and animesh in
|
||||
that order. Copy and retain the upstream Linden assets only after reviewing
|
||||
their separate `cc-by-sa-3.0.txt` obligations.
|
||||
- Animation interpolation, joint transforms, skinning, attachment rigs, and
|
||||
physics must use the original test tolerances and matrix multiplication order.
|
||||
|
||||
#### Objects, parcels, terrain, environment and rendering model
|
||||
|
||||
- Port `Primitive`, texture entry, media, particles, permissions, materials,
|
||||
and simulator collections before `ObjectManager`.
|
||||
- Object updates need an explicit decode/apply split; preserve terse/compressed/
|
||||
full update semantics, stale-update behavior, and parent-child linking.
|
||||
- Parcel, estate, grid, terrain, sound, environment, interest-list, directory,
|
||||
friends, groups, marketplace, and experience managers each get a transport-
|
||||
independent state/model layer plus network handlers.
|
||||
- Terrain codecs and GLTF/material handling use golden fixtures and size limits.
|
||||
GLTF JSON may use Serde, but binary buffers/accessors require checked offset
|
||||
arithmetic.
|
||||
|
||||
#### Archives and import/export
|
||||
|
||||
- `OarFile`, tar reader/writer, region settings, and asset archiver use `tar` and
|
||||
`flate2` with path normalization, extraction-root enforcement, and size/count
|
||||
limits.
|
||||
- Collada/OBJ/GLTF import/export keeps format DTOs separate from world objects.
|
||||
Deterministic output and round-trip fixtures are required; live upload is a
|
||||
later layer.
|
||||
|
||||
#### Threading and utilities
|
||||
|
||||
- Replace custom events/semaphores/read-write locks with Tokio or `std::sync`
|
||||
primitives where semantics match. Port custom optimistic/spin locks only if a
|
||||
benchmark and correctness test show they are still required.
|
||||
- Replace array pools with `Vec`/`Bytes` reuse first. Add pooling only after
|
||||
profiling and with a bounded pool.
|
||||
- Observable/event dictionaries expose change streams without executing user
|
||||
callbacks while holding internal locks.
|
||||
|
||||
Completion gate for the core crate: no public core shim remains; all offline
|
||||
parity tests pass; malformed-input fuzz targets do not panic; fake-server login,
|
||||
caps, UDP, reconnect, and cancellation scenarios pass on all supported OSes.
|
||||
|
||||
### 5.7 Rendering crates
|
||||
|
||||
`libremetaverse-rendering-simple` ports `SimpleRenderer` first as the reference
|
||||
implementation of the core rendering trait. It should favor clarity and serve
|
||||
as a fixture generator.
|
||||
|
||||
`libremetaverse-rendering-mesh-foundry` then ports the complete MeshFoundry
|
||||
pipeline using types, OSD, core assets, PrimMesher, and imaging. Preserve face
|
||||
grouping, materials, transforms, skin weights, normals/tangents, texture
|
||||
coordinates, and output ordering. Do not merge the two crates: users should be
|
||||
able to choose the small renderer without the full mesh pipeline.
|
||||
|
||||
### 5.8 `libremetaverse-lsl-tools`
|
||||
|
||||
- Treat `Tools/` as the parser-generator implementation and `YYClass/` as
|
||||
generated grammar output. Do not line-by-line port all generated classes.
|
||||
- First translate lexer tokens, source locations, errors, comments, precedence,
|
||||
grammar productions, and recovery behavior into compatibility tests.
|
||||
- Re-express the grammar in LALRPOP only if it reproduces accepted/rejected
|
||||
programs and diagnostic locations. Otherwise port the existing table machine
|
||||
with generated Rust tables.
|
||||
- Generated tables are committed and deterministically reproducible. Keep public
|
||||
token/parser entry points recognizable for migration, while Rust iterators and
|
||||
`Result` replace C# enumerators/exceptions.
|
||||
|
||||
Completion gate: all grammar fixtures and error-recovery cases match, and the
|
||||
generated code is warning-free without 150 hand-maintained class shims.
|
||||
|
||||
### 5.9 `libremetaverse-rlv`
|
||||
|
||||
- Port message/command parsing and enums first, preserving case, separators,
|
||||
attachment point aliases, option handling, and invalid-command errors.
|
||||
- Port blacklist/provider/callback interfaces as narrow traits. Return typed
|
||||
actions from pure parsing/decision code; execute viewer actions through an
|
||||
injected callback adapter.
|
||||
- Port restriction state, camera settings, shared folders, inventory maps,
|
||||
locked folders, attachment requests, permissions, and service orchestration.
|
||||
- Use immutable snapshots or short locks for restriction reads; callbacks run
|
||||
after locks are released.
|
||||
|
||||
Completion gate: every RLV command/query/restriction/exception test case is a
|
||||
semantic Rust test and all original expected error distinctions remain visible.
|
||||
|
||||
### 5.10 `libremetaverse-utilities`
|
||||
|
||||
Port the convenience functions only after their owning lower-level APIs exist.
|
||||
Prefer free functions or extension traits with explicit client references. Do
|
||||
not use this crate to bypass ownership, cancellation, or error handling in the
|
||||
core crate.
|
||||
|
||||
### 5.11 Voice crates
|
||||
|
||||
For `libremetaverse-voice-vivox`:
|
||||
|
||||
- port XML/control definitions, account/session/participant models, TCP pipe,
|
||||
connector, gateway, and manager as a protocol adapter;
|
||||
- keep blocking compatibility APIs as thin wrappers in the program layer, not
|
||||
library methods that create runtimes;
|
||||
- Vivox SDK/server availability and licensing are external prerequisites. The
|
||||
crate can be protocol-complete without shipping proprietary native binaries.
|
||||
|
||||
For `libremetaverse-voice-webrtc`:
|
||||
|
||||
- port signaling messages and data-channel framing before peer/media handling;
|
||||
- adapt `webrtc`, `cpal`, and `opus` behind local traits so protocol tests do not
|
||||
require audio hardware;
|
||||
- make device selection, sample format/rate/channel conversion, mute, reconnect,
|
||||
ICE/SDP negotiation, and teardown explicit state transitions;
|
||||
- CI uses virtual/fake audio; real-device and live voice validation is a manual
|
||||
gate on macOS, Linux, and Windows.
|
||||
|
||||
Completion gate: recorded signaling/data-channel fixtures pass offline, fake
|
||||
audio round-trips have bounded latency/loss behavior, teardown leaves no tasks,
|
||||
and opt-in live validation succeeds without logging tokens.
|
||||
|
||||
## 6. Test-first migration workflow
|
||||
|
||||
The compatibility suite is deliberately red in the shell stage. First finish
|
||||
the complete signature gate in section 3.1 across all crates. Then translate the
|
||||
complete suite in section 3.2 across all crates. Only after both global gates
|
||||
may implementation proceed in dependency order.
|
||||
|
||||
For each generated test:
|
||||
|
||||
1. Open the exact C# source/line and verify its method-body SHA-256 still matches
|
||||
`tests/upstream-tests.json`.
|
||||
2. Identify setup, action, assertions, parameterized cases, fixtures, categories,
|
||||
and intended exception/event/timeout behavior.
|
||||
3. Replace only that Rust `pending(...)` body with semantically equivalent Rust.
|
||||
Keep the generated Rust test name or update the ledger in the same change.
|
||||
4. Port literal bytes, UUIDs, timestamps, tolerances, culture assumptions, and
|
||||
ordering exactly. Do not weaken an assertion merely because the shim differs.
|
||||
5. If the needed API is absent, stop and treat that as a signature-coverage
|
||||
defect: add the missing catalog/mapping entry and final failing signature,
|
||||
then rerun the 100% API coverage gate.
|
||||
6. Keep the test red at the standardized unimplemented member boundary; do not
|
||||
implement behavior during the test-translation stage.
|
||||
7. Mark the JSON entry with a future `status: semantic` field only after review;
|
||||
enhance the generator/checker to preserve those reviewed entries rather than
|
||||
overwrite them.
|
||||
|
||||
Test translation rules:
|
||||
|
||||
- NUnit `[TestCase]` invocations remain separate Rust tests so failures identify
|
||||
the exact case.
|
||||
- `Assert.Throws` maps to matching a specific Rust error variant, not any error.
|
||||
- floating assertions retain the original tolerance and NaN behavior.
|
||||
- async tests use paused/injected time where possible; no arbitrary sleeps.
|
||||
- Moq interaction tests prefer a small fake recording calls; use Mockall only
|
||||
when a trait contract has many independent expectations.
|
||||
- `RequiresLiveServer` tests become `#[ignore]` and require explicit grid URL,
|
||||
first/last name, and secret environment variables. They never run in normal CI.
|
||||
- benchmark-category methods move to Criterion and are removed from correctness
|
||||
pass/fail counts.
|
||||
- malformed wire/asset tests become fuzz seeds after direct parity is passing.
|
||||
|
||||
Required gates:
|
||||
|
||||
```sh
|
||||
cargo fmt --all -- --check
|
||||
cargo check --workspace --all-targets
|
||||
cargo test --workspace --no-run
|
||||
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||
cargo doc --workspace --no-deps
|
||||
```
|
||||
|
||||
During Stage 0, `cargo test --workspace` fails in generated parity placeholders.
|
||||
During the full test-suite gate, `pending(` is the explicit remaining-work
|
||||
count and CI prohibits increases. At the end of that gate, tests still fail,
|
||||
but only because callable production signatures report standardized
|
||||
not-implemented member IDs. During implementation, that failure inventory then
|
||||
shrinks as behavior turns the fixed suite green.
|
||||
|
||||
## 7. Program and live-grid validation plan
|
||||
|
||||
The `programs` package contains these binary targets:
|
||||
|
||||
| Rust binary | C# project | Port purpose |
|
||||
|---|---|---|
|
||||
| `simple-bot` | `SimpleBot` | login, IM/chat events, movement and animations |
|
||||
| `prim-inspector` | `PrimInspector` | object discovery/properties and transforms |
|
||||
| `inventory-explorer` | `InventoryExplorer` | inventory fetch, search, stats and export |
|
||||
| `packet-dump` | `PacketDump` | raw/decoded packet subscription and logging |
|
||||
| `irc-gateway` | `IRCGateway` | concurrent external chat bridge |
|
||||
| `test-client` | `TestClient` | full interactive client and all 128 command targets |
|
||||
| `vivox-test` | `VivoxTest` | Vivox account/session/participant validation |
|
||||
| `webrtc-test` | `WebRtcTest` | WebRTC signaling, devices and audio validation |
|
||||
| `osd-inspector` | `OSDInspector` | offline LLSD inspect/convert/validate/primitive round trip |
|
||||
|
||||
Port `osd-inspector` first because it is offline and validates StructuredData.
|
||||
Then port `simple-bot`, `packet-dump`, `prim-inspector`, and
|
||||
`inventory-explorer` as vertical grid slices. Port TestClient commands by folder
|
||||
(system/login first, then communication, inventory, objects, movement, land,
|
||||
groups, friends, directory, appearance, voice, stats) while keeping
|
||||
`commands::TEST_CLIENT_COMMANDS` as the completeness inventory. Voice programs
|
||||
come last because they need native/device and service prerequisites.
|
||||
|
||||
Live validation must use a dedicated test account and an explicitly supplied
|
||||
grid URI. Secrets are read from environment variables or the OS secret store,
|
||||
never source/config committed to Git. A live smoke run records sanitized:
|
||||
|
||||
- server/grid/version and Rust commit;
|
||||
- login and initial simulator/capabilities completion;
|
||||
- IM/chat send/receive;
|
||||
- movement/teleport/region crossing;
|
||||
- inventory fetch and one reversible test-folder operation;
|
||||
- object discovery/property request;
|
||||
- asset/texture download and decode;
|
||||
- clean logout and task/socket teardown.
|
||||
|
||||
Operations that spend currency, modify estate/parcel state, upload assets,
|
||||
delete inventory, or affect other users require separate opt-in flags and are
|
||||
not part of the default smoke test.
|
||||
|
||||
## 8. Ordered delivery milestones
|
||||
|
||||
1. **Structural inventory shell (current Stage 0):** crates, type/declaration
|
||||
ledgers, failing test parity entries, program targets, license, and this plan
|
||||
compile. This is not the callable API or semantic test suite.
|
||||
2. **Full public API signature shim:** extract authoritative compiled metadata,
|
||||
map 100% of public members to reviewed Rust signatures, and implement only
|
||||
standardized failing bodies. All signatures and downstream compile fixtures
|
||||
build before test translation begins.
|
||||
3. **Full semantic test translation:** replace all 1,295 `pending` cases with
|
||||
exact reviewed Rust equivalents using the public shim. The entire suite
|
||||
compiles and fails only at standardized unimplemented member boundaries.
|
||||
4. **Types implementation:** UUID, math, enums, conversions, collections,
|
||||
cache/token bucket; turn only the already-existing Types tests green.
|
||||
5. **StructuredData implementation:** OSD plus binary/notation/XML/JSON/protobuf
|
||||
compatibility; turn only the already-existing StructuredData tests green.
|
||||
6. **Imaging and PrimMesher implementation:** managed images, codec trait,
|
||||
TGA/J2K adapter,
|
||||
extrusion/sculpt/OBJ and rendering tests.
|
||||
7. **Wire codegen:** deterministic Rust packet/visual/skeleton/tree/genepool/
|
||||
attention generation and golden packet bytes.
|
||||
8. **Networking/client lifecycle:** UDP, caps, login, simulators, settings,
|
||||
cancellation, reconnect and logout against fake servers.
|
||||
9. **World vertical slices:** agent, inventory, assets/appearance, objects,
|
||||
parcels/terrain/environment, social/directory/groups, marketplace/estate.
|
||||
10. **Rendering, RLV, LSL tools:** implement against their already-translated
|
||||
suites and add public migration docs.
|
||||
11. **Programs/live grid:** offline inspector, four core examples, TestClient,
|
||||
then voice adapters and opt-in live/device validation.
|
||||
12. **Hardening/release:** fuzz untrusted parsers, cross-platform native CI,
|
||||
MSRV, docs/examples, semver/API audit, license/asset audit, performance
|
||||
comparison against the separately run pinned C# implementation, and an
|
||||
artifact/dependency audit proving there is no CLR, .NET assembly, subprocess,
|
||||
RPC, or LibreMetaverse foreign-interface dependency.
|
||||
|
||||
Each milestone ends with zero unreviewed generated shims in its owned modules,
|
||||
no weakened parity assertions, formatted/clippy-clean code, public Rust docs
|
||||
that cite the corresponding C# concept, and a refreshed completeness report
|
||||
from the checked-in ledgers.
|
||||
Reference in New Issue
Block a user