MetaCrate/README.md

# MetaCrate

MetaCrate is a test-driven, clean native Rust reimplementation of
[LibreMetaverse](https://github.com/cinderblocks/libremetaverse). LibreMetaverse
is a behavioral and API reference only: the finished project must not load,
host, bind to, invoke, or ship the .NET implementation. The current
stage is a compiling structural shell: crate boundaries mirror the .NET library
projects, public C# type names have Rust declarations, every upstream NUnit
invocation has a traceable catalog entry, and every sample/tool project has a
Rust binary target. Types, StructuredData, Imaging, PrimMesher, both rendering adapters, and
the main assembly's packet/message/asset/primitive wire-data, core
runtime/networking, avatar-facing manager, world/social/service manager, RLV,
LSL tools, Utilities, Vivox, and WebRTC slices now have complete callable,
failure-only signatures. The independent downstream fixture compiles every
cataloged type and member with zero exclusions, and all 1,289 NUnit invocations
now have reviewed Rust parity cases. Production behavior implementation is the
next stage.

A case counts as translated only when an explicit Rust test body calls the
mapped API and retains the upstream identity and body hash. The parity ledger
now contains zero pending or unreviewed cases.

The source snapshot, compatibility rules, dependency research, and ordered
implementation plan are in [RUSTREWRITE.md](RUSTREWRITE.md).

The authoritative compiled .NET surface is checked in as
[`api/public-api.json`](api/public-api.json). Every type and member has a
reviewed Rust destination in [`api/RUST-TYPES.tsv`](api/RUST-TYPES.tsv) and
[`api/RUST-MAPPING.tsv`](api/RUST-MAPPING.tsv); [`api/README.md`](api/README.md)
documents deterministic regeneration and validation.
All API crates expose the same typed `Error` for fallible unimplemented members;
infallible shims use `libremetaverse_types::unimplemented_api!` so placeholders
cannot look like successful behavior.

Building MetaCrate requires Rust 1.96 or newer.

```sh
cargo build --workspace
cargo test --workspace --no-run
python3 tools/generate_rust_mapping.py --check
python3 tools/generate_api_shims.py --check
python3 tools/check_test_parity.py
python3 tools/audit_red_suite.py
```

### Optional codec features

The default build needs no system image-codec library. It includes the
project-owned TGA and DDS implementations; the default `dds-bc67` feature uses
the pure-Rust `bcdec_rs` dependency for BC6H and BC7. Build without BC6H/BC7
support with `cargo build -p libremetaverse --no-default-features`.

The currently available opt-in codec backends are:

| Capability | Feature | Build command | External prerequisite |
| --- | --- | --- | --- |
| Raw J2K and JP2 decoding/encoding through OpenJPEG | `libremetaverse/jpeg2000` | `cargo build -p libremetaverse --features jpeg2000` | OpenJPEG 2.5.4 or newer |
| BMP, GIF, ICO, JPEG, PNG, WBMP, and WebP decoding through Skia | `libremetaverse-imaging-skia/skia` | `cargo build -p libremetaverse-imaging-skia --features skia` | A matching rust-skia binary cache, or the documented native source-build tools |

Run their focused test suites with:

```sh
cargo test -p libremetaverse-imaging --features jpeg2000
cargo test -p libremetaverse-imaging-skia --features skia
```

`cargo build --workspace --all-features` enables both optional backends and
therefore requires both sets of native prerequisites. Platform installation,
offline-build, cache, and redistribution details are in the
[`OpenJPEG adapter guide`](crates/libremetaverse-openjpeg/README.md) and
[`Skia adapter guide`](crates/libremetaverse-imaging-skia/README.md).

An independent optional `rust-j2k` backend is planned but has not been
implemented yet. There is deliberately no enablement command for it today;
the existing `jpeg2000` feature always selects the OpenJPEG backend.

`tests/upstream-tests.json` is the machine-readable NUnit parity catalog.
Translated tests live in hand-written Rust files with the `parity-case` marker
documented in `tests/PARITY.md`; `python3 tools/generate_surface.py --check`
verifies the catalog against the pinned adjacent LibreMetaverse checkout without
overwriting those files.

Running `cargo test --workspace` is intentionally red during the shim stage.

## Native implementation progress

### Milestone 04

Milestone 04 is complete. `libremetaverse-types` now provides
native UUID, incremental CRC-32, little-endian floating-point byte helpers,
Vector2/Vector3/Vector3d/Vector4 arithmetic and serialization, Color4 color
conversion, Ray storage, Quaternion rotation/interpolation/serialization, and
Matrix4 construction, transforms, inversion, and decomposition. UUID protocol
bytes use explicit network order while `compat::Guid` conversions preserve the
mixed-endian .NET byte-array layout. Vector, color, and normalized quaternion
byte encodings are explicitly little-endian and do not depend on host
architecture. Matrix4 preserves LibreMetaverse's row-major storage, row-vector
transform direction, and translation in the fourth row; quaternion components
are stored as `(x, y, z, w)` and products use the reference Hamilton-product
direction. All Types enum discriminants retain their C# widths, extensible
protocol flags preserve unknown bits, and the native conversion layer covers
numeric and floating-point byte order, UTF-8
and hex text, hashes and PBKDF2, timestamps, IP addresses, region handles, enum
metadata, and cross-platform OS detection. The Types collections,
including duplicate-preserving multi-value and synchronized double-key maps,
bounded LRU/MRU caches, deterministic absolute and sliding expiration, and
hierarchical token-bucket throttling are native Rust as well. Cache and bucket
tests use injected monotonic clocks, and concurrency invariants run without
sleep-based timing. All 45 Types and all 942 mapped members now resolve to
native implementations; no generated Types shims remain.

### Milestone 05

Milestone 05 now has a native, format-neutral OSD value model with structural
equality and hashing, permissive `LibreMetaverse` conversion defaults, explicit
integer/date byte order, synchronized snapshot-based arrays and maps, and
bounded parser dispatch. OSD map order is intentionally unspecified at the
public model boundary; encoders must choose and document any stable ordering
they require. Untrusted dispatch limits input bytes, nesting depth, decoded
nodes, and aggregate binary allocation before returning a value. The native
Binary LLSD codec covers every reference marker, both accepted headers, exact
numeric and date byte order, seekable stream overloads, and position-bearing
errors for malformed or truncated input. Its parser and encoder enforce the
same byte, depth, node, and allocation bounds; map keys are encoded in sorted
order to make output stable despite the model's intentionally unordered maps.
The bounded Notation LLSD codec preserves UTF-16 length-prefix semantics, the
reference escape rules and accepted scalar spellings, base64, base16 and
length-prefixed binary forms, and both compact and formatted output. Malformed
notation reports UTF-16 offsets with parser context, while input, output,
nesting, node, and aggregate allocation limits apply to untrusted text.
The native XML LLSD codec covers the wrapper and inner-element APIs, all scalar
and container elements, namespace-local names, XML declarations, comments,
CDATA, entity decoding, and the reference's nonstandard Linden processing
instruction handling. Serialization is compact and deterministic for maps.
DTD declarations and named entity expansion are disabled; input, output,
nesting, node, and aggregate allocation limits are enforced with byte-positioned
parse errors.
JSON conversion is explicit rather than a derived Serde mapping: integral
width coercion, empty strings, binary arrays, typed values, and the reference's
default omission rules retain their C# behavior. Its reader rejects duplicate
properties and enforces byte, depth, node, and allocation limits while compact
output uses System.Text.Json-compatible escaping and stable map-key ordering.
The private OSD Protobuf schema is implemented directly with fixed field tags,
ZigZag `int32`, little-endian IEEE-754 fixed64 values, 16-byte UUIDs, bounded
length-delimited containers, and wire-type-aware unknown-field skipping. The
schema remains an internal compatibility format; native encoders emit stable
sorted maps and accept the exact optional LLSD Protobuf header.
Cross-format conversion follows the reference's format-specific boundaries:

| Format | Round-trip behavior |
| --- | --- |
| Binary LLSD | All regular OSD variants are lossless; raw `OSDLlsdXml` is unsupported. |
| Notation LLSD | All regular OSD variants are lossless; raw `OSDLlsdXml` is unsupported. |
| XML LLSD | Regular variants are lossless; raw `OSDLlsdXml` is injected as an element and therefore decodes as that element's ordinary OSD value. |
| JSON OSD | Booleans, integers, reals, nonempty strings, maps, and arrays retain structure; UUID, date, and URI become strings, binary becomes an integer array, and undefined, empty strings, and raw XML become JSON `null`. |
| Protobuf OSD | Regular variants are lossless within the reference schema's date range and whole-second precision; raw `OSDLlsdXml` maps to undefined. |

Checked-in malformed corpora and bounded deterministic mutations exercise all
five parsers without an external fuzzing runtime. They cover truncation, tags,
lengths, payloads, delimiters, nesting, entity rejection, contextual errors,
and the parser resource limits. Every encoder sorts map keys where necessary,
so serialization is deterministic across platforms.

### Milestone 06

Milestone 06 starts with a native `ManagedImage`: top-left-origin,
byte-per-sample planar storage has an explicit width-byte row stride, with gray
stored in the red plane and optional color, alpha, and bump planes. Checked
constructors and resizers cap decoded images at 4096 by 4096 worth of pixels,
codec adapters cap buffered encoded input at 64 MiB, and canonical
interleaving accepts explicit row strides without exposing codec-library types
through the core abstraction.
The native TGA/Pfim-compatible path decodes uncompressed and RLE true-color,
grayscale, and indexed TGA data with all descriptor orientations, plus bounded
legacy and DX10 DDS packed luminance/RGB(A) layouts and BC1 through BC7 blocks.
TGA output preserves the reference header, BGR(A) order, descriptor choice,
alpha-only expansion, and deterministic trailing padding. File and stream
inputs are capped before decoding, and all dimensions, packed rows, palettes,
packets, masks, and block indices use checked arithmetic with positioned parse
errors.
Legacy and DX10 BC1 through BC5 decoding is built in. The default `dds-bc67`
feature adds BC6H/BC7 decoding through the safe, pure-Rust `bcdec_rs` crate;
disabling default features removes that optional dependency and makes those
two formats return a typed unsupported-format error.
JPEG 2000 is available through the opt-in `jpeg2000` feature. It provides raw
J2K and JP2 lossless/lossy encoding and decoding, preserves one through five
component order, precision, signedness, and alpha metadata until explicit byte
conversion, and bounds encoded input, output, dimensions, and decoded samples.
The compatibility encoder reproduces CoreJ2K's four-plane RGB/alpha view,
including its alpha-only and opaque-alpha substitutions. See
[`crates/libremetaverse-openjpeg/README.md`](crates/libremetaverse-openjpeg/README.md)
for OpenJPEG prerequisites, licensing, and deployment details. Default builds
do not discover or link OpenJPEG.
The optional Skia adapter decodes BMP, GIF, ICO, JPEG, PNG, WBMP, and WebP into
the same checked planar representation. Its `skia` feature uses pinned
rust-skia binary-cache configurations on macOS, Linux, and Windows, normalizes
premultiplied color at the imaging boundary, and keeps encoded input, decoded
dimensions, strides, and allocations bounded. See
[`crates/libremetaverse-imaging-skia/README.md`](crates/libremetaverse-imaging-skia/README.md)
for native prerequisites, cache/source-build controls, licenses, and packaging
details. Default workspace builds do not compile or link Skia.
Primitive profile and path meshing is implemented in native Rust from the
pinned `PrimMesher.cs` reference. It covers checked linear, flexible, and
circular extrusion; profile cuts and hollows; twist, taper, shear, skew,
radius, and revolutions; cap and side winding; normals, UVs, viewer faces, and
prim-face indexing. Geometry allocation and index arithmetic are bounded, and
identical finite inputs produce deterministic mesh output on every supported
platform.
Sculpt maps are sampled through the same checked imaging abstraction and can
produce plane, sphere, torus, and cylinder topology with reference seam,
mirror, inversion, LOD, normal, UV, and viewer-face behavior. Native viewer
indexing deduplicates vertices per prim face, while bounded Wavefront OBJ
ingestion preserves object/group output and position/UV/normal associations.
Malformed dimensions, channel layouts, topology, and indices return typed
errors without partial meshes or panics.
The milestone-owned feature matrix, rendering-data contract, reproducible
audit, cross-platform CI commands, bounds, and large-input performance
baselines are documented in
[`docs/imaging-meshing.md`](docs/imaging-meshing.md).
The controlled audit aggregates every expected failure by standardized C#
member ID and rejects unrelated fixture, assertion, compile, or symbol errors.

### Milestone 07

The deterministic, Rust-only source-data generator framework and its complete
pinned input inventory are documented in
[`codegen/README.md`](codegen/README.md).
The first native data generator now parses the pinned protocol template into a
checked-in catalog covering all 483 packets, 905 blocks, exact field widths,
repetition rules, flags, IDs, and frequencies. It supplies the public
`PacketType`, deterministic lookup tables, and real default construction and
sizing behavior for every mapped packet/block while validating the complete
shape against the pinned compiled C# API catalog. The generated native packet
codec implements the reference header frequencies and IDs, flags, sequences,
appended ACKs, little-endian fields, big-endian ports, fixed and length-prefixed
data, block repetition, zerocoding, bounded decoding, and MTU packet splitting.
Golden-byte tests and generated round trips cover every packet type without
requiring the C# toolchain at build or test time.
The native XML generators also emit all 672 visual parameters and all 21 tree
and 6 grass definitions from the pinned character and foliage inputs. Generated
types preserve ranges, defaults, groups, wearables, color/alpha data, driven
relationships, skeletal/volume morphs, and every foliage rendering property.
Schema validation rejects malformed or ambiguous input, while sorted visual-ID
and source-indexed foliage lookups preserve the ordering observable in the
reference APIs and packet bytes.
The remaining avatar-data generators emit the full 133-bone/26-collision-volume
skeleton hierarchy, both masculine and feminine versions of the default and
updated 11-slot attention tables, and all 24 genepool archetypes with 3,360
visual-parameter values. Native skeleton traversal, alias lookup, expanded mesh
joint lists, custom XML loading, attention indexing, and archetype lookup now
implement the mapped behavior directly in Rust. Each checked-in catalog retains
its own input hash and license provenance.

### Milestone 08

The native client core now provides the exact grouped `Settings` defaults from
the golden C# implementation, validates endpoints, durations, limits, cache
policy, and download policy, and exposes typed configuration and lifecycle
errors. `GridClient` construction and drop have no network side effects and do
not create an async runtime. Explicitly composed services share a cancellation
token and are shut down idempotently in network, manager, HTTP, then rate-limiter
order. Injected clocks support deterministic tests, while `Debug` output omits
endpoint values and service internals. The ownership and executor requirements
are documented in [`docs/client-core.md`](docs/client-core.md).
The native Tokio UDP layer now implements the C# packet buffers and throttle
encoding plus bounded socket receive, coordination, and single-writer tasks.
It assigns wrapping protocol sequences, aggregates and consumes ACKs, retries
reliable packets with duplicate suppression, preserves zerocoding and the
1,200-byte MTU contract, applies independent task/texture/asset token buckets,
and exposes payload-free transport statistics. All queues, peer state,
zerocode expansion, ACK state, and reliable windows have explicit limits;
linked cancellation and final drop release every socket task. The executor,
wire, backpressure, retry, and security contracts are documented in
[`docs/udp-transport.md`](docs/udp-transport.md).
The native network manager now layers ordered packet and CAPS callback
registries, typed RAII subscriptions, synchronized simulator collections,
cancellable connection events, ACK-gated circuit setup, current-simulator and
seed-capability selection, CAPS `EnableSimulator`, UDP `DisableSimulator`, ping
handling, typed region-handshake replies, two-interval keepalive detection, and
deterministic disconnect reasons on that transport. Callback lists are
snapshotted before invocation, and manager/transport workers retain no dropped
client owner. The ownership,
dispatch, lifecycle, and fake-server verification contracts are documented in
[`docs/network-manager.md`](docs/network-manager.md).
The capability transport is now native Rust as well. `HttpCapsClient` supports
the C# GET/POST/PUT/PATCH/DELETE and LLSD overloads through an injectable
`reqwest`/fake-handler boundary, with HTTPS, bounded redirects, gzip/deflate,
streaming progress, cancellation, explicit request/response/decompression
limits, and rejection of outbound non-HTTP capability URIs. The per-category
oldest-first token buckets retain the reference defaults and cap-name mapping.
The download manager adds a bounded queue and concurrency gate, canonical-URI
deduplication, subscriber progress fanout, shared cancellation, and transient
retry handling while preserving permanent 401/403/404/410 failures. Capability
URLs and tokens are absent from errors and diagnostics. Ownership, limits,
injection, and offline fake-server coverage are documented in
[`docs/caps-http.md`](docs/caps-http.md).
Login is implemented against that transport with the C# password-hashing and
token rules, viewer/channel/platform fields, start-location normalization,
bounded redirects, cancellation, and typed status transitions. Native response
parsing populates session, simulator, inventory, buddy, account-benefit, and
service fields before the initial UDP circuit and seed capability are installed.
Credentials and session/capability tokens are redacted from diagnostics, and
offline fake login/simulator tests cover success, rejection, redirect, timeout,
and cancellation cleanup. The login contract is documented with the network
lifecycle in [`docs/network-manager.md`](docs/network-manager.md).
Region handoff and logout now follow the same packet-level lifecycle as the
golden implementation: a promoted simulator receives `UseCircuitCode` and
`CompleteAgentMovement`, while blocking, asynchronous, and nonblocking logout
send `LogoutRequest`, validate `LogoutReply`, preserve callback ordering, and
perform bounded idempotent teardown. Shutdown cancels login/logout work before
closing every simulator and worker, clears session and capability secrets, and
allows the manager to reconnect cleanly afterward. Loopback fake-server tests
cover handoff, reconnect, reply, timeout, cancellation, and repeated shutdown.
Seed-cap discovery and `EventQueueGet` are native Rust too: the full reference
capability list is posted as LLSD/XML, accepted URIs are rate-categorized, and
bounded long polls preserve ack IDs, reconnect retries, shutdown `done`, and
event order. Typed Linden messages share the CAPS dispatch registry; unknown
messages remain observable and then use the generated caps-to-packet catalog so
packet handlers receive them through the same bounded pipeline as UDP traffic.