Chili Palmer b413cd6f4d
Some checks failed
Native code generation / deterministic (push) Failing after 6m8s
Imaging and meshing gate / native (push) Failing after 17s
JPEG 2000 feature / linux (push) Failing after 59s
Skia feature / linux (push) Failing after 1m33s
Implement native client core lifecycle (#51)
2026-08-09 10:53:33 +00:00
2026-08-08 09:49:22 +02:00
2026-08-08 18:30:51 +02:00
2026-08-09 07:07:13 +00:00

MetaCrate

MetaCrate is a test-driven, clean native Rust reimplementation of LibreMetaverse. LibreMetaverse is a behavioral and API reference only: the finished project must not load, host, bind to, invoke, or ship the .NET implementation. The current stage is a compiling structural shell: crate boundaries mirror the .NET library projects, public C# type names have Rust declarations, every upstream NUnit invocation has a traceable catalog entry, and every sample/tool project has a Rust binary target. Types, StructuredData, Imaging, PrimMesher, both rendering adapters, and the main assembly's packet/message/asset/primitive wire-data, core runtime/networking, avatar-facing manager, world/social/service manager, RLV, LSL tools, Utilities, Vivox, and WebRTC slices now have complete callable, failure-only signatures. The independent downstream fixture compiles every cataloged type and member with zero exclusions, and all 1,289 NUnit invocations now have reviewed Rust parity cases. Production behavior implementation is the next stage.

A case counts as translated only when an explicit Rust test body calls the mapped API and retains the upstream identity and body hash. The parity ledger now contains zero pending or unreviewed cases.

The source snapshot, compatibility rules, dependency research, and ordered implementation plan are in RUSTREWRITE.md.

The authoritative compiled .NET surface is checked in as api/public-api.json. Every type and member has a reviewed Rust destination in api/RUST-TYPES.tsv and api/RUST-MAPPING.tsv; api/README.md documents deterministic regeneration and validation. All API crates expose the same typed Error for fallible unimplemented members; infallible shims use libremetaverse_types::unimplemented_api! so placeholders cannot look like successful behavior.

Building MetaCrate requires Rust 1.96 or newer.

cargo build --workspace
cargo test --workspace --no-run
python3 tools/generate_rust_mapping.py --check
python3 tools/generate_api_shims.py --check
python3 tools/check_test_parity.py
python3 tools/audit_red_suite.py

Optional codec features

The default build needs no system image-codec library. It includes the project-owned TGA and DDS implementations; the default dds-bc67 feature uses the pure-Rust bcdec_rs dependency for BC6H and BC7. Build without BC6H/BC7 support with cargo build -p libremetaverse --no-default-features.

The currently available opt-in codec backends are:

Capability Feature Build command External prerequisite
Raw J2K and JP2 decoding/encoding through OpenJPEG libremetaverse/jpeg2000 cargo build -p libremetaverse --features jpeg2000 OpenJPEG 2.5.4 or newer
BMP, GIF, ICO, JPEG, PNG, WBMP, and WebP decoding through Skia libremetaverse-imaging-skia/skia cargo build -p libremetaverse-imaging-skia --features skia A matching rust-skia binary cache, or the documented native source-build tools

Run their focused test suites with:

cargo test -p libremetaverse-imaging --features jpeg2000
cargo test -p libremetaverse-imaging-skia --features skia

cargo build --workspace --all-features enables both optional backends and therefore requires both sets of native prerequisites. Platform installation, offline-build, cache, and redistribution details are in the OpenJPEG adapter guide and Skia adapter guide.

An independent optional rust-j2k backend is planned but has not been implemented yet. There is deliberately no enablement command for it today; the existing jpeg2000 feature always selects the OpenJPEG backend.

tests/upstream-tests.json is the machine-readable NUnit parity catalog. Translated tests live in hand-written Rust files with the parity-case marker documented in tests/PARITY.md; python3 tools/generate_surface.py --check verifies the catalog against the pinned adjacent LibreMetaverse checkout without overwriting those files.

Running cargo test --workspace is intentionally red during the shim stage.

Native implementation progress

Milestone 04

Milestone 04 is complete. libremetaverse-types now provides native UUID, incremental CRC-32, little-endian floating-point byte helpers, Vector2/Vector3/Vector3d/Vector4 arithmetic and serialization, Color4 color conversion, Ray storage, Quaternion rotation/interpolation/serialization, and Matrix4 construction, transforms, inversion, and decomposition. UUID protocol bytes use explicit network order while compat::Guid conversions preserve the mixed-endian .NET byte-array layout. Vector, color, and normalized quaternion byte encodings are explicitly little-endian and do not depend on host architecture. Matrix4 preserves LibreMetaverse's row-major storage, row-vector transform direction, and translation in the fourth row; quaternion components are stored as (x, y, z, w) and products use the reference Hamilton-product direction. All Types enum discriminants retain their C# widths, extensible protocol flags preserve unknown bits, and the native conversion layer covers numeric and floating-point byte order, UTF-8 and hex text, hashes and PBKDF2, timestamps, IP addresses, region handles, enum metadata, and cross-platform OS detection. The Types collections, including duplicate-preserving multi-value and synchronized double-key maps, bounded LRU/MRU caches, deterministic absolute and sliding expiration, and hierarchical token-bucket throttling are native Rust as well. Cache and bucket tests use injected monotonic clocks, and concurrency invariants run without sleep-based timing. All 45 Types and all 942 mapped members now resolve to native implementations; no generated Types shims remain.

Milestone 05

Milestone 05 now has a native, format-neutral OSD value model with structural equality and hashing, permissive LibreMetaverse conversion defaults, explicit integer/date byte order, synchronized snapshot-based arrays and maps, and bounded parser dispatch. OSD map order is intentionally unspecified at the public model boundary; encoders must choose and document any stable ordering they require. Untrusted dispatch limits input bytes, nesting depth, decoded nodes, and aggregate binary allocation before returning a value. The native Binary LLSD codec covers every reference marker, both accepted headers, exact numeric and date byte order, seekable stream overloads, and position-bearing errors for malformed or truncated input. Its parser and encoder enforce the same byte, depth, node, and allocation bounds; map keys are encoded in sorted order to make output stable despite the model's intentionally unordered maps. The bounded Notation LLSD codec preserves UTF-16 length-prefix semantics, the reference escape rules and accepted scalar spellings, base64, base16 and length-prefixed binary forms, and both compact and formatted output. Malformed notation reports UTF-16 offsets with parser context, while input, output, nesting, node, and aggregate allocation limits apply to untrusted text. The native XML LLSD codec covers the wrapper and inner-element APIs, all scalar and container elements, namespace-local names, XML declarations, comments, CDATA, entity decoding, and the reference's nonstandard Linden processing instruction handling. Serialization is compact and deterministic for maps. DTD declarations and named entity expansion are disabled; input, output, nesting, node, and aggregate allocation limits are enforced with byte-positioned parse errors. JSON conversion is explicit rather than a derived Serde mapping: integral width coercion, empty strings, binary arrays, typed values, and the reference's default omission rules retain their C# behavior. Its reader rejects duplicate properties and enforces byte, depth, node, and allocation limits while compact output uses System.Text.Json-compatible escaping and stable map-key ordering. The private OSD Protobuf schema is implemented directly with fixed field tags, ZigZag int32, little-endian IEEE-754 fixed64 values, 16-byte UUIDs, bounded length-delimited containers, and wire-type-aware unknown-field skipping. The schema remains an internal compatibility format; native encoders emit stable sorted maps and accept the exact optional LLSD Protobuf header. Cross-format conversion follows the reference's format-specific boundaries:

Format Round-trip behavior
Binary LLSD All regular OSD variants are lossless; raw OSDLlsdXml is unsupported.
Notation LLSD All regular OSD variants are lossless; raw OSDLlsdXml is unsupported.
XML LLSD Regular variants are lossless; raw OSDLlsdXml is injected as an element and therefore decodes as that element's ordinary OSD value.
JSON OSD Booleans, integers, reals, nonempty strings, maps, and arrays retain structure; UUID, date, and URI become strings, binary becomes an integer array, and undefined, empty strings, and raw XML become JSON null.
Protobuf OSD Regular variants are lossless within the reference schema's date range and whole-second precision; raw OSDLlsdXml maps to undefined.

Checked-in malformed corpora and bounded deterministic mutations exercise all five parsers without an external fuzzing runtime. They cover truncation, tags, lengths, payloads, delimiters, nesting, entity rejection, contextual errors, and the parser resource limits. Every encoder sorts map keys where necessary, so serialization is deterministic across platforms.

Milestone 06

Milestone 06 starts with a native ManagedImage: top-left-origin, byte-per-sample planar storage has an explicit width-byte row stride, with gray stored in the red plane and optional color, alpha, and bump planes. Checked constructors and resizers cap decoded images at 4096 by 4096 worth of pixels, codec adapters cap buffered encoded input at 64 MiB, and canonical interleaving accepts explicit row strides without exposing codec-library types through the core abstraction. The native TGA/Pfim-compatible path decodes uncompressed and RLE true-color, grayscale, and indexed TGA data with all descriptor orientations, plus bounded legacy and DX10 DDS packed luminance/RGB(A) layouts and BC1 through BC7 blocks. TGA output preserves the reference header, BGR(A) order, descriptor choice, alpha-only expansion, and deterministic trailing padding. File and stream inputs are capped before decoding, and all dimensions, packed rows, palettes, packets, masks, and block indices use checked arithmetic with positioned parse errors. Legacy and DX10 BC1 through BC5 decoding is built in. The default dds-bc67 feature adds BC6H/BC7 decoding through the safe, pure-Rust bcdec_rs crate; disabling default features removes that optional dependency and makes those two formats return a typed unsupported-format error. JPEG 2000 is available through the opt-in jpeg2000 feature. It provides raw J2K and JP2 lossless/lossy encoding and decoding, preserves one through five component order, precision, signedness, and alpha metadata until explicit byte conversion, and bounds encoded input, output, dimensions, and decoded samples. The compatibility encoder reproduces CoreJ2K's four-plane RGB/alpha view, including its alpha-only and opaque-alpha substitutions. See crates/libremetaverse-openjpeg/README.md for OpenJPEG prerequisites, licensing, and deployment details. Default builds do not discover or link OpenJPEG. The optional Skia adapter decodes BMP, GIF, ICO, JPEG, PNG, WBMP, and WebP into the same checked planar representation. Its skia feature uses pinned rust-skia binary-cache configurations on macOS, Linux, and Windows, normalizes premultiplied color at the imaging boundary, and keeps encoded input, decoded dimensions, strides, and allocations bounded. See crates/libremetaverse-imaging-skia/README.md for native prerequisites, cache/source-build controls, licenses, and packaging details. Default workspace builds do not compile or link Skia. Primitive profile and path meshing is implemented in native Rust from the pinned PrimMesher.cs reference. It covers checked linear, flexible, and circular extrusion; profile cuts and hollows; twist, taper, shear, skew, radius, and revolutions; cap and side winding; normals, UVs, viewer faces, and prim-face indexing. Geometry allocation and index arithmetic are bounded, and identical finite inputs produce deterministic mesh output on every supported platform. Sculpt maps are sampled through the same checked imaging abstraction and can produce plane, sphere, torus, and cylinder topology with reference seam, mirror, inversion, LOD, normal, UV, and viewer-face behavior. Native viewer indexing deduplicates vertices per prim face, while bounded Wavefront OBJ ingestion preserves object/group output and position/UV/normal associations. Malformed dimensions, channel layouts, topology, and indices return typed errors without partial meshes or panics. The milestone-owned feature matrix, rendering-data contract, reproducible audit, cross-platform CI commands, bounds, and large-input performance baselines are documented in docs/imaging-meshing.md. The controlled audit aggregates every expected failure by standardized C# member ID and rejects unrelated fixture, assertion, compile, or symbol errors.

Milestone 07

The deterministic, Rust-only source-data generator framework and its complete pinned input inventory are documented in codegen/README.md. The first native data generator now parses the pinned protocol template into a checked-in catalog covering all 483 packets, 905 blocks, exact field widths, repetition rules, flags, IDs, and frequencies. It supplies the public PacketType, deterministic lookup tables, and real default construction and sizing behavior for every mapped packet/block while validating the complete shape against the pinned compiled C# API catalog. The generated native packet codec implements the reference header frequencies and IDs, flags, sequences, appended ACKs, little-endian fields, big-endian ports, fixed and length-prefixed data, block repetition, zerocoding, bounded decoding, and MTU packet splitting. Golden-byte tests and generated round trips cover every packet type without requiring the C# toolchain at build or test time. The native XML generators also emit all 672 visual parameters and all 21 tree and 6 grass definitions from the pinned character and foliage inputs. Generated types preserve ranges, defaults, groups, wearables, color/alpha data, driven relationships, skeletal/volume morphs, and every foliage rendering property. Schema validation rejects malformed or ambiguous input, while sorted visual-ID and source-indexed foliage lookups preserve the ordering observable in the reference APIs and packet bytes. The remaining avatar-data generators emit the full 133-bone/26-collision-volume skeleton hierarchy, both masculine and feminine versions of the default and updated 11-slot attention tables, and all 24 genepool archetypes with 3,360 visual-parameter values. Native skeleton traversal, alias lookup, expanded mesh joint lists, custom XML loading, attention indexing, and archetype lookup now implement the mapped behavior directly in Rust. Each checked-in catalog retains its own input hash and license provenance.

Milestone 08

The native client core now provides the exact grouped Settings defaults from the golden C# implementation, validates endpoints, durations, limits, cache policy, and download policy, and exposes typed configuration and lifecycle errors. GridClient construction and drop have no network side effects and do not create an async runtime. Explicitly composed services share a cancellation token and are shut down idempotently in network, manager, HTTP, then rate-limiter order. Injected clocks support deterministic tests, while Debug output omits endpoint values and service internals. The ownership and executor requirements are documented in docs/client-core.md.

Description
reimplementation of libremetaverse in Rust to enable compact and fast metaverse tools.
Readme 107 MiB
Languages
Rust 92.2%
Python 5.4%
omnetpp-msg 1.9%
C# 0.4%