MetaCrate/codegen/README.md

Native data generation

libremetaverse-codegen is the Rust-only framework for the six source generators present at LibreMetaverse commit 2aa70bb68513b39795da5d13c88f31b86e85a3ba. Normal builds compile only the checked-in Rust outputs. They do not run this tool, invoke .NET or Roslyn, read an environment-dependent path, access the network, or open the sibling reference checkout.

Provenance inventory

sources.json is the machine-readable inventory. It records the upstream repository and commit, source-generator hashes, every actual AdditionalFile, its vendored path, SHA-256, format, and applicable license. The generator implementations and message_template.msg are covered by the pinned upstream BSD-3-Clause notice. The seven inputs copied from LibreMetaverse/linden/ are separately licensed CC BY-SA 3.0 and retain that license through generated data tables; see NOTICE.md.

Golden generator	Pinned data inputs
PacketSourceGenerator	message_template.msg
VisualParamGenerator	avatar_lad.xml
SkeletonGenerator	avatar_skeleton.xml
AttentionsGenerator	attentions.xml, attentionsN.xml
TreesGenerator	trees.xml, grass.xml
GenepoolGenerator	genepool.xml

The C# project does not provide the optional visualparamtemplate.cs file. The pinned VisualParam generator therefore uses the template embedded in its own source; the inventory records that generator source and hash as provenance.

The files under inputs/ are exact, unmodified snapshots. Their distinct license and copyright attribution are covered by NOTICE.md, and the generated source manifest repeats each hash and license in its header and data table. MetaCrate does not vendor the sibling Linden textures, meshes, animations, gestures, clothing, body parts, or other viewer assets.

Packet generation

The packet generator tokenizes the pinned message_template.msg directly in Rust. It preserves source order, all 483 packet names, the 434 low/17 medium/32 high frequency assignments, 16-bit IDs (including the low-frequency Fixed IDs), trust and zerocode flags, optional protocol flags, 905 blocks, repetition rules, and every field width. Braces, identifiers, numeric values, field kinds, prefix widths, duplicate names, and duplicate frequency/ID pairs are validated with stable source-located PGxxx diagnostics.

packet_catalog.rs is the checked-in native output. It contains the public PacketType discriminants, ordered immutable schema descriptors, name/type/frequency-ID dispatch tables, and native construction and sizing implementations for every mapped packet and block. Variable blocks start empty, fixed-repeat blocks receive their declared number of independently constructed elements, and scalar/vector/UUID fields receive the same zero defaults as the golden generator. The generated Length metadata intentionally preserves the golden generator's variable-block count semantics.

The same output contains the native wire implementation for every generated packet and block. It preserves high-, medium-, and low-frequency headers, network-order packet IDs, sequence numbers and appended ACKs; the reference field endianness and widths; one- and two-byte variable prefixes; fixed-field padding and truncation; fixed and variable block counts; normalized quaternion encoding; zerocoding boundaries; and the reference ToBytesMultiple splitting rules. Readers use checked bounds and allocation arithmetic and reject unknown IDs or truncated payloads without panicking. Generated tests round-trip all 483 packet types and exercise every truncation boundary, while focused fixtures assert exact reference bytes and malformed-input behavior.

The generated packet and block APIs also implement the golden generator's PacketToOSD, PacketFromOSD, ToOSD, and FromOSD behavior. This includes little-endian binary encoding for unsigned 32- and 64-bit values, permissive LLSD conversions, omission of absent map fields, one-element arrays for single blocks, exact replacement of variable blocks, and bounded updates of fixed-repeat blocks. The all-fields test round-trips every packet through both wire bytes and LLSD maps.

Generation also compares all packet, block, field, field-type, and PacketType entries against the pinned compiled public API catalog. This rejects a template parser or naming change even if the emitted Rust would otherwise compile. The Rust API compile fixture then compiles every mapped generated member, while the packet catalog test exercises representative low/medium/high, fixed, multiple, variable, and unknown dispatch behavior.

Visual parameters and foliage

The visual-parameter generator parses the pinned avatar_lad.xml with explicit root, attribute, numeric, vector, color, relationship, and duplicate-ID validation. Its checked-in visual_catalog.rs output contains all 672 public parameters, including ranges and defaults, groups, wearable associations, alpha/color data, driven relationships, skeletal distortions, and volume morphs. VisualParams::params() exposes a BTreeMap-backed sorted list keyed by parameter ID. The 253 group-0/group-3 transmitted IDs are kept in ascending numeric order because that order is observable in AvatarAppearance bytes; no other XML order is retained.

The foliage generator validates trees.xml and grass.xml, including UUIDs, all numeric properties, unique IDs, and the contiguous source-order species index required by the reference Get implementation. Its checked-in foliage_catalog.rs output uses source-ordered vectors, so tree and grass enum discriminants provide constant-time indexed lookup. Golden fixtures cover all catalog counts, representative complete records, public getters/setters, appearance decoding, and every generated relationship category.

Skeleton, attentions, and genepool

The skeleton generator validates the pinned hierarchy root, declared counts, finite three-component transforms, unique joint names and aliases, collision volumes, support categories, and child element schema. Its checked-in skeleton_catalog.rs output constructs all 133 joints and 26 collision volumes in source hierarchy order. The native skeleton records also implement depth-first enumeration, canonical and alias lookup, expanded mesh-joint lists, the built-in loader, and validated custom XML loading without a serializer or managed runtime.

The attention generator validates both pinned attention inputs independently, requiring masculine and feminine sets with each of the nine named parameters exactly once. The generated attention_catalog.rs keeps the eleven LookAtType slots in enum-discriminant order, including the observable zero-initialized None and Clear entries.

The genepool generator validates unique archetype names and parameter IDs, finite values, and each ignored-but-schema-relevant texture UUID and slot. The generated genepool_catalog.rs preserves source archetype and parameter order for all 24 archetypes and 3,360 parameter values; exact-name lookup returns the original source index. Separate generated headers retain the input hashes and CC-BY-SA-3.0 provenance for the skeleton, both attention files, and genepool asset.

Commands

Regenerate every currently registered Rust output using only checked-in data:

cargo run -p libremetaverse-codegen -- generate

Verify hashes and reject stale output without writing:

cargo run -p libremetaverse-codegen -- check

Generate into an empty temporary tree while continuing to read the pinned inputs from this checkout:

cargo run -p libremetaverse-codegen -- generate-to /tmp/metacrate-codegen

Maintainers can refresh the vendored snapshots from an explicitly supplied checkout. The command verifies all six golden generator hashes and all input hashes before writing, so a different upstream revision is rejected:

cargo run -p libremetaverse-codegen -- vendor ../libremetaverse

Generation uses UTF-8 with an optional BOM, normalizes CRLF/CR to LF, rejects NUL input, reports stable path:line:column: severity[code] diagnostics, sorts manifest records by stable IDs, emits provenance-rich generated headers, and always terminates output with one LF. Tests generate every registered output twice in memory and require byte identity before comparing with the checked-in files.

Completion gate

The complete milestone gate regenerates source_manifest.rs, packet_catalog.rs, visual_catalog.rs, foliage_catalog.rs, skeleton_catalog.rs, attention_catalog.rs, and genepool_catalog.rs in an isolated temporary directory and byte-compares them with the checked-in files. It also verifies the six golden generator records (PacketSourceGenerator, VisualParamGenerator, SkeletonGenerator, AttentionsGenerator, TreesGenerator, and GenepoolGenerator), all eight pinned input names and hashes, generated headers and licenses, complete public API ledgers/shims and the compile fixture, native packet LLSD implementations, and the absence of .NET/Roslyn dependencies or build-script invocations.

Run the normal gate after generating a clean Rust release build:

cargo build --workspace --release --locked
python3 tools/check_codegen_gate.py --release-dir target/release
cargo check --manifest-path tests/api-compile/Cargo.toml --locked

The release-directory option additionally scans actual artifacts for managed runtime names and markers. The Gitea workflow runs these commands from a clean checkout using only ubuntu-latest.