188 lines
9.3 KiB
Markdown
188 lines
9.3 KiB
Markdown
# Native data generation
|
|
|
|
`libremetaverse-codegen` is the Rust-only framework for the six source
|
|
generators present at LibreMetaverse commit
|
|
`2aa70bb68513b39795da5d13c88f31b86e85a3ba`. Normal builds compile only the
|
|
checked-in Rust outputs. They do not run this tool, invoke .NET or Roslyn, read
|
|
an environment-dependent path, access the network, or open the sibling
|
|
reference checkout.
|
|
|
|
## Provenance inventory
|
|
|
|
[`sources.json`](sources.json) is the machine-readable inventory. It records
|
|
the upstream repository and commit, source-generator hashes, every actual
|
|
`AdditionalFile`, its vendored path, SHA-256, format, and BSD-3-Clause license.
|
|
|
|
| Golden generator | Pinned data inputs |
|
|
| --- | --- |
|
|
| PacketSourceGenerator | `message_template.msg` |
|
|
| VisualParamGenerator | `avatar_lad.xml` |
|
|
| SkeletonGenerator | `avatar_skeleton.xml` |
|
|
| AttentionsGenerator | `attentions.xml`, `attentionsN.xml` |
|
|
| TreesGenerator | `trees.xml`, `grass.xml` |
|
|
| GenepoolGenerator | `genepool.xml` |
|
|
|
|
The C# project does not provide the optional `visualparamtemplate.cs` file.
|
|
The pinned VisualParam generator therefore uses the template embedded in its
|
|
own source; the inventory records that generator source and hash as provenance.
|
|
|
|
The files under `inputs/` are exact, unmodified snapshots. Their license and
|
|
copyright attribution are covered by the repository
|
|
[`LICENSE.md`](../LICENSE.md). The generated source manifest repeats each hash
|
|
and license in its header and data table.
|
|
|
|
## Packet generation
|
|
|
|
The packet generator tokenizes the pinned `message_template.msg` directly in
|
|
Rust. It preserves source order, all 483 packet names, the 434 low/17 medium/32
|
|
high frequency assignments, 16-bit IDs (including the low-frequency `Fixed`
|
|
IDs), trust and zerocode flags, optional protocol flags, 905 blocks, repetition
|
|
rules, and every field width. Braces, identifiers, numeric values, field kinds,
|
|
prefix widths, duplicate names, and duplicate frequency/ID pairs are validated
|
|
with stable source-located `PGxxx` diagnostics.
|
|
|
|
[`packet_catalog.rs`](../crates/libremetaverse/src/packet_catalog.rs) is the
|
|
checked-in native output. It contains the public `PacketType` discriminants,
|
|
ordered immutable schema descriptors, name/type/frequency-ID dispatch tables,
|
|
and native construction and sizing implementations for every mapped packet and
|
|
block. Variable blocks start empty, fixed-repeat blocks receive their declared
|
|
number of independently constructed elements, and scalar/vector/UUID fields
|
|
receive the same zero defaults as the golden generator. The generated `Length`
|
|
metadata intentionally preserves the golden generator's variable-block count
|
|
semantics.
|
|
|
|
The same output contains the native wire implementation for every generated
|
|
packet and block. It preserves high-, medium-, and low-frequency headers,
|
|
network-order packet IDs, sequence numbers and appended ACKs; the reference
|
|
field endianness and widths; one- and two-byte variable prefixes; fixed-field
|
|
padding and truncation; fixed and variable block counts; normalized quaternion
|
|
encoding; zerocoding boundaries; and the reference `ToBytesMultiple` splitting
|
|
rules. Readers use checked bounds and allocation arithmetic and reject unknown
|
|
IDs or truncated payloads without panicking. Generated tests round-trip all 483
|
|
packet types and exercise every truncation boundary, while focused fixtures
|
|
assert exact reference bytes and malformed-input behavior.
|
|
|
|
The generated packet and block APIs also implement the golden generator's
|
|
`PacketToOSD`, `PacketFromOSD`, `ToOSD`, and `FromOSD` behavior. This includes
|
|
little-endian binary encoding for unsigned 32- and 64-bit values, permissive
|
|
LLSD conversions, omission of absent map fields, one-element arrays for single
|
|
blocks, exact replacement of variable blocks, and bounded updates of
|
|
fixed-repeat blocks. The all-fields test round-trips every packet through both
|
|
wire bytes and LLSD maps.
|
|
|
|
Generation also compares all packet, block, field, field-type, and `PacketType`
|
|
entries against the pinned compiled public API catalog. This rejects a template
|
|
parser or naming change even if the emitted Rust would otherwise compile. The
|
|
Rust API compile fixture then compiles every mapped generated member, while the
|
|
packet catalog test exercises representative low/medium/high, fixed, multiple,
|
|
variable, and unknown dispatch behavior.
|
|
|
|
## Visual parameters and foliage
|
|
|
|
The visual-parameter generator parses the pinned `avatar_lad.xml` with explicit
|
|
root, attribute, numeric, vector, color, relationship, and duplicate-ID
|
|
validation. Its checked-in
|
|
[`visual_catalog.rs`](../crates/libremetaverse/src/visual_catalog.rs) output
|
|
contains all 672 public parameters, including ranges and defaults, groups,
|
|
wearable associations, alpha/color data, driven relationships, skeletal
|
|
distortions, and volume morphs. `VisualParams::params()` exposes a
|
|
`BTreeMap`-backed sorted list keyed by parameter ID. The 253 group-0/group-3
|
|
transmitted IDs are kept in ascending numeric order because that order is
|
|
observable in `AvatarAppearance` bytes; no other XML order is retained.
|
|
|
|
The foliage generator validates `trees.xml` and `grass.xml`, including UUIDs,
|
|
all numeric properties, unique IDs, and the contiguous source-order species
|
|
index required by the reference `Get` implementation. Its checked-in
|
|
[`foliage_catalog.rs`](../crates/libremetaverse/src/foliage_catalog.rs) output
|
|
uses source-ordered vectors, so tree and grass enum discriminants provide
|
|
constant-time indexed lookup. Golden fixtures cover all catalog counts,
|
|
representative complete records, public getters/setters, appearance decoding,
|
|
and every generated relationship category.
|
|
|
|
## Skeleton, attentions, and genepool
|
|
|
|
The skeleton generator validates the pinned hierarchy root, declared counts,
|
|
finite three-component transforms, unique joint names and aliases, collision
|
|
volumes, support categories, and child element schema. Its checked-in
|
|
[`skeleton_catalog.rs`](../crates/libremetaverse/src/skeleton_catalog.rs) output
|
|
constructs all 133 joints and 26 collision volumes in source hierarchy order.
|
|
The native skeleton records also implement depth-first enumeration, canonical
|
|
and alias lookup, expanded mesh-joint lists, the built-in loader, and validated
|
|
custom XML loading without a serializer or managed runtime.
|
|
|
|
The attention generator validates both pinned attention inputs independently,
|
|
requiring masculine and feminine sets with each of the nine named parameters
|
|
exactly once. The generated
|
|
[`attention_catalog.rs`](../crates/libremetaverse/src/attention_catalog.rs)
|
|
keeps the eleven `LookAtType` slots in enum-discriminant order, including the
|
|
observable zero-initialized `None` and `Clear` entries.
|
|
|
|
The genepool generator validates unique archetype names and parameter IDs,
|
|
finite values, and each ignored-but-schema-relevant texture UUID and slot. The
|
|
generated [`genepool_catalog.rs`](../crates/libremetaverse/src/genepool_catalog.rs)
|
|
preserves source archetype and parameter order for all 24 archetypes and 3,360
|
|
parameter values; exact-name lookup returns the original source index. Separate
|
|
generated headers retain the input hashes and BSD-3-Clause provenance for the
|
|
skeleton, both attention files, and genepool asset.
|
|
|
|
## Commands
|
|
|
|
Regenerate every currently registered Rust output using only checked-in data:
|
|
|
|
```console
|
|
cargo run -p libremetaverse-codegen -- generate
|
|
```
|
|
|
|
Verify hashes and reject stale output without writing:
|
|
|
|
```console
|
|
cargo run -p libremetaverse-codegen -- check
|
|
```
|
|
|
|
Generate into an empty temporary tree while continuing to read the pinned
|
|
inputs from this checkout:
|
|
|
|
```console
|
|
cargo run -p libremetaverse-codegen -- generate-to /tmp/metacrate-codegen
|
|
```
|
|
|
|
Maintainers can refresh the vendored snapshots from an explicitly supplied
|
|
checkout. The command verifies all six golden generator hashes and all input
|
|
hashes before writing, so a different upstream revision is rejected:
|
|
|
|
```console
|
|
cargo run -p libremetaverse-codegen -- vendor ../libremetaverse
|
|
```
|
|
|
|
Generation uses UTF-8 with an optional BOM, normalizes CRLF/CR to LF, rejects
|
|
NUL input, reports stable `path:line:column: severity[code]` diagnostics, sorts
|
|
manifest records by stable IDs, emits provenance-rich generated headers, and
|
|
always terminates output with one LF. Tests generate every registered output
|
|
twice in memory and require byte identity before comparing with the checked-in
|
|
files.
|
|
|
|
## Completion gate
|
|
|
|
The complete milestone gate regenerates `source_manifest.rs`,
|
|
`packet_catalog.rs`, `visual_catalog.rs`, `foliage_catalog.rs`,
|
|
`skeleton_catalog.rs`, `attention_catalog.rs`, and `genepool_catalog.rs` in an
|
|
isolated temporary directory and byte-compares them with the checked-in files.
|
|
It also verifies the six golden generator records (`PacketSourceGenerator`,
|
|
`VisualParamGenerator`, `SkeletonGenerator`, `AttentionsGenerator`,
|
|
`TreesGenerator`, and `GenepoolGenerator`), all eight pinned input names and
|
|
hashes, generated headers and licenses, complete public API ledgers/shims and
|
|
the compile fixture, native packet LLSD implementations, and the absence of
|
|
.NET/Roslyn dependencies or build-script invocations.
|
|
|
|
Run the normal gate after generating a clean Rust release build:
|
|
|
|
```console
|
|
cargo build --workspace --release --locked
|
|
python3 tools/check_codegen_gate.py --release-dir target/release
|
|
cargo check --manifest-path tests/api-compile/Cargo.toml --locked
|
|
```
|
|
|
|
The release-directory option additionally scans actual artifacts for managed
|
|
runtime names and markers. The Gitea workflow runs these commands from a clean
|
|
checkout using only `ubuntu-latest`.
|