Implement OXP export and import #110

New Issue

hugo · 2026-08-09T10:46:31Z

hugo commented

2026-08-09 10:46:31 +00:00

Objective

Implement complete, native Rust import and export of OXP (Onrez Package) object backups. An OXP file is a self-contained object-backup document containing linksets, prim properties, optional object inventory, and optional embedded assets.

The implementation must be usable without any external reference checkout, viewer process, .NET runtime, cache directory, or network service. This issue contains the format contract required for implementation.

Crate naming and workspace structure

Add two native MetaCrate crates:

metacrate-oxp: the format implementation, typed document model, validation, bounded decoder, and encoder.
metacrate: the native umbrella crate, analogous to the existing compatibility umbrella, re-exporting metacrate_oxp as oxp.

The libremetaverse-* prefix is reserved for compatibility-layer crates. New native MetaCrate functionality uses the metacrate-* prefix. Do not rename existing crates in this issue, and do not put the OXP implementation into the compatibility umbrella.

Both crates must follow the workspace MSRV/lints and build on Linux, Windows, and macOS without platform-specific APIs.

Scope

metacrate-oxp must:

decode OXP from bytes and any std::io::Read;
encode OXP to bytes and any std::io::Write;
provide ordinary path-based convenience functions using std::fs;
expose a typed OxpDocument model for manifest metadata, linksets, prims, inventory records, and embedded assets;
retain unknown LLSD keys/values so a decode/encode round trip does not silently discard compatible extensions;
validate structural references and report typed, contextual errors;
apply configurable resource limits before allocating attacker-controlled sizes;
preserve embedded asset bytes exactly.

This crate owns the file format only. Rezzing objects, uploading assets, enforcing grid permissions, converting JPEG 2000, parsing mesh payloads, and presenting UI are outside this issue. Assets are returned as typed metadata plus opaque bytes; no automatic extraction to disk is required.

File framing

An OXP file has no outer magic number, filename table, or archive directory.

Serialize one LLSD value using the binary LLSD encoding below. The root value must be a map.
Compress the complete binary LLSD byte stream as an RFC 1950 zlib stream (zlib header and Adler-32 trailer around DEFLATE data).
Store those compressed bytes directly in the .oxp file.

This is zlib, not gzip and not raw DEFLATE. The established writer uses maximum zlib compression, but interoperability is semantic: valid encoders need not produce byte-identical compressed output.

The decoder must validate the zlib wrapper/checksum, enforce compressed and decompressed-size limits while streaming, parse exactly one root value, and reject truncated or trailing decompressed data.

Binary LLSD encoding

Every value begins with a one-byte ASCII tag. All counts and numeric payloads are big-endian/network order.

LLSD value	Encoding
undefined	`!`
false / true	`0` / `1`
signed integer	`i` + 4-byte two's-complement i32
real	`r` + 8-byte IEEE-754 f64
UUID	`u` + 16 RFC 4122 bytes in network order
UTF-8 string	`s` + u32 byte length + bytes
date	`d` + 8-byte IEEE-754 seconds since Unix epoch
URI	`l` + u32 UTF-8 byte length + bytes
binary	`b` + u32 byte length + bytes
array	`[` + u32 element count + encoded values + `]`
map	`{` + u32 pair count + pairs + `}`

Each map key is encoded as k + u32 UTF-8 byte length + key bytes, immediately followed by its encoded value. Readers may accept the historical quoted notation-key form, but writers must emit k keys. Counts describe contained elements/pairs and the closing delimiter is still mandatory.

Reject negative/overflowing effective lengths, invalid UTF-8 in keys/strings/URIs, invalid UUID sizes, missing delimiters, count/delimiter disagreement, excessive nesting/node counts, and non-finite values where a typed OXP field requires a finite number.

Root document schema

The root is an LLSD map with these keys:

Key	Type	Required	Meaning
`format_version`	i32	yes	OXP container version. Current writer value is `2`. Import historical supported versions up to 2; reject missing, non-positive, and greater values as unsupported.
`client`	string	no	Exporting client/channel and version. Informational.
`creation_date`	string	no	Export date, conventionally UTC `YYYY-MM-DD`. This is a string, not LLSD Date.
`author`	string	no	Informational exporter/avatar name; omitted by anonymized exports.
`grid`	string	no	Informational grid label; omitted by anonymized exports.
`linkset`	array of arrays of UUID	yes	Ordered linksets. Each inner array contains the root UUID first, then child UUIDs in link order.
`prim`	map UUID-string -> prim map	yes	Prim records referenced by `linkset`.
`asset`	map UUID-string -> asset map	no	Embedded textures, materials, and object-content assets.
`inventory`	map UUID-string -> inventory map	no	Inventory item records referenced by prim `content`.
`lo_feature_version`	i32	no	Enhanced-feature schema version. Defined value is `1`.
`lo_asset_mask`	i32 bitmask	no	Export selections: `0x01` contents, `0x02` textures, `0x04` materials, `0x08` meshes.
`mesh_asset`	map UUID-string -> asset map	no	Embedded raw mesh assets in enhanced exports.

Root map ordering is not significant. Writers should use stable ordering for reproducible uncompressed LLSD. Unknown root keys must be retained.

The asset mask is descriptive, not proof that every referenced asset exists: permissions, unavailable cache data, or download failures can produce partial packages.

Prim schema

Each prim key is the original prim UUID in canonical text form. A prim map contains:

Core placement and shape

parent: UUID for a child prim; omitted for linkset roots.
attachment_point: i32 attachment-point identifier, roots only when applicable.
position: three f64 values [x, y, z]. Child positions are root-relative; root positions are region/world placement metadata.
scale: three f64 values.
rotation: four f64 values [x, y, z, w].
flags: LLSD binary containing exactly one big-endian u32.
material: i32 legacy prim material code.
clickaction: optional i32.
volume: map with:
- path: curve i32; begin, end, scale_x, scale_y, shear_x, shear_y, twist, twist_begin, radius_offset, taper_x, taper_y, revolutions, and skew as f64.
- profile: curve i32; begin, end, and hollow as f64.

A permissions-substituted default prim may contain only placement plus flags, volume, and material; do not require optional visual/metadata fields to exist.

Faces and legacy materials

texture: array indexed by face. Each entry contains:
- imageid UUID;
- colors four-f64 RGBA array;
- scales, scalet, offsets, offsett, imagerot, and optional glow as f64;
- bump, fullbright, and media_flags as i32;
- optional media_data map;
- optional gltf_override string containing a glTF material override JSON document;
- enhanced optional texgen i32: 0 default or 1 planar.
materials: optional array of legacy per-face material maps. Enhanced exports pad this array with default/null material records so its index matches the face index. A material map uses:
- UUIDs NormMap, SpecMap;
- i32 fixed-point fields NormOffsetX, NormOffsetY, NormRepeatX, NormRepeatY, NormRotation, SpecOffsetX, SpecOffsetY, SpecRepeatX, SpecRepeatY, SpecRotation; divide by 10000.0 for the represented value;
- SpecColor four-value color array;
- i32 SpecExp, EnvIntensity, DiffuseAlphaMode, and AlphaMaskCutoff.

Preserve array order exactly. Do not compact padded materials.

Optional prim parameter blocks

sculpt: { texture: UUID, type: i32 } for sculpted prims.
mesh: enhanced mesh-placeholder reference with the same shape, normally type 5; texture is the mesh UUID and the matching bytes are in mesh_asset.
flexible: air_friction, gravity, tension, wind_sensitivity f64; simulate_lod i32; user_force three-f64 array.
light: color four-f64 linear RGBA, plus f64 radius, falloff, and cutoff.
light_texture: texture UUID and params three-f64 array.
extended_mesh: flags i32; bit 0x01 means animated mesh.
render_material: { entries: [{ te_idx: i32, id: UUID }, ...] }, mapping face indices 0..255 to PBR material asset UUIDs.
reflection_probe: f64 ambiance, f64 clip_distance, i32 flags; bits are 0x01 box volume, 0x02 dynamic content, and 0x04 mirror.
particle: a map containing integer 0/1 flag keys LL_PART_INTERP_COLOR_MASK, LL_PART_INTERP_SCALE_MASK, LL_PART_BOUNCE_MASK, LL_PART_WIND_MASK, LL_PART_FOLLOW_SRC_MASK, LL_PART_FOLLOW_VELOCITY_MASK, LL_PART_TARGET_POS_MASK, LL_PART_TARGET_LINEAR_MASK, LL_PART_EMISSIVE_MASK, LL_PART_BEAM_MASK, LL_PART_SRC_PATTERN_DROP, LL_PART_SRC_PATTERN_EXPLODE, LL_PART_SRC_PATTERN_ANGLE, LL_PART_SRC_PATTERN_ANGLE_CONE, and LL_PART_SRC_PATTERN_ANGLE_CONE_EMPTY; f64 ParticleMaxAge, SourceMaxAge, SourceStartAge, InnerAngle, OuterAngle, BurstRate, BurstSpeedMin, BurstSpeedMax, BurstRadius; i32 BurstPartCount; arrays StartColor/ EndColor (four f64), StartScale/ EndScale (two f64), AngularVelocity (three f64); and UUID strings PartImageID, TargetId.

Physics, identity, permissions, and contents

ExtraPhysics: PhysicsShapeType i32 and f64 Density, Friction, Restitution, GravityMultiplier.
name, description, touch_name, sit_name: strings.
creation_date: LLSD binary containing exactly one big-endian u64.
Optional identity metadata: creator_id, owner_id, group_id, last_owner_id UUIDs and corresponding *_name LLSD maps. These may be absent after anonymization.
base_mask, owner_mask, group_mask, everyone_mask, next_owner_mask: LLSD binary containing exactly one big-endian u32 each.
sale_info: { sale_type: string, sale_price: i32 }.
content: ordered array of inventory-item UUIDs whose records are under root inventory.

Unknown prim keys and unknown keys inside recognized parameter maps must survive a round trip.

Embedded assets

Both asset and mesh_asset map canonical UUID strings to:

{
  name: string,
  description: string,
  type: string,
  data: binary
}

Known type values include texture, material, mesh, and ordinary inventory asset types such as scripts, notecards, sounds, animations, gestures, clothing, and body parts. Preserve unknown type strings.

Payload rules:

texture: original JPEG 2000/J2C bytes. System/default textures may be referenced but intentionally not embedded.
material: a second, uncompressed binary-LLSD document in data with root { version: "1.1", type: "GLTF 2.0", data: <glTF JSON string> }. Readers should also tolerate material asset version 1.0.
mesh: raw simulator mesh/SLM asset bytes with the viewer-cache-only 12-byte preamble removed. OXP treats this payload as opaque and must not add/remove a preamble.
Other types: original asset bytes, unchanged.

Do not derive filenames from untrusted metadata or write assets during decode. UUID-keyed extraction can be a later, explicit consumer concern.

Inventory schema

The root inventory map is keyed by inventory item UUID. Each record contains:

item_id, parent_id, asset_id: UUID;
name, desc, type, inv_type: string;
flags, created_at: i32;
sale_info: sale_type string and sale_price i32;
permissions: UUID creator_id, owner_id, last_owner_id, group_id; boolean is_owner_group; i32 base_mask, owner_mask, group_mask, everyone_mask, next_owner_mask.

Anonymized packages may zero identity UUIDs and set created_at to zero. A blank notecard may have its null asset ID replaced by a generated asset UUID whose asset payload is an empty serialized notecard. Missing assets must be represented as missing, not fabricated.

Import validation and compatibility

Validate every UUID map key.
Every UUID in every linkset must resolve to exactly one prim.
A prim must occur in at most one linkset; each linkset must be non-empty.
The first UUID is the linkset root. Child parent references, when present, must resolve inside that linkset and must not create cycles.
Asset and inventory references may be unresolved because valid exports can be partial. Report these as structured diagnostics on the decoded document, not fatal container errors.
Absence of lo_feature_version means standard OXP, but readers must still recognize enhanced fields by presence because early enhanced files were unlabelled.
For lo_feature_version >= 1, decode the known version-1 fields and preserve unknown future fields. Do not interpret lo_asset_mask as a completeness guarantee.
Reject unsupported format_version; do not silently guess another container format.
Never panic on malformed input.

Export behavior

Emit format_version = 2.
Emit standard OXP without the lo_*, mesh, or mesh_asset fields when no enhanced features are requested.
When enhanced features are present, emit lo_feature_version = 1, the accurate lo_asset_mask, face-aligned padded legacy materials, texgen, enhanced prim blocks, complete untruncated texture payloads, and mesh bytes under mesh_asset.
Preserve linkset order and root-first order.
Preserve original UUID references and embedded bytes; do not silently remap IDs.
Metadata/anonymization is caller-controlled. Omitting author/grid and identity maps must not change structural data.
Return an error rather than writing a partially serialized document when structural validation fails.

Minimal public API

Keep the API small and conventional:

OxpDocument plus typed child records;
OxpLimits with safe defaults for compressed bytes, decompressed bytes, nesting depth, total LLSD nodes, collection lengths, and total embedded asset bytes;
a non-exhaustive OxpError distinguishing I/O, zlib, binary LLSD, unsupported version, limit exceeded, invalid UUID/type, and structural-reference failures;
from_reader / from_bytes and to_writer / to_vec;
path helpers may delegate directly to those functions.

Do not add async APIs, plugin traits, factories, viewer-specific callbacks, asset conversion services, or a second archive abstraction in this issue.

Tests

Add deterministic, offline tests that cover:

Binary LLSD golden vectors for every tag, big-endian i32/f64/UUID/length handling, nested maps/arrays, UTF-8, and all malformed length/count/delimiter cases.
Zlib framing: valid RFC 1950 stream, checksum failure, truncation, gzip/raw-DEFLATE rejection, compressed/decompressed limits, and a high-expansion-ratio input.
A minimal standard version-2 OXP with one root and child, face data, legacy material data, physics, particle data, inventory, a J2C asset, and an ordinary content asset.
An enhanced version-1 feature document with lo_asset_mask = 0x0f, mesh placeholder plus opaque mesh_asset, animated-mesh flag, face-aligned materials, texgen, light texture, PBR material plus its nested binary LLSD/glTF JSON, glTF override, and reflection probe flags.
Decode -> encode -> decode semantic equality, exact preservation of embedded bytes and array ordering, and preservation of unknown fields.
Partial-but-valid packages with referenced default/missing assets and anonymized metadata.
Rejection of a future container version, invalid UUID keys, missing prims, duplicate prim placement, parent cycles, non-finite typed transforms, and limit violations.

Check in small repository-owned fixtures or construct them directly from the byte grammar above. Tests and implementation must not call another program, access sibling repositories, use a viewer cache, or require a network connection.

Documentation

Document in metacrate-oxp crate docs:

that OXP is zlib-wrapped binary LLSD rather than ZIP;
the root/prim/asset/inventory schema and feature-version rules above;
standard versus enhanced exports;
partial asset behavior and opaque mesh/J2C payloads;
resource-limit defaults and safe handling of untrusted files;
one import and one export example through the metacrate::oxp re-export.

Update the workspace README crate table and naming explanation: libremetaverse-* is compatibility-facing, while native functionality is metacrate-* and is collected by the metacrate umbrella.

Validation gates

The issue is complete only when all relevant workspace gates and these focused gates pass:

cargo test -p metacrate-oxp
cargo test -p metacrate
cargo check --workspace --all-targets
cargo build --workspace --all-features
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo fmt --all -- --check
python3 tools/check_test_parity.py
python3 tools/audit_red_suite.py

Also run the focused crate tests on Linux, Windows, and macOS. No platform may use a different on-disk representation.

Definition of done

A consumer can import standard and enhanced OXP bytes into a validated, typed native Rust document and export that document back to an interoperable .oxp file through metacrate::oxp. Linkset/face ordering, UUID references, unknown compatible fields, nested material data, and all embedded asset bytes survive round trips; malformed or oversized untrusted inputs fail with typed errors; and implementation/testing requires only this repository.

## Objective Implement complete, native Rust import and export of OXP (Onrez Package) object backups. An OXP file is a self-contained object-backup document containing linksets, prim properties, optional object inventory, and optional embedded assets. The implementation must be usable without any external reference checkout, viewer process, .NET runtime, cache directory, or network service. This issue contains the format contract required for implementation. ## Crate naming and workspace structure Add two native MetaCrate crates: - `metacrate-oxp`: the format implementation, typed document model, validation, bounded decoder, and encoder. - `metacrate`: the native umbrella crate, analogous to the existing compatibility umbrella, re-exporting `metacrate_oxp` as `oxp`. The `libremetaverse-*` prefix is reserved for compatibility-layer crates. New native MetaCrate functionality uses the `metacrate-*` prefix. Do not rename existing crates in this issue, and do not put the OXP implementation into the compatibility umbrella. Both crates must follow the workspace MSRV/lints and build on Linux, Windows, and macOS without platform-specific APIs. ## Scope `metacrate-oxp` must: - decode OXP from bytes and any `std::io::Read`; - encode OXP to bytes and any `std::io::Write`; - provide ordinary path-based convenience functions using `std::fs`; - expose a typed `OxpDocument` model for manifest metadata, linksets, prims, inventory records, and embedded assets; - retain unknown LLSD keys/values so a decode/encode round trip does not silently discard compatible extensions; - validate structural references and report typed, contextual errors; - apply configurable resource limits before allocating attacker-controlled sizes; - preserve embedded asset bytes exactly. This crate owns the file format only. Rezzing objects, uploading assets, enforcing grid permissions, converting JPEG 2000, parsing mesh payloads, and presenting UI are outside this issue. Assets are returned as typed metadata plus opaque bytes; no automatic extraction to disk is required. ## File framing An OXP file has no outer magic number, filename table, or archive directory. 1. Serialize one LLSD value using the binary LLSD encoding below. The root value must be a map. 2. Compress the complete binary LLSD byte stream as an RFC 1950 zlib stream (zlib header and Adler-32 trailer around DEFLATE data). 3. Store those compressed bytes directly in the `.oxp` file. This is zlib, not gzip and not raw DEFLATE. The established writer uses maximum zlib compression, but interoperability is semantic: valid encoders need not produce byte-identical compressed output. The decoder must validate the zlib wrapper/checksum, enforce compressed and decompressed-size limits while streaming, parse exactly one root value, and reject truncated or trailing decompressed data. ## Binary LLSD encoding Every value begins with a one-byte ASCII tag. All counts and numeric payloads are big-endian/network order. | LLSD value | Encoding | |---|---| | undefined | `!` | | false / true | `0` / `1` | | signed integer | `i` + 4-byte two's-complement i32 | | real | `r` + 8-byte IEEE-754 f64 | | UUID | `u` + 16 RFC 4122 bytes in network order | | UTF-8 string | `s` + u32 byte length + bytes | | date | `d` + 8-byte IEEE-754 seconds since Unix epoch | | URI | `l` + u32 UTF-8 byte length + bytes | | binary | `b` + u32 byte length + bytes | | array | `[` + u32 element count + encoded values + `]` | | map | `{` + u32 pair count + pairs + `}` | Each map key is encoded as `k` + u32 UTF-8 byte length + key bytes, immediately followed by its encoded value. Readers may accept the historical quoted notation-key form, but writers must emit `k` keys. Counts describe contained elements/pairs and the closing delimiter is still mandatory. Reject negative/overflowing effective lengths, invalid UTF-8 in keys/strings/URIs, invalid UUID sizes, missing delimiters, count/delimiter disagreement, excessive nesting/node counts, and non-finite values where a typed OXP field requires a finite number. ## Root document schema The root is an LLSD map with these keys: | Key | Type | Required | Meaning | |---|---|---:|---| | `format_version` | i32 | yes | OXP container version. Current writer value is `2`. Import historical supported versions up to 2; reject missing, non-positive, and greater values as unsupported. | | `client` | string | no | Exporting client/channel and version. Informational. | | `creation_date` | string | no | Export date, conventionally UTC `YYYY-MM-DD`. This is a string, not LLSD Date. | | `author` | string | no | Informational exporter/avatar name; omitted by anonymized exports. | | `grid` | string | no | Informational grid label; omitted by anonymized exports. | | `linkset` | array of arrays of UUID | yes | Ordered linksets. Each inner array contains the root UUID first, then child UUIDs in link order. | | `prim` | map UUID-string -> prim map | yes | Prim records referenced by `linkset`. | | `asset` | map UUID-string -> asset map | no | Embedded textures, materials, and object-content assets. | | `inventory` | map UUID-string -> inventory map | no | Inventory item records referenced by prim `content`. | | `lo_feature_version` | i32 | no | Enhanced-feature schema version. Defined value is `1`. | | `lo_asset_mask` | i32 bitmask | no | Export selections: `0x01` contents, `0x02` textures, `0x04` materials, `0x08` meshes. | | `mesh_asset` | map UUID-string -> asset map | no | Embedded raw mesh assets in enhanced exports. | Root map ordering is not significant. Writers should use stable ordering for reproducible uncompressed LLSD. Unknown root keys must be retained. The asset mask is descriptive, not proof that every referenced asset exists: permissions, unavailable cache data, or download failures can produce partial packages. ## Prim schema Each `prim` key is the original prim UUID in canonical text form. A prim map contains: ### Core placement and shape - `parent`: UUID for a child prim; omitted for linkset roots. - `attachment_point`: i32 attachment-point identifier, roots only when applicable. - `position`: three f64 values `[x, y, z]`. Child positions are root-relative; root positions are region/world placement metadata. - `scale`: three f64 values. - `rotation`: four f64 values `[x, y, z, w]`. - `flags`: LLSD binary containing exactly one big-endian u32. - `material`: i32 legacy prim material code. - `clickaction`: optional i32. - `volume`: map with: - `path`: `curve` i32; `begin`, `end`, `scale_x`, `scale_y`, `shear_x`, `shear_y`, `twist`, `twist_begin`, `radius_offset`, `taper_x`, `taper_y`, `revolutions`, and `skew` as f64. - `profile`: `curve` i32; `begin`, `end`, and `hollow` as f64. A permissions-substituted default prim may contain only placement plus `flags`, `volume`, and `material`; do not require optional visual/metadata fields to exist. ### Faces and legacy materials - `texture`: array indexed by face. Each entry contains: - `imageid` UUID; - `colors` four-f64 RGBA array; - `scales`, `scalet`, `offsets`, `offsett`, `imagerot`, and optional `glow` as f64; - `bump`, `fullbright`, and `media_flags` as i32; - optional `media_data` map; - optional `gltf_override` string containing a glTF material override JSON document; - enhanced optional `texgen` i32: `0` default or `1` planar. - `materials`: optional array of legacy per-face material maps. Enhanced exports pad this array with default/null material records so its index matches the face index. A material map uses: - UUIDs `NormMap`, `SpecMap`; - i32 fixed-point fields `NormOffsetX`, `NormOffsetY`, `NormRepeatX`, `NormRepeatY`, `NormRotation`, `SpecOffsetX`, `SpecOffsetY`, `SpecRepeatX`, `SpecRepeatY`, `SpecRotation`; divide by `10000.0` for the represented value; - `SpecColor` four-value color array; - i32 `SpecExp`, `EnvIntensity`, `DiffuseAlphaMode`, and `AlphaMaskCutoff`. Preserve array order exactly. Do not compact padded `materials`. ### Optional prim parameter blocks - `sculpt`: `{ texture: UUID, type: i32 }` for sculpted prims. - `mesh`: enhanced mesh-placeholder reference with the same shape, normally type `5`; `texture` is the mesh UUID and the matching bytes are in `mesh_asset`. - `flexible`: `air_friction`, `gravity`, `tension`, `wind_sensitivity` f64; `simulate_lod` i32; `user_force` three-f64 array. - `light`: `color` four-f64 linear RGBA, plus f64 `radius`, `falloff`, and `cutoff`. - `light_texture`: `texture` UUID and `params` three-f64 array. - `extended_mesh`: `flags` i32; bit `0x01` means animated mesh. - `render_material`: `{ entries: [{ te_idx: i32, id: UUID }, ...] }`, mapping face indices 0..255 to PBR material asset UUIDs. - `reflection_probe`: f64 `ambiance`, f64 `clip_distance`, i32 `flags`; bits are `0x01` box volume, `0x02` dynamic content, and `0x04` mirror. - `particle`: a map containing integer 0/1 flag keys `LL_PART_INTERP_COLOR_MASK`, `LL_PART_INTERP_SCALE_MASK`, `LL_PART_BOUNCE_MASK`, `LL_PART_WIND_MASK`, `LL_PART_FOLLOW_SRC_MASK`, `LL_PART_FOLLOW_VELOCITY_MASK`, `LL_PART_TARGET_POS_MASK`, `LL_PART_TARGET_LINEAR_MASK`, `LL_PART_EMISSIVE_MASK`, `LL_PART_BEAM_MASK`, `LL_PART_SRC_PATTERN_DROP`, `LL_PART_SRC_PATTERN_EXPLODE`, `LL_PART_SRC_PATTERN_ANGLE`, `LL_PART_SRC_PATTERN_ANGLE_CONE`, and `LL_PART_SRC_PATTERN_ANGLE_CONE_EMPTY`; f64 `ParticleMaxAge`, `SourceMaxAge`, `SourceStartAge`, `InnerAngle`, `OuterAngle`, `BurstRate`, `BurstSpeedMin`, `BurstSpeedMax`, `BurstRadius`; i32 `BurstPartCount`; arrays `StartColor`/ `EndColor` (four f64), `StartScale`/ `EndScale` (two f64), `AngularVelocity` (three f64); and UUID strings `PartImageID`, `TargetId`. ### Physics, identity, permissions, and contents - `ExtraPhysics`: `PhysicsShapeType` i32 and f64 `Density`, `Friction`, `Restitution`, `GravityMultiplier`. - `name`, `description`, `touch_name`, `sit_name`: strings. - `creation_date`: LLSD binary containing exactly one big-endian u64. - Optional identity metadata: `creator_id`, `owner_id`, `group_id`, `last_owner_id` UUIDs and corresponding `*_name` LLSD maps. These may be absent after anonymization. - `base_mask`, `owner_mask`, `group_mask`, `everyone_mask`, `next_owner_mask`: LLSD binary containing exactly one big-endian u32 each. - `sale_info`: `{ sale_type: string, sale_price: i32 }`. - `content`: ordered array of inventory-item UUIDs whose records are under root `inventory`. Unknown prim keys and unknown keys inside recognized parameter maps must survive a round trip. ## Embedded assets Both `asset` and `mesh_asset` map canonical UUID strings to: ```text { name: string, description: string, type: string, data: binary } ``` Known `type` values include `texture`, `material`, `mesh`, and ordinary inventory asset types such as scripts, notecards, sounds, animations, gestures, clothing, and body parts. Preserve unknown type strings. Payload rules: - `texture`: original JPEG 2000/J2C bytes. System/default textures may be referenced but intentionally not embedded. - `material`: a second, uncompressed binary-LLSD document in `data` with root `{ version: "1.1", type: "GLTF 2.0", data: <glTF JSON string> }`. Readers should also tolerate material asset version `1.0`. - `mesh`: raw simulator mesh/SLM asset bytes with the viewer-cache-only 12-byte preamble removed. OXP treats this payload as opaque and must not add/remove a preamble. - Other types: original asset bytes, unchanged. Do not derive filenames from untrusted metadata or write assets during decode. UUID-keyed extraction can be a later, explicit consumer concern. ## Inventory schema The root `inventory` map is keyed by inventory item UUID. Each record contains: - `item_id`, `parent_id`, `asset_id`: UUID; - `name`, `desc`, `type`, `inv_type`: string; - `flags`, `created_at`: i32; - `sale_info`: `sale_type` string and `sale_price` i32; - `permissions`: UUID `creator_id`, `owner_id`, `last_owner_id`, `group_id`; boolean `is_owner_group`; i32 `base_mask`, `owner_mask`, `group_mask`, `everyone_mask`, `next_owner_mask`. Anonymized packages may zero identity UUIDs and set `created_at` to zero. A blank notecard may have its null asset ID replaced by a generated asset UUID whose asset payload is an empty serialized notecard. Missing assets must be represented as missing, not fabricated. ## Import validation and compatibility - Validate every UUID map key. - Every UUID in every `linkset` must resolve to exactly one `prim`. - A prim must occur in at most one linkset; each linkset must be non-empty. - The first UUID is the linkset root. Child `parent` references, when present, must resolve inside that linkset and must not create cycles. - Asset and inventory references may be unresolved because valid exports can be partial. Report these as structured diagnostics on the decoded document, not fatal container errors. - Absence of `lo_feature_version` means standard OXP, but readers must still recognize enhanced fields by presence because early enhanced files were unlabelled. - For `lo_feature_version >= 1`, decode the known version-1 fields and preserve unknown future fields. Do not interpret `lo_asset_mask` as a completeness guarantee. - Reject unsupported `format_version`; do not silently guess another container format. - Never panic on malformed input. ## Export behavior - Emit `format_version = 2`. - Emit standard OXP without the `lo_*`, `mesh`, or `mesh_asset` fields when no enhanced features are requested. - When enhanced features are present, emit `lo_feature_version = 1`, the accurate `lo_asset_mask`, face-aligned padded legacy materials, `texgen`, enhanced prim blocks, complete untruncated texture payloads, and mesh bytes under `mesh_asset`. - Preserve linkset order and root-first order. - Preserve original UUID references and embedded bytes; do not silently remap IDs. - Metadata/anonymization is caller-controlled. Omitting author/grid and identity maps must not change structural data. - Return an error rather than writing a partially serialized document when structural validation fails. ## Minimal public API Keep the API small and conventional: - `OxpDocument` plus typed child records; - `OxpLimits` with safe defaults for compressed bytes, decompressed bytes, nesting depth, total LLSD nodes, collection lengths, and total embedded asset bytes; - a non-exhaustive `OxpError` distinguishing I/O, zlib, binary LLSD, unsupported version, limit exceeded, invalid UUID/type, and structural-reference failures; - `from_reader` / `from_bytes` and `to_writer` / `to_vec`; - path helpers may delegate directly to those functions. Do not add async APIs, plugin traits, factories, viewer-specific callbacks, asset conversion services, or a second archive abstraction in this issue. ## Tests Add deterministic, offline tests that cover: 1. Binary LLSD golden vectors for every tag, big-endian i32/f64/UUID/length handling, nested maps/arrays, UTF-8, and all malformed length/count/delimiter cases. 2. Zlib framing: valid RFC 1950 stream, checksum failure, truncation, gzip/raw-DEFLATE rejection, compressed/decompressed limits, and a high-expansion-ratio input. 3. A minimal standard version-2 OXP with one root and child, face data, legacy material data, physics, particle data, inventory, a J2C asset, and an ordinary content asset. 4. An enhanced version-1 feature document with `lo_asset_mask = 0x0f`, mesh placeholder plus opaque `mesh_asset`, animated-mesh flag, face-aligned materials, texgen, light texture, PBR material plus its nested binary LLSD/glTF JSON, glTF override, and reflection probe flags. 5. Decode -> encode -> decode semantic equality, exact preservation of embedded bytes and array ordering, and preservation of unknown fields. 6. Partial-but-valid packages with referenced default/missing assets and anonymized metadata. 7. Rejection of a future container version, invalid UUID keys, missing prims, duplicate prim placement, parent cycles, non-finite typed transforms, and limit violations. Check in small repository-owned fixtures or construct them directly from the byte grammar above. Tests and implementation must not call another program, access sibling repositories, use a viewer cache, or require a network connection. ## Documentation Document in `metacrate-oxp` crate docs: - that OXP is zlib-wrapped binary LLSD rather than ZIP; - the root/prim/asset/inventory schema and feature-version rules above; - standard versus enhanced exports; - partial asset behavior and opaque mesh/J2C payloads; - resource-limit defaults and safe handling of untrusted files; - one import and one export example through the `metacrate::oxp` re-export. Update the workspace README crate table and naming explanation: `libremetaverse-*` is compatibility-facing, while native functionality is `metacrate-*` and is collected by the `metacrate` umbrella. ## Validation gates The issue is complete only when all relevant workspace gates and these focused gates pass: ```sh cargo test -p metacrate-oxp cargo test -p metacrate cargo check --workspace --all-targets cargo build --workspace --all-features cargo clippy --workspace --all-targets --all-features -- -D warnings cargo fmt --all -- --check python3 tools/check_test_parity.py python3 tools/audit_red_suite.py ``` Also run the focused crate tests on Linux, Windows, and macOS. No platform may use a different on-disk representation. ## Definition of done A consumer can import standard and enhanced OXP bytes into a validated, typed native Rust document and export that document back to an interoperable `.oxp` file through `metacrate::oxp`. Linkset/face ordering, UUID references, unknown compatible fields, nested material data, and all embedded asset bytes survive round trips; malformed or oversized untrusted inputs fail with typed errors; and implementation/testing requires only this repository.

hugo added this to the 13 - Extensions milestone 2026-08-09 10:46:31 +00:00

hugo added the enhancement label 2026-08-09 10:46:31 +00:00

hugo referenced this issue

2026-08-09 11:12:18 +00:00

Add metacrate-oxp extract/create/list CLI #111

hugo added idea and removed enhancement labels 2026-08-13 04:25:26 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/MetaCrate#110