Add metacrate-oxp extract/create/list CLI #111

New Issue

Open

opened 2026-08-09 11:12:18 +00:00 by hugo · 0 comments

hugo commented 2026-08-09 11:12:18 +00:00

Owner

Copy Link

Objective

Add a cross-platform metacrate-oxp command-line tool with exactly three subcommands:

metacrate-oxp extract <file.oxp>
metacrate-oxp create <folder>
metacrate-oxp list <file.oxp>

This issue depends on #110. Implement the binary target in the same metacrate-oxp Cargo package as the OXP library; do not create another parser, archive abstraction, or CLI crate. All OXP decoding, validation, limits, and encoding must call the library from #110.

Path behavior

• extract /path/house.oxp creates /path/house/.
• create /path/house/ creates /path/house.oxp.
• The OXP extension check is ASCII case-insensitive; only the final .oxp extension is removed.
• extract and create must refuse to overwrite an existing target file or directory.
• Failed operations must not leave a partly written target. Write to a unique temporary sibling and rename it only after every file and the final OXP document have been validated and flushed.
• Accept relative or absolute paths through ordinary cross-platform std::path/std::fs; do not use platform-specific APIs.
• No stdin/stdout archive mode, recursive folder discovery, batch mode, or conversion flags are required.

Extracted directory format

Every extracted folder contains:

house/
  manifest.json
  <asset-uuid>.<type-extension>
  <asset-uuid>.<type-extension>
  ...

Asset filenames use the canonical lowercase hyphenated UUID from the OXP asset-map key. The original asset name and description are metadata only and must never be used as a path.

manifest.json is pretty-printed UTF-8 JSON with a final newline and this versioned top-level shape:

{
  "directory_format": "metacrate-oxp",
  "directory_format_version": 1,
  "source_file": "house.oxp",
  "document": {
    "...": "all OXP metadata except embedded payload bytes"
  },
  "assets": [
    {
      "uuid": "01234567-89ab-cdef-0123-456789abcdef",
      "source_map": "asset",
      "type": "texture",
      "name": "Brick",
      "description": "",
      "file": "01234567-89ab-cdef-0123-456789abcdef.j2c",
      "extensions": {}
    }
  ]
}

Requirements:

• document contains every non-payload part of OxpDocument: container/client metadata, feature version and asset mask, ordered linksets, all prim data, inventory, and all unknown compatible fields retained by #110.
• assets contains one record for every entry in both root asset and mesh_asset maps. source_map is exactly "asset" or "mesh_asset"; this is required to recreate the same root map.
• Replace each OXP asset entry's binary data value with its file reference. Retain name, description, declared type, and unknown asset-entry fields.
• File references are relative basenames only. Reject absolute paths, separators, ., .., empty names, non-canonical UUID stems, and extensions inconsistent with the mapping below.
• The JSON representation of known typed OXP fields should be natural and readable. Unknown LLSD extension values must use a documented tagged JSON representation for LLSD-only types (undefined, UUID, URI, date, binary, and values that plain JSON cannot preserve) so extract/create does not change their type or value.
• Any non-asset binary extension data remaining in metadata must be encoded losslessly in tagged JSON, using base64 with an explicit encoding tag. Embedded asset payloads themselves are always separate files, never base64 in manifest.json.
• Sort the assets array by source_map then UUID and serialize maps deterministically where ordering is not semantically significant. Preserve linkset, prim-face, material, render-material, content, and all other semantic array order.
• source_file is informational. create derives its output name from the folder, not from this field.

Do not emit a second metadata file or a copy of the compressed OXP.

Asset extension mapping

Choose the extension from the OXP asset record's declared type string. mesh_asset entries and declared mesh assets always use .asset.

OXP type string	Extracted extension	Payload
mesh or any mesh_asset entry	.asset	Raw simulator mesh/SLM asset bytes
texture	.j2c	JPEG 2000 codestream
sound	.ogg	Ogg Vorbis audio
animatn	.animatn	Simulator animation asset
notecard	.txt	Serialized notecard text; preserve bytes and line endings
lsltext, script	.lsl	LSL source/legacy script
lslbyte	.lslb	LSL bytecode
landmark	.landmark	Serialized landmark
callcard	.callingcard	Serialized calling card
clothing	.clothing	Serialized wearable
bodypart	.bodypart	Serialized body part
gesture	.gesture	Serialized gesture
txtr_tga, img_tga	.tga	Targa image
snd_wav	.wav	RIFF/WAVE audio
jpeg	.jpg	JPEG image
material	.material	Binary-LLSD-wrapped render material; do not unwrap to JSON
gltf	.gltf	glTF JSON asset
glbin	.glb	Binary glTF asset
settings	.settings	Serialized environment/settings asset
simstate	.simstate	Simulator-state asset
link	.link	Inventory link asset
link_f	.linkfolder	Inventory-folder link asset
category	.category	Inventory category asset
widget	.widget	Viewer widget asset
person	.person	Person-reference asset
object	.object	Serialized object asset from a nonstandard/older package
invalid, -1, reserved or unknown strings	.bin	Opaque bytes

The established object exporter normally skips embedded object and none inventory assets, stores meshes under mesh_asset, and most commonly embeds textures, sounds, landmarks, clothing/body parts, notecards, LSL source, animations, gestures, settings, and materials. The CLI must nevertheless extract every asset record actually present, including legacy, custom, reserved, and unknown types.

Extensions describe the payload but never authorize transcoding. extract writes the exact OXP data bytes; create reads them unchanged. In particular:

• do not decode/re-encode J2C, Ogg, WAV, TGA, JPEG, animation, mesh, or glTF;
• do not strip the notecard serialization header or normalize text/line endings;
• do not unwrap/rebuild .material payloads;
• do not add the viewer-cache preamble to .asset mesh files.

If the same UUID appears in both asset maps, one UUID filename may be shared only when declared type, chosen extension, and payload bytes are identical. Otherwise extraction must fail instead of overwriting one payload.

extract

extract must:

1. Open and decode the OXP through the bounded #110 API.
2. Validate the document before writing anything visible at the target path.
3. Create a temporary sibling directory.
4. Write every embedded asset to its computed UUID filename.
5. Write the version-1 manifest.json after all asset files succeed.
6. Flush/close files and atomically rename the temporary directory to the final same-named folder.
7. Print a concise success line with the folder path and extracted asset count.

Missing referenced assets are valid OXP diagnostics and do not create placeholder files. Only entries actually carrying embedded data appear in assets.

create

create must:

1. Require exactly one manifest.json in the supplied folder.
2. Reject an unsupported directory format/version before reading asset payloads.
3. Validate every asset UUID, source_map, declared type, expected filename, and safe relative basename.
4. Read only files explicitly referenced by assets. Ignore unrelated files such as .DS_Store; never recursively scan.
5. Reject missing files, directories, symbolic links, non-regular files, duplicate conflicting records, and files exceeding #110 limits.
6. Reconstruct the complete OxpDocument, restoring each payload to its original asset or mesh_asset record.
7. Run the same structural validation used by normal OXP encoding.
8. Encode to a temporary sibling file and atomically rename it to <folder-name>.oxp.
9. Print a concise success line with the OXP path and embedded asset count.

Creation must be semantically lossless, not compressed-byte-identical: decoding the original and recreated OXP must produce equal OXP documents, including unknown fields, and every embedded payload must be byte-identical.

JSON parse/type errors must report the JSON path or record/field name. Missing asset errors must include UUID and expected file path.

list

list decodes the OXP through #110 but writes no files. It lists only embedded asset records, from both asset and mesh_asset, sorted by source_map then UUID.

Write a stable tab-separated header and rows to stdout:

SOURCE  UUID  TYPE  BYTES  FILE  NAME  DESCRIPTION

• FILE is the filename extract would use.
• Escape tabs, carriage returns, newlines, and backslashes in name/description so each asset occupies exactly one output line.
• BYTES is the uncompressed payload length.
• Print the header and no data rows for a valid OXP with zero embedded assets.
• Diagnostics and errors go to stderr.
• list must not inspect or convert payload contents.

CLI behavior

• metacrate-oxp --help and each <command> --help document usage and path derivation.
• No command or invalid arguments: print concise usage to stderr and exit nonzero.
• Success exits 0; usage errors and data/I/O failures exit nonzero.
• Preserve the typed library error chain in human-readable diagnostics.
• Three fixed commands do not justify a plugin system, async runtime, progress framework, or service layer. Reuse an already accepted workspace argument parser if one exists when implemented; otherwise straightforward argument parsing is sufficient.

Tests

Add deterministic offline integration tests covering:

1. Path derivation for relative/absolute paths, multiple dots, spaces, Unicode, and uppercase .OXP.
2. The exact extension mapping above, including both mesh-map handling and unknown .bin fallback.
3. Extraction of a document containing at least texture, mesh, sound, animation, notecard, script, gesture, material, and unknown payloads; assert exact bytes and canonical filenames.
4. Original OXP -> extract -> create -> decoded-document semantic equality, unknown LLSD field retention, exact payload equality, and preserved semantic array order.
5. list ordering, columns, byte counts, escaping, empty-document output, and absence of filesystem writes.
6. Existing-target refusal and cleanup of temporary siblings after injected read/write/encode failures.
7. Invalid/unsupported JSON, unsafe paths, extension/type mismatch, missing files, symlinks, non-regular files, conflicting duplicate UUIDs, and resource-limit failures.
8. CLI help, success/error exit codes, stdout/stderr separation, and actionable errors.

Tests must use temporary directories and call the built binary as a subprocess. They must not access sibling repositories, viewer caches, network services, or platform-specific tools.

Documentation

Update metacrate-oxp crate and workspace documentation with:

• the three commands and exact path behavior;
• the version-1 manifest.json schema;
• the complete extension table;
• the guarantee that payloads are never transcoded;
• the overwrite, symlink, traversal, size-limit, and atomic-write behavior;
• a short extract/edit/create workflow example.

Validation gates

cargo test -p metacrate-oxp --all-targets
cargo run -p metacrate-oxp -- --help
cargo run -p metacrate-oxp -- extract --help
cargo run -p metacrate-oxp -- create --help
cargo run -p metacrate-oxp -- list --help
cargo check --workspace --all-targets
cargo build --workspace --all-features
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo fmt --all -- --check
python3 tools/check_test_parity.py
python3 tools/audit_red_suite.py

Run the focused CLI integration tests on Linux, Windows, and macOS.

Definition of done

A user can losslessly extract any supported OXP into a same-named folder with canonical UUID asset filenames and one complete manifest.json, inspect its embedded assets with list, and recreate a same-named OXP with create. The recreated document is semantically equal, asset payloads are byte-identical, no target is overwritten or left partial, and the same directory representation works on all supported platforms.

## Objective Add a cross-platform `metacrate-oxp` command-line tool with exactly three subcommands: ```text metacrate-oxp extract <file.oxp> metacrate-oxp create <folder> metacrate-oxp list <file.oxp> ``` This issue depends on #110. Implement the binary target in the same `metacrate-oxp` Cargo package as the OXP library; do not create another parser, archive abstraction, or CLI crate. All OXP decoding, validation, limits, and encoding must call the library from #110. ## Path behavior - `extract /path/house.oxp` creates `/path/house/`. - `create /path/house/` creates `/path/house.oxp`. - The OXP extension check is ASCII case-insensitive; only the final `.oxp` extension is removed. - `extract` and `create` must refuse to overwrite an existing target file or directory. - Failed operations must not leave a partly written target. Write to a unique temporary sibling and rename it only after every file and the final OXP document have been validated and flushed. - Accept relative or absolute paths through ordinary cross-platform `std::path`/`std::fs`; do not use platform-specific APIs. - No stdin/stdout archive mode, recursive folder discovery, batch mode, or conversion flags are required. ## Extracted directory format Every extracted folder contains: ```text house/ manifest.json <asset-uuid>.<type-extension> <asset-uuid>.<type-extension> ... ``` Asset filenames use the canonical lowercase hyphenated UUID from the OXP asset-map key. The original asset name and description are metadata only and must never be used as a path. `manifest.json` is pretty-printed UTF-8 JSON with a final newline and this versioned top-level shape: ```json { "directory_format": "metacrate-oxp", "directory_format_version": 1, "source_file": "house.oxp", "document": { "...": "all OXP metadata except embedded payload bytes" }, "assets": [ { "uuid": "01234567-89ab-cdef-0123-456789abcdef", "source_map": "asset", "type": "texture", "name": "Brick", "description": "", "file": "01234567-89ab-cdef-0123-456789abcdef.j2c", "extensions": {} } ] } ``` Requirements: - `document` contains every non-payload part of `OxpDocument`: container/client metadata, feature version and asset mask, ordered linksets, all prim data, inventory, and all unknown compatible fields retained by #110. - `assets` contains one record for every entry in both root `asset` and `mesh_asset` maps. `source_map` is exactly `"asset"` or `"mesh_asset"`; this is required to recreate the same root map. - Replace each OXP asset entry's binary `data` value with its `file` reference. Retain `name`, `description`, declared `type`, and unknown asset-entry fields. - File references are relative basenames only. Reject absolute paths, separators, `.`, `..`, empty names, non-canonical UUID stems, and extensions inconsistent with the mapping below. - The JSON representation of known typed OXP fields should be natural and readable. Unknown LLSD extension values must use a documented tagged JSON representation for LLSD-only types (undefined, UUID, URI, date, binary, and values that plain JSON cannot preserve) so extract/create does not change their type or value. - Any non-asset binary extension data remaining in metadata must be encoded losslessly in tagged JSON, using base64 with an explicit encoding tag. Embedded asset payloads themselves are always separate files, never base64 in `manifest.json`. - Sort the `assets` array by `source_map` then UUID and serialize maps deterministically where ordering is not semantically significant. Preserve linkset, prim-face, material, render-material, content, and all other semantic array order. - `source_file` is informational. `create` derives its output name from the folder, not from this field. Do not emit a second metadata file or a copy of the compressed OXP. ## Asset extension mapping Choose the extension from the OXP asset record's declared type string. `mesh_asset` entries and declared `mesh` assets always use `.asset`. | OXP type string | Extracted extension | Payload | |---|---|---| | `mesh` or any `mesh_asset` entry | `.asset` | Raw simulator mesh/SLM asset bytes | | `texture` | `.j2c` | JPEG 2000 codestream | | `sound` | `.ogg` | Ogg Vorbis audio | | `animatn` | `.animatn` | Simulator animation asset | | `notecard` | `.txt` | Serialized notecard text; preserve bytes and line endings | | `lsltext`, `script` | `.lsl` | LSL source/legacy script | | `lslbyte` | `.lslb` | LSL bytecode | | `landmark` | `.landmark` | Serialized landmark | | `callcard` | `.callingcard` | Serialized calling card | | `clothing` | `.clothing` | Serialized wearable | | `bodypart` | `.bodypart` | Serialized body part | | `gesture` | `.gesture` | Serialized gesture | | `txtr_tga`, `img_tga` | `.tga` | Targa image | | `snd_wav` | `.wav` | RIFF/WAVE audio | | `jpeg` | `.jpg` | JPEG image | | `material` | `.material` | Binary-LLSD-wrapped render material; do not unwrap to JSON | | `gltf` | `.gltf` | glTF JSON asset | | `glbin` | `.glb` | Binary glTF asset | | `settings` | `.settings` | Serialized environment/settings asset | | `simstate` | `.simstate` | Simulator-state asset | | `link` | `.link` | Inventory link asset | | `link_f` | `.linkfolder` | Inventory-folder link asset | | `category` | `.category` | Inventory category asset | | `widget` | `.widget` | Viewer widget asset | | `person` | `.person` | Person-reference asset | | `object` | `.object` | Serialized object asset from a nonstandard/older package | | `invalid`, `-1`, reserved or unknown strings | `.bin` | Opaque bytes | The established object exporter normally skips embedded `object` and `none` inventory assets, stores meshes under `mesh_asset`, and most commonly embeds textures, sounds, landmarks, clothing/body parts, notecards, LSL source, animations, gestures, settings, and materials. The CLI must nevertheless extract every asset record actually present, including legacy, custom, reserved, and unknown types. Extensions describe the payload but never authorize transcoding. `extract` writes the exact OXP `data` bytes; `create` reads them unchanged. In particular: - do not decode/re-encode J2C, Ogg, WAV, TGA, JPEG, animation, mesh, or glTF; - do not strip the notecard serialization header or normalize text/line endings; - do not unwrap/rebuild `.material` payloads; - do not add the viewer-cache preamble to `.asset` mesh files. If the same UUID appears in both asset maps, one UUID filename may be shared only when declared type, chosen extension, and payload bytes are identical. Otherwise extraction must fail instead of overwriting one payload. ## `extract` `extract` must: 1. Open and decode the OXP through the bounded #110 API. 2. Validate the document before writing anything visible at the target path. 3. Create a temporary sibling directory. 4. Write every embedded asset to its computed UUID filename. 5. Write the version-1 `manifest.json` after all asset files succeed. 6. Flush/close files and atomically rename the temporary directory to the final same-named folder. 7. Print a concise success line with the folder path and extracted asset count. Missing referenced assets are valid OXP diagnostics and do not create placeholder files. Only entries actually carrying embedded `data` appear in `assets`. ## `create` `create` must: 1. Require exactly one `manifest.json` in the supplied folder. 2. Reject an unsupported directory format/version before reading asset payloads. 3. Validate every asset UUID, `source_map`, declared type, expected filename, and safe relative basename. 4. Read only files explicitly referenced by `assets`. Ignore unrelated files such as `.DS_Store`; never recursively scan. 5. Reject missing files, directories, symbolic links, non-regular files, duplicate conflicting records, and files exceeding #110 limits. 6. Reconstruct the complete `OxpDocument`, restoring each payload to its original `asset` or `mesh_asset` record. 7. Run the same structural validation used by normal OXP encoding. 8. Encode to a temporary sibling file and atomically rename it to `<folder-name>.oxp`. 9. Print a concise success line with the OXP path and embedded asset count. Creation must be semantically lossless, not compressed-byte-identical: decoding the original and recreated OXP must produce equal OXP documents, including unknown fields, and every embedded payload must be byte-identical. JSON parse/type errors must report the JSON path or record/field name. Missing asset errors must include UUID and expected file path. ## `list` `list` decodes the OXP through #110 but writes no files. It lists only embedded asset records, from both `asset` and `mesh_asset`, sorted by `source_map` then UUID. Write a stable tab-separated header and rows to stdout: ```text SOURCE UUID TYPE BYTES FILE NAME DESCRIPTION ``` - `FILE` is the filename `extract` would use. - Escape tabs, carriage returns, newlines, and backslashes in name/description so each asset occupies exactly one output line. - `BYTES` is the uncompressed payload length. - Print the header and no data rows for a valid OXP with zero embedded assets. - Diagnostics and errors go to stderr. - `list` must not inspect or convert payload contents. ## CLI behavior - `metacrate-oxp --help` and each `<command> --help` document usage and path derivation. - No command or invalid arguments: print concise usage to stderr and exit nonzero. - Success exits 0; usage errors and data/I/O failures exit nonzero. - Preserve the typed library error chain in human-readable diagnostics. - Three fixed commands do not justify a plugin system, async runtime, progress framework, or service layer. Reuse an already accepted workspace argument parser if one exists when implemented; otherwise straightforward argument parsing is sufficient. ## Tests Add deterministic offline integration tests covering: 1. Path derivation for relative/absolute paths, multiple dots, spaces, Unicode, and uppercase `.OXP`. 2. The exact extension mapping above, including both mesh-map handling and unknown `.bin` fallback. 3. Extraction of a document containing at least texture, mesh, sound, animation, notecard, script, gesture, material, and unknown payloads; assert exact bytes and canonical filenames. 4. Original OXP -> `extract` -> `create` -> decoded-document semantic equality, unknown LLSD field retention, exact payload equality, and preserved semantic array order. 5. `list` ordering, columns, byte counts, escaping, empty-document output, and absence of filesystem writes. 6. Existing-target refusal and cleanup of temporary siblings after injected read/write/encode failures. 7. Invalid/unsupported JSON, unsafe paths, extension/type mismatch, missing files, symlinks, non-regular files, conflicting duplicate UUIDs, and resource-limit failures. 8. CLI help, success/error exit codes, stdout/stderr separation, and actionable errors. Tests must use temporary directories and call the built binary as a subprocess. They must not access sibling repositories, viewer caches, network services, or platform-specific tools. ## Documentation Update `metacrate-oxp` crate and workspace documentation with: - the three commands and exact path behavior; - the version-1 `manifest.json` schema; - the complete extension table; - the guarantee that payloads are never transcoded; - the overwrite, symlink, traversal, size-limit, and atomic-write behavior; - a short extract/edit/create workflow example. ## Validation gates ```sh cargo test -p metacrate-oxp --all-targets cargo run -p metacrate-oxp -- --help cargo run -p metacrate-oxp -- extract --help cargo run -p metacrate-oxp -- create --help cargo run -p metacrate-oxp -- list --help cargo check --workspace --all-targets cargo build --workspace --all-features cargo clippy --workspace --all-targets --all-features -- -D warnings cargo fmt --all -- --check python3 tools/check_test_parity.py python3 tools/audit_red_suite.py ``` Run the focused CLI integration tests on Linux, Windows, and macOS. ## Definition of done A user can losslessly extract any supported OXP into a same-named folder with canonical UUID asset filenames and one complete `manifest.json`, inspect its embedded assets with `list`, and recreate a same-named OXP with `create`. The recreated document is semantically equal, asset payloads are byte-identical, no target is overwritten or left partial, and the same directory representation works on all supported platforms.

hugo added this to the 13 - Extensions milestone 2026-08-09 11:12:18 +00:00

hugo added the enhancement label 2026-08-09 11:12:18 +00:00

hugo added idea and removed enhancement labels 2026-08-13 04:24:00 +00:00

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

No results found.

Labels

Clear labels

bug

Something is not working

critical

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

idea

ideas for later, not yet for implementation

invalid

Something is wrong

question

More information is needed

wontfix

This won't be fixed

No Label idea

Milestone

No items

No Milestone 13 - Extensions

Projects

Clear projects

No project

Assignees

Clear assignees

hugo

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/MetaCrate#111