hugo/MetaCrate
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Imaging extension: add optional pure-Rust JPEG 2000 backend #109

New Issue
Closed
opened 2026-08-09 06:45:03 +00:00 by hugo · 1 comment
hugo commented 2026-08-09 06:45:03 +00:00
Owner
Copy Link

Objective

Add a complete optional pure-Rust JPEG 2000 backend behind a Cargo feature named rust-j2k. Use the pure-Rust j2k crate from the current compatible 0.8 release line as the codec engine. The backend must implement the same LibreMetaverse/CoreJ2K behavior and the same bounded imaging contract as the existing OpenJPEG backend; it must not be a stub, decode-only fallback, or 8-bit convenience wrapper.

The existing C# source and translated/golden tests are the behavioral reference. Do not run or depend on .NET code.

Current architecture

  • libremetaverse-imaging/jpeg2000 enables the optional project-owned libremetaverse-openjpeg adapter.
  • crates/libremetaverse-imaging/src/jpeg2000.rs currently owns J2kCodec, J2kFormat, J2kCompression, J2kDecodeOptions, J2kEncodeOptions, component conversion, bounds enforcement, and OpenJPEG adaptation.
  • The umbrella libremetaverse/jpeg2000 feature forwards to that backend.
  • The workspace MSRV is Rust 1.96. Keep the selected j2k version compatible with that floor and record the exact audited version in Cargo.lock.

Feature and API design

  • Add optional dependency j2k and feature rust-j2k = ["dep:j2k"] to libremetaverse-imaging.
  • Add rust-j2k = ["libremetaverse-imaging/rust-j2k"] to the umbrella libremetaverse crate.
  • The rust-j2k feature must not enable, link, or transitively require libremetaverse-openjpeg, OpenJPEG, CMake, pkg-config, vcpkg, or another native codec.
  • Preserve the existing jpeg2000 feature and J2kCodec behavior without a breaking rename. Expose the Rust backend explicitly, for example as RustJ2kCodec, implementing ITextureCodec and the same byte/interleaved encode/decode operations.
  • Move shared public options, format/compression types, validation, bounds checks, and ManagedImage/component conversion into backend-neutral code rather than duplicating behavior.
  • Enabling jpeg2000 and rust-j2k together must compile and expose both backends. --all-features must remain valid. Do not use feature ordering or a conditional type alias that silently changes which backend J2kCodec means.

Required behavior

Decode

  • Accept raw Part 1 J2K/J2C codestreams and JP2 containers.
  • Preserve image dimensions and native component planes before ManagedImage conversion.
  • Preserve component count, ordering, precision, signedness, and JP2 alpha metadata. Support the 1 through 5 component layouts exercised by the existing codec abstraction and reject unsupported/subsampled layouts with the same typed behavior as the OpenJPEG path.
  • Correctly convert the reference gray, RGB, RGBA, alpha-only, and bump/alpha compatibility cases into ManagedImage.
  • Honor discard_levels, quality_layers, and strict/truncated-input behavior. Zero quality layers means all layers. Do not accept and ignore an option: if the upstream facade lacks a required control, add a real bounded adapter or contribute the missing capability upstream before completing this issue.
  • Detect malformed containers/codestreams and return existing typed Error variants without panic.

Encode

  • Produce both raw J2K codestream and JP2 output according to J2kFormat.
  • Implement reversible 5/3 lossless encoding with exact sample reconstruction.
  • Implement irreversible 9/7 lossy encoding using the requested finite compression ratio. Preserve the existing validation that rejects ratios below 1.0, NaN, and infinity.
  • Preserve the four-component CoreJ2K compatibility view used by the current backend: RGB plus optional alpha, alpha-only substitution, opaque alpha insertion, and omission of bump as an encoded component.
  • Enforce max_encoded_bytes and return a typed failure rather than truncating output.

Resource safety

  • Reject empty or oversized encoded input before buffering.
  • Validate dimensions, component counts, pixel counts, sample lengths, strides, and all integer conversions before allocation or indexing.
  • Use bounded/streaming j2k APIs where available. Decoder-controlled dimensions must never cause allocation before max_pixels and DEFAULT_MAX_PIXELS are enforced.
  • Preserve the workspace prohibition on unsafe code in project crates. Any unsafe code inside dependencies must remain encapsulated and be covered by the dependency/license audit.
  • Do not add platform-specific APIs. The same Rust implementation must build on Linux, Windows, and macOS.

Tests

Refactor the codec contract tests so the same assertions run independently against OpenJPEG and RustJ2kCodec, while retaining backend-specific tests where needed. Add at least:

  • Raw J2K and JP2 lossless round trips for gray, RGB, RGBA, and alpha substitution.
  • Lossy 9/7 encoding at multiple compression ratios with decoded quality and size assertions that are stable across platforms.
  • Native precision/signedness/component-order fixtures, including the existing five-component interleaved fixture.
  • Reduced-resolution decode and progressive quality-layer decode using multi-resolution/multi-layer fixtures.
  • Strict versus permissive truncated-input behavior.
  • Empty, malformed, oversized, impossible-dimension, excessive-component, and output-limit failures.
  • Interoperability tests in the combined feature build: Rust-encoded streams decode through OpenJPEG and existing OpenJPEG fixtures decode through Rust. Require pixel equivalence for lossless data; do not require byte-identical codestreams unless the reference fixture explicitly does.

Tests must be deterministic, offline, and must not generate fixtures by invoking C#, OpenJPEG command-line programs, or network services at test time. Small audited fixtures may be checked in with provenance.

Validation gates

The implementation is complete only when all of the following pass:

cargo check --workspace --all-targets
cargo test -p libremetaverse-imaging --no-default-features --features rust-j2k
cargo test -p libremetaverse --no-default-features --features rust-j2k
cargo test -p libremetaverse-imaging --no-default-features --features jpeg2000,rust-j2k
cargo build --workspace --all-features
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo fmt --all -- --check
python3 tools/check_test_parity.py
python3 tools/audit_red_suite.py

Also verify the pure-Rust feature dependency tree contains neither libremetaverse-openjpeg nor a native OpenJPEG binding, and add CI coverage for Rust 1.96 plus current stable on Linux, Windows, and macOS.

Documentation and audit

  • Update the main README optional codec table with rust-j2k build and test commands and explain how it differs from jpeg2000.
  • Update docs/imaging-meshing.md and relevant crate documentation with backend selection, supported formats/options, feature coexistence, MSRV, limits, and deployment implications.
  • Record the j2k version, license, repository, safety posture, and relevant transitive dependencies in the dependency/license audit.
  • Remove the current README statement saying that rust-j2k is only planned once the feature is actually usable.

Definition of done

A user can build MetaCrate with --no-default-features --features rust-j2k, encode and decode compatible J2K and JP2 textures entirely in Rust, and pass the existing C#-derived imaging requirements without installing OpenJPEG or any other native JPEG 2000 library. Existing default and OpenJPEG builds remain unchanged, and every feature combination above is tested in isolation.

## Objective Add a complete optional pure-Rust JPEG 2000 backend behind a Cargo feature named `rust-j2k`. Use the pure-Rust `j2k` crate from the current compatible 0.8 release line as the codec engine. The backend must implement the same LibreMetaverse/CoreJ2K behavior and the same bounded imaging contract as the existing OpenJPEG backend; it must not be a stub, decode-only fallback, or 8-bit convenience wrapper. The existing C# source and translated/golden tests are the behavioral reference. Do not run or depend on .NET code. ## Current architecture - `libremetaverse-imaging/jpeg2000` enables the optional project-owned `libremetaverse-openjpeg` adapter. - `crates/libremetaverse-imaging/src/jpeg2000.rs` currently owns `J2kCodec`, `J2kFormat`, `J2kCompression`, `J2kDecodeOptions`, `J2kEncodeOptions`, component conversion, bounds enforcement, and OpenJPEG adaptation. - The umbrella `libremetaverse/jpeg2000` feature forwards to that backend. - The workspace MSRV is Rust 1.96. Keep the selected `j2k` version compatible with that floor and record the exact audited version in `Cargo.lock`. ## Feature and API design - Add optional dependency `j2k` and feature `rust-j2k = ["dep:j2k"]` to `libremetaverse-imaging`. - Add `rust-j2k = ["libremetaverse-imaging/rust-j2k"]` to the umbrella `libremetaverse` crate. - The `rust-j2k` feature must not enable, link, or transitively require `libremetaverse-openjpeg`, OpenJPEG, CMake, pkg-config, vcpkg, or another native codec. - Preserve the existing `jpeg2000` feature and `J2kCodec` behavior without a breaking rename. Expose the Rust backend explicitly, for example as `RustJ2kCodec`, implementing `ITextureCodec` and the same byte/interleaved encode/decode operations. - Move shared public options, format/compression types, validation, bounds checks, and ManagedImage/component conversion into backend-neutral code rather than duplicating behavior. - Enabling `jpeg2000` and `rust-j2k` together must compile and expose both backends. `--all-features` must remain valid. Do not use feature ordering or a conditional type alias that silently changes which backend `J2kCodec` means. ## Required behavior ### Decode - Accept raw Part 1 J2K/J2C codestreams and JP2 containers. - Preserve image dimensions and native component planes before ManagedImage conversion. - Preserve component count, ordering, precision, signedness, and JP2 alpha metadata. Support the 1 through 5 component layouts exercised by the existing codec abstraction and reject unsupported/subsampled layouts with the same typed behavior as the OpenJPEG path. - Correctly convert the reference gray, RGB, RGBA, alpha-only, and bump/alpha compatibility cases into `ManagedImage`. - Honor `discard_levels`, `quality_layers`, and strict/truncated-input behavior. Zero quality layers means all layers. Do not accept and ignore an option: if the upstream facade lacks a required control, add a real bounded adapter or contribute the missing capability upstream before completing this issue. - Detect malformed containers/codestreams and return existing typed `Error` variants without panic. ### Encode - Produce both raw J2K codestream and JP2 output according to `J2kFormat`. - Implement reversible 5/3 lossless encoding with exact sample reconstruction. - Implement irreversible 9/7 lossy encoding using the requested finite compression ratio. Preserve the existing validation that rejects ratios below 1.0, NaN, and infinity. - Preserve the four-component CoreJ2K compatibility view used by the current backend: RGB plus optional alpha, alpha-only substitution, opaque alpha insertion, and omission of bump as an encoded component. - Enforce `max_encoded_bytes` and return a typed failure rather than truncating output. ### Resource safety - Reject empty or oversized encoded input before buffering. - Validate dimensions, component counts, pixel counts, sample lengths, strides, and all integer conversions before allocation or indexing. - Use bounded/streaming `j2k` APIs where available. Decoder-controlled dimensions must never cause allocation before `max_pixels` and `DEFAULT_MAX_PIXELS` are enforced. - Preserve the workspace prohibition on unsafe code in project crates. Any unsafe code inside dependencies must remain encapsulated and be covered by the dependency/license audit. - Do not add platform-specific APIs. The same Rust implementation must build on Linux, Windows, and macOS. ## Tests Refactor the codec contract tests so the same assertions run independently against OpenJPEG and `RustJ2kCodec`, while retaining backend-specific tests where needed. Add at least: - Raw J2K and JP2 lossless round trips for gray, RGB, RGBA, and alpha substitution. - Lossy 9/7 encoding at multiple compression ratios with decoded quality and size assertions that are stable across platforms. - Native precision/signedness/component-order fixtures, including the existing five-component interleaved fixture. - Reduced-resolution decode and progressive quality-layer decode using multi-resolution/multi-layer fixtures. - Strict versus permissive truncated-input behavior. - Empty, malformed, oversized, impossible-dimension, excessive-component, and output-limit failures. - Interoperability tests in the combined feature build: Rust-encoded streams decode through OpenJPEG and existing OpenJPEG fixtures decode through Rust. Require pixel equivalence for lossless data; do not require byte-identical codestreams unless the reference fixture explicitly does. Tests must be deterministic, offline, and must not generate fixtures by invoking C#, OpenJPEG command-line programs, or network services at test time. Small audited fixtures may be checked in with provenance. ## Validation gates The implementation is complete only when all of the following pass: ```sh cargo check --workspace --all-targets cargo test -p libremetaverse-imaging --no-default-features --features rust-j2k cargo test -p libremetaverse --no-default-features --features rust-j2k cargo test -p libremetaverse-imaging --no-default-features --features jpeg2000,rust-j2k cargo build --workspace --all-features cargo clippy --workspace --all-targets --all-features -- -D warnings cargo fmt --all -- --check python3 tools/check_test_parity.py python3 tools/audit_red_suite.py ``` Also verify the pure-Rust feature dependency tree contains neither `libremetaverse-openjpeg` nor a native OpenJPEG binding, and add CI coverage for Rust 1.96 plus current stable on Linux, Windows, and macOS. ## Documentation and audit - Update the main README optional codec table with `rust-j2k` build and test commands and explain how it differs from `jpeg2000`. - Update `docs/imaging-meshing.md` and relevant crate documentation with backend selection, supported formats/options, feature coexistence, MSRV, limits, and deployment implications. - Record the `j2k` version, license, repository, safety posture, and relevant transitive dependencies in the dependency/license audit. - Remove the current README statement saying that `rust-j2k` is only planned once the feature is actually usable. ## Definition of done A user can build MetaCrate with `--no-default-features --features rust-j2k`, encode and decode compatible J2K and JP2 textures entirely in Rust, and pass the existing C#-derived imaging requirements without installing OpenJPEG or any other native JPEG 2000 library. Existing default and OpenJPEG builds remain unchanged, and every feature combination above is tested in isolation.
hugo added this to the 13 - Extensions milestone 2026-08-09 06:45:03 +00:00
hugo added the enhancement label 2026-08-09 06:45:03 +00:00
hugo commented 2026-08-13 06:39:58 +00:00
Author
Owner
Copy Link

Implemented the optional pure-Rust JPEG 2000 backend and verified it end to end.

Implementation:

  • Added the independently selectable rust-j2k feature using pinned j2k 0.8.1 without OpenJPEG or native codec dependencies.
  • Added RustJ2kCodec with raw J2K and JP2 lossless 5/3 and lossy 9/7 encode/decode, native precision and signed component planes, gray/RGB/RGBA/alpha/bump compatibility conversion, real discard-level and progressive quality-layer handling, strict/permissive truncation behavior, and typed resource-limit failures.
  • Preserved J2kCodec as the OpenJPEG backend and made both backends coexist under all-features, including umbrella texture workflows.
  • Added deterministic contract and interoperability coverage for 1-5 components, signed/mixed precision, subsampling and excessive-component rejection, progressive layers, reduced resolution, malformed/truncated inputs, and output caps.
  • Updated feature matrices, MSRV/stable and portable cross-target coverage, API/provenance/dependency evidence, and user/deployment documentation.

Verification:

  • 19 rust-j2k imaging tests passed independently.
  • 26 combined jpeg2000,rust-j2k tests passed, including bidirectional lossless interoperability.
  • Umbrella rust-j2k tests, Rust 1.96, stable Linux, Windows GNU cross-check, and macOS cross-check passed.
  • Workspace check, all-features build, all-target/all-feature clippy with warnings denied, formatting, parity audit, API/dependency/provenance audits, and pure-Rust dependency-tree review passed.
  • Gitea required CI run #401 passed on ubuntu-latest ARM64 in 10m44s: https://git.rfc1437.de/hugo/MetaCrate/actions/runs/401

Primary implementation commit: 0bab5b4. Protected-main follow-up commits refresh deterministic audit evidence and serialize ARM64 Cargo work to avoid all-features clippy OOM; final verified tip: e1744f4.

Implemented the optional pure-Rust JPEG 2000 backend and verified it end to end. Implementation: - Added the independently selectable rust-j2k feature using pinned j2k 0.8.1 without OpenJPEG or native codec dependencies. - Added RustJ2kCodec with raw J2K and JP2 lossless 5/3 and lossy 9/7 encode/decode, native precision and signed component planes, gray/RGB/RGBA/alpha/bump compatibility conversion, real discard-level and progressive quality-layer handling, strict/permissive truncation behavior, and typed resource-limit failures. - Preserved J2kCodec as the OpenJPEG backend and made both backends coexist under all-features, including umbrella texture workflows. - Added deterministic contract and interoperability coverage for 1-5 components, signed/mixed precision, subsampling and excessive-component rejection, progressive layers, reduced resolution, malformed/truncated inputs, and output caps. - Updated feature matrices, MSRV/stable and portable cross-target coverage, API/provenance/dependency evidence, and user/deployment documentation. Verification: - 19 rust-j2k imaging tests passed independently. - 26 combined jpeg2000,rust-j2k tests passed, including bidirectional lossless interoperability. - Umbrella rust-j2k tests, Rust 1.96, stable Linux, Windows GNU cross-check, and macOS cross-check passed. - Workspace check, all-features build, all-target/all-feature clippy with warnings denied, formatting, parity audit, API/dependency/provenance audits, and pure-Rust dependency-tree review passed. - Gitea required CI run #401 passed on ubuntu-latest ARM64 in 10m44s: https://git.rfc1437.de/hugo/MetaCrate/actions/runs/401 Primary implementation commit: 0bab5b4. Protected-main follow-up commits refresh deterministic audit evidence and serialize ARM64 Cargo work to avoid all-features clippy OOM; final verified tip: e1744f4.
hugo closed this issue 2026-08-13 06:40:07 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something is not working
 critical
duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 idea
ideas for later, not yet for implementation
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No Label enhancement
Milestone
No items
No Milestone 13 - Extensions
Projects
Clear projects
No project
Assignees
Clear assignees
hugo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: hugo/MetaCrate#109
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?