Generate Types and StructuredData API shims

This commit is contained in:
2026-08-08 11:56:13 +02:00
parent d3f5e27f52
commit 0a2e7dd057
15 changed files with 12120 additions and 5154 deletions

View File

@@ -7,9 +7,9 @@ host, bind to, invoke, or ship the .NET implementation. The current
stage is a compiling structural shell: crate boundaries mirror the .NET library stage is a compiling structural shell: crate boundaries mirror the .NET library
projects, public C# type names have Rust declarations, every upstream NUnit case projects, public C# type names have Rust declarations, every upstream NUnit case
has a traceable placeholder, and every sample/tool project has a Rust binary has a traceable placeholder, and every sample/tool project has a Rust binary
target. Callable member signatures and semantic Rust test translations are the target. The Types and StructuredData assemblies now have complete callable,
next two required gates; the current generated declarations are not yet the failure-only signatures; the remaining assembly slices and semantic Rust test
complete public API shim. translations are still required before the public API shim is complete.
The source snapshot, compatibility rules, dependency research, and ordered The source snapshot, compatibility rules, dependency research, and ordered
implementation plan are in [RUSTREWRITE.md](RUSTREWRITE.md). implementation plan are in [RUSTREWRITE.md](RUSTREWRITE.md).
@@ -27,6 +27,7 @@ cannot look like successful behavior.
cargo build --workspace cargo build --workspace
cargo test --workspace --no-run cargo test --workspace --no-run
python3 tools/generate_rust_mapping.py --check python3 tools/generate_rust_mapping.py --check
python3 tools/generate_api_shims.py --check
``` ```
Running `cargo test --workspace` is intentionally red during the shim stage. Running `cargo test --workspace` is intentionally red during the shim stage.

View File

@@ -45,23 +45,27 @@ in the same change.
The current workspace is a structural baseline, not a working client: The current workspace is a structural baseline, not a working client:
- all 13 public .NET library projects have corresponding Rust crates; - all 13 public .NET library projects have corresponding Rust crates;
- 1,705 public C# type declarations produce compiling Rust type/trait shims; - 1,705 source-discovered public C# type declarations produce compiling Rust
type/trait shims;
- 12,195 public declaration lines are retained in the API ledger; - 12,195 public declaration lines are retained in the API ledger;
- compiled metadata records 3,066 public types and 30,789 public members with - compiled metadata records 3,066 public types and 30,789 public members with
matching independent ECMA-335 counts, with all 142 external signature types matching independent ECMA-335 counts, with all 142 external signature types
resolved to cross-platform Rust or project-owned boundary types; resolved to cross-platform Rust or project-owned boundary types;
- the Types and StructuredData slices expose 61 final Rust-facing types and all
1,237 callable members with real fields/constants/enum values and standardized
failing bodies;
- all 1,295 NUnit `[Test]`/`[TestCase]` invocations have compiling Rust test - all 1,295 NUnit `[Test]`/`[TestCase]` invocations have compiling Rust test
entries that intentionally panic with their source identity and body hash; entries that intentionally panic with their source identity and body hash;
- all nine sample/tool projects have compiling Rust binary targets; - all nine sample/tool projects have compiling Rust binary targets;
- the 128 TestClient command source files are retained as a command inventory. - the 128 TestClient command source files are retained as a command inventory.
The generated type shims preserve names and module placement, but they do not The remaining generated type shims preserve names and module placement, but do
yet expose callable C# methods, properties, fields, event payloads, enum values, not yet expose their callable members. `api/SURFACE.tsv` is an inventory aid,
or delegate signatures. `api/SURFACE.tsv` is an inventory aid, not proof of API not proof of API coverage: it records declaration lines but does not fully parse
coverage: it records declaration lines but does not fully parse multiline multiline signatures. Likewise, a generated `pending` test is a parity
signatures. Likewise, a generated `pending` test is a parity placeholder, not a placeholder, not a semantic translation. **This repository is therefore at
semantic translation. **This repository is therefore at Stage 0 and does not Stage 0 and does not yet satisfy either the full API-shim gate or the full
yet satisfy either the full API-shim gate or the full test-suite gate.** test-suite gate.**
No functional porting starts until both missing gates are complete across the No functional porting starts until both missing gates are complete across the
entire workspace, rather than one crate at a time: entire workspace, rather than one crate at a time:

View File

@@ -48,7 +48,7 @@ External member APIs are not copied. Each signature type resolves to Rust core/s
- Delegates and flags enums: **40 / 46** - Delegates and flags enums: **40 / 46**
- Optional/ref/out/in parameters: **470 / 6,144 / 555 / 0** - Optional/ref/out/in parameters: **470 / 6,144 / 555 / 0**
- Async Rust signatures: **360** - Async Rust signatures: **360**
- Descriptively renamed overloads: **7,131** - Descriptively renamed overloads: **7,137**
- Explicit CLR delegate APM differences: **80** - Explicit CLR delegate APM differences: **80**
Run `python3 tools/generate_rust_mapping.py --check` to regenerate in memory and enforce the complete gate. Run `python3 tools/generate_rust_mapping.py --check` to regenerate in memory and enforce the complete gate.

View File

@@ -37,6 +37,8 @@ Regenerate or validate the Rust mapping with only Python's standard library:
```sh ```sh
python3 tools/generate_rust_mapping.py python3 tools/generate_rust_mapping.py
python3 tools/generate_rust_mapping.py --check python3 tools/generate_rust_mapping.py --check
python3 tools/generate_api_shims.py
python3 tools/generate_api_shims.py --check
``` ```
The checker rejects duplicate C# IDs, duplicate Rust destinations, invalid The checker rejects duplicate C# IDs, duplicate Rust destinations, invalid
@@ -48,3 +50,9 @@ fallbacks, and generated function bodies that do not use the standardized
The two support traits in `RUST-TYPES.tsv` are included because compiled public The two support traits in `RUST-TYPES.tsv` are included because compiled public
inheritance records reference them even though they are absent from the public inheritance records reference them even though they are absent from the public
type table. type table.
`generate_api_shims.py` consumes the catalog and reviewed mapping ledger. It
currently owns the completed Types and StructuredData slices, formats their
generated Rust deterministically, and rejects any missing or stale type/member
documentation ID. `SHIM-COVERAGE.md` distinguishes callable slices from the
remaining milestone work; mapping coverage alone is not callable coverage.

File diff suppressed because it is too large Load Diff

View File

@@ -3092,7 +3092,7 @@ T:System.Action`1 System.Action<T> System.Private.CoreLib delegate core callback
T:System.Action`2 System.Action<T1, T2> System.Private.CoreLib delegate core callback Box<dyn Fn(T1, T2) + Send + Sync> language not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Action`2 System.Action<T1, T2> System.Private.CoreLib delegate core callback Box<dyn Fn(T1, T2) + Send + Sync> language not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Action`3 System.Action<T1, T2, T3> System.Private.CoreLib delegate core callback Box<dyn Fn(T1, T2, T3) + Send + Sync> language not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Action`3 System.Action<T1, T2, T3> System.Private.CoreLib delegate core callback Box<dyn Fn(T1, T2, T3) + Send + Sync> language not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.ApplicationException System.ApplicationException System.Private.CoreLib class libremetaverse-types compat libremetaverse_types::compat::ApplicationException native_cross_platform_replacement not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.ApplicationException System.ApplicationException System.Private.CoreLib class libremetaverse-types compat libremetaverse_types::compat::ApplicationException native_cross_platform_replacement not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Array System.Array System.Private.CoreLib class std Vec<T> stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Array System.Array System.Private.CoreLib class libremetaverse-types libremetaverse_types::compat::Array stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.AsyncCallback System.AsyncCallback System.Private.CoreLib delegate Box<dyn Fn(&dyn std Box<dyn Fn(&dyn std::any::Any) + Send + Sync> stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.AsyncCallback System.AsyncCallback System.Private.CoreLib delegate Box<dyn Fn(&dyn std Box<dyn Fn(&dyn std::any::Any) + Send + Sync> stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Attribute System.Attribute System.Private.CoreLib class libremetaverse-types libremetaverse_types::compat::Attribute stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Attribute System.Attribute System.Private.CoreLib class libremetaverse-types libremetaverse_types::compat::Attribute stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Boolean System.Boolean System.Private.CoreLib struct core primitive bool language not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Boolean System.Boolean System.Private.CoreLib struct core primitive bool language not_applicable not_applicable not_applicable mapped external member surface is not copied
@@ -3135,7 +3135,7 @@ T:System.Func`1 System.Func<TResult> System.Private.CoreLib delegate core callba
T:System.Func`2 System.Func<T, TResult> System.Private.CoreLib delegate core callback Box<dyn Fn(T) -> TResult + Send + Sync> language not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Func`2 System.Func<T, TResult> System.Private.CoreLib delegate core callback Box<dyn Fn(T) -> TResult + Send + Sync> language not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Globalization.CultureInfo System.Globalization.CultureInfo System.Private.CoreLib class libremetaverse-types compat libremetaverse_types::compat::CultureInfo native_cross_platform_replacement not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Globalization.CultureInfo System.Globalization.CultureInfo System.Private.CoreLib class libremetaverse-types compat libremetaverse_types::compat::CultureInfo native_cross_platform_replacement not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Globalization.UnicodeCategory System.Globalization.UnicodeCategory System.Private.CoreLib enum libremetaverse-types compat libremetaverse_types::compat::UnicodeCategory native_cross_platform_replacement not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Globalization.UnicodeCategory System.Globalization.UnicodeCategory System.Private.CoreLib enum libremetaverse-types compat libremetaverse_types::compat::UnicodeCategory native_cross_platform_replacement not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Guid System.Guid System.Private.CoreLib struct uuid uuid::Uuid stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Guid System.Guid System.Private.CoreLib struct libremetaverse-types libremetaverse_types::compat::Guid stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.IAsyncDisposable System.IAsyncDisposable System.Private.CoreLib interface libremetaverse-types compat libremetaverse_types::compat::IAsyncDisposable native_cross_platform_replacement not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.IAsyncDisposable System.IAsyncDisposable System.Private.CoreLib interface libremetaverse-types compat libremetaverse_types::compat::IAsyncDisposable native_cross_platform_replacement not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.IAsyncResult System.IAsyncResult System.Private.CoreLib interface Box<dyn std Box<dyn std::any::Any + Send + Sync> stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.IAsyncResult System.IAsyncResult System.Private.CoreLib interface Box<dyn std Box<dyn std::any::Any + Send + Sync> stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.ICloneable System.ICloneable System.Private.CoreLib interface std Clone stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.ICloneable System.ICloneable System.Private.CoreLib interface std Clone stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied
@@ -3197,7 +3197,7 @@ T:System.Type System.Type System.Private.CoreLib class libremetaverse-types lib
T:System.UInt16 System.UInt16 System.Private.CoreLib struct core primitive u16 language not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.UInt16 System.UInt16 System.Private.CoreLib struct core primitive u16 language not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.UInt32 System.UInt32 System.Private.CoreLib struct core primitive u32 language not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.UInt32 System.UInt32 System.Private.CoreLib struct core primitive u32 language not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.UInt64 System.UInt64 System.Private.CoreLib struct core primitive u64 language not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.UInt64 System.UInt64 System.Private.CoreLib struct core primitive u64 language not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Uri System.Uri System.Private.Uri class url url::Url stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Uri System.Uri System.Private.Uri class libremetaverse-types libremetaverse_types::compat::Uri stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.ValueTuple`2 System.ValueTuple<T1, T2> System.Private.CoreLib struct core tuple (T1, T2) language not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.ValueTuple`2 System.ValueTuple<T1, T2> System.Private.CoreLib struct core tuple (T1, T2) language not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.ValueTuple`3 System.ValueTuple<T1, T2, T3> System.Private.CoreLib struct core tuple (T1, T2, T3) language not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.ValueTuple`3 System.ValueTuple<T1, T2, T3> System.Private.CoreLib struct core tuple (T1, T2, T3) language not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.ValueTuple`4 System.ValueTuple<T1, T2, T3, T4> System.Private.CoreLib struct core tuple (T1, T2, T3, T4) language not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.ValueTuple`4 System.ValueTuple<T1, T2, T3, T4> System.Private.CoreLib struct core tuple (T1, T2, T3, T4) language not_applicable not_applicable not_applicable mapped external member surface is not copied
@@ -3205,7 +3205,7 @@ T:System.ValueTuple`6 System.ValueTuple<T1, T2, T3, T4, T5, T6> System.Private.C
T:System.ValueType System.ValueType System.Private.CoreLib class libremetaverse-types libremetaverse_types::compat::Value stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.ValueType System.ValueType System.Private.CoreLib class libremetaverse-types libremetaverse_types::compat::Value stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Void System.Void System.Private.CoreLib struct core primitive () language not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Void System.Void System.Private.CoreLib struct core primitive () language not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Xml.XmlElement System.Xml.XmlElement System.Private.Xml class libremetaverse-structured-data libremetaverse_structured_data::xml::Element stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Xml.XmlElement System.Xml.XmlElement System.Private.Xml class libremetaverse-structured-data libremetaverse_structured_data::xml::Element stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Xml.XmlReader System.Xml.XmlReader System.Private.Xml class libremetaverse-structured-data libremetaverse_structured_data::xml::Reader<R> stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Xml.XmlReader System.Xml.XmlReader System.Private.Xml class libremetaverse-structured-data libremetaverse_structured_data::xml::Reader stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Xml.XmlTextReader System.Xml.XmlTextReader System.Private.Xml class libremetaverse-types compat libremetaverse_types::compat::XmlTextReader native_cross_platform_replacement not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Xml.XmlTextReader System.Xml.XmlTextReader System.Private.Xml class libremetaverse-types compat libremetaverse_types::compat::XmlTextReader native_cross_platform_replacement not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Xml.XmlTextWriter System.Xml.XmlTextWriter System.Private.Xml class libremetaverse-types compat libremetaverse_types::compat::XmlTextWriter native_cross_platform_replacement not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Xml.XmlTextWriter System.Xml.XmlTextWriter System.Private.Xml class libremetaverse-types compat libremetaverse_types::compat::XmlTextWriter native_cross_platform_replacement not_applicable not_applicable not_applicable mapped external member surface is not copied
T:System.Xml.XmlWriter System.Xml.XmlWriter System.Private.Xml class libremetaverse-structured-data libremetaverse_structured_data::xml::Writer<W> stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied T:System.Xml.XmlWriter System.Xml.XmlWriter System.Private.Xml class libremetaverse-structured-data libremetaverse_structured_data::xml::Writer stdlib_or_adopted_crate not_applicable not_applicable not_applicable mapped external member surface is not copied
Can't render this file because it is too large.

21
api/SHIM-COVERAGE.md Normal file
View File

@@ -0,0 +1,21 @@
# Callable shim coverage
Generated by `python3 tools/generate_api_shims.py`; do not edit by hand.
| Assembly | Types | Members | Status |
|---|---:|---:|---|
| `LibreMetaverse` | 2,711 | 27,281 | pending milestone issue |
| `LibreMetaverse.Imaging.Abstractions` | 3 | 20 | pending milestone issue |
| `LibreMetaverse.Imaging.Skia` | 1 | 3 | pending milestone issue |
| `LibreMetaverse.LslTools` | 164 | 768 | pending milestone issue |
| `LibreMetaverse.PrimMesher` | 17 | 207 | pending milestone issue |
| `LibreMetaverse.RLV` | 28 | 499 | pending milestone issue |
| `LibreMetaverse.Rendering.MeshFoundry` | 1 | 14 | pending milestone issue |
| `LibreMetaverse.Rendering.Simple` | 1 | 6 | pending milestone issue |
| `LibreMetaverse.StructuredData` | 16 | 295 | callable failure-only shim |
| `LibreMetaverse.Types` | 45 | 942 | callable failure-only shim |
| `LibreMetaverse.Utilities` | 3 | 13 | pending milestone issue |
| `LibreMetaverse.Voice.Vivox` | 64 | 531 | pending milestone issue |
| `LibreMetaverse.Voice.WebRTC` | 12 | 210 | pending milestone issue |
Current callable coverage: **61 types / 1,237 members**.

File diff suppressed because it is too large Load Diff

View File

@@ -1,7 +1,17 @@
//! Structured-data types corresponding to `LibreMetaverse.StructuredData`. //! Structured-data types corresponding to `LibreMetaverse.StructuredData`.
extern crate self as libremetaverse_structured_data;
mod generated; mod generated;
pub mod xml {
/// Native XML reader boundary; external XML APIs are not copied.
pub struct Reader;
/// Native XML writer boundary; external XML APIs are not copied.
pub struct Writer;
}
pub use generated::*; pub use generated::*;
pub use libremetaverse_types as types; pub use libremetaverse_types as types;
pub use libremetaverse_types::NotImplemented as Error; pub use libremetaverse_types::NotImplemented as Error;

View File

@@ -0,0 +1,61 @@
//! Project-owned replacements for external types exposed by C# signatures.
//!
//! These types preserve Rust API identity without copying external member
//! surfaces. Their behavior is implemented only when the owning API slice is
//! ported.
use std::marker::PhantomData;
pub trait Collection<T> {}
pub trait ReadWrite: std::io::Read + std::io::Write {}
impl<T: std::io::Read + std::io::Write> ReadWrite for T {}
#[derive(Clone, Debug, Eq, Hash, PartialEq)]
pub struct Object(u8);
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
pub struct Utf16CodeUnit(pub u16);
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
pub struct Guid(pub [u8; 16]);
#[derive(Clone, Debug, Eq, Hash, PartialEq)]
pub struct Uri(pub String);
#[derive(Clone, Debug, Eq, Hash, PartialEq)]
pub struct Regex(pub String);
#[derive(Clone, Debug, Eq, Hash, PartialEq)]
pub struct CultureInfo(pub String);
#[derive(Clone, Debug, Eq, Hash, PartialEq)]
pub struct ExternalError(pub String);
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
pub struct EnumValue(pub i64);
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct ArrayList(pub Vec<Object>);
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct Array(pub Vec<Object>);
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct Hashtable(pub std::collections::HashMap<Object, Object>);
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
pub struct IDictionaryEnumerator;
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
pub struct IEqualityComparer<T>(pub PhantomData<fn(T)>);
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct StringReader(pub String);
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct StringWriter(pub String);
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
pub struct TypeId(pub &'static str);

File diff suppressed because it is too large Load Diff

View File

@@ -1,5 +1,8 @@
//! Core value types corresponding to `LibreMetaverse.Types`. //! Core value types corresponding to `LibreMetaverse.Types`.
extern crate self as libremetaverse_types;
pub mod compat;
mod generated; mod generated;
pub mod shim; pub mod shim;

View File

@@ -0,0 +1,33 @@
use libremetaverse_structured_data::{OSD, OSDParser};
use libremetaverse_types::{AssetType, Color4, InventoryType, PrimFlags, Utils};
#[test]
fn catalog_values_and_inert_layouts_are_real() {
assert_eq!(AssetType::Material as i8, 57);
assert_eq!(InventoryType::CATEGORY.0, 8);
assert_eq!(InventoryType::FOLDER.0, 8);
assert_eq!(PrimFlags::PHYSICS.0, 1);
assert_eq!(Utils::PI.to_bits(), 0x4049_0fdb);
let color = Color4 {
r: 0.25,
g: 0.5,
b: 0.75,
a: 1.0,
};
assert_eq!((color.r, color.g, color.b, color.a), (0.25, 0.5, 0.75, 1.0));
let value = OSD::Binary(vec![1, 2, 3]);
assert!(matches!(value, OSD::Binary(bytes) if bytes == [1, 2, 3]));
}
#[test]
fn serializer_entry_point_returns_its_exact_member_id() {
let Err(error) = OSDParser::deserialize_with_bytes(Vec::new()) else {
panic!("failure-only serializer unexpectedly returned a value");
};
assert_eq!(
error.csharp_member(),
"M:LibreMetaverse.StructuredData.OSDParser.Deserialize(System.Byte[])"
);
}

280
tools/generate_api_shims.py Normal file
View File

@@ -0,0 +1,280 @@
#!/usr/bin/env python3
"""Generate callable, failure-only Rust shims from the authoritative catalog."""
from __future__ import annotations
import argparse
import csv
import json
import re
import subprocess
from collections import Counter
from pathlib import Path
import generate_rust_mapping as mapping
ROOT = Path(__file__).resolve().parents[1]
CATALOG = ROOT / "api/public-api.json"
LEDGER = ROOT / "api/RUST-MAPPING.tsv"
COVERAGE = ROOT / "api/SHIM-COVERAGE.md"
TARGETS = {
"LibreMetaverse.Types": ROOT / "crates/libremetaverse-types/src/generated.rs",
"LibreMetaverse.StructuredData": ROOT / "crates/libremetaverse-structured-data/src/generated.rs",
}
UNDERLYING = {
"System.SByte": "i8", "System.Byte": "u8", "System.Int16": "i16", "System.UInt16": "u16",
"System.Int32": "i32", "System.UInt32": "u32", "System.Int64": "i64", "System.UInt64": "u64",
}
PRIVATE_LAYOUTS = {
"T:LibreMetaverse.CacheDictionary`2": [("entries", "std::collections::HashMap<TKey, TValue>")],
"T:LibreMetaverse.DoubleDictionary`3": [
("by_first", "std::collections::HashMap<TKey1, TValue>"),
("by_second", "std::collections::HashMap<TKey2, TValue>"),
],
"T:LibreMetaverse.EmptyRemovalStrategy`1": [("keys", "Vec<TKey>")],
"T:LibreMetaverse.ExpiringCache`2": [("entries", "std::collections::HashMap<TKey, TValue>")],
"T:LibreMetaverse.LruRemovalStrategy`1": [("keys", "Vec<TKey>")],
"T:LibreMetaverse.MruRemovalStrategy`1": [("keys", "Vec<TKey>")],
"T:LibreMetaverse.MultiValueDictionary`2": [("entries", "std::collections::HashMap<TKey, Vec<TValue>>")],
"T:LibreMetaverse.TokenBucket": [("content", "i32"), ("max_burst", "i32"), ("drip_rate", "i32")],
"T:LibreMetaverse.UUID": [("bytes", "[u8; 16]")],
"T:LibreMetaverse.StructuredData.OSDArray": [("values", "Vec<OSD>")],
"T:LibreMetaverse.StructuredData.OSDBinary": [("value", "Vec<u8>")],
"T:LibreMetaverse.StructuredData.OSDBoolean": [("value", "bool")],
"T:LibreMetaverse.StructuredData.OSDDate": [("value", "std::time::SystemTime")],
"T:LibreMetaverse.StructuredData.OSDException": [("message", "String")],
"T:LibreMetaverse.StructuredData.OSDInteger": [("value", "i32")],
"T:LibreMetaverse.StructuredData.OSDMap": [("values", "std::collections::HashMap<String, OSD>")],
"T:LibreMetaverse.StructuredData.OSDReal": [("value", "f64")],
"T:LibreMetaverse.StructuredData.OSDString": [("value", "String")],
"T:LibreMetaverse.StructuredData.OSDUUID": [("value", "libremetaverse_types::UUID")],
"T:LibreMetaverse.StructuredData.OSDUri": [("value", "libremetaverse_types::compat::Uri")],
}
def generic_names(item: dict) -> list[str]:
return [parameter["name"] for parameter in item.get("generic_parameters", [])]
def format_rust(source: str) -> str:
result = subprocess.run(
["rustfmt", "--edition", "2024", "--emit", "stdout"],
input=source,
text=True,
capture_output=True,
check=True,
)
return result.stdout
def type_name(type_row: dict) -> str:
return type_row["rust_path"].rsplit("::", 1)[-1]
def generic_suffix(names: list[str]) -> str:
return "<" + ", ".join(names) + ">" if names else ""
def flags_enum(item: dict) -> bool:
return any(attribute["type"] == "System.FlagsAttribute" for attribute in item.get("attributes", []))
def duplicate_enum_values(item: dict) -> bool:
values = [member["value"]["value"] for member in item["members"] if member["kind"] == "enum_value"]
return len(values) != len(set(values))
def render_enum(item: dict, rust_name: str) -> list[str]:
underlying = UNDERLYING[item["enum_underlying_type"]]
values = [member for member in item["members"] if member["kind"] == "enum_value"]
lines = [f"/// C# type: `{item['doc_id']}`."]
if flags_enum(item) or duplicate_enum_values(item):
lines += [
"#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]",
"#[repr(transparent)]",
f"pub struct {rust_name}(pub {underlying});",
f"impl {rust_name} {{",
]
for member in values:
name = mapping.snake(member["name"]).upper()
value = member["value"]["value"]
lines += [f" /// C# member: `{member['doc_id']}`.", f" pub const {name}: Self = Self({value});"]
lines.append("}")
return lines
lines += [
"#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]",
f"#[repr({underlying})]",
f"pub enum {rust_name} {{",
]
for member in values:
lines += [f" /// C# member: `{member['doc_id']}`.", f" {mapping.pascal(member['name'])} = {member['value']['value']},"]
lines.append("}")
return lines
def render_body(signature: str, member_id: str, error_model: str, trait: bool) -> str:
if trait:
signature = signature.removeprefix("pub ")
failure = (
f"libremetaverse_types::not_implemented({json.dumps(member_id)})"
if error_model.startswith("Result")
else f"libremetaverse_types::unimplemented_api!({json.dumps(member_id)})"
)
return f" {signature} {{ {failure} }}"
def render_type(item: dict, type_row: dict, member_rows: dict[str, dict[str, str]], mapper: mapping.Mapper) -> str:
rust_name = type_name(type_row)
if item["kind"] == "enum":
return "\n".join(render_enum(item, rust_name))
names = generic_names(item)
suffix = generic_suffix(names)
trait = item["kind"] == "interface"
lines = [f"/// C# type: `{item['doc_id']}`."]
if trait:
lines.append(f"pub trait {rust_name}{suffix} {{")
else:
fields = [member for member in item["members"] if member["kind"] == "field" and not member.get("static")]
private_fields = PRIVATE_LAYOUTS.get(item["doc_id"], [])
if item["doc_id"] == "T:LibreMetaverse.StructuredData.OSD":
lines += [
"#[non_exhaustive]",
"pub enum OSD {",
" Undefined, Boolean(bool), Integer(i32), Real(f64), String(String),",
" UUID(libremetaverse_types::UUID), Date(std::time::SystemTime),",
" Uri(libremetaverse_types::compat::Uri), Binary(Vec<u8>),",
" Array(Vec<OSD>), Map(std::collections::HashMap<String, OSD>), LlsdXml(String),",
"}",
]
elif not fields and not names and not private_fields:
lines.append(f"pub struct {rust_name};")
else:
lines.append(f"pub struct {rust_name}{suffix} {{")
generics = set(names)
for field in fields:
field_type = mapper.type(field["type"], field.get("nullability"), generics)
lines += [f" /// C# member: `{field['doc_id']}`.", f" pub {mapping.snake(field['name'])}: {field_type},"]
for field_name, field_type in private_fields:
lines.append(f" {field_name}: {field_type},")
if names:
tuple_type = "(" + ", ".join(names) + ("," if len(names) == 1 else "") + ")"
lines.append(f" _marker: std::marker::PhantomData<{tuple_type}>,")
lines.append("}")
lines += [f"impl{suffix} {rust_name}{suffix} {{"]
for member in item["members"]:
if member["kind"] in {"enum_value"} or (member["kind"] == "field" and not member.get("static")):
continue
row = member_rows[member["doc_id"]]
lines.append(f" /// C# member: `{member['doc_id']}`.")
if member["kind"] == "constant":
value = member["value"]["value"]
lines.append(f" {row['rust_signature']} = {value};")
continue
signatures = row["rust_signature"].split(" ; ")
for index, signature in enumerate(signatures):
if index:
lines.append(f" /// Setter for C# member: `{member['doc_id']}`.")
lines.append(render_body(signature, member["doc_id"], row["error_model"], trait))
lines.append("}")
if not trait:
for interface in item.get("interfaces", []):
base, arguments, _ = mapping.type_parts(interface)
if base not in mapper.interfaces:
continue
mapped_arguments = [mapper.type(argument, None, set(names)) for argument in arguments]
trait_path = mapper.resolved[base] + generic_suffix(mapped_arguments)
lines.append(f"impl{suffix} {trait_path} for {rust_name}{suffix} {{}}")
return "\n".join(lines)
def generate_sources(catalog: dict) -> tuple[dict[Path, str], dict[str, tuple[int, int]]]:
type_rows, resolved = mapping.build_type_rows(catalog)
type_by_id = {row["csharp_type_id"]: row for row in type_rows}
interfaces = {mapping.type_parts(item["signature"])[0] for assembly in catalog["assemblies"] for item in assembly["types"] if item["kind"] == "interface"}
mapper = mapping.Mapper(resolved, interfaces)
member_rows = {row["csharp_id"]: row for row in csv.DictReader(LEDGER.open(), delimiter="\t")}
outputs = {}
coverage = {}
for assembly in catalog["assemblies"]:
name = assembly["identity"]["name"]
if name not in TARGETS:
continue
chunks = [
"// @generated by tools/generate_api_shims.py; do not edit by hand.",
"#![allow(clippy::missing_errors_doc)]",
"#![allow(clippy::must_use_candidate)]",
"#![allow(clippy::all, clippy::pedantic, dead_code, unused_variables)]",
"#![allow(non_camel_case_types)]",
"#![allow(non_snake_case)]",
"",
]
for item in assembly["types"]:
chunks += [render_type(item, type_by_id[item["doc_id"]], member_rows, mapper), ""]
source = format_rust("\n".join(chunks).rstrip() + "\n")
expected_types = {item["doc_id"] for item in assembly["types"]}
expected_members = {member["doc_id"] for item in assembly["types"] for member in item["members"]}
emitted_types = set(re.findall(r"C# type: `(.*)`\.", source))
emitted_members = set(re.findall(r"C# member: `(.*)`\.", source))
if emitted_types != expected_types or emitted_members != expected_members:
raise ValueError(
f"generated catalog ID mismatch for {name}: "
f"types missing/stale={len(expected_types - emitted_types)}/{len(emitted_types - expected_types)}, "
f"members missing/stale={len(expected_members - emitted_members)}/{len(emitted_members - expected_members)}"
)
outputs[TARGETS[name]] = source
coverage[name] = (len(assembly["types"]), sum(len(item["members"]) for item in assembly["types"]))
return outputs, coverage
def coverage_report(catalog: dict, coverage: dict[str, tuple[int, int]]) -> str:
lines = [
"# Callable shim coverage", "", "Generated by `python3 tools/generate_api_shims.py`; do not edit by hand.", "",
"| Assembly | Types | Members | Status |", "|---|---:|---:|---|",
]
total_types = total_members = 0
for assembly in catalog["assemblies"]:
name = assembly["identity"]["name"]
types = len(assembly["types"])
members = sum(len(item["members"]) for item in assembly["types"])
if name in coverage:
got_types, got_members = coverage[name]
if (types, members) != (got_types, got_members):
raise ValueError(f"incomplete generated slice for {name}")
status = "callable failure-only shim"
total_types += types
total_members += members
else:
status = "pending milestone issue"
lines.append(f"| `{name}` | {types:,} | {members:,} | {status} |")
lines += ["", f"Current callable coverage: **{total_types:,} types / {total_members:,} members**.", ""]
return "\n".join(lines)
def generate() -> dict[Path, str]:
catalog = json.loads(CATALOG.read_text())
sources, coverage = generate_sources(catalog)
sources[COVERAGE] = coverage_report(catalog, coverage)
return sources
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--check", action="store_true")
args = parser.parse_args()
outputs = generate()
if args.check:
stale = [str(path.relative_to(ROOT)) for path, text in outputs.items() if not path.exists() or path.read_text() != text]
if stale:
raise SystemExit("stale generated API shims: " + ", ".join(stale))
else:
for path, text in outputs.items():
path.write_text(text)
print("generated " + ", ".join(str(path.relative_to(ROOT)) for path in outputs))
if __name__ == "__main__":
main()

View File

@@ -54,7 +54,7 @@ PRIMITIVES = {
} }
STDLIB = { STDLIB = {
"System.Array": "Vec<T>", "System.Array": "libremetaverse_types::compat::Array",
"System.ArraySegment": "libremetaverse_types::compat::ArraySegment<T>", "System.ArraySegment": "libremetaverse_types::compat::ArraySegment<T>",
"System.Collections.BitArray": "Vec<bool>", "System.Collections.BitArray": "Vec<bool>",
"System.Collections.Generic.Dictionary": "std::collections::HashMap<K, V>", "System.Collections.Generic.Dictionary": "std::collections::HashMap<K, V>",
@@ -96,8 +96,8 @@ STDLIB = {
"System.DateTime": "std::time::SystemTime", "System.DateTime": "std::time::SystemTime",
"System.DateTimeOffset": "std::time::SystemTime", "System.DateTimeOffset": "std::time::SystemTime",
"System.TimeSpan": "std::time::Duration", "System.TimeSpan": "std::time::Duration",
"System.Guid": "uuid::Uuid", "System.Guid": "libremetaverse_types::compat::Guid",
"System.Uri": "url::Url", "System.Uri": "libremetaverse_types::compat::Uri",
"System.Version": "libremetaverse_types::compat::Version", "System.Version": "libremetaverse_types::compat::Version",
"System.Random": "libremetaverse_types::compat::RandomSource", "System.Random": "libremetaverse_types::compat::RandomSource",
"System.Exception": "libremetaverse_types::compat::ExternalError", "System.Exception": "libremetaverse_types::compat::ExternalError",
@@ -114,7 +114,7 @@ STDLIB = {
"System.ValueType": "libremetaverse_types::compat::Value", "System.ValueType": "libremetaverse_types::compat::Value",
"System.Text.Encoding": "libremetaverse_types::compat::TextEncoding", "System.Text.Encoding": "libremetaverse_types::compat::TextEncoding",
"System.Text.StringBuilder": "String", "System.Text.StringBuilder": "String",
"System.Text.RegularExpressions.Regex": "regex::Regex", "System.Text.RegularExpressions.Regex": "libremetaverse_types::compat::Regex",
"System.IO.Stream": "Box<dyn libremetaverse_types::compat::ReadWrite + Send>", "System.IO.Stream": "Box<dyn libremetaverse_types::compat::ReadWrite + Send>",
"System.IO.MemoryStream": "std::io::Cursor<Vec<u8>>", "System.IO.MemoryStream": "std::io::Cursor<Vec<u8>>",
"System.IO.TextReader": "Box<dyn std::io::BufRead + Send>", "System.IO.TextReader": "Box<dyn std::io::BufRead + Send>",
@@ -142,8 +142,8 @@ STDLIB = {
"System.ValueTuple": "(T1, T2)", "System.ValueTuple": "(T1, T2)",
"System.Xml.XmlNode": "libremetaverse_structured_data::xml::Node", "System.Xml.XmlNode": "libremetaverse_structured_data::xml::Node",
"System.Xml.XmlElement": "libremetaverse_structured_data::xml::Element", "System.Xml.XmlElement": "libremetaverse_structured_data::xml::Element",
"System.Xml.XmlReader": "libremetaverse_structured_data::xml::Reader<R>", "System.Xml.XmlReader": "libremetaverse_structured_data::xml::Reader",
"System.Xml.XmlWriter": "libremetaverse_structured_data::xml::Writer<W>", "System.Xml.XmlWriter": "libremetaverse_structured_data::xml::Writer",
"System.Text.Json.JsonElement": "libremetaverse_voice_webrtc::signaling::JsonValue", "System.Text.Json.JsonElement": "libremetaverse_voice_webrtc::signaling::JsonValue",
} }
@@ -459,8 +459,9 @@ def build_type_rows(catalog: dict) -> tuple[list[dict[str, str]], dict[str, str]
class Mapper: class Mapper:
def __init__(self, resolved: dict[str, str]): def __init__(self, resolved: dict[str, str], interfaces: set[str] | None = None):
self.resolved = resolved self.resolved = resolved
self.interfaces = interfaces or set()
def type(self, value: str, nullability: dict | None = None, generics: set[str] | None = None) -> str: def type(self, value: str, nullability: dict | None = None, generics: set[str] | None = None) -> str:
base, arguments, suffix = type_parts(value) base, arguments, suffix = type_parts(value)
@@ -498,8 +499,10 @@ class Mapper:
"System.ValueTuple", "System.Threading.Tasks.Task", "System.Threading.Tasks.ValueTask", "System.ValueTuple", "System.Threading.Tasks.Task", "System.Threading.Tasks.ValueTask",
}: }:
target = substitute(target, mapped_arguments) target = substitute(target, mapped_arguments)
if "<" not in target and target not in {"Ord", "Eq", "Clone", "std::fmt::Display"}: if "<" not in target and not target.startswith("(") and target not in {"Ord", "Eq", "Clone", "std::fmt::Display"}:
target += "<" + ", ".join(mapped_arguments) + ">" target += "<" + ", ".join(mapped_arguments) + ">"
if base in self.interfaces:
target = f"Box<dyn {target}>"
for _ in re.findall(r"\[[,]*\]", suffix): for _ in re.findall(r"\[[,]*\]", suffix):
target = f"Vec<{target}>" target = f"Vec<{target}>"
if "*" in suffix: if "*" in suffix:
@@ -551,7 +554,7 @@ def map_parameter(mapper: Mapper, parameter: dict, generics: set[str]) -> tuple[
def member_base_name(item: dict) -> str: def member_base_name(item: dict) -> str:
if item["kind"] == "enum_value": if item["kind"] == "enum_value":
return pascal(item["name"]) return pascal(item["name"])
if item["kind"] == "constant" or (item["kind"] == "field" and item.get("static")): if item["kind"] == "constant":
return snake(item["name"]).upper() return snake(item["name"]).upper()
if item["kind"] == "constructor": if item["kind"] == "constructor":
return "new" return "new"
@@ -566,6 +569,14 @@ def member_base_name(item: dict) -> str:
return operators.get(item["name"], snake(item["name"].removesuffix("Async"))) return operators.get(item["name"], snake(item["name"].removesuffix("Async")))
def parameter_slug(value: str) -> str:
base, _, suffix = type_parts(value)
slug = snake(base.rsplit(".", 1)[-1])
if "[" in suffix:
return "bytes" if base == "System.Byte" else f"{slug}_array"
return slug
def member_signature(mapper: Mapper, owner: dict, item: dict, rust_name: str) -> tuple[str, str, str, str, str]: def member_signature(mapper: Mapper, owner: dict, item: dict, rust_name: str) -> tuple[str, str, str, str, str]:
generics = generic_names(item, owner) generics = generic_names(item, owner)
mapped_parameters = [map_parameter(mapper, parameter, generics) for parameter in item.get("parameters", [])] mapped_parameters = [map_parameter(mapper, parameter, generics) for parameter in item.get("parameters", [])]
@@ -581,7 +592,8 @@ def member_signature(mapper: Mapper, owner: dict, item: dict, rust_name: str) ->
elif kind == "constant": elif kind == "constant":
signature = f"pub const {rust_name}: {value_type}" signature = f"pub const {rust_name}: {value_type}"
elif item.get("static"): elif item.get("static"):
signature = f"pub static {rust_name}: {value_type}" signature = f"pub fn {rust_name}() -> {value_type}"
return signature, "none", asyncness, error_model, "static_field_accessor"
else: else:
signature = f"pub {rust_name}: {value_type}" signature = f"pub {rust_name}: {value_type}"
return signature, "owned_value", asyncness, "none", "direct" return signature, "owned_value", asyncness, "none", "direct"
@@ -653,16 +665,22 @@ def build_member_rows(catalog: dict, mapper: Mapper) -> list[dict[str, str]]:
for owner in assembly["types"]: for owner in assembly["types"]:
_, module, owner_path = declared_type_path(assembly_name, owner) _, module, owner_path = declared_type_path(assembly_name, owner)
groups = Counter(member_base_name(member) for member in owner["members"]) groups = Counter(member_base_name(member) for member in owner["members"])
enum_values = [member.get("value", {}).get("value") for member in owner["members"] if member["kind"] == "enum_value"]
enum_constants = owner["kind"] == "enum" and (
any(attribute["type"] == "System.FlagsAttribute" for attribute in owner.get("attributes", []))
or len(enum_values) != len(set(enum_values))
)
used = set() used = set()
for item in owner["members"]: for item in owner["members"]:
base_name = member_base_name(item) base_name = member_base_name(item)
if item["kind"] == "enum_value" and enum_constants:
base_name = snake(item["name"]).upper()
rust_name = base_name rust_name = base_name
decision = "direct_snake_case" decision = "direct_snake_case"
if groups[base_name] > 1: if groups[base_name] > 1:
suffixes = [] suffixes = []
for parameter in item.get("parameters", []): for parameter in item.get("parameters", []):
base, _, _ = type_parts(parameter["type"]) suffixes.append(parameter_slug(parameter["type"]))
suffixes.append(snake(base.rsplit(".", 1)[-1]))
suffix = "_".join(suffixes) or item["kind"] suffix = "_".join(suffixes) or item["kind"]
rust_name = f"{base_name}_with_{suffix}" rust_name = f"{base_name}_with_{suffix}"
decision = "descriptive_overload_name" decision = "descriptive_overload_name"
@@ -808,7 +826,8 @@ def coverage_report(catalog: dict, type_rows: list[dict[str, str]], member_rows:
def generate() -> dict[Path, str]: def generate() -> dict[Path, str]:
catalog = json.loads(CATALOG.read_text()) catalog = json.loads(CATALOG.read_text())
type_rows, resolved = build_type_rows(catalog) type_rows, resolved = build_type_rows(catalog)
mapper = Mapper(resolved) interfaces = {type_parts(item["signature"])[0] for assembly in catalog["assemblies"] for item in assembly["types"] if item["kind"] == "interface"}
mapper = Mapper(resolved, interfaces)
member_rows = build_member_rows(catalog, mapper) member_rows = build_member_rows(catalog, mapper)
validate(catalog, type_rows, member_rows, mapper) validate(catalog, type_rows, member_rows, mapper)
type_fields = ["csharp_type_id", "csharp_signature", "source_assembly", "source_kind", "rust_crate", "rust_module", "rust_path", "mapping_decision", "generic_bounds", "inheritance_mapping", "enum_mapping", "status", "notes"] type_fields = ["csharp_type_id", "csharp_signature", "source_assembly", "source_kind", "rust_crate", "rust_module", "rust_path", "mapping_decision", "generic_bounds", "inheritance_mapping", "enum_mapping", "status", "notes"]