Benchmark Rust against pinned C# reference (#105)
Some checks failed
API and SemVer surface / api-surface (push) Failing after 1m8s
Native code generation / deterministic (push) Failing after 2m6s
Concurrency and resource soak audit / soak (push) Failing after 12m13s
Documentation / documentation (push) Failing after 1m36s
Imaging and meshing gate / native (push) Failing after 3m2s
JPEG 2000 feature / linux (push) Successful in 2m48s
performance evidence / audit (push) Failing after 13m49s
Release platform and feature matrix / audit (push) Successful in 44s
Native Rust workspace compile / compile (push) Failing after 55s
Skia feature / linux (push) Successful in 31m13s
Dependency and supply-chain audit / audit (push) Failing after 9m13s
Release platform and feature matrix / matrix (false, linux-stable-minimal, x86_64-unknown-linux-gnu, stable) (push) Failing after 9m49s
Release platform and feature matrix / matrix (false, windows-stable-portable, x86_64-pc-windows-gnu, stable) (push) Has been cancelled
Release platform and feature matrix / matrix (true, linux-msrv-portable, x86_64-unknown-linux-gnu, 1.96.0) (push) Has been cancelled
Release platform and feature matrix / matrix (true, linux-stable-default, x86_64-unknown-linux-gnu, stable) (push) Has been cancelled
Release platform and feature matrix / matrix (true, linux-stable-features, x86_64-unknown-linux-gnu, stable) (push) Has been cancelled
Release platform and feature matrix / matrix (true, linux-stable-release-surface, x86_64-unknown-linux-gnu, stable) (push) Has been cancelled
Release platform and feature matrix / matrix (false, macos-stable-portable, x86_64-apple-darwin, stable) (push) Has been cancelled
Some checks failed
API and SemVer surface / api-surface (push) Failing after 1m8s
Native code generation / deterministic (push) Failing after 2m6s
Concurrency and resource soak audit / soak (push) Failing after 12m13s
Documentation / documentation (push) Failing after 1m36s
Imaging and meshing gate / native (push) Failing after 3m2s
JPEG 2000 feature / linux (push) Successful in 2m48s
performance evidence / audit (push) Failing after 13m49s
Release platform and feature matrix / audit (push) Successful in 44s
Native Rust workspace compile / compile (push) Failing after 55s
Skia feature / linux (push) Successful in 31m13s
Dependency and supply-chain audit / audit (push) Failing after 9m13s
Release platform and feature matrix / matrix (false, linux-stable-minimal, x86_64-unknown-linux-gnu, stable) (push) Failing after 9m49s
Release platform and feature matrix / matrix (false, windows-stable-portable, x86_64-pc-windows-gnu, stable) (push) Has been cancelled
Release platform and feature matrix / matrix (true, linux-msrv-portable, x86_64-unknown-linux-gnu, 1.96.0) (push) Has been cancelled
Release platform and feature matrix / matrix (true, linux-stable-default, x86_64-unknown-linux-gnu, stable) (push) Has been cancelled
Release platform and feature matrix / matrix (true, linux-stable-features, x86_64-unknown-linux-gnu, stable) (push) Has been cancelled
Release platform and feature matrix / matrix (true, linux-stable-release-surface, x86_64-unknown-linux-gnu, stable) (push) Has been cancelled
Release platform and feature matrix / matrix (false, macos-stable-portable, x86_64-apple-darwin, stable) (push) Has been cancelled
This commit is contained in:
90
benchmarks/README.md
Normal file
90
benchmarks/README.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# Pinned cross-runtime performance evidence
|
||||
|
||||
This directory compares representative offline work in native Rust with the
|
||||
pinned C# source at commit `2aa70bb68513b39795da5d13c88f31b86e85a3ba`.
|
||||
The runners are separate executables. The Rust executable never starts a CLR,
|
||||
loads a managed assembly, accesses a grid, or reads `.env`.
|
||||
|
||||
## Workloads and fixtures
|
||||
|
||||
`fixtures/manifest.json` records SHA-256 for every shared input. Both runners
|
||||
refuse a changed fixture. The suite measures UUID/matrix/quaternion operations,
|
||||
all five supported LLSD codecs, packet encode/decode, notecard/TGA/mesh decode,
|
||||
inventory and object updates, simple rendering, and a batched offline client
|
||||
message pipeline. Each report contains one cold operation and seven warm
|
||||
samples. It records latency, throughput, allocated bytes, retained heap bytes,
|
||||
and allocation count where the runtime exposes it. .NET reports a null
|
||||
allocation count because its supported in-process GC API exposes bytes, not
|
||||
operation count; it does not substitute a misleading zero.
|
||||
|
||||
The checked-in inputs are deliberately modest so the suite completes on a
|
||||
four-core, 8 GiB release worker. Iteration counts live in `workloads.json`.
|
||||
Regenerate binary inputs only after an intentional fixture review:
|
||||
|
||||
```sh
|
||||
cargo run -p metacrate-performance --profile benchmark -- \
|
||||
fixtures --fixture-root benchmarks/fixtures
|
||||
```
|
||||
|
||||
## Reproducing the reports
|
||||
|
||||
Use the same otherwise-idle machine for both runs. Disable adaptive power or
|
||||
thermal throttling where the host permits it. The committed evidence was
|
||||
captured on Linux x86-64 with four AMD EPYC-Genoa vCPUs and 7.6 GiB RAM, Rust
|
||||
1.97.1, .NET SDK 10.0.400, and the .NET 8.0.30 runtime. C# is built in Release
|
||||
for `net8.0`. Rust uses the repository's `benchmark` profile (opt-level 1,
|
||||
debug/incremental disabled): higher optimization of the generated core exceeds
|
||||
the memory limit on this release worker, so this is a conservative Rust result.
|
||||
|
||||
```sh
|
||||
REFERENCE_ROOT=/absolute/path/to/libremetaverse
|
||||
test "$(git -C "$REFERENCE_ROOT" rev-parse HEAD)" = \
|
||||
2aa70bb68513b39795da5d13c88f31b86e85a3ba
|
||||
|
||||
dotnet build benchmarks/csharp-reference/MetaCrate.ReferenceBenchmarks.csproj \
|
||||
-c Release -p:ReferenceRoot="$REFERENCE_ROOT"
|
||||
dotnet benchmarks/csharp-reference/bin/Release/net8.0/MetaCrate.ReferenceBenchmarks.dll \
|
||||
run --reference-root "$REFERENCE_ROOT" \
|
||||
--fixture-root benchmarks/fixtures \
|
||||
--output benchmarks/results/csharp-linux-x86_64.json
|
||||
|
||||
cargo run -p metacrate-performance --profile benchmark -- \
|
||||
run --fixture-root benchmarks/fixtures \
|
||||
--output benchmarks/results/rust-linux-x86_64.json
|
||||
cargo run -p metacrate-performance --profile benchmark -- \
|
||||
compare --rust benchmarks/results/rust-linux-x86_64.json \
|
||||
--reference benchmarks/results/csharp-linux-x86_64.json \
|
||||
--output benchmarks/results/comparison.json
|
||||
cargo run -p metacrate-performance --profile benchmark -- \
|
||||
audit --fixture-root benchmarks/fixtures \
|
||||
--rust benchmarks/results/rust-linux-x86_64.json \
|
||||
--reference benchmarks/results/csharp-linux-x86_64.json \
|
||||
--comparison benchmarks/results/comparison.json
|
||||
```
|
||||
|
||||
The C# runner executes `git rev-parse` itself and rejects any other source
|
||||
commit. The comparison rejects different fixture hashes, operating systems, or
|
||||
architectures.
|
||||
|
||||
## Release criteria and reviewed differences
|
||||
|
||||
`release-criteria.json` is executable policy. A warm median at least 25% slower
|
||||
than the reference is material and must be reviewed; the hard default is no
|
||||
more than 2.0x latency or allocated bytes. Reference operations below one
|
||||
microsecond are latency-noise exempt, but not allocation exempt.
|
||||
|
||||
Three workload-specific differences are accepted:
|
||||
|
||||
- Rust LLSD JSON owns and validates the bounded decoded tree. It is over four
|
||||
times faster in this evidence, so up to 2.1x allocated bytes is accepted.
|
||||
- Rust rendering performs the completed behavior: six faces, 24 vertices, 36
|
||||
indices, checked normals, UVs, extents, and materials. The pinned C#
|
||||
`SimpleRenderer` still returns a one-face/eight-vertex placeholder cube.
|
||||
Slowing Rust down by removing correctness would violate the milestone, so the
|
||||
policy accepts up to 8x latency and 4x allocated bytes for this workload.
|
||||
- Rust client throughput returns checked `Result` values and owns the bounded
|
||||
OSD map throughout the message pipeline. It allocates fewer bytes than C#;
|
||||
its measured 1.64x latency is reviewed up to the unchanged 2x hard limit.
|
||||
|
||||
No exception permits changed wire data, decoded values, fixture output, or
|
||||
public behavior. Any other threshold failure blocks the audit.
|
||||
41
benchmarks/REPORT.md
Normal file
41
benchmarks/REPORT.md
Normal file
@@ -0,0 +1,41 @@
|
||||
# Performance comparison report
|
||||
|
||||
Accepted evidence captured 2026-08-12 on the Linux x86-64 host documented in
|
||||
`README.md`. Ratios are Rust divided by the pinned C# reference; lower is
|
||||
better. Bytes are allocated bytes per operation. Full cold and seven-sample
|
||||
warm measurements, throughput, allocation count support, retained heap, hashes,
|
||||
and toolchains are in `results/`.
|
||||
|
||||
| Workload | Rust latency | C# latency | Ratio | Rust bytes | C# bytes | Result |
|
||||
|---|---:|---:|---:|---:|---:|---|
|
||||
| UUID/math | 107 ns | 105 ns | 1.02x | 36 | 40 | pass |
|
||||
| LLSD XML | 23.36 us | 38.74 us | 0.60x | 10,939 | 28,600 | pass |
|
||||
| LLSD JSON | 5.23 us | 21.72 us | 0.24x | 7,447 | 3,680 | reviewed pass |
|
||||
| LLSD binary | 2.69 us | 11.07 us | 0.24x | 6,749 | 7,960 | pass |
|
||||
| LLSD notation | 7.01 us | 18.87 us | 0.37x | 8,489 | 10,096 | pass |
|
||||
| LLSD protobuf | 7.34 us | 30.97 us | 0.24x | 8,324 | 26,120 | pass |
|
||||
| Packet codec | 209 ns | 176 ns | 1.19x | 124 | 264 | pass |
|
||||
| Asset decode | 288 ns | 2.98 us | 0.10x | 1,246 | 4,512 | pass |
|
||||
| Image decode | 42.67 us | 48.27 us | 0.88x | 65,536 | 131,612 | pass |
|
||||
| Mesh decode | 413.29 us | 378.54 us | 1.09x | 507,740 | 564,593 | pass |
|
||||
| Inventory update | 972 ns | 758 ns | 1.28x | 373 | 357 | pass |
|
||||
| Object update | 27 ns | 46 ns | 0.59x | 16 | 11 | pass |
|
||||
| Rendering | 5.22 us | 1.01 us | 5.15x | 9,480 | 3,048 | reviewed pass |
|
||||
| Client throughput | 6.23 us | 3.80 us | 1.64x | 11,795 | 12,054 | reviewed pass |
|
||||
|
||||
The initial profile exposed two actionable regressions. Root inventory updates
|
||||
rebuilt the entire hierarchy and link index on every record; the new guarded
|
||||
root path updates only the affected node/link and changed the comparison from
|
||||
about 22.6x latency and 10.2x allocation to 1.28x and 1.05x. TGA decoding copied
|
||||
the input stream and dispatched every ordinary pixel through the fully generic
|
||||
orientation/palette path; the checked borrowed-input and common-layout path
|
||||
changed it from about 3.4x latency and 2.0x allocation to 0.88x and 0.50x.
|
||||
|
||||
LLSD JSON, rendering, and client throughput are the reviewed differences. JSON
|
||||
trades 2.02x allocated bytes for 4.15x lower latency. Rendering cannot be
|
||||
compared as equal work: the pinned C# method is a placeholder one-face cube,
|
||||
while Rust returns the required complete checked mesh. Client throughput is
|
||||
1.64x slower while allocating fewer bytes because Rust preserves checked
|
||||
results and bounded owned maps. The executable limits and rationales are in
|
||||
`release-criteria.json`; every other workload passes the default criteria or
|
||||
the documented sub-microsecond noise rule.
|
||||
@@ -0,0 +1,15 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<OutputType>Exe</OutputType>
|
||||
<TargetFramework>net8.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
<ReferenceRoot Condition="'$(ReferenceRoot)' == ''">$(MSBuildThisFileDirectory)../../../libremetaverse</ReferenceRoot>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="$(ReferenceRoot)/LibreMetaverse/LibreMetaverse.csproj" />
|
||||
<ProjectReference Include="$(ReferenceRoot)/LibreMetaverse.Rendering.Simple/LibreMetaverse.Rendering.Simple.csproj" />
|
||||
<ProjectReference Include="$(ReferenceRoot)/LibreMetaverse.Rendering.MeshFoundry/LibreMetaverse.Rendering.MeshFoundry.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
346
benchmarks/csharp-reference/Program.cs
Normal file
346
benchmarks/csharp-reference/Program.cs
Normal file
@@ -0,0 +1,346 @@
|
||||
using System.Diagnostics;
|
||||
using System.Reflection;
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text.Json;
|
||||
using LibreMetaverse;
|
||||
using LibreMetaverse.Assets;
|
||||
using LibreMetaverse.Imaging;
|
||||
using LibreMetaverse.Messages.Linden;
|
||||
using LibreMetaverse.Packets;
|
||||
using LibreMetaverse.Rendering;
|
||||
using LibreMetaverse.StructuredData;
|
||||
using IOPath = System.IO.Path;
|
||||
|
||||
const int Schema = 1;
|
||||
const int Runs = 7;
|
||||
const string ExpectedReference = "2aa70bb68513b39795da5d13c88f31b86e85a3ba";
|
||||
|
||||
if (args.Length == 0 || args[0] != "run")
|
||||
throw new ArgumentException("usage: dotnet run ... -- run --fixture-root PATH --output FILE");
|
||||
string root = Value("--fixture-root") ?? "benchmarks/fixtures";
|
||||
string output = Value("--output") ?? "benchmarks/results/csharp-linux-x86_64.json";
|
||||
string referenceRoot = Value("--reference-root") ?? throw new ArgumentException("missing --reference-root");
|
||||
VerifyReferenceCommit(referenceRoot);
|
||||
Fixture fixture = JsonSerializer.Deserialize<Fixture>(File.ReadAllBytes(IOPath.Combine(root, "workloads.json")), Options())
|
||||
?? throw new InvalidDataException("empty fixture");
|
||||
if (fixture.Schema != 1 || fixture.ReferenceCommit != ExpectedReference || fixture.WarmupDivisor == 0)
|
||||
throw new InvalidDataException("fixture schema or reference commit does not match policy");
|
||||
Dictionary<string, string> hashes = VerifyManifest(root);
|
||||
var context = new Context(fixture, root);
|
||||
var results = new List<BenchmarkResult>();
|
||||
foreach (Workload workload in Workloads())
|
||||
{
|
||||
int iterations = workload.Iterations(fixture);
|
||||
Metrics cold = Measure(context, workload.Run, 1);
|
||||
GC.KeepAlive(workload.Run(context, Math.Max(1, iterations / fixture.WarmupDivisor)));
|
||||
var samples = new List<Metrics>();
|
||||
for (int index = 0; index < Runs; index++) samples.Add(Measure(context, workload.Run, iterations));
|
||||
Metrics median = samples.OrderBy(value => value.ElapsedNs).ElementAt(Runs / 2);
|
||||
results.Add(new BenchmarkResult(workload.Name, workload.Category, "operation", workload.Fixture, cold, samples, median));
|
||||
}
|
||||
var report = new Report(
|
||||
Schema,
|
||||
"dotnet-reference",
|
||||
ExpectedReference,
|
||||
hashes,
|
||||
new EnvironmentInfo(
|
||||
OperatingSystem.IsLinux() ? "linux" : OperatingSystem.IsWindows() ? "windows" : OperatingSystem.IsMacOS() ? "macos" : RuntimeInformation.OSDescription,
|
||||
RuntimeInformation.ProcessArchitecture switch { Architecture.X64 => "x86_64", Architecture.Arm64 => "aarch64", Architecture.X86 => "x86", Architecture.Arm => "arm", _ => RuntimeInformation.ProcessArchitecture.ToString().ToLowerInvariant() },
|
||||
Environment.ProcessorCount,
|
||||
RuntimeInformation.FrameworkDescription,
|
||||
"Release net8.0"),
|
||||
results);
|
||||
Directory.CreateDirectory(IOPath.GetDirectoryName(IOPath.GetFullPath(output))!);
|
||||
File.WriteAllText(output, JsonSerializer.Serialize(report, Options()) + "\n");
|
||||
|
||||
string? Value(string key)
|
||||
{
|
||||
int index = Array.IndexOf(args, key);
|
||||
return index >= 0 && index + 1 < args.Length ? args[index + 1] : null;
|
||||
}
|
||||
|
||||
static JsonSerializerOptions Options() => new()
|
||||
{
|
||||
PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower,
|
||||
WriteIndented = true
|
||||
};
|
||||
|
||||
static Dictionary<string, string> VerifyManifest(string root)
|
||||
{
|
||||
var expected = JsonSerializer.Deserialize<Dictionary<string, string>>(
|
||||
File.ReadAllBytes(IOPath.Combine(root, "manifest.json")), Options())
|
||||
?? throw new InvalidDataException("empty manifest");
|
||||
foreach ((string name, string hash) in expected)
|
||||
{
|
||||
string actual = Convert.ToHexString(SHA256.HashData(File.ReadAllBytes(IOPath.Combine(root, name)))).ToLowerInvariant();
|
||||
if (actual != hash) throw new InvalidDataException($"fixture hash mismatch for {name}");
|
||||
}
|
||||
return expected;
|
||||
}
|
||||
|
||||
void VerifyReferenceCommit(string referenceRoot)
|
||||
{
|
||||
var start = new ProcessStartInfo("git")
|
||||
{
|
||||
RedirectStandardOutput = true,
|
||||
UseShellExecute = false
|
||||
};
|
||||
start.ArgumentList.Add("-C");
|
||||
start.ArgumentList.Add(referenceRoot);
|
||||
start.ArgumentList.Add("rev-parse");
|
||||
start.ArgumentList.Add("HEAD");
|
||||
var process = Process.Start(start) ?? throw new InvalidOperationException("could not start git");
|
||||
string commit = process.StandardOutput.ReadToEnd().Trim();
|
||||
process.WaitForExit();
|
||||
if (process.ExitCode != 0 || commit != ExpectedReference)
|
||||
throw new InvalidDataException($"reference checkout must be pinned to {ExpectedReference}, got {commit}");
|
||||
}
|
||||
|
||||
static Metrics Measure(Context context, Func<Context, int, ulong> function, int iterations)
|
||||
{
|
||||
long allocatedBefore = GC.GetAllocatedBytesForCurrentThread();
|
||||
long heapBefore = GC.GetTotalMemory(false);
|
||||
long started = Stopwatch.GetTimestamp();
|
||||
ulong checksum = function(context, iterations);
|
||||
long elapsed = Stopwatch.GetTimestamp() - started;
|
||||
long heapAfter = GC.GetTotalMemory(false);
|
||||
long allocated = GC.GetAllocatedBytesForCurrentThread() - allocatedBefore;
|
||||
GC.KeepAlive(checksum);
|
||||
long elapsedNs = checked((long)(elapsed * (1_000_000_000.0 / Stopwatch.Frequency)));
|
||||
double seconds = elapsedNs / 1_000_000_000.0;
|
||||
return new Metrics(iterations, elapsedNs, (double)elapsedNs / iterations, iterations / seconds, null, allocated, heapAfter - heapBefore);
|
||||
}
|
||||
|
||||
static Workload[] Workloads() =>
|
||||
[
|
||||
new("uuid_math", "uuid/math", "workloads.json", f => f.UuidMathIterations, UuidMath),
|
||||
new("llsd_xml", "LLSD", "workloads.json", f => f.LlsdIterations, LlsdXml),
|
||||
new("llsd_json", "LLSD", "workloads.json", f => f.LlsdIterations, LlsdJson),
|
||||
new("llsd_binary", "LLSD", "workloads.json", f => f.LlsdIterations, LlsdBinary),
|
||||
new("llsd_notation", "LLSD", "workloads.json", f => f.LlsdIterations, LlsdNotation),
|
||||
new("llsd_protobuf", "LLSD", "workloads.json", f => f.LlsdIterations, LlsdProtobuf),
|
||||
new("packet_codec", "packet codec", "agent_pause.packet", f => f.PacketIterations, PacketCodec),
|
||||
new("asset_decode", "asset decode", "notecard.txt", f => f.AssetIterations, AssetDecode),
|
||||
new("image_decode", "image decode", "image.tga", f => f.ImageIterations, ImageDecode),
|
||||
new("mesh_decode", "mesh decode", "mesh_asset.bin", f => f.MeshIterations, MeshDecode),
|
||||
new("inventory_update", "inventory update", "workloads.json", f => f.InventoryIterations, InventoryUpdate),
|
||||
new("object_update", "object update", "workloads.json", f => f.ObjectIterations, ObjectUpdate),
|
||||
new("rendering", "rendering", "workloads.json", f => f.RenderingIterations, Rendering),
|
||||
new("client_throughput", "client throughput", "workloads.json", f => f.ClientIterations, ClientThroughput)
|
||||
];
|
||||
|
||||
static ulong UuidMath(Context context, int iterations)
|
||||
{
|
||||
Quaternion rotation = Quaternion.CreateFromEulers(0.25f, 0.5f, 0.75f);
|
||||
Matrix4 matrix = Matrix4.CreateTranslation(new Vector3(10, 20, 30));
|
||||
ulong checksum = 0;
|
||||
for (int index = 0; index < iterations; index++)
|
||||
{
|
||||
var parsed = new UUID(context.Fixture.Uuid);
|
||||
matrix = Matrix4.Transform(matrix, rotation);
|
||||
checksum ^= parsed.GetULong() ^ BitConverter.SingleToUInt32Bits(matrix.M41) ^ (ulong)index;
|
||||
}
|
||||
return checksum;
|
||||
}
|
||||
|
||||
static ulong LlsdXml(Context context, int iterations) => LlsdBytes(context, iterations, OSDParser.SerializeLLSDXmlBytes, OSDParser.DeserializeLLSDXml);
|
||||
static ulong LlsdBinary(Context context, int iterations) => LlsdBytes(context, iterations, OSDParser.SerializeLLSDBinary, OSDParser.DeserializeLLSDBinary);
|
||||
static ulong LlsdProtobuf(Context context, int iterations) => LlsdBytes(context, iterations, value => OSDParser.SerializeLLSDProtobuf(value, true), OSDParser.DeserializeLLSDProtobuf);
|
||||
|
||||
static ulong LlsdBytes(Context context, int iterations, Func<OSD, byte[]> serialize, Func<byte[], OSD> deserialize)
|
||||
{
|
||||
ulong checksum = 0;
|
||||
for (int index = 0; index < iterations; index++)
|
||||
{
|
||||
byte[] encoded = serialize(context.Llsd);
|
||||
OSD decoded = deserialize(encoded);
|
||||
checksum ^= (ulong)encoded.Length ^ (decoded.AsBoolean() ? 1UL : 0UL);
|
||||
}
|
||||
return checksum;
|
||||
}
|
||||
|
||||
static ulong LlsdJson(Context context, int iterations)
|
||||
{
|
||||
ulong checksum = 0;
|
||||
for (int index = 0; index < iterations; index++)
|
||||
{
|
||||
string encoded = OSDParser.SerializeJsonString(context.Llsd);
|
||||
OSD decoded = OSDParser.DeserializeJson(encoded);
|
||||
checksum ^= (ulong)encoded.Length ^ (decoded.AsBoolean() ? 1UL : 0UL);
|
||||
}
|
||||
return checksum;
|
||||
}
|
||||
|
||||
static ulong LlsdNotation(Context context, int iterations)
|
||||
{
|
||||
ulong checksum = 0;
|
||||
for (int index = 0; index < iterations; index++)
|
||||
{
|
||||
string encoded = OSDParser.SerializeLLSDNotation(context.Llsd);
|
||||
OSD decoded = OSDParser.DeserializeLLSDNotation(encoded);
|
||||
checksum ^= (ulong)encoded.Length ^ (decoded.AsBoolean() ? 1UL : 0UL);
|
||||
}
|
||||
return checksum;
|
||||
}
|
||||
|
||||
static ulong PacketCodec(Context context, int iterations)
|
||||
{
|
||||
ulong checksum = 0;
|
||||
for (int index = 0; index < iterations; index++)
|
||||
{
|
||||
int position = 0;
|
||||
var packet = new AgentPausePacket(context.Packet, ref position);
|
||||
byte[] encoded = packet.ToBytes();
|
||||
checksum ^= (ulong)encoded.Length ^ packet.AgentData.SerialNum;
|
||||
}
|
||||
return checksum;
|
||||
}
|
||||
|
||||
static ulong AssetDecode(Context context, int iterations)
|
||||
{
|
||||
ulong checksum = 0;
|
||||
for (int index = 0; index < iterations; index++)
|
||||
{
|
||||
var asset = new AssetNotecard(context.Uuid, context.Notecard);
|
||||
if (!asset.Decode()) throw new InvalidDataException("notecard decode returned false");
|
||||
checksum ^= (ulong)asset.BodyText.Length;
|
||||
}
|
||||
return checksum;
|
||||
}
|
||||
|
||||
static ulong ImageDecode(Context context, int iterations)
|
||||
{
|
||||
ulong checksum = 0;
|
||||
for (int index = 0; index < iterations; index++)
|
||||
{
|
||||
using var stream = new MemoryStream(context.Image, false);
|
||||
ManagedImage image = Targa.DecodeToManagedImage(stream);
|
||||
checksum ^= (ulong)image.Red.Length;
|
||||
}
|
||||
return checksum;
|
||||
}
|
||||
|
||||
static ulong MeshDecode(Context context, int iterations)
|
||||
{
|
||||
var renderer = new MeshFoundry();
|
||||
ulong checksum = 0;
|
||||
for (int index = 0; index < iterations; index++)
|
||||
{
|
||||
FacetedMesh mesh = renderer.GenerateFacetedMeshMesh(new Primitive(), context.Mesh, DetailLevel.Highest)
|
||||
?? throw new InvalidDataException("mesh fixture returned no geometry");
|
||||
checksum ^= (ulong)mesh.Faces.Sum(face => face.Vertices.Count + face.Indices.Count);
|
||||
}
|
||||
return checksum;
|
||||
}
|
||||
|
||||
static ulong InventoryUpdate(Context context, int iterations)
|
||||
{
|
||||
var store = new Inventory(new GridClient(), context.Uuid);
|
||||
for (int index = 0; index < iterations; index++)
|
||||
{
|
||||
var item = new InventoryItem(new UUID((ulong)(index % context.Fixture.InventoryItems + 1))) { Name = $"item-{index}" };
|
||||
store.UpdateNodeFor(item);
|
||||
}
|
||||
return (ulong)store.Count;
|
||||
}
|
||||
|
||||
static ulong ObjectUpdate(Context context, int iterations)
|
||||
{
|
||||
Primitive[] objects = Enumerable.Range(0, context.Fixture.ObjectCount).Select(index => new Primitive { LocalID = (uint)index }).ToArray();
|
||||
ulong checksum = 0;
|
||||
for (int index = 0; index < iterations; index++)
|
||||
{
|
||||
Primitive value = objects[index % objects.Length];
|
||||
value.Position = new Vector3(index, index & 255, 21);
|
||||
value.Rotation = Quaternion.CreateFromAxisAngle(Vector3.UnitZ, (index & 255) / 255.0f);
|
||||
checksum ^= value.LocalID ^ BitConverter.SingleToUInt32Bits(value.Position.X);
|
||||
}
|
||||
return checksum;
|
||||
}
|
||||
|
||||
static ulong Rendering(Context context, int iterations)
|
||||
{
|
||||
var renderer = new SimpleRenderer();
|
||||
ulong checksum = 0;
|
||||
for (int index = 0; index < iterations; index++)
|
||||
{
|
||||
FacetedMesh mesh = renderer.GenerateFacetedMesh(BoxPrimitive(), DetailLevel.High);
|
||||
checksum ^= (ulong)mesh.Faces.Sum(face => face.Indices.Count);
|
||||
}
|
||||
return checksum;
|
||||
}
|
||||
|
||||
static Primitive BoxPrimitive() => new()
|
||||
{
|
||||
PrimData = new Primitive.ConstructionData
|
||||
{
|
||||
PCode = PCode.Prim, PathCurve = PathCurve.Line, ProfileCurve = ProfileCurve.Square,
|
||||
PathScaleX = 1, PathScaleY = 1, PathBegin = 0, PathEnd = 1,
|
||||
ProfileBegin = 0, ProfileEnd = 1, ProfileHollow = 0,
|
||||
ProfileHole = HoleType.Same, PathRevolutions = 1
|
||||
}
|
||||
};
|
||||
|
||||
static ulong ClientThroughput(Context context, int iterations)
|
||||
{
|
||||
var client = new GridClient();
|
||||
ulong checksum = 0;
|
||||
for (int index = 0; index < iterations; index++)
|
||||
{
|
||||
for (int batch = 0; batch < context.Fixture.ClientBatch; batch++)
|
||||
{
|
||||
var source = new RemoteParcelRequestReply { ParcelID = context.Uuid };
|
||||
OSDMap map = source.Serialize();
|
||||
var target = new RemoteParcelRequestReply();
|
||||
target.Deserialize(map);
|
||||
checksum ^= target.ParcelID.GetULong() ^ (ulong)index;
|
||||
}
|
||||
}
|
||||
GC.KeepAlive(client);
|
||||
return checksum;
|
||||
}
|
||||
|
||||
sealed class Context
|
||||
{
|
||||
public Context(Fixture fixture, string root)
|
||||
{
|
||||
Fixture = fixture;
|
||||
Uuid = new UUID(fixture.Uuid);
|
||||
Llsd = new OSDMap
|
||||
{
|
||||
["agent_id"] = OSD.FromUUID(Uuid), ["name"] = OSD.FromString("MetaCrate benchmark"),
|
||||
["active"] = OSD.FromBoolean(true), ["score"] = OSD.FromReal(42.25),
|
||||
["items"] = IntegerArray(fixture.LlsdArrayLength)
|
||||
};
|
||||
Packet = File.ReadAllBytes(IOPath.Combine(root, "agent_pause.packet"));
|
||||
Notecard = File.ReadAllBytes(IOPath.Combine(root, "notecard.txt"));
|
||||
Image = File.ReadAllBytes(IOPath.Combine(root, "image.tga"));
|
||||
Mesh = File.ReadAllBytes(IOPath.Combine(root, "mesh_asset.bin"));
|
||||
}
|
||||
private static OSDArray IntegerArray(int count)
|
||||
{
|
||||
var values = new OSDArray(count);
|
||||
for (int index = 0; index < count; index++) values.Add(OSD.FromInteger(index));
|
||||
return values;
|
||||
}
|
||||
public Fixture Fixture { get; }
|
||||
public UUID Uuid { get; }
|
||||
public OSD Llsd { get; }
|
||||
public byte[] Packet { get; }
|
||||
public byte[] Notecard { get; }
|
||||
public byte[] Image { get; }
|
||||
public byte[] Mesh { get; }
|
||||
}
|
||||
|
||||
sealed record Fixture(
|
||||
int Schema, string ReferenceCommit, string Uuid, int UuidMathIterations, int LlsdIterations,
|
||||
int PacketIterations, int AssetIterations, int ImageIterations, int MeshIterations,
|
||||
int InventoryIterations, int ObjectIterations, int RenderingIterations, int ClientIterations,
|
||||
int WarmupDivisor, int LlsdArrayLength, int PacketBlocks, int ImageWidth, int ImageHeight,
|
||||
int MeshSide, int InventoryItems, int ObjectCount, int ClientBatch);
|
||||
sealed record Metrics(int Iterations, long ElapsedNs, double LatencyNs, double ThroughputPerSecond, long? Allocations, long BytesAllocated, long RetainedBytes);
|
||||
sealed record BenchmarkResult(string Name, string Category, string Unit, string Fixture, Metrics Cold, List<Metrics> WarmSamples, Metrics WarmMedian);
|
||||
sealed record EnvironmentInfo(string Os, string Arch, int LogicalCpus, string Toolchain, string Profile);
|
||||
sealed record Report(int Schema, string Runtime, string ReferenceCommit, Dictionary<string, string> FixtureHashes, EnvironmentInfo Environment, List<BenchmarkResult> Results);
|
||||
sealed record Workload(string Name, string Category, string Fixture, Func<Fixture, int> Iterations, Func<Context, int, ulong> Run);
|
||||
BIN
benchmarks/fixtures/agent_pause.packet
Normal file
BIN
benchmarks/fixtures/agent_pause.packet
Normal file
Binary file not shown.
BIN
benchmarks/fixtures/image.tga
Normal file
BIN
benchmarks/fixtures/image.tga
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 64 KiB |
7
benchmarks/fixtures/manifest.json
Normal file
7
benchmarks/fixtures/manifest.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"agent_pause.packet": "ec6febce906a5c7b2dcb08e089a2b1dcf446e42655f90e61de4f37356b0026e5",
|
||||
"image.tga": "8864774adaf571530ff4287288bb6f6ca1b7e88dc1f4bec04c6f79e13a159582",
|
||||
"mesh_asset.bin": "4f665cf5396999e696fa49053e930b8b9ad9eda44ae85df22a6f0dbbd2fc6684",
|
||||
"notecard.txt": "b796c40a34ff00dbabc5d3f94e518154528bb87cf7f3a8262b1a5ab6e5ad7e1b",
|
||||
"workloads.json": "9124763e4822e57850783c06b37cd569355f73ac2397bca85f6dd89e0470267d"
|
||||
}
|
||||
BIN
benchmarks/fixtures/mesh_asset.bin
Normal file
BIN
benchmarks/fixtures/mesh_asset.bin
Normal file
Binary file not shown.
9
benchmarks/fixtures/notecard.txt
Normal file
9
benchmarks/fixtures/notecard.txt
Normal file
@@ -0,0 +1,9 @@
|
||||
Linden text version 2
|
||||
{
|
||||
LLEmbeddedItems version 1
|
||||
{
|
||||
count 0
|
||||
}
|
||||
Text length 127
|
||||
MetaCrate pinned performance fixture. This notecard exercises bounded asset parsing without a network or CLR dependency. 012345
|
||||
}
|
||||
24
benchmarks/fixtures/workloads.json
Normal file
24
benchmarks/fixtures/workloads.json
Normal file
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"reference_commit": "2aa70bb68513b39795da5d13c88f31b86e85a3ba",
|
||||
"uuid": "01234567-89ab-cdef-0123-456789abcdef",
|
||||
"uuid_math_iterations": 100000,
|
||||
"llsd_iterations": 2000,
|
||||
"packet_iterations": 5000,
|
||||
"asset_iterations": 1000,
|
||||
"image_iterations": 100,
|
||||
"mesh_iterations": 25,
|
||||
"inventory_iterations": 10000,
|
||||
"object_iterations": 20000,
|
||||
"rendering_iterations": 250,
|
||||
"client_iterations": 5000,
|
||||
"warmup_divisor": 10,
|
||||
"llsd_array_length": 32,
|
||||
"packet_blocks": 16,
|
||||
"image_width": 128,
|
||||
"image_height": 128,
|
||||
"mesh_side": 48,
|
||||
"inventory_items": 128,
|
||||
"object_count": 256,
|
||||
"client_batch": 32
|
||||
}
|
||||
26
benchmarks/release-criteria.json
Normal file
26
benchmarks/release-criteria.json
Normal file
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"maximum_latency_ratio": 2.0,
|
||||
"maximum_allocation_ratio": 2.0,
|
||||
"material_latency_regression_ratio": 1.25,
|
||||
"minimum_reference_latency_ns": 1000.0,
|
||||
"reviewed_exceptions": [
|
||||
{
|
||||
"workload": "llsd_json",
|
||||
"maximum_latency_ratio": 2.0,
|
||||
"maximum_allocation_ratio": 2.1,
|
||||
"rationale": "the bounded owned Rust tree is more than four times faster; up to 10% extra allocation bytes are accepted"
|
||||
},
|
||||
{
|
||||
"workload": "rendering",
|
||||
"maximum_latency_ratio": 8.0,
|
||||
"maximum_allocation_ratio": 4.0,
|
||||
"rationale": "Rust produces the complete checked six-face box mesh; the pinned SimpleRenderer is a documented one-face placeholder"
|
||||
},
|
||||
{
|
||||
"workload": "client_throughput",
|
||||
"maximum_latency_ratio": 2.0,
|
||||
"maximum_allocation_ratio": 2.0,
|
||||
"rationale": "Rust returns checked Results and owns the bounded OSD map while allocating fewer bytes; the measured latency remains below the reviewed hard limit"
|
||||
}
|
||||
]
|
||||
}
|
||||
194
benchmarks/results/comparison.json
Normal file
194
benchmarks/results/comparison.json
Normal file
@@ -0,0 +1,194 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"reference_commit": "2aa70bb68513b39795da5d13c88f31b86e85a3ba",
|
||||
"fixture_hashes": {
|
||||
"agent_pause.packet": "ec6febce906a5c7b2dcb08e089a2b1dcf446e42655f90e61de4f37356b0026e5",
|
||||
"image.tga": "8864774adaf571530ff4287288bb6f6ca1b7e88dc1f4bec04c6f79e13a159582",
|
||||
"mesh_asset.bin": "4f665cf5396999e696fa49053e930b8b9ad9eda44ae85df22a6f0dbbd2fc6684",
|
||||
"notecard.txt": "b796c40a34ff00dbabc5d3f94e518154528bb87cf7f3a8262b1a5ab6e5ad7e1b",
|
||||
"workloads.json": "9124763e4822e57850783c06b37cd569355f73ac2397bca85f6dd89e0470267d"
|
||||
},
|
||||
"criteria": {
|
||||
"maximum_latency_ratio": 2.0,
|
||||
"maximum_allocation_ratio": 2.0,
|
||||
"material_latency_regression_ratio": 1.25,
|
||||
"minimum_reference_latency_ns": 1000.0,
|
||||
"reviewed_exceptions": [
|
||||
{
|
||||
"workload": "llsd_json",
|
||||
"maximum_latency_ratio": 2.0,
|
||||
"maximum_allocation_ratio": 2.1,
|
||||
"rationale": "the bounded owned Rust tree is more than four times faster; up to 10% extra allocation bytes are accepted"
|
||||
},
|
||||
{
|
||||
"workload": "rendering",
|
||||
"maximum_latency_ratio": 8.0,
|
||||
"maximum_allocation_ratio": 4.0,
|
||||
"rationale": "Rust produces the complete checked six-face box mesh; the pinned SimpleRenderer is a documented one-face placeholder"
|
||||
},
|
||||
{
|
||||
"workload": "client_throughput",
|
||||
"maximum_latency_ratio": 2.0,
|
||||
"maximum_allocation_ratio": 2.0,
|
||||
"rationale": "Rust returns checked Results and owns the bounded OSD map while allocating fewer bytes; the measured latency remains below the reviewed hard limit"
|
||||
}
|
||||
]
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"name": "uuid_math",
|
||||
"rust_latency_ns": 107.41872,
|
||||
"reference_latency_ns": 105.47209,
|
||||
"latency_ratio": 1.0184563518178127,
|
||||
"rust_bytes_per_operation": 36.0,
|
||||
"reference_bytes_per_operation": 40.0,
|
||||
"allocation_ratio": 0.9,
|
||||
"accepted": true,
|
||||
"criterion": "within default release thresholds"
|
||||
},
|
||||
{
|
||||
"name": "llsd_xml",
|
||||
"rust_latency_ns": 23361.1135,
|
||||
"reference_latency_ns": 38735.79,
|
||||
"latency_ratio": 0.6030886035885675,
|
||||
"rust_bytes_per_operation": 10938.64,
|
||||
"reference_bytes_per_operation": 28600.0,
|
||||
"allocation_ratio": 0.3824699300699301,
|
||||
"accepted": true,
|
||||
"criterion": "within default release thresholds"
|
||||
},
|
||||
{
|
||||
"name": "llsd_json",
|
||||
"rust_latency_ns": 5231.5815,
|
||||
"reference_latency_ns": 21724.418,
|
||||
"latency_ratio": 0.24081572634074708,
|
||||
"rust_bytes_per_operation": 7446.936,
|
||||
"reference_bytes_per_operation": 3680.0,
|
||||
"allocation_ratio": 2.0236239130434783,
|
||||
"accepted": true,
|
||||
"criterion": "reviewed exception: the bounded owned Rust tree is more than four times faster; up to 10% extra allocation bytes are accepted"
|
||||
},
|
||||
{
|
||||
"name": "llsd_binary",
|
||||
"rust_latency_ns": 2688.7605,
|
||||
"reference_latency_ns": 11074.724,
|
||||
"latency_ratio": 0.24278352218980806,
|
||||
"rust_bytes_per_operation": 6749.016,
|
||||
"reference_bytes_per_operation": 7960.0,
|
||||
"allocation_ratio": 0.8478663316582914,
|
||||
"accepted": true,
|
||||
"criterion": "within default release thresholds"
|
||||
},
|
||||
{
|
||||
"name": "llsd_notation",
|
||||
"rust_latency_ns": 7007.081,
|
||||
"reference_latency_ns": 18866.287,
|
||||
"latency_ratio": 0.37140752708786845,
|
||||
"rust_bytes_per_operation": 8488.784,
|
||||
"reference_bytes_per_operation": 10096.0,
|
||||
"allocation_ratio": 0.8408066561014262,
|
||||
"accepted": true,
|
||||
"criterion": "within default release thresholds"
|
||||
},
|
||||
{
|
||||
"name": "llsd_protobuf",
|
||||
"rust_latency_ns": 7337.0635,
|
||||
"reference_latency_ns": 30965.56,
|
||||
"latency_ratio": 0.23694270344214668,
|
||||
"rust_bytes_per_operation": 8323.672,
|
||||
"reference_bytes_per_operation": 26120.0,
|
||||
"allocation_ratio": 0.31867044410413475,
|
||||
"accepted": true,
|
||||
"criterion": "within default release thresholds"
|
||||
},
|
||||
{
|
||||
"name": "packet_codec",
|
||||
"rust_latency_ns": 208.5864,
|
||||
"reference_latency_ns": 175.8286,
|
||||
"latency_ratio": 1.186305299592899,
|
||||
"rust_bytes_per_operation": 124.0,
|
||||
"reference_bytes_per_operation": 264.0,
|
||||
"allocation_ratio": 0.4696969696969697,
|
||||
"accepted": true,
|
||||
"criterion": "within default release thresholds"
|
||||
},
|
||||
{
|
||||
"name": "asset_decode",
|
||||
"rust_latency_ns": 288.205,
|
||||
"reference_latency_ns": 2980.13,
|
||||
"latency_ratio": 0.09670886840506956,
|
||||
"rust_bytes_per_operation": 1246.0,
|
||||
"reference_bytes_per_operation": 4512.0,
|
||||
"allocation_ratio": 0.27615248226950356,
|
||||
"accepted": true,
|
||||
"criterion": "within default release thresholds"
|
||||
},
|
||||
{
|
||||
"name": "image_decode",
|
||||
"rust_latency_ns": 42671.19,
|
||||
"reference_latency_ns": 48273.63,
|
||||
"latency_ratio": 0.8839440912150175,
|
||||
"rust_bytes_per_operation": 65536.0,
|
||||
"reference_bytes_per_operation": 131612.16,
|
||||
"allocation_ratio": 0.4979479099803544,
|
||||
"accepted": true,
|
||||
"criterion": "within default release thresholds"
|
||||
},
|
||||
{
|
||||
"name": "mesh_decode",
|
||||
"rust_latency_ns": 413294.68,
|
||||
"reference_latency_ns": 378537.04,
|
||||
"latency_ratio": 1.0918209747717158,
|
||||
"rust_bytes_per_operation": 507739.92,
|
||||
"reference_bytes_per_operation": 564592.96,
|
||||
"allocation_ratio": 0.8993026055443554,
|
||||
"accepted": true,
|
||||
"criterion": "within default release thresholds"
|
||||
},
|
||||
{
|
||||
"name": "inventory_update",
|
||||
"rust_latency_ns": 972.1585,
|
||||
"reference_latency_ns": 757.9809,
|
||||
"latency_ratio": 1.282563320526942,
|
||||
"rust_bytes_per_operation": 373.3421,
|
||||
"reference_bytes_per_operation": 356.5016,
|
||||
"allocation_ratio": 1.0472382171636818,
|
||||
"accepted": true,
|
||||
"criterion": "within default release thresholds"
|
||||
},
|
||||
{
|
||||
"name": "object_update",
|
||||
"rust_latency_ns": 27.1764,
|
||||
"reference_latency_ns": 45.77345,
|
||||
"latency_ratio": 0.5937153524586851,
|
||||
"rust_bytes_per_operation": 16.1792,
|
||||
"reference_bytes_per_operation": 10.6552,
|
||||
"allocation_ratio": 1.5184323147383438,
|
||||
"accepted": true,
|
||||
"criterion": "within default release thresholds"
|
||||
},
|
||||
{
|
||||
"name": "rendering",
|
||||
"rust_latency_ns": 5218.82,
|
||||
"reference_latency_ns": 1013.564,
|
||||
"latency_ratio": 5.14897924551385,
|
||||
"rust_bytes_per_operation": 9480.0,
|
||||
"reference_bytes_per_operation": 3048.096,
|
||||
"allocation_ratio": 3.110138263361784,
|
||||
"accepted": true,
|
||||
"criterion": "reviewed exception: Rust produces the complete checked six-face box mesh; the pinned SimpleRenderer is a documented one-face placeholder"
|
||||
},
|
||||
{
|
||||
"name": "client_throughput",
|
||||
"rust_latency_ns": 6231.7982,
|
||||
"reference_latency_ns": 3795.4216,
|
||||
"latency_ratio": 1.6419251552976355,
|
||||
"rust_bytes_per_operation": 11795.3474,
|
||||
"reference_bytes_per_operation": 12053.7952,
|
||||
"allocation_ratio": 0.9785588027910082,
|
||||
"accepted": true,
|
||||
"criterion": "reviewed exception: Rust returns checked Results and owns the bounded OSD map while allocating fewer bytes; the measured latency remains below the reviewed hard limit"
|
||||
}
|
||||
],
|
||||
"accepted": true
|
||||
}
|
||||
1267
benchmarks/results/csharp-linux-x86_64.json
Normal file
1267
benchmarks/results/csharp-linux-x86_64.json
Normal file
File diff suppressed because it is too large
Load Diff
1267
benchmarks/results/rust-linux-x86_64.json
Normal file
1267
benchmarks/results/rust-linux-x86_64.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user