RuDS/RELEASEBUILD.md

Portable Gitea Release Builds from a Linux ARM64 Runner

This document is a blueprint for producing native Rust application releases for Linux ARM64, Linux x64, macOS ARM64, macOS x64, and Windows x64 from a Gitea Actions runner whose host is Ubuntu ARM64. It records the design and the failure modes proven while preparing the RuDS 0.9.0 release.

The important distinction is that the release tooling is portable Rust, not that every application dependency is pure Rust. GTK and WebKitGTK are normal Linux system libraries and should be linked from the runner's Ubuntu packages. Apple bundles need portable construction because no macOS runner is available.

The implementation in this repository is split between:

• .gitea/workflows/release.yml: orchestration and target-specific build setup.
• .gitea/scripts/install-linux-build-dependencies.sh: native ARM64 packages and the extracted x64 Ubuntu sysroot.
• .gitea/scripts/x86_64-linux-gnu-gcc-pic and .gitea/scripts/x86_64-linux-gnu-g++-pic: cross-compiler wrappers.
• crates/bds-release: release notes, Gitea API access, dependency downloads, archive creation, Apple bundle construction, signing, and upload.

Resulting artifact matrix

Build CLI and desktop artifacts separately so users do not have to download a GUI application to obtain command-line tools. RuDS creates two assets per target:

Target	CLI	Desktop
macOS ARM64	.tar.gz	.dmg containing an .app
macOS x64	.tar.gz	.dmg containing an .app
Linux ARM64	.tar.gz	.tar.gz
Linux x64	.tar.gz	.tar.gz
Windows x64	.zip	portable .zip

Compile all final programs for a target in one Cargo invocation. In this repository those are bds-ui, bds-cli, and bds-mcp. This keeps their feature selection and shared runtime libraries consistent.

Pipeline shape

Use a tag-triggered, draft-first pipeline:

1. Check out full history, test the release helper, generate notes, and create or update a draft Gitea release.
2. Build and upload every platform into that release ID.
3. Publish only after every build job succeeds.
4. Query Gitea afterward and verify the public release, target commit, and asset manifest.

A minimal dependency graph is:

prepare draft
    |
    +-- Linux x64
            |
            +-- Linux ARM64
            +-- macOS ARM64 + x64
            +-- Windows x64
                    |
                    +-- publish

RuDS deliberately runs the most failure-prone cross-link first. On a single runner all jobs serialize anyway, and failing early avoids spending hours on other targets first. With several independent runners, make all platform jobs depend only on preparation and let them run in parallel.

The draft is the transaction boundary. Never make a release public before all assets exist. The final publisher should ideally fetch the draft and compare the exact expected filenames and non-zero sizes before setting draft: false. RuDS currently relies on job dependencies plus strict per-package input checks; adding an explicit final manifest check would make that boundary stronger.

Gitea runner and permissions

Register a label that describes the real machine, for example linux-arm64:host, and use runs-on: linux-arm64. Do not treat a name such as ubuntu-latest as an implicit x64 machine. Gitea runner labels have the form label[:schema[:args]].

The runner needs:

• Enough disk for Cargo registries, several target trees, cross SDKs, sysroots, and container layers.
• Outbound HTTPS access to Rust distribution servers, crates.io, Gitea, NuGet, GitHub release assets, and the configured container registry.
• Passwordless access to install Ubuntu build packages, or a prebuilt runner image containing them.
• Container support for the macOS and Windows build images.
• A URL by which build containers can reach Gitea. localhost inside a container is the container itself, not the runner host.

Container images must publish an ARM64 image even when the target they build is x64. Pin images by digest, not a floating tag. Also pin the Rust toolchain and always build with --locked.

Grant the workflow only the permissions it needs: repository contents read, issues read if release notes inspect issues, and releases write. Gitea clamps workflow permissions to repository or owner limits. If the built-in secrets.GITEA_TOKEN cannot write releases on the deployed Gitea version, use a repository secret containing a narrowly scoped token.

Gitea Actions resembles GitHub Actions but is not identical. Validate contexts, permissions, output files, and action compatibility against the deployed Gitea and act_runner versions. Non-fully-qualified actions may be downloaded from GitHub depending on DEFAULT_ACTIONS_URL; mirror or fully qualify them on restricted installations.

Pass the draft release ID through a job output. Gitea supports the familiar key=value step-output file convention; this repository writes to the GITHUB_OUTPUT compatibility variable and maps that step output to a job output. Do not scrape an ID from human-readable command output.

Specialized build containers may not run JavaScript-based checkout actions reliably. RuDS performs an authenticated, depth-one git fetch of the exact tag inside those containers. Fetch the tag ref explicitly and check out FETCH_HEAD; do not build a mutable branch tip.

Linux desktop builds

There is no architectural reason to exclude the GTK/WebKit desktop application from Linux artifacts. The native ARM64 build is ordinary compilation on the ARM64 Ubuntu runner. Install:

build-essential
cmake
libgtk-3-dev
libwebkit2gtk-4.1-dev
libxdo-dev
pkg-config

The resulting application dynamically links the distro GTK/WebKit stack. Do not put the entire GTK/WebKit dependency closure into the application archive. Bundle application-owned shared libraries, dynamically linked Rust standard libraries, and explicitly redistributable runtimes; document the required Linux runtime packages for users.

Build on the oldest distro baseline that the release promises to support. Using a rolling ubuntu-latest silently raises the minimum glibc and WebKitGTK versions when the runner image changes. A fixed Ubuntu release or container is more reproducible.

Add an $ORIGIN runtime search path so executables find shared libraries placed beside them:

-C link-arg=-Wl,-rpath,$ORIGIN

Cross-linking Linux x64 on ARM64

Rust cross-compilation is the easy part. Native dependencies selected through pkg-config, C/C++ build scripts, the linker, and the x64 libc are the parts that require deliberate configuration.

On an Ubuntu ARM64 host:

1. Restrict the existing Ubuntu deb822 sources to arm64.
2. Add amd64 as a foreign architecture and add matching amd64 archive and security sources for the same Ubuntu codename.
3. Install native host tools such as gcc-x86-64-linux-gnu, g++-x86-64-linux-gnu, cmake, and pkg-config.
4. Download the GTK, WebKitGTK, and xdo amd64 development packages and their dependencies with APT's --download-only mode.
5. Extract the .deb files with dpkg-deb --extract into an isolated sysroot, such as /opt/ruds-x64-sysroot.
6. Point Cargo, the GNU cross-linker, and pkg-config exclusively at that sysroot.

Do not normally install the foreign development packages on the ARM64 host. Some packages contain target-architecture helper executables or maintainer scripts; package installation can attempt to execute x64 programs on ARM64. Downloading and extracting produces the headers, libraries, and metadata needed for linking without running target code.

The essential environment is:

AR_x86_64_unknown_linux_gnu=x86_64-linux-gnu-ar
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc
PKG_CONFIG_ALLOW_CROSS=1
PKG_CONFIG_SYSROOT_DIR=/opt/ruds-x64-sysroot
PKG_CONFIG_LIBDIR=/opt/ruds-x64-sysroot/usr/lib/x86_64-linux-gnu/pkgconfig:/opt/ruds-x64-sysroot/usr/lib/pkgconfig:/opt/ruds-x64-sysroot/usr/share/pkgconfig

Pass --sysroot to the linker, add the target library directories with Cargo -L native=..., and retain the $ORIGIN rpath. Do not leave host pkg-config paths in the search list: that can produce an ARM64/x64 mixture which may fail late or, worse, appear to link successfully.

Ubuntu libc development linker scripts may refer to absolute legacy paths such as /lib/x86_64-linux-gnu. Those paths are interpreted inside the sysroot but may not exist in an extracted modern Ubuntu package layout. Materialize the required libc, libm, libmvec, and loader paths within the sysroot. Do not create host-level x64 symlinks.

Architecture-independent .pc files may already be installed on the host, so APT may not download them again for amd64. They contain metadata rather than target executables and can be copied into the sysroot; PKG_CONFIG_SYSROOT_DIR then redirects their paths.

Some native Rust dependencies build static objects that are later linked into application shared libraries. They need position-independent code. Compiler wrappers should append -fPIC as the final flag so it overrides earlier flags. Target-specific Cargo environment names can contain hyphens and therefore are not valid shell assignment identifiers. Set them through env, for example:

env \
  "CC_x86_64-unknown-linux-gnu=$PWD/.gitea/scripts/x86_64-linux-gnu-gcc-pic" \
  "CXX_x86_64-unknown-linux-gnu=$PWD/.gitea/scripts/x86_64-linux-gnu-g++-pic" \
  cargo build --release --locked --target x86_64-unknown-linux-gnu

Audit every -sys crate for pkg-config overrides. For RuDS, ZSTD_SYS_USE_PKG_CONFIG must be unset in the x64 build so zstd does not select an incompatible host library.

macOS without a macOS runner

Use cargo-zigbuild in a digest-pinned ARM64-compatible container to compile the aarch64-apple-darwin and x86_64-apple-darwin targets. The container must contain a legally usable Apple SDK. Review the SDK and redistribution licenses; portable tooling does not remove those obligations.

RuDS supplies these linker requirements:

• Both @executable_path and @executable_path/../Resources rpaths. The first serves the flat CLI archive and the second serves the .app layout.
• Header padding for later Mach-O load-command rewriting.
• The chosen minimum macOS deployment version.
• -Wl,-undefined,dynamic_lookup where Apple symbols are intentionally resolved at runtime.

Zig's Darwin linker does not implement every ld64 option. In particular, symbol-specific -U handling was not usable here. Broad -undefined,dynamic_lookup works when late-bound framework symbols are truly required, but it can also mask accidental unresolved symbols; use it only with dependency inspection and runtime testing.

The Zig Apple link path expected a compiler runtime archive alias. A valid empty libclang_rt.osx.a archive (!<arch>\n) in an explicit native library directory satisfied that lookup.

Intel macOS and version-sensitive native runtimes

Do not assume that the latest machine-learning runtime publishes every Apple architecture. The ONNX Runtime release used by newer Rust crates no longer had an Intel macOS binary. RuDS keeps a compatible dependency line:

fastembed 5.12.0
ort/ort-sys 2.0.0-rc.11
ONNX Runtime 1.23.2 for macOS x64

The Rust release helper downloads Microsoft's x64 archive, verifies its pinned SHA-256 digest, extracts the versioned dylib and license, and creates an unversioned linker copy. With ort-sys rc.11, the working build variables are:

ORT_LIB_LOCATION=<directory-containing-libonnxruntime.dylib>
ORT_PREFER_DYNAMIC_LINK=1

ORT_LIB_PATH, documented or accepted by other versions, is ignored by this specific prerelease. When a native dependency behaves unexpectedly, read the build script for the exact locked crate version rather than documentation for a newer release.

Pure-Rust Apple packaging

Build the application bundle and DMG with Rust crates rather than hdiutil, codesign, shell archive tricks, or a simulated macOS environment. RuDS uses Rust libraries to:

1. Create Contents/MacOS, Contents/Resources, Info.plist, the icon, bundle metadata, and URL handlers.
2. Put executables in Contents/MacOS and dylibs/licenses in Resources.
3. Parse 64-bit little-endian Mach-O load commands and rewrite application dylib IDs and references to @rpath/<basename>.
4. Sign after all binary mutations.
5. Construct and sign the DMG.

Mach-O load commands have fixed allocated space. Reserve header padding at link time or a longer rewritten install name will not fit. Rewrite every applicable load command, including weak, re-exported, lazy, and upward dylib loads, and never attempt to parse licenses or other resource files as Mach-O binaries.

The current output is ad-hoc signed. Developer ID signing and Apple notarization need Apple-issued credentials and a supported submission path; a Linux-only pipeline cannot claim notarized output merely because it can construct and ad-hoc-sign a valid bundle.

Windows x64 from Linux ARM64

Use cargo-xwin in a digest-pinned ARM64-compatible container. It provides the Windows MSVC SDK/import-library environment without requiring a Windows host. Install the Rust x86_64-pc-windows-msvc target and NASM when native assembly dependencies require it, then populate the xwin cache before looking up SDK libraries.

RuDS's DirectML-enabled build needs:

• Microsoft DirectML 1.15.4 downloaded from NuGet and verified by SHA-256.
• DirectML.lib for linking, plus DirectML.dll and its license in the final package.
• A correctly cased PathCch.lib alias copied from the xwin SDK, because some build tooling expects that spelling.
• The target release directory on Cargo's native library search path.

Do not force target-feature=+crt-static merely to make the archive look more self-contained. Windows system and SDK dependencies may require the dynamic CRT, and forcing static CRT can produce incompatible link combinations. Bundle the redistributable application runtime DLLs that are actually required and retain their licenses. RuDS statically links ONNX Runtime on Windows but ships DirectML dynamically.

Portable ZIPs are the simplest cross-host Windows result. MSI, MSIX, or signed installer production introduces additional tooling, signing certificates, and Windows policy requirements and should be treated as separate scope.

Packaging rules

Keep packaging in a small host-native Rust binary. bds-release runs on the ARM64 builder even while it packages foreign target files. It uses Rust for HTTP, hashing, tar/gzip, ZIP, plist, ICNS, Apple bundle/DMG construction, Mach-O mutation, signing, and Gitea API calls.

For each target:

• Require every expected executable and application shared library before creating an archive.
• Copy dynamic Rust standard libraries from rustc --target <target> --print target-libdir when the project links Rust's standard library dynamically.
• Include application-owned shared libraries and required redistributable third-party runtimes with license files.
• Give archives deterministic, tag-derived names and a single top-level directory.
• Reject unsafe tags used as filenames and refuse to overwrite existing output.
• Stream large uploads with a content length rather than loading them into RAM.
• Retry network failures, HTTP 429, and server errors with bounded backoff.
• Verify every downloaded native dependency with a pinned cryptographic digest.

Packaging must follow the target's runtime model. Linux should use the distro's GTK/WebKit libraries; macOS .app bundles should carry their private dylibs in Resources with corrected install names; Windows packages should carry approved redistributable DLLs. “Put every shared library beside the executable” is not a portable policy.

Release creation and rebuilding

The preparation job uses full Git history to locate the previous tag, reads issue numbers from commit messages, fetches closed issues through Gitea's API, and creates categorized release notes. It writes the draft release ID as a step output so every build uploads to the same release.

Rebuilding an existing published tag is deliberately explicit. In RuDS, REBUILD_PUBLISHED_RELEASE=true changes the release back to a draft, deletes its assets, regenerates its metadata, and uploads a consistent new set. This is destructive and temporarily removes the public release. Prefer a new immutable tag unless repairing an existing release is intentional. Rerunning a failed draft also clears its partial asset set before rebuilding, which prevents stale and new files from being mixed.

Before pushing a release tag:

1. Update the workspace version, lockfile, README, and any visible version metadata together.
2. Run formatting, the full workspace tests, and relevant release-helper tests.
3. Build locally for the host where practical.
4. Commit and push the release commit to the default branch.
5. Create an annotated tag at that exact commit, then push the tag.

After the workflow finishes, verify through the Gitea API or tea:

• Every job is successful rather than merely the workflow appearing complete.
• The tag and its peeled commit match the intended default-branch commit.
• The release is public (draft: false) and has the expected prerelease flag.
• The release targets the exact tagged commit.
• The expected asset names are present exactly once and have non-zero sizes.
• The repository worktree is clean and local/remote refs agree.

Useful Gitea API endpoints are:

GET /repos/{owner}/{repo}/actions/runs
GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs
GET /repos/{owner}/{repo}/actions/jobs/{job_id}/logs
GET /repos/{owner}/{repo}/releases/tags/{tag}

Job logs can be very large. Download or query them once, then filter locally for the failing step and nearby compiler/linker diagnostics instead of repeatedly printing the entire log.

Common failures

Symptom	Cause	Fix
Exec format error during x64 dependency installation	A foreign package's executable or maintainer script ran on ARM64	Download packages only and extract them into a sysroot
Linker reports wrong ELF architecture	Host libraries leaked through pkg-config or a build script	Restrict PKG_CONFIG_LIBDIR, set its sysroot, and audit -sys crate overrides
Relocation errors while creating a .so	A C/C++ static object was built without PIC	Append -fPIC in target compiler wrappers
libc or loader missing despite downloaded packages	Ubuntu linker scripts contain absolute legacy paths	Materialize those target paths inside the sysroot
Linux binary starts but cannot find project .so files	No relative runtime search path	Link with an $ORIGIN rpath and package the libraries beside the binary
macOS link rejects -U	Zig's Darwin linker differs from Apple ld64	Use supported options; use dynamic lookup only when justified
Mach-O install-name rewrite does not fit	No load-command header padding	Add header padding before linking, then mutate before signing
Intel macOS ORT download fails	The selected upstream release dropped x64 macOS	Pin a compatible crate/runtime pair and supply a verified dylib
ort-sys ignores a supplied runtime	Environment variable belongs to another crate version	Inspect the exact locked crate build script; rc.11 uses ORT_LIB_LOCATION
Windows library is “missing” despite SDK contents	Import-library case/name differs from build-script expectation	Add an explicit linker alias such as PathCch.lib
Windows CRT link conflicts	Static CRT was forced across incompatible dependencies	Use the target's normal dynamic CRT unless the full graph supports static CRT
Checkout fails only inside a build container	The image cannot execute the checkout action or cannot reach Gitea	Fetch the exact tag with authenticated Git in the container
Release is public but incomplete	Publication happened before asset completion	Create a draft first and publish only behind all build jobs and manifest checks
A release takes hours	Cold Rust/native builds on one physical ARM64 runner serialize	Add real runners, carefully cache immutable inputs, or prebake dependencies

Performance and reproducibility

A cold five-target build with GTK/WebKit, ONNX Runtime, and ML dependencies can take about three hours on a single ARM64 runner. The workflow itself is simple; compilation and native dependency setup dominate the time.

Safe ways to reduce it include:

• Add separate runners with the same label, or platform-specific labels, so independent target jobs truly run concurrently.
• Prebuild a versioned runner/container image containing system dependencies and the Linux x64 sysroot.
• Cache Cargo registry, Git checkouts, target artifacts, Rust toolchains, and cross-SDK caches with keys including the lockfile, Rust version, target, feature set, distro version, and relevant build scripts.
• Avoid concurrent memory-heavy jobs on one physical host even if several logical runners are registered there.

Prefer a slow clean build over a fast cache that can mix architectures or stale native libraries. Keep caches disabled until their keys and invalidation rules are demonstrably correct.

Validation depth

Cross-built programs cannot be fully exercised on the ARM64 host, but artifacts should still receive structural checks:

• ELF: inspect architecture, DT_NEEDED, rpath, and unexpected build-host paths with file and readelf.
• Mach-O: inspect architectures, load commands, bundle layout, plist, signatures, and DMG contents with parsers that run on Linux.
• PE: inspect machine type and imports with LLVM or MinGW object tools.
• Archives: extract into temporary directories, compare the manifest, and check executable modes and licenses.

Then smoke-test on real target systems. QEMU or Wine can catch basic loader failures but is not authoritative for GTK/WebKit behavior, macOS signing and Gatekeeper, or Windows GPU/DirectML behavior.

References

• Gitea Actions
• Gitea Actions design and runner labels
• Gitea Actions variables
• Gitea API
• cargo-zigbuild
• cargo-xwin

Treat the checked-in workflow and Rust helper as the executable source of truth. This document explains why their unusual-looking pieces exist so that future agents do not remove them as apparent complexity and rediscover the same target, linker, packaging, and release-state failures.