# Portable Gitea Release Builds from a Linux ARM64 Runner This document is a blueprint for producing native Rust application releases for Linux ARM64, Linux x64, macOS ARM64, macOS x64, and Windows x64 from a Gitea Actions runner whose host is Ubuntu ARM64. It records the design and the failure modes proven while preparing the RuDS 0.9.0 release. The important distinction is that the release tooling is portable Rust, not that every application dependency is pure Rust. GTK and WebKitGTK are normal Linux system libraries and should be linked from the runner's Ubuntu packages. Apple bundles need portable construction because no macOS runner is available. The implementation in this repository is split between: - `.gitea/workflows/release.yml`: orchestration and target-specific build setup. - `.gitea/scripts/install-linux-build-dependencies.sh`: native ARM64 packages and the extracted x64 Ubuntu sysroot. - `.gitea/scripts/x86_64-linux-gnu-gcc-pic` and `.gitea/scripts/x86_64-linux-gnu-g++-pic`: cross-compiler wrappers. - `crates/bds-release`: release notes, Gitea API access, dependency downloads, archive creation, Apple bundle construction, signing, and upload. ## Resulting artifact matrix Build CLI and desktop artifacts separately so users do not have to download a GUI application to obtain command-line tools. RuDS creates two assets per target: | Target | CLI | Desktop | | --- | --- | --- | | macOS ARM64 | `.tar.gz` | `.dmg` containing an `.app` | | macOS x64 | `.tar.gz` | `.dmg` containing an `.app` | | Linux ARM64 | `.tar.gz` | `.tar.gz` | | Linux x64 | `.tar.gz` | `.tar.gz` | | Windows x64 | `.zip` | portable `.zip` | Compile all final programs for a target in one Cargo invocation. In this repository those are `bds-ui`, `bds-cli`, and `bds-mcp`. This keeps their feature selection and shared runtime libraries consistent. ## Pipeline shape Use a tag-triggered, draft-first pipeline: 1. Check out full history, test the release helper, generate notes, and create or update a draft Gitea release. 2. Build and upload every platform into that release ID. 3. Publish only after every build job succeeds. 4. Query Gitea afterward and verify the public release, target commit, and asset manifest. A minimal dependency graph is: ```text prepare draft | +-- Linux x64 | +-- Linux ARM64 +-- macOS ARM64 + x64 +-- Windows x64 | +-- publish ``` RuDS deliberately runs the most failure-prone cross-link first. On a single runner all jobs serialize anyway, and failing early avoids spending hours on other targets first. With several independent runners, make all platform jobs depend only on preparation and let them run in parallel. The draft is the transaction boundary. Never make a release public before all assets exist. The final publisher should ideally fetch the draft and compare the exact expected filenames and non-zero sizes before setting `draft: false`. RuDS currently relies on job dependencies plus strict per-package input checks; adding an explicit final manifest check would make that boundary stronger. ## Gitea runner and permissions Register a label that describes the real machine, for example `linux-arm64:host`, and use `runs-on: linux-arm64`. Do not treat a name such as `ubuntu-latest` as an implicit x64 machine. Gitea runner labels have the form `label[:schema[:args]]`. The runner needs: - Enough disk for Cargo registries, several target trees, cross SDKs, sysroots, and container layers. - Outbound HTTPS access to Rust distribution servers, crates.io, Gitea, NuGet, GitHub release assets, and the configured container registry. - Passwordless access to install Ubuntu build packages, or a prebuilt runner image containing them. - Container support for the macOS and Windows build images. - A URL by which build containers can reach Gitea. `localhost` inside a container is the container itself, not the runner host. Container images must publish an ARM64 image even when the target they build is x64. Pin images by digest, not a floating tag. Also pin the Rust toolchain and always build with `--locked`. Grant the workflow only the permissions it needs: repository contents read, issues read if release notes inspect issues, and releases write. Gitea clamps workflow permissions to repository or owner limits. If the built-in `secrets.GITEA_TOKEN` cannot write releases on the deployed Gitea version, use a repository secret containing a narrowly scoped token. Gitea Actions resembles GitHub Actions but is not identical. Validate contexts, permissions, output files, and action compatibility against the deployed Gitea and `act_runner` versions. Non-fully-qualified actions may be downloaded from GitHub depending on `DEFAULT_ACTIONS_URL`; mirror or fully qualify them on restricted installations. Pass the draft release ID through a job output. Gitea supports the familiar `key=value` step-output file convention; this repository writes to the `GITHUB_OUTPUT` compatibility variable and maps that step output to a job output. Do not scrape an ID from human-readable command output. Specialized build containers may not run JavaScript-based checkout actions reliably. RuDS performs an authenticated, depth-one `git fetch` of the exact tag inside those containers. Fetch the tag ref explicitly and check out `FETCH_HEAD`; do not build a mutable branch tip. ## Linux desktop builds There is no architectural reason to exclude the GTK/WebKit desktop application from Linux artifacts. The native ARM64 build is ordinary compilation on the ARM64 Ubuntu runner. Install: ```text build-essential cmake libgtk-3-dev libwebkit2gtk-4.1-dev libxdo-dev pkg-config ``` The resulting application dynamically links the distro GTK/WebKit stack. Do not put the entire GTK/WebKit dependency closure into the application archive. Bundle application-owned shared libraries, dynamically linked Rust standard libraries, and explicitly redistributable runtimes; document the required Linux runtime packages for users. Build on the oldest distro baseline that the release promises to support. Using a rolling `ubuntu-latest` silently raises the minimum glibc and WebKitGTK versions when the runner image changes. A fixed Ubuntu release or container is more reproducible. Add an `$ORIGIN` runtime search path so executables find shared libraries placed beside them: ```text -C link-arg=-Wl,-rpath,$ORIGIN ``` ### Cross-linking Linux x64 on ARM64 Rust cross-compilation is the easy part. Native dependencies selected through `pkg-config`, C/C++ build scripts, the linker, and the x64 libc are the parts that require deliberate configuration. On an Ubuntu ARM64 host: 1. Restrict the existing Ubuntu deb822 sources to `arm64`. 2. Add `amd64` as a foreign architecture and add matching amd64 archive and security sources for the same Ubuntu codename. 3. Install native host tools such as `gcc-x86-64-linux-gnu`, `g++-x86-64-linux-gnu`, `cmake`, and `pkg-config`. 4. Download the GTK, WebKitGTK, and xdo amd64 development packages and their dependencies with APT's `--download-only` mode. 5. Extract the `.deb` files with `dpkg-deb --extract` into an isolated sysroot, such as `/opt/ruds-x64-sysroot`. 6. Point Cargo, the GNU cross-linker, and `pkg-config` exclusively at that sysroot. Do not normally install the foreign development packages on the ARM64 host. Some packages contain target-architecture helper executables or maintainer scripts; package installation can attempt to execute x64 programs on ARM64. Downloading and extracting produces the headers, libraries, and metadata needed for linking without running target code. The essential environment is: ```text AR_x86_64_unknown_linux_gnu=x86_64-linux-gnu-ar CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc PKG_CONFIG_ALLOW_CROSS=1 PKG_CONFIG_SYSROOT_DIR=/opt/ruds-x64-sysroot PKG_CONFIG_LIBDIR=/opt/ruds-x64-sysroot/usr/lib/x86_64-linux-gnu/pkgconfig:/opt/ruds-x64-sysroot/usr/lib/pkgconfig:/opt/ruds-x64-sysroot/usr/share/pkgconfig ``` Pass `--sysroot` to the linker, add the target library directories with Cargo `-L native=...`, and retain the `$ORIGIN` rpath. Do not leave host pkg-config paths in the search list: that can produce an ARM64/x64 mixture which may fail late or, worse, appear to link successfully. Ubuntu libc development linker scripts may refer to absolute legacy paths such as `/lib/x86_64-linux-gnu`. Those paths are interpreted inside the sysroot but may not exist in an extracted modern Ubuntu package layout. Materialize the required libc, libm, libmvec, and loader paths within the sysroot. Do not create host-level x64 symlinks. Architecture-independent `.pc` files may already be installed on the host, so APT may not download them again for amd64. They contain metadata rather than target executables and can be copied into the sysroot; `PKG_CONFIG_SYSROOT_DIR` then redirects their paths. Some native Rust dependencies build static objects that are later linked into application shared libraries. They need position-independent code. Compiler wrappers should append `-fPIC` as the final flag so it overrides earlier flags. Target-specific Cargo environment names can contain hyphens and therefore are not valid shell assignment identifiers. Set them through `env`, for example: ```sh env \ "CC_x86_64-unknown-linux-gnu=$PWD/.gitea/scripts/x86_64-linux-gnu-gcc-pic" \ "CXX_x86_64-unknown-linux-gnu=$PWD/.gitea/scripts/x86_64-linux-gnu-g++-pic" \ cargo build --release --locked --target x86_64-unknown-linux-gnu ``` Audit every `-sys` crate for pkg-config overrides. For RuDS, `ZSTD_SYS_USE_PKG_CONFIG` must be unset in the x64 build so zstd does not select an incompatible host library. ## macOS without a macOS runner Use `cargo-zigbuild` in a digest-pinned ARM64-compatible container to compile the `aarch64-apple-darwin` and `x86_64-apple-darwin` targets. The container must contain a legally usable Apple SDK. Review the SDK and redistribution licenses; portable tooling does not remove those obligations. RuDS supplies these linker requirements: - Both `@executable_path` and `@executable_path/../Resources` rpaths. The first serves the flat CLI archive and the second serves the `.app` layout. - Header padding for later Mach-O load-command rewriting. - The chosen minimum macOS deployment version. - `-Wl,-undefined,dynamic_lookup` where Apple symbols are intentionally resolved at runtime. Zig's Darwin linker does not implement every `ld64` option. In particular, symbol-specific `-U` handling was not usable here. Broad `-undefined,dynamic_lookup` works when late-bound framework symbols are truly required, but it can also mask accidental unresolved symbols; use it only with dependency inspection and runtime testing. The Zig Apple link path expected a compiler runtime archive alias. A valid empty `libclang_rt.osx.a` archive (`!\n`) in an explicit native library directory satisfied that lookup. ### Intel macOS and version-sensitive native runtimes Do not assume that the latest machine-learning runtime publishes every Apple architecture. The ONNX Runtime release used by newer Rust crates no longer had an Intel macOS binary. RuDS keeps a compatible dependency line: ```text fastembed 5.12.0 ort/ort-sys 2.0.0-rc.11 ONNX Runtime 1.23.2 for macOS x64 ``` The Rust release helper downloads Microsoft's x64 archive, verifies its pinned SHA-256 digest, extracts the versioned dylib and license, and creates an unversioned linker copy. With `ort-sys` rc.11, the working build variables are: ```text ORT_LIB_LOCATION= ORT_PREFER_DYNAMIC_LINK=1 ``` `ORT_LIB_PATH`, documented or accepted by other versions, is ignored by this specific prerelease. When a native dependency behaves unexpectedly, read the build script for the exact locked crate version rather than documentation for a newer release. ### Pure-Rust Apple packaging Build the application bundle and DMG with Rust crates rather than `hdiutil`, `codesign`, shell archive tricks, or a simulated macOS environment. RuDS uses Rust libraries to: 1. Create `Contents/MacOS`, `Contents/Resources`, `Info.plist`, the icon, bundle metadata, and URL handlers. 2. Put executables in `Contents/MacOS` and dylibs/licenses in Resources. 3. Parse 64-bit little-endian Mach-O load commands and rewrite application dylib IDs and references to `@rpath/`. 4. Sign after all binary mutations. 5. Construct and sign the DMG. Mach-O load commands have fixed allocated space. Reserve header padding at link time or a longer rewritten install name will not fit. Rewrite every applicable load command, including weak, re-exported, lazy, and upward dylib loads, and never attempt to parse licenses or other resource files as Mach-O binaries. The current output is ad-hoc signed. Developer ID signing and Apple notarization need Apple-issued credentials and a supported submission path; a Linux-only pipeline cannot claim notarized output merely because it can construct and ad-hoc-sign a valid bundle. ## Windows x64 from Linux ARM64 Use `cargo-xwin` in a digest-pinned ARM64-compatible container. It provides the Windows MSVC SDK/import-library environment without requiring a Windows host. Install the Rust `x86_64-pc-windows-msvc` target and NASM when native assembly dependencies require it, then populate the xwin cache before looking up SDK libraries. RuDS's DirectML-enabled build needs: - Microsoft DirectML 1.15.4 downloaded from NuGet and verified by SHA-256. - `DirectML.lib` for linking, plus `DirectML.dll` and its license in the final package. - A correctly cased `PathCch.lib` alias copied from the xwin SDK, because some build tooling expects that spelling. - The target release directory on Cargo's native library search path. Do not force `target-feature=+crt-static` merely to make the archive look more self-contained. Windows system and SDK dependencies may require the dynamic CRT, and forcing static CRT can produce incompatible link combinations. Bundle the redistributable application runtime DLLs that are actually required and retain their licenses. RuDS statically links ONNX Runtime on Windows but ships DirectML dynamically. Portable ZIPs are the simplest cross-host Windows result. MSI, MSIX, or signed installer production introduces additional tooling, signing certificates, and Windows policy requirements and should be treated as separate scope. ## Packaging rules Keep packaging in a small host-native Rust binary. `bds-release` runs on the ARM64 builder even while it packages foreign target files. It uses Rust for HTTP, hashing, tar/gzip, ZIP, plist, ICNS, Apple bundle/DMG construction, Mach-O mutation, signing, and Gitea API calls. For each target: - Require every expected executable and application shared library before creating an archive. - Copy dynamic Rust standard libraries from `rustc --target --print target-libdir` when the project links Rust's standard library dynamically. - Include application-owned shared libraries and required redistributable third-party runtimes with license files. - Give archives deterministic, tag-derived names and a single top-level directory. - Reject unsafe tags used as filenames and refuse to overwrite existing output. - Stream large uploads with a content length rather than loading them into RAM. - Retry network failures, HTTP 429, and server errors with bounded backoff. - Verify every downloaded native dependency with a pinned cryptographic digest. Packaging must follow the target's runtime model. Linux should use the distro's GTK/WebKit libraries; macOS `.app` bundles should carry their private dylibs in Resources with corrected install names; Windows packages should carry approved redistributable DLLs. “Put every shared library beside the executable” is not a portable policy. ## Release creation and rebuilding The preparation job uses full Git history to locate the previous tag, reads issue numbers from commit messages, fetches closed issues through Gitea's API, and creates categorized release notes. It writes the draft release ID as a step output so every build uploads to the same release. Rebuilding an existing published tag is deliberately explicit. In RuDS, `REBUILD_PUBLISHED_RELEASE=true` changes the release back to a draft, deletes its assets, regenerates its metadata, and uploads a consistent new set. This is destructive and temporarily removes the public release. Prefer a new immutable tag unless repairing an existing release is intentional. Rerunning a failed draft also clears its partial asset set before rebuilding, which prevents stale and new files from being mixed. Before pushing a release tag: 1. Update the workspace version, lockfile, README, and any visible version metadata together. 2. Run formatting, the full workspace tests, and relevant release-helper tests. 3. Build locally for the host where practical. 4. Commit and push the release commit to the default branch. 5. Create an annotated tag at that exact commit, then push the tag. After the workflow finishes, verify through the Gitea API or `tea`: - Every job is successful rather than merely the workflow appearing complete. - The tag and its peeled commit match the intended default-branch commit. - The release is public (`draft: false`) and has the expected prerelease flag. - The release targets the exact tagged commit. - The expected asset names are present exactly once and have non-zero sizes. - The repository worktree is clean and local/remote refs agree. Useful Gitea API endpoints are: ```text GET /repos/{owner}/{repo}/actions/runs GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs GET /repos/{owner}/{repo}/actions/jobs/{job_id}/logs GET /repos/{owner}/{repo}/releases/tags/{tag} ``` Job logs can be very large. Download or query them once, then filter locally for the failing step and nearby compiler/linker diagnostics instead of repeatedly printing the entire log. ## Common failures | Symptom | Cause | Fix | | --- | --- | --- | | `Exec format error` during x64 dependency installation | A foreign package's executable or maintainer script ran on ARM64 | Download packages only and extract them into a sysroot | | Linker reports wrong ELF architecture | Host libraries leaked through pkg-config or a build script | Restrict `PKG_CONFIG_LIBDIR`, set its sysroot, and audit `-sys` crate overrides | | Relocation errors while creating a `.so` | A C/C++ static object was built without PIC | Append `-fPIC` in target compiler wrappers | | libc or loader missing despite downloaded packages | Ubuntu linker scripts contain absolute legacy paths | Materialize those target paths inside the sysroot | | Linux binary starts but cannot find project `.so` files | No relative runtime search path | Link with an `$ORIGIN` rpath and package the libraries beside the binary | | macOS link rejects `-U` | Zig's Darwin linker differs from Apple `ld64` | Use supported options; use dynamic lookup only when justified | | Mach-O install-name rewrite does not fit | No load-command header padding | Add header padding before linking, then mutate before signing | | Intel macOS ORT download fails | The selected upstream release dropped x64 macOS | Pin a compatible crate/runtime pair and supply a verified dylib | | `ort-sys` ignores a supplied runtime | Environment variable belongs to another crate version | Inspect the exact locked crate build script; rc.11 uses `ORT_LIB_LOCATION` | | Windows library is “missing” despite SDK contents | Import-library case/name differs from build-script expectation | Add an explicit linker alias such as `PathCch.lib` | | Windows CRT link conflicts | Static CRT was forced across incompatible dependencies | Use the target's normal dynamic CRT unless the full graph supports static CRT | | Checkout fails only inside a build container | The image cannot execute the checkout action or cannot reach Gitea | Fetch the exact tag with authenticated Git in the container | | Release is public but incomplete | Publication happened before asset completion | Create a draft first and publish only behind all build jobs and manifest checks | | A release takes hours | Cold Rust/native builds on one physical ARM64 runner serialize | Add real runners, carefully cache immutable inputs, or prebake dependencies | ## Performance and reproducibility A cold five-target build with GTK/WebKit, ONNX Runtime, and ML dependencies can take about three hours on a single ARM64 runner. The workflow itself is simple; compilation and native dependency setup dominate the time. Safe ways to reduce it include: - Add separate runners with the same label, or platform-specific labels, so independent target jobs truly run concurrently. - Prebuild a versioned runner/container image containing system dependencies and the Linux x64 sysroot. - Cache Cargo registry, Git checkouts, target artifacts, Rust toolchains, and cross-SDK caches with keys including the lockfile, Rust version, target, feature set, distro version, and relevant build scripts. - Avoid concurrent memory-heavy jobs on one physical host even if several logical runners are registered there. Prefer a slow clean build over a fast cache that can mix architectures or stale native libraries. Keep caches disabled until their keys and invalidation rules are demonstrably correct. ## Validation depth Cross-built programs cannot be fully exercised on the ARM64 host, but artifacts should still receive structural checks: - ELF: inspect architecture, `DT_NEEDED`, rpath, and unexpected build-host paths with `file` and `readelf`. - Mach-O: inspect architectures, load commands, bundle layout, plist, signatures, and DMG contents with parsers that run on Linux. - PE: inspect machine type and imports with LLVM or MinGW object tools. - Archives: extract into temporary directories, compare the manifest, and check executable modes and licenses. Then smoke-test on real target systems. QEMU or Wine can catch basic loader failures but is not authoritative for GTK/WebKit behavior, macOS signing and Gatekeeper, or Windows GPU/DirectML behavior. ## References - [Gitea Actions](https://docs.gitea.com/usage/actions) - [Gitea Actions design and runner labels](https://docs.gitea.com/usage/actions/design) - [Gitea Actions variables](https://docs.gitea.com/usage/actions/actions-variables) - [Gitea API](https://docs.gitea.com/api/1.24/) - [cargo-zigbuild](https://github.com/rust-cross/cargo-zigbuild) - [cargo-xwin](https://github.com/rust-cross/cargo-xwin) Treat the checked-in workflow and Rust helper as the executable source of truth. This document explains why their unusual-looking pieces exist so that future agents do not remove them as apparent complexity and rediscover the same target, linker, packaging, and release-state failures.