RuDS/RELEASEBUILD.md

# Portable Gitea Release Builds from a Linux ARM64 Runner

This document is a blueprint for producing native Rust application releases for
Linux ARM64, Linux x64, macOS ARM64, macOS x64, and Windows x64 from a Gitea
Actions runner whose host is Ubuntu ARM64. It records the design and the failure
modes proven while preparing the RuDS 0.9.0 release.

The important distinction is that the release tooling is portable Rust, not
that every application dependency is pure Rust. GTK and WebKitGTK are normal
Linux system libraries and should be linked from the runner's Ubuntu packages.
Apple bundles need portable construction because no macOS runner is available.

The implementation in this repository is split between:

- `.gitea/workflows/release.yml`: orchestration and target-specific build setup.
- `.gitea/scripts/install-linux-build-dependencies.sh`: native ARM64 packages
  and the extracted x64 Ubuntu sysroot.
- `.gitea/scripts/x86_64-linux-gnu-gcc-pic` and
  `.gitea/scripts/x86_64-linux-gnu-g++-pic`: cross-compiler wrappers.
- `crates/bds-release`: release notes, Gitea API access, dependency downloads,
  archive creation, Apple bundle construction, signing, and upload.

## Resulting artifact matrix

Build CLI and desktop artifacts separately so users do not have to download a
GUI application to obtain command-line tools. RuDS creates two assets per target:

| Target | CLI | Desktop |
| --- | --- | --- |
| macOS ARM64 | `.tar.gz` | `.dmg` containing an `.app` |
| macOS x64 | `.tar.gz` | `.dmg` containing an `.app` |
| Linux ARM64 | `.tar.gz` | `.tar.gz` |
| Linux x64 | `.tar.gz` | `.tar.gz` |
| Windows x64 | `.zip` | portable `.zip` |

Compile all final programs for a target in one Cargo invocation. In this
repository those are `bds-ui`, `bds-cli`, and `bds-mcp`. This keeps their feature
selection and shared runtime libraries consistent.

## Pipeline shape

Use a tag-triggered, draft-first pipeline:

1. Check out full history, test the release helper, generate notes, and create
   or update a draft Gitea release.
2. Build and upload every platform into that release ID.
3. Publish only after every build job succeeds.
4. Query Gitea afterward and verify the public release, target commit, and asset
   manifest.

A minimal dependency graph is:

```text
prepare draft
    |
    +-- Linux x64
            |
            +-- Linux ARM64
            +-- macOS ARM64 + x64
            +-- Windows x64
                    |
                    +-- publish
```

RuDS deliberately runs the most failure-prone cross-link first. On a single
runner all jobs serialize anyway, and failing early avoids spending hours on
other targets first. With several independent runners, make all platform jobs
depend only on preparation and let them run in parallel.

The draft is the transaction boundary. Never make a release public before all
assets exist. The final publisher should ideally fetch the draft and compare
the exact expected filenames and non-zero sizes before setting `draft: false`.
RuDS currently relies on job dependencies plus strict per-package input checks;
adding an explicit final manifest check would make that boundary stronger.

## Gitea runner and permissions

Register a label that describes the real machine, for example
`linux-arm64:host`, and use `runs-on: linux-arm64`. Do not treat a name such as
`ubuntu-latest` as an implicit x64 machine. Gitea runner labels have the form
`label[:schema[:args]]`.

The runner needs:

- Enough disk for Cargo registries, several target trees, cross SDKs, sysroots,
  and container layers.
- Outbound HTTPS access to Rust distribution servers, crates.io, Gitea, NuGet,
  GitHub release assets, and the configured container registry.
- Passwordless access to install Ubuntu build packages, or a prebuilt runner
  image containing them.
- Container support for the macOS and Windows build images.
- A URL by which build containers can reach Gitea. `localhost` inside a
  container is the container itself, not the runner host.

Container images must publish an ARM64 image even when the target they build is
x64. Pin images by digest, not a floating tag. Also pin the Rust toolchain and
always build with `--locked`.

Grant the workflow only the permissions it needs: repository contents read,
issues read if release notes inspect issues, and releases write. Gitea clamps
workflow permissions to repository or owner limits. If the built-in
`secrets.GITEA_TOKEN` cannot write releases on the deployed Gitea version, use a
repository secret containing a narrowly scoped token.

Gitea Actions resembles GitHub Actions but is not identical. Validate contexts,
permissions, output files, and action compatibility against the deployed Gitea
and `act_runner` versions. Non-fully-qualified actions may be downloaded from
GitHub depending on `DEFAULT_ACTIONS_URL`; mirror or fully qualify them on
restricted installations.

Pass the draft release ID through a job output. Gitea supports the familiar
`key=value` step-output file convention; this repository writes to the
`GITHUB_OUTPUT` compatibility variable and maps that step output to a job
output. Do not scrape an ID from human-readable command output.

Specialized build containers may not run JavaScript-based checkout actions
reliably. RuDS performs an authenticated, depth-one `git fetch` of the exact tag
inside those containers. Fetch the tag ref explicitly and check out
`FETCH_HEAD`; do not build a mutable branch tip.

## Linux desktop builds

There is no architectural reason to exclude the GTK/WebKit desktop application
from Linux artifacts. The native ARM64 build is ordinary compilation on the
ARM64 Ubuntu runner. Install:

```text
build-essential
cmake
libgtk-3-dev
libwebkit2gtk-4.1-dev
libxdo-dev
pkg-config
```

The resulting application dynamically links the distro GTK/WebKit stack. Do
not put the entire GTK/WebKit dependency closure into the application archive.
Bundle application-owned shared libraries, dynamically linked Rust standard
libraries, and explicitly redistributable runtimes; document the required
Linux runtime packages for users.

Build on the oldest distro baseline that the release promises to support.
Using a rolling `ubuntu-latest` silently raises the minimum glibc and WebKitGTK
versions when the runner image changes. A fixed Ubuntu release or container is
more reproducible.

Add an `$ORIGIN` runtime search path so executables find shared libraries placed
beside them:

```text
-C link-arg=-Wl,-rpath,$ORIGIN
```

### Cross-linking Linux x64 on ARM64

Rust cross-compilation is the easy part. Native dependencies selected through
`pkg-config`, C/C++ build scripts, the linker, and the x64 libc are the parts
that require deliberate configuration.

On an Ubuntu ARM64 host:

1. Restrict the existing Ubuntu deb822 sources to `arm64`.
2. Add `amd64` as a foreign architecture and add matching amd64 archive and
   security sources for the same Ubuntu codename.
3. Install native host tools such as `gcc-x86-64-linux-gnu`,
   `g++-x86-64-linux-gnu`, `cmake`, and `pkg-config`.
4. Download the GTK, WebKitGTK, and xdo amd64 development packages and their
   dependencies with APT's `--download-only` mode.
5. Extract the `.deb` files with `dpkg-deb --extract` into an isolated sysroot,
   such as `/opt/ruds-x64-sysroot`.
6. Point Cargo, the GNU cross-linker, and `pkg-config` exclusively at that
   sysroot.

Do not normally install the foreign development packages on the ARM64 host.
Some packages contain target-architecture helper executables or maintainer
scripts; package installation can attempt to execute x64 programs on ARM64.
Downloading and extracting produces the headers, libraries, and metadata needed
for linking without running target code.

The essential environment is:

```text
AR_x86_64_unknown_linux_gnu=x86_64-linux-gnu-ar
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc
PKG_CONFIG_ALLOW_CROSS=1
PKG_CONFIG_SYSROOT_DIR=/opt/ruds-x64-sysroot
PKG_CONFIG_LIBDIR=/opt/ruds-x64-sysroot/usr/lib/x86_64-linux-gnu/pkgconfig:/opt/ruds-x64-sysroot/usr/lib/pkgconfig:/opt/ruds-x64-sysroot/usr/share/pkgconfig
```

Pass `--sysroot` to the linker, add the target library directories with Cargo
`-L native=...`, and retain the `$ORIGIN` rpath. Do not leave host pkg-config
paths in the search list: that can produce an ARM64/x64 mixture which may fail
late or, worse, appear to link successfully.

Ubuntu libc development linker scripts may refer to absolute legacy paths such
as `/lib/x86_64-linux-gnu`. Those paths are interpreted inside the sysroot but
may not exist in an extracted modern Ubuntu package layout. Materialize the
required libc, libm, libmvec, and loader paths within the sysroot. Do not create
host-level x64 symlinks.

Architecture-independent `.pc` files may already be installed on the host, so
APT may not download them again for amd64. They contain metadata rather than
target executables and can be copied into the sysroot; `PKG_CONFIG_SYSROOT_DIR`
then redirects their paths.

Some native Rust dependencies build static objects that are later linked into
application shared libraries. They need position-independent code. Compiler
wrappers should append `-fPIC` as the final flag so it overrides earlier flags.
Target-specific Cargo environment names can contain hyphens and therefore are
not valid shell assignment identifiers. Set them through `env`, for example:

```sh
env \
  "CC_x86_64-unknown-linux-gnu=$PWD/.gitea/scripts/x86_64-linux-gnu-gcc-pic" \
  "CXX_x86_64-unknown-linux-gnu=$PWD/.gitea/scripts/x86_64-linux-gnu-g++-pic" \
  cargo build --release --locked --target x86_64-unknown-linux-gnu
```

Audit every `-sys` crate for pkg-config overrides. For RuDS,
`ZSTD_SYS_USE_PKG_CONFIG` must be unset in the x64 build so zstd does not select
an incompatible host library.

## macOS without a macOS runner

Use `cargo-zigbuild` in a digest-pinned ARM64-compatible container to compile
the `aarch64-apple-darwin` and `x86_64-apple-darwin` targets. The container must
contain a legally usable Apple SDK. Review the SDK and redistribution licenses;
portable tooling does not remove those obligations.

RuDS supplies these linker requirements:

- Both `@executable_path` and `@executable_path/../Resources` rpaths. The first
  serves the flat CLI archive and the second serves the `.app` layout.
- Header padding for later Mach-O load-command rewriting.
- The chosen minimum macOS deployment version.
- `-Wl,-undefined,dynamic_lookup` where Apple symbols are intentionally resolved
  at runtime.

Zig's Darwin linker does not implement every `ld64` option. In particular,
symbol-specific `-U` handling was not usable here. Broad
`-undefined,dynamic_lookup` works when late-bound framework symbols are truly
required, but it can also mask accidental unresolved symbols; use it only with
dependency inspection and runtime testing.

The Zig Apple link path expected a compiler runtime archive alias. A valid empty
`libclang_rt.osx.a` archive (`!<arch>\n`) in an explicit native library directory
satisfied that lookup.

### Intel macOS and version-sensitive native runtimes

Do not assume that the latest machine-learning runtime publishes every Apple
architecture. The ONNX Runtime release used by newer Rust crates no longer had
an Intel macOS binary. RuDS keeps a compatible dependency line:

```text
fastembed 5.12.0
ort/ort-sys 2.0.0-rc.11
ONNX Runtime 1.23.2 for macOS x64
```

The Rust release helper downloads Microsoft's x64 archive, verifies its pinned
SHA-256 digest, extracts the versioned dylib and license, and creates an
unversioned linker copy. With `ort-sys` rc.11, the working build variables are:

```text
ORT_LIB_LOCATION=<directory-containing-libonnxruntime.dylib>
ORT_PREFER_DYNAMIC_LINK=1
```

`ORT_LIB_PATH`, documented or accepted by other versions, is ignored by this
specific prerelease. When a native dependency behaves unexpectedly, read the
build script for the exact locked crate version rather than documentation for a
newer release.

### Pure-Rust Apple packaging

Build the application bundle and DMG with Rust crates rather than `hdiutil`,
`codesign`, shell archive tricks, or a simulated macOS environment. RuDS uses
Rust libraries to:

1. Create `Contents/MacOS`, `Contents/Resources`, `Info.plist`, the icon, bundle
   metadata, and URL handlers.
2. Put executables in `Contents/MacOS` and dylibs/licenses in Resources.
3. Parse 64-bit little-endian Mach-O load commands and rewrite application
   dylib IDs and references to `@rpath/<basename>`.
4. Sign after all binary mutations.
5. Construct and sign the DMG.

Mach-O load commands have fixed allocated space. Reserve header padding at link
time or a longer rewritten install name will not fit. Rewrite every applicable
load command, including weak, re-exported, lazy, and upward dylib loads, and
never attempt to parse licenses or other resource files as Mach-O binaries.

The current output is ad-hoc signed. Developer ID signing and Apple notarization
need Apple-issued credentials and a supported submission path; a Linux-only
pipeline cannot claim notarized output merely because it can construct and
ad-hoc-sign a valid bundle.

## Windows x64 from Linux ARM64

Use `cargo-xwin` in a digest-pinned ARM64-compatible container. It provides the
Windows MSVC SDK/import-library environment without requiring a Windows host.
Install the Rust `x86_64-pc-windows-msvc` target and NASM when native assembly
dependencies require it, then populate the xwin cache before looking up SDK
libraries.

RuDS's DirectML-enabled build needs:

- Microsoft DirectML 1.15.4 downloaded from NuGet and verified by SHA-256.
- `DirectML.lib` for linking, plus `DirectML.dll` and its license in the final
  package.
- A correctly cased `PathCch.lib` alias copied from the xwin SDK, because some
  build tooling expects that spelling.
- The target release directory on Cargo's native library search path.

Do not force `target-feature=+crt-static` merely to make the archive look more
self-contained. Windows system and SDK dependencies may require the dynamic
CRT, and forcing static CRT can produce incompatible link combinations. Bundle
the redistributable application runtime DLLs that are actually required and
retain their licenses. RuDS statically links ONNX Runtime on Windows but ships
DirectML dynamically.

Portable ZIPs are the simplest cross-host Windows result. MSI, MSIX, or signed
installer production introduces additional tooling, signing certificates, and
Windows policy requirements and should be treated as separate scope.

## Packaging rules

Keep packaging in a small host-native Rust binary. `bds-release` runs on the
ARM64 builder even while it packages foreign target files. It uses Rust for
HTTP, hashing, tar/gzip, ZIP, plist, ICNS, Apple bundle/DMG construction, Mach-O
mutation, signing, and Gitea API calls.

For each target:

- Require every expected executable and application shared library before
  creating an archive.
- Copy dynamic Rust standard libraries from
  `rustc --target <target> --print target-libdir` when the project links Rust's
  standard library dynamically.
- Include application-owned shared libraries and required redistributable
  third-party runtimes with license files.
- Give archives deterministic, tag-derived names and a single top-level
  directory.
- Reject unsafe tags used as filenames and refuse to overwrite existing output.
- Stream large uploads with a content length rather than loading them into RAM.
- Retry network failures, HTTP 429, and server errors with bounded backoff.
- Verify every downloaded native dependency with a pinned cryptographic digest.

Packaging must follow the target's runtime model. Linux should use the distro's
GTK/WebKit libraries; macOS `.app` bundles should carry their private dylibs in
Resources with corrected install names; Windows packages should carry approved
redistributable DLLs. “Put every shared library beside the executable” is not a
portable policy.

## Release creation and rebuilding

The preparation job uses full Git history to locate the previous tag, reads
issue numbers from commit messages, fetches closed issues through Gitea's API,
and creates categorized release notes. It writes the draft release ID as a step
output so every build uploads to the same release.

Rebuilding an existing published tag is deliberately explicit. In RuDS,
`REBUILD_PUBLISHED_RELEASE=true` changes the release back to a draft, deletes
its assets, regenerates its metadata, and uploads a consistent new set. This is
destructive and temporarily removes the public release. Prefer a new immutable
tag unless repairing an existing release is intentional. Rerunning a failed
draft also clears its partial asset set before rebuilding, which prevents stale
and new files from being mixed.

Before pushing a release tag:

1. Update the workspace version, lockfile, README, and any visible version
   metadata together.
2. Run formatting, the full workspace tests, and relevant release-helper tests.
3. Build locally for the host where practical.
4. Commit and push the release commit to the default branch.
5. Create an annotated tag at that exact commit, then push the tag.

After the workflow finishes, verify through the Gitea API or `tea`:

- Every job is successful rather than merely the workflow appearing complete.
- The tag and its peeled commit match the intended default-branch commit.
- The release is public (`draft: false`) and has the expected prerelease flag.
- The release targets the exact tagged commit.
- The expected asset names are present exactly once and have non-zero sizes.
- The repository worktree is clean and local/remote refs agree.

Useful Gitea API endpoints are:

```text
GET /repos/{owner}/{repo}/actions/runs
GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs
GET /repos/{owner}/{repo}/actions/jobs/{job_id}/logs
GET /repos/{owner}/{repo}/releases/tags/{tag}
```

Job logs can be very large. Download or query them once, then filter locally for
the failing step and nearby compiler/linker diagnostics instead of repeatedly
printing the entire log.

## Common failures

| Symptom | Cause | Fix |
| --- | --- | --- |
| `Exec format error` during x64 dependency installation | A foreign package's executable or maintainer script ran on ARM64 | Download packages only and extract them into a sysroot |
| Linker reports wrong ELF architecture | Host libraries leaked through pkg-config or a build script | Restrict `PKG_CONFIG_LIBDIR`, set its sysroot, and audit `-sys` crate overrides |
| Relocation errors while creating a `.so` | A C/C++ static object was built without PIC | Append `-fPIC` in target compiler wrappers |
| libc or loader missing despite downloaded packages | Ubuntu linker scripts contain absolute legacy paths | Materialize those target paths inside the sysroot |
| Linux binary starts but cannot find project `.so` files | No relative runtime search path | Link with an `$ORIGIN` rpath and package the libraries beside the binary |
| macOS link rejects `-U` | Zig's Darwin linker differs from Apple `ld64` | Use supported options; use dynamic lookup only when justified |
| Mach-O install-name rewrite does not fit | No load-command header padding | Add header padding before linking, then mutate before signing |
| Intel macOS ORT download fails | The selected upstream release dropped x64 macOS | Pin a compatible crate/runtime pair and supply a verified dylib |
| `ort-sys` ignores a supplied runtime | Environment variable belongs to another crate version | Inspect the exact locked crate build script; rc.11 uses `ORT_LIB_LOCATION` |
| Windows library is “missing” despite SDK contents | Import-library case/name differs from build-script expectation | Add an explicit linker alias such as `PathCch.lib` |
| Windows CRT link conflicts | Static CRT was forced across incompatible dependencies | Use the target's normal dynamic CRT unless the full graph supports static CRT |
| Checkout fails only inside a build container | The image cannot execute the checkout action or cannot reach Gitea | Fetch the exact tag with authenticated Git in the container |
| Release is public but incomplete | Publication happened before asset completion | Create a draft first and publish only behind all build jobs and manifest checks |
| A release takes hours | Cold Rust/native builds on one physical ARM64 runner serialize | Add real runners, carefully cache immutable inputs, or prebake dependencies |

## Performance and reproducibility

A cold five-target build with GTK/WebKit, ONNX Runtime, and ML dependencies can
take about three hours on a single ARM64 runner. The workflow itself is simple;
compilation and native dependency setup dominate the time.

Safe ways to reduce it include:

- Add separate runners with the same label, or platform-specific labels, so
  independent target jobs truly run concurrently.
- Prebuild a versioned runner/container image containing system dependencies and
  the Linux x64 sysroot.
- Cache Cargo registry, Git checkouts, target artifacts, Rust toolchains, and
  cross-SDK caches with keys including the lockfile, Rust version, target,
  feature set, distro version, and relevant build scripts.
- Avoid concurrent memory-heavy jobs on one physical host even if several
  logical runners are registered there.

Prefer a slow clean build over a fast cache that can mix architectures or stale
native libraries. Keep caches disabled until their keys and invalidation rules
are demonstrably correct.

## Validation depth

Cross-built programs cannot be fully exercised on the ARM64 host, but artifacts
should still receive structural checks:

- ELF: inspect architecture, `DT_NEEDED`, rpath, and unexpected build-host paths
  with `file` and `readelf`.
- Mach-O: inspect architectures, load commands, bundle layout, plist, signatures,
  and DMG contents with parsers that run on Linux.
- PE: inspect machine type and imports with LLVM or MinGW object tools.
- Archives: extract into temporary directories, compare the manifest, and check
  executable modes and licenses.

Then smoke-test on real target systems. QEMU or Wine can catch basic loader
failures but is not authoritative for GTK/WebKit behavior, macOS signing and
Gatekeeper, or Windows GPU/DirectML behavior.

## References

- [Gitea Actions](https://docs.gitea.com/usage/actions)
- [Gitea Actions design and runner labels](https://docs.gitea.com/usage/actions/design)
- [Gitea Actions variables](https://docs.gitea.com/usage/actions/actions-variables)
- [Gitea API](https://docs.gitea.com/api/1.24/)
- [cargo-zigbuild](https://github.com/rust-cross/cargo-zigbuild)
- [cargo-xwin](https://github.com/rust-cross/cargo-xwin)

Treat the checked-in workflow and Rust helper as the executable source of truth.
This document explains why their unusual-looking pieces exist so that future
agents do not remove them as apparent complexity and rediscover the same target,
linker, packaging, and release-state failures.