From 1b43014abdeabf2000935a1de2654579d862acb2 Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Sat, 15 Aug 2026 09:21:59 +0000 Subject: [PATCH] doc: documented problems in released build --- RELEASEBUILD.md | 472 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 472 insertions(+) create mode 100644 RELEASEBUILD.md diff --git a/RELEASEBUILD.md b/RELEASEBUILD.md new file mode 100644 index 0000000..51c7a9f --- /dev/null +++ b/RELEASEBUILD.md @@ -0,0 +1,472 @@ +# Portable Gitea Release Builds from a Linux ARM64 Runner + +This document is a blueprint for producing native Rust application releases for +Linux ARM64, Linux x64, macOS ARM64, macOS x64, and Windows x64 from a Gitea +Actions runner whose host is Ubuntu ARM64. It records the design and the failure +modes proven while preparing the RuDS 0.9.0 release. + +The important distinction is that the release tooling is portable Rust, not +that every application dependency is pure Rust. GTK and WebKitGTK are normal +Linux system libraries and should be linked from the runner's Ubuntu packages. +Apple bundles need portable construction because no macOS runner is available. + +The implementation in this repository is split between: + +- `.gitea/workflows/release.yml`: orchestration and target-specific build setup. +- `.gitea/scripts/install-linux-build-dependencies.sh`: native ARM64 packages + and the extracted x64 Ubuntu sysroot. +- `.gitea/scripts/x86_64-linux-gnu-gcc-pic` and + `.gitea/scripts/x86_64-linux-gnu-g++-pic`: cross-compiler wrappers. +- `crates/bds-release`: release notes, Gitea API access, dependency downloads, + archive creation, Apple bundle construction, signing, and upload. + +## Resulting artifact matrix + +Build CLI and desktop artifacts separately so users do not have to download a +GUI application to obtain command-line tools. RuDS creates two assets per target: + +| Target | CLI | Desktop | +| --- | --- | --- | +| macOS ARM64 | `.tar.gz` | `.dmg` containing an `.app` | +| macOS x64 | `.tar.gz` | `.dmg` containing an `.app` | +| Linux ARM64 | `.tar.gz` | `.tar.gz` | +| Linux x64 | `.tar.gz` | `.tar.gz` | +| Windows x64 | `.zip` | portable `.zip` | + +Compile all final programs for a target in one Cargo invocation. In this +repository those are `bds-ui`, `bds-cli`, and `bds-mcp`. This keeps their feature +selection and shared runtime libraries consistent. + +## Pipeline shape + +Use a tag-triggered, draft-first pipeline: + +1. Check out full history, test the release helper, generate notes, and create + or update a draft Gitea release. +2. Build and upload every platform into that release ID. +3. Publish only after every build job succeeds. +4. Query Gitea afterward and verify the public release, target commit, and asset + manifest. + +A minimal dependency graph is: + +```text +prepare draft + | + +-- Linux x64 + | + +-- Linux ARM64 + +-- macOS ARM64 + x64 + +-- Windows x64 + | + +-- publish +``` + +RuDS deliberately runs the most failure-prone cross-link first. On a single +runner all jobs serialize anyway, and failing early avoids spending hours on +other targets first. With several independent runners, make all platform jobs +depend only on preparation and let them run in parallel. + +The draft is the transaction boundary. Never make a release public before all +assets exist. The final publisher should ideally fetch the draft and compare +the exact expected filenames and non-zero sizes before setting `draft: false`. +RuDS currently relies on job dependencies plus strict per-package input checks; +adding an explicit final manifest check would make that boundary stronger. + +## Gitea runner and permissions + +Register a label that describes the real machine, for example +`linux-arm64:host`, and use `runs-on: linux-arm64`. Do not treat a name such as +`ubuntu-latest` as an implicit x64 machine. Gitea runner labels have the form +`label[:schema[:args]]`. + +The runner needs: + +- Enough disk for Cargo registries, several target trees, cross SDKs, sysroots, + and container layers. +- Outbound HTTPS access to Rust distribution servers, crates.io, Gitea, NuGet, + GitHub release assets, and the configured container registry. +- Passwordless access to install Ubuntu build packages, or a prebuilt runner + image containing them. +- Container support for the macOS and Windows build images. +- A URL by which build containers can reach Gitea. `localhost` inside a + container is the container itself, not the runner host. + +Container images must publish an ARM64 image even when the target they build is +x64. Pin images by digest, not a floating tag. Also pin the Rust toolchain and +always build with `--locked`. + +Grant the workflow only the permissions it needs: repository contents read, +issues read if release notes inspect issues, and releases write. Gitea clamps +workflow permissions to repository or owner limits. If the built-in +`secrets.GITEA_TOKEN` cannot write releases on the deployed Gitea version, use a +repository secret containing a narrowly scoped token. + +Gitea Actions resembles GitHub Actions but is not identical. Validate contexts, +permissions, output files, and action compatibility against the deployed Gitea +and `act_runner` versions. Non-fully-qualified actions may be downloaded from +GitHub depending on `DEFAULT_ACTIONS_URL`; mirror or fully qualify them on +restricted installations. + +Pass the draft release ID through a job output. Gitea supports the familiar +`key=value` step-output file convention; this repository writes to the +`GITHUB_OUTPUT` compatibility variable and maps that step output to a job +output. Do not scrape an ID from human-readable command output. + +Specialized build containers may not run JavaScript-based checkout actions +reliably. RuDS performs an authenticated, depth-one `git fetch` of the exact tag +inside those containers. Fetch the tag ref explicitly and check out +`FETCH_HEAD`; do not build a mutable branch tip. + +## Linux desktop builds + +There is no architectural reason to exclude the GTK/WebKit desktop application +from Linux artifacts. The native ARM64 build is ordinary compilation on the +ARM64 Ubuntu runner. Install: + +```text +build-essential +cmake +libgtk-3-dev +libwebkit2gtk-4.1-dev +libxdo-dev +pkg-config +``` + +The resulting application dynamically links the distro GTK/WebKit stack. Do +not put the entire GTK/WebKit dependency closure into the application archive. +Bundle application-owned shared libraries, dynamically linked Rust standard +libraries, and explicitly redistributable runtimes; document the required +Linux runtime packages for users. + +Build on the oldest distro baseline that the release promises to support. +Using a rolling `ubuntu-latest` silently raises the minimum glibc and WebKitGTK +versions when the runner image changes. A fixed Ubuntu release or container is +more reproducible. + +Add an `$ORIGIN` runtime search path so executables find shared libraries placed +beside them: + +```text +-C link-arg=-Wl,-rpath,$ORIGIN +``` + +### Cross-linking Linux x64 on ARM64 + +Rust cross-compilation is the easy part. Native dependencies selected through +`pkg-config`, C/C++ build scripts, the linker, and the x64 libc are the parts +that require deliberate configuration. + +On an Ubuntu ARM64 host: + +1. Restrict the existing Ubuntu deb822 sources to `arm64`. +2. Add `amd64` as a foreign architecture and add matching amd64 archive and + security sources for the same Ubuntu codename. +3. Install native host tools such as `gcc-x86-64-linux-gnu`, + `g++-x86-64-linux-gnu`, `cmake`, and `pkg-config`. +4. Download the GTK, WebKitGTK, and xdo amd64 development packages and their + dependencies with APT's `--download-only` mode. +5. Extract the `.deb` files with `dpkg-deb --extract` into an isolated sysroot, + such as `/opt/ruds-x64-sysroot`. +6. Point Cargo, the GNU cross-linker, and `pkg-config` exclusively at that + sysroot. + +Do not normally install the foreign development packages on the ARM64 host. +Some packages contain target-architecture helper executables or maintainer +scripts; package installation can attempt to execute x64 programs on ARM64. +Downloading and extracting produces the headers, libraries, and metadata needed +for linking without running target code. + +The essential environment is: + +```text +AR_x86_64_unknown_linux_gnu=x86_64-linux-gnu-ar +CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc +PKG_CONFIG_ALLOW_CROSS=1 +PKG_CONFIG_SYSROOT_DIR=/opt/ruds-x64-sysroot +PKG_CONFIG_LIBDIR=/opt/ruds-x64-sysroot/usr/lib/x86_64-linux-gnu/pkgconfig:/opt/ruds-x64-sysroot/usr/lib/pkgconfig:/opt/ruds-x64-sysroot/usr/share/pkgconfig +``` + +Pass `--sysroot` to the linker, add the target library directories with Cargo +`-L native=...`, and retain the `$ORIGIN` rpath. Do not leave host pkg-config +paths in the search list: that can produce an ARM64/x64 mixture which may fail +late or, worse, appear to link successfully. + +Ubuntu libc development linker scripts may refer to absolute legacy paths such +as `/lib/x86_64-linux-gnu`. Those paths are interpreted inside the sysroot but +may not exist in an extracted modern Ubuntu package layout. Materialize the +required libc, libm, libmvec, and loader paths within the sysroot. Do not create +host-level x64 symlinks. + +Architecture-independent `.pc` files may already be installed on the host, so +APT may not download them again for amd64. They contain metadata rather than +target executables and can be copied into the sysroot; `PKG_CONFIG_SYSROOT_DIR` +then redirects their paths. + +Some native Rust dependencies build static objects that are later linked into +application shared libraries. They need position-independent code. Compiler +wrappers should append `-fPIC` as the final flag so it overrides earlier flags. +Target-specific Cargo environment names can contain hyphens and therefore are +not valid shell assignment identifiers. Set them through `env`, for example: + +```sh +env \ + "CC_x86_64-unknown-linux-gnu=$PWD/.gitea/scripts/x86_64-linux-gnu-gcc-pic" \ + "CXX_x86_64-unknown-linux-gnu=$PWD/.gitea/scripts/x86_64-linux-gnu-g++-pic" \ + cargo build --release --locked --target x86_64-unknown-linux-gnu +``` + +Audit every `-sys` crate for pkg-config overrides. For RuDS, +`ZSTD_SYS_USE_PKG_CONFIG` must be unset in the x64 build so zstd does not select +an incompatible host library. + +## macOS without a macOS runner + +Use `cargo-zigbuild` in a digest-pinned ARM64-compatible container to compile +the `aarch64-apple-darwin` and `x86_64-apple-darwin` targets. The container must +contain a legally usable Apple SDK. Review the SDK and redistribution licenses; +portable tooling does not remove those obligations. + +RuDS supplies these linker requirements: + +- Both `@executable_path` and `@executable_path/../Resources` rpaths. The first + serves the flat CLI archive and the second serves the `.app` layout. +- Header padding for later Mach-O load-command rewriting. +- The chosen minimum macOS deployment version. +- `-Wl,-undefined,dynamic_lookup` where Apple symbols are intentionally resolved + at runtime. + +Zig's Darwin linker does not implement every `ld64` option. In particular, +symbol-specific `-U` handling was not usable here. Broad +`-undefined,dynamic_lookup` works when late-bound framework symbols are truly +required, but it can also mask accidental unresolved symbols; use it only with +dependency inspection and runtime testing. + +The Zig Apple link path expected a compiler runtime archive alias. A valid empty +`libclang_rt.osx.a` archive (`!\n`) in an explicit native library directory +satisfied that lookup. + +### Intel macOS and version-sensitive native runtimes + +Do not assume that the latest machine-learning runtime publishes every Apple +architecture. The ONNX Runtime release used by newer Rust crates no longer had +an Intel macOS binary. RuDS keeps a compatible dependency line: + +```text +fastembed 5.12.0 +ort/ort-sys 2.0.0-rc.11 +ONNX Runtime 1.23.2 for macOS x64 +``` + +The Rust release helper downloads Microsoft's x64 archive, verifies its pinned +SHA-256 digest, extracts the versioned dylib and license, and creates an +unversioned linker copy. With `ort-sys` rc.11, the working build variables are: + +```text +ORT_LIB_LOCATION= +ORT_PREFER_DYNAMIC_LINK=1 +``` + +`ORT_LIB_PATH`, documented or accepted by other versions, is ignored by this +specific prerelease. When a native dependency behaves unexpectedly, read the +build script for the exact locked crate version rather than documentation for a +newer release. + +### Pure-Rust Apple packaging + +Build the application bundle and DMG with Rust crates rather than `hdiutil`, +`codesign`, shell archive tricks, or a simulated macOS environment. RuDS uses +Rust libraries to: + +1. Create `Contents/MacOS`, `Contents/Resources`, `Info.plist`, the icon, bundle + metadata, and URL handlers. +2. Put executables in `Contents/MacOS` and dylibs/licenses in Resources. +3. Parse 64-bit little-endian Mach-O load commands and rewrite application + dylib IDs and references to `@rpath/`. +4. Sign after all binary mutations. +5. Construct and sign the DMG. + +Mach-O load commands have fixed allocated space. Reserve header padding at link +time or a longer rewritten install name will not fit. Rewrite every applicable +load command, including weak, re-exported, lazy, and upward dylib loads, and +never attempt to parse licenses or other resource files as Mach-O binaries. + +The current output is ad-hoc signed. Developer ID signing and Apple notarization +need Apple-issued credentials and a supported submission path; a Linux-only +pipeline cannot claim notarized output merely because it can construct and +ad-hoc-sign a valid bundle. + +## Windows x64 from Linux ARM64 + +Use `cargo-xwin` in a digest-pinned ARM64-compatible container. It provides the +Windows MSVC SDK/import-library environment without requiring a Windows host. +Install the Rust `x86_64-pc-windows-msvc` target and NASM when native assembly +dependencies require it, then populate the xwin cache before looking up SDK +libraries. + +RuDS's DirectML-enabled build needs: + +- Microsoft DirectML 1.15.4 downloaded from NuGet and verified by SHA-256. +- `DirectML.lib` for linking, plus `DirectML.dll` and its license in the final + package. +- A correctly cased `PathCch.lib` alias copied from the xwin SDK, because some + build tooling expects that spelling. +- The target release directory on Cargo's native library search path. + +Do not force `target-feature=+crt-static` merely to make the archive look more +self-contained. Windows system and SDK dependencies may require the dynamic +CRT, and forcing static CRT can produce incompatible link combinations. Bundle +the redistributable application runtime DLLs that are actually required and +retain their licenses. RuDS statically links ONNX Runtime on Windows but ships +DirectML dynamically. + +Portable ZIPs are the simplest cross-host Windows result. MSI, MSIX, or signed +installer production introduces additional tooling, signing certificates, and +Windows policy requirements and should be treated as separate scope. + +## Packaging rules + +Keep packaging in a small host-native Rust binary. `bds-release` runs on the +ARM64 builder even while it packages foreign target files. It uses Rust for +HTTP, hashing, tar/gzip, ZIP, plist, ICNS, Apple bundle/DMG construction, Mach-O +mutation, signing, and Gitea API calls. + +For each target: + +- Require every expected executable and application shared library before + creating an archive. +- Copy dynamic Rust standard libraries from + `rustc --target --print target-libdir` when the project links Rust's + standard library dynamically. +- Include application-owned shared libraries and required redistributable + third-party runtimes with license files. +- Give archives deterministic, tag-derived names and a single top-level + directory. +- Reject unsafe tags used as filenames and refuse to overwrite existing output. +- Stream large uploads with a content length rather than loading them into RAM. +- Retry network failures, HTTP 429, and server errors with bounded backoff. +- Verify every downloaded native dependency with a pinned cryptographic digest. + +Packaging must follow the target's runtime model. Linux should use the distro's +GTK/WebKit libraries; macOS `.app` bundles should carry their private dylibs in +Resources with corrected install names; Windows packages should carry approved +redistributable DLLs. “Put every shared library beside the executable” is not a +portable policy. + +## Release creation and rebuilding + +The preparation job uses full Git history to locate the previous tag, reads +issue numbers from commit messages, fetches closed issues through Gitea's API, +and creates categorized release notes. It writes the draft release ID as a step +output so every build uploads to the same release. + +Rebuilding an existing published tag is deliberately explicit. In RuDS, +`REBUILD_PUBLISHED_RELEASE=true` changes the release back to a draft, deletes +its assets, regenerates its metadata, and uploads a consistent new set. This is +destructive and temporarily removes the public release. Prefer a new immutable +tag unless repairing an existing release is intentional. Rerunning a failed +draft also clears its partial asset set before rebuilding, which prevents stale +and new files from being mixed. + +Before pushing a release tag: + +1. Update the workspace version, lockfile, README, and any visible version + metadata together. +2. Run formatting, the full workspace tests, and relevant release-helper tests. +3. Build locally for the host where practical. +4. Commit and push the release commit to the default branch. +5. Create an annotated tag at that exact commit, then push the tag. + +After the workflow finishes, verify through the Gitea API or `tea`: + +- Every job is successful rather than merely the workflow appearing complete. +- The tag and its peeled commit match the intended default-branch commit. +- The release is public (`draft: false`) and has the expected prerelease flag. +- The release targets the exact tagged commit. +- The expected asset names are present exactly once and have non-zero sizes. +- The repository worktree is clean and local/remote refs agree. + +Useful Gitea API endpoints are: + +```text +GET /repos/{owner}/{repo}/actions/runs +GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs +GET /repos/{owner}/{repo}/actions/jobs/{job_id}/logs +GET /repos/{owner}/{repo}/releases/tags/{tag} +``` + +Job logs can be very large. Download or query them once, then filter locally for +the failing step and nearby compiler/linker diagnostics instead of repeatedly +printing the entire log. + +## Common failures + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `Exec format error` during x64 dependency installation | A foreign package's executable or maintainer script ran on ARM64 | Download packages only and extract them into a sysroot | +| Linker reports wrong ELF architecture | Host libraries leaked through pkg-config or a build script | Restrict `PKG_CONFIG_LIBDIR`, set its sysroot, and audit `-sys` crate overrides | +| Relocation errors while creating a `.so` | A C/C++ static object was built without PIC | Append `-fPIC` in target compiler wrappers | +| libc or loader missing despite downloaded packages | Ubuntu linker scripts contain absolute legacy paths | Materialize those target paths inside the sysroot | +| Linux binary starts but cannot find project `.so` files | No relative runtime search path | Link with an `$ORIGIN` rpath and package the libraries beside the binary | +| macOS link rejects `-U` | Zig's Darwin linker differs from Apple `ld64` | Use supported options; use dynamic lookup only when justified | +| Mach-O install-name rewrite does not fit | No load-command header padding | Add header padding before linking, then mutate before signing | +| Intel macOS ORT download fails | The selected upstream release dropped x64 macOS | Pin a compatible crate/runtime pair and supply a verified dylib | +| `ort-sys` ignores a supplied runtime | Environment variable belongs to another crate version | Inspect the exact locked crate build script; rc.11 uses `ORT_LIB_LOCATION` | +| Windows library is “missing” despite SDK contents | Import-library case/name differs from build-script expectation | Add an explicit linker alias such as `PathCch.lib` | +| Windows CRT link conflicts | Static CRT was forced across incompatible dependencies | Use the target's normal dynamic CRT unless the full graph supports static CRT | +| Checkout fails only inside a build container | The image cannot execute the checkout action or cannot reach Gitea | Fetch the exact tag with authenticated Git in the container | +| Release is public but incomplete | Publication happened before asset completion | Create a draft first and publish only behind all build jobs and manifest checks | +| A release takes hours | Cold Rust/native builds on one physical ARM64 runner serialize | Add real runners, carefully cache immutable inputs, or prebake dependencies | + +## Performance and reproducibility + +A cold five-target build with GTK/WebKit, ONNX Runtime, and ML dependencies can +take about three hours on a single ARM64 runner. The workflow itself is simple; +compilation and native dependency setup dominate the time. + +Safe ways to reduce it include: + +- Add separate runners with the same label, or platform-specific labels, so + independent target jobs truly run concurrently. +- Prebuild a versioned runner/container image containing system dependencies and + the Linux x64 sysroot. +- Cache Cargo registry, Git checkouts, target artifacts, Rust toolchains, and + cross-SDK caches with keys including the lockfile, Rust version, target, + feature set, distro version, and relevant build scripts. +- Avoid concurrent memory-heavy jobs on one physical host even if several + logical runners are registered there. + +Prefer a slow clean build over a fast cache that can mix architectures or stale +native libraries. Keep caches disabled until their keys and invalidation rules +are demonstrably correct. + +## Validation depth + +Cross-built programs cannot be fully exercised on the ARM64 host, but artifacts +should still receive structural checks: + +- ELF: inspect architecture, `DT_NEEDED`, rpath, and unexpected build-host paths + with `file` and `readelf`. +- Mach-O: inspect architectures, load commands, bundle layout, plist, signatures, + and DMG contents with parsers that run on Linux. +- PE: inspect machine type and imports with LLVM or MinGW object tools. +- Archives: extract into temporary directories, compare the manifest, and check + executable modes and licenses. + +Then smoke-test on real target systems. QEMU or Wine can catch basic loader +failures but is not authoritative for GTK/WebKit behavior, macOS signing and +Gatekeeper, or Windows GPU/DirectML behavior. + +## References + +- [Gitea Actions](https://docs.gitea.com/usage/actions) +- [Gitea Actions design and runner labels](https://docs.gitea.com/usage/actions/design) +- [Gitea Actions variables](https://docs.gitea.com/usage/actions/actions-variables) +- [Gitea API](https://docs.gitea.com/api/1.24/) +- [cargo-zigbuild](https://github.com/rust-cross/cargo-zigbuild) +- [cargo-xwin](https://github.com/rust-cross/cargo-xwin) + +Treat the checked-in workflow and Rust helper as the executable source of truth. +This document explains why their unusual-looking pieces exist so that future +agents do not remove them as apparent complexity and rediscover the same target, +linker, packaging, and release-state failures.