doc: documented problems in released build

RELEASEBUILD.md

@@ -0,0 +1,472 @@
+# Portable Gitea Release Builds from a Linux ARM64 Runner
+
+This document is a blueprint for producing native Rust application releases for
+Linux ARM64, Linux x64, macOS ARM64, macOS x64, and Windows x64 from a Gitea
+Actions runner whose host is Ubuntu ARM64. It records the design and the failure
+modes proven while preparing the RuDS 0.9.0 release.
+
+The important distinction is that the release tooling is portable Rust, not
+that every application dependency is pure Rust. GTK and WebKitGTK are normal
+Linux system libraries and should be linked from the runner's Ubuntu packages.
+Apple bundles need portable construction because no macOS runner is available.
+
+The implementation in this repository is split between:
+
+- `.gitea/workflows/release.yml`: orchestration and target-specific build setup.
+- `.gitea/scripts/install-linux-build-dependencies.sh`: native ARM64 packages
+  and the extracted x64 Ubuntu sysroot.
+- `.gitea/scripts/x86_64-linux-gnu-gcc-pic` and
+  `.gitea/scripts/x86_64-linux-gnu-g++-pic`: cross-compiler wrappers.
+- `crates/bds-release`: release notes, Gitea API access, dependency downloads,
+  archive creation, Apple bundle construction, signing, and upload.
+
+## Resulting artifact matrix
+
+Build CLI and desktop artifacts separately so users do not have to download a
+GUI application to obtain command-line tools. RuDS creates two assets per target:
+
+| Target | CLI | Desktop |
+| --- | --- | --- |
+| macOS ARM64 | `.tar.gz` | `.dmg` containing an `.app` |
+| macOS x64 | `.tar.gz` | `.dmg` containing an `.app` |
+| Linux ARM64 | `.tar.gz` | `.tar.gz` |
+| Linux x64 | `.tar.gz` | `.tar.gz` |
+| Windows x64 | `.zip` | portable `.zip` |
+
+Compile all final programs for a target in one Cargo invocation. In this
+repository those are `bds-ui`, `bds-cli`, and `bds-mcp`. This keeps their feature
+selection and shared runtime libraries consistent.
+
+## Pipeline shape
+
+Use a tag-triggered, draft-first pipeline:
+
+1. Check out full history, test the release helper, generate notes, and create
+   or update a draft Gitea release.
+2. Build and upload every platform into that release ID.
+3. Publish only after every build job succeeds.
+4. Query Gitea afterward and verify the public release, target commit, and asset
+   manifest.
+
+A minimal dependency graph is:
+
+```text
+prepare draft
+    |
+    +-- Linux x64
+            |
+            +-- Linux ARM64
+            +-- macOS ARM64 + x64
+            +-- Windows x64
+                    |
+                    +-- publish
+```
+
+RuDS deliberately runs the most failure-prone cross-link first. On a single
+runner all jobs serialize anyway, and failing early avoids spending hours on
+other targets first. With several independent runners, make all platform jobs
+depend only on preparation and let them run in parallel.
+
+The draft is the transaction boundary. Never make a release public before all
+assets exist. The final publisher should ideally fetch the draft and compare
+the exact expected filenames and non-zero sizes before setting `draft: false`.
+RuDS currently relies on job dependencies plus strict per-package input checks;
+adding an explicit final manifest check would make that boundary stronger.
+
+## Gitea runner and permissions
+
+Register a label that describes the real machine, for example
+`linux-arm64:host`, and use `runs-on: linux-arm64`. Do not treat a name such as
+`ubuntu-latest` as an implicit x64 machine. Gitea runner labels have the form
+`label[:schema[:args]]`.
+
+The runner needs:
+
+- Enough disk for Cargo registries, several target trees, cross SDKs, sysroots,
+  and container layers.
+- Outbound HTTPS access to Rust distribution servers, crates.io, Gitea, NuGet,
+  GitHub release assets, and the configured container registry.
+- Passwordless access to install Ubuntu build packages, or a prebuilt runner
+  image containing them.
+- Container support for the macOS and Windows build images.
+- A URL by which build containers can reach Gitea. `localhost` inside a
+  container is the container itself, not the runner host.
+
+Container images must publish an ARM64 image even when the target they build is
+x64. Pin images by digest, not a floating tag. Also pin the Rust toolchain and
+always build with `--locked`.
+
+Grant the workflow only the permissions it needs: repository contents read,
+issues read if release notes inspect issues, and releases write. Gitea clamps
+workflow permissions to repository or owner limits. If the built-in
+`secrets.GITEA_TOKEN` cannot write releases on the deployed Gitea version, use a
+repository secret containing a narrowly scoped token.
+
+Gitea Actions resembles GitHub Actions but is not identical. Validate contexts,
+permissions, output files, and action compatibility against the deployed Gitea
+and `act_runner` versions. Non-fully-qualified actions may be downloaded from
+GitHub depending on `DEFAULT_ACTIONS_URL`; mirror or fully qualify them on
+restricted installations.
+
+Pass the draft release ID through a job output. Gitea supports the familiar
+`key=value` step-output file convention; this repository writes to the
+`GITHUB_OUTPUT` compatibility variable and maps that step output to a job
+output. Do not scrape an ID from human-readable command output.
+
+Specialized build containers may not run JavaScript-based checkout actions
+reliably. RuDS performs an authenticated, depth-one `git fetch` of the exact tag
+inside those containers. Fetch the tag ref explicitly and check out
+`FETCH_HEAD`; do not build a mutable branch tip.
+
+## Linux desktop builds
+
+There is no architectural reason to exclude the GTK/WebKit desktop application
+from Linux artifacts. The native ARM64 build is ordinary compilation on the
+ARM64 Ubuntu runner. Install:
+
+```text
+build-essential
+cmake
+libgtk-3-dev
+libwebkit2gtk-4.1-dev
+libxdo-dev
+pkg-config
+```
+
+The resulting application dynamically links the distro GTK/WebKit stack. Do
+not put the entire GTK/WebKit dependency closure into the application archive.
+Bundle application-owned shared libraries, dynamically linked Rust standard
+libraries, and explicitly redistributable runtimes; document the required
+Linux runtime packages for users.
+
+Build on the oldest distro baseline that the release promises to support.
+Using a rolling `ubuntu-latest` silently raises the minimum glibc and WebKitGTK
+versions when the runner image changes. A fixed Ubuntu release or container is
+more reproducible.
+
+Add an `$ORIGIN` runtime search path so executables find shared libraries placed
+beside them:
+
+```text
+-C link-arg=-Wl,-rpath,$ORIGIN
+```
+
+### Cross-linking Linux x64 on ARM64
+
+Rust cross-compilation is the easy part. Native dependencies selected through
+`pkg-config`, C/C++ build scripts, the linker, and the x64 libc are the parts
+that require deliberate configuration.
+
+On an Ubuntu ARM64 host:
+
+1. Restrict the existing Ubuntu deb822 sources to `arm64`.
+2. Add `amd64` as a foreign architecture and add matching amd64 archive and
+   security sources for the same Ubuntu codename.
+3. Install native host tools such as `gcc-x86-64-linux-gnu`,
+   `g++-x86-64-linux-gnu`, `cmake`, and `pkg-config`.
+4. Download the GTK, WebKitGTK, and xdo amd64 development packages and their
+   dependencies with APT's `--download-only` mode.
+5. Extract the `.deb` files with `dpkg-deb --extract` into an isolated sysroot,
+   such as `/opt/ruds-x64-sysroot`.
+6. Point Cargo, the GNU cross-linker, and `pkg-config` exclusively at that
+   sysroot.
+
+Do not normally install the foreign development packages on the ARM64 host.
+Some packages contain target-architecture helper executables or maintainer
+scripts; package installation can attempt to execute x64 programs on ARM64.
+Downloading and extracting produces the headers, libraries, and metadata needed
+for linking without running target code.
+
+The essential environment is:
+
+```text
+AR_x86_64_unknown_linux_gnu=x86_64-linux-gnu-ar
+CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc
+PKG_CONFIG_ALLOW_CROSS=1
+PKG_CONFIG_SYSROOT_DIR=/opt/ruds-x64-sysroot
+PKG_CONFIG_LIBDIR=/opt/ruds-x64-sysroot/usr/lib/x86_64-linux-gnu/pkgconfig:/opt/ruds-x64-sysroot/usr/lib/pkgconfig:/opt/ruds-x64-sysroot/usr/share/pkgconfig
+```
+
+Pass `--sysroot` to the linker, add the target library directories with Cargo
+`-L native=...`, and retain the `$ORIGIN` rpath. Do not leave host pkg-config
+paths in the search list: that can produce an ARM64/x64 mixture which may fail
+late or, worse, appear to link successfully.
+
+Ubuntu libc development linker scripts may refer to absolute legacy paths such
+as `/lib/x86_64-linux-gnu`. Those paths are interpreted inside the sysroot but
+may not exist in an extracted modern Ubuntu package layout. Materialize the
+required libc, libm, libmvec, and loader paths within the sysroot. Do not create
+host-level x64 symlinks.
+
+Architecture-independent `.pc` files may already be installed on the host, so
+APT may not download them again for amd64. They contain metadata rather than
+target executables and can be copied into the sysroot; `PKG_CONFIG_SYSROOT_DIR`
+then redirects their paths.
+
+Some native Rust dependencies build static objects that are later linked into
+application shared libraries. They need position-independent code. Compiler
+wrappers should append `-fPIC` as the final flag so it overrides earlier flags.
+Target-specific Cargo environment names can contain hyphens and therefore are
+not valid shell assignment identifiers. Set them through `env`, for example:
+
+```sh
+env \
+  "CC_x86_64-unknown-linux-gnu=$PWD/.gitea/scripts/x86_64-linux-gnu-gcc-pic" \
+  "CXX_x86_64-unknown-linux-gnu=$PWD/.gitea/scripts/x86_64-linux-gnu-g++-pic" \
+  cargo build --release --locked --target x86_64-unknown-linux-gnu
+```
+
+Audit every `-sys` crate for pkg-config overrides. For RuDS,
+`ZSTD_SYS_USE_PKG_CONFIG` must be unset in the x64 build so zstd does not select
+an incompatible host library.
+
+## macOS without a macOS runner
+
+Use `cargo-zigbuild` in a digest-pinned ARM64-compatible container to compile
+the `aarch64-apple-darwin` and `x86_64-apple-darwin` targets. The container must
+contain a legally usable Apple SDK. Review the SDK and redistribution licenses;
+portable tooling does not remove those obligations.
+
+RuDS supplies these linker requirements:
+
+- Both `@executable_path` and `@executable_path/../Resources` rpaths. The first
+  serves the flat CLI archive and the second serves the `.app` layout.
+- Header padding for later Mach-O load-command rewriting.
+- The chosen minimum macOS deployment version.
+- `-Wl,-undefined,dynamic_lookup` where Apple symbols are intentionally resolved
+  at runtime.
+
+Zig's Darwin linker does not implement every `ld64` option. In particular,
+symbol-specific `-U` handling was not usable here. Broad
+`-undefined,dynamic_lookup` works when late-bound framework symbols are truly
+required, but it can also mask accidental unresolved symbols; use it only with
+dependency inspection and runtime testing.
+
+The Zig Apple link path expected a compiler runtime archive alias. A valid empty
+`libclang_rt.osx.a` archive (`!<arch>\n`) in an explicit native library directory
+satisfied that lookup.
+
+### Intel macOS and version-sensitive native runtimes
+
+Do not assume that the latest machine-learning runtime publishes every Apple
+architecture. The ONNX Runtime release used by newer Rust crates no longer had
+an Intel macOS binary. RuDS keeps a compatible dependency line:
+
+```text
+fastembed 5.12.0
+ort/ort-sys 2.0.0-rc.11
+ONNX Runtime 1.23.2 for macOS x64
+```
+
+The Rust release helper downloads Microsoft's x64 archive, verifies its pinned
+SHA-256 digest, extracts the versioned dylib and license, and creates an
+unversioned linker copy. With `ort-sys` rc.11, the working build variables are:
+
+```text
+ORT_LIB_LOCATION=<directory-containing-libonnxruntime.dylib>
+ORT_PREFER_DYNAMIC_LINK=1
+```
+
+`ORT_LIB_PATH`, documented or accepted by other versions, is ignored by this
+specific prerelease. When a native dependency behaves unexpectedly, read the
+build script for the exact locked crate version rather than documentation for a
+newer release.
+
+### Pure-Rust Apple packaging
+
+Build the application bundle and DMG with Rust crates rather than `hdiutil`,
+`codesign`, shell archive tricks, or a simulated macOS environment. RuDS uses
+Rust libraries to:
+
+1. Create `Contents/MacOS`, `Contents/Resources`, `Info.plist`, the icon, bundle
+   metadata, and URL handlers.
+2. Put executables in `Contents/MacOS` and dylibs/licenses in Resources.
+3. Parse 64-bit little-endian Mach-O load commands and rewrite application
+   dylib IDs and references to `@rpath/<basename>`.
+4. Sign after all binary mutations.
+5. Construct and sign the DMG.
+
+Mach-O load commands have fixed allocated space. Reserve header padding at link
+time or a longer rewritten install name will not fit. Rewrite every applicable
+load command, including weak, re-exported, lazy, and upward dylib loads, and
+never attempt to parse licenses or other resource files as Mach-O binaries.
+
+The current output is ad-hoc signed. Developer ID signing and Apple notarization
+need Apple-issued credentials and a supported submission path; a Linux-only
+pipeline cannot claim notarized output merely because it can construct and
+ad-hoc-sign a valid bundle.
+
+## Windows x64 from Linux ARM64
+
+Use `cargo-xwin` in a digest-pinned ARM64-compatible container. It provides the
+Windows MSVC SDK/import-library environment without requiring a Windows host.
+Install the Rust `x86_64-pc-windows-msvc` target and NASM when native assembly
+dependencies require it, then populate the xwin cache before looking up SDK
+libraries.
+
+RuDS's DirectML-enabled build needs:
+
+- Microsoft DirectML 1.15.4 downloaded from NuGet and verified by SHA-256.
+- `DirectML.lib` for linking, plus `DirectML.dll` and its license in the final
+  package.
+- A correctly cased `PathCch.lib` alias copied from the xwin SDK, because some
+  build tooling expects that spelling.
+- The target release directory on Cargo's native library search path.
+
+Do not force `target-feature=+crt-static` merely to make the archive look more
+self-contained. Windows system and SDK dependencies may require the dynamic
+CRT, and forcing static CRT can produce incompatible link combinations. Bundle
+the redistributable application runtime DLLs that are actually required and
+retain their licenses. RuDS statically links ONNX Runtime on Windows but ships
+DirectML dynamically.
+
+Portable ZIPs are the simplest cross-host Windows result. MSI, MSIX, or signed
+installer production introduces additional tooling, signing certificates, and
+Windows policy requirements and should be treated as separate scope.
+
+## Packaging rules
+
+Keep packaging in a small host-native Rust binary. `bds-release` runs on the
+ARM64 builder even while it packages foreign target files. It uses Rust for
+HTTP, hashing, tar/gzip, ZIP, plist, ICNS, Apple bundle/DMG construction, Mach-O
+mutation, signing, and Gitea API calls.
+
+For each target:
+
+- Require every expected executable and application shared library before
+  creating an archive.
+- Copy dynamic Rust standard libraries from
+  `rustc --target <target> --print target-libdir` when the project links Rust's
+  standard library dynamically.
+- Include application-owned shared libraries and required redistributable
+  third-party runtimes with license files.
+- Give archives deterministic, tag-derived names and a single top-level
+  directory.
+- Reject unsafe tags used as filenames and refuse to overwrite existing output.
+- Stream large uploads with a content length rather than loading them into RAM.
+- Retry network failures, HTTP 429, and server errors with bounded backoff.
+- Verify every downloaded native dependency with a pinned cryptographic digest.
+
+Packaging must follow the target's runtime model. Linux should use the distro's
+GTK/WebKit libraries; macOS `.app` bundles should carry their private dylibs in
+Resources with corrected install names; Windows packages should carry approved
+redistributable DLLs. “Put every shared library beside the executable” is not a
+portable policy.
+
+## Release creation and rebuilding
+
+The preparation job uses full Git history to locate the previous tag, reads
+issue numbers from commit messages, fetches closed issues through Gitea's API,
+and creates categorized release notes. It writes the draft release ID as a step
+output so every build uploads to the same release.
+
+Rebuilding an existing published tag is deliberately explicit. In RuDS,
+`REBUILD_PUBLISHED_RELEASE=true` changes the release back to a draft, deletes
+its assets, regenerates its metadata, and uploads a consistent new set. This is
+destructive and temporarily removes the public release. Prefer a new immutable
+tag unless repairing an existing release is intentional. Rerunning a failed
+draft also clears its partial asset set before rebuilding, which prevents stale
+and new files from being mixed.
+
+Before pushing a release tag:
+
+1. Update the workspace version, lockfile, README, and any visible version
+   metadata together.
+2. Run formatting, the full workspace tests, and relevant release-helper tests.
+3. Build locally for the host where practical.
+4. Commit and push the release commit to the default branch.
+5. Create an annotated tag at that exact commit, then push the tag.
+
+After the workflow finishes, verify through the Gitea API or `tea`:
+
+- Every job is successful rather than merely the workflow appearing complete.
+- The tag and its peeled commit match the intended default-branch commit.
+- The release is public (`draft: false`) and has the expected prerelease flag.
+- The release targets the exact tagged commit.
+- The expected asset names are present exactly once and have non-zero sizes.
+- The repository worktree is clean and local/remote refs agree.
+
+Useful Gitea API endpoints are:
+
+```text
+GET /repos/{owner}/{repo}/actions/runs
+GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs
+GET /repos/{owner}/{repo}/actions/jobs/{job_id}/logs
+GET /repos/{owner}/{repo}/releases/tags/{tag}
+```
+
+Job logs can be very large. Download or query them once, then filter locally for
+the failing step and nearby compiler/linker diagnostics instead of repeatedly
+printing the entire log.
+
+## Common failures
+
+| Symptom | Cause | Fix |
+| --- | --- | --- |
+| `Exec format error` during x64 dependency installation | A foreign package's executable or maintainer script ran on ARM64 | Download packages only and extract them into a sysroot |
+| Linker reports wrong ELF architecture | Host libraries leaked through pkg-config or a build script | Restrict `PKG_CONFIG_LIBDIR`, set its sysroot, and audit `-sys` crate overrides |
+| Relocation errors while creating a `.so` | A C/C++ static object was built without PIC | Append `-fPIC` in target compiler wrappers |
+| libc or loader missing despite downloaded packages | Ubuntu linker scripts contain absolute legacy paths | Materialize those target paths inside the sysroot |
+| Linux binary starts but cannot find project `.so` files | No relative runtime search path | Link with an `$ORIGIN` rpath and package the libraries beside the binary |
+| macOS link rejects `-U` | Zig's Darwin linker differs from Apple `ld64` | Use supported options; use dynamic lookup only when justified |
+| Mach-O install-name rewrite does not fit | No load-command header padding | Add header padding before linking, then mutate before signing |
+| Intel macOS ORT download fails | The selected upstream release dropped x64 macOS | Pin a compatible crate/runtime pair and supply a verified dylib |
+| `ort-sys` ignores a supplied runtime | Environment variable belongs to another crate version | Inspect the exact locked crate build script; rc.11 uses `ORT_LIB_LOCATION` |
+| Windows library is “missing” despite SDK contents | Import-library case/name differs from build-script expectation | Add an explicit linker alias such as `PathCch.lib` |
+| Windows CRT link conflicts | Static CRT was forced across incompatible dependencies | Use the target's normal dynamic CRT unless the full graph supports static CRT |
+| Checkout fails only inside a build container | The image cannot execute the checkout action or cannot reach Gitea | Fetch the exact tag with authenticated Git in the container |
+| Release is public but incomplete | Publication happened before asset completion | Create a draft first and publish only behind all build jobs and manifest checks |
+| A release takes hours | Cold Rust/native builds on one physical ARM64 runner serialize | Add real runners, carefully cache immutable inputs, or prebake dependencies |
+
+## Performance and reproducibility
+
+A cold five-target build with GTK/WebKit, ONNX Runtime, and ML dependencies can
+take about three hours on a single ARM64 runner. The workflow itself is simple;
+compilation and native dependency setup dominate the time.
+
+Safe ways to reduce it include:
+
+- Add separate runners with the same label, or platform-specific labels, so
+  independent target jobs truly run concurrently.
+- Prebuild a versioned runner/container image containing system dependencies and
+  the Linux x64 sysroot.
+- Cache Cargo registry, Git checkouts, target artifacts, Rust toolchains, and
+  cross-SDK caches with keys including the lockfile, Rust version, target,
+  feature set, distro version, and relevant build scripts.
+- Avoid concurrent memory-heavy jobs on one physical host even if several
+  logical runners are registered there.
+
+Prefer a slow clean build over a fast cache that can mix architectures or stale
+native libraries. Keep caches disabled until their keys and invalidation rules
+are demonstrably correct.
+
+## Validation depth
+
+Cross-built programs cannot be fully exercised on the ARM64 host, but artifacts
+should still receive structural checks:
+
+- ELF: inspect architecture, `DT_NEEDED`, rpath, and unexpected build-host paths
+  with `file` and `readelf`.
+- Mach-O: inspect architectures, load commands, bundle layout, plist, signatures,
+  and DMG contents with parsers that run on Linux.
+- PE: inspect machine type and imports with LLVM or MinGW object tools.
+- Archives: extract into temporary directories, compare the manifest, and check
+  executable modes and licenses.
+
+Then smoke-test on real target systems. QEMU or Wine can catch basic loader
+failures but is not authoritative for GTK/WebKit behavior, macOS signing and
+Gatekeeper, or Windows GPU/DirectML behavior.
+
+## References
+
+- [Gitea Actions](https://docs.gitea.com/usage/actions)
+- [Gitea Actions design and runner labels](https://docs.gitea.com/usage/actions/design)
+- [Gitea Actions variables](https://docs.gitea.com/usage/actions/actions-variables)
+- [Gitea API](https://docs.gitea.com/api/1.24/)
+- [cargo-zigbuild](https://github.com/rust-cross/cargo-zigbuild)
+- [cargo-xwin](https://github.com/rust-cross/cargo-xwin)
+
+Treat the checked-in workflow and Rust helper as the executable source of truth.
+This document explains why their unusual-looking pieces exist so that future
+agents do not remove them as apparent complexity and rediscover the same target,
+linker, packaging, and release-state failures.