CI: consolidate 13 Gitea workflows into one bounded 15-minute required gate #115

New Issue

hugo · 2026-08-12T15:21:13Z

hugo commented

2026-08-12 15:21:13 +00:00

Problem

MetaCrate currently has 13 active Gitea Actions workflows. Broad and overlapping path filters cause a normal source change to start several independent jobs on the same ARM64 VPS runner. Each job repeats checkout, toolchain setup, package installation, native dependency preparation, and overlapping Cargo work. This produces long queues, repeated compilation, and poor time-to-signal.

This is a critical CI correctness and runtime bug, not only a cosmetic workflow cleanup. A green result must still mean that the code is ready, but routine validation must complete in about 15 minutes maximum.

Measured baseline

The Gitea API currently reports 13 active workflows and 333 historical runs. Completed runs with valid timestamps show:

Workflow	Observed runtime
Skia feature	31.2 min median successful; 32.0 min maximum
Release platform and feature matrix	86.0 min maximum
Native release artifact audit	66.6 min maximum
Native code generation	13.2 min median successful; 18.5 min maximum
Native Rust workspace compile	22.9 min maximum successful
Performance evidence	13.8 min
Supply-chain audit	about 9.2 min
Imaging and meshing gate	5.1 min median successful
JPEG 2000 feature	2.8 min median successful

Examples: Skia run 324, release-matrix run 322, artifact-audit run 315, and workspace run 236.

The YAML inventory also shows:

all 13 workflows repeat checkout and Rust toolchain setup;
11 workflows run an apt update/install;
8 workflows build or install the pinned OpenJPEG dependency;
6 workflows repeat formatting checks;
5 workflows run clippy and 4 build documentation;
8 workflows globally force single-job Cargo execution;
only 2 workflows configure any cache;
5 workflows set FORCE_SKIA_BINARIES_DOWNLOAD.

Current workflow inventory

Workflow	Responsibility that must be preserved or deliberately reassigned
`api-surface.yml`	generated API mapping, SemVer baseline/audit, downstream consumer compilation, focused lint
`artifact-audit.yml`	source packages, release binaries, native linkage and artifact/provenance inspection
`codegen.yml`	deterministic generation, generated wire/catalog tests, API fixture, clean generator artifact
`concurrency-audit.yml`	focused race regressions and bounded resource soak
`documentation.yml`	documentation audit, warning-free docs, doctests, examples, documented program tests
`imaging-meshing.yml`	milestone 06 checks, imaging/meshing tests, parity cases, benchmark compilation, lint/docs
`jpeg2000.yml`	optional OpenJPEG/JPEG 2000 feature build and tests
`performance.yml`	pinned C# comparison harness and committed performance evidence audit
`release-candidate.yml`	aggregate release-readiness gate and evidence
`release-matrix.yml`	MSRV/default/minimal/features/release profiles and portable cross-target checks
`rust-workspace.yml`	milestone checks, workspace compilation, program smoke, audio feature, extension lint/docs
`skia.yml`	default and Skia feature tests plus package listing
`supply-chain.yml`	dependency graph, advisory, license, source, provenance, duplicate and unused dependency audits

Required outcome

Replace the current push/pull-request fan-out with one required ci.yml workflow that is the authoritative code-ready signal.
Prefer a single job on the VPS so checkout, native setup, Cargo artifacts, and caches are shared. Multiple jobs are acceptable only if the complete workflow still stays within the runtime budget and does not duplicate setup or compilation.
Keep no more than a very small number of workflows overall. Expensive release packaging, full release matrices, long soak tests, or reference performance regeneration may remain separate only when they are manual, scheduled, or tag/release-only and do not run on every push or pull request. The goal is one routine CI workflow and at most two clearly justified non-routine workflows.
Put the validation command graph in one maintained entry point, preferably the existing metacrate-ci-matrix tool or a cross-platform repository-owned CI driver. YAML must orchestrate it rather than duplicate long command lists.
Create a coverage manifest mapping every check in the table above to:
- the required CI gate;
- a manual/scheduled release gate; or
- an explicitly documented retirement with rationale.
Order the required gate for fast failure: static/generated checks and formatting first, then compile/test/features, then clippy/docs/audits.
Run every Gitea job on ubuntu-latest only. Portable Windows and macOS mappings must remain cross-compiled or statically checked from Ubuntu; do not add macOS or Windows Gitea runners.

Runtime and caching work

Use a coherent cache strategy for Cargo registry/git data, compiled Rust artifacts or sccache, native artifacts, and pinned audit tools. Keys must include the Rust version, target triple, Cargo.lock, relevant feature/native configuration, and native installer inputs.
Install system packages once per workflow, not once per concern.
Stop rebuilding OpenJPEG 2.5.4 for every workflow. Use a verified runner image, architecture-keyed cache, or one shared audited install while preserving the exact-version and provenance checks.
Do not run equivalent Cargo test/check/clippy/doc invocations more than once. Reuse compatible artifacts and avoid a release-candidate job that simply recompiles everything already proven by CI.
Tune bounded concurrency for the ARM64 VPS instead of globally forcing CARGO_BUILD_JOBS=1 everywhere. Keep memory use safe and prove the selected setting with measurements.
Add workflow concurrency/cancellation so a newer commit cancels obsolete queued or running validation for the same branch or pull request.
Cache or preinstall pinned cargo-deny and cargo-machete; do not compile both tools from scratch on every source push.

Skia regression that must be fixed

The exact MetaCrate Skia 0.99.0 binary exists for aarch64-unknown-linux-gnu with the requested JPEG/PDF/SVG/textlayout/Vulkan/WebP feature tuple:

skia-binaries-a25a0fdb7d90429aa2d1-aarch64-unknown-linux-gnu-jpegd-jpege-pdf-svg-textlayout-vulkan-webpd-webpe.tar.gz

The current downstream workflows set FORCE_SKIA_BINARIES_DOWNLOAD. In rust-skia 0.99.0 this variable is a repository-testing switch: it asks for a Git hash from the skia-bindings package directory. A crates.io dependency directory is not a Git checkout, so selection can return no key and fall through to STARTING A FULL BUILD. Merely setting the variable to 0 still activates it because rust-skia checks for presence.

The consolidated workflow must:

remove this misuse and use the normal crates.io .cargo_vcs_info.json hash lookup, or provide an explicitly pinned/mirrored SKIA_BINARIES_URL;
verify the actual target is aarch64-unknown-linux-gnu, not musl or another triple;
verify that GitHub and its release-assets redirect host are reachable, or use the mirror;
fail fast rather than silently compile Skia from source when the expected archive is unavailable;
assert in CI evidence that DOWNLOAD AND INSTALL SUCCEEDED occurred and STARTING A FULL BUILD did not occur.

Acceptance criteria

Exactly one workflow is required and automatically triggered for routine pushes and pull requests.
No more than three active workflow files remain, and any workflow beyond the required CI gate is manual, scheduled, or release/tag-only with a documented reason.
The old 13-workflow check inventory is mapped to the consolidated design, with no silent loss of functional, feature, API, documentation, security, packaging, or cross-target coverage.
The required gate includes the code-ready subset of formatting, deterministic generation, API/SemVer validation, workspace and feature compilation, unit/integration/parity tests, Skia and JPEG 2000 coverage, program fake/offline smoke, clippy, warning-free documentation, and dependency/provenance policy checks.
Long soak, full packaging, full release matrix, and C# performance regeneration are either made fast enough for the required gate or moved to an explicitly non-routine workflow; their fast regression/evidence validation remains in routine CI where applicable.
Every job uses ubuntu-latest; no macOS or Windows Gitea jobs are introduced.
The required workflow has a hard timeout-minutes: 15. A practical internal target of 12 minutes leaves shutdown/upload variance below the hard limit.
On the actual Linux ARM64 Gitea VPS, five consecutive validation runs complete in 15:00 or less from started_at to completed_at. The sample must include at least one deliberate cold dependency/native cache run and at least four normal warm-cache runs.
The validation PR records before/after elapsed time for every stage and links all five qualifying Gitea runs. Evidence includes runner architecture, Rust host/target, cache hit/miss state, peak memory if available, and Skia download outcome.
No qualifying run builds Skia from source. Logs/evidence prove the exact ARM64 GNU archive or approved mirror was installed.
A cold or invalid native cache cannot silently use stale OpenJPEG/Skia artifacts; exact versions, hashes/provenance, target triple, and feature keys are validated.
One commit creates at most one routine runner-consuming workflow run, and obsolete same-branch runs are canceled.
Branch protection is updated to require only the consolidated CI result after coverage and runtime parity are proven.
CI/release documentation is updated to describe the final workflow split, local reproduction command, cache invalidation, runtime budget, and troubleshooting.
The old workflows are removed or disabled only after the consolidated gate passes the coverage comparison and ARM64 runtime validation.

A design that is functionally complete but regularly exceeds 15 minutes is not accepted. A design that meets 15 minutes by dropping required coverage is also not accepted.

## Problem MetaCrate currently has 13 active Gitea Actions workflows. Broad and overlapping path filters cause a normal source change to start several independent jobs on the same ARM64 VPS runner. Each job repeats checkout, toolchain setup, package installation, native dependency preparation, and overlapping Cargo work. This produces long queues, repeated compilation, and poor time-to-signal. This is a critical CI correctness and runtime bug, not only a cosmetic workflow cleanup. A green result must still mean that the code is ready, but routine validation must complete in about 15 minutes maximum. ## Measured baseline The Gitea API currently reports 13 active workflows and 333 historical runs. Completed runs with valid timestamps show: | Workflow | Observed runtime | | --- | ---: | | Skia feature | 31.2 min median successful; 32.0 min maximum | | Release platform and feature matrix | 86.0 min maximum | | Native release artifact audit | 66.6 min maximum | | Native code generation | 13.2 min median successful; 18.5 min maximum | | Native Rust workspace compile | 22.9 min maximum successful | | Performance evidence | 13.8 min | | Supply-chain audit | about 9.2 min | | Imaging and meshing gate | 5.1 min median successful | | JPEG 2000 feature | 2.8 min median successful | Examples: [Skia run 324](https://git.rfc1437.de/hugo/MetaCrate/actions/runs/324), [release-matrix run 322](https://git.rfc1437.de/hugo/MetaCrate/actions/runs/322), [artifact-audit run 315](https://git.rfc1437.de/hugo/MetaCrate/actions/runs/315), and [workspace run 236](https://git.rfc1437.de/hugo/MetaCrate/actions/runs/236). The YAML inventory also shows: - all 13 workflows repeat checkout and Rust toolchain setup; - 11 workflows run an apt update/install; - 8 workflows build or install the pinned OpenJPEG dependency; - 6 workflows repeat formatting checks; - 5 workflows run clippy and 4 build documentation; - 8 workflows globally force single-job Cargo execution; - only 2 workflows configure any cache; - 5 workflows set `FORCE_SKIA_BINARIES_DOWNLOAD`. ## Current workflow inventory | Workflow | Responsibility that must be preserved or deliberately reassigned | | --- | --- | | `api-surface.yml` | generated API mapping, SemVer baseline/audit, downstream consumer compilation, focused lint | | `artifact-audit.yml` | source packages, release binaries, native linkage and artifact/provenance inspection | | `codegen.yml` | deterministic generation, generated wire/catalog tests, API fixture, clean generator artifact | | `concurrency-audit.yml` | focused race regressions and bounded resource soak | | `documentation.yml` | documentation audit, warning-free docs, doctests, examples, documented program tests | | `imaging-meshing.yml` | milestone 06 checks, imaging/meshing tests, parity cases, benchmark compilation, lint/docs | | `jpeg2000.yml` | optional OpenJPEG/JPEG 2000 feature build and tests | | `performance.yml` | pinned C# comparison harness and committed performance evidence audit | | `release-candidate.yml` | aggregate release-readiness gate and evidence | | `release-matrix.yml` | MSRV/default/minimal/features/release profiles and portable cross-target checks | | `rust-workspace.yml` | milestone checks, workspace compilation, program smoke, audio feature, extension lint/docs | | `skia.yml` | default and Skia feature tests plus package listing | | `supply-chain.yml` | dependency graph, advisory, license, source, provenance, duplicate and unused dependency audits | ## Required outcome 1. Replace the current push/pull-request fan-out with one required `ci.yml` workflow that is the authoritative code-ready signal. 2. Prefer a single job on the VPS so checkout, native setup, Cargo artifacts, and caches are shared. Multiple jobs are acceptable only if the complete workflow still stays within the runtime budget and does not duplicate setup or compilation. 3. Keep no more than a very small number of workflows overall. Expensive release packaging, full release matrices, long soak tests, or reference performance regeneration may remain separate only when they are manual, scheduled, or tag/release-only and do not run on every push or pull request. The goal is one routine CI workflow and at most two clearly justified non-routine workflows. 4. Put the validation command graph in one maintained entry point, preferably the existing `metacrate-ci-matrix` tool or a cross-platform repository-owned CI driver. YAML must orchestrate it rather than duplicate long command lists. 5. Create a coverage manifest mapping every check in the table above to: - the required CI gate; - a manual/scheduled release gate; or - an explicitly documented retirement with rationale. 6. Order the required gate for fast failure: static/generated checks and formatting first, then compile/test/features, then clippy/docs/audits. 7. Run every Gitea job on `ubuntu-latest` only. Portable Windows and macOS mappings must remain cross-compiled or statically checked from Ubuntu; do not add macOS or Windows Gitea runners. ## Runtime and caching work - Use a coherent cache strategy for Cargo registry/git data, compiled Rust artifacts or `sccache`, native artifacts, and pinned audit tools. Keys must include the Rust version, target triple, `Cargo.lock`, relevant feature/native configuration, and native installer inputs. - Install system packages once per workflow, not once per concern. - Stop rebuilding OpenJPEG 2.5.4 for every workflow. Use a verified runner image, architecture-keyed cache, or one shared audited install while preserving the exact-version and provenance checks. - Do not run equivalent Cargo test/check/clippy/doc invocations more than once. Reuse compatible artifacts and avoid a release-candidate job that simply recompiles everything already proven by CI. - Tune bounded concurrency for the ARM64 VPS instead of globally forcing `CARGO_BUILD_JOBS=1` everywhere. Keep memory use safe and prove the selected setting with measurements. - Add workflow concurrency/cancellation so a newer commit cancels obsolete queued or running validation for the same branch or pull request. - Cache or preinstall pinned `cargo-deny` and `cargo-machete`; do not compile both tools from scratch on every source push. ## Skia regression that must be fixed The exact MetaCrate Skia 0.99.0 binary exists for `aarch64-unknown-linux-gnu` with the requested JPEG/PDF/SVG/textlayout/Vulkan/WebP feature tuple: `skia-binaries-a25a0fdb7d90429aa2d1-aarch64-unknown-linux-gnu-jpegd-jpege-pdf-svg-textlayout-vulkan-webpd-webpe.tar.gz` The current downstream workflows set `FORCE_SKIA_BINARIES_DOWNLOAD`. In rust-skia 0.99.0 this variable is a repository-testing switch: it asks for a Git hash from the `skia-bindings` package directory. A crates.io dependency directory is not a Git checkout, so selection can return no key and fall through to `STARTING A FULL BUILD`. Merely setting the variable to `0` still activates it because rust-skia checks for presence. The consolidated workflow must: - remove this misuse and use the normal crates.io `.cargo_vcs_info.json` hash lookup, or provide an explicitly pinned/mirrored `SKIA_BINARIES_URL`; - verify the actual target is `aarch64-unknown-linux-gnu`, not musl or another triple; - verify that GitHub and its release-assets redirect host are reachable, or use the mirror; - fail fast rather than silently compile Skia from source when the expected archive is unavailable; - assert in CI evidence that `DOWNLOAD AND INSTALL SUCCEEDED` occurred and `STARTING A FULL BUILD` did not occur. ## Acceptance criteria - [ ] Exactly one workflow is required and automatically triggered for routine pushes and pull requests. - [ ] No more than three active workflow files remain, and any workflow beyond the required CI gate is manual, scheduled, or release/tag-only with a documented reason. - [ ] The old 13-workflow check inventory is mapped to the consolidated design, with no silent loss of functional, feature, API, documentation, security, packaging, or cross-target coverage. - [ ] The required gate includes the code-ready subset of formatting, deterministic generation, API/SemVer validation, workspace and feature compilation, unit/integration/parity tests, Skia and JPEG 2000 coverage, program fake/offline smoke, clippy, warning-free documentation, and dependency/provenance policy checks. - [ ] Long soak, full packaging, full release matrix, and C# performance regeneration are either made fast enough for the required gate or moved to an explicitly non-routine workflow; their fast regression/evidence validation remains in routine CI where applicable. - [ ] Every job uses `ubuntu-latest`; no macOS or Windows Gitea jobs are introduced. - [ ] The required workflow has a hard `timeout-minutes: 15`. A practical internal target of 12 minutes leaves shutdown/upload variance below the hard limit. - [ ] On the actual Linux ARM64 Gitea VPS, five consecutive validation runs complete in 15:00 or less from `started_at` to `completed_at`. The sample must include at least one deliberate cold dependency/native cache run and at least four normal warm-cache runs. - [ ] The validation PR records before/after elapsed time for every stage and links all five qualifying Gitea runs. Evidence includes runner architecture, Rust host/target, cache hit/miss state, peak memory if available, and Skia download outcome. - [ ] No qualifying run builds Skia from source. Logs/evidence prove the exact ARM64 GNU archive or approved mirror was installed. - [ ] A cold or invalid native cache cannot silently use stale OpenJPEG/Skia artifacts; exact versions, hashes/provenance, target triple, and feature keys are validated. - [ ] One commit creates at most one routine runner-consuming workflow run, and obsolete same-branch runs are canceled. - [ ] Branch protection is updated to require only the consolidated CI result after coverage and runtime parity are proven. - [ ] CI/release documentation is updated to describe the final workflow split, local reproduction command, cache invalidation, runtime budget, and troubleshooting. - [ ] The old workflows are removed or disabled only after the consolidated gate passes the coverage comparison and ARM64 runtime validation. A design that is functionally complete but regularly exceeds 15 minutes is not accepted. A design that meets 15 minutes by dropping required coverage is also not accepted.

hugo added this to the 13 - Extensions milestone 2026-08-12 15:21:13 +00:00

hugo added the bug critical labels 2026-08-12 15:21:13 +00:00

hugo modified the milestone from 13 - Extensions to 12 - Hardening and release

2026-08-12 16:21:02 +00:00

hugo referenced this issue from a commit