Match bDS2 full-site generation performance #108
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
RuDS full generation performs substantially more repeated work than bDS2 on large projects. Align the rendering pipeline by bulk-loading translations, links, and post-media into a reusable render context; precompute project-wide maps and taxonomy data once; cache parsed Liquid templates and reusable rendered content; render single pages concurrently across available cores; and preload generated-file hashes plus batch changed-hash upserts. Avoid retaining duplicate HTML copies when output can be streamed. Do not add a large-site benchmark or query-count regression test before changing behavior, that would take far too long and expand unit test runtime. do a proper evaluation of the ../bDS2 code on how that works and implement the RuDS code the same way. You have a working base to read to know how this should work.
Details on the problems of the RuDS code:
RuDS does per-post translation and media queries for every language and every generation section (site.rs (line 279), site.rs (line 314)). For the active 10,305-post, two-language project, that is roughly 103,050 translation queries plus 103,050 media queries across five sections. bDS2 bulk-loads and groups these once in its RenderContext (line 58).
Each RuDS single-page render performs two link queries, rebuilds an all-post lookup map, and recalculates taxonomy counts (site.rs (line 1116)). That creates quadratic project-wide work. bDS2 precomputes these maps.
RuDS renders single posts serially (site.rs (line 387)); bDS2 distributes them across all scheduler cores (outputs.ex (line 572), outputs.ex (line 684)).
RuDS rebuilds the Liquid parser, recompiles partials, and parses the template for every page (page_renderer.rs (line 48)). bDS2 memoizes parsed template ASTs (template_selection.ex (line 146)). RuDS also queries/upserts generated hashes per file, whereas bDS2 preloads and batch-upserts them.
Resolved in
d9085cf. RuDS now matches bDS2's full-generation execution model: all online CPU workers are used, rendering sections run independently, reusable render data and compiled templates are shared, page renders receive only route-local post data, and generated-file hashes are loaded and written in batches. Added regression coverage and updated the Allium task specification. Verified with the full workspace test suite, Clippy, build, formatting, and Allium validation.