bDS/WORKER_PLAN.md

Worker Threads Architecture for Blog Generation

Problem

Cmd-R (Render Site) is slow with 10k+ posts / ~20k pages. The rendering pipeline is CPU-bound (Liquid templates + Markdown parsing). All current Promise.all parallelism just interleaves I/O on a single core. Actual multi-core parallelism via worker_threads is needed.

Current Architecture

blogHandlers.ts (IPC entry)
  → preloadGenerationData() — loads all posts, translations, menu
  → Promise.all([5 section tasks])  ← runs on ONE CPU core
      ├── core     (root pages, sitemap, feeds, assets)
      ├── single   (one page per post — THE bottleneck, ~20k pages)
      ├── category (paginated category index pages)
      ├── tag      (paginated tag index pages)
      └── date     (year/month/day archive pages)

Each section calls BlogGenerationEngine.generate() which:
  1. Builds GenerationPostIndex (tags, categories, date buckets)
  2. Bulk-loads file hashes from DB
  3. Creates route renderer (Liquid + PreviewServer + cached engines)
  4. Renders pages sequentially/batched → writes files if hash changed

Shared mutable state across sections:

• SQLite database (libsql, WAL mode) — singleton connection
• File system output directory (but sections write to disjoint paths)
• Template caches (Liquid cache: true) — populated once, read-only after
• PreloadedGenerationData — read-only after creation

Existing worker_threads usage: Pyodide macro workers (pythonMacro.worker.ts, BlogmarkPythonWorkerRuntime.ts) already use worker_threads successfully.

Target Architecture

MAIN THREAD                              WORKER THREADS (N = cpu_count - 1)
───────────                              ──────────────────────────────────
blogHandlers.ts                          
  preloadGenerationData()               
  serialize + partition posts            
       │                                 
       ├──► Worker 1: posts[0..chunk]    → own DB conn, own Liquid, render + write
       ├──► Worker 2: posts[chunk..2chunk] → own DB conn, own Liquid, render + write
       ├──► Worker 3: posts[2chunk..3chunk] → own DB conn, own Liquid, render + write
       └──► Worker N: posts[...rest]     → own DB conn, own Liquid, render + write
       │
       ├── receive progress messages → TaskManager.emit()
       ├── receive results → merge stats
       └── return merged BlogGenerationResult

Phased Implementation

Phase 1 — Single Post Worker Pool (highest impact)

Move generateSinglePostPages to a worker pool. Single posts are the bottleneck (~20k of ~20k pages). Other sections stay in main thread.

1.1 Spike: Verify dependencies work in worker_threads

• Test @libsql/client opens a second connection in a worker thread (WAL mode)
• Test liquidjs renders a template in a worker thread
• Measure memory overhead per worker with 10k post metadata

1.2 Create generation.worker.ts

New worker entry point that:

• Receives via workerData: serialized options, post chunk, template roots, DB path, hash cache
• Opens its own @libsql/client connection (WAL mode allows concurrent readers/writers)
• Creates its own Liquid instance with cache: true + registers custom filters
• Creates its own PageRenderer, PreviewServer, route renderer
• Renders assigned posts → writes HTML files + updates file hashes in DB
• Sends progress via parentPort.postMessage({ type: 'progress', ... })
• Sends result via parentPort.postMessage({ type: 'result', stats })

1.3 Serialize PreloadedGenerationData

• PostData[] contains Date objects → serialize to ISO strings, parse back in worker
• Post content is lazy-loaded from filesystem → workers read post files directly
• HtmlRewriteContext maps → pass as plain Record<string, string> (already partially converted)
• Each worker bulk-loads its own generatedHashCache from DB

1.4 Extract PageRenderer factory for workers

• Extract filter registration (markdown, i18n) into a shared createPageRenderer(config) function
• Workers call this with their own DB-backed engines
• Keep macroTemplateCache and macroLiquid as worker-local singletons (they self-populate)

1.5 Create GenerationWorkerPool

New class that:

• Spawns N workers (os.cpus().length - 1, configurable, min 1)
• Distributes post chunks to workers (round-robin or equal split)
• Collects progress messages → relays to onProgress callback
• Collects results → merges stats (pagesWritten, pagesSkipped)
• Handles worker errors/crashes gracefully
• Tears down workers when generation completes

1.6 Refactor BlogGenerationEngine.generate() coordinator

• Split into coordinator (main thread) + worker payload
• Coordinator: loads data, partitions posts, manages worker pool, merges results
• Multi-language subtree loop: each language pass creates a new set of worker tasks
• Non-single sections (core, category, tag, date) remain in main thread

1.7 Progress reporting

• Workers: parentPort.postMessage({ type: 'progress', value, message })
• Main thread: listen on each worker, relay to TaskManager.emit() → IPC → renderer
• Aggregate progress across all workers for accurate progress bar

1.8 Testing

• Unit tests: mock worker pool, test coordinator logic
• Integration test: spawn real workers with in-memory SQLite + template files
• Verify existing BlogGenerationEngine.test.ts tests still pass (they mock at engine boundary)

Phase 2 — All Sections in Workers

Move category/tag/date sections to workers too. Each section gets one worker.

• Category pages → one worker
• Tag pages → one worker
• Date archive pages → one worker
• Core pages stay in main thread (sitemap/feeds/assets are one-time + small)

Phase 3 — Python Macro Handling

Handle posts with Python macros across worker boundaries.

Recommended approach: Pre-expansion pass

1. Before distributing posts to workers, scan for Python macro markers
2. Expand macros in main thread (Pyodide is already in a worker — reuse existing PythonMacroWorkerRuntime)
3. Cache expanded content
4. Pass pre-expanded content to generation workers

Alternative approaches (if pre-expansion is too slow):

• Workers send macro calls back to main thread via messages (RPC pattern)
• Workers skip macro posts; main thread renders them in a second pass

Key Files to Modify

File	Change
src/main/engine/generation.worker.ts	NEW — worker entry point
src/main/engine/GenerationWorkerPool.ts	NEW — worker pool manager
src/main/engine/BlogGenerationEngine.ts	Refactor generate() into coordinator
src/main/engine/PageRenderer.ts	Extract filter registration into factory function
src/main/engine/GenerationRouteRendererFactory.ts	Make usable from worker context
src/main/ipc/blogHandlers.ts	Pass DB path + template roots to worker pool
src/main/engine/RoutePageGenerationService.ts	generateSinglePostPages moves to worker
vite.config.ts / tsconfig.main.json	Worker entry point build config

Data Serialization Requirements

Data	Size (10k posts)	Strategy
BlogGenerationOptions	~1KB	Pass as workerData (plain object)
PreloadedGenerationData	~2-5MB	Serialize Date→ISO string, pass via workerData
Post content (body)	N/A	Workers read from filesystem (lazy)
HtmlRewriteContext	~500KB	Pass as Record<string, string> in workerData
generatedHashCache	~1MB	Each worker bulk-loads from DB independently
Template files	~50KB	Workers read from filesystem
Progress updates	tiny	parentPort.postMessage()

Risks & Mitigations

Risk	Mitigation
@libsql/client native bindings may not work in workers	Spike first (1.1). Fallback: use better-sqlite3 directly in workers.
Memory pressure (N copies of post metadata)	Measure in spike. Could use SharedArrayBuffer or reduce per-worker data.
Pyodide macros can't run in generation workers	Phase 3 pre-expansion pass. Most posts don't use Python macros.
Worker crashes lose progress	Pool manager catches errors, reports partial results, coordinator can retry failed chunks.
Template root paths differ in packaged app	Pass process.resourcesPath via workerData. Already has CWD fallback.
Build configuration for worker entry point	Add worker to Vite/esbuild config (existing pattern from pythonMacro.worker.ts).

Success Criteria

• Render Site with 10k posts uses all available CPU cores
• Wall-clock time scales roughly linearly with core count (e.g., 4 cores → ~4x faster)
• No regression in output correctness (identical HTML output)
• Progress bar still works smoothly
• Memory usage stays under 2GB total