bDS/PYTHON_SCRIPTING.md

Python Scripting — Remaining Work (Implementation-First)

Last verified: 24 Feb 2026

This document is intentionally reduced to what is still left to implement. When plan and code differ, code is the source of truth.

Implemented (checked)

• Pyodide dependency integrated.
• Renderer worker runtime exists (pythonRuntime.worker.ts) with ready/error/stdout/run protocol.
• Runtime timeout watchdog + reset/recovery implemented in PythonRuntimeManager.
• Renderer runtime request queueing implemented (concurrent calls are serialized in manager).
• ABI v1 schemas and validation for macro context/result implemented (abiV1.ts).
• Benchmark harness implemented (npm run bench:python-runtime -- <iterations>).
• Script persistence model implemented (scripts DB table + scripts/*.py files).
• Script metadata model implemented (kind, entrypoint, enabled, version, etc.).
• Main process script CRUD engine + IPC handlers implemented.
• Preload + shared API typings for scripts implemented.
• Renderer scripts UX implemented (sidebar list, editor, save, run, delete).
• Script syntax check + entrypoint discovery integrated in editor UX.
• Blogmark transform pipeline executes Python transform scripts (kind='transform') via a queued worker runtime by default.
• Project preference pythonRuntimeMode added with Settings → Technology section.

Confirmed Deviations from Original Plan

These are current realities and should be treated as authoritative unless we explicitly decide to change them.

1. Transform runtime is now configurable (project-level)

   • Default: webworker (worker-thread based Python runtime with queued requests).
   • Optional fallback: main-thread legacy execution mode.

2. Built-in render-time macros remain JS-based by design

   • Existing JS macro path remains valid (PageRenderer.renderMacro and renderer macro registry/definitions).
   • Python macro support is additive for new macros, not a migration requirement.

3. Macro ABI exists but is not yet wired for additive Python macro creation in the production render path

   • ABI v1 + runtime manager support exist.
   • Main page generation path still uses existing JS macro rendering.

4. Scripts rebuild/sync parity is implemented (simple policy)

   • ScriptEngine.rebuildDatabaseFromFiles() now rebuilds DB metadata from scripts/*.py.
   • ScriptEngine.reconcileScriptsFromGitChanges() now handles added/modified/deleted/renamed script files after git pull.
   • Settings → Data now includes Rebuild Scripts button (scripts:rebuildFromFiles) for manual parity with posts/media rebuild.

Remaining Work Only

1) Python runtime boundary (P0) — Implemented

• Worker model introduced for Blogmark transform execution with queued communication.
• Runtime mode made project-configurable via Settings → Technology (pythonRuntimeMode).
• Legacy main-thread mode retained as explicit fallback option.

2) Add scripts file-system rebuild/sync (P1) — Implemented

• Implement rebuild/meta-diff style synchronization for scripts/ so external file edits are detected.
• Define conflict handling policy between DB metadata and script file frontmatter/body.
• Add tests for create/edit/delete performed outside app while app is closed/open.

Implemented policy (simple)

• Source of truth: script file + docstring frontmatter when present/valid.
• Rebuild path: delete current scripts rows for active project and re-import from scripts/*.py.
• Reconcile path (git pull): apply file deltas (added|modified|deleted|renamed) and upsert/delete rows.
• Conflict behavior: prefer file metadata/body; fall back to safe defaults when values are missing/invalid.

3) Additive Python macro support in render pipeline (P1)

• Add macro-to-script resolution (token/hook -> script id/slug) for Python-backed macros.
• Execute Python macro scripts from the active render path when a macro resolves to a Python script.
• Preserve existing JS macro behavior for built-in/current macros.
• Add explicit fallback rules so unresolved/failed Python macros do not break JS macro rendering.
• Reuse runtime cache keys across repeated Python macro invocations in generation loops.
• Add guardrails for timeout/error fallback during render.

4) Coexistence hardening + tests (P2)

• Add integration tests proving Python-based and JS-based macros can be used together in one post/page.
• Add fixtures/golden tests for mixed macro rendering stability.
• Document precedence/dispatch behavior when macro names overlap (Python script vs JS built-in).

5) Default script seeding/versioning (P2)

• Bundle default scripts in repo/app resources.
• Seed missing defaults on project init/startup deterministically.
• Add versioned update strategy for default scripts.

6) Diagnostics and performance visibility (P3)

• Add macro execution counters (count, timeout/error counts, p50/p95) for real render path.
• Define regression thresholds based on benchmark trends.

Out of Scope Until Core Gaps Close

• AI assistant tooling exposure from Python scripts.
• General Python package/dependency policy expansion.
• Advanced bridge optimizations (only if metrics prove need).

Acceptance Gate Before Marking Python Scripting “Complete”

• Users can create new Python macros that execute in production generation flow.
• Python-based and JS-based macros coexist in production generation flow.
• Scripts directory external changes are synchronized reliably.
• Runtime boundary decision implemented and protected by tests.
• Coexistence/dispatch behavior is documented and covered by tests.
• npm test and npm run build pass.