bDS/PYTHON_SCRIPTING.md

# Python Scripting — Remaining Work (Implementation-First)

Last verified: 24 Feb 2026

This document is intentionally reduced to **what is still left to implement**.
When plan and code differ, code is the source of truth.

## Implemented (checked)

- [x] Pyodide dependency integrated.
- [x] Renderer worker runtime exists (`pythonRuntime.worker.ts`) with ready/error/stdout/run protocol.
- [x] Runtime timeout watchdog + reset/recovery implemented in `PythonRuntimeManager`.
- [x] Renderer runtime request queueing implemented (concurrent calls are serialized in manager).
- [x] ABI v1 schemas and validation for macro context/result implemented (`abiV1.ts`).
- [x] Benchmark harness implemented (`npm run bench:python-runtime -- <iterations>`).
- [x] Script persistence model implemented (`scripts` DB table + `scripts/*.py` files).
- [x] Script metadata model implemented (`kind`, `entrypoint`, `enabled`, `version`, etc.).
- [x] Main process script CRUD engine + IPC handlers implemented.
- [x] Preload + shared API typings for scripts implemented.
- [x] Renderer scripts UX implemented (sidebar list, editor, save, run, delete).
- [x] Script syntax check + entrypoint discovery integrated in editor UX.
- [x] Blogmark transform pipeline executes Python transform scripts (`kind='transform'`) via a queued worker runtime by default.
- [x] Project preference `pythonRuntimeMode` added with Settings → Technology section.

## Confirmed Deviations from Original Plan

These are current realities and should be treated as authoritative unless we explicitly decide to change them.

1. **Transform runtime is now configurable (project-level)**
   - Default: `webworker` (worker-thread based Python runtime with queued requests).
   - Optional fallback: `main-thread` legacy execution mode.

2. **Built-in render-time macros remain JS-based by design**
   - Existing JS macro path remains valid (`PageRenderer.renderMacro` and renderer macro registry/definitions).
   - Python macro support is additive for new macros, not a migration requirement.

3. **Macro ABI exists but is not yet wired for additive Python macro creation in the production render path**
   - ABI v1 + runtime manager support exist.
   - Main page generation path still uses existing JS macro rendering.

4. **Scripts rebuild/sync parity is implemented (simple policy)**
   - `ScriptEngine.rebuildDatabaseFromFiles()` now rebuilds DB metadata from `scripts/*.py`.
   - `ScriptEngine.reconcileScriptsFromGitChanges()` now handles added/modified/deleted/renamed script files after git pull.
   - Settings → Data now includes **Rebuild Scripts** button (`scripts:rebuildFromFiles`) for manual parity with posts/media rebuild.

## Remaining Work Only

## 1) Python runtime boundary (P0) — Implemented

- [x] Worker model introduced for Blogmark transform execution with queued communication.
- [x] Runtime mode made project-configurable via Settings → Technology (`pythonRuntimeMode`).
- [x] Legacy main-thread mode retained as explicit fallback option.

## 2) Add scripts file-system rebuild/sync (P1) — Implemented

- [x] Implement rebuild/meta-diff style synchronization for `scripts/` so external file edits are detected.
- [x] Define conflict handling policy between DB metadata and script file frontmatter/body.
- [x] Add tests for create/edit/delete performed outside app while app is closed/open.

### Implemented policy (simple)

- Source of truth: script file + docstring frontmatter when present/valid.
- Rebuild path: delete current `scripts` rows for active project and re-import from `scripts/*.py`.
- Reconcile path (git pull): apply file deltas (`added|modified|deleted|renamed`) and upsert/delete rows.
- Conflict behavior: prefer file metadata/body; fall back to safe defaults when values are missing/invalid.

## 3) Additive Python macro support in render pipeline (P1)

- [ ] Add macro-to-script resolution (token/hook -> script id/slug) for Python-backed macros.
- [ ] Execute Python macro scripts from the active render path when a macro resolves to a Python script.
- [ ] Preserve existing JS macro behavior for built-in/current macros.
- [ ] Add explicit fallback rules so unresolved/failed Python macros do not break JS macro rendering.
- [ ] Reuse runtime cache keys across repeated Python macro invocations in generation loops.
- [ ] Add guardrails for timeout/error fallback during render.

## 4) Coexistence hardening + tests (P2)

- [ ] Add integration tests proving Python-based and JS-based macros can be used together in one post/page.
- [ ] Add fixtures/golden tests for mixed macro rendering stability.
- [ ] Document precedence/dispatch behavior when macro names overlap (Python script vs JS built-in).

## 5) Default script seeding/versioning (P2)

- [ ] Bundle default scripts in repo/app resources.
- [ ] Seed missing defaults on project init/startup deterministically.
- [ ] Add versioned update strategy for default scripts.

## 6) Diagnostics and performance visibility (P3)

- [ ] Add macro execution counters (count, timeout/error counts, p50/p95) for real render path.
- [ ] Define regression thresholds based on benchmark trends.

## Out of Scope Until Core Gaps Close

- [ ] AI assistant tooling exposure from Python scripts.
- [ ] General Python package/dependency policy expansion.
- [ ] Advanced bridge optimizations (only if metrics prove need).

## Acceptance Gate Before Marking Python Scripting “Complete”

- [ ] Users can create new Python macros that execute in production generation flow.
- [ ] Python-based and JS-based macros coexist in production generation flow.
- [x] Scripts directory external changes are synchronized reliably.
- [x] Runtime boundary decision implemented and protected by tests.
- [ ] Coexistence/dispatch behavior is documented and covered by tests.
- [x] `npm test` and `npm run build` pass.