123 lines
9.1 KiB
Markdown
123 lines
9.1 KiB
Markdown
# Python Scripting — Remaining Work (Implementation-First)
|
|
|
|
Last verified: 26 Feb 2026
|
|
|
|
This document is intentionally reduced to **what is still left to implement**.
|
|
When plan and code differ, code is the source of truth.
|
|
|
|
## Implemented (checked)
|
|
|
|
- [x] Pyodide dependency integrated.
|
|
- [x] Renderer worker runtime exists (`pythonRuntime.worker.ts`) with ready/error/stdout/run protocol.
|
|
- [x] Runtime timeout watchdog + reset/recovery implemented in `PythonRuntimeManager`.
|
|
- [x] Renderer runtime request queueing implemented (concurrent calls are serialized in manager).
|
|
- [x] ABI v1 schemas and validation for macro context/result implemented (`abiV1.ts`).
|
|
- [x] Benchmark harness implemented (`npm run bench:python-runtime -- <iterations>`).
|
|
- [x] Script persistence model implemented (`scripts` DB table + `scripts/*.py` files).
|
|
- [x] Script metadata model implemented (`kind`, `entrypoint`, `enabled`, `version`, etc.).
|
|
- [x] Main process script CRUD engine + IPC handlers implemented.
|
|
- [x] Preload + shared API typings for scripts implemented.
|
|
- [x] Renderer scripts UX implemented (sidebar list, editor, save, run, delete).
|
|
- [x] Script syntax check + entrypoint discovery integrated in editor UX.
|
|
- [x] Blogmark transform pipeline executes Python transform scripts (`kind='transform'`) via a queued worker runtime by default.
|
|
- [x] Project preference `pythonRuntimeMode` added with Settings → Technology section.
|
|
|
|
## Confirmed Deviations from Original Plan
|
|
|
|
These are current realities and should be treated as authoritative unless we explicitly decide to change them.
|
|
|
|
1. **Transform runtime is now configurable (project-level)**
|
|
- Default: `webworker` (worker-thread based Python runtime with queued requests).
|
|
- Optional fallback: `main-thread` legacy execution mode.
|
|
|
|
2. **Built-in render-time macros remain JS-based by design**
|
|
- Existing JS macro path remains valid (`PageRenderer.renderMacro` and renderer macro registry/definitions).
|
|
- Python macro support is additive for new macros, not a migration requirement.
|
|
|
|
3. **Macro ABI is now wired for Python macro execution in both preview and production render paths**
|
|
- ABI v1 + runtime manager support exist.
|
|
- Main page generation path now supports Python macro rendering via `PythonMacroWorkerRuntime`.
|
|
- Renderer preview path supports Python macro rendering via `PythonRuntimeManager`.
|
|
- JS built-in macros always take priority; Python macros only resolve for names not in the JS registry.
|
|
|
|
4. **Scripts rebuild/sync parity is implemented (simple policy)**
|
|
- `ScriptEngine.rebuildDatabaseFromFiles()` now rebuilds DB metadata from `scripts/*.py`.
|
|
- `ScriptEngine.reconcileScriptsFromGitChanges()` now handles added/modified/deleted/renamed script files after git pull.
|
|
- Settings → Data now includes **Rebuild Scripts** button (`scripts:rebuildFromFiles`) for manual parity with posts/media rebuild.
|
|
|
|
## Remaining Work Only
|
|
|
|
## 1) Python runtime boundary (P0) — Implemented
|
|
|
|
- [x] Worker model introduced for Blogmark transform execution with queued communication.
|
|
- [x] Runtime mode made project-configurable via Settings → Technology (`pythonRuntimeMode`).
|
|
- [x] Legacy main-thread mode retained as explicit fallback option.
|
|
|
|
## 2) Add scripts file-system rebuild/sync (P1) — Implemented
|
|
|
|
- [x] Implement rebuild/meta-diff style synchronization for `scripts/` so external file edits are detected.
|
|
- [x] Define conflict handling policy between DB metadata and script file frontmatter/body.
|
|
- [x] Add tests for create/edit/delete performed outside app while app is closed/open.
|
|
|
|
### Implemented policy (simple)
|
|
|
|
- Source of truth: script file + docstring frontmatter when present/valid.
|
|
- Rebuild path: delete current `scripts` rows for active project and re-import from `scripts/*.py`.
|
|
- Reconcile path (git pull): apply file deltas (`added|modified|deleted|renamed`) and upsert/delete rows.
|
|
- Conflict behavior: prefer file metadata/body; fall back to safe defaults when values are missing/invalid.
|
|
|
|
## 3) Additive Python macro support in render pipeline (P1) — Implemented
|
|
|
|
- [x] Add macro-to-script resolution (token/hook -> script id/slug) for Python-backed macros.
|
|
- [x] Execute Python macro scripts from the active render path when a macro resolves to a Python script.
|
|
- [x] Preserve existing JS macro behavior for built-in/current macros.
|
|
- [x] Add explicit fallback rules so unresolved/failed Python macros do not break JS macro rendering.
|
|
- [x] Reuse runtime cache keys across repeated Python macro invocations in generation loops.
|
|
- [x] Add guardrails for timeout/error fallback during render.
|
|
|
|
### Implementation details
|
|
|
|
- **Calling convention**: Python macro entrypoints use a two-argument signature: `def render(context, post)`. `context` is the macro context dict (with `params`, `language`, `post_slug`, etc.). `post` is the full `PostData` dict for the post containing the macro (with `title`, `slug`, `content`, `tags`, `categories`, `createdAt`, etc.), or `None` if post data is unavailable.
|
|
|
|
- **Production path**: `PageRenderer` now accepts an optional `PythonMacroRendererContract`. Macro replacement in the Liquid `markdown` filter is async via `replaceAllMacrosAsync()`. When an unknown macro name is encountered and a Python macro renderer is provided, the renderer resolves enabled macro scripts by slug and executes them via `PythonMacroWorkerRuntime` (Node.js worker thread + Pyodide). Script resolution only occurs when at least one non-built-in macro is present in the content, avoiding overhead for posts with only JS macros. PostData is serialized per-post via `serializePostDataForMacro()` and threaded through the Liquid `markdown` filter as `post_data_json_by_id`.
|
|
|
|
- **Preview path**: The renderer macro registry (`registry.ts`) supports `setPythonMacroResolver()`. When a macro is not found in the JS registry, the Python resolver is consulted. If a matching script is found, the renderer delegates to `PythonMacroRendererFn` which uses the existing `PythonRuntimeManager` (web worker + Pyodide).
|
|
|
|
- **`bds_api` module**: Python macros can call app APIs via the `bds_api` module. In the main-process worker (`PythonMacroWorkerRuntime`), API calls are dispatched via `mainProcessPythonApiInvoker.ts` which routes to engine methods directly. In the renderer worker, API calls go through the existing IPC bridge. The `bds_api` module is auto-generated by `generatePythonApiModuleV1.ts` and installed in both workers at bootstrap.
|
|
|
|
- **Editor macro detection**: The editor macro plugin uses `hasMacro()` to distinguish known macros from unknown ones. `hasMacro()` checks both the JS macro registry and a set of known Python macro slugs fetched via `scripts:getEnabledMacroSlugs` IPC. The slug set is refreshed on startup and whenever scripts change (`BDS_EVENT_SCRIPTS_CHANGED`).
|
|
|
|
- **Precedence**: JS built-in macros (youtube, vimeo, gallery, photo_archive, tag_cloud) always take priority over Python scripts with the same slug. Python macros only activate for names not registered in the JS macro registry.
|
|
|
|
- **Error handling**: Python macro execution errors are caught and result in empty string output, preserving the rest of the document. Script resolution errors are also caught gracefully.
|
|
|
|
- **Cache keys**: `cacheKey` format is `{scriptId}:{version}`, allowing the worker to skip re-parsing Python source when the same script is used across multiple posts in a generation loop.
|
|
|
|
## 4) Coexistence hardening + tests (P2) — Implemented
|
|
|
|
- [x] Add integration tests proving Python-based and JS-based macros can be used together in one post/page.
|
|
- [x] Add fixtures/golden tests for mixed macro rendering stability.
|
|
- [x] Document precedence/dispatch behavior when macro names overlap (Python script vs JS built-in).
|
|
|
|
### Test coverage
|
|
|
|
- `tests/engine/PageRenderer.pythonMacros.test.ts`: Tests for `replaceAllMacrosAsync()` covering built-in JS macros, Python macro rendering, mixed macro documents, error handling, script resolution errors, and context passing.
|
|
- `tests/renderer/macros/pythonMacroCoexistence.test.ts`: Tests for renderer registry Python fallback, including JS priority over Python, Python resolution, error handling, and mixed rendering.
|
|
- `tests/engine/PythonMacroWorkerRuntime.test.ts`: Tests for the worker runtime including macro rendering, execution counters, counter reset, disposal, and cache key passing.
|
|
- `tests/engine/ScriptEngine.test.ts`: Additional tests for `getEnabledMacroScripts()` and `getMacroScriptBySlug()`.
|
|
|
|
## 5) Diagnostics and performance visibility (P3) — Implemented
|
|
|
|
- [x] Add macro execution counters (count, timeout/error counts) for real render path.
|
|
- [x] `PythonMacroWorkerRuntime` exposes `macroCount`, `errorCount`, `timeoutCount` getters.
|
|
- [x] `resetCounters()` method for clean state between generation runs.
|
|
|
|
## Acceptance Gate Before Marking Python Scripting “Complete”
|
|
|
|
- [x] Users can create new Python macros that execute in production generation flow.
|
|
- [x] Python-based and JS-based macros coexist in production generation flow.
|
|
- [x] Scripts directory external changes are synchronized reliably.
|
|
- [x] Runtime boundary decision implemented and protected by tests.
|
|
- [x] Coexistence/dispatch behavior is documented and covered by tests.
|
|
- [x] `npm test` and `npm run build` pass.
|