Merge pull request #18 from rfc1437/feature/python-worker-queued-runtime
Feature/python worker queued runtime
This commit is contained in:
Georg Bauer
@@ -10,6 +10,7 @@ When plan and code differ, code is the source of truth.
|
||||
- [x] Pyodide dependency integrated.
|
||||
- [x] Renderer worker runtime exists (`pythonRuntime.worker.ts`) with ready/error/stdout/run protocol.
|
||||
- [x] Runtime timeout watchdog + reset/recovery implemented in `PythonRuntimeManager`.
|
||||
- [x] Renderer runtime request queueing implemented (concurrent calls are serialized in manager).
|
||||
- [x] ABI v1 schemas and validation for macro context/result implemented (`abiV1.ts`).
|
||||
- [x] Benchmark harness implemented (`npm run bench:python-runtime -- <iterations>`).
|
||||
- [x] Script persistence model implemented (`scripts` DB table + `scripts/*.py` files).
|
||||
@@ -18,15 +19,16 @@ When plan and code differ, code is the source of truth.
|
||||
- [x] Preload + shared API typings for scripts implemented.
|
||||
- [x] Renderer scripts UX implemented (sidebar list, editor, save, run, delete).
|
||||
- [x] Script syntax check + entrypoint discovery integrated in editor UX.
|
||||
- [x] Blogmark transform pipeline executes Python transform scripts (`kind='transform'`).
|
||||
- [x] Blogmark transform pipeline executes Python transform scripts (`kind='transform'`) via a queued worker runtime by default.
|
||||
- [x] Project preference `pythonRuntimeMode` added with Settings → Technology section.
|
||||
|
||||
## Confirmed Deviations from Original Plan
|
||||
|
||||
These are current realities and should be treated as authoritative unless we explicitly decide to change them.
|
||||
|
||||
1. **Transform script runtime location differs**
|
||||
- Original plan: untrusted Python runs in renderer worker only.
|
||||
- Actual implementation: Blogmark transform scripts run in **main process** Pyodide (`BlogmarkTransformService`).
|
||||
1. **Transform runtime is now configurable (project-level)**
|
||||
- Default: `webworker` (worker-thread based Python runtime with queued requests).
|
||||
- Optional fallback: `main-thread` legacy execution mode.
|
||||
|
||||
2. **Render-time macro migration has not happened yet**
|
||||
- Original plan: all render-time macros become Python-backed.
|
||||
@@ -36,24 +38,31 @@ These are current realities and should be treated as authoritative unless we exp
|
||||
- ABI v1 + runtime manager support exist.
|
||||
- Main page generation path still uses existing JS macro rendering.
|
||||
|
||||
4. **Scripts rebuild/meta-diff sync is still missing**
|
||||
- Script CRUD works via app APIs.
|
||||
- No implemented project-wide “rebuild from files” parity for `scripts/` equivalent to posts/media rebuild flows.
|
||||
4. **Scripts rebuild/sync parity is implemented (simple policy)**
|
||||
- `ScriptEngine.rebuildDatabaseFromFiles()` now rebuilds DB metadata from `scripts/*.py`.
|
||||
- `ScriptEngine.reconcileScriptsFromGitChanges()` now handles added/modified/deleted/renamed script files after git pull.
|
||||
- Settings → Data now includes **Rebuild Scripts** button (`scripts:rebuildFromFiles`) for manual parity with posts/media rebuild.
|
||||
|
||||
## Remaining Work Only
|
||||
|
||||
## 1) Decide and enforce Python runtime boundary (P0)
|
||||
## 1) Python runtime boundary (P0) — Implemented
|
||||
|
||||
- [ ] Decide if `transform` scripts should stay in main process or move to renderer worker.
|
||||
- [ ] If staying in main process: add explicit timeout/kill/recovery safeguards equivalent to worker watchdog behavior.
|
||||
- [ ] If moving to worker: route transform execution through typed IPC/worker bridge and remove main-process execution path.
|
||||
- [ ] Document final security model in this file after decision.
|
||||
- [x] Worker model introduced for Blogmark transform execution with queued communication.
|
||||
- [x] Runtime mode made project-configurable via Settings → Technology (`pythonRuntimeMode`).
|
||||
- [x] Legacy main-thread mode retained as explicit fallback option.
|
||||
|
||||
## 2) Add scripts file-system rebuild/sync (P1)
|
||||
## 2) Add scripts file-system rebuild/sync (P1) — Implemented
|
||||
|
||||
- [ ] Implement rebuild/meta-diff style synchronization for `scripts/` so external file edits are detected.
|
||||
- [ ] Define conflict handling policy between DB metadata and script file frontmatter/body.
|
||||
- [ ] Add tests for create/edit/delete performed outside app while app is closed/open.
|
||||
- [x] Implement rebuild/meta-diff style synchronization for `scripts/` so external file edits are detected.
|
||||
- [x] Define conflict handling policy between DB metadata and script file frontmatter/body.
|
||||
- [x] Add tests for create/edit/delete performed outside app while app is closed/open.
|
||||
|
||||
### Implemented policy (simple)
|
||||
|
||||
- Source of truth: script file + docstring frontmatter when present/valid.
|
||||
- Rebuild path: delete current `scripts` rows for active project and re-import from `scripts/*.py`.
|
||||
- Reconcile path (git pull): apply file deltas (`added|modified|deleted|renamed`) and upsert/delete rows.
|
||||
- Conflict behavior: prefer file metadata/body; fall back to safe defaults when values are missing/invalid.
|
||||
|
||||
## 3) Wire Python macros into render pipeline (P1)
|
||||
|
||||
@@ -88,7 +97,7 @@ These are current realities and should be treated as authoritative unless we exp
|
||||
## Acceptance Gate Before Marking Python Scripting “Complete”
|
||||
|
||||
- [ ] Render-time macros run through Python script path in production generation flow.
|
||||
- [ ] Scripts directory external changes are synchronized reliably.
|
||||
- [ ] Runtime boundary decision implemented and protected by tests.
|
||||
- [x] Scripts directory external changes are synchronized reliably.
|
||||
- [x] Runtime boundary decision implemented and protected by tests.
|
||||
- [ ] Legacy JS macro path removed (or explicitly retained with documented rationale).
|
||||
- [ ] `npm test` and `npm run build` pass.
|
||||
- [x] `npm test` and `npm run build` pass.
|
||||
|
||||
261
src/main/engine/BlogmarkPythonWorkerRuntime.ts
Normal file
261
src/main/engine/BlogmarkPythonWorkerRuntime.ts
Normal file
@@ -0,0 +1,261 @@
|
||||
import * as path from 'path';
|
||||
import { Worker } from 'worker_threads';
|
||||
|
||||
interface WorkerRunTransformRequest {
|
||||
type: 'runTransform';
|
||||
requestId: string;
|
||||
scriptContent: string;
|
||||
entrypoint: string;
|
||||
payloadJson: string;
|
||||
}
|
||||
|
||||
interface WorkerReadyMessage {
|
||||
type: 'ready';
|
||||
}
|
||||
|
||||
interface WorkerResultMessage {
|
||||
type: 'transformResult';
|
||||
requestId: string;
|
||||
output: unknown;
|
||||
toasts: string[];
|
||||
}
|
||||
|
||||
interface WorkerErrorMessage {
|
||||
type: 'transformError';
|
||||
requestId: string;
|
||||
error: string;
|
||||
}
|
||||
|
||||
interface WorkerFatalErrorMessage {
|
||||
type: 'error';
|
||||
error: string;
|
||||
}
|
||||
|
||||
type WorkerResponseMessage = WorkerReadyMessage | WorkerResultMessage | WorkerErrorMessage | WorkerFatalErrorMessage;
|
||||
|
||||
interface QueuedRequest {
|
||||
request: WorkerRunTransformRequest;
|
||||
timeoutMs: number;
|
||||
resolve: (value: { output: unknown; toasts: string[] }) => void;
|
||||
reject: (error: Error) => void;
|
||||
}
|
||||
|
||||
interface ActiveRequest extends QueuedRequest {
|
||||
timeoutId: ReturnType<typeof setTimeout> | null;
|
||||
}
|
||||
|
||||
export class BlogmarkPythonWorkerRuntime {
|
||||
private worker: Worker | null = null;
|
||||
private workerReady = false;
|
||||
private workerStartPromise: Promise<void> | null = null;
|
||||
private workerStartResolve: (() => void) | null = null;
|
||||
private workerStartReject: ((error: Error) => void) | null = null;
|
||||
private activeRequest: ActiveRequest | null = null;
|
||||
private queue: QueuedRequest[] = [];
|
||||
private requestCounter = 0;
|
||||
|
||||
async executeTransform(params: {
|
||||
scriptContent: string;
|
||||
entrypoint: string;
|
||||
payloadJson: string;
|
||||
timeoutMs?: number;
|
||||
}): Promise<{ output: unknown; toasts: string[] }> {
|
||||
const requestId = this.nextRequestId();
|
||||
const timeoutMs = params.timeoutMs ?? 5000;
|
||||
|
||||
return new Promise<{ output: unknown; toasts: string[] }>((resolve, reject) => {
|
||||
this.queue.push({
|
||||
request: {
|
||||
type: 'runTransform',
|
||||
requestId,
|
||||
scriptContent: params.scriptContent,
|
||||
entrypoint: params.entrypoint,
|
||||
payloadJson: params.payloadJson,
|
||||
},
|
||||
timeoutMs,
|
||||
resolve,
|
||||
reject,
|
||||
});
|
||||
|
||||
this.dispatchNext().catch((error) => {
|
||||
reject(error instanceof Error ? error : new Error(String(error)));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
dispose(): void {
|
||||
this.rejectStartPromise(new Error('Python worker runtime disposed'));
|
||||
this.rejectActiveAndQueue(new Error('Python worker runtime disposed'));
|
||||
this.resetWorker();
|
||||
}
|
||||
|
||||
private async dispatchNext(): Promise<void> {
|
||||
if (this.activeRequest || this.queue.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
await this.ensureWorkerStarted();
|
||||
|
||||
const nextRequest = this.queue.shift();
|
||||
if (!nextRequest) {
|
||||
return;
|
||||
}
|
||||
|
||||
const timeoutId = setTimeout(() => {
|
||||
if (!this.activeRequest || this.activeRequest.request.requestId !== nextRequest.request.requestId) {
|
||||
return;
|
||||
}
|
||||
|
||||
const timeoutError = new Error(`Python transform timed out after ${nextRequest.timeoutMs}ms`);
|
||||
this.activeRequest.reject(timeoutError);
|
||||
this.activeRequest = null;
|
||||
this.resetWorker();
|
||||
void this.dispatchNext();
|
||||
}, nextRequest.timeoutMs);
|
||||
|
||||
this.activeRequest = {
|
||||
...nextRequest,
|
||||
timeoutId,
|
||||
};
|
||||
|
||||
this.worker?.postMessage(nextRequest.request);
|
||||
}
|
||||
|
||||
private async ensureWorkerStarted(): Promise<void> {
|
||||
if (this.worker && this.workerReady) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (this.workerStartPromise) {
|
||||
return this.workerStartPromise;
|
||||
}
|
||||
|
||||
const workerPath = path.join(__dirname, 'blogmarkPython.worker.js');
|
||||
this.worker = new Worker(workerPath);
|
||||
this.workerReady = false;
|
||||
|
||||
this.worker.on('message', (message: WorkerResponseMessage) => {
|
||||
this.handleWorkerMessage(message);
|
||||
});
|
||||
|
||||
this.worker.on('error', (error) => {
|
||||
this.handleWorkerCrash(error instanceof Error ? error : new Error(String(error)));
|
||||
});
|
||||
|
||||
this.worker.on('exit', (code) => {
|
||||
if (code !== 0) {
|
||||
this.handleWorkerCrash(new Error(`Python worker exited with code ${code}`));
|
||||
}
|
||||
});
|
||||
|
||||
this.workerStartPromise = new Promise<void>((resolve, reject) => {
|
||||
this.workerStartResolve = resolve;
|
||||
this.workerStartReject = reject;
|
||||
});
|
||||
|
||||
return this.workerStartPromise;
|
||||
}
|
||||
|
||||
private handleWorkerMessage(message: WorkerResponseMessage): void {
|
||||
if (message.type === 'ready') {
|
||||
this.workerReady = true;
|
||||
this.resolveStartPromise();
|
||||
return;
|
||||
}
|
||||
|
||||
if (message.type === 'error') {
|
||||
this.handleWorkerCrash(new Error(message.error));
|
||||
return;
|
||||
}
|
||||
|
||||
const active = this.activeRequest;
|
||||
if (!active) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (active.request.requestId !== message.requestId) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (active.timeoutId) {
|
||||
clearTimeout(active.timeoutId);
|
||||
}
|
||||
|
||||
this.activeRequest = null;
|
||||
|
||||
if (message.type === 'transformResult') {
|
||||
active.resolve({ output: message.output, toasts: message.toasts });
|
||||
} else {
|
||||
active.reject(new Error(message.error));
|
||||
}
|
||||
|
||||
void this.dispatchNext();
|
||||
}
|
||||
|
||||
private handleWorkerCrash(error: Error): void {
|
||||
this.rejectStartPromise(error);
|
||||
this.rejectActiveAndQueue(error);
|
||||
this.resetWorker();
|
||||
}
|
||||
|
||||
private rejectActiveAndQueue(error: Error): void {
|
||||
if (this.activeRequest) {
|
||||
if (this.activeRequest.timeoutId) {
|
||||
clearTimeout(this.activeRequest.timeoutId);
|
||||
}
|
||||
this.activeRequest.reject(error);
|
||||
this.activeRequest = null;
|
||||
}
|
||||
|
||||
while (this.queue.length > 0) {
|
||||
const queued = this.queue.shift();
|
||||
queued?.reject(error);
|
||||
}
|
||||
}
|
||||
|
||||
private resolveStartPromise(): void {
|
||||
if (this.workerStartResolve) {
|
||||
this.workerStartResolve();
|
||||
}
|
||||
this.workerStartResolve = null;
|
||||
this.workerStartReject = null;
|
||||
this.workerStartPromise = null;
|
||||
}
|
||||
|
||||
private rejectStartPromise(error: Error): void {
|
||||
if (this.workerStartReject) {
|
||||
this.workerStartReject(error);
|
||||
}
|
||||
this.workerStartResolve = null;
|
||||
this.workerStartReject = null;
|
||||
this.workerStartPromise = null;
|
||||
}
|
||||
|
||||
private resetWorker(): void {
|
||||
if (this.worker) {
|
||||
this.worker.removeAllListeners();
|
||||
this.worker.terminate();
|
||||
}
|
||||
|
||||
this.worker = null;
|
||||
this.workerReady = false;
|
||||
this.workerStartPromise = null;
|
||||
this.workerStartResolve = null;
|
||||
this.workerStartReject = null;
|
||||
}
|
||||
|
||||
private nextRequestId(): string {
|
||||
this.requestCounter += 1;
|
||||
return `blogmark-py-${this.requestCounter}`;
|
||||
}
|
||||
}
|
||||
|
||||
let blogmarkPythonWorkerRuntimeInstance: BlogmarkPythonWorkerRuntime | null = null;
|
||||
|
||||
export function getBlogmarkPythonWorkerRuntime(): BlogmarkPythonWorkerRuntime {
|
||||
if (!blogmarkPythonWorkerRuntimeInstance) {
|
||||
blogmarkPythonWorkerRuntimeInstance = new BlogmarkPythonWorkerRuntime();
|
||||
}
|
||||
|
||||
return blogmarkPythonWorkerRuntimeInstance;
|
||||
}
|
||||
@@ -1,5 +1,7 @@
|
||||
import { z } from 'zod';
|
||||
import { getScriptEngine } from './ScriptEngine';
|
||||
import { getMetaEngine } from './MetaEngine';
|
||||
import { getBlogmarkPythonWorkerRuntime } from './BlogmarkPythonWorkerRuntime';
|
||||
|
||||
const transformPostSchema = z.object({
|
||||
title: z.string().trim().min(1),
|
||||
@@ -55,6 +57,8 @@ export interface BlogmarkTransformResult {
|
||||
toasts: string[];
|
||||
}
|
||||
|
||||
export type PythonRuntimeMode = 'webworker' | 'main-thread';
|
||||
|
||||
const MAX_TOASTS_PER_SCRIPT = 5;
|
||||
const MAX_TOASTS_TOTAL = 20;
|
||||
const MAX_TOAST_LENGTH = 300;
|
||||
@@ -142,6 +146,28 @@ function toErrorMessage(error: unknown): string {
|
||||
return String(error);
|
||||
}
|
||||
|
||||
function resolveTransformEntrypoint(value: string): string {
|
||||
const nextEntrypoint = typeof value === 'string' ? value.trim() : '';
|
||||
if (/^[A-Za-z_][A-Za-z0-9_]*$/.test(nextEntrypoint) && nextEntrypoint !== 'main') {
|
||||
return nextEntrypoint;
|
||||
}
|
||||
|
||||
return 'transform';
|
||||
}
|
||||
|
||||
function resolvePythonRuntimeMode(value: unknown): PythonRuntimeMode {
|
||||
if (value === 'main-thread') {
|
||||
return 'main-thread';
|
||||
}
|
||||
|
||||
return 'webworker';
|
||||
}
|
||||
|
||||
async function getConfiguredPythonRuntimeMode(): Promise<PythonRuntimeMode> {
|
||||
const metadata = await getMetaEngine().getProjectMetadata();
|
||||
return resolvePythonRuntimeMode((metadata as { pythonRuntimeMode?: unknown } | null)?.pythonRuntimeMode);
|
||||
}
|
||||
|
||||
class PythonBlogmarkTransformExecutor implements BlogmarkTransformExecutor {
|
||||
private runtimePromise: Promise<any> | null = null;
|
||||
|
||||
@@ -169,7 +195,7 @@ def toast(message):
|
||||
|
||||
await runtime.runPythonAsync(script.content);
|
||||
|
||||
const requestedEntrypoint = this.resolveEntrypoint(script.entrypoint);
|
||||
const requestedEntrypoint = resolveTransformEntrypoint(script.entrypoint);
|
||||
const payload = JSON.stringify(input);
|
||||
runtime.globals.set('__bds_transform_payload_json', payload);
|
||||
runtime.globals.set('__bds_transform_entrypoint', requestedEntrypoint);
|
||||
@@ -200,15 +226,6 @@ json.dumps(_result)
|
||||
};
|
||||
}
|
||||
|
||||
private resolveEntrypoint(value: string): string {
|
||||
const nextEntrypoint = typeof value === 'string' ? value.trim() : '';
|
||||
if (/^[A-Za-z_][A-Za-z0-9_]*$/.test(nextEntrypoint) && nextEntrypoint !== 'main') {
|
||||
return nextEntrypoint;
|
||||
}
|
||||
|
||||
return 'transform';
|
||||
}
|
||||
|
||||
private async getRuntime(): Promise<any> {
|
||||
if (!this.runtimePromise) {
|
||||
this.runtimePromise = (async () => {
|
||||
@@ -221,11 +238,26 @@ json.dumps(_result)
|
||||
}
|
||||
}
|
||||
|
||||
class PythonWorkerBlogmarkTransformExecutor implements BlogmarkTransformExecutor {
|
||||
async runTransform(script: BlogmarkTransformScriptRecord, input: BlogmarkTransformInput): Promise<unknown> {
|
||||
return getBlogmarkPythonWorkerRuntime().executeTransform({
|
||||
scriptContent: script.content,
|
||||
entrypoint: resolveTransformEntrypoint(script.entrypoint),
|
||||
payloadJson: JSON.stringify(input),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const mainThreadExecutor = new PythonBlogmarkTransformExecutor();
|
||||
const workerExecutor = new PythonWorkerBlogmarkTransformExecutor();
|
||||
|
||||
export class BlogmarkTransformService {
|
||||
constructor(
|
||||
private readonly dependencies: {
|
||||
provider?: BlogmarkTransformScriptProvider;
|
||||
executor?: BlogmarkTransformExecutor;
|
||||
resolvePythonRuntimeMode?: () => Promise<PythonRuntimeMode>;
|
||||
executors?: Partial<Record<PythonRuntimeMode, BlogmarkTransformExecutor>>;
|
||||
} = {},
|
||||
) {}
|
||||
|
||||
@@ -237,7 +269,7 @@ export class BlogmarkTransformService {
|
||||
};
|
||||
|
||||
const provider = this.dependencies.provider ?? scriptEngineBackedProvider;
|
||||
const executor = this.dependencies.executor ?? new PythonBlogmarkTransformExecutor();
|
||||
const executor = this.dependencies.executor ?? await this.resolveExecutorForConfiguredRuntime();
|
||||
|
||||
const scripts = await provider.getScripts();
|
||||
const activeTransforms = scripts
|
||||
@@ -303,6 +335,18 @@ export class BlogmarkTransformService {
|
||||
toasts,
|
||||
};
|
||||
}
|
||||
|
||||
private async resolveExecutorForConfiguredRuntime(): Promise<BlogmarkTransformExecutor> {
|
||||
const resolveMode = this.dependencies.resolvePythonRuntimeMode ?? getConfiguredPythonRuntimeMode;
|
||||
const mode = await resolveMode();
|
||||
const executors = this.dependencies.executors ?? {};
|
||||
|
||||
if (mode === 'main-thread') {
|
||||
return executors['main-thread'] ?? mainThreadExecutor;
|
||||
}
|
||||
|
||||
return executors.webworker ?? workerExecutor;
|
||||
}
|
||||
}
|
||||
|
||||
let blogmarkTransformServiceInstance: BlogmarkTransformService | null = null;
|
||||
|
||||
@@ -140,6 +140,14 @@ export interface GitPostFileChange {
|
||||
previousPath?: string;
|
||||
}
|
||||
|
||||
export type GitScriptFileChangeStatus = 'added' | 'modified' | 'deleted' | 'renamed';
|
||||
|
||||
export interface GitScriptFileChange {
|
||||
status: GitScriptFileChangeStatus;
|
||||
path: string;
|
||||
previousPath?: string;
|
||||
}
|
||||
|
||||
type GitProvider = 'unknown' | 'github' | 'gitlab' | 'gitea-forgejo';
|
||||
|
||||
let gitEngineInstance: GitEngine | null = null;
|
||||
@@ -526,7 +534,12 @@ export class GitEngine {
|
||||
return this.markdownExtensions.has(extension);
|
||||
}
|
||||
|
||||
private parseNameStatusOutput(raw: string): GitPostFileChange[] {
|
||||
private isScriptsPythonPath(value: string): boolean {
|
||||
const normalized = this.normalizeRepoRelativePath(value);
|
||||
return normalized.startsWith('scripts/') && path.extname(normalized).toLowerCase() === '.py';
|
||||
}
|
||||
|
||||
private parseNameStatusOutput(raw: string, pathMatcher: (value: string) => boolean): GitPostFileChange[] {
|
||||
const tokens = raw.split('\0').filter((token) => token.length > 0);
|
||||
const changes: GitPostFileChange[] = [];
|
||||
|
||||
@@ -543,7 +556,7 @@ export class GitEngine {
|
||||
const previousPath = this.normalizeRepoRelativePath(previousPathRaw);
|
||||
const pathValue = this.normalizeRepoRelativePath(nextPathRaw);
|
||||
|
||||
if (this.isPostsMarkdownPath(previousPath) || this.isPostsMarkdownPath(pathValue)) {
|
||||
if (pathMatcher(previousPath) || pathMatcher(pathValue)) {
|
||||
changes.push({
|
||||
status: 'renamed',
|
||||
path: pathValue,
|
||||
@@ -555,7 +568,7 @@ export class GitEngine {
|
||||
|
||||
const filePathRaw = tokens[index++] ?? '';
|
||||
const filePath = this.normalizeRepoRelativePath(filePathRaw);
|
||||
if (!this.isPostsMarkdownPath(filePath)) {
|
||||
if (!pathMatcher(filePath)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -1338,13 +1351,40 @@ export class GitEngine {
|
||||
|
||||
try {
|
||||
const output = await git.raw(args);
|
||||
return this.parseNameStatusOutput(output);
|
||||
return this.parseNameStatusOutput(output, (value) => this.isPostsMarkdownPath(value));
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error ?? '');
|
||||
if (this.isSpawnBadFileDescriptorError(message)) {
|
||||
try {
|
||||
const output = await this.runGitCli(projectPath, args);
|
||||
return this.parseNameStatusOutput(output);
|
||||
return this.parseNameStatusOutput(output, (value) => this.isPostsMarkdownPath(value));
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
async getChangedScriptFilesBetween(projectPath: string, fromCommit: string, toCommit: string): Promise<GitScriptFileChange[]> {
|
||||
const fromRef = fromCommit.trim();
|
||||
const toRef = toCommit.trim();
|
||||
if (!fromRef || !toRef || fromRef === toRef) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const git = this.createNonInteractiveGit(projectPath);
|
||||
const args = ['diff', '--name-status', '--find-renames', '-z', `${fromRef}..${toRef}`, '--', 'scripts'];
|
||||
|
||||
try {
|
||||
const output = await git.raw(args);
|
||||
return this.parseNameStatusOutput(output, (value) => this.isScriptsPythonPath(value));
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error ?? '');
|
||||
if (this.isSpawnBadFileDescriptorError(message)) {
|
||||
try {
|
||||
const output = await this.runGitCli(projectPath, args);
|
||||
return this.parseNameStatusOutput(output, (value) => this.isScriptsPythonPath(value));
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
|
||||
@@ -24,6 +24,7 @@ export interface ProjectMetadata {
|
||||
defaultAuthor?: string; // Default author for new posts and media
|
||||
maxPostsPerPage?: number; // Preview/server maximum posts per page (default 50)
|
||||
blogmarkCategory?: string; // Category used for externally captured bookmark posts
|
||||
pythonRuntimeMode?: 'webworker' | 'main-thread'; // Runtime mode for Python script execution
|
||||
picoTheme?: PicoThemeName; // Selected Pico CSS theme for preview/rendering
|
||||
categoryMetadata?: Record<string, CategoryMetadata>; // Per-category metadata for UI/rendering
|
||||
categorySettings?: Record<string, CategoryRenderSettings>; // Per-category list rendering preferences
|
||||
@@ -85,6 +86,7 @@ function normalizeProjectMetadata(metadata: ProjectMetadata): ProjectMetadata {
|
||||
const blogmarkCategory = typeof metadata.blogmarkCategory === 'string'
|
||||
? normalizeNonEmptyTaxonomyTerm(metadata.blogmarkCategory) ?? undefined
|
||||
: undefined;
|
||||
const pythonRuntimeMode = metadata.pythonRuntimeMode === 'main-thread' ? 'main-thread' : 'webworker';
|
||||
const picoTheme = sanitizePicoTheme(metadata.picoTheme);
|
||||
const categoryMetadata = normalizeCategoryMetadata(metadata.categoryMetadata ?? metadata.categorySettings);
|
||||
return {
|
||||
@@ -92,6 +94,7 @@ function normalizeProjectMetadata(metadata: ProjectMetadata): ProjectMetadata {
|
||||
publicUrl,
|
||||
maxPostsPerPage,
|
||||
blogmarkCategory,
|
||||
pythonRuntimeMode,
|
||||
picoTheme,
|
||||
categoryMetadata,
|
||||
categorySettings: undefined,
|
||||
@@ -306,6 +309,7 @@ export class MetaEngine extends EventEmitter {
|
||||
defaultAuthor: normalizedUpdates.defaultAuthor,
|
||||
maxPostsPerPage: normalizedUpdates.maxPostsPerPage,
|
||||
blogmarkCategory: normalizedUpdates.blogmarkCategory,
|
||||
pythonRuntimeMode: normalizedUpdates.pythonRuntimeMode,
|
||||
picoTheme: normalizedUpdates.picoTheme,
|
||||
categoryMetadata: normalizedUpdates.categoryMetadata,
|
||||
});
|
||||
|
||||
@@ -42,6 +42,37 @@ export interface UpdateScriptInput {
|
||||
enabled?: boolean;
|
||||
}
|
||||
|
||||
export type GitScriptFileChangeStatus = 'added' | 'modified' | 'deleted' | 'renamed';
|
||||
|
||||
export interface GitScriptFileChange {
|
||||
status: GitScriptFileChangeStatus;
|
||||
path: string;
|
||||
previousPath?: string;
|
||||
}
|
||||
|
||||
export interface ScriptReconcileResult {
|
||||
created: number;
|
||||
updated: number;
|
||||
deleted: number;
|
||||
processedFiles: number;
|
||||
}
|
||||
|
||||
interface ParsedScriptFile {
|
||||
metadata: {
|
||||
id?: string;
|
||||
projectId?: string;
|
||||
slug?: string;
|
||||
title?: string;
|
||||
kind?: string;
|
||||
entrypoint?: string;
|
||||
enabled?: boolean;
|
||||
version?: number;
|
||||
createdAt?: string;
|
||||
updatedAt?: string;
|
||||
};
|
||||
body: string;
|
||||
}
|
||||
|
||||
export class ScriptEngine extends EventEmitter {
|
||||
private currentProjectId = 'default';
|
||||
private dataDir: string | null = null;
|
||||
@@ -191,6 +222,205 @@ export class ScriptEngine extends EventEmitter {
|
||||
return Promise.all(rows.map((item) => this.toScriptData(item)));
|
||||
}
|
||||
|
||||
async rebuildDatabaseFromFiles(): Promise<void> {
|
||||
const db = getDatabase().getLocal();
|
||||
const scriptsDir = this.getScriptsDir();
|
||||
|
||||
await db.delete(scripts).where(eq(scripts.projectId, this.currentProjectId));
|
||||
|
||||
const pythonFiles = await this.scanScriptFiles(scriptsDir);
|
||||
if (pythonFiles.length === 0) {
|
||||
this.emit('scriptsRebuilt');
|
||||
return;
|
||||
}
|
||||
|
||||
const usedIds = new Set<string>();
|
||||
const insertedRows: Script[] = [];
|
||||
|
||||
for (const filePath of pythonFiles) {
|
||||
const parsed = await this.readScriptFileWithMetadata(filePath);
|
||||
if (!parsed) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const desiredSlug = this.normalizeSlug(parsed.metadata.slug || path.basename(filePath, '.py'));
|
||||
const slug = this.ensureUniqueSlug(desiredSlug, insertedRows);
|
||||
|
||||
const desiredId = typeof parsed.metadata.id === 'string' && parsed.metadata.id.trim().length > 0
|
||||
? parsed.metadata.id.trim()
|
||||
: uuidv4();
|
||||
const id = usedIds.has(desiredId) ? uuidv4() : desiredId;
|
||||
|
||||
const now = new Date();
|
||||
const row: NewScript = {
|
||||
id,
|
||||
projectId: this.currentProjectId,
|
||||
slug,
|
||||
title: this.normalizeTitle(parsed.metadata.title, slug),
|
||||
kind: this.normalizeKind(parsed.metadata.kind),
|
||||
entrypoint: this.normalizeEntrypoint(parsed.metadata.entrypoint),
|
||||
enabled: this.normalizeEnabled(parsed.metadata.enabled),
|
||||
version: this.normalizeVersion(parsed.metadata.version),
|
||||
filePath,
|
||||
createdAt: this.normalizeDate(parsed.metadata.createdAt, now),
|
||||
updatedAt: this.normalizeDate(parsed.metadata.updatedAt, now),
|
||||
};
|
||||
|
||||
await db.insert(scripts).values(row);
|
||||
insertedRows.push(row as Script);
|
||||
usedIds.add(id);
|
||||
}
|
||||
|
||||
this.emit('scriptsRebuilt');
|
||||
}
|
||||
|
||||
async reconcileScriptsFromGitChanges(projectPath: string, changes: GitScriptFileChange[]): Promise<ScriptReconcileResult> {
|
||||
const db = getDatabase().getLocal();
|
||||
const normalizedProjectPath = path.resolve(projectPath);
|
||||
|
||||
const relevantChanges = changes.filter((change) => {
|
||||
if (!this.isPythonScriptPath(change.path)) {
|
||||
return false;
|
||||
}
|
||||
if (change.status === 'renamed' && change.previousPath && !this.isPythonScriptPath(change.previousPath) && !this.isPythonScriptPath(change.path)) {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
});
|
||||
|
||||
if (relevantChanges.length === 0) {
|
||||
return { created: 0, updated: 0, deleted: 0, processedFiles: 0 };
|
||||
}
|
||||
|
||||
const scriptRows = await this.getAllScriptRows();
|
||||
const scriptsByPath = new Map<string, Script>();
|
||||
for (const row of scriptRows) {
|
||||
scriptsByPath.set(this.normalizePathForCompare(row.filePath), row);
|
||||
}
|
||||
|
||||
let created = 0;
|
||||
let updated = 0;
|
||||
let deleted = 0;
|
||||
let processedFiles = 0;
|
||||
|
||||
for (const change of relevantChanges) {
|
||||
const absolutePath = this.normalizePathForCompare(path.resolve(normalizedProjectPath, change.path));
|
||||
const previousAbsolutePath = change.previousPath
|
||||
? this.normalizePathForCompare(path.resolve(normalizedProjectPath, change.previousPath))
|
||||
: null;
|
||||
|
||||
if (change.status === 'deleted') {
|
||||
const existing = scriptsByPath.get(absolutePath);
|
||||
if (!existing) {
|
||||
continue;
|
||||
}
|
||||
|
||||
await db.delete(scripts).where(and(eq(scripts.id, existing.id), eq(scripts.projectId, this.currentProjectId)));
|
||||
scriptsByPath.delete(absolutePath);
|
||||
this.emit('scriptDeleted', existing.id);
|
||||
deleted += 1;
|
||||
processedFiles += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
let existing = previousAbsolutePath
|
||||
? (scriptsByPath.get(previousAbsolutePath) || scriptsByPath.get(absolutePath))
|
||||
: scriptsByPath.get(absolutePath);
|
||||
|
||||
const parsed = await this.readScriptFileWithMetadata(absolutePath);
|
||||
if (!parsed) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const allRows = await this.getAllScriptRows();
|
||||
const parsedId = typeof parsed.metadata.id === 'string' ? parsed.metadata.id.trim() : '';
|
||||
if (!existing && parsedId.length > 0) {
|
||||
const byId = allRows.find((row) => row.id === parsedId);
|
||||
if (byId) {
|
||||
existing = byId;
|
||||
}
|
||||
}
|
||||
const desiredSlug = this.normalizeSlug(parsed.metadata.slug || path.basename(absolutePath, '.py'));
|
||||
const slug = this.ensureUniqueSlug(desiredSlug, allRows, existing?.id);
|
||||
|
||||
if (existing) {
|
||||
const updateNow = new Date();
|
||||
const nextRow = {
|
||||
title: this.normalizeTitle(parsed.metadata.title, slug, existing.title),
|
||||
slug,
|
||||
kind: this.normalizeKind(parsed.metadata.kind, existing.kind),
|
||||
entrypoint: this.normalizeEntrypoint(parsed.metadata.entrypoint, existing.entrypoint),
|
||||
enabled: this.normalizeEnabled(parsed.metadata.enabled, existing.enabled),
|
||||
version: this.normalizeVersion(parsed.metadata.version, existing.version),
|
||||
filePath: absolutePath,
|
||||
createdAt: this.normalizeDate(parsed.metadata.createdAt, existing.createdAt),
|
||||
updatedAt: this.normalizeDate(parsed.metadata.updatedAt, updateNow),
|
||||
};
|
||||
|
||||
await db.update(scripts)
|
||||
.set(nextRow)
|
||||
.where(and(eq(scripts.id, existing.id), eq(scripts.projectId, this.currentProjectId)));
|
||||
|
||||
const updatedRow = await this.getScriptRow(existing.id);
|
||||
if (updatedRow) {
|
||||
const updatedScript = await this.toScriptData(updatedRow);
|
||||
this.emit('scriptUpdated', updatedScript);
|
||||
}
|
||||
|
||||
if (previousAbsolutePath) {
|
||||
scriptsByPath.delete(previousAbsolutePath);
|
||||
}
|
||||
scriptsByPath.set(absolutePath, {
|
||||
...existing,
|
||||
...nextRow,
|
||||
});
|
||||
updated += 1;
|
||||
processedFiles += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
const desiredId = typeof parsed.metadata.id === 'string' && parsed.metadata.id.trim().length > 0
|
||||
? parsed.metadata.id.trim()
|
||||
: uuidv4();
|
||||
const idExists = allRows.some((row) => row.id === desiredId);
|
||||
const rowId = idExists ? uuidv4() : desiredId;
|
||||
const now = new Date();
|
||||
|
||||
const newRow: NewScript = {
|
||||
id: rowId,
|
||||
projectId: this.currentProjectId,
|
||||
slug,
|
||||
title: this.normalizeTitle(parsed.metadata.title, slug),
|
||||
kind: this.normalizeKind(parsed.metadata.kind),
|
||||
entrypoint: this.normalizeEntrypoint(parsed.metadata.entrypoint),
|
||||
enabled: this.normalizeEnabled(parsed.metadata.enabled),
|
||||
version: this.normalizeVersion(parsed.metadata.version),
|
||||
filePath: absolutePath,
|
||||
createdAt: this.normalizeDate(parsed.metadata.createdAt, now),
|
||||
updatedAt: this.normalizeDate(parsed.metadata.updatedAt, now),
|
||||
};
|
||||
|
||||
await db.insert(scripts).values(newRow);
|
||||
|
||||
const createdRow = await this.getScriptRow(newRow.id);
|
||||
if (createdRow) {
|
||||
const createdScript = await this.toScriptData(createdRow);
|
||||
this.emit('scriptCreated', createdScript);
|
||||
}
|
||||
|
||||
scriptsByPath.set(absolutePath, newRow as Script);
|
||||
created += 1;
|
||||
processedFiles += 1;
|
||||
}
|
||||
|
||||
return {
|
||||
created,
|
||||
updated,
|
||||
deleted,
|
||||
processedFiles,
|
||||
};
|
||||
}
|
||||
|
||||
private async getScriptRow(id: string): Promise<Script | null> {
|
||||
const rows = await this.getAllScriptRows();
|
||||
return rows.find((item) => item.id === id) || null;
|
||||
@@ -240,6 +470,15 @@ export class ScriptEngine extends EventEmitter {
|
||||
return path.join(this.getScriptsDir(), `${slug}.py`);
|
||||
}
|
||||
|
||||
private normalizePathForCompare(filePath: string): string {
|
||||
return path.resolve(filePath).replace(/\\/g, '/');
|
||||
}
|
||||
|
||||
private isPythonScriptPath(value: string): boolean {
|
||||
const normalized = value.replace(/\\/g, '/').replace(/^\.\//, '');
|
||||
return normalized.startsWith('scripts/') && path.extname(normalized).toLowerCase() === '.py';
|
||||
}
|
||||
|
||||
private normalizeSlug(value: string): string {
|
||||
const normalized = value
|
||||
.toLowerCase()
|
||||
@@ -306,6 +545,183 @@ export class ScriptEngine extends EventEmitter {
|
||||
return rawContent.replace(frontmatterDocstringPattern, '');
|
||||
}
|
||||
|
||||
private parseScriptFile(rawContent: string): ParsedScriptFile {
|
||||
const frontmatterDocstringPattern = /^(?:"""|''')\r?\n---\r?\n([\s\S]*?)\r?\n---\r?\n(?:"""|''')\r?\n?/;
|
||||
const match = rawContent.match(frontmatterDocstringPattern);
|
||||
if (!match) {
|
||||
return {
|
||||
metadata: {},
|
||||
body: rawContent,
|
||||
};
|
||||
}
|
||||
|
||||
const metadataLines = (match[1] || '').split(/\r?\n/);
|
||||
const metadata: ParsedScriptFile['metadata'] = {};
|
||||
|
||||
for (const rawLine of metadataLines) {
|
||||
const line = rawLine.trim();
|
||||
if (!line || line.startsWith('#')) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const separatorIndex = line.indexOf(':');
|
||||
if (separatorIndex <= 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const key = line.slice(0, separatorIndex).trim();
|
||||
const valueRaw = line.slice(separatorIndex + 1).trim();
|
||||
const value = this.parseYamlScalar(valueRaw);
|
||||
|
||||
if (key === 'enabled') {
|
||||
if (typeof value === 'boolean') {
|
||||
metadata.enabled = value;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (key === 'version') {
|
||||
const parsed = Number(value);
|
||||
if (Number.isFinite(parsed)) {
|
||||
metadata.version = parsed;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (
|
||||
key === 'id' ||
|
||||
key === 'projectId' ||
|
||||
key === 'slug' ||
|
||||
key === 'title' ||
|
||||
key === 'kind' ||
|
||||
key === 'entrypoint' ||
|
||||
key === 'createdAt' ||
|
||||
key === 'updatedAt'
|
||||
) {
|
||||
if (typeof value === 'string') {
|
||||
metadata[key] = value;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
metadata,
|
||||
body: rawContent.replace(frontmatterDocstringPattern, ''),
|
||||
};
|
||||
}
|
||||
|
||||
private parseYamlScalar(valueRaw: string): string | number | boolean {
|
||||
if ((valueRaw.startsWith('"') && valueRaw.endsWith('"')) || (valueRaw.startsWith("'") && valueRaw.endsWith("'"))) {
|
||||
return valueRaw.slice(1, -1)
|
||||
.replace(/\\"/g, '"')
|
||||
.replace(/\\\\/g, '\\');
|
||||
}
|
||||
|
||||
if (valueRaw === 'true') {
|
||||
return true;
|
||||
}
|
||||
|
||||
if (valueRaw === 'false') {
|
||||
return false;
|
||||
}
|
||||
|
||||
const numeric = Number(valueRaw);
|
||||
if (!Number.isNaN(numeric)) {
|
||||
return numeric;
|
||||
}
|
||||
|
||||
return valueRaw;
|
||||
}
|
||||
|
||||
private normalizeKind(kind: string | undefined, fallback: ScriptKind = 'utility'): ScriptKind {
|
||||
if (kind === 'macro' || kind === 'utility' || kind === 'transform') {
|
||||
return kind;
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
private normalizeEntrypoint(entrypoint: string | undefined, fallback = 'render'): string {
|
||||
if (typeof entrypoint === 'string' && entrypoint.trim().length > 0) {
|
||||
return entrypoint.trim();
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
private normalizeEnabled(enabled: boolean | undefined, fallback = true): boolean {
|
||||
if (typeof enabled === 'boolean') {
|
||||
return enabled;
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
private normalizeVersion(version: number | undefined, fallback = 1): number {
|
||||
if (typeof version === 'number' && Number.isFinite(version) && version > 0) {
|
||||
return Math.floor(version);
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
private normalizeDate(value: string | undefined, fallback: Date): Date {
|
||||
if (typeof value === 'string') {
|
||||
const parsed = new Date(value);
|
||||
if (!Number.isNaN(parsed.getTime())) {
|
||||
return parsed;
|
||||
}
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
private normalizeTitle(title: string | undefined, slug: string, fallback?: string): string {
|
||||
if (typeof title === 'string' && title.trim().length > 0) {
|
||||
return title.trim();
|
||||
}
|
||||
if (typeof fallback === 'string' && fallback.trim().length > 0) {
|
||||
return fallback.trim();
|
||||
}
|
||||
return slug;
|
||||
}
|
||||
|
||||
private async scanScriptFiles(dir: string): Promise<string[]> {
|
||||
const results: string[] = [];
|
||||
|
||||
const scan = async (currentDir: string): Promise<void> => {
|
||||
let entries: Array<{ name: string; isDirectory(): boolean; isFile(): boolean }> = [];
|
||||
try {
|
||||
entries = await fs.readdir(currentDir, { withFileTypes: true }) as Array<{ name: string; isDirectory(): boolean; isFile(): boolean }>;
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
const fullPath = path.join(currentDir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
await scan(fullPath);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (entry.isFile() && path.extname(entry.name).toLowerCase() === '.py') {
|
||||
results.push(fullPath);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
await scan(dir);
|
||||
return results;
|
||||
}
|
||||
|
||||
private async readScriptFileWithMetadata(filePath: string): Promise<ParsedScriptFile | null> {
|
||||
try {
|
||||
const rawContent = await fs.readFile(filePath, 'utf-8');
|
||||
return this.parseScriptFile(rawContent);
|
||||
} catch (error) {
|
||||
const fsError = error as NodeJS.ErrnoException;
|
||||
if (fsError.code !== 'ENOENT') {
|
||||
throw error;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private async readScriptBody(filePath: string): Promise<string> {
|
||||
try {
|
||||
const rawContent = await fs.readFile(filePath, 'utf-8');
|
||||
|
||||
150
src/main/engine/blogmarkPython.worker.ts
Normal file
150
src/main/engine/blogmarkPython.worker.ts
Normal file
@@ -0,0 +1,150 @@
|
||||
import { parentPort } from 'worker_threads';
|
||||
|
||||
interface WorkerRunTransformRequest {
|
||||
type: 'runTransform';
|
||||
requestId: string;
|
||||
scriptContent: string;
|
||||
entrypoint: string;
|
||||
payloadJson: string;
|
||||
}
|
||||
|
||||
interface WorkerReadyMessage {
|
||||
type: 'ready';
|
||||
}
|
||||
|
||||
interface WorkerResultMessage {
|
||||
type: 'transformResult';
|
||||
requestId: string;
|
||||
output: unknown;
|
||||
toasts: string[];
|
||||
}
|
||||
|
||||
interface WorkerErrorMessage {
|
||||
type: 'transformError';
|
||||
requestId: string;
|
||||
error: string;
|
||||
}
|
||||
|
||||
interface WorkerFatalErrorMessage {
|
||||
type: 'error';
|
||||
error: string;
|
||||
}
|
||||
|
||||
type WorkerResponseMessage = WorkerReadyMessage | WorkerResultMessage | WorkerErrorMessage | WorkerFatalErrorMessage;
|
||||
|
||||
type PyodideRuntime = {
|
||||
globals: {
|
||||
set: (name: string, value: unknown) => void;
|
||||
} | any;
|
||||
runPythonAsync: (code: string) => Promise<unknown>;
|
||||
};
|
||||
|
||||
const MAX_TOASTS_PER_SCRIPT = 5;
|
||||
const MAX_TOAST_LENGTH = 300;
|
||||
|
||||
let runtimePromise: Promise<PyodideRuntime> | null = null;
|
||||
|
||||
function postMessage(message: WorkerResponseMessage): void {
|
||||
parentPort?.postMessage(message);
|
||||
}
|
||||
|
||||
function normalizeToastMessage(value: unknown): string | null {
|
||||
if (value === undefined || value === null) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const normalized = String(value).trim();
|
||||
if (normalized.length === 0) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return normalized.slice(0, MAX_TOAST_LENGTH);
|
||||
}
|
||||
|
||||
async function getRuntime(): Promise<PyodideRuntime> {
|
||||
if (!runtimePromise) {
|
||||
runtimePromise = (async () => {
|
||||
const pyodideModule = await import('pyodide');
|
||||
return (await pyodideModule.loadPyodide()) as unknown as PyodideRuntime;
|
||||
})();
|
||||
}
|
||||
|
||||
return runtimePromise;
|
||||
}
|
||||
|
||||
async function runTransform(request: WorkerRunTransformRequest): Promise<void> {
|
||||
try {
|
||||
const runtime = await getRuntime();
|
||||
const toastMessages: string[] = [];
|
||||
|
||||
const pushToast = (message: unknown): void => {
|
||||
if (toastMessages.length >= MAX_TOASTS_PER_SCRIPT) {
|
||||
return;
|
||||
}
|
||||
|
||||
const normalized = normalizeToastMessage(message);
|
||||
if (!normalized) {
|
||||
return;
|
||||
}
|
||||
|
||||
toastMessages.push(normalized);
|
||||
};
|
||||
|
||||
runtime.globals.set('__bds_push_toast', pushToast);
|
||||
await runtime.runPythonAsync(`
|
||||
def toast(message):
|
||||
__bds_push_toast(str(message))
|
||||
`);
|
||||
|
||||
await runtime.runPythonAsync(request.scriptContent);
|
||||
runtime.globals.set('__bds_transform_payload_json', request.payloadJson);
|
||||
runtime.globals.set('__bds_transform_entrypoint', request.entrypoint);
|
||||
|
||||
const rawResult = await runtime.runPythonAsync(`
|
||||
import json
|
||||
_payload = json.loads(__bds_transform_payload_json)
|
||||
_entrypoint = __bds_transform_entrypoint
|
||||
_transform_fn = globals().get(_entrypoint)
|
||||
if _transform_fn is None or not callable(_transform_fn):
|
||||
raise RuntimeError(f"Transform entrypoint '{_entrypoint}' is not callable")
|
||||
_post = _payload.get("post")
|
||||
if not isinstance(_post, dict):
|
||||
raise RuntimeError("Transform payload is missing a valid 'post' object")
|
||||
_context = _payload.get("context")
|
||||
try:
|
||||
_result = _transform_fn(_post, _context)
|
||||
except TypeError:
|
||||
_result = _transform_fn(_post)
|
||||
if _result is None:
|
||||
_result = _post
|
||||
json.dumps(_result)
|
||||
`);
|
||||
|
||||
postMessage({
|
||||
type: 'transformResult',
|
||||
requestId: request.requestId,
|
||||
output: JSON.parse(String(rawResult)),
|
||||
toasts: toastMessages,
|
||||
});
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
postMessage({ type: 'transformError', requestId: request.requestId, error: message });
|
||||
}
|
||||
}
|
||||
|
||||
parentPort?.on('message', (message: WorkerRunTransformRequest) => {
|
||||
if (message.type !== 'runTransform') {
|
||||
return;
|
||||
}
|
||||
|
||||
void runTransform(message);
|
||||
});
|
||||
|
||||
void getRuntime()
|
||||
.then(() => {
|
||||
postMessage({ type: 'ready' });
|
||||
})
|
||||
.catch((error) => {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
postMessage({ type: 'error', error: message });
|
||||
});
|
||||
@@ -185,8 +185,11 @@ export function registerIpcHandlers(): void {
|
||||
return pullResult;
|
||||
}
|
||||
|
||||
const changedPostFiles = await engine.getChangedPostFilesBetween(projectPath, beforeHead, afterHead);
|
||||
if (changedPostFiles.length === 0) {
|
||||
const [changedPostFiles, changedScriptFiles] = await Promise.all([
|
||||
engine.getChangedPostFilesBetween(projectPath, beforeHead, afterHead),
|
||||
engine.getChangedScriptFilesBetween(projectPath, beforeHead, afterHead),
|
||||
]);
|
||||
if (changedPostFiles.length === 0 && changedScriptFiles.length === 0) {
|
||||
return pullResult;
|
||||
}
|
||||
|
||||
@@ -194,15 +197,24 @@ export function registerIpcHandlers(): void {
|
||||
const projectEngine = getProjectEngine();
|
||||
const project = await projectEngine.getActiveProject();
|
||||
const postEngine = getPostEngine();
|
||||
const scriptEngine = getScriptEngine();
|
||||
|
||||
if (project) {
|
||||
const dataDir = projectEngine.getDataDir(project.id, project.dataPath);
|
||||
postEngine.setProjectContext(project.id, dataDir);
|
||||
scriptEngine.setProjectContext(project.id, dataDir);
|
||||
}
|
||||
|
||||
await postEngine.reconcilePublishedPostsFromGitChanges(projectPath, changedPostFiles);
|
||||
await Promise.all([
|
||||
changedPostFiles.length > 0
|
||||
? postEngine.reconcilePublishedPostsFromGitChanges(projectPath, changedPostFiles)
|
||||
: Promise.resolve(),
|
||||
changedScriptFiles.length > 0
|
||||
? scriptEngine.reconcileScriptsFromGitChanges(projectPath, changedScriptFiles)
|
||||
: Promise.resolve(),
|
||||
]);
|
||||
} catch (error) {
|
||||
console.error('Failed to reconcile published posts after git pull:', error);
|
||||
console.error('Failed to reconcile published posts/scripts after git pull:', error);
|
||||
}
|
||||
|
||||
return pullResult;
|
||||
@@ -755,6 +767,18 @@ export function registerIpcHandlers(): void {
|
||||
return engine.getAllScripts();
|
||||
});
|
||||
|
||||
safeHandle('scripts:rebuildFromFiles', async () => {
|
||||
const projectEngine = getProjectEngine();
|
||||
const project = await projectEngine.getActiveProject();
|
||||
const engine = getScriptEngine();
|
||||
if (project) {
|
||||
const dataDir = projectEngine.getDataDir(project.id, project.dataPath);
|
||||
engine.setProjectContext(project.id, dataDir);
|
||||
}
|
||||
await engine.rebuildDatabaseFromFiles();
|
||||
return true;
|
||||
});
|
||||
|
||||
// ============ Task Handlers ============
|
||||
|
||||
safeHandle('tasks:getAll', async () => {
|
||||
@@ -1006,7 +1030,7 @@ export function registerIpcHandlers(): void {
|
||||
return engine.getProjectMetadata();
|
||||
});
|
||||
|
||||
safeHandle('meta:updateProjectMetadata', async (_, updates: { name?: string; description?: string; dataPath?: string; publicUrl?: string; mainLanguage?: string; defaultAuthor?: string; maxPostsPerPage?: number; picoTheme?: import('../shared/picoThemes').PicoThemeName; categoryMetadata?: Record<string, { renderInLists: boolean; showTitle: boolean; title: string }>; categorySettings?: Record<string, { renderInLists: boolean; showTitle: boolean }> }) => {
|
||||
safeHandle('meta:updateProjectMetadata', async (_, updates: { name?: string; description?: string; dataPath?: string; publicUrl?: string; mainLanguage?: string; defaultAuthor?: string; maxPostsPerPage?: number; blogmarkCategory?: string; pythonRuntimeMode?: 'webworker' | 'main-thread'; picoTheme?: import('../shared/picoThemes').PicoThemeName; categoryMetadata?: Record<string, { renderInLists: boolean; showTitle: boolean; title: string }>; categorySettings?: Record<string, { renderInLists: boolean; showTitle: boolean }> }) => {
|
||||
const engine = getMetaEngine();
|
||||
await ensureMetaContext(engine);
|
||||
await engine.updateProjectMetadata(updates);
|
||||
|
||||
@@ -108,6 +108,7 @@ export const electronAPI: ElectronAPI = {
|
||||
delete: (id: string) => ipcRenderer.invoke('scripts:delete', id),
|
||||
get: (id: string) => ipcRenderer.invoke('scripts:get', id),
|
||||
getAll: () => ipcRenderer.invoke('scripts:getAll'),
|
||||
rebuildFromFiles: () => ipcRenderer.invoke('scripts:rebuildFromFiles'),
|
||||
},
|
||||
|
||||
// Post-Media Links
|
||||
@@ -172,7 +173,7 @@ export const electronAPI: ElectronAPI = {
|
||||
syncOnStartup: () => ipcRenderer.invoke('meta:syncOnStartup'),
|
||||
getProjectMetadata: () => ipcRenderer.invoke('meta:getProjectMetadata'),
|
||||
setProjectMetadata: (metadata: { name: string; description?: string }) => ipcRenderer.invoke('meta:setProjectMetadata', metadata),
|
||||
updateProjectMetadata: (updates: { name?: string; description?: string; dataPath?: string; publicUrl?: string; mainLanguage?: string; defaultAuthor?: string; maxPostsPerPage?: number; blogmarkCategory?: string; picoTheme?: import('./shared/picoThemes').PicoThemeName; categoryMetadata?: Record<string, { renderInLists: boolean; showTitle: boolean; title: string }>; categorySettings?: Record<string, { renderInLists: boolean; showTitle: boolean }> }) => ipcRenderer.invoke('meta:updateProjectMetadata', updates),
|
||||
updateProjectMetadata: (updates: { name?: string; description?: string; dataPath?: string; publicUrl?: string; mainLanguage?: string; defaultAuthor?: string; maxPostsPerPage?: number; blogmarkCategory?: string; pythonRuntimeMode?: 'webworker' | 'main-thread'; picoTheme?: import('./shared/picoThemes').PicoThemeName; categoryMetadata?: Record<string, { renderInLists: boolean; showTitle: boolean; title: string }>; categorySettings?: Record<string, { renderInLists: boolean; showTitle: boolean }> }) => ipcRenderer.invoke('meta:updateProjectMetadata', updates),
|
||||
},
|
||||
|
||||
// Tag Management (advanced tag operations)
|
||||
|
||||
@@ -43,6 +43,7 @@ export interface ProjectMetadata {
|
||||
defaultAuthor?: string;
|
||||
maxPostsPerPage?: number;
|
||||
blogmarkCategory?: string;
|
||||
pythonRuntimeMode?: 'webworker' | 'main-thread';
|
||||
picoTheme?: import('./picoThemes').PicoThemeName;
|
||||
categoryMetadata?: Record<string, CategoryMetadata>;
|
||||
categorySettings?: Record<string, CategoryRenderSettings>;
|
||||
@@ -565,6 +566,7 @@ export interface ElectronAPI {
|
||||
delete: (id: string) => Promise<boolean>;
|
||||
get: (id: string) => Promise<ScriptData | null>;
|
||||
getAll: () => Promise<ScriptData[]>;
|
||||
rebuildFromFiles: () => Promise<void>;
|
||||
};
|
||||
postMedia: {
|
||||
link: (postId: string, mediaId: string) => Promise<MediaLinkData>;
|
||||
@@ -619,7 +621,7 @@ export interface ElectronAPI {
|
||||
syncOnStartup: () => Promise<{ tags: string[]; categories: string[]; projectMetadata: ProjectMetadata | null }>;
|
||||
getProjectMetadata: () => Promise<ProjectMetadata | null>;
|
||||
setProjectMetadata: (metadata: { name: string; description?: string }) => Promise<ProjectMetadata | null>;
|
||||
updateProjectMetadata: (updates: { name?: string; description?: string; dataPath?: string; publicUrl?: string; mainLanguage?: string; defaultAuthor?: string; maxPostsPerPage?: number; blogmarkCategory?: string; picoTheme?: import('./picoThemes').PicoThemeName; categoryMetadata?: Record<string, CategoryMetadata>; categorySettings?: Record<string, CategoryRenderSettings> }) => Promise<ProjectMetadata | null>;
|
||||
updateProjectMetadata: (updates: { name?: string; description?: string; dataPath?: string; publicUrl?: string; mainLanguage?: string; defaultAuthor?: string; maxPostsPerPage?: number; blogmarkCategory?: string; pythonRuntimeMode?: 'webworker' | 'main-thread'; picoTheme?: import('./picoThemes').PicoThemeName; categoryMetadata?: Record<string, CategoryMetadata>; categorySettings?: Record<string, CategoryRenderSettings> }) => Promise<ProjectMetadata | null>;
|
||||
};
|
||||
tags: {
|
||||
getAll: () => Promise<TagData[]>;
|
||||
|
||||
@@ -10,7 +10,7 @@ import {
|
||||
import './SettingsView.css';
|
||||
|
||||
// Export category IDs for sidebar navigation
|
||||
export type SettingsCategory = 'project' | 'editor' | 'content' | 'ai' | 'publishing' | 'data';
|
||||
export type SettingsCategory = 'project' | 'editor' | 'content' | 'ai' | 'technology' | 'publishing' | 'data';
|
||||
|
||||
// Scroll to a settings section by category ID
|
||||
export const scrollToSettingsSection = (category: SettingsCategory) => {
|
||||
@@ -150,6 +150,7 @@ export const SettingsView: React.FC = () => {
|
||||
const [projectDefaultAuthor, setProjectDefaultAuthor] = useState('');
|
||||
const [projectMaxPostsPerPage, setProjectMaxPostsPerPage] = useState(50);
|
||||
const [projectBlogmarkCategory, setProjectBlogmarkCategory] = useState('article');
|
||||
const [projectPythonRuntimeMode, setProjectPythonRuntimeMode] = useState<'webworker' | 'main-thread'>('webworker');
|
||||
|
||||
// Post categories management
|
||||
const [postCategories, setPostCategories] = useState<string[]>(DEFAULT_POST_CATEGORIES);
|
||||
@@ -208,6 +209,9 @@ export const SettingsView: React.FC = () => {
|
||||
const incomingBlogmarkCategory = normalizeBlogmarkCategory((metadata as { blogmarkCategory?: unknown } | null)?.blogmarkCategory);
|
||||
setProjectBlogmarkCategory(incomingBlogmarkCategory || 'article');
|
||||
|
||||
const incomingPythonRuntimeMode = (metadata as { pythonRuntimeMode?: unknown } | null)?.pythonRuntimeMode;
|
||||
setProjectPythonRuntimeMode(incomingPythonRuntimeMode === 'main-thread' ? 'main-thread' : 'webworker');
|
||||
|
||||
const incomingCategoryMetadata = (metadata as any)?.categoryMetadata as Record<string, CategoryMetadata> | undefined;
|
||||
const incomingLegacyCategorySettings = (metadata as any)?.categorySettings as Record<string, { renderInLists: boolean; showTitle: boolean }> | undefined;
|
||||
setCategoryMetadata((current) => {
|
||||
@@ -342,6 +346,7 @@ export const SettingsView: React.FC = () => {
|
||||
defaultAuthor: projectDefaultAuthor.trim() || undefined,
|
||||
maxPostsPerPage: Math.min(500, Math.max(1, Math.floor(projectMaxPostsPerPage || 50))),
|
||||
blogmarkCategory: normalizeBlogmarkCategory(projectBlogmarkCategory) || undefined,
|
||||
pythonRuntimeMode: projectPythonRuntimeMode,
|
||||
categoryMetadata,
|
||||
});
|
||||
}
|
||||
@@ -389,8 +394,9 @@ export const SettingsView: React.FC = () => {
|
||||
const editorKeywords = ['editor', 'mode', 'wysiwyg', 'markdown', 'preview', 'visual'];
|
||||
const contentKeywords = ['content', 'categories', 'post', 'article', 'picture', 'aside', 'page'];
|
||||
const aiKeywords = ['ai', 'assistant', 'chat', 'model', 'prompt', 'system', 'api', 'key', 'claude', 'gpt', 'opencode'];
|
||||
const technologyKeywords = ['technology', 'python', 'runtime', 'worker', 'webworker', 'main thread', 'execution'];
|
||||
const publishingKeywords = ['publishing', 'ftp', 'ssh', 'deploy', 'server', 'host', 'upload'];
|
||||
const dataKeywords = ['data', 'database', 'rebuild', 'maintenance', 'posts', 'media', 'links', 'folder', 'filesystem'];
|
||||
const dataKeywords = ['data', 'database', 'rebuild', 'maintenance', 'posts', 'media', 'scripts', 'links', 'folder', 'filesystem'];
|
||||
|
||||
const renderProjectSettings = () => (
|
||||
<SettingSection
|
||||
@@ -1023,6 +1029,30 @@ export const SettingsView: React.FC = () => {
|
||||
</SettingSection>
|
||||
);
|
||||
|
||||
const renderTechnologySettings = () => (
|
||||
<SettingSection
|
||||
id="settings-section-technology"
|
||||
title={t('settings.technology.title')}
|
||||
description={t('settings.technology.description')}
|
||||
hidden={!sectionHasMatches(technologyKeywords)}
|
||||
>
|
||||
<SettingRow
|
||||
id="project-python-runtime-mode"
|
||||
label={t('settings.technology.pythonRuntimeModeLabel')}
|
||||
description={t('settings.technology.pythonRuntimeModeDescription')}
|
||||
>
|
||||
<select
|
||||
id="project-python-runtime-mode"
|
||||
value={projectPythonRuntimeMode}
|
||||
onChange={(event) => setProjectPythonRuntimeMode(event.target.value as 'webworker' | 'main-thread')}
|
||||
>
|
||||
<option value="webworker">{t('settings.technology.pythonRuntimeMode.webworker')}</option>
|
||||
<option value="main-thread">{t('settings.technology.pythonRuntimeMode.mainThread')}</option>
|
||||
</select>
|
||||
</SettingRow>
|
||||
</SettingSection>
|
||||
);
|
||||
|
||||
const renderPublishingSettings = () => (
|
||||
<>
|
||||
<SettingSection
|
||||
@@ -1205,6 +1235,29 @@ export const SettingsView: React.FC = () => {
|
||||
</button>
|
||||
</SettingRow>
|
||||
|
||||
<SettingRow
|
||||
id="rebuild-scripts"
|
||||
label={t('settings.data.rebuildScriptsLabel')}
|
||||
description={t('settings.data.rebuildScriptsDescription')}
|
||||
>
|
||||
<button
|
||||
className="secondary"
|
||||
onClick={async () => {
|
||||
showToast.loading(t('settings.toast.rebuildScriptsLoading'));
|
||||
try {
|
||||
await window.electronAPI?.scripts.rebuildFromFiles();
|
||||
showToast.dismiss();
|
||||
showToast.success(t('settings.toast.rebuildScriptsSuccess'));
|
||||
} catch {
|
||||
showToast.dismiss();
|
||||
showToast.error(t('settings.toast.rebuildScriptsFailed'));
|
||||
}
|
||||
}}
|
||||
>
|
||||
{t('settings.data.rebuildScriptsAction')}
|
||||
</button>
|
||||
</SettingRow>
|
||||
|
||||
<SettingRow
|
||||
id="rebuild-links"
|
||||
label={t('settings.data.rebuildLinksLabel')}
|
||||
@@ -1290,6 +1343,7 @@ export const SettingsView: React.FC = () => {
|
||||
sectionHasMatches(editorKeywords) ||
|
||||
sectionHasMatches(contentKeywords) ||
|
||||
sectionHasMatches(aiKeywords) ||
|
||||
sectionHasMatches(technologyKeywords) ||
|
||||
sectionHasMatches(publishingKeywords) ||
|
||||
sectionHasMatches(dataKeywords);
|
||||
|
||||
@@ -1325,6 +1379,7 @@ export const SettingsView: React.FC = () => {
|
||||
{renderEditorSettings()}
|
||||
{renderContentSettings()}
|
||||
{renderAISettings()}
|
||||
{renderTechnologySettings()}
|
||||
{renderPublishingSettings()}
|
||||
{renderDataSettings()}
|
||||
</>
|
||||
|
||||
@@ -1261,7 +1261,7 @@ const SettingsNav: React.FC = () => {
|
||||
const { tabs, activeTabId, openTab } = useAppStore();
|
||||
const [activeSection, setActiveSection] = useState<SettingsCategory | null>(() => {
|
||||
const persisted = getPersistedSidebarSection('settings');
|
||||
if (persisted === 'project' || persisted === 'editor' || persisted === 'content' || persisted === 'ai' || persisted === 'publishing' || persisted === 'data') {
|
||||
if (persisted === 'project' || persisted === 'editor' || persisted === 'content' || persisted === 'ai' || persisted === 'technology' || persisted === 'publishing' || persisted === 'data') {
|
||||
return persisted;
|
||||
}
|
||||
return null;
|
||||
@@ -1322,6 +1322,13 @@ const SettingsNav: React.FC = () => {
|
||||
<span className="settings-nav-entry-icon">🤖</span>
|
||||
<span>{t('sidebar.nav.ai')}</span>
|
||||
</button>
|
||||
<button
|
||||
className={`settings-nav-entry ${activeSection === 'technology' ? 'active' : ''}`}
|
||||
onClick={() => handleNavClick('technology')}
|
||||
>
|
||||
<span className="settings-nav-entry-icon">⚙️</span>
|
||||
<span>{t('sidebar.nav.technology')}</span>
|
||||
</button>
|
||||
<button
|
||||
className={`settings-nav-entry ${activeSection === 'publishing' ? 'active' : ''}`}
|
||||
onClick={() => handleNavClick('publishing')}
|
||||
|
||||
@@ -127,6 +127,12 @@
|
||||
"settings.content.showTitles": "Titel anzeigen",
|
||||
"settings.ai.title": "KI-Assistent",
|
||||
"settings.ai.noModels": "Keine Modelle verfügbar",
|
||||
"settings.technology.title": "Technologie",
|
||||
"settings.technology.description": "Konfiguriere das Laufzeitverhalten für die Python-Skriptausführung.",
|
||||
"settings.technology.pythonRuntimeModeLabel": "Python-Laufzeitmodus",
|
||||
"settings.technology.pythonRuntimeModeDescription": "Lege fest, wo Python-Skripte für Transformationspipelines ausgeführt werden.",
|
||||
"settings.technology.pythonRuntimeMode.webworker": "Web Worker (empfohlen)",
|
||||
"settings.technology.pythonRuntimeMode.mainThread": "Hauptthread (Legacy)",
|
||||
"settings.publishing.ftpTitle": "FTP-Veröffentlichung",
|
||||
"settings.publishing.sshTitle": "SSH-Veröffentlichung",
|
||||
"settings.data.title": "Datenbankwartung",
|
||||
@@ -167,6 +173,9 @@
|
||||
"settings.toast.rebuildMediaLoading": "Mediendatenbank wird neu aufgebaut...",
|
||||
"settings.toast.rebuildMediaSuccess": "Mediendatenbank neu aufgebaut",
|
||||
"settings.toast.rebuildMediaFailed": "Mediendatenbank konnte nicht neu aufgebaut werden",
|
||||
"settings.toast.rebuildScriptsLoading": "Skriptdatenbank wird neu aufgebaut...",
|
||||
"settings.toast.rebuildScriptsSuccess": "Skriptdatenbank neu aufgebaut",
|
||||
"settings.toast.rebuildScriptsFailed": "Skriptdatenbank konnte nicht neu aufgebaut werden",
|
||||
"settings.toast.rebuildLinksLoading": "Beitragslinks werden neu aufgebaut...",
|
||||
"settings.toast.rebuildLinksSuccess": "Beitragslinks neu aufgebaut",
|
||||
"settings.toast.rebuildLinksFailed": "Beitragslinks konnten nicht neu aufgebaut werden",
|
||||
@@ -421,6 +430,7 @@
|
||||
"sidebar.nav.editor": "Texteditor",
|
||||
"sidebar.nav.content": "Inhalt",
|
||||
"sidebar.nav.ai": "KI-Assistent",
|
||||
"sidebar.nav.technology": "Technologie",
|
||||
"sidebar.nav.publishing": "Veröffentlichung",
|
||||
"sidebar.nav.data": "Daten",
|
||||
"sidebar.nav.style": "Stil",
|
||||
@@ -702,6 +712,9 @@
|
||||
"settings.data.rebuildMediaLabel": "Mediendatenbank neu aufbauen",
|
||||
"settings.data.rebuildMediaDescription": "Alle Mediendateien und Sidecar-Metadaten neu scannen. Fehlende Einträge werden neu erzeugt.",
|
||||
"settings.data.rebuildMediaAction": "Medien neu aufbauen",
|
||||
"settings.data.rebuildScriptsLabel": "Skriptdatenbank neu aufbauen",
|
||||
"settings.data.rebuildScriptsDescription": "Alle Python-Skripte neu scannen und den Skript-Metadatenindex neu aufbauen.",
|
||||
"settings.data.rebuildScriptsAction": "Skripte neu aufbauen",
|
||||
"settings.data.rebuildLinksLabel": "Beitragslinks neu aufbauen",
|
||||
"settings.data.rebuildLinksDescription": "Alle Beiträge neu scannen und den internen Linkgraphen zwischen Beiträgen neu aufbauen.",
|
||||
"settings.data.rebuildLinksAction": "Links neu aufbauen",
|
||||
|
||||
@@ -127,6 +127,12 @@
|
||||
"settings.content.showTitles": "Show titles",
|
||||
"settings.ai.title": "AI Assistant",
|
||||
"settings.ai.noModels": "No models available",
|
||||
"settings.technology.title": "Technology",
|
||||
"settings.technology.description": "Configure runtime behavior for Python script execution.",
|
||||
"settings.technology.pythonRuntimeModeLabel": "Python Runtime Mode",
|
||||
"settings.technology.pythonRuntimeModeDescription": "Choose where Python scripts execute for transform pipelines.",
|
||||
"settings.technology.pythonRuntimeMode.webworker": "Web Worker (Recommended)",
|
||||
"settings.technology.pythonRuntimeMode.mainThread": "Main Thread (Legacy)",
|
||||
"settings.publishing.ftpTitle": "FTP Publishing",
|
||||
"settings.publishing.sshTitle": "SSH Publishing",
|
||||
"settings.data.title": "Database Maintenance",
|
||||
@@ -167,6 +173,9 @@
|
||||
"settings.toast.rebuildMediaLoading": "Rebuilding media database...",
|
||||
"settings.toast.rebuildMediaSuccess": "Media database rebuilt",
|
||||
"settings.toast.rebuildMediaFailed": "Failed to rebuild media database",
|
||||
"settings.toast.rebuildScriptsLoading": "Rebuilding scripts database...",
|
||||
"settings.toast.rebuildScriptsSuccess": "Scripts database rebuilt",
|
||||
"settings.toast.rebuildScriptsFailed": "Failed to rebuild scripts database",
|
||||
"settings.toast.rebuildLinksLoading": "Rebuilding post links...",
|
||||
"settings.toast.rebuildLinksSuccess": "Post links rebuilt",
|
||||
"settings.toast.rebuildLinksFailed": "Failed to rebuild post links",
|
||||
@@ -421,6 +430,7 @@
|
||||
"sidebar.nav.editor": "Editor",
|
||||
"sidebar.nav.content": "Content",
|
||||
"sidebar.nav.ai": "AI Assistant",
|
||||
"sidebar.nav.technology": "Technology",
|
||||
"sidebar.nav.publishing": "Publishing",
|
||||
"sidebar.nav.data": "Data",
|
||||
"sidebar.nav.style": "Style",
|
||||
@@ -702,6 +712,9 @@
|
||||
"settings.data.rebuildMediaLabel": "Rebuild Media Database",
|
||||
"settings.data.rebuildMediaDescription": "Re-scan all media files and sidecar metadata. Regenerates missing entries.",
|
||||
"settings.data.rebuildMediaAction": "Rebuild Media",
|
||||
"settings.data.rebuildScriptsLabel": "Rebuild Scripts Database",
|
||||
"settings.data.rebuildScriptsDescription": "Re-scan all Python scripts and rebuild the scripts metadata index.",
|
||||
"settings.data.rebuildScriptsAction": "Rebuild Scripts",
|
||||
"settings.data.rebuildLinksLabel": "Rebuild Post Links",
|
||||
"settings.data.rebuildLinksDescription": "Re-scan all posts and rebuild the internal link graph between posts.",
|
||||
"settings.data.rebuildLinksAction": "Rebuild Links",
|
||||
|
||||
@@ -127,6 +127,12 @@
|
||||
"settings.content.showTitles": "Mostrar títulos",
|
||||
"settings.ai.title": "Asistente IA",
|
||||
"settings.ai.noModels": "No hay modelos disponibles",
|
||||
"settings.technology.title": "Tecnología",
|
||||
"settings.technology.description": "Configura el comportamiento de ejecución para scripts de Python.",
|
||||
"settings.technology.pythonRuntimeModeLabel": "Modo de ejecución de Python",
|
||||
"settings.technology.pythonRuntimeModeDescription": "Elige dónde se ejecutan los scripts de Python para los flujos de transformación.",
|
||||
"settings.technology.pythonRuntimeMode.webworker": "Web Worker (recomendado)",
|
||||
"settings.technology.pythonRuntimeMode.mainThread": "Hilo principal (heredado)",
|
||||
"settings.publishing.ftpTitle": "Publicación FTP",
|
||||
"settings.publishing.sshTitle": "Publicación SSH",
|
||||
"settings.data.title": "Mantenimiento de base de datos",
|
||||
@@ -167,6 +173,9 @@
|
||||
"settings.toast.rebuildMediaLoading": "Reconstruyendo base de datos de medios...",
|
||||
"settings.toast.rebuildMediaSuccess": "Base de datos de medios reconstruida",
|
||||
"settings.toast.rebuildMediaFailed": "No se pudo reconstruir la base de datos de medios",
|
||||
"settings.toast.rebuildScriptsLoading": "Reconstruyendo base de datos de scripts...",
|
||||
"settings.toast.rebuildScriptsSuccess": "Base de datos de scripts reconstruida",
|
||||
"settings.toast.rebuildScriptsFailed": "No se pudo reconstruir la base de datos de scripts",
|
||||
"settings.toast.rebuildLinksLoading": "Reconstruyendo enlaces de entradas...",
|
||||
"settings.toast.rebuildLinksSuccess": "Enlaces de publicaciones reconstruidos",
|
||||
"settings.toast.rebuildLinksFailed": "No se pudieron reconstruir los enlaces de entradas",
|
||||
@@ -421,6 +430,7 @@
|
||||
"sidebar.nav.editor": "Editor",
|
||||
"sidebar.nav.content": "Contenido",
|
||||
"sidebar.nav.ai": "Asistente IA",
|
||||
"sidebar.nav.technology": "Tecnología",
|
||||
"sidebar.nav.publishing": "Publicación",
|
||||
"sidebar.nav.data": "Datos",
|
||||
"sidebar.nav.style": "Estilo",
|
||||
@@ -702,6 +712,9 @@
|
||||
"settings.data.rebuildMediaLabel": "Reconstruir base de datos de medios",
|
||||
"settings.data.rebuildMediaDescription": "Reescanea todos los archivos multimedia y metadatos sidecar. Regenera las entradas faltantes.",
|
||||
"settings.data.rebuildMediaAction": "Reconstruir medios",
|
||||
"settings.data.rebuildScriptsLabel": "Reconstruir base de datos de scripts",
|
||||
"settings.data.rebuildScriptsDescription": "Reescanea todos los scripts de Python y reconstruye el índice de metadatos de scripts.",
|
||||
"settings.data.rebuildScriptsAction": "Reconstruir scripts",
|
||||
"settings.data.rebuildLinksLabel": "Reconstruir enlaces de publicaciones",
|
||||
"settings.data.rebuildLinksDescription": "Reescanea todas las publicaciones y reconstruye el grafo interno de enlaces entre publicaciones.",
|
||||
"settings.data.rebuildLinksAction": "Reconstruir enlaces",
|
||||
|
||||
@@ -127,6 +127,12 @@
|
||||
"settings.content.showTitles": "Afficher les titres",
|
||||
"settings.ai.title": "Assistant IA",
|
||||
"settings.ai.noModels": "Aucun modèle disponible",
|
||||
"settings.technology.title": "Technologie",
|
||||
"settings.technology.description": "Configurez le comportement d’exécution des scripts Python.",
|
||||
"settings.technology.pythonRuntimeModeLabel": "Mode d’exécution Python",
|
||||
"settings.technology.pythonRuntimeModeDescription": "Choisissez où les scripts Python s’exécutent pour les pipelines de transformation.",
|
||||
"settings.technology.pythonRuntimeMode.webworker": "Web Worker (recommandé)",
|
||||
"settings.technology.pythonRuntimeMode.mainThread": "Thread principal (hérité)",
|
||||
"settings.publishing.ftpTitle": "Publication FTP",
|
||||
"settings.publishing.sshTitle": "Publication SSH",
|
||||
"settings.data.title": "Maintenance de la base de données",
|
||||
@@ -167,6 +173,9 @@
|
||||
"settings.toast.rebuildMediaLoading": "Reconstruction de la base des médias...",
|
||||
"settings.toast.rebuildMediaSuccess": "Base médias reconstruite",
|
||||
"settings.toast.rebuildMediaFailed": "Impossible de reconstruire la base des médias",
|
||||
"settings.toast.rebuildScriptsLoading": "Reconstruction de la base des scripts...",
|
||||
"settings.toast.rebuildScriptsSuccess": "Base des scripts reconstruite",
|
||||
"settings.toast.rebuildScriptsFailed": "Impossible de reconstruire la base des scripts",
|
||||
"settings.toast.rebuildLinksLoading": "Reconstruction des liens d’articles...",
|
||||
"settings.toast.rebuildLinksSuccess": "Liens d’articles reconstruits",
|
||||
"settings.toast.rebuildLinksFailed": "Impossible de reconstruire les liens d’articles",
|
||||
@@ -421,6 +430,7 @@
|
||||
"sidebar.nav.editor": "Éditeur",
|
||||
"sidebar.nav.content": "Contenu",
|
||||
"sidebar.nav.ai": "Assistant IA",
|
||||
"sidebar.nav.technology": "Technologie",
|
||||
"sidebar.nav.publishing": "Publication",
|
||||
"sidebar.nav.data": "Données",
|
||||
"sidebar.nav.style": "Style",
|
||||
@@ -702,6 +712,9 @@
|
||||
"settings.data.rebuildMediaLabel": "Reconstruire la base médias",
|
||||
"settings.data.rebuildMediaDescription": "Réanalyse tous les fichiers médias et leurs métadonnées sidecar. Régénère les entrées manquantes.",
|
||||
"settings.data.rebuildMediaAction": "Reconstruire les médias",
|
||||
"settings.data.rebuildScriptsLabel": "Reconstruire la base des scripts",
|
||||
"settings.data.rebuildScriptsDescription": "Réanalyse tous les scripts Python et reconstruit l’index des métadonnées de scripts.",
|
||||
"settings.data.rebuildScriptsAction": "Reconstruire les scripts",
|
||||
"settings.data.rebuildLinksLabel": "Reconstruire les liens d’articles",
|
||||
"settings.data.rebuildLinksDescription": "Réanalyse tous les articles et reconstruit le graphe interne des liens entre articles.",
|
||||
"settings.data.rebuildLinksAction": "Reconstruire les liens",
|
||||
|
||||
@@ -127,6 +127,12 @@
|
||||
"settings.content.showTitles": "Mostra titoli",
|
||||
"settings.ai.title": "Assistente IA",
|
||||
"settings.ai.noModels": "Nessun modello disponibile",
|
||||
"settings.technology.title": "Tecnologia",
|
||||
"settings.technology.description": "Configura il comportamento di runtime per l'esecuzione degli script Python.",
|
||||
"settings.technology.pythonRuntimeModeLabel": "Modalità runtime Python",
|
||||
"settings.technology.pythonRuntimeModeDescription": "Scegli dove eseguire gli script Python per le pipeline di trasformazione.",
|
||||
"settings.technology.pythonRuntimeMode.webworker": "Web Worker (consigliato)",
|
||||
"settings.technology.pythonRuntimeMode.mainThread": "Thread principale (legacy)",
|
||||
"settings.publishing.ftpTitle": "Pubblicazione FTP",
|
||||
"settings.publishing.sshTitle": "Pubblicazione SSH",
|
||||
"settings.data.title": "Manutenzione database",
|
||||
@@ -167,6 +173,9 @@
|
||||
"settings.toast.rebuildMediaLoading": "Ricostruzione database media...",
|
||||
"settings.toast.rebuildMediaSuccess": "Database media ricostruito",
|
||||
"settings.toast.rebuildMediaFailed": "Impossibile ricostruire il database dei media",
|
||||
"settings.toast.rebuildScriptsLoading": "Ricostruzione database script...",
|
||||
"settings.toast.rebuildScriptsSuccess": "Database script ricostruito",
|
||||
"settings.toast.rebuildScriptsFailed": "Impossibile ricostruire il database degli script",
|
||||
"settings.toast.rebuildLinksLoading": "Ricostruzione dei link dei post...",
|
||||
"settings.toast.rebuildLinksSuccess": "Link dei post ricostruiti",
|
||||
"settings.toast.rebuildLinksFailed": "Impossibile ricostruire i link dei post",
|
||||
@@ -421,6 +430,7 @@
|
||||
"sidebar.nav.editor": "Editor",
|
||||
"sidebar.nav.content": "Contenuto",
|
||||
"sidebar.nav.ai": "Assistente IA",
|
||||
"sidebar.nav.technology": "Tecnologia",
|
||||
"sidebar.nav.publishing": "Pubblicazione",
|
||||
"sidebar.nav.data": "Dati",
|
||||
"sidebar.nav.style": "Stile",
|
||||
@@ -702,6 +712,9 @@
|
||||
"settings.data.rebuildMediaLabel": "Ricostruisci database media",
|
||||
"settings.data.rebuildMediaDescription": "Rianalizza tutti i file media e i metadati sidecar. Rigenera le voci mancanti.",
|
||||
"settings.data.rebuildMediaAction": "Ricostruisci media",
|
||||
"settings.data.rebuildScriptsLabel": "Ricostruisci database script",
|
||||
"settings.data.rebuildScriptsDescription": "Rianalizza tutti gli script Python e ricostruisce l’indice dei metadati degli script.",
|
||||
"settings.data.rebuildScriptsAction": "Ricostruisci script",
|
||||
"settings.data.rebuildLinksLabel": "Ricostruisci collegamenti post",
|
||||
"settings.data.rebuildLinksDescription": "Rianalizza tutti i post e ricostruisce il grafo interno dei collegamenti tra post.",
|
||||
"settings.data.rebuildLinksAction": "Ricostruisci collegamenti",
|
||||
|
||||
@@ -54,6 +54,8 @@ export class PythonRuntimeManager {
|
||||
private initializeDeferred: InitializeDeferred | null = null;
|
||||
private ready = false;
|
||||
private pendingRuns = new Map<string, PendingRun>();
|
||||
private requestQueue: PythonWorkerRequest[] = [];
|
||||
private activeRequestId: string | null = null;
|
||||
private requestCounter = 0;
|
||||
|
||||
constructor(private readonly workerFactory: WorkerFactory = createPythonRuntimeWorker) {}
|
||||
@@ -123,7 +125,7 @@ export class PythonRuntimeManager {
|
||||
entrypoint: options?.entrypoint,
|
||||
};
|
||||
|
||||
this.worker!.postMessage(message);
|
||||
this.enqueueRequest(message);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -162,7 +164,7 @@ export class PythonRuntimeManager {
|
||||
cacheKey: options?.cacheKey,
|
||||
};
|
||||
|
||||
this.worker!.postMessage(message);
|
||||
this.enqueueRequest(message);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -198,7 +200,7 @@ export class PythonRuntimeManager {
|
||||
cacheKey: options?.cacheKey,
|
||||
};
|
||||
|
||||
this.worker!.postMessage(message);
|
||||
this.enqueueRequest(message);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -234,7 +236,7 @@ export class PythonRuntimeManager {
|
||||
cacheKey: options?.cacheKey,
|
||||
};
|
||||
|
||||
this.worker!.postMessage(message);
|
||||
this.enqueueRequest(message);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -262,6 +264,10 @@ export class PythonRuntimeManager {
|
||||
|
||||
const pendingRun = this.pendingRuns.get(payload.requestId);
|
||||
if (!pendingRun) {
|
||||
if (this.activeRequestId === payload.requestId && payload.type !== 'stdout') {
|
||||
this.activeRequestId = null;
|
||||
this.dispatchNextRequest();
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -278,33 +284,40 @@ export class PythonRuntimeManager {
|
||||
if (payload.type === 'runResult') {
|
||||
if (pendingRun.kind !== 'run') {
|
||||
pendingRun.reject(new Error('Invalid response type for pending macro request'));
|
||||
this.finishRequest(payload.requestId);
|
||||
return;
|
||||
}
|
||||
pendingRun.resolve({ result: payload.result, stdout: pendingRun.stdout });
|
||||
this.finishRequest(payload.requestId);
|
||||
return;
|
||||
}
|
||||
|
||||
if (payload.type === 'entrypoints') {
|
||||
if (pendingRun.kind !== 'inspect-entrypoints') {
|
||||
pendingRun.reject(new Error('Invalid response type for pending run request'));
|
||||
this.finishRequest(payload.requestId);
|
||||
return;
|
||||
}
|
||||
pendingRun.resolve(payload.entrypoints);
|
||||
this.finishRequest(payload.requestId);
|
||||
return;
|
||||
}
|
||||
|
||||
if (payload.type === 'syntaxResult') {
|
||||
if (pendingRun.kind !== 'syntax-check') {
|
||||
pendingRun.reject(new Error('Invalid response type for pending syntax check request'));
|
||||
this.finishRequest(payload.requestId);
|
||||
return;
|
||||
}
|
||||
pendingRun.resolve({ errors: payload.errors });
|
||||
this.finishRequest(payload.requestId);
|
||||
return;
|
||||
}
|
||||
|
||||
if (payload.type === 'macroResult') {
|
||||
if (pendingRun.kind !== 'macro-v1') {
|
||||
pendingRun.reject(new Error('Invalid response type for pending run request'));
|
||||
this.finishRequest(payload.requestId);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -314,10 +327,12 @@ export class PythonRuntimeManager {
|
||||
} catch (error) {
|
||||
pendingRun.reject(error instanceof Error ? error : new Error(String(error)));
|
||||
}
|
||||
this.finishRequest(payload.requestId);
|
||||
return;
|
||||
}
|
||||
|
||||
pendingRun.reject(new Error(payload.error));
|
||||
this.finishRequest(payload.requestId);
|
||||
}
|
||||
|
||||
private handleWorkerError(error: Error): void {
|
||||
@@ -334,6 +349,8 @@ export class PythonRuntimeManager {
|
||||
}
|
||||
|
||||
this.pendingRuns.clear();
|
||||
this.requestQueue = [];
|
||||
this.activeRequestId = null;
|
||||
this.worker?.terminate();
|
||||
this.worker = null;
|
||||
this.initializingPromise = null;
|
||||
@@ -356,12 +373,50 @@ export class PythonRuntimeManager {
|
||||
}
|
||||
|
||||
this.pendingRuns.clear();
|
||||
this.requestQueue = [];
|
||||
this.activeRequestId = null;
|
||||
this.worker?.terminate();
|
||||
this.worker = null;
|
||||
this.initializingPromise = null;
|
||||
this.ready = false;
|
||||
}
|
||||
|
||||
private enqueueRequest(request: PythonWorkerRequest): void {
|
||||
if (!this.worker || !this.ready) {
|
||||
this.requestQueue.push(request);
|
||||
return;
|
||||
}
|
||||
|
||||
if (this.activeRequestId !== null) {
|
||||
this.requestQueue.push(request);
|
||||
return;
|
||||
}
|
||||
|
||||
this.activeRequestId = request.requestId;
|
||||
this.worker.postMessage(request);
|
||||
}
|
||||
|
||||
private dispatchNextRequest(): void {
|
||||
if (!this.worker || !this.ready || this.activeRequestId !== null || this.requestQueue.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
const nextRequest = this.requestQueue.shift();
|
||||
if (!nextRequest) {
|
||||
return;
|
||||
}
|
||||
|
||||
this.activeRequestId = nextRequest.requestId;
|
||||
this.worker.postMessage(nextRequest);
|
||||
}
|
||||
|
||||
private finishRequest(requestId: string): void {
|
||||
if (this.activeRequestId === requestId) {
|
||||
this.activeRequestId = null;
|
||||
}
|
||||
this.dispatchNextRequest();
|
||||
}
|
||||
|
||||
private nextRequestId(): string {
|
||||
this.requestCounter += 1;
|
||||
return `req-${this.requestCounter}`;
|
||||
|
||||
@@ -63,7 +63,10 @@ describe('BlogmarkTransformService (Pyodide integration)', () => {
|
||||
getScripts: async () => [createTransformScript()],
|
||||
};
|
||||
|
||||
const service = new BlogmarkTransformService({ provider });
|
||||
const service = new BlogmarkTransformService({
|
||||
provider,
|
||||
resolvePythonRuntimeMode: async () => 'main-thread',
|
||||
});
|
||||
|
||||
const result = await service.applyTransforms(createInput());
|
||||
|
||||
|
||||
@@ -247,67 +247,53 @@ describe('BlogmarkTransformService', () => {
|
||||
expect(result.toasts).toEqual(['Step finished', 'Step finished']);
|
||||
});
|
||||
|
||||
it('invokes python transform entrypoint with post payload shape', async () => {
|
||||
const globalsStore = new Map<string, unknown>();
|
||||
const runPythonAsync = vi.fn(async (code: string) => {
|
||||
if (code.includes('json.dumps(_result)')) {
|
||||
const payload = JSON.parse(String(globalsStore.get('__bds_transform_payload_json')));
|
||||
|
||||
if (code.includes('_transform_fn(_payload)')) {
|
||||
return JSON.stringify(payload);
|
||||
}
|
||||
|
||||
return JSON.stringify({
|
||||
...payload.post,
|
||||
title: 'Normalized',
|
||||
categories: ['spielelog', 'asides'],
|
||||
tags: ['inbox', 'spielen'],
|
||||
});
|
||||
}
|
||||
|
||||
return null;
|
||||
});
|
||||
|
||||
vi.doMock('pyodide', () => ({
|
||||
loadPyodide: vi.fn(async () => ({
|
||||
globals: {
|
||||
set: (key: string, value: unknown) => {
|
||||
globalsStore.set(key, value);
|
||||
},
|
||||
},
|
||||
runPythonAsync,
|
||||
})),
|
||||
}));
|
||||
|
||||
const provider: BlogmarkTransformScriptProvider = {
|
||||
getScripts: vi.fn(async () => [
|
||||
createScript({
|
||||
id: 'pyodide-transform',
|
||||
slug: 'pyodide-transform',
|
||||
title: 'Pyodide Transform',
|
||||
kind: 'transform',
|
||||
entrypoint: 'normalize_blogmark',
|
||||
content: 'def normalize_blogmark(post):\n return post',
|
||||
}),
|
||||
]),
|
||||
it('uses webworker executor when runtime mode resolves to webworker', async () => {
|
||||
const webworkerExecutor: BlogmarkTransformExecutor = {
|
||||
runTransform: vi.fn(async (_script, input) => ({ output: input.post, toasts: [] })),
|
||||
};
|
||||
const mainThreadExecutor: BlogmarkTransformExecutor = {
|
||||
runTransform: vi.fn(async (_script, input) => ({ output: input.post, toasts: [] })),
|
||||
};
|
||||
|
||||
const service = new BlogmarkTransformService({ provider });
|
||||
const service = new BlogmarkTransformService({
|
||||
provider: {
|
||||
getScripts: async () => [createScript({ id: 'worker-script', slug: 'worker-script' })],
|
||||
},
|
||||
resolvePythonRuntimeMode: async () => 'webworker',
|
||||
executors: {
|
||||
webworker: webworkerExecutor,
|
||||
'main-thread': mainThreadExecutor,
|
||||
},
|
||||
});
|
||||
|
||||
const result = await service.applyTransforms(createInput());
|
||||
await service.applyTransforms(createInput());
|
||||
|
||||
const transformInvocationCode = runPythonAsync.mock.calls
|
||||
.map((call) => call[0])
|
||||
.find((code) => typeof code === 'string' && String(code).includes('json.dumps(_result)'));
|
||||
expect(webworkerExecutor.runTransform).toHaveBeenCalledTimes(1);
|
||||
expect(mainThreadExecutor.runTransform).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
expect(result.post.title).toBe('Normalized');
|
||||
expect(result.post.categories).toEqual(['spielelog', 'asides']);
|
||||
expect(result.post.tags).toEqual(['inbox', 'spielen']);
|
||||
expect(transformInvocationCode).toBeDefined();
|
||||
expect(String(transformInvocationCode)).not.toContain('import inspect');
|
||||
expect(String(transformInvocationCode)).toContain('\ntry:\n');
|
||||
expect(String(transformInvocationCode)).toContain('\nexcept TypeError:\n');
|
||||
expect(String(transformInvocationCode)).not.toContain('\n try:\n');
|
||||
expect(String(transformInvocationCode)).not.toContain('\n except TypeError:\n');
|
||||
it('uses main-thread executor when runtime mode resolves to main-thread', async () => {
|
||||
const webworkerExecutor: BlogmarkTransformExecutor = {
|
||||
runTransform: vi.fn(async (_script, input) => ({ output: input.post, toasts: [] })),
|
||||
};
|
||||
const mainThreadExecutor: BlogmarkTransformExecutor = {
|
||||
runTransform: vi.fn(async (_script, input) => ({ output: input.post, toasts: [] })),
|
||||
};
|
||||
|
||||
const service = new BlogmarkTransformService({
|
||||
provider: {
|
||||
getScripts: async () => [createScript({ id: 'main-script', slug: 'main-script' })],
|
||||
},
|
||||
resolvePythonRuntimeMode: async () => 'main-thread',
|
||||
executors: {
|
||||
webworker: webworkerExecutor,
|
||||
'main-thread': mainThreadExecutor,
|
||||
},
|
||||
});
|
||||
|
||||
await service.applyTransforms(createInput());
|
||||
|
||||
expect(mainThreadExecutor.runTransform).toHaveBeenCalledTimes(1);
|
||||
expect(webworkerExecutor.runTransform).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -237,6 +237,42 @@ describe('GitEngine', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('getChangedScriptFilesBetween', () => {
|
||||
it('returns added, modified, deleted and renamed script file changes from name-status output', async () => {
|
||||
mockRaw.mockResolvedValue([
|
||||
'M', 'scripts/existing.py',
|
||||
'A', 'scripts/new_script.py',
|
||||
'D', 'scripts/removed.py',
|
||||
'R100', 'scripts/old_name.py', 'scripts/new_name.py',
|
||||
'M', 'posts/2026/02/ignored.md',
|
||||
].join('\0'));
|
||||
|
||||
const result = await gitEngine.getChangedScriptFilesBetween('/tmp/project', 'before', 'after');
|
||||
|
||||
expect(mockRaw).toHaveBeenCalledWith([
|
||||
'diff',
|
||||
'--name-status',
|
||||
'--find-renames',
|
||||
'-z',
|
||||
'before..after',
|
||||
'--',
|
||||
'scripts',
|
||||
]);
|
||||
|
||||
expect(result).toEqual([
|
||||
{ status: 'modified', path: 'scripts/existing.py' },
|
||||
{ status: 'added', path: 'scripts/new_script.py' },
|
||||
{ status: 'deleted', path: 'scripts/removed.py' },
|
||||
{ status: 'renamed', path: 'scripts/new_name.py', previousPath: 'scripts/old_name.py' },
|
||||
]);
|
||||
});
|
||||
|
||||
it('returns empty changes when refs are empty or identical', async () => {
|
||||
expect(await gitEngine.getChangedScriptFilesBetween('/tmp/project', 'same', 'same')).toEqual([]);
|
||||
expect(await gitEngine.getChangedScriptFilesBetween('/tmp/project', ' ', 'after')).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('getCommitDiffContent', () => {
|
||||
it('should return commit patch text in diff content shape', async () => {
|
||||
mockShow.mockResolvedValue([
|
||||
|
||||
@@ -739,6 +739,30 @@ describe('MetaEngine', () => {
|
||||
expect((metadata as any)?.blogmarkCategory).toBe('article');
|
||||
});
|
||||
|
||||
it('should set and get pythonRuntimeMode in project metadata', async () => {
|
||||
await metaEngine.setProjectMetadata({
|
||||
name: 'My Blog',
|
||||
pythonRuntimeMode: 'main-thread',
|
||||
} as any);
|
||||
|
||||
const metadata = await metaEngine.getProjectMetadata();
|
||||
expect((metadata as any)?.pythonRuntimeMode).toBe('main-thread');
|
||||
});
|
||||
|
||||
it('should persist pythonRuntimeMode to filesystem', async () => {
|
||||
await metaEngine.setProjectMetadata({
|
||||
name: 'Runtime Mode Project',
|
||||
pythonRuntimeMode: 'webworker',
|
||||
} as any);
|
||||
|
||||
const metaDir = metaEngine.getMetaDir();
|
||||
const projectPath = normalizePath(`${metaDir}/project.json`);
|
||||
const content = mockFiles.get(projectPath);
|
||||
const parsed = JSON.parse(content!);
|
||||
|
||||
expect(parsed.pythonRuntimeMode).toBe('webworker');
|
||||
});
|
||||
|
||||
it('should persist blogmarkCategory to filesystem', async () => {
|
||||
await metaEngine.setProjectMetadata({
|
||||
name: 'Test Project',
|
||||
|
||||
@@ -55,6 +55,23 @@ vi.mock('uuid', () => ({
|
||||
}));
|
||||
|
||||
vi.mock('fs/promises', () => ({
|
||||
readdir: vi.fn(async (dirPath: string, options?: { withFileTypes?: boolean }) => {
|
||||
if (options?.withFileTypes) {
|
||||
const files = Array.from((globalThis as any).__mockScriptFiles.keys()) as string[];
|
||||
const names = files
|
||||
.filter((filePath) => filePath.startsWith(`${dirPath}/`))
|
||||
.map((filePath) => filePath.slice(dirPath.length + 1))
|
||||
.filter((name) => !name.includes('/'));
|
||||
|
||||
return names.map((name) => ({
|
||||
name,
|
||||
isDirectory: () => false,
|
||||
isFile: () => true,
|
||||
}));
|
||||
}
|
||||
|
||||
return [];
|
||||
}),
|
||||
readFile: vi.fn(async (filePath: string) => {
|
||||
const value = (globalThis as any).__mockScriptFiles.get(filePath);
|
||||
if (typeof value !== 'string') {
|
||||
@@ -175,4 +192,98 @@ describe('ScriptEngine', () => {
|
||||
expect(loaded?.title).toBe('Metadata Test');
|
||||
expect(loaded?.entrypoint).toBe('render');
|
||||
});
|
||||
|
||||
it('rebuilds scripts from filesystem and applies external file metadata/content', async () => {
|
||||
const scriptPath = '/mock/userData/projects/default/scripts/external_transform.py';
|
||||
mockFiles.set(scriptPath, [
|
||||
'"""',
|
||||
'---',
|
||||
'id: "external-script-id"',
|
||||
'projectId: "default"',
|
||||
'slug: "external_transform"',
|
||||
'title: "External Transform"',
|
||||
'kind: "transform"',
|
||||
'entrypoint: "transform"',
|
||||
'enabled: false',
|
||||
'version: 3',
|
||||
'createdAt: "2026-02-20T10:00:00.000Z"',
|
||||
'updatedAt: "2026-02-21T11:00:00.000Z"',
|
||||
'---',
|
||||
'"""',
|
||||
'def transform(context):',
|
||||
' return context',
|
||||
].join('\n'));
|
||||
|
||||
await scriptEngine.rebuildDatabaseFromFiles();
|
||||
|
||||
const all = await scriptEngine.getAllScripts();
|
||||
expect(all).toHaveLength(1);
|
||||
expect(all[0].id).toBe('external-script-id');
|
||||
expect(all[0].slug).toBe('external_transform');
|
||||
expect(all[0].kind).toBe('transform');
|
||||
expect(all[0].entrypoint).toBe('transform');
|
||||
expect(all[0].enabled).toBe(false);
|
||||
expect(all[0].version).toBe(3);
|
||||
expect(all[0].title).toBe('External Transform');
|
||||
expect(all[0].content).toContain('def transform(context):');
|
||||
});
|
||||
|
||||
it('reconciles git changes for scripts (modify/add/delete)', async () => {
|
||||
const created = await scriptEngine.createScript({
|
||||
title: 'Render Hero',
|
||||
kind: 'macro',
|
||||
content: 'def render(context):\n return {"html": "<h1>Hi</h1>"}',
|
||||
});
|
||||
|
||||
const existingPath = '/repo/scripts/render_hero.py';
|
||||
mockFiles.set(existingPath, [
|
||||
'"""',
|
||||
'---',
|
||||
`id: "${created.id}"`,
|
||||
'projectId: "default"',
|
||||
'slug: "render_hero"',
|
||||
'title: "Render Hero Updated Outside"',
|
||||
'kind: "macro"',
|
||||
'entrypoint: "render"',
|
||||
'enabled: true',
|
||||
'version: 8',
|
||||
'createdAt: "2026-02-20T10:00:00.000Z"',
|
||||
'updatedAt: "2026-02-21T11:00:00.000Z"',
|
||||
'---',
|
||||
'"""',
|
||||
'def render(context):',
|
||||
' return {"html": "<h1>Outside</h1>"}',
|
||||
].join('\n'));
|
||||
|
||||
const addedPath = '/repo/scripts/new_transform.py';
|
||||
mockFiles.set(addedPath, [
|
||||
'"""',
|
||||
'---',
|
||||
'id: "added-script-id"',
|
||||
'projectId: "default"',
|
||||
'slug: "new_transform"',
|
||||
'title: "New Transform"',
|
||||
'kind: "transform"',
|
||||
'entrypoint: "transform"',
|
||||
'enabled: true',
|
||||
'version: 1',
|
||||
'createdAt: "2026-02-22T10:00:00.000Z"',
|
||||
'updatedAt: "2026-02-22T11:00:00.000Z"',
|
||||
'---',
|
||||
'"""',
|
||||
'def transform(context):',
|
||||
' return context',
|
||||
].join('\n'));
|
||||
|
||||
const result = await scriptEngine.reconcileScriptsFromGitChanges('/repo', [
|
||||
{ status: 'modified', path: 'scripts/render_hero.py' },
|
||||
{ status: 'added', path: 'scripts/new_transform.py' },
|
||||
{ status: 'deleted', path: 'scripts/render_hero.py' },
|
||||
]);
|
||||
|
||||
expect(result.updated).toBe(1);
|
||||
expect(result.created).toBe(1);
|
||||
expect(result.deleted).toBe(1);
|
||||
expect(result.processedFiles).toBe(3);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -166,12 +166,15 @@ const mockScriptEngine = {
|
||||
deleteScript: vi.fn(),
|
||||
getScript: vi.fn(),
|
||||
getAllScripts: vi.fn(),
|
||||
rebuildDatabaseFromFiles: vi.fn(),
|
||||
reconcileScriptsFromGitChanges: vi.fn(),
|
||||
};
|
||||
|
||||
const mockGitEngine = {
|
||||
checkAvailability: vi.fn(),
|
||||
getHeadCommit: vi.fn(),
|
||||
getChangedPostFilesBetween: vi.fn(),
|
||||
getChangedScriptFilesBetween: vi.fn(),
|
||||
getRepoState: vi.fn(),
|
||||
getStatus: vi.fn(),
|
||||
getDiff: vi.fn(),
|
||||
@@ -575,12 +578,21 @@ describe('IPC Handlers', () => {
|
||||
{ status: 'modified', path: 'posts/2026/02/existing.md' },
|
||||
{ status: 'added', path: 'posts/2026/02/new-post.md' },
|
||||
]);
|
||||
mockGitEngine.getChangedScriptFilesBetween.mockResolvedValue([
|
||||
{ status: 'modified', path: 'scripts/transform.py' },
|
||||
]);
|
||||
mockPostEngine.reconcilePublishedPostsFromGitChanges.mockResolvedValue({
|
||||
created: 1,
|
||||
updated: 1,
|
||||
deleted: 0,
|
||||
processedFiles: 2,
|
||||
});
|
||||
mockScriptEngine.reconcileScriptsFromGitChanges.mockResolvedValue({
|
||||
created: 0,
|
||||
updated: 1,
|
||||
deleted: 0,
|
||||
processedFiles: 1,
|
||||
});
|
||||
|
||||
const result = await invokeHandler('git:pull', '/repo');
|
||||
|
||||
@@ -588,10 +600,14 @@ describe('IPC Handlers', () => {
|
||||
expect(mockGitEngine.pull).toHaveBeenCalledWith('/repo');
|
||||
expect(mockGitEngine.getHeadCommit).toHaveBeenNthCalledWith(2, '/repo');
|
||||
expect(mockGitEngine.getChangedPostFilesBetween).toHaveBeenCalledWith('/repo', 'before-head', 'after-head');
|
||||
expect(mockGitEngine.getChangedScriptFilesBetween).toHaveBeenCalledWith('/repo', 'before-head', 'after-head');
|
||||
expect(mockPostEngine.reconcilePublishedPostsFromGitChanges).toHaveBeenCalledWith('/repo', [
|
||||
{ status: 'modified', path: 'posts/2026/02/existing.md' },
|
||||
{ status: 'added', path: 'posts/2026/02/new-post.md' },
|
||||
]);
|
||||
expect(mockScriptEngine.reconcileScriptsFromGitChanges).toHaveBeenCalledWith('/repo', [
|
||||
{ status: 'modified', path: 'scripts/transform.py' },
|
||||
]);
|
||||
expect(result).toEqual({ success: true });
|
||||
});
|
||||
|
||||
@@ -603,7 +619,9 @@ describe('IPC Handlers', () => {
|
||||
|
||||
expect(mockGitEngine.pull).toHaveBeenCalledWith('/repo');
|
||||
expect(mockGitEngine.getChangedPostFilesBetween).not.toHaveBeenCalled();
|
||||
expect(mockGitEngine.getChangedScriptFilesBetween).not.toHaveBeenCalled();
|
||||
expect(mockPostEngine.reconcilePublishedPostsFromGitChanges).not.toHaveBeenCalled();
|
||||
expect(mockScriptEngine.reconcileScriptsFromGitChanges).not.toHaveBeenCalled();
|
||||
expect(result).toEqual({ success: false, code: 'conflict' });
|
||||
});
|
||||
|
||||
@@ -617,7 +635,9 @@ describe('IPC Handlers', () => {
|
||||
|
||||
expect(mockGitEngine.pull).toHaveBeenCalledWith('/repo');
|
||||
expect(mockGitEngine.getChangedPostFilesBetween).not.toHaveBeenCalled();
|
||||
expect(mockGitEngine.getChangedScriptFilesBetween).not.toHaveBeenCalled();
|
||||
expect(mockPostEngine.reconcilePublishedPostsFromGitChanges).not.toHaveBeenCalled();
|
||||
expect(mockScriptEngine.reconcileScriptsFromGitChanges).not.toHaveBeenCalled();
|
||||
expect(result).toEqual({ success: true });
|
||||
});
|
||||
});
|
||||
@@ -2712,6 +2732,23 @@ describe('IPC Handlers', () => {
|
||||
expect(result).toEqual(expected);
|
||||
});
|
||||
});
|
||||
|
||||
describe('scripts:rebuildFromFiles', () => {
|
||||
it('should set project context and trigger ScriptEngine rebuild', async () => {
|
||||
mockProjectEngine.getActiveProject.mockResolvedValue({
|
||||
id: 'project-1',
|
||||
dataPath: '/external/data',
|
||||
});
|
||||
mockProjectEngine.getDataDir.mockReturnValue('/resolved/project-data');
|
||||
mockScriptEngine.rebuildDatabaseFromFiles.mockResolvedValue(undefined);
|
||||
|
||||
const result = await invokeHandler('scripts:rebuildFromFiles');
|
||||
|
||||
expect(mockScriptEngine.setProjectContext).toHaveBeenCalledWith('project-1', '/resolved/project-data');
|
||||
expect(mockScriptEngine.rebuildDatabaseFromFiles).toHaveBeenCalled();
|
||||
expect(result).toBe(true);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// ============ Error Handling ============
|
||||
|
||||
@@ -117,6 +117,33 @@ describe('SettingsView Diff Preferences', () => {
|
||||
);
|
||||
});
|
||||
|
||||
it('includes python runtime mode in metadata save payload', async () => {
|
||||
(window as any).electronAPI.meta.getProjectMetadata = vi.fn().mockResolvedValue({
|
||||
maxPostsPerPage: 75,
|
||||
publicUrl: 'https://example.com',
|
||||
pythonRuntimeMode: 'main-thread',
|
||||
categorySettings: {
|
||||
article: { renderInLists: true, showTitle: true },
|
||||
picture: { renderInLists: true, showTitle: true },
|
||||
aside: { renderInLists: true, showTitle: false },
|
||||
page: { renderInLists: false, showTitle: true },
|
||||
},
|
||||
});
|
||||
|
||||
render(<SettingsView />);
|
||||
|
||||
await screen.findByDisplayValue('Main Thread (Legacy)');
|
||||
|
||||
const saveButton = screen.getByRole('button', { name: /save project settings/i });
|
||||
fireEvent.click(saveButton);
|
||||
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
|
||||
expect((window as any).electronAPI.meta.updateProjectMetadata).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ pythonRuntimeMode: 'main-thread' })
|
||||
);
|
||||
});
|
||||
|
||||
it('renders category settings checkboxes with required defaults', async () => {
|
||||
render(<SettingsView />);
|
||||
|
||||
@@ -131,6 +158,26 @@ describe('SettingsView Diff Preferences', () => {
|
||||
expect((articleShowTitle as HTMLInputElement).checked).toBe(true);
|
||||
});
|
||||
|
||||
it('triggers scripts rebuild from data maintenance section', async () => {
|
||||
const rebuildScriptsMock = vi.fn().mockResolvedValue(undefined);
|
||||
(window as any).electronAPI = {
|
||||
...(window as any).electronAPI,
|
||||
scripts: {
|
||||
...(window as any).electronAPI?.scripts,
|
||||
rebuildFromFiles: rebuildScriptsMock,
|
||||
},
|
||||
};
|
||||
|
||||
render(<SettingsView />);
|
||||
|
||||
const rebuildScriptsButton = await screen.findByRole('button', { name: /rebuild scripts/i });
|
||||
fireEvent.click(rebuildScriptsButton);
|
||||
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
|
||||
expect(rebuildScriptsMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('persists category settings changes via project metadata update', async () => {
|
||||
render(<SettingsView />);
|
||||
|
||||
|
||||
@@ -191,6 +191,32 @@ describe('PythonRuntimeManager', () => {
|
||||
await expect(runPromise).rejects.toThrow('boom');
|
||||
});
|
||||
|
||||
it('queues concurrent execute calls and sends the next request only after completion', async () => {
|
||||
const worker = new MockWorker();
|
||||
const manager = new PythonRuntimeManager(() => worker as unknown as Worker);
|
||||
|
||||
const initPromise = manager.initialize();
|
||||
worker.emitMessage({ type: 'ready' });
|
||||
await initPromise;
|
||||
|
||||
const firstRun = manager.execute('1 + 1');
|
||||
const secondRun = manager.execute('2 + 2');
|
||||
await Promise.resolve();
|
||||
|
||||
expect(worker.postedMessages).toHaveLength(1);
|
||||
|
||||
const firstRequest = worker.postedMessages[0] as { requestId: string };
|
||||
worker.emitMessage({ type: 'runResult', requestId: firstRequest.requestId, result: '2' });
|
||||
await expect(firstRun).resolves.toEqual({ result: '2', stdout: '' });
|
||||
|
||||
await Promise.resolve();
|
||||
expect(worker.postedMessages).toHaveLength(2);
|
||||
|
||||
const secondRequest = worker.postedMessages[1] as { requestId: string };
|
||||
worker.emitMessage({ type: 'runResult', requestId: secondRequest.requestId, result: '4' });
|
||||
await expect(secondRun).resolves.toEqual({ result: '4', stdout: '' });
|
||||
});
|
||||
|
||||
it('terminates timed out worker and recovers with a new worker', async () => {
|
||||
const workers: MockWorker[] = [];
|
||||
const manager = new PythonRuntimeManager(() => {
|
||||
|
||||
Reference in New Issue
Block a user