MLXServer/docs/chat-document-plan.md

Chat Document Format And Implementation Plan

Recommendation

Use a custom macOS document package with the extension .mlxchat.

• User-facing behavior: appears as a single document in Finder and standard Open/Save panels.
• Internal structure: a file package directory containing a versioned JSON manifest plus binary attachments.
• Primary app integration: keep the current WindowGroup app structure initially, add explicit Open/Save/Revert flows, and only consider a later move to DocumentGroup if document-centric window management becomes a product goal.

This is the best fit for MLX Server because it needs to preserve more than transcript text:

• full message history
• pasted or dragged images
• assistant thinking blocks
• model and generation context
• future tool calls and tool results
• enough metadata to migrate old documents safely

Why Not Other Formats

Flat JSON file

Good for text-only chats. Weak once images and future binary artifacts need to travel with the conversation.

ZIP container

Portable, but less native on macOS. Finder and AppKit document workflows work better with a package document.

SQLite

Reasonable for an internal database, but a poor user-facing interchange format for a single chat document.

Binary plist or NSKeyedArchiver

Too opaque and too brittle for a long-lived app document format.

Proposed Package Layout

Conversation.mlxchat/
  manifest.json
  attachments/
    9E8B7C7B-3F6E-4F77-8D17-0C7C0E8D5E62.png
    91A2A1D4-55C8-4D0D-BECB-5A2B5D276D8B.jpg
  previews/
    thumbnail.png

Notes:

• manifest.json is the canonical source of truth.
• attachments/ contains lossless binary payloads referenced by manifest IDs.
• previews/thumbnail.png is optional and can be added later for Quick Look or Finder previews.

UTType And Document Identity

Define a custom exported type.

• Identifier: de.rfc1437.mlxserver.chat
• Conforms to: com.apple.package, public.data
• Filename extension: mlxchat
• Display name: MLX Server Chat Document

In this project, that will require updates in project.yml so the generated Info.plist exports the document type.

Manifest Schema

The manifest should stay plain JSON and be versioned from day one.

Top-level shape

{
  "schemaVersion": 1,
  "documentId": "1F8A10B6-7D1B-4D2B-9D3B-18B9818D3C17",
  "createdAt": "2026-03-18T12:00:00Z",
  "updatedAt": "2026-03-18T12:34:56Z",
  "appVersion": "1.0.0",
  "model": {
    "id": "qwen3.5-9b",
    "displayName": "Qwen3.5 9B",
    "repoId": "mlx-community/Qwen3.5-9B-4bit"
  },
  "settings": {
    "systemPrompt": "",
    "thinkingEnabled": true,
    "temperature": 0.7
  },
  "messages": [],
  "toolEvents": [],
  "uiState": {
    "draftInput": "",
    "scrollAnchorMessageId": null
  }
}

Message shape

{
  "id": "7A0D764B-D40D-4A7E-9655-441EF79C2B44",
  "role": "assistant",
  "createdAt": "2026-03-18T12:01:00Z",
  "content": "Visible response text",
  "rawContent": "<think>reasoning</think>Visible response text",
  "thinkingContent": "reasoning",
  "streamingState": "completed",
  "attachments": [
    {
      "id": "9E8B7C7B-3F6E-4F77-8D17-0C7C0E8D5E62",
      "type": "image",
      "relativePath": "attachments/9E8B7C7B-3F6E-4F77-8D17-0C7C0E8D5E62.png",
      "mimeType": "image/png",
      "pixelWidth": 1024,
      "pixelHeight": 768,
      "sha256": "..."
    }
  ]
}

Schema rules

• Never serialize NSImage directly.
• Store timestamps in ISO 8601 UTC.
• Store stable UUIDs for document, messages, and attachments.
• Treat rawContent as canonical for assistant messages so future parsing changes do not destroy original output.
• Keep thinkingContent denormalized for fast restore and forward compatibility.
• Ignore unknown keys on read so older app versions degrade gracefully.

State To Preserve

Minimum state required for a true restore:

• all Conversation.messages
• message timestamps
• user-attached images
• assistant rawContent, content, thinkingContent, and streaming completion state
• selected model identity at the time of the chat
• relevant generation settings used to produce the conversation
• current system prompt

State that should not be persisted initially:

• active ChatSession or KV cache
• live token counters
• running API server state
• in-flight generation tasks

Restored chats should reproduce visible history, not attempt to resume an MLX runtime session.

Required Refactor Before Saving

The current chat model is UI-oriented and not suitable as the storage contract.

Relevant existing code:

• MLXServer/Models/ChatMessage.swift
• MLXServer/ViewModels/ChatViewModel.swift
• MLXServer/Utilities/ChatExporter.swift
• MLXServer/ContentView.swift

Problem areas in the current model

• ChatMessage.id is regenerated on every launch because it is not decoded from persisted data.
• timestamp is assigned in the initializer and cannot currently be restored.
• images uses NSImage, which is a view/runtime type rather than a storage type.
• Conversation has no import/export path other than Markdown and RTF export.

Recommended split

Introduce two distinct layers:

1. Persistent document model
2. Runtime/UI model

Suggested types:

• ChatDocumentPackage: FileDocument implementation for .mlxchat
• ChatDocumentManifest: Codable root object
• StoredChatMessage: Codable message payload
• StoredAttachment: Codable attachment metadata
• ChatAttachment: runtime representation with file URL and lazily loaded image

This keeps the JSON schema stable while allowing UI and MLX runtime concerns to evolve independently.

Swift API Design

New files

• MLXServer/Documents/ChatDocumentPackage.swift
• MLXServer/Documents/ChatDocumentManifest.swift
• MLXServer/Documents/ChatDocumentMigration.swift

Adjust existing files

• MLXServer/Models/ChatMessage.swift: make the runtime model restorable from stored values and stop using implicit-only timestamps for persisted messages.
• MLXServer/ViewModels/ChatViewModel.swift: add load/save hooks and dirty-state tracking.
• MLXServer/ContentView.swift: add open/save/save-as/revert/import wiring.
• MLXServer/Commands/SaveChatCommands.swift: expand from export-only commands to proper document commands.
• project.yml: register exported document type and include new source files.

FileDocument responsibilities

Write path:

• serialize manifest JSON
• encode attachments into deterministic filenames
• build package directory with FileWrapper(directoryWithFileWrappers:)

Read path:

• validate package structure
• decode manifest
• resolve referenced attachments
• migrate old schema versions before handing data to the app

Integration Strategy

Phase 1: Add document support without changing the app architecture

Recommended first implementation.

• Keep MLXServerApp as a WindowGroup app.
• Add Open Chat…, Save Chat…, Save Chat As…, and Revert To Saved commands.
• Use fileImporter and fileExporter from the current main window.
• Track the current document URL and dirty state inside ChatViewModel.

Why this first:

• minimal disruption to model loading and existing window behavior
• faster path to a working restore feature
• avoids forcing document-window semantics into the app before the document model is proven

Required view model additions

Add to ChatViewModel:

• currentDocumentURL: URL?
• hasUnsavedChanges: Bool
• lastSavedSnapshotHash: String?
• loadDocument(from:)
• saveDocument(to:)
• markDirtyIfNeeded()

Mark the conversation dirty when:

• a message is added
• an attachment is added or removed
• streamed assistant output is finalized
• document-level settings that belong in the manifest change

Phase 2: Consider migration to DocumentGroup

Optional and only worth doing if you want standard multi-document macOS behavior.

Benefits:

• native new/open/reopen document lifecycle
• automatic recent documents support
• better fit for multiple chat windows

Costs:

• larger refactor to inject ModelManager into document scenes cleanly
• more work to preserve current app-global settings and API server assumptions

Recommendation: do not start here.

Save Semantics

Use standard desktop-document behavior.

• Save: overwrite current .mlxchat if the chat already has a document URL.
• Save As: always prompt for a new destination.
• Open: replace the current conversation after unsaved-changes confirmation.
• New Chat: if dirty, prompt before clearing.
• autosave: optional second step after manual save/open works reliably.

Unsaved change detection can be implemented by hashing a canonical serialized manifest without volatile fields such as updatedAt.

Attachment Handling

Store images as files in attachments/, not inline base64 JSON.

Recommended rules:

• Prefer PNG for screenshots and lossless clipboard images.
• Preserve original file format when importing from disk if feasible.
• Compute SHA-256 for deduplication and integrity checks.
• Do not rely on original absolute file paths after import.
• Restore images from document-local paths only.

Runtime API suggestion:

• ChatAttachment.imageURL
• ChatAttachment.loadNSImage()

This avoids embedding heavy NSImage values in the persistence model.

Migration Strategy

Implement migrations immediately, even with only one schema version.

• schemaVersion = 1 for the first release
• central migration entry point that upgrades any older manifest to current model structs
• log and surface a clear user error if a future version is unsupported

This prevents format lock-in and avoids dangerous ad hoc decode logic later.

Error Handling

User-facing errors should distinguish between:

• invalid document package
• missing manifest
• corrupted attachment reference
• unsupported schema version
• failed attachment decode

Do not partially mutate the current conversation until a full document read succeeds.

Testing Plan

Add focused tests for document round-tripping.

Unit tests

• manifest encode/decode round trip
• message round trip with thinking content
• attachment reference round trip
• migration from schema version 1 fixture
• dirty-state hash stability

Integration tests

• save a chat with images, reopen it, and verify visible conversation equality
• save a chat with assistant thinking blocks, reopen it, and verify both visible and thinking content
• open a malformed package and verify the app shows a recoverable error

Manual verification

• create a new chat, save as .mlxchat, quit app, reopen document, verify full restore
• duplicate the file in Finder and open the copy
• rename the file in Finder and reopen
• test a large image attachment document

Implementation Sequence

1. Introduce Codable stored document types and a versioned manifest.
2. Refactor runtime chat state so messages and attachments can be reconstructed from stored data.
3. Implement .mlxchat package read/write using FileDocument.
4. Register UTType and document type metadata in project.yml.
5. Add open/save/save-as/revert commands and wire them into MLXServer/ContentView.swift and MLXServer/Commands/SaveChatCommands.swift.
6. Add unsaved-change prompts around new/open flows.
7. Keep Markdown/RTF export as a separate export feature, not the primary restore format.
8. Add round-trip tests and a small set of fixture documents.

Concrete Recommendation For This Codebase

Implement .mlxchat as a package document now, but keep the current WindowGroup app structure for the first pass.

That gives MLX Server a native macOS document format without taking on an unnecessary app-architecture rewrite. Once the format and save/open behavior are stable, reassess whether the product would benefit from a full DocumentGroup migration.