Add DeepSeek-style hierarchical AGENTS.md instruction lifecycle #74

New Issue

hugo · 2026-08-29T17:56:05Z

hugo commented

2026-08-29 17:56:05 +00:00

Goal

Upgrade the root-only AGENTS.md support delivered by #29 into the durable, hierarchical workspace-instruction lifecycle used by DeepSeek Harness, adapted to DS4Server's Rust-native session and tool architecture.

Today src/app/generation.rs reads only <project>/AGENTS.md on the opening turn and reuses that frozen text. It has no nested scope discovery, update/removal handling, prompt budget, or resume reconciliation.

Required behavior

Discovery and precedence

Load the user-global <Application Support>/de.rfc1437.ds4server/AGENTS.md, then every applicable project instruction from the selected project root to the session working directory, broadest to most specific.
In each project directory load AGENTS.md, then optional AGENTS.local.md. More-specific files and local overlays take precedence, while system/developer/direct-user instructions remain authoritative.
After a successful structured read, write, or edit reaches a deeper directory, discover every newly crossed scope before the next model request. Do not infer filesystem activity by parsing bash commands.
Suppress trimmed-content duplicates within one directory and unchanged content across turns. Keep discovery and state isolated per session.
Project instruction files and resolved symlinks must remain inside the selected project root; the explicit application-support file is the only user-global source.

Durable lifecycle

Put the initial instruction chain into the first request as one hidden, durable conversation message with structured source metadata (path, scope, action, digest), not as mutable process-global state.
Persist enough metadata to reconstruct the visible instruction state after app restart. Resume must reconcile the current files with durable history and append a complete replacement only when the baseline is incompatible.
A newly reached scope appends an Additional instructions from: <path> message. A changed file appends an explicit replacement; a confirmed deletion appends a removal notice. Temporary read/stat failures do not mean removal.
Re-arm compacted instructions when they are no longer present in the model-visible surface. Tool-result reconciliation must commit only after the successful tool result itself is durable.
Include the effective baseline identity in KV/prompt identity. Later changes remain append-only so existing prefix cache entries stay reusable; new sessions use current file content.

Rendering and limits

Use a model-visible <system-reminder> frame that names every source, explains broad-to-specific precedence, and states that workspace files cannot override higher-authority instructions. Escape literal closing reminder tags from file content.
Cap each source read at 1 MiB and each rendered baseline/change batch at 64 KiB, measured as UTF-8 bytes. Under pressure, omit whole broader files before UTF-8-safe truncation of the most-specific file, and include a bounded diagnostic naming omitted/truncated paths.
Missing or empty instruction chains add no prompt content. Unreadable, oversized, or malformed candidates fail closed without blocking the user turn; surface a concise diagnostic without injecting partial untracked state.

Implementation plan

Extract the current ad-hoc loader from src/app/generation.rs into a small Rust instruction module responsible for discovery, bounded reads, rendering, digests, and reconciliation.
Extend persisted hidden-message metadata so instruction baselines and set/replace/remove transitions are distinguishable without parsing prompt prose.
Have successful path-aware filesystem tools report normalized touched paths through the existing tool-result lifecycle; reconcile those paths after the result commits and before the next generation step.
Integrate baseline/replacement messages with session opening, continuation, compaction, resume, system-prompt reminders, and content-addressed KV identity. Preserve first-title behavior and keep instruction messages out of the visible transcript.
Add focused tests for ordering/precedence, global/root/local/nested scopes, duplicate suppression, UTF-8 budgets, frame escaping, root confinement, unavailable files, successful versus failed tools, change/removal notices, per-session isolation, restart/resume, compaction recovery, and KV reuse/invalidation.

Boundaries

Rust only; do not add JavaScript, a DeepSeek Harness runtime dependency, a file watcher, shell-command parsing, or an external agent process.
Support AGENTS.md and AGENTS.local.md only. CLAUDE.md, imports, lowercase aliases, and generic workflow configuration are out of scope.
Instruction text is guidance, never a filesystem sandbox or permission boundary; existing tool path and approval enforcement remains authoritative.

Acceptance criteria

The first model request receives the bounded global-to-working-directory chain with deterministic broad-to-specific ordering.
Entering a nested path through read/write/edit makes its instructions effective on the next request exactly once; edits and confirmed removals produce durable replacement/removal messages.
Restart, resume, compaction, queued prompts, and tool continuations retain or correctly restore the effective instruction state without stale root-only text or duplicate injections.
Prompt framing cannot be closed by repository content, all reads/renders respect their byte caps, and project sources cannot escape the selected root.
Existing #29 behavior remains intact: no model work before first submission, instructions are hidden from the transcript/title input, missing files are harmless, and compatible deterministic prefixes reuse KV cache.
cargo fmt --all -- --check, Clippy with -D warnings, make bundle, and cargo test --all-features pass.

## Goal Upgrade the root-only `AGENTS.md` support delivered by #29 into the durable, hierarchical workspace-instruction lifecycle used by [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness/tree/cd5ef8148158c3a752a658978873241fdf8e2bbc/packages/context/agent-instructions), adapted to DS4Server's Rust-native session and tool architecture. Today `src/app/generation.rs` reads only `<project>/AGENTS.md` on the opening turn and reuses that frozen text. It has no nested scope discovery, update/removal handling, prompt budget, or resume reconciliation. ## Required behavior ### Discovery and precedence - Load the user-global `<Application Support>/de.rfc1437.ds4server/AGENTS.md`, then every applicable project instruction from the selected project root to the session working directory, broadest to most specific. - In each project directory load `AGENTS.md`, then optional `AGENTS.local.md`. More-specific files and local overlays take precedence, while system/developer/direct-user instructions remain authoritative. - After a successful structured `read`, `write`, or `edit` reaches a deeper directory, discover every newly crossed scope before the next model request. Do not infer filesystem activity by parsing `bash` commands. - Suppress trimmed-content duplicates within one directory and unchanged content across turns. Keep discovery and state isolated per session. - Project instruction files and resolved symlinks must remain inside the selected project root; the explicit application-support file is the only user-global source. ### Durable lifecycle - Put the initial instruction chain into the first request as one hidden, durable conversation message with structured source metadata (`path`, scope, action, digest), not as mutable process-global state. - Persist enough metadata to reconstruct the visible instruction state after app restart. Resume must reconcile the current files with durable history and append a complete replacement only when the baseline is incompatible. - A newly reached scope appends an `Additional instructions from: <path>` message. A changed file appends an explicit replacement; a confirmed deletion appends a removal notice. Temporary read/stat failures do not mean removal. - Re-arm compacted instructions when they are no longer present in the model-visible surface. Tool-result reconciliation must commit only after the successful tool result itself is durable. - Include the effective baseline identity in KV/prompt identity. Later changes remain append-only so existing prefix cache entries stay reusable; new sessions use current file content. ### Rendering and limits - Use a model-visible `<system-reminder>` frame that names every source, explains broad-to-specific precedence, and states that workspace files cannot override higher-authority instructions. Escape literal closing reminder tags from file content. - Cap each source read at 1 MiB and each rendered baseline/change batch at 64 KiB, measured as UTF-8 bytes. Under pressure, omit whole broader files before UTF-8-safe truncation of the most-specific file, and include a bounded diagnostic naming omitted/truncated paths. - Missing or empty instruction chains add no prompt content. Unreadable, oversized, or malformed candidates fail closed without blocking the user turn; surface a concise diagnostic without injecting partial untracked state. ## Implementation plan 1. Extract the current ad-hoc loader from `src/app/generation.rs` into a small Rust instruction module responsible for discovery, bounded reads, rendering, digests, and reconciliation. 2. Extend persisted hidden-message metadata so instruction baselines and `set`/`replace`/`remove` transitions are distinguishable without parsing prompt prose. 3. Have successful path-aware filesystem tools report normalized touched paths through the existing tool-result lifecycle; reconcile those paths after the result commits and before the next generation step. 4. Integrate baseline/replacement messages with session opening, continuation, compaction, resume, system-prompt reminders, and content-addressed KV identity. Preserve first-title behavior and keep instruction messages out of the visible transcript. 5. Add focused tests for ordering/precedence, global/root/local/nested scopes, duplicate suppression, UTF-8 budgets, frame escaping, root confinement, unavailable files, successful versus failed tools, change/removal notices, per-session isolation, restart/resume, compaction recovery, and KV reuse/invalidation. ## Boundaries - Rust only; do not add JavaScript, a DeepSeek Harness runtime dependency, a file watcher, shell-command parsing, or an external agent process. - Support `AGENTS.md` and `AGENTS.local.md` only. `CLAUDE.md`, imports, lowercase aliases, and generic workflow configuration are out of scope. - Instruction text is guidance, never a filesystem sandbox or permission boundary; existing tool path and approval enforcement remains authoritative. ## Acceptance criteria - The first model request receives the bounded global-to-working-directory chain with deterministic broad-to-specific ordering. - Entering a nested path through `read`/`write`/`edit` makes its instructions effective on the next request exactly once; edits and confirmed removals produce durable replacement/removal messages. - Restart, resume, compaction, queued prompts, and tool continuations retain or correctly restore the effective instruction state without stale root-only text or duplicate injections. - Prompt framing cannot be closed by repository content, all reads/renders respect their byte caps, and project sources cannot escape the selected root. - Existing #29 behavior remains intact: no model work before first submission, instructions are hidden from the transcript/title input, missing files are harmless, and compatible deterministic prefixes reuse KV cache. - `cargo fmt --all -- --check`, Clippy with `-D warnings`, `make bundle`, and `cargo test --all-features` pass.

hugo added the enhancement label 2026-08-29 17:56:05 +00:00

hugo commented

2026-08-29 19:57:38 +00:00

Implemented in commit 1597ddb.

Added a Rust-native hierarchical AGENTS.md/AGENTS.local.md reconciler with global-to-local precedence, source confinement, duplicate suppression, structured digests/actions, 1 MiB source limits, 64 KiB UTF-8-safe prompt batches, escaped system-reminder framing, and bounded diagnostics.
Persisted hidden instruction metadata for restart/resume, append-only changes, replacements/removals, and compaction re-arming.
Made successful structured read/write/edit tools report normalized project paths. Tool results commit first; reached-scope instruction transitions commit durably before the next model operation.
Kept instruction rows out of visible transcripts and title input while including their identity in content-addressed KV history.

Verified in isolation:

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
make bundle
cargo test --all-features (186 passed, 14 ignored by existing hardware/browser annotations)

Implemented in commit 1597ddb. - Added a Rust-native hierarchical AGENTS.md/AGENTS.local.md reconciler with global-to-local precedence, source confinement, duplicate suppression, structured digests/actions, 1 MiB source limits, 64 KiB UTF-8-safe prompt batches, escaped system-reminder framing, and bounded diagnostics. - Persisted hidden instruction metadata for restart/resume, append-only changes, replacements/removals, and compaction re-arming. - Made successful structured read/write/edit tools report normalized project paths. Tool results commit first; reached-scope instruction transitions commit durably before the next model operation. - Kept instruction rows out of visible transcripts and title input while including their identity in content-addressed KV history. Verified in isolation: - cargo fmt --all -- --check - cargo clippy --all-targets --all-features -- -D warnings - make bundle - cargo test --all-features (186 passed, 14 ignored by existing hardware/browser annotations)

hugo closed this issue

2026-08-29 19:57:42 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/DS4Server#74