mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-27 19:35:13 +01:00
DottaandPaperclip
125ddcabe5
## Thinking Path > - Paperclip lets people manage AI agents and their work. > - The HTTP server also owns durable Runner state. > - Each committed event rewrites and flushes that state on the HTTP thread. > - Large journals and slow writes delay unrelated navigation and saves. > - Event acknowledgements must still wait for durable storage. > - This PR stages output writes asynchronously and joins them before writer replacement. > - The board can serve requests while Runner output waits for storage. ## Linked Issues or Issue Description Refs: #17507. Related: #17514 reduces writes per event burst. This PR moves individual output writes and file flushes outside the HTTP thread. **What happened?** A live CPU profile spent about three seconds of a 13-second sample serializing and synchronously writing Runner state. Representative journals were about 9 MB. Unrelated task requests had multi-second delays. **Expected behavior** A slow output-journal write must leave unrelated HTTP requests responsive. An event must not be acknowledged before it is durable. **Steps to reproduce** Hold an authenticated output event's file flush in the durable protocol test. Send an unrelated HTTP request during that flush. Also save a command, reconnect, or retire the controller during the write. ## What Changed - Stage `item.delta` snapshots with asynchronous file writes and file flushes. Keep lifecycle, control, authentication, and tool events synchronous so transport readers observe durable boundaries. - Fence publication by saved revision. An older snapshot cannot replace newer commands or connection state. A failed write fences the writer. - Write compact JSON and share private-file checks. Existing journals remain readable. - Stop external ingress before joining admitted writes. Checkpoint only after retirement succeeds. Close reports a failed detach consistently and releases notification consumers. - Add real HTTP and Rust transport cases for held writes, reconnects, commands, failure, restart, close, and timeout. - Add immutable Copilot source identity v61 for the changed shared transport. Preserve v60, historical decoding, exact-current launch checks, and qualification flags. No new live Copilot qualification is claimed. ## Verification - Head `6f5c92e778`: workspace `pnpm -r typecheck` and `pnpm build` passed. All CI gates passed in [run 40356159484](https://github.com/paperclipai/paperclip/actions/runs/40356159484), including both full Runner shards. Fresh Greptile review is 5/5; all findings are resolved. - Current-base focused verification passed 300 cases across protocol, real Rust restart/continuation, permission, backend, and immutable-profile suites. The complete transport suite also passed 286 cases before the base refresh. - Earlier iterations exposed lifecycle visibility, late writer publication, and close-order failures. Controlled regressions reproduced each before repair. CI also found missing test-peer drain methods and a stale source fingerprint; both are corrected. An older broad local test run was stopped after its source was superseded. No full local-suite pass is claimed; full coverage passed in isolated CI. - A synthetic 9 MB journal comparison used 20 writes per sample and concurrent HTTP probes, with alternating baseline/treatment order. HTTP median changed from 31 ms to 2–3 ms; p95 changed from 51–85 ms to 12–21 ms. Parsed output was equal. The asynchronous loop served 125–153 probes versus 19–20, and took longer overall (992–1069 ms versus 675–678 ms). This proves local I/O responsiveness, not deployed application latency. - Deployment and live save/archive acceptance remain pending. ## Risks Persistence interleavings affect restart authority. The tests cover concurrent commands, replacement, failed storage, and reopening the journal. JSON serialization, atomic rename, directory flush, and control-event writes remain synchronous. This change still rewrites whole journals. Incomplete retirement retains recovery evidence and fails the close. A new exact-current Copilot source identity can reject an older launch identity; historical decoding remains available. No migration is required. ## Model Used OpenAI GPT-6 (Codex), with reasoning, tool use, SSH and browser diagnostics, code editing, and test execution. The exact serving model suffix and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>