127 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	33ca62f776	Keep external-agent feature reads on their admission transaction (#19750) ... ## Thinking Path > - Paperclip manages work for AI agents. > - External native agents must pass current feature and authority checks. > - The broker checks feature settings inside its admission transaction. > - The old feature reader uses the root database pool. > - Two active transactions can leave no connection for that read. > - This change uses the admitted transaction for the same policy read. > - It keeps the existing company, principal and controller checks. ## Linked Issues or Issue Description Refs: #19340. That merged change fixes retained native Retry ownership. This change leaves that recovery contract intact. Related: #18744 and #17833 fix caller-owned transaction reads in other domains. Root searched live GitHub for open externalBotEnabled changes and found no duplicate. **What happened?** An external-agent admission can wait for another pool connection while it holds a row lock. A second transaction can occupy the remaining connection and wait for that lock. The feature read must use the transaction that already owns admission. **Expected behavior** Dot, Grok and Muse read current feature settings on the admitted transaction. Disabled providers remain disabled. Current authority checks remain in force. **Steps to reproduce** 1. Use a real PostgreSQL pool with two connections. 2. Lock the settings row in the admission transaction. 3. Occupy the second connection with a transaction that waits for that row. 4. Read the existing provider feature policy through the admitted owner. 5. Require the read to finish before the competing lock is acquired. **Paperclip version or commit** The fix was tested at `d8dece250b75cffee67d3b7f4f0e669826282164`, based on master `839d41aabbc6d042250548aedcaa5c50d369dca3`. It now includes master `131b566ca4fa866c4348739c75cfe48c805b3398` through normal merge `c2031b0fb3e5d1b34ebff1d5af8cc050d129bd9a`. All seven fix files retain their tested blobs and modes. ## What Changed - Pass the admitted select owner through the broker identity contract. - Use that owner for Dot, Grok and Muse feature settings. - Add a read-only settings projection that reuses the existing managed and staging view. It does not create the settings row. - Add six real PostgreSQL controls for enabled and disabled providers. - Add three broker-path PostgreSQL regressions for exact transaction forwarding through Dot, Grok and Muse. - Keep all newer master broker changes and existing authority checks. ## Verification Local controlled regression passed at tested revision `d8dece250b75cffee67d3b7f4f0e669826282164`: - Baseline: restore only the five original master production leaves. Keep the same read-only settings projection and regression file. One enabled-Dot case failed with the intended `57014`; it still read the correct `true` policy. Five cases were unselected. - The trace held the admission lock at 2.441 ms. The second connection waited for that lock at 9.494 ms. The root-pool feature read finished at 2015.585 ms, after the competing statement timed out at 2014.982 ms. - Candidate: all six enabled/disabled Dot, Grok and Muse controls passed, with zero skips, in 14.528 s. Each read returned the configured policy before the competing lock was acquired. Every competing transaction committed without an error. - Fresh normal candidate production declarations passed in 17.080 s. The complete 615-root owning test program passed in 4.756 s. Own dependency setup and TypeScript export builds had passed before these controls. - Known worker groups and sockets closed. The single post-owner audit found no other sessions in the test database. No forensic database was changed. An earlier selector attempt skipped all six cases and opened no test database connection. It is not a regression pass or failure. Source inspection proved that Vitest 5 filters names with ` > ` separators, while its JSON reporter joins names with spaces. The selected control then used the unique title-only pattern; its separate baseline failure is preserved above. The later normal master merge preserves all seven tested files. The 46 incoming paths change Native/workspace recovery, types and presentation. They do not change the direct feature/settings read graph. The direct-policy evidence therefore carries from its tested revision. It was **not rerun at the merged revision**. Fresh merged-head checks at `c2031b0fb3e5d1b34ebff1d5af8cc050d129bd9a` passed: the content-aware Shared/SDK producer (1.325 s), normal Runner core and surfaces TypeScript programs (0.936 s and 0.693 s), and the full normal App graph (28.933 s). The App check included all production layers and all four test programs: 616 normal test roots and 397/409/412 tree roots. Known build owners and sockets closed. Database/Eval and provider asset outputs remained unchanged. No install, Cargo, binary build, provider execution or PG test repeated. Fresh final-head remote CI and review remain **pending**. The test entry is `server/src/modules/external-agents/feature-reader-owner.postgres.test.ts`. It requires a dedicated external `PAPERCLIP_FEATURE_OWNER_TEST_DATABASE_URL` ending in `_test` and an owned `PAPERCLIP_FEATURE_OWNER_TRACE` path. Its normal migration hook, two-connection pool, case deadlines and assertions remain unchanged. No provider or Runner binary is needed for these controls. The review also identified a wiring coverage gap: the six contention controls call the readers directly. A new whole-three file now traverses the actual Dot/Grok/Muse broker factories, normal pending-pairing producers, dispatch and real execution-admission transaction. Pass-through observations compare the exact admission transaction with the owner supplied to the normal readonly settings projection. Each provider is disabled through normal settings and must return real 403, read actual false policy, leave the queued run without Native authority, and create no assignment. No grant, ready binding, policy result or successful authorization is mocked. With only the original five master production leaves restored, all three new cases failed at the exact missing-owner assertion (`readOwners` expected length 1, actual 0) after passing the real 403 and admission-scope checks. The full baseline had0 pass / 3 fail / 0 skip in 12.691 s. With the exact candidate leaves restored, the unchanged whole-three file passed 3/3 with zero skips in 10.101 s. Both variants used actual Node 24 / Vitest 5 forks and a real PG pool of two; all known owners/groups/sockets closed. The candidate's private temporary directory was removed; the failed baseline database/home remain retained. Root independently read all three raw baseline failures, rehashed source and outputs, verified known owner/group closure, then observed no remaining test sessions. The successful database was normally removed; the original failing database and home remain retained. The new test entry is `server/src/modules/external-agents/broker-feature-owner.postgres.test.ts`, using dedicated `PAPERCLIP_BROKER_FEATURE_OWNER_TEST_DATABASE_URL`. The complete normal 617-root owning test program passed in 4.883 s after a normal exported config refresh; prior production declarations and unrelated configs remained byte-identical. No compiler or dependency/Rust work repeated for the PG variants. These results were obtained with the seven production/source files at c203 plus the new test; they are not claimed as a new published-head remote CI result. The review repair is committed at `d043860eb93a26c4416f184697a212f027d538e2`; it adds only this 117-line test file to `c2031b0f`. Tracked production bytes are unchanged. Exact published head `d043860eb93a26c4416f184697a212f027d538e2` now passes 56 checks with four expected skips. Greptile reviewed this exact head at 5/5 and confirms that the broker-test gap is addressed. The complete review thread is resolved. Ponytail stopped without a verdict; one normal retry was requested and no active turn or new finding was observed. No earlier-head result is used as a new-head pass. The controls prove feature policy and transaction ownership. They do not prove deployed suspension, hosted Dot/Muse eligibility or full Native task execution. The separate private useful-case pass does not supply missing current-master or accounting evidence. ## Risks The broker identity contract changes for all three providers. A missed adapter could use the wrong owner. Current semantic, real-policy and exact broker-owner identity tests cover that error. The read-only projection reuses the normal settings view. It preserves managed and staging rules. An absent row is projected without a write; the existing initializing getter stays unchanged. No schema, public API, queue, timer, rate limit or provider deadline changes. No historical Retry or physical-stop proof is changed. ## Model Used OpenAI `gpt-6.1-sol` assisted with source changes and tool use. An OpenAI GPT-6 family coordinator reviewed the work; its exact runtime ID was not exposed. Context-window sizes were unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (tested revision and source-carry limits stated above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no user-facing command or contract change) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 01:02:03 -06:00
DottaandClaude Opus 5.5	e620ce3678	fix(heartbeat): deliver each accepted follow-up exactly once through restarts and admission holds (#19741) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can comment, click Retry, or rely on automatic recovery while an agent works on a task > - Restart-chaos waves showed accepted follow-ups being dropped, or delivered twice, around restarts and temporary company admission holds (accounting still settling after a restart) > - A dropped comment means the user's request is never acted on. A duplicate reopens work that is already done > - This pull request makes accepted work wait through a temporary hold and resume when it clears. It also stops a comment from being promoted again after a completed run already received it > - The benefit is that each accepted follow-up is delivered exactly once, eventually, unless the task is cancelled or a reached hard stop applies ## Linked Issues or Issue Description Refs #19702 (never-started tasks during a temporary hold), #19700 (bounded accounting hold), #14910 (stale reopen from a deferred wake; no file overlap). **What happened?** Restart-chaos waves found these shapes: 1. **Comment wake cancelled.** A comment was saved behind a run and promoted when the run ended. A temporary company hold refused the unstarted queued run, and claim cancelled it terminally. The recovery wake that followed was cancelled the same way, and the task rested `blocked` without the comment. 2. **Dead Retry.** A Retry click during the hold stayed `deferred_issue_execution` after the hold cleared. Nothing woke it, and the issue held no execution lock. 3. **Refused assignment.** A new task's first wake was refused during the hold. Nothing woke stranded-work recovery when the hold cleared, so the task stayed `todo` with no run. 4. **Silent comment after recovery.** A comment was saved behind a run that restart recovery finalized. A company-scoped hint waited out the 30 s deferred-receipt rescan, so the task looked done with the message unread. 5. **Comment delivered twice.** A run whose provider process was lost had a saved comment. Its retry started after the comment, received it in its prompt (task history), did the work, and finished the task. The saved wake was then promoted anyway, which reopened the done task. **Expected behavior** An accepted follow-up (comment, Retry, recovery) is delivered exactly once, eventually, unless the task is cancelled. A temporary refusal never ends in a terminal cancel, skip, or block. Reaching a hard stop remains the board's decision. ## What Changed - `heartbeat/queue.ts`: on a budget block at claim, an unstarted queued run stays queued when the block is temporary. A reached hard stop or blocked unpriced usage still cancels it. - `recovery/service.ts`: stranded-work recovery waits out a temporary hold for any task. It does not escalate to `blocked`. This extends #19702 beyond never-started tasks. - `budget-block-ownership.ts`: a shared `isBoardOwnedBudgetBlock`, moved out of the recovery service. `budgets.ts` is unchanged. - `run-cost-accounting.ts`: settling a run's accounting wakes queued-run dispatch, the deferred-receipt worker, and stranded-work recovery. Held work waits passively; this is what resumes it. - `deferred-execution-notifications.ts`: new `notifyDeferredWakesForIssue`. `native-run-projection.ts` (a run reaching a terminal status) and `environments.ts` (lease release) now hint each saved wake of the issue by id, so delivery does not wait out the company rescan interval. - `wake-queue/adapters/postgres.ts`: release treats a saved comment as delivered only when a succeeded run of the wake's agent on that issue started after the comment's latest revision, and either received the comment as its own wake input (`wakeCommentIds`) or was a retry of the interrupted turn whose task history included it. In that case it cancels the saved wake with "Queued messages were already delivered to a completed run" and leaves the task as it is. Task history alone never consumes a message saved for its own turn. One query checks the whole batch. - `run-cost-accounting.ts`: settling late receipt debt also sends the hold-release notifications. - `doc/execution-semantics.md`: documents the rule. ## Verification Red on `master` and green with this change, on embedded Postgres: - `server/src/__tests__/heartbeat-held-comment-wake.test.ts`: - A promoted comment refused by the hold stays pending, with no cancelled run and no `blocked` task. It is delivered once after the hold clears through real `accountRunCost`. - A Retry clicked during the hold starts exactly once after accounting settles, which wakes its owner. - A new task whose assignment was refused during the hold starts once accounting settles, which wakes stranded-work recovery. - A comment that a later succeeded retry already received as history, or that a later run received as wake input, is consumed without reopening the task. - Guard tests (green on both): - A comment is still delivered in each of these cases: the later run never received it, it was edited after that run started, it was posted while the run was already running, or a non-retry run only saw it as history. - A task whose continuation was refused waits out the hold. - A reached hard stop is still refused. - `server/src/__tests__/native-run-projection-work.test.ts`: projecting a recovered run terminal hints the saved comment's own wake (red on `master`). - Related suites pass locally (14 files, 482 tests): wake-queue, queued comments routes, refused assignment wake, native run projection. - Live replays (local, unmasked, kept instances): - `--replay w-20261107T031005-s7501/t04` (Claude C1, K1): PASS. The comment's run was created at 03:56:24 during the hold, stayed queued, started at 04:06:24 when the hold cleared, and succeeded. It is the only run carrying the comment. - `--replay w-20261107T002911-s7401/t01` (before the per-wake hint): it reproduced shape 4. Run 592d772f was killed 250 ms after `run.result.accepted`, released and finalized at restart, and the task rested `done` with the comment still saved. Calling the existing sweep on the kept database promoted it, which confirmed that the missing piece was the trigger. ## Risks - A queued run can now sit queued for the length of a temporary hold, which #19700 bounds. It does not hold the issue's execution lock, so later comments still coalesce into it. - The delivered-comment rule depends on the run's recorded prompt context. A succeeded run that received the comment in its history counts as delivery, even if the agent did not act on it. This matches exactly-once delivery semantics. - Settling accounting now emits three extra notifications per run. They are coalesced per company. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, live restart-chaos replays). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 00:24:09 -06:00
DottaandClaude Opus 5.5	8a34c1b62c	fix(budgets): bound the accounting hold so one missing receipt cannot wedge a company (#19700) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Pending run accounting holds new company work until it settles (#19497, #19606, #19673) > - A run settles only with closure evidence: a verified local stop, or a terminated remote sandbox. Without that evidence it held the whole company forever. In restart-chaos storm 4, a single run held 35 tasks for hours > - Two more shapes kept the hold alive. A run whose result had already been accepted never counted as closed. A recovery that already owned its held reservation was refused by other tasks' recovering accounting, so concurrent recoveries waited on each other > - This pull request bounds the hold, closes accepted-result runs on a verified stop, and lets an owned recovery finish > - The benefit is that no single missing receipt can wedge a company, while genuinely unknown spend still surfaces under the blocking policy ## Linked Issues or Issue Description Follow-up to #19497, #19606 and #19673. **What happened?** - Storm 4: run `f879895f` timed out in `session.open`. Its local stop was never verified (its process identity is still recorded), recovery ended in `terminal_failure`, and its receipt never became ready. From then on, every new-work wake got 409 "New work must wait while a native run recovers its unfinished accounting". That was 93 refusals, 35 blocked tasks, and no Retry. - Storm 4: run `6d3c76bf` was resuming as the recovery owner with its own held reservation. `reserveRunBudget` refused it because other tasks' runs were recovering. - Wave s7401: run `0a3a849d` failed after its result was accepted, with a verified local stop. It had 10 receipts across 2 sources, all incomplete with 0 tokens. It stayed "completed runs await accounting" because closure only covered coordinators in terminal failure with no result. **What did you expect?** A run without closure evidence holds new work only for a bounded time, then settles visibly. An accepted-result run with a verified stop closes like a terminal failure. A recovery that already owns admitted work can finish. **Steps to reproduce** 1. Restart the control plane during several native turns so that one run's local stop is never verified. 2. Observe the run stay "recovering" and every new wake refused indefinitely. ## What Changed - `run-cost-accounting.ts`: - A coordinator counts as settled when it ended in terminal failure without a result, or when it has an accepted result. The verified-stop and remote-termination closures use that rule, which covers the s7401 shape. - A native run whose provider never started settles at once, like a pre-provider failure, with no charge and its reservation released. Its recovery has settled, no usage receipt was ever journaled (a provider turn journals one as soon as it starts), and its local process never launched or was verified stopped (wave s7403, run 3c1d7b70, `session.open` timeout). - New bound `ACCOUNTING_CLOSURE_TIMEOUT_MS`: the remote runner reconnect grace (7 minutes) plus 3 minutes. A terminal native run with a settled coordinator and no closure evidence settles at the bound: - its known usage is recorded as unpriced (or estimated from the rate card), with provenance evidence "closure evidence never arrived…"; - it is marked `accountingClosureTimedOut`; - a `cost.accounting_closure_timed_out` activity entry names the run. - The reconciler selects these runs, and `readCostAccountingWork` schedules a wake at the bound. - `budget-exposure.ts`: a timed-out closure's final charge is exempt from none of the unpriced-block exemptions. Under the `block` policy it raises the normal budget notice ("recorded usage has no reliable price") instead of a silent hold. - `budgets.ts`, `budget-reservations.ts`, `run-dispatch/adapters/postgres.ts`: when a validated recovery owner continues admitted work, admission and both dispatch rechecks apply only hard stops and pauses. Pending and recovering accounting holds still gate new work. - `doc/SPEC-implementation.md`: documents the bound, the accepted-result closure and the provenance. ## Verification - `closed-native-receipt-accounting.test.ts` (embedded Postgres). All of these fail on master and pass here: - s7401 shape (accepted result, verified stop, all receipts incomplete with 0 tokens): settles and clears admission. - A run without closure evidence: held before the bound with a scheduled wake, then settled with the "closure evidence never arrived" provenance, the `accountingClosureTimedOut` flag and an activity entry. Admission clears under `allow`. - Storm 4 `f879895f` shape (terminal failure, recorded process identity, no stop event): held, then released after the bound. - The blocking policy still blocks on a timed-out closure. - s7403 shape (no receipts, model unknown): a run whose process never launched, or was verified stopped, settles at once with no charge and its reservation released. With an unverified process it waits for the bound. - Storm 4 `6d3c76bf` shape: a recovery owner with its held reservation reserves past another task's recovering accounting, while new work for another task still waits. - Existing suites pass locally: budget-enforcement-scoping, budget-enforcement-worker, native-stop-accounting, cost-data-quality, heartbeat-cost-accounting, run-cost-accounting, budgets-service. Server typecheck passes. - The other storm 4 terminal-failure runs (`2e477ae9`, `e4d2e14d`, `6d3c76bf`, `7d21a44c`, `4bc49df0`, `1d85fbd9`) were already settled by #19673 in that database (`cost_accounting_pending=false`) and were not holding admission. `f879895f` alone held it. ## Risks - After the bound, a run whose provider might still be running settles as unpriced. Under the `block` policy that raises a hard-stop budget notice, by design. Under `allow`, work continues and a late receipt still reconciles. - A recovery owner skips accounting holds, so concurrent recoveries can spend while other tasks' accounting is unknown. Hard stops still apply, consistent with best-effort budget enforcement. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 21:02:01 -06:00
3 people	51a65af9b8	fix(recovery): continue preserved preparation after process loss (#19464) ... ## Thinking Path > - Paperclip manages AI agents and their execution state. > - A server restart can stop workspace preparation before provider execution. > - Workspace recovery must preserve files before the task can continue. > - Recovery handled interrupted preparations but skipped failed process-loss records with the same no-provider certificate. > - A withheld first retry could also hide the remaining retry budget. > - This change resumes certified preparations through normal admission after file recovery. ## Linked Issues or Issue Description **What happened?** A process-loss attempt remained failed after workspace recovery completed. Its first retry had been cancelled before dispatch by the workspace hold. The task had no running attempt or automatic continuation. **Expected behavior** Continue once after the server proves that provider work did not start, file preservation completed, and the exact leases stopped. Retain the existing failure budget and all task admission gates. **Steps to reproduce** 1. Stop a Runner preparation before native runtime selection. Record the restart recovery bootstrap certificate. 2. Retain the workspace and withhold the first automatic retry before dispatch. 3. Complete file preservation and restart the recovery worker. 4. Observe that the old code schedules no successor. The regression test reproduces this with PostgreSQL and real filesystem operations. **Paperclip version or commit** Reproduced against master at `2f35eb1080`. Related prior work: #19422 continues saved user input after verified copy-back. This change handles automatic continuation of certified process-loss preparation. ## What Changed - Accept failed process-loss preparation only with the durable no-provider bootstrap certificate and existing execution, lease and file checks. - Discover and settle the recovered source through the existing preparation recovery path. - Continue from a verified withheld retry without rewriting its history or resetting its failure count. Adopt it only as the sole retry child; multiple-child history reuses the newest existing child deterministically. - Exercise automatic dispatch, preserved-file readback, one provider turn, accepted result and task completion in the local native-session fixture. - Keep a durable continuation receipt after file-hold settlement until a successor is recorded. Known gates wait for existing eligibility events. Unresolved delivery keeps its shared recovery deadline. - Document the recovery and retry-budget rules. ## Verification - Red: both new PostgreSQL/filesystem regressions failed with zero scheduled successors where one was required. - Green: 54 focused recovery, automatic-dispatch, interruption, multiple-child history and admission safety cases pass on `13c27c2c2f`. The full PostgreSQL admission adapter suite passes all 83 tests; server source declaration typecheck also passes. Local environment: Darwin arm64, Node 26.4.0, PostgreSQL 18.1, Vitest 5.0.3. - Additional red/green: interruption after committed settlement, pause/resume, failed enqueue after cursor advancement, and absence of an earlier retry all failed before their repairs. Restart before enqueue, restart after the successor claim, and pause/resume now preserve one continuation. A rolled-back eligibility change cannot start execution. Deferred and delivered records have no retry deadline. - Local acceptance uses the real heartbeat dispatcher, isolated files and PostgreSQL, with a simulated native provider. It reads the preserved file, writes continued work, accepts the structured result and completes the task once. - Negative cases retain no-certificate, provider-evidence, live-owner, Stop, missing-stop-receipt, wrong-snapshot, human-block, pause, reassignment, approval and exhaustion fences. - CI fixture regression: 4 error-propagation/concurrent-resolution cases pass after adding the new obligation read to their transaction stubs; existing assertions remain unchanged. - Existing idle-drain and notification contract regressions: 4 tests pass after restricting recovery wakes to actual durable obligations. Late approvals block scheduled, queued and final dispatch. - Additional existing retry-scheduling suite: 62 cases pass locally. The pre-provider quota case exceeds its existing 5-second terminal wait on this loaded Darwin host, including when tested alone and with the preceding retry implementation. It remains a local timing limitation; the assertion and timeout were not changed. All Linux CI server shards pass on the combined head, including this unchanged suite. - Existing run-control, legacy-recovery and shared execution-control lifecycle suites: 128 tests pass with the review repair. Required CI supplies final-revision broad coverage. - Real-Codex acceptance was not run: automatic approval review rejected authenticated provider execution because it could incur cost/external effects and required explicit approval. No provider call ran; the controlled-provider acceptance covers the causal restart and durable scheduling behavior. - Hosted acceptance and recovery of existing tasks require rollout. They have not been claimed from the local provider fixture. - Final-head review: Greptile 5/5 on `13c27c2c2f`, no outstanding findings. Another contributor added the multiple-child history safety repair in `13c27c2c2f` while CI was running. The delta was inspected and the 54 focused recovery cases rerun successfully. Required CI passes on that exact combined head: https://github.com/paperclipai/paperclip/actions/runs/42241794918. All 55 successful checks include aggregate `ci / verify`, typecheck, build, canary, all server and E2E shards, and Runner package/Rust/static checks. The separate Paperclip Review service check remains queued waiting for its assigned agent; no validated assessment has been returned. ## Risks A malformed preparation certificate or a false no-execution receipt could permit duplicate work. The existing checks still reject provider execution records, native coordinators, live controllers, unsettled leases and unverified files. The pending continuation must survive a restart without polling known admission blockers. A withheld retry retains its original row and consumes its remaining failure budget. Normal admission checks current ownership, approvals, pause and budgets. ## Model Used OpenAI GPT-6 family via Codex. The exact model ID and context window are unavailable in session metadata. Used reasoning, code editing and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal task id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing>	2026-11-06 09:36:46 -06:00
DottaandClaude Opus 5.5	7e3caa7bac	fix(tree-control): resume work a pause gate refused after release (#19421) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The board can pause an issue tree. A pause stops work. Resume or release must continue that work once. > - PR #19394 made release re-dispatch a run that a pause Stop cancelled. > - That rule covers only a run that an operator Stop ended and that the hold snapshot named. > - A scheduled retry that comes due during a pause is not stopped. The dispatch gate cancels it with `issue_paused`. The release did not see it as owed work, so the task stalled with no run. > - This pull request makes every gate that refuses work for a pause record the covering hold. Release then re-dispatches that work exactly once. > - The benefit is that a paused task with a parked retry or a queued run continues after release, with no manual wake. ## Linked Issues or Issue Description Refs #19394 **What happened?** A child task had a `workspace_busy` scheduled retry. The board paused the parent tree. The retry came due during the pause. The scheduled-retry gate cancelled it with `issue_paused` ("Scheduled retry suppressed because the issue is held by an active subtree pause hold"). The board released the hold. Four minutes later the child was still `in_progress` with no run, no execution blocker, and no new activity. **Expected behavior** Release (or `mode: "resume"`) re-dispatches the refused work exactly once. No reconciliation and no user message are necessary. Stranded-issue recovery does not add a second run. **Steps to reproduce** 1. Give an assigned child task a parked `workspace_busy` scheduled retry. 2. Pause the parent tree. 3. Let the retry come due while the pause is active. The gate cancels it with `issue_paused`. 4. Release the pause. No run starts for the child. **Paperclip version or commit** master at 4d49a31286 (includes #19394). **Agent adapter(s) involved** Not adapter-specific (core bug). ## What Changed - `issue-tree-control.ts`: add `ISSUE_PAUSED_ERROR_CODE` and `pauseSuppressionResult()`. A gate records `treeHoldSuppression: { holdId, rootIssueId }` on a run that it refused for a pause. - `findOwedPauseResume` now treats the latest run as owed when it is an `issue_paused` cancellation under a pause hold. It uses the recorded hold. For an unmarked run (a scheduled retry cancelled before this change), it uses a snapshot pause that was active at cancellation. The operator Stop rule does not change. - Scheduled-retry gate (`run-dispatch/adapters/postgres.ts`, `cancelSuppressedRetryInTx`): locks the pause row `FOR SHARE` and records the covering hold. A concurrent release either sees the refusal, or the retry sees no pause and runs normally. - Queued-run claim gate (`heartbeat/queue.ts`): cancels with `issue_paused` and the hold marker, after the agent start lock is released. Before, the cancel re-entered that lock from inside it, and the agent's queue stalled until the 30-second stale-lock timeout. After the refusal is saved, the gate re-checks the hold and resumes the task itself if release already finished (`resumeIfPauseAlreadyReleased`). - Release (`issue-tree-resume.ts`, route): revisits the snapshot members and every task with a run refused for that hold, also when `wakeAgents` is set. This covers tasks created under the tree after the snapshot and tasks moved out before release. - `packages/db`: partial index `heartbeat_runs_tree_hold_suppression_idx` (migration 0436, `CREATE INDEX CONCURRENTLY`, registered as a top-level migration) for that lookup. - Docs: `doc/execution-semantics.md` (gate-refused work, race handling, non-assignee and pre-fix scope) and `doc/DATABASE.md` (index). ## Verification - Red before the fix, green after: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/issue-tree-pause-resume-run.test.ts` (21 tests). - "resumes a parked retry the pause suppressed exactly once after release / resume / legacy": before, `expected [] to have a length of 1` (no resumed run). After, one `issue_tree_resumed` run with `interruptedRunId` = the suppressed retry, no execution blocker, and the task finished. `reconcileStrandedAssignedIssues` added no run during the hold or after release. - "resumes a queued run the claim gate refused during the hold exactly once": before, the test timed out at 15 s (start-lock re-entry). After, the run is cancelled with `issue_paused` and the hold marker, and release re-dispatches it once. - "resumes a refused retry on a task moved out of the paused tree before release" and "... on a task outside the snapshot when release also wakes members": review findings, one resume each. - "leaves a due retry parked when the pause is released while the gate refuses it": the test waits until the gate blocks on an uncommitted release. The retry is not cancelled and runs once. Without the `FOR SHARE` lock the gate does not wait and the test fails. - "resumes work a gate refused after the release that should have resumed it had finished": the late-refusal re-check resumes once, and a repeated check does not dispatch again. - "does not resume a paused-code cancellation that no pause hold covered": negative case. - Related suites pass locally: `src/__tests__/issue-tree*`, `src/modules/run-dispatch`, `src/services/heartbeat/queue.test.ts`, `heartbeat-dependency-scheduling`, `heartbeat-workspace-busy`, and `packages/db` `client.test.ts`. - I did not run a live staging check here. Staging APIs are not reachable from this environment. ## Risks - Migration 0436 adds a partial index on `heartbeat_runs`. It is built concurrently and covers only cancelled runs with the marker, so it is small. - The scheduled-retry gate now takes a `FOR SHARE` row lock on the pause hold before it cancels. Release updates that row, so the two serialize briefly. Release does not lock issues or runs, so the lock order cannot deadlock with the gate's issue-then-run locks. - The queued-run claim gate now records `issue_paused` in place of the generic `cancelled`. Stranded recovery already classifies `issue_paused` as non-retryable. The pause resume owns that work, so recovery does not add a second run. - Scope limits (documented): a non-assignee run refused by a pause is not replayed, the same as non-assignee wakes that admission drops during a hold. Among pre-fix rows, only scheduled retries carry `issue_paused`, so only those fall back to the snapshot. A pre-fix row whose hold is already released does not self-recover; the staging task that showed this bug needs one manual wake, or a new pause and release. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 03:03:15 -06:00
DottaandClaude Opus 5.5	90b8190c33	fix(native-runtime): detect half-open runner ingress and project Stop for live runs (#19407) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs in a managed sandbox talk to the server through a provider-ingress WebSocket. The server opens this socket to the runner (`paperclip-runnerd`) through the sandbox preview proxy. > - A staging run ran a 20-minute foreground shell command that printed nothing. After the first second, no run events arrived. The run stayed "Working" for over 35 minutes. No watchdog acted, and the run projection said `canStop: false`. > - The server only detected a lost runner connection from a `close` or `error` event. It never sent pings. A half-open route, for example a proxy that drops the connection without a FIN, never emits `close`. The run then waits forever and the 7-minute reconnect grace never starts. > - A long silent tool call is legitimate. The fix must tell "quiet but alive" apart from "nothing alive". A protocol ping does that: runnerd answers every ping with a pong while its provider waits on the tool. > - This pull request pings authenticated runner connections and terminates the socket after 90 seconds of total silence. The existing reconnect grace and transport-failure path then recover the run. It also makes the execution projection report `canStop: true` for every queued or running run. > - The benefit is that a run with a dead runner connection now ends in a bounded time instead of hanging. API readers also see the Stop that the board already offers. ## Linked Issues or Issue Description **What happened?** A native-runner run (managed sandbox, OpenCode with an OpenRouter model) ran a foreground shell command for about 20 minutes with no output. The last run event came at about T+1s. The command finished on disk, but the run never got another event and stayed `running` / "Working" for over 35 minutes. Its projection had `canStop: false` and `permittedActions: ["inspect_run"]`. A board cancel through the API worked. **Expected behavior** The server detects a dead runner connection and recovers or fails the run in a bounded time. A runner that is alive but quiet stays attached. The projection reports that a live run can be stopped. **Steps to reproduce** 1. Start a native run in a managed sandbox. 2. Make the agent run a long foreground command that prints nothing. 3. Let the preview route drop the WebSocket without a close frame. 4. Before this fix, the run stays "Working" forever. ## What Changed - `server/src/realtime/runner-prp-outbound.ts`: after PRP authentication, `WsJsonWireConnection.monitorLiveness` sends a WebSocket ping every 20 s. Any inbound frame or pong counts as liveness. After 90 s with nothing inbound, it terminates the socket. That emits an ordinary `close`, so the PRP authority and the reconnect loop use the existing 7-minute grace and then fail the transport. If the server's own event loop stalls, the window restarts instead of blaming the runner. The 90 s bound is longer than runnerd's own 60 s connection lease, which already requires runnerd to read the socket. - `server/src/services/execution-projection.ts`: `canStop` is now `true` for every `queued` or `running` run. The old failed-native-retry rule is unchanged. - `server/src/services/execution-projection.ts`: a queued run now shows "Waiting for previous run to stop" (cause `previous_run_settling`) when a predecessor that admission checks (the task execution owner, the retried run, or the agent's latest settled run on the task) still has pending cleanup by `hasPendingRunCleanup`. Before, a turn queued after a Stop showed only "Queued" with no reason while the stopped run's controller still waited on the dead runner. That in-process wait (`claimQueuedRun` settling-owner gate) also made Retry a no-op. The liveness bound above makes that controller settle. Its existing cleanup then calls `startNextQueuedRunForAgent`, which starts the queued turn. - `server/src/services/run-cancellation.ts` and `server/src/modules/run-dispatch/adapters/postgres.ts`: a user-stopped native run whose runner was unreachable can later settle as `failed` (`native_provider_terminal_failed`) when its transport fails closed. The dispatch gate's message deferral required a `cancelled` source and compared the message with `finishedAt`. The user's follow-up message was therefore marked `skipped`, and the task asked for a second message. The new `recordedUserStop` helper honors the recorded user Stop on such a failed source, so the gate saves the message for the deferred-receipt worker. - `packages/shared/src/types/execution-projection.ts`: documents the `canStop` meaning. - `doc/execution-semantics.md`: documents the ingress liveness bound and the new `canStop` meaning. ## Verification - Red then green: `runner-prp-outbound.test.ts` "detects a half-open authenticated ingress and fails the run within a bounded window". On the old code it failed because no ping was sent and the socket was never terminated. It now passes and the active-run `failure` rejects with `runner_ingress_unavailable`. - Safety: "keeps a quiet but live runner attached through a long silent tool call" checks that a runner answering pings stays attached for 20 simulated minutes with no termination and no failure. - Red then green: `execution-projection.test.ts` "offers Stop for every live run, including a long-silent native turn". On the old code it failed with `canStop: false`. - Red then green: `execution-projection-recovery.test.ts` "names the wait when a stopped predecessor still holds the task's runner" (real Postgres). On the old code the successor projected a plain "Queued". It also checks that the wait clears on lease release and ignores another agent's run. - Red then green: `workspace-recovery.test.ts` "continues a message sent after Stop when the stopped run later failed closed". On the old code it failed 3 of 3 runs with the receipt `skipped`. It now passes, and the full file passes twice (392 tests). The test now pins the message after the recorded Stop. Before, the fixture could write both in the same millisecond, which made the sibling variants flaky. - Passed: run-cancellation, explicit-native-continuation, run-dispatch and run-control suites (622 tests). - Passed: `npx vitest run src/realtime/ src/services/execution-projection*.test.ts src/__tests__/heartbeat-run-privacy-routes.test.ts`, the native-runner-execution-driver tests, and the plugin-runner-task tests. - Passed: `pnpm --filter @paperclipai/server typecheck`. - I read runnerd (`runner-core/src/durable/transport.rs`) and confirmed it answers `Ping` with `Pong` in its non-blocking read loop. - Not run: a live managed-sandbox reproduction. Staging and Daytona were not reachable from this environment. `plugin-runner-probe.test.ts` needs a local cargo build of runnerd, and it did not start in this worktree. ## Risks - A false termination is possible if a runner fails to answer pings for 90 s while it is still alive. runnerd already loses its own connection lease after 60 s without reading, so this bound does not add a stricter requirement. A terminated live runner reconnects within the existing grace. - If the preview proxy drops WebSocket control frames, ping alone does not prove liveness. Inbound PRP frames (runnerd's lease renewals, about every 30 s) still count, so a live runner stays attached. - `canStop` now means "the board can Stop now" for live runs. The UI already shows Stop for running and queued runs without reading this hint, so the UI does not change. - Out of scope: the inbound runner WebSocket server (`runner-prp-ws.ts`), and a lost OpenCode SSE event while the runner is alive. A stopped run whose controller hangs for a reason other than a dead transport still holds the queued successor. The projection now names that wait, but this PR does not force-release it, because the controller has no termination evidence. A queued reason for the agent concurrency cap is also not covered. The wait label calls the admission predicate (`hasPendingRunCleanup`) for up to three predecessor runs per queued run. This adds a few reads to projections of queued runs only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 00:52:01 -06:00
DottaandClaude Opus 5.5	6ca7b151f7	feat: compact an agent's context from the task Properties panel (#19378) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task's Properties panel shows the assignee agent's context window (paperclipai/paperclip#19273) > - The board can see that the context is nearly full, but it cannot act on it > - A full context makes the agent compact at an unplanned time, often in the middle of work > - This pull request adds "Compact now" to the Context hover card > - The benefit is that the board can compact the agent's context between runs, at a time it chooses ## Linked Issues or Issue Description Refs paperclipai/paperclip#19273 **Problem** The Context row shows how full the agent's context window is. The board cannot ask the agent to compact it. **Proposed solution** A "Compact now" button in the Context hover card. It runs the harness's own compaction: Codex `thread/compact/start`, and Claude's `/compact` command through ACP. **Alternatives considered** A separate PRP `session.compact` command. Both harnesses run compaction as a turn, so a compaction turn keeps the existing durable turn binding, interruption, and recovery. A separate command needs a second turn lifecycle. ## What Changed - UI: the Context hover card shows Compact now with three states: available, Compacting…, and unavailable with a reason (agent mid-run, harness unsupported, task closed). The Storybook story shows each state. - Server: `POST /api/issues/:id/context/compact` (board only, `decideIssueAccess` `issue:mutate`, activity log `issue.context_compaction_requested`). It queues a short run marked by `contextSnapshot.contextCompactionRequestId`. `GET /issues/:id/context-usage` returns the `compaction` state. - Server: a compaction run does no task work. The native status arbiter preserves the task status. The lifecycle skips handoff, liveness continuation, review-path disposition and the issue comment, as it does for goal-control runs. - Runner TS: `startTurn({ compact: true })`. The Codex harness session sends `thread/compact/start`. The runnerd transport maps it to PRP `turn.start` with `compact: true` (Codex) or `text: "/compact"` (Claude/ACPX). The native runtime option `sessionCompaction` requires a resumed session and settles with a synthesized yielded result. - Runner Rust: `turn.start` accepts `compact: true` for Codex. `CodexProvider::start_compaction_turn` sends `thread/compact/start` and binds the turn from the `turn/started` that follows the empty response. - Fix (from #19273): Codex emitted `context.compacted` on both the start and the end of one compaction item. It now emits it once, and the count is distinct by `compactionId`. - Fix (from #19273): the conversation event view dropped `context.compacted`, so a finished run never showed the divider. The view now keeps compactions, plus context readings for runs that compacted. A finished run's compactions show as thread dividers. - Fix: the Codex harness stamps before/after sizes on the compaction event. Codex reports the compacted size before the compaction item ends, so the old neighbour-based guess showed the wrong size. ## Verification - `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, `node scripts/storybook-catalog.mjs`: pass. - New tests: `server/src/__tests__/context-compaction.test.ts`, `status-arbiter.test.ts`, `issue-context-usage.test.ts` (duplicate compaction counts once), `ui/.../context-usage-row.test.tsx`, `native-session-runtime.test.ts` (context compaction), Codex driver and runnerd transport tests, Rust `codex_compaction_turn_binds_the_turn_reported_after_its_empty_response`. Rust `codex_provider` suite: 115 passed. - `pnpm test:run` locally: one failure in `chat-channels.integration.test.ts`. It reads `.claude/skills/slack/SKILL.md` from the main checkout, which is not present in this worktree. It is not related to this change. - Live, local worktree instance, Codex agent on the Paperclip runner: clicked Compact now. The run succeeded. Context went from 35k to 9.4k tokens, and later 9.4k to 9.2k. The task stayed In Review. The thread shows "Context compacted · 9.4k → 9.2k tokens" and the meter dropped. - Live Claude proof: pending. The local Claude agent needs an Anthropic API key. claude-agent-acp handles `/compact` as an echo-less turn, and the runner already maps its "Compacting completed." notice to `context.compacted`. ## Risks - A compaction run resumes the agent's session. If the session cannot resume, the run fails with `native_context_compaction_unavailable` and the task is unchanged. - Compact now is not available while the agent runs on the task, because compaction interrupts the active turn. - The conversation view now returns context readings for runs that compacted. Runs that never compacted return the same events as before. - An old runner binary ignores `compact` and rejects the `turn.start` because it has no text. It does not start an ordinary turn. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 22:07:12 -06:00
3 people	f2593b7905	feat: make Coordinator and Agent Chat standard and speed up navigation (#19360) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People enter that work through the workspace, agent conversations, and tasks. > - Coordinator and Agent Chat still require experimental enrollment. > - Task link prefetches do not warm the compact queries used by the default task view. Chat also waits for a request when its navigation list already holds the conversation. > - This pull request makes Coordinator and Agent Chat standard and reuses the reads needed for navigation. > - People can choose Classic workspace in Experimental settings. Cached work can appear while fresh data loads. ## Linked Issues or Issue Description **What existing behavior does this improve?** Workspace selection, Agent Chat availability, and navigation between Chat, Tasks, and task details. **Subsystem affected** Cross-cutting: UI, workspace preferences, instance settings, and the database migration. **Current behavior** People must enable Coordinator and Agent Chat in Experimental settings. Opening a task can repeat requests because prefetch warms the full detail and comment queries, while the default view reads compact task data and a recent comment window. Opening Chat waits for a detail request even when the current person's conversation list contains it. **Proposed behavior** Coordinator is the standard workspace for everyone. Experimental settings offer Classic workspace, off by default. Old Coordinator experiment values do not select Classic. Agent Chat is available to everyone, including instances with an old saved or managed off value. Task link intent warms the queries used by the mounted view. Chat can use the current person's company-scoped conversation list during refresh. **Reason and benefit** People can use the standard workspace without setup. Repeat navigation can show cached content while the server refreshes it. **Breaking changes** Existing Coordinator experiment and Agent Chat off values no longer disable these surfaces. Classic requires an explicit new opt-in. The workspace API keeps its existing response shape and audit checks. The migration retains legacy values and adds a separate Classic preference. Execution permissions, company access, and chat privacy checks still apply. **Additional context** Roadmap check: this graduates existing surfaces and improves their navigation. It does not add a new agent execution capability. Duplicate searches found no open Coordinator workspace PR. Related server cache work, Refs: #17828, addresses compact issue-list invalidation; this PR addresses client navigation queries. ## What Changed - Add a default-off Classic workspace preference. Derive the existing workspace API response from that choice. Keep personal persistence, account isolation, and audited writes in the workspace preference service. - Replace the Coordinator experiment control with Classic workspace. Use Coordinator while preferences load. Update the existing settings examples and navigation recipe. - Retire the Agent Chat experiment control. Ignore old stored and managed off values. Retain the compatibility key and the existing execution and access checks. - Reuse account- and company-scoped chat list data with its original freshness timestamp. - Share compact task and recent-comment query options between prefetch and the task view. Keep full-history prefetch for Classic tasks and expanded history. - Add upgrade, replay, cache, account isolation, and browser switching regressions. Update product, database, development, and roadmap documentation. ## Verification - Targeted preference, settings, migration, cache, and UI regression tests passed during implementation. The cache tests hold refresh requests pending and check that content still renders. They also check account isolation and stale refresh behavior. - A real browser test verified fresh Coordinator defaults, Classic opt-in persistence, and return to Coordinator. It used an isolated source checkout with fresh data on macOS. - Token, migration numbering and safety, map, and Storybook catalog checks pass after the rebase. - Repository typecheck and build pass. Typecheck and token gates also pass after review repairs. The broad local run passed all six chat integration shards (1,931 tests). It then found the host-CLI-dependent recovery fixture described below. Its remaining duplicate work was stopped after the full current-head CI matrix passed; this is not a claim that the broad local command completed green. - Review repairs passed 419 UI/migration fixture tests, 19 Coordinator navigation tests, and all 214 tests in the five affected server suites. The 24 Chat/workspace browser tests pass, including standard Coordinator switching and explicit Classic persistence. - The broader browser repair run passed 85 tests. It retained two failed assertions and skipped five later serial cases. The affected files then passed all 11 tests after the fixes: reopen the desktop panel after reload and open More for Audit and Connectors. This includes the five serial cases that had not run. - The broad local suite exposed a recovery fixture that executed the host Codex CLI during a simulated remote probe. On a host with Codex 0.154.0, model fallback broke its fixed-model assertion. The fake sandbox now reports Codex 0.160.0. All 20 selected recovery/provider compatibility cases pass after the repair. - A CLI company import/export suite timed out during server startup in the second CI run. The port fallback is intentional in that fixture. No root cause was established. That suite passes on the final commit in CI. This records recovery, not a claimed product fix for the earlier timeout. - The first CI run exposed remaining assumptions about the retired flags and Classic UI. The repair commit updates those fixtures and navigation steps. It also restores the missing enabled Cases destination in Coordinator. Earlier local recovery attempts used the wrong fixture notification and advanced fake time during live execution; the corrected tests use the subscribed recovery queue and wall time during execution. - Greptile reviewed commit `98972b030d4505d838427a2fca6b196b8cad6d47` at **5/5** with no new comments. The original finding is resolved. - Current-head CI is green: **56 successful checks**, with three checks skipped by workflow filters. This includes typecheck, build, the complete Vitest matrix, all eight browser shards, Runner verification, release checks, and security checks. [Final CI run](https://github.com/paperclipai/paperclip/actions/runs/42132651414). - Typecheck caught missing `pid` and `startedAt` fields in the simulated command result. The fixture now returns the complete process contract, and the server test-layer typecheck passes. - Storybook catalog alignment was checked. Storybook play tests were not run. Hosted latency and cold bundle loading were not measured. ## Risks - The new default changes the workspace for people who previously left the Coordinator experiment off. They can opt into Classic in Experimental settings. - The additive migration must run before the new server code. It is safe to replay and preserves explicit Classic choices. - Cached content can be stale during background refresh. Existing query invalidation and source timestamps still control freshness. - These changes reduce avoidable loading waits. They do not establish a measured improvement in hosted request latency or initial bundle download time. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, code execution, and browser automation. The exact serving revision and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-05 16:28:25 -08:00
DottaandClaude Opus 5.5	a20dfbc695	fix(recovery): continue a message sent during Stop settlement after recovery (#19355) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can stop or interrupt a running native task and then send a message to give new direction > - When the stopped run left a workspace hold, the dispatch gate withholds the follow-up run until recovery saves the work > - The gate must save the user's message on its receipt so that the deferred-receipt worker continues it after recovery > - Two conditions made the gate skip the message instead of saving it, so the task stayed in progress with "Saved work recovered. Send a message to continue." although the user already sent one > - This pull request saves the message in both conditions > - The benefit is that the message the user already sent continues once, automatically, after recovery ## Linked Issues or Issue Description **What happened?** After a user stopped or interrupted a native task with an isolated sandbox workspace and sent a message, the follow-up run was cancelled with `execution_reconciliation_required`. Its error said "Saved work recovered. Send a message to continue." Nothing re-woke the task. In the interrupt case, a periodic sweep started a run about four minutes later. In the case where the follow-up first waited for the stopped run's workspace save (`workspace_busy`), each later retry parked on the same resolved recovery action, and the task did not continue. **Expected behavior** When the recovery that the follow-up waited on resolves, the message that the user already sent continues once, automatically. The product does not ask for a second message. **Steps to reproduce** 1. Run a native task in a managed sandbox with an isolated workspace. 2. Stop the run (or send a comment with interrupt), then send a message while the stop and workspace save are still settling. 3. Wait for the workspace recovery to resolve. 4. See that the follow-up run is cancelled with `execution_reconciliation_required` and that no run continues the message. **Paperclip version or commit** `master` before this change. **Root cause** `deferUnstartedUserComment` (run dispatch) refused to save the withheld message in two cases: 1. It compared the message time with the source run's `finishedAt`. A native run finishes only after its provider stop and workspace save settle, which takes seconds. A message written after the Stop request but before that settlement was treated as input written before the Stop. The receipt was marked `skipped`. 2. It required the run's own receipt to be user-owned. A workspace-busy retry or an admission retry carries the user's message under a system-owned receipt, so the message was dropped with the retry. ## What Changed - Add `userStopBoundaryAt(run)` in `run-cancellation.ts`. For a run cancelled by a user Stop or interrupt, it returns the recorded Stop request time (`cancellation.recordedAt`) when that is earlier than `finishedAt`. For all other runs it returns `finishedAt`. - Use this boundary in the dispatch gate's message deferral, in native explicit-continuation admission, and in the remote-Stop continuation check, so the three places agree on "message written after the Stop". - When the withheld run is a system-owned retry, walk a bounded chain (max 8) of retries that did not reach the provider (`workspace_busy` with `providerWorkStarted: false`, or a verified pre-dispatch `execution_reconciliation_required`) to the originating user receipt. Save the message on that receipt with the user's original wake context. Mark the retry's system receipt `skipped`. - Update the "input written before the Stop" test so that the Stop request, and not only the settlement, comes after the message. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` (`continues a message ... without a second message, after recovery already resolved`): a direct follow-up and a workspace-busy retry. Each runs recovery to completion first, then lets the gate withhold the follow-up, then runs only the deferred-receipt worker. Each asserts that exactly one successor run is queued with the user's message and the user's authority, and that no provider work starts. - Red before the fix: both tests fail because the receipt is `skipped` instead of `deferred_issue_execution`. Green after the fix. - `npx vitest run src/services/workspace-recovery.test.ts` (384 passed). - `npx vitest run src/services/explicit-native-continuation.test.ts src/modules/run-dispatch src/services/execution-continuation.test.ts src/services/run-cancellation.test.ts` (581 passed). - 19 more heartbeat, retry, queued-comment, and native recovery suites (1272 passed). One failure, `heartbeat-process-recovery > redacts opaque environment-bound credentials from Sentry diagnostics: result`, also fails on unchanged `master` and is not related. ## Risks - A message written after the Stop request but before the stopped run settled now continues. The stopping run cannot consume new input after its Stop request, and admission still removes comments that a run already delivered (`undeliveredLegacyUserCommentIds`). All admission gates (holds, stop proofs, approvals, budget) still apply. - Only user Stops and interrupts move the boundary. Other cancellations keep `finishedAt`. - The retry walk is bounded, stays in one company, agent, and issue, and accepts only retries without provider work. It revives a user receipt only from `cancelled` or `skipped`, so a second retry cannot save the same receipt twice. - Not in scope: a run that fails during native preparation with no user message (for example `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=unexpected_root_entry` on a system wake). After its recovery resolves, the task still waits for a user message. That is a separate root cause in the native session executor's prior-owner retirement proof. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 18:08:24 -06:00
DottaandClaude Opus 5.5	002455ea06	fix(issues): finish a reassignment whose native stop settles after the request (#19354) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can stop or interrupt a running native task and then send a message to give new direction > - When the stopped run left a workspace hold, the dispatch gate withholds the follow-up run until recovery saves the work > - The gate must save the user's message on its receipt so that the deferred-receipt worker continues it after recovery > - Two conditions made the gate skip the message instead of saving it, so the task stayed in progress with "Saved work recovered. Send a message to continue." although the user already sent one > - This pull request saves the message in both conditions > - The benefit is that the message the user already sent continues once, automatically, after recovery ## Linked Issues or Issue Description **What happened?** After a user stopped or interrupted a native task with an isolated sandbox workspace and sent a message, the follow-up run was cancelled with `execution_reconciliation_required`. Its error said "Saved work recovered. Send a message to continue." Nothing re-woke the task. In the interrupt case, a periodic sweep started a run about four minutes later. In the case where the follow-up first waited for the stopped run's workspace save (`workspace_busy`), each later retry parked on the same resolved recovery action, and the task did not continue. **Expected behavior** When the recovery that the follow-up waited on resolves, the message that the user already sent continues once, automatically. The product does not ask for a second message. **Steps to reproduce** 1. Run a native task in a managed sandbox with an isolated workspace. 2. Stop the run (or send a comment with interrupt), then send a message while the stop and workspace save are still settling. 3. Wait for the workspace recovery to resolve. 4. See that the follow-up run is cancelled with `execution_reconciliation_required` and that no run continues the message. **Paperclip version or commit** `master` before this change. **Root cause** `deferUnstartedUserComment` (run dispatch) refused to save the withheld message in two cases: 1. It compared the message time with the source run's `finishedAt`. A native run finishes only after its provider stop and workspace save settle, which takes seconds. A message written after the Stop request but before that settlement was treated as input written before the Stop. The receipt was marked `skipped`. 2. It required the run's own receipt to be user-owned. A workspace-busy retry or an admission retry carries the user's message under a system-owned receipt, so the message was dropped with the retry. ## What Changed - Add `userStopBoundaryAt(run)` in `run-cancellation.ts`. For a run cancelled by a user Stop or interrupt, it returns the recorded Stop request time (`cancellation.recordedAt`) when that is earlier than `finishedAt`. For all other runs it returns `finishedAt`. - Use this boundary in the dispatch gate's message deferral, in native explicit-continuation admission, and in the remote-Stop continuation check, so the three places agree on "message written after the Stop". - When the withheld run is a system-owned retry, walk a bounded chain (max 8) of retries that did not reach the provider (`workspace_busy` with `providerWorkStarted: false`, or a verified pre-dispatch `execution_reconciliation_required`) to the originating user receipt. Save the message on that receipt with the user's original wake context. Mark the retry's system receipt `skipped`. - Update the "input written before the Stop" test so that the Stop request, and not only the settlement, comes after the message. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` (`continues a message ... without a second message, after recovery already resolved`): a direct follow-up and a workspace-busy retry. Each runs recovery to completion first, then lets the gate withhold the follow-up, then runs only the deferred-receipt worker. Each asserts that exactly one successor run is queued with the user's message and the user's authority, and that no provider work starts. - Red before the fix: both tests fail because the receipt is `skipped` instead of `deferred_issue_execution`. Green after the fix. - `npx vitest run src/services/workspace-recovery.test.ts` (384 passed). - `npx vitest run src/services/explicit-native-continuation.test.ts src/modules/run-dispatch src/services/execution-continuation.test.ts src/services/run-cancellation.test.ts` (581 passed). - 19 more heartbeat, retry, queued-comment, and native recovery suites (1272 passed). One failure, `heartbeat-process-recovery > redacts opaque environment-bound credentials from Sentry diagnostics: result`, also fails on unchanged `master` and is not related. ## Risks - A message written after the Stop request but before the stopped run settled now continues. The stopping run cannot consume new input after its Stop request, and admission still removes comments that a run already delivered (`undeliveredLegacyUserCommentIds`). All admission gates (holds, stop proofs, approvals, budget) still apply. - Only user Stops and interrupts move the boundary. Other cancellations keep `finishedAt`. - The retry walk is bounded, stays in one company, agent, and issue, and accepts only retries without provider work. It revives a user receipt only from `cancelled` or `skipped`, so a second retry cannot save the same receipt twice. - Not in scope: a run that fails during native preparation with no user message (for example `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=unexpected_root_entry` on a system wake). After its recovery resolves, the task still waits for a user message. That is a separate root cause in the native session executor's prior-owner retirement proof. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:02:39 -06:00
Nicky LeachandClaude Opus 5.5	f9d366acdf	fix(runner): require a managed AI connection for plugin Runner-task environments (#19341) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A plugin can provide Runner execution through an environment driver that declares `supportsTasks: true`. Agent computers use this path. > - Core let an agent without a managed AI connection run on that path. The run then used the account that was installed on the plugin's execution host. > - That is not a supported credential source. When the host Runner did not match, the run failed three times, 30 seconds apart, and then blocked the task with a generic error. The error did not name the fix. > - The plugin cannot refuse these runs itself. `describe` has no connection data, and Core replaces plugin error text before the user sees it. > - This pull request makes a managed AI connection a requirement for every plugin Runner-task environment. Core refuses a run without one before it acquires a lease or calls the plugin. > - The benefit is one clear notice that tells the operator to choose an AI connection, and no wasted retries or plugin work. ## Linked Issues or Issue Description Refs #19326. That pull request makes a Runner revision mismatch retry. It noted that runs without a managed AI connection reach the Runner check on a different path, which still blocks after 3 attempts. This pull request removes that path for plugin Runner-task environments. **Problem or motivation** A `paperclip_runner` agent with no managed AI connection can run in a plugin environment that provides Runner tasks. The run uses the account installed on the execution host. If the host Runner does not match, the native session retries 3 times and then blocks the task with "Plugin-provided Runner execution operation unavailable". The operator cannot tell what to fix. **Proposed solution** Refuse these runs in Core before any lease, plugin call, secret resolution, or provider work. Use one stable reason, `ai_connection_required`, for run admission, the setup probe, and agent verification. Show a task notice that names the fix. **Alternatives considered** A refusal in the plugin. It cannot see the connection at `describe`, which runs first. Its error text is also replaced twice before the user sees it. A new structured plugin error code would still not cover `describe`. ## What Changed - Add `server/src/services/plugin-runner-ai-connection.ts`. It holds the `ai_connection_required` code and message, the probe result, and the manifest lookup for a Runner-task driver. The lookup moves here from `agent-runner-selection.ts`, so run admission and runner selection use the same rule. - `heartbeat/execution-configuration.ts`: refuse a new `paperclip_runner` run in a Runner-task environment when it has no managed AI connection. This also applies when `homeMode: "preconfigured"` drops the connection. The run fails as `configuration_incomplete` with reason `ai_connection_required` and an action URL to the agent's runtime settings. The check runs before observability, secrets, the managed runtime, the workspace, and the lease. A resumed native run keeps its admitted contract. - `recovery/stranded-notice.ts`: add an "AI connection required" task notice. - `run-failure-report.ts`: treat this pre-dispatch refusal as an owner action, not an unexpected failure. - `adapters/registry.ts`: the unmanaged probe for a Runner-task target now returns a failed `ai_connection_required` check. It calls no task operation. Managed probes still go through `testManagedEnvironment`. - `agent-harness-verification.ts` and `agent-lifecycle/application/failure.ts`: classify the check as `ai_connection_required` with a fixed message. Verification waits for the configuration to change instead of retrying. - Docs: `PLUGIN_RUNNER_EXECUTION.md` states the requirement. `PLUGIN_ENVIRONMENT_TASKS.md` no longer mentions installed accounts. `agent-config-ui.md` notes the requirement. - Tests: the "plugin-owned account" heartbeat test described the behavior that this change removes. It now checks the assembled heartbeat refusal: no plugin call, no native boundary, no adapter, no workspace files. ## Verification - `npx vitest run` in `server/` for these files. All tests passed: - `src/__tests__/heartbeat-execution-configuration.test.ts`. New cases: refused without a connection, refused for a preconfigured home, managed passes, resumed native input passes, non-task plugin driver passes. The refused cases also check that observability, secrets, and the managed runtime did not start. - `src/__tests__/heartbeat-plugin-environment.test.ts`. One test, "inherits the instance default environment across companies…", also fails on unmodified `master` in my environment. This change does not touch it, because its driver does not declare `supportsTasks`. - `src/adapters/registry.test.ts`, `src/__tests__/agent-harness-verification.test.ts`, `src/modules/agent-lifecycle/__tests__/diagnostics.test.ts`, `src/services/recovery/stranded-notice.test.ts`, and `src/services/__tests__/run-failure-report.test.ts`. - 17 more suites that use plugin Runner-task environments, with 705 tests: `plugin-runner-task.integration`, `plugin-warm-tasks.integration`, `task-creation-recovery.integration`, `native-startup-cleanup-recovery`, `environment-task-authority.integration`, `plugin-task-credential-writeback`, agent-lifecycle `lifecycle`, `setup`, and `environment-plugins`, `workspace-preparation`, `agent-runner-selection`, `agent-harness-verification-provider-api`, `agent-harness-verification-remote`, `plugin-environment-driver-seam`, `environment-task-runtime`, `agent-test-environment-routes`, and `adapter-registry`. - `pnpm typecheck` in `server/` reported no errors in the changed files. Its only errors came from sandbox-provider packages that my local install did not link. - Not run: `plugin-runner-probe.test.ts`, because it needs a Cargo build. This change does not modify `probePluginRunnerEnvironment`. Also not run: the full `pnpm test:run` and `pnpm build`. CI runs them. ## Risks - Behavior change: an agent in a plugin Runner-task environment with no managed AI connection now stops before dispatch. Before, it ran with the host's installed account. The agent's owner must choose an AI connection. This includes agents with `homeMode: "preconfigured"` in these environments. - Runs that were already admitted and resume from saved native input are not refused. They keep the earlier behavior until they finish. - The check reads the plugin manifest on each new run admission for plugin environments. This is the same lookup that runner selection already does. - This pull request and #19326 both edit `stranded-notice.ts` and `failure.ts`. The changes are additive, but whichever merges second needs a small rebase. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (file edits, shell, test runs). Default context window, extended thinking on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 14:59:40 -08:00
Nicky LeachandClaude Opus 5.5	a6ab6a301c	fix(server): retry a run when its remote Runner is out of date (#19326) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can run on a remote Runner that an environment plugin provides > - A release can update the app before the remote Runner, so for a short time the app sees an older Runner > - In that window the plugin can refuse the task, but Core cannot tell this refusal from other errors > - Today the run fails as `setup_failed` with no retry, and the task goes to board recovery. Before #19246, it was also reported as a project authentication conflict > - Agent verification also counts the refusal as a failed harness test, and it parks the agent after three tries > - This pull request gives the plugin a stable error code for a Runner mismatch, and Core waits and retries when it gets that code > - The benefit is that a release no longer blocks tasks or parks agents, and a real authentication conflict keeps its own clear notice ## Linked Issues or Issue Description Refs #19246 (it stopped reporting every precheck error as a project authentication conflict) Refs #19249 (the same bounded bootstrap retry for a stopped plugin worker) Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope is unchanged) Refs #19172 (related open PR: it reports an undescribable plugin Runner as a retryable check in the same precheck) Refs #18667 (a real `ai_connection_incompatible` conflict, which now gets its own notice) **What happened?** A run started while its remote Runner was older than the app. The environment plugin refused the task because the installed Runner did not match the revision it expects. Core recorded a generic setup failure, scheduled no retry, and blocked the task. Before #19246, Core reported this as a project authentication conflict. An agent in `verifying` counted the same refusal as a failed harness test. After three tries it stopped verification until a person chose Retry. **Expected behavior** Core treats a Runner mismatch as temporary. A run retries without blocking its task. A `verifying` agent checks again until the Runner matches. Only a real `ai_connection_incompatible` result blocks the task, with a notice that names the conflict. **Steps to reproduce** 1. Use an agent on a plugin environment whose installed remote Runner revision differs from the revision the plugin expects. 2. Wake the agent on a task, or put a new agent through verification. 3. The run fails with `setup_failed` and the task is blocked. Verification parks after three tries. **Paperclip version or commit** master at 377e98f1a2. **Deployment mode** Authenticated deployment with a plugin environment that provides a remote Runner. ## What Changed - Plugin SDK: add `PluginEnvironmentTaskError` and `ENVIRONMENT_TASK_ERROR_CODES` with one code, `runner_revision_mismatch`. The worker RPC host sends it as structured `environmentTask` error data with the schema `paperclip/environment-task-error-code/v1` (`environmentTaskErrorCodeData`, read with `readEnvironmentTaskErrorCode`). This uses the same envelope method as the other environment RPC errors. Core reads only the code, never the message. The source refusal diagnostic from #19302 keeps its own schema and API unchanged, and it takes precedence on a source `submit`. - `executeEnvironmentTask` still replaces each worker error with a generic error. After the #19302 diagnostic check, it now keeps only that closed code. - Run setup: a Runner mismatch before provider dispatch gets the error code `runner_revision_mismatch`. Setup schedules a bounded retry with its own reason, `runner_revision_wait`, and budget (10 attempts, 60 s apart). The reason is in the same resource-wait lane as workspace and AI connection waits, so a wait does not spend the failure retry budget. The guards are the same as for `plugin_worker_unavailable`. Recovery and legacy reconciliation treat the code like the other bootstrap-only codes. `scheduleBoundedRetryForRun` uses the Runner wait reason and budget for a mismatch run when a caller gives no reason, and the recovery sweeper reports exhaustion from that budget (`failedRunRetryBudgetSpent`). An answered interaction that hits a mismatch also waits for the Runner, and its successor keeps the resolved interaction continuation identity. - Move the managed project auth classification from `heartbeat.ts` to `classifyManagedProjectAuthFailure` in `run-preparation.ts`. Only a found `ai_connection_incompatible` conflict becomes a configuration failure. All other errors keep their own code. - Recovery notices: `ai_connection_incompatible` now has its own notice instead of the "secret/env bindings" text. An exhausted Runner wait gets the notice "Remote Runner is out of date". - Agent verification: the Runner probe and the managed project auth precheck report a mismatch as a retryable check. `assertHarnessTestPassed` maps it to the new lifecycle code `runner_revision_mismatch`. That code never counts toward the three-failure park. The agent checks again every 60 s and becomes ready when the Runner matches. - Document the error code in `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`. ## Verification - `cd packages/plugins/sdk && npx vitest run tests/environment-task-error-code.test.ts tests/environment-task-error.test.ts tests/worker-rpc-host.test.ts`: the code crosses the real `environmentTask` RPC. A plain error with the same message gets no code. Other methods get no envelope. The #19302 diagnostic tests still pass. - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts`: a fresh or wrapped mismatch gets `runner_revision_mismatch` and the runner retry budget, with no configuration failure. A started, untrusted (message only) or native run keeps `setup_failed` with no retry. A real `ai_connection_incompatible` stays `configuration_incomplete` with no retry. Other errors, such as an unreachable execution host, are not reported as an authentication conflict. - `npx vitest run src/services/execution-recovery-attempt.test.ts`: Runner waits do not change `failureRetries`, and earlier failures do not shorten the wait. - `npx vitest run src/services/heartbeat/retries.test.ts src/modules/run-dispatch` (embedded Postgres): a recovery caller without options gets the Runner wait budget and its exhaustion. An answered interaction waits for the Runner, keeps its continuation identity and does not spend failure retries. - `npx vitest run src/__tests__/environment-task-runtime.test.ts`: sanitization keeps only the code. No worker text or cause survives. - `npx vitest run src/modules/agent-lifecycle/__tests__/verification-retry.test.ts` (embedded Postgres): a `verifying` agent keeps checking after five mismatches. It never gets `retryRequired`. It becomes ready when the next check passes. - `npx vitest run src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts src/__tests__/agent-harness-verification-remote.test.ts src/services/recovery/stranded-notice.test.ts src/services/recovery/service.pause-durability.test.ts src/services/legacy-execution-recovery.test.ts`. - `npx tsc --noEmit` passes for `packages/plugins/sdk` and `server`. - One test in `src/__tests__/heartbeat-process-recovery.test.ts` ("redacts opaque environment-bound credentials from Sentry diagnostics") fails on my machine. It fails the same way on the base commit without this change. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. ## Risks - Low to medium. The change adds one new setup failure code and one lifecycle code. Other failures keep their current behavior. - A Runner mismatch must come from a plugin that throws `PluginEnvironmentTaskError`. Until an environment plugin does that, behavior is the same as today. - The run retry budget is 10 minutes. If a Runner update takes longer, the task goes to board recovery with the Runner notice. - A remote Runner run without a managed AI connection reaches the Runner check only after native dispatch. That configuration is not supported, so this change does not handle it. Such a run keeps the native session retry (3 attempts, 30 s apart). - #19172 edits the same precheck call in `agent-environment-test.ts`. Whichever PR merges second needs a small rebase. Keep the mismatch check before its generic retryable check. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (code search, shell and test runs). Reasoning mode: standard agentic coding. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 14:34:23 -08:00
DottaandClaude Opus 5.5	00c7738513	fix(task-chat): keep messages sendable after a declined connection request (#19331) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The task chat composer shows a queue of saved input that waits for the agent's next turn > - When a person declines a connection request (for example Slack), the server queues a read-only receipt so the agent learns the outcome > - The composer showed that receipt with its raw JSON receipt in hover text, and the receipt hid the person's next message > - The next message could not be edited, discarded, or sent, so the chat looked stuck after a simple "Decline" > - This pull request shows a readable status for receipts and keeps the person's own later messages visible and actionable > - The benefit is that declining a connection resolves cleanly and the conversation continues ## Linked Issues or Issue Description **What happened?** During an onboarding test on staging, a person declined the agent's "Connect Slack" request. A queued steering item then stayed above the composer. It could not be edited. Its hover text showed raw JSON. Sending the next message produced an error. **Steps to reproduce** 1. Give a task to a Paperclip runner agent whose run is active or waits on a busy workspace. 2. Let the agent request a connection, then decline it. 3. See the queued item "Rejected: Connect Slack". Hover it: the title is the full JSON receipt. 4. Send another message. The message never appears in the queue. Its row controls stay disabled. Send or Interrupt on the receipt queue returns `409 This queue does not support interruption`. **Expected behavior** A declined connection request resolves cleanly. It shows a readable status. The next message can be sent. **Evidence (staging wake diagnostics for the test task)** - 17:52:09: the decline wake for the Slack request. It stayed deferred, was never claimed, and was skipped at 18:21. - 17:52:10: a later retry wake for the plan-approval continuation. - `GET /issues/:id/queued-comments` projects the oldest matching wake. From 17:52 to 18:21 the composer showed the Slack receipt. Refs: related to the separate plan-approval queued-message fix, which changes the recovery sweep. This PR only touches the queue projection and the receipt label. ## What Changed - `server/src/services/queued-interaction-response.ts`: add `queuedInteractionSummary()`. A receipt now has one readable line, for example `Declined: Slack connection`, `Connected: GitHub connection`, or `Accepted: Build the app`. The entry `source.summary` carries it. The steering body for the agent still includes the typed JSON receipt. - `server/src/routes/issues.ts` (`findQueuedCommentWake`): a queued user-message wake now takes precedence over a receipt wake. A receipt shows only when it is the only queued input. The receipt wake itself is unchanged and is still delivered to the agent. - `server/src/modules/wake-queue/adapters/postgres.ts` (`findExistingDeferredWake`): skip interaction-receipt wakes when a new message picks the deferred wake it merges into. Before, the receipt was the oldest deferred wake, the merge was skipped, and every later message started its own wake. Only the first message then appeared in the queue. Receipts still never absorb user input. - `packages/shared/src/types/issue.ts`: add the optional `summary` field to `IssueQueuedCommentEntry.source`. - `ui/src/components/task-chat/TaskChatQueuedMessages.tsx`: add `queuedMessageLabel()`. The row text, the hover title, and all aria-labels use the summary. They never use the raw steering body. ## Verification Red/green tests were written first and run against the unchanged code. New tests: - `server/src/__tests__/issue-queued-comments-routes.test.ts` - `shows a declined connection receipt as a readable status, not its JSON` - `keeps the next message visible and actionable after a declined connection (running run | stopped run)`: declines Slack, posts the next message, and asserts that the queue lists that message with `canEdit`/`canDiscard`, that the discard works, and that the receipt wake is still pending for the agent. - `queues every later message together after a declined connection (legacy run | stopped run)`: two messages after a decline both appear in one editable queue, and the receipt wake keeps no message ids. - `ui/src/components/task-chat/TaskChatQueuedMessages.test.tsx` - `shows a declined connection request as readable text, never its raw JSON receipt (BUG-08)` **Red (unchanged code):** ``` server: × shows a declined connection receipt as a readable status, not its JSON AssertionError: expected { kind: 'interaction', …(3) } to match object { …, summary: 'Declined: Slack connection' } server: × keeps the next message visible and actionable after a declined connection (running run) server: × keeps the next message visible and actionable after a declined connection (stopped run) AssertionError: ["Declined: Slack connection"]: expected undefined to match object { canEdit: true, canDiscard: true } (the queue lists only the receipt; the new message is missing) Tests 3 failed | 85 skipped (88) ui: × shows a declined connection request as readable text, never its raw JSON receipt (BUG-08) AssertionError: expected 'Reorder queued message: Rejected: Con…' not to contain '{' Tests 1 failed | 23 skipped (24) ``` An exploratory probe on the unchanged code also showed the send error on this path: `POST /queued-comments/interrupt` on the receipt queue with a running native run returns `409 {"error":"This queue does not support interruption"}`. The exact UI text that the tester saw was not available. Second commit (review follow-up), unchanged `findExistingDeferredWake`: ``` server: × queues every later message together after a declined connection (legacy run) AssertionError: expected [ Array(1) ] to deeply equal [ …(2) ] Tests 1 failed | 1 passed | 88 skipped (90) ``` With the change: `Tests 2 passed | 88 skipped (90)`. **Green (with this change):** ``` pnpm --filter server exec vitest run src/__tests__/issue-queued-comments-routes.test.ts Test Files 1 passed (1) Tests 90 passed (90) server: heartbeat-comment-wake-batching + src/modules/wake-queue Test Files 11 passed Tests 322 passed (322) server: retry-scheduling, wake-diagnostics, question-response-delivery, comment-wake-batching, dot-runner, responsible-user-invariant, stale-queue-invalidation, queued-comments routes Test Files 8 passed Tests 357 passed (357) pnpm --filter ui exec vitest run src/components/task-chat/TaskChatQueuedMessages.test.tsx src/lib/issue-queued-comment-queue.test.ts Test Files 2 passed (2) Tests 31 passed (31) server: question-response-delivery, connection-intents-service, satisfied-connection-intents Test Files 3 passed Tests 199 passed (199) ``` A first full run of the route file hit a 30s `beforeAll` timeout while embedded Postgres started on a loaded machine. One new stopped-run case was also flaky, because the agent could dispatch the new message before the assertion. I fixed the flake: the test now occupies the agent on another task (`maxConcurrentRuns: 1`), which is the existing pattern in this file. I confirmed again that this test fails without the route fix. Typecheck: `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter @paperclipai/server typecheck`, and `pnpm --filter @paperclipai/ui typecheck` pass. `pnpm check:token-gates`: all gates clean. I did not run the full `pnpm test:run` or `pnpm build` locally. CI runs them. ## Risks - Behavior change: when a receipt and a user message wait at the same time, the composer now shows the message queue. The receipt is no longer shown there until it is the only queued input. The receipt wake and its delivery do not change. The resolved card in the thread still shows the outcome. - The first line of the steering body for connection receipts changes from `Rejected: Connect Slack` to `Declined: Slack connection`. The JSON receipt for the agent does not change. - Out of scope (unchanged on master, reproduced without any receipt): while a native run is running, each new message gets its own queued run, and the queue shows only the first one. Messages that join a `scheduled_retry` run's context also do not appear in the queue projection. - Overlap: the plan-approval queued-message fix (BUG-03) changes the recovery sweep, not this code. Both fixes touch the same symptom area. They do not edit the same functions. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, browser inspection of staging, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 21:03:37 +00:00
3 people	b929ad43de	fix: keep dependency waits neutral and explain GitHub access failures (#19315) ... ## Thinking Path > - Paperclip manages agent work and shows users what can move it forward. > - Task status and tool approval cards describe different parts of that work. > - A queued dependency wait has no provider attempt to retry. > - GitHub can reject an approved action while other connection tools still work. > - These cases showed a false Retry banner and an unhelpful provider error. > - This pull request keeps dependency waits neutral and gives GitHub access failures a direct repair path. ## Linked Issues or Issue Description Related: #19268 preserves pending Retry preparation and current failure causes. **What happened?** A queued wake cancelled by the dependency gate appeared as a failed attempt with Retry. A GitHub MCP permission denial appeared as raw HTTP error text after an approved repository action. **Expected behavior** Keep the task's dependency state. Show GitHub access failures with a clear explanation and a link to the selected connection's Permissions page. Preserve actual failures, provider details, and approval gates. **Steps to reproduce** 1. Queue a wake, then leave its task dependent on unfinished work before dispatch. 2. Open the task after the dependency gate cancels that wake. 3. Separately approve a GitHub action whose MCP result contains `403 Resource not accessible by integration`. 4. Reopen the saved tool approval. ## What Changed - Recognize both queues' unstarted dependency-wait receipts in cancellation provenance and task projections. No dispatch or recovery gate changes. - Classify GitHub's specific permission error after MCP result normalization. Keep result validation, the raw diagnostic, and working tools available. - Apply the same display code to both historical GitHub error codes. Identify the provider from company-scoped connection metadata, including renamed apps, without rewriting stored decisions. - Show a token-based inline access card with **Manage GitHub access** and collapsed **Error details**. - Give agents trusted recovery steps after an approved GitHub action fails. Keep provider text in the untrusted result and prevent automatic replay. - Add regressions, a production-component Storybook case, and execution/connector documentation. ## Verification - Red before the fix: cancellation provenance, dependency-wait projection, the real queue cancellation, approved MCP failure classification, saved interaction projection, and the error card assertions failed at the expected behavior. - Green focused tests: queue cancellation, one ready-task executor, approval-before-dispatch, one failed GitHub request, connection reuse, retained provider diagnostics, saved interaction reads, task list/detail projection and redaction. - UI: 106 component tests passed. Token gates and Storybook catalog checks passed. - `node scripts/typecheck-low-memory.mjs`: passed. - Local source instance: verified a saved dependency wait stays neutral after reload; a saved GitHub error shows repair guidance, expands its original error, and opens the correct connection Permissions page. PostgreSQL and HTTP APIs were real. The GitHub denial was a controlled MCP fixture; no account or provider permission changed. - `pnpm build`: passed for all workspace packages, the Runner binary, server, CLI and UI. The server build and all server typecheck layers also passed after the review repair. - Additional safety checks passed for realistic repository arguments, active recovery, failed Retry preparation, and bounded provider diagnostics. - Review repair: 45 pure tests and 7 PostgreSQL cases passed. These cover both historical error codes, renamed apps, and company/provider/application identity boundaries. A second local browser check verified the renamed app, retained error detail, and correct Permissions destination. - Final review regressions: the dispatcher and metadata cases failed before the repair. Afterward, 25 focused cases and 59 projection/identity cases passed, including connection-key precedence. The native wait and app-only GitHub source also passed a local browser check. - The historical read adds at most one connection query per interaction read batch. It selects IDs and the provider key only and runs only for candidate permission failures. It adds no provider requests. - Full local `pnpm test:run` stopped at the chat suite's `afterAll` cleanup timeout. All 268 assertions in that group and the three prior chat groups passed. The failure is retained here; it is not counted as a full-suite pass. CI passed on `e8a692e1e2` after one workspace-startup rerun. Fresh CI is pending for the combined head. Its first workspace-a attempt timed out in the CLI company import/export test's child-server startup wait; 4,968 UI tests and 287 CLI tests passed in that job. The same shard passed on the preceding commit. The workspace shard passed on its one rerun. All other test, build and browser jobs passed. The [first failure](https://github.com/paperclipai/paperclip/actions/runs/42093823928/job/127478869941) is retained in the Actions history. Greptile reviewed the preceding head at 5/5. Fresh review is pending for the combined head. ## Risks The GitHub classifier matches the documented permission-denial phrase, not every 403. Rate limits, missing resources and other providers retain their existing error behavior. Historical normalization affects display only. A dependency cancellation needs an unstarted run and a supported queue receipt; current user gates, Retry preparation and recovery remain authoritative. No schema migration or credential changes. ## Model Used OpenAI GPT-6, through the Codex agent. The exact runtime model ID and context-window size are not exposed in this session. Used reasoning, repository tools, TypeScript, PostgreSQL, MCP fixtures and browser verification. The parallel agent-recovery commit records Anthropic Claude Opus 5.5 as co-author. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the bug template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run the focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open findings - [x] I will address reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 20:55:34 +00:00
DottaandClaude Opus 5.5	7644045bc5	fix(server): remove a settle deadlock and three server test races (#19314) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server Vitest shards gate every pull request > - Three open pull requests (#19289, #19267, #19266) failed server tests in code that they do not change > - Each failure had a different cause. One cause is a real product deadlock, and three causes are test races > - Random CI failures slow every contributor and hide real regressions > - This pull request removes each root cause. It does not add retries or skips > - The benefit is fewer false CI failures and one less deadlock in external-agent settlement ## Linked Issues or Issue Description Refs #19289, #19267, #19266 **What happened?** Server tests failed at random on pull requests that did not touch them: - #19289, job 127412331924: `dot-runner.test.ts` > "signed readiness, native tools and finalization through Dot local Rust" failed with `PostgresError: deadlock detected` on the run-event append (`select ... from heartbeat_runs ... for update`). - #19267, job 127427384499: `tool-access-service.test.ts` > "keeps an organization audience fail-closed when empty replacement races with member archive" received a widened grant instead of the 409 error. - #19267, job 127410158948: `heartbeat-process-recovery.test.ts` > "does not reap active adapter executions started by another heartbeat service instance" failed with "Timed out waiting for adapter execution to start". - #19266, job 127404184242: `voice-session-store.integration.test.ts` > "schedules phone reply retries at the saved rate-limit deadline without early database scans" failed with a duplicate key on `companies_issue_prefix_idx`. **Root causes** 1. Dot deadlock (product bug). `settle` in `native-broker.ts` takes the external admission guard. Then it updates the same assignment row in two or more statements. PostgreSQL re-runs the foreign-key checks when a transaction updates a row version that it wrote earlier, so the second UPDATE takes `FOR KEY SHARE` on the heartbeat run. Execution admission locks the issue and the run `FOR UPDATE` before it takes the guard. Thus settle holds the guard and waits for the run, and admission holds the run and waits for the guard. The run-event append can queue behind them as a third participant, and the database can kill it. A local reproduction with `log_lock_waits` showed the RI check: `SELECT 1 FROM ONLY heartbeat_runs x WHERE id = $1 FOR KEY SHARE OF x`, during `update dot_runner_assignments set status='settled'`. 2. Tool-access race (test bug). The empty replacement and the member cleanup both waited for the first replacement to commit. Both waiters woke at the same time and raced for the membership row. A replacement that wins that race widens access before the cleanup, and the product code documents that outcome as valid ("Whichever transaction wins is then authoritative"). The test only expected the cleanup to win. 3. Heartbeat timeout (test bug). The run start includes real runtime skill preparation for a new company. I measured 3.3 s to 5.5 s locally before the adapter starts. The test gave this step a fixed 3 s budget. 4. Voice prefix collision (test bug). Each of about 110 fixture companies used `V` plus seven random hex digits as its unique issue prefix, so two companies could get the same prefix. **Master:** These tests do not fail on recent master runs (the master workflows do not run the server shards). The failures are timing-dependent and are not specific to a branch. ## What Changed - `server/src/modules/external-agents/native-broker.ts`: `settle` writes the assignment in one UPDATE. This applies to both Dot and Muse. The update sets the same columns under the same conditions as before. - `server/src/__tests__/dot-runner.test.ts`: added "settles accept/finish under the admission guard without waiting on a run that admission holds". The test holds the run `FOR UPDATE` like admission does, then runs settle, then takes the guard. Before the fix, settle waits and PostgreSQL reports a deadlock. After the fix, settle completes. - `server/src/__tests__/tool-access-service.test.ts`: start the empty replacement after the cleanup holds the membership row. Then wait until the replacement is blocked by the cleanup. The test still proves that the replacement re-checks the member status after the cleanup and fails closed. - `server/src/__tests__/heartbeat-process-recovery.test.ts`: wait for the adapter to start. Fail at once if the run settles before the adapter starts. The test timeout is now 30 s, because the start (3 s to 6 s) and the existing 5 s settle wait came close to the 15 s default. - `server/src/__tests__/voice-session-store.integration.test.ts`: each fixture prefix includes a per-run sequence, so prefixes in one run are always distinct and stay at 10 characters. ## Verification - Red/green for the deadlock: with the `native-broker.ts` change removed, both new tests fail with `deadlock detected`. With the change, both pass. - `npx vitest run src/__tests__/dot-runner.test.ts` (server): 57 passed. - Before the fix, I ran the Dot Rust test 8 times with `deadlock_timeout=50ms` and got the deadlock in about 1 of 8 runs. Two further runs of 16 iterations each passed. - `npx vitest run src/__tests__/tool-access-service.test.ts -t "fail-closed when empty replacement races"`: 3 passed, run 4 times. - `npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "does not reap active adapter executions"`: this failed locally before the change (start took 3.3 s to 5.5 s). It passes after the change. - `npx vitest run src/__tests__/voice-session-store.integration.test.ts`: 132 passed. - `npx vitest run src/__tests__/muse-control.test.ts`: 256 passed. This file covers the Muse settle path. - Not run locally: the full `pnpm -r typecheck` / `pnpm test:run` / `pnpm build`. CI runs them. A server `tsc --noEmit` shows no errors in the changed lines. The existing errors come from the unbuilt runner package in this worktree. ## Risks - Low risk. The settle change keeps the same column writes and conditions, now in one statement. It removes a lock wait. It does not add a lock. - The other changes are only in tests. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, and GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 13:27:21 -06:00
DottaandClaude Opus 5.5	10a7d039c8	fix(dispatch): keep a workspace wait parked while an ended writer's save settles (#19285) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run that finds its task workspace held by another run waits as a `workspace_busy` scheduled retry. #19143 keeps that wait as one parked retry while the holder is live. > - A remote task workspace stays held by its last writer run until the writer's save and sandbox release settle. This hold continues after the writer run ends, for example when a reassignment stops the writer. > - The parked-wait check counted only a `running` holder. For an ended writer whose save had not settled, it released the waiter at each due check. Preparation deferred again, cancelled the run, and created a new retry run. > - After a mid-run reassignment, the new owner got five cancelled `workspace_busy` runs in eleven minutes, and the task did no work. > - This pull request makes the parked wait use the same writer-hold rule as workspace reservation, so the waiter stays parked until the hold releases. > - The benefit is one durable wait, one start when the hold releases, and no churn of cancelled runs. ## Linked Issues or Issue Description Refs #19143 (parks a workspace-busy wait). Refs #19184 (releases a recovered hold for a new assignee after reassignment). **What happened?** A board user reassigned a task in the middle of a run. The old run was cancelled with `issue_reassigned`. The new assignee's runs were each created, started and cancelled with `workspace_busy`: "Shared execution workspace is busy: run … ended, but its workspace save has not settled". This repeated every 1–2 minutes (five cancelled runs in 11 minutes). Every cancellation named the stopped run of the previous assignee as the holder. The new assignee's other tasks were not the holder. **Expected behavior** The reassigned task waits as one parked retry. It starts once when the stopped writer's save and sandbox release settle. The wait stops at the one-hour holder cutoff. **Steps to reproduce** 1. Give a task a remote execution workspace whose writer run has ended, but whose save has not settled (`writerRunId` set, `dirty: true`). 2. Wake the task's assignee. The run defers with `workspace_busy` and schedules a retry. 3. Make the retry due and promote scheduled retries. Before this fix, the retry is promoted and dispatched, defers again, and creates another run. ## What Changed - New `server/src/services/remote-workspace-writer-hold.ts`: `readRemoteWorkspaceWriterHold` holds the rule for who holds a remote workspace (a running writer, or an ended writer whose save or sandbox release has not settled). The code is moved from `reserveRemoteWorkspace` without a behavior change. - `reserveRemoteWorkspace` uses that function. It throws the same errors as before. - `parkWorkspaceBusyRetryInTx` (from #19143): when the recorded holder is not running, it reads the same hold for the waited execution workspace. If the recorded holder is an ended writer that still holds it, the retry stays parked. The cutoff clock starts at the writer's end, the same value that reservation measures. Earlier waiters keep their arrival order. - New `WORKSPACE_BUSY_SETTLING_RECHECK_MS` (60 s). Settling has no single completion point that can wake the waiter, so a wait behind a settling writer re-checks every minute. Each check creates no run. In the promotion transaction it reads the workspace, the writer run, its task, and the writer's leases (one row per sandbox lease of that run). The checks last only while the wait exists, up to the one-hour cutoff. - `doc/execution-semantics.md`: documents the parked wait behind a settling writer. ## Verification - Red before the fix: `cd server && npx vitest run src/__tests__/heartbeat-remote-workspace-busy.test.ts`. The two new tests failed: the due retry was promoted (`expected [ … ] to not include '<retry id>'`). - Green after the fix, with: - `parks one waiter while an ended writer's save settles, then admits it once the hold releases`: three due checks keep one parked run, each re-check is bounded by the writer's end plus one hour, there is one `workspace_busy` cancellation in total and no new retry run. After the hold releases, the next check promotes the same run. - `stops parking behind an ended writer once its unsettled save reaches the holder cutoff`: parked before the cutoff, promoted after it (preparation then fails with the existing unfinished-writer error). - Existing suites pass: `heartbeat-workspace-busy.test.ts` (live-holder parking, arrival order, stop, workspace move), `modules/run-dispatch`, `remote-workspace-reservation.test.ts`, `remote-workspaces.integration.test.ts`, `isolated-remote-workspace-source.test.ts`, `heartbeat/workspace-preparation.test.ts` (476 tests in all). - `tsc --noEmit` for `server` shows no errors in the changed files. - Not done: a live run on staging. The fix is not deployed. ## Risks - Low. Parking applies only to a retry that recorded a `workspace_busy` wait on an execution workspace whose recorded writer is that holder. If the hold settles, moves to another run, or the task changes workspace, the retry is released to normal dispatch as before. - The wait stays bounded by the existing one-hour holder cutoff from the writer's end. After that, preparation fails as before. A user can still stop the parked run. - One writer per workspace is unchanged. Reservation still decides admission with the same rule. - The one-minute re-check runs only while a wait exists. It is not idle polling. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 11:48:29 -06:00
Nicky Leach	53937be283	fix(agents): select environment plugins only for agents on that environment (#19173)	2026-11-04 20:59:02 -08:00
DottaandClaude Opus 5.5	06826915d7	fix(dispatch): park a workspace-busy wait instead of recreating runs (#19143) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Child tasks can reuse their parent's execution workspace. Only one run at a time may write in a shared sandbox workspace. > - A run that finds the workspace held by another live run waits with a `workspace_busy` scheduled retry. > - That wait cancelled the run and created a new retry run every 60–120 seconds, until the holder ended. Each check made one cancelled run record. > - Three sibling tasks that waited for a parent made 15 cancelled runs in 13 minutes. They also got the workspace in random order, not in arrival order. > - This pull request keeps the waiting run as one parked retry while its holder is live, and wakes it when the holder ends. > - The benefit is a clean run history, a prompt handoff, and fair arrival-order access to a shared workspace. ## Linked Issues or Issue Description Refs #18691 (the workspace wait for a live holder). Refs #19075 (sends a stopped writer with a pending save into the same wait). Refs #12587 (an earlier proposal about fairness for this retry). **What happened?** A parent task delegated three subtasks that inherit its execution workspace. Each subtask run was created, then cancelled with `workspace_busy` ("Shared execution workspace is busy: run … is still running"), then created again about every 1.5–2 minutes. One subtask got 8 cancelled runs, one got 5, and one got 2. Every cancellation named a holder that was really running: first the parent run, then the sibling that got the workspace. So serialization was correct. The cost was the churn of run records, and the subtask that waited first ran last. **Expected behavior** A run that needs a busy shared workspace waits as one durable record. It continues when the holder finishes. Siblings that wait for one workspace get it one at a time, in arrival order. Two writers never share the workspace, and the user can still stop a waiting run. **Steps to reproduce** 1. Bind two or more issues to one shared execution workspace in a sandbox environment. 2. Keep a run live on one issue (the holder). 3. Wake the other issues. Before this change, each due check promotes the retry, cancels it with `workspace_busy`, and creates a new retry run. After this change, the retry stays parked until the holder ends. **Paperclip version or commit** Observed on staging at `53735db7`. The code is unchanged on `master`. ## What Changed - `finalizeWorkspaceBusyDeferral` records `contextSnapshot.workspaceBusyWait` = `{ key, holderRunId, waitingSince }`. The key is `execution:<id>` or `project:<id>`. `waitingSince` carries over when the run defers again for the same workspace. - New `server/src/modules/run-dispatch/adapters/workspace-busy-wait.ts`: - `parkWorkspaceBusyRetryInTx` runs in the scheduled-retry promotion transaction, after the existing gate allows the retry. If the recorded holder is still running and is not stale, or an earlier waiter for the same workspace is still scheduled, queued, or running, the retry stays `scheduled_retry`. Only its deadline moves. It does not create a run, an event, or a wake. The fallback deadline is 5 minutes, and never later than the holder's one-hour stale point. - `wakeWorkspaceBusyWaitersInTx` runs in the transaction that ends a running run (`writeRunStatus`). It makes due the retries that wait for that run, and the retries that wait for the same workspace as that run. It registers the scheduled-retry dispatch notification before the write. - `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` moves to the run-dispatch domain, so the deferral gate and the parked check use one value. `heartbeat/retries.ts` exports it again. - A retry without a recorded wait (for example, a retry from before this change) keeps the old behavior. - `doc/execution-semantics.md` describes the parked wait. ## Verification - Red before the fix, green after (`server/src/__tests__/heartbeat-workspace-busy.test.ts`, "reused shared execution workspace in a sandbox"): - "waits on a live holder in one parked run, then runs once when the holder ends": before, the retry was promoted while the holder was live. After, it stays parked through due checks at +1 s, +10 min, and +30 min. Ending the holder makes it due at once. It runs exactly once, and there is only one `workspace_busy` cancellation. - "hands the workspace to three waiting siblings in arrival order, one writer at a time": before, nothing woke the waiters when the holder ended. After, exactly one waiter is admitted per handoff, in arrival order. Each runs once, and there are 3 cancellations in total (one first deferral for each sibling). - "still lets the user stop a parked waiter": before, the waiter was promoted while the holder was live. After, Stop cancels the parked retry, and nothing brings it back. - Safety case (green before and after): "lets a stale holder reach the existing quiet-holder failure instead of parking forever". - `npx vitest run src/__tests__/heartbeat-workspace-busy.test.ts`: 35 passed. - `heartbeat-remote-workspace-busy`, `heartbeat-retry-scheduling`, `src/modules/run-dispatch`, `heartbeat/retries.test.ts`, `execution-projection.test.ts`: 266 of 267 passed. The one failure, "records pre-provider quota rejection…", is a timing flake on a loaded machine and passes when it runs alone. It does not use this path. - `tsc --noEmit` reports no errors in the changed files. - Staging evidence was read only. This change is not deployed. ## Risks - One extra indexed `SELECT … FOR UPDATE` on `scheduled_retry` rows when a running run ends. The update and the notification happen only when waiters exist. - If an earlier waiter stays queued for a long time (for example, its agent is paused), later waiters for the same workspace wait behind it. Each check is a deadline update, not a run. The fallback check runs every 5 minutes. - A remote writer that has ended but has not finished its save (#19075) is not parked, because its run is no longer `running`. Such a wait can still make a small number of deferrals until the save settles. - Low migration risk: no schema change. Old retries without the new context key keep the old behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, browser read of staging run records) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 22:01:37 -06:00
3 people	19c710fe93	fix(issues): record an interrupt comment as the user's Stop (#19149) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A board user can interrupt a running agent with a comment (`interrupt: true`) to redirect the work > - An interrupt is a Stop plus a message, but it did not record the user's stop identity the way the Stop control does > - The release path therefore treated an interrupted native run as a failed run: it blocked the task and took the finalization lease from the runner that was still settling > - Workspace preservation then stopped the sandbox under that live runner, so the runner reconnected for about 6 minutes and the stop could not be verified afterward > - The redirect message stayed saved, but it waited on workspace repair that could not finish > - This pull request records the board user's stop identity on interrupts, the same as the Stop control and terminalization stops > - The benefit is that an interrupt now settles like a Stop and the redirect can continue ## Linked Issues or Issue Description Refs #19122 (the Stop-then-redirect fix for the dispatch gate; this PR covers the interrupt-comment path). **What happened?** A board user posted a comment with `interrupt: true` on a task with a running native run (Daytona sandbox). The run was cancelled with `operator_interrupted` and the native cancel was acknowledged. Within a second, the task went to `blocked` and a `native_continuation_requires_reconciliation` hold was created. The runner transport then logged `state=reconnecting` for about 6 minutes, and the run ended with `runner_process_termination_unconfirmed: ... Daytona sandbox lease ... is no longer active`. Workspace recovery started about 10–17 minutes later and failed with `agent_stop_unconfirmed` / `provider_unavailable`. The redirect comment stayed saved as a deferred wake waiting on `workspace_repair_required`. No follow-up run started. **Expected behavior** An interrupt behaves like Stop followed by a message. The stopped runner settles normally. The task is not blocked as a failure. The saved message continues once the previous execution has stopped. **Steps to reproduce** 1. Start a native run on a remote sandbox. 2. Post a board comment with `interrupt: true`. 3. Observe the task become `blocked` with a `native-continuation:<runId>` hold, and the run's finalization row rewritten to `terminal_failure` with no lease owner while the executor is still settling. 4. Observe workspace preservation stop the sandbox, the runner transport reconnect until its grace ends, and recovery fail to confirm the stop. **Root cause** `recordNativeTerminalRecoveryIfNeeded` (`server/src/modules/wake-queue/adapters/postgres.ts`) skips intentional stops through `isAcknowledgedNativeStop`. That check requires `resultJson.cancelledByActorType === "user"` and `cancelledByUserId`. The Stop control and terminalization stops record these fields. `operatorInterruptCancelOptions` (`server/src/routes/issues.ts`) recorded only `interruptedByActorType` / `interruptedByActorId`. So an acknowledged interrupt took the failure branch. That branch blocks the task and sets `nativeRunFinalizations` to `terminal_failure` with `leaseOwner: null`. Cancelled-workspace preservation then saw a terminal row with no owner and stopped the sandbox while the runner was still settling. **Paperclip version or commit** Seen on staging at 69dea878c0. The touched code is unchanged on current `master`. **Agent adapter(s) involved** Not adapter-specific (core bug). Seen with the Paperclip runner adapter on a Daytona sandbox. ## What Changed - `operatorInterruptCancelOptions` now records `cancelledByActorType: "user"` and `cancelledByUserId` for a board user, in addition to the existing interrupt fields. All three interrupt call sites (comment POST, issue PATCH, queued-comment interrupt) use it. - Route regression in `issue-update-comment-wakeup-routes.test.ts`: an `interrupt: true` comment on a running native run sends these fields, and the route's actual fields satisfy `isAcknowledgedNativeStop` with an acknowledged native stop receipt. - Release-path regression in `wake-queue/adapters/postgres.test.ts`: an acknowledged interrupt Stop, while the executor still owns finalization, releases the task without blocking it, creates no recovery hold, keeps the finalization lease owner, and keeps the saved redirect deferred. ## Verification - Red before the fix: - The route test failed because `cancelRun` received no `cancelledByActorType` / `cancelledByUserId`. - The release-path test failed with the old interrupt fields: the task was blocked, a hold was created, and the finalization owner was cleared. - Green after the fix: - `cd server && npx vitest run src/__tests__/issue-update-comment-wakeup-routes.test.ts src/modules/wake-queue` - Related suites: `acknowledged-native-stop`, `issue-queued-comments-routes`, `issue-comment-reopen-routes`, `issue-execution-policy-routes`, `explicit-native-continuation`, `heartbeat-process-recovery`. - The staging incident was read through the read-only run, event, log, recovery-action and wake diagnostics APIs. - Not done: a live staging reproduction at the fixed revision. Deployment is out of scope. ## Risks - An interrupt now counts as the user's Stop everywhere the Stop control does. This includes case-handoff supersession in `cancelRun`: an interrupt supersedes a pending case handoff, like Stop. - Only board users get the stop identity. Agent actors cannot use `interrupt: true` (the route returns 403). - Runs interrupted before this change keep their old fields. They are not migrated. - Not changed here, and suitable for follow-ups: - `recordNativeTerminalRecoveryIfNeeded` can still clear a live finalization `leaseOwner` for other failed native runs. - The runner PRP ingress reconnect loop (`server/src/realtime/runner-prp-outbound.ts`) does not stop early when the sandbox lease is closed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, subagent code tracing, browser for read-only staging diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 21:58:51 -06:00
3 people	1a7eb70a37	fix(run-dispatch): save a redirect sent while a stopped run settles (#19122) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can stop a running agent and then send a new message that redirects the work > - The stopped run keeps settling for some seconds after Stop (backup and checkpoint persist) > - A message sent in that window is queued as a normal run before settlement records the workspace recovery hold > - The dispatch gate then withholds the unstarted run, but it saved the message only for sources with a native workspace-sync baseline > - For other sources (remote workspace, snapshot-only) it marked the wake `skipped`, so the message was lost and the task stayed in progress with no run and no wake > - This pull request saves the withheld message for any source when the message was written after the Stop > - The benefit is that a redirect after Stop is acted on exactly once instead of disappearing ## Linked Issues or Issue Description **What happened?** After Stop, a user posted a redirect comment with resume intent. The follow-up run was created while the stopped run was still persisting its checkpoint. When settlement finished, the dispatch gate cancelled the follow-up run with `execution_reconciliation_required` and an empty-looking error ("Recovering saved work…"). The user's wake was marked `skipped`. Nothing woke the task again. It stayed `in_progress` with no active or queued run. **Expected behavior** The message sent after Stop runs exactly once after the stopped run settles. The task never rests in progress with no run and no pending wake. **Steps to reproduce** 1. Start a native run whose stopped source has no native workspace-sync baseline (for example a remote-workspace sandbox run). 2. Stop the run. 3. Post a comment with resume intent while the stopped run still persists its checkpoint. 4. Let settlement record the workspace recovery hold. 5. Observe that the queued follow-up run is cancelled with `execution_reconciliation_required` and its wake is `skipped`. **Root cause** `deferUnstartedUserComment` in `server/src/modules/run-dispatch/adapters/postgres.ts` saved a withheld user comment only when the stopped source had a native workspace-sync baseline (the copyback replay case). The enqueue path saves or admits the same message when it arrives after the hold exists. The gap was only timing. Refs #18332 (added the native-baseline-only deferral this PR widens). **Paperclip version or commit** Reproduced on `master` at e107a75755. The touched code is unchanged on current `master`. **Agent adapter(s) involved** Not adapter-specific (core bug). Seen with the Paperclip runner adapter. ## What Changed - `deferUnstartedUserComment` now saves a withheld user comment as a `deferred_issue_execution` receipt for any stopped native source when the latest saved comment was written after the source's Stop (`finishedAt`). - The native-baseline copyback path is unchanged. - Input written before the Stop keeps the old behavior (`skipped`). Older queued input cannot undo a Stop. - Delivery uses the existing deferred-receipt path (`resumeRemoteStopComments` / `resumeDeferredExecutionReceipts` → `admitExplicitNativeContinuation` with `queuedCommentRequestId`). All continuation gates still apply. - New regression in `workspace-recovery.test.ts` that controls the timing: the message is queued while the stopped run is still active, then the hold is recorded, then the gate runs. - The existing snapshot-only queued-reset tests asserted the dropped message. They now assert that the message is saved and delivered once, like the native-baseline case. ## Verification - Red before the fix: the new test failed with the wake `status: "skipped"`. - Green after the fix: - `cd server && npx vitest run src/services/workspace-recovery.test.ts` (308 passed) - `cd server && npx vitest run src/modules/run-dispatch src/__tests__/heartbeat-undispatched-execution-wait.test.ts src/__tests__/issue-queued-comments-routes.test.ts src/services/explicit-native-continuation.test.ts src/__tests__/heartbeat-process-recovery.test.ts src/services/execution-continuation.test.ts` - Safety cases in the new test: - The redirect is saved, then delivered as exactly one successor run with the original user's authority. The old provider run is not executed. - Input written before the Stop stays skipped and creates no successor. - The UI Stop button uses the same `POST /heartbeat-runs/:id/cancel` path. A comment with `interrupt: true` cancels through the same `cancelRun` and then queues the comment wake. Both reach the same gate. These were checked by code reading, not by a separate test. - Not done: a live staging reproduction at the fixed revision. Deployment is out of scope. ## Risks - Behavior change: for sources without a native baseline, a post-Stop message is now saved and delivered after recovery, instead of dropped. Admission still blocks it until workspace repair is verified and the previous execution has stopped. While blocked, the task shows "Your message is saved." - If admission never succeeds, the message stays saved and visible. Before, it was silently lost. - No schema or API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, browser for read-only staging diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 17:39:48 -06:00
DottaandPaperclip	0eaabe5c0b	refactor(accounting): remove company serialization from paid settlement (#18917) ... Settle paid operations through operation-scoped transactions. Preserve exact ledger evidence and replay while moving aggregate budget decisions to durable deferred work. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-04 07:29:28 -06:00
DottaandPaperclip	d8c1b9d4b5	fix(runtime): reclaim failed remote checkpoint retries (#18877) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native recovery must preserve the original execution and provider state. > - A failed remote run can retain its provider PID after heartbeat releases the task execution lock. > - Normal resumption excludes that PID, while restart recovery rejects the released lock. > - Recovery must reclaim that exact execution only after current admission and durable ownership checks pass. > - Failed retries and admission holds need separate finite budgets. Neither can prove a provider stopped. ## Linked Issues or Issue Description **What happened?** A failed remote native run remains in retryable failure without a task execution owner. Normal resumption and restart recovery both reject it before inspecting the preserved remote provider. The current remote workspace reference format also misses the due-retry routing path. **Expected behavior** A due exact-checkpoint retry reclaims its original task execution only when current ownership and safety gates permit it. Newer task input prevents an old claim. Lost or ambiguous provider state still prevents replay or replacement. Ineligible recovery eventually becomes a quiet manual reconciliation obligation. **Steps to reproduce** 1. Start a remote native run and persist its provider checkpoint and encrypted Runner authority. 2. Fail with exact-checkpoint recovery and a retained remote PID. 3. Release the task execution lock at heartbeat completion. 4. Run ordinary resumption and restart recovery after the retry deadline. 5. Observe that both reject the same eligible run before remote inspection. **Paperclip version or commit** Reproduced from source at `d59791828482`; integrated with upstream `db9be4c5a6`. **Deployment mode** Remote workspaces with PostgreSQL. Local regressions use disposable PostgreSQL and encrypted checkpoint persistence. Related public repairs #18846, #18854, #18860 and #18841 cover finished per-turn owners, pre-provider failures, unstarted follow-ups and storage migration. They do not reclaim this failed run's released execution lock. Merged #18879 also adds scheduled-resume lock retention and reclaim. This PR keeps that compatibility and requires encrypted Runner authority, canonical source ordering and a charged failed retry before reclaim. Its scheduled-resume regression now uses a real encrypted authority and retains a snapshot-only refusal control. ## What Changed - Share normal admission checks and one guarded execution-binding restore/event helper between normal and remote resumption. - Keep agent-name normalization in a module with no imports. Preserve heartbeat's existing export. Native admission no longer loads heartbeat orchestration for a string key. - Recheck company, assignment, agent eligibility, Stop, pause, approval, dependencies, execution and checkout ownership before reclaim. - Reject newer task input using PostgreSQL creation-time and ID ordering, preserving timestamp precision. - Require the recorded remote target, encrypted Runner authority, exact checkpoint identity and existing controller fence. - Commit task binding, controller lease, run status and recovery event atomically. Recognize both remote workspace reference formats. - Charge failed reattachments against the existing three-provider-attempt budget, including retained-lock crash windows. Active controller adoption preserves its charged attempt. - Use the existing five-retry/30-minute system-recovery policy for admission holds. Ended tasks and newer sources exhaust it immediately. Preserve PID, checkpoint and task bindings, suppress resource settlement, and assign manual reconciliation with executor stop/state explicitly unverified. - Register recovery intent before deadline writes; preserve live/future deadlines quietly and roll back zero-row writes. Record changed hold reasons once and terminal dispositions separately. - Keep historical provider-budget exhaustion behind current admission/source/controller/authority checks. Make an existing orphan-cleanup fixture explicitly due. - Include exact plugin task leases that have no workspace reference. Reuse every reclaim and retry-budget guard, with a bounded company/task/run lease projection. - Share strict manual ownership proof across stale-lock cleanup, adoption, queue release and wake admission. Require termination receipts for every exact remote lease before normal cleanup can release either binding. - Preserve governing Stop through the existing audited cancellation path. Its acknowledgement does not release an unverified provider. Ordinary Retry can wait for exact Stop and file preservation, then queue one continuation. ## Verification - Current head is `5d7baf74aa89d00d7ac6e1493d38a6ce2ae7b7b5`. Its [current CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41782198858) is terminal and clean: 54 successful, three skipped and two neutral contexts. Required `ci / verify` and `ci / e2e` pass. Greptile is 5/5 on this exact head with no open findings. No fresh Paperclip 5/5 assessment is claimed. - The [prior complete CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41754365388) passed on `3e641466d3edcaa6dd5021d979627082d388eb05`, with Greptile and Paperclip both 5/5 and no open threads. - All 233 tests across the six affected heartbeat, restart-isolation, remote-admission, scheduled-resume and session-binding families pass on production-identical `a5104fa89f9cb493091e000bf21ed3858871ae46` under Node 24.22.2 and pnpm 9.15.4. They cover both workspace formats and plugin task leases, competing controllers, exact rollback, Stop, assignment/company boundaries, attempts, work-intent ordering, finite backoff and quiet held state. - The merge preserves #18931's per-run audit-failure isolation and this PR's unchanged-transition hold. Two initial composition cases failed because their setup or expected result described the older contract. The rollback test now expects the exact per-run deferral while retaining full coordinator equality and no work intent. The routine fixture writes a real encrypted checkpoint through the normal owner before reaching the actual routine unique constraint. No production admission guard or rollback assertion was weakened. A separate agent reviewed this exact merge and fixture delta without finding a blocker. - Full recursive `pnpm -r typecheck` and `pnpm build` pass on production merge `89c616fe45ceff75db450f67fd34224b4df7441d`. Subsequent commits change only three test files. Both the two-file `a5104fa` and final `5d7baf` server test-layer typechecks pass. - Four production-identical `a5104fa` probes pass with the rebuilt actual Rust Runner, a scripted ACPX subprocess and encrypted PostgreSQL checkpoints under Node 24.22.2. Workspace-reference and plugin-lease admission each cover surviving and lost providers. Replacement preserves one provider launch, one original side effect, the charged attempt and restored task execution binding. Lost providers reject snapshot recovery without replacement. This is protocol-boundary evidence, not a live Claude or hosted provisioning claim. - Initial `a5104fa` CI found one additional routine integration fixture missing the persisted `exact_checkpoint_resume` mode. Its encrypted checkpoint was already present. The exact case fails locally before repair and passes after recording that mode and expecting the charged provider attempt. Every exact routine conflict, rollback, checkpoint, task binding, successor-count and sibling-startup assertion remains. Final head changes only that fixture. The correct native-server group passes all nine files: 175 executed cases and 14 container-only skips, matching the now-successful native-server CI gate. - The earlier canonical HTTP Retry acceptance passed on the prior reviewed head with real PostgreSQL, the recovery worker and real files. Pending Stop protects both bindings and starts no successor. Exact termination and copyback queue one preserved-workspace continuation. No browser or live-model outcome is inferred from that test. - The predecessor's full local `pnpm test:run` failed in an unchanged Discord Gateway fixture with a 200 ms lease and a 422 initial ownership error. Focused predecessor and untouched-base controls pass; that full-run cause remains unproved. The required command was also started on production-identical `a5104fa`. All six local chat shards passed, covering 1,850 cases. The duplicate local invocation was interrupted with SIGINT while the general-server group was still running after this head's complete CI and the combined repair qualification had passed. The wrapper exited 1 on cancellation, no process retained its test log, and no full local pass is claimed. An earlier attempt to select the native lane through the CI wrapper instead selected local package sharding: 2,631 cases passed and 13 failed. Those separate failures are retained without causal attribution or unrelated source changes; that run is not counted as native-server qualification. The direct native group and current complete CI matrix pass. - Live Claude Opus 5.5 continuation, hosted plugin provisioning, deployment and recovery of historical tasks remain unverified. No original task was retried or mutated for this verification. ## Risks - Restoring a task binding grants remote inspection; it does not prove that the provider survived. - Unknown effects, lost providers and missing semantic results retain their existing reconciliation requirements. - Exhausted admission holds preserve execution bindings and resource evidence for manual reconciliation. They do not authorize a fresh executor or provider/resource release. - The source guard is conservative: newer task input prevents recovery of an older failed source. - No database migration or provider profile change is required. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, code editing and shell execution. The runtime does not expose the exact model snapshot or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:18:07 -06:00
DottaandPaperclip	f211d44c00	fix(heartbeat): preserve queued coalesced comments at claim (#18913) ... ## Thinking Path > - Paperclip manages agents and their work. > - The wake queue records task comments before an agent starts. > - A queued run can accept another comment through coalescing. > - Claim rebuilds that run's input from its primary wake. > - Coalescing updated the run and its receipt but left the primary batch unchanged. > - This change saves the accepted batch atomically in the owning writer. > - The benefit is complete input at claim without repeating old work. ## Linked Issues or Issue Description Related: #18887 preserves offline queued turns. #13948 addresses equivalent automatic wake deduplication. This change repairs the accepted comment batch at the existing claim boundary. **What happened?** A queued run started with comment A. Admission accepted comment B, saved `[A, B]` in the run context, and finished B's coalesced receipt. The primary wake still held `[A]`. Actual claim rebuilt input from that wake and dropped B. **Expected behavior** Claim must retain every accepted live comment in order. Stop, changed source authority, deleted comments, and company boundaries must still apply. Already claimed wakes must retain their existing recovery source. **Steps to reproduce** 1. Queue a task run whose primary wake has an authoritative comment batch `[A]`. 2. Admit comment B through `createHeartbeatQueue.enqueueWakeup` and the real Postgres admission writer. 3. Confirm that admission accepts `[A, B]` and records B's coalesced receipt. 4. Claim the original queued run. Before this fix, the claimed input contains only A. **Paperclip version or commit** The untouched producer regression fails on `d7283e875772ca8c684dc86e471e6b48d543b7b1` with both process and native Runner/Codex adapter selection. Provider execution is disabled in the fixture. ## What Changed - Save the accepted comment batch in the unclaimed primary wake, run context, and coalesced receipt in one transaction. - Lock the primary wake before the run. Recheck the expected run status and same-company, same-agent and run bindings; apply the existing actor equality fence only to durable receipts before mutation. - Preserve incoming input as normal deferred work when the locked source changed. Preserve receipt identity and actor fields. - Preserve the original primary actor and trigger comment while updating its accepted batch. Each incoming receipt retains its actual actor. Keep claimed source payloads and legacy envelopes on their existing recovery paths. - Safely defer a same-named different agent without rewriting the original agent's run, wake or receipt. Missing and foreign-company targets remain rejected. Do not rebuild input from historical receipts. - Add actual process/native producer, real database rollback, Stop, deletion, foreign-source, and fallback controls. Update three direct writer fixtures for the required expected-status contract. ## Verification - The untouched process and native producer red on `d7283e875772ca8c684dc86e471e6b48d543b7b1` loses accepted comment B at actual claim. The initial incomplete mock is retained separately and excluded from behavioral proof. - Current process/native matrix: 22 passed, including mixed authors, system primary, same-named different agents, changed status, claimed sources, Stop, deletion and foreign-source boundaries. Queue, Postgres admission and use-case suites passed 234 cases before the source-preserving serializer follow-up; its 22 actual producer cases passed again afterward. - CI on the previous head found a real ordinary mixed-author workspace-copyback regression. The earlier 13-case selection omitted this test; that coverage gap is preserved. The first correction exposed a second real source-provenance/copyback failure. After preserving the primary trigger and actor while updating only its batch, the complete workspace recovery file passed all 188 cases, including ordinary mixed-author input and native copyback. - Real HTTP queued-comment consumption, discard/reorder and Stop boundaries: 101 passed before these review corrections. Existing chat and Slack consumers each passed their selected real writer case. - Previous-head CI also timed out in the CLI child-server startup fixture and a ZIP transfer case. The full untouched transfer file passed 17 cases locally. Startup remains unclassified; its intentionally reserved port is part of the fixture and is not diagnosed as the cause. No CI retry was issued. - An earlier queue run passed its 231 assertions but timed out during teardown; its preserved replay passed all 60 queue cases. This is distinct from the behavioral reds. - Exact `ab59d944895774d89f950248e356e93af00f4832` server type layers and production server/Runner build passed. The unchanged-default serializer file passed all 11 cases. Current-head CI and both substantive reviews remain required; previous grades and checks do not qualify this head. - No hosted task was retried or changed. Local tests do not establish any hosted deferred event's cause, provider execution, native acknowledgement, physical cleanup, or serving revision. ## Risks - Admission now performs bounded row reads and locks for each coalesced wake. It adds no idle scan or remote request. The existing task transaction owns the operation; lock order follows queued claim. - Claimed-wake recovery uses source identity. Its payload remains unchanged, and the existing actual native copyback tests cover that boundary. - A source that changes before the lock can defer the new input. This preserves work instead of consuming it into a stale run. No migration or historical backfill is required. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, shell execution, and test tools. The runtime does not report the context-window size. A separate OpenAI coordinator reviewed the source. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 22:09:58 -06:00
Devin FoleyandClaude Opus 5.5	ad5cd785b1	fix(wake-queue): keep promoted Retry runs scoped to their issue (#18876) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The wake queue starts the next run for a task when the task's current run releases its execution lock. > - A board Retry saves a receipt with only `{ issueId, taskRetry }`. This receipt has no saved wake context. > - When the release drain promotes that receipt, the promoted context gets `taskKey` from the payload, but it does not get `issueId` or `taskId`. > - The new run takes the task's execution lock, but the database saves it as a company-scoped run with no issue. > - Low-trust workspace preparation then has no issue reference. The run fails before provider work with "Low-trust execution issue is outside the active trust boundary." > - This pull request restores the issue scope of a payload-only receipt during promotion. > - The benefit is that a Retry on a low-trust task runs in its task boundary and does not fail during setup. ## Linked Issues or Issue Description **What happened?** A user clicked Retry on a low-trust task while a run was busy. When the busy run ended, the release drain promoted the saved Retry receipt. The promoted run had `issue_id = NULL`, `scope_kind = company`, and a context snapshot with only `taskKey`, `wakeReason: task_retry`, `wakeSource`, and `wakeTriggerDetail`. Workspace preparation failed with `low_trust_boundary_mismatch` (HTTP 422) in the setup phase. **Expected behavior** A promoted run for a task must keep that task's issue scope. This is true also when the deferred receipt has no saved wake context. Workspace preparation must then evaluate the task's own trust boundary. **Steps to reproduce** 1. Use a task with a low-trust execution policy whose assignee is an agent. 2. While a run on that task is busy, click Retry on the task. 3. Let the busy run end. The release drain promotes the deferred Retry receipt. 4. The promoted run is company-scoped with no `issueId`. It fails in setup with `low_trust_boundary_mismatch`. **Paperclip version or commit** `master` at 51d1d4ce07. **Agent adapter(s) involved** Not adapter-specific (core bug). ## What Changed - `enrichPromotedWakeContext` (`server/src/modules/wake-queue/domain/context.ts`) now restores `issueId` and `taskId` from the payload when the deferred context does not have them. This is the same rule that the original wake enrichment uses. The function does not overwrite a value that the queued context already has. - I added a domain test for the payload-only receipt and for the no-overwrite rule. - I added a Postgres adapter test (`adapters/promoted-retry-scope.test.ts`). It runs the real release drain with a Retry receipt that has no saved wake context. It checks that the promoted run is issue-scoped and holds the task's execution lock. The database trigger `set_heartbeat_run_scope_kind` derives `issue_id` and `scope_kind` from the restored context `issueId`. ## Verification - Red before the fix: both new tests fail on unchanged `master`. The adapter test shows the incident shape: `issueId: null`, `scopeKind: "company"`, and `taskKey` equal to the issue UUID. - Green after the fix: - `pnpm vitest run server/src/modules/wake-queue/` passes (10 files, 289 tests). - The related suites pass (11 files, 468 tests): `heartbeat/queue`, `heartbeat/run-control`, `heartbeat/run-lifecycle`, `deferred-execution-receipt-work`, `execution-lock-orphan-cleanup`, `heartbeat-comment-wake-batching`, `issue-stale-execution-lock-routes`, `chat-failed-run-retry-boundary`, `task-retry`, `execution-control-work`, and `workspace-recovery`. - Server typecheck (`node server/scripts/typecheck-layers.mjs`) passes. - I did not run a live low-trust Retry on a deployed stack. The adapter test uses the real release drain, the real database trigger, and the persisted receipt shape that `requestTaskRetry` writes. ## Risks - Low risk. The change only adds `issueId` and `taskId` when they are missing and the payload has them. Deferred wakes that already have a queued context do not change. - The release drain still promotes a board Retry receipt directly. It does not go through `resumeTaskRetry`. This PR does not change that path. A promoted Retry therefore does not get `forceFreshSession` or `taskRetryRequestId`. This behavior existed before this PR. It is a possible follow-up, not part of this fix. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (code search, shell, and test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 16:57:44 -08:00
4 people	ce332ecec0	fix(execution-admission): keep admitted project sets after settlement (#18871) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - #18746 added execution-admission storage: one admission row per attempt and one charge row per admitted project. > - Settlement deletes every charge row. The admission row keeps only the primary project and an opaque context revision. > - Thus, after settlement, the complete admitted project set is lost. > - A retained provider session still carries the references of its previous admission. Example: a task mentions A and B, the admission settles, and then a user removes B from the task text. The next admission must still count B, or start a fresh context. The primary project or the context token cannot supply B. > - This pull request keeps the complete admitted set on the admission row after settlement. Charge rows stay the only active capacity. > - The benefit is that later retained-session resolution can read the previous set. Settled history does not hold capacity or block project deletion. ## Linked Issues or Issue Description Refs #18746 (corrects its storage). Related: #2831, #14333. **Subsystem affected** Server execution admission (`server/src/modules/execution-admission/`) and the `packages/db` schema. **Problem or motivation** `settleAdmission` deletes all charges of an admission. No other storage records the complete project set. After settlement, the set cannot be recovered. Current task text is not a replacement, because it can change after admission. **Proposed solution** Add `execution_admissions.admitted_project_ids uuid[]`. It holds the sorted, distinct set. It is written once at admission and a trigger blocks changes. It has no project foreign key. `{}` is a known empty set. Null means unknown. Only settled rows can have null. Migration `0428` fills the set of unsettled `0426` rows from their charges, because those charges are still complete. Settled `0426` rows stay unknown. **Alternatives considered** - Keep charge rows after settlement with a `released_at` column and a partial active index. The project foreign key would then block project deletion permanently, or the key must be removed from active charges. One table would also hold two different facts. - A separate history table with one row per project. This is more rows, joins and constraints for data that is only read as a whole set by admission ID. - Encode the set in `context_revision`. That token is opaque and owned by the context owner. The database cannot validate it. - Use `heartbeat_runs.context_snapshot`. It is mutable and is per run, not per admission attempt. ## What Changed - Schema and migration `0428_retained_admitted_project_sets`: add `admitted_project_ids`, a check (null only when settled, no null elements, at most 100 entries) and an immutability trigger. The backfill fills unsettled rows from their charges and leaves settled rows null. The migration is safe to run again. Migration `0426` is not changed. - `recordAdmission` writes the set. A replay compares against the stored set. `settleAdmission` still deletes the whole charge set in one transaction and does not change the stored set. - Add `readAdmittedProjects` (company-scoped, by admission ID). It returns `known` with the set, or `unknown`. - Add a pure `retainedSessionProjects` rule: a retained session uses previous ∪ current. An unknown previous set returns `fresh_context_required`. No consumer uses it yet; C1b integrates retained sessions. - Update `doc/DATABASE.md` and `server/src/modules/README.md`. Not changed: no public cap writes, no claim or dispatch integration, no session integration, no provider activation, no `serialize` → cap 1 migration. Active counts still use only the indexed charge rows. ## Verification Red, on unchanged `master` code, with the new tests: - `server`: `npx vitest run src/modules/execution-admission/adapters/postgres.test.ts`. The new tests fail. With a temporary reader that uses the only storage `master` has (charges), the settled A+B admission reads `[]` instead of `[A, B]`. - `packages/db`: `npx vitest run src/execution-admission-migration.test.ts` fails, because no `0428` migration exists. Green, with this change: - `server`: `npx vitest run src/modules/execution-admission`: 27 passed. The new PostgreSQL test covers: A+B admission → settlement → set still readable, charges and counts zero, settlement replay keeps the set, a second attempt with a different set, primaryless and empty sets, company isolation, rollback, immutability, the check constraint, and deletion of a project that has only settled history (its ID stays in the set). It also checks that a project with an active charge still cannot be deleted. - `packages/db`: `npx vitest run src/execution-admission-migration.test.ts`: 2 passed. The upgrade test starts from the shipped `0426` schema with an unsettled A+B row, an unsettled empty row and a settled row. After the upgrade the sets are `[A, B]`, `{}` and null. A second run of `0428` changes nothing. - `pnpm --filter @paperclipai/db typecheck` and `check:migrations` passed. `node scripts/check-module-boundaries.mjs` passed. - Not run locally: the server layered typecheck. The local process was killed for memory (a 5.8 GiB limit shared with other jobs). CI runs it. This PR does not prove C2 claim races or C3 lifecycle authority. ## Risks - Migration: one `ADD COLUMN`, an `UPDATE` that only touches unsettled rows (none in production, because nothing writes admissions yet), a check and a trigger. It does not delete or rewrite data. - Settled rows created before `0428` stay unknown. A future consumer must start a fresh context for them. It must not treat them as empty or primary-only. - The stored set has no project foreign key. Company scope is checked at admission time, under the same project locks and the charge foreign keys. Size: about 70 production lines (schema 8, migration SQL 22, adapter 28, domain 15), 105 test lines and 24 doc lines. Generated metadata (`0428_snapshot.json`, `_journal.json`, removal of `0422_snapshot.json` by the standard pruning) is separate. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, embedded PostgreSQL tests). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: nickyleach <331803+nickyleach@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 15:31:10 -08:00
DottaandPaperclip	451d06e2fc	fix: resume interrupted Runner preparation and confirm failure report delivery (#18853) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns run recovery and task continuation after a restart. > - A shutdown can interrupt Runner preparation before any provider work starts. > - Workspace recovery can save the files but leave that task behind a legacy execution hold. > - Retry then accepts a request without starting a successor. Startup also leaves the task blocked. > - This pull request verifies the interrupted preparation, preserves its files, and admits one bounded continuation. > - It also keeps a dropped Sentry capture eligible for a later report. ## Linked Issues or Issue Description **What happened?** A hosted task remained blocked after a graceful server shutdown interrupted Runner preparation. Its remote workspace was recovered. Retry returned an accepted request, but no successor started. Startup did not resume it. The task also had no matching Sentry event. The recovery cause is verified. The exact cause of the missing historical Sentry event is unknown. A separate regression with the real Sentry SDK proves that processing backpressure can drop a capture while the task cache marks it as reported forever. **Expected behavior** Retry must continue eligible interrupted work from its saved workspace. Startup must schedule one bounded retry when durable evidence proves that no provider started. A later task read must be able to retry an unconfirmed Sentry report. **Steps to reproduce** 1. Interrupt Runner preparation during a server shutdown before native runtime selection. 2. Retain and recover its remote workspace under the legacy recovery hold. 3. Press Retry, or restart after file recovery has completed. 4. Observe that the task remains blocked without a successor. For the reporting case, hold the first Sentry event in its asynchronous enrichment step. Fill the SDK processing buffer. Capture a second task failure, release the first event, and read the second failed task again later. **Paperclip version or commit** Reproduced against source commit `4d31588dea`. The hosted incident served `6ef37398c8d799c5142a3a6590ecc70ad352b663`. **Deployment mode** Hosted Linux server with remote workspaces and external PostgreSQL. Local regressions use the source checkout on macOS arm64, Node 26.4.0, PostgreSQL 17.10, and the audited optional Sentry SDK 10.71.0. Related public changes: #18541 added task-state failure reports. #18678 retained remote exports before Retry. #18803 fenced remote workspace writers before continuation. The small backend-port fixture correction follows #18739. It is included here because the same existing setup defect blocked CI. ## What Changed - Verify pre-native shutdown interruption against the immutable adapter claim, controller ownership, run events, native records, stopped leases, and exact workspace recovery receipts. - Retire only the server's unchanged bootstrap failure block. Admit explicit Retry through the existing continuation path. - Discover completed workspace recovery at startup and schedule one retry under the existing failure budget. Let saved input own its continuation. - Apply the existing safe replacement dispatch checks to preparation retries. A later human block, competing execution, dependency, pause, approval, or budget gate must still prevent dispatch. - Keep execution evidence available through the bounded run projection. - Confirm Sentry task-report delivery from a successful SDK HTTP response. Keep unconfirmed captures pending for 60 seconds before a later task read can retry them. Both caches remain bounded and start no timer. - Document recovery and report delivery semantics. - Align existing handoff/budget assertions with the guarded preparation retry reason. Keep the owned-runtime fixture outside reserved proxy ports, with bounded allocation attempts. ## Verification - Red before the recovery fix: all three new Retry/startup cases failed because no successor was created. Green after the fix: all three passed with real saved workspace archives and PostgreSQL. - The pre-merge revision passes all 384 tests in the six affected recovery, run-control, run-state, retry, and dispatch suites. - The late-block regression failed before its repair. Its four focused recovery/dispatch cases then passed, including final dispatch cancellation. - Review found a second stop check that still rejected verified local and lease-free preparation. Both new cases failed before that repair. All 17 interrupted-preparation regressions passed on the pre-merge revision. - The Sentry regression failed with the real SDK before the fix. All 91 affected reporting tests then passed. The transport stays in memory. No test sends events to Sentry. - Full repository typecheck passed with `node scripts/typecheck-low-memory.mjs`. The full `pnpm build` and module boundary check passed. - CI found untyped delivery-hook parameters when the optional Sentry SDK was absent. Explicit callback types fix that build. The SDK-absent server build and all 91 SDK-present reporting tests pass. - Server typecheck passes with the SDK installed and the real Sentry SDK contract CI job passes. Greptile reviewed the pre-merge commit `103e175c44` at 5/5 with no open findings. - The local full-suite attempt initially overlapped the rebase and stopped at test-line selection. Its unchanged-head rerun passed the first 309-test chat shard. I stopped the old-head run before applying review fixes. Current-head CI owns the remaining full-suite verification. - CI exposed two old retry-reason expectations. The handoff and retry-budget checks pass with the new guarded reason. A separate existing process-adoption fixture chose port 42953 in the reserved 42000–42999 proxy range. The fixture now rejects reserved ports, following #18739. Its real process/adoption/termination test passes locally. - The final Runner package job [initially failed](https://github.com/paperclipai/paperclip/actions/runs/41718507919/job/126205912033) one unchanged Codex credential case after it could not acquire its port quorum, then timed out. The complete unchanged 42-case credential suite passes locally. One unchanged CI retry passed; the original competing port owner was not identified. No Runner implementation or credential test was changed. - Required CI passes on `103e175c4490e1bdf804c64a8635b0e4568545ae`, including `ci / verify` and `ci / e2e`. That revision has Greptile 5/5 with no open findings. The optional Paperclip Review check on that earlier head remained queued, waiting for its assigned agent. - Merged current `master` without replacing published commits. The retry gate retains both shutdown interruption checks and the new workspace preparation timeout policy. All 311 tests in the five combined recovery, retry scheduling, completion, and native startup suites passed after the conflict resolution. Server typecheck passed. Final head `005d380b00d45a2393c4a2a190eab27611168d3c` includes the next conflict-free master update. All 17 interrupted-preparation regressions passed again on that final head. Greptile reviewed that exact head at 5/5 with no new findings. Required CI passes on that exact head, including `ci / verify`, `ci / e2e`, typecheck, build, the full test shards, and the real Sentry SDK isolation check. The optional Paperclip Review remains queued, waiting for its assigned agent; it is not a required branch check. - On the final head, six test jobs were cancelled after their spot runners received shutdown signals at the same time. Another server shard passed 2,419 tests, but its unchanged 103-record cleanup backlog case exceeded the 15-second limit. The unchanged full cleanup suite then passed all 29 tests locally. One unchanged retry of only the seven unsuccessful jobs passed. The [original cancellation and timeout logs](https://github.com/paperclipai/paperclip/actions/runs/41724898038/attempts/1) remain in the workflow history. No cleanup implementation or test was changed. - The signed-in hosted UI reproduced the failed Retry before the fix. The fixed path is covered through the real HTTP Retry route and assembled heartbeat/recovery services. This PR has not been deployed to the hosted incident. A live repaired Retry remains unverified until rollout. ## Risks - Automatic continuation must not repeat provider work. It requires positive preparation evidence and rejects retained provider, process-launch, coordinator, result, live-controller, unsafe lease, or mismatched workspace evidence. - Repeated recovery calls must create one successor. Tests cover concurrent requests, repeated startup visits, pending Retry, retry exhaustion, and changed task ownership. - Sentry delivery state stays local to the process. A restart or cache eviction can report a failure again. An unconfirmed event needs a later task read to retry. - No schema or API contract changes are required. ## Model Used OpenAI GPT-6 through Codex. This session does not expose a more specific deployment ID or context window size. Reasoning, code editing, test execution, and browser diagnostics assisted the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 16:10:49 -06:00
DottaandPaperclip	1f55073f96	fix(cursor-cloud): support API-only controllers on managed stacks (#18310) ... ## Thinking Path > - Paperclip helps people manage AI agents at work. > - Cursor Cloud hosts execution behind its API. > - Product setup still expected a Paperclip Computer. > - Managed-only tenants therefore rejected valid setup. > - This admits the validated API-owned controller. > - Authority, Stop proof, budgets and cleanup remain enforced. ## Linked Issues or Issue Description Refs #18068 and #18069; both dependencies have merged. **What happened?** Managed-only setup still requested a Computer. Live tests exposed missing Cursor Stop proof and duplicate replay. **Expected behavior** Use Cursor’s API controller. Verify provider/controller termination and replay each log identity once. **Steps to reproduce** Enable native Runner/managed-only. Configure Cursor model/repository/key without a Computer, then Test, finish setup, run and Stop. ## What Changed - Derive API-only placement from validated native Cursor configuration; reject placement/command/workspace/home overrides and keep Kubernetes-only execution closed. - Skip Computers/checkout/workspace realization. Keep scoped tools, completion, private staging, budgets, leases and recovery. Persist placement; reject contradictions. - Align Cloud cards, Test, Save and Finish setup; hold unready drafts. Legacy keeps its Computer, credential and policy checks. - Verify Stop from exact API ownership, audited intent, provider-terminal proof, no Computer/process ownership and controller retirement. Recheck immediate/deferred retirement; wake existing recovery on durable evidence. - Clear warnings when the actual stopped target later has authoritative confirmation. Keep missing/other-run negatives and the existing predicate. - Publish safe status refresh after full-proof ACK. Retain bounded native/physical cancellation history; add no poller or weaker proof path. ## Verification Core merged as `3910352c923a`; Product merged as `27d35314be8e`. Public branch `5f86eb66b081` has tree `67344151d303`. Its exact-head Greptile summary is 5/5; the fresh audit found zero unresolved threads and complete pagination. The controlled workspace retry and required summary passed. All 59 current-head contexts are terminal and clean. Prior green-36 evidence keeps its identity. [Current CI](https://github.com/paperclipai/paperclip/actions/runs/41680511731) first failed workspaces-a 1/2: the CLI company-import/export beforeAll 30-second runtime-info/child-PID/HTTP readiness wait expired. All 431 UI files/4,714 tests and 285 other CLI tests passed (one skip). The cause remains UNKNOWN. Child output records startup recovery complete, but buffered CI prefixes do not establish its relation to the waiter deadline; runtime-file/PID/last-health state was not captured. Both unchanged local probes were RED. The second observed dependency preflight still pending at the deadline; observer overhead was not quantified. Neither proves the CI cause. After all functional jobs finished, only the queued metadata summary was cancelled; the other 48 records stayed unchanged. Attempt 1 retains 46 successes, the original failure, one skip and the cancelled summary. One same-head debug retry was admitted at 18:42 UTC on November 3. Its workspace job passed at 19:23 UTC on the same checkout: UI 431 files/4,714 tests and CLI 33 files/286 tests, including company import/export. Test caching was disabled. The new `ci / verify` summary passed; workflow attempt 2 completed SUCCESS. Required `ci / e2e` also passed. The final check at 19:53 UTC on November 3 found all 59 current-head contexts terminal and clean: zero pending and zero failing. The 47 inherited completed/skipped rows are not new executions. No code, assertion or timeout changed; success does not explain the original failure or erase the local REDs. CI checkout is `1ea9d8608ced`/tree `78cb1ec13d59` (parents `2e37c73c05fe`, 5f86); both workflows use 2e37. Its 20 extra paths add low-memory build/Rust task-memory. Scripts cohere; later matrix changes only labels, with no newer debug-cache steps. Static audit found no blocker. Supported API arguments omit the memory flag: no default monitor/instruction changes. Flagged external launchers can change Cursor instructions/digests and remain unqualified. The merge added 41 master-identical paths, retained the green-36 inbox hunk and found UI fixtures matching upstream. Startup/backup, recovery projection, sleep diagnostics and binary backup change production. Static audit found eight Stop/ACK owner blobs and 37 sleep-work SQL templates unchanged, with sleep fences/single-flight accounting intact. Exact precommit tree: 416/416 across eight server files, binary backup 1/1, server types PASS; repeated scopes are not summed. The deferred-retirement P1 reproduced three real-DB/owned-process failures; missing proof stayed unverified. Recovery passed 47 process cases/33 proof negatives. Warning repair binds actual targets to same-run proof. Validation passed 729 scoped UI cases, a separate 359-case thread run, 111 backend cases (84 CAS, 6 assembled, 21 activity/privacy), types/tokens. Proof validators/predicate are unchanged; no new poller or increased timeout. The d8d3 attempt stays FAILED: 45 passes, two workspaces-a failures, derived verify failure and one skip. Both shards shared three failures: two successful mocks omitted the account guard’s required userId. The two-line fixture repair changed no guard, assertion or timeout. Three failures reproduced; 54 access/auth tests then passed, including account-change/denial negatives, plus UI types/tokens. The 69285 attempt stays FAILED: 49 jobs, 46 passed, serialized shard 9 failed, its verify aggregate failed and one selection job skipped. The shard passed 575/576 cases. Its aborted-snapshot case expected nine computes but saw eight: a final HTTP 200 coalesced while seven ordinary reads had not proved the aborted compute settled. Earlier CI and the unmodified local baseline passed with a miss; no local RED is claimed. The repair awaits the snapshot, allows exactly one extra GET if coalesced, then requires a miss and retains nine computes/all HTTP429 assertions. No production code, hook, loop, sleep or timeout changed. Six inbox and seven cache/admission cases passed locally, plus server types; 30 unrelated cases were filtered. Green 36 passed 576/576 without taking the extra-GET branch; local runs also missed that schedule. Current CI event merge 1ea9 now passed 576/576 and naturally exercised the former failure: coalesced (count 2, 1ms), then exactly one fresh miss (count 1, 16ms), with strict nine-compute/HTTP429 assertions intact. No instrumentation or retry was added. Earlier compatibility passed scoped remote/inbox, routine, API-placement and UI tests, plus types/migrations/snapshots/maps/tokens. The 89-path adoption restored fleet routing without workflow overrides or added skips. Each result retains its source identity. Live qualification remains actual `9af1a5de42a2`: unchanged Artifact grader PASSED 17/17 and active-Stop grader PASSED 21/21. All three provider test resources were deleted canonically with GET404; post-delete download returned the exact 27 bytes. Histories stayed stable (Artifact 121 and Stop 67 canonical identities), with 21 company runs, zero active work and no accounting/cancellation/reservation debt or integrity findings. Saved settings reopened; no-prompt Test/Linux admission passed with `claude-sonnet-4-5`. No new Save is claimed. The first Stop sampling case stays FAILED on a 10.373-second gap. Unchanged retry kept all 24 samples: 35.076 seconds quiet, maximum gap 4.584 seconds, plus a separate CANCELLED/artifact-absent read after 180 seconds. Initial grading rejected locator spelling before functional checks; metadata changed only after same-file/inode/byte proof, with raw evidence/oracles unchanged. Original Question/Plan/reconnect and historical deleted-artifact results keep their identities. Git trees verified 11 history-only conflicts and six earlier mainline paths. Hosted 9af to the public BRANCH differs in 222 paths; 13,500 other mode/blob leaves match. Runner runtime implementations and six observer/driver pins match 9af on that branch only; inherited `drivers/acpx/runtime-host.test.ts` differs. No all-Runner/src, CI/latest-master runtime or image equality is claimed. Later static audits through `8ab89f41e499` found no concrete API-only blocker. That base adds completed-history sleep proofs and an MCP cleanup migration; protected Stop/ACK owners are unchanged. These later changes are outside this CI and have no live qualification here. Host duplicate-card and later recovery/UI repairs retain separate proof, not deployed 9af identity. Direct-key coverage requires absent aiConnection; API-only placement skips inherited remote-workspace disk-full paths, and config-read gates match their parent. Other retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; title/startup/custody/Stop/deletion unknown causes and bounded recoveries; superseded warning/50 duplicate identities; pre-repair ACK/Core CI failures. Fixture RED/GREEN passed 7/7 browser and 121/121 shutdown tests with a failing broken-unsubscribe control; stable WSP wrapper passed 128 after physical teardown failure. No full local-suite pass is inferred. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and behavior checked live; unpriced provider dollar cost stays UNKNOWN. The placement exception applies only to validated native Cursor API execution. Other local work remains denied under managed-only policy. Missing/stale Stop or cleanup proof stays pending; damaged archives remain failed. Explicit Legacy keeps its existing policy owners. ## Model Used OpenAI Codex/GPT-6 family: tools, execution, browser checks, delegated review. API ID/context window unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; retained failures are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 13:55:39 -06:00
4 people	fef86e32e9	feat(execution-admission): add project concurrency policy and admission storage (#18746) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Projects group tasks. Agents run tasks as heartbeat runs. > - Today, workspace mode and shared-workspace flags decide whether tasks run at the same time. A project cannot set its own limit on concurrent task executions. > - A correct project limit must count every project that a task uses, including @-mentioned projects. It must also keep the count when a project is archived or deleted, and when a run is admitted more than one time. > - This change cannot go into one pull request with claims and settlement. Thus this pull request adds only the policy semantics and the durable accounting storage that later claims will use. > - This pull request adds a new `execution-admission` module, three tables and the deletion guards. No route writes a cap and no claim path enforces it. > - The benefit is a tested storage contract (constraints, idempotency, rollback, indexed counts, lifecycle preservation) before any behavior changes. ## Linked Issues or Issue Description Refs #2831, Refs #14333 (related agent-level and adapter-level limits; not duplicates). **Subsystem affected** Server execution admission (new module `server/src/modules/execution-admission/`) and the `packages/db` schema. **Problem or motivation** A project cannot limit how many of its task executions are active. The current shared-workspace flags mix resource safety with capacity. A future project limit needs storage that records each admission attempt and every project that the attempt charges. Run IDs cannot be the identity, because a run can be released before dispatch and admitted again. **Proposed solution** Add a separate policy table that keeps "never configured" apart from "explicit no limit". Add admission epochs with one charge row per distinct project. Settlement deletes the charge set in one transaction. Later pull requests integrate claims, settlement, waits and the public control. **Alternatives considered** Columns on `projects` would leak an unenforced cap through every project reader. A project column or a `uuid[]` on `heartbeat_runs` cannot enforce per-project uniqueness or company scope. `budget_reservations` has no project set. A counter would be a second source of truth. ## What Changed - Added `project_task_execution_policies`. No row means never configured. A null maximum means an explicit "no limit". A check accepts only positive maximums. A composite foreign key keeps the project and company together. The row is deleted with its project. - Added `execution_admissions`. Each row is one admission attempt of a run: `(run_id, attempt)` is unique, and a partial unique index allows one unsettled attempt per run. It keeps the context revision, the nullable primary project (provenance, no foreign key) and the settlement identity. A check sets `settlement_id` and `settled_at` together. - Added `execution_admission_project_charges`. It has one row per distinct admitted project, unlimited projects included. Composite foreign keys keep the admission and the project in one company. The project foreign key has no cascade, so a charge cannot disappear with its project. `(company_id, project_id)` is indexed for active counts. - Added migration `0426_absent_guardian.sql` (renumbered from `0423`, then `0425`, as master added `0423`–`0425`). The generated SQL was reordered so the unique indexes exist before the composite foreign keys that need them. Snapshots `0419` and `0420` were pruned to keep five snapshots. - Added the `execution-admission` module (domain → application → adapters): - Domain: policy patch parsing and application (omitted keeps, null clears, partial updates keep other fields, malformed values are rejected), canonical project-set preparation (primary ∪ references, deduplicated, ascending, bounded with an explicit `too_many_projects` result), and the settlement replay decision. - Application: internal `readTaskExecutionPolicy` and `updateTaskExecutionPolicy` commands. No route, tool, CLI or import calls them. - Adapter: the project-set lock (`FOR NO KEY UPDATE`, ascending, one statement), the policy store, and the storage transitions `recordAdmission`, `settleAdmission` and `countActiveProjectCharges`. These transitions are not exported from the module entry point. Claims (later) must add the capacity check and the lifecycle evidence. - Project deletion returns `409 project_execution_admission_pending` while the project has a charge. Agent deletion returns `409 agent_execution_admission_pending` while its runs have an unsettled admission, and removes settled history. Company deletion calls a new module participant before it deletes runs and projects. - Documented the tables, the lifecycle rules and the proposed global lock order in `doc/DATABASE.md`. Added module ownership to `server/src/modules/README.md`. ## Verification - `cd server && npx vitest run src/modules/execution-admission` → 2 files, 26 tests pass (the Greptile finding on an empty primary ID added an assertion to an existing test). The adapter tests use embedded PostgreSQL and cover: - policy semantics, the maximum check and the company foreign key; - charges for every distinct project, a primaryless set and an empty set (no rows); - replay of one admission ID, conflict on a changed record, refusal of a second live attempt and the partial unique index; - settlement of the whole set once, replay, a different settlement, then attempt 2 for the same run; - company isolation of projects, runs and admission IDs, the duplicate-charge key, and rollback of an aborted transaction; - active counts per project, and an `EXPLAIN` that uses `execution_admission_project_charges_project_idx`; - archive keeps charges, `projectService.remove` returns 409, a direct project delete fails on the foreign key, and delete succeeds after settlement; - `deleteAgentDependencies` returns 409 while unsettled, and `deleteCompany` removes only that company's admissions. - `cd packages/db && npx vitest run src/execution-admission-migration.test.ts` → pass. It removes the new tables and receipt, seeds a legacy project and run, applies pending migrations, and compares the existing rows. - `pnpm --filter @paperclipai/db check:migrations` → pass. Five snapshots remain (`0421`, `0422`, `0424`, `0425`, `0426`; `0423` is SQL-only on master). - `node scripts/check-module-boundaries.mjs` → pass. - `packages/db` `tsc --noEmit` → pass. A scoped `tsc` of the module entry points and domain tests → pass. - Existing tests for the changed deletion services → 6 files, 161 tests pass (`resource-lifecycle-events`, `prompt-injection-prevention`, `agent-lifecycle/lifecycle`, `case-deletion`, `project-workspace-root-provenance`, `companies-service`). - Not run locally: the full `server` typecheck. `tsc` and `tsgo` were stopped by the out-of-memory killer on the 7 GB machine (exit 137, also single-threaded). The full `pnpm -r typecheck`, `pnpm test:run` and `pnpm build` were not run locally; CI ran them on head `520b69d553`: all gates pass, including Typecheck, Build and the 12 server test shards. After merging master (`1c9ca5829f`) into head `c088729361`, the migration check, migration upgrade test, module tests, module-boundary check and `packages/db` typecheck were rerun locally and pass; CI is rerunning the full suite. ## Risks - Migration risk is low. It creates three empty tables and indexes. It does not change existing tables or rows. - Behavior changes only when admission rows exist. Nothing writes them yet, so project, agent and company deletion behave as before. - Later claim work must take the project-set lock in the documented order. A claim that takes project locks after issue locks could deadlock with project deletion. - Settlement deletes charge rows, so a settled admission keeps its primary project and context revision but not its full project set. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, embedded PostgreSQL tests). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: nickyleach <331803+nickyleach@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 11:35:28 -08:00
DottaandPaperclip	ccc63ea458	fix(server): prove completed history before idle sleep (#18646) ... Reviewed and verified at 592bce69c6. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 12:37:25 -06:00
DottaandPaperclip	86b365f992	feat(auth): add independent agent configuration read permission (#18766) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents need to inspect configuration and instructions when they set up teammates. > - Configuration reads currently depend on configure or suggestion grants. > - Legacy standard agents did not receive those newer grants. Some self-read routes also lose the target agent identity. > - This pull request adds an independent read grant and applies only that grant to eligible existing agents. > - The benefit is that agents can inspect and verify configuration without new write authority. ## Linked Issues or Issue Description Refs #15283. Related earlier proposals: #8064 and #12050. Those proposals target older authorization code; this change uses the current configure/suggest model and adds a read-only rollout for existing agents. Related bounded peer metadata proposal: #14160. **What existing behavior does this improve?** Agent configuration, configured skill, instruction, and revision reads through the existing APIs and native instruction tools. **Current behavior** An agent needs `agents:configure` or `agents:suggest-changes` to inspect a peer. Older standard agents can hire a teammate but cannot verify the saved instructions. Configuration and revision routes also reject self-reads because they authorize a company resource instead of the target agent. **Proposed behavior** Add `agents:read` for redacted configuration and instruction reads. Keep `agents:configure` as read/write and preserve suggestion-grant compatibility. Standard new agents receive the read grant. A one-time migration grants only reads to eligible existing standard agents. Individual routes authorize the target agent and grant scope. **Reason and benefit** Read access can be granted without configuration changes, rollback, or environment probes. Legacy agents can verify teammates without receiving new write permissions. ## What Changed - Add `agents:read` to the shared permission contract and standard-agent defaults. - Accept read, configure, or suggestion grants for configuration reads, with target scope and the responsible-user ceiling. - Fix target-aware self and scoped configuration, skill, and revision reads. - Add an audited data migration. Skip low-trust, built-in, inactive, pending, terminated, setup, and scoped configuration/suggestion cases. Preserve existing read scopes and later revocation. - Update the specifications, default-permission documentation, and agent access map. ## Verification - Focused acceptance: 276 tests passed across seven suites on commit `a37314df88`: authorization, real-key HTTP access, route authorization, canonical instructions, default grants, migration replay, and lifecycle configuration. - The HTTP journey uses a real hashed agent API key and a disposable PostgreSQL database. It verifies redaction, scoped reads, denied edits and rollback, cross-company isolation, and both agent and responsible-user revocation. - New read-grant regressions failed before the authorization change. A realistic paused/pausing/resuming migration fixture also failed before the lifecycle filter correction and passed afterward. Setup-held paused agents stay excluded. - `pnpm -r typecheck` and `pnpm build` passed on the rebased branch; the final lifecycle migration correction also passed the database package typecheck and build. The latest commit changes only a test assertion. - `pnpm --filter @paperclipai/db check:migrations`, `node scripts/check-maps.mjs`, and `git diff --check` passed. - Full current-head [CI run](https://github.com/paperclipai/paperclip/actions/runs/41641531426) passed: typecheck, build, all general and serialized test lanes, Runner checks, all eight browser shards, canary dry run, policy, and aggregate gates. The local full-suite attempt was stopped after exposing the outdated cross-company 403 assertion; interrupted local full-suite attempts are not counted as passes. ## Risks The migration deliberately expands redacted configuration and instruction visibility for eligible existing standard agents in the same company. An absent older write grant does not deny the new read capability. It adds no write grant. Scoped configure/suggestion grants and all existing read-grant scopes stay intact. Administrators can remove the independent read grant after the one-time rollout; ordinary updates and startup do not restore it. Configure and suggestion grants still imply reads within their own scopes. Cross-company configuration/history reads now return 404 through the common target-aware helper. No schema change or credential-value disclosure is added. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The host does not expose a more specific runtime model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 07:26:27 -06:00
DottaandPaperclip	16643a6e03	fix(runner): recover pending Stop cleanup after controller restart (#18682) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns execution, Stop delivery, and usage settlement. > - A Stop can remain pending when the controller restarts. > - Recovery must reach the existing session without starting more paid work. > - Startup and budget fences currently block that cleanup, and interrupted runs leave the restart selection. > - This pull request gives exact audited Stop cleanup a shared authority check and records its real acknowledgement. > - The benefit is safe cleanup while usage debt remains honest. ## Linked Issues or Issue Description **What happened?** A pending native run-only Stop can retain an unfinished execution owner after controller loss. Its startup cancellation fence prevents recovery admission. If teardown marks the run interrupted and clears its task lock, restart selection also omits it. A recovered session can receive cancellation without the durable acknowledgement that the normal Stop request would save. **Expected behavior** Recover only the existing remote session for the audited pending Stop. Keep fresh provider work blocked, including during budget pause. Keep incomplete usage pending until a real complete receipt arrives. **Steps to reproduce** 1. Start a native run with a held reservation and an authenticated remote checkpoint. 2. Persist a run-only Stop and its startup and execution cancellation fences before acknowledgement. 3. Lose the controller. Leave the coordinator unfinished, mark the run interrupted, and clear its task lock. 4. Attempt restart recovery and observe the missing claim or cancellation admission rejection. Related work: #14733 binds caller Stop requests. #18658 handles plugin startup cleanup obligations. #18035 proposes broader accounting changes. This change repairs existing-session Stop recovery and preserves current accounting guards. ## What Changed - Add one shared check for the exact run, company, issue, pending Stop, intent audit, live recovery lease, generation, and remote reattachment disposition. - Include eligible interrupted Stop owners in restart selection. Keep generic interruptions and another task execution owner fenced. - Reuse only the original held reservation for cleanup. Keep budget pause effective for fresh, bootstrap, dead-runner, and unproven recovery work. - Apply the same cleanup authority at heartbeat runtime selection and native entry. Skip attachment staging and warm-owner retirement only for that exact cleanup. Revalidate at session publication and fail closed if the proof changed. - Send the existing Stop through the audited cancellation producer before turn submission. Close the coordinator only after acknowledgement, including system-owned run-only Stops. - Give verified cleanup a transport purpose that retains pending semantic inputs without replay, dispatch, or a semantic reply. Add an explicit Core retain-only intake mode; preserve authenticated identity, schema, digest, sequence and replay checks. Ordinary execution still requires its dispatcher. - Add PostgreSQL, native protocol, and startup-fence regressions. Update the execution contract and recovery maps. ## Verification - The combined Stop regression fails on unchanged production at the cancellation reservation guard. Interrupted restart selection also fails before the repair while its running sibling passes. - An assembled `heartbeatService.recoverNativeRunsAfterRestart` regression exposes the earlier runtime-selection fence with the rest of the repair present. The fixed path reaches selector/claim, runtime selection, original held-budget reuse during pause, dispatch, real native protocol, audited cancellation, and heartbeat settlement. It starts no provider turn, acquires no new sandbox, and does not reclaim the task lock. - Fail-if-called attachment staging and turn boundaries remain untouched. A retained warm-owner regression prevents retirement. Invalidation at session publication fails closed. Actual backend/driver constructor tests verify cleanup-purpose forwarding, absent default options, and denial under foreign ownership. - Missing usage retains accounting debt and the hold. A later complete receipt records one 25-cent charge and settles that same hold. Concurrent duplicate cleanup is denied. Both user-owned and system-owned Stops pass. - Real transport tests: 9 pass on Node 24.22.1. They preserve ordinary saved-tool and completion recovery, prove the authority-denial race before cancellation, and qualify cleanup with pending ordinary/completion inputs and either handler ordering. - Real authenticated Core intake: 11 pass. Callback-present and callback-absent retain-only cases persist and acknowledge unanswered input without dispatch, recovery or result. Default missing-dispatcher rejection and digest-integrity/suspension guards remain effective. Omitting the callback alone first failed by disconnecting; the explicit retain-only mode fixes that defect. - Repository `pnpm -r typecheck`, the server/runner-vendor build, map checks, and module boundary checks passed locally before integrating the latest retained-startup change. That integration passes 11 native startup/cleanup compatibility cases and 3 assembled recovery/accounting cases. The final broad affected-suite attempt was stopped without a verdict; it is incomplete. Full repository tests and build are delegated to current-head GitHub CI. - Review exposed an additional real-process close failure with unanswered input. The corrected path requires a fresh exact provider-stop receipt confirming exit, followed by durable event drain and exact runner suspension. All 11 focused transport cases pass, including missing/false exit-proof negatives and ordinary recovery siblings. Runner TypeScript and map validation pass after this correction. - The assembled test controls provider, workspace-transfer and GitHub bootstrap I/O. It does not qualify hosted credentials, physical provider termination, workspace preservation, or historical receipt recovery. Local socket-permission setup failures are separate from semantic regression results. No hosted replay has run on this repair. - CI exposed an ordinary factory-call compatibility regression and an obsolete authority-read test fixture. The repair preserves single-argument ordinary calls and updates the race fixture without changing its Stop assertions. The complete native executor suite now passes all 1,049 tests; the ordinary factory boundary and all five Stop-accounting cases pass. A separate local environment-default case twice exceeded its five-second wait while other work ran; its prior-head CI case passed. The same unchanged case passed an isolated diagnostic without changing its assertion or timeout; the precise contention source remains unproven. - The final fixture edit initially used the wrong overloaded promise type. CI rejected it; the annotation-only correction passes the exact `pnpm run typecheck:build-gaps` entry point locally, including server service tests and remaining workspaces. No runtime assertion changed. - Final exact head `007fd4d04a2d6338c6dc073e97ab470a277fc3d0`: required `ci / verify` and `ci / e2e`, build, typecheck and test groups pass. Greptile is 5/5 with both findings resolved. Three expected checks are skipped; the optional native check is neutral, while its final current-head comment gives 5/5 from source inspection without a checkout or test execution. The first server-shard attempt lost its self-hosted runner connection; one scoped rerun passed. Its original annotation and unavailable-log evidence are retained. ## Risks - This changes execution authority. Cleanup requires an audited, exactly bound pending Stop, an unfinished coordinator, a live remote reattachment claim, and the original held reservation. It never authorizes a replacement provider turn. - Missing checkpoint, original held reservation, termination, or usage evidence can keep cleanup or accounting pending. The repair does not manufacture acknowledgement, replies, receipts, or zero cost. - Retain-only intake is an internal trusted constructor option. It changes no agent tool permission. Ordinary callback requirements and semantic dispatch remain unchanged. - No schema, public API, provider profile, or deployment setting changes are required. ## Model Used OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source inspection, code execution, and regression testing. The context-window capacity is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run affected tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 18:22:40 -06:00
commitperclip[bot]	fd0d61b8e3	fix: Compare native finaliser lease expiry at millisecond precision when releasing an expired owner (#17214) ... > [!NOTE] > Opened by Prospector's automated issue-fix pipeline. A maintainer reviews and decides; nothing here merges on its own. ## Thinking Path - Issue #16715 reports a defect in paperclipai/paperclip. - Root cause, as the fixing agent read it: Lease renewal writes 'lease_expires_at' from the database clock ('now() + interval'), so the stored value has microseconds. 'agentExecutionsHaveStopped' reads that value into a millisecond JS 'Date' and then releases the row with an exact 'eq' on it. The update never matches, so the stop check returns false on every tick and the agent stays in 'pausing'. - This pull request: Compare native finaliser lease expiry at millisecond precision when releasing an expired owner ## What Changed - `server/src/services/agent-execution-stop.ts`: The release update now truncates the stored lease expiry to milliseconds before comparing it with the value it read, using the same 'date_trunc('milliseconds', ...) = ...::timestamptz' pattern as 'chat-channels/activity.ts'. The other guards stay the same: owner, generation, phase and the null result. This was the only exact-match comparison on a lease expiry in 'server/src'. ## Verification - Reproducing test(s), added in this PR: `server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts` - They fail on the base commit and pass with the fix, each twice in a fresh sandbox. - 3 independent attempts wrote 3 reproductions; this fix passes every one of them. - The repository's compile command passes with the fix applied. - The full test suite passes except for the 10 file(s) that already fail on the base commit. ## Risks - Written by an automated pipeline, not a person: review it as you would an outside contribution. - Risk tier of the touched paths: 2. - The scope reviewer found it also changes, beyond the report: agentExecutionsHaveStopped for an expired finaliser row whose lease_expires_at is already millisecond-precise (written from a JS Date) - The scope reviewer found it also changes, beyond the report: Finaliser row whose lease was renewed or taken over between the select and the update - The scope reviewer found it also changes, beyond the report: Rows with null leaseExpiresAt ## Model Used - codex:gpt-6.1-sol, opus, sonnet (Anthropic Claude, via Claude Code; OpenAI, via the Codex CLI), writing and checking the change in an isolated sandbox. - opus (Anthropic Claude, via Claude Code), reviewing it independently. ## Linked Issues Fixes #16715 Other pull requests that name #16715: - #17204 (open): fix(heartbeat): stop loading full run rows and stale pids when pausing an agent - #16745 (open): fix(agent-lifecycle): release an expired native finalizer lease at millisecond precision ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with 'Fixes: #' / 'Closes #' / 'Refs #' OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub '#NNN' / 'github.com/paperclipai/paperclip' URLs) - [ ] My branch name describes the change (e.g. 'docs/...', 'fix/...') and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge <!-- prospector:issue-fix v1 issue=16715 base=c7ea5c0216b8 report=1a4e13ecd7c6cdb6 --> Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>	2026-11-02 15:01:52 -08:00
DottaandPaperclip	a92f616cd0	fix(db): attribute bounded captures to execution owners (#18634) ... ## Thinking Path > - Paperclip manages agent work through shared server and database services. > - High query volume can spend main-thread time on SQL construction and result decoding. > - The existing bounded response capture counts rows and bytes, but it cannot name the operation that executed a query. > - Bundled callers can be missing, and measuring every parsed JSON value adds work to the diagnostic. > - This pull request carries a semantic execution owner into each measured query and bounds response sizing. > - Operators can rank query volume by owner before choosing a performance fix. ## Linked Issues or Issue Description Refs #17504, #17522, #17537, #17564. **What existing behavior does this improve?** The opt-in `PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW` diagnostic. **Subsystem affected** Database diagnostics and existing server execution boundaries. **Current behavior** The capture groups by table, operation, and builder caller. It cannot distinguish execution owners that reuse a prepared query. Bundled deployments need source maps for caller locations. The capture visits every returned value and serializes parsed JSON again to estimate its size. **Proposed behavior** Version 2 captures use fixed semantic owners for HTTP requests, delivery queues, execution phases, and budget work. They count completed calls, returned rows, errors, and completion wall time. They sample successful results 1, 17, 33, and so on, with fixed cell and string/byte limits. Complex values remain explicitly unmeasured. Semantic ownership works without source maps. **Reason and benefit** A short census can identify which operations issue the most queries or return the most rows. It preserves each query's execution owner through pooled connections, concurrent operations, and prepared-query reuse. It reduces diagnostic work without claiming per-query CPU attribution. **Breaking changes** Opt-in capture output moves from `entry.v1`/`summary.v1` to `entry.v2`/`summary.v2`. Sampled primitive byte lower bounds replace all-result byte estimates. Consumers must use the new fields and must not extrapolate them into a total. There is no database migration or application API change. ## What Changed - Added `withDatabaseOperation(owner, action)` to the DB package. The server owns semantic labels. Queries retain execution ownership in their completion closure. - Added fixed owners at existing HTTP, delivery, budget, and execution timing boundaries. Delivery work replaces the context of the producer that woke it. - Added complete call/row/error counters and completion wall durations within the existing bounded window. Durations include waits and are not CPU measurements. - Sampled and bounded primitive-value sizing. Complex and oversized values remain unmeasured. Known owners skip repeated stack extraction. - Preserved the deadline, group limit, paced output, query outcomes, and existing coverage exclusions. Updated the operator documentation. - Added real PostgreSQL owner-isolation and prepared-query tests, server scope tests, and plain bundled-runtime proof. - Moved projection-test and benchmark full-response sizing into an explicit test helper. Existing payload-size thresholds remain unchanged; these synthetic checks do not use the production sampler. ## Verification - Final-commit DB capture suites: 43 tests passed in 13.86 seconds against the actual PostgreSQL driver, including concurrent owners, prepared reuse, transaction failure, a dedicated client, sampling, and a query crossing the deadline. DB typecheck passed. - Server attribution and existing delivery/execution/budget suites: 53 tests passed. Final concurrent HTTP GET/JSON POST scope refinement: 5 tests passed. - Runtime bundle suite: 8 existing tests passed. The new no-source-map owner test passed after its fixture was corrected to use a driver thenable. An ordinary native Promise can bypass an overridden `.then`; the real PostgreSQL tests separately cover driver integration. - Observer-only experiment: three alternating pairs per case on Node 24.22.2, macOS arm64, 3,000 calls per run after warm-up. For an already-materialized synthetic JSON result, median capture CPU fell from 82.37 to 1.87 microseconds/call. For a 100-row primitive result, it fell from 8.10 to 4.40 microseconds/call. Every result retained identity and all calls were counted. The new diagnostic intentionally measures less byte data; these are observer costs, not database latency or a hosted application speedup. Disabled-case timings were below one microsecond/call and too small for a performance conclusion. - Final lifecycle regression: 28 unit tests and DB typecheck passed. Closing the last capture disables and replaces owner storage; overlapping captures retain it. - Final-commit full recursive typecheck passed in 38.38 seconds and full build passed in 40.81 seconds. The build stamp matches `83ed4ba9cebd68d48929c46f15ab32d5c0b9687b`. Focused server attribution tests also passed 5/5 before the test-helper-only migration. Full test acceptance will use the repository's parallel CI gates; the unsharded local command takes hours. - Verification failures retained: initial sandbox runs could not open local IPC/PostgreSQL listeners; permitted reruns exercised the real services. One old test expected JSON serialization and was updated to assert explicit unmeasured complex values. The first bundle fixture used the wrong promise boundary, as described above. A retained-callback regression also demonstrated that disabling and reusing the same AsyncLocalStorage instance could resurrect an old owner; replacing the instance after the last capture closes fixes that case. - The first CI run found older projection tests consuming the previous full-response fields. Those consumers now use the full-response test helper, preserving their byte thresholds. The two complete affected route suites plus helper tests passed 115/115 in 73.08 seconds. A temporary full-row query negative control then failed the unchanged 2,048-byte limit at 5,000,812 bytes and was restored; five other service suites passed 34/34 and the affected chat cases passed 3/3. All three previously failing CI shards passed on the corrected head. A local skill-list benchmark smoke check completed 255 captured queries with no in-flight queries or overflow; this was integration verification under concurrent test load, not a timing benchmark. - Two final-head runner jobs lost communication with GitHub before reporting a test result. One targeted retry of those jobs and their dependent aggregate passed. All 53 final-head checks passed, with 3 intentional skips; previously passing suites were preserved. - Hosted capture remains pending until the reviewed revision is deployed and a bounded window is explicitly configured. Reproduce the focused checks with Node 24 and the repository pnpm version: ```sh pnpm exec vitest run packages/db/src/response-capture.test.ts packages/db/src/response-capture-postgres.test.ts pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/database-operation-attribution.test.ts scripts/bundle-runtime.test.mjs pnpm -r typecheck pnpm build ``` ## Risks - Owner context is active while a valid future capture window is armed, up to 30 minutes before accounting starts. No-window and expired calls take the direct path. Diagnostics still add bounded work while enabled. - A nearest nested owner is reported. Broad `http.request` and unattributed groups remain visible; this does not identify every service callsite or SQL shape. - The existing census excludes tagged/raw explicit-execute/streaming paths and internal protocol commands. Its counts are not PostgreSQL transaction totals. - Completion wall time includes queue/database/network/scheduling waits. Concurrent durations overlap. Sampled bytes exclude complex values, large strings, and values beyond the sizing limits. - Activation is startup-only. Operators must choose the window after artifact preparation, allow rollout/warm-up time, and remove the expired setting afterward. ## Model Used OpenAI Codex, GPT-6 family, with code execution, repository inspection, and parallel agents. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 12:12:21 -06:00
DottaandPaperclip	9ca9af544a	Drive remaining live run and resource recovery from durable work (#18419) ... Reviewed and verified at 6a4c193906. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 16:00:35 -05:00
DottaandPaperclip	eb12cfc96b	perf(github): activate connection work from events and deadlines (#18395) ... Reviewed and verified at f77620b9a4. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 15:38:43 -05:00
DottaandPaperclip	dce383c733	perf(server): stop recurring attention retention scans (#18388) ... Reviewed and verified at a61af9888e. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 12:28:40 -05:00
DottaandPaperclip	b6b57079b9	fix: evaluate status cards on committed changes and deadlines (#18401) ... Reviewed and verified at 21795e222a. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 11:41:59 -05:00
DottaandPaperclip	3ee4c5b613	Make saved execution receipt recovery react to durable work (#18409) ... Reviewed and verified at 8f306e0a6e. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 10:03:59 -05:00
DottaandPaperclip	baf651c8d0	perf(heartbeat): drive retry promotion from durable deadlines (#18387) ... Reviewed and verified at f1c48c924d. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 09:07:52 -05:00
Devin FoleyandPaperclip	81ecaf0654	fix(server): bound failed lifecycle verification (#18383) ... Failed lifecycle verification could allocate another test environment every minute indefinitely. Count failed host verification tests separately from claims, persist a repair wait for confirmed configuration failures or three generic failed tests, and clear it through authorized retry or execution configuration changes. Preserve typed temporary execution-target failures through adapter checks so timeouts, transport failures and interrupted probes keep automatic recovery. Retain existing readiness, authorization, termination and cleanup gates. Document the policy and add real PostgreSQL persistence and actual OpenCode adapter regressions. Verified 398 focused tests (one Windows-only test skipped on macOS), repository-wide typecheck and build, structural checks, fully green current-head CI and Greptile 5/5 with no unresolved comments. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-30 15:29:43 -07:00
DottaandPaperclip	f5e5b05a33	Preserve native workspace recovery across Stop and sandbox reuse (#18332) ... Land the thirteen-file native recovery fix independently from #18245. Retain verified Stop evidence and queued user comments. Retire stale preparation ownership atomically and require paired sandbox lease handoff receipts before workspace recovery. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-30 10:57:53 -05:00
3 people	ba07f85cad	perf: schedule agent timers from durable deadlines (#17876) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/58/changes/paperclip/58 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent timers must start due work and preserve queue admission rules. > - The server woke their scan on every heartbeat interval, even with no eligible timer. > - Idle scans keep the database and instance active. > - This pull request derives the next deadline from saved policy and work records. > - Supported producer transactions notify the shared coordinator when eligibility changes. > - An idle timer worker stops scanning and resumes when real work or a deadline requires it. ## Linked Issues or Issue Description Refs #17863. This is slice 63 of the recurring-work migration and follows the merged issue-monitor deadline PR. Related: #16632 changes timer issue scope; #16881 changes live-run eligibility. This PR preserves the current admission policy. Those policy changes are separate work. A future time-sensitive predicate must expose its next transition deadline. **What existing behavior does this improve?** Agent timer heartbeat scheduling in the server. **Current behavior** A recurring heartbeat wakes the timer scan even when companies are paused, timers are disabled, or no actionable task exists. **Proposed behavior** Register the earliest eligible interval or positive daily-cap reset deadline. Re-read on supported producer commits and startup. Leave no recurring database scan when no timer obligation remains. ## What Changed - Keep policy and deadline calculation in the heartbeat queue. Scheduling retains company, reporting-chain, worktree, and CAS gates. - Use the existing deadline coordinator. Remove only the timer wake from the server heartbeat loop. - Notify before existing transaction writes that change eligibility or deadlines. Cover configuration, lifecycle, company budgets, task ownership, recovery, conversations, accounting corrections, and completion. - Use the lifecycle status-change hook for saved hire rejection. Cover checkout's second adoption transaction after an initial failed CAS. - Preserve pending work through idle and maintenance drains. Check cancellation between due agents. Reconstruct on restart. - Document the supported writer model and producer inventory. Add no schema, table, or provider branch. ## Verification - **1,710 passing cases across 49 complete files**, no skips, on accepted source `32db86c1974dcbf794f01a8c5e42024e2ad071fc`. Coverage includes real timer policy/CAS/producers, monitor and shared deadline consumers, native reconstruction, Slack lifecycle, skills/settings/configuration, queue isolation and agent lifecycle. - Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. Twenty-nine fresh PostgreSQL 17 schemas plus per-file fixtures applied all 412 journal entries through index 413; all selected source hashes and temporary fixture imports were restored exactly. - Four new deadline composition cases verify future/overdue timing, minimum scan spacing and disarming. A controlled mutation dropping only the inherited continuation option fails; the composed source passes with identical assertions and timers. - Normal parent reconciliation preserves task-drain expiry and both deadline contracts. The final monitor-parent publication changes only its audit, preserving every qualified runtime and selected test source. - Historical failures and review corrections remain in [the timer audit](doc/idle-agent-timers.md). No full local repository-suite, live provider, measured suspend/wake, external-alarm or fleet-savings claim. Final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - Process-local notifications cover supported service/API writers in one server process. Direct SQL and independent replicas need an explicit ingress contract. Startup reconstruction is not live cross-process notification. - Notifications can batch for up to the configured heartbeat interval. Known deadlines retain their scheduler path; failed reconciliation retains bounded retry. - Enabled timers without an actionable-work requirement remain real recurring obligations. Zero caps require policy changes; positive reached caps use UTC midnight. - The timer fixture substitutes adapter enqueue. Existing admission tests also pass, but this is not deployed provider or sleep qualification. - Cloud does not gain legacy agent timer alarms in this PR. Existing sleep-safety blockers remain. No database/host suspension or fleet savings claim is made. - Parent-stack CI and review must pass before merge. This PR does not authorize merging its dependencies. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs, a controlled red/green regression and complete PostgreSQL integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 08:58:27 -05:00
3 people	0f6f0a3fee	fix(server): schedule issue monitors without heartbeat polling (#17863) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/56/changes/paperclip/56 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Issue monitors save a future check on an existing issue. > - The heartbeat timer still scans monitors on every tick, even when none are due. > - The saved deadline and claim already provide durable scheduling state. > - This pull request gives monitors their own deadline worker and connects state changes to it. > - Idle monitor work stops querying. Existing admission and recovery rules remain in force. ## Linked Issues or Issue Description Refs #17840, #16325. Related: #17171 addresses rearming a monitor after its admitted run fails. This PR changes when monitor inspection runs and follows the merged routine prerequisite. **What existing behavior does this improve?** Issue-monitor scheduling and idle database use. **Current behavior** Heartbeat ticks call the combined agent and monitor sweep. The existing next-deadline query only retains future monitors. Polling covers overdue leftovers, stale claims, and changes that make a saved monitor eligible. **Proposed behavior** Use the shared deadline worker. Retain overdue work with bounded retries. Wake after committed changes to monitor state, issue eligibility, company state, or native ownership. Keep legacy agent ticks separate for the next slice. ## What Changed - Add monitor-specific sweep and deadline methods to the existing scheduling owner. Keep the manual combined method. - Include overdue work and the strict five-minute claim fence. Exclude active native owners until their state changes. - Register monitors with the shared deadline worker. Heartbeat ticks wake only the legacy agent timer worker. - Connect direct issue-state writers and existing execution-control notifications to monitor work. Checkout uses a short transaction around its existing CAS. Provider work stays outside it. - Add real database lifecycle tests and retain routine, startup, admission, execution-control, and native-projection coverage. - Record the obligation, producer inventory, recovery behavior, and limits in `doc/plans/2026-10-27-issue-monitor-sleep-audit.md`. ## Verification - **1,233 passing cases in 35 complete files**, no skips, on accepted source `c24aadfc91e9394d73c799f9e05fc67202160557`. Coverage includes real monitor scheduling/admission, native settlement, committed producers, retained routines and all shared deadline consumer integration suites. - Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. Twenty-four fresh PostgreSQL 17 schemas plus per-file fixtures applied all 412 journal entries through index 413; every selected source hash and temporary fixture import was restored exactly. - Normal parent reconciliation preserves master's admitted-wake result and scoped issue deletion while keeping committed monitor notifications. All other paths match the automatic merge; packages remain byte-exact to the qualified parent. The final routine-parent publication changes only its audit. - Historical failures and corrections remain in [the monitor audit](doc/plans/2026-10-27-issue-monitor-sleep-audit.md). Old delegated-Slack rate and Runner drain CI failures are retained; the current inherited clock fixture and package source are byte-identical to merged #17835. Fresh final-head CI remains required. - No full local repository-suite, live provider, deployed suspend/wake, external-alarm, production-load or fleet-savings claim. Final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - A missing producer notification can strand a monitor after polling is removed. The audit covers ordinary writes, direct status/assignment writers, recovery, and native ownership changes. - The writer model is process-local. Independent SQL writers and other replicas need a supported notification or reconciliation contract. - Overdue work uses the configured heartbeat interval as a retry floor for the monitor worker. New notifications during that backoff wait for its next pass. Normal state-change bursts coalesce for one second. - Notification fanout can inspect monitors during other active work. An unchanged idle subsystem has no recurring monitor scan. - The legacy agent timer still polls. Cloud sleep proofs and external alarm coverage are unchanged. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, ordinary Git merges, composition proofs, PostgreSQL fixtures and complete local integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 08:40:56 -05:00
3 people	829f9e7eaf	fix(server): settle Muse terminal assignments from committed changes (#17793) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/12/changes/paperclip/12 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its external-agent module owns Muse assignments, operation receipts, and worker holds. > - Running assignments and retained terminal history must not cause an empty maintenance scan every five seconds. > - Terminal run changes and operation results already have durable database transactions. > - This pull request wakes the existing maintenance worker from those commits and reconstructs work at startup. > - The worker retries only concrete settlement debt. It retains qualification deadlines and authority checks. ## Linked Issues or Issue Description Refs: #17772. This is slice 57. Parent #17772 has merged. Related: #17000, #17525, #17752, #17758. The publication targets master and preserves the merged parent. **What existing behavior does this improve?** Muse terminal assignment settlement and the existing Muse maintenance worker. **Current behavior** A running assignment can keep maintenance scanning every five seconds. Terminal reconciliation rewrites unchanged history. Native finalization also calls that reconciliation during periodic replay. Discovery excludes a revoked original binding even when its assignment still needs settlement. **Proposed behavior** Committed run changes and operation receipts wake the existing worker with the original company, run or binding identity. Startup reconstructs actionable work from existing rows. Maintenance processes bounded batches and disarms after settlement. Default live and manual callbacks remain. The periodic callback delegates this one duty. **Reason and benefit** Configured but idle instances stop querying for this obligation. Late receipts, restarts, drain and historical bindings still have recovery paths. No table or independent timer is added. **Breaking changes** No schema, endpoint or provider protocol changes. Explicit manual reconciliation retains its unbounded default. This does not alter Cloud sleep admission or claim that all instance work can sleep. ## What Changed - Add an owner predicate shared by discovery and locked settlement. Fenced private-worker uncertainty remains held without an automatic retry loop. - Preserve original binding generation and company/agent/run identity, including revoked bindings. Fence only the terminal assignment during reconciliation. - Extend existing maintenance with scoped hints, bounded batches, concrete-debt retry, startup reconstruction and quiet drain behavior. - Register notifications inside existing terminal, projection, cancellation, operation-result and send-failure transactions. Keep qualification and OAuth signals explicitly global when coalesced with scoped changes. - Serialize only four domain IDs. Real native-finalizer tests caught and prevent a circular-object serialization failure at this boundary. - Use existing restore/worktree suppression and settings reactivation. Delegate both periodic finalizer branches while retaining resource callbacks and default live/manual settlement. - Add the producer audit and regression evidence in `doc/plans/2026-10-26-muse-terminal-settlement-sleep.md`. ## Verification Accepted production source: `584015018ed4a9d57d2b0f192b75847503b54e53`. Publication head: `305014b030d6c5eb891c70bc57036f4192d4fdad`. Normal adoption of the actual parent squash preserves the complete accepted child tree. Only audit text, a migration-count clarification, and the proved Photon test extension follow. - **3,715 distinct cases in 52 complete server files pass**, plus **20 selected chat cases** (3,735 selected total). The main 49-file batch passes 3,008; process recovery passes 410; review response passes 273; the full Photon file passes 24. The other 1,794 chat cases were not run locally. - **Quiet producer proof:** moving only the Muse fanout outside its parent guard produces six candidate SELECTs and fails the regression. Keeping the guard yields zero and still settles a real terminal run. The same test hash spans the controlled red/green pair. - **Photon review evidence:** repeated active callbacks produce zero settlement hints/candidate SQL. Three concurrent attention recoveries produce exactly one of each. The existing locked attention guard was correct; production needed no fix. The parent security check accepted the resolved finding before merge. This PR keeps the additional settlement queue in the existing regression. - **Static checks pass:** database build; server typecheck/build; module boundaries; migration order; current Copilot identity. Packages equal the accepted parent byte for byte. No new live provider qualification is claimed. - **Fixtures and failures:** 34 fresh PostgreSQL 17 fixtures have 412 applied migration receipts through tag 0413_tired_kronos, plus fresh per-call helper fixtures. All 52 original test hashes restore exactly. The additional Photon test hash matches its passing proof. Earlier remote chat passive-idle failure remains recorded; both passive/missing cases pass in the new selection. Parent handoff timeout/fresh-file-pass accounting remains in its audit. No assertion or timeout was weakened. - [Integration and quiet producer audit](https://github.com/paperclipai/paperclip/blob/305014b030d6c5eb891c70bc57036f4192d4fdad/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md#current-master-integration-and-quiet-producers-2026-10-29). Fresh CI, security, and exact-head Greptile 5/5 must pass before merge. <details><summary>Earlier revision verification and failure history</summary> Receiver failure fairness: a controlled 1,100 ms batch reproduced a 30-second delay for healthy due receivers. The worker now supplies its existing per-item deadline through `nextRunAtOnError`. All 20 receiver lifecycle tests pass on the owner and combined tip `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production typechecking passes on the owner. The failed receiver retains its own backoff, and settled receivers return to zero recurring SQL. The five-second assertion budget is unchanged. See [the owning audit](https://github.com/paperclipai/paperclip/blob/f7477933469fdc960f555425dce66eff25a94e54/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch). Current-head CI and review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/bbd31dea5ef3a4875950a87b4636af8b6bde858e/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Inherited voice-cleanup verification: the controlled slow-provider regression and all 13 lifecycle tests passed on its owner and combined tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only the fixture clock advancement and its audit note. Current head: `f7477933469fdc960f555425dce66eff25a94e54`. This remains a single-duty PR stacked on #17772. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-24-recurring-server-sleep-stack.md), [2026-10-26-muse-terminal-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. </details> ## Risks - In-process signals do not capture independent SQL or other replicas. Supported external events still need ingress. - Pending assignments and operation receipts retain their original binding identity. A new agent configuration cannot take their ownership. - Genuine qualification, OAuth, and recovery deadlines remain obligations. They must not be discarded merely to make the worker quiet. - The parent coarse-hint guard prevents recurring Muse discovery; real company/binding authority changes retain their signals. - No full local repository suite, full chat suite, live Muse or Copilot qualification, deployed sleep, database suspension, external alarm coverage, or cost savings is claimed. Existing Cloud blockers remain. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, real PostgreSQL tests, and GitHub tools. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 06:18:40 -05:00
3 people	4d0847c4ca	fix(server): settle external conversations from committed work (#17772) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/6/changes/paperclip/6 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip manages AI agents and the tasks that they execute. > - External conversations must become idle after an authorized response is published and all related work is complete. > - Native replay and stranded-task recovery still discover this settlement work through recurring scans. > - A new deadline worker must preserve response ordering, monitor handoffs, and unfinished work. > - This pull request gives settlement one shared readiness predicate and activates its worker from committed changes. > - Existing rows provide recovery truth. No new table or per-feature timer is added. > - The benefit is a settlement worker that becomes quiet and resumes when relevant work changes. ## Linked Issues or Issue Description Refs #17758. This is slice 56. Slice 55 has merged. Related: #17606 (chat milestones), #17663 (bounded run reads), #17665 (monitor proof), and #13809 (answered Slack conversation lifecycle). The provider applicability remains Slack and Speko-owned conversations. This does not extend Telegram settlement from #16559. **What happened?** Conversation settlement depended on periodic native replay and generic task recovery. An exhausted monitor also cleared its schedule before recovery admission. During that gap, an answered conversation could look finished. Failed or deferred admission lost the monitor schedule. **Expected behavior** Relevant committed changes activate settlement. The worker uses the owning service's evidence and becomes quiet when no settlement is owed. Monitor recovery stays pending until admission, and restart does not admit an already accepted recovery twice. **Steps to reproduce** 1. Create an active external conversation with a successful run and an authorized published response. 2. Keep execution, approval, a goal, or a monitor pending. Let the settlement worker become quiet. 3. Resolve that prerequisite through its owning service. Verify that the conversation settles without another periodic sweep. 4. Delay or fail exhausted-monitor recovery admission. Verify that the monitor remains pending and the conversation cannot settle during the handoff. **Paperclip version or commit** Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`. Parent #17758 merged as `8ac5a128e66fb672e88106412f76cf278599bfc4`. The publication adopts that actual squash through a normal merge with an unchanged complete child tree. ## What Changed - Preserve quiet after coarse company-only maintenance hints. The shared inbound-notice and response-availability helpers keep settlement enabled by default. Only the milestone dispatcher suppresses their duplicate settlement fanout. Its scoped issue-review signal and the explicit company-authority signal still wake real work. Four database cases prove both quiet and real company-wide producer delivery. - Preserve source-generation checks for chat handoff, task-before-comment locking, issue-scoped decision invalidation, and current conversation event columns from master. - Track initial external conversation settlement through shutdown without blocking HTTP readiness on endpoint or task locks. - Reuse the real queue's coalesced recovery receipt after restart, without admitting the same monitor recovery again. - Share the existing conversation readiness predicate between bounded discovery and locked settlement. Preserve current provider policy, run/publication proof, company boundaries, and endpoint-to-issue lock order. - Register a scoped deadline worker through the delivery coordinator. Preserve startup, restore/worktree suppression, drain/resume, failure recovery, and joined shutdown. - Extend existing review, milestone, wake-notice, and finalization signals. Add notifications at independent issue, comment, goal, session, and wake-receipt unblocks. - Keep monitor owner recovery pending until guarded admission. Reuse an accepted receipt for the same monitor schedule after restart, even when its limit reason changes; retain compatibility with legacy receipt keys. Commit board escalation and its monitor clear together. - Delegate periodic native settlement while preserving resource callbacks. Keep direct/manual defaults and preserve generic recovery ordering with a read-only readiness guard. - Add the producer and lifecycle audit in `doc/plans/2026-10-26-external-conversation-settlement-sleep.md` and update the stack ledger. ## Verification Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`. Publication head: `b256bb52647affd9f5279ea8c40f48bbe40f031f`. The complete accepted child tree stays unchanged when adopting merged parent #17758. This publication adds only the audit document. - **3,129 distinct cases in 37 complete server files pass**, plus **18 selected chat-response cases** (3,147 selected cases). The other 1,796 chat cases were not run locally. Full settlement, queue, scheduling, run-state, goals, interactions, task Retry, decision targets, status reassessment, workspace, process recovery, and review consumers are covered. - **Preserved failure:** the first 35-file command passed 2,445 cases and timed out one restored-work handoff at its existing 15-second limit. That aggregate command failed. The unchanged full 89-case actions file then passed on a fresh fixture; the affected case took 22 ms. Root cause was not established. No timeout or assertion changed. The 34 other files passed originally. Distinct counts exclude duplicate actions passes. Fresh CI remains a separate gate. - **Indirect wake regression:** the original unconditional baseline failed two quiet checks. Guards alone left the chat case failing with three candidate queries. The final controlled indirect baseline fails one of four cases; the repair passes all four with the same test hash. Positive controls preserve real company-only producers. - **Source and fixture controls:** all 38 selected test files restore to their exact committed hashes after fixture imports. Fresh PostgreSQL 17 fixtures use 413 migrations. All five manual merge resolutions preserve current authority, locking, and event-column contracts. - **Static checks pass:** database build; server typecheck and build; module boundaries; migration order; Copilot profile identity. Runner packages equal the accepted parent byte for byte. No new Runner/package qualification is claimed. - [Current integration and wake audit](https://github.com/paperclipai/paperclip/blob/b256bb52647affd9f5279ea8c40f48bbe40f031f/doc/plans/2026-10-26-external-conversation-settlement-sleep.md#current-master-integration-and-indirect-wake-audit-2026-10-29). Latest-head CI and exact-head Greptile 5/5 must pass before merge. <details><summary>Earlier revision verification and preserved failure history</summary> Receiver failure fairness: a controlled 1,100 ms batch reproduced a 30-second delay for healthy due receivers. The worker now supplies its existing per-item deadline through `nextRunAtOnError`. All 20 receiver lifecycle tests pass on the owner and combined tip `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production typechecking passes on the owner. The failed receiver retains its own backoff, and settled receivers return to zero recurring SQL. The five-second assertion budget is unchanged. See [the owning audit](https://github.com/paperclipai/paperclip/blob/83204a67880d980cfbea3771cee19a3cda0022b7/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch). Current-head CI and review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/ef9ae5767430d0093efc736ae0b303ba5afaae65/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Inherited voice-cleanup verification: the controlled slow-provider regression and all 13 lifecycle tests passed on its owner and combined tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only the fixture clock advancement and its audit note. Current head: `83204a67880d980cfbea3771cee19a3cda0022b7`. This remains a single-duty PR stacked on #17758. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-24-recurring-server-sleep-stack.md), [2026-10-26-external-conversation-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-26-external-conversation-settlement-sleep.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. </details> ## Risks - Signals cover supported in-process writers. They do not supply change capture for independent SQL or other replicas. - Real company-wide changes can inspect active candidates. Coarse maintenance hints do not activate settlement. Configured idle and retained history must remain quiet. - Monitor receipt binding and admission order must preserve company, agent, issue, and exact schedule identity. Current parent handoff locks and source-generation checks remain intact. - The local restored-work timeout is retained above. A fresh full-file pass does not establish its cause; latest-head CI must also pass. - Generic recovery and Muse/resource duties remain separate slices. This does not make the whole server idle. Existing Cloud sleep blockers remain. No deployed sleep, database suspension, external alarm coverage, fleet cost, or live provider qualification claim is made. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, PostgreSQL tests, and GitHub tools. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 06:02:43 -05:00
3 people	df6156c2e3	fix(server): wake native status reassessment from committed changes (#17709) ... ## Thinking Path > - Paperclip manages agents, tasks and their durable native results. > - Changed evidence, task authority or review retirement can require a new status assessment. > - Periodic orphan recovery repeatedly reads history to find these changes. > - This change gives reassessment an owner driven by committed changes and saved deadlines. > - Existing result, assessment, decision and task rows remain durable truth. > - The status committer still verifies current sources and authority before writing. > - Settled or insufficient evidence becomes quiet until a supported producer changes it. ## Linked Issues or Issue Description Refs #17702 (merged), #17679 and #17680. This is slice 50, now based on master. It owns native status reassessment; workspace settlement, retained sessions and other orphan duties remain separate. **What happened?** Periodic recovery repeatedly inspects completed native results. Removing that scan without complete writer coverage or current authority checks could strand new evidence or commit an old result over newer work. **Expected behavior** Reassess from existing durable facts after relevant outer transactions commit. Retain real retry/lease deadlines and restart/drain recovery. Preserve current authority, quality-review and supersession checks. Make no recurring SQL when work is settled or dormant. **Steps to reproduce** Complete a native result awaiting evidence or review, leave it idle, then approve a real work product or retire the selected automatic review. Its saved result should advance once when authorized, or record the existing nonterminal business gate, without a periodic scan. ## What Changed - Extract reassessment and its pure status policy from finalization recovery into their owning modules. Add a scoped worker using the existing delivery coordinator and scheduler; no queue table or migration is added. - Retain the three durable discovery triggers: changed work-product history, authoritative task status/version, or retirement of the selected automatic completion review. Preserve selected-source, ownership, effects, retry and lease fences under the existing committer. - Notify from actual task/work-product, claim release, approval, interaction, OAuth, event evidence and workspace-settlement owners before their transaction writes. Keep provider calls outside short transactions and nested writes behind outer commit. - Scope ordinary milestone/finalization/coalesced-input wakes to an issue/run. Keep genuine company authority restoration explicit. Real Photon and coalescer regressions cover all three response/status queues. - Move the current parent's required quality policy into the owner and apply it to every reassessment that could complete a task. An unmet review records the existing in-progress decision and next action, avoiding an operational retry loop. Preserve conversation/session reset locks and the workspace omission-barrier signal. - Skip reassessment in periodic recovery while preserving explicit/startup recovery and the historical supersession guard before independent workspace work. Document the source contract, producer inventory, failures and qualifications in the linked audit. ## Verification - Broad source `9fddb3f0b1157df32b64fb1f22e9d6e9624470e8` passes **2731 distinct cases in 28 complete files**, with no skips: 2458 producer/owner/authority/workspace/quality cases in 394.43 seconds and all 273 governed-response cases in 53.74 seconds. Nineteen dedicated fresh PostgreSQL 17 databases migrate through 0413; eighteen embedded-only invocations preserve fresh-per-call isolation through the existing recovery helper. All 22 temporary imports restore exactly and all 28 hashes match. - Final runtime source `5ea3b0fd016bf7e63ba82a7044825a46a0cd2082` passes **1211 cases in five complete affected files** in 108.23 seconds after the narrowly reproduced evidence-trigger fix: all 43 worker, 145 corpus, 32 finalization-recovery, 968 native-session and 23 quality-review cases. Four fresh databases plus the quality helper's own fresh fixture use PostgreSQL 17. Its one adapted import restores exactly; all five hashes match. The other 27 broad-run test files are unchanged. These qualifications are source-scoped and not cumulatively double-counted. - Required/optional real regressions reproduce retry loops on retired-review and evidence-approval paths before repair; both become SQL quiet afterward with unchanged assertions/time limits. Controlled removal of only ordinary scope guards reproduces two coarse-hint SQL failures; exact guard restoration passes. Real producer, outer transaction, claim, authority, 103-item backlog, retry and drain cases pass. Photon transport is mocked; callbacks, locks and observed SQL are real. - Normal merges preserve 12661 parent/unrelated entries, 35 owned entries and 21 automatic shared results, with five original conflicts reviewed explicitly. The parent's actual squash reconciliation preserves the entire held child tree. The final quality fix changes only its owner and tests. Fourteen added comments are removed with identical nontrivia tokens; no directives are removed and the final comment check is empty. - Database build, server types/build, boundaries and exact-head migration order pass. The database tree is identical to merged #17702: no new SQL, queue table or snapshot is generated. Earlier benchmarks, host limitations, failed assertions and their repairs remain in `doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md` and `doc/idle-native-status-reassessment.md`. - This qualifies targeted external PostgreSQL behavior, not embedded bootstrap on the semaphore-exhausted host, a full local repository suite, deployed sleep or measured fleet savings. Fresh exact-head CI and review remain required. ## Risks - Notifications retain the process-local supported-writer contract. Independent SQL writers and uncoordinated replicas are not covered by a new transport. - Reassessment competes with current authority changes. Existing row/source locks and retries remain required; arbitrary missing identities have no row to lock, so this is not a general predicate-lock guarantee. - Discovery pages 100 rows but retains outstanding candidates in memory. It remains proportional to backlog size, and cancellation occurs between queries. - Other orphan/workspace/session duties and Cloud sleep eligibility remain unchanged. Missing supported producer coverage could strand dormant work, so the producer audit and real callback/transaction tests remain part of acceptance. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution and PostgreSQL tests. The runtime does not expose a more specific serving snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted scope is stated above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 03:40:01 -05:00
3 people	79ce86630a	fix(server): schedule governed review response recovery (#17702) ... ## Thinking Path > - Paperclip manages agents, tasks and their external chat responses. > - A completed chat turn can owe a response while an earlier review gate remains pending. > - Periodic recovery revisits that response history even when no work can advance. > - This change uses the existing delivery coordinator and saved retry deadlines. > - Approval, interaction and authority changes wake the affected issue or run after commit. > - The existing presentation owner verifies the durable result and current authority. > - Settled and dormant history becomes quiet without losing future eligible responses. ## Linked Issues or Issue Description Refs #17696 (merged) and #15625. This is slice 49, now based on master. It owns governed-review response presentation. Accepted responses and pending decision effects keep their existing owners. **What happened?** Native finalization recovery repeatedly inspects committed review responses. Removing the scan without covering approval, interaction, task-policy and authority changes would strand responses that become eligible later. **Expected behavior** Recover an owed response once from existing durable facts. Enforce authority at publication, wake after relevant outer transactions commit, retain real retry deadlines and restart/drain recovery, and stop recurring SQL when no response can advance. **Steps to reproduce** Complete an external chat turn while an earlier completion review remains pending. Block its response behind an approval or interaction, then resolve that blocker. The response must appear once without a periodic scan or resolving the original review gate. ## What Changed - Add a scoped governed-review worker using the existing coordinator, finalizations, decisions and comment receipts. No queue table or transaction wrapper is added. - Keep recovery and publication policy in the native finalizer. Preserve binding, source/session authority, ownership, deadline and deleted-comment fences; match coordinator-before-issue lock ordering. - Notify through actual approval, interaction, OAuth, task-policy and lifecycle producers before their owned writes. Serialize approval resolution with task linking so a concurrent link retains its wake. Nested transactions still settle at outer commit. - Scope ordinary milestones, finalization and coalesced input to their issue/run. Use the explicit company authority boundary for genuine global restoration, including membership changes. Both response workers share this boundary. - Register under maintenance and drain gates, disable the two periodic repair entrances, and retain fresh finalization and explicit recovery. Add the concurrent 0413 comment-receipt index and migration registration; retain five snapshots. - Preserve the producer inventory, source qualifications and earlier failures in the owner audit. Retain a real Photon regression proving routine ownership assertions stay quiet and concurrent authority restoration wakes each worker once. ## Verification - Scoped repair source `2341714720b5c273f0ce011b48345af0ef39227f` passes **2053 distinct cases**: 1703 in 15 complete producer/authority/recovery files, including all 273 governed-response cases and 95 wake-queue adapter cases, plus 350 selected chat cases. Another 1463 chat cases are outside that selection. Thirteen fresh PostgreSQL 17 fixtures migrate through unchanged 0413; the one temporary queue-fixture import restores exactly. All sixteen captured hashes match. - Two actual repeated coarse-hint SQL regressions fail before repair and pass afterward. Actual membership restoration passes before and after. The same assertions and time limits prove dormant work stays quiet and real authority restoration still publishes once. Real coalesced input covers both queues after commit. - Combined source `586f5674cd9798b33cd22437255e0480b5726d20`, including independent master #18129, passes **1297 distinct cases**: 1093 in five complete server files, 185 in five runner files, 14 Node protocol cases and five UI registry cases. This includes the complete 22-case Photon file for both response workers. The external Photon transport is mocked; Paperclip callbacks, transaction fences and observed SQL are real. The earlier and combined case counts are separate, with no cumulative double-counting. - A normal merge retains both complete owned and independent master binary patches. All twelve new-context hashes and sixteen earlier repair hashes match. Three dedicated fresh 0413 fixtures plus the live-event helper's own fresh database use PostgreSQL 17 without source adaptations. Database build, server types/build, boundaries, full-SHA migration order and generated TS/Rust profiles pass. The new inherited Runner vendor inputs rebuild successfully. - Database bytes remain identical to the previously qualified 0413 source. The recorded populated 0412-to0413 upgrade retains all 411 prior receipts and 38 companies, appends one receipt and none on replay; the index is valid and ready. The earlier 2828-case acceptance, CLI journey, setup failures and their repairs retain their recorded source scope in `doc/plans/2026-10-26-governed-review-response-sleep-audit.md`. - Parent #17696 merged with 58 clean checks, Greptile 5/5 and Ponytail 5/5. Its Photon warning was disproved by real callback/SQL evidence and the security check became successful. This PR still requires its own fresh exact-head CI and review gates. No deployed sleep, external alarm or fleet savings claim is made. ## Risks - Notifications use the existing process-local writer contract. Independent SQL writers and uncoordinated replicas are outside its coverage. - Genuine startup and company authority reconstruction remain proportional to the owed backlog. Queries page 100 rows, but discovery accumulates the candidates; this is not a constant-cost global operation. - Current publication authorization, durable binding, ownership and lock ordering remain mandatory. The worker does not resolve the original review gate. - The concurrent index uses existing migration replay handling. Other recovery duties and Cloud sleep eligibility remain separate. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution and PostgreSQL tests. The runtime does not expose a more specific serving snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 02:54:46 -05:00
3 people	ec3b558277	fix(server): schedule accepted native response recovery (#17696) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs store accepted results and their finalization decisions. > - A completed run can still owe its public answer after an interruption. > - Periodic recovery scans for that answer even when no new work exists. > - This pull request wakes recovery from committed changes and saved retry deadlines. > - The existing owner still verifies the result, source, and current publication authority. > - Settled or dormant response work no longer needs its own periodic scan. ## Linked Issues or Issue Description Refs #17690 (merged). This is slice 48 of the sleep migration stack, now based on master. Related: #17661 handles GitHub terminal publication; this change handles native accepted-response presentation. Governed-review presentation remains a separate duty. **What happened?** Native finalization recovery repeatedly checks already accepted results for a missing answer. Removing that check without complete notifications would lose responses when evidence arrives late, an agent resumes, or admission settles after execution. **Expected behavior** Recover the answer from its saved result when a supported change makes it eligible. Retain real retry deadlines. Stop querying when the answer is selected or cannot currently be published. Reconstruct work after restart. **Steps to reproduce** 1. Commit a native accepted result with no selected response comment. 2. Run finalization recovery while its source is unavailable or locked. 3. Restore that source or release the lock. The answer must appear once without repeating provider execution. ## What Changed - Honor recorded non-create decisions and skip-comment policy in both discovery and recovery. Apply durable conversation/result/continuation guards before recovery locks. Preserve genuine unanswered and legacy recovery. - Scope routine response wakes to the affected issue/run. Coalesced input names its active run; pending-effect settlement names its issue. Sixteen existing global authority/configuration hooks use one explicit company invalidation boundary. No new transaction wrappers or tables are added. - Add an accepted-response worker using the existing delivery coordinator and scoped deadline helper. Preserve saved retry deadlines and pending-effect fences under the owner's lock. - Share the finalization notification boundary. Wake on finalization settlement, exact accepted-result/final-message evidence, agent availability, retry admission, and existing chat authority changes. - Keep lifecycle policy in services. A narrow pre-write status hook lets lifecycle retain ownership of its transaction and status transitions. - Register the worker during enabled maintenance startup. Remove its authority subscription on shutdown. Periodic replay skips this duty on both committed and fresh-commit paths; ordinary finalization and explicit recovery retain their behavior. - Add a partial index and expression statistics for unpresented native runs. Migration 0412 supports concurrent replay and adds no queue table. - Add real PostgreSQL and lifecycle coverage. Preserve the implementation and producer audit in the plan document. ## Verification - Additional real Photon callback acceptance on the published production code disproves the security warning: ten active assertions cause zero response-recovery hints/SELECTs, while three concurrent assertions during an actual attention-to-active recovery cause exactly one wake/SELECT. The existing locked status guard already enforces this boundary. No production change was needed; the resolved finding now has a successful current-head security check. The regression will be retained with the next slice. - Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883` passes **1489 distinct server cases**: 1139 in six complete recovery/finalizer/monitor/summary/workspace suites (81.02 seconds), plus 350 selected chat cases (297.68s). Another 1463 chat cases are outside the pattern; the complete 1813-case file is not claimed. All eleven new regressions are included in that count. Five dedicated fresh PostgreSQL 17 databases migrate through unchanged 0412, with no fixture-source adaptations. Database build, server types/build, boundaries and full-SHA migration ordering pass. All seven test source hashes match. - The exact published baseline first fails eight behavioral assertions while the three genuine-answer cases pass. The same corrected test source passes all eleven after repair: recorded none/reuse remain suppressed; five ineligible history families take zero recovery row locks across repeated issue hints; missing/legacy/create-null answers still publish once; fresh periodic finalization defers presentation to its worker. Earlier fixture setup and variable/compiler failures remain explained in the audit, rather than erased by later success. - Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821 server cases**: 1496 in 13 complete producer/lifecycle/authority suites (110.63 seconds), plus 325 selected chat cases (278.99 seconds). Coverage includes all native-session cases, queue admission, status decisions, membership/board access, the complete GitHub workflow, accepted-response lifecycle and real authority restoration. The other 1477 chat cases are outside this selection; the complete 1802-case file is not claimed for this repair run. - Actual PostgreSQL regressions first reproduce a coalesced-input hint missing its run scope and two routine hint paths producing 12 response-history SELECTs each after three notifications. The same assertions pass with run-scoped coalescing and zero response-history SELECTs for unscoped routine hints. Actual membership restoration still publishes the waiting response once. Assertion time limits are unchanged; this measures real driver SQL and persisted producer work. - Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412. One embedded-only wake-queue fixture import temporarily uses its own external database and restores byte for byte. All 14 selected source files match pre-run hashes. This qualifies external-database behavior rather than reproducing embedded CI locally. - Master advanced independently with #18126's Pi archive retry fix. Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both complete binary patches. All accepted server/database/UI/test/CLI trees stay identical; the three independent runner installer/manifest files are retained. All 13 installer tests pass, including stalled header/body cancellation. Combined server types/build, boundaries and full-SHA migration order pass. A subsequent normal merge of #18128 at `e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only file changes with both complete patches preserved. The changed HTTPS backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178 outside the pattern), and combined types/build, boundaries and migration order pass again. All 14 repair source hashes remain unchanged. The independent browser fixture changes are preserved; no local browser run is claimed. - The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed the 4171-case full baseline before this scope repair. That scope is preserved in the audit rather than represented as a current full rerun. All 0412 database bytes remain unchanged: original concurrent index/statistics/ANALYZE, prior executable history, journal entries and retained snapshots. Five snapshots 0408 through 0412 remain. The populated 0411-to0412 upgrade preserved 410 prior receipts and 38 companies, appended one receipt and none on replay; index/statistic checks passed. - The earlier P2 security failure, Greptile's former conditional dependency on #17702, and the subsequent actual Ponytail eligibility findings are recorded in `doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This PR now includes the issue-scoped pending-effect notification fix itself, so it is independently correct before #17702. Fresh published-head CI, Greptile 5/5 and a clean security scan remain required. No deployed-sleep, external-wake or fleet-savings claim is made. ## Risks - The writer contract is process-local. Independent SQL writers and other replicas are not covered by these notifications. - Unpresented backlog reconstruction still has memory and time cost proportional to candidate count. Pages contain at most 100 rows; execution uses the shared fair batch. Selected history is excluded. Health-only signals do not trigger reconstruction. - Publication policy remains with its current owner. Digest, company/run binding, source authorization, selected-answer suppression, and ownership fences remain required. - The rest of native recovery and governed-review presentation still run separately. This PR does not relax Cloud sleep eligibility or add external wake coverage. - The stack parent has merged and its actual ancestry is reconciled. This publication still needs fresh current-head CI and review. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution, and PostgreSQL tests. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-10-29 02:31:20 -05:00
DottaandPaperclip	4e715eb7ea	fix(server): schedule historical unsafe export recovery (#17687) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs can retain accepted results after an unsafe workspace export fails. > - Historical recovery can retire an old notice or resume the accepted result without another provider turn. > - The periodic reaper repeatedly scans failures that cannot currently recover. > - Existing rows and committed changes can identify the actual recovery obligation. > - This pull request wakes recovery from those changes and retries only owed work. > - Dormant history becomes quiet while ownership and explicit operator decisions remain protected. ## Linked Issues or Issue Description **What happened?** The native finalization reaper repeatedly visits historical unsafe workspace-export failures. Cancelled tasks, newer runs, changed contracts, and explicit operator decisions can make those rows permanently ineligible. They still produce periodic database reads. **What did you expect?** Retire stale unsafe-export notices and recover eligible accepted results. Stop reading dormant history until a relevant committed change occurs. Keep a retry while a real obligation is blocked by ownership or a transient failure. **Steps to reproduce** Retain an unsafe-export terminal failure whose task is cancelled or superseded. Leave the instance idle after startup. The periodic reaper continues reading that history. Prerequisite #17680 is merged into master. Related: #14402 established safe export finalization; #14329 was an earlier historical recovery proposal. #17679 owns separate policy-review cleanup. This PR changes only historical unsafe-export scheduling and its producers. ## What Changed - Register a scoped worker through the existing coordinator and scheduler. Remove this duty from periodic reaping. Preserve the manual/startup recovery entry point. - Distinguish stale notices, eligible accepted results, dormant history, and concrete ownership contention. Reuse the existing mutation owner and lock order. Preserve explicit settlement vetoes and exact result/contract identity. - Signal relevant task, run, action, and deletion writes before they commit. Existing transaction boundaries handle nested commits and uncertain outcomes. Notifications contain only scope IDs. - Retain a retry only for owed work. Reconstruct work at startup, preserve drain/resume, and stop through the shared lifecycle. - Add concurrent, idempotent partial index migration 0410 for unsafe terminal failures. Keep the five newest generated schema snapshots. - Document producer coverage, query plans, supported writer boundaries, and test failures in the unsafe-export audit. ## Verification - Source `2a13dd08a6cc52f0adab09e3df9ce31e97b746b9` passes all 2188 cases across 24 complete files, without skips. The main selection passes 2129 cases in 192.58 seconds; safe-replacement passes 59 in 26.04 seconds. Coverage includes all 968 native-session cases, real export/action producers, explicit continuation, workspace reconstruction, interaction routing, Stop, queue and Slack consumers. The audit gives per-file counts. - Twenty separate fresh PostgreSQL 17 databases migrate through 0410 before setup. Four embedded-only fixture imports temporarily use their own external database on this IPC-exhausted host, then are restored byte for byte. Assertions and timing limits stay unchanged. This is qualified external-database acceptance, not an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-head migration order pass. Normal integration preserves the complete original runtime/schema patch and unrelated upstream blobs. Original concurrent index SQL, all prior executable history, journal entries and retained snapshot bytes are unchanged; five snapshots 0406 through 0410 remain. Added comment removal preserves the full non-comment TypeScript token streams. - A populated 0409-to0410 upgrade preserves all 408 prior migration receipts and 38 companies. It appends exactly one receipt; replay adds none. The index is valid and ready on fresh and upgraded databases. - The owning audit retains earlier fixture failures, skipped checks and notice-retirement regression history. Older combined stack checks retain their original scope in their owning audits. Actual parent `73d24aa5acfcf17b3f676b4da7cd05eb70b60b99` is reconciled through a normal merge. Its tree equals the reviewed ancestor and the complete accepted child tree is unchanged. Fresh published-head CI and Greptile 5/5 remain required. No deployed sleep, fleet-cost or live provider qualification claim is made. - Inherited prefix correction: current integration `8f43c9f81fac23b50997e7f0e1d0d9cf383a8109` changes only two values in a previously unselected corpus fixture. All production code, the complete package tree and all 24 accepted test files remain byte-identical to the 2188-case source above. The additional complete 72-case corpus passes in 14.00 seconds and again on the same database in 13.08 seconds. Retained PSC/OUT and sentinel rows remain unchanged. The repeat is not double-counted. The parent audit retains the original collision and rejected cleanup attempt. - Latest parent context: integration `4d51231abe8b68c21ce896d0d2ac16865978c8ab` carries only the parent's one-argument GitHub observation correction and an independent master cleanup assertion. All production code, packages, 24 accepted suite files and the additional corpus are unchanged. The two focused native-presentation cases pass in 15.14 seconds and the complete cleanup-removal suite passes all six cases in 8.31 seconds on two fresh 0410 PostgreSQL fixtures. This does not claim the full parent workflow was rerun here. Current head `b28399babac72ba9db1c4a4ef882c5e87aded38c` has passing CI and Greptile 5/5 with no actionable findings. The first CI attempt lost runners: fourteen jobs explicitly received simultaneous shutdown signals, two reported lost runner communication, and two aggregate gates failed downstream. One failed-job retry on unchanged source passes. The underlying host shutdown cause remains unconfirmed; no code fix or extra retry is claimed. [CI run and attempts](https://github.com/paperclipai/paperclip/actions/runs/40819408528). No Ponytail verdict is present. ## Risks - A missing future producer can delay dormant-history discovery until restart. The audit names current producers. Independent SQL writers and other server processes remain outside the process-local signal contract. - Physical ownership and uncertain operations retain retries. Time alone does not authorize taking workspace ownership. No provider turn or unsafe archive retry is added. - The recovery owner remains authoritative if state changes after discovery. Terminal UI status does not authorize resource destruction or accepted-result replay. - The partial index adds matching-row write/storage cost. It builds concurrently through the existing restart-safe migration runner. - This proves the changed worker becomes quiet. It does not prove host suspension, external wake coverage, or fleet cost savings. Other sleep migrations remain. This slice still requires fresh current-head CI and reviews. ## Model Used OpenAI Codex, based on GPT-6, with reasoning and code-execution tools. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 00:13:22 -05:00
DottaandPaperclip	51c3f8924c	fix(server): schedule cancelled workspace preservation (#17677) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A cancelled native run may still owe preservation of prepared workspace files. > - The existing recovery receipt and source lease record that obligation. > - The orphan reaper still searches for these cancellations on every pass. > - This PR registers committed cancellations with the shared deadline worker. > - The existing retention owner keeps file pinning and ownership checks. > - Idle and completed history no longer causes this preservation scan. ## Linked Issues or Issue Description Refs #17674 and #17145. Stopped instruction collection is merged into master. It owns file-preservation discovery only. The existing environment cleanup worker continues to own stop-only teardown. **What needs to be improved?** `preserveCancelledNativeWorkspaces` runs from every orphan-reaper pass. It queries even when no cancelled native workspace needs a file-recovery hold. Current foreground Stop can also defer retention after a lock or database failure, so startup-only recovery would be insufficient. **Why is this improvement important?** Empty queries keep idle tenants active. Removing them must not lose user files, weaken Stop ownership, replay cancelled work, or turn a retained allocation into destructive cleanup. **Proposed solution** Register the existing durable obligation from committed run-status, cancellation-acknowledgement and workspace-reference writes. Reconstruct it at startup. Keep retries while actual preservation remains unresolved. Disarm after the existing owner records the hold. Related-PR search and ROADMAP.md review found no duplicate of this scheduling migration. ## What Changed - Add a company/run-scoped preservation worker with 100-row keyset reconstruction and fair batches of five. - Separate pending file debt from immediate coordinator/lease readiness. Retain a 30-second retry for known unfinished preservation and shared failure backoff for errors. - Signal the existing status, acknowledgement, workspace-reference, deletion and settings transactions. Add no table or independent timer. - Keep exact source checks, no-workspace contracts, task/coordinator/run lock order, retention receipt and no-replay recovery card with their owners. - Recheck cancellation acknowledgement, run status and coordinator phase under the retention locks. Preserve the existing foreground preterminal API. - Keep startup/manual recovery; skip only this duty in periodic reaping. Preserve drain/standby and shutdown behavior. - Add concurrent, replay-safe partial index migration 0409 and retain the five newest snapshots. - Add the producer and lifecycle audit in `doc/plans/2026-10-25-cancelled-workspace-preservation-sleep-audit.md`. ## Verification - Current source `21b46a8e4293ca670235117458b984493a181759`: all 1472 cases in 16 complete files pass without skips. The owner passes 84 cases in 14.93 seconds. Fifteen consumer files pass 1388 cases in 133.46 seconds, including all 968 native-session cases and Stop, dispatch, restart, workspace-sync and upgrade-readiness coverage. - Twelve separate fresh PostgreSQL 17 databases migrated through 0409 before setup. The embedded-only lifecycle fixture temporarily used its own external database on this IPC-exhausted host and was restored byte for byte. Assertions and timing limits stayed unchanged. This qualifies external-database behavior; it does not claim an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-head migration order pass. The complete original runtime/schema patch and unrelated current-parent blobs are preserved. Original concurrent index SQL, all earlier executable history and retained snapshot bytes are unchanged; five snapshots 0405 through 0409 remain. - A populated 0408-to0409 upgrade preserves all 407 prior receipts and 38 companies. Exactly one receipt is appended; replay adds none. The preservation index is valid and ready on fresh and upgraded databases. - Actual parent `82529741418e138d9d623bf13d2fa22c1ff79d36` is merged through a normal ancestry reconciliation. Its complete tree equals the reviewed parent. The entire accepted child tree and all source hashes are preserved. The audit retains earlier failures, qualifications and per-file counts. Published head `b34f0589847f33754ecde9413fd36bf007353d2c` has Greptile 5/5 and all 57 checks passing or skipped. No Ponytail review is present. These checks do not establish deployed host/database suspension or Cloud alarm delivery. ## Risks - Signals cover instrumented single-process writers. Independent SQL writers and additional replicas need integration. - Status signals may inspect a run without relevant file debt once. They do not create recurring work. Discovery and memory still scale with actual candidates. - Missing ownership or stop evidence remains pending. Active retries are intentional while a concrete file obligation is unresolved. - Retry timing is process-local; startup reconstructs durable debt. The worker does not overwrite another native-finalizer deadline. - Admitted database operations may finish after drain begins. Shutdown joins them. Provider teardown remains with the existing cleanup owner. - The partial index adds storage and write cost. Its concurrent migration uses the existing interrupted-index repair path. - Cloud alarm coverage, sleep eligibility and host suspension are unchanged. This proves worker behavior, not fleet savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 22:27:11 -05:00