mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
0b8ba1ac92931512bea6dc3424b34eb41b414085
55
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
0b20ea6fe9 |
fix(accounting): keep budget admission and usage handoff alive under run-row updates (#20484)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Before a run starts provider work, Paperclip reserves its budget and binds a usage recorder. Both write the run's `heartbeat_runs` row. > - Run-log appends, output progress and lifecycle events update that same row while the run prepares and streams. > - #20385 fixed this race for the standalone usage-receipt write only. Budget admission and the recorder handoff still take a repeatable-read snapshot before they lock the run row. > - Each concurrent row update makes that snapshot stale. Four retries run out with SQLSTATE `40001`, and the run ends `adapter_failed` with raw SQL as its error. > - This pull request applies the #20385 pattern (READ COMMITTED under the existing locks) to those writers and to the final dispatch gate. It also gives exhausted retries a bounded cause. > - The benefit is that busy runs start and account normally, and a remaining failure names its cause. ## Linked Issues or Issue Description Refs #20385. Builds on #20483 (bounded failure cause and `withoutSqlStatementText`). **What happened?** The restart-chaos L1 workload with `--db-pressure 4` (harness writers that update each running run row, one row per statement, like a fast stream) failed on master. In one wave, 0 of 4 tasks passed. Every first run failed in `reserveRunBudget` before spawn. In an earlier wave, a run failed in `createRunUsageRecorder`. The stack in both cases is `could not serialize access due to concurrent update` (`40001`) on the run-row `FOR UPDATE`. The board `runError` showed the full `select ... from heartbeat_runs ... for update` statement. The server log line `heartbeat execution failed` dropped the database cause. Realism: the harness writers are synthetic, but the product has the same writers on the same row. `appendHeartbeatRunEvent` increments `next_event_seq`, output progress updates `last_output_*`, and lifecycle events land while a run prepares. On a hosted database the window between snapshot and row lock is wider than locally, so a busy run reaches the same race without the harness. **Expected behavior** Budget admission and the recorder handoff commit while other writers update the run row. Budget invariants stay intact: one reservation per run, a reached hard stop still refuses work, and committed holds count against later admissions. Usage receipts are journaled once. If retries still run out, the run shows a bounded cause without SQL, and the server log records the SQLSTATE. **Steps to reproduce** Run `server/src/services/run-accounting-contention.test.ts` against the parent commit. The deterministic cases commit a real `appendHeartbeatRunEvent` after the writer's transaction checks the company and before it locks the run. Six of the eight cases fail with `40001`. ## What Changed - `reserveRunBudget` runs at READ COMMITTED. All mutable reads follow the shared admission fences, the agent `FOR SHARE` lock and the run-row `FOR UPDATE`. Exposure stays one SQL statement (hold-to-ledger atomic). The `budget_reservations_run_idx` unique index and the run lock still prevent a second hold. - The final dispatch gate's accounting phase (`dispatchResolvedInteractionIfCurrent` through `withIssueThenRunLocks` in `run-dispatch/adapters/postgres.ts`) runs at READ COMMITTED. Its ownership phase already read the same staleness facts at READ COMMITTED under the same issue and run locks. Retry promotion and stale-run cancellation keep their repeatable-read snapshot. The first L1 proof wave found this writer exhausting `40001` once the other three were fixed. - `createRunUsageRecorder` runs its receipt-source handoff and its capture-failure fence at READ COMMITTED. Both lock the run row first, the same as `persistUsageReceipt` after #20385. - `withOperationAccountingTransaction` raises `AccountingContentionError` when retries run out. It has a bounded message, `code` keeps the SQLSTATE (`40001`/`40P01`), `cause` keeps the database error, and a warning logs both. - `failRun` shows that bounded message. The code stays `adapter_failed`, so chat and recovery retry paths still apply. #20483's `failureCause` records `40001`. All other SQL text handling comes from #20483 (`withoutSqlStatementText`), unchanged here. - `doc/DATABASE.md` documents the isolation choice and the failure contract. ### Inventory of REPEATABLE READ / SERIALIZABLE transactions that write run-scoped rows | Writer | Status | Notes | | --- | --- | --- | | `persistUsageReceipt` (`usage-receipts.ts`) | Fixed in #20385 | READ COMMITTED under the run lock. | | `reserveRunBudget` (`budget-reservations.ts`) | Fixed here | Seen in waves (`budget-reservations.ts:90`). | | `createRunUsageRecorder` handoff (`usage-receipts.ts`) | Fixed here | Seen in waves (`usage-receipts.ts:572`). | | `createRunUsageRecorder` capture-failure fence (`persistFailure`) | Fixed here | Same shape: run lock first, runs mid-stream. | | Native finalizer `claimCoordinator` (`native-run-finalizer.ts`) | Safe | A default READ COMMITTED transaction with `FOR UPDATE` on the coordinator row. It cannot raise `40001`. The wave errors there were `native_finalization_missing`, a symptom of the failed reservation (no coordinator existed yet). | | Final dispatch gate, accounting phase (`dispatchResolvedInteractionIfCurrent`) | Fixed here | Seen in proof wave `w-20261110T075857-s20386` (runs `140ddb10`, `c028c6c4`, `fea0ee5a`). Its ownership phase already used READ COMMITTED under the same locks with the same staleness reads. | | Retry promotion and stale-run cancellation (`promoteOrCancelDueRetry`, `cancelStaleQueuedRun`, same lock helper) | Follow-up | Kept at repeatable read: `loadGateFacts` combines wakes, comments, dependencies, tree holds and repair state in one decision. They act on queued or scheduled runs, not streaming ones. Not seen failing in waves. If retries run out, they now fail with the bounded cause. | | Run settlement `accountRunCost` (`run-cost-accounting.ts`) | Follow-up | Seen with `40001` in a wave log. Leaves the run pending; `reconcileRunCosts` retries it durably. Multi-aggregate (coordinator, receipts, images, closure). | | Tool cost events (`paperclip-runner-tool-authority.ts`, `costs.ts`) | Safe | Insert `cost_events` only; the run row is referenced, not locked or updated. | | Budget enforcement and evaluation (`budgets.ts`), billing reconciliation, accounting-integrity repair | Safe | Lock policy, invoice or company rows, not a streaming run row. Run reads are plain selects. | | Browser use, decisions, prompt-injection, fast responses | Safe | Write their own invocation/reservation rows, not `heartbeat_runs`. | | Read-only RR snapshots (`withAccountingReadSnapshot`, idle-sleep, tool gateway, customer success, `routes/decisions.ts`) | Safe | Read only, or no run-scoped writes. | ## Verification - Red on the parent commit, green on this branch: `server/src/services/run-accounting-contention.test.ts` (real embedded PostgreSQL). - `reserveRunBudget > reserves once when run events commit between its company check and its run lock` (red: `40001`) - `reserveRunBudget > reserves every run while back-to-back writers update the run row` (red: `40001`) - `reserveRunBudget > counts every committed hold, so successive admissions never exceed the budget` (red) - `reserveRunBudget > still refuses a run once the hard stop is reached` (red: `40001` instead of `409`) - `createRunUsageRecorder > binds its receipt source when run events commit between its company check and its run lock` (red: `40001`; green asserts the spooled receipt is journaled exactly once) - `createRunUsageRecorder > binds while back-to-back writers update the run row` (red: `40001`) - `reserveRunBudget > never creates a second reservation for one run when two dispatches race` (invariant; passes before and after) - `final dispatch gate > admits and hands off a reserved run when run events commit between its company check and its run lock` (red: `AccountingContentionError` after 4 × `40001`) - `createRunUsageRecorder > persists its capture-failure fence once when run events commit between its company check and its run lock` - `final dispatch gate > admits while back-to-back writers update the run row` (red: `40001`) - `exhausted retries > fail with a bounded, SQL-free cause that keeps the SQLSTATE` (new contract) - `run-completion.test.ts > saves a bounded cause and keeps the retryable generic code when accounting retries are exhausted` (saved code, message and `failureCause`, and log fields, through `failRun`) - Related suites pass: `usage-receipts-read`, `admission-independent-of-cleanup`, `budget-live-attempts`, `idle-accounting-work`, `budget-enforcement-scoping`, `closed-native-receipt-accounting`, `budget-evaluation-work`, `run-dispatch/adapters/postgres`, `heartbeat/run-completion` (all 126 run-completion cases pass; its `afterAll` embedded-PostgreSQL teardown exceeded 30 s on this loaded host, the known local flake). - `pnpm --filter @paperclipai/server typecheck` passes. - Latency of `reserveRunBudget` on the normal path (no contention, embedded PostgreSQL, 350 measured reservations per run, 3 runs each): before p50 14.7 / 15.0 / 18.6 ms, p95 25.6 / 31.4 / 39.4 ms; after p50 14.2 / 14.0 / 16.6 ms, p95 29.7 / 24.0 / 29.3 ms. No added latency: the statements are the same, only the isolation level changes. Under contention it removes the retry backoff. - Live restart-chaos L1 waves (`--seed 20386 --db-pressure 4`, codex and claude, K1): - Master before this change: `w-20261110T050818-s20386`, 0/4. Every first run failed in `reserveRunBudget` with `40001`. - Head `90530789b0` (first three writers fixed): `w-20261110T075857-s20386`, 2/4. Both codex tasks passed. Both claude failures (t02, t03) trace to the dispatch gate, now fixed. Those runs already showed the bounded cause, with no SQL on the board. The failure was logged with `sqlState` 40001. No `reserveRunBudget` or recorder `40001` remained. - Head `d5ed21926f` (dispatch gate fixed): `w-20261110T090531-s20386`. All 4 tasks reached done with no retry and R1–R6 passed. There were no failed runs and zero `40001` in the server log, under 310k pressure updates. Each task still failed the T-oracle `company_spend_settled` with `pending_runs=1`. That pending run is t02's continuation run `601ae80a`, cancelled before provider work as `issue_terminal_status`, whose accounting debt was not yet settled when graded about 15 s later. It is not a contention failure. Caveat: I edited the worktree during this wave (`worktree_changed_mid_wave`), so a clean wave on the final head follows. - Final head `9f1c480717`, clean worktree: `w-20261110T094417-s20386`. All 4 tasks reached done with no retry, and R1–R6 passed for each. There were no failed runs and zero serialization failures in the server log, under about 523k pressure updates. Each task still fails only the T-oracle `company_spend_settled` (`pending_runs=1`). It is the same cancelled pre-provider continuation run (`issue_terminal_status`), whose accounting debt was still unsettled at grading. That is a separate settlement-timing follow-up, not database contention. Master was 0/4 on the same seed. ## Risks - READ COMMITTED gives each statement its own snapshot. Admission reads policy, scope and exposure in separate statements. The shared company/project fences and the agent share lock exclude concurrent pause and policy edits, and exposure stays one statement, so each read is at least as fresh as the old snapshot. - Concurrent admissions of different runs stay best-effort, as `doc/SPEC-implementation.md` already states ("concurrent starts can exceed the same observed balance"). This change does not widen that window: each statement snapshot is newer than the old transaction snapshot. It does not add a company mutex. - A run that fails after exhausted retries now records `accounting_contention` instead of `adapter_failed`. Retry rules that key on `adapter_failed` spawn-like messages did not match this failure before, so no automatic retry changes. - No schema, migration or persisted-format change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with shell, file-edit and code-execution tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4a2eb920cf |
fix(storage): wait out a store outage on a run's critical path instead of failing it (#20299)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A cloud instance keeps run logs and evidence in object storage, so a run touches the store as soon as it starts > - In storage chaos wave `w-20261109T145628-s9902` (ST1: 75–176 s windows where every S3 call returns 503), only 1 of 8 tasks finished. A run starting or resuming during an outage failed within a second as `adapter_failed`, generic recovery burned its attempts seconds apart, the task was escalated to the board, and the user saw the raw S3 error > - This pull request treats a transient storage error as an infrastructure wait: the run is retried on a minute-scale schedule, rapid recovery is suppressed, the board shows it as working, and the run error says storage was unavailable > - The benefit is that a store outage of several minutes delays work instead of failing and stranding it ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (master `610b54f551`, `--storage s3`, ST1). **Where the run fails.** At run start or resume, `run-output` `begin` opens the run log: `getRunLogStore().begin` → object run log store → object byte journal `claim` → S3 conditional PUT. The journal retries only 3 times within about 300 ms, so a 503 escapes and fails the run as `adapter_failed`. The async stack ends in the SDK middleware with no app frame. `log_store` stays null on the failed run, which pins the claim. **What followed.** - `failRun` released the issue into generic immediate recovery. Recovery retried 1–2 s apart inside the outage (t01: 14:58:33, :35, :59, 14:59:01), then stopped and opened a board-owned `stranded_assigned_issue` action. These are the open recovery actions on CHA-1, CHA-4, CHA-7 and CHA-8; a later Retry hit the outage again and escalated the same way. - The run error was the raw S3 text ("Injected storage outage"). **Slow store (ST2, `w-20261109T151946-s9903`).** With about 9.7 s of added latency per call, the same claim hit the journal's 15 s per-operation deadline (`AbortSignal.timeout(15_000)`). The run failed after about 36 s with `AbortError: Request aborted` and burned 3 runs the same way; 4 of 8 tasks needed a Retry. t02/t03 (Retry parked with `process_identity_missing`) are a separate native-recovery class and are handled in a follow-up. **Daytona storage chaos ST1 (`w-20261109T161202-s9912`, 1/8):** with S3 returning 503 for 72 to 154 s at a time: - Workspace restore failed at once (`remote_workspace_restore_failed: step=snapshot_download; storage=provider_error`), three times in 40 s, then `setup_failed`. It left an active recovery action behind. - Runs reattached after a restart died within 2 s on the run-log claim, then became reconciliation-required. - Setup failed on other storage reads. - 6 of 7 failures needed a UI Retry. So the rule here is one class rule: a transient store error on a run's critical path waits, it never fails the run. ## What Changed - New `server/src/storage/storage-unavailable.ts`: - `isStorageUnavailableError` reads the recorded storage failure diagnostics first, then the SDK error shape. Unavailable means 429/500/502/503/504, the S3 throttling and timeout names, network codes (`ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, `EPIPE`, `EAI_AGAIN`, `ENOTFOUND`), or an `AbortError` whose cause is a `TimeoutError` (a per-operation deadline). 403, 404 and 412 are not outages, and neither is an abort caused by the run's own Stop. - It defines the `storage_unavailable` code, the `storage_unavailable_wait` retry reason, the retry policy (60 s × 10 attempts, about 10 minutes) and the user-facing message. - `server/src/services/heartbeat/run-completion.ts`: - `failRun` classifies such a failure as `storage_unavailable` with the plain message. - Unless native recovery owns the run, it schedules the storage wait with `scheduleBoundedRetryForRun`, skips the interaction-continuation retry, and releases the issue with `suppressImmediateRecovery`. - `failRunSetup` uses the same policy for this code. - `server/src/services/heartbeat/retries.ts`: `storage_unavailable` joins the transient infrastructure codes. It gets its own retry budget and policy, like `runner_revision_mismatch`. - `server/src/services/task-interruption-presentation.ts`: `storage_unavailable_wait` is a resource wait, so the board shows the task as working with no Retry while it waits. - **One shared outage wait for the run's critical path** (`server/src/storage/storage-unavailable.ts`): - `retryThroughStorageOutage` retries a transient store failure with backoff (2 s doubling to 30 s) until that storage operation's outage has lasted 5 minutes, then gives the error back. The outermost wait of an operation records its first failure, and nested and parallel waits inside it share that start; no other request's success extends it. Shared probes and shared reads run under their own scope, so one run's Stop never cancels work another run waits on. A stop signal ends the wait, and non-outage errors are never retried. - Inside `waitOutStorageOutages`, every replayable S3 request waits instead of failing: get, head, delete, buffered and file puts, and multipart create, file parts and complete. A consumed upload stream is never resent, and requests outside the scope keep failing fast. - The scope covers run execution (`heartbeat.ts` `executeRun`: setup, workspace restore, run-log claim and the run itself), plus `loadWorkspaceSnapshotArchive` and `uploadWorkspaceSnapshot`, which copy-back and recovery workers use too. - The run-log claim retries as a whole, because the journal bounds each operation with a 15 s deadline. - Past the budget, the run takes the bounded `storage_unavailable` wait below. - The classifier follows the error's `cause` chain, so a restore failure wrapping an outage counts. `failRunSetup` checks storage before restore failures, so that case takes the storage wait instead of two quick retries and an escalation (the R6 leak). - `object-byte-journal.ts`: when its deadline expires, the journal rethrows the recorded storage error instead of a bare `TimeoutError`. ## Verification Red/green (each new test fails against master and passes with this PR): - `run-completion.test.ts` "waits a minute and suppresses rapid recovery when a run fails because storage is unavailable (storage chaos ST1)". An S3-shaped 503 records `storage_unavailable` with the plain message (no raw S3 text), schedules `STORAGE_UNAVAILABLE_RETRY`, releases with `suppressImmediateRecovery: true`, and skips the continuation retry. "keeps an ordinary adapter failure on the normal recovery path" guards the default. - `task-interruption-query.test.ts` "does not present an overdue storage_unavailable_wait wait as a failed attempt": working, `canRetry: false`. - `storage-unavailable.test.ts`: which errors count as an outage and which do not, including recorded diagnostics, a deadline-aborted call (ST2), and a Stop-aborted call that must not count. Static checks: - `tsc --noEmit` passes for `server`. Live verification: Phase 3 will rerun ST1 (seed 9902) with the MinIO fault proxy after merge. - `server/src/__tests__/storage-outage-wait.test.ts` runs against a local S3 that answers 503 for 3 s: - Inside the scope, a snapshot `getObject` and a run-log conditional PUT succeed once the store is back. - Outside it, a request fails fast. - A stop ends the wait. - Unit cases cover the budget, non-outage errors, and a `RemoteWorkspaceRestoreError` wrapping an outage. - `run-completion.test.ts`: a setup failure from a restore that wrapped an outage takes the storage wait (red before). - `object-byte-journal.test.ts`: a real `claim` whose head PUT hits its deadline is classified as a storage outage (red before). ## Risks - A storage outage now holds a task for up to about 10 minutes of automatic waits instead of escalating within seconds. A longer outage still ends in the existing board recovery. - A run whose failure is misclassified as storage-unavailable waits instead of using fast recovery. The classifier accepts only server-side or network failures from the storage SDK, never 4xx answers. - No schema change. - A run can now wait until a storage outage has lasted 5 minutes before it fails over to the bounded wait. It shows as working throughout, and Stop ends the wait. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
3cc1032f6a |
fix: deliver saved answers to dependency-blocked tasks (#20363)
## Thinking Path > - Paperclip lets people direct agent work through tasks and structured answers. > - Dependencies stop task execution until prerequisite work finishes. > - A person must still be able to answer or redirect the assignee of a blocked task. > - Ordinary comments already get a bounded answer/triage turn. A saved question response has no comment ID, so the same gates reject it. > - Later dispatch also requires an active task and tries to check it out. That conflicts with bounded interaction on a blocked task. > - This pull request verifies the stored human response at each gate and keeps the turn in bounded interaction mode. > - The assignee can receive the answer without starting dependency work or removing its blockers. ## Linked Issues or Issue Description Refs #19923. That change covers dependency waits and comment retries. This change covers stored interaction responses that have no comment ID. **What happened?** A parent task waits on a parked child. The assignee asked a setup question. After a person answers, the response stays queued. Sending that saved response is refused as `issue_dependencies_blocked`. Ordinary new messages can reach the same assignee. Passing admission alone is insufficient. The dispatch status check rejects a blocked task, and automatic checkout then tries to start work whose prerequisites are unfinished. **Expected behavior** Deliver the stored human answer or confirmation to its requesting assignee as a bounded answer/triage turn. Keep the task blocked and keep its dependency relationships. Preserve current-owner, pause, approval, budget, recovery, and execution gates. **Steps to reproduce** 1. Assign a parent task and make a backlog child block it. 2. Save a question on the parent and answer it as a person. 3. Dispatch its resolved-interaction wake, or send the saved answer from the task queue. 4. Observe that the answer does not reach the provider while an ordinary comment can run in answer/triage mode. **Paperclip version or commit** Reproduced on master at `653ab8b18f76`. This branch is rebased onto master at `0e3fdebea8`. ## What Changed - Verify the stored response by company, task, requesting agent, status, kind, human resolver, and resolution time. A context flag alone is insufficient. - Allow a verified response through dependency admission and claim in the existing bounded interaction mode. - Recheck that response during scheduled retry promotion and final provider dispatch. - Keep bounded responses from checking out the blocked task or setting it to `in_progress`, including when the last blocker finishes during preparation. - Preserve continuation classification through workspace and AI-account resource waits, so the final response and owner checks still run. - Correct the task-case API summary so the existing `case for task` discovery regression passes on the first search page. This fixes a pre-existing failure exposed by full CI, without changing search ranking or assertions. - Ignore obsolete Retry notices after newer conversation work, without acknowledging saved input or changing execution authority. - Document the response contract. Add real PostgreSQL and assembled-heartbeat regressions. ## Verification - Final commit `ba4b337cf1bb0236b7c0a742665b222ee894ff51`: [required CI passed, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/42706363120/attempts/2), including typecheck, production build, server/chat tests, Runner checks, and all 8 browser shards. PR status: 59 passing checks and 3 normal skips. - Red: answered questions and accepted/rejected confirmations return no admitted run before the fix. All three new positive cases fail at the dependency gate. A resource-wait retry also reproduces as refused before its gate is fixed. - Green: at delivery revision `9375a603e4`, focused heartbeat integration and classification passed 14 tests; retry scheduling and dispatch safety passed 241 tests. These use real PostgreSQL and controlled adapter and resource-availability boundaries. The wider initial dependency, native-continuation, and queue suites also passed before the final refinements. - Negative cases keep pending, missing, wrong-task, and agent-resolved responses blocked. Changing the response proof immediately before dispatch cancels the run without calling the provider. Policy controls retain the current assignee, task status, and pending approval checks. - Positive cases execute once, leave the parent blocked, preserve the dependency edge, and leave the child in backlog with no child execution. The same saved response survives a resource wait; removing its human proof refuses promotion. - The resource-wait provider-boundary and blocker-completion regressions also failed before their fixes. All 14 focused integration and classifier cases passed at delivery revision `9375a603e4`, including a changed response during retry preparation. - Red: a retained Retry for an older run makes a newer successful conversation display as failed. Green: all 78 database-backed query and interruption-presentation tests pass, including a later real failure and independent queued input controls. - Server layered typecheck and module-boundary checks passed at `11992aad0c7fbaf13e7548d71ee633560bd8bb6f`, before the one-line API summary correction. That CI passed build, typecheck and every browser shard, but exposed the pre-existing case-search assertion and a report-assets test timeout. Final-head CI passed as linked above. - Fresh Greptile review is 5/5 on final commit `ba4b337cf1bb0236b7c0a742665b222ee894ff51`, with no unresolved threads. - The existing API discovery suite reproduced its first-page assertion before the summary correction; all 3 cases pass afterward. The timed-out report-assets privacy test passed locally in isolation and the full support job passed on the final-head rerun. Runner shutdowns interrupted the first attempt; affected jobs were rerun once without code changes. A separate onboarding timeout also passed on that rerun. - `pnpm verify --explain` requests broad CI because the execution contract is a global input. The final CI run supplied that coverage. - Separately recovered the reported staging conversation through an ordinary message and the existing steering control. The agent acknowledged the message and the saved-answer queue drained. The fresh Notebook action subsequently succeeded. This was on the previously deployed revision; it does not claim this code is deployed. ## Risks - This changes admission and dispatch. Each gate checks the persisted response; no caller-supplied marker grants the exception. - The bounded turn does not check out the task, resolve a dependency, or approve another action. Existing pause, ownership, budget, approval, and stopped-execution controls still apply. - Each additional lookup is one primary-key read and runs only for a response on a dependency-blocked path. There is no new polling worker. - No migration is required. Older queued contexts get the exception only when their stored human response still verifies. No prior run or action is replayed by this change. - This PR does not roll out the code to staging. ## Model Used OpenAI Codex, GPT-6 family, with repository tools, code execution, and browser inspection. The runtime does not expose the exact model ID, context-window limit, or reasoning-mode label; those values are not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
c49ab6b9ee |
feat(issues): wake the parent once when a child or blocker stalls on the board (#20098)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task delegates to children and waits on blockers. Its agent learns of progress only through wakes > - Today only completion (`issue_children_completed`) and resolution (`issue_blockers_resolved`) wake a waiting issue > - In restart chaos `w-20261108T181313-s9701` t05, a blocker stopped on a recovery action that needs the board, and its parent sat silently blocked for 50+ minutes > - This pull request wakes the waiting agent once per such stall, tells it what it can do, and shows the stall on the parent's page with the child's Retry > - The benefit is that a stuck child or blocker no longer silently strands its parent ## Linked Issues or Issue Description Refs #19831 (durable ownership plan; restart chaos campaign). Restart chaos wave `w-20261108T181313-s9701`, task t05 (M1, claude). **Problem** From the kept database: - CHA-8 `blocks` CHA-5. - CHA-8's board-owned recovery action (`native_session_retry_exhausted`) was created at 18:44:02.909 and resolved with outcome `blocked` at 18:44:08, when CHA-8 moved to `blocked` for the board. - CHA-5's only wake in the whole wave was its `issue_assigned` at 18:20:48. Its agent never heard about CHA-8. **Expected behavior** - When a child or blocker stalls on the board, the waiting agent is woken once. It can then re-delegate, escalate, or block with a reason. - The board sees the stall on the waiting task, with the child's Retry. ## What Changed - **Shared:** - `ISSUE_BLOCKER_STALLED_WAKE_REASON` (`issue_blocker_stalled`), `ISSUE_BLOCKER_STALL_RELATIONS`, and the types `IssueBlockerStallRelation` and `IssueBlockerStalledWakeContext`; - `Issue.stalledDependencies`. - **Server, `services/recovery/blocker-stall-wakeups.ts`:** - A stall is an issue that is not done or cancelled, whose latest recovery action needs the board. That means either open (`active`/`escalated`) and board-owned or escalated, or board-owned and resolved as `blocked` while the issue is still `blocked`. The second case is what t05 needed: recovery resolves its action that way when it hands the issue to the board. - Waiters are the parent, and every issue the stalled one `blocks`, when they are open, visible and agent-assigned. - Each waiter gets one wake per stall. The idempotency key `issue_blocker_stalled:{waiter}:{stalled}:{recoveryActionId}` is checked through the indexed `(company_id, payload->>'issueId')` path. A later stall has a new action id, so it wakes again. - Each wake writes activity `issue.blocker_stalled_wake`. - The pass runs from the recovery-action sweep (`reconcileActiveRecoveryActions`) and from the per-issue blocked sweep (`reconcileResolvedDependencyWakeBackstop`, which visits each blocked issue). No new worker. - A waiter's agent that cannot read the stalled issue (`issue:read`, which honors private visibility) is woken without its title, identifier or recovery details; the activity entry withholds them too. - **Database:** migration `0450` adds the partial unique index `agent_wakeup_requests_issue_blocker_stalled_uq` on `(company_id, idempotency_key)` for `issue_blocker_stalled:` keys in delivered or pending statuses, so two concurrent sweeps cannot both deliver a stall. It is built `CONCURRENTLY IF NOT EXISTS` outside a transaction (listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`). - **Queue:** `allowsDependencyBlockedWake` (in `run-dispatch/domain/wake-context.ts`) lets a stall wake run while the waiter's blockers are unresolved, at both dependency gates. Without it the wake was parked until the stalled blocker resolved, which is never. Pause holds are unchanged. - **Agent guidance:** the wake payload carries `blockerStall`, and `renderPaperclipWakePrompt` adds a "Stalled dependency" section. It names the task and its recovery action and the board's next action, then the three choices: re-delegate, escalate with a comment, or block with a reason naming the stalled task. - **Other reason lists:** `issue_blocker_stalled` joins the responsible-user wake reasons, the conversation execution-wake reasons (it never drives a chat turn) and the wake diagnostic's known reasons. - **Board surface:** - `GET /issues/:id` returns `stalledDependencies`, readable issues only (`readableIssueEdgeIds`). - `StalledDependencyNotice` shows on the issue's chat notices, next to `IssueBlockedNotice`: each stalled child or blocker, its board next action, a link, and the child's own Retry (`useTaskRetry`), offered only when the child's interruption allows it. ## Verification Red/green: - `server/src/__tests__/issue-blocker-stall-wakeups.test.ts` (14 tests). - On master: no `issue_blocker_stalled` wake is raised. - With this PR: the parent is woken once per stall, and re-woken after a new stall. A blocker's dependent is woken. - The t05 shape (an action resolved as `blocked`) wakes once, and nothing more after a Retry. - No wake for an agent-owned action, a done, unassigned or hidden waiter, or an action resolved another way. - `listStalledDependencies` lists children and blockers. - `heartbeat-dependency-scheduling.test.ts` "admits an issue_blocker_stalled wake while a blocker is unresolved". On master the wake was parked (`null`); once admitted, the run-start gate cancelled it. With this PR it runs with `dependencyBlockedInteraction`. The file: 14 passed. - `packages/adapter-utils` `server-utils.test.ts`: the new prompt test failed on master (no `blockerStall`). The file: 141 passed. - `ui/src/components/StalledDependencyNotice.test.tsx`: 3 passed. `IssueChatThread.test.tsx`: 109 passed. Regression and static checks: - Nearby server suites (recovery durable wait, dependency wake routes, recovery actions, run-preparation): 149 passed. - `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`. - `pnpm check:token-gates` is clean. Review round 1 (Greptile, Superagent), each with a regression: - A backlog waiter is never woken. The waiter's status and assignee are guarded at enqueue. - One stall per run: no coalescing, so a merged run cannot keep only the last stall's context. - Only a delivered or pending wake ends a stall. A refused one (wake-on-demand off) is retried after 10 minutes, and an agent that takes no wakes is skipped without writing a row. - Company-wide passes resume after the last stall a capped pass reached. The per-issue sweep also checks the issue's own children and blockers, which covers an unassigned stalled blocker. - The dependency gates no longer trust the reason string. `verifyBlockerStallWake` requires a system request, the stalled issue's current stall action, and a real child or blocker edge. An agent-requested wake with the same context is parked (tested). - The notice shows a refused Retry's reason, and a lost response as "Taking longer than usual…", with Retry still available (tested). Review round 2 (Greptile), each with a regression: - Race: two passes claiming one stall leave one delivered row (unique index; the loser's insert is skipped). - The company recovery sweep selects companies whose only stall is a resolved-`blocked` board action, so an unassigned stalled blocker with a todo waiter is reached. - Private stalled issue: the waiter's wake and activity carry no title, identifier or recovery details. - `GET /issues/:id` always returns `stalledDependencies`, so a cleared stall replaces the cached list. - The notice shows a failed child read with a Reload button (tested). - `pnpm --filter @paperclipai/db check:migrations` passes. Review round 3 (Greptile), each with a regression: - A stall wake that fails to enqueue is retried: the recovery-action pass reports `blockerStallWakeFailures` and the dependency pass counts it in `enqueueFailed`, so the worker throws and retries at its backoff (`issue-recovery-work-lifecycle.test.ts`). - The notice drops a stall once the child's refreshed status has moved on (`StalledDependencyNotice.test.tsx`, 7 tests). ## Risks - A deploy wakes the waiters of every issue that is stalled today, once each. That is intended (they are stranded now), and it's bounded at 200 stalls per pass. - A stall wake may run while blockers are unresolved, like a comment wake. It is bounded to one per stall, and pause holds still apply. - The agent skill (`skills/paperclip`, CODEOWNER-owned) does not yet list the new reason. The wake prompt carries the guidance. - One additive index, built concurrently; no persisted run state change. Existing rows cannot violate it before deploy, since no `issue_blocker_stalled:` keys exist yet. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7c56147bb3 |
fix(workspaces): a busy-workspace holder whose controller lease is held is live, however silent (#20023)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Two runs must never write one workspace at once. A run that finds a workspace held (a remote workspace's writer run, or a running holder of a shared local tree) waits behind it > - The holder counted as live only while it produced output: after 60 minutes of silence it was "stale" > - For the shared local tree, that meant the waiter went ahead while the silent holder might still be working, which is a second writer. > - For a remote workspace, the waiter failed instead of waiting. > - Silence is not proof a run stopped. A long tool call can be silent and alive > - The durable ownership plan proves ownership by identity or by a held fence, never by age > - This pull request treats a holder whose controller still renews its lease (the legacy controller lease or the native coordinator lease) as live, however silent. The 60-minute rule applies only once that fence lapses > - The benefit is that a silent but live holder is never overtaken. A dead holder is still settled by the run reapers, which wake the waiters ## Linked Issues or Issue Description Refs #19831 (plan, Phase 2 sections A and C, and Phase 3 E), #19838 (inventory: "Remote workspace writer lane", and the "Released only by TTL/age" list: "Remote writer lane: 60 min"). **What happened?** `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` (60 min of output silence) decided whether a running holder still held a workspace. This applied in four places: - the remote reservation deferral; - the parked busy-workspace wait; - `findSharedWorkspaceHolder`, which stopped counting the holder at all; - `findExecutionWorkspaceHolder`, which flagged it stale. **Expected behavior** A running holder whose controller lease is held is live by fence. Only a holder with no held fence is judged by the silence cutoff, which keeps the visible failure instead of parking forever. The check is database-only (`runOwnerFenceHeldSql`): no process probe on the workspace-preparation path before a turn starts. ## What Changed - `remote-workspace-writer-hold.ts`: new `runOwnerFenceHeldSql`. It is true while `heartbeat_runs.controller_lease_expires_at > now()` or the run's native coordinator lease is held. `RemoteWorkspaceWriter.ownerFenceHeld` is reported for a running writer. - `heartbeat/workspace-preparation.ts`: the reservation keeps deferring behind a silent writer whose fence is held. - `run-dispatch/adapters/workspace-busy-wait.ts`: a parked wait behind a running holder with a held fence stays parked and re-checks at the parked cadence. - `heartbeat/retries.ts`: `findSharedWorkspaceHolder` counts a silent holder whose fence is held. `findExecutionWorkspaceHolder` flags a holder stale only when it is silent and its fence lapsed. ## Verification - `heartbeat/workspace-preparation.test.ts`: 138 pass. The extended "waits for the live writer of an inherited remote … at reservation" (both modes) fails on `master` and passes here: a silent writer with a held lease defers, and once the lease lapses it is the 409 as before. - `heartbeat-workspace-busy.test.ts`: 41 pass. The new "still defers behind a silent holder whose controller renews its lease" fails on `master` and passes here. The existing "does not defer when the only holder has been silent past the staleness threshold" (no lease) still passes. - The server package typecheck passes. ## Risks - A silent holder whose controller is alive now blocks waiters for as long as it holds its lease. That is the safe direction. A hung but live run is the active-run silence watchdog's to handle, and a dead controller's lease lapses within a minute. - The extra predicate is an indexed lookup of the run's own row and its coordinator row. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
25273aa69b |
fix(onboarding): start the agent when the opening question is answered (#19886)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A new company starts with an onboarding first task. The task opens
with a greeting and an opening question card that the assignee agent
posts.
> - Onboarding does not wake the agent when it creates the task. The
user's answer must start the agent.
> - When the user clicks an answer, run dispatch cancels the agent's run
0.4 seconds later with `issue_not_in_progress`. Answer continuations
only start on `in_progress` or `in_review` tasks, and the onboarding
task is still `todo`.
> - The user sees nothing until an unrelated bounded retry starts the
agent about two minutes later. This is the first thing every new user
does.
> - This pull request lets an answer start a `todo` task when the task
has never started and no run asked the question.
> - The benefit is that the agent starts immediately after the first
answer. The protection for parked tasks does not change.
## Linked Issues or Issue Description
**What happened?**
I created a new company and clicked an option on the onboarding opening
question ("What would you like to do?"). The agent's run was cancelled
immediately with `issue_not_in_progress`: "Cancelled because
resolved-interaction continuation issue is no longer in_progress
(current status: todo) before the queued run could start". The task
stayed in `todo`. About two minutes later, a `transient_failure` bounded
retry of the cancelled run started the agent through the normal checkout
path.
**Expected behavior**
The agent starts as soon as the user answers the opening question.
**Steps to reproduce**
1. Create a new company and hire the first agent through onboarding.
2. Open the onboarding first task.
3. Click an option on the opening question card and submit.
4. Look at the agent's runs. The first run is `cancelled` with
`issue_not_in_progress`.
**Paperclip version or commit**
`master` at `86bc3db6938`. The same code is on `00ad9503fe3`.
**Relevant logs or output**
```
heartbeat_runs: status=cancelled error_code=issue_not_in_progress
"Cancelled because resolved-interaction continuation issue is no longer in_progress (current status: todo) before the queued run could start"
two minutes later: "Scheduled bounded retry 1/2" (retryReason transient_failure)
```
Related: #13068 added the onboarding opening question. #13318 made
answer continuations require `in_progress` or `in_review`.
## What Changed
- New `server/src/services/unstarted-task-question-answer.ts`.
`isUnstartedTaskQuestionAnswer` returns true only when all of these are
true:
- the interaction is an answered question (`ask_user_questions`, status
`answered`) on this task, and it has no `sourceRunId` (no run asked it);
- the task is `todo`;
- the task's `startedAt` is null.
It reads only server-owned rows. `startedAt` is never cleared, so a task
that started and was later moved back to `todo` does not match.
- `run-dispatch/adapters/postgres.ts`: the staleness facts include
`isUnstartedTaskQuestionAnswer`. The adapter reads it under the issue
lock, and only for a resolved-interaction continuation on a `todo` task.
- `run-dispatch/domain/policy.ts`: `decideQueuedRunStaleness` lets such
a continuation start from `todo`. The other answer continuations still
require `in_progress` or `in_review`.
- `services/heartbeat.ts`: the final continuation checkout adds `todo`
to the expected statuses only for the same case. The task then moves to
`in_progress` in the same guarded update.
## Verification
- `heartbeat-stale-queue-invalidation.test.ts` (embedded Postgres):
- New test: an answer to a question that no run asked, on a
never-started `todo` task, starts the assignee. The task moves to
`in_progress` under the run's execution lock. With the fix reverted,
this test fails with the reported error (`cancelled`,
`issue_not_in_progress`).
- New test: three regression cases stay cancelled with
`issue_not_in_progress`:
- an answer on a task that started and was moved back to `todo`;
- an answer to a question that an agent run asked;
- a rejected confirmation that no run asked.
Without the kind and status check, the confirmation case fails.
- `policy.test.ts`: the new fact allows `todo`. The same fact on a
`backlog` task is still cancelled.
- `pnpm --filter @paperclipai/server typecheck`: exit 0.
- `npx vitest run` on these suites: 8 files, 497 tests passed.
- stale-queue invalidation
- run-dispatch policy
- onboarding first-task routes
- onboarding first-task admission
- question-response delivery
- issue-thread interaction routes
- heartbeat retry scheduling
- legacy continuation authority
## Risks
- Low risk. The change only affects answer continuations on `todo` tasks
that have never started, where the question has no source run. All other
continuations behave as before.
- The exception is not limited to the onboarding card. Any answered
question that no run asked, on a never-started `todo` task, can now
start the assignee. This matches what a plain comment wake does on a
`todo` task. Confirmations and other interaction kinds keep the existing
rule. For example, rejecting a confirmation that no run asked still does
not start the task.
- The final checkout does one extra indexed read. It only runs for
resolved-interaction continuations on `todo` tasks.
- I did not find which component scheduled the two-minute
`transient_failure` retry. After this change, onboarding does not depend
on that retry.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs). Investigation, fix, and tests were
AI-assisted, with human direction.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
docs describe this behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
DottaandClaude Opus 5.5
|
8a34c1b62c |
fix(budgets): bound the accounting hold so one missing receipt cannot wedge a company (#19700)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Pending run accounting holds new company work until it settles (#19497, #19606, #19673) > - A run settles only with closure evidence: a verified local stop, or a terminated remote sandbox. Without that evidence it held the whole company forever. In restart-chaos storm 4, a single run held 35 tasks for hours > - Two more shapes kept the hold alive. A run whose result had already been accepted never counted as closed. A recovery that already owned its held reservation was refused by other tasks' recovering accounting, so concurrent recoveries waited on each other > - This pull request bounds the hold, closes accepted-result runs on a verified stop, and lets an owned recovery finish > - The benefit is that no single missing receipt can wedge a company, while genuinely unknown spend still surfaces under the blocking policy ## Linked Issues or Issue Description Follow-up to #19497, #19606 and #19673. **What happened?** - Storm 4: run `f879895f` timed out in `session.open`. Its local stop was never verified (its process identity is still recorded), recovery ended in `terminal_failure`, and its receipt never became ready. From then on, every new-work wake got 409 "New work must wait while a native run recovers its unfinished accounting". That was 93 refusals, 35 blocked tasks, and no Retry. - Storm 4: run `6d3c76bf` was resuming as the recovery owner with its own held reservation. `reserveRunBudget` refused it because other tasks' runs were recovering. - Wave s7401: run `0a3a849d` failed after its result was accepted, with a verified local stop. It had 10 receipts across 2 sources, all incomplete with 0 tokens. It stayed "completed runs await accounting" because closure only covered coordinators in terminal failure with no result. **What did you expect?** A run without closure evidence holds new work only for a bounded time, then settles visibly. An accepted-result run with a verified stop closes like a terminal failure. A recovery that already owns admitted work can finish. **Steps to reproduce** 1. Restart the control plane during several native turns so that one run's local stop is never verified. 2. Observe the run stay "recovering" and every new wake refused indefinitely. ## What Changed - `run-cost-accounting.ts`: - A coordinator counts as settled when it ended in terminal failure without a result, or when it has an accepted result. The verified-stop and remote-termination closures use that rule, which covers the s7401 shape. - A native run whose provider never started settles at once, like a pre-provider failure, with no charge and its reservation released. Its recovery has settled, no usage receipt was ever journaled (a provider turn journals one as soon as it starts), and its local process never launched or was verified stopped (wave s7403, run 3c1d7b70, `session.open` timeout). - New bound `ACCOUNTING_CLOSURE_TIMEOUT_MS`: the remote runner reconnect grace (7 minutes) plus 3 minutes. A terminal native run with a settled coordinator and no closure evidence settles at the bound: - its known usage is recorded as unpriced (or estimated from the rate card), with provenance evidence "closure evidence never arrived…"; - it is marked `accountingClosureTimedOut`; - a `cost.accounting_closure_timed_out` activity entry names the run. - The reconciler selects these runs, and `readCostAccountingWork` schedules a wake at the bound. - `budget-exposure.ts`: a timed-out closure's final charge is exempt from none of the unpriced-block exemptions. Under the `block` policy it raises the normal budget notice ("recorded usage has no reliable price") instead of a silent hold. - `budgets.ts`, `budget-reservations.ts`, `run-dispatch/adapters/postgres.ts`: when a validated recovery owner continues admitted work, admission and both dispatch rechecks apply only hard stops and pauses. Pending and recovering accounting holds still gate new work. - `doc/SPEC-implementation.md`: documents the bound, the accepted-result closure and the provenance. ## Verification - `closed-native-receipt-accounting.test.ts` (embedded Postgres). All of these fail on master and pass here: - s7401 shape (accepted result, verified stop, all receipts incomplete with 0 tokens): settles and clears admission. - A run without closure evidence: held before the bound with a scheduled wake, then settled with the "closure evidence never arrived" provenance, the `accountingClosureTimedOut` flag and an activity entry. Admission clears under `allow`. - Storm 4 `f879895f` shape (terminal failure, recorded process identity, no stop event): held, then released after the bound. - The blocking policy still blocks on a timed-out closure. - s7403 shape (no receipts, model unknown): a run whose process never launched, or was verified stopped, settles at once with no charge and its reservation released. With an unverified process it waits for the bound. - Storm 4 `6d3c76bf` shape: a recovery owner with its held reservation reserves past another task's recovering accounting, while new work for another task still waits. - Existing suites pass locally: budget-enforcement-scoping, budget-enforcement-worker, native-stop-accounting, cost-data-quality, heartbeat-cost-accounting, run-cost-accounting, budgets-service. Server typecheck passes. - The other storm 4 terminal-failure runs (`2e477ae9`, `e4d2e14d`, `6d3c76bf`, `7d21a44c`, `4bc49df0`, `1d85fbd9`) were already settled by #19673 in that database (`cost_accounting_pending=false`) and were not holding admission. `f879895f` alone held it. ## Risks - After the bound, a run whose provider might still be running settles as unpriced. Under the `block` policy that raises a hard-stop budget notice, by design. Under `allow`, work continues and a late receipt still reconciles. - A recovery owner skips accounting holds, so concurrent recoveries can spend while other tasks' accounting is unknown. Hard stops still apply, consistent with best-effort budget enforcement. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
51a65af9b8 |
fix(recovery): continue preserved preparation after process loss (#19464)
## Thinking Path > - Paperclip manages AI agents and their execution state. > - A server restart can stop workspace preparation before provider execution. > - Workspace recovery must preserve files before the task can continue. > - Recovery handled interrupted preparations but skipped failed process-loss records with the same no-provider certificate. > - A withheld first retry could also hide the remaining retry budget. > - This change resumes certified preparations through normal admission after file recovery. ## Linked Issues or Issue Description **What happened?** A process-loss attempt remained failed after workspace recovery completed. Its first retry had been cancelled before dispatch by the workspace hold. The task had no running attempt or automatic continuation. **Expected behavior** Continue once after the server proves that provider work did not start, file preservation completed, and the exact leases stopped. Retain the existing failure budget and all task admission gates. **Steps to reproduce** 1. Stop a Runner preparation before native runtime selection. Record the restart recovery bootstrap certificate. 2. Retain the workspace and withhold the first automatic retry before dispatch. 3. Complete file preservation and restart the recovery worker. 4. Observe that the old code schedules no successor. The regression test reproduces this with PostgreSQL and real filesystem operations. **Paperclip version or commit** Reproduced against master at `2f35eb1080`. Related prior work: #19422 continues saved user input after verified copy-back. This change handles automatic continuation of certified process-loss preparation. ## What Changed - Accept failed process-loss preparation only with the durable no-provider bootstrap certificate and existing execution, lease and file checks. - Discover and settle the recovered source through the existing preparation recovery path. - Continue from a verified withheld retry without rewriting its history or resetting its failure count. Adopt it only as the sole retry child; multiple-child history reuses the newest existing child deterministically. - Exercise automatic dispatch, preserved-file readback, one provider turn, accepted result and task completion in the local native-session fixture. - Keep a durable continuation receipt after file-hold settlement until a successor is recorded. Known gates wait for existing eligibility events. Unresolved delivery keeps its shared recovery deadline. - Document the recovery and retry-budget rules. ## Verification - Red: both new PostgreSQL/filesystem regressions failed with zero scheduled successors where one was required. - Green: 54 focused recovery, automatic-dispatch, interruption, multiple-child history and admission safety cases pass on `13c27c2c2f`. The full PostgreSQL admission adapter suite passes all 83 tests; server source declaration typecheck also passes. Local environment: Darwin arm64, Node 26.4.0, PostgreSQL 18.1, Vitest 5.0.3. - Additional red/green: interruption after committed settlement, pause/resume, failed enqueue after cursor advancement, and absence of an earlier retry all failed before their repairs. Restart before enqueue, restart after the successor claim, and pause/resume now preserve one continuation. A rolled-back eligibility change cannot start execution. Deferred and delivered records have no retry deadline. - Local acceptance uses the real heartbeat dispatcher, isolated files and PostgreSQL, with a simulated native provider. It reads the preserved file, writes continued work, accepts the structured result and completes the task once. - Negative cases retain no-certificate, provider-evidence, live-owner, Stop, missing-stop-receipt, wrong-snapshot, human-block, pause, reassignment, approval and exhaustion fences. - CI fixture regression: 4 error-propagation/concurrent-resolution cases pass after adding the new obligation read to their transaction stubs; existing assertions remain unchanged. - Existing idle-drain and notification contract regressions: 4 tests pass after restricting recovery wakes to actual durable obligations. Late approvals block scheduled, queued and final dispatch. - Additional existing retry-scheduling suite: 62 cases pass locally. The pre-provider quota case exceeds its existing 5-second terminal wait on this loaded Darwin host, including when tested alone and with the preceding retry implementation. It remains a local timing limitation; the assertion and timeout were not changed. All Linux CI server shards pass on the combined head, including this unchanged suite. - Existing run-control, legacy-recovery and shared execution-control lifecycle suites: 128 tests pass with the review repair. Required CI supplies final-revision broad coverage. - Real-Codex acceptance was not run: automatic approval review rejected authenticated provider execution because it could incur cost/external effects and required explicit approval. No provider call ran; the controlled-provider acceptance covers the causal restart and durable scheduling behavior. - Hosted acceptance and recovery of existing tasks require rollout. They have not been claimed from the local provider fixture. - Final-head review: Greptile 5/5 on `13c27c2c2f`, no outstanding findings. Another contributor added the multiple-child history safety repair in `13c27c2c2f` while CI was running. The delta was inspected and the 54 focused recovery cases rerun successfully. Required CI passes on that exact combined head: https://github.com/paperclipai/paperclip/actions/runs/42241794918. All 55 successful checks include aggregate `ci / verify`, typecheck, build, canary, all server and E2E shards, and Runner package/Rust/static checks. The separate Paperclip Review service check remains queued waiting for its assigned agent; no validated assessment has been returned. ## Risks A malformed preparation certificate or a false no-execution receipt could permit duplicate work. The existing checks still reject provider execution records, native coordinators, live controllers, unsettled leases and unverified files. The pending continuation must survive a restart without polling known admission blockers. A withheld retry retains its original row and consumes its remaining failure budget. Normal admission checks current ownership, approvals, pause and budgets. ## Model Used OpenAI GPT-6 family via Codex. The exact model ID and context window are unavailable in session metadata. Used reasoning, code editing and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal task id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
7e3caa7bac |
fix(tree-control): resume work a pause gate refused after release (#19421)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The board can pause an issue tree. A pause stops work. Resume or
release must continue that work once.
> - PR #19394 made release re-dispatch a run that a pause Stop
cancelled.
> - That rule covers only a run that an operator Stop ended and that the
hold snapshot named.
> - A scheduled retry that comes due during a pause is not stopped. The
dispatch gate cancels it with `issue_paused`. The release did not see it
as owed work, so the task stalled with no run.
> - This pull request makes every gate that refuses work for a pause
record the covering hold. Release then re-dispatches that work exactly
once.
> - The benefit is that a paused task with a parked retry or a queued
run continues after release, with no manual wake.
## Linked Issues or Issue Description
Refs #19394
**What happened?**
A child task had a `workspace_busy` scheduled retry. The board paused
the parent tree. The retry came due during the pause. The
scheduled-retry gate cancelled it with `issue_paused` ("Scheduled retry
suppressed because the issue is held by an active subtree pause hold").
The board released the hold. Four minutes later the child was still
`in_progress` with no run, no execution blocker, and no new activity.
**Expected behavior**
Release (or `mode: "resume"`) re-dispatches the refused work exactly
once. No reconciliation and no user message are necessary.
Stranded-issue recovery does not add a second run.
**Steps to reproduce**
1. Give an assigned child task a parked `workspace_busy` scheduled
retry.
2. Pause the parent tree.
3. Let the retry come due while the pause is active. The gate cancels it
with `issue_paused`.
4. Release the pause. No run starts for the child.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
90b8190c33 |
fix(native-runtime): detect half-open runner ingress and project Stop for live runs (#19407)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs in a managed sandbox talk to the server through a provider-ingress WebSocket. The server opens this socket to the runner (`paperclip-runnerd`) through the sandbox preview proxy. > - A staging run ran a 20-minute foreground shell command that printed nothing. After the first second, no run events arrived. The run stayed "Working" for over 35 minutes. No watchdog acted, and the run projection said `canStop: false`. > - The server only detected a lost runner connection from a `close` or `error` event. It never sent pings. A half-open route, for example a proxy that drops the connection without a FIN, never emits `close`. The run then waits forever and the 7-minute reconnect grace never starts. > - A long silent tool call is legitimate. The fix must tell "quiet but alive" apart from "nothing alive". A protocol ping does that: runnerd answers every ping with a pong while its provider waits on the tool. > - This pull request pings authenticated runner connections and terminates the socket after 90 seconds of total silence. The existing reconnect grace and transport-failure path then recover the run. It also makes the execution projection report `canStop: true` for every queued or running run. > - The benefit is that a run with a dead runner connection now ends in a bounded time instead of hanging. API readers also see the Stop that the board already offers. ## Linked Issues or Issue Description **What happened?** A native-runner run (managed sandbox, OpenCode with an OpenRouter model) ran a foreground shell command for about 20 minutes with no output. The last run event came at about T+1s. The command finished on disk, but the run never got another event and stayed `running` / "Working" for over 35 minutes. Its projection had `canStop: false` and `permittedActions: ["inspect_run"]`. A board cancel through the API worked. **Expected behavior** The server detects a dead runner connection and recovers or fails the run in a bounded time. A runner that is alive but quiet stays attached. The projection reports that a live run can be stopped. **Steps to reproduce** 1. Start a native run in a managed sandbox. 2. Make the agent run a long foreground command that prints nothing. 3. Let the preview route drop the WebSocket without a close frame. 4. Before this fix, the run stays "Working" forever. ## What Changed - `server/src/realtime/runner-prp-outbound.ts`: after PRP authentication, `WsJsonWireConnection.monitorLiveness` sends a WebSocket ping every 20 s. Any inbound frame or pong counts as liveness. After 90 s with nothing inbound, it terminates the socket. That emits an ordinary `close`, so the PRP authority and the reconnect loop use the existing 7-minute grace and then fail the transport. If the server's own event loop stalls, the window restarts instead of blaming the runner. The 90 s bound is longer than runnerd's own 60 s connection lease, which already requires runnerd to read the socket. - `server/src/services/execution-projection.ts`: `canStop` is now `true` for every `queued` or `running` run. The old failed-native-retry rule is unchanged. - `server/src/services/execution-projection.ts`: a queued run now shows "Waiting for previous run to stop" (cause `previous_run_settling`) when a predecessor that admission checks (the task execution owner, the retried run, or the agent's latest settled run on the task) still has pending cleanup by `hasPendingRunCleanup`. Before, a turn queued after a Stop showed only "Queued" with no reason while the stopped run's controller still waited on the dead runner. That in-process wait (`claimQueuedRun` settling-owner gate) also made Retry a no-op. The liveness bound above makes that controller settle. Its existing cleanup then calls `startNextQueuedRunForAgent`, which starts the queued turn. - `server/src/services/run-cancellation.ts` and `server/src/modules/run-dispatch/adapters/postgres.ts`: a user-stopped native run whose runner was unreachable can later settle as `failed` (`native_provider_terminal_failed`) when its transport fails closed. The dispatch gate's message deferral required a `cancelled` source and compared the message with `finishedAt`. The user's follow-up message was therefore marked `skipped`, and the task asked for a second message. The new `recordedUserStop` helper honors the recorded user Stop on such a failed source, so the gate saves the message for the deferred-receipt worker. - `packages/shared/src/types/execution-projection.ts`: documents the `canStop` meaning. - `doc/execution-semantics.md`: documents the ingress liveness bound and the new `canStop` meaning. ## Verification - Red then green: `runner-prp-outbound.test.ts` "detects a half-open authenticated ingress and fails the run within a bounded window". On the old code it failed because no ping was sent and the socket was never terminated. It now passes and the active-run `failure` rejects with `runner_ingress_unavailable`. - Safety: "keeps a quiet but live runner attached through a long silent tool call" checks that a runner answering pings stays attached for 20 simulated minutes with no termination and no failure. - Red then green: `execution-projection.test.ts` "offers Stop for every live run, including a long-silent native turn". On the old code it failed with `canStop: false`. - Red then green: `execution-projection-recovery.test.ts` "names the wait when a stopped predecessor still holds the task's runner" (real Postgres). On the old code the successor projected a plain "Queued". It also checks that the wait clears on lease release and ignores another agent's run. - Red then green: `workspace-recovery.test.ts` "continues a message sent after Stop when the stopped run later failed closed". On the old code it failed 3 of 3 runs with the receipt `skipped`. It now passes, and the full file passes twice (392 tests). The test now pins the message after the recorded Stop. Before, the fixture could write both in the same millisecond, which made the sibling variants flaky. - Passed: run-cancellation, explicit-native-continuation, run-dispatch and run-control suites (622 tests). - Passed: `npx vitest run src/realtime/ src/services/execution-projection*.test.ts src/__tests__/heartbeat-run-privacy-routes.test.ts`, the native-runner-execution-driver tests, and the plugin-runner-task tests. - Passed: `pnpm --filter @paperclipai/server typecheck`. - I read runnerd (`runner-core/src/durable/transport.rs`) and confirmed it answers `Ping` with `Pong` in its non-blocking read loop. - Not run: a live managed-sandbox reproduction. Staging and Daytona were not reachable from this environment. `plugin-runner-probe.test.ts` needs a local cargo build of runnerd, and it did not start in this worktree. ## Risks - A false termination is possible if a runner fails to answer pings for 90 s while it is still alive. runnerd already loses its own connection lease after 60 s without reading, so this bound does not add a stricter requirement. A terminated live runner reconnects within the existing grace. - If the preview proxy drops WebSocket control frames, ping alone does not prove liveness. Inbound PRP frames (runnerd's lease renewals, about every 30 s) still count, so a live runner stays attached. - `canStop` now means "the board can Stop now" for live runs. The UI already shows Stop for running and queued runs without reading this hint, so the UI does not change. - Out of scope: the inbound runner WebSocket server (`runner-prp-ws.ts`), and a lost OpenCode SSE event while the runner is alive. A stopped run whose controller hangs for a reason other than a dead transport still holds the queued successor. The projection now names that wait, but this PR does not force-release it, because the controller has no termination evidence. A queued reason for the agent concurrency cap is also not covered. The wait label calls the admission predicate (`hasPendingRunCleanup`) for up to three predecessor runs per queued run. This adds a few reads to projections of queued runs only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a20dfbc695 |
fix(recovery): continue a message sent during Stop settlement after recovery (#19355)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can stop or interrupt a running native task and then send a message to give new direction > - When the stopped run left a workspace hold, the dispatch gate withholds the follow-up run until recovery saves the work > - The gate must save the user's message on its receipt so that the deferred-receipt worker continues it after recovery > - Two conditions made the gate skip the message instead of saving it, so the task stayed in progress with "Saved work recovered. Send a message to continue." although the user already sent one > - This pull request saves the message in both conditions > - The benefit is that the message the user already sent continues once, automatically, after recovery ## Linked Issues or Issue Description **What happened?** After a user stopped or interrupted a native task with an isolated sandbox workspace and sent a message, the follow-up run was cancelled with `execution_reconciliation_required`. Its error said "Saved work recovered. Send a message to continue." Nothing re-woke the task. In the interrupt case, a periodic sweep started a run about four minutes later. In the case where the follow-up first waited for the stopped run's workspace save (`workspace_busy`), each later retry parked on the same resolved recovery action, and the task did not continue. **Expected behavior** When the recovery that the follow-up waited on resolves, the message that the user already sent continues once, automatically. The product does not ask for a second message. **Steps to reproduce** 1. Run a native task in a managed sandbox with an isolated workspace. 2. Stop the run (or send a comment with interrupt), then send a message while the stop and workspace save are still settling. 3. Wait for the workspace recovery to resolve. 4. See that the follow-up run is cancelled with `execution_reconciliation_required` and that no run continues the message. **Paperclip version or commit** `master` before this change. **Root cause** `deferUnstartedUserComment` (run dispatch) refused to save the withheld message in two cases: 1. It compared the message time with the source run's `finishedAt`. A native run finishes only after its provider stop and workspace save settle, which takes seconds. A message written after the Stop request but before that settlement was treated as input written before the Stop. The receipt was marked `skipped`. 2. It required the run's own receipt to be user-owned. A workspace-busy retry or an admission retry carries the user's message under a system-owned receipt, so the message was dropped with the retry. ## What Changed - Add `userStopBoundaryAt(run)` in `run-cancellation.ts`. For a run cancelled by a user Stop or interrupt, it returns the recorded Stop request time (`cancellation.recordedAt`) when that is earlier than `finishedAt`. For all other runs it returns `finishedAt`. - Use this boundary in the dispatch gate's message deferral, in native explicit-continuation admission, and in the remote-Stop continuation check, so the three places agree on "message written after the Stop". - When the withheld run is a system-owned retry, walk a bounded chain (max 8) of retries that did not reach the provider (`workspace_busy` with `providerWorkStarted: false`, or a verified pre-dispatch `execution_reconciliation_required`) to the originating user receipt. Save the message on that receipt with the user's original wake context. Mark the retry's system receipt `skipped`. - Update the "input written before the Stop" test so that the Stop request, and not only the settlement, comes after the message. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` (`continues a message ... without a second message, after recovery already resolved`): a direct follow-up and a workspace-busy retry. Each runs recovery to completion first, then lets the gate withhold the follow-up, then runs only the deferred-receipt worker. Each asserts that exactly one successor run is queued with the user's message and the user's authority, and that no provider work starts. - Red before the fix: both tests fail because the receipt is `skipped` instead of `deferred_issue_execution`. Green after the fix. - `npx vitest run src/services/workspace-recovery.test.ts` (384 passed). - `npx vitest run src/services/explicit-native-continuation.test.ts src/modules/run-dispatch src/services/execution-continuation.test.ts src/services/run-cancellation.test.ts` (581 passed). - 19 more heartbeat, retry, queued-comment, and native recovery suites (1272 passed). One failure, `heartbeat-process-recovery > redacts opaque environment-bound credentials from Sentry diagnostics: result`, also fails on unchanged `master` and is not related. ## Risks - A message written after the Stop request but before the stopped run settled now continues. The stopping run cannot consume new input after its Stop request, and admission still removes comments that a run already delivered (`undeliveredLegacyUserCommentIds`). All admission gates (holds, stop proofs, approvals, budget) still apply. - Only user Stops and interrupts move the boundary. Other cancellations keep `finishedAt`. - The retry walk is bounded, stays in one company, agent, and issue, and accepts only retries without provider work. It revives a user receipt only from `cancelled` or `skipped`, so a second retry cannot save the same receipt twice. - Not in scope: a run that fails during native preparation with no user message (for example `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=unexpected_root_entry` on a system wake). After its recovery resolves, the task still waits for a user message. That is a separate root cause in the native session executor's prior-owner retirement proof. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
a6ab6a301c |
fix(server): retry a run when its remote Runner is out of date (#19326)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can run on a remote Runner that an environment plugin
provides
> - A release can update the app before the remote Runner, so for a
short time the app sees an older Runner
> - In that window the plugin can refuse the task, but Core cannot tell
this refusal from other errors
> - Today the run fails as `setup_failed` with no retry, and the task
goes to board recovery. Before #19246, it was also reported as a project
authentication conflict
> - Agent verification also counts the refusal as a failed harness test,
and it parks the agent after three tries
> - This pull request gives the plugin a stable error code for a Runner
mismatch, and Core waits and retries when it gets that code
> - The benefit is that a release no longer blocks tasks or parks
agents, and a real authentication conflict keeps its own clear notice
## Linked Issues or Issue Description
Refs #19246 (it stopped reporting every precheck error as a project
authentication conflict)
Refs #19249 (the same bounded bootstrap retry for a stopped plugin
worker)
Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope
is unchanged)
Refs #19172 (related open PR: it reports an undescribable plugin Runner
as a retryable check in the same precheck)
Refs #18667 (a real `ai_connection_incompatible` conflict, which now
gets its own notice)
**What happened?**
A run started while its remote Runner was older than the app. The
environment plugin refused the task because the installed Runner did not
match the revision it expects. Core recorded a generic setup failure,
scheduled no retry, and blocked the task. Before #19246, Core reported
this as a project authentication conflict. An agent in `verifying`
counted the same refusal as a failed harness test. After three tries it
stopped verification until a person chose Retry.
**Expected behavior**
Core treats a Runner mismatch as temporary. A run retries without
blocking its task. A `verifying` agent checks again until the Runner
matches. Only a real `ai_connection_incompatible` result blocks the
task, with a notice that names the conflict.
**Steps to reproduce**
1. Use an agent on a plugin environment whose installed remote Runner
revision differs from the revision the plugin expects.
2. Wake the agent on a task, or put a new agent through verification.
3. The run fails with `setup_failed` and the task is blocked.
Verification parks after three tries.
**Paperclip version or commit**
master at
|
||
|
|
b929ad43de |
fix: keep dependency waits neutral and explain GitHub access failures (#19315)
## Thinking Path > - Paperclip manages agent work and shows users what can move it forward. > - Task status and tool approval cards describe different parts of that work. > - A queued dependency wait has no provider attempt to retry. > - GitHub can reject an approved action while other connection tools still work. > - These cases showed a false Retry banner and an unhelpful provider error. > - This pull request keeps dependency waits neutral and gives GitHub access failures a direct repair path. ## Linked Issues or Issue Description Related: #19268 preserves pending Retry preparation and current failure causes. **What happened?** A queued wake cancelled by the dependency gate appeared as a failed attempt with Retry. A GitHub MCP permission denial appeared as raw HTTP error text after an approved repository action. **Expected behavior** Keep the task's dependency state. Show GitHub access failures with a clear explanation and a link to the selected connection's Permissions page. Preserve actual failures, provider details, and approval gates. **Steps to reproduce** 1. Queue a wake, then leave its task dependent on unfinished work before dispatch. 2. Open the task after the dependency gate cancels that wake. 3. Separately approve a GitHub action whose MCP result contains `403 Resource not accessible by integration`. 4. Reopen the saved tool approval. ## What Changed - Recognize both queues' unstarted dependency-wait receipts in cancellation provenance and task projections. No dispatch or recovery gate changes. - Classify GitHub's specific permission error after MCP result normalization. Keep result validation, the raw diagnostic, and working tools available. - Apply the same display code to both historical GitHub error codes. Identify the provider from company-scoped connection metadata, including renamed apps, without rewriting stored decisions. - Show a token-based inline access card with **Manage GitHub access** and collapsed **Error details**. - Give agents trusted recovery steps after an approved GitHub action fails. Keep provider text in the untrusted result and prevent automatic replay. - Add regressions, a production-component Storybook case, and execution/connector documentation. ## Verification - Red before the fix: cancellation provenance, dependency-wait projection, the real queue cancellation, approved MCP failure classification, saved interaction projection, and the error card assertions failed at the expected behavior. - Green focused tests: queue cancellation, one ready-task executor, approval-before-dispatch, one failed GitHub request, connection reuse, retained provider diagnostics, saved interaction reads, task list/detail projection and redaction. - UI: 106 component tests passed. Token gates and Storybook catalog checks passed. - `node scripts/typecheck-low-memory.mjs`: passed. - Local source instance: verified a saved dependency wait stays neutral after reload; a saved GitHub error shows repair guidance, expands its original error, and opens the correct connection Permissions page. PostgreSQL and HTTP APIs were real. The GitHub denial was a controlled MCP fixture; no account or provider permission changed. - `pnpm build`: passed for all workspace packages, the Runner binary, server, CLI and UI. The server build and all server typecheck layers also passed after the review repair. - Additional safety checks passed for realistic repository arguments, active recovery, failed Retry preparation, and bounded provider diagnostics. - Review repair: 45 pure tests and 7 PostgreSQL cases passed. These cover both historical error codes, renamed apps, and company/provider/application identity boundaries. A second local browser check verified the renamed app, retained error detail, and correct Permissions destination. - Final review regressions: the dispatcher and metadata cases failed before the repair. Afterward, 25 focused cases and 59 projection/identity cases passed, including connection-key precedence. The native wait and app-only GitHub source also passed a local browser check. - The historical read adds at most one connection query per interaction read batch. It selects IDs and the provider key only and runs only for candidate permission failures. It adds no provider requests. - Full local `pnpm test:run` stopped at the chat suite's `afterAll` cleanup timeout. All 268 assertions in that group and the three prior chat groups passed. The failure is retained here; it is not counted as a full-suite pass. CI passed on `e8a692e1e2` after one workspace-startup rerun. Fresh CI is pending for the combined head. Its first workspace-a attempt timed out in the CLI company import/export test's child-server startup wait; 4,968 UI tests and 287 CLI tests passed in that job. The same shard passed on the preceding commit. The workspace shard passed on its one rerun. All other test, build and browser jobs passed. The [first failure](https://github.com/paperclipai/paperclip/actions/runs/42093823928/job/127478869941) is retained in the Actions history. Greptile reviewed the preceding head at 5/5. Fresh review is pending for the combined head. ## Risks The GitHub classifier matches the documented permission-denial phrase, not every 403. Rate limits, missing resources and other providers retain their existing error behavior. Historical normalization affects display only. A dependency cancellation needs an unstarted run and a supported queue receipt; current user gates, Retry preparation and recovery remain authoritative. No schema migration or credential changes. ## Model Used OpenAI GPT-6, through the Codex agent. The exact runtime model ID and context-window size are not exposed in this session. Used reasoning, repository tools, TypeScript, PostgreSQL, MCP fixtures and browser verification. The parallel agent-recovery commit records Anthropic Claude Opus 5.5 as co-author. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the bug template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run the focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open findings - [x] I will address reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
10a7d039c8 |
fix(dispatch): keep a workspace wait parked while an ended writer's save settles (#19285)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run that finds its task workspace held by another run waits as a `workspace_busy` scheduled retry. #19143 keeps that wait as one parked retry while the holder is live. > - A remote task workspace stays held by its last writer run until the writer's save and sandbox release settle. This hold continues after the writer run ends, for example when a reassignment stops the writer. > - The parked-wait check counted only a `running` holder. For an ended writer whose save had not settled, it released the waiter at each due check. Preparation deferred again, cancelled the run, and created a new retry run. > - After a mid-run reassignment, the new owner got five cancelled `workspace_busy` runs in eleven minutes, and the task did no work. > - This pull request makes the parked wait use the same writer-hold rule as workspace reservation, so the waiter stays parked until the hold releases. > - The benefit is one durable wait, one start when the hold releases, and no churn of cancelled runs. ## Linked Issues or Issue Description Refs #19143 (parks a workspace-busy wait). Refs #19184 (releases a recovered hold for a new assignee after reassignment). **What happened?** A board user reassigned a task in the middle of a run. The old run was cancelled with `issue_reassigned`. The new assignee's runs were each created, started and cancelled with `workspace_busy`: "Shared execution workspace is busy: run … ended, but its workspace save has not settled". This repeated every 1–2 minutes (five cancelled runs in 11 minutes). Every cancellation named the stopped run of the previous assignee as the holder. The new assignee's other tasks were not the holder. **Expected behavior** The reassigned task waits as one parked retry. It starts once when the stopped writer's save and sandbox release settle. The wait stops at the one-hour holder cutoff. **Steps to reproduce** 1. Give a task a remote execution workspace whose writer run has ended, but whose save has not settled (`writerRunId` set, `dirty: true`). 2. Wake the task's assignee. The run defers with `workspace_busy` and schedules a retry. 3. Make the retry due and promote scheduled retries. Before this fix, the retry is promoted and dispatched, defers again, and creates another run. ## What Changed - New `server/src/services/remote-workspace-writer-hold.ts`: `readRemoteWorkspaceWriterHold` holds the rule for who holds a remote workspace (a running writer, or an ended writer whose save or sandbox release has not settled). The code is moved from `reserveRemoteWorkspace` without a behavior change. - `reserveRemoteWorkspace` uses that function. It throws the same errors as before. - `parkWorkspaceBusyRetryInTx` (from #19143): when the recorded holder is not running, it reads the same hold for the waited execution workspace. If the recorded holder is an ended writer that still holds it, the retry stays parked. The cutoff clock starts at the writer's end, the same value that reservation measures. Earlier waiters keep their arrival order. - New `WORKSPACE_BUSY_SETTLING_RECHECK_MS` (60 s). Settling has no single completion point that can wake the waiter, so a wait behind a settling writer re-checks every minute. Each check creates no run. In the promotion transaction it reads the workspace, the writer run, its task, and the writer's leases (one row per sandbox lease of that run). The checks last only while the wait exists, up to the one-hour cutoff. - `doc/execution-semantics.md`: documents the parked wait behind a settling writer. ## Verification - Red before the fix: `cd server && npx vitest run src/__tests__/heartbeat-remote-workspace-busy.test.ts`. The two new tests failed: the due retry was promoted (`expected [ … ] to not include '<retry id>'`). - Green after the fix, with: - `parks one waiter while an ended writer's save settles, then admits it once the hold releases`: three due checks keep one parked run, each re-check is bounded by the writer's end plus one hour, there is one `workspace_busy` cancellation in total and no new retry run. After the hold releases, the next check promotes the same run. - `stops parking behind an ended writer once its unsettled save reaches the holder cutoff`: parked before the cutoff, promoted after it (preparation then fails with the existing unfinished-writer error). - Existing suites pass: `heartbeat-workspace-busy.test.ts` (live-holder parking, arrival order, stop, workspace move), `modules/run-dispatch`, `remote-workspace-reservation.test.ts`, `remote-workspaces.integration.test.ts`, `isolated-remote-workspace-source.test.ts`, `heartbeat/workspace-preparation.test.ts` (476 tests in all). - `tsc --noEmit` for `server` shows no errors in the changed files. - Not done: a live run on staging. The fix is not deployed. ## Risks - Low. Parking applies only to a retry that recorded a `workspace_busy` wait on an execution workspace whose recorded writer is that holder. If the hold settles, moves to another run, or the task changes workspace, the retry is released to normal dispatch as before. - The wait stays bounded by the existing one-hour holder cutoff from the writer's end. After that, preparation fails as before. A user can still stop the parked run. - One writer per workspace is unchanged. Reservation still decides admission with the same rule. - The one-minute re-check runs only while a wait exists. It is not idle polling. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
06826915d7 |
fix(dispatch): park a workspace-busy wait instead of recreating runs (#19143)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Child tasks can reuse their parent's execution workspace. Only one run at a time may write in a shared sandbox workspace. > - A run that finds the workspace held by another live run waits with a `workspace_busy` scheduled retry. > - That wait cancelled the run and created a new retry run every 60–120 seconds, until the holder ended. Each check made one cancelled run record. > - Three sibling tasks that waited for a parent made 15 cancelled runs in 13 minutes. They also got the workspace in random order, not in arrival order. > - This pull request keeps the waiting run as one parked retry while its holder is live, and wakes it when the holder ends. > - The benefit is a clean run history, a prompt handoff, and fair arrival-order access to a shared workspace. ## Linked Issues or Issue Description Refs #18691 (the workspace wait for a live holder). Refs #19075 (sends a stopped writer with a pending save into the same wait). Refs #12587 (an earlier proposal about fairness for this retry). **What happened?** A parent task delegated three subtasks that inherit its execution workspace. Each subtask run was created, then cancelled with `workspace_busy` ("Shared execution workspace is busy: run … is still running"), then created again about every 1.5–2 minutes. One subtask got 8 cancelled runs, one got 5, and one got 2. Every cancellation named a holder that was really running: first the parent run, then the sibling that got the workspace. So serialization was correct. The cost was the churn of run records, and the subtask that waited first ran last. **Expected behavior** A run that needs a busy shared workspace waits as one durable record. It continues when the holder finishes. Siblings that wait for one workspace get it one at a time, in arrival order. Two writers never share the workspace, and the user can still stop a waiting run. **Steps to reproduce** 1. Bind two or more issues to one shared execution workspace in a sandbox environment. 2. Keep a run live on one issue (the holder). 3. Wake the other issues. Before this change, each due check promotes the retry, cancels it with `workspace_busy`, and creates a new retry run. After this change, the retry stays parked until the holder ends. **Paperclip version or commit** Observed on staging at `53735db7`. The code is unchanged on `master`. ## What Changed - `finalizeWorkspaceBusyDeferral` records `contextSnapshot.workspaceBusyWait` = `{ key, holderRunId, waitingSince }`. The key is `execution:<id>` or `project:<id>`. `waitingSince` carries over when the run defers again for the same workspace. - New `server/src/modules/run-dispatch/adapters/workspace-busy-wait.ts`: - `parkWorkspaceBusyRetryInTx` runs in the scheduled-retry promotion transaction, after the existing gate allows the retry. If the recorded holder is still running and is not stale, or an earlier waiter for the same workspace is still scheduled, queued, or running, the retry stays `scheduled_retry`. Only its deadline moves. It does not create a run, an event, or a wake. The fallback deadline is 5 minutes, and never later than the holder's one-hour stale point. - `wakeWorkspaceBusyWaitersInTx` runs in the transaction that ends a running run (`writeRunStatus`). It makes due the retries that wait for that run, and the retries that wait for the same workspace as that run. It registers the scheduled-retry dispatch notification before the write. - `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` moves to the run-dispatch domain, so the deferral gate and the parked check use one value. `heartbeat/retries.ts` exports it again. - A retry without a recorded wait (for example, a retry from before this change) keeps the old behavior. - `doc/execution-semantics.md` describes the parked wait. ## Verification - Red before the fix, green after (`server/src/__tests__/heartbeat-workspace-busy.test.ts`, "reused shared execution workspace in a sandbox"): - "waits on a live holder in one parked run, then runs once when the holder ends": before, the retry was promoted while the holder was live. After, it stays parked through due checks at +1 s, +10 min, and +30 min. Ending the holder makes it due at once. It runs exactly once, and there is only one `workspace_busy` cancellation. - "hands the workspace to three waiting siblings in arrival order, one writer at a time": before, nothing woke the waiters when the holder ended. After, exactly one waiter is admitted per handoff, in arrival order. Each runs once, and there are 3 cancellations in total (one first deferral for each sibling). - "still lets the user stop a parked waiter": before, the waiter was promoted while the holder was live. After, Stop cancels the parked retry, and nothing brings it back. - Safety case (green before and after): "lets a stale holder reach the existing quiet-holder failure instead of parking forever". - `npx vitest run src/__tests__/heartbeat-workspace-busy.test.ts`: 35 passed. - `heartbeat-remote-workspace-busy`, `heartbeat-retry-scheduling`, `src/modules/run-dispatch`, `heartbeat/retries.test.ts`, `execution-projection.test.ts`: 266 of 267 passed. The one failure, "records pre-provider quota rejection…", is a timing flake on a loaded machine and passes when it runs alone. It does not use this path. - `tsc --noEmit` reports no errors in the changed files. - Staging evidence was read only. This change is not deployed. ## Risks - One extra indexed `SELECT … FOR UPDATE` on `scheduled_retry` rows when a running run ends. The update and the notification happen only when waiters exist. - If an earlier waiter stays queued for a long time (for example, its agent is paused), later waiters for the same workspace wait behind it. Each check is a deadline update, not a run. The fallback check runs every 5 minutes. - A remote writer that has ended but has not finished its save (#19075) is not parked, because its run is no longer `running`. Such a wait can still make a small number of deferrals until the save settles. - Low migration risk: no schema change. Old retries without the new context key keep the old behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, browser read of staging run records) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1a7eb70a37 |
fix(run-dispatch): save a redirect sent while a stopped run settles (#19122)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can stop a running agent and then send a new message that
redirects the work
> - The stopped run keeps settling for some seconds after Stop (backup
and checkpoint persist)
> - A message sent in that window is queued as a normal run before
settlement records the workspace recovery hold
> - The dispatch gate then withholds the unstarted run, but it saved the
message only for sources with a native workspace-sync baseline
> - For other sources (remote workspace, snapshot-only) it marked the
wake `skipped`, so the message was lost and the task stayed in progress
with no run and no wake
> - This pull request saves the withheld message for any source when the
message was written after the Stop
> - The benefit is that a redirect after Stop is acted on exactly once
instead of disappearing
## Linked Issues or Issue Description
**What happened?**
After Stop, a user posted a redirect comment with resume intent. The
follow-up run was created while the stopped run was still persisting its
checkpoint. When settlement finished, the dispatch gate cancelled the
follow-up run with `execution_reconciliation_required` and an
empty-looking error ("Recovering saved work…"). The user's wake was
marked `skipped`. Nothing woke the task again. It stayed `in_progress`
with no active or queued run.
**Expected behavior**
The message sent after Stop runs exactly once after the stopped run
settles. The task never rests in progress with no run and no pending
wake.
**Steps to reproduce**
1. Start a native run whose stopped source has no native workspace-sync
baseline (for example a remote-workspace sandbox run).
2. Stop the run.
3. Post a comment with resume intent while the stopped run still
persists its checkpoint.
4. Let settlement record the workspace recovery hold.
5. Observe that the queued follow-up run is cancelled with
`execution_reconciliation_required` and its wake is `skipped`.
**Root cause**
`deferUnstartedUserComment` in
`server/src/modules/run-dispatch/adapters/postgres.ts` saved a withheld
user comment only when the stopped source had a native workspace-sync
baseline (the copyback replay case). The enqueue path saves or admits
the same message when it arrives after the hold exists. The gap was only
timing.
Refs #18332 (added the native-baseline-only deferral this PR widens).
**Paperclip version or commit**
Reproduced on `master` at
|
||
|
DottaandPaperclip
|
0eaabe5c0b |
refactor(accounting): remove company serialization from paid settlement (#18917)
Settle paid operations through operation-scoped transactions. Preserve exact ledger evidence and replay while moving aggregate budget decisions to durable deferred work. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
451d06e2fc |
fix: resume interrupted Runner preparation and confirm failure report delivery (#18853)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns run recovery and task continuation after a restart. > - A shutdown can interrupt Runner preparation before any provider work starts. > - Workspace recovery can save the files but leave that task behind a legacy execution hold. > - Retry then accepts a request without starting a successor. Startup also leaves the task blocked. > - This pull request verifies the interrupted preparation, preserves its files, and admits one bounded continuation. > - It also keeps a dropped Sentry capture eligible for a later report. ## Linked Issues or Issue Description **What happened?** A hosted task remained blocked after a graceful server shutdown interrupted Runner preparation. Its remote workspace was recovered. Retry returned an accepted request, but no successor started. Startup did not resume it. The task also had no matching Sentry event. The recovery cause is verified. The exact cause of the missing historical Sentry event is unknown. A separate regression with the real Sentry SDK proves that processing backpressure can drop a capture while the task cache marks it as reported forever. **Expected behavior** Retry must continue eligible interrupted work from its saved workspace. Startup must schedule one bounded retry when durable evidence proves that no provider started. A later task read must be able to retry an unconfirmed Sentry report. **Steps to reproduce** 1. Interrupt Runner preparation during a server shutdown before native runtime selection. 2. Retain and recover its remote workspace under the legacy recovery hold. 3. Press Retry, or restart after file recovery has completed. 4. Observe that the task remains blocked without a successor. For the reporting case, hold the first Sentry event in its asynchronous enrichment step. Fill the SDK processing buffer. Capture a second task failure, release the first event, and read the second failed task again later. **Paperclip version or commit** Reproduced against source commit `4d31588dea`. The hosted incident served `6ef37398c8d799c5142a3a6590ecc70ad352b663`. **Deployment mode** Hosted Linux server with remote workspaces and external PostgreSQL. Local regressions use the source checkout on macOS arm64, Node 26.4.0, PostgreSQL 17.10, and the audited optional Sentry SDK 10.71.0. Related public changes: #18541 added task-state failure reports. #18678 retained remote exports before Retry. #18803 fenced remote workspace writers before continuation. The small backend-port fixture correction follows #18739. It is included here because the same existing setup defect blocked CI. ## What Changed - Verify pre-native shutdown interruption against the immutable adapter claim, controller ownership, run events, native records, stopped leases, and exact workspace recovery receipts. - Retire only the server's unchanged bootstrap failure block. Admit explicit Retry through the existing continuation path. - Discover completed workspace recovery at startup and schedule one retry under the existing failure budget. Let saved input own its continuation. - Apply the existing safe replacement dispatch checks to preparation retries. A later human block, competing execution, dependency, pause, approval, or budget gate must still prevent dispatch. - Keep execution evidence available through the bounded run projection. - Confirm Sentry task-report delivery from a successful SDK HTTP response. Keep unconfirmed captures pending for 60 seconds before a later task read can retry them. Both caches remain bounded and start no timer. - Document recovery and report delivery semantics. - Align existing handoff/budget assertions with the guarded preparation retry reason. Keep the owned-runtime fixture outside reserved proxy ports, with bounded allocation attempts. ## Verification - Red before the recovery fix: all three new Retry/startup cases failed because no successor was created. Green after the fix: all three passed with real saved workspace archives and PostgreSQL. - The pre-merge revision passes all 384 tests in the six affected recovery, run-control, run-state, retry, and dispatch suites. - The late-block regression failed before its repair. Its four focused recovery/dispatch cases then passed, including final dispatch cancellation. - Review found a second stop check that still rejected verified local and lease-free preparation. Both new cases failed before that repair. All 17 interrupted-preparation regressions passed on the pre-merge revision. - The Sentry regression failed with the real SDK before the fix. All 91 affected reporting tests then passed. The transport stays in memory. No test sends events to Sentry. - Full repository typecheck passed with `node scripts/typecheck-low-memory.mjs`. The full `pnpm build` and module boundary check passed. - CI found untyped delivery-hook parameters when the optional Sentry SDK was absent. Explicit callback types fix that build. The SDK-absent server build and all 91 SDK-present reporting tests pass. - Server typecheck passes with the SDK installed and the real Sentry SDK contract CI job passes. Greptile reviewed the pre-merge commit `103e175c44` at 5/5 with no open findings. - The local full-suite attempt initially overlapped the rebase and stopped at test-line selection. Its unchanged-head rerun passed the first 309-test chat shard. I stopped the old-head run before applying review fixes. Current-head CI owns the remaining full-suite verification. - CI exposed two old retry-reason expectations. The handoff and retry-budget checks pass with the new guarded reason. A separate existing process-adoption fixture chose port 42953 in the reserved 42000–42999 proxy range. The fixture now rejects reserved ports, following #18739. Its real process/adoption/termination test passes locally. - The final Runner package job [initially failed](https://github.com/paperclipai/paperclip/actions/runs/41718507919/job/126205912033) one unchanged Codex credential case after it could not acquire its port quorum, then timed out. The complete unchanged 42-case credential suite passes locally. One unchanged CI retry passed; the original competing port owner was not identified. No Runner implementation or credential test was changed. - Required CI passes on `103e175c4490e1bdf804c64a8635b0e4568545ae`, including `ci / verify` and `ci / e2e`. That revision has Greptile 5/5 with no open findings. The optional Paperclip Review check on that earlier head remained queued, waiting for its assigned agent. - Merged current `master` without replacing published commits. The retry gate retains both shutdown interruption checks and the new workspace preparation timeout policy. All 311 tests in the five combined recovery, retry scheduling, completion, and native startup suites passed after the conflict resolution. Server typecheck passed. Final head `005d380b00d45a2393c4a2a190eab27611168d3c` includes the next conflict-free master update. All 17 interrupted-preparation regressions passed again on that final head. Greptile reviewed that exact head at 5/5 with no new findings. Required CI passes on that exact head, including `ci / verify`, `ci / e2e`, typecheck, build, the full test shards, and the real Sentry SDK isolation check. The optional Paperclip Review remains queued, waiting for its assigned agent; it is not a required branch check. - On the final head, six test jobs were cancelled after their spot runners received shutdown signals at the same time. Another server shard passed 2,419 tests, but its unchanged 103-record cleanup backlog case exceeded the 15-second limit. The unchanged full cleanup suite then passed all 29 tests locally. One unchanged retry of only the seven unsuccessful jobs passed. The [original cancellation and timeout logs](https://github.com/paperclipai/paperclip/actions/runs/41724898038/attempts/1) remain in the workflow history. No cleanup implementation or test was changed. - The signed-in hosted UI reproduced the failed Retry before the fix. The fixed path is covered through the real HTTP Retry route and assembled heartbeat/recovery services. This PR has not been deployed to the hosted incident. A live repaired Retry remains unverified until rollout. ## Risks - Automatic continuation must not repeat provider work. It requires positive preparation evidence and rejects retained provider, process-launch, coordinator, result, live-controller, unsafe lease, or mismatched workspace evidence. - Repeated recovery calls must create one successor. Tests cover concurrent requests, repeated startup visits, pending Retry, retry exhaustion, and changed task ownership. - Sentry delivery state stays local to the process. A restart or cache eviction can report a failure again. An unconfirmed event needs a later task read to retry. - No schema or API contract changes are required. ## Model Used OpenAI GPT-6 through Codex. This session does not expose a more specific deployment ID or context window size. Reasoning, code editing, test execution, and browser diagnostics assisted the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
16643a6e03 |
fix(runner): recover pending Stop cleanup after controller restart (#18682)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns execution, Stop delivery, and usage settlement. > - A Stop can remain pending when the controller restarts. > - Recovery must reach the existing session without starting more paid work. > - Startup and budget fences currently block that cleanup, and interrupted runs leave the restart selection. > - This pull request gives exact audited Stop cleanup a shared authority check and records its real acknowledgement. > - The benefit is safe cleanup while usage debt remains honest. ## Linked Issues or Issue Description **What happened?** A pending native run-only Stop can retain an unfinished execution owner after controller loss. Its startup cancellation fence prevents recovery admission. If teardown marks the run interrupted and clears its task lock, restart selection also omits it. A recovered session can receive cancellation without the durable acknowledgement that the normal Stop request would save. **Expected behavior** Recover only the existing remote session for the audited pending Stop. Keep fresh provider work blocked, including during budget pause. Keep incomplete usage pending until a real complete receipt arrives. **Steps to reproduce** 1. Start a native run with a held reservation and an authenticated remote checkpoint. 2. Persist a run-only Stop and its startup and execution cancellation fences before acknowledgement. 3. Lose the controller. Leave the coordinator unfinished, mark the run interrupted, and clear its task lock. 4. Attempt restart recovery and observe the missing claim or cancellation admission rejection. Related work: #14733 binds caller Stop requests. #18658 handles plugin startup cleanup obligations. #18035 proposes broader accounting changes. This change repairs existing-session Stop recovery and preserves current accounting guards. ## What Changed - Add one shared check for the exact run, company, issue, pending Stop, intent audit, live recovery lease, generation, and remote reattachment disposition. - Include eligible interrupted Stop owners in restart selection. Keep generic interruptions and another task execution owner fenced. - Reuse only the original held reservation for cleanup. Keep budget pause effective for fresh, bootstrap, dead-runner, and unproven recovery work. - Apply the same cleanup authority at heartbeat runtime selection and native entry. Skip attachment staging and warm-owner retirement only for that exact cleanup. Revalidate at session publication and fail closed if the proof changed. - Send the existing Stop through the audited cancellation producer before turn submission. Close the coordinator only after acknowledgement, including system-owned run-only Stops. - Give verified cleanup a transport purpose that retains pending semantic inputs without replay, dispatch, or a semantic reply. Add an explicit Core retain-only intake mode; preserve authenticated identity, schema, digest, sequence and replay checks. Ordinary execution still requires its dispatcher. - Add PostgreSQL, native protocol, and startup-fence regressions. Update the execution contract and recovery maps. ## Verification - The combined Stop regression fails on unchanged production at the cancellation reservation guard. Interrupted restart selection also fails before the repair while its running sibling passes. - An assembled `heartbeatService.recoverNativeRunsAfterRestart` regression exposes the earlier runtime-selection fence with the rest of the repair present. The fixed path reaches selector/claim, runtime selection, original held-budget reuse during pause, dispatch, real native protocol, audited cancellation, and heartbeat settlement. It starts no provider turn, acquires no new sandbox, and does not reclaim the task lock. - Fail-if-called attachment staging and turn boundaries remain untouched. A retained warm-owner regression prevents retirement. Invalidation at session publication fails closed. Actual backend/driver constructor tests verify cleanup-purpose forwarding, absent default options, and denial under foreign ownership. - Missing usage retains accounting debt and the hold. A later complete receipt records one 25-cent charge and settles that same hold. Concurrent duplicate cleanup is denied. Both user-owned and system-owned Stops pass. - Real transport tests: 9 pass on Node 24.22.1. They preserve ordinary saved-tool and completion recovery, prove the authority-denial race before cancellation, and qualify cleanup with pending ordinary/completion inputs and either handler ordering. - Real authenticated Core intake: 11 pass. Callback-present and callback-absent retain-only cases persist and acknowledge unanswered input without dispatch, recovery or result. Default missing-dispatcher rejection and digest-integrity/suspension guards remain effective. Omitting the callback alone first failed by disconnecting; the explicit retain-only mode fixes that defect. - Repository `pnpm -r typecheck`, the server/runner-vendor build, map checks, and module boundary checks passed locally before integrating the latest retained-startup change. That integration passes 11 native startup/cleanup compatibility cases and 3 assembled recovery/accounting cases. The final broad affected-suite attempt was stopped without a verdict; it is incomplete. Full repository tests and build are delegated to current-head GitHub CI. - Review exposed an additional real-process close failure with unanswered input. The corrected path requires a fresh exact provider-stop receipt confirming exit, followed by durable event drain and exact runner suspension. All 11 focused transport cases pass, including missing/false exit-proof negatives and ordinary recovery siblings. Runner TypeScript and map validation pass after this correction. - The assembled test controls provider, workspace-transfer and GitHub bootstrap I/O. It does not qualify hosted credentials, physical provider termination, workspace preservation, or historical receipt recovery. Local socket-permission setup failures are separate from semantic regression results. No hosted replay has run on this repair. - CI exposed an ordinary factory-call compatibility regression and an obsolete authority-read test fixture. The repair preserves single-argument ordinary calls and updates the race fixture without changing its Stop assertions. The complete native executor suite now passes all 1,049 tests; the ordinary factory boundary and all five Stop-accounting cases pass. A separate local environment-default case twice exceeded its five-second wait while other work ran; its prior-head CI case passed. The same unchanged case passed an isolated diagnostic without changing its assertion or timeout; the precise contention source remains unproven. - The final fixture edit initially used the wrong overloaded promise type. CI rejected it; the annotation-only correction passes the exact `pnpm run typecheck:build-gaps` entry point locally, including server service tests and remaining workspaces. No runtime assertion changed. - Final exact head `007fd4d04a2d6338c6dc073e97ab470a277fc3d0`: required `ci / verify` and `ci / e2e`, build, typecheck and test groups pass. Greptile is 5/5 with both findings resolved. Three expected checks are skipped; the optional native check is neutral, while its final current-head comment gives 5/5 from source inspection without a checkout or test execution. The first server-shard attempt lost its self-hosted runner connection; one scoped rerun passed. Its original annotation and unavailable-log evidence are retained. ## Risks - This changes execution authority. Cleanup requires an audited, exactly bound pending Stop, an unfinished coordinator, a live remote reattachment claim, and the original held reservation. It never authorizes a replacement provider turn. - Missing checkpoint, original held reservation, termination, or usage evidence can keep cleanup or accounting pending. The repair does not manufacture acknowledgement, replies, receipts, or zero cost. - Retain-only intake is an internal trusted constructor option. It changes no agent tool permission. Ordinary callback requirements and semantic dispatch remain unchanged. - No schema, public API, provider profile, or deployment setting changes are required. ## Model Used OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source inspection, code execution, and regression testing. The context-window capacity is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run affected tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9ca9af544a |
Drive remaining live run and resource recovery from durable work (#18419)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
3ee4c5b613 |
Make saved execution receipt recovery react to durable work (#18409)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
baf651c8d0 |
perf(heartbeat): drive retry promotion from durable deadlines (#18387)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
f5e5b05a33 |
Preserve native workspace recovery across Stop and sandbox reuse (#18332)
Land the thirteen-file native recovery fix independently from #18245. Retain verified Stop evidence and queued user comments. Retire stale preparation ownership atomically and require paired sandbox lease handoff receipts before workspace recovery. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
829f9e7eaf |
fix(server): settle Muse terminal assignments from committed changes (#17793)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/12/changes/paperclip/12 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its external-agent module owns Muse assignments, operation receipts, and worker holds. > - Running assignments and retained terminal history must not cause an empty maintenance scan every five seconds. > - Terminal run changes and operation results already have durable database transactions. > - This pull request wakes the existing maintenance worker from those commits and reconstructs work at startup. > - The worker retries only concrete settlement debt. It retains qualification deadlines and authority checks. ## Linked Issues or Issue Description Refs: #17772. This is slice 57. Parent #17772 has merged. Related: #17000, #17525, #17752, #17758. The publication targets master and preserves the merged parent. **What existing behavior does this improve?** Muse terminal assignment settlement and the existing Muse maintenance worker. **Current behavior** A running assignment can keep maintenance scanning every five seconds. Terminal reconciliation rewrites unchanged history. Native finalization also calls that reconciliation during periodic replay. Discovery excludes a revoked original binding even when its assignment still needs settlement. **Proposed behavior** Committed run changes and operation receipts wake the existing worker with the original company, run or binding identity. Startup reconstructs actionable work from existing rows. Maintenance processes bounded batches and disarms after settlement. Default live and manual callbacks remain. The periodic callback delegates this one duty. **Reason and benefit** Configured but idle instances stop querying for this obligation. Late receipts, restarts, drain and historical bindings still have recovery paths. No table or independent timer is added. **Breaking changes** No schema, endpoint or provider protocol changes. Explicit manual reconciliation retains its unbounded default. This does not alter Cloud sleep admission or claim that all instance work can sleep. ## What Changed - Add an owner predicate shared by discovery and locked settlement. Fenced private-worker uncertainty remains held without an automatic retry loop. - Preserve original binding generation and company/agent/run identity, including revoked bindings. Fence only the terminal assignment during reconciliation. - Extend existing maintenance with scoped hints, bounded batches, concrete-debt retry, startup reconstruction and quiet drain behavior. - Register notifications inside existing terminal, projection, cancellation, operation-result and send-failure transactions. Keep qualification and OAuth signals explicitly global when coalesced with scoped changes. - Serialize only four domain IDs. Real native-finalizer tests caught and prevent a circular-object serialization failure at this boundary. - Use existing restore/worktree suppression and settings reactivation. Delegate both periodic finalizer branches while retaining resource callbacks and default live/manual settlement. - Add the producer audit and regression evidence in `doc/plans/2026-10-26-muse-terminal-settlement-sleep.md`. ## Verification Accepted production source: `584015018ed4a9d57d2b0f192b75847503b54e53`. Publication head: `305014b030d6c5eb891c70bc57036f4192d4fdad`. Normal adoption of the actual parent squash preserves the complete accepted child tree. Only audit text, a migration-count clarification, and the proved Photon test extension follow. - **3,715 distinct cases in 52 complete server files pass**, plus **20 selected chat cases** (3,735 selected total). The main 49-file batch passes 3,008; process recovery passes 410; review response passes 273; the full Photon file passes 24. The other 1,794 chat cases were not run locally. - **Quiet producer proof:** moving only the Muse fanout outside its parent guard produces six candidate SELECTs and fails the regression. Keeping the guard yields zero and still settles a real terminal run. The same test hash spans the controlled red/green pair. - **Photon review evidence:** repeated active callbacks produce zero settlement hints/candidate SQL. Three concurrent attention recoveries produce exactly one of each. The existing locked attention guard was correct; production needed no fix. The parent security check accepted the resolved finding before merge. This PR keeps the additional settlement queue in the existing regression. - **Static checks pass:** database build; server typecheck/build; module boundaries; migration order; current Copilot identity. Packages equal the accepted parent byte for byte. No new live provider qualification is claimed. - **Fixtures and failures:** 34 fresh PostgreSQL 17 fixtures have 412 applied migration receipts through tag 0413_tired_kronos, plus fresh per-call helper fixtures. All 52 original test hashes restore exactly. The additional Photon test hash matches its passing proof. Earlier remote chat passive-idle failure remains recorded; both passive/missing cases pass in the new selection. Parent handoff timeout/fresh-file-pass accounting remains in its audit. No assertion or timeout was weakened. - [Integration and quiet producer audit](https://github.com/paperclipai/paperclip/blob/305014b030d6c5eb891c70bc57036f4192d4fdad/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md#current-master-integration-and-quiet-producers-2026-10-29). Fresh CI, security, and exact-head Greptile 5/5 must pass before merge. <details><summary>Earlier revision verification and failure history</summary> Receiver failure fairness: a controlled 1,100 ms batch reproduced a 30-second delay for healthy due receivers. The worker now supplies its existing per-item deadline through `nextRunAtOnError`. All 20 receiver lifecycle tests pass on the owner and combined tip `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production typechecking passes on the owner. The failed receiver retains its own backoff, and settled receivers return to zero recurring SQL. The five-second assertion budget is unchanged. See [the owning audit](https://github.com/paperclipai/paperclip/blob/f7477933469fdc960f555425dce66eff25a94e54/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch). Current-head CI and review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/bbd31dea5ef3a4875950a87b4636af8b6bde858e/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Inherited voice-cleanup verification: the controlled slow-provider regression and all 13 lifecycle tests passed on its owner and combined tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only the fixture clock advancement and its audit note. Current head: `f7477933469fdc960f555425dce66eff25a94e54`. This remains a single-duty PR stacked on #17772. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-24-recurring-server-sleep-stack.md), [2026-10-26-muse-terminal-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. </details> ## Risks - In-process signals do not capture independent SQL or other replicas. Supported external events still need ingress. - Pending assignments and operation receipts retain their original binding identity. A new agent configuration cannot take their ownership. - Genuine qualification, OAuth, and recovery deadlines remain obligations. They must not be discarded merely to make the worker quiet. - The parent coarse-hint guard prevents recurring Muse discovery; real company/binding authority changes retain their signals. - No full local repository suite, full chat suite, live Muse or Copilot qualification, deployed sleep, database suspension, external alarm coverage, or cost savings is claimed. Existing Cloud blockers remain. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, real PostgreSQL tests, and GitHub tools. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
ec3b558277 |
fix(server): schedule accepted native response recovery (#17696)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs store accepted results and their finalization decisions. > - A completed run can still owe its public answer after an interruption. > - Periodic recovery scans for that answer even when no new work exists. > - This pull request wakes recovery from committed changes and saved retry deadlines. > - The existing owner still verifies the result, source, and current publication authority. > - Settled or dormant response work no longer needs its own periodic scan. ## Linked Issues or Issue Description Refs #17690 (merged). This is slice 48 of the sleep migration stack, now based on master. Related: #17661 handles GitHub terminal publication; this change handles native accepted-response presentation. Governed-review presentation remains a separate duty. **What happened?** Native finalization recovery repeatedly checks already accepted results for a missing answer. Removing that check without complete notifications would lose responses when evidence arrives late, an agent resumes, or admission settles after execution. **Expected behavior** Recover the answer from its saved result when a supported change makes it eligible. Retain real retry deadlines. Stop querying when the answer is selected or cannot currently be published. Reconstruct work after restart. **Steps to reproduce** 1. Commit a native accepted result with no selected response comment. 2. Run finalization recovery while its source is unavailable or locked. 3. Restore that source or release the lock. The answer must appear once without repeating provider execution. ## What Changed - Honor recorded non-create decisions and skip-comment policy in both discovery and recovery. Apply durable conversation/result/continuation guards before recovery locks. Preserve genuine unanswered and legacy recovery. - Scope routine response wakes to the affected issue/run. Coalesced input names its active run; pending-effect settlement names its issue. Sixteen existing global authority/configuration hooks use one explicit company invalidation boundary. No new transaction wrappers or tables are added. - Add an accepted-response worker using the existing delivery coordinator and scoped deadline helper. Preserve saved retry deadlines and pending-effect fences under the owner's lock. - Share the finalization notification boundary. Wake on finalization settlement, exact accepted-result/final-message evidence, agent availability, retry admission, and existing chat authority changes. - Keep lifecycle policy in services. A narrow pre-write status hook lets lifecycle retain ownership of its transaction and status transitions. - Register the worker during enabled maintenance startup. Remove its authority subscription on shutdown. Periodic replay skips this duty on both committed and fresh-commit paths; ordinary finalization and explicit recovery retain their behavior. - Add a partial index and expression statistics for unpresented native runs. Migration 0412 supports concurrent replay and adds no queue table. - Add real PostgreSQL and lifecycle coverage. Preserve the implementation and producer audit in the plan document. ## Verification - Additional real Photon callback acceptance on the published production code disproves the security warning: ten active assertions cause zero response-recovery hints/SELECTs, while three concurrent assertions during an actual attention-to-active recovery cause exactly one wake/SELECT. The existing locked status guard already enforces this boundary. No production change was needed; the resolved finding now has a successful current-head security check. The regression will be retained with the next slice. - Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883` passes **1489 distinct server cases**: 1139 in six complete recovery/finalizer/monitor/summary/workspace suites (81.02 seconds), plus 350 selected chat cases (297.68s). Another 1463 chat cases are outside the pattern; the complete 1813-case file is not claimed. All eleven new regressions are included in that count. Five dedicated fresh PostgreSQL 17 databases migrate through unchanged 0412, with no fixture-source adaptations. Database build, server types/build, boundaries and full-SHA migration ordering pass. All seven test source hashes match. - The exact published baseline first fails eight behavioral assertions while the three genuine-answer cases pass. The same corrected test source passes all eleven after repair: recorded none/reuse remain suppressed; five ineligible history families take zero recovery row locks across repeated issue hints; missing/legacy/create-null answers still publish once; fresh periodic finalization defers presentation to its worker. Earlier fixture setup and variable/compiler failures remain explained in the audit, rather than erased by later success. - Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821 server cases**: 1496 in 13 complete producer/lifecycle/authority suites (110.63 seconds), plus 325 selected chat cases (278.99 seconds). Coverage includes all native-session cases, queue admission, status decisions, membership/board access, the complete GitHub workflow, accepted-response lifecycle and real authority restoration. The other 1477 chat cases are outside this selection; the complete 1802-case file is not claimed for this repair run. - Actual PostgreSQL regressions first reproduce a coalesced-input hint missing its run scope and two routine hint paths producing 12 response-history SELECTs each after three notifications. The same assertions pass with run-scoped coalescing and zero response-history SELECTs for unscoped routine hints. Actual membership restoration still publishes the waiting response once. Assertion time limits are unchanged; this measures real driver SQL and persisted producer work. - Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412. One embedded-only wake-queue fixture import temporarily uses its own external database and restores byte for byte. All 14 selected source files match pre-run hashes. This qualifies external-database behavior rather than reproducing embedded CI locally. - Master advanced independently with #18126's Pi archive retry fix. Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both complete binary patches. All accepted server/database/UI/test/CLI trees stay identical; the three independent runner installer/manifest files are retained. All 13 installer tests pass, including stalled header/body cancellation. Combined server types/build, boundaries and full-SHA migration order pass. A subsequent normal merge of #18128 at `e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only file changes with both complete patches preserved. The changed HTTPS backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178 outside the pattern), and combined types/build, boundaries and migration order pass again. All 14 repair source hashes remain unchanged. The independent browser fixture changes are preserved; no local browser run is claimed. - The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed the 4171-case full baseline before this scope repair. That scope is preserved in the audit rather than represented as a current full rerun. All 0412 database bytes remain unchanged: original concurrent index/statistics/ANALYZE, prior executable history, journal entries and retained snapshots. Five snapshots 0408 through 0412 remain. The populated 0411-to0412 upgrade preserved 410 prior receipts and 38 companies, appended one receipt and none on replay; index/statistic checks passed. - The earlier P2 security failure, Greptile's former conditional dependency on #17702, and the subsequent actual Ponytail eligibility findings are recorded in `doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This PR now includes the issue-scoped pending-effect notification fix itself, so it is independently correct before #17702. Fresh published-head CI, Greptile 5/5 and a clean security scan remain required. No deployed-sleep, external-wake or fleet-savings claim is made. ## Risks - The writer contract is process-local. Independent SQL writers and other replicas are not covered by these notifications. - Unpresented backlog reconstruction still has memory and time cost proportional to candidate count. Pages contain at most 100 rows; execution uses the shared fair batch. Selected history is excluded. Health-only signals do not trigger reconstruction. - Publication policy remains with its current owner. Digest, company/run binding, source authorization, selected-answer suppression, and ownership fences remain required. - The rest of native recovery and governed-review presentation still run separately. This PR does not relax Cloud sleep eligibility or add external wake coverage. - The stack parent has merged and its actual ancestry is reconciled. This publication still needs fresh current-head CI and review. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution, and PostgreSQL tests. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandPaperclip
|
4e715eb7ea |
fix(server): schedule historical unsafe export recovery (#17687)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs can retain accepted results after an unsafe workspace export fails. > - Historical recovery can retire an old notice or resume the accepted result without another provider turn. > - The periodic reaper repeatedly scans failures that cannot currently recover. > - Existing rows and committed changes can identify the actual recovery obligation. > - This pull request wakes recovery from those changes and retries only owed work. > - Dormant history becomes quiet while ownership and explicit operator decisions remain protected. ## Linked Issues or Issue Description **What happened?** The native finalization reaper repeatedly visits historical unsafe workspace-export failures. Cancelled tasks, newer runs, changed contracts, and explicit operator decisions can make those rows permanently ineligible. They still produce periodic database reads. **What did you expect?** Retire stale unsafe-export notices and recover eligible accepted results. Stop reading dormant history until a relevant committed change occurs. Keep a retry while a real obligation is blocked by ownership or a transient failure. **Steps to reproduce** Retain an unsafe-export terminal failure whose task is cancelled or superseded. Leave the instance idle after startup. The periodic reaper continues reading that history. Prerequisite #17680 is merged into master. Related: #14402 established safe export finalization; #14329 was an earlier historical recovery proposal. #17679 owns separate policy-review cleanup. This PR changes only historical unsafe-export scheduling and its producers. ## What Changed - Register a scoped worker through the existing coordinator and scheduler. Remove this duty from periodic reaping. Preserve the manual/startup recovery entry point. - Distinguish stale notices, eligible accepted results, dormant history, and concrete ownership contention. Reuse the existing mutation owner and lock order. Preserve explicit settlement vetoes and exact result/contract identity. - Signal relevant task, run, action, and deletion writes before they commit. Existing transaction boundaries handle nested commits and uncertain outcomes. Notifications contain only scope IDs. - Retain a retry only for owed work. Reconstruct work at startup, preserve drain/resume, and stop through the shared lifecycle. - Add concurrent, idempotent partial index migration 0410 for unsafe terminal failures. Keep the five newest generated schema snapshots. - Document producer coverage, query plans, supported writer boundaries, and test failures in the unsafe-export audit. ## Verification - Source `2a13dd08a6cc52f0adab09e3df9ce31e97b746b9` passes all 2188 cases across 24 complete files, without skips. The main selection passes 2129 cases in 192.58 seconds; safe-replacement passes 59 in 26.04 seconds. Coverage includes all 968 native-session cases, real export/action producers, explicit continuation, workspace reconstruction, interaction routing, Stop, queue and Slack consumers. The audit gives per-file counts. - Twenty separate fresh PostgreSQL 17 databases migrate through 0410 before setup. Four embedded-only fixture imports temporarily use their own external database on this IPC-exhausted host, then are restored byte for byte. Assertions and timing limits stay unchanged. This is qualified external-database acceptance, not an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-head migration order pass. Normal integration preserves the complete original runtime/schema patch and unrelated upstream blobs. Original concurrent index SQL, all prior executable history, journal entries and retained snapshot bytes are unchanged; five snapshots 0406 through 0410 remain. Added comment removal preserves the full non-comment TypeScript token streams. - A populated 0409-to0410 upgrade preserves all 408 prior migration receipts and 38 companies. It appends exactly one receipt; replay adds none. The index is valid and ready on fresh and upgraded databases. - The owning audit retains earlier fixture failures, skipped checks and notice-retirement regression history. Older combined stack checks retain their original scope in their owning audits. Actual parent `73d24aa5acfcf17b3f676b4da7cd05eb70b60b99` is reconciled through a normal merge. Its tree equals the reviewed ancestor and the complete accepted child tree is unchanged. Fresh published-head CI and Greptile 5/5 remain required. No deployed sleep, fleet-cost or live provider qualification claim is made. - Inherited prefix correction: current integration `8f43c9f81fac23b50997e7f0e1d0d9cf383a8109` changes only two values in a previously unselected corpus fixture. All production code, the complete package tree and all 24 accepted test files remain byte-identical to the 2188-case source above. The additional complete 72-case corpus passes in 14.00 seconds and again on the same database in 13.08 seconds. Retained PSC/OUT and sentinel rows remain unchanged. The repeat is not double-counted. The parent audit retains the original collision and rejected cleanup attempt. - Latest parent context: integration `4d51231abe8b68c21ce896d0d2ac16865978c8ab` carries only the parent's one-argument GitHub observation correction and an independent master cleanup assertion. All production code, packages, 24 accepted suite files and the additional corpus are unchanged. The two focused native-presentation cases pass in 15.14 seconds and the complete cleanup-removal suite passes all six cases in 8.31 seconds on two fresh 0410 PostgreSQL fixtures. This does not claim the full parent workflow was rerun here. Current head `b28399babac72ba9db1c4a4ef882c5e87aded38c` has passing CI and Greptile 5/5 with no actionable findings. The first CI attempt lost runners: fourteen jobs explicitly received simultaneous shutdown signals, two reported lost runner communication, and two aggregate gates failed downstream. One failed-job retry on unchanged source passes. The underlying host shutdown cause remains unconfirmed; no code fix or extra retry is claimed. [CI run and attempts](https://github.com/paperclipai/paperclip/actions/runs/40819408528). No Ponytail verdict is present. ## Risks - A missing future producer can delay dormant-history discovery until restart. The audit names current producers. Independent SQL writers and other server processes remain outside the process-local signal contract. - Physical ownership and uncertain operations retain retries. Time alone does not authorize taking workspace ownership. No provider turn or unsafe archive retry is added. - The recovery owner remains authoritative if state changes after discovery. Terminal UI status does not authorize resource destruction or accepted-result replay. - The partial index adds matching-row write/storage cost. It builds concurrently through the existing restart-safe migration runner. - This proves the changed worker becomes quiet. It does not prove host suspension, external wake coverage, or fleet cost savings. Other sleep migrations remain. This slice still requires fresh current-head CI and reviews. ## Model Used OpenAI Codex, based on GPT-6, with reasoning and code-execution tools. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
51c3f8924c |
fix(server): schedule cancelled workspace preservation (#17677)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A cancelled native run may still owe preservation of prepared workspace files. > - The existing recovery receipt and source lease record that obligation. > - The orphan reaper still searches for these cancellations on every pass. > - This PR registers committed cancellations with the shared deadline worker. > - The existing retention owner keeps file pinning and ownership checks. > - Idle and completed history no longer causes this preservation scan. ## Linked Issues or Issue Description Refs #17674 and #17145. Stopped instruction collection is merged into master. It owns file-preservation discovery only. The existing environment cleanup worker continues to own stop-only teardown. **What needs to be improved?** `preserveCancelledNativeWorkspaces` runs from every orphan-reaper pass. It queries even when no cancelled native workspace needs a file-recovery hold. Current foreground Stop can also defer retention after a lock or database failure, so startup-only recovery would be insufficient. **Why is this improvement important?** Empty queries keep idle tenants active. Removing them must not lose user files, weaken Stop ownership, replay cancelled work, or turn a retained allocation into destructive cleanup. **Proposed solution** Register the existing durable obligation from committed run-status, cancellation-acknowledgement and workspace-reference writes. Reconstruct it at startup. Keep retries while actual preservation remains unresolved. Disarm after the existing owner records the hold. Related-PR search and ROADMAP.md review found no duplicate of this scheduling migration. ## What Changed - Add a company/run-scoped preservation worker with 100-row keyset reconstruction and fair batches of five. - Separate pending file debt from immediate coordinator/lease readiness. Retain a 30-second retry for known unfinished preservation and shared failure backoff for errors. - Signal the existing status, acknowledgement, workspace-reference, deletion and settings transactions. Add no table or independent timer. - Keep exact source checks, no-workspace contracts, task/coordinator/run lock order, retention receipt and no-replay recovery card with their owners. - Recheck cancellation acknowledgement, run status and coordinator phase under the retention locks. Preserve the existing foreground preterminal API. - Keep startup/manual recovery; skip only this duty in periodic reaping. Preserve drain/standby and shutdown behavior. - Add concurrent, replay-safe partial index migration 0409 and retain the five newest snapshots. - Add the producer and lifecycle audit in `doc/plans/2026-10-25-cancelled-workspace-preservation-sleep-audit.md`. ## Verification - Current source `21b46a8e4293ca670235117458b984493a181759`: all 1472 cases in 16 complete files pass without skips. The owner passes 84 cases in 14.93 seconds. Fifteen consumer files pass 1388 cases in 133.46 seconds, including all 968 native-session cases and Stop, dispatch, restart, workspace-sync and upgrade-readiness coverage. - Twelve separate fresh PostgreSQL 17 databases migrated through 0409 before setup. The embedded-only lifecycle fixture temporarily used its own external database on this IPC-exhausted host and was restored byte for byte. Assertions and timing limits stayed unchanged. This qualifies external-database behavior; it does not claim an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-head migration order pass. The complete original runtime/schema patch and unrelated current-parent blobs are preserved. Original concurrent index SQL, all earlier executable history and retained snapshot bytes are unchanged; five snapshots 0405 through 0409 remain. - A populated 0408-to0409 upgrade preserves all 407 prior receipts and 38 companies. Exactly one receipt is appended; replay adds none. The preservation index is valid and ready on fresh and upgraded databases. - Actual parent `82529741418e138d9d623bf13d2fa22c1ff79d36` is merged through a normal ancestry reconciliation. Its complete tree equals the reviewed parent. The entire accepted child tree and all source hashes are preserved. The audit retains earlier failures, qualifications and per-file counts. Published head `b34f0589847f33754ecde9413fd36bf007353d2c` has Greptile 5/5 and all 57 checks passing or skipped. No Ponytail review is present. These checks do not establish deployed host/database suspension or Cloud alarm delivery. ## Risks - Signals cover instrumented single-process writers. Independent SQL writers and additional replicas need integration. - Status signals may inspect a run without relevant file debt once. They do not create recurring work. Discovery and memory still scale with actual candidates. - Missing ownership or stop evidence remains pending. Active retries are intentional while a concrete file obligation is unresolved. - Retry timing is process-local; startup reconstructs durable debt. The worker does not overwrite another native-finalizer deadline. - Admitted database operations may finish after drain begins. Shutdown joins them. Provider teardown remains with the existing cleanup owner. - The partial index adds storage and write cost. Its concurrent migration uses the existing interrupted-index repair path. - Cloud alarm coverage, sleep eligibility and host suspension are unchanged. This proves worker behavior, not fleet savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8252974141 |
fix(server): schedule stopped instruction collection (#17674)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runs can stop while their private instruction edits still need collection. > - Existing copy rows and stop evidence describe that unfinished work. > - The orphan reaper still queries for it on every maintenance pass. > - This PR activates collection from committed copy and run transitions. > - It uses saved retry deadlines when stop proof or collection remains pending. > - Active copies and completed history no longer cause collection polling. ## Linked Issues or Issue Description Refs #17671 and #17670. Both prerequisites are merged into master. Captured saves and post-save release remain separate reasons with their own workers. **What needs to be improved?** `recoverStopped` queries instruction copies on every orphan-reaper pass, even when no terminal run owes collection. A run can become terminal without changing its copy row, so copy-only notifications would miss real work. **Why is this improvement important?** Repeated empty queries prevent idle tenant instances from becoming quiet. Removing them must preserve edits, stop proof, current ownership, and recovery after restart. **Proposed solution** Use the existing transaction signals, scoped deadline worker, and scheduler. Reconstruct eligible copies at startup, signal supported copy and terminal-run writers, and retain saved retry deadlines for unresolved work. Preserve the existing collection and directory owners. Related PR search found the warm agent-file ownership work (#14695), verified stop boundary (#14311), and task-workspace cleanup (#16464). This PR does not replace those contracts. ROADMAP.md has no duplicate migration. ## What Changed - Add scoped stopped-instruction collection with keyset reconstruction and fair batches of 20. - Signal before copy preparation, mutation and adoption writes, and before terminal run writes in their existing transactions. - Cover heartbeat, native finalization/restart/session recovery, execution-control, queue and dispatch producers. - Signal settings changes and company/agent deletion. Preserve drain, standby, restart and shutdown ownership. - Recheck terminal-run eligibility in the due-time claim. Keep terminal status separate from process-stop proof. - Retain existing stop checks, attempt limits, saved candidate bytes and remote-resource fencing. - Skip only this lane in periodic orphan recovery. Preserve manual/startup recovery. - Add concurrent, replay-safe partial index 0408 and retain the five newest snapshots. Add no table. - Record the owner and producer audit in `doc/plans/2026-10-25-stopped-instruction-collection-sleep-audit.md`. ## Verification - Source `cb2818a8f46ef961f744977a258f0e9042e31a42`: all 1469 cases across 14 complete files pass without skips in 141.69 seconds. This includes all 968 native-session cases, filesystem saves, startup, terminal-run producers, dispatch, queue, execution-control, finalization and lifecycle. - Eleven separate fresh PostgreSQL 17 databases migrated through 0408. The embedded-only lifecycle fixture temporarily used its own external database on this IPC-exhausted host and was restored byte for byte. Assertions and timing limits stayed unchanged. This qualifies external-database behavior; it does not claim an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-current migration order pass. The original concurrent index SQL is unchanged. All prior executable migration history, journal entries and retained snapshot bytes are preserved; the five newest snapshots are 0404 through 0408. - A populated 0407-to0408 upgrade preserves all 406 prior migration receipts and 38 companies. It appends exactly one receipt. Replay adds none. The collection index is valid and ready on fresh and upgraded databases. - The normal merge of actual parent `267842fc3c6e6542deed8a1bcc2c78bec8df7862` preserves both complete patches, all independent sidebar changes from #18117, and the full accepted server/package/test trees. The complete sidebar test file passes all 36 cases on the combined tree. React act warnings remain. - Prior failures and fixture qualifications remain in the audit. Published head `d176d731f083c7edd2ede688cb901d1e66d8f1d8` has Greptile 5/5, all 57 checks passing or skipped, and no unresolved review threads. No Ponytail review is present. These local tests do not establish deployed host/database suspension or Cloud alarm delivery. ## Risks - Process-local signals cover instrumented writers. Independent SQL writers and additional replicas still need integration. - Terminal-run signals can cause one scoped inspection even if no instruction copy exists. They do not create a recurring scan. - Reconstruction and memory scale with actual pending candidates; active candidates are filtered by the run join. - Admitted file/provider operations can outlive a later drain request. Shutdown joins them. Uncertain ownership remains pending rather than being discarded. - The partial index adds storage and write cost. Its concurrent migration uses the existing interrupted-index repair path. - Cloud alarms, sleep eligibility and provider suspension are unchanged. Worker quiet is not a claim of deployed host/database sleep or measured savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL/filesystem tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
27270e9bf8 |
fix(server): wake interrupted retry cancellation from committed intent (#17667)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A parked scheduled retry has no executor to finish an interrupted Stop request. > - Its saved cancellation fence already records the work that remains. > - The orphan reaper currently scans for that work on every heartbeat, including idle periods. > - This PR registers the existing records with the shared delivery scheduler and wakes recovery after committed writes. > - Recovery keeps the existing cancellation guards and settlement behavior, then becomes quiet. ## Linked Issues or Issue Description Refs #17666, the merged preceding PR in the recurring-server-work migration stack. This PR targets master. Related cancellation work found in the dedup search: #12301 and #7340. This PR changes the trigger for interrupted parked-retry cancellation; it does not replace their cancellation or retry policies. **What happened?** The periodic orphan reaper queries scheduled retries with cancellation fences even when no cancellation needs recovery. Its first-20-row scan can also stop behind malformed evidence. **Expected behavior** Committed cancellation intent wakes bounded recovery. Failed work retains a retry. Startup reconstructs durable work. Empty or settled work performs no recurring SQL for this duty. **Steps to reproduce** 1. Start a server with heartbeat scheduling enabled and no interrupted retry cancellations. 2. Observe the orphan reaper repeatedly query for cancellation fences. 3. Cancel a scheduled retry and fail the final status write after its fence commits. Recovery must complete the existing Stop request without running a provider or dropping its metadata. **Paperclip version or commit** Current parent squash `7afee8f0622e273d10506b0febfb68f143d168b1`. Source acceptance `66107ae0a7ad85fd0476f91c4f8f98ae74a7a048`. **Deployment mode** Self-hosted server with the existing single-process transaction signal model. Local evidence uses PostgreSQL 17. ## What Changed - Add one domain-owned recovery worker over existing heartbeat run rows. No new table. It delegates to run control and preserves original cancellation, budget, and operator metadata. - Signal cancellation fences, lifecycle transitions, dispatch deferral/promotion/suppression, relevant settings changes, and deletion within their owning transactions. Wrap only the ordinary database fence in a short transaction; provider calls stay outside it. - Keep startup/manual recovery compatible and remove this duty from the periodic orphan sweep. Other reaper duties remain for separate PRs. - Extend the shared scoped worker to retain pending work with a memory-only admission retry while its owner is paused. Without pending work, it schedules nothing. Existing GitHub consumers retain their behavior. - Add a concurrent idempotent partial index and lifecycle/producer tests. Enable guarded disposable PostgreSQL URLs in affected test harnesses. - Reconstruct one hundred candidates per scope per scheduler pass. Resume from the saved scope cursor and retain fair twenty-item execution batches. Manual recovery starts cancellation before reading the full backlog. - Require the recovered browser reply to be saved before testing reload. Keep all original assertions and time limits. - Record the full producer audit, lifecycle limits, and failed attempts in `doc/plans/2026-10-25-scheduled-retry-cancellation-sleep-audit.md`. ## Verification Published head: `405e533394b26215e22d0452e9b0207d3f784a79`. Source acceptance: `a35fa43b98e2bd378d4a920d2775ce243654838b`. The final commit changes the audit only. - All 363 backend cases in ten complete owner and consumer files pass without skips. This includes actual Stop/retry producers, transaction settlement, interrupted-write recovery, pause/resume, shutdown, fair backlog processing and quiet after settlement. Seven separate fresh PostgreSQL 17 databases prevent fixture interference. - The two embedded-only backend fixtures temporarily use separate external databases on this IPC-exhausted host. Their fixture imports and native client teardown are restored byte for byte afterward. Assertions and timing limits are unchanged. CI must still pass with the original embedded fixtures. - Two paging regressions fail on the prior worker and pass on the fix. They cover cancellation before full reconstruction, a malformed first page, fair batches, a newly committed intent behind the cursor, and no later idle SQL. Per-candidate admission reads remain deliberate safety checks. - The repeated CI reload failure is reproduced with a temporary twelve-second fixture settlement delay. The trace retains the same five-second missing-reply assertion, even though a cleanup race masked it in the top-level error. Requiring exactly one saved reply from the resumed run passes under the identical delay. The fault is removed. All six browser journeys pass on the final source in 40.5 seconds, with zero retries and the original 120-second case and five-second reload limits. - Browser acceptance uses the canonical CLI/server/static-UI path and a fresh external database. Only fixture database-URI setup is temporarily adapted and then restored. This is not an unmodified embedded CI-environment reproduction. The six cases include the controlled case; repeated runs are not added to the distinct count. - Actual database build, server typecheck/build, module boundaries and exact-current migration order pass. The database source tree is unchanged from the verified 0405-to0406 upgrade: all 404 prior receipts and 300 companies stay intact, exactly one receipt is added, replay adds none, and the index is valid on upgraded and fresh databases. The five snapshots remain 0402 through 0406. - All earlier failures remain recorded in the audit. Prior head `48b3fcb83f305616df02a4337a42ec0867bb4b80` had exact-head Greptile 5/5 but failed the repeated browser assertion. It was not retried blindly. Its completed Ponytail review's backlog concern and old migration paragraph are addressed here. Old scores do not satisfy new-head gates. - Current-head CI passes. Exact-head Greptile and Ponytail both give 5/5, with no remaining actionable findings. These checks do not prove deployed instance or database suspension. ## Risks - Only supported in-process writers produce local transaction signals. Independent SQL writers or other replicas need a separate ingress contract. Startup reconstructs saved obligations. - A real failure retains bounded retries. Unknown commit outcomes retain the shared intent hold. Neither is discarded to report quiet. - Per-owner drain uses a memory-only admission recheck. The shared helper change affects existing consumers, which have focused regression coverage. - The partial index is concurrent and idempotent; it adds index storage. Each scoped load reads at most one hundred rows. The shared helper retains discovered pending IDs between passes, so a large unresolved backlog still uses memory. - Cloud alarms, idle-safety exemptions, and other reaper duties remain separate. Merge requires passing current-head CI and review gates. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and local PostgreSQL tests. The runtime does not expose a more precise model identifier or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5f0da562db |
fix(server): wake chat lifecycle milestones from committed changes (#17606)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected chats show safe notices when a run starts, waits, finishes, or fails. > - A recurring scan projected these notices even when no eligible work existed. > - Eligibility depends on committed run state, conversation lineage, and current authority. > - Existing database rows can reconstruct that work after restart. > - This PR wakes a scoped worker from those writes and removes lifecycle polling. > - Native progress events keep their separate existing owner until the next migration. ## Linked Issues or Issue Description **What happened?** The chat maintenance timer repeatedly scanned run lifecycle state. Empty and settled instances still queried the database. A run-status event alone cannot replace the scan: prompt receipts, continuation receipts, identity changes, and restored authority can make a run eligible later. **What did you expect?** Committed changes wake projection. Startup reconstructs eligible work. A completed pass stops until another relevant change arrives. Failed work retains recovery without scanning settled history forever. **Steps to reproduce** Configure an automatic chat endpoint. Finish its work, or leave it unused. Observe recurring lifecycle projection queries from the chat maintenance callback. Stacked on #17592. Related foundation: #15625. This PR owns lifecycle milestones only. ## What Changed - Preserve upstream coalesced Cloud membership writes together with locked membership restoration and its committed chat notification. - Signal milestone eligibility from the real Slack account-link transaction. - Register the lifecycle worker with the existing delivery coordinator. Remove lifecycle projection from the periodic chat callback. Keep native progress in a separately named lane and preserve the explicit combined API. - Reconstruct company/issue scopes from the existing selector. Drain bounded batches with company fairness. Retry failures, retain writes through drain/resume, and disarm settled work. - Signal from supported run, presentation, interaction, receipt, lineage, conversation, recovery, and authority writers. Preserve their transaction ownership, including nested savepoints and uncertain settlement. No new table is added. - Preserve provider authorization, authored-final suppression, responsible-user continuity, deduplication, the 24-hour selection window, and raw-output restrictions. Keep provider interpretation in its owner. - Preserve PostgreSQL timestamp precision in lifecycle pagination. Rejected submillisecond candidates must not repeat the same page and starve later notices. - Document producer coverage and alternate-writer dispositions in `doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`. ## Verification Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting master after #17592 merged at `7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates the actual landed parent. The entire 77-path milestone patch is binary-identical to the locally verified preparation across that ancestry reconciliation; both unrelated master paths are preserved. The four earlier producer/test integration conflicts retain scoped milestone and voice notices, current onboarding completion admission, and expanded identity query/read-scope coverage. - **330 distinct selected cases pass on the verified unchanged patch:** 305 across 14 complete suites (all five milestone worker suites, run identity, board claim, Cloud provisioning, auth session routes, Cloud actor authentication, stale-lock recovery, GitHub credentials, project Git authorization, and app admission wiring), plus 25 selected real chat producer cases. The remaining 1,590 chat integration cases were skipped, not passed. - The real producer cases cover saved milestone discovery, GitHub and Slack identity changes, account-link rollback/reconfirmation, ordinary ingress reactivation, post-close action/delivery receipt settlement, all ten authority-restoration paths including Cloud synchronization, and changed native review metadata. - Worker coverage verifies outer commit, rollback, uncertain-write recovery, restart, bounded batching/fairness, timestamp precision, retries, drain/resume, and settled SQL quiet. Authorization and existing publication deduplication remain with their owning services. - Server production typechecking, build, and module-boundary checks pass. The tests above were executed separately. Production typechecking excludes `src/__tests__` and does not replace those runs. Each database owner uses its own newly created and app-migrated PostgreSQL fixture. - [Current-parent acceptance and preserved failure ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28) records commands, counts, producer scope, conflict dispositions, and limits. Earlier combined-stack counts and failures keep their original revision scope; local acceptance does not erase old CI failures. - No new migration, queue table, dependency, workflow, or shared scheduler contract is introduced. Parent migration 0397, bounded consent recovery, voice cleanup/retention repairs, and runner checkpoint restoration remain present. - **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No Ponytail review is present, which is an explicitly authorized nonblocker; no failing review is waived. Older reviews and parent checks are not current passes. Actual failing Ponytail findings must be fixed; an absent/stopped/paused turn without a verdict is an authorized nonblocker. Active unfinished Ponytail uses its original request/start timer, with a 25-minute monitor cutoff respecting the user's 30-minute maximum. - No complete local repository matrix, live provider tenant, external alarm, physical host/database suspension, or fleet savings claim is made. Native progress remains a separate following slice. ## Risks - A future eligibility writer must notify the existing projection boundary. Independent SQL writers or another active server process are outside this process-local signal contract. - Authority and lineage remain sensitive. The worker uses the existing selectors and publication policy; it does not grant execution or disclosure authority. - Native progress and other publication maintenance still have recurring work. This PR removes one reason, not the entire chat maintenance timer. - Worker quiet does not prove database suspension, host sleep, or external alarm coverage. Cloud sleep blockers remain unchanged. - Merge eligibility requires passing current-head CI and Greptile 5/5 after this actual-master reconciliation. Subsequent stack slices retain their own gates. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window size are not exposed. Used repository inspection, reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details available in this session) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the documented focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
30b753f614 |
perf(budgets): scope accounting and deliver enforcement asynchronously (#17046)
## Thinking Path > - Paperclip manages AI agents and their paid work. > - Budgets must prevent new work and stop reported overruns. > - Cost settlement also ran serial lifecycle reconciliation for every eligible company agent. > - That made request latency grow with unrelated idle agents and delayed results on stop delivery. > - This change separates atomic admission and accounting from durable asynchronous enforcement. > - Scoped live usage detects overruns without writing provisional ledger charges. > - The result is constant query scope for ordinary settlement and recoverable stop delivery. ## Linked Issues or Issue Description **What happened?** Settlement and budget checks entered company-wide lifecycle reconciliation. A company with 1,000 unrelated idle agents performed 1,001 accounting transactions for one ordinary cost evaluation. User requests could also wait for provider termination. Running usage checkpoints did not enforce reported monetary exposure. **Expected behavior** Keep the final reservation gate authoritative. Commit charges, scope blocks, and enforcement intent before returning. Deliver stop requests after commit. Read only applicable policies and execution scopes for ordinary charges. Stop long-running work when supported reported or priced usage establishes an overrun. **Steps to reproduce** 1. Create a disposable company with one company budget and idle agents. 2. Evaluate a charge below the budget with 0, 17, and 1,000 unrelated idle agents. 3. Compare query counts, company accounting transactions, and duration. 4. Use the controlled partial-usage and delayed-stop fixtures to check enforcement and request responsiveness. **Paperclip version or commit** The measured baseline is `6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837` from `master`. Related public work: Refs #16976, #2006, #3539, #13763, #14296, #16853, #13075. These cover the affected service charge, cancellation, and accounting paths. This change removes their shared reconciliation overhead. ## What Changed - Batch applicable policy exposure under the existing company accounting lock. Preserve exact fractional money and zero-reservation dispatch fences. - Commit a durable final budget dispatch claim before provider I/O. Retain a separate final issue/run ownership transaction through synchronous handoff, without holding the company accounting lock or awaiting provider work. Preserve native recovery lease checks and duplicate handoff fencing. Browser handoff rechecks current policy; it reuses a consumed rate slot only for the identical policy generation. - Separate ledger spend, live observed spend, and effective held capacity. Use the maximum of reservation and live observation for each unfinished operation. - Deliver budget cancellation through the existing coordinator. Revalidate generations and persist queued/running stop fences before external termination. Bound stop concurrency to four. - Apply agent holds only to the affected agent. Project budgets leave other projects running. Project real company transitions in bounded lifecycle batches. - Replace periodic broad reconciliation with durable pending work, startup repair, receipt replay, and concrete retry/window deadlines. Retain drain and idle-safety checks. - Enforce accepted cumulative usage checkpoints without provisional charges. Final accounting atomically replaces live exposure and settles reservations. - Add operator-gated budget spans, regression coverage, map updates, and a real-PostgreSQL comparison harness. No schema or first-party telemetry change. ## Verification - Full workspace `pnpm -r typecheck` passes on the final ownership fix, rebased onto `bbcf5dfe98`. Production `pnpm build` passes on final head `4c5f4de8b1`. Map, module-boundary, token, and Storybook catalog checks pass. - Focused real-PostgreSQL coverage includes the 292-case rebased accounting/dispatch run, 63 browser cases, 52 run-control/cancellation cases, 42 policy-generation cases, 77 lifecycle/commentary cases, and the exact Muse confirmation regression. Native executor unit coverage passes all 768 cases. Final ownership coverage passes: 74 adapter/use-case cases, nine public regression cases, and the additional deleted-project race. These exercise competing ownership, budget changes and charges between phases, zero/nonzero duplicate claims, rollback, recovery contention, read-only capacity rechecks, and controlled HTTP startup after commit. Another 74 pure/startup/tracing cases pass. The complete CI matrix and both 5/5 reviews passed on `1ac9b58`; its external security scan identified a stale-owner handoff race. The final fix restores synchronous issue/run ownership fencing after the accounting transaction commits. Final head `8e84f55ce8` has fresh Greptile 5/5 with zero new comments, Paperclip review 5/5, and a passing security scan. The complete latest-head [CI matrix](https://github.com/paperclipai/paperclip/actions/runs/39535189808) passes, including budget, browser, Runner, build, and canary jobs. The unchanged Runner close fixture passed on its sole same-head job retry. - Broad local `pnpm test:run` invocations were stopped before required source updates (rebase and review fixes). They are not claimed as full passes. Final full-suite coverage comes from the complete latest-head CI matrix; local focused coverage and the incomplete broad invocations are reported separately. - Reproduce performance with `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-enforcement.ts --baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837`; repeat with `--reverse`. The harness uses disposable real PostgreSQL, 20 warm samples per variant/count, and records raw timings, SQL/lock counts, agent reads, unchanged agent state, and query plans. Final production-source measurement revision: `eff713a4f72c7f4b7a4002a701cb481d4bfccd95`; the extended harness is committed in `4c5f4de8b1`. | Unrelated idle agents | Baseline median / p95 | New median / p95 | Baseline SQL / locks | New SQL / locks | New agent reads | | --- | ---: | ---: | ---: | ---: | ---: | | 0 | 1.80 / 2.33 ms | 2.99 / 3.88 ms | 5 / 1 | 7 / 1 | 0 | | 17 | 15.53 / 19.55 ms | 2.24 / 2.60 ms | 90 / 18 | 7 / 1 | 0 | | 1,000 | 978.35 / 1,790.43 ms | 2.16 / 2.52 ms | 5,005 / 1,001 | 7 / 1 | 0 | - These measurements use local PostgreSQL 18.1 and Node 26.4.0 arm64. They isolate the cost-evaluation core and complete admission path, without hosted-region or provider latency. Prior-revision reversed-order controls also preserved the constant query counts; the final measurements above use the final production source. - Complete reserve → final ownership gate → immediate offline provider result is measured separately, with one responsible agent and one active operation at each unrelated-agent count. Admission p95 baseline/new: 0 agents **11.54 / 29.77 ms**; 17 agents **9.31 / 27.62 ms**; 1,000 agents **24.51 / 74.05 ms**. Baseline admission was already scoped; the stronger final fence adds bounded overhead. New counts stay constant at 53 Drizzle statements, three transactions, two company-accounting locks, and five agent-reading statements bound to the responsible agent. Zero unrelated-agent reads, exactly one provider callback per sample, and no leftover runs or holds. All admission scenarios stay below 250 ms. No external provider calls are included. - The partial-usage/delayed-termination fixture passed 20/20: receipt persistence p95 13 ms, receipt return to observed durable Stop fence p95 28 ms, complete receipt-start-to-fence p95 50 ms, unrelated database request p95 1 ms while termination was blocked. After releasing the controlled termination fixture, acknowledgement p95 was 14 ms. These are separately measured phases; fixture acknowledgement is not live-provider termination latency. - Models and provider deadlines are unchanged. No deployment, live inference, or staging acceptance is claimed. A quiet 30-minute follow-up waits for the merged revision on staging, then repeats classifier and controlled budget-stop acceptance in staging, recording hosted bookkeeping, queue delay, and actual provider acknowledgement separately. - Development failures retained: finalizer notification regressions (fixed; 27/27 affected cases passed), invalid browser admission fixture assumptions (fixed; browser coverage now passes 63/63), and first-head lifecycle/dispatch CI regressions (reproduced and repaired). A prior-head Cases E2E test hit its unchanged announcement-dismiss handler race while opening a filter; the same journey passed on `1ac9b58`. The external security review then found a stale-owner race between accounting commit and provider entry; final verification covers a competing run in that gap while retaining provider I/O outside the accounting lock. Early PostgreSQL startup attempts hit local semaphore pressure; serial reruns succeeded. The final-head Runner lane had one failure among 1,823 tests: the unchanged `timeout-attachment` fixture failed at its first process close after 6.28 seconds. The sole same-head job retry passed all 1,819 active tests (four skipped), including that fixture in 1.85 seconds. Three isolated local probes could not run correctly because the Rust fake-provider fixture binary was absent; cleanup masked the original startup failure. CI builds that fixture through its normal lane. These probes are inconclusive and are not counted as passes. ## Risks - This changes accounting, lifecycle, and cancellation ordering. Durable generations, source/sequence checks, ownership leases, and company locking remain required for correctness. - Provider stop acknowledgement can be slow or fail. Stop intent remains pending until existing termination proof succeeds; cancelled row status alone does not acknowledge delivery. - Mid-run limits depend on provider usage reports or supported price calculations. Providers that report only at completion cannot provide exact mid-call detection. - Signals are process-local wake hints for the supported single-active-writer model. Durable state and startup recovery remain authoritative. - There are no migrations, new queue tables, or cached balances. Repeat the classifier and controlled budget-stop acceptance after normal rollout. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution, and parallel coding agents. The host does not expose the exact backend model revision or context-window size, so those values are not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
f3af9887ca |
fix(chat): keep each chat conversation in its own wake batch (#17054)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat channels let a linked user talk to an agent from Slack, GitHub and other platforms. Each external thread is a chat conversation that is bound to one issue. > - One issue can have many conversations. A GitHub pull request review makes one conversation for each inline finding thread. > - When replies arrive behind an active run, the wake queue merges them into one deferred wake. The merge did not look at the conversation. > - A reviewed chat run can attest only one conversation. A merged batch from two threads always failed attestation with `setup_failed` / `reviewed_chat_execution_binding_not_authorized`, so no reply got an answer. > - This pull request keeps each chat conversation in its own wake batch. The single-conversation attestation check stays the same. > - The benefit is that replies in different review threads each get their own attested run, one after the other. ## Linked Issues or Issue Description Refs #16793 (related: chat origin during coalescing). No public issue exists, so the bug report follows. **What happened?** A linked user replied in three different inline review-finding threads of one pull request while the agent's review run was still active. The three wakes merged into one deferred wake. The promoted run failed in about 2 seconds with `setup_failed` / `reviewed_chat_execution_binding_not_authorized` and `providerWorkStarted=false`. None of the three replies got an answer. The failed-chat retry cannot recover this batch, because a retry source holds one conversation. **Steps to reproduce** 1. Bind two chat conversations (two GitHub finding threads) to one `in_review` issue. 2. Start a run for the assigned agent on that issue. 3. As the linked user, reply once in each thread. 4. Finish the active run. 5. See one promoted run that fails with `reviewed_chat_execution_binding_not_authorized`. **Expected behavior** Each thread gets its own run. Each run attests its one conversation and answers its reply. **Cause** `findExistingDeferredWake` matched on company, agent, issue and actor only. The active-run coalesce path had the same gap. `authorizeChatConversationForBoundRun` correctly denies a batch with more than one conversation. ## What Changed - `run-dispatch/domain/wake-context.ts`: add the `chatConversationId` wake-context key, a reader, and `wakeChatConversationsConflict`. A conflict means that both wakes have a conversation and the two conversations are different. - `issue-assignment-wakeup.ts` and `chat-channels/inbound/wakeups.ts`: stamp the conversation ID on each inbound chat wake. - `chat-channels/failed-run-retry.ts`: stamp the conversation ID on failed-chat retry wakes, so later replies do not merge into a retry from another conversation. - `wake-queue` admission: do not coalesce into an active run of another conversation. The deferred-wake lookup now matches only wakes of the same conversation or wakes without a conversation. - `heartbeat/queue.ts`: the same-scope queued or running coalesce target is skipped when the conversations conflict. - `doc/execution-semantics.md`: describe per-conversation chat wake batching. - Tests: an integration regression in the GitHub review block of `chat-channels.integration.test.ts`, and unit cases for the active-run branch in `wake-queue/application/use-cases.test.ts`. ## Verification - Red before the fix: `pnpm --dir server exec vitest run src/__tests__/chat-channels.integration.test.ts -t "replies in review finding threads"`. The main case fails with one run: `{ status: "failed", errorCode: "setup_failed", error: "reviewed_chat_execution_binding_not_authorized" }`. - Green after the fix, same command, 3 of 3 pass: - Replies in two finding threads give two successive runs. Each run succeeds, has `paperclipExternalChatExecutionBound: true`, and holds only its own comment and conversation. - Two replies in the same thread still give one run with both comments. - After the linked identity is revoked, each separated run is still denied (`setup_failed`) and the adapter does not execute. - `vitest run src/modules/wake-queue src/modules/run-dispatch`: all pass, with the new cases (different conversation, same conversation, run without a conversation, non-chat wake). - Related suites pass: `heartbeat-comment-wake-batching`, `durable-chat-wakeup`, `chat-failed-run-retry-boundary`, `issue-queued-comments-routes`, `heartbeat-stale-queue-invalidation`. - Full `chat-channels.integration.test.ts`: 823 passed, 1 timed out. The timed-out test (`GitHub terminal receipt composes pinned HTTP and durable source authority (held_token)`) passes alone with and without this change. The sandbox has a 4 GB memory cap. - Not run locally: `pnpm -r typecheck` and `pnpm build`. The server `tsc` process was killed at the 4 GB sandbox memory cap. CI runs both. - Runtime dependency note: the deferred-wake lookup adds one JSONB predicate to the existing deferred-wake query. It reads the same rows. The query count does not change. The integration regression runs it against embedded Postgres. ## Risks - Low risk. No schema change and no migration. - Behavior change: chat replies from different conversations on one issue now run one after the other, not in one batch. Before this change, such a batch always failed, so no working flow becomes slower. - Wakes without a conversation (non-chat wakes, and wakes queued before this deploy) keep the old coalescing. A deferred wake from before the deploy has no conversation key, so one later reply can still merge into it. That window closes when the queue drains. - The check keeps the single-conversation attestation in `authorizeChatConversationForBoundRun`. Revoked identities are still denied. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code through the Paperclip agent harness. It used tool use (shell, file editing, test runs) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
d20cb69ce5 |
fix(heartbeat): preserve active retries when comments arrive (#16855)
## Thinking Path > - Paperclip manages agents and their work. > - Heartbeat retries preserve task ownership after a recoverable failure. > - The issue read model includes retries that are scheduled, queued, or running. > - The human comment route treated every retry in that model as parked work. > - A normal status comment could therefore cancel a running continuation and return its task to Todo. > - This pull request limits comment supersession to parked retries and checks that state at the cancellation fence. > - The agent can keep working when a person adds context. Explicit Stop and Interrupt remain available. ## Linked Issues or Issue Description **What happened?** A Board comment on an in-progress task cancelled its running recovery continuation. The handler read a live retry from `getCurrentScheduledRetry` but did not check its status. The cancellation also forced workspace recovery when copy-back did not complete. **Expected behavior** A comment must preserve queued and running execution. Only a retry that is still `scheduled_retry` may be superseded. If promotion wins the race, the comment must keep the task in progress and follow the normal wake path. **Steps to reproduce** 1. Let a task retry advance from `scheduled_retry` to `running`. 2. Keep its retry reason and source run identity in the stored row. 3. Post a normal human comment, without an Interrupt request. 4. Observe the old handler cancel the run and move the task to Todo. **Paperclip version or commit** Reproduced on master base `9966f31bee`. The staging incident served `f3cd3f4ea89539b68410841d39ec15a5c626007f`. **Deployment mode** Staging server with a remote runner. The defect is in the core comment and cancellation path. Refs #16835 and #16843. Those repairs cover runner health and workspace preservation. This change prevents a normal comment from requesting the wrong cancellation. Related #9732 promotes human wakes on parked retries; it does not protect a retry that is already running. ## What Changed - Check the live retry status in both comment routes: POST comments and PATCH issue with a comment. - Add an optional expected-status fence to cancellation. A concurrent promotion or terminal outcome returns a no-op result. - Make due-retry promotion honor a cancellation claim under its issue and run row locks. A scheduler cannot promote work after cancellation has won. - Finish persisted cancellations of parked retries through the existing startup and periodic recovery sweep. Preserve their original request and retry a failed settlement without starting provider work. - Keep the task active when guarded cancellation does not win. Do not record a cancellation that did not occur. - Add route regressions, real PostgreSQL race tests, and pending-retry and explicit Stop safety coverage. - Exercise both normal HTTP comment routes with real services, stored Board membership, PostgreSQL, and a live child process. Assert that the process can still perform work, the task keeps its original execution, and no second executor starts. - Document the comment supersession contract and update the task feature/access maps with the local regression coverage and deployed acceptance recipe. ## Verification - Before the fix, all four POST/PATCH regressions cancelled queued or running work. - Before the fix, the real PostgreSQL race test cancelled a retry that became running after lookup. - The comment-route suite passes **103 tests**. The final dispatch, cancellation, restart recovery and HTTP/live-process suites pass **119 tests**, for **222 affected tests** across these runs. They cover comment delivery, both sides of the promotion race, concurrent terminal outcomes, pending retries, explicit Stop, and a live process that can continue work. - Command: `node node_modules/vitest/vitest.mjs run server/src/__tests__/issue-comment-reopen-routes.test.ts server/src/services/heartbeat/run-control.test.ts`. - The normal HTTP/live-process acceptance suite passes all **14 tests**, including both comment endpoints with stored Board membership and deterministic wakeup teardown. - Review exposed a second race after the cancellation claim. Its new PostgreSQL test failed before the dispatch guard with `promoted` and passes afterward with `not_promoted`, followed by completed cancellation. - A real PostgreSQL constraint interrupts the terminal cancellation write after its intent commits. The assembled recovery service failed to settle that row before the repair. Afterward a fresh service settles it, clears its task lock, cancels its saved wake, preserves the original request and leaves unrelated parked retries alone. A repeated database failure preserves the intent for a later successful sweep; a second successful sweep adds no duplicate event. - Full `pnpm -r typecheck` and `pnpm build` pass. Server typecheck and build pass again after the final server-only review fix. A local full-suite attempt started on `f8773bf` did not return an overall result; its log is preserved and no local full-suite pass is claimed. The new recovery change passes the focused suites and fresh server typecheck/build. Latest-head CI on `8f928adee67c089e1ac405d2ad3b169eedb8da27` is green: the full PR workflow and commitperclip review passed; only the two optional Storybook jobs skipped. - Greptile reviewed final head `8f928adee67c089e1ac405d2ad3b169eedb8da27` at **5/5**, with no unresolved findings. All actual latest-head CI checks passed; the two optional Storybook jobs skipped. - `node scripts/check-maps.mjs` passes after the map updates (34 areas, 172 capability IDs); it checks structure only. - Fixed-revision staging acceptance is pending deployment. The original workspace remains held and its latest source checkpoint is backed up and verified. ## Risks - A promoted retry must keep its execution and ownership when a comment arrives. The database status predicate and route no-op handling enforce this at the race boundary. - A cancellation intent must prevent promotion without leaving work parked forever after a restart. The recovery sweep consumes at most 20 such intents per pass through guarded cancellation, and a failed settlement keeps its intent for retry. - A parked retry must still accept fresh instructions. Its existing cancellation and wake behavior remains covered. - Ordinary Stop and Interrupt calls omit the expected-status option. Their cancellation authority is unchanged and remains covered by the run-control suite. - No schema migration or API request contract changes are required. ## Model Used OpenAI `gpt-6.1-sol` in Codex, with extra-high reasoning, tool use, and code execution. The session does not expose its context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7c7d2d8d2c |
fix(runtime): continue from saved sources after workspace loss (#16690)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The runtime must preserve saved work and prevent duplicate execution. > - Older cleanup could destroy a sandbox before its workspace export finished. > - The existing recovery controls require either the original sandbox or recovered original files. > - Surviving attachments can support a rebuild, but they do not prove that missing edits were recovered. > - This pull request adds an explicit reconstruction decision and one fresh continuation through the normal execution gates. > - Operators can continue useful work while the original failed run remains truthful. ## Linked Issues or Issue Description **What happened?** A historical failed export can have an accepted result, a prepared workspace descriptor, and a confirmed destroyed sandbox. Its notice can also say `new_source_execution_path` and disappear. The task then has neither a truthful original-file repair path nor a supported reconstruction path. **Expected behavior** Keep the missing-file obligation visible. Let an authorized operator choose surviving task attachments and record lost work and external action outcomes. Start one fresh continuation after current execution checks. Keep the old result and failed export intact. **Steps to reproduce** 1. Seed a failed native export with a prepared descriptor and an exact destroyed-lease receipt. 2. Resolve its notice with the historical `new_source_execution_path` bookkeeping value. 3. Read the task. The baseline hides the hold even though no source reconciliation exists. 4. With this change, choose **Continue from saved sources** and submit a reconstruction decision. Related public work: #16451 preserves healthy warm sessions. #16583 prevents destructive cleanup of unexported work. This change handles historical source loss that those fixes cannot undo. Searches of open and closed work found no equivalent reconstruction endpoint. ## What Changed - Add a board-only reconstruction endpoint with `runtime:manage` and task visibility checks. - Validate the exact run, result, descriptor and destroyed lease. Reject active copyback, competing owners, newer work, stale task status, and foreign attachments. - Record attachment hashes, missing work, and external action outcomes. Preserve the original failed run, accepted result, prepared descriptor and terminal failure. - Hold other continuation paths until the durable reconstruction intent admits its successor. Record that successor in the same transaction and preserve the retry count. - Add an operator control that distinguishes reconstruction from original-file recovery. Update the recovery runbook and capability maps. ## Verification - Exact baseline source returns no blocker for the destroyed-source fixture. The same fixture returns a required recovery hold with this change. - New real PostgreSQL and assembled API/heartbeat suite: 25/25 pass. It covers duplicate submissions, lost acknowledgement, context reset, source identity, task ownership, concurrent execution attachment scope, later human blocks, deleted sources and agent pause. - Existing export retry, explicit continuation and recovery-action suites: 386 cases pass. One new-fixture constraint error in the combined run was corrected; the full new suite then passed. - UI recovery suites: 46/46 pass across three files. They cover disabled actions, loss acknowledgement, failure retention and runtime authority. - `pnpm -r typecheck` and `pnpm build` pass. The first sandboxed typecheck attempt could not open the TypeScript IPC socket; the authorized rerun passed. - Module boundaries, map checks, token gates and diff whitespace checks pass. - OpenAPI route coverage: 21/21 pass. The first CI head exposed a missing endpoint registration; this head adds the registration and the focused suite passes. - Review identified a missing dispatch-time check and an isolated attachment cache. Both are fixed. Claim and final handoff recheck the exact successor, sources and current execution gates. A controlled post-claim operator block prevents the assembled heartbeat from entering the provider. The shared attachment cache refreshes uploads and removes deleted selections. - Final focused run: 180/180 pass across six suites, including reconstruction, existing dispatch/stale-queue contracts and UI recovery. - A second review reproduced cancellation by an already satisfied historical connection card while the task was `todo`. Read-only connection availability now works before checkout; active-task mutation checks remain strict. The baseline failed two cases; after the fix all 74 connection and reconstruction tests pass, including revoked access, pending questions and pending approvals. - All current-head CI gates pass on `9d197236da11fa24f1ceadf5589cb2b81179d824`, including general/serialized tests, Runner, build and browser gates. Greptile is 5/5 for that exact head with no unresolved threads. The full local `pnpm test:run` is still running its serial server group; no full local pass is claimed. Live staging recovery is pending deployment. ## Risks - Reconstruction cannot restore missing bytes. The operator must identify surviving sources and actions that must not be repeated. The agent must verify those attachments before rebuilding. - This path is limited to exact, confirmed destroyed remote sources with an accepted result. Retained or uncertain sources keep their existing repair path. - No schema migration, credential change, budget increase or warm-session policy change is included. Normal pause, ownership, budget, approval and dispatch gates still apply. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The session does not expose an exact backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; full local run limitation recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
5847f047c7 |
fix(connections): retire requests for already available tools (#15856)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connection cards let users provide the access an agent needs. > - Users can complete setup in Apps while an earlier card stays pending. > - A stale card can then cancel recovery despite valid existing access. > - This change verifies existing access and expires the obsolete request. > - It records a system outcome and sends one durable continuation. > - Approval, identity, permission, and workspace recovery gates remain in place. ## Linked Issues or Issue Description **What happened?** An ordinary tool connection request stayed pending after the addressed user connected the service in Apps. A later conversation retry was cancelled with `issue_waiting_for_response`. **Expected behavior** A request expires when the same user and requesting agent already have usable access. The agent gets a durable tool refresh. The system records no new human approval. **Steps to reproduce** 1. Have an agent request a tool connection on an active task. 2. Connect the service through Apps with the addressed user's existing grant, agent install, and allowed tools. 3. Let a conversation run stop and retry. The pending card still blocks the retry on the base branch. Related: #12345 introduced the connection request flow. This fix uses its existing access checks and delivery queue. ## What Changed - Verify company, task ownership, user identity, membership, health, grants, installs, and task tool policy before expiring an ordinary request. - Preserve AI account, channel, additional permission, and upstream provider requests on their existing paths. - Save the system outcome, audit event, and continuation in one transaction. Notify an idle delivery worker when a request commits. - Recheck access and other questions or approvals before delivery, retry, final dispatch, and native projection. Refresh system observations for an authorized replacement account. - Rearm delivery when access changes before provider work starts. Verify the host's admission receipt against execution evidence before allowing retry without a repair incident. - Require the current delivery claim when marking an attempt complete. An old completion cannot overwrite a rearmed continuation. - Add embedded-Postgres regressions for permissions, policy, concurrency, restart, replacement accounts, both dispatch gates, and the delivery race. Keep genuine workspace repair holds blocking. ## Verification - Latest-head connection, wake-context, dispatch, and legacy recovery suites: 124 tests passed. The connection suite includes 44 real-Postgres cases. - Server typecheck passed. Full `pnpm -r typecheck` and `pnpm build` passed on the latest head. - `pnpm test:run` was run locally. Its general-server group passed 18,583 tests and failed 20. That long run started before the review fixes, then loaded new test cases against modules transformed before those fixes. The affected latest-head suites pass in a fresh process. - One full-run failure also reproduces on unchanged base commit |
||
|
Devin FoleyandPaperclip
|
06484b3c41 |
fix: preserve conversation retries through execution cleanup (#15463)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery schedules bounded retries after a provider disconnects. > - A stopped run can still hold its environment lease while cleanup runs. > - Retrying before that lease is released cancels the new run before it starts and spends another retry. > - Restoring the task can also send the worker repair instructions from an already resolved recovery action. > - This pull request preserves the waiting retry and removes settled recovery instructions from later wakes. > - The task can continue after cleanup without an operator repairing the same incident again. ## Linked Issues or Issue Description **What happened?** A legacy conversation run disconnected while it was doing ordinary work. Its environment cleanup took longer than the retry delay. Two retries were cancelled before dispatch with `execution_reconciliation_required`. Those cancellations exhausted the failure budget. After an operator restored the task, the wake still told the original worker to repair the runtime and hand the task back to itself. **Expected behavior** Cleanup waits preserve the pending attempt. The same retry can continue after ownership is released, subject to all current gates. Once a recovery action is resolved or cancelled, subsequent task wakes omit its repair instructions. **Steps to reproduce** 1. Fail a legacy conversation run while its environment lease remains in `pending_cleanup`. 2. Schedule a bounded retry and run promotion before cleanup releases that lease. 3. Repeat the scheduler sweep. Before this fix, retries promote and then cancel without starting. 4. Resolve a stranded-task recovery action and build the restored task wake with that action ID. Before this fix, the wake still includes the settled repair instructions. **Paperclip version or commit** Reproduced with database regressions against `ceabc3bc880` on master. Related: #15019 restores a skipped assignment handoff after lease release. #15235 filters stale handoff evidence in the recovery sweep. This change preserves an existing scheduled conversation retry and corrects restored wake content. It does not create a new handoff wake. ## What Changed - Keep an unstarted legacy conversation retry on the same durable row while prior execution ownership remains active. Recheck after 30 seconds without increasing retry accounting. - Return a queued retry to scheduled state if it encounters that hold at the claim gate. Retain its issue claim and publish the status change. - Record one local lifecycle diagnostic per blocking run. Remove that wait marker on promotion. - Include recovery action metadata only while the referenced action is active or escalated. - Return an explicit `waiting` response and the saved schedule when Retry now meets cleanup. Show the wait inline without a false success or disabled button. - Add database, rendered-prompt, route, and UI regressions. Document the execution and run-log contracts. ## Verification - Five cleanup and restored-wake regressions fail against the original production code. The Retry now route and UI regressions also fail before their correction. - Related retry, dispatch, stale-queue, and recovery suites: 321 tests pass across seven files. All ten focused cleanup/restored-wake cases pass after rebase. The final dispatch adjustment passes all 46 adapter tests. - Retry now routes and affected UI suites: all 46 tests pass. The tests cover repeated clicks, the saved schedule, unchanged accounting, promotion after release, and no false success or error state. - `pnpm -r typecheck`, `pnpm build`, `pnpm build-storybook`, and `pnpm check:token-gates` pass. - Full local `pnpm test:run` was started and then stopped after the final commit passed all GitHub CI test shards. No complete local full-suite result is claimed; CI supplies the complete test result for the final commit. - Final head `6f1058a332c039e33c4f002b296d20a5554e760e`: all 55 GitHub checks are green or intentionally skipped. Apex review is 5/5 after two reviews, with no unresolved threads. The PR has no merge conflicts. ## Risks The wait applies only to unstarted legacy conversation retries. Native runs and non-conversation execution keep their existing recovery rules. Cleanup must actually release ownership before execution can resume. The wait does not fix a cleanup service that never finishes. Promotion and dispatch still enforce cancellation, reassignment, pause, budget, and reconciliation gates. No schema change is required. The Retry now response adds a `waiting` outcome; the shared contract and all three UI controls handle it. ## Model Used OpenAI Codex based on GPT-6, with repository analysis, tool use, and local code execution. The runtime does not expose the precise serving model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; complete final-head test coverage in CI) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
799e4d556f |
fix: make accounting durable and synchronize cost reporting (#14997)
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f2715e02bb |
fix(native): surface model capacity errors and retry automatically (#15347)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs preserve provider events and results, then decide task state. > - Codex can fail a turn because the selected model is temporarily at capacity. > - Paperclip displayed a generic native-session failure and left the task in recovery. > - A known capacity failure needs a clear message and a durable delayed retry. > - This pull request uses committed terminal evidence, the status-effect ledger, and existing dispatch gates. > - The task can continue automatically without an unbounded retry loop or a model switch. ## Linked Issues or Issue Description Related: #13993 adds launcher capacity deferrals before provider startup. This change handles a committed native Codex `serverOverloaded` terminal after provider work has started. **What happened?** A native Codex turn ended with `codexErrorInfo: serverOverloaded` and “Selected model is at capacity. Please try a different model.” Paperclip saved the result, showed a generic failure, and required recovery instead of waiting and retrying. **Expected behavior** Show the capacity error directly. Schedule bounded retries after a short delay. Preserve saved work and honor current execution gates. **Steps to reproduce** 1. Run a native Codex task. 2. Commit a `turn.failed` event with `serverOverloaded`, then accept a failed result for the same turn. 3. Inspect the task error and its recovery state. The regression test reproduces this without a paid provider call. **Paperclip version or commit** Reproduced against master `16b7db35ffa0f9a95913c8cbdeea3d595435691f`. ## What Changed - Classify capacity failures from committed runner events and the pinned execution identity. Preserve accepted results and display the specific capacity error. - Atomically persist one scheduled successor with the status decision. Retry after one minute, then two minutes. Share the existing failure budget and stop after two automatic retries. - Reuse dispatch gates for ownership, task holds, dependencies, budget, and locks. Wait for predecessor execution, finalization, and cleanup before claiming a retry. - Preserve pending reviewer authority and suppress retries after reassignment or a successor claim. - Consume the failed run's resume receipt and delivered wake input. Rebuild ordinary continuation from the failed run so explicitly resumed tasks can retry without borrowing one-run authorization. - Label scheduled retries “Model at capacity.” Add a Storybook example and avoid duplicate punctuation in the existing retry card. - Add regression coverage and document the runtime contract. ## Verification - Targeted recovery and UI suites: 125 tests passed. Additional final cleanup and lock regressions passed. - Latest-head continuation and authorization regressions: 57 tests passed, including all four capacity integration tests against an isolated PostgreSQL database. - Rechecked all four capacity integration tests with the full runner's isolated `PAPERCLIP_HOME`, config, temporary directory, and serial fork settings: passed. - `pnpm -r typecheck`: passed. Final server typecheck passed. - `pnpm build`: passed. - `pnpm check:token-gates`: passed. - Browser: checked the real Storybook card. It shows the capacity message, automatic retry time, and existing Retry now action. - `pnpm test:run`: attempted and restarted after an interruption. The resumed run started before the final review correction and was stopped after recorded workspace/native test failures and the five-minute Git streaming timeout already documented on master. Exit 130; no complete local full-suite pass is claimed. Current-head isolated capacity tests and the complete CI suite pass. - A combined local recovery-suite attempt also hit PostgreSQL initialization failures at this macOS host's global shared-memory limit (32 slots). The focused recovery suites passed separately; final-head capacity tests also pass with the full runner's isolated environment settings. - Latest-head GitHub checks: all 56 checks green, including general and serialized test shards, browser E2E, typecheck, build, Runner verification, and canary dry run. No merge conflicts. - Greptile: 5/5 on `813a470b8c18c05aeb7e31e63e573c3a3a2f5cac`, with zero unresolved findings after fixing the resumed-task continuation issue. ## Risks - Capacity retries can repeat a task turn after partial work. They start a fresh provider session with task history and wait for predecessor cleanup. They do not resume the failed turn. - Retries retain the configured model unless an operator changes configuration. Persistent overload consumes the existing failure budget and then needs an explicit retry or model change. - Only run-bound native Codex `serverOverloaded` failures qualify. Usage-limit exhaustion, model/account incompatibility, unbound text, and unknown provider failures retain their current recovery behavior. - No schema migration is required. ## Model Used - OpenAI GPT-6 through Codex. The exact model ID and context-window size are not exposed to this session. Used reasoning, repository tools, code execution, and browser verification. No subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0e0b63e5a5 |
feat(connections): add experimental task-pinned AI routing (#14967)
## Thinking Path > - Paperclip manages AI agents and their work. > - AI Connections separate account access from models and harnesses. > - A pool must act as one connection while retaining each task’s account. > - Core must enforce member access and preserve session and recovery rules. > - A plugin supplies rotation policy without receiving credentials. > - This change adds durable routing and native connector setup and management. ## Linked Issues or Issue Description **Subsystem affected** AI Connections, Connectors, plugins, run dispatch, and session compatibility. **Problem or motivation** Operators need to rotate new tasks across saved accounts while each task keeps its account and session. Pool setup must fit the existing connector catalog and account workflow. **Proposed solution** Add an experimental router binding, a capability-gated plugin hook, and transactional task pins. Plugins declare native pooled connectors through `aiConnectionRouter`. Core hosts the existing-account picker, ordering step, and account settings. Related usage contract: #14936. Companion private plugin: https://github.com/paperclipai/paperclip-cloud/pull/643. **Roadmap alignment** This extends Apps and AI Connections. Core supplies generic enforcement and native connector UI; the private plugin owns rotation and quota policy. The prior duplicate search found no matching router implementation. ## What Changed - Add a router binding without changing existing concrete bindings. Keep the instance flag and new pools disabled by default. Require manual operator configuration. Show no routing toggle in Experimental settings on either open-source or Cloud installs, even after routing is enabled. - Persist company-scoped pools, one shared cursor per pool, and pins keyed by company, pool, agent, and task. Commit pins and cursor advances together with revision checks and bounded retries. Persist run-ID affinity before allocation. - Pass only authorized metadata and normalized usage to plugins. Core retains credential handling, member access checks, runtime qualification, and recovery evidence. Probe outside locks with a shared 15-second budget and freshness cache. - Resolve routing before credential preparation and backend selection. Preserve pins through turns, session resets, removed members, and quota waits. Retain admitted recovery after disable or uninstall. - Separate credential session epochs from token generations. Verified refresh preserves the epoch; reconnect and manual replacement change it. Include the credential slot ID in session and usage-cache identity, so reconnecting an indexed legacy account invalidates its old session even when both epochs are zero. - Validate pool member installations before accepting saved-agent bindings and recheck compatibility when the harness changes. Install only authorized members in the new-agent transaction and record their IDs in local activity. Pool membership cannot install a restricted shared connection. - Preserve pool bindings when agents hire teammates through either creation API or native caller runtime inheritance. Block stale manager credential references; retain explicit child authentication precedence and reject incompatible inherited pools. - Add native connector registration through plugin metadata. Reuse the Connectors catalog, setup header, account header, sidebar, dialogs, and usage display. Setup selects and orders saved connections. Advanced settings hold usage rules and member runtime defaults. New-account setup opens in another tab. - Use revision-checked pool archival from the Connectors catalog and account page. Keep task pins, cursors, recovery evidence, and underlying connections. Reject ordinary connection updates or removals that bypass pool revisions. - Add pool selectors, composer models, override notes, quota status, run details, activity records, and local run-log records. Keep session-adoption copy minimal. - Show **Used by** below the pool connections. List current company agents with shared avatars and profile links. Include paused agents; exclude terminated agents and agents using another pool. - Add Core stories for the generic connector workflow and runtime surfaces. Cloud stories reuse these production routes and tokens through a preview-only alias. ## Verification - Final head `73cb953bca30ed83e4505dd820edd9b5edffd28b`: full workspace `pnpm -r typecheck`, `pnpm build`, and `pnpm check:token-gates` pass locally. - All 422 focused connector/settings/shared-contract/migration tests and all 156 database-backed AI connection, hiring, reconnect, and durable-routing cases pass (69 hiring cases rerun after the final auth-precedence fix). The merged shared contract retains connection instructions and pool metadata. The pool migration is generated at sequence 0299 after the latest upstream migrations; this PR makes no lockfile changes. - All four full-app Playwright tests pass on the final head after a cold restart and migration, against the installed private plugin and isolated database, with no route or pool-API mocks. They cover hidden routing controls after manual opt-in, native pool creation, ordering, membership edits, rename, paused defaults, enabling/save/refresh persistence, stale edits, cancellation/removal, preserved underlying accounts, unavailable routers, and Used by avatars and profile links. Exact command: `PAPERCLIP_CONNECTION_POOL_E2E=1 AI_CONNECTIONS_TEST_COMPANY_ID=a37b9625-5ecf-4e29-8081-04df3d6e7d6f AI_CONNECTIONS_TEST_URL=http://127.0.0.1:3108 pnpm exec playwright test --config tests/ai-connections-app/playwright.config.ts connection-pools.spec.ts`. - [Native setup, ordering, and management screenshots](https://github.com/paperclipai/paperclip/pull/14967#issuecomment-6006976278) address the review follow-up. [Earlier selector, quota, and run-detail screenshots](https://github.com/paperclipai/paperclip/pull/14967#issuecomment-5971537316) show the runtime surfaces. Core previews: `pnpm --filter @paperclipai/ui storybook`, then **Connectors / Pool host** or **AI Connections / Connection pools**. Cloud owns its host-backed plugin stories; both repositories’ Operator Setup Required story assertions pass. - Live acceptance used OpenAI/Codex and Anthropic/Claude ACPX, resumed both exact sessions after restart, preserved pinned accounts through explicit reset and controlled quota deferral/recovery, and committed only two allocations across fourteen runs. A later UI-created task test again rotated OpenAI then Anthropic and resumed OpenAI through follow-up/restart/quota recovery. That later Anthropic execution was blocked by its saved OAuth token expiring (provider 401). No live usage probes ran. - The full local `pnpm test:run` was attempted earlier and did not complete because of macOS embedded PostgreSQL bootstrap/shared-memory failures and the 40,000-file Git fixture timeout. The focused database suites above now pass; full-suite verification is provided by the split CI lanes. The preceding CI run had one runtime readiness timeout; it passes locally both alone and inside the larger runtime suite. That larger local suite also encountered an embedded PostgreSQL setup failure and two macOS temporary-path alias assertions; those two assertions pass with canonical TMPDIR=/private/tmp. All final-head CI checks are terminal green, including full general/serialized server suites, Runner checks, browser E2E shards, canary verification, build, and typecheck. Greptile is 5/5 on that exact head with no unresolved threads. ## Risks - The migration adds routing tables and a credential epoch column. Install the private plugin only with the compatible Core contract. - Routing and each pool require opt-in. Production distribution and fleet defaults remain unchanged. - Unknown usage stays eligible. Known pinned exhaustion waits; revoked access requires operator repair. - Legacy adapters require compatible members. Runner model and effort overrides remain limited by qualified backend support. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository editing, code execution, and browser testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted suites; full-suite limitations are reported above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
83076d7e7c |
feat: return completed handoffs to Agent Chat (#14408)
Return completed Agent Chat handoffs through a durable outbox and scope each generated update to its supplied tasks. Add recovery, browser delivery, result access, and calibrated quality coverage. Validated with two consecutive ten-case Claude/Codex campaigns, all CI checks, and a 5/5 review. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
96bf004a79 |
fix: use persisted state for lifecycle continuation and retry budgets (#13888)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its control plane decides when a task can continue, wait, stop, or complete. > - Legacy continuation could change when an agent changed its wording without changing task state. > - Shared attempt counts also let repair and infrastructure retries affect each other's limits. > - This pull request uses persisted state and separate, bounded allowances for these decisions. > - If automatic repair stops, the task explains what happened and offers a guarded retry. > - Paired tests and real-provider evaluations verify that Stop, approvals, ownership, and spending limits remain authoritative. ## Linked Issues or Issue Description Related work: Refs #13761, Refs #11126, Refs #13610. These cover obsolete continuation dispatch and retry storms. Open and closed issues and PRs were searched for related lifecycle, continuation, and retry work. **What happened?** Legacy continuation depended on English wording and progress heuristics. Repair, failure retry, and productive continuation could consume shared counts. When bounded repair stopped, the task showed a technical recovery message without a clear next action. **Expected behavior** Persisted disposition and owned execution paths determine the next action. Missing disposition prompts bounded agent repair. Explicit work mode determines planning mode. Narrative changes and raw activity counts cannot replenish allowances. An exhausted repair shows a readable notice. An explicit retry checks current controls and preserves the assigned agent. **Steps to reproduce** Run `pnpm test:lifecycle-baseline`. The paired probes keep structured state constant while varying completion, planning, blocker, and progress prose. Run the explicit `lifecycle-baseline` and `continuation-accounting` Product E2E suites for real-provider coverage. In Storybook, open **Design previews / Recovery notice** to inspect the production component's normal, pending, acknowledged, unavailable, failure, and mobile states. ## What Changed - Hide the image attachment button, icon, and drop/paste hint in answer composers. Image paste and drop support remains available. - Merge current master and retain both browser regression sets. Use a production-stamped service worker in the offline recovery browser fixture. - Share one state-based legacy continuation decision across immediate, delayed, and recovered dispatch. Bind bounded repairs to their source run and episode. - Remove title and description wording from work-mode authority. Agents can still write requested plans in execution mode. - Persist separate failure-retry and productive-continuation counters. Disposition repair and resource waits cannot consume or reset those allowances. - Validate delayed repair identity, then recheck current gates before provider dispatch. Fence native startup cancellation. - Show **Agent needs attention**, a plain-language explanation, **Retry agent**, and expandable details in both task interfaces. Report request progress, acknowledgement, and errors inline. - Store typed recovery notice metadata. Recognize older active notices only through exact stored action and run IDs. Notice text never grants retry authority. - Use the existing recovery-action endpoint for retry. Recheck current action, status, owner, agent availability, dependencies, active runs, pending questions and confirmations, approvals, pause controls, and budget. Duplicate requests do not wake twice. - Add component, page, route, database, contract, and Storybook coverage. Keep the scenario inventory and executable evals here. Historical reports and snapshots live in the [commit-pinned paperclip-evals archive](https://github.com/paperclipai/paperclip-evals/blob/ce3e5afcd4a1184650f586a2b5b8be5874c66c8b/experiments/2026-09-lifecycle-authority/README.md). - Preserve unsaved project fields while the same project URL changes to its canonical alias. Do not reuse data across projects or companies. This separate fix addresses the repeated repository-editor browser failure without changing the browser test. - Keep the development service worker from intercepting Vite module reloads. Update the connection-intent browser fixture to record progress and completion through the agent API. ## Verification Merge preparation on September 25, commit `c1e8e4b7ddd9fbc4913ed55ce21b8e12906c2f97`: - Merged master `bd2030932` and resolved the browser test-list conflict by keeping both sets of regressions. - Deterministic lifecycle baseline: 1,090/1,090 assertions passed; no failures, skips, or missing selected evidence. Unit 423, runner 184, database integration 397, grading 86. - Browser support: 17/17 passed. The offline recovery test first failed with an unstamped development worker, then passed with the production stamp. Its assertions are unchanged. - Focused interaction UI and offline fallback tests: 19/19 passed. Verified the custom-answer composer in Storybook: no attachment controls or hint; entering an answer enables Next. - Recursive typecheck, production build, token gates, and diff checks passed. The worktree is clean. No new real-provider campaign was run. - Current CI and review: [Current PR CI passed](https://github.com/paperclipai/paperclip/actions/runs/36166011243): 55 successful checks and two optional Storybook skips. Greptile scored this exact commit 5/5. Hiding the question attachment controls is an intentional UI change; paste/drop remains available. Earlier recovery UI verification, commit `21be0fec0e90e86b6d662b8ee4831847cd041cdb`: - Recursive typecheck, production build, token gates, and diff checks passed. - Focused UI coverage: 338 tests passed across six suites (336 before the interaction guard, with the two affected suites rerun at 149 passed after it). Covers both task interfaces, the real page mutation, pending/error acknowledgement, stale state, and unavailable controls. - Recovery database integration: 352 tests passed before the interaction guard. The complete recovery-action and mutation-route suites passed 181 tests after it. The two new pending question/confirmation regressions failed before the fix and passed afterward, including resolved-interaction controls. Shared validator suite: 31 passed. E2E catalog suites: 34 passed. - Browser inspection passed for light/dark themes, mobile layout, expandable details, pending retry, acknowledgement, failure, and disabled retry. Storybook renders the production component; its request is simulated. - The broad local run hit two chat callback-order wait failures and was stopped after all CI unit/database/runner shards passed. Both local failures passed when rerun without the competing full-suite process. - CI exposed a repeated project-repository draft-loss race during canonical redirects. A new unit regression failed before the fix; all nine project-page tests now pass, including controls for other projects and companies. Both unchanged repository browser tests passed against a fresh local server. UI typecheck, production UI build, and token gates passed after this fix. - [Earlier PR CI passed](https://github.com/paperclipai/paperclip/actions/runs/36072486798) on `21be0fec0e90e86b6d662b8ee4831847cd041cdb`: 55 successful checks, two optional Storybook skips, and no failed or pending checks. The repository browser shard passed with the production fix. Greptile is 5/5 on this exact commit with no unresolved review threads. The PR is mergeable. Historical, source-qualified lifecycle evidence: - Lifecycle baseline: 1,074 assertions. Native session coverage: 447 tests. Product E2E support: 515 tests. Browser support: 11 tests. Full earlier verification is retained in the archive. - [Real-provider campaign: 8/8 passed, zero retries](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35881382080-1/index.html), source `e88d210417280140b44a36449027290adcb1aeaa`. Evidence and cleanup checks passed. This includes deliberately exhausted repair cases that correctly remain blocked; it does not mean every task finished Done. This campaign predates the recovery UI change. - Archive migration verified all 16 original JSON files byte-for-byte and all 24 checksum entries. App tests do not need private archive access. [Archive PR #27](https://github.com/paperclipai/paperclip-evals/pull/27) is merged. ## Risks - Agents that omit durable disposition receive at most two repair attempts by default. Prose-only completion exposes missing state rather than silently changing scheduling. - A retry is an explicit board action. The server rechecks current controls. A successful response confirms the task returned to To do; it does not claim that the provider has already started. - Existing notice metadata remains valid. Only older active notices with matching structured evidence receive the new UI. Historical notices without that evidence keep their existing rendering. No schema migration is required. - Old run records require conservative retry accounting. Tests cover old counters, alternating retry lanes, restarts, and exhausted repairs. - Historical snapshots require private `paperclip-evals` access. The app index retains public campaign links. Live campaigns qualify specific sources and scenarios; no new real-provider campaign has run for the recovery UI commit. > This fixes existing lifecycle and recovery behavior and does not duplicate planned core work. ## Model Used OpenAI GPT-6 through Codex assisted implementation, reasoning, code execution, and review. The exact serving model ID and context window are not exposed in this task. Historical real-provider evaluations used Codex model `gpt-5.6-sol`, separately from the implementation assistant. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
84fe89906d |
fix: complete native agent review handoffs (#13581)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native execution uses durable runs, issue locks, wake requests, and typed tool authority > - A child can finish with a native agent review request while its original assignee stays responsible for the work > - The reviewer then needs a bounded execution path that can inspect the child, record one decision, and finish safely > - Before this change, assignee-only gates rejected the reviewer or left the parent waiting after the child review ended > - This pull request adds typed reviewer admission, scoped reviewer tools, durable wake and recovery handling, and parent continuation evidence > - The benefit is that native review handoffs complete without changing child ownership or granting broad mutation access ## Linked Issues or Issue Description Refs: #13314 Refs: #13574 **What happened?** A native child run could report `needs_review` for an agent reviewer. The reviewer wake then failed assignee and execution-lock checks. The child remained in review and the parent remained waiting. **Expected behavior** The named reviewer should receive one durable wake. The reviewer should inspect the child and resolve the exact review card. The child assignee should stay unchanged. The parent should receive the recorded review outcome after the child reaches its terminal state. **Steps to reproduce** 1. Run a native task with a different named agent reviewer. 2. Keep the child assigned to its original worker. 3. Let the worker finish with a native completion review request. 4. Start the durable reviewer wake. 5. Resolve the review and finish the reviewer run. 6. Observe the child and parent state. **Paperclip version or commit** Base: `e926b1301`. PR head: `b31ad9ab8`. Live reviewer verification source: `eea171aae`. **Deployment mode** Built from source. **Installation method** Built from source (pnpm build). **Agent adapter(s) involved** Not adapter-specific (core bug). **Access context** Both. **Database mode** Embedded PostgreSQL in the isolated live test fixtures. ## What Changed - Add server-validated native review assignment facts. - Admit only the exact company, issue, source run, decision, revision, addressee, and resolver policy. - Give reviewer runs a narrow set of Paperclip read and resolve tools. File and shell access follow the configured agent and environment policy, so reviewers can run tests. - Separate server-owned reviewer instructions from untrusted persisted review data. Escape the data boundary; retain server-enforced authorization. - Keep the child assignee unchanged. Atomically claim the reviewer run, wake request, and issue execution lock. A competing lock prevents provider startup. - Require the exact running reviewer session and current issue lock to resolve its assigned card. Reject missing, unrelated, or terminal reviewer runs. - Add durable reviewer wake, lock, stale-card, and abandoned-run recovery handling. - Prevent duplicate native wake dispatches during deferred admission and recovery. - Carry accepted or rejected child review outcomes into parent task context and continuation evidence. - Add focused server, runner, and native protocol coverage. - Preserve upstream continuation rules. Add child review decisions as separate evidence, while keeping real human answers in their own field. - Return actionable completion validation feedback to both providers. Permit a corrected completion after rejection. Keep strict terminal acknowledgment validation. - Apply exclusive shared-workspace locks to sandbox environments. Local and SSH folders can run concurrently, including when old settings request serialization. - Repair test timing, native event parsing, and the review artifact assertion. Allow a valid reject, correct, and accept review sequence. Check the accepted card against its reviewer run and decision. Keep polling within the existing deadline when review acceptance precedes the parent wake projection; report a specific missing-continuation error at timeout. - Apply the ACPX pending-call limit to reserved finish/block calls, with capacity-release and cancellation tests. ## Verification - `pnpm build`: passed on `eea171aae`. - `pnpm -r typecheck`: passed on `eea171aae`. - `pnpm test:e2e:runner:unit`: 359 tests passed in 30 files on `b31ad9ab8`; runner E2E typecheck also passed. - `pnpm check:token-gates`: passed. - Focused DB review, reviewer authority, and prompt-boundary checks: 31 tests passed. They cover invalid reviewer runs, competing locks, atomic admission, duplicate claims, and valid resolution. - Heartbeat, workspace, and recovery checks: 30 tests passed. - ACPX sidecar suite: 27 tests passed. Moving the capacity guard back below reserved handling makes both new regression cases fail. - Four focused live continuation checks passed on their first attempt at `f15f55e0a`: answer updates scope (6/6 each on Codex and Claude) and question tool guidance (12/12 each). These cases do not use the reviewer prompt path changed afterward. - Fresh Codex and Claude review-handoff checks passed all 29 native checks each on their first attempt at `eea171aae`. Both runs received the expected fixed prompt and completed cleanup. Only the six selected live flows were tested; no full paid provider catalog run. - The final commit only extracts the existing test-harness timeout diagnostic into a shared helper and adds positive and negative coverage. Removing the accepted-review guard makes two regression assertions fail; restoring it passes all six timeout tests. Production runtime code, prompts, deadlines, and grading criteria are unchanged by this final commit. - Deadline regressions: a valid continuation delayed 20 seconds succeeds within its 30-second unit-test deadline; an absent wake returns a specific candidate-failure diagnostic at that same deadline. Both assertions failed before the fix. Production E2E deadlines remain unchanged. - Historical native failures remain recorded: Docker availability failures; a valid reject/correct/accept sequence that the first-card grader misread; and a test that rejected the gap between accepted child review and parent wake projection. No failed result was regraded. The latest tests use a protected reference to the pinned Docker image and the unchanged artifact oracle and time limits. - Full repository verification runs in GitHub CI. Local verification uses the focused suites above, full build, and full typecheck. An unchanged Codex shutdown timing test failed once in CI, passed in isolation, and its full shard passed on the final commit without changes to that test or its causal code path. The original failure is retained in the verification record. Greptile reviewed `b31ad9ab8` at 5/5 with no outstanding actionable findings. All review threads are resolved. All current-head CI gates passed, including the isolated native runner Docker build (55 successful checks; two skipped by the workflow). ## Risks - Reviewer admission depends on exact persisted decision and interaction bindings. A stale or changed card is rejected. - Paperclip control-plane tools are limited to inspection and review resolution. This is not a filesystem permission boundary; provider file and shell access retain the configured policy. - Deferred wake recovery changes dispatch receipt coalescing. A scheduler regression could delay a continuation if the receipt state is wrong. - Parent review outcomes are evidence for the model. They do not grant tool authority or change issue ownership. - This change does not address legacy lease-hold handoff behavior. > Roadmap review: native execution, review gates, and durable recovery are existing roadmap capabilities. This PR completes a narrow reliability path for those capabilities. ## Model Used OpenAI `gpt-6-astra` with reasoning, tool use, and code execution. OpenAI `gpt-5.6-luna` assisted with bounded implementation, review, and journal work. Context window size is not exposed by this session. Live test subjects use `gpt-5.6-sol` and `claude-sonnet-5`; they are not the PR authors. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0e9b24c821 |
fix: resume subscription comment wakes and preserve retry status (#13433)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Concurrent task runs can share a managed AI subscription with one credential lease. > - #13438 added durable retries for busy subscriptions and task-lock checks. > - A comment wake can run for an agent who is not the task assignee. Such a run never owns the task lock, so the new checks suppress its retry. > - This pull request preserves those comment wakes while keeping the lock checks for assignee runs. > - It also shows the subscription wait in task status and preserves the failure count through the database projection. ## Linked Issues or Issue Description Refs #13438. **What happened?** When a subscription is busy, a non-assignee comment wake is cancelled without a successor. Repeated subscription waits also display an inflated attempt count because the execution query omits the preserved failure count. **Expected behavior** An eligible comment wake waits and resumes without claiming the assignee's task lock. The task shows “Waiting for AI subscription”. Waiting does not consume provider-failure retries. Assignee retries still stop when the task lock is cleared or transferred. **Steps to reproduce** 1. Configure an agent with a managed subscription and concurrent runs. 2. Hold its credential lease in one run. 3. Mention the agent on a task assigned to another actor. 4. Release the lease and inspect whether the comment wake has a scheduled retry. The test uses a real embedded Postgres database and a real credential lease. Provider execution uses a fixture. The reassignment test applies a database mutation after the real checkout. No live provider account is needed. **Paperclip version or commit** Based on `f912ecaac` from #13438. Before the reconciliation fix, the rebased regression at `2063cfd13` failed because the comment wake had no scheduled retry. The original configuration failure was reproduced on `5282cabde` before #13438 merged. **Deployment mode** Built from source with embedded Postgres for local verification. ## What Changed - Record non-assignee comment-wake authority at admission while holding the task and run locks. A later reassignment cannot grant this exception. Preserve it through repeated subscription waits. - Keep master’s lock checks for assignee retries, including the check inside the scheduling transaction. - Show the subscription wait and preserve its failure count in the execution projection query. - Limit pre-provider wait receipts to fresh executions. A persisted native execution input must retain its recovery path. - Add lease, retry, projection, cancellation, reassignment, pause, revocation, service-recreation, and ownership regression coverage. - Use master’s 60–120 second retry interval and document the resulting behavior. ## Verification - Red: after rebase, the comment-wake test failed with no scheduled retry; the other 50 contention and retry-scheduling tests passed. - Red/green: reassigning the task immediately after real checkout reproduced an unwanted successor for both assignment and comment wakes. Both tests pass after recording authority at admission. - Green: all 285 targeted tests pass across 12 suites, including #13438’s four cancellation-race phases, its hiring/connection suite, run-dispatch integration tests, and both reassignment regressions. - `pnpm -r typecheck` passes after rebase. - `pnpm build` passes on the final revision. - The full general and serialized test suites pass in [CI run 34899514681](https://github.com/paperclipai/paperclip/actions/runs/34899514681) on `99f1e4c1d`. Full-suite verification ran in CI; local verification used the 285 targeted tests. - All 32 active PR checks pass, including build, typecheck, native runner verification, canary dry run, and all browser shards. Two Storybook checks are skipped by path filters. - Greptile reviewed `99f1e4c1d` at 5/5 with no outstanding findings. All review threads are resolved. - `git diff --check` and `node scripts/check-module-boundaries.mjs` pass. ## Risks - The non-assignee exception requires authority recorded under admission locks or its server-created subscription retry. Tests cover caller-supplied flags, assignment and comment reassignment races, repeated comment waits, and assignee lock protections. - A wait uses master’s 60–120 second interval. A long-held lease can produce multiple wait records. - Existing native sessions can have provider effects. They must not receive a fresh-execution receipt that permits replay. - No schema migration or credential permission change is required. ## Model Used OpenAI Codex, model `gpt-6-astra`, with repository search, code editing, shell execution, and test tools. The session does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f912ecaacf |
fix: carry AI connections through hiring and unblock task execution (#13438)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents hire other agents and assign tasks to them. > - Managed AI connections must follow those hires across legacy and native runners. > - Missing accounts should pause task execution and let the user connect from the task. > - Subscription contention must wait without asking for new credentials. > - This pull request fixes these paths and the native tool and Daytona staging failures found during live tests. > - The result is a working hire, subtask, and connection setup flow on local and remote runners. ## Linked Issues or Issue Description **What happened?** A managed Claude or Codex agent could hire a teammate without a usable AI binding. Cross-provider hiring could fail before the user had a chance to connect the new provider. First-time task setup did not show the existing AI credential form inline. A busy subscription could request a new connection. Native API replies could stop the parent after a hire had already committed. Fresh Daytona sandboxes could fail to extract read-only skill directories created on macOS. **Expected behavior** Compatible hires inherit the managed connection choice. A hire for another provider uses the responsible user's default. If that account is missing, the hire succeeds and the task asks for a connection. Completing setup in the task resumes work automatically. Explicit child auth settings and existing unmanaged login paths keep precedence. Shared-account access checks remain in force. **Steps to reproduce** 1. Connect a Claude or Codex parent with a managed AI account. 2. Ask it to hire one agent of each provider and create a self-assigned subtask. 3. Assign work to both hires without connecting the second provider first. 4. Connect the missing provider from its task card. 5. Check that all tasks finish and same-provider work uses the original account. 6. Repeat with native runners and fresh Daytona sandboxes. The opt-in browser suite in `tests/hiring-ai-connections/README.md` performs these steps. **Paperclip version or commit** The live failures were reproduced from `f2c5e54dc`. The branch is rebased onto `5282cabde`. **Deployment mode** Isolated local development instance. Legacy CLI and native runners. Local execution and ephemeral Daytona sandboxes. Related work: Refs #13247 for managed AI connections. Refs #13268 for legacy credential-reference inheritance, which this branch preserves. Refs #13432 for a concurrent managed-inheritance fix. This PR also covers cross-provider task setup, subscription waits, native API replies, and Daytona extraction. It permits missing responsible-user defaults at hire time; restricted shared selections still fail. ## What Changed - Apply managed connection defaults to both agent creation routes. Preserve explicit auth choices and legacy credential-reference inheritance. - Allow hires before their responsible user connects the provider. Keep approval gates, company boundaries, and shared-account access checks. - Reuse the production AI credential form inside the pending task card. Resume the task after setup. - Retry subscription lease contention without consuming the provider-failure allowance or creating a connection request. - Require task execution-lock ownership when scheduling, promoting, and dispatching subscription retries. Recheck ownership under the issue row lock. - Rename the HTTP operation identity at the native tool boundary so it cannot override the runner's operation identity. - Delay directory permission restoration during Daytona extraction. Preserve the final read-only modes. - Add database-backed regressions, real browser acceptance tests, and Storybook states. Document setup and run-log behavior. ## Verification - Six real browser scenarios passed: both parent providers on legacy local and legacy Daytona; native Codex locally; native Claude on Daytona. Each scenario hires both providers, completes a self-subtask and assigned work, and connects the missing provider inline with automatic continuation. - Successful runs verify the account, responsible user, runner mode, and Daytona lease. All 18 test sandboxes were deleted. - Live authentication used API keys. Subscription inheritance, lease contention, and retry have integration coverage. Fresh subscription OAuth sign-in was not automated. - Red/green tests reproduced missing bindings, missing inline forms, subscription contention, native API reply failure, and GNU tar permission failure. - Seven Storybook browser checks passed. They cover both providers, method selection, narrow layout, completion, cancellation, and invalid credentials. - Full local suite coverage completed before rebase. Initial timing and fixture startup failures passed unchanged on isolated reruns. The first full command did not exit cleanly; the remaining workspace and serialized groups were completed separately. - After rebase, 107 hiring/auth/retry tests and 59 native API, task-card, and Daytona tests passed. The full workspace typecheck, production build, and token gates passed again. Storybook build passed before rebase. - Review fixes: 169 hiring/retry/dispatch tests, 37 adjacent tests, and four explicit cancellation-race cases passed. Eight cross-provider cases cover stale auth keys on both creation routes and both runner types. Server typecheck and build passed. - Final CI on `ee4890837a8a4913e07453392b9a75969580dae1`: 32 checks passed. Two optional Storybook jobs were skipped. The full server, workspace, browser, native runner, build, typecheck, and release checks passed. - Three unchanged tests initially failed on a busy port, a chat row-lock race, and preview-server readiness. Each affected job passed after one CI rerun. Isolated local checks also passed: 41 credential tests, the chat-concurrency case, and 25 preview-runtime tests. - Greptile reviewed the final commit at 5/5. Both review threads are resolved. GitHub reports no merge conflicts. ## Risks - A missing personal account now defers authentication to the first task. Explicit incompatible bindings and restricted shared accounts still fail at hire time. - An inherited personal default uses the responsible user's existing authorization to install access for the new agent. It never copies credentials or another user's identity. - Subscription contention retries after a delay and rechecks task eligibility. It does not consume the provider-failure budget. - Native hiring uses the existing managed API-tools opt-in. Remote native runners require a matching Linux binary and provider pack, as documented in the acceptance README. - No schema changes. Live tests make paid provider calls and remain opt-in. ## Model Used OpenAI GPT-6 (`gpt-6-astra`) in Codex, with reasoning, repository inspection, code execution, browser automation, and API tools. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
422287eecd |
fix: preserve runner recovery, warm sessions, and task outcomes (#13338)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The native runner connects task messages, provider execution, and task outcomes. > - First-time user tests exposed gaps in recovery, completion permissions, message delivery, and Stop behavior. > - These gaps left usable output hidden, completed work waiting for bookkeeping, or safe work unable to continue. > - This pull request fixes the shared lifecycle and receipt paths while preserving process ownership and action checks. > - Users can continue work with accurate task state and durable messages. ## Linked Issues or Issue Description **What happened?** A stopped local Codex execution could remain blocked even after its processes had stopped and its complete transcript proved that no external action needed replay. Claude under Conservative permissions could fail to call task completion tools. Recovery could reuse an assistant item ID and overwrite prior output. A delivered comment could remain marked uncertain after navigation. Stop could look like Pause or a new recovery incident. Workspace contention could look like cancellation. A direct reply reopening Done could enter a clarification loop. **Expected behavior** Recover automatically only with verified termination and complete action receipts. Preserve answers and messages. Keep task completion available under Conservative permissions without broad tool access. Show crashes as Blocked, actual human decisions as In Review, and ordinary workspace contention as waiting. Stop the current response and allow a new direction. **Steps to reproduce** 1. Create ordinary response tasks with local Codex and Claude Code, then send follow-up messages through the task composer. 2. Interrupt a disposable local Codex runner during text-only work. Verify automatic continuation and retained output. 3. Stop a response, send a new request, answer a clarification, and reopen completed work with another message. 4. Navigate or reload while a comment submission is pending. Confirm the exact persisted request receipt settles it without removing newer draft text. 5. Run two tasks in a shared Daytona workspace. Confirm waiting does not appear as failure. **Paperclip version or commit** Initial acceptance baseline: `c9021c6721f91e2c74bd9fee9d3fd41c999d17b7`. Current integration base: `6cef9743c`. Both operator-interruption and workspace-waiting guards are preserved; native restart and legacy permission rules remain documented. **Deployment mode** An isolated source-built test-drive instance, with real local Codex and Claude Code providers and disposable Daytona environments. Related work: #13314, #13316, #13327, #13344, #13239, #13254, #13163. This PR addresses additional failures from ordinary task journeys, including controller restart handoff and repeated warm sandbox setup. Historical task status reconciliation is excluded. ## What Changed - Persist runner ownership immediately at spawn and resume an explicitly adopted runner even when the controller crashed before the first driver checkpoint. Detach the controller safely across graceful restarts, including session startup. Prevent an old finalizer from suspending or signaling an adopted runner. Checkpoint idle warm sessions before shutdown. Preserve the same run and queued follow-up messages. - Scope saved legacy queue successor checks to the queue owner while preserving ordinary task locks, operator identity, assignment gates, and exactly-once delivery. - Preserve managed Codex credential files when an old session is detached for restart; normal owned cleanup still copies refreshed auth back and removes the scoped copy. - Reuse the bound warm shared sandbox and fully verify an existing staged provider pack before using it. This avoids repeated uploads when the pack is already valid. - Add a narrow local Codex replacement path with stopped-process proof, a closed transcript inventory, exact completion receipts, and fresh-session lineage. Preserve no-replay holds when evidence is incomplete. Recovery may clear only the same run's recorded Blocked status version; manual re-blocking and dependency changes invalidate that receipt, while queued comments do not. Later blocks stop scheduled, queued, and final dispatch; queued/final checks re-read dependencies even when the task status stays In Progress. - Permit only task delivery and human-input tools through the isolated Claude runner's exact task bridge. - Scope assistant item identity to the provider turn and ignore only authority-free Codex skill-change notifications during startup. - Reconcile composer submissions by client request ID across response loss, navigation, and reload. Retain text typed during delivery. - Keep acknowledged run-only Stop neutral and show workspace contention as waiting. Project exhausted native failures as Blocked. - Restore the guarded task-page retry action for failed legacy runs, including the server-supported explicit new-attempt path for stopped conversation adapters. Preserve native/process recovery holds and avoid promising Retry while a decision or execution gate hides it. - Refresh delivered artifacts and handle direct user replies that reopen completed work without a clarification loop. - Check the embedded PostgreSQL PID, data directory, and actual port before connecting or migrating. - Document accepted behavior and add focused regressions at lifecycle, route, transcript, and UI boundaries. ## Verification - Final head `fece606ac2` passes the complete GitHub CI matrix: **34 green checks, two expected Storybook skips, no failures or pending checks**, including `ci / verify`, `ci / e2e`, full runner verification, typecheck, build, every server/workspace shard, and all browser shards. [CI run](https://github.com/paperclipai/paperclip/actions/runs/34727183287). Greptile is **5/5 with no open findings**. The final two commits only refine test fixtures; both affected suites pass 24/24 locally and in CI, with server typecheck green. - Complete local Vitest coverage uses the canonical groups/shards: all 635 general server suites, all 145 serialized suites, and all workspace packages. The aggregate began on `0a8001c18` while the final queue fix arrived: 23,903 passed, five failed, 87 skipped. The five port/socket/timing failures passed unchanged in follow-ups (60 tests in the exposure/file suites and 412 tests covering the serialized failures and unrun tails). The final queue/operator-identity suites separately passed 52/52. This is aggregate coverage plus explicit reruns, not a pristine single-command final-head run. - After integration with current master, queue/operator-identity/continuation suites passed 162/162 and affected UI suites passed 140/140. ACP Stop/continuation and legacy task/Inbox/message browser suites passed 9/9, including both task recovery Retry and thread Try again, automatic saved-message delivery, exactly one new run, Done, and retained output after reload. The default process Stop/Pause/Resume browser case passed (the native-provider case is opt-in and skipped by default). The complete Board attachment/receipt browser suite passed 11/11 on a disposable instance, covering both composers, exact receipts after lost responses, no replay, bound attachments, and newer drafts after reload. - Blocking-intent regressions cover pre-existing Blocked, a mismatched run/cause, an explicit manual re-block, changed dependencies, a queued comment after failure, and a block arriving between scheduling and provider dispatch. The negative cases reproduced before the fix. All 478 affected executor/recovery/dispatch tests passed; both database suites ran separately after availability-probe skips in the first combined command. The final late-dependency check passed all 143 affected recovery/dispatch tests (zero skips) after two new negative cases reproduced the bug. - Focused runtime regressions cover awaited runner ownership publication, authenticated adoption before the first checkpoint, old-finalizer detachment, idle and busy warm-session shutdown, rejected checkpoint propagation, provider-pack verification, and managed-Codex credential preservation. Four managed credential detachment cases reproduced the bug before the fix; normal owned cleanup still succeeds exactly once. - Live local Claude: SIGKILL 2.6 seconds into startup recovered the same run automatically in 53 seconds, then a normal follow-up completed in 24 seconds. SIGTERM 2.5 seconds into startup preserved the same run (54 seconds) and its queued follow-up (21 seconds). Answers remained visible and the task reached Done. - Live Claude Daytona: a warm follow-up retained its sandbox and fell from 121 seconds to 44 seconds. A separate cold turn took 127 seconds; after controller shutdown and checkpointing, its follow-up completed in 33 seconds with the same sandbox, workspace, native session, and runner. Both answers remained visible and the task was Done. - Other live journeys covered task completion and follow-up with local and Daytona Codex, local Codex crash recovery, Stop then new direction, clarification response, live artifact refresh, and shared-workspace waiting. - Validation limits: the opt-in native composer Stop/Pause→subtree Resume fixture exposes terminal/result ordering and subtree-cancellation attribution bugs that can leave a child task blocked; that new finding is assigned to a separate follow-up and is not claimed fixed here. Default CI skips this optional native-provider fixture. Managed-Codex credential handoff and the queue-agent integration use automated regression evidence. Cold custom provider-pack uploads still add startup latency. ## Risks - Automatic replacement remains deliberately narrow: local Codex, verified stopped identities, unchanged retained state, and a complete text/completion-only turn. Unknown actions, partial history, or changed ownership remain blocked. - Claude completion permission handling changes an upstream package patch. The exact isolated task bridge must remain pinned; unrelated tools keep their existing permissions. - New task failure projection changes user-visible status. No historical status backfill or database migration is included. - This is a broad lifecycle fix across server and UI. Live proof covers graceful local Claude restart during startup and idle Claude Daytona session recovery across controller shutdown. Live abrupt SIGKILL during local Claude startup also recovered the same run. Unknown ownership or missing action evidence still blocks reuse. Cold custom provider-pack uploads still add startup latency; this change avoids unnecessary repeat uploads. ## Model Used OpenAI GPT-6 (Codex), with reasoning, code execution, browser automation, and tool use. The exact hosted model ID and context window are not exposed in this task. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
24007a9804 |
fix: promote review tasks when continuations start (#13318)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service starts and tracks agent work on issues. > - An issue can be in review before a user comment starts more agent work. > - The previous checkout rule did not claim an issue from review for this continuation. > - The issue could stay in review while an agent actively worked on it. > - This pull request lets a resolved interaction continuation claim an issue from review. > - The existing checkout update then sets the issue to in progress. > - The benefit is that issue status now shows active agent work correctly. ## Linked Issues or Issue Description **What happened?** An issue stayed in review when a new agent continuation started work on it. **Expected behavior** The issue must move to in progress when an agent starts work. An idle issue must stay in review. **Steps to reproduce** 1. Put an assigned issue in review. 2. Resolve an interaction that starts a continuation. 3. Start the heartbeat run. 4. Observe that the issue remains in review while the run works. **Paperclip version or commit** The problem reproduces on the master branch before this change. ## What Changed - Allow a resolved interaction continuation to claim an assigned issue from review. - Keep idle review issues unchanged. - Add regression tests for both behaviors. ## Verification - Run `node_modules/.bin/vitest run server/src/__tests__/heartbeat-auto-checkout.test.ts`. - Confirm that all four tests pass. ## Risks - Low risk. The change only expands checkout eligibility for resolved interaction continuations. - The existing guarded checkout update still controls ownership and the status update. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex with GPT-5.5. The model used reasoning, tool use, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ab15aff390 |
feat: add experimental persistent agent chat (#13284)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Conversations must use the same tasks, controls, and execution history. > - Users need an ongoing chat with an agent without managing task properties. > - Agents should clarify and plan work, then hand execution to assigned project tasks. > - This pull request combines the reviewed Agent Chat stack for one squash merge. > - The benefit is persistent conversation with normal task governance and shared UI. ## Linked Issues or Issue Description **Subsystem affected** Task lifecycle, agent runtime tools, shared task UI, and browser/paid runner tests. **Problem or motivation** Users need one persistent conversation with each agent. A separate chat store or renderer would duplicate task behavior and bypass existing controls. **Proposed solution** Use a task-backed chat per company, user, and agent. Reuse the task composer and transcript. Clarify and plan in chat, then create assigned project tasks with the relevant plan. Keep Agent Chat behind its own disabled-by-default experimental setting. **Roadmap alignment** This implements the task-backed direction in [CEO Chat](https://github.com/paperclipai/paperclip/blob/master/ROADMAP.md#-ceo-chat). Related proposals: #2504 and #9693. Related request: #7981. The maintainer requested one squash merge of the complete stack. Consolidates the reviewed runtime [#13281](https://github.com/paperclipai/paperclip/pull/13281), backend [#13282](https://github.com/paperclipai/paperclip/pull/13282), and UI [#13283](https://github.com/paperclipai/paperclip/pull/13283) layers with this PR's E2E coverage. All four layers passed CI and received Greptile 5/5 before consolidation. This PR targets master and includes the complete feature. ## What Changed - Add personal canonical chat tasks with ordinary company visibility, immutable identity, idempotent first sends, and an idle waiting state. - Process `/new` in queue order. Preserve history, release a chat pause, and fence old provider context and delayed writes. - Keep chat lifecycle rules across recovery, finalization, assignment, task lists, and rollups. - Support research and plan revision in chat. Hand plans to ordinary assigned project tasks before execution starts. Reject new chat subtasks. - Add repository-aware project creation and discovery tools, including multiple repository IDs and GitHub URLs, authorization, idempotency, and durable project-created cards. - Reuse task UI components for chat, with starred/recent agent navigation and a separate `enableAgentChat` experimental flag. - Add deterministic browser tests and 24 paid chat cells across four Codex/Claude profiles, with validated reports and screenshots. - Integrate current master recovery, controller lease, queued-message, and task UI changes. Gate chat interruption and deferred promotion on ownership/feature policy. Guarantee lease renewal and active controls are stopped even if teardown fails. - Preserve master's migration 0273 and generate chat migration 0274 with idempotent replay for development databases. ## Verification - Prior exact heads of all four PRs passed Linux CI, including build, typecheck, general/serialized tests, and browser E2E. Each had Greptile 5/5 and no unresolved findings. - Integrated local verification passed: full repository typecheck and production build, Storybook build, token gates, 340 focused UI tests, all 20 deterministic chat browser tests, two migration replay tests, 88 focused chat/queue/native/controller tests, and provider/session regressions including real lease expiry. These include the three lifecycle regressions for the final admission/teardown fixes; server typecheck also passes. Current head `1268eda16cc2af892055917e7292f068820be135` has Greptile 5/5 with no unresolved findings and passing security scans. All final-head CI gates passed: build, full Runner verification, typecheck/release registry, canary, all general/serialized test shards, and all browser E2E shards ([CI run](https://github.com/paperclipai/paperclip/actions/runs/34696739927)). Local PostgreSQL startup contention required serialized retries; skipped fixtures do not count as passing coverage. - The earlier paid campaign passed all 24 chat cells and retained 32 screenshots: [report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-34648511170-1/index.html?report=agent-chat#suite-agent-chat). It tested `abacbdfd2f660709ec37312cdb758284c8399d04`; it is prior evidence, not a paid run of this integrated head. - Manual check: enable Agent Chat in Experimental settings, open an agent, clarify and revise a plan, then hand off to an assigned project task. Stop a reply, send `/new`, and verify fresh context with retained history. Disable the setting and verify agent shortcuts/new chat turns are blocked. ## Risks - Queue/session integration can affect retries and delayed writes. Tests cover ownership, cancellation, reset boundaries, idle recovery, and ordinary task behavior. - Migration 0274 adds conversation fields and constraints. Replay is idempotent and preserves existing development chat history. - This combines the previously reviewed stack at the maintainer's request. Agent Chat remains off by default and is separate from Conference Room. ## Model Used OpenAI Codex, GPT-6 Astra (`gpt-6-astra`), with reasoning, code execution, browser tools, and parallel review. The exact context-window size is not exposed in this session. Codex and Claude also ran as test subjects in the linked paid campaign. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
1c4bcff2b1 |
fix(test): await issue lock before retry race assertions (#13273)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Retry decisions must respect changes to an issue owner. > - Database tests verify this with two concurrent transactions. > - The test started the competing operation before its fixture held the lock. > - That race can fail a correct source verification run and block deployment. > - This PR waits for lock acquisition before starting the competing operation. ## Linked Issues or Issue Description Refs #13257 for the deployment verification work that exposed this test race. No duplicate fix was found. **What happened?** [Cloud verification job 103424158921](https://github.com/paperclipai/paperclip/actions/runs/34648268409/job/103424158921) failed because the test did not observe a concurrent issue-row lock waiter. The fixture and retry operation both started without an ordering guarantee. **Expected behavior** The fixture must hold the issue lock before the competing retry operation starts. The test must still prove that the retry waits for the lock and observes the reassignment. **Steps to reproduce** 1. Add a temporary 50 ms delay before the fixture acquires the issue lock. 2. Run the promoteOrCancelDueRetry issue-lock test. 3. The original fixture fails with the same missing-waiter error as CI. 4. The synchronized fixture passes with that delay. The delay is not part of this PR. ## What Changed - Separate fixture readiness from its transaction completion promise. - Wait for readiness at both callers before starting the retry decision. - Propagate transaction failure during setup through Promise.race. ## Verification - The original fixture fails under the temporary delayed-lock probe. The fixed fixture passes the same probe. - All 31 tests in server/src/modules/run-dispatch/adapters/postgres.test.ts pass after removing the probe. - The real concurrent waiter, lock-order, and reassignment assertions remain intact. No timeout was increased. - Full local typecheck and build pass (221s and 50s). The full local test command stopped in its server phase after 10,610 passes, 65 skips, and 14 failures: 13 existing macOS skill-cache rename/permission failures and one unchanged Telegram test assertion. The Telegram test passes in a focused rerun. Later local phases did not run after that failure. Current-head Greptile is 5/5 with no findings. All 32 current-head checks pass, including full Linux typecheck, build, native verification, server suites, browser suites, and canary packaging. The unchanged GitHub browser mock assertion passed its single failed-shard retry. - git diff --check passes. ## Risks - Only test synchronization changes. Production database behavior is unchanged. - Awaiting the transaction itself during setup would deadlock the test. Returning the completion promise inside an object avoids that problem. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — all 31 database adapter tests pass; full local verification limits are disclosed above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b1efd65edc |
fix: continue interrupted task conversations with bounded retries (#13237)
## Thinking Path > - Paperclip manages AI agents and their tasks. > - A task can outlive a provider process or a server restart. > - Legacy recovery treated unknown tool outcomes as a permanent execution hold. > - That hold could also reject a later user message. > - A conversation turn can use prior history without replaying prior tool calls. > - This pull request lets supported conversation adapters continue within the existing retry budget. > - Users can send a new message after automatic attempts stop. ## Linked Issues or Issue Description **What happened?** A server restart could interrupt a local ACP run and leave its task behind a permanent recovery hold. A later user message could be cancelled before the provider answered. The immediate recovery path could also create a successor outside the durable failure counter. **Expected behavior** Continue with a bounded new conversation turn. Preserve a compatible provider session or use full task context when it is unavailable. Do not replay recorded tools. When automatic attempts stop, allow a new user request through the normal execution gates. **Steps to reproduce** 1. Start a task with a local conversation adapter. 2. Restart the server while the provider is working. 3. Let the previous run become interrupted. 4. Send a follow-up message and observe the recovery hold on the old behavior. Related work: Refs #13075 for durable task recovery. Refs #12946 for retry-limit and checkout-lock handling. This change routes conversation recovery through the existing bounded scheduler. ## What Changed - Mark supported local conversation failures for continuation. Keep native-runner and non-conversation recovery rules. - Carry an interruption notice into the next turn. Retain stopped ACP session history even when a write outcome is unknown. - Clear unavailable ACP sessions so the next bounded attempt can use full task context. - Route immediate failure recovery through the same durable scheduler as process-loss recovery. Release only the predecessor checkout when its retry takes ownership. - Retire obsolete conversation holds using immutable run evidence, in bounded batches with an activity record. Preserve outcome evidence and do not wake historical tasks. - Block actual admission and Resume while a predecessor process or environment lease is still active. Keep the original interruption notice after a rejected wake. Preserve the upstream blocked-wake waiting contract: bounded retry planning can happen during cleanup, while deferred messages and execution remain gated. - Add subprocess and database regression tests. Update the execution contract. - Add the current thread-status field to the native recovery provider fixture so its damaged-journal test reaches the intended boundary. Tolerate an already-exited fixture process during test cleanup while still asserting both processes terminate. ## Verification - Workspace typecheck passed: `pnpm -r typecheck`. - Build passed: `pnpm build`. - Module boundaries passed: `pnpm check:module-boundaries`. - Focused tests passed: 293 recovery/session/dispatch tests, 66 retry and response-gate tests, and 37 native-session tests. Some suites overlap. - Tests cover interrupted writes, missing sessions, concurrent retries, restart persistence, pending questions and approvals, execution gates, and historical holds. - Built the Rust test executables with `pnpm --filter @paperclipai/paperclip-runner build:rust` for native-runner verification. - Full Vitest coverage verified locally using the repository’s general and serialized shards, with focused reruns for failures and files not reached after a shard stopped. The ownership-gate regression is fixed and the complete affected server shard passes (1,390 tests). Local parallel runs also hit temporary-directory, resource, and timing failures; those suites pass with canonical temporary paths and sequential reruns. No test timeouts were increased. - Final merged-branch regression run: 577 tests pass across process recovery, retry scheduling, liveness, durable chat, wake-queue application/adapter, dispatch, continuation, native sessions, and task chat. Earlier focused verification also passed 19 native control tests. Token gates and whitespace validation pass. - Browser verification passed all three ACP Stop/continue/pause scenarios, including a rerun after merging the upstream waiting behavior: `PAPERCLIP_E2E_PORT=3397 pnpm test:e2e tests/e2e/acp-stop-continuation.spec.ts`. The interrupted-write case verifies that follow-up completes without a repeated write. - Final-head [CI run 34625037394](https://github.com/paperclipai/paperclip/actions/runs/34625037394) passed on `06ac4bd9d150f8b209a96e5fd609c696958794a0`: all 31 reported checks are green, including server/workspace suites, all browser shards, native runner verification, build, typecheck, release dry run, and aggregate gates. The two conditional Storybook checks were skipped. Greptile reviewed this exact commit at 5/5; all review threads are resolved. ## Risks - A new model turn can choose to repeat an action. Paperclip does not replay recorded tool calls and does not certify unknown action outcomes. - Conversation adapters now stop after their retry budget instead of requiring action reconciliation. Explicit Stop, pause, dependency, approval, budget, and ownership gates remain in force. - No schema migration or dependency changes. Historical holds are folded without changing task status or waking work. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and test execution. The session does not expose a more specific model build ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
eb640ec129 |
fix(execution): keep blocked wakes waiting without repeated runs (#13236)
## Thinking Path > - Paperclip manages AI agents and their work. > - Wake admission decides when a task can create an execution run. > - Recovery can prohibit replay while the previous execution needs review. > - Dependency reconciliation kept creating runs before dispatch rejected that same hold. > - Each rejected run added another startup notice without doing useful work. > - This change checks the hold during admission and records repeated automatic waits once. > - Tasks keep their messages and can resume when the current gates permit execution. ## Linked Issues or Issue Description Related changes: Refs #13173 (stale completed-task continuations). Refs #12651 (dependency waits during recovery). **What happened?** A blocked task with completed dependencies can remain under a durable execution reconciliation hold. Each scheduler pass created a queued run. Dispatch then cancelled it before the adapter started. The skipped wake did not satisfy dependency wake deduplication, so this repeated and filled the conversation with “Couldn't start” notices. **Expected behavior** A known execution hold creates a waiting diagnostic without a run. Repeated automatic observations share that diagnostic. Clearing the hold permits a new wake only after the other gates pass. New comments remain available for the next eligible execution. **Steps to reproduce** 1. Assign a blocked task with a completed blocker. 2. Give the task an active reconciliation action, or a resolved action whose automatic recovery evidence still prohibits replay. 3. Run dependency reconciliation repeatedly. 4. Observe repeated cancelled pre-start runs on the base branch. This branch creates no runs while held and admits work after the effective hold clears. ## What Changed - Check effective execution holds under the issue admission lock before inserting runs. Keep the final dispatch check for races. - Share automatic wait diagnostics across producers, wake keys, and service restarts. Apply the helper to reconciliation, dependencies, pause holds, availability, budgets, and disabled heartbeats. - Preserve ordinary comment and interaction receipts during execution holds. Prevent release from draining them while replay is blocked. Keep external-chat receipt authorization intact. - Group empty pre-start reconciliation cancellations into a neutral waiting notice. Keep started runs and the full run history. - Document the waiting contract and add database-backed, UI, and browser regressions. - Stabilize two existing verification tests: allow the asynchronous chat lease transition a bounded five-second wait, and accept either legitimate damaged-session refusal while retaining exact archive-evidence assertions. ## Verification Passed targeted tests: - `pnpm exec vitest run server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts server/src/modules/wake-queue/adapters/postgres.test.ts` — 28 tests. - `pnpm exec vitest run ui/src/components/TaskChatThread.test.tsx` — covered in the initial combined test run; UI suite passed. - Run-dispatch adapter tests passed in the combined gate regression run. - `pnpm exec vitest run server/src/__tests__/durable-chat-wakeup.test.ts` — 41 tests, including held receipt replay, promotion, and revoked access. - `PAPERCLIP_E2E_PORT=3294 pnpm test:e2e tests/e2e/acp-stop-continuation.spec.ts` — all 3 browser scenarios pass. Repeated held messages create no additional runs or provider prompts and do not replay writes. - `pnpm check:token-gates` - `pnpm check:module-boundaries` - `git diff --check` `pnpm -r typecheck` and `pnpm build` pass. The full local `pnpm test:run` invocation did not finish green: it encountered exhausted local PostgreSQL shared-memory slots, a missing fresh-worktree runner test binary, and tests loaded across in-flight edits. The affected chat/database suites passed on rerun (81 tests), and the targeted lifecycle/recovery verification passed (3 tests). After building the runner test binary, the full native session suite also passed (37 tests). Final-head [CI run 34621288475](https://github.com/paperclipai/paperclip/actions/runs/34621288475) passed on `8659618b0ed2b98df002a28f4c1bd97321b0db04`, including all server/workspace test shards, all three browser shards, runner verification, typecheck, build, release dry run, and the aggregate verification gates. All 31 reported checks passed; the two conditional Storybook checks were skipped as intended. Greptile reviewed that exact commit at 5/5 with no unresolved review threads. ## Risks The wait record is diagnostic only. It must never count as a delivered wake or bypass a current gate. Tests cover repeated and concurrent admission, resolved no-replay evidence, a remaining dependency after hold clearance, deferred comments, and release gating. Explicit user requests and authorized chat receipts do not share automatic diagnostics. No migration or historical data deletion is required. Existing provider retry budgets remain unchanged. ## Model Used OpenAI GPT-6 through Codex. The runtime does not expose the exact hosted snapshot ID or context-window size. Used repository inspection, code editing, command execution, tests, and review tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |