DottaandPaperclip
8a3d2cb04f
## Thinking Path > - Paperclip manages AI agents and their work. > - Queue admission must stop work while a previous execution needs recovery. > - The final dispatch gate can cancel a queued run before it starts. > - Legacy recovery treats this known wait as an unknown provider outcome and creates another hold. > - The extra hold can survive repair of the original execution and leave saved messages stuck. > - This pull request uses one database proof to prevent the extra incident and recognize old false holds. > - Saved input can then pass through normal admission with the original work and safety gates intact. ## Linked Issues or Issue Description Refs #15483. **What happened?** The dispatcher writes an `execution_reconciliation_required` cancellation when an existing recovery hold denies a queued run. No runtime is selected and no provider starts. Legacy recovery later creates a `legacy_execution_requires_reconciliation` action for that cancelled admission. Its settled `replay: blocked` record remains an effective hold after the real source recovery finishes. The task has saved messages but no eligible Retry or Continue action. **Expected behavior** A verified admission denial stays a wait. It must not create another provider-outcome incident. Existing false incidents must stop blocking after the server verifies their recorded source. Real execution, uncertain outcomes and file recovery stay held. The harness determines these infrastructure facts; the user does not certify them in a form. **Steps to reproduce** 1. Persist a queued-run gate cancellation with its task-bound `executionWait` receipt and no execution authority or resource evidence. 2. Run legacy reconciliation. On the old code, it creates a second incident. 3. Persist the old incident as resolved with replay blocked, then clear the original source hold. 4. Restart queue recovery with saved user messages. On the old code, no successor is admitted. Related work: #15965 covers other pre-start dispatch cancellations but explicitly excludes this existing-hold code and leaves historical holds unchanged. #16634 covers stale-cancel classification and queue drain, without re-evaluating historical effective holds. #15383 covers an operator Stop during conversation startup. This change addresses the existing-hold receipt and its persisted false incident; it does not copy those broader changes. ## What Changed - Add one database proof for a legacy admission denied by an existing execution hold. Require the exact dispatcher receipt, matching task identity, no runtime or process claim, no lease or finalization coordinator, and no execution events. - Use the proof before legacy recovery creates an incident and when it computes effective historical holds. Keep the old audit rows. - Preserve source file obligations and the original recovery action. Keep the normal queue, pause, ownership, company and budget checks. - Add real PostgreSQL regressions for incident prevention, old state, concurrent queue recovery, message order, conflicting evidence and scope boundaries. - Update the execution contract and both recovery maps. ## Verification - Red: the regression on unfixed source fails on incident classification, historical hold projection and actual periodic saved-message admission. The other 24 safety cases pass. - Green: all 27 new database cases pass. Two safety assertions were aligned with the existing contract: daily limits stop dispatch after queue promotion; a human comment can intentionally reopen a terminal task. The final regression checks the real daily dispatch gate and agent pause. - Existing `explicit-native-continuation.test.ts`: 255 tests pass. Existing legacy recovery and PostgreSQL dispatcher suites: 61 tests pass. - `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs` and `git diff --check` pass. - GitHub PR CI run `39313204660` passed on `ef6009e656b7c9b4c788480dce322874565960ac`. All 54 successful check-runs and the Snyk status are green. Two optional Storybook checks were skipped. The independent Paperclip Review completed with a neutral check and a 5/5 assessment with no findings. Greptile also reports 5/5 on the same commit. There are no unresolved review threads. - `pnpm test:run` is still running locally. Its overall result is pending; no full local suite pass is claimed. The exact-head GitHub CI suite passed. - A read-only probe of an affected hosted task verifies the same database proof and projects no false blocker with the working source. No live rows were changed. Deployment and a real resumed provider run remain unverified. ## Risks - An incorrect exclusion could admit work with uncertain effects. The proof fails closed on missing receipts, source mismatches, runtime/process ownership, any retained lease, coordinator, execution event or file obligation. Real upstream file recovery remains blocking in the regression. - The effective-hold query adds indexed existence checks scoped to the recorded run and company. It makes no provider calls and performs no database repair writes. - Historical audit records keep their recorded outcome. Only their effective blocking status changes when the server can prove the admission never ran. - No schema, permission, credential or UI form changes are included. Live recovery still needs verification after rollout. ## Model Used OpenAI Codex, model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
20 KiB
navigation
| navigation | ||||||
|---|---|---|---|---|---|---|
|
Recovery
People can see why a task stopped, who owns the next action, and whether a safe continuation is available. Recovery retains task context and ownership, honors pause and budget gates, and distinguishes confirmed failure from an uncertain external outcome. A retry acknowledgement alone is not proof that work resumed.
Agent access: companion.
Sub-features
diagnosis: a task/run exposes the recorded cause, evidence, owner, attempt progress, and next action without requiring a raw-log interpretation.bounded-continuation: supported interrupted work continues with preserved conversation context and bounded attempts, without blindly replaying tools.source-action: source-scoped recovery actions remain attached to the task; independent repair work may have its own issue and return owner.exhaustion: exhausted or overdue recovery exposes the current gate and permitted operator action; it does not claim an automatic path is still live.workspace: workspace divergence/export failures show the applicable repair, reconcile, or isolated reissue action with evidence and authority checks.unknown-outcome: uncertain external side effects require inspection and an explicit decision, not automatic replay.user-continuation: saved user messages after a stop are reconsidered once cleanup proves the old execution is stopped; duplicate wakes are avoided.reconnect: browser stream recovery refreshes persisted state without claiming the agent itself restarted or that a failed run succeeded.
How to get to it (user POV)
source-task
Open the affected task from Tasks, Inbox/Blocked, search, or a direct task link. Inspect the recovery notice/card in its history, expand details if needed, and use the currently offered action. Follow any linked repair task and return owner.
An authorized board user can expand Recovery details on a stopped-run notice to read the decision that still blocks replay, including a settled recovery record. The notice matches the current action and run; it does not show another run's historical decision or authorize a retry. Source-reviewed behavior; the notice regressions cover empty active recovery, a mismatched source, and failed history reads.
run-detail
Open the agent's run (/agents/:agentId/runs/:runId) from activity or the task's
run link. The run page hosts workspace recovery controls only for a failed run
with error code workspace_validation_failed and a linked source task that still
has a live workspace_validation recovery action. Use the source task for other
stopped-run recovery checks.
operator-retry
On an exhausted disposition or workspace recovery card, inspect the current reason and use the permitted retry/repair/reconcile action. Some repairs require confirmation; unavailable actions must explain the gate or remain absent.
post-stop-message
After a stop, send a new instruction on the same task or inspect one already saved while cleanup was pending. Observe the waiting message, continuation, and eventual result rather than manually changing the task status to look complete.
A continuation denied before startup by an existing execution hold remains a wait. Once the original hold clears, the server verifies the dispatch receipt and absence of execution evidence, ignores any historical false incident for that unused admission, and reconsiders saved input through normal admission. No infrastructure-certification form is needed. The database regressions cover incident prevention, retained audit history, concurrent restart admission, ordered messages, contradictory execution evidence, and unchanged pause and task gates. These tests do not qualify a live provider or prove task completion.
browser-reconnect
While viewing an active or recovering task, interrupt the browser connection and restore it, or suspend and return to the tab. Reload is a separate check. The current task and run state should reconcile with the server.
Driving it
Preconditions: follow the baseline. Use a disposable instance and reproducible failure. Record source task/run, adapter/runtime mode, cause, attempt count, current owner, and verified stop evidence. Do not kill an unrelated server or induce provider side effects just to get a failure screen. Use the existing deterministic recovery suite first.
source-task
Automated: recovery actions covers source scoping, bounded attempts, operator stops, quota monitoring, ownership, stale action retirement, and continuation deduplication. Recovery cards cover diagnosis, retry progress, exhaustion, and action availability. The fixture-backed execution-recovery journeys start their own test-drive instances and exercise safe, uncertain, restart, manager-lineage, and legacy-unknown paths:
pnpm exec vitest run server/src/__tests__/issue-recovery-actions.test.ts ui/src/components/IssueRecoveryActionCard.test.tsx
pnpm exec playwright test -c tests/e2e/execution-recovery/playwright.config.ts recovery.spec.ts
Manual: open a stopped task through each affected navigation path. Confirm its explanation, original owner, recovery owner if different, and next action. Follow the supported recovery until the source task produces a useful result or a specific escalation. Reload and verify that resolved notices retire and attempt history remains. A fixture pass does not qualify every live harness/provider.
The native restart integration suite also covers a prepared remote workspace interrupted before the executor's first claim. Real database tests assert one same-run bootstrap claim under concurrent controllers and reject prior claims, launch/provider evidence, and malformed workspace references. These cases prove classification, not a live remote provider restart. For that journey, interrupt a disposable deployment after workspace preparation and before its first executor claim, then confirm one provider turn starts on the original run through the normal admission gates.
Active remote process supervision distinguishes a failed health read from a confirmed exit. The executor tests exercise timeout recovery with a real listening process, unchanged ownership, and rejected identity evidence. The Runner restart tests require positive exit evidence before replacement. This coverage does not prove recovery of a historical retained sandbox or behavior during a live provider outage.
The same database suite pauses an original preparation while recovery claims the remote run. It then verifies that the old preparation yields without opening a provider or changing the recovery owner, for both live and expired leases. Heartbeat boundary tests verify that the detachment signal transfers recovery ownership before cleanup. The suite also rejects an older unmarked preparation and verifies that calling preparation again cannot add the fencing marker from supplied profile data.
For a native workspace owner hold, repeat issue reads and recovery checks after the task becomes done or cancelled. Verify one unchanged action ID, attempt count, and timestamps; task completion must not authorize copyback. Clear only the exact receipt after operator stop verification, then confirm stale action cleanup. The route regressions and heartbeat regression cover this sequence without a live operator recovery claim.
An intentionally Done onboarding parent can still need its validated
child-completion report. Recovery must not mark that callback successful before
execution solely because the parent is Done. The existing
stale-lock sweep suite
covers delayed native/Legacy startup, invalid parent/child/company/assignee
bindings and a dead owner. Process-death and ownership backstops remain active.
Repeat the real onboarding journey and correlate callback turn events with its
reply and downloadable child file; status alone does not prove delivery. The
literal ea6c8eb5e16d61111968f633a66748b397b64a3e live failure is retained,
and the revised behavior still needs live qualification.
run-detail
Automated: run workspace recovery tests this host; it is component coverage, not an end-to-end repair.
Manual: prepare a failed run with error code workspace_validation_failed, a
linked source task, and a live workspace_validation recovery action on that
task. Open that exact run and compare its recovery diagnosis to the source task.
Use an eligible action from the run surface, return to the task,
and verify the same action state, persisted outcome, and restored execution.
Repeat with an actor who lacks authority. Verify that the run-page controls are
absent for other error codes or after the source action retires; continue other
recovery checks on the source task. Full run-detail navigation remains a manual
gap.
operator-retry
Automated: disposition notices tests pending/error/gated states and waiting for the real retry acknowledgement. The recovery-card suite covers workspace divergence, reissue, reconcile, and confirmation. Workspace export covers quiet restoration, legacy/operator assistance and resolved states.
For a retained legacy or native file hold, open the source task after its recovery action has resolved but its files still need recovery. The task shows a compact operator-attention notice with inspection and collapsed diagnostics, without a file-certification form. A source-bound live recovery uses the ordinary working state; a missed or exhausted retry and an unknown action outcome remain visible. An authorized board operator can use the resolution API to record verified files and a backup link without changing task status or starting another run. The server independently validates the original stopped-execution receipt. Agent Chat retains the hold rather than suggesting another message. The notice and repair route cover quiet live recovery, read-only access, stale or missing stop proof, source-only repair, and preserved task status. These tests use fixtures; separately verify original files and their attached artifact before recording a real repair.
Repeat source-only repair with a succeeded legacy run that still has a retained
file obligation. Its result must remain successful, its current task owner and
status must stay unchanged, and later saved input must become eligible for normal
admission. A running or queued source and a normal retry of a successful turn
remain denied. The reconciliation regressions
cover these cases and stop-receipt validation without claiming live provider proof.
For an abandoned native copyback owner, the harness verifies the exact stopped
deployment and the reconciler restores saved work automatically. Task users see
no process details or shutdown form. The documented operator API handles older
receipts after independent platform verification, through a separate administrator
gate. The authorization regressions
and route regressions
cover administrator admission, ordinary-member and agent denial, suspended and
foreign membership, missing role data, and stale cloud admin rows.
The native workspace finalizer suite
runs the real reconciler with a valid saved yield from result_accepted,
workspace_finalizing, and retryable_failure. It verifies physical fencing
before verified stop, automatic and operator recovery, successful export and commitment, hold resolution,
unchanged saved results, retained approval/monitor waits, and no new provider wake.
These local regressions do not prove a deployment platform stopped a foreign writer.
Manual: reproduce an exhausted disposable action. Click once, verify pending state prevents repeated clicks, and inspect a successful continuation or an explicit rejection. Reload to confirm the durable outcome. For a workspace repair, record the compared branches/revisions and preserve uncommitted work; confirm the repair's actual result before resuming. Check the denied and stale action paths. A changed card label is insufficient proof of workspace integrity.
For graceful legacy shutdown, interrupt a disposable sandbox run while file export is in flight. Delay settlement past the first 30-second wait. Confirm the lease stays available, no repair hold or successor appears before export, and the finalizer drain waits for the same owner. After export, confirm saved files, successful lease cleanup, and one bounded continuation. Reload before checking the successor. Repeat with controller loss: the expired fence must retain the exact source and require repair, never claim restored files or start a second executor. The database-backed shutdown regression checks finalizer joining, release order, durable retry uniqueness, and reload. The settlement regressions cover other-controller fencing and owner loss. Provider boundaries are controlled in these tests; they do not qualify a live sandbox export.
For an abandoned native controller, follow the stop-verification procedure with a disposable deployment. A new platform-bound owner recovers automatically only after the harness signs proof that the exact deployment fully stopped. Legacy owners require independently verified stop evidence through the operator API; the task has no shutdown form. Confirm the original accepted result commits, the recovery hold resolves, saved files remain intact, and existing approval waits or monitor checks remain. No new agent turn should start. Repeat with a retry-phase owner and an ordinary runtime manager; the first can resume through automatic proof or authorized operator confirmation, while the second cannot use the operator API.
post-stop-message
Automated: legacy continuation, queue routes, and native restart recovery cover their respective recovery paths. Legacy failure continuation adds a browser journey; it is not native runtime proof.
Queued Stop database regressions verify that an operator cancellation before runtime selection releases its own locks, preserves deferred input without launching it, and leaves another active cycle of the same routine unchanged. Provider cancellations and already selected or dispatched runs retain their existing release behavior.
Manual: stop a disposable run, submit a distinguishable follow-up, and observe any cleanup wait. After confirmed stop, verify the follow-up reaches one eligible continuation and produces the requested answer. Repeat with a task/ancestor pause and a budget hold: recovery must not bypass the gate. For unknown external outcomes, inspect the provider before authorizing another operation.
browser-reconnect
Automated: live-update recovery covers client transport recovery with mocks. It does not prove runner restart.
Manual: go offline during a disposable run, let the server reach a new state, then reconnect. Confirm task/run/attention data refresh without duplicate comments or decisions; reload and compare. A healthy websocket proves transport only. Separately inspect the underlying execution if it is still stuck.
Gotchas
- Legacy and native runtimes have different ownership and restart paths. Record which one was exercised instead of inferring coverage from the adapter name.
- A process disappearance, elapsed timeout, or successful API retry does not prove an external tool operation never happened. Unknown outcomes stay unknown until corroborated; automatic replay can duplicate a side effect.
- Active-run watchdogs, task watchdogs, reviewers, and recovery owners have different authority. No recovery card grants board permission by itself.
- Idle Agent Chat is not stranded work. An intentional operator stop must not silently trigger recovery that fights the user's instruction.
- Source inspection and mocked component tests do not qualify real server restart continuity. Use the appropriate runtime suite for that claim.
- See execution semantics for the governing recovery contract and steering for saved follow-ups.
Storybook verification
exhaustion
Scope: ui Storybook: Exhausted/overdue attempts, operator retry pending/error, blocked authority/gate and specific escalation. Production: Inspect durable attempt bounds and actual eligible retry/repair; acknowledgment alone does not prove continued execution. Sources: Source: IssueRecoveryActionCard.tsx, DispositionRecoveryNotice.tsx, issue-recovery-actions.ts. Tests: IssueRecoveryActionCard.test.tsx, DispositionRecoveryNotice.test.tsx, issue-recovery-actions.test.ts, recovery.spec.ts
unknown-outcome
Scope: ui Storybook: Uncertain external outcome, required inspection/evidence and explicit permitted operator decision. Production: Corroborate provider outcome before authorization; enforce no automatic external side-effect replay and reject stale/unauthorized decisions. Sources: Source: IssueRecoveryActionCard.tsx, issue-recovery-actions.ts. Tests: IssueRecoveryActionCard.test.tsx, issue-recovery-actions.test.ts, recovery.spec.ts
user-continuation
Scope: ui Storybook: Saved follow-up waiting on stop cleanup, pause/budget gate, one eligible continuation and result/failure. Production: Confirm previous execution stopped before delivery, preserve message, avoid duplicate wakes and honor actual task/ancestor/budget gate in separate legacy/native paths. Sources: Source: TaskChatQueuedMessages.tsx, legacy-continuation.ts, native-restart-recovery.ts. Tests: legacy-continuation.test.ts, native-restart-recovery.test.ts, issue-queued-comments-routes.test.ts, legacy-failure-continuation.spec.ts