DottaandPaperclip
d33b40a5c3
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The control plane must retain work when an execution or workspace export fails. > - Retained-workspace holds can leave saved input waiting with no usable Retry action. > - Users cannot verify process ownership or repair provider allocations. > - This pull request gives system recovery a durable budget and gives each deliberate Retry a durable intent. > - Safe recovery can preserve files, settle an accepted result, or prepare an independent workspace. > - One task state keeps infrastructure diagnostics out of the ordinary conversation while preserving real permission and approval gates. ## Linked Issues or Issue Description **What happened?** A task with a retained workspace can show recovery, failed-run and saved-message warnings at the same time. Historical runs without native workspace descriptors cannot use the native restore path. Old recovery metadata can prevent a new attempt even when an independent workspace would be safe. **Expected behavior** Paperclip owns bounded infrastructure retries and file preservation. A user can request another attempt with ordinary Retry. Thinking appears immediately. The service preserves completed effects and current user controls. It does not claim that starting fresh recovered the old files. **Steps to reproduce** 1. Retain a failed run and its workspace after export or process-stop failure. 2. Save a follow-up while the recovery hold is active. 3. Open the task and try to continue without infrastructure credentials or operator certification. Refs #16906, #16835, #16883. This extends their recovery and receipt contracts. Related work: #16962 and #16963. ## What Changed - Add task-level Retry with company-scoped authorization, all-status idempotency and durable admission before physical work. Keep exact-run Retry compatible. - Add typed recovery phases, finite persisted backoff, historical adoption and durable local failure diagnostics. - Preserve native workspaces or verified historical snapshots. Admit an independent fresh workspace only after execution containment. Keep old-source preservation and cleanup obligations separate. - Carry completed publication and API receipts forward. Fence uncertain mutation retries across replacement runs. Preserve connected-chat identity and authorization. - Re-enter normal dispatch for saved input. Reuse accepted results through the existing finalizer. - Use one task interruption projection across the thread, composer and task list. Show optimistic Thinking, ordinary Retry and genuine user gates. - Add PostgreSQL, filesystem, API, browser-fixture and controlled provider coverage. Document the behavior and update the feature/access maps. ## Verification - All GitHub Actions CI jobs pass on `6d8c6d27d50586d8548b31d03b47f6d73a0ee11d`: typecheck, all server/chat/workspace and serialized suites, Runner checks, eight browser shards, canary and build. The required merge gate passes. - Greptile reviewed the final commit at 5/5, with no actionable findings or unresolved review threads. Its two earlier P1 findings are fixed. - Full recursive typecheck and build pass locally on the final commit. All 16 task Retry cases pass against isolated real PostgreSQL. The unrelated-success case and both stale Retry cases fail on the previous code and pass after the fix. - The corrected gateway fixture passes its uncertain-mutation and independent-read case. Four real HTTP API cases pass for definite 403/422 rejection, completed-effect reuse, and uncertain-effect fences across deliberate and automatic retries. These focused gateway/API checks used a temporary database lifecycle harness on an existing healthy local PostgreSQL process. The standard repository helper is unchanged. - Earlier qualification covers 291 task UI cases, saved-file copyback and ancestry, current Agent Chat ownership races, nine chat Retry API browser cases, and six assembled browser recovery fixtures. A real API/PostgreSQL/native-runner journey showed Thinking in 25 ms despite a delayed response, admitted one successor in a separate workspace, retained original files, and survived reload. - The controlled Daytona test interrupts export, joins the controller, restarts it in a separate process, and verifies one agent start, one external sentinel operation, retained file contents and no premature cleanup. It does not establish abrupt-kill or live LLM/GitHub acceptance. - The full local `pnpm test:run` was attempted and was not green: an OAuth row-lock timing assertion failed, and a later suite exhausted macOS PostgreSQL semaphores. The ten OAuth timing cases passed on rerun against a separate database. Later embedded database initialization failures remain recorded; the final CI matrix passes. - Security scanning passed on production-code commit `63953edf069b111c6becc3144dffa6deb52f68b5`. Its only successor changes one test-fixture identity line. The final scan is neutral/inconclusive because its pinned merge-base checkout failed or timed out. Both rerun API requests returned 404, so no fresh successful scan is claimed. Socket and Snyk checks pass on the final head. - The optional Paperclip Review remains queued behind an earlier run in workspace finalization. Its prior completed assessment was incomplete because a comparison tool was blocked by the platform filter. It is not counted as a completed code review. GitHub does not require this check for merge. - Hosted acceptance remains pending deployment. Local and CI results do not establish recovery of a hosted task or its retained files. ## Risks - Recovery changes span admission, physical ownership and dispatch. Incorrect evidence could permit overlapping work. Existing physical locks and authoritative stop requirements remain in place. - The migration adds concurrent indexes. It requires the existing top-level migration runner and retries incomplete index creation safely. - A fresh attempt can complete while old-source preservation remains unresolved. The service retains that obligation and does not report those files as recovered. - Unknown tool mutations remain blocked until their authoritative receipt owner resolves them. Generic upstream tools have no universal result-query protocol. - Recovery and diagnostics stay inside the stack. No Cloud incident endpoint, notification worker or new outbound reporting channel is included. ## Model Used OpenAI Codex, GPT-6 family. The runtime does not expose a more specific model version or context-window size. The model used reasoning, shell execution, source editing and browser tools. No model subagents were used for this implementation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; the full local run limitation is documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
6.5 KiB
Automatic task recovery
Infrastructure recovery belongs to the system. A task shows one working state, one scheduled retry, or an ordinary Retry action. Approval, credentials, assignment, budget and deliberate pause remain separate user decisions.
Admission and execution
POST /api/issues/:id/retry accepts { idempotencyKey, observedRunId }.
It requires authenticated board task-mutation authority. It records an
agent_wakeup_requests receipt before physical work. Company/key uniqueness
covers completed and failed receipts as well as pending ones. Concurrent clicks
attach to accepted work. A new deliberate click after exhaustion can create a
new generation; a repeated HTTP request cannot.
The existing exact-run retry endpoint retains its source-bound contract. Connected conversations use their existing inbound authorization owner. A board Retry changes the workspace strategy, not the external principal, source batch, conversation access or publication authority.
Agent Chat additionally requires the current conversation owner and enabled Agent Chat. Admission and dispatch both recheck these conditions under the issue lock. An unadmitted receipt from a former owner cannot own the current owner's Retry.
The normal dispatch queue rechecks task eligibility and claims saved input. An accepted native result goes back to its existing finalizer. Recovery does not create a second agent to reproduce that result.
Files, ownership and effects
The source recovery action holds a validated systemRecovery record. Its
revision and generation fence late bookkeeping writes. Physical exports still
use the existing workspace ownership lock and controller stop receipts.
Execution containment and file preservation are independent facts:
- Another agent needs authoritative containment of the old executor.
- Copyback needs a quiescent writer and a verified destination.
- A fresh attempt uses a new execution workspace and the separate
isolated-workspace-attemptsnamespace. It does not modify the original directory or satisfy that source's preservation/cleanup obligation.
Fresh attempts currently require Core-managed storage. Plugin-owned storage does not yet support importing these preserved snapshots. Preparation stops before calling that provider or acquiring an environment. It retains the original workspace rather than reopening it as a fresh attempt.
Valid native baselines use the existing merge. A bound historical source without a merge baseline is exported to configured storage as an immutable, verified snapshot. Upload operations have stable identities and manifests. Recovered snapshots attach separately with provenance; they never become an invented baseline for overwriting newer files.
Known publication receipts and exact API mutation receipts carry forward. Uncertain API mutations and upstream tool mutations remain fenced across run replacement. Reads and independent work remain available. Generic upstream tools have no universal result-query protocol, so their unknown mutations require authoritative reconciliation rather than a new call key. Existing GitHub publication reconciliation stays with its dedicated receipt owner.
Retry policy and historical adoption
Maintenance allows an initial attempt and five retries at 30, 60, 120, 240 and 480 seconds with persisted ±20% jitter. Scheduling stops 30 minutes after the first failure. A longer provider cooldown takes precedence. Restarts, phase changes and rescans do not replenish the budget. Existing stricter execution retry limits remain separate.
The maintenance sweep classifies at most 50 historical records per tick. Indexed source queries and durable policy markers prevent repeat adoption. It includes retained leases, legacy runs without native descriptors, exhausted actions and resolved actions that still owe preservation. Historical failure evidence remains intact.
Saved authorized follow-ups may continue through normal dispatch. A task with no waiting request requires Retry before another agent turn. Completed, cancelled, deliberately paused and superseded tasks do not reopen merely because old files need maintenance.
historicalRecoveryCounts reports classification, preservation, input delivery,
waiting-for-Retry and unresolved-source counts for protected diagnostics.
Adoption alone is not proof that files were recovered.
Local diagnostics and presentation
Recovery and its durable failure evidence stay inside the stack. Exhausted attempts remain on the system-owned source recovery record with their failure code, evidence fingerprint and retry budget. The existing local server log records preservation failures. Recovery does not add a Cloud incident endpoint, notification worker or outbound reporting channel.
The task interruption projection drives thread, composer and list status. Thinking is optimistic presentation, not a fake run or provider-start record. Unknown admission reconciles the same request key. Backoff and real user gates replace Thinking when work is not active. Saved input is editable; Steer and Interrupt require a live target. Technical diagnostics remain inspectable.
Qualification
Run the focused PostgreSQL/filesystem suites from server/:
../node_modules/.bin/vitest run --config vitest.config.ts \
src/services/system-recovery-state.test.ts \
src/services/task-retry.test.ts \
src/services/workspace-recovery.test.ts \
src/services/explicit-native-continuation.test.ts \
src/services/heartbeat/workspace-preparation.test.ts \
src/services/native-runtime/native-chat-workspace.test.ts
server/scripts/workspace-recovery-provider.test.mjs is an opt-in controlled
Daytona probe. It requires PAPERCLIP_RECOVERY_LIVE=1, DAYTONA_API_KEY,
PAPERCLIP_DAYTONA_SDK_PATH and an immutable PAPERCLIP_RECOVERY_IMAGE.
It interrupts export,
joins the first controller, starts a second controller and verifies files,
preservation receipts, agent-start count and external-operation count before
cleanup. It does not claim an abrupt controller kill or a live LLM/GitHub test.
The local browser fixture uses the actual API, PostgreSQL and native runner with a deterministic model process. Verify immediate Thinking with delayed HTTP, one successor, the independent directory, unchanged source files, a useful result and stable completion after reload.
Neither this probe nor a green build proves hosted recovery. Hosted acceptance requires the actual served revision, the original saved input, readable preserved files, and confirmation that completed publications were not repeated.