mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-24 18:19:05 +02:00
DottaandPaperclip
eaacadeba3
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery must save retained workspace files before ordinary execution can resume. > - The environment runtime blocks access to retained sources to prevent unsafe reuse or deletion. > - The recovery worker also used that ordinary resume path. > - The guard returned no lease, so recovery reported an invalid source binding before it could save files. > - This pull request gives the physical preservation owner a separate, source-bound resume path. > - Ordinary execution and destructive cleanup keep their existing guards. ## Linked Issues or Issue Description Refs #16965. **What happened?** A retained workspace with a workspaceRestoreRecovery marker could not recover through the real environment runtime. The runtime returned null before calling the provider. Recovery classified this as source_binding_invalid. Earlier recovery fixtures replaced the runtime facade, so they did not exercise this guard. **Expected behavior** The verified preservation owner can reopen its exact stopped allocation to save files. Other callers remain blocked. Recovery preserves newer local files and does not start another agent or repeat a completed external action. **Steps to reproduce** 1. Seed a terminal source run with a retained sandbox, a recovery action, and the workspaceRestoreRecovery lease marker. 2. Give the source a verified stop receipt. 3. Call recoverRetainedWorkspace through environmentRuntimeService. 4. Before this change, the recovery action returns to blocked with source_binding_invalid. No files are copied. **Paperclip version or commit** Reproduced on master at65f59e339a. The same failure was observed in a served build containing #16965. ## What Changed - Add an internal preservation resume method. It validates the exact company, task, run, action, lease, environment, current physical owner, and stop-only cleanup claim before provider access. - Keep ordinary resume and destructive cleanup blocked for protected sources. - Run all retained-workspace fixtures through the assembled environment runtime. Add a failing-before/passing-after protected-source regression and four rejection cases. - Use the same runtime path and protection marker in the opt-in Daytona interruption probe. Allow a verified immutable snapshot UUID as its input. - Document the authority boundary and verification recipes. ## Verification - Red: the protected-source regression failed with expected recovered, received blocked. It recorded source_binding_invalid before provider resume. - Green: 235 tests passed across workspace recovery, durable task Retry, and environment runtime. These include saved files, newer local files, bounded retries, competing owners, and no extra run or wake. - pnpm -r typecheck and pnpm build passed. The initial sandboxed typecheck could not open a tsx IPC pipe; the same command passed with local process access. - node scripts/check-maps.mjs and git diff --check passed. - Live Daytona qualification passed using the same immutable snapshot ID as the affected deployment. Export was interrupted after a real download; a second controller restored the files and verified one original agent start, one simulated external action, one saved snapshot, and a stopped retained source. The disposable sandbox was deleted after verification. - Greptile reviewed804dfe24efat 5/5 with no actionable findings or unresolved review threads. - Full current-commit CI passed: https://github.com/paperclipai/paperclip/actions/runs/39540771747. Required ci / verify and ci / e2e passed. Superagent, Socket, and Snyk checks passed. - Full local pnpm test:run did not pass: 22,992 tests passed, one managed-run identity test timed out at 60 seconds, and 100 tests skipped across the first general-server phase. The command stopped before its later phases. The unchanged agent-identity.test.ts suite then passed all 11 tests in isolation in 15.53 seconds. The original timeout remains recorded; it is not reclassified as a full local pass. The same commit's complete CI test matrix passed. - The optional Paperclip Review completed with a neutral advisory 5/5 on the same head and no actionable findings. It inspected all eight changed files and connected code. It did not independently run tests or the provider probe. - Hosted ordinary Retry recorded one durable intent and no successor run while file preservation remained blocked. The existing published review stayed unchanged. Hosted recovery remains unverified until this follow-up is deployed. A Retry acknowledgement alone does not prove recovery. ## Risks - Recovery can reopen a retained sandbox. The dedicated path requires the current physical ownership claim and the exact source's stop-only intent. It does not clear the ordinary protection marker. - Provider failures retain the source and use the existing bounded recovery budget. The provider cannot substitute another allocation. - The live probe uses a disposable sandbox and simulated agent/external-effect counters. It does not prove a live LLM or GitHub review journey. - No schema, public API, permission, budget, pause, or approval changes. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code execution, browser verification, and test tools. The exact served model variant and context window are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; full suite still running) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
137 lines
7.0 KiB
Markdown
137 lines
7.0 KiB
Markdown
# Automatic task recovery
|
|
|
|
Infrastructure recovery belongs to the system. A task shows one working state,
|
|
one scheduled retry, or an ordinary **Retry** action. Approval, credentials,
|
|
assignment, budget and deliberate pause remain separate user decisions.
|
|
|
|
## Admission and execution
|
|
|
|
`POST /api/issues/:id/retry` accepts `{ idempotencyKey, observedRunId }`.
|
|
It requires authenticated board task-mutation authority. It records an
|
|
`agent_wakeup_requests` receipt before physical work. Company/key uniqueness
|
|
covers completed and failed receipts as well as pending ones. Concurrent clicks
|
|
attach to accepted work. A new deliberate click after exhaustion can create a
|
|
new generation; a repeated HTTP request cannot.
|
|
|
|
The existing exact-run retry endpoint retains its source-bound contract.
|
|
Connected conversations use their existing inbound authorization owner. A board
|
|
Retry changes the workspace strategy, not the external principal, source batch,
|
|
conversation access or publication authority.
|
|
|
|
Agent Chat additionally requires the current conversation owner and enabled
|
|
Agent Chat. Admission and dispatch both recheck these conditions under the
|
|
issue lock. An unadmitted receipt from a former owner cannot own the current
|
|
owner's Retry.
|
|
|
|
The normal dispatch queue rechecks task eligibility and claims saved input.
|
|
An accepted native result goes back to its existing finalizer. Recovery does
|
|
not create a second agent to reproduce that result.
|
|
|
|
## Files, ownership and effects
|
|
|
|
The source recovery action holds a validated `systemRecovery` record. Its
|
|
revision and generation fence late bookkeeping writes. Physical exports still
|
|
use the existing workspace ownership lock and controller stop receipts.
|
|
|
|
The export owner can reopen its exact stopped sandbox through the environment
|
|
runtime's recovery path. This path checks the current source action, run,
|
|
environment, allocation, stop-only intent, and physical ownership claim before
|
|
calling the provider. The retained-source fence still blocks ordinary resume
|
|
and destruction. Recovery does not clear it to gain access to the saved files.
|
|
|
|
Execution containment and file preservation are independent facts:
|
|
|
|
- Another agent needs authoritative containment of the old executor.
|
|
- Copyback needs a quiescent writer and a verified destination.
|
|
- A fresh attempt uses a new execution workspace and the separate
|
|
`isolated-workspace-attempts` namespace. It does not modify the original
|
|
directory or satisfy that source's preservation/cleanup obligation.
|
|
|
|
Fresh attempts currently require Core-managed storage. Plugin-owned storage
|
|
does not yet support importing these preserved snapshots. Preparation stops
|
|
before calling that provider or acquiring an environment. It retains the
|
|
original workspace rather than reopening it as a fresh attempt.
|
|
|
|
Valid native baselines use the existing merge. A bound historical source
|
|
without a merge baseline is exported to configured storage as an immutable,
|
|
verified snapshot. Upload operations have stable identities and manifests.
|
|
Recovered snapshots attach separately with provenance; they never become an
|
|
invented baseline for overwriting newer files.
|
|
|
|
Known publication receipts and exact API mutation receipts carry forward.
|
|
Uncertain API mutations and upstream tool mutations remain fenced across run
|
|
replacement. Reads and independent work remain available. Generic upstream
|
|
tools have no universal result-query protocol, so their unknown mutations
|
|
require authoritative reconciliation rather than a new call key. Existing
|
|
GitHub publication reconciliation stays with its dedicated receipt owner.
|
|
|
|
## Retry policy and historical adoption
|
|
|
|
Maintenance allows an initial attempt and five retries at 30, 60, 120, 240 and
|
|
480 seconds with persisted ±20% jitter. Scheduling stops 30 minutes after the
|
|
first failure. A longer provider cooldown takes precedence. Restarts, phase
|
|
changes and rescans do not replenish the budget. Existing stricter execution
|
|
retry limits remain separate.
|
|
|
|
The maintenance sweep classifies at most 50 historical records per tick.
|
|
Indexed source queries and durable policy markers prevent repeat adoption.
|
|
It includes retained leases, legacy runs without native descriptors, exhausted
|
|
actions and resolved actions that still owe preservation. Historical failure
|
|
evidence remains intact.
|
|
|
|
Saved authorized follow-ups may continue through normal dispatch. A task with
|
|
no waiting request requires Retry before another agent turn. Completed,
|
|
cancelled, deliberately paused and superseded tasks do not reopen merely
|
|
because old files need maintenance.
|
|
|
|
`historicalRecoveryCounts` reports classification, preservation, input delivery,
|
|
waiting-for-Retry and unresolved-source counts for protected diagnostics.
|
|
Adoption alone is not proof that files were recovered.
|
|
|
|
## Local diagnostics and presentation
|
|
|
|
Recovery and its durable failure evidence stay inside the stack. Exhausted
|
|
attempts remain on the system-owned source recovery record with their failure
|
|
code, evidence fingerprint and retry budget. The existing local server log
|
|
records preservation failures. Recovery does not add a Cloud incident endpoint,
|
|
notification worker or outbound reporting channel.
|
|
|
|
The task interruption projection drives thread, composer and list status.
|
|
Thinking is optimistic presentation, not a fake run or provider-start record.
|
|
Unknown admission reconciles the same request key. Backoff and real user gates
|
|
replace Thinking when work is not active. Saved input is editable; Steer and
|
|
Interrupt require a live target. Technical diagnostics remain inspectable.
|
|
|
|
## Qualification
|
|
|
|
Run the focused PostgreSQL/filesystem suites from `server/`:
|
|
|
|
```sh
|
|
../node_modules/.bin/vitest run --config vitest.config.ts \
|
|
src/services/system-recovery-state.test.ts \
|
|
src/services/task-retry.test.ts \
|
|
src/services/workspace-recovery.test.ts \
|
|
src/services/explicit-native-continuation.test.ts \
|
|
src/services/heartbeat/workspace-preparation.test.ts \
|
|
src/services/native-runtime/native-chat-workspace.test.ts
|
|
```
|
|
|
|
`server/scripts/workspace-recovery-provider.test.mjs` is an opt-in controlled
|
|
Daytona probe. It requires `PAPERCLIP_RECOVERY_LIVE=1`, `DAYTONA_API_KEY`,
|
|
`PAPERCLIP_DAYTONA_SDK_PATH` and an immutable `PAPERCLIP_RECOVERY_IMAGE`.
|
|
An immutable provider snapshot UUID in `PAPERCLIP_RECOVERY_SNAPSHOT_ID` can
|
|
replace the image digest. The probe checks its ID and active state first.
|
|
It interrupts export,
|
|
joins the first controller, starts a second controller and verifies files,
|
|
preservation receipts, agent-start count and external-operation count before
|
|
cleanup. It does not claim an abrupt controller kill or a live LLM/GitHub test.
|
|
|
|
The local browser fixture uses the actual API, PostgreSQL and native runner
|
|
with a deterministic model process. Verify immediate Thinking with delayed
|
|
HTTP, one successor, the independent directory, unchanged source files, a
|
|
useful result and stable completion after reload.
|
|
|
|
Neither this probe nor a green build proves hosted recovery. Hosted acceptance
|
|
requires the actual served revision, the original saved input, readable
|
|
preserved files, and confirmation that completed publications were not repeated.
|