DottaandPaperclip
3f1ad6b25b
## Thinking Path > - Paperclip manages agent work through task and execution ownership. > - A task Retry must retain the user's request while the system verifies an old executor. > - Connected conversations also retain their external principal and accepted source batch. > - Chat Retry admission required containment before it could save a wait for that recovery. > - A saved chat receipt could also remain deferred after containment became available. > - This pull request lets the exact recovery-bound request wait and resume through its chat owner. > - Normal admission still checks containment, current access and policy before one successor starts. ## Linked Issues or Issue Description **What happened?** An ordinary task Retry on a connected conversation failed with `failed_chat_retry_not_authorized` while its retained remote executor still needed verification. The new recovery generation existed, but the chat action failed before it could save a dispatch wait. This occurred after the repairs in #17267 were confirmed live. The local regression found a second gap. A processed chat action with a deferred wake was not reconsidered after containment. Re-entering normal admission also tried to insert its existing durable receipt again. A historical hold for the same failed run could also hide the current Retry's workspace strategy, so the request still waited after the old executor was contained. Preparation could then read that sibling instead of the recovery record that owned the fresh successor, losing the fresh-workspace decision. **Expected behavior** Save the exact authorized Retry while its matching recovery generation verifies the executor. Once containment is recorded, reconsider the same chat receipt through normal admission. Start one successor with the original source batch. Keep publication, principal, pause and budget checks. **Steps to reproduce** 1. Admit a GitHub or Telegram request and record a failed native run with a retained remote lease. 2. Leave the executor's termination unconfirmed and exhaust its prior recovery generation. 3. Use ordinary task Retry to create a new recovery generation. 4. Process the connected Retry. The old admission code fails instead of retaining a wait. 5. With a deferred receipt, record exact containment and reconcile again. The old dispatch path leaves it stranded. **Paperclip version or commit** Reproduced from `412914cfa78d5abefcc34d3eb8d4bb86e461caac`. The branch is rebased onto current master. **Deployment mode** Hosted tenant for the reported failure. Isolated local PostgreSQL and the assembled heartbeat/chat services for the regression. Refs #17267 and #16965. This change does not add a Cloud dependency or change the absence protocol. ## What Changed - Permit a saved admission wait only for the exact current task Retry and active source recovery generation. - Keep promotion and execution dependent on independent containment proof. - Bind a Retry to active automatic recovery without replacing its generation, attempt count or deadline. - Share the current exact Retry strategy across historical holds for the same run. Retain each hold's stop and effect checks. - Select the canonical recovery record that proves the committed fresh successor during workspace preparation. - Let task receipt reconciliation return the same deferred action to its chat owner. - Recheck current board and external authority, then update the existing wake atomically with its successor. - Preserve the exact chat source batch. Keep generic board-comment filtering unchanged. - Add GitHub and Telegram cases for waiting, containment, one successor, stale generations, source edits, endpoint pause and supersession. - Update recovery documentation and capability maps. ## Verification - Red: real PostgreSQL tests reproduced the admission rejection for GitHub and Telegram. Six active-recovery cases then failed on the generation-ID assumption. Two same-run historical-hold cases remained deferred after containment on the old strategy lookup. A preparation regression then returned no readiness when the historical hold was stored first. - Green: 73 selected chat integration tests and 394 continuation, task Retry, workspace recovery, recovery-state and workspace-preparation tests passed. The chat tests use the real chat admission and heartbeat queue. Unrelated capacity stays occupied so no real provider starts. They assert no executor before containment, one successor afterward, unchanged source leases while waiting, preserved recovery budgets, and denial after authority changes. A separate source hold still blocks work. Both sibling insertion orders pass through the committed-readiness lookup and real workspace preparation: the new directory has independent Git storage and files, and unpublished old work remains intact. - `pnpm -r typecheck`, `node scripts/check-maps.mjs` and `git diff --check` passed after the review fixes. The source build passed on `f6660efa48e37a80cd9857217a537606de85b5c8`. - The local host reached its PostgreSQL semaphore limit. Successful runs used a test-only loader to reduce disposable cluster connection and worker limits. No repository database helper or application setting changed. - The full local `pnpm test:run` began before the final review edits, failed in its first server phase and was stopped after those failures. Later phases did not run. This is not a full local pass. The transient Retry-notice case passes after fixture isolation; the four Discord close-owned publication cases also pass in isolation. Other broad-run failure causes are not established. - Exact-head CI and security checks pass on `f6660efa48e37a80cd9857217a537606de85b5c8`; Greptile is 5/5 with zero unresolved review threads. Optional Paperclip Review completed neutral with a 5/5 static assessment of all 11 changed files and no findings; it did not run tests or hosted acceptance. All 57 current-head checks are terminal: 54 success, two skipped and the optional neutral review, with no failures or pending checks. - The real hosted workflow confirmed that the deployed absence repair recorded containment. A later ordinary UI Retry was accepted and saved a durable request, but it remains behind the historical source hold. The useful existing review remains unchanged. This follow-up is not deployed; a live successful successor and final task disposition remain unverified. No missing files are claimed recovered. ## Risks - The existing task reconciliation sweep can restage a deferred chat action. This reuses its receipt and generation; it does not grant new execution or publication authority. - Dispatch must keep the board Retry identity separate from the connected request's principal. The regression checks these bindings and preserves current chat authorization at each phase. - Historical holds reuse a strategy only from the locked same-run recovery that names the current Retry intent. A different run, malformed recovery record or independent source hold remains fenced. - Unknown executors without a matching active recovery still fail admission. Promotion and execution require containment even when admission saved a wait. - No schema or API shape changes. No credential changes, manual deployment or service restart. ## Model Used OpenAI GPT-6 through Codex, with repository tools, code execution and browser verification. The exact served model variant and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
8.7 KiB
Automatic task recovery
Infrastructure recovery belongs to the system. A task shows one working state, one scheduled retry, or an ordinary Retry action. Approval, credentials, assignment, budget and deliberate pause remain separate user decisions.
Admission and execution
POST /api/issues/:id/retry accepts { idempotencyKey, observedRunId }.
It requires authenticated board task-mutation authority. It records an
agent_wakeup_requests receipt before physical work. Company/key uniqueness
covers completed and failed receipts as well as pending ones. Concurrent clicks
attach to accepted work. A new deliberate click after exhaustion can create a
new generation; a repeated HTTP request cannot.
The existing exact-run retry endpoint retains its source-bound contract. Connected conversations use their existing inbound authorization owner. A board Retry changes the workspace strategy, not the external principal, source batch, conversation access or publication authority.
A connected-conversation Retry can save its exact request while the matching system recovery generation verifies the old executor. Saving that wait grants no execution authority. The chat owner rechecks the same board intent, external principal and source batch when it resumes the deferred receipt. Containment, current access, task eligibility, pause and budget gates must still pass before one successor starts. Restarts reuse the receipt; they do not create another Retry generation or another external publication.
Several retained recovery holds can describe the same failed execution. A historical hold without a system recovery record uses the current exact Retry's workspace strategy for that run. It does not borrow another run's strategy or skip any hold's independent containment and effect checks. Attaching a Retry to active automatic recovery preserves its generation, attempt count and deadline.
Agent Chat additionally requires the current conversation owner and enabled Agent Chat. Admission and dispatch both recheck these conditions under the issue lock. An unadmitted receipt from a former owner cannot own the current owner's Retry.
The normal dispatch queue rechecks task eligibility and claims saved input. An accepted native result goes back to its existing finalizer. Recovery does not create a second agent to reproduce that result.
Files, ownership and effects
The source recovery action holds a validated systemRecovery record. Its
revision and generation fence late bookkeeping writes. Physical exports still
use the existing workspace ownership lock and controller stop receipts.
The export owner can reopen its exact stopped sandbox through the environment runtime's recovery path. This path checks the current source action, run, environment, allocation, stop-only intent, and physical ownership claim before calling the provider. The retained-source fence still blocks ordinary resume and destruction. Recovery does not clear it to gain access to the saved files.
Execution containment and file preservation are independent facts:
- Another agent needs authoritative containment of the old executor.
- Copyback needs a quiescent writer and a verified destination.
- A fresh attempt uses a new execution workspace and the separate
isolated-workspace-attemptsnamespace. It does not modify the original directory or satisfy that source's preservation/cleanup obligation.
A missing managed sandbox cannot answer another stop request. Core can request a signed absence observation from Cloud under a host-bound server capability. Cloud reads the exact allocation in the historical credential's organization. Only a fresh signature for the exact stack, company, issue, run, lease, allocation and physical recovery owner can settle execution absence. Permission failures, a changed organization, unsigned answers and provider timeouts remain inconclusive. This protocol requires Cloud support before the Core consumer.
An absence receipt does not recover files or complete the task. The original run stays failed and its source remains protected. Existing authorized input may use normal fresh workspace admission only after every executor is contained. Known publications and uncertain external effects retain their normal receipt fences.
Fresh attempts currently require Core-managed storage. Plugin-owned storage does not yet support importing these preserved snapshots. Preparation stops before calling that provider or acquiring an environment. It retains the original workspace rather than reopening it as a fresh attempt.
Valid native baselines use the existing merge. A bound historical source without a merge baseline is exported to configured storage as an immutable, verified snapshot. Upload operations have stable identities and manifests. Recovered snapshots attach separately with provenance; they never become an invented baseline for overwriting newer files.
Known publication receipts and exact API mutation receipts carry forward. Uncertain API mutations and upstream tool mutations remain fenced across run replacement. Reads and independent work remain available. Generic upstream tools have no universal result-query protocol, so their unknown mutations require authoritative reconciliation rather than a new call key. Existing GitHub publication reconciliation stays with its dedicated receipt owner.
Retry policy and historical adoption
Maintenance allows an initial attempt and five retries at 30, 60, 120, 240 and 480 seconds with persisted ±20% jitter. Scheduling stops 30 minutes after the first failure. A longer provider cooldown takes precedence. Restarts, phase changes and rescans do not replenish the budget. Existing stricter execution retry limits remain separate.
The maintenance sweep classifies at most 50 historical records per tick. Indexed source queries and durable policy markers prevent repeat adoption. It includes retained leases, legacy runs without native descriptors, exhausted actions and resolved actions that still owe preservation. Historical failure evidence remains intact.
Saved authorized follow-ups may continue through normal dispatch. A task with no waiting request requires Retry before another agent turn. Completed, cancelled, deliberately paused and superseded tasks do not reopen merely because old files need maintenance.
historicalRecoveryCounts reports classification, preservation, input delivery,
waiting-for-Retry and unresolved-source counts for protected diagnostics.
Adoption alone is not proof that files were recovered.
Local diagnostics and presentation
Recovery and its durable failure evidence stay inside the stack. Exhausted attempts remain on the system-owned source recovery record with their failure code, evidence fingerprint and retry budget. The existing local server log records preservation failures. Recovery does not add a Cloud incident endpoint, notification worker or outbound reporting channel.
The task interruption projection drives thread, composer and list status. Thinking is optimistic presentation, not a fake run or provider-start record. Unknown admission reconciles the same request key. Backoff and real user gates replace Thinking when work is not active. Saved input is editable; Steer and Interrupt require a live target. Technical diagnostics remain inspectable.
Qualification
Run the focused PostgreSQL/filesystem suites from server/:
../node_modules/.bin/vitest run --config vitest.config.ts \
src/services/system-recovery-state.test.ts \
src/services/task-retry.test.ts \
src/services/workspace-recovery.test.ts \
src/services/explicit-native-continuation.test.ts \
src/services/heartbeat/workspace-preparation.test.ts \
src/services/native-runtime/native-chat-workspace.test.ts
server/scripts/workspace-recovery-provider.test.mjs is an opt-in controlled
Daytona probe. It requires PAPERCLIP_RECOVERY_LIVE=1, DAYTONA_API_KEY,
PAPERCLIP_DAYTONA_SDK_PATH and an immutable PAPERCLIP_RECOVERY_IMAGE.
An immutable provider snapshot UUID in PAPERCLIP_RECOVERY_SNAPSHOT_ID can
replace the image digest. The probe checks its ID and active state first.
It interrupts export,
joins the first controller, starts a second controller and verifies files,
preservation receipts, agent-start count and external-operation count before
cleanup. It does not claim an abrupt controller kill or a live LLM/GitHub test.
The local browser fixture uses the actual API, PostgreSQL and native runner with a deterministic model process. Verify immediate Thinking with delayed HTTP, one successor, the independent directory, unchanged source files, a useful result and stable completion after reload.
Neither this probe nor a green build proves hosted recovery. Hosted acceptance requires the actual served revision, the original saved input, readable preserved files, and confirmation that completed publications were not repeated.