mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 12:04:41 +01:00
DottaandPaperclip
338bf75761
## Thinking Path > - Paperclip manages agents and their tasks. > - Task Retry must preserve saved input and enforce current access. > - A GitHub reconnect invalidates old runtime callbacks. > - Retry reused that old callback authority and rejected valid new intent. > - Retry also replayed an older failed request when a follow-up was already saved. > - This change admits a deliberate Retry under current authority and transfers saved input once. ## Linked Issues or Issue Description Refs #16965, #17053 and #17317. These repairs preserve work and contain old executors. This change fixes the subsequent chat admission boundary. A search found no overlapping open reconnect/admission fix. **What happened?** Ordinary task Retry returned `chat_failed_run_retry_not_authorized` after the same GitHub App reconnected. The endpoint was healthy, but the saved request still carried its old runtime generation. A newer queued follow-up also made Retry reject the older failed request instead of continuing the follow-up. No successor was admitted. HTTP qualification found two related defects. Concurrent clicks could omit the canonical chat action from a response. Signed-in board dispatch also passed an incomplete task resource to the permission resolver, which rejected a permitted mutation. **Expected behavior** A deliberate Retry checks current authority for the same App, sender, repository, conversation and task. It continues saved input once. A later reconnect, changed App, revoked access or changed source prevents dispatch. Old runtime callbacks remain invalid. **Steps to reproduce** 1. Admit a GitHub request and record a failed native run. 2. Save a follow-up while execution is blocked. 3. Reconnect the same GitHub App. 4. Click ordinary task Retry. 5. Before this change, admission returns 409. Without reconnect, the newer input alone also causes 409. **Paperclip version or commit** Baseline source: `41c42895fa862c586ff841f8c50200d93ffc83a4`. **Deployment mode** Source checkout on macOS. Regressions use real PostgreSQL transactions and the assembled chat, HTTP route and heartbeat admission services. Provider boundaries are simulated. No hosted recovery is claimed. ## What Changed - Record the admitting App identity for new GitHub deliveries. Historical deliveries can use their original working-comment binding. Reauthorize deliberate Retry under a pinned current generation and credential-reference fingerprint. - Select the complete saved-input queue from the failed run's actual conversation before replaying the older failed batch. Keep exact-run Retry bounded while ordinary task Retry supports larger queues. Transfer its receipts with the dispatch transaction. Recover the same ownership after restart or a lost response. Release unadmitted input after a definitive rejection. - Preserve canonical chat action IDs across concurrent clicks. Supply current task status and assignment fields to board mutation checks. - Keep exact-run Retry, stale webhook, containment, source integrity, publication, pause and budget checks. Update recovery contracts and verification recipes. ## Verification - Red: all three new PostgreSQL regressions failed on unchanged production code. Reconnect failed at chat authorization. Newer input failed at the obsolete-source guard. The combined case also failed. - Green: 61 selected chat integration cases passed. These include 27 new cases and 34 existing cases for containment, retained-source recovery, exact batches and source revocation. - A further red/green PostgreSQL regression reproduces the 51-message overflow identified during review. All 51 messages reach one successor, including a later Retry of that successor. A separate 52-message consumed-batch regression also passes. - A competing conversation retains its own deferred message. Distinct automatic policy contexts are checked independently; revoking only the later policy blocks the whole unadmitted batch. Batch validation still rejects a later message with a changed App binding or deleted source. A real PostgreSQL query capture reduced the 51-message preparation from 1,103 calls to fewer than 180. - All 16 durable task Retry tests passed. - The real HTTP route returns 202 and admits one queued successor for concurrent deliveries. Fault injection covers rollback during input handoff, response loss after dispatch commit, and permission loss when resuming an existing durable wait. A wait only adopts input after current admission succeeds; a one-shot settlement marker keeps status projection read-only. A new deliberate Retry works after another reconnect or a failed successor. - Current-App, repository, sender, source edit/delete, reassignment, endpoint pause and board-permission controls passed. Historical deliveries remain unchanged. - Final-head CI: 55 checks passed, three expected skips. The unchanged minute-bucket rate-limit assertion passed locally against PostgreSQL and on its one CI rerun. Greptile reviewed the final commit at 5/5 with no unresolved threads. - Workspace typecheck and full build passed. Map alignment, module boundaries, UI token gates and `git diff --check` passed. - Hosted task recovery and a live provider turn remain unverified. The local tests hold agent capacity to inspect admission without starting a model or publishing a GitHub response. ## Risks - A deliberate GitHub Retry can admit old saved input under current connection authority. It requires the same App binding and current access. It pins that admission; it does not rewrite the historical delivery or make stale callbacks valid. - The durable chat queue has an optional admission callback. It transfers input only when the transaction contains a valid queued execution or saved wait. Other chat consumers have no callback and retain their behavior. - No migration, Cloud dependency, deployment, configuration change or workspace cleanup is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source inspection, code execution and GitHub tools. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
182 lines
10 KiB
Markdown
182 lines
10 KiB
Markdown
# Automatic task recovery
|
|
|
|
Infrastructure recovery belongs to the system. A task shows one working state,
|
|
one scheduled retry, or an ordinary **Retry** action. Approval, credentials,
|
|
assignment, budget and deliberate pause remain separate user decisions.
|
|
|
|
## Admission and execution
|
|
|
|
`POST /api/issues/:id/retry` accepts `{ idempotencyKey, observedRunId }`.
|
|
It requires authenticated board task-mutation authority. It records an
|
|
`agent_wakeup_requests` receipt before physical work. Company/key uniqueness
|
|
covers completed and failed receipts as well as pending ones. Concurrent clicks
|
|
attach to accepted work. A new deliberate click after exhaustion can create a
|
|
new generation; a repeated HTTP request cannot.
|
|
|
|
The existing exact-run retry endpoint retains its source-bound contract.
|
|
Connected conversations use their existing inbound authorization owner. A board
|
|
Retry preserves external principal, conversation access and publication checks.
|
|
A GitHub Retry selects already-saved, undelivered input before replaying the
|
|
failed run's older request. The selected receipts transfer to the retry owner
|
|
under the dispatch transaction. They link to its one successor, including after
|
|
a restart or a lost dispatch response. Rejected, unadmitted retries release the
|
|
saved input for another deliberate Retry.
|
|
|
|
A deliberate GitHub Retry can admit saved input after the same App reconnects.
|
|
It verifies the App binding in the accepted delivery or its historical working
|
|
comment receipt, then checks current sender, repository, conversation and task
|
|
access. The retry records the current runtime generation and credential-reference
|
|
fingerprint. Another reconnect invalidates that admission. Historical deliveries,
|
|
stale webhook callbacks and the exact-run retry contract retain their fences.
|
|
|
|
Task Retry keeps the complete saved-input batch, including batches larger than
|
|
the exact-run endpoint's limit. Source rows, receipts and invalidations are read
|
|
in batches. Within one transaction, messages with the same principal and automatic
|
|
policy context share a current-access check; every message still proves its App,
|
|
destination, comment link and immutable source. Different automatic policy contexts
|
|
receive separate access checks. Input in another conversation remains queued.
|
|
|
|
A connected-conversation Retry can save its exact request while the matching
|
|
system recovery generation verifies the old executor. Saving that wait grants
|
|
no execution authority. The chat owner rechecks the same board intent, external
|
|
principal and source batch when it resumes the deferred receipt. Containment,
|
|
current access, task eligibility, pause and budget gates must still pass before
|
|
one successor starts. Restarts reuse the receipt; they do not create another
|
|
Retry generation or another external publication.
|
|
|
|
Several retained recovery holds can describe the same failed execution. A
|
|
historical hold without a system recovery record uses the current exact Retry's
|
|
workspace strategy for that run. It does not borrow another run's strategy or
|
|
skip any hold's independent containment and effect checks. Attaching a Retry to
|
|
active automatic recovery preserves its generation, attempt count and deadline.
|
|
|
|
Agent Chat additionally requires the current conversation owner and enabled
|
|
Agent Chat. Admission and dispatch both recheck these conditions under the
|
|
issue lock. An unadmitted receipt from a former owner cannot own the current
|
|
owner's Retry.
|
|
|
|
The normal dispatch queue rechecks task eligibility and claims saved input.
|
|
An accepted native result goes back to its existing finalizer. Recovery does
|
|
not create a second agent to reproduce that result.
|
|
|
|
## Files, ownership and effects
|
|
|
|
The source recovery action holds a validated `systemRecovery` record. Its
|
|
revision and generation fence late bookkeeping writes. Physical exports still
|
|
use the existing workspace ownership lock and controller stop receipts.
|
|
|
|
The export owner can reopen its exact stopped sandbox through the environment
|
|
runtime's recovery path. This path checks the current source action, run,
|
|
environment, allocation, stop-only intent, and physical ownership claim before
|
|
calling the provider. The retained-source fence still blocks ordinary resume
|
|
and destruction. Recovery does not clear it to gain access to the saved files.
|
|
|
|
Execution containment and file preservation are independent facts:
|
|
|
|
- Another agent needs authoritative containment of the old executor.
|
|
- Copyback needs a quiescent writer and a verified destination.
|
|
- A fresh attempt uses a new execution workspace and the separate
|
|
`isolated-workspace-attempts` namespace. It does not modify the original
|
|
directory or satisfy that source's preservation/cleanup obligation.
|
|
|
|
A missing managed sandbox cannot answer another stop request. Core can request
|
|
a signed absence observation from Cloud under a host-bound server capability.
|
|
Cloud reads the exact allocation in the historical credential's organization.
|
|
Only a fresh signature for the exact stack, company, issue, run, lease,
|
|
allocation and physical recovery owner can settle execution absence. Permission
|
|
failures, a changed organization, unsigned answers and provider timeouts remain
|
|
inconclusive. This protocol requires Cloud support before the Core consumer.
|
|
|
|
An absence receipt does not recover files or complete the task. The original run
|
|
stays failed and its source remains protected. Existing authorized input may use
|
|
normal fresh workspace admission only after every executor is contained. Known
|
|
publications and uncertain external effects retain their normal receipt fences.
|
|
|
|
Fresh attempts currently require Core-managed storage. Plugin-owned storage
|
|
does not yet support importing these preserved snapshots. Preparation stops
|
|
before calling that provider or acquiring an environment. It retains the
|
|
original workspace rather than reopening it as a fresh attempt.
|
|
|
|
Valid native baselines use the existing merge. A bound historical source
|
|
without a merge baseline is exported to configured storage as an immutable,
|
|
verified snapshot. Upload operations have stable identities and manifests.
|
|
Recovered snapshots attach separately with provenance; they never become an
|
|
invented baseline for overwriting newer files.
|
|
|
|
Known publication receipts and exact API mutation receipts carry forward.
|
|
Uncertain API mutations and upstream tool mutations remain fenced across run
|
|
replacement. Reads and independent work remain available. Generic upstream
|
|
tools have no universal result-query protocol, so their unknown mutations
|
|
require authoritative reconciliation rather than a new call key. Existing
|
|
GitHub publication reconciliation stays with its dedicated receipt owner.
|
|
|
|
## Retry policy and historical adoption
|
|
|
|
Maintenance allows an initial attempt and five retries at 30, 60, 120, 240 and
|
|
480 seconds with persisted ±20% jitter. Scheduling stops 30 minutes after the
|
|
first failure. A longer provider cooldown takes precedence. Restarts, phase
|
|
changes and rescans do not replenish the budget. Existing stricter execution
|
|
retry limits remain separate.
|
|
|
|
The maintenance sweep classifies at most 50 historical records per tick.
|
|
Indexed source queries and durable policy markers prevent repeat adoption.
|
|
It includes retained leases, legacy runs without native descriptors, exhausted
|
|
actions and resolved actions that still owe preservation. Historical failure
|
|
evidence remains intact.
|
|
|
|
Saved authorized follow-ups may continue through normal dispatch. A task with
|
|
no waiting request requires Retry before another agent turn. Completed,
|
|
cancelled, deliberately paused and superseded tasks do not reopen merely
|
|
because old files need maintenance.
|
|
|
|
`historicalRecoveryCounts` reports classification, preservation, input delivery,
|
|
waiting-for-Retry and unresolved-source counts for protected diagnostics.
|
|
Adoption alone is not proof that files were recovered.
|
|
|
|
## Local diagnostics and presentation
|
|
|
|
Recovery and its durable failure evidence stay inside the stack. Exhausted
|
|
attempts remain on the system-owned source recovery record with their failure
|
|
code, evidence fingerprint and retry budget. The existing local server log
|
|
records preservation failures. Recovery does not add a Cloud incident endpoint,
|
|
notification worker or outbound reporting channel.
|
|
|
|
The task interruption projection drives thread, composer and list status.
|
|
Thinking is optimistic presentation, not a fake run or provider-start record.
|
|
Unknown admission reconciles the same request key. Backoff and real user gates
|
|
replace Thinking when work is not active. Saved input is editable; Steer and
|
|
Interrupt require a live target. Technical diagnostics remain inspectable.
|
|
|
|
## Qualification
|
|
|
|
Run the focused PostgreSQL/filesystem suites from `server/`:
|
|
|
|
```sh
|
|
../node_modules/.bin/vitest run --config vitest.config.ts \
|
|
src/services/system-recovery-state.test.ts \
|
|
src/services/task-retry.test.ts \
|
|
src/services/workspace-recovery.test.ts \
|
|
src/services/explicit-native-continuation.test.ts \
|
|
src/services/heartbeat/workspace-preparation.test.ts \
|
|
src/services/native-runtime/native-chat-workspace.test.ts
|
|
```
|
|
|
|
`server/scripts/workspace-recovery-provider.test.mjs` is an opt-in controlled
|
|
Daytona probe. It requires `PAPERCLIP_RECOVERY_LIVE=1`, `DAYTONA_API_KEY`,
|
|
`PAPERCLIP_DAYTONA_SDK_PATH` and an immutable `PAPERCLIP_RECOVERY_IMAGE`.
|
|
An immutable provider snapshot UUID in `PAPERCLIP_RECOVERY_SNAPSHOT_ID` can
|
|
replace the image digest. The probe checks its ID and active state first.
|
|
It interrupts export,
|
|
joins the first controller, starts a second controller and verifies files,
|
|
preservation receipts, agent-start count and external-operation count before
|
|
cleanup. It does not claim an abrupt controller kill or a live LLM/GitHub test.
|
|
|
|
The local browser fixture uses the actual API, PostgreSQL and native runner
|
|
with a deterministic model process. Verify immediate Thinking with delayed
|
|
HTTP, one successor, the independent directory, unchanged source files, a
|
|
useful result and stable completion after reload.
|
|
|
|
Neither this probe nor a green build proves hosted recovery. Hosted acceptance
|
|
requires the actual served revision, the original saved input, readable
|
|
preserved files, and confirmation that completed publications were not repeated.
|