mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
fix(recovery): verify cross-task warm sandbox handoffs and say plainly when recovery refuses (#20282)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes keep a warm sandbox and hand it to
the next run through a chain of reusable leases
> - Acquisition binds that chain to the execution workspace, so a task
can inherit a warm sandbox from an earlier task in the same workspace
> - Saved-files recovery verifies the same chain, but it required every
earlier lease, run, and finalization to belong to the recovering task
> - A valid chain that started with another task therefore always failed
with `allocation_ownership_ambiguous`. That code is `service_required`,
so recovery stopped after one attempt, and each board Retry repeated the
same refusal
> - This pull request makes the chain check match the acquisition rule,
and it checks each predecessor against its own task
> - It also makes the board say plainly when recovery refuses the saved
state, because "failed after automatic retries" was not true and Retry
looked like it did nothing
> - The benefit is that a task blocked with "Saving the agent's files
failed after automatic retries" can recover its saved files, and a
refusal that Retry cannot change is shown with its reason
## Linked Issues or Issue Description
**What happened?**
A native run in a remote sandbox finished its turn, and its result was
accepted. The copy of its files back to the host failed and was
scheduled to retry. A graceful server restart then interrupted the
retry, so the run settled with
`native_workspace_sync_out_retry_exhausted`, and the task became
blocked. Every saved-files recovery generation then failed on its first
attempt with `allocation_ownership_ambiguous` (`service_required`). The
board showed "Saving the agent's files failed after automatic retries.
Retry to try again". Each Retry replaced the generation
(`issue.recovery_generation_replaced`), and the new generation failed in
the same way within one second. To the user, Retry did nothing. The
sandbox lease chain was valid: same agent, same execution workspace,
same sandbox, and matching handoff receipts on all eight leases. But its
first three leases belonged to an earlier task in the same workspace.
**Expected behavior**
Recovery must accept a warm sandbox chain that acquisition accepted.
Acquisition binds reuse to the execution workspace, and it requires the
same task only when the lease has no workspace. Recovery must check each
predecessor run and finalization against that predecessor's own task. It
must still refuse a chain whose records disagree or that has no shared
workspace.
**Steps to reproduce**
1. Run a native task in a remote sandbox with `reuse_by_environment`,
with an execution workspace that another task also uses.
2. Run the other task in the same workspace first, so the sandbox lease
is handed from that task to this one.
3. Make the last run of the second task fail to copy its files back (for
example, restart the server during the copy-back retry).
4. Wait for saved-files recovery, or click Retry. Recovery fails with
`allocation_ownership_ambiguous`, and Retry repeats the failure.
**Paperclip version or commit**
`master` at a3a25ce6c9 (the same code as the 2026.1106.0 staging build
for every file this change touches).
## What Changed
- `retiredReusableLeasePredecessors` (`environment-lease-handoff.ts`)
requires the same task only when the lease has no execution workspace.
This matches the binding that `acquireLease` uses when it replaces a
reusable lease.
- `inspectWorkspaceRecoverySource` (`workspace-recovery-source.ts`)
checks each predecessor run and its finalization against the predecessor
lease's own task, not the recovering task. A predecessor from another
task is accepted only when both leases have the same execution
workspace. All other checks stay: same agent, terminal run, no live
controller or lease owner, a settled finalization, and a saved snapshot
at an older epoch on the same allocation, workspace, and remote
directory.
- The other callers of the chain walk (remote termination resolution and
retired backup restore) get the same rule.
- The test fixture `warmHandoffSource` gets an option that starts the
chain with an earlier task in a shared workspace.
- Honest copy for exhausted recovery (`exhaustedSystemRecoveryAction`):
- A deterministic refusal (`service_required`) now says "Paperclip
refused to recover the saved files", names the reason in plain words and
the code, and says that Retry runs the same check again and succeeds
only after Paperclip is updated.
- Other exhausted recovery says "stopped after N attempts" only when
there were several attempts. One attempt says "failed".
- Continuation (`explicit-native-continuation.ts`) reports a new wait
reason, `saved_work_recovery_refused`, when the exhausted recovery is a
deterministic refusal. The task presentation shows its own message for
it. `saved_work_recovery_failed` no longer claims automatic retries.
- Retry stays available for a refusal. It is the action that runs
recovery again after an update. After each Retry, the board shows the
same refusal and its reason, not an unchanged generic notice.
### Recovery refusal classes
| Retry class | Failure codes | Automatic recovery | Board Retry |
| --- | --- | --- | --- |
| `retry_transient` | `provider_unavailable`, `export_interrupted` and
other transient codes | Retries with backoff, up to 5 retries in 30
minutes | Starts a new recovery round |
| `recheck_evidence` | `agent_stop_unconfirmed`,
`workspace_writer_active`, `effect_unresolved`,
`copyback_stop_unconfirmed` | Checks the evidence again later | Starts a
new round (the copy-back stop case has its own wait text) |
| `use_alternative_strategy` | `historical_source_missing`,
`source_unavailable` (no recorded source) | Stops. The retained-source
fallback can continue from the last save | Starts a fresh attempt |
| `service_required` (deterministic) | `allocation_ownership_ambiguous`,
`source_binding_invalid`, `snapshot_verification_failed`,
`archive_unsafe`, `source_unavailable` (recorded source gone) | Stops
after one attempt. The same saved state gives the same result | Runs the
same check again. The board now says this and names the reason |
## Verification
- Red before the fix: `recovers saved work after 1|2 warm handoffs that
began with another task in the same workspace` failed with `expected
'allocation_ownership_ambiguous' to be null`.
- Green after the fix: `npx vitest run
src/services/workspace-recovery-warm-handoff.test.ts
src/services/workspace-recovery-source.test.ts` passed (60 tests).
- Safety cases that stay fenced: a predecessor lease that names a
different task than its run (`lease_task`), and a cross-task predecessor
with no shared workspace (`unshared_workspace`). The existing `task`
fault (another task with no workspace) also stays fenced.
- Sibling and retry suites passed: environment runtime and orphan
sandbox, cancelled workspace preservation, workspace recovery fallback,
task retry, remote workspace lifecycle, explicit native continuation,
native cleanup admission, and remote retry admission.
- Copy, red before and green after: `system-recovery-state.test.ts`
(`states a deterministic refusal plainly and never claims retries that
did not happen`), and the end-to-end `names a deterministic saved-files
refusal behind Retry when no fallback can continue, and another Retry
shows it again` in `workspace-recovery-preparation-gates.test.ts`. On
the old code the wait reason was `saved_work_recovery_failed`. With the
change it is `saved_work_recovery_refused`, with the refusal reason, for
both Retry clicks.
- Updated expectation: the board handoff in
`workspace-recovery-retry-copyback.test.ts` now expects the refusal text
for its `source_binding_invalid` source (79 tests pass).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.
- I compared the real incident's lease chain with every predecessor
condition through read-only diagnostics. All eight leases match on
agent, sandbox, workspace, and remote directory, and each predecessor
run saved a snapshot at an older epoch.
## Risks
- Recovery now trusts a predecessor from another task in the same
execution workspace. Acquisition already allows that handoff, and
recovery still requires the predecessor's own run and finalization to be
settled and saved. A workspace is the unit that owns the files, so this
does not let one workspace read another workspace's files.
- I could not read one predecessor's finalization row in the real
incident. If that row is not in a settled phase, recovery still refuses
with `allocation_ownership_ambiguous`, which is the old behavior.
- `snapshot_verification_failed` is labeled `service_required` at its 19
call sites, so it now shows the refusal text. If one of those sites is
really transient, the text says "Retry runs the check again", which is
still true.
- A second cause kept the same task stuck: a remote snapshot save that a
restart interrupted stayed `running`, and that blocks the
retained-source fallback. That fix comes in a separate pull request.
- No schema or API change. The new wait reason is a new string in an
existing field. The UI shows the message as it receives it.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only browser diagnostics).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6958cb112c
commit
87e755ce8e
10 files changed
+142
-24
No files matched your search
@@ -115,7 +115,8 @@ export function useWorkspaceRecoveryFixture(executionTargets: Map<string, Adapte
|
||||
});
|
||||
}
|
||||
|
||||
export async function seed(mode = "native", nativeDescriptor = true, resourceDisposition: "destroy" | "keep_running" = "destroy", reusable = false, ephemeral = false) {
|
||||
export async function seed(mode = "native", nativeDescriptor = true, resourceDisposition: "destroy" | "keep_running" = "destroy", reusable = false, ephemeral = false,
|
||||
descriptorWorkspaceId?: string) {
|
||||
const companyId = randomUUID(), issueId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), actionId: string = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Workspace recovery", defaultResponsibleUserId: "board", issuePrefix: `R${companyId.slice(0, 6)}` });
|
||||
fixtureCompanyIds.add(companyId);
|
||||
@@ -147,7 +148,7 @@ export async function seed(mode = "native", nativeDescriptor = true, resourceDis
|
||||
}) } };
|
||||
targets.set(lease.id, target);
|
||||
if (nativeDescriptor) {
|
||||
const prepared = await prepareNativeWorkspaceSync({ db, companyId, workspaceId: issueId, workspaceLocalDir: host,
|
||||
const prepared = await prepareNativeWorkspaceSync({ db, companyId, workspaceId: descriptorWorkspaceId ?? issueId, workspaceLocalDir: host,
|
||||
runId, lease: lease as EnvironmentLease, target, resourceDisposition });
|
||||
expect(prepared).not.toBeNull();
|
||||
} else {
|
||||
@@ -395,8 +396,20 @@ export async function stoppedWorkspaceWorker(f: { runId: string }) {
|
||||
return { proof, stopProof, owner, deploymentWorker };
|
||||
}
|
||||
|
||||
export async function warmHandoffSource(hops = 1, beforeStop?: (lease: typeof environmentLeases.$inferSelect) => Promise<void>) {
|
||||
const prior = await seed("native", true, "keep_running", true), source = await run(prior.runId);
|
||||
export async function warmHandoffSource(hops = 1, beforeStop?: (lease: typeof environmentLeases.$inferSelect) => Promise<void>,
|
||||
options: { firstOwnerInSharedWorkspace?: boolean } = {}) {
|
||||
const sharedWorkspaceId = options.firstOwnerInSharedWorkspace ? randomUUID() : undefined;
|
||||
const prior = await seed("native", true, "keep_running", true, false, sharedWorkspaceId), source = await run(prior.runId);
|
||||
const descriptorWorkspaceId = sharedWorkspaceId ?? prior.issueId;
|
||||
let executionWorkspaceId: string | null = null, taskId: string = prior.issueId;
|
||||
if (sharedWorkspaceId) {
|
||||
const [workspace] = await db.insert(executionWorkspaces).values({ id: sharedWorkspaceId, companyId: prior.companyId,
|
||||
mode: "isolated_workspace", strategyType: "test", name: "Shared task workspace" }).returning();
|
||||
const [later] = await db.insert(issues).values({ companyId: prior.companyId, assigneeAgentId: source.agentId,
|
||||
status: "blocked", title: "Later task in the same workspace" }).returning();
|
||||
executionWorkspaceId = workspace.id; taskId = later.id;
|
||||
await db.update(environmentLeases).set({ executionWorkspaceId }).where(eq(environmentLeases.id, prior.lease.id));
|
||||
}
|
||||
let previousTarget = targets.get(prior.lease.id)!, previousLeaseId = prior.lease.id;
|
||||
let previousRunId: string = prior.runId;
|
||||
let lease = prior.lease, current = source;
|
||||
@@ -409,17 +422,17 @@ export async function warmHandoffSource(hops = 1, beforeStop?: (lease: typeof en
|
||||
await environmentService(db).releaseLease(previousLeaseId, "released", { cleanupStatus: "success" });
|
||||
const [released] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, previousLeaseId));
|
||||
[current] = await db.insert(heartbeatRuns).values({ companyId: prior.companyId, agentId: source.agentId,
|
||||
issueId: prior.issueId, nativeIssueId: prior.issueId, runtimeMode: "native", status: "cancelled", finishedAt: new Date(),
|
||||
contextSnapshot: { issueId: prior.issueId } }).returning();
|
||||
await db.insert(nativeRunFinalizations).values({ companyId: prior.companyId, issueId: prior.issueId, runId: current.id, phase: "terminal_failure" });
|
||||
issueId: taskId, nativeIssueId: taskId, runtimeMode: "native", status: "cancelled", finishedAt: new Date(),
|
||||
contextSnapshot: { issueId: taskId } }).returning();
|
||||
await db.insert(nativeRunFinalizations).values({ companyId: prior.companyId, issueId: taskId, runId: current.id, phase: "terminal_failure" });
|
||||
const acquired = await environmentService(db).acquireLease({ companyId: prior.companyId, environmentId: prior.lease.environmentId!,
|
||||
issueId: prior.issueId, heartbeatRunId: current.id, provider: "daytona", providerLeaseId: prior.providerLeaseId,
|
||||
leasePolicy: "reuse_by_environment", replacesReusableLeaseId: previousLeaseId,
|
||||
issueId: taskId, heartbeatRunId: current.id, provider: "daytona", providerLeaseId: prior.providerLeaseId,
|
||||
executionWorkspaceId, leasePolicy: "reuse_by_environment", replacesReusableLeaseId: previousLeaseId,
|
||||
metadata: { ...released.metadata, sandboxLeaseAcquisition: { outcome: "resumed" } } });
|
||||
[lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, acquired.id));
|
||||
const target = { ...previousTarget, leaseId: lease.id, sandboxLeaseAcquisition: { outcome: "resumed" as const, providerLeaseId: prior.providerLeaseId } };
|
||||
targets.set(lease.id, target);
|
||||
expect(await prepareNativeWorkspaceSync({ db, companyId: prior.companyId, workspaceId: prior.issueId,
|
||||
expect(await prepareNativeWorkspaceSync({ db, companyId: prior.companyId, workspaceId: descriptorWorkspaceId,
|
||||
workspaceLocalDir: prior.host, runId: current.id, lease: lease as EnvironmentLease, target, resourceDisposition: "keep_running" })).not.toBeNull();
|
||||
leaseIds.push(lease.id); previousRunId = current.id; previousLeaseId = lease.id; previousTarget = target;
|
||||
}
|
||||
@@ -434,8 +447,8 @@ export async function warmHandoffSource(hops = 1, beforeStop?: (lease: typeof en
|
||||
expect(stopping).not.toBeNull();
|
||||
expect(await settleStopOnlyCleanup(db, stopping!, { attemptId: String(stopping!.metadata!.pendingCleanupAttemptId),
|
||||
receipt: { providerLeaseId: prior.providerLeaseId, state: "stopped" } })).not.toBeNull();
|
||||
return { ...prior, priorRunId: prior.runId, priorLeaseId: prior.lease.id, leaseIds, runId: current.id, lease, actionId: action.id,
|
||||
input: { ...prior.input, runId: current.id, actionId: action.id } };
|
||||
return { ...prior, issueId: taskId, priorRunId: prior.runId, priorLeaseId: prior.lease.id, firstOwnerIssueId: prior.issueId, executionWorkspaceId,
|
||||
leaseIds, runId: current.id, lease, actionId: action.id, input: { ...prior.input, issueId: taskId, runId: current.id, actionId: action.id } };
|
||||
}
|
||||
|
||||
export async function makeRetryDue(actionId: string) {
|
||||
|
||||
@@ -61,7 +61,8 @@ export function retiredReusableLeasePredecessors(current: Lease, history: Lease[
|
||||
(outgoing as Record<string, unknown>)[key] !== value) ||
|
||||
!previous.environmentId || !previous.issueId || typeof previous.metadata?.agentId !== "string" ||
|
||||
previous.companyId !== current.companyId || previous.environmentId !== current.environmentId ||
|
||||
previous.executionWorkspaceId !== current.executionWorkspaceId || previous.issueId !== current.issueId ||
|
||||
previous.executionWorkspaceId !== current.executionWorkspaceId ||
|
||||
(current.executionWorkspaceId === null && previous.issueId !== current.issueId) ||
|
||||
previous.metadata.agentId !== current.metadata?.agentId ||
|
||||
previous.provider !== current.provider || previous.providerLeaseId !== current.providerLeaseId ||
|
||||
previous.releasedAt?.getTime() !== successor.acquiredAt.getTime() ||
|
||||
|
||||
@@ -412,6 +412,7 @@ export async function admitExplicitNativeContinuation(input: {
|
||||
ownerHold: sql<boolean>`${issueRecoveryActions.evidence}#>>'{workspaceOwnerRecovery,state}' = 'operator_required'
|
||||
and ${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`,
|
||||
recoveryState: sql<string | null>`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}'`,
|
||||
retryClass: sql<string | null>`${issueRecoveryActions.evidence}#>>'{systemRecovery,failure,retryClass}'`,
|
||||
}).from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
||||
inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
||||
@@ -423,7 +424,9 @@ export async function admitExplicitNativeContinuation(input: {
|
||||
// starts one new recovery generation (task-retry), so name that action
|
||||
// instead of an endless finalizing wait (RC-50b, s14).
|
||||
if (live.length && live.every(row => row.recoveryState === "exhausted"))
|
||||
return blocked("saved_work_recovery_failed", "Saving the agent's files failed after automatic retries. Retry to try again.");
|
||||
return live.some(row => row.retryClass === "service_required")
|
||||
? blocked("saved_work_recovery_refused", "Paperclip refused to recover the saved files because a safety check on them cannot pass. Retry runs the same check again.")
|
||||
: blocked("saved_work_recovery_failed", "Saving the agent's files failed. Retry to try again.");
|
||||
return blocked("finalizing_saved_result", "Paperclip is finishing the saved result.");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { createSystemRecovery, failSystemRecovery, readSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "./system-recovery-state.js";
|
||||
import { createSystemRecovery, failSystemRecovery, isDeterministicRecoveryRefusal, readSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "./system-recovery-state.js";
|
||||
import { exhaustedSystemRecoveryAction } from "./system-recovery.js";
|
||||
import { workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js";
|
||||
|
||||
@@ -59,17 +59,36 @@ describe("durable system recovery budget", () => {
|
||||
message: "Automatic recovery of saved work stopped after 1 attempts (last failure: allocation_ownership_ambiguous). "
|
||||
+ "Retry recovery to export and merge from the retained sandbox again; no provider turn is submitted. "
|
||||
+ "If Retry keeps failing, inspect the run and its environment." } }, runId);
|
||||
expect(display?.message).toContain("Retry to try again");
|
||||
expect(display?.message).toContain("Paperclip refused to recover the saved files");
|
||||
expect(display?.message).not.toMatch(manual);
|
||||
});
|
||||
|
||||
it("states a deterministic refusal plainly and never claims retries that did not happen", () => {
|
||||
const refused = failSystemRecovery(reserveSystemRecoveryAttempt(createSystemRecovery(randomUUID()), 0)!,
|
||||
new SystemRecoveryFailure("allocation_ownership_ambiguous", "service_required"), { now: 0 });
|
||||
expect(refused).toMatchObject({ state: "exhausted", attempts: 1, retriesUsed: 0 });
|
||||
expect(isDeterministicRecoveryRefusal(refused)).toBe(true);
|
||||
const text = exhaustedSystemRecoveryAction(refused);
|
||||
expect(text).toContain("Paperclip refused to recover the saved files");
|
||||
expect(text).toContain("Paperclip cannot prove which runs used the sandbox that holds the saved files (allocation_ownership_ambiguous)");
|
||||
expect(text).toContain("Retry runs the check again and succeeds only after Paperclip is updated");
|
||||
expect(text).not.toMatch(/after automatic retries|stopped after|Retry to try again/);
|
||||
|
||||
const once = failSystemRecovery(reserveSystemRecoveryAttempt(createSystemRecovery(randomUUID()), 0)!,
|
||||
new SystemRecoveryFailure("historical_source_missing", "use_alternative_strategy"), { now: 0 });
|
||||
expect(isDeterministicRecoveryRefusal(once)).toBe(false);
|
||||
expect(exhaustedSystemRecoveryAction(once)).toBe("Automatic recovery of saved work failed (last failure: historical_source_missing). "
|
||||
+ "Retry to try again; the agent will not repeat its work.");
|
||||
expect(exhaustedSystemRecoveryAction({ ...once, attempts: 6 })).toContain("stopped after 6 attempts");
|
||||
});
|
||||
|
||||
it("names the last failure and its cause in the board handoff", () => {
|
||||
const state = reserveSystemRecoveryAttempt(createSystemRecovery(randomUUID()), 0)!;
|
||||
const withCause = failSystemRecovery(state, new SystemRecoveryFailure("provider_unavailable", "retry_transient"),
|
||||
{ now: 0, cause: { label: "sandbox_not_running", code: null, status: null } });
|
||||
expect(exhaustedSystemRecoveryAction(withCause)).toContain("(last failure: provider_unavailable, cause: sandbox_not_running)");
|
||||
const withoutCause = failSystemRecovery(state, new SystemRecoveryFailure("archive_unsafe", "service_required"), { now: 0 });
|
||||
expect(exhaustedSystemRecoveryAction(withoutCause)).toContain("(last failure: archive_unsafe)");
|
||||
expect(exhaustedSystemRecoveryAction(withoutCause)).toContain("(archive_unsafe)");
|
||||
});
|
||||
it("reconciles a recorded attempt after restart without spending another retry", () => {
|
||||
const running = reserveSystemRecoveryAttempt(createSystemRecovery(randomUUID()), 0)!;
|
||||
|
||||
@@ -135,7 +135,26 @@ export function exhaustedSystemRecoveryAction(state: SystemRecoveryState): strin
|
||||
return "Recovery of saved work is waiting for the previous server's file copy to stop. "
|
||||
+ "Recovery resumes once Paperclip can confirm that it stopped. No provider turn is submitted.";
|
||||
const cause = state.failure?.cause ? `, cause: ${state.failure.cause.label}` : "";
|
||||
if (isDeterministicRecoveryRefusal(state)) {
|
||||
const code = state.failure!.code;
|
||||
return `Paperclip refused to recover the saved files: ${recoveryRefusalReasons[code] ?? "a safety check on the saved files did not pass"} `
|
||||
+ `(${code}${cause}). Automatic recovery does not repeat this check, because the same saved state gives the same result. `
|
||||
+ "Retry runs the check again and succeeds only after Paperclip is updated. The agent's result is saved, and the agent will not repeat its work.";
|
||||
}
|
||||
const reason = state.failure?.code ? ` (last failure: ${state.failure.code}${cause})` : "";
|
||||
return `Automatic recovery of saved work stopped after ${state.attempts} attempts${reason}. `
|
||||
const outcome = state.attempts > 1 ? `stopped after ${state.attempts} attempts` : "failed";
|
||||
return `Automatic recovery of saved work ${outcome}${reason}. `
|
||||
+ "Retry to try again; the agent will not repeat its work.";
|
||||
}
|
||||
|
||||
export function isDeterministicRecoveryRefusal(state: Pick<SystemRecoveryState, "state" | "failure"> | null | undefined) {
|
||||
return state?.state === "exhausted" && state.failure?.retryClass === "service_required";
|
||||
}
|
||||
|
||||
const recoveryRefusalReasons: Partial<Record<RecoveryFailureCode, string>> = {
|
||||
allocation_ownership_ambiguous: "Paperclip cannot prove which runs used the sandbox that holds the saved files",
|
||||
source_binding_invalid: "the recorded source of the saved files does not match the run that saved them",
|
||||
snapshot_verification_failed: "the saved snapshot of the files did not pass verification",
|
||||
archive_unsafe: "the saved file archive contains entries that are not safe to apply",
|
||||
source_unavailable: "the saved files are no longer available where the run recorded them",
|
||||
};
|
||||
@@ -123,7 +123,9 @@ const retryWaitMessages: Record<string, string> = {
|
||||
finalizing_saved_result: "Paperclip is finishing the saved result before Retry can continue.",
|
||||
// Another click cannot release a copyback owner that stopped without reporting.
|
||||
// Retry starts a new recovery generation; the agent does not repeat its work.
|
||||
saved_work_recovery_failed: "Saving the agent's files failed after automatic retries. Retry to try again; the agent will not repeat its work.",
|
||||
saved_work_recovery_failed: "Saving the agent's files failed. Retry to try again; the agent will not repeat its work.",
|
||||
saved_work_recovery_refused: "Paperclip refused to recover the saved files because a safety check on them cannot pass. "
|
||||
+ "Retry runs the same check again and succeeds only after Paperclip is updated. The agent's result is saved, and the agent will not repeat its work.",
|
||||
workspace_owner_stop_required: "Retry is waiting until Paperclip confirms that the previous server stopped saving files. The agent's result and your Retry are saved.",
|
||||
cleanup_quarantined: "Retry cannot continue this attempt. Send a new message after the previous execution has stopped.",
|
||||
cancelled_by_operator: "This attempt was stopped. Send a new message to continue from saved work.",
|
||||
|
||||
@@ -416,6 +416,36 @@ describe("automatic retained workspace recovery", () => {
|
||||
.toEqual([{ id: settled.runId, previousRunId: f.runId }]);
|
||||
}, 60_000);
|
||||
|
||||
it("names a deterministic saved-files refusal behind Retry when no fallback can continue, and another Retry shows it again", async () => {
|
||||
const f = await acceptedRemoteExport();
|
||||
const issue = (await db.select().from(issues).where(eq(issues.id, f.issueId)))[0];
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
const [other] = await db.insert(issues).values({ companyId: f.companyId, title: "Earlier task" }).returning();
|
||||
await db.insert(environmentLeases).values({ companyId: f.companyId, environmentId: lease.environmentId, issueId: other.id,
|
||||
provider: "daytona", providerLeaseId: f.providerLeaseId, status: "expired", leasePolicy: "reuse_by_environment",
|
||||
cleanupStatus: "success", releasedAt: new Date(), metadata: { remoteCwd: path.dirname(f.remote) } });
|
||||
await db.insert(workspaceOperations).values({ companyId: f.companyId, executionWorkspaceId: f.workspaceId, heartbeatRunId: f.runId,
|
||||
issueId: f.issueId, phase: "remote_workspace_snapshot", status: "running", metadata: { boundary: "final" } });
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.source.agentId, status: "running" });
|
||||
const app = boardRecoveryApp();
|
||||
for (const click of [1, 2]) {
|
||||
const retry = (await request(app).post(`/api/issues/${f.issueId}/retry`)
|
||||
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body;
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "exhausted", attempts: 1,
|
||||
failure: { code: "allocation_ownership_ambiguous", retryClass: "service_required" } });
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, retry.requestId!);
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!)))[0], `click ${click}`)
|
||||
.toMatchObject({ runId: null, payload: { executionWait: { reason: "saved_work_recovery_refused" } } });
|
||||
const shown = await taskInterruptionPresentation(db, issue);
|
||||
expect(shown).toMatchObject({ state: "failed", canRetry: true, retryWait: { reason: "saved_work_recovery_refused" } });
|
||||
expect(shown?.retryWait?.message).toContain("Paperclip refused to recover the saved files");
|
||||
expect(shown?.retryWait?.message).not.toMatch(/after automatic retries/);
|
||||
expect((await getExecutionBlocker(db, f.companyId, f.issueId))?.nextAction)
|
||||
.toContain("Paperclip cannot prove which runs used the sandbox that holds the saved files");
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
it("preserves an explicitly requested superseded remote export without applying its result over newer work", async () => {
|
||||
const f = await acceptedRemoteExport();
|
||||
const [later] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.source.agentId,
|
||||
|
||||
@@ -508,7 +508,7 @@ describe("automatic retained workspace recovery", () => {
|
||||
const stable = (row: typeof after) => ({ ...row, updatedAt: null, ownerType: null, nextAction: null,
|
||||
evidence: { ...row.evidence, workspaceRecovery: { ...(row.evidence.workspaceRecovery as object), message: null } } });
|
||||
if (spent) {
|
||||
expect(after).toMatchObject({ ownerType: "board", nextAction: expect.stringMatching(/Retry to try again/) });
|
||||
expect(after).toMatchObject({ ownerType: "board", nextAction: expect.stringMatching(/Paperclip refused to recover the saved files: .*Retry runs the check again/) });
|
||||
expect(stable(after)).toEqual(stable(before));
|
||||
} else expect(after).toEqual(before);
|
||||
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].leasePolicy).toBe("reuse_by_environment");
|
||||
|
||||
@@ -190,9 +190,8 @@ export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRec
|
||||
const priors = predecessorRuns.length ? await db.select().from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, input.companyId), inArray(heartbeatRuns.id, predecessorRuns))) : [];
|
||||
const priorOwners = predecessorRuns.length ? await db.select({ runId: nativeRunFinalizations.runId,
|
||||
phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner }).from(nativeRunFinalizations).where(and(
|
||||
eq(nativeRunFinalizations.companyId, input.companyId), inArray(nativeRunFinalizations.runId, predecessorRuns),
|
||||
eq(nativeRunFinalizations.issueId, input.issueId))) : [];
|
||||
issueId: nativeRunFinalizations.issueId, phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner }).from(nativeRunFinalizations).where(and(
|
||||
eq(nativeRunFinalizations.companyId, input.companyId), inArray(nativeRunFinalizations.runId, predecessorRuns))) : [];
|
||||
for (const predecessor of predecessors) {
|
||||
const prior = priors.find(item => item.id === predecessor.heartbeatRunId);
|
||||
const priorReference = readNativeWorkspaceSyncReference(prior?.runnerProfileJson?.nativeWorkspaceSync);
|
||||
@@ -203,7 +202,8 @@ export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRec
|
||||
: priorRemote?.state === "saved" && Boolean(priorRemote.snapshotId) && Boolean(remoteDescriptor)
|
||||
&& priorRemote.epoch < remoteDescriptor!.epoch;
|
||||
const priorOwner = priorOwners.find(item => item.runId === predecessor.heartbeatRunId);
|
||||
if (!prior || prior.runtimeMode !== "native" || prior.agentId !== run.agentId || prior.nativeIssueId !== input.issueId ||
|
||||
if (!prior || prior.runtimeMode !== "native" || prior.agentId !== run.agentId ||
|
||||
!predecessor.issueId || prior.nativeIssueId !== predecessor.issueId || priorOwner?.issueId !== predecessor.issueId ||
|
||||
!terminalWorkspaceRunStates.includes(prior.status) || adapterExecutionControls.has(prior.id) ||
|
||||
(prior.controllerLeaseExpiresAt && prior.controllerLeaseExpiresAt > new Date()) ||
|
||||
!priorOwner || priorOwner.leaseOwner || !["committed", "terminal_failure", "applied"].includes(priorOwner.phase) ||
|
||||
|
||||
@@ -186,6 +186,37 @@ describe("automatic retained workspace recovery", () => {
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it.each([1, 2])("recovers saved work after %s warm handoffs that began with another task in the same workspace", async hops => {
|
||||
const f = await warmHandoffSource(hops, undefined, { firstOwnerInSharedWorkspace: true });
|
||||
const [first] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.priorLeaseId));
|
||||
expect(first).toMatchObject({ issueId: f.firstOwnerIssueId, executionWorkspaceId: f.executionWorkspaceId, status: "expired" });
|
||||
expect(new Set([f.firstOwnerIssueId, f.issueId, f.executionWorkspaceId]).size).toBe(3);
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const state = readSystemRecovery((await action(f.actionId)).evidence)!;
|
||||
expect(state.failure?.code ?? null).toBeNull();
|
||||
expect(state.state).toBe("complete");
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
|
||||
expect(await readFile(path.join(f.host, "after-warm-stop.txt"), "utf8")).toBe("stopped turn work\n");
|
||||
}, 30_000);
|
||||
|
||||
it.each(["lease_task", "unshared_workspace", "other_workspace"])(
|
||||
"keeps a cross-task predecessor fenced unless its own records prove the shared workspace: %s", async fault => {
|
||||
const f = await warmHandoffSource(1, undefined, { firstOwnerInSharedWorkspace: true });
|
||||
const [other] = await db.insert(issues).values({ companyId: f.companyId, title: "Unrelated task" }).returning();
|
||||
if (fault === "lease_task") await db.update(environmentLeases).set({ issueId: other.id }).where(eq(environmentLeases.id, f.priorLeaseId));
|
||||
if (fault === "unshared_workspace") await db.update(environmentLeases).set({ executionWorkspaceId: null })
|
||||
.where(eq(environmentLeases.id, f.priorLeaseId));
|
||||
if (fault === "other_workspace") {
|
||||
const [otherWorkspace] = await db.insert(executionWorkspaces).values({ companyId: f.companyId, mode: "isolated_workspace",
|
||||
strategyType: "test", name: "Other workspace" }).returning();
|
||||
await db.update(environmentLeases).set({ issueId: f.issueId, executionWorkspaceId: otherWorkspace.id })
|
||||
.where(eq(environmentLeases.id, f.priorLeaseId));
|
||||
}
|
||||
expect(await inspectWorkspaceSource(db, { companyId: f.companyId, issueId: f.issueId, runId: f.runId, actionId: f.actionId }))
|
||||
.toMatchObject({ status: "blocked", failureCode: "allocation_ownership_ambiguous" });
|
||||
await expect(readFile(path.join(f.host, "after-warm-stop.txt"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
}, 30_000);
|
||||
|
||||
it.each(["active", "active_run", "active_control", "foreign_company", "environment", "workspace", "task", "agent", "unfinalized", "owner", "missing_proof", "conflicting_proof", "dangling_proof", "extra_history"])(
|
||||
"keeps ambiguous warm source history fenced: %s", async fault => {
|
||||
const f = await warmHandoffSource();
|
||||
|
||||
Reference in new issue
Block a user