fix(native): apply a saved accepted result on Retry instead of a new turn (#20819)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can finish with an accepted result,
and then fail to save its workspace. Saved-work recovery saves the files
again from the retained sandbox, with no provider turn.
> - #20758 made a save that finds the sandbox disk full stop at once as
`sandbox_disk_full` and hand the task to the board with a Retry.
> - When the user frees space and presses Retry, the save now succeeds.
But the Retry then started a new agent turn (`strategy: continue`). That
turn repeats work that already finished and spends provider budget
again. The accepted result was not applied (`replacementDenied:
explicit_user_continuation`).
> - The decision for this case: when a durable record proves that the
run's result was already accepted, Retry applies that saved result and
starts no new turn.
> - This pull request makes Retry admission wait for the run's
saved-work holds and then applies the accepted result through the
existing preserved-result finalization.
> - The benefit is that a Retry after a failed save finishes the task
from the work that was already done, with no repeated provider work or
spend.

## Linked Issues or Issue Description

Refs #20758 (full-disk save rests as `sandbox_disk_full` with a Retry),
Refs #20810 (chaos fault DF1 for a full disk on the final workspace
save)

**What happened?**
A native run had an accepted, succeeded result. Its workspace save
failed because the sandbox disk was full, and the task rested at
`sandbox_disk_full` with a Retry. The user freed space and pressed
Retry. The save succeeded, and then Retry admission started a new run
with `strategy: continue`. The accepted result stayed unapplied (the
finalization coordinator stayed in `terminal_failure`).

Two paths caused this:
1. Retry admission let the Retry through as soon as the saved files were
verified. That can happen before preserved-result finalization runs: the
recovery worker dispatches the Retry right after one hold settles, while
the hold that Retry added for the same run is still scheduled. On master
this is deterministic in that order, and a race otherwise.
2. Preserved-result finalization reads the newest saved-work hold of the
run. While that hold was still scheduled, it declined, and nothing held
the Retry back.

**Expected behavior**
After the save succeeds, and a durable record proves that the run's
result was already accepted, Retry applies that result. The run and task
finish from it. No new `turn.start` and no new provider spend. If the
save fails again with a full disk, the task keeps the
`sandbox_disk_full` rest and a working Retry. If there is no proven
accepted result, Retry keeps the previous behaviour and starts one run.

**Steps to reproduce**
1. Give a native run an accepted, succeeded result whose copy-back
failed (`native_workspace_sync_out_retry_exhausted`).
2. Make the saved-work recovery save fail with
`remote_workspace_sandbox_disk_full`. The hold goes to the board.
3. Free space. Press Retry. Let recovery save the files, then dispatch
the Retry before preserved-result finalization runs.
4. On master, a new run starts with `previousRunId` set to the source
run, and the accepted result is not applied.

## What Changed

- `preserved-result-finalization.ts`: the eligibility checks move into
one read (`readPreservedResult`). They are unchanged: an accepted,
succeeded result of this run, a completed saved-work recovery with a
verified receipt, no Stop, no newer run, no current execution, and no
applied generation. A new read-only
`canResumePreservedResultFinalization` uses the same read without locks.
`resumePreservedResultFinalization` keeps the finalizer's lock order
(coordinator first) and takes the issue and run rows with NOWAIT, so it
cannot deadlock with the finalizer, owner-stop confirmation or Retry
dispatch; a busy row is retried briefly, and the last attempt waits for
its locks. It also writes an activity entry,
`issue.saved_result_finalization_resumed`, when it applies a result.
- `explicit-native-continuation.ts`: Retry admission holds an accepted
Retry (`evaluate`) of a run whose saved files are verified and whose
accepted result is still held, with the wait `finalizing_saved_result`.
It holds while a saved-work hold of that run is still `scheduled`,
`running` or `awaiting_evidence` and the result is otherwise applicable
(no Stop, newer run, other owner or current execution), or once
`canResumePreservedResultFinalization` is true. In every other case
admission is unchanged.
- `heartbeat/run-control.ts` (`resumeTaskRetry`): after dispatch, if
admission held the Retry with `finalizing_saved_result`, it calls
`resumePreservedResultFinalization`. That coalesces the Retry into the
source run with `strategy: finalize_result` and re-arms finalization.
This is one key read after admission; the resumption runs only for a
held Retry. The recovery worker already calls the same function after
each saved hold, so either path finishes the work.
- Tests: `workspace-recovery-disk-full.test.ts` now proves the new Retry
behaviour (red on master), a dispatch that lands while another hold of
the run is still saving (red on master), a Retry while the disk is still
full, and three negative cases that keep one new run.
`workspace-recovery-preparation-gates.test.ts` (s12 and RC-50b) and
`explicit-native-continuation.test.ts` (copy-back + `task_retry`)
encoded the old "new run" result for the same situation. They now expect
the saved result to be applied. A fixture helper `recoverRunHolds` runs
the recovery steps for every hold of a run, as the worker does.
- `ui/src/components/WorkspaceExportRecovery.tsx`: at the
`sandbox_disk_full` rest, the retained-file panel no longer shows its
own "Recover saved files" button and no longer hides the task's Retry.
Before, the API said `canRetry: true` and the note said "Retry saves the
files again", but the board showed only "Recover saved files" (seen in
the DF1 chaos wave). The panel now says to free space and use Retry. A
new UI test covers it. `pnpm check:token-gates` is clean.
- `doc/execution-semantics.md`: documents the Retry behaviour and its
conditions.

## Verification

Red/green (`cd server && npx vitest run
src/services/workspace-recovery-disk-full.test.ts`):

- On master source with the new tests: 3 failed, 3 passed (the 3
negative cases pass on master, as they should). Each failure shows a
second heartbeat run (the new agent turn): `expected [ …(2) ] to deeply
equal [ Array(1) ]`.
- On this branch: 7 passed. A fourth negative case (an operator Stop
while a hold is still settling keeps one new run) was added after
review; it also passes on master.
- Guard removal 1: with the `canResumePreservedResultFinalization` check
replaced by `true`, the negative test "an operator Stop recorded on the
run" fails (`expected [] to have a length of 1 but got +0`: the Retry
waits forever instead of starting its one run).
- Guard removal 2: with the "hold still saving" check removed, the test
"holds a Retry dispatched while another saved-work hold of the run is
still settling" fails (a second run starts).

Related suites pass on this branch: `workspace-recovery-disk-full`,
`workspace-recovery-preparation-gates`, `workspace-recovery-admission`,
`native-exhausted-finalization-retry`, `explicit-native-continuation`
(641 tests).

On the first revision, `task-retry`,
`workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`,
`preserved-native-response-wait.integration`, `deferred-execution-wait`
and `deferred-execution-receipt-work` passed (317 tests). Other retry
and recovery suites were not run locally; required CI `general_tests`
covers them. After the review fixes, the five related suites above
passed again (642 tests).

`npx tsc --noEmit -p server/tsconfig.json` is clean.

`pnpm verify --explain` selects a broad plan (exit 2) because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

## Risks

- Behaviour change: an accepted Retry of a terminal run with an
accepted, succeeded result and verified saved files now applies that
result and starts no run. Before, it started one new run. This is the
intended change. It also covers the s12 and RC-50b recovery-handoff
cases, which used to start a new run.
- A held Retry waits while any saved-work hold of the run is still
saving. The recovery worker drives those holds to `complete` or
`exhausted`. When a hold ends `exhausted` or the records do not prove
the result, the wait ends and the Retry starts one run, as before. A
hold that stays in `awaiting_evidence` keeps the Retry waiting with
"Paperclip is finishing the saved result."
- Upgrade-safe: no persisted shape changes. The checks read existing
records (`native_run_results`, `native_run_finalizations`, the recovery
action evidence, workspace receipts). A record written by an older
server that lacks any required fact keeps the old behaviour. An older
server that reads the new state sees a coalesced Retry and a re-armed
finalization, which it already handles.
- No new work before `turn.start` on the normal path. Admission adds the
eligibility read only when it has already proven an accepted result and
verified saved files for a Retry of that run. `resumeTaskRetry` adds one
key read after admission.
- Lock order: `resumePreservedResultFinalization` keeps the finalizer's
order (coordinator, then issue, then run). Because it can now run beside
Retry dispatch (which locks the issue first), it takes the issue and run
rows with NOWAIT. A busy row ends the attempt, and it retries up to 9
times at 100 ms. The 10th attempt waits for its locks, as this
resumption always did, so a recovered result is never left unapplied.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed 2026-11-11 10:03:53 -06:00
1 parent 369e238295
commit 889b33da72
10 files changed
+398 -107

No files matched your search

+12 -1
View File
@@ -4695,7 +4695,18 @@ generation ends at once as
`user_action_required`) and goes to the board. It is not retried on the automatic
schedule, and it never falls back to the last save, so the run's unsaved files stay
in the retained sandbox and the workspace keeps its previous saved version. The
board's Retry saves again; once the save succeeds, the accepted Retry starts one run.
board's Retry saves again. Once the save succeeds, the accepted Retry applies the
run's saved accepted result and finishes the run and task from it, with no new
agent turn and no provider spend. Retry admission waits (`finalizing_saved_result`)
while any saved-work hold of that run is still saving, then preserved-result
finalization coalesces the Retry into the source run (`finalize_result`, activity
`issue.saved_result_finalization_resumed`). This applies to any accepted Retry
(`evaluate`) of a terminal run whose coordinator still holds an accepted,
succeeded result, whose saved-work recovery completed with a verified receipt,
and which has no recorded Stop, newer run, current execution or other owner. A
record that lacks any of these, including one written by an older server, keeps
the previous behaviour: the Retry starts one run. If the save still finds the
disk full, the hold rests again as `sandbox_disk_full` with its Retry.
A reclaim whose outcome is unknown keeps the scheduled retry, because its journal
resumes on the next attempt and can still free space.
@@ -15,6 +15,7 @@ import { environmentService } from "../environments.js";
import { classifyEnvironmentCapabilities, environmentRuntimeService } from "../environment-runtime.js";
import { readSystemRecovery, createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "../system-recovery-state.js";
import { adoptSystemRecovery } from "../system-recovery.js";
import { recoverRetainedWorkspace } from "../workspace-recovery.js";
import { requestTaskRetry } from "../task-retry.js";
import { execFile } from "node:child_process";
import { randomUUID } from "node:crypto";
@@ -457,3 +458,12 @@ export async function makeRetryDue(actionId: string) {
await db.update(issueRecoveryActions).set({ evidence: { ...current.evidence,
systemRecovery: { ...state, nextAttemptAt: 0 } } }).where(eq(issueRecoveryActions.id, actionId));
}
/** The recovery worker's steps for every saved-work hold of one run (Retry can
* add one beside the board hold), without its post-recovery dispatch. */
export async function recoverRunHolds(f: Parameters<typeof recoverRetainedWorkspace>[0]) {
const holds = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, f.companyId), sql`${issueRecoveryActions.evidence}->>'runId' = ${f.runId}`,
sql`${issueRecoveryActions.evidence} ? 'systemRecovery'`)).orderBy(issueRecoveryActions.createdAt);
for (const hold of holds) await recoverRetainedWorkspace({ ...f, actionId: hold.id });
}
@@ -26,6 +26,7 @@ import {
import { startEmbeddedPostgresTestDatabase, getEmbeddedPostgresTestSupport } from "../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
import { admitExplicitNativeContinuation } from "./explicit-native-continuation.js";
import { resumePreservedResultFinalization } from "./preserved-result-finalization.js";
import { adapterExecutionControls, createAdapterExecutionControl } from "./adapter-execution-control.js";
import { CONVERSATION_CONTINUATION_POLICY } from "./conversation-continuation.js";
import { buildExecutionContinuation } from "./execution-continuation.js";
@@ -2806,7 +2807,28 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
.toMatchObject({ status: "deferred_issue_execution", runId: null });
});
it.each(["message", "queued_message", "task_retry"] as const)("continues a terminal retained result after verified copy-back: %s", async origin => {
// A Retry asks for the accepted result, which now has its saved files. It is
// applied by preserved-result finalization, never replaced by a new turn.
it("holds a Retry of a terminal retained result after verified copy-back until its accepted result is applied", async () => {
const f = await recoveredCopybackFixture("task_retry");
const [before] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId));
const waits: string[] = [];
expect(await db.transaction(async tx => {
await tx.select().from(issues).where(eq(issues.id, f.issueId)).for("update");
return admitExplicitNativeContinuation({ ...f, db: tx as unknown as typeof db, onBlocked: reason => { waits.push(reason); } });
})).toBeNull();
expect(waits).toEqual(["finalizing_saved_result"]);
expect(await resumePreservedResultFinalization(db, f.companyId, f.sourceRunId)).toBe(true);
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.requestId)))[0])
.toMatchObject({ status: "coalesced", runId: f.sourceRunId, payload: { taskRetry: { strategy: "finalize_result" } } });
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)))[0])
.toMatchObject({ phase: "workspace_finalizing", resultId: f.resultId });
expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0]).toEqual(before);
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
sql`${heartbeatRuns.id} <> ${f.sourceRunId}`))).toEqual([]);
});
it.each(["message", "queued_message"] as const)("continues a terminal retained result after verified copy-back: %s", async origin => {
const f = await recoveredCopybackFixture(origin);
const [before] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId));
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
@@ -35,6 +35,7 @@ import { activeIssueInteractionCondition } from "./issue-question-context.js";
import { historicalAdapterType, isConversationAdapter } from "./conversation-continuation.js";
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
import { firstTurnDeliveredCommentIds } from "./queued-comment-delivery.js";
import { canResumePreservedResultFinalization } from "./preserved-result-finalization.js";
type Run = Pick<typeof heartbeatRuns.$inferSelect,
"id" | "companyId" | "agentId" | "status" | "startedAt" | "finishedAt" | "createdAt" |
@@ -404,6 +405,16 @@ export async function admitExplicitNativeContinuation(input: {
actionId: action.id, operationIds: recovery.preservationOperationIds });
const repairedTerminal = completedResult && coordinator.phase === "terminal_failure"
&& await hasVerifiedWorkspaceRepair(db, run);
// A Retry of this run whose files are saved applies the accepted result
// it already has, never a new agent turn. Saved files can be verified
// before their recovery attempt settles, so wait for that attempt too.
// The wait ends when preserved-result finalization coalesces this intent
// (resumeTaskRetry or the recovery sweep). Records that do not prove an
// applicable result (Stop, newer run, other owner) keep today's turn.
if (repairedTerminal && taskRetry?.sourceRunId === run.id && taskRetry.strategy === "evaluate"
&& await canResumePreservedResultFinalization(db, companyId, run.id,
["scheduled", "running", "awaiting_evidence"].includes(recovery?.state ?? "") ? "pending" : "ready"))
return blocked("finalizing_saved_result", "Paperclip is finishing the saved result.");
if (completedResult && !repairedTerminal && !(freshIntent && snapshotSettled)) {
// Nothing finishes the saved result while a stopped copyback owner is
// unverified. The run's live retained recovery records that hold; name
@@ -129,6 +129,7 @@ import {
} from "../native-runtime/index.js";
import { runningProcesses } from "../../adapters/index.js";
import { parseObject } from "../../adapters/utils.js";
import { resumePreservedResultFinalization } from "../preserved-result-finalization.js";
import { isNativeRunnerOwnershipHeld, nativeManualAdmissionOwnerColumns, nativeRunnerOwnershipNotHeldCondition } from "../native-runtime/native-runner-ownership.js";
import { readNativeStartupCleanup, acknowledgeNativeStartupCleanup } from "../native-runtime/native-startup-cleanup.js";
import {
@@ -851,6 +852,14 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
responsibleUserId: intent.actorId },
issueStateGuard: { assigneeAgentId: wake.agentId, statuses: ["todo", "in_progress", "in_review", "blocked"] },
idempotencyKey: `task-retry-dispatch:${wake.id}` }, wake.id);
// Admission held this Retry because its source run's files are saved and
// its accepted result can be applied. Apply it now instead of a new turn.
// One key read after admission; the resumption runs only for a held Retry.
const [held] = await db.select({ status: agentWakeupRequests.status, payload: agentWakeupRequests.payload })
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.id, wake.id)));
if (held?.status === "deferred_issue_execution"
&& parseObject(held.payload?.executionWait).reason === "finalizing_saved_result")
await resumePreservedResultFinalization(db, companyId, intent.sourceRunId);
}
async function resumeExecutionWaitComments(scope: DeferredReceiptRecoveryOptions = {}) {
@@ -4,54 +4,123 @@ import { agentWakeupRequests, heartbeatRuns, issueRecoveryActions, issues, nativ
import { readSystemRecovery } from "./system-recovery-state.js";
import { preservedResultHasStop } from "./native-runtime/preserved-native-result-finalization.js";
import { readWorkspaceRecovery, verifiedWorkspaceRepairReceipt } from "./workspace-recovery-receipt.js";
import { persistActivity, publishActivity } from "./activity-log.js";
import { logger } from "../middleware/logger.js";
function isLockNotAvailable(error: unknown): boolean {
for (let current = error, depth = 0; current && depth < 4; depth++) {
if ((current as { code?: unknown }).code === "55P03") return true;
current = (current as { cause?: unknown }).cause;
}
return false;
}
/** The durable facts that let a terminal run's accepted result be applied once
* its files are saved: the coordinator names an accepted, succeeded result of
* this run; the run's own saved-work recovery completed with a verified receipt;
* nobody stopped the run; and no newer run, owner or applied generation exists.
* Any missing or contradicting fact returns null, so callers keep their existing
* behaviour. `lock` takes the row locks that resumption needs. */
type Lock = false | "nowait" | "wait";
async function readPreservedResult(db: Db, companyId: string, runId: string, lock: Lock) {
const applicable = await readApplicableResult(db, companyId, runId, lock);
if (!applicable) return null;
const { coordinator, issue, run, result } = applicable;
const [action] = await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issue.id),
sql`${issueRecoveryActions.evidence}->>'runId' = ${runId}`,
)).orderBy(desc(issueRecoveryActions.createdAt)).limit(1);
const recovery = readSystemRecovery(action?.evidence);
const workspace = readWorkspaceRecovery(action?.evidence ?? null);
if (recovery?.state !== "complete" || recovery.sourceRunId !== run.id || !recovery.preservationOperationIds.length
|| workspace?.status !== "recovered" || workspace.runId !== run.id
|| !await verifiedWorkspaceRepairReceipt(db, run)) return null;
const [intent] = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.agentId, run.agentId),
eq(agentWakeupRequests.status, "deferred_issue_execution"), eq(agentWakeupRequests.reason, "task_retry"),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
sql`${agentWakeupRequests.payload}#>>'{taskRetry,sourceRunId}' = ${runId}`,
)).orderBy(desc(agentWakeupRequests.requestedAt)).limit(1);
const generationId = intent?.id ?? recovery.generationId;
if (coordinator.recoveryHistory.some(entry => entry.kind === "preserved_result_finalization" &&
entry.generationId === generationId)) return null;
return { coordinator, issue, run, result, recovery, intent, generationId };
}
/** The facts that do not depend on the saved-work proof: an accepted, succeeded
* result held by a terminal coordinator, no Stop, no newer run, and the same
* agent owner with no current execution. */
async function readApplicableResult(db: Db, companyId: string, runId: string, lock: Lock) {
// The finalizer and owner-stop confirmation lock the coordinator before the
// issue; Retry dispatch locks the issue first. This takes the coordinator
// first (like the finalizer). With "nowait" it does not wait for the issue or
// run while it holds the coordinator: a busy row fails the attempt
// (lock_not_available) and the caller tries again.
const coordinatorQuery = db.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, runId)));
const [coordinator] = lock ? await coordinatorQuery.for("update") : await coordinatorQuery;
if (!coordinator?.resultId || coordinator.phase !== "terminal_failure" || coordinator.leaseOwner ||
coordinator.failureDetail?.successorRunId) return null;
const issueQuery = db.select().from(issues).where(and(eq(issues.companyId, companyId), eq(issues.id, coordinator.issueId)));
const [issue] = lock === "nowait" ? await issueQuery.for("update", { noWait: true })
: lock ? await issueQuery.for("update") : await issueQuery;
const runQuery = db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId)));
const [run] = lock === "nowait" ? await runQuery.for("update", { noWait: true })
: lock ? await runQuery.for("update") : await runQuery;
if (!issue || !run || run.status === "cancelled" || preservedResultHasStop(run.resultJson)
|| issue.assigneeAgentId !== run.agentId || issue.assigneeUserId || ["done", "cancelled"].includes(issue.status) ||
issue.executionRunId || issue.checkoutRunId) return null;
const [newer] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), ne(heartbeatRuns.id, runId),
or(eq(heartbeatRuns.issueId, issue.id), eq(heartbeatRuns.nativeIssueId, issue.id),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`),
gte(heartbeatRuns.createdAt, run.createdAt),
)).limit(1);
if (newer) return null;
const [result] = await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
eq(nativeRunResults.companyId, companyId), eq(nativeRunResults.issueId, issue.id),
eq(nativeRunResults.runId, runId), eq(nativeRunResults.id, coordinator.resultId),
eq(nativeRunResults.schemaStatus, "accepted"),
sql`${nativeRunResults.resultJson}#>>'{terminal,runTerminalState}' = 'succeeded'`,
));
return result ? { coordinator, issue, run, result } : null;
}
/** Read-only: whether resumePreservedResultFinalization would apply this run's
* saved accepted result now (`ready`), or could once the run's saved-work
* recovery completes (`pending`). Retry admission uses it to wait for that
* result instead of starting a new agent turn. It takes no locks. */
export async function canResumePreservedResultFinalization(db: Db, companyId: string, runId: string,
when: "ready" | "pending" = "ready") {
return (when === "ready" ? await readPreservedResult(db, companyId, runId, false)
: await readApplicableResult(db, companyId, runId, false)) !== null;
}
/** Re-enter the existing finalizer only after preservation. This does not
* recreate a result, grant an executor budget, or apply it over newer work. */
export async function resumePreservedResultFinalization(db: Db, companyId: string, runId: string) {
return db.transaction(async tx => {
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, runId),
)).for("update");
if (!coordinator?.resultId || coordinator.phase !== "terminal_failure" || coordinator.leaseOwner ||
coordinator.failureDetail?.successorRunId) return false;
const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, companyId),
eq(issues.id, coordinator.issueId))).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.id, runId))).for("update");
if (!issue || !run || run.status === "cancelled" || preservedResultHasStop(run.resultJson)
|| issue.assigneeAgentId !== run.agentId || issue.assigneeUserId || ["done", "cancelled"].includes(issue.status) ||
issue.executionRunId || issue.checkoutRunId) return false;
const [newer] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), ne(heartbeatRuns.id, runId),
or(eq(heartbeatRuns.issueId, issue.id), eq(heartbeatRuns.nativeIssueId, issue.id),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`),
gte(heartbeatRuns.createdAt, run.createdAt),
)).limit(1);
if (newer) return false;
const [result] = await tx.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
eq(nativeRunResults.companyId, companyId), eq(nativeRunResults.issueId, issue.id),
eq(nativeRunResults.runId, runId), eq(nativeRunResults.id, coordinator.resultId),
eq(nativeRunResults.schemaStatus, "accepted"),
sql`${nativeRunResults.resultJson}#>>'{terminal,runTerminalState}' = 'succeeded'`,
));
const [action] = await tx.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issue.id),
sql`${issueRecoveryActions.evidence}->>'runId' = ${runId}`,
)).orderBy(desc(issueRecoveryActions.createdAt)).limit(1);
const recovery = readSystemRecovery(action?.evidence);
const workspace = readWorkspaceRecovery(action?.evidence ?? null);
if (!result || recovery?.state !== "complete" || recovery.sourceRunId !== run.id || !recovery.preservationOperationIds.length
|| workspace?.status !== "recovered" || workspace.runId !== run.id
|| !await verifiedWorkspaceRepairReceipt(tx as unknown as Db, run)) return false;
const [intent] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.agentId, run.agentId),
eq(agentWakeupRequests.status, "deferred_issue_execution"), eq(agentWakeupRequests.reason, "task_retry"),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
sql`${agentWakeupRequests.payload}#>>'{taskRetry,sourceRunId}' = ${runId}`,
)).orderBy(desc(agentWakeupRequests.requestedAt)).limit(1);
const generationId = intent?.id ?? recovery.generationId;
if (coordinator.recoveryHistory.some(entry => entry.kind === "preserved_result_finalization" &&
entry.generationId === generationId)) return false;
// Each attempt rereads every fact. A row held by another writer (usually a
// Retry dispatch for this task) is briefly retried without waiting while the
// coordinator is held. The last attempt waits for its locks, as this
// resumption always did, so a recovered result is never left unapplied
// because a row stayed busy.
for (let attempt = 1; ; attempt += 1) {
const wait = attempt >= PRESERVED_RESULT_LOCK_ATTEMPTS;
const resumed = await attemptPreservedResultFinalization(db, companyId, runId, wait);
if (resumed !== "busy") return resumed;
await new Promise(resolve => setTimeout(resolve, PRESERVED_RESULT_LOCK_DELAY_MS));
}
}
const PRESERVED_RESULT_LOCK_ATTEMPTS = 10;
const PRESERVED_RESULT_LOCK_DELAY_MS = 100;
async function attemptPreservedResultFinalization(db: Db, companyId: string, runId: string, wait: boolean): Promise<boolean | "busy"> {
const resumed = await db.transaction(async tx => {
const found = await readPreservedResult(tx as unknown as Db, companyId, runId, wait ? "wait" : "nowait");
if (!found) return null;
const { coordinator, issue, run, result, recovery, intent, generationId } = found;
await notifyNativeFinalizationWork(tx, { companyId, runId });
await tx.update(nativeRunFinalizations).set({
phase: coordinator.assessmentId ? "arbitrating" : "workspace_finalizing",
@@ -67,6 +136,16 @@ export async function resumePreservedResultFinalization(db: Db, companyId: strin
status: "coalesced", runId, finishedAt: new Date(), updatedAt: new Date(),
payload: sql`jsonb_set(${agentWakeupRequests.payload}, '{taskRetry,strategy}', '"finalize_result"'::jsonb)`,
}).where(eq(agentWakeupRequests.id, intent.id));
return true;
return persistActivity(tx as unknown as Db, { companyId, actorType: "system", actorId: "task-recovery",
action: "issue.saved_result_finalization_resumed", entityType: "issue", entityId: issue.id, runId: run.id,
details: { resultId: result.id, preservationGenerationId: recovery.generationId,
...(intent ? { requestId: intent.id, strategy: "finalize_result" } : {}) } });
}).catch(error => {
if (isLockNotAvailable(error)) return "busy" as const;
throw error;
});
if (resumed === "busy") return "busy";
if (!resumed) return false;
try { publishActivity(resumed.publication); } catch { logger.warn({ runId }, "Saved result activity delivery deferred"); }
return true;
}
@@ -2,8 +2,8 @@ import { randomUUID } from "node:crypto";
import { readdir, readFile } from "node:fs/promises";
import path from "node:path";
import request from "supertest";
import { and, eq } from "drizzle-orm";
import { agentWakeupRequests, executionWorkspaces, heartbeatRuns, issues } from "@paperclipai/db";
import { and, eq, sql } from "drizzle-orm";
import { activityLog, agentWakeupRequests, costEvents, executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults } from "@paperclipai/db";
import { heartbeatService } from "./heartbeat.js";
import { describe, expect, it, vi } from "vitest";
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
@@ -13,8 +13,9 @@ import { readWorkspaceRecovery } from "./workspace-recovery-receipt.js";
import { REMOTE_WORKSPACE_SANDBOX_DISK_FULL } from "./remote-workspace-transport.js";
import { getExecutionBlocker } from "./execution-blocker.js";
import { taskInterruptionPresentation } from "./task-interruption-presentation.js";
import { finalizeNativeRun } from "./native-runtime/native-run-finalizer.js";
import { createSavedWorkspaceLoader, readRemoteWorkspaceAuthority } from "./remote-workspaces.js";
import { useWorkspaceRecoveryFixture, acceptedRemoteExport, action, db, receiptOnlyRecoveryApp } from "./__tests__/workspace-recovery-fixture.js";
import { useWorkspaceRecoveryFixture, acceptedRemoteExport, action, db, receiptOnlyRecoveryApp, recoverRunHolds, run } from "./__tests__/workspace-recovery-fixture.js";
const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
vi.mock("./environment-execution-target.js", async importOriginal => {
const actual = await importOriginal<typeof import("./environment-execution-target.js")>();
@@ -22,31 +23,67 @@ vi.mock("./environment-execution-target.js", async importOriginal => {
(input.leaseId ? targets.get(input.leaseId) : undefined) ?? actual.resolveEnvironmentExecutionTarget(input) };
});
/** An accepted run whose save failed on a full sandbox disk, handed to the board. */
async function diskFullHold() {
const f = await acceptedRemoteExport();
const target = targets.get(f.lease.id)!, syncOut = target.runner!.syncOut!;
const disk = { full: true, saves: 0 };
target.runner!.syncOut = async operations => {
disk.saves++;
if (disk.full) throw new Error(REMOTE_WORKSPACE_SANDBOX_DISK_FULL);
return syncOut(operations);
};
await recoverRetainedWorkspace(f.input);
await recoverRetainedWorkspace(f.input);
expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "exhausted",
failure: { retryClass: "user_action_required", cause: { label: "sandbox_disk_full" } } });
// The route's own dispatch is replaced, so each test decides when it lands.
const retry = async () => (await request(receiptOnlyRecoveryApp()).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body as { requestId: string; accepted: boolean; observedRunId: string };
const runIds = async () => (await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(eq(heartbeatRuns.companyId, f.companyId))).map(row => row.id).sort();
const recoverAll = () => recoverRunHolds(f.input);
return { ...f, disk, retry, runIds, recoverAll };
}
const wake = async (id: string) => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, id)))[0]!;
const coordinator = async (runId: string) => (await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0]!;
/** The Retry applied the saved accepted result: no new run, and the source run and task finish from it. */
async function expectSavedResultApplied(f: Awaited<ReturnType<typeof diskFullHold>>, requestId: string, runsBefore: string[]) {
expect(await f.runIds()).toEqual(runsBefore);
expect(await wake(requestId)).toMatchObject({ status: "coalesced", runId: f.runId,
payload: { taskRetry: { strategy: "finalize_result" } } });
expect(await coordinator(f.runId)).toMatchObject({ phase: "workspace_finalizing", resultId: f.resultId,
recoveryHistory: [expect.objectContaining({ kind: "preserved_result_finalization", generationId: requestId, resultId: f.resultId })] });
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, f.companyId),
eq(activityLog.action, "issue.saved_result_finalization_resumed")))).toEqual([expect.objectContaining({
actorType: "system", entityId: f.issueId, runId: f.runId,
details: expect.objectContaining({ requestId, resultId: f.resultId, strategy: "finalize_result" }) })]);
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
.toMatchObject({ status: "done", executionRunId: null, checkoutRunId: null });
expect((await run(f.runId)).status).toBe("succeeded");
expect(await f.runIds()).toEqual(runsBefore);
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toEqual([]);
}
describe("saved-work recovery on a full sandbox disk", () => {
useWorkspaceRecoveryFixture(targets);
it("ends recovery at the first full-disk save with a named cause and a Retry that saves once there is room", async () => {
const f = await acceptedRemoteExport();
const target = targets.get(f.lease.id)!, syncOut = target.runner!.syncOut!;
let full = true, saves = 0;
target.runner!.syncOut = async operations => {
saves++;
if (full) throw new Error(REMOTE_WORKSPACE_SANDBOX_DISK_FULL);
return syncOut(operations);
};
it("ends recovery at the first full-disk save with a named cause and a Retry that saves and applies the accepted result without a new turn", async () => {
const f = await diskFullHold();
const workspaceBefore = (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId)))[0];
await recoverRetainedWorkspace(f.input);
const failed = readSystemRecovery((await action(f.actionId)).evidence);
expect(failed).toMatchObject({ state: "exhausted", nextAttemptAt: null, attempts: 1, failure: {
code: "export_interrupted", retryClass: "user_action_required",
cause: { label: "sandbox_disk_full", code: "ENOSPC", status: null } } });
const savesAtFailure = saves;
const savesAtFailure = f.disk.saves;
expect(savesAtFailure).toBeGreaterThan(0);
await recoverRetainedWorkspace(f.input);
await recoverRetainedWorkspace(f.input);
expect(saves).toBe(savesAtFailure);
expect(f.disk.saves).toBe(savesAtFailure);
const handedOff = await action(f.actionId);
expect(handedOff).toMatchObject({ status: "active", ownerType: "board" });
expect(handedOff.nextAction).toMatch(/sandbox disk is full/);
@@ -64,14 +101,12 @@ describe("saved-work recovery on a full sandbox disk", () => {
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ canRetry: true, observedRunId: f.runId });
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.runId,
workspaceRecovery: { canRetry: true } });
const runsBefore = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
const runsBefore = await f.runIds();
full = false;
const app = receiptOnlyRecoveryApp();
const receipt = (await request(app).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body;
f.disk.full = false;
const receipt = await f.retry();
expect(receipt).toMatchObject({ accepted: true, observedRunId: f.runId });
await recoverRetainedWorkspace(f.input);
await f.recoverAll();
const recovered = await action(f.actionId);
expect(readWorkspaceRecovery(recovered.evidence)?.status).toBe("recovered");
expect(readSystemRecovery(recovered.evidence)?.preservationOperationIds).toHaveLength(1);
@@ -81,20 +116,108 @@ describe("saved-work recovery on a full sandbox disk", () => {
snapshotId: authority.snapshotId! });
try { expect(await readFile(path.join(saved.root, "saved.txt"), "utf8")).toBe("finished work\n"); }
finally { await saved.cleanup(); }
const savesAfterRetry = saves;
await recoverRetainedWorkspace(f.input);
expect(saves).toBe(savesAfterRetry);
expect(await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)))
.toEqual(runsBefore);
const savesAfterRetry = f.disk.saves;
await f.recoverAll();
expect(f.disk.saves).toBe(savesAfterRetry);
expect(await f.runIds()).toEqual(runsBefore);
// Retry dispatch lands after the save, before anything applied the result
// (also replayed and concurrent). It applies the accepted result and
// never starts another agent turn to repeat the finished work.
const heartbeat = heartbeatService(db);
await Promise.all([heartbeat.resumeTaskRetry(f.companyId, receipt.requestId),
heartbeat.resumeTaskRetry(f.companyId, receipt.requestId)]);
await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId);
await expectSavedResultApplied(f, receipt.requestId, runsBefore);
}, 60_000);
it("holds a Retry dispatched while another saved-work hold of the run is still settling, then applies the accepted result", async () => {
const f = await diskFullHold();
const runsBefore = await f.runIds();
f.disk.full = false;
const receipt = await f.retry();
// The recovery worker saves the files for the board hold first and then
// dispatches, while the hold Retry added for the same run is still
// scheduled. The saved files are already verified at that point.
await recoverRetainedWorkspace(f.input);
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
const pending = (await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, f.companyId),
sql`${issueRecoveryActions.evidence}->>'runId' = ${f.runId}`))).filter(hold => hold.id !== f.actionId);
expect(pending.map(hold => readSystemRecovery(hold.evidence)?.state)).toContain("scheduled");
const heartbeat = heartbeatService(db);
await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId);
expect(await f.runIds()).toEqual(runsBefore);
expect(await wake(receipt.requestId)).toMatchObject({ status: "deferred_issue_execution",
payload: { executionWait: { reason: "finalizing_saved_result" } } });
await f.recoverAll();
await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId);
await expectSavedResultApplied(f, receipt.requestId, runsBefore);
}, 60_000);
it("does not hold a Retry for a settling hold when the run records an operator Stop", async () => {
const f = await diskFullHold();
const source = await run(f.runId);
await db.update(heartbeatRuns).set({ resultJson: { ...source.resultJson, cancelledByUserId: "local-board" } })
.where(eq(heartbeatRuns.id, f.runId));
const runsBefore = await f.runIds();
f.disk.full = false;
const receipt = await f.retry();
await recoverRetainedWorkspace(f.input);
await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId);
expect((await f.runIds()).filter(id => !runsBefore.includes(id))).toHaveLength(1);
expect(await coordinator(f.runId)).toMatchObject({ phase: "terminal_failure", recoveryHistory: [] });
}, 60_000);
it("keeps the named full-disk rest and a working Retry when the disk is still full", async () => {
const f = await diskFullHold();
const runsBefore = await f.runIds();
const first = await f.retry();
await f.recoverAll();
await heartbeatService(db).resumeTaskRetry(f.companyId, first.requestId);
expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "exhausted",
failure: { retryClass: "user_action_required", cause: { label: "sandbox_disk_full" } } });
await f.recoverAll();
expect((await action(f.actionId)).nextAction).toMatch(/sandbox disk is full/);
expect(await f.runIds()).toEqual(runsBefore);
expect(await coordinator(f.runId)).toMatchObject({ phase: "terminal_failure", recoveryHistory: [] });
const issue = (await db.select().from(issues).where(eq(issues.id, f.issueId)))[0];
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ canRetry: true });
// Once there is room, the next Retry saves and applies the result.
f.disk.full = false;
const second = await f.retry();
await f.recoverAll();
await heartbeatService(db).resumeTaskRetry(f.companyId, second.requestId);
await expectSavedResultApplied(f, second.requestId, runsBefore);
}, 60_000);
it.each([
{ record: "no accepted result", prepare: async (f: Awaited<ReturnType<typeof diskFullHold>>) => {
await db.update(nativeRunResults).set({ schemaStatus: "rejected" }).where(eq(nativeRunResults.id, f.resultId));
} },
{ record: "an accepted result that did not succeed", prepare: async (f: Awaited<ReturnType<typeof diskFullHold>>) => {
await db.update(nativeRunResults).set({ resultJson: sql`jsonb_set(${nativeRunResults.resultJson}, '{terminal,runTerminalState}', '"failed"'::jsonb)` })
.where(eq(nativeRunResults.id, f.resultId));
} },
// The result and saved files are proven, but the run records an operator
// Stop, so applying the result is not provably wanted.
{ record: "an operator Stop recorded on the run", prepare: async (f: Awaited<ReturnType<typeof diskFullHold>>) => {
const source = await run(f.runId);
await db.update(heartbeatRuns).set({ resultJson: { ...source.resultJson, cancelledByUserId: "local-board" } })
.where(eq(heartbeatRuns.id, f.runId));
} },
])("keeps today's Retry, one new run, with $record", async ({ prepare }) => {
const f = await diskFullHold();
await prepare(f);
const runsBefore = await f.runIds();
f.disk.full = false;
const receipt = await f.retry();
await f.recoverAll();
expect(readWorkspaceRecovery((await action(f.actionId)).evidence)?.status).toBe("recovered");
await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId);
const successors = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)))
.filter(row => row.id !== f.runId);
.filter(row => !runsBefore.includes(row.id));
expect(successors).toHaveLength(1);
expect(successors[0].contextSnapshot).toMatchObject({ previousRunId: f.runId, taskRetryRequestId: receipt.requestId });
expect(await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
eq(agentWakeupRequests.idempotencyKey, `task-retry-dispatch:${receipt.requestId}`)))).toHaveLength(1);
expect(await coordinator(f.runId)).toMatchObject({ phase: "terminal_failure", recoveryHistory: [] });
}, 60_000);
});
@@ -30,7 +30,7 @@ import { preserveRemoteWorkspacePreparationFailure } from "./native-runtime/nati
import { getExecutionBlocker } from "./execution-blocker.js";
import { taskInterruptionPresentation } from "./task-interruption-presentation.js";
import { resumePreservedResultFinalization } from "./preserved-result-finalization.js";
import { useWorkspaceRecoveryFixture, acceptedRemoteExport, action, boardRecoveryApp, db, interruptedRemotePreparation, makeRetryDue, root, run, seed, withheldPreparationRetry } from "./__tests__/workspace-recovery-fixture.js";
import { useWorkspaceRecoveryFixture, acceptedRemoteExport, action, boardRecoveryApp, db, interruptedRemotePreparation, makeRetryDue, recoverRunHolds, root, run, seed, withheldPreparationRetry } from "./__tests__/workspace-recovery-fixture.js";
const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
vi.mock("./environment-execution-target.js", async importOriginal => {
const actual = await importOriginal<typeof import("./environment-execution-target.js")>();
@@ -289,10 +289,10 @@ describe("automatic retained workspace recovery", () => {
// "exhausted before upgrade": the wave databases, where earlier code already
// spent the whole budget on rechecks and recorded no hold.
it.each([
{ wave: "s13", spentAttempts: 0, expectedStrategy: "finalize_result", exhaustedBeforeUpgrade: false },
{ wave: "s12", spentAttempts: 1, expectedStrategy: "continue", exhaustedBeforeUpgrade: false },
{ wave: "exhausted before upgrade", spentAttempts: 0, expectedStrategy: "finalize_result", exhaustedBeforeUpgrade: true },
])("lets an accepted Retry finish after a killed copyback owner instead of waiting forever ($wave)", async ({ spentAttempts, expectedStrategy, exhaustedBeforeUpgrade }) => {
{ wave: "s13", spentAttempts: 0, exhaustedBeforeUpgrade: false },
{ wave: "s12", spentAttempts: 1, exhaustedBeforeUpgrade: false },
{ wave: "exhausted before upgrade", spentAttempts: 0, exhaustedBeforeUpgrade: true },
])("lets an accepted Retry finish after a killed copyback owner instead of waiting forever ($wave)", async ({ spentAttempts, exhaustedBeforeUpgrade }) => {
const f = await acceptedRemoteExport();
const issue = (await db.select().from(issues).where(eq(issues.id, f.issueId)))[0];
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.source.agentId, status: "running" });
@@ -354,25 +354,19 @@ describe("automatic retained workspace recovery", () => {
stopEvidence: "The killed control-plane process exited and its host was replaced.",
}).expect(202);
expect((await run(f.runId)).runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeUndefined();
// The same steps the periodic sweep runs for this action.
await recoverRetainedWorkspace(f.input);
// The same steps the periodic sweep runs for this run's holds.
await recoverRunHolds(f.input);
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
await resumePreservedResultFinalization(db, f.companyId, f.runId);
await heartbeatService(db).resumeTaskRetry(f.companyId, retry.requestId!);
// The one accepted click now does work: it finishes the saved result, or,
// after an earlier recovery handoff (s12), continues in exactly one new run.
// The one accepted click now does work: it finishes the saved result,
// also after an earlier recovery failure (s12), with no new agent turn.
const [settled] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!));
expect(settled).toMatchObject({ status: "coalesced", runId: expect.any(String) });
if (expectedStrategy === "finalize_result") {
expect(settled).toMatchObject({ runId: f.runId, payload: { taskRetry: { strategy: "finalize_result" } } });
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "done" });
} else {
expect(settled.payload).toMatchObject({ taskRetry: { strategy: "continue" } });
expect(await db.select({ id: heartbeatRuns.id, previousRunId: sql<string>`${heartbeatRuns.contextSnapshot}->>'previousRunId'` })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued"))))
.toEqual([{ id: settled.runId, previousRunId: f.runId }]);
}
expect(settled).toMatchObject({ status: "coalesced", runId: f.runId, payload: { taskRetry: { strategy: "finalize_result" } } });
expect(await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
eq(heartbeatRuns.status, "queued")))).toEqual([]);
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "done" });
}, 60_000);
// RC-50b (wave s14/t05): Retry was accepted while saved-work recovery ran.
@@ -403,17 +397,17 @@ describe("automatic retained workspace recovery", () => {
const second = (await request(app).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body;
expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "scheduled", attempts: 0 });
await recoverRetainedWorkspace(f.input);
await recoverRunHolds(f.input);
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
await resumePreservedResultFinalization(db, f.companyId, f.runId);
await heartbeatService(db).resumeTaskRetry(f.companyId, second.requestId!);
// After a board handoff the Retry continues from the recovered files in
// exactly one new run (as in the s12 case), never another wait.
// After a board handoff the Retry applies the accepted result from the
// recovered files, never another wait and never a new agent turn.
const [settled] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, second.requestId!));
expect(settled).toMatchObject({ status: "coalesced", runId: expect.any(String), payload: { taskRetry: { strategy: "continue" } } });
expect(await db.select({ id: heartbeatRuns.id, previousRunId: sql<string>`${heartbeatRuns.contextSnapshot}->>'previousRunId'` })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued"))))
.toEqual([{ id: settled.runId, previousRunId: f.runId }]);
expect(settled).toMatchObject({ status: "coalesced", runId: f.runId, payload: { taskRetry: { strategy: "finalize_result" } } });
expect(await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
eq(heartbeatRuns.status, "queued")))).toEqual([]);
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "done" });
}, 60_000);
it("names a deterministic saved-files refusal behind Retry when no fallback can continue, and another Retry shows it again", async () => {
@@ -373,6 +373,25 @@ describe("governed retained-file recovery", () => {
expect(localStorage.getItem("workspace-file-recovery:company-1:issue-1:action-1:run-1:board-1")).toContain(previous);
});
it("hands a full-disk save rest to the task's Retry instead of its own recovery button", async () => {
const diskFull = { ...action, ownerType: "board", evidence: { ...action.evidence, systemRecovery: {
state: "exhausted", preservationOperationIds: [], failure: { code: "export_interrupted",
retryClass: "user_action_required", cause: { label: "sandbox_disk_full", code: "ENOSPC", status: null } } } },
} as unknown as IssueRecoveryAction;
api.inspect.mockResolvedValue(inspection(diskFull));
const owns = vi.fn();
await act(async () => root.render(<QueryClientProvider client={client}>
<WorkspaceExportRecovery action={diskFull} issueId="issue-1" companyId="company-1" onRecoveryControlChange={owns} />
</QueryClientProvider>));
await settle();
expect(button("Recover saved files")).toBeUndefined();
expect(button("Try file recovery again")).toBeUndefined();
expect(owns).toHaveBeenLastCalledWith(false);
expect(owns).not.toHaveBeenCalledWith(true);
expect(container.textContent).toContain("use Retry to save them");
expect(api.recover).not.toHaveBeenCalled();
});
it("only reports recovered files after the durable preservation receipt is present", async () => {
const recovered = { ...action, evidence: { ...action.evidence, workspaceRecovery: {
...(action.evidence.workspaceRecovery as object), status: "recovered",
+15 -2
View File
@@ -125,6 +125,14 @@ function isRetainedExport(action: IssueRecoveryAction, issueId: string, companyI
&& typeof action.evidence.runId === "string" && action.evidence.runId.length > 0;
}
/** Saved-work recovery ended because the sandbox disk was still full (#20758). */
function isSandboxDiskFullRest(action: IssueRecoveryAction) {
const system = asObject(action.evidence.systemRecovery);
const failure = asObject(system.failure);
return system.state === "exhausted" && failure.retryClass === "user_action_required"
&& asObject(failure.cause).label === "sandbox_disk_full";
}
function isControllerStopRequired(action: IssueRecoveryAction, issueId: string, companyId: string) {
return action.companyId === companyId && action.sourceIssueId === issueId
&& action.ownerType === "board" && ["active", "escalated"].includes(action.status)
@@ -343,21 +351,26 @@ function RetainedFilesRecovery({ action, issueId, companyId, className, onRecove
const waiting = !failedReceipt && !saved && (restoring || acknowledged || serverAcknowledged);
const unconfirmed = Boolean(pending && !acknowledged && !serverAcknowledged);
const sourceLoading = sourceView.kind === "loading" || sourceView.kind === "reconnecting";
// A save that rested on a full sandbox disk is resumed by the task's Retry,
// which saves the files again and then applies the saved result. Its note
// names Retry, so this panel offers no separate control for it.
const diskFullRest = isSandboxDiskFullRest(current ?? action);
// A verified source makes this panel the governed recovery path. Otherwise the
// task's own Retry (renewed, re-verified recovery) remains the next action.
const ownsRecoveryControl = canMutate && (sourceLoading || Boolean(current && verified && !saved));
const ownsRecoveryControl = canMutate && !diskFullRest && (sourceLoading || Boolean(current && verified && !saved));
// `waiting` already excludes a failed accepted request, so the reason stays
// visible after a failed attempt and hides only while recovery is running.
const unverified = Boolean(current && !verified && !saved && !waiting);
return <div className={cn("flex flex-col gap-2 text-sm", className)} data-testid="retained-file-recovery">
<ReportRecoveryControl owns={ownsRecoveryControl} onChange={onRecoveryControlChange} />
<p role="status">{saved ? "Saved files recovered. Task recovery is still being checked."
: diskFullRest && !waiting ? "The sandbox disk was full when Paperclip saved the files. Free space, then use Retry to save them and finish the task."
: failedReceipt ? "File recovery has not finished. The saved source remains protected."
: restoring ? "Recovering saved files. Work remains paused until recovery is verified."
: waiting ? "File recovery is queued. Waiting for verified saved files."
: "The saved files need recovery before work can continue."}</p>
{canMutate ? <>
{current && verified && retryable && !waiting && (!pending || unconfirmed && checkedUnconfirmed || confirmedFailedAttempt) ? <Button variant="outline" size="sm"
{current && verified && retryable && !waiting && !diskFullRest && (!pending || unconfirmed && checkedUnconfirmed || confirmedFailedAttempt) ? <Button variant="outline" size="sm"
className="self-start" disabled={request.isPending || check.isPending}
onClick={() => request.mutate(confirmedFailedAttempt)}>{request.isPending ? "Requesting file recovery…"
: confirmedFailedAttempt ? "Try file recovery again" : "Recover saved files"}</Button> : null}