Files
PaperClipAI/server/src/services/preserved-result-finalization.ts
T
DottaandClaude Opus 5.5 df89950eb6 fix(native): save or set aside a superseded run's unsaved workspace so its task queue moves (#21136)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result. Then a finalizer exports the
run's workspace files and applies the result to the task.
> - When a newer run already decided the task, the finalizer retires the
older result as superseded. Since #21106, this also ends the run.
> - If that retirement interrupts a failed workspace export, the run's
files stay unsaved. The run also stays the writer of the task's
workspace. Nothing records those files for recovery, so nothing ever
releases the workspace.
> - Every run queued on the task then waits behind a save that never
settles. After one hour, each queued run fails setup instead.
> - This pull request records those files for the existing automatic
workspace recovery. Recovery saves them from the retained sandbox, or,
when every bounded attempt fails, continues the task from its last save
with the files attached.
> - The benefit is that the task's queue moves again by itself, and the
run's unsaved work is saved or kept findable. The superseded result is
never applied.

## Linked Issues or Issue Description

Refs #21106

**What happened?**
On a hosted staging server, a native run accepted its result at
11:31:08Z. Its workspace export failed once with
`remote_workspace_archive_fs_eacces`. The finalizer then retired it as
`native_finalization_superseded`, because another run had already
decided the task. #21106 ended that run at 14:59:17Z (`failed`). The
task still could not move. The next queued run was cancelled with
`workspace_busy` ("ended, but its workspace save has not settled"). Its
retry re-armed every minute. Four more runs stayed queued. The task
showed no recovery action.

**Expected behavior**
The ended run's unsaved files are saved from its retained sandbox, and
the workspace is released. If they cannot be saved within the bounded
retries, the task continues from its last committed save, and the files
are attached to the task. The queued runs proceed in both cases. The
superseded result is never applied.

**Steps to reproduce**
1. Accept a native result for a run that writes a remote workspace.
2. Let the workspace export fail once, while a newer run has already
committed the task's decision.
3. See that the finalizer is `terminal_failure` /
`native_finalization_superseded` and the run is `failed`.
4. See that the workspace still names the run as its dirty writer, and
that no retained-file recovery action exists.
5. Queue another run on the task. It defers with `workspace_busy`, and
after one hour it fails setup with "The workspace still has an
unfinished writer or save operation."

**Root cause**
`recordRetryableFailure` (`native-run-finalizer.ts`) retires a
superseded finalizer without preserving the export.
`preserveNativeFailedWorkspace` records a retained-file hold for an
accepted result only when the failure code is
`native_workspace_sync_out_retry_exhausted`. A
`native_finalization_superseded` coordinator never qualifies.
`readRemoteWorkspaceWriterHold` keeps an ended dirty writer as the
holder until its save settles. Without a hold, no worker ever settles
it.

**Paperclip version or commit**
Staging Core with #21106 (`81d2e4d8d9`). Master `ef2b53b675` has the
same behavior.

## What Changed

- `native-superseded-workspace-export.ts` (new): identifies a superseded
finalizer that failed in the workspace export. It selects ended runs
that still hold their workspace with unsaved files and no retained-file
hold. It records the hold through `preserveNativeFailedWorkspace`. It
checks again only while a just-ended run's sandbox stop receipt is still
arriving (at most one hour).
- `native-failed-workspace-recovery.ts`: `preserveNativeFailedWorkspace`
treats a superseded workspace export like an exhausted copy-back, with
the same proofs (run ended, no live controller, accepted result,
confirmed sandbox stop-and-retain, source still bound to the workspace).
It marks the hold `supersededWorkspaceExport`.
- `native-run-projection-work.ts`: the native run projection worker also
selects these runs. It loads them at startup and wakes when a finalizer
changes. Rows written by older releases are therefore settled after
deploy without a person.
- `workspace-recovery-fallback-discovery.ts` and
`workspace-recovery-fallback.ts`: a marked hold whose bounded re-export
is spent (at most 5 retries in 30 minutes) is eligible for the existing
continue-from-last-save fallback. That fallback sets aside only this
run's workspace epoch and keeps its location on the workspace record. It
attaches the readable files to the task unapplied. The task message says
Paperclip "could not save" the previous run's unsaved files.
- `preserved-result-finalization.ts`: a superseded result is never
resumed for application, even after its files are saved.
- `doc/execution-semantics.md`: document the rule.
- No schema change. Nothing changes on the path that starts a run or a
turn.

## Verification

- `server/src/services/superseded-workspace-export-recovery.test.ts`
(new). The first two tests fail on master and pass on this branch:
- "records the files for automatic recovery, saves them from the
retained sandbox and frees the workspace". It uses the staging row
shape. The projection worker records the hold, and a second pass changes
nothing. The workspace waits for that save. The system recovery saves
`saved.txt` into a committed snapshot. The workspace becomes free. The
finalizer, the result and the task's decision do not change, and the
result is not resumed. On master the worker selects nothing (`expected 0
to be 1`).
- "continues from the last save with the files attached when every
re-export fails". After an exhausted re-export budget, the hold is
resolved `restored`. The files are attached (`quarantine.state =
captured`). The workspace moves to epoch 2 with no writer, and the
abandoned epoch keeps its lease location. The message has no request for
a person to fix anything.
- "leaves a superseded result alone when it never failed in the
workspace export".
- Related suites pass: `native-run-projection-work`,
`native-finalization-recovery`, `workspace-recovery-fallback`,
`native-failed-workspace-recovery`, `workspace-recovery-retry-copyback`.
- `tsc --noEmit` for `server`.

## Risks

- Low to medium risk. The change acts only on runs whose finalizer is
already terminal and superseded, that failed in the workspace export,
and that still own a dirty workspace. All existing preservation proofs
still apply.
- The saved files of a superseded run become the workspace's next saved
version. The run's result (the task decision) is still never applied.
- When the export can never succeed, the task continues from its last
save. Files that cannot be read are not attached. The workspace record
keeps the set-aside epoch's location. Saving the readable part of a
partly unreadable workspace is not in scope.
- Runs that already failed setup before this deploys are not restarted
by this change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:20:54 -06:00

153 lines
9.7 KiB
TypeScript

import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js";
import { and, desc, eq, gte, ne, or, sql } from "drizzle-orm";
import { agentWakeupRequests, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
import { readSystemRecovery } from "./system-recovery-state.js";
import { preservedResultHasStop } from "./native-runtime/preserved-native-result-finalization.js";
import { readWorkspaceRecovery, verifiedWorkspaceRepairReceipt } from "./workspace-recovery-receipt.js";
import { persistActivity, publishActivity } from "./activity-log.js";
import { logger } from "../middleware/logger.js";
function isLockNotAvailable(error: unknown): boolean {
for (let current = error, depth = 0; current && depth < 4; depth++) {
if ((current as { code?: unknown }).code === "55P03") return true;
current = (current as { cause?: unknown }).cause;
}
return false;
}
/** The durable facts that let a terminal run's accepted result be applied once
* its files are saved: the coordinator names an accepted, succeeded result of
* this run; the run's own saved-work recovery completed with a verified receipt;
* nobody stopped the run; and no newer run, owner or applied generation exists.
* Any missing or contradicting fact returns null, so callers keep their existing
* behaviour. `lock` takes the row locks that resumption needs. */
type Lock = false | "nowait" | "wait";
async function readPreservedResult(db: Db, companyId: string, runId: string, lock: Lock) {
const applicable = await readApplicableResult(db, companyId, runId, lock);
if (!applicable) return null;
const { coordinator, issue, run, result } = applicable;
const [action] = await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issue.id),
sql`${issueRecoveryActions.evidence}->>'runId' = ${runId}`,
)).orderBy(desc(issueRecoveryActions.createdAt)).limit(1);
const recovery = readSystemRecovery(action?.evidence);
const workspace = readWorkspaceRecovery(action?.evidence ?? null);
if (recovery?.state !== "complete" || recovery.sourceRunId !== run.id || !recovery.preservationOperationIds.length
|| workspace?.status !== "recovered" || workspace.runId !== run.id
|| !await verifiedWorkspaceRepairReceipt(db, run)) return null;
const [intent] = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.agentId, run.agentId),
eq(agentWakeupRequests.status, "deferred_issue_execution"), eq(agentWakeupRequests.reason, "task_retry"),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
sql`${agentWakeupRequests.payload}#>>'{taskRetry,sourceRunId}' = ${runId}`,
)).orderBy(desc(agentWakeupRequests.requestedAt)).limit(1);
const generationId = intent?.id ?? recovery.generationId;
if (coordinator.recoveryHistory.some(entry => entry.kind === "preserved_result_finalization" &&
entry.generationId === generationId)) return null;
return { coordinator, issue, run, result, recovery, intent, generationId };
}
/** The facts that do not depend on the saved-work proof: an accepted, succeeded
* result held by a terminal coordinator, no Stop, no newer run, and the same
* agent owner with no current execution. */
async function readApplicableResult(db: Db, companyId: string, runId: string, lock: Lock) {
// The finalizer and owner-stop confirmation lock the coordinator before the
// issue; Retry dispatch locks the issue first. This takes the coordinator
// first (like the finalizer). With "nowait" it does not wait for the issue or
// run while it holds the coordinator: a busy row fails the attempt
// (lock_not_available) and the caller tries again.
const coordinatorQuery = db.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, runId)));
const [coordinator] = lock ? await coordinatorQuery.for("update") : await coordinatorQuery;
// A result retired as superseded is never applied, even after its files are saved.
if (!coordinator?.resultId || coordinator.phase !== "terminal_failure" || coordinator.leaseOwner ||
coordinator.failureDetail?.successorRunId || coordinator.failureCode === "native_finalization_superseded") return null;
const issueQuery = db.select().from(issues).where(and(eq(issues.companyId, companyId), eq(issues.id, coordinator.issueId)));
const [issue] = lock === "nowait" ? await issueQuery.for("update", { noWait: true })
: lock ? await issueQuery.for("update") : await issueQuery;
const runQuery = db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId)));
const [run] = lock === "nowait" ? await runQuery.for("update", { noWait: true })
: lock ? await runQuery.for("update") : await runQuery;
if (!issue || !run || run.status === "cancelled" || preservedResultHasStop(run.resultJson)
|| issue.assigneeAgentId !== run.agentId || issue.assigneeUserId || ["done", "cancelled"].includes(issue.status) ||
issue.executionRunId || issue.checkoutRunId) return null;
const [newer] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), ne(heartbeatRuns.id, runId),
or(eq(heartbeatRuns.issueId, issue.id), eq(heartbeatRuns.nativeIssueId, issue.id),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`),
gte(heartbeatRuns.createdAt, run.createdAt),
)).limit(1);
if (newer) return null;
const [result] = await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
eq(nativeRunResults.companyId, companyId), eq(nativeRunResults.issueId, issue.id),
eq(nativeRunResults.runId, runId), eq(nativeRunResults.id, coordinator.resultId),
eq(nativeRunResults.schemaStatus, "accepted"),
sql`${nativeRunResults.resultJson}#>>'{terminal,runTerminalState}' = 'succeeded'`,
));
return result ? { coordinator, issue, run, result } : null;
}
/** Read-only: whether resumePreservedResultFinalization would apply this run's
* saved accepted result now (`ready`), or could once the run's saved-work
* recovery completes (`pending`). Retry admission uses it to wait for that
* result instead of starting a new agent turn. It takes no locks. */
export async function canResumePreservedResultFinalization(db: Db, companyId: string, runId: string,
when: "ready" | "pending" = "ready") {
return (when === "ready" ? await readPreservedResult(db, companyId, runId, false)
: await readApplicableResult(db, companyId, runId, false)) !== null;
}
/** Re-enter the existing finalizer only after preservation. This does not
* recreate a result, grant an executor budget, or apply it over newer work. */
export async function resumePreservedResultFinalization(db: Db, companyId: string, runId: string) {
// Each attempt rereads every fact. A row held by another writer (usually a
// Retry dispatch for this task) is briefly retried without waiting while the
// coordinator is held. The last attempt waits for its locks, as this
// resumption always did, so a recovered result is never left unapplied
// because a row stayed busy.
for (let attempt = 1; ; attempt += 1) {
const wait = attempt >= PRESERVED_RESULT_LOCK_ATTEMPTS;
const resumed = await attemptPreservedResultFinalization(db, companyId, runId, wait);
if (resumed !== "busy") return resumed;
await new Promise(resolve => setTimeout(resolve, PRESERVED_RESULT_LOCK_DELAY_MS));
}
}
const PRESERVED_RESULT_LOCK_ATTEMPTS = 10;
const PRESERVED_RESULT_LOCK_DELAY_MS = 100;
async function attemptPreservedResultFinalization(db: Db, companyId: string, runId: string, wait: boolean): Promise<boolean | "busy"> {
const resumed = await db.transaction(async tx => {
const found = await readPreservedResult(tx as unknown as Db, companyId, runId, wait ? "wait" : "nowait");
if (!found) return null;
const { coordinator, issue, run, result, recovery, intent, generationId } = found;
await notifyNativeFinalizationWork(tx, { companyId, runId });
await tx.update(nativeRunFinalizations).set({
phase: coordinator.assessmentId ? "arbitrating" : "workspace_finalizing",
nextAttemptAt: new Date(), failureCode: null, failureDetail: null, updatedAt: new Date(),
recoveryHistory: [...coordinator.recoveryHistory, {
kind: "preserved_result_finalization", generationId,
preservationGenerationId: recovery.generationId,
resultId: result.id, previousFailureCode: coordinator.failureCode,
previousFailureDetail: coordinator.failureDetail, at: new Date().toISOString(),
}],
}).where(eq(nativeRunFinalizations.runId, runId));
if (intent) await tx.update(agentWakeupRequests).set({
status: "coalesced", runId, finishedAt: new Date(), updatedAt: new Date(),
payload: sql`jsonb_set(${agentWakeupRequests.payload}, '{taskRetry,strategy}', '"finalize_result"'::jsonb)`,
}).where(eq(agentWakeupRequests.id, intent.id));
return persistActivity(tx as unknown as Db, { companyId, actorType: "system", actorId: "task-recovery",
action: "issue.saved_result_finalization_resumed", entityType: "issue", entityId: issue.id, runId: run.id,
details: { resultId: result.id, preservationGenerationId: recovery.generationId,
...(intent ? { requestId: intent.id, strategy: "finalize_result" } : {}) } });
}).catch(error => {
if (isLockNotAvailable(error)) return "busy" as const;
throw error;
});
if (resumed === "busy") return "busy";
if (!resumed) return false;
try { publishActivity(resumed.publication); } catch { logger.warn({ runId }, "Saved result activity delivery deferred"); }
return true;
}