mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 10:52:06 +01:00
fix(native): save or set aside a superseded run's unsaved workspace so its task queue moves (#21136)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run saves its accepted result. Then a finalizer exports the run's workspace files and applies the result to the task. > - When a newer run already decided the task, the finalizer retires the older result as superseded. Since #21106, this also ends the run. > - If that retirement interrupts a failed workspace export, the run's files stay unsaved. The run also stays the writer of the task's workspace. Nothing records those files for recovery, so nothing ever releases the workspace. > - Every run queued on the task then waits behind a save that never settles. After one hour, each queued run fails setup instead. > - This pull request records those files for the existing automatic workspace recovery. Recovery saves them from the retained sandbox, or, when every bounded attempt fails, continues the task from its last save with the files attached. > - The benefit is that the task's queue moves again by itself, and the run's unsaved work is saved or kept findable. The superseded result is never applied. ## Linked Issues or Issue Description Refs #21106 **What happened?** On a hosted staging server, a native run accepted its result at 11:31:08Z. Its workspace export failed once with `remote_workspace_archive_fs_eacces`. The finalizer then retired it as `native_finalization_superseded`, because another run had already decided the task. #21106 ended that run at 14:59:17Z (`failed`). The task still could not move. The next queued run was cancelled with `workspace_busy` ("ended, but its workspace save has not settled"). Its retry re-armed every minute. Four more runs stayed queued. The task showed no recovery action. **Expected behavior** The ended run's unsaved files are saved from its retained sandbox, and the workspace is released. If they cannot be saved within the bounded retries, the task continues from its last committed save, and the files are attached to the task. The queued runs proceed in both cases. The superseded result is never applied. **Steps to reproduce** 1. Accept a native result for a run that writes a remote workspace. 2. Let the workspace export fail once, while a newer run has already committed the task's decision. 3. See that the finalizer is `terminal_failure` / `native_finalization_superseded` and the run is `failed`. 4. See that the workspace still names the run as its dirty writer, and that no retained-file recovery action exists. 5. Queue another run on the task. It defers with `workspace_busy`, and after one hour it fails setup with "The workspace still has an unfinished writer or save operation." **Root cause** `recordRetryableFailure` (`native-run-finalizer.ts`) retires a superseded finalizer without preserving the export. `preserveNativeFailedWorkspace` records a retained-file hold for an accepted result only when the failure code is `native_workspace_sync_out_retry_exhausted`. A `native_finalization_superseded` coordinator never qualifies. `readRemoteWorkspaceWriterHold` keeps an ended dirty writer as the holder until its save settles. Without a hold, no worker ever settles it. **Paperclip version or commit** Staging Core with #21106 (`81d2e4d8d9`). Master `ef2b53b675` has the same behavior. ## What Changed - `native-superseded-workspace-export.ts` (new): identifies a superseded finalizer that failed in the workspace export. It selects ended runs that still hold their workspace with unsaved files and no retained-file hold. It records the hold through `preserveNativeFailedWorkspace`. It checks again only while a just-ended run's sandbox stop receipt is still arriving (at most one hour). - `native-failed-workspace-recovery.ts`: `preserveNativeFailedWorkspace` treats a superseded workspace export like an exhausted copy-back, with the same proofs (run ended, no live controller, accepted result, confirmed sandbox stop-and-retain, source still bound to the workspace). It marks the hold `supersededWorkspaceExport`. - `native-run-projection-work.ts`: the native run projection worker also selects these runs. It loads them at startup and wakes when a finalizer changes. Rows written by older releases are therefore settled after deploy without a person. - `workspace-recovery-fallback-discovery.ts` and `workspace-recovery-fallback.ts`: a marked hold whose bounded re-export is spent (at most 5 retries in 30 minutes) is eligible for the existing continue-from-last-save fallback. That fallback sets aside only this run's workspace epoch and keeps its location on the workspace record. It attaches the readable files to the task unapplied. The task message says Paperclip "could not save" the previous run's unsaved files. - `preserved-result-finalization.ts`: a superseded result is never resumed for application, even after its files are saved. - `doc/execution-semantics.md`: document the rule. - No schema change. Nothing changes on the path that starts a run or a turn. ## Verification - `server/src/services/superseded-workspace-export-recovery.test.ts` (new). The first two tests fail on master and pass on this branch: - "records the files for automatic recovery, saves them from the retained sandbox and frees the workspace". It uses the staging row shape. The projection worker records the hold, and a second pass changes nothing. The workspace waits for that save. The system recovery saves `saved.txt` into a committed snapshot. The workspace becomes free. The finalizer, the result and the task's decision do not change, and the result is not resumed. On master the worker selects nothing (`expected 0 to be 1`). - "continues from the last save with the files attached when every re-export fails". After an exhausted re-export budget, the hold is resolved `restored`. The files are attached (`quarantine.state = captured`). The workspace moves to epoch 2 with no writer, and the abandoned epoch keeps its lease location. The message has no request for a person to fix anything. - "leaves a superseded result alone when it never failed in the workspace export". - Related suites pass: `native-run-projection-work`, `native-finalization-recovery`, `workspace-recovery-fallback`, `native-failed-workspace-recovery`, `workspace-recovery-retry-copyback`. - `tsc --noEmit` for `server`. ## Risks - Low to medium risk. The change acts only on runs whose finalizer is already terminal and superseded, that failed in the workspace export, and that still own a dirty workspace. All existing preservation proofs still apply. - The saved files of a superseded run become the workspace's next saved version. The run's result (the task decision) is still never applied. - When the export can never succeed, the task continues from its last save. Files that cannot be read are not attached. The workspace record keeps the set-aside epoch's location. Saving the readable part of a partly unreadable workspace is not in scope. - Runs that already failed setup before this deploys are not restarted by this change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
782d6bead4
commit
df89950eb6
8 files changed
+319
-9
No files matched your search
@@ -19,6 +19,7 @@ import { pinFailedObservedWorkspace } from "./native-observed-workspace-preserva
|
||||
import { hasConfirmedSandboxStopAndRetain } from "../sandbox-stop-and-retain.js";
|
||||
import { preparedRemoteWorkspaceSource } from "../remote-workspace-lifecycle.js";
|
||||
import { releaseUnchangedRemoteWorkspaceWriterById } from "../remote-workspaces.js";
|
||||
import { isSupersededWorkspaceExport, markSupersededWorkspaceExportHold } from "./native-superseded-workspace-export.js";
|
||||
|
||||
const reusableLeasePolicies = ["reuse_by_environment", "reuse_by_execution_workspace"];
|
||||
|
||||
@@ -103,8 +104,12 @@ export async function preserveNativeFailedWorkspace(db: Db, selected: Pick<typeo
|
||||
&& (claim.kind !== "reattach_remote_runner" || (claim.remote.kind !== "plugin_task" && claim.remote.providerLeaseId === reference.providerLeaseId
|
||||
&& claim.remote.remoteCwd === reference.remoteCwd)));
|
||||
if ((coordinator?.leaseOwner || (claim && coordinator?.phase === "observed")) && !ownsClaim) throw new NativeWorkspaceFinalizationBusyError();
|
||||
// A result retired as superseded after its workspace export failed owes
|
||||
// the same files as an exhausted copy-back: nothing else exports them, and
|
||||
// its writer keeps the workspace until they are saved or set aside.
|
||||
const supersededExport = Boolean(coordinator && isSupersededWorkspaceExport(coordinator));
|
||||
const exhaustedAcceptedCopyback = Boolean(coordinator?.resultId && coordinator.phase === "terminal_failure"
|
||||
&& coordinator.failureCode === "native_workspace_sync_out_retry_exhausted");
|
||||
&& coordinator.failureCode === "native_workspace_sync_out_retry_exhausted") || supersededExport;
|
||||
if (!coordinator || (coordinator.resultId && !exhaustedAcceptedCopyback
|
||||
&& !(options.requireCancelledStop && stoppedAcceptedCopyback(coordinator)))
|
||||
|| !(options.allowPreterminal ? ["observed", "terminal_failure"] : ["terminal_failure"]).includes(coordinator.phase)) return;
|
||||
@@ -170,6 +175,7 @@ export async function preserveNativeFailedWorkspace(db: Db, selected: Pick<typeo
|
||||
[retained] = await tx.update(heartbeatRuns).set({ nativePhase: "terminal_failure", nativePhaseUpdatedAt: new Date() })
|
||||
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId))).returning();
|
||||
}
|
||||
if (supersededExport && retainFiles) await markSupersededWorkspaceExportHold(tx as unknown as Db, retained);
|
||||
if (retainFiles && !alreadyRetained) await logActivity(tx as unknown as Db, {
|
||||
companyId: run.companyId, actorType: "system", actorId: "native-workspace-recovery",
|
||||
action: "issue.workspace_recovery_required", entityType: "issue",
|
||||
|
||||
@@ -8,6 +8,8 @@ import { pendingNativeStatusEffectsCondition } from "./native-pending-effect-wor
|
||||
import { pendingNativeRunProjectionCondition, projectCommittedRun } from "./native-run-projection.js";
|
||||
import { endSupersededNativeFinalizationRuns, NATIVE_FINALIZATION_SUPERSEDED } from "./native-run-finalizer.js";
|
||||
import { nativeRunnerOwnershipNotHeldCondition } from "./native-runner-ownership.js";
|
||||
import { preserveNativeFailedWorkspace } from "./native-failed-workspace-recovery.js";
|
||||
import { preserveSupersededWorkspaceExport, supersededWorkspaceExportUnpreservedCondition } from "./native-superseded-workspace-export.js";
|
||||
|
||||
type Entry = { id: string; companyId: string; issueId: string; dueAt: number };
|
||||
|
||||
@@ -25,7 +27,9 @@ function supersededRunStillRunningCondition() {
|
||||
}
|
||||
|
||||
export function nativeRunProjectionWork(db: Db, project = projectCommittedRun,
|
||||
endSuperseded: (db: Db, runIds: readonly string[]) => Promise<string[]> = endSupersededNativeFinalizationRuns) {
|
||||
endSuperseded: (db: Db, runIds: readonly string[]) => Promise<string[]> = endSupersededNativeFinalizationRuns,
|
||||
preserveExport: (db: Db, runId: string) => Promise<{ recheckAt: number | null }> =
|
||||
(db, runId) => preserveSupersededWorkspaceExport(db, runId, preserveNativeFailedWorkspace)) {
|
||||
function candidates(scope?: NativeRunProjectionScope, after?: string) {
|
||||
return db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, issueId: nativeRunFinalizations.issueId })
|
||||
.from(nativeRunFinalizations)
|
||||
@@ -34,7 +38,8 @@ export function nativeRunProjectionWork(db: Db, project = projectCommittedRun,
|
||||
.innerJoin(issues, and(eq(issues.id, nativeRunFinalizations.issueId), eq(issues.companyId, nativeRunFinalizations.companyId)))
|
||||
.where(and(or(and(eq(nativeRunFinalizations.phase, "committed"),
|
||||
pendingNativeRunProjectionCondition(db),
|
||||
not(pendingNativeStatusEffectsCondition(db))), supersededRunStillRunningCondition()),
|
||||
not(pendingNativeStatusEffectsCondition(db))), supersededRunStillRunningCondition(),
|
||||
supersededWorkspaceExportUnpreservedCondition()),
|
||||
scope ? eq(heartbeatRuns.companyId, scope.companyId) : undefined,
|
||||
scope?.issueId ? eq(nativeRunFinalizations.issueId, scope.issueId) : undefined,
|
||||
scope?.runId ? eq(heartbeatRuns.id, scope.runId) : undefined,
|
||||
@@ -66,6 +71,18 @@ export function nativeRunProjectionWork(db: Db, project = projectCommittedRun,
|
||||
await endSuperseded(db, [entry.id]);
|
||||
return (await candidates(scope)).length ? { ...entry, dueAt: Date.now() + 30_000 } : null;
|
||||
}
|
||||
const [unsavedExport] = await db.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
|
||||
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
|
||||
eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId)))
|
||||
.where(and(eq(nativeRunFinalizations.runId, entry.id), eq(nativeRunFinalizations.companyId, entry.companyId),
|
||||
supersededWorkspaceExportUnpreservedCondition()));
|
||||
if (unsavedExport) {
|
||||
// An ended superseded run still holds its workspace with unsaved files.
|
||||
// Record the files for automatic recovery; a refusal rests until the
|
||||
// next start or signal, except while its sandbox stop is still arriving.
|
||||
const { recheckAt } = await preserveExport(db, entry.id);
|
||||
return recheckAt === null ? null : { ...entry, dueAt: recheckAt };
|
||||
}
|
||||
const [run] = await db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId,
|
||||
status: heartbeatRuns.status, finishedAt: heartbeatRuns.finishedAt, nativePhase: heartbeatRuns.nativePhase,
|
||||
runtimeMode: heartbeatRuns.runtimeMode, nativeIssueId: heartbeatRuns.nativeIssueId,
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
import { and, eq, inArray, isNotNull, isNull, sql } from "drizzle-orm";
|
||||
import { environmentLeases, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import { remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared";
|
||||
import { SUPERSEDED_WORKSPACE_EXPORT_KEY, SUPERSEDED_WORKSPACE_EXPORT_SCHEMA } from "../workspace-recovery-fallback-discovery.js";
|
||||
import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
|
||||
import { hasRequiredWorkspaceRecovery } from "../workspace-restore-recovery-state.js";
|
||||
|
||||
/** The finalizer's verdict for a result a newer run already decided (see native-run-finalizer). */
|
||||
const SUPERSEDED = "native_finalization_superseded";
|
||||
/** How long a just-ended run's source may wait for its sandbox stop receipt before the check rests. */
|
||||
const STOP_RECEIPT_WAIT_MS = 60 * 60_000;
|
||||
const STOP_RECEIPT_RECHECK_MS = 60_000;
|
||||
|
||||
type Coordinator = Pick<typeof nativeRunFinalizations.$inferSelect, "resultId" | "phase" | "failureCode" | "failureDetail">;
|
||||
|
||||
/**
|
||||
* A finalizer retired as superseded while it was exporting the run's
|
||||
* workspace. The result is never applied, but the run's files were not saved
|
||||
* either, and the run still holds its workspace as the writer.
|
||||
*/
|
||||
export function isSupersededWorkspaceExport(coordinator: Coordinator) {
|
||||
const detail = coordinator.failureDetail;
|
||||
return Boolean(coordinator.resultId) && coordinator.phase === "terminal_failure" && coordinator.failureCode === SUPERSEDED
|
||||
&& typeof detail?.workspaceFinalizeAttempt === "number";
|
||||
}
|
||||
|
||||
/**
|
||||
* Names the retained-file hold of a superseded export, so that a spent
|
||||
* re-export budget continues the task from its last save (the retained-source
|
||||
* fallback) instead of waiting for a person: no result waits for these files.
|
||||
*/
|
||||
export async function markSupersededWorkspaceExportHold(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId">) {
|
||||
const marker = { schema: SUPERSEDED_WORKSPACE_EXPORT_SCHEMA, runId: run.id };
|
||||
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence}
|
||||
|| ${JSON.stringify({ [SUPERSEDED_WORKSPACE_EXPORT_KEY]: marker })}::jsonb` })
|
||||
.where(and(eq(issueRecoveryActions.companyId, run.companyId),
|
||||
eq(issueRecoveryActions.fingerprint, `native-workspace:${run.id}`),
|
||||
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
|
||||
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery' is not null`,
|
||||
sql`not (${issueRecoveryActions.evidence} ? ${SUPERSEDED_WORKSPACE_EXPORT_KEY})`));
|
||||
}
|
||||
|
||||
/**
|
||||
* An ended run whose superseded export left its files unsaved and its
|
||||
* workspace held, with no retained-file obligation recorded yet. Rows written
|
||||
* by releases before this rule match too, so the new build settles them itself.
|
||||
*/
|
||||
export function supersededWorkspaceExportUnpreservedCondition() {
|
||||
return and(eq(nativeRunFinalizations.phase, "terminal_failure"), eq(nativeRunFinalizations.failureCode, SUPERSEDED),
|
||||
isNotNull(nativeRunFinalizations.resultId), isNull(nativeRunFinalizations.leaseOwner),
|
||||
sql`${nativeRunFinalizations.failureDetail} ? 'workspaceFinalizeAttempt'`,
|
||||
eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "failed"),
|
||||
sql`${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,state}' = 'prepared'`,
|
||||
sql`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync' is null`,
|
||||
sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb)->'workspaceRestoreRecovery' is null`,
|
||||
sql`exists (select 1 from execution_workspaces workspace where workspace.company_id = ${heartbeatRuns.companyId}
|
||||
and workspace.id::text = ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,workspaceId}'
|
||||
and workspace.metadata#>>'{remoteWorkspace,writerRunId}' = ${heartbeatRuns.id}::text
|
||||
and workspace.metadata#>>'{remoteWorkspace,dirty}' = 'true')`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Records the retained-file obligation of an ended superseded export (the
|
||||
* system workspace recovery then re-exports it from the retained sandbox).
|
||||
* Returns when to check again: only while a just-ended run's sandbox has not
|
||||
* yet confirmed its stop, the one precondition that arrives by itself.
|
||||
*/
|
||||
export async function preserveSupersededWorkspaceExport(db: Db, runId: string,
|
||||
preserve: (db: Db, run: typeof heartbeatRuns.$inferSelect) => Promise<unknown>): Promise<{ recheckAt: number | null }> {
|
||||
const [row] = await db.select({ run: heartbeatRuns }).from(nativeRunFinalizations)
|
||||
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
|
||||
eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId)))
|
||||
.where(and(eq(nativeRunFinalizations.runId, runId), supersededWorkspaceExportUnpreservedCondition()));
|
||||
if (!row) return { recheckAt: null };
|
||||
const preserved = await preserve(db, row.run) as typeof heartbeatRuns.$inferSelect | undefined;
|
||||
if (preserved && hasRequiredWorkspaceRecovery(preserved.resultJson)) return { recheckAt: null };
|
||||
const reference = remoteWorkspaceRunReferenceSchema.safeParse(row.run.runnerProfileJson?.remoteWorkspace);
|
||||
if (!reference.success) return { recheckAt: null };
|
||||
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, row.run.companyId),
|
||||
eq(environmentLeases.id, reference.data.leaseId), eq(environmentLeases.heartbeatRunId, row.run.id),
|
||||
inArray(environmentLeases.status, ["active", "pending_cleanup", "released"])));
|
||||
const ended = row.run.finishedAt?.getTime() ?? 0;
|
||||
return { recheckAt: lease && !hasRemoteTerminationReceipt(lease) && Date.now() < ended + STOP_RECEIPT_WAIT_MS
|
||||
? Date.now() + STOP_RECEIPT_RECHECK_MS : null };
|
||||
}
|
||||
Reference in new issue
Block a user