Files
PaperClipAI/server/src/services/native-runtime/native-workspace-finalization-state.ts
T
DottaandClaude Opus 5.5 0477f622d6 fix(recovery): keep the sandbox that holds abandoned unsaved files out of replacement and cleanup (#21304)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes, and the workspace files live in
those sandboxes until a save
> - When a run cannot resume exactly, the fresh continuation fallback
(#21135) abandons the old workspace epoch and keeps its unsaved files
"recorded and unapplied" on the old sandbox
> - The abandonment clears the dirty flag, the writer and the location,
so the workspace looks clean. No code reads the abandonment receipt
> - A later failed resume can then replace the sandbox, and cleanup can
destroy it. That sandbox holds the only copy of the unsaved files
> - This pull request adds one shared hold. The task continues on a new
sandbox, and the old sandbox is parked until its files are saved,
discarded, or kept for 7 days
> - The benefit is that the fallback keeps its promise. Abandoned files
are not lost

## Linked Issues or Issue Description

Refs #21135
Refs #21297

**What happened?**
After the fresh continuation fallback abandoned a workspace epoch, the
receipt in `remoteWorkspaceAbandonedEpochs` was write-only.
`reusableSandboxReplacementIsSafe` saw a clean workspace with a
committed save and allowed replacement.
`retireReplacedReusableSandboxLease` then moved the old lease to
`pending_cleanup`, and the cleanup sweep destroyed the sandbox with the
unsaved files. `nativeWorkspaceExportPendingForLease` (the check that
the sweep, the retained orphan settlement and the scoped destroy paths
use) also answered "nothing pending".

**What did you expect?**
A sandbox that holds abandoned, unsaved files stays until the files are
saved, a person discards them, or a stated retention period ends.

**Steps to reproduce**
1. Abandon a remote workspace epoch whose location names sandbox L.
2. Let the next run fail to resume L.
3. The replacement decision says safe, L goes to `pending_cleanup`, and
the sweep destroys it.

## What Changed

- New `server/src/services/remote-workspace-abandoned-files-hold.ts`
holds the single predicate for "this lease holds abandoned files". It is
one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript
wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a
workspace in its company has an abandoned-epoch receipt whose
`location.providerLeaseId` equals the lease's provider lease id, and
that receipt has no `released` object. A receipt written by the current
release (no new fields) counts as unreleased.
- Replace and park. A failed resume of a held sandbox still lets the run
replace it, so the task continues at once on a new sandbox restored from
the last committed save. `retireReplacedReusableSandboxLease` parks the
old lease as `retained` with the `abandonedWorkspaceFilesParked` marker
(failure reason `abandoned_workspace_files_held`) instead of
`pending_cleanup`. `reusableLeaseCanBeResumed` and
`reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never
select it again. The check is a metadata read with no query, so the run
path does no extra work.
- Every path that can destroy a sandbox now honours the hold:
- `nativeWorkspaceExportPendingForLease` answers pending for a held
lease. This covers the pending cleanup sweep (defer), the orphaned
active lease sweep, the retained orphan settlement `heldBy`,
`destroyClaimedSandboxLease` (terminal workspace reaper
`destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the
environment delete path, and run lease release with `destroy`), the
`destroyRunLease` dispatcher, and the plugin sandbox teardown retry.
- The run lease settlement (`environment_run_lease_active`) is the one
caller that ignores the hold (`{ destructive: false }`). It never
destroys a sandbox. It stops the sandbox and keeps it, or it moves a
lease without a sandbox to `pending_cleanup`, where the sweep parks a
held lease. If it waited for the hold, a dead run's lease would stay
`active` and block the retry for the whole retention period.
- `destroyReusableSandboxLease` parks the lease and returns without a
provider call. This covers obsolete reusable lease cleanup, the destroy
after an expired or mismatched resume, release with `expired`, and the
sandbox driver's `destroyRunLease`.
- The `retryPendingSandboxTeardown` dispatcher refuses a destructive
teardown of a held lease. Stop-only cleanup still runs, because a stop
keeps the filesystem.
- The retained orphan listing (`alwaysHeldSql`) uses the same SQL
fragment, so held orphans are not polled.
- Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released`
with a reason (`recovered`, `discarded`, `retention_expired`) and an
actor. It moves the parked leases for that sandbox to `pending_cleanup`
when no other unreleased receipt holds them, and it writes a
`workspace.abandoned_files_released` activity entry on the task in plain
language. No export path for abandoned epochs exists today. The
`recovered` reason is the hook for one; this pull request adds no new
export mechanism.
- Retention expiry. New receipts carry `retainUntil` (7 days). An old
receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt`
includes the earliest retention end of any unreleased receipt. The
shared cleanup worker therefore wakes at that real deadline, and a
restart rebuilds it from the database. At the deadline the sweep runs
`expireAbandonedWorkspaceFileHolds`. This releases each due receipt as
an attributed system decision (actor `workspace-retention`) and logs the
held receipts and parked leases for each company. Each expiry always
writes a task entry: "Unsaved files from an interrupted run were kept
for N days, and their retention period has ended." The entry adds "They
are scheduled for removal." only when a parked lease was queued for
cleanup. A separate entry, "... kept for N days and have now been
removed.", is written only after the provider confirms the delete. Each
receipt is released in its own transaction. A failed receipt is logged
and skipped, and it is retried after a backoff (5 minutes, doubling, at
most 6 hours). The other receipts are still processed. The entries
reference a run or a task only if it still exists. If the task was
deleted, the decision goes to the server log only.
- Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that
an unreleased receipt holds goes back to `retained` and is parked by the
sweep. A sandbox that a release before this change already destroyed
cannot be recovered. Its receipt ends at its retention time like any
other receipt.
- Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch`
trims only released receipts and keeps the last 20 of them. It never
drops an unreleased receipt. A new abandonment of the same sandbox and
root replaces the unreleased receipt for that root. So a workspace holds
at most one unreleased receipt for each distinct sandbox and root
abandoned in the last 7 days.

## Verification

- `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19
tests). It follows the lease through the real acquisition path with a
failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the
real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`:
- `gives the next run a new sandbox at once and parks the held one,
which cleanup and later runs leave alone`
- `pulls a held lease already queued for cleanup back to retained, and
leaves it on the provider`
- `treats a receipt written before release fields existed as unreleased`
  - `records a retention expiry on the receipt`
- `hands the parked lease to normal cleanup once the files are
recovered` / `... discarded`
- `removes the files only after the retention period, and says removed
only after the provider confirms it`
  - `does not report removal when the provider delete fails`
- `records the end of retention without scheduling removal when another
receipt still holds the sandbox`
- `records the end of retention when it expires before the lease was
ever parked`
- `wakes the cleanup worker at the receipt's retention end, also after a
restart, and then goes quiet`
- `releases a receipt whose run was deleted and records it without the
run reference`
- `releases a receipt whose task was deleted without writing a task
entry`
- `keeps expiring later receipts when one fails, and retries the failed
one after a backoff`
- `keeps a held sandbox when the retained orphan settlement reviews it`
- `lists the sandbox for the retained orphan settlement again once the
receipt is released`
- `keeps holding the first sandbox after many later abandonments on
another sandbox`
- `replaces and cleans the sandbox as before when the receipt names a
different sandbox`
- `replaces and cleans the sandbox as before when no epoch was
abandoned`
- Red on the base revision (62d673cc3f): the park, upgrade, old-receipt,
expiry, many-abandonments and scheduling tests fail. The controls pass.
- Green: 19 of 19. Also
`environment-runtime-reusable-resume-replacement.test.ts` 17/17,
`environment-retained-orphan-sandbox.test.ts` 32/32,
`heartbeat-pending-cleanup-sweep.test.ts` 51/51,
`fresh-continuation-fallback.test.ts` 58/58.
- `tsc --noEmit -p server/tsconfig.json` and a tsc pass over the new
test file: clean. `environment-cleanup-work.test.ts` +
`heartbeat-pending-cleanup-sweep.test.ts`: 80/80.

- No existing test expectation was changed. Three CI failures on bb43f52
came from this change. Two came from the hold blocking the run lease
settlement: `ownership-reconciliation-report.test.ts > releases the
lease and frees the workspace for the retry once a preparation-only
epoch ends` and `remote-workspace-save-concurrency.test.ts > lets a
retry prepare the workspace again after its first writer was killed
during preparation`. The third, `native-session-executor.test.ts >
restores a warm backup into a created lease only when a settled restore
hold retained the stopped source`, came from a mocked database that has
no `execute`; the predicate now runs through `select`. All three pass
locally now. So do `heartbeat-orphaned-active-lease-sweep`,
`environment-run-lease-ownership`, `native-workspace-warm-reuse` and
`remote-workspaces.integration`.

## Risks

- Cost of a parked sandbox. Parking makes no provider call and changes
only the lease row. A sandbox that could not be resumed is not running.
A reusable sandbox that was released is already stopped and kept. In
both cases the cost is stopped storage for up to 7 days. One case is
different: if a destructive path finds a held warm sandbox that is still
running (`retained` after a keep-running release), the sandbox stays
running until the receipt is released. The expiry log line counts parked
leases for each company, so this cost can be seen.
- Run lease release for an `ephemeral` lease calls the provider's
release. Some providers delete the sandbox on that call. A held sandbox
normally has a reusable lease, so this pull request does not gate that
call.
- No schema change. The receipt is JSON in
`execution_workspaces.metadata`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:00:20 -06:00

148 lines
10 KiB
TypeScript

import { environmentLeases, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
import { and, eq, exists, sql } from "drizzle-orm";
import { hasRequiredWorkspaceRecovery, nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js";
import { recordRequiredWorkspaceRestoreFailure } from "../legacy-execution-recovery.js";
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
import { abandonedWorkspaceFilesHeldOnLease } from "../remote-workspace-abandoned-files-hold.js";
import { matchPreparedRemoteWorkspaceSource, preparedRemoteWorkspaceSource, type WorkspaceSourceLease } from "../remote-workspace-lifecycle.js";
/** File-only recovery for setup that never reached native admission. The live
* preparer must still own its controller; orphan repair accepts terminal runs
* only. This records no provider result and never starts a provider turn. */
export async function preserveRemoteWorkspacePreparationFailure(db: Db, lease: WorkspaceSourceLease,
options: { assertOwned?: () => Promise<void> } = {}) {
const candidate = await preparedRemoteWorkspaceSource(db, lease);
if (!candidate || candidate.run.runtimeMode !== "legacy") return null;
// This callback renews the run on the liveness connection. Finish it before
// locking that row; the captured owner is rechecked under the lock below.
await options.assertOwned?.();
return db.transaction(async tx => {
await tx.execute(sql`select set_config('statement_timeout', '15000', true), set_config('lock_timeout', '1000', true)`);
const [task] = await tx.select({ id: issues.id }).from(issues).where(and(
eq(issues.companyId, lease.companyId), eq(issues.id, candidate.issueId))).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, lease.companyId), eq(heartbeatRuns.id, candidate.run.id))).for("update");
if (!task || !run || run.runtimeMode !== "legacy" || run.runnerProfileJson?.nativeExecutionInput != null
|| options.assertOwned && (run.controllerBootId !== candidate.run.controllerBootId
|| run.controllerBootId !== null && (!run.controllerLeaseExpiresAt || run.controllerLeaseExpiresAt.getTime() <= Date.now()))
|| (!options.assertOwned && (!["failed", "cancelled", "timed_out", "interrupted"].includes(run.status)
|| run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt.getTime() > Date.now()))) return null;
const [coordinator] = await tx.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.where(and(eq(nativeRunFinalizations.companyId, lease.companyId), eq(nativeRunFinalizations.runId, run.id)));
if (coordinator) return null;
await tx.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, lease.companyId), eq(executionWorkspaces.id, candidate.reference.workspaceId))).for("update");
await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, lease.companyId), eq(environmentLeases.id, lease.id))).for("update");
const current = await preparedRemoteWorkspaceSource(tx as unknown as Db, lease);
if (!current || current.issueId !== task.id || current.run.id !== run.id) return null;
if (options.assertOwned && run.controllerBootId !== null
&& (!run.controllerLeaseExpiresAt || run.controllerLeaseExpiresAt.getTime() <= Date.now())) return null;
if (hasRequiredWorkspaceRecovery(run.resultJson)) return run;
return await recordRequiredWorkspaceRestoreFailure(tx as unknown as Db, run,
{ workspaceRestoreFailure: "restore_failed" }, current.lease);
});
}
function nativeWorkspaceRunProjection() {
const input = sql`${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'`;
return {
id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
issueId: heartbeatRuns.issueId, nativeIssueId: heartbeatRuns.nativeIssueId,
runtimeMode: heartbeatRuns.runtimeMode, driverKind: heartbeatRuns.driverKind,
reference: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'`,
observedRemoteWorkspace: sql<boolean>`${heartbeatRuns.status} = 'failed'
and ${heartbeatRuns.runnerProfileJson}->'remoteWorkspace' is not null
and ${heartbeatRuns.runnerProfileJson}->'remoteWorkspace' <> 'null'::jsonb
and exists (select 1 from ${nativeRunFinalizations}
where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId}
and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
and ${nativeRunFinalizations.issueId} = ${heartbeatRuns.nativeIssueId}
and ${nativeRunFinalizations.phase} = 'observed' and ${nativeRunFinalizations.resultId} is null)`,
executionScope: sql<Record<string, unknown>>`jsonb_build_object(
'schema', ${input}->'schema', 'binding', ${input}->'binding', 'workspace', ${input}->'workspace',
'provider', jsonb_build_object('kind', ${input}#>'{provider,kind}', 'bridgeRevision', ${input}#>'{provider,bridgeRevision}'))`,
controllerBootId: heartbeatRuns.controllerBootId,
contextIssueId: sql<string | null>`${heartbeatRuns.contextSnapshot}->>'issueId'`,
remoteReference: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'`,
};
}
function nativeWorkspaceSourcePending(run: Pick<typeof heartbeatRuns.$inferSelect,
"id" | "companyId" | "agentId" | "issueId" | "nativeIssueId" | "runtimeMode" | "driverKind"> & {
reference: unknown; executionScope: Record<string, unknown>; observedRemoteWorkspace: boolean;
}, lease: WorkspaceSourceLease) {
if (run.runtimeMode !== "native") return false;
if (run.observedRemoteWorkspace) return true;
if (nativeRunHasNoWorkspaceAccess({ ...run, runnerProfileJson: { nativeExecutionInput: run.executionScope } })) return false;
const reference = readNativeWorkspaceSyncReference(run.reference);
return reference?.state === "prepared" && reference.leaseId === lease.id
&& reference.providerLeaseId === lease.providerLeaseId;
}
export async function nativeWorkspaceExportPendingForLease(db: Db, lease: WorkspaceSourceLease,
options: { destructive?: boolean } = {}): Promise<boolean> {
if (!lease.provider || lease.provider === "local" || !lease.providerLeaseId) return false;
const held = () => options.destructive === false ? Promise.resolve(false) : abandonedWorkspaceFilesHeldOnLease(db, lease);
if (!lease.heartbeatRunId) return await held();
const [run] = await db.select(nativeWorkspaceRunProjection()).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, lease.heartbeatRunId), eq(heartbeatRuns.companyId, lease.companyId),
eq(heartbeatRuns.runtimeMode, "native"),
)).limit(1);
if (run?.observedRemoteWorkspace) return true;
if (await preparedRemoteWorkspaceSource(db, lease)) return true;
if (run && nativeWorkspaceSourcePending(run, lease)) return true;
return await held();
}
export async function nativeWorkspaceExportPendingForResource(db: Db, lease: WorkspaceSourceLease, resumingRunId: string | null) {
if (!lease.provider || !lease.providerLeaseId || lease.provider === "local") return false;
const sources = await db.select({ lease: { id: environmentLeases.id, companyId: environmentLeases.companyId,
heartbeatRunId: environmentLeases.heartbeatRunId, provider: environmentLeases.provider, providerLeaseId: environmentLeases.providerLeaseId,
issueId: environmentLeases.issueId, executionWorkspaceId: environmentLeases.executionWorkspaceId,
environmentId: environmentLeases.environmentId, metadata: environmentLeases.metadata },
run: nativeWorkspaceRunProjection(),
workspace: { id: executionWorkspaces.id, companyId: executionWorkspaces.companyId, sourceIssueId: executionWorkspaces.sourceIssueId,
mode: executionWorkspaces.mode, metadata: executionWorkspaces.metadata },
}).from(environmentLeases).innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, environmentLeases.companyId),
eq(heartbeatRuns.id, environmentLeases.heartbeatRunId))).leftJoin(executionWorkspaces, and(
eq(executionWorkspaces.companyId, heartbeatRuns.companyId),
sql`${executionWorkspaces.id}::text = ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,workspaceId}'`,
)).where(and(eq(environmentLeases.provider, lease.provider), eq(environmentLeases.providerLeaseId, lease.providerLeaseId)));
for (const source of sources) {
if (source.lease.id === lease.id && source.lease.heartbeatRunId === resumingRunId) continue;
const remote = matchPreparedRemoteWorkspaceSource({ lease: source.lease, run: { ...source.run,
issueId: source.run.nativeIssueId ?? source.run.issueId ?? source.run.contextIssueId,
reference: source.run.remoteReference, legacyReference: source.run.reference } }, source.workspace);
if (remote || nativeWorkspaceSourcePending(source.run, source.lease)) return true;
}
return false;
}
/**
* An accepted result transfers recovery to the workspace finalizer. A dead
* provider PID cannot authorize discarding its unfinished workspace suffix,
* including the interval before physical copyback acquires its owner receipt.
* The heartbeat's display phase and arbitrary profile JSON are not evidence.
*/
export function pendingNativeWorkspaceFinalizationCondition(db: Db) {
return and(
eq(heartbeatRuns.runtimeMode, "native"),
exists(db.select({ runId: nativeRunFinalizations.runId })
.from(nativeRunFinalizations)
.innerJoin(nativeRunResults, and(
eq(nativeRunResults.id, nativeRunFinalizations.resultId),
eq(nativeRunResults.runId, nativeRunFinalizations.runId),
eq(nativeRunResults.companyId, nativeRunFinalizations.companyId),
eq(nativeRunResults.issueId, nativeRunFinalizations.issueId),
eq(nativeRunResults.schemaStatus, "accepted"),
))
.where(and(
eq(nativeRunFinalizations.runId, heartbeatRuns.id),
eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
eq(nativeRunFinalizations.issueId, heartbeatRuns.nativeIssueId),
eq(nativeRunFinalizations.phase, "workspace_finalizing"),
))),
)!;
}