mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in remote sandboxes, and the workspace files live in those sandboxes until a save > - When a run cannot resume exactly, the fresh continuation fallback (#21135) abandons the old workspace epoch and keeps its unsaved files "recorded and unapplied" on the old sandbox > - The abandonment clears the dirty flag, the writer and the location, so the workspace looks clean. No code reads the abandonment receipt > - A later failed resume can then replace the sandbox, and cleanup can destroy it. That sandbox holds the only copy of the unsaved files > - This pull request adds one shared hold. The task continues on a new sandbox, and the old sandbox is parked until its files are saved, discarded, or kept for 7 days > - The benefit is that the fallback keeps its promise. Abandoned files are not lost ## Linked Issues or Issue Description Refs #21135 Refs #21297 **What happened?** After the fresh continuation fallback abandoned a workspace epoch, the receipt in `remoteWorkspaceAbandonedEpochs` was write-only. `reusableSandboxReplacementIsSafe` saw a clean workspace with a committed save and allowed replacement. `retireReplacedReusableSandboxLease` then moved the old lease to `pending_cleanup`, and the cleanup sweep destroyed the sandbox with the unsaved files. `nativeWorkspaceExportPendingForLease` (the check that the sweep, the retained orphan settlement and the scoped destroy paths use) also answered "nothing pending". **What did you expect?** A sandbox that holds abandoned, unsaved files stays until the files are saved, a person discards them, or a stated retention period ends. **Steps to reproduce** 1. Abandon a remote workspace epoch whose location names sandbox L. 2. Let the next run fail to resume L. 3. The replacement decision says safe, L goes to `pending_cleanup`, and the sweep destroys it. ## What Changed - New `server/src/services/remote-workspace-abandoned-files-hold.ts` holds the single predicate for "this lease holds abandoned files". It is one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a workspace in its company has an abandoned-epoch receipt whose `location.providerLeaseId` equals the lease's provider lease id, and that receipt has no `released` object. A receipt written by the current release (no new fields) counts as unreleased. - Replace and park. A failed resume of a held sandbox still lets the run replace it, so the task continues at once on a new sandbox restored from the last committed save. `retireReplacedReusableSandboxLease` parks the old lease as `retained` with the `abandonedWorkspaceFilesParked` marker (failure reason `abandoned_workspace_files_held`) instead of `pending_cleanup`. `reusableLeaseCanBeResumed` and `reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never select it again. The check is a metadata read with no query, so the run path does no extra work. - Every path that can destroy a sandbox now honours the hold: - `nativeWorkspaceExportPendingForLease` answers pending for a held lease. This covers the pending cleanup sweep (defer), the orphaned active lease sweep, the retained orphan settlement `heldBy`, `destroyClaimedSandboxLease` (terminal workspace reaper `destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the environment delete path, and run lease release with `destroy`), the `destroyRunLease` dispatcher, and the plugin sandbox teardown retry. - The run lease settlement (`environment_run_lease_active`) is the one caller that ignores the hold (`{ destructive: false }`). It never destroys a sandbox. It stops the sandbox and keeps it, or it moves a lease without a sandbox to `pending_cleanup`, where the sweep parks a held lease. If it waited for the hold, a dead run's lease would stay `active` and block the retry for the whole retention period. - `destroyReusableSandboxLease` parks the lease and returns without a provider call. This covers obsolete reusable lease cleanup, the destroy after an expired or mismatched resume, release with `expired`, and the sandbox driver's `destroyRunLease`. - The `retryPendingSandboxTeardown` dispatcher refuses a destructive teardown of a held lease. Stop-only cleanup still runs, because a stop keeps the filesystem. - The retained orphan listing (`alwaysHeldSql`) uses the same SQL fragment, so held orphans are not polled. - Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released` with a reason (`recovered`, `discarded`, `retention_expired`) and an actor. It moves the parked leases for that sandbox to `pending_cleanup` when no other unreleased receipt holds them, and it writes a `workspace.abandoned_files_released` activity entry on the task in plain language. No export path for abandoned epochs exists today. The `recovered` reason is the hook for one; this pull request adds no new export mechanism. - Retention expiry. New receipts carry `retainUntil` (7 days). An old receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt` includes the earliest retention end of any unreleased receipt. The shared cleanup worker therefore wakes at that real deadline, and a restart rebuilds it from the database. At the deadline the sweep runs `expireAbandonedWorkspaceFileHolds`. This releases each due receipt as an attributed system decision (actor `workspace-retention`) and logs the held receipts and parked leases for each company. Each expiry always writes a task entry: "Unsaved files from an interrupted run were kept for N days, and their retention period has ended." The entry adds "They are scheduled for removal." only when a parked lease was queued for cleanup. A separate entry, "... kept for N days and have now been removed.", is written only after the provider confirms the delete. Each receipt is released in its own transaction. A failed receipt is logged and skipped, and it is retried after a backoff (5 minutes, doubling, at most 6 hours). The other receipts are still processed. The entries reference a run or a task only if it still exists. If the task was deleted, the decision goes to the server log only. - Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that an unreleased receipt holds goes back to `retained` and is parked by the sweep. A sandbox that a release before this change already destroyed cannot be recovered. Its receipt ends at its retention time like any other receipt. - Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch` trims only released receipts and keeps the last 20 of them. It never drops an unreleased receipt. A new abandonment of the same sandbox and root replaces the unreleased receipt for that root. So a workspace holds at most one unreleased receipt for each distinct sandbox and root abandoned in the last 7 days. ## Verification - `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19 tests). It follows the lease through the real acquisition path with a failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`: - `gives the next run a new sandbox at once and parks the held one, which cleanup and later runs leave alone` - `pulls a held lease already queued for cleanup back to retained, and leaves it on the provider` - `treats a receipt written before release fields existed as unreleased` - `records a retention expiry on the receipt` - `hands the parked lease to normal cleanup once the files are recovered` / `... discarded` - `removes the files only after the retention period, and says removed only after the provider confirms it` - `does not report removal when the provider delete fails` - `records the end of retention without scheduling removal when another receipt still holds the sandbox` - `records the end of retention when it expires before the lease was ever parked` - `wakes the cleanup worker at the receipt's retention end, also after a restart, and then goes quiet` - `releases a receipt whose run was deleted and records it without the run reference` - `releases a receipt whose task was deleted without writing a task entry` - `keeps expiring later receipts when one fails, and retries the failed one after a backoff` - `keeps a held sandbox when the retained orphan settlement reviews it` - `lists the sandbox for the retained orphan settlement again once the receipt is released` - `keeps holding the first sandbox after many later abandonments on another sandbox` - `replaces and cleans the sandbox as before when the receipt names a different sandbox` - `replaces and cleans the sandbox as before when no epoch was abandoned` - Red on the base revision (62d673cc3f): the park, upgrade, old-receipt, expiry, many-abandonments and scheduling tests fail. The controls pass. - Green: 19 of 19. Also `environment-runtime-reusable-resume-replacement.test.ts` 17/17, `environment-retained-orphan-sandbox.test.ts` 32/32, `heartbeat-pending-cleanup-sweep.test.ts` 51/51, `fresh-continuation-fallback.test.ts` 58/58. - `tsc --noEmit -p server/tsconfig.json` and a tsc pass over the new test file: clean. `environment-cleanup-work.test.ts` + `heartbeat-pending-cleanup-sweep.test.ts`: 80/80. - No existing test expectation was changed. Three CI failures onbb43f52came from this change. Two came from the hold blocking the run lease settlement: `ownership-reconciliation-report.test.ts > releases the lease and frees the workspace for the retry once a preparation-only epoch ends` and `remote-workspace-save-concurrency.test.ts > lets a retry prepare the workspace again after its first writer was killed during preparation`. The third, `native-session-executor.test.ts > restores a warm backup into a created lease only when a settled restore hold retained the stopped source`, came from a mocked database that has no `execute`; the predicate now runs through `select`. All three pass locally now. So do `heartbeat-orphaned-active-lease-sweep`, `environment-run-lease-ownership`, `native-workspace-warm-reuse` and `remote-workspaces.integration`. ## Risks - Cost of a parked sandbox. Parking makes no provider call and changes only the lease row. A sandbox that could not be resumed is not running. A reusable sandbox that was released is already stopped and kept. In both cases the cost is stopped storage for up to 7 days. One case is different: if a destructive path finds a held warm sandbox that is still running (`retained` after a keep-running release), the sandbox stays running until the receipt is released. The expiry log line counts parked leases for each company, so this cost can be seen. - Run lease release for an `ephemeral` lease calls the provider's release. Some providers delete the sandbox on that call. A held sandbox normally has a reusable lease, so this pull request does not gate that call. - No schema change. The receipt is JSON in `execution_workspaces.metadata`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edit) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
148 lines
10 KiB
TypeScript
148 lines
10 KiB
TypeScript
import { environmentLeases, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
|
|
import { and, eq, exists, sql } from "drizzle-orm";
|
|
import { hasRequiredWorkspaceRecovery, nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js";
|
|
import { recordRequiredWorkspaceRestoreFailure } from "../legacy-execution-recovery.js";
|
|
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
|
|
import { abandonedWorkspaceFilesHeldOnLease } from "../remote-workspace-abandoned-files-hold.js";
|
|
import { matchPreparedRemoteWorkspaceSource, preparedRemoteWorkspaceSource, type WorkspaceSourceLease } from "../remote-workspace-lifecycle.js";
|
|
|
|
/** File-only recovery for setup that never reached native admission. The live
|
|
* preparer must still own its controller; orphan repair accepts terminal runs
|
|
* only. This records no provider result and never starts a provider turn. */
|
|
export async function preserveRemoteWorkspacePreparationFailure(db: Db, lease: WorkspaceSourceLease,
|
|
options: { assertOwned?: () => Promise<void> } = {}) {
|
|
const candidate = await preparedRemoteWorkspaceSource(db, lease);
|
|
if (!candidate || candidate.run.runtimeMode !== "legacy") return null;
|
|
// This callback renews the run on the liveness connection. Finish it before
|
|
// locking that row; the captured owner is rechecked under the lock below.
|
|
await options.assertOwned?.();
|
|
return db.transaction(async tx => {
|
|
await tx.execute(sql`select set_config('statement_timeout', '15000', true), set_config('lock_timeout', '1000', true)`);
|
|
const [task] = await tx.select({ id: issues.id }).from(issues).where(and(
|
|
eq(issues.companyId, lease.companyId), eq(issues.id, candidate.issueId))).for("update");
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, lease.companyId), eq(heartbeatRuns.id, candidate.run.id))).for("update");
|
|
if (!task || !run || run.runtimeMode !== "legacy" || run.runnerProfileJson?.nativeExecutionInput != null
|
|
|| options.assertOwned && (run.controllerBootId !== candidate.run.controllerBootId
|
|
|| run.controllerBootId !== null && (!run.controllerLeaseExpiresAt || run.controllerLeaseExpiresAt.getTime() <= Date.now()))
|
|
|| (!options.assertOwned && (!["failed", "cancelled", "timed_out", "interrupted"].includes(run.status)
|
|
|| run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt.getTime() > Date.now()))) return null;
|
|
const [coordinator] = await tx.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
|
|
.where(and(eq(nativeRunFinalizations.companyId, lease.companyId), eq(nativeRunFinalizations.runId, run.id)));
|
|
if (coordinator) return null;
|
|
await tx.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, lease.companyId), eq(executionWorkspaces.id, candidate.reference.workspaceId))).for("update");
|
|
await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, lease.companyId), eq(environmentLeases.id, lease.id))).for("update");
|
|
const current = await preparedRemoteWorkspaceSource(tx as unknown as Db, lease);
|
|
if (!current || current.issueId !== task.id || current.run.id !== run.id) return null;
|
|
if (options.assertOwned && run.controllerBootId !== null
|
|
&& (!run.controllerLeaseExpiresAt || run.controllerLeaseExpiresAt.getTime() <= Date.now())) return null;
|
|
if (hasRequiredWorkspaceRecovery(run.resultJson)) return run;
|
|
return await recordRequiredWorkspaceRestoreFailure(tx as unknown as Db, run,
|
|
{ workspaceRestoreFailure: "restore_failed" }, current.lease);
|
|
});
|
|
}
|
|
|
|
function nativeWorkspaceRunProjection() {
|
|
const input = sql`${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'`;
|
|
return {
|
|
id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
|
issueId: heartbeatRuns.issueId, nativeIssueId: heartbeatRuns.nativeIssueId,
|
|
runtimeMode: heartbeatRuns.runtimeMode, driverKind: heartbeatRuns.driverKind,
|
|
reference: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'`,
|
|
observedRemoteWorkspace: sql<boolean>`${heartbeatRuns.status} = 'failed'
|
|
and ${heartbeatRuns.runnerProfileJson}->'remoteWorkspace' is not null
|
|
and ${heartbeatRuns.runnerProfileJson}->'remoteWorkspace' <> 'null'::jsonb
|
|
and exists (select 1 from ${nativeRunFinalizations}
|
|
where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId}
|
|
and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
|
|
and ${nativeRunFinalizations.issueId} = ${heartbeatRuns.nativeIssueId}
|
|
and ${nativeRunFinalizations.phase} = 'observed' and ${nativeRunFinalizations.resultId} is null)`,
|
|
executionScope: sql<Record<string, unknown>>`jsonb_build_object(
|
|
'schema', ${input}->'schema', 'binding', ${input}->'binding', 'workspace', ${input}->'workspace',
|
|
'provider', jsonb_build_object('kind', ${input}#>'{provider,kind}', 'bridgeRevision', ${input}#>'{provider,bridgeRevision}'))`,
|
|
controllerBootId: heartbeatRuns.controllerBootId,
|
|
contextIssueId: sql<string | null>`${heartbeatRuns.contextSnapshot}->>'issueId'`,
|
|
remoteReference: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'`,
|
|
};
|
|
}
|
|
|
|
function nativeWorkspaceSourcePending(run: Pick<typeof heartbeatRuns.$inferSelect,
|
|
"id" | "companyId" | "agentId" | "issueId" | "nativeIssueId" | "runtimeMode" | "driverKind"> & {
|
|
reference: unknown; executionScope: Record<string, unknown>; observedRemoteWorkspace: boolean;
|
|
}, lease: WorkspaceSourceLease) {
|
|
if (run.runtimeMode !== "native") return false;
|
|
if (run.observedRemoteWorkspace) return true;
|
|
if (nativeRunHasNoWorkspaceAccess({ ...run, runnerProfileJson: { nativeExecutionInput: run.executionScope } })) return false;
|
|
const reference = readNativeWorkspaceSyncReference(run.reference);
|
|
return reference?.state === "prepared" && reference.leaseId === lease.id
|
|
&& reference.providerLeaseId === lease.providerLeaseId;
|
|
}
|
|
|
|
export async function nativeWorkspaceExportPendingForLease(db: Db, lease: WorkspaceSourceLease,
|
|
options: { destructive?: boolean } = {}): Promise<boolean> {
|
|
if (!lease.provider || lease.provider === "local" || !lease.providerLeaseId) return false;
|
|
const held = () => options.destructive === false ? Promise.resolve(false) : abandonedWorkspaceFilesHeldOnLease(db, lease);
|
|
if (!lease.heartbeatRunId) return await held();
|
|
const [run] = await db.select(nativeWorkspaceRunProjection()).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, lease.heartbeatRunId), eq(heartbeatRuns.companyId, lease.companyId),
|
|
eq(heartbeatRuns.runtimeMode, "native"),
|
|
)).limit(1);
|
|
if (run?.observedRemoteWorkspace) return true;
|
|
if (await preparedRemoteWorkspaceSource(db, lease)) return true;
|
|
if (run && nativeWorkspaceSourcePending(run, lease)) return true;
|
|
return await held();
|
|
}
|
|
|
|
export async function nativeWorkspaceExportPendingForResource(db: Db, lease: WorkspaceSourceLease, resumingRunId: string | null) {
|
|
if (!lease.provider || !lease.providerLeaseId || lease.provider === "local") return false;
|
|
const sources = await db.select({ lease: { id: environmentLeases.id, companyId: environmentLeases.companyId,
|
|
heartbeatRunId: environmentLeases.heartbeatRunId, provider: environmentLeases.provider, providerLeaseId: environmentLeases.providerLeaseId,
|
|
issueId: environmentLeases.issueId, executionWorkspaceId: environmentLeases.executionWorkspaceId,
|
|
environmentId: environmentLeases.environmentId, metadata: environmentLeases.metadata },
|
|
run: nativeWorkspaceRunProjection(),
|
|
workspace: { id: executionWorkspaces.id, companyId: executionWorkspaces.companyId, sourceIssueId: executionWorkspaces.sourceIssueId,
|
|
mode: executionWorkspaces.mode, metadata: executionWorkspaces.metadata },
|
|
}).from(environmentLeases).innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, environmentLeases.companyId),
|
|
eq(heartbeatRuns.id, environmentLeases.heartbeatRunId))).leftJoin(executionWorkspaces, and(
|
|
eq(executionWorkspaces.companyId, heartbeatRuns.companyId),
|
|
sql`${executionWorkspaces.id}::text = ${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,workspaceId}'`,
|
|
)).where(and(eq(environmentLeases.provider, lease.provider), eq(environmentLeases.providerLeaseId, lease.providerLeaseId)));
|
|
for (const source of sources) {
|
|
if (source.lease.id === lease.id && source.lease.heartbeatRunId === resumingRunId) continue;
|
|
const remote = matchPreparedRemoteWorkspaceSource({ lease: source.lease, run: { ...source.run,
|
|
issueId: source.run.nativeIssueId ?? source.run.issueId ?? source.run.contextIssueId,
|
|
reference: source.run.remoteReference, legacyReference: source.run.reference } }, source.workspace);
|
|
if (remote || nativeWorkspaceSourcePending(source.run, source.lease)) return true;
|
|
}
|
|
return false;
|
|
}
|
|
|
|
/**
|
|
* An accepted result transfers recovery to the workspace finalizer. A dead
|
|
* provider PID cannot authorize discarding its unfinished workspace suffix,
|
|
* including the interval before physical copyback acquires its owner receipt.
|
|
* The heartbeat's display phase and arbitrary profile JSON are not evidence.
|
|
*/
|
|
export function pendingNativeWorkspaceFinalizationCondition(db: Db) {
|
|
return and(
|
|
eq(heartbeatRuns.runtimeMode, "native"),
|
|
exists(db.select({ runId: nativeRunFinalizations.runId })
|
|
.from(nativeRunFinalizations)
|
|
.innerJoin(nativeRunResults, and(
|
|
eq(nativeRunResults.id, nativeRunFinalizations.resultId),
|
|
eq(nativeRunResults.runId, nativeRunFinalizations.runId),
|
|
eq(nativeRunResults.companyId, nativeRunFinalizations.companyId),
|
|
eq(nativeRunResults.issueId, nativeRunFinalizations.issueId),
|
|
eq(nativeRunResults.schemaStatus, "accepted"),
|
|
))
|
|
.where(and(
|
|
eq(nativeRunFinalizations.runId, heartbeatRuns.id),
|
|
eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
|
|
eq(nativeRunFinalizations.issueId, heartbeatRuns.nativeIssueId),
|
|
eq(nativeRunFinalizations.phase, "workspace_finalizing"),
|
|
))),
|
|
)!;
|
|
}
|