fix(recovery): keep the sandbox that holds abandoned unsaved files out of replacement and cleanup (#21304)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes, and the workspace files live in
those sandboxes until a save
> - When a run cannot resume exactly, the fresh continuation fallback
(#21135) abandons the old workspace epoch and keeps its unsaved files
"recorded and unapplied" on the old sandbox
> - The abandonment clears the dirty flag, the writer and the location,
so the workspace looks clean. No code reads the abandonment receipt
> - A later failed resume can then replace the sandbox, and cleanup can
destroy it. That sandbox holds the only copy of the unsaved files
> - This pull request adds one shared hold. The task continues on a new
sandbox, and the old sandbox is parked until its files are saved,
discarded, or kept for 7 days
> - The benefit is that the fallback keeps its promise. Abandoned files
are not lost

## Linked Issues or Issue Description

Refs #21135
Refs #21297

**What happened?**
After the fresh continuation fallback abandoned a workspace epoch, the
receipt in `remoteWorkspaceAbandonedEpochs` was write-only.
`reusableSandboxReplacementIsSafe` saw a clean workspace with a
committed save and allowed replacement.
`retireReplacedReusableSandboxLease` then moved the old lease to
`pending_cleanup`, and the cleanup sweep destroyed the sandbox with the
unsaved files. `nativeWorkspaceExportPendingForLease` (the check that
the sweep, the retained orphan settlement and the scoped destroy paths
use) also answered "nothing pending".

**What did you expect?**
A sandbox that holds abandoned, unsaved files stays until the files are
saved, a person discards them, or a stated retention period ends.

**Steps to reproduce**
1. Abandon a remote workspace epoch whose location names sandbox L.
2. Let the next run fail to resume L.
3. The replacement decision says safe, L goes to `pending_cleanup`, and
the sweep destroys it.

## What Changed

- New `server/src/services/remote-workspace-abandoned-files-hold.ts`
holds the single predicate for "this lease holds abandoned files". It is
one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript
wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a
workspace in its company has an abandoned-epoch receipt whose
`location.providerLeaseId` equals the lease's provider lease id, and
that receipt has no `released` object. A receipt written by the current
release (no new fields) counts as unreleased.
- Replace and park. A failed resume of a held sandbox still lets the run
replace it, so the task continues at once on a new sandbox restored from
the last committed save. `retireReplacedReusableSandboxLease` parks the
old lease as `retained` with the `abandonedWorkspaceFilesParked` marker
(failure reason `abandoned_workspace_files_held`) instead of
`pending_cleanup`. `reusableLeaseCanBeResumed` and
`reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never
select it again. The check is a metadata read with no query, so the run
path does no extra work.
- Every path that can destroy a sandbox now honours the hold:
- `nativeWorkspaceExportPendingForLease` answers pending for a held
lease. This covers the pending cleanup sweep (defer), the orphaned
active lease sweep, the retained orphan settlement `heldBy`,
`destroyClaimedSandboxLease` (terminal workspace reaper
`destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the
environment delete path, and run lease release with `destroy`), the
`destroyRunLease` dispatcher, and the plugin sandbox teardown retry.
- The run lease settlement (`environment_run_lease_active`) is the one
caller that ignores the hold (`{ destructive: false }`). It never
destroys a sandbox. It stops the sandbox and keeps it, or it moves a
lease without a sandbox to `pending_cleanup`, where the sweep parks a
held lease. If it waited for the hold, a dead run's lease would stay
`active` and block the retry for the whole retention period.
- `destroyReusableSandboxLease` parks the lease and returns without a
provider call. This covers obsolete reusable lease cleanup, the destroy
after an expired or mismatched resume, release with `expired`, and the
sandbox driver's `destroyRunLease`.
- The `retryPendingSandboxTeardown` dispatcher refuses a destructive
teardown of a held lease. Stop-only cleanup still runs, because a stop
keeps the filesystem.
- The retained orphan listing (`alwaysHeldSql`) uses the same SQL
fragment, so held orphans are not polled.
- Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released`
with a reason (`recovered`, `discarded`, `retention_expired`) and an
actor. It moves the parked leases for that sandbox to `pending_cleanup`
when no other unreleased receipt holds them, and it writes a
`workspace.abandoned_files_released` activity entry on the task in plain
language. No export path for abandoned epochs exists today. The
`recovered` reason is the hook for one; this pull request adds no new
export mechanism.
- Retention expiry. New receipts carry `retainUntil` (7 days). An old
receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt`
includes the earliest retention end of any unreleased receipt. The
shared cleanup worker therefore wakes at that real deadline, and a
restart rebuilds it from the database. At the deadline the sweep runs
`expireAbandonedWorkspaceFileHolds`. This releases each due receipt as
an attributed system decision (actor `workspace-retention`) and logs the
held receipts and parked leases for each company. Each expiry always
writes a task entry: "Unsaved files from an interrupted run were kept
for N days, and their retention period has ended." The entry adds "They
are scheduled for removal." only when a parked lease was queued for
cleanup. A separate entry, "... kept for N days and have now been
removed.", is written only after the provider confirms the delete. Each
receipt is released in its own transaction. A failed receipt is logged
and skipped, and it is retried after a backoff (5 minutes, doubling, at
most 6 hours). The other receipts are still processed. The entries
reference a run or a task only if it still exists. If the task was
deleted, the decision goes to the server log only.
- Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that
an unreleased receipt holds goes back to `retained` and is parked by the
sweep. A sandbox that a release before this change already destroyed
cannot be recovered. Its receipt ends at its retention time like any
other receipt.
- Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch`
trims only released receipts and keeps the last 20 of them. It never
drops an unreleased receipt. A new abandonment of the same sandbox and
root replaces the unreleased receipt for that root. So a workspace holds
at most one unreleased receipt for each distinct sandbox and root
abandoned in the last 7 days.

## Verification

- `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19
tests). It follows the lease through the real acquisition path with a
failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the
real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`:
- `gives the next run a new sandbox at once and parks the held one,
which cleanup and later runs leave alone`
- `pulls a held lease already queued for cleanup back to retained, and
leaves it on the provider`
- `treats a receipt written before release fields existed as unreleased`
  - `records a retention expiry on the receipt`
- `hands the parked lease to normal cleanup once the files are
recovered` / `... discarded`
- `removes the files only after the retention period, and says removed
only after the provider confirms it`
  - `does not report removal when the provider delete fails`
- `records the end of retention without scheduling removal when another
receipt still holds the sandbox`
- `records the end of retention when it expires before the lease was
ever parked`
- `wakes the cleanup worker at the receipt's retention end, also after a
restart, and then goes quiet`
- `releases a receipt whose run was deleted and records it without the
run reference`
- `releases a receipt whose task was deleted without writing a task
entry`
- `keeps expiring later receipts when one fails, and retries the failed
one after a backoff`
- `keeps a held sandbox when the retained orphan settlement reviews it`
- `lists the sandbox for the retained orphan settlement again once the
receipt is released`
- `keeps holding the first sandbox after many later abandonments on
another sandbox`
- `replaces and cleans the sandbox as before when the receipt names a
different sandbox`
- `replaces and cleans the sandbox as before when no epoch was
abandoned`
- Red on the base revision (62d673cc3f): the park, upgrade, old-receipt,
expiry, many-abandonments and scheduling tests fail. The controls pass.
- Green: 19 of 19. Also
`environment-runtime-reusable-resume-replacement.test.ts` 17/17,
`environment-retained-orphan-sandbox.test.ts` 32/32,
`heartbeat-pending-cleanup-sweep.test.ts` 51/51,
`fresh-continuation-fallback.test.ts` 58/58.
- `tsc --noEmit -p server/tsconfig.json` and a tsc pass over the new
test file: clean. `environment-cleanup-work.test.ts` +
`heartbeat-pending-cleanup-sweep.test.ts`: 80/80.

- No existing test expectation was changed. Three CI failures on bb43f52
came from this change. Two came from the hold blocking the run lease
settlement: `ownership-reconciliation-report.test.ts > releases the
lease and frees the workspace for the retry once a preparation-only
epoch ends` and `remote-workspace-save-concurrency.test.ts > lets a
retry prepare the workspace again after its first writer was killed
during preparation`. The third, `native-session-executor.test.ts >
restores a warm backup into a created lease only when a settled restore
hold retained the stopped source`, came from a mocked database that has
no `execute`; the predicate now runs through `select`. All three pass
locally now. So do `heartbeat-orphaned-active-lease-sweep`,
`environment-run-lease-ownership`, `native-workspace-warm-reuse` and
`remote-workspaces.integration`.

## Risks

- Cost of a parked sandbox. Parking makes no provider call and changes
only the lease row. A sandbox that could not be resumed is not running.
A reusable sandbox that was released is already stopped and kept. In
both cases the cost is stopped storage for up to 7 days. One case is
different: if a destructive path finds a held warm sandbox that is still
running (`retained` after a keep-running release), the sandbox stays
running until the receipt is released. The expiry log line counts parked
leases for each company, so this cost can be seen.
- Run lease release for an `ephemeral` lease calls the provider's
release. Some providers delete the sandbox on that call. A held sandbox
normally has a reusable lease, so this pull request does not gate that
call.
- No schema change. The receipt is JSON in
`execution_workspaces.metadata`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 01:00:20 -06:00
1 parent e363b0e52d
commit 0477f622d6
10 files changed
+845 -12

No files matched your search

@@ -0,0 +1,524 @@
import { JsonRpcCallError } from "@paperclipai/plugin-sdk";
import { randomUUID } from "node:crypto";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { eq, sql } from "drizzle-orm";
import {
activityLog,
agents,
companies,
createDb,
environmentLeases,
environments,
executionWorkspaceSnapshots,
executionWorkspaces,
heartbeatRuns,
issues,
plugins,
projects,
} from "@paperclipai/db";
import type { Environment, EnvironmentLease } from "@paperclipai/shared";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { environmentRuntimeService } from "../services/environment-runtime.ts";
import { environmentService } from "../services/environments.ts";
import type { PluginWorkerManager } from "../services/plugin-worker-manager.ts";
import { abandonUnverifiedRemoteWorkspaceEpoch } from "../services/remote-workspace-epoch-abandonment.ts";
import { nativeWorkspaceExportPendingForLease } from "../services/native-runtime/native-workspace-finalization-state.ts";
import { retireReplacedReusableSandboxLease } from "../services/reusable-sandbox-resume-replacement.ts";
import { heartbeatService } from "../services/heartbeat.ts";
import { nextEnvironmentCleanupAt } from "../services/environment-cleanup-state.ts";
import { releasableRetainedOrphanCondition, settleRetainedOrphanSandbox } from "../services/environment-retained-orphan-settlement.ts";
import {
ABANDONED_WORKSPACE_FILES_PARKED_KEY, ABANDONED_WORKSPACE_FILES_PARKED_REASON, ABANDONED_WORKSPACE_FILES_RETENTION_DAYS,
expireAbandonedWorkspaceFileHolds, nextAbandonedWorkspaceFileHoldExpiryAt, releaseAbandonedWorkspaceFiles,
} from "../services/remote-workspace-abandoned-files-hold.ts";
vi.mock("../services/native-runtime/native-harness-backup-storage.js", () => ({
verifyRecoverableNativeHarnessBackup: vi.fn(async () => true),
}));
const support = await getEmbeddedPostgresTestSupport();
const REFUSAL = "Daytona sandbox is still stopping at its start deadline";
describe.skipIf(!support.supported)("sandbox that holds files from an abandoned workspace epoch", () => {
let stopDb: (() => Promise<void>) | null = null;
let db!: ReturnType<typeof createDb>;
let connectionString: string;
beforeAll(async () => {
const started = await startEmbeddedPostgresTestDatabase("abandoned-workspace-files-hold");
stopDb = started.cleanup;
connectionString = started.connectionString;
});
beforeEach(() => { db = createDb(connectionString); });
afterEach(async () => {
vi.restoreAllMocks();
await db.delete(activityLog);
await db.delete(executionWorkspaceSnapshots);
await db.delete(environmentLeases);
await db.delete(heartbeatRuns);
await db.delete(issues);
await db.delete(agents);
await db.delete(environments);
await db.delete(executionWorkspaces);
await db.delete(plugins);
await db.delete(projects);
await db.delete(companies);
await db.$client.end({ timeout: 1 });
});
afterAll(async () => { await stopDb?.(); });
async function seed() {
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID();
const abandonedRunId = randomUUID(), runId = randomUUID(), projectId = randomUUID(), issueId = randomUUID();
const executionWorkspaceId = randomUUID();
const providerLeaseId = randomUUID();
const providerConfig = { provider: "fake-plugin", image: "fake:test", timeoutMs: 300_000, reuseLease: true };
await db.insert(companies).values({ id: companyId, name: "Acme", status: "active", issuePrefix: `H${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "active",
adapterType: "paperclip_runner", adapterConfig: {}, runtimeConfig: {}, permissions: {} });
await db.insert(issues).values({ id: issueId, companyId, title: "Repair", status: "in_progress", assigneeAgentId: agentId });
await db.insert(environments).values({ id: environmentId, name: "Paperclip Sandbox", driver: "sandbox", status: "active",
config: providerConfig });
await db.insert(plugins).values({
id: pluginId, pluginKey: "acme.hold-provider", packageName: "@acme/hold-provider",
version: "1.0.0", apiVersion: 1, categories: ["automation"], status: "ready", installOrder: 1, updatedAt: new Date(),
manifestJson: {
id: "acme.hold-provider", apiVersion: 1, version: "1.0.0", displayName: "Hold Provider",
description: "Test provider", author: "Paperclip", categories: ["automation"],
capabilities: ["environment.drivers.register"], entrypoints: { worker: "dist/worker.js" },
environmentDrivers: [{ driverKey: "fake-plugin", kind: "sandbox_provider", displayName: "Fake Plugin",
supportsReusableLeases: true,
configSchema: { type: "object", properties: { image: { type: "string" }, timeoutMs: { type: "number" }, reuseLease: { type: "boolean" } } } }],
},
} as never);
await db.insert(projects).values({ id: projectId, companyId, name: "Project", status: "active" });
await db.insert(executionWorkspaces).values({ id: executionWorkspaceId, companyId, projectId, mode: "isolated_workspace",
strategyType: "project_primary", name: "Task workspace", status: "active", providerType: "cloud_sandbox",
cwd: `/workspaces/${executionWorkspaceId}`, sourceIssueId: issueId });
await db.insert(heartbeatRuns).values([
{ id: abandonedRunId, companyId, agentId, issueId, invocationSource: "assignment", status: "running" },
{ id: runId, companyId, agentId, issueId, invocationSource: "assignment", status: "running" },
]);
const environment: Environment = { id: environmentId, name: "Paperclip Sandbox", description: null, driver: "sandbox",
status: "active", config: providerConfig, envVars: {}, metadata: null, createdAt: new Date(), updatedAt: new Date() };
const priming = environmentRuntimeService(db, { pluginWorkerManager: {
isRunning: () => true,
call: async (_pluginId: string, method: string) => {
if (method !== "environmentAcquireLease") throw new Error(`Unexpected plugin method: ${method}`);
return { providerLeaseId, metadata: { ...providerConfig, sandboxId: providerLeaseId, remoteCwd: "/home/daytona/paperclip-workspace" } };
},
getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }),
} as unknown as PluginWorkerManager });
const created = await priming.acquireRunLease({ companyId, environment, issueId, agentId, adapterType: "paperclip_runner",
heartbeatRunId: abandonedRunId, persistedExecutionWorkspace: { id: executionWorkspaceId, mode: "isolated_workspace" } });
await db.update(environmentLeases).set({ status: "retained" }).where(eq(environmentLeases.id, created.lease.id));
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, abandonedRunId));
const snapshotId = randomUUID();
await db.insert(executionWorkspaceSnapshots).values({ id: snapshotId, companyId, executionWorkspaceId,
sourceOperationId: randomUUID(), sourceRunId: abandonedRunId, generation: 1, epoch: 1, status: "committed" });
const location = { environmentId, leaseId: created.lease.id, providerLeaseId,
remoteCwd: "/home/daytona/paperclip-workspace/.paperclip-workspaces/abandoned" };
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: {
schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId, source: { kind: "empty" },
writerRunId: abandonedRunId, dirty: true, location } } }).where(eq(executionWorkspaces.id, executionWorkspaceId));
const lease = (await environmentService(db).getLeaseById(created.lease.id))!;
return { companyId, agentId, issueId, environment, pluginId, abandonedRunId, runId, executionWorkspaceId,
lease, providerLeaseId, snapshotId, location };
}
type Seeded = Awaited<ReturnType<typeof seed>>;
async function abandon(seeded: Seeded) {
await db.transaction(tx => abandonUnverifiedRemoteWorkspaceEpoch(tx, { companyId: seeded.companyId,
workspaceId: seeded.executionWorkspaceId, runId: seeded.abandonedRunId, epoch: 1, snapshotId: seeded.snapshotId,
location: seeded.location }));
await reserveFor(seeded, seeded.runId);
}
async function reserveFor(seeded: Seeded, runId: string) {
const workspace = await readWorkspace(seeded);
await db.update(executionWorkspaces).set({ metadata: { ...workspace.metadata,
remoteWorkspace: { ...(workspace.metadata!.remoteWorkspace as Record<string, unknown>), writerRunId: runId } } })
.where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
}
async function readWorkspace(seeded: Seeded) {
return (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, seeded.executionWorkspaceId)))[0]!;
}
async function setReceipts(seeded: Seeded, patch: (receipt: Record<string, unknown>) => Record<string, unknown>) {
const workspace = await readWorkspace(seeded);
const receipts = workspace.metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>>;
await db.update(executionWorkspaces).set({ metadata: { ...workspace.metadata,
remoteWorkspaceAbandonedEpochs: receipts.map(patch) } }).where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
}
function worker(seeded: Seeded) {
const call = vi.fn(async (_pluginId: string, method: string, _params?: unknown) => {
if (method === "environmentResumeLease") throw new JsonRpcCallError({ code: -32000, message: REFUSAL });
if (method === "environmentAcquireLease") {
return { providerLeaseId: "replacement-sandbox", metadata: { provider: "fake-plugin", image: "fake:test",
timeoutMs: 300_000, reuseLease: true, remoteCwd: "/home/daytona/paperclip-workspace" } };
}
if (method === "environmentDestroyLease" || method === "environmentReleaseLease") return {};
throw new Error(`Unexpected plugin method: ${method}`);
});
const manager = { isRunning: vi.fn((id: string) => id === seeded.pluginId), call,
getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })),
} as unknown as PluginWorkerManager;
return { call, runtime: environmentRuntimeService(db, { pluginWorkerManager: manager }) };
}
function acquire(seeded: Seeded, runtime: ReturnType<typeof environmentRuntimeService>) {
let clock = Date.now();
vi.spyOn(Date, "now").mockImplementation(() => (clock += 40_000));
return runtime.acquireRunLease({ companyId: seeded.companyId, environment: seeded.environment, issueId: seeded.issueId,
agentId: seeded.agentId, adapterType: "paperclip_runner", heartbeatRunId: seeded.runId,
persistedExecutionWorkspace: { id: seeded.executionWorkspaceId, mode: "isolated_workspace" } });
}
async function leaseRow(seeded: Seeded) {
return (await environmentService(db).getLeaseById(seeded.lease.id))!;
}
function destroyCalls(call: ReturnType<typeof worker>["call"]) {
return call.mock.calls.filter(([, method]) => method === "environmentDestroyLease");
}
async function sweep(runtime: ReturnType<typeof environmentRuntimeService>) {
return await heartbeatService(db, { environmentRuntime: runtime as never }).sweepPendingCleanupLeases({ backoffMs: 0 });
}
async function newRun(seeded: Seeded) {
const runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: seeded.companyId, agentId: seeded.agentId,
issueId: seeded.issueId, invocationSource: "assignment", status: "running" });
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, seeded.runId));
await reserveFor(seeded, runId);
return runId;
}
it("gives the next run a new sandbox at once and parks the held one, which cleanup and later runs leave alone", async () => {
const seeded = await seed();
await abandon(seeded);
const { call, runtime } = worker(seeded);
const acquired = await acquire(seeded, runtime);
expect(acquired.lease.providerLeaseId).toBe("replacement-sandbox");
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "replacement", reason: "resume_failed" });
const parked = await leaseRow(seeded);
expect(parked).toMatchObject({ status: "retained", failureReason: ABANDONED_WORKSPACE_FILES_PARKED_REASON });
expect(parked.metadata).toHaveProperty(ABANDONED_WORKSPACE_FILES_PARKED_KEY);
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(true);
await sweep(runtime);
await expect(runtime.destroyRunLease({ environment: seeded.environment, lease: parked,
failureReason: "pending_cleanup_retry" })).resolves.toBeNull();
await expect(runtime.retryPendingSandboxTeardown({ environment: seeded.environment, lease: parked })).rejects.toThrow();
expect(destroyCalls(call)).toEqual([]);
expect(await leaseRow(seeded)).toMatchObject({ status: "retained" });
await db.update(environmentLeases).set({ status: "released" }).where(eq(environmentLeases.providerLeaseId, "replacement-sandbox"));
const nextRunId = await newRun(seeded);
const later = worker(seeded);
let clock = Date.now();
vi.spyOn(Date, "now").mockImplementation(() => (clock += 40_000));
later.call.mockImplementation(async (_pluginId: string, method: string) => {
if (method === "environmentResumeLease") return { providerLeaseId: "replacement-sandbox", metadata: { provider: "fake-plugin",
image: "fake:test", timeoutMs: 300_000, reuseLease: true, remoteCwd: "/home/daytona/paperclip-workspace" } };
throw new Error(`Unexpected plugin method: ${method}`);
});
const next = await later.runtime.acquireRunLease({ companyId: seeded.companyId, environment: seeded.environment,
issueId: seeded.issueId, agentId: seeded.agentId, adapterType: "paperclip_runner", heartbeatRunId: nextRunId,
persistedExecutionWorkspace: { id: seeded.executionWorkspaceId, mode: "isolated_workspace" } });
expect(next.lease.providerLeaseId).toBe("replacement-sandbox");
const resumed = later.call.mock.calls.filter(([, method]) => method === "environmentResumeLease");
expect(resumed.map(([, , params]) => (params as { providerLeaseId: string }).providerLeaseId)).toEqual(["replacement-sandbox"]);
expect(await leaseRow(seeded)).toMatchObject({ status: "retained" });
});
it("pulls a held lease already queued for cleanup back to retained, and leaves it on the provider", async () => {
const seeded = await seed();
await abandon(seeded);
await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", failureReason: "resume_failed" })
.where(eq(environmentLeases.id, seeded.lease.id));
const { call, runtime } = worker(seeded);
await sweep(runtime);
expect(destroyCalls(call)).toEqual([]);
expect(await leaseRow(seeded)).toMatchObject({ status: "retained", failureReason: ABANDONED_WORKSPACE_FILES_PARKED_REASON });
});
it("treats a receipt written before release fields existed as unreleased", async () => {
const seeded = await seed();
await abandon(seeded);
await setReceipts(seeded, ({ retainUntil: _retainUntil, released: _released, ...receipt }) => receipt);
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(true);
expect(await retireReplacedReusableSandboxLease({ db, lease: seeded.lease as EnvironmentLease,
companyId: seeded.companyId, heartbeatRunId: seeded.runId })).toBe(true);
expect(await leaseRow(seeded)).toMatchObject({ status: "retained" });
});
it("records a retention expiry on the receipt", async () => {
const seeded = await seed();
await abandon(seeded);
const [receipt] = (await readWorkspace(seeded)).metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>>;
expect(Date.parse(receipt!.retainUntil as string) - Date.parse(receipt!.abandonedAt as string))
.toBe(ABANDONED_WORKSPACE_FILES_RETENTION_DAYS * 86_400_000);
});
it.each([
{ reason: "recovered" as const, actorType: "system" as const, actorId: "workspace-recovery" },
{ reason: "discarded" as const, actorType: "user" as const, actorId: "board-user" },
])("hands the parked lease to normal cleanup once the files are $reason", async ({ reason, actorType, actorId }) => {
const seeded = await seed();
await abandon(seeded);
const { call, runtime } = worker(seeded);
await acquire(seeded, runtime);
expect(await releaseAbandonedWorkspaceFiles(db, { companyId: seeded.companyId, workspaceId: seeded.executionWorkspaceId,
epoch: 1, reason, actorType, actorId })).toBe(true);
expect(await leaseRow(seeded)).toMatchObject({ status: "pending_cleanup" });
vi.restoreAllMocks();
await sweep(runtime);
expect(destroyCalls(call).map(([, , params]) => (params as { providerLeaseId: string }).providerLeaseId))
.toEqual([seeded.providerLeaseId]);
expect(await leaseRow(seeded)).toMatchObject({ status: "expired" });
});
async function entries(action: string) {
return await db.select().from(activityLog).where(eq(activityLog.action, action));
}
it("removes the files only after the retention period, and says removed only after the provider confirms it", async () => {
const seeded = await seed();
await abandon(seeded);
const { call, runtime } = worker(seeded);
await acquire(seeded, runtime);
vi.restoreAllMocks();
expect(await expireAbandonedWorkspaceFileHolds(db, new Date(Date.now() + 86_400_000))).toEqual({ expired: 0 });
await sweep(runtime);
expect(destroyCalls(call)).toEqual([]);
expect(await entries("workspace.abandoned_files_released")).toEqual([]);
expect(await expireAbandonedWorkspaceFileHolds(db, new Date(Date.now() + 8 * 86_400_000))).toEqual({ expired: 1 });
const [released] = await entries("workspace.abandoned_files_released");
expect(released).toMatchObject({ entityType: "issue", entityId: seeded.issueId, actorType: "system", actorId: "workspace-retention" });
expect(released!.details).toMatchObject({ reason: "retention_expired",
summary: "Unsaved files from an interrupted run were kept for 8 days, and their retention period has ended. They are scheduled for removal." });
expect(await entries("workspace.abandoned_files_removed")).toEqual([]);
await sweep(runtime);
expect(destroyCalls(call)).toHaveLength(1);
expect(await leaseRow(seeded)).toMatchObject({ status: "expired" });
const [removed] = await entries("workspace.abandoned_files_removed");
expect(removed).toMatchObject({ entityType: "issue", entityId: seeded.issueId, actorType: "system" });
expect(removed!.details).toMatchObject({
summary: "Unsaved files from an interrupted run were kept for 8 days and have now been removed." });
});
it("does not report removal when the provider delete fails", async () => {
const seeded = await seed();
await abandon(seeded);
const { call, runtime } = worker(seeded);
await acquire(seeded, runtime);
vi.restoreAllMocks();
await setReceipts(seeded, receipt => ({ ...receipt, retainUntil: new Date(Date.now() - 1_000).toISOString() }));
call.mockImplementation(async (_pluginId: string, method: string) => {
throw new Error(`provider refused ${method}`);
});
await sweep(runtime);
expect(await entries("workspace.abandoned_files_released")).toHaveLength(1);
expect(await entries("workspace.abandoned_files_removed")).toEqual([]);
expect(await leaseRow(seeded)).toMatchObject({ status: "pending_cleanup" });
});
it("records the end of retention without scheduling removal when another receipt still holds the sandbox", async () => {
const seeded = await seed();
await abandon(seeded);
const { call, runtime } = worker(seeded);
await acquire(seeded, runtime);
vi.restoreAllMocks();
const workspace = await readWorkspace(seeded);
const [first] = workspace.metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>>;
await db.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspaceAbandonedEpochs: [
{ ...first, retainUntil: new Date(Date.now() - 1_000).toISOString() },
{ ...first, epoch: 7, location: { ...(first!.location as Record<string, unknown>), remoteCwd: "/other-root" },
retainUntil: new Date(Date.now() + 86_400_000).toISOString() },
] } }).where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
await sweep(runtime);
const [ended] = await entries("workspace.abandoned_files_released");
expect(ended).toMatchObject({ entityType: "issue", entityId: seeded.issueId, actorType: "system", actorId: "workspace-retention" });
expect(ended!.details).toMatchObject({ reason: "retention_expired", leasesHandedToCleanup: 0,
summary: expect.not.stringContaining("scheduled for removal") });
expect((ended!.details as { summary: string }).summary).toContain("retention period has ended");
expect(destroyCalls(call)).toEqual([]);
expect(await leaseRow(seeded)).toMatchObject({ status: "retained" });
});
it("records the end of retention when it expires before the lease was ever parked", async () => {
const seeded = await seed();
await abandon(seeded);
const { runtime } = worker(seeded);
await setReceipts(seeded, receipt => ({ ...receipt, retainUntil: new Date(Date.now() - 1_000).toISOString() }));
await sweep(runtime);
const [ended] = await entries("workspace.abandoned_files_released");
expect(ended).toMatchObject({ entityType: "issue", entityId: seeded.issueId, actorId: "workspace-retention" });
expect(ended!.details).toMatchObject({ reason: "retention_expired", leasesHandedToCleanup: 0 });
expect((ended!.details as { summary: string }).summary).not.toContain("scheduled for removal");
expect(await entries("workspace.abandoned_files_removed")).toEqual([]);
});
it("releases a receipt whose run was deleted and records it without the run reference", async () => {
const seeded = await seed();
await abandon(seeded);
await setReceipts(seeded, receipt => ({ ...receipt, runId: randomUUID(), retainUntil: new Date(Date.now() - 1_000).toISOString() }));
expect(await expireAbandonedWorkspaceFileHolds(db)).toEqual({ expired: 1 });
const [ended] = await entries("workspace.abandoned_files_released");
expect(ended).toMatchObject({ entityType: "issue", entityId: seeded.issueId, runId: null });
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(false);
});
it("releases a receipt whose task was deleted without writing a task entry", async () => {
const seeded = await seed();
await abandon(seeded);
await setReceipts(seeded, receipt => ({ ...receipt, runId: randomUUID(), retainUntil: new Date(Date.now() - 1_000).toISOString() }));
await db.update(executionWorkspaces).set({ sourceIssueId: null }).where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
expect(await expireAbandonedWorkspaceFileHolds(db)).toEqual({ expired: 1 });
expect(await entries("workspace.abandoned_files_released")).toEqual([]);
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(false);
});
it("keeps expiring later receipts when one fails, and retries the failed one after a backoff", async () => {
const seeded = await seed();
await abandon(seeded);
await abandonOnOtherSandbox(seeded, "/later-root");
await setReceipts(seeded, receipt => ({ ...receipt, retainUntil: new Date(Date.now() - 1_000).toISOString() }));
await db.execute(sql`create function reject_abandoned_epoch_one() returns trigger language plpgsql as $$
begin if new.details->>'abandonedEpoch' = '1' then raise exception 'test activity failure'; end if; return new; end $$`);
await db.execute(sql`create trigger reject_abandoned_epoch_one before insert on activity_log
for each row execute function reject_abandoned_epoch_one()`);
try {
const before = Date.now();
expect(await expireAbandonedWorkspaceFileHolds(db)).toEqual({ expired: 1 });
const receipts = (await readWorkspace(seeded)).metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>>;
const failed = receipts.find(receipt => receipt.epoch === 1)!;
expect(failed.released).toBeUndefined();
expect(failed.expiryAttempts).toBe(1);
expect(receipts.find(receipt => receipt.epoch !== 1)!.released).toMatchObject({ reason: "retention_expired" });
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(true);
expect(await nextAbandonedWorkspaceFileHoldExpiryAt(db)).toBeGreaterThanOrEqual(before + 5 * 60_000);
} finally {
await db.execute(sql`drop trigger reject_abandoned_epoch_one on activity_log`);
await db.execute(sql`drop function reject_abandoned_epoch_one()`);
}
expect(await expireAbandonedWorkspaceFileHolds(db, new Date(Date.now() + 10 * 60_000))).toEqual({ expired: 1 });
});
it("wakes the cleanup worker at the receipt's retention end, also after a restart, and then goes quiet", async () => {
const seeded = await seed();
await abandon(seeded);
const { call, runtime } = worker(seeded);
await acquire(seeded, runtime);
vi.restoreAllMocks();
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success" })
.where(eq(environmentLeases.providerLeaseId, "replacement-sandbox"));
const [receipt] = (await readWorkspace(seeded)).metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>>;
const retainUntil = Date.parse(receipt!.retainUntil as string);
expect(await nextEnvironmentCleanupAt(db)).toBe(retainUntil);
const restarted = createDb(connectionString);
try {
expect(await nextEnvironmentCleanupAt(restarted)).toBe(retainUntil);
} finally {
await restarted.$client.end({ timeout: 1 });
}
await setReceipts(seeded, value => ({ ...value, retainUntil: new Date(Date.now() - 1_000).toISOString() }));
await sweep(runtime);
expect(destroyCalls(call)).toHaveLength(1);
expect(await entries("workspace.abandoned_files_removed")).toHaveLength(1);
expect(await nextEnvironmentCleanupAt(db)).toBeNull();
});
async function stoppedWarmOrphan(seeded: Seeded) {
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, seeded.runId));
await db.update(environmentLeases).set({ status: "failed", cleanupStatus: "success", leasePolicy: "reuse_by_environment",
metadata: { ...seeded.lease.metadata, remoteExecutionTermination: { state: "stopped", leaseId: seeded.lease.id,
runId: seeded.abandonedRunId } } }).where(eq(environmentLeases.id, seeded.lease.id));
const listed = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(releasableRetainedOrphanCondition());
const settled = await settleRetainedOrphanSandbox(db, { recordId: seeded.lease.id, runId: seeded.abandonedRunId } as never,
{ now: new Date(), proof: { reason: "owner_gone" } as never });
return { listed: listed.map(row => row.id), settled };
}
it("keeps a held sandbox when the retained orphan settlement reviews it", async () => {
const seeded = await seed();
await abandon(seeded);
const { listed, settled } = await stoppedWarmOrphan(seeded);
expect(listed).toEqual([]);
expect(settled).toEqual({ settled: false, reason: "workspace_preservation_pending" });
expect(await leaseRow(seeded)).toMatchObject({ status: "failed" });
});
it("lists the sandbox for the retained orphan settlement again once the receipt is released", async () => {
const seeded = await seed();
await abandon(seeded);
await setReceipts(seeded, receipt => ({ ...receipt,
released: { reason: "discarded", releasedAt: new Date().toISOString(), actorType: "user", actorId: "board-user" } }));
const { listed, settled } = await stoppedWarmOrphan(seeded);
expect(listed).toEqual([seeded.lease.id]);
expect(settled.reason).not.toBe("workspace_preservation_pending");
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(false);
});
async function abandonOnOtherSandbox(seeded: Seeded, remoteCwd: string) {
const workspace = await readWorkspace(seeded);
const authority = workspace.metadata!.remoteWorkspace as Record<string, unknown> & { epoch: number };
const location = { environmentId: seeded.environment.id, leaseId: randomUUID(), providerLeaseId: "sandbox-b", remoteCwd };
await db.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspace: { ...authority,
writerRunId: seeded.runId, dirty: true, location } } }).where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
await db.transaction(tx => abandonUnverifiedRemoteWorkspaceEpoch(tx, { companyId: seeded.companyId,
workspaceId: seeded.executionWorkspaceId, runId: seeded.runId, epoch: authority.epoch,
snapshotId: authority.snapshotId as string, location }));
}
it("keeps holding the first sandbox after many later abandonments on another sandbox", async () => {
const seeded = await seed();
await abandon(seeded);
for (let index = 0; index < 21; index++) await abandonOnOtherSandbox(seeded, `/root-${index}`);
for (let index = 0; index < 5; index++) await abandonOnOtherSandbox(seeded, "/root-0");
const receipts = (await readWorkspace(seeded)).metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>>;
expect(receipts).toHaveLength(22);
expect(receipts.filter(receipt => (receipt.location as { remoteCwd: string }).remoteCwd === "/root-0")).toHaveLength(1);
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(true);
expect(await releaseAbandonedWorkspaceFiles(db, { companyId: seeded.companyId, workspaceId: seeded.executionWorkspaceId,
epoch: 1, reason: "discarded", actorType: "user", actorId: "board-user" })).toBe(true);
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(false);
});
it("replaces and cleans the sandbox as before when the receipt names a different sandbox", async () => {
const seeded = await seed();
await abandon(seeded);
await setReceipts(seeded, receipt => ({ ...receipt,
location: { ...(receipt.location as Record<string, unknown>), providerLeaseId: "another-sandbox" } }));
const { runtime } = worker(seeded);
const acquired = await acquire(seeded, runtime);
expect(acquired.lease.providerLeaseId).toBe("replacement-sandbox");
expect(await leaseRow(seeded)).toMatchObject({ status: "pending_cleanup", failureReason: "resume_failed" });
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(false);
});
it("replaces and cleans the sandbox as before when no epoch was abandoned", async () => {
const seeded = await seed();
const workspace = await readWorkspace(seeded);
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: {
...(workspace.metadata!.remoteWorkspace as Record<string, unknown>), epoch: 2, dirty: false, writerRunId: seeded.runId } } })
.where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
const { runtime } = worker(seeded);
const acquired = await acquire(seeded, runtime);
expect(acquired.lease.providerLeaseId).toBe("replacement-sandbox");
expect(await leaseRow(seeded)).toMatchObject({ status: "pending_cleanup" });
});
});
@@ -1,6 +1,7 @@
import { and, eq, or, sql } from "drizzle-orm";
import { environmentLeases, sandboxCreationOperations, type Db } from "@paperclipai/db";
import { hasDeferredEnvironmentCleanup } from "./environment-cleanup-notifications.js";
import { nextAbandonedWorkspaceFileHoldExpiryAt } from "./remote-workspace-abandoned-files-hold.js";
export const ENVIRONMENT_CLEANUP_RETRY_MS = 5 * 60_000;
@@ -86,7 +87,8 @@ export async function nextEnvironmentCleanupAt(db: Db, backoffMs = ENVIRONMENT_C
${pendingCleanupDeadlineSql()}) end)` }).from(environmentLeases).where(eq(environmentLeases.status, "pending_cleanup"));
const [creation] = await db.select({ next: sql<string | null>`min(extract(epoch from ${sandboxCreationOperations.nextAttemptAt}) * 1000)` })
.from(sandboxCreationOperations).where(recoverableSandboxCreationCondition());
const deadlines = [row?.next, creation?.next].filter((value): value is string => value != null).map(Number);
const expiry = await nextAbandonedWorkspaceFileHoldExpiryAt(db);
const deadlines = [row?.next, creation?.next, expiry].filter((value): value is string | number => value != null).map(Number);
const saved = deadlines.length ? Math.min(...deadlines) : null;
const buffered = hasDeferredEnvironmentCleanup(db) ? Date.now() + backoffMs : null;
return saved === null ? buffered : buffered === null ? saved : Math.min(saved, buffered);
@@ -2,6 +2,7 @@ import { and, eq, inArray, ne, notInArray, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, issueRecoveryActions, type Db } from "@paperclipai/db";
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
import { nativeWorkspaceExportPendingForLease } from "./native-runtime/native-workspace-finalization-state.js";
import { abandonedWorkspaceFilesHeldOnLeaseRowSql } from "./remote-workspace-abandoned-files-hold.js";
import type { OwnerProof } from "./owner-proof.js";
import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
@@ -111,6 +112,7 @@ function softlyHeldSql(now?: Date) {
function alwaysHeldSql() {
return sql`(${environmentLeases.metadata} ?| array['nativeWorkspaceExportResume', 'sandboxStopAndRetain',
'workspaceRestoreRecovery', 'workspaceRecoveryCleanup', 'runnerTaskSourceConsumer']
or ${abandonedWorkspaceFilesHeldOnLeaseRowSql()}
or coalesce(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') = 'true'
or coalesce(${environmentLeases.metadata}#>>'{legacyShutdownWorkspaceSettlement,state}' = 'pending', false)
or exists (select 1 from heartbeat_runs run where run.id = ${environmentLeases.heartbeatRunId}
@@ -113,7 +113,7 @@ async function currentBootOwnsRun(tx: Db, lease: LeaseRow, runControllerBootId:
}
async function keptByAnotherObligation(tx: Db, lease: LeaseRow): Promise<string | null> {
if (await nativeWorkspaceExportPendingForLease(tx, lease)) return "workspace_export_pending";
if (await nativeWorkspaceExportPendingForLease(tx, lease, { destructive: false })) return "workspace_export_pending";
if (!(await allowLegacyShutdownWorkspaceCleanup(tx, lease))) return "legacy_shutdown_fence";
// One provider resource id names one physical sandbox; another live lease
// on it still owns that sandbox.
+12 -2
View File
@@ -22,6 +22,7 @@ import { flushDeferredEnvironmentCleanup, registerDeferredEnvironmentCleanup, wr
import { executeEnvironmentTask, pluginEnvironmentSupportsTasks, settleDispatchedEnvironmentTask } from "./environment-task-runtime.js";
import { allowLegacyShutdownWorkspaceCleanup, legacyShutdownWorkspaceResourceProtected } from "./legacy-shutdown-workspace-settlement.js";
import { nativeWorkspaceExportPendingForLease, nativeWorkspaceExportPendingForResource } from "./native-runtime/native-workspace-finalization-state.js";
import { abandonedWorkspaceFilesHeldOnLease, abandonedWorkspaceFilesParked, parkAbandonedWorkspaceFilesLease } from "./remote-workspace-abandoned-files-hold.js";
import { beginIdleTrackedWork } from "./task-admission.js";
import { historicalProviderCredentials } from "./historical-provider-credentials.js";
import { ownedWorkspaceRecoveryLease, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
@@ -1137,14 +1138,16 @@ async function waitForSettlingReusableLease(db: Db, input: {
}
function reusableLeaseCanBeResumed(input: {
lease: Pick<EnvironmentLease, "status" | "heartbeatRunId">;
lease: Pick<EnvironmentLease, "status" | "heartbeatRunId" | "metadata">;
heartbeatRunId: string | null;
}): boolean {
if (abandonedWorkspaceFilesParked(input.lease)) return false;
if (input.lease.status === "released" || input.lease.status === "retained") return true;
return input.lease.status === "active" && input.heartbeatRunId !== null && input.lease.heartbeatRunId === input.heartbeatRunId;
}
function reusableLeaseCanBeCleanedUp(lease: Pick<EnvironmentLease, "status">): boolean {
function reusableLeaseCanBeCleanedUp(lease: Pick<EnvironmentLease, "status" | "metadata">): boolean {
if (abandonedWorkspaceFilesParked(lease)) return false;
return lease.status === "released" || lease.status === "retained";
}
@@ -3214,6 +3217,10 @@ function createSandboxEnvironmentDriver(
failureReason: string;
}): Promise<EnvironmentLease | null> {
if (await legacyShutdownWorkspaceResourceProtected(db, input.lease)) return null;
if (await abandonedWorkspaceFilesHeldOnLease(db, input.lease)) {
await parkAbandonedWorkspaceFilesLease(db, input.lease, ["released", "retained", "failed", "pending_cleanup"]);
return null;
}
let cleanupStatus: "success" | "failed" = "success";
let termination: ReturnType<typeof remoteTerminationReceipt>;
const metadata = input.lease.metadata ?? {};
@@ -4341,6 +4348,9 @@ export function environmentRuntimeService(
if (!(await allowLegacyShutdownWorkspaceCleanup(db, input.lease))) {
throw new Error("Shutdown workspace settlement is still pending.");
}
if (!hasStopOnlyCleanup(input.lease) && await abandonedWorkspaceFilesHeldOnLease(db, input.lease)) {
throw new Error("Unsaved files from an abandoned workspace are held on this sandbox; destructive cleanup is forbidden.");
}
if (input.lease.metadata?.legacyShutdownWorkspaceSettlement) {
const current = await environmentsSvc.getLeaseById(input.lease.id);
if (!current) throw new Error("Shutdown workspace source is unavailable.");
+11 -1
View File
@@ -31,6 +31,7 @@ import { createHeartbeatAgentStatus } from "./agent-status.js";
import { preserveWorkspaceRestoreRecoveryMetadataSql } from "../legacy-workspace-restore-recovery.js";
import { preserveCancelledNativeWorkspaces, preserveOrphanedNativeWorkspace } from "../native-runtime/native-failed-workspace-recovery.js";
import { nativeWorkspaceExportPendingForLease, pendingNativeWorkspaceFinalizationCondition } from "../native-runtime/native-workspace-finalization-state.js";
import { expireAbandonedWorkspaceFileHoldsWhenDue, parkAbandonedWorkspaceFilesLease, recordAbandonedWorkspaceFilesRemoved } from "../remote-workspace-abandoned-files-hold.js";
import { CHAT_COMPLETION_WAKE_REASON } from "../chat-completion-delivery.js";
import {
hasStopOnlyCleanup,
@@ -1747,6 +1748,10 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
logger.warn({ errorKind: "teardown_claim_settle_failed" }, "teardown claim settle failed; a later sweep retries");
});
if (!manualRetry) await expireAbandonedWorkspaceFileHoldsWhenDue(db).catch(() => {
logger.warn({ errorKind: "abandoned_workspace_files_expiry_failed" }, "abandoned workspace file expiry failed; a later sweep retries");
});
const rows = await db
.select()
.from(environmentLeases)
@@ -1767,6 +1772,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
let capped = 0;
for (const row of rows) {
if (pendingCleanupAttemptsInFlight.has(row.id)) continue;
if (!hasStopOnlyCleanup(row) && await parkAbandonedWorkspaceFilesLease(db, row, ["pending_cleanup"])) continue;
if (!hasStopOnlyCleanup(row) && await nativeWorkspaceExportPendingForLease(db, row)) {
await deferPendingCleanupLease(row.id);
continue;
@@ -1881,7 +1887,10 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
failureReason: "pending_cleanup_retry",
remoteExecutionTermination: remoteTerminationReceipt(lease, receipt),
});
if (released) destroyed += 1;
if (released) {
destroyed += 1;
await recordAbandonedWorkspaceFilesRemoved(db, row);
}
} else if (environment) {
const result = await environmentRuntime.destroyRunLease({
environment,
@@ -1890,6 +1899,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
});
if (result && result.status !== "pending_cleanup") {
destroyed += 1;
if (result.cleanupStatus === "success") await recordAbandonedWorkspaceFilesRemoved(db, row);
}
}
} catch (error) {
@@ -3,6 +3,7 @@ import { and, eq, exists, sql } from "drizzle-orm";
import { hasRequiredWorkspaceRecovery, nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js";
import { recordRequiredWorkspaceRestoreFailure } from "../legacy-execution-recovery.js";
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
import { abandonedWorkspaceFilesHeldOnLease } from "../remote-workspace-abandoned-files-hold.js";
import { matchPreparedRemoteWorkspaceSource, preparedRemoteWorkspaceSource, type WorkspaceSourceLease } from "../remote-workspace-lifecycle.js";
/** File-only recovery for setup that never reached native admission. The live
@@ -79,15 +80,19 @@ function nativeWorkspaceSourcePending(run: Pick<typeof heartbeatRuns.$inferSelec
&& reference.providerLeaseId === lease.providerLeaseId;
}
export async function nativeWorkspaceExportPendingForLease(db: Db, lease: WorkspaceSourceLease): Promise<boolean> {
if (!lease.heartbeatRunId || !lease.provider || lease.provider === "local" || !lease.providerLeaseId) return false;
export async function nativeWorkspaceExportPendingForLease(db: Db, lease: WorkspaceSourceLease,
options: { destructive?: boolean } = {}): Promise<boolean> {
if (!lease.provider || lease.provider === "local" || !lease.providerLeaseId) return false;
const held = () => options.destructive === false ? Promise.resolve(false) : abandonedWorkspaceFilesHeldOnLease(db, lease);
if (!lease.heartbeatRunId) return await held();
const [run] = await db.select(nativeWorkspaceRunProjection()).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, lease.heartbeatRunId), eq(heartbeatRuns.companyId, lease.companyId),
eq(heartbeatRuns.runtimeMode, "native"),
)).limit(1);
if (run?.observedRemoteWorkspace) return true;
if (await preparedRemoteWorkspaceSource(db, lease)) return true;
return Boolean(run && nativeWorkspaceSourcePending(run, lease));
if (run && nativeWorkspaceSourcePending(run, lease)) return true;
return await held();
}
export async function nativeWorkspaceExportPendingForResource(db: Db, lease: WorkspaceSourceLease, resumingRunId: string | null) {
@@ -0,0 +1,262 @@
import { and, eq, inArray, sql, type SQL } from "drizzle-orm";
import { companies, environmentLeases, executionWorkspaces, heartbeatRuns, issues, type Db } from "@paperclipai/db";
import { logger } from "../middleware/logger.js";
import { logActivity } from "./activity-log.js";
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
import {
ABANDONED_WORKSPACE_FILES_RETENTION_DAYS, ABANDONED_WORKSPACE_FILES_RETENTION_MS, REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA,
} from "./remote-workspace-epoch-abandonment.js";
export { ABANDONED_WORKSPACE_FILES_RETENTION_DAYS, ABANDONED_WORKSPACE_FILES_RETENTION_MS };
export const ABANDONED_WORKSPACE_FILES_PARKED_KEY = "abandonedWorkspaceFilesParked";
export const ABANDONED_WORKSPACE_FILES_PARKED_REASON = "abandoned_workspace_files_held";
export const ABANDONED_WORKSPACE_FILES_RELEASED_REASON = "abandoned_workspace_files_released";
export const ABANDONED_WORKSPACE_FILES_RELEASE_KEY = "abandonedWorkspaceFilesRelease";
const ABANDONED_WORKSPACE_FILES_EXPIRY_BACKOFF_MS = 5 * 60_000;
const ABANDONED_WORKSPACE_FILES_EXPIRY_MAX_BACKOFF_MS = 6 * 60 * 60_000;
async function existingIssueId(db: Db, companyId: string, issueId: string | null): Promise<string | null> {
if (!issueId) return null;
const [issue] = await db.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)));
return issue?.id ?? null;
}
async function existingRunId(db: Db, companyId: string, runId: string | null | undefined): Promise<string | undefined> {
if (!runId) return undefined;
const [run] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId)));
return run?.id;
}
function days(count: number) {
return `${count} ${count === 1 ? "day" : "days"}`;
}
export type AbandonedWorkspaceFilesReleaseReason = "recovered" | "discarded" | "retention_expired";
type HoldLease = Pick<typeof environmentLeases.$inferSelect, "companyId" | "provider" | "providerLeaseId">;
type Receipt = Record<string, unknown> & { location?: { providerLeaseId?: unknown }; released?: unknown };
export function abandonedWorkspaceFilesHeldSql(companyId: SQL | string, providerLeaseId: SQL | string): SQL {
return sql`exists (select 1 from execution_workspaces held_workspace
cross join lateral jsonb_array_elements(case when jsonb_typeof(held_workspace.metadata->'remoteWorkspaceAbandonedEpochs') = 'array'
then held_workspace.metadata->'remoteWorkspaceAbandonedEpochs' else '[]'::jsonb end) held_receipt
where held_workspace.company_id = ${companyId}
and held_workspace.metadata ? 'remoteWorkspaceAbandonedEpochs'
and held_receipt->>'schema' = ${REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA}
and held_receipt->'location'->>'providerLeaseId' = ${providerLeaseId}
and coalesce(jsonb_typeof(held_receipt->'released'), 'null') = 'null')`;
}
export function abandonedWorkspaceFilesHeldOnLeaseRowSql(): SQL {
return sql`(${environmentLeases.providerLeaseId} is not null and coalesce(${environmentLeases.provider}, 'local') <> 'local'
and ${abandonedWorkspaceFilesHeldSql(sql`${environmentLeases.companyId}`, sql`${environmentLeases.providerLeaseId}`)})`;
}
export async function abandonedWorkspaceFilesHeldOnLease(db: Pick<Db, "select">, lease: HoldLease): Promise<boolean> {
if (!lease.providerLeaseId || !lease.provider || lease.provider === "local") return false;
const [row] = await db.select({ held: sql<boolean>`${abandonedWorkspaceFilesHeldSql(sql`${companies.id}`, lease.providerLeaseId)}` })
.from(companies).where(eq(companies.id, lease.companyId)).limit(1);
return row?.held === true;
}
export function abandonedWorkspaceFilesParked(lease: { metadata?: Record<string, unknown> | null }): boolean {
return lease.metadata?.[ABANDONED_WORKSPACE_FILES_PARKED_KEY] != null;
}
export async function parkAbandonedWorkspaceFilesLease(db: Db, lease: Pick<typeof environmentLeases.$inferSelect,
"id" | "companyId" | "heartbeatRunId">, statuses: string[]): Promise<boolean> {
const now = new Date();
const parked = await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({
status: "retained", cleanupStatus: "failed", failureReason: ABANDONED_WORKSPACE_FILES_PARKED_REASON, updatedAt: now,
metadata: sql`(case when jsonb_typeof(${environmentLeases.metadata}) = 'object' then ${environmentLeases.metadata} else '{}'::jsonb end)
|| ${JSON.stringify({ [ABANDONED_WORKSPACE_FILES_PARKED_KEY]: { parkedAt: now.toISOString() }, pendingCleanupInFlight: false })}::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),
inArray(environmentLeases.status, statuses),
sql`coalesce(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') != 'true'`,
abandonedWorkspaceFilesHeldOnLeaseRowSql())).returning({ id: environmentLeases.id }),
{ companyId: lease.companyId, runId: lease.heartbeatRunId ?? undefined, leaseId: lease.id });
return parked.length > 0;
}
function receiptDueAt(receipt: Receipt): number {
const retry = typeof receipt.expiryRetryAt === "string" ? Date.parse(receipt.expiryRetryAt) : Number.NaN;
return Math.max(receiptRetainUntil(receipt), Number.isFinite(retry) ? retry : 0);
}
function receiptRetainUntil(receipt: Receipt): number {
const explicit = typeof receipt.retainUntil === "string" ? Date.parse(receipt.retainUntil) : Number.NaN;
if (Number.isFinite(explicit)) return explicit;
const abandonedAt = typeof receipt.abandonedAt === "string" ? Date.parse(receipt.abandonedAt) : Number.NaN;
return (Number.isFinite(abandonedAt) ? abandonedAt : 0) + ABANDONED_WORKSPACE_FILES_RETENTION_MS;
}
function unreleased(receipt: unknown): receipt is Receipt {
const value = receipt as Receipt | null;
return value?.schema === REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA && value.released == null
&& typeof value.location?.providerLeaseId === "string";
}
export async function releaseAbandonedWorkspaceFiles(db: Db, input: {
companyId: string; workspaceId: string; epoch: number; reason: AbandonedWorkspaceFilesReleaseReason;
actorType: "system" | "user" | "agent"; actorId: string; now?: Date;
}): Promise<boolean> {
const now = input.now ?? new Date();
return db.transaction(async transaction => {
const tx = transaction as unknown as Db;
const [workspace] = await tx.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId))).for("update");
const receipts = Array.isArray(workspace?.metadata?.remoteWorkspaceAbandonedEpochs)
? workspace!.metadata!.remoteWorkspaceAbandonedEpochs as unknown[] : [];
const target = receipts.find(receipt => unreleased(receipt) && receipt.epoch === input.epoch) as Receipt | undefined;
if (!workspace || !target) return false;
const released = { reason: input.reason, releasedAt: now.toISOString(), actorType: input.actorType, actorId: input.actorId };
await tx.update(executionWorkspaces).set({ updatedAt: now, metadata: { ...workspace.metadata,
remoteWorkspaceAbandonedEpochs: receipts.map(receipt => receipt === target ? { ...target, released } : receipt) } })
.where(and(eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, workspace.id)));
const providerLeaseId = target.location!.providerLeaseId as string;
const recordedRunId = typeof target.runId === "string" ? target.runId : undefined;
const [run] = recordedRunId ? await tx.select({ id: heartbeatRuns.id, issueId: heartbeatRuns.issueId }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, recordedRunId))) : [];
const runId = run?.id;
const issueId = await existingIssueId(tx, input.companyId, workspace.sourceIssueId ?? run?.issueId ?? null);
const keptDays = Math.max(1, Math.round((now.getTime() - Date.parse(String(target.abandonedAt ?? now.toISOString()))) / 86_400_000));
const marker = { issueId, workspaceId: workspace.id, abandonedEpoch: input.epoch, reason: input.reason, keptDays, runId: runId ?? null };
const unparked = await writeEnvironmentCleanup(tx, write => write.update(environmentLeases).set({
status: "pending_cleanup", cleanupStatus: "failed", failureReason: ABANDONED_WORKSPACE_FILES_RELEASED_REASON,
releasedAt: now, updatedAt: now,
metadata: sql`(${environmentLeases.metadata} - ${ABANDONED_WORKSPACE_FILES_PARKED_KEY})
|| ${JSON.stringify({ [ABANDONED_WORKSPACE_FILES_RELEASE_KEY]: marker })}::jsonb`,
}).where(and(eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.providerLeaseId, providerLeaseId),
eq(environmentLeases.status, "retained"), sql`${environmentLeases.metadata} ? ${ABANDONED_WORKSPACE_FILES_PARKED_KEY}`,
sql`not ${abandonedWorkspaceFilesHeldOnLeaseRowSql()}`)).returning({ id: environmentLeases.id }),
{ companyId: input.companyId });
const scheduled = unparked.length > 0;
const summary = input.reason === "retention_expired"
? `Unsaved files from an interrupted run were kept for ${days(keptDays)}, and their retention period has ended.${scheduled ? " They are scheduled for removal." : ""}`
: input.reason === "discarded"
? `Unsaved files from an interrupted run were discarded.${scheduled ? " They are scheduled for removal." : ""}`
: "Unsaved files from an interrupted run were saved to the task's recovered files.";
logger.info({ companyId: input.companyId, workspaceId: workspace.id, abandonedEpoch: input.epoch, reason: input.reason,
actorType: input.actorType, actorId: input.actorId, leasesHandedToCleanup: unparked.length, issueId, runId: runId ?? null,
summary }, "abandoned workspace files released");
if (issueId) await logActivity(tx, { companyId: input.companyId, actorType: input.actorType, actorId: input.actorId,
action: "workspace.abandoned_files_released", entityType: "issue", entityId: issueId, runId,
details: { summary, reason: input.reason, workspaceId: workspace.id, abandonedEpoch: input.epoch,
retainUntil: new Date(receiptRetainUntil(target)).toISOString(), leasesHandedToCleanup: unparked.length } });
return true;
});
}
export async function expireAbandonedWorkspaceFileHolds(db: Db, now = new Date()): Promise<{ expired: number }> {
const rows = await db.select({ id: executionWorkspaces.id, companyId: executionWorkspaces.companyId,
receipts: sql<unknown>`${executionWorkspaces.metadata}->'remoteWorkspaceAbandonedEpochs'` }).from(executionWorkspaces)
.where(sql`jsonb_typeof(${executionWorkspaces.metadata}->'remoteWorkspaceAbandonedEpochs') = 'array'
and exists (select 1 from jsonb_array_elements(${executionWorkspaces.metadata}->'remoteWorkspaceAbandonedEpochs') receipt
where receipt->>'schema' = ${REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA}
and coalesce(jsonb_typeof(receipt->'released'), 'null') = 'null')`);
let expired = 0;
let failed = 0;
const held = new Map<string, number>();
for (const row of rows) {
for (const receipt of Array.isArray(row.receipts) ? row.receipts : []) {
if (!unreleased(receipt)) continue;
if (receiptDueAt(receipt) > now.getTime() || typeof receipt.epoch !== "number") {
held.set(row.companyId, (held.get(row.companyId) ?? 0) + 1);
continue;
}
try {
if (await releaseAbandonedWorkspaceFiles(db, { companyId: row.companyId, workspaceId: row.id, epoch: receipt.epoch,
reason: "retention_expired", actorType: "system", actorId: "workspace-retention", now })) expired += 1;
} catch (error) {
failed += 1;
const retryAt = await deferAbandonedWorkspaceFilesExpiry(db, row.companyId, row.id, receipt, now);
logger.warn({ err: error, companyId: row.companyId, workspaceId: row.id, abandonedEpoch: receipt.epoch, retryAt },
"abandoned workspace file expiry failed; it retries later");
}
}
}
if (expired > 0 || failed > 0 || held.size > 0) {
const parked = await db.select({ companyId: environmentLeases.companyId, count: sql<number>`count(*)::int` })
.from(environmentLeases).where(sql`${environmentLeases.metadata} ? ${ABANDONED_WORKSPACE_FILES_PARKED_KEY}`)
.groupBy(environmentLeases.companyId);
logger.info({ expired, failed, retentionDays: ABANDONED_WORKSPACE_FILES_RETENTION_DAYS,
heldReceiptsByCompany: Object.fromEntries(held), parkedLeasesByCompany: Object.fromEntries(parked.map(p => [p.companyId, p.count])) },
"abandoned workspace file holds checked");
}
return { expired };
}
async function deferAbandonedWorkspaceFilesExpiry(db: Db, companyId: string, workspaceId: string, receipt: Receipt, now: Date) {
const attempts = (typeof receipt.expiryAttempts === "number" ? receipt.expiryAttempts : 0) + 1;
const retryAt = new Date(now.getTime() + Math.min(ABANDONED_WORKSPACE_FILES_EXPIRY_MAX_BACKOFF_MS,
ABANDONED_WORKSPACE_FILES_EXPIRY_BACKOFF_MS * 2 ** (attempts - 1))).toISOString();
try {
await db.transaction(async transaction => {
const tx = transaction as unknown as Db;
const [workspace] = await tx.select({ metadata: executionWorkspaces.metadata }).from(executionWorkspaces)
.where(and(eq(executionWorkspaces.companyId, companyId), eq(executionWorkspaces.id, workspaceId))).for("update");
const receipts = Array.isArray(workspace?.metadata?.remoteWorkspaceAbandonedEpochs)
? workspace!.metadata!.remoteWorkspaceAbandonedEpochs as unknown[] : [];
if (!workspace) return;
await tx.update(executionWorkspaces).set({ metadata: { ...workspace.metadata,
remoteWorkspaceAbandonedEpochs: receipts.map(value => unreleased(value) && value.epoch === receipt.epoch
&& value.location?.providerLeaseId === receipt.location?.providerLeaseId
? { ...value, expiryAttempts: attempts, expiryRetryAt: retryAt } : value) } })
.where(and(eq(executionWorkspaces.companyId, companyId), eq(executionWorkspaces.id, workspaceId)));
});
} catch (error) {
logger.warn({ err: error, companyId, workspaceId }, "abandoned workspace file expiry retry could not be recorded");
}
return retryAt;
}
function receiptExpirySql() {
return sql`coalesce(case when receipt->>'retainUntil' ~ '^[0-9]{4}-[0-9]{2}-[0-9]{2}T' then (receipt->>'retainUntil')::timestamptz end,
case when receipt->>'abandonedAt' ~ '^[0-9]{4}-[0-9]{2}-[0-9]{2}T'
then (receipt->>'abandonedAt')::timestamptz + ${`${ABANDONED_WORKSPACE_FILES_RETENTION_DAYS} days`}::interval end,
'epoch'::timestamptz)`;
}
function receiptDueSql() {
return sql`greatest(${receiptExpirySql()},
case when receipt->>'expiryRetryAt' ~ '^[0-9]{4}-[0-9]{2}-[0-9]{2}T' then (receipt->>'expiryRetryAt')::timestamptz
else 'epoch'::timestamptz end)`;
}
export async function nextAbandonedWorkspaceFileHoldExpiryAt(db: Pick<Db, "execute">): Promise<number | null> {
const [row] = await db.execute<{ next: string | Date | null }>(sql`select min(${receiptDueSql()}) as next
from execution_workspaces workspace
cross join lateral jsonb_array_elements(case when jsonb_typeof(workspace.metadata->'remoteWorkspaceAbandonedEpochs') = 'array'
then workspace.metadata->'remoteWorkspaceAbandonedEpochs' else '[]'::jsonb end) receipt
where workspace.metadata ? 'remoteWorkspaceAbandonedEpochs'
and receipt->>'schema' = ${REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA}
and jsonb_typeof(receipt->'location'->'providerLeaseId') = 'string'
and coalesce(jsonb_typeof(receipt->'released'), 'null') = 'null'`);
return row?.next ? new Date(row.next).getTime() : null;
}
export async function expireAbandonedWorkspaceFileHoldsWhenDue(db: Db, now = new Date()): Promise<{ expired: number }> {
const next = await nextAbandonedWorkspaceFileHoldExpiryAt(db);
if (next === null || next > now.getTime()) return { expired: 0 };
return await expireAbandonedWorkspaceFileHolds(db, now);
}
export async function recordAbandonedWorkspaceFilesRemoved(db: Db, lease: Pick<typeof environmentLeases.$inferSelect,
"id" | "companyId" | "metadata">): Promise<void> {
const marker = lease.metadata?.[ABANDONED_WORKSPACE_FILES_RELEASE_KEY] as {
issueId?: string | null; workspaceId?: string; abandonedEpoch?: number; reason?: string; keptDays?: number; runId?: string | null;
} | undefined;
if (!marker || (marker.reason !== "retention_expired" && marker.reason !== "discarded")) return;
const issueId = await existingIssueId(db, lease.companyId, marker.issueId ?? null);
const runId = await existingRunId(db, lease.companyId, marker.runId);
const summary = marker.reason === "retention_expired"
? `Unsaved files from an interrupted run were kept for ${days(marker.keptDays ?? ABANDONED_WORKSPACE_FILES_RETENTION_DAYS)} and have now been removed.`
: "Discarded unsaved files from an interrupted run have now been removed.";
logger.info({ companyId: lease.companyId, leaseId: lease.id, workspaceId: marker.workspaceId, issueId, summary },
"abandoned workspace files removed");
if (!issueId) return;
await logActivity(db, { companyId: lease.companyId, actorType: "system", actorId: "workspace-retention",
action: "workspace.abandoned_files_removed", entityType: "issue", entityId: issueId, runId,
details: { summary, reason: marker.reason, workspaceId: marker.workspaceId, abandonedEpoch: marker.abandonedEpoch, leaseId: lease.id } });
}
@@ -5,6 +5,8 @@ import { persistActivity } from "./activity-log.js";
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
export const REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA = "paperclip.remote-workspace-abandoned-epoch.v1";
export const ABANDONED_WORKSPACE_FILES_RETENTION_DAYS = 7;
export const ABANDONED_WORKSPACE_FILES_RETENTION_MS = ABANDONED_WORKSPACE_FILES_RETENTION_DAYS * 24 * 60 * 60 * 1000;
/**
* Ends an epoch whose unsaved files cannot be verified, without applying them.
@@ -35,9 +37,20 @@ export async function abandonUnverifiedRemoteWorkspaceEpoch(tx: Transaction, inp
const abandoned = { schema: REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA, epoch: current.epoch, runId: input.runId,
snapshotId: current.snapshotId, location, providerRef: workspace.providerRef,
...(workspace.metadata?.nativeWorkspaceMigration !== undefined ? { nativeWorkspaceMigration: workspace.metadata.nativeWorkspaceMigration } : {}),
abandonedAt: new Date().toISOString() };
abandonedAt: new Date().toISOString(), retainUntil: new Date(Date.now() + ABANDONED_WORKSPACE_FILES_RETENTION_MS).toISOString() };
const held = (receipt: unknown) => {
const value = receipt as { schema?: unknown; released?: unknown; location?: { providerLeaseId?: unknown } } | null;
return value?.schema === REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA && value.released == null
&& typeof value.location?.providerLeaseId === "string";
};
const sameRoot = (receipt: unknown) => {
const value = receipt as { location?: { providerLeaseId?: unknown; remoteCwd?: unknown } };
return held(receipt) && value.location?.providerLeaseId === location.providerLeaseId && value.location?.remoteCwd === location.remoteCwd;
};
const kept = [...prior.filter(receipt => !sameRoot(receipt)), abandoned];
const releasedHistory = kept.filter(receipt => !held(receipt)).slice(-20);
const metadata: Record<string, unknown> = { ...workspace.metadata, remoteWorkspace: next,
remoteWorkspaceAbandonedEpochs: [...prior, abandoned].slice(-20) };
remoteWorkspaceAbandonedEpochs: kept.filter(receipt => held(receipt) || releasedHistory.includes(receipt)) };
// The migration receipt names the old root as the next preparation's root.
delete metadata.nativeWorkspaceMigration;
const [saved] = await tx.update(executionWorkspaces).set({ providerRef: null, cwd: `/workspaces/${workspace.id}`, metadata,
@@ -7,6 +7,7 @@ import type { EnvironmentLease } from "@paperclipai/shared";
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
import { verifyRecoverableNativeHarnessBackup } from "./native-runtime/native-harness-backup-storage.js";
import { findPluginWorkerUnavailableError } from "./plugin-worker-unavailable-error.js";
import { abandonedWorkspaceFilesHeldOnLeaseRowSql, ABANDONED_WORKSPACE_FILES_PARKED_KEY, ABANDONED_WORKSPACE_FILES_PARKED_REASON } from "./remote-workspace-abandoned-files-hold.js";
import { sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
/**
@@ -131,13 +132,17 @@ export async function retireReplacedReusableSandboxLease(input: {
const { db, lease } = input;
if (!lease.providerLeaseId) return false;
const now = new Date();
const held = abandonedWorkspaceFilesHeldOnLeaseRowSql();
const [claimed] = await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({
status: "pending_cleanup",
status: sql`case when ${held} then 'retained' else 'pending_cleanup' end`,
cleanupStatus: "failed",
failureReason: REUSABLE_SANDBOX_RESUME_REPLACED_REASON,
failureReason: sql`case when ${held} then ${ABANDONED_WORKSPACE_FILES_PARKED_REASON} else ${REUSABLE_SANDBOX_RESUME_REPLACED_REASON} end`,
releasedAt: now,
lastUsedAt: now,
updatedAt: now,
metadata: sql`case when ${held} then coalesce(${environmentLeases.metadata}, '{}'::jsonb)
|| ${JSON.stringify({ [ABANDONED_WORKSPACE_FILES_PARKED_KEY]: { parkedAt: now.toISOString() } })}::jsonb
else ${environmentLeases.metadata} end`,
}).where(and(
eq(environmentLeases.id, lease.id),
eq(environmentLeases.companyId, input.companyId),