mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
fix(recovery): keep the sandbox that holds abandoned unsaved files out of replacement and cleanup (#21304)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in remote sandboxes, and the workspace files live in those sandboxes until a save > - When a run cannot resume exactly, the fresh continuation fallback (#21135) abandons the old workspace epoch and keeps its unsaved files "recorded and unapplied" on the old sandbox > - The abandonment clears the dirty flag, the writer and the location, so the workspace looks clean. No code reads the abandonment receipt > - A later failed resume can then replace the sandbox, and cleanup can destroy it. That sandbox holds the only copy of the unsaved files > - This pull request adds one shared hold. The task continues on a new sandbox, and the old sandbox is parked until its files are saved, discarded, or kept for 7 days > - The benefit is that the fallback keeps its promise. Abandoned files are not lost ## Linked Issues or Issue Description Refs #21135 Refs #21297 **What happened?** After the fresh continuation fallback abandoned a workspace epoch, the receipt in `remoteWorkspaceAbandonedEpochs` was write-only. `reusableSandboxReplacementIsSafe` saw a clean workspace with a committed save and allowed replacement. `retireReplacedReusableSandboxLease` then moved the old lease to `pending_cleanup`, and the cleanup sweep destroyed the sandbox with the unsaved files. `nativeWorkspaceExportPendingForLease` (the check that the sweep, the retained orphan settlement and the scoped destroy paths use) also answered "nothing pending". **What did you expect?** A sandbox that holds abandoned, unsaved files stays until the files are saved, a person discards them, or a stated retention period ends. **Steps to reproduce** 1. Abandon a remote workspace epoch whose location names sandbox L. 2. Let the next run fail to resume L. 3. The replacement decision says safe, L goes to `pending_cleanup`, and the sweep destroys it. ## What Changed - New `server/src/services/remote-workspace-abandoned-files-hold.ts` holds the single predicate for "this lease holds abandoned files". It is one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a workspace in its company has an abandoned-epoch receipt whose `location.providerLeaseId` equals the lease's provider lease id, and that receipt has no `released` object. A receipt written by the current release (no new fields) counts as unreleased. - Replace and park. A failed resume of a held sandbox still lets the run replace it, so the task continues at once on a new sandbox restored from the last committed save. `retireReplacedReusableSandboxLease` parks the old lease as `retained` with the `abandonedWorkspaceFilesParked` marker (failure reason `abandoned_workspace_files_held`) instead of `pending_cleanup`. `reusableLeaseCanBeResumed` and `reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never select it again. The check is a metadata read with no query, so the run path does no extra work. - Every path that can destroy a sandbox now honours the hold: - `nativeWorkspaceExportPendingForLease` answers pending for a held lease. This covers the pending cleanup sweep (defer), the orphaned active lease sweep, the retained orphan settlement `heldBy`, `destroyClaimedSandboxLease` (terminal workspace reaper `destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the environment delete path, and run lease release with `destroy`), the `destroyRunLease` dispatcher, and the plugin sandbox teardown retry. - The run lease settlement (`environment_run_lease_active`) is the one caller that ignores the hold (`{ destructive: false }`). It never destroys a sandbox. It stops the sandbox and keeps it, or it moves a lease without a sandbox to `pending_cleanup`, where the sweep parks a held lease. If it waited for the hold, a dead run's lease would stay `active` and block the retry for the whole retention period. - `destroyReusableSandboxLease` parks the lease and returns without a provider call. This covers obsolete reusable lease cleanup, the destroy after an expired or mismatched resume, release with `expired`, and the sandbox driver's `destroyRunLease`. - The `retryPendingSandboxTeardown` dispatcher refuses a destructive teardown of a held lease. Stop-only cleanup still runs, because a stop keeps the filesystem. - The retained orphan listing (`alwaysHeldSql`) uses the same SQL fragment, so held orphans are not polled. - Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released` with a reason (`recovered`, `discarded`, `retention_expired`) and an actor. It moves the parked leases for that sandbox to `pending_cleanup` when no other unreleased receipt holds them, and it writes a `workspace.abandoned_files_released` activity entry on the task in plain language. No export path for abandoned epochs exists today. The `recovered` reason is the hook for one; this pull request adds no new export mechanism. - Retention expiry. New receipts carry `retainUntil` (7 days). An old receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt` includes the earliest retention end of any unreleased receipt. The shared cleanup worker therefore wakes at that real deadline, and a restart rebuilds it from the database. At the deadline the sweep runs `expireAbandonedWorkspaceFileHolds`. This releases each due receipt as an attributed system decision (actor `workspace-retention`) and logs the held receipts and parked leases for each company. Each expiry always writes a task entry: "Unsaved files from an interrupted run were kept for N days, and their retention period has ended." The entry adds "They are scheduled for removal." only when a parked lease was queued for cleanup. A separate entry, "... kept for N days and have now been removed.", is written only after the provider confirms the delete. Each receipt is released in its own transaction. A failed receipt is logged and skipped, and it is retried after a backoff (5 minutes, doubling, at most 6 hours). The other receipts are still processed. The entries reference a run or a task only if it still exists. If the task was deleted, the decision goes to the server log only. - Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that an unreleased receipt holds goes back to `retained` and is parked by the sweep. A sandbox that a release before this change already destroyed cannot be recovered. Its receipt ends at its retention time like any other receipt. - Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch` trims only released receipts and keeps the last 20 of them. It never drops an unreleased receipt. A new abandonment of the same sandbox and root replaces the unreleased receipt for that root. So a workspace holds at most one unreleased receipt for each distinct sandbox and root abandoned in the last 7 days. ## Verification - `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19 tests). It follows the lease through the real acquisition path with a failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`: - `gives the next run a new sandbox at once and parks the held one, which cleanup and later runs leave alone` - `pulls a held lease already queued for cleanup back to retained, and leaves it on the provider` - `treats a receipt written before release fields existed as unreleased` - `records a retention expiry on the receipt` - `hands the parked lease to normal cleanup once the files are recovered` / `... discarded` - `removes the files only after the retention period, and says removed only after the provider confirms it` - `does not report removal when the provider delete fails` - `records the end of retention without scheduling removal when another receipt still holds the sandbox` - `records the end of retention when it expires before the lease was ever parked` - `wakes the cleanup worker at the receipt's retention end, also after a restart, and then goes quiet` - `releases a receipt whose run was deleted and records it without the run reference` - `releases a receipt whose task was deleted without writing a task entry` - `keeps expiring later receipts when one fails, and retries the failed one after a backoff` - `keeps a held sandbox when the retained orphan settlement reviews it` - `lists the sandbox for the retained orphan settlement again once the receipt is released` - `keeps holding the first sandbox after many later abandonments on another sandbox` - `replaces and cleans the sandbox as before when the receipt names a different sandbox` - `replaces and cleans the sandbox as before when no epoch was abandoned` - Red on the base revision (62d673cc3f): the park, upgrade, old-receipt, expiry, many-abandonments and scheduling tests fail. The controls pass. - Green: 19 of 19. Also `environment-runtime-reusable-resume-replacement.test.ts` 17/17, `environment-retained-orphan-sandbox.test.ts` 32/32, `heartbeat-pending-cleanup-sweep.test.ts` 51/51, `fresh-continuation-fallback.test.ts` 58/58. - `tsc --noEmit -p server/tsconfig.json` and a tsc pass over the new test file: clean. `environment-cleanup-work.test.ts` + `heartbeat-pending-cleanup-sweep.test.ts`: 80/80. - No existing test expectation was changed. Three CI failures onbb43f52came from this change. Two came from the hold blocking the run lease settlement: `ownership-reconciliation-report.test.ts > releases the lease and frees the workspace for the retry once a preparation-only epoch ends` and `remote-workspace-save-concurrency.test.ts > lets a retry prepare the workspace again after its first writer was killed during preparation`. The third, `native-session-executor.test.ts > restores a warm backup into a created lease only when a settled restore hold retained the stopped source`, came from a mocked database that has no `execute`; the predicate now runs through `select`. All three pass locally now. So do `heartbeat-orphaned-active-lease-sweep`, `environment-run-lease-ownership`, `native-workspace-warm-reuse` and `remote-workspaces.integration`. ## Risks - Cost of a parked sandbox. Parking makes no provider call and changes only the lease row. A sandbox that could not be resumed is not running. A reusable sandbox that was released is already stopped and kept. In both cases the cost is stopped storage for up to 7 days. One case is different: if a destructive path finds a held warm sandbox that is still running (`retained` after a keep-running release), the sandbox stays running until the receipt is released. The expiry log line counts parked leases for each company, so this cost can be seen. - Run lease release for an `ephemeral` lease calls the provider's release. Some providers delete the sandbox on that call. A held sandbox normally has a reusable lease, so this pull request does not gate that call. - No schema change. The receipt is JSON in `execution_workspaces.metadata`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edit) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e363b0e52d
commit
0477f622d6
10 files changed
+845
-12
No files matched your search
@@ -0,0 +1,524 @@
|
||||
import { JsonRpcCallError } from "@paperclipai/plugin-sdk";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import {
|
||||
activityLog,
|
||||
agents,
|
||||
companies,
|
||||
createDb,
|
||||
environmentLeases,
|
||||
environments,
|
||||
executionWorkspaceSnapshots,
|
||||
executionWorkspaces,
|
||||
heartbeatRuns,
|
||||
issues,
|
||||
plugins,
|
||||
projects,
|
||||
} from "@paperclipai/db";
|
||||
import type { Environment, EnvironmentLease } from "@paperclipai/shared";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { environmentRuntimeService } from "../services/environment-runtime.ts";
|
||||
import { environmentService } from "../services/environments.ts";
|
||||
import type { PluginWorkerManager } from "../services/plugin-worker-manager.ts";
|
||||
import { abandonUnverifiedRemoteWorkspaceEpoch } from "../services/remote-workspace-epoch-abandonment.ts";
|
||||
import { nativeWorkspaceExportPendingForLease } from "../services/native-runtime/native-workspace-finalization-state.ts";
|
||||
import { retireReplacedReusableSandboxLease } from "../services/reusable-sandbox-resume-replacement.ts";
|
||||
import { heartbeatService } from "../services/heartbeat.ts";
|
||||
import { nextEnvironmentCleanupAt } from "../services/environment-cleanup-state.ts";
|
||||
import { releasableRetainedOrphanCondition, settleRetainedOrphanSandbox } from "../services/environment-retained-orphan-settlement.ts";
|
||||
import {
|
||||
ABANDONED_WORKSPACE_FILES_PARKED_KEY, ABANDONED_WORKSPACE_FILES_PARKED_REASON, ABANDONED_WORKSPACE_FILES_RETENTION_DAYS,
|
||||
expireAbandonedWorkspaceFileHolds, nextAbandonedWorkspaceFileHoldExpiryAt, releaseAbandonedWorkspaceFiles,
|
||||
} from "../services/remote-workspace-abandoned-files-hold.ts";
|
||||
|
||||
vi.mock("../services/native-runtime/native-harness-backup-storage.js", () => ({
|
||||
verifyRecoverableNativeHarnessBackup: vi.fn(async () => true),
|
||||
}));
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const REFUSAL = "Daytona sandbox is still stopping at its start deadline";
|
||||
|
||||
describe.skipIf(!support.supported)("sandbox that holds files from an abandoned workspace epoch", () => {
|
||||
let stopDb: (() => Promise<void>) | null = null;
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let connectionString: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
const started = await startEmbeddedPostgresTestDatabase("abandoned-workspace-files-hold");
|
||||
stopDb = started.cleanup;
|
||||
connectionString = started.connectionString;
|
||||
});
|
||||
beforeEach(() => { db = createDb(connectionString); });
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks();
|
||||
await db.delete(activityLog);
|
||||
await db.delete(executionWorkspaceSnapshots);
|
||||
await db.delete(environmentLeases);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(issues);
|
||||
await db.delete(agents);
|
||||
await db.delete(environments);
|
||||
await db.delete(executionWorkspaces);
|
||||
await db.delete(plugins);
|
||||
await db.delete(projects);
|
||||
await db.delete(companies);
|
||||
await db.$client.end({ timeout: 1 });
|
||||
});
|
||||
afterAll(async () => { await stopDb?.(); });
|
||||
|
||||
async function seed() {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID();
|
||||
const abandonedRunId = randomUUID(), runId = randomUUID(), projectId = randomUUID(), issueId = randomUUID();
|
||||
const executionWorkspaceId = randomUUID();
|
||||
const providerLeaseId = randomUUID();
|
||||
const providerConfig = { provider: "fake-plugin", image: "fake:test", timeoutMs: 300_000, reuseLease: true };
|
||||
await db.insert(companies).values({ id: companyId, name: "Acme", status: "active", issuePrefix: `H${companyId.slice(0, 6)}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "active",
|
||||
adapterType: "paperclip_runner", adapterConfig: {}, runtimeConfig: {}, permissions: {} });
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Repair", status: "in_progress", assigneeAgentId: agentId });
|
||||
await db.insert(environments).values({ id: environmentId, name: "Paperclip Sandbox", driver: "sandbox", status: "active",
|
||||
config: providerConfig });
|
||||
await db.insert(plugins).values({
|
||||
id: pluginId, pluginKey: "acme.hold-provider", packageName: "@acme/hold-provider",
|
||||
version: "1.0.0", apiVersion: 1, categories: ["automation"], status: "ready", installOrder: 1, updatedAt: new Date(),
|
||||
manifestJson: {
|
||||
id: "acme.hold-provider", apiVersion: 1, version: "1.0.0", displayName: "Hold Provider",
|
||||
description: "Test provider", author: "Paperclip", categories: ["automation"],
|
||||
capabilities: ["environment.drivers.register"], entrypoints: { worker: "dist/worker.js" },
|
||||
environmentDrivers: [{ driverKey: "fake-plugin", kind: "sandbox_provider", displayName: "Fake Plugin",
|
||||
supportsReusableLeases: true,
|
||||
configSchema: { type: "object", properties: { image: { type: "string" }, timeoutMs: { type: "number" }, reuseLease: { type: "boolean" } } } }],
|
||||
},
|
||||
} as never);
|
||||
await db.insert(projects).values({ id: projectId, companyId, name: "Project", status: "active" });
|
||||
await db.insert(executionWorkspaces).values({ id: executionWorkspaceId, companyId, projectId, mode: "isolated_workspace",
|
||||
strategyType: "project_primary", name: "Task workspace", status: "active", providerType: "cloud_sandbox",
|
||||
cwd: `/workspaces/${executionWorkspaceId}`, sourceIssueId: issueId });
|
||||
await db.insert(heartbeatRuns).values([
|
||||
{ id: abandonedRunId, companyId, agentId, issueId, invocationSource: "assignment", status: "running" },
|
||||
{ id: runId, companyId, agentId, issueId, invocationSource: "assignment", status: "running" },
|
||||
]);
|
||||
const environment: Environment = { id: environmentId, name: "Paperclip Sandbox", description: null, driver: "sandbox",
|
||||
status: "active", config: providerConfig, envVars: {}, metadata: null, createdAt: new Date(), updatedAt: new Date() };
|
||||
const priming = environmentRuntimeService(db, { pluginWorkerManager: {
|
||||
isRunning: () => true,
|
||||
call: async (_pluginId: string, method: string) => {
|
||||
if (method !== "environmentAcquireLease") throw new Error(`Unexpected plugin method: ${method}`);
|
||||
return { providerLeaseId, metadata: { ...providerConfig, sandboxId: providerLeaseId, remoteCwd: "/home/daytona/paperclip-workspace" } };
|
||||
},
|
||||
getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }),
|
||||
} as unknown as PluginWorkerManager });
|
||||
const created = await priming.acquireRunLease({ companyId, environment, issueId, agentId, adapterType: "paperclip_runner",
|
||||
heartbeatRunId: abandonedRunId, persistedExecutionWorkspace: { id: executionWorkspaceId, mode: "isolated_workspace" } });
|
||||
await db.update(environmentLeases).set({ status: "retained" }).where(eq(environmentLeases.id, created.lease.id));
|
||||
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, abandonedRunId));
|
||||
const snapshotId = randomUUID();
|
||||
await db.insert(executionWorkspaceSnapshots).values({ id: snapshotId, companyId, executionWorkspaceId,
|
||||
sourceOperationId: randomUUID(), sourceRunId: abandonedRunId, generation: 1, epoch: 1, status: "committed" });
|
||||
const location = { environmentId, leaseId: created.lease.id, providerLeaseId,
|
||||
remoteCwd: "/home/daytona/paperclip-workspace/.paperclip-workspaces/abandoned" };
|
||||
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: {
|
||||
schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId, source: { kind: "empty" },
|
||||
writerRunId: abandonedRunId, dirty: true, location } } }).where(eq(executionWorkspaces.id, executionWorkspaceId));
|
||||
const lease = (await environmentService(db).getLeaseById(created.lease.id))!;
|
||||
return { companyId, agentId, issueId, environment, pluginId, abandonedRunId, runId, executionWorkspaceId,
|
||||
lease, providerLeaseId, snapshotId, location };
|
||||
}
|
||||
|
||||
type Seeded = Awaited<ReturnType<typeof seed>>;
|
||||
|
||||
async function abandon(seeded: Seeded) {
|
||||
await db.transaction(tx => abandonUnverifiedRemoteWorkspaceEpoch(tx, { companyId: seeded.companyId,
|
||||
workspaceId: seeded.executionWorkspaceId, runId: seeded.abandonedRunId, epoch: 1, snapshotId: seeded.snapshotId,
|
||||
location: seeded.location }));
|
||||
await reserveFor(seeded, seeded.runId);
|
||||
}
|
||||
|
||||
async function reserveFor(seeded: Seeded, runId: string) {
|
||||
const workspace = await readWorkspace(seeded);
|
||||
await db.update(executionWorkspaces).set({ metadata: { ...workspace.metadata,
|
||||
remoteWorkspace: { ...(workspace.metadata!.remoteWorkspace as Record<string, unknown>), writerRunId: runId } } })
|
||||
.where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
|
||||
}
|
||||
|
||||
async function readWorkspace(seeded: Seeded) {
|
||||
return (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, seeded.executionWorkspaceId)))[0]!;
|
||||
}
|
||||
|
||||
async function setReceipts(seeded: Seeded, patch: (receipt: Record<string, unknown>) => Record<string, unknown>) {
|
||||
const workspace = await readWorkspace(seeded);
|
||||
const receipts = workspace.metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>>;
|
||||
await db.update(executionWorkspaces).set({ metadata: { ...workspace.metadata,
|
||||
remoteWorkspaceAbandonedEpochs: receipts.map(patch) } }).where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
|
||||
}
|
||||
|
||||
function worker(seeded: Seeded) {
|
||||
const call = vi.fn(async (_pluginId: string, method: string, _params?: unknown) => {
|
||||
if (method === "environmentResumeLease") throw new JsonRpcCallError({ code: -32000, message: REFUSAL });
|
||||
if (method === "environmentAcquireLease") {
|
||||
return { providerLeaseId: "replacement-sandbox", metadata: { provider: "fake-plugin", image: "fake:test",
|
||||
timeoutMs: 300_000, reuseLease: true, remoteCwd: "/home/daytona/paperclip-workspace" } };
|
||||
}
|
||||
if (method === "environmentDestroyLease" || method === "environmentReleaseLease") return {};
|
||||
throw new Error(`Unexpected plugin method: ${method}`);
|
||||
});
|
||||
const manager = { isRunning: vi.fn((id: string) => id === seeded.pluginId), call,
|
||||
getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })),
|
||||
} as unknown as PluginWorkerManager;
|
||||
return { call, runtime: environmentRuntimeService(db, { pluginWorkerManager: manager }) };
|
||||
}
|
||||
|
||||
function acquire(seeded: Seeded, runtime: ReturnType<typeof environmentRuntimeService>) {
|
||||
let clock = Date.now();
|
||||
vi.spyOn(Date, "now").mockImplementation(() => (clock += 40_000));
|
||||
return runtime.acquireRunLease({ companyId: seeded.companyId, environment: seeded.environment, issueId: seeded.issueId,
|
||||
agentId: seeded.agentId, adapterType: "paperclip_runner", heartbeatRunId: seeded.runId,
|
||||
persistedExecutionWorkspace: { id: seeded.executionWorkspaceId, mode: "isolated_workspace" } });
|
||||
}
|
||||
|
||||
async function leaseRow(seeded: Seeded) {
|
||||
return (await environmentService(db).getLeaseById(seeded.lease.id))!;
|
||||
}
|
||||
|
||||
function destroyCalls(call: ReturnType<typeof worker>["call"]) {
|
||||
return call.mock.calls.filter(([, method]) => method === "environmentDestroyLease");
|
||||
}
|
||||
|
||||
async function sweep(runtime: ReturnType<typeof environmentRuntimeService>) {
|
||||
return await heartbeatService(db, { environmentRuntime: runtime as never }).sweepPendingCleanupLeases({ backoffMs: 0 });
|
||||
}
|
||||
|
||||
async function newRun(seeded: Seeded) {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId: seeded.companyId, agentId: seeded.agentId,
|
||||
issueId: seeded.issueId, invocationSource: "assignment", status: "running" });
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
await reserveFor(seeded, runId);
|
||||
return runId;
|
||||
}
|
||||
|
||||
it("gives the next run a new sandbox at once and parks the held one, which cleanup and later runs leave alone", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
const { call, runtime } = worker(seeded);
|
||||
|
||||
const acquired = await acquire(seeded, runtime);
|
||||
|
||||
expect(acquired.lease.providerLeaseId).toBe("replacement-sandbox");
|
||||
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "replacement", reason: "resume_failed" });
|
||||
const parked = await leaseRow(seeded);
|
||||
expect(parked).toMatchObject({ status: "retained", failureReason: ABANDONED_WORKSPACE_FILES_PARKED_REASON });
|
||||
expect(parked.metadata).toHaveProperty(ABANDONED_WORKSPACE_FILES_PARKED_KEY);
|
||||
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(true);
|
||||
|
||||
await sweep(runtime);
|
||||
await expect(runtime.destroyRunLease({ environment: seeded.environment, lease: parked,
|
||||
failureReason: "pending_cleanup_retry" })).resolves.toBeNull();
|
||||
await expect(runtime.retryPendingSandboxTeardown({ environment: seeded.environment, lease: parked })).rejects.toThrow();
|
||||
expect(destroyCalls(call)).toEqual([]);
|
||||
expect(await leaseRow(seeded)).toMatchObject({ status: "retained" });
|
||||
|
||||
await db.update(environmentLeases).set({ status: "released" }).where(eq(environmentLeases.providerLeaseId, "replacement-sandbox"));
|
||||
const nextRunId = await newRun(seeded);
|
||||
const later = worker(seeded);
|
||||
let clock = Date.now();
|
||||
vi.spyOn(Date, "now").mockImplementation(() => (clock += 40_000));
|
||||
later.call.mockImplementation(async (_pluginId: string, method: string) => {
|
||||
if (method === "environmentResumeLease") return { providerLeaseId: "replacement-sandbox", metadata: { provider: "fake-plugin",
|
||||
image: "fake:test", timeoutMs: 300_000, reuseLease: true, remoteCwd: "/home/daytona/paperclip-workspace" } };
|
||||
throw new Error(`Unexpected plugin method: ${method}`);
|
||||
});
|
||||
const next = await later.runtime.acquireRunLease({ companyId: seeded.companyId, environment: seeded.environment,
|
||||
issueId: seeded.issueId, agentId: seeded.agentId, adapterType: "paperclip_runner", heartbeatRunId: nextRunId,
|
||||
persistedExecutionWorkspace: { id: seeded.executionWorkspaceId, mode: "isolated_workspace" } });
|
||||
expect(next.lease.providerLeaseId).toBe("replacement-sandbox");
|
||||
const resumed = later.call.mock.calls.filter(([, method]) => method === "environmentResumeLease");
|
||||
expect(resumed.map(([, , params]) => (params as { providerLeaseId: string }).providerLeaseId)).toEqual(["replacement-sandbox"]);
|
||||
expect(await leaseRow(seeded)).toMatchObject({ status: "retained" });
|
||||
});
|
||||
|
||||
it("pulls a held lease already queued for cleanup back to retained, and leaves it on the provider", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", failureReason: "resume_failed" })
|
||||
.where(eq(environmentLeases.id, seeded.lease.id));
|
||||
const { call, runtime } = worker(seeded);
|
||||
await sweep(runtime);
|
||||
expect(destroyCalls(call)).toEqual([]);
|
||||
expect(await leaseRow(seeded)).toMatchObject({ status: "retained", failureReason: ABANDONED_WORKSPACE_FILES_PARKED_REASON });
|
||||
});
|
||||
|
||||
it("treats a receipt written before release fields existed as unreleased", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
await setReceipts(seeded, ({ retainUntil: _retainUntil, released: _released, ...receipt }) => receipt);
|
||||
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(true);
|
||||
expect(await retireReplacedReusableSandboxLease({ db, lease: seeded.lease as EnvironmentLease,
|
||||
companyId: seeded.companyId, heartbeatRunId: seeded.runId })).toBe(true);
|
||||
expect(await leaseRow(seeded)).toMatchObject({ status: "retained" });
|
||||
});
|
||||
|
||||
it("records a retention expiry on the receipt", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
const [receipt] = (await readWorkspace(seeded)).metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>>;
|
||||
expect(Date.parse(receipt!.retainUntil as string) - Date.parse(receipt!.abandonedAt as string))
|
||||
.toBe(ABANDONED_WORKSPACE_FILES_RETENTION_DAYS * 86_400_000);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ reason: "recovered" as const, actorType: "system" as const, actorId: "workspace-recovery" },
|
||||
{ reason: "discarded" as const, actorType: "user" as const, actorId: "board-user" },
|
||||
])("hands the parked lease to normal cleanup once the files are $reason", async ({ reason, actorType, actorId }) => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
const { call, runtime } = worker(seeded);
|
||||
await acquire(seeded, runtime);
|
||||
expect(await releaseAbandonedWorkspaceFiles(db, { companyId: seeded.companyId, workspaceId: seeded.executionWorkspaceId,
|
||||
epoch: 1, reason, actorType, actorId })).toBe(true);
|
||||
expect(await leaseRow(seeded)).toMatchObject({ status: "pending_cleanup" });
|
||||
vi.restoreAllMocks();
|
||||
await sweep(runtime);
|
||||
expect(destroyCalls(call).map(([, , params]) => (params as { providerLeaseId: string }).providerLeaseId))
|
||||
.toEqual([seeded.providerLeaseId]);
|
||||
expect(await leaseRow(seeded)).toMatchObject({ status: "expired" });
|
||||
});
|
||||
|
||||
async function entries(action: string) {
|
||||
return await db.select().from(activityLog).where(eq(activityLog.action, action));
|
||||
}
|
||||
|
||||
it("removes the files only after the retention period, and says removed only after the provider confirms it", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
const { call, runtime } = worker(seeded);
|
||||
await acquire(seeded, runtime);
|
||||
vi.restoreAllMocks();
|
||||
|
||||
expect(await expireAbandonedWorkspaceFileHolds(db, new Date(Date.now() + 86_400_000))).toEqual({ expired: 0 });
|
||||
await sweep(runtime);
|
||||
expect(destroyCalls(call)).toEqual([]);
|
||||
expect(await entries("workspace.abandoned_files_released")).toEqual([]);
|
||||
|
||||
expect(await expireAbandonedWorkspaceFileHolds(db, new Date(Date.now() + 8 * 86_400_000))).toEqual({ expired: 1 });
|
||||
const [released] = await entries("workspace.abandoned_files_released");
|
||||
expect(released).toMatchObject({ entityType: "issue", entityId: seeded.issueId, actorType: "system", actorId: "workspace-retention" });
|
||||
expect(released!.details).toMatchObject({ reason: "retention_expired",
|
||||
summary: "Unsaved files from an interrupted run were kept for 8 days, and their retention period has ended. They are scheduled for removal." });
|
||||
expect(await entries("workspace.abandoned_files_removed")).toEqual([]);
|
||||
|
||||
await sweep(runtime);
|
||||
expect(destroyCalls(call)).toHaveLength(1);
|
||||
expect(await leaseRow(seeded)).toMatchObject({ status: "expired" });
|
||||
const [removed] = await entries("workspace.abandoned_files_removed");
|
||||
expect(removed).toMatchObject({ entityType: "issue", entityId: seeded.issueId, actorType: "system" });
|
||||
expect(removed!.details).toMatchObject({
|
||||
summary: "Unsaved files from an interrupted run were kept for 8 days and have now been removed." });
|
||||
});
|
||||
|
||||
it("does not report removal when the provider delete fails", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
const { call, runtime } = worker(seeded);
|
||||
await acquire(seeded, runtime);
|
||||
vi.restoreAllMocks();
|
||||
await setReceipts(seeded, receipt => ({ ...receipt, retainUntil: new Date(Date.now() - 1_000).toISOString() }));
|
||||
call.mockImplementation(async (_pluginId: string, method: string) => {
|
||||
throw new Error(`provider refused ${method}`);
|
||||
});
|
||||
await sweep(runtime);
|
||||
expect(await entries("workspace.abandoned_files_released")).toHaveLength(1);
|
||||
expect(await entries("workspace.abandoned_files_removed")).toEqual([]);
|
||||
expect(await leaseRow(seeded)).toMatchObject({ status: "pending_cleanup" });
|
||||
});
|
||||
|
||||
it("records the end of retention without scheduling removal when another receipt still holds the sandbox", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
const { call, runtime } = worker(seeded);
|
||||
await acquire(seeded, runtime);
|
||||
vi.restoreAllMocks();
|
||||
const workspace = await readWorkspace(seeded);
|
||||
const [first] = workspace.metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>>;
|
||||
await db.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspaceAbandonedEpochs: [
|
||||
{ ...first, retainUntil: new Date(Date.now() - 1_000).toISOString() },
|
||||
{ ...first, epoch: 7, location: { ...(first!.location as Record<string, unknown>), remoteCwd: "/other-root" },
|
||||
retainUntil: new Date(Date.now() + 86_400_000).toISOString() },
|
||||
] } }).where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
|
||||
await sweep(runtime);
|
||||
const [ended] = await entries("workspace.abandoned_files_released");
|
||||
expect(ended).toMatchObject({ entityType: "issue", entityId: seeded.issueId, actorType: "system", actorId: "workspace-retention" });
|
||||
expect(ended!.details).toMatchObject({ reason: "retention_expired", leasesHandedToCleanup: 0,
|
||||
summary: expect.not.stringContaining("scheduled for removal") });
|
||||
expect((ended!.details as { summary: string }).summary).toContain("retention period has ended");
|
||||
expect(destroyCalls(call)).toEqual([]);
|
||||
expect(await leaseRow(seeded)).toMatchObject({ status: "retained" });
|
||||
});
|
||||
|
||||
it("records the end of retention when it expires before the lease was ever parked", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
const { runtime } = worker(seeded);
|
||||
await setReceipts(seeded, receipt => ({ ...receipt, retainUntil: new Date(Date.now() - 1_000).toISOString() }));
|
||||
await sweep(runtime);
|
||||
const [ended] = await entries("workspace.abandoned_files_released");
|
||||
expect(ended).toMatchObject({ entityType: "issue", entityId: seeded.issueId, actorId: "workspace-retention" });
|
||||
expect(ended!.details).toMatchObject({ reason: "retention_expired", leasesHandedToCleanup: 0 });
|
||||
expect((ended!.details as { summary: string }).summary).not.toContain("scheduled for removal");
|
||||
expect(await entries("workspace.abandoned_files_removed")).toEqual([]);
|
||||
});
|
||||
|
||||
it("releases a receipt whose run was deleted and records it without the run reference", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
await setReceipts(seeded, receipt => ({ ...receipt, runId: randomUUID(), retainUntil: new Date(Date.now() - 1_000).toISOString() }));
|
||||
expect(await expireAbandonedWorkspaceFileHolds(db)).toEqual({ expired: 1 });
|
||||
const [ended] = await entries("workspace.abandoned_files_released");
|
||||
expect(ended).toMatchObject({ entityType: "issue", entityId: seeded.issueId, runId: null });
|
||||
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(false);
|
||||
});
|
||||
|
||||
it("releases a receipt whose task was deleted without writing a task entry", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
await setReceipts(seeded, receipt => ({ ...receipt, runId: randomUUID(), retainUntil: new Date(Date.now() - 1_000).toISOString() }));
|
||||
await db.update(executionWorkspaces).set({ sourceIssueId: null }).where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
|
||||
expect(await expireAbandonedWorkspaceFileHolds(db)).toEqual({ expired: 1 });
|
||||
expect(await entries("workspace.abandoned_files_released")).toEqual([]);
|
||||
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(false);
|
||||
});
|
||||
|
||||
it("keeps expiring later receipts when one fails, and retries the failed one after a backoff", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
await abandonOnOtherSandbox(seeded, "/later-root");
|
||||
await setReceipts(seeded, receipt => ({ ...receipt, retainUntil: new Date(Date.now() - 1_000).toISOString() }));
|
||||
await db.execute(sql`create function reject_abandoned_epoch_one() returns trigger language plpgsql as $$
|
||||
begin if new.details->>'abandonedEpoch' = '1' then raise exception 'test activity failure'; end if; return new; end $$`);
|
||||
await db.execute(sql`create trigger reject_abandoned_epoch_one before insert on activity_log
|
||||
for each row execute function reject_abandoned_epoch_one()`);
|
||||
try {
|
||||
const before = Date.now();
|
||||
expect(await expireAbandonedWorkspaceFileHolds(db)).toEqual({ expired: 1 });
|
||||
const receipts = (await readWorkspace(seeded)).metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>>;
|
||||
const failed = receipts.find(receipt => receipt.epoch === 1)!;
|
||||
expect(failed.released).toBeUndefined();
|
||||
expect(failed.expiryAttempts).toBe(1);
|
||||
expect(receipts.find(receipt => receipt.epoch !== 1)!.released).toMatchObject({ reason: "retention_expired" });
|
||||
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(true);
|
||||
expect(await nextAbandonedWorkspaceFileHoldExpiryAt(db)).toBeGreaterThanOrEqual(before + 5 * 60_000);
|
||||
} finally {
|
||||
await db.execute(sql`drop trigger reject_abandoned_epoch_one on activity_log`);
|
||||
await db.execute(sql`drop function reject_abandoned_epoch_one()`);
|
||||
}
|
||||
expect(await expireAbandonedWorkspaceFileHolds(db, new Date(Date.now() + 10 * 60_000))).toEqual({ expired: 1 });
|
||||
});
|
||||
|
||||
it("wakes the cleanup worker at the receipt's retention end, also after a restart, and then goes quiet", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
const { call, runtime } = worker(seeded);
|
||||
await acquire(seeded, runtime);
|
||||
vi.restoreAllMocks();
|
||||
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success" })
|
||||
.where(eq(environmentLeases.providerLeaseId, "replacement-sandbox"));
|
||||
const [receipt] = (await readWorkspace(seeded)).metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>>;
|
||||
const retainUntil = Date.parse(receipt!.retainUntil as string);
|
||||
|
||||
expect(await nextEnvironmentCleanupAt(db)).toBe(retainUntil);
|
||||
const restarted = createDb(connectionString);
|
||||
try {
|
||||
expect(await nextEnvironmentCleanupAt(restarted)).toBe(retainUntil);
|
||||
} finally {
|
||||
await restarted.$client.end({ timeout: 1 });
|
||||
}
|
||||
|
||||
await setReceipts(seeded, value => ({ ...value, retainUntil: new Date(Date.now() - 1_000).toISOString() }));
|
||||
await sweep(runtime);
|
||||
expect(destroyCalls(call)).toHaveLength(1);
|
||||
expect(await entries("workspace.abandoned_files_removed")).toHaveLength(1);
|
||||
expect(await nextEnvironmentCleanupAt(db)).toBeNull();
|
||||
});
|
||||
|
||||
async function stoppedWarmOrphan(seeded: Seeded) {
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
await db.update(environmentLeases).set({ status: "failed", cleanupStatus: "success", leasePolicy: "reuse_by_environment",
|
||||
metadata: { ...seeded.lease.metadata, remoteExecutionTermination: { state: "stopped", leaseId: seeded.lease.id,
|
||||
runId: seeded.abandonedRunId } } }).where(eq(environmentLeases.id, seeded.lease.id));
|
||||
const listed = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(releasableRetainedOrphanCondition());
|
||||
const settled = await settleRetainedOrphanSandbox(db, { recordId: seeded.lease.id, runId: seeded.abandonedRunId } as never,
|
||||
{ now: new Date(), proof: { reason: "owner_gone" } as never });
|
||||
return { listed: listed.map(row => row.id), settled };
|
||||
}
|
||||
|
||||
it("keeps a held sandbox when the retained orphan settlement reviews it", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
const { listed, settled } = await stoppedWarmOrphan(seeded);
|
||||
expect(listed).toEqual([]);
|
||||
expect(settled).toEqual({ settled: false, reason: "workspace_preservation_pending" });
|
||||
expect(await leaseRow(seeded)).toMatchObject({ status: "failed" });
|
||||
});
|
||||
|
||||
it("lists the sandbox for the retained orphan settlement again once the receipt is released", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
await setReceipts(seeded, receipt => ({ ...receipt,
|
||||
released: { reason: "discarded", releasedAt: new Date().toISOString(), actorType: "user", actorId: "board-user" } }));
|
||||
const { listed, settled } = await stoppedWarmOrphan(seeded);
|
||||
expect(listed).toEqual([seeded.lease.id]);
|
||||
expect(settled.reason).not.toBe("workspace_preservation_pending");
|
||||
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(false);
|
||||
});
|
||||
|
||||
async function abandonOnOtherSandbox(seeded: Seeded, remoteCwd: string) {
|
||||
const workspace = await readWorkspace(seeded);
|
||||
const authority = workspace.metadata!.remoteWorkspace as Record<string, unknown> & { epoch: number };
|
||||
const location = { environmentId: seeded.environment.id, leaseId: randomUUID(), providerLeaseId: "sandbox-b", remoteCwd };
|
||||
await db.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspace: { ...authority,
|
||||
writerRunId: seeded.runId, dirty: true, location } } }).where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
|
||||
await db.transaction(tx => abandonUnverifiedRemoteWorkspaceEpoch(tx, { companyId: seeded.companyId,
|
||||
workspaceId: seeded.executionWorkspaceId, runId: seeded.runId, epoch: authority.epoch,
|
||||
snapshotId: authority.snapshotId as string, location }));
|
||||
}
|
||||
|
||||
it("keeps holding the first sandbox after many later abandonments on another sandbox", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
for (let index = 0; index < 21; index++) await abandonOnOtherSandbox(seeded, `/root-${index}`);
|
||||
for (let index = 0; index < 5; index++) await abandonOnOtherSandbox(seeded, "/root-0");
|
||||
const receipts = (await readWorkspace(seeded)).metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>>;
|
||||
expect(receipts).toHaveLength(22);
|
||||
expect(receipts.filter(receipt => (receipt.location as { remoteCwd: string }).remoteCwd === "/root-0")).toHaveLength(1);
|
||||
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(true);
|
||||
|
||||
expect(await releaseAbandonedWorkspaceFiles(db, { companyId: seeded.companyId, workspaceId: seeded.executionWorkspaceId,
|
||||
epoch: 1, reason: "discarded", actorType: "user", actorId: "board-user" })).toBe(true);
|
||||
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(false);
|
||||
});
|
||||
|
||||
it("replaces and cleans the sandbox as before when the receipt names a different sandbox", async () => {
|
||||
const seeded = await seed();
|
||||
await abandon(seeded);
|
||||
await setReceipts(seeded, receipt => ({ ...receipt,
|
||||
location: { ...(receipt.location as Record<string, unknown>), providerLeaseId: "another-sandbox" } }));
|
||||
const { runtime } = worker(seeded);
|
||||
const acquired = await acquire(seeded, runtime);
|
||||
expect(acquired.lease.providerLeaseId).toBe("replacement-sandbox");
|
||||
expect(await leaseRow(seeded)).toMatchObject({ status: "pending_cleanup", failureReason: "resume_failed" });
|
||||
expect(await nativeWorkspaceExportPendingForLease(db, seeded.lease)).toBe(false);
|
||||
});
|
||||
|
||||
it("replaces and cleans the sandbox as before when no epoch was abandoned", async () => {
|
||||
const seeded = await seed();
|
||||
const workspace = await readWorkspace(seeded);
|
||||
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: {
|
||||
...(workspace.metadata!.remoteWorkspace as Record<string, unknown>), epoch: 2, dirty: false, writerRunId: seeded.runId } } })
|
||||
.where(eq(executionWorkspaces.id, seeded.executionWorkspaceId));
|
||||
const { runtime } = worker(seeded);
|
||||
const acquired = await acquire(seeded, runtime);
|
||||
expect(acquired.lease.providerLeaseId).toBe("replacement-sandbox");
|
||||
expect(await leaseRow(seeded)).toMatchObject({ status: "pending_cleanup" });
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,7 @@
|
||||
import { and, eq, or, sql } from "drizzle-orm";
|
||||
import { environmentLeases, sandboxCreationOperations, type Db } from "@paperclipai/db";
|
||||
import { hasDeferredEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { nextAbandonedWorkspaceFileHoldExpiryAt } from "./remote-workspace-abandoned-files-hold.js";
|
||||
|
||||
export const ENVIRONMENT_CLEANUP_RETRY_MS = 5 * 60_000;
|
||||
|
||||
@@ -86,7 +87,8 @@ export async function nextEnvironmentCleanupAt(db: Db, backoffMs = ENVIRONMENT_C
|
||||
${pendingCleanupDeadlineSql()}) end)` }).from(environmentLeases).where(eq(environmentLeases.status, "pending_cleanup"));
|
||||
const [creation] = await db.select({ next: sql<string | null>`min(extract(epoch from ${sandboxCreationOperations.nextAttemptAt}) * 1000)` })
|
||||
.from(sandboxCreationOperations).where(recoverableSandboxCreationCondition());
|
||||
const deadlines = [row?.next, creation?.next].filter((value): value is string => value != null).map(Number);
|
||||
const expiry = await nextAbandonedWorkspaceFileHoldExpiryAt(db);
|
||||
const deadlines = [row?.next, creation?.next, expiry].filter((value): value is string | number => value != null).map(Number);
|
||||
const saved = deadlines.length ? Math.min(...deadlines) : null;
|
||||
const buffered = hasDeferredEnvironmentCleanup(db) ? Date.now() + backoffMs : null;
|
||||
return saved === null ? buffered : buffered === null ? saved : Math.min(saved, buffered);
|
||||
|
||||
@@ -2,6 +2,7 @@ import { and, eq, inArray, ne, notInArray, or, sql } from "drizzle-orm";
|
||||
import { environmentLeases, heartbeatRuns, issueRecoveryActions, type Db } from "@paperclipai/db";
|
||||
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { nativeWorkspaceExportPendingForLease } from "./native-runtime/native-workspace-finalization-state.js";
|
||||
import { abandonedWorkspaceFilesHeldOnLeaseRowSql } from "./remote-workspace-abandoned-files-hold.js";
|
||||
import type { OwnerProof } from "./owner-proof.js";
|
||||
import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
|
||||
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
@@ -111,6 +112,7 @@ function softlyHeldSql(now?: Date) {
|
||||
function alwaysHeldSql() {
|
||||
return sql`(${environmentLeases.metadata} ?| array['nativeWorkspaceExportResume', 'sandboxStopAndRetain',
|
||||
'workspaceRestoreRecovery', 'workspaceRecoveryCleanup', 'runnerTaskSourceConsumer']
|
||||
or ${abandonedWorkspaceFilesHeldOnLeaseRowSql()}
|
||||
or coalesce(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') = 'true'
|
||||
or coalesce(${environmentLeases.metadata}#>>'{legacyShutdownWorkspaceSettlement,state}' = 'pending', false)
|
||||
or exists (select 1 from heartbeat_runs run where run.id = ${environmentLeases.heartbeatRunId}
|
||||
|
||||
@@ -113,7 +113,7 @@ async function currentBootOwnsRun(tx: Db, lease: LeaseRow, runControllerBootId:
|
||||
}
|
||||
|
||||
async function keptByAnotherObligation(tx: Db, lease: LeaseRow): Promise<string | null> {
|
||||
if (await nativeWorkspaceExportPendingForLease(tx, lease)) return "workspace_export_pending";
|
||||
if (await nativeWorkspaceExportPendingForLease(tx, lease, { destructive: false })) return "workspace_export_pending";
|
||||
if (!(await allowLegacyShutdownWorkspaceCleanup(tx, lease))) return "legacy_shutdown_fence";
|
||||
// One provider resource id names one physical sandbox; another live lease
|
||||
// on it still owns that sandbox.
|
||||
|
||||
@@ -22,6 +22,7 @@ import { flushDeferredEnvironmentCleanup, registerDeferredEnvironmentCleanup, wr
|
||||
import { executeEnvironmentTask, pluginEnvironmentSupportsTasks, settleDispatchedEnvironmentTask } from "./environment-task-runtime.js";
|
||||
import { allowLegacyShutdownWorkspaceCleanup, legacyShutdownWorkspaceResourceProtected } from "./legacy-shutdown-workspace-settlement.js";
|
||||
import { nativeWorkspaceExportPendingForLease, nativeWorkspaceExportPendingForResource } from "./native-runtime/native-workspace-finalization-state.js";
|
||||
import { abandonedWorkspaceFilesHeldOnLease, abandonedWorkspaceFilesParked, parkAbandonedWorkspaceFilesLease } from "./remote-workspace-abandoned-files-hold.js";
|
||||
import { beginIdleTrackedWork } from "./task-admission.js";
|
||||
import { historicalProviderCredentials } from "./historical-provider-credentials.js";
|
||||
import { ownedWorkspaceRecoveryLease, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
|
||||
@@ -1137,14 +1138,16 @@ async function waitForSettlingReusableLease(db: Db, input: {
|
||||
}
|
||||
|
||||
function reusableLeaseCanBeResumed(input: {
|
||||
lease: Pick<EnvironmentLease, "status" | "heartbeatRunId">;
|
||||
lease: Pick<EnvironmentLease, "status" | "heartbeatRunId" | "metadata">;
|
||||
heartbeatRunId: string | null;
|
||||
}): boolean {
|
||||
if (abandonedWorkspaceFilesParked(input.lease)) return false;
|
||||
if (input.lease.status === "released" || input.lease.status === "retained") return true;
|
||||
return input.lease.status === "active" && input.heartbeatRunId !== null && input.lease.heartbeatRunId === input.heartbeatRunId;
|
||||
}
|
||||
|
||||
function reusableLeaseCanBeCleanedUp(lease: Pick<EnvironmentLease, "status">): boolean {
|
||||
function reusableLeaseCanBeCleanedUp(lease: Pick<EnvironmentLease, "status" | "metadata">): boolean {
|
||||
if (abandonedWorkspaceFilesParked(lease)) return false;
|
||||
return lease.status === "released" || lease.status === "retained";
|
||||
}
|
||||
|
||||
@@ -3214,6 +3217,10 @@ function createSandboxEnvironmentDriver(
|
||||
failureReason: string;
|
||||
}): Promise<EnvironmentLease | null> {
|
||||
if (await legacyShutdownWorkspaceResourceProtected(db, input.lease)) return null;
|
||||
if (await abandonedWorkspaceFilesHeldOnLease(db, input.lease)) {
|
||||
await parkAbandonedWorkspaceFilesLease(db, input.lease, ["released", "retained", "failed", "pending_cleanup"]);
|
||||
return null;
|
||||
}
|
||||
let cleanupStatus: "success" | "failed" = "success";
|
||||
let termination: ReturnType<typeof remoteTerminationReceipt>;
|
||||
const metadata = input.lease.metadata ?? {};
|
||||
@@ -4341,6 +4348,9 @@ export function environmentRuntimeService(
|
||||
if (!(await allowLegacyShutdownWorkspaceCleanup(db, input.lease))) {
|
||||
throw new Error("Shutdown workspace settlement is still pending.");
|
||||
}
|
||||
if (!hasStopOnlyCleanup(input.lease) && await abandonedWorkspaceFilesHeldOnLease(db, input.lease)) {
|
||||
throw new Error("Unsaved files from an abandoned workspace are held on this sandbox; destructive cleanup is forbidden.");
|
||||
}
|
||||
if (input.lease.metadata?.legacyShutdownWorkspaceSettlement) {
|
||||
const current = await environmentsSvc.getLeaseById(input.lease.id);
|
||||
if (!current) throw new Error("Shutdown workspace source is unavailable.");
|
||||
|
||||
@@ -31,6 +31,7 @@ import { createHeartbeatAgentStatus } from "./agent-status.js";
|
||||
import { preserveWorkspaceRestoreRecoveryMetadataSql } from "../legacy-workspace-restore-recovery.js";
|
||||
import { preserveCancelledNativeWorkspaces, preserveOrphanedNativeWorkspace } from "../native-runtime/native-failed-workspace-recovery.js";
|
||||
import { nativeWorkspaceExportPendingForLease, pendingNativeWorkspaceFinalizationCondition } from "../native-runtime/native-workspace-finalization-state.js";
|
||||
import { expireAbandonedWorkspaceFileHoldsWhenDue, parkAbandonedWorkspaceFilesLease, recordAbandonedWorkspaceFilesRemoved } from "../remote-workspace-abandoned-files-hold.js";
|
||||
import { CHAT_COMPLETION_WAKE_REASON } from "../chat-completion-delivery.js";
|
||||
import {
|
||||
hasStopOnlyCleanup,
|
||||
@@ -1747,6 +1748,10 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
logger.warn({ errorKind: "teardown_claim_settle_failed" }, "teardown claim settle failed; a later sweep retries");
|
||||
});
|
||||
|
||||
if (!manualRetry) await expireAbandonedWorkspaceFileHoldsWhenDue(db).catch(() => {
|
||||
logger.warn({ errorKind: "abandoned_workspace_files_expiry_failed" }, "abandoned workspace file expiry failed; a later sweep retries");
|
||||
});
|
||||
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(environmentLeases)
|
||||
@@ -1767,6 +1772,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
let capped = 0;
|
||||
for (const row of rows) {
|
||||
if (pendingCleanupAttemptsInFlight.has(row.id)) continue;
|
||||
if (!hasStopOnlyCleanup(row) && await parkAbandonedWorkspaceFilesLease(db, row, ["pending_cleanup"])) continue;
|
||||
if (!hasStopOnlyCleanup(row) && await nativeWorkspaceExportPendingForLease(db, row)) {
|
||||
await deferPendingCleanupLease(row.id);
|
||||
continue;
|
||||
@@ -1881,7 +1887,10 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
failureReason: "pending_cleanup_retry",
|
||||
remoteExecutionTermination: remoteTerminationReceipt(lease, receipt),
|
||||
});
|
||||
if (released) destroyed += 1;
|
||||
if (released) {
|
||||
destroyed += 1;
|
||||
await recordAbandonedWorkspaceFilesRemoved(db, row);
|
||||
}
|
||||
} else if (environment) {
|
||||
const result = await environmentRuntime.destroyRunLease({
|
||||
environment,
|
||||
@@ -1890,6 +1899,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
});
|
||||
if (result && result.status !== "pending_cleanup") {
|
||||
destroyed += 1;
|
||||
if (result.cleanupStatus === "success") await recordAbandonedWorkspaceFilesRemoved(db, row);
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
|
||||
@@ -3,6 +3,7 @@ import { and, eq, exists, sql } from "drizzle-orm";
|
||||
import { hasRequiredWorkspaceRecovery, nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js";
|
||||
import { recordRequiredWorkspaceRestoreFailure } from "../legacy-execution-recovery.js";
|
||||
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
|
||||
import { abandonedWorkspaceFilesHeldOnLease } from "../remote-workspace-abandoned-files-hold.js";
|
||||
import { matchPreparedRemoteWorkspaceSource, preparedRemoteWorkspaceSource, type WorkspaceSourceLease } from "../remote-workspace-lifecycle.js";
|
||||
|
||||
/** File-only recovery for setup that never reached native admission. The live
|
||||
@@ -79,15 +80,19 @@ function nativeWorkspaceSourcePending(run: Pick<typeof heartbeatRuns.$inferSelec
|
||||
&& reference.providerLeaseId === lease.providerLeaseId;
|
||||
}
|
||||
|
||||
export async function nativeWorkspaceExportPendingForLease(db: Db, lease: WorkspaceSourceLease): Promise<boolean> {
|
||||
if (!lease.heartbeatRunId || !lease.provider || lease.provider === "local" || !lease.providerLeaseId) return false;
|
||||
export async function nativeWorkspaceExportPendingForLease(db: Db, lease: WorkspaceSourceLease,
|
||||
options: { destructive?: boolean } = {}): Promise<boolean> {
|
||||
if (!lease.provider || lease.provider === "local" || !lease.providerLeaseId) return false;
|
||||
const held = () => options.destructive === false ? Promise.resolve(false) : abandonedWorkspaceFilesHeldOnLease(db, lease);
|
||||
if (!lease.heartbeatRunId) return await held();
|
||||
const [run] = await db.select(nativeWorkspaceRunProjection()).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, lease.heartbeatRunId), eq(heartbeatRuns.companyId, lease.companyId),
|
||||
eq(heartbeatRuns.runtimeMode, "native"),
|
||||
)).limit(1);
|
||||
if (run?.observedRemoteWorkspace) return true;
|
||||
if (await preparedRemoteWorkspaceSource(db, lease)) return true;
|
||||
return Boolean(run && nativeWorkspaceSourcePending(run, lease));
|
||||
if (run && nativeWorkspaceSourcePending(run, lease)) return true;
|
||||
return await held();
|
||||
}
|
||||
|
||||
export async function nativeWorkspaceExportPendingForResource(db: Db, lease: WorkspaceSourceLease, resumingRunId: string | null) {
|
||||
|
||||
@@ -0,0 +1,262 @@
|
||||
import { and, eq, inArray, sql, type SQL } from "drizzle-orm";
|
||||
import { companies, environmentLeases, executionWorkspaces, heartbeatRuns, issues, type Db } from "@paperclipai/db";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { logActivity } from "./activity-log.js";
|
||||
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import {
|
||||
ABANDONED_WORKSPACE_FILES_RETENTION_DAYS, ABANDONED_WORKSPACE_FILES_RETENTION_MS, REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA,
|
||||
} from "./remote-workspace-epoch-abandonment.js";
|
||||
|
||||
export { ABANDONED_WORKSPACE_FILES_RETENTION_DAYS, ABANDONED_WORKSPACE_FILES_RETENTION_MS };
|
||||
export const ABANDONED_WORKSPACE_FILES_PARKED_KEY = "abandonedWorkspaceFilesParked";
|
||||
export const ABANDONED_WORKSPACE_FILES_PARKED_REASON = "abandoned_workspace_files_held";
|
||||
export const ABANDONED_WORKSPACE_FILES_RELEASED_REASON = "abandoned_workspace_files_released";
|
||||
export const ABANDONED_WORKSPACE_FILES_RELEASE_KEY = "abandonedWorkspaceFilesRelease";
|
||||
const ABANDONED_WORKSPACE_FILES_EXPIRY_BACKOFF_MS = 5 * 60_000;
|
||||
const ABANDONED_WORKSPACE_FILES_EXPIRY_MAX_BACKOFF_MS = 6 * 60 * 60_000;
|
||||
|
||||
async function existingIssueId(db: Db, companyId: string, issueId: string | null): Promise<string | null> {
|
||||
if (!issueId) return null;
|
||||
const [issue] = await db.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)));
|
||||
return issue?.id ?? null;
|
||||
}
|
||||
|
||||
async function existingRunId(db: Db, companyId: string, runId: string | null | undefined): Promise<string | undefined> {
|
||||
if (!runId) return undefined;
|
||||
const [run] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId)));
|
||||
return run?.id;
|
||||
}
|
||||
|
||||
function days(count: number) {
|
||||
return `${count} ${count === 1 ? "day" : "days"}`;
|
||||
}
|
||||
|
||||
export type AbandonedWorkspaceFilesReleaseReason = "recovered" | "discarded" | "retention_expired";
|
||||
|
||||
type HoldLease = Pick<typeof environmentLeases.$inferSelect, "companyId" | "provider" | "providerLeaseId">;
|
||||
type Receipt = Record<string, unknown> & { location?: { providerLeaseId?: unknown }; released?: unknown };
|
||||
|
||||
export function abandonedWorkspaceFilesHeldSql(companyId: SQL | string, providerLeaseId: SQL | string): SQL {
|
||||
return sql`exists (select 1 from execution_workspaces held_workspace
|
||||
cross join lateral jsonb_array_elements(case when jsonb_typeof(held_workspace.metadata->'remoteWorkspaceAbandonedEpochs') = 'array'
|
||||
then held_workspace.metadata->'remoteWorkspaceAbandonedEpochs' else '[]'::jsonb end) held_receipt
|
||||
where held_workspace.company_id = ${companyId}
|
||||
and held_workspace.metadata ? 'remoteWorkspaceAbandonedEpochs'
|
||||
and held_receipt->>'schema' = ${REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA}
|
||||
and held_receipt->'location'->>'providerLeaseId' = ${providerLeaseId}
|
||||
and coalesce(jsonb_typeof(held_receipt->'released'), 'null') = 'null')`;
|
||||
}
|
||||
|
||||
export function abandonedWorkspaceFilesHeldOnLeaseRowSql(): SQL {
|
||||
return sql`(${environmentLeases.providerLeaseId} is not null and coalesce(${environmentLeases.provider}, 'local') <> 'local'
|
||||
and ${abandonedWorkspaceFilesHeldSql(sql`${environmentLeases.companyId}`, sql`${environmentLeases.providerLeaseId}`)})`;
|
||||
}
|
||||
|
||||
export async function abandonedWorkspaceFilesHeldOnLease(db: Pick<Db, "select">, lease: HoldLease): Promise<boolean> {
|
||||
if (!lease.providerLeaseId || !lease.provider || lease.provider === "local") return false;
|
||||
const [row] = await db.select({ held: sql<boolean>`${abandonedWorkspaceFilesHeldSql(sql`${companies.id}`, lease.providerLeaseId)}` })
|
||||
.from(companies).where(eq(companies.id, lease.companyId)).limit(1);
|
||||
return row?.held === true;
|
||||
}
|
||||
|
||||
export function abandonedWorkspaceFilesParked(lease: { metadata?: Record<string, unknown> | null }): boolean {
|
||||
return lease.metadata?.[ABANDONED_WORKSPACE_FILES_PARKED_KEY] != null;
|
||||
}
|
||||
|
||||
export async function parkAbandonedWorkspaceFilesLease(db: Db, lease: Pick<typeof environmentLeases.$inferSelect,
|
||||
"id" | "companyId" | "heartbeatRunId">, statuses: string[]): Promise<boolean> {
|
||||
const now = new Date();
|
||||
const parked = await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({
|
||||
status: "retained", cleanupStatus: "failed", failureReason: ABANDONED_WORKSPACE_FILES_PARKED_REASON, updatedAt: now,
|
||||
metadata: sql`(case when jsonb_typeof(${environmentLeases.metadata}) = 'object' then ${environmentLeases.metadata} else '{}'::jsonb end)
|
||||
|| ${JSON.stringify({ [ABANDONED_WORKSPACE_FILES_PARKED_KEY]: { parkedAt: now.toISOString() }, pendingCleanupInFlight: false })}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),
|
||||
inArray(environmentLeases.status, statuses),
|
||||
sql`coalesce(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') != 'true'`,
|
||||
abandonedWorkspaceFilesHeldOnLeaseRowSql())).returning({ id: environmentLeases.id }),
|
||||
{ companyId: lease.companyId, runId: lease.heartbeatRunId ?? undefined, leaseId: lease.id });
|
||||
return parked.length > 0;
|
||||
}
|
||||
|
||||
function receiptDueAt(receipt: Receipt): number {
|
||||
const retry = typeof receipt.expiryRetryAt === "string" ? Date.parse(receipt.expiryRetryAt) : Number.NaN;
|
||||
return Math.max(receiptRetainUntil(receipt), Number.isFinite(retry) ? retry : 0);
|
||||
}
|
||||
|
||||
function receiptRetainUntil(receipt: Receipt): number {
|
||||
const explicit = typeof receipt.retainUntil === "string" ? Date.parse(receipt.retainUntil) : Number.NaN;
|
||||
if (Number.isFinite(explicit)) return explicit;
|
||||
const abandonedAt = typeof receipt.abandonedAt === "string" ? Date.parse(receipt.abandonedAt) : Number.NaN;
|
||||
return (Number.isFinite(abandonedAt) ? abandonedAt : 0) + ABANDONED_WORKSPACE_FILES_RETENTION_MS;
|
||||
}
|
||||
|
||||
function unreleased(receipt: unknown): receipt is Receipt {
|
||||
const value = receipt as Receipt | null;
|
||||
return value?.schema === REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA && value.released == null
|
||||
&& typeof value.location?.providerLeaseId === "string";
|
||||
}
|
||||
|
||||
export async function releaseAbandonedWorkspaceFiles(db: Db, input: {
|
||||
companyId: string; workspaceId: string; epoch: number; reason: AbandonedWorkspaceFilesReleaseReason;
|
||||
actorType: "system" | "user" | "agent"; actorId: string; now?: Date;
|
||||
}): Promise<boolean> {
|
||||
const now = input.now ?? new Date();
|
||||
return db.transaction(async transaction => {
|
||||
const tx = transaction as unknown as Db;
|
||||
const [workspace] = await tx.select().from(executionWorkspaces).where(and(
|
||||
eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId))).for("update");
|
||||
const receipts = Array.isArray(workspace?.metadata?.remoteWorkspaceAbandonedEpochs)
|
||||
? workspace!.metadata!.remoteWorkspaceAbandonedEpochs as unknown[] : [];
|
||||
const target = receipts.find(receipt => unreleased(receipt) && receipt.epoch === input.epoch) as Receipt | undefined;
|
||||
if (!workspace || !target) return false;
|
||||
const released = { reason: input.reason, releasedAt: now.toISOString(), actorType: input.actorType, actorId: input.actorId };
|
||||
await tx.update(executionWorkspaces).set({ updatedAt: now, metadata: { ...workspace.metadata,
|
||||
remoteWorkspaceAbandonedEpochs: receipts.map(receipt => receipt === target ? { ...target, released } : receipt) } })
|
||||
.where(and(eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, workspace.id)));
|
||||
const providerLeaseId = target.location!.providerLeaseId as string;
|
||||
const recordedRunId = typeof target.runId === "string" ? target.runId : undefined;
|
||||
const [run] = recordedRunId ? await tx.select({ id: heartbeatRuns.id, issueId: heartbeatRuns.issueId }).from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, recordedRunId))) : [];
|
||||
const runId = run?.id;
|
||||
const issueId = await existingIssueId(tx, input.companyId, workspace.sourceIssueId ?? run?.issueId ?? null);
|
||||
const keptDays = Math.max(1, Math.round((now.getTime() - Date.parse(String(target.abandonedAt ?? now.toISOString()))) / 86_400_000));
|
||||
const marker = { issueId, workspaceId: workspace.id, abandonedEpoch: input.epoch, reason: input.reason, keptDays, runId: runId ?? null };
|
||||
const unparked = await writeEnvironmentCleanup(tx, write => write.update(environmentLeases).set({
|
||||
status: "pending_cleanup", cleanupStatus: "failed", failureReason: ABANDONED_WORKSPACE_FILES_RELEASED_REASON,
|
||||
releasedAt: now, updatedAt: now,
|
||||
metadata: sql`(${environmentLeases.metadata} - ${ABANDONED_WORKSPACE_FILES_PARKED_KEY})
|
||||
|| ${JSON.stringify({ [ABANDONED_WORKSPACE_FILES_RELEASE_KEY]: marker })}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.providerLeaseId, providerLeaseId),
|
||||
eq(environmentLeases.status, "retained"), sql`${environmentLeases.metadata} ? ${ABANDONED_WORKSPACE_FILES_PARKED_KEY}`,
|
||||
sql`not ${abandonedWorkspaceFilesHeldOnLeaseRowSql()}`)).returning({ id: environmentLeases.id }),
|
||||
{ companyId: input.companyId });
|
||||
const scheduled = unparked.length > 0;
|
||||
const summary = input.reason === "retention_expired"
|
||||
? `Unsaved files from an interrupted run were kept for ${days(keptDays)}, and their retention period has ended.${scheduled ? " They are scheduled for removal." : ""}`
|
||||
: input.reason === "discarded"
|
||||
? `Unsaved files from an interrupted run were discarded.${scheduled ? " They are scheduled for removal." : ""}`
|
||||
: "Unsaved files from an interrupted run were saved to the task's recovered files.";
|
||||
logger.info({ companyId: input.companyId, workspaceId: workspace.id, abandonedEpoch: input.epoch, reason: input.reason,
|
||||
actorType: input.actorType, actorId: input.actorId, leasesHandedToCleanup: unparked.length, issueId, runId: runId ?? null,
|
||||
summary }, "abandoned workspace files released");
|
||||
if (issueId) await logActivity(tx, { companyId: input.companyId, actorType: input.actorType, actorId: input.actorId,
|
||||
action: "workspace.abandoned_files_released", entityType: "issue", entityId: issueId, runId,
|
||||
details: { summary, reason: input.reason, workspaceId: workspace.id, abandonedEpoch: input.epoch,
|
||||
retainUntil: new Date(receiptRetainUntil(target)).toISOString(), leasesHandedToCleanup: unparked.length } });
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
export async function expireAbandonedWorkspaceFileHolds(db: Db, now = new Date()): Promise<{ expired: number }> {
|
||||
const rows = await db.select({ id: executionWorkspaces.id, companyId: executionWorkspaces.companyId,
|
||||
receipts: sql<unknown>`${executionWorkspaces.metadata}->'remoteWorkspaceAbandonedEpochs'` }).from(executionWorkspaces)
|
||||
.where(sql`jsonb_typeof(${executionWorkspaces.metadata}->'remoteWorkspaceAbandonedEpochs') = 'array'
|
||||
and exists (select 1 from jsonb_array_elements(${executionWorkspaces.metadata}->'remoteWorkspaceAbandonedEpochs') receipt
|
||||
where receipt->>'schema' = ${REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA}
|
||||
and coalesce(jsonb_typeof(receipt->'released'), 'null') = 'null')`);
|
||||
let expired = 0;
|
||||
let failed = 0;
|
||||
const held = new Map<string, number>();
|
||||
for (const row of rows) {
|
||||
for (const receipt of Array.isArray(row.receipts) ? row.receipts : []) {
|
||||
if (!unreleased(receipt)) continue;
|
||||
if (receiptDueAt(receipt) > now.getTime() || typeof receipt.epoch !== "number") {
|
||||
held.set(row.companyId, (held.get(row.companyId) ?? 0) + 1);
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
if (await releaseAbandonedWorkspaceFiles(db, { companyId: row.companyId, workspaceId: row.id, epoch: receipt.epoch,
|
||||
reason: "retention_expired", actorType: "system", actorId: "workspace-retention", now })) expired += 1;
|
||||
} catch (error) {
|
||||
failed += 1;
|
||||
const retryAt = await deferAbandonedWorkspaceFilesExpiry(db, row.companyId, row.id, receipt, now);
|
||||
logger.warn({ err: error, companyId: row.companyId, workspaceId: row.id, abandonedEpoch: receipt.epoch, retryAt },
|
||||
"abandoned workspace file expiry failed; it retries later");
|
||||
}
|
||||
}
|
||||
}
|
||||
if (expired > 0 || failed > 0 || held.size > 0) {
|
||||
const parked = await db.select({ companyId: environmentLeases.companyId, count: sql<number>`count(*)::int` })
|
||||
.from(environmentLeases).where(sql`${environmentLeases.metadata} ? ${ABANDONED_WORKSPACE_FILES_PARKED_KEY}`)
|
||||
.groupBy(environmentLeases.companyId);
|
||||
logger.info({ expired, failed, retentionDays: ABANDONED_WORKSPACE_FILES_RETENTION_DAYS,
|
||||
heldReceiptsByCompany: Object.fromEntries(held), parkedLeasesByCompany: Object.fromEntries(parked.map(p => [p.companyId, p.count])) },
|
||||
"abandoned workspace file holds checked");
|
||||
}
|
||||
return { expired };
|
||||
}
|
||||
|
||||
async function deferAbandonedWorkspaceFilesExpiry(db: Db, companyId: string, workspaceId: string, receipt: Receipt, now: Date) {
|
||||
const attempts = (typeof receipt.expiryAttempts === "number" ? receipt.expiryAttempts : 0) + 1;
|
||||
const retryAt = new Date(now.getTime() + Math.min(ABANDONED_WORKSPACE_FILES_EXPIRY_MAX_BACKOFF_MS,
|
||||
ABANDONED_WORKSPACE_FILES_EXPIRY_BACKOFF_MS * 2 ** (attempts - 1))).toISOString();
|
||||
try {
|
||||
await db.transaction(async transaction => {
|
||||
const tx = transaction as unknown as Db;
|
||||
const [workspace] = await tx.select({ metadata: executionWorkspaces.metadata }).from(executionWorkspaces)
|
||||
.where(and(eq(executionWorkspaces.companyId, companyId), eq(executionWorkspaces.id, workspaceId))).for("update");
|
||||
const receipts = Array.isArray(workspace?.metadata?.remoteWorkspaceAbandonedEpochs)
|
||||
? workspace!.metadata!.remoteWorkspaceAbandonedEpochs as unknown[] : [];
|
||||
if (!workspace) return;
|
||||
await tx.update(executionWorkspaces).set({ metadata: { ...workspace.metadata,
|
||||
remoteWorkspaceAbandonedEpochs: receipts.map(value => unreleased(value) && value.epoch === receipt.epoch
|
||||
&& value.location?.providerLeaseId === receipt.location?.providerLeaseId
|
||||
? { ...value, expiryAttempts: attempts, expiryRetryAt: retryAt } : value) } })
|
||||
.where(and(eq(executionWorkspaces.companyId, companyId), eq(executionWorkspaces.id, workspaceId)));
|
||||
});
|
||||
} catch (error) {
|
||||
logger.warn({ err: error, companyId, workspaceId }, "abandoned workspace file expiry retry could not be recorded");
|
||||
}
|
||||
return retryAt;
|
||||
}
|
||||
|
||||
function receiptExpirySql() {
|
||||
return sql`coalesce(case when receipt->>'retainUntil' ~ '^[0-9]{4}-[0-9]{2}-[0-9]{2}T' then (receipt->>'retainUntil')::timestamptz end,
|
||||
case when receipt->>'abandonedAt' ~ '^[0-9]{4}-[0-9]{2}-[0-9]{2}T'
|
||||
then (receipt->>'abandonedAt')::timestamptz + ${`${ABANDONED_WORKSPACE_FILES_RETENTION_DAYS} days`}::interval end,
|
||||
'epoch'::timestamptz)`;
|
||||
}
|
||||
|
||||
function receiptDueSql() {
|
||||
return sql`greatest(${receiptExpirySql()},
|
||||
case when receipt->>'expiryRetryAt' ~ '^[0-9]{4}-[0-9]{2}-[0-9]{2}T' then (receipt->>'expiryRetryAt')::timestamptz
|
||||
else 'epoch'::timestamptz end)`;
|
||||
}
|
||||
|
||||
export async function nextAbandonedWorkspaceFileHoldExpiryAt(db: Pick<Db, "execute">): Promise<number | null> {
|
||||
const [row] = await db.execute<{ next: string | Date | null }>(sql`select min(${receiptDueSql()}) as next
|
||||
from execution_workspaces workspace
|
||||
cross join lateral jsonb_array_elements(case when jsonb_typeof(workspace.metadata->'remoteWorkspaceAbandonedEpochs') = 'array'
|
||||
then workspace.metadata->'remoteWorkspaceAbandonedEpochs' else '[]'::jsonb end) receipt
|
||||
where workspace.metadata ? 'remoteWorkspaceAbandonedEpochs'
|
||||
and receipt->>'schema' = ${REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA}
|
||||
and jsonb_typeof(receipt->'location'->'providerLeaseId') = 'string'
|
||||
and coalesce(jsonb_typeof(receipt->'released'), 'null') = 'null'`);
|
||||
return row?.next ? new Date(row.next).getTime() : null;
|
||||
}
|
||||
|
||||
export async function expireAbandonedWorkspaceFileHoldsWhenDue(db: Db, now = new Date()): Promise<{ expired: number }> {
|
||||
const next = await nextAbandonedWorkspaceFileHoldExpiryAt(db);
|
||||
if (next === null || next > now.getTime()) return { expired: 0 };
|
||||
return await expireAbandonedWorkspaceFileHolds(db, now);
|
||||
}
|
||||
|
||||
export async function recordAbandonedWorkspaceFilesRemoved(db: Db, lease: Pick<typeof environmentLeases.$inferSelect,
|
||||
"id" | "companyId" | "metadata">): Promise<void> {
|
||||
const marker = lease.metadata?.[ABANDONED_WORKSPACE_FILES_RELEASE_KEY] as {
|
||||
issueId?: string | null; workspaceId?: string; abandonedEpoch?: number; reason?: string; keptDays?: number; runId?: string | null;
|
||||
} | undefined;
|
||||
if (!marker || (marker.reason !== "retention_expired" && marker.reason !== "discarded")) return;
|
||||
const issueId = await existingIssueId(db, lease.companyId, marker.issueId ?? null);
|
||||
const runId = await existingRunId(db, lease.companyId, marker.runId);
|
||||
const summary = marker.reason === "retention_expired"
|
||||
? `Unsaved files from an interrupted run were kept for ${days(marker.keptDays ?? ABANDONED_WORKSPACE_FILES_RETENTION_DAYS)} and have now been removed.`
|
||||
: "Discarded unsaved files from an interrupted run have now been removed.";
|
||||
logger.info({ companyId: lease.companyId, leaseId: lease.id, workspaceId: marker.workspaceId, issueId, summary },
|
||||
"abandoned workspace files removed");
|
||||
if (!issueId) return;
|
||||
await logActivity(db, { companyId: lease.companyId, actorType: "system", actorId: "workspace-retention",
|
||||
action: "workspace.abandoned_files_removed", entityType: "issue", entityId: issueId, runId,
|
||||
details: { summary, reason: marker.reason, workspaceId: marker.workspaceId, abandonedEpoch: marker.abandonedEpoch, leaseId: lease.id } });
|
||||
}
|
||||
@@ -5,6 +5,8 @@ import { persistActivity } from "./activity-log.js";
|
||||
|
||||
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
export const REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA = "paperclip.remote-workspace-abandoned-epoch.v1";
|
||||
export const ABANDONED_WORKSPACE_FILES_RETENTION_DAYS = 7;
|
||||
export const ABANDONED_WORKSPACE_FILES_RETENTION_MS = ABANDONED_WORKSPACE_FILES_RETENTION_DAYS * 24 * 60 * 60 * 1000;
|
||||
|
||||
/**
|
||||
* Ends an epoch whose unsaved files cannot be verified, without applying them.
|
||||
@@ -35,9 +37,20 @@ export async function abandonUnverifiedRemoteWorkspaceEpoch(tx: Transaction, inp
|
||||
const abandoned = { schema: REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA, epoch: current.epoch, runId: input.runId,
|
||||
snapshotId: current.snapshotId, location, providerRef: workspace.providerRef,
|
||||
...(workspace.metadata?.nativeWorkspaceMigration !== undefined ? { nativeWorkspaceMigration: workspace.metadata.nativeWorkspaceMigration } : {}),
|
||||
abandonedAt: new Date().toISOString() };
|
||||
abandonedAt: new Date().toISOString(), retainUntil: new Date(Date.now() + ABANDONED_WORKSPACE_FILES_RETENTION_MS).toISOString() };
|
||||
const held = (receipt: unknown) => {
|
||||
const value = receipt as { schema?: unknown; released?: unknown; location?: { providerLeaseId?: unknown } } | null;
|
||||
return value?.schema === REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA && value.released == null
|
||||
&& typeof value.location?.providerLeaseId === "string";
|
||||
};
|
||||
const sameRoot = (receipt: unknown) => {
|
||||
const value = receipt as { location?: { providerLeaseId?: unknown; remoteCwd?: unknown } };
|
||||
return held(receipt) && value.location?.providerLeaseId === location.providerLeaseId && value.location?.remoteCwd === location.remoteCwd;
|
||||
};
|
||||
const kept = [...prior.filter(receipt => !sameRoot(receipt)), abandoned];
|
||||
const releasedHistory = kept.filter(receipt => !held(receipt)).slice(-20);
|
||||
const metadata: Record<string, unknown> = { ...workspace.metadata, remoteWorkspace: next,
|
||||
remoteWorkspaceAbandonedEpochs: [...prior, abandoned].slice(-20) };
|
||||
remoteWorkspaceAbandonedEpochs: kept.filter(receipt => held(receipt) || releasedHistory.includes(receipt)) };
|
||||
// The migration receipt names the old root as the next preparation's root.
|
||||
delete metadata.nativeWorkspaceMigration;
|
||||
const [saved] = await tx.update(executionWorkspaces).set({ providerRef: null, cwd: `/workspaces/${workspace.id}`, metadata,
|
||||
|
||||
@@ -7,6 +7,7 @@ import type { EnvironmentLease } from "@paperclipai/shared";
|
||||
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { verifyRecoverableNativeHarnessBackup } from "./native-runtime/native-harness-backup-storage.js";
|
||||
import { findPluginWorkerUnavailableError } from "./plugin-worker-unavailable-error.js";
|
||||
import { abandonedWorkspaceFilesHeldOnLeaseRowSql, ABANDONED_WORKSPACE_FILES_PARKED_KEY, ABANDONED_WORKSPACE_FILES_PARKED_REASON } from "./remote-workspace-abandoned-files-hold.js";
|
||||
import { sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
|
||||
|
||||
/**
|
||||
@@ -131,13 +132,17 @@ export async function retireReplacedReusableSandboxLease(input: {
|
||||
const { db, lease } = input;
|
||||
if (!lease.providerLeaseId) return false;
|
||||
const now = new Date();
|
||||
const held = abandonedWorkspaceFilesHeldOnLeaseRowSql();
|
||||
const [claimed] = await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({
|
||||
status: "pending_cleanup",
|
||||
status: sql`case when ${held} then 'retained' else 'pending_cleanup' end`,
|
||||
cleanupStatus: "failed",
|
||||
failureReason: REUSABLE_SANDBOX_RESUME_REPLACED_REASON,
|
||||
failureReason: sql`case when ${held} then ${ABANDONED_WORKSPACE_FILES_PARKED_REASON} else ${REUSABLE_SANDBOX_RESUME_REPLACED_REASON} end`,
|
||||
releasedAt: now,
|
||||
lastUsedAt: now,
|
||||
updatedAt: now,
|
||||
metadata: sql`case when ${held} then coalesce(${environmentLeases.metadata}, '{}'::jsonb)
|
||||
|| ${JSON.stringify({ [ABANDONED_WORKSPACE_FILES_PARKED_KEY]: { parkedAt: now.toISOString() } })}::jsonb
|
||||
else ${environmentLeases.metadata} end`,
|
||||
}).where(and(
|
||||
eq(environmentLeases.id, lease.id),
|
||||
eq(environmentLeases.companyId, input.companyId),
|
||||
|
||||
Reference in new issue
Block a user