diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index 56fbb5b762..04aaa912d5 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -1718,3 +1718,15 @@ Agents cannot read or change these preferences. The legacy instance general setting is retained for API compatibility but no longer controls shortcut behavior in the app; users opt in individually after the upgrade. + +### Unsafe native workspace exports + +An unsafe workspace link does not fail an accepted native task result. Retry +export automatically with confined entries only and keep archive confinement in +place. If the export remains unsafe, omit it and finish the saved result under +normal completion rules. Record diagnostics only in run logs; do not add a task +warning or manual repair action. This also applies to historical unsafe failures: +omit the already-rejected export, clear stale repair notices, and finalize the +accepted result without another provider turn, even when its old sandbox is +unavailable. Preserve current ownership and newer-work fences. See +`native-workspace-finalization-recovery.md`. diff --git a/doc/SPEC.md b/doc/SPEC.md index 28ec1662cd..3e5fa410fe 100644 --- a/doc/SPEC.md +++ b/doc/SPEC.md @@ -619,3 +619,15 @@ Agents cannot read or change these preferences. The legacy instance general setting is retained for API compatibility but no longer controls shortcut behavior in the app; users opt in individually after the upgrade. + +### Unsafe native workspace exports + +An unsafe workspace link does not fail an accepted native task result. Retry +export automatically with confined entries only and keep archive confinement in +place. If the export remains unsafe, omit it and finish the saved result under +normal completion rules. Record diagnostics only in run logs; do not add a task +warning or manual repair action. This also applies to historical unsafe failures: +omit the already-rejected export, clear stale repair notices, and finalize the +accepted result without another provider turn, even when its old sandbox is +unavailable. Preserve current ownership and newer-work fences. See +`native-workspace-finalization-recovery.md`. diff --git a/doc/architecture/native-status-arbitration.md b/doc/architecture/native-status-arbitration.md index 43cc102f19..cd8732da70 100644 --- a/doc/architecture/native-status-arbitration.md +++ b/doc/architecture/native-status-arbitration.md @@ -214,7 +214,10 @@ commit, and retryable or terminal failure. Examples: -- a workspace-finalization failure preserves the claim and records a retryable +- an unsafe workspace archive triggers one automatic export of confined entries, + omitting unsafe links; if it remains unsafe, discard the export and settle the saved + result with an informational run-log entry and no task warning or repair action; +- other workspace-finalization failures preserve the claim and record a retryable error rather than falsely completing the issue; - a failed provider run preserves partial evidence and schedules recovery; - a status-version race causes bounded reassessment against current issue diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md new file mode 100644 index 0000000000..7db6b4222f --- /dev/null +++ b/doc/native-workspace-finalization-recovery.md @@ -0,0 +1,203 @@ +# Native workspace finalization ownership and recovery + +Native workspace export and merge acquire a PostgreSQL advisory lock scoped to +company and run before the first physical copyback. The live heartbeat and the +reconciler share that lock. Recovery skips a busy owner without recording another +workspace operation or spending a retry. Recovery also rechecks the coordinator's terminal state and retry time under ownership: an earlier sweep snapshot cannot +start another export after live finalization publishes a permanent repair or delay. +A completed workspace barrier is reread +under ownership before export, and a committed coordinator cannot be overwritten +by a late failure receipt. + +The lock transaction holds no row locks. Ordinary progress and finalization +receipts remain visible through the normal database pool. A dedicated connection outside the application pool is +reserved for the duration of copyback and closed when it settles; even a one-connection application pool remains available for progress writes. Its run profile carries +`nativeWorkspaceFinalizationOwner`, an exact token, host, PID, and process-start +receipt. Losing the lock connection does not prove physical copyback stopped: +a contender still refuses a receipt whose controller is alive. The original +callback joins before its token is released, and publication checks the lock +connection and token. Graceful completion clears the receipt. If that cleanup write fails after the +callback joins, only the same exact controller boot retains positive in-process +join evidence and may resume after reconnecting; an unknown token or a new boot +does not inherit that authority. A controller that +has exited on the same host can be recovered automatically only when a durable +successful workspace barrier proves its copyback finished. A dead parent can +leave tar/Git children alive, so incomplete copyback requires operator stop +verification even on the same host. PID reuse is checked against its recorded +start time rather than trusted by PID alone. + +## Unverified copyback after controller replacement + +The controller cannot verify a process on a foreign or unknown host, or orphaned +copyback children after an abrupt parent death before the success barrier. It surfaces +`native_workspace_finalization_owner_unverified` as board-owned recovery, with no +automatic provider wake. This is an intentional limit: elapsed time or a missing +database connection never proves the old copyback process stopped. + +An instance operator must first verify through the deployment platform that the +exact prior controller **and its copyback subprocesses** have stopped. Retain the +sandbox, accepted native result, and workspace descriptor. Do not run a new +provider turn, delete workspace contents, or relax archive confinement. + +After that platform verification, use a database maintenance transaction to +release only the exact receipt shown by the recovery action. Replace the four +placeholders with the action's company, run, token, and source issue. The advisory +lock prevents concurrent acquisition during this change; the token comparison +prevents clearing a newer owner. A zero-row update means ownership changed and +requires fresh inspection. This maintenance operation is for a full-control +instance operator, not an agent tool. + +```sql +BEGIN; +SELECT pg_advisory_xact_lock(hashtextextended( + 'native-workspace-finalization::', 0)); +UPDATE heartbeat_runs +SET runner_profile_json = runner_profile_json - 'nativeWorkspaceFinalizationOwner' +WHERE company_id = ''::uuid + AND id = ''::uuid + AND native_issue_id = ''::uuid + AND runtime_mode = 'native' + AND runner_profile_json->'nativeWorkspaceFinalizationOwner'->>'token' = '' +RETURNING id; +COMMIT; +``` + +Resolve the existing board recovery action with a note containing the platform +stop evidence. The ordinary reconciliation sweep then resumes workspace +finalization from the accepted result. Confirm the run's native phase and +`resultJson.finalizationPhase` are `committed`, there is no `nextAttemptAt`, and +no workspace operation is still running. The accepted provider result is reused. + +## Automatic unsafe archive recovery + +An unsafe link in a native workspace must not fail a completed task or require +an operator to repair a sandbox. The accepted agent result remains authoritative. + +Daytona validates every exported archive before extraction. If validation rejects +an archive, export once more using files, directories, and relative symlinks +whose resolved targets stay inside the workspace. The fallback does not follow +or delete symlinks. It omits unsafe or unresolvable links, stores hard-linked +files as ordinary bytes, and preserves directory exclusions and empty directories. The second archive passes the same confinement checks. +A fixed informational message records the fallback in the provider log. + +If native copyback still rejects the archive or its source confinement check, +Paperclip discards that export, records `workspace_export_omitted` at info level +in the local run log, and completes finalization using the original accepted +result. It does not create a task warning, recovery card, repair request, or new +provider turn. The normal completion policy still enforces ownership and explicit +workflow constraints. Lost remote files are an accepted tradeoff. + +The live path and restart finalizer use the same policy under workspace +finalization ownership. Ownership loss, transport failures, missing sandboxes, +and other unrelated errors retain their existing handling. No unsafe archive is +extracted. No host path or link target is copied into the informational run event. + + +## Repairing a failed workspace export without rerunning the agent + +Transient export failures retry three times, then produce +`native_workspace_sync_out_retry_exhausted`. The accepted result stays saved, +and the exact sandbox is stopped and retained for export-only recovery. The +reason identifies a transport/copyback failure, not an inferred disk-space cause. +New unsafe archives use the automatic policy above and do not create this hold. +Historical unsafe-archive holds recover automatically as described below; they do not use this operator flow. + +A board member with runtime management access can complete the saved result: + +1. Read the recovery action, run, and environment lease. Verify the company, run, + provider sandbox ID, and stopped-provider receipt. Preserve that exact sandbox; + do not acquire a replacement or seed the host workspace over it. +2. Through the provider console or official SDK, resume that sandbox and inspect + the export failure while preserving all user files. Restore transport access + or other provider prerequisites before retrying. Extending the sandbox + retention window can be necessary during this repair. +3. In the task's **Workspace export needs repair** notice, describe the repair and + choose **Retry workspace export**. The equivalent board API is + `POST /api/issues/:id/recovery-actions/retry-workspace-export` with + `{ "actionId": "", "runId": "", "repairNote": "" }`. +4. Confirm the same run commits, the saved result determines the task outcome, + and the recovery action resolves. No new provider turn or wake is created. + A reusable repair sandbox is stopped and retained again. An ephemeral sandbox follows its original destroy-after-turn policy only after the exact result commits and the workspace reference contains a finalized host-copy receipt. + +Admission requires the same accepted result, task owner, descriptor, and lease; +confirmed prior provider stop; an available repaired sandbox; and no newer task +execution or competing current lease. It shares finalization ownership and +resumes only the recorded provider lease through its verified lifecycle method. +That method drains old activity and verifies the saved workspace identity before +reopening the provider's controller admission gate; an external console restart +alone does not reopen that gate. Admission rejects a missing or replacement +sandbox and revalidates after probing the exact workspace. A changed binding or unavailable sandbox +returns `409` without reopening work. A duplicate queued request is idempotent. +Unsafe archives use automatic salvage or omission during retry. Generic +ordinary recovery resolution cannot retry, mark done, or send the task for review +in place of export-only retry. Explicit board false-positive/cancellation +dispositions remain deliberate overrides; they do not claim successful copyback. + +Before publishing terminal export failure, the finalizer atomically records a stop-only intent with the failed run. This includes ephemeral allocations: ordinary release policy cannot delete their unexported files. A crash before cleanup leaves enough exact lease, result, and plugin authority for restart recovery. The same intent remains while export-only retry is active. + +Before resuming the retained sandbox, the controller durably records a stop-only +cleanup intent on that exact lease and removes the old stopped receipt. If the +resume reply is lost, probing fails, or admission changes, it stops and retains +the sandbox. A failed stop remains `pending_cleanup`; a restarted controller's +bounded cleanup sweep retries the verified stop without destroying saved files. +An in-flight request holds a 15-minute cleanup claim, so a controller crash may +delay that sweep until the claim expires. Neither an unconfirmed stop nor a +stale receipt grants admission, and a changed lease or competing sandbox owner +prevents cleanup from taking ownership. Retry export after the lease has a new +confirmed stopped receipt. No provider turn is created by this recovery. +Cleanup and its readiness probe use the plugin ID recorded on that lease. Another +plugin with the same provider name cannot take over; an unavailable original +plugin defers cleanup without consuming an attempt. Stop-only cleanup requires the exact plugin worker to advertise `environmentStopLease`. Older plugins without that hook receive neither release nor destroy; resume admission also defers until safe compensation is available. +New intents use schema v2 with an explicit plugin pin. A v1 intent created before +that field existed remains recoverable using only the plugin ID already recorded +on its exact lease; an explicit mismatched pin is still rejected. + +The opt-in `native-workspace-export-resume.live.test.ts` is a provider-boundary +fault integration, separate from the browser Product E2E. After building the +Daytona plugin, run that exact Vitest file with `PAPERCLIP_LIVE_EXPORT_RESUME=1`, +`DAYTONA_API_KEY`, and `PAPERCLIP_LIVE_EXPORT_RESUME_IMAGE` set to an immutable +image digest. It creates one disposable ephemeral sandbox and database, injects probe and stop transport failures, and verifies a fresh runtime can stop the sandbox while preserving exact nonce bytes. It also injects three transient failures at the production finalizer boundary and verifies retained work plus export-only retry admission. This boundary test does not claim physical copyback or result commitment; historical browser Product E2E supplied that proof before automatic unsafe-export recovery. It deletes only that owned fixture after proof. + +### Historical unsafe exports + +The reconciliation sweep automatically recovers accepted results that an older +controller terminalized with `native_workspace_sync_out_unsafe_archive`. It +omits the already-rejected export without accessing the old provider, so a +stopped, unavailable, or deleted sandbox cannot require user repair. The +omission barrier and result re-admission are atomic under finalization ownership. +The normal status arbiter still enforces current ownership, contracts, and gates. + +This works with active, incorrectly resolved, or missing old repair actions. It +clears the matching stale unsafe notices instead of restoring them. Newer runs, +changed task ownership or contracts, another active recovery action, and explicit +operator dispositions prevent replay of old results. No provider turn is created. +Diagnostics appear only in the local run log; no unsafe-export activity notice, +repair card, or recovery chip is created. Existing stop-only allocation intents +retain their exact ownership and cleanup protections. + +Daytona stop-only preservation disables provider auto-delete and refreshes the provider record to confirm the disabled policy before stopping. An unavailable or unconfirmed policy leaves cleanup pending; it never falls back to stop or delete. The original ephemeral destroy policy applies only after exact accepted-result copyback and commitment. + +New ephemeral native allocations receive an acquisition-time workspace sentinel bound to their run and provider allocation. Export-only resume requires that proof and the unchanged durable intent. A legacy ephemeral allocation without this proof remains blocked; recovery never creates a sentinel or accepts a replacement during resume. The reusable workspace identity hash is unchanged. + +Disabling auto-delete retains provider storage while repair waits. No automatic lease-age sweep deletes these released, confirmed-stopped allocations. Storage costs can continue until committed copyback applies the original cleanup policy or an operator explicitly deletes the allocation. + +### Generic stop-only cleanup + +An explicit sandbox `stop_and_retain` also requires the dedicated provider hook +when no export repair is involved, including successful per-turn reusable runs +and startup cancellation. Before dispatch, the controller stores a separate +`sandboxStopAndRetain` intent bound to the company, run, lease, provider allocation, +and original plugin. The initial cleanup and restart sweep both require that +plugin to advertise `environmentStopLease`. Missing capability or failed stop +leaves pending cleanup; ordinary release and destroy are never fallback methods. + +A matching stopped receipt clears the pending intent, leaves the lease released +and resumable, and saves `sandboxStopAndRetainReceipt`. Its request identity, +original plugin and `method: "environmentStopLease"` attest this dispatch route. +The accepted-result export intent and its commitment requirements are unchanged. + +The built-in fake provider uses its explicit `stopLease` operation rather than a +plugin RPC. Its intent pins the registered built-in provider and its receipt says +`builtin.stopLease`. This provider owns no real process or filesystem; the receipt +models its lifecycle. Missing built-in stop support or an unconfirmed receipt +also remains pending through restart, without calling release or destroy. diff --git a/doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md b/doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md index 8cc2ea4100..c3988f98f1 100644 --- a/doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md +++ b/doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md @@ -201,3 +201,9 @@ Earlier drafts listed two concurrency keys, `concurrentSyncAndExec` and read either key, so a declaration had no effect. The host removed both keys. The strict capability validator now rejects them as unknown keys. The host can reintroduce a concurrency capability when a runtime path enforces it. + +### Preserving an unexported workspace + +Native export recovery requires the separately discovered `environmentStopLease` RPC (`onEnvironmentStopLease`). It stops the exact allocation and preserves files regardless of its normal release policy. A failed or unconfirmed stop must throw and must never fall back to deletion. The host uses the recorded plugin ID and defers when that worker does not advertise the hook; a generic release method is insufficient evidence of stop-only support. Normal successful ephemeral release remains destructive after verified copyback. + +Daytona stop-only preservation disables provider auto-delete and refreshes the provider record to confirm the disabled policy before stopping. An unavailable or unconfirmed policy leaves cleanup pending; it never falls back to stop or delete. The original ephemeral destroy policy applies only after exact accepted-result copyback and commitment. diff --git a/doc/run-log-events.md b/doc/run-log-events.md index e468d7ff86..d9cf3faf66 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -34,6 +34,15 @@ credential material are never written to the run log. These records remain run-log events. They do not create an OpenTelemetry or Paperclip Telemetry export, and legacy adapters do not use this writer. +## Omitted Unsafe Workspace Export + +`workspace_export_omitted` is an informational system event in the local run log. +Its payload is `{ "reason": "restore_unsafe_archive" }`, with `"legacy": true` +when recovering an unsafe failure from an older controller. It records that native +finalization discarded an unsafe export and continued with the accepted result. +It contains no archive names, link targets, or raw error details. It does not +create a task warning, recovery action, Telemetry event, or OpenTelemetry export. + ## Native Restart Recovery Run-Log Event Paperclip writes a `native.recovery.transition` event for every native restart diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 70030b051d..ecfd795442 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -255,6 +255,19 @@ export function postgresJsOptions(options: DatabaseClientOptions): Record Db>(); + +export async function withDedicatedDbConnection(db: Db, action: (dedicated: Db) => Promise): Promise { + const factory = dedicatedDbFactories.get(db); + if (!factory) throw new Error("dedicated_connection_requires_create_db"); + const dedicated = factory(); + try { return await action(dedicated); } + finally { await dedicated.$client.end({ timeout: 1 }); } +} + export function createDb(url: string, options?: DatabaseClientOptions) { const resolved = resolveDatabaseClientOptions(options ?? databaseClientOptionsFromEnv()); const sql = postgres(url, postgresJsOptions(resolved)); @@ -263,7 +276,11 @@ export function createDb(url: string, options?: DatabaseClientOptions) { // The registry keeps the real client (teardown must end the actual pool); // drizzle gets the retrying face so a pooler-recycled socket replays the // query instead of failing the request that happened to draw it. - return drizzlePg(withTransientWriteRetry(sql), { schema }); + const db = drizzlePg(withTransientWriteRetry(sql), { schema }); + dedicatedDbFactories.set(db, () => createDb(url, { + ...resolved, maxConnections: 1, applicationName: "paperclip-workspace-finalization-lock", + })); + return db; } export async function getPostgresDataDirectory(url: string): Promise { diff --git a/packages/db/src/index.ts b/packages/db/src/index.ts index fb62bcf9cc..378c9bd37b 100644 --- a/packages/db/src/index.ts +++ b/packages/db/src/index.ts @@ -1,5 +1,6 @@ export { createDb, + withDedicatedDbConnection, closeRegisteredClients, getPostgresDataDirectory, ensurePostgresDatabase, diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts index dd4e8f4efc..ff1c419f55 100644 --- a/packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync.test.ts @@ -17,7 +17,7 @@ vi.mock("@daytonaio/sdk", () => ({ DaytonaTimeoutError: class MockDaytonaTimeoutError extends Error {}, })); -import { performSyncIn } from "./file-sync.js"; +import { performSyncIn, performSyncOut } from "./file-sync.js"; import { __setDaytonaPluginContextForTest } from "./plugin.js"; import type { PluginContext, PluginSyncOperation } from "@paperclipai/plugin-sdk"; @@ -1082,3 +1082,78 @@ describe("daytona file-sync inbound zstd transport compression", () => { }); }); }); + + +describe.skipIf(!gnuTar)("automatic unsafe workspace export recovery", () => { + const cleanupDirs: string[] = []; + afterEach(async () => { + await Promise.all(cleanupDirs.splice(0).map((root) => fs.rm(root, { recursive: true, force: true }))); + }); + it.each(["/usr/bin/pnpm", "../../outside.txt", "ambiguous", "ambiguous-target"])("preserves files and safe links when a link is unsafe: %s", async (target) => { + const root = await fs.mkdtemp("/tmp/paperclip-export-recovery-"); + cleanupDirs.push(root); + const remoteDir = path.join(root, "remote"); + const output = path.join(root, "output"); + const bin = path.join(root, "bin"); + await fs.mkdir(path.join(remoteDir, "nested"), { recursive: true }); + await fs.mkdir(path.join(remoteDir, "cache")); + await fs.mkdir(path.join(remoteDir, "empty")); + await fs.mkdir(bin); + await fs.symlink(gnuTar!, path.join(bin, "tar")); + await fs.writeFile(path.join(root, "outside.txt"), "private bytes"); + await fs.writeFile(path.join(remoteDir, "nested", "--safe [file].txt"), "saved work", { mode: 0o600 }); + await fs.writeFile(path.join(remoteDir, "cache", "ignored.txt"), "excluded"); + const linkName = target === "ambiguous" ? "tool -> decoy" : "tool"; + const linkTarget = target === "ambiguous" ? "--safe [file].txt" : target === "ambiguous-target" ? "saved -> work.txt" : target; + if (target === "ambiguous-target") await fs.writeFile(path.join(remoteDir, "nested", linkTarget), "more saved work"); + await fs.symlink(linkTarget, path.join(remoteDir, "nested", linkName)); + await fs.symlink("nested/--safe [file].txt", path.join(remoteDir, "safe-link")); + await fs.link(path.join(remoteDir, "nested", "--safe [file].txt"), path.join(remoteDir, "hard-link")); + const downloads = vi.fn(async (requests: Array<{ source: string; destination: string }>) => { + for (const request of requests) await fs.copyFile(request.source, request.destination); + return requests.map(({ source }) => ({ source })); + }); + const recovery = vi.fn(); + const { sandbox } = createRealExecSandbox({ commandEnv: { ...process.env, PATH: `${bin}:${process.env.PATH}` } }); + Object.assign(sandbox.fs, { downloadFiles: downloads, deleteFile: (file: string) => fs.rm(file, { force: true }) }); + await expect(performSyncOut({ sandbox: sandbox as never, remoteDir, timeoutSeconds: 30, onArchiveRecovery: recovery, + operations: [{ operationId: "export", files: [{ sourcePath: remoteDir, targetPath: output, kind: "directory", exclude: ["cache"] }] }], + })).resolves.toMatchObject({ operations: [{ operationId: "export", filesTransferred: target === "ambiguous-target" ? 4 : 3 }] }); + expect(downloads).toHaveBeenCalledTimes(2); + expect(recovery).toHaveBeenCalledOnce(); + expect(await fs.readFile(path.join(output, "nested", "--safe [file].txt"), "utf8")).toBe("saved work"); + expect((await fs.stat(path.join(output, "nested", "--safe [file].txt"))).mode & 0o777).toBe(0o600); + expect(await fs.readFile(path.join(output, "hard-link"), "utf8")).toBe("saved work"); + await expect(fs.lstat(path.join(output, "nested", linkName))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await fs.readlink(path.join(output, "safe-link"))).toBe("nested/--safe [file].txt"); + expect(await fs.readdir(path.join(output, "empty"))).toEqual([]); + await expect(fs.stat(path.join(output, "cache", "ignored.txt"))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await fs.readFile(path.join(root, "outside.txt"), "utf8")).toBe("private bytes"); + expect((await fs.lstat(path.join(remoteDir, "nested", linkName))).isSymbolicLink()).toBe(true); + expect((await fs.readdir(remoteDir)).filter((name) => name.startsWith(".paperclip-upload"))).toEqual([]); + }); + + it("finishes an export containing only unsafe links with an empty directory", async () => { + const root = await fs.mkdtemp("/tmp/paperclip-export-empty-"); + cleanupDirs.push(root); + const remoteDir = path.join(root, "remote"); + const output = path.join(root, "output"); + const bin = path.join(root, "bin"); + await fs.mkdir(remoteDir); + await fs.mkdir(bin); + await fs.symlink(gnuTar!, path.join(bin, "tar")); + await fs.symlink("/usr/bin/pnpm", path.join(remoteDir, "pnpm")); + const { sandbox } = createRealExecSandbox({ commandEnv: { ...process.env, PATH: `${bin}:${process.env.PATH}` } }); + Object.assign(sandbox.fs, { + downloadFiles: async (requests: Array<{ source: string; destination: string }>) => { + for (const request of requests) await fs.copyFile(request.source, request.destination); + return requests.map(({ source }) => ({ source })); + }, + deleteFile: (file: string) => fs.rm(file, { force: true }), + }); + await performSyncOut({ sandbox: sandbox as never, remoteDir, timeoutSeconds: 30, + operations: [{ operationId: "empty", files: [{ sourcePath: remoteDir, targetPath: output, kind: "directory" }] }], + }); + expect(await fs.readdir(output)).toEqual([]); + }); +}); diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts index f235076753..ad84d90d39 100644 --- a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts @@ -2,7 +2,7 @@ import path from "node:path"; import os from "node:os"; import { promises as fs, createReadStream, createWriteStream } from "node:fs"; import { randomUUID } from "node:crypto"; -import { execFile } from "node:child_process"; +import { execFile, spawn } from "node:child_process"; import { promisify } from "node:util"; import zlib from "node:zlib"; import { pipeline } from "node:stream/promises"; @@ -250,43 +250,131 @@ export function splitLinkEntryOnce(field: string, delimiter: string): { name: st * preserved. Parses the `-tvf` verbose listing so both member names and link * targets are inspected; any unparseable line fails closed. */ -async function assertTarballEntriesConfined(archivePath: string): Promise { - const { stdout } = await execFileAsync("tar", ["-tvf", archivePath], { +class UnsafeOutboundArchiveError extends Error {} + +function assertTarListingLineConfined(line: string): void { + if (line.trim().length === 0) return; + const parsed = parseTarVerboseListingLine(line); + if (!parsed) { + throw new UnsafeOutboundArchiveError(`Daytona syncOut refusing tarball with an unparseable entry listing: ${line}`); + } + const typeFlag = parsed.typeFlag; + let name = parsed.rest; + let linkTarget: string | null = null; + if (typeFlag === "l") { + const split = splitLinkEntryOnce(name, " -> "); + if (!split) throw new UnsafeOutboundArchiveError(`Daytona syncOut refusing unparseable or ambiguous symlink entry: ${line}`); + name = split.name; + linkTarget = split.target; + } else if (typeFlag === "h") { + const split = splitLinkEntryOnce(name, " link to "); + if (!split) throw new UnsafeOutboundArchiveError(`Daytona syncOut refusing unparseable or ambiguous hardlink entry: ${line}`); + name = split.name; + linkTarget = split.target; + } + const cleanName = name.replace(/\/+$/, ""); + if (cleanName.length > 0 && posixPathEscapes(cleanName)) { + throw new UnsafeOutboundArchiveError(`Daytona syncOut refusing tarball member that escapes the extraction dir: ${name}`); + } + if (linkTarget !== null) { + const resolved = path.posix.join(path.posix.dirname(cleanName), linkTarget); + if (path.posix.isAbsolute(linkTarget) || posixPathEscapes(resolved)) { + throw new UnsafeOutboundArchiveError( + `Daytona syncOut refusing tarball link whose target escapes the extraction dir: ${name} -> ${linkTarget}`, + ); + } + } +} + +const TAR_LISTING_MAX_LINE_BYTES = 64 * 1024; +const TAR_LISTING_MAX_STDERR_BYTES = 64 * 1024; +// Full workspace exports are larger than provider checkpoints. These quotas +// admit the supported 60k-file / 39.8 MB-name export and 145k-entry regression, +// while bounding work on untrusted metadata independently of the wall deadline. +// The byte quota matches the native workspace descriptor's 64 MiB ceiling; +// it is an admission counter, never a buffer allocation. +const TAR_LISTING_MAX_TOTAL_BYTES = 64 * 1024 * 1024; +const TAR_LISTING_MAX_ENTRIES = 250_000; +const TAR_LISTING_TIMEOUT_MS = 120_000; + +export async function assertTarballEntriesConfined( + archivePath: string, + timeoutMs = TAR_LISTING_TIMEOUT_MS, +): Promise { + // A valid large workspace can exceed execFile's buffer. Stream within both + // aggregate admission quotas and per-entry/diagnostic memory bounds, checking + // every entry before extraction. Keep bytes until a full line to preserve + // UTF-8 characters split across pipe chunks. + const child = spawn("tar", ["-tvf", archivePath], { env: { ...process.env, COPYFILE_DISABLE: "1" }, - maxBuffer: 32 * 1024 * 1024, + stdio: ["ignore", "pipe", "pipe"], }); - const lines = stdout.split("\n").filter((line) => line.trim().length > 0); - for (const line of lines) { - const parsed = parseTarVerboseListingLine(line); - if (!parsed) { - throw new Error(`Daytona syncOut refusing tarball with an unparseable entry listing: ${line}`); + let spawnError: Error | undefined; + let failure: Error | undefined; + let stderr = Buffer.alloc(0); + let pending: Buffer = Buffer.alloc(0); + let totalBytes = 0; + let entries = 0; + const validateLine = (line: Buffer) => { + // Count empty lines too, so whitespace cannot evade the parsing-work quota. + if (++entries > TAR_LISTING_MAX_ENTRIES) { + throw new Error("Daytona syncOut tar listing entry limit exceeded (250000)"); } - const typeFlag = parsed.typeFlag; - let name = parsed.rest; - let linkTarget: string | null = null; - if (typeFlag === "l") { - const split = splitLinkEntryOnce(name, " -> "); - if (!split) throw new Error(`Daytona syncOut refusing unparseable or ambiguous symlink entry: ${line}`); - name = split.name; - linkTarget = split.target; - } else if (typeFlag === "h") { - const split = splitLinkEntryOnce(name, " link to "); - if (!split) throw new Error(`Daytona syncOut refusing unparseable or ambiguous hardlink entry: ${line}`); - name = split.name; - linkTarget = split.target; + assertTarListingLineConfined(line.toString("utf8")); + }; + const closed = new Promise<{ code: number | null; signal: NodeJS.Signals | null }>((resolve) => { + child.once("error", (error) => { spawnError = error; }); + child.once("close", (code, signal) => resolve({ code, signal })); + }); + const stop = (error: Error) => { + failure ??= error; + child.kill("SIGKILL"); + }; + const timer = setTimeout(() => { + stop(new Error("Daytona syncOut tar listing validation timed out")); + }, Math.max(1, Math.min(timeoutMs, TAR_LISTING_TIMEOUT_MS))); + child.stderr.on("data", (chunk: Buffer) => { + if (stderr.length + chunk.length > TAR_LISTING_MAX_STDERR_BYTES) { + stop(new Error("Daytona syncOut tar listing diagnostics exceed the byte limit")); + return; } - const cleanName = name.replace(/\/+$/, ""); - if (cleanName.length > 0 && posixPathEscapes(cleanName)) { - throw new Error(`Daytona syncOut refusing tarball member that escapes the extraction dir: ${name}`); - } - if (linkTarget !== null) { - const resolved = path.posix.join(path.posix.dirname(cleanName), linkTarget); - if (path.posix.isAbsolute(linkTarget) || posixPathEscapes(resolved)) { - throw new Error( - `Daytona syncOut refusing tarball link whose target escapes the extraction dir: ${name} -> ${linkTarget}`, - ); + stderr = Buffer.concat([stderr, chunk]); + }); + try { + for await (const chunk of child.stdout) { + if (failure) break; + const bytes = chunk as Buffer; + totalBytes += bytes.length; + if (totalBytes > TAR_LISTING_MAX_TOTAL_BYTES) { + throw new Error("Daytona syncOut tar total listing byte limit exceeded (64 MiB)"); + } + let start = 0; + while (start < bytes.length) { + const newline = bytes.indexOf(10, start); + const end = newline < 0 ? bytes.length : newline; + if (pending.length + end - start > TAR_LISTING_MAX_LINE_BYTES) { + throw new Error("Daytona syncOut refusing tarball with an entry listing exceeding the byte limit"); + } + pending = Buffer.concat([pending, bytes.subarray(start, end)]); + if (newline < 0) break; + validateLine(pending); + pending = Buffer.alloc(0); + start = newline + 1; } } + if (!failure && pending.length > 0) validateLine(pending); + const result = await closed; + if (failure) throw failure; + if (spawnError) throw spawnError; + if (result.code !== 0) { + throw new Error(`Daytona syncOut tar listing failed (${result.signal ?? result.code}): ${stderr.toString("utf8").trim()}`); + } + } catch (error) { + stop(error instanceof Error ? error : new Error(String(error))); + await closed; + throw failure; + } finally { + clearTimeout(timer); } } @@ -1160,6 +1248,7 @@ async function syncOutDirectoryMapping(input: { mapping: PluginSyncFileMapping; remoteDir: string; timeoutSeconds: number; + onArchiveRecovery?: () => void; }): Promise<{ filesTransferred: number; bytesTransferred: number }> { const { sandbox, mapping, remoteDir, timeoutSeconds } = input; assertConfinedSandboxPath(remoteDir, mapping.sourcePath, "source"); @@ -1177,7 +1266,10 @@ async function syncOutDirectoryMapping(input: { return withHostTempDir(async (tmp) => { const remoteTar = path.posix.join(remoteDir, scratchName(".tar")); - const excludeFlags = ["._*", ...(mapping.exclude ?? [])] + const remoteList = path.posix.join(remoteDir, scratchName(".list")); + const excludes = ["._*", `${SCRATCH_PREFIX}*`, ...(mapping.exclude ?? [])]; + const excludeFlags = excludes + .flatMap((entry) => [entry, `${entry.replace(/\/$/, "")}/*`]) .map((entry) => `--exclude ${shellQuote(entry)}`) .join(" "); // Tar the source in-sandbox (naming top-level entries so no "." self-entry is @@ -1191,37 +1283,66 @@ async function syncOutDirectoryMapping(input: { `if [ "$#" -eq 0 ]; then dd if=/dev/zero of=${shellQuote(remoteTar)} bs=1024 count=1; ` + `else tar -c --no-xattrs ${mapping.followSymlinks ? "-h " : ""}${excludeFlags} -f ${shellQuote(remoteTar)} -- "$@"; fi`, ].join(" && "); - await assertSandboxCommandOk(sandbox, `sh -c ${shellQuote(tarScript)}`, timeoutSeconds, "syncOut tar"); - guardRoundTrips += 1; + // Rebuild the archive with files, directories, and relative links whose + // resolved targets remain inside this mapping. Nothing is deleted or + // dereferenced. Host validation still checks the rebuilt archive, including + // links changed by the sandbox between enumeration and tar creation. + const prunePaths = excludes.flatMap((entry) => [ + `-path ${shellQuote(`./${entry}`)}`, `-path ${shellQuote(`*/${entry}`)}`, + ]).join(" -o "); + const filterLinks = [ + ...canonicalizerPreamble(shellQuote(mapping.sourcePath)), + 'for _pc_link do', + ' _pc_target=$(readlink -- "$_pc_link") || continue;', + ' case "$_pc_link" in *" -> "*) continue ;; esac;', + ' case "$_pc_target" in /*|*" -> "*) continue ;; esac;', + ' _pc_real=$(_pc_resolve "$_pc_link" 2>/dev/null) || continue;', + ` case "$_pc_real/" in "$_pc_root"/*) printf '%s\\0' "$_pc_link" ;; esac;`, + 'done', + ].join("\n"); + const confinedEntriesScript = [ + `cd ${shellQuote(mapping.sourcePath)}`, + `find . -mindepth 1 \\( ${prunePaths} \\) -prune -o -type l -exec sh -c ${shellQuote(filterLinks)} sh {} + -o \\( -type f -o -type d \\) -print0 > ${shellQuote(remoteList)}`, + `tar -c --no-xattrs --hard-dereference --no-recursion --null ${excludeFlags} -f ${shellQuote(remoteTar)} -T ${shellQuote(remoteList)}`, + ].join(" && "); const localTar = path.join(tmp, "sync-out.tar"); let bytesTransferred = 0; try { - // `transfer` span: the real byte download — `sandbox.fs.downloadFiles`. - const responses = await withProviderSpan({ - name: "transfer", - wallMsAttr: SPAN_ATTR.transferWallMs, - attributes: { - [SPAN_ATTR.transferGuardCount]: guardRoundTrips, - [SPAN_ATTR.transferDirection]: "outbound", - }, - run: () => - sandbox.fs.downloadFiles([{ source: remoteTar, destination: localTar }], timeoutSeconds), - }); - const response = responses.find((entry) => entry.source === remoteTar) ?? responses[0]; - if (!response || response.error) { - throw new Error( - `Daytona syncOut directory download failed for ${mapping.sourcePath}: ${response?.error ?? "no response returned"}`, - ); + for (let attempt = 0; attempt < 2; attempt += 1) { + await assertSandboxCommandOk(sandbox, `sh -c ${shellQuote(attempt === 0 ? tarScript : confinedEntriesScript)}`, timeoutSeconds, "syncOut tar"); + guardRoundTrips += 1; + // `transfer` span: the real byte download — `sandbox.fs.downloadFiles`. + const responses = await withProviderSpan({ + name: "transfer", + wallMsAttr: SPAN_ATTR.transferWallMs, + attributes: { + [SPAN_ATTR.transferGuardCount]: guardRoundTrips, + [SPAN_ATTR.transferDirection]: "outbound", + }, + run: () => + sandbox.fs.downloadFiles([{ source: remoteTar, destination: localTar }], timeoutSeconds), + }); + const response = responses.find((entry) => entry.source === remoteTar) ?? responses[0]; + if (!response || response.error) { + throw new Error( + `Daytona syncOut directory download failed for ${mapping.sourcePath}: ${response?.error ?? "no response returned"}`, + ); + } + bytesTransferred += (await fs.stat(localTar)).size; + try { + await extractHostTarball({ archivePath: localTar, localDir: mapping.targetPath }); + break; + } catch (error) { + if (!(error instanceof UnsafeOutboundArchiveError) || attempt > 0) throw error; + // Diagnostic only: no warning, task action, or additional agent turn. + try { input.onArchiveRecovery?.(); } catch { /* logging is best effort */ } + } } - bytesTransferred = (await fs.stat(localTar)).size; - await extractHostTarball({ archivePath: localTar, localDir: mapping.targetPath }); } finally { // Best-effort remove the sandbox-side scratch tar; the host temp dir is // cleaned by withHostTempDir. - await sandbox.fs - .deleteFile(remoteTar) - .catch(() => undefined); + await Promise.all([remoteTar, remoteList].map((file) => sandbox.fs.deleteFile(file).catch(() => undefined))); } const filesTransferred = await countHostFiles(mapping.targetPath, mapping.exclude); return { filesTransferred, bytesTransferred }; @@ -1233,6 +1354,7 @@ export async function performSyncOut(input: { operations: PluginSyncOperation[]; remoteDir: string; timeoutSeconds: number; + onArchiveRecovery?: () => void; }): Promise { const operations: PluginEnvironmentSyncResult["operations"] = []; for (const operation of input.operations) { @@ -1255,6 +1377,7 @@ export async function performSyncOut(input: { const dirResult = await syncOutDirectoryMapping({ sandbox: input.sandbox, mapping, + onArchiveRecovery: input.onArchiveRecovery, remoteDir: input.remoteDir, timeoutSeconds: input.timeoutSeconds, }); diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts index 358f4f5974..7d4c5aaaff 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts @@ -60,6 +60,7 @@ function createMockSandbox(overrides: { // `refreshData`. The default mock leaves it unset (no TTL configured). autoDestroyAt: overrides.autoDestroyAt ?? undefined, updatedAt: overrides.updatedAt, + autoDeleteInterval: -1, getWorkDir: vi.fn().mockResolvedValue(overrides.workDir ?? "/home/daytona"), getUserHomeDir: vi.fn().mockResolvedValue("/home/daytona"), start: vi.fn().mockResolvedValue(undefined), @@ -1366,6 +1367,32 @@ describe("Daytona sandbox provider plugin", () => { expect(lease?.metadata).not.toHaveProperty("gpu"); }); + it.each(["matched", "mismatch", "missing", "foreign_run", "foreign_allocation", "legacy"])("verifies an ephemeral run-bound workspace before export recovery: %s", async outcome => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox({ id: "ephemeral-owned" }); + mockCreate.mockResolvedValue(sandbox); + const base = { driverKey: "daytona", companyId: "company-1", environmentId: "env-1", agentId: "agent-1", + issueId: "issue-1", adapterType: "paperclip_runner", config: { reuseLease: false } }; + const lease = await plugin.definition.onEnvironmentAcquireLease!({ ...base, runId: "run-1" }); + const sentinel = lease.metadata!.workspaceSentinel as { token: string; result: string }; + expect(sentinel).toMatchObject({ token: expect.stringMatching(/^[a-f0-9]{64}$/), result: "written" }); + const second = await plugin.definition.onEnvironmentAcquireLease!({ ...base, runId: "run-2" }); + expect((second.metadata!.workspaceSentinel as { token: string }).token).not.toBe(sentinel.token); + sandbox.process.executeCommand.mockResolvedValueOnce({ exitCode: outcome === "missing" ? 1 : 0, + result: JSON.stringify({ token: outcome === "matched" ? sentinel.token : "foreign" }), artifacts: { stdout: "" } }); + const metadata = { ...lease.metadata, + ...(outcome === "foreign_run" ? { nativeWorkspaceExportResume: { runId: "foreign-run" } } : {}), + ...(outcome === "legacy" ? { workspaceSentinel: { result: "skipped", token: null } } : {}) }; + sandbox.fs.createFolder.mockClear(); sandbox.fs.uploadFile.mockClear(); sandbox.stop.mockClear(); + const resumed = await plugin.definition.onEnvironmentResumeLease!({ ...base, providerLeaseId: outcome === "foreign_allocation" ? "replacement" : lease.providerLeaseId!, leaseMetadata: metadata }); + expect(resumed).toMatchObject(outcome === "matched" ? { providerLeaseId: sandbox.id, metadata: { workspaceSentinel: { result: "matched" } } } + : { providerLeaseId: null, metadata: { expired: true, workspaceSentinel: { result: ["foreign_run", "foreign_allocation", "legacy"].includes(outcome) ? "mismatch" : outcome } } }); + expect(sandbox.fs.createFolder).not.toHaveBeenCalled(); + expect(sandbox.fs.uploadFile).not.toHaveBeenCalled(); + expect(sandbox.stop).not.toHaveBeenCalled(); + expect(sandbox.delete).not.toHaveBeenCalled(); + }); + it("changes reusable-lease sentinel identity when resources change", async () => { process.env.DAYTONA_API_KEY = "host-key"; @@ -1738,6 +1765,21 @@ describe("Daytona sandbox provider plugin", () => { })).rejects.toThrow("delete failed"); }); + it.each([false, true])("explicit stop retains an ephemeral sandbox even when stop fails: %s", async fail => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox({ id: "sandbox-stop-only", state: "started" }); + if (fail) sandbox.stop.mockRejectedValueOnce(new Error("stop failed")); + mockGet.mockResolvedValue(sandbox); + const stop = plugin.definition.onEnvironmentStopLease!({ + driverKey: "daytona", companyId: "company-1", environmentId: "env-1", + providerLeaseId: sandbox.id, config: { reuseLease: false }, + }); + if (fail) await expect(stop).rejects.toThrow("stop failed"); + else await expect(stop).resolves.toEqual({ providerLeaseId: sandbox.id, state: "stopped" }); + expect(sandbox.stop).toHaveBeenCalledTimes(1); + expect(sandbox.delete).not.toHaveBeenCalled(); + }); + it("stops reusable leases and deletes ephemeral leases on release", async () => { process.env.DAYTONA_API_KEY = "host-key"; const reusable = createMockSandbox({ id: "sandbox-reusable" }); @@ -1772,6 +1814,54 @@ describe("Daytona sandbox provider plugin", () => { expect(ephemeral.delete).toHaveBeenCalledWith(300, true); }); + it.each([false, true])("preserves an ephemeral completed workspace on explicit stop-and-retain (stop fails: %s)", async (stopFails) => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox({ id: "sandbox-saved-work", state: "started" }); + if (stopFails) sandbox.stop.mockRejectedValueOnce(new Error("injected stop transport failure")); + mockGet.mockResolvedValue(sandbox); + const stop = plugin.definition.onEnvironmentReleaseLease?.({ + driverKey: "daytona", companyId: "company-1", environmentId: "env-1", + providerLeaseId: sandbox.id, config: { timeoutMs: 300000, reuseLease: false }, + resourceDisposition: "stop_and_retain", + } as Parameters>[0]); + if (stopFails) await expect(stop).rejects.toThrow("injected stop transport failure"); + else await expect(stop).resolves.toEqual({ providerLeaseId: sandbox.id, state: "stopped" }); + expect(sandbox.stop).toHaveBeenCalledOnce(); + expect(sandbox.delete).not.toHaveBeenCalled(); + expect(sandbox.archive).not.toHaveBeenCalled(); + }); + + it.each(["confirmed", "setter_failed", "refresh_failed", "not_confirmed"])("disables provider auto-delete before preserving stopped work: %s", async kind => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = Object.assign(createMockSandbox({ id: "sandbox-auto-delete", state: "started" }), { autoDeleteInterval: 0 }); + let providerAutoDelete = 0, implicitlyDeleted = false; + sandbox.setAutoDeleteInterval.mockImplementation(async () => { + if (kind === "setter_failed") throw new Error("setter unavailable"); + sandbox.autoDeleteInterval = -1; // SDK setter updates its local field even without a fresh provider read. + if (kind !== "not_confirmed") providerAutoDelete = -1; + }); + sandbox.refreshData.mockImplementation(async () => { + if (kind === "refresh_failed") throw new Error("refresh unavailable"); + sandbox.autoDeleteInterval = providerAutoDelete; + }); + sandbox.stop.mockImplementation(async () => { implicitlyDeleted = providerAutoDelete === 0; }); + mockGet.mockResolvedValue(sandbox); + const stop = plugin.definition.onEnvironmentStopLease!({ driverKey: "daytona", companyId: "company-1", environmentId: "env-1", + providerLeaseId: sandbox.id, config: { reuseLease: false, autoDeleteInterval: 0 } }); + if (kind === "confirmed") { + await expect(stop).resolves.toEqual({ providerLeaseId: sandbox.id, state: "stopped" }); + expect(sandbox.setAutoDeleteInterval).toHaveBeenCalledWith(-1); + expect(sandbox.setAutoDeleteInterval.mock.invocationCallOrder[0]).toBeLessThan(sandbox.refreshData.mock.invocationCallOrder.at(-1)!); + expect(sandbox.refreshData.mock.invocationCallOrder.at(-1)!).toBeLessThan(sandbox.stop.mock.invocationCallOrder[0]); + } else { + await expect(stop).rejects.toThrow(); + expect(sandbox.stop).not.toHaveBeenCalled(); + } + expect(implicitlyDeleted).toBe(false); + expect(sandbox.delete).not.toHaveBeenCalled(); + expect(sandbox.archive).not.toHaveBeenCalled(); + }); + it("archives instead of deleting when the lease was acquired with archiveOnRelease", async () => { process.env.DAYTONA_API_KEY = "host-key"; const sandbox = createMockSandbox({ id: "sandbox-test-probe", state: "started" }); diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.ts index ac8465254b..84babf4986 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts @@ -182,6 +182,8 @@ interface DaytonaDriverConfig { type WorkspaceSentinelResult = { path: string; token: string | null; + runId?: string; + providerLeaseId?: string; result: "written" | "matched" | "missing" | "mismatch" | "skipped"; }; @@ -511,12 +513,12 @@ function hasMissingSandboxContainer(sandbox: Sandbox): boolean { `not found: failed to inspect sandbox container ${sandbox.id}: Error response from daemon: No such container: ${sandbox.id}`; } -async function resolveSandboxWorkingDirectory(sandbox: Sandbox): Promise { +async function resolveSandboxWorkingDirectory(sandbox: Sandbox, create = true): Promise { const root = (await sandbox.getWorkDir())?.trim() || (await sandbox.getUserHomeDir())?.trim() || "/home/daytona"; const remoteCwd = path.posix.join(root, "paperclip-workspace"); - await sandbox.fs.createFolder(remoteCwd, "755"); + if (create) await sandbox.fs.createFolder(remoteCwd, "755"); return remoteCwd; } @@ -544,15 +546,17 @@ function parseProbeInteger(value: string | undefined | null): number | null { } function workspaceSentinelToken(input: { - params: Pick; + params: Pick; + providerLeaseId: string; config: DaytonaDriverConfig; }): string | null { - if (!input.config.reuseLease || !input.params.agentId || (!input.params.executionWorkspaceId && !input.params.issueId)) { + if ((!input.config.reuseLease && !input.params.runId) || !input.params.agentId || (!input.params.executionWorkspaceId && !input.params.issueId)) { return null; } return createHash("sha256") .update(stableStringify({ provider: "daytona", + ...(!input.config.reuseLease ? { ephemeralRunId: input.params.runId, ephemeralProviderLeaseId: input.providerLeaseId } : {}), companyId: input.params.companyId, environmentId: input.params.environmentId, agentId: input.params.agentId, @@ -585,7 +589,7 @@ async function writeWorkspaceSentinel(input: { timeoutSeconds: number; }): Promise { const sentinelPath = workspaceSentinelPath(input.remoteCwd); - const token = workspaceSentinelToken({ params: input.params, config: input.config }); + const token = workspaceSentinelToken({ params: input.params, config: input.config, providerLeaseId: input.sandbox.id }); if (!token) { return { path: sentinelPath, token: null, result: "skipped" }; } @@ -606,7 +610,8 @@ async function writeWorkspaceSentinel(input: { sentinelPath, input.timeoutSeconds, ); - return { path: sentinelPath, token, result: "written" }; + return { path: sentinelPath, token, result: "written", + ...(!input.config.reuseLease ? { runId: input.params.runId!, providerLeaseId: input.sandbox.id } : {}) }; } async function verifyWorkspaceSentinel(input: { @@ -621,6 +626,8 @@ async function verifyWorkspaceSentinel(input: { const sentinelPath = typeof metadataSentinel?.path === "string" ? metadataSentinel.path : workspaceSentinelPath(input.remoteCwd); + const binding = typeof metadataSentinel?.runId === "string" && typeof metadataSentinel?.providerLeaseId === "string" + ? { runId: metadataSentinel.runId, providerLeaseId: metadataSentinel.providerLeaseId } : {}; const expectedToken = typeof metadataSentinel?.token === "string" ? metadataSentinel.token : null; if (!expectedToken) { return { path: sentinelPath, token: null, result: "missing" }; @@ -641,6 +648,7 @@ async function verifyWorkspaceSentinel(input: { return { path: sentinelPath, token: expectedToken, + ...binding, result: actualToken === expectedToken ? "matched" : "mismatch", }; } catch { @@ -2347,6 +2355,17 @@ const plugin = definePlugin({ params: PluginEnvironmentResumeLeaseParams, ): Promise { const config = parseDriverConfig(params.config); + if (params.leaseMetadata?.reuseLease === false) { + const sentinel = isRecord(params.leaseMetadata.workspaceSentinel) ? params.leaseMetadata.workspaceSentinel : null; + const intent = isRecord(params.leaseMetadata.nativeWorkspaceExportResume) ? params.leaseMetadata.nativeWorkspaceExportResume : null; + // Ephemeral recovery is bound at acquisition. Never mint proof during + // resume, accept a copied sentinel for a replacement, or waive legacy proof. + if (typeof sentinel?.token !== "string" || !sentinel.token || typeof sentinel.runId !== "string" || !sentinel.runId + || sentinel.providerLeaseId !== params.providerLeaseId || params.leaseMetadata.sandboxId !== params.providerLeaseId + || (intent && intent.runId !== sentinel.runId)) { + return { providerLeaseId: null, metadata: { expired: true, workspaceSentinel: { result: "mismatch" } } }; + } + } const scope: SandboxScope = { driverKey: params.driverKey, companyId: params.companyId, @@ -2403,7 +2422,7 @@ const plugin = definePlugin({ const resumedFromState = sandbox.state ?? null; await ensureSandboxStarted(sandbox, toTimeoutSeconds(config.timeoutMs)); try { - const remoteCwd = await resolveSandboxWorkingDirectory(sandbox); + const remoteCwd = await resolveSandboxWorkingDirectory(sandbox, false); // C3: a resumed lease must clear the workspace sentinel before it is // trusted, even when the handle came from the cache. On any non-match we // evict the cached handle and expire the lease so a stale/foreign sandbox @@ -2453,10 +2472,41 @@ const plugin = definePlugin({ ); }, + async onEnvironmentStopLease(params: PluginEnvironmentReleaseLeaseParams): Promise { + if (!params.providerLeaseId) throw new Error("Daytona stop requires an exact sandbox identity"); + const config = parseDriverConfig(params.config); + const scope: SandboxScope = { driverKey: params.driverKey, companyId: params.companyId, + environmentId: params.environmentId, providerLeaseId: params.providerLeaseId, config }; + const teardownGate = sandboxHandleTeardownGates.begin(scope); + sandboxHandleLeaseAdmissionStates.close(scope); + try { + const sandbox = await getSandboxOrNull(scope, { bypassTeardownGate: true }); + if (!sandbox) throw new Error("Daytona retained sandbox is unavailable"); + evictSandboxHandle(scope); + // Provider auto-delete can destroy a stopped sandbox without an explicit + // delete call. Confirm the provider disabled it before stopping saved work; + // the SDK setter also updates a local field, which is not sufficient proof. + const deadline = Math.min(config.livenessTimeoutMs, 30_000); + await withLivenessTimeout("sandbox.setAutoDeleteInterval", deadline, () => sandbox.setAutoDeleteInterval(-1)); + await withLivenessTimeout("sandbox.refreshData", deadline, () => sandbox.refreshData()); + if (sandbox.autoDeleteInterval !== -1) throw new Error("Daytona retention policy was not confirmed"); + if (sandbox.state !== "stopped") { + await terminateAtProvider(scope, "sandbox.stop", () => sandbox.stop(Math.min(toTimeoutSeconds(config.timeoutMs), 30))); + } + sandboxHandleSessionStore.clear(scope); + await closeDaytonaDuplexChannelsForLease(params.providerLeaseId); + return { providerLeaseId: params.providerLeaseId, state: "stopped" }; + } finally { + sandboxHandleTeardownGates.end(scope, teardownGate); + evictSandboxHandle(scope); + } + }, + async onEnvironmentReleaseLease( params: PluginEnvironmentReleaseLeaseParams, ): Promise { if (!params.providerLeaseId) return; + if (params.resourceDisposition === "stop_and_retain") return plugin.definition.onEnvironmentStopLease!(params); const config = parseDriverConfig(params.config); const scope: SandboxScope = { driverKey: params.driverKey, @@ -3061,6 +3111,7 @@ const plugin = definePlugin({ operations: params.operations, remoteDir, timeoutSeconds, + onArchiveRecovery: () => pluginContext?.logger.info("Workspace export omitted unsafe links; retrying with confined entries."), }); sandboxHandleCache.markFresh(scope); return result; diff --git a/packages/plugins/sandbox-providers/daytona/src/tar-listing.test.ts b/packages/plugins/sandbox-providers/daytona/src/tar-listing.test.ts new file mode 100644 index 0000000000..796e93bc00 --- /dev/null +++ b/packages/plugins/sandbox-providers/daytona/src/tar-listing.test.ts @@ -0,0 +1,191 @@ +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, expect, it, vi } from "vitest"; + +vi.mock("./plugin.js", () => ({ getPluginTracer: () => undefined })); +import { assertTarballEntriesConfined } from "./file-sync.js"; + +const temporaryDirectories: string[] = []; +afterEach(async () => { + vi.unstubAllEnvs(); + await Promise.all(temporaryDirectories.splice(0).map((dir) => fs.rm(dir, { recursive: true, force: true }))); +}); + +async function temporaryDirectory() { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), "daytona-tar-listing-")); + temporaryDirectories.push(dir); + return dir; +} + +// Empty USTAR members are sufficient to exercise the real host tar listing. +// Repeating a member is valid tar and avoids creating 145,000 host files just +// to reproduce the archive metadata buffer limit. +function tarHeader(name: string, prefix = "", target?: string): Buffer { + const header = Buffer.alloc(512); + header.write(name, 0, 100); + header.write("0000644\0", 100); + header.write("0000000\0", 108); + header.write("0000000\0", 116); + header.write("00000000000\0", 124); + header.write("00000000000\0", 136); + header.fill(32, 148, 156); + header.write(target === undefined ? "0" : "2", 156); + if (target !== undefined) header.write(target, 157, 100); + header.write("ustar\0", 257); + header.write("00", 263); + header.write(prefix, 345, 155); + const sum = header.reduce((total, value) => total + value, 0); + header.write(`${sum.toString(8).padStart(6, "0")}\0 `, 148); + return header; +} + +it.each([false, true])("validates every member beyond 32 MiB of real tar names (unsafe suffix: %s)", async (unsafeSuffix) => { + const archive = path.join(await temporaryDirectory(), "large.tar"); + const prefix = "nested-".repeat(21); + const name = "asset-".repeat(15); + const count = 145_000; + expect((prefix.length + 1 + name.length) * count).toBeGreaterThan(32 * 1024 * 1024); + const header = tarHeader(name, prefix); + const batch = Buffer.concat(Array.from({ length: 1_000 }, () => header)); + const file = await fs.open(archive, "w"); + try { + for (let written = 0; written < count; written += 1_000) await file.write(batch); + if (unsafeSuffix) await file.write(tarHeader("escape", "", "../../outside")); + await file.write(Buffer.alloc(1_024)); + } finally { + await file.close(); + } + if (unsafeSuffix) { + await expect(assertTarballEntriesConfined(archive)).rejects.toThrow("link whose target escapes"); + } else { + await expect(assertTarballEntriesConfined(archive)).resolves.toBeUndefined(); + } +}, 30_000); + +async function fakeTar(script: string) { + const directory = await temporaryDirectory(); + const pidFile = path.join(directory, "pid"); + await fs.writeFile(path.join(directory, "tar"), `#!${process.execPath}\n` + + `require("node:fs").appendFileSync(${JSON.stringify(pidFile)}, String(process.pid) + "\\n");\n${script}\n`, { mode: 0o755 }); + vi.stubEnv("PATH", `${directory}${path.delimiter}${process.env.PATH}`); + return async () => { + const pids = (await fs.readFile(pidFile, "utf8")).trim().split("\n").map(Number); + for (const pid of pids) expect(() => process.kill(pid, 0)).toThrow(); + }; +} + +const safeLine = "-rw-r--r-- 0/0 0 2026-09-27 12:00 nested/café.txt"; + +// The child streams batches with pipe backpressure; quota tests do not allocate +// a complete oversized listing in either the parent or the child. +function listingWriter(line: string, count: number, tail = "") { + return ` + const { once } = require("node:events"); + const line = Buffer.from(${JSON.stringify(line)}); + (async () => { + let left = ${count}; + while (left > 0) { + const size = Math.min(left, Math.max(1, Math.floor(65536 / line.length))); + const batch = Buffer.concat(Array.from({ length: size }, () => line)); + if (!process.stdout.write(batch)) await once(process.stdout, "drain"); + left -= size; + } + process.stdout.write(${JSON.stringify(tail)}); + })(); + `; +} + +it.each([false, true])("bounds aggregate listing bytes at 64 MiB (over quota: %s)", async (overQuota) => { + const line = "-rw-r--r-- 0/0 0 2026-09-27 12:00 ".padEnd(4095, "x") + "\n"; + const count = 64 * 1024 * 1024 / Buffer.byteLength(line) + Number(overQuota); + const checkReaped = await fakeTar(listingWriter(line, count)); + const result = assertTarballEntriesConfined("unused.tar"); + if (overQuota) await expect(result).rejects.toThrow("total listing byte limit"); + else await expect(result).resolves.toBeUndefined(); + await checkReaped(); +}, 30_000); + +it.each([false, true])("bounds aggregate listing entries at 250,000 (over quota: %s)", async (overQuota) => { + // The over-quota member has no newline: EOF must pass the same admission gate. + const checkReaped = await fakeTar(listingWriter(safeLine + "\n", 250_000, overQuota ? safeLine : "")); + const result = assertTarballEntriesConfined("unused.tar"); + if (overQuota) await expect(result).rejects.toThrow("listing entry limit"); + else await expect(result).resolves.toBeUndefined(); + await checkReaped(); +}, 30_000); + +it("counts blank lines against the aggregate parsing quota", async () => { + const checkReaped = await fakeTar(listingWriter("\n", 250_001)); + await expect(assertTarballEntriesConfined("unused.tar")).rejects.toThrow("listing entry limit"); + await checkReaped(); +}); + +it("enforces quotas independently on repeated and concurrent listings and reaps every child", async () => { + const checkReaped = await fakeTar(` + if (process.argv[3] === "small.tar") process.stdout.write(${JSON.stringify(safeLine + "\n")}); + else { ${listingWriter(safeLine + "\n", 250_001)} } + `); + const results = await Promise.allSettled([ + assertTarballEntriesConfined("large-a.tar"), + assertTarballEntriesConfined("small.tar"), + assertTarballEntriesConfined("large-b.tar"), + ]); + expect(results[0]).toMatchObject({ status: "rejected", reason: expect.objectContaining({ message: expect.stringContaining("listing entry limit") }) }); + expect(results[1]).toMatchObject({ status: "fulfilled" }); + expect(results[2]).toMatchObject({ status: "rejected", reason: expect.objectContaining({ message: expect.stringContaining("listing entry limit") }) }); + await expect(assertTarballEntriesConfined("large-again.tar")).rejects.toThrow("listing entry limit"); + await checkReaped(); +}, 30_000); +it("preserves UTF-8 split across chunks and checks a final line without a newline", async () => { + const checkReaped = await fakeTar(` + const bytes = Buffer.from(${JSON.stringify(safeLine)}); + const split = bytes.indexOf(Buffer.from("é")) + 1; + process.stdout.write(bytes.subarray(0, split)); + setTimeout(() => process.stdout.write(bytes.subarray(split)), 30); + `); + await expect(assertTarballEntriesConfined("unused.tar")).resolves.toBeUndefined(); + await checkReaped(); +}); + +it.each([ + "not a tar listing", + "-rw-r--r-- 0/0 0 2026-09-27 12:00 ../escape", + "lrwxrwxrwx 0/0 0 2026-09-27 12:00 link -> /outside", + "hrw-r--r-- 0/0 0 2026-09-27 12:00 link link to ../../outside", + "lrwxrwxrwx 0/0 0 2026-09-27 12:00 link -> decoy -> ../../outside", +])("rejects an unsafe final listing and reaps the child: %s", async (line) => { + const checkReaped = await fakeTar(`process.stdout.write(${JSON.stringify(line)});`); + await expect(assertTarballEntriesConfined("unused.tar")).rejects.toThrow("Daytona syncOut refusing"); + await checkReaped(); +}); + +it("bounds an unterminated entry instead of buffering it indefinitely", async () => { + const checkReaped = await fakeTar(`process.stdout.write("x".repeat(65_537)); setInterval(() => {}, 1_000);`); + await expect(assertTarballEntriesConfined("unused.tar")).rejects.toThrow("entry listing exceeding the byte limit"); + await checkReaped(); +}); + +it("bounds stderr and reaps a noisy tar process", async () => { + const checkReaped = await fakeTar(`process.stderr.write("x".repeat(65_537)); setInterval(() => {}, 1_000);`); + await expect(assertTarballEntriesConfined("unused.tar")).rejects.toThrow("diagnostics exceed the byte limit"); + await checkReaped(); +}); + +it("rejects nonzero tar exit even after a valid listing", async () => { + const checkReaped = await fakeTar(`process.stdout.write(${JSON.stringify(safeLine + "\n")}); process.exitCode = 2;`); + await expect(assertTarballEntriesConfined("unused.tar")).rejects.toThrow("tar listing failed (2)"); + await checkReaped(); +}); + +it("times out and reaps a stalled listing process", async () => { + const checkReaped = await fakeTar(`setInterval(() => {}, 1_000);`); + // Leave enough startup time for the PID witness under parallel test load. + await expect(assertTarballEntriesConfined("unused.tar", 2_000)).rejects.toThrow("timed out"); + await checkReaped(); +}); + +it("rejects spawn failures without hanging", async () => { + vi.stubEnv("PATH", await temporaryDirectory()); + await expect(assertTarballEntriesConfined("unused.tar")).rejects.toThrow("ENOENT"); +}); diff --git a/packages/plugins/sdk/src/define-plugin.ts b/packages/plugins/sdk/src/define-plugin.ts index 5d2763f92f..ef5c5dc6f9 100644 --- a/packages/plugins/sdk/src/define-plugin.ts +++ b/packages/plugins/sdk/src/define-plugin.ts @@ -387,6 +387,13 @@ export interface PluginDefinition { params: PluginEnvironmentReleaseLeaseParams, ): Promise; + /** Stop this exact allocation and retain all files, regardless of release + * policy. Throw if stop cannot be confirmed; never destroy as a fallback. + * Separate worker discovery lets the host safely defer older providers. */ + onEnvironmentStopLease?( + params: PluginEnvironmentReleaseLeaseParams, + ): Promise; + /** Called when the host needs to force-destroy provider state. */ onEnvironmentDestroyLease?( params: PluginEnvironmentDestroyLeaseParams, diff --git a/packages/plugins/sdk/src/protocol.ts b/packages/plugins/sdk/src/protocol.ts index a599a8fc7e..4605866261 100644 --- a/packages/plugins/sdk/src/protocol.ts +++ b/packages/plugins/sdk/src/protocol.ts @@ -662,6 +662,9 @@ export interface PluginEnvironmentResumeLeaseParams extends PluginEnvironmentDri } export interface PluginEnvironmentReleaseLeaseParams extends PluginEnvironmentDriverBaseParams { + /** Stop the exact allocation while preserving its files, regardless of its + * ordinary release policy. A failed stop must throw, never fall back to delete. */ + resourceDisposition?: "stop_and_retain"; /** Explicit operator cancellation: terminate active work instead of waiting * for command/sync activity to drain. Still requires a provider receipt. */ cancelActiveWork?: boolean; @@ -1375,6 +1378,10 @@ export interface HostToWorkerMethods { params: PluginEnvironmentReleaseLeaseParams, result: PluginEnvironmentTerminationReceipt | void, ]; + environmentStopLease: [ + params: PluginEnvironmentReleaseLeaseParams, + result: PluginEnvironmentTerminationReceipt, + ]; environmentDestroyLease: [ params: PluginEnvironmentDestroyLeaseParams, result: PluginEnvironmentTerminationReceipt | void, @@ -1478,6 +1485,7 @@ export const HOST_TO_WORKER_OPTIONAL_METHODS: readonly HostToWorkerMethodName[] "environmentAcquireLease", "environmentResumeLease", "environmentReleaseLease", + "environmentStopLease", "environmentDestroyLease", "environmentRealizeWorkspace", "environmentExecute", diff --git a/packages/plugins/sdk/src/testing.ts b/packages/plugins/sdk/src/testing.ts index 799ec106c8..07b4cc0d5e 100644 --- a/packages/plugins/sdk/src/testing.ts +++ b/packages/plugins/sdk/src/testing.ts @@ -161,6 +161,7 @@ export interface EnvironmentEventRecord { | "acquireLease" | "resumeLease" | "releaseLease" + | "stopLease" | "destroyLease" | "realizeWorkspace" | "execute" @@ -187,6 +188,7 @@ export interface EnvironmentTestHarnessOptions extends TestHarnessOptions { onAcquireLease?: (params: PluginEnvironmentAcquireLeaseParams) => Promise; onResumeLease?: (params: PluginEnvironmentResumeLeaseParams) => Promise; onReleaseLease?: (params: PluginEnvironmentReleaseLeaseParams) => Promise; + onStopLease?: (params: PluginEnvironmentReleaseLeaseParams) => Promise; onDestroyLease?: (params: PluginEnvironmentDestroyLeaseParams) => Promise; onRealizeWorkspace?: (params: PluginEnvironmentRealizeWorkspaceParams) => Promise; onExecute?: (params: PluginEnvironmentExecuteParams) => Promise; @@ -212,6 +214,8 @@ export interface EnvironmentTestHarness extends TestHarness { resumeLease(params: PluginEnvironmentResumeLeaseParams): Promise; /** Invoke the environment driver's releaseLease hook. */ releaseLease(params: PluginEnvironmentReleaseLeaseParams): Promise; + /** Stop and preserve an allocation independently of its release policy. */ + stopLease(params: PluginEnvironmentReleaseLeaseParams): Promise; /** Invoke the environment driver's destroyLease hook. */ destroyLease(params: PluginEnvironmentDestroyLeaseParams): Promise; /** Invoke the environment driver's realizeWorkspace hook. */ @@ -2727,6 +2731,9 @@ export function createEnvironmentTestHarness(options: EnvironmentTestHarnessOpti async releaseLease(params) { return callHook("releaseLease", driver.onReleaseLease, params, "onReleaseLease"); }, + async stopLease(params) { + return callHook("stopLease", driver.onStopLease, params, "onStopLease"); + }, async destroyLease(params) { return callHook("destroyLease", driver.onDestroyLease, params, "onDestroyLease"); }, diff --git a/packages/plugins/sdk/src/worker-rpc-host.ts b/packages/plugins/sdk/src/worker-rpc-host.ts index edf9634901..78d8308a29 100644 --- a/packages/plugins/sdk/src/worker-rpc-host.ts +++ b/packages/plugins/sdk/src/worker-rpc-host.ts @@ -1639,6 +1639,9 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost case "environmentReleaseLease": return handleEnvironmentReleaseLease(params as PluginEnvironmentReleaseLeaseParams); + case "environmentStopLease": + if (!plugin.definition.onEnvironmentStopLease) throw methodNotImplemented("environmentStopLease"); + return plugin.definition.onEnvironmentStopLease(params as PluginEnvironmentReleaseLeaseParams); case "environmentDestroyLease": return handleEnvironmentDestroyLease(params as PluginEnvironmentDestroyLeaseParams); @@ -1740,6 +1743,7 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost if (plugin.definition.onEnvironmentAcquireLease) supportedMethods.push("environmentAcquireLease"); if (plugin.definition.onEnvironmentResumeLease) supportedMethods.push("environmentResumeLease"); if (plugin.definition.onEnvironmentReleaseLease) supportedMethods.push("environmentReleaseLease"); + if (plugin.definition.onEnvironmentStopLease) supportedMethods.push("environmentStopLease"); if (plugin.definition.onEnvironmentDestroyLease) supportedMethods.push("environmentDestroyLease"); if (plugin.definition.onEnvironmentRealizeWorkspace) supportedMethods.push("environmentRealizeWorkspace"); if (plugin.definition.onEnvironmentExecute) supportedMethods.push("environmentExecute"); diff --git a/packages/plugins/sdk/tests/environment-sync-negotiation.test.ts b/packages/plugins/sdk/tests/environment-sync-negotiation.test.ts index 786ca64d98..383247023a 100644 --- a/packages/plugins/sdk/tests/environment-sync-negotiation.test.ts +++ b/packages/plugins/sdk/tests/environment-sync-negotiation.test.ts @@ -249,3 +249,29 @@ describe("environment sync verb negotiation", () => { } }); }); + + +describe("environment stop-and-retain negotiation", () => { + it.each([false, true])("dispatches only an explicitly advertised stop hook: %s", async supported => { + let releases = 0; + const worker = startTestWorker(definePlugin({ + async setup() {}, + async onEnvironmentReleaseLease() { releases += 1; }, + ...(supported ? { async onEnvironmentStopLease(params: { providerLeaseId: string | null }) { + return { providerLeaseId: params.providerLeaseId!, state: "stopped" as const }; + } } : {}), + })); + try { + const result = await worker.callWorker<{ supportedMethods: string[] }>("initialize", { + manifest: MANIFEST, config: {}, databaseNamespace: null, + }); + expect(result.supportedMethods.includes("environmentStopLease")).toBe(supported); + const call = worker.callWorker("environmentStopLease", { + driverKey: "daytona", companyId: "company", environmentId: "env", config: {}, providerLeaseId: "lease-1", + }); + if (supported) await expect(call).resolves.toEqual({ providerLeaseId: "lease-1", state: "stopped" }); + else await expect(call).rejects.toMatchObject({ code: PLUGIN_RPC_ERROR_CODES.METHOD_NOT_IMPLEMENTED }); + expect(releases).toBe(0); + } finally { worker.stop(); } + }); +}); diff --git a/packages/plugins/sdk/tests/worker-rpc-host.test.ts b/packages/plugins/sdk/tests/worker-rpc-host.test.ts index 6652995631..b504a70362 100644 --- a/packages/plugins/sdk/tests/worker-rpc-host.test.ts +++ b/packages/plugins/sdk/tests/worker-rpc-host.test.ts @@ -342,7 +342,7 @@ describe("worker configChanged cross-tenant guard", () => { }); async function initialize() { - await callWorker("initialize", { + return await callWorker("initialize", { manifest: { id: "paperclip.config-guard-test", apiVersion: 1, @@ -369,6 +369,23 @@ describe("worker configChanged cross-tenant guard", () => { return { callWorker, initialize, stop }; } + it.each([false, true])("advertises and dispatches stop-only only with an explicit hook: %s", async supported => { + let releases = 0, stops = 0; + const worker = makeWorker(definePlugin({ async setup() {}, + async onEnvironmentReleaseLease() { releases++; return { providerLeaseId: "allocation", state: "destroyed" }; }, + ...(supported ? { async onEnvironmentStopLease() { stops++; return { providerLeaseId: "allocation", state: "stopped" as const }; } } : {}), + })); + try { + const initialized = await worker.initialize() as { supportedMethods: string[] }; + expect(initialized.supportedMethods.includes("environmentStopLease")).toBe(supported); + const stopped = worker.callWorker("environmentStopLease", { driverKey: "fixture", companyId: "company", environmentId: "environment", providerLeaseId: "allocation", config: {} }); + if (supported) await expect(stopped).resolves.toEqual({ providerLeaseId: "allocation", state: "stopped" }); + else await expect(stopped).rejects.toThrow(); + expect(stops).toBe(supported ? 1 : 0); + expect(releases).toBe(0); + } finally { worker.stop(); } + }); + it("fails closed when a second, distinct company's config would overwrite a single-tenant worker", async () => { const applied: Array<{ companyId: string | null; token: unknown }> = []; const plugin = definePlugin({ diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index 9ae73b4504..04b1216786 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -1960,6 +1960,7 @@ export { issueExecutionPolicySchema, issueExecutionStateSchema, resolveIssueRecoveryActionSchema, + retryWorkspaceExportSchema, issueReviewRequestSchema, issueExecutionWorkspaceSettingsSchema, checkoutIssueSchema, @@ -2787,4 +2788,4 @@ export * from "./slack-tools.js"; export { MEMORY_CONNECTOR_IDS, isMemoryConnectorId, type MemoryConnectorId } from "./memory-connectors.js"; export * from "./connection-routing.js"; -export { WORKSPACE_RESTORE_FAILURE_CODES, hasWorkspaceRestoreFailure, safeWorkspaceRestorePath } from "./workspace-restore.js"; +export { WORKSPACE_RESTORE_FAILURE_CODES, hasWorkspaceRestoreFailure, safeWorkspaceRestorePath, isNativeWorkspaceExportRepairCause } from "./workspace-restore.js"; diff --git a/packages/shared/src/validators/index.ts b/packages/shared/src/validators/index.ts index 09c4fef7c4..a39388a080 100644 --- a/packages/shared/src/validators/index.ts +++ b/packages/shared/src/validators/index.ts @@ -433,6 +433,7 @@ export { issueExecutionStateSchema, issueRecoveryActionReadModelSchema, resolveIssueRecoveryActionSchema, + retryWorkspaceExportSchema, issueReviewRequestSchema, issueExecutionWorkspaceSettingsSchema, checkoutIssueSchema, diff --git a/packages/shared/src/validators/issue.ts b/packages/shared/src/validators/issue.ts index dfedf02d22..96ab22cbe9 100644 --- a/packages/shared/src/validators/issue.ts +++ b/packages/shared/src/validators/issue.ts @@ -555,6 +555,12 @@ const RESOLVE_ISSUE_RECOVERY_ACTION_OUTCOMES = [ "cancelled", ] as const; +export const retryWorkspaceExportSchema = z.object({ + actionId: z.string().guid(), + runId: z.string().guid(), + repairNote: z.string().trim().min(20).max(12000), +}).strict(); + export const resolveIssueRecoveryActionSchema = z .object({ executionReconciliation: z diff --git a/packages/shared/src/workspace-restore.ts b/packages/shared/src/workspace-restore.ts index caa01b366c..65da970896 100644 --- a/packages/shared/src/workspace-restore.ts +++ b/packages/shared/src/workspace-restore.ts @@ -6,6 +6,11 @@ export const WORKSPACE_RESTORE_FAILURE_CODES = [ "restore_failed", ] as const; +/** These failures preserve an accepted native result for board-owned export repair. */ +export function isNativeWorkspaceExportRepairCause(value: unknown): boolean { + return value === "native_workspace_sync_out_retry_exhausted"; +} + export function hasWorkspaceRestoreFailure(result: Record | null | undefined): boolean { return WORKSPACE_RESTORE_FAILURE_CODES.some((code) => result?.workspaceRestoreFailure === code); } diff --git a/server/src/__tests__/environment-runtime.test.ts b/server/src/__tests__/environment-runtime.test.ts index 8ec03b56d3..7886155e22 100644 --- a/server/src/__tests__/environment-runtime.test.ts +++ b/server/src/__tests__/environment-runtime.test.ts @@ -483,14 +483,14 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { if (method === "environmentResumeLease") return { providerLeaseId: warmProviderLeaseId, metadata: { remoteCwd: "/workspace" }, }; - if (method === "environmentDestroyLease" || method === "environmentReleaseLease") return { + if (method === "environmentDestroyLease" || method === "environmentStopLease" || method === "environmentReleaseLease") return { providerLeaseId: input.providerLeaseId, state: method === "environmentDestroyLease" ? "destroyed" : "stopped", }; throw new Error(`Unexpected lifecycle method: ${method}`); }); const runtime = environmentRuntimeService(db, { pluginWorkerManager: { isRunning: () => true, call, - getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }), + getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentStopLease", "environmentDestroyLease"] }), } as unknown as PluginWorkerManager }); const input = { companyId: seeded.companyId, agentId: seeded.agentId, issueId, @@ -531,11 +531,11 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }); const workerManager = { isRunning: () => true, call, - getWorker: () => ({ supportedMethods: ["environmentReleaseLease"] }), + getWorker: () => ({ supportedMethods: ["environmentStopLease"] }), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); await runtimeWithPlugin.releaseRunLeases(runId, "released", undefined, "stop_and_retain", true); - expect(call).toHaveBeenCalledWith(pluginId, "environmentReleaseLease", expect.objectContaining({ + expect(call).toHaveBeenCalledWith(pluginId, "environmentStopLease", expect.objectContaining({ companyId, providerLeaseId: reusableLease.providerLeaseId, cancelActiveWork: true, }), expect.any(Number)); expect(call).toHaveBeenCalledOnce(); @@ -544,6 +544,66 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }, ); + it.each(["release_only", "stopped", "stop_failed", "unconfirmed", "restart", "competing_owner", "missing_pin", "late_receipt"])( + "never dispatches destructive release for an untagged stop-and-retain request: %s", async outcome => { + const seeded = await seedReusablePluginSandboxLease("paperclip_runner"); + const lease = seeded.reusableLease; + await db.update(environmentLeases).set({ leasePolicy: "ephemeral", metadata: { + ...lease.metadata, reuseLease: false, ...(outcome === "missing_pin" ? { pluginId: undefined } : {}), + } }).where(eq(environmentLeases.id, lease.id)); + await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, seeded.runId)); + let supported = !["release_only", "restart"].includes(outcome); + let calls = 0; + const call = vi.fn(async (_id: string, method: string) => { + if (method !== "environmentStopLease") return { providerLeaseId: lease.providerLeaseId, state: "destroyed" }; + if (++calls === 1 && outcome === "stop_failed") throw new Error("injected stop failure"); + if (calls === 1 && outcome === "unconfirmed") return undefined; + if (outcome === "late_receipt") await db.update(environmentLeases).set({ + status: "active", heartbeatRunId: null, cleanupStatus: null, metadata: { newOwner: true }, + }).where(eq(environmentLeases.id, lease.id)); + return { providerLeaseId: lease.providerLeaseId, state: "stopped" }; + }); + const runtime = () => environmentRuntimeService(db, { pluginWorkerManager: { + isRunning: () => true, call, + getWorker: () => ({ supportedMethods: ["environmentReleaseLease", "environmentDestroyLease", ...(supported ? ["environmentStopLease"] : [])] }), + } as unknown as PluginWorkerManager }); + if (outcome === "competing_owner") await environmentService(db).acquireLease({ + companyId: seeded.companyId, environmentId: seeded.environment.id, heartbeatRunId: null, + leasePolicy: "ephemeral", provider: lease.provider, providerLeaseId: lease.providerLeaseId, + }); + await runtime().releaseRunLeases(seeded.runId, "failed", undefined, "stop_and_retain"); + expect(call.mock.calls.every(entry => entry[1] === "environmentStopLease")).toBe(true); + if (outcome === "late_receipt") { + expect(await environmentService(db).getLeaseById(lease.id)).toMatchObject({ status: "active", heartbeatRunId: null, metadata: { newOwner: true } }); + return; + } + if (outcome === "missing_pin") { + expect(call).not.toHaveBeenCalled(); + await heartbeatService(db, { environmentRuntime: runtime() }).sweepPendingCleanupLeases({ backoffMs: 0 }); + expect(call).not.toHaveBeenCalled(); + expect(await environmentService(db).getLeaseById(lease.id)).toMatchObject({ status: "pending_cleanup" }); + return; + } + if (["release_only", "restart", "competing_owner"].includes(outcome)) { + expect(call).not.toHaveBeenCalled(); + expect(await environmentService(db).getLeaseById(lease.id)).toMatchObject({ status: "pending_cleanup", metadata: { + sandboxStopAndRetain: { pluginId: seeded.pluginId, runId: seeded.runId, providerLeaseId: lease.providerLeaseId }, + } }); + await heartbeatService(db, { environmentRuntime: runtime() }).sweepPendingCleanupLeases({ backoffMs: 0 }); + expect(call).not.toHaveBeenCalled(); + if (outcome === "competing_owner") return; + supported = true; + } + if (["release_only", "stop_failed", "unconfirmed", "restart"].includes(outcome)) { + await heartbeatService(db, { environmentRuntime: runtime() }).sweepPendingCleanupLeases({ backoffMs: 0 }); + } + expect(await environmentService(db).getLeaseById(lease.id)).toMatchObject({ status: "released", cleanupStatus: "success", failureReason: null, + metadata: { remoteExecutionTermination: { state: "stopped", providerLeaseId: lease.providerLeaseId }, + sandboxStopAndRetainReceipt: { method: "environmentStopLease", runId: seeded.runId, pluginId: seeded.pluginId } } }); + expect(call.mock.calls.every(entry => entry[1] === "environmentStopLease")).toBe(true); + }, + ); + it("reports an unwired manager separately from a stopped sandbox worker", async () => { const { companyId, environment, runId } = await seedReusablePluginSandboxLease(); const input = { @@ -609,13 +669,14 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { const workerManager = { isRunning: vi.fn((id: string) => id === pluginId), call: vi.fn(async (_pluginId: string, method: string) => { - if (method === "environmentReleaseLease") return undefined; + if (method === "environmentStopLease") return { providerLeaseId: reusableLease.providerLeaseId, state: "stopped" }; throw new Error(`Unexpected plugin method while stopping lease: ${method}`); }), getWorker: vi.fn(() => ({ supportedMethods: [ "environmentResumeLease", "environmentReleaseLease", + "environmentStopLease", "environmentDestroyLease", ], })), @@ -635,7 +696,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }); expect(workerManager.call).toHaveBeenCalledWith( pluginId, - "environmentReleaseLease", + "environmentStopLease", expect.objectContaining({ providerLeaseId: reusableLease.providerLeaseId }), expect.any(Number), ); @@ -663,12 +724,12 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { if (method === "environmentResumeLease") return { providerLeaseId: "projectless-1", metadata: { remoteCwd: "/workspace" }, }; - if (method === "environmentReleaseLease") return undefined; + if (method === "environmentStopLease") return { providerLeaseId: "projectless-1", state: "stopped" }; throw new Error(`Unexpected projectless lease method: ${method}`); }); const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: { isRunning: () => true, call, - getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }), + getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentStopLease", "environmentDestroyLease"] }), } as unknown as PluginWorkerManager }); const first = await runtimeWithPlugin.acquireRunLease({ companyId: seeded.companyId, environment: seeded.environment, issueId, @@ -841,7 +902,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }, }; } - if (method === "environmentReleaseLease") return undefined; + if (method === "environmentStopLease") return { providerLeaseId: "sandbox-exact-resume", state: "stopped" }; if (method === "environmentResumeLease") { return { providerLeaseId: "sandbox-exact-resume", @@ -860,6 +921,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { supportedMethods: [ "environmentResumeLease", "environmentReleaseLease", + "environmentStopLease", "environmentDestroyLease", ], })), @@ -1264,6 +1326,64 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { expect(released[0]?.lease.status).toBe("released"); }); + it.each([false, true])("confirms a built-in fake stop without release or destroy fallback: cancel=%s", async cancel => { + const { companyId, environment, runId } = await seedEnvironment({ + driver: "sandbox", config: { provider: "fake", image: "ubuntu:24.04", reuseLease: false }, + }); + const acquired = await runtime.acquireRunLease({ companyId, environment, issueId: null, + heartbeatRunId: runId, persistedExecutionWorkspace: null }); + const provider = sandboxProviderRuntime.requireSandboxProvider("fake"); + const release = vi.spyOn(provider, "releaseLease"), destroy = vi.spyOn(provider, "destroyLease"); + const released = await runtime.releaseRunLeases(runId, "released", undefined, "stop_and_retain", cancel); + expect(released).toHaveLength(1); + expect(released[0]?.lease).toMatchObject({ status: "released", cleanupStatus: "success", failureReason: null, + metadata: { remoteExecutionTermination: { state: "stopped", providerLeaseId: acquired.lease.providerLeaseId }, + sandboxStopAndRetainReceipt: { method: "builtin.stopLease", builtinProvider: "fake", runId } } }); + expect(await remoteExecutionHasStopped(db, companyId, runId)).toBe(true); + expect(release).not.toHaveBeenCalled(); + expect(destroy).not.toHaveBeenCalled(); + }); + + it.each(["stop_failed", "wrong_receipt", "missing_capability", "changed_binding"])( + "preserves a built-in stop through restart without destructive fallback: %s", async outcome => { + const { companyId, environment, runId } = await seedEnvironment({ + driver: "sandbox", config: { provider: "fake", image: "ubuntu:24.04", reuseLease: false }, + }); + const acquired = await runtime.acquireRunLease({ companyId, environment, issueId: null, + heartbeatRunId: runId, persistedExecutionWorkspace: null }); + const provider = sandboxProviderRuntime.requireSandboxProvider("fake"), originalStop = provider.stopLease!; + const release = vi.spyOn(provider, "releaseLease"), destroy = vi.spyOn(provider, "destroyLease"); + try { + provider.stopLease = outcome === "missing_capability" ? undefined : vi.fn(async () => { + if (outcome === "wrong_receipt") return { providerLeaseId: "sandbox://fake/other", state: "stopped" as const }; + throw new Error("injected built-in stop failure"); + }); + await runtime.releaseRunLeases(runId, "released", undefined, "stop_and_retain", true); + expect(await environmentService(db).getLeaseById(acquired.lease.id)).toMatchObject({ status: "pending_cleanup" }); + expect(await remoteExecutionHasStopped(db, companyId, runId)).toBe(false); + if (outcome === "missing_capability") { + await heartbeatService(db, { environmentRuntime: environmentRuntimeService(db) }).sweepPendingCleanupLeases({ backoffMs: 0 }); + expect(await environmentService(db).getLeaseById(acquired.lease.id)).toMatchObject({ status: "pending_cleanup" }); + } + const stop = vi.fn(originalStop.bind(provider)); + provider.stopLease = stop; + if (outcome === "changed_binding") await db.update(environmentLeases).set({ providerLeaseId: "sandbox://fake/replacement" }) + .where(eq(environmentLeases.id, acquired.lease.id)); + await heartbeatService(db, { environmentRuntime: environmentRuntimeService(db) }).sweepPendingCleanupLeases({ backoffMs: 0 }); + if (outcome === "changed_binding") { + expect(stop).not.toHaveBeenCalled(); + expect(await environmentService(db).getLeaseById(acquired.lease.id)).toMatchObject({ status: "pending_cleanup" }); + } else { + expect(stop).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ providerLeaseId: acquired.lease.providerLeaseId })); + expect(await environmentService(db).getLeaseById(acquired.lease.id)).toMatchObject({ status: "released", cleanupStatus: "success" }); + expect(await remoteExecutionHasStopped(db, companyId, runId)).toBe(true); + } + expect(release).not.toHaveBeenCalled(); + expect(destroy).not.toHaveBeenCalled(); + } finally { provider.stopLease = originalStop; } + }, + ); + it("releases the remote sandbox when the lease insert rejects a foreign-company binding", async () => { const { companyId, environment, runId } = await seedEnvironment({ driver: "sandbox", @@ -5015,6 +5135,113 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }); }); + it("resumes only the exact stopped sandbox lifecycle without acquiring or reseeding", async () => { + const seeded = await seedReusablePluginSandboxLease("paperclip_runner"); + await environmentService(db).releaseLease(seeded.reusableLease.id, "released", { cleanupStatus: "success" }); + const lease = (await environmentService(db).getLeaseById(seeded.reusableLease.id))!; + const call = vi.fn(async (_id: string, method: string) => { + if (method !== "environmentResumeLease") throw new Error("Only exact resume is permitted"); + return { providerLeaseId: lease.providerLeaseId, metadata: { remoteCwd: "/workspace" } }; + }); + const workerManager = { isRunning: () => true, call, + getWorker: () => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] }), + } as unknown as PluginWorkerManager; + const runtime = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + await expect(runtime.resumeRunLease({ environment: seeded.environment, lease })).resolves.toMatchObject({ providerLeaseId: lease.providerLeaseId }); + expect(call).toHaveBeenCalledOnce(); + expect(call).toHaveBeenCalledWith(seeded.pluginId, "environmentResumeLease", expect.objectContaining({ + companyId: seeded.companyId, environmentId: seeded.environment.id, providerLeaseId: lease.providerLeaseId, + leaseMetadata: lease.metadata, + }), expect.any(Number)); + expect((await environmentService(db).getLeaseById(lease.id))?.status).toBe("released"); + expect(await db.select().from(environmentLeases)).toHaveLength(1); + }); + + it.each(["stopped", "unconfirmed", "foreign_marker", "competing_owner", "retained_owner", "no_stop_capability", "live_request", "late_receipt", "duplicate_key", "owner_stopped", "owner_driver_changed", "owner_missing", "legacy_intent", "foreign_plugin_pin", "v2_missing_pin"])("recovers a pending export resume with stop-only cleanup after restart: %s", async outcome => { + const seeded = await seedReusablePluginSandboxLease("paperclip_runner"); + const lease = seeded.reusableLease; + const requestId = randomUUID(); + if (["duplicate_key", "owner_stopped", "owner_driver_changed", "owner_missing", "legacy_intent"].includes(outcome)) { + const [owner] = await db.select().from(plugins).where(eq(plugins.id, seeded.pluginId)); + await db.insert(plugins).values({ ...owner, id: randomUUID(), pluginKey: "other.same-driver", packageName: "@other/same-driver", + installOrder: -1, manifestJson: { ...owner.manifestJson, id: "other.same-driver" } }); + if (outcome === "owner_driver_changed") await db.update(plugins).set({ manifestJson: { ...owner.manifestJson, environmentDrivers: [] } }).where(eq(plugins.id, seeded.pluginId)); + if (outcome === "owner_missing") await db.delete(plugins).where(eq(plugins.id, seeded.pluginId)); + } + await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", metadata: { + ...lease.metadata, + nativeWorkspaceExportResume: { schema: outcome === "v2_missing_pin" ? "paperclip.workspace-export-resume.v2" : "paperclip.workspace-export-resume.v1", requestId, companyId: seeded.companyId, + ...(["legacy_intent", "v2_missing_pin"].includes(outcome) ? {} : { pluginId: outcome === "foreign_plugin_pin" ? randomUUID() : seeded.pluginId }), + runId: outcome === "foreign_marker" ? randomUUID() : lease.heartbeatRunId, leaseId: lease.id, + provider: lease.provider, providerLeaseId: lease.providerLeaseId, resultId: randomUUID() }, + pendingCleanupAttemptId: requestId, pendingCleanupInFlight: true, + pendingCleanupLeaseExpiresAtMs: Date.now() + (outcome === "live_request" ? 60_000 : -1), + } }).where(eq(environmentLeases.id, lease.id)); + if (["competing_owner", "retained_owner"].includes(outcome)) await db.insert(environmentLeases).values({ companyId: seeded.companyId, + environmentId: seeded.environment.id, status: outcome === "retained_owner" ? "retained" : "active", provider: lease.provider, providerLeaseId: lease.providerLeaseId }); + const call = vi.fn(async (_id: string, method: string) => { + if (method !== "environmentStopLease") throw new Error("Saved workspace must never be destroyed"); + if (outcome === "late_receipt") await db.update(environmentLeases).set({ status: "active", heartbeatRunId: null, + cleanupStatus: null, metadata: { newOwner: true } }).where(eq(environmentLeases.id, lease.id)); + return outcome === "unconfirmed" ? undefined : { providerLeaseId: lease.providerLeaseId, state: "stopped" }; + }); + const workerManager = { isRunning: (id: string) => !(outcome === "owner_stopped" && id === seeded.pluginId), call, + getWorker: () => ({ supportedMethods: outcome === "no_stop_capability" ? ["environmentReleaseLease", "environmentDestroyLease"] : ["environmentStopLease", "environmentDestroyLease"] }), + } as unknown as PluginWorkerManager; + const restarted = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + await heartbeatService(db, { environmentRuntime: restarted }).sweepPendingCleanupLeases({ backoffMs: 0 }); + expect(call.mock.calls.some((entry) => entry[1] === "environmentDestroyLease")).toBe(false); + const persisted = (await environmentService(db).getLeaseById(lease.id))!; + if (["stopped", "duplicate_key", "legacy_intent"].includes(outcome)) { + expect(call).toHaveBeenCalledWith(seeded.pluginId, "environmentStopLease", expect.objectContaining({ providerLeaseId: lease.providerLeaseId, cancelActiveWork: true }), expect.any(Number)); + expect(persisted).toMatchObject({ status: "released", cleanupStatus: "success", metadata: { remoteExecutionTermination: { state: "stopped", runId: lease.heartbeatRunId } } }); + expect(persisted.metadata?.nativeWorkspaceExportResume).toBeUndefined(); + } else if (outcome === "late_receipt") { + expect(persisted).toMatchObject({ status: "active", heartbeatRunId: null, cleanupStatus: null, metadata: { newOwner: true } }); + } else { + expect(persisted).toMatchObject({ status: "pending_cleanup", cleanupStatus: "failed" }); + expect(persisted.metadata?.remoteExecutionTermination).toBeUndefined(); + if (["foreign_marker", "foreign_plugin_pin", "v2_missing_pin", "competing_owner", "retained_owner", "no_stop_capability", "live_request", "owner_stopped", "owner_driver_changed", "owner_missing"].includes(outcome)) expect(call).not.toHaveBeenCalled(); + if (["owner_stopped", "owner_driver_changed", "owner_missing"].includes(outcome)) { + expect(persisted.metadata?.pendingCleanupAttemptId).toBe(requestId); + } + } + }); + + it.each(["stopped", "stop_failed", "unconfirmed", "restart", "committed_release", "inflight"])("preserves terminal ephemeral export work through release and restart: %s", async outcome => { + const seeded = await seedReusablePluginSandboxLease("paperclip_runner"); + const lease = seeded.reusableLease, requestId = randomUUID(); + await db.update(heartbeatRuns).set({ status: outcome === "committed_release" ? "succeeded" : "failed" }).where(eq(heartbeatRuns.id, seeded.runId)); + await db.update(environmentLeases).set({ status: outcome === "committed_release" ? "active" : "pending_cleanup", leasePolicy: "ephemeral", cleanupStatus: "failed", metadata: { + ...lease.metadata, reuseLease: false, + nativeWorkspaceExportResume: { schema: "paperclip.workspace-export-resume.v2", purpose: "terminal_export", requestId, + companyId: seeded.companyId, pluginId: seeded.pluginId, runId: seeded.runId, resultId: randomUUID(), + leaseId: lease.id, provider: lease.provider, providerLeaseId: lease.providerLeaseId }, + pendingCleanupAttemptId: requestId, pendingCleanupInFlight: outcome === "inflight", + pendingCleanupLeaseExpiresAtMs: outcome === "inflight" ? Date.now() + 60_000 : 0, + } }).where(eq(environmentLeases.id, lease.id)); + let calls = 0; + const call = vi.fn(async (_id: string, method: string, params: any) => { + expect(method).toBe("environmentStopLease"); + expect(params).toMatchObject({ providerLeaseId: lease.providerLeaseId, resourceDisposition: "stop_and_retain", config: { reuseLease: false } }); + if (++calls === 1 && outcome === "stop_failed") throw new Error("injected stop failure"); + if (calls === 1 && outcome === "unconfirmed") return undefined; + return { providerLeaseId: lease.providerLeaseId, state: "stopped" }; + }); + const runtime = () => environmentRuntimeService(db, { pluginWorkerManager: { isRunning: () => true, call, + getWorker: () => ({ supportedMethods: ["environmentStopLease", "environmentDestroyLease"] }), + } as unknown as PluginWorkerManager }); + if (outcome === "restart") await heartbeatService(db, { environmentRuntime: runtime() }).sweepPendingCleanupLeases({ backoffMs: 0 }); + else await runtime().releaseRunLeases(seeded.runId, "failed", undefined, "stop_and_retain"); + if (outcome === "inflight") { expect(call).not.toHaveBeenCalled(); return; } + if (["stop_failed", "unconfirmed"].includes(outcome)) { + expect(await environmentService(db).getLeaseById(lease.id)).toMatchObject({ status: "pending_cleanup", metadata: { pendingCleanupInFlight: false } }); + await heartbeatService(db, { environmentRuntime: runtime() }).sweepPendingCleanupLeases({ backoffMs: 0 }); + } + expect(await environmentService(db).getLeaseById(lease.id)).toMatchObject({ status: "released", cleanupStatus: "success", metadata: { remoteExecutionTermination: { state: "stopped" } } }); + expect(call.mock.calls.every(entry => entry[1] === "environmentStopLease")).toBe(true); + }); + it("does not allocate a native-runner replacement without a verified backup", async () => { const seeded = await seedReusablePluginSandboxLease("paperclip_runner"); const workerManager = { diff --git a/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts b/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts index 1564aa3a17..004ed36b8c 100644 --- a/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts +++ b/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts @@ -161,7 +161,10 @@ const mockIssueTreeControlService = vi.hoisted(() => ({ getActivePauseHoldGate: const mockLogActivity = vi.hoisted(() => vi.fn(async () => undefined)); const mockObserveCrossIssueInfluence = vi.hoisted(() => vi.fn(async () => null)); +const mockRetryWorkspaceExport = vi.hoisted(() => vi.fn()); + function registerRouteMocks() { + vi.doMock("../services/native-runtime/native-workspace-export-retry.js", () => ({ retryNativeWorkspaceExport: mockRetryWorkspaceExport })); vi.doMock("@paperclipai/shared/telemetry", () => ({ trackAgentTaskCompleted: vi.fn(), trackErrorHandlerCrash: vi.fn(), @@ -1993,6 +1996,43 @@ describe("agent issue mutation checkout ownership", () => { ); }); + it("queues board export-only recovery without calling provider wake", async () => { + mockAccessService.decide.mockResolvedValue({ allowed: true }); + mockRetryWorkspaceExport.mockResolvedValue({ runId: ownerRunId, status: "queued" }); + const res = await request(await createApp(boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/retry-workspace-export`) + .send({ actionId: recoveryActionId, runId: ownerRunId, repairNote: "Restored provider connectivity and preserved all saved files." }); + expect(res.status).toBe(202); + expect(mockRetryWorkspaceExport).toHaveBeenCalledWith(expect.objectContaining({ companyId, issueId, runId: ownerRunId, actionId: recoveryActionId, actorId: "board-user" })); + expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled(); + }); + + it.each(["agent", "viewer"])("rejects %s export-only retries before admission", async kind => { + mockAccessService.decide.mockResolvedValue({ allowed: false, explanation: "No runtime access" }); + const res = await request(await createApp(kind === "agent" ? ownerActor() : boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/retry-workspace-export`) + .send({ actionId: recoveryActionId, runId: ownerRunId, repairNote: "Restored provider connectivity and preserved all saved files." }); + expect(res.status).toBe(403); expect(mockRetryWorkspaceExport).not.toHaveBeenCalled(); + }); + + it.each(["todo", "done", "in_review"].flatMap(sourceIssueStatus => ["native_workspace_sync_out_unsafe_archive", "native_workspace_sync_out_retry_exhausted"].map(cause => ({ sourceIssueStatus, cause }))))("does not resolve accepted export recovery through an ordinary $sourceIssueStatus transition: $cause", async ({ sourceIssueStatus, cause }) => { + const sourceIssue = makeIssue({ status: "blocked", assigneeAgentId: ownerAgentId }); + mockIssueService.getById.mockResolvedValue(sourceIssue); + mockIssueService.update.mockImplementation(async (_id: string, patch: Record) => ({ ...sourceIssue, ...patch })); + mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({ + id: recoveryActionId, status: "active", kind: "active_run_watchdog", ownerType: "board", + ownerAgentId: null, returnOwnerAgentId: ownerAgentId, + cause, evidence: { runId: ownerRunId }, + }); + const res = await request(await createApp(boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/resolve`) + .send({ actionId: recoveryActionId, outcome: "restored", sourceIssueStatus }); + expect.soft(res.status).toBe(409); + expect.soft(res.body.details?.code).toBe(cause === "native_workspace_sync_out_unsafe_archive" ? "workspace_export_automatic_recovery" : "workspace_export_retry_required"); + expect.soft(mockIssueService.update).not.toHaveBeenCalled(); + expect.soft(mockHeartbeatService.wakeup).not.toHaveBeenCalled(); + }); + it.each(["active", "waiting", "completed", "unavailable", "endpoint_removed"])( "rejects restoring a %s chat task before changing its issue or recovery action", async (state) => { diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index e9f6ab8906..653c5e7965 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -1,3 +1,5 @@ +import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared"; +import { retryNativeWorkspaceExport } from "../services/native-runtime/native-workspace-export-retry.js"; import { queuedInteractionId, readQueuedInteractionResponse, hasQueuedInteractionResponse } from "../services/queued-interaction-response.js"; import { deliverConversationComments, isConversation } from "../services/agent-conversations.js"; import { issueRecoveryActionReadModel } from "../services/issue-recovery-actions.js"; @@ -78,6 +80,7 @@ import { createIssueSchema, resolveCreateIssueStatusDefault, resolveIssueRecoveryActionSchema, + retryWorkspaceExportSchema, runnerGoalActionRequestSchema, feedbackTargetTypeSchema, feedbackTraceStatusSchema, @@ -9109,6 +9112,18 @@ export function issueRoutes( }); }); + router.post("/issues/:id/recovery-actions/retry-workspace-export", validate(retryWorkspaceExportSchema), async (req, res) => { + assertBoard(req); + const issue = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Issue not found"); + if (!issue || !(await assertIssueReadAllowed(req, res, issue))) return; + const decision = await access.decide({ actor: req.actor, action: "runtime:manage", resource: { type: "company", companyId: issue.companyId } }); + if (!decision.allowed) throw forbidden(decision.explanation, authorizationDeniedDetails(decision)); + const receipt = await retryNativeWorkspaceExport({ db, companyId: issue.companyId, issueId: issue.id, + actionId: req.body.actionId, runId: req.body.runId, repairNote: req.body.repairNote, + actorId: getActorInfo(req).actorId, environmentRuntime }); + res.status(202).json(receipt); + }); + router.post( "/issues/:id/recovery-actions/resolve", validate(resolveIssueRecoveryActionSchema), @@ -9257,6 +9272,14 @@ export function issueRoutes( { source: "recovery_action_resolution" }, ); + if (outcome === "restored" && activeRecoveryAction.cause === "native_workspace_sync_out_unsafe_archive") { + throw conflict("Unsafe workspace export recovers automatically without another provider turn.", { code: "workspace_export_automatic_recovery" }); + } + + if (outcome === "restored" && isNativeWorkspaceExportRepairCause(activeRecoveryAction.cause)) { + throw conflict("Repair the retained sandbox, then use Retry workspace export to finish the accepted result without another provider turn.", { code: "workspace_export_retry_required" }); + } + // Retrying an exhausted disposition repair is an explicit retry of the // recorded owner, never permission to reopen a stopped/completed task or // silently retry a new assignee from an old notice. All admission gates diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index 439262a6fc..9cd65e6f6d 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -207,6 +207,7 @@ import { // Issue recovery and decomposition createAcceptedPlanDecompositionSchema, resolveIssueRecoveryActionSchema, + retryWorkspaceExportSchema, cancelIssueThreadInteractionSchema, // Secret provider configs and remote import createSecretProviderConfigSchema, @@ -9978,6 +9979,15 @@ registerCurrentRoute({ summary: "List issue recovery actions", }); +registerCurrentRoute({ + method: "post", + path: "/api/issues/{id}/recovery-actions/retry-workspace-export", + tags: ["issues"], + summary: "Retry only workspace export for a repaired accepted native result", + body: retryWorkspaceExportSchema, + responses: { 202: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound, 409: r.conflict }, +}); + registerCurrentRoute({ method: "post", path: "/api/issues/{id}/recovery-actions/resolve", diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts index cb484cc5ba..4105ec6642 100644 --- a/server/src/services/environment-runtime.ts +++ b/server/src/services/environment-runtime.ts @@ -1,7 +1,9 @@ +import { hasStopOnlyCleanup, prepareSandboxStopAndRetain, readStopOnlyCleanup, settleStopOnlyCleanup, stopOnlyCleanupKey } from "./sandbox-stop-and-retain.js"; import { readEnvironmentCreationCleanupError } from "@paperclipai/plugin-sdk"; import { remoteTerminationReceipt } from "./remote-execution-termination.js"; +import { hasNativeWorkspaceExportResume, releaseCompletedNativeWorkspaceExportRetention } from "./native-runtime/native-workspace-export-resume.js"; import { createHash, randomUUID } from "node:crypto"; -import { and, eq, inArray, sql } from "drizzle-orm"; +import { and, eq, inArray, ne, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { companySecrets, companySecretVersions, environmentLeases, heartbeatRuns } from "@paperclipai/db"; import type { @@ -478,6 +480,7 @@ export interface EnvironmentDriverAcquireInput { } export interface EnvironmentDriverReleaseInput { + resourceDisposition?: "stop_and_retain"; /** Explicit Stop may terminate in-flight setup rather than drain it. */ cancelActiveWork?: boolean; environment: Environment; @@ -1670,7 +1673,14 @@ function createSandboxEnvironmentDriver( return runParent !== undefined ? runWithRuntimeParent(runParent, call) : call(); } - async function resolveSandboxProviderPlugin(input: { provider: string }) { + async function resolveSandboxProviderPlugin(input: { provider: string; pluginId?: string }) { + if (input.pluginId) { + const pinned = await resolvePluginSandboxProviderDriverById({ db, pluginId: input.pluginId, driverKey: input.provider }); + if (!pinned) return { state: "missing" as const, resolved: null }; + if (pinned.plugin.status !== "ready") return { state: "not_ready" as const, resolved: pinned }; + if (!pluginWorkerManager?.isRunning(pinned.plugin.id)) return { state: "worker_unavailable" as const, resolved: pinned }; + return { state: "running" as const, resolved: pinned }; + } const running = await resolvePluginSandboxProviderDriverByKey({ db, driverKey: input.provider, @@ -2564,6 +2574,23 @@ function createSandboxEnvironmentDriver( }, async retryPendingSandboxTeardown(input) { + const exportResume = hasStopOnlyCleanup(input.lease); + const resumeIntent = readStopOnlyCleanup(input.lease); + const assertExportResumeOwnership = async () => { + if (!resumeIntent) throw new Error("Workspace export resume ownership is invalid."); + const [current] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, input.lease.id)).limit(1); + const [otherOwner] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and( + ne(environmentLeases.id, input.lease.id), eq(environmentLeases.provider, input.lease.provider!), + eq(environmentLeases.providerLeaseId, input.lease.providerLeaseId!), inArray(environmentLeases.status, ["active", "retained", "pending_cleanup"]), + )).limit(1); + if (!current || current.status !== "pending_cleanup" || current.companyId !== input.lease.companyId + || current.metadata?.pendingCleanupAttemptId !== input.lease.metadata?.pendingCleanupAttemptId + || readStopOnlyCleanup(current)?.requestId !== resumeIntent.requestId + || readStopOnlyCleanup(current)?.pluginId !== resumeIntent.pluginId || otherOwner) { + throw new Error("Workspace export resume ownership changed before cleanup."); + } + }; + if (exportResume) await assertExportResumeOwnership(); // Resolve the teardown from the immutable orphan lease row, not from the // current environment. The row keeps the provider, the provider lease id, // and the sandbox config in its metadata. A provider change re-points the @@ -2597,7 +2624,9 @@ function createSandboxEnvironmentDriver( `Sandbox provider "${recordedProvider}" needs a plugin worker manager for cleanup, but none is available.`, ); } - const pluginProvider = await resolveSandboxProviderPlugin({ provider: recordedProvider }); + const pinnedPluginId = exportResume ? readString(input.lease.metadata?.pluginId) : null; + if (exportResume && !pinnedPluginId) throw new Error("Workspace export cleanup has no recorded provider plugin."); + const pluginProvider = await resolveSandboxProviderPlugin({ provider: recordedProvider, ...(pinnedPluginId ? { pluginId: pinnedPluginId } : {}) }); if (pluginProvider.state !== "running") { throw new Error( `Sandbox provider plugin for "${recordedProvider}" is ${pluginProvider.state}, so the cleanup teardown cannot run yet.`, @@ -2614,6 +2643,23 @@ function createSandboxEnvironmentDriver( { issueId: input.lease.issueId, heartbeatRunId: input.lease.heartbeatRunId }, ); const workerConfig = stripSandboxProviderEnvelope(config as SandboxEnvironmentConfig); + if (exportResume) { + const pluginId = pluginProvider.resolved.plugin.id; + if (!pluginWorkerVerifiesLifecycleMethod(pluginId, "environmentStopLease")) { + throw new Error("Workspace export recovery requires verified stop-only cleanup."); + } + await assertExportResumeOwnership(); + const receipt = await runLeaseReleaseWithRunParent(input.lease.id, () => pluginWorkerManager.call(pluginId, "environmentStopLease", { + driverKey: recordedProvider, companyId: input.lease.companyId, + environmentId: input.lease.environmentId ?? "", issueId: input.lease.issueId, + config: workerConfig, providerLeaseId: input.lease.providerLeaseId, + leaseMetadata: input.lease.metadata ?? {}, cancelActiveWork: true, resourceDisposition: "stop_and_retain", + }, Math.min(resolvePluginSandboxRpcTimeoutMs(workerConfig) ?? 60_000, 60_000))); + if (remoteTerminationReceipt(input.lease, receipt)?.state !== "stopped") { + throw new Error("Workspace export recovery did not confirm the retained sandbox stopped."); + } + return receipt; + } const failedCreation = readEnvironmentCreationCleanupError({ data: { schema: "paperclip/environment-creation-cleanup/v1", cleanup: input.lease.metadata?.failedCreateCleanup, @@ -2655,6 +2701,18 @@ function createSandboxEnvironmentDriver( metadataConfig, { issueId: input.lease.issueId, heartbeatRunId: input.lease.heartbeatRunId }, ); + if (exportResume) { + const provider = getBuiltinSandboxProvider(recordedProvider); + if (resumeIntent?.pluginId !== null || !provider?.stopLease) { + throw new Error("Sandbox preservation requires verified built-in stop-only cleanup."); + } + await assertExportResumeOwnership(); + const receipt = await provider.stopLease({ config: cleanupConfig, providerLeaseId: input.lease.providerLeaseId }); + if (remoteTerminationReceipt(input.lease, receipt)?.state !== "stopped") { + throw new Error("Built-in sandbox stop did not confirm the exact retained allocation."); + } + return receipt; + } await destroySandboxProviderLease({ config: cleanupConfig, providerLeaseId: input.lease.providerLeaseId, @@ -2671,11 +2729,20 @@ function createSandboxEnvironmentDriver( // teardown runs, throws, and counts toward the cap. if (!recordedProvider) return true; // A built-in provider has no plugin worker, so it is always ready. - if (isBuiltinSandboxProvider(recordedProvider)) return true; + if (isBuiltinSandboxProvider(recordedProvider)) { + return !hasStopOnlyCleanup(input.lease) || typeof getBuiltinSandboxProvider(recordedProvider)?.stopLease === "function"; + } // No worker manager is a permanent condition here. Report ready, so the // teardown runs, throws its own "no worker manager" error, and counts // toward the cap. if (!pluginWorkerManager) return true; + if (hasStopOnlyCleanup(input.lease)) { + const pinnedPluginId = readString(input.lease.metadata?.pluginId); + if (!pinnedPluginId) return true; // Permanent invalid intent, never a by-key fallback. + const pinned = await resolvePluginSandboxProviderDriverById({ db, pluginId: pinnedPluginId, driverKey: recordedProvider }); + return Boolean(pinned?.plugin.status === "ready" && pluginWorkerManager.isRunning(pinned.plugin.id) + && pluginWorkerVerifiesLifecycleMethod(pinned.plugin.id, "environmentStopLease")); + } // Resolve the installed plugin without a wait. A plugin reload or a plugin // reinstall can remove the plugin row for a short window, so a missing // plugin is a transient condition, not a permanent one. Report not ready, @@ -2778,6 +2845,31 @@ function createSandboxEnvironmentDriver( }; }, + async resumeRunLease(input) { + const pluginId = readString(input.lease.metadata?.pluginId); + const providerKey = readString(input.lease.metadata?.provider); + if (!input.lease.metadata?.sandboxProviderPlugin || !pluginWorkerManager || !pluginId || !providerKey + || !input.lease.providerLeaseId || input.lease.environmentId !== input.environment.id + || !pluginWorkerVerifiesLifecycleMethod(pluginId, "environmentResumeLease") + || (hasNativeWorkspaceExportResume(input.lease) && !pluginWorkerVerifiesLifecycleMethod(pluginId, "environmentStopLease"))) { + throw new Error("The exact sandbox lease cannot be resumed by its verified provider."); + } + // This is a lifecycle resume, never acquisition: no replacement, host + // seed, lease mutation, or provider turn is allowed at this boundary. + const config = stripSandboxProviderEnvelope(await resolvePluginSandboxRuntimeConfig({ + environment: input.environment, lease: input.lease, provider: providerKey, + }) as SandboxEnvironmentConfig); + const resumed = await pluginWorkerManager.call(pluginId, "environmentResumeLease", { + driverKey: providerKey, companyId: input.lease.companyId, environmentId: input.environment.id, + issueId: input.lease.issueId, config, providerLeaseId: input.lease.providerLeaseId, + leaseMetadata: input.lease.metadata ?? undefined, + }, Math.min(resolvePluginSandboxRpcTimeoutMs(config) ?? 60_000, 60_000)); + if (resumed?.providerLeaseId !== input.lease.providerLeaseId) { + throw new Error("The provider did not confirm the exact retained sandbox. No replacement was acquired."); + } + return resumed; + }, + async execute(input) { // Plugin-backed sandbox providers: delegate command execution. if (input.lease.metadata?.sandboxProviderPlugin && pluginWorkerManager) { @@ -3137,6 +3229,9 @@ function createSandboxEnvironmentDriver( async function releasePluginBackedSandboxLease( input: EnvironmentDriverReleaseInput, ): Promise { + if (input.resourceDisposition === "stop_and_retain") { + throw new Error("Stop-only sandbox cleanup requires a durable stop intent."); + } const metadata = input.lease.metadata ?? {}; const pluginId = readString(metadata.pluginId); const providerKey = readString(metadata.provider); @@ -3848,7 +3943,8 @@ export function environmentRuntimeService( .where( and( eq(environmentLeases.heartbeatRunId, heartbeatRunId), - inArray(environmentLeases.status, ["active"]), + or(eq(environmentLeases.status, "active"), and(eq(environmentLeases.status, "pending_cleanup"), + sql`(${environmentLeases.metadata} ? 'nativeWorkspaceExportResume' or ${environmentLeases.metadata} ? 'sandboxStopAndRetain')`)), ), ); if (leaseRows.length === 0) { @@ -3865,9 +3961,45 @@ export function environmentRuntimeService( const environment = leaseRow.environmentId ? await environmentsSvc.getById(leaseRow.environmentId) : null; + let leaseSnapshot = toEnvironmentLeaseSnapshot(leaseRow); + if (hasNativeWorkspaceExportResume(leaseSnapshot) && providerResourceDisposition === "destroy") { + const completed = await releaseCompletedNativeWorkspaceExportRetention(db, leaseSnapshot); + if (completed) leaseSnapshot = toEnvironmentLeaseSnapshot(completed); + } + if (providerResourceDisposition === "stop_and_retain" && !hasStopOnlyCleanup(leaseSnapshot) + && getLeaseDriverKey(leaseSnapshot, environment) === "sandbox") { + const prepared = await prepareSandboxStopAndRetain(db, leaseSnapshot); + if (!prepared) continue; + leaseSnapshot = toEnvironmentLeaseSnapshot(prepared); + } + if (hasStopOnlyCleanup(leaseSnapshot)) { + const intentKey = stopOnlyCleanupKey(leaseSnapshot); + const attemptId = randomUUID(); + const [claimed] = await db.update(environmentLeases).set({ + status: "pending_cleanup", cleanupStatus: "failed", releasedAt: new Date(), + metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ pendingCleanupAttemptId: attemptId, + pendingCleanupInFlight: true, pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000 })}::jsonb`, + }).where(and(eq(environmentLeases.id, leaseRow.id), eq(environmentLeases.heartbeatRunId, heartbeatRunId), + inArray(environmentLeases.status, ["active", "pending_cleanup"]), + sql`coalesce(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') = 'false'`, + sql`${environmentLeases.metadata}->${intentKey} = ${JSON.stringify(leaseSnapshot.metadata?.[intentKey])}::jsonb`, + )).returning(); + if (!claimed) continue; + const snapshot = toEnvironmentLeaseSnapshot(claimed); + try { + const driver = requireDriverKey(getLeaseDriverKey(snapshot, environment)); + if (!driver.retryPendingSandboxTeardown) throw new Error("Workspace export recovery requires stop-only cleanup."); + const receipt = await driver.retryPendingSandboxTeardown({ environment, lease: snapshot }); + const lease = await settleStopOnlyCleanup(db, snapshot, { attemptId, receipt }); + if (lease && environment) released.push({ environment, lease: toEnvironmentLeaseSnapshot(lease), + leaseContext: { executionWorkspaceId: lease.executionWorkspaceId, executionWorkspaceMode: null } }); + } catch (error) { + await settleStopOnlyCleanup(db, snapshot, { attemptId }); + throw error; + } + continue; + } if (!environment) continue; - - const leaseSnapshot = toEnvironmentLeaseSnapshot(leaseRow); if ( providerResourceDisposition === "keep_running" && leaseSnapshot.leasePolicy === "reuse_by_environment" @@ -3924,6 +4056,7 @@ export function environmentRuntimeService( : driver ? await driver.releaseRunLease({ ...(cancelActiveWork ? { cancelActiveWork: true } : {}), + ...(providerResourceDisposition === "stop_and_retain" ? { resourceDisposition: "stop_and_retain" as const } : {}), environment, lease: leaseSnapshot, // A stopped reusable provider resource must remain eligible diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 91b1d36282..b08f639466 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,11 @@ +import { restoreNativeWorkspaceBestEffort } from "./native-runtime/native-workspace-best-effort.js"; +import { + withNativeWorkspaceFinalizationOwnership, + NativeWorkspaceFinalizationBusyError, + NativeWorkspaceFinalizationOwnershipLostError, + type NativeWorkspaceFinalizationOwnership, +} from "./native-runtime/native-workspace-finalization-ownership.js"; +import { hasStopOnlyCleanup, settleStopOnlyCleanup } from "./sandbox-stop-and-retain.js"; import { applyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-result"; import { hasWorkspaceRestoreFailure } from "@paperclipai/shared"; import { externalConversationStateSql, nonIdleSlackIssueCondition } from "./slack-conversation-state.js"; @@ -18573,7 +18581,7 @@ export function heartbeatService( // path uses the provider and configuration recorded on the lease first; // the environment is lifecycle context and only a legacy fallback. const isOrphanEphemeralLease = lease.leasePolicy === "ephemeral"; - const useRecordedTeardown = isOrphanEphemeralLease || !environment; + const useRecordedTeardown = isOrphanEphemeralLease || !environment || hasStopOnlyCleanup(lease); // Do not consume a finite cleanup attempt while the provider plugin is // briefly unavailable. A plugin worker restart, a plugin reload, or a @@ -18642,7 +18650,9 @@ export function heartbeatService( environment, lease, }); - const released = await environmentsSvc.releaseLease(lease.id, "expired", { + const released = hasStopOnlyCleanup(lease) + ? await settleStopOnlyCleanup(db, lease, { attemptId: claimed, receipt }) + : await environmentsSvc.releaseLease(lease.id, "expired", { expectedPendingCleanupAttemptId: claimed, cleanupStatus: "success", failureReason: "pending_cleanup_retry", @@ -18800,7 +18810,7 @@ export function heartbeatService( agentId: input.agentId, status: settledRun?.status, failureReason: settledRun?.error ?? undefined, - providerResourceDisposition: input.succeeded + providerResourceDisposition: input.succeeded && (!parseObject(settledRun?.resultJson).workspaceExportRetry || workspaceSyncReference?.resourceDisposition === "destroy") ? (workspaceSyncReference?.resourceDisposition ?? "stop_and_retain") : "stop_and_retain", }); @@ -24437,44 +24447,78 @@ export function heartbeatService( // If recording the barrier itself fails, propagate as a run failure // rather than silently leaving dependents stranded behind a missing // finalize row. - if (nativeWorkspaceSync) { - await nativeWorkspaceSync.restoreWorkspace(); - } - await db - .update(heartbeatRuns) - .set({ executionControlDeadlineAt: new Date(Date.now() + 60_000) }) - .where( - and( - eq(heartbeatRuns.id, run.id), - eq(heartbeatRuns.status, "running"), - ), - ); - const workspaceFinalizeStatus = hasWorkspaceRestoreFailure(adapterResult.resultJson) ? "failed" : "succeeded"; - await recordWorkspaceFinalize(workspaceFinalizeStatus); - if (adapterResult.nativeFinalization) { - adapterResult.nativeFinalization.workspaceFinalizeStatus = - workspaceFinalizeStatus; + const completeWorkspace = async (ownership?: NativeWorkspaceFinalizationOwnership) => { try { - const finalized = await finalizeNativeRun({ - db, - runId: run.id, - workspaceFinalizeStatus, - preserveProviderAttempt: Boolean(nativeWorkspaceSync), - }); - await dispatchPendingNativeStatusWakeups({ - companyId: run.companyId, - }); - if (finalized.phase === "committed") { - await nativeWorkspaceSync?.cleanup(); + if (nativeWorkspaceSync) { + const exported = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and( + eq(workspaceOperations.companyId, run.companyId), + eq(workspaceOperations.heartbeatRunId, run.id), + eq(workspaceOperations.phase, "workspace_finalize"), + eq(workspaceOperations.status, "succeeded"), + )).limit(1); + if (exported.length) adapterFinalizeOutcome = "succeeded"; + else await restoreNativeWorkspaceBestEffort({ + db, runId: run.id, assertOwnership: ownership?.assertHeld, + restore: () => nativeWorkspaceSync!.restoreWorkspace(ownership?.assertHeld), + }); } - } catch (finalizeErr) { - logger.warn( - { err: finalizeErr, runId: run.id }, - "native result persisted but finalization did not apply; the reconciliation loop will retry", - ); + await ownership?.assertHeld(); + await db + .update(heartbeatRuns) + .set({ executionControlDeadlineAt: new Date(Date.now() + 60_000) }) + .where( + and( + eq(heartbeatRuns.id, run.id), + eq(heartbeatRuns.status, "running"), + ), + ); + const workspaceFinalizeStatus = hasWorkspaceRestoreFailure(adapterResult.resultJson) ? "failed" : "succeeded"; + await recordWorkspaceFinalize(workspaceFinalizeStatus); + if (adapterResult.nativeFinalization) { + adapterResult.nativeFinalization.workspaceFinalizeStatus = + workspaceFinalizeStatus; + try { + const finalized = await finalizeNativeRun({ + db, + runId: run.id, + workspaceFinalizeStatus, + preserveProviderAttempt: Boolean(nativeWorkspaceSync), + }); + await dispatchPendingNativeStatusWakeups({ + companyId: run.companyId, + }); + if (finalized.phase === "committed") { + await nativeWorkspaceSync?.cleanup(); + } + } catch (finalizeErr) { + logger.warn( + { err: finalizeErr, runId: run.id }, + "native result persisted but finalization did not apply; the reconciliation loop will retry", + ); + } + } + } catch (error) { + if (ownership) { + await ownership.assertHeld(); + await recordWorkspaceFinalize("failed"); + } + throw error; } + }; + if (nativeWorkspaceSync) { + const owned = await withNativeWorkspaceFinalizationOwnership({ + db, companyId: run.companyId, runId: run.id, + }, completeWorkspace); + if (!owned.acquired) throw new NativeWorkspaceFinalizationBusyError(); + } else { + await completeWorkspace(); } } catch (adapterErr) { + if (adapterErr instanceof NativeWorkspaceFinalizationBusyError + || adapterErr instanceof NativeWorkspaceFinalizationOwnershipLostError) { + nativeWorkspaceFinalizeScheduled = true; + throw adapterErr; + } if (adapterErr instanceof NativeControllerDetachedForRestartError) { // Preserve the provider and its run for the new controller. This // also keeps generic teardown from terminalizing/releasing its lease. @@ -25414,6 +25458,14 @@ export function heartbeatService( } return; } + if (err instanceof NativeWorkspaceFinalizationBusyError + || err instanceof NativeWorkspaceFinalizationOwnershipLostError) { + // Another exact owner is finishing copyback, or this owner lost its + // lock connection. Preserve the accepted result and let reconciliation + // inspect durable ownership; neither case consumes an export retry. + logger.info({ runId: run.id, reason: err.message }, "native workspace finalization deferred to its durable owner"); + return; + } if (err instanceof NativeWorkspaceFinalizeScheduledError) { const coordinator = await db .select({ @@ -25429,7 +25481,9 @@ export function heartbeatService( stream: "system", level: err.terminalFailure ? "error" : "warn", message: err.terminalFailure - ? "native result is durable, but the sandbox containing unexported workspace changes is unrecoverable" + ? err.reasonCode === "workspace_sync_out_failed" + ? "native result is durable; automatic workspace copy-back retries stopped and saved work is retained for export repair" + : "native result is durable, but the sandbox containing unexported workspace changes is unrecoverable" : "native result is durable; workspace copy-back will retry without another provider turn", payload: { attempt: coordinator?.attempt ?? null, @@ -25441,13 +25495,13 @@ export function heartbeatService( if (err.terminalFailure) { // The durable coordinator already failed the run, blocked the // issue, and cleared its execution lock. Let ordinary teardown - // release the now-useless lease and return the agent to service. + // release the lease while retaining the sandbox and its unexported work. nativeWorkspaceFinalizeScheduled = false; providerResourceDispositionForRun = "stop_and_retain"; await finalizeAgentStatus( run.agentId, "failed", - "native_workspace_sync_out_unrecoverable", + `native_${err.reasonCode}`, { wasFirstHeartbeat: timerClaimWasFirstHeartbeat(run) }, ).catch(() => undefined); } diff --git a/server/src/services/native-runtime/native-finalization-reconciler.ts b/server/src/services/native-runtime/native-finalization-reconciler.ts index c88479166b..095bb6115d 100644 --- a/server/src/services/native-runtime/native-finalization-reconciler.ts +++ b/server/src/services/native-runtime/native-finalization-reconciler.ts @@ -1,3 +1,4 @@ +import { recoverLegacyUnsafeWorkspaceExports } from "./native-workspace-export-recovery.js"; import { dismissAutomaticCompletionReviews, decisionHasRetiredAutomaticReview } from "./automatic-completion-reviews.js"; import { logger } from "../../middleware/logger.js"; import { createHash, randomUUID } from "node:crypto"; @@ -546,6 +547,9 @@ export async function reconcileNativeFinalizations( }) => Promise; } = {}, ) { + await recoverLegacyUnsafeWorkspaceExports(db, runIds).catch((err) => { + logger.warn({ err }, "Historical unsafe export recovery remains pending"); + }); await dismissObsoleteNativePolicyReviews(db, runIds).catch((err) => { logger.warn({ err }, "Obsolete native policy review lookup failed; continuing native reconciliation"); }); @@ -855,6 +859,8 @@ export async function reconcileNativeFinalizations( runId: row.runId, environmentRuntime: options.environmentRuntime, }); + // Busy is ownership, not another failed export or retry-budget debit. + if (!operation) continue; const workspaceFinalizeStatus = operation.status === "succeeded" ? "succeeded" : "failed"; if (workspaceFinalizeStatus === "failed") { diff --git a/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts b/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts index aeaa9f0f35..b6fe6445e5 100644 --- a/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts +++ b/server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts @@ -7,10 +7,15 @@ import { completionContracts, createDb, heartbeatRuns, + heartbeatRunEvents, issues, nativeRunFinalizations, nativeRunResults, workAssessments, + issueRecoveryActions, + agentWakeupRequests, + workspaceOperations, + activityLog, } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, @@ -47,6 +52,9 @@ function captureRunFailureCallsFrom(fromIndex: number) { } import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js"; +import { restoreNativeWorkspaceBestEffort } from "./native-workspace-best-effort.js"; +import { reconcileNativeFinalizations } from "./native-finalization-reconciler.js"; +import { recoverLegacyUnsafeWorkspaceExports } from "./native-workspace-export-recovery.js"; import { finalizeNativeRun, recordNativeFinalizationFailure } from "./native-run-finalizer.js"; import { deliverExecutionStatuses } from "../execution-status-delivery.js"; import { commitNativeStatusDecision } from "./status-decision-committer.js"; @@ -155,6 +163,7 @@ describeEmbeddedPostgres("native run finalizer / status decision committer — a async function driveToCompleteResult( fixture: Awaited>, terminal: typeof CONTROL_PLANE_CONFORMANCE_TERMINAL = CONTROL_PLANE_CONFORMANCE_TERMINAL, + result = CONTROL_PLANE_CONFORMANCE_RESULT, ) { const port = newPort(fixture); await port.openRun({ @@ -169,7 +178,7 @@ describeEmbeddedPostgres("native run finalizer / status decision committer — a sourceInstanceId: fixture.runnerInstanceId, }); await port.completeRun({ - result: CONTROL_PLANE_CONFORMANCE_RESULT, + result, terminal, callerResultId: `${fixture.runId}:result`, }); @@ -508,6 +517,83 @@ describeEmbeddedPostgres("native run finalizer / status decision committer — a expect(replayed).toEqual(repaired); }); + it("retires this run's scheduled finalization retry after successful commit", async () => { + const fixture = await seedNativeRun(); + await driveToCompleteResult(fixture, CONTROL_PLANE_CONFORMANCE_TERMINAL); + await recordNativeFinalizationFailure({ db, runId: fixture.runId, + error: new Error("native_workspace_sync_out_failed"), failureScope: "workspace", projectRunStatus: true }); + const [pending] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId)); + expect(pending).toMatchObject({ status: "active", evidence: { runId: fixture.runId } }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + const [settled] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, pending.id)); + expect(settled.status).toBe("resolved"); + }); + + it("repairs stale retry metadata on an already committed successful run", async () => { + const fixture = await seedNativeRun(); + await driveToCompleteResult(fixture, CONTROL_PLANE_CONFORMANCE_TERMINAL); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + await db.update(heartbeatRuns).set({ resultJson: { + ...before.resultJson, finalizationPhase: "retryable_failure", failureCode: "native_finalization_invalid", + originalFailureCode: "native_finalization_invalid", nextAttemptAt: new Date().toISOString(), + } }).where(eq(heartbeatRuns.id, fixture.runId)); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + const [after] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(after).toMatchObject({ status: "succeeded", nativePhase: "committed", resultJson: { + finalizationPhase: "committed", failureCode: null, originalFailureCode: null, nextAttemptAt: null, + } }); + }); + + it("ignores a stale workspace failure after the same run committed successfully", async () => { + const fixture = await seedNativeRun(); + await driveToCompleteResult(fixture, CONTROL_PLANE_CONFORMANCE_TERMINAL); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const late = await recordNativeFinalizationFailure({ + db, runId: fixture.runId, error: new Error("native_workspace_sync_out_failed"), + failureScope: "workspace", projectRunStatus: true, + }); + expect(late.phase).toBe("committed"); + const [after] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(after).toEqual(before); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId))).toEqual([]); + }); + + it("materializes recovery for an agent-owned invalid-result outcome", async () => { + const fixture = await seedNativeRun(); + await db.insert(nativeRunFinalizations).values({ + runId: fixture.runId, companyId, issueId: fixture.issueId, + phase: "terminal_failure", failureCode: "native_finalization_invalid", + failureDetail: { recoveryOwner: { kind: "agent", agentId } }, + }); + const outcome = await recordNativeFinalizationFailure({ + db, runId: fixture.runId, error: new Error("native_finalization_invalid"), + }); + expect(outcome.phase).toBe("retryable_failure"); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId))) + .toEqual([expect.objectContaining({ status: "active", ownerType: "agent", cause: "native_finalization_invalid" })]); + }); + + it.each(["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_unsafe_archive"])( + "does not reopen board-owned %s repair for a late failure", async (failureCode) => { + const fixture = await seedNativeRun(); + await db.insert(nativeRunFinalizations).values({ + runId: fixture.runId, companyId, issueId: fixture.issueId, phase: "workspace_finalizing", + }); + await recordNativeFinalizationFailure({ db, runId: fixture.runId, + error: new Error(failureCode), failureScope: "workspace", permanent: true }); + const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)); + const recoveryBefore = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId)); + for (const failureScope of [undefined, "workspace"] as const) { + await recordNativeFinalizationFailure({ db, runId: fixture.runId, + error: new Error("native_workspace_sync_out_failed"), failureScope }); + } + const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)); + expect(after).toEqual(before); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId))).toEqual(recoveryBefore); + }); + it("emits zero events when a retryable-failure write's conditional status spread is omitted", async () => { const fixture = await seedNativeRun(); // recordNativeFinalizationFailure only needs the run and coordinator rows @@ -626,6 +712,80 @@ describeEmbeddedPostgres("native run finalizer / status decision committer — a ).resolves.toBe("blocked"); }); + it("commits the saved result when unsafe workspace export is omitted, with only a run log", async () => { + const fixture = await seedNativeRun(); + await db.update(completionContracts).set({ risk: "low", completionAuthority: "agent_claim_policy" }) + .where(eq(completionContracts.id, fixture.contractId)); + await driveToCompleteResult(fixture, CONTROL_PLANE_CONFORMANCE_TERMINAL, { + ...CONTROL_PLANE_CONFORMANCE_RESULT, + completionClaim: { ...CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim!, contractRevision: "telemetry-v1" }, + }); + const [original] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId)); + const restore = vi.fn(async () => { + throw new Error("Daytona syncOut refusing tarball link whose target escapes the extraction dir: tools/pnpm -> /private/tool"); + }); + const assertOwnership = vi.fn(async () => {}); + await restoreNativeWorkspaceBestEffort({ db, runId: fixture.runId, restore, assertOwnership }); + const outcome = await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true, preserveProviderAttempt: true }); + expect(outcome.phase).toBe("committed"); + expect(restore).toHaveBeenCalledOnce(); + expect(assertOwnership).toHaveBeenCalledOnce(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const [issue] = await db.select().from(issues).where(eq(issues.id, fixture.issueId)); + expect(run).toMatchObject({ status: "succeeded", error: null, errorCode: null }); + expect(issue.status).toBe("done"); + const results = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId)); + expect(results).toEqual([original]); + const actions = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId)); + expect(actions).toEqual([]); + const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, fixture.runId)); + expect(events.filter(event => event.eventType === "workspace_export_omitted")).toMatchObject([ + { level: "info", payload: { reason: "restore_unsafe_archive" } }, + ]); + expect(JSON.stringify(events)).not.toContain("/private/tool"); + }); + + it.each(["active", "resolved", "missing"])("automatically completes a legacy unsafe result with a %s repair action and no sandbox", async actionState => { + const fixture = await seedNativeRun(); + await db.update(completionContracts).set({ risk: "low", completionAuthority: "agent_claim_policy" }).where(eq(completionContracts.id, fixture.contractId)); + await driveToCompleteResult(fixture, CONTROL_PLANE_CONFORMANCE_TERMINAL, { + ...CONTROL_PLANE_CONFORMANCE_RESULT, + completionClaim: { ...CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim!, contractRevision: "telemetry-v1" }, + }); + const originalResults = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId)); + await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", failureCode: "native_workspace_sync_out_unsafe_archive" }).where(eq(nativeRunFinalizations.runId, fixture.runId)); + await db.update(heartbeatRuns).set({ status: "failed", nativePhase: "terminal_failure", errorCode: "native_workspace_sync_out_unsafe_archive", + runnerProfileJson: { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v1", state: "prepared", + descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null, + workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: "missing-old-sandbox", remoteCwd: "/work", resourceDisposition: "destroy" } }, + }).where(eq(heartbeatRuns.id, fixture.runId)); + await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, fixture.issueId)); + if (actionState !== "missing") await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: fixture.issueId, + kind: "active_run_watchdog", ownerType: "board", cause: "native_workspace_sync_out_unsafe_archive", fingerprint: fixture.runId, + evidence: { runId: fixture.runId }, nextAction: "Repair the unsafe link manually", status: actionState, + ...(actionState === "resolved" ? { outcome: "restored", resolutionNote: "new_source_execution_path", resolvedAt: new Date() } : {}), + }); + const priorWakeups = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId)); + const priorActivity = await db.select().from(activityLog).where(eq(activityLog.entityId, fixture.issueId)); + await recoverLegacyUnsafeWorkspaceExports(db, [fixture.runId]); + expect(await db.select().from(activityLog).where(eq(activityLog.entityId, fixture.issueId))).toEqual(priorActivity); + // No environment runtime is supplied: a stopped, deleted, or unavailable + // provider cannot prevent omission and accepted-result commitment. + await reconcileNativeFinalizations(db, [fixture.runId]); + await reconcileNativeFinalizations(db, [fixture.runId]); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)))[0]).toMatchObject({ status: "succeeded", nativePhase: "committed", error: null, errorCode: null }); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0].status).toBe("done"); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId))).toEqual(originalResults); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).toEqual(priorWakeups); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.nativeIssueId, fixture.issueId))).toHaveLength(1); + const actions = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId)); + expect(actions.every(action => action.status === "resolved" && action.nextAction === "")).toBe(true); + expect((await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, fixture.runId)))[0]).toMatchObject({ status: "succeeded", metadata: { workspaceSync: { omitted: true, legacy: true } } }); + const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, fixture.runId)); + expect(events.filter(event => event.eventType === "workspace_export_omitted")).toMatchObject([{ level: "info", payload: { reason: "restore_unsafe_archive", legacy: true } }]); + expect(events.filter(event => event.eventType === "workspace_export_omitted")).toHaveLength(1); + }); + it("emits exactly one event for a cancel_continuations write (trap 2: :685/:518 overlap)", async () => { const fixture = await seedNativeRun(); // Build the minimal real rows commitNativeStatusDecision's foreign keys diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index afef715688..1e61872f4b 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -3,6 +3,7 @@ import { dismissAutomaticCompletionReviews } from "./automatic-completion-review import { getNativeReviewAssignment, readNativeReviewAssignmentContext } from "./native-review-participant.js"; import { conversationNativeDecision, isConversation } from "../agent-conversations.js"; import { randomUUID } from "node:crypto"; +import { preserveNativeWorkspaceExportLease } from "./native-workspace-export-resume.js"; import { and, eq, inArray, isNotNull, isNull, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { @@ -12,12 +13,14 @@ import { heartbeatRuns, heartbeatRunEvents, issueApprovals, + issueRecoveryActions, issueThreadInteractions, issues, nativeRunFinalizations, nativeRunResults, statusDecisions, workAssessments, + workspaceOperations, } from "@paperclipai/db"; import { classifyNativeEvidence } from "./evidence-classifier.js"; import type { PrpIgnoredAttentionRequest } from "@paperclipai/paperclip-runner"; @@ -282,6 +285,39 @@ async function recordRetryableFailure(input: { : ("failed" as const); let terminalRunToEmit: typeof heartbeatRuns.$inferSelect | null = null; const outcome = await input.db.transaction(async (tx) => { + // Admission snapshots cannot authorize a failure write after a different + // owner committed success. Match status-committer lock ordering. + const current = await tx.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.runId, input.run.id), + eq(nativeRunFinalizations.companyId, input.run.companyId), + )).for("update").limit(1).then((rows) => rows[0] ?? null); + if (!current) throw new Error("native_finalization_missing"); + // Audit-only attention first records an agent-owned invalid-result outcome; + // its caller still needs to materialize the normal bounded recovery action. + // Board-owned terminal repairs and late snapshots remain settled. + const currentExportRetry = record(record(current.failureDetail).workspaceExportRetry).requestId; + if (currentExportRetry && currentExportRetry !== record(record(input.coordinator.failureDetail).workspaceExportRetry).requestId) return current; + const recoveryOwner = record(record(current.failureDetail).recoveryOwner); + const pendingAgentRecovery = current.phase === "terminal_failure" + && input.coordinator.phase === "terminal_failure" + && recoveryOwner.kind === "agent" && recoveryOwner.agentId === input.run.agentId; + if (current.phase === "committed" + || (current.phase === "terminal_failure" && !pendingAgentRecovery) + || current.leaseOwner !== input.coordinator.leaseOwner) return current; + const [currentRun] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.run.id)).limit(1); + if (!currentRun) throw new Error("native_finalization_run_missing"); + if (input.failureScope === "workspace") { + const exported = await tx.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and( + eq(workspaceOperations.companyId, input.run.companyId), + eq(workspaceOperations.heartbeatRunId, input.run.id), + eq(workspaceOperations.phase, "workspace_finalize"), + eq(workspaceOperations.status, "succeeded"), + )).limit(1); + // The live exporter may have acquired ownership since the stale error + // was raised, or already durably published a successful copyback. + if (exported.length || record(currentRun.runnerProfileJson).nativeWorkspaceFinalizationOwner) return current; + } + input = { ...input, coordinator: current, run: currentRun }; const issue = await tx .select({ lastStatusDecisionId: issues.lastStatusDecisionId, @@ -372,6 +408,7 @@ async function recordRetryableFailure(input: { failureDetail: { message: input.message.slice(0, 2_000), originalFailureCode: input.failureCode, + ...(priorFailureDetail.workspaceExportRetry ? { workspaceExportRetry: priorFailureDetail.workspaceExportRetry } : {}), ...(workspaceFinalizeAttempt === null ? {} : { workspaceFinalizeAttempt }), @@ -388,6 +425,9 @@ async function recordRetryableFailure(input: { .where(eq(nativeRunFinalizations.runId, input.run.id)); const projectsTerminalStatus = exhausted && !supersededByNewerRun && input.projectRunStatus; + if (projectsTerminalStatus && input.failureScope === "workspace") { + await preserveNativeWorkspaceExportLease(tx as unknown as Db, input.run, input.coordinator.resultId); + } const [updatedRun] = await tx .update(heartbeatRuns) .set({ @@ -442,7 +482,7 @@ async function recordRetryableFailure(input: { input.coordinator.issueId, { status: - input.permanent && input.failureScope === "workspace" + input.failureScope === "workspace" ? "blocked" : "in_review", }, @@ -525,6 +565,7 @@ export async function recordNativeFinalizationFailure(input: { if (!run || run.runtimeMode !== "native" || !coordinator) throw input.error; const message = input.error instanceof Error ? input.error.message : String(input.error); + if (message === "native_finalization_lease_busy") return coordinator; const failureCode = message.startsWith("native_") ? message : "native_finalization_invalid"; @@ -546,6 +587,19 @@ export async function recordNativeFinalizationFailure(input: { }); } +async function resolveCommittedFinalizationRecovery(db: Db, run: typeof heartbeatRuns.$inferSelect, issueId: string) { + const actions = await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issueId), + eq(issueRecoveryActions.kind, "active_run_watchdog"), inArray(issueRecoveryActions.status, ["active", "escalated"]), + sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, + sql`${issueRecoveryActions.wakePolicy}->>'kind' = 'resume_native_run'`, + )); + for (const action of actions) await issueRecoveryActionService(db).resolveActiveForIssue({ + companyId: run.companyId, sourceIssueId: issueId, actionId: action.id, + status: "resolved", outcome: "restored", resolutionNote: "The accepted native result and workspace finalization committed successfully; this run needs no further finalization retry.", + }); +} + async function projectCommittedRun(input: { db: Db; run: typeof heartbeatRuns.$inferSelect; @@ -599,10 +653,10 @@ async function projectCommittedRun(input: { 'observedAt', ${heartbeatRuns.updatedAt} ) ) - else ${heartbeatRuns.resultJson} - end`, + else coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) + end || jsonb_build_object('finalizationPhase', 'committed', 'failureCode', null, 'originalFailureCode', null, 'nextAttemptAt', null)`, } - : {}), + : { resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || jsonb_build_object('finalizationPhase', 'committed', 'failureCode', null, 'originalFailureCode', null, 'nextAttemptAt', null)` }), updatedAt: now, }) .where( @@ -618,6 +672,10 @@ async function projectCommittedRun(input: { isNotNull(heartbeatRuns.errorCode), isNull(heartbeatRuns.finishedAt), sql`${heartbeatRuns.nativePhase} is distinct from 'committed'`, + sql`${heartbeatRuns.resultJson}->>'finalizationPhase' is distinct from 'committed'`, + sql`${heartbeatRuns.resultJson}->>'nextAttemptAt' is not null`, + sql`${heartbeatRuns.resultJson}->>'failureCode' is not null`, + sql`${heartbeatRuns.resultJson}->>'originalFailureCode' is not null`, ), ), ), @@ -629,6 +687,10 @@ async function projectCommittedRun(input: { sql`${heartbeatRuns.status} is distinct from ${projectedStatus}`, isNull(heartbeatRuns.finishedAt), sql`${heartbeatRuns.nativePhase} is distinct from 'committed'`, + sql`${heartbeatRuns.resultJson}->>'finalizationPhase' is distinct from 'committed'`, + sql`${heartbeatRuns.resultJson}->>'nextAttemptAt' is not null`, + sql`${heartbeatRuns.resultJson}->>'failureCode' is not null`, + sql`${heartbeatRuns.resultJson}->>'originalFailureCode' is not null`, ...(terminalState === "succeeded" ? [isNotNull(heartbeatRuns.error), isNotNull(heartbeatRuns.errorCode)] : []), @@ -1045,6 +1107,7 @@ export async function finalizeNativeRun(input: { logger.warn({ err, runId: run.id }, "Slack conversation settlement deferred to reconciliation"); }); } + await resolveCommittedFinalizationRecovery(input.db, run, coordinator.issueId); return coordinator; } const [resultRow, contractRow] = await Promise.all([ @@ -1319,6 +1382,7 @@ export async function finalizeNativeRun(input: { resultJson: { ...record(run.resultJson), finalizationPhase, + ...(finalizationPhase === "committed" ? { failureCode: null, originalFailureCode: null, nextAttemptAt: null } : {}), assessmentId: assessmentRow.id, decisionId: committed.decision.id, authoritativeDecision: decision.toStatus, @@ -1374,6 +1438,7 @@ export async function finalizeNativeRun(input: { logger.warn({ err, runId: run.id }, "Slack conversation settlement deferred to reconciliation"); }); } + if (finalizationPhase === "committed") await resolveCommittedFinalizationRecovery(input.db, run, coordinator.issueId); return { ...coordinator, phase: finalizationPhase, diff --git a/server/src/services/native-runtime/native-workspace-best-effort.test.ts b/server/src/services/native-runtime/native-workspace-best-effort.test.ts new file mode 100644 index 0000000000..f067117c09 --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-best-effort.test.ts @@ -0,0 +1,55 @@ +import { describe, expect, it, vi, beforeEach } from "vitest"; +import type { Db } from "@paperclipai/db"; +const log = vi.hoisted(() => vi.fn()); +vi.mock("../heartbeat-run-events.js", () => ({ appendHeartbeatRunEvent: log })); +import { restoreNativeWorkspaceBestEffort } from "./native-workspace-best-effort.js"; + +function fixture() { + const limit = vi.fn(async () => [{ companyId: "company", agentId: "agent" }]); + const db = { select: vi.fn(() => ({ from: () => ({ where: () => ({ limit }) }) })) }; + return { db: db as unknown as Db, select: db.select, limit }; +} + +describe("best effort native workspace export", () => { + beforeEach(() => log.mockReset()); + + it("preserves the restore result on success", async () => { + const { db, select } = fixture(); + await expect(restoreNativeWorkspaceBestEffort({ db, runId: "run", restore: async () => true })).resolves.toBe(true); + expect(select).not.toHaveBeenCalled(); + expect(log).not.toHaveBeenCalled(); + }); + + it.each([ + new Error("Daytona syncOut refusing tarball member that escapes the extraction dir: ../private"), + new Error("Daytona outbound symlink-escape guard command failed (exit 42): private"), + Object.assign(new Error("private detail"), { code: "WORKSPACE_RESTORE_UNSAFE_ARCHIVE" }), + ])("omits unsafe exports with only a redacted info event: %s", async (error) => { + const { db } = fixture(); + await expect(restoreNativeWorkspaceBestEffort({ db, runId: "run", restore: async () => { throw error; } })).resolves.toBeUndefined(); + expect(log).toHaveBeenCalledExactlyOnceWith(db, expect.objectContaining({ + companyId: "company", agentId: "agent", runId: "run", level: "info", eventType: "workspace_export_omitted", + })); + expect(JSON.stringify(log.mock.calls[0][1])).not.toContain("private"); + }); + + it("does not hide transport failures or ownership loss", async () => { + const { db, select } = fixture(); + const restore = async () => { throw new Error("transport unavailable"); }; + await expect(restoreNativeWorkspaceBestEffort({ db, runId: "run", restore })).rejects.toThrow("transport unavailable"); + await expect(restoreNativeWorkspaceBestEffort({ db, runId: "run", + restore: async () => { throw Object.assign(new Error("unsafe"), { code: "WORKSPACE_RESTORE_UNSAFE_ARCHIVE" }); }, + assertOwnership: async () => { throw new Error("ownership lost"); }, + })).rejects.toThrow("ownership lost"); + expect(select).not.toHaveBeenCalled(); + expect(log).not.toHaveBeenCalled(); + }); + + it("does not fail the task when the informational log is unavailable", async () => { + const { db } = fixture(); + log.mockRejectedValueOnce(new Error("log unavailable")); + await expect(restoreNativeWorkspaceBestEffort({ db, runId: "run", + restore: async () => { throw Object.assign(new Error("unsafe"), { code: "WORKSPACE_RESTORE_UNSAFE_ARCHIVE" }); }, + })).resolves.toBeUndefined(); + }); +}); diff --git a/server/src/services/native-runtime/native-workspace-best-effort.ts b/server/src/services/native-runtime/native-workspace-best-effort.ts new file mode 100644 index 0000000000..95df6df6c1 --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-best-effort.ts @@ -0,0 +1,41 @@ +import { eq } from "drizzle-orm"; +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { classifyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-merge"; +import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; +import { logger } from "../../middleware/logger.js"; + +/** An unsafe export cannot invalidate an already accepted agent result. */ +export async function restoreNativeWorkspaceBestEffort(input: { + db: Db; + runId: string; + restore: () => Promise; + assertOwnership?: () => Promise; +}): Promise { + try { + return await input.restore(); + } catch (error) { + // Losing finalization ownership must never open the workspace barrier. + await input.assertOwnership?.(); + if (classifyWorkspaceRestoreFailure(error) !== "restore_unsafe_archive") throw error; + // The provider salvages confined archive entries when possible. A source + // that itself escapes the workspace cannot be exported at all. Both cases + // intentionally allow lost remote files: unsafe persistence must not fail + // an accepted task result. This diagnostic stays in run logs, never the task. + try { + const [run] = await input.db.select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId }) + .from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)).limit(1); + if (run) await appendHeartbeatRunEvent(input.db, { + ...run, + runId: input.runId, + eventType: "workspace_export_omitted", + stream: "system", + level: "info", + message: "Unsafe workspace export omitted; continuing with the accepted result.", + payload: { reason: "restore_unsafe_archive" }, + }); + } catch { + logger.info({ runId: input.runId }, "Unsafe workspace export omitted; run diagnostic could not be persisted"); + } + return undefined; + } +} diff --git a/server/src/services/native-runtime/native-workspace-export-recovery.ts b/server/src/services/native-runtime/native-workspace-export-recovery.ts new file mode 100644 index 0000000000..1c97e2ea2e --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-export-recovery.ts @@ -0,0 +1,111 @@ +import { and, asc, desc, eq, gt, inArray, ne, or, sql } from "drizzle-orm"; +import { completionContracts, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, workspaceOperations, type Db } from "@paperclipai/db"; +import { publishLiveEvent } from "../live-events.js"; +import { buildHeartbeatRunStatusLiveEventPayload } from "../heartbeat-run-status-payload.js"; +import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; +import { withNativeWorkspaceFinalizationOwnership } from "./native-workspace-finalization-ownership.js"; + +const UNSAFE_EXPORT = "native_workspace_sync_out_unsafe_archive"; +const scanCursors = new WeakMap(); + +/** Older controllers terminalized unsafe archives. Omit that already-rejected + * export and replay only the accepted result through the normal status arbiter. + * No provider access is necessary, even if the old sandbox no longer exists. */ +export async function recoverLegacyUnsafeWorkspaceExports(db: Db, runIds?: string[]) { + const selectCandidates = (cursor?: string) => db.select({ runId: nativeRunFinalizations.runId, companyId: nativeRunFinalizations.companyId, issueId: nativeRunFinalizations.issueId }) + .from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.phase, "terminal_failure"), eq(nativeRunFinalizations.failureCode, UNSAFE_EXPORT), + ...(runIds?.length ? [inArray(nativeRunFinalizations.runId, runIds)] : []), + ...(cursor ? [gt(nativeRunFinalizations.runId, cursor)] : []), + )).orderBy(asc(nativeRunFinalizations.runId)).limit(25); + let candidates = await selectCandidates(runIds?.length ? undefined : scanCursors.get(db)); + if (!candidates.length && !runIds?.length && scanCursors.has(db)) { + scanCursors.delete(db); + candidates = await selectCandidates(); + } + for (const candidate of candidates) { + if (!runIds?.length) scanCursors.set(db, candidate.runId); + const owned = await withNativeWorkspaceFinalizationOwnership({ db, ...candidate }, async ownership => { + await ownership.assertHeld(); + return db.transaction(async tx => { + const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, candidate.companyId), eq(issues.id, candidate.issueId))).for("update").limit(1); + const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, candidate.companyId), eq(nativeRunFinalizations.runId, candidate.runId), + )).for("update").limit(1); + if (!coordinator || coordinator.phase !== "terminal_failure" || coordinator.failureCode !== UNSAFE_EXPORT + || !coordinator.resultId || coordinator.leaseOwner || coordinator.issueId !== issue?.id) return null; + const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, candidate.companyId), eq(heartbeatRuns.id, candidate.runId))).for("update").limit(1); + if (!issue || !run || run.runtimeMode !== "native" || run.nativeIssueId !== issue.id) return null; + const [result] = await tx.select().from(nativeRunResults).where(and( + eq(nativeRunResults.id, coordinator.resultId), eq(nativeRunResults.companyId, run.companyId), + eq(nativeRunResults.runId, run.id), eq(nativeRunResults.issueId, issue.id), + eq(nativeRunResults.completionContractId, run.completionContractId ?? "00000000-0000-0000-0000-000000000000"), + eq(nativeRunResults.schemaStatus, "accepted"), + )).limit(1); + if (!result) return null; + const priorActions = await tx.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issue.id), + eq(issueRecoveryActions.cause, UNSAFE_EXPORT), sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, + )); + const explicitlySettled = priorActions.some(action => action.status === "cancelled" + || (action.status === "resolved" && !["new_source_execution_path", "unsafe_export_automatic_recovery"].includes(action.resolutionNote ?? ""))); + const now = new Date(); + // A stale unsafe notice is never an instruction to repair or start a + // provider turn, including when newer work prevents replay of this run. + await tx.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: now, + resolutionNote: "unsafe_export_automatic_recovery", nextAction: "", wakePolicy: null, updatedAt: now, + }).where(and(eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issue.id), + eq(issueRecoveryActions.cause, UNSAFE_EXPORT), sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, + or(inArray(issueRecoveryActions.status, ["active", "escalated"]), + and(eq(issueRecoveryActions.status, "resolved"), eq(issueRecoveryActions.resolutionNote, "new_source_execution_path"))), + )); + const [newerRun] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, run.companyId), ne(heartbeatRuns.id, run.id), + or(eq(heartbeatRuns.nativeIssueId, issue.id), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`), + or(gt(heartbeatRuns.createdAt, run.createdAt), inArray(heartbeatRuns.status, ["queued", "running"])), + )).limit(1); + const [otherAction] = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issue.id), + inArray(issueRecoveryActions.status, ["active", "escalated"]), + )).limit(1); + const [latestContract] = await tx.select({ id: completionContracts.id }).from(completionContracts).where(and( + eq(completionContracts.companyId, run.companyId), eq(completionContracts.issueId, issue.id), + )).orderBy(desc(completionContracts.revision)).limit(1); + const eligible = !explicitlySettled && !newerRun && !otherAction && latestContract?.id === result.completionContractId + && ["blocked", "in_review"].includes(issue.status) && issue.assigneeAgentId === run.agentId + && ["failed", "succeeded"].includes(run.status) && run.nativePhase === "terminal_failure" + && (!issue.executionRunId || issue.executionRunId === run.id) + && (!issue.checkoutRunId || issue.checkoutRunId === run.id); + let updatedRun: typeof heartbeatRuns.$inferSelect | undefined; + if (eligible) { + await ownership.assertHeld(); + // Publish the omission barrier atomically with re-admission. This is + // explicitly not a receipt claiming that files were copied to the host. + await tx.insert(workspaceOperations).values({ companyId: run.companyId, heartbeatRunId: run.id, issueId: issue.id, + phase: "workspace_finalize", status: "succeeded", exitCode: 0, finishedAt: now, + metadata: { owningService: "native_workspace_finalizer", workspaceSync: { omitted: true, reason: "restore_unsafe_archive", legacy: true } }, + }); + await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null, + failureDetail: { workspaceFinalizeAttempt: 0, legacyUnsafeExportOmitted: { resultId: result.id, recoveredAt: now.toISOString() } }, updatedAt: now, + }).where(eq(nativeRunFinalizations.runId, run.id)); + [updatedRun] = await tx.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null, + nativePhase: "result_accepted", nativePhaseUpdatedAt: now, + resultJson: { ...run.resultJson, finalizationPhase: "result_accepted", failureCode: null, originalFailureCode: null, nextAttemptAt: null }, updatedAt: now, + }).where(eq(heartbeatRuns.id, run.id)).returning(); + await tx.update(issues).set({ executionRunId: run.id, updatedAt: now }).where(eq(issues.id, issue.id)); + await appendHeartbeatRunEvent(tx as unknown as Db, { companyId: run.companyId, agentId: run.agentId, runId: run.id, + eventType: "workspace_export_omitted", stream: "system", level: "info", + message: "Historical unsafe workspace export omitted; continuing with the accepted result.", + payload: { reason: "restore_unsafe_archive", legacy: true }, + }); + } + // Normal finalization publishes task status. Unsafe-export diagnostics + // belong only in run logs, not activity history or task notifications. + return updatedRun ?? null; + }); + }); + if (owned.acquired && owned.value) publishLiveEvent({ companyId: owned.value.companyId, + type: "heartbeat.run.status", payload: buildHeartbeatRunStatusLiveEventPayload(owned.value), + }); + } +} diff --git a/server/src/services/native-runtime/native-workspace-export-resume.live.test.ts b/server/src/services/native-runtime/native-workspace-export-resume.live.test.ts new file mode 100644 index 0000000000..69348c40b4 --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-export-resume.live.test.ts @@ -0,0 +1,221 @@ +// Opt-in provider-boundary fault integration, not a Product E2E workflow. +// Real Daytona acquire/resume/stop/execute + production retry/reaper services; +// only the probe transport and first stop reply are deliberately unavailable. +import { randomUUID, createHash } from "node:crypto"; +import { writeFile } from "node:fs/promises"; +import { createRequire } from "node:module"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { eq } from "drizzle-orm"; +import { agents, companies, completionContracts, createDb, environmentLeases, environments, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, plugins } from "@paperclipai/db"; +import type { Environment, EnvironmentLease } from "@paperclipai/shared"; +import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { environmentRuntimeService } from "../environment-runtime.js"; +import type { PluginWorkerManager } from "../plugin-worker-manager.js"; +import { heartbeatService } from "../heartbeat.js"; +import { remoteTerminationReceipt } from "../remote-execution-termination.js"; +import { retryNativeWorkspaceExport } from "./native-workspace-export-retry.js"; +import { recordNativeFinalizationFailure } from "./native-run-finalizer.js"; +import { classifyNativeWorkspaceFailure } from "./native-workspace-failure.js"; + +const enabled = process.env.PAPERCLIP_LIVE_EXPORT_RESUME === "1"; +const describeLive = enabled ? describe : describe.skip; +const image = process.env.PAPERCLIP_LIVE_EXPORT_RESUME_IMAGE ?? ""; +const quote = (s: string) => `'${s.replaceAll("'", "'\\''")}'`; + +describeLive("live Daytona export-resume failure recovery", () => { + let temporary: Awaited>; + let db: ReturnType; + let plugin: any; + let manifest: any; + let sandbox: any; + let lease: EnvironmentLease; + let environment: Environment; + const ids = { company: randomUUID(), agent: randomUUID(), environment: randomUUID(), run: randomUUID(), issue: randomUUID(), + result: randomUUID(), contract: randomUUID(), lease: randomUUID(), action: randomUUID(), plugin: randomUUID() }; + const nonce = randomUUID(); + const bytes = `preserved-export-resume-${nonce}\n`; + const digest = createHash("sha256").update(bytes).digest("hex"); + const config = { provider: "daytona", image, reuseLease: false, timeoutMs: 120_000, autoStopInterval: 10, autoDeleteInterval: 0 }; + const evidence: Record = { kind: "live_provider_boundary_fault_integration", image, nonceSha256: digest }; + const methods: string[] = []; + const providerCalls: unknown[] = []; + evidence.providerCalls = providerCalls; + let failProbe = false; + let failStop = false; + const handlers: Record = { environmentResumeLease: "onEnvironmentResumeLease", environmentStopLease: "onEnvironmentStopLease", environmentExecute: "onEnvironmentExecute" }; + const manager = () => ({ isRunning: () => true, + getWorker: () => ({ supportedMethods: Object.keys(handlers) }), + call: async (_id: string, method: string, input: any) => { + methods.push(method); + if (method === "environmentExecute" && failProbe) { failProbe = false; evidence.probeFaultsInjected = Number(evidence.probeFaultsInjected ?? 0) + 1; throw new Error("Injected probe transport failure"); } + if (method === "environmentStopLease" && failStop) { failStop = false; throw new Error("Injected stop transport failure"); } + if (!handlers[method]) throw new Error("Unexpected provider operation"); + try { + const result = await plugin[handlers[method]](input); + providerCalls.push({ method, ...(method === "environmentExecute" ? { exitCode: result.exitCode, timedOut: result.timedOut } : {}), + ...(method === "environmentResumeLease" ? { exactAllocation: result.providerLeaseId === lease.providerLeaseId, exactRoot: result.metadata?.remoteCwd === lease.metadata?.remoteCwd } : {}) }); + return result; + } catch (error) { + evidence.providerFailure = { method, errorName: (error as Error).name, + // Function locations diagnose fixture failures without copying provider messages or credentials. + frames: (error as Error).stack?.split("\n").slice(1, 6) }; + throw new Error("Live Daytona operation failed; provider details withheld"); + } + }, + }) as unknown as PluginWorkerManager; + function runtime() { + const service = environmentRuntimeService(db, { pluginWorkerManager: manager() }); + const execute = service.execute.bind(service); + service.execute = async input => { + try { return await execute(input); } + catch (error) { + evidence.executionFailure = { errorName: (error as Error).name, frames: (error as Error).stack?.split("\n").slice(1, 6) }; + throw error; + } + }; + return service; + } + const scope = () => ({ driverKey: "daytona", companyId: ids.company, environmentId: ids.environment, issueId: ids.issue, + config, providerLeaseId: lease.providerLeaseId, leaseMetadata: lease.metadata }); + async function persistedLease() { + return (await db.select().from(environmentLeases).where(eq(environmentLeases.id, ids.lease)))[0]; + } + async function confirmStoppedAndPreserved() { + await sandbox.refreshData(); + expect(sandbox.state).toBe("stopped"); + expect(sandbox.autoDeleteInterval).toBe(-1); + evidence.providerAutoDeleteDisabled = true; + await sandbox.start(60); + const result = await sandbox.process.executeCommand(`sha256sum ${quote(`${lease.metadata!.remoteCwd}/preserved-${nonce}.txt`)}`, "/", undefined, 15); + expect(result.exitCode).toBe(0); + expect(result.result.split(/\s/)[0]).toBe(digest); + await sandbox.stop(60); + await sandbox.refreshData(); + expect(sandbox.state).toBe("stopped"); + } + beforeAll(async () => { + if (!process.env.DAYTONA_API_KEY || !/@sha256:[a-f0-9]{64}$/.test(image)) throw new Error("Explicit Daytona credentials and immutable image required"); + temporary = await startEmbeddedPostgresTestDatabase("live-export-resume-"); db = createDb(temporary.connectionString); + ({ default: plugin } = await import(new URL("../../../../packages/plugins/sandbox-providers/daytona/dist/plugin.js", import.meta.url).href)); + plugin = plugin.definition; + ({ default: manifest } = await import(new URL("../../../../packages/plugins/sandbox-providers/daytona/dist/manifest.js", import.meta.url).href)); + await db.insert(companies).values({ id: ids.company, name: "Disposable export lifecycle", issuePrefix: "LXR" }); + await db.insert(agents).values({ id: ids.agent, companyId: ids.company, name: "No provider turn", adapterType: "paperclip_runner" }); + await db.insert(environments).values({ id: ids.environment, name: `Lifecycle ${nonce}`, driver: "sandbox", config }); + environment = (await db.select().from(environments).where(eq(environments.id, ids.environment)))[0] as unknown as Environment; + await db.insert(plugins).values({ id: ids.plugin, pluginKey: manifest.id, packageName: "@paperclipai/plugin-daytona", version: manifest.version, + apiVersion: 1, categories: ["automation"], manifestJson: manifest, status: "ready", installOrder: 1 }); + await db.insert(issues).values({ id: ids.issue, companyId: ids.company, title: "Preserved accepted result", status: "blocked", assigneeAgentId: ids.agent }); + await db.insert(completionContracts).values({ id: ids.contract, companyId: ids.company, issueId: ids.issue, revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "live-test", risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: { objective: "Preserve saved work" }, canonicalSha256: digest, createdByActorType: "system", createdByActorId: "live-test" }); + await db.insert(heartbeatRuns).values({ id: ids.run, companyId: ids.company, agentId: ids.agent, status: "failed", runtimeMode: "native", nativeIssueId: ids.issue, nativePhase: "terminal_failure", completionContractId: ids.contract }); + await db.insert(nativeRunResults).values({ id: ids.result, companyId: ids.company, issueId: ids.issue, runId: ids.run, completionContractId: ids.contract, serverFingerprint: digest, schemaStatus: "accepted", resultJson: { work: "preserve saved work" }, canonicalSha256: digest }); + await db.insert(nativeRunFinalizations).values({ runId: ids.run, companyId: ids.company, issueId: ids.issue, phase: "terminal_failure", resultId: ids.result, failureCode: "native_workspace_sync_out_retry_exhausted" }); + await db.insert(issueRecoveryActions).values({ id: ids.action, companyId: ids.company, sourceIssueId: ids.issue, kind: "active_run_watchdog", ownerType: "board", returnOwnerAgentId: ids.agent, cause: "native_workspace_sync_out_retry_exhausted", fingerprint: ids.run, evidence: { runId: ids.run }, nextAction: "Repair saved files" }); + let setupPhase = "acquire"; + try { + const acquired = await plugin.onEnvironmentAcquireLease({ driverKey: "daytona", companyId: ids.company, environmentId: ids.environment, + issueId: ids.issue, runId: ids.run, agentId: ids.agent, adapterType: "paperclip_runner", config }); + setupPhase = "sdk_lookup"; + const require = createRequire(new URL("../../../../packages/plugins/sandbox-providers/daytona/package.json", import.meta.url)); + const { Daytona } = require("@daytonaio/sdk"); + sandbox = await new Daytona({ apiKey: process.env.DAYTONA_API_KEY }).get(acquired.providerLeaseId); + const metadata = { ...acquired.metadata, provider: "daytona", pluginId: ids.plugin, sandboxProviderPlugin: true }; + await db.insert(environmentLeases).values({ id: ids.lease, companyId: ids.company, environmentId: ids.environment, issueId: ids.issue, + heartbeatRunId: ids.run, status: "active", leasePolicy: "ephemeral", provider: "daytona", providerLeaseId: acquired.providerLeaseId, metadata }); + lease = await persistedLease() as unknown as EnvironmentLease; + setupPhase = "write"; + const wrote = await sandbox.process.executeCommand(`printf %s ${quote(bytes)} > ${quote(`${metadata.remoteCwd}/preserved-${nonce}.txt`)}`, "/", undefined, 15); + if (wrote.exitCode !== 0) throw new Error("Write failed"); + setupPhase = "initial_stop"; + const receipt = await plugin.onEnvironmentStopLease(scope()); + expect(receipt.state).toBe("stopped"); + await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(), + metadata: { ...metadata, remoteExecutionTermination: remoteTerminationReceipt(lease, receipt) } }).where(eq(environmentLeases.id, ids.lease)); + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v1", state: "prepared", descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId: ids.lease, providerLeaseId: lease.providerLeaseId, remoteCwd: metadata.remoteCwd, resourceDisposition: "destroy" } } }).where(eq(heartbeatRuns.id, ids.run)); + } catch (error) { + evidence.setupFailure = { phase: setupPhase, errorType: (error as Error).name }; + throw new Error(`Live fixture setup failed at ${setupPhase}; provider details withheld`); + } + }, 180_000); + afterAll(async () => { + try { + if (sandbox) { + if (sandbox.labels?.["paperclip-company-id"] !== ids.company || sandbox.labels?.["paperclip-environment-id"] !== ids.environment || sandbox.labels?.["paperclip-run-id"] !== ids.run) throw new Error("Fixture cleanup ownership mismatch"); + await sandbox.delete(60); + evidence.cleanupPassed = true; + } + } catch { throw new Error("Exact live fixture cleanup failed; provider details withheld"); } + finally { + if (temporary) await temporary.cleanup(); + if (process.env.PAPERCLIP_EXPORT_RESUME_EVIDENCE_PATH) await writeFile(process.env.PAPERCLIP_EXPORT_RESUME_EVIDENCE_PATH, JSON.stringify(evidence, null, 2), { mode: 0o600 }); + } + }, 90_000); + it("stops after a failed probe, and recovers a failed stop through a fresh runtime", async () => { + const request = () => ({ db, companyId: ids.company, issueId: ids.issue, actionId: ids.action, runId: ids.run, actorId: "live-test", + repairNote: "Provider-boundary fault integration preserves exact synthetic nonce bytes", environmentRuntime: runtime() }); + failProbe = true; + await expect(retryNativeWorkspaceExport(request())).rejects.toThrow("Resume and repair"); + expect(await persistedLease()).toMatchObject({ status: "released", cleanupStatus: "success", metadata: { remoteExecutionTermination: { state: "stopped" } } }); + await confirmStoppedAndPreserved(); + evidence.failedProbeCompensated = true; + failProbe = true; failStop = true; + await expect(retryNativeWorkspaceExport(request())).rejects.toThrow("Resume and repair"); + const pending = await persistedLease(); + expect(pending).toMatchObject({ status: "pending_cleanup", cleanupStatus: "failed", metadata: { pendingCleanupInFlight: false, nativeWorkspaceExportResume: { runId: ids.run, leaseId: ids.lease } } }); + expect(pending.metadata?.remoteExecutionTermination).toBeUndefined(); + await sandbox.refreshData(); expect(sandbox.state).toBe("started"); + evidence.failedStopDurablyTracked = true; + // Load a fresh worker module as well as a fresh controller service. The + // reaper must recover from durable rows with no cached provider handle. + const restartedPlugin = (await import(`${new URL("../../../../packages/plugins/sandbox-providers/daytona/dist/plugin.js", import.meta.url).href}?restart=${nonce}`)).default.definition; + expect(restartedPlugin).not.toBe(plugin); + plugin = restartedPlugin; + const fresh = runtime(); + await heartbeatService(db, { environmentRuntime: fresh }).sweepPendingCleanupLeases({ backoffMs: 0 }); + expect(await persistedLease()).toMatchObject({ status: "released", cleanupStatus: "success", metadata: { remoteExecutionTermination: { state: "stopped" } } }); + expect((await persistedLease()).metadata?.nativeWorkspaceExportResume).toBeUndefined(); + await confirmStoppedAndPreserved(); + expect(await db.select().from(agentWakeupRequests)).toHaveLength(0); + expect(await db.select().from(heartbeatRuns)).toHaveLength(1); + expect((await db.select().from(nativeRunFinalizations))[0]).toMatchObject({ phase: "terminal_failure", resultId: ids.result }); + expect(methods.filter(m => m === "environmentResumeLease")).toHaveLength(2); + expect(methods.filter(m => m === "environmentStopLease")).toHaveLength(3); + expect(evidence.probeFaultsInjected).toBe(2); + evidence.restartStopPreservedBytes = true; evidence.noNewProviderTurn = true; evidence.noDestroyBeforeFixtureCleanup = true; + }, 180_000); + it("preserves an ephemeral allocation after injected transient export exhaustion and a failed initial stop", async () => { + await plugin.onEnvironmentResumeLease(scope()); + await db.update(environmentLeases).set({ status: "active", cleanupStatus: null, releasedAt: null, + metadata: { ...(await persistedLease()).metadata, remoteExecutionTermination: undefined } }).where(eq(environmentLeases.id, ids.lease)); + await db.update(heartbeatRuns).set({ status: "running", nativePhase: "result_accepted" }).where(eq(heartbeatRuns.id, ids.run)); + await db.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, failureDetail: null }).where(eq(nativeRunFinalizations.runId, ids.run)); + const acceptedBefore = await db.select().from(nativeRunResults); + // Deliberate fixture-only transport failure at the finalizer boundary. No + // disk pressure, provider outage, or successful physical copyback is claimed. + for (let attempt = 0; attempt < 3; attempt++) { + const failure = classifyNativeWorkspaceFailure(new Error("Injected fixture export transport unavailable")); + await recordNativeFinalizationFailure({ db, runId: ids.run, error: new Error(failure.failureCode), + failureScope: "workspace", projectRunStatus: true, permanent: failure.permanent }); + } + expect(await persistedLease()).toMatchObject({ status: "pending_cleanup", leasePolicy: "ephemeral", metadata: { reuseLease: false, nativeWorkspaceExportResume: { purpose: "terminal_export", resultId: ids.result } } }); + failStop = true; + await runtime().releaseRunLeases(ids.run, "failed", undefined, "stop_and_retain"); + expect(await persistedLease()).toMatchObject({ status: "pending_cleanup", metadata: { pendingCleanupInFlight: false } }); + await sandbox.refreshData(); expect(sandbox.state).toBe("started"); + plugin = (await import(`${new URL("../../../../packages/plugins/sandbox-providers/daytona/dist/plugin.js", import.meta.url).href}?exhaustion-restart=${nonce}`)).default.definition; + await heartbeatService(db, { environmentRuntime: runtime() }).sweepPendingCleanupLeases({ backoffMs: 0 }); + expect(await persistedLease()).toMatchObject({ status: "released", metadata: { remoteExecutionTermination: { providerLeaseId: lease.providerLeaseId, state: "stopped" } } }); + await confirmStoppedAndPreserved(); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.cause, "native_workspace_sync_out_retry_exhausted")); + expect(action).toMatchObject({ status: "active", ownerType: "board" }); + const queued = await retryNativeWorkspaceExport({ db, companyId: ids.company, issueId: ids.issue, runId: ids.run, actionId: action.id, + actorId: "live-test", repairNote: "Fixture-only export transport fault removed; exact nonce and allocation preserved", environmentRuntime: runtime() }); + expect(queued).toMatchObject({ resultId: ids.result, leaseId: ids.lease, status: "queued" }); + expect(await db.select().from(nativeRunResults)).toEqual(acceptedBefore); + expect(await db.select().from(agentWakeupRequests)).toHaveLength(0); + expect(await db.select().from(heartbeatRuns)).toHaveLength(1); + evidence.ephemeralExhaustionRestartPreserved = true; + evidence.genericExportRetryAdmittedWithoutProviderTurn = true; + evidence.genericCopybackNotAttempted = true; + }, 180_000); +}); diff --git a/server/src/services/native-runtime/native-workspace-export-resume.ts b/server/src/services/native-runtime/native-workspace-export-resume.ts new file mode 100644 index 0000000000..94b3d46c07 --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-export-resume.ts @@ -0,0 +1,125 @@ +import { randomUUID } from "node:crypto"; +import { and, eq, inArray, ne, sql } from "drizzle-orm"; +import { environmentLeases, heartbeatRuns, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { remoteTerminationReceipt } from "../remote-execution-termination.js"; +import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js"; + +type Lease = { + id: string; companyId: string; heartbeatRunId: string | null; + provider: string | null; providerLeaseId: string | null; + metadata: Record | null; +}; + +export const NATIVE_WORKSPACE_EXPORT_RESUME_KEY = "nativeWorkspaceExportResume"; + +export function hasNativeWorkspaceExportResume(lease: Pick): boolean { + return Object.prototype.hasOwnProperty.call(lease.metadata ?? {}, NATIVE_WORKSPACE_EXPORT_RESUME_KEY); +} + +/** This intent is written before resuming a retained sandbox or terminalizing an + * accepted result with unexported work. Recovery may stop + * this exact allocation, but must never fall through to destructive cleanup. */ +export function readNativeWorkspaceExportResume(lease: Lease) { + const value = lease.metadata?.[NATIVE_WORKSPACE_EXPORT_RESUME_KEY]; + if (!value || typeof value !== "object" || Array.isArray(value)) return null; + const marker = value as Record; + if (!["paperclip.workspace-export-resume.v1", "paperclip.workspace-export-resume.v2"].includes(String(marker.schema)) + || marker.companyId !== lease.companyId || marker.runId !== lease.heartbeatRunId || !lease.heartbeatRunId + || marker.leaseId !== lease.id || marker.provider !== lease.provider || !lease.provider + || marker.providerLeaseId !== lease.providerLeaseId || !lease.providerLeaseId + || typeof marker.requestId !== "string" || !marker.requestId + || typeof marker.resultId !== "string" || !marker.resultId) return null; + // v1 preceded the intent's explicit plugin pin. Its exact lease already + // recorded the acquiring plugin; never reconstruct that owner from a driver + // name, and never override an explicit (even invalid) intent pin. + const pluginId = marker.schema === "paperclip.workspace-export-resume.v1" && marker.pluginId === undefined + ? lease.metadata?.pluginId : marker.pluginId; + if (typeof pluginId !== "string" || !pluginId || pluginId !== lease.metadata?.pluginId) return null; + return { ...marker, requestId: marker.requestId, resultId: marker.resultId, pluginId }; +} + +/** Called inside the finalizer's transaction, before its terminal failure is + * visible to orphan cleanup. A crash cannot expose an untagged failed lease. */ +export async function preserveNativeWorkspaceExportLease(db: Db, run: typeof heartbeatRuns.$inferSelect, resultId: string | null) { + const reference = readNativeWorkspaceSyncReference(run.runnerProfileJson?.nativeWorkspaceSync); + if (!reference || reference.state !== "prepared" || !resultId) return; + const [accepted] = await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and( + eq(nativeRunResults.id, resultId), eq(nativeRunResults.companyId, run.companyId), + eq(nativeRunResults.runId, run.id), eq(nativeRunResults.completionContractId, run.completionContractId!), + eq(nativeRunResults.schemaStatus, "accepted"), + )).limit(1); + if (!accepted) return; + const [lease] = await db.select().from(environmentLeases).where(and( + eq(environmentLeases.id, reference.leaseId), eq(environmentLeases.companyId, run.companyId), + eq(environmentLeases.heartbeatRunId, run.id), eq(environmentLeases.providerLeaseId, reference.providerLeaseId), + )).for("update").limit(1); + if (!lease || lease.status !== "active") return; + const requestId = randomUUID(), now = new Date(); + await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", + failureReason: "workspace_export_stop_pending", releasedAt: now, updatedAt: now, + metadata: { ...lease.metadata, remoteExecutionTermination: undefined, + [NATIVE_WORKSPACE_EXPORT_RESUME_KEY]: { schema: "paperclip.workspace-export-resume.v2", requestId, + purpose: "terminal_export", companyId: run.companyId, runId: run.id, resultId, + leaseId: lease.id, provider: lease.provider, providerLeaseId: lease.providerLeaseId, + pluginId: lease.metadata?.pluginId }, + pendingCleanupAttemptId: requestId, pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0, + pendingCleanupRetryAfterMs: 0, pendingCleanupRetryAttempts: 0, pendingCleanupRetryCapWarned: false }, + }).where(eq(environmentLeases.id, lease.id)); +} + +/** Only verified copyback plus commitment permits the original ephemeral + * destroy policy. A terminal failure or merely accepted result is insufficient. */ +export async function releaseCompletedNativeWorkspaceExportRetention(db: Db, lease: Lease) { + const marker = readNativeWorkspaceExportResume(lease); + if (!marker) return null; + const [bound] = await db.select({ run: heartbeatRuns, coordinator: nativeRunFinalizations }).from(heartbeatRuns) + .innerJoin(nativeRunFinalizations, and(eq(nativeRunFinalizations.runId, heartbeatRuns.id), eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId))) + .where(and(eq(heartbeatRuns.id, lease.heartbeatRunId!), eq(heartbeatRuns.companyId, lease.companyId), + inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled", "timed_out", "interrupted"]), + eq(heartbeatRuns.nativePhase, "committed"), + eq(nativeRunFinalizations.phase, "committed"), eq(nativeRunFinalizations.resultId, marker.resultId))).limit(1); + const reference = readNativeWorkspaceSyncReference(bound?.run.runnerProfileJson?.nativeWorkspaceSync); + if (!reference || reference.state !== "finalized" || !reference.finalHostSha256 + || reference.resourceDisposition !== "destroy" || reference.leaseId !== lease.id || reference.providerLeaseId !== lease.providerLeaseId) return null; + const [otherOwner] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and( + ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider!), + eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), inArray(environmentLeases.status, ["active", "retained", "pending_cleanup"]), + )).limit(1); + if (otherOwner) return null; + const [released] = await db.update(environmentLeases).set({ + metadata: sql`${environmentLeases.metadata} - 'nativeWorkspaceExportResume'`, + }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId), + eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), + eq(environmentLeases.provider, lease.provider!), sql`${environmentLeases.metadata}->>'pluginId' = ${marker.pluginId}`, + eq(environmentLeases.status, "active"), + sql`${environmentLeases.metadata}->'nativeWorkspaceExportResume'->>'requestId' = ${marker.requestId}`, + )).returning(); + return released ?? null; +} + +/** A late cleanup receipt cannot rewrite a rebound lease or a newer attempt. */ +export async function settleNativeWorkspaceExportResume(db: Db, lease: Lease, options: { + attemptId: string; receipt?: unknown; status?: "released" | "expired"; +}) { + const marker = readNativeWorkspaceExportResume(lease); + if (!marker) return null; + const receipt = remoteTerminationReceipt(lease, options.receipt); + const stopped = receipt?.state === "stopped"; + const now = new Date(); + const [row] = await db.update(environmentLeases).set({ + status: stopped ? options.status ?? "released" : "pending_cleanup", cleanupStatus: stopped ? "success" : "failed", + releasedAt: now, lastUsedAt: now, updatedAt: now, + metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination' + ${stopped ? sql`- 'nativeWorkspaceExportResume'` : sql``}) || ${JSON.stringify({ + ...(stopped ? { remoteExecutionTermination: receipt } : {}), + pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0, + })}::jsonb`, + }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId), + eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.provider, lease.provider!), + eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), eq(environmentLeases.status, "pending_cleanup"), + sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.attemptId}`, + sql`${environmentLeases.metadata}->'nativeWorkspaceExportResume'->>'requestId' = ${marker.requestId}`, + sql`${environmentLeases.metadata}->>'pluginId' = ${marker.pluginId}`, + )).returning(); + return row ?? null; +} diff --git a/server/src/services/native-runtime/native-workspace-export-retry.test.ts b/server/src/services/native-runtime/native-workspace-export-retry.test.ts new file mode 100644 index 0000000000..d8e664fba0 --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-export-retry.test.ts @@ -0,0 +1,291 @@ +import { randomUUID } from "node:crypto"; +import { beforeAll, afterAll, describe, expect, it, vi } from "vitest"; +import { eq } from "drizzle-orm"; +import { agents, companies, completionContracts, createDb, environmentLeases, environments, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, activityLog } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { remoteTerminationReceipt } from "../remote-execution-termination.js"; +import { withNativeWorkspaceFinalizationOwnership } from "./native-workspace-finalization-ownership.js"; +const probe = vi.hoisted(() => vi.fn()); +vi.mock("../environment-execution-target.js", () => ({ resolveEnvironmentExecutionTarget: async () => ({ kind: "remote", transport: "sandbox", remoteCwd: "/work", runner: { execute: probe } }) })); +import { recoverLegacyUnsafeWorkspaceExports } from "./native-workspace-export-recovery.js"; +import { recordNativeFinalizationFailure } from "./native-run-finalizer.js"; +import { recoveryService } from "../recovery/service.js"; +import { retryNativeWorkspaceExport } from "./native-workspace-export-retry.js"; +import { releaseCompletedNativeWorkspaceExportRetention } from "./native-workspace-export-resume.js"; + +describe("board retry of accepted workspace export", () => { + let temporary: Awaited>; + let db: ReturnType; + const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(); + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("workspace-export-retry-"); db = createDb(temporary.connectionString); + await db.insert(companies).values({ id: companyId, name: "Export repair", issuePrefix: "EXP" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Exporter", adapterType: "paperclip_runner" }); + await db.insert(environments).values({ id: environmentId, name: `Retained ${environmentId}`, driver: "sandbox" }); + }, 30_000); + afterAll(async () => { await temporary.cleanup(); }); + async function seed(cause = "native_workspace_sync_out_retry_exhausted") { + probe.mockReset().mockResolvedValue({ exitCode: 0, timedOut: false }); + const issueId = randomUUID(), runId = randomUUID(), resultId = randomUUID(), contractId = randomUUID(), leaseId = randomUUID(), actionId = randomUUID(), providerLeaseId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Preserve accepted work", status: "blocked", assigneeAgentId: agentId }); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: { objective: "Preserve accepted work" }, canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "failed", runtimeMode: "native", nativeIssueId: issueId, nativePhase: "terminal_failure", completionContractId: contractId, + runnerProfileJson: { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v1", state: "prepared", descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId, providerLeaseId, remoteCwd: "/work", resourceDisposition: "keep_running" } } }); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId, completionContractId: contractId, serverFingerprint: resultId, schemaStatus: "accepted", resultJson: { work: "already finished" }, canonicalSha256: resultId }); + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "terminal_failure", resultId, failureCode: cause, failureDetail: { workspaceFinalizeAttempt: 1 } }); + const identity = { id: leaseId, companyId, heartbeatRunId: runId, provider: "daytona", providerLeaseId }; + await db.insert(environmentLeases).values({ ...identity, environmentId, issueId, status: "released", releasedAt: new Date(), cleanupStatus: "success", leasePolicy: "reuse_by_environment", metadata: { pluginId: randomUUID(), remoteExecutionTermination: remoteTerminationReceipt(identity, { providerLeaseId, state: "stopped" }) } }); + await db.insert(issueRecoveryActions).values({ id: actionId, companyId, sourceIssueId: issueId, kind: "active_run_watchdog", ownerType: "board", returnOwnerAgentId: agentId, cause, fingerprint: runId, evidence: { runId }, nextAction: "Repair saved files" }); + const request = { db, companyId, issueId, actionId, runId, actorId: "board", repairNote: "Restored provider transport availability and preserved the saved work.", environmentRuntime: { + resumeRunLease: vi.fn().mockResolvedValue({ providerLeaseId, metadata: { remoteCwd: "/work" } }), + retryPendingSandboxTeardown: vi.fn().mockResolvedValue({ providerLeaseId, state: "stopped" }), + } as never }; + return { ...request, request, resultId, leaseId }; + } + it("rejects manual unsafe-export admission without contacting the provider", async () => { + const f = await seed("native_workspace_sync_out_unsafe_archive"); + await expect(retryNativeWorkspaceExport(f.request)).rejects.toThrow("no longer current"); + expect(probe).not.toHaveBeenCalled(); + expect((f.request.environmentRuntime as { resumeRunLease: ReturnType }).resumeRunLease).not.toHaveBeenCalled(); + }); + it("queues only the existing result and lease, audits once, and deduplicates a pending click", async () => { + const f = await seed(); + const accepted = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId)); + expect(await retryNativeWorkspaceExport(f.request)).toMatchObject({ runId: f.runId, resultId: f.resultId, leaseId: f.leaseId, status: "queued" }); + await retryNativeWorkspaceExport(f.request); + expect(probe).toHaveBeenCalledOnce(); + expect(probe).toHaveBeenCalledWith(expect.objectContaining({ args: ["-c", "true"], bypassSession: true })); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toEqual(accepted); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).toHaveLength(0); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.nativeIssueId, f.issueId))).toHaveLength(1); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toMatchObject({ phase: "result_accepted", resultId: f.resultId, nextAttemptAt: null }); + expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)))[0]).toMatchObject({ status: "active", releasedAt: null }); + expect(await db.select().from(activityLog).where(eq(activityLog.entityId, f.issueId))).toHaveLength(1); + }); + it("reopens the stopped provider lifecycle before probing its exact sandbox", async () => { + const f = await seed(); let providerAdmissionOpen = false; + const resume = (f.request.environmentRuntime as { resumeRunLease: ReturnType }).resumeRunLease; + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + resume.mockImplementation(async () => { providerAdmissionOpen = true; return { providerLeaseId: lease.providerLeaseId, metadata: { remoteCwd: "/work" } }; }); + probe.mockImplementation(async () => { if (!providerAdmissionOpen) throw new Error("Released provider admission is closed"); return { exitCode: 0, timedOut: false }; }); + await expect(retryNativeWorkspaceExport(f.request)).resolves.toMatchObject({ status: "queued" }); + expect(resume).toHaveBeenCalledOnce(); + expect(resume).toHaveBeenCalledWith(expect.objectContaining({ lease: expect.objectContaining({ id: f.leaseId, providerLeaseId: lease.providerLeaseId }) })); + }); + it("retries exhausted transient export of an ephemeral allocation without another provider turn", async () => { + const f = await seed(); + await db.update(nativeRunFinalizations).set({ failureCode: "native_workspace_sync_out_retry_exhausted", failureDetail: { workspaceFinalizeAttempt: 3 } }).where(eq(nativeRunFinalizations.runId, f.runId)); + await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted" }).where(eq(issueRecoveryActions.id, f.actionId)); + await db.update(environmentLeases).set({ leasePolicy: "ephemeral" }).where(eq(environmentLeases.id, f.leaseId)); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson, nativeWorkspaceSync: { ...(run.runnerProfileJson!.nativeWorkspaceSync as object), resourceDisposition: "destroy" } } }).where(eq(heartbeatRuns.id, f.runId)); + await expect(retryNativeWorkspaceExport(f.request)).resolves.toMatchObject({ status: "queued", resultId: f.resultId, leaseId: f.leaseId }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).toHaveLength(0); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.nativeIssueId, f.issueId))).toHaveLength(1); + }); + it.each(["unsafe", "transient"])("records stop-only preservation atomically with %s terminal export failure", async kind => { + const f = await seed(); + await db.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, failureDetail: null }).where(eq(nativeRunFinalizations.runId, f.runId)); + await db.update(heartbeatRuns).set({ status: "running", nativePhase: "result_accepted" }).where(eq(heartbeatRuns.id, f.runId)); + await db.update(environmentLeases).set({ status: "active", leasePolicy: "ephemeral", releasedAt: null, metadata: { pluginId: "plugin-test", pendingCleanupRetryAfterMs: Date.now() + 600_000, pendingCleanupRetryAttempts: 7 } }).where(eq(environmentLeases.id, f.leaseId)); + for (let attempt = 1; attempt <= (kind === "unsafe" ? 1 : 3); attempt++) { + await recordNativeFinalizationFailure({ db, runId: f.runId, + error: new Error(kind === "unsafe" ? "native_workspace_sync_out_unsafe_archive" : "native_workspace_sync_out_failed"), + failureScope: "workspace", projectRunStatus: true, permanent: kind === "unsafe" }); + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + if (kind === "transient" && attempt < 3) expect(lease.status).toBe("active"); + else { + expect(lease).toMatchObject({ status: "pending_cleanup", metadata: { + nativeWorkspaceExportResume: { runId: f.runId, leaseId: f.leaseId, resultId: f.resultId }, + pendingCleanupInFlight: false, pendingCleanupRetryAfterMs: 0, pendingCleanupRetryAttempts: 0, + } }); + expect(lease.metadata?.remoteExecutionTermination).toBeUndefined(); + expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("blocked"); + } + } + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, f.resultId)))[0].schemaStatus).toBe("accepted"); + }); + it.each(["complete", "committed_failed", "committed_cancelled", "running", "unexported", "wrong_result", "uncommitted_run", "wrong_allocation", "competing_owner"])("releases ephemeral retention only after exact committed copyback: %s", async kind => { + const f = await seed(); + await retryNativeWorkspaceExport(f.request); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + await db.update(heartbeatRuns).set({ status: kind === "running" ? "running" : kind === "committed_cancelled" ? "cancelled" : ["uncommitted_run", "committed_failed"].includes(kind) ? "failed" : "succeeded", + nativePhase: kind === "uncommitted_run" ? "terminal_failure" : "committed", + runnerProfileJson: { ...run.runnerProfileJson, nativeWorkspaceSync: { ...(run.runnerProfileJson!.nativeWorkspaceSync as object), + state: kind === "unexported" ? "prepared" : "finalized", finalHostSha256: "c".repeat(64), resourceDisposition: "destroy", + ...(kind === "wrong_allocation" ? { providerLeaseId: randomUUID() } : {}) } }, + }).where(eq(heartbeatRuns.id, f.runId)); + await db.update(nativeRunFinalizations).set({ phase: "committed", ...(kind === "wrong_result" ? { resultId: null } : {}) }).where(eq(nativeRunFinalizations.runId, f.runId)); + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + if (kind === "competing_owner") await db.insert(environmentLeases).values({ companyId, environmentId, status: "active", provider: lease.provider, providerLeaseId: lease.providerLeaseId }); + const released = await releaseCompletedNativeWorkspaceExportRetention(db, lease); + const committed = ["complete", "committed_failed", "committed_cancelled"].includes(kind); + if (committed) expect(released).toMatchObject({ id: lease.id, metadata: expect.not.objectContaining({ nativeWorkspaceExportResume: expect.anything() }) }); + else expect(released).toBeNull(); + const [after] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + expect(Boolean(after.metadata?.nativeWorkspaceExportResume)).toBe(!committed); + }); + it("records a recoverable stop-only intent before the provider can resume", async () => { + const f = await seed(); + const runtime = f.request.environmentRuntime as { resumeRunLease: ReturnType; retryPendingSandboxTeardown: ReturnType }; + runtime.resumeRunLease.mockImplementation(async () => { + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + expect(lease).toMatchObject({ status: "pending_cleanup", cleanupStatus: "failed", metadata: { + nativeWorkspaceExportResume: { runId: f.runId, leaseId: f.leaseId, resultId: f.resultId }, + pendingCleanupInFlight: true, + } }); + expect(lease.metadata?.remoteExecutionTermination).toBeUndefined(); + return { providerLeaseId: lease.providerLeaseId, metadata: { remoteCwd: "/work" } }; + }); + await expect(retryNativeWorkspaceExport(f.request)).resolves.toMatchObject({ status: "queued" }); + expect(runtime.retryPendingSandboxTeardown).not.toHaveBeenCalled(); + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + expect(lease.status).toBe("active"); + expect(lease.metadata?.nativeWorkspaceExportResume).toMatchObject({ runId: f.runId, resultId: f.resultId }); + }); + it.each(["resume_reply_lost", "probe_failed", "admission_changed", "cancelled", "completed", "stop_failed"])("tracks and compensates a failed export resume: %s", async kind => { + const f = await seed(); + const runtime = f.request.environmentRuntime as { resumeRunLease: ReturnType; retryPendingSandboxTeardown: ReturnType }; + if (kind === "resume_reply_lost") runtime.resumeRunLease.mockRejectedValueOnce(new Error("reply lost after provider resumed")); + else probe.mockImplementationOnce(async () => { + if (kind === "admission_changed") { + await db.insert(heartbeatRuns).values({ companyId, agentId, nativeIssueId: f.issueId, status: "failed", createdAt: new Date(Date.now() + 1000) }); + return { exitCode: 0, timedOut: false }; + } + if (kind === "cancelled" || kind === "completed") { + await db.update(heartbeatRuns).set({ status: kind === "cancelled" ? "cancelled" : "succeeded" }).where(eq(heartbeatRuns.id, f.runId)); + return { exitCode: 0, timedOut: false }; + } + throw new Error("probe unavailable"); + }); + if (kind === "stop_failed") runtime.retryPendingSandboxTeardown.mockRejectedValueOnce(new Error("provider stop unavailable")); + await expect(retryNativeWorkspaceExport(f.request)).rejects.toThrow(); + expect(runtime.retryPendingSandboxTeardown).toHaveBeenCalledOnce(); + expect(runtime.retryPendingSandboxTeardown).toHaveBeenCalledWith(expect.objectContaining({ lease: expect.objectContaining({ id: f.leaseId, heartbeatRunId: f.runId }) })); + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + expect(lease).toMatchObject(kind === "stop_failed" ? { status: "pending_cleanup", cleanupStatus: "failed" } : { status: "released", cleanupStatus: "success" }); + if (kind === "stop_failed") { + expect(lease.metadata?.remoteExecutionTermination).toBeUndefined(); + expect(lease.metadata?.pendingCleanupInFlight).toBe(false); + expect(lease.metadata?.nativeWorkspaceExportResume).toMatchObject({ runId: f.runId, leaseId: f.leaseId }); + } else expect(lease.metadata?.remoteExecutionTermination).toMatchObject({ runId: f.runId, leaseId: f.leaseId, state: "stopped" }); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toMatchObject({ phase: "terminal_failure", resultId: f.resultId }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).toHaveLength(0); + }); + it.each(["rebound_lease", "competing_owner", "retained_owner", "late_stop_receipt"])("does not stop or overwrite a newer sandbox owner: %s", async kind => { + const f = await seed(); + const runtime = f.request.environmentRuntime as { retryPendingSandboxTeardown: ReturnType }; + const rebound = async () => db.update(environmentLeases).set({ status: "active", heartbeatRunId: null, + metadata: { newOwner: true }, cleanupStatus: null, releasedAt: null }).where(eq(environmentLeases.id, f.leaseId)); + probe.mockImplementationOnce(async () => { + if (kind === "rebound_lease") await rebound(); + if (kind === "competing_owner" || kind === "retained_owner") { + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + await db.insert(environmentLeases).values({ companyId, environmentId, status: kind === "retained_owner" ? "retained" : "active", provider: lease.provider, providerLeaseId: lease.providerLeaseId }); + } + throw new Error("probe failed after ownership changed"); + }); + if (kind === "late_stop_receipt") runtime.retryPendingSandboxTeardown.mockImplementationOnce(async ({ lease }) => { + await rebound(); return { providerLeaseId: lease.providerLeaseId, state: "stopped" }; + }); + await expect(retryNativeWorkspaceExport(f.request)).rejects.toThrow(); + expect(runtime.retryPendingSandboxTeardown).toHaveBeenCalledTimes(kind === "late_stop_receipt" ? 1 : 0); + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + if (kind !== "competing_owner" && kind !== "retained_owner") expect(lease).toMatchObject({ status: "active", heartbeatRunId: null, metadata: { newOwner: true }, cleanupStatus: null }); + expect(lease.metadata?.remoteExecutionTermination).toBeUndefined(); + }); + it.each(["missing", "replacement", "wrong_root"])("refuses an unproven resume without probing or acquiring replacement: %s", async kind => { + const f = await seed(); + const resume = (f.request.environmentRuntime as { resumeRunLease: ReturnType }).resumeRunLease; + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + resume.mockResolvedValue({ providerLeaseId: kind === "missing" ? null : kind === "replacement" ? randomUUID() : lease.providerLeaseId, + metadata: { remoteCwd: kind === "wrong_root" ? "/other" : "/work" } }); + await expect(retryNativeWorkspaceExport(f.request)).rejects.toThrow("Resume and repair"); + expect(probe).not.toHaveBeenCalled(); + expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)))[0].status).toBe("released"); + }); + it("fences an old failure whose transaction arrives after explicit repair admission", async () => { + const f = await seed(); + let release!: () => void, captured!: () => void; + const ready = new Promise(resolve => { captured = resolve; }); + const wait = new Promise(resolve => { release = resolve; }); + const transaction = db.transaction.bind(db); + const interception = vi.spyOn(db, "transaction").mockImplementationOnce(async (...args) => { + captured(); await wait; return transaction(...args); + }); + const lateFailure = recordNativeFinalizationFailure({ db, runId: f.runId, + error: new Error("native_workspace_sync_out_failed"), failureScope: "workspace", projectRunStatus: true }); + await ready; + try { await retryNativeWorkspaceExport(f.request); } finally { release(); } + await lateFailure; interception.mockRestore(); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toMatchObject({ phase: "result_accepted", failureCode: null, nextAttemptAt: null }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toMatchObject({ status: "running", nativePhase: "result_accepted" }); + }); + it.each(["foreign_company", "missing_result", "newer_run", "lease_rebound", "other_lease", "retained_lease", "unconfirmed_stop", "destroyed"])("rejects %s before probing or changing finalization", async (kind) => { + const f = await seed(); + if (kind === "foreign_company") f.request.companyId = randomUUID(); + if (kind === "missing_result") await db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, f.runId)); + if (kind === "newer_run") await db.insert(heartbeatRuns).values({ companyId, agentId, nativeIssueId: f.issueId, status: "failed", createdAt: new Date(Date.now() + 1000) }); + if (kind === "other_lease" || kind === "retained_lease") { + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + await db.insert(environmentLeases).values({ companyId, environmentId, issueId: f.issueId, status: kind === "retained_lease" ? "retained" : "active", provider: lease.provider, providerLeaseId: lease.providerLeaseId }); + } + if (kind === "lease_rebound") await db.update(environmentLeases).set({ heartbeatRunId: null }).where(eq(environmentLeases.id, f.leaseId)); + if (kind === "unconfirmed_stop" || kind === "destroyed") { + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + await db.update(environmentLeases).set({ metadata: kind === "unconfirmed_stop" ? {} : { remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }) } }).where(eq(environmentLeases.id, f.leaseId)); + } + await expect(retryNativeWorkspaceExport(f.request)).rejects.toThrow(); + expect(probe).not.toHaveBeenCalled(); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0].phase).toBe("terminal_failure"); + }); + it("leaves repair intact when the exact sandbox is not running", async () => { + const f = await seed(); probe.mockRejectedValueOnce(new Error("stopped")); + await expect(retryNativeWorkspaceExport(f.request)).rejects.toThrow("Resume and repair"); + expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)))[0].status).toBe("released"); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0].phase).toBe("terminal_failure"); + }); + it("revalidates the lease after the read-only probe", async () => { + const f = await seed(); probe.mockImplementationOnce(async () => { await db.update(environmentLeases).set({ heartbeatRunId: null }).where(eq(environmentLeases.id, f.leaseId)); return { exitCode: 0 }; }); + await expect(retryNativeWorkspaceExport(f.request)).rejects.toThrow("no longer current"); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0].phase).toBe("terminal_failure"); + }); + it("does not take ownership from an active exporter", async () => { + const f = await seed(); + await withNativeWorkspaceFinalizationOwnership(f, async () => { + await expect(retryNativeWorkspaceExport(f.request)).rejects.toThrow("still owned"); + }); + expect(probe).not.toHaveBeenCalled(); + }); + it.each(["native_workspace_sync_out_retry_exhausted", "native_workspace_sync_out_unsafe_archive"])("does not dispatch another provider turn from a pending accepted export: %s", async cause => { + const f = await seed(cause); + await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "assignment", status: "claimed", runId: f.runId, payload: { issueId: f.issueId } }); + await recoveryService(db, { enqueueWakeup: vi.fn() }).reconcileStrandedAssignedIssues(); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId)))[0]).toMatchObject({ status: "active", ownerType: "board" }); + }); + + it.each(["eligible", "newer_run", "competing_action", "other_lease", "unaccepted_result", "changed_contract", "cancelled", "owned"])("quietly recovers historical unsafe exports with ownership fences: %s", async kind => { + const f = await seed("native_workspace_sync_out_unsafe_archive"); + await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolutionNote: "new_source_execution_path", resolvedAt: new Date() }).where(eq(issueRecoveryActions.id, f.actionId)); + if (kind === "newer_run") await db.insert(heartbeatRuns).values({ companyId, agentId, nativeIssueId: f.issueId, status: "failed", createdAt: new Date(Date.now() + 1000) }); + if (kind === "competing_action") await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog", ownerType: "board", cause: "other_repair", fingerprint: "other", nextAction: "Preserve another repair" }); + if (kind === "other_lease") { + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + await db.insert(environmentLeases).values({ companyId, environmentId, status: "active", provider: lease.provider, providerLeaseId: lease.providerLeaseId }); + } + if (kind === "unaccepted_result") await db.update(nativeRunResults).set({ schemaStatus: "rejected" }).where(eq(nativeRunResults.id, f.resultId)); + if (kind === "changed_contract") { + const [old] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, f.issueId)); + await db.insert(completionContracts).values({ ...old, id: randomUUID(), canonicalSha256: randomUUID(), revision: old.revision + 1 }); + } + if (kind === "cancelled") await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId)); + if (kind === "owned") await withNativeWorkspaceFinalizationOwnership(f, async () => recoverLegacyUnsafeWorkspaceExports(db, [f.runId])); + else await recoverLegacyUnsafeWorkspaceExports(db, [f.runId]); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId)); + expect(action.status).toBe("resolved"); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0].phase).toBe(["eligible", "other_lease"].includes(kind) ? "result_accepted" : "terminal_failure"); + expect(probe).not.toHaveBeenCalled(); + expect((f.request.environmentRuntime as { resumeRunLease: ReturnType }).resumeRunLease).not.toHaveBeenCalled(); + }); + +}); diff --git a/server/src/services/native-runtime/native-workspace-export-retry.ts b/server/src/services/native-runtime/native-workspace-export-retry.ts new file mode 100644 index 0000000000..c8c4ff598e --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-export-retry.ts @@ -0,0 +1,188 @@ +import { randomUUID } from "node:crypto"; +import type { Environment, EnvironmentLease } from "@paperclipai/shared"; +import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared"; +import { and, eq, gt, inArray, ne, or, sql } from "drizzle-orm"; +import { environmentLeases, environments, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { conflict } from "../../errors.js"; +import { persistActivity, publishActivity } from "../activity-log.js"; +import { hasRemoteTerminationReceipt, remoteTerminationReceipt } from "../remote-execution-termination.js"; +import { NATIVE_WORKSPACE_EXPORT_RESUME_KEY, readNativeWorkspaceExportResume, settleNativeWorkspaceExportResume } from "./native-workspace-export-resume.js"; +import { withNativeWorkspaceFinalizationOwnership } from "./native-workspace-finalization-ownership.js"; +import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js"; +import type { EnvironmentRuntimeService } from "../environment-runtime.js"; +import { resolveEnvironmentExecutionTarget } from "../environment-execution-target.js"; + +const record = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +const changed = () => conflict("The recorded export repair is no longer current. Refresh the task and inspect its run.", { code: "workspace_export_retry_stale" }); + +/** Explicit board admission reopens only the accepted result's finalization suffix. */ +export async function retryNativeWorkspaceExport(input: { + db: Db; companyId: string; issueId: string; actionId: string; runId: string; + actorId: string; repairNote: string; environmentRuntime: EnvironmentRuntimeService; +}) { + async function inspect(db: Db, lock = false, resumeRequestId?: string) { + const one = async (query: PromiseLike & { for(mode: "update"): PromiseLike }): Promise => + (await (lock ? query.for("update") : query))[0] ?? null; + // Keep the same issue -> coordinator -> run order as status commitment. + const issue = await one(db.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).limit(1)); + const coordinator = await one(db.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId), eq(nativeRunFinalizations.issueId, input.issueId))).limit(1)); + const run = await one(db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.nativeIssueId, input.issueId))).limit(1)); + const action = await one(db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId))).limit(1)); + if (!issue || !run || !coordinator || !action || action.evidence?.runId !== run.id + || action.kind !== "active_run_watchdog" || !isNativeWorkspaceExportRepairCause(action.cause) || action.ownerType !== "board" + || !["active", "escalated"].includes(action.status) || run.runtimeMode !== "native" + || !(issue.status === "blocked" || (issue.status === "in_review" && action.cause === "native_workspace_sync_out_retry_exhausted")) || issue.assigneeAgentId !== run.agentId + || (issue.executionRunId && issue.executionRunId !== run.id) + || (issue.checkoutRunId && issue.checkoutRunId !== run.id) || coordinator.leaseOwner) throw changed(); + const reference = readNativeWorkspaceSyncReference(record(run.runnerProfileJson).nativeWorkspaceSync); + if (!reference || reference.state !== "prepared") throw changed(); + const lease = await one(db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.id, reference.leaseId))).limit(1)); + if (!lease || lease.heartbeatRunId !== run.id || lease.issueId !== issue.id + || lease.providerLeaseId !== reference.providerLeaseId || !lease.environmentId + || typeof lease.metadata?.pluginId !== "string" || !lease.metadata.pluginId) throw changed(); + const [otherLease] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and( + ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider!), eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), inArray(environmentLeases.status, ["active", "retained", "pending_cleanup"]), + )).limit(1); + if (otherLease) throw changed(); + const [result] = await db.select().from(nativeRunResults).where(and(eq(nativeRunResults.companyId, input.companyId), eq(nativeRunResults.issueId, issue.id), eq(nativeRunResults.runId, run.id), eq(nativeRunResults.id, coordinator.resultId ?? "00000000-0000-0000-0000-000000000000"))).limit(1); + if (!result || result.schemaStatus !== "accepted" || result.completionContractId !== run.completionContractId) throw changed(); + const [newer] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, input.companyId), ne(heartbeatRuns.id, run.id), + or(eq(heartbeatRuns.nativeIssueId, issue.id), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`), + or(gt(heartbeatRuns.createdAt, run.createdAt), inArray(heartbeatRuns.status, ["queued", "running"])), + )).limit(1); + if (newer) throw changed(); + const pending = record(action.evidence?.workspaceExportRetry); + const alreadyQueued = coordinator.phase !== "terminal_failure" && pending.runId === run.id + && pending.resultId === result.id && pending.leaseId === lease.id; + const resumeIntent = readNativeWorkspaceExportResume(lease); + const ownsResume = Boolean(resumeRequestId && lease.status === "pending_cleanup" + && lease.metadata?.pendingCleanupAttemptId === resumeRequestId + && lease.metadata?.pendingCleanupInFlight === true && resumeIntent?.requestId === resumeRequestId + && Number(lease.metadata?.pendingCleanupLeaseExpiresAtMs) > Date.now() + && resumeIntent.resultId === result.id); + if (!alreadyQueued && (coordinator.phase !== "terminal_failure" || coordinator.failureCode !== action.cause + || run.status !== "failed" || (!ownsResume && (!hasRemoteTerminationReceipt(lease) + || record(lease.metadata?.remoteExecutionTermination).state !== "stopped")))) throw changed(); + if (resumeRequestId && !ownsResume) throw changed(); + const [environment] = await db.select().from(environments).where(eq(environments.id, lease.environmentId)).limit(1); + if (!environment) throw changed(); + return { issue, coordinator, run, action, reference, lease, result, environment, alreadyQueued }; + } + + const owned = await withNativeWorkspaceFinalizationOwnership(input, async (ownership) => { + const initial = await inspect(input.db); + let resumeRequestId: string | undefined; + try { + if (!initial.alreadyQueued) { + // The operator resumes and repairs the exact retained sandbox through its + // provider console. Never acquire a replacement or seed over saved work. + const target = await resolveEnvironmentExecutionTarget({ db: input.db, companyId: input.companyId, + adapterType: "paperclip_runner", environment: initial.environment, leaseId: initial.lease.id, + leaseMetadata: initial.lease.metadata, lease: initial.lease as EnvironmentLease, environmentRuntime: input.environmentRuntime }); + if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner + || target.remoteCwd !== initial.reference.remoteCwd) throw changed(); + await ownership.assertHeld(); + const requestId = randomUUID(); + const claimedLease = await input.db.transaction(async tx => { + const current = await inspect(tx as unknown as Db, true); + if (current.alreadyQueued || current.result.id !== initial.result.id || current.lease.id !== initial.lease.id + || current.reference.descriptorSha256 !== initial.reference.descriptorSha256) throw changed(); + const now = new Date(); + // Persist cleanup authority before any provider call. A lost resume + // reply or controller restart must not leave a running sandbox behind + // an old stopped receipt. The pending claim fences the cleanup sweeper. + const [lease] = await tx.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", + releasedAt: now, failureReason: "workspace_export_resume_pending", updatedAt: now, + metadata: { ...current.lease.metadata, remoteExecutionTermination: undefined, + [NATIVE_WORKSPACE_EXPORT_RESUME_KEY]: { schema: "paperclip.workspace-export-resume.v2", requestId, + companyId: input.companyId, runId: input.runId, leaseId: current.lease.id, + pluginId: current.lease.metadata?.pluginId, + provider: current.lease.provider, providerLeaseId: current.lease.providerLeaseId, resultId: current.result.id }, + pendingCleanupAttemptId: requestId, pendingCleanupInFlight: true, + pendingCleanupRetryAfterMs: 0, pendingCleanupRetryAttempts: 0, pendingCleanupRetryCapWarned: false, + pendingCleanupLeaseExpiresAtMs: now.getTime() + 15 * 60_000 }, + }).where(eq(environmentLeases.id, current.lease.id)).returning(); + return lease; + }); + resumeRequestId = requestId; + try { + await ownership.assertHeld(); + await inspect(input.db, false, requestId); + // A provider stop also closes its controller-side activity gate. The + // operator's external repair does not reopen that gate. Resume only the + // recorded lease through its provider so sentinel and drain checks run. + const resumed = await input.environmentRuntime.resumeRunLease({ environment: initial.environment as unknown as Environment, + lease: claimedLease as EnvironmentLease }); + if (resumed?.providerLeaseId !== initial.lease.providerLeaseId + || resumed.metadata?.remoteCwd !== initial.reference.remoteCwd) throw new Error("sandbox_resume_identity_unproven"); + const probe = await target.runner.execute({ command: target.shellCommand ?? "sh", args: ["-c", "true"], cwd: "/", timeoutMs: 10_000, bypassSession: true }); + if (probe.timedOut || probe.exitCode !== 0) throw new Error("sandbox_not_running"); + } catch { + throw conflict("Resume and repair the retained sandbox before retrying workspace export. No provider turn was started.", { code: "workspace_export_sandbox_unavailable" }); + } + } + await ownership.assertHeld(); + const admitted = await input.db.transaction(async tx => { + const current = await inspect(tx as unknown as Db, true, resumeRequestId); + if (current.result.id !== initial.result.id || current.lease.id !== initial.lease.id + || current.reference.descriptorSha256 !== initial.reference.descriptorSha256) throw changed(); + const receipt = { runId: current.run.id, resultId: current.result.id, leaseId: current.lease.id, status: "queued" as const }; + if (current.alreadyQueued) return { receipt, publication: null }; + const now = new Date(); + const retry = { ...receipt, requestId: randomUUID(), actorId: input.actorId, repairNote: input.repairNote, requestedAt: now.toISOString(), + stoppedProvider: initial.lease.metadata?.remoteExecutionTermination }; + await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null, + failureDetail: { ...current.coordinator.failureDetail, workspaceFinalizeAttempt: 0, workspaceExportRetry: retry }, updatedAt: now }).where(eq(nativeRunFinalizations.runId, current.run.id)); + await tx.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null, + nativePhase: "result_accepted", nativePhaseUpdatedAt: now, + resultJson: { ...current.run.resultJson, workspaceExportRetry: retry, finalizationPhase: "result_accepted", failureCode: null, originalFailureCode: null, nextAttemptAt: null }, updatedAt: now }).where(eq(heartbeatRuns.id, current.run.id)); + await tx.update(issues).set({ executionRunId: current.run.id, updatedAt: now }).where(eq(issues.id, current.issue.id)); + // Reclaim the same lease only for control-plane copyback. The normal + // finalization settlement will stop/retain it again on success or failure. + await tx.update(environmentLeases).set({ status: "active", releasedAt: null, cleanupStatus: null, failureReason: null, + metadata: { ...current.lease.metadata, remoteExecutionTermination: undefined, + // Keep stop-only authority through copyback and commitment. A crash + // before final release must not make the orphan sweeper destroy it. + pendingCleanupAttemptId: undefined, pendingCleanupInFlight: false, + pendingCleanupLeaseExpiresAtMs: 0 }, updatedAt: now }).where(eq(environmentLeases.id, current.lease.id)); + await tx.update(issueRecoveryActions).set({ evidence: { ...current.action.evidence, workspaceExportRetry: retry }, + nextAction: "Workspace export is queued for the accepted result. No provider turn will run.", + wakePolicy: { kind: "resume_native_run", runId: current.run.id }, updatedAt: now }).where(eq(issueRecoveryActions.id, current.action.id)); + const publication = await persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "user", actorId: input.actorId, + action: "issue.workspace_export_retry_requested", entityType: "issue", entityId: input.issueId, runId: input.runId, + details: { recoveryActionId: input.actionId, resultId: current.result.id, leaseId: current.lease.id, repairNote: input.repairNote } }); + return { receipt, publication }; + }); + if (admitted.publication) publishActivity(admitted.publication.publication); + return admitted.receipt; + } catch (error) { + if (resumeRequestId) { + // Admission may have committed before a connection failed. Stop only + // while our durable intent is still present; never stop a later owner. + try { + const [lease] = await input.db.select().from(environmentLeases).where(and(eq(environmentLeases.id, initial.lease.id), + eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.heartbeatRunId, input.runId), + eq(environmentLeases.status, "pending_cleanup"), + sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${resumeRequestId}`)).limit(1); + const [otherOwner] = await input.db.select({ id: environmentLeases.id }).from(environmentLeases).where(and( + ne(environmentLeases.id, initial.lease.id), eq(environmentLeases.provider, initial.lease.provider!), + eq(environmentLeases.providerLeaseId, initial.lease.providerLeaseId!), inArray(environmentLeases.status, ["active", "retained", "pending_cleanup"]), + )).limit(1); + if (lease && !otherOwner && readNativeWorkspaceExportResume(lease)?.requestId === resumeRequestId) { + let termination: ReturnType; + try { + const receipt = await input.environmentRuntime.retryPendingSandboxTeardown({ environment: initial.environment as unknown as Environment, lease: lease as EnvironmentLease }); + termination = remoteTerminationReceipt(lease, receipt); + if (termination?.state !== "stopped") termination = undefined; + } catch { /* The durable stop-only intent remains recoverable. */ } + await settleNativeWorkspaceExportResume(input.db, lease, { attemptId: resumeRequestId, receipt: termination }); + } + } catch { /* A database outage leaves the pre-resume intent for recovery. */ } + } + throw error; + } + }); + if (!owned.acquired) throw conflict("Workspace finalization is still owned by another operation. Wait for it to stop before retrying.", { code: "workspace_export_retry_busy" }); + return owned.value; +} diff --git a/server/src/services/native-runtime/native-workspace-failure.ts b/server/src/services/native-runtime/native-workspace-failure.ts new file mode 100644 index 0000000000..36eade3f3b --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-failure.ts @@ -0,0 +1,22 @@ +import { classifyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-merge"; + +export type NativeWorkspaceFailureCode = + | "workspace_sync_out_failed" + | "workspace_sync_out_unrecoverable" + | "workspace_sync_out_unsafe_archive"; + +/** Keep initial and recovered copy-back failures on the same retry policy. */ +export function classifyNativeWorkspaceFailure(error: unknown): { + code: NativeWorkspaceFailureCode; + failureCode: `native_${NativeWorkspaceFailureCode}`; + permanent: boolean; +} { + const message = error instanceof Error ? error.message.trim() : ""; + const code: NativeWorkspaceFailureCode = + message === "workspace_sync_out_unrecoverable" || message.includes("daytona_sandbox_not_found") + ? "workspace_sync_out_unrecoverable" + : message === "workspace_sync_out_unsafe_archive" || classifyWorkspaceRestoreFailure(error) === "restore_unsafe_archive" + ? "workspace_sync_out_unsafe_archive" + : "workspace_sync_out_failed"; + return { code, failureCode: `native_${code}`, permanent: code !== "workspace_sync_out_failed" }; +} diff --git a/server/src/services/native-runtime/native-workspace-finalization-ownership.ts b/server/src/services/native-runtime/native-workspace-finalization-ownership.ts new file mode 100644 index 0000000000..84227de004 --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-finalization-ownership.ts @@ -0,0 +1,161 @@ +import { issueRecoveryActionService } from "../issue-recovery-actions.js"; +import { nativeSha256 } from "./canonical.js"; +import { randomUUID } from "node:crypto"; +import os from "node:os"; +import { and, eq, sql } from "drizzle-orm"; +import { heartbeatRuns, workspaceOperations, withDedicatedDbConnection, type Db } from "@paperclipai/db"; +import { + currentNativeControllerIdentity, + evaluateNativeControllerTakeover, +} from "./native-restart-recovery.js"; + +const OWNER_KEY = "nativeWorkspaceFinalizationOwner"; +const activeOwnerTokens = new Set(); +// Positive same-boot evidence, retained only when durable cleanup could not be +// confirmed. Absence from the active set alone never proves another token joined. +const joinedOwnerTokens = new Set(); +type Owner = { + token: string; + controllerBootId?: string; + hostname: string; + pid: number; + processStartedAt: string; +}; +export type NativeWorkspaceFinalizationOwnership = { + token: string; + /** Recheck both the physical lock connection and the durable owner before publishing. */ + assertHeld(): Promise; +}; +export class NativeWorkspaceFinalizationBusyError extends Error { + constructor() { super("native_workspace_finalization_busy"); } +} +export class NativeWorkspaceFinalizationOwnershipLostError extends Error { + constructor(options?: ErrorOptions) { + super("native_workspace_finalization_ownership_lost", options); + } +} + +function readOwner(value: unknown): Owner | null { + if (!value || typeof value !== "object") return null; + const owner = value as Partial; + return typeof owner.token === "string" && typeof owner.hostname === "string" + && Number.isInteger(owner.pid) && Number(owner.pid) > 0 + && typeof owner.processStartedAt === "string" && Number.isFinite(Date.parse(owner.processStartedAt)) + ? owner as Owner : null; +} + +/** + * The advisory lock serializes controllers without holding row locks across I/O. + * The durable receipt additionally prevents a disconnected lock session from + * authorizing takeover while its controller is still doing physical copyback. + * There is deliberately no wall-clock timeout that can steal a slow export. + */ +export async function withNativeWorkspaceFinalizationOwnership( + input: { db: Db; companyId: string; runId: string }, + action: (ownership: NativeWorkspaceFinalizationOwnership) => Promise, +): Promise<{ acquired: false } | { acquired: true; value: T }> { + const identity = await currentNativeControllerIdentity(); + const owner: Owner = { + token: randomUUID(), controllerBootId: identity.bootId, hostname: os.hostname(), pid: identity.pid, + processStartedAt: identity.processStartedAt.toISOString(), + }; + const scope = and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.runtimeMode, "native")); + const owned = and(scope, sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${owner.token}`); + let claimed = false; + let actionPromise: Promise | undefined; + try { + return await withDedicatedDbConnection(input.db, (dedicated) => dedicated.transaction(async (lock) => { + const rows = await lock.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${input.companyId}:${input.runId}`}, 0)) as acquired`); + if (rows[0]?.acquired !== true) return { acquired: false } as const; + // Reads and durable progress use the ordinary pool, not the lock transaction. + const [run] = await input.db.select({ profile: heartbeatRuns.runnerProfileJson, issueId: heartbeatRuns.nativeIssueId, agentId: heartbeatRuns.agentId }).from(heartbeatRuns).where(scope).limit(1); + if (!run) throw new Error("native_workspace_finalization_binding_missing"); + const rawPrior = run.profile?.[OWNER_KEY]; + const prior = readOwner(rawPrior); + if (rawPrior != null) { + const requireStopVerification = async () => { + if (run.issueId) await issueRecoveryActionService(input.db).upsertSourceScoped({ + companyId: input.companyId, sourceIssueId: run.issueId, + kind: "active_run_watchdog", ownerType: "board", returnOwnerAgentId: run.agentId, + cause: "native_workspace_finalization_owner_unverified", + fingerprint: nativeSha256({ runId: input.runId, owner: rawPrior }), + evidence: { runId: input.runId, owner: rawPrior }, + nextAction: "Verify the previous controller and its workspace-copyback processes have stopped, then release only this exact workspace owner receipt using doc/native-workspace-finalization-recovery.md. Resume workspace finalization without another provider turn.", + wakePolicy: null, maxAttempts: 1, supersedeOnIdentityChange: true, + }); + return { acquired: false } as const; + }; + // A foreign host or malformed receipt cannot prove the old physical owner stopped. + if (!prior || prior.hostname !== owner.hostname) return requireStopVerification(); + const knownJoinedHere = prior.controllerBootId === identity.bootId + && prior.pid === identity.pid && joinedOwnerTokens.has(prior.token) + && !activeOwnerTokens.has(prior.token); + if (!knownJoinedHere) { + const takeover = await evaluateNativeControllerTakeover({ + owner: { leaseOwner: prior.token, leaseExpiresAt: new Date(0), controllerPid: prior.pid, + controllerProcessStartedAt: new Date(prior.processStartedAt) }, now: new Date(), + }); + if (!takeover.allowed) return { acquired: false } as const; + // A dead parent may have orphaned tar/Git children. Only a durable + // completed-copyback barrier proves they joined; otherwise board proof + // is required even on the same host. Both callers reuse this barrier. + const [completedCopyback] = await input.db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and( + eq(workspaceOperations.companyId, input.companyId), + eq(workspaceOperations.heartbeatRunId, input.runId), + eq(workspaceOperations.issueId, run.issueId!), + eq(workspaceOperations.phase, "workspace_finalize"), + eq(workspaceOperations.status, "succeeded"), + )).limit(1); + if (!completedCopyback) return requireStopVerification(); + } + } + // Claim before the potentially ambiguous write so finally can remove + // only our token even if the server committed but its response was lost. + activeOwnerTokens.add(owner.token); + claimed = true; + const [claimedRun] = await input.db.update(heartbeatRuns).set({ + runnerProfileJson: sql`jsonb_set(coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb), array[${OWNER_KEY}], ${JSON.stringify(owner)}::jsonb)`, + }).where(and(scope, prior + ? sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${prior.token}` + : sql`${heartbeatRuns.runnerProfileJson}->>${OWNER_KEY} is null` + )).returning({ id: heartbeatRuns.id }); + if (!claimedRun) return { acquired: false } as const; + if (prior) joinedOwnerTokens.delete(prior.token); + const ownership: NativeWorkspaceFinalizationOwnership = { + token: owner.token, + assertHeld: async () => { + try { + // A failed transaction is never transparently reconnected by the DB client. + await lock.execute(sql`select 1`); + const [current] = await input.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(owned).limit(1); + if (!current) throw new Error("owner_replaced"); + } catch (cause) { + throw new NativeWorkspaceFinalizationOwnershipLostError({ cause }); + } + }, + }; + await ownership.assertHeld(); + actionPromise = action(ownership); + const value = await actionPromise; + await ownership.assertHeld(); + return { acquired: true, value } as const; + })); + } finally { + // postgres.js may reject the transaction when the socket dies before the + // callback settles. Join that callback before releasing its durable fence. + await actionPromise?.catch(() => undefined); + if (claimed) { + activeOwnerTokens.delete(owner.token); + joinedOwnerTokens.add(owner.token); + try { + await input.db.update(heartbeatRuns).set({ + runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) - ${OWNER_KEY}`, + }).where(owned); + joinedOwnerTokens.delete(owner.token); + } catch { + // A reconnect in this exact controller boot can use the positive join + // receipt. A new boot still needs durable barrier or operator proof. + } + } + } +} diff --git a/server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts b/server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts new file mode 100644 index 0000000000..a1c5ae1ec3 --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts @@ -0,0 +1,64 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import type { Db } from "@paperclipai/db"; + +const sync = vi.hoisted(() => ({ resume: vi.fn() })); +vi.mock("./native-workspace-finalization-ownership.js", () => ({ + withNativeWorkspaceFinalizationOwnership: async (_input: unknown, action: (owner: unknown) => unknown) => + ({ acquired: true, value: await action({ token: "fixture-owner", assertHeld: async () => {} }) }), +})); +vi.mock("./native-workspace-sync.js", () => ({ + readNativeWorkspaceSyncReference: () => ({ leaseId: "lease", providerLeaseId: "sandbox", workspaceId: "workspace" }), + resumeNativeWorkspaceSync: sync.resume, +})); +vi.mock("../environments.js", () => ({ environmentService: () => ({ + getLeaseById: async () => ({ id: "lease", companyId: "company", environmentId: "environment", providerLeaseId: "sandbox", status: "active" }), + getById: async () => ({ id: "environment", companyId: "company" }), +}) })); +vi.mock("../environment-execution-target.js", () => ({ resolveEnvironmentExecutionTarget: async () => ({ kind: "remote" }) })); +vi.mock("../workspace-operations.js", () => ({ workspaceOperationService: () => ({ + createRecorder: () => ({ recordOperation: async (input: { run: () => Promise }) => input.run() }), +}) })); +import { resumeNativeWorkspaceFinalization } from "./native-workspace-finalizer.js"; + +function fixtureDb(): Db { + const responses = [[{ + companyId: "company", runtimeMode: "native", issueId: "issue", resultId: "result", + runnerProfileJson: { nativeWorkspaceSync: {}, nativeExecutionInput: { binding: {} } }, + }], [{ phase: "result_accepted", nextAttemptAt: null, resultId: "result" }], []]; + return { select: () => { + const query = { + from: () => query, innerJoin: () => query, where: () => query, orderBy: () => query, + limit: async () => responses.shift() ?? [], + }; + return query; + } } as unknown as Db; +} + +describe("native workspace finalization without a manual export repair", () => { + beforeEach(() => sync.resume.mockReset()); + it.each([ + "Daytona syncOut refusing tarball link whose target escapes the extraction dir: tools/pnpm -> /usr/bin/pnpm", + "Daytona syncOut refusing tarball member that escapes the extraction dir: ../private", + ])("settles unsafe exports successfully after restart: %s", async (message) => { + sync.resume.mockRejectedValueOnce(new Error(message)); + const result = await resumeNativeWorkspaceFinalization({ db: fixtureDb(), runId: "run", environmentRuntime: {} as never }); + expect(result).toMatchObject({ status: "succeeded", exitCode: 0 }); + expect(sync.resume).toHaveBeenCalledOnce(); + expect(JSON.stringify(result)).not.toContain("tools/pnpm"); + }); + + it.each([ + ["daytona_sandbox_not_found", "workspace_sync_out_unrecoverable"], + ["Daytona syncOut directory download failed: timeout", "workspace_sync_out_failed"], + ])("retains unrelated failure handling: %s", async (message, code) => { + sync.resume.mockRejectedValueOnce(new Error(message)); + const result = await resumeNativeWorkspaceFinalization({ db: fixtureDb(), runId: "run", environmentRuntime: {} as never }); + expect(result).toMatchObject({ status: "failed", exitCode: 1, stderr: `${code}\n` }); + }); + + it("does not treat a missing workspace reference as an omitted unsafe archive", async () => { + sync.resume.mockResolvedValueOnce(false); + const result = await resumeNativeWorkspaceFinalization({ db: fixtureDb(), runId: "run", environmentRuntime: {} as never }); + expect(result).toMatchObject({ status: "failed", exitCode: 1, stderr: "workspace_sync_out_unrecoverable\n" }); + }); +}); diff --git a/server/src/services/native-runtime/native-workspace-finalizer.test.ts b/server/src/services/native-runtime/native-workspace-finalizer.test.ts index 7221370023..766dd1cf53 100644 --- a/server/src/services/native-runtime/native-workspace-finalizer.test.ts +++ b/server/src/services/native-runtime/native-workspace-finalizer.test.ts @@ -1,9 +1,13 @@ +import { spawn } from "node:child_process"; +import { once } from "node:events"; +import { readProcessStartedAt } from "../hot-restart.js"; +import { withNativeWorkspaceFinalizationOwnership } from "./native-workspace-finalization-ownership.js"; import { randomUUID } from "node:crypto"; import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { afterAll, beforeAll, describe, expect, it } from "vitest"; -import { and, desc, eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { and, desc, eq, sql } from "drizzle-orm"; import { agents, companies, @@ -14,6 +18,7 @@ import { issues, nativeRunFinalizations, nativeRunResults, + issueRecoveryActions, projects, workspaceOperations, } from "@paperclipai/db"; @@ -154,6 +159,209 @@ describe("native workspace finalization recovery", () => { } }); + it("reuses a successful export even if an old controller wrote a later failed barrier", async () => { + const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Late failed barrier" }); + const [success] = await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: seeded.runId, + issueId: seeded.issueId, phase: "workspace_finalize", status: "succeeded" }).returning(); + await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: seeded.runId, + issueId: seeded.issueId, phase: "workspace_finalize", status: "failed", createdAt: new Date(Date.now() + 1000) }); + const operation = await resumeNativeWorkspaceFinalization({ db, runId: seeded.runId }); + expect(operation?.id).toBe(success.id); + expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, seeded.runId))).toHaveLength(2); + }); + + it.each(["terminal_failure", "retryable_failure"] as const)( + "rechecks %s admission before a delayed recovery starts another export", + async (phase) => { + const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Delayed recovery admission" }); + await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: seeded.runId, + issueId: seeded.issueId, phase: "workspace_finalize", status: "failed", + stderrExcerpt: "workspace_sync_out_unsafe_archive\n" }); + // A sweep can select its run before live copyback publishes this outcome, + // then enter the export path only after the live owner releases its lock. + await db.update(nativeRunFinalizations).set({ phase, + failureCode: "native_workspace_sync_out_unsafe_archive", + failureDetail: { workspaceFinalizeAttempt: 1, recoveryOwner: { kind: "board" } }, + nextAttemptAt: phase === "retryable_failure" ? new Date(Date.now() + 60_000) : null, + }).where(eq(nativeRunFinalizations.runId, seeded.runId)); + const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seeded.runId)); + expect(await resumeNativeWorkspaceFinalization({ db, runId: seeded.runId })).toBeNull(); + expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, seeded.runId))).toHaveLength(1); + const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seeded.runId)); + expect(after).toEqual(before); + }, + ); + + it("keeps ordinary progress writable even when the application pool has one connection", async () => { + const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Small application pool" }); + const smallPool = createDb(temporary.connectionString, { maxConnections: 1 }); + const result = await withNativeWorkspaceFinalizationOwnership({ db: smallPool, companyId, runId: seeded.runId }, async (ownership) => { + await ownership.assertHeld(); + await smallPool.update(heartbeatRuns).set({ updatedAt: new Date() }).where(eq(heartbeatRuns.id, seeded.runId)); + return "progress saved"; + }); + expect(result).toEqual({ acquired: true, value: "progress saved" }); + }); + + it("skips recovery while the live heartbeat owns slow copyback", async () => { + const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Live copyback" }); + const live = await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async (ownership) => { + await ownership.assertHeld(); + expect(await resumeNativeWorkspaceFinalization({ db: createDb(temporary.connectionString), runId: seeded.runId })).toBeNull(); + expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, seeded.runId))).toEqual([]); + return "copied"; + }); + expect(live).toEqual({ acquired: true, value: "copied" }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); + expect(run.runnerProfileJson).not.toHaveProperty("nativeWorkspaceFinalizationOwner"); + }); + + it("keeps the physical owner fenced after its lock connection dies until copyback joins", async () => { + const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Disconnected copyback" }); + let enter!: () => void; + const entered = new Promise((resolve) => { enter = resolve; }); + let finish!: () => void; + const finished = new Promise((resolve) => { finish = resolve; }); + let published = false; + const first = withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async (ownership) => { + enter(); await finished; await ownership.assertHeld(); published = true; + }); + // Observe rejection immediately, while retaining the promise until its callback joins. + const outcome = first.then(() => null, (error: unknown) => error); + try { + await entered; + await db.execute(sql`select pg_terminate_backend(pid) from pg_locks where locktype = 'advisory' and database = (select oid from pg_database where datname = current_database()) and pid <> pg_backend_pid()`); + const second = await withNativeWorkspaceFinalizationOwnership({ db: createDb(temporary.connectionString), companyId, runId: seeded.runId }, async () => "must not run"); + expect(second).toEqual({ acquired: false }); + const [held] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); + expect(held.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeTruthy(); + } finally { finish(); } + expect(await outcome).toBeTruthy(); + expect(published).toBe(false); + expect(await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async () => "recovered")) + .toEqual({ acquired: true, value: "recovered" }); + }); + + it("recovers its own joined copyback receipt after the application pool disconnects during cleanup", async () => { + const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Reconnect joined copyback" }); + const interruptedDb = createDb(temporary.connectionString); + let joined = false; + await expect(withNativeWorkspaceFinalizationOwnership({ db: interruptedDb, companyId, runId: seeded.runId }, async () => { + await fs.writeFile(path.join(workspaceRoot, "joined-copyback.txt"), "durable work"); + joined = true; + await interruptedDb.$client.end({ timeout: 1 }); + })).rejects.toThrow(); + expect(joined).toBe(true); + const [beforeReconnect] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); + const receipt = beforeReconnect.runnerProfileJson?.nativeWorkspaceFinalizationOwner as Record; + expect(receipt).toBeTruthy(); + for (const unknown of [{ ...receipt, controllerBootId: randomUUID() }, { ...receipt, token: randomUUID() }]) { + await db.update(heartbeatRuns).set({ runnerProfileJson: { ...beforeReconnect.runnerProfileJson, nativeWorkspaceFinalizationOwner: unknown } }) + .where(eq(heartbeatRuns.id, seeded.runId)); + expect(await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async () => "must not run")) + .toEqual({ acquired: false }); + } + await db.update(heartbeatRuns).set({ runnerProfileJson: beforeReconnect.runnerProfileJson }).where(eq(heartbeatRuns.id, seeded.runId)); + expect(await withNativeWorkspaceFinalizationOwnership({ db: createDb(temporary.connectionString), companyId, runId: seeded.runId }, async () => "resumed")) + .toEqual({ acquired: true, value: "resumed" }); + }); + + it("never steals pending copyback from an orphan child after its controller is killed", async () => { + const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Orphan copyback child" }); + const output = path.join(workspaceRoot, "orphan-copyback.txt"); + const childScript = `const fs = require('node:fs'); setInterval(() => fs.appendFileSync(${JSON.stringify(output)}, 'copying\\n'), 25);`; + const parentScript = `const {spawn} = require('node:child_process'); const child = spawn(process.execPath, ['-e', ${JSON.stringify(childScript)}], {detached:true, stdio:'ignore'}); console.log(child.pid); child.unref(); setInterval(() => {}, 1000);`; + const parent = spawn(process.execPath, ["-e", parentScript], { stdio: ["ignore", "pipe", "ignore"] }); + const [chunk] = await once(parent.stdout!, "data"); + const childPid = Number(String(chunk).trim()); + expect(childPid).toBeGreaterThan(0); + try { + const startedAt = await readProcessStartedAt(parent.pid!); + const exited = once(parent, "close"); + parent.kill("SIGKILL"); await exited; + process.kill(childPid, 0); // The copyback child outlived the exact controller. + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: { + token: randomUUID(), hostname: os.hostname(), pid: parent.pid, processStartedAt: startedAt, + } } }).where(eq(heartbeatRuns.id, seeded.runId)); + const work = vi.fn(async () => "second physical writer"); + expect(await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, work)).toEqual({ acquired: false }); + expect(work).not.toHaveBeenCalled(); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, seeded.issueId)); + expect(action).toMatchObject({ ownerType: "board", cause: "native_workspace_finalization_owner_unverified", wakePolicy: null }); + } finally { + parent.kill("SIGKILL"); + try { process.kill(-childPid, "SIGKILL"); } catch { /* fixture already exited */ } + } + }); + + it("recovers completed copyback after the exact controller process on this host exited", async () => { + const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Dead copyback controller" }); + await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: seeded.runId, + issueId: seeded.issueId, phase: "workspace_finalize", status: "succeeded" }); + const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" }); + await once(child, "spawn"); + const startedAt = await readProcessStartedAt(child.pid!); + expect(startedAt).toBeTruthy(); + const exited = once(child, "close"); + child.kill("SIGTERM"); await exited; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: { + token: randomUUID(), hostname: os.hostname(), pid: child.pid, processStartedAt: startedAt, + } } }).where(eq(heartbeatRuns.id, seeded.runId)); + expect(await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async () => "recovered")) + .toEqual({ acquired: true, value: "recovered" }); + }); + + it("surfaces an unverifiable foreign controller as board recovery without physical writes", async () => { + const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Foreign copyback controller" }); + const owner = { token: randomUUID(), hostname: "other-controller.invalid", pid: 42, processStartedAt: new Date().toISOString() }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seeded.runId)); + const work = vi.fn(); + expect(await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, work)).toEqual({ acquired: false }); + expect(work).not.toHaveBeenCalled(); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, seeded.issueId)); + expect(action).toMatchObject({ ownerType: "board", cause: "native_workspace_finalization_owner_unverified", wakePolicy: null, evidence: { runId: seeded.runId, owner } }); + expect(action.nextAction).toContain("previous controller"); + await expect(withNativeWorkspaceFinalizationOwnership({ db, companyId: foreignCompanyId, runId: seeded.runId }, work)) + .rejects.toThrow("native_workspace_finalization_binding_missing"); + const [held] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); + expect(held.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(owner); + }); + + it("does not start another recovery while workspace finalization is still running", async () => { + const cwd = path.join(workspaceRoot, "slow-workspace"); + await fs.mkdir(cwd); + const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Slow workspace export" }); + await db.insert(workspaceOperations).values({ + companyId, heartbeatRunId: seeded.runId, issueId: seeded.issueId, + phase: "workspace_finalize", cwd, status: "failed", + }); + let entered!: () => void; + const started = new Promise((resolve) => { entered = resolve; }); + let release!: () => void; + const held = new Promise((resolve) => { release = resolve; }); + const stat = fs.stat.bind(fs); + let calls = 0; + const spy = vi.spyOn(fs, "stat").mockImplementation(async (...args: Parameters) => { + if (args[0] === cwd) { calls += 1; entered(); await held; } + return stat(...args); + }); + const first = resumeNativeWorkspaceFinalization({ db, runId: seeded.runId }); + try { + await started; + const second = resumeNativeWorkspaceFinalization({ db, runId: seeded.runId }); + // A second invocation must return busy without waiting for or executing the slow I/O. + const outcome = await Promise.race([second, new Promise((resolve) => setTimeout(() => resolve("still-running"), 1_000))]); + release(); + await Promise.all([first, second]); + expect(outcome).toBeNull(); + expect(calls).toBe(1); + const operations = await db.select().from(workspaceOperations).where(and( + eq(workspaceOperations.heartbeatRunId, seeded.runId), eq(workspaceOperations.status, "succeeded"), + )); + expect(operations).toHaveLength(1); + } finally { release(); await first; spy.mockRestore(); } + }); + it("records directory-only recovery without inventing an execution-workspace foreign key", async () => { const cwd = path.join(workspaceRoot, "directory-only"); await fs.mkdir(cwd); @@ -195,7 +403,7 @@ describe("native workspace finalization recovery", () => { cwd, status: "succeeded", }); - expect(operation.metadata).toMatchObject({ + expect(operation?.metadata).toMatchObject({ owningService: "native_workspace_finalizer", observation: "workspace_directory", }); @@ -301,15 +509,15 @@ describe("native workspace finalization recovery", () => { cwd: authorizedCwd, status: "succeeded", }); - expect(operation.cwd).not.toBe(foreignCwd); - expect(operation.cwd).not.toBe(mismatchedCwd); + expect(operation!.cwd).not.toBe(foreignCwd); + expect(operation!.cwd).not.toBe(mismatchedCwd); const persisted = await db .select() .from(workspaceOperations) .where( and( - eq(workspaceOperations.id, operation.id), + eq(workspaceOperations.id, operation!.id), eq(workspaceOperations.companyId, companyId), eq(workspaceOperations.issueId, seeded.issueId), ), diff --git a/server/src/services/native-runtime/native-workspace-finalizer.ts b/server/src/services/native-runtime/native-workspace-finalizer.ts index 5dbf526f6a..f0d7b657f1 100644 --- a/server/src/services/native-runtime/native-workspace-finalizer.ts +++ b/server/src/services/native-runtime/native-workspace-finalizer.ts @@ -1,3 +1,5 @@ +import { restoreNativeWorkspaceBestEffort } from "./native-workspace-best-effort.js"; +import { withNativeWorkspaceFinalizationOwnership } from "./native-workspace-finalization-ownership.js"; import fs from "node:fs/promises"; import { and, desc, eq } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -71,6 +73,35 @@ export async function resumeNativeWorkspaceFinalization(input: { throw new Error("native_workspace_finalization_binding_missing"); } + const owned = await withNativeWorkspaceFinalizationOwnership({ + db: input.db, companyId: bound.companyId, runId: input.runId, + }, async (ownership) => { + // A sweep can be admitted before the live owner publishes a permanent + // failure or retry delay, then acquire ownership after that owner releases. + // Recheck admission inside the lock before any copyback or operation receipt. + const admission = await input.db.select({ + phase: nativeRunFinalizations.phase, + nextAttemptAt: nativeRunFinalizations.nextAttemptAt, + resultId: nativeRunFinalizations.resultId, + }).from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.runId, input.runId), + eq(nativeRunFinalizations.companyId, bound.companyId), + eq(nativeRunFinalizations.issueId, bound.issueId), + )).limit(1).then((rows) => rows[0] ?? null); + if (!admission || admission.resultId !== bound.resultId) { + throw new Error("native_workspace_finalization_binding_missing"); + } + if (admission.phase === "terminal_failure" + || (admission.nextAttemptAt && admission.nextAttemptAt > new Date())) return null; + const successful = await input.db.select().from(workspaceOperations).where(and( + eq(workspaceOperations.companyId, bound.companyId), + eq(workspaceOperations.heartbeatRunId, input.runId), + eq(workspaceOperations.issueId, bound.issueId), + eq(workspaceOperations.phase, "workspace_finalize"), + eq(workspaceOperations.status, "succeeded"), + )).orderBy(desc(workspaceOperations.createdAt)).limit(1).then((rows) => rows[0] ?? null); + // A stale failure from a pre-fencing controller cannot invalidate exported work. + if (successful) return successful; const previous = await input.db.select().from(workspaceOperations).where(and( eq(workspaceOperations.companyId, bound.companyId), eq(workspaceOperations.heartbeatRunId, input.runId), @@ -116,6 +147,7 @@ export async function resumeNativeWorkspaceFinalization(input: { : "workspace_directory", }, run: async () => { + await ownership.assertHeld(); if (nativeWorkspaceSync) { if (!input.environmentRuntime) { return { @@ -167,19 +199,25 @@ export async function resumeNativeWorkspaceFinalization(input: { }; } try { - const restored = await resumeNativeWorkspaceSync({ + const restored = await restoreNativeWorkspaceBestEffort({ db: input.db, runId: input.runId, - target, + assertOwnership: ownership.assertHeld, + restore: () => resumeNativeWorkspaceSync({ + db: input.db, runId: input.runId, target, assertOwnership: ownership.assertHeld, + }), }); - if (!restored) { + await ownership.assertHeld(); + if (restored === false) { return workspaceSyncFailure("workspace_sync_out_unrecoverable"); } return { status: "succeeded", exitCode: 0, system: - "Native workspace finalization restored the remote workspace.\n", + restored + ? "Native workspace finalization restored the remote workspace.\n" + : "Unsafe workspace export omitted; finalization completed.\n", metadata: { workspaceSync: { schema: nativeWorkspaceSync.schema, @@ -189,6 +227,7 @@ export async function resumeNativeWorkspaceFinalization(input: { }, }; } catch (error) { + await ownership.assertHeld(); const code = error instanceof Error && (error.message === "workspace_sync_out_unrecoverable" || @@ -206,6 +245,7 @@ export async function resumeNativeWorkspaceFinalization(input: { }; } const isDirectory = await fs.stat(cwd).then((stat) => stat.isDirectory()).catch(() => false); + await ownership.assertHeld(); if (!isDirectory) { return { status: "failed", @@ -241,4 +281,6 @@ export async function resumeNativeWorkspaceFinalization(input: { }; }, }); + }); + return owned.acquired ? owned.value : null; } diff --git a/server/src/services/native-runtime/native-workspace-sync.ts b/server/src/services/native-runtime/native-workspace-sync.ts index a618f612eb..9953d1473f 100644 --- a/server/src/services/native-runtime/native-workspace-sync.ts +++ b/server/src/services/native-runtime/native-workspace-sync.ts @@ -87,7 +87,7 @@ export interface NativeWorkspaceSyncReference { export interface PreparedNativeWorkspaceSync { mode: WorkspaceInboundMode; reference: NativeWorkspaceSyncReference; - restoreWorkspace(): Promise; + restoreWorkspace(assertOwnership?: () => Promise): Promise; cleanup(): Promise; } @@ -772,8 +772,10 @@ async function finalizePreparedRuntime(input: { target: Extract; runtime: PreparedAdapterExecutionTargetRuntime; descriptor: NativeWorkspaceSyncDescriptor; + assertOwnership?: () => Promise; }): Promise { await input.runtime.restoreWorkspace(); + await input.assertOwnership?.(); const finalSnapshot = await import("@paperclipai/adapter-utils/workspace-restore-merge").then( ({ captureDirectorySnapshot }) => @@ -796,6 +798,7 @@ async function finalizePreparedRuntime(input: { finalizedAt: new Date().toISOString(), finalHostSha256, }; + await input.assertOwnership?.(); const reference = await writeDescriptor(finalizedDescriptor); await persistRunReference(input.db, input.runId, reference); await persistLeaseStamp({ @@ -993,7 +996,7 @@ export async function prepareNativeWorkspaceSync(input: { get reference() { return reference; }, - restoreWorkspace: async () => { + restoreWorkspace: async (assertOwnership) => { if (!restorePromise) { restorePromise = finalizePreparedRuntime({ db: input.db, @@ -1001,6 +1004,7 @@ export async function prepareNativeWorkspaceSync(input: { target, runtime: preparedRuntime, descriptor, + assertOwnership, }) .then((finalizedReference) => { reference = finalizedReference; @@ -1025,6 +1029,7 @@ export async function resumeNativeWorkspaceSync(input: { db: Db; runId: string; target: AdapterExecutionTarget; + assertOwnership?: () => Promise; }): Promise { if (input.target.kind !== "remote" || input.target.transport !== "sandbox") { throw new Error("workspace_sync_out_unrecoverable"); @@ -1049,6 +1054,7 @@ export async function resumeNativeWorkspaceSync(input: { ) { throw new Error("workspace_sync_out_unrecoverable"); } + await input.assertOwnership?.(); if ( existing.descriptor.state === "finalized" && existing.descriptor.finalHostSha256 @@ -1083,6 +1089,7 @@ export async function resumeNativeWorkspaceSync(input: { target: input.target, runtime, descriptor: existing.descriptor, + assertOwnership: input.assertOwnership, }); return true; } diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 705297ac57..10ab3d9824 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -1,3 +1,4 @@ +import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared"; import { settleSlackConversation } from "../slack-conversation-lifecycle.js"; import { externalConversationStateSql } from "../slack-conversation-state.js"; import { executionRetryAccounting } from "../execution-recovery-attempt.js"; @@ -3437,7 +3438,11 @@ export function recoveryService( // A queued comment or healthy child cannot establish what the stopped // provider already did. Only execution reconciliation can clear this hold. - if (requiresExecutionReconciliation(action.cause)) { + if (requiresExecutionReconciliation(action.cause) + || isNativeWorkspaceExportRepairCause(action.cause) + || action.cause === "native_workspace_sync_out_unsafe_archive") { + // A queued wake or healthy child does not export this accepted result. + // Only its native finalizer or an explicit board disposition can settle it. result.skipped += 1; continue; } diff --git a/server/src/services/sandbox-provider-runtime.ts b/server/src/services/sandbox-provider-runtime.ts index ad0d690f48..955954c608 100644 --- a/server/src/services/sandbox-provider-runtime.ts +++ b/server/src/services/sandbox-provider-runtime.ts @@ -97,6 +97,8 @@ export interface SandboxProvider { probe(config: SandboxEnvironmentConfig): Promise; acquireLease(input: AcquireSandboxLeaseInput): Promise; resumeLease(input: ResumeSandboxLeaseInput): Promise; + /** Explicit stop-only support. Absence must never fall back to release/destroy. */ + stopLease?(input: DestroySandboxLeaseInput): Promise<{ providerLeaseId: string; state: "stopped" }>; releaseLease(input: ReleaseSandboxLeaseInput): Promise; destroyLease(input: DestroySandboxLeaseInput): Promise; matchesReusableLease(input: { @@ -181,6 +183,14 @@ class FakeSandboxProvider implements SandboxProvider { }; } + async stopLease(input: DestroySandboxLeaseInput): Promise<{ providerLeaseId: string; state: "stopped" }> { + assertProviderConfig(this.provider, input.config); + if (!input.providerLeaseId?.startsWith("sandbox://fake/")) throw new Error("Fake sandbox stop needs its exact allocation."); + // The fake provider owns no process or filesystem; its explicit stop receipt + // models the lifecycle without invoking its ordinary release or destroy. + return { providerLeaseId: input.providerLeaseId, state: "stopped" }; + } + async releaseLease(): Promise { return; } diff --git a/server/src/services/sandbox-stop-and-retain.ts b/server/src/services/sandbox-stop-and-retain.ts new file mode 100644 index 0000000000..e9823dcabd --- /dev/null +++ b/server/src/services/sandbox-stop-and-retain.ts @@ -0,0 +1,95 @@ +import { randomUUID } from "node:crypto"; +import { and, eq, sql, type SQL } from "drizzle-orm"; +import { environmentLeases, type Db } from "@paperclipai/db"; +import { isBuiltinSandboxProvider } from "./sandbox-provider-runtime.js"; +import { remoteTerminationReceipt } from "./remote-execution-termination.js"; +import { hasNativeWorkspaceExportResume, readNativeWorkspaceExportResume, settleNativeWorkspaceExportResume } from "./native-runtime/native-workspace-export-resume.js"; + +type Lease = Pick; +export const SANDBOX_STOP_AND_RETAIN_KEY = "sandboxStopAndRetain"; + +export function hasStopOnlyCleanup(lease: Pick): boolean { + return hasNativeWorkspaceExportResume(lease) + || Object.prototype.hasOwnProperty.call(lease.metadata ?? {}, SANDBOX_STOP_AND_RETAIN_KEY); +} + +export function stopOnlyCleanupKey(lease: Pick) { + return hasNativeWorkspaceExportResume(lease) ? "nativeWorkspaceExportResume" : SANDBOX_STOP_AND_RETAIN_KEY; +} + +export function readStopOnlyCleanup(lease: Lease) { + if (hasNativeWorkspaceExportResume(lease)) return readNativeWorkspaceExportResume(lease); + const value = lease.metadata?.[SANDBOX_STOP_AND_RETAIN_KEY]; + if (!value || typeof value !== "object" || Array.isArray(value)) return null; + const intent = value as Record; + if (intent.schema !== "paperclip.sandbox-stop-and-retain.v1" + || intent.companyId !== lease.companyId || intent.runId !== lease.heartbeatRunId || !lease.heartbeatRunId + || intent.leaseId !== lease.id || intent.provider !== lease.provider || !lease.provider + || intent.providerLeaseId !== lease.providerLeaseId || !lease.providerLeaseId + || typeof intent.requestId !== "string" || !intent.requestId) return null; + if (intent.builtinProvider !== undefined) { + if (intent.builtinProvider !== lease.provider || !isBuiltinSandboxProvider(lease.provider) + || intent.pluginId !== undefined || lease.metadata?.pluginId != null || lease.metadata?.sandboxProviderPlugin) return null; + return { ...intent, requestId: intent.requestId, pluginId: null, builtinProvider: lease.provider }; + } + if (typeof intent.pluginId !== "string" || !intent.pluginId || intent.pluginId !== lease.metadata?.pluginId) return null; + return { ...intent, requestId: intent.requestId, pluginId: intent.pluginId, builtinProvider: null }; +} + +/** Persist before provider dispatch so an older worker or a restart cannot turn + * an explicit stop into ordinary destructive release. Invalid pins stay pending. */ +export async function prepareSandboxStopAndRetain(db: Db, lease: Lease) { + const requestId = randomUUID(), now = new Date(); + const intent = { schema: "paperclip.sandbox-stop-and-retain.v1", requestId, + companyId: lease.companyId, runId: lease.heartbeatRunId, leaseId: lease.id, + provider: lease.provider, providerLeaseId: lease.providerLeaseId, + ...(lease.provider && isBuiltinSandboxProvider(lease.provider) && !lease.metadata?.sandboxProviderPlugin && lease.metadata?.pluginId == null + ? { builtinProvider: lease.provider } : { pluginId: lease.metadata?.pluginId }) }; + const [updated] = await db.update(environmentLeases).set({ + status: "pending_cleanup", cleanupStatus: "failed", failureReason: "sandbox_stop_pending", releasedAt: now, updatedAt: now, + metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'remoteExecutionTermination' - 'sandboxStopAndRetainReceipt') || ${JSON.stringify({ + [SANDBOX_STOP_AND_RETAIN_KEY]: intent, pendingCleanupAttemptId: requestId, + pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0, + pendingCleanupRetryAfterMs: 0, pendingCleanupRetryAttempts: 0, pendingCleanupRetryCapWarned: false, + })}::jsonb`, + }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId), + eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.status, "active"), + sql`${environmentLeases.provider} is not distinct from ${lease.provider}`, + sql`${environmentLeases.providerLeaseId} is not distinct from ${lease.providerLeaseId}`, + sql`${environmentLeases.metadata} is not distinct from ${lease.metadata === null ? null : JSON.stringify(lease.metadata)}::jsonb`, + )).returning(); + return updated ?? null; +} + +export async function settleStopOnlyCleanup(db: Db, lease: Lease, options: { attemptId: string; receipt?: unknown }) { + if (hasNativeWorkspaceExportResume(lease)) return settleNativeWorkspaceExportResume(db, lease, options); + const intent = readStopOnlyCleanup(lease); + if (!intent) return null; + const receipt = remoteTerminationReceipt(lease, options.receipt), stopped = receipt?.state === "stopped"; + const now = new Date(); + const removeIntent: SQL = stopped ? sql`- 'sandboxStopAndRetain'` : sql``; + const [updated] = await db.update(environmentLeases).set({ + status: stopped ? "released" : "pending_cleanup", cleanupStatus: stopped ? "success" : "failed", + failureReason: stopped ? null : "sandbox_stop_pending", + releasedAt: now, lastUsedAt: now, updatedAt: now, + metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination' ${removeIntent}) || ${JSON.stringify({ + ...(stopped ? { remoteExecutionTermination: receipt, sandboxStopAndRetainReceipt: { + schema: "paperclip.sandbox-stop-and-retain-receipt.v1", requestId: intent.requestId, + companyId: lease.companyId, runId: lease.heartbeatRunId, leaseId: lease.id, + provider: lease.provider, providerLeaseId: lease.providerLeaseId, + ...(intent.pluginId ? { pluginId: intent.pluginId, method: "environmentStopLease" } + : { builtinProvider: lease.provider, method: "builtin.stopLease" }), + confirmedAt: receipt.confirmedAt, + } } : {}), + pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0, + })}::jsonb`, + }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId), + eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.provider, lease.provider!), + eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), eq(environmentLeases.status, "pending_cleanup"), + sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.attemptId}`, + sql`${environmentLeases.metadata}->'sandboxStopAndRetain' = ${JSON.stringify(lease.metadata?.[SANDBOX_STOP_AND_RETAIN_KEY])}::jsonb`, + sql`${environmentLeases.metadata}->>'pluginId' is not distinct from ${intent.pluginId}`, + )).returning(); + return updated ?? null; +} diff --git a/ui/src/api/issues.ts b/ui/src/api/issues.ts index 7d8bf562a3..b4f0671acc 100644 --- a/ui/src/api/issues.ts +++ b/ui/src/api/issues.ts @@ -271,6 +271,8 @@ export const issuesApi = { `/issues/${id}/stalled-review-decision`, data, ), + retryWorkspaceExport: (id: string, data: { actionId: string; runId: string; repairNote: string }) => + api.post<{ runId: string; resultId: string; leaseId: string; status: "queued" }>(`/issues/${id}/recovery-actions/retry-workspace-export`, data), resolveRecoveryAction: ( id: string, data: { diff --git a/ui/src/components/IssueRecoveryActionCard.test.tsx b/ui/src/components/IssueRecoveryActionCard.test.tsx index 75e4b3330c..88e6cc7b13 100644 --- a/ui/src/components/IssueRecoveryActionCard.test.tsx +++ b/ui/src/components/IssueRecoveryActionCard.test.tsx @@ -1145,3 +1145,19 @@ describe("IssueRecoveryActionCard owner-sticky retry lineage", () => { expect(node.textContent).toContain("→ Returns to:"); }); }); + +it.each(["active", "escalated", "resolved"] as const)("does not show a historical unsafe-export repair card: %s", status => { + const node = render( {}} />); + expect(node.textContent).toBe(""); +}); + +it("requires export retry for ordinary restoration while keeping explicit board overrides", () => { + const node = render( {}} canFalsePositive />); + click(node.querySelector("[data-testid='recovery-action-resolve-trigger']")); + expect(document.body.textContent).not.toContain("Try again"); + expect(document.body.textContent).not.toContain("Mark task done"); + expect(document.body.textContent).not.toContain("Send for review"); + expect(document.body.textContent).toContain("False positive, done"); +}); diff --git a/ui/src/components/IssueRecoveryActionCard.tsx b/ui/src/components/IssueRecoveryActionCard.tsx index 7b61f04135..beb84076e1 100644 --- a/ui/src/components/IssueRecoveryActionCard.tsx +++ b/ui/src/components/IssueRecoveryActionCard.tsx @@ -1,3 +1,4 @@ +import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared"; import { useWorkspaceIsolationControls } from "@/hooks/useWorkspaceIsolationControls"; import { requiresExecutionReconciliation } from "@paperclipai/shared"; import { useMemo, useState } from "react"; @@ -1065,12 +1066,13 @@ export function IssueRecoveryActionCard({ resolved: "resolved", } satisfies Record)[cardState]; - const showResolveActions = onResolve !== undefined && cardState !== "resolved"; const visibleResolveOptions = RESOLVE_OPTIONS.filter((option) => { + if (isNativeWorkspaceExportRepairCause(action.cause) && ["todo", "done", "in_review"].includes(option.outcome)) return false; if (option.outcome === "todo" && requiresExecutionReconciliation(action.cause)) return false; if (option.boardOnly && !canFalsePositive) return false; return true; }); + const showResolveActions = onResolve !== undefined && cardState !== "resolved" && visibleResolveOptions.length > 0; const reissueBaseRef = divergence?.reissueBaseRef ?? null; const showReissueAction = workspaceIsolationControlsVisible && @@ -1117,7 +1119,7 @@ export function IssueRecoveryActionCard({ showBreakGlass || showRepairAction; - if (requiresExecutionReconciliation(action.cause)) return null; + if (requiresExecutionReconciliation(action.cause) || action.cause === "native_workspace_sync_out_unsafe_archive") return null; return (
vi.fn()); +vi.mock("../api/issues", () => ({ issuesApi: { retryWorkspaceExport: retry } })); +const action = { id: "action", cause: "native_workspace_sync_out_retry_exhausted", status: "active", updatedAt: "2026-01-01T00:00:00Z", ownerType: "board", evidence: { runId: "run" }, wakePolicy: null } as unknown as IssueRecoveryAction; +describe("WorkspaceExportRecovery", () => { + let root: Root, container: HTMLDivElement, client: QueryClient; + const onQueued = vi.fn(); + beforeEach(() => { + retry.mockReset().mockResolvedValue({ status: "queued" }); onQueued.mockReset(); + container = document.createElement("div"); document.body.append(container); root = createRoot(container); + client = new QueryClient({ defaultOptions: { mutations: { retry: false } } }); + }); + afterEach(async () => { await act(async () => root.unmount()); client.clear(); container.remove(); }); + async function mount(overrides: Partial[0]> = {}) { + await act(async () => root.render()); + } + async function enterNote() { + const textarea = container.querySelector("textarea")!; + await act(async () => { + Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!.call(textarea, "Restored provider connectivity while preserving all saved files."); + textarea.dispatchEvent(new Event("input", { bubbles: true })); + }); + } + async function submit() { + await act(async () => container.querySelector("button")!.click()); + await act(async () => { await new Promise(resolve => setTimeout(resolve, 10)); }); + } + it("requires a repair note and submits only the exact recorded run", async () => { + await mount(); expect(container.querySelector("button")!.disabled).toBe(true); + await enterNote(); await submit(); + expect(retry).toHaveBeenCalledWith("issue", { actionId: "action", runId: "run", repairNote: "Restored provider connectivity while preserving all saved files." }); + expect(onQueued).toHaveBeenCalledOnce(); + expect(container.querySelector('[role="status"]')?.textContent).toContain("agent will not repeat its work"); + }); + it("offers repair again when a queued export publishes a new permanent failure", async () => { + await mount(); await enterNote(); await submit(); + expect(container.querySelector("button")).toBeNull(); + await mount({ action: { ...action, updatedAt: "2026-01-01T00:01:00Z", wakePolicy: null } }); + expect(container.querySelector("button")?.textContent).toBe("Retry workspace export"); + }); + it("keeps unavailable-sandbox errors actionable inline", async () => { + retry.mockRejectedValueOnce(new Error("Resume the retained sandbox before retrying.")); await mount(); + await enterNote(); await submit(); + expect(container.querySelector('[role="alert"]')?.textContent).toContain("Resume the retained sandbox"); + }); + it("does not offer the action without runtime access", async () => { + await mount({ canManage: false }); expect(container.querySelector("button")).toBeNull(); + }); + it("offers saved-result export after transient retries are exhausted without claiming an unsafe link", async () => { + await mount({ action: { ...action, cause: "native_workspace_sync_out_retry_exhausted" } }); + expect(container.querySelector("button")?.textContent).toBe("Retry workspace export"); + expect(container.textContent).toContain("export failure"); + expect(container.textContent).not.toContain("unsafe link"); + await enterNote(); await submit(); + expect(retry).toHaveBeenCalledWith("issue", expect.objectContaining({ runId: "run" })); + }); + it("never asks users to repair historical unsafe exports", async () => { + await mount({ action: { ...action, cause: "native_workspace_sync_out_unsafe_archive" } }); + expect(container.textContent).toBe(""); + expect(retry).not.toHaveBeenCalled(); + }); + it("does not offer a second retry while the same export is queued", async () => { + await mount({ action: { ...action, wakePolicy: { kind: "resume_native_run" } } }); + expect(container.querySelector('[role="status"]')?.textContent).toContain("Export is queued"); expect(container.querySelector("button")).toBeNull(); + }); +}); diff --git a/ui/src/components/WorkspaceExportRecovery.tsx b/ui/src/components/WorkspaceExportRecovery.tsx new file mode 100644 index 0000000000..23ae74319b --- /dev/null +++ b/ui/src/components/WorkspaceExportRecovery.tsx @@ -0,0 +1,43 @@ +import { useId, useState } from "react"; +import { useMutation } from "@tanstack/react-query"; +import type { IssueRecoveryAction } from "@paperclipai/shared"; +import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared"; +import { issuesApi } from "../api/issues"; +import { Button } from "./ui/button"; +import { Label } from "./ui/label"; +import { Textarea } from "./ui/textarea"; + +/** Copyback repair preserves the accepted result and never wakes the provider. */ +export function WorkspaceExportRecovery({ issueId, action, canManage, onQueued }: { + issueId: string; action: IssueRecoveryAction | null; canManage: boolean; onQueued: () => void; +}) { + const noteId = useId(); + const [repairNote, setRepairNote] = useState(""); + const [queuedActionVersion, setQueuedActionVersion] = useState(null); + const retry = useMutation({ + mutationFn: () => issuesApi.retryWorkspaceExport(issueId, { + actionId: action!.id, runId: action!.evidence.runId as string, repairNote: repairNote.trim(), + }), + onSuccess: () => { setQueuedActionVersion(String(action!.updatedAt)); onQueued(); }, + }); + if (!action || !isNativeWorkspaceExportRepairCause(action.cause) || action.ownerType !== "board" + || !["active", "escalated"].includes(action.status) || typeof action.evidence.runId !== "string") return null; + const queued = queuedActionVersion === String(action.updatedAt) || action.wakePolicy?.kind === "resume_native_run"; + return
+

Workspace export needs repair

+ {queued ?

Export is queued for the saved result. The agent will not repeat its work.

: <> +

{"Automatic workspace export retries stopped. Inspect the export failure, restore provider or destination availability, and preserve the saved files in the retained sandbox. Retry export here when the cause is resolved."}

+ {canManage ? <> + +