From 7351a9d74ff8e413a1ea1bfd47a1e3c26a81cc06 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 5 Nov 2026 12:20:10 -0600 Subject: [PATCH] fix(native): restore retired backups after warm lease handoffs (#19299) ## Thinking Path > - Paperclip manages AI agents and keeps their work across turns. > - A native session can keep the same sandbox through several warm lease handoffs. > - Each handoff keeps the old lease row and records matching receipts on both owners. > - The retired-backup guard required exactly one row for a provider allocation. > - That rule rejected valid history after Paperclip destroyed the sandbox. > - This change uses the existing handoff proof to select the final owner. > - A verified backup can then restore without discarding ownership checks. ## Linked Issues or Issue Description Refs #19208. **What happened?** A follow-up received a new sandbox after its old sandbox was destroyed. The old allocation had a normal warm lease handoff. Both lease rows remained in PostgreSQL. The guard rejected them with `runner_harness_state_mismatch: backup_without_reusable_lease`. Its later same-run retry failed with `native_remote_recovery_lease_mismatch`. **Expected behavior** Restore the verified backup when one final owner has a valid destroyed receipt and every earlier owner has a matching handoff receipt. Keep uncertain ownership blocked. **Steps to reproduce** 1. Complete a native warm turn. 2. Hand the reusable sandbox to another turn. 3. Complete that turn and save its backup. 4. Close the workspace and destroy the sandbox. 5. Send a follow-up. The new allocation must restore the saved session. **Paperclip version or commit** Reproduced in the PostgreSQL regression on the parent of this change. The affected deployed build also contains #19208. ## What Changed - Select the single final lease owner. Validate all predecessor rows with `retiredReusableLeasePredecessors`. - Keep the destroyed receipt, exact backup stamp, and newer-provider-work checks. - Add database regressions for one and three handoffs. Reject missing or changed receipts, an active predecessor, missing history, an unrelated owner, a stopped source, another backup, and later provider work. - Document the valid lease-history shape. ## Verification - Red before the source change: the two valid-handoff cases returned false. The other 16 database cases passed. - Green after the change: all 18 database cases passed. They also passed on current head `d16e098b8c2e6c0dda70984c1df21e3dc792ab34`. - Command: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-retired-backup-source.integration.test.ts`. - The database, executor, retry-admission, and lease suites passed: 4 files, 1,308 tests. - `pnpm -r typecheck` and `pnpm build` passed. The build stamp names head `4ee733c3a979436196522f36537ff182e9df5398`. The full local suite stopped in chat shard 3 after a 15-second timeout in `materializes direct external-chat finals once and accepts the next turn without agent bookkeeping writes`. Earlier shards passed 702 cases; that shard passed 303 other cases. The isolated test passes on this branch and unchanged master `10a7d039c8efeac0fd655c24970df5ae3658414b`. All current-head CI chat shards also passed. The earlier full-suite attempt remains failed; its timeout cause is not established. No test or timeout was changed. CI is waiting for its last Runner job. - Current head `d16e098b8c2e6c0dda70984c1df21e3dc792ab34` removes four comment lines from the previously tested head. Its 18 database cases passed. Greptile reviewed this head at 5/5 with no actionable findings: https://github.com/paperclipai/paperclip/pull/19299#issuecomment-6665446098. CI remains pending. - Live inspection found the matching predecessor/successor receipts and the final owner's destroyed receipt. This was read-only evidence from the affected build. - A live provider run at the fixed revision has not been exercised. Local Daytona credentials and an immutable test image are unavailable. No deployment or original-task retry is claimed. - Initial wider tests could not resolve the unbuilt plugin SDK. The SDK build is now available. A sandboxed database launch could not bind loopback; the same regression passed with the disposable database outside that restriction. ## Risks - A wrong owner could allow an old backup to rewind a session. The shared validator must account for every lease row and match both sides of each handoff. - Live, unrelated, incomplete, or conflicting history stays blocked. The existing backup integrity and provider-progress checks remain. - No migration, permission, model, trust, timeout, or retry-policy change is included. - The original task still needs the fixed build and normal recovery. Hosted acceptance remains unverified. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code execution, and tool use. The session does not expose the exact model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 5 +- ...-retired-backup-source.integration.test.ts | 62 ++++++++++++++++++- .../native-retired-backup-source.ts | 8 ++- 3 files changed, 68 insertions(+), 7 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 65cf2148a2b..6d50ff984fd 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -2074,7 +2074,10 @@ backup. Paperclip restores that backup into the new sandbox only when all of these are true: the backup's source lease is expired or failed with a provider `destroyed` termination receipt, the source lease's backup stamp names this exact backup manifest, and no provider event for the session was recorded after -the backup completed. Any other state is ambiguous. The run then fails closed +the backup completed. Normal warm handoffs can leave multiple lease records +for that allocation. Restore requires one final owner and the existing matching +predecessor/successor receipts for every earlier owner. Unrelated, live, or +incompletely linked owners remain ambiguous. The run then fails closed with `runner_harness_state_mismatch: backup_without_reusable_lease`. A per-turn sandbox turn can finish through recovery without a Runner diff --git a/server/src/services/native-runtime/native-retired-backup-source.integration.test.ts b/server/src/services/native-runtime/native-retired-backup-source.integration.test.ts index f725280f4ba..7a8ed9f1322 100644 --- a/server/src/services/native-runtime/native-retired-backup-source.integration.test.ts +++ b/server/src/services/native-runtime/native-retired-backup-source.integration.test.ts @@ -1,7 +1,8 @@ import { randomUUID } from "node:crypto"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; -import { agents, companies, createDb, environmentLeases, heartbeatRunEvents, heartbeatRuns } from "@paperclipai/db"; +import { agents, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, issues } from "@paperclipai/db"; import { recoveryPostgresSupport, startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js"; +import { createReusableLeaseHandoff } from "../environment-lease-handoff.js"; import { retiredSourceAuthorizesNativeBackupRestore } from "./native-retired-backup-source.js"; const describeDb = recoveryPostgresSupport.supported ? describe : describe.skip; @@ -19,7 +20,8 @@ describeDb("retired native backup source authority", () => { const completedAt = new Date("2026-11-05T05:02:50.000Z"); async function fixture(options: { termination?: "destroyed" | "stopped" | null; status?: string; stampSha?: string; - providerEventAt?: Date; secondLease?: boolean } = {}) { + providerEventAt?: Date; secondLease?: boolean; handoffs?: number; + handoffFault?: "missing_receipt" | "changed_receipt" | "active_predecessor" | "missing_predecessor" } = {}) { const companyId = randomUUID(), agentId = randomUUID(), sessionId = randomUUID(), runnerInstanceId = randomUUID(); const sourceRunId = randomUUID(), currentRunId = randomUUID(), leaseId = randomUUID(), providerLeaseId = randomUUID(); await db.insert(companies).values({ id: companyId, name: "Retired source", issuePrefix: `R${companyId.slice(0, 5)}` }); @@ -37,7 +39,43 @@ describeDb("retired native backup source authority", () => { runId: sourceRunId, leaseId: id, provider: "daytona", providerLeaseId, state: options.termination ?? "destroyed", confirmedAt: new Date().toISOString() } }), } }); - await db.insert(environmentLeases).values(lease(leaseId)); + const sourceLease = lease(leaseId); + if (options.handoffs) { + const environmentId = randomUUID(), issueId = randomUUID(), workspaceId = randomUUID(); + await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Retained session" }); + await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId, + mode: "shared_workspace", strategyType: "project_primary", name: "Retained workspace" }); + const history: Array = []; + for (let index = 0; index <= options.handoffs; index++) { + const ownerRunId = index === options.handoffs ? sourceRunId : randomUUID(); + if (ownerRunId !== sourceRunId) await db.insert(heartbeatRuns).values({ id: ownerRunId, companyId, agentId, + status: "succeeded", runtimeMode: "native", nativeSessionId: sessionId, + invocationSource: "assignment", triggerDetail: "system" }); + const acquiredAt = new Date(completedAt.getTime() - (options.handoffs - index + 1) * 60_000); + const row = { ...sourceLease, id: index === options.handoffs ? leaseId : randomUUID(), + heartbeatRunId: ownerRunId, environmentId, issueId, executionWorkspaceId: workspaceId, acquiredAt, + leasePolicy: "reuse_by_environment", metadata: { ...sourceLease.metadata, agentId } }; + if (index < options.handoffs) { + const { nativeHarnessBackup: _backup, remoteExecutionTermination: _termination, ...metadata } = row.metadata; + history.push({ ...row, metadata }); + } else history.push(row); + } + for (let index = 1; index < history.length; index++) { + const previous = history[index - 1]!, current = history[index]!; + const handoff = createReusableLeaseHandoff(previous as typeof environmentLeases.$inferSelect, + current as typeof environmentLeases.$inferSelect); + previous.releasedAt = current.acquiredAt; + previous.metadata = { ...previous.metadata, reusableLeaseSuccessor: handoff }; + current.metadata = { ...current.metadata, reusableLeasePredecessor: handoff }; + } + if (options.handoffFault === "missing_receipt") delete history[0]!.metadata!.reusableLeaseSuccessor; + if (options.handoffFault === "changed_receipt") history[0]!.metadata!.reusableLeaseSuccessor = { + ...(history[0]!.metadata!.reusableLeaseSuccessor as Record), successorRunId: randomUUID() }; + if (options.handoffFault === "active_predecessor") history[0]!.status = "active"; + if (options.handoffFault === "missing_predecessor") history.shift(); + await db.insert(environmentLeases).values(history); + } else await db.insert(environmentLeases).values(sourceLease); if (options.secondLease) await db.insert(environmentLeases).values(lease(randomUUID())); // The backup turn's own provider events precede the checkpoint. await db.insert(heartbeatRunEvents).values({ companyId, runId: sourceRunId, agentId, seq: 1, eventType: "turn.started", @@ -54,6 +92,24 @@ describeDb("retired native backup source authority", () => { await expect(fixture()).resolves.toBe(true); }); + it.each([1, 3])("authorizes a destroyed allocation after %i verified warm lease handoffs", async (handoffs) => { + await expect(fixture({ handoffs })).resolves.toBe(true); + }); + + it.each(["missing_receipt", "changed_receipt", "active_predecessor", "missing_predecessor"] as const)( + "fails closed for a warm lease chain with %s", async (handoffFault) => { + await expect(fixture({ handoffs: 2, handoffFault })).resolves.toBe(false); + }); + + it.each([ + ["an unrelated owner", { secondLease: true }], + ["a stopped source", { termination: "stopped" as const }], + ["another checkpoint", { stampSha: `sha256:${"c".repeat(64)}` }], + ["later provider work", { providerEventAt: new Date(completedAt.getTime() + 60_000) }], + ])("keeps the backup restore held after warm handoffs with %s", async (_label, options) => { + await expect(fixture({ handoffs: 2, ...options })).resolves.toBe(false); + }); + it.each([ ["the provider only stopped the source", { termination: "stopped" as const }], ["no termination receipt exists", { termination: null }], diff --git a/server/src/services/native-runtime/native-retired-backup-source.ts b/server/src/services/native-runtime/native-retired-backup-source.ts index 865a2f43322..bd250afc76d 100644 --- a/server/src/services/native-runtime/native-retired-backup-source.ts +++ b/server/src/services/native-runtime/native-retired-backup-source.ts @@ -1,6 +1,7 @@ import { createHash } from "node:crypto"; import { and, eq, gt, inArray, ne } from "drizzle-orm"; import { environmentLeases, heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db"; +import { retiredReusableLeasePredecessors } from "../environment-lease-handoff.js"; import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js"; import { readNativeHarnessBackupStamp } from "./native-harness-backup-stamp.js"; @@ -42,9 +43,10 @@ export async function retiredSourceAuthorizesNativeBackupRestore(db: Db, input: )); const source = leases.filter(lease => lease.companyId === input.companyId && lease.providerLeaseId === input.manifest.sourceProviderLeaseId); - // One provider allocation maps to one host lease. Anything else is ambiguous. - if (source.length !== 1) return false; - const lease = source[0]!; + const tips = source.filter(lease => !Object.hasOwn(lease.metadata ?? {}, "reusableLeaseSuccessor")); + if (tips.length !== 1) return false; + const lease = tips[0]!; + if (!retiredReusableLeasePredecessors(lease, source.filter(row => row.id !== lease.id))) return false; const termination = lease.metadata?.remoteExecutionTermination as { state?: unknown } | undefined; // A released lease can still be selected for resume; only a retired one is final. if (!["expired", "failed"].includes(lease.status) || !hasRemoteTerminationReceipt(lease)