diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.md index 12f8dada5c4..8334e3b1685 100644 --- a/doc/idle-sleep-safety.md +++ b/doc/idle-sleep-safety.md @@ -133,6 +133,17 @@ fingerprint and RPC company scope for later read-only verification. Retained allocations, unfinished runs/accounting/retries, export intents and live setup credentials continue to block sleep. +For a reused remote allocation, a completed predecessor can resolve termination +through the final owner's destruction receipt only when the full host-recorded +lease handoff chain matches. Company, agent, task, workspace, environment, provider +and allocation identities must agree. A stopped successor, broken handoff or +unrelated owner grants no cleanup authority. This resolution does not rewrite +historical leases, execute remotely or claim that uncollected files were saved. +The normal agent-file recovery worker retains the no-save outcome and completes +cleanup after confirmed destruction. A completed cleanup receipt with a stale +retry deadline is reconciled once through the existing deadline worker. Clearing +that deadline compares the receipt so a concurrent cleanup obligation is kept. + Pending human-only connection setup can wait asleep when no active enabled connection of the requested purpose exists. Available access reblocks sleep so the existing reconciliation path can run; answering a setup card continues to diff --git a/doc/plans/2026-10-31-zero-downtime-web-deploys.md b/doc/plans/2026-10-31-zero-downtime-web-deploys.md index 71e4bf658b2..842a6e5854d 100644 --- a/doc/plans/2026-10-31-zero-downtime-web-deploys.md +++ b/doc/plans/2026-10-31-zero-downtime-web-deploys.md @@ -1,10 +1,152 @@ # Zero-downtime web deployments -Date: 2026-10-31. Status (2026-11-01 15:39 UTC): existing-volume staging enrollment -and manual sleep/wake preserve saved files and read/save behavior. The full local -active-agent journey now passes on `cc23019691`; hosted browser continuity and -the ten-deploy staging campaign remain unqualified. Natural database autosuspension -passes after abandoned test-resource cleanup. No production canary yet. +Date: 2026-10-31. Status (2026-11-02 17:23 UTC): Core #18554 is merged as +`1a6017d6460267ad89d7e1784e2b521c28d8e9ac`. The ten-roll and native-agent +proof used Cloud `461c4aea28`. Separately reviewed Cloud #1091/#1092 subsequently +reached staging as `89dd0cfec2`; their changes concern workspace cleanup and +Agent Computer resource fencing. Railway, fleet and deployment workflow paths +are unchanged. Production remains on its original controller revision; the +next manual controller deployment must use current reviewed master. +Both immutable Cloud distributions built from that fixed source passed packaged +Linux rehearsals through the Railway wrapper: 260 reads, 53 saves, zero failures, +legacy-volume migration, an unready replacement, and removal of both predecessor +homes. Each fixture cleaned its containers. These rehearsals do not exercise an +active agent or an actual browser. The two distributions have distinct image +digests but identical application source; their compatible rollback qualifies +artifact replacement, not compatibility between different source revisions. + +The final Core PR has two 5/5 reviews and all required checks passed. One legacy +browser CI shard failed because its deferred-wake fixture observed no queued wake; +the unchanged case passed locally on isolated PostgreSQL and its single CI retry +passed. The original failure remains in the PR evidence. The full local live-agent +journey passed on `cc23019691` before the final non-suspended dispatch refinement; +that refinement passed its three focused Linux regressions, server typecheck, +build and final CI. Do not describe the earlier live run as a rerun of the merge. + +The fixed canonical package baseline passed on the pinned migrated staging fixture +using the allowed one-time maintenance path. An actual Chromium tab running the +older UI reconnected, retained an unsaved title edit, and saved it through the new +server; an independent API read confirmed the title. That old tab opened the +Agents page, and all four previously captured asset hashes still match after the +old deployment was removed. Baseline maintenance interruptions are preserved and +are not counted as zero-downtime qualification. + +The first ten-roll attempt was cancelled at step one, before Railway replacement. +The controller's pre-existing hot-upgrade allowlist named an older test stack, +so it drained rather than negotiated preservation of the new fixture's active run. +Continuous read/save and browser probes remained healthy, but this is NOT active +execution continuity evidence. The original command completed normally once; its +downloaded output bytes and native terminal were independently verified. The +owned fixture has been added to the existing allowlist without changing fleet +defaults. The normal staging controller workflow is applying that correction. +The rollout harness now checks both canary policy and positive tenant readiness +before submitting a campaign. The cancelled attempt remains failed evidence, +with no automatic campaign retry. +The configuration deployment then failed before Railway submission when Sentry +release registration failed or timed out. Its source checks passed; the original +logs are retained. One unchanged whole-workflow retry preserves attempt-bound +preflight requirements. That retry failed at the same pre-Railway Sentry request. +A read-only Sentry lookup confirms the release exists (HTTP 200). The canary-only +configuration was then applied by redeploying the existing successful controller +deployment: the source and replacement image digests are identical, and health +attests unchanged Cloud `461c4aea28`. Both failed workflow attempts are retained; +no Sentry gate was disabled and no new application release was built in this +configuration recovery. The new active-command attempt and gated sequence are +starting with the corrected controller. The first preflight stopped before task +admission on a missing optional plan's 404. `useIssuePlanDocument` explicitly +maps that response to null; an independent read confirmed no plan on the owned +completed task. The raw response remains recorded and the browser gate permits +only that verified path. The configuration redeploy's actual browser reconnect +was 1.22 seconds, with no probe read/save failure. + +All ten Railway replacements are independently qualified. Final continuous proof +passed 12,521 reads and 1,566 saves, with every acknowledged save present exactly +once and all 34 file hashes matching. There were zero HTTP failures. Actual +browser reconnects across the ten deployments ranged from 1.16 to 1.29 seconds. +The old UI assets and original agent artifact remain readable after deployment +ten and provider cleanup. Both traffic observers have stopped cleanly. The first +replaced the webserver while the original eight-minute remote command was +running. Its one execution ID started before replacement and completed afterward +with exit zero. The second replaced the server after completion; a later follow-up +retained the same native/provider session and runner, with accepted native +terminals and finalized accounting for both runs. The 91-byte artifact was +downloaded and independently checked for exactly one start and completion. +Actual browser reconnects were 1.28, 1.16 and 1.25 seconds; continuous reads and +saves have no failures so far. Previously unopened Projects and Skills pages rendered in the +retained old tab. A title draft survived the third cutover and was saved through +the UI, then independently read back from the replacement. + +The harness stopped after deployment three on another owned task's expected +missing optional-plan 404. The UI explicitly treats this as null. Both exact task +paths were independently checked, and the provider deployment, package eligibility, +serving identity, removal of the predecessor, browser timing and continuous probe +were revalidated. The failed original receipt remains unchanged alongside an +explicit classification review. The remaining seven replacements continue without +redeploying or retrying deployment three. An announcement also obscured one +sidebar click; its ordinary Dismiss control resolved that retained interaction +timeout. A separate large run-log diagnostic timed out; a narrower public-events +read succeeded and proves the original command's single execution identity. +A later bounded 64-KiB log read succeeded from shared storage after both the +originating webserver and remote sandbox were removed. + +The native recovery event records SIGTERM without a retry run, then adoption of +the same runner at controller generation 2. Each of the two runs has exactly one +accepted result and terminal event. The independent accounting inspector has no +findings or pending work; the task totals 52.941 cents across two runs. Including +the earlier no-replacement attempt, staging has three estimated cost events totaling +113.84284 cents. The temporary agent is paused, its secret removed and the prior +company budget restored; all three provider resources are independently confirmed absent. The first delete +was refused while the sandbox stopped; that failure remains alongside the +successful cleanup after a fresh stopped-state observation. Both owned local +PostgreSQL/object-storage containers are removed; test evidence is retained. + +Deployment nine also completed, but the local test reader observed the browser +snapshot while its writer had truncated the file. The campaign response was +already parsed and successful; the only non-atomic JSON read in that failing +checkpoint is the local browser report. A concurrent-write regression reproduced +the problem. The reader now retries only local JSON syntax errors for at most +100 ms; persistent corruption still fails, and HTTP/acceptance failures are never +retried. Deployment nine was independently revalidated without redeployment. +Both raw failed sequence receipts and their explicit classification reviews remain; +the ten provider deployments themselves have not been retried. The final staging +driver stopped at the harness failure and is being resumed from the retained +sequence. The normal UI's 256,000-byte run-log read also passes after replacement. + +The first repair check stopped before mutation because an unrelated staging +campaign was active. That campaign has finished, and the pinned canary remains +active and unlocked on the qualified final image. Both one-target configuration repair campaigns passed: backup policy was +restored, the temporary marker removed, and durable files/shared assets remained +enabled on the same volume-free image. The current-image eight-minute natural-idle check failed with all browser pages and +traffic probes closed. Bounded SQL diagnosis found three completed native runs +still in agent-file collection recovery. Normal archive of both completed test +workspaces independently passed close-readiness and preserved saved workspace +files. It produced provider-confirmed destruction receipts for the latest leases. +Two copies then completed no-save cleanup, but retained stale retry deadlines; +the earlier run of a reused sandbox still lacks direct termination evidence even +though its validated successor allocation is destroyed. These are source recovery +gaps, not an idle-proof pass. No historical rows were manually rewritten. The +failed idle and cancelled browser lifecycle receipts remain unchanged. Fix +ownership belongs to remote termination proof resolution and agent-directory +cleanup scheduling; no polling cadence or sleep-safety check should be relaxed. + +The isolated source branch is now based on current master `97c4dc969e`. The +recovery fix resolves the destroyed allocation through the existing validated +handoff-chain contract at read time; it does not stamp historical leases or infer +stop from task status. Agent-file cleanup also reconciles stale retry deadlines +with a receipt comparison. Both owning service suites passed 169 tests, and the +server typecheck passed. The first new deadline test caught a public guard that +skipped settlement; that failed attempt is retained and the corrected five-case +selection passed. The final seven-case selection passed, including a concurrent receipt change and +cross-company rejection. Repository-wide checks and PR review are pending. +This source has not been deployed and is not covered by the prior ten-roll proof. + +Next: review and ship the cleanup fix, complete current-image idle/sleep/wake, then the +isolated production forward/rollback canary. No production deployment has occurred. +Dotta completed the authorized GitHub sign-in; the production browser is back on +the ordinary organization list. The production canary still waits for staging. Its new organization form is +prepared but unsubmitted. Production remains on its original controller revision, +with no campaigns, the single-volume default and continuous deployment disabled. Earlier +staging sleep/wake and natural database-idle proofs remain valid for their image. Core #18543 merged as `7751fc3107` after 55 successful CI checks, Greptile 5/5 and no unresolved threads. It supplies runtime admission, immutable-cache @@ -202,7 +344,7 @@ Measure one-time setup separately from repeated attempts. Keep failed evidence. | Item | State | Next action | | --- | --- | --- | | Prior prerequisites | #18403 and all five storage workstreams merged | Retain the ephemeral filesystem contract | -| Core implementation | #18518, #18526, #18529, #18530, #18531, #18533, #18543 and #18552 merged; #18554 reviewed at 5/5 with CI pending; local packaged migration, browser overlap, cold restoration and durable command proofs recorded below | Publish and qualify the exact fixed release after CI/review merge | +| Core implementation | #18518, #18526, #18529, #18530, #18531, #18533, #18543 and #18552 merged; #18554 merged with all required checks and 5/5 reviews; both exact fixed images passed packaged migration and deletion; active hosted output verified | Finish hosted continuation and accounting reconciliation | | Cloud implementation | #1096, #1098, #1099, #1100, #1101 and #1102 merged; #1102 serving on staging | Verify repeated Railway overlap behavior after Core recovery qualification | | Staging | Original enrollment complete; migrated data and signing key preserved; volume detached and retained | Sleep/wake and natural database idle pass; verify repair and ten app-only replacements with browser continuity | | Production | No canary mutation performed | Proceed only after the staging and active-agent acceptance gates pass | diff --git a/server/src/__tests__/agent-directory-working-copies.test.ts b/server/src/__tests__/agent-directory-working-copies.test.ts index fcf3c5ebd6a..415781328d3 100644 --- a/server/src/__tests__/agent-directory-working-copies.test.ts +++ b/server/src/__tests__/agent-directory-working-copies.test.ts @@ -16,7 +16,7 @@ import { createHash, randomUUID } from "node:crypto"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { eq, sql } from "drizzle-orm"; import { agentFileStore, fileHash, inspectAgentFile, inspectAgentDirectory, snapshotAgentFiles, MAX_AGENT_FILE_BYTES, MAX_AGENT_DIRECTORY_BYTES, MAX_AGENT_DIRECTORY_ENTRIES } from "../services/agent-file-store.js"; -import { agents, companies, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, environmentLeases, environments, agentInstructionWorkingCopies, agentInstructionRevisions, agentInstructionHeads, createDb, fileStorageDb } from "@paperclipai/db"; +import { agents, companies, issues, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, environmentLeases, environments, agentInstructionWorkingCopies, agentInstructionRevisions, agentInstructionHeads, createDb, fileStorageDb } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js"; import { agentInstructionWorkingCopyService, instructionWorkingCopyGuidance } from "../services/agent-instruction-working-copies.js"; @@ -24,6 +24,7 @@ import { resolveManagedInstructionsRoot } from "../services/agent-instructions.j import { buildNativeRuntimeContext } from "../services/native-runtime/runtime-context.js"; import type { EnvironmentRuntimeService } from "../services/environment-runtime.js"; import { remoteTerminationReceipt } from "../services/remote-execution-termination.js"; +import { createReusableLeaseHandoff } from "../services/environment-lease-handoff.js"; import { AgentDirectoryReuseInvalidatedError, agentDirectoryWorkingCopyService } from "../services/agent-directory-working-copies.js"; import { withDirectoryMergeLock } from "@paperclipai/adapter-utils/workspace-restore-merge"; import { heartbeatRunEvents } from "@paperclipai/db"; @@ -123,6 +124,44 @@ describe("persistent agent directories", () => { for (const copy of [active, completed, transferred]) expect((await fs.stat(copy.localRoot)).isDirectory()).toBe(true); } finally { await worker.stop(); } }); + it("retires a stale retry on an already cleaned receipt after restart without filesystem work", async () => { + const copy = await run(); + await db.update(agentInstructionWorkingCopies).set({ state: "unavailable", processStoppedAt: new Date(), + nextAttemptAt: new Date(), receipt: { schema: "paperclip.agent-files.v1", cleanupPending: false } }) + .where(eq(agentInstructionWorkingCopies.runId, copy.runId)); + clock(); const worker = coordinator(), removes = vi.spyOn(fs, "rm"); + try { + await agentInstructionWorkingCopyService(db).registerDirectoryCleanupWork(worker, () => true, async () => true).ready; + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(100); + expect((await copies.get(companyId, copy.runId))?.nextAttemptAt).toBeNull(); + }); + await expect.poll(() => idleWorkSnapshot().active).toBe(0); + const reads = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); expect(removes).not.toHaveBeenCalled(); + expect(worker.nextWakeAt()).toBeNull(); + expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial); + } finally { await worker.stop(); } + }); + it("honors a cleanup obligation published before stale-deadline settlement", async () => { + const copy = await run(), dueAt = new Date(Date.now() + 30_000); + await db.update(agentInstructionWorkingCopies).set({ state: "unavailable", processStoppedAt: new Date(), + nextAttemptAt: dueAt, receipt: { schema: "paperclip.agent-files.v1", cleanupPending: false } }) + .where(eq(agentInstructionWorkingCopies.runId, copy.runId)); + const transact = db.transaction.bind(db); + const race = vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => { + await db.update(agentInstructionWorkingCopies).set({ receipt: { ...copy.receipt, cleanupPending: true } }) + .where(eq(agentInstructionWorkingCopies.runId, copy.runId)); + return transact(callback, config); + }); + try { + await copies.release(companyId, copy.runId); + expect(await copies.get(companyId, copy.runId)).toMatchObject({ receipt: { cleanupPending: false }, nextAttemptAt: null }); + await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" }); + expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial); + } finally { race.mockRestore(); } + }); it("reactivates from real collection, preserves saved files, retries failed cleanup, and becomes quiet", async () => { const copy = await run(); await fs.writeFile(path.join(copy.localRoot, "memory.txt"), "saved memory"); clock(); const worker = coordinator(); @@ -1461,6 +1500,53 @@ describe("persistent agent directories", () => { return { copy: (await copies.get(companyId, copy.runId))!, lease, environmentId }; } + it.each(["destroyed", "stopped", "broken_handoff", "extra_owner", "foreign_company"] as const)( + "recovers an earlier remote copy only from the exact destroyed successor chain (%s)", async scenario => { + const { copy, lease } = await unavailableRemote(); + const issueId = randomUUID(), successorRunId = randomUUID(), successorId = randomUUID(); + const acquiredAt = new Date(Date.now() - 2000), handoffAt = new Date(Date.now() - 1000); + await db.insert(issues).values({ id: issueId, companyId, title: "Completed retained workspace", status: "done" }); + await db.insert(heartbeatRuns).values({ id: successorRunId, companyId, agentId, invocationSource: "on_demand", status: "succeeded" }); + const [previous] = await db.update(environmentLeases).set({ status: "expired", leasePolicy: "reuse_by_environment", + issueId, acquiredAt, releasedAt: handoffAt, cleanupStatus: "success", metadata: { agentId } }) + .where(eq(environmentLeases.id, lease.id)).returning(); + const successor = { ...previous!, id: successorId, heartbeatRunId: successorRunId, acquiredAt: handoffAt, + releasedAt: new Date(), metadata: { agentId } }; + if (scenario === "foreign_company") { + const foreignCompanyId = randomUUID(); + await db.insert(companies).values({ id: foreignCompanyId, name: "Unrelated owner", issuePrefix: randomUUID().slice(0, 8) }); + successor.companyId = foreignCompanyId; + } + const handoff = createReusableLeaseHandoff(previous!, successor); + const termination = remoteTerminationReceipt(successor, { providerLeaseId: lease.providerLeaseId, + state: scenario === "stopped" ? "stopped" : "destroyed" }); + await db.update(environmentLeases).set({ metadata: { agentId, reusableLeaseSuccessor: handoff } }) + .where(eq(environmentLeases.id, lease.id)); + await db.insert(environmentLeases).values({ ...successor, metadata: { agentId, remoteExecutionTermination: termination, + reusableLeasePredecessor: scenario === "broken_handoff" ? { ...handoff, handoffId: randomUUID() } : handoff } }); + if (scenario === "extra_owner") await db.insert(environmentLeases).values({ ...lease, id: randomUUID(), status: "active" }); + await db.update(agentInstructionWorkingCopies).set({ state: "pending_collection", attempts: 0, candidateHash: null, candidateBase64: null }) + .where(eq(agentInstructionWorkingCopies.runId, copy.runId)); + const shell = vi.spyOn(executionTargetTools, "runAdapterExecutionTargetShellCommand").mockRejectedValue(new Error("must not execute")); + try { + await copies.recoverStopped(); + const recovered = (await copies.get(companyId, copy.runId))!; + if (scenario === "destroyed") { + expect(recovered).toMatchObject({ state: "unavailable", errorCode: "INSTRUCTION_COLLECTION_UNAVAILABLE", + receipt: { cleanupPending: false }, nextAttemptAt: null }); + expect(recovered.processStoppedAt).toBeInstanceOf(Date); + await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" }); + } else { + expect(recovered).toMatchObject({ state: "pending_collection", processStoppedAt: null }); + expect(await fs.readFile(path.join(copy.localRoot, entryFile), "utf8")).toBe(initial); + } + expect(shell).not.toHaveBeenCalled(); + expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial); + const [historical] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)); + expect(historical!.metadata).not.toHaveProperty("remoteExecutionTermination"); + } finally { shell.mockRestore(); } + }); + it.each([undefined, "stopped", "companyId", "runId", "leaseId", "provider", "providerLeaseId"])( "defers unavailable cleanup without exact destruction proof (%s)", async proof => { const { copy } = await unavailableRemote({ state: proof === undefined ? undefined : proof === "stopped" ? "stopped" : "destroyed", diff --git a/server/src/services/agent-directory-working-copies.ts b/server/src/services/agent-directory-working-copies.ts index c1fe658aab6..b1c8de72e5d 100644 --- a/server/src/services/agent-directory-working-copies.ts +++ b/server/src/services/agent-directory-working-copies.ts @@ -2,7 +2,7 @@ import { notifyInstructionCollection } from "./instruction-collection-notificati import fs from "node:fs/promises"; import path from "node:path"; import { isDeepStrictEqual } from "node:util"; -import { and, eq, inArray, isNull, sql } from "drizzle-orm"; +import { and, eq, inArray, isNull, sql, type SQL } from "drizzle-orm"; import { agents, environmentLeases, environments, agentInstructionWorkingCopies as copies, withDedicatedDbConnection, type Db } from "@paperclipai/db"; import { syncDirectoryToSsh, restoreWorkspaceFromSshExecution } from "@paperclipai/adapter-utils/ssh"; import { prepareAdapterExecutionTargetRuntime, runAdapterExecutionTargetShellCommand, type AdapterExecutionTarget, type PreparedAdapterExecutionTargetRuntime } from "@paperclipai/adapter-utils/execution-target"; @@ -16,7 +16,7 @@ import type { AuthorizationActor } from "./authorization.js"; import type { EnvironmentRuntimeService } from "./environment-runtime.js"; import type { Environment, EnvironmentLease } from "@paperclipai/shared"; import { agentDirectoryBaselineDigest, agentDirectoryProbeProgram, observeLocalAgentDirectory } from "./agent-directory-probe.js"; -import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; +import { hasRemoteTerminationReceipt, resolveRemoteTerminationLease } from "./remote-execution-termination.js"; import { cachedAgentFileManifest, captureAgentFileCheckpoint, checkpointBaseline, checkpointSnapshot, type AgentFileManifest } from "./agent-file-checkpoints.js"; import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js"; import { logger } from "../middleware/logger.js"; @@ -46,7 +46,7 @@ function baseline(row: Copy) { function actor(row: Copy): AuthorizationActor { return { type: "agent", companyId: row.companyId, agentId: row.agentId, runId: row.runId, onBehalfOfUserId: row.responsibleUserId }; } -export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string, runId: string) => Promise, patch: (row: Copy, values: Partial) => Promise, environmentRuntime?: EnvironmentRuntimeService) { +export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string, runId: string) => Promise, patch: (row: Copy, values: Partial, condition?: SQL) => Promise, environmentRuntime?: EnvironmentRuntimeService) { const store = agentFileStore(db); async function insertCopy(values: typeof copies.$inferInsert) { return db.transaction(async tx => { @@ -503,7 +503,15 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string return row.state === "superseded" || !row.processStoppedAt || completed.has(row.state) && row.receipt?.cleanupPending === false; } + async function settleCompletedCleanupRetry(row: Copy) { + if (completed.has(row.state) && row.processStoppedAt && !row.receipt?.retainedByRunId && row.receipt?.cleanupPending === false && row.nextAttemptAt) { + return patch(row, { nextAttemptAt: null }, sql`${copies.receipt} = ${JSON.stringify(row.receipt)}::jsonb + AND ${copies.processStoppedAt} IS NOT NULL`); + } + return row; + } async function release(row: Copy, target?: AdapterExecutionTarget | null) { + row = await settleCompletedCleanupRetry(row); if (releaseIsNoop(row) || !await owns(row)) return; const destroyedOnly = row.receipt?.cleanupDestroyedOnly === true; if (row.receipt?.retainedByRunId || row.state === "superseded" || !await owns(row) || !row.processStoppedAt) return; @@ -559,7 +567,7 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string : eq(environmentLeases.environmentId, row.location.slice("remote:".length)), )); if (leases.length !== 1) return false; - const lease = leases[0]!; + const lease = await resolveRemoteTerminationLease(db, leases[0]!); if (lease.environmentId !== null && row.location !== `remote:${lease.environmentId}`) return false; const remoteCwd = cleanup?.remoteCwd ?? lease.metadata?.remoteCwd; return typeof remoteCwd === "string" && @@ -588,7 +596,7 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string cleanup?.leaseId ? eq(environmentLeases.id, cleanup.leaseId) : eq(environmentLeases.environmentId, row.location.slice("remote:".length)), )); if (leases.length !== 1) return null; - const lease = leases[0]!; + const lease = await resolveRemoteTerminationLease(db, leases[0]!); const destroyedWithoutEnvironment = lease.environmentId === null && hasRemoteTerminationReceipt(lease) && (lease.metadata?.remoteExecutionTermination as { state?: string } | undefined)?.state === "destroyed"; if (row.location !== `remote:${lease.environmentId}` && !destroyedWithoutEnvironment) return null; @@ -658,10 +666,12 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string checkpointWarm: (row: Copy, target?: AdapterExecutionTarget | null) => serial(row, current => current.receipt?.warm === true ? checkpoint(current, target) : Promise.resolve(current), "agent_directory_checkpoint"), collectStopped: (row: Copy, target?: AdapterExecutionTarget | null) => serial(row, current => collectStopped(current, target), "agent_directory_collect"), release: async (row: Copy) => { - const current = await get(row.companyId, row.runId); + let current = await get(row.companyId, row.runId); + if (!current) return; + current = await settleCompletedCleanupRetry(current); // This check changes no receipt or bytes. A concurrent stop merely leaves // cleanup to its owner or the recovery sweep; it grants no new authority. - if (!current || releaseIsNoop(current)) return; + if (releaseIsNoop(current)) return; await serial(current, release, "agent_directory_release"); }, }; diff --git a/server/src/services/agent-instruction-working-copies.ts b/server/src/services/agent-instruction-working-copies.ts index 0d990331d4a..586bd63dbfb 100644 --- a/server/src/services/agent-instruction-working-copies.ts +++ b/server/src/services/agent-instruction-working-copies.ts @@ -129,8 +129,8 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen }; return db.transaction(tx => write(tx as unknown as Db)); } - async function patch(row: Copy, values: Partial) { - const updated = await tryPatch(row, values); + async function patch(row: Copy, values: Partial, condition?: SQL) { + const updated = await tryPatch(row, values, false, condition); // A late cleanup must not overwrite an explicit resolution or a newer // baseline. Canonical content has its own independent head CAS. return updated ?? (await get(row.companyId, row.runId))!; @@ -499,8 +499,8 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen function directoryCleanupEntry(row: Copy | null): DirectoryCleanupEntry | null { return row && ["saved", "unchanged", "resolved", "unavailable"].includes(row.state) && row.processStoppedAt && isAgentDirectoryCopy(row) && !row.receipt?.retainedByRunId - && row.receipt?.cleanupPending !== false - && (Object.hasOwn(row.receipt!, "baseline") || row.receipt?.cleanupPending === true) + && (row.receipt?.cleanupPending === false ? row.nextAttemptAt !== null + : Object.hasOwn(row.receipt!, "baseline") || row.receipt?.cleanupPending === true) ? { id: row.runId, companyId: row.companyId, dueAt: row.nextAttemptAt?.getTime() ?? 0 } : null; } function directoryCleanupPredicate() { @@ -508,8 +508,9 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen AND ${copies.processStoppedAt} IS NOT NULL AND ${copies.receipt}->>'schema' = 'paperclip.agent-files.v1' AND coalesce(${copies.receipt}->>'retainedByRunId', '') = '' - AND ${copies.receipt}->>'cleanupPending' IS DISTINCT FROM 'false' - AND (${copies.receipt} ? 'baseline' OR ${copies.receipt}->>'cleanupPending' = 'true')`; + AND ((${copies.receipt}->>'cleanupPending' = 'false' AND ${copies.nextAttemptAt} IS NOT NULL) + OR (${copies.receipt}->>'cleanupPending' IS DISTINCT FROM 'false' + AND (${copies.receipt} ? 'baseline' OR ${copies.receipt}->>'cleanupPending' = 'true')))`; } async function loadDirectoryCleanup(scope: DirectoryCleanupScope | undefined, admitted: () => boolean) { const entries: DirectoryCleanupEntry[] = []; diff --git a/server/src/services/remote-execution-termination.ts b/server/src/services/remote-execution-termination.ts index f9fae854c69..d30b60ae697 100644 --- a/server/src/services/remote-execution-termination.ts +++ b/server/src/services/remote-execution-termination.ts @@ -1,5 +1,8 @@ import { and, eq } from "drizzle-orm"; import { environmentLeases, type Db } from "@paperclipai/db"; +import { retiredReusableLeasePredecessors } from "./environment-lease-handoff.js"; + +type Lease = typeof environmentLeases.$inferSelect; type LeaseIdentity = { id: string; companyId: string; heartbeatRunId: string | null; @@ -39,12 +42,36 @@ export function remoteLeaseCleanupScope(lease: Pick { + if (hasRemoteTerminationReceipt(lease) || lease.status !== "expired" || + lease.leasePolicy !== "reuse_by_environment" || !lease.provider || !lease.providerLeaseId) return lease; + const history = await db.select().from(environmentLeases).where(and( + eq(environmentLeases.provider, lease.provider), eq(environmentLeases.providerLeaseId, lease.providerLeaseId), + )); + const current = history.find(row => row.id === lease.id); + if (!current || current.companyId !== lease.companyId || current.heartbeatRunId !== lease.heartbeatRunId) return lease; + const tips = history.filter(row => !Object.hasOwn(row.metadata ?? {}, "reusableLeaseSuccessor")); + if (tips.length !== 1) return current; + const tip = tips[0]!; + const receipt = tip.metadata?.remoteExecutionTermination as Record | undefined; + if (!hasRemoteTerminationReceipt(tip) || receipt?.state !== "destroyed") return current; + const predecessors = retiredReusableLeasePredecessors(tip, history.filter(row => row.id !== tip.id)); + if (!predecessors?.some(row => row.id === current.id)) return current; + const termination = remoteTerminationReceipt(current, { providerLeaseId: current.providerLeaseId, state: "destroyed" }); + if (!termination) return current; + return { ...current, metadata: { ...current.metadata, remoteExecutionTermination: { + ...termination, confirmedAt: receipt.confirmedAt, + } } }; +} + export async function stoppedRemoteCleanupScopes(db: Db, companyId: string, runId: string) { const leases = await db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId), )); - if (leases.length === 0 || !leases.every(hasRemoteTerminationReceipt)) return null; - return [...new Set(leases.map(lease => remoteLeaseCleanupScope(lease)!))]; + if (leases.length === 0) return null; + const resolved = await Promise.all(leases.map(lease => resolveRemoteTerminationLease(db, lease))); + if (!resolved.every(hasRemoteTerminationReceipt)) return null; + return [...new Set(resolved.map(lease => remoteLeaseCleanupScope(lease)!))]; } export async function remoteExecutionHasStopped(db: Db, companyId: string, runId: string) {