From 60a99307b3bb1062078d35af6677e3f377d8bab8 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 2 Nov 2026 11:53:44 -0600 Subject: [PATCH] fix(server): settle cleanup after reusable sandbox destruction (#18633) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A web-server replacement must preserve work and let completed cleanup become idle. > - Several task turns can use one remote sandbox through recorded lease handoffs. > - The last owner can destroy that sandbox while an earlier owner still waits for its own termination receipt. > - A completed agent-file cleanup can also retain a retry deadline from a competing callback. > - This pull request resolves confirmed destruction through the existing handoff contract and settles obsolete cleanup retries. > - The benefit is correct cleanup recovery without deleting history or weakening sleep checks. ## Linked Issues or Issue Description Refs: #18554. Related work: #14695 handles warm agent-file ownership; #13291 handles interrupted conversation continuation. This PR fixes cleanup recovery after confirmed final destruction. It does not change provider process reuse or start a new run. **What happened?** After ten qualified staging web-server replacements, a bounded database-idle check failed. Completed native runs still owned pending agent-file cleanup. Normal workspace archive confirmed final sandbox destruction. An earlier lease in the same recorded handoff chain remained unresolved. Two completed cleanup receipts also kept stale retry deadlines. **Expected behavior** Resolve an earlier owner's cleanup only when the complete recorded handoff chain leads to an exact provider-confirmed destruction receipt. Clear a retry when its cleanup is already complete. Keep real or uncertain work pending. **Steps to reproduce** 1. Complete a remote run, reuse its sandbox for a second run, then archive the completed workspace. 2. Recover agent-file collection for the first run after the final lease confirms destruction. 3. Separately, reconstruct a completed cleanup receipt that still has a retry deadline. 4. Observe that both records remain unfinished without this fix. **Paperclip version or commit** Observed on `1a6017d6460267ad89d7e1784e2b521c28d8e9ac`. The changed source was unchanged in master `97c4dc969e13364464765ffbde74955005bf44d3`, which is this branch's base. **Deployment mode** A managed ephemeral web server with PostgreSQL, object storage and a remote sandbox. Regression tests use local PostgreSQL and no live provider or model calls. ## What Changed - Resolve remote termination at read time through the existing strict reusable-lease handoff validator. Require one final owner and a valid destruction receipt. Preserve historical lease rows. - Use the same proof in agent-directory cleanup and remote-stop inspection. Keep stopped, ambiguous, broken and cross-company chains blocked. - Reconcile completed cleanup receipts with stale retry deadlines through the existing deadline worker. Compare the receipt before clearing a deadline. - Add recovery, refusal, restart quietness and concurrent-obligation regressions. - Document the cleanup contract and preserve deployment evidence, failed checks and remaining acceptance work in the existing plan. ## Verification - Both owning service suites passed 169 tests before the final receipt comparison was added. - The seven-case selection passed again after the release-race review fix: `pnpm exec vitest run server/src/__tests__/agent-directory-working-copies.test.ts -t 'stale retry|destroyed successor chain|stale-deadline settlement'`. - Repository-wide `pnpm -r typecheck` and `pnpm build` passed on final review fix `7935d99bed`. The full `pnpm test:run` is still running. Latest-head CI has 55 successful checks, three skips, and one neutral review result. Greptile is 5/5. The Paperclip reviewer’s race finding is fixed; its optional query-deduplication suggestion is documented and the fresh lock-boundary checks are retained. - The reviewer identified an early return after a lost receipt comparison. The strengthened race regression failed before the fix and passed afterward: release now acts on the returned authoritative row, completes the new cleanup obligation, and preserves canonical files. - The first focused attempt failed the new stale-deadline test because the public release guard skipped settlement. That guard was fixed. The original failure is retained. A sandboxed typecheck could not fetch pnpm signature metadata; the normal network-enabled command passed without changing verification policy. - Prior deployed evidence: ten Railway replacements, 12,521 reads, 1,566 verified saves, 34 matching file hashes, no HTTP failures, actual browser reconnects of 1.16–1.29 seconds, and one remote execution retained across replacement. Two harness false alarms were separately adjudicated without redeploying those steps. - That deployment evidence predates this PR. Current-image natural idle, automatic sleep/wake browser return and the isolated production canary remain unverified. Production has not changed. ## Risks - Cleanup authority must not cross a company or allocation boundary. The existing validator checks the entire recorded chain; stopped successors and unexpected owners remain unresolved. - Confirmed destruction permits no-save cleanup. It does not claim that uncollected agent files were saved. - Read-time resolution adds a scoped lease-history query only for expired reusable leases without direct termination proof. No new recurring worker or polling cadence is added. - The deadline update must not erase new work. It requires the same cleanup receipt and preserves concurrent changes. - No schema or public API changes are included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code execution and test tools. The precise backend revision and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-sleep-safety.md | 11 ++ .../2026-10-31-zero-downtime-web-deploys.md | 154 +++++++++++++++++- .../agent-directory-working-copies.test.ts | 88 +++++++++- .../agent-directory-working-copies.ts | 24 ++- .../agent-instruction-working-copies.ts | 13 +- .../services/remote-execution-termination.ts | 31 +++- 6 files changed, 299 insertions(+), 22 deletions(-) diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.md index 12f8dada5c4..8334e3b1685 100644 --- a/doc/idle-sleep-safety.md +++ b/doc/idle-sleep-safety.md @@ -133,6 +133,17 @@ fingerprint and RPC company scope for later read-only verification. Retained allocations, unfinished runs/accounting/retries, export intents and live setup credentials continue to block sleep. +For a reused remote allocation, a completed predecessor can resolve termination +through the final owner's destruction receipt only when the full host-recorded +lease handoff chain matches. Company, agent, task, workspace, environment, provider +and allocation identities must agree. A stopped successor, broken handoff or +unrelated owner grants no cleanup authority. This resolution does not rewrite +historical leases, execute remotely or claim that uncollected files were saved. +The normal agent-file recovery worker retains the no-save outcome and completes +cleanup after confirmed destruction. A completed cleanup receipt with a stale +retry deadline is reconciled once through the existing deadline worker. Clearing +that deadline compares the receipt so a concurrent cleanup obligation is kept. + Pending human-only connection setup can wait asleep when no active enabled connection of the requested purpose exists. Available access reblocks sleep so the existing reconciliation path can run; answering a setup card continues to diff --git a/doc/plans/2026-10-31-zero-downtime-web-deploys.md b/doc/plans/2026-10-31-zero-downtime-web-deploys.md index 71e4bf658b2..842a6e5854d 100644 --- a/doc/plans/2026-10-31-zero-downtime-web-deploys.md +++ b/doc/plans/2026-10-31-zero-downtime-web-deploys.md @@ -1,10 +1,152 @@ # Zero-downtime web deployments -Date: 2026-10-31. Status (2026-11-01 15:39 UTC): existing-volume staging enrollment -and manual sleep/wake preserve saved files and read/save behavior. The full local -active-agent journey now passes on `cc23019691`; hosted browser continuity and -the ten-deploy staging campaign remain unqualified. Natural database autosuspension -passes after abandoned test-resource cleanup. No production canary yet. +Date: 2026-10-31. Status (2026-11-02 17:23 UTC): Core #18554 is merged as +`1a6017d6460267ad89d7e1784e2b521c28d8e9ac`. The ten-roll and native-agent +proof used Cloud `461c4aea28`. Separately reviewed Cloud #1091/#1092 subsequently +reached staging as `89dd0cfec2`; their changes concern workspace cleanup and +Agent Computer resource fencing. Railway, fleet and deployment workflow paths +are unchanged. Production remains on its original controller revision; the +next manual controller deployment must use current reviewed master. +Both immutable Cloud distributions built from that fixed source passed packaged +Linux rehearsals through the Railway wrapper: 260 reads, 53 saves, zero failures, +legacy-volume migration, an unready replacement, and removal of both predecessor +homes. Each fixture cleaned its containers. These rehearsals do not exercise an +active agent or an actual browser. The two distributions have distinct image +digests but identical application source; their compatible rollback qualifies +artifact replacement, not compatibility between different source revisions. + +The final Core PR has two 5/5 reviews and all required checks passed. One legacy +browser CI shard failed because its deferred-wake fixture observed no queued wake; +the unchanged case passed locally on isolated PostgreSQL and its single CI retry +passed. The original failure remains in the PR evidence. The full local live-agent +journey passed on `cc23019691` before the final non-suspended dispatch refinement; +that refinement passed its three focused Linux regressions, server typecheck, +build and final CI. Do not describe the earlier live run as a rerun of the merge. + +The fixed canonical package baseline passed on the pinned migrated staging fixture +using the allowed one-time maintenance path. An actual Chromium tab running the +older UI reconnected, retained an unsaved title edit, and saved it through the new +server; an independent API read confirmed the title. That old tab opened the +Agents page, and all four previously captured asset hashes still match after the +old deployment was removed. Baseline maintenance interruptions are preserved and +are not counted as zero-downtime qualification. + +The first ten-roll attempt was cancelled at step one, before Railway replacement. +The controller's pre-existing hot-upgrade allowlist named an older test stack, +so it drained rather than negotiated preservation of the new fixture's active run. +Continuous read/save and browser probes remained healthy, but this is NOT active +execution continuity evidence. The original command completed normally once; its +downloaded output bytes and native terminal were independently verified. The +owned fixture has been added to the existing allowlist without changing fleet +defaults. The normal staging controller workflow is applying that correction. +The rollout harness now checks both canary policy and positive tenant readiness +before submitting a campaign. The cancelled attempt remains failed evidence, +with no automatic campaign retry. +The configuration deployment then failed before Railway submission when Sentry +release registration failed or timed out. Its source checks passed; the original +logs are retained. One unchanged whole-workflow retry preserves attempt-bound +preflight requirements. That retry failed at the same pre-Railway Sentry request. +A read-only Sentry lookup confirms the release exists (HTTP 200). The canary-only +configuration was then applied by redeploying the existing successful controller +deployment: the source and replacement image digests are identical, and health +attests unchanged Cloud `461c4aea28`. Both failed workflow attempts are retained; +no Sentry gate was disabled and no new application release was built in this +configuration recovery. The new active-command attempt and gated sequence are +starting with the corrected controller. The first preflight stopped before task +admission on a missing optional plan's 404. `useIssuePlanDocument` explicitly +maps that response to null; an independent read confirmed no plan on the owned +completed task. The raw response remains recorded and the browser gate permits +only that verified path. The configuration redeploy's actual browser reconnect +was 1.22 seconds, with no probe read/save failure. + +All ten Railway replacements are independently qualified. Final continuous proof +passed 12,521 reads and 1,566 saves, with every acknowledged save present exactly +once and all 34 file hashes matching. There were zero HTTP failures. Actual +browser reconnects across the ten deployments ranged from 1.16 to 1.29 seconds. +The old UI assets and original agent artifact remain readable after deployment +ten and provider cleanup. Both traffic observers have stopped cleanly. The first +replaced the webserver while the original eight-minute remote command was +running. Its one execution ID started before replacement and completed afterward +with exit zero. The second replaced the server after completion; a later follow-up +retained the same native/provider session and runner, with accepted native +terminals and finalized accounting for both runs. The 91-byte artifact was +downloaded and independently checked for exactly one start and completion. +Actual browser reconnects were 1.28, 1.16 and 1.25 seconds; continuous reads and +saves have no failures so far. Previously unopened Projects and Skills pages rendered in the +retained old tab. A title draft survived the third cutover and was saved through +the UI, then independently read back from the replacement. + +The harness stopped after deployment three on another owned task's expected +missing optional-plan 404. The UI explicitly treats this as null. Both exact task +paths were independently checked, and the provider deployment, package eligibility, +serving identity, removal of the predecessor, browser timing and continuous probe +were revalidated. The failed original receipt remains unchanged alongside an +explicit classification review. The remaining seven replacements continue without +redeploying or retrying deployment three. An announcement also obscured one +sidebar click; its ordinary Dismiss control resolved that retained interaction +timeout. A separate large run-log diagnostic timed out; a narrower public-events +read succeeded and proves the original command's single execution identity. +A later bounded 64-KiB log read succeeded from shared storage after both the +originating webserver and remote sandbox were removed. + +The native recovery event records SIGTERM without a retry run, then adoption of +the same runner at controller generation 2. Each of the two runs has exactly one +accepted result and terminal event. The independent accounting inspector has no +findings or pending work; the task totals 52.941 cents across two runs. Including +the earlier no-replacement attempt, staging has three estimated cost events totaling +113.84284 cents. The temporary agent is paused, its secret removed and the prior +company budget restored; all three provider resources are independently confirmed absent. The first delete +was refused while the sandbox stopped; that failure remains alongside the +successful cleanup after a fresh stopped-state observation. Both owned local +PostgreSQL/object-storage containers are removed; test evidence is retained. + +Deployment nine also completed, but the local test reader observed the browser +snapshot while its writer had truncated the file. The campaign response was +already parsed and successful; the only non-atomic JSON read in that failing +checkpoint is the local browser report. A concurrent-write regression reproduced +the problem. The reader now retries only local JSON syntax errors for at most +100 ms; persistent corruption still fails, and HTTP/acceptance failures are never +retried. Deployment nine was independently revalidated without redeployment. +Both raw failed sequence receipts and their explicit classification reviews remain; +the ten provider deployments themselves have not been retried. The final staging +driver stopped at the harness failure and is being resumed from the retained +sequence. The normal UI's 256,000-byte run-log read also passes after replacement. + +The first repair check stopped before mutation because an unrelated staging +campaign was active. That campaign has finished, and the pinned canary remains +active and unlocked on the qualified final image. Both one-target configuration repair campaigns passed: backup policy was +restored, the temporary marker removed, and durable files/shared assets remained +enabled on the same volume-free image. The current-image eight-minute natural-idle check failed with all browser pages and +traffic probes closed. Bounded SQL diagnosis found three completed native runs +still in agent-file collection recovery. Normal archive of both completed test +workspaces independently passed close-readiness and preserved saved workspace +files. It produced provider-confirmed destruction receipts for the latest leases. +Two copies then completed no-save cleanup, but retained stale retry deadlines; +the earlier run of a reused sandbox still lacks direct termination evidence even +though its validated successor allocation is destroyed. These are source recovery +gaps, not an idle-proof pass. No historical rows were manually rewritten. The +failed idle and cancelled browser lifecycle receipts remain unchanged. Fix +ownership belongs to remote termination proof resolution and agent-directory +cleanup scheduling; no polling cadence or sleep-safety check should be relaxed. + +The isolated source branch is now based on current master `97c4dc969e`. The +recovery fix resolves the destroyed allocation through the existing validated +handoff-chain contract at read time; it does not stamp historical leases or infer +stop from task status. Agent-file cleanup also reconciles stale retry deadlines +with a receipt comparison. Both owning service suites passed 169 tests, and the +server typecheck passed. The first new deadline test caught a public guard that +skipped settlement; that failed attempt is retained and the corrected five-case +selection passed. The final seven-case selection passed, including a concurrent receipt change and +cross-company rejection. Repository-wide checks and PR review are pending. +This source has not been deployed and is not covered by the prior ten-roll proof. + +Next: review and ship the cleanup fix, complete current-image idle/sleep/wake, then the +isolated production forward/rollback canary. No production deployment has occurred. +Dotta completed the authorized GitHub sign-in; the production browser is back on +the ordinary organization list. The production canary still waits for staging. Its new organization form is +prepared but unsubmitted. Production remains on its original controller revision, +with no campaigns, the single-volume default and continuous deployment disabled. Earlier +staging sleep/wake and natural database-idle proofs remain valid for their image. Core #18543 merged as `7751fc3107` after 55 successful CI checks, Greptile 5/5 and no unresolved threads. It supplies runtime admission, immutable-cache @@ -202,7 +344,7 @@ Measure one-time setup separately from repeated attempts. Keep failed evidence. | Item | State | Next action | | --- | --- | --- | | Prior prerequisites | #18403 and all five storage workstreams merged | Retain the ephemeral filesystem contract | -| Core implementation | #18518, #18526, #18529, #18530, #18531, #18533, #18543 and #18552 merged; #18554 reviewed at 5/5 with CI pending; local packaged migration, browser overlap, cold restoration and durable command proofs recorded below | Publish and qualify the exact fixed release after CI/review merge | +| Core implementation | #18518, #18526, #18529, #18530, #18531, #18533, #18543 and #18552 merged; #18554 merged with all required checks and 5/5 reviews; both exact fixed images passed packaged migration and deletion; active hosted output verified | Finish hosted continuation and accounting reconciliation | | Cloud implementation | #1096, #1098, #1099, #1100, #1101 and #1102 merged; #1102 serving on staging | Verify repeated Railway overlap behavior after Core recovery qualification | | Staging | Original enrollment complete; migrated data and signing key preserved; volume detached and retained | Sleep/wake and natural database idle pass; verify repair and ten app-only replacements with browser continuity | | Production | No canary mutation performed | Proceed only after the staging and active-agent acceptance gates pass | diff --git a/server/src/__tests__/agent-directory-working-copies.test.ts b/server/src/__tests__/agent-directory-working-copies.test.ts index fcf3c5ebd6a..415781328d3 100644 --- a/server/src/__tests__/agent-directory-working-copies.test.ts +++ b/server/src/__tests__/agent-directory-working-copies.test.ts @@ -16,7 +16,7 @@ import { createHash, randomUUID } from "node:crypto"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { eq, sql } from "drizzle-orm"; import { agentFileStore, fileHash, inspectAgentFile, inspectAgentDirectory, snapshotAgentFiles, MAX_AGENT_FILE_BYTES, MAX_AGENT_DIRECTORY_BYTES, MAX_AGENT_DIRECTORY_ENTRIES } from "../services/agent-file-store.js"; -import { agents, companies, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, environmentLeases, environments, agentInstructionWorkingCopies, agentInstructionRevisions, agentInstructionHeads, createDb, fileStorageDb } from "@paperclipai/db"; +import { agents, companies, issues, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, environmentLeases, environments, agentInstructionWorkingCopies, agentInstructionRevisions, agentInstructionHeads, createDb, fileStorageDb } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js"; import { agentInstructionWorkingCopyService, instructionWorkingCopyGuidance } from "../services/agent-instruction-working-copies.js"; @@ -24,6 +24,7 @@ import { resolveManagedInstructionsRoot } from "../services/agent-instructions.j import { buildNativeRuntimeContext } from "../services/native-runtime/runtime-context.js"; import type { EnvironmentRuntimeService } from "../services/environment-runtime.js"; import { remoteTerminationReceipt } from "../services/remote-execution-termination.js"; +import { createReusableLeaseHandoff } from "../services/environment-lease-handoff.js"; import { AgentDirectoryReuseInvalidatedError, agentDirectoryWorkingCopyService } from "../services/agent-directory-working-copies.js"; import { withDirectoryMergeLock } from "@paperclipai/adapter-utils/workspace-restore-merge"; import { heartbeatRunEvents } from "@paperclipai/db"; @@ -123,6 +124,44 @@ describe("persistent agent directories", () => { for (const copy of [active, completed, transferred]) expect((await fs.stat(copy.localRoot)).isDirectory()).toBe(true); } finally { await worker.stop(); } }); + it("retires a stale retry on an already cleaned receipt after restart without filesystem work", async () => { + const copy = await run(); + await db.update(agentInstructionWorkingCopies).set({ state: "unavailable", processStoppedAt: new Date(), + nextAttemptAt: new Date(), receipt: { schema: "paperclip.agent-files.v1", cleanupPending: false } }) + .where(eq(agentInstructionWorkingCopies.runId, copy.runId)); + clock(); const worker = coordinator(), removes = vi.spyOn(fs, "rm"); + try { + await agentInstructionWorkingCopyService(db).registerDirectoryCleanupWork(worker, () => true, async () => true).ready; + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(100); + expect((await copies.get(companyId, copy.runId))?.nextAttemptAt).toBeNull(); + }); + await expect.poll(() => idleWorkSnapshot().active).toBe(0); + const reads = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); expect(removes).not.toHaveBeenCalled(); + expect(worker.nextWakeAt()).toBeNull(); + expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial); + } finally { await worker.stop(); } + }); + it("honors a cleanup obligation published before stale-deadline settlement", async () => { + const copy = await run(), dueAt = new Date(Date.now() + 30_000); + await db.update(agentInstructionWorkingCopies).set({ state: "unavailable", processStoppedAt: new Date(), + nextAttemptAt: dueAt, receipt: { schema: "paperclip.agent-files.v1", cleanupPending: false } }) + .where(eq(agentInstructionWorkingCopies.runId, copy.runId)); + const transact = db.transaction.bind(db); + const race = vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => { + await db.update(agentInstructionWorkingCopies).set({ receipt: { ...copy.receipt, cleanupPending: true } }) + .where(eq(agentInstructionWorkingCopies.runId, copy.runId)); + return transact(callback, config); + }); + try { + await copies.release(companyId, copy.runId); + expect(await copies.get(companyId, copy.runId)).toMatchObject({ receipt: { cleanupPending: false }, nextAttemptAt: null }); + await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" }); + expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial); + } finally { race.mockRestore(); } + }); it("reactivates from real collection, preserves saved files, retries failed cleanup, and becomes quiet", async () => { const copy = await run(); await fs.writeFile(path.join(copy.localRoot, "memory.txt"), "saved memory"); clock(); const worker = coordinator(); @@ -1461,6 +1500,53 @@ describe("persistent agent directories", () => { return { copy: (await copies.get(companyId, copy.runId))!, lease, environmentId }; } + it.each(["destroyed", "stopped", "broken_handoff", "extra_owner", "foreign_company"] as const)( + "recovers an earlier remote copy only from the exact destroyed successor chain (%s)", async scenario => { + const { copy, lease } = await unavailableRemote(); + const issueId = randomUUID(), successorRunId = randomUUID(), successorId = randomUUID(); + const acquiredAt = new Date(Date.now() - 2000), handoffAt = new Date(Date.now() - 1000); + await db.insert(issues).values({ id: issueId, companyId, title: "Completed retained workspace", status: "done" }); + await db.insert(heartbeatRuns).values({ id: successorRunId, companyId, agentId, invocationSource: "on_demand", status: "succeeded" }); + const [previous] = await db.update(environmentLeases).set({ status: "expired", leasePolicy: "reuse_by_environment", + issueId, acquiredAt, releasedAt: handoffAt, cleanupStatus: "success", metadata: { agentId } }) + .where(eq(environmentLeases.id, lease.id)).returning(); + const successor = { ...previous!, id: successorId, heartbeatRunId: successorRunId, acquiredAt: handoffAt, + releasedAt: new Date(), metadata: { agentId } }; + if (scenario === "foreign_company") { + const foreignCompanyId = randomUUID(); + await db.insert(companies).values({ id: foreignCompanyId, name: "Unrelated owner", issuePrefix: randomUUID().slice(0, 8) }); + successor.companyId = foreignCompanyId; + } + const handoff = createReusableLeaseHandoff(previous!, successor); + const termination = remoteTerminationReceipt(successor, { providerLeaseId: lease.providerLeaseId, + state: scenario === "stopped" ? "stopped" : "destroyed" }); + await db.update(environmentLeases).set({ metadata: { agentId, reusableLeaseSuccessor: handoff } }) + .where(eq(environmentLeases.id, lease.id)); + await db.insert(environmentLeases).values({ ...successor, metadata: { agentId, remoteExecutionTermination: termination, + reusableLeasePredecessor: scenario === "broken_handoff" ? { ...handoff, handoffId: randomUUID() } : handoff } }); + if (scenario === "extra_owner") await db.insert(environmentLeases).values({ ...lease, id: randomUUID(), status: "active" }); + await db.update(agentInstructionWorkingCopies).set({ state: "pending_collection", attempts: 0, candidateHash: null, candidateBase64: null }) + .where(eq(agentInstructionWorkingCopies.runId, copy.runId)); + const shell = vi.spyOn(executionTargetTools, "runAdapterExecutionTargetShellCommand").mockRejectedValue(new Error("must not execute")); + try { + await copies.recoverStopped(); + const recovered = (await copies.get(companyId, copy.runId))!; + if (scenario === "destroyed") { + expect(recovered).toMatchObject({ state: "unavailable", errorCode: "INSTRUCTION_COLLECTION_UNAVAILABLE", + receipt: { cleanupPending: false }, nextAttemptAt: null }); + expect(recovered.processStoppedAt).toBeInstanceOf(Date); + await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" }); + } else { + expect(recovered).toMatchObject({ state: "pending_collection", processStoppedAt: null }); + expect(await fs.readFile(path.join(copy.localRoot, entryFile), "utf8")).toBe(initial); + } + expect(shell).not.toHaveBeenCalled(); + expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial); + const [historical] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)); + expect(historical!.metadata).not.toHaveProperty("remoteExecutionTermination"); + } finally { shell.mockRestore(); } + }); + it.each([undefined, "stopped", "companyId", "runId", "leaseId", "provider", "providerLeaseId"])( "defers unavailable cleanup without exact destruction proof (%s)", async proof => { const { copy } = await unavailableRemote({ state: proof === undefined ? undefined : proof === "stopped" ? "stopped" : "destroyed", diff --git a/server/src/services/agent-directory-working-copies.ts b/server/src/services/agent-directory-working-copies.ts index c1fe658aab6..b1c8de72e5d 100644 --- a/server/src/services/agent-directory-working-copies.ts +++ b/server/src/services/agent-directory-working-copies.ts @@ -2,7 +2,7 @@ import { notifyInstructionCollection } from "./instruction-collection-notificati import fs from "node:fs/promises"; import path from "node:path"; import { isDeepStrictEqual } from "node:util"; -import { and, eq, inArray, isNull, sql } from "drizzle-orm"; +import { and, eq, inArray, isNull, sql, type SQL } from "drizzle-orm"; import { agents, environmentLeases, environments, agentInstructionWorkingCopies as copies, withDedicatedDbConnection, type Db } from "@paperclipai/db"; import { syncDirectoryToSsh, restoreWorkspaceFromSshExecution } from "@paperclipai/adapter-utils/ssh"; import { prepareAdapterExecutionTargetRuntime, runAdapterExecutionTargetShellCommand, type AdapterExecutionTarget, type PreparedAdapterExecutionTargetRuntime } from "@paperclipai/adapter-utils/execution-target"; @@ -16,7 +16,7 @@ import type { AuthorizationActor } from "./authorization.js"; import type { EnvironmentRuntimeService } from "./environment-runtime.js"; import type { Environment, EnvironmentLease } from "@paperclipai/shared"; import { agentDirectoryBaselineDigest, agentDirectoryProbeProgram, observeLocalAgentDirectory } from "./agent-directory-probe.js"; -import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; +import { hasRemoteTerminationReceipt, resolveRemoteTerminationLease } from "./remote-execution-termination.js"; import { cachedAgentFileManifest, captureAgentFileCheckpoint, checkpointBaseline, checkpointSnapshot, type AgentFileManifest } from "./agent-file-checkpoints.js"; import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js"; import { logger } from "../middleware/logger.js"; @@ -46,7 +46,7 @@ function baseline(row: Copy) { function actor(row: Copy): AuthorizationActor { return { type: "agent", companyId: row.companyId, agentId: row.agentId, runId: row.runId, onBehalfOfUserId: row.responsibleUserId }; } -export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string, runId: string) => Promise, patch: (row: Copy, values: Partial) => Promise, environmentRuntime?: EnvironmentRuntimeService) { +export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string, runId: string) => Promise, patch: (row: Copy, values: Partial, condition?: SQL) => Promise, environmentRuntime?: EnvironmentRuntimeService) { const store = agentFileStore(db); async function insertCopy(values: typeof copies.$inferInsert) { return db.transaction(async tx => { @@ -503,7 +503,15 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string return row.state === "superseded" || !row.processStoppedAt || completed.has(row.state) && row.receipt?.cleanupPending === false; } + async function settleCompletedCleanupRetry(row: Copy) { + if (completed.has(row.state) && row.processStoppedAt && !row.receipt?.retainedByRunId && row.receipt?.cleanupPending === false && row.nextAttemptAt) { + return patch(row, { nextAttemptAt: null }, sql`${copies.receipt} = ${JSON.stringify(row.receipt)}::jsonb + AND ${copies.processStoppedAt} IS NOT NULL`); + } + return row; + } async function release(row: Copy, target?: AdapterExecutionTarget | null) { + row = await settleCompletedCleanupRetry(row); if (releaseIsNoop(row) || !await owns(row)) return; const destroyedOnly = row.receipt?.cleanupDestroyedOnly === true; if (row.receipt?.retainedByRunId || row.state === "superseded" || !await owns(row) || !row.processStoppedAt) return; @@ -559,7 +567,7 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string : eq(environmentLeases.environmentId, row.location.slice("remote:".length)), )); if (leases.length !== 1) return false; - const lease = leases[0]!; + const lease = await resolveRemoteTerminationLease(db, leases[0]!); if (lease.environmentId !== null && row.location !== `remote:${lease.environmentId}`) return false; const remoteCwd = cleanup?.remoteCwd ?? lease.metadata?.remoteCwd; return typeof remoteCwd === "string" && @@ -588,7 +596,7 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string cleanup?.leaseId ? eq(environmentLeases.id, cleanup.leaseId) : eq(environmentLeases.environmentId, row.location.slice("remote:".length)), )); if (leases.length !== 1) return null; - const lease = leases[0]!; + const lease = await resolveRemoteTerminationLease(db, leases[0]!); const destroyedWithoutEnvironment = lease.environmentId === null && hasRemoteTerminationReceipt(lease) && (lease.metadata?.remoteExecutionTermination as { state?: string } | undefined)?.state === "destroyed"; if (row.location !== `remote:${lease.environmentId}` && !destroyedWithoutEnvironment) return null; @@ -658,10 +666,12 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string checkpointWarm: (row: Copy, target?: AdapterExecutionTarget | null) => serial(row, current => current.receipt?.warm === true ? checkpoint(current, target) : Promise.resolve(current), "agent_directory_checkpoint"), collectStopped: (row: Copy, target?: AdapterExecutionTarget | null) => serial(row, current => collectStopped(current, target), "agent_directory_collect"), release: async (row: Copy) => { - const current = await get(row.companyId, row.runId); + let current = await get(row.companyId, row.runId); + if (!current) return; + current = await settleCompletedCleanupRetry(current); // This check changes no receipt or bytes. A concurrent stop merely leaves // cleanup to its owner or the recovery sweep; it grants no new authority. - if (!current || releaseIsNoop(current)) return; + if (releaseIsNoop(current)) return; await serial(current, release, "agent_directory_release"); }, }; diff --git a/server/src/services/agent-instruction-working-copies.ts b/server/src/services/agent-instruction-working-copies.ts index 0d990331d4a..586bd63dbfb 100644 --- a/server/src/services/agent-instruction-working-copies.ts +++ b/server/src/services/agent-instruction-working-copies.ts @@ -129,8 +129,8 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen }; return db.transaction(tx => write(tx as unknown as Db)); } - async function patch(row: Copy, values: Partial) { - const updated = await tryPatch(row, values); + async function patch(row: Copy, values: Partial, condition?: SQL) { + const updated = await tryPatch(row, values, false, condition); // A late cleanup must not overwrite an explicit resolution or a newer // baseline. Canonical content has its own independent head CAS. return updated ?? (await get(row.companyId, row.runId))!; @@ -499,8 +499,8 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen function directoryCleanupEntry(row: Copy | null): DirectoryCleanupEntry | null { return row && ["saved", "unchanged", "resolved", "unavailable"].includes(row.state) && row.processStoppedAt && isAgentDirectoryCopy(row) && !row.receipt?.retainedByRunId - && row.receipt?.cleanupPending !== false - && (Object.hasOwn(row.receipt!, "baseline") || row.receipt?.cleanupPending === true) + && (row.receipt?.cleanupPending === false ? row.nextAttemptAt !== null + : Object.hasOwn(row.receipt!, "baseline") || row.receipt?.cleanupPending === true) ? { id: row.runId, companyId: row.companyId, dueAt: row.nextAttemptAt?.getTime() ?? 0 } : null; } function directoryCleanupPredicate() { @@ -508,8 +508,9 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen AND ${copies.processStoppedAt} IS NOT NULL AND ${copies.receipt}->>'schema' = 'paperclip.agent-files.v1' AND coalesce(${copies.receipt}->>'retainedByRunId', '') = '' - AND ${copies.receipt}->>'cleanupPending' IS DISTINCT FROM 'false' - AND (${copies.receipt} ? 'baseline' OR ${copies.receipt}->>'cleanupPending' = 'true')`; + AND ((${copies.receipt}->>'cleanupPending' = 'false' AND ${copies.nextAttemptAt} IS NOT NULL) + OR (${copies.receipt}->>'cleanupPending' IS DISTINCT FROM 'false' + AND (${copies.receipt} ? 'baseline' OR ${copies.receipt}->>'cleanupPending' = 'true')))`; } async function loadDirectoryCleanup(scope: DirectoryCleanupScope | undefined, admitted: () => boolean) { const entries: DirectoryCleanupEntry[] = []; diff --git a/server/src/services/remote-execution-termination.ts b/server/src/services/remote-execution-termination.ts index f9fae854c69..d30b60ae697 100644 --- a/server/src/services/remote-execution-termination.ts +++ b/server/src/services/remote-execution-termination.ts @@ -1,5 +1,8 @@ import { and, eq } from "drizzle-orm"; import { environmentLeases, type Db } from "@paperclipai/db"; +import { retiredReusableLeasePredecessors } from "./environment-lease-handoff.js"; + +type Lease = typeof environmentLeases.$inferSelect; type LeaseIdentity = { id: string; companyId: string; heartbeatRunId: string | null; @@ -39,12 +42,36 @@ export function remoteLeaseCleanupScope(lease: Pick { + if (hasRemoteTerminationReceipt(lease) || lease.status !== "expired" || + lease.leasePolicy !== "reuse_by_environment" || !lease.provider || !lease.providerLeaseId) return lease; + const history = await db.select().from(environmentLeases).where(and( + eq(environmentLeases.provider, lease.provider), eq(environmentLeases.providerLeaseId, lease.providerLeaseId), + )); + const current = history.find(row => row.id === lease.id); + if (!current || current.companyId !== lease.companyId || current.heartbeatRunId !== lease.heartbeatRunId) return lease; + const tips = history.filter(row => !Object.hasOwn(row.metadata ?? {}, "reusableLeaseSuccessor")); + if (tips.length !== 1) return current; + const tip = tips[0]!; + const receipt = tip.metadata?.remoteExecutionTermination as Record | undefined; + if (!hasRemoteTerminationReceipt(tip) || receipt?.state !== "destroyed") return current; + const predecessors = retiredReusableLeasePredecessors(tip, history.filter(row => row.id !== tip.id)); + if (!predecessors?.some(row => row.id === current.id)) return current; + const termination = remoteTerminationReceipt(current, { providerLeaseId: current.providerLeaseId, state: "destroyed" }); + if (!termination) return current; + return { ...current, metadata: { ...current.metadata, remoteExecutionTermination: { + ...termination, confirmedAt: receipt.confirmedAt, + } } }; +} + export async function stoppedRemoteCleanupScopes(db: Db, companyId: string, runId: string) { const leases = await db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId), )); - if (leases.length === 0 || !leases.every(hasRemoteTerminationReceipt)) return null; - return [...new Set(leases.map(lease => remoteLeaseCleanupScope(lease)!))]; + if (leases.length === 0) return null; + const resolved = await Promise.all(leases.map(lease => resolveRemoteTerminationLease(db, lease))); + if (!resolved.every(hasRemoteTerminationReceipt)) return null; + return [...new Set(resolved.map(lease => remoteLeaseCleanupScope(lease)!))]; } export async function remoteExecutionHasStopped(db: Db, companyId: string, runId: string) {