From afe86e9dedf1197737f5cae3249017a1f4987a9f Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Thu, 12 Nov 2026 20:20:45 -0800 Subject: [PATCH] fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A plugin environment runs each Paperclip Runner turn as a provider task, and a follow-up run can continue the same Claude/ACPX conversation from the saved checkpoint > - Core authorizes that continuation only when the predecessor run's remote execution has stopped (`remoteExecutionHasStopped`), so two Runners never write one checkpoint > - The predecessor's own settlement already proves the stop: the provider reports `executionStopped`, and Core then completes the task > - On a per-turn (ephemeral) lease, Core records the stop receipt only when it releases the lease, and that release can be held for a long time (for example by a pending workspace export) > - In that window, an ordinary follow-up fails with `adapter_failed: Plugin Runner retained session is not authorized`, even though the stop is already proven > - This pull request records the stop receipt when the stop is proven, and the stop check accepts it before the release > - The benefit is that follow-ups continue as soon as the predecessor is proven stopped, and the release gates keep their meaning ## Linked Issues or Issue Description Fixes #21211 Refs #21214 and #21223. Those cover the creation fence for same-run recovery, which still requires a released lease. Production evidence for this change: in both observed failures, the predecessor was a per-turn plugin run. Its `task.settle` completed with all provider calls successful. Core then did not release its lease for 30 minutes or more, and no release call was made in that time. The follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint` → `resolvePluginTaskResume`) within seconds of admission. ## What Changed - `server/src/services/environment-task-runtime.ts` - After a recorded provider stop, `complete` writes the bound stop receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the current lease for every lease policy. Before, it did this only for `reuse_by_environment` leases. Reusable tasks still confirm every owner and still move retained owners to released. - New work on a lease (`submit`, `copilot_metadata`, first portable resume) removes the stop receipt in the same write that clears `runnerTaskExecutionStopped`. A replacement physical task does not copy the old receipt. - `server/src/services/remote-execution-termination.ts` - New `hasBoundRemoteStopReceipt` checks only the receipt binding (company, run, lease, provider and physical task). `hasRemoteTerminationReceipt` uses it and keeps its released-only semantics. Resource cleanup gates do not change. - `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a lease that is released with its receipt, or that is still `active` (not released) with a bound stop receipt. A lease whose release started and did not settle does not count. After release starts, the release owns the receipt. - New `remoteExecutionHasReleased` gives the release-based answer. - `server/src/services/environment-run-orchestrator.ts`: same-run plugin recovery uses `remoteExecutionHasReleased`. The same-run resume check after it requires released receipts, so its behavior does not change. - `resolvePluginTaskResume` keeps its checks and their order. Its refusal now names the first failing check: `Plugin Runner retained session is not authorized (reason: )`, for example `predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The old message stays as the prefix, and no ids go into it. The stale-availability recheck in `environment-task-runtime.ts` names its refusal the same way. Production telemetry for #21211 could not tell which check fired. The next occurrence will show it. - `doc/execution-semantics.md` and `doc/testing-native-remote-continuity.md` describe the stop receipt and when it is recorded. Caller audit for the changed meaning of `remoteExecutionHasStopped`: | Caller | Needs | Decision | | --- | --- | --- | | `agent-instruction-working-copies.ts` | stop | Accepts. Directory cleanup still requires a released "destroyed" receipt. | | `environment-run-orchestrator.ts` | release | Moved to `remoteExecutionHasReleased`. | | `plugin-task-resume.ts` | stop | Accepts. This is the fix. | | `native-runtime/native-runner-ownership.ts` | stop | Accepts. `hasPendingRunCleanup` still counts the active lease as pending. | | `native-runtime/preserved-native-result-finalization.ts` | stop | Accepts. Workspace recovery has its own gate. | | `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts. `hasPendingRunCleanup` still holds active leases. | | `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop acknowledgement and quarantined ownership retirement. Holds and blockers still apply on the normal path. | | `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors still fail it. | ## Verification From `server/`, after `node scripts/agent-preflight.mjs`: - `npx vitest run src/__tests__/environment-task-authority.integration.test.ts`: 10/10 pass. The 2 new tests fail without the fix. - The plugin task and environment runtime suites (`environment-task-authority.integration`, `plugin-runner-task.integration`, `plugin-warm-tasks.integration`, `environment-task-runtime`, `plugin-task-source-recovery.integration`, `environment-runtime`): 6 files, 366 pass. - The audited caller suites (`agent-instruction-working-copies`, case handoff legacy and native, `environment-run-orchestrator`, `heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`, `native-remote-retry-admission`, `preserved-native-response-wait`): 9 files, 806 pass. - `pnpm --filter @paperclipai/server typecheck`: pass. The new integration tests cover: - `complete` writes a bound stop receipt on a still-active ephemeral lease. `remoteExecutionHasStopped` is true and `hasRemoteTerminationReceipt` is still false. - `resolvePluginTaskResume` (inspect and launch) authorizes a follow-up's new task, and the real host `read_state` checkpoint read succeeds, while the predecessor's release is held. - Refusals: no provider stop, a receipt for another provider task, an unstopped sibling lease, and a release in progress. - New work on the lease removes the receipt. Release overwrites the receipt with its own receipt. Warm owners keep their behavior. I did not add an executor-level test. `native-session-executor.test.ts` mocks `resolvePluginTaskResume`, so the integration test runs the real checkpoint read path instead. I did not run the full suite locally. CI runs it. ## Risks - `remoteExecutionHasStopped` now returns true earlier for a run whose leases are still held. Each caller is audited above. Callers that need released resources use the release-based predicates. - The stop receipt can now be on an `active` lease. The SQL readers of `remoteExecutionTermination` (cost accounting, execution blockers, idle-work proof, orphan settlement) also require a released status. `pinFailedObservedWorkspace` treats an active ephemeral lease with a receipt as not pinnable. That path applies only to the remote-workspace flow and falls back to `preserveNativeFailedWorkspace`. - Leases that settled before this change have no early receipt. They continue to wait for their release. - Nothing compares the refusal message exactly. Matchers use substrings, and run failure classification does not use the message. - A follow-up still fails closed if the predecessor's stop was never proven. This change does not add a recoverable wait. - The reason the release was held in production is not addressed here. - No migration and no API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, sub-agents, Honeycomb trace queries) and extended thinking. One sub-agent wrote the change. A second session reviewed it before this pull request. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 17 ++ doc/testing-native-remote-continuity.md | 8 +- ...ronment-task-authority.integration.test.ts | 251 +++++++++++++++++- .../services/environment-run-orchestrator.ts | 6 +- .../src/services/environment-task-runtime.ts | 29 +- server/src/services/plugin-task-resume.ts | 85 ++++-- .../services/remote-execution-termination.ts | 59 +++- 7 files changed, 396 insertions(+), 59 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 29aefe9693b..1bb81b0e687 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -3483,6 +3483,23 @@ agent, workspace, retained session, and termination of prior execution. The checkpoint must match the exact Runner, lease, run, session, turn, and item. A missing restart claim does not remove these checks or authorize a fresh session. +A plugin task records its stop receipt when it settles. When the provider +reports that the task stopped, completing the task writes a receipt bound to +the exact company, run, lease, provider and task, even before the lease is +released. Lease release can be held long after that, for example by a pending +workspace export. A per-turn follow-up in a later run can then resume the +retained checkpoint: for continuation, every lease of the predecessor run must +be released with its termination receipt or still be active with this stop +receipt and its release not started. Once release or destroy of a lease starts, +release owns the receipt: the lease does not count until release settles with +its own receipt, because the provider can be releasing the task's resources +while the lease is still active. Dispatching new work on a lease removes its +stop receipt. Resource cleanup gates, and a same-run retry's replacement of earlier +leases, still require the released termination receipt. +When the server refuses the resume, the error keeps the message +`Plugin Runner retained session is not authorized` and names the first failed +check, for example `(reason: predecessor_not_stopped)`. + These checks can refuse the exact checkpoint before the retry claims a new provider attempt. For example, the interrupted attempt can leave the provider state not suspended. The run error then names the interrupted attempt first, as diff --git a/doc/testing-native-remote-continuity.md b/doc/testing-native-remote-continuity.md index 2e98d99556f..3631fa6dcb3 100644 --- a/doc/testing-native-remote-continuity.md +++ b/doc/testing-native-remote-continuity.md @@ -80,7 +80,13 @@ The executor tests cover Claude and Codex, including reopening blocked work. The lease-authority integration test uses PostgreSQL to check that checkpoint discovery cannot launch work or bypass a termination receipt. A stale negative availability flag requires a fresh provider status before the checkpoint can -be read for continuation. +be read for continuation. It also checks that a follow-up can resume while the +predecessor's lease release is held, using the stop receipt that the task +recorded when it settled. A receipt for another task, a second predecessor +lease without stop proof, or a release that has started and not settled still +refuses continuation. The test holds a real lease release inside the provider +call and checks that the follow-up is refused until that release settles. +Each refusal is checked by the reason that it names. The executor can restore one quarantined journal in the same session scope. It requires a terminal prior run, completed commands, a terminal event, and an diff --git a/server/src/__tests__/environment-task-authority.integration.test.ts b/server/src/__tests__/environment-task-authority.integration.test.ts index ffb37044169..4b0c151513e 100644 --- a/server/src/__tests__/environment-task-authority.integration.test.ts +++ b/server/src/__tests__/environment-task-authority.integration.test.ts @@ -3,7 +3,9 @@ import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { eq } from "drizzle-orm"; import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns, plugins, sandboxCreationOperations } from "@paperclipai/db"; import { resolvePluginTaskResume } from "../services/environment-task-runtime.js"; -import { remoteTerminationReceipt } from "../services/remote-execution-termination.js"; +import type { PluginTaskResumeRefusal } from "../services/plugin-task-resume.js"; +import { hasBoundRemoteStopReceipt, hasRemoteTerminationReceipt, remoteExecutionHasReleased, remoteExecutionHasStopped, + remoteTerminationReceipt } from "../services/remote-execution-termination.js"; import { environmentRuntimeService } from "../services/environment-runtime.js"; import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; import { createPluginRunnerTaskOwner } from "../services/native-runtime/plugin-runner-task.js"; @@ -12,6 +14,8 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro const support = await getEmbeddedPostgresTestSupport(); const databaseSuite = support.supported ? describe : describe.skip; +/** A resume refusal keeps its original message and names the check that failed. */ +const refused = (reason: PluginTaskResumeRefusal) => `Plugin Runner retained session is not authorized (reason: ${reason})`; databaseSuite("environment task lease authority", () => { let db: ReturnType; @@ -227,17 +231,29 @@ databaseSuite("environment task lease authority", () => { await expect(authorize()).resolves.toBe("old-task"); for (const field of ["runnerInstanceId", "environmentLeaseId", "normalizedSessionId", "runId"]) { await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, - priorIdentity: { ...priorIdentity, [field]: randomUUID() } })).rejects.toThrow("not authorized"); + priorIdentity: { ...priorIdentity, [field]: randomUUID() } })).rejects.toThrow(refused("predecessor_candidate_missing")); } await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, oldRunId)); - await expect(authorize()).rejects.toThrow("not authorized"); + await expect(authorize()).rejects.toThrow(refused("predecessor_run_active")); await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, oldRunId)); - await expect(resolvePluginTaskResume(db, { companyId: randomUUID(), leaseId: nextLeaseId, priorIdentity })).rejects.toThrow("not authorized"); + await expect(resolvePluginTaskResume(db, { companyId: randomUUID(), leaseId: nextLeaseId, priorIdentity })).rejects.toThrow(refused("next_lease_missing")); await db.update(environmentLeases).set({ metadata: { ...pin, pluginId: randomUUID() } }).where(eq(environmentLeases.id, nextLeaseId)); - await expect(authorize()).rejects.toThrow("not authorized"); + await expect(authorize()).rejects.toThrow(refused("provider_mismatch")); await db.update(environmentLeases).set({ metadata: pin }).where(eq(environmentLeases.id, nextLeaseId)); + await db.update(environmentLeases).set({ providerLeaseId: "old-task" }).where(eq(environmentLeases.id, nextLeaseId)); + await expect(authorize()).rejects.toThrow(refused("same_provider_task")); + await db.update(environmentLeases).set({ providerLeaseId: "new-task" }).where(eq(environmentLeases.id, nextLeaseId)); + const otherEnvironmentId = randomUUID(); + await db.insert(environments).values({ id: otherEnvironmentId, name: `Fixture ${otherEnvironmentId}`, driver: "plugin", config: {} }); + await db.update(environmentLeases).set({ environmentId: otherEnvironmentId }).where(eq(environmentLeases.id, nextLeaseId)); + await expect(authorize()).rejects.toThrow(refused("scope_mismatch")); + await db.update(environmentLeases).set({ environmentId }).where(eq(environmentLeases.id, nextLeaseId)); + await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, nextRunId)); + await expect(authorize()).rejects.toThrow(refused("next_run_not_running")); + await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, nextRunId)); + await expect(authorize()).resolves.toBe("old-task"); await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, oldLeaseId)); - await expect(authorize()).rejects.toThrow("not authorized"); + await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped")); await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId)); let sessionStateAvailable = false; const operations: string[] = []; @@ -259,8 +275,8 @@ databaseSuite("environment task lease authority", () => { operation: { kind: "ensure_ownership", operationId: portableTargetLeaseId, sourceTaskId: "old-task", sourceGeneration: "source-generation", targetGeneration: "target-generation", policy: "ordinary", identity: checkpointIdentity }, } } }).where(eq(environmentLeases.id, oldLeaseId)); - await expect(authorize()).rejects.toThrow("not authorized"); - await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).rejects.toThrow("not authorized"); + await expect(authorize()).rejects.toThrow(refused("predecessor_candidate_missing")); + await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).rejects.toThrow(refused("predecessor_candidate_missing")); await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: { kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task", } })).rejects.toThrow("not authorized"); @@ -269,19 +285,19 @@ databaseSuite("environment task lease authority", () => { } })).rejects.toThrow("not authorized"); expect(call).not.toHaveBeenCalled(); await db.update(environmentLeases).set({ metadata: { ...source.metadata, runnerTaskSessionRetained: false } }).where(eq(environmentLeases.id, oldLeaseId)); - await expect(authorize()).rejects.toThrow("not authorized"); + await expect(authorize()).rejects.toThrow(refused("predecessor_candidate_missing")); await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).resolves.toBe("old-task"); const inspect = () => runtime.task({ companyId, leaseId: nextLeaseId, operation: { kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task", } }); - await expect(inspect()).rejects.toThrow("not authorized"); - await expect(authorize()).rejects.toThrow("not authorized"); + await expect(inspect()).rejects.toThrow(refused("checkpoint_status_unconfirmed")); + await expect(authorize()).rejects.toThrow(refused("predecessor_candidate_missing")); // Discovery cannot launch, and cannot bypass the original termination receipt. await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: { kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task", } })).rejects.toThrow("resume identity mismatch"); await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, oldLeaseId)); - await expect(inspect()).rejects.toThrow("not authorized"); + await expect(inspect()).rejects.toThrow(refused("predecessor_not_stopped")); expect(operations).toEqual(["status"]); await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId)); sessionStateAvailable = true; @@ -350,4 +366,215 @@ databaseSuite("environment task lease authority", () => { }); + async function seedTaskPlugin(name: string) { + const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID(); + await db.insert(companies).values({ id: companyId, name, issuePrefix: `T${companyId.slice(0, 7).replace(/-/g, "").toUpperCase()}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", adapterType: "paperclip_runner" }); + await db.insert(environments).values({ id: environmentId, name: `Fixture ${environmentId}`, driver: "plugin", + config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } }); + await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: `fixture-${name}`, version: "1.0.0", status: "ready", manifestJson: { + capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }], + } as never }); + const pin = { driver: "plugin", pluginId, driverKey: "tasks", environmentId, runnerTaskLease: true, + runnerTaskProjectAuthValidation: { configurationId: "fixture", provider: "openai", projectIds: [] } }; + return { companyId, agentId, environmentId, pin }; + } + + it("records a per-turn task's stop receipt when it settles, before its held lease release", async () => { + const { companyId, agentId, environmentId, pin } = await seedTaskPlugin("per-turn-stop"); + const oldRunId = randomUUID(), nextRunId = randomUUID(), oldLeaseId = randomUUID(), nextLeaseId = randomUUID(); + await db.insert(heartbeatRuns).values([ + { id: oldRunId, companyId, agentId, invocationSource: "manual", status: "running" }, + { id: nextRunId, companyId, agentId, invocationSource: "manual", status: "running" }, + ]); + const prior = { id: oldLeaseId, companyId, environmentId, heartbeatRunId: oldRunId, provider: "plugin:fixture:tasks", providerLeaseId: "old-task" }; + const oldIdentity = { runnerId: "runner", leaseId: oldLeaseId, runId: oldRunId, sessionId: "session", turnId: "prior-turn", itemId: "prior-item" }; + await db.insert(environmentLeases).values([ + { ...prior, status: "active", leasePolicy: "ephemeral", metadata: { ...pin, runnerTaskDispatched: true, + runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false, runnerTaskIdentity: oldIdentity } }, + { id: nextLeaseId, companyId, environmentId, heartbeatRunId: nextRunId, provider: prior.provider, providerLeaseId: "new-task", + status: "active", leasePolicy: "ephemeral", metadata: pin }, + ]); + let executionStopped = false; + const call = vi.fn(async (_id: string, method: string, params: any) => method !== "environmentTask" ? undefined + : ["read_state", "archive_state"].includes(params.operation.kind) + ? { kind: "state", taskId: params.taskId, state: { schema: "paperclip.runner.durable.state.v1", lifecycle: "suspended", + ...(params.operation.identity ?? params.operation.priorIdentity) } } + : params.operation.kind === "status" + ? { kind: "status", taskId: params.taskId, phase: executionStopped ? "completed" : "running", executionStopped, sessionStateAvailable: executionStopped } + : { kind: "accepted", taskId: params.taskId }); + const workers = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentTask", "environmentReleaseLease"] }), call } as unknown as PluginWorkerManager; + const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers }); + const readLease = async (id: string) => (await db.select().from(environmentLeases).where(eq(environmentLeases.id, id)))[0]!; + const priorIdentity = { runnerInstanceId: "runner", normalizedSessionId: "session", environmentLeaseId: oldLeaseId, runId: oldRunId }; + const checkpointIdentity = { ...priorIdentity, turnId: "prior-turn", itemId: "prior-item" }; + const runner = { runnerId: "runner", leaseId: oldLeaseId, sessionId: "session", runId: nextRunId, + turnId: "turn", itemId: "item", harness: "codex", protocolMin: 1, protocolMax: 3 }; + const authorize = (inspectCheckpoint?: boolean) => resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint }); + + // Neither a running status nor a completion without a provider stop is stop proof. + await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "status" } }); + await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "complete" } }); + expect((await readLease(oldLeaseId)).metadata).not.toHaveProperty("remoteExecutionTermination"); + expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(false); + + // The turn checkpoint stops and completes the task; release of the lease is still held. + executionStopped = true; + await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "stop" } }); + await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "status" } }); + await runtime.task({ companyId, leaseId: oldLeaseId, operation: { kind: "complete" } }); + await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, oldRunId)); + const settled = await readLease(oldLeaseId); + expect(settled).toMatchObject({ status: "active", releasedAt: null, cleanupStatus: null, metadata: { + runnerTaskExecutionStopped: true, runnerTaskSessionRetained: true, runnerTaskDispatched: false, + remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId, runId: oldRunId, leaseId: oldLeaseId, + provider: prior.provider, providerLeaseId: "old-task", state: "stopped" } } }); + expect(hasBoundRemoteStopReceipt(settled)).toBe(true); + expect(hasRemoteTerminationReceipt(settled)).toBe(false); + expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(true); + expect(await remoteExecutionHasReleased(db, companyId, oldRunId)).toBe(false); + + // The follow-up can inspect and launch from the checkpoint while the predecessor is unreleased. + await expect(authorize(true)).resolves.toBe("old-task"); + await expect(authorize()).resolves.toBe("old-task"); + await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: { + kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task", + } })).resolves.toMatchObject({ kind: "state", state: { ...checkpointIdentity, lifecycle: "suspended" } }); + + // A receipt for another physical task, a second unstopped predecessor lease, or a release + // that started without settling still refuses authority. + await db.update(environmentLeases).set({ metadata: { ...settled.metadata, remoteExecutionTermination: + remoteTerminationReceipt({ ...prior, providerLeaseId: "other-task" }, { providerLeaseId: "other-task", state: "stopped" }) } }) + .where(eq(environmentLeases.id, oldLeaseId)); + await expect(authorize(true)).rejects.toThrow(refused("predecessor_not_stopped")); + await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped")); + await db.update(environmentLeases).set({ metadata: settled.metadata }).where(eq(environmentLeases.id, oldLeaseId)); + const siblingId = randomUUID(); + await db.insert(environmentLeases).values({ id: siblingId, companyId, environmentId, heartbeatRunId: oldRunId, + provider: prior.provider, providerLeaseId: "sibling-task", status: "active", leasePolicy: "ephemeral", metadata: pin }); + await expect(authorize(true)).rejects.toThrow(refused("predecessor_not_stopped")); + await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped")); + await db.delete(environmentLeases).where(eq(environmentLeases.id, siblingId)); + await db.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", releasedAt: new Date() }) + .where(eq(environmentLeases.id, oldLeaseId)); + await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped")); + await db.update(environmentLeases).set({ status: "active", cleanupStatus: null, releasedAt: null }).where(eq(environmentLeases.id, oldLeaseId)); + await expect(authorize()).resolves.toBe("old-task"); + + // Dispatching new execution invalidates a stop receipt on that lease. + const next = await readLease(nextLeaseId); + await db.update(environmentLeases).set({ metadata: { ...next.metadata, runnerTaskExecutionStopped: true, + remoteExecutionTermination: remoteTerminationReceipt(next, { providerLeaseId: "new-task", state: "stopped" }) } }) + .where(eq(environmentLeases.id, nextLeaseId)); + expect(await remoteExecutionHasStopped(db, companyId, nextRunId)).toBe(true); + await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: { + kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task", + } })).resolves.toMatchObject({ kind: "accepted" }); + expect((await readLease(nextLeaseId)).metadata).not.toHaveProperty("remoteExecutionTermination"); + expect(await remoteExecutionHasStopped(db, companyId, nextRunId)).toBe(false); + + // Release still writes its own receipt over the settlement receipt. + const releaseErrors: unknown[] = []; + await runtime.releaseRunLeases(oldRunId, "released", (_id, error) => { releaseErrors.push(error); }); + expect(releaseErrors).toEqual([]); + const released = await readLease(oldLeaseId); + expect(released).toMatchObject({ status: "released", cleanupStatus: "success", + metadata: { remoteExecutionTermination: { leaseId: oldLeaseId, providerLeaseId: "old-task", state: "stopped" } } }); + expect(hasRemoteTerminationReceipt(released)).toBe(true); + expect(await remoteExecutionHasReleased(db, companyId, oldRunId)).toBe(true); + }); + + it("stops counting a settled task's stop receipt once its lease release starts", async () => { + const { companyId, agentId, environmentId, pin } = await seedTaskPlugin("release-owns-receipt"); + const oldRunId = randomUUID(), nextRunId = randomUUID(), oldLeaseId = randomUUID(), nextLeaseId = randomUUID(); + await db.insert(heartbeatRuns).values([ + { id: oldRunId, companyId, agentId, invocationSource: "manual", status: "succeeded" }, + { id: nextRunId, companyId, agentId, invocationSource: "manual", status: "running" }, + ]); + const prior = { id: oldLeaseId, companyId, environmentId, heartbeatRunId: oldRunId, provider: "plugin:fixture:tasks", providerLeaseId: "old-task" }; + // The predecessor's task settled in its run: stopped, session retained, stop receipt recorded. + const settledMetadata = { ...pin, runnerTaskDispatched: false, runnerTaskExecutionStopped: true, runnerTaskSessionRetained: true, + runnerTaskIdentity: { runnerId: "runner", leaseId: oldLeaseId, runId: oldRunId, sessionId: "session", turnId: "turn", itemId: "item" }, + remoteExecutionTermination: remoteTerminationReceipt(prior, { providerLeaseId: "old-task", state: "stopped" }) }; + await db.insert(environmentLeases).values([ + { ...prior, status: "active", leasePolicy: "ephemeral", metadata: settledMetadata }, + { id: nextLeaseId, companyId, environmentId, heartbeatRunId: nextRunId, provider: prior.provider, providerLeaseId: "new-task", + status: "active", leasePolicy: "ephemeral", metadata: pin }, + ]); + let releaseEntered!: () => void, finishRelease!: () => void; + const entered = new Promise(resolve => { releaseEntered = resolve; }); + const held = new Promise(resolve => { finishRelease = resolve; }); + const call = vi.fn(async (_id: string, method: string) => { + if (method !== "environmentReleaseLease") throw new Error(`unexpected ${method}`); + releaseEntered(); + await held; + }); + const workers = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentTask", "environmentReleaseLease"] }), call } as unknown as PluginWorkerManager; + const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers }); + const priorIdentity = { runnerInstanceId: "runner", normalizedSessionId: "session", environmentLeaseId: oldLeaseId, runId: oldRunId }; + const authorize = (inspectCheckpoint?: boolean) => resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint }); + await expect(authorize()).resolves.toBe("old-task"); + + // Each marker that release, destroy, or startup and creation cleanup write first takes the receipt over. + for (const marker of [{ pluginLeaseCleanupMethod: "release" }, { pluginLeaseCleanupMethod: "destroy" }, { runnerTaskRetiring: true }]) { + await db.update(environmentLeases).set({ metadata: { ...settledMetadata, ...marker } }).where(eq(environmentLeases.id, oldLeaseId)); + expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(false); + await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped")); + } + await db.update(environmentLeases).set({ metadata: settledMetadata }).where(eq(environmentLeases.id, oldLeaseId)); + + // Hold the real release inside the provider call: the lease is still active with its settlement receipt. + const releaseErrors: unknown[] = []; + const releasing = runtime.releaseRunLeases(oldRunId, "released", (_id, error) => { releaseErrors.push(error); }); + await entered; + const inFlight = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId)))[0]!; + expect(inFlight).toMatchObject({ status: "active", releasedAt: null, + metadata: { pluginLeaseCleanupMethod: "release", runnerTaskRetiring: true } }); + expect(hasBoundRemoteStopReceipt(inFlight)).toBe(true); + expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(false); + await expect(authorize(true)).rejects.toThrow(refused("predecessor_not_stopped")); + await expect(authorize()).rejects.toThrow(refused("predecessor_not_stopped")); + + // Release settles with its own receipt, which authorizes the follow-up. + finishRelease(); + await releasing; + expect(releaseErrors).toEqual([]); + const released = (await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId)))[0]!; + expect(released).toMatchObject({ status: "released", cleanupStatus: "success" }); + expect(hasRemoteTerminationReceipt(released)).toBe(true); + expect(await remoteExecutionHasStopped(db, companyId, oldRunId)).toBe(true); + await expect(authorize(true)).resolves.toBe("old-task"); + await expect(authorize()).resolves.toBe("old-task"); + }); + + it("still records stop receipts for every warm owner and retires retained owners", async () => { + const { companyId, agentId, environmentId, pin } = await seedTaskPlugin("warm-stop"); + const priorRunId = randomUUID(), currentRunId = randomUUID(), priorLeaseId = randomUUID(), currentLeaseId = randomUUID(); + await db.insert(heartbeatRuns).values([ + { id: priorRunId, companyId, agentId, invocationSource: "manual", status: "succeeded" }, + { id: currentRunId, companyId, agentId, invocationSource: "manual", status: "running" }, + ]); + const shared = { companyId, environmentId, provider: "plugin:fixture:tasks", providerLeaseId: "warm-task", leasePolicy: "reuse_by_environment" as const }; + await db.insert(environmentLeases).values([ + { ...shared, id: priorLeaseId, heartbeatRunId: priorRunId, status: "retained", metadata: pin }, + { ...shared, id: currentLeaseId, heartbeatRunId: currentRunId, status: "active", metadata: { ...pin, runnerTaskDispatched: true } }, + ]); + const call = vi.fn(async (_id: string, _method: string, params: any) => params.operation.kind === "status" + ? { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true, sessionStateAvailable: false } + : { kind: "accepted", taskId: params.taskId }); + const runtime = environmentRuntimeService(db, { pluginWorkerManager: { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call } as unknown as PluginWorkerManager }); + await runtime.task({ companyId, leaseId: currentLeaseId, operation: { kind: "status" } }); + await runtime.task({ companyId, leaseId: currentLeaseId, operation: { kind: "complete" } }); + const leases = await db.select().from(environmentLeases).where(eq(environmentLeases.providerLeaseId, "warm-task")); + const priorLease = leases.find(lease => lease.id === priorLeaseId)!, currentLease = leases.find(lease => lease.id === currentLeaseId)!; + expect(priorLease).toMatchObject({ status: "released", cleanupStatus: "success", + metadata: { remoteExecutionTermination: { leaseId: priorLeaseId, runId: priorRunId, state: "stopped" } } }); + expect(currentLease).toMatchObject({ status: "active", + metadata: { remoteExecutionTermination: { leaseId: currentLeaseId, runId: currentRunId, state: "stopped" } } }); + expect(hasRemoteTerminationReceipt(priorLease)).toBe(true); + expect(await remoteExecutionHasStopped(db, companyId, priorRunId)).toBe(true); + expect(await remoteExecutionHasStopped(db, companyId, currentRunId)).toBe(true); + expect(await remoteExecutionHasReleased(db, companyId, currentRunId)).toBe(false); + }); + }); diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts index 77abc489223..50a33190335 100644 --- a/server/src/services/environment-run-orchestrator.ts +++ b/server/src/services/environment-run-orchestrator.ts @@ -20,7 +20,7 @@ import { prepareRemoteWorkspace, readRemoteWorkspaceAuthority, remoteWorkspaceRe import { environmentLeases, type Db } from "@paperclipai/db"; import { and, eq } from "drizzle-orm"; -import { remoteExecutionHasStopped } from "./remote-execution-termination.js"; +import { remoteExecutionHasReleased } from "./remote-execution-termination.js"; import { remoteRecoveryLeaseMismatch, type RemoteRecoveryLeaseMismatchField } from "./native-runtime/remote-recovery-lease-mismatch.js"; import { EnvironmentProviderUnavailableError, readTransientLeaseAcquisitionFailure } from "./environment-provider-unavailable-error.js"; import { injectLeaseAcquisitionFault } from "./environment-lease-fault-injection.js"; @@ -347,7 +347,9 @@ export function environmentRunOrchestrator( lease.metadata?.agentId !== input.agentId)) throw new Error("native_plugin_recovery_lease_mismatch"); const errors: unknown[] = []; await environmentRuntime.releaseRunLeases(input.heartbeatRunId, "released", (_id, error) => { errors.push(error); }); - if (errors.length || (prior.length && !await remoteExecutionHasStopped(db, input.companyId, input.heartbeatRunId))) { + // Same-run task resume requires every prior lease released with its receipt; + // a stop recorded before a held release is not enough here. + if (errors.length || (prior.length && !await remoteExecutionHasReleased(db, input.companyId, input.heartbeatRunId))) { throw new Error("native_plugin_recovery_termination_unconfirmed", { cause: errors[0] }); } } diff --git a/server/src/services/environment-task-runtime.ts b/server/src/services/environment-task-runtime.ts index 87020bb274f..f6c40f8fb63 100644 --- a/server/src/services/environment-task-runtime.ts +++ b/server/src/services/environment-task-runtime.ts @@ -1,4 +1,4 @@ -import { resolvePluginTaskResume } from "./plugin-task-resume.js"; +import { pluginTaskResumeRefused, resolvePluginTaskResume } from "./plugin-task-resume.js"; export { resolvePluginTaskResume } from "./plugin-task-resume.js"; import { attachSandboxCreation, environmentService } from "./environments.js"; import { createOwnedSandbox, queueSandboxCreationCleanup, recordSandboxCreationResource } from "./sandbox-creation-operations.js"; @@ -198,11 +198,13 @@ export async function executeEnvironmentTask(db: Db, workers: Pick>'runnerTaskRetiring', 'false') != 'true'`, requiredProjectAuthProof ? sql`${environmentLeases.metadata}->'runnerTaskProjectAuthValidation' = ${JSON.stringify(requiredProjectAuthProof)}::jsonb` : undefined, @@ -420,7 +423,7 @@ export async function executeEnvironmentTask(db: Db, workers: Pick 1) return "predecessor_candidate_ambiguous"; + const old = candidates[0]!; + const identity = old.lease.metadata?.runnerTaskIdentity as Record | undefined; + if (!old.lease.providerLeaseId) return "predecessor_task_missing"; + if (!identity || identity.runnerId !== priorIdentity.runnerInstanceId || identity.leaseId !== priorIdentity.environmentLeaseId || + identity.sessionId !== priorIdentity.normalizedSessionId || identity.runId !== priorIdentity.runId) return "identity_mismatch"; + if (old.lease.metadata?.driver !== "plugin" || next.lease.metadata?.driver !== "plugin") return "provider_mismatch"; + if (!next.lease.providerLeaseId) return "next_task_missing"; + if (next.lease.providerLeaseId === old.lease.providerLeaseId) return "same_provider_task"; + if (old.run.id !== next.run.id && !TERMINAL_RUN_STATUSES.includes(old.run.status)) return "predecessor_run_active"; + if (old.run.agentId !== next.run.agentId || old.lease.issueId !== next.lease.issueId || + old.lease.executionWorkspaceId !== next.lease.executionWorkspaceId || + !old.lease.environmentId || old.lease.environmentId !== next.lease.environmentId) return "scope_mismatch"; + if (!next.lease.provider?.startsWith("plugin:") || old.lease.provider !== next.lease.provider || + old.lease.metadata?.pluginId !== next.lease.metadata?.pluginId || + old.lease.metadata?.driverKey !== next.lease.metadata?.driverKey) return "provider_mismatch"; + if (next.lease.status !== "active" || next.lease.releasedAt || + (next.lease.expiresAt && next.lease.expiresAt.getTime() <= Date.now())) return "next_lease_inactive"; + if (next.run.status !== "running") return "next_run_not_running"; + return null; +} + /** A saved provider task can resume only within the same authorized agent workspace. */ export async function resolvePluginTaskResume(db: Db, input: { companyId: string; leaseId: string; - priorIdentity: { runnerInstanceId: string; environmentLeaseId: string; runId: string; normalizedSessionId: string }; + priorIdentity: PriorIdentity; /** Read-only discovery; launching still requires an attested retained checkpoint. */ inspectCheckpoint?: boolean; }): Promise { @@ -28,25 +82,12 @@ export async function resolvePluginTaskResume(db: Db, input: { !rows.some(successor => successor.lease.id !== input.leaseId && successor.lease.metadata?.runnerTaskResumeFrom === row.lease.providerLeaseId); }); - const old = candidates.length === 1 ? candidates[0] : undefined; - const identity = old?.lease.metadata?.runnerTaskIdentity as Record | undefined; - if (!old || !next || !old.lease.providerLeaseId || !identity || - identity.runnerId !== input.priorIdentity.runnerInstanceId || identity.leaseId !== input.priorIdentity.environmentLeaseId || - identity.sessionId !== input.priorIdentity.normalizedSessionId || identity.runId !== input.priorIdentity.runId || - old.lease.metadata?.driver !== "plugin" || next.lease.metadata?.driver !== "plugin" || - !next.lease.providerLeaseId || next.lease.providerLeaseId === old.lease.providerLeaseId || - (old.run.id !== next.run.id && !["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes(old.run.status)) || - old.run.agentId !== next.run.agentId || old.lease.issueId !== next.lease.issueId || - old.lease.executionWorkspaceId !== next.lease.executionWorkspaceId || - !old.lease.environmentId || old.lease.environmentId !== next.lease.environmentId || - !next.lease.provider?.startsWith("plugin:") || old.lease.provider !== next.lease.provider || - old.lease.metadata?.pluginId !== next.lease.metadata?.pluginId || old.lease.metadata?.driverKey !== next.lease.metadata?.driverKey || - next.lease.status !== "active" || next.lease.releasedAt || next.run.status !== "running" || - (next.lease.expiresAt && next.lease.expiresAt.getTime() <= Date.now()) || - !(old.run.id === next.run.id - ? rows.filter(row => row.lease.id !== next.lease.id).every(row => hasRemoteTerminationReceipt(row.lease)) - : await remoteExecutionHasStopped(db, input.companyId, old.run.id))) { - throw new Error("Plugin Runner retained session is not authorized"); - } - return old.lease.providerLeaseId; + const refusal = resumeRefusal(input.priorIdentity, next, candidates); + if (refusal) throw pluginTaskResumeRefused(refusal); + const old = candidates[0]!; + const stopped = old.run.id === next!.run.id + ? rows.filter(row => row.lease.id !== next!.lease.id).every(row => hasRemoteTerminationReceipt(row.lease)) + : await remoteExecutionHasStopped(db, input.companyId, old.run.id); + if (!stopped) throw pluginTaskResumeRefused("predecessor_not_stopped"); + return old.lease.providerLeaseId!; } diff --git a/server/src/services/remote-execution-termination.ts b/server/src/services/remote-execution-termination.ts index e18a6fdaae9..147982f4746 100644 --- a/server/src/services/remote-execution-termination.ts +++ b/server/src/services/remote-execution-termination.ts @@ -24,19 +24,47 @@ export function remoteTerminationReceipt(lease: LeaseIdentity, value: unknown) { }; } -export function hasRemoteTerminationReceipt(lease: LeaseIdentity & { +type ReceiptLease = LeaseIdentity & { releasedAt: unknown; cleanupStatus: string | null; status: string; metadata: Record | null; -}): boolean { +}; + +/** The lease carries a provider stop receipt bound to its exact company, run, + * lease, provider and physical task. It proves execution there stopped, not + * that the lease's resources were released. */ +export function hasBoundRemoteStopReceipt(lease: Pick): boolean { const receipt = lease.metadata?.remoteExecutionTermination as Record | undefined; - return Boolean(lease.releasedAt && lease.cleanupStatus === "success" && - ["released", "expired", "failed"].includes(lease.status) && receipt && - receipt.schema === "paperclip.remote-termination.v1" && + return Boolean(receipt && receipt.schema === "paperclip.remote-termination.v1" && receipt.companyId === lease.companyId && receipt.runId === lease.heartbeatRunId && receipt.leaseId === lease.id && receipt.provider === lease.provider && remoteTerminationReceipt(lease, receipt)); } +/** The lease was released with successful cleanup and its bound stop receipt. + * Resource cleanup gates use this; it never accepts an unreleased lease. */ +export function hasRemoteTerminationReceipt(lease: ReceiptLease): boolean { + return Boolean(lease.releasedAt && lease.cleanupStatus === "success" && + ["released", "expired", "failed"].includes(lease.status) && hasBoundRemoteStopReceipt(lease)); +} + +/** Release or destroy of this lease has started: plugin release and destroy + * record their cleanup method first, and task settlement, startup cleanup and + * interrupted-creation cleanup mark the task retiring. The provider can then be + * releasing the task's resources while the lease row is still active. */ +function leaseReleaseStarted(lease: Pick): boolean { + return lease.metadata?.pluginLeaseCleanupMethod !== undefined || lease.metadata?.runnerTaskRetiring === true; +} + +/** Execution on this lease has stopped: it was released with its termination + * receipt, or it is still active, its release has not started, and its plugin + * task recorded the provider's stop when it settled. Release can be held long + * after that stop (for example by a pending workspace export); once release + * starts, it owns the receipt until the lease is released. */ +function remoteLeaseExecutionStopped(lease: ReceiptLease): boolean { + return hasRemoteTerminationReceipt(lease) || + lease.status === "active" && !lease.releasedAt && !leaseReleaseStarted(lease) && hasBoundRemoteStopReceipt(lease); +} + /** Workspace recovery saved this run's files, re-stopped its sandbox, and then * queued the stopped sandbox for destroy. The destroy is resource cleanup: the * stop receipt it carried forward still proves no provider can run there. */ @@ -77,16 +105,27 @@ export async function resolveRemoteTerminationLease(db: Db, lease: Lease): Promi } } }; } -export async function stoppedRemoteCleanupScopes(db: Db, companyId: string, runId: string) { +async function runRemoteLeases(db: Db, companyId: string, runId: string) { const leases = await db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId), )); - if (leases.length === 0) return null; - const resolved = await Promise.all(leases.map(lease => resolveRemoteTerminationLease(db, lease))); - if (!resolved.every(hasRemoteTerminationReceipt)) return null; - return [...new Set(resolved.map(lease => remoteLeaseCleanupScope(lease)!))]; + return Promise.all(leases.map(lease => resolveRemoteTerminationLease(db, lease))); } +/** Cleanup scopes of a run whose every lease has stopped execution, or null. */ +export async function stoppedRemoteCleanupScopes(db: Db, companyId: string, runId: string) { + const leases = await runRemoteLeases(db, companyId, runId); + if (leases.length === 0 || !leases.every(remoteLeaseExecutionStopped)) return null; + return [...new Set(leases.map(lease => remoteLeaseCleanupScope(lease)!))]; +} + +/** Every lease of the run has stopped execution. Its resources can still be held. */ export async function remoteExecutionHasStopped(db: Db, companyId: string, runId: string) { return await stoppedRemoteCleanupScopes(db, companyId, runId) !== null; } + +/** Every lease of the run was released with its termination receipt. */ +export async function remoteExecutionHasReleased(db: Db, companyId: string, runId: string) { + const leases = await runRemoteLeases(db, companyId, runId); + return leases.length > 0 && leases.every(hasRemoteTerminationReceipt); +}