From a924d2e2c7e14d2390e037b84aea4bd74ca06e1e Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Thu, 12 Nov 2026 10:43:48 -0800 Subject: [PATCH] Persist portable Runner ownership through controller recovery (#21066) ## Thinking Path > - Paperclip delegates remote Runner execution to environment plugins. > - A plugin can move a session to another task allocation. > - The Core controller can lose an ownership reply or restart before the Runner starts. > - The source and target need a durable record of the same request. > - This change binds restart and attachment to that request and the original unlaunched task lease. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of plugin Runner sessions on replacement compute. **Current behavior** Core cannot replay a session ownership request from saved lease state during startup. **Proposed behavior** Record the operation on both leases and check the matched ownership receipt before attaching. Preserve the original pre-admission lease across controller restart. Managed environment configuration is in [#21065](https://github.com/paperclipai/paperclip/pull/21065). Related warm sandbox cleanup is in [#20065](https://github.com/paperclipai/paperclip/pull/20065). This layer preserves the same-sandbox fallback from [#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace placement follows in [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Add the optional `sessionHandoffVersion` capability and the ownership task operation. - Save matching ownership claims and fence source checkpoint access. Replay a pending request on the original target. - Restore a portable session using the durable claim and the current credentials for the same account. Start a fresh conversation when the provider reports an unavailable checkpoint, or when Core's persisted compatibility decision selects a different session after transfer. The latter requires an unlaunched target; subsequent recovery must match the fresh task's saved identity. Both paths retain the workspace transfer and source fencing. - Reuse the original unlaunched task lease after hard or graceful controller restart. After a move has attached, validate the surviving Runner through the saved claim rather than requiring its logical lease ID to equal the new physical lease. - Keep older retained leases without a saved account identity on the existing resume path. - Apply the existing supported MCP-only compatibility rule to warm recovery and predecessor accounting. Persist the capability discovered at admission and reuse it after restart without repeating discovery. - Keep predecessor-retirement fallback scoped to the same physical sandbox. A refusal after a portable move preserves the session instead of requesting a replacement that the plugin driver cannot enforce. ## Verification - On `1eb527be27`, server typecheck (including Runner build and SDK dependencies) passed. - Independent final verification passed 168 tests across portable source recovery, session resume, controller recovery, and task authority; two existing tests were skipped. The full portable source integration suite passed 64 tests. - The new integration follows confirmed ownership through an incompatible instruction change, persisted fresh admission, target-only prepare/auth/submit without the source checkpoint, and reattachment after controller replacement. It also rejects unpersisted session selection, ambiguous target leases, and incorrect saved identities. - Before this final change, independent verification passed 1,667 tests across executor, session resume, recovery usage, remote warm recovery, runtime selection, and runtime execution; the environment lifecycle suite passed 208 tests. - Existing regressions cover previous-release account metadata, supported and unsupported MCP changes, provider/instruction incompatibility, and admission capability reuse without another discovery call. - On `51d7eec373`, all four S3 multipart throttle tests passed independently. CI exposed an existing timing-sensitive test twice; its 20 ms overlap assumption now uses synchronized retries and an event-loop yield. Production upload code is unchanged. Temporary concurrency caps of one and eight both fail the retained exact-four assertion. - Exact SDK source-pin checks passed for both stack heads. The upper #20743 code patch remains unchanged; its documentation preserves both layers' behavior. ## Risks - The handoff protocol is opt-in. Existing plugin leases without portable ownership metadata keep the previous release's resume path. Live attachment still requires end-to-end qualification. An unverified claim blocks startup. Git workspaces prepared on a portable allocation also need the workspace placement layer. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip --- doc/plugins/PLUGIN_RUNNER_EXECUTION.md | 9 + packages/plugins/sdk/src/environment-tasks.ts | 36 +- .../environment-task-release-resume.test.ts | 22 ++ packages/shared/src/types/plugin.ts | 1 + packages/shared/src/validators/plugin.ts | 1 + .../src/__tests__/environment-runtime.test.ts | 79 +++++ ...ronment-task-authority.integration.test.ts | 22 +- .../heartbeat-process-recovery.test.ts | 47 +++ .../plugin-task-credential-writeback.test.ts | 14 + .../storage-s3-multipart-throttle.test.ts | 17 +- .../services/environment-execution-target.ts | 2 + .../services/environment-run-orchestrator.ts | 22 +- server/src/services/environment-runtime.ts | 41 ++- .../src/services/environment-task-runtime.ts | 107 +++++- server/src/services/environments.ts | 3 +- server/src/services/heartbeat.ts | 19 +- .../heartbeat/instruction-working-copy.ts | 2 +- .../src/services/heartbeat/recovery.test.ts | 35 ++ server/src/services/heartbeat/recovery.ts | 23 +- .../services/heartbeat/runtime-execution.ts | 4 +- .../heartbeat/runtime-selection.test.ts | 63 +++- .../services/heartbeat/runtime-selection.ts | 53 +-- .../services/heartbeat/session-preparation.ts | 23 +- .../heartbeat/workspace-preparation.ts | 18 +- .../src/services/legacy-controller-lease.ts | 49 ++- ...ve-controller-recovery.integration.test.ts | 12 +- .../native-runtime/native-restart-recovery.ts | 24 +- .../native-runtime/native-runner-ownership.ts | 2 + .../native-session-executor.test.ts | 224 +++++++++--- .../native-runtime/native-session-executor.ts | 198 ++++++++--- .../native-runtime/native-session-resume.ts | 18 +- .../native-source-recovery-ownership.ts | 25 +- .../native-runtime/plugin-runner-task.test.ts | 12 + .../native-runtime/plugin-runner-task.ts | 8 +- ...n-task-source-recovery.integration.test.ts | 219 +++++++++++- .../remote-warm-session-recovery.ts | 22 +- .../plugin-task-credential-writeback.ts | 6 + server/src/services/plugin-task-ownership.ts | 319 ++++++++++++++++++ server/src/services/plugin-task-resume.ts | 1 + .../runner-e2e/source-sdk-public-inputs.json | 6 +- 40 files changed, 1585 insertions(+), 223 deletions(-) create mode 100644 server/src/services/plugin-task-ownership.ts diff --git a/doc/plugins/PLUGIN_RUNNER_EXECUTION.md b/doc/plugins/PLUGIN_RUNNER_EXECUTION.md index 2fb9c87b637..ee582ef6f6d 100644 --- a/doc/plugins/PLUGIN_RUNNER_EXECUTION.md +++ b/doc/plugins/PLUGIN_RUNNER_EXECUTION.md @@ -172,6 +172,15 @@ It does not qualify a deployed plugin, live account, or production WSS proxy. ## Workspace completion and maintenance +A portable restore uses the durable ownership claim and the same account +identity after a credential refresh or controller replacement. It starts a +new Runner on the target. Run attachment applies the current tools and +runtime configuration to the restored provider session. After a confirmed +workspace transfer, Core can use the same target to start a new conversation +when its session compatibility check requires one. The new conversation +starts without the prior provider checkpoint, and the ownership claim +continues to fence the source. + A storage provider can declare `workspaceProvider.workspaceVerificationVersion: 1`. Before creating a fresh workspace for an admitted Retry, Core may send the original task-owned workspace's `prepare` request with `verifyExisting: true`. diff --git a/packages/plugins/sdk/src/environment-tasks.ts b/packages/plugins/sdk/src/environment-tasks.ts index 98e5f69a00b..de583d80918 100644 --- a/packages/plugins/sdk/src/environment-tasks.ts +++ b/packages/plugins/sdk/src/environment-tasks.ts @@ -25,12 +25,13 @@ const githubBroker = z.object({ url: z.string().url().max(2000).refine(value => return ["http:", "https:"].includes(url.protocol) && !url.username && !url.password && !url.search && !url.hash; } catch { return false; } }, "Expected a credential-free HTTP broker URL"), token: credentialValue }).strict(); +const claudeCredentialEnvironment = z.object({ + CLAUDE_CODE_OAUTH_TOKEN: credentialValue.optional(), ANTHROPIC_API_KEY: credentialValue.optional(), +}).strict().refine(env => Object.values(env).filter(Boolean).length === 1, "Select one Claude credential"); /** Only provider authentication is transferred. Paths and executable settings remain provider-owned. */ export const environmentTaskCredentialSchema = z.discriminatedUnion("type", [ z.object({ type: z.literal("codex-auth-json"), authJson: credentialValue }).strict(), - z.object({ type: z.literal("claude-env"), environment: z.object({ - CLAUDE_CODE_OAUTH_TOKEN: credentialValue.optional(), ANTHROPIC_API_KEY: credentialValue.optional(), - }).strict().refine(env => Object.values(env).filter(Boolean).length === 1, "Select one Claude credential") }).strict(), + z.object({ type: z.literal("claude-env"), environment: claudeCredentialEnvironment }).strict(), ]); export type PluginEnvironmentTaskCredential = z.infer; @@ -114,7 +115,18 @@ export const environmentTaskOperationSchema = z.discriminatedUnion("kind", [ z.object({ kind: z.literal("status") }).strict(), z.object({ kind: z.literal("connection") }).strict(), z.object({ kind: z.literal("release"), identity: durableIdentity }).strict(), - z.object({ kind: z.literal("resume"), identity: durableIdentity, bootstrapTicket: credentialValue }).strict(), + z.object({ kind: z.literal("resume"), identity: durableIdentity, bootstrapTicket: credentialValue, + handoffCredential: z.object({ accountIdentity: z.string().min(1).max(1024), environment: claudeCredentialEnvironment }).strict().optional(), + }).strict(), + z.object({ + kind: z.literal("ensure_ownership"), + operationId: z.string().uuid(), + identity: durableIdentity, + sourceTaskId: providerId.max(1024), + sourceGeneration: providerId.max(1024), + targetGeneration: providerId.max(1024), + policy: z.enum(["ordinary", "reclaim_if_needed"]), + }).strict(), z.object({ kind: z.literal("complete") }).strict(), z.object({ kind: z.literal("stop") }).strict(), ]); @@ -125,6 +137,14 @@ export const environmentTaskResultSchema = z.discriminatedUnion("kind", [ reason: z.enum(["session_state_unavailable", "account_binding_mismatch"]) }).strict(), z.object({ kind: z.literal("released"), taskId: providerId }).strict(), z.object({ kind: z.literal("resume"), taskId: providerId, outcome: z.enum(["resumed", "unavailable"]) }).strict(), + z.object({ + kind: z.literal("ownership"), + taskId: providerId, + operationId: z.string().uuid(), + targetGeneration: providerId.max(1024), + status: z.enum(["pending", "ready"]), + checkpointUnavailable: z.literal(true).optional(), + }).strict(), z.object({ kind: z.literal("copilot_metadata"), taskId: providerId, metadata: z.record(z.string(), z.unknown()), executionStopped: z.literal(true) }).strict(), z.object({ kind: z.literal("project_auth"), taskId: providerId, configurationId: providerId.max(1024), @@ -142,6 +162,7 @@ export const environmentTaskResultSchema = z.discriminatedUnion("kind", [ runner: z.object({ version: z.string().min(1), digest: z.string().regex(/^sha256:[a-f0-9]{64}$/) }).strict(), warmTaskVersion: z.literal(1).optional(), releaseResumeVersion: z.literal(1).optional(), + sessionHandoffVersion: z.literal(1).optional(), launchPreparationVersion: z.literal(1).optional(), executionContextVersion: z.literal(1).optional(), /** Each version attests stopped-source recovery for its provider independently. */ @@ -211,6 +232,13 @@ export interface PluginEnvironmentTaskParams extends Omit Promise) { @@ -89,6 +93,24 @@ describe("release and resume task contract", () => { .toMatchObject({ result: { kind: "resume", taskId: "task", outcome: "unavailable" } }); }); + it("binds a declarative ownership response to the exact worker request", async () => { + for (const status of ["pending", "ready"] as const) { + const result = { kind: "ownership", status, taskId: "task", operationId: ownership.operationId, + targetGeneration: ownership.targetGeneration } as const; + expect(await invoke(ownership, async params => { + expect(params.operation).toEqual(ownership); + return result; + })).toMatchObject({ result }); + for (const changes of [ + { taskId: "another-task" }, + { operationId: "1f083ad0-a357-4f14-ad46-e9df981845b5" }, + { targetGeneration: "stale-generation" }, + ]) { + expect(await invoke(ownership, async () => ({ ...result, ...changes }))).toHaveProperty("error"); + } + } + }); + it.each(["source mismatch", "claim conflict", "authentication failed", "transport timeout"])("keeps %s as an RPC error", async message => { expect(await invoke(resume, async () => { throw new Error(message); })).toHaveProperty("error"); }); diff --git a/packages/shared/src/types/plugin.ts b/packages/shared/src/types/plugin.ts index 016059b873c..22e1e35307f 100644 --- a/packages/shared/src/types/plugin.ts +++ b/packages/shared/src/types/plugin.ts @@ -201,6 +201,7 @@ export interface PluginEnvironmentDriverDeclaration { creationOperationVersion?: 1; /** Codex task retention and authorized run attachment, version 1. */ warmTaskVersion?: 1; + sessionHandoffVersion?: 1; /** Stable driver key, unique within the plugin. Namespaced by plugin ID at runtime. */ driverKey: string; /** diff --git a/packages/shared/src/validators/plugin.ts b/packages/shared/src/validators/plugin.ts index 2a90d3d00b6..782e0515677 100644 --- a/packages/shared/src/validators/plugin.ts +++ b/packages/shared/src/validators/plugin.ts @@ -183,6 +183,7 @@ export const pluginEnvironmentDriverDeclarationSchema = z.object({ freshWorkspaceVersion: z.literal(1).optional(), creationOperationVersion: z.literal(1).optional(), warmTaskVersion: z.literal(1).optional(), + sessionHandoffVersion: z.literal(1).optional(), driverKey: z.string().min(1).regex( /^[a-z0-9][a-z0-9._-]*$/, "Environment driver key must start with a lowercase alphanumeric and contain only lowercase letters, digits, dots, hyphens, or underscores", diff --git a/server/src/__tests__/environment-runtime.test.ts b/server/src/__tests__/environment-runtime.test.ts index b016d5e98ee..93d4e6c2e5c 100644 --- a/server/src/__tests__/environment-runtime.test.ts +++ b/server/src/__tests__/environment-runtime.test.ts @@ -33,6 +33,7 @@ import { executionWorkspaces, heartbeatRuns, issues, + nativeRunFinalizations, plugins, projects, } from "@paperclipai/db"; @@ -68,6 +69,9 @@ import { traceparentFromContextToken } from "../instrumentation.ts"; import { ROOT_CONTEXT, trace } from "@opentelemetry/api"; import { buildNativeHarnessBackupManifest } from "../services/native-runtime/native-session-executor.ts"; import { createNativeHarnessBackupStamp } from "../services/native-runtime/native-harness-backup-stamp.ts"; +import { environmentRunOrchestrator } from "../services/environment-run-orchestrator.js"; +import { claimNativeRestartRecoveries } from "../services/native-runtime/native-restart-recovery.js"; +import { NATIVE_PREPARATION_FENCING_VERSION } from "../services/native-runtime/native-active-runs.js"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; @@ -1300,6 +1304,81 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { ); }); + it.each(["handoff", "fresh", "legacy resume"] as const)("preserves the portable task placement on a run retry (%s)", async mode => { + const resumeSession = mode !== "fresh"; + const pluginId = randomUUID(); + const pluginKey = "paperclip.test-portable-tasks"; + const seeded = await seedEnvironment({ driver: "plugin", config: { pluginKey, driverKey: "tasks", runnerLifecycleMode: "warm" } }); + await insertSandboxProviderPlugin(pluginId, { pluginKey, environmentDrivers: [{ + driverKey: "tasks", kind: "sandbox_provider", supportsTasks: true, warmTaskVersion: 1, sessionHandoffVersion: 1, + }] }); + const issueId = randomUUID(), sessionId = randomUUID(), nextRunId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId: seeded.companyId, title: "Portable session", status: "in_progress" }); + await db.insert(heartbeatRuns).values({ id: nextRunId, companyId: seeded.companyId, agentId: seeded.agentId, status: "running" }); + let acquisitions = 0; + const call = vi.fn(async (_plugin: string, method: string) => { + if (method === "environmentAcquireLease") return { providerLeaseId: `sandbox-${++acquisitions}`, + metadata: { ownershipGeneration: `generation-${acquisitions}` } }; + if (method === "environmentResumeLease") return { providerLeaseId: "sandbox-1", metadata: {} }; + if (method === "environmentReleaseLease") return { providerLeaseId: "sandbox-1", state: "stopped" }; + throw new Error(`Unexpected plugin method: ${method}`); + }); + const runtime = environmentRuntimeService(db, { pluginWorkerManager: { + isRunning: () => true, call, getWorker: () => ({ supportedMethods: ["environmentResumeLease"] }), + } as unknown as PluginWorkerManager }); + const input = { companyId: seeded.companyId, environment: seeded.environment, issueId, agentId: seeded.agentId, + persistedExecutionWorkspace: null, adapterType: "paperclip_runner" }; + const orchestrator = environmentRunOrchestrator(db, { environmentRuntime: runtime }); + const acquisitionInput = { ...input, selectedEnvironmentId: seeded.environment.id, localEnvironmentId: randomUUID(), + admittedLifecycleMode: "warm" as const, executionWorkspaceSettings: null }; + const first = await orchestrator.acquireForRun({ ...acquisitionInput, heartbeatRunId: seeded.runId }); + await db.update(environmentLeases).set({ status: "retained", metadata: { + ...first.lease.metadata, runnerTaskIdentity: { sessionId }, + ...(mode === "legacy resume" ? {} : { runnerTaskAccountIdentity: "test-account" }), + } }).where(eq(environmentLeases.id, first.lease.id)); + const nextInput = { ...acquisitionInput, heartbeatRunId: nextRunId, runnerTaskResumeSessionId: resumeSession ? sessionId : null }; + const next = await orchestrator.acquireForRun(nextInput); + expect(next.lease.providerLeaseId).toBe(mode === "legacy resume" ? "sandbox-1" : "sandbox-2"); + expect(next.lease.metadata?.portableTaskPreviousLeaseId).toBe(mode === "handoff" ? first.lease.id : undefined); + await expect(orchestrator.acquireForRun(nextInput)).resolves.toMatchObject({ lease: { id: next.lease.id, providerLeaseId: next.lease.providerLeaseId } }); + if (mode === "handoff") { + await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, seeded.agentId)); + await db.update(issues).set({ assigneeAgentId: seeded.agentId, executionRunId: nextRunId }).where(eq(issues.id, issueId)); + await db.update(heartbeatRuns).set({ scopeKind: "issue", issueId, nativeIssueId: issueId, runtimeMode: "native", + nativeSessionId: sessionId, runnerInstanceId: randomUUID(), runnerProfileJson: { + nativePreparationFencingVersion: NATIVE_PREPARATION_FENCING_VERSION, + sessionCheckpoint: { identity: { companyId: seeded.companyId, agentId: seeded.agentId, issueId, runId: nextRunId, sessionId }, + providerSessionId: "inherited-provider-session" }, + } }).where(eq(heartbeatRuns.id, nextRunId)); + await db.insert(nativeRunFinalizations).values({ runId: nextRunId, companyId: seeded.companyId, issueId, phase: "observed" }); + try { + const [claim] = await claimNativeRestartRecoveries({ db, restartKind: "hard", runIds: [nextRunId] }); + expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: nextRunId, portablePreparationLeaseId: next.lease.id }); + if (claim.kind !== "bootstrap_incomplete" || !claim.portablePreparationLeaseId) throw new Error("Expected portable bootstrap recovery"); + const preparation = { kind: "plugin_task" as const, leaseId: claim.portablePreparationLeaseId, preparation: true as const }; + await expect(orchestrator.acquireForRun({ ...nextInput, reattachRemoteLease: preparation })) + .resolves.toMatchObject({ lease: { id: next.lease.id, providerLeaseId: "sandbox-2", heartbeatRunId: nextRunId } }); + await expect(orchestrator.acquireForRun({ ...nextInput, reattachRemoteLease: preparation, issueId: null })) + .rejects.toThrow("field=lease_scope"); + await expect(orchestrator.acquireForRun({ ...nextInput, reattachRemoteLease: preparation, + persistedExecutionWorkspace: { id: randomUUID(), mode: "isolated_workspace" } })).rejects.toThrow("field=lease_scope"); + const otherEnvironmentId = randomUUID(); + await db.insert(environments).values({ id: otherEnvironmentId, name: otherEnvironmentId, driver: "plugin", + status: "active", config: seeded.environment.config }); + await expect(orchestrator.acquireForRun({ ...nextInput, reattachRemoteLease: preparation, + selectedEnvironmentId: otherEnvironmentId })).rejects.toThrow("field=lease_scope"); + } finally { + await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, nextRunId)); + } + } + await expect(orchestrator.acquireForRun({ ...nextInput, issueId: null })).rejects.toThrow("Active plugin task placement changed"); + expect(call.mock.calls.map(([, method]) => method)).toEqual({ + handoff: ["environmentAcquireLease", "environmentAcquireLease"], + fresh: ["environmentAcquireLease", "environmentReleaseLease", "environmentAcquireLease"], + "legacy resume": ["environmentAcquireLease", "environmentResumeLease"], + }[mode]); + }); + it.each(["same task", "different task", "different agent", "no task"])( "scopes projectless reusable leases to their task and agent: %s", async (scenario) => { const seeded = await seedReusablePluginSandboxLease(); diff --git a/server/src/__tests__/environment-task-authority.integration.test.ts b/server/src/__tests__/environment-task-authority.integration.test.ts index 0f1ad4c9ff7..ffb37044169 100644 --- a/server/src/__tests__/environment-task-authority.integration.test.ts +++ b/server/src/__tests__/environment-task-authority.integration.test.ts @@ -241,19 +241,33 @@ databaseSuite("environment task lease authority", () => { await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId)); let sessionStateAvailable = false; const operations: string[] = []; - const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }), - call: async (_id: string, _method: string, params: any) => ["read_state", "archive_state"].includes(params.operation.kind) + const call = vi.fn(async (_id: string, _method: string, params: any) => ["read_state", "archive_state"].includes(params.operation.kind) ? { kind: "state", taskId: params.taskId, state: { schema: "paperclip.runner.durable.state.v1", lifecycle: "suspended", ...(params.operation.identity ?? params.operation.priorIdentity) } } : params.operation.kind === "status" ? (operations.push("status"), { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true, sessionStateAvailable }) - : { kind: "accepted", taskId: params.taskId }, - } as unknown as PluginWorkerManager; + : { kind: "accepted", taskId: params.taskId }); + const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call } as unknown as PluginWorkerManager; const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers }); const runner = { runnerId: "runner", leaseId: oldLeaseId, sessionId: "session", runId: nextRunId, turnId: "turn", itemId: "item", harness: "codex", protocolMin: 1, protocolMax: 3 }; const checkpointIdentity = { ...priorIdentity, turnId: "prior-turn", itemId: "prior-item" }; const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId)); + const portableTargetLeaseId = randomUUID(); + await db.update(environmentLeases).set({ metadata: { ...source.metadata, portableTaskSuccessor: { + sourceLeaseId: oldLeaseId, targetLeaseId: portableTargetLeaseId, + operation: { kind: "ensure_ownership", operationId: portableTargetLeaseId, sourceTaskId: "old-task", + sourceGeneration: "source-generation", targetGeneration: "target-generation", policy: "ordinary", identity: checkpointIdentity }, + } } }).where(eq(environmentLeases.id, oldLeaseId)); + await expect(authorize()).rejects.toThrow("not authorized"); + await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).rejects.toThrow("not authorized"); + await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: { + kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task", + } })).rejects.toThrow("not authorized"); + await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: { + kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task", + } })).rejects.toThrow("not authorized"); + expect(call).not.toHaveBeenCalled(); await db.update(environmentLeases).set({ metadata: { ...source.metadata, runnerTaskSessionRetained: false } }).where(eq(environmentLeases.id, oldLeaseId)); await expect(authorize()).rejects.toThrow("not authorized"); await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).resolves.toBe("old-task"); diff --git a/server/src/__tests__/heartbeat-process-recovery.test.ts b/server/src/__tests__/heartbeat-process-recovery.test.ts index 3dc360b993b..c90676328e5 100644 --- a/server/src/__tests__/heartbeat-process-recovery.test.ts +++ b/server/src/__tests__/heartbeat-process-recovery.test.ts @@ -7,6 +7,7 @@ import { createSystemRecovery } from "../services/system-recovery-state.js"; import * as conversationContinuation from "../services/conversation-continuation.js"; import { classifyEnvironmentCapabilities, environmentRuntimeService } from "../services/environment-runtime.js"; import * as workspaceRuntime from "../services/workspace-runtime.js"; +import * as workspacePreparation from "../services/heartbeat/workspace-preparation.js"; import { nativeWorkspaceSyncInternals } from "../services/native-runtime/native-workspace-sync.js"; import { ensureNativeCompletionContract } from "../services/native-runtime/completion-contracts.js"; import { readNativePlanWait, hasCommittedNativePlanWait } from "../services/native-runtime/native-plan-wait.js"; @@ -5278,6 +5279,52 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { ]); }); + it("hands off the same portable task lease during in-flight workspace preparation on graceful shutdown", async () => { + const f = await seedRunFixture({ adapterType: "paperclip_runner", agentStatus: "idle", runStatus: "queued", runtimeMode: "legacy" }); + await db.update(heartbeatRuns).set({ scopeKind: "issue", issueId: f.issueId }).where(eq(heartbeatRuns.id, f.runId)); + await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId)); + const { leaseId } = await seedEnvironmentLeaseFixture({ ...f, driver: "plugin", provider: "plugin:fixture:tasks" }); + await db.update(environmentLeases).set({ providerLeaseId: "same-portable-task", metadata: { + driver: "plugin", agentId: f.agentId, runnerTaskLease: true, portableTaskPreviousLeaseId: randomUUID(), + } }).where(eq(environmentLeases.id, leaseId)); + let finishPreparation!: () => void; + const providerReply = new Promise(resolve => { finishPreparation = resolve; }); + const reachedPreparation = vi.fn(); + const preparation = vi.spyOn(workspacePreparation, "prepareHeartbeatWorkspace").mockImplementationOnce(async (_db, input) => { + await input.controllerLease.assertOwned(); + reachedPreparation(); + await providerReply; + await input.controllerLease.assertOwned(); + throw new Error("Old controller unexpectedly kept preparation authority"); + }); + const runtime = environmentRuntimeService(db); + const release = vi.spyOn(runtime, "releaseRunLeases"); + const heartbeat = heartbeatService(db, { environmentRuntime: runtime }); + try { + await heartbeat.resumeQueuedRunsForAgent(f.companyId, f.agentId); + await vi.waitFor(() => expect(reachedPreparation).toHaveBeenCalledOnce(), { timeout: 10_000 }); + expect(await heartbeat.drainRunningRunsForShutdown("SIGTERM", new Date(), [f.runId])).toMatchObject({ + restartSuspendedRunIds: [f.runId], interruptedRunIds: [], retryRunIds: [], + }); + finishPreparation(); + await heartbeat.drainActiveRunExecutions(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + expect(run).toMatchObject({ status: "running", runtimeMode: "legacy", executionStage: "preparing" }); + expect(run.controllerLeaseExpiresAt!.getTime()).toBeLessThanOrEqual(Date.now()); + expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)))[0]) + .toMatchObject({ heartbeatRunId: f.runId, status: "active", providerLeaseId: "same-portable-task" }); + expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(f.runId); + expect(release).not.toHaveBeenCalled(); + expect(mockAdapterExecute).not.toHaveBeenCalled(); + expect(mockExecutePaperclipNativeSession).not.toHaveBeenCalled(); + } finally { + finishPreparation(); + await heartbeat.drainActiveRunExecutions(); + preparation.mockRestore(); + release.mockRestore(); + } + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + it.each([ { name: "still preparing", resumes: true }, { name: "admitted to dispatch", stage: "dispatching", resumes: false }, diff --git a/server/src/__tests__/plugin-task-credential-writeback.test.ts b/server/src/__tests__/plugin-task-credential-writeback.test.ts index c1ac37bc105..936ef101f31 100644 --- a/server/src/__tests__/plugin-task-credential-writeback.test.ts +++ b/server/src/__tests__/plugin-task-credential-writeback.test.ts @@ -83,6 +83,20 @@ suite("durable plugin task credential handoff", () => { failExport(value: boolean) { failExport = value; }, credential: () => service.credential(selection!), settle: (executionStopObserved = false) => settleDispatchedEnvironmentTask(db, workers, { companyId, id: leaseId }, { executionStopObserved }) }; } + it("completes a moved task's saved credential handoff after confirming stop", async () => { + const f = await fixture(); + await db.update(environmentLeases).set({ metadata: + sql`${environmentLeases.metadata} || '{"portableTaskSuccessor":{"targetLeaseId":"replacement"}}'::jsonb`, + }).where(eq(environmentLeases.id, f.leaseId)); + await expect(f.synchronize()).rejects.toThrow("session ownership unavailable"); + expect(f.calls).toEqual([]); + await expect(f.settle()).resolves.toBe("completed"); + expect(f.calls).toEqual(["stop", "status", "credentials", "complete"]); + expect(await f.credential()).toBe(auth("fixture-refreshed", 2021)); + await expect(f.synchronize()).resolves.toBeUndefined(); + expect(f.calls).toHaveLength(4); + }); + it.each(["codex", "grok"] as const)("writes managed %s refreshes without any plugin or invocation home", async provider => { const f = await fixture(true, provider); const read = vi.fn(async () => f.accountCredential("fixture-refreshed", 2021)); diff --git a/server/src/__tests__/storage-s3-multipart-throttle.test.ts b/server/src/__tests__/storage-s3-multipart-throttle.test.ts index 0572359cd79..fda0a85f8d2 100644 --- a/server/src/__tests__/storage-s3-multipart-throttle.test.ts +++ b/server/src/__tests__/storage-s3-multipart-throttle.test.ts @@ -1,5 +1,6 @@ import { AbortMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, S3Client } from "@aws-sdk/client-s3"; import { Readable } from "node:stream"; +import { setImmediate as nextEventLoopTurn } from "node:timers/promises"; import { createHash } from "node:crypto"; import { createServer } from "node:http"; import type { AddressInfo } from "node:net"; @@ -10,6 +11,7 @@ import { describe, expect, it, vi } from "vitest"; import { FILE_PART_CONCURRENCY, THROTTLED_PART_ATTEMPTS, putS3MultipartFile } from "../storage/s3-multipart.js"; import { createS3StorageProvider } from "../storage/s3-provider.js"; import { createStorageService } from "../storage/service.js"; +import * as fileCrypto from "../storage/file-crypto-pool.js"; const partBytes = 8 * 1024 * 1024; @@ -89,17 +91,24 @@ describe("file-backed S3 multipart uploads under a request-rate limit", () => { const client = new S3Client({ region: "us-east-1" }); const attempts = new Map(); let laterActive = 0, laterPeak = 0; + let firstRoundRetries = 0; + const firstRoundRetried = Promise.withResolvers(); + // Keep disk I/O out of the concurrency measurement. The real-client test checks MD5. + using digest = vi.spyOn(fileCrypto, "digestFileOffThread").mockResolvedValue({ hex: "0".repeat(32), bytes: partBytes }); const random = vi.spyOn(Math, "random").mockReturnValue(0); const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => { if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" }; if (command instanceof UploadPartCommand) { const number = command.input.PartNumber!; attempts.set(number, (attempts.get(number) ?? 0) + 1); - await drain(command.input.Body); - if (number <= FILE_PART_CONCURRENCY && attempts.get(number) === 1) throw slowDown(); - if (number > FILE_PART_CONCURRENCY) { + if (number <= FILE_PART_CONCURRENCY) { + if (attempts.get(number) === 1) throw slowDown(); + // Finish the already-dispatched retries together so every eligible slot can take new work. + if (++firstRoundRetries === FILE_PART_CONCURRENCY) firstRoundRetried.resolve(); + await firstRoundRetried.promise; + } else { laterActive++; laterPeak = Math.max(laterPeak, laterActive); - await new Promise(resolve => setTimeout(resolve, 20)); + await nextEventLoopTurn(); laterActive--; } return { ETag: `part-${number}` }; diff --git a/server/src/services/environment-execution-target.ts b/server/src/services/environment-execution-target.ts index 6d78e7f00a9..3d41d3ebb2f 100644 --- a/server/src/services/environment-execution-target.ts +++ b/server/src/services/environment-execution-target.ts @@ -237,6 +237,7 @@ export async function resolveEnvironmentExecutionTarget(input: { leaseMetadata: Record | null; lease?: EnvironmentLease | null; environmentRuntime?: EnvironmentRuntimeService | null; + signal?: AbortSignal; // The startup tracer for the provider-exec span. Defaults to the endpoint- // gated server tracer, which is a no-op when tracing is off. Tests inject a // recording tracer. @@ -278,6 +279,7 @@ export async function resolveEnvironmentExecutionTarget(input: { const result = await runtime.task({ companyId: input.companyId, leaseId: lease.id, operation: environmentTaskOperationSchema.parse(operation), ...(context?.executionContext ? { executionContext: context.executionContext } : {}), + ...(input.signal ? { ownershipSignal: input.signal } : {}), ...(context?.expectedTaskId !== undefined ? { expectedTaskId: context.expectedTaskId } : {}) }); if (result.kind === "description" && result.taskId !== acquisition.providerLeaseId) { acquisition.outcome = "replacement"; diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts index aa1c0273f62..da8d141adc1 100644 --- a/server/src/services/environment-run-orchestrator.ts +++ b/server/src/services/environment-run-orchestrator.ts @@ -41,7 +41,7 @@ import { runnerLifecycleDefaultNeedsLeaseProbe, resolveWarmRunnerIdleTimeoutMsForRun, } from "./runner-environment-lifecycle.js"; -import { assertEnvironmentCompanyBinding, environmentService } from "./environments.js"; +import { assertEnvironmentCompanyBinding, environmentService, toEnvironmentLease } from "./environments.js"; import { environmentRuntimeService, buildEnvironmentLeaseContext, @@ -50,6 +50,7 @@ import { type ProviderResourceDisposition, } from "./environment-runtime.js"; import { ENVIRONMENT_DRIVER_TRAITS } from "./environment-driver-traits.js"; +import { unlaunchedPortableTaskLease } from "./plugin-task-ownership.js"; import { resolveEnvironmentExecutionTarget, resolveEnvironmentExecutionTransport, @@ -308,9 +309,10 @@ export function environmentRunOrchestrator( persistedExecutionWorkspace: Pick | null; executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null; runnerTaskProjectIds?: readonly string[]; + runnerTaskResumeSessionId?: string | null; /** Existing live runner recovery must use its original active lease, including ephemeral leases. */ reattachRemoteLease?: { kind?: "sandbox"; leaseId: string; providerLeaseId: string; remoteCwd: string } - | { kind: "plugin_task"; leaseId: string }; + | { kind: "plugin_task"; leaseId: string; preparation?: true }; /** A new controller must stop the previous plugin task before allocating a replacement. */ recoverPluginTask?: boolean; }): Promise { @@ -365,16 +367,22 @@ export function environmentRunOrchestrator( persistedExecutionWorkspace: input.persistedExecutionWorkspace, executionWorkspaceSettings: input.executionWorkspaceSettings, runnerTaskProjectIds: input.runnerTaskProjectIds, + runnerTaskResumeSessionId: input.runnerTaskResumeSessionId, adapterType: input.adapterType ?? null, warmRunnerIdleTimeoutMs, runnerReconnectGraceMs, }; let leaseRecord: EnvironmentRuntimeLeaseRecord; - if (input.reattachRemoteLease) { + const expected = input.reattachRemoteLease; + if (expected) { // Reattachment keeps the original lease, including ephemeral leases. // Never reacquire: acquisition can replace a non-reusable sandbox. - const expected = input.reattachRemoteLease; - const lease = await environmentsSvc.getLeaseById(expected.leaseId); + const replayPreparation = expected.kind === "plugin_task" && expected.preparation; + const lease = replayPreparation + ? await db.select().from(environmentLeases).where(and(eq(environmentLeases.id, expected.leaseId), + eq(environmentLeases.companyId, input.companyId), unlaunchedPortableTaskLease())) + .then(([row]) => row ? toEnvironmentLease(row) : null) + : await environmentsSvc.getLeaseById(expected.leaseId); const mismatch: RemoteRecoveryLeaseMismatchField | null = !lease ? "lease" : lease.companyId !== input.companyId || lease.environmentId !== environment.id || lease.heartbeatRunId !== input.heartbeatRunId || lease.issueId !== input.issueId || @@ -385,6 +393,8 @@ export function environmentRunOrchestrator( : expected.kind === "plugin_task" ? environment.driver !== "plugin" ? "environment_driver" : input.adapterType !== "paperclip_runner" ? "adapter_type" + : replayPreparation ? environment.config.pluginKey !== lease.metadata?.pluginKey || + environment.config.driverKey !== lease.metadata?.driverKey ? "plugin_task" : null : lease.metadata?.runnerTaskLease !== true || lease.metadata?.runnerTaskDispatched !== true || lease.metadata?.runnerTaskExecutionStopped === true ? "plugin_task" : null : environment.driver !== "sandbox" ? "environment_driver" @@ -483,6 +493,7 @@ export function environmentRunOrchestrator( effectiveExecutionWorkspaceMode: string | null; persistedExecutionWorkspace: ExecutionWorkspace | null; runnerTaskProjectIds?: readonly string[]; + signal?: AbortSignal; assertOwned?: () => Promise; /** * The host duplex observability recorder for this run. The orchestrator threads @@ -719,6 +730,7 @@ export function environmentRunOrchestrator( leaseMetadata: (lease.metadata as Record | null) ?? null, lease, environmentRuntime, + signal: input.signal, duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null, }); // The native plugin task bridge owns guest paths. Never fall back to host execution. diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts index d3366b8bb0d..b637afc081f 100644 --- a/server/src/services/environment-runtime.ts +++ b/server/src/services/environment-runtime.ts @@ -20,7 +20,6 @@ import { createOwnedSandbox, recoverSandboxCreations, recoverablePluginCreation import { dueRunLeaseReleaseCondition, pendingRunLeaseReleaseCondition } from "./environment-lease-release-state.js"; import { flushDeferredEnvironmentCleanup, registerDeferredEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; import { executeEnvironmentTask, pluginEnvironmentSupportsTasks, settleDispatchedEnvironmentTask } from "./environment-task-runtime.js"; -import type { PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk"; import { allowLegacyShutdownWorkspaceCleanup, legacyShutdownWorkspaceResourceProtected } from "./legacy-shutdown-workspace-settlement.js"; import { nativeWorkspaceExportPendingForLease, nativeWorkspaceExportPendingForResource } from "./native-runtime/native-workspace-finalization-state.js"; import { beginIdleTrackedWork } from "./task-admission.js"; @@ -484,6 +483,7 @@ export interface EnvironmentDriverAcquireInput { executionWorkspaceMode: ExecutionWorkspace["mode"] | null; executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null; runnerTaskProjectIds?: readonly string[]; + runnerTaskResumeSessionId?: string | null; /** * The harness/adapter type for this run (the agent's adapter). Drivers that * materialize a per-run sandbox use it to select the runtime image so a single @@ -3516,6 +3516,23 @@ function createPluginEnvironmentDriver( workerManager.getWorker(plugin.id)?.supportedMethods.includes("environmentResumeLease"); const lifecyclePolicy = warm ? { mode: "warm", idleTimeoutMs: parsed.config.runnerIdleTimeoutMs ?? 300_000 } : { mode: "per_turn", idleTimeoutMs: null }; const fingerprint = createHash("sha256").update(JSON.stringify([parsed.config, input.agentId, input.runnerTaskProjectIds ?? [], ...(input.purpose ? [input.purpose, input.loginCommandKey] : [])])).digest("hex"); + if (driver.sessionHandoffVersion === 1 && input.adapterType === "paperclip_runner" && input.heartbeatRunId && input.purpose !== "login") { + // The run lease already records the placement to replay after a lost preparation reply. + const active = await db.select().from(environmentLeases).where(and( + eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.environmentId, input.environment.id), + eq(environmentLeases.heartbeatRunId, input.heartbeatRunId), eq(environmentLeases.status, "active"), + )).limit(2); + if (active.length > 0) { + const lease = active[0]!; + if (active.length !== 1 || !lease.providerLeaseId || lease.releasedAt || lease.cleanupStatus || + lease.expiresAt && lease.expiresAt.getTime() <= Date.now() || + lease.issueId !== input.issueId || lease.executionWorkspaceId !== (input.executionWorkspaceId ?? null) || + lease.metadata?.runnerTaskLease !== true || lease.metadata?.runnerTaskRetiring === true || + lease.metadata?.pluginId !== plugin.id || lease.metadata?.driverKey !== parsed.config.driverKey || + lease.metadata?.runnerTaskFingerprint !== fingerprint) throw new Error("Active plugin task placement changed"); + return toEnvironmentLeaseSnapshot(lease); + } + } const candidates = warm ? await db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.environmentId, input.environment.id), eq(environmentLeases.status, "retained"), eq(environmentLeases.issueId, input.issueId!), @@ -3523,14 +3540,25 @@ function createPluginEnvironmentDriver( sql`${environmentLeases.metadata}->>'agentId' = ${input.agentId!}`, sql`${environmentLeases.metadata}->>'pluginId' = ${plugin.id}`, sql`${environmentLeases.metadata}->>'driverKey' = ${parsed.config.driverKey}`, + sql`${environmentLeases.metadata}->'portableTaskSuccessor' is null`, )) : []; if (candidates.length > 1) throw new Error("Ambiguous retained plugin task ownership"); const previous = candidates[0]; + const previousSessionId = (previous?.metadata?.runnerTaskIdentity as { sessionId?: unknown } | undefined)?.sessionId; + const previousAccountIdentity = previous?.metadata?.runnerTaskAccountIdentity; + const freshPortableSession = driver.sessionHandoffVersion === 1 && input.runnerTaskResumeSessionId !== undefined && + (input.runnerTaskResumeSessionId === null || input.runnerTaskResumeSessionId !== previousSessionId); + const handoffPrevious = !freshPortableSession && driver.sessionHandoffVersion === 1 && previous?.providerLeaseId && + previous.metadata?.runnerTaskFingerprint === fingerprint && + typeof previousAccountIdentity === "string" && previousAccountIdentity.trim().length > 0 && + typeof (previous.metadata?.providerMetadata as { ownershipGeneration?: unknown } | undefined)?.ownershipGeneration === "string" + ? previous : undefined; let resumed: PluginEnvironmentLease | undefined; - if (previous?.providerLeaseId && previous.metadata?.runnerTaskFingerprint !== fingerprint) { - // A changed mount scope must retire the old process before checkpoint recovery. + if (!handoffPrevious && previous?.providerLeaseId && + (freshPortableSession || previous.metadata?.runnerTaskFingerprint !== fingerprint)) { + // A fresh logical session or changed mount scope must retire the old process before provisioning. await this.releaseRunLease({environment:input.environment,lease:toEnvironmentLeaseSnapshot(previous),status:"released"}); - } else if (previous?.providerLeaseId) { + } else if (!handoffPrevious && previous?.providerLeaseId) { // Resume only inspects the exact computer; an unavailable reply must not create a replacement. resumed = await workerManager.call(plugin.id, "environmentResumeLease", { driverKey: parsed.config.driverKey, companyId: input.companyId, environmentId: input.environment.id, @@ -3589,6 +3617,7 @@ function createPluginEnvironmentDriver( ...(warm ? {runnerTaskProjectIds:[...input.runnerTaskProjectIds ?? []]} : {}),remoteCwd:".", ...(input.agentId ? { agentId: input.agentId } : {}), providerMetadata: providerLease.metadata ?? {}, + ...(handoffPrevious ? { portableTaskPreviousLeaseId: handoffPrevious.id } : {}), ...(input.purpose ? { purpose: input.purpose, loginCommandKey: input.loginCommandKey } : {}), environmentId: input.environment.id, ...(input.purpose !== "login" && input.adapterType === "paperclip_runner" && driver.supportsTasks === true ? { runnerTaskLease: true } : {}), @@ -3996,7 +4025,7 @@ export function environmentRuntimeService( }, /** Dispatch plugin-provided Runner execution, including resource preparation and task lifecycle. */ - async task(input: { companyId: string; leaseId: string; operation: PluginEnvironmentTaskOperation; executionContext?: unknown; expectedTaskId?: string }) { + async task(input: Parameters[2]) { if (!options.pluginWorkerManager) throw new Error("Plugin-provided Runner execution worker manager unavailable"); return executeEnvironmentTask(db, options.pluginWorkerManager, input); }, @@ -4015,6 +4044,7 @@ export function environmentRuntimeService( persistedExecutionWorkspace: Pick | null; executionWorkspaceSettings?: IssueExecutionWorkspaceSettings | null; runnerTaskProjectIds?: readonly string[]; + runnerTaskResumeSessionId?: string | null; /** The agent's adapter type for this run (mixed-harness environments). */ adapterType?: string | null; resolveSnapshotIdentity?: boolean; @@ -4060,6 +4090,7 @@ export function environmentRuntimeService( executionWorkspaceMode: leaseContext.executionWorkspaceMode, executionWorkspaceSettings: input.executionWorkspaceSettings ?? null, runnerTaskProjectIds: input.runnerTaskProjectIds, + runnerTaskResumeSessionId: input.runnerTaskResumeSessionId, purpose: input.purpose, loginCommandKey: input.loginCommandKey, adapterType: input.adapterType ?? null, diff --git a/server/src/services/environment-task-runtime.ts b/server/src/services/environment-task-runtime.ts index 12f7e54913d..87020bb274f 100644 --- a/server/src/services/environment-task-runtime.ts +++ b/server/src/services/environment-task-runtime.ts @@ -5,19 +5,22 @@ import { createOwnedSandbox, queueSandboxCreationCleanup, recordSandboxCreationR import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; import { releaseProviderRecoveryTransfers, stageProviderFreshWorkspace } from "./provider-fresh-workspace.js"; import { readWorkspaceProviderBinding } from "@paperclipai/shared"; -import { settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js"; +import { canReadStoppedPluginTaskCredential, settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js"; import { and, eq, inArray, sql } from "drizzle-orm"; import { environmentLeases, executionWorkspaces, environments, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db"; import { freshWorkspaceSchema, environmentTaskOperationSchema, parseEnvironmentTaskResult, environmentTaskErrorData, readEnvironmentTaskErrorDiagnostic, JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, environmentTaskErrorCodeData, PluginEnvironmentTaskError, readEnvironmentTaskErrorCode, - type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk"; + type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation, type PluginEnvironmentTaskResult } from "@paperclipai/plugin-sdk"; import { pluginRegistryService } from "./plugin-registry.js"; import { remoteTerminationReceipt } from "./remote-execution-termination.js"; import type { PluginWorkerManager } from "./plugin-worker-manager.js"; import { NativeExecutionContext } from "./native-runtime/native-execution-context.js"; import { assertPluginTaskSourceAttachment, assertPluginTaskSourceRecovery } from "./native-runtime/plugin-task-source-recovery.js"; import { pluginAgentRuntimeIdentity } from "./plugin-agent-runtime-state.js"; +import { assertPluginTaskOwnershipDispatch, PluginTaskOwnershipPendingError, readyPortableTaskSource, reconcilePluginTaskAdmission } from "./plugin-task-ownership.js"; + +export type PluginTaskDescription = Extract; export function isSettledReadyRunnerState(state: Record): boolean { const commands = state.processedCommands; @@ -48,14 +51,15 @@ export async function pluginEnvironmentSupportsTasks(db: Db, workers: PluginWork * never a plugin identity, resource endpoint, or agent/project authority. * A timeout is ambiguous: retry the same operation against the same lease. */ -export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManager, input: { +export async function executeEnvironmentTask(db: Db, workers: Pick, input: { companyId: string; leaseId: string; operation: PluginEnvironmentTaskOperation; executionContext?: unknown; + ownershipSignal?: AbortSignal; /** Physical task captured by an existing owner, independent of the mutable lease. */ expectedTaskId?: string; -}) { +}): Promise { const operation = environmentTaskOperationSchema.parse(input.operation); const [row] = await db.select({ lease: environmentLeases, environment: environments, run: heartbeatRuns }) .from(environmentLeases) @@ -74,6 +78,27 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage const taskId = row.lease.providerLeaseId; if (input.expectedTaskId !== undefined && input.expectedTaskId !== taskId) throw new Error("Plugin Runner physical task changed"); const metadata = lease.metadata ?? {}; + const ownership = operation.kind === "ensure_ownership"; + if (operation.kind === "describe" && metadata.portableTaskOwnershipReady !== true && metadata.portableTaskSuccessor === undefined && + (metadata.portableTaskPredecessor !== undefined || typeof metadata.portableTaskPreviousLeaseId === "string")) { + await reconcilePluginTaskAdmission(db, lease, request => + executeEnvironmentTask(db, workers, { ...input, operation: request }), context?.signal ?? input.ownershipSignal); + return executeEnvironmentTask(db, workers, input); + } + const requiresOwnership = !["ensure_ownership", "status", "stop", "complete"].includes(operation.kind); + const stoppedCredentialHandoff = operation.kind === "credentials" && input.expectedTaskId === taskId && + canReadStoppedPluginTaskCredential(metadata, operation.accountIdentity); + if (requiresOwnership && (metadata.portableTaskSuccessor !== undefined && !stoppedCredentialHandoff || + (metadata.portableTaskPredecessor !== undefined || metadata.portableTaskPreviousLeaseId !== undefined) && metadata.portableTaskOwnershipReady !== true)) { + throw new Error("Plugin Runner session ownership unavailable"); + } + if (ownership) { + if (!environment || lease.status !== "active" || lease.releasedAt || run?.status !== "running" || + metadata.runnerTaskRetiring === true || lease.expiresAt && lease.expiresAt.getTime() <= Date.now()) { + throw new Error("Plugin-provided Runner execution lease is not active"); + } + assertPluginTaskOwnershipDispatch(lease, operation); + } // A different launch identity is authorized by the saved consumer claim, never by the mismatch alone. const sourceRecovery = run && ((operation.kind === "prepare" && operation.requirements.identity.runId !== run.id && metadata.runnerTaskSourceRecovery) || (operation.kind === "submit" && operation.runner.runId !== run.id && metadata.runnerTaskSourceRecovery)) @@ -98,12 +123,21 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage const savedIdentity = metadata.runnerTaskIdentity as Record | undefined; const prior = operation.kind === "prepare" ? operation.priorIdentity : undefined; const priorSaved = metadata.runnerTaskPriorIdentity as Record | undefined; + const firstPortableResume = operation.kind === "resume" && !savedIdentity && + metadata.portableTaskPredecessor !== undefined; + if (firstPortableResume && (run?.status !== "running" || + !await readyPortableTaskSource(db, input.companyId, lease.id, operation.identity))) { + throw new Error("Plugin Runner session ownership unavailable"); + } const matches = (identity: typeof prior, saved: Record | undefined) => !!identity && !!saved && identity.runnerInstanceId === saved.runnerId && identity.environmentLeaseId === saved.leaseId && identity.normalizedSessionId === saved.sessionId && identity.runId === saved.runId && identity.turnId === saved.turnId && identity.itemId === saved.itemId; + const portableCheckpoint = operation.kind === "read_state" && !resumeFrom && !savedIdentity && + metadata.portableTaskPredecessor !== undefined + ? (await readyPortableTaskSource(db, input.companyId, lease.id, operation.identity))?.taskId : undefined; const ownCheckpoint = checkpointIdentity && !resumeFrom && - (matches(checkpointIdentity,savedIdentity) || operation.kind !== "read_file" && matches(checkpointIdentity,priorSaved)); + (portableCheckpoint || matches(checkpointIdentity,savedIdentity) || operation.kind !== "read_file" && matches(checkpointIdentity,priorSaved)); if (checkpointIdentity && !resumeFrom && !ownCheckpoint) throw new Error("Plugin Runner state identity mismatch"); const retainedCheckpointRead = operation.kind === "read_state" && ownCheckpoint && lease.leasePolicy === "reuse_by_environment" && ["retained", "expired"].includes(lease.status) && @@ -122,10 +156,21 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage const sourceAttachment = operation.kind === "prepare" && prior && sourceReference?.identity?.runId === prior.runId ? await assertPluginTaskSourceAttachment(db, { companyId: input.companyId, leaseId: lease.id, context: context instanceof NativeExecutionContext ? context : undefined, operation }) : undefined; - const warmAttachment = !!prior && (metadata.runnerTaskReused === true || !!sourceAttachment) && - (matches(prior, savedIdentity) || matches(prior, priorSaved) && operation.kind === "prepare" && matches(operation.requirements.identity, savedIdentity)) && runner?.runnerId === prior.runnerInstanceId && + const portableAttachment = prior && !sourceAttachment && metadata.runnerTaskReused !== true && + metadata.portableTaskPredecessor !== undefined + ? (await readyPortableTaskSource(db, input.companyId, lease.id, prior))?.taskId : undefined; + const warmAttachment = !!prior && (metadata.runnerTaskReused === true || !!sourceAttachment || !!portableAttachment) && + (portableAttachment && !savedIdentity || matches(prior, savedIdentity) || + matches(prior, priorSaved) && operation.kind === "prepare" && matches(operation.requirements.identity, savedIdentity)) && + runner?.runnerId === prior.runnerInstanceId && runner?.leaseId === prior.environmentLeaseId && runner?.sessionId === prior.normalizedSessionId; if (prior && !warmAttachment) throw new Error("Plugin warm task prior identity mismatch"); + const portableAuthority = firstPortableResume || portableAttachment + ? sql`${environmentLeases.metadata}->'portableTaskSuccessor' is null + and ${environmentLeases.metadata}->'portableTaskOwnershipReady' = 'true'::jsonb + and coalesce(${environmentLeases.metadata}->>'portableTaskCheckpointUnavailable', 'false') != 'true' + and ${environmentLeases.metadata}->'portableTaskPredecessor' = ${JSON.stringify(metadata.portableTaskPredecessor)}::jsonb` + : undefined; if (runner && !sourceRecovery && ((!checkpointIdentity || !resumeFrom) && !(ownCheckpoint && operation.kind !== "read_file") && runner.runId !== (activeProbe ? probe.runId : run?.id) || (!resumeFrom && runner.leaseId !== lease.id && !ownCheckpoint && !warmAttachment) || (activeProbe && (runner.sessionId !== probe.sessionId || (launches && operation.projectIds.length > 0))))) { throw new Error("Plugin-provided Runner execution identity mismatch"); @@ -168,10 +213,18 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage const plugin = await pluginRegistryService(db).getById(metadata.pluginId); const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === metadata.driverKey); if (!plugin || plugin.status !== "ready" || !driver?.supportsTasks || - !plugin.manifestJson.capabilities.includes("environment.drivers.register") || - !workers.getWorker(plugin.id)?.supportedMethods.includes("environmentTask")) { + !plugin.manifestJson.capabilities.includes("environment.drivers.register")) { throw new Error("Plugin-provided Runner execution provider unavailable"); } + const worker = workers.getWorker(plugin.id); + if (ownership && !worker && driver.sessionHandoffVersion === 1) throw new PluginTaskOwnershipPendingError(); + if (!worker?.supportedMethods.includes("environmentTask")) { + throw new Error("Plugin-provided Runner execution provider unavailable"); + } + if (operation.kind === "resume" && operation.handoffCredential && driver.sessionHandoffVersion !== 1) { + throw new Error("Plugin Runner credential handoff unavailable"); + } + if (ownership && driver.sessionHandoffVersion !== 1) throw new Error("Plugin Runner ownership reconciliation unavailable"); const callWorker = async (method: M, params: HostToWorkerMethods[M][0], timeout?: number): Promise => { if (!context) { const cleanup = metadata.runnerTaskStartupCleanup as { contextId?: unknown; requiresPreparationStop?: unknown } | undefined; @@ -255,13 +308,23 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage operation.launch.maxRuntimeMs = Math.min(remainingMs, ...requestedLimits); delete operation.launch.maxLifetimeMs; } - if (operation.kind === "submit" || operation.kind === "copilot_metadata") { + if (operation.kind === "submit" || operation.kind === "copilot_metadata" || firstPortableResume) { // No ticket, headers, environment values, or launch paths belong in metadata. + const dispatched = { + runnerTaskLease: true, runnerTaskDispatched: true, + runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false, + ...(operation.kind === "submit" ? { + runnerTaskIdentity: operation.runner, + runnerTaskResumeFrom: operation.resumeFrom ?? null, + runnerTaskAccountIdentity: operation.accountIdentity, + } : {}), + }; const marked = await db.update(environmentLeases).set({ - metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskAdmissionRejection') || ${JSON.stringify({ runnerTaskLease: true, runnerTaskDispatched: true, runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false, ...(operation.kind === "submit" ? { runnerTaskIdentity: operation.runner, runnerTaskResumeFrom: operation.resumeFrom ?? null } : {}) })}::jsonb`, + metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskAdmissionRejection') || ${JSON.stringify(dispatched)}::jsonb`, }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`, requiredProjectAuthProof ? sql`${environmentLeases.metadata}->'runnerTaskProjectAuthValidation' = ${JSON.stringify(requiredProjectAuthProof)}::jsonb` : undefined, + firstPortableResume ? portableAuthority : undefined, )).returning({ id: environmentLeases.id }); if (marked.length !== 1) throw new Error("Plugin-provided Runner execution lease is not active"); } @@ -276,7 +339,9 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage ...(startup ? { runnerTaskStartup: startup } : {}), })}::jsonb` }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`, - sql`${environmentLeases.metadata}->'runnerTaskIdentity' = ${JSON.stringify(savedIdentity)}::jsonb`)).returning({id:environmentLeases.id}); + portableAttachment ? portableAuthority : undefined, + savedIdentity ? sql`${environmentLeases.metadata}->'runnerTaskIdentity' = ${JSON.stringify(savedIdentity)}::jsonb` + : sql`${environmentLeases.metadata}->'runnerTaskIdentity' is null`)).returning({id:environmentLeases.id}); if (marked.length !== 1) throw new Error("Plugin warm task attachment lost authority"); } async function recordTaskStop(result: Extract, {kind:"status" | "copilot_metadata"}>) { @@ -322,6 +387,7 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage ...(recoverableCreation ? { creationOperationVersion: 1, providerMetadata: metadata.providerMetadata } : {}) }, }, async identity => { const params = { driverKey: replacementDriverKey, companyId: input.companyId, environmentId: environment!.id, + executionWorkspaceId: lease.executionWorkspaceId ?? undefined, issueId: lease.issueId, config: driverConfig, runId: identity.runId, creationOperationId: identity.id, agentId: run!.agentId, adapterType: "paperclip_runner", ...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? { requestedExpiresAt: metadata.runnerTaskRequestedExpiresAt } : {}) }; @@ -414,8 +480,15 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage ...(executionContext ? { executionContext } : {}), }; const timeout = operation.kind === "copilot_metadata" ? Math.max(1, Math.min(40_000, Number(probe?.expiresAtMs) - Date.now())) - : executionContext ? Math.max(1, context!.deadlineAtMs - Date.now()) : 15_000; - const result = await callWorker("environmentTask", params, timeout); + : executionContext ? Math.max(1, context!.deadlineAtMs - Date.now()) + : driver.sessionHandoffVersion === 1 && (ownership || operation.kind === "describe") ? 90_000 : 15_000; + const result = await callWorker("environmentTask", params, timeout).catch(error => { + // The plugin may have applied the saved claim before its reply was lost. + // Keep the same operation ID; the RPC response below still needs validation. + if (ownership && (!(error instanceof JsonRpcCallError) || + error.code === PLUGIN_RPC_ERROR_CODES.TIMEOUT || error.code === PLUGIN_RPC_ERROR_CODES.WORKER_UNAVAILABLE)) throw new PluginTaskOwnershipPendingError(); + throw error; + }); if (executionContext) context!.assertOpen(); const parsed = parseEnvironmentTaskResult(operation, taskId, result); if (operation.kind === "submit" && parsed.kind === "admission_rejected") { @@ -443,6 +516,11 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage await assertPluginTaskSourceAttachment(db, { companyId: input.companyId, leaseId: lease.id, context: context instanceof NativeExecutionContext ? context : undefined, operation }); } + if (portableCheckpoint && operation.kind === "read_state" || portableAttachment && operation.kind === "prepare") { + const identity = operation.kind === "read_state" ? operation.identity : operation.priorIdentity; + const sourceTaskId = (await readyPortableTaskSource(db, input.companyId, lease.id, identity))?.taskId; + if (sourceTaskId !== (portableCheckpoint || portableAttachment)) throw new Error("Plugin Runner session ownership changed"); + } if (operation.kind === "read_file") { const [current] = await db.select({ lease: environmentLeases, run: heartbeatRuns }).from(environmentLeases) .leftJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, environmentLeases.companyId))) @@ -499,6 +577,7 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage } return parsed; } catch (error) { + if (error instanceof PluginTaskOwnershipPendingError) throw error; const diagnostic = operation.kind === "submit" && operation.resumeFrom ? readEnvironmentTaskErrorDiagnostic(error) : null; if (diagnostic) throw new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR, message: "Plugin Runner source state unavailable", diff --git a/server/src/services/environments.ts b/server/src/services/environments.ts index 29b73fc9eb9..cf8e197aee1 100644 --- a/server/src/services/environments.ts +++ b/server/src/services/environments.ts @@ -274,7 +274,7 @@ function resolveKubernetesConfig( return companyIdOrConfig; } -function toEnvironmentLease(row: EnvironmentLeaseRow): EnvironmentLease { +export function toEnvironmentLease(row: EnvironmentLeaseRow): EnvironmentLease { return { id: row.id, companyId: row.companyId, @@ -1649,6 +1649,7 @@ export function environmentService(db: Db) { // lease holds older session state. That claim and this // handoff compete on this row, so only one of them wins. sql`not (coalesce(${environmentLeases.metadata}, '{}'::jsonb) ? 'reusableSandboxLeaseSuperseded')`, + sql`not (coalesce(${environmentLeases.metadata}, '{}'::jsonb) ? 'portableTaskSuccessor')`, ), ) .returning(); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 220bc6aa7a5..32481819937 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1897,6 +1897,7 @@ export function heartbeatService( runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; + portablePreparationLeaseId?: string; onNativeExecutionLeaseAcquired?: () => Promise; } = {}, ) { @@ -1914,7 +1915,8 @@ export function heartbeatService( async function executeRunAttempt( runId: string, - runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; onNativeExecutionLeaseAcquired?: () => Promise }, + runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; portablePreparationLeaseId?: string; + onNativeExecutionLeaseAcquired?: () => Promise }, ) { const attemptStartedAtMs = Date.now(); let attestedQuestionResponseAtMs: number | null = null; @@ -2087,8 +2089,9 @@ export function heartbeatService( let dispatchGateEntered = false; const nativeDispatchMayHaveStarted = () => nativeDispatchStarted || dispatchGateEntered || nativeOwnershipHeld || Boolean(runOptions.nativeRestartRecovery || runOptions.nativeLeaseOwner); + const preparationHandedOff = () => shutdownHandedOff() && !legacyAdapterEntered && !nativeDispatchStarted; const leaveToNextController = async (error: unknown) => { - if (!shutdownHandedOff() || legacyAdapterEntered || nativeDispatchStarted) return false; + if (!preparationHandedOff()) return false; nativeSessionResumeScheduled = true; const reason = error instanceof Error ? error.message : String(error); const errorCode = /^[a-z][a-z0-9_]{0,80}$/.test(reason) ? reason : null; @@ -2895,7 +2898,7 @@ export function heartbeatService( executionTarget, workspaceRestoreSource, } = await timePreparation("workspace", () => prepareHeartbeatWorkspace(db, { - run, agent, controllerLease, nativeRunnerPreparationSpans, + run, agent, controllerLease, nativeRunnerPreparationSpans, signal: executionControl.controller.signal, task: { issueRef, issueId, @@ -2930,6 +2933,7 @@ export function heartbeatService( configSnapshot, secretManifest, managedAiCredentialIdentity: managedAiRuntime?.identity, + runnerTaskResumeSessionId: run.nativeSessionId ?? preparedSession.taskResumeSessionId, }, reuse: { requestedShouldReuseExisting, @@ -3295,7 +3299,7 @@ export function heartbeatService( context.projectId = executionWorkspace.projectId; } const { - previousSessionDisplayId, runtimeSessionParamsForAdapter, taskSessionCredentialCompatible, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction, + previousSessionDisplayId, runtimeSessionParamsForAdapter, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction, } = await resolveHeartbeatAdapterSession({ agent, task: { context, taskKey, issueId, continuationSummary, executionContinuation }, @@ -3507,9 +3511,7 @@ export function heartbeatService( session: { persistedNativeExecutionInput, persistedRunnerProfile, - taskSessionCredentialCompatible, - taskSessionDecodedParams, - taskSessionForRun, + taskResumeSessionId: preparedSession.taskResumeSessionId, isFailedChatRunRetry, getFreshSessionHandoff, }, @@ -3585,6 +3587,8 @@ export function heartbeatService( adapter, runtimeForAdapter, getNativeFreshSessionHandoff, + pluginTaskDescription: runtimeSelection.pluginTaskDescription, + toolRefreshOnResume: runtimeSelection.toolRefreshOnResume, getFreshSessionHandoff, runOptions, }, @@ -3999,6 +4003,7 @@ export function heartbeatService( await instructionWorkingCopy?.nativeInstructionReservation?.release().catch(error => logger.warn({ runId: run.id, err: error }, "Managed warm session preparation cleanup failed")); if (managedAiRuntime) await managedAiRuntime.cleanup().catch(() => logger.warn({ runId: run.id }, "AI connection refresh or cleanup failed")); let latestRun = await getRun(run.id).catch(() => null); + if (preparationHandedOff() && (!latestRun || latestRun.status === "running")) nativeSessionResumeScheduled = true; try { if (latestRun && isHeartbeatRunTerminalStatus(latestRun.status)) { await writeExecutionControl(db, DELIVERY_QUEUES.executionFinalization, tx => tx diff --git a/server/src/services/heartbeat/instruction-working-copy.ts b/server/src/services/heartbeat/instruction-working-copy.ts index d9a98f6393b..a3aeb34c717 100644 --- a/server/src/services/heartbeat/instruction-working-copy.ts +++ b/server/src/services/heartbeat/instruction-working-copy.ts @@ -64,7 +64,7 @@ interface InstructionWorkingCopyInput { context: Selection["task"]["context"]; session: { taskSession: { lastRunId: string | null } | null; - taskSessionForRun: Selection["session"]["taskSessionForRun"]; + taskSessionForRun: { lastRunId: string | null } | null; sessionConfigMetadata: EffectiveRunSessionConfigMetadata; }; workspace: Pick; diff --git a/server/src/services/heartbeat/recovery.test.ts b/server/src/services/heartbeat/recovery.test.ts index e26bb112ccf..bf894afacab 100644 --- a/server/src/services/heartbeat/recovery.test.ts +++ b/server/src/services/heartbeat/recovery.test.ts @@ -30,6 +30,7 @@ import { hasRemoteTerminationReceipt, remoteTerminationReceipt } from "../remote import { persistHeartbeatRunProcessMetadata } from "./run-lifecycle.js"; import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; import { hasNativeLocalProcessStop, PROCESS_START_REQUESTED, releaseSettledNativeRunProcess } from "../native-local-process-stop.js"; +import { legacyControllerBootId } from "../legacy-controller-lease.js"; type Run = typeof heartbeatRuns.$inferSelect; @@ -212,6 +213,40 @@ describePostgres("heartbeat recovery module ownership", () => { return (await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, invocationSource: "automation", status: "running", runtimeMode: "legacy" }).returning())[0]; } + it.each(["unlaunched", "dispatched", "reassigned", "ambiguous"] as const)("claims the same expired portable preparation only with its original task lock (%s)", async state => { + const run = await seedRun(), deps = callbacks(db); + const environmentId = randomUUID(), leaseId = randomUUID(); + const oldBootId = randomUUID(); + await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId)); + await db.update(heartbeatRuns).set({ scopeKind: "issue", controllerBootId: oldBootId, controllerLeaseExpiresAt: new Date(0), + executionStage: "preparing" }).where(eq(heartbeatRuns.id, run.id)); + if (state !== "reassigned") await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, run.issueId!)); + await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "plugin", config: {} }); + await db.insert(environmentLeases).values({ id: leaseId, companyId: run.companyId, environmentId, heartbeatRunId: run.id, + issueId: run.issueId, provider: "plugin:fixture:tasks", providerLeaseId: "portable-task", status: "active", + metadata: { agentId: run.agentId, driver: "plugin", runnerTaskLease: true, portableTaskPreviousLeaseId: randomUUID(), + ...(state === "dispatched" ? { runnerTaskDispatched: true } : {}) } }); + if (state === "ambiguous") await db.insert(environmentLeases).values({ companyId: run.companyId, environmentId, + heartbeatRunId: run.id, issueId: run.issueId, provider: "plugin:fixture:tasks", providerLeaseId: "competing-task", status: "active" }); + const scope = { companyId: run.companyId, runId: run.id }; + const recovery = createHeartbeatRecovery(db, deps); + await Promise.all([recovery.recoverActiveExecutions(scope), recovery.recoverActiveExecutions(scope)]); + if (state === "unlaunched") { + expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(run.id, { portablePreparationLeaseId: leaseId }); + expect(deps.setRunStatusFromLive).not.toHaveBeenCalled(); + const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + expect(current).toMatchObject({ status: "running", controllerBootId: legacyControllerBootId, executionStage: "preparing" }); + expect(current.controllerLeaseExpiresAt!.getTime()).toBeGreaterThan(Date.now()); + expect((await db.select().from(issues).where(eq(issues.id, run.issueId!)))[0].executionRunId).toBe(run.id); + expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)))[0]).toMatchObject({ + status: "active", providerLeaseId: "portable-task", heartbeatRunId: run.id, + }); + } else { + expect(deps.executeRun).not.toHaveBeenCalled(); + expect(deps.setRunStatusFromLive).toHaveBeenCalledOnce(); + } + }); + it.each([false, true])("delivers scoped recovery to a preparation without a runner PID or checkpoint (locally tracked=%s)", async tracked => { const run = await seedRun(), deps = callbacks(db); await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId)); diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index 393a39c8fff..d2f4fe09b02 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -38,9 +38,11 @@ import { } from "../sandbox-stop-and-retain.js"; import { discoverSettledNativeRunProcesses, PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED, recordNativeLocalProcessStopInTransaction, releaseSettledNativeRunProcess, retryPendingSettledNativeRunProcesses } from "../native-local-process-stop.js"; import { pendingCaseHandoffForRun } from "./case-handoff-lifecycle.js"; +import { portableTaskPreparationLeaseId } from "../plugin-task-ownership.js"; import { retireInheritedRemoteRunnerIdentity } from "../native-runtime/native-inherited-runner-retry.js"; import { LEGACY_LIVE_OWNER_SETTLE_GRACE_MS, + handoffLegacyControllerPreparation, legacyControllerBootId, liveLegacyControllerExpiresAt, proveLegacyController, @@ -157,6 +159,7 @@ export interface HeartbeatRecoveryDependencies extends Pick Promise; }) => Promise; scheduleNativeSessionResumeDispatch: (runId: string, nextAttemptAt: Date, mode?: "restart") => void; @@ -1269,6 +1272,13 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD restartSuspendedRunIds.push(run.id); return; } + if (agent.adapterType === "paperclip_runner" && run.runtimeMode === "legacy" && run.scopeKind === "issue" && + run.issueId && run.executionStage === "preparing" && run.controllerBootId === legacyControllerBootId && + await handoffLegacyControllerPreparation(db, run, portableTaskPreparationLeaseId(db, run), + () => markRunHandedOffForShutdown(run.id))) { + restartSuspendedRunIds.push(run.id); + return; + } const message = `Interrupted by graceful server shutdown (${signal})`; const running = runningProcesses.get(run.id); const control = run.runtimeMode === "legacy" ? adapterExecutionControls.get(run.id) : undefined; @@ -2467,6 +2477,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD nativeWorkspaceFinalizationPending, } of activeRuns) { if (run.runtimeMode === "native" && readNativeStartupCleanup(run)) continue; + if (run.runtimeMode === "legacy" && isRunHandedOffForShutdown(run.id)) continue; if (parseObject(run.contextSnapshot).nativeQuestionCancellation) continue; // Native recovery owns these dispositions. Its writes happened after this // inventory was read; do not reap or overwrite them using the old row. @@ -2686,8 +2697,18 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD readNonEmptyString(runContext.wakeReason) === "issue_monitor_due" && monitorNextCheckAt !== undefined && (!monitorNextCheckAt || monitorNextCheckAt.getTime() <= now.getTime()); - const revokedState = await revokeExpiredLegacyControllerWithState(db, run); + const reclaimPortablePreparation = priorBootSetupExpired && adapterType === "paperclip_runner" && run.scopeKind === "issue" && run.issueId + ? portableTaskPreparationLeaseId(db, run) : undefined; + const revokedState = await revokeExpiredLegacyControllerWithState(db, run, reclaimPortablePreparation); if (!revokedState) continue; + if (revokedState.portablePreparationLeaseId) { + const execution = executeRun(run.id, { portablePreparationLeaseId: revokedState.portablePreparationLeaseId }).catch(error => { + logger.error({ err: error, runId: run.id }, "portable task preparation recovery failed"); + }); + activeRunExecutionPromises.add(execution); + void execution.finally(() => activeRunExecutionPromises.delete(execution)); + continue; + } // The scan snapshot can predate a child the old controller saved just // before it died. The child checks above used that snapshot, so a proven // gone controller is never enough here: when revocation reports a child diff --git a/server/src/services/heartbeat/runtime-execution.ts b/server/src/services/heartbeat/runtime-execution.ts index 7880cff8c51..a8ef0e0bcdb 100644 --- a/server/src/services/heartbeat/runtime-execution.ts +++ b/server/src/services/heartbeat/runtime-execution.ts @@ -128,7 +128,7 @@ export interface HeartbeatRuntimeExecutionInput { issueId: string | null; taskKey: string | null; }; - runtime: Pick & { + runtime: Pick & { nativeRuntimeResolution: Selection["nativeRuntimeResolution"]; adapter: ServerAdapterModule; runtimeForAdapter: AdapterExecutionContext["runtime"]; @@ -636,9 +636,11 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe observability, getFreshSessionHandoff: getNativeFreshSessionHandoff, refreshTools: context.refreshTools === true, + toolRefreshOnResume: input.runtime.toolRefreshOnResume, conversationMode: isConversation(issueContext), turnTimeoutMs: Math.max(0, asNumber(runtimeConfig.timeoutSec, 0)) * 1_000, runnerInstanceId: nativeRunnerInstanceId, + pluginTaskDescription: input.runtime.pluginTaskDescription, leaseOwner: runOptions.nativeLeaseOwner, restartRecovery: runOptions.nativeRestartRecovery, onExecutionLeaseAcquired: runOptions.onNativeExecutionLeaseAcquired, diff --git a/server/src/services/heartbeat/runtime-selection.test.ts b/server/src/services/heartbeat/runtime-selection.test.ts index 0d8239399ec..3268ad3412b 100644 --- a/server/src/services/heartbeat/runtime-selection.test.ts +++ b/server/src/services/heartbeat/runtime-selection.test.ts @@ -25,6 +25,7 @@ import { prepareNativeWorkspaceSync } from "../native-runtime/native-workspace-s import { resolveHeartbeatNativeRuntimeMode } from "../native-runtime/runtime-mode.js"; import { claimNativeRestartRecoveries } from "../native-runtime/native-restart-recovery.js"; import { buildNativeModelEnvelope, canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js"; +import * as runner from "../../vendor/paperclip-runner/index.js"; import { prepareAndSelectHeartbeatRuntime, selectHeartbeatRuntime, type HeartbeatRuntimeSelectionInput } from "./runtime-selection.js"; import { resolveRunnerEnvironmentForRun, resolveWarmRunnerIdleTimeoutMsForRun } from "../runner-environment-lifecycle.js"; import { reserveRunBudget } from "../budget-reservations.js"; @@ -82,7 +83,7 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = run, agent, nativeRuntimeResolution: resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: null, workspaceId: null }), task: { issueRef: issue, issueContext: null, context: { paperclipTaskMarkdown: "Complete this task" }, executionContinuation: null, safeWakeComments: [], safeWakeCommentContext: null, paperclipWakePayload: null, acceptedPlanContinuationWake: false }, - session: { persistedNativeExecutionInput: null, persistedRunnerProfile: {}, taskSessionCredentialCompatible: true, taskSessionDecodedParams: null, taskSessionForRun: null, isFailedChatRunRetry: false, getFreshSessionHandoff: vi.fn(async () => "Earlier context") }, + session: { persistedNativeExecutionInput: null, persistedRunnerProfile: {}, taskResumeSessionId: null, isFailedChatRunRetry: false, getFreshSessionHandoff: vi.fn(async () => "Earlier context") }, workspace: { persistedExecutionWorkspace: null, executionWorkspace: { cwd: "/tmp/paperclip-runtime-selection", repoUrl: null, repoRef: null, branchName: null }, executionTarget: null, isDotRun: false, projectRepositoryPaths: [], activeEnvironmentLease: { lease: { id: randomUUID(), companyId: company.id, environmentId: null, executionWorkspaceId: null, issueId: issue.id, heartbeatRunId: run.id, status: "active", leasePolicy: "ephemeral", provider: null, providerLeaseId: null, acquiredAt: now, lastUsedAt: now, expiresAt: null, releasedAt: null, failureReason: null, cleanupStatus: null, metadata: null, createdAt: now, updatedAt: now } }, }, @@ -97,6 +98,58 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = const coordinators = (id: string) => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, id)); const sandbox = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", reusableLeaseConfigured: false }; + it.each(["uninspected", "confirmed", "replaced"])("selects an independent native session after %s disk-only recovery", async placement => { + const { input, run, issue } = await fixture(); + const priorSession = randomUUID(); + const lease = input.workspace.activeEnvironmentLease!.lease; + const description = { kind: "description" as const, taskId: randomUUID(), runner: { version: "test", digest: `sha256:${"a".repeat(64)}` } }; + const previousDescription = placement === "replaced" ? { ...description, taskId: randomUUID() } : description; + const pending = { ...lease, providerLeaseId: description.taskId, metadata: { portableTaskPreviousLeaseId: randomUUID(), + portableTaskPredecessor: { operation: { identity: { normalizedSessionId: priorSession } } }, + } }; + const ready = { ...pending.metadata, portableTaskOwnershipReady: true, portableTaskCheckpointUnavailable: true }; + await db.insert(environmentLeases).values(placement === "confirmed" ? { ...pending, metadata: ready } : pending); + input.workspace.activeEnvironmentLease!.lease = pending; + if (placement !== "uninspected") input.workspace.activeEnvironmentLease!.pluginTaskDescription = previousDescription; + const taskRpc = vi.fn(async ({ kind }: { kind: string }) => { + expect(kind).toBe("describe"); + await db.update(environmentLeases).set({ + metadata: ready, + }).where(eq(environmentLeases.id, lease.id)); + return description; + }); + input.workspace.executionTarget = { ...sandbox, leaseId: lease.id, runnerTask: taskRpc, + reusableLeaseConfigured: true, effectiveCapabilities: { reusableLeases: true }, + runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } } as never; + input.session.taskResumeSessionId = priorSession; + const network = vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("Cloud is unavailable")); + try { + const selected = await selectHeartbeatRuntime(db, input); + expect(selected.selected).toBe(true); + if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection"); + const saved = await read(run.id); + expect(saved.nativeSessionId).toBeTruthy(); + expect(saved.nativeSessionId).not.toBe(priorSession); + expect(await selected.getNativeFreshSessionHandoff?.()).toBe("Earlier context"); + expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0]).toEqual(issue); + expect(selected.pluginTaskDescription).toEqual(description); + expect(taskRpc).toHaveBeenCalledTimes(placement === "confirmed" ? 0 : 1); + expect(network).not.toHaveBeenCalled(); + + // A lost reply after committing selection must allow the same replacement run to retry. + input.run = saved; + input.session.persistedRunnerProfile = saved.runnerProfileJson ?? {}; + input.session.persistedNativeExecutionInput = selected.nativeExecution; + await expect(selectHeartbeatRuntime(db, input)).resolves.toMatchObject({ + selected: true, nativeExecution: selected.nativeExecution, + }); + input.run = { ...saved, nativeSessionId: priorSession }; + await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native_portable_reseed_session_already_attached"); + } finally { + network.mockRestore(); + } + }); + it("rejects invalid local Runner configuration before native authority and retains the preparation owner", async () => { const f = await fixture(); await claimPreparation(f); @@ -623,10 +676,14 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = }); it("restores the immutable admitted input and saved checkpoint when newer context arrives", async () => { + using discovery = vi.spyOn(runner, "describeRunnerdNativeSessionBackend"); const { input, run } = await fixture(); const selected = await selectHeartbeatRuntime(db, input); if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection"); + expect(selected.toolRefreshOnResume).toBe(true); + expect(discovery).toHaveBeenCalledTimes(1); const saved = await read(run.id); + expect(saved.runnerProfileJson?.nativeToolRefreshOnResume).toBe(true); const profile = { ...saved.runnerProfileJson, sessionCheckpoint: { retained: "checkpoint" } }; await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, run.id)); input.run = await read(run.id); @@ -636,11 +693,13 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = input.task.safeWakeComments = [{ id: randomUUID(), body: "New direction belongs to a separate wake" }]; vi.mocked(buildNativeRuntimeContext).mockClear(); const resumed = await selectHeartbeatRuntime(db, input); - expect(resumed).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution, nativeRunnerInstanceId: selected.nativeRunnerInstanceId }); + expect(resumed).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution, nativeRunnerInstanceId: selected.nativeRunnerInstanceId, + toolRefreshOnResume: true }); expect((await read(run.id)).runnerProfileJson).toMatchObject(profile); expect(await db.select().from(completionContracts)).toHaveLength(1); expect(await coordinators(run.id)).toHaveLength(1); expect(buildNativeRuntimeContext).not.toHaveBeenCalled(); + expect(discovery).toHaveBeenCalledTimes(1); }); it("rejects an admitted input bound to another run without replacing the saved runtime", async () => { diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts index 3e6655b2077..18880ba4f57 100644 --- a/server/src/services/heartbeat/runtime-selection.ts +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -39,6 +39,8 @@ import { } from "@paperclipai/db"; import { getStartupTracer } from "../../instrumentation.js"; import { documentService } from "../documents.js"; +import { readPortableTaskReseed } from "../plugin-task-ownership.js"; +import type { PluginTaskDescription } from "../environment-task-runtime.js"; import { managedAgentProfileService } from "../managed-agent-profiles.js"; import { remoteAgentProfileService } from "../remote-agent-profiles.js"; import { @@ -47,7 +49,6 @@ import { buildNativeRuntimeContext, ensureNativeCompletionContract, findNativeSessionResumeRun, - isNativeSessionId, isUnusedNativeSessionBootstrap, isUnusedLegacyNativeRetryReplacement, materializeNativeInteractionResponses, @@ -117,9 +118,7 @@ export interface HeartbeatRuntimeSelectionInput { session: { persistedNativeExecutionInput: NativeExecutionInput | null; persistedRunnerProfile: Record; - taskSessionCredentialCompatible: boolean; - taskSessionDecodedParams: Record | null; - taskSessionForRun: { lastRunId: string | null } | null; + taskResumeSessionId: string | null; isFailedChatRunRetry: boolean; getFreshSessionHandoff: (() => Promise) | undefined; }; @@ -127,7 +126,9 @@ export interface HeartbeatRuntimeSelectionInput { persistedExecutionWorkspace: { id: string; metadata?: Record | null } | null; executionWorkspace: Omit[0]["workspace"], "id">; executionTarget: Parameters[0]["target"]; - activeEnvironmentLease: Pick[0], "lease"> | null; + activeEnvironmentLease: (Pick[0], "lease"> & { + pluginTaskDescription?: PluginTaskDescription; + }) | null; isDotRun: boolean; projectRepositoryPaths: string[]; onCheckpoint?: Parameters[0]["onCheckpoint"]; @@ -161,6 +162,8 @@ export type HeartbeatRuntimeSelectionResult = { selected: false } | { selected: true; nativeExecution: NativeExecutionInput | null; nativeRunnerInstanceId: string | null; + pluginTaskDescription?: PluginTaskDescription; + toolRefreshOnResume?: boolean; getNativeFreshSessionHandoff: (() => Promise) | undefined; nativeWorkspaceSync: Awaited>; }; @@ -313,16 +316,17 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele onNativeLifecycleSelected, onProviderResourceDisposition, onNativeOwnershipHeld, stopControllerLease } = input; const { issueRef, issueContext, context, executionContinuation, safeWakeComments, safeWakeCommentContext, paperclipWakePayload, acceptedPlanContinuationWake } = input.task; - const { persistedNativeExecutionInput, persistedRunnerProfile, taskSessionCredentialCompatible, taskSessionDecodedParams, - taskSessionForRun, isFailedChatRunRetry, getFreshSessionHandoff } = input.session; + const { persistedNativeExecutionInput, persistedRunnerProfile, taskResumeSessionId, isFailedChatRunRetry, getFreshSessionHandoff } = input.session; const { persistedExecutionWorkspace, executionWorkspace, executionTarget, activeEnvironmentLease, isDotRun, projectRepositoryPaths } = input.workspace; const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config; const runOptions = input.recovery; const { providerTraceRequested, providerTraceCapture } = input.trace; if (!isDotRun) assertEphemeralNativeRuntime(nativeRuntimeResolution.kind === "native"); let nativeExecution: NativeExecutionInput | null = null; + let toolRefreshOnResume: boolean | undefined; let getNativeFreshSessionHandoff: (() => Promise) | undefined; let nativeRunnerInstanceId: string | null = null; + let pluginTaskDescription = activeEnvironmentLease?.pluginTaskDescription; let nativeWorkspaceSync: Awaited> = null; let providerResourceDispositionForRun: ProviderResourceDisposition | undefined; if (nativeRuntimeResolution.kind === "native") { @@ -451,9 +455,20 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele immediateRequests: immediateCompletion.requests, immediateRequestSources: immediateCompletion.sources, })); - const taskNativeSessionId = !taskSessionCredentialCompatible ? null : readNonEmptyString( - taskSessionDecodedParams?.sessionId, - ); + const portableLease = activeEnvironmentLease?.lease; + const runnerTask = executionTarget?.kind === "remote" && executionTarget.transport === "sandbox" && + executionTarget.leaseId === portableLease?.id ? executionTarget.runnerTask : undefined; + const portable = portableLease && runnerTask ? await readPortableTaskReseed(db, { + companyId: agent.companyId, runId: run.id, lease: portableLease, description: pluginTaskDescription, + runnerTask, + }) : null; + if (portable) pluginTaskDescription = portable.description; + const reseed = portable?.reseed; + if (reseed && run.nativeSessionId && (!reseed.previousSessionId || run.nativeSessionId === reseed.previousSessionId || + persistedNativeExecutionInput?.session.normalizedSessionId !== run.nativeSessionId)) { + throw new Error("native_portable_reseed_session_already_attached"); + } + const reseedPortableSession = Boolean(reseed); // Compatibility for native retry rows created before same-run restart // recovery existed. Only an entirely unused replacement row may // inherit its source checkpoint; any process/provider evidence on the @@ -516,15 +531,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele : null; const legacyRetrySessionId = compatibleLegacyRetrySource?.nativeSessionId; - const taskResumeRunId = - taskSessionForRun?.lastRunId && - taskSessionForRun.lastRunId !== run.id && - isNativeSessionId(taskNativeSessionId) - ? taskSessionForRun.lastRunId - : null; - const resumableTaskSessionId = isDotRun ? null : taskResumeRunId - ? taskNativeSessionId - : (legacyRetrySessionId ?? null); + const resumableTaskSessionId = reseedPortableSession || isDotRun ? null : taskResumeSessionId ?? legacyRetrySessionId ?? null; const requestedNativeSessionId = run.nativeSessionId ?? resumableTaskSessionId; // A task-session lastRunId can lag a failed turn or point at an older @@ -570,6 +577,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele const persistedProfile = persistedRunnerProfile; if (persistedNativeExecutionInput) { nativeExecution = persistedNativeExecutionInput; + toolRefreshOnResume = persistedProfile.nativeToolRefreshOnResume === true; if ( nativeExecution.binding.companyId !== agent.companyId || nativeExecution.binding.runId !== run.id || @@ -896,11 +904,12 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele const backendDescriptor = await measureRunPhase("runtime_selection.provider_discovery", () => describeRunnerdNativeSessionBackend(buildExecution({ normalizedSessionId: nativeSessionId, resumedSession: previousNativeRun !== null, }))); + toolRefreshOnResume = backendDescriptor.capabilities.toolRefreshOnResume === true; const nativeExecutionWithCheckpoint = buildNativeExecutionWithCheckpoint({ previousRun: previousNativeRun, normalizedSessionId: nativeSessionId, executionTargetKind: executionTarget?.kind ?? "local", - toolRefreshOnResume: backendDescriptor.capabilities.toolRefreshOnResume === true, + toolRefreshOnResume, refreshTools: context.refreshTools === true, buildExecution, }); @@ -1037,6 +1046,8 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele : {}), nativeExecutionInput: lockedProfile.nativeExecutionInput ?? nativeExecution, + nativeToolRefreshOnResume: lockedProfile.nativeExecutionInput + ? lockedProfile.nativeToolRefreshOnResume === true : toolRefreshOnResume === true, ...(typeof lockedProfile.inheritedProcessIdentityRunId === "string" ? { inheritedProcessIdentityRunId: lockedProfile.inheritedProcessIdentityRunId } : lockedRun.processPid == null && lockedRun.processGroupId == null && lockedRun.processStartedAt == null @@ -1167,5 +1178,5 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele }) .where(eq(heartbeatRuns.id, run.id))); } - return { selected: true, nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff, nativeWorkspaceSync }; + return { selected: true, nativeExecution, nativeRunnerInstanceId, pluginTaskDescription, toolRefreshOnResume, getNativeFreshSessionHandoff, nativeWorkspaceSync }; } diff --git a/server/src/services/heartbeat/session-preparation.ts b/server/src/services/heartbeat/session-preparation.ts index 04ec0654fec..9d7729f0954 100644 --- a/server/src/services/heartbeat/session-preparation.ts +++ b/server/src/services/heartbeat/session-preparation.ts @@ -1,3 +1,4 @@ +import { isNativeSessionId } from "../native-runtime/index.js"; import { and, desc, eq } from "drizzle-orm"; import { agentConfigRevisions, type Db, type agents, type agentTaskSessions, type heartbeatRuns, type issues, type projects } from "@paperclipai/db"; import type { AiConnectionBinding, AiRuntimeConnectionBinding } from "@paperclipai/shared"; @@ -29,7 +30,7 @@ type TaskSession = typeof agentTaskSessions.$inferSelect | null; type SessionCodec = ReturnType; export interface HeartbeatSessionPreparationInput { - run: Pick; + run: Pick; agent: typeof agents.$inferSelect; task: { context: Record; @@ -204,6 +205,14 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre const sessionResetReason = sessionConfigFreshness.reasons.join("; ") || null; const taskSessionForRun = resetTaskSession ? null : taskSession; + const taskSessionCredentialCompatible = !managedAiRuntime || managedAiSessionIdentityCompatible( + taskSession?.sessionParamsJson?.paperclipAiCredentialIdentity, + managedAiRuntime.sessionIdentity, + managedAiRuntime.identity, + ); + const taskSessionId = readNonEmptyString(taskSessionDecodedParams?.sessionId); + const taskResumeSessionId = taskSessionCredentialCompatible && taskSessionForRun?.lastRunId && + taskSessionForRun.lastRunId !== run.id && isNativeSessionId(taskSessionId) ? taskSessionId : null; const getFreshSessionHandoff = issueRef ? createNativeSessionHandoffLoader({ db, companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id, before: run.createdAt, throughCommentId: readNonEmptyString(context.conversationReplayThroughCommentId) ?? (context.interactionKind ? null : wakeCommentId), @@ -230,7 +239,8 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre const handoff = await getFreshSessionHandoff(); if (handoff) context.paperclipFreshSessionHandoffMarkdown = handoff; } - return { sessionConfigMetadata, configuredModel, sessionConfigFreshness, resetTaskSession, sessionResetReason, taskSessionForRun, getFreshSessionHandoff, previousSessionParams }; + return { sessionConfigMetadata, configuredModel, sessionConfigFreshness, resetTaskSession, sessionResetReason, taskSessionForRun, + taskSessionCredentialCompatible, taskResumeSessionId, getFreshSessionHandoff, previousSessionParams }; } export interface HeartbeatAdapterSessionInput { @@ -264,7 +274,7 @@ export async function resolveHeartbeatAdapterSession(input: HeartbeatAdapterSess const { agent, evaluateSessionCompaction } = input; const { context, taskKey, issueId, continuationSummary, executionContinuation } = input.task; const { runtime, taskSession, sessionCodec, explicitResumeSessionDisplayId, runtimeSessionParams, managedAiRuntime, prepared } = input.session; - const { sessionConfigMetadata, sessionConfigFreshness, resetTaskSession, taskSessionForRun, previousSessionParams } = prepared; + const { sessionConfigMetadata, sessionConfigFreshness, resetTaskSession, taskSessionForRun, taskSessionCredentialCompatible, previousSessionParams } = prepared; const { latestWorkspaceConfigMetadata, workspaceConfigFreshness, reusedExecutionWorkspace, resolvedWorkspaceReusePolicy, persistedExecutionWorkspace, runtimeWorkspaceWarnings, requestedShouldReuseExisting, workspaceReuseRequest } = input.workspace; const runtimeSessionFallback = taskKey || resetTaskSession @@ -328,11 +338,6 @@ export async function resolveHeartbeatAdapterSession(input: HeartbeatAdapterSess delete context.paperclipPreviousSessionId; } - const taskSessionCredentialCompatible = !managedAiRuntime || managedAiSessionIdentityCompatible( - taskSession?.sessionParamsJson?.paperclipAiCredentialIdentity, - managedAiRuntime.sessionIdentity, - managedAiRuntime.identity, - ); if (managedAiRuntime) { sessionConfigMetadata.aiCredentialIdentity = managedAiRuntime.sessionIdentity; if (!taskSessionCredentialCompatible) { @@ -396,5 +401,5 @@ export async function resolveHeartbeatAdapterSession(input: HeartbeatAdapterSess }, }; - return { previousSessionDisplayId, runtimeSessionParamsForAdapter, taskSessionCredentialCompatible, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction }; + return { previousSessionDisplayId, runtimeSessionParamsForAdapter, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction }; } diff --git a/server/src/services/heartbeat/workspace-preparation.ts b/server/src/services/heartbeat/workspace-preparation.ts index 63e539e5024..6b18919a9af 100644 --- a/server/src/services/heartbeat/workspace-preparation.ts +++ b/server/src/services/heartbeat/workspace-preparation.ts @@ -104,6 +104,7 @@ const RETAINED_SANDBOX_GONE = new RegExp(`^${NATIVE_REMOTE_RECOVERY_LEASE_MISMAT /** Prepares the workspace and environment; dispatch and terminal lease release stay with the caller. */ export interface HeartbeatWorkspacePreparationInput { + signal?: AbortSignal; run: typeof heartbeatRuns.$inferSelect; agent: typeof agents.$inferSelect; task: { @@ -145,6 +146,7 @@ export interface HeartbeatWorkspacePreparationInput { configSnapshot: WorkspaceMetadataInput["configSnapshot"]; secretManifest: WorkspaceMetadataInput["secretManifest"]; managedAiCredentialIdentity?: string; + runnerTaskResumeSessionId?: string | null; }; reuse: { requestedShouldReuseExisting: boolean; @@ -154,7 +156,7 @@ export interface HeartbeatWorkspacePreparationInput { nativeRecoveryExecutionWorkspaceId: string | null; persistedNativeExecutionInput: NativeExecutionInput | null; isDotRun: boolean; - runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim }; + runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; portablePreparationLeaseId?: string }; }; services: { workspaceWorkers?: WorkspaceWorkers; @@ -253,6 +255,12 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace appendRunEvent, } = input.services; const pluginTaskWorkspace = selectedEnvironmentForConfig?.driver === "plugin" && agent.adapterType === "paperclip_runner"; + const restartRecovery = runOptions.nativeRestartRecovery; + const portablePreparationLeaseId = runOptions.portablePreparationLeaseId ?? + (restartRecovery?.kind === "bootstrap_incomplete" ? restartRecovery.portablePreparationLeaseId : undefined); + if (portablePreparationLeaseId && (!pluginTaskWorkspace || controllerOnly)) { + throw remoteRecoveryLeaseMismatch("workspace_preparation", "environment_driver"); + } for (const workspaceId of new Set([existingExecutionWorkspace?.id, reusableExistingExecutionWorkspace?.id].filter(Boolean))) { const [workspace] = await db.select({ metadata: executionWorkspacesTable.metadata }).from(executionWorkspacesTable) .where(and(eq(executionWorkspacesTable.companyId, agent.companyId), eq(executionWorkspacesTable.id, workspaceId!))); @@ -1301,7 +1309,8 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace await controllerLease.assertOwned(); const remoteRecovery = runOptions.nativeRestartRecovery?.kind === "reattach_remote_runner" ? runOptions.nativeRestartRecovery : null; - const pluginRecovery = remoteRecovery?.remote.kind === "plugin_task" ? remoteRecovery.remote : null; + const pluginRecovery = remoteRecovery?.remote.kind === "plugin_task" ? remoteRecovery.remote : portablePreparationLeaseId + ? { kind: "plugin_task" as const, leaseId: portablePreparationLeaseId, preparation: true as const } : undefined; const sandboxRecovery = remoteRecovery && remoteRecovery.remote.kind !== "plugin_task" ? remoteRecovery.remote : null; const recoveryWorkspace = sandboxRecovery ? readNativeWorkspaceSyncReference(parseObject(run.runnerProfileJson).nativeWorkspaceSync) : null; @@ -1340,10 +1349,12 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace adapterConfig: parseObject(agent.adapterConfig), admittedLifecycleMode: persistedNativeExecutionInput?.session.lifecyclePolicy.mode, managedAiCredentialIdentity: input.config.managedAiCredentialIdentity, + runnerTaskResumeSessionId: input.config.runnerTaskResumeSessionId, runnerTaskProjectIds:[...new Set([executionWorkspace.projectId,...resolvedWorkspace.runnerTaskProjectIds ?? [], ...(executionWorkspace.additionalWorkspaces ?? []).map(workspace=>workspace.projectId)] .filter((id):id is string=>Boolean(id)))], - recoverPluginTask: runOptions.nativeRestartRecovery?.kind === "resume_dead_runner" || runOptions.nativeRestartRecovery?.kind === "bootstrap_incomplete", + recoverPluginTask: !portablePreparationLeaseId && + (restartRecovery?.kind === "resume_dead_runner" || restartRecovery?.kind === "bootstrap_incomplete"), issueId: issueId ?? null, heartbeatRunId: run.id, agentId: agent.id, @@ -1441,6 +1452,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace effectiveExecutionWorkspaceMode, persistedExecutionWorkspace, runnerTaskProjectIds: resolvedWorkspace.runnerTaskProjectIds, + ...(input.signal ? { signal: input.signal } : {}), assertOwned: remoteAuthority ? assertRemotePreparationOwned : controllerLease.assertOwned, duplexObservabilityRecorder, })); diff --git a/server/src/services/legacy-controller-lease.ts b/server/src/services/legacy-controller-lease.ts index 36dcf0af0ad..407c579a792 100644 --- a/server/src/services/legacy-controller-lease.ts +++ b/server/src/services/legacy-controller-lease.ts @@ -1,5 +1,5 @@ import { notifyLiveRunRecovery } from "./live-run-recovery-notifications.js"; -import { and, eq, gt, lte, sql } from "drizzle-orm"; +import { and, eq, gt, isNull, lte, sql, type SQL } from "drizzle-orm"; import { controllerLeaseDb, heartbeatRuns, type Db } from "@paperclipai/db"; import type { LegacyControllerLeaseLossCause, LegacyControllerLeaseLossTiming } from "./run-cancellation.js"; import { randomUUID } from "node:crypto"; @@ -176,7 +176,25 @@ export async function hasLiveLegacyController(db: Db, run: Run): Promise; +export type RevokedLegacyControllerState = Pick & { portablePreparationLeaseId?: string }; + +function legacyPreparationStage() { + return and(eq(heartbeatRuns.runtimeMode, "legacy"), eq(heartbeatRuns.executionStage, "preparing"), + isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), isNull(heartbeatRuns.processStartedAt)); +} + +export async function handoffLegacyControllerPreparation(db: Db, run: Run, preparationLease: SQL, onHandoff: () => void) { + return db.transaction(async tx => { + await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id }); + const [released] = await tx.update(heartbeatRuns).set({ controllerLeaseExpiresAt: sql`clock_timestamp()` }) + .where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), + eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.controllerBootId, legacyControllerBootId), + legacyPreparationStage(), sql`${preparationLease} is not null`)).returning({ id: heartbeatRuns.id }); + if (!released) return false; + onHandoff(); + return true; + }); +} /** Atomically revoke an expired controller. Renewal and revocation serialize on * the run row. Expiry permits cleanup, never dispatch of a replacement agent. @@ -186,6 +204,7 @@ export type RevokedLegacyControllerState = Pick, ): Promise { if (run.runtimeMode === "native" || !run.controllerBootId) { return { executionStage: run.executionStage, processPid: run.processPid, processGroupId: run.processGroupId }; @@ -193,19 +212,19 @@ export async function revokeExpiredLegacyControllerWithState( const controllerBootId = run.controllerBootId; const [revoked] = await db.transaction(async tx => { await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id }); - return tx.update(heartbeatRuns).set({ - controllerBootId: randomUUID(), - controllerLeaseExpiresAt: sql`clock_timestamp() + interval '60 seconds'`, - }).where(and( - eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), - eq(heartbeatRuns.status, "running"), - eq(heartbeatRuns.controllerBootId, controllerBootId), - lte(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`), - )).returning({ - executionStage: heartbeatRuns.executionStage, - processPid: heartbeatRuns.processPid, - processGroupId: heartbeatRuns.processGroupId, - }); + const expired = and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), + eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.controllerBootId, controllerBootId), + lte(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`)); + const state = { executionStage: heartbeatRuns.executionStage, processPid: heartbeatRuns.processPid, + processGroupId: heartbeatRuns.processGroupId }; + if (reclaimPreparation) { + const [reclaimed] = await tx.update(heartbeatRuns).set(legacyControllerClaim("legacy")).where(and(expired, legacyPreparationStage(), + sql`${reclaimPreparation} is not null`)).returning({ ...state, portablePreparationLeaseId: reclaimPreparation }); + if (reclaimed?.portablePreparationLeaseId) return [{ ...reclaimed, portablePreparationLeaseId: reclaimed.portablePreparationLeaseId }]; + } + return tx.update(heartbeatRuns).set({ controllerBootId: randomUUID(), + controllerLeaseExpiresAt: sql`clock_timestamp() + interval '60 seconds'`, + }).where(expired).returning(state); }); return revoked ?? null; } diff --git a/server/src/services/native-runtime/native-controller-recovery.integration.test.ts b/server/src/services/native-runtime/native-controller-recovery.integration.test.ts index 49cac6d9fbb..e91afae1b21 100644 --- a/server/src/services/native-runtime/native-controller-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-controller-recovery.integration.test.ts @@ -509,8 +509,16 @@ describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable d await db.update(nativeRunFinalizations).set({ phase: "committed", leaseOwner: null, leaseExpiresAt: null }) .where(eq(nativeRunFinalizations.runId, fixture.scope.runId)); await expect(readSandbox()).resolves.toEqual({ remoteCwd: "/workspace" }); - await expect(inspect({ readControlState: () => ({ identity, commands: [{ status: "pending" }], - committedEvents: [{ eventType: "run.terminal", envelope: identity }] }) })).rejects.toBeInstanceOf(NativePredecessorRetirementUnprovenError); + const readUnsettledControlState = () => ({ identity, commands: [{ status: "pending" }], + committedEvents: [{ eventType: "run.terminal", envelope: identity }] }); + await expect(inspect({ readControlState: readUnsettledControlState })) + .rejects.toBeInstanceOf(NativePredecessorRetirementUnprovenError); + const movedFailure = await inspect({ + target: { ...target, sandboxLeaseAcquisition: { outcome: "replacement", providerLeaseId: "new-sandbox" } } as never, + portableSourceTaskId: "retained-sandbox", readControlState: readUnsettledControlState, + }).catch(error => error); + expect(movedFailure).not.toBeInstanceOf(NativePredecessorRetirementUnprovenError); + expect(movedFailure.message).toBe(`${denied}; warm_recovery_refused=journal_unsettled`); // A created or replacement sandbox holds no Runner to adopt. Live adoption // steps aside so ordinary admission can retire the finished owner, instead // of denying with prior_owner_terminal_state_indeterminate (Nov 10 staging). diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index d71af628d3e..450e63c531a 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -123,6 +123,7 @@ export type NativeRestartRecoveryClaim = providerAttempt: number; restartKind: NativeRestartKind; recoveryRequestId: string | null; + portablePreparationLeaseId?: string; /** False when the prior controller suspended this run in a clean drain. */ chargesProviderAttempt?: boolean; }; @@ -1546,6 +1547,12 @@ export async function claimNativeRestartRecoveries(input: { const continuedSessionOnly = hasCheckpoint && checkpointIdentityMatches && providerEvents.length === 0 && !pendingInheritedRequests && checkpointRecord.activeTurnId == null && checkpointRecord.semanticResult == null && checkpointRecord.terminal == null; + const portablePreparation = pluginTask?.unlaunchedPortable === true && row.run.status === "running" && + row.coordinator.phase === "observed" && !durableAuthority && !row.run.sessionIdAfter && + row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION && + (hasCheckpoint ? admissionCheckpointOnly && providerEvents.length === 0 : !hasProviderEvidence) && + (!row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt || await carriesInheritedRunnerIdentity(tx, row.run)) && + !(await hasNativeExecutionStartEvidence(tx, row.run)); const remoteRunnerNeverLaunched = !unclaimedBootstrap && remoteWorkspace?.state === "prepared" && !pluginTask && !durableAuthority && stagedAttachment === null && row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION @@ -1672,11 +1679,11 @@ export async function claimNativeRestartRecoveries(input: { : await restartResumeKeepsAttempt(tx, { runId: row.run.id, coordinator: row.coordinator }); const resumeKeepsAttempt = restartCredit.keepsAttempt; const classification = classifyNativeRunnerRecoveryEvidence({ - remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority), - unclaimedBootstrap, + remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority) || portablePreparation, + unclaimedBootstrap: unclaimedBootstrap || portablePreparation && row.coordinator.attempt === 0, remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched - || remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands || remoteFollowUpNeverAttached, - remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched ? "remote_runner_never_launched" + || remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands || remoteFollowUpNeverAttached || portablePreparation, + remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched || portablePreparation ? "remote_runner_never_launched" : remoteRunnerExitedBeforeProviderWork ? "remote_runner_exited_before_provider_work" : remoteRunnerNeverReceivedCommands ? "remote_runner_never_received_commands" : remoteFollowUpNeverAttached ? "remote_follow_up_never_attached" @@ -1687,13 +1694,13 @@ export async function claimNativeRestartRecoveries(input: { knownProviderProcessAlive: providerProcesses.livePids.length > 0, knownProviderProcessIdentityAmbiguous: providerProcesses.ambiguousLivePids.length > 0, - hasCheckpoint: hasCheckpoint && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly) + hasCheckpoint: hasCheckpoint && !portablePreparation && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly) && !remoteRunnerNeverLaunched && !remoteFollowUpNeverAttached && !(remoteBootstrapProvenIncomplete && reboundSessionOnly), stagedAttachmentVerified: stagedAttachment !== null, checkpointIdentityMatches: checkpointIdentityMatches && (!remoteSandbox || remoteWorkspace !== null), - hasProviderEvidence: hasProviderEvidence && !(unclaimedBootstrap && inheritedCheckpointOnly), + hasProviderEvidence: hasProviderEvidence && !portablePreparation && !(unclaimedBootstrap && inheritedCheckpointOnly), checkpointFailed: (checkpointRecord.terminal as Record | undefined) ?.runTerminalState === "failed" && @@ -1893,7 +1900,7 @@ export async function claimNativeRestartRecoveries(input: { leaseOwner, // Cleanup and earlier candidates can outlive the batch observation. // The owner row is locked: grant the full lease at this write. - leaseExpiresAt: sql`clock_timestamp() + ${durableAuthority ? REMOTE_CONTROLLER_LEASE_MS : 20 * 60_000} * interval '1 millisecond'`, + leaseExpiresAt: sql`clock_timestamp() + ${durableAuthority || portablePreparation ? REMOTE_CONTROLLER_LEASE_MS : 20 * 60_000} * interval '1 millisecond'`, controllerBootId: controller.bootId, controllerHostHash, controllerPid: controller.pid, @@ -1981,7 +1988,7 @@ export async function claimNativeRestartRecoveries(input: { await restoreNativeSessionExecution(tx as unknown as Db, { issue, run: row.run, agentName: resumptionAgent.name, now }); } - claimScope = { companyId: row.run.companyId, issueId: row.coordinator.issueId, controller, remote: Boolean(durableAuthority) }; + claimScope = { companyId: row.run.companyId, issueId: row.coordinator.issueId, controller, remote: Boolean(durableAuthority || portablePreparation) }; const common = { runId: row.run.id, leaseOwner, @@ -2021,6 +2028,7 @@ export async function claimNativeRestartRecoveries(input: { return { kind: claimKind, ...common, + ...(claimKind === "bootstrap_incomplete" && portablePreparation && pluginTask ? { portablePreparationLeaseId: pluginTask.id } : {}), } satisfies NativeRestartRecoveryClaim; }).catch((error: unknown) => { // The whole candidate transaction rolled back, so this run's state is diff --git a/server/src/services/native-runtime/native-runner-ownership.ts b/server/src/services/native-runtime/native-runner-ownership.ts index 78a191b3026..9e4852dc9ec 100644 --- a/server/src/services/native-runtime/native-runner-ownership.ts +++ b/server/src/services/native-runtime/native-runner-ownership.ts @@ -3,6 +3,7 @@ import { environmentLeases, heartbeatRuns, nativeRunFinalizations, type Db } fro import { and, desc, eq, inArray, isNull, or, sql } from "drizzle-orm"; import { remoteExecutionHasStopped } from "../remote-execution-termination.js"; import { readSystemRecovery } from "../system-recovery-state.js"; +import { unlaunchedPortableTaskLease } from "../plugin-task-ownership.js"; type NativeRemoteExecutionScope = { run: Pick; @@ -13,6 +14,7 @@ export async function readNativeRemoteExecutionOwnershipForRuns(db: Db, inputs: const key = (companyId: string, issueId: string, runId: string) => `${companyId}:${issueId}:${runId}`; const leases = inputs.length ? await db.select({ id: environmentLeases.id, companyId: environmentLeases.companyId, issueId: environmentLeases.issueId, runId: environmentLeases.heartbeatRunId, + unlaunchedPortable: unlaunchedPortableTaskLease(), metadata: sql>`jsonb_build_object('runnerTaskStartup', ${environmentLeases.metadata}->'runnerTaskStartup')`, }).from(environmentLeases).where(and( or(...inputs.map(input => and(eq(environmentLeases.companyId, input.run.companyId), diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index c5a7a79eb32..06c2ec0115a 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -9,8 +9,10 @@ import * as stagedAttachmentRecovery from "./native-staged-attachment-recovery.j import * as controllerRecovery from "./native-controller-recovery.js"; import { isSettledRemoteWarmControlState } from "./remote-warm-session-recovery.js"; import * as remoteWarmRecoveryRuntime from "./remote-warm-session-recovery.js"; +import * as pluginTaskOwnership from "../plugin-task-ownership.js"; import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js"; import * as environmentTaskRuntime from "../environment-task-runtime.js"; +import * as pluginCredentialWriteback from "../plugin-task-credential-writeback.js"; import * as pluginTaskRuntime from "./plugin-runner-task.js"; import * as completionFeedback from "./native-completion-feedback.js"; import { environmentTaskOperationSchema } from "@paperclipai/plugin-sdk"; @@ -19,7 +21,7 @@ import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@open import { configuredEnvironment, configuredEnvironmentProjection } from "../../vendor/paperclip-runner/index.js"; import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from "vitest"; import { createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "../system-recovery-state.js"; -import { readFileSync } from "node:fs"; +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; import { once } from "node:events"; import { PgDialect } from "drizzle-orm/pg-core"; import type { SQL } from "drizzle-orm"; @@ -14176,15 +14178,20 @@ describe("native warm session supervision", () => { { name: "rotated provider key", firstEnvironment: { OPENAI_API_KEY: "fixture-key" }, secondEnvironment: { OPENAI_API_KEY: "fixture-rotated-key" }, reuse: false }, { name: "rotated Claude token", firstEnvironment: { CLAUDE_CODE_OAUTH_TOKEN: "fixture-token" }, secondEnvironment: { CLAUDE_CODE_OAUTH_TOKEN: "fixture-rotated-token" }, reuse: false }, { name: "tool refresh", refreshTools: true, reuse: false }, + { name: "unsupported tool refresh", refreshTools: true, unsupportedRefresh: true, reuse: false }, { name: "network access change", firstEnvironment: { PAPERCLIP_RUNNER_NETWORK_ACCESS: "disabled" }, secondEnvironment: { PAPERCLIP_RUNNER_NETWORK_ACCESS: "enabled" }, reuse: false }, { name: "run-scoped environment broker", firstEnvironment: { PAPERCLIP_GITHUB_BROKER_TOKEN: "first-run-capability" }, secondEnvironment: { PAPERCLIP_GITHUB_BROKER_TOKEN: "second-run-capability" }, reuse: false }, { name: "Core-hosted GitHub access", managedGitHub: true, reuse: true, adoptionReuse: false }, ] as Array<{ name: string; firstEnvironment?: Record; secondEnvironment?: Record; - refreshTools?: boolean; managedGitHub?: boolean; reuse: boolean; adoptionReuse?: boolean }>)( + refreshTools?: boolean; unsupportedRefresh?: boolean; managedGitHub?: boolean; reuse: boolean; adoptionReuse?: boolean }>)( "decides warm and adopted process reuse from one key: $name", - async ({ name, firstEnvironment = {}, secondEnvironment = {}, refreshTools, managedGitHub = false, reuse, adoptionReuse = reuse }) => { + async ({ name, firstEnvironment = {}, secondEnvironment = {}, refreshTools, unsupportedRefresh = false, + managedGitHub = false, reuse, adoptionReuse = reuse }) => { const decisions: Record = {}; - for (const path of ["warm", "adoption"] as const) { + for (const path of ["warm", "adoption", ...(refreshTools ? ["portable", "portable-bootstrap", "legacy-plugin"] as const : [])] as const) { + const portable = path === "portable" || path === "portable-bootstrap"; + const bootstrap = path === "portable-bootstrap"; + const pluginRecovery = portable || path === "legacy-plugin"; githubAccess.create.mockReset().mockImplementation(async () => ({ env: {}, ready: true, duplexRequested: false, activate: githubAccess.activate.mockReset().mockImplementation(() => vi.fn()), stop: githubAccess.stop.mockReset().mockResolvedValue(undefined) })); const stateBase = await mkdtemp(join(tmpdir(), "paperclip-runnerd-reuse-key-")); @@ -14195,15 +14202,24 @@ describe("native warm session supervision", () => { const scope = `${path}-${name.replaceAll(" ", "-")}`; const first = { ...execution, + ...(pluginRecovery ? { schema: "paperclip.native-execution-input.v5", executionMode: "default", + provider: { ...execution.provider, approvalPolicy: "never" }, planningContext: null, runtimeContext: nativeRuntimeContextFixture() } as const : {}), binding: { ...execution.binding, runId: `run-reuse-key-first-${scope}`, executionWorkspaceId: "workspace-reuse-key" }, workspace: { cwd: "/tmp/runnerd-reuse-key", repoUrl: null, repoRef: null, branchName: null }, session: { normalizedSessionId: `session-reuse-key-${scope}`, driverKind: "codex_app_server" as const, protocolVersion: 1 as const, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } }, - } as NativeExecutionInputV1; - const second = { ...first, binding: { ...first.binding, runId: `run-reuse-key-second-${scope}` } } as NativeExecutionInputV1; + } as NativeExecutionInput; + if (pluginRecovery) parseNativeExecutionInput(first); + const second = { ...first, binding: { ...first.binding, runId: `run-reuse-key-second-${scope}` } }; + if ("runtimeContext" in second) { + second.runtimeContext = structuredClone(second.runtimeContext); + second.runtimeContext.mcp.digest = "1".repeat(64); + second.runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(second.runtimeContext); + } const target = { kind: "remote" as const, transport: "sandbox" as const, environmentId: "environment-reuse-key", - remoteCwd: "/home/daytona/paperclip-workspace", runner: { execute: vi.fn() } } as never; + remoteCwd: "/home/daytona/paperclip-workspace", runner: { execute: vi.fn() } }; const firstSession = { close: vi.fn(async () => undefined) }; + const recoveredSession = { close: vi.fn(async () => undefined), snapshot: vi.fn() }; const result = { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" }, turnId: "turn", normalizedSessionId: first.session.normalizedSessionId, providerSessionId: "provider-reuse-key", driverKind: "test", driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null }; @@ -14219,36 +14235,138 @@ describe("native warm session supervision", () => { return result; }) .mockImplementationOnce(async (options) => { - decisions[path] = options.existingSession === firstSession; + decisions[path] = options.existingSession === (portable ? recoveredSession : firstSession); return result; }); const inspected = vi.spyOn(remoteWarmRecoveryRuntime, "inspectRemoteWarmSessionRecovery").mockImplementation(async request => { - decisions[path] = request.canReuseProcess(first); + const reuseProcess = request.canReuseProcess(first); + if (pluginRecovery) { + expect(reuseProcess).toBe(false); + expect(request.matchesExecution(first)).toBe(!unsupportedRefresh); + if (unsupportedRefresh) throw new Error("fixture unsupported tool refresh refused"); + expect(request.matchesExecution({ ...first, provider: { ...first.provider, model: "another-model" } } as NativeExecutionInput)).toBe(false); + if ("runtimeContext" in first) { + const changed = structuredClone(first); + changed.runtimeContext.instructions.bundle.digest = "2".repeat(64); + changed.runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(changed.runtimeContext); + expect(request.matchesExecution(changed)).toBe(false); + } + const checkpoint = request.loadCheckpoint(first); + expect(checkpoint?.identity.runId).toBe(first.binding.runId); + recoveredSession.snapshot.mockResolvedValue(checkpoint); + return { identity: request.identity as Record, checkpoint: checkpoint!, + previousExecution: first, reuseProcess, ...(portable ? { portableSourceTaskId: "source-task" } : {}), + remote: { providerLeaseId: "target-task", remoteCwd: target.remoteCwd }, revalidate: async () => {} }; + } + decisions[path] = reuseProcess; throw new Error("fixture adoption decision recorded"); }); + const ownership = portable + ? vi.spyOn(pluginTaskOwnership, "readyPortableTaskSource").mockImplementation(async (_db, _company, _lease, identity, scope) => { + expect(scope?.accountIdentity).toBe("fixture-account"); + return { taskId: "source-task", identity: identity as NonNullable>>["identity"] }; + }) : undefined; + const projection = vi.spyOn(controllerRecovery, "resolveNativeControllerRecovery").mockResolvedValue({ kind: "no_prior_execution" }); + const restore = path === "legacy-plugin" ? vi.spyOn(runnerCheckpoint, "restoreNativeRunnerCheckpoint") + .mockImplementation(async (_db, binding) => binding.runId === first.binding.runId && recoveredSession.close.mock.calls.length ? {} as never : null) : undefined; + const issueProjection = path === "legacy-plugin" ? vi.spyOn(issueServiceModule, "issueService") + .mockReturnValue({ update: vi.fn(async () => null) } as never) : undefined; try { await executePaperclipNativeSession({ db: leaseDb(first), execution: first, runnerInstanceId: "runner-reuse-key", - useRunnerd: true, managedGitHub, runnerExecutionTarget: target, + useRunnerd: true, managedGitHub, runnerExecutionTarget: target as never, + ...(portable ? { managedAiAccountIdentity: "fixture-account", managedAiCredentialIdentity: "old-revision" } : {}), runnerEnvironment: { PAPERCLIP_GITHUB_AUTH_MODE: "managed", ...firstEnvironment } }); const durableRoot = (await readdir(stateBase, { withFileTypes: true })) .filter(entry => entry.isDirectory() && /^[a-f0-9]{64}$/.test(entry.name)).map(entry => join(stateBase, entry.name))[0]!; const durableIdentity = { runId: first.binding.runId, normalizedSessionId: first.session.normalizedSessionId, - runnerInstanceId: "runner-reuse-key", environmentLeaseId: first.binding.executionWorkspaceId }; + runnerInstanceId: "runner-reuse-key", environmentLeaseId: first.binding.executionWorkspaceId, + turnId: "prior-turn", itemId: "prior-item" }; await mkdir(join(durableRoot, "control-plane"), { recursive: true }); await writeFile(join(durableRoot, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(durableIdentity))); await mkdir(join(durableRoot, "runner"), { recursive: true }); await writeFile(join(durableRoot, "runner", "runner-state.json"), JSON.stringify(durableRunnerState(durableIdentity, "ready"))); - if (path === "adoption") expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] }); + if (path !== "warm") expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] }); + if (portable) { + const directory = join(resolvePaperclipInstanceRoot(), "runtime/paperclip-runner/sessions"); + const path = join(directory, (await readdir(directory))[0]!); + const { snapshot, configDigest, processConfigDigest } = JSON.parse(await readFile(path, "utf8")); + await rm(path); + await rm(durableRoot, { recursive: true }); + projection.mockResolvedValueOnce({ + kind: "recoverable", snapshot, + metadata: controllerRecovery.createNativeControllerRecovery(snapshot, { configDigest, processConfigDigest }, null), + checkpoint: { restoreSessionDirectory(root: string) { + mkdirSync(join(root, "control-plane"), { recursive: true }); + writeFileSync(join(root, "control-plane/control-plane-state.json"), JSON.stringify(durableControlPlaneState(durableIdentity))); + } } as never, + }); + } + if (pluginRecovery) state.createBackend.mockReturnValueOnce({ + kind: "test", recoverSession: async () => ({ recovered: true, session: recoveredSession }), + } as never); + const pluginTaskDescription = { kind: "description", taskId: "target-task", warmTaskVersion: 1, + launchPreparationVersion: 1, projectAuthCheckVersion: 1, releaseResumeVersion: 1, + runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } } as const; + const portableTarget = { ...target, runner: undefined, leaseId: "target-lease", + sandboxLeaseAcquisition: { outcome: path === "legacy-plugin" ? "resumed" : "created", providerLeaseId: "target-task" }, + runnerTask: vi.fn(async (operation: { kind: string }) => { + if (path === "legacy-plugin") { + expect(recoveredSession.close).toHaveBeenCalledOnce(); + expect(operation.kind).toBe("describe"); + throw new Error("fixture replacement inspection reached"); + } + return pluginTaskDescription; + }), + }; + if (pluginRecovery) await writeFile(join(stateBase, "auth.json"), + JSON.stringify({ OPENAI_API_KEY: "synthetic-fixture" }), { mode: 0o600 }); + const leaseOwner = bootstrap ? "bootstrap-recovery-owner" : undefined; + const controller = bootstrap ? await import("./native-restart-recovery.js").then(module => module.currentNativeControllerIdentity()) : undefined; + const db = leaseDb(second, controller ? { leaseOwner, controllerBootId: controller.bootId, + controllerPid: controller.pid, controllerGeneration: 1, leaseExpiresAt: new Date(Date.now() + 60_000) } : {}); + const select = db.select.bind(db); + db.select = ((fields?: Record) => { + if (fields?.status === heartbeatRuns.status && fields?.runnerProfileJson === heartbeatRuns.runnerProfileJson) { + return { from: () => ({ where: () => ({ limit: async () => + [{ status: "succeeded", runnerProfileJson: { nativeExecutionInput: first } }] }) }) }; + } + if (pluginRecovery && fields?.metadata === environmentLeases.metadata) { + return { from: () => ({ where: async () => [{ providerLeaseId: pluginTaskDescription.taskId, unlaunchedPortable: bootstrap, + metadata: ownership ? { portableTaskPreviousLeaseId: "source-lease", portableTaskOwnershipReady: true } : {} }] }) }; + } + return select(fields as never); + }) as Db["select"]; const continuation = executePaperclipNativeSession({ - db: { ...leaseDb(second), select: () => ({ from: () => ({ where: () => ({ limit: () => - Promise.resolve([{ status: "succeeded", runnerProfileJson: { nativeExecutionInput: first } }]) }) }) }) } as unknown as Db, - execution: second, runnerInstanceId: "runner-reuse-key", useRunnerd: true, managedGitHub, refreshTools, - runnerExecutionTarget: target, runnerEnvironment: { PAPERCLIP_GITHUB_AUTH_MODE: "managed", ...secondEnvironment } }); + db, execution: second, runnerInstanceId: "runner-reuse-key", useRunnerd: true, managedGitHub, refreshTools, toolRefreshOnResume: !unsupportedRefresh, + runnerExecutionTarget: (pluginRecovery ? portableTarget : target) as never, + ...(leaseOwner ? { leaseOwner, restartRecovery: { kind: "bootstrap_incomplete" as const, runId: second.binding.runId, + leaseOwner, controllerGeneration: 1, providerAttempt: 0, restartKind: "hard" as const, recoveryRequestId: null, + portablePreparationLeaseId: portableTarget.leaseId } } : {}), + ...(pluginRecovery ? { pluginTaskDescription, runnerIngressAuthorized: true, managedAiCredentialHome: stateBase, + managedAiAccountIdentity: "fixture-account", managedAiCredentialIdentity: "current-revision" } : {}), + runnerEnvironment: { PAPERCLIP_GITHUB_AUTH_MODE: "managed", ...secondEnvironment } }); + if (pluginRecovery && unsupportedRefresh) { + await expect(continuation).rejects.toThrow("fixture unsupported tool refresh refused"); + continue; + } + if (path === "legacy-plugin") { + await expect(continuation).rejects.toThrow("fixture replacement inspection reached"); + expect(portableTarget.runnerTask).toHaveBeenCalledOnce(); + continue; + } if (path === "adoption") await expect(continuation).rejects.toThrow("fixture adoption decision recorded"); else await continuation; - expect(decisions[path]).toBe(path === "adoption" ? adoptionReuse : reuse); + expect(decisions[path]).toBe(portable ? true : path === "adoption" ? adoptionReuse : reuse); + if (portable) { + expect(portableTarget.runnerTask).not.toHaveBeenCalled(); + expect(recoveredSession.close).not.toHaveBeenCalled(); + } if (path === "warm") expect(firstSession.close).toHaveBeenCalledTimes(reuse ? 0 : 1); } finally { + projection.mockRestore(); + restore?.mockRestore(); + issueProjection?.mockRestore(); + ownership?.mockRestore(); inspected.mockRestore(); await closeIdleWarmNativeSessionsForRestart(); if (previousStateDirectory === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; @@ -14258,7 +14376,7 @@ describe("native warm session supervision", () => { await rm(stateBase, { recursive: true, force: true }); } } - expect(decisions).toEqual({ warm: reuse, adoption: adoptionReuse }); + expect(decisions).toEqual({ warm: reuse, adoption: adoptionReuse, ...(refreshTools && !unsupportedRefresh ? { portable: true, "portable-bootstrap": true } : {}) }); }, ); @@ -17963,8 +18081,9 @@ describe("runnerd provider runtime wiring", () => { } finally { resume.mockRestore(); } }); - it.each(["verified", "wrong-state", "wrong-lease"] as const)("reattaches a surviving plugin task through its owner: %s", async mode => { + it.each(["verified", "portable", "wrong-claim", "wrong-state", "wrong-lease"] as const)("reattaches a surviving plugin task through its owner: %s", async mode => { const current = scopedNativeExecution(`plugin-live-${mode}`); + const moved = mode === "portable" || mode === "wrong-claim"; const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!, runnerInstanceId: "plugin-runner", environmentLeaseId: "same-task-lease", turnId: "admitted-turn", itemId: "admitted-item" }; const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId); @@ -17980,24 +18099,35 @@ describe("runnerd provider runtime wiring", () => { runner: { version: "new-artifact", digest: `sha256:${"b".repeat(64)}` } }; }); state.createBackend.mockClear(); state.createTransport.mockClear(); - const backend = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId, - runnerIngressAuthorized: true, runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", - leaseId: mode === "wrong-lease" ? "replacement-task" : identity.environmentLeaseId, runnerTask: task }, - controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-artifact", digest: `sha256:${"a".repeat(64)}` } }, - restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller", controllerGeneration: 2, - providerAttempt: 1, restartKind: "hard", recoveryRequestId: null, remote: { kind: "plugin_task", leaseId: identity.environmentLeaseId } } }); - if (mode !== "verified") { - await expect(backend).rejects.toThrow(mode === "wrong-state" ? "adoption identity mismatch" : "recovery_lease_mismatch"); - expect(state.createBackend).not.toHaveBeenCalled(); - return; - } - await backend; - state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); - const options = state.createTransport.mock.calls.at(-1)![0]; - expect(options).toMatchObject({ prpIdentity: identity, adoptExistingRunner: { pid: null, startedAt: null }, - runnerArtifact: { version: "admitted-artifact" } }); - expect(options).toMatchObject({ runnerReconnectGraceMs: 420_000 }); - expect(task.mock.calls.some(([op]) => ["prepare", "submit", "stop"].includes(op.kind))).toBe(false); + const portable = moved ? vi.spyOn(pluginTaskOwnership, "assertAttachedPortableTask") + .mockImplementation(async () => { + if (mode === "wrong-claim") throw new Error("native_portable_task_ownership_unverified"); + }) : undefined; + const leaseId = moved ? "moved-task-lease" : identity.environmentLeaseId; + try { + const backend = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId, + runnerIngressAuthorized: true, runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", + leaseId: mode === "wrong-lease" ? "replacement-task" : leaseId, runnerTask: task, + ...(moved ? { sandboxLeaseAcquisition: { outcome: "resumed" as const, providerLeaseId: "same-task" } } : {}) }, + controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-artifact", digest: `sha256:${"a".repeat(64)}` } }, + restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller", controllerGeneration: 2, + providerAttempt: 1, restartKind: "hard", recoveryRequestId: null, remote: { kind: "plugin_task", leaseId } } }); + if (mode !== "verified" && mode !== "portable") { + await expect(backend).rejects.toThrow(mode === "wrong-state" ? "adoption identity mismatch" : mode === "wrong-claim" ? "ownership_unverified" : "recovery_lease_mismatch"); + expect(state.createBackend).not.toHaveBeenCalled(); + return; + } + await backend; + if (moved) expect(portable).toHaveBeenCalledWith(expect.anything(), { + companyId: current.binding.companyId, leaseId, taskId: "same-task", runId: current.binding.runId, identity, + }); + state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); + const options = state.createTransport.mock.calls.at(-1)![0]; + expect(options).toMatchObject({ prpIdentity: identity, adoptExistingRunner: { pid: null, startedAt: null }, + runnerArtifact: { version: "admitted-artifact" } }); + expect(options).toMatchObject({ runnerReconnectGraceMs: 420_000 }); + expect(task.mock.calls.some(([op]) => ["prepare", "submit", "stop"].includes(op.kind))).toBe(false); + } finally { portable?.mockRestore(); } }); it.each(["codex", "acpx-codex", "grok"] as const)("writes managed %s events through the current warm exporter without per-turn reads", async provider => { @@ -18175,8 +18305,11 @@ describe("runnerd provider runtime wiring", () => { } finally {owner.mockRestore();await rm(home,{recursive:true,force:true});} }); - it.each((["codex", "claude"] as const).flatMap(provider => [true, false].map(reuseProcess => [provider, reuseProcess] as const)))("reconnects task-only %s warm recovery under the saved identity before reuse or retirement: %s", async (provider, reuseProcess) => { - const current=scopedNativeExecution(`plugin-restart-${provider}-${reuseProcess}`); + it.each([ + ...(["codex", "claude"] as const).flatMap(provider => [true, false].map(reuseProcess => [provider, reuseProcess, false] as const)), + ["codex", true, true] as const, + ])("reconnects task-only %s warm recovery under the saved identity before reuse or retirement: %s (portable=%s)", async (provider, reuseProcess, portable) => { + const current=scopedNativeExecution(`plugin-restart-${provider}-${reuseProcess}-${portable}`); if(provider === "claude") current.provider={kind:"acpx",agent:"claude",model:"fixture",permissionMode:"deny-all"} as never; current.session.lifecyclePolicy={mode:"warm",idleTimeoutMs:300000}; const identity={runnerInstanceId:"old-runner",environmentLeaseId:"old-lease",runId:"old-run", @@ -18184,22 +18317,29 @@ describe("runnerd provider runtime wiring", () => { const home=await mkdtemp(join(tmpdir(),"plugin-restart-account-")); await writeFile(join(home,"auth.json"),JSON.stringify({OPENAI_API_KEY:"synthetic-fixture"}),{mode:0o600}); const owner=vi.spyOn(pluginTaskRuntime,"createPluginRunnerTaskOwner"); + const register = vi.spyOn(pluginCredentialWriteback, "registerPluginTaskCredentialHandoff").mockResolvedValue(undefined); + const binding = { grantId: randomUUID(), secretId: randomUUID(), sessionEpoch: 1, versionSelector: "latest" as const }; const task=vi.fn(async()=>({kind:"description",taskId:"saved-task",launchPreparationVersion:1,projectAuthCheckVersion:1, warmTaskVersion:1,runner:{version:"fixture",digest:`sha256:${"a".repeat(64)}`}})); state.createBackend.mockClear();state.createTransport.mockClear(); try { await createRunnerdBackend({db:leaseDb(current),execution:current,runnerInstanceId:"new-runner",runnerIngressAuthorized:true, runnerEnvironment:{ANTHROPIC_API_KEY:"synthetic-fixture"},managedAiCredentialHome:home,managedAiCredentialIdentity:reuseProcess ? "old-revision" : "new-revision",managedAiAccountIdentity:"account", - managedAiCredentialWriteback:{} as never, + managedAiCredentialWriteback: binding, remoteWarmRecovery:{identity,reuseProcess,previousExecution:current,checkpoint:{} as never, - remote:{providerLeaseId:"saved-task",remoteCwd:"."},revalidate:async()=>{}}, + remote:{providerLeaseId:"saved-task",remoteCwd:"."},revalidate:async()=>{}, + ...(portable ? { portableSourceTaskId: "old-task" } : {})}, runnerExecutionTarget:{kind:"remote",transport:"sandbox",leaseId:"new-lease",remoteCwd:".",runnerTask:task, sandboxLeaseAcquisition:{outcome:"resumed",providerLeaseId:"saved-task"}}}); state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); expect(state.createTransport.mock.calls.at(-1)![0].prpIdentity).toEqual(identity); expect(owner.mock.calls.at(-1)![0].adoptIdentity).toEqual(identity); expect(task.mock.calls).toEqual([[{kind:"describe"}]]); - } finally {owner.mockRestore();await rm(home,{recursive:true,force:true});} + if (portable) expect(register).toHaveBeenCalledExactlyOnceWith(expect.anything(), { + companyId: current.binding.companyId, leaseId: "new-lease", + }, "account", binding); + else expect(register).not.toHaveBeenCalled(); + } finally {register.mockRestore(); owner.mockRestore();await rm(home,{recursive:true,force:true});} }); it.each((["codex", "claude"] as const).flatMap(provider => [ diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index fd2e9d55671..db4c578406e 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -54,7 +54,8 @@ import { writeExecutionControl } from "../execution-control-notifications.js"; import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { createRunUsageRecorder } from "../usage-receipts.js"; import { syncPluginTaskCredentialHandoff, registerPluginTaskCredentialHandoff } from "../plugin-task-credential-writeback.js"; -import { resolvePluginTaskResume } from "../environment-task-runtime.js"; +import { resolvePluginTaskResume, type PluginTaskDescription } from "../environment-task-runtime.js"; +import { assertAttachedPortableTask, readyPortableTaskSource, unlaunchedPortableTaskLease } from "../plugin-task-ownership.js"; import { environmentTaskOperationSchema, environmentTaskResultSchema } from "@paperclipai/plugin-sdk"; import { preparePluginRunnerCredentials } from "./plugin-runner-credentials.js"; import { createPluginRunnerTaskOwner, verifyPluginRunnerTaskReattachment } from "./plugin-runner-task.js"; @@ -278,7 +279,7 @@ import { renderNativeRunnerStagedAttachmentPrompt, stageNativeRunnerWakeAttachments, } from "./native-runner-file-handoff.js"; -import { NATIVE_PROVIDER_SIDE_EFFECT_PENDING, NATIVE_PROVIDER_STATE_INCOMPATIBLE, nativeRuntimeContractForProvider, nativeToolContractFingerprintForTarget } from "./native-session-resume.js"; +import { NATIVE_PROVIDER_SIDE_EFFECT_PENDING, NATIVE_PROVIDER_STATE_INCOMPATIBLE, nativeRuntimeContextsMatch, nativeRuntimeContractForProvider, nativeToolContractFingerprintForTarget } from "./native-session-resume.js"; import { clearRunProviderStateUnsaved, markAcceptedRunProviderStateUnsaved } from "./native-provider-state-unsaved.js"; import { verifyRetainedMaintenanceNoLaunch } from "./native-maintenance-no-launch.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; @@ -6247,6 +6248,29 @@ async function recordPriorOwnerFenceRefusal(input: { db: Db; execution: NativeEx }).catch(() => undefined); } +async function assertPluginTaskReattachmentBinding(input: { + db: Db; + execution: NativeExecutionInput; + restartRecovery?: NativeRestartRecoveryClaim; +}, target: Extract, identity: Record | null | undefined) { + const claim = input.restartRecovery; + const leaseId = target.leaseId; + if (claim?.kind !== "reattach_remote_runner" || claim.remote.kind !== "plugin_task" || + claim.runId !== input.execution.binding.runId || !leaseId || claim.remote.leaseId !== leaseId || + !identity || !durableIdentityMatchesExecution(identity, input.execution) || + typeof identity.turnId !== "string" || !identity.turnId || typeof identity.itemId !== "string" || !identity.itemId) { + throw new Error("native_plugin_recovery_lease_mismatch"); + } + const taskId = target.transport === "sandbox" ? target.sandboxLeaseAcquisition?.providerLeaseId : undefined; + if (identity.environmentLeaseId !== leaseId) { + if (!taskId) throw new Error("native_plugin_recovery_lease_mismatch"); + await assertAttachedPortableTask(input.db, { + companyId: input.execution.binding.companyId, leaseId, taskId, runId: claim.runId, identity, + }); + } + return taskId; +} + async function migrateRunnerdStateRootForExecution(input: { db: Db; execution: NativeExecutionInput; @@ -6370,12 +6394,12 @@ async function migrateRunnerdStateRootForExecution(input: { if (input.restartRecovery?.kind === "reattach_remote_runner") { if (input.restartRecovery.remote.kind === "plugin_task") { const target = input.runnerExecutionTarget; - if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runnerTask - || target.leaseId !== input.restartRecovery.remote.leaseId || identity.environmentLeaseId !== target.leaseId - || !durableIdentityMatchesExecution(identity, input.execution)) { + if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runnerTask) { throw new Error("native_plugin_recovery_lease_mismatch"); } - await verifyPluginRunnerTaskReattachment(target.runnerTask, identity); + const taskId = await assertPluginTaskReattachmentBinding(input, target, identity); + await verifyPluginRunnerTaskReattachment(operation => target.runnerTask!(operation, + taskId ? { expectedTaskId: taskId } : undefined), identity); return; } try { @@ -7281,6 +7305,19 @@ function nativeSessionConfigDigest( .digest("hex")}`; } +function matchesRemoteWarmExecution( + previous: NativeExecutionInput, + current: NativeExecutionInput, + toolRefreshOnResume?: boolean, +): boolean { + if (nativeSessionScopeKey(previous) !== nativeSessionScopeKey(current)) return false; + const currentDigest = nativeSessionConfigDigest(current, "remote"); + if (nativeSessionConfigDigest(previous, "remote") === currentDigest) return true; + return toolRefreshOnResume === true && "runtimeContext" in previous && "runtimeContext" in current && + nativeRuntimeContextsMatch(previous.runtimeContext, current.runtimeContext, toolRefreshOnResume) && + nativeSessionConfigDigest({ ...previous, runtimeContext: current.runtimeContext }, "remote") === currentDigest; +} + export function nativeControllerMigrationDescriptor(execution: NativeExecutionInput, recordedToolContractFingerprint: unknown) { if (typeof recordedToolContractFingerprint !== "string" || !/^sha256:[a-f0-9]{64}$/.test(recordedToolContractFingerprint)) { throw new Error("native_controller_migration_tool_contract_missing"); @@ -10203,6 +10240,7 @@ export async function executePaperclipNativeSession(input: { getFreshSessionHandoff?: () => Promise; /** Retire a retained transport so recovery can replace provider tool declarations. */ refreshTools?: boolean; + toolRefreshOnResume?: boolean; db: Db; execution: NativeExecutionInput; /** Private server preparation proof; never persisted or accepted from a tool. */ @@ -10259,6 +10297,7 @@ export async function executePaperclipNativeSession(input: { managedAiCredentialWriteback?: ManagedAiCredentialWritebackBinding; billingIdentity?: Pick; onTaskDispatched?: () => Promise | void; + pluginTaskDescription?: PluginTaskDescription; runnerExecutionTarget?: AdapterExecutionTarget | null; /** Resolved per-run authorization; not an independent instance setting. */ runnerIngressAuthorized?: boolean; @@ -10657,7 +10696,22 @@ async function executePaperclipNativeSessionAttempt( const stagedRestart = input.restartRecovery?.kind === "reattach_remote_runner" ? await verifyStagedAttachmentRestart(input) : null; const timeController = createRunPreparationTiming(input.execution.binding.runId, "native.controller"); - if (input.useRunnerd) await timeController("warm_restore", () => restoreWarmRunnerdController(input)); + const target = input.runnerExecutionTarget; + const portableTarget = target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask ? target : undefined; + const companyId = input.execution.binding.companyId; + const portablePreparationLeaseId = input.restartRecovery?.kind === "bootstrap_incomplete" + ? input.restartRecovery.portablePreparationLeaseId : undefined; + const targetLease = portableTarget?.leaseId ? (await input.db.select({ + metadata: environmentLeases.metadata, providerLeaseId: environmentLeases.providerLeaseId, + ...(portablePreparationLeaseId ? { unlaunchedPortable: unlaunchedPortableTaskLease() } : {}), + }).from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId), eq(environmentLeases.id, portableTarget.leaseId))))[0] : undefined; + if (portablePreparationLeaseId && (portableTarget?.leaseId !== portablePreparationLeaseId || targetLease?.unlaunchedPortable !== true)) { + throw remoteRecoveryLeaseMismatch("runner_verification", "plugin_task"); + } + const replayPortablePreparation = portablePreparationLeaseId !== undefined; + const canRecoverWarmOwner = !input.restartRecovery || replayPortablePreparation; + if (input.useRunnerd) await timeController("warm_restore", () => restoreWarmRunnerdController( + replayPortablePreparation ? { ...input, restartRecovery: undefined } : input)); const identityRedactor = createAgentIdentityRedactor(input.runnerEnvironment?.PAPERCLIP_AGENT_PRIVATE_KEY); const redactIdentityText = (text: string) => redactSensitiveText( identityRedactor.chunk("diagnostic", text) + identityRedactor.finish("diagnostic"), @@ -10757,29 +10811,51 @@ async function executePaperclipNativeSessionAttempt( }); let retainedTransition: VerifiedWarmTransitionBinding | undefined; let remoteWarmRecovery: RemoteWarmSessionRecovery | undefined; - let pluginTaskDescription: Extract,{kind:"description"}> | undefined; + let pluginTaskDescription: PluginTaskDescription | undefined; + let portableSource: Awaited>; + if (portableTarget?.runnerTask && portableTarget.leaseId) { + if (input.pluginTaskDescription && input.pluginTaskDescription.taskId === targetLease?.providerLeaseId) { + pluginTaskDescription = input.pluginTaskDescription; + } + if (targetLease?.metadata?.portableTaskPreviousLeaseId || targetLease?.metadata?.portableTaskPredecessor) { + const description = targetLease.metadata.portableTaskOwnershipReady === true && pluginTaskDescription + ? pluginTaskDescription : environmentTaskResultSchema.parse(await portableTarget.runnerTask({ kind: "describe" })); + if (description.kind !== "description") throw new Error("Plugin Runner description unavailable"); + pluginTaskDescription = description; + portableSource = await readyPortableTaskSource(input.db, companyId, portableTarget.leaseId, + readRunnerdDurableIdentity(scopedRunnerdStateRoot(input.execution)), { + attachedRunId: input.execution.binding.runId, + selectedSessionId: nativeSessionKey(input.execution), + accountIdentity: input.managedAiAccountIdentity ?? input.managedAiCredentialIdentity, + }); + } + } const cachedReleasedOwner = warmNativeSessions.get(nativeSessionScopeKey(input.execution)); if (cachedReleasedOwner?.released) { - const proof = releasedWarmRunner(readWarmNativeEnvelope(input.execution)); + const proof = portableSource ? undefined : releasedWarmRunner(readWarmNativeEnvelope(input.execution)); const identity = readRunnerdDurableIdentity(scopedRunnerdStateRoot(input.execution)); const target = input.runnerExecutionTarget; - if (proof && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId - && target.sandboxLeaseAcquisition?.outcome === "created" && proof.taskId !== target.sandboxLeaseAcquisition.providerLeaseId) { + const source = portableSource ?? (proof && { taskId: proof.taskId, identity: proof.source }); + if (source && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId + && (portableSource || target.sandboxLeaseAcquisition?.outcome === "created") + && source.taskId !== target.sandboxLeaseAcquisition?.providerLeaseId) { const ownInstructionPreparation = cachedReleasedOwner.instructionPreparationRunId === input.execution.binding.runId && input.instructionWorkingCopy?.runId === input.execution.binding.runId; if ((cachedReleasedOwner.busy && !ownInstructionPreparation) || cachedReleasedOwner.preparingRunId && cachedReleasedOwner.preparingRunId !== input.execution.binding.runId || cachedReleasedOwner.instructionPreparationRunId && !ownInstructionPreparation || cachedReleasedOwner.closeOnReleaseReason) throw new Error("native_session_supervisor_busy"); - if (proof.taskId !== cachedReleasedOwner.pluginTaskId || proof.scopeDigest !== releasedWarmScope(input) - || canonicalJson(proof.source) !== canonicalJson(identity)) throw new Error("native_session_released_owner_scope_mismatch"); + if (source.taskId !== cachedReleasedOwner.pluginTaskId || cachedReleasedOwner.runId !== identity?.runId || + (proof && (proof.scopeDigest !== releasedWarmScope(input) || canonicalJson(proof.source) !== canonicalJson(identity)))) { + throw new Error("native_session_released_owner_scope_mismatch"); + } const wasBusy = cachedReleasedOwner.busy; cachedReleasedOwner.busy = true; try { - // Only physical stop releases the cached controller; ordinary checkpoint admission still proves continuation. - const resumeFrom = await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId, - leaseId: target.leaseId, priorIdentity: proof.source, inspectCheckpoint: true }); - if (resumeFrom !== proof.taskId) throw new Error("native_session_released_owner_scope_mismatch"); + // Ordinary replacement still needs a stop receipt for the old task. + const resumeFrom = portableSource?.taskId ?? await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId, + leaseId: target.leaseId, priorIdentity: source.identity, inspectCheckpoint: true }); + if (resumeFrom !== source.taskId) throw new Error("native_session_released_owner_scope_mismatch"); try { await detachReleasedWarmNativeSession(cachedReleasedOwner); } catch (error) { @@ -10790,7 +10866,7 @@ async function executePaperclipNativeSessionAttempt( } finally { cachedReleasedOwner.busy = wasBusy; } } else requireReleasedWarmScope(input, proof, identity); } - if (input.useRunnerd && !input.restartRecovery && !input.backend && + if (input.useRunnerd && canRecoverWarmOwner && !input.backend && input.runnerExecutionTarget?.kind === "remote" && input.execution.session.lifecyclePolicy.mode === "warm" && !idleWarmNativeSessionOwnerRunId(input)) { @@ -10800,10 +10876,12 @@ async function executePaperclipNativeSessionAttempt( if (taskTarget.transport === "sandbox" && taskTarget.runnerTask && taskTarget.leaseId && durableIdentityMatchesSession(identity,input.execution) && identity.runId !== input.execution.binding.runId) { // Inspection can replace a stopped retained task before deciding between adoption and checkpoint recovery. - const description = environmentTaskResultSchema.parse(await taskTarget.runnerTask({kind:"describe"})); - if (description.kind !== "description") throw new Error("Plugin Runner description unavailable"); - pluginTaskDescription = description; - if (taskTarget.sandboxLeaseAcquisition?.outcome !== "resumed") { + if (!pluginTaskDescription || pluginTaskDescription.taskId !== taskTarget.sandboxLeaseAcquisition?.providerLeaseId) { + const description = environmentTaskResultSchema.parse(await taskTarget.runnerTask({ kind: "describe" })); + if (description.kind !== "description") throw new Error("Plugin Runner description unavailable"); + pluginTaskDescription = description; + } + if (taskTarget.sandboxLeaseAcquisition?.outcome !== "resumed" && !portableSource) { await recoverPluginRunnerCheckpoint(input, root); } } @@ -10817,22 +10895,23 @@ async function executePaperclipNativeSessionAttempt( } if (durableIdentityMatchesSession(identity, input.execution) && identity.runId !== input.execution.binding.runId && runnerdAuthorityLifecycleWithVerifiedBackup({ root, identity, execution: input.execution, allowVerifiedBackup: true }) !== "suspended") { - let releasedSource: DurableRecoveryIdentity | undefined; + let releasedSource = portableSource?.identity; remoteWarmRecovery = await inspectRemoteWarmSessionRecovery({ db: input.db, execution: input.execution, target: input.runnerExecutionTarget, identity, + portableSourceTaskId: portableSource?.taskId, readControlState: () => record(JSON.parse(readBoundedNativeFile( resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES, "runner_durable_identity_too_large").toString("utf8"))), - matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) && - nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(input.execution, "remote"), + matchesExecution: previous => matchesRemoteWarmExecution(previous, input.execution, input.toolRefreshOnResume), loadCheckpoint: previous => { - const proof = releasedWarmRunner(readWarmNativeEnvelope(previous)); - if (proof) { - releasedSource = requireReleasedWarmScope(input, proof, identity); + if (!portableSource) { + const proof = releasedWarmRunner(readWarmNativeEnvelope(previous)); + if (proof) releasedSource = requireReleasedWarmScope(input, proof, identity); } return loadWarmNativeCheckpoint(previous, nativeSessionConfigDigest(previous, "remote"), "remote"); }, - canReuseProcess: previous => !needsCoreGitHubAccess(input) && canReuseNativeProcess(input, + canReuseProcess: previous => !portableSource && !needsCoreGitHubAccess(input) && + nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(input.execution, "remote") && canReuseNativeProcess(input, record(JSON.parse(readBoundedNativeFile(nativeSessionCheckpointPath(previous), NATIVE_WARM_CHECKPOINT_MAX_BYTES, "native_session_supervisor_checkpoint_too_large").toString("utf8"))).processConfigDigest), }) ?? undefined; @@ -11061,7 +11140,7 @@ async function executePaperclipNativeSessionAttempt( : coordinator.controllerGeneration + 1; const target = input.runnerExecutionTarget; const retained = warmNativeSessions.get(nativeSessionScopeKey(input.execution)); - if (!input.restartRecovery && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId && + if (canRecoverWarmOwner && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId && ((retained?.pluginTaskId && idleWarmNativeSessionOwnerRunId(input) !== undefined) || remoteWarmRecovery?.releasedSource) && (input.execution.provider.kind === "codex" || (input.execution.provider.kind === "acpx" && input.execution.provider.agent === "claude"))) { claimedStartupContext = new NativeExecutionContext(input.execution, Math.min(executionLeaseTtlMs, @@ -11801,7 +11880,7 @@ async function executePaperclipNativeSessionAttempt( executionContext = undefined; startupForAdmission!.executionContext = undefined; }; - if (warmSessionId !== null && warmConfigDigest !== null && !input.restartRecovery) { + if (warmSessionId !== null && warmConfigDigest !== null && canRecoverWarmOwner) { const registered = warmNativeSessions.get(warmSessionId); const entry = registered && forgetUnretiredOwnerOnOtherSandbox(warmSessionId, registered, input.runnerExecutionTarget) ? undefined : registered; @@ -12088,11 +12167,15 @@ async function executePaperclipNativeSessionAttempt( signal: executionContext?.signal ?? new AbortController().signal, }); if (!recovered?.recovered || !recovered.session) throw new Error("runner_state_identity_mismatch: remote_warm_session_unproven"); - if (remoteWarmRecovery.reuseProcess) { + const movedSession = remoteWarmRecovery.portableSourceTaskId !== undefined; + // A portable resume already starts a new Runner. Its run attachment + // prepares the current tools and runtime on the verified target. + if (movedSession || remoteWarmRecovery.reuseProcess) { existingWarmSession = recovered.session; - await emitStartupReuse(input.onEvent, "runner", { state: "reused", reason: "verified_reuse" }); + await emitStartupReuse(input.onEvent, "runner", movedSession + ? { state: "fresh", reason: "replacement" } : { state: "reused", reason: "verified_reuse" }); await emitStartupReuse(input.onEvent, "provider_session", { state: "reused", reason: "verified_reuse" }); - trace.setBudgetStart("warm"); + if (!movedSession) trace.setBudgetStart("warm"); persistedWarmSession = null; } else { await recovered.session.close({ reason: "recovered warm session requires current process configuration" }); @@ -12104,12 +12187,22 @@ async function executePaperclipNativeSessionAttempt( remoteWarmRecovery.previousExecution.binding, scopedRunnerdStateRoot(input.execution), nativeSessionKey(input.execution)); if (!suspendedController) throw new Error("native_runner_checkpoint_missing_after_suspension"); - persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution, - nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote"), await recovered.session.snapshot()); - persistedWarmSession = loadWarmNativeCheckpoint(input.execution, warmConfigDigest!, "remote"); - runnerdBackend = await createRunnerdBackend({ ...input, onDiagnosticLease: ownDiagnosticLease, providerPackAdmissionSource, recoveryPurpose: claimedStopCleanup ? "stop_cleanup" : "execution", onAccountingEvent: producerUsage?.capture, runnerInstanceId: effectiveRunnerInstanceId, + const verifiedConfigDigest = nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote"); + persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution, verifiedConfigDigest, await recovered.session.snapshot()); + persistedWarmSession = loadWarmNativeCheckpoint(input.execution, verifiedConfigDigest, "remote"); + runnerdBackend = await createRunnerdBackend({ + ...input, + pluginTaskDescription: undefined, + onDiagnosticLease: ownDiagnosticLease, + providerPackAdmissionSource, + recoveryPurpose: claimedStopCleanup ? "stop_cleanup" : "execution", + onAccountingEvent: producerUsage?.capture, + runnerInstanceId: effectiveRunnerInstanceId, controllerPersistence: input.runnerExecutionTarget?.kind === "remote" ? { bootId: controller.bootId } : undefined, - durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId, trace, toolTrace }); + durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId, + trace, + toolTrace, + }); } } const readRemoteCleanupLease = async () => input.runnerExecutionTarget?.kind === "remote" && @@ -15710,6 +15803,7 @@ export function remoteCheckpointIncompleteFailure( export async function createRunnerdBackend(input: { executionContext?: NativeExecutionContext; casesEnabled?: boolean; + toolRefreshOnResume?: boolean; db: Db; providerPackAdmissionSource?: NativeProviderPackAdmissionSource; controllerPersistence?: { bootId: string; runnerArtifact?: { version: string; digest: string } }; @@ -15724,7 +15818,7 @@ export async function createRunnerdBackend(input: { remoteWarmRecovery?: Omit & { revalidate: () => Promise }; retainedPluginTaskOwner?: ReturnType; beginRetainedRunAttachment?: () => void; - pluginTaskDescription?: Extract,{kind:"description"}>; + pluginTaskDescription?: PluginTaskDescription; onSpawn?: (meta: { pid: number; processGroupId: number | null; @@ -16128,8 +16222,10 @@ async function createRunnerdBackendWithinSessionClaim( const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity ? createNativeRecoveryUsageDelivery({ db: input.db, execution: input.execution, sourceIdentity: { ...recoveryUsageIdentity }, - matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) && - nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind), + matchesExecution: previous => input.remoteWarmRecovery + ? matchesRemoteWarmExecution(previous, input.execution, input.toolRefreshOnResume) + : nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) && + nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind), }) : undefined; const accountingTransportOptions = { @@ -16185,15 +16281,19 @@ async function createRunnerdBackendWithinSessionClaim( const { root, durableIdentity, durableBinding } = durableState ??= readDurableState(); const provider = input.execution.provider; if (provider.kind !== "codex" && provider.kind !== "opencode" && provider.kind !== "acpx") throw new Error("Plugin Runner tasks require a workspace harness"); + if (input.restartRecovery?.kind === "reattach_remote_runner") { + await assertPluginTaskReattachmentBinding(input, remoteTarget, durableBinding ? durableIdentity : null); + } + let priorDescription = input.pluginTaskDescription?.taskId === remoteTarget.sandboxLeaseAcquisition?.providerLeaseId + ? input.pluginTaskDescription : undefined; if (remoteTarget.sandboxLeaseAcquisition?.outcome === "resumed" && !durableBinding && !input.retainedPluginTaskOwner && !input.remoteWarmRecovery) { // A new logical session (including fresh retries) cannot inherit the old process authority. await pluginTask({kind:"stop"}); const stopped=environmentTaskResultSchema.parse(await pluginTask({kind:"status"})); if(stopped.kind!=="status" || stopped.executionStopped!==true) throw new Error("Plugin Runner termination is unconfirmed; reconcile the saved task"); await pluginTask({kind:"complete"}); + priorDescription = undefined; } - const priorDescription = input.pluginTaskDescription?.taskId === remoteTarget.sandboxLeaseAcquisition?.providerLeaseId - ? input.pluginTaskDescription : undefined; const description = priorDescription ?? environmentTaskResultSchema.parse(await pluginTask({ kind: "describe" })); if (description.kind !== "description" || description.launchPreparationVersion !== 1 || description.projectAuthCheckVersion !== 1) throw new Error("Plugin Runner launch preparation v1 unavailable"); const saved = durableBinding ? durableIdentity as RunnerdDurableIdentity : undefined; @@ -16212,9 +16312,7 @@ async function createRunnerdBackendWithinSessionClaim( normalizedSessionId: input.remoteWarmRecovery.identity.normalizedSessionId, turnId: input.remoteWarmRecovery.identity.turnId, itemId: input.remoteWarmRecovery.identity.itemId, } : undefined; - if (input.restartRecovery?.kind === "reattach_remote_runner" && (!prior || !prior.turnId || !prior.itemId || prior.environmentLeaseId !== remoteTarget.leaseId)) { - throw new Error("native_plugin_recovery_lease_mismatch"); - } + const resumeFrom = !retainedOwner && !adopting && prior && prior.environmentLeaseId !== remoteTarget.leaseId ? await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId, leaseId: remoteTarget.leaseId!, priorIdentity: prior }) : undefined; @@ -16224,7 +16322,8 @@ async function createRunnerdBackendWithinSessionClaim( home: input.managedAiCredentialHome, environment: input.runnerEnvironment ?? {}, }); if (credentials.needsWriteback && !input.managedAiCredentialWriteback) throw new Error("Plugin Runner credential writeback authority unavailable"); - if (!adopting && input.managedAiCredentialWriteback && credentials.accountIdentity) { + // Same-lease recovery already has a writeback binding. A portable move creates a new lease. + if ((!adopting || input.remoteWarmRecovery?.portableSourceTaskId) && input.managedAiCredentialWriteback && credentials.accountIdentity) { if (!remoteTarget.leaseId) throw new Error("Plugin Runner credential lease unavailable"); await registerPluginTaskCredentialHandoff(input.db, { companyId: input.execution.binding.companyId, leaseId: remoteTarget.leaseId }, credentials.accountIdentity, input.managedAiCredentialWriteback); @@ -16251,7 +16350,8 @@ async function createRunnerdBackendWithinSessionClaim( releaseResumeVersion: description.releaseResumeVersion, authorityRunId: description.releaseResumeVersion === 1 ? input.execution.binding.runId : undefined, releasedSource: input.remoteWarmRecovery?.releasedSource, - onResumed: async source => updateReleasedWarmRunner(input.execution, source, description.taskId, null), + sessionHandoff: !!input.remoteWarmRecovery?.portableSourceTaskId, + onResumed: async source => updateReleasedWarmRunner(input.execution, source, input.remoteWarmRecovery?.portableSourceTaskId ?? description.taskId, null), onRetired: async source => updateReleasedWarmRunner(input.execution, source, description.taskId, null), adoptIdentity: adoptionIdentity ?? (adopting && prior ? { runnerInstanceId: prior.runnerInstanceId, environmentLeaseId: prior.environmentLeaseId, diff --git a/server/src/services/native-runtime/native-session-resume.ts b/server/src/services/native-runtime/native-session-resume.ts index 9cd6fddbe16..bb9fe4f458d 100644 --- a/server/src/services/native-runtime/native-session-resume.ts +++ b/server/src/services/native-runtime/native-session-resume.ts @@ -8,7 +8,7 @@ import type { NativeExecutionInput, PersistedNativeSession, } from "../../vendor/paperclip-runner/index.js"; -import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js"; +import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput, type NativeRuntimeContextSnapshot } from "../../vendor/paperclip-runner/index.js"; export type NativeToolExecutionTargetKind = "local" | "remote"; @@ -612,6 +612,15 @@ export function buildNativeExecutionWithCheckpoint(input: { }; } +export function nativeRuntimeContextsMatch( + previous: NativeRuntimeContextSnapshot, + current: NativeRuntimeContextSnapshot, + toolRefreshOnResume?: boolean, +): boolean { + return previous.aggregateDigest === current.aggregateDigest || + (toolRefreshOnResume === true && canonicalNativeRuntimeContextDigest({ ...previous, mcp: current.mcp }) === current.aggregateDigest); +} + /** * Rebind a completed prior run's provider checkpoint to a new heartbeat run. * The provider/driver session identity is retained, while every per-turn and @@ -697,12 +706,7 @@ export function rebindNativeSessionCheckpoint(input: { ("homeMode" in previousExecution ? previousExecution.homeMode : "isolated") !== ("homeMode" in current ? current.homeMode : "isolated") || ("runtimeContext" in previousExecution && "runtimeContext" in current && - previousExecution.runtimeContext.aggregateDigest !== current.runtimeContext.aggregateDigest && - !(input.toolRefreshOnResume === true && - canonicalNativeRuntimeContextDigest({ - ...previousExecution.runtimeContext, - mcp: current.runtimeContext.mcp, - }) === current.runtimeContext.aggregateDigest)) + !nativeRuntimeContextsMatch(previousExecution.runtimeContext, current.runtimeContext, input.toolRefreshOnResume)) ) return null; diff --git a/server/src/services/native-runtime/native-source-recovery-ownership.ts b/server/src/services/native-runtime/native-source-recovery-ownership.ts index 51b4eb56f0c..5d5ff564584 100644 --- a/server/src/services/native-runtime/native-source-recovery-ownership.ts +++ b/server/src/services/native-runtime/native-source-recovery-ownership.ts @@ -1,6 +1,7 @@ -import { and, asc, eq, exists, inArray, sql } from "drizzle-orm"; +import { and, asc, eq, exists, inArray, or, sql } from "drizzle-orm"; import { environmentLeases, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db"; import { inspectDurablePrpCheckpointHeader, type DurablePrpCheckpoint } from "../../vendor/paperclip-runner/index.js"; +import { nativeSha256 } from "./canonical.js"; type Transaction = Parameters[0]>[0]; const record = (value: unknown): Record => value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; @@ -18,7 +19,8 @@ export async function readNativeSourceConsumers(tx: Transaction, companyId: stri sql`${nativeRunnerCheckpoints.material}->>'schema' like 'paperclip.native-runner-checkpoint-retirement.%'`))); return tx.select({ id: environmentLeases.id, metadata: environmentLeases.metadata, retired }).from(environmentLeases).where(and( eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId), - sql`${environmentLeases.metadata}->'runnerTaskSourceConsumer' is not null`)); + or(sql`${environmentLeases.metadata}->'runnerTaskSourceConsumer' is not null`, + sql`${environmentLeases.metadata}->'portableTaskSuccessor' is not null`))); } /** Physical source consumption revokes the completed controller's late writer. */ @@ -30,13 +32,26 @@ export async function assertNativeSourceCheckpointConsumer(tx: Transaction, owne if (!rows.length) return; const deny = (): never => { throw new Error("native_runner_checkpoint_owner_lost"); }; if (owner.runId === owner.authorityRunId || rows.length !== 1 || !checkpoint) return deny(); - const source = rows[0]!, consumer = record(source.metadata?.runnerTaskSourceConsumer); + const source = rows[0]!; + const header = checkpoint.records.get("header"); + const identity = header ? inspectDurablePrpCheckpointHeader(header) : null; + const portable = record(source.metadata?.portableTaskSuccessor); + if (source.metadata?.portableTaskSuccessor !== undefined) { + if (typeof portable.targetLeaseId !== "string" || portable.sourceLeaseId !== source.id || + !identity || nativeSha256(record(portable.operation).identity ?? null) !== nativeSha256(identity)) return deny(); + const [target] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, owner.companyId), + eq(environmentLeases.id, portable.targetLeaseId))); + if (!target || target.heartbeatRunId !== owner.authorityRunId || target.status !== "active" || target.releasedAt || + target.expiresAt && target.expiresAt.getTime() <= Date.now() || + target.metadata?.portableTaskOwnershipReady !== true || target.metadata.portableTaskSuccessor !== undefined || + nativeSha256(target.metadata.portableTaskPredecessor ?? null) !== nativeSha256(portable)) return deny(); + return; + } + const consumer = record(source.metadata?.runnerTaskSourceConsumer); if (typeof consumer.leaseId !== "string" || consumer.runId !== owner.authorityRunId) return deny(); const [lease] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, owner.companyId), eq(environmentLeases.id, consumer.leaseId))); const recovery = record(lease?.metadata?.runnerTaskSourceRecovery); - const header = checkpoint.records.get("header"); - const identity = header ? inspectDurablePrpCheckpointHeader(header) : null; const expected = record(recovery.identity); if (!lease || lease.heartbeatRunId !== owner.authorityRunId || lease.status !== "active" || lease.releasedAt || lease.expiresAt && lease.expiresAt.getTime() <= Date.now() || diff --git a/server/src/services/native-runtime/plugin-runner-task.test.ts b/server/src/services/native-runtime/plugin-runner-task.test.ts index 75c980061e1..6ee44595a4e 100644 --- a/server/src/services/native-runtime/plugin-runner-task.test.ts +++ b/server/src/services/native-runtime/plugin-runner-task.test.ts @@ -553,6 +553,18 @@ describe("plugin Runner release/resume facade", () => { ? { kind: "released", taskId: "task" } : { kind: "resume", taskId: "task", outcome: "resumed" }); + it.each([true, false])("sends the current Claude credential only for session handoff (%s)", async sessionHandoff => { + const task = lifecycleTask(); + const environment = { CLAUDE_CODE_OAUTH_TOKEN: "fixture-current-token" }; + const owner = createPluginRunnerTaskOwner({ ...input("claude"), task, sessionHandoff, + credential: { type: "claude-env", environment } }); + await expect(owner.resume(requirements.identity, "transient-ticket")).resolves.toBe("resumed"); + expect(task).toHaveBeenCalledExactlyOnceWith({ + kind: "resume", identity: requirements.identity, bootstrapTicket: "transient-ticket", + ...(sessionHandoff ? { handoffCredential: { accountIdentity: "selected-account", environment } } : {}), + }, { expectedTaskId: "task" }); + }); + it.each(["codex", "claude"])("quiesces %s before release and closes only after its ACK", async harness => { const { task } = provider(); const perform = task.getMockImplementation()!; diff --git a/server/src/services/native-runtime/plugin-runner-task.ts b/server/src/services/native-runtime/plugin-runner-task.ts index 573c0e276f0..91a9b491c30 100644 --- a/server/src/services/native-runtime/plugin-runner-task.ts +++ b/server/src/services/native-runtime/plugin-runner-task.ts @@ -50,6 +50,7 @@ export function createPluginRunnerTaskOwner(input: { warmTaskVersion?: 1; releaseResumeVersion?: 1; releasedSource?: DurableRecoveryIdentity; + sessionHandoff?: boolean; onResumed?: (identity: DurableRecoveryIdentity) => Promise; onRetired?: (identity: DurableRecoveryIdentity) => Promise; githubBroker?: { url: string; token: string }; @@ -418,7 +419,12 @@ export function createPluginRunnerTaskOwner(input: { }; const resume = async (identity: DurableRecoveryIdentity, bootstrapTicket: string): Promise<"resumed" | "unavailable"> => { requireReleaseResume(); - const result = await call(environmentTaskOperationSchema.parse({ kind: "resume", identity, bootstrapTicket })); + const handoffCredential = input.sessionHandoff && input.credential?.type === "claude-env" + ? { accountIdentity: input.accountIdentity, environment: input.credential.environment } + : undefined; + const result = await call(environmentTaskOperationSchema.parse({ kind: "resume", identity, bootstrapTicket, + ...(handoffCredential ? { handoffCredential } : {}), + })); requireReleaseResume(); if (result.kind !== "resume") throw new Error("Plugin Runner resume result unavailable"); if (result.outcome === "unavailable") { diff --git a/server/src/services/native-runtime/plugin-task-source-recovery.integration.test.ts b/server/src/services/native-runtime/plugin-task-source-recovery.integration.test.ts index e1de7eb4391..dc6e988fdc4 100644 --- a/server/src/services/native-runtime/plugin-task-source-recovery.integration.test.ts +++ b/server/src/services/native-runtime/plugin-task-source-recovery.integration.test.ts @@ -3,19 +3,23 @@ import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { eq, sql } from "drizzle-orm"; -import { environmentTaskOperationSchema } from "@paperclipai/plugin-sdk"; +import { environmentTaskOperationSchema, type PluginEnvironmentTaskResult } from "@paperclipai/plugin-sdk"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { agents, companies, completionContracts, createDb, environmentLeases, environments, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, nativeRunnerCheckpoints, plugins } from "@paperclipai/db"; -import { parseNativeExecutionInput, runnerProviderLaunchRequirements, DurablePrpControlPlane, type DurablePrpCheckpoint, type NativeExecutionInput } from "../../vendor/paperclip-runner/index.js"; +import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput, runnerProviderLaunchRequirements, DurablePrpControlPlane, type DurablePrpCheckpoint, type NativeExecutionInput } from "../../vendor/paperclip-runner/index.js"; import { recoveryPostgresSupport, startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js"; import { createIsolatedTestHome } from "../../__tests__/helpers/isolated-test-home.js"; import { executeEnvironmentTask, resolvePluginTaskResume } from "../environment-task-runtime.js"; import type { PluginWorkerManager } from "../plugin-worker-manager.js"; +import { assertAttachedPortableTask, readyPortableTaskSource, reconcilePluginTaskOwnership } from "../plugin-task-ownership.js"; import { remoteTerminationReceipt } from "../remote-execution-termination.js"; +import * as runtimeApi from "../../runtime-api.js"; import { NativeExecutionContext } from "./native-execution-context.js"; import { buildNativeExecutionInput } from "./native-execution-input.js"; import { currentNativeControllerIdentity } from "./native-restart-recovery.js"; +import { createRunnerdBackend, nativeControllerMigrationDescriptor } from "./native-session-executor.js"; +import { buildNativeExecutionWithCheckpoint, nativeToolContractFingerprintForTarget } from "./native-session-resume.js"; import { loadNativeRunnerCheckpoint, nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js"; import { claimPluginTaskSourceRecovery } from "./plugin-task-source-recovery.js"; import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; @@ -83,7 +87,7 @@ describe.skipIf(!recoveryPostgresSupport.supported)("stopped source admission un leaseOwner: owner.leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000) }); await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "source-fixture", version: "1.0.0", status: "ready", - manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }] } as never }); + manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true, sessionHandoffVersion: 1 }] } as never }); await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "plugin", status: "active", config: { pluginKey: pluginId, driverKey: "tasks" } }); const metadata = { driver: "plugin", pluginId, driverKey: "tasks", runnerTaskLease: true }; @@ -98,7 +102,7 @@ describe.skipIf(!recoveryPostgresSupport.supported)("stopped source admission un const input = { companyId, leaseId, context, identity, resumeFrom: "source-task", ...(provider === "codex" ? { codexSourceRecoveryVersion: 1 as const } : { acpxSourceRecoveryVersion: 1 as const }), accountIdentity: "current-selected-account", matchesExecution: (previous: typeof sourceExecution) => previous.provider.kind === execution.provider.kind && previous.workspace.cwd === execution.workspace.cwd }; - const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call: vi.fn(async (_id: string, _method: string, p: any) => { + const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call: vi.fn(async (_id: string, _method: string, p: any): Promise => { if (p.operation.kind === "prepare") return { kind: "prepared", taskId: p.taskId, launch: { configurationId: "source-configuration", requirementsDigest: `sha256:${"a".repeat(64)}`, provider: {} } }; if (p.operation.kind === "check_project_auth") return { kind: "project_auth", taskId: p.taskId, provider: p.operation.provider, configurationId: p.operation.configurationId, allowed: true }; @@ -125,6 +129,213 @@ describe.skipIf(!recoveryPostgresSupport.supported)("stopped source admission un harness: provider === "claude" ? "claude" as const : "codex" as const, authProvider: provider === "claude" ? "anthropic" : "openai" }; } + it.each([ + { policy: "ordinary", scenario: "ordinary handoff" }, + { policy: "reclaim_if_needed", scenario: "idle source reclaim" }, + { policy: "ordinary", scenario: "fresh conversation after instructions change", freshSession: true }, + ] as const)("reconciles ownership after a lost response: $scenario", async scenario => { + const { policy } = scenario; + const f = await fixture(); + await db.update(heartbeatRuns).set({ scopeKind: "issue", issueId: f.issueId }).where(eq(heartbeatRuns.id, f.runId)); + await db.update(environmentLeases).set({ status: "retained", releasedAt: null, + metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination') || ${JSON.stringify({ + providerMetadata: { ownershipGeneration: "source-generation" }, runnerTaskAccountIdentity: f.accountIdentity, + })}::jsonb`, + }).where(eq(environmentLeases.id, f.sourceLeaseId)); + await db.update(environmentLeases).set({ + metadata: sql`${environmentLeases.metadata} || '{"providerMetadata":{"ownershipGeneration":"target-generation"}}'::jsonb`, + }).where(eq(environmentLeases.id, f.leaseId)); + const request = { companyId: f.companyId, sourceLeaseId: f.sourceLeaseId, targetLeaseId: f.leaseId, policy }; + const send = (database: typeof db) => reconcilePluginTaskOwnership(database, request, operation => + executeEnvironmentTask(database, f.workers as unknown as PluginWorkerManager, { companyId: f.companyId, leaseId: f.leaseId, operation })); + const describe = (database: typeof db, ownershipSignal?: AbortSignal) => executeEnvironmentTask(database, f.workers as unknown as PluginWorkerManager, + { companyId: f.companyId, leaseId: f.leaseId, ownershipSignal, operation: { kind: "describe" } }); + if (policy === "ordinary") await db.update(environmentLeases).set({ + metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ portableTaskPreviousLeaseId: f.sourceLeaseId })}::jsonb`, + }).where(eq(environmentLeases.id, f.leaseId)); + f.workers.call.mockRejectedValueOnce(new Error("worker transport lost")); + await expect(policy === "ordinary" ? describe(db) : send(db)).rejects.toThrow(); + const operation = f.workers.call.mock.calls[0][2].operation; + expect(operation).toMatchObject({ kind: "ensure_ownership", operationId: f.leaseId, identity: f.identity, + sourceTaskId: "source-task", sourceGeneration: "source-generation", targetGeneration: "target-generation", policy }); + await expect(f.task({ ...operation, policy: policy === "ordinary" ? "reclaim_if_needed" : "ordinary" })).rejects.toThrow("ownership_unverified"); + await expect(f.task()).rejects.toThrow("session ownership unavailable"); + await expect(executeEnvironmentTask(db, f.workers as unknown as PluginWorkerManager, { + companyId: f.companyId, leaseId: f.sourceLeaseId, operation: { kind: "resume", identity: f.identity, bootstrapTicket: "fixture-ticket" }, + })).rejects.toThrow("session ownership unavailable"); + expect(f.workers.call).toHaveBeenCalledTimes(1); + const reply = { kind: "ownership", taskId: "replacement-task", operationId: f.leaseId, + targetGeneration: "target-generation" } as const; + f.workers.call.mockResolvedValueOnce({ ...reply, status: "pending" }); + await expect(describe(db)).rejects.toThrow("native_portable_task_ownership_pending"); + const targetWriter = nativeRunnerCheckpointWriter(db, { companyId: f.companyId, issueId: f.issueId, + runId: f.sourceRunId, authorityRunId: f.runId, ...f.owner }); + await expect(targetWriter(f.checkpoint)).rejects.toThrow("native_runner_checkpoint_owner_lost"); + const restarted = createDb(database.connectionString); + f.workers.call.mockResolvedValueOnce({ ...reply, status: "ready", targetGeneration: "another-generation" }); + await expect(describe(restarted)).rejects.toThrow("reconcile the same task"); + await expect(f.task()).rejects.toThrow("session ownership unavailable"); + const description = { kind: "description", taskId: "replacement-task", + runner: { version: "test", digest: `sha256:${"a".repeat(64)}` } } as const; + f.workers.call.mockRejectedValueOnce(new Error("worker transport lost again")); + f.workers.call.mockResolvedValueOnce({ ...reply, status: "pending" }); + f.workers.call.mockResolvedValueOnce({ ...reply, status: "ready" }); + f.workers.call.mockResolvedValueOnce(description); + await expect(describe(restarted, new AbortController().signal)).resolves.toEqual(description); + expect(f.workers.call.mock.calls.map(([, , call]) => call.operation)).toEqual([...Array(6).fill(operation), { kind: "describe" }]); + const [target] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + expect(target.metadata?.portableTaskOwnershipReady).toBe(true); + expect(target.metadata?.runnerTaskAccountIdentity).toBe(f.accountIdentity); + const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.sourceLeaseId)); + expect(source.metadata?.remoteExecutionTermination).toBeUndefined(); + await expect(f.oldWriter(f.checkpoint)).rejects.toThrow("native_runner_checkpoint_owner_lost"); + await expect(targetWriter(f.checkpoint)).resolves.toBeUndefined(); + + async function recoverAttachedRunner(execution: NativeExecutionInput, identity: typeof f.identity) { + const { root: stateDirectory } = nativeControllerMigrationDescriptor(execution, + nativeToolContractFingerprintForTarget("remote", execution.provider.kind)); + const durable = new DurablePrpControlPlane({ stateDirectory: join(stateDirectory, "control-plane"), + identity: identity, expectedRunnerVersion: description.runner.version, expectedRunnerDigest: description.runner.digest }); + durable.persistRunAttachTemplate({ provider: { instructions: "Run using the admitted instructions.", + runtimeContext: "runtimeContext" in execution ? execution.runtimeContext : null } }); + await durable.flushCheckpoint(); + await durable.stop(); + const recover = () => createRunnerdBackend({ db: restarted, execution, + runnerInstanceId: identity.runnerInstanceId, runnerIngressAuthorized: true, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", leaseId: f.leaseId, + sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "replacement-task" }, + runnerTask: (operation, options) => executeEnvironmentTask(restarted, f.workers as unknown as PluginWorkerManager, { + companyId: f.companyId, leaseId: f.leaseId, operation: environmentTaskOperationSchema.parse(operation), + expectedTaskId: options?.expectedTaskId, + }), + }, + restartRecovery: { kind: "reattach_remote_runner", remote: { kind: "plugin_task", leaseId: f.leaseId }, + runId: f.runId, leaseOwner: "successor", controllerGeneration: f.owner.generation + 1, + providerAttempt: 1, restartKind: "hard", recoveryRequestId: null }, + }); + f.workers.call.mockClear(); + f.workers.call + .mockResolvedValueOnce({ kind: "status", taskId: "replacement-task", phase: "running", executionStopped: false }) + .mockResolvedValueOnce({ kind: "state", taskId: "replacement-task", + state: { schema: "paperclip.runner.durable.state.v1", ...identity } }) + .mockResolvedValueOnce({ ...description, launchPreparationVersion: 1, projectAuthCheckVersion: 1 }); + const api = vi.spyOn(runtimeApi, "resolveLocalRuntimeApiUrl").mockReturnValue("http://127.0.0.1:3100"); + try { await expect(recover()).resolves.toBeDefined(); } + finally { api.mockRestore(); } + expect(f.workers.call.mock.calls.map(([, , call]) => [call.taskId, call.operation.kind])).toEqual([ + ["replacement-task", "status"], ["replacement-task", "read_state"], ["replacement-task", "describe"], + ]); + return recover; + } + + if ("freshSession" in scenario) { + const previousExecution = f.sourceExecution; + const nativeToolContractFingerprint = nativeToolContractFingerprintForTarget("remote", f.execution.provider.kind); + const normalizedSessionId = f.identity.normalizedSessionId; + const previousRun = { + id: f.sourceRunId, companyId: f.companyId, agentId: f.execution.binding.agentId, nativeSessionId: normalizedSessionId, + runnerProfileJson: { nativeToolContractFingerprint, nativeExecutionInput: previousExecution, + sessionCheckpoint: { backendKind: "runner", driverKind: previousExecution.session.driverKind, + sessionId: "previous-provider-thread", providerSessionId: "previous-provider-thread", + identity: { ...previousExecution.binding, sessionId: normalizedSessionId }, pendingRuntimeRequests: [] } }, + }; + const select = (current: NativeExecutionInput) => buildNativeExecutionWithCheckpoint({ + previousRun, normalizedSessionId, executionTargetKind: "remote", toolRefreshOnResume: true, + buildExecution: ({ normalizedSessionId }) => ({ ...current, session: { ...current.session, normalizedSessionId } } as NativeExecutionInput), + }); + expect(select(f.execution).checkpoint?.providerSessionId).toBe("previous-provider-thread"); + const changed = structuredClone(f.execution); + if (!("runtimeContext" in changed)) throw new Error("fixture requires runtime context"); + changed.runtimeContext.instructions.bundle.digest = "1".repeat(64); + changed.runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(changed.runtimeContext); + const selected = select(changed); + expect(selected.checkpoint).toBeNull(); + expect(selected.normalizedSessionId).not.toBe(normalizedSessionId); + const scope = { attachedRunId: f.runId, accountIdentity: f.accountIdentity, selectedSessionId: selected.normalizedSessionId }; + const sourceFor = (identity: unknown) => readyPortableTaskSource(restarted, f.companyId, f.leaseId, identity, scope); + await expect(sourceFor(null)).rejects.toThrow("ownership_unverified"); + await db.update(heartbeatRuns).set({ nativeSessionId: selected.normalizedSessionId }).where(eq(heartbeatRuns.id, f.runId)); + await expect(sourceFor(null)).rejects.toThrow("ownership_unverified"); + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: selected.execution } }) + .where(eq(heartbeatRuns.id, f.runId)); + await expect(sourceFor(null)).resolves.toBeUndefined(); + await expect(sourceFor(f.identity)).rejects.toThrow("ownership_unverified"); + const duplicate = randomUUID(); + await db.insert(environmentLeases).values({ ...target, id: duplicate, providerLeaseId: "ambiguous-task" }); + await expect(sourceFor(null)).rejects.toThrow("ownership_unverified"); + await db.delete(environmentLeases).where(eq(environmentLeases.id, duplicate)); + + const fresh = { ...f.identity, runId: f.runId, environmentLeaseId: f.leaseId, runnerInstanceId: randomUUID(), + normalizedSessionId: selected.normalizedSessionId, turnId: "fresh-turn", itemId: "fresh-item" }; + const prepare = { ...f.operation, resumeFrom: undefined, requirements: { ...f.operation.requirements, identity: fresh } }; + const submit = { ...f.submit, resumeFrom: undefined, runner: { ...f.submit.runner, + runnerId: fresh.runnerInstanceId, leaseId: fresh.environmentLeaseId, runId: fresh.runId, + sessionId: fresh.normalizedSessionId, turnId: fresh.turnId, itemId: fresh.itemId } }; + f.workers.call.mockClear(); + await expect(f.task(prepare)).resolves.toMatchObject({ kind: "prepared" }); + await expect(f.task({ kind: "check_project_auth", provider: "openai", projectIds: [], + configurationId: submit.preparedLaunchId })).resolves.toMatchObject({ kind: "project_auth", allowed: true }); + await expect(f.task(submit)).resolves.toMatchObject({ kind: "accepted" }); + expect(f.workers.call.mock.calls.map(([, , call]) => call.operation.kind)).toEqual(["prepare", "check_project_auth", "submit"]); + expect(f.workers.call.mock.calls.every(([, , call]) => call.taskId === "replacement-task" && + call.operation.resumeFrom === undefined && call.operation.priorIdentity === undefined)).toBe(true); + await expect(sourceFor(null)).rejects.toThrow("ownership_unverified"); + await expect(sourceFor({ ...fresh, turnId: "wrong-turn" })).rejects.toThrow("ownership_unverified"); + await expect(sourceFor(fresh)).resolves.toBeUndefined(); + await recoverAttachedRunner(selected.execution, fresh); + return; + } + + const resume = { kind: "resume", identity: f.identity, bootstrapTicket: "fixture-ticket" } as const; + f.workers.call.mockResolvedValueOnce({ kind: "resume", taskId: "replacement-task", outcome: "resumed" }); + await expect(f.task(resume)).resolves.toMatchObject({ kind: "resume", outcome: "resumed" }); + const [dispatched] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + expect(dispatched.metadata?.runnerTaskDispatched).toBe(true); + + const readState = { kind: "read_state", identity: f.identity } as const; + await expect(f.task({ ...readState, identity: { ...f.identity, itemId: "wrong-item" } })) + .rejects.toThrow("ownership_unverified"); + f.workers.call.mockResolvedValueOnce({ kind: "state", taskId: "replacement-task", state: { ...f.identity } }); + await expect(f.task(readState)).resolves.toMatchObject({ kind: "state", state: f.identity }); + const nextIdentity = { ...f.identity, runId: f.runId, turnId: "target-turn", itemId: "target-item" }; + const attach = { ...f.operation, resumeFrom: undefined, priorIdentity: f.identity, + requirements: { ...f.operation.requirements, identity: nextIdentity } }; + await expect(f.task({ ...attach, requirements: { ...attach.requirements, + identity: { ...nextIdentity, normalizedSessionId: randomUUID() } } })).rejects.toThrow("prior identity mismatch"); + await expect(f.task(attach)).resolves.toMatchObject({ kind: "prepared" }); + await expect(f.task(attach)).resolves.toMatchObject({ kind: "prepared" }); + const [attached] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)); + expect(attached.metadata?.runnerTaskIdentity).toMatchObject({ runId: f.runId, sessionId: f.identity.normalizedSessionId }); + expect(attached.metadata?.runnerTaskPriorIdentity).toMatchObject({ runId: f.sourceRunId, itemId: f.identity.itemId }); + const scope = { attachedRunId: f.runId, accountIdentity: f.accountIdentity }; + await expect(readyPortableTaskSource(db, f.companyId, f.leaseId, nextIdentity, scope)).resolves.toBeUndefined(); + await expect(readyPortableTaskSource(db, f.companyId, f.leaseId, { ...nextIdentity, turnId: "wrong-turn" }, scope)) + .rejects.toThrow("ownership_unverified"); + await expect(readyPortableTaskSource(db, f.companyId, f.leaseId, f.identity, scope)) + .resolves.toEqual({ taskId: "source-task", identity: f.identity }); + await expect(readyPortableTaskSource(db, f.companyId, f.leaseId, f.identity, { ...scope, accountIdentity: "another-account" })) + .rejects.toThrow("ownership_unverified"); + + const binding = { companyId: f.companyId, leaseId: f.leaseId, taskId: "replacement-task", runId: f.runId, identity: nextIdentity }; + await expect(assertAttachedPortableTask(restarted, binding)).resolves.toBeUndefined(); + for (const invalid of [ + { ...binding, leaseId: f.sourceLeaseId }, + { ...binding, taskId: "source-task" }, + { ...binding, identity: f.identity }, + { ...binding, identity: { ...nextIdentity, environmentLeaseId: f.leaseId } }, + ]) await expect(assertAttachedPortableTask(restarted, invalid)).rejects.toThrow("ownership_unverified"); + + if (policy === "ordinary") { + const recover = await recoverAttachedRunner(f.execution, nextIdentity); + await db.update(environmentLeases).set({ + metadata: sql`${environmentLeases.metadata} - 'portableTaskSuccessor'`, + }).where(eq(environmentLeases.id, f.sourceLeaseId)); + f.workers.call.mockClear(); + await expect(recover()).rejects.toThrow("ownership_unverified"); + expect(f.workers.call).not.toHaveBeenCalled(); + } + }); + it.each(["codex", "claude"] as const)("authorizes the claimed %s source while preserving A's accepted outcome", async provider => { const f = await fixture(provider); const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)); diff --git a/server/src/services/native-runtime/remote-warm-session-recovery.ts b/server/src/services/native-runtime/remote-warm-session-recovery.ts index 99d3d4207ea..65c4bd151e9 100644 --- a/server/src/services/native-runtime/remote-warm-session-recovery.ts +++ b/server/src/services/native-runtime/remote-warm-session-recovery.ts @@ -5,6 +5,7 @@ import { parseNativeExecutionInput, type NativeExecutionInput, type PersistedNat import { currentNativeControllerIdentity, evaluateNativeControllerTakeover, recordLegacyHostFallbackProof } from "./native-restart-recovery.js"; import { NativeControllerRecoveryWaitingError, readNativeControllerRecovery } from "./native-controller-recovery.js"; import { NativePredecessorRetirementUnprovenError } from "./predecessor-retirement.js"; +import { readyPortableTaskSource } from "../plugin-task-ownership.js"; export type RemoteWarmSessionRecovery = { identity: Record; @@ -12,6 +13,7 @@ export type RemoteWarmSessionRecovery = { previousExecution: NativeExecutionInput; reuseProcess: boolean; releasedSource?: DurableRecoveryIdentity; + portableSourceTaskId?: string; remote: { providerLeaseId: string; remoteCwd: string }; revalidate: (leaseOwner: string) => Promise; }; @@ -128,6 +130,7 @@ export async function inspectRemoteWarmSessionRecovery(input: { matchesExecution: (previous: NativeExecutionInput) => boolean; loadCheckpoint: (previous: NativeExecutionInput) => PersistedNativeSession | null; canReuseProcess: (previous: NativeExecutionInput) => boolean; + portableSourceTaskId?: string; }): Promise { const { binding } = input.execution; const { target } = input; @@ -152,13 +155,15 @@ export async function inspectRemoteWarmSessionRecovery(input: { // sandbox (for example after a restart or deploy replaced the controller) // holds no Runner to adopt, so live adoption steps aside. Ordinary admission // then retires the finished owner or keeps it held with a named reason. - if (target.transport === "sandbox" && ["created", "replacement"].includes(target.sandboxLeaseAcquisition?.outcome ?? "")) { + if (!input.portableSourceTaskId && target.transport === "sandbox" && ["created", "replacement"].includes(target.sandboxLeaseAcquisition?.outcome ?? "")) { return null; } if (target.transport !== "sandbox" || !target.leaseId || !target.environmentId || - !target.providerKey || !target.sandboxLeaseAcquisition || target.sandboxLeaseAcquisition.outcome !== "resumed" || + !target.providerKey || !target.sandboxLeaseAcquisition || + (!input.portableSourceTaskId && target.sandboxLeaseAcquisition.outcome !== "resumed") || identity.runId === binding.runId) return deny("target_not_resumed_sandbox"); const providerLeaseId = target.sandboxLeaseAcquisition.providerLeaseId; + const sourceProviderLeaseId = input.portableSourceTaskId ?? providerLeaseId; const [prior] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations }) .from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) .where(and(eq(heartbeatRuns.id, identity.runId), eq(heartbeatRuns.companyId, binding.companyId), @@ -171,7 +176,7 @@ export async function inspectRemoteWarmSessionRecovery(input: { throw new NativeControllerRecoveryWaitingError(prior.owner.leaseExpiresAt); } if (prior.owner.phase !== "committed" || prior.owner.leaseOwner) return deny("owner_unsettled"); - settledPredecessor = { runId: identity.runId, providerLeaseId }; + if (sourceProviderLeaseId === providerLeaseId) settledPredecessor = { runId: identity.runId, providerLeaseId }; const checkpoint = input.loadCheckpoint(previous); if (!checkpoint || checkpoint.identity.runId !== identity.runId || checkpoint.identity.companyId !== binding.companyId || checkpoint.identity.issueId !== binding.issueId || @@ -194,15 +199,22 @@ export async function inspectRemoteWarmSessionRecovery(input: { const priorLeases = await input.db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, binding.companyId), eq(environmentLeases.heartbeatRunId, identity.runId), eq(environmentLeases.issueId, binding.issueId), eq(environmentLeases.environmentId, target.environmentId), - eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, providerLeaseId), + eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, sourceProviderLeaseId), )); if (priorLeases.length !== 1 || priorLeases[0].metadata?.remoteCwd !== target.remoteCwd || - !["released", "expired", "active"].includes(priorLeases[0].status)) return deny("prior_lease_mismatch"); + !["released", "expired", "active", ...(input.portableSourceTaskId ? ["retained"] : [])].includes(priorLeases[0].status)) { + return deny("prior_lease_mismatch"); + } return { identity, checkpoint, previousExecution: previous, reuseProcess: input.canReuseProcess(previous), + portableSourceTaskId: input.portableSourceTaskId, remote: { providerLeaseId, remoteCwd: target.remoteCwd }, revalidate: async leaseOwner => { await assertPriorStopped(); + if (input.portableSourceTaskId && + (await readyPortableTaskSource(input.db, binding.companyId, target.leaseId!, identity))?.taskId !== input.portableSourceTaskId) { + return deny("prior_lease_mismatch"); + } const controller = await currentNativeControllerIdentity(); const [current] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations, issue: issues, agent: agents, company: companies }) .from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) diff --git a/server/src/services/plugin-task-credential-writeback.ts b/server/src/services/plugin-task-credential-writeback.ts index bcc01ed56ce..a431f307d86 100644 --- a/server/src/services/plugin-task-credential-writeback.ts +++ b/server/src/services/plugin-task-credential-writeback.ts @@ -13,6 +13,12 @@ const handoffSchema = z.object({ const key = "runnerTaskCredentialHandoff"; type Scope = { companyId: string; leaseId: string; expectedTaskId?: string }; +export function canReadStoppedPluginTaskCredential(metadata: Record, accountIdentity: string): boolean { + const handoff = handoffSchema.safeParse(metadata[key]); + return metadata.runnerTaskExecutionStopped === true && handoff.success && + handoff.data.state === "pending" && handoff.data.accountIdentity === accountIdentity; +} + /** Save authority, never credential bytes or a controller-local temporary path. */ export async function registerPluginTaskCredentialHandoff( db: Db, diff --git a/server/src/services/plugin-task-ownership.ts b/server/src/services/plugin-task-ownership.ts new file mode 100644 index 00000000000..c7f86b60217 --- /dev/null +++ b/server/src/services/plugin-task-ownership.ts @@ -0,0 +1,319 @@ +import { setTimeout as delay } from "node:timers/promises"; +import { and, asc, eq, inArray, isNotNull, isNull, or, sql } from "drizzle-orm"; +import { environmentLeases, heartbeatRuns, issues, type Db } from "@paperclipai/db"; +import { environmentTaskOperationSchema, environmentTaskResultSchema, type PluginEnvironmentTaskOperation, type PluginEnvironmentTaskResult } from "@paperclipai/plugin-sdk"; +import { nativeSha256 } from "./native-runtime/canonical.js"; +import { lockSourceRecoveryCoordinators } from "./native-runtime/native-source-recovery-ownership.js"; +import type { PluginTaskDescription } from "./environment-task-runtime.js"; + +type OwnershipOperation = Extract; +type OwnershipResult = Extract; +type Lease = typeof environmentLeases.$inferSelect; +type OwnershipClaim = { + sourceLeaseId: string; + targetLeaseId: string; + operation: OwnershipOperation; +}; + +export function unlaunchedPortableTaskLease() { + const { metadata } = environmentLeases; + const unlaunched = and( + eq(environmentLeases.status, "active"), isNull(environmentLeases.releasedAt), isNull(environmentLeases.cleanupStatus), + isNotNull(environmentLeases.providerLeaseId), + or(isNull(environmentLeases.expiresAt), sql`${environmentLeases.expiresAt} > clock_timestamp()`), + sql`${metadata}->>'driver' = 'plugin' and ${metadata}->>'runnerTaskLease' = 'true' + and jsonb_typeof(${metadata}->'portableTaskPreviousLeaseId') = 'string' + and not (${metadata} ?| array['portableTaskSuccessor', 'runnerTaskIdentity', 'runnerTaskStartup', 'runnerTaskStartupCleanup']) + and coalesce(${metadata}->>'runnerTaskDispatched', 'false') = 'false' + and coalesce(${metadata}->>'runnerTaskRetiring', 'false') = 'false'`, + )!; + return sql`case when ${unlaunched} then not exists (select 1 from environment_leases other + where other.company_id = ${environmentLeases.companyId} and other.heartbeat_run_id = ${environmentLeases.heartbeatRunId} + and other.status = 'active' and other.id <> ${environmentLeases.id}) else false end`; +} + +export function portableTaskPreparationLeaseId(db: Db, run: Pick) { + return sql`(${db.select({ id: environmentLeases.id }).from(issues).innerJoin(environmentLeases, and( + eq(environmentLeases.companyId, issues.companyId), eq(environmentLeases.issueId, issues.id), + eq(environmentLeases.heartbeatRunId, issues.executionRunId), + )).where(and(eq(issues.companyId, run.companyId), run.issueId ? eq(issues.id, run.issueId) : sql`false`, + eq(issues.executionRunId, run.id), eq(issues.assigneeAgentId, run.agentId), eq(issues.status, "in_progress"), + sql`${environmentLeases.metadata}->>'agentId' = ${run.agentId}`, unlaunchedPortableTaskLease()))})`; +} + +const object = (value: unknown): Record => + value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; + +export class PluginTaskOwnershipPendingError extends Error { + constructor() { + super("native_portable_task_ownership_pending"); + } +} + +function unavailable(): never { + throw new Error("native_portable_task_ownership_unverified"); +} + +function leaseGeneration(lease: Lease) { + return object(lease.metadata?.providerMetadata).ownershipGeneration; +} + +function readOperation(source: Lease, target: Lease, policy: OwnershipOperation["policy"]): OwnershipOperation { + const identity = object(source.metadata?.runnerTaskIdentity); + const operation = environmentTaskOperationSchema.safeParse({ + kind: "ensure_ownership", operationId: target.id, sourceTaskId: source.providerLeaseId, + sourceGeneration: leaseGeneration(source), targetGeneration: leaseGeneration(target), policy, + identity: { + runnerInstanceId: identity.runnerId, environmentLeaseId: identity.leaseId, + runId: identity.runId, normalizedSessionId: identity.sessionId, + turnId: identity.turnId, itemId: identity.itemId, + }, + }); + return operation.success && operation.data.kind === "ensure_ownership" ? operation.data : unavailable(); +} + +async function ownershipClaim(db: Db, input: { + companyId: string; + sourceLeaseId: string; + targetLeaseId: string; + policy: OwnershipOperation["policy"]; +}, ready?: OwnershipResult): Promise { + const { companyId, sourceLeaseId, targetLeaseId, policy } = input; + if (sourceLeaseId === targetLeaseId) return unavailable(); + return db.transaction(async tx => { + const scope = and(eq(environmentLeases.companyId, companyId), inArray(environmentLeases.id, [sourceLeaseId, targetLeaseId])); + const initial = await tx.select().from(environmentLeases).where(scope); + const sourceRunId = initial.find(lease => lease.id === sourceLeaseId)?.heartbeatRunId; + const targetRunId = initial.find(lease => lease.id === targetLeaseId)?.heartbeatRunId; + const issueId = initial.find(lease => lease.id === sourceLeaseId)?.issueId; + if (!sourceRunId || !targetRunId || sourceRunId === targetRunId || !issueId) return unavailable(); + const owners = await lockSourceRecoveryCoordinators(tx, companyId, [sourceRunId, targetRunId]); + const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, companyId), eq(issues.id, issueId))).for("update", { noWait: true }); + const leases = await tx.select().from(environmentLeases).where(scope).orderBy(asc(environmentLeases.id)).for("update"); + const source = leases.find(lease => lease.id === sourceLeaseId); + const target = leases.find(lease => lease.id === targetLeaseId); + const runs = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), + inArray(heartbeatRuns.id, [sourceRunId, targetRunId]))); + const prior = runs.find(run => run.id === sourceRunId); + const current = runs.find(run => run.id === targetRunId); + const sourceOwner = owners.find(owner => owner.runId === sourceRunId); + if (!source || !target || source.heartbeatRunId !== sourceRunId || target.heartbeatRunId !== targetRunId || + !source.provider?.startsWith("plugin:") || source.provider !== target.provider || + !source.providerLeaseId || !target.providerLeaseId || source.providerLeaseId === target.providerLeaseId || + !source.environmentId || source.environmentId !== target.environmentId || + source.issueId !== issueId || target.issueId !== issueId || source.executionWorkspaceId !== target.executionWorkspaceId || + source.metadata?.driver !== "plugin" || target.metadata?.driver !== "plugin" || + typeof source.metadata.pluginId !== "string" || source.metadata.pluginId !== target.metadata.pluginId || + typeof source.metadata.driverKey !== "string" || source.metadata.driverKey !== target.metadata.driverKey || + source.metadata.runnerTaskLease !== true || target.metadata.runnerTaskLease !== true || + source.metadata.runnerTaskRetiring === true || source.metadata.reusableLeaseSuccessor !== undefined || + source.metadata.runnerTaskSourceConsumer !== undefined || + !["retained", "released", "expired"].includes(source.status) || + target.status !== "active" || target.releasedAt || target.metadata.runnerTaskRetiring === true || + target.expiresAt && target.expiresAt.getTime() <= Date.now() || + prior?.status !== "succeeded" || prior.nativeIssueId !== issueId || + sourceOwner?.phase !== "committed" || !sourceOwner.resultId || sourceOwner.leaseOwner || + current?.status !== "running" || current.scopeKind !== "issue" || current.issueId !== issueId || + current.nativeIssueId !== null && current.nativeIssueId !== issueId || current.resultJson?.nativeCancellation || + prior.agentId !== current.agentId || issue?.executionRunId !== targetRunId || issue.assigneeAgentId !== current.agentId || + issue.status !== "in_progress") return unavailable(); + + const operation = readOperation(source, target, policy); + if (operation.identity.runId !== prior.id || operation.identity.normalizedSessionId !== prior.nativeSessionId || + current.nativeSessionId !== null && current.nativeSessionId !== prior.nativeSessionId) return unavailable(); + const claim = { sourceLeaseId, targetLeaseId, operation }; + const existing = [source.metadata.portableTaskSuccessor, target.metadata.portableTaskPredecessor]; + if (existing.some(saved => saved !== undefined && nativeSha256(saved) !== nativeSha256(claim)) || + (existing[0] === undefined) !== (existing[1] === undefined)) return unavailable(); + if (ready && (existing[0] === undefined || ready.taskId !== target.providerLeaseId || + ready.operationId !== operation.operationId || ready.targetGeneration !== operation.targetGeneration || + ready.status !== "ready")) return unavailable(); + if (!ready && existing[0] === undefined) { + await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ portableTaskSuccessor: claim })}::jsonb` }) + .where(eq(environmentLeases.id, sourceLeaseId)); + await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ portableTaskPredecessor: claim })}::jsonb` }) + .where(eq(environmentLeases.id, targetLeaseId)); + } else if (ready && target.metadata.portableTaskOwnershipReady === true && + (target.metadata.portableTaskCheckpointUnavailable === true) !== (ready.checkpointUnavailable === true)) { + return unavailable(); + } else if (ready && target.metadata.portableTaskOwnershipReady !== true) { + await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ + portableTaskOwnershipReady: true, ...(ready.checkpointUnavailable ? { portableTaskCheckpointUnavailable: true } + : { runnerTaskAccountIdentity: source.metadata.runnerTaskAccountIdentity }), + })}::jsonb` }) + .where(eq(environmentLeases.id, targetLeaseId)); + } + return claim; + }); +} + +export async function reconcilePluginTaskOwnership(db: Db, input: Parameters[1], + dispatch: (operation: OwnershipOperation) => Promise): Promise { + try { + const claim = await ownershipClaim(db, input); + const result = await dispatch(claim.operation); + if (result.kind !== "ownership" || result.operationId !== claim.operation.operationId || + result.targetGeneration !== claim.operation.targetGeneration) return unavailable(); + if (result.status === "ready") await ownershipClaim(db, input, result); + return result; + } catch (error) { + if ((object(object(error).cause).code ?? object(error).code) === "55P03") throw new PluginTaskOwnershipPendingError(); + throw error; + } +} + +export function assertPluginTaskOwnershipDispatch(lease: Lease, operation: OwnershipOperation) { + const claim = object(lease.metadata?.portableTaskPredecessor); + if (claim.targetLeaseId !== lease.id || claim.operation === undefined || + operation.targetGeneration !== leaseGeneration(lease) || + nativeSha256(claim.operation) !== nativeSha256(operation)) return unavailable(); +} + +/** Reconcile the declared predecessor before admitting the task on this lease. */ +export async function reconcilePluginTaskAdmission(db: Db, lease: Lease, + dispatch: (operation: OwnershipOperation) => Promise, signal?: AbortSignal): Promise { + let sourceLeaseId = lease.metadata?.portableTaskPreviousLeaseId; + let policy: OwnershipOperation["policy"] = "ordinary"; + if (lease.metadata?.portableTaskPredecessor !== undefined) { + const claim = object(lease.metadata.portableTaskPredecessor); + const operation = environmentTaskOperationSchema.safeParse(claim.operation); + if (!operation.success || operation.data.kind !== "ensure_ownership") return unavailable(); + assertPluginTaskOwnershipDispatch(lease, operation.data); + sourceLeaseId = claim.sourceLeaseId; + policy = operation.data.policy; + } + if (typeof sourceLeaseId !== "string") return unavailable(); + const request = { companyId: lease.companyId, sourceLeaseId, targetLeaseId: lease.id, policy }; + let backoffMs = 250; + for (;;) { + signal?.throwIfAborted(); + try { + const result = await reconcilePluginTaskOwnership(db, request, dispatch); + if (result.status === "ready") { + signal?.throwIfAborted(); + return; + } + } catch (error) { + if (!(error instanceof PluginTaskOwnershipPendingError)) throw error; + } + if (!signal) throw new PluginTaskOwnershipPendingError(); + await delay(backoffMs, undefined, { signal }); + backoffMs = Math.min(backoffMs * 2, 5_000); + } +} + +export async function readPortableTaskReseed(db: Db, input: { + companyId: string; + runId: string; + lease: Pick; + description?: PluginTaskDescription; + runnerTask: (operation: { kind: "describe" }) => Promise; +}): Promise<{ description: PluginTaskDescription; reseed: { previousSessionId: string | null } | null } | null> { + if (!input.lease.metadata?.portableTaskPreviousLeaseId) return null; + const readCurrent = async () => (await db.select({ + metadata: environmentLeases.metadata, providerLeaseId: environmentLeases.providerLeaseId, + }).from(environmentLeases).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.companyId), + eq(environmentLeases.heartbeatRunId, input.runId))))[0]; + + let description = input.description; + let current = description ? await readCurrent() : undefined; + if (!description || current?.providerLeaseId !== description.taskId || current.metadata?.portableTaskOwnershipReady !== true) { + const result = environmentTaskResultSchema.parse(await input.runnerTask({ kind: "describe" })); + if (result.kind !== "description") throw new Error("Plugin Runner description unavailable"); + description = result; + current = await readCurrent(); + } + if (current?.providerLeaseId !== description.taskId || current.metadata?.portableTaskOwnershipReady !== true) return unavailable(); + if (current.metadata.portableTaskCheckpointUnavailable !== true) return { description, reseed: null }; + + const predecessor = object(current.metadata.portableTaskPredecessor); + const identity = object(object(predecessor.operation).identity); + const sessionId = identity.normalizedSessionId; + return { description, reseed: { previousSessionId: typeof sessionId === "string" && sessionId.trim() ? sessionId : null } }; +} + +async function readReadyPortableTaskClaim(db: Db, companyId: string, targetLeaseId: string) { + const [target] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId), + eq(environmentLeases.id, targetLeaseId))); + if (!target?.metadata?.portableTaskPredecessor) return undefined; + if (target.metadata.portableTaskCheckpointUnavailable === true) return undefined; + const claim = object(target.metadata.portableTaskPredecessor); + const operation = environmentTaskOperationSchema.safeParse(claim.operation); + if (target.status !== "active" || target.releasedAt || target.metadata.portableTaskOwnershipReady !== true || + target.metadata.portableTaskSuccessor !== undefined || target.metadata.runnerTaskRetiring === true || + target.expiresAt && target.expiresAt.getTime() <= Date.now() || + !operation.success || operation.data.kind !== "ensure_ownership" || + typeof claim.sourceLeaseId !== "string") return unavailable(); + assertPluginTaskOwnershipDispatch(target, operation.data); + const [source] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId), + eq(environmentLeases.id, claim.sourceLeaseId))); + if (!source || source.provider !== target.provider || source.providerLeaseId !== operation.data.sourceTaskId || + leaseGeneration(source) !== operation.data.sourceGeneration || + nativeSha256(source.metadata?.portableTaskSuccessor ?? null) !== nativeSha256(claim)) return unavailable(); + return { source, target, operation: operation.data }; +} + +function matchesAttachedPortableTaskIdentity( + target: Lease, identity: Record, runId: string, +) { + const saved = object(target.metadata?.runnerTaskIdentity); + const attached = { + runnerInstanceId: saved.runnerId, environmentLeaseId: saved.leaseId, + runId: saved.runId, normalizedSessionId: saved.sessionId, turnId: saved.turnId, itemId: saved.itemId, + }; + return target.heartbeatRunId === runId && identity.runId === runId && + Object.entries(attached).every(([key, value]) => typeof value === "string" && value.length > 0 && identity[key] === value); +} + +function matchesPortableLineage(source: OwnershipOperation["identity"], identity: Record) { + return source.environmentLeaseId === identity.environmentLeaseId && source.runnerInstanceId === identity.runnerInstanceId && + source.normalizedSessionId === identity.normalizedSessionId; +} + +export async function assertAttachedPortableTask(db: Db, input: { + companyId: string; leaseId: string; taskId: string; runId: string; identity: Record; +}) { + const claim = await readReadyPortableTaskClaim(db, input.companyId, input.leaseId); + if (!claim || claim.target.providerLeaseId !== input.taskId || + !matchesAttachedPortableTaskIdentity(claim.target, input.identity, input.runId) || + !matchesPortableLineage(claim.operation.identity, input.identity)) return unavailable(); +} + +export async function readyPortableTaskSource(db: Db, companyId: string, targetLeaseId: string, identity: unknown, + scope: { attachedRunId?: string; selectedSessionId?: string; accountIdentity?: string } = {}) { + const claim = await readReadyPortableTaskClaim(db, companyId, targetLeaseId); + if (!claim) return undefined; + const { source, target, operation } = claim; + if ("accountIdentity" in scope && (!scope.accountIdentity || source.metadata?.runnerTaskAccountIdentity !== scope.accountIdentity)) + return unavailable(); + const candidate = object(identity); + const { attachedRunId, selectedSessionId } = scope; + if (selectedSessionId && selectedSessionId !== operation.identity.normalizedSessionId) { + if (!attachedRunId || target.heartbeatRunId !== attachedRunId) return unavailable(); + const [admitted] = await db.select({ lease: environmentLeases, unlaunched: unlaunchedPortableTaskLease() }) + .from(environmentLeases).innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), + eq(heartbeatRuns.companyId, environmentLeases.companyId))) + .innerJoin(issues, and(eq(issues.id, environmentLeases.issueId), eq(issues.companyId, environmentLeases.companyId))) + .where(and(eq(environmentLeases.id, targetLeaseId), eq(environmentLeases.companyId, companyId), + eq(environmentLeases.providerLeaseId, target.providerLeaseId!), + sql`${environmentLeases.metadata}->'portableTaskPredecessor' = ${JSON.stringify(target.metadata?.portableTaskPredecessor)}::jsonb`, + eq(heartbeatRuns.id, attachedRunId), eq(heartbeatRuns.status, "running"), + sql`${heartbeatRuns.resultJson}->'nativeCancellation' is null`, eq(heartbeatRuns.nativeSessionId, selectedSessionId), + sql`${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'session'->>'normalizedSessionId' = ${selectedSessionId}`, + eq(heartbeatRuns.nativeIssueId, issues.id), eq(issues.executionRunId, heartbeatRuns.id), + eq(issues.assigneeAgentId, heartbeatRuns.agentId), eq(issues.status, "in_progress"))); + if (!admitted) return unavailable(); + if (identity == null && admitted.unlaunched) return undefined; + if (admitted.lease.metadata?.runnerTaskDispatched === true && + candidate.normalizedSessionId === selectedSessionId && candidate.environmentLeaseId === target.id && + matchesAttachedPortableTaskIdentity(admitted.lease, candidate, attachedRunId)) return undefined; + return unavailable(); + } + if (attachedRunId && target.heartbeatRunId === attachedRunId && candidate.runId === attachedRunId) { + if (matchesAttachedPortableTaskIdentity(target, candidate, attachedRunId) && + matchesPortableLineage(operation.identity, candidate)) return undefined; + return unavailable(); + } + if (Object.entries(operation.identity).some(([key, value]) => candidate[key] !== value)) return unavailable(); + return { taskId: operation.sourceTaskId, identity: operation.identity }; +} diff --git a/server/src/services/plugin-task-resume.ts b/server/src/services/plugin-task-resume.ts index d5326f3d0dd..716a5e7ba21 100644 --- a/server/src/services/plugin-task-resume.ts +++ b/server/src/services/plugin-task-resume.ts @@ -19,6 +19,7 @@ export async function resolvePluginTaskResume(db: Db, input: { const identity = row.lease.metadata?.runnerTaskIdentity as Record | undefined; const consumer = row.lease.metadata?.runnerTaskSourceConsumer as { leaseId?: unknown; runId?: unknown; resumeFrom?: unknown } | null | undefined; return row.lease.id !== input.leaseId && row.run.id === input.priorIdentity.runId && + row.lease.metadata?.portableTaskSuccessor === undefined && (!Object.hasOwn(row.lease.metadata ?? {}, "runnerTaskSourceConsumer") || consumer?.leaseId === input.leaseId && consumer.runId === next?.run.id && consumer.resumeFrom === row.lease.providerLeaseId) && (input.inspectCheckpoint || row.lease.metadata?.runnerTaskSessionRetained === true) && diff --git a/tests/runner-e2e/source-sdk-public-inputs.json b/tests/runner-e2e/source-sdk-public-inputs.json index a850e464474..cc3eda6c434 100644 --- a/tests/runner-e2e/source-sdk-public-inputs.json +++ b/tests/runner-e2e/source-sdk-public-inputs.json @@ -13,7 +13,7 @@ "packages/plugins/sdk/src/environment-stop-error.ts": "b0e03f8aa2b2170ba095da5c7138050fced3f3cb", "packages/plugins/sdk/src/environment-sync-error.ts": "7d6a51a95da49a7da4525aebbab6382dc5c44ae8", "packages/plugins/sdk/src/environment-task-error.ts": "98aae8a06743b38884b68cb9a3ec44c57b34c0e4", - "packages/plugins/sdk/src/environment-tasks.ts": "98e5f69a00b81eb6e8c25bdb5fcdc22e5485ef18", + "packages/plugins/sdk/src/environment-tasks.ts": "de583d809187f0677e25b95f22c3c42961a9d2e2", "packages/plugins/sdk/src/host-client-factory.ts": "e0301c390ea13f2340b669383f07af2455e48c8a", "packages/plugins/sdk/src/idle-drain.test.ts": "cc59f7881e7d46c89395a1d93d74f6c6b9b0a64a", "packages/plugins/sdk/src/idle-drain.ts": "ae6e4c6be4657a3d541396922b3bc551b8db557f", @@ -408,7 +408,7 @@ "packages/shared/src/types/issue.ts": "725bd6f677b9d461460697d1ce10dee739a66ebe", "packages/shared/src/types/live.ts": "b90e55e4de8600c6fc17d7fa8d89079d4ae655ec", "packages/shared/src/types/native-finalization.ts": "c026a73003ae232fea4f7624d561493b383df358", - "packages/shared/src/types/plugin.ts": "016059b873c7189b8e65bb0dfa1ffa039c97c2ca", + "packages/shared/src/types/plugin.ts": "22e1e35307f91fdf5c210b86bae7746594669516", "packages/shared/src/types/project.ts": "78eeea3694ce860d5d0be59c158c399c03368078", "packages/shared/src/types/quota.ts": "a0686a9afb3111a4de093f66200dc0c9152e883e", "packages/shared/src/types/resource-memberships.ts": "9eab6fad50facf3e489f8c40e5bedd92e1d1faf8", @@ -490,7 +490,7 @@ "packages/shared/src/validators/onboarding-seed.ts": "de302a8400d4199053f379d9ed853522a3ac6d32", "packages/shared/src/validators/partial.ts": "fa60ec7481fc14175365e9e500c814b159f1ab53", "packages/shared/src/validators/plugin.test.ts": "23bc443850d46d815877a71a44625fd42aebb00a", - "packages/shared/src/validators/plugin.ts": "2a90d3d00b64f802b9c30f957e97702250ab2db4", + "packages/shared/src/validators/plugin.ts": "782e051567729c2eb91a3b80e883b4a8ed996646", "packages/shared/src/validators/project.ts": "6ea168f5faf3161e6004ea47274f60634c77b219", "packages/shared/src/validators/provider-trace.ts": "b38b6200a2e51513f5dca0f8b008f2fe68175e82", "packages/shared/src/validators/resource-memberships.ts": "b124ac8a8f24b4dbb03ae82bcc87541f6b4ecdfd",