Persist portable Runner ownership through controller recovery (#21066)

## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed 2026-11-12 10:43:48 -08:00
1 parent a4a0668a5c
commit a924d2e2c7
40 files changed
+1585 -223

No files matched your search

+9
View File
@@ -172,6 +172,15 @@ It does not qualify a deployed plugin, live account, or production WSS proxy.
## Workspace completion and maintenance
A portable restore uses the durable ownership claim and the same account
identity after a credential refresh or controller replacement. It starts a
new Runner on the target. Run attachment applies the current tools and
runtime configuration to the restored provider session. After a confirmed
workspace transfer, Core can use the same target to start a new conversation
when its session compatibility check requires one. The new conversation
starts without the prior provider checkpoint, and the ownership claim
continues to fence the source.
A storage provider can declare `workspaceProvider.workspaceVerificationVersion: 1`.
Before creating a fresh workspace for an admitted Retry, Core may send the
original task-owned workspace's `prepare` request with `verifyExisting: true`.
+32 -4
View File
@@ -25,12 +25,13 @@ const githubBroker = z.object({ url: z.string().url().max(2000).refine(value =>
return ["http:", "https:"].includes(url.protocol) && !url.username && !url.password && !url.search && !url.hash;
} catch { return false; }
}, "Expected a credential-free HTTP broker URL"), token: credentialValue }).strict();
const claudeCredentialEnvironment = z.object({
CLAUDE_CODE_OAUTH_TOKEN: credentialValue.optional(), ANTHROPIC_API_KEY: credentialValue.optional(),
}).strict().refine(env => Object.values(env).filter(Boolean).length === 1, "Select one Claude credential");
/** Only provider authentication is transferred. Paths and executable settings remain provider-owned. */
export const environmentTaskCredentialSchema = z.discriminatedUnion("type", [
z.object({ type: z.literal("codex-auth-json"), authJson: credentialValue }).strict(),
z.object({ type: z.literal("claude-env"), environment: z.object({
CLAUDE_CODE_OAUTH_TOKEN: credentialValue.optional(), ANTHROPIC_API_KEY: credentialValue.optional(),
}).strict().refine(env => Object.values(env).filter(Boolean).length === 1, "Select one Claude credential") }).strict(),
z.object({ type: z.literal("claude-env"), environment: claudeCredentialEnvironment }).strict(),
]);
export type PluginEnvironmentTaskCredential = z.infer<typeof environmentTaskCredentialSchema>;
@@ -114,7 +115,18 @@ export const environmentTaskOperationSchema = z.discriminatedUnion("kind", [
z.object({ kind: z.literal("status") }).strict(),
z.object({ kind: z.literal("connection") }).strict(),
z.object({ kind: z.literal("release"), identity: durableIdentity }).strict(),
z.object({ kind: z.literal("resume"), identity: durableIdentity, bootstrapTicket: credentialValue }).strict(),
z.object({ kind: z.literal("resume"), identity: durableIdentity, bootstrapTicket: credentialValue,
handoffCredential: z.object({ accountIdentity: z.string().min(1).max(1024), environment: claudeCredentialEnvironment }).strict().optional(),
}).strict(),
z.object({
kind: z.literal("ensure_ownership"),
operationId: z.string().uuid(),
identity: durableIdentity,
sourceTaskId: providerId.max(1024),
sourceGeneration: providerId.max(1024),
targetGeneration: providerId.max(1024),
policy: z.enum(["ordinary", "reclaim_if_needed"]),
}).strict(),
z.object({ kind: z.literal("complete") }).strict(),
z.object({ kind: z.literal("stop") }).strict(),
]);
@@ -125,6 +137,14 @@ export const environmentTaskResultSchema = z.discriminatedUnion("kind", [
reason: z.enum(["session_state_unavailable", "account_binding_mismatch"]) }).strict(),
z.object({ kind: z.literal("released"), taskId: providerId }).strict(),
z.object({ kind: z.literal("resume"), taskId: providerId, outcome: z.enum(["resumed", "unavailable"]) }).strict(),
z.object({
kind: z.literal("ownership"),
taskId: providerId,
operationId: z.string().uuid(),
targetGeneration: providerId.max(1024),
status: z.enum(["pending", "ready"]),
checkpointUnavailable: z.literal(true).optional(),
}).strict(),
z.object({ kind: z.literal("copilot_metadata"), taskId: providerId, metadata: z.record(z.string(), z.unknown()),
executionStopped: z.literal(true) }).strict(),
z.object({ kind: z.literal("project_auth"), taskId: providerId, configurationId: providerId.max(1024),
@@ -142,6 +162,7 @@ export const environmentTaskResultSchema = z.discriminatedUnion("kind", [
runner: z.object({ version: z.string().min(1), digest: z.string().regex(/^sha256:[a-f0-9]{64}$/) }).strict(),
warmTaskVersion: z.literal(1).optional(),
releaseResumeVersion: z.literal(1).optional(),
sessionHandoffVersion: z.literal(1).optional(),
launchPreparationVersion: z.literal(1).optional(),
executionContextVersion: z.literal(1).optional(),
/** Each version attests stopped-source recovery for its provider independently. */
@@ -211,6 +232,13 @@ export interface PluginEnvironmentTaskParams extends Omit<PluginEnvironmentDrive
/** An accepted stop is not proof of process termination. */
export function parseEnvironmentTaskResult(operation: PluginEnvironmentTaskOperation, taskId: string, value: unknown): PluginEnvironmentTaskResult {
const result = environmentTaskResultSchema.parse(value);
if (operation.kind === "ensure_ownership") {
if (result.kind !== "ownership" || result.taskId !== taskId || result.operationId !== operation.operationId ||
result.targetGeneration !== operation.targetGeneration) {
throw new Error("Invalid plugin-provided session ownership response");
}
return result;
}
const expected = operation.kind === "release" ? "released" : operation.kind === "resume" ? "resume" : operation.kind === "read_file" ? "file" : operation.kind === "describe" ? "description" : operation.kind === "prepare" ? "prepared" : operation.kind === "check_project_auth" ? "project_auth" : ["read_state", "archive_state"].includes(operation.kind) ? "state" : ["status", "connection", "credentials", "copilot_metadata"].includes(operation.kind) ? operation.kind : "accepted";
const rejectedSubmit = operation.kind === "submit" && result.kind === "admission_rejected";
if (result.taskId !== taskId || (result.kind !== expected && !rejectedSubmit)) throw new Error("Invalid plugin-provided Runner execution response");
@@ -17,6 +17,10 @@ const identity = { runnerInstanceId: "runner", environmentLeaseId: "lease", runI
normalizedSessionId: "session", turnId: "turn", itemId: "item" };
const release = { kind: "release", identity } as const;
const resume = { kind: "resume", identity, bootstrapTicket: "fixture-ticket" } as const;
const ownership = {
kind: "ensure_ownership", operationId: "9df4526f-36c5-4fba-9cf6-3c4fd317f531", identity,
sourceTaskId: "source-task", sourceGeneration: "source-generation", targetGeneration: "target-generation", policy: "reclaim_if_needed",
} as const;
const description = { kind: "description", taskId: "task", runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } };
async function invoke(operation: unknown, handler: (params: PluginEnvironmentTaskParams) => Promise<PluginEnvironmentTaskResult>) {
@@ -89,6 +93,24 @@ describe("release and resume task contract", () => {
.toMatchObject({ result: { kind: "resume", taskId: "task", outcome: "unavailable" } });
});
it("binds a declarative ownership response to the exact worker request", async () => {
for (const status of ["pending", "ready"] as const) {
const result = { kind: "ownership", status, taskId: "task", operationId: ownership.operationId,
targetGeneration: ownership.targetGeneration } as const;
expect(await invoke(ownership, async params => {
expect(params.operation).toEqual(ownership);
return result;
})).toMatchObject({ result });
for (const changes of [
{ taskId: "another-task" },
{ operationId: "1f083ad0-a357-4f14-ad46-e9df981845b5" },
{ targetGeneration: "stale-generation" },
]) {
expect(await invoke(ownership, async () => ({ ...result, ...changes }))).toHaveProperty("error");
}
}
});
it.each(["source mismatch", "claim conflict", "authentication failed", "transport timeout"])("keeps %s as an RPC error", async message => {
expect(await invoke(resume, async () => { throw new Error(message); })).toHaveProperty("error");
});
+1
View File
@@ -201,6 +201,7 @@ export interface PluginEnvironmentDriverDeclaration {
creationOperationVersion?: 1;
/** Codex task retention and authorized run attachment, version 1. */
warmTaskVersion?: 1;
sessionHandoffVersion?: 1;
/** Stable driver key, unique within the plugin. Namespaced by plugin ID at runtime. */
driverKey: string;
/**
+1
View File
@@ -183,6 +183,7 @@ export const pluginEnvironmentDriverDeclarationSchema = z.object({
freshWorkspaceVersion: z.literal(1).optional(),
creationOperationVersion: z.literal(1).optional(),
warmTaskVersion: z.literal(1).optional(),
sessionHandoffVersion: z.literal(1).optional(),
driverKey: z.string().min(1).regex(
/^[a-z0-9][a-z0-9._-]*$/,
"Environment driver key must start with a lowercase alphanumeric and contain only lowercase letters, digits, dots, hyphens, or underscores",
@@ -33,6 +33,7 @@ import {
executionWorkspaces,
heartbeatRuns,
issues,
nativeRunFinalizations,
plugins,
projects,
} from "@paperclipai/db";
@@ -68,6 +69,9 @@ import { traceparentFromContextToken } from "../instrumentation.ts";
import { ROOT_CONTEXT, trace } from "@opentelemetry/api";
import { buildNativeHarnessBackupManifest } from "../services/native-runtime/native-session-executor.ts";
import { createNativeHarnessBackupStamp } from "../services/native-runtime/native-harness-backup-stamp.ts";
import { environmentRunOrchestrator } from "../services/environment-run-orchestrator.js";
import { claimNativeRestartRecoveries } from "../services/native-runtime/native-restart-recovery.js";
import { NATIVE_PREPARATION_FENCING_VERSION } from "../services/native-runtime/native-active-runs.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
@@ -1300,6 +1304,81 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
);
});
it.each(["handoff", "fresh", "legacy resume"] as const)("preserves the portable task placement on a run retry (%s)", async mode => {
const resumeSession = mode !== "fresh";
const pluginId = randomUUID();
const pluginKey = "paperclip.test-portable-tasks";
const seeded = await seedEnvironment({ driver: "plugin", config: { pluginKey, driverKey: "tasks", runnerLifecycleMode: "warm" } });
await insertSandboxProviderPlugin(pluginId, { pluginKey, environmentDrivers: [{
driverKey: "tasks", kind: "sandbox_provider", supportsTasks: true, warmTaskVersion: 1, sessionHandoffVersion: 1,
}] });
const issueId = randomUUID(), sessionId = randomUUID(), nextRunId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId: seeded.companyId, title: "Portable session", status: "in_progress" });
await db.insert(heartbeatRuns).values({ id: nextRunId, companyId: seeded.companyId, agentId: seeded.agentId, status: "running" });
let acquisitions = 0;
const call = vi.fn(async (_plugin: string, method: string) => {
if (method === "environmentAcquireLease") return { providerLeaseId: `sandbox-${++acquisitions}`,
metadata: { ownershipGeneration: `generation-${acquisitions}` } };
if (method === "environmentResumeLease") return { providerLeaseId: "sandbox-1", metadata: {} };
if (method === "environmentReleaseLease") return { providerLeaseId: "sandbox-1", state: "stopped" };
throw new Error(`Unexpected plugin method: ${method}`);
});
const runtime = environmentRuntimeService(db, { pluginWorkerManager: {
isRunning: () => true, call, getWorker: () => ({ supportedMethods: ["environmentResumeLease"] }),
} as unknown as PluginWorkerManager });
const input = { companyId: seeded.companyId, environment: seeded.environment, issueId, agentId: seeded.agentId,
persistedExecutionWorkspace: null, adapterType: "paperclip_runner" };
const orchestrator = environmentRunOrchestrator(db, { environmentRuntime: runtime });
const acquisitionInput = { ...input, selectedEnvironmentId: seeded.environment.id, localEnvironmentId: randomUUID(),
admittedLifecycleMode: "warm" as const, executionWorkspaceSettings: null };
const first = await orchestrator.acquireForRun({ ...acquisitionInput, heartbeatRunId: seeded.runId });
await db.update(environmentLeases).set({ status: "retained", metadata: {
...first.lease.metadata, runnerTaskIdentity: { sessionId },
...(mode === "legacy resume" ? {} : { runnerTaskAccountIdentity: "test-account" }),
} }).where(eq(environmentLeases.id, first.lease.id));
const nextInput = { ...acquisitionInput, heartbeatRunId: nextRunId, runnerTaskResumeSessionId: resumeSession ? sessionId : null };
const next = await orchestrator.acquireForRun(nextInput);
expect(next.lease.providerLeaseId).toBe(mode === "legacy resume" ? "sandbox-1" : "sandbox-2");
expect(next.lease.metadata?.portableTaskPreviousLeaseId).toBe(mode === "handoff" ? first.lease.id : undefined);
await expect(orchestrator.acquireForRun(nextInput)).resolves.toMatchObject({ lease: { id: next.lease.id, providerLeaseId: next.lease.providerLeaseId } });
if (mode === "handoff") {
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, seeded.agentId));
await db.update(issues).set({ assigneeAgentId: seeded.agentId, executionRunId: nextRunId }).where(eq(issues.id, issueId));
await db.update(heartbeatRuns).set({ scopeKind: "issue", issueId, nativeIssueId: issueId, runtimeMode: "native",
nativeSessionId: sessionId, runnerInstanceId: randomUUID(), runnerProfileJson: {
nativePreparationFencingVersion: NATIVE_PREPARATION_FENCING_VERSION,
sessionCheckpoint: { identity: { companyId: seeded.companyId, agentId: seeded.agentId, issueId, runId: nextRunId, sessionId },
providerSessionId: "inherited-provider-session" },
} }).where(eq(heartbeatRuns.id, nextRunId));
await db.insert(nativeRunFinalizations).values({ runId: nextRunId, companyId: seeded.companyId, issueId, phase: "observed" });
try {
const [claim] = await claimNativeRestartRecoveries({ db, restartKind: "hard", runIds: [nextRunId] });
expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: nextRunId, portablePreparationLeaseId: next.lease.id });
if (claim.kind !== "bootstrap_incomplete" || !claim.portablePreparationLeaseId) throw new Error("Expected portable bootstrap recovery");
const preparation = { kind: "plugin_task" as const, leaseId: claim.portablePreparationLeaseId, preparation: true as const };
await expect(orchestrator.acquireForRun({ ...nextInput, reattachRemoteLease: preparation }))
.resolves.toMatchObject({ lease: { id: next.lease.id, providerLeaseId: "sandbox-2", heartbeatRunId: nextRunId } });
await expect(orchestrator.acquireForRun({ ...nextInput, reattachRemoteLease: preparation, issueId: null }))
.rejects.toThrow("field=lease_scope");
await expect(orchestrator.acquireForRun({ ...nextInput, reattachRemoteLease: preparation,
persistedExecutionWorkspace: { id: randomUUID(), mode: "isolated_workspace" } })).rejects.toThrow("field=lease_scope");
const otherEnvironmentId = randomUUID();
await db.insert(environments).values({ id: otherEnvironmentId, name: otherEnvironmentId, driver: "plugin",
status: "active", config: seeded.environment.config });
await expect(orchestrator.acquireForRun({ ...nextInput, reattachRemoteLease: preparation,
selectedEnvironmentId: otherEnvironmentId })).rejects.toThrow("field=lease_scope");
} finally {
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, nextRunId));
}
}
await expect(orchestrator.acquireForRun({ ...nextInput, issueId: null })).rejects.toThrow("Active plugin task placement changed");
expect(call.mock.calls.map(([, method]) => method)).toEqual({
handoff: ["environmentAcquireLease", "environmentAcquireLease"],
fresh: ["environmentAcquireLease", "environmentReleaseLease", "environmentAcquireLease"],
"legacy resume": ["environmentAcquireLease", "environmentResumeLease"],
}[mode]);
});
it.each(["same task", "different task", "different agent", "no task"])(
"scopes projectless reusable leases to their task and agent: %s", async (scenario) => {
const seeded = await seedReusablePluginSandboxLease();
@@ -241,19 +241,33 @@ databaseSuite("environment task lease authority", () => {
await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId));
let sessionStateAvailable = false;
const operations: string[] = [];
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }),
call: async (_id: string, _method: string, params: any) => ["read_state", "archive_state"].includes(params.operation.kind)
const call = vi.fn(async (_id: string, _method: string, params: any) => ["read_state", "archive_state"].includes(params.operation.kind)
? { kind: "state", taskId: params.taskId, state: { schema: "paperclip.runner.durable.state.v1", lifecycle: "suspended",
...(params.operation.identity ?? params.operation.priorIdentity) } }
: params.operation.kind === "status"
? (operations.push("status"), { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true, sessionStateAvailable })
: { kind: "accepted", taskId: params.taskId },
} as unknown as PluginWorkerManager;
: { kind: "accepted", taskId: params.taskId });
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call } as unknown as PluginWorkerManager;
const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
const runner = { runnerId: "runner", leaseId: oldLeaseId, sessionId: "session", runId: nextRunId,
turnId: "turn", itemId: "item", harness: "codex", protocolMin: 1, protocolMax: 3 };
const checkpointIdentity = { ...priorIdentity, turnId: "prior-turn", itemId: "prior-item" };
const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId));
const portableTargetLeaseId = randomUUID();
await db.update(environmentLeases).set({ metadata: { ...source.metadata, portableTaskSuccessor: {
sourceLeaseId: oldLeaseId, targetLeaseId: portableTargetLeaseId,
operation: { kind: "ensure_ownership", operationId: portableTargetLeaseId, sourceTaskId: "old-task",
sourceGeneration: "source-generation", targetGeneration: "target-generation", policy: "ordinary", identity: checkpointIdentity },
} } }).where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).rejects.toThrow("not authorized");
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).rejects.toThrow("not authorized");
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task",
} })).rejects.toThrow("not authorized");
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task",
} })).rejects.toThrow("not authorized");
expect(call).not.toHaveBeenCalled();
await db.update(environmentLeases).set({ metadata: { ...source.metadata, runnerTaskSessionRetained: false } }).where(eq(environmentLeases.id, oldLeaseId));
await expect(authorize()).rejects.toThrow("not authorized");
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).resolves.toBe("old-task");
@@ -7,6 +7,7 @@ import { createSystemRecovery } from "../services/system-recovery-state.js";
import * as conversationContinuation from "../services/conversation-continuation.js";
import { classifyEnvironmentCapabilities, environmentRuntimeService } from "../services/environment-runtime.js";
import * as workspaceRuntime from "../services/workspace-runtime.js";
import * as workspacePreparation from "../services/heartbeat/workspace-preparation.js";
import { nativeWorkspaceSyncInternals } from "../services/native-runtime/native-workspace-sync.js";
import { ensureNativeCompletionContract } from "../services/native-runtime/completion-contracts.js";
import { readNativePlanWait, hasCommittedNativePlanWait } from "../services/native-runtime/native-plan-wait.js";
@@ -5278,6 +5279,52 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
]);
});
it("hands off the same portable task lease during in-flight workspace preparation on graceful shutdown", async () => {
const f = await seedRunFixture({ adapterType: "paperclip_runner", agentStatus: "idle", runStatus: "queued", runtimeMode: "legacy" });
await db.update(heartbeatRuns).set({ scopeKind: "issue", issueId: f.issueId }).where(eq(heartbeatRuns.id, f.runId));
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId));
const { leaseId } = await seedEnvironmentLeaseFixture({ ...f, driver: "plugin", provider: "plugin:fixture:tasks" });
await db.update(environmentLeases).set({ providerLeaseId: "same-portable-task", metadata: {
driver: "plugin", agentId: f.agentId, runnerTaskLease: true, portableTaskPreviousLeaseId: randomUUID(),
} }).where(eq(environmentLeases.id, leaseId));
let finishPreparation!: () => void;
const providerReply = new Promise<void>(resolve => { finishPreparation = resolve; });
const reachedPreparation = vi.fn();
const preparation = vi.spyOn(workspacePreparation, "prepareHeartbeatWorkspace").mockImplementationOnce(async (_db, input) => {
await input.controllerLease.assertOwned();
reachedPreparation();
await providerReply;
await input.controllerLease.assertOwned();
throw new Error("Old controller unexpectedly kept preparation authority");
});
const runtime = environmentRuntimeService(db);
const release = vi.spyOn(runtime, "releaseRunLeases");
const heartbeat = heartbeatService(db, { environmentRuntime: runtime });
try {
await heartbeat.resumeQueuedRunsForAgent(f.companyId, f.agentId);
await vi.waitFor(() => expect(reachedPreparation).toHaveBeenCalledOnce(), { timeout: 10_000 });
expect(await heartbeat.drainRunningRunsForShutdown("SIGTERM", new Date(), [f.runId])).toMatchObject({
restartSuspendedRunIds: [f.runId], interruptedRunIds: [], retryRunIds: [],
});
finishPreparation();
await heartbeat.drainActiveRunExecutions();
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
expect(run).toMatchObject({ status: "running", runtimeMode: "legacy", executionStage: "preparing" });
expect(run.controllerLeaseExpiresAt!.getTime()).toBeLessThanOrEqual(Date.now());
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)))[0])
.toMatchObject({ heartbeatRunId: f.runId, status: "active", providerLeaseId: "same-portable-task" });
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(f.runId);
expect(release).not.toHaveBeenCalled();
expect(mockAdapterExecute).not.toHaveBeenCalled();
expect(mockExecutePaperclipNativeSession).not.toHaveBeenCalled();
} finally {
finishPreparation();
await heartbeat.drainActiveRunExecutions();
preparation.mockRestore();
release.mockRestore();
}
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
it.each([
{ name: "still preparing", resumes: true },
{ name: "admitted to dispatch", stage: "dispatching", resumes: false },
@@ -83,6 +83,20 @@ suite("durable plugin task credential handoff", () => {
failExport(value: boolean) { failExport = value; }, credential: () => service.credential(selection!),
settle: (executionStopObserved = false) => settleDispatchedEnvironmentTask(db, workers, { companyId, id: leaseId }, { executionStopObserved }) };
}
it("completes a moved task's saved credential handoff after confirming stop", async () => {
const f = await fixture();
await db.update(environmentLeases).set({ metadata:
sql`${environmentLeases.metadata} || '{"portableTaskSuccessor":{"targetLeaseId":"replacement"}}'::jsonb`,
}).where(eq(environmentLeases.id, f.leaseId));
await expect(f.synchronize()).rejects.toThrow("session ownership unavailable");
expect(f.calls).toEqual([]);
await expect(f.settle()).resolves.toBe("completed");
expect(f.calls).toEqual(["stop", "status", "credentials", "complete"]);
expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
await expect(f.synchronize()).resolves.toBeUndefined();
expect(f.calls).toHaveLength(4);
});
it.each(["codex", "grok"] as const)("writes managed %s refreshes without any plugin or invocation home", async provider => {
const f = await fixture(true, provider);
const read = vi.fn(async () => f.accountCredential("fixture-refreshed", 2021));
@@ -1,5 +1,6 @@
import { AbortMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, S3Client } from "@aws-sdk/client-s3";
import { Readable } from "node:stream";
import { setImmediate as nextEventLoopTurn } from "node:timers/promises";
import { createHash } from "node:crypto";
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";
@@ -10,6 +11,7 @@ import { describe, expect, it, vi } from "vitest";
import { FILE_PART_CONCURRENCY, THROTTLED_PART_ATTEMPTS, putS3MultipartFile } from "../storage/s3-multipart.js";
import { createS3StorageProvider } from "../storage/s3-provider.js";
import { createStorageService } from "../storage/service.js";
import * as fileCrypto from "../storage/file-crypto-pool.js";
const partBytes = 8 * 1024 * 1024;
@@ -89,17 +91,24 @@ describe("file-backed S3 multipart uploads under a request-rate limit", () => {
const client = new S3Client({ region: "us-east-1" });
const attempts = new Map<number, number>();
let laterActive = 0, laterPeak = 0;
let firstRoundRetries = 0;
const firstRoundRetried = Promise.withResolvers<void>();
// Keep disk I/O out of the concurrency measurement. The real-client test checks MD5.
using digest = vi.spyOn(fileCrypto, "digestFileOffThread").mockResolvedValue({ hex: "0".repeat(32), bytes: partBytes });
const random = vi.spyOn(Math, "random").mockReturnValue(0);
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
const number = command.input.PartNumber!;
attempts.set(number, (attempts.get(number) ?? 0) + 1);
await drain(command.input.Body);
if (number <= FILE_PART_CONCURRENCY && attempts.get(number) === 1) throw slowDown();
if (number > FILE_PART_CONCURRENCY) {
if (number <= FILE_PART_CONCURRENCY) {
if (attempts.get(number) === 1) throw slowDown();
// Finish the already-dispatched retries together so every eligible slot can take new work.
if (++firstRoundRetries === FILE_PART_CONCURRENCY) firstRoundRetried.resolve();
await firstRoundRetried.promise;
} else {
laterActive++; laterPeak = Math.max(laterPeak, laterActive);
await new Promise(resolve => setTimeout(resolve, 20));
await nextEventLoopTurn();
laterActive--;
}
return { ETag: `part-${number}` };
@@ -237,6 +237,7 @@ export async function resolveEnvironmentExecutionTarget(input: {
leaseMetadata: Record<string, unknown> | null;
lease?: EnvironmentLease | null;
environmentRuntime?: EnvironmentRuntimeService | null;
signal?: AbortSignal;
// The startup tracer for the provider-exec span. Defaults to the endpoint-
// gated server tracer, which is a no-op when tracing is off. Tests inject a
// recording tracer.
@@ -278,6 +279,7 @@ export async function resolveEnvironmentExecutionTarget(input: {
const result = await runtime.task({ companyId: input.companyId, leaseId: lease.id,
operation: environmentTaskOperationSchema.parse(operation),
...(context?.executionContext ? { executionContext: context.executionContext } : {}),
...(input.signal ? { ownershipSignal: input.signal } : {}),
...(context?.expectedTaskId !== undefined ? { expectedTaskId: context.expectedTaskId } : {}) });
if (result.kind === "description" && result.taskId !== acquisition.providerLeaseId) {
acquisition.outcome = "replacement";
@@ -41,7 +41,7 @@ import {
runnerLifecycleDefaultNeedsLeaseProbe,
resolveWarmRunnerIdleTimeoutMsForRun,
} from "./runner-environment-lifecycle.js";
import { assertEnvironmentCompanyBinding, environmentService } from "./environments.js";
import { assertEnvironmentCompanyBinding, environmentService, toEnvironmentLease } from "./environments.js";
import {
environmentRuntimeService,
buildEnvironmentLeaseContext,
@@ -50,6 +50,7 @@ import {
type ProviderResourceDisposition,
} from "./environment-runtime.js";
import { ENVIRONMENT_DRIVER_TRAITS } from "./environment-driver-traits.js";
import { unlaunchedPortableTaskLease } from "./plugin-task-ownership.js";
import {
resolveEnvironmentExecutionTarget,
resolveEnvironmentExecutionTransport,
@@ -308,9 +309,10 @@ export function environmentRunOrchestrator(
persistedExecutionWorkspace: Pick<ExecutionWorkspace, "id" | "mode"> | null;
executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
runnerTaskProjectIds?: readonly string[];
runnerTaskResumeSessionId?: string | null;
/** Existing live runner recovery must use its original active lease, including ephemeral leases. */
reattachRemoteLease?: { kind?: "sandbox"; leaseId: string; providerLeaseId: string; remoteCwd: string }
| { kind: "plugin_task"; leaseId: string };
| { kind: "plugin_task"; leaseId: string; preparation?: true };
/** A new controller must stop the previous plugin task before allocating a replacement. */
recoverPluginTask?: boolean;
}): Promise<EnvironmentAcquisitionResult> {
@@ -365,16 +367,22 @@ export function environmentRunOrchestrator(
persistedExecutionWorkspace: input.persistedExecutionWorkspace,
executionWorkspaceSettings: input.executionWorkspaceSettings,
runnerTaskProjectIds: input.runnerTaskProjectIds,
runnerTaskResumeSessionId: input.runnerTaskResumeSessionId,
adapterType: input.adapterType ?? null,
warmRunnerIdleTimeoutMs,
runnerReconnectGraceMs,
};
let leaseRecord: EnvironmentRuntimeLeaseRecord;
if (input.reattachRemoteLease) {
const expected = input.reattachRemoteLease;
if (expected) {
// Reattachment keeps the original lease, including ephemeral leases.
// Never reacquire: acquisition can replace a non-reusable sandbox.
const expected = input.reattachRemoteLease;
const lease = await environmentsSvc.getLeaseById(expected.leaseId);
const replayPreparation = expected.kind === "plugin_task" && expected.preparation;
const lease = replayPreparation
? await db.select().from(environmentLeases).where(and(eq(environmentLeases.id, expected.leaseId),
eq(environmentLeases.companyId, input.companyId), unlaunchedPortableTaskLease()))
.then(([row]) => row ? toEnvironmentLease(row) : null)
: await environmentsSvc.getLeaseById(expected.leaseId);
const mismatch: RemoteRecoveryLeaseMismatchField | null = !lease ? "lease"
: lease.companyId !== input.companyId || lease.environmentId !== environment.id ||
lease.heartbeatRunId !== input.heartbeatRunId || lease.issueId !== input.issueId ||
@@ -385,6 +393,8 @@ export function environmentRunOrchestrator(
: expected.kind === "plugin_task"
? environment.driver !== "plugin" ? "environment_driver"
: input.adapterType !== "paperclip_runner" ? "adapter_type"
: replayPreparation ? environment.config.pluginKey !== lease.metadata?.pluginKey ||
environment.config.driverKey !== lease.metadata?.driverKey ? "plugin_task" : null
: lease.metadata?.runnerTaskLease !== true || lease.metadata?.runnerTaskDispatched !== true
|| lease.metadata?.runnerTaskExecutionStopped === true ? "plugin_task" : null
: environment.driver !== "sandbox" ? "environment_driver"
@@ -483,6 +493,7 @@ export function environmentRunOrchestrator(
effectiveExecutionWorkspaceMode: string | null;
persistedExecutionWorkspace: ExecutionWorkspace | null;
runnerTaskProjectIds?: readonly string[];
signal?: AbortSignal;
assertOwned?: () => Promise<void>;
/**
* The host duplex observability recorder for this run. The orchestrator threads
@@ -719,6 +730,7 @@ export function environmentRunOrchestrator(
leaseMetadata: (lease.metadata as Record<string, unknown> | null) ?? null,
lease,
environmentRuntime,
signal: input.signal,
duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
});
// The native plugin task bridge owns guest paths. Never fall back to host execution.
+36 -5
View File
@@ -20,7 +20,6 @@ import { createOwnedSandbox, recoverSandboxCreations, recoverablePluginCreation
import { dueRunLeaseReleaseCondition, pendingRunLeaseReleaseCondition } from "./environment-lease-release-state.js";
import { flushDeferredEnvironmentCleanup, registerDeferredEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
import { executeEnvironmentTask, pluginEnvironmentSupportsTasks, settleDispatchedEnvironmentTask } from "./environment-task-runtime.js";
import type { PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
import { allowLegacyShutdownWorkspaceCleanup, legacyShutdownWorkspaceResourceProtected } from "./legacy-shutdown-workspace-settlement.js";
import { nativeWorkspaceExportPendingForLease, nativeWorkspaceExportPendingForResource } from "./native-runtime/native-workspace-finalization-state.js";
import { beginIdleTrackedWork } from "./task-admission.js";
@@ -484,6 +483,7 @@ export interface EnvironmentDriverAcquireInput {
executionWorkspaceMode: ExecutionWorkspace["mode"] | null;
executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
runnerTaskProjectIds?: readonly string[];
runnerTaskResumeSessionId?: string | null;
/**
* The harness/adapter type for this run (the agent's adapter). Drivers that
* materialize a per-run sandbox use it to select the runtime image so a single
@@ -3516,6 +3516,23 @@ function createPluginEnvironmentDriver(
workerManager.getWorker(plugin.id)?.supportedMethods.includes("environmentResumeLease");
const lifecyclePolicy = warm ? { mode: "warm", idleTimeoutMs: parsed.config.runnerIdleTimeoutMs ?? 300_000 } : { mode: "per_turn", idleTimeoutMs: null };
const fingerprint = createHash("sha256").update(JSON.stringify([parsed.config, input.agentId, input.runnerTaskProjectIds ?? [], ...(input.purpose ? [input.purpose, input.loginCommandKey] : [])])).digest("hex");
if (driver.sessionHandoffVersion === 1 && input.adapterType === "paperclip_runner" && input.heartbeatRunId && input.purpose !== "login") {
// The run lease already records the placement to replay after a lost preparation reply.
const active = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.environmentId, input.environment.id),
eq(environmentLeases.heartbeatRunId, input.heartbeatRunId), eq(environmentLeases.status, "active"),
)).limit(2);
if (active.length > 0) {
const lease = active[0]!;
if (active.length !== 1 || !lease.providerLeaseId || lease.releasedAt || lease.cleanupStatus ||
lease.expiresAt && lease.expiresAt.getTime() <= Date.now() ||
lease.issueId !== input.issueId || lease.executionWorkspaceId !== (input.executionWorkspaceId ?? null) ||
lease.metadata?.runnerTaskLease !== true || lease.metadata?.runnerTaskRetiring === true ||
lease.metadata?.pluginId !== plugin.id || lease.metadata?.driverKey !== parsed.config.driverKey ||
lease.metadata?.runnerTaskFingerprint !== fingerprint) throw new Error("Active plugin task placement changed");
return toEnvironmentLeaseSnapshot(lease);
}
}
const candidates = warm ? await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.environmentId, input.environment.id),
eq(environmentLeases.status, "retained"), eq(environmentLeases.issueId, input.issueId!),
@@ -3523,14 +3540,25 @@ function createPluginEnvironmentDriver(
sql`${environmentLeases.metadata}->>'agentId' = ${input.agentId!}`,
sql`${environmentLeases.metadata}->>'pluginId' = ${plugin.id}`,
sql`${environmentLeases.metadata}->>'driverKey' = ${parsed.config.driverKey}`,
sql`${environmentLeases.metadata}->'portableTaskSuccessor' is null`,
)) : [];
if (candidates.length > 1) throw new Error("Ambiguous retained plugin task ownership");
const previous = candidates[0];
const previousSessionId = (previous?.metadata?.runnerTaskIdentity as { sessionId?: unknown } | undefined)?.sessionId;
const previousAccountIdentity = previous?.metadata?.runnerTaskAccountIdentity;
const freshPortableSession = driver.sessionHandoffVersion === 1 && input.runnerTaskResumeSessionId !== undefined &&
(input.runnerTaskResumeSessionId === null || input.runnerTaskResumeSessionId !== previousSessionId);
const handoffPrevious = !freshPortableSession && driver.sessionHandoffVersion === 1 && previous?.providerLeaseId &&
previous.metadata?.runnerTaskFingerprint === fingerprint &&
typeof previousAccountIdentity === "string" && previousAccountIdentity.trim().length > 0 &&
typeof (previous.metadata?.providerMetadata as { ownershipGeneration?: unknown } | undefined)?.ownershipGeneration === "string"
? previous : undefined;
let resumed: PluginEnvironmentLease | undefined;
if (previous?.providerLeaseId && previous.metadata?.runnerTaskFingerprint !== fingerprint) {
// A changed mount scope must retire the old process before checkpoint recovery.
if (!handoffPrevious && previous?.providerLeaseId &&
(freshPortableSession || previous.metadata?.runnerTaskFingerprint !== fingerprint)) {
// A fresh logical session or changed mount scope must retire the old process before provisioning.
await this.releaseRunLease({environment:input.environment,lease:toEnvironmentLeaseSnapshot(previous),status:"released"});
} else if (previous?.providerLeaseId) {
} else if (!handoffPrevious && previous?.providerLeaseId) {
// Resume only inspects the exact computer; an unavailable reply must not create a replacement.
resumed = await workerManager.call(plugin.id, "environmentResumeLease", {
driverKey: parsed.config.driverKey, companyId: input.companyId, environmentId: input.environment.id,
@@ -3589,6 +3617,7 @@ function createPluginEnvironmentDriver(
...(warm ? {runnerTaskProjectIds:[...input.runnerTaskProjectIds ?? []]} : {}),remoteCwd:".",
...(input.agentId ? { agentId: input.agentId } : {}),
providerMetadata: providerLease.metadata ?? {},
...(handoffPrevious ? { portableTaskPreviousLeaseId: handoffPrevious.id } : {}),
...(input.purpose ? { purpose: input.purpose, loginCommandKey: input.loginCommandKey } : {}),
environmentId: input.environment.id,
...(input.purpose !== "login" && input.adapterType === "paperclip_runner" && driver.supportsTasks === true ? { runnerTaskLease: true } : {}),
@@ -3996,7 +4025,7 @@ export function environmentRuntimeService(
},
/** Dispatch plugin-provided Runner execution, including resource preparation and task lifecycle. */
async task(input: { companyId: string; leaseId: string; operation: PluginEnvironmentTaskOperation; executionContext?: unknown; expectedTaskId?: string }) {
async task(input: Parameters<typeof executeEnvironmentTask>[2]) {
if (!options.pluginWorkerManager) throw new Error("Plugin-provided Runner execution worker manager unavailable");
return executeEnvironmentTask(db, options.pluginWorkerManager, input);
},
@@ -4015,6 +4044,7 @@ export function environmentRuntimeService(
persistedExecutionWorkspace: Pick<ExecutionWorkspace, "id" | "mode"> | null;
executionWorkspaceSettings?: IssueExecutionWorkspaceSettings | null;
runnerTaskProjectIds?: readonly string[];
runnerTaskResumeSessionId?: string | null;
/** The agent's adapter type for this run (mixed-harness environments). */
adapterType?: string | null;
resolveSnapshotIdentity?: boolean;
@@ -4060,6 +4090,7 @@ export function environmentRuntimeService(
executionWorkspaceMode: leaseContext.executionWorkspaceMode,
executionWorkspaceSettings: input.executionWorkspaceSettings ?? null,
runnerTaskProjectIds: input.runnerTaskProjectIds,
runnerTaskResumeSessionId: input.runnerTaskResumeSessionId,
purpose: input.purpose,
loginCommandKey: input.loginCommandKey,
adapterType: input.adapterType ?? null,
+93 -14
View File
@@ -5,19 +5,22 @@ import { createOwnedSandbox, queueSandboxCreationCleanup, recordSandboxCreationR
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
import { releaseProviderRecoveryTransfers, stageProviderFreshWorkspace } from "./provider-fresh-workspace.js";
import { readWorkspaceProviderBinding } from "@paperclipai/shared";
import { settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js";
import { canReadStoppedPluginTaskCredential, settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js";
import { and, eq, inArray, sql } from "drizzle-orm";
import { environmentLeases, executionWorkspaces, environments, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db";
import { freshWorkspaceSchema, environmentTaskOperationSchema, parseEnvironmentTaskResult,
environmentTaskErrorData, readEnvironmentTaskErrorDiagnostic, JsonRpcCallError, PLUGIN_RPC_ERROR_CODES,
environmentTaskErrorCodeData, PluginEnvironmentTaskError, readEnvironmentTaskErrorCode,
type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation, type PluginEnvironmentTaskResult } from "@paperclipai/plugin-sdk";
import { pluginRegistryService } from "./plugin-registry.js";
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
import { NativeExecutionContext } from "./native-runtime/native-execution-context.js";
import { assertPluginTaskSourceAttachment, assertPluginTaskSourceRecovery } from "./native-runtime/plugin-task-source-recovery.js";
import { pluginAgentRuntimeIdentity } from "./plugin-agent-runtime-state.js";
import { assertPluginTaskOwnershipDispatch, PluginTaskOwnershipPendingError, readyPortableTaskSource, reconcilePluginTaskAdmission } from "./plugin-task-ownership.js";
export type PluginTaskDescription = Extract<PluginEnvironmentTaskResult, { kind: "description" }>;
export function isSettledReadyRunnerState(state: Record<string, unknown>): boolean {
const commands = state.processedCommands;
@@ -48,14 +51,15 @@ export async function pluginEnvironmentSupportsTasks(db: Db, workers: PluginWork
* never a plugin identity, resource endpoint, or agent/project authority.
* A timeout is ambiguous: retry the same operation against the same lease.
*/
export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManager, input: {
export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerManager, "getWorker" | "call">, input: {
companyId: string;
leaseId: string;
operation: PluginEnvironmentTaskOperation;
executionContext?: unknown;
ownershipSignal?: AbortSignal;
/** Physical task captured by an existing owner, independent of the mutable lease. */
expectedTaskId?: string;
}) {
}): Promise<PluginEnvironmentTaskResult> {
const operation = environmentTaskOperationSchema.parse(input.operation);
const [row] = await db.select({ lease: environmentLeases, environment: environments, run: heartbeatRuns })
.from(environmentLeases)
@@ -74,6 +78,27 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
const taskId = row.lease.providerLeaseId;
if (input.expectedTaskId !== undefined && input.expectedTaskId !== taskId) throw new Error("Plugin Runner physical task changed");
const metadata = lease.metadata ?? {};
const ownership = operation.kind === "ensure_ownership";
if (operation.kind === "describe" && metadata.portableTaskOwnershipReady !== true && metadata.portableTaskSuccessor === undefined &&
(metadata.portableTaskPredecessor !== undefined || typeof metadata.portableTaskPreviousLeaseId === "string")) {
await reconcilePluginTaskAdmission(db, lease, request =>
executeEnvironmentTask(db, workers, { ...input, operation: request }), context?.signal ?? input.ownershipSignal);
return executeEnvironmentTask(db, workers, input);
}
const requiresOwnership = !["ensure_ownership", "status", "stop", "complete"].includes(operation.kind);
const stoppedCredentialHandoff = operation.kind === "credentials" && input.expectedTaskId === taskId &&
canReadStoppedPluginTaskCredential(metadata, operation.accountIdentity);
if (requiresOwnership && (metadata.portableTaskSuccessor !== undefined && !stoppedCredentialHandoff ||
(metadata.portableTaskPredecessor !== undefined || metadata.portableTaskPreviousLeaseId !== undefined) && metadata.portableTaskOwnershipReady !== true)) {
throw new Error("Plugin Runner session ownership unavailable");
}
if (ownership) {
if (!environment || lease.status !== "active" || lease.releasedAt || run?.status !== "running" ||
metadata.runnerTaskRetiring === true || lease.expiresAt && lease.expiresAt.getTime() <= Date.now()) {
throw new Error("Plugin-provided Runner execution lease is not active");
}
assertPluginTaskOwnershipDispatch(lease, operation);
}
// A different launch identity is authorized by the saved consumer claim, never by the mismatch alone.
const sourceRecovery = run && ((operation.kind === "prepare" && operation.requirements.identity.runId !== run.id && metadata.runnerTaskSourceRecovery) ||
(operation.kind === "submit" && operation.runner.runId !== run.id && metadata.runnerTaskSourceRecovery))
@@ -98,12 +123,21 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
const savedIdentity = metadata.runnerTaskIdentity as Record<string, unknown> | undefined;
const prior = operation.kind === "prepare" ? operation.priorIdentity : undefined;
const priorSaved = metadata.runnerTaskPriorIdentity as Record<string, unknown> | undefined;
const firstPortableResume = operation.kind === "resume" && !savedIdentity &&
metadata.portableTaskPredecessor !== undefined;
if (firstPortableResume && (run?.status !== "running" ||
!await readyPortableTaskSource(db, input.companyId, lease.id, operation.identity))) {
throw new Error("Plugin Runner session ownership unavailable");
}
const matches = (identity: typeof prior, saved: Record<string, unknown> | undefined) => !!identity && !!saved &&
identity.runnerInstanceId === saved.runnerId && identity.environmentLeaseId === saved.leaseId &&
identity.normalizedSessionId === saved.sessionId && identity.runId === saved.runId &&
identity.turnId === saved.turnId && identity.itemId === saved.itemId;
const portableCheckpoint = operation.kind === "read_state" && !resumeFrom && !savedIdentity &&
metadata.portableTaskPredecessor !== undefined
? (await readyPortableTaskSource(db, input.companyId, lease.id, operation.identity))?.taskId : undefined;
const ownCheckpoint = checkpointIdentity && !resumeFrom &&
(matches(checkpointIdentity,savedIdentity) || operation.kind !== "read_file" && matches(checkpointIdentity,priorSaved));
(portableCheckpoint || matches(checkpointIdentity,savedIdentity) || operation.kind !== "read_file" && matches(checkpointIdentity,priorSaved));
if (checkpointIdentity && !resumeFrom && !ownCheckpoint) throw new Error("Plugin Runner state identity mismatch");
const retainedCheckpointRead = operation.kind === "read_state" && ownCheckpoint &&
lease.leasePolicy === "reuse_by_environment" && ["retained", "expired"].includes(lease.status) &&
@@ -122,10 +156,21 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
const sourceAttachment = operation.kind === "prepare" && prior && sourceReference?.identity?.runId === prior.runId
? await assertPluginTaskSourceAttachment(db, { companyId: input.companyId, leaseId: lease.id,
context: context instanceof NativeExecutionContext ? context : undefined, operation }) : undefined;
const warmAttachment = !!prior && (metadata.runnerTaskReused === true || !!sourceAttachment) &&
(matches(prior, savedIdentity) || matches(prior, priorSaved) && operation.kind === "prepare" && matches(operation.requirements.identity, savedIdentity)) && runner?.runnerId === prior.runnerInstanceId &&
const portableAttachment = prior && !sourceAttachment && metadata.runnerTaskReused !== true &&
metadata.portableTaskPredecessor !== undefined
? (await readyPortableTaskSource(db, input.companyId, lease.id, prior))?.taskId : undefined;
const warmAttachment = !!prior && (metadata.runnerTaskReused === true || !!sourceAttachment || !!portableAttachment) &&
(portableAttachment && !savedIdentity || matches(prior, savedIdentity) ||
matches(prior, priorSaved) && operation.kind === "prepare" && matches(operation.requirements.identity, savedIdentity)) &&
runner?.runnerId === prior.runnerInstanceId &&
runner?.leaseId === prior.environmentLeaseId && runner?.sessionId === prior.normalizedSessionId;
if (prior && !warmAttachment) throw new Error("Plugin warm task prior identity mismatch");
const portableAuthority = firstPortableResume || portableAttachment
? sql`${environmentLeases.metadata}->'portableTaskSuccessor' is null
and ${environmentLeases.metadata}->'portableTaskOwnershipReady' = 'true'::jsonb
and coalesce(${environmentLeases.metadata}->>'portableTaskCheckpointUnavailable', 'false') != 'true'
and ${environmentLeases.metadata}->'portableTaskPredecessor' = ${JSON.stringify(metadata.portableTaskPredecessor)}::jsonb`
: undefined;
if (runner && !sourceRecovery && ((!checkpointIdentity || !resumeFrom) && !(ownCheckpoint && operation.kind !== "read_file") && runner.runId !== (activeProbe ? probe.runId : run?.id) || (!resumeFrom && runner.leaseId !== lease.id && !ownCheckpoint && !warmAttachment)
|| (activeProbe && (runner.sessionId !== probe.sessionId || (launches && operation.projectIds.length > 0))))) {
throw new Error("Plugin-provided Runner execution identity mismatch");
@@ -168,10 +213,18 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
const plugin = await pluginRegistryService(db).getById(metadata.pluginId);
const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === metadata.driverKey);
if (!plugin || plugin.status !== "ready" || !driver?.supportsTasks ||
!plugin.manifestJson.capabilities.includes("environment.drivers.register") ||
!workers.getWorker(plugin.id)?.supportedMethods.includes("environmentTask")) {
!plugin.manifestJson.capabilities.includes("environment.drivers.register")) {
throw new Error("Plugin-provided Runner execution provider unavailable");
}
const worker = workers.getWorker(plugin.id);
if (ownership && !worker && driver.sessionHandoffVersion === 1) throw new PluginTaskOwnershipPendingError();
if (!worker?.supportedMethods.includes("environmentTask")) {
throw new Error("Plugin-provided Runner execution provider unavailable");
}
if (operation.kind === "resume" && operation.handoffCredential && driver.sessionHandoffVersion !== 1) {
throw new Error("Plugin Runner credential handoff unavailable");
}
if (ownership && driver.sessionHandoffVersion !== 1) throw new Error("Plugin Runner ownership reconciliation unavailable");
const callWorker = async <M extends HostToWorkerMethodName>(method: M, params: HostToWorkerMethods[M][0], timeout?: number): Promise<HostToWorkerMethods[M][1]> => {
if (!context) {
const cleanup = metadata.runnerTaskStartupCleanup as { contextId?: unknown; requiresPreparationStop?: unknown } | undefined;
@@ -255,13 +308,23 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
operation.launch.maxRuntimeMs = Math.min(remainingMs, ...requestedLimits);
delete operation.launch.maxLifetimeMs;
}
if (operation.kind === "submit" || operation.kind === "copilot_metadata") {
if (operation.kind === "submit" || operation.kind === "copilot_metadata" || firstPortableResume) {
// No ticket, headers, environment values, or launch paths belong in metadata.
const dispatched = {
runnerTaskLease: true, runnerTaskDispatched: true,
runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false,
...(operation.kind === "submit" ? {
runnerTaskIdentity: operation.runner,
runnerTaskResumeFrom: operation.resumeFrom ?? null,
runnerTaskAccountIdentity: operation.accountIdentity,
} : {}),
};
const marked = await db.update(environmentLeases).set({
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskAdmissionRejection') || ${JSON.stringify({ runnerTaskLease: true, runnerTaskDispatched: true, runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false, ...(operation.kind === "submit" ? { runnerTaskIdentity: operation.runner, runnerTaskResumeFrom: operation.resumeFrom ?? null } : {}) })}::jsonb`,
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskAdmissionRejection') || ${JSON.stringify(dispatched)}::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
requiredProjectAuthProof ? sql`${environmentLeases.metadata}->'runnerTaskProjectAuthValidation' = ${JSON.stringify(requiredProjectAuthProof)}::jsonb` : undefined,
firstPortableResume ? portableAuthority : undefined,
)).returning({ id: environmentLeases.id });
if (marked.length !== 1) throw new Error("Plugin-provided Runner execution lease is not active");
}
@@ -276,7 +339,9 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
...(startup ? { runnerTaskStartup: startup } : {}),
})}::jsonb` }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.status, "active"),
sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
sql`${environmentLeases.metadata}->'runnerTaskIdentity' = ${JSON.stringify(savedIdentity)}::jsonb`)).returning({id:environmentLeases.id});
portableAttachment ? portableAuthority : undefined,
savedIdentity ? sql`${environmentLeases.metadata}->'runnerTaskIdentity' = ${JSON.stringify(savedIdentity)}::jsonb`
: sql`${environmentLeases.metadata}->'runnerTaskIdentity' is null`)).returning({id:environmentLeases.id});
if (marked.length !== 1) throw new Error("Plugin warm task attachment lost authority");
}
async function recordTaskStop(result: Extract<ReturnType<typeof parseEnvironmentTaskResult>, {kind:"status" | "copilot_metadata"}>) {
@@ -322,6 +387,7 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
...(recoverableCreation ? { creationOperationVersion: 1, providerMetadata: metadata.providerMetadata } : {}) },
}, async identity => {
const params = { driverKey: replacementDriverKey, companyId: input.companyId, environmentId: environment!.id,
executionWorkspaceId: lease.executionWorkspaceId ?? undefined,
issueId: lease.issueId, config: driverConfig, runId: identity.runId, creationOperationId: identity.id,
agentId: run!.agentId, adapterType: "paperclip_runner",
...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? { requestedExpiresAt: metadata.runnerTaskRequestedExpiresAt } : {}) };
@@ -414,8 +480,15 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
...(executionContext ? { executionContext } : {}),
};
const timeout = operation.kind === "copilot_metadata" ? Math.max(1, Math.min(40_000, Number(probe?.expiresAtMs) - Date.now()))
: executionContext ? Math.max(1, context!.deadlineAtMs - Date.now()) : 15_000;
const result = await callWorker("environmentTask", params, timeout);
: executionContext ? Math.max(1, context!.deadlineAtMs - Date.now())
: driver.sessionHandoffVersion === 1 && (ownership || operation.kind === "describe") ? 90_000 : 15_000;
const result = await callWorker("environmentTask", params, timeout).catch(error => {
// The plugin may have applied the saved claim before its reply was lost.
// Keep the same operation ID; the RPC response below still needs validation.
if (ownership && (!(error instanceof JsonRpcCallError) ||
error.code === PLUGIN_RPC_ERROR_CODES.TIMEOUT || error.code === PLUGIN_RPC_ERROR_CODES.WORKER_UNAVAILABLE)) throw new PluginTaskOwnershipPendingError();
throw error;
});
if (executionContext) context!.assertOpen();
const parsed = parseEnvironmentTaskResult(operation, taskId, result);
if (operation.kind === "submit" && parsed.kind === "admission_rejected") {
@@ -443,6 +516,11 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
await assertPluginTaskSourceAttachment(db, { companyId: input.companyId, leaseId: lease.id,
context: context instanceof NativeExecutionContext ? context : undefined, operation });
}
if (portableCheckpoint && operation.kind === "read_state" || portableAttachment && operation.kind === "prepare") {
const identity = operation.kind === "read_state" ? operation.identity : operation.priorIdentity;
const sourceTaskId = (await readyPortableTaskSource(db, input.companyId, lease.id, identity))?.taskId;
if (sourceTaskId !== (portableCheckpoint || portableAttachment)) throw new Error("Plugin Runner session ownership changed");
}
if (operation.kind === "read_file") {
const [current] = await db.select({ lease: environmentLeases, run: heartbeatRuns }).from(environmentLeases)
.leftJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, environmentLeases.companyId)))
@@ -499,6 +577,7 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
}
return parsed;
} catch (error) {
if (error instanceof PluginTaskOwnershipPendingError) throw error;
const diagnostic = operation.kind === "submit" && operation.resumeFrom ? readEnvironmentTaskErrorDiagnostic(error) : null;
if (diagnostic) throw new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
message: "Plugin Runner source state unavailable",
+2 -1
View File
@@ -274,7 +274,7 @@ function resolveKubernetesConfig(
return companyIdOrConfig;
}
function toEnvironmentLease(row: EnvironmentLeaseRow): EnvironmentLease {
export function toEnvironmentLease(row: EnvironmentLeaseRow): EnvironmentLease {
return {
id: row.id,
companyId: row.companyId,
@@ -1649,6 +1649,7 @@ export function environmentService(db: Db) {
// lease holds older session state. That claim and this
// handoff compete on this row, so only one of them wins.
sql`not (coalesce(${environmentLeases.metadata}, '{}'::jsonb) ? 'reusableSandboxLeaseSuperseded')`,
sql`not (coalesce(${environmentLeases.metadata}, '{}'::jsonb) ? 'portableTaskSuccessor')`,
),
)
.returning();
+12 -7
View File
@@ -1897,6 +1897,7 @@ export function heartbeatService(
runOptions: {
nativeLeaseOwner?: string;
nativeRestartRecovery?: NativeRestartRecoveryClaim;
portablePreparationLeaseId?: string;
onNativeExecutionLeaseAcquired?: () => Promise<void>;
} = {},
) {
@@ -1914,7 +1915,8 @@ export function heartbeatService(
async function executeRunAttempt(
runId: string,
runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; onNativeExecutionLeaseAcquired?: () => Promise<void> },
runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; portablePreparationLeaseId?: string;
onNativeExecutionLeaseAcquired?: () => Promise<void> },
) {
const attemptStartedAtMs = Date.now();
let attestedQuestionResponseAtMs: number | null = null;
@@ -2087,8 +2089,9 @@ export function heartbeatService(
let dispatchGateEntered = false;
const nativeDispatchMayHaveStarted = () => nativeDispatchStarted || dispatchGateEntered || nativeOwnershipHeld
|| Boolean(runOptions.nativeRestartRecovery || runOptions.nativeLeaseOwner);
const preparationHandedOff = () => shutdownHandedOff() && !legacyAdapterEntered && !nativeDispatchStarted;
const leaveToNextController = async (error: unknown) => {
if (!shutdownHandedOff() || legacyAdapterEntered || nativeDispatchStarted) return false;
if (!preparationHandedOff()) return false;
nativeSessionResumeScheduled = true;
const reason = error instanceof Error ? error.message : String(error);
const errorCode = /^[a-z][a-z0-9_]{0,80}$/.test(reason) ? reason : null;
@@ -2895,7 +2898,7 @@ export function heartbeatService(
executionTarget,
workspaceRestoreSource,
} = await timePreparation("workspace", () => prepareHeartbeatWorkspace(db, {
run, agent, controllerLease, nativeRunnerPreparationSpans,
run, agent, controllerLease, nativeRunnerPreparationSpans, signal: executionControl.controller.signal,
task: {
issueRef,
issueId,
@@ -2930,6 +2933,7 @@ export function heartbeatService(
configSnapshot,
secretManifest,
managedAiCredentialIdentity: managedAiRuntime?.identity,
runnerTaskResumeSessionId: run.nativeSessionId ?? preparedSession.taskResumeSessionId,
},
reuse: {
requestedShouldReuseExisting,
@@ -3295,7 +3299,7 @@ export function heartbeatService(
context.projectId = executionWorkspace.projectId;
}
const {
previousSessionDisplayId, runtimeSessionParamsForAdapter, taskSessionCredentialCompatible, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction,
previousSessionDisplayId, runtimeSessionParamsForAdapter, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction,
} = await resolveHeartbeatAdapterSession({
agent,
task: { context, taskKey, issueId, continuationSummary, executionContinuation },
@@ -3507,9 +3511,7 @@ export function heartbeatService(
session: {
persistedNativeExecutionInput,
persistedRunnerProfile,
taskSessionCredentialCompatible,
taskSessionDecodedParams,
taskSessionForRun,
taskResumeSessionId: preparedSession.taskResumeSessionId,
isFailedChatRunRetry,
getFreshSessionHandoff,
},
@@ -3585,6 +3587,8 @@ export function heartbeatService(
adapter,
runtimeForAdapter,
getNativeFreshSessionHandoff,
pluginTaskDescription: runtimeSelection.pluginTaskDescription,
toolRefreshOnResume: runtimeSelection.toolRefreshOnResume,
getFreshSessionHandoff,
runOptions,
},
@@ -3999,6 +4003,7 @@ export function heartbeatService(
await instructionWorkingCopy?.nativeInstructionReservation?.release().catch(error => logger.warn({ runId: run.id, err: error }, "Managed warm session preparation cleanup failed"));
if (managedAiRuntime) await managedAiRuntime.cleanup().catch(() => logger.warn({ runId: run.id }, "AI connection refresh or cleanup failed"));
let latestRun = await getRun(run.id).catch(() => null);
if (preparationHandedOff() && (!latestRun || latestRun.status === "running")) nativeSessionResumeScheduled = true;
try {
if (latestRun && isHeartbeatRunTerminalStatus(latestRun.status)) {
await writeExecutionControl(db, DELIVERY_QUEUES.executionFinalization, tx => tx
@@ -64,7 +64,7 @@ interface InstructionWorkingCopyInput {
context: Selection["task"]["context"];
session: {
taskSession: { lastRunId: string | null } | null;
taskSessionForRun: Selection["session"]["taskSessionForRun"];
taskSessionForRun: { lastRunId: string | null } | null;
sessionConfigMetadata: EffectiveRunSessionConfigMetadata;
};
workspace: Pick<Selection["workspace"], "executionTarget" | "executionWorkspace" | "persistedExecutionWorkspace">;
@@ -30,6 +30,7 @@ import { hasRemoteTerminationReceipt, remoteTerminationReceipt } from "../remote
import { persistHeartbeatRunProcessMetadata } from "./run-lifecycle.js";
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
import { hasNativeLocalProcessStop, PROCESS_START_REQUESTED, releaseSettledNativeRunProcess } from "../native-local-process-stop.js";
import { legacyControllerBootId } from "../legacy-controller-lease.js";
type Run = typeof heartbeatRuns.$inferSelect;
@@ -212,6 +213,40 @@ describePostgres("heartbeat recovery module ownership", () => {
return (await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, invocationSource: "automation", status: "running", runtimeMode: "legacy" }).returning())[0];
}
it.each(["unlaunched", "dispatched", "reassigned", "ambiguous"] as const)("claims the same expired portable preparation only with its original task lock (%s)", async state => {
const run = await seedRun(), deps = callbacks(db);
const environmentId = randomUUID(), leaseId = randomUUID();
const oldBootId = randomUUID();
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
await db.update(heartbeatRuns).set({ scopeKind: "issue", controllerBootId: oldBootId, controllerLeaseExpiresAt: new Date(0),
executionStage: "preparing" }).where(eq(heartbeatRuns.id, run.id));
if (state !== "reassigned") await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, run.issueId!));
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "plugin", config: {} });
await db.insert(environmentLeases).values({ id: leaseId, companyId: run.companyId, environmentId, heartbeatRunId: run.id,
issueId: run.issueId, provider: "plugin:fixture:tasks", providerLeaseId: "portable-task", status: "active",
metadata: { agentId: run.agentId, driver: "plugin", runnerTaskLease: true, portableTaskPreviousLeaseId: randomUUID(),
...(state === "dispatched" ? { runnerTaskDispatched: true } : {}) } });
if (state === "ambiguous") await db.insert(environmentLeases).values({ companyId: run.companyId, environmentId,
heartbeatRunId: run.id, issueId: run.issueId, provider: "plugin:fixture:tasks", providerLeaseId: "competing-task", status: "active" });
const scope = { companyId: run.companyId, runId: run.id };
const recovery = createHeartbeatRecovery(db, deps);
await Promise.all([recovery.recoverActiveExecutions(scope), recovery.recoverActiveExecutions(scope)]);
if (state === "unlaunched") {
expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(run.id, { portablePreparationLeaseId: leaseId });
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
expect(current).toMatchObject({ status: "running", controllerBootId: legacyControllerBootId, executionStage: "preparing" });
expect(current.controllerLeaseExpiresAt!.getTime()).toBeGreaterThan(Date.now());
expect((await db.select().from(issues).where(eq(issues.id, run.issueId!)))[0].executionRunId).toBe(run.id);
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)))[0]).toMatchObject({
status: "active", providerLeaseId: "portable-task", heartbeatRunId: run.id,
});
} else {
expect(deps.executeRun).not.toHaveBeenCalled();
expect(deps.setRunStatusFromLive).toHaveBeenCalledOnce();
}
});
it.each([false, true])("delivers scoped recovery to a preparation without a runner PID or checkpoint (locally tracked=%s)", async tracked => {
const run = await seedRun(), deps = callbacks(db);
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
+22 -1
View File
@@ -38,9 +38,11 @@ import {
} from "../sandbox-stop-and-retain.js";
import { discoverSettledNativeRunProcesses, PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED, recordNativeLocalProcessStopInTransaction, releaseSettledNativeRunProcess, retryPendingSettledNativeRunProcesses } from "../native-local-process-stop.js";
import { pendingCaseHandoffForRun } from "./case-handoff-lifecycle.js";
import { portableTaskPreparationLeaseId } from "../plugin-task-ownership.js";
import { retireInheritedRemoteRunnerIdentity } from "../native-runtime/native-inherited-runner-retry.js";
import {
LEGACY_LIVE_OWNER_SETTLE_GRACE_MS,
handoffLegacyControllerPreparation,
legacyControllerBootId,
liveLegacyControllerExpiresAt,
proveLegacyController,
@@ -157,6 +159,7 @@ export interface HeartbeatRecoveryDependencies extends Pick<HeartbeatRetryDepend
executeRun: (runId: string, options?: {
nativeLeaseOwner?: string;
nativeRestartRecovery?: NativeRestartRecoveryClaim;
portablePreparationLeaseId?: string;
onNativeExecutionLeaseAcquired?: () => Promise<void>;
}) => Promise<void>;
scheduleNativeSessionResumeDispatch: (runId: string, nextAttemptAt: Date, mode?: "restart") => void;
@@ -1269,6 +1272,13 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
restartSuspendedRunIds.push(run.id);
return;
}
if (agent.adapterType === "paperclip_runner" && run.runtimeMode === "legacy" && run.scopeKind === "issue" &&
run.issueId && run.executionStage === "preparing" && run.controllerBootId === legacyControllerBootId &&
await handoffLegacyControllerPreparation(db, run, portableTaskPreparationLeaseId(db, run),
() => markRunHandedOffForShutdown(run.id))) {
restartSuspendedRunIds.push(run.id);
return;
}
const message = `Interrupted by graceful server shutdown (${signal})`;
const running = runningProcesses.get(run.id);
const control = run.runtimeMode === "legacy" ? adapterExecutionControls.get(run.id) : undefined;
@@ -2467,6 +2477,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
nativeWorkspaceFinalizationPending,
} of activeRuns) {
if (run.runtimeMode === "native" && readNativeStartupCleanup(run)) continue;
if (run.runtimeMode === "legacy" && isRunHandedOffForShutdown(run.id)) continue;
if (parseObject(run.contextSnapshot).nativeQuestionCancellation) continue;
// Native recovery owns these dispositions. Its writes happened after this
// inventory was read; do not reap or overwrite them using the old row.
@@ -2686,8 +2697,18 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
readNonEmptyString(runContext.wakeReason) === "issue_monitor_due" &&
monitorNextCheckAt !== undefined &&
(!monitorNextCheckAt || monitorNextCheckAt.getTime() <= now.getTime());
const revokedState = await revokeExpiredLegacyControllerWithState(db, run);
const reclaimPortablePreparation = priorBootSetupExpired && adapterType === "paperclip_runner" && run.scopeKind === "issue" && run.issueId
? portableTaskPreparationLeaseId(db, run) : undefined;
const revokedState = await revokeExpiredLegacyControllerWithState(db, run, reclaimPortablePreparation);
if (!revokedState) continue;
if (revokedState.portablePreparationLeaseId) {
const execution = executeRun(run.id, { portablePreparationLeaseId: revokedState.portablePreparationLeaseId }).catch(error => {
logger.error({ err: error, runId: run.id }, "portable task preparation recovery failed");
});
activeRunExecutionPromises.add(execution);
void execution.finally(() => activeRunExecutionPromises.delete(execution));
continue;
}
// The scan snapshot can predate a child the old controller saved just
// before it died. The child checks above used that snapshot, so a proven
// gone controller is never enough here: when revocation reports a child
@@ -128,7 +128,7 @@ export interface HeartbeatRuntimeExecutionInput {
issueId: string | null;
taskKey: string | null;
};
runtime: Pick<Selected, "nativeExecution" | "nativeRunnerInstanceId" | "getNativeFreshSessionHandoff"> & {
runtime: Pick<Selected, "nativeExecution" | "nativeRunnerInstanceId" | "pluginTaskDescription" | "toolRefreshOnResume" | "getNativeFreshSessionHandoff"> & {
nativeRuntimeResolution: Selection["nativeRuntimeResolution"];
adapter: ServerAdapterModule;
runtimeForAdapter: AdapterExecutionContext["runtime"];
@@ -636,9 +636,11 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
observability,
getFreshSessionHandoff: getNativeFreshSessionHandoff,
refreshTools: context.refreshTools === true,
toolRefreshOnResume: input.runtime.toolRefreshOnResume,
conversationMode: isConversation(issueContext),
turnTimeoutMs: Math.max(0, asNumber(runtimeConfig.timeoutSec, 0)) * 1_000,
runnerInstanceId: nativeRunnerInstanceId,
pluginTaskDescription: input.runtime.pluginTaskDescription,
leaseOwner: runOptions.nativeLeaseOwner,
restartRecovery: runOptions.nativeRestartRecovery,
onExecutionLeaseAcquired: runOptions.onNativeExecutionLeaseAcquired,
@@ -25,6 +25,7 @@ import { prepareNativeWorkspaceSync } from "../native-runtime/native-workspace-s
import { resolveHeartbeatNativeRuntimeMode } from "../native-runtime/runtime-mode.js";
import { claimNativeRestartRecoveries } from "../native-runtime/native-restart-recovery.js";
import { buildNativeModelEnvelope, canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js";
import * as runner from "../../vendor/paperclip-runner/index.js";
import { prepareAndSelectHeartbeatRuntime, selectHeartbeatRuntime, type HeartbeatRuntimeSelectionInput } from "./runtime-selection.js";
import { resolveRunnerEnvironmentForRun, resolveWarmRunnerIdleTimeoutMsForRun } from "../runner-environment-lifecycle.js";
import { reserveRunBudget } from "../budget-reservations.js";
@@ -82,7 +83,7 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
run, agent,
nativeRuntimeResolution: resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: null, workspaceId: null }),
task: { issueRef: issue, issueContext: null, context: { paperclipTaskMarkdown: "Complete this task" }, executionContinuation: null, safeWakeComments: [], safeWakeCommentContext: null, paperclipWakePayload: null, acceptedPlanContinuationWake: false },
session: { persistedNativeExecutionInput: null, persistedRunnerProfile: {}, taskSessionCredentialCompatible: true, taskSessionDecodedParams: null, taskSessionForRun: null, isFailedChatRunRetry: false, getFreshSessionHandoff: vi.fn(async () => "Earlier context") },
session: { persistedNativeExecutionInput: null, persistedRunnerProfile: {}, taskResumeSessionId: null, isFailedChatRunRetry: false, getFreshSessionHandoff: vi.fn(async () => "Earlier context") },
workspace: { persistedExecutionWorkspace: null, executionWorkspace: { cwd: "/tmp/paperclip-runtime-selection", repoUrl: null, repoRef: null, branchName: null }, executionTarget: null, isDotRun: false, projectRepositoryPaths: [],
activeEnvironmentLease: { lease: { id: randomUUID(), companyId: company.id, environmentId: null, executionWorkspaceId: null, issueId: issue.id, heartbeatRunId: run.id, status: "active", leasePolicy: "ephemeral", provider: null, providerLeaseId: null, acquiredAt: now, lastUsedAt: now, expiresAt: null, releasedAt: null, failureReason: null, cleanupStatus: null, metadata: null, createdAt: now, updatedAt: now } },
},
@@ -97,6 +98,58 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
const coordinators = (id: string) => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, id));
const sandbox = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", reusableLeaseConfigured: false };
it.each(["uninspected", "confirmed", "replaced"])("selects an independent native session after %s disk-only recovery", async placement => {
const { input, run, issue } = await fixture();
const priorSession = randomUUID();
const lease = input.workspace.activeEnvironmentLease!.lease;
const description = { kind: "description" as const, taskId: randomUUID(), runner: { version: "test", digest: `sha256:${"a".repeat(64)}` } };
const previousDescription = placement === "replaced" ? { ...description, taskId: randomUUID() } : description;
const pending = { ...lease, providerLeaseId: description.taskId, metadata: { portableTaskPreviousLeaseId: randomUUID(),
portableTaskPredecessor: { operation: { identity: { normalizedSessionId: priorSession } } },
} };
const ready = { ...pending.metadata, portableTaskOwnershipReady: true, portableTaskCheckpointUnavailable: true };
await db.insert(environmentLeases).values(placement === "confirmed" ? { ...pending, metadata: ready } : pending);
input.workspace.activeEnvironmentLease!.lease = pending;
if (placement !== "uninspected") input.workspace.activeEnvironmentLease!.pluginTaskDescription = previousDescription;
const taskRpc = vi.fn(async ({ kind }: { kind: string }) => {
expect(kind).toBe("describe");
await db.update(environmentLeases).set({
metadata: ready,
}).where(eq(environmentLeases.id, lease.id));
return description;
});
input.workspace.executionTarget = { ...sandbox, leaseId: lease.id, runnerTask: taskRpc,
reusableLeaseConfigured: true, effectiveCapabilities: { reusableLeases: true },
runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } } as never;
input.session.taskResumeSessionId = priorSession;
const network = vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("Cloud is unavailable"));
try {
const selected = await selectHeartbeatRuntime(db, input);
expect(selected.selected).toBe(true);
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
const saved = await read(run.id);
expect(saved.nativeSessionId).toBeTruthy();
expect(saved.nativeSessionId).not.toBe(priorSession);
expect(await selected.getNativeFreshSessionHandoff?.()).toBe("Earlier context");
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0]).toEqual(issue);
expect(selected.pluginTaskDescription).toEqual(description);
expect(taskRpc).toHaveBeenCalledTimes(placement === "confirmed" ? 0 : 1);
expect(network).not.toHaveBeenCalled();
// A lost reply after committing selection must allow the same replacement run to retry.
input.run = saved;
input.session.persistedRunnerProfile = saved.runnerProfileJson ?? {};
input.session.persistedNativeExecutionInput = selected.nativeExecution;
await expect(selectHeartbeatRuntime(db, input)).resolves.toMatchObject({
selected: true, nativeExecution: selected.nativeExecution,
});
input.run = { ...saved, nativeSessionId: priorSession };
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native_portable_reseed_session_already_attached");
} finally {
network.mockRestore();
}
});
it("rejects invalid local Runner configuration before native authority and retains the preparation owner", async () => {
const f = await fixture();
await claimPreparation(f);
@@ -623,10 +676,14 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
});
it("restores the immutable admitted input and saved checkpoint when newer context arrives", async () => {
using discovery = vi.spyOn(runner, "describeRunnerdNativeSessionBackend");
const { input, run } = await fixture();
const selected = await selectHeartbeatRuntime(db, input);
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
expect(selected.toolRefreshOnResume).toBe(true);
expect(discovery).toHaveBeenCalledTimes(1);
const saved = await read(run.id);
expect(saved.runnerProfileJson?.nativeToolRefreshOnResume).toBe(true);
const profile = { ...saved.runnerProfileJson, sessionCheckpoint: { retained: "checkpoint" } };
await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, run.id));
input.run = await read(run.id);
@@ -636,11 +693,13 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
input.task.safeWakeComments = [{ id: randomUUID(), body: "New direction belongs to a separate wake" }];
vi.mocked(buildNativeRuntimeContext).mockClear();
const resumed = await selectHeartbeatRuntime(db, input);
expect(resumed).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution, nativeRunnerInstanceId: selected.nativeRunnerInstanceId });
expect(resumed).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution, nativeRunnerInstanceId: selected.nativeRunnerInstanceId,
toolRefreshOnResume: true });
expect((await read(run.id)).runnerProfileJson).toMatchObject(profile);
expect(await db.select().from(completionContracts)).toHaveLength(1);
expect(await coordinators(run.id)).toHaveLength(1);
expect(buildNativeRuntimeContext).not.toHaveBeenCalled();
expect(discovery).toHaveBeenCalledTimes(1);
});
it("rejects an admitted input bound to another run without replacing the saved runtime", async () => {
@@ -39,6 +39,8 @@ import {
} from "@paperclipai/db";
import { getStartupTracer } from "../../instrumentation.js";
import { documentService } from "../documents.js";
import { readPortableTaskReseed } from "../plugin-task-ownership.js";
import type { PluginTaskDescription } from "../environment-task-runtime.js";
import { managedAgentProfileService } from "../managed-agent-profiles.js";
import { remoteAgentProfileService } from "../remote-agent-profiles.js";
import {
@@ -47,7 +49,6 @@ import {
buildNativeRuntimeContext,
ensureNativeCompletionContract,
findNativeSessionResumeRun,
isNativeSessionId,
isUnusedNativeSessionBootstrap,
isUnusedLegacyNativeRetryReplacement,
materializeNativeInteractionResponses,
@@ -117,9 +118,7 @@ export interface HeartbeatRuntimeSelectionInput {
session: {
persistedNativeExecutionInput: NativeExecutionInput | null;
persistedRunnerProfile: Record<string, unknown>;
taskSessionCredentialCompatible: boolean;
taskSessionDecodedParams: Record<string, unknown> | null;
taskSessionForRun: { lastRunId: string | null } | null;
taskResumeSessionId: string | null;
isFailedChatRunRetry: boolean;
getFreshSessionHandoff: (() => Promise<string | null>) | undefined;
};
@@ -127,7 +126,9 @@ export interface HeartbeatRuntimeSelectionInput {
persistedExecutionWorkspace: { id: string; metadata?: Record<string, unknown> | null } | null;
executionWorkspace: Omit<Parameters<typeof buildNativeExecutionInput>[0]["workspace"], "id">;
executionTarget: Parameters<typeof prepareNativeWorkspaceSync>[0]["target"];
activeEnvironmentLease: Pick<Parameters<typeof prepareNativeWorkspaceSync>[0], "lease"> | null;
activeEnvironmentLease: (Pick<Parameters<typeof prepareNativeWorkspaceSync>[0], "lease"> & {
pluginTaskDescription?: PluginTaskDescription;
}) | null;
isDotRun: boolean;
projectRepositoryPaths: string[];
onCheckpoint?: Parameters<typeof prepareNativeWorkspaceSync>[0]["onCheckpoint"];
@@ -161,6 +162,8 @@ export type HeartbeatRuntimeSelectionResult = { selected: false } | {
selected: true;
nativeExecution: NativeExecutionInput | null;
nativeRunnerInstanceId: string | null;
pluginTaskDescription?: PluginTaskDescription;
toolRefreshOnResume?: boolean;
getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
nativeWorkspaceSync: Awaited<ReturnType<typeof prepareNativeWorkspaceSync>>;
};
@@ -313,16 +316,17 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
onNativeLifecycleSelected, onProviderResourceDisposition, onNativeOwnershipHeld, stopControllerLease } = input;
const { issueRef, issueContext, context, executionContinuation, safeWakeComments, safeWakeCommentContext,
paperclipWakePayload, acceptedPlanContinuationWake } = input.task;
const { persistedNativeExecutionInput, persistedRunnerProfile, taskSessionCredentialCompatible, taskSessionDecodedParams,
taskSessionForRun, isFailedChatRunRetry, getFreshSessionHandoff } = input.session;
const { persistedNativeExecutionInput, persistedRunnerProfile, taskResumeSessionId, isFailedChatRunRetry, getFreshSessionHandoff } = input.session;
const { persistedExecutionWorkspace, executionWorkspace, executionTarget, activeEnvironmentLease, isDotRun, projectRepositoryPaths } = input.workspace;
const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config;
const runOptions = input.recovery;
const { providerTraceRequested, providerTraceCapture } = input.trace;
if (!isDotRun) assertEphemeralNativeRuntime(nativeRuntimeResolution.kind === "native");
let nativeExecution: NativeExecutionInput | null = null;
let toolRefreshOnResume: boolean | undefined;
let getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
let nativeRunnerInstanceId: string | null = null;
let pluginTaskDescription = activeEnvironmentLease?.pluginTaskDescription;
let nativeWorkspaceSync: Awaited<ReturnType<typeof prepareNativeWorkspaceSync>> = null;
let providerResourceDispositionForRun: ProviderResourceDisposition | undefined;
if (nativeRuntimeResolution.kind === "native") {
@@ -451,9 +455,20 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
immediateRequests: immediateCompletion.requests,
immediateRequestSources: immediateCompletion.sources,
}));
const taskNativeSessionId = !taskSessionCredentialCompatible ? null : readNonEmptyString(
taskSessionDecodedParams?.sessionId,
);
const portableLease = activeEnvironmentLease?.lease;
const runnerTask = executionTarget?.kind === "remote" && executionTarget.transport === "sandbox" &&
executionTarget.leaseId === portableLease?.id ? executionTarget.runnerTask : undefined;
const portable = portableLease && runnerTask ? await readPortableTaskReseed(db, {
companyId: agent.companyId, runId: run.id, lease: portableLease, description: pluginTaskDescription,
runnerTask,
}) : null;
if (portable) pluginTaskDescription = portable.description;
const reseed = portable?.reseed;
if (reseed && run.nativeSessionId && (!reseed.previousSessionId || run.nativeSessionId === reseed.previousSessionId ||
persistedNativeExecutionInput?.session.normalizedSessionId !== run.nativeSessionId)) {
throw new Error("native_portable_reseed_session_already_attached");
}
const reseedPortableSession = Boolean(reseed);
// Compatibility for native retry rows created before same-run restart
// recovery existed. Only an entirely unused replacement row may
// inherit its source checkpoint; any process/provider evidence on the
@@ -516,15 +531,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
: null;
const legacyRetrySessionId =
compatibleLegacyRetrySource?.nativeSessionId;
const taskResumeRunId =
taskSessionForRun?.lastRunId &&
taskSessionForRun.lastRunId !== run.id &&
isNativeSessionId(taskNativeSessionId)
? taskSessionForRun.lastRunId
: null;
const resumableTaskSessionId = isDotRun ? null : taskResumeRunId
? taskNativeSessionId
: (legacyRetrySessionId ?? null);
const resumableTaskSessionId = reseedPortableSession || isDotRun ? null : taskResumeSessionId ?? legacyRetrySessionId ?? null;
const requestedNativeSessionId =
run.nativeSessionId ?? resumableTaskSessionId;
// A task-session lastRunId can lag a failed turn or point at an older
@@ -570,6 +577,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
const persistedProfile = persistedRunnerProfile;
if (persistedNativeExecutionInput) {
nativeExecution = persistedNativeExecutionInput;
toolRefreshOnResume = persistedProfile.nativeToolRefreshOnResume === true;
if (
nativeExecution.binding.companyId !== agent.companyId ||
nativeExecution.binding.runId !== run.id ||
@@ -896,11 +904,12 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
const backendDescriptor = await measureRunPhase("runtime_selection.provider_discovery", () => describeRunnerdNativeSessionBackend(buildExecution({
normalizedSessionId: nativeSessionId, resumedSession: previousNativeRun !== null,
})));
toolRefreshOnResume = backendDescriptor.capabilities.toolRefreshOnResume === true;
const nativeExecutionWithCheckpoint = buildNativeExecutionWithCheckpoint({
previousRun: previousNativeRun,
normalizedSessionId: nativeSessionId,
executionTargetKind: executionTarget?.kind ?? "local",
toolRefreshOnResume: backendDescriptor.capabilities.toolRefreshOnResume === true,
toolRefreshOnResume,
refreshTools: context.refreshTools === true,
buildExecution,
});
@@ -1037,6 +1046,8 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
: {}),
nativeExecutionInput:
lockedProfile.nativeExecutionInput ?? nativeExecution,
nativeToolRefreshOnResume: lockedProfile.nativeExecutionInput
? lockedProfile.nativeToolRefreshOnResume === true : toolRefreshOnResume === true,
...(typeof lockedProfile.inheritedProcessIdentityRunId === "string"
? { inheritedProcessIdentityRunId: lockedProfile.inheritedProcessIdentityRunId }
: lockedRun.processPid == null && lockedRun.processGroupId == null && lockedRun.processStartedAt == null
@@ -1167,5 +1178,5 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
})
.where(eq(heartbeatRuns.id, run.id)));
}
return { selected: true, nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff, nativeWorkspaceSync };
return { selected: true, nativeExecution, nativeRunnerInstanceId, pluginTaskDescription, toolRefreshOnResume, getNativeFreshSessionHandoff, nativeWorkspaceSync };
}
@@ -1,3 +1,4 @@
import { isNativeSessionId } from "../native-runtime/index.js";
import { and, desc, eq } from "drizzle-orm";
import { agentConfigRevisions, type Db, type agents, type agentTaskSessions, type heartbeatRuns, type issues, type projects } from "@paperclipai/db";
import type { AiConnectionBinding, AiRuntimeConnectionBinding } from "@paperclipai/shared";
@@ -29,7 +30,7 @@ type TaskSession = typeof agentTaskSessions.$inferSelect | null;
type SessionCodec = ReturnType<typeof getAdapterSessionCodec>;
export interface HeartbeatSessionPreparationInput {
run: Pick<typeof heartbeatRuns.$inferSelect, "createdAt">;
run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "createdAt">;
agent: typeof agents.$inferSelect;
task: {
context: Record<string, unknown>;
@@ -204,6 +205,14 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre
const sessionResetReason =
sessionConfigFreshness.reasons.join("; ") || null;
const taskSessionForRun = resetTaskSession ? null : taskSession;
const taskSessionCredentialCompatible = !managedAiRuntime || managedAiSessionIdentityCompatible(
taskSession?.sessionParamsJson?.paperclipAiCredentialIdentity,
managedAiRuntime.sessionIdentity,
managedAiRuntime.identity,
);
const taskSessionId = readNonEmptyString(taskSessionDecodedParams?.sessionId);
const taskResumeSessionId = taskSessionCredentialCompatible && taskSessionForRun?.lastRunId &&
taskSessionForRun.lastRunId !== run.id && isNativeSessionId(taskSessionId) ? taskSessionId : null;
const getFreshSessionHandoff = issueRef ? createNativeSessionHandoffLoader({
db, companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id, before: run.createdAt,
throughCommentId: readNonEmptyString(context.conversationReplayThroughCommentId) ?? (context.interactionKind ? null : wakeCommentId),
@@ -230,7 +239,8 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre
const handoff = await getFreshSessionHandoff();
if (handoff) context.paperclipFreshSessionHandoffMarkdown = handoff;
}
return { sessionConfigMetadata, configuredModel, sessionConfigFreshness, resetTaskSession, sessionResetReason, taskSessionForRun, getFreshSessionHandoff, previousSessionParams };
return { sessionConfigMetadata, configuredModel, sessionConfigFreshness, resetTaskSession, sessionResetReason, taskSessionForRun,
taskSessionCredentialCompatible, taskResumeSessionId, getFreshSessionHandoff, previousSessionParams };
}
export interface HeartbeatAdapterSessionInput {
@@ -264,7 +274,7 @@ export async function resolveHeartbeatAdapterSession(input: HeartbeatAdapterSess
const { agent, evaluateSessionCompaction } = input;
const { context, taskKey, issueId, continuationSummary, executionContinuation } = input.task;
const { runtime, taskSession, sessionCodec, explicitResumeSessionDisplayId, runtimeSessionParams, managedAiRuntime, prepared } = input.session;
const { sessionConfigMetadata, sessionConfigFreshness, resetTaskSession, taskSessionForRun, previousSessionParams } = prepared;
const { sessionConfigMetadata, sessionConfigFreshness, resetTaskSession, taskSessionForRun, taskSessionCredentialCompatible, previousSessionParams } = prepared;
const { latestWorkspaceConfigMetadata, workspaceConfigFreshness, reusedExecutionWorkspace, resolvedWorkspaceReusePolicy, persistedExecutionWorkspace, runtimeWorkspaceWarnings, requestedShouldReuseExisting, workspaceReuseRequest } = input.workspace;
const runtimeSessionFallback =
taskKey || resetTaskSession
@@ -328,11 +338,6 @@ export async function resolveHeartbeatAdapterSession(input: HeartbeatAdapterSess
delete context.paperclipPreviousSessionId;
}
const taskSessionCredentialCompatible = !managedAiRuntime || managedAiSessionIdentityCompatible(
taskSession?.sessionParamsJson?.paperclipAiCredentialIdentity,
managedAiRuntime.sessionIdentity,
managedAiRuntime.identity,
);
if (managedAiRuntime) {
sessionConfigMetadata.aiCredentialIdentity = managedAiRuntime.sessionIdentity;
if (!taskSessionCredentialCompatible) {
@@ -396,5 +401,5 @@ export async function resolveHeartbeatAdapterSession(input: HeartbeatAdapterSess
},
};
return { previousSessionDisplayId, runtimeSessionParamsForAdapter, taskSessionCredentialCompatible, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction };
return { previousSessionDisplayId, runtimeSessionParamsForAdapter, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction };
}
@@ -104,6 +104,7 @@ const RETAINED_SANDBOX_GONE = new RegExp(`^${NATIVE_REMOTE_RECOVERY_LEASE_MISMAT
/** Prepares the workspace and environment; dispatch and terminal lease release stay with the caller. */
export interface HeartbeatWorkspacePreparationInput {
signal?: AbortSignal;
run: typeof heartbeatRuns.$inferSelect;
agent: typeof agents.$inferSelect;
task: {
@@ -145,6 +146,7 @@ export interface HeartbeatWorkspacePreparationInput {
configSnapshot: WorkspaceMetadataInput["configSnapshot"];
secretManifest: WorkspaceMetadataInput["secretManifest"];
managedAiCredentialIdentity?: string;
runnerTaskResumeSessionId?: string | null;
};
reuse: {
requestedShouldReuseExisting: boolean;
@@ -154,7 +156,7 @@ export interface HeartbeatWorkspacePreparationInput {
nativeRecoveryExecutionWorkspaceId: string | null;
persistedNativeExecutionInput: NativeExecutionInput | null;
isDotRun: boolean;
runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim };
runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; portablePreparationLeaseId?: string };
};
services: {
workspaceWorkers?: WorkspaceWorkers;
@@ -253,6 +255,12 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
appendRunEvent,
} = input.services;
const pluginTaskWorkspace = selectedEnvironmentForConfig?.driver === "plugin" && agent.adapterType === "paperclip_runner";
const restartRecovery = runOptions.nativeRestartRecovery;
const portablePreparationLeaseId = runOptions.portablePreparationLeaseId ??
(restartRecovery?.kind === "bootstrap_incomplete" ? restartRecovery.portablePreparationLeaseId : undefined);
if (portablePreparationLeaseId && (!pluginTaskWorkspace || controllerOnly)) {
throw remoteRecoveryLeaseMismatch("workspace_preparation", "environment_driver");
}
for (const workspaceId of new Set([existingExecutionWorkspace?.id, reusableExistingExecutionWorkspace?.id].filter(Boolean))) {
const [workspace] = await db.select({ metadata: executionWorkspacesTable.metadata }).from(executionWorkspacesTable)
.where(and(eq(executionWorkspacesTable.companyId, agent.companyId), eq(executionWorkspacesTable.id, workspaceId!)));
@@ -1301,7 +1309,8 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
await controllerLease.assertOwned();
const remoteRecovery = runOptions.nativeRestartRecovery?.kind === "reattach_remote_runner"
? runOptions.nativeRestartRecovery : null;
const pluginRecovery = remoteRecovery?.remote.kind === "plugin_task" ? remoteRecovery.remote : null;
const pluginRecovery = remoteRecovery?.remote.kind === "plugin_task" ? remoteRecovery.remote : portablePreparationLeaseId
? { kind: "plugin_task" as const, leaseId: portablePreparationLeaseId, preparation: true as const } : undefined;
const sandboxRecovery = remoteRecovery && remoteRecovery.remote.kind !== "plugin_task" ? remoteRecovery.remote : null;
const recoveryWorkspace = sandboxRecovery
? readNativeWorkspaceSyncReference(parseObject(run.runnerProfileJson).nativeWorkspaceSync) : null;
@@ -1340,10 +1349,12 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
adapterConfig: parseObject(agent.adapterConfig),
admittedLifecycleMode: persistedNativeExecutionInput?.session.lifecyclePolicy.mode,
managedAiCredentialIdentity: input.config.managedAiCredentialIdentity,
runnerTaskResumeSessionId: input.config.runnerTaskResumeSessionId,
runnerTaskProjectIds:[...new Set([executionWorkspace.projectId,...resolvedWorkspace.runnerTaskProjectIds ?? [],
...(executionWorkspace.additionalWorkspaces ?? []).map(workspace=>workspace.projectId)]
.filter((id):id is string=>Boolean(id)))],
recoverPluginTask: runOptions.nativeRestartRecovery?.kind === "resume_dead_runner" || runOptions.nativeRestartRecovery?.kind === "bootstrap_incomplete",
recoverPluginTask: !portablePreparationLeaseId &&
(restartRecovery?.kind === "resume_dead_runner" || restartRecovery?.kind === "bootstrap_incomplete"),
issueId: issueId ?? null,
heartbeatRunId: run.id,
agentId: agent.id,
@@ -1441,6 +1452,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
effectiveExecutionWorkspaceMode,
persistedExecutionWorkspace,
runnerTaskProjectIds: resolvedWorkspace.runnerTaskProjectIds,
...(input.signal ? { signal: input.signal } : {}),
assertOwned: remoteAuthority ? assertRemotePreparationOwned : controllerLease.assertOwned,
duplexObservabilityRecorder,
}));
+34 -15
View File
@@ -1,5 +1,5 @@
import { notifyLiveRunRecovery } from "./live-run-recovery-notifications.js";
import { and, eq, gt, lte, sql } from "drizzle-orm";
import { and, eq, gt, isNull, lte, sql, type SQL } from "drizzle-orm";
import { controllerLeaseDb, heartbeatRuns, type Db } from "@paperclipai/db";
import type { LegacyControllerLeaseLossCause, LegacyControllerLeaseLossTiming } from "./run-cancellation.js";
import { randomUUID } from "node:crypto";
@@ -176,7 +176,25 @@ export async function hasLiveLegacyController(db: Db, run: Run): Promise<boolean
return (await liveLegacyControllerExpiresAt(db, run)) !== null;
}
export type RevokedLegacyControllerState = Pick<Run, "executionStage" | "processPid" | "processGroupId">;
export type RevokedLegacyControllerState = Pick<Run, "executionStage" | "processPid" | "processGroupId"> & { portablePreparationLeaseId?: string };
function legacyPreparationStage() {
return and(eq(heartbeatRuns.runtimeMode, "legacy"), eq(heartbeatRuns.executionStage, "preparing"),
isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), isNull(heartbeatRuns.processStartedAt));
}
export async function handoffLegacyControllerPreparation(db: Db, run: Run, preparationLease: SQL<string | null>, onHandoff: () => void) {
return db.transaction(async tx => {
await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id });
const [released] = await tx.update(heartbeatRuns).set({ controllerLeaseExpiresAt: sql`clock_timestamp()` })
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.controllerBootId, legacyControllerBootId),
legacyPreparationStage(), sql`${preparationLease} is not null`)).returning({ id: heartbeatRuns.id });
if (!released) return false;
onHandoff();
return true;
});
}
/** Atomically revoke an expired controller. Renewal and revocation serialize on
* the run row. Expiry permits cleanup, never dispatch of a replacement agent.
@@ -186,6 +204,7 @@ export type RevokedLegacyControllerState = Pick<Run, "executionStage" | "process
export async function revokeExpiredLegacyControllerWithState(
db: Db,
run: Run,
reclaimPreparation?: SQL<string | null>,
): Promise<RevokedLegacyControllerState | null> {
if (run.runtimeMode === "native" || !run.controllerBootId) {
return { executionStage: run.executionStage, processPid: run.processPid, processGroupId: run.processGroupId };
@@ -193,19 +212,19 @@ export async function revokeExpiredLegacyControllerWithState(
const controllerBootId = run.controllerBootId;
const [revoked] = await db.transaction(async tx => {
await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id });
return tx.update(heartbeatRuns).set({
controllerBootId: randomUUID(),
controllerLeaseExpiresAt: sql`clock_timestamp() + interval '60 seconds'`,
}).where(and(
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
eq(heartbeatRuns.status, "running"),
eq(heartbeatRuns.controllerBootId, controllerBootId),
lte(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`),
)).returning({
executionStage: heartbeatRuns.executionStage,
processPid: heartbeatRuns.processPid,
processGroupId: heartbeatRuns.processGroupId,
});
const expired = and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.controllerBootId, controllerBootId),
lte(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`));
const state = { executionStage: heartbeatRuns.executionStage, processPid: heartbeatRuns.processPid,
processGroupId: heartbeatRuns.processGroupId };
if (reclaimPreparation) {
const [reclaimed] = await tx.update(heartbeatRuns).set(legacyControllerClaim("legacy")).where(and(expired, legacyPreparationStage(),
sql`${reclaimPreparation} is not null`)).returning({ ...state, portablePreparationLeaseId: reclaimPreparation });
if (reclaimed?.portablePreparationLeaseId) return [{ ...reclaimed, portablePreparationLeaseId: reclaimed.portablePreparationLeaseId }];
}
return tx.update(heartbeatRuns).set({ controllerBootId: randomUUID(),
controllerLeaseExpiresAt: sql`clock_timestamp() + interval '60 seconds'`,
}).where(expired).returning(state);
});
return revoked ?? null;
}
@@ -509,8 +509,16 @@ describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable d
await db.update(nativeRunFinalizations).set({ phase: "committed", leaseOwner: null, leaseExpiresAt: null })
.where(eq(nativeRunFinalizations.runId, fixture.scope.runId));
await expect(readSandbox()).resolves.toEqual({ remoteCwd: "/workspace" });
await expect(inspect({ readControlState: () => ({ identity, commands: [{ status: "pending" }],
committedEvents: [{ eventType: "run.terminal", envelope: identity }] }) })).rejects.toBeInstanceOf(NativePredecessorRetirementUnprovenError);
const readUnsettledControlState = () => ({ identity, commands: [{ status: "pending" }],
committedEvents: [{ eventType: "run.terminal", envelope: identity }] });
await expect(inspect({ readControlState: readUnsettledControlState }))
.rejects.toBeInstanceOf(NativePredecessorRetirementUnprovenError);
const movedFailure = await inspect({
target: { ...target, sandboxLeaseAcquisition: { outcome: "replacement", providerLeaseId: "new-sandbox" } } as never,
portableSourceTaskId: "retained-sandbox", readControlState: readUnsettledControlState,
}).catch(error => error);
expect(movedFailure).not.toBeInstanceOf(NativePredecessorRetirementUnprovenError);
expect(movedFailure.message).toBe(`${denied}; warm_recovery_refused=journal_unsettled`);
// A created or replacement sandbox holds no Runner to adopt. Live adoption
// steps aside so ordinary admission can retire the finished owner, instead
// of denying with prior_owner_terminal_state_indeterminate (Nov 10 staging).
@@ -123,6 +123,7 @@ export type NativeRestartRecoveryClaim =
providerAttempt: number;
restartKind: NativeRestartKind;
recoveryRequestId: string | null;
portablePreparationLeaseId?: string;
/** False when the prior controller suspended this run in a clean drain. */
chargesProviderAttempt?: boolean;
};
@@ -1546,6 +1547,12 @@ export async function claimNativeRestartRecoveries(input: {
const continuedSessionOnly = hasCheckpoint && checkpointIdentityMatches && providerEvents.length === 0
&& !pendingInheritedRequests && checkpointRecord.activeTurnId == null && checkpointRecord.semanticResult == null
&& checkpointRecord.terminal == null;
const portablePreparation = pluginTask?.unlaunchedPortable === true && row.run.status === "running" &&
row.coordinator.phase === "observed" && !durableAuthority && !row.run.sessionIdAfter &&
row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION &&
(hasCheckpoint ? admissionCheckpointOnly && providerEvents.length === 0 : !hasProviderEvidence) &&
(!row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt || await carriesInheritedRunnerIdentity(tx, row.run)) &&
!(await hasNativeExecutionStartEvidence(tx, row.run));
const remoteRunnerNeverLaunched = !unclaimedBootstrap && remoteWorkspace?.state === "prepared" && !pluginTask
&& !durableAuthority && stagedAttachment === null
&& row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION
@@ -1672,11 +1679,11 @@ export async function claimNativeRestartRecoveries(input: {
: await restartResumeKeepsAttempt(tx, { runId: row.run.id, coordinator: row.coordinator });
const resumeKeepsAttempt = restartCredit.keepsAttempt;
const classification = classifyNativeRunnerRecoveryEvidence({
remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority),
unclaimedBootstrap,
remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority) || portablePreparation,
unclaimedBootstrap: unclaimedBootstrap || portablePreparation && row.coordinator.attempt === 0,
remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched
|| remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands || remoteFollowUpNeverAttached,
remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched ? "remote_runner_never_launched"
|| remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands || remoteFollowUpNeverAttached || portablePreparation,
remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched || portablePreparation ? "remote_runner_never_launched"
: remoteRunnerExitedBeforeProviderWork ? "remote_runner_exited_before_provider_work"
: remoteRunnerNeverReceivedCommands ? "remote_runner_never_received_commands"
: remoteFollowUpNeverAttached ? "remote_follow_up_never_attached"
@@ -1687,13 +1694,13 @@ export async function claimNativeRestartRecoveries(input: {
knownProviderProcessAlive: providerProcesses.livePids.length > 0,
knownProviderProcessIdentityAmbiguous:
providerProcesses.ambiguousLivePids.length > 0,
hasCheckpoint: hasCheckpoint && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly)
hasCheckpoint: hasCheckpoint && !portablePreparation && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly)
&& !remoteRunnerNeverLaunched && !remoteFollowUpNeverAttached && !(remoteBootstrapProvenIncomplete && reboundSessionOnly),
stagedAttachmentVerified: stagedAttachment !== null,
checkpointIdentityMatches:
checkpointIdentityMatches &&
(!remoteSandbox || remoteWorkspace !== null),
hasProviderEvidence: hasProviderEvidence && !(unclaimedBootstrap && inheritedCheckpointOnly),
hasProviderEvidence: hasProviderEvidence && !portablePreparation && !(unclaimedBootstrap && inheritedCheckpointOnly),
checkpointFailed:
(checkpointRecord.terminal as Record<string, unknown> | undefined)
?.runTerminalState === "failed" &&
@@ -1893,7 +1900,7 @@ export async function claimNativeRestartRecoveries(input: {
leaseOwner,
// Cleanup and earlier candidates can outlive the batch observation.
// The owner row is locked: grant the full lease at this write.
leaseExpiresAt: sql`clock_timestamp() + ${durableAuthority ? REMOTE_CONTROLLER_LEASE_MS : 20 * 60_000} * interval '1 millisecond'`,
leaseExpiresAt: sql`clock_timestamp() + ${durableAuthority || portablePreparation ? REMOTE_CONTROLLER_LEASE_MS : 20 * 60_000} * interval '1 millisecond'`,
controllerBootId: controller.bootId,
controllerHostHash,
controllerPid: controller.pid,
@@ -1981,7 +1988,7 @@ export async function claimNativeRestartRecoveries(input: {
await restoreNativeSessionExecution(tx as unknown as Db, { issue, run: row.run, agentName: resumptionAgent.name, now });
}
claimScope = { companyId: row.run.companyId, issueId: row.coordinator.issueId, controller, remote: Boolean(durableAuthority) };
claimScope = { companyId: row.run.companyId, issueId: row.coordinator.issueId, controller, remote: Boolean(durableAuthority || portablePreparation) };
const common = {
runId: row.run.id,
leaseOwner,
@@ -2021,6 +2028,7 @@ export async function claimNativeRestartRecoveries(input: {
return {
kind: claimKind,
...common,
...(claimKind === "bootstrap_incomplete" && portablePreparation && pluginTask ? { portablePreparationLeaseId: pluginTask.id } : {}),
} satisfies NativeRestartRecoveryClaim;
}).catch((error: unknown) => {
// The whole candidate transaction rolled back, so this run's state is
@@ -3,6 +3,7 @@ import { environmentLeases, heartbeatRuns, nativeRunFinalizations, type Db } fro
import { and, desc, eq, inArray, isNull, or, sql } from "drizzle-orm";
import { remoteExecutionHasStopped } from "../remote-execution-termination.js";
import { readSystemRecovery } from "../system-recovery-state.js";
import { unlaunchedPortableTaskLease } from "../plugin-task-ownership.js";
type NativeRemoteExecutionScope = {
run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "runnerProfileJson">;
@@ -13,6 +14,7 @@ export async function readNativeRemoteExecutionOwnershipForRuns(db: Db, inputs:
const key = (companyId: string, issueId: string, runId: string) => `${companyId}:${issueId}:${runId}`;
const leases = inputs.length ? await db.select({ id: environmentLeases.id, companyId: environmentLeases.companyId,
issueId: environmentLeases.issueId, runId: environmentLeases.heartbeatRunId,
unlaunchedPortable: unlaunchedPortableTaskLease(),
metadata: sql<Record<string, unknown>>`jsonb_build_object('runnerTaskStartup', ${environmentLeases.metadata}->'runnerTaskStartup')`,
}).from(environmentLeases).where(and(
or(...inputs.map(input => and(eq(environmentLeases.companyId, input.run.companyId),
@@ -9,8 +9,10 @@ import * as stagedAttachmentRecovery from "./native-staged-attachment-recovery.j
import * as controllerRecovery from "./native-controller-recovery.js";
import { isSettledRemoteWarmControlState } from "./remote-warm-session-recovery.js";
import * as remoteWarmRecoveryRuntime from "./remote-warm-session-recovery.js";
import * as pluginTaskOwnership from "../plugin-task-ownership.js";
import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js";
import * as environmentTaskRuntime from "../environment-task-runtime.js";
import * as pluginCredentialWriteback from "../plugin-task-credential-writeback.js";
import * as pluginTaskRuntime from "./plugin-runner-task.js";
import * as completionFeedback from "./native-completion-feedback.js";
import { environmentTaskOperationSchema } from "@paperclipai/plugin-sdk";
@@ -19,7 +21,7 @@ import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@open
import { configuredEnvironment, configuredEnvironmentProjection } from "../../vendor/paperclip-runner/index.js";
import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from "vitest";
import { createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "../system-recovery-state.js";
import { readFileSync } from "node:fs";
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { once } from "node:events";
import { PgDialect } from "drizzle-orm/pg-core";
import type { SQL } from "drizzle-orm";
@@ -14176,15 +14178,20 @@ describe("native warm session supervision", () => {
{ name: "rotated provider key", firstEnvironment: { OPENAI_API_KEY: "fixture-key" }, secondEnvironment: { OPENAI_API_KEY: "fixture-rotated-key" }, reuse: false },
{ name: "rotated Claude token", firstEnvironment: { CLAUDE_CODE_OAUTH_TOKEN: "fixture-token" }, secondEnvironment: { CLAUDE_CODE_OAUTH_TOKEN: "fixture-rotated-token" }, reuse: false },
{ name: "tool refresh", refreshTools: true, reuse: false },
{ name: "unsupported tool refresh", refreshTools: true, unsupportedRefresh: true, reuse: false },
{ name: "network access change", firstEnvironment: { PAPERCLIP_RUNNER_NETWORK_ACCESS: "disabled" }, secondEnvironment: { PAPERCLIP_RUNNER_NETWORK_ACCESS: "enabled" }, reuse: false },
{ name: "run-scoped environment broker", firstEnvironment: { PAPERCLIP_GITHUB_BROKER_TOKEN: "first-run-capability" }, secondEnvironment: { PAPERCLIP_GITHUB_BROKER_TOKEN: "second-run-capability" }, reuse: false },
{ name: "Core-hosted GitHub access", managedGitHub: true, reuse: true, adoptionReuse: false },
] as Array<{ name: string; firstEnvironment?: Record<string, string>; secondEnvironment?: Record<string, string>;
refreshTools?: boolean; managedGitHub?: boolean; reuse: boolean; adoptionReuse?: boolean }>)(
refreshTools?: boolean; unsupportedRefresh?: boolean; managedGitHub?: boolean; reuse: boolean; adoptionReuse?: boolean }>)(
"decides warm and adopted process reuse from one key: $name",
async ({ name, firstEnvironment = {}, secondEnvironment = {}, refreshTools, managedGitHub = false, reuse, adoptionReuse = reuse }) => {
async ({ name, firstEnvironment = {}, secondEnvironment = {}, refreshTools, unsupportedRefresh = false,
managedGitHub = false, reuse, adoptionReuse = reuse }) => {
const decisions: Record<string, boolean> = {};
for (const path of ["warm", "adoption"] as const) {
for (const path of ["warm", "adoption", ...(refreshTools ? ["portable", "portable-bootstrap", "legacy-plugin"] as const : [])] as const) {
const portable = path === "portable" || path === "portable-bootstrap";
const bootstrap = path === "portable-bootstrap";
const pluginRecovery = portable || path === "legacy-plugin";
githubAccess.create.mockReset().mockImplementation(async () => ({ env: {}, ready: true, duplexRequested: false,
activate: githubAccess.activate.mockReset().mockImplementation(() => vi.fn()), stop: githubAccess.stop.mockReset().mockResolvedValue(undefined) }));
const stateBase = await mkdtemp(join(tmpdir(), "paperclip-runnerd-reuse-key-"));
@@ -14195,15 +14202,24 @@ describe("native warm session supervision", () => {
const scope = `${path}-${name.replaceAll(" ", "-")}`;
const first = {
...execution,
...(pluginRecovery ? { schema: "paperclip.native-execution-input.v5", executionMode: "default",
provider: { ...execution.provider, approvalPolicy: "never" }, planningContext: null, runtimeContext: nativeRuntimeContextFixture() } as const : {}),
binding: { ...execution.binding, runId: `run-reuse-key-first-${scope}`, executionWorkspaceId: "workspace-reuse-key" },
workspace: { cwd: "/tmp/runnerd-reuse-key", repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: `session-reuse-key-${scope}`, driverKind: "codex_app_server" as const,
protocolVersion: 1 as const, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
} as NativeExecutionInputV1;
const second = { ...first, binding: { ...first.binding, runId: `run-reuse-key-second-${scope}` } } as NativeExecutionInputV1;
} as NativeExecutionInput;
if (pluginRecovery) parseNativeExecutionInput(first);
const second = { ...first, binding: { ...first.binding, runId: `run-reuse-key-second-${scope}` } };
if ("runtimeContext" in second) {
second.runtimeContext = structuredClone(second.runtimeContext);
second.runtimeContext.mcp.digest = "1".repeat(64);
second.runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(second.runtimeContext);
}
const target = { kind: "remote" as const, transport: "sandbox" as const, environmentId: "environment-reuse-key",
remoteCwd: "/home/daytona/paperclip-workspace", runner: { execute: vi.fn() } } as never;
remoteCwd: "/home/daytona/paperclip-workspace", runner: { execute: vi.fn() } };
const firstSession = { close: vi.fn(async () => undefined) };
const recoveredSession = { close: vi.fn(async () => undefined), snapshot: vi.fn() };
const result = { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" }, turnId: "turn",
normalizedSessionId: first.session.normalizedSessionId, providerSessionId: "provider-reuse-key", driverKind: "test",
driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
@@ -14219,36 +14235,138 @@ describe("native warm session supervision", () => {
return result;
})
.mockImplementationOnce(async (options) => {
decisions[path] = options.existingSession === firstSession;
decisions[path] = options.existingSession === (portable ? recoveredSession : firstSession);
return result;
});
const inspected = vi.spyOn(remoteWarmRecoveryRuntime, "inspectRemoteWarmSessionRecovery").mockImplementation(async request => {
decisions[path] = request.canReuseProcess(first);
const reuseProcess = request.canReuseProcess(first);
if (pluginRecovery) {
expect(reuseProcess).toBe(false);
expect(request.matchesExecution(first)).toBe(!unsupportedRefresh);
if (unsupportedRefresh) throw new Error("fixture unsupported tool refresh refused");
expect(request.matchesExecution({ ...first, provider: { ...first.provider, model: "another-model" } } as NativeExecutionInput)).toBe(false);
if ("runtimeContext" in first) {
const changed = structuredClone(first);
changed.runtimeContext.instructions.bundle.digest = "2".repeat(64);
changed.runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(changed.runtimeContext);
expect(request.matchesExecution(changed)).toBe(false);
}
const checkpoint = request.loadCheckpoint(first);
expect(checkpoint?.identity.runId).toBe(first.binding.runId);
recoveredSession.snapshot.mockResolvedValue(checkpoint);
return { identity: request.identity as Record<string, string>, checkpoint: checkpoint!,
previousExecution: first, reuseProcess, ...(portable ? { portableSourceTaskId: "source-task" } : {}),
remote: { providerLeaseId: "target-task", remoteCwd: target.remoteCwd }, revalidate: async () => {} };
}
decisions[path] = reuseProcess;
throw new Error("fixture adoption decision recorded");
});
const ownership = portable
? vi.spyOn(pluginTaskOwnership, "readyPortableTaskSource").mockImplementation(async (_db, _company, _lease, identity, scope) => {
expect(scope?.accountIdentity).toBe("fixture-account");
return { taskId: "source-task", identity: identity as NonNullable<Awaited<ReturnType<typeof pluginTaskOwnership.readyPortableTaskSource>>>["identity"] };
}) : undefined;
const projection = vi.spyOn(controllerRecovery, "resolveNativeControllerRecovery").mockResolvedValue({ kind: "no_prior_execution" });
const restore = path === "legacy-plugin" ? vi.spyOn(runnerCheckpoint, "restoreNativeRunnerCheckpoint")
.mockImplementation(async (_db, binding) => binding.runId === first.binding.runId && recoveredSession.close.mock.calls.length ? {} as never : null) : undefined;
const issueProjection = path === "legacy-plugin" ? vi.spyOn(issueServiceModule, "issueService")
.mockReturnValue({ update: vi.fn(async () => null) } as never) : undefined;
try {
await executePaperclipNativeSession({ db: leaseDb(first), execution: first, runnerInstanceId: "runner-reuse-key",
useRunnerd: true, managedGitHub, runnerExecutionTarget: target,
useRunnerd: true, managedGitHub, runnerExecutionTarget: target as never,
...(portable ? { managedAiAccountIdentity: "fixture-account", managedAiCredentialIdentity: "old-revision" } : {}),
runnerEnvironment: { PAPERCLIP_GITHUB_AUTH_MODE: "managed", ...firstEnvironment } });
const durableRoot = (await readdir(stateBase, { withFileTypes: true }))
.filter(entry => entry.isDirectory() && /^[a-f0-9]{64}$/.test(entry.name)).map(entry => join(stateBase, entry.name))[0]!;
const durableIdentity = { runId: first.binding.runId, normalizedSessionId: first.session.normalizedSessionId,
runnerInstanceId: "runner-reuse-key", environmentLeaseId: first.binding.executionWorkspaceId };
runnerInstanceId: "runner-reuse-key", environmentLeaseId: first.binding.executionWorkspaceId,
turnId: "prior-turn", itemId: "prior-item" };
await mkdir(join(durableRoot, "control-plane"), { recursive: true });
await writeFile(join(durableRoot, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(durableIdentity)));
await mkdir(join(durableRoot, "runner"), { recursive: true });
await writeFile(join(durableRoot, "runner", "runner-state.json"), JSON.stringify(durableRunnerState(durableIdentity, "ready")));
if (path === "adoption") expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
if (path !== "warm") expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
if (portable) {
const directory = join(resolvePaperclipInstanceRoot(), "runtime/paperclip-runner/sessions");
const path = join(directory, (await readdir(directory))[0]!);
const { snapshot, configDigest, processConfigDigest } = JSON.parse(await readFile(path, "utf8"));
await rm(path);
await rm(durableRoot, { recursive: true });
projection.mockResolvedValueOnce({
kind: "recoverable", snapshot,
metadata: controllerRecovery.createNativeControllerRecovery(snapshot, { configDigest, processConfigDigest }, null),
checkpoint: { restoreSessionDirectory(root: string) {
mkdirSync(join(root, "control-plane"), { recursive: true });
writeFileSync(join(root, "control-plane/control-plane-state.json"), JSON.stringify(durableControlPlaneState(durableIdentity)));
} } as never,
});
}
if (pluginRecovery) state.createBackend.mockReturnValueOnce({
kind: "test", recoverSession: async () => ({ recovered: true, session: recoveredSession }),
} as never);
const pluginTaskDescription = { kind: "description", taskId: "target-task", warmTaskVersion: 1,
launchPreparationVersion: 1, projectAuthCheckVersion: 1, releaseResumeVersion: 1,
runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } } as const;
const portableTarget = { ...target, runner: undefined, leaseId: "target-lease",
sandboxLeaseAcquisition: { outcome: path === "legacy-plugin" ? "resumed" : "created", providerLeaseId: "target-task" },
runnerTask: vi.fn(async (operation: { kind: string }) => {
if (path === "legacy-plugin") {
expect(recoveredSession.close).toHaveBeenCalledOnce();
expect(operation.kind).toBe("describe");
throw new Error("fixture replacement inspection reached");
}
return pluginTaskDescription;
}),
};
if (pluginRecovery) await writeFile(join(stateBase, "auth.json"),
JSON.stringify({ OPENAI_API_KEY: "synthetic-fixture" }), { mode: 0o600 });
const leaseOwner = bootstrap ? "bootstrap-recovery-owner" : undefined;
const controller = bootstrap ? await import("./native-restart-recovery.js").then(module => module.currentNativeControllerIdentity()) : undefined;
const db = leaseDb(second, controller ? { leaseOwner, controllerBootId: controller.bootId,
controllerPid: controller.pid, controllerGeneration: 1, leaseExpiresAt: new Date(Date.now() + 60_000) } : {});
const select = db.select.bind(db);
db.select = ((fields?: Record<string, unknown>) => {
if (fields?.status === heartbeatRuns.status && fields?.runnerProfileJson === heartbeatRuns.runnerProfileJson) {
return { from: () => ({ where: () => ({ limit: async () =>
[{ status: "succeeded", runnerProfileJson: { nativeExecutionInput: first } }] }) }) };
}
if (pluginRecovery && fields?.metadata === environmentLeases.metadata) {
return { from: () => ({ where: async () => [{ providerLeaseId: pluginTaskDescription.taskId, unlaunchedPortable: bootstrap,
metadata: ownership ? { portableTaskPreviousLeaseId: "source-lease", portableTaskOwnershipReady: true } : {} }] }) };
}
return select(fields as never);
}) as Db["select"];
const continuation = executePaperclipNativeSession({
db: { ...leaseDb(second), select: () => ({ from: () => ({ where: () => ({ limit: () =>
Promise.resolve([{ status: "succeeded", runnerProfileJson: { nativeExecutionInput: first } }]) }) }) }) } as unknown as Db,
execution: second, runnerInstanceId: "runner-reuse-key", useRunnerd: true, managedGitHub, refreshTools,
runnerExecutionTarget: target, runnerEnvironment: { PAPERCLIP_GITHUB_AUTH_MODE: "managed", ...secondEnvironment } });
db, execution: second, runnerInstanceId: "runner-reuse-key", useRunnerd: true, managedGitHub, refreshTools, toolRefreshOnResume: !unsupportedRefresh,
runnerExecutionTarget: (pluginRecovery ? portableTarget : target) as never,
...(leaseOwner ? { leaseOwner, restartRecovery: { kind: "bootstrap_incomplete" as const, runId: second.binding.runId,
leaseOwner, controllerGeneration: 1, providerAttempt: 0, restartKind: "hard" as const, recoveryRequestId: null,
portablePreparationLeaseId: portableTarget.leaseId } } : {}),
...(pluginRecovery ? { pluginTaskDescription, runnerIngressAuthorized: true, managedAiCredentialHome: stateBase,
managedAiAccountIdentity: "fixture-account", managedAiCredentialIdentity: "current-revision" } : {}),
runnerEnvironment: { PAPERCLIP_GITHUB_AUTH_MODE: "managed", ...secondEnvironment } });
if (pluginRecovery && unsupportedRefresh) {
await expect(continuation).rejects.toThrow("fixture unsupported tool refresh refused");
continue;
}
if (path === "legacy-plugin") {
await expect(continuation).rejects.toThrow("fixture replacement inspection reached");
expect(portableTarget.runnerTask).toHaveBeenCalledOnce();
continue;
}
if (path === "adoption") await expect(continuation).rejects.toThrow("fixture adoption decision recorded");
else await continuation;
expect(decisions[path]).toBe(path === "adoption" ? adoptionReuse : reuse);
expect(decisions[path]).toBe(portable ? true : path === "adoption" ? adoptionReuse : reuse);
if (portable) {
expect(portableTarget.runnerTask).not.toHaveBeenCalled();
expect(recoveredSession.close).not.toHaveBeenCalled();
}
if (path === "warm") expect(firstSession.close).toHaveBeenCalledTimes(reuse ? 0 : 1);
} finally {
projection.mockRestore();
restore?.mockRestore();
issueProjection?.mockRestore();
ownership?.mockRestore();
inspected.mockRestore();
await closeIdleWarmNativeSessionsForRestart();
if (previousStateDirectory === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR;
@@ -14258,7 +14376,7 @@ describe("native warm session supervision", () => {
await rm(stateBase, { recursive: true, force: true });
}
}
expect(decisions).toEqual({ warm: reuse, adoption: adoptionReuse });
expect(decisions).toEqual({ warm: reuse, adoption: adoptionReuse, ...(refreshTools && !unsupportedRefresh ? { portable: true, "portable-bootstrap": true } : {}) });
},
);
@@ -17963,8 +18081,9 @@ describe("runnerd provider runtime wiring", () => {
} finally { resume.mockRestore(); }
});
it.each(["verified", "wrong-state", "wrong-lease"] as const)("reattaches a surviving plugin task through its owner: %s", async mode => {
it.each(["verified", "portable", "wrong-claim", "wrong-state", "wrong-lease"] as const)("reattaches a surviving plugin task through its owner: %s", async mode => {
const current = scopedNativeExecution(`plugin-live-${mode}`);
const moved = mode === "portable" || mode === "wrong-claim";
const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!,
runnerInstanceId: "plugin-runner", environmentLeaseId: "same-task-lease", turnId: "admitted-turn", itemId: "admitted-item" };
const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId);
@@ -17980,24 +18099,35 @@ describe("runnerd provider runtime wiring", () => {
runner: { version: "new-artifact", digest: `sha256:${"b".repeat(64)}` } };
});
state.createBackend.mockClear(); state.createTransport.mockClear();
const backend = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
runnerIngressAuthorized: true, runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".",
leaseId: mode === "wrong-lease" ? "replacement-task" : identity.environmentLeaseId, runnerTask: task },
controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-artifact", digest: `sha256:${"a".repeat(64)}` } },
restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller", controllerGeneration: 2,
providerAttempt: 1, restartKind: "hard", recoveryRequestId: null, remote: { kind: "plugin_task", leaseId: identity.environmentLeaseId } } });
if (mode !== "verified") {
await expect(backend).rejects.toThrow(mode === "wrong-state" ? "adoption identity mismatch" : "recovery_lease_mismatch");
expect(state.createBackend).not.toHaveBeenCalled();
return;
}
await backend;
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
const options = state.createTransport.mock.calls.at(-1)![0];
expect(options).toMatchObject({ prpIdentity: identity, adoptExistingRunner: { pid: null, startedAt: null },
runnerArtifact: { version: "admitted-artifact" } });
expect(options).toMatchObject({ runnerReconnectGraceMs: 420_000 });
expect(task.mock.calls.some(([op]) => ["prepare", "submit", "stop"].includes(op.kind))).toBe(false);
const portable = moved ? vi.spyOn(pluginTaskOwnership, "assertAttachedPortableTask")
.mockImplementation(async () => {
if (mode === "wrong-claim") throw new Error("native_portable_task_ownership_unverified");
}) : undefined;
const leaseId = moved ? "moved-task-lease" : identity.environmentLeaseId;
try {
const backend = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
runnerIngressAuthorized: true, runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".",
leaseId: mode === "wrong-lease" ? "replacement-task" : leaseId, runnerTask: task,
...(moved ? { sandboxLeaseAcquisition: { outcome: "resumed" as const, providerLeaseId: "same-task" } } : {}) },
controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-artifact", digest: `sha256:${"a".repeat(64)}` } },
restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller", controllerGeneration: 2,
providerAttempt: 1, restartKind: "hard", recoveryRequestId: null, remote: { kind: "plugin_task", leaseId } } });
if (mode !== "verified" && mode !== "portable") {
await expect(backend).rejects.toThrow(mode === "wrong-state" ? "adoption identity mismatch" : mode === "wrong-claim" ? "ownership_unverified" : "recovery_lease_mismatch");
expect(state.createBackend).not.toHaveBeenCalled();
return;
}
await backend;
if (moved) expect(portable).toHaveBeenCalledWith(expect.anything(), {
companyId: current.binding.companyId, leaseId, taskId: "same-task", runId: current.binding.runId, identity,
});
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
const options = state.createTransport.mock.calls.at(-1)![0];
expect(options).toMatchObject({ prpIdentity: identity, adoptExistingRunner: { pid: null, startedAt: null },
runnerArtifact: { version: "admitted-artifact" } });
expect(options).toMatchObject({ runnerReconnectGraceMs: 420_000 });
expect(task.mock.calls.some(([op]) => ["prepare", "submit", "stop"].includes(op.kind))).toBe(false);
} finally { portable?.mockRestore(); }
});
it.each(["codex", "acpx-codex", "grok"] as const)("writes managed %s events through the current warm exporter without per-turn reads", async provider => {
@@ -18175,8 +18305,11 @@ describe("runnerd provider runtime wiring", () => {
} finally {owner.mockRestore();await rm(home,{recursive:true,force:true});}
});
it.each((["codex", "claude"] as const).flatMap(provider => [true, false].map(reuseProcess => [provider, reuseProcess] as const)))("reconnects task-only %s warm recovery under the saved identity before reuse or retirement: %s", async (provider, reuseProcess) => {
const current=scopedNativeExecution(`plugin-restart-${provider}-${reuseProcess}`);
it.each([
...(["codex", "claude"] as const).flatMap(provider => [true, false].map(reuseProcess => [provider, reuseProcess, false] as const)),
["codex", true, true] as const,
])("reconnects task-only %s warm recovery under the saved identity before reuse or retirement: %s (portable=%s)", async (provider, reuseProcess, portable) => {
const current=scopedNativeExecution(`plugin-restart-${provider}-${reuseProcess}-${portable}`);
if(provider === "claude") current.provider={kind:"acpx",agent:"claude",model:"fixture",permissionMode:"deny-all"} as never;
current.session.lifecyclePolicy={mode:"warm",idleTimeoutMs:300000};
const identity={runnerInstanceId:"old-runner",environmentLeaseId:"old-lease",runId:"old-run",
@@ -18184,22 +18317,29 @@ describe("runnerd provider runtime wiring", () => {
const home=await mkdtemp(join(tmpdir(),"plugin-restart-account-"));
await writeFile(join(home,"auth.json"),JSON.stringify({OPENAI_API_KEY:"synthetic-fixture"}),{mode:0o600});
const owner=vi.spyOn(pluginTaskRuntime,"createPluginRunnerTaskOwner");
const register = vi.spyOn(pluginCredentialWriteback, "registerPluginTaskCredentialHandoff").mockResolvedValue(undefined);
const binding = { grantId: randomUUID(), secretId: randomUUID(), sessionEpoch: 1, versionSelector: "latest" as const };
const task=vi.fn(async()=>({kind:"description",taskId:"saved-task",launchPreparationVersion:1,projectAuthCheckVersion:1,
warmTaskVersion:1,runner:{version:"fixture",digest:`sha256:${"a".repeat(64)}`}}));
state.createBackend.mockClear();state.createTransport.mockClear();
try {
await createRunnerdBackend({db:leaseDb(current),execution:current,runnerInstanceId:"new-runner",runnerIngressAuthorized:true,
runnerEnvironment:{ANTHROPIC_API_KEY:"synthetic-fixture"},managedAiCredentialHome:home,managedAiCredentialIdentity:reuseProcess ? "old-revision" : "new-revision",managedAiAccountIdentity:"account",
managedAiCredentialWriteback:{} as never,
managedAiCredentialWriteback: binding,
remoteWarmRecovery:{identity,reuseProcess,previousExecution:current,checkpoint:{} as never,
remote:{providerLeaseId:"saved-task",remoteCwd:"."},revalidate:async()=>{}},
remote:{providerLeaseId:"saved-task",remoteCwd:"."},revalidate:async()=>{},
...(portable ? { portableSourceTaskId: "old-task" } : {})},
runnerExecutionTarget:{kind:"remote",transport:"sandbox",leaseId:"new-lease",remoteCwd:".",runnerTask:task,
sandboxLeaseAcquisition:{outcome:"resumed",providerLeaseId:"saved-task"}}});
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
expect(state.createTransport.mock.calls.at(-1)![0].prpIdentity).toEqual(identity);
expect(owner.mock.calls.at(-1)![0].adoptIdentity).toEqual(identity);
expect(task.mock.calls).toEqual([[{kind:"describe"}]]);
} finally {owner.mockRestore();await rm(home,{recursive:true,force:true});}
if (portable) expect(register).toHaveBeenCalledExactlyOnceWith(expect.anything(), {
companyId: current.binding.companyId, leaseId: "new-lease",
}, "account", binding);
else expect(register).not.toHaveBeenCalled();
} finally {register.mockRestore(); owner.mockRestore();await rm(home,{recursive:true,force:true});}
});
it.each((["codex", "claude"] as const).flatMap(provider => [
@@ -54,7 +54,8 @@ import { writeExecutionControl } from "../execution-control-notifications.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import { createRunUsageRecorder } from "../usage-receipts.js";
import { syncPluginTaskCredentialHandoff, registerPluginTaskCredentialHandoff } from "../plugin-task-credential-writeback.js";
import { resolvePluginTaskResume } from "../environment-task-runtime.js";
import { resolvePluginTaskResume, type PluginTaskDescription } from "../environment-task-runtime.js";
import { assertAttachedPortableTask, readyPortableTaskSource, unlaunchedPortableTaskLease } from "../plugin-task-ownership.js";
import { environmentTaskOperationSchema, environmentTaskResultSchema } from "@paperclipai/plugin-sdk";
import { preparePluginRunnerCredentials } from "./plugin-runner-credentials.js";
import { createPluginRunnerTaskOwner, verifyPluginRunnerTaskReattachment } from "./plugin-runner-task.js";
@@ -278,7 +279,7 @@ import {
renderNativeRunnerStagedAttachmentPrompt,
stageNativeRunnerWakeAttachments,
} from "./native-runner-file-handoff.js";
import { NATIVE_PROVIDER_SIDE_EFFECT_PENDING, NATIVE_PROVIDER_STATE_INCOMPATIBLE, nativeRuntimeContractForProvider, nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
import { NATIVE_PROVIDER_SIDE_EFFECT_PENDING, NATIVE_PROVIDER_STATE_INCOMPATIBLE, nativeRuntimeContextsMatch, nativeRuntimeContractForProvider, nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
import { clearRunProviderStateUnsaved, markAcceptedRunProviderStateUnsaved } from "./native-provider-state-unsaved.js";
import { verifyRetainedMaintenanceNoLaunch } from "./native-maintenance-no-launch.js";
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
@@ -6247,6 +6248,29 @@ async function recordPriorOwnerFenceRefusal(input: { db: Db; execution: NativeEx
}).catch(() => undefined);
}
async function assertPluginTaskReattachmentBinding(input: {
db: Db;
execution: NativeExecutionInput;
restartRecovery?: NativeRestartRecoveryClaim;
}, target: Extract<AdapterExecutionTarget, { kind: "remote" }>, identity: Record<string, unknown> | null | undefined) {
const claim = input.restartRecovery;
const leaseId = target.leaseId;
if (claim?.kind !== "reattach_remote_runner" || claim.remote.kind !== "plugin_task" ||
claim.runId !== input.execution.binding.runId || !leaseId || claim.remote.leaseId !== leaseId ||
!identity || !durableIdentityMatchesExecution(identity, input.execution) ||
typeof identity.turnId !== "string" || !identity.turnId || typeof identity.itemId !== "string" || !identity.itemId) {
throw new Error("native_plugin_recovery_lease_mismatch");
}
const taskId = target.transport === "sandbox" ? target.sandboxLeaseAcquisition?.providerLeaseId : undefined;
if (identity.environmentLeaseId !== leaseId) {
if (!taskId) throw new Error("native_plugin_recovery_lease_mismatch");
await assertAttachedPortableTask(input.db, {
companyId: input.execution.binding.companyId, leaseId, taskId, runId: claim.runId, identity,
});
}
return taskId;
}
async function migrateRunnerdStateRootForExecution(input: {
db: Db;
execution: NativeExecutionInput;
@@ -6370,12 +6394,12 @@ async function migrateRunnerdStateRootForExecution(input: {
if (input.restartRecovery?.kind === "reattach_remote_runner") {
if (input.restartRecovery.remote.kind === "plugin_task") {
const target = input.runnerExecutionTarget;
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runnerTask
|| target.leaseId !== input.restartRecovery.remote.leaseId || identity.environmentLeaseId !== target.leaseId
|| !durableIdentityMatchesExecution(identity, input.execution)) {
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runnerTask) {
throw new Error("native_plugin_recovery_lease_mismatch");
}
await verifyPluginRunnerTaskReattachment(target.runnerTask, identity);
const taskId = await assertPluginTaskReattachmentBinding(input, target, identity);
await verifyPluginRunnerTaskReattachment(operation => target.runnerTask!(operation,
taskId ? { expectedTaskId: taskId } : undefined), identity);
return;
}
try {
@@ -7281,6 +7305,19 @@ function nativeSessionConfigDigest(
.digest("hex")}`;
}
function matchesRemoteWarmExecution(
previous: NativeExecutionInput,
current: NativeExecutionInput,
toolRefreshOnResume?: boolean,
): boolean {
if (nativeSessionScopeKey(previous) !== nativeSessionScopeKey(current)) return false;
const currentDigest = nativeSessionConfigDigest(current, "remote");
if (nativeSessionConfigDigest(previous, "remote") === currentDigest) return true;
return toolRefreshOnResume === true && "runtimeContext" in previous && "runtimeContext" in current &&
nativeRuntimeContextsMatch(previous.runtimeContext, current.runtimeContext, toolRefreshOnResume) &&
nativeSessionConfigDigest({ ...previous, runtimeContext: current.runtimeContext }, "remote") === currentDigest;
}
export function nativeControllerMigrationDescriptor(execution: NativeExecutionInput, recordedToolContractFingerprint: unknown) {
if (typeof recordedToolContractFingerprint !== "string" || !/^sha256:[a-f0-9]{64}$/.test(recordedToolContractFingerprint)) {
throw new Error("native_controller_migration_tool_contract_missing");
@@ -10203,6 +10240,7 @@ export async function executePaperclipNativeSession(input: {
getFreshSessionHandoff?: () => Promise<string | null>;
/** Retire a retained transport so recovery can replace provider tool declarations. */
refreshTools?: boolean;
toolRefreshOnResume?: boolean;
db: Db;
execution: NativeExecutionInput;
/** Private server preparation proof; never persisted or accepted from a tool. */
@@ -10259,6 +10297,7 @@ export async function executePaperclipNativeSession(input: {
managedAiCredentialWriteback?: ManagedAiCredentialWritebackBinding;
billingIdentity?: Pick<AdapterExecutionResult, "provider" | "biller" | "billingType">;
onTaskDispatched?: () => Promise<void> | void;
pluginTaskDescription?: PluginTaskDescription;
runnerExecutionTarget?: AdapterExecutionTarget | null;
/** Resolved per-run authorization; not an independent instance setting. */
runnerIngressAuthorized?: boolean;
@@ -10657,7 +10696,22 @@ async function executePaperclipNativeSessionAttempt(
const stagedRestart = input.restartRecovery?.kind === "reattach_remote_runner"
? await verifyStagedAttachmentRestart(input) : null;
const timeController = createRunPreparationTiming(input.execution.binding.runId, "native.controller");
if (input.useRunnerd) await timeController("warm_restore", () => restoreWarmRunnerdController(input));
const target = input.runnerExecutionTarget;
const portableTarget = target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask ? target : undefined;
const companyId = input.execution.binding.companyId;
const portablePreparationLeaseId = input.restartRecovery?.kind === "bootstrap_incomplete"
? input.restartRecovery.portablePreparationLeaseId : undefined;
const targetLease = portableTarget?.leaseId ? (await input.db.select({
metadata: environmentLeases.metadata, providerLeaseId: environmentLeases.providerLeaseId,
...(portablePreparationLeaseId ? { unlaunchedPortable: unlaunchedPortableTaskLease() } : {}),
}).from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId), eq(environmentLeases.id, portableTarget.leaseId))))[0] : undefined;
if (portablePreparationLeaseId && (portableTarget?.leaseId !== portablePreparationLeaseId || targetLease?.unlaunchedPortable !== true)) {
throw remoteRecoveryLeaseMismatch("runner_verification", "plugin_task");
}
const replayPortablePreparation = portablePreparationLeaseId !== undefined;
const canRecoverWarmOwner = !input.restartRecovery || replayPortablePreparation;
if (input.useRunnerd) await timeController("warm_restore", () => restoreWarmRunnerdController(
replayPortablePreparation ? { ...input, restartRecovery: undefined } : input));
const identityRedactor = createAgentIdentityRedactor(input.runnerEnvironment?.PAPERCLIP_AGENT_PRIVATE_KEY);
const redactIdentityText = (text: string) => redactSensitiveText(
identityRedactor.chunk("diagnostic", text) + identityRedactor.finish("diagnostic"),
@@ -10757,29 +10811,51 @@ async function executePaperclipNativeSessionAttempt(
});
let retainedTransition: VerifiedWarmTransitionBinding | undefined;
let remoteWarmRecovery: RemoteWarmSessionRecovery | undefined;
let pluginTaskDescription: Extract<ReturnType<typeof environmentTaskResultSchema.parse>,{kind:"description"}> | undefined;
let pluginTaskDescription: PluginTaskDescription | undefined;
let portableSource: Awaited<ReturnType<typeof readyPortableTaskSource>>;
if (portableTarget?.runnerTask && portableTarget.leaseId) {
if (input.pluginTaskDescription && input.pluginTaskDescription.taskId === targetLease?.providerLeaseId) {
pluginTaskDescription = input.pluginTaskDescription;
}
if (targetLease?.metadata?.portableTaskPreviousLeaseId || targetLease?.metadata?.portableTaskPredecessor) {
const description = targetLease.metadata.portableTaskOwnershipReady === true && pluginTaskDescription
? pluginTaskDescription : environmentTaskResultSchema.parse(await portableTarget.runnerTask({ kind: "describe" }));
if (description.kind !== "description") throw new Error("Plugin Runner description unavailable");
pluginTaskDescription = description;
portableSource = await readyPortableTaskSource(input.db, companyId, portableTarget.leaseId,
readRunnerdDurableIdentity(scopedRunnerdStateRoot(input.execution)), {
attachedRunId: input.execution.binding.runId,
selectedSessionId: nativeSessionKey(input.execution),
accountIdentity: input.managedAiAccountIdentity ?? input.managedAiCredentialIdentity,
});
}
}
const cachedReleasedOwner = warmNativeSessions.get(nativeSessionScopeKey(input.execution));
if (cachedReleasedOwner?.released) {
const proof = releasedWarmRunner(readWarmNativeEnvelope(input.execution));
const proof = portableSource ? undefined : releasedWarmRunner(readWarmNativeEnvelope(input.execution));
const identity = readRunnerdDurableIdentity(scopedRunnerdStateRoot(input.execution));
const target = input.runnerExecutionTarget;
if (proof && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId
&& target.sandboxLeaseAcquisition?.outcome === "created" && proof.taskId !== target.sandboxLeaseAcquisition.providerLeaseId) {
const source = portableSource ?? (proof && { taskId: proof.taskId, identity: proof.source });
if (source && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId
&& (portableSource || target.sandboxLeaseAcquisition?.outcome === "created")
&& source.taskId !== target.sandboxLeaseAcquisition?.providerLeaseId) {
const ownInstructionPreparation = cachedReleasedOwner.instructionPreparationRunId === input.execution.binding.runId
&& input.instructionWorkingCopy?.runId === input.execution.binding.runId;
if ((cachedReleasedOwner.busy && !ownInstructionPreparation)
|| cachedReleasedOwner.preparingRunId && cachedReleasedOwner.preparingRunId !== input.execution.binding.runId
|| cachedReleasedOwner.instructionPreparationRunId && !ownInstructionPreparation
|| cachedReleasedOwner.closeOnReleaseReason) throw new Error("native_session_supervisor_busy");
if (proof.taskId !== cachedReleasedOwner.pluginTaskId || proof.scopeDigest !== releasedWarmScope(input)
|| canonicalJson(proof.source) !== canonicalJson(identity)) throw new Error("native_session_released_owner_scope_mismatch");
if (source.taskId !== cachedReleasedOwner.pluginTaskId || cachedReleasedOwner.runId !== identity?.runId ||
(proof && (proof.scopeDigest !== releasedWarmScope(input) || canonicalJson(proof.source) !== canonicalJson(identity)))) {
throw new Error("native_session_released_owner_scope_mismatch");
}
const wasBusy = cachedReleasedOwner.busy;
cachedReleasedOwner.busy = true;
try {
// Only physical stop releases the cached controller; ordinary checkpoint admission still proves continuation.
const resumeFrom = await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
leaseId: target.leaseId, priorIdentity: proof.source, inspectCheckpoint: true });
if (resumeFrom !== proof.taskId) throw new Error("native_session_released_owner_scope_mismatch");
// Ordinary replacement still needs a stop receipt for the old task.
const resumeFrom = portableSource?.taskId ?? await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
leaseId: target.leaseId, priorIdentity: source.identity, inspectCheckpoint: true });
if (resumeFrom !== source.taskId) throw new Error("native_session_released_owner_scope_mismatch");
try {
await detachReleasedWarmNativeSession(cachedReleasedOwner);
} catch (error) {
@@ -10790,7 +10866,7 @@ async function executePaperclipNativeSessionAttempt(
} finally { cachedReleasedOwner.busy = wasBusy; }
} else requireReleasedWarmScope(input, proof, identity);
}
if (input.useRunnerd && !input.restartRecovery && !input.backend &&
if (input.useRunnerd && canRecoverWarmOwner && !input.backend &&
input.runnerExecutionTarget?.kind === "remote" &&
input.execution.session.lifecyclePolicy.mode === "warm" &&
!idleWarmNativeSessionOwnerRunId(input)) {
@@ -10800,10 +10876,12 @@ async function executePaperclipNativeSessionAttempt(
if (taskTarget.transport === "sandbox" && taskTarget.runnerTask && taskTarget.leaseId &&
durableIdentityMatchesSession(identity,input.execution) && identity.runId !== input.execution.binding.runId) {
// Inspection can replace a stopped retained task before deciding between adoption and checkpoint recovery.
const description = environmentTaskResultSchema.parse(await taskTarget.runnerTask({kind:"describe"}));
if (description.kind !== "description") throw new Error("Plugin Runner description unavailable");
pluginTaskDescription = description;
if (taskTarget.sandboxLeaseAcquisition?.outcome !== "resumed") {
if (!pluginTaskDescription || pluginTaskDescription.taskId !== taskTarget.sandboxLeaseAcquisition?.providerLeaseId) {
const description = environmentTaskResultSchema.parse(await taskTarget.runnerTask({ kind: "describe" }));
if (description.kind !== "description") throw new Error("Plugin Runner description unavailable");
pluginTaskDescription = description;
}
if (taskTarget.sandboxLeaseAcquisition?.outcome !== "resumed" && !portableSource) {
await recoverPluginRunnerCheckpoint(input, root);
}
}
@@ -10817,22 +10895,23 @@ async function executePaperclipNativeSessionAttempt(
}
if (durableIdentityMatchesSession(identity, input.execution) && identity.runId !== input.execution.binding.runId &&
runnerdAuthorityLifecycleWithVerifiedBackup({ root, identity, execution: input.execution, allowVerifiedBackup: true }) !== "suspended") {
let releasedSource: DurableRecoveryIdentity | undefined;
let releasedSource = portableSource?.identity;
remoteWarmRecovery = await inspectRemoteWarmSessionRecovery({
db: input.db, execution: input.execution, target: input.runnerExecutionTarget, identity,
portableSourceTaskId: portableSource?.taskId,
readControlState: () => record(JSON.parse(readBoundedNativeFile(
resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
"runner_durable_identity_too_large").toString("utf8"))),
matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(input.execution, "remote"),
matchesExecution: previous => matchesRemoteWarmExecution(previous, input.execution, input.toolRefreshOnResume),
loadCheckpoint: previous => {
const proof = releasedWarmRunner(readWarmNativeEnvelope(previous));
if (proof) {
releasedSource = requireReleasedWarmScope(input, proof, identity);
if (!portableSource) {
const proof = releasedWarmRunner(readWarmNativeEnvelope(previous));
if (proof) releasedSource = requireReleasedWarmScope(input, proof, identity);
}
return loadWarmNativeCheckpoint(previous, nativeSessionConfigDigest(previous, "remote"), "remote");
},
canReuseProcess: previous => !needsCoreGitHubAccess(input) && canReuseNativeProcess(input,
canReuseProcess: previous => !portableSource && !needsCoreGitHubAccess(input) &&
nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(input.execution, "remote") && canReuseNativeProcess(input,
record(JSON.parse(readBoundedNativeFile(nativeSessionCheckpointPath(previous),
NATIVE_WARM_CHECKPOINT_MAX_BYTES, "native_session_supervisor_checkpoint_too_large").toString("utf8"))).processConfigDigest),
}) ?? undefined;
@@ -11061,7 +11140,7 @@ async function executePaperclipNativeSessionAttempt(
: coordinator.controllerGeneration + 1;
const target = input.runnerExecutionTarget;
const retained = warmNativeSessions.get(nativeSessionScopeKey(input.execution));
if (!input.restartRecovery && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId &&
if (canRecoverWarmOwner && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId &&
((retained?.pluginTaskId && idleWarmNativeSessionOwnerRunId(input) !== undefined) || remoteWarmRecovery?.releasedSource) &&
(input.execution.provider.kind === "codex" || (input.execution.provider.kind === "acpx" && input.execution.provider.agent === "claude"))) {
claimedStartupContext = new NativeExecutionContext(input.execution, Math.min(executionLeaseTtlMs,
@@ -11801,7 +11880,7 @@ async function executePaperclipNativeSessionAttempt(
executionContext = undefined;
startupForAdmission!.executionContext = undefined;
};
if (warmSessionId !== null && warmConfigDigest !== null && !input.restartRecovery) {
if (warmSessionId !== null && warmConfigDigest !== null && canRecoverWarmOwner) {
const registered = warmNativeSessions.get(warmSessionId);
const entry = registered && forgetUnretiredOwnerOnOtherSandbox(warmSessionId, registered, input.runnerExecutionTarget)
? undefined : registered;
@@ -12088,11 +12167,15 @@ async function executePaperclipNativeSessionAttempt(
signal: executionContext?.signal ?? new AbortController().signal,
});
if (!recovered?.recovered || !recovered.session) throw new Error("runner_state_identity_mismatch: remote_warm_session_unproven");
if (remoteWarmRecovery.reuseProcess) {
const movedSession = remoteWarmRecovery.portableSourceTaskId !== undefined;
// A portable resume already starts a new Runner. Its run attachment
// prepares the current tools and runtime on the verified target.
if (movedSession || remoteWarmRecovery.reuseProcess) {
existingWarmSession = recovered.session;
await emitStartupReuse(input.onEvent, "runner", { state: "reused", reason: "verified_reuse" });
await emitStartupReuse(input.onEvent, "runner", movedSession
? { state: "fresh", reason: "replacement" } : { state: "reused", reason: "verified_reuse" });
await emitStartupReuse(input.onEvent, "provider_session", { state: "reused", reason: "verified_reuse" });
trace.setBudgetStart("warm");
if (!movedSession) trace.setBudgetStart("warm");
persistedWarmSession = null;
} else {
await recovered.session.close({ reason: "recovered warm session requires current process configuration" });
@@ -12104,12 +12187,22 @@ async function executePaperclipNativeSessionAttempt(
remoteWarmRecovery.previousExecution.binding, scopedRunnerdStateRoot(input.execution),
nativeSessionKey(input.execution));
if (!suspendedController) throw new Error("native_runner_checkpoint_missing_after_suspension");
persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution,
nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote"), await recovered.session.snapshot());
persistedWarmSession = loadWarmNativeCheckpoint(input.execution, warmConfigDigest!, "remote");
runnerdBackend = await createRunnerdBackend({ ...input, onDiagnosticLease: ownDiagnosticLease, providerPackAdmissionSource, recoveryPurpose: claimedStopCleanup ? "stop_cleanup" : "execution", onAccountingEvent: producerUsage?.capture, runnerInstanceId: effectiveRunnerInstanceId,
const verifiedConfigDigest = nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote");
persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution, verifiedConfigDigest, await recovered.session.snapshot());
persistedWarmSession = loadWarmNativeCheckpoint(input.execution, verifiedConfigDigest, "remote");
runnerdBackend = await createRunnerdBackend({
...input,
pluginTaskDescription: undefined,
onDiagnosticLease: ownDiagnosticLease,
providerPackAdmissionSource,
recoveryPurpose: claimedStopCleanup ? "stop_cleanup" : "execution",
onAccountingEvent: producerUsage?.capture,
runnerInstanceId: effectiveRunnerInstanceId,
controllerPersistence: input.runnerExecutionTarget?.kind === "remote" ? { bootId: controller.bootId } : undefined,
durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId, trace, toolTrace });
durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId,
trace,
toolTrace,
});
}
}
const readRemoteCleanupLease = async () => input.runnerExecutionTarget?.kind === "remote" &&
@@ -15710,6 +15803,7 @@ export function remoteCheckpointIncompleteFailure(
export async function createRunnerdBackend(input: {
executionContext?: NativeExecutionContext;
casesEnabled?: boolean;
toolRefreshOnResume?: boolean;
db: Db;
providerPackAdmissionSource?: NativeProviderPackAdmissionSource;
controllerPersistence?: { bootId: string; runnerArtifact?: { version: string; digest: string } };
@@ -15724,7 +15818,7 @@ export async function createRunnerdBackend(input: {
remoteWarmRecovery?: Omit<RemoteWarmSessionRecovery, "revalidate"> & { revalidate: () => Promise<void> };
retainedPluginTaskOwner?: ReturnType<typeof createPluginRunnerTaskOwner>;
beginRetainedRunAttachment?: () => void;
pluginTaskDescription?: Extract<ReturnType<typeof environmentTaskResultSchema.parse>,{kind:"description"}>;
pluginTaskDescription?: PluginTaskDescription;
onSpawn?: (meta: {
pid: number;
processGroupId: number | null;
@@ -16128,8 +16222,10 @@ async function createRunnerdBackendWithinSessionClaim(
const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity
? createNativeRecoveryUsageDelivery({
db: input.db, execution: input.execution, sourceIdentity: { ...recoveryUsageIdentity },
matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind),
matchesExecution: previous => input.remoteWarmRecovery
? matchesRemoteWarmExecution(previous, input.execution, input.toolRefreshOnResume)
: nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind),
})
: undefined;
const accountingTransportOptions = {
@@ -16185,15 +16281,19 @@ async function createRunnerdBackendWithinSessionClaim(
const { root, durableIdentity, durableBinding } = durableState ??= readDurableState();
const provider = input.execution.provider;
if (provider.kind !== "codex" && provider.kind !== "opencode" && provider.kind !== "acpx") throw new Error("Plugin Runner tasks require a workspace harness");
if (input.restartRecovery?.kind === "reattach_remote_runner") {
await assertPluginTaskReattachmentBinding(input, remoteTarget, durableBinding ? durableIdentity : null);
}
let priorDescription = input.pluginTaskDescription?.taskId === remoteTarget.sandboxLeaseAcquisition?.providerLeaseId
? input.pluginTaskDescription : undefined;
if (remoteTarget.sandboxLeaseAcquisition?.outcome === "resumed" && !durableBinding && !input.retainedPluginTaskOwner && !input.remoteWarmRecovery) {
// A new logical session (including fresh retries) cannot inherit the old process authority.
await pluginTask({kind:"stop"});
const stopped=environmentTaskResultSchema.parse(await pluginTask({kind:"status"}));
if(stopped.kind!=="status" || stopped.executionStopped!==true) throw new Error("Plugin Runner termination is unconfirmed; reconcile the saved task");
await pluginTask({kind:"complete"});
priorDescription = undefined;
}
const priorDescription = input.pluginTaskDescription?.taskId === remoteTarget.sandboxLeaseAcquisition?.providerLeaseId
? input.pluginTaskDescription : undefined;
const description = priorDescription ?? environmentTaskResultSchema.parse(await pluginTask({ kind: "describe" }));
if (description.kind !== "description" || description.launchPreparationVersion !== 1 || description.projectAuthCheckVersion !== 1) throw new Error("Plugin Runner launch preparation v1 unavailable");
const saved = durableBinding ? durableIdentity as RunnerdDurableIdentity : undefined;
@@ -16212,9 +16312,7 @@ async function createRunnerdBackendWithinSessionClaim(
normalizedSessionId: input.remoteWarmRecovery.identity.normalizedSessionId,
turnId: input.remoteWarmRecovery.identity.turnId, itemId: input.remoteWarmRecovery.identity.itemId,
} : undefined;
if (input.restartRecovery?.kind === "reattach_remote_runner" && (!prior || !prior.turnId || !prior.itemId || prior.environmentLeaseId !== remoteTarget.leaseId)) {
throw new Error("native_plugin_recovery_lease_mismatch");
}
const resumeFrom = !retainedOwner && !adopting && prior && prior.environmentLeaseId !== remoteTarget.leaseId
? await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
leaseId: remoteTarget.leaseId!, priorIdentity: prior }) : undefined;
@@ -16224,7 +16322,8 @@ async function createRunnerdBackendWithinSessionClaim(
home: input.managedAiCredentialHome, environment: input.runnerEnvironment ?? {},
});
if (credentials.needsWriteback && !input.managedAiCredentialWriteback) throw new Error("Plugin Runner credential writeback authority unavailable");
if (!adopting && input.managedAiCredentialWriteback && credentials.accountIdentity) {
// Same-lease recovery already has a writeback binding. A portable move creates a new lease.
if ((!adopting || input.remoteWarmRecovery?.portableSourceTaskId) && input.managedAiCredentialWriteback && credentials.accountIdentity) {
if (!remoteTarget.leaseId) throw new Error("Plugin Runner credential lease unavailable");
await registerPluginTaskCredentialHandoff(input.db, { companyId: input.execution.binding.companyId, leaseId: remoteTarget.leaseId },
credentials.accountIdentity, input.managedAiCredentialWriteback);
@@ -16251,7 +16350,8 @@ async function createRunnerdBackendWithinSessionClaim(
releaseResumeVersion: description.releaseResumeVersion,
authorityRunId: description.releaseResumeVersion === 1 ? input.execution.binding.runId : undefined,
releasedSource: input.remoteWarmRecovery?.releasedSource,
onResumed: async source => updateReleasedWarmRunner(input.execution, source, description.taskId, null),
sessionHandoff: !!input.remoteWarmRecovery?.portableSourceTaskId,
onResumed: async source => updateReleasedWarmRunner(input.execution, source, input.remoteWarmRecovery?.portableSourceTaskId ?? description.taskId, null),
onRetired: async source => updateReleasedWarmRunner(input.execution, source, description.taskId, null),
adoptIdentity: adoptionIdentity ?? (adopting && prior ? {
runnerInstanceId: prior.runnerInstanceId, environmentLeaseId: prior.environmentLeaseId,
@@ -8,7 +8,7 @@ import type {
NativeExecutionInput,
PersistedNativeSession,
} from "../../vendor/paperclip-runner/index.js";
import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput, type NativeRuntimeContextSnapshot } from "../../vendor/paperclip-runner/index.js";
export type NativeToolExecutionTargetKind = "local" | "remote";
@@ -612,6 +612,15 @@ export function buildNativeExecutionWithCheckpoint(input: {
};
}
export function nativeRuntimeContextsMatch(
previous: NativeRuntimeContextSnapshot,
current: NativeRuntimeContextSnapshot,
toolRefreshOnResume?: boolean,
): boolean {
return previous.aggregateDigest === current.aggregateDigest ||
(toolRefreshOnResume === true && canonicalNativeRuntimeContextDigest({ ...previous, mcp: current.mcp }) === current.aggregateDigest);
}
/**
* Rebind a completed prior run's provider checkpoint to a new heartbeat run.
* The provider/driver session identity is retained, while every per-turn and
@@ -697,12 +706,7 @@ export function rebindNativeSessionCheckpoint(input: {
("homeMode" in previousExecution ? previousExecution.homeMode : "isolated") !== ("homeMode" in current ? current.homeMode : "isolated") ||
("runtimeContext" in previousExecution &&
"runtimeContext" in current &&
previousExecution.runtimeContext.aggregateDigest !== current.runtimeContext.aggregateDigest &&
!(input.toolRefreshOnResume === true &&
canonicalNativeRuntimeContextDigest({
...previousExecution.runtimeContext,
mcp: current.runtimeContext.mcp,
}) === current.runtimeContext.aggregateDigest))
!nativeRuntimeContextsMatch(previousExecution.runtimeContext, current.runtimeContext, input.toolRefreshOnResume))
)
return null;
@@ -1,6 +1,7 @@
import { and, asc, eq, exists, inArray, sql } from "drizzle-orm";
import { and, asc, eq, exists, inArray, or, sql } from "drizzle-orm";
import { environmentLeases, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db";
import { inspectDurablePrpCheckpointHeader, type DurablePrpCheckpoint } from "../../vendor/paperclip-runner/index.js";
import { nativeSha256 } from "./canonical.js";
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
const record = (value: unknown): Record<string, unknown> => value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
@@ -18,7 +19,8 @@ export async function readNativeSourceConsumers(tx: Transaction, companyId: stri
sql`${nativeRunnerCheckpoints.material}->>'schema' like 'paperclip.native-runner-checkpoint-retirement.%'`)));
return tx.select({ id: environmentLeases.id, metadata: environmentLeases.metadata, retired }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId),
sql`${environmentLeases.metadata}->'runnerTaskSourceConsumer' is not null`));
or(sql`${environmentLeases.metadata}->'runnerTaskSourceConsumer' is not null`,
sql`${environmentLeases.metadata}->'portableTaskSuccessor' is not null`)));
}
/** Physical source consumption revokes the completed controller's late writer. */
@@ -30,13 +32,26 @@ export async function assertNativeSourceCheckpointConsumer(tx: Transaction, owne
if (!rows.length) return;
const deny = (): never => { throw new Error("native_runner_checkpoint_owner_lost"); };
if (owner.runId === owner.authorityRunId || rows.length !== 1 || !checkpoint) return deny();
const source = rows[0]!, consumer = record(source.metadata?.runnerTaskSourceConsumer);
const source = rows[0]!;
const header = checkpoint.records.get("header");
const identity = header ? inspectDurablePrpCheckpointHeader(header) : null;
const portable = record(source.metadata?.portableTaskSuccessor);
if (source.metadata?.portableTaskSuccessor !== undefined) {
if (typeof portable.targetLeaseId !== "string" || portable.sourceLeaseId !== source.id ||
!identity || nativeSha256(record(portable.operation).identity ?? null) !== nativeSha256(identity)) return deny();
const [target] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, owner.companyId),
eq(environmentLeases.id, portable.targetLeaseId)));
if (!target || target.heartbeatRunId !== owner.authorityRunId || target.status !== "active" || target.releasedAt ||
target.expiresAt && target.expiresAt.getTime() <= Date.now() ||
target.metadata?.portableTaskOwnershipReady !== true || target.metadata.portableTaskSuccessor !== undefined ||
nativeSha256(target.metadata.portableTaskPredecessor ?? null) !== nativeSha256(portable)) return deny();
return;
}
const consumer = record(source.metadata?.runnerTaskSourceConsumer);
if (typeof consumer.leaseId !== "string" || consumer.runId !== owner.authorityRunId) return deny();
const [lease] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, owner.companyId),
eq(environmentLeases.id, consumer.leaseId)));
const recovery = record(lease?.metadata?.runnerTaskSourceRecovery);
const header = checkpoint.records.get("header");
const identity = header ? inspectDurablePrpCheckpointHeader(header) : null;
const expected = record(recovery.identity);
if (!lease || lease.heartbeatRunId !== owner.authorityRunId || lease.status !== "active" || lease.releasedAt ||
lease.expiresAt && lease.expiresAt.getTime() <= Date.now() ||
@@ -553,6 +553,18 @@ describe("plugin Runner release/resume facade", () => {
? { kind: "released", taskId: "task" }
: { kind: "resume", taskId: "task", outcome: "resumed" });
it.each([true, false])("sends the current Claude credential only for session handoff (%s)", async sessionHandoff => {
const task = lifecycleTask();
const environment = { CLAUDE_CODE_OAUTH_TOKEN: "fixture-current-token" };
const owner = createPluginRunnerTaskOwner({ ...input("claude"), task, sessionHandoff,
credential: { type: "claude-env", environment } });
await expect(owner.resume(requirements.identity, "transient-ticket")).resolves.toBe("resumed");
expect(task).toHaveBeenCalledExactlyOnceWith({
kind: "resume", identity: requirements.identity, bootstrapTicket: "transient-ticket",
...(sessionHandoff ? { handoffCredential: { accountIdentity: "selected-account", environment } } : {}),
}, { expectedTaskId: "task" });
});
it.each(["codex", "claude"])("quiesces %s before release and closes only after its ACK", async harness => {
const { task } = provider();
const perform = task.getMockImplementation()!;
@@ -50,6 +50,7 @@ export function createPluginRunnerTaskOwner(input: {
warmTaskVersion?: 1;
releaseResumeVersion?: 1;
releasedSource?: DurableRecoveryIdentity;
sessionHandoff?: boolean;
onResumed?: (identity: DurableRecoveryIdentity) => Promise<void>;
onRetired?: (identity: DurableRecoveryIdentity) => Promise<void>;
githubBroker?: { url: string; token: string };
@@ -418,7 +419,12 @@ export function createPluginRunnerTaskOwner(input: {
};
const resume = async (identity: DurableRecoveryIdentity, bootstrapTicket: string): Promise<"resumed" | "unavailable"> => {
requireReleaseResume();
const result = await call(environmentTaskOperationSchema.parse({ kind: "resume", identity, bootstrapTicket }));
const handoffCredential = input.sessionHandoff && input.credential?.type === "claude-env"
? { accountIdentity: input.accountIdentity, environment: input.credential.environment }
: undefined;
const result = await call(environmentTaskOperationSchema.parse({ kind: "resume", identity, bootstrapTicket,
...(handoffCredential ? { handoffCredential } : {}),
}));
requireReleaseResume();
if (result.kind !== "resume") throw new Error("Plugin Runner resume result unavailable");
if (result.outcome === "unavailable") {
@@ -3,19 +3,23 @@ import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { eq, sql } from "drizzle-orm";
import { environmentTaskOperationSchema } from "@paperclipai/plugin-sdk";
import { environmentTaskOperationSchema, type PluginEnvironmentTaskResult } from "@paperclipai/plugin-sdk";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, companies, completionContracts, createDb, environmentLeases, environments, heartbeatRuns, issues, nativeRunFinalizations,
nativeRunResults, nativeRunnerCheckpoints, plugins } from "@paperclipai/db";
import { parseNativeExecutionInput, runnerProviderLaunchRequirements, DurablePrpControlPlane, type DurablePrpCheckpoint, type NativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput, runnerProviderLaunchRequirements, DurablePrpControlPlane, type DurablePrpCheckpoint, type NativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
import { recoveryPostgresSupport, startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js";
import { createIsolatedTestHome } from "../../__tests__/helpers/isolated-test-home.js";
import { executeEnvironmentTask, resolvePluginTaskResume } from "../environment-task-runtime.js";
import type { PluginWorkerManager } from "../plugin-worker-manager.js";
import { assertAttachedPortableTask, readyPortableTaskSource, reconcilePluginTaskOwnership } from "../plugin-task-ownership.js";
import { remoteTerminationReceipt } from "../remote-execution-termination.js";
import * as runtimeApi from "../../runtime-api.js";
import { NativeExecutionContext } from "./native-execution-context.js";
import { buildNativeExecutionInput } from "./native-execution-input.js";
import { currentNativeControllerIdentity } from "./native-restart-recovery.js";
import { createRunnerdBackend, nativeControllerMigrationDescriptor } from "./native-session-executor.js";
import { buildNativeExecutionWithCheckpoint, nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
import { loadNativeRunnerCheckpoint, nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js";
import { claimPluginTaskSourceRecovery } from "./plugin-task-source-recovery.js";
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
@@ -83,7 +87,7 @@ describe.skipIf(!recoveryPostgresSupport.supported)("stopped source admission un
leaseOwner: owner.leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000) });
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "source-fixture", version: "1.0.0", status: "ready",
manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }] } as never });
manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true, sessionHandoffVersion: 1 }] } as never });
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "plugin", status: "active",
config: { pluginKey: pluginId, driverKey: "tasks" } });
const metadata = { driver: "plugin", pluginId, driverKey: "tasks", runnerTaskLease: true };
@@ -98,7 +102,7 @@ describe.skipIf(!recoveryPostgresSupport.supported)("stopped source admission un
const input = { companyId, leaseId, context, identity, resumeFrom: "source-task",
...(provider === "codex" ? { codexSourceRecoveryVersion: 1 as const } : { acpxSourceRecoveryVersion: 1 as const }),
accountIdentity: "current-selected-account", matchesExecution: (previous: typeof sourceExecution) => previous.provider.kind === execution.provider.kind && previous.workspace.cwd === execution.workspace.cwd };
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call: vi.fn(async (_id: string, _method: string, p: any) => {
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call: vi.fn(async (_id: string, _method: string, p: any): Promise<PluginEnvironmentTaskResult> => {
if (p.operation.kind === "prepare") return { kind: "prepared", taskId: p.taskId,
launch: { configurationId: "source-configuration", requirementsDigest: `sha256:${"a".repeat(64)}`, provider: {} } };
if (p.operation.kind === "check_project_auth") return { kind: "project_auth", taskId: p.taskId, provider: p.operation.provider, configurationId: p.operation.configurationId, allowed: true };
@@ -125,6 +129,213 @@ describe.skipIf(!recoveryPostgresSupport.supported)("stopped source admission un
harness: provider === "claude" ? "claude" as const : "codex" as const, authProvider: provider === "claude" ? "anthropic" : "openai" };
}
it.each([
{ policy: "ordinary", scenario: "ordinary handoff" },
{ policy: "reclaim_if_needed", scenario: "idle source reclaim" },
{ policy: "ordinary", scenario: "fresh conversation after instructions change", freshSession: true },
] as const)("reconciles ownership after a lost response: $scenario", async scenario => {
const { policy } = scenario;
const f = await fixture();
await db.update(heartbeatRuns).set({ scopeKind: "issue", issueId: f.issueId }).where(eq(heartbeatRuns.id, f.runId));
await db.update(environmentLeases).set({ status: "retained", releasedAt: null,
metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination') || ${JSON.stringify({
providerMetadata: { ownershipGeneration: "source-generation" }, runnerTaskAccountIdentity: f.accountIdentity,
})}::jsonb`,
}).where(eq(environmentLeases.id, f.sourceLeaseId));
await db.update(environmentLeases).set({
metadata: sql`${environmentLeases.metadata} || '{"providerMetadata":{"ownershipGeneration":"target-generation"}}'::jsonb`,
}).where(eq(environmentLeases.id, f.leaseId));
const request = { companyId: f.companyId, sourceLeaseId: f.sourceLeaseId, targetLeaseId: f.leaseId, policy };
const send = (database: typeof db) => reconcilePluginTaskOwnership(database, request, operation =>
executeEnvironmentTask(database, f.workers as unknown as PluginWorkerManager, { companyId: f.companyId, leaseId: f.leaseId, operation }));
const describe = (database: typeof db, ownershipSignal?: AbortSignal) => executeEnvironmentTask(database, f.workers as unknown as PluginWorkerManager,
{ companyId: f.companyId, leaseId: f.leaseId, ownershipSignal, operation: { kind: "describe" } });
if (policy === "ordinary") await db.update(environmentLeases).set({
metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ portableTaskPreviousLeaseId: f.sourceLeaseId })}::jsonb`,
}).where(eq(environmentLeases.id, f.leaseId));
f.workers.call.mockRejectedValueOnce(new Error("worker transport lost"));
await expect(policy === "ordinary" ? describe(db) : send(db)).rejects.toThrow();
const operation = f.workers.call.mock.calls[0][2].operation;
expect(operation).toMatchObject({ kind: "ensure_ownership", operationId: f.leaseId, identity: f.identity,
sourceTaskId: "source-task", sourceGeneration: "source-generation", targetGeneration: "target-generation", policy });
await expect(f.task({ ...operation, policy: policy === "ordinary" ? "reclaim_if_needed" : "ordinary" })).rejects.toThrow("ownership_unverified");
await expect(f.task()).rejects.toThrow("session ownership unavailable");
await expect(executeEnvironmentTask(db, f.workers as unknown as PluginWorkerManager, {
companyId: f.companyId, leaseId: f.sourceLeaseId, operation: { kind: "resume", identity: f.identity, bootstrapTicket: "fixture-ticket" },
})).rejects.toThrow("session ownership unavailable");
expect(f.workers.call).toHaveBeenCalledTimes(1);
const reply = { kind: "ownership", taskId: "replacement-task", operationId: f.leaseId,
targetGeneration: "target-generation" } as const;
f.workers.call.mockResolvedValueOnce({ ...reply, status: "pending" });
await expect(describe(db)).rejects.toThrow("native_portable_task_ownership_pending");
const targetWriter = nativeRunnerCheckpointWriter(db, { companyId: f.companyId, issueId: f.issueId,
runId: f.sourceRunId, authorityRunId: f.runId, ...f.owner });
await expect(targetWriter(f.checkpoint)).rejects.toThrow("native_runner_checkpoint_owner_lost");
const restarted = createDb(database.connectionString);
f.workers.call.mockResolvedValueOnce({ ...reply, status: "ready", targetGeneration: "another-generation" });
await expect(describe(restarted)).rejects.toThrow("reconcile the same task");
await expect(f.task()).rejects.toThrow("session ownership unavailable");
const description = { kind: "description", taskId: "replacement-task",
runner: { version: "test", digest: `sha256:${"a".repeat(64)}` } } as const;
f.workers.call.mockRejectedValueOnce(new Error("worker transport lost again"));
f.workers.call.mockResolvedValueOnce({ ...reply, status: "pending" });
f.workers.call.mockResolvedValueOnce({ ...reply, status: "ready" });
f.workers.call.mockResolvedValueOnce(description);
await expect(describe(restarted, new AbortController().signal)).resolves.toEqual(description);
expect(f.workers.call.mock.calls.map(([, , call]) => call.operation)).toEqual([...Array(6).fill(operation), { kind: "describe" }]);
const [target] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
expect(target.metadata?.portableTaskOwnershipReady).toBe(true);
expect(target.metadata?.runnerTaskAccountIdentity).toBe(f.accountIdentity);
const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.sourceLeaseId));
expect(source.metadata?.remoteExecutionTermination).toBeUndefined();
await expect(f.oldWriter(f.checkpoint)).rejects.toThrow("native_runner_checkpoint_owner_lost");
await expect(targetWriter(f.checkpoint)).resolves.toBeUndefined();
async function recoverAttachedRunner(execution: NativeExecutionInput, identity: typeof f.identity) {
const { root: stateDirectory } = nativeControllerMigrationDescriptor(execution,
nativeToolContractFingerprintForTarget("remote", execution.provider.kind));
const durable = new DurablePrpControlPlane({ stateDirectory: join(stateDirectory, "control-plane"),
identity: identity, expectedRunnerVersion: description.runner.version, expectedRunnerDigest: description.runner.digest });
durable.persistRunAttachTemplate({ provider: { instructions: "Run using the admitted instructions.",
runtimeContext: "runtimeContext" in execution ? execution.runtimeContext : null } });
await durable.flushCheckpoint();
await durable.stop();
const recover = () => createRunnerdBackend({ db: restarted, execution,
runnerInstanceId: identity.runnerInstanceId, runnerIngressAuthorized: true,
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", leaseId: f.leaseId,
sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "replacement-task" },
runnerTask: (operation, options) => executeEnvironmentTask(restarted, f.workers as unknown as PluginWorkerManager, {
companyId: f.companyId, leaseId: f.leaseId, operation: environmentTaskOperationSchema.parse(operation),
expectedTaskId: options?.expectedTaskId,
}),
},
restartRecovery: { kind: "reattach_remote_runner", remote: { kind: "plugin_task", leaseId: f.leaseId },
runId: f.runId, leaseOwner: "successor", controllerGeneration: f.owner.generation + 1,
providerAttempt: 1, restartKind: "hard", recoveryRequestId: null },
});
f.workers.call.mockClear();
f.workers.call
.mockResolvedValueOnce({ kind: "status", taskId: "replacement-task", phase: "running", executionStopped: false })
.mockResolvedValueOnce({ kind: "state", taskId: "replacement-task",
state: { schema: "paperclip.runner.durable.state.v1", ...identity } })
.mockResolvedValueOnce({ ...description, launchPreparationVersion: 1, projectAuthCheckVersion: 1 });
const api = vi.spyOn(runtimeApi, "resolveLocalRuntimeApiUrl").mockReturnValue("http://127.0.0.1:3100");
try { await expect(recover()).resolves.toBeDefined(); }
finally { api.mockRestore(); }
expect(f.workers.call.mock.calls.map(([, , call]) => [call.taskId, call.operation.kind])).toEqual([
["replacement-task", "status"], ["replacement-task", "read_state"], ["replacement-task", "describe"],
]);
return recover;
}
if ("freshSession" in scenario) {
const previousExecution = f.sourceExecution;
const nativeToolContractFingerprint = nativeToolContractFingerprintForTarget("remote", f.execution.provider.kind);
const normalizedSessionId = f.identity.normalizedSessionId;
const previousRun = {
id: f.sourceRunId, companyId: f.companyId, agentId: f.execution.binding.agentId, nativeSessionId: normalizedSessionId,
runnerProfileJson: { nativeToolContractFingerprint, nativeExecutionInput: previousExecution,
sessionCheckpoint: { backendKind: "runner", driverKind: previousExecution.session.driverKind,
sessionId: "previous-provider-thread", providerSessionId: "previous-provider-thread",
identity: { ...previousExecution.binding, sessionId: normalizedSessionId }, pendingRuntimeRequests: [] } },
};
const select = (current: NativeExecutionInput) => buildNativeExecutionWithCheckpoint({
previousRun, normalizedSessionId, executionTargetKind: "remote", toolRefreshOnResume: true,
buildExecution: ({ normalizedSessionId }) => ({ ...current, session: { ...current.session, normalizedSessionId } } as NativeExecutionInput),
});
expect(select(f.execution).checkpoint?.providerSessionId).toBe("previous-provider-thread");
const changed = structuredClone(f.execution);
if (!("runtimeContext" in changed)) throw new Error("fixture requires runtime context");
changed.runtimeContext.instructions.bundle.digest = "1".repeat(64);
changed.runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(changed.runtimeContext);
const selected = select(changed);
expect(selected.checkpoint).toBeNull();
expect(selected.normalizedSessionId).not.toBe(normalizedSessionId);
const scope = { attachedRunId: f.runId, accountIdentity: f.accountIdentity, selectedSessionId: selected.normalizedSessionId };
const sourceFor = (identity: unknown) => readyPortableTaskSource(restarted, f.companyId, f.leaseId, identity, scope);
await expect(sourceFor(null)).rejects.toThrow("ownership_unverified");
await db.update(heartbeatRuns).set({ nativeSessionId: selected.normalizedSessionId }).where(eq(heartbeatRuns.id, f.runId));
await expect(sourceFor(null)).rejects.toThrow("ownership_unverified");
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: selected.execution } })
.where(eq(heartbeatRuns.id, f.runId));
await expect(sourceFor(null)).resolves.toBeUndefined();
await expect(sourceFor(f.identity)).rejects.toThrow("ownership_unverified");
const duplicate = randomUUID();
await db.insert(environmentLeases).values({ ...target, id: duplicate, providerLeaseId: "ambiguous-task" });
await expect(sourceFor(null)).rejects.toThrow("ownership_unverified");
await db.delete(environmentLeases).where(eq(environmentLeases.id, duplicate));
const fresh = { ...f.identity, runId: f.runId, environmentLeaseId: f.leaseId, runnerInstanceId: randomUUID(),
normalizedSessionId: selected.normalizedSessionId, turnId: "fresh-turn", itemId: "fresh-item" };
const prepare = { ...f.operation, resumeFrom: undefined, requirements: { ...f.operation.requirements, identity: fresh } };
const submit = { ...f.submit, resumeFrom: undefined, runner: { ...f.submit.runner,
runnerId: fresh.runnerInstanceId, leaseId: fresh.environmentLeaseId, runId: fresh.runId,
sessionId: fresh.normalizedSessionId, turnId: fresh.turnId, itemId: fresh.itemId } };
f.workers.call.mockClear();
await expect(f.task(prepare)).resolves.toMatchObject({ kind: "prepared" });
await expect(f.task({ kind: "check_project_auth", provider: "openai", projectIds: [],
configurationId: submit.preparedLaunchId })).resolves.toMatchObject({ kind: "project_auth", allowed: true });
await expect(f.task(submit)).resolves.toMatchObject({ kind: "accepted" });
expect(f.workers.call.mock.calls.map(([, , call]) => call.operation.kind)).toEqual(["prepare", "check_project_auth", "submit"]);
expect(f.workers.call.mock.calls.every(([, , call]) => call.taskId === "replacement-task" &&
call.operation.resumeFrom === undefined && call.operation.priorIdentity === undefined)).toBe(true);
await expect(sourceFor(null)).rejects.toThrow("ownership_unverified");
await expect(sourceFor({ ...fresh, turnId: "wrong-turn" })).rejects.toThrow("ownership_unverified");
await expect(sourceFor(fresh)).resolves.toBeUndefined();
await recoverAttachedRunner(selected.execution, fresh);
return;
}
const resume = { kind: "resume", identity: f.identity, bootstrapTicket: "fixture-ticket" } as const;
f.workers.call.mockResolvedValueOnce({ kind: "resume", taskId: "replacement-task", outcome: "resumed" });
await expect(f.task(resume)).resolves.toMatchObject({ kind: "resume", outcome: "resumed" });
const [dispatched] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
expect(dispatched.metadata?.runnerTaskDispatched).toBe(true);
const readState = { kind: "read_state", identity: f.identity } as const;
await expect(f.task({ ...readState, identity: { ...f.identity, itemId: "wrong-item" } }))
.rejects.toThrow("ownership_unverified");
f.workers.call.mockResolvedValueOnce({ kind: "state", taskId: "replacement-task", state: { ...f.identity } });
await expect(f.task(readState)).resolves.toMatchObject({ kind: "state", state: f.identity });
const nextIdentity = { ...f.identity, runId: f.runId, turnId: "target-turn", itemId: "target-item" };
const attach = { ...f.operation, resumeFrom: undefined, priorIdentity: f.identity,
requirements: { ...f.operation.requirements, identity: nextIdentity } };
await expect(f.task({ ...attach, requirements: { ...attach.requirements,
identity: { ...nextIdentity, normalizedSessionId: randomUUID() } } })).rejects.toThrow("prior identity mismatch");
await expect(f.task(attach)).resolves.toMatchObject({ kind: "prepared" });
await expect(f.task(attach)).resolves.toMatchObject({ kind: "prepared" });
const [attached] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
expect(attached.metadata?.runnerTaskIdentity).toMatchObject({ runId: f.runId, sessionId: f.identity.normalizedSessionId });
expect(attached.metadata?.runnerTaskPriorIdentity).toMatchObject({ runId: f.sourceRunId, itemId: f.identity.itemId });
const scope = { attachedRunId: f.runId, accountIdentity: f.accountIdentity };
await expect(readyPortableTaskSource(db, f.companyId, f.leaseId, nextIdentity, scope)).resolves.toBeUndefined();
await expect(readyPortableTaskSource(db, f.companyId, f.leaseId, { ...nextIdentity, turnId: "wrong-turn" }, scope))
.rejects.toThrow("ownership_unverified");
await expect(readyPortableTaskSource(db, f.companyId, f.leaseId, f.identity, scope))
.resolves.toEqual({ taskId: "source-task", identity: f.identity });
await expect(readyPortableTaskSource(db, f.companyId, f.leaseId, f.identity, { ...scope, accountIdentity: "another-account" }))
.rejects.toThrow("ownership_unverified");
const binding = { companyId: f.companyId, leaseId: f.leaseId, taskId: "replacement-task", runId: f.runId, identity: nextIdentity };
await expect(assertAttachedPortableTask(restarted, binding)).resolves.toBeUndefined();
for (const invalid of [
{ ...binding, leaseId: f.sourceLeaseId },
{ ...binding, taskId: "source-task" },
{ ...binding, identity: f.identity },
{ ...binding, identity: { ...nextIdentity, environmentLeaseId: f.leaseId } },
]) await expect(assertAttachedPortableTask(restarted, invalid)).rejects.toThrow("ownership_unverified");
if (policy === "ordinary") {
const recover = await recoverAttachedRunner(f.execution, nextIdentity);
await db.update(environmentLeases).set({
metadata: sql`${environmentLeases.metadata} - 'portableTaskSuccessor'`,
}).where(eq(environmentLeases.id, f.sourceLeaseId));
f.workers.call.mockClear();
await expect(recover()).rejects.toThrow("ownership_unverified");
expect(f.workers.call).not.toHaveBeenCalled();
}
});
it.each(["codex", "claude"] as const)("authorizes the claimed %s source while preserving A's accepted outcome", async provider => {
const f = await fixture(provider);
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
@@ -5,6 +5,7 @@ import { parseNativeExecutionInput, type NativeExecutionInput, type PersistedNat
import { currentNativeControllerIdentity, evaluateNativeControllerTakeover, recordLegacyHostFallbackProof } from "./native-restart-recovery.js";
import { NativeControllerRecoveryWaitingError, readNativeControllerRecovery } from "./native-controller-recovery.js";
import { NativePredecessorRetirementUnprovenError } from "./predecessor-retirement.js";
import { readyPortableTaskSource } from "../plugin-task-ownership.js";
export type RemoteWarmSessionRecovery = {
identity: Record<string, string>;
@@ -12,6 +13,7 @@ export type RemoteWarmSessionRecovery = {
previousExecution: NativeExecutionInput;
reuseProcess: boolean;
releasedSource?: DurableRecoveryIdentity;
portableSourceTaskId?: string;
remote: { providerLeaseId: string; remoteCwd: string };
revalidate: (leaseOwner: string) => Promise<void>;
};
@@ -128,6 +130,7 @@ export async function inspectRemoteWarmSessionRecovery(input: {
matchesExecution: (previous: NativeExecutionInput) => boolean;
loadCheckpoint: (previous: NativeExecutionInput) => PersistedNativeSession | null;
canReuseProcess: (previous: NativeExecutionInput) => boolean;
portableSourceTaskId?: string;
}): Promise<RemoteWarmSessionRecovery | null> {
const { binding } = input.execution;
const { target } = input;
@@ -152,13 +155,15 @@ export async function inspectRemoteWarmSessionRecovery(input: {
// sandbox (for example after a restart or deploy replaced the controller)
// holds no Runner to adopt, so live adoption steps aside. Ordinary admission
// then retires the finished owner or keeps it held with a named reason.
if (target.transport === "sandbox" && ["created", "replacement"].includes(target.sandboxLeaseAcquisition?.outcome ?? "")) {
if (!input.portableSourceTaskId && target.transport === "sandbox" && ["created", "replacement"].includes(target.sandboxLeaseAcquisition?.outcome ?? "")) {
return null;
}
if (target.transport !== "sandbox" || !target.leaseId || !target.environmentId ||
!target.providerKey || !target.sandboxLeaseAcquisition || target.sandboxLeaseAcquisition.outcome !== "resumed" ||
!target.providerKey || !target.sandboxLeaseAcquisition ||
(!input.portableSourceTaskId && target.sandboxLeaseAcquisition.outcome !== "resumed") ||
identity.runId === binding.runId) return deny("target_not_resumed_sandbox");
const providerLeaseId = target.sandboxLeaseAcquisition.providerLeaseId;
const sourceProviderLeaseId = input.portableSourceTaskId ?? providerLeaseId;
const [prior] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations })
.from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
.where(and(eq(heartbeatRuns.id, identity.runId), eq(heartbeatRuns.companyId, binding.companyId),
@@ -171,7 +176,7 @@ export async function inspectRemoteWarmSessionRecovery(input: {
throw new NativeControllerRecoveryWaitingError(prior.owner.leaseExpiresAt);
}
if (prior.owner.phase !== "committed" || prior.owner.leaseOwner) return deny("owner_unsettled");
settledPredecessor = { runId: identity.runId, providerLeaseId };
if (sourceProviderLeaseId === providerLeaseId) settledPredecessor = { runId: identity.runId, providerLeaseId };
const checkpoint = input.loadCheckpoint(previous);
if (!checkpoint || checkpoint.identity.runId !== identity.runId ||
checkpoint.identity.companyId !== binding.companyId || checkpoint.identity.issueId !== binding.issueId ||
@@ -194,15 +199,22 @@ export async function inspectRemoteWarmSessionRecovery(input: {
const priorLeases = await input.db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, binding.companyId), eq(environmentLeases.heartbeatRunId, identity.runId),
eq(environmentLeases.issueId, binding.issueId), eq(environmentLeases.environmentId, target.environmentId),
eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, providerLeaseId),
eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, sourceProviderLeaseId),
));
if (priorLeases.length !== 1 || priorLeases[0].metadata?.remoteCwd !== target.remoteCwd ||
!["released", "expired", "active"].includes(priorLeases[0].status)) return deny("prior_lease_mismatch");
!["released", "expired", "active", ...(input.portableSourceTaskId ? ["retained"] : [])].includes(priorLeases[0].status)) {
return deny("prior_lease_mismatch");
}
return {
identity, checkpoint, previousExecution: previous, reuseProcess: input.canReuseProcess(previous),
portableSourceTaskId: input.portableSourceTaskId,
remote: { providerLeaseId, remoteCwd: target.remoteCwd },
revalidate: async leaseOwner => {
await assertPriorStopped();
if (input.portableSourceTaskId &&
(await readyPortableTaskSource(input.db, binding.companyId, target.leaseId!, identity))?.taskId !== input.portableSourceTaskId) {
return deny("prior_lease_mismatch");
}
const controller = await currentNativeControllerIdentity();
const [current] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations, issue: issues, agent: agents, company: companies })
.from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
@@ -13,6 +13,12 @@ const handoffSchema = z.object({
const key = "runnerTaskCredentialHandoff";
type Scope = { companyId: string; leaseId: string; expectedTaskId?: string };
export function canReadStoppedPluginTaskCredential(metadata: Record<string, unknown>, accountIdentity: string): boolean {
const handoff = handoffSchema.safeParse(metadata[key]);
return metadata.runnerTaskExecutionStopped === true && handoff.success &&
handoff.data.state === "pending" && handoff.data.accountIdentity === accountIdentity;
}
/** Save authority, never credential bytes or a controller-local temporary path. */
export async function registerPluginTaskCredentialHandoff(
db: Db,
@@ -0,0 +1,319 @@
import { setTimeout as delay } from "node:timers/promises";
import { and, asc, eq, inArray, isNotNull, isNull, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, issues, type Db } from "@paperclipai/db";
import { environmentTaskOperationSchema, environmentTaskResultSchema, type PluginEnvironmentTaskOperation, type PluginEnvironmentTaskResult } from "@paperclipai/plugin-sdk";
import { nativeSha256 } from "./native-runtime/canonical.js";
import { lockSourceRecoveryCoordinators } from "./native-runtime/native-source-recovery-ownership.js";
import type { PluginTaskDescription } from "./environment-task-runtime.js";
type OwnershipOperation = Extract<PluginEnvironmentTaskOperation, { kind: "ensure_ownership" }>;
type OwnershipResult = Extract<PluginEnvironmentTaskResult, { kind: "ownership" }>;
type Lease = typeof environmentLeases.$inferSelect;
type OwnershipClaim = {
sourceLeaseId: string;
targetLeaseId: string;
operation: OwnershipOperation;
};
export function unlaunchedPortableTaskLease() {
const { metadata } = environmentLeases;
const unlaunched = and(
eq(environmentLeases.status, "active"), isNull(environmentLeases.releasedAt), isNull(environmentLeases.cleanupStatus),
isNotNull(environmentLeases.providerLeaseId),
or(isNull(environmentLeases.expiresAt), sql`${environmentLeases.expiresAt} > clock_timestamp()`),
sql`${metadata}->>'driver' = 'plugin' and ${metadata}->>'runnerTaskLease' = 'true'
and jsonb_typeof(${metadata}->'portableTaskPreviousLeaseId') = 'string'
and not (${metadata} ?| array['portableTaskSuccessor', 'runnerTaskIdentity', 'runnerTaskStartup', 'runnerTaskStartupCleanup'])
and coalesce(${metadata}->>'runnerTaskDispatched', 'false') = 'false'
and coalesce(${metadata}->>'runnerTaskRetiring', 'false') = 'false'`,
)!;
return sql<boolean>`case when ${unlaunched} then not exists (select 1 from environment_leases other
where other.company_id = ${environmentLeases.companyId} and other.heartbeat_run_id = ${environmentLeases.heartbeatRunId}
and other.status = 'active' and other.id <> ${environmentLeases.id}) else false end`;
}
export function portableTaskPreparationLeaseId(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "issueId" | "agentId">) {
return sql<string | null>`(${db.select({ id: environmentLeases.id }).from(issues).innerJoin(environmentLeases, and(
eq(environmentLeases.companyId, issues.companyId), eq(environmentLeases.issueId, issues.id),
eq(environmentLeases.heartbeatRunId, issues.executionRunId),
)).where(and(eq(issues.companyId, run.companyId), run.issueId ? eq(issues.id, run.issueId) : sql`false`,
eq(issues.executionRunId, run.id), eq(issues.assigneeAgentId, run.agentId), eq(issues.status, "in_progress"),
sql`${environmentLeases.metadata}->>'agentId' = ${run.agentId}`, unlaunchedPortableTaskLease()))})`;
}
const object = (value: unknown): Record<string, unknown> =>
value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
export class PluginTaskOwnershipPendingError extends Error {
constructor() {
super("native_portable_task_ownership_pending");
}
}
function unavailable(): never {
throw new Error("native_portable_task_ownership_unverified");
}
function leaseGeneration(lease: Lease) {
return object(lease.metadata?.providerMetadata).ownershipGeneration;
}
function readOperation(source: Lease, target: Lease, policy: OwnershipOperation["policy"]): OwnershipOperation {
const identity = object(source.metadata?.runnerTaskIdentity);
const operation = environmentTaskOperationSchema.safeParse({
kind: "ensure_ownership", operationId: target.id, sourceTaskId: source.providerLeaseId,
sourceGeneration: leaseGeneration(source), targetGeneration: leaseGeneration(target), policy,
identity: {
runnerInstanceId: identity.runnerId, environmentLeaseId: identity.leaseId,
runId: identity.runId, normalizedSessionId: identity.sessionId,
turnId: identity.turnId, itemId: identity.itemId,
},
});
return operation.success && operation.data.kind === "ensure_ownership" ? operation.data : unavailable();
}
async function ownershipClaim(db: Db, input: {
companyId: string;
sourceLeaseId: string;
targetLeaseId: string;
policy: OwnershipOperation["policy"];
}, ready?: OwnershipResult): Promise<OwnershipClaim> {
const { companyId, sourceLeaseId, targetLeaseId, policy } = input;
if (sourceLeaseId === targetLeaseId) return unavailable();
return db.transaction(async tx => {
const scope = and(eq(environmentLeases.companyId, companyId), inArray(environmentLeases.id, [sourceLeaseId, targetLeaseId]));
const initial = await tx.select().from(environmentLeases).where(scope);
const sourceRunId = initial.find(lease => lease.id === sourceLeaseId)?.heartbeatRunId;
const targetRunId = initial.find(lease => lease.id === targetLeaseId)?.heartbeatRunId;
const issueId = initial.find(lease => lease.id === sourceLeaseId)?.issueId;
if (!sourceRunId || !targetRunId || sourceRunId === targetRunId || !issueId) return unavailable();
const owners = await lockSourceRecoveryCoordinators(tx, companyId, [sourceRunId, targetRunId]);
const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, companyId), eq(issues.id, issueId))).for("update", { noWait: true });
const leases = await tx.select().from(environmentLeases).where(scope).orderBy(asc(environmentLeases.id)).for("update");
const source = leases.find(lease => lease.id === sourceLeaseId);
const target = leases.find(lease => lease.id === targetLeaseId);
const runs = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
inArray(heartbeatRuns.id, [sourceRunId, targetRunId])));
const prior = runs.find(run => run.id === sourceRunId);
const current = runs.find(run => run.id === targetRunId);
const sourceOwner = owners.find(owner => owner.runId === sourceRunId);
if (!source || !target || source.heartbeatRunId !== sourceRunId || target.heartbeatRunId !== targetRunId ||
!source.provider?.startsWith("plugin:") || source.provider !== target.provider ||
!source.providerLeaseId || !target.providerLeaseId || source.providerLeaseId === target.providerLeaseId ||
!source.environmentId || source.environmentId !== target.environmentId ||
source.issueId !== issueId || target.issueId !== issueId || source.executionWorkspaceId !== target.executionWorkspaceId ||
source.metadata?.driver !== "plugin" || target.metadata?.driver !== "plugin" ||
typeof source.metadata.pluginId !== "string" || source.metadata.pluginId !== target.metadata.pluginId ||
typeof source.metadata.driverKey !== "string" || source.metadata.driverKey !== target.metadata.driverKey ||
source.metadata.runnerTaskLease !== true || target.metadata.runnerTaskLease !== true ||
source.metadata.runnerTaskRetiring === true || source.metadata.reusableLeaseSuccessor !== undefined ||
source.metadata.runnerTaskSourceConsumer !== undefined ||
!["retained", "released", "expired"].includes(source.status) ||
target.status !== "active" || target.releasedAt || target.metadata.runnerTaskRetiring === true ||
target.expiresAt && target.expiresAt.getTime() <= Date.now() ||
prior?.status !== "succeeded" || prior.nativeIssueId !== issueId ||
sourceOwner?.phase !== "committed" || !sourceOwner.resultId || sourceOwner.leaseOwner ||
current?.status !== "running" || current.scopeKind !== "issue" || current.issueId !== issueId ||
current.nativeIssueId !== null && current.nativeIssueId !== issueId || current.resultJson?.nativeCancellation ||
prior.agentId !== current.agentId || issue?.executionRunId !== targetRunId || issue.assigneeAgentId !== current.agentId ||
issue.status !== "in_progress") return unavailable();
const operation = readOperation(source, target, policy);
if (operation.identity.runId !== prior.id || operation.identity.normalizedSessionId !== prior.nativeSessionId ||
current.nativeSessionId !== null && current.nativeSessionId !== prior.nativeSessionId) return unavailable();
const claim = { sourceLeaseId, targetLeaseId, operation };
const existing = [source.metadata.portableTaskSuccessor, target.metadata.portableTaskPredecessor];
if (existing.some(saved => saved !== undefined && nativeSha256(saved) !== nativeSha256(claim)) ||
(existing[0] === undefined) !== (existing[1] === undefined)) return unavailable();
if (ready && (existing[0] === undefined || ready.taskId !== target.providerLeaseId ||
ready.operationId !== operation.operationId || ready.targetGeneration !== operation.targetGeneration ||
ready.status !== "ready")) return unavailable();
if (!ready && existing[0] === undefined) {
await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ portableTaskSuccessor: claim })}::jsonb` })
.where(eq(environmentLeases.id, sourceLeaseId));
await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ portableTaskPredecessor: claim })}::jsonb` })
.where(eq(environmentLeases.id, targetLeaseId));
} else if (ready && target.metadata.portableTaskOwnershipReady === true &&
(target.metadata.portableTaskCheckpointUnavailable === true) !== (ready.checkpointUnavailable === true)) {
return unavailable();
} else if (ready && target.metadata.portableTaskOwnershipReady !== true) {
await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({
portableTaskOwnershipReady: true, ...(ready.checkpointUnavailable ? { portableTaskCheckpointUnavailable: true }
: { runnerTaskAccountIdentity: source.metadata.runnerTaskAccountIdentity }),
})}::jsonb` })
.where(eq(environmentLeases.id, targetLeaseId));
}
return claim;
});
}
export async function reconcilePluginTaskOwnership(db: Db, input: Parameters<typeof ownershipClaim>[1],
dispatch: (operation: OwnershipOperation) => Promise<PluginEnvironmentTaskResult>): Promise<OwnershipResult> {
try {
const claim = await ownershipClaim(db, input);
const result = await dispatch(claim.operation);
if (result.kind !== "ownership" || result.operationId !== claim.operation.operationId ||
result.targetGeneration !== claim.operation.targetGeneration) return unavailable();
if (result.status === "ready") await ownershipClaim(db, input, result);
return result;
} catch (error) {
if ((object(object(error).cause).code ?? object(error).code) === "55P03") throw new PluginTaskOwnershipPendingError();
throw error;
}
}
export function assertPluginTaskOwnershipDispatch(lease: Lease, operation: OwnershipOperation) {
const claim = object(lease.metadata?.portableTaskPredecessor);
if (claim.targetLeaseId !== lease.id || claim.operation === undefined ||
operation.targetGeneration !== leaseGeneration(lease) ||
nativeSha256(claim.operation) !== nativeSha256(operation)) return unavailable();
}
/** Reconcile the declared predecessor before admitting the task on this lease. */
export async function reconcilePluginTaskAdmission(db: Db, lease: Lease,
dispatch: (operation: OwnershipOperation) => Promise<PluginEnvironmentTaskResult>, signal?: AbortSignal): Promise<void> {
let sourceLeaseId = lease.metadata?.portableTaskPreviousLeaseId;
let policy: OwnershipOperation["policy"] = "ordinary";
if (lease.metadata?.portableTaskPredecessor !== undefined) {
const claim = object(lease.metadata.portableTaskPredecessor);
const operation = environmentTaskOperationSchema.safeParse(claim.operation);
if (!operation.success || operation.data.kind !== "ensure_ownership") return unavailable();
assertPluginTaskOwnershipDispatch(lease, operation.data);
sourceLeaseId = claim.sourceLeaseId;
policy = operation.data.policy;
}
if (typeof sourceLeaseId !== "string") return unavailable();
const request = { companyId: lease.companyId, sourceLeaseId, targetLeaseId: lease.id, policy };
let backoffMs = 250;
for (;;) {
signal?.throwIfAborted();
try {
const result = await reconcilePluginTaskOwnership(db, request, dispatch);
if (result.status === "ready") {
signal?.throwIfAborted();
return;
}
} catch (error) {
if (!(error instanceof PluginTaskOwnershipPendingError)) throw error;
}
if (!signal) throw new PluginTaskOwnershipPendingError();
await delay(backoffMs, undefined, { signal });
backoffMs = Math.min(backoffMs * 2, 5_000);
}
}
export async function readPortableTaskReseed(db: Db, input: {
companyId: string;
runId: string;
lease: Pick<Lease, "id" | "metadata">;
description?: PluginTaskDescription;
runnerTask: (operation: { kind: "describe" }) => Promise<unknown>;
}): Promise<{ description: PluginTaskDescription; reseed: { previousSessionId: string | null } | null } | null> {
if (!input.lease.metadata?.portableTaskPreviousLeaseId) return null;
const readCurrent = async () => (await db.select({
metadata: environmentLeases.metadata, providerLeaseId: environmentLeases.providerLeaseId,
}).from(environmentLeases).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.heartbeatRunId, input.runId))))[0];
let description = input.description;
let current = description ? await readCurrent() : undefined;
if (!description || current?.providerLeaseId !== description.taskId || current.metadata?.portableTaskOwnershipReady !== true) {
const result = environmentTaskResultSchema.parse(await input.runnerTask({ kind: "describe" }));
if (result.kind !== "description") throw new Error("Plugin Runner description unavailable");
description = result;
current = await readCurrent();
}
if (current?.providerLeaseId !== description.taskId || current.metadata?.portableTaskOwnershipReady !== true) return unavailable();
if (current.metadata.portableTaskCheckpointUnavailable !== true) return { description, reseed: null };
const predecessor = object(current.metadata.portableTaskPredecessor);
const identity = object(object(predecessor.operation).identity);
const sessionId = identity.normalizedSessionId;
return { description, reseed: { previousSessionId: typeof sessionId === "string" && sessionId.trim() ? sessionId : null } };
}
async function readReadyPortableTaskClaim(db: Db, companyId: string, targetLeaseId: string) {
const [target] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId),
eq(environmentLeases.id, targetLeaseId)));
if (!target?.metadata?.portableTaskPredecessor) return undefined;
if (target.metadata.portableTaskCheckpointUnavailable === true) return undefined;
const claim = object(target.metadata.portableTaskPredecessor);
const operation = environmentTaskOperationSchema.safeParse(claim.operation);
if (target.status !== "active" || target.releasedAt || target.metadata.portableTaskOwnershipReady !== true ||
target.metadata.portableTaskSuccessor !== undefined || target.metadata.runnerTaskRetiring === true ||
target.expiresAt && target.expiresAt.getTime() <= Date.now() ||
!operation.success || operation.data.kind !== "ensure_ownership" ||
typeof claim.sourceLeaseId !== "string") return unavailable();
assertPluginTaskOwnershipDispatch(target, operation.data);
const [source] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId),
eq(environmentLeases.id, claim.sourceLeaseId)));
if (!source || source.provider !== target.provider || source.providerLeaseId !== operation.data.sourceTaskId ||
leaseGeneration(source) !== operation.data.sourceGeneration ||
nativeSha256(source.metadata?.portableTaskSuccessor ?? null) !== nativeSha256(claim)) return unavailable();
return { source, target, operation: operation.data };
}
function matchesAttachedPortableTaskIdentity(
target: Lease, identity: Record<string, unknown>, runId: string,
) {
const saved = object(target.metadata?.runnerTaskIdentity);
const attached = {
runnerInstanceId: saved.runnerId, environmentLeaseId: saved.leaseId,
runId: saved.runId, normalizedSessionId: saved.sessionId, turnId: saved.turnId, itemId: saved.itemId,
};
return target.heartbeatRunId === runId && identity.runId === runId &&
Object.entries(attached).every(([key, value]) => typeof value === "string" && value.length > 0 && identity[key] === value);
}
function matchesPortableLineage(source: OwnershipOperation["identity"], identity: Record<string, unknown>) {
return source.environmentLeaseId === identity.environmentLeaseId && source.runnerInstanceId === identity.runnerInstanceId &&
source.normalizedSessionId === identity.normalizedSessionId;
}
export async function assertAttachedPortableTask(db: Db, input: {
companyId: string; leaseId: string; taskId: string; runId: string; identity: Record<string, unknown>;
}) {
const claim = await readReadyPortableTaskClaim(db, input.companyId, input.leaseId);
if (!claim || claim.target.providerLeaseId !== input.taskId ||
!matchesAttachedPortableTaskIdentity(claim.target, input.identity, input.runId) ||
!matchesPortableLineage(claim.operation.identity, input.identity)) return unavailable();
}
export async function readyPortableTaskSource(db: Db, companyId: string, targetLeaseId: string, identity: unknown,
scope: { attachedRunId?: string; selectedSessionId?: string; accountIdentity?: string } = {}) {
const claim = await readReadyPortableTaskClaim(db, companyId, targetLeaseId);
if (!claim) return undefined;
const { source, target, operation } = claim;
if ("accountIdentity" in scope && (!scope.accountIdentity || source.metadata?.runnerTaskAccountIdentity !== scope.accountIdentity))
return unavailable();
const candidate = object(identity);
const { attachedRunId, selectedSessionId } = scope;
if (selectedSessionId && selectedSessionId !== operation.identity.normalizedSessionId) {
if (!attachedRunId || target.heartbeatRunId !== attachedRunId) return unavailable();
const [admitted] = await db.select({ lease: environmentLeases, unlaunched: unlaunchedPortableTaskLease() })
.from(environmentLeases).innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId),
eq(heartbeatRuns.companyId, environmentLeases.companyId)))
.innerJoin(issues, and(eq(issues.id, environmentLeases.issueId), eq(issues.companyId, environmentLeases.companyId)))
.where(and(eq(environmentLeases.id, targetLeaseId), eq(environmentLeases.companyId, companyId),
eq(environmentLeases.providerLeaseId, target.providerLeaseId!),
sql`${environmentLeases.metadata}->'portableTaskPredecessor' = ${JSON.stringify(target.metadata?.portableTaskPredecessor)}::jsonb`,
eq(heartbeatRuns.id, attachedRunId), eq(heartbeatRuns.status, "running"),
sql`${heartbeatRuns.resultJson}->'nativeCancellation' is null`, eq(heartbeatRuns.nativeSessionId, selectedSessionId),
sql`${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'session'->>'normalizedSessionId' = ${selectedSessionId}`,
eq(heartbeatRuns.nativeIssueId, issues.id), eq(issues.executionRunId, heartbeatRuns.id),
eq(issues.assigneeAgentId, heartbeatRuns.agentId), eq(issues.status, "in_progress")));
if (!admitted) return unavailable();
if (identity == null && admitted.unlaunched) return undefined;
if (admitted.lease.metadata?.runnerTaskDispatched === true &&
candidate.normalizedSessionId === selectedSessionId && candidate.environmentLeaseId === target.id &&
matchesAttachedPortableTaskIdentity(admitted.lease, candidate, attachedRunId)) return undefined;
return unavailable();
}
if (attachedRunId && target.heartbeatRunId === attachedRunId && candidate.runId === attachedRunId) {
if (matchesAttachedPortableTaskIdentity(target, candidate, attachedRunId) &&
matchesPortableLineage(operation.identity, candidate)) return undefined;
return unavailable();
}
if (Object.entries(operation.identity).some(([key, value]) => candidate[key] !== value)) return unavailable();
return { taskId: operation.sourceTaskId, identity: operation.identity };
}
@@ -19,6 +19,7 @@ export async function resolvePluginTaskResume(db: Db, input: {
const identity = row.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
const consumer = row.lease.metadata?.runnerTaskSourceConsumer as { leaseId?: unknown; runId?: unknown; resumeFrom?: unknown } | null | undefined;
return row.lease.id !== input.leaseId && row.run.id === input.priorIdentity.runId &&
row.lease.metadata?.portableTaskSuccessor === undefined &&
(!Object.hasOwn(row.lease.metadata ?? {}, "runnerTaskSourceConsumer") ||
consumer?.leaseId === input.leaseId && consumer.runId === next?.run.id && consumer.resumeFrom === row.lease.providerLeaseId) &&
(input.inspectCheckpoint || row.lease.metadata?.runnerTaskSessionRetained === true) &&
@@ -13,7 +13,7 @@
"packages/plugins/sdk/src/environment-stop-error.ts": "b0e03f8aa2b2170ba095da5c7138050fced3f3cb",
"packages/plugins/sdk/src/environment-sync-error.ts": "7d6a51a95da49a7da4525aebbab6382dc5c44ae8",
"packages/plugins/sdk/src/environment-task-error.ts": "98aae8a06743b38884b68cb9a3ec44c57b34c0e4",
"packages/plugins/sdk/src/environment-tasks.ts": "98e5f69a00b81eb6e8c25bdb5fcdc22e5485ef18",
"packages/plugins/sdk/src/environment-tasks.ts": "de583d809187f0677e25b95f22c3c42961a9d2e2",
"packages/plugins/sdk/src/host-client-factory.ts": "e0301c390ea13f2340b669383f07af2455e48c8a",
"packages/plugins/sdk/src/idle-drain.test.ts": "cc59f7881e7d46c89395a1d93d74f6c6b9b0a64a",
"packages/plugins/sdk/src/idle-drain.ts": "ae6e4c6be4657a3d541396922b3bc551b8db557f",
@@ -408,7 +408,7 @@
"packages/shared/src/types/issue.ts": "725bd6f677b9d461460697d1ce10dee739a66ebe",
"packages/shared/src/types/live.ts": "b90e55e4de8600c6fc17d7fa8d89079d4ae655ec",
"packages/shared/src/types/native-finalization.ts": "c026a73003ae232fea4f7624d561493b383df358",
"packages/shared/src/types/plugin.ts": "016059b873c7189b8e65bb0dfa1ffa039c97c2ca",
"packages/shared/src/types/plugin.ts": "22e1e35307f91fdf5c210b86bae7746594669516",
"packages/shared/src/types/project.ts": "78eeea3694ce860d5d0be59c158c399c03368078",
"packages/shared/src/types/quota.ts": "a0686a9afb3111a4de093f66200dc0c9152e883e",
"packages/shared/src/types/resource-memberships.ts": "9eab6fad50facf3e489f8c40e5bedd92e1d1faf8",
@@ -490,7 +490,7 @@
"packages/shared/src/validators/onboarding-seed.ts": "de302a8400d4199053f379d9ed853522a3ac6d32",
"packages/shared/src/validators/partial.ts": "fa60ec7481fc14175365e9e500c814b159f1ab53",
"packages/shared/src/validators/plugin.test.ts": "23bc443850d46d815877a71a44625fd42aebb00a",
"packages/shared/src/validators/plugin.ts": "2a90d3d00b64f802b9c30f957e97702250ab2db4",
"packages/shared/src/validators/plugin.ts": "782e051567729c2eb91a3b80e883b4a8ed996646",
"packages/shared/src/validators/project.ts": "6ea168f5faf3161e6004ea47274f60634c77b219",
"packages/shared/src/validators/provider-trace.ts": "b38b6200a2e51513f5dca0f8b008f2fe68175e82",
"packages/shared/src/validators/resource-memberships.ts": "b124ac8a8f24b4dbb03ae82bcc87541f6b4ecdfd",