mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path > - Paperclip delegates remote Runner execution to environment plugins. > - A plugin can move a session to another task allocation. > - The Core controller can lose an ownership reply or restart before the Runner starts. > - The source and target need a durable record of the same request. > - This change binds restart and attachment to that request and the original unlaunched task lease. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of plugin Runner sessions on replacement compute. **Current behavior** Core cannot replay a session ownership request from saved lease state during startup. **Proposed behavior** Record the operation on both leases and check the matched ownership receipt before attaching. Preserve the original pre-admission lease across controller restart. Managed environment configuration is in [#21065](https://github.com/paperclipai/paperclip/pull/21065). Related warm sandbox cleanup is in [#20065](https://github.com/paperclipai/paperclip/pull/20065). This layer preserves the same-sandbox fallback from [#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace placement follows in [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Add the optional `sessionHandoffVersion` capability and the ownership task operation. - Save matching ownership claims and fence source checkpoint access. Replay a pending request on the original target. - Restore a portable session using the durable claim and the current credentials for the same account. Start a fresh conversation when the provider reports an unavailable checkpoint, or when Core's persisted compatibility decision selects a different session after transfer. The latter requires an unlaunched target; subsequent recovery must match the fresh task's saved identity. Both paths retain the workspace transfer and source fencing. - Reuse the original unlaunched task lease after hard or graceful controller restart. After a move has attached, validate the surviving Runner through the saved claim rather than requiring its logical lease ID to equal the new physical lease. - Keep older retained leases without a saved account identity on the existing resume path. - Apply the existing supported MCP-only compatibility rule to warm recovery and predecessor accounting. Persist the capability discovered at admission and reuse it after restart without repeating discovery. - Keep predecessor-retirement fallback scoped to the same physical sandbox. A refusal after a portable move preserves the session instead of requesting a replacement that the plugin driver cannot enforce. ## Verification - On `1eb527be27`, server typecheck (including Runner build and SDK dependencies) passed. - Independent final verification passed 168 tests across portable source recovery, session resume, controller recovery, and task authority; two existing tests were skipped. The full portable source integration suite passed 64 tests. - The new integration follows confirmed ownership through an incompatible instruction change, persisted fresh admission, target-only prepare/auth/submit without the source checkpoint, and reattachment after controller replacement. It also rejects unpersisted session selection, ambiguous target leases, and incorrect saved identities. - Before this final change, independent verification passed 1,667 tests across executor, session resume, recovery usage, remote warm recovery, runtime selection, and runtime execution; the environment lifecycle suite passed 208 tests. - Existing regressions cover previous-release account metadata, supported and unsupported MCP changes, provider/instruction incompatibility, and admission capability reuse without another discovery call. - On `51d7eec373`, all four S3 multipart throttle tests passed independently. CI exposed an existing timing-sensitive test twice; its 20 ms overlap assumption now uses synchronized retries and an event-loop yield. Production upload code is unchanged. Temporary concurrency caps of one and eight both fail the retained exact-four assertion. - Exact SDK source-pin checks passed for both stack heads. The upper #20743 code patch remains unchanged; its documentation preserves both layers' behavior. ## Risks - The handoff protocol is opt-in. Existing plugin leases without portable ownership metadata keep the previous release's resume path. Live attachment still requires end-to-end qualification. An unverified claim blocks startup. Git workspaces prepared on a portable allocation also need the workspace placement layer. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
a4a0668a5c
commit
a924d2e2c7
40 files changed
+1585
-223
No files matched your search
@@ -172,6 +172,15 @@ It does not qualify a deployed plugin, live account, or production WSS proxy.
|
||||
|
||||
## Workspace completion and maintenance
|
||||
|
||||
A portable restore uses the durable ownership claim and the same account
|
||||
identity after a credential refresh or controller replacement. It starts a
|
||||
new Runner on the target. Run attachment applies the current tools and
|
||||
runtime configuration to the restored provider session. After a confirmed
|
||||
workspace transfer, Core can use the same target to start a new conversation
|
||||
when its session compatibility check requires one. The new conversation
|
||||
starts without the prior provider checkpoint, and the ownership claim
|
||||
continues to fence the source.
|
||||
|
||||
A storage provider can declare `workspaceProvider.workspaceVerificationVersion: 1`.
|
||||
Before creating a fresh workspace for an admitted Retry, Core may send the
|
||||
original task-owned workspace's `prepare` request with `verifyExisting: true`.
|
||||
|
||||
@@ -25,12 +25,13 @@ const githubBroker = z.object({ url: z.string().url().max(2000).refine(value =>
|
||||
return ["http:", "https:"].includes(url.protocol) && !url.username && !url.password && !url.search && !url.hash;
|
||||
} catch { return false; }
|
||||
}, "Expected a credential-free HTTP broker URL"), token: credentialValue }).strict();
|
||||
const claudeCredentialEnvironment = z.object({
|
||||
CLAUDE_CODE_OAUTH_TOKEN: credentialValue.optional(), ANTHROPIC_API_KEY: credentialValue.optional(),
|
||||
}).strict().refine(env => Object.values(env).filter(Boolean).length === 1, "Select one Claude credential");
|
||||
/** Only provider authentication is transferred. Paths and executable settings remain provider-owned. */
|
||||
export const environmentTaskCredentialSchema = z.discriminatedUnion("type", [
|
||||
z.object({ type: z.literal("codex-auth-json"), authJson: credentialValue }).strict(),
|
||||
z.object({ type: z.literal("claude-env"), environment: z.object({
|
||||
CLAUDE_CODE_OAUTH_TOKEN: credentialValue.optional(), ANTHROPIC_API_KEY: credentialValue.optional(),
|
||||
}).strict().refine(env => Object.values(env).filter(Boolean).length === 1, "Select one Claude credential") }).strict(),
|
||||
z.object({ type: z.literal("claude-env"), environment: claudeCredentialEnvironment }).strict(),
|
||||
]);
|
||||
export type PluginEnvironmentTaskCredential = z.infer<typeof environmentTaskCredentialSchema>;
|
||||
|
||||
@@ -114,7 +115,18 @@ export const environmentTaskOperationSchema = z.discriminatedUnion("kind", [
|
||||
z.object({ kind: z.literal("status") }).strict(),
|
||||
z.object({ kind: z.literal("connection") }).strict(),
|
||||
z.object({ kind: z.literal("release"), identity: durableIdentity }).strict(),
|
||||
z.object({ kind: z.literal("resume"), identity: durableIdentity, bootstrapTicket: credentialValue }).strict(),
|
||||
z.object({ kind: z.literal("resume"), identity: durableIdentity, bootstrapTicket: credentialValue,
|
||||
handoffCredential: z.object({ accountIdentity: z.string().min(1).max(1024), environment: claudeCredentialEnvironment }).strict().optional(),
|
||||
}).strict(),
|
||||
z.object({
|
||||
kind: z.literal("ensure_ownership"),
|
||||
operationId: z.string().uuid(),
|
||||
identity: durableIdentity,
|
||||
sourceTaskId: providerId.max(1024),
|
||||
sourceGeneration: providerId.max(1024),
|
||||
targetGeneration: providerId.max(1024),
|
||||
policy: z.enum(["ordinary", "reclaim_if_needed"]),
|
||||
}).strict(),
|
||||
z.object({ kind: z.literal("complete") }).strict(),
|
||||
z.object({ kind: z.literal("stop") }).strict(),
|
||||
]);
|
||||
@@ -125,6 +137,14 @@ export const environmentTaskResultSchema = z.discriminatedUnion("kind", [
|
||||
reason: z.enum(["session_state_unavailable", "account_binding_mismatch"]) }).strict(),
|
||||
z.object({ kind: z.literal("released"), taskId: providerId }).strict(),
|
||||
z.object({ kind: z.literal("resume"), taskId: providerId, outcome: z.enum(["resumed", "unavailable"]) }).strict(),
|
||||
z.object({
|
||||
kind: z.literal("ownership"),
|
||||
taskId: providerId,
|
||||
operationId: z.string().uuid(),
|
||||
targetGeneration: providerId.max(1024),
|
||||
status: z.enum(["pending", "ready"]),
|
||||
checkpointUnavailable: z.literal(true).optional(),
|
||||
}).strict(),
|
||||
z.object({ kind: z.literal("copilot_metadata"), taskId: providerId, metadata: z.record(z.string(), z.unknown()),
|
||||
executionStopped: z.literal(true) }).strict(),
|
||||
z.object({ kind: z.literal("project_auth"), taskId: providerId, configurationId: providerId.max(1024),
|
||||
@@ -142,6 +162,7 @@ export const environmentTaskResultSchema = z.discriminatedUnion("kind", [
|
||||
runner: z.object({ version: z.string().min(1), digest: z.string().regex(/^sha256:[a-f0-9]{64}$/) }).strict(),
|
||||
warmTaskVersion: z.literal(1).optional(),
|
||||
releaseResumeVersion: z.literal(1).optional(),
|
||||
sessionHandoffVersion: z.literal(1).optional(),
|
||||
launchPreparationVersion: z.literal(1).optional(),
|
||||
executionContextVersion: z.literal(1).optional(),
|
||||
/** Each version attests stopped-source recovery for its provider independently. */
|
||||
@@ -211,6 +232,13 @@ export interface PluginEnvironmentTaskParams extends Omit<PluginEnvironmentDrive
|
||||
/** An accepted stop is not proof of process termination. */
|
||||
export function parseEnvironmentTaskResult(operation: PluginEnvironmentTaskOperation, taskId: string, value: unknown): PluginEnvironmentTaskResult {
|
||||
const result = environmentTaskResultSchema.parse(value);
|
||||
if (operation.kind === "ensure_ownership") {
|
||||
if (result.kind !== "ownership" || result.taskId !== taskId || result.operationId !== operation.operationId ||
|
||||
result.targetGeneration !== operation.targetGeneration) {
|
||||
throw new Error("Invalid plugin-provided session ownership response");
|
||||
}
|
||||
return result;
|
||||
}
|
||||
const expected = operation.kind === "release" ? "released" : operation.kind === "resume" ? "resume" : operation.kind === "read_file" ? "file" : operation.kind === "describe" ? "description" : operation.kind === "prepare" ? "prepared" : operation.kind === "check_project_auth" ? "project_auth" : ["read_state", "archive_state"].includes(operation.kind) ? "state" : ["status", "connection", "credentials", "copilot_metadata"].includes(operation.kind) ? operation.kind : "accepted";
|
||||
const rejectedSubmit = operation.kind === "submit" && result.kind === "admission_rejected";
|
||||
if (result.taskId !== taskId || (result.kind !== expected && !rejectedSubmit)) throw new Error("Invalid plugin-provided Runner execution response");
|
||||
|
||||
@@ -17,6 +17,10 @@ const identity = { runnerInstanceId: "runner", environmentLeaseId: "lease", runI
|
||||
normalizedSessionId: "session", turnId: "turn", itemId: "item" };
|
||||
const release = { kind: "release", identity } as const;
|
||||
const resume = { kind: "resume", identity, bootstrapTicket: "fixture-ticket" } as const;
|
||||
const ownership = {
|
||||
kind: "ensure_ownership", operationId: "9df4526f-36c5-4fba-9cf6-3c4fd317f531", identity,
|
||||
sourceTaskId: "source-task", sourceGeneration: "source-generation", targetGeneration: "target-generation", policy: "reclaim_if_needed",
|
||||
} as const;
|
||||
const description = { kind: "description", taskId: "task", runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } };
|
||||
|
||||
async function invoke(operation: unknown, handler: (params: PluginEnvironmentTaskParams) => Promise<PluginEnvironmentTaskResult>) {
|
||||
@@ -89,6 +93,24 @@ describe("release and resume task contract", () => {
|
||||
.toMatchObject({ result: { kind: "resume", taskId: "task", outcome: "unavailable" } });
|
||||
});
|
||||
|
||||
it("binds a declarative ownership response to the exact worker request", async () => {
|
||||
for (const status of ["pending", "ready"] as const) {
|
||||
const result = { kind: "ownership", status, taskId: "task", operationId: ownership.operationId,
|
||||
targetGeneration: ownership.targetGeneration } as const;
|
||||
expect(await invoke(ownership, async params => {
|
||||
expect(params.operation).toEqual(ownership);
|
||||
return result;
|
||||
})).toMatchObject({ result });
|
||||
for (const changes of [
|
||||
{ taskId: "another-task" },
|
||||
{ operationId: "1f083ad0-a357-4f14-ad46-e9df981845b5" },
|
||||
{ targetGeneration: "stale-generation" },
|
||||
]) {
|
||||
expect(await invoke(ownership, async () => ({ ...result, ...changes }))).toHaveProperty("error");
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["source mismatch", "claim conflict", "authentication failed", "transport timeout"])("keeps %s as an RPC error", async message => {
|
||||
expect(await invoke(resume, async () => { throw new Error(message); })).toHaveProperty("error");
|
||||
});
|
||||
|
||||
@@ -201,6 +201,7 @@ export interface PluginEnvironmentDriverDeclaration {
|
||||
creationOperationVersion?: 1;
|
||||
/** Codex task retention and authorized run attachment, version 1. */
|
||||
warmTaskVersion?: 1;
|
||||
sessionHandoffVersion?: 1;
|
||||
/** Stable driver key, unique within the plugin. Namespaced by plugin ID at runtime. */
|
||||
driverKey: string;
|
||||
/**
|
||||
|
||||
@@ -183,6 +183,7 @@ export const pluginEnvironmentDriverDeclarationSchema = z.object({
|
||||
freshWorkspaceVersion: z.literal(1).optional(),
|
||||
creationOperationVersion: z.literal(1).optional(),
|
||||
warmTaskVersion: z.literal(1).optional(),
|
||||
sessionHandoffVersion: z.literal(1).optional(),
|
||||
driverKey: z.string().min(1).regex(
|
||||
/^[a-z0-9][a-z0-9._-]*$/,
|
||||
"Environment driver key must start with a lowercase alphanumeric and contain only lowercase letters, digits, dots, hyphens, or underscores",
|
||||
|
||||
@@ -33,6 +33,7 @@ import {
|
||||
executionWorkspaces,
|
||||
heartbeatRuns,
|
||||
issues,
|
||||
nativeRunFinalizations,
|
||||
plugins,
|
||||
projects,
|
||||
} from "@paperclipai/db";
|
||||
@@ -68,6 +69,9 @@ import { traceparentFromContextToken } from "../instrumentation.ts";
|
||||
import { ROOT_CONTEXT, trace } from "@opentelemetry/api";
|
||||
import { buildNativeHarnessBackupManifest } from "../services/native-runtime/native-session-executor.ts";
|
||||
import { createNativeHarnessBackupStamp } from "../services/native-runtime/native-harness-backup-stamp.ts";
|
||||
import { environmentRunOrchestrator } from "../services/environment-run-orchestrator.js";
|
||||
import { claimNativeRestartRecoveries } from "../services/native-runtime/native-restart-recovery.js";
|
||||
import { NATIVE_PREPARATION_FENCING_VERSION } from "../services/native-runtime/native-active-runs.js";
|
||||
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
|
||||
@@ -1300,6 +1304,81 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it.each(["handoff", "fresh", "legacy resume"] as const)("preserves the portable task placement on a run retry (%s)", async mode => {
|
||||
const resumeSession = mode !== "fresh";
|
||||
const pluginId = randomUUID();
|
||||
const pluginKey = "paperclip.test-portable-tasks";
|
||||
const seeded = await seedEnvironment({ driver: "plugin", config: { pluginKey, driverKey: "tasks", runnerLifecycleMode: "warm" } });
|
||||
await insertSandboxProviderPlugin(pluginId, { pluginKey, environmentDrivers: [{
|
||||
driverKey: "tasks", kind: "sandbox_provider", supportsTasks: true, warmTaskVersion: 1, sessionHandoffVersion: 1,
|
||||
}] });
|
||||
const issueId = randomUUID(), sessionId = randomUUID(), nextRunId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId: seeded.companyId, title: "Portable session", status: "in_progress" });
|
||||
await db.insert(heartbeatRuns).values({ id: nextRunId, companyId: seeded.companyId, agentId: seeded.agentId, status: "running" });
|
||||
let acquisitions = 0;
|
||||
const call = vi.fn(async (_plugin: string, method: string) => {
|
||||
if (method === "environmentAcquireLease") return { providerLeaseId: `sandbox-${++acquisitions}`,
|
||||
metadata: { ownershipGeneration: `generation-${acquisitions}` } };
|
||||
if (method === "environmentResumeLease") return { providerLeaseId: "sandbox-1", metadata: {} };
|
||||
if (method === "environmentReleaseLease") return { providerLeaseId: "sandbox-1", state: "stopped" };
|
||||
throw new Error(`Unexpected plugin method: ${method}`);
|
||||
});
|
||||
const runtime = environmentRuntimeService(db, { pluginWorkerManager: {
|
||||
isRunning: () => true, call, getWorker: () => ({ supportedMethods: ["environmentResumeLease"] }),
|
||||
} as unknown as PluginWorkerManager });
|
||||
const input = { companyId: seeded.companyId, environment: seeded.environment, issueId, agentId: seeded.agentId,
|
||||
persistedExecutionWorkspace: null, adapterType: "paperclip_runner" };
|
||||
const orchestrator = environmentRunOrchestrator(db, { environmentRuntime: runtime });
|
||||
const acquisitionInput = { ...input, selectedEnvironmentId: seeded.environment.id, localEnvironmentId: randomUUID(),
|
||||
admittedLifecycleMode: "warm" as const, executionWorkspaceSettings: null };
|
||||
const first = await orchestrator.acquireForRun({ ...acquisitionInput, heartbeatRunId: seeded.runId });
|
||||
await db.update(environmentLeases).set({ status: "retained", metadata: {
|
||||
...first.lease.metadata, runnerTaskIdentity: { sessionId },
|
||||
...(mode === "legacy resume" ? {} : { runnerTaskAccountIdentity: "test-account" }),
|
||||
} }).where(eq(environmentLeases.id, first.lease.id));
|
||||
const nextInput = { ...acquisitionInput, heartbeatRunId: nextRunId, runnerTaskResumeSessionId: resumeSession ? sessionId : null };
|
||||
const next = await orchestrator.acquireForRun(nextInput);
|
||||
expect(next.lease.providerLeaseId).toBe(mode === "legacy resume" ? "sandbox-1" : "sandbox-2");
|
||||
expect(next.lease.metadata?.portableTaskPreviousLeaseId).toBe(mode === "handoff" ? first.lease.id : undefined);
|
||||
await expect(orchestrator.acquireForRun(nextInput)).resolves.toMatchObject({ lease: { id: next.lease.id, providerLeaseId: next.lease.providerLeaseId } });
|
||||
if (mode === "handoff") {
|
||||
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, seeded.agentId));
|
||||
await db.update(issues).set({ assigneeAgentId: seeded.agentId, executionRunId: nextRunId }).where(eq(issues.id, issueId));
|
||||
await db.update(heartbeatRuns).set({ scopeKind: "issue", issueId, nativeIssueId: issueId, runtimeMode: "native",
|
||||
nativeSessionId: sessionId, runnerInstanceId: randomUUID(), runnerProfileJson: {
|
||||
nativePreparationFencingVersion: NATIVE_PREPARATION_FENCING_VERSION,
|
||||
sessionCheckpoint: { identity: { companyId: seeded.companyId, agentId: seeded.agentId, issueId, runId: nextRunId, sessionId },
|
||||
providerSessionId: "inherited-provider-session" },
|
||||
} }).where(eq(heartbeatRuns.id, nextRunId));
|
||||
await db.insert(nativeRunFinalizations).values({ runId: nextRunId, companyId: seeded.companyId, issueId, phase: "observed" });
|
||||
try {
|
||||
const [claim] = await claimNativeRestartRecoveries({ db, restartKind: "hard", runIds: [nextRunId] });
|
||||
expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: nextRunId, portablePreparationLeaseId: next.lease.id });
|
||||
if (claim.kind !== "bootstrap_incomplete" || !claim.portablePreparationLeaseId) throw new Error("Expected portable bootstrap recovery");
|
||||
const preparation = { kind: "plugin_task" as const, leaseId: claim.portablePreparationLeaseId, preparation: true as const };
|
||||
await expect(orchestrator.acquireForRun({ ...nextInput, reattachRemoteLease: preparation }))
|
||||
.resolves.toMatchObject({ lease: { id: next.lease.id, providerLeaseId: "sandbox-2", heartbeatRunId: nextRunId } });
|
||||
await expect(orchestrator.acquireForRun({ ...nextInput, reattachRemoteLease: preparation, issueId: null }))
|
||||
.rejects.toThrow("field=lease_scope");
|
||||
await expect(orchestrator.acquireForRun({ ...nextInput, reattachRemoteLease: preparation,
|
||||
persistedExecutionWorkspace: { id: randomUUID(), mode: "isolated_workspace" } })).rejects.toThrow("field=lease_scope");
|
||||
const otherEnvironmentId = randomUUID();
|
||||
await db.insert(environments).values({ id: otherEnvironmentId, name: otherEnvironmentId, driver: "plugin",
|
||||
status: "active", config: seeded.environment.config });
|
||||
await expect(orchestrator.acquireForRun({ ...nextInput, reattachRemoteLease: preparation,
|
||||
selectedEnvironmentId: otherEnvironmentId })).rejects.toThrow("field=lease_scope");
|
||||
} finally {
|
||||
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, nextRunId));
|
||||
}
|
||||
}
|
||||
await expect(orchestrator.acquireForRun({ ...nextInput, issueId: null })).rejects.toThrow("Active plugin task placement changed");
|
||||
expect(call.mock.calls.map(([, method]) => method)).toEqual({
|
||||
handoff: ["environmentAcquireLease", "environmentAcquireLease"],
|
||||
fresh: ["environmentAcquireLease", "environmentReleaseLease", "environmentAcquireLease"],
|
||||
"legacy resume": ["environmentAcquireLease", "environmentResumeLease"],
|
||||
}[mode]);
|
||||
});
|
||||
|
||||
it.each(["same task", "different task", "different agent", "no task"])(
|
||||
"scopes projectless reusable leases to their task and agent: %s", async (scenario) => {
|
||||
const seeded = await seedReusablePluginSandboxLease();
|
||||
|
||||
@@ -241,19 +241,33 @@ databaseSuite("environment task lease authority", () => {
|
||||
await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
let sessionStateAvailable = false;
|
||||
const operations: string[] = [];
|
||||
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }),
|
||||
call: async (_id: string, _method: string, params: any) => ["read_state", "archive_state"].includes(params.operation.kind)
|
||||
const call = vi.fn(async (_id: string, _method: string, params: any) => ["read_state", "archive_state"].includes(params.operation.kind)
|
||||
? { kind: "state", taskId: params.taskId, state: { schema: "paperclip.runner.durable.state.v1", lifecycle: "suspended",
|
||||
...(params.operation.identity ?? params.operation.priorIdentity) } }
|
||||
: params.operation.kind === "status"
|
||||
? (operations.push("status"), { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true, sessionStateAvailable })
|
||||
: { kind: "accepted", taskId: params.taskId },
|
||||
} as unknown as PluginWorkerManager;
|
||||
: { kind: "accepted", taskId: params.taskId });
|
||||
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call } as unknown as PluginWorkerManager;
|
||||
const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
|
||||
const runner = { runnerId: "runner", leaseId: oldLeaseId, sessionId: "session", runId: nextRunId,
|
||||
turnId: "turn", itemId: "item", harness: "codex", protocolMin: 1, protocolMax: 3 };
|
||||
const checkpointIdentity = { ...priorIdentity, turnId: "prior-turn", itemId: "prior-item" };
|
||||
const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, oldLeaseId));
|
||||
const portableTargetLeaseId = randomUUID();
|
||||
await db.update(environmentLeases).set({ metadata: { ...source.metadata, portableTaskSuccessor: {
|
||||
sourceLeaseId: oldLeaseId, targetLeaseId: portableTargetLeaseId,
|
||||
operation: { kind: "ensure_ownership", operationId: portableTargetLeaseId, sourceTaskId: "old-task",
|
||||
sourceGeneration: "source-generation", targetGeneration: "target-generation", policy: "ordinary", identity: checkpointIdentity },
|
||||
} } }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
await expect(authorize()).rejects.toThrow("not authorized");
|
||||
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).rejects.toThrow("not authorized");
|
||||
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
|
||||
kind: "read_state", identity: checkpointIdentity, resumeFrom: "old-task",
|
||||
} })).rejects.toThrow("not authorized");
|
||||
await expect(runtime.task({ companyId, leaseId: nextLeaseId, operation: {
|
||||
kind: "submit", preparedLaunchId: "fixture", projectIds: [], bootstrapTicket: "fixture", runner, resumeFrom: "old-task",
|
||||
} })).rejects.toThrow("not authorized");
|
||||
expect(call).not.toHaveBeenCalled();
|
||||
await db.update(environmentLeases).set({ metadata: { ...source.metadata, runnerTaskSessionRetained: false } }).where(eq(environmentLeases.id, oldLeaseId));
|
||||
await expect(authorize()).rejects.toThrow("not authorized");
|
||||
await expect(resolvePluginTaskResume(db, { companyId, leaseId: nextLeaseId, priorIdentity, inspectCheckpoint: true })).resolves.toBe("old-task");
|
||||
|
||||
@@ -7,6 +7,7 @@ import { createSystemRecovery } from "../services/system-recovery-state.js";
|
||||
import * as conversationContinuation from "../services/conversation-continuation.js";
|
||||
import { classifyEnvironmentCapabilities, environmentRuntimeService } from "../services/environment-runtime.js";
|
||||
import * as workspaceRuntime from "../services/workspace-runtime.js";
|
||||
import * as workspacePreparation from "../services/heartbeat/workspace-preparation.js";
|
||||
import { nativeWorkspaceSyncInternals } from "../services/native-runtime/native-workspace-sync.js";
|
||||
import { ensureNativeCompletionContract } from "../services/native-runtime/completion-contracts.js";
|
||||
import { readNativePlanWait, hasCommittedNativePlanWait } from "../services/native-runtime/native-plan-wait.js";
|
||||
@@ -5278,6 +5279,52 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it("hands off the same portable task lease during in-flight workspace preparation on graceful shutdown", async () => {
|
||||
const f = await seedRunFixture({ adapterType: "paperclip_runner", agentStatus: "idle", runStatus: "queued", runtimeMode: "legacy" });
|
||||
await db.update(heartbeatRuns).set({ scopeKind: "issue", issueId: f.issueId }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId));
|
||||
const { leaseId } = await seedEnvironmentLeaseFixture({ ...f, driver: "plugin", provider: "plugin:fixture:tasks" });
|
||||
await db.update(environmentLeases).set({ providerLeaseId: "same-portable-task", metadata: {
|
||||
driver: "plugin", agentId: f.agentId, runnerTaskLease: true, portableTaskPreviousLeaseId: randomUUID(),
|
||||
} }).where(eq(environmentLeases.id, leaseId));
|
||||
let finishPreparation!: () => void;
|
||||
const providerReply = new Promise<void>(resolve => { finishPreparation = resolve; });
|
||||
const reachedPreparation = vi.fn();
|
||||
const preparation = vi.spyOn(workspacePreparation, "prepareHeartbeatWorkspace").mockImplementationOnce(async (_db, input) => {
|
||||
await input.controllerLease.assertOwned();
|
||||
reachedPreparation();
|
||||
await providerReply;
|
||||
await input.controllerLease.assertOwned();
|
||||
throw new Error("Old controller unexpectedly kept preparation authority");
|
||||
});
|
||||
const runtime = environmentRuntimeService(db);
|
||||
const release = vi.spyOn(runtime, "releaseRunLeases");
|
||||
const heartbeat = heartbeatService(db, { environmentRuntime: runtime });
|
||||
try {
|
||||
await heartbeat.resumeQueuedRunsForAgent(f.companyId, f.agentId);
|
||||
await vi.waitFor(() => expect(reachedPreparation).toHaveBeenCalledOnce(), { timeout: 10_000 });
|
||||
expect(await heartbeat.drainRunningRunsForShutdown("SIGTERM", new Date(), [f.runId])).toMatchObject({
|
||||
restartSuspendedRunIds: [f.runId], interruptedRunIds: [], retryRunIds: [],
|
||||
});
|
||||
finishPreparation();
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
expect(run).toMatchObject({ status: "running", runtimeMode: "legacy", executionStage: "preparing" });
|
||||
expect(run.controllerLeaseExpiresAt!.getTime()).toBeLessThanOrEqual(Date.now());
|
||||
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)))[0])
|
||||
.toMatchObject({ heartbeatRunId: f.runId, status: "active", providerLeaseId: "same-portable-task" });
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(f.runId);
|
||||
expect(release).not.toHaveBeenCalled();
|
||||
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
||||
expect(mockExecutePaperclipNativeSession).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
finishPreparation();
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
preparation.mockRestore();
|
||||
release.mockRestore();
|
||||
}
|
||||
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
||||
|
||||
it.each([
|
||||
{ name: "still preparing", resumes: true },
|
||||
{ name: "admitted to dispatch", stage: "dispatching", resumes: false },
|
||||
|
||||
@@ -83,6 +83,20 @@ suite("durable plugin task credential handoff", () => {
|
||||
failExport(value: boolean) { failExport = value; }, credential: () => service.credential(selection!),
|
||||
settle: (executionStopObserved = false) => settleDispatchedEnvironmentTask(db, workers, { companyId, id: leaseId }, { executionStopObserved }) };
|
||||
}
|
||||
it("completes a moved task's saved credential handoff after confirming stop", async () => {
|
||||
const f = await fixture();
|
||||
await db.update(environmentLeases).set({ metadata:
|
||||
sql`${environmentLeases.metadata} || '{"portableTaskSuccessor":{"targetLeaseId":"replacement"}}'::jsonb`,
|
||||
}).where(eq(environmentLeases.id, f.leaseId));
|
||||
await expect(f.synchronize()).rejects.toThrow("session ownership unavailable");
|
||||
expect(f.calls).toEqual([]);
|
||||
await expect(f.settle()).resolves.toBe("completed");
|
||||
expect(f.calls).toEqual(["stop", "status", "credentials", "complete"]);
|
||||
expect(await f.credential()).toBe(auth("fixture-refreshed", 2021));
|
||||
await expect(f.synchronize()).resolves.toBeUndefined();
|
||||
expect(f.calls).toHaveLength(4);
|
||||
});
|
||||
|
||||
it.each(["codex", "grok"] as const)("writes managed %s refreshes without any plugin or invocation home", async provider => {
|
||||
const f = await fixture(true, provider);
|
||||
const read = vi.fn(async () => f.accountCredential("fixture-refreshed", 2021));
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { AbortMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, S3Client } from "@aws-sdk/client-s3";
|
||||
import { Readable } from "node:stream";
|
||||
import { setImmediate as nextEventLoopTurn } from "node:timers/promises";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createServer } from "node:http";
|
||||
import type { AddressInfo } from "node:net";
|
||||
@@ -10,6 +11,7 @@ import { describe, expect, it, vi } from "vitest";
|
||||
import { FILE_PART_CONCURRENCY, THROTTLED_PART_ATTEMPTS, putS3MultipartFile } from "../storage/s3-multipart.js";
|
||||
import { createS3StorageProvider } from "../storage/s3-provider.js";
|
||||
import { createStorageService } from "../storage/service.js";
|
||||
import * as fileCrypto from "../storage/file-crypto-pool.js";
|
||||
|
||||
const partBytes = 8 * 1024 * 1024;
|
||||
|
||||
@@ -89,17 +91,24 @@ describe("file-backed S3 multipart uploads under a request-rate limit", () => {
|
||||
const client = new S3Client({ region: "us-east-1" });
|
||||
const attempts = new Map<number, number>();
|
||||
let laterActive = 0, laterPeak = 0;
|
||||
let firstRoundRetries = 0;
|
||||
const firstRoundRetried = Promise.withResolvers<void>();
|
||||
// Keep disk I/O out of the concurrency measurement. The real-client test checks MD5.
|
||||
using digest = vi.spyOn(fileCrypto, "digestFileOffThread").mockResolvedValue({ hex: "0".repeat(32), bytes: partBytes });
|
||||
const random = vi.spyOn(Math, "random").mockReturnValue(0);
|
||||
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
||||
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
||||
if (command instanceof UploadPartCommand) {
|
||||
const number = command.input.PartNumber!;
|
||||
attempts.set(number, (attempts.get(number) ?? 0) + 1);
|
||||
await drain(command.input.Body);
|
||||
if (number <= FILE_PART_CONCURRENCY && attempts.get(number) === 1) throw slowDown();
|
||||
if (number > FILE_PART_CONCURRENCY) {
|
||||
if (number <= FILE_PART_CONCURRENCY) {
|
||||
if (attempts.get(number) === 1) throw slowDown();
|
||||
// Finish the already-dispatched retries together so every eligible slot can take new work.
|
||||
if (++firstRoundRetries === FILE_PART_CONCURRENCY) firstRoundRetried.resolve();
|
||||
await firstRoundRetried.promise;
|
||||
} else {
|
||||
laterActive++; laterPeak = Math.max(laterPeak, laterActive);
|
||||
await new Promise(resolve => setTimeout(resolve, 20));
|
||||
await nextEventLoopTurn();
|
||||
laterActive--;
|
||||
}
|
||||
return { ETag: `part-${number}` };
|
||||
|
||||
@@ -237,6 +237,7 @@ export async function resolveEnvironmentExecutionTarget(input: {
|
||||
leaseMetadata: Record<string, unknown> | null;
|
||||
lease?: EnvironmentLease | null;
|
||||
environmentRuntime?: EnvironmentRuntimeService | null;
|
||||
signal?: AbortSignal;
|
||||
// The startup tracer for the provider-exec span. Defaults to the endpoint-
|
||||
// gated server tracer, which is a no-op when tracing is off. Tests inject a
|
||||
// recording tracer.
|
||||
@@ -278,6 +279,7 @@ export async function resolveEnvironmentExecutionTarget(input: {
|
||||
const result = await runtime.task({ companyId: input.companyId, leaseId: lease.id,
|
||||
operation: environmentTaskOperationSchema.parse(operation),
|
||||
...(context?.executionContext ? { executionContext: context.executionContext } : {}),
|
||||
...(input.signal ? { ownershipSignal: input.signal } : {}),
|
||||
...(context?.expectedTaskId !== undefined ? { expectedTaskId: context.expectedTaskId } : {}) });
|
||||
if (result.kind === "description" && result.taskId !== acquisition.providerLeaseId) {
|
||||
acquisition.outcome = "replacement";
|
||||
|
||||
@@ -41,7 +41,7 @@ import {
|
||||
runnerLifecycleDefaultNeedsLeaseProbe,
|
||||
resolveWarmRunnerIdleTimeoutMsForRun,
|
||||
} from "./runner-environment-lifecycle.js";
|
||||
import { assertEnvironmentCompanyBinding, environmentService } from "./environments.js";
|
||||
import { assertEnvironmentCompanyBinding, environmentService, toEnvironmentLease } from "./environments.js";
|
||||
import {
|
||||
environmentRuntimeService,
|
||||
buildEnvironmentLeaseContext,
|
||||
@@ -50,6 +50,7 @@ import {
|
||||
type ProviderResourceDisposition,
|
||||
} from "./environment-runtime.js";
|
||||
import { ENVIRONMENT_DRIVER_TRAITS } from "./environment-driver-traits.js";
|
||||
import { unlaunchedPortableTaskLease } from "./plugin-task-ownership.js";
|
||||
import {
|
||||
resolveEnvironmentExecutionTarget,
|
||||
resolveEnvironmentExecutionTransport,
|
||||
@@ -308,9 +309,10 @@ export function environmentRunOrchestrator(
|
||||
persistedExecutionWorkspace: Pick<ExecutionWorkspace, "id" | "mode"> | null;
|
||||
executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
|
||||
runnerTaskProjectIds?: readonly string[];
|
||||
runnerTaskResumeSessionId?: string | null;
|
||||
/** Existing live runner recovery must use its original active lease, including ephemeral leases. */
|
||||
reattachRemoteLease?: { kind?: "sandbox"; leaseId: string; providerLeaseId: string; remoteCwd: string }
|
||||
| { kind: "plugin_task"; leaseId: string };
|
||||
| { kind: "plugin_task"; leaseId: string; preparation?: true };
|
||||
/** A new controller must stop the previous plugin task before allocating a replacement. */
|
||||
recoverPluginTask?: boolean;
|
||||
}): Promise<EnvironmentAcquisitionResult> {
|
||||
@@ -365,16 +367,22 @@ export function environmentRunOrchestrator(
|
||||
persistedExecutionWorkspace: input.persistedExecutionWorkspace,
|
||||
executionWorkspaceSettings: input.executionWorkspaceSettings,
|
||||
runnerTaskProjectIds: input.runnerTaskProjectIds,
|
||||
runnerTaskResumeSessionId: input.runnerTaskResumeSessionId,
|
||||
adapterType: input.adapterType ?? null,
|
||||
warmRunnerIdleTimeoutMs,
|
||||
runnerReconnectGraceMs,
|
||||
};
|
||||
let leaseRecord: EnvironmentRuntimeLeaseRecord;
|
||||
if (input.reattachRemoteLease) {
|
||||
const expected = input.reattachRemoteLease;
|
||||
if (expected) {
|
||||
// Reattachment keeps the original lease, including ephemeral leases.
|
||||
// Never reacquire: acquisition can replace a non-reusable sandbox.
|
||||
const expected = input.reattachRemoteLease;
|
||||
const lease = await environmentsSvc.getLeaseById(expected.leaseId);
|
||||
const replayPreparation = expected.kind === "plugin_task" && expected.preparation;
|
||||
const lease = replayPreparation
|
||||
? await db.select().from(environmentLeases).where(and(eq(environmentLeases.id, expected.leaseId),
|
||||
eq(environmentLeases.companyId, input.companyId), unlaunchedPortableTaskLease()))
|
||||
.then(([row]) => row ? toEnvironmentLease(row) : null)
|
||||
: await environmentsSvc.getLeaseById(expected.leaseId);
|
||||
const mismatch: RemoteRecoveryLeaseMismatchField | null = !lease ? "lease"
|
||||
: lease.companyId !== input.companyId || lease.environmentId !== environment.id ||
|
||||
lease.heartbeatRunId !== input.heartbeatRunId || lease.issueId !== input.issueId ||
|
||||
@@ -385,6 +393,8 @@ export function environmentRunOrchestrator(
|
||||
: expected.kind === "plugin_task"
|
||||
? environment.driver !== "plugin" ? "environment_driver"
|
||||
: input.adapterType !== "paperclip_runner" ? "adapter_type"
|
||||
: replayPreparation ? environment.config.pluginKey !== lease.metadata?.pluginKey ||
|
||||
environment.config.driverKey !== lease.metadata?.driverKey ? "plugin_task" : null
|
||||
: lease.metadata?.runnerTaskLease !== true || lease.metadata?.runnerTaskDispatched !== true
|
||||
|| lease.metadata?.runnerTaskExecutionStopped === true ? "plugin_task" : null
|
||||
: environment.driver !== "sandbox" ? "environment_driver"
|
||||
@@ -483,6 +493,7 @@ export function environmentRunOrchestrator(
|
||||
effectiveExecutionWorkspaceMode: string | null;
|
||||
persistedExecutionWorkspace: ExecutionWorkspace | null;
|
||||
runnerTaskProjectIds?: readonly string[];
|
||||
signal?: AbortSignal;
|
||||
assertOwned?: () => Promise<void>;
|
||||
/**
|
||||
* The host duplex observability recorder for this run. The orchestrator threads
|
||||
@@ -719,6 +730,7 @@ export function environmentRunOrchestrator(
|
||||
leaseMetadata: (lease.metadata as Record<string, unknown> | null) ?? null,
|
||||
lease,
|
||||
environmentRuntime,
|
||||
signal: input.signal,
|
||||
duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
|
||||
});
|
||||
// The native plugin task bridge owns guest paths. Never fall back to host execution.
|
||||
|
||||
@@ -20,7 +20,6 @@ import { createOwnedSandbox, recoverSandboxCreations, recoverablePluginCreation
|
||||
import { dueRunLeaseReleaseCondition, pendingRunLeaseReleaseCondition } from "./environment-lease-release-state.js";
|
||||
import { flushDeferredEnvironmentCleanup, registerDeferredEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { executeEnvironmentTask, pluginEnvironmentSupportsTasks, settleDispatchedEnvironmentTask } from "./environment-task-runtime.js";
|
||||
import type { PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
|
||||
import { allowLegacyShutdownWorkspaceCleanup, legacyShutdownWorkspaceResourceProtected } from "./legacy-shutdown-workspace-settlement.js";
|
||||
import { nativeWorkspaceExportPendingForLease, nativeWorkspaceExportPendingForResource } from "./native-runtime/native-workspace-finalization-state.js";
|
||||
import { beginIdleTrackedWork } from "./task-admission.js";
|
||||
@@ -484,6 +483,7 @@ export interface EnvironmentDriverAcquireInput {
|
||||
executionWorkspaceMode: ExecutionWorkspace["mode"] | null;
|
||||
executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
|
||||
runnerTaskProjectIds?: readonly string[];
|
||||
runnerTaskResumeSessionId?: string | null;
|
||||
/**
|
||||
* The harness/adapter type for this run (the agent's adapter). Drivers that
|
||||
* materialize a per-run sandbox use it to select the runtime image so a single
|
||||
@@ -3516,6 +3516,23 @@ function createPluginEnvironmentDriver(
|
||||
workerManager.getWorker(plugin.id)?.supportedMethods.includes("environmentResumeLease");
|
||||
const lifecyclePolicy = warm ? { mode: "warm", idleTimeoutMs: parsed.config.runnerIdleTimeoutMs ?? 300_000 } : { mode: "per_turn", idleTimeoutMs: null };
|
||||
const fingerprint = createHash("sha256").update(JSON.stringify([parsed.config, input.agentId, input.runnerTaskProjectIds ?? [], ...(input.purpose ? [input.purpose, input.loginCommandKey] : [])])).digest("hex");
|
||||
if (driver.sessionHandoffVersion === 1 && input.adapterType === "paperclip_runner" && input.heartbeatRunId && input.purpose !== "login") {
|
||||
// The run lease already records the placement to replay after a lost preparation reply.
|
||||
const active = await db.select().from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.environmentId, input.environment.id),
|
||||
eq(environmentLeases.heartbeatRunId, input.heartbeatRunId), eq(environmentLeases.status, "active"),
|
||||
)).limit(2);
|
||||
if (active.length > 0) {
|
||||
const lease = active[0]!;
|
||||
if (active.length !== 1 || !lease.providerLeaseId || lease.releasedAt || lease.cleanupStatus ||
|
||||
lease.expiresAt && lease.expiresAt.getTime() <= Date.now() ||
|
||||
lease.issueId !== input.issueId || lease.executionWorkspaceId !== (input.executionWorkspaceId ?? null) ||
|
||||
lease.metadata?.runnerTaskLease !== true || lease.metadata?.runnerTaskRetiring === true ||
|
||||
lease.metadata?.pluginId !== plugin.id || lease.metadata?.driverKey !== parsed.config.driverKey ||
|
||||
lease.metadata?.runnerTaskFingerprint !== fingerprint) throw new Error("Active plugin task placement changed");
|
||||
return toEnvironmentLeaseSnapshot(lease);
|
||||
}
|
||||
}
|
||||
const candidates = warm ? await db.select().from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.environmentId, input.environment.id),
|
||||
eq(environmentLeases.status, "retained"), eq(environmentLeases.issueId, input.issueId!),
|
||||
@@ -3523,14 +3540,25 @@ function createPluginEnvironmentDriver(
|
||||
sql`${environmentLeases.metadata}->>'agentId' = ${input.agentId!}`,
|
||||
sql`${environmentLeases.metadata}->>'pluginId' = ${plugin.id}`,
|
||||
sql`${environmentLeases.metadata}->>'driverKey' = ${parsed.config.driverKey}`,
|
||||
sql`${environmentLeases.metadata}->'portableTaskSuccessor' is null`,
|
||||
)) : [];
|
||||
if (candidates.length > 1) throw new Error("Ambiguous retained plugin task ownership");
|
||||
const previous = candidates[0];
|
||||
const previousSessionId = (previous?.metadata?.runnerTaskIdentity as { sessionId?: unknown } | undefined)?.sessionId;
|
||||
const previousAccountIdentity = previous?.metadata?.runnerTaskAccountIdentity;
|
||||
const freshPortableSession = driver.sessionHandoffVersion === 1 && input.runnerTaskResumeSessionId !== undefined &&
|
||||
(input.runnerTaskResumeSessionId === null || input.runnerTaskResumeSessionId !== previousSessionId);
|
||||
const handoffPrevious = !freshPortableSession && driver.sessionHandoffVersion === 1 && previous?.providerLeaseId &&
|
||||
previous.metadata?.runnerTaskFingerprint === fingerprint &&
|
||||
typeof previousAccountIdentity === "string" && previousAccountIdentity.trim().length > 0 &&
|
||||
typeof (previous.metadata?.providerMetadata as { ownershipGeneration?: unknown } | undefined)?.ownershipGeneration === "string"
|
||||
? previous : undefined;
|
||||
let resumed: PluginEnvironmentLease | undefined;
|
||||
if (previous?.providerLeaseId && previous.metadata?.runnerTaskFingerprint !== fingerprint) {
|
||||
// A changed mount scope must retire the old process before checkpoint recovery.
|
||||
if (!handoffPrevious && previous?.providerLeaseId &&
|
||||
(freshPortableSession || previous.metadata?.runnerTaskFingerprint !== fingerprint)) {
|
||||
// A fresh logical session or changed mount scope must retire the old process before provisioning.
|
||||
await this.releaseRunLease({environment:input.environment,lease:toEnvironmentLeaseSnapshot(previous),status:"released"});
|
||||
} else if (previous?.providerLeaseId) {
|
||||
} else if (!handoffPrevious && previous?.providerLeaseId) {
|
||||
// Resume only inspects the exact computer; an unavailable reply must not create a replacement.
|
||||
resumed = await workerManager.call(plugin.id, "environmentResumeLease", {
|
||||
driverKey: parsed.config.driverKey, companyId: input.companyId, environmentId: input.environment.id,
|
||||
@@ -3589,6 +3617,7 @@ function createPluginEnvironmentDriver(
|
||||
...(warm ? {runnerTaskProjectIds:[...input.runnerTaskProjectIds ?? []]} : {}),remoteCwd:".",
|
||||
...(input.agentId ? { agentId: input.agentId } : {}),
|
||||
providerMetadata: providerLease.metadata ?? {},
|
||||
...(handoffPrevious ? { portableTaskPreviousLeaseId: handoffPrevious.id } : {}),
|
||||
...(input.purpose ? { purpose: input.purpose, loginCommandKey: input.loginCommandKey } : {}),
|
||||
environmentId: input.environment.id,
|
||||
...(input.purpose !== "login" && input.adapterType === "paperclip_runner" && driver.supportsTasks === true ? { runnerTaskLease: true } : {}),
|
||||
@@ -3996,7 +4025,7 @@ export function environmentRuntimeService(
|
||||
},
|
||||
|
||||
/** Dispatch plugin-provided Runner execution, including resource preparation and task lifecycle. */
|
||||
async task(input: { companyId: string; leaseId: string; operation: PluginEnvironmentTaskOperation; executionContext?: unknown; expectedTaskId?: string }) {
|
||||
async task(input: Parameters<typeof executeEnvironmentTask>[2]) {
|
||||
if (!options.pluginWorkerManager) throw new Error("Plugin-provided Runner execution worker manager unavailable");
|
||||
return executeEnvironmentTask(db, options.pluginWorkerManager, input);
|
||||
},
|
||||
@@ -4015,6 +4044,7 @@ export function environmentRuntimeService(
|
||||
persistedExecutionWorkspace: Pick<ExecutionWorkspace, "id" | "mode"> | null;
|
||||
executionWorkspaceSettings?: IssueExecutionWorkspaceSettings | null;
|
||||
runnerTaskProjectIds?: readonly string[];
|
||||
runnerTaskResumeSessionId?: string | null;
|
||||
/** The agent's adapter type for this run (mixed-harness environments). */
|
||||
adapterType?: string | null;
|
||||
resolveSnapshotIdentity?: boolean;
|
||||
@@ -4060,6 +4090,7 @@ export function environmentRuntimeService(
|
||||
executionWorkspaceMode: leaseContext.executionWorkspaceMode,
|
||||
executionWorkspaceSettings: input.executionWorkspaceSettings ?? null,
|
||||
runnerTaskProjectIds: input.runnerTaskProjectIds,
|
||||
runnerTaskResumeSessionId: input.runnerTaskResumeSessionId,
|
||||
purpose: input.purpose,
|
||||
loginCommandKey: input.loginCommandKey,
|
||||
adapterType: input.adapterType ?? null,
|
||||
|
||||
@@ -5,19 +5,22 @@ import { createOwnedSandbox, queueSandboxCreationCleanup, recordSandboxCreationR
|
||||
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { releaseProviderRecoveryTransfers, stageProviderFreshWorkspace } from "./provider-fresh-workspace.js";
|
||||
import { readWorkspaceProviderBinding } from "@paperclipai/shared";
|
||||
import { settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js";
|
||||
import { canReadStoppedPluginTaskCredential, settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js";
|
||||
import { and, eq, inArray, sql } from "drizzle-orm";
|
||||
import { environmentLeases, executionWorkspaces, environments, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db";
|
||||
import { freshWorkspaceSchema, environmentTaskOperationSchema, parseEnvironmentTaskResult,
|
||||
environmentTaskErrorData, readEnvironmentTaskErrorDiagnostic, JsonRpcCallError, PLUGIN_RPC_ERROR_CODES,
|
||||
environmentTaskErrorCodeData, PluginEnvironmentTaskError, readEnvironmentTaskErrorCode,
|
||||
type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
|
||||
type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation, type PluginEnvironmentTaskResult } from "@paperclipai/plugin-sdk";
|
||||
import { pluginRegistryService } from "./plugin-registry.js";
|
||||
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
|
||||
import { NativeExecutionContext } from "./native-runtime/native-execution-context.js";
|
||||
import { assertPluginTaskSourceAttachment, assertPluginTaskSourceRecovery } from "./native-runtime/plugin-task-source-recovery.js";
|
||||
import { pluginAgentRuntimeIdentity } from "./plugin-agent-runtime-state.js";
|
||||
import { assertPluginTaskOwnershipDispatch, PluginTaskOwnershipPendingError, readyPortableTaskSource, reconcilePluginTaskAdmission } from "./plugin-task-ownership.js";
|
||||
|
||||
export type PluginTaskDescription = Extract<PluginEnvironmentTaskResult, { kind: "description" }>;
|
||||
|
||||
export function isSettledReadyRunnerState(state: Record<string, unknown>): boolean {
|
||||
const commands = state.processedCommands;
|
||||
@@ -48,14 +51,15 @@ export async function pluginEnvironmentSupportsTasks(db: Db, workers: PluginWork
|
||||
* never a plugin identity, resource endpoint, or agent/project authority.
|
||||
* A timeout is ambiguous: retry the same operation against the same lease.
|
||||
*/
|
||||
export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManager, input: {
|
||||
export async function executeEnvironmentTask(db: Db, workers: Pick<PluginWorkerManager, "getWorker" | "call">, input: {
|
||||
companyId: string;
|
||||
leaseId: string;
|
||||
operation: PluginEnvironmentTaskOperation;
|
||||
executionContext?: unknown;
|
||||
ownershipSignal?: AbortSignal;
|
||||
/** Physical task captured by an existing owner, independent of the mutable lease. */
|
||||
expectedTaskId?: string;
|
||||
}) {
|
||||
}): Promise<PluginEnvironmentTaskResult> {
|
||||
const operation = environmentTaskOperationSchema.parse(input.operation);
|
||||
const [row] = await db.select({ lease: environmentLeases, environment: environments, run: heartbeatRuns })
|
||||
.from(environmentLeases)
|
||||
@@ -74,6 +78,27 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
const taskId = row.lease.providerLeaseId;
|
||||
if (input.expectedTaskId !== undefined && input.expectedTaskId !== taskId) throw new Error("Plugin Runner physical task changed");
|
||||
const metadata = lease.metadata ?? {};
|
||||
const ownership = operation.kind === "ensure_ownership";
|
||||
if (operation.kind === "describe" && metadata.portableTaskOwnershipReady !== true && metadata.portableTaskSuccessor === undefined &&
|
||||
(metadata.portableTaskPredecessor !== undefined || typeof metadata.portableTaskPreviousLeaseId === "string")) {
|
||||
await reconcilePluginTaskAdmission(db, lease, request =>
|
||||
executeEnvironmentTask(db, workers, { ...input, operation: request }), context?.signal ?? input.ownershipSignal);
|
||||
return executeEnvironmentTask(db, workers, input);
|
||||
}
|
||||
const requiresOwnership = !["ensure_ownership", "status", "stop", "complete"].includes(operation.kind);
|
||||
const stoppedCredentialHandoff = operation.kind === "credentials" && input.expectedTaskId === taskId &&
|
||||
canReadStoppedPluginTaskCredential(metadata, operation.accountIdentity);
|
||||
if (requiresOwnership && (metadata.portableTaskSuccessor !== undefined && !stoppedCredentialHandoff ||
|
||||
(metadata.portableTaskPredecessor !== undefined || metadata.portableTaskPreviousLeaseId !== undefined) && metadata.portableTaskOwnershipReady !== true)) {
|
||||
throw new Error("Plugin Runner session ownership unavailable");
|
||||
}
|
||||
if (ownership) {
|
||||
if (!environment || lease.status !== "active" || lease.releasedAt || run?.status !== "running" ||
|
||||
metadata.runnerTaskRetiring === true || lease.expiresAt && lease.expiresAt.getTime() <= Date.now()) {
|
||||
throw new Error("Plugin-provided Runner execution lease is not active");
|
||||
}
|
||||
assertPluginTaskOwnershipDispatch(lease, operation);
|
||||
}
|
||||
// A different launch identity is authorized by the saved consumer claim, never by the mismatch alone.
|
||||
const sourceRecovery = run && ((operation.kind === "prepare" && operation.requirements.identity.runId !== run.id && metadata.runnerTaskSourceRecovery) ||
|
||||
(operation.kind === "submit" && operation.runner.runId !== run.id && metadata.runnerTaskSourceRecovery))
|
||||
@@ -98,12 +123,21 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
const savedIdentity = metadata.runnerTaskIdentity as Record<string, unknown> | undefined;
|
||||
const prior = operation.kind === "prepare" ? operation.priorIdentity : undefined;
|
||||
const priorSaved = metadata.runnerTaskPriorIdentity as Record<string, unknown> | undefined;
|
||||
const firstPortableResume = operation.kind === "resume" && !savedIdentity &&
|
||||
metadata.portableTaskPredecessor !== undefined;
|
||||
if (firstPortableResume && (run?.status !== "running" ||
|
||||
!await readyPortableTaskSource(db, input.companyId, lease.id, operation.identity))) {
|
||||
throw new Error("Plugin Runner session ownership unavailable");
|
||||
}
|
||||
const matches = (identity: typeof prior, saved: Record<string, unknown> | undefined) => !!identity && !!saved &&
|
||||
identity.runnerInstanceId === saved.runnerId && identity.environmentLeaseId === saved.leaseId &&
|
||||
identity.normalizedSessionId === saved.sessionId && identity.runId === saved.runId &&
|
||||
identity.turnId === saved.turnId && identity.itemId === saved.itemId;
|
||||
const portableCheckpoint = operation.kind === "read_state" && !resumeFrom && !savedIdentity &&
|
||||
metadata.portableTaskPredecessor !== undefined
|
||||
? (await readyPortableTaskSource(db, input.companyId, lease.id, operation.identity))?.taskId : undefined;
|
||||
const ownCheckpoint = checkpointIdentity && !resumeFrom &&
|
||||
(matches(checkpointIdentity,savedIdentity) || operation.kind !== "read_file" && matches(checkpointIdentity,priorSaved));
|
||||
(portableCheckpoint || matches(checkpointIdentity,savedIdentity) || operation.kind !== "read_file" && matches(checkpointIdentity,priorSaved));
|
||||
if (checkpointIdentity && !resumeFrom && !ownCheckpoint) throw new Error("Plugin Runner state identity mismatch");
|
||||
const retainedCheckpointRead = operation.kind === "read_state" && ownCheckpoint &&
|
||||
lease.leasePolicy === "reuse_by_environment" && ["retained", "expired"].includes(lease.status) &&
|
||||
@@ -122,10 +156,21 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
const sourceAttachment = operation.kind === "prepare" && prior && sourceReference?.identity?.runId === prior.runId
|
||||
? await assertPluginTaskSourceAttachment(db, { companyId: input.companyId, leaseId: lease.id,
|
||||
context: context instanceof NativeExecutionContext ? context : undefined, operation }) : undefined;
|
||||
const warmAttachment = !!prior && (metadata.runnerTaskReused === true || !!sourceAttachment) &&
|
||||
(matches(prior, savedIdentity) || matches(prior, priorSaved) && operation.kind === "prepare" && matches(operation.requirements.identity, savedIdentity)) && runner?.runnerId === prior.runnerInstanceId &&
|
||||
const portableAttachment = prior && !sourceAttachment && metadata.runnerTaskReused !== true &&
|
||||
metadata.portableTaskPredecessor !== undefined
|
||||
? (await readyPortableTaskSource(db, input.companyId, lease.id, prior))?.taskId : undefined;
|
||||
const warmAttachment = !!prior && (metadata.runnerTaskReused === true || !!sourceAttachment || !!portableAttachment) &&
|
||||
(portableAttachment && !savedIdentity || matches(prior, savedIdentity) ||
|
||||
matches(prior, priorSaved) && operation.kind === "prepare" && matches(operation.requirements.identity, savedIdentity)) &&
|
||||
runner?.runnerId === prior.runnerInstanceId &&
|
||||
runner?.leaseId === prior.environmentLeaseId && runner?.sessionId === prior.normalizedSessionId;
|
||||
if (prior && !warmAttachment) throw new Error("Plugin warm task prior identity mismatch");
|
||||
const portableAuthority = firstPortableResume || portableAttachment
|
||||
? sql`${environmentLeases.metadata}->'portableTaskSuccessor' is null
|
||||
and ${environmentLeases.metadata}->'portableTaskOwnershipReady' = 'true'::jsonb
|
||||
and coalesce(${environmentLeases.metadata}->>'portableTaskCheckpointUnavailable', 'false') != 'true'
|
||||
and ${environmentLeases.metadata}->'portableTaskPredecessor' = ${JSON.stringify(metadata.portableTaskPredecessor)}::jsonb`
|
||||
: undefined;
|
||||
if (runner && !sourceRecovery && ((!checkpointIdentity || !resumeFrom) && !(ownCheckpoint && operation.kind !== "read_file") && runner.runId !== (activeProbe ? probe.runId : run?.id) || (!resumeFrom && runner.leaseId !== lease.id && !ownCheckpoint && !warmAttachment)
|
||||
|| (activeProbe && (runner.sessionId !== probe.sessionId || (launches && operation.projectIds.length > 0))))) {
|
||||
throw new Error("Plugin-provided Runner execution identity mismatch");
|
||||
@@ -168,10 +213,18 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
const plugin = await pluginRegistryService(db).getById(metadata.pluginId);
|
||||
const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === metadata.driverKey);
|
||||
if (!plugin || plugin.status !== "ready" || !driver?.supportsTasks ||
|
||||
!plugin.manifestJson.capabilities.includes("environment.drivers.register") ||
|
||||
!workers.getWorker(plugin.id)?.supportedMethods.includes("environmentTask")) {
|
||||
!plugin.manifestJson.capabilities.includes("environment.drivers.register")) {
|
||||
throw new Error("Plugin-provided Runner execution provider unavailable");
|
||||
}
|
||||
const worker = workers.getWorker(plugin.id);
|
||||
if (ownership && !worker && driver.sessionHandoffVersion === 1) throw new PluginTaskOwnershipPendingError();
|
||||
if (!worker?.supportedMethods.includes("environmentTask")) {
|
||||
throw new Error("Plugin-provided Runner execution provider unavailable");
|
||||
}
|
||||
if (operation.kind === "resume" && operation.handoffCredential && driver.sessionHandoffVersion !== 1) {
|
||||
throw new Error("Plugin Runner credential handoff unavailable");
|
||||
}
|
||||
if (ownership && driver.sessionHandoffVersion !== 1) throw new Error("Plugin Runner ownership reconciliation unavailable");
|
||||
const callWorker = async <M extends HostToWorkerMethodName>(method: M, params: HostToWorkerMethods[M][0], timeout?: number): Promise<HostToWorkerMethods[M][1]> => {
|
||||
if (!context) {
|
||||
const cleanup = metadata.runnerTaskStartupCleanup as { contextId?: unknown; requiresPreparationStop?: unknown } | undefined;
|
||||
@@ -255,13 +308,23 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
operation.launch.maxRuntimeMs = Math.min(remainingMs, ...requestedLimits);
|
||||
delete operation.launch.maxLifetimeMs;
|
||||
}
|
||||
if (operation.kind === "submit" || operation.kind === "copilot_metadata") {
|
||||
if (operation.kind === "submit" || operation.kind === "copilot_metadata" || firstPortableResume) {
|
||||
// No ticket, headers, environment values, or launch paths belong in metadata.
|
||||
const dispatched = {
|
||||
runnerTaskLease: true, runnerTaskDispatched: true,
|
||||
runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false,
|
||||
...(operation.kind === "submit" ? {
|
||||
runnerTaskIdentity: operation.runner,
|
||||
runnerTaskResumeFrom: operation.resumeFrom ?? null,
|
||||
runnerTaskAccountIdentity: operation.accountIdentity,
|
||||
} : {}),
|
||||
};
|
||||
const marked = await db.update(environmentLeases).set({
|
||||
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskAdmissionRejection') || ${JSON.stringify({ runnerTaskLease: true, runnerTaskDispatched: true, runnerTaskExecutionStopped: false, runnerTaskSessionRetained: false, ...(operation.kind === "submit" ? { runnerTaskIdentity: operation.runner, runnerTaskResumeFrom: operation.resumeFrom ?? null } : {}) })}::jsonb`,
|
||||
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'runnerTaskAdmissionRejection') || ${JSON.stringify(dispatched)}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId),
|
||||
eq(environmentLeases.status, "active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
|
||||
requiredProjectAuthProof ? sql`${environmentLeases.metadata}->'runnerTaskProjectAuthValidation' = ${JSON.stringify(requiredProjectAuthProof)}::jsonb` : undefined,
|
||||
firstPortableResume ? portableAuthority : undefined,
|
||||
)).returning({ id: environmentLeases.id });
|
||||
if (marked.length !== 1) throw new Error("Plugin-provided Runner execution lease is not active");
|
||||
}
|
||||
@@ -276,7 +339,9 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
...(startup ? { runnerTaskStartup: startup } : {}),
|
||||
})}::jsonb` }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.status, "active"),
|
||||
sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`,
|
||||
sql`${environmentLeases.metadata}->'runnerTaskIdentity' = ${JSON.stringify(savedIdentity)}::jsonb`)).returning({id:environmentLeases.id});
|
||||
portableAttachment ? portableAuthority : undefined,
|
||||
savedIdentity ? sql`${environmentLeases.metadata}->'runnerTaskIdentity' = ${JSON.stringify(savedIdentity)}::jsonb`
|
||||
: sql`${environmentLeases.metadata}->'runnerTaskIdentity' is null`)).returning({id:environmentLeases.id});
|
||||
if (marked.length !== 1) throw new Error("Plugin warm task attachment lost authority");
|
||||
}
|
||||
async function recordTaskStop(result: Extract<ReturnType<typeof parseEnvironmentTaskResult>, {kind:"status" | "copilot_metadata"}>) {
|
||||
@@ -322,6 +387,7 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
...(recoverableCreation ? { creationOperationVersion: 1, providerMetadata: metadata.providerMetadata } : {}) },
|
||||
}, async identity => {
|
||||
const params = { driverKey: replacementDriverKey, companyId: input.companyId, environmentId: environment!.id,
|
||||
executionWorkspaceId: lease.executionWorkspaceId ?? undefined,
|
||||
issueId: lease.issueId, config: driverConfig, runId: identity.runId, creationOperationId: identity.id,
|
||||
agentId: run!.agentId, adapterType: "paperclip_runner",
|
||||
...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? { requestedExpiresAt: metadata.runnerTaskRequestedExpiresAt } : {}) };
|
||||
@@ -414,8 +480,15 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
...(executionContext ? { executionContext } : {}),
|
||||
};
|
||||
const timeout = operation.kind === "copilot_metadata" ? Math.max(1, Math.min(40_000, Number(probe?.expiresAtMs) - Date.now()))
|
||||
: executionContext ? Math.max(1, context!.deadlineAtMs - Date.now()) : 15_000;
|
||||
const result = await callWorker("environmentTask", params, timeout);
|
||||
: executionContext ? Math.max(1, context!.deadlineAtMs - Date.now())
|
||||
: driver.sessionHandoffVersion === 1 && (ownership || operation.kind === "describe") ? 90_000 : 15_000;
|
||||
const result = await callWorker("environmentTask", params, timeout).catch(error => {
|
||||
// The plugin may have applied the saved claim before its reply was lost.
|
||||
// Keep the same operation ID; the RPC response below still needs validation.
|
||||
if (ownership && (!(error instanceof JsonRpcCallError) ||
|
||||
error.code === PLUGIN_RPC_ERROR_CODES.TIMEOUT || error.code === PLUGIN_RPC_ERROR_CODES.WORKER_UNAVAILABLE)) throw new PluginTaskOwnershipPendingError();
|
||||
throw error;
|
||||
});
|
||||
if (executionContext) context!.assertOpen();
|
||||
const parsed = parseEnvironmentTaskResult(operation, taskId, result);
|
||||
if (operation.kind === "submit" && parsed.kind === "admission_rejected") {
|
||||
@@ -443,6 +516,11 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
await assertPluginTaskSourceAttachment(db, { companyId: input.companyId, leaseId: lease.id,
|
||||
context: context instanceof NativeExecutionContext ? context : undefined, operation });
|
||||
}
|
||||
if (portableCheckpoint && operation.kind === "read_state" || portableAttachment && operation.kind === "prepare") {
|
||||
const identity = operation.kind === "read_state" ? operation.identity : operation.priorIdentity;
|
||||
const sourceTaskId = (await readyPortableTaskSource(db, input.companyId, lease.id, identity))?.taskId;
|
||||
if (sourceTaskId !== (portableCheckpoint || portableAttachment)) throw new Error("Plugin Runner session ownership changed");
|
||||
}
|
||||
if (operation.kind === "read_file") {
|
||||
const [current] = await db.select({ lease: environmentLeases, run: heartbeatRuns }).from(environmentLeases)
|
||||
.leftJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId), eq(heartbeatRuns.companyId, environmentLeases.companyId)))
|
||||
@@ -499,6 +577,7 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
}
|
||||
return parsed;
|
||||
} catch (error) {
|
||||
if (error instanceof PluginTaskOwnershipPendingError) throw error;
|
||||
const diagnostic = operation.kind === "submit" && operation.resumeFrom ? readEnvironmentTaskErrorDiagnostic(error) : null;
|
||||
if (diagnostic) throw new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.WORKER_ERROR,
|
||||
message: "Plugin Runner source state unavailable",
|
||||
|
||||
@@ -274,7 +274,7 @@ function resolveKubernetesConfig(
|
||||
return companyIdOrConfig;
|
||||
}
|
||||
|
||||
function toEnvironmentLease(row: EnvironmentLeaseRow): EnvironmentLease {
|
||||
export function toEnvironmentLease(row: EnvironmentLeaseRow): EnvironmentLease {
|
||||
return {
|
||||
id: row.id,
|
||||
companyId: row.companyId,
|
||||
@@ -1649,6 +1649,7 @@ export function environmentService(db: Db) {
|
||||
// lease holds older session state. That claim and this
|
||||
// handoff compete on this row, so only one of them wins.
|
||||
sql`not (coalesce(${environmentLeases.metadata}, '{}'::jsonb) ? 'reusableSandboxLeaseSuperseded')`,
|
||||
sql`not (coalesce(${environmentLeases.metadata}, '{}'::jsonb) ? 'portableTaskSuccessor')`,
|
||||
),
|
||||
)
|
||||
.returning();
|
||||
|
||||
@@ -1897,6 +1897,7 @@ export function heartbeatService(
|
||||
runOptions: {
|
||||
nativeLeaseOwner?: string;
|
||||
nativeRestartRecovery?: NativeRestartRecoveryClaim;
|
||||
portablePreparationLeaseId?: string;
|
||||
onNativeExecutionLeaseAcquired?: () => Promise<void>;
|
||||
} = {},
|
||||
) {
|
||||
@@ -1914,7 +1915,8 @@ export function heartbeatService(
|
||||
|
||||
async function executeRunAttempt(
|
||||
runId: string,
|
||||
runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; onNativeExecutionLeaseAcquired?: () => Promise<void> },
|
||||
runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; portablePreparationLeaseId?: string;
|
||||
onNativeExecutionLeaseAcquired?: () => Promise<void> },
|
||||
) {
|
||||
const attemptStartedAtMs = Date.now();
|
||||
let attestedQuestionResponseAtMs: number | null = null;
|
||||
@@ -2087,8 +2089,9 @@ export function heartbeatService(
|
||||
let dispatchGateEntered = false;
|
||||
const nativeDispatchMayHaveStarted = () => nativeDispatchStarted || dispatchGateEntered || nativeOwnershipHeld
|
||||
|| Boolean(runOptions.nativeRestartRecovery || runOptions.nativeLeaseOwner);
|
||||
const preparationHandedOff = () => shutdownHandedOff() && !legacyAdapterEntered && !nativeDispatchStarted;
|
||||
const leaveToNextController = async (error: unknown) => {
|
||||
if (!shutdownHandedOff() || legacyAdapterEntered || nativeDispatchStarted) return false;
|
||||
if (!preparationHandedOff()) return false;
|
||||
nativeSessionResumeScheduled = true;
|
||||
const reason = error instanceof Error ? error.message : String(error);
|
||||
const errorCode = /^[a-z][a-z0-9_]{0,80}$/.test(reason) ? reason : null;
|
||||
@@ -2895,7 +2898,7 @@ export function heartbeatService(
|
||||
executionTarget,
|
||||
workspaceRestoreSource,
|
||||
} = await timePreparation("workspace", () => prepareHeartbeatWorkspace(db, {
|
||||
run, agent, controllerLease, nativeRunnerPreparationSpans,
|
||||
run, agent, controllerLease, nativeRunnerPreparationSpans, signal: executionControl.controller.signal,
|
||||
task: {
|
||||
issueRef,
|
||||
issueId,
|
||||
@@ -2930,6 +2933,7 @@ export function heartbeatService(
|
||||
configSnapshot,
|
||||
secretManifest,
|
||||
managedAiCredentialIdentity: managedAiRuntime?.identity,
|
||||
runnerTaskResumeSessionId: run.nativeSessionId ?? preparedSession.taskResumeSessionId,
|
||||
},
|
||||
reuse: {
|
||||
requestedShouldReuseExisting,
|
||||
@@ -3295,7 +3299,7 @@ export function heartbeatService(
|
||||
context.projectId = executionWorkspace.projectId;
|
||||
}
|
||||
const {
|
||||
previousSessionDisplayId, runtimeSessionParamsForAdapter, taskSessionCredentialCompatible, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction,
|
||||
previousSessionDisplayId, runtimeSessionParamsForAdapter, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction,
|
||||
} = await resolveHeartbeatAdapterSession({
|
||||
agent,
|
||||
task: { context, taskKey, issueId, continuationSummary, executionContinuation },
|
||||
@@ -3507,9 +3511,7 @@ export function heartbeatService(
|
||||
session: {
|
||||
persistedNativeExecutionInput,
|
||||
persistedRunnerProfile,
|
||||
taskSessionCredentialCompatible,
|
||||
taskSessionDecodedParams,
|
||||
taskSessionForRun,
|
||||
taskResumeSessionId: preparedSession.taskResumeSessionId,
|
||||
isFailedChatRunRetry,
|
||||
getFreshSessionHandoff,
|
||||
},
|
||||
@@ -3585,6 +3587,8 @@ export function heartbeatService(
|
||||
adapter,
|
||||
runtimeForAdapter,
|
||||
getNativeFreshSessionHandoff,
|
||||
pluginTaskDescription: runtimeSelection.pluginTaskDescription,
|
||||
toolRefreshOnResume: runtimeSelection.toolRefreshOnResume,
|
||||
getFreshSessionHandoff,
|
||||
runOptions,
|
||||
},
|
||||
@@ -3999,6 +4003,7 @@ export function heartbeatService(
|
||||
await instructionWorkingCopy?.nativeInstructionReservation?.release().catch(error => logger.warn({ runId: run.id, err: error }, "Managed warm session preparation cleanup failed"));
|
||||
if (managedAiRuntime) await managedAiRuntime.cleanup().catch(() => logger.warn({ runId: run.id }, "AI connection refresh or cleanup failed"));
|
||||
let latestRun = await getRun(run.id).catch(() => null);
|
||||
if (preparationHandedOff() && (!latestRun || latestRun.status === "running")) nativeSessionResumeScheduled = true;
|
||||
try {
|
||||
if (latestRun && isHeartbeatRunTerminalStatus(latestRun.status)) {
|
||||
await writeExecutionControl(db, DELIVERY_QUEUES.executionFinalization, tx => tx
|
||||
|
||||
@@ -64,7 +64,7 @@ interface InstructionWorkingCopyInput {
|
||||
context: Selection["task"]["context"];
|
||||
session: {
|
||||
taskSession: { lastRunId: string | null } | null;
|
||||
taskSessionForRun: Selection["session"]["taskSessionForRun"];
|
||||
taskSessionForRun: { lastRunId: string | null } | null;
|
||||
sessionConfigMetadata: EffectiveRunSessionConfigMetadata;
|
||||
};
|
||||
workspace: Pick<Selection["workspace"], "executionTarget" | "executionWorkspace" | "persistedExecutionWorkspace">;
|
||||
|
||||
@@ -30,6 +30,7 @@ import { hasRemoteTerminationReceipt, remoteTerminationReceipt } from "../remote
|
||||
import { persistHeartbeatRunProcessMetadata } from "./run-lifecycle.js";
|
||||
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
|
||||
import { hasNativeLocalProcessStop, PROCESS_START_REQUESTED, releaseSettledNativeRunProcess } from "../native-local-process-stop.js";
|
||||
import { legacyControllerBootId } from "../legacy-controller-lease.js";
|
||||
|
||||
type Run = typeof heartbeatRuns.$inferSelect;
|
||||
|
||||
@@ -212,6 +213,40 @@ describePostgres("heartbeat recovery module ownership", () => {
|
||||
return (await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, invocationSource: "automation", status: "running", runtimeMode: "legacy" }).returning())[0];
|
||||
}
|
||||
|
||||
it.each(["unlaunched", "dispatched", "reassigned", "ambiguous"] as const)("claims the same expired portable preparation only with its original task lock (%s)", async state => {
|
||||
const run = await seedRun(), deps = callbacks(db);
|
||||
const environmentId = randomUUID(), leaseId = randomUUID();
|
||||
const oldBootId = randomUUID();
|
||||
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
|
||||
await db.update(heartbeatRuns).set({ scopeKind: "issue", controllerBootId: oldBootId, controllerLeaseExpiresAt: new Date(0),
|
||||
executionStage: "preparing" }).where(eq(heartbeatRuns.id, run.id));
|
||||
if (state !== "reassigned") await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, run.issueId!));
|
||||
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "plugin", config: {} });
|
||||
await db.insert(environmentLeases).values({ id: leaseId, companyId: run.companyId, environmentId, heartbeatRunId: run.id,
|
||||
issueId: run.issueId, provider: "plugin:fixture:tasks", providerLeaseId: "portable-task", status: "active",
|
||||
metadata: { agentId: run.agentId, driver: "plugin", runnerTaskLease: true, portableTaskPreviousLeaseId: randomUUID(),
|
||||
...(state === "dispatched" ? { runnerTaskDispatched: true } : {}) } });
|
||||
if (state === "ambiguous") await db.insert(environmentLeases).values({ companyId: run.companyId, environmentId,
|
||||
heartbeatRunId: run.id, issueId: run.issueId, provider: "plugin:fixture:tasks", providerLeaseId: "competing-task", status: "active" });
|
||||
const scope = { companyId: run.companyId, runId: run.id };
|
||||
const recovery = createHeartbeatRecovery(db, deps);
|
||||
await Promise.all([recovery.recoverActiveExecutions(scope), recovery.recoverActiveExecutions(scope)]);
|
||||
if (state === "unlaunched") {
|
||||
expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(run.id, { portablePreparationLeaseId: leaseId });
|
||||
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
|
||||
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
|
||||
expect(current).toMatchObject({ status: "running", controllerBootId: legacyControllerBootId, executionStage: "preparing" });
|
||||
expect(current.controllerLeaseExpiresAt!.getTime()).toBeGreaterThan(Date.now());
|
||||
expect((await db.select().from(issues).where(eq(issues.id, run.issueId!)))[0].executionRunId).toBe(run.id);
|
||||
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)))[0]).toMatchObject({
|
||||
status: "active", providerLeaseId: "portable-task", heartbeatRunId: run.id,
|
||||
});
|
||||
} else {
|
||||
expect(deps.executeRun).not.toHaveBeenCalled();
|
||||
expect(deps.setRunStatusFromLive).toHaveBeenCalledOnce();
|
||||
}
|
||||
});
|
||||
|
||||
it.each([false, true])("delivers scoped recovery to a preparation without a runner PID or checkpoint (locally tracked=%s)", async tracked => {
|
||||
const run = await seedRun(), deps = callbacks(db);
|
||||
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
|
||||
|
||||
@@ -38,9 +38,11 @@ import {
|
||||
} from "../sandbox-stop-and-retain.js";
|
||||
import { discoverSettledNativeRunProcesses, PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED, recordNativeLocalProcessStopInTransaction, releaseSettledNativeRunProcess, retryPendingSettledNativeRunProcesses } from "../native-local-process-stop.js";
|
||||
import { pendingCaseHandoffForRun } from "./case-handoff-lifecycle.js";
|
||||
import { portableTaskPreparationLeaseId } from "../plugin-task-ownership.js";
|
||||
import { retireInheritedRemoteRunnerIdentity } from "../native-runtime/native-inherited-runner-retry.js";
|
||||
import {
|
||||
LEGACY_LIVE_OWNER_SETTLE_GRACE_MS,
|
||||
handoffLegacyControllerPreparation,
|
||||
legacyControllerBootId,
|
||||
liveLegacyControllerExpiresAt,
|
||||
proveLegacyController,
|
||||
@@ -157,6 +159,7 @@ export interface HeartbeatRecoveryDependencies extends Pick<HeartbeatRetryDepend
|
||||
executeRun: (runId: string, options?: {
|
||||
nativeLeaseOwner?: string;
|
||||
nativeRestartRecovery?: NativeRestartRecoveryClaim;
|
||||
portablePreparationLeaseId?: string;
|
||||
onNativeExecutionLeaseAcquired?: () => Promise<void>;
|
||||
}) => Promise<void>;
|
||||
scheduleNativeSessionResumeDispatch: (runId: string, nextAttemptAt: Date, mode?: "restart") => void;
|
||||
@@ -1269,6 +1272,13 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
restartSuspendedRunIds.push(run.id);
|
||||
return;
|
||||
}
|
||||
if (agent.adapterType === "paperclip_runner" && run.runtimeMode === "legacy" && run.scopeKind === "issue" &&
|
||||
run.issueId && run.executionStage === "preparing" && run.controllerBootId === legacyControllerBootId &&
|
||||
await handoffLegacyControllerPreparation(db, run, portableTaskPreparationLeaseId(db, run),
|
||||
() => markRunHandedOffForShutdown(run.id))) {
|
||||
restartSuspendedRunIds.push(run.id);
|
||||
return;
|
||||
}
|
||||
const message = `Interrupted by graceful server shutdown (${signal})`;
|
||||
const running = runningProcesses.get(run.id);
|
||||
const control = run.runtimeMode === "legacy" ? adapterExecutionControls.get(run.id) : undefined;
|
||||
@@ -2467,6 +2477,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
nativeWorkspaceFinalizationPending,
|
||||
} of activeRuns) {
|
||||
if (run.runtimeMode === "native" && readNativeStartupCleanup(run)) continue;
|
||||
if (run.runtimeMode === "legacy" && isRunHandedOffForShutdown(run.id)) continue;
|
||||
if (parseObject(run.contextSnapshot).nativeQuestionCancellation) continue;
|
||||
// Native recovery owns these dispositions. Its writes happened after this
|
||||
// inventory was read; do not reap or overwrite them using the old row.
|
||||
@@ -2686,8 +2697,18 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
readNonEmptyString(runContext.wakeReason) === "issue_monitor_due" &&
|
||||
monitorNextCheckAt !== undefined &&
|
||||
(!monitorNextCheckAt || monitorNextCheckAt.getTime() <= now.getTime());
|
||||
const revokedState = await revokeExpiredLegacyControllerWithState(db, run);
|
||||
const reclaimPortablePreparation = priorBootSetupExpired && adapterType === "paperclip_runner" && run.scopeKind === "issue" && run.issueId
|
||||
? portableTaskPreparationLeaseId(db, run) : undefined;
|
||||
const revokedState = await revokeExpiredLegacyControllerWithState(db, run, reclaimPortablePreparation);
|
||||
if (!revokedState) continue;
|
||||
if (revokedState.portablePreparationLeaseId) {
|
||||
const execution = executeRun(run.id, { portablePreparationLeaseId: revokedState.portablePreparationLeaseId }).catch(error => {
|
||||
logger.error({ err: error, runId: run.id }, "portable task preparation recovery failed");
|
||||
});
|
||||
activeRunExecutionPromises.add(execution);
|
||||
void execution.finally(() => activeRunExecutionPromises.delete(execution));
|
||||
continue;
|
||||
}
|
||||
// The scan snapshot can predate a child the old controller saved just
|
||||
// before it died. The child checks above used that snapshot, so a proven
|
||||
// gone controller is never enough here: when revocation reports a child
|
||||
|
||||
@@ -128,7 +128,7 @@ export interface HeartbeatRuntimeExecutionInput {
|
||||
issueId: string | null;
|
||||
taskKey: string | null;
|
||||
};
|
||||
runtime: Pick<Selected, "nativeExecution" | "nativeRunnerInstanceId" | "getNativeFreshSessionHandoff"> & {
|
||||
runtime: Pick<Selected, "nativeExecution" | "nativeRunnerInstanceId" | "pluginTaskDescription" | "toolRefreshOnResume" | "getNativeFreshSessionHandoff"> & {
|
||||
nativeRuntimeResolution: Selection["nativeRuntimeResolution"];
|
||||
adapter: ServerAdapterModule;
|
||||
runtimeForAdapter: AdapterExecutionContext["runtime"];
|
||||
@@ -636,9 +636,11 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
|
||||
observability,
|
||||
getFreshSessionHandoff: getNativeFreshSessionHandoff,
|
||||
refreshTools: context.refreshTools === true,
|
||||
toolRefreshOnResume: input.runtime.toolRefreshOnResume,
|
||||
conversationMode: isConversation(issueContext),
|
||||
turnTimeoutMs: Math.max(0, asNumber(runtimeConfig.timeoutSec, 0)) * 1_000,
|
||||
runnerInstanceId: nativeRunnerInstanceId,
|
||||
pluginTaskDescription: input.runtime.pluginTaskDescription,
|
||||
leaseOwner: runOptions.nativeLeaseOwner,
|
||||
restartRecovery: runOptions.nativeRestartRecovery,
|
||||
onExecutionLeaseAcquired: runOptions.onNativeExecutionLeaseAcquired,
|
||||
|
||||
@@ -25,6 +25,7 @@ import { prepareNativeWorkspaceSync } from "../native-runtime/native-workspace-s
|
||||
import { resolveHeartbeatNativeRuntimeMode } from "../native-runtime/runtime-mode.js";
|
||||
import { claimNativeRestartRecoveries } from "../native-runtime/native-restart-recovery.js";
|
||||
import { buildNativeModelEnvelope, canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js";
|
||||
import * as runner from "../../vendor/paperclip-runner/index.js";
|
||||
import { prepareAndSelectHeartbeatRuntime, selectHeartbeatRuntime, type HeartbeatRuntimeSelectionInput } from "./runtime-selection.js";
|
||||
import { resolveRunnerEnvironmentForRun, resolveWarmRunnerIdleTimeoutMsForRun } from "../runner-environment-lifecycle.js";
|
||||
import { reserveRunBudget } from "../budget-reservations.js";
|
||||
@@ -82,7 +83,7 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
|
||||
run, agent,
|
||||
nativeRuntimeResolution: resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: null, workspaceId: null }),
|
||||
task: { issueRef: issue, issueContext: null, context: { paperclipTaskMarkdown: "Complete this task" }, executionContinuation: null, safeWakeComments: [], safeWakeCommentContext: null, paperclipWakePayload: null, acceptedPlanContinuationWake: false },
|
||||
session: { persistedNativeExecutionInput: null, persistedRunnerProfile: {}, taskSessionCredentialCompatible: true, taskSessionDecodedParams: null, taskSessionForRun: null, isFailedChatRunRetry: false, getFreshSessionHandoff: vi.fn(async () => "Earlier context") },
|
||||
session: { persistedNativeExecutionInput: null, persistedRunnerProfile: {}, taskResumeSessionId: null, isFailedChatRunRetry: false, getFreshSessionHandoff: vi.fn(async () => "Earlier context") },
|
||||
workspace: { persistedExecutionWorkspace: null, executionWorkspace: { cwd: "/tmp/paperclip-runtime-selection", repoUrl: null, repoRef: null, branchName: null }, executionTarget: null, isDotRun: false, projectRepositoryPaths: [],
|
||||
activeEnvironmentLease: { lease: { id: randomUUID(), companyId: company.id, environmentId: null, executionWorkspaceId: null, issueId: issue.id, heartbeatRunId: run.id, status: "active", leasePolicy: "ephemeral", provider: null, providerLeaseId: null, acquiredAt: now, lastUsedAt: now, expiresAt: null, releasedAt: null, failureReason: null, cleanupStatus: null, metadata: null, createdAt: now, updatedAt: now } },
|
||||
},
|
||||
@@ -97,6 +98,58 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
|
||||
const coordinators = (id: string) => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, id));
|
||||
const sandbox = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", reusableLeaseConfigured: false };
|
||||
|
||||
it.each(["uninspected", "confirmed", "replaced"])("selects an independent native session after %s disk-only recovery", async placement => {
|
||||
const { input, run, issue } = await fixture();
|
||||
const priorSession = randomUUID();
|
||||
const lease = input.workspace.activeEnvironmentLease!.lease;
|
||||
const description = { kind: "description" as const, taskId: randomUUID(), runner: { version: "test", digest: `sha256:${"a".repeat(64)}` } };
|
||||
const previousDescription = placement === "replaced" ? { ...description, taskId: randomUUID() } : description;
|
||||
const pending = { ...lease, providerLeaseId: description.taskId, metadata: { portableTaskPreviousLeaseId: randomUUID(),
|
||||
portableTaskPredecessor: { operation: { identity: { normalizedSessionId: priorSession } } },
|
||||
} };
|
||||
const ready = { ...pending.metadata, portableTaskOwnershipReady: true, portableTaskCheckpointUnavailable: true };
|
||||
await db.insert(environmentLeases).values(placement === "confirmed" ? { ...pending, metadata: ready } : pending);
|
||||
input.workspace.activeEnvironmentLease!.lease = pending;
|
||||
if (placement !== "uninspected") input.workspace.activeEnvironmentLease!.pluginTaskDescription = previousDescription;
|
||||
const taskRpc = vi.fn(async ({ kind }: { kind: string }) => {
|
||||
expect(kind).toBe("describe");
|
||||
await db.update(environmentLeases).set({
|
||||
metadata: ready,
|
||||
}).where(eq(environmentLeases.id, lease.id));
|
||||
return description;
|
||||
});
|
||||
input.workspace.executionTarget = { ...sandbox, leaseId: lease.id, runnerTask: taskRpc,
|
||||
reusableLeaseConfigured: true, effectiveCapabilities: { reusableLeases: true },
|
||||
runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } } as never;
|
||||
input.session.taskResumeSessionId = priorSession;
|
||||
const network = vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("Cloud is unavailable"));
|
||||
try {
|
||||
const selected = await selectHeartbeatRuntime(db, input);
|
||||
expect(selected.selected).toBe(true);
|
||||
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
|
||||
const saved = await read(run.id);
|
||||
expect(saved.nativeSessionId).toBeTruthy();
|
||||
expect(saved.nativeSessionId).not.toBe(priorSession);
|
||||
expect(await selected.getNativeFreshSessionHandoff?.()).toBe("Earlier context");
|
||||
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0]).toEqual(issue);
|
||||
expect(selected.pluginTaskDescription).toEqual(description);
|
||||
expect(taskRpc).toHaveBeenCalledTimes(placement === "confirmed" ? 0 : 1);
|
||||
expect(network).not.toHaveBeenCalled();
|
||||
|
||||
// A lost reply after committing selection must allow the same replacement run to retry.
|
||||
input.run = saved;
|
||||
input.session.persistedRunnerProfile = saved.runnerProfileJson ?? {};
|
||||
input.session.persistedNativeExecutionInput = selected.nativeExecution;
|
||||
await expect(selectHeartbeatRuntime(db, input)).resolves.toMatchObject({
|
||||
selected: true, nativeExecution: selected.nativeExecution,
|
||||
});
|
||||
input.run = { ...saved, nativeSessionId: priorSession };
|
||||
await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native_portable_reseed_session_already_attached");
|
||||
} finally {
|
||||
network.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects invalid local Runner configuration before native authority and retains the preparation owner", async () => {
|
||||
const f = await fixture();
|
||||
await claimPreparation(f);
|
||||
@@ -623,10 +676,14 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
|
||||
});
|
||||
|
||||
it("restores the immutable admitted input and saved checkpoint when newer context arrives", async () => {
|
||||
using discovery = vi.spyOn(runner, "describeRunnerdNativeSessionBackend");
|
||||
const { input, run } = await fixture();
|
||||
const selected = await selectHeartbeatRuntime(db, input);
|
||||
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
|
||||
expect(selected.toolRefreshOnResume).toBe(true);
|
||||
expect(discovery).toHaveBeenCalledTimes(1);
|
||||
const saved = await read(run.id);
|
||||
expect(saved.runnerProfileJson?.nativeToolRefreshOnResume).toBe(true);
|
||||
const profile = { ...saved.runnerProfileJson, sessionCheckpoint: { retained: "checkpoint" } };
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, run.id));
|
||||
input.run = await read(run.id);
|
||||
@@ -636,11 +693,13 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
|
||||
input.task.safeWakeComments = [{ id: randomUUID(), body: "New direction belongs to a separate wake" }];
|
||||
vi.mocked(buildNativeRuntimeContext).mockClear();
|
||||
const resumed = await selectHeartbeatRuntime(db, input);
|
||||
expect(resumed).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution, nativeRunnerInstanceId: selected.nativeRunnerInstanceId });
|
||||
expect(resumed).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution, nativeRunnerInstanceId: selected.nativeRunnerInstanceId,
|
||||
toolRefreshOnResume: true });
|
||||
expect((await read(run.id)).runnerProfileJson).toMatchObject(profile);
|
||||
expect(await db.select().from(completionContracts)).toHaveLength(1);
|
||||
expect(await coordinators(run.id)).toHaveLength(1);
|
||||
expect(buildNativeRuntimeContext).not.toHaveBeenCalled();
|
||||
expect(discovery).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("rejects an admitted input bound to another run without replacing the saved runtime", async () => {
|
||||
|
||||
@@ -39,6 +39,8 @@ import {
|
||||
} from "@paperclipai/db";
|
||||
import { getStartupTracer } from "../../instrumentation.js";
|
||||
import { documentService } from "../documents.js";
|
||||
import { readPortableTaskReseed } from "../plugin-task-ownership.js";
|
||||
import type { PluginTaskDescription } from "../environment-task-runtime.js";
|
||||
import { managedAgentProfileService } from "../managed-agent-profiles.js";
|
||||
import { remoteAgentProfileService } from "../remote-agent-profiles.js";
|
||||
import {
|
||||
@@ -47,7 +49,6 @@ import {
|
||||
buildNativeRuntimeContext,
|
||||
ensureNativeCompletionContract,
|
||||
findNativeSessionResumeRun,
|
||||
isNativeSessionId,
|
||||
isUnusedNativeSessionBootstrap,
|
||||
isUnusedLegacyNativeRetryReplacement,
|
||||
materializeNativeInteractionResponses,
|
||||
@@ -117,9 +118,7 @@ export interface HeartbeatRuntimeSelectionInput {
|
||||
session: {
|
||||
persistedNativeExecutionInput: NativeExecutionInput | null;
|
||||
persistedRunnerProfile: Record<string, unknown>;
|
||||
taskSessionCredentialCompatible: boolean;
|
||||
taskSessionDecodedParams: Record<string, unknown> | null;
|
||||
taskSessionForRun: { lastRunId: string | null } | null;
|
||||
taskResumeSessionId: string | null;
|
||||
isFailedChatRunRetry: boolean;
|
||||
getFreshSessionHandoff: (() => Promise<string | null>) | undefined;
|
||||
};
|
||||
@@ -127,7 +126,9 @@ export interface HeartbeatRuntimeSelectionInput {
|
||||
persistedExecutionWorkspace: { id: string; metadata?: Record<string, unknown> | null } | null;
|
||||
executionWorkspace: Omit<Parameters<typeof buildNativeExecutionInput>[0]["workspace"], "id">;
|
||||
executionTarget: Parameters<typeof prepareNativeWorkspaceSync>[0]["target"];
|
||||
activeEnvironmentLease: Pick<Parameters<typeof prepareNativeWorkspaceSync>[0], "lease"> | null;
|
||||
activeEnvironmentLease: (Pick<Parameters<typeof prepareNativeWorkspaceSync>[0], "lease"> & {
|
||||
pluginTaskDescription?: PluginTaskDescription;
|
||||
}) | null;
|
||||
isDotRun: boolean;
|
||||
projectRepositoryPaths: string[];
|
||||
onCheckpoint?: Parameters<typeof prepareNativeWorkspaceSync>[0]["onCheckpoint"];
|
||||
@@ -161,6 +162,8 @@ export type HeartbeatRuntimeSelectionResult = { selected: false } | {
|
||||
selected: true;
|
||||
nativeExecution: NativeExecutionInput | null;
|
||||
nativeRunnerInstanceId: string | null;
|
||||
pluginTaskDescription?: PluginTaskDescription;
|
||||
toolRefreshOnResume?: boolean;
|
||||
getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
|
||||
nativeWorkspaceSync: Awaited<ReturnType<typeof prepareNativeWorkspaceSync>>;
|
||||
};
|
||||
@@ -313,16 +316,17 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
|
||||
onNativeLifecycleSelected, onProviderResourceDisposition, onNativeOwnershipHeld, stopControllerLease } = input;
|
||||
const { issueRef, issueContext, context, executionContinuation, safeWakeComments, safeWakeCommentContext,
|
||||
paperclipWakePayload, acceptedPlanContinuationWake } = input.task;
|
||||
const { persistedNativeExecutionInput, persistedRunnerProfile, taskSessionCredentialCompatible, taskSessionDecodedParams,
|
||||
taskSessionForRun, isFailedChatRunRetry, getFreshSessionHandoff } = input.session;
|
||||
const { persistedNativeExecutionInput, persistedRunnerProfile, taskResumeSessionId, isFailedChatRunRetry, getFreshSessionHandoff } = input.session;
|
||||
const { persistedExecutionWorkspace, executionWorkspace, executionTarget, activeEnvironmentLease, isDotRun, projectRepositoryPaths } = input.workspace;
|
||||
const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config;
|
||||
const runOptions = input.recovery;
|
||||
const { providerTraceRequested, providerTraceCapture } = input.trace;
|
||||
if (!isDotRun) assertEphemeralNativeRuntime(nativeRuntimeResolution.kind === "native");
|
||||
let nativeExecution: NativeExecutionInput | null = null;
|
||||
let toolRefreshOnResume: boolean | undefined;
|
||||
let getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
|
||||
let nativeRunnerInstanceId: string | null = null;
|
||||
let pluginTaskDescription = activeEnvironmentLease?.pluginTaskDescription;
|
||||
let nativeWorkspaceSync: Awaited<ReturnType<typeof prepareNativeWorkspaceSync>> = null;
|
||||
let providerResourceDispositionForRun: ProviderResourceDisposition | undefined;
|
||||
if (nativeRuntimeResolution.kind === "native") {
|
||||
@@ -451,9 +455,20 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
|
||||
immediateRequests: immediateCompletion.requests,
|
||||
immediateRequestSources: immediateCompletion.sources,
|
||||
}));
|
||||
const taskNativeSessionId = !taskSessionCredentialCompatible ? null : readNonEmptyString(
|
||||
taskSessionDecodedParams?.sessionId,
|
||||
);
|
||||
const portableLease = activeEnvironmentLease?.lease;
|
||||
const runnerTask = executionTarget?.kind === "remote" && executionTarget.transport === "sandbox" &&
|
||||
executionTarget.leaseId === portableLease?.id ? executionTarget.runnerTask : undefined;
|
||||
const portable = portableLease && runnerTask ? await readPortableTaskReseed(db, {
|
||||
companyId: agent.companyId, runId: run.id, lease: portableLease, description: pluginTaskDescription,
|
||||
runnerTask,
|
||||
}) : null;
|
||||
if (portable) pluginTaskDescription = portable.description;
|
||||
const reseed = portable?.reseed;
|
||||
if (reseed && run.nativeSessionId && (!reseed.previousSessionId || run.nativeSessionId === reseed.previousSessionId ||
|
||||
persistedNativeExecutionInput?.session.normalizedSessionId !== run.nativeSessionId)) {
|
||||
throw new Error("native_portable_reseed_session_already_attached");
|
||||
}
|
||||
const reseedPortableSession = Boolean(reseed);
|
||||
// Compatibility for native retry rows created before same-run restart
|
||||
// recovery existed. Only an entirely unused replacement row may
|
||||
// inherit its source checkpoint; any process/provider evidence on the
|
||||
@@ -516,15 +531,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
|
||||
: null;
|
||||
const legacyRetrySessionId =
|
||||
compatibleLegacyRetrySource?.nativeSessionId;
|
||||
const taskResumeRunId =
|
||||
taskSessionForRun?.lastRunId &&
|
||||
taskSessionForRun.lastRunId !== run.id &&
|
||||
isNativeSessionId(taskNativeSessionId)
|
||||
? taskSessionForRun.lastRunId
|
||||
: null;
|
||||
const resumableTaskSessionId = isDotRun ? null : taskResumeRunId
|
||||
? taskNativeSessionId
|
||||
: (legacyRetrySessionId ?? null);
|
||||
const resumableTaskSessionId = reseedPortableSession || isDotRun ? null : taskResumeSessionId ?? legacyRetrySessionId ?? null;
|
||||
const requestedNativeSessionId =
|
||||
run.nativeSessionId ?? resumableTaskSessionId;
|
||||
// A task-session lastRunId can lag a failed turn or point at an older
|
||||
@@ -570,6 +577,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
|
||||
const persistedProfile = persistedRunnerProfile;
|
||||
if (persistedNativeExecutionInput) {
|
||||
nativeExecution = persistedNativeExecutionInput;
|
||||
toolRefreshOnResume = persistedProfile.nativeToolRefreshOnResume === true;
|
||||
if (
|
||||
nativeExecution.binding.companyId !== agent.companyId ||
|
||||
nativeExecution.binding.runId !== run.id ||
|
||||
@@ -896,11 +904,12 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
|
||||
const backendDescriptor = await measureRunPhase("runtime_selection.provider_discovery", () => describeRunnerdNativeSessionBackend(buildExecution({
|
||||
normalizedSessionId: nativeSessionId, resumedSession: previousNativeRun !== null,
|
||||
})));
|
||||
toolRefreshOnResume = backendDescriptor.capabilities.toolRefreshOnResume === true;
|
||||
const nativeExecutionWithCheckpoint = buildNativeExecutionWithCheckpoint({
|
||||
previousRun: previousNativeRun,
|
||||
normalizedSessionId: nativeSessionId,
|
||||
executionTargetKind: executionTarget?.kind ?? "local",
|
||||
toolRefreshOnResume: backendDescriptor.capabilities.toolRefreshOnResume === true,
|
||||
toolRefreshOnResume,
|
||||
refreshTools: context.refreshTools === true,
|
||||
buildExecution,
|
||||
});
|
||||
@@ -1037,6 +1046,8 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
|
||||
: {}),
|
||||
nativeExecutionInput:
|
||||
lockedProfile.nativeExecutionInput ?? nativeExecution,
|
||||
nativeToolRefreshOnResume: lockedProfile.nativeExecutionInput
|
||||
? lockedProfile.nativeToolRefreshOnResume === true : toolRefreshOnResume === true,
|
||||
...(typeof lockedProfile.inheritedProcessIdentityRunId === "string"
|
||||
? { inheritedProcessIdentityRunId: lockedProfile.inheritedProcessIdentityRunId }
|
||||
: lockedRun.processPid == null && lockedRun.processGroupId == null && lockedRun.processStartedAt == null
|
||||
@@ -1167,5 +1178,5 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
|
||||
})
|
||||
.where(eq(heartbeatRuns.id, run.id)));
|
||||
}
|
||||
return { selected: true, nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff, nativeWorkspaceSync };
|
||||
return { selected: true, nativeExecution, nativeRunnerInstanceId, pluginTaskDescription, toolRefreshOnResume, getNativeFreshSessionHandoff, nativeWorkspaceSync };
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { isNativeSessionId } from "../native-runtime/index.js";
|
||||
import { and, desc, eq } from "drizzle-orm";
|
||||
import { agentConfigRevisions, type Db, type agents, type agentTaskSessions, type heartbeatRuns, type issues, type projects } from "@paperclipai/db";
|
||||
import type { AiConnectionBinding, AiRuntimeConnectionBinding } from "@paperclipai/shared";
|
||||
@@ -29,7 +30,7 @@ type TaskSession = typeof agentTaskSessions.$inferSelect | null;
|
||||
type SessionCodec = ReturnType<typeof getAdapterSessionCodec>;
|
||||
|
||||
export interface HeartbeatSessionPreparationInput {
|
||||
run: Pick<typeof heartbeatRuns.$inferSelect, "createdAt">;
|
||||
run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "createdAt">;
|
||||
agent: typeof agents.$inferSelect;
|
||||
task: {
|
||||
context: Record<string, unknown>;
|
||||
@@ -204,6 +205,14 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre
|
||||
const sessionResetReason =
|
||||
sessionConfigFreshness.reasons.join("; ") || null;
|
||||
const taskSessionForRun = resetTaskSession ? null : taskSession;
|
||||
const taskSessionCredentialCompatible = !managedAiRuntime || managedAiSessionIdentityCompatible(
|
||||
taskSession?.sessionParamsJson?.paperclipAiCredentialIdentity,
|
||||
managedAiRuntime.sessionIdentity,
|
||||
managedAiRuntime.identity,
|
||||
);
|
||||
const taskSessionId = readNonEmptyString(taskSessionDecodedParams?.sessionId);
|
||||
const taskResumeSessionId = taskSessionCredentialCompatible && taskSessionForRun?.lastRunId &&
|
||||
taskSessionForRun.lastRunId !== run.id && isNativeSessionId(taskSessionId) ? taskSessionId : null;
|
||||
const getFreshSessionHandoff = issueRef ? createNativeSessionHandoffLoader({
|
||||
db, companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id, before: run.createdAt,
|
||||
throughCommentId: readNonEmptyString(context.conversationReplayThroughCommentId) ?? (context.interactionKind ? null : wakeCommentId),
|
||||
@@ -230,7 +239,8 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre
|
||||
const handoff = await getFreshSessionHandoff();
|
||||
if (handoff) context.paperclipFreshSessionHandoffMarkdown = handoff;
|
||||
}
|
||||
return { sessionConfigMetadata, configuredModel, sessionConfigFreshness, resetTaskSession, sessionResetReason, taskSessionForRun, getFreshSessionHandoff, previousSessionParams };
|
||||
return { sessionConfigMetadata, configuredModel, sessionConfigFreshness, resetTaskSession, sessionResetReason, taskSessionForRun,
|
||||
taskSessionCredentialCompatible, taskResumeSessionId, getFreshSessionHandoff, previousSessionParams };
|
||||
}
|
||||
|
||||
export interface HeartbeatAdapterSessionInput {
|
||||
@@ -264,7 +274,7 @@ export async function resolveHeartbeatAdapterSession(input: HeartbeatAdapterSess
|
||||
const { agent, evaluateSessionCompaction } = input;
|
||||
const { context, taskKey, issueId, continuationSummary, executionContinuation } = input.task;
|
||||
const { runtime, taskSession, sessionCodec, explicitResumeSessionDisplayId, runtimeSessionParams, managedAiRuntime, prepared } = input.session;
|
||||
const { sessionConfigMetadata, sessionConfigFreshness, resetTaskSession, taskSessionForRun, previousSessionParams } = prepared;
|
||||
const { sessionConfigMetadata, sessionConfigFreshness, resetTaskSession, taskSessionForRun, taskSessionCredentialCompatible, previousSessionParams } = prepared;
|
||||
const { latestWorkspaceConfigMetadata, workspaceConfigFreshness, reusedExecutionWorkspace, resolvedWorkspaceReusePolicy, persistedExecutionWorkspace, runtimeWorkspaceWarnings, requestedShouldReuseExisting, workspaceReuseRequest } = input.workspace;
|
||||
const runtimeSessionFallback =
|
||||
taskKey || resetTaskSession
|
||||
@@ -328,11 +338,6 @@ export async function resolveHeartbeatAdapterSession(input: HeartbeatAdapterSess
|
||||
delete context.paperclipPreviousSessionId;
|
||||
}
|
||||
|
||||
const taskSessionCredentialCompatible = !managedAiRuntime || managedAiSessionIdentityCompatible(
|
||||
taskSession?.sessionParamsJson?.paperclipAiCredentialIdentity,
|
||||
managedAiRuntime.sessionIdentity,
|
||||
managedAiRuntime.identity,
|
||||
);
|
||||
if (managedAiRuntime) {
|
||||
sessionConfigMetadata.aiCredentialIdentity = managedAiRuntime.sessionIdentity;
|
||||
if (!taskSessionCredentialCompatible) {
|
||||
@@ -396,5 +401,5 @@ export async function resolveHeartbeatAdapterSession(input: HeartbeatAdapterSess
|
||||
},
|
||||
};
|
||||
|
||||
return { previousSessionDisplayId, runtimeSessionParamsForAdapter, taskSessionCredentialCompatible, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction };
|
||||
return { previousSessionDisplayId, runtimeSessionParamsForAdapter, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction };
|
||||
}
|
||||
@@ -104,6 +104,7 @@ const RETAINED_SANDBOX_GONE = new RegExp(`^${NATIVE_REMOTE_RECOVERY_LEASE_MISMAT
|
||||
|
||||
/** Prepares the workspace and environment; dispatch and terminal lease release stay with the caller. */
|
||||
export interface HeartbeatWorkspacePreparationInput {
|
||||
signal?: AbortSignal;
|
||||
run: typeof heartbeatRuns.$inferSelect;
|
||||
agent: typeof agents.$inferSelect;
|
||||
task: {
|
||||
@@ -145,6 +146,7 @@ export interface HeartbeatWorkspacePreparationInput {
|
||||
configSnapshot: WorkspaceMetadataInput["configSnapshot"];
|
||||
secretManifest: WorkspaceMetadataInput["secretManifest"];
|
||||
managedAiCredentialIdentity?: string;
|
||||
runnerTaskResumeSessionId?: string | null;
|
||||
};
|
||||
reuse: {
|
||||
requestedShouldReuseExisting: boolean;
|
||||
@@ -154,7 +156,7 @@ export interface HeartbeatWorkspacePreparationInput {
|
||||
nativeRecoveryExecutionWorkspaceId: string | null;
|
||||
persistedNativeExecutionInput: NativeExecutionInput | null;
|
||||
isDotRun: boolean;
|
||||
runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim };
|
||||
runOptions: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; portablePreparationLeaseId?: string };
|
||||
};
|
||||
services: {
|
||||
workspaceWorkers?: WorkspaceWorkers;
|
||||
@@ -253,6 +255,12 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
|
||||
appendRunEvent,
|
||||
} = input.services;
|
||||
const pluginTaskWorkspace = selectedEnvironmentForConfig?.driver === "plugin" && agent.adapterType === "paperclip_runner";
|
||||
const restartRecovery = runOptions.nativeRestartRecovery;
|
||||
const portablePreparationLeaseId = runOptions.portablePreparationLeaseId ??
|
||||
(restartRecovery?.kind === "bootstrap_incomplete" ? restartRecovery.portablePreparationLeaseId : undefined);
|
||||
if (portablePreparationLeaseId && (!pluginTaskWorkspace || controllerOnly)) {
|
||||
throw remoteRecoveryLeaseMismatch("workspace_preparation", "environment_driver");
|
||||
}
|
||||
for (const workspaceId of new Set([existingExecutionWorkspace?.id, reusableExistingExecutionWorkspace?.id].filter(Boolean))) {
|
||||
const [workspace] = await db.select({ metadata: executionWorkspacesTable.metadata }).from(executionWorkspacesTable)
|
||||
.where(and(eq(executionWorkspacesTable.companyId, agent.companyId), eq(executionWorkspacesTable.id, workspaceId!)));
|
||||
@@ -1301,7 +1309,8 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
|
||||
await controllerLease.assertOwned();
|
||||
const remoteRecovery = runOptions.nativeRestartRecovery?.kind === "reattach_remote_runner"
|
||||
? runOptions.nativeRestartRecovery : null;
|
||||
const pluginRecovery = remoteRecovery?.remote.kind === "plugin_task" ? remoteRecovery.remote : null;
|
||||
const pluginRecovery = remoteRecovery?.remote.kind === "plugin_task" ? remoteRecovery.remote : portablePreparationLeaseId
|
||||
? { kind: "plugin_task" as const, leaseId: portablePreparationLeaseId, preparation: true as const } : undefined;
|
||||
const sandboxRecovery = remoteRecovery && remoteRecovery.remote.kind !== "plugin_task" ? remoteRecovery.remote : null;
|
||||
const recoveryWorkspace = sandboxRecovery
|
||||
? readNativeWorkspaceSyncReference(parseObject(run.runnerProfileJson).nativeWorkspaceSync) : null;
|
||||
@@ -1340,10 +1349,12 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
|
||||
adapterConfig: parseObject(agent.adapterConfig),
|
||||
admittedLifecycleMode: persistedNativeExecutionInput?.session.lifecyclePolicy.mode,
|
||||
managedAiCredentialIdentity: input.config.managedAiCredentialIdentity,
|
||||
runnerTaskResumeSessionId: input.config.runnerTaskResumeSessionId,
|
||||
runnerTaskProjectIds:[...new Set([executionWorkspace.projectId,...resolvedWorkspace.runnerTaskProjectIds ?? [],
|
||||
...(executionWorkspace.additionalWorkspaces ?? []).map(workspace=>workspace.projectId)]
|
||||
.filter((id):id is string=>Boolean(id)))],
|
||||
recoverPluginTask: runOptions.nativeRestartRecovery?.kind === "resume_dead_runner" || runOptions.nativeRestartRecovery?.kind === "bootstrap_incomplete",
|
||||
recoverPluginTask: !portablePreparationLeaseId &&
|
||||
(restartRecovery?.kind === "resume_dead_runner" || restartRecovery?.kind === "bootstrap_incomplete"),
|
||||
issueId: issueId ?? null,
|
||||
heartbeatRunId: run.id,
|
||||
agentId: agent.id,
|
||||
@@ -1441,6 +1452,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
|
||||
effectiveExecutionWorkspaceMode,
|
||||
persistedExecutionWorkspace,
|
||||
runnerTaskProjectIds: resolvedWorkspace.runnerTaskProjectIds,
|
||||
...(input.signal ? { signal: input.signal } : {}),
|
||||
assertOwned: remoteAuthority ? assertRemotePreparationOwned : controllerLease.assertOwned,
|
||||
duplexObservabilityRecorder,
|
||||
}));
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { notifyLiveRunRecovery } from "./live-run-recovery-notifications.js";
|
||||
import { and, eq, gt, lte, sql } from "drizzle-orm";
|
||||
import { and, eq, gt, isNull, lte, sql, type SQL } from "drizzle-orm";
|
||||
import { controllerLeaseDb, heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import type { LegacyControllerLeaseLossCause, LegacyControllerLeaseLossTiming } from "./run-cancellation.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
@@ -176,7 +176,25 @@ export async function hasLiveLegacyController(db: Db, run: Run): Promise<boolean
|
||||
return (await liveLegacyControllerExpiresAt(db, run)) !== null;
|
||||
}
|
||||
|
||||
export type RevokedLegacyControllerState = Pick<Run, "executionStage" | "processPid" | "processGroupId">;
|
||||
export type RevokedLegacyControllerState = Pick<Run, "executionStage" | "processPid" | "processGroupId"> & { portablePreparationLeaseId?: string };
|
||||
|
||||
function legacyPreparationStage() {
|
||||
return and(eq(heartbeatRuns.runtimeMode, "legacy"), eq(heartbeatRuns.executionStage, "preparing"),
|
||||
isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), isNull(heartbeatRuns.processStartedAt));
|
||||
}
|
||||
|
||||
export async function handoffLegacyControllerPreparation(db: Db, run: Run, preparationLease: SQL<string | null>, onHandoff: () => void) {
|
||||
return db.transaction(async tx => {
|
||||
await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id });
|
||||
const [released] = await tx.update(heartbeatRuns).set({ controllerLeaseExpiresAt: sql`clock_timestamp()` })
|
||||
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
|
||||
eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.controllerBootId, legacyControllerBootId),
|
||||
legacyPreparationStage(), sql`${preparationLease} is not null`)).returning({ id: heartbeatRuns.id });
|
||||
if (!released) return false;
|
||||
onHandoff();
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
/** Atomically revoke an expired controller. Renewal and revocation serialize on
|
||||
* the run row. Expiry permits cleanup, never dispatch of a replacement agent.
|
||||
@@ -186,6 +204,7 @@ export type RevokedLegacyControllerState = Pick<Run, "executionStage" | "process
|
||||
export async function revokeExpiredLegacyControllerWithState(
|
||||
db: Db,
|
||||
run: Run,
|
||||
reclaimPreparation?: SQL<string | null>,
|
||||
): Promise<RevokedLegacyControllerState | null> {
|
||||
if (run.runtimeMode === "native" || !run.controllerBootId) {
|
||||
return { executionStage: run.executionStage, processPid: run.processPid, processGroupId: run.processGroupId };
|
||||
@@ -193,19 +212,19 @@ export async function revokeExpiredLegacyControllerWithState(
|
||||
const controllerBootId = run.controllerBootId;
|
||||
const [revoked] = await db.transaction(async tx => {
|
||||
await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id });
|
||||
return tx.update(heartbeatRuns).set({
|
||||
controllerBootId: randomUUID(),
|
||||
controllerLeaseExpiresAt: sql`clock_timestamp() + interval '60 seconds'`,
|
||||
}).where(and(
|
||||
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
|
||||
eq(heartbeatRuns.status, "running"),
|
||||
eq(heartbeatRuns.controllerBootId, controllerBootId),
|
||||
lte(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`),
|
||||
)).returning({
|
||||
executionStage: heartbeatRuns.executionStage,
|
||||
processPid: heartbeatRuns.processPid,
|
||||
processGroupId: heartbeatRuns.processGroupId,
|
||||
});
|
||||
const expired = and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
|
||||
eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.controllerBootId, controllerBootId),
|
||||
lte(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`));
|
||||
const state = { executionStage: heartbeatRuns.executionStage, processPid: heartbeatRuns.processPid,
|
||||
processGroupId: heartbeatRuns.processGroupId };
|
||||
if (reclaimPreparation) {
|
||||
const [reclaimed] = await tx.update(heartbeatRuns).set(legacyControllerClaim("legacy")).where(and(expired, legacyPreparationStage(),
|
||||
sql`${reclaimPreparation} is not null`)).returning({ ...state, portablePreparationLeaseId: reclaimPreparation });
|
||||
if (reclaimed?.portablePreparationLeaseId) return [{ ...reclaimed, portablePreparationLeaseId: reclaimed.portablePreparationLeaseId }];
|
||||
}
|
||||
return tx.update(heartbeatRuns).set({ controllerBootId: randomUUID(),
|
||||
controllerLeaseExpiresAt: sql`clock_timestamp() + interval '60 seconds'`,
|
||||
}).where(expired).returning(state);
|
||||
});
|
||||
return revoked ?? null;
|
||||
}
|
||||
|
||||
@@ -509,8 +509,16 @@ describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable d
|
||||
await db.update(nativeRunFinalizations).set({ phase: "committed", leaseOwner: null, leaseExpiresAt: null })
|
||||
.where(eq(nativeRunFinalizations.runId, fixture.scope.runId));
|
||||
await expect(readSandbox()).resolves.toEqual({ remoteCwd: "/workspace" });
|
||||
await expect(inspect({ readControlState: () => ({ identity, commands: [{ status: "pending" }],
|
||||
committedEvents: [{ eventType: "run.terminal", envelope: identity }] }) })).rejects.toBeInstanceOf(NativePredecessorRetirementUnprovenError);
|
||||
const readUnsettledControlState = () => ({ identity, commands: [{ status: "pending" }],
|
||||
committedEvents: [{ eventType: "run.terminal", envelope: identity }] });
|
||||
await expect(inspect({ readControlState: readUnsettledControlState }))
|
||||
.rejects.toBeInstanceOf(NativePredecessorRetirementUnprovenError);
|
||||
const movedFailure = await inspect({
|
||||
target: { ...target, sandboxLeaseAcquisition: { outcome: "replacement", providerLeaseId: "new-sandbox" } } as never,
|
||||
portableSourceTaskId: "retained-sandbox", readControlState: readUnsettledControlState,
|
||||
}).catch(error => error);
|
||||
expect(movedFailure).not.toBeInstanceOf(NativePredecessorRetirementUnprovenError);
|
||||
expect(movedFailure.message).toBe(`${denied}; warm_recovery_refused=journal_unsettled`);
|
||||
// A created or replacement sandbox holds no Runner to adopt. Live adoption
|
||||
// steps aside so ordinary admission can retire the finished owner, instead
|
||||
// of denying with prior_owner_terminal_state_indeterminate (Nov 10 staging).
|
||||
|
||||
@@ -123,6 +123,7 @@ export type NativeRestartRecoveryClaim =
|
||||
providerAttempt: number;
|
||||
restartKind: NativeRestartKind;
|
||||
recoveryRequestId: string | null;
|
||||
portablePreparationLeaseId?: string;
|
||||
/** False when the prior controller suspended this run in a clean drain. */
|
||||
chargesProviderAttempt?: boolean;
|
||||
};
|
||||
@@ -1546,6 +1547,12 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
const continuedSessionOnly = hasCheckpoint && checkpointIdentityMatches && providerEvents.length === 0
|
||||
&& !pendingInheritedRequests && checkpointRecord.activeTurnId == null && checkpointRecord.semanticResult == null
|
||||
&& checkpointRecord.terminal == null;
|
||||
const portablePreparation = pluginTask?.unlaunchedPortable === true && row.run.status === "running" &&
|
||||
row.coordinator.phase === "observed" && !durableAuthority && !row.run.sessionIdAfter &&
|
||||
row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION &&
|
||||
(hasCheckpoint ? admissionCheckpointOnly && providerEvents.length === 0 : !hasProviderEvidence) &&
|
||||
(!row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt || await carriesInheritedRunnerIdentity(tx, row.run)) &&
|
||||
!(await hasNativeExecutionStartEvidence(tx, row.run));
|
||||
const remoteRunnerNeverLaunched = !unclaimedBootstrap && remoteWorkspace?.state === "prepared" && !pluginTask
|
||||
&& !durableAuthority && stagedAttachment === null
|
||||
&& row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION
|
||||
@@ -1672,11 +1679,11 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
: await restartResumeKeepsAttempt(tx, { runId: row.run.id, coordinator: row.coordinator });
|
||||
const resumeKeepsAttempt = restartCredit.keepsAttempt;
|
||||
const classification = classifyNativeRunnerRecoveryEvidence({
|
||||
remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority),
|
||||
unclaimedBootstrap,
|
||||
remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority) || portablePreparation,
|
||||
unclaimedBootstrap: unclaimedBootstrap || portablePreparation && row.coordinator.attempt === 0,
|
||||
remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched
|
||||
|| remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands || remoteFollowUpNeverAttached,
|
||||
remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched ? "remote_runner_never_launched"
|
||||
|| remoteRunnerExitedBeforeProviderWork || remoteRunnerNeverReceivedCommands || remoteFollowUpNeverAttached || portablePreparation,
|
||||
remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched || portablePreparation ? "remote_runner_never_launched"
|
||||
: remoteRunnerExitedBeforeProviderWork ? "remote_runner_exited_before_provider_work"
|
||||
: remoteRunnerNeverReceivedCommands ? "remote_runner_never_received_commands"
|
||||
: remoteFollowUpNeverAttached ? "remote_follow_up_never_attached"
|
||||
@@ -1687,13 +1694,13 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
knownProviderProcessAlive: providerProcesses.livePids.length > 0,
|
||||
knownProviderProcessIdentityAmbiguous:
|
||||
providerProcesses.ambiguousLivePids.length > 0,
|
||||
hasCheckpoint: hasCheckpoint && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly)
|
||||
hasCheckpoint: hasCheckpoint && !portablePreparation && !remoteRunnerNeverReceivedCommands && !(unclaimedBootstrap && inheritedCheckpointOnly)
|
||||
&& !remoteRunnerNeverLaunched && !remoteFollowUpNeverAttached && !(remoteBootstrapProvenIncomplete && reboundSessionOnly),
|
||||
stagedAttachmentVerified: stagedAttachment !== null,
|
||||
checkpointIdentityMatches:
|
||||
checkpointIdentityMatches &&
|
||||
(!remoteSandbox || remoteWorkspace !== null),
|
||||
hasProviderEvidence: hasProviderEvidence && !(unclaimedBootstrap && inheritedCheckpointOnly),
|
||||
hasProviderEvidence: hasProviderEvidence && !portablePreparation && !(unclaimedBootstrap && inheritedCheckpointOnly),
|
||||
checkpointFailed:
|
||||
(checkpointRecord.terminal as Record<string, unknown> | undefined)
|
||||
?.runTerminalState === "failed" &&
|
||||
@@ -1893,7 +1900,7 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
leaseOwner,
|
||||
// Cleanup and earlier candidates can outlive the batch observation.
|
||||
// The owner row is locked: grant the full lease at this write.
|
||||
leaseExpiresAt: sql`clock_timestamp() + ${durableAuthority ? REMOTE_CONTROLLER_LEASE_MS : 20 * 60_000} * interval '1 millisecond'`,
|
||||
leaseExpiresAt: sql`clock_timestamp() + ${durableAuthority || portablePreparation ? REMOTE_CONTROLLER_LEASE_MS : 20 * 60_000} * interval '1 millisecond'`,
|
||||
controllerBootId: controller.bootId,
|
||||
controllerHostHash,
|
||||
controllerPid: controller.pid,
|
||||
@@ -1981,7 +1988,7 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
await restoreNativeSessionExecution(tx as unknown as Db, { issue, run: row.run, agentName: resumptionAgent.name, now });
|
||||
}
|
||||
|
||||
claimScope = { companyId: row.run.companyId, issueId: row.coordinator.issueId, controller, remote: Boolean(durableAuthority) };
|
||||
claimScope = { companyId: row.run.companyId, issueId: row.coordinator.issueId, controller, remote: Boolean(durableAuthority || portablePreparation) };
|
||||
const common = {
|
||||
runId: row.run.id,
|
||||
leaseOwner,
|
||||
@@ -2021,6 +2028,7 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
return {
|
||||
kind: claimKind,
|
||||
...common,
|
||||
...(claimKind === "bootstrap_incomplete" && portablePreparation && pluginTask ? { portablePreparationLeaseId: pluginTask.id } : {}),
|
||||
} satisfies NativeRestartRecoveryClaim;
|
||||
}).catch((error: unknown) => {
|
||||
// The whole candidate transaction rolled back, so this run's state is
|
||||
|
||||
@@ -3,6 +3,7 @@ import { environmentLeases, heartbeatRuns, nativeRunFinalizations, type Db } fro
|
||||
import { and, desc, eq, inArray, isNull, or, sql } from "drizzle-orm";
|
||||
import { remoteExecutionHasStopped } from "../remote-execution-termination.js";
|
||||
import { readSystemRecovery } from "../system-recovery-state.js";
|
||||
import { unlaunchedPortableTaskLease } from "../plugin-task-ownership.js";
|
||||
|
||||
type NativeRemoteExecutionScope = {
|
||||
run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "runnerProfileJson">;
|
||||
@@ -13,6 +14,7 @@ export async function readNativeRemoteExecutionOwnershipForRuns(db: Db, inputs:
|
||||
const key = (companyId: string, issueId: string, runId: string) => `${companyId}:${issueId}:${runId}`;
|
||||
const leases = inputs.length ? await db.select({ id: environmentLeases.id, companyId: environmentLeases.companyId,
|
||||
issueId: environmentLeases.issueId, runId: environmentLeases.heartbeatRunId,
|
||||
unlaunchedPortable: unlaunchedPortableTaskLease(),
|
||||
metadata: sql<Record<string, unknown>>`jsonb_build_object('runnerTaskStartup', ${environmentLeases.metadata}->'runnerTaskStartup')`,
|
||||
}).from(environmentLeases).where(and(
|
||||
or(...inputs.map(input => and(eq(environmentLeases.companyId, input.run.companyId),
|
||||
|
||||
@@ -9,8 +9,10 @@ import * as stagedAttachmentRecovery from "./native-staged-attachment-recovery.j
|
||||
import * as controllerRecovery from "./native-controller-recovery.js";
|
||||
import { isSettledRemoteWarmControlState } from "./remote-warm-session-recovery.js";
|
||||
import * as remoteWarmRecoveryRuntime from "./remote-warm-session-recovery.js";
|
||||
import * as pluginTaskOwnership from "../plugin-task-ownership.js";
|
||||
import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js";
|
||||
import * as environmentTaskRuntime from "../environment-task-runtime.js";
|
||||
import * as pluginCredentialWriteback from "../plugin-task-credential-writeback.js";
|
||||
import * as pluginTaskRuntime from "./plugin-runner-task.js";
|
||||
import * as completionFeedback from "./native-completion-feedback.js";
|
||||
import { environmentTaskOperationSchema } from "@paperclipai/plugin-sdk";
|
||||
@@ -19,7 +21,7 @@ import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@open
|
||||
import { configuredEnvironment, configuredEnvironmentProjection } from "../../vendor/paperclip-runner/index.js";
|
||||
import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from "vitest";
|
||||
import { createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "../system-recovery-state.js";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { once } from "node:events";
|
||||
import { PgDialect } from "drizzle-orm/pg-core";
|
||||
import type { SQL } from "drizzle-orm";
|
||||
@@ -14176,15 +14178,20 @@ describe("native warm session supervision", () => {
|
||||
{ name: "rotated provider key", firstEnvironment: { OPENAI_API_KEY: "fixture-key" }, secondEnvironment: { OPENAI_API_KEY: "fixture-rotated-key" }, reuse: false },
|
||||
{ name: "rotated Claude token", firstEnvironment: { CLAUDE_CODE_OAUTH_TOKEN: "fixture-token" }, secondEnvironment: { CLAUDE_CODE_OAUTH_TOKEN: "fixture-rotated-token" }, reuse: false },
|
||||
{ name: "tool refresh", refreshTools: true, reuse: false },
|
||||
{ name: "unsupported tool refresh", refreshTools: true, unsupportedRefresh: true, reuse: false },
|
||||
{ name: "network access change", firstEnvironment: { PAPERCLIP_RUNNER_NETWORK_ACCESS: "disabled" }, secondEnvironment: { PAPERCLIP_RUNNER_NETWORK_ACCESS: "enabled" }, reuse: false },
|
||||
{ name: "run-scoped environment broker", firstEnvironment: { PAPERCLIP_GITHUB_BROKER_TOKEN: "first-run-capability" }, secondEnvironment: { PAPERCLIP_GITHUB_BROKER_TOKEN: "second-run-capability" }, reuse: false },
|
||||
{ name: "Core-hosted GitHub access", managedGitHub: true, reuse: true, adoptionReuse: false },
|
||||
] as Array<{ name: string; firstEnvironment?: Record<string, string>; secondEnvironment?: Record<string, string>;
|
||||
refreshTools?: boolean; managedGitHub?: boolean; reuse: boolean; adoptionReuse?: boolean }>)(
|
||||
refreshTools?: boolean; unsupportedRefresh?: boolean; managedGitHub?: boolean; reuse: boolean; adoptionReuse?: boolean }>)(
|
||||
"decides warm and adopted process reuse from one key: $name",
|
||||
async ({ name, firstEnvironment = {}, secondEnvironment = {}, refreshTools, managedGitHub = false, reuse, adoptionReuse = reuse }) => {
|
||||
async ({ name, firstEnvironment = {}, secondEnvironment = {}, refreshTools, unsupportedRefresh = false,
|
||||
managedGitHub = false, reuse, adoptionReuse = reuse }) => {
|
||||
const decisions: Record<string, boolean> = {};
|
||||
for (const path of ["warm", "adoption"] as const) {
|
||||
for (const path of ["warm", "adoption", ...(refreshTools ? ["portable", "portable-bootstrap", "legacy-plugin"] as const : [])] as const) {
|
||||
const portable = path === "portable" || path === "portable-bootstrap";
|
||||
const bootstrap = path === "portable-bootstrap";
|
||||
const pluginRecovery = portable || path === "legacy-plugin";
|
||||
githubAccess.create.mockReset().mockImplementation(async () => ({ env: {}, ready: true, duplexRequested: false,
|
||||
activate: githubAccess.activate.mockReset().mockImplementation(() => vi.fn()), stop: githubAccess.stop.mockReset().mockResolvedValue(undefined) }));
|
||||
const stateBase = await mkdtemp(join(tmpdir(), "paperclip-runnerd-reuse-key-"));
|
||||
@@ -14195,15 +14202,24 @@ describe("native warm session supervision", () => {
|
||||
const scope = `${path}-${name.replaceAll(" ", "-")}`;
|
||||
const first = {
|
||||
...execution,
|
||||
...(pluginRecovery ? { schema: "paperclip.native-execution-input.v5", executionMode: "default",
|
||||
provider: { ...execution.provider, approvalPolicy: "never" }, planningContext: null, runtimeContext: nativeRuntimeContextFixture() } as const : {}),
|
||||
binding: { ...execution.binding, runId: `run-reuse-key-first-${scope}`, executionWorkspaceId: "workspace-reuse-key" },
|
||||
workspace: { cwd: "/tmp/runnerd-reuse-key", repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: `session-reuse-key-${scope}`, driverKind: "codex_app_server" as const,
|
||||
protocolVersion: 1 as const, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
|
||||
} as NativeExecutionInputV1;
|
||||
const second = { ...first, binding: { ...first.binding, runId: `run-reuse-key-second-${scope}` } } as NativeExecutionInputV1;
|
||||
} as NativeExecutionInput;
|
||||
if (pluginRecovery) parseNativeExecutionInput(first);
|
||||
const second = { ...first, binding: { ...first.binding, runId: `run-reuse-key-second-${scope}` } };
|
||||
if ("runtimeContext" in second) {
|
||||
second.runtimeContext = structuredClone(second.runtimeContext);
|
||||
second.runtimeContext.mcp.digest = "1".repeat(64);
|
||||
second.runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(second.runtimeContext);
|
||||
}
|
||||
const target = { kind: "remote" as const, transport: "sandbox" as const, environmentId: "environment-reuse-key",
|
||||
remoteCwd: "/home/daytona/paperclip-workspace", runner: { execute: vi.fn() } } as never;
|
||||
remoteCwd: "/home/daytona/paperclip-workspace", runner: { execute: vi.fn() } };
|
||||
const firstSession = { close: vi.fn(async () => undefined) };
|
||||
const recoveredSession = { close: vi.fn(async () => undefined), snapshot: vi.fn() };
|
||||
const result = { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" }, turnId: "turn",
|
||||
normalizedSessionId: first.session.normalizedSessionId, providerSessionId: "provider-reuse-key", driverKind: "test",
|
||||
driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
@@ -14219,36 +14235,138 @@ describe("native warm session supervision", () => {
|
||||
return result;
|
||||
})
|
||||
.mockImplementationOnce(async (options) => {
|
||||
decisions[path] = options.existingSession === firstSession;
|
||||
decisions[path] = options.existingSession === (portable ? recoveredSession : firstSession);
|
||||
return result;
|
||||
});
|
||||
const inspected = vi.spyOn(remoteWarmRecoveryRuntime, "inspectRemoteWarmSessionRecovery").mockImplementation(async request => {
|
||||
decisions[path] = request.canReuseProcess(first);
|
||||
const reuseProcess = request.canReuseProcess(first);
|
||||
if (pluginRecovery) {
|
||||
expect(reuseProcess).toBe(false);
|
||||
expect(request.matchesExecution(first)).toBe(!unsupportedRefresh);
|
||||
if (unsupportedRefresh) throw new Error("fixture unsupported tool refresh refused");
|
||||
expect(request.matchesExecution({ ...first, provider: { ...first.provider, model: "another-model" } } as NativeExecutionInput)).toBe(false);
|
||||
if ("runtimeContext" in first) {
|
||||
const changed = structuredClone(first);
|
||||
changed.runtimeContext.instructions.bundle.digest = "2".repeat(64);
|
||||
changed.runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(changed.runtimeContext);
|
||||
expect(request.matchesExecution(changed)).toBe(false);
|
||||
}
|
||||
const checkpoint = request.loadCheckpoint(first);
|
||||
expect(checkpoint?.identity.runId).toBe(first.binding.runId);
|
||||
recoveredSession.snapshot.mockResolvedValue(checkpoint);
|
||||
return { identity: request.identity as Record<string, string>, checkpoint: checkpoint!,
|
||||
previousExecution: first, reuseProcess, ...(portable ? { portableSourceTaskId: "source-task" } : {}),
|
||||
remote: { providerLeaseId: "target-task", remoteCwd: target.remoteCwd }, revalidate: async () => {} };
|
||||
}
|
||||
decisions[path] = reuseProcess;
|
||||
throw new Error("fixture adoption decision recorded");
|
||||
});
|
||||
const ownership = portable
|
||||
? vi.spyOn(pluginTaskOwnership, "readyPortableTaskSource").mockImplementation(async (_db, _company, _lease, identity, scope) => {
|
||||
expect(scope?.accountIdentity).toBe("fixture-account");
|
||||
return { taskId: "source-task", identity: identity as NonNullable<Awaited<ReturnType<typeof pluginTaskOwnership.readyPortableTaskSource>>>["identity"] };
|
||||
}) : undefined;
|
||||
const projection = vi.spyOn(controllerRecovery, "resolveNativeControllerRecovery").mockResolvedValue({ kind: "no_prior_execution" });
|
||||
const restore = path === "legacy-plugin" ? vi.spyOn(runnerCheckpoint, "restoreNativeRunnerCheckpoint")
|
||||
.mockImplementation(async (_db, binding) => binding.runId === first.binding.runId && recoveredSession.close.mock.calls.length ? {} as never : null) : undefined;
|
||||
const issueProjection = path === "legacy-plugin" ? vi.spyOn(issueServiceModule, "issueService")
|
||||
.mockReturnValue({ update: vi.fn(async () => null) } as never) : undefined;
|
||||
try {
|
||||
await executePaperclipNativeSession({ db: leaseDb(first), execution: first, runnerInstanceId: "runner-reuse-key",
|
||||
useRunnerd: true, managedGitHub, runnerExecutionTarget: target,
|
||||
useRunnerd: true, managedGitHub, runnerExecutionTarget: target as never,
|
||||
...(portable ? { managedAiAccountIdentity: "fixture-account", managedAiCredentialIdentity: "old-revision" } : {}),
|
||||
runnerEnvironment: { PAPERCLIP_GITHUB_AUTH_MODE: "managed", ...firstEnvironment } });
|
||||
const durableRoot = (await readdir(stateBase, { withFileTypes: true }))
|
||||
.filter(entry => entry.isDirectory() && /^[a-f0-9]{64}$/.test(entry.name)).map(entry => join(stateBase, entry.name))[0]!;
|
||||
const durableIdentity = { runId: first.binding.runId, normalizedSessionId: first.session.normalizedSessionId,
|
||||
runnerInstanceId: "runner-reuse-key", environmentLeaseId: first.binding.executionWorkspaceId };
|
||||
runnerInstanceId: "runner-reuse-key", environmentLeaseId: first.binding.executionWorkspaceId,
|
||||
turnId: "prior-turn", itemId: "prior-item" };
|
||||
await mkdir(join(durableRoot, "control-plane"), { recursive: true });
|
||||
await writeFile(join(durableRoot, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(durableIdentity)));
|
||||
await mkdir(join(durableRoot, "runner"), { recursive: true });
|
||||
await writeFile(join(durableRoot, "runner", "runner-state.json"), JSON.stringify(durableRunnerState(durableIdentity, "ready")));
|
||||
if (path === "adoption") expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
if (path !== "warm") expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] });
|
||||
if (portable) {
|
||||
const directory = join(resolvePaperclipInstanceRoot(), "runtime/paperclip-runner/sessions");
|
||||
const path = join(directory, (await readdir(directory))[0]!);
|
||||
const { snapshot, configDigest, processConfigDigest } = JSON.parse(await readFile(path, "utf8"));
|
||||
await rm(path);
|
||||
await rm(durableRoot, { recursive: true });
|
||||
projection.mockResolvedValueOnce({
|
||||
kind: "recoverable", snapshot,
|
||||
metadata: controllerRecovery.createNativeControllerRecovery(snapshot, { configDigest, processConfigDigest }, null),
|
||||
checkpoint: { restoreSessionDirectory(root: string) {
|
||||
mkdirSync(join(root, "control-plane"), { recursive: true });
|
||||
writeFileSync(join(root, "control-plane/control-plane-state.json"), JSON.stringify(durableControlPlaneState(durableIdentity)));
|
||||
} } as never,
|
||||
});
|
||||
}
|
||||
if (pluginRecovery) state.createBackend.mockReturnValueOnce({
|
||||
kind: "test", recoverSession: async () => ({ recovered: true, session: recoveredSession }),
|
||||
} as never);
|
||||
const pluginTaskDescription = { kind: "description", taskId: "target-task", warmTaskVersion: 1,
|
||||
launchPreparationVersion: 1, projectAuthCheckVersion: 1, releaseResumeVersion: 1,
|
||||
runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } } as const;
|
||||
const portableTarget = { ...target, runner: undefined, leaseId: "target-lease",
|
||||
sandboxLeaseAcquisition: { outcome: path === "legacy-plugin" ? "resumed" : "created", providerLeaseId: "target-task" },
|
||||
runnerTask: vi.fn(async (operation: { kind: string }) => {
|
||||
if (path === "legacy-plugin") {
|
||||
expect(recoveredSession.close).toHaveBeenCalledOnce();
|
||||
expect(operation.kind).toBe("describe");
|
||||
throw new Error("fixture replacement inspection reached");
|
||||
}
|
||||
return pluginTaskDescription;
|
||||
}),
|
||||
};
|
||||
if (pluginRecovery) await writeFile(join(stateBase, "auth.json"),
|
||||
JSON.stringify({ OPENAI_API_KEY: "synthetic-fixture" }), { mode: 0o600 });
|
||||
const leaseOwner = bootstrap ? "bootstrap-recovery-owner" : undefined;
|
||||
const controller = bootstrap ? await import("./native-restart-recovery.js").then(module => module.currentNativeControllerIdentity()) : undefined;
|
||||
const db = leaseDb(second, controller ? { leaseOwner, controllerBootId: controller.bootId,
|
||||
controllerPid: controller.pid, controllerGeneration: 1, leaseExpiresAt: new Date(Date.now() + 60_000) } : {});
|
||||
const select = db.select.bind(db);
|
||||
db.select = ((fields?: Record<string, unknown>) => {
|
||||
if (fields?.status === heartbeatRuns.status && fields?.runnerProfileJson === heartbeatRuns.runnerProfileJson) {
|
||||
return { from: () => ({ where: () => ({ limit: async () =>
|
||||
[{ status: "succeeded", runnerProfileJson: { nativeExecutionInput: first } }] }) }) };
|
||||
}
|
||||
if (pluginRecovery && fields?.metadata === environmentLeases.metadata) {
|
||||
return { from: () => ({ where: async () => [{ providerLeaseId: pluginTaskDescription.taskId, unlaunchedPortable: bootstrap,
|
||||
metadata: ownership ? { portableTaskPreviousLeaseId: "source-lease", portableTaskOwnershipReady: true } : {} }] }) };
|
||||
}
|
||||
return select(fields as never);
|
||||
}) as Db["select"];
|
||||
const continuation = executePaperclipNativeSession({
|
||||
db: { ...leaseDb(second), select: () => ({ from: () => ({ where: () => ({ limit: () =>
|
||||
Promise.resolve([{ status: "succeeded", runnerProfileJson: { nativeExecutionInput: first } }]) }) }) }) } as unknown as Db,
|
||||
execution: second, runnerInstanceId: "runner-reuse-key", useRunnerd: true, managedGitHub, refreshTools,
|
||||
runnerExecutionTarget: target, runnerEnvironment: { PAPERCLIP_GITHUB_AUTH_MODE: "managed", ...secondEnvironment } });
|
||||
db, execution: second, runnerInstanceId: "runner-reuse-key", useRunnerd: true, managedGitHub, refreshTools, toolRefreshOnResume: !unsupportedRefresh,
|
||||
runnerExecutionTarget: (pluginRecovery ? portableTarget : target) as never,
|
||||
...(leaseOwner ? { leaseOwner, restartRecovery: { kind: "bootstrap_incomplete" as const, runId: second.binding.runId,
|
||||
leaseOwner, controllerGeneration: 1, providerAttempt: 0, restartKind: "hard" as const, recoveryRequestId: null,
|
||||
portablePreparationLeaseId: portableTarget.leaseId } } : {}),
|
||||
...(pluginRecovery ? { pluginTaskDescription, runnerIngressAuthorized: true, managedAiCredentialHome: stateBase,
|
||||
managedAiAccountIdentity: "fixture-account", managedAiCredentialIdentity: "current-revision" } : {}),
|
||||
runnerEnvironment: { PAPERCLIP_GITHUB_AUTH_MODE: "managed", ...secondEnvironment } });
|
||||
if (pluginRecovery && unsupportedRefresh) {
|
||||
await expect(continuation).rejects.toThrow("fixture unsupported tool refresh refused");
|
||||
continue;
|
||||
}
|
||||
if (path === "legacy-plugin") {
|
||||
await expect(continuation).rejects.toThrow("fixture replacement inspection reached");
|
||||
expect(portableTarget.runnerTask).toHaveBeenCalledOnce();
|
||||
continue;
|
||||
}
|
||||
if (path === "adoption") await expect(continuation).rejects.toThrow("fixture adoption decision recorded");
|
||||
else await continuation;
|
||||
expect(decisions[path]).toBe(path === "adoption" ? adoptionReuse : reuse);
|
||||
expect(decisions[path]).toBe(portable ? true : path === "adoption" ? adoptionReuse : reuse);
|
||||
if (portable) {
|
||||
expect(portableTarget.runnerTask).not.toHaveBeenCalled();
|
||||
expect(recoveredSession.close).not.toHaveBeenCalled();
|
||||
}
|
||||
if (path === "warm") expect(firstSession.close).toHaveBeenCalledTimes(reuse ? 0 : 1);
|
||||
} finally {
|
||||
projection.mockRestore();
|
||||
restore?.mockRestore();
|
||||
issueProjection?.mockRestore();
|
||||
ownership?.mockRestore();
|
||||
inspected.mockRestore();
|
||||
await closeIdleWarmNativeSessionsForRestart();
|
||||
if (previousStateDirectory === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
||||
@@ -14258,7 +14376,7 @@ describe("native warm session supervision", () => {
|
||||
await rm(stateBase, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
expect(decisions).toEqual({ warm: reuse, adoption: adoptionReuse });
|
||||
expect(decisions).toEqual({ warm: reuse, adoption: adoptionReuse, ...(refreshTools && !unsupportedRefresh ? { portable: true, "portable-bootstrap": true } : {}) });
|
||||
},
|
||||
);
|
||||
|
||||
@@ -17963,8 +18081,9 @@ describe("runnerd provider runtime wiring", () => {
|
||||
} finally { resume.mockRestore(); }
|
||||
});
|
||||
|
||||
it.each(["verified", "wrong-state", "wrong-lease"] as const)("reattaches a surviving plugin task through its owner: %s", async mode => {
|
||||
it.each(["verified", "portable", "wrong-claim", "wrong-state", "wrong-lease"] as const)("reattaches a surviving plugin task through its owner: %s", async mode => {
|
||||
const current = scopedNativeExecution(`plugin-live-${mode}`);
|
||||
const moved = mode === "portable" || mode === "wrong-claim";
|
||||
const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!,
|
||||
runnerInstanceId: "plugin-runner", environmentLeaseId: "same-task-lease", turnId: "admitted-turn", itemId: "admitted-item" };
|
||||
const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId);
|
||||
@@ -17980,24 +18099,35 @@ describe("runnerd provider runtime wiring", () => {
|
||||
runner: { version: "new-artifact", digest: `sha256:${"b".repeat(64)}` } };
|
||||
});
|
||||
state.createBackend.mockClear(); state.createTransport.mockClear();
|
||||
const backend = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
|
||||
runnerIngressAuthorized: true, runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".",
|
||||
leaseId: mode === "wrong-lease" ? "replacement-task" : identity.environmentLeaseId, runnerTask: task },
|
||||
controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-artifact", digest: `sha256:${"a".repeat(64)}` } },
|
||||
restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller", controllerGeneration: 2,
|
||||
providerAttempt: 1, restartKind: "hard", recoveryRequestId: null, remote: { kind: "plugin_task", leaseId: identity.environmentLeaseId } } });
|
||||
if (mode !== "verified") {
|
||||
await expect(backend).rejects.toThrow(mode === "wrong-state" ? "adoption identity mismatch" : "recovery_lease_mismatch");
|
||||
expect(state.createBackend).not.toHaveBeenCalled();
|
||||
return;
|
||||
}
|
||||
await backend;
|
||||
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
|
||||
const options = state.createTransport.mock.calls.at(-1)![0];
|
||||
expect(options).toMatchObject({ prpIdentity: identity, adoptExistingRunner: { pid: null, startedAt: null },
|
||||
runnerArtifact: { version: "admitted-artifact" } });
|
||||
expect(options).toMatchObject({ runnerReconnectGraceMs: 420_000 });
|
||||
expect(task.mock.calls.some(([op]) => ["prepare", "submit", "stop"].includes(op.kind))).toBe(false);
|
||||
const portable = moved ? vi.spyOn(pluginTaskOwnership, "assertAttachedPortableTask")
|
||||
.mockImplementation(async () => {
|
||||
if (mode === "wrong-claim") throw new Error("native_portable_task_ownership_unverified");
|
||||
}) : undefined;
|
||||
const leaseId = moved ? "moved-task-lease" : identity.environmentLeaseId;
|
||||
try {
|
||||
const backend = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
|
||||
runnerIngressAuthorized: true, runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".",
|
||||
leaseId: mode === "wrong-lease" ? "replacement-task" : leaseId, runnerTask: task,
|
||||
...(moved ? { sandboxLeaseAcquisition: { outcome: "resumed" as const, providerLeaseId: "same-task" } } : {}) },
|
||||
controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-artifact", digest: `sha256:${"a".repeat(64)}` } },
|
||||
restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller", controllerGeneration: 2,
|
||||
providerAttempt: 1, restartKind: "hard", recoveryRequestId: null, remote: { kind: "plugin_task", leaseId } } });
|
||||
if (mode !== "verified" && mode !== "portable") {
|
||||
await expect(backend).rejects.toThrow(mode === "wrong-state" ? "adoption identity mismatch" : mode === "wrong-claim" ? "ownership_unverified" : "recovery_lease_mismatch");
|
||||
expect(state.createBackend).not.toHaveBeenCalled();
|
||||
return;
|
||||
}
|
||||
await backend;
|
||||
if (moved) expect(portable).toHaveBeenCalledWith(expect.anything(), {
|
||||
companyId: current.binding.companyId, leaseId, taskId: "same-task", runId: current.binding.runId, identity,
|
||||
});
|
||||
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
|
||||
const options = state.createTransport.mock.calls.at(-1)![0];
|
||||
expect(options).toMatchObject({ prpIdentity: identity, adoptExistingRunner: { pid: null, startedAt: null },
|
||||
runnerArtifact: { version: "admitted-artifact" } });
|
||||
expect(options).toMatchObject({ runnerReconnectGraceMs: 420_000 });
|
||||
expect(task.mock.calls.some(([op]) => ["prepare", "submit", "stop"].includes(op.kind))).toBe(false);
|
||||
} finally { portable?.mockRestore(); }
|
||||
});
|
||||
|
||||
it.each(["codex", "acpx-codex", "grok"] as const)("writes managed %s events through the current warm exporter without per-turn reads", async provider => {
|
||||
@@ -18175,8 +18305,11 @@ describe("runnerd provider runtime wiring", () => {
|
||||
} finally {owner.mockRestore();await rm(home,{recursive:true,force:true});}
|
||||
});
|
||||
|
||||
it.each((["codex", "claude"] as const).flatMap(provider => [true, false].map(reuseProcess => [provider, reuseProcess] as const)))("reconnects task-only %s warm recovery under the saved identity before reuse or retirement: %s", async (provider, reuseProcess) => {
|
||||
const current=scopedNativeExecution(`plugin-restart-${provider}-${reuseProcess}`);
|
||||
it.each([
|
||||
...(["codex", "claude"] as const).flatMap(provider => [true, false].map(reuseProcess => [provider, reuseProcess, false] as const)),
|
||||
["codex", true, true] as const,
|
||||
])("reconnects task-only %s warm recovery under the saved identity before reuse or retirement: %s (portable=%s)", async (provider, reuseProcess, portable) => {
|
||||
const current=scopedNativeExecution(`plugin-restart-${provider}-${reuseProcess}-${portable}`);
|
||||
if(provider === "claude") current.provider={kind:"acpx",agent:"claude",model:"fixture",permissionMode:"deny-all"} as never;
|
||||
current.session.lifecyclePolicy={mode:"warm",idleTimeoutMs:300000};
|
||||
const identity={runnerInstanceId:"old-runner",environmentLeaseId:"old-lease",runId:"old-run",
|
||||
@@ -18184,22 +18317,29 @@ describe("runnerd provider runtime wiring", () => {
|
||||
const home=await mkdtemp(join(tmpdir(),"plugin-restart-account-"));
|
||||
await writeFile(join(home,"auth.json"),JSON.stringify({OPENAI_API_KEY:"synthetic-fixture"}),{mode:0o600});
|
||||
const owner=vi.spyOn(pluginTaskRuntime,"createPluginRunnerTaskOwner");
|
||||
const register = vi.spyOn(pluginCredentialWriteback, "registerPluginTaskCredentialHandoff").mockResolvedValue(undefined);
|
||||
const binding = { grantId: randomUUID(), secretId: randomUUID(), sessionEpoch: 1, versionSelector: "latest" as const };
|
||||
const task=vi.fn(async()=>({kind:"description",taskId:"saved-task",launchPreparationVersion:1,projectAuthCheckVersion:1,
|
||||
warmTaskVersion:1,runner:{version:"fixture",digest:`sha256:${"a".repeat(64)}`}}));
|
||||
state.createBackend.mockClear();state.createTransport.mockClear();
|
||||
try {
|
||||
await createRunnerdBackend({db:leaseDb(current),execution:current,runnerInstanceId:"new-runner",runnerIngressAuthorized:true,
|
||||
runnerEnvironment:{ANTHROPIC_API_KEY:"synthetic-fixture"},managedAiCredentialHome:home,managedAiCredentialIdentity:reuseProcess ? "old-revision" : "new-revision",managedAiAccountIdentity:"account",
|
||||
managedAiCredentialWriteback:{} as never,
|
||||
managedAiCredentialWriteback: binding,
|
||||
remoteWarmRecovery:{identity,reuseProcess,previousExecution:current,checkpoint:{} as never,
|
||||
remote:{providerLeaseId:"saved-task",remoteCwd:"."},revalidate:async()=>{}},
|
||||
remote:{providerLeaseId:"saved-task",remoteCwd:"."},revalidate:async()=>{},
|
||||
...(portable ? { portableSourceTaskId: "old-task" } : {})},
|
||||
runnerExecutionTarget:{kind:"remote",transport:"sandbox",leaseId:"new-lease",remoteCwd:".",runnerTask:task,
|
||||
sandboxLeaseAcquisition:{outcome:"resumed",providerLeaseId:"saved-task"}}});
|
||||
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
|
||||
expect(state.createTransport.mock.calls.at(-1)![0].prpIdentity).toEqual(identity);
|
||||
expect(owner.mock.calls.at(-1)![0].adoptIdentity).toEqual(identity);
|
||||
expect(task.mock.calls).toEqual([[{kind:"describe"}]]);
|
||||
} finally {owner.mockRestore();await rm(home,{recursive:true,force:true});}
|
||||
if (portable) expect(register).toHaveBeenCalledExactlyOnceWith(expect.anything(), {
|
||||
companyId: current.binding.companyId, leaseId: "new-lease",
|
||||
}, "account", binding);
|
||||
else expect(register).not.toHaveBeenCalled();
|
||||
} finally {register.mockRestore(); owner.mockRestore();await rm(home,{recursive:true,force:true});}
|
||||
});
|
||||
|
||||
it.each((["codex", "claude"] as const).flatMap(provider => [
|
||||
|
||||
@@ -54,7 +54,8 @@ import { writeExecutionControl } from "../execution-control-notifications.js";
|
||||
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
|
||||
import { createRunUsageRecorder } from "../usage-receipts.js";
|
||||
import { syncPluginTaskCredentialHandoff, registerPluginTaskCredentialHandoff } from "../plugin-task-credential-writeback.js";
|
||||
import { resolvePluginTaskResume } from "../environment-task-runtime.js";
|
||||
import { resolvePluginTaskResume, type PluginTaskDescription } from "../environment-task-runtime.js";
|
||||
import { assertAttachedPortableTask, readyPortableTaskSource, unlaunchedPortableTaskLease } from "../plugin-task-ownership.js";
|
||||
import { environmentTaskOperationSchema, environmentTaskResultSchema } from "@paperclipai/plugin-sdk";
|
||||
import { preparePluginRunnerCredentials } from "./plugin-runner-credentials.js";
|
||||
import { createPluginRunnerTaskOwner, verifyPluginRunnerTaskReattachment } from "./plugin-runner-task.js";
|
||||
@@ -278,7 +279,7 @@ import {
|
||||
renderNativeRunnerStagedAttachmentPrompt,
|
||||
stageNativeRunnerWakeAttachments,
|
||||
} from "./native-runner-file-handoff.js";
|
||||
import { NATIVE_PROVIDER_SIDE_EFFECT_PENDING, NATIVE_PROVIDER_STATE_INCOMPATIBLE, nativeRuntimeContractForProvider, nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
|
||||
import { NATIVE_PROVIDER_SIDE_EFFECT_PENDING, NATIVE_PROVIDER_STATE_INCOMPATIBLE, nativeRuntimeContextsMatch, nativeRuntimeContractForProvider, nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
|
||||
import { clearRunProviderStateUnsaved, markAcceptedRunProviderStateUnsaved } from "./native-provider-state-unsaved.js";
|
||||
import { verifyRetainedMaintenanceNoLaunch } from "./native-maintenance-no-launch.js";
|
||||
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
|
||||
@@ -6247,6 +6248,29 @@ async function recordPriorOwnerFenceRefusal(input: { db: Db; execution: NativeEx
|
||||
}).catch(() => undefined);
|
||||
}
|
||||
|
||||
async function assertPluginTaskReattachmentBinding(input: {
|
||||
db: Db;
|
||||
execution: NativeExecutionInput;
|
||||
restartRecovery?: NativeRestartRecoveryClaim;
|
||||
}, target: Extract<AdapterExecutionTarget, { kind: "remote" }>, identity: Record<string, unknown> | null | undefined) {
|
||||
const claim = input.restartRecovery;
|
||||
const leaseId = target.leaseId;
|
||||
if (claim?.kind !== "reattach_remote_runner" || claim.remote.kind !== "plugin_task" ||
|
||||
claim.runId !== input.execution.binding.runId || !leaseId || claim.remote.leaseId !== leaseId ||
|
||||
!identity || !durableIdentityMatchesExecution(identity, input.execution) ||
|
||||
typeof identity.turnId !== "string" || !identity.turnId || typeof identity.itemId !== "string" || !identity.itemId) {
|
||||
throw new Error("native_plugin_recovery_lease_mismatch");
|
||||
}
|
||||
const taskId = target.transport === "sandbox" ? target.sandboxLeaseAcquisition?.providerLeaseId : undefined;
|
||||
if (identity.environmentLeaseId !== leaseId) {
|
||||
if (!taskId) throw new Error("native_plugin_recovery_lease_mismatch");
|
||||
await assertAttachedPortableTask(input.db, {
|
||||
companyId: input.execution.binding.companyId, leaseId, taskId, runId: claim.runId, identity,
|
||||
});
|
||||
}
|
||||
return taskId;
|
||||
}
|
||||
|
||||
async function migrateRunnerdStateRootForExecution(input: {
|
||||
db: Db;
|
||||
execution: NativeExecutionInput;
|
||||
@@ -6370,12 +6394,12 @@ async function migrateRunnerdStateRootForExecution(input: {
|
||||
if (input.restartRecovery?.kind === "reattach_remote_runner") {
|
||||
if (input.restartRecovery.remote.kind === "plugin_task") {
|
||||
const target = input.runnerExecutionTarget;
|
||||
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runnerTask
|
||||
|| target.leaseId !== input.restartRecovery.remote.leaseId || identity.environmentLeaseId !== target.leaseId
|
||||
|| !durableIdentityMatchesExecution(identity, input.execution)) {
|
||||
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runnerTask) {
|
||||
throw new Error("native_plugin_recovery_lease_mismatch");
|
||||
}
|
||||
await verifyPluginRunnerTaskReattachment(target.runnerTask, identity);
|
||||
const taskId = await assertPluginTaskReattachmentBinding(input, target, identity);
|
||||
await verifyPluginRunnerTaskReattachment(operation => target.runnerTask!(operation,
|
||||
taskId ? { expectedTaskId: taskId } : undefined), identity);
|
||||
return;
|
||||
}
|
||||
try {
|
||||
@@ -7281,6 +7305,19 @@ function nativeSessionConfigDigest(
|
||||
.digest("hex")}`;
|
||||
}
|
||||
|
||||
function matchesRemoteWarmExecution(
|
||||
previous: NativeExecutionInput,
|
||||
current: NativeExecutionInput,
|
||||
toolRefreshOnResume?: boolean,
|
||||
): boolean {
|
||||
if (nativeSessionScopeKey(previous) !== nativeSessionScopeKey(current)) return false;
|
||||
const currentDigest = nativeSessionConfigDigest(current, "remote");
|
||||
if (nativeSessionConfigDigest(previous, "remote") === currentDigest) return true;
|
||||
return toolRefreshOnResume === true && "runtimeContext" in previous && "runtimeContext" in current &&
|
||||
nativeRuntimeContextsMatch(previous.runtimeContext, current.runtimeContext, toolRefreshOnResume) &&
|
||||
nativeSessionConfigDigest({ ...previous, runtimeContext: current.runtimeContext }, "remote") === currentDigest;
|
||||
}
|
||||
|
||||
export function nativeControllerMigrationDescriptor(execution: NativeExecutionInput, recordedToolContractFingerprint: unknown) {
|
||||
if (typeof recordedToolContractFingerprint !== "string" || !/^sha256:[a-f0-9]{64}$/.test(recordedToolContractFingerprint)) {
|
||||
throw new Error("native_controller_migration_tool_contract_missing");
|
||||
@@ -10203,6 +10240,7 @@ export async function executePaperclipNativeSession(input: {
|
||||
getFreshSessionHandoff?: () => Promise<string | null>;
|
||||
/** Retire a retained transport so recovery can replace provider tool declarations. */
|
||||
refreshTools?: boolean;
|
||||
toolRefreshOnResume?: boolean;
|
||||
db: Db;
|
||||
execution: NativeExecutionInput;
|
||||
/** Private server preparation proof; never persisted or accepted from a tool. */
|
||||
@@ -10259,6 +10297,7 @@ export async function executePaperclipNativeSession(input: {
|
||||
managedAiCredentialWriteback?: ManagedAiCredentialWritebackBinding;
|
||||
billingIdentity?: Pick<AdapterExecutionResult, "provider" | "biller" | "billingType">;
|
||||
onTaskDispatched?: () => Promise<void> | void;
|
||||
pluginTaskDescription?: PluginTaskDescription;
|
||||
runnerExecutionTarget?: AdapterExecutionTarget | null;
|
||||
/** Resolved per-run authorization; not an independent instance setting. */
|
||||
runnerIngressAuthorized?: boolean;
|
||||
@@ -10657,7 +10696,22 @@ async function executePaperclipNativeSessionAttempt(
|
||||
const stagedRestart = input.restartRecovery?.kind === "reattach_remote_runner"
|
||||
? await verifyStagedAttachmentRestart(input) : null;
|
||||
const timeController = createRunPreparationTiming(input.execution.binding.runId, "native.controller");
|
||||
if (input.useRunnerd) await timeController("warm_restore", () => restoreWarmRunnerdController(input));
|
||||
const target = input.runnerExecutionTarget;
|
||||
const portableTarget = target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask ? target : undefined;
|
||||
const companyId = input.execution.binding.companyId;
|
||||
const portablePreparationLeaseId = input.restartRecovery?.kind === "bootstrap_incomplete"
|
||||
? input.restartRecovery.portablePreparationLeaseId : undefined;
|
||||
const targetLease = portableTarget?.leaseId ? (await input.db.select({
|
||||
metadata: environmentLeases.metadata, providerLeaseId: environmentLeases.providerLeaseId,
|
||||
...(portablePreparationLeaseId ? { unlaunchedPortable: unlaunchedPortableTaskLease() } : {}),
|
||||
}).from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId), eq(environmentLeases.id, portableTarget.leaseId))))[0] : undefined;
|
||||
if (portablePreparationLeaseId && (portableTarget?.leaseId !== portablePreparationLeaseId || targetLease?.unlaunchedPortable !== true)) {
|
||||
throw remoteRecoveryLeaseMismatch("runner_verification", "plugin_task");
|
||||
}
|
||||
const replayPortablePreparation = portablePreparationLeaseId !== undefined;
|
||||
const canRecoverWarmOwner = !input.restartRecovery || replayPortablePreparation;
|
||||
if (input.useRunnerd) await timeController("warm_restore", () => restoreWarmRunnerdController(
|
||||
replayPortablePreparation ? { ...input, restartRecovery: undefined } : input));
|
||||
const identityRedactor = createAgentIdentityRedactor(input.runnerEnvironment?.PAPERCLIP_AGENT_PRIVATE_KEY);
|
||||
const redactIdentityText = (text: string) => redactSensitiveText(
|
||||
identityRedactor.chunk("diagnostic", text) + identityRedactor.finish("diagnostic"),
|
||||
@@ -10757,29 +10811,51 @@ async function executePaperclipNativeSessionAttempt(
|
||||
});
|
||||
let retainedTransition: VerifiedWarmTransitionBinding | undefined;
|
||||
let remoteWarmRecovery: RemoteWarmSessionRecovery | undefined;
|
||||
let pluginTaskDescription: Extract<ReturnType<typeof environmentTaskResultSchema.parse>,{kind:"description"}> | undefined;
|
||||
let pluginTaskDescription: PluginTaskDescription | undefined;
|
||||
let portableSource: Awaited<ReturnType<typeof readyPortableTaskSource>>;
|
||||
if (portableTarget?.runnerTask && portableTarget.leaseId) {
|
||||
if (input.pluginTaskDescription && input.pluginTaskDescription.taskId === targetLease?.providerLeaseId) {
|
||||
pluginTaskDescription = input.pluginTaskDescription;
|
||||
}
|
||||
if (targetLease?.metadata?.portableTaskPreviousLeaseId || targetLease?.metadata?.portableTaskPredecessor) {
|
||||
const description = targetLease.metadata.portableTaskOwnershipReady === true && pluginTaskDescription
|
||||
? pluginTaskDescription : environmentTaskResultSchema.parse(await portableTarget.runnerTask({ kind: "describe" }));
|
||||
if (description.kind !== "description") throw new Error("Plugin Runner description unavailable");
|
||||
pluginTaskDescription = description;
|
||||
portableSource = await readyPortableTaskSource(input.db, companyId, portableTarget.leaseId,
|
||||
readRunnerdDurableIdentity(scopedRunnerdStateRoot(input.execution)), {
|
||||
attachedRunId: input.execution.binding.runId,
|
||||
selectedSessionId: nativeSessionKey(input.execution),
|
||||
accountIdentity: input.managedAiAccountIdentity ?? input.managedAiCredentialIdentity,
|
||||
});
|
||||
}
|
||||
}
|
||||
const cachedReleasedOwner = warmNativeSessions.get(nativeSessionScopeKey(input.execution));
|
||||
if (cachedReleasedOwner?.released) {
|
||||
const proof = releasedWarmRunner(readWarmNativeEnvelope(input.execution));
|
||||
const proof = portableSource ? undefined : releasedWarmRunner(readWarmNativeEnvelope(input.execution));
|
||||
const identity = readRunnerdDurableIdentity(scopedRunnerdStateRoot(input.execution));
|
||||
const target = input.runnerExecutionTarget;
|
||||
if (proof && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId
|
||||
&& target.sandboxLeaseAcquisition?.outcome === "created" && proof.taskId !== target.sandboxLeaseAcquisition.providerLeaseId) {
|
||||
const source = portableSource ?? (proof && { taskId: proof.taskId, identity: proof.source });
|
||||
if (source && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId
|
||||
&& (portableSource || target.sandboxLeaseAcquisition?.outcome === "created")
|
||||
&& source.taskId !== target.sandboxLeaseAcquisition?.providerLeaseId) {
|
||||
const ownInstructionPreparation = cachedReleasedOwner.instructionPreparationRunId === input.execution.binding.runId
|
||||
&& input.instructionWorkingCopy?.runId === input.execution.binding.runId;
|
||||
if ((cachedReleasedOwner.busy && !ownInstructionPreparation)
|
||||
|| cachedReleasedOwner.preparingRunId && cachedReleasedOwner.preparingRunId !== input.execution.binding.runId
|
||||
|| cachedReleasedOwner.instructionPreparationRunId && !ownInstructionPreparation
|
||||
|| cachedReleasedOwner.closeOnReleaseReason) throw new Error("native_session_supervisor_busy");
|
||||
if (proof.taskId !== cachedReleasedOwner.pluginTaskId || proof.scopeDigest !== releasedWarmScope(input)
|
||||
|| canonicalJson(proof.source) !== canonicalJson(identity)) throw new Error("native_session_released_owner_scope_mismatch");
|
||||
if (source.taskId !== cachedReleasedOwner.pluginTaskId || cachedReleasedOwner.runId !== identity?.runId ||
|
||||
(proof && (proof.scopeDigest !== releasedWarmScope(input) || canonicalJson(proof.source) !== canonicalJson(identity)))) {
|
||||
throw new Error("native_session_released_owner_scope_mismatch");
|
||||
}
|
||||
const wasBusy = cachedReleasedOwner.busy;
|
||||
cachedReleasedOwner.busy = true;
|
||||
try {
|
||||
// Only physical stop releases the cached controller; ordinary checkpoint admission still proves continuation.
|
||||
const resumeFrom = await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
|
||||
leaseId: target.leaseId, priorIdentity: proof.source, inspectCheckpoint: true });
|
||||
if (resumeFrom !== proof.taskId) throw new Error("native_session_released_owner_scope_mismatch");
|
||||
// Ordinary replacement still needs a stop receipt for the old task.
|
||||
const resumeFrom = portableSource?.taskId ?? await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
|
||||
leaseId: target.leaseId, priorIdentity: source.identity, inspectCheckpoint: true });
|
||||
if (resumeFrom !== source.taskId) throw new Error("native_session_released_owner_scope_mismatch");
|
||||
try {
|
||||
await detachReleasedWarmNativeSession(cachedReleasedOwner);
|
||||
} catch (error) {
|
||||
@@ -10790,7 +10866,7 @@ async function executePaperclipNativeSessionAttempt(
|
||||
} finally { cachedReleasedOwner.busy = wasBusy; }
|
||||
} else requireReleasedWarmScope(input, proof, identity);
|
||||
}
|
||||
if (input.useRunnerd && !input.restartRecovery && !input.backend &&
|
||||
if (input.useRunnerd && canRecoverWarmOwner && !input.backend &&
|
||||
input.runnerExecutionTarget?.kind === "remote" &&
|
||||
input.execution.session.lifecyclePolicy.mode === "warm" &&
|
||||
!idleWarmNativeSessionOwnerRunId(input)) {
|
||||
@@ -10800,10 +10876,12 @@ async function executePaperclipNativeSessionAttempt(
|
||||
if (taskTarget.transport === "sandbox" && taskTarget.runnerTask && taskTarget.leaseId &&
|
||||
durableIdentityMatchesSession(identity,input.execution) && identity.runId !== input.execution.binding.runId) {
|
||||
// Inspection can replace a stopped retained task before deciding between adoption and checkpoint recovery.
|
||||
const description = environmentTaskResultSchema.parse(await taskTarget.runnerTask({kind:"describe"}));
|
||||
if (description.kind !== "description") throw new Error("Plugin Runner description unavailable");
|
||||
pluginTaskDescription = description;
|
||||
if (taskTarget.sandboxLeaseAcquisition?.outcome !== "resumed") {
|
||||
if (!pluginTaskDescription || pluginTaskDescription.taskId !== taskTarget.sandboxLeaseAcquisition?.providerLeaseId) {
|
||||
const description = environmentTaskResultSchema.parse(await taskTarget.runnerTask({ kind: "describe" }));
|
||||
if (description.kind !== "description") throw new Error("Plugin Runner description unavailable");
|
||||
pluginTaskDescription = description;
|
||||
}
|
||||
if (taskTarget.sandboxLeaseAcquisition?.outcome !== "resumed" && !portableSource) {
|
||||
await recoverPluginRunnerCheckpoint(input, root);
|
||||
}
|
||||
}
|
||||
@@ -10817,22 +10895,23 @@ async function executePaperclipNativeSessionAttempt(
|
||||
}
|
||||
if (durableIdentityMatchesSession(identity, input.execution) && identity.runId !== input.execution.binding.runId &&
|
||||
runnerdAuthorityLifecycleWithVerifiedBackup({ root, identity, execution: input.execution, allowVerifiedBackup: true }) !== "suspended") {
|
||||
let releasedSource: DurableRecoveryIdentity | undefined;
|
||||
let releasedSource = portableSource?.identity;
|
||||
remoteWarmRecovery = await inspectRemoteWarmSessionRecovery({
|
||||
db: input.db, execution: input.execution, target: input.runnerExecutionTarget, identity,
|
||||
portableSourceTaskId: portableSource?.taskId,
|
||||
readControlState: () => record(JSON.parse(readBoundedNativeFile(
|
||||
resolve(root, "control-plane", "control-plane-state.json"), NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
|
||||
"runner_durable_identity_too_large").toString("utf8"))),
|
||||
matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
|
||||
nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(input.execution, "remote"),
|
||||
matchesExecution: previous => matchesRemoteWarmExecution(previous, input.execution, input.toolRefreshOnResume),
|
||||
loadCheckpoint: previous => {
|
||||
const proof = releasedWarmRunner(readWarmNativeEnvelope(previous));
|
||||
if (proof) {
|
||||
releasedSource = requireReleasedWarmScope(input, proof, identity);
|
||||
if (!portableSource) {
|
||||
const proof = releasedWarmRunner(readWarmNativeEnvelope(previous));
|
||||
if (proof) releasedSource = requireReleasedWarmScope(input, proof, identity);
|
||||
}
|
||||
return loadWarmNativeCheckpoint(previous, nativeSessionConfigDigest(previous, "remote"), "remote");
|
||||
},
|
||||
canReuseProcess: previous => !needsCoreGitHubAccess(input) && canReuseNativeProcess(input,
|
||||
canReuseProcess: previous => !portableSource && !needsCoreGitHubAccess(input) &&
|
||||
nativeSessionConfigDigest(previous, "remote") === nativeSessionConfigDigest(input.execution, "remote") && canReuseNativeProcess(input,
|
||||
record(JSON.parse(readBoundedNativeFile(nativeSessionCheckpointPath(previous),
|
||||
NATIVE_WARM_CHECKPOINT_MAX_BYTES, "native_session_supervisor_checkpoint_too_large").toString("utf8"))).processConfigDigest),
|
||||
}) ?? undefined;
|
||||
@@ -11061,7 +11140,7 @@ async function executePaperclipNativeSessionAttempt(
|
||||
: coordinator.controllerGeneration + 1;
|
||||
const target = input.runnerExecutionTarget;
|
||||
const retained = warmNativeSessions.get(nativeSessionScopeKey(input.execution));
|
||||
if (!input.restartRecovery && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId &&
|
||||
if (canRecoverWarmOwner && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask && target.leaseId &&
|
||||
((retained?.pluginTaskId && idleWarmNativeSessionOwnerRunId(input) !== undefined) || remoteWarmRecovery?.releasedSource) &&
|
||||
(input.execution.provider.kind === "codex" || (input.execution.provider.kind === "acpx" && input.execution.provider.agent === "claude"))) {
|
||||
claimedStartupContext = new NativeExecutionContext(input.execution, Math.min(executionLeaseTtlMs,
|
||||
@@ -11801,7 +11880,7 @@ async function executePaperclipNativeSessionAttempt(
|
||||
executionContext = undefined;
|
||||
startupForAdmission!.executionContext = undefined;
|
||||
};
|
||||
if (warmSessionId !== null && warmConfigDigest !== null && !input.restartRecovery) {
|
||||
if (warmSessionId !== null && warmConfigDigest !== null && canRecoverWarmOwner) {
|
||||
const registered = warmNativeSessions.get(warmSessionId);
|
||||
const entry = registered && forgetUnretiredOwnerOnOtherSandbox(warmSessionId, registered, input.runnerExecutionTarget)
|
||||
? undefined : registered;
|
||||
@@ -12088,11 +12167,15 @@ async function executePaperclipNativeSessionAttempt(
|
||||
signal: executionContext?.signal ?? new AbortController().signal,
|
||||
});
|
||||
if (!recovered?.recovered || !recovered.session) throw new Error("runner_state_identity_mismatch: remote_warm_session_unproven");
|
||||
if (remoteWarmRecovery.reuseProcess) {
|
||||
const movedSession = remoteWarmRecovery.portableSourceTaskId !== undefined;
|
||||
// A portable resume already starts a new Runner. Its run attachment
|
||||
// prepares the current tools and runtime on the verified target.
|
||||
if (movedSession || remoteWarmRecovery.reuseProcess) {
|
||||
existingWarmSession = recovered.session;
|
||||
await emitStartupReuse(input.onEvent, "runner", { state: "reused", reason: "verified_reuse" });
|
||||
await emitStartupReuse(input.onEvent, "runner", movedSession
|
||||
? { state: "fresh", reason: "replacement" } : { state: "reused", reason: "verified_reuse" });
|
||||
await emitStartupReuse(input.onEvent, "provider_session", { state: "reused", reason: "verified_reuse" });
|
||||
trace.setBudgetStart("warm");
|
||||
if (!movedSession) trace.setBudgetStart("warm");
|
||||
persistedWarmSession = null;
|
||||
} else {
|
||||
await recovered.session.close({ reason: "recovered warm session requires current process configuration" });
|
||||
@@ -12104,12 +12187,22 @@ async function executePaperclipNativeSessionAttempt(
|
||||
remoteWarmRecovery.previousExecution.binding, scopedRunnerdStateRoot(input.execution),
|
||||
nativeSessionKey(input.execution));
|
||||
if (!suspendedController) throw new Error("native_runner_checkpoint_missing_after_suspension");
|
||||
persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution,
|
||||
nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote"), await recovered.session.snapshot());
|
||||
persistedWarmSession = loadWarmNativeCheckpoint(input.execution, warmConfigDigest!, "remote");
|
||||
runnerdBackend = await createRunnerdBackend({ ...input, onDiagnosticLease: ownDiagnosticLease, providerPackAdmissionSource, recoveryPurpose: claimedStopCleanup ? "stop_cleanup" : "execution", onAccountingEvent: producerUsage?.capture, runnerInstanceId: effectiveRunnerInstanceId,
|
||||
const verifiedConfigDigest = nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote");
|
||||
persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution, verifiedConfigDigest, await recovered.session.snapshot());
|
||||
persistedWarmSession = loadWarmNativeCheckpoint(input.execution, verifiedConfigDigest, "remote");
|
||||
runnerdBackend = await createRunnerdBackend({
|
||||
...input,
|
||||
pluginTaskDescription: undefined,
|
||||
onDiagnosticLease: ownDiagnosticLease,
|
||||
providerPackAdmissionSource,
|
||||
recoveryPurpose: claimedStopCleanup ? "stop_cleanup" : "execution",
|
||||
onAccountingEvent: producerUsage?.capture,
|
||||
runnerInstanceId: effectiveRunnerInstanceId,
|
||||
controllerPersistence: input.runnerExecutionTarget?.kind === "remote" ? { bootId: controller.bootId } : undefined,
|
||||
durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId, trace, toolTrace });
|
||||
durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId,
|
||||
trace,
|
||||
toolTrace,
|
||||
});
|
||||
}
|
||||
}
|
||||
const readRemoteCleanupLease = async () => input.runnerExecutionTarget?.kind === "remote" &&
|
||||
@@ -15710,6 +15803,7 @@ export function remoteCheckpointIncompleteFailure(
|
||||
export async function createRunnerdBackend(input: {
|
||||
executionContext?: NativeExecutionContext;
|
||||
casesEnabled?: boolean;
|
||||
toolRefreshOnResume?: boolean;
|
||||
db: Db;
|
||||
providerPackAdmissionSource?: NativeProviderPackAdmissionSource;
|
||||
controllerPersistence?: { bootId: string; runnerArtifact?: { version: string; digest: string } };
|
||||
@@ -15724,7 +15818,7 @@ export async function createRunnerdBackend(input: {
|
||||
remoteWarmRecovery?: Omit<RemoteWarmSessionRecovery, "revalidate"> & { revalidate: () => Promise<void> };
|
||||
retainedPluginTaskOwner?: ReturnType<typeof createPluginRunnerTaskOwner>;
|
||||
beginRetainedRunAttachment?: () => void;
|
||||
pluginTaskDescription?: Extract<ReturnType<typeof environmentTaskResultSchema.parse>,{kind:"description"}>;
|
||||
pluginTaskDescription?: PluginTaskDescription;
|
||||
onSpawn?: (meta: {
|
||||
pid: number;
|
||||
processGroupId: number | null;
|
||||
@@ -16128,8 +16222,10 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
const recoveryUsage = input.onAccountingEvent && recoveryUsageIdentity
|
||||
? createNativeRecoveryUsageDelivery({
|
||||
db: input.db, execution: input.execution, sourceIdentity: { ...recoveryUsageIdentity },
|
||||
matchesExecution: previous => nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
|
||||
nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind),
|
||||
matchesExecution: previous => input.remoteWarmRecovery
|
||||
? matchesRemoteWarmExecution(previous, input.execution, input.toolRefreshOnResume)
|
||||
: nativeSessionScopeKey(previous) === nativeSessionScopeKey(input.execution) &&
|
||||
nativeSessionConfigDigest(previous, target.kind) === nativeSessionConfigDigest(input.execution, target.kind),
|
||||
})
|
||||
: undefined;
|
||||
const accountingTransportOptions = {
|
||||
@@ -16185,15 +16281,19 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
const { root, durableIdentity, durableBinding } = durableState ??= readDurableState();
|
||||
const provider = input.execution.provider;
|
||||
if (provider.kind !== "codex" && provider.kind !== "opencode" && provider.kind !== "acpx") throw new Error("Plugin Runner tasks require a workspace harness");
|
||||
if (input.restartRecovery?.kind === "reattach_remote_runner") {
|
||||
await assertPluginTaskReattachmentBinding(input, remoteTarget, durableBinding ? durableIdentity : null);
|
||||
}
|
||||
let priorDescription = input.pluginTaskDescription?.taskId === remoteTarget.sandboxLeaseAcquisition?.providerLeaseId
|
||||
? input.pluginTaskDescription : undefined;
|
||||
if (remoteTarget.sandboxLeaseAcquisition?.outcome === "resumed" && !durableBinding && !input.retainedPluginTaskOwner && !input.remoteWarmRecovery) {
|
||||
// A new logical session (including fresh retries) cannot inherit the old process authority.
|
||||
await pluginTask({kind:"stop"});
|
||||
const stopped=environmentTaskResultSchema.parse(await pluginTask({kind:"status"}));
|
||||
if(stopped.kind!=="status" || stopped.executionStopped!==true) throw new Error("Plugin Runner termination is unconfirmed; reconcile the saved task");
|
||||
await pluginTask({kind:"complete"});
|
||||
priorDescription = undefined;
|
||||
}
|
||||
const priorDescription = input.pluginTaskDescription?.taskId === remoteTarget.sandboxLeaseAcquisition?.providerLeaseId
|
||||
? input.pluginTaskDescription : undefined;
|
||||
const description = priorDescription ?? environmentTaskResultSchema.parse(await pluginTask({ kind: "describe" }));
|
||||
if (description.kind !== "description" || description.launchPreparationVersion !== 1 || description.projectAuthCheckVersion !== 1) throw new Error("Plugin Runner launch preparation v1 unavailable");
|
||||
const saved = durableBinding ? durableIdentity as RunnerdDurableIdentity : undefined;
|
||||
@@ -16212,9 +16312,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
normalizedSessionId: input.remoteWarmRecovery.identity.normalizedSessionId,
|
||||
turnId: input.remoteWarmRecovery.identity.turnId, itemId: input.remoteWarmRecovery.identity.itemId,
|
||||
} : undefined;
|
||||
if (input.restartRecovery?.kind === "reattach_remote_runner" && (!prior || !prior.turnId || !prior.itemId || prior.environmentLeaseId !== remoteTarget.leaseId)) {
|
||||
throw new Error("native_plugin_recovery_lease_mismatch");
|
||||
}
|
||||
|
||||
const resumeFrom = !retainedOwner && !adopting && prior && prior.environmentLeaseId !== remoteTarget.leaseId
|
||||
? await resolvePluginTaskResume(input.db, { companyId: input.execution.binding.companyId,
|
||||
leaseId: remoteTarget.leaseId!, priorIdentity: prior }) : undefined;
|
||||
@@ -16224,7 +16322,8 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
home: input.managedAiCredentialHome, environment: input.runnerEnvironment ?? {},
|
||||
});
|
||||
if (credentials.needsWriteback && !input.managedAiCredentialWriteback) throw new Error("Plugin Runner credential writeback authority unavailable");
|
||||
if (!adopting && input.managedAiCredentialWriteback && credentials.accountIdentity) {
|
||||
// Same-lease recovery already has a writeback binding. A portable move creates a new lease.
|
||||
if ((!adopting || input.remoteWarmRecovery?.portableSourceTaskId) && input.managedAiCredentialWriteback && credentials.accountIdentity) {
|
||||
if (!remoteTarget.leaseId) throw new Error("Plugin Runner credential lease unavailable");
|
||||
await registerPluginTaskCredentialHandoff(input.db, { companyId: input.execution.binding.companyId, leaseId: remoteTarget.leaseId },
|
||||
credentials.accountIdentity, input.managedAiCredentialWriteback);
|
||||
@@ -16251,7 +16350,8 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
releaseResumeVersion: description.releaseResumeVersion,
|
||||
authorityRunId: description.releaseResumeVersion === 1 ? input.execution.binding.runId : undefined,
|
||||
releasedSource: input.remoteWarmRecovery?.releasedSource,
|
||||
onResumed: async source => updateReleasedWarmRunner(input.execution, source, description.taskId, null),
|
||||
sessionHandoff: !!input.remoteWarmRecovery?.portableSourceTaskId,
|
||||
onResumed: async source => updateReleasedWarmRunner(input.execution, source, input.remoteWarmRecovery?.portableSourceTaskId ?? description.taskId, null),
|
||||
onRetired: async source => updateReleasedWarmRunner(input.execution, source, description.taskId, null),
|
||||
adoptIdentity: adoptionIdentity ?? (adopting && prior ? {
|
||||
runnerInstanceId: prior.runnerInstanceId, environmentLeaseId: prior.environmentLeaseId,
|
||||
|
||||
@@ -8,7 +8,7 @@ import type {
|
||||
NativeExecutionInput,
|
||||
PersistedNativeSession,
|
||||
} from "../../vendor/paperclip-runner/index.js";
|
||||
import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
|
||||
import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput, type NativeRuntimeContextSnapshot } from "../../vendor/paperclip-runner/index.js";
|
||||
|
||||
export type NativeToolExecutionTargetKind = "local" | "remote";
|
||||
|
||||
@@ -612,6 +612,15 @@ export function buildNativeExecutionWithCheckpoint(input: {
|
||||
};
|
||||
}
|
||||
|
||||
export function nativeRuntimeContextsMatch(
|
||||
previous: NativeRuntimeContextSnapshot,
|
||||
current: NativeRuntimeContextSnapshot,
|
||||
toolRefreshOnResume?: boolean,
|
||||
): boolean {
|
||||
return previous.aggregateDigest === current.aggregateDigest ||
|
||||
(toolRefreshOnResume === true && canonicalNativeRuntimeContextDigest({ ...previous, mcp: current.mcp }) === current.aggregateDigest);
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebind a completed prior run's provider checkpoint to a new heartbeat run.
|
||||
* The provider/driver session identity is retained, while every per-turn and
|
||||
@@ -697,12 +706,7 @@ export function rebindNativeSessionCheckpoint(input: {
|
||||
("homeMode" in previousExecution ? previousExecution.homeMode : "isolated") !== ("homeMode" in current ? current.homeMode : "isolated") ||
|
||||
("runtimeContext" in previousExecution &&
|
||||
"runtimeContext" in current &&
|
||||
previousExecution.runtimeContext.aggregateDigest !== current.runtimeContext.aggregateDigest &&
|
||||
!(input.toolRefreshOnResume === true &&
|
||||
canonicalNativeRuntimeContextDigest({
|
||||
...previousExecution.runtimeContext,
|
||||
mcp: current.runtimeContext.mcp,
|
||||
}) === current.runtimeContext.aggregateDigest))
|
||||
!nativeRuntimeContextsMatch(previousExecution.runtimeContext, current.runtimeContext, input.toolRefreshOnResume))
|
||||
)
|
||||
return null;
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { and, asc, eq, exists, inArray, sql } from "drizzle-orm";
|
||||
import { and, asc, eq, exists, inArray, or, sql } from "drizzle-orm";
|
||||
import { environmentLeases, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db";
|
||||
import { inspectDurablePrpCheckpointHeader, type DurablePrpCheckpoint } from "../../vendor/paperclip-runner/index.js";
|
||||
import { nativeSha256 } from "./canonical.js";
|
||||
|
||||
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
const record = (value: unknown): Record<string, unknown> => value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
||||
@@ -18,7 +19,8 @@ export async function readNativeSourceConsumers(tx: Transaction, companyId: stri
|
||||
sql`${nativeRunnerCheckpoints.material}->>'schema' like 'paperclip.native-runner-checkpoint-retirement.%'`)));
|
||||
return tx.select({ id: environmentLeases.id, metadata: environmentLeases.metadata, retired }).from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId),
|
||||
sql`${environmentLeases.metadata}->'runnerTaskSourceConsumer' is not null`));
|
||||
or(sql`${environmentLeases.metadata}->'runnerTaskSourceConsumer' is not null`,
|
||||
sql`${environmentLeases.metadata}->'portableTaskSuccessor' is not null`)));
|
||||
}
|
||||
|
||||
/** Physical source consumption revokes the completed controller's late writer. */
|
||||
@@ -30,13 +32,26 @@ export async function assertNativeSourceCheckpointConsumer(tx: Transaction, owne
|
||||
if (!rows.length) return;
|
||||
const deny = (): never => { throw new Error("native_runner_checkpoint_owner_lost"); };
|
||||
if (owner.runId === owner.authorityRunId || rows.length !== 1 || !checkpoint) return deny();
|
||||
const source = rows[0]!, consumer = record(source.metadata?.runnerTaskSourceConsumer);
|
||||
const source = rows[0]!;
|
||||
const header = checkpoint.records.get("header");
|
||||
const identity = header ? inspectDurablePrpCheckpointHeader(header) : null;
|
||||
const portable = record(source.metadata?.portableTaskSuccessor);
|
||||
if (source.metadata?.portableTaskSuccessor !== undefined) {
|
||||
if (typeof portable.targetLeaseId !== "string" || portable.sourceLeaseId !== source.id ||
|
||||
!identity || nativeSha256(record(portable.operation).identity ?? null) !== nativeSha256(identity)) return deny();
|
||||
const [target] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, owner.companyId),
|
||||
eq(environmentLeases.id, portable.targetLeaseId)));
|
||||
if (!target || target.heartbeatRunId !== owner.authorityRunId || target.status !== "active" || target.releasedAt ||
|
||||
target.expiresAt && target.expiresAt.getTime() <= Date.now() ||
|
||||
target.metadata?.portableTaskOwnershipReady !== true || target.metadata.portableTaskSuccessor !== undefined ||
|
||||
nativeSha256(target.metadata.portableTaskPredecessor ?? null) !== nativeSha256(portable)) return deny();
|
||||
return;
|
||||
}
|
||||
const consumer = record(source.metadata?.runnerTaskSourceConsumer);
|
||||
if (typeof consumer.leaseId !== "string" || consumer.runId !== owner.authorityRunId) return deny();
|
||||
const [lease] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, owner.companyId),
|
||||
eq(environmentLeases.id, consumer.leaseId)));
|
||||
const recovery = record(lease?.metadata?.runnerTaskSourceRecovery);
|
||||
const header = checkpoint.records.get("header");
|
||||
const identity = header ? inspectDurablePrpCheckpointHeader(header) : null;
|
||||
const expected = record(recovery.identity);
|
||||
if (!lease || lease.heartbeatRunId !== owner.authorityRunId || lease.status !== "active" || lease.releasedAt ||
|
||||
lease.expiresAt && lease.expiresAt.getTime() <= Date.now() ||
|
||||
|
||||
@@ -553,6 +553,18 @@ describe("plugin Runner release/resume facade", () => {
|
||||
? { kind: "released", taskId: "task" }
|
||||
: { kind: "resume", taskId: "task", outcome: "resumed" });
|
||||
|
||||
it.each([true, false])("sends the current Claude credential only for session handoff (%s)", async sessionHandoff => {
|
||||
const task = lifecycleTask();
|
||||
const environment = { CLAUDE_CODE_OAUTH_TOKEN: "fixture-current-token" };
|
||||
const owner = createPluginRunnerTaskOwner({ ...input("claude"), task, sessionHandoff,
|
||||
credential: { type: "claude-env", environment } });
|
||||
await expect(owner.resume(requirements.identity, "transient-ticket")).resolves.toBe("resumed");
|
||||
expect(task).toHaveBeenCalledExactlyOnceWith({
|
||||
kind: "resume", identity: requirements.identity, bootstrapTicket: "transient-ticket",
|
||||
...(sessionHandoff ? { handoffCredential: { accountIdentity: "selected-account", environment } } : {}),
|
||||
}, { expectedTaskId: "task" });
|
||||
});
|
||||
|
||||
it.each(["codex", "claude"])("quiesces %s before release and closes only after its ACK", async harness => {
|
||||
const { task } = provider();
|
||||
const perform = task.getMockImplementation()!;
|
||||
|
||||
@@ -50,6 +50,7 @@ export function createPluginRunnerTaskOwner(input: {
|
||||
warmTaskVersion?: 1;
|
||||
releaseResumeVersion?: 1;
|
||||
releasedSource?: DurableRecoveryIdentity;
|
||||
sessionHandoff?: boolean;
|
||||
onResumed?: (identity: DurableRecoveryIdentity) => Promise<void>;
|
||||
onRetired?: (identity: DurableRecoveryIdentity) => Promise<void>;
|
||||
githubBroker?: { url: string; token: string };
|
||||
@@ -418,7 +419,12 @@ export function createPluginRunnerTaskOwner(input: {
|
||||
};
|
||||
const resume = async (identity: DurableRecoveryIdentity, bootstrapTicket: string): Promise<"resumed" | "unavailable"> => {
|
||||
requireReleaseResume();
|
||||
const result = await call(environmentTaskOperationSchema.parse({ kind: "resume", identity, bootstrapTicket }));
|
||||
const handoffCredential = input.sessionHandoff && input.credential?.type === "claude-env"
|
||||
? { accountIdentity: input.accountIdentity, environment: input.credential.environment }
|
||||
: undefined;
|
||||
const result = await call(environmentTaskOperationSchema.parse({ kind: "resume", identity, bootstrapTicket,
|
||||
...(handoffCredential ? { handoffCredential } : {}),
|
||||
}));
|
||||
requireReleaseResume();
|
||||
if (result.kind !== "resume") throw new Error("Plugin Runner resume result unavailable");
|
||||
if (result.outcome === "unavailable") {
|
||||
|
||||
+215
-4
@@ -3,19 +3,23 @@ import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { environmentTaskOperationSchema } from "@paperclipai/plugin-sdk";
|
||||
import { environmentTaskOperationSchema, type PluginEnvironmentTaskResult } from "@paperclipai/plugin-sdk";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, completionContracts, createDb, environmentLeases, environments, heartbeatRuns, issues, nativeRunFinalizations,
|
||||
nativeRunResults, nativeRunnerCheckpoints, plugins } from "@paperclipai/db";
|
||||
import { parseNativeExecutionInput, runnerProviderLaunchRequirements, DurablePrpControlPlane, type DurablePrpCheckpoint, type NativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
|
||||
import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput, runnerProviderLaunchRequirements, DurablePrpControlPlane, type DurablePrpCheckpoint, type NativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
|
||||
import { recoveryPostgresSupport, startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js";
|
||||
import { createIsolatedTestHome } from "../../__tests__/helpers/isolated-test-home.js";
|
||||
import { executeEnvironmentTask, resolvePluginTaskResume } from "../environment-task-runtime.js";
|
||||
import type { PluginWorkerManager } from "../plugin-worker-manager.js";
|
||||
import { assertAttachedPortableTask, readyPortableTaskSource, reconcilePluginTaskOwnership } from "../plugin-task-ownership.js";
|
||||
import { remoteTerminationReceipt } from "../remote-execution-termination.js";
|
||||
import * as runtimeApi from "../../runtime-api.js";
|
||||
import { NativeExecutionContext } from "./native-execution-context.js";
|
||||
import { buildNativeExecutionInput } from "./native-execution-input.js";
|
||||
import { currentNativeControllerIdentity } from "./native-restart-recovery.js";
|
||||
import { createRunnerdBackend, nativeControllerMigrationDescriptor } from "./native-session-executor.js";
|
||||
import { buildNativeExecutionWithCheckpoint, nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
|
||||
import { loadNativeRunnerCheckpoint, nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js";
|
||||
import { claimPluginTaskSourceRecovery } from "./plugin-task-source-recovery.js";
|
||||
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
||||
@@ -83,7 +87,7 @@ describe.skipIf(!recoveryPostgresSupport.supported)("stopped source admission un
|
||||
leaseOwner: owner.leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000) });
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "source-fixture", version: "1.0.0", status: "ready",
|
||||
manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }] } as never });
|
||||
manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true, sessionHandoffVersion: 1 }] } as never });
|
||||
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "plugin", status: "active",
|
||||
config: { pluginKey: pluginId, driverKey: "tasks" } });
|
||||
const metadata = { driver: "plugin", pluginId, driverKey: "tasks", runnerTaskLease: true };
|
||||
@@ -98,7 +102,7 @@ describe.skipIf(!recoveryPostgresSupport.supported)("stopped source admission un
|
||||
const input = { companyId, leaseId, context, identity, resumeFrom: "source-task",
|
||||
...(provider === "codex" ? { codexSourceRecoveryVersion: 1 as const } : { acpxSourceRecoveryVersion: 1 as const }),
|
||||
accountIdentity: "current-selected-account", matchesExecution: (previous: typeof sourceExecution) => previous.provider.kind === execution.provider.kind && previous.workspace.cwd === execution.workspace.cwd };
|
||||
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call: vi.fn(async (_id: string, _method: string, p: any) => {
|
||||
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask"] }), call: vi.fn(async (_id: string, _method: string, p: any): Promise<PluginEnvironmentTaskResult> => {
|
||||
if (p.operation.kind === "prepare") return { kind: "prepared", taskId: p.taskId,
|
||||
launch: { configurationId: "source-configuration", requirementsDigest: `sha256:${"a".repeat(64)}`, provider: {} } };
|
||||
if (p.operation.kind === "check_project_auth") return { kind: "project_auth", taskId: p.taskId, provider: p.operation.provider, configurationId: p.operation.configurationId, allowed: true };
|
||||
@@ -125,6 +129,213 @@ describe.skipIf(!recoveryPostgresSupport.supported)("stopped source admission un
|
||||
harness: provider === "claude" ? "claude" as const : "codex" as const, authProvider: provider === "claude" ? "anthropic" : "openai" };
|
||||
}
|
||||
|
||||
it.each([
|
||||
{ policy: "ordinary", scenario: "ordinary handoff" },
|
||||
{ policy: "reclaim_if_needed", scenario: "idle source reclaim" },
|
||||
{ policy: "ordinary", scenario: "fresh conversation after instructions change", freshSession: true },
|
||||
] as const)("reconciles ownership after a lost response: $scenario", async scenario => {
|
||||
const { policy } = scenario;
|
||||
const f = await fixture();
|
||||
await db.update(heartbeatRuns).set({ scopeKind: "issue", issueId: f.issueId }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.update(environmentLeases).set({ status: "retained", releasedAt: null,
|
||||
metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination') || ${JSON.stringify({
|
||||
providerMetadata: { ownershipGeneration: "source-generation" }, runnerTaskAccountIdentity: f.accountIdentity,
|
||||
})}::jsonb`,
|
||||
}).where(eq(environmentLeases.id, f.sourceLeaseId));
|
||||
await db.update(environmentLeases).set({
|
||||
metadata: sql`${environmentLeases.metadata} || '{"providerMetadata":{"ownershipGeneration":"target-generation"}}'::jsonb`,
|
||||
}).where(eq(environmentLeases.id, f.leaseId));
|
||||
const request = { companyId: f.companyId, sourceLeaseId: f.sourceLeaseId, targetLeaseId: f.leaseId, policy };
|
||||
const send = (database: typeof db) => reconcilePluginTaskOwnership(database, request, operation =>
|
||||
executeEnvironmentTask(database, f.workers as unknown as PluginWorkerManager, { companyId: f.companyId, leaseId: f.leaseId, operation }));
|
||||
const describe = (database: typeof db, ownershipSignal?: AbortSignal) => executeEnvironmentTask(database, f.workers as unknown as PluginWorkerManager,
|
||||
{ companyId: f.companyId, leaseId: f.leaseId, ownershipSignal, operation: { kind: "describe" } });
|
||||
if (policy === "ordinary") await db.update(environmentLeases).set({
|
||||
metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ portableTaskPreviousLeaseId: f.sourceLeaseId })}::jsonb`,
|
||||
}).where(eq(environmentLeases.id, f.leaseId));
|
||||
f.workers.call.mockRejectedValueOnce(new Error("worker transport lost"));
|
||||
await expect(policy === "ordinary" ? describe(db) : send(db)).rejects.toThrow();
|
||||
const operation = f.workers.call.mock.calls[0][2].operation;
|
||||
expect(operation).toMatchObject({ kind: "ensure_ownership", operationId: f.leaseId, identity: f.identity,
|
||||
sourceTaskId: "source-task", sourceGeneration: "source-generation", targetGeneration: "target-generation", policy });
|
||||
await expect(f.task({ ...operation, policy: policy === "ordinary" ? "reclaim_if_needed" : "ordinary" })).rejects.toThrow("ownership_unverified");
|
||||
await expect(f.task()).rejects.toThrow("session ownership unavailable");
|
||||
await expect(executeEnvironmentTask(db, f.workers as unknown as PluginWorkerManager, {
|
||||
companyId: f.companyId, leaseId: f.sourceLeaseId, operation: { kind: "resume", identity: f.identity, bootstrapTicket: "fixture-ticket" },
|
||||
})).rejects.toThrow("session ownership unavailable");
|
||||
expect(f.workers.call).toHaveBeenCalledTimes(1);
|
||||
const reply = { kind: "ownership", taskId: "replacement-task", operationId: f.leaseId,
|
||||
targetGeneration: "target-generation" } as const;
|
||||
f.workers.call.mockResolvedValueOnce({ ...reply, status: "pending" });
|
||||
await expect(describe(db)).rejects.toThrow("native_portable_task_ownership_pending");
|
||||
const targetWriter = nativeRunnerCheckpointWriter(db, { companyId: f.companyId, issueId: f.issueId,
|
||||
runId: f.sourceRunId, authorityRunId: f.runId, ...f.owner });
|
||||
await expect(targetWriter(f.checkpoint)).rejects.toThrow("native_runner_checkpoint_owner_lost");
|
||||
const restarted = createDb(database.connectionString);
|
||||
f.workers.call.mockResolvedValueOnce({ ...reply, status: "ready", targetGeneration: "another-generation" });
|
||||
await expect(describe(restarted)).rejects.toThrow("reconcile the same task");
|
||||
await expect(f.task()).rejects.toThrow("session ownership unavailable");
|
||||
const description = { kind: "description", taskId: "replacement-task",
|
||||
runner: { version: "test", digest: `sha256:${"a".repeat(64)}` } } as const;
|
||||
f.workers.call.mockRejectedValueOnce(new Error("worker transport lost again"));
|
||||
f.workers.call.mockResolvedValueOnce({ ...reply, status: "pending" });
|
||||
f.workers.call.mockResolvedValueOnce({ ...reply, status: "ready" });
|
||||
f.workers.call.mockResolvedValueOnce(description);
|
||||
await expect(describe(restarted, new AbortController().signal)).resolves.toEqual(description);
|
||||
expect(f.workers.call.mock.calls.map(([, , call]) => call.operation)).toEqual([...Array(6).fill(operation), { kind: "describe" }]);
|
||||
const [target] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
|
||||
expect(target.metadata?.portableTaskOwnershipReady).toBe(true);
|
||||
expect(target.metadata?.runnerTaskAccountIdentity).toBe(f.accountIdentity);
|
||||
const [source] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.sourceLeaseId));
|
||||
expect(source.metadata?.remoteExecutionTermination).toBeUndefined();
|
||||
await expect(f.oldWriter(f.checkpoint)).rejects.toThrow("native_runner_checkpoint_owner_lost");
|
||||
await expect(targetWriter(f.checkpoint)).resolves.toBeUndefined();
|
||||
|
||||
async function recoverAttachedRunner(execution: NativeExecutionInput, identity: typeof f.identity) {
|
||||
const { root: stateDirectory } = nativeControllerMigrationDescriptor(execution,
|
||||
nativeToolContractFingerprintForTarget("remote", execution.provider.kind));
|
||||
const durable = new DurablePrpControlPlane({ stateDirectory: join(stateDirectory, "control-plane"),
|
||||
identity: identity, expectedRunnerVersion: description.runner.version, expectedRunnerDigest: description.runner.digest });
|
||||
durable.persistRunAttachTemplate({ provider: { instructions: "Run using the admitted instructions.",
|
||||
runtimeContext: "runtimeContext" in execution ? execution.runtimeContext : null } });
|
||||
await durable.flushCheckpoint();
|
||||
await durable.stop();
|
||||
const recover = () => createRunnerdBackend({ db: restarted, execution,
|
||||
runnerInstanceId: identity.runnerInstanceId, runnerIngressAuthorized: true,
|
||||
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", leaseId: f.leaseId,
|
||||
sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "replacement-task" },
|
||||
runnerTask: (operation, options) => executeEnvironmentTask(restarted, f.workers as unknown as PluginWorkerManager, {
|
||||
companyId: f.companyId, leaseId: f.leaseId, operation: environmentTaskOperationSchema.parse(operation),
|
||||
expectedTaskId: options?.expectedTaskId,
|
||||
}),
|
||||
},
|
||||
restartRecovery: { kind: "reattach_remote_runner", remote: { kind: "plugin_task", leaseId: f.leaseId },
|
||||
runId: f.runId, leaseOwner: "successor", controllerGeneration: f.owner.generation + 1,
|
||||
providerAttempt: 1, restartKind: "hard", recoveryRequestId: null },
|
||||
});
|
||||
f.workers.call.mockClear();
|
||||
f.workers.call
|
||||
.mockResolvedValueOnce({ kind: "status", taskId: "replacement-task", phase: "running", executionStopped: false })
|
||||
.mockResolvedValueOnce({ kind: "state", taskId: "replacement-task",
|
||||
state: { schema: "paperclip.runner.durable.state.v1", ...identity } })
|
||||
.mockResolvedValueOnce({ ...description, launchPreparationVersion: 1, projectAuthCheckVersion: 1 });
|
||||
const api = vi.spyOn(runtimeApi, "resolveLocalRuntimeApiUrl").mockReturnValue("http://127.0.0.1:3100");
|
||||
try { await expect(recover()).resolves.toBeDefined(); }
|
||||
finally { api.mockRestore(); }
|
||||
expect(f.workers.call.mock.calls.map(([, , call]) => [call.taskId, call.operation.kind])).toEqual([
|
||||
["replacement-task", "status"], ["replacement-task", "read_state"], ["replacement-task", "describe"],
|
||||
]);
|
||||
return recover;
|
||||
}
|
||||
|
||||
if ("freshSession" in scenario) {
|
||||
const previousExecution = f.sourceExecution;
|
||||
const nativeToolContractFingerprint = nativeToolContractFingerprintForTarget("remote", f.execution.provider.kind);
|
||||
const normalizedSessionId = f.identity.normalizedSessionId;
|
||||
const previousRun = {
|
||||
id: f.sourceRunId, companyId: f.companyId, agentId: f.execution.binding.agentId, nativeSessionId: normalizedSessionId,
|
||||
runnerProfileJson: { nativeToolContractFingerprint, nativeExecutionInput: previousExecution,
|
||||
sessionCheckpoint: { backendKind: "runner", driverKind: previousExecution.session.driverKind,
|
||||
sessionId: "previous-provider-thread", providerSessionId: "previous-provider-thread",
|
||||
identity: { ...previousExecution.binding, sessionId: normalizedSessionId }, pendingRuntimeRequests: [] } },
|
||||
};
|
||||
const select = (current: NativeExecutionInput) => buildNativeExecutionWithCheckpoint({
|
||||
previousRun, normalizedSessionId, executionTargetKind: "remote", toolRefreshOnResume: true,
|
||||
buildExecution: ({ normalizedSessionId }) => ({ ...current, session: { ...current.session, normalizedSessionId } } as NativeExecutionInput),
|
||||
});
|
||||
expect(select(f.execution).checkpoint?.providerSessionId).toBe("previous-provider-thread");
|
||||
const changed = structuredClone(f.execution);
|
||||
if (!("runtimeContext" in changed)) throw new Error("fixture requires runtime context");
|
||||
changed.runtimeContext.instructions.bundle.digest = "1".repeat(64);
|
||||
changed.runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(changed.runtimeContext);
|
||||
const selected = select(changed);
|
||||
expect(selected.checkpoint).toBeNull();
|
||||
expect(selected.normalizedSessionId).not.toBe(normalizedSessionId);
|
||||
const scope = { attachedRunId: f.runId, accountIdentity: f.accountIdentity, selectedSessionId: selected.normalizedSessionId };
|
||||
const sourceFor = (identity: unknown) => readyPortableTaskSource(restarted, f.companyId, f.leaseId, identity, scope);
|
||||
await expect(sourceFor(null)).rejects.toThrow("ownership_unverified");
|
||||
await db.update(heartbeatRuns).set({ nativeSessionId: selected.normalizedSessionId }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await expect(sourceFor(null)).rejects.toThrow("ownership_unverified");
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: selected.execution } })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
await expect(sourceFor(null)).resolves.toBeUndefined();
|
||||
await expect(sourceFor(f.identity)).rejects.toThrow("ownership_unverified");
|
||||
const duplicate = randomUUID();
|
||||
await db.insert(environmentLeases).values({ ...target, id: duplicate, providerLeaseId: "ambiguous-task" });
|
||||
await expect(sourceFor(null)).rejects.toThrow("ownership_unverified");
|
||||
await db.delete(environmentLeases).where(eq(environmentLeases.id, duplicate));
|
||||
|
||||
const fresh = { ...f.identity, runId: f.runId, environmentLeaseId: f.leaseId, runnerInstanceId: randomUUID(),
|
||||
normalizedSessionId: selected.normalizedSessionId, turnId: "fresh-turn", itemId: "fresh-item" };
|
||||
const prepare = { ...f.operation, resumeFrom: undefined, requirements: { ...f.operation.requirements, identity: fresh } };
|
||||
const submit = { ...f.submit, resumeFrom: undefined, runner: { ...f.submit.runner,
|
||||
runnerId: fresh.runnerInstanceId, leaseId: fresh.environmentLeaseId, runId: fresh.runId,
|
||||
sessionId: fresh.normalizedSessionId, turnId: fresh.turnId, itemId: fresh.itemId } };
|
||||
f.workers.call.mockClear();
|
||||
await expect(f.task(prepare)).resolves.toMatchObject({ kind: "prepared" });
|
||||
await expect(f.task({ kind: "check_project_auth", provider: "openai", projectIds: [],
|
||||
configurationId: submit.preparedLaunchId })).resolves.toMatchObject({ kind: "project_auth", allowed: true });
|
||||
await expect(f.task(submit)).resolves.toMatchObject({ kind: "accepted" });
|
||||
expect(f.workers.call.mock.calls.map(([, , call]) => call.operation.kind)).toEqual(["prepare", "check_project_auth", "submit"]);
|
||||
expect(f.workers.call.mock.calls.every(([, , call]) => call.taskId === "replacement-task" &&
|
||||
call.operation.resumeFrom === undefined && call.operation.priorIdentity === undefined)).toBe(true);
|
||||
await expect(sourceFor(null)).rejects.toThrow("ownership_unverified");
|
||||
await expect(sourceFor({ ...fresh, turnId: "wrong-turn" })).rejects.toThrow("ownership_unverified");
|
||||
await expect(sourceFor(fresh)).resolves.toBeUndefined();
|
||||
await recoverAttachedRunner(selected.execution, fresh);
|
||||
return;
|
||||
}
|
||||
|
||||
const resume = { kind: "resume", identity: f.identity, bootstrapTicket: "fixture-ticket" } as const;
|
||||
f.workers.call.mockResolvedValueOnce({ kind: "resume", taskId: "replacement-task", outcome: "resumed" });
|
||||
await expect(f.task(resume)).resolves.toMatchObject({ kind: "resume", outcome: "resumed" });
|
||||
const [dispatched] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
|
||||
expect(dispatched.metadata?.runnerTaskDispatched).toBe(true);
|
||||
|
||||
const readState = { kind: "read_state", identity: f.identity } as const;
|
||||
await expect(f.task({ ...readState, identity: { ...f.identity, itemId: "wrong-item" } }))
|
||||
.rejects.toThrow("ownership_unverified");
|
||||
f.workers.call.mockResolvedValueOnce({ kind: "state", taskId: "replacement-task", state: { ...f.identity } });
|
||||
await expect(f.task(readState)).resolves.toMatchObject({ kind: "state", state: f.identity });
|
||||
const nextIdentity = { ...f.identity, runId: f.runId, turnId: "target-turn", itemId: "target-item" };
|
||||
const attach = { ...f.operation, resumeFrom: undefined, priorIdentity: f.identity,
|
||||
requirements: { ...f.operation.requirements, identity: nextIdentity } };
|
||||
await expect(f.task({ ...attach, requirements: { ...attach.requirements,
|
||||
identity: { ...nextIdentity, normalizedSessionId: randomUUID() } } })).rejects.toThrow("prior identity mismatch");
|
||||
await expect(f.task(attach)).resolves.toMatchObject({ kind: "prepared" });
|
||||
await expect(f.task(attach)).resolves.toMatchObject({ kind: "prepared" });
|
||||
const [attached] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
|
||||
expect(attached.metadata?.runnerTaskIdentity).toMatchObject({ runId: f.runId, sessionId: f.identity.normalizedSessionId });
|
||||
expect(attached.metadata?.runnerTaskPriorIdentity).toMatchObject({ runId: f.sourceRunId, itemId: f.identity.itemId });
|
||||
const scope = { attachedRunId: f.runId, accountIdentity: f.accountIdentity };
|
||||
await expect(readyPortableTaskSource(db, f.companyId, f.leaseId, nextIdentity, scope)).resolves.toBeUndefined();
|
||||
await expect(readyPortableTaskSource(db, f.companyId, f.leaseId, { ...nextIdentity, turnId: "wrong-turn" }, scope))
|
||||
.rejects.toThrow("ownership_unverified");
|
||||
await expect(readyPortableTaskSource(db, f.companyId, f.leaseId, f.identity, scope))
|
||||
.resolves.toEqual({ taskId: "source-task", identity: f.identity });
|
||||
await expect(readyPortableTaskSource(db, f.companyId, f.leaseId, f.identity, { ...scope, accountIdentity: "another-account" }))
|
||||
.rejects.toThrow("ownership_unverified");
|
||||
|
||||
const binding = { companyId: f.companyId, leaseId: f.leaseId, taskId: "replacement-task", runId: f.runId, identity: nextIdentity };
|
||||
await expect(assertAttachedPortableTask(restarted, binding)).resolves.toBeUndefined();
|
||||
for (const invalid of [
|
||||
{ ...binding, leaseId: f.sourceLeaseId },
|
||||
{ ...binding, taskId: "source-task" },
|
||||
{ ...binding, identity: f.identity },
|
||||
{ ...binding, identity: { ...nextIdentity, environmentLeaseId: f.leaseId } },
|
||||
]) await expect(assertAttachedPortableTask(restarted, invalid)).rejects.toThrow("ownership_unverified");
|
||||
|
||||
if (policy === "ordinary") {
|
||||
const recover = await recoverAttachedRunner(f.execution, nextIdentity);
|
||||
await db.update(environmentLeases).set({
|
||||
metadata: sql`${environmentLeases.metadata} - 'portableTaskSuccessor'`,
|
||||
}).where(eq(environmentLeases.id, f.sourceLeaseId));
|
||||
f.workers.call.mockClear();
|
||||
await expect(recover()).rejects.toThrow("ownership_unverified");
|
||||
expect(f.workers.call).not.toHaveBeenCalled();
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["codex", "claude"] as const)("authorizes the claimed %s source while preserving A's accepted outcome", async provider => {
|
||||
const f = await fixture(provider);
|
||||
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
|
||||
|
||||
@@ -5,6 +5,7 @@ import { parseNativeExecutionInput, type NativeExecutionInput, type PersistedNat
|
||||
import { currentNativeControllerIdentity, evaluateNativeControllerTakeover, recordLegacyHostFallbackProof } from "./native-restart-recovery.js";
|
||||
import { NativeControllerRecoveryWaitingError, readNativeControllerRecovery } from "./native-controller-recovery.js";
|
||||
import { NativePredecessorRetirementUnprovenError } from "./predecessor-retirement.js";
|
||||
import { readyPortableTaskSource } from "../plugin-task-ownership.js";
|
||||
|
||||
export type RemoteWarmSessionRecovery = {
|
||||
identity: Record<string, string>;
|
||||
@@ -12,6 +13,7 @@ export type RemoteWarmSessionRecovery = {
|
||||
previousExecution: NativeExecutionInput;
|
||||
reuseProcess: boolean;
|
||||
releasedSource?: DurableRecoveryIdentity;
|
||||
portableSourceTaskId?: string;
|
||||
remote: { providerLeaseId: string; remoteCwd: string };
|
||||
revalidate: (leaseOwner: string) => Promise<void>;
|
||||
};
|
||||
@@ -128,6 +130,7 @@ export async function inspectRemoteWarmSessionRecovery(input: {
|
||||
matchesExecution: (previous: NativeExecutionInput) => boolean;
|
||||
loadCheckpoint: (previous: NativeExecutionInput) => PersistedNativeSession | null;
|
||||
canReuseProcess: (previous: NativeExecutionInput) => boolean;
|
||||
portableSourceTaskId?: string;
|
||||
}): Promise<RemoteWarmSessionRecovery | null> {
|
||||
const { binding } = input.execution;
|
||||
const { target } = input;
|
||||
@@ -152,13 +155,15 @@ export async function inspectRemoteWarmSessionRecovery(input: {
|
||||
// sandbox (for example after a restart or deploy replaced the controller)
|
||||
// holds no Runner to adopt, so live adoption steps aside. Ordinary admission
|
||||
// then retires the finished owner or keeps it held with a named reason.
|
||||
if (target.transport === "sandbox" && ["created", "replacement"].includes(target.sandboxLeaseAcquisition?.outcome ?? "")) {
|
||||
if (!input.portableSourceTaskId && target.transport === "sandbox" && ["created", "replacement"].includes(target.sandboxLeaseAcquisition?.outcome ?? "")) {
|
||||
return null;
|
||||
}
|
||||
if (target.transport !== "sandbox" || !target.leaseId || !target.environmentId ||
|
||||
!target.providerKey || !target.sandboxLeaseAcquisition || target.sandboxLeaseAcquisition.outcome !== "resumed" ||
|
||||
!target.providerKey || !target.sandboxLeaseAcquisition ||
|
||||
(!input.portableSourceTaskId && target.sandboxLeaseAcquisition.outcome !== "resumed") ||
|
||||
identity.runId === binding.runId) return deny("target_not_resumed_sandbox");
|
||||
const providerLeaseId = target.sandboxLeaseAcquisition.providerLeaseId;
|
||||
const sourceProviderLeaseId = input.portableSourceTaskId ?? providerLeaseId;
|
||||
const [prior] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations })
|
||||
.from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
|
||||
.where(and(eq(heartbeatRuns.id, identity.runId), eq(heartbeatRuns.companyId, binding.companyId),
|
||||
@@ -171,7 +176,7 @@ export async function inspectRemoteWarmSessionRecovery(input: {
|
||||
throw new NativeControllerRecoveryWaitingError(prior.owner.leaseExpiresAt);
|
||||
}
|
||||
if (prior.owner.phase !== "committed" || prior.owner.leaseOwner) return deny("owner_unsettled");
|
||||
settledPredecessor = { runId: identity.runId, providerLeaseId };
|
||||
if (sourceProviderLeaseId === providerLeaseId) settledPredecessor = { runId: identity.runId, providerLeaseId };
|
||||
const checkpoint = input.loadCheckpoint(previous);
|
||||
if (!checkpoint || checkpoint.identity.runId !== identity.runId ||
|
||||
checkpoint.identity.companyId !== binding.companyId || checkpoint.identity.issueId !== binding.issueId ||
|
||||
@@ -194,15 +199,22 @@ export async function inspectRemoteWarmSessionRecovery(input: {
|
||||
const priorLeases = await input.db.select().from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, binding.companyId), eq(environmentLeases.heartbeatRunId, identity.runId),
|
||||
eq(environmentLeases.issueId, binding.issueId), eq(environmentLeases.environmentId, target.environmentId),
|
||||
eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, providerLeaseId),
|
||||
eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, sourceProviderLeaseId),
|
||||
));
|
||||
if (priorLeases.length !== 1 || priorLeases[0].metadata?.remoteCwd !== target.remoteCwd ||
|
||||
!["released", "expired", "active"].includes(priorLeases[0].status)) return deny("prior_lease_mismatch");
|
||||
!["released", "expired", "active", ...(input.portableSourceTaskId ? ["retained"] : [])].includes(priorLeases[0].status)) {
|
||||
return deny("prior_lease_mismatch");
|
||||
}
|
||||
return {
|
||||
identity, checkpoint, previousExecution: previous, reuseProcess: input.canReuseProcess(previous),
|
||||
portableSourceTaskId: input.portableSourceTaskId,
|
||||
remote: { providerLeaseId, remoteCwd: target.remoteCwd },
|
||||
revalidate: async leaseOwner => {
|
||||
await assertPriorStopped();
|
||||
if (input.portableSourceTaskId &&
|
||||
(await readyPortableTaskSource(input.db, binding.companyId, target.leaseId!, identity))?.taskId !== input.portableSourceTaskId) {
|
||||
return deny("prior_lease_mismatch");
|
||||
}
|
||||
const controller = await currentNativeControllerIdentity();
|
||||
const [current] = await input.db.select({ run: heartbeatRuns, owner: nativeRunFinalizations, issue: issues, agent: agents, company: companies })
|
||||
.from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
|
||||
|
||||
@@ -13,6 +13,12 @@ const handoffSchema = z.object({
|
||||
const key = "runnerTaskCredentialHandoff";
|
||||
type Scope = { companyId: string; leaseId: string; expectedTaskId?: string };
|
||||
|
||||
export function canReadStoppedPluginTaskCredential(metadata: Record<string, unknown>, accountIdentity: string): boolean {
|
||||
const handoff = handoffSchema.safeParse(metadata[key]);
|
||||
return metadata.runnerTaskExecutionStopped === true && handoff.success &&
|
||||
handoff.data.state === "pending" && handoff.data.accountIdentity === accountIdentity;
|
||||
}
|
||||
|
||||
/** Save authority, never credential bytes or a controller-local temporary path. */
|
||||
export async function registerPluginTaskCredentialHandoff(
|
||||
db: Db,
|
||||
|
||||
@@ -0,0 +1,319 @@
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { and, asc, eq, inArray, isNotNull, isNull, or, sql } from "drizzle-orm";
|
||||
import { environmentLeases, heartbeatRuns, issues, type Db } from "@paperclipai/db";
|
||||
import { environmentTaskOperationSchema, environmentTaskResultSchema, type PluginEnvironmentTaskOperation, type PluginEnvironmentTaskResult } from "@paperclipai/plugin-sdk";
|
||||
import { nativeSha256 } from "./native-runtime/canonical.js";
|
||||
import { lockSourceRecoveryCoordinators } from "./native-runtime/native-source-recovery-ownership.js";
|
||||
import type { PluginTaskDescription } from "./environment-task-runtime.js";
|
||||
|
||||
type OwnershipOperation = Extract<PluginEnvironmentTaskOperation, { kind: "ensure_ownership" }>;
|
||||
type OwnershipResult = Extract<PluginEnvironmentTaskResult, { kind: "ownership" }>;
|
||||
type Lease = typeof environmentLeases.$inferSelect;
|
||||
type OwnershipClaim = {
|
||||
sourceLeaseId: string;
|
||||
targetLeaseId: string;
|
||||
operation: OwnershipOperation;
|
||||
};
|
||||
|
||||
export function unlaunchedPortableTaskLease() {
|
||||
const { metadata } = environmentLeases;
|
||||
const unlaunched = and(
|
||||
eq(environmentLeases.status, "active"), isNull(environmentLeases.releasedAt), isNull(environmentLeases.cleanupStatus),
|
||||
isNotNull(environmentLeases.providerLeaseId),
|
||||
or(isNull(environmentLeases.expiresAt), sql`${environmentLeases.expiresAt} > clock_timestamp()`),
|
||||
sql`${metadata}->>'driver' = 'plugin' and ${metadata}->>'runnerTaskLease' = 'true'
|
||||
and jsonb_typeof(${metadata}->'portableTaskPreviousLeaseId') = 'string'
|
||||
and not (${metadata} ?| array['portableTaskSuccessor', 'runnerTaskIdentity', 'runnerTaskStartup', 'runnerTaskStartupCleanup'])
|
||||
and coalesce(${metadata}->>'runnerTaskDispatched', 'false') = 'false'
|
||||
and coalesce(${metadata}->>'runnerTaskRetiring', 'false') = 'false'`,
|
||||
)!;
|
||||
return sql<boolean>`case when ${unlaunched} then not exists (select 1 from environment_leases other
|
||||
where other.company_id = ${environmentLeases.companyId} and other.heartbeat_run_id = ${environmentLeases.heartbeatRunId}
|
||||
and other.status = 'active' and other.id <> ${environmentLeases.id}) else false end`;
|
||||
}
|
||||
|
||||
export function portableTaskPreparationLeaseId(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "issueId" | "agentId">) {
|
||||
return sql<string | null>`(${db.select({ id: environmentLeases.id }).from(issues).innerJoin(environmentLeases, and(
|
||||
eq(environmentLeases.companyId, issues.companyId), eq(environmentLeases.issueId, issues.id),
|
||||
eq(environmentLeases.heartbeatRunId, issues.executionRunId),
|
||||
)).where(and(eq(issues.companyId, run.companyId), run.issueId ? eq(issues.id, run.issueId) : sql`false`,
|
||||
eq(issues.executionRunId, run.id), eq(issues.assigneeAgentId, run.agentId), eq(issues.status, "in_progress"),
|
||||
sql`${environmentLeases.metadata}->>'agentId' = ${run.agentId}`, unlaunchedPortableTaskLease()))})`;
|
||||
}
|
||||
|
||||
const object = (value: unknown): Record<string, unknown> =>
|
||||
value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
||||
|
||||
export class PluginTaskOwnershipPendingError extends Error {
|
||||
constructor() {
|
||||
super("native_portable_task_ownership_pending");
|
||||
}
|
||||
}
|
||||
|
||||
function unavailable(): never {
|
||||
throw new Error("native_portable_task_ownership_unverified");
|
||||
}
|
||||
|
||||
function leaseGeneration(lease: Lease) {
|
||||
return object(lease.metadata?.providerMetadata).ownershipGeneration;
|
||||
}
|
||||
|
||||
function readOperation(source: Lease, target: Lease, policy: OwnershipOperation["policy"]): OwnershipOperation {
|
||||
const identity = object(source.metadata?.runnerTaskIdentity);
|
||||
const operation = environmentTaskOperationSchema.safeParse({
|
||||
kind: "ensure_ownership", operationId: target.id, sourceTaskId: source.providerLeaseId,
|
||||
sourceGeneration: leaseGeneration(source), targetGeneration: leaseGeneration(target), policy,
|
||||
identity: {
|
||||
runnerInstanceId: identity.runnerId, environmentLeaseId: identity.leaseId,
|
||||
runId: identity.runId, normalizedSessionId: identity.sessionId,
|
||||
turnId: identity.turnId, itemId: identity.itemId,
|
||||
},
|
||||
});
|
||||
return operation.success && operation.data.kind === "ensure_ownership" ? operation.data : unavailable();
|
||||
}
|
||||
|
||||
async function ownershipClaim(db: Db, input: {
|
||||
companyId: string;
|
||||
sourceLeaseId: string;
|
||||
targetLeaseId: string;
|
||||
policy: OwnershipOperation["policy"];
|
||||
}, ready?: OwnershipResult): Promise<OwnershipClaim> {
|
||||
const { companyId, sourceLeaseId, targetLeaseId, policy } = input;
|
||||
if (sourceLeaseId === targetLeaseId) return unavailable();
|
||||
return db.transaction(async tx => {
|
||||
const scope = and(eq(environmentLeases.companyId, companyId), inArray(environmentLeases.id, [sourceLeaseId, targetLeaseId]));
|
||||
const initial = await tx.select().from(environmentLeases).where(scope);
|
||||
const sourceRunId = initial.find(lease => lease.id === sourceLeaseId)?.heartbeatRunId;
|
||||
const targetRunId = initial.find(lease => lease.id === targetLeaseId)?.heartbeatRunId;
|
||||
const issueId = initial.find(lease => lease.id === sourceLeaseId)?.issueId;
|
||||
if (!sourceRunId || !targetRunId || sourceRunId === targetRunId || !issueId) return unavailable();
|
||||
const owners = await lockSourceRecoveryCoordinators(tx, companyId, [sourceRunId, targetRunId]);
|
||||
const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, companyId), eq(issues.id, issueId))).for("update", { noWait: true });
|
||||
const leases = await tx.select().from(environmentLeases).where(scope).orderBy(asc(environmentLeases.id)).for("update");
|
||||
const source = leases.find(lease => lease.id === sourceLeaseId);
|
||||
const target = leases.find(lease => lease.id === targetLeaseId);
|
||||
const runs = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
||||
inArray(heartbeatRuns.id, [sourceRunId, targetRunId])));
|
||||
const prior = runs.find(run => run.id === sourceRunId);
|
||||
const current = runs.find(run => run.id === targetRunId);
|
||||
const sourceOwner = owners.find(owner => owner.runId === sourceRunId);
|
||||
if (!source || !target || source.heartbeatRunId !== sourceRunId || target.heartbeatRunId !== targetRunId ||
|
||||
!source.provider?.startsWith("plugin:") || source.provider !== target.provider ||
|
||||
!source.providerLeaseId || !target.providerLeaseId || source.providerLeaseId === target.providerLeaseId ||
|
||||
!source.environmentId || source.environmentId !== target.environmentId ||
|
||||
source.issueId !== issueId || target.issueId !== issueId || source.executionWorkspaceId !== target.executionWorkspaceId ||
|
||||
source.metadata?.driver !== "plugin" || target.metadata?.driver !== "plugin" ||
|
||||
typeof source.metadata.pluginId !== "string" || source.metadata.pluginId !== target.metadata.pluginId ||
|
||||
typeof source.metadata.driverKey !== "string" || source.metadata.driverKey !== target.metadata.driverKey ||
|
||||
source.metadata.runnerTaskLease !== true || target.metadata.runnerTaskLease !== true ||
|
||||
source.metadata.runnerTaskRetiring === true || source.metadata.reusableLeaseSuccessor !== undefined ||
|
||||
source.metadata.runnerTaskSourceConsumer !== undefined ||
|
||||
!["retained", "released", "expired"].includes(source.status) ||
|
||||
target.status !== "active" || target.releasedAt || target.metadata.runnerTaskRetiring === true ||
|
||||
target.expiresAt && target.expiresAt.getTime() <= Date.now() ||
|
||||
prior?.status !== "succeeded" || prior.nativeIssueId !== issueId ||
|
||||
sourceOwner?.phase !== "committed" || !sourceOwner.resultId || sourceOwner.leaseOwner ||
|
||||
current?.status !== "running" || current.scopeKind !== "issue" || current.issueId !== issueId ||
|
||||
current.nativeIssueId !== null && current.nativeIssueId !== issueId || current.resultJson?.nativeCancellation ||
|
||||
prior.agentId !== current.agentId || issue?.executionRunId !== targetRunId || issue.assigneeAgentId !== current.agentId ||
|
||||
issue.status !== "in_progress") return unavailable();
|
||||
|
||||
const operation = readOperation(source, target, policy);
|
||||
if (operation.identity.runId !== prior.id || operation.identity.normalizedSessionId !== prior.nativeSessionId ||
|
||||
current.nativeSessionId !== null && current.nativeSessionId !== prior.nativeSessionId) return unavailable();
|
||||
const claim = { sourceLeaseId, targetLeaseId, operation };
|
||||
const existing = [source.metadata.portableTaskSuccessor, target.metadata.portableTaskPredecessor];
|
||||
if (existing.some(saved => saved !== undefined && nativeSha256(saved) !== nativeSha256(claim)) ||
|
||||
(existing[0] === undefined) !== (existing[1] === undefined)) return unavailable();
|
||||
if (ready && (existing[0] === undefined || ready.taskId !== target.providerLeaseId ||
|
||||
ready.operationId !== operation.operationId || ready.targetGeneration !== operation.targetGeneration ||
|
||||
ready.status !== "ready")) return unavailable();
|
||||
if (!ready && existing[0] === undefined) {
|
||||
await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ portableTaskSuccessor: claim })}::jsonb` })
|
||||
.where(eq(environmentLeases.id, sourceLeaseId));
|
||||
await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ portableTaskPredecessor: claim })}::jsonb` })
|
||||
.where(eq(environmentLeases.id, targetLeaseId));
|
||||
} else if (ready && target.metadata.portableTaskOwnershipReady === true &&
|
||||
(target.metadata.portableTaskCheckpointUnavailable === true) !== (ready.checkpointUnavailable === true)) {
|
||||
return unavailable();
|
||||
} else if (ready && target.metadata.portableTaskOwnershipReady !== true) {
|
||||
await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({
|
||||
portableTaskOwnershipReady: true, ...(ready.checkpointUnavailable ? { portableTaskCheckpointUnavailable: true }
|
||||
: { runnerTaskAccountIdentity: source.metadata.runnerTaskAccountIdentity }),
|
||||
})}::jsonb` })
|
||||
.where(eq(environmentLeases.id, targetLeaseId));
|
||||
}
|
||||
return claim;
|
||||
});
|
||||
}
|
||||
|
||||
export async function reconcilePluginTaskOwnership(db: Db, input: Parameters<typeof ownershipClaim>[1],
|
||||
dispatch: (operation: OwnershipOperation) => Promise<PluginEnvironmentTaskResult>): Promise<OwnershipResult> {
|
||||
try {
|
||||
const claim = await ownershipClaim(db, input);
|
||||
const result = await dispatch(claim.operation);
|
||||
if (result.kind !== "ownership" || result.operationId !== claim.operation.operationId ||
|
||||
result.targetGeneration !== claim.operation.targetGeneration) return unavailable();
|
||||
if (result.status === "ready") await ownershipClaim(db, input, result);
|
||||
return result;
|
||||
} catch (error) {
|
||||
if ((object(object(error).cause).code ?? object(error).code) === "55P03") throw new PluginTaskOwnershipPendingError();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
export function assertPluginTaskOwnershipDispatch(lease: Lease, operation: OwnershipOperation) {
|
||||
const claim = object(lease.metadata?.portableTaskPredecessor);
|
||||
if (claim.targetLeaseId !== lease.id || claim.operation === undefined ||
|
||||
operation.targetGeneration !== leaseGeneration(lease) ||
|
||||
nativeSha256(claim.operation) !== nativeSha256(operation)) return unavailable();
|
||||
}
|
||||
|
||||
/** Reconcile the declared predecessor before admitting the task on this lease. */
|
||||
export async function reconcilePluginTaskAdmission(db: Db, lease: Lease,
|
||||
dispatch: (operation: OwnershipOperation) => Promise<PluginEnvironmentTaskResult>, signal?: AbortSignal): Promise<void> {
|
||||
let sourceLeaseId = lease.metadata?.portableTaskPreviousLeaseId;
|
||||
let policy: OwnershipOperation["policy"] = "ordinary";
|
||||
if (lease.metadata?.portableTaskPredecessor !== undefined) {
|
||||
const claim = object(lease.metadata.portableTaskPredecessor);
|
||||
const operation = environmentTaskOperationSchema.safeParse(claim.operation);
|
||||
if (!operation.success || operation.data.kind !== "ensure_ownership") return unavailable();
|
||||
assertPluginTaskOwnershipDispatch(lease, operation.data);
|
||||
sourceLeaseId = claim.sourceLeaseId;
|
||||
policy = operation.data.policy;
|
||||
}
|
||||
if (typeof sourceLeaseId !== "string") return unavailable();
|
||||
const request = { companyId: lease.companyId, sourceLeaseId, targetLeaseId: lease.id, policy };
|
||||
let backoffMs = 250;
|
||||
for (;;) {
|
||||
signal?.throwIfAborted();
|
||||
try {
|
||||
const result = await reconcilePluginTaskOwnership(db, request, dispatch);
|
||||
if (result.status === "ready") {
|
||||
signal?.throwIfAborted();
|
||||
return;
|
||||
}
|
||||
} catch (error) {
|
||||
if (!(error instanceof PluginTaskOwnershipPendingError)) throw error;
|
||||
}
|
||||
if (!signal) throw new PluginTaskOwnershipPendingError();
|
||||
await delay(backoffMs, undefined, { signal });
|
||||
backoffMs = Math.min(backoffMs * 2, 5_000);
|
||||
}
|
||||
}
|
||||
|
||||
export async function readPortableTaskReseed(db: Db, input: {
|
||||
companyId: string;
|
||||
runId: string;
|
||||
lease: Pick<Lease, "id" | "metadata">;
|
||||
description?: PluginTaskDescription;
|
||||
runnerTask: (operation: { kind: "describe" }) => Promise<unknown>;
|
||||
}): Promise<{ description: PluginTaskDescription; reseed: { previousSessionId: string | null } | null } | null> {
|
||||
if (!input.lease.metadata?.portableTaskPreviousLeaseId) return null;
|
||||
const readCurrent = async () => (await db.select({
|
||||
metadata: environmentLeases.metadata, providerLeaseId: environmentLeases.providerLeaseId,
|
||||
}).from(environmentLeases).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.companyId),
|
||||
eq(environmentLeases.heartbeatRunId, input.runId))))[0];
|
||||
|
||||
let description = input.description;
|
||||
let current = description ? await readCurrent() : undefined;
|
||||
if (!description || current?.providerLeaseId !== description.taskId || current.metadata?.portableTaskOwnershipReady !== true) {
|
||||
const result = environmentTaskResultSchema.parse(await input.runnerTask({ kind: "describe" }));
|
||||
if (result.kind !== "description") throw new Error("Plugin Runner description unavailable");
|
||||
description = result;
|
||||
current = await readCurrent();
|
||||
}
|
||||
if (current?.providerLeaseId !== description.taskId || current.metadata?.portableTaskOwnershipReady !== true) return unavailable();
|
||||
if (current.metadata.portableTaskCheckpointUnavailable !== true) return { description, reseed: null };
|
||||
|
||||
const predecessor = object(current.metadata.portableTaskPredecessor);
|
||||
const identity = object(object(predecessor.operation).identity);
|
||||
const sessionId = identity.normalizedSessionId;
|
||||
return { description, reseed: { previousSessionId: typeof sessionId === "string" && sessionId.trim() ? sessionId : null } };
|
||||
}
|
||||
|
||||
async function readReadyPortableTaskClaim(db: Db, companyId: string, targetLeaseId: string) {
|
||||
const [target] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId),
|
||||
eq(environmentLeases.id, targetLeaseId)));
|
||||
if (!target?.metadata?.portableTaskPredecessor) return undefined;
|
||||
if (target.metadata.portableTaskCheckpointUnavailable === true) return undefined;
|
||||
const claim = object(target.metadata.portableTaskPredecessor);
|
||||
const operation = environmentTaskOperationSchema.safeParse(claim.operation);
|
||||
if (target.status !== "active" || target.releasedAt || target.metadata.portableTaskOwnershipReady !== true ||
|
||||
target.metadata.portableTaskSuccessor !== undefined || target.metadata.runnerTaskRetiring === true ||
|
||||
target.expiresAt && target.expiresAt.getTime() <= Date.now() ||
|
||||
!operation.success || operation.data.kind !== "ensure_ownership" ||
|
||||
typeof claim.sourceLeaseId !== "string") return unavailable();
|
||||
assertPluginTaskOwnershipDispatch(target, operation.data);
|
||||
const [source] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId),
|
||||
eq(environmentLeases.id, claim.sourceLeaseId)));
|
||||
if (!source || source.provider !== target.provider || source.providerLeaseId !== operation.data.sourceTaskId ||
|
||||
leaseGeneration(source) !== operation.data.sourceGeneration ||
|
||||
nativeSha256(source.metadata?.portableTaskSuccessor ?? null) !== nativeSha256(claim)) return unavailable();
|
||||
return { source, target, operation: operation.data };
|
||||
}
|
||||
|
||||
function matchesAttachedPortableTaskIdentity(
|
||||
target: Lease, identity: Record<string, unknown>, runId: string,
|
||||
) {
|
||||
const saved = object(target.metadata?.runnerTaskIdentity);
|
||||
const attached = {
|
||||
runnerInstanceId: saved.runnerId, environmentLeaseId: saved.leaseId,
|
||||
runId: saved.runId, normalizedSessionId: saved.sessionId, turnId: saved.turnId, itemId: saved.itemId,
|
||||
};
|
||||
return target.heartbeatRunId === runId && identity.runId === runId &&
|
||||
Object.entries(attached).every(([key, value]) => typeof value === "string" && value.length > 0 && identity[key] === value);
|
||||
}
|
||||
|
||||
function matchesPortableLineage(source: OwnershipOperation["identity"], identity: Record<string, unknown>) {
|
||||
return source.environmentLeaseId === identity.environmentLeaseId && source.runnerInstanceId === identity.runnerInstanceId &&
|
||||
source.normalizedSessionId === identity.normalizedSessionId;
|
||||
}
|
||||
|
||||
export async function assertAttachedPortableTask(db: Db, input: {
|
||||
companyId: string; leaseId: string; taskId: string; runId: string; identity: Record<string, unknown>;
|
||||
}) {
|
||||
const claim = await readReadyPortableTaskClaim(db, input.companyId, input.leaseId);
|
||||
if (!claim || claim.target.providerLeaseId !== input.taskId ||
|
||||
!matchesAttachedPortableTaskIdentity(claim.target, input.identity, input.runId) ||
|
||||
!matchesPortableLineage(claim.operation.identity, input.identity)) return unavailable();
|
||||
}
|
||||
|
||||
export async function readyPortableTaskSource(db: Db, companyId: string, targetLeaseId: string, identity: unknown,
|
||||
scope: { attachedRunId?: string; selectedSessionId?: string; accountIdentity?: string } = {}) {
|
||||
const claim = await readReadyPortableTaskClaim(db, companyId, targetLeaseId);
|
||||
if (!claim) return undefined;
|
||||
const { source, target, operation } = claim;
|
||||
if ("accountIdentity" in scope && (!scope.accountIdentity || source.metadata?.runnerTaskAccountIdentity !== scope.accountIdentity))
|
||||
return unavailable();
|
||||
const candidate = object(identity);
|
||||
const { attachedRunId, selectedSessionId } = scope;
|
||||
if (selectedSessionId && selectedSessionId !== operation.identity.normalizedSessionId) {
|
||||
if (!attachedRunId || target.heartbeatRunId !== attachedRunId) return unavailable();
|
||||
const [admitted] = await db.select({ lease: environmentLeases, unlaunched: unlaunchedPortableTaskLease() })
|
||||
.from(environmentLeases).innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId),
|
||||
eq(heartbeatRuns.companyId, environmentLeases.companyId)))
|
||||
.innerJoin(issues, and(eq(issues.id, environmentLeases.issueId), eq(issues.companyId, environmentLeases.companyId)))
|
||||
.where(and(eq(environmentLeases.id, targetLeaseId), eq(environmentLeases.companyId, companyId),
|
||||
eq(environmentLeases.providerLeaseId, target.providerLeaseId!),
|
||||
sql`${environmentLeases.metadata}->'portableTaskPredecessor' = ${JSON.stringify(target.metadata?.portableTaskPredecessor)}::jsonb`,
|
||||
eq(heartbeatRuns.id, attachedRunId), eq(heartbeatRuns.status, "running"),
|
||||
sql`${heartbeatRuns.resultJson}->'nativeCancellation' is null`, eq(heartbeatRuns.nativeSessionId, selectedSessionId),
|
||||
sql`${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'session'->>'normalizedSessionId' = ${selectedSessionId}`,
|
||||
eq(heartbeatRuns.nativeIssueId, issues.id), eq(issues.executionRunId, heartbeatRuns.id),
|
||||
eq(issues.assigneeAgentId, heartbeatRuns.agentId), eq(issues.status, "in_progress")));
|
||||
if (!admitted) return unavailable();
|
||||
if (identity == null && admitted.unlaunched) return undefined;
|
||||
if (admitted.lease.metadata?.runnerTaskDispatched === true &&
|
||||
candidate.normalizedSessionId === selectedSessionId && candidate.environmentLeaseId === target.id &&
|
||||
matchesAttachedPortableTaskIdentity(admitted.lease, candidate, attachedRunId)) return undefined;
|
||||
return unavailable();
|
||||
}
|
||||
if (attachedRunId && target.heartbeatRunId === attachedRunId && candidate.runId === attachedRunId) {
|
||||
if (matchesAttachedPortableTaskIdentity(target, candidate, attachedRunId) &&
|
||||
matchesPortableLineage(operation.identity, candidate)) return undefined;
|
||||
return unavailable();
|
||||
}
|
||||
if (Object.entries(operation.identity).some(([key, value]) => candidate[key] !== value)) return unavailable();
|
||||
return { taskId: operation.sourceTaskId, identity: operation.identity };
|
||||
}
|
||||
@@ -19,6 +19,7 @@ export async function resolvePluginTaskResume(db: Db, input: {
|
||||
const identity = row.lease.metadata?.runnerTaskIdentity as Record<string, unknown> | undefined;
|
||||
const consumer = row.lease.metadata?.runnerTaskSourceConsumer as { leaseId?: unknown; runId?: unknown; resumeFrom?: unknown } | null | undefined;
|
||||
return row.lease.id !== input.leaseId && row.run.id === input.priorIdentity.runId &&
|
||||
row.lease.metadata?.portableTaskSuccessor === undefined &&
|
||||
(!Object.hasOwn(row.lease.metadata ?? {}, "runnerTaskSourceConsumer") ||
|
||||
consumer?.leaseId === input.leaseId && consumer.runId === next?.run.id && consumer.resumeFrom === row.lease.providerLeaseId) &&
|
||||
(input.inspectCheckpoint || row.lease.metadata?.runnerTaskSessionRetained === true) &&
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"packages/plugins/sdk/src/environment-stop-error.ts": "b0e03f8aa2b2170ba095da5c7138050fced3f3cb",
|
||||
"packages/plugins/sdk/src/environment-sync-error.ts": "7d6a51a95da49a7da4525aebbab6382dc5c44ae8",
|
||||
"packages/plugins/sdk/src/environment-task-error.ts": "98aae8a06743b38884b68cb9a3ec44c57b34c0e4",
|
||||
"packages/plugins/sdk/src/environment-tasks.ts": "98e5f69a00b81eb6e8c25bdb5fcdc22e5485ef18",
|
||||
"packages/plugins/sdk/src/environment-tasks.ts": "de583d809187f0677e25b95f22c3c42961a9d2e2",
|
||||
"packages/plugins/sdk/src/host-client-factory.ts": "e0301c390ea13f2340b669383f07af2455e48c8a",
|
||||
"packages/plugins/sdk/src/idle-drain.test.ts": "cc59f7881e7d46c89395a1d93d74f6c6b9b0a64a",
|
||||
"packages/plugins/sdk/src/idle-drain.ts": "ae6e4c6be4657a3d541396922b3bc551b8db557f",
|
||||
@@ -408,7 +408,7 @@
|
||||
"packages/shared/src/types/issue.ts": "725bd6f677b9d461460697d1ce10dee739a66ebe",
|
||||
"packages/shared/src/types/live.ts": "b90e55e4de8600c6fc17d7fa8d89079d4ae655ec",
|
||||
"packages/shared/src/types/native-finalization.ts": "c026a73003ae232fea4f7624d561493b383df358",
|
||||
"packages/shared/src/types/plugin.ts": "016059b873c7189b8e65bb0dfa1ffa039c97c2ca",
|
||||
"packages/shared/src/types/plugin.ts": "22e1e35307f91fdf5c210b86bae7746594669516",
|
||||
"packages/shared/src/types/project.ts": "78eeea3694ce860d5d0be59c158c399c03368078",
|
||||
"packages/shared/src/types/quota.ts": "a0686a9afb3111a4de093f66200dc0c9152e883e",
|
||||
"packages/shared/src/types/resource-memberships.ts": "9eab6fad50facf3e489f8c40e5bedd92e1d1faf8",
|
||||
@@ -490,7 +490,7 @@
|
||||
"packages/shared/src/validators/onboarding-seed.ts": "de302a8400d4199053f379d9ed853522a3ac6d32",
|
||||
"packages/shared/src/validators/partial.ts": "fa60ec7481fc14175365e9e500c814b159f1ab53",
|
||||
"packages/shared/src/validators/plugin.test.ts": "23bc443850d46d815877a71a44625fd42aebb00a",
|
||||
"packages/shared/src/validators/plugin.ts": "2a90d3d00b64f802b9c30f957e97702250ab2db4",
|
||||
"packages/shared/src/validators/plugin.ts": "782e051567729c2eb91a3b80e883b4a8ed996646",
|
||||
"packages/shared/src/validators/project.ts": "6ea168f5faf3161e6004ea47274f60634c77b219",
|
||||
"packages/shared/src/validators/provider-trace.ts": "b38b6200a2e51513f5dca0f8b008f2fe68175e82",
|
||||
"packages/shared/src/validators/resource-memberships.ts": "b124ac8a8f24b4dbb03ae82bcc87541f6b4ecdfd",
|
||||
|
||||
Reference in new issue
Block a user