mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 17:51:58 +01:00
fix(runner): preserve cleanup ownership across startup failures (#18658)
## Thinking Path > - Paperclip controls which run owns an agent task. > - A plugin can continue accepted work after Core stops waiting. > - A late task-creation reply can therefore arrive after startup cancellation. > - Core must keep that work owned until its resource identity and cleanup outcome are known. > - This PR supplies the cleanup and recovery support before shared startup is enabled. > - The dependent activation PR can then reuse transport data within a bounded startup context. ## Linked Issues or Issue Description Refs #18405. This is the independently mergeable prerequisite for that stacked activation PR. **What existing behavior does this improve?** Cleanup of explicitly recorded startup work that outlives its caller. **Current behavior** Caller timeout, worker exit, or failed cleanup-intent persistence can leave work without a complete recovery path. **Proposed behavior** Retain the cleanup obligation, use existing Stop and recovery queues, and release ownership only after the exact task and accepted work have stopped. Ordinary calls keep their existing startup payloads and budgets. This PR does not create startup contexts or advertise transport reuse. **Reason and benefit** Separate the failure-handling contract from activation so each can be reviewed on its own. This support PR has **no standalone TTFT speedup**. ## What Changed - Add the startup cleanup marker, exact ownership checks, verified completion, and acknowledgement helpers. - Connect recorded cleanup obligations to existing Stop, deadline reconciliation, restart recovery, and retry scheduling. - Retain typed cleanup responsibility when a database write fails. Keep the nonterminal run from being treated as ordinary completed cleanup. - Observe late acquisition results and save resource identities before retiring them. Retain real host writes and received results across worker exit or persistence failure. - Queue abandoned replacement tasks through the existing creation journal and cleanup machinery. Require the replacement's own Stop proof. - Accept an explicit startup context in the internal bridge for lifetime tracking. Automatic context creation, marker production, capability advertisement, transport capture, and executor activation belong to #18405. - Add PostgreSQL and worker IPC regressions, including legacy payload assertions. ## Verification Initial split source `8fb482d8be9e70101b0cfdd0830f41cf5b8e3cf4` had tree `95508bd1859e991670cad8afd4de8d2167009ac5`. **190 distinct targeted tests pass**, along with server typechecking. The selected checks cover real PostgreSQL cleanup/ownership, worker IPC, the bridge and creation journal, ordinary/distributed Stop, restart and deadline recovery, idle-sleep behavior, and the unchanged legacy warm-task suite. Independent recovery and worker reviews found no blocking issue. The first correction, `b6342d42042601b376da38dc8e1dba9ca80e9539`, fixes the replacement cleanup finding with the existing startup-cleanup marker, scoped to contextual abandoned task creations. The replacement was never prepared, so preparation-stop proof is not required. No dispatch is fabricated and no normal-startup query is added. The actual queue/sweep/driver/receipt/cancellation regression fails before the fix and passes after it: void, running or foreign-task status retains ownership; exact-task stopped status permits completion even when Release returns void. **64 affected targeted cases and server typechecking pass**. Independent review found no blocker. Current source is `f53af72af76f263096c3c2e788506e6faef9474f`. It fixes both recovery findings from the prior head's Apex 3/5: orphan recovery now preserves pending or completed-but-not-cancelled cleanup through its existing final-write fence, and exact late identities can reach verified cleanup after reconciliation or restart. The regression selection went from four failing assertions to six passing cases. Affected selections cover 217 distinct cases; the exact final six-case regression and server typecheck pass. Independent reviewers found no blocker in either fix. Current-head standard review is 5/5 and all CI checks pass (54 successes, three expected skips, plus Snyk). Apex also completed at **5/5** on this exact head, with zero unresolved threads. The requested independent availability audit is complete on this head. It found an activation-dependent operability gap: an acquisition response that is never received can leave an unknown creation identity with no supported task inspection/reconciliation path, while cleanup keeps the run owner reserved. This does not occur in ordinary lower-only calls. The contract and recovery policy need follow-up before activation; the external review scores do not resolve that availability tradeoff. The split preserves the accepted implementation at `21db7b3851` after its clean merge with frozen master `946735d03a55c1fb4cc5434d6bf1d87742daa9df`. The initial dependent activation PR restored exact aggregate tree `f68c6895f87c7b93ae03e6324d955d2659a99dce`. Both subsequent cleanup corrections are integrated into upper head `27978f294a02da3992585c7f4daa73cd8fcf99f8`; its current aggregate tree is `a0b66c33f5e22745f034f45d24d213f5c210cf16`. Lower-only payload tests cover the absence of capability advertisement and transport projection. Only targeted local tests and typechecking are used. Full regression coverage runs in GitHub CI. No deployment or ROM qualification is part of this change. The lower audit covered connected worker, creation-journal, Stop, restart, deadline and orphan-recovery paths. A new isolated real-PostgreSQL characterization reproduced the unknown-ID hold through actual recovery, cleanup verification and run-control cancellation. It models a lost acquisition response; it is not a live provider or complete IPC test. No source edits were made by the audit. The upper/composition audit remains required after its review cycle. ## Risks - Recording a replacement result now commits with the existing cleanup notification. This conditional replacement path adds transaction/notification responses; common retained startup is unchanged. - This adds cleanup and recovery state handling. A missing receipt or permanently inaccessible old backend can keep ownership held indefinitely. Temporary outages retry; an ordinary retry is not evidence that a deleted or replaced backend stopped. - A caller timeout does not undo work already accepted by a plugin. Unknown resource creation remains unresolved until its identity/outcome is established. Current task creation has no supported inspect-by-operation or operator reconciliation API for a permanently lost identity; repeated Stop alone cannot resolve it. - The support code understands records produced by the dependent activation PR. It does not enable the shared startup policy by itself. - Existing legacy callers must retain their behavior; context-only automatic replacement cleanup remains explicitly scoped. ## Model Used OpenAI Codex, GPT-6 family. The exact model ID and context window are not exposed in this session. Tool-assisted implementation, targeted testing, and independent subagent review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, with available version and capability details - [x] I have checked ROADMAP.md and confirmed this is the requested split of existing work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue with the required fields - [x] I have not referenced internal issues or private links - [x] My branch name describes the change and contains no internal ticket ID - [x] Targeted local tests pass on this source - [x] I have added or updated relevant tests - [x] I have updated relevant documentation - [x] I have documented the risks - [x] All current-head GitHub CI gates are green - [x] Current-head standard Greptile and Apex are 5/5 with no unresolved findings - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
b4315943e8
commit
a9b18e1cbe
24 files changed
+1829
-106
No files matched your search
@@ -1996,6 +1996,24 @@ That is the execution contract Paperclip should present to operators.
|
||||
|
||||
### Cancellation during native startup
|
||||
|
||||
Core supports verified cleanup for explicitly recorded startup obligations. This
|
||||
support does not create startup contexts, advertise transport support, or capture
|
||||
a transport binding. Existing callers retain their legacy payloads and startup
|
||||
budgets. This support alone makes no startup-latency claim.
|
||||
|
||||
Restart recovery uses the claim breadcrumb only while the current run has no
|
||||
first session checkpoint and the prior controller and context deadlines have
|
||||
expired. A later preparation breadcrumb requires the stronger preparation-stop
|
||||
proof. A failed cleanup-intent write retains the original typed failure and
|
||||
rearms the existing restart retry after 30 seconds; the running run continues to
|
||||
block idle sleep. Recovery queues cleanup rather than replaying a provider. A completed result
|
||||
or a newer owner wins. The existing adopted-authentication and chat-workspace
|
||||
ownership holds retain their separate behavior. The startup budget does not bound
|
||||
the duration of remote cleanup. Existing Stop receipts and process cleanup still gate
|
||||
destructive lifecycle or binding changes. Operator plugin restarts and external
|
||||
machine changes can still interrupt startup.
|
||||
|
||||
|
||||
Cancellation records a preparation fence while holding the run row lock. Native
|
||||
runtime selection checks that fence, the running status, and the current startup
|
||||
controller lease in the same transaction that creates the native coordinator.
|
||||
@@ -2034,6 +2052,15 @@ preserving history, unknown action outcomes, and attempt counts. Pauses, approva
|
||||
budgets, task ownership, and terminal task status still gate admission. No
|
||||
automatic provider replay is authorized by a cancelled startup.
|
||||
|
||||
A failed cleanup attempt saves its next retry time in the existing control-work
|
||||
deadlines. The startup marker retains the original admission deadline. Both
|
||||
candidate batches use stable deadline order, and pending cleanup cannot enter
|
||||
ordinary forced finalization. A late replacement-task reply remains owned until
|
||||
its resource identity is durable. It is queued for cleanup instead of attached
|
||||
after cancellation. The old task's Stop receipt cannot settle cleanup of a new
|
||||
replacement task. A lost reply with no known resource identity remains unresolved
|
||||
until provider inspection or operator reconciliation establishes its outcome.
|
||||
|
||||
### Delivering queued messages after a legacy run stops
|
||||
|
||||
The legacy queued-message Interrupt action accepts a null `targetRunId` when
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
|
||||
import { createOwnedSandbox } from "../services/sandbox-creation-operations.js";
|
||||
import { createOwnedSandbox, recordSandboxCreationResource } from "../services/sandbox-creation-operations.js";
|
||||
import { mkdtemp, writeFile, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
@@ -116,15 +116,18 @@ describe.skipIf(!support.supported)("environment cleanup scheduling", () => {
|
||||
]));
|
||||
await quiet();
|
||||
});
|
||||
it("settles a deleted-company creation as manual history across restart without idle polling", async () => {
|
||||
it.each([false, true])("settles a deleted-company creation as manual history across restart without idle polling (contextual=%s)", async contextual => {
|
||||
await start(); await quiet();
|
||||
await createOwnedSandbox(db, { companyId, environmentId, provider: "fixture", driverKey: "fixture", config: {}, metadata: {} },
|
||||
const created = await createOwnedSandbox(db, { companyId, environmentId, provider: "fixture", driverKey: "fixture", config: {},
|
||||
metadata: contextual ? { runnerTaskLease: true, runnerTaskCreationContextId: randomUUID() } : {} },
|
||||
async () => ({ providerLeaseId: "fixture-deleted-company" }));
|
||||
await db.delete(companies).where(eq(companies.id, companyId));
|
||||
await settle(31 * 60_000);
|
||||
await vi.waitFor(async () => expect(await db.select().from(sandboxCreationOperations)).toEqual([
|
||||
expect.objectContaining({ state: "needs_reconciliation", metadata: expect.objectContaining({ creationReconciliationReason: "company_deleted" }) }),
|
||||
]));
|
||||
if (contextual) await recordSandboxCreationResource(db, { id: String(created.metadata!.creationOperationId), companyId, environmentId,
|
||||
providerLeaseId: created.providerLeaseId });
|
||||
await quiet();
|
||||
worker!.stop(); await coordinator!.stop();
|
||||
await start(); await quiet();
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns, plugins } from "@paperclipai/db";
|
||||
import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns, plugins, sandboxCreationOperations } from "@paperclipai/db";
|
||||
import { resolvePluginTaskResume } from "../services/environment-task-runtime.js";
|
||||
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
|
||||
import { environmentRuntimeService } from "../services/environment-runtime.js";
|
||||
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
|
||||
import { NativeExecutionContext } from "../services/native-runtime/native-execution-context.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
@@ -20,6 +21,108 @@ databaseSuite("environment task lease authority", () => {
|
||||
});
|
||||
afterAll(async () => { await stopDb?.(); });
|
||||
|
||||
it("replaces a stopped warm candidate within its tracked startup context", async () => {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID(), runId = randomUUID(), leaseId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Warm candidate", issuePrefix: `T${companyId.slice(0, 7).replace(/-/g, "").toUpperCase()}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", adapterType: "paperclip_runner" });
|
||||
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "fixture-warm-replacement", version: "1", status: "ready",
|
||||
manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }] } as never });
|
||||
await db.insert(environments).values({ id: environmentId, companyId, name: `Fixture ${environmentId}`, driver: "plugin",
|
||||
config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "manual", status: "running" });
|
||||
await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, heartbeatRunId: runId,
|
||||
provider: "plugin:fixture:tasks", providerLeaseId: "stopped-task", status: "active", leasePolicy: "reuse_by_environment",
|
||||
metadata: { driver: "plugin", pluginId, pluginKey: pluginId, driverKey: "tasks", runnerTaskLease: true, runnerTaskReused: true } });
|
||||
const calls: string[] = [];
|
||||
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask", "environmentAcquireLease"] }),
|
||||
call: async (_plugin: string, method: string, params: any, _timeout?: number, _sink?: unknown, settled?: () => void) => {
|
||||
calls.push(method === "environmentTask" ? params.operation.kind : method);
|
||||
try {
|
||||
if (method === "environmentAcquireLease") return { providerLeaseId: "replacement-task" };
|
||||
if (params.operation.kind === "status") return { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true };
|
||||
if (params.operation.kind === "describe") return { kind: "description", taskId: params.taskId,
|
||||
runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` }, executionContextVersion: 1 };
|
||||
return { kind: "accepted", taskId: params.taskId };
|
||||
} finally { settled?.(); }
|
||||
},
|
||||
} as unknown as PluginWorkerManager;
|
||||
const context = new NativeExecutionContext({ binding: { companyId, agentId, runId } } as never, 1_000);
|
||||
try {
|
||||
await expect(environmentRuntimeService(db, { pluginWorkerManager: workers }).task({ companyId, leaseId,
|
||||
operation: { kind: "describe" }, executionContext: context })).resolves.toMatchObject({ kind: "description", taskId: "replacement-task" });
|
||||
expect(calls).toEqual(["status", "status", "complete", "environmentAcquireLease", "describe"]);
|
||||
expect(context.pending).toBe(false);
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId));
|
||||
expect(lease.providerLeaseId).toBe("replacement-task");
|
||||
expect(lease.metadata?.runnerTaskReused).toBe(false);
|
||||
expect(JSON.stringify(lease.metadata)).not.toContain(context.id);
|
||||
} finally { context.finish(); }
|
||||
});
|
||||
|
||||
it.each(["late", "post-result", "retiring"])("queues a replacement instead of attaching after %s startup cancellation", async mode => {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID(), runId = randomUUID(), leaseId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Warm candidate", issuePrefix: `T${companyId.slice(0, 7).replace(/-/g, "").toUpperCase()}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", adapterType: "paperclip_runner" });
|
||||
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "fixture-warm-replacement", version: "1", status: "ready",
|
||||
manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }] } as never });
|
||||
await db.insert(environments).values({ id: environmentId, companyId, name: `Fixture ${environmentId}`, driver: "plugin",
|
||||
config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "manual", status: "running" });
|
||||
await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, heartbeatRunId: runId,
|
||||
provider: "plugin:fixture:tasks", providerLeaseId: "stopped-task", status: "active", leasePolicy: "reuse_by_environment",
|
||||
metadata: { driver: "plugin", pluginId, pluginKey: pluginId, driverKey: "tasks", runnerTaskLease: true, runnerTaskReused: true } });
|
||||
const calls: string[] = [];
|
||||
let reply!: () => Promise<void>, acquiring!: () => void;
|
||||
const entered = new Promise<void>(resolve => { acquiring = resolve; });
|
||||
const workers = { getWorker: () => ({ supportedMethods: ["environmentTask", "environmentAcquireLease"] }),
|
||||
call: async (_plugin: string, method: string, params: any, _timeout?: number, _sink?: unknown, settled?: () => void, _contextId?: string, observe?: (result: any) => Promise<void>) => {
|
||||
calls.push(method === "environmentTask" ? params.operation.kind : method);
|
||||
try {
|
||||
if (method === "environmentAcquireLease") return await new Promise(resolve => {
|
||||
reply = async () => { const result = { providerLeaseId: "replacement-task", metadata: { binding: "fresh" } };
|
||||
await observe?.(result);
|
||||
if (mode === "retiring") await db.update(environmentLeases).set({ metadata: { driver: "plugin", pluginId, driverKey: "tasks", runnerTaskRetiring: true } }).where(eq(environmentLeases.id, leaseId));
|
||||
resolve(result); };
|
||||
acquiring();
|
||||
});
|
||||
if (params.operation.kind === "status") return { kind: "status", taskId: params.taskId, phase: "completed", executionStopped: true };
|
||||
if (params.operation.kind === "describe") return { kind: "description", taskId: params.taskId,
|
||||
runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` }, executionContextVersion: 1 };
|
||||
return { kind: "accepted", taskId: params.taskId };
|
||||
} finally { settled?.(); }
|
||||
},
|
||||
} as unknown as PluginWorkerManager;
|
||||
const context = new NativeExecutionContext({ binding: { companyId, agentId, runId } } as never, 1_000);
|
||||
if (mode === "post-result") {
|
||||
const wait = context.wait.bind(context);
|
||||
vi.spyOn(context, "wait").mockImplementation(operation => wait(async () => {
|
||||
const value = await operation(); context.abort(new Error("Cancelled after recording")); return value;
|
||||
}));
|
||||
}
|
||||
try {
|
||||
const request = environmentRuntimeService(db, { pluginWorkerManager: workers }).task({ companyId, leaseId,
|
||||
operation: { kind: "describe" }, executionContext: context });
|
||||
const rejected = expect(request).rejects.toThrow("operation unavailable");
|
||||
await entered;
|
||||
if (mode === "late") {
|
||||
context.abort(new Error("Startup cancelled"));
|
||||
await rejected;
|
||||
expect(context.pending).toBe(true);
|
||||
}
|
||||
await reply(); await rejected;
|
||||
expect(context.pending).toBe(false);
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId));
|
||||
expect(lease.providerLeaseId).toBe("stopped-task");
|
||||
const [creation] = await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.runId, runId));
|
||||
expect(creation.state).toBe("cleanup_pending");
|
||||
expect(creation.metadata.runnerTaskCreationContextId).toBe(context.id);
|
||||
const [cleanup] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, creation.leaseId!));
|
||||
expect(cleanup).toMatchObject({ heartbeatRunId: runId, providerLeaseId: "replacement-task", status: "pending_cleanup" });
|
||||
expect(cleanup.metadata).toMatchObject({ runnerTaskLease: true, providerMetadata: { binding: "fresh" } });
|
||||
expect(calls).not.toContain("describe");
|
||||
} finally { context.finish(); }
|
||||
});
|
||||
|
||||
it("authorizes a new task lease only after scoped prior execution has stopped", async () => {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), pluginId = randomUUID();
|
||||
const oldRunId = randomUUID(), nextRunId = randomUUID(), oldLeaseId = randomUUID(), nextLeaseId = randomUUID();
|
||||
|
||||
@@ -2,14 +2,16 @@ import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { PgDialect } from "drizzle-orm/pg-core";
|
||||
import { executionWorkspaces, environmentLeases, type Db } from "@paperclipai/db";
|
||||
import { environmentTaskOperationSchema, parseEnvironmentTaskResult } from "@paperclipai/plugin-sdk";
|
||||
import { executeEnvironmentTask } from "../services/environment-task-runtime.js";
|
||||
import { executeEnvironmentTask, settleDispatchedEnvironmentTask } from "../services/environment-task-runtime.js";
|
||||
import { NativeExecutionContext } from "../services/native-runtime/native-execution-context.js";
|
||||
|
||||
const state = vi.hoisted(() => ({ plugin: {} as any, release: vi.fn() }));
|
||||
const state = vi.hoisted(() => ({ plugin: {} as any, release: vi.fn(), identity: vi.fn(), config: vi.fn() }));
|
||||
vi.mock("../services/provider-fresh-workspace.js", () => ({
|
||||
stageProviderFreshWorkspace: async (input: { existing: unknown }) => input.existing,
|
||||
releaseProviderRecoveryTransfers: state.release,
|
||||
}));
|
||||
vi.mock("../services/plugin-registry.js", () => ({ pluginRegistryService: () => ({ getById: vi.fn(async () => state.plugin) }) }));
|
||||
vi.mock("../services/plugin-registry.js", () => ({ pluginRegistryService: () => ({ getById: vi.fn(async () => state.plugin), getConfig: state.config }) }));
|
||||
vi.mock("../services/plugin-agent-runtime-state.js", () => ({ pluginAgentRuntimeIdentity: state.identity }));
|
||||
const leaseId = "10000000-0000-4000-8000-000000000001";
|
||||
const projectId = "10000000-0000-4000-8000-000000000002";
|
||||
const secondProjectId = "10000000-0000-4000-8000-000000000003";
|
||||
@@ -25,7 +27,7 @@ function database(value: ReturnType<typeof row> | null = row(), projectRows = [{
|
||||
const query: any = { from: (table: unknown) => { selectedRows = table === executionWorkspaces ? workspaceRows : table === environmentLeases ? ownerRows : projectRows; return query; }, leftJoin: () => query, where: vi.fn(() => query), limit: () => Promise.resolve(results.shift()), then: (resolve: (rows: typeof projectRows) => unknown) => Promise.resolve(selectedRows).then(resolve as (rows: unknown[]) => unknown) };
|
||||
const sets: Record<string, unknown>[] = [];
|
||||
const update: any = { set: (value: Record<string, unknown>) => { sets.push(value); return update; }, where: () => update,
|
||||
returning: async () => [{ id: leaseId }], then: (resolve: (value: unknown) => unknown) => Promise.resolve([]).then(resolve) };
|
||||
returning: async () => [{ id: leaseId, metadata: value?.lease.metadata, heartbeatRunId: value?.lease.heartbeatRunId }], then: (resolve: (value: unknown) => unknown) => Promise.resolve([]).then(resolve) };
|
||||
const db = { select: () => query, update: () => update,
|
||||
transaction: vi.fn(async (write: (tx: Db) => Promise<unknown>) => write(db as unknown as Db)) };
|
||||
return { db: db as unknown as Db, query, sets, transaction: db.transaction };
|
||||
@@ -37,9 +39,65 @@ const submit = { kind: "submit" as const, preparedLaunchId: "configuration", pro
|
||||
|
||||
beforeEach(() => {
|
||||
state.release.mockReset();
|
||||
state.identity.mockReset().mockResolvedValue({ instanceSecrets: { launcher: "synthetic-transient-binding" } });
|
||||
state.config.mockReset().mockResolvedValue({ configJson: { endpoint: "synthetic" } });
|
||||
state.plugin = { id: "original", pluginKey: "test.provider", status: "ready", manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }] } };
|
||||
});
|
||||
describe("plugin-provided Runner execution", () => {
|
||||
it.each([false, true])("retires an undispatched warm task only with required preparation proof (%s)", async preparationStopped => {
|
||||
const value = row();
|
||||
Object.assign(value.lease.metadata, { runnerTaskDispatched: false,
|
||||
runnerTaskStartupCleanup: { contextId: "startup", requiresPreparationStop: true } });
|
||||
const fixture = database(value), workers = worker();
|
||||
fixture.db.select = ((...args: unknown[]) => (database(value).db.select as (...values: unknown[]) => unknown)(...args)) as Db["select"];
|
||||
let release!: () => void;
|
||||
const drain = new Promise<void>(resolve => { release = resolve; });
|
||||
const drainStartup = vi.fn(() => drain);
|
||||
Object.assign(workers, { drainStartup });
|
||||
const operations: string[] = [];
|
||||
workers.call.mockImplementation(async (...args: any[]) => {
|
||||
const operation = args[2].operation.kind; operations.push(operation);
|
||||
return operation === "status" ? { kind: "status", taskId: "attempt-1", phase: "cancelled", executionStopped: true, preparationStopped }
|
||||
: { kind: "accepted", taskId: "attempt-1" };
|
||||
});
|
||||
const retiring = settleDispatchedEnvironmentTask(fixture.db, workers as never, { companyId: "company", id: leaseId });
|
||||
const checked = retiring.then(value => ({ value, error: null }), error => ({ value: false, error }));
|
||||
await vi.waitFor(() => expect(operations).toContain("stop"));
|
||||
await vi.waitFor(() => expect(operations).toContain("status"));
|
||||
expect(drainStartup).toHaveBeenCalledWith("startup");
|
||||
expect(operations).not.toContain("complete");
|
||||
release();
|
||||
const result = await checked;
|
||||
if (preparationStopped) { expect(result).toEqual({ value: true, error: null }); expect(operations).toEqual(["stop", "status", "complete"]); }
|
||||
else { expect(result.error?.message).toContain("termination is unconfirmed"); expect(fixture.sets).toContainEqual(expect.objectContaining({ status: "pending_cleanup", cleanupStatus: "failed" })); }
|
||||
});
|
||||
it("keeps legacy payloads passive without capability or transport activation", async () => {
|
||||
const context = new NativeExecutionContext({ binding: { companyId: "company", agentId: "agent", runId: "run" } } as never, 1_000);
|
||||
const workers = worker();
|
||||
workers.call.mockImplementation(async (...args: any[]) => { args[5]?.(); return { kind: "description", taskId: "attempt-1", runner: { version: "fixture", digest: `sha256:${"a".repeat(64)}` } }; });
|
||||
try {
|
||||
for (const executionContext of [undefined, context]) {
|
||||
await executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId,
|
||||
operation: { kind: "describe" }, ...(executionContext ? { executionContext } : {}) });
|
||||
const sent = workers.call.mock.calls.at(-1) as unknown as any[];
|
||||
expect(sent[2]).not.toHaveProperty("executionContextSupported");
|
||||
expect(sent[2]).not.toHaveProperty("executionContext");
|
||||
if (!executionContext) expect(sent).toHaveLength(4);
|
||||
}
|
||||
expect(state.identity).not.toHaveBeenCalled();
|
||||
expect(state.config).not.toHaveBeenCalled();
|
||||
expect(context.pending).toBe(false);
|
||||
} finally { context.finish(); }
|
||||
});
|
||||
it.each(["json", "wrong-run"])("rejects a %s context before the worker", async mode => {
|
||||
const context = new NativeExecutionContext({ binding: { companyId: "company", agentId: "agent", runId: mode === "wrong-run" ? "other" : "run" } } as never, 1_000);
|
||||
const workers = worker();
|
||||
try {
|
||||
await expect(executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: { kind: "describe" },
|
||||
executionContext: mode === "json" ? JSON.parse(JSON.stringify(context)) : context })).rejects.toThrow("native_execution_context");
|
||||
expect(workers.call).not.toHaveBeenCalled();
|
||||
} finally { context.finish(); }
|
||||
});
|
||||
it("forwards a retained fresh directory intent on later launch calls and checks provider support", async () => {
|
||||
const value = row();
|
||||
value.lease.metadata.runnerTaskProjectAuthValidation.projectIds = [];
|
||||
|
||||
@@ -1,10 +1,15 @@
|
||||
import { EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, authUsers, companies, companyMemberships, createDb, heartbeatRuns, issues } from "@paperclipai/db";
|
||||
import { agents, authUsers, companies, companyMemberships, createDb, environmentLeases, environments, heartbeatRuns, issues, nativeRunFinalizations } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { NativeStartupCleanupPendingError, readNativeStartupCleanup } from "../services/native-runtime/native-startup-cleanup.js";
|
||||
import { prepareNativeHeartbeatRun } from "../services/native-runtime/prepare-native-run.js";
|
||||
import { environmentService } from "../services/environments.js";
|
||||
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
|
||||
import { createIsolatedTestHome } from "./helpers/isolated-test-home.js";
|
||||
|
||||
const execute = vi.hoisted(() => vi.fn());
|
||||
vi.mock("../adapters/index.js", async () => {
|
||||
@@ -70,4 +75,108 @@ suite("native terminal-run cleanup admission", () => {
|
||||
expect((await heartbeat.getRun(nextId))?.status).toBe("succeeded");
|
||||
expect(execute).toHaveBeenCalledTimes(3);
|
||||
}, 30_000);
|
||||
|
||||
it("retries a failed startup intent through timer failure and drain before verified cancellation", async () => {
|
||||
const home = createIsolatedTestHome(), originalHome = process.env.PAPERCLIP_HOME;
|
||||
process.env.PAPERCLIP_HOME = home.home;
|
||||
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), leaseId = randomUUID();
|
||||
const contextId = randomUUID(), leaseOwner = randomUUID(), taskId = randomUUID(), deadlineAtMs = Date.now() - 1;
|
||||
const originalSetTimeout = globalThis.setTimeout;
|
||||
const retries: Array<{ timer: ReturnType<typeof setTimeout>; fire(): void }> = [];
|
||||
const timeout = vi.spyOn(globalThis, "setTimeout").mockImplementation(((callback: (...args: unknown[]) => void, delay?: number, ...args: unknown[]) => {
|
||||
if (typeof delay === "number" && delay <= 30_000 && new Error().stack?.includes("scheduleNativeSessionResumeDispatch")) {
|
||||
const timer = originalSetTimeout(() => {}, 3_600_000);
|
||||
retries.push({ timer, fire() { clearTimeout(timer); callback(...args); } });
|
||||
return timer;
|
||||
}
|
||||
return originalSetTimeout(callback, delay, ...args);
|
||||
}) as typeof setTimeout);
|
||||
let failSuppression = false, runId = "";
|
||||
const runtimeEnv = new Proxy<Record<string, string | undefined>>({}, { get(_target, key) {
|
||||
if (key === "PAPERCLIP_DATABASE_RESTORE_IN_PROGRESS" && failSuppression) {
|
||||
failSuppression = false;
|
||||
throw new Error("Scheduling dependency unavailable");
|
||||
}
|
||||
return undefined;
|
||||
} });
|
||||
const heartbeat = heartbeatService(db, { runtimeEnv });
|
||||
try {
|
||||
await db.insert(companies).values({ id: companyId, name: "Intent retry", issuePrefix: "CIR", defaultResponsibleUserId: "owner" });
|
||||
await db.insert(authUsers).values({ id: "owner", name: "Fixture user", email: "owner@example.test", createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing();
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "owner", status: "active", membershipRole: "member" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", role: "engineer", status: "idle", adapterType: "process",
|
||||
adapterConfig: {}, permissions: {}, runtimeConfig: { heartbeat: { enabled: false, wakeOnDemand: true } } });
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Recover startup", status: "todo", assigneeAgentId: agentId, responsibleUserId: "owner" });
|
||||
const [environment] = await db.insert(environments).values({ companyId, name: "Controlled provider", driver: "plugin", config: {} }).returning();
|
||||
await db.execute(sql`create table test_cleanup_intent_fault (enabled boolean not null)`);
|
||||
await db.execute(sql`insert into test_cleanup_intent_fault values (true)`);
|
||||
await db.execute(sql`create sequence test_cleanup_intent_attempts`);
|
||||
await db.execute(sql.raw(`create function test_reject_cleanup_intent() returns trigger language plpgsql as $$ begin
|
||||
if new.company_id = '${companyId}'::uuid and new.error_code = 'native_startup_cleanup_pending'
|
||||
and exists (select 1 from test_cleanup_intent_fault where enabled) then
|
||||
perform nextval('test_cleanup_intent_attempts'); raise exception 'Cleanup intent write unavailable';
|
||||
end if; return new; end $$`));
|
||||
await db.execute(sql`create trigger test_cleanup_intent_fault before update on heartbeat_runs for each row execute function test_reject_cleanup_intent()`);
|
||||
execute.mockReset();
|
||||
execute.mockImplementation(async (context: { runId: string }) => {
|
||||
runId = context.runId;
|
||||
const [run] = await db.update(heartbeatRuns).set({ runtimeMode: "native", runtimeModeResolvedAt: null }).where(eq(heartbeatRuns.id, runId)).returning();
|
||||
await prepareNativeHeartbeatRun({ db, run, issue: { id: issueId, title: "Recover startup", description: null }, environmentLeaseId: leaseId });
|
||||
await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: { provider: "codex" } }).where(eq(agents.id, agentId));
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt: 1,
|
||||
controllerGeneration: 1, leaseOwner, leaseExpiresAt: new Date(Date.now() - 1_000) });
|
||||
await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId: environment.id, heartbeatRunId: runId,
|
||||
issueId, status: "active", provider: "plugin", providerLeaseId: taskId, leasePolicy: "reuse_by_environment", metadata: { runnerTaskLease: true } });
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) || ${JSON.stringify({
|
||||
nativeStartupContext: { schema: "paperclip.native-startup-context.v1", companyId, agentId, runId,
|
||||
contextId, deadlineAtMs, leaseId, leaseOwner, controllerGeneration: 1, attempt: 1 },
|
||||
})}::jsonb` }).where(eq(heartbeatRuns.id, runId));
|
||||
throw new NativeStartupCleanupPendingError(contextId, deadlineAtMs, leaseOwner, false, leaseId);
|
||||
});
|
||||
await heartbeat.invoke(agentId, "on_demand", { issueId }, "manual");
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
const owner = () => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)).then(rows => rows[0]!);
|
||||
const lease = () => db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)).then(rows => rows[0]!);
|
||||
expect((await db.execute<{ last_value: string }>(sql`select last_value from test_cleanup_intent_attempts`))[0].last_value).toBe("2");
|
||||
expect(await heartbeat.getRun(runId)).toMatchObject({ status: "running", errorCode: null });
|
||||
expect((await owner()).leaseOwner).toBe(leaseOwner);
|
||||
expect(await lease()).toMatchObject({ status: "active", cleanupStatus: null });
|
||||
expect(readNativeStartupCleanup((await heartbeat.getRun(runId))!)).toBeNull();
|
||||
const drainedRetries = retries.length;
|
||||
expect(drainedRetries).toBeGreaterThan(0);
|
||||
await heartbeat.resumeQueuedRunsAfterDrain();
|
||||
expect(retries.length).toBeGreaterThan(drainedRetries);
|
||||
const resumedRetries = retries.length;
|
||||
failSuppression = true;
|
||||
retries.at(-1)!.fire();
|
||||
await vi.waitFor(() => expect(retries.length).toBeGreaterThan(resumedRetries));
|
||||
expect((await owner()).leaseOwner).toBe(leaseOwner);
|
||||
expect((await heartbeat.getRun(runId))?.status).toBe("running");
|
||||
await db.execute(sql`update test_cleanup_intent_fault set enabled = false`);
|
||||
retries.at(-1)!.fire();
|
||||
await vi.waitFor(async () => expect(readNativeStartupCleanup((await heartbeat.getRun(runId))!)?.state).toBe("pending"));
|
||||
await vi.waitFor(async () => expect((await heartbeat.listEvents(runId)).some(event => event.eventType === "native.recovery.transition")).toBe(true));
|
||||
const current = await lease();
|
||||
await environmentService(db).releaseLease(leaseId, "released", { cleanupStatus: "success",
|
||||
remoteExecutionTermination: remoteTerminationReceipt(current, { providerLeaseId: taskId, state: "stopped" }) });
|
||||
await heartbeat.cancelRun(runId, "Startup cleanup retry completed");
|
||||
expect(await heartbeat.getRun(runId)).toMatchObject({ status: "cancelled" });
|
||||
expect(readNativeStartupCleanup((await heartbeat.getRun(runId))!)?.state).toBe("complete");
|
||||
expect(await owner()).toMatchObject({ leaseOwner: null, controlDeadlineAt: null });
|
||||
expect(execute).toHaveBeenCalledOnce();
|
||||
} finally {
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
timeout.mockRestore();
|
||||
for (const retry of retries) clearTimeout(retry.timer);
|
||||
await db.execute(sql`drop trigger if exists test_cleanup_intent_fault on heartbeat_runs`);
|
||||
await db.execute(sql`drop function if exists test_reject_cleanup_intent()`);
|
||||
await db.execute(sql`drop table if exists test_cleanup_intent_fault`);
|
||||
await db.execute(sql`drop sequence if exists test_cleanup_intent_attempts`);
|
||||
if (originalHome === undefined) delete process.env.PAPERCLIP_HOME;
|
||||
else process.env.PAPERCLIP_HOME = originalHome;
|
||||
home.cleanup();
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
});
|
||||
@@ -204,6 +204,19 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps a failed startup intent awake after its local worker counter settles", async () => {
|
||||
const f = await seed();
|
||||
expect(await read()).toEqual(none);
|
||||
const finish = beginIdleTrackedWork();
|
||||
await db.update(heartbeatRuns).set({ status: "running", runtimeMode: "native", nativePhase: "observed",
|
||||
runnerProfileJson: { nativeStartupContext: { contextId: randomUUID(), deadlineAtMs: now - 1 } } }).where(eq(heartbeatRuns.id, f.runId));
|
||||
finish();
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(await read()).toEqual(present);
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.runId));
|
||||
expect(await read()).toEqual(none);
|
||||
});
|
||||
|
||||
it("retains committed budget stop intent until its exact generation is delivered", async () => {
|
||||
const f = await seed();
|
||||
const [policy] = await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
|
||||
|
||||
@@ -0,0 +1,729 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { databaseWorkPending } from "@paperclipai/db/work-signals";
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import type { PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
|
||||
import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
activityLog, agents, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases,
|
||||
environments, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, plugins, sandboxCreationOperations,
|
||||
} from "@paperclipai/db";
|
||||
import { recoveryPostgresSupport, startRecoveryPostgres } from "./helpers/recovery-postgres.js";
|
||||
import { createIsolatedTestHome } from "./helpers/isolated-test-home.js";
|
||||
import { prepareNativeHeartbeatRun } from "../services/native-runtime/prepare-native-run.js";
|
||||
import {
|
||||
acknowledgeNativeStartupCleanup, NATIVE_STARTUP_CLEANUP_PENDING, readNativeStartupCleanup,
|
||||
requestNativeStartupCleanup, verifyNativeStartupCleanup, clearNativeStartupContext,
|
||||
} from "../services/native-runtime/native-startup-cleanup.js";
|
||||
import { isNativeRunnerOwnershipHeld } from "../services/native-runtime/native-runner-ownership.js";
|
||||
import { cancelNativeSession } from "../services/native-runtime/native-session-executor.js";
|
||||
import * as nativeSessionExecutor from "../services/native-runtime/native-session-executor.js";
|
||||
import { NativeExecutionContext } from "../services/native-runtime/native-execution-context.js";
|
||||
import type { NativeExecutionInput } from "../vendor/paperclip-runner/index.js";
|
||||
import { callbacks } from "../services/heartbeat/__tests__/run-control-fixtures.js";
|
||||
import { createHeartbeatRunControl } from "../services/heartbeat/run-control.js";
|
||||
import { nextEnvironmentCleanupAt } from "../services/environment-cleanup-state.js";
|
||||
import { executionControlSchedule } from "../services/execution-control-work.js";
|
||||
import { claimNativeRestartRecoveries } from "../services/native-runtime/native-restart-recovery.js";
|
||||
import { reconcileAbandonedExecutionControl } from "../services/execution-control-reconciliation.js";
|
||||
import { createPluginWorkerManager, type PluginWorkerManager } from "../services/plugin-worker-manager.js";
|
||||
import { hasRemoteTerminationReceipt, remoteTerminationReceipt } from "../services/remote-execution-termination.js";
|
||||
|
||||
import { createOwnedSandbox, queueSandboxCreationCleanup, recordSandboxCreationResource, recoverSandboxCreations } from "../services/sandbox-creation-operations.js";
|
||||
import { environmentService } from "../services/environments.js";
|
||||
import { environmentRuntimeService } from "../services/environment-runtime.js";
|
||||
import { createHeartbeatRecovery, type HeartbeatRecoveryDependencies } from "../services/heartbeat/recovery.js";
|
||||
|
||||
const describePostgres = recoveryPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
describePostgres("durable cleanup after a native startup timeout", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startRecoveryPostgres>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let home: ReturnType<typeof createIsolatedTestHome>;
|
||||
const originalHome = process.env.PAPERCLIP_HOME;
|
||||
const companyId = randomUUID(), agentId = randomUUID();
|
||||
|
||||
beforeAll(async () => {
|
||||
home = createIsolatedTestHome();
|
||||
process.env.PAPERCLIP_HOME = home.home;
|
||||
temporary = await startRecoveryPostgres("native-startup-cleanup-");
|
||||
db = createDb(temporary.connectionString);
|
||||
await db.insert(companies).values({ id: companyId, name: "Startup cleanup", issuePrefix: "NSC" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Runner", role: "engineer", status: "running",
|
||||
adapterType: "paperclip_runner", adapterConfig: { provider: "codex" }, runtimeConfig: {}, permissions: {} });
|
||||
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
||||
|
||||
afterAll(async () => {
|
||||
await temporary?.cleanup();
|
||||
home?.cleanup();
|
||||
if (originalHome === undefined) delete process.env.PAPERCLIP_HOME;
|
||||
else process.env.PAPERCLIP_HOME = originalHome;
|
||||
});
|
||||
|
||||
async function fixture(requiresPreparationStop = false) {
|
||||
const runId = randomUUID(), issueId = randomUUID(), leaseId = randomUUID(), taskId = randomUUID();
|
||||
const contextId = randomUUID(), leaseOwner = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, identifier: `NSC-${issueId}`, title: "Timed out startup",
|
||||
status: "in_progress", priority: "medium", assigneeAgentId: agentId });
|
||||
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running",
|
||||
runtimeMode: "native", nativeIssueId: issueId, invocationSource: "manual", contextSnapshot: { issueId } }).returning();
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
await prepareNativeHeartbeatRun({ db, run, issue: { id: issueId, title: "Timed out startup", description: null }, environmentLeaseId: leaseId });
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt: 1,
|
||||
controllerGeneration: 1, controllerBootId: randomUUID(), controllerPid: process.pid,
|
||||
controllerProcessStartedAt: new Date(), leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000) });
|
||||
const [environment] = await db.insert(environments).values({ name: `Cleanup ${runId}`, driver: "plugin", status: "active", config: {} }).returning();
|
||||
await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId: environment.id, heartbeatRunId: runId,
|
||||
issueId, status: "active", provider: "plugin", providerLeaseId: taskId, leasePolicy: "reuse_by_environment",
|
||||
metadata: { runnerTaskLease: true, runnerTaskReused: true } });
|
||||
const request = { companyId, agentId, runId, contextId, leaseOwner, deadlineAtMs: Date.now() - 1, requiresPreparationStop };
|
||||
const reload = () => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then(rows => rows[0]!);
|
||||
const owner = () => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)).then(rows => rows[0]!);
|
||||
const lease = () => db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)).then(rows => rows[0]!);
|
||||
const stopReceipt = async (override: Record<string, unknown> = {}, preparationStopped = false) => {
|
||||
const current = await lease();
|
||||
const receipt = remoteTerminationReceipt(current, { providerLeaseId: taskId, state: "stopped" });
|
||||
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success",
|
||||
metadata: { ...current.metadata, ...(preparationStopped ? { runnerTaskPreparationStopped: true } : {}), remoteExecutionTermination: { ...receipt, ...override } } }).where(eq(environmentLeases.id, leaseId));
|
||||
};
|
||||
const acceptedResult = async () => {
|
||||
const current = await reload(), resultId = randomUUID();
|
||||
await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId,
|
||||
completionContractId: current.completionContractId!, serverFingerprint: resultId, schemaStatus: "accepted",
|
||||
resultJson: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "Completed work" }, canonicalSha256: "a".repeat(64) });
|
||||
await db.update(nativeRunFinalizations).set({ resultId, phase: "workspace_finalizing" }).where(eq(nativeRunFinalizations.runId, runId));
|
||||
return resultId;
|
||||
};
|
||||
return { request, reload, owner, lease, stopReceipt, acceptedResult };
|
||||
}
|
||||
|
||||
async function saveEarlyContext(f: Awaited<ReturnType<typeof fixture>>) {
|
||||
const run = await f.reload(), owner = await f.owner(), lease = await f.lease();
|
||||
const context = { schema: "paperclip.native-startup-context.v1", companyId, agentId, runId: run.id,
|
||||
contextId: f.request.contextId, deadlineAtMs: f.request.deadlineAtMs, leaseId: lease.id,
|
||||
leaseOwner: owner.leaseOwner, controllerGeneration: owner.controllerGeneration, attempt: owner.attempt };
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson, nativeStartupContext: context } }).where(eq(heartbeatRuns.id, run.id));
|
||||
return context;
|
||||
}
|
||||
|
||||
it.each(["pending", "needs_reconciliation"])("keeps ownership until a %s replacement queued after the first teardown snapshot is verified stopped", async state => {
|
||||
const f = await fixture(true), original = await f.lease();
|
||||
let reply!: (result: { providerLeaseId: string }) => void, creating!: () => void;
|
||||
const entered = new Promise<void>(resolve => { creating = resolve; });
|
||||
const acquiring = createOwnedSandbox(db, { companyId, environmentId: original.environmentId!, heartbeatRunId: f.request.runId,
|
||||
issueId: original.issueId, provider: original.provider!, driverKey: "tasks", config: {}, requestKey: original.providerLeaseId!,
|
||||
metadata: { runnerTaskLease: true, runnerTaskCreationContextId: f.request.contextId } }, async () => new Promise(resolve => { reply = resolve; creating(); }));
|
||||
await entered;
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
await f.stopReceipt({}, true);
|
||||
await expect(verifyNativeStartupCleanup(db, await f.reload(), async () => {})).rejects.toThrow("cleanup is still pending");
|
||||
const [pending] = await db.select().from(sandboxCreationOperations).where(and(
|
||||
eq(sandboxCreationOperations.runId, f.request.runId), eq(sandboxCreationOperations.requestKey, original.providerLeaseId!)));
|
||||
if (state === "needs_reconciliation") {
|
||||
await db.update(sandboxCreationOperations).set({ nextAttemptAt: new Date(0) }).where(eq(sandboxCreationOperations.id, pending.id));
|
||||
await recoverSandboxCreations(db, async () => null);
|
||||
expect((await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id, pending.id)))[0].state).toBe(state);
|
||||
const inspect = vi.fn(async () => null);
|
||||
await recoverSandboxCreations(db, inspect);
|
||||
expect(inspect).not.toHaveBeenCalled();
|
||||
await expect(queueSandboxCreationCleanup(db, { id: pending.id, companyId, environmentId: original.environmentId!,
|
||||
providerLeaseId: "unproven-task" })).rejects.toThrow("ownership changed");
|
||||
}
|
||||
reply({ providerLeaseId: "late-replacement" });
|
||||
const fresh = await acquiring;
|
||||
await recordSandboxCreationResource(db, { id: String(fresh.metadata!.creationOperationId), companyId,
|
||||
environmentId: original.environmentId!, providerLeaseId: fresh.providerLeaseId, providerMetadata: { binding: "actual" } });
|
||||
if (state === "pending") await db.update(sandboxCreationOperations).set({ nextAttemptAt: new Date(0) })
|
||||
.where(eq(sandboxCreationOperations.id, String(fresh.metadata!.creationOperationId)));
|
||||
expect(await nextEnvironmentCleanupAt(db)).not.toBeNull();
|
||||
const restarted = createDb(temporary.connectionString);
|
||||
try { await recoverSandboxCreations(restarted, async () => { throw new Error("A recorded task identity needs no new provider inspection"); }); }
|
||||
finally { await restarted.$client.end(); }
|
||||
const [creation] = await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id, String(fresh.metadata!.creationOperationId)));
|
||||
await expect(verifyNativeStartupCleanup(db, await f.reload(), async () => {})).rejects.toThrow("cleanup is still pending");
|
||||
const replacement = (await environmentService(db).getLeaseById(creation.leaseId!))!;
|
||||
await queueSandboxCreationCleanup(db, { id: creation.id, companyId, environmentId: original.environmentId!, providerLeaseId: fresh.providerLeaseId });
|
||||
expect((await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id, creation.id)))[0].leaseId).toBe(replacement.id);
|
||||
await expect(queueSandboxCreationCleanup(db, { id: creation.id, companyId, environmentId: original.environmentId!, providerLeaseId: "foreign-task" })).rejects.toThrow("ownership changed");
|
||||
await environmentService(db).releaseLease(replacement.id, "released", { cleanupStatus: "success",
|
||||
remoteExecutionTermination: remoteTerminationReceipt(replacement, { providerLeaseId: replacement.providerLeaseId, state: "stopped" }) });
|
||||
expect((await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id, creation.id)))[0].state).toBe("resolved");
|
||||
await queueSandboxCreationCleanup(db, { id: creation.id, companyId, environmentId: original.environmentId!, providerLeaseId: fresh.providerLeaseId });
|
||||
expect((await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id, creation.id)))[0])
|
||||
.toMatchObject({ state: "resolved", leaseId: replacement.id });
|
||||
await verifyNativeStartupCleanup(db, await f.reload(), async () => {});
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.request.runId));
|
||||
await acknowledgeNativeStartupCleanup(db, f.request);
|
||||
expect((await f.owner()).leaseOwner).toBeNull();
|
||||
});
|
||||
|
||||
it.each(["before_inventory", "before_write", "complete_before_write", "ordinary"])("preserves startup cleanup through orphan recovery: %s", async phase => {
|
||||
const f = await fixture(), deps = callbacks(db);
|
||||
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", leaseExpiresAt: new Date(0) })
|
||||
.where(eq(nativeRunFinalizations.runId, f.request.runId));
|
||||
if (phase === "before_inventory") await requestNativeStartupCleanup(db, f.request);
|
||||
const write = deps.setRunStatusFromLive.getMockImplementation()!;
|
||||
deps.setRunStatusFromLive.mockImplementation(async (...args) => {
|
||||
if (phase === "before_write" || phase === "complete_before_write") {
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
if (phase === "complete_before_write") await verifyNativeStartupCleanup(db, await f.reload(), () => f.stopReceipt());
|
||||
}
|
||||
return write(...args);
|
||||
});
|
||||
const recovery = createHeartbeatRecovery(db, { ...deps, environmentRuntime: environmentRuntimeService(db),
|
||||
environmentsSvc: environmentService(db), runtimeEnv: {}, activeRunExecutionPromises: new Set<Promise<void>>(),
|
||||
mergeRunStopMetadataForAgent: (_agent: unknown, _status: unknown, options: { resultJson?: object }) => options?.resultJson,
|
||||
classifyAndPersistRunLiveness: async (run: unknown) => run, releaseEnvironmentLeasesForRun: vi.fn(),
|
||||
releaseIssueExecutionAndPromote: vi.fn(), scheduleInteractionContinuationInfrastructureRetryIfEligible: async () => null,
|
||||
} as unknown as HeartbeatRecoveryDependencies);
|
||||
await recovery.recoverActiveExecutions({ runId: f.request.runId }, { staleThresholdMs: 0 });
|
||||
expect((await f.reload()).status).toBe(phase === "ordinary" ? "failed" : "running");
|
||||
if (phase === "ordinary") return;
|
||||
expect((await f.owner()).leaseOwner).toBe(f.request.leaseOwner);
|
||||
deps.verifyStartupCleanup = run => verifyNativeStartupCleanup(db, run, () => f.stopReceipt());
|
||||
expect(await createHeartbeatRunControl(db, deps).cancelRunInternal(f.request.runId, "Startup cleanup retry"))
|
||||
.toMatchObject({ status: "cancelled" });
|
||||
expect((await f.owner()).leaseOwner).toBeNull();
|
||||
});
|
||||
|
||||
it.each(["request", "acknowledge"])("registers queue intent before cleanup %s writes", async phase => {
|
||||
const f = await fixture(true);
|
||||
const writes: boolean[][] = [];
|
||||
const observe = async (queues: string[], work: () => Promise<unknown>) => {
|
||||
const transaction = db.transaction.bind(db);
|
||||
const spy = vi.spyOn(db, "transaction").mockImplementation(((callback: (tx: typeof db) => Promise<unknown>) => transaction(async tx => {
|
||||
const update = tx.update.bind(tx);
|
||||
tx.update = ((table: unknown) => {
|
||||
if ([heartbeatRuns, nativeRunFinalizations, environmentLeases].includes(table as never)) {
|
||||
writes.push(queues.map(queue => databaseWorkPending(db, queue)));
|
||||
}
|
||||
return update(table as never);
|
||||
}) as typeof tx.update;
|
||||
return callback(tx as unknown as typeof db);
|
||||
})) as typeof db.transaction);
|
||||
try { await work(); } finally { spy.mockRestore(); }
|
||||
};
|
||||
if (phase === "request") {
|
||||
await observe([DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.environmentCleanup], () => requestNativeStartupCleanup(db, f.request));
|
||||
expect(writes).toEqual([[true, true], [true, true], [true, true]]);
|
||||
await verifyNativeStartupCleanup(db, await f.reload(), () => f.stopReceipt({}, true));
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.request.runId));
|
||||
await acknowledgeNativeStartupCleanup(db, f.request);
|
||||
} else {
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
await verifyNativeStartupCleanup(db, await f.reload(), () => f.stopReceipt({}, true));
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.request.runId));
|
||||
await observe([DELIVERY_QUEUES.agentLifecycle, DELIVERY_QUEUES.environmentCleanup], () => acknowledgeNativeStartupCleanup(db, f.request));
|
||||
expect(writes).toEqual([[true, true]]);
|
||||
}
|
||||
});
|
||||
|
||||
it("returns the deadline queue to quiet and wakes again for the next startup failure", async () => {
|
||||
const schedule = executionControlSchedule(db);
|
||||
expect(await schedule.finalization()).toBeNull();
|
||||
for (let failure = 0; failure < 2; failure += 1) {
|
||||
const f = await fixture(true);
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
expect(await schedule.finalization()).not.toBeNull();
|
||||
const deps = callbacks(db);
|
||||
deps.verifyStartupCleanup = run => verifyNativeStartupCleanup(db, run, () => f.stopReceipt({}, true));
|
||||
const control = createHeartbeatRunControl(db, deps);
|
||||
await reconcileAbandonedExecutionControl(db, new Date(Date.now() + 1_000), { startupCleanup: async scope => {
|
||||
if (scope.runId !== f.request.runId) return;
|
||||
await control.cancelRunInternal(scope.runId, "Startup cleanup retry");
|
||||
await acknowledgeNativeStartupCleanup(db, scope);
|
||||
} });
|
||||
expect((await f.reload()).status).toBe("cancelled");
|
||||
expect(await f.owner()).toMatchObject({ leaseOwner: null, controlDeadlineAt: null });
|
||||
expect(await schedule.finalization()).toBeNull();
|
||||
}
|
||||
});
|
||||
|
||||
it("discovers a crash after warm prepare dispatch but before a cleanup intent was recorded", async () => {
|
||||
const f = await fixture(true), lease = await f.lease();
|
||||
await db.update(environmentLeases).set({ metadata: { ...lease.metadata, runnerTaskStartup: {
|
||||
contextId: f.request.contextId, deadlineAtMs: f.request.deadlineAtMs, runId: f.request.runId,
|
||||
} } }).where(eq(environmentLeases.id, lease.id));
|
||||
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) })
|
||||
.where(eq(nativeRunFinalizations.runId, f.request.runId));
|
||||
expect(readNativeStartupCleanup(await f.reload())).toBeNull();
|
||||
const onClaim = vi.fn();
|
||||
const result = await claimNativeRestartRecoveries({ db, restartKind: "hard", runIds: [f.request.runId],
|
||||
controller: { bootId: randomUUID(), pid: process.pid, processStartedAt: new Date() }, onClaim });
|
||||
expect(result).toEqual([{ kind: "blocked", runId: f.request.runId, reason: NATIVE_STARTUP_CLEANUP_PENDING }]);
|
||||
expect(onClaim).not.toHaveBeenCalled();
|
||||
expect(readNativeStartupCleanup(await f.reload())).toMatchObject({ contextId: f.request.contextId,
|
||||
state: "pending", requiresPreparationStop: true });
|
||||
expect(isNativeRunnerOwnershipHeld(await f.reload())).toBe(false);
|
||||
});
|
||||
|
||||
it.each(["wrong_run", "future_deadline", "live_owner", "checkpointed", "missing_marker"])(
|
||||
"does not infer a timed-out startup from %s", async variant => {
|
||||
const f = await fixture(true), lease = await f.lease();
|
||||
if (variant !== "missing_marker") await db.update(environmentLeases).set({ metadata: { ...lease.metadata,
|
||||
runnerTaskStartup: { contextId: f.request.contextId, runId: variant === "wrong_run" ? randomUUID() : f.request.runId,
|
||||
deadlineAtMs: variant === "future_deadline" ? Date.now() + 60_000 : f.request.deadlineAtMs },
|
||||
} }).where(eq(environmentLeases.id, lease.id));
|
||||
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() + (variant === "live_owner" ? 60_000 : -1_000)) })
|
||||
.where(eq(nativeRunFinalizations.runId, f.request.runId));
|
||||
if (variant === "checkpointed") await db.update(heartbeatRuns).set({ sessionIdAfter: "current-run-checkpoint" })
|
||||
.where(eq(heartbeatRuns.id, f.request.runId));
|
||||
await claimNativeRestartRecoveries({ db, restartKind: "hard", runIds: [f.request.runId],
|
||||
controller: { bootId: randomUUID(), pid: process.pid, processStartedAt: new Date() } });
|
||||
expect(readNativeStartupCleanup(await f.reload())).toBeNull();
|
||||
});
|
||||
|
||||
it("persists an audited cancellation intent that a new database handle can recover without a generic hold", async () => {
|
||||
const f = await fixture();
|
||||
expect(await requestNativeStartupCleanup(db, f.request)).toBe(true);
|
||||
const run = await f.reload(), marker = readNativeStartupCleanup(run)!;
|
||||
expect(marker).toMatchObject({ state: "pending", contextId: f.request.contextId, leaseOwner: f.request.leaseOwner,
|
||||
controllerGeneration: 1, attempt: 1, taskId: (await f.lease()).providerLeaseId });
|
||||
expect(run).toMatchObject({ status: "running", errorCode: NATIVE_STARTUP_CLEANUP_PENDING });
|
||||
expect(run.resultJson?.startupCancellation).toMatchObject({ requestedAt: expect.any(String) });
|
||||
expect(isNativeRunnerOwnershipHeld(run)).toBe(false);
|
||||
expect(await db.select({ id: activityLog.id }).from(activityLog).where(and(eq(activityLog.id, marker.auditId),
|
||||
eq(activityLog.companyId, companyId), eq(activityLog.action, "native.startup_cleanup_requested")))).toHaveLength(1);
|
||||
const restarted = createDb(temporary.connectionString), release = vi.fn(f.stopReceipt);
|
||||
await verifyNativeStartupCleanup(restarted, await f.reload(), release);
|
||||
await verifyNativeStartupCleanup(restarted, await f.reload(), release);
|
||||
expect(release).toHaveBeenCalledOnce();
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("complete");
|
||||
await acknowledgeNativeStartupCleanup(restarted, { companyId, runId: run.id });
|
||||
expect((await f.owner()).controlDeadlineAt).not.toBeNull();
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, run.id));
|
||||
await acknowledgeNativeStartupCleanup(restarted, { companyId, runId: run.id });
|
||||
expect(await f.owner()).toMatchObject({ leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null, recoveryState: null });
|
||||
});
|
||||
|
||||
it("keeps cleanup bound to the explicit current lease when a released alias belongs to the same run", async () => {
|
||||
const f = await fixture(true), current = await f.lease();
|
||||
const aliasId = randomUUID();
|
||||
await db.insert(environmentLeases).values({ ...current, id: aliasId, providerLeaseId: randomUUID(),
|
||||
status: "released", releasedAt: new Date(), cleanupStatus: "success" });
|
||||
expect(await requestNativeStartupCleanup(db, { ...f.request, leaseId: current.id })).toBe(true);
|
||||
expect(readNativeStartupCleanup(await f.reload())).toMatchObject({ leaseId: current.id, taskId: current.providerLeaseId });
|
||||
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, aliasId)))[0]?.metadata?.runnerTaskStartupCleanup).toBeUndefined();
|
||||
});
|
||||
|
||||
it("recovers an exact Stop receipt committed before Core records completion", async () => {
|
||||
const f = await fixture();
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
await f.stopReceipt();
|
||||
const release = vi.fn(async () => { throw new Error("Already retired task must not be released twice"); });
|
||||
await verifyNativeStartupCleanup(createDb(temporary.connectionString), await f.reload(), release);
|
||||
expect(release).not.toHaveBeenCalled();
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("complete");
|
||||
});
|
||||
|
||||
it("keeps accelerated preparation pending when only the provider has stopped", async () => {
|
||||
const f = await fixture(true);
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
await expect(verifyNativeStartupCleanup(db, await f.reload(), () => f.stopReceipt())).rejects.toThrow(/pending|preparation/i);
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("pending");
|
||||
expect((await f.owner()).leaseOwner).toBe(f.request.leaseOwner);
|
||||
});
|
||||
|
||||
it("automatically finishes pending cleanup after a late real worker reply and exact preparation Stop proof", async () => {
|
||||
const f = await fixture(true), lease = await f.lease();
|
||||
const entrypointPath = path.join(home.home, `cleanup-worker-${f.request.runId}.cjs`);
|
||||
await writeFile(entrypointPath, `
|
||||
const rl=require("node:readline").createInterface({input:process.stdin,crlfDelay:Infinity});
|
||||
const held=[];let stopped=false;
|
||||
const send=(id,result)=>process.stdout.write(JSON.stringify({jsonrpc:"2.0",id,result})+"\\n");
|
||||
rl.on("line",line=>{const m=JSON.parse(line);
|
||||
if(m.method==="initialize")send(m.id,{ok:true,supportedMethods:["environmentTask","configChanged"]});
|
||||
else if(m.method==="environmentTask"){
|
||||
const kind=m.params.operation.kind;
|
||||
if(kind==="prepare")held.push(m.id);
|
||||
else if(kind==="status")send(m.id,{kind:"status",taskId:m.params.taskId,phase:stopped?"cancelled":"running",executionStopped:stopped,preparationStopped:stopped&&held.length===0});
|
||||
else {if(kind==="stop")stopped=true;send(m.id,{kind:"accepted",taskId:m.params.taskId});}
|
||||
}else if(m.method==="configChanged"){for(const id of held.splice(0))send(id,{});send(m.id,{});}
|
||||
else if(m.method==="shutdown"){send(m.id,{});setImmediate(()=>process.exit(0));}
|
||||
});
|
||||
`, { mode: 0o600 });
|
||||
const manifest: PaperclipPluginManifestV1 = { id: "test.startup-cleanup", apiVersion: 1, version: "1.0.0",
|
||||
displayName: "Cleanup fixture", description: "Cleanup fixture", author: "Paperclip", categories: ["automation"],
|
||||
capabilities: [], entrypoints: { worker: "worker.cjs" } };
|
||||
const manager = createPluginWorkerManager();
|
||||
await manager.startWorker(manifest.id, { entrypointPath, manifest, config: {},
|
||||
instanceInfo: { instanceId: "cleanup", hostVersion: "1.0.0" }, apiVersion: 1, hostHandlers: {} });
|
||||
const context = new NativeExecutionContext({ binding: { companyId, agentId, runId: f.request.runId } } as NativeExecutionInput, 100);
|
||||
const params = { companyId, agentId, runId: f.request.runId, environmentId: lease.environmentId, issueId: lease.issueId,
|
||||
driverKey: "tasks", taskId: lease.providerLeaseId!, config: {}, projectIds: [], lease: { providerLeaseId: lease.providerLeaseId! } };
|
||||
const operation = { kind: "prepare", projectIds: [], requirements: { schema: "paperclip.runner.launch-requirements.v1",
|
||||
identity: { runnerInstanceId: "fixture", environmentLeaseId: lease.id, runId: f.request.runId,
|
||||
normalizedSessionId: "session", turnId: "turn", itemId: "item" }, provider: "codex", model: null,
|
||||
instructions: "Synthetic", runtimeContext: null, settings: {}, approvalPolicy: "never", readOnly: false,
|
||||
preparedConversation: false, includeCollaborationModeInstructions: false } } as PluginEnvironmentTaskOperation;
|
||||
const release = async () => {
|
||||
await manager.call(manifest.id, "environmentTask", { ...params, operation: { kind: "stop" } });
|
||||
const status = await manager.call(manifest.id, "environmentTask", { ...params, operation: { kind: "status" } });
|
||||
if (status.kind === "status" && status.executionStopped && status.preparationStopped) {
|
||||
await manager.call(manifest.id, "environmentTask", { ...params, operation: { kind: "complete" } });
|
||||
await f.stopReceipt({}, true);
|
||||
}
|
||||
};
|
||||
try {
|
||||
await expect(manager.call(manifest.id, "environmentTask", { ...params, operation }, 10, undefined, context.trackWorker())).rejects.toThrow("timed out");
|
||||
if (!context.signal.aborted) await new Promise<void>(resolve => context.signal.addEventListener("abort", () => resolve(), { once: true }));
|
||||
expect(context.pending).toBe(true);
|
||||
expect(await requestNativeStartupCleanup(db, { ...f.request, contextId: context.id, deadlineAtMs: context.deadlineAtMs })).toBe(true);
|
||||
expect(isNativeRunnerOwnershipHeld(await f.reload())).toBe(false);
|
||||
await expect(cancelNativeSession(f.request.runId, "Startup timeout", { db, scope: "run" })).resolves.toMatchObject({ auditId: expect.any(String) });
|
||||
await expect(verifyNativeStartupCleanup(db, await f.reload(), release)).rejects.toThrow("pending");
|
||||
expect((await f.reload()).status).toBe("running");
|
||||
await manager.call(manifest.id, "configChanged", { companyId, config: { release: true } });
|
||||
expect(context.pending).toBe(false);
|
||||
await verifyNativeStartupCleanup(createDb(temporary.connectionString), await f.reload(), release);
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("complete");
|
||||
expect((await f.lease()).metadata?.runnerTaskPreparationStopped).toBe(true);
|
||||
} finally { context.finish(); await manager.stopAll(); }
|
||||
});
|
||||
|
||||
it.each([false, true])("verifies an abandoned ephemeral startup without an inactive-owner acknowledgement (receipt=%s)", async stopped => {
|
||||
const prior = process.env.PAPERCLIP_EPHEMERAL_SERVER;
|
||||
process.env.PAPERCLIP_EPHEMERAL_SERVER = "true";
|
||||
try {
|
||||
const f = await fixture(true);
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
const deps = callbacks(db);
|
||||
deps.verifyStartupCleanup = vi.fn(run => verifyNativeStartupCleanup(db, run, async () => { if (stopped) await f.stopReceipt({}, true); }));
|
||||
const control = createHeartbeatRunControl(db, deps);
|
||||
if (stopped) {
|
||||
expect(await control.cancelRunInternal(f.request.runId, "Startup timeout")).toMatchObject({ status: "cancelled" });
|
||||
expect((await f.owner()).leaseOwner).toBeNull();
|
||||
} else {
|
||||
await expect(control.cancelRunInternal(f.request.runId, "Startup timeout")).rejects.toThrow("Task cleanup is still pending");
|
||||
expect((await f.reload()).status).toBe("running");
|
||||
expect((await f.owner()).leaseOwner).toBe(f.request.leaseOwner);
|
||||
}
|
||||
expect(deps.verifyStartupCleanup).toHaveBeenCalled();
|
||||
} finally {
|
||||
if (prior === undefined) delete process.env.PAPERCLIP_EPHEMERAL_SERVER;
|
||||
else process.env.PAPERCLIP_EPHEMERAL_SERVER = prior;
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it("terminalizes through assembled run control only after verified cleanup succeeds", async () => {
|
||||
const f = await fixture(true);
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
let drained = false;
|
||||
const deps = callbacks(db);
|
||||
deps.verifyStartupCleanup = run => verifyNativeStartupCleanup(db, run, () => f.stopReceipt({}, drained));
|
||||
const control = createHeartbeatRunControl(db, deps);
|
||||
await expect(control.cancelRunInternal(f.request.runId, "Startup timeout")).rejects.toThrow("pending");
|
||||
expect(await f.reload()).toMatchObject({ status: "running", errorCode: NATIVE_STARTUP_CLEANUP_PENDING });
|
||||
expect((await f.owner()).leaseOwner).toBe(f.request.leaseOwner);
|
||||
drained = true;
|
||||
const result = await control.cancelRunInternal(f.request.runId, "Startup timeout");
|
||||
expect(result).toMatchObject({ status: "cancelled" });
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("complete");
|
||||
expect(await f.owner()).toMatchObject({ leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null });
|
||||
});
|
||||
|
||||
it("joins cleanup recorded while an operator Stop waits for native startup", async () => {
|
||||
const f = await fixture(true);
|
||||
let entered!: () => void, resume!: () => void;
|
||||
const waiting = new Promise<void>(resolve => { entered = resolve; });
|
||||
const settled = new Promise<void>(resolve => { resume = resolve; });
|
||||
const cancel = nativeSessionExecutor.cancelNativeSession;
|
||||
const spy = vi.spyOn(nativeSessionExecutor, "cancelNativeSession").mockImplementation(async (...args) => {
|
||||
const result = await cancel(...args);
|
||||
if (args[0] === f.request.runId) { entered(); await settled; }
|
||||
return result;
|
||||
});
|
||||
let drained = false;
|
||||
const deps = callbacks(db);
|
||||
deps.verifyStartupCleanup = vi.fn(run => verifyNativeStartupCleanup(db, run, () => f.stopReceipt({}, drained)));
|
||||
const control = createHeartbeatRunControl(db, deps);
|
||||
try {
|
||||
const stopping = control.cancelRunInternal(f.request.runId, "Operator Stop");
|
||||
const outcome = stopping.then(value => ({ value }), error => ({ error }));
|
||||
await waiting;
|
||||
expect(await requestNativeStartupCleanup(db, f.request)).toBe(true);
|
||||
resume();
|
||||
expect(await outcome).toMatchObject({ error: expect.objectContaining({ message: "Task cleanup is still pending" }) });
|
||||
expect(deps.verifyStartupCleanup).toHaveBeenCalledOnce();
|
||||
expect((await f.reload()).status).toBe("running");
|
||||
expect((await f.owner()).leaseOwner).toBe(f.request.leaseOwner);
|
||||
drained = true;
|
||||
expect(await control.cancelRunInternal(f.request.runId, "Operator Stop")).toMatchObject({ status: "cancelled" });
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("complete");
|
||||
expect((await f.owner()).leaseOwner).toBeNull();
|
||||
} finally { resume(); spy.mockRestore(); }
|
||||
});
|
||||
|
||||
it("rejects a cancellation commit when cleanup arrives after its final read", async () => {
|
||||
const f = await fixture(true);
|
||||
const deps = callbacks(db), commit = deps.setRunStatusFromLive.getMockImplementation()!;
|
||||
deps.setRunStatusFromLive.mockImplementationOnce(async (...args) => {
|
||||
expect(await requestNativeStartupCleanup(db, f.request)).toBe(true);
|
||||
return commit(...args);
|
||||
});
|
||||
deps.verifyStartupCleanup = vi.fn(run => verifyNativeStartupCleanup(db, run, () => f.stopReceipt({}, true)));
|
||||
const control = createHeartbeatRunControl(db, deps);
|
||||
expect(await control.cancelRunInternal(f.request.runId, "Operator Stop")).toMatchObject({ status: "running" });
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("pending");
|
||||
expect((await f.owner()).leaseOwner).toBe(f.request.leaseOwner);
|
||||
expect(deps.startNextQueuedRunForAgent).not.toHaveBeenCalled();
|
||||
expect(await control.cancelRunInternal(f.request.runId, "Operator Stop")).toMatchObject({ status: "cancelled" });
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("complete");
|
||||
expect((await f.owner()).leaseOwner).toBeNull();
|
||||
});
|
||||
|
||||
it("restarts the real cancellation flow after remote Stop committed before Core acknowledgement", async () => {
|
||||
const f = await fixture(true);
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
await f.stopReceipt({}, true);
|
||||
const restarted = createDb(temporary.connectionString), release = vi.fn(async () => { throw new Error("Stop receipt already persisted"); });
|
||||
const deps = callbacks(restarted);
|
||||
deps.verifyStartupCleanup = run => verifyNativeStartupCleanup(restarted, run, release);
|
||||
const control = createHeartbeatRunControl(restarted, deps);
|
||||
const reconcile = vi.fn(async (scope: { companyId: string; runId: string }) => {
|
||||
if (scope.runId !== f.request.runId) return;
|
||||
expect(scope.companyId).toBe(companyId);
|
||||
await control.cancelRunInternal(scope.runId, "Startup timeout recovered after restart");
|
||||
await acknowledgeNativeStartupCleanup(restarted, scope);
|
||||
});
|
||||
await reconcileAbandonedExecutionControl(restarted, new Date(Date.now() + 1_000), { startupCleanup: reconcile });
|
||||
expect(reconcile.mock.calls.some(([scope]) => scope.runId === f.request.runId)).toBe(true);
|
||||
expect(release).not.toHaveBeenCalled();
|
||||
expect((await f.reload()).status).toBe("cancelled");
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("complete");
|
||||
expect(await f.owner()).toMatchObject({ leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null });
|
||||
});
|
||||
|
||||
it.each(["providerLeaseId", "runId", "leaseId"])("rejects a Stop receipt for another %s", async field => {
|
||||
const f = await fixture();
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
await expect(verifyNativeStartupCleanup(db, await f.reload(), () => f.stopReceipt({ [field]: randomUUID() }))).rejects.toThrow("pending");
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("pending");
|
||||
});
|
||||
|
||||
it("does not issue a Stop against a newer controller generation", async () => {
|
||||
const f = await fixture();
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
await db.update(nativeRunFinalizations).set({ controllerGeneration: 2 }).where(eq(nativeRunFinalizations.runId, f.request.runId));
|
||||
const release = vi.fn(f.stopReceipt);
|
||||
await expect(verifyNativeStartupCleanup(db, await f.reload(), release)).rejects.toThrow("ownership changed");
|
||||
expect(release).not.toHaveBeenCalled();
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("pending");
|
||||
});
|
||||
|
||||
it("rejects a stale generation that changes while Stop is in flight", async () => {
|
||||
const f = await fixture();
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
await expect(verifyNativeStartupCleanup(db, await f.reload(), async () => {
|
||||
await f.stopReceipt();
|
||||
await db.update(nativeRunFinalizations).set({ controllerGeneration: 2 }).where(eq(nativeRunFinalizations.runId, f.request.runId));
|
||||
})).rejects.toThrow("ownership changed");
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("pending");
|
||||
});
|
||||
|
||||
it("preserves a final result that wins while verified cleanup is in flight", async () => {
|
||||
const f = await fixture();
|
||||
await requestNativeStartupCleanup(db, f.request);
|
||||
let resultId: string;
|
||||
await expect(verifyNativeStartupCleanup(db, await f.reload(), async () => {
|
||||
await f.stopReceipt();
|
||||
resultId = await f.acceptedResult();
|
||||
})).rejects.toThrow("ownership changed");
|
||||
expect(await f.owner()).toMatchObject({ resultId: resultId!, phase: "workspace_finalizing" });
|
||||
expect(readNativeStartupCleanup(await f.reload())?.state).toBe("pending");
|
||||
});
|
||||
|
||||
it.each(["native_adopted_runner_authentication_timeout", "native_execution_ownership_unverified"])("preserves the existing %s hold", async errorCode => {
|
||||
const f = await fixture();
|
||||
await db.update(heartbeatRuns).set({ errorCode, nativePhase: "terminal_failure" }).where(eq(heartbeatRuns.id, f.request.runId));
|
||||
expect(await requestNativeStartupCleanup(db, f.request)).toBe(false);
|
||||
const current = await f.reload();
|
||||
expect(current).toMatchObject({ status: "running", errorCode, nativePhase: "terminal_failure" });
|
||||
expect(readNativeStartupCleanup(current)).toBeNull();
|
||||
});
|
||||
|
||||
it("does not replace a final result with a timeout intent", async () => {
|
||||
const f = await fixture();
|
||||
const resultId = await f.acceptedResult();
|
||||
expect(await requestNativeStartupCleanup(db, f.request)).toBe(false);
|
||||
expect((await f.owner()).resultId).toBe(resultId);
|
||||
expect(readNativeStartupCleanup(await f.reload())).toBeNull();
|
||||
});
|
||||
it("promotes a pre-capability context before an ordinary deadline after the audited intent write fails", async () => {
|
||||
const f = await fixture();
|
||||
await saveEarlyContext(f);
|
||||
await db.execute(sql`create function reject_startup_intent() returns trigger language plpgsql as $$ begin
|
||||
if new.action = 'native.startup_cleanup_requested' then raise exception 'Startup intent write unavailable'; end if; return new; end $$`);
|
||||
await db.execute(sql`create trigger reject_startup_intent before insert on activity_log for each row execute function reject_startup_intent()`);
|
||||
try { await expect(requestNativeStartupCleanup(db, f.request)).rejects.toThrow(); }
|
||||
finally { await db.execute(sql`drop trigger reject_startup_intent on activity_log`); await db.execute(sql`drop function reject_startup_intent()`); }
|
||||
expect(readNativeStartupCleanup(await f.reload())).toBeNull();
|
||||
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) }).where(eq(nativeRunFinalizations.runId, f.request.runId));
|
||||
await db.update(heartbeatRuns).set({ executionControlDeadlineAt: new Date(Date.now() - 1_000) }).where(eq(heartbeatRuns.id, f.request.runId));
|
||||
await reconcileAbandonedExecutionControl(db);
|
||||
expect(readNativeStartupCleanup(await f.reload())).toMatchObject({ contextId: f.request.contextId, requiresPreparationStop: false, state: "pending" });
|
||||
expect((await f.reload()).status).toBe("running");
|
||||
expect((await f.owner()).leaseOwner).toBe(f.request.leaseOwner);
|
||||
await verifyNativeStartupCleanup(db, await f.reload(), () => f.stopReceipt());
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.request.runId));
|
||||
await acknowledgeNativeStartupCleanup(db, f.request);
|
||||
});
|
||||
|
||||
it.each([false, true])("recovers an early context without replacing stronger preparation proof (prepared=%s)", async prepared => {
|
||||
const f = await fixture();
|
||||
await saveEarlyContext(f);
|
||||
if (prepared) { const lease = await f.lease(); await db.update(environmentLeases).set({ metadata: { ...lease.metadata,
|
||||
runnerTaskStartup: { contextId: f.request.contextId, deadlineAtMs: f.request.deadlineAtMs, runId: f.request.runId } } }).where(eq(environmentLeases.id, lease.id)); }
|
||||
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) }).where(eq(nativeRunFinalizations.runId, f.request.runId));
|
||||
const onClaim = vi.fn();
|
||||
const result = await claimNativeRestartRecoveries({ db, restartKind: "hard", runIds: [f.request.runId],
|
||||
controller: { bootId: randomUUID(), pid: process.pid, processStartedAt: new Date() }, onClaim });
|
||||
expect(result).toEqual([{ kind: "blocked", runId: f.request.runId, reason: NATIVE_STARTUP_CLEANUP_PENDING }]);
|
||||
expect(onClaim).not.toHaveBeenCalled();
|
||||
expect(readNativeStartupCleanup(await f.reload())).toMatchObject({ requiresPreparationStop: prepared, state: "pending" });
|
||||
await verifyNativeStartupCleanup(db, await f.reload(), () => f.stopReceipt({}, prepared));
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.request.runId));
|
||||
await acknowledgeNativeStartupCleanup(db, f.request);
|
||||
});
|
||||
|
||||
it.each([false, true])("defers an early context to a concrete owner deadline (locally active=%s)", async active => {
|
||||
const f = await fixture();
|
||||
await saveEarlyContext(f);
|
||||
const owner = await f.owner();
|
||||
const result = await claimNativeRestartRecoveries({ db, restartKind: "hard", runIds: [f.request.runId],
|
||||
liveRunExecutions: active ? new Set([f.request.runId]) : undefined,
|
||||
controller: { bootId: randomUUID(), pid: process.pid, processStartedAt: new Date() } });
|
||||
expect(result).toMatchObject([{ kind: "awaiting_evidence", runId: f.request.runId, retryAt: owner.leaseExpiresAt }]);
|
||||
expect(readNativeStartupCleanup(await f.reload())).toBeNull();
|
||||
});
|
||||
|
||||
it("does not turn an early context into cleanup after the current-run checkpoint", async () => {
|
||||
const f = await fixture();
|
||||
await saveEarlyContext(f);
|
||||
await db.update(heartbeatRuns).set({ sessionIdAfter: "current-checkpoint" }).where(eq(heartbeatRuns.id, f.request.runId));
|
||||
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) }).where(eq(nativeRunFinalizations.runId, f.request.runId));
|
||||
await claimNativeRestartRecoveries({ db, restartKind: "hard", runIds: [f.request.runId],
|
||||
controller: { bootId: randomUUID(), pid: process.pid, processStartedAt: new Date() } });
|
||||
expect(readNativeStartupCleanup(await f.reload())).toBeNull();
|
||||
});
|
||||
|
||||
|
||||
it("locks the selected lease before choosing the stronger preparation proof", async () => {
|
||||
const f = await fixture(), lease = await f.lease();
|
||||
let changed!: () => void, commit!: () => void;
|
||||
const written = new Promise<void>(resolve => { changed = resolve; });
|
||||
const release = new Promise<void>(resolve => { commit = resolve; });
|
||||
const preparing = db.transaction(async tx => {
|
||||
await tx.update(environmentLeases).set({ metadata: { ...lease.metadata, runnerTaskStartup: {
|
||||
contextId: f.request.contextId, deadlineAtMs: f.request.deadlineAtMs, runId: f.request.runId,
|
||||
} } }).where(eq(environmentLeases.id, lease.id));
|
||||
changed();
|
||||
await release;
|
||||
});
|
||||
await written;
|
||||
const requesting = requestNativeStartupCleanup(db, f.request);
|
||||
try {
|
||||
await vi.waitFor(async () => {
|
||||
const rows = await db.execute(sql`select query from pg_stat_activity where wait_event_type = 'Lock'
|
||||
and query like 'select%from "environment_leases"%'`);
|
||||
expect(rows.length).toBeGreaterThan(0);
|
||||
});
|
||||
} finally { commit(); }
|
||||
await preparing;
|
||||
await requesting;
|
||||
expect(readNativeStartupCleanup(await f.reload())).toMatchObject({ requiresPreparationStop: true });
|
||||
await expect(verifyNativeStartupCleanup(db, await f.reload(), () => f.stopReceipt())).rejects.toThrow("pending");
|
||||
await verifyNativeStartupCleanup(db, await f.reload(), () => f.stopReceipt({}, true));
|
||||
});
|
||||
|
||||
it("clears only its owned startup context before fallback and leaves no expired cleanup authority", async () => {
|
||||
const f = await fixture();
|
||||
await saveEarlyContext(f);
|
||||
const scope = { companyId, runId: f.request.runId, contextId: f.request.contextId,
|
||||
leaseOwner: f.request.leaseOwner, controllerGeneration: 1 };
|
||||
await expect(clearNativeStartupContext(db, { ...scope, controllerGeneration: 2 })).rejects.toThrow("lost authority");
|
||||
expect((await f.reload()).runnerProfileJson?.nativeStartupContext).toBeDefined();
|
||||
await clearNativeStartupContext(db, scope);
|
||||
expect((await f.reload()).runnerProfileJson?.nativeStartupContext).toBeUndefined();
|
||||
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) }).where(eq(nativeRunFinalizations.runId, f.request.runId));
|
||||
await claimNativeRestartRecoveries({ db, restartKind: "hard", runIds: [f.request.runId],
|
||||
controller: { bootId: randomUUID(), pid: process.pid, processStartedAt: new Date() } });
|
||||
expect(readNativeStartupCleanup(await f.reload())).toBeNull();
|
||||
});
|
||||
|
||||
it.each(["cleanup", "terminal"])("does not clear the startup breadcrumb after %s wins", async winner => {
|
||||
const f = await fixture();
|
||||
await saveEarlyContext(f);
|
||||
if (winner === "cleanup") await requestNativeStartupCleanup(db, f.request);
|
||||
else await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.request.runId));
|
||||
await expect(clearNativeStartupContext(db, { companyId, runId: f.request.runId, contextId: f.request.contextId,
|
||||
leaseOwner: f.request.leaseOwner, controllerGeneration: 1 })).rejects.toThrow("lost authority");
|
||||
expect((await f.reload()).runnerProfileJson?.nativeStartupContext).toBeDefined();
|
||||
if (winner === "cleanup") expect(readNativeStartupCleanup(await f.reload())).toMatchObject({ state: "pending" });
|
||||
});
|
||||
|
||||
it("requires exact task Stop proof when retrying cleanup of a never-dispatched replacement", async () => {
|
||||
const f = await fixture(true), original = await f.lease(), pluginId = randomUUID();
|
||||
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "fixture-abandoned-task", version: "1", status: "ready",
|
||||
manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", supportsTasks: true }] } as never });
|
||||
await db.update(environments).set({ companyId, config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } })
|
||||
.where(eq(environments.id, original.environmentId!));
|
||||
const fresh = await createOwnedSandbox(db, { companyId, environmentId: original.environmentId!, heartbeatRunId: f.request.runId,
|
||||
issueId: original.issueId, provider: original.provider!, pluginId, driverKey: "tasks", config: {}, requestKey: original.providerLeaseId!,
|
||||
metadata: { driver: "plugin", pluginId, pluginKey: pluginId, driverKey: "tasks", runnerTaskLease: true, agentId,
|
||||
runnerTaskCreationContextId: f.request.contextId } }, async () => ({ providerLeaseId: "late-never-dispatched" }));
|
||||
await queueSandboxCreationCleanup(db, { id: String(fresh.metadata!.creationOperationId), companyId,
|
||||
environmentId: original.environmentId!, providerLeaseId: fresh.providerLeaseId });
|
||||
const creation = async () => (await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id, String(fresh.metadata!.creationOperationId))))[0]!;
|
||||
const replacement = async () => (await environmentService(db).getLeaseById((await creation()).leaseId!))!;
|
||||
expect((await replacement()).metadata?.runnerTaskDispatched).not.toBe(true);
|
||||
await requestNativeStartupCleanup(db, { ...f.request, leaseId: original.id });
|
||||
await f.stopReceipt({}, true);
|
||||
let proof: "void" | "running" | "foreign" | "exact" = "void";
|
||||
const calls: string[] = [];
|
||||
const workers = { isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentTask", "environmentReleaseLease"] }),
|
||||
drainStartup: vi.fn(async () => undefined),
|
||||
call: async (_plugin: string, method: string, params: any) => {
|
||||
calls.push(method === "environmentTask" ? params.operation.kind : method);
|
||||
if (method === "environmentReleaseLease") return undefined;
|
||||
if (params.operation.kind === "status") return proof === "void" ? undefined : {
|
||||
kind: "status", taskId: proof === "foreign" ? "another-task" : params.taskId,
|
||||
phase: proof === "running" ? "running" : "cancelled", executionStopped: proof !== "running",
|
||||
};
|
||||
return { kind: "accepted", taskId: params.taskId };
|
||||
},
|
||||
} as unknown as PluginWorkerManager;
|
||||
const runtime = environmentRuntimeService(db, { pluginWorkerManager: workers });
|
||||
const recovery = createHeartbeatRecovery(db, { ...callbacks(db), environmentRuntime: runtime,
|
||||
environmentsSvc: environmentService(db), runtimeEnv: {}, activeRunExecutionPromises: new Set<Promise<void>>(),
|
||||
acknowledgeRemoteStop: async () => undefined, resumeRemoteStopComments: async () => undefined,
|
||||
} as unknown as HeartbeatRecoveryDependencies);
|
||||
const stopDeps = callbacks(db);
|
||||
stopDeps.verifyStartupCleanup = run => verifyNativeStartupCleanup(db, run, async () => {});
|
||||
const control = createHeartbeatRunControl(db, stopDeps);
|
||||
for (const missing of ["void", "running", "foreign"] as const) {
|
||||
proof = missing;
|
||||
const swept = await recovery.sweepPendingCleanupLeases({ explicitRetry: { companyId, runId: f.request.runId, actorId: "operator" } });
|
||||
const cancelled = await control.cancelRunInternal(f.request.runId, "Startup cleanup retry").then(() => true, () => false);
|
||||
const lease = await replacement();
|
||||
expect({ destroyed: swept.destroyed, status: lease.status, verifiedReceipt: hasRemoteTerminationReceipt(lease),
|
||||
creationState: (await creation()).state, cancelled, leaseOwner: (await f.owner()).leaseOwner }).toEqual({
|
||||
destroyed: 0, status: "pending_cleanup", verifiedReceipt: false, creationState: "cleanup_pending", cancelled: false, leaseOwner: f.request.leaseOwner,
|
||||
});
|
||||
}
|
||||
proof = "exact";
|
||||
expect((await recovery.sweepPendingCleanupLeases({ explicitRetry: { companyId, runId: f.request.runId, actorId: "operator" } })).destroyed).toBe(1);
|
||||
expect(calls).toContain("stop");
|
||||
expect(calls).toContain("status");
|
||||
expect(calls).toContain("complete");
|
||||
expect(workers!.drainStartup).toHaveBeenCalledWith(f.request.contextId);
|
||||
expect(hasRemoteTerminationReceipt(await replacement())).toBe(true);
|
||||
expect((await creation()).state).toBe("resolved");
|
||||
expect(await control.cancelRunInternal(f.request.runId, "Startup cleanup retry")).toMatchObject({ status: "cancelled" });
|
||||
expect((await f.owner()).leaseOwner).toBeNull();
|
||||
});
|
||||
|
||||
});
|
||||
+7
-1
@@ -81,6 +81,7 @@ import { deliverExecutionStatuses } from "./services/execution-status-delivery.j
|
||||
import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./services/execution-recovery-resolution.js";
|
||||
import { reconcileSafeNativeReplacements } from "./services/native-runtime/native-safe-replacement.js";
|
||||
import { reconcileAbandonedExecutionControl } from "./services/execution-control-reconciliation.js";
|
||||
import { acknowledgeNativeStartupCleanup } from "./services/native-runtime/native-startup-cleanup.js";
|
||||
import { executionControlSchedule, registerExecutionControlWork } from "./services/execution-control-work.js";
|
||||
import { connectionIntentDeliveryService } from "./services/connection-intent-delivery.js";
|
||||
import { existsSync, readFileSync, rmSync } from "node:fs";
|
||||
@@ -1376,7 +1377,12 @@ async function startServerWithDatabaseTeardown(
|
||||
const unsubscribeLocalLoginCleanup = localLoginCleanup.subscribeCleanupReady(localLoginCleanupWork.wake);
|
||||
server.on("close", unsubscribeLocalLoginCleanup);
|
||||
registerExecutionControlWork(deliveryWork, [
|
||||
{ queue: DELIVERY_QUEUES.executionFinalization, run: () => reconcileAbandonedExecutionControl(db), next: executionSchedule.finalization },
|
||||
{ queue: DELIVERY_QUEUES.executionFinalization, run: () => reconcileAbandonedExecutionControl(db, new Date(), {
|
||||
startupCleanup: async scope => {
|
||||
await accountingHeartbeat.cancelRun(scope.runId, "Native startup timed out; cancelling and verifying cleanup");
|
||||
await acknowledgeNativeStartupCleanup(db, scope);
|
||||
},
|
||||
}), next: executionSchedule.finalization },
|
||||
{ queue: DELIVERY_QUEUES.executionStatus, run: () => deliverExecutionStatuses(db), next: executionSchedule.status },
|
||||
{ queue: DELIVERY_QUEUES.executionDisposition, run: () => settleUnrecoverableExecutions(db), next: executionSchedule.disposition },
|
||||
...(heartbeat ? [
|
||||
|
||||
@@ -1,9 +1,19 @@
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { and, eq, or, sql } from "drizzle-orm";
|
||||
import { environmentLeases, sandboxCreationOperations, type Db } from "@paperclipai/db";
|
||||
import { hasDeferredEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
|
||||
export const ENVIRONMENT_CLEANUP_RETRY_MS = 5 * 60_000;
|
||||
|
||||
export function recoverableSandboxCreationCondition() {
|
||||
return or(eq(sandboxCreationOperations.state, "pending"), and(
|
||||
eq(sandboxCreationOperations.state, "needs_reconciliation"),
|
||||
sql`coalesce(${sandboxCreationOperations.metadata}->>'creationReconciliationReason', '') <> 'company_deleted'`,
|
||||
sql`${sandboxCreationOperations.metadata}->>'runnerTaskLease' = 'true'`,
|
||||
sql`jsonb_typeof(${sandboxCreationOperations.metadata}->'runnerTaskCreationContextId') = 'string' and ${sandboxCreationOperations.metadata}->>'runnerTaskCreationContextId' <> ''`,
|
||||
sql`jsonb_typeof(${sandboxCreationOperations.metadata}->'creationProviderLeaseId') = 'string' and ${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' <> ''`,
|
||||
));
|
||||
}
|
||||
|
||||
export function pendingCleanupDeadlineSql(now = Date.now()) {
|
||||
const deadline = sql`case
|
||||
when ${environmentLeases.metadata}->>'pendingCleanupInFlight' = 'true'
|
||||
@@ -21,7 +31,7 @@ export async function nextEnvironmentCleanupAt(db: Db, backoffMs = ENVIRONMENT_C
|
||||
extract(epoch from ${environmentLeases.updatedAt}) * 1000 + ${backoffMs},
|
||||
${pendingCleanupDeadlineSql()}))` }).from(environmentLeases).where(eq(environmentLeases.status, "pending_cleanup"));
|
||||
const [creation] = await db.select({ next: sql<string | null>`min(extract(epoch from ${sandboxCreationOperations.nextAttemptAt}) * 1000)` })
|
||||
.from(sandboxCreationOperations).where(eq(sandboxCreationOperations.state, "pending"));
|
||||
.from(sandboxCreationOperations).where(recoverableSandboxCreationCondition());
|
||||
const deadlines = [row?.next, creation?.next].filter((value): value is string => value != null).map(Number);
|
||||
const saved = deadlines.length ? Math.min(...deadlines) : null;
|
||||
const buffered = hasDeferredEnvironmentCleanup(db) ? Date.now() + backoffMs : null;
|
||||
|
||||
@@ -3382,7 +3382,7 @@ function createPluginEnvironmentDriver(
|
||||
async releaseRunLease(input) {
|
||||
await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"pluginLeaseCleanupMethod":"release"}'::jsonb` }).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.lease.companyId)));
|
||||
const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
|
||||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true);
|
||||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true && !input.lease.metadata?.runnerTaskStartup && !input.lease.metadata?.runnerTaskStartupCleanup);
|
||||
const { plugin, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
|
||||
await workerManager.call(plugin.id, "environmentReleaseLease", {
|
||||
driverKey,
|
||||
@@ -3425,7 +3425,7 @@ function createPluginEnvironmentDriver(
|
||||
async destroyRunLease(input) {
|
||||
await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"pluginLeaseCleanupMethod":"destroy"}'::jsonb` }).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.lease.companyId)));
|
||||
const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
|
||||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true);
|
||||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true && !input.lease.metadata?.runnerTaskStartup && !input.lease.metadata?.runnerTaskStartupCleanup);
|
||||
const { pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
|
||||
await destroyPluginEnvironmentLease({
|
||||
db,
|
||||
@@ -3462,7 +3462,7 @@ function createPluginEnvironmentDriver(
|
||||
|
||||
async retryPendingSandboxTeardown(input) {
|
||||
const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
|
||||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true);
|
||||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true && !input.lease.metadata?.runnerTaskStartup && !input.lease.metadata?.runnerTaskStartupCleanup);
|
||||
const { plugin, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
|
||||
const environmentId = input.environment?.id ?? readString(input.lease.metadata?.environmentId);
|
||||
if (!environmentId) throw new Error("Plugin lease cleanup has no saved environment identity");
|
||||
@@ -3778,7 +3778,7 @@ export function environmentRuntimeService(
|
||||
},
|
||||
|
||||
/** Dispatch plugin-provided Runner execution, including resource preparation and task lifecycle. */
|
||||
async task(input: { companyId: string; leaseId: string; operation: PluginEnvironmentTaskOperation }) {
|
||||
async task(input: { companyId: string; leaseId: string; operation: PluginEnvironmentTaskOperation; executionContext?: unknown }) {
|
||||
if (!options.pluginWorkerManager) throw new Error("Plugin-provided Runner execution worker manager unavailable");
|
||||
return executeEnvironmentTask(db, options.pluginWorkerManager, input);
|
||||
},
|
||||
|
||||
@@ -1,15 +1,17 @@
|
||||
import { attachSandboxCreation, environmentService } from "./environments.js";
|
||||
import { createOwnedSandbox, recordSandboxCreationResource } from "./sandbox-creation-operations.js";
|
||||
import { createOwnedSandbox, queueSandboxCreationCleanup, recordSandboxCreationResource } from "./sandbox-creation-operations.js";
|
||||
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { releaseProviderRecoveryTransfers, stageProviderFreshWorkspace } from "./provider-fresh-workspace.js";
|
||||
import { readWorkspaceProviderBinding } from "@paperclipai/shared";
|
||||
import { settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js";
|
||||
import { and, eq, inArray, or, sql } from "drizzle-orm";
|
||||
import { environmentLeases, executionWorkspaces, environments, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db";
|
||||
import { freshWorkspaceSchema, environmentTaskOperationSchema, parseEnvironmentTaskResult, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
|
||||
import { freshWorkspaceSchema, environmentTaskOperationSchema, parseEnvironmentTaskResult,
|
||||
type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
|
||||
import { pluginRegistryService } from "./plugin-registry.js";
|
||||
import { hasRemoteTerminationReceipt, remoteExecutionHasStopped, remoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
|
||||
import { NativeExecutionContext } from "./native-runtime/native-execution-context.js";
|
||||
|
||||
export async function pluginEnvironmentSupportsTasks(db: Db, workers: PluginWorkerManager, lease: { companyId: string; metadata: Record<string, unknown> | null }) {
|
||||
const metadata = lease.metadata ?? {};
|
||||
@@ -31,6 +33,7 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
companyId: string;
|
||||
leaseId: string;
|
||||
operation: PluginEnvironmentTaskOperation;
|
||||
executionContext?: unknown;
|
||||
}) {
|
||||
const operation = environmentTaskOperationSchema.parse(input.operation);
|
||||
const [row] = await db.select({ lease: environmentLeases, environment: environments, run: heartbeatRuns })
|
||||
@@ -40,6 +43,13 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
.where(and(eq(environmentLeases.id, input.leaseId), eq(environmentLeases.companyId, input.companyId))).limit(1);
|
||||
if (!row || !row.lease.providerLeaseId) throw new Error("Plugin-provided Runner execution lease unavailable");
|
||||
const { lease, environment, run } = row;
|
||||
const context = input.executionContext;
|
||||
if (context !== undefined) {
|
||||
if (!(context instanceof NativeExecutionContext) || !run) {
|
||||
throw new Error("native_execution_context_unavailable");
|
||||
}
|
||||
context.assertScope({ companyId: input.companyId, agentId: run.agentId, runId: run.id });
|
||||
}
|
||||
const taskId = row.lease.providerLeaseId;
|
||||
const metadata = lease.metadata ?? {};
|
||||
const probe = metadata.runnerTaskProbe as { runId?: unknown; sessionId?: unknown; expiresAtMs?: unknown } | undefined;
|
||||
@@ -128,6 +138,19 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
!workers.getWorker(plugin.id)?.supportedMethods.includes("environmentTask")) {
|
||||
throw new Error("Plugin-provided Runner execution provider unavailable");
|
||||
}
|
||||
const callWorker = async <M extends HostToWorkerMethodName>(method: M, params: HostToWorkerMethods[M][0], timeout?: number): Promise<HostToWorkerMethods[M][1]> => {
|
||||
if (!context) {
|
||||
const cleanup = metadata.runnerTaskStartupCleanup as { contextId?: unknown; requiresPreparationStop?: unknown } | undefined;
|
||||
if (["stop", "status", "complete", "credentials"].includes(operation.kind) && typeof cleanup?.contextId === "string" && typeof cleanup.requiresPreparationStop === "boolean") {
|
||||
return workers.call(plugin.id, method, params, timeout, undefined, undefined, cleanup.contextId);
|
||||
}
|
||||
return timeout === undefined ? workers.call(plugin.id, method, params) : workers.call(plugin.id, method, params, timeout);
|
||||
}
|
||||
const settled = context.trackWorker();
|
||||
const result = await workers.call(plugin.id, method, params, Math.max(1, Math.min(timeout ?? 15_000, context.deadlineAtMs - Date.now())), undefined, settled, context.id);
|
||||
context.assertOpen();
|
||||
return result;
|
||||
};
|
||||
const issue = scopesProjects && lease.issueId ? await db.select({ id: issues.id }).from(issues)
|
||||
.where(and(eq(issues.id, lease.issueId), eq(issues.companyId, input.companyId))).limit(1).then(rows => rows[0]) : null;
|
||||
if (scopesProjects && lease.issueId && !issue) throw new Error("Plugin-provided Runner execution issue unavailable");
|
||||
@@ -219,7 +242,7 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
}
|
||||
try {
|
||||
if (operation.kind === "describe" && metadata.runnerTaskReused === true) {
|
||||
const state = parseEnvironmentTaskResult({kind:"status"}, taskId, await workers.call(plugin.id, "environmentTask", {
|
||||
const state = parseEnvironmentTaskResult({kind:"status"}, taskId, await callWorker("environmentTask", {
|
||||
driverKey: metadata.driverKey, companyId: input.companyId, environmentId: environment?.id ?? null,
|
||||
issueId: lease.issueId, config: driverConfig, taskId, runId: run?.id ?? null, agentId: run?.agentId ?? null,
|
||||
projectIds, lease: {providerLeaseId: taskId, metadata}, operation:{kind:"status"},
|
||||
@@ -233,37 +256,74 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
runnerTaskSessionRetained:state.sessionStateAvailable === true})}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, metadata.runnerTaskPreviousLeaseId), eq(environmentLeases.companyId, input.companyId)));
|
||||
const replacementDriverKey = metadata.driverKey;
|
||||
let abandoned = false;
|
||||
const fresh = await createOwnedSandbox(db, {
|
||||
companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId,
|
||||
issueId: lease.issueId, executionWorkspaceId: lease.executionWorkspaceId, provider: lease.provider!,
|
||||
pluginId: plugin.id, driverKey: metadata.driverKey, config: {}, requestKey: taskId,
|
||||
metadata: { driver: "plugin", pluginId: plugin.id, pluginKey: plugin.pluginKey, driverKey: metadata.driverKey, runnerTaskLease: true },
|
||||
}, identity => workers.call(plugin.id, "environmentAcquireLease", {driverKey:replacementDriverKey,
|
||||
companyId:input.companyId,environmentId:environment!.id,issueId:lease.issueId,config:driverConfig,
|
||||
runId:identity.runId,creationOperationId:identity.id,agentId:run!.agentId,adapterType:"paperclip_runner",
|
||||
...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? {requestedExpiresAt:metadata.runnerTaskRequestedExpiresAt} : {})}));
|
||||
metadata: { driver: "plugin", pluginId: plugin.id, pluginKey: plugin.pluginKey, driverKey: metadata.driverKey,
|
||||
runnerTaskLease: true, agentId: run!.agentId,
|
||||
...(context ? { runnerTaskCreationContextId: context.id } : {}) },
|
||||
}, async identity => {
|
||||
const params = { driverKey: replacementDriverKey, companyId: input.companyId, environmentId: environment!.id,
|
||||
issueId: lease.issueId, config: driverConfig, runId: identity.runId, creationOperationId: identity.id,
|
||||
agentId: run!.agentId, adapterType: "paperclip_runner",
|
||||
...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? { requestedExpiresAt: metadata.runnerTaskRequestedExpiresAt } : {}) };
|
||||
if (!context) return workers.call(plugin.id, "environmentAcquireLease", params, 30_000);
|
||||
const settled = context.trackWorker();
|
||||
let observedResource: Parameters<typeof queueSandboxCreationCleanup>[1] | undefined;
|
||||
let dispatched = false;
|
||||
try {
|
||||
return await context.wait(() => {
|
||||
dispatched = true;
|
||||
return workers.call(plugin.id, "environmentAcquireLease", params, 30_000, undefined, settled, context.id,
|
||||
async result => {
|
||||
if (!result.providerLeaseId || result.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable");
|
||||
const resource = { id: identity.id, companyId: input.companyId, environmentId: environment!.id,
|
||||
providerLeaseId: result.providerLeaseId, providerMetadata: result.metadata };
|
||||
await recordSandboxCreationResource(db, resource);
|
||||
observedResource = resource;
|
||||
if (abandoned || !context.active || context.signal.aborted || Date.now() >= context.deadlineAtMs) {
|
||||
await queueSandboxCreationCleanup(db, resource);
|
||||
}
|
||||
});
|
||||
});
|
||||
} catch (error) {
|
||||
if (!dispatched) settled();
|
||||
abandoned = true;
|
||||
if (observedResource) await queueSandboxCreationCleanup(db, observedResource);
|
||||
throw error;
|
||||
}
|
||||
});
|
||||
if (!fresh.providerLeaseId || fresh.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable");
|
||||
await recordSandboxCreationResource(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId,
|
||||
environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId });
|
||||
const {runnerTaskIdentity:_identity,runnerTaskPriorIdentity:_prior,runnerTaskCredentialHandoff:_handoff,
|
||||
runnerTaskResumeFrom:_resume,runnerTaskPreviousLeaseId:_previous,...rest}=metadata;
|
||||
const replacementMetadata={...rest,providerMetadata:fresh.metadata ?? {},runnerTaskReused:false,runnerTaskDispatched:false,
|
||||
runnerTaskExecutionStopped:false,runnerTaskSessionRetained:false};
|
||||
{
|
||||
try {
|
||||
context?.assertOpen();
|
||||
if (!context) await recordSandboxCreationResource(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId,
|
||||
environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId });
|
||||
const expiresAt=fresh.expiresAt ? new Date(fresh.expiresAt) : null;
|
||||
const deadline=typeof metadata.runnerTaskRequestedExpiresAt === "string" ? Date.parse(metadata.runnerTaskRequestedExpiresAt) : null;
|
||||
if ((expiresAt && (!Number.isFinite(expiresAt.getTime()) || expiresAt.getTime() <= Date.now())) ||
|
||||
(deadline !== null && (!Number.isFinite(deadline) || !expiresAt || expiresAt.getTime() > deadline)))
|
||||
throw new Error("Plugin replacement task expiry is not bounded");
|
||||
await attachSandboxCreation(db, String(fresh.metadata!.creationOperationId), { companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId }, async tx => {
|
||||
context?.assertOpen();
|
||||
const [updated] = await tx.update(environmentLeases).set({providerLeaseId:fresh.providerLeaseId,expiresAt,
|
||||
metadata:{...replacementMetadata,creationOperationId:fresh.metadata!.creationOperationId},
|
||||
}).where(and(eq(environmentLeases.id,lease.id),eq(environmentLeases.providerLeaseId,taskId),
|
||||
eq(environmentLeases.status,"active"))).returning({id:environmentLeases.id});
|
||||
eq(environmentLeases.status,"active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`)).returning({id:environmentLeases.id});
|
||||
if (!updated) throw new Error("Plugin task replacement ownership changed");
|
||||
return (await environmentService(tx).getLeaseById(updated.id))!;
|
||||
const attached = (await environmentService(tx).getLeaseById(updated.id))!;
|
||||
context?.assertOpen();
|
||||
return attached;
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
if (context) await queueSandboxCreationCleanup(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId,
|
||||
environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId, providerMetadata: fresh.metadata });
|
||||
throw error;
|
||||
}
|
||||
return executeEnvironmentTask(db, workers, input);
|
||||
}
|
||||
@@ -275,14 +335,17 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
return snapshot.credential;
|
||||
});
|
||||
}
|
||||
const result = await workers.call(plugin.id, "environmentTask", {
|
||||
const params = {
|
||||
driverKey: metadata.driverKey, companyId: input.companyId, environmentId: environment?.id ?? null,
|
||||
issueId: lease.issueId, config: driverConfig,
|
||||
taskId, runId: run?.id ?? (activeProbe ? String(probe.runId) : null), agentId: run?.agentId ?? null, projectIds,
|
||||
lease: { providerLeaseId: lease.providerLeaseId, metadata: lease.metadata ?? undefined },
|
||||
...(workspaceBinding ? { workspace: workspaceBinding } : {}),
|
||||
...(freshWorkspace && scopesProjects ? { fresh: freshWorkspace } : {}), operation,
|
||||
}, operation.kind === "copilot_metadata" ? Math.max(1, Math.min(40_000, Number(probe?.expiresAtMs) - Date.now())) : 15_000);
|
||||
};
|
||||
const timeout = operation.kind === "copilot_metadata" ? Math.max(1, Math.min(40_000, Number(probe?.expiresAtMs) - Date.now()))
|
||||
: 15_000;
|
||||
const result = await callWorker("environmentTask", params, timeout);
|
||||
const parsed = parseEnvironmentTaskResult(operation, taskId, result);
|
||||
if (operation.kind === "read_file") {
|
||||
const [current] = await db.select({ lease: environmentLeases, run: heartbeatRuns }).from(environmentLeases)
|
||||
@@ -316,7 +379,9 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
}
|
||||
if ((parsed.kind === "status" || parsed.kind === "copilot_metadata") && parsed.executionStopped === true) {
|
||||
await db.update(environmentLeases).set({
|
||||
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ runnerTaskExecutionStopped: true, runnerTaskSessionRetained: parsed.kind === "status" && parsed.sessionStateAvailable === true })}::jsonb`,
|
||||
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ runnerTaskExecutionStopped: true,
|
||||
runnerTaskSessionRetained: parsed.kind === "status" && parsed.sessionStateAvailable === true,
|
||||
...(parsed.kind === "status" && parsed.preparationStopped === true ? { runnerTaskPreparationStopped: true } : {}) })}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId)));
|
||||
}
|
||||
if (operation.kind === "complete") {
|
||||
@@ -353,16 +418,22 @@ export async function settleDispatchedEnvironmentTask(db: Db, workers: PluginWor
|
||||
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"runnerTaskRetiring":true}'::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId)))
|
||||
.returning({ metadata: environmentLeases.metadata, heartbeatRunId: environmentLeases.heartbeatRunId });
|
||||
if (current?.metadata?.runnerTaskDispatched !== true) {
|
||||
const startup = current?.metadata?.runnerTaskStartup;
|
||||
const startupCleanup = current?.metadata?.runnerTaskStartupCleanup as { requiresPreparationStop?: boolean } | undefined;
|
||||
if (current?.metadata?.runnerTaskDispatched !== true && !startup && !startupCleanup) {
|
||||
if (current?.metadata) await settlePluginTaskCredentialHandoff(db, { companyId: lease.companyId, leaseId: lease.id }, current.metadata,
|
||||
async () => { throw new Error("Undispatched task must not request credentials"); });
|
||||
return current?.metadata?.runnerTaskExecutionStopped === true;
|
||||
}
|
||||
const scope = { companyId: lease.companyId, leaseId: lease.id };
|
||||
try {
|
||||
const contextId = (startupCleanup as { contextId?: unknown } | undefined)?.contextId ?? (startup as { contextId?: unknown } | undefined)?.contextId;
|
||||
const draining = typeof contextId === "string" ? workers.drainStartup?.(contextId) : undefined;
|
||||
void draining?.catch(() => undefined);
|
||||
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "stop" } });
|
||||
const state = await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "status" } });
|
||||
if (state.kind !== "status" || state.executionStopped !== true) throw new Error("Plugin Runner termination is unconfirmed");
|
||||
if (state.kind !== "status" || state.executionStopped !== true || ((startup || startupCleanup?.requiresPreparationStop) && state.preparationStopped !== true)) throw new Error("Plugin Runner termination is unconfirmed");
|
||||
await draining;
|
||||
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "complete" } });
|
||||
return true;
|
||||
} catch {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
||||
import { enableRunFailureReporting } from "../__tests__/helpers/run-failure-reporting.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { eq, inArray } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, completionContracts, createDb, environmentLeases, environments, heartbeatRuns, issues, issueRecoveryActions, issueThreadInteractions, nativeRunFinalizations, nativeRunResults, statusDecisions, workAssessments } from "@paperclipai/db";
|
||||
import {
|
||||
@@ -78,6 +78,55 @@ describeEmbeddedPostgres("reconcileAbandonedExecutionControl reports a genuine f
|
||||
return { companyId, agentId, issueId, runId };
|
||||
}
|
||||
|
||||
it("recognizes pending cleanup selected only by the run deadline batch", async () => {
|
||||
const f = await seedAbandonedRunFixture(), now = new Date();
|
||||
const cleanupIds = Array.from({ length: 51 }, () => randomUUID());
|
||||
const hidden = cleanupIds[50]!;
|
||||
await db.insert(heartbeatRuns).values(cleanupIds.map(id => ({ id, companyId: f.companyId, agentId: f.agentId,
|
||||
nativeIssueId: f.issueId, runtimeMode: "native", status: "running",
|
||||
executionControlDeadlineAt: id === hidden ? new Date(now.getTime() - 120_000) : null })));
|
||||
await db.insert(nativeRunFinalizations).values(cleanupIds.map(runId => ({ runId, companyId: f.companyId,
|
||||
issueId: f.issueId, phase: "terminal_failure", failureCode: "native_startup_cleanup_pending",
|
||||
controlDeadlineAt: new Date(now.getTime() - (runId === hidden ? 60_000 : 120_000)) })));
|
||||
const attempted: string[] = [];
|
||||
try {
|
||||
await reconcileAbandonedExecutionControl(db, now, { startupCleanup: async ({ runId }) => { attempted.push(runId); throw new Error("Provider unavailable"); } });
|
||||
expect(attempted).toContain(hidden);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, hidden)))[0].status).toBe("running");
|
||||
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, hidden)))[0].controlDeadlineAt! > now).toBe(true);
|
||||
} finally {
|
||||
await db.update(nativeRunFinalizations).set({ controlDeadlineAt: null }).where(inArray(nativeRunFinalizations.runId, cleanupIds));
|
||||
await db.update(heartbeatRuns).set({ executionControlDeadlineAt: null }).where(inArray(heartbeatRuns.id, cleanupIds));
|
||||
}
|
||||
});
|
||||
|
||||
it("moves failed startup cleanup retries behind later expired runs", async () => {
|
||||
const f = await seedAbandonedRunFixture();
|
||||
const now = new Date();
|
||||
const cleanupIds = Array.from({ length: 50 }, () => randomUUID());
|
||||
const firstDeadline = new Date(now.getTime() - 120_000);
|
||||
await db.insert(heartbeatRuns).values(cleanupIds.map(id => ({ id, companyId: f.companyId, agentId: f.agentId,
|
||||
nativeIssueId: f.issueId, runtimeMode: "native", status: "running", executionControlDeadlineAt: firstDeadline,
|
||||
contextSnapshot: { issueId: f.issueId } })));
|
||||
await db.insert(nativeRunFinalizations).values(cleanupIds.map(runId => ({ runId, companyId: f.companyId,
|
||||
issueId: f.issueId, phase: "terminal_failure", failureCode: "native_startup_cleanup_pending", controlDeadlineAt: firstDeadline })));
|
||||
const attempted: string[] = [];
|
||||
const startupCleanup = async ({ runId }: { runId: string }) => { attempted.push(runId); throw new Error("Provider unavailable"); };
|
||||
try {
|
||||
await reconcileAbandonedExecutionControl(db, now, { startupCleanup });
|
||||
await reconcileAbandonedExecutionControl(db, now, { startupCleanup });
|
||||
expect(new Set(attempted)).toEqual(new Set(cleanupIds));
|
||||
expect(attempted).toHaveLength(50);
|
||||
const [later] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
expect(later?.status).toBe("failed");
|
||||
const retries = await db.select().from(nativeRunFinalizations).where(inArray(nativeRunFinalizations.runId, cleanupIds));
|
||||
expect(retries.every(row => row.controlDeadlineAt && row.controlDeadlineAt > now)).toBe(true);
|
||||
} finally {
|
||||
await db.update(nativeRunFinalizations).set({ controlDeadlineAt: null }).where(inArray(nativeRunFinalizations.runId, cleanupIds));
|
||||
await db.update(heartbeatRuns).set({ executionControlDeadlineAt: null }).where(inArray(heartbeatRuns.id, cleanupIds));
|
||||
}
|
||||
});
|
||||
|
||||
it("reports exactly one Sentry event for a genuine finalization-deadline failure", async () => {
|
||||
const { runId } = await seedAbandonedRunFixture();
|
||||
const captureCallsBefore = mockCaptureRunFailure.mock.calls.length;
|
||||
|
||||
@@ -8,7 +8,7 @@ import { notifyInstructionCollection } from "./instruction-collection-notificati
|
||||
import { notifyExecutionRunWork } from "./execution-control-notifications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { and, eq, isNotNull, lte, sql } from "drizzle-orm";
|
||||
import { and, asc, eq, isNotNull, lte, sql } from "drizzle-orm";
|
||||
import {
|
||||
agents,
|
||||
heartbeatRuns,
|
||||
@@ -22,17 +22,25 @@ import { hasLiveLegacyController } from "./legacy-controller-lease.js";
|
||||
import { reportRunFailure } from "./run-failure-report.js";
|
||||
import { getNativeReviewAssignment, readNativeReviewAssignmentContext } from "./native-runtime/native-review-participant.js";
|
||||
import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js";
|
||||
import { EXECUTION_RECONCILIATION_INTERVAL_MS } from "./execution-control-deadline.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import { NATIVE_STARTUP_CLEANUP_PENDING, readNativeStartupContext, promoteNativeStartupCleanup } from "./native-runtime/native-startup-cleanup.js";
|
||||
|
||||
/** Only newly recorded control deadlines are eligible. Upgrades never replay ambiguous historical runs. */
|
||||
export async function reconcileAbandonedExecutionControl(
|
||||
db: Db,
|
||||
now = new Date(),
|
||||
options: { startupCleanup?: (scope: { companyId: string; runId: string }) => Promise<unknown> } = {},
|
||||
) {
|
||||
const nativeDue = await db
|
||||
.select({
|
||||
runId: nativeRunFinalizations.runId,
|
||||
issueId: nativeRunFinalizations.issueId,
|
||||
companyId: nativeRunFinalizations.companyId,
|
||||
failureCode: nativeRunFinalizations.failureCode,
|
||||
leaseOwner: nativeRunFinalizations.leaseOwner,
|
||||
controllerGeneration: nativeRunFinalizations.controllerGeneration,
|
||||
attempt: nativeRunFinalizations.attempt,
|
||||
})
|
||||
.from(nativeRunFinalizations)
|
||||
.where(
|
||||
@@ -41,20 +49,28 @@ export async function reconcileAbandonedExecutionControl(
|
||||
lte(nativeRunFinalizations.controlDeadlineAt, now),
|
||||
),
|
||||
)
|
||||
.orderBy(asc(nativeRunFinalizations.controlDeadlineAt), asc(nativeRunFinalizations.runId))
|
||||
.limit(50);
|
||||
const controlDue = await db
|
||||
.select({
|
||||
runId: heartbeatRuns.id,
|
||||
companyId: heartbeatRuns.companyId,
|
||||
context: heartbeatRuns.contextSnapshot,
|
||||
nativeIssueId: nativeRunFinalizations.issueId,
|
||||
failureCode: nativeRunFinalizations.failureCode,
|
||||
leaseOwner: nativeRunFinalizations.leaseOwner,
|
||||
controllerGeneration: nativeRunFinalizations.controllerGeneration,
|
||||
attempt: nativeRunFinalizations.attempt,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.leftJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
|
||||
.where(
|
||||
and(
|
||||
isNotNull(heartbeatRuns.executionControlDeadlineAt),
|
||||
lte(heartbeatRuns.executionControlDeadlineAt, now),
|
||||
),
|
||||
)
|
||||
.orderBy(asc(heartbeatRuns.executionControlDeadlineAt), asc(heartbeatRuns.id))
|
||||
.limit(50);
|
||||
const due = [
|
||||
...new Map(
|
||||
@@ -62,10 +78,10 @@ export async function reconcileAbandonedExecutionControl(
|
||||
...nativeDue,
|
||||
...controlDue.map((row) => ({
|
||||
...row,
|
||||
issueId:
|
||||
issueId: row.nativeIssueId ?? (
|
||||
typeof row.context?.issueId === "string"
|
||||
? row.context.issueId
|
||||
: null,
|
||||
: null),
|
||||
})),
|
||||
].map((row) => [row.runId, row]),
|
||||
).values(),
|
||||
@@ -78,6 +94,25 @@ export async function reconcileAbandonedExecutionControl(
|
||||
while (nextCandidate < due.length) {
|
||||
const candidate = due[nextCandidate++]!;
|
||||
try {
|
||||
if (candidate.failureCode === NATIVE_STARTUP_CLEANUP_PENDING) {
|
||||
if (!options.startupCleanup) continue;
|
||||
try {
|
||||
await options.startupCleanup({ companyId: candidate.companyId, runId: candidate.runId });
|
||||
} finally {
|
||||
await db.transaction(async tx => {
|
||||
if (candidate.issueId) await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, candidate.issueId)).for("update");
|
||||
const [owner] = await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, candidate.runId)).for("update");
|
||||
if (owner?.failureCode !== NATIVE_STARTUP_CLEANUP_PENDING || !owner.controlDeadlineAt || owner.controlDeadlineAt > now ||
|
||||
owner.leaseOwner !== candidate.leaseOwner || owner.controllerGeneration !== candidate.controllerGeneration || owner.attempt !== candidate.attempt) return;
|
||||
const cleanupRetryAt = new Date(Math.max(now.getTime(), Date.now()) + EXECUTION_RECONCILIATION_INTERVAL_MS);
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization);
|
||||
await tx.update(nativeRunFinalizations).set({ controlDeadlineAt: cleanupRetryAt }).where(eq(nativeRunFinalizations.runId, candidate.runId));
|
||||
await tx.update(heartbeatRuns).set({ executionControlDeadlineAt: cleanupRetryAt }).where(and(eq(heartbeatRuns.id, candidate.runId),
|
||||
isNotNull(heartbeatRuns.executionControlDeadlineAt), lte(heartbeatRuns.executionControlDeadlineAt, now)));
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// Set inside the transaction only when the write below genuinely
|
||||
// transitions the run into "failed". Read after the transaction
|
||||
// commits, so a rolled-back write never reports a false failure.
|
||||
@@ -128,6 +163,18 @@ export async function reconcileAbandonedExecutionControl(
|
||||
].some((deadline) => deadline && deadline <= now)
|
||||
)
|
||||
return false;
|
||||
if (coordinator?.failureCode === NATIVE_STARTUP_CLEANUP_PENDING) return false;
|
||||
const startupContext = coordinator && readNativeStartupContext(run, coordinator);
|
||||
if (startupContext) {
|
||||
if (!await promoteNativeStartupCleanup(tx as unknown as Db, run, coordinator!, now)) {
|
||||
const retryAt = new Date(Math.max(now.getTime() + EXECUTION_RECONCILIATION_INTERVAL_MS,
|
||||
startupContext.deadlineAtMs, coordinator!.leaseExpiresAt?.getTime() ?? 0));
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization);
|
||||
await tx.update(nativeRunFinalizations).set({ controlDeadlineAt: retryAt }).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
await tx.update(heartbeatRuns).set({ executionControlDeadlineAt: retryAt }).where(eq(heartbeatRuns.id, run.id));
|
||||
}
|
||||
return false;
|
||||
}
|
||||
await tx
|
||||
.update(heartbeatRuns)
|
||||
.set({ executionControlDeadlineAt: null })
|
||||
|
||||
@@ -398,6 +398,7 @@ import {
|
||||
isNativeRunnerOwnershipHeld,
|
||||
NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE,
|
||||
} from "./native-runtime/native-runner-ownership.js";
|
||||
import { NativeStartupCleanupPendingError, requestNativeStartupCleanup, verifyNativeStartupCleanup, readNativeStartupCleanup } from "./native-runtime/native-startup-cleanup.js";
|
||||
import {
|
||||
emitAgentTaskRun,
|
||||
emitAgentTaskRunById,
|
||||
@@ -1436,6 +1437,14 @@ export function heartbeatService(
|
||||
}
|
||||
|
||||
const runControl = createHeartbeatRunControl(db, {
|
||||
verifyStartupCleanup: async run => {
|
||||
await verifyNativeStartupCleanup(db, run, () => envOrchestrator.releaseForRun({
|
||||
heartbeatRunId: run.id, companyId: run.companyId, agentId: run.agentId, status: "expired",
|
||||
cancelActiveWork: true, providerResourceDisposition: "destroy",
|
||||
}));
|
||||
const marker = readNativeStartupCleanup(run)!;
|
||||
await options.pluginWorkerManager?.retireStartup?.(marker.contextId);
|
||||
},
|
||||
getRun,
|
||||
getRunCleanupState,
|
||||
getRunExecutionWait,
|
||||
@@ -1759,7 +1768,7 @@ export function heartbeatService(
|
||||
},
|
||||
});
|
||||
})().catch((error) => {
|
||||
if (!nativeRecoveryDeadlines.has(runId)) {
|
||||
if (!nativeRecoveryDeadlines.has(runId) || nativeRecoveryDeadlines.get(runId) === pending) {
|
||||
scheduleNativeSessionResumeDispatch(runId, new Date(Date.now() + 30_000), mode);
|
||||
}
|
||||
logger.error(
|
||||
@@ -1956,6 +1965,19 @@ export function heartbeatService(
|
||||
Parameters<typeof cleanupGitHubOperationLaunchers>[0] | null = null;
|
||||
let nativeSessionResumeScheduled = false;
|
||||
let nativeOwnershipHeld = false;
|
||||
let nativeStartupCleanupScheduled = false;
|
||||
let nativeStartupCleanupError: NativeStartupCleanupPendingError | undefined;
|
||||
const requestStartupCleanup = async (error: NativeStartupCleanupPendingError) => {
|
||||
nativeStartupCleanupError = error;
|
||||
try {
|
||||
nativeStartupCleanupScheduled = await requestNativeStartupCleanup(db, { companyId: run.companyId, agentId: run.agentId, runId: run.id,
|
||||
contextId: error.contextId, deadlineAtMs: error.deadlineAtMs, leaseOwner: error.leaseOwner,
|
||||
leaseId: error.leaseId, requiresPreparationStop: error.requiresPreparationStop });
|
||||
} catch (writeError) {
|
||||
scheduleNativeSessionResumeDispatch(run.id, new Date(Date.now() + 30_000), "restart");
|
||||
throw writeError;
|
||||
}
|
||||
};
|
||||
let instructionWorkingCopy: ReturnType<typeof createHeartbeatInstructionWorkingCopy> | null = null;
|
||||
let nativeDispatchStarted = false;
|
||||
let nativeWorkspaceFinalizeScheduled = false;
|
||||
@@ -3527,6 +3549,10 @@ export function heartbeatService(
|
||||
}));
|
||||
} catch (err) {
|
||||
await persistUsageCaptureFailure?.();
|
||||
if (err instanceof NativeStartupCleanupPendingError) {
|
||||
await requestStartupCleanup(err);
|
||||
return;
|
||||
}
|
||||
if (err instanceof NativeControllerDetachedForRestartError) {
|
||||
nativeSessionResumeScheduled = true;
|
||||
return;
|
||||
@@ -3659,7 +3685,12 @@ export function heartbeatService(
|
||||
});
|
||||
}
|
||||
} catch (outerErr) {
|
||||
if (
|
||||
if (nativeStartupCleanupScheduled || nativeStartupCleanupError || outerErr instanceof NativeStartupCleanupPendingError) {
|
||||
const pending = outerErr instanceof NativeStartupCleanupPendingError ? outerErr : nativeStartupCleanupError;
|
||||
if (!nativeStartupCleanupScheduled && pending) await requestStartupCleanup(pending).catch(error => {
|
||||
logger.warn({ err: error, runId: run.id }, "Native startup cleanup intent will retry through owned recovery");
|
||||
});
|
||||
} else if (
|
||||
nativeOwnershipHeld ||
|
||||
outerErr instanceof NativeRunnerOwnershipUnverifiedError
|
||||
) {
|
||||
@@ -3696,7 +3727,7 @@ export function heartbeatService(
|
||||
);
|
||||
} else {
|
||||
await preserveNativeWorkspaceOnFailure();
|
||||
if (!nativeOwnershipHeld) await runCompletion.failRunSetup({
|
||||
if (!nativeOwnershipHeld && !nativeStartupCleanupScheduled && !nativeStartupCleanupError) await runCompletion.failRunSetup({
|
||||
run, runId, outerErr, signal: executionControl.controller.signal,
|
||||
providerDispatchStarted: legacyAdapterEntered || nativeDispatchStarted,
|
||||
identityRedactor, readFailureReportSecrets,
|
||||
@@ -3746,7 +3777,7 @@ export function heartbeatService(
|
||||
latestRun &&
|
||||
!nativeSessionResumeScheduled &&
|
||||
!nativeWorkspaceFinalizeScheduled &&
|
||||
!nativeOwnershipHeld
|
||||
!nativeOwnershipHeld && !nativeStartupCleanupScheduled && !nativeStartupCleanupError
|
||||
) {
|
||||
latestRun = await terminalizeRunOnLeaseRelease(latestRun).catch(
|
||||
(terminalizeErr) => {
|
||||
@@ -3772,7 +3803,7 @@ export function heartbeatService(
|
||||
if (
|
||||
!nativeSessionResumeScheduled &&
|
||||
!nativeWorkspaceFinalizeScheduled &&
|
||||
!nativeOwnershipHeld
|
||||
!nativeOwnershipHeld && !nativeStartupCleanupScheduled && !nativeStartupCleanupError
|
||||
) {
|
||||
// Keep launchers during same-run recovery. At a terminal boundary all
|
||||
// operations have settled; clean before the remote lease can be stopped.
|
||||
@@ -3895,7 +3926,7 @@ export function heartbeatService(
|
||||
}).where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "cancelled")));
|
||||
}
|
||||
if (latestRun?.runtimeMode === "native" && isHeartbeatRunTerminalStatus(latestRun.status) &&
|
||||
!nativeSessionResumeScheduled && !nativeWorkspaceFinalizeScheduled && !nativeOwnershipHeld) {
|
||||
!nativeSessionResumeScheduled && !nativeWorkspaceFinalizeScheduled && !nativeOwnershipHeld && !nativeStartupCleanupScheduled && !nativeStartupCleanupError) {
|
||||
await settleNativeRunProcess(db, run.id).catch(error => {
|
||||
logger.warn({ err: error, runId: run.id }, "failed to release settled native process identity");
|
||||
});
|
||||
@@ -3932,7 +3963,7 @@ export function heartbeatService(
|
||||
caseHandoffCleanupSettled = hasStoppedLocalProcessTree(latestRun);
|
||||
}
|
||||
if (caseHandoffCleanupSettled && latestRun && !nativeSessionResumeScheduled && !nativeWorkspaceFinalizeScheduled &&
|
||||
!nativeOwnershipHeld && isHeartbeatRunTerminalStatus(latestRun.status)) {
|
||||
!nativeOwnershipHeld && !nativeStartupCleanupScheduled && !nativeStartupCleanupError && isHeartbeatRunTerminalStatus(latestRun.status)) {
|
||||
const handoff = await pendingCaseHandoffForRun(db, run.id);
|
||||
if (handoff) await recordCaseHandoffExecutionReleased(db, run.id, handoff.issueId);
|
||||
}
|
||||
|
||||
@@ -168,13 +168,13 @@ describe("heartbeat recovery module callbacks", () => {
|
||||
expect(database.access).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("persists an authentication ownership hold before its event without terminating or retrying", async () => {
|
||||
it.each(["adopted_runner_authentication_timeout"] as const)("persists a %s ownership hold before its event without terminating or retrying", async reason => {
|
||||
const database = guardedDatabase();
|
||||
const deps = callbacks(database.db);
|
||||
const run = { id: "run", runtimeMode: "native", status: "running" } as Run;
|
||||
const blocked = { ...run, nativePhase: "terminal_failure", errorCode: NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE };
|
||||
deps.setRunStatusFromLive.mockResolvedValue({ updated: true, run: blocked });
|
||||
expect(await createHeartbeatRecovery(database.db, deps).markNativeOwnershipUnverified(run, { reason: "adopted_runner_authentication_timeout" })).toBe(blocked);
|
||||
expect(await createHeartbeatRecovery(database.db, deps).markNativeOwnershipUnverified(run, { reason })).toBe(blocked);
|
||||
expect(deps.setRunStatusFromLive).toHaveBeenCalledWith("run", "running", ["running"], expect.objectContaining({ nativePhase: "terminal_failure" }));
|
||||
expect(deps.appendRunEvent).toHaveBeenCalledAfter(deps.setRunStatusFromLive);
|
||||
expect(deps.cancelHeartbeatNativeRun).not.toHaveBeenCalled();
|
||||
@@ -207,6 +207,42 @@ describePostgres("heartbeat recovery module ownership", () => {
|
||||
return (await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, invocationSource: "automation", status: "running", runtimeMode: "legacy" }).returning())[0];
|
||||
}
|
||||
|
||||
it("routes an expired pre-capability startup to cleanup before generic orphan ownership holds", async () => {
|
||||
const run = await seedRun(), leaseId = randomUUID(), environmentId = randomUUID(), leaseOwner = randomUUID();
|
||||
const deadlineAtMs = Date.now() - 1_000, contextId = randomUUID();
|
||||
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
|
||||
await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, run.issueId!));
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, nativePhase: "observed",
|
||||
runnerInstanceId: randomUUID(), nativeSessionId: randomUUID(), runnerProfileJson: { nativeStartupContext: {
|
||||
schema: "paperclip.native-startup-context.v1", companyId: run.companyId, agentId: run.agentId, runId: run.id,
|
||||
contextId, deadlineAtMs, leaseId, leaseOwner, controllerGeneration: 1, attempt: 1 } } }).where(eq(heartbeatRuns.id, run.id));
|
||||
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
|
||||
phase: "observed", attempt: 1, leaseOwner, controllerGeneration: 1, leaseExpiresAt: new Date(Date.now() - 1_000),
|
||||
controllerBootId: randomUUID(), controllerPid: process.pid, controllerProcessStartedAt: new Date() });
|
||||
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "plugin", config: {} });
|
||||
await db.insert(environmentLeases).values({ id: leaseId, companyId: run.companyId, environmentId, heartbeatRunId: run.id,
|
||||
issueId: run.issueId, provider: "plugin", providerLeaseId: "early-task", status: "active", metadata: { runnerTaskLease: true } });
|
||||
const deps = callbacks(db);
|
||||
deps.getRun.mockImplementation(async id => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0] ?? null);
|
||||
native.claims.mockImplementationOnce(claimNativeRestartRecoveries);
|
||||
try {
|
||||
await createHeartbeatRecovery(db, deps).recoverActiveExecutions({ runId: run.id });
|
||||
expect(native.claims).toHaveBeenCalledWith(expect.objectContaining({ runIds: [run.id] }));
|
||||
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
|
||||
expect(current).toMatchObject({ status: "running", errorCode: "native_startup_cleanup_pending",
|
||||
resultJson: { nativeStartupCleanup: { contextId, state: "pending" } } });
|
||||
expect(deps.executeRun).not.toHaveBeenCalled();
|
||||
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
await db.delete(environmentLeases).where(eq(environmentLeases.id, leaseId));
|
||||
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, run.id));
|
||||
await db.delete(activityLog).where(eq(activityLog.runId, run.id));
|
||||
await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, run.issueId!));
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
|
||||
}
|
||||
});
|
||||
|
||||
async function ownedProcess() {
|
||||
const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)", `paperclip-recovery-${randomUUID()}`], {
|
||||
stdio: "ignore", detached: true, env: { PATH: process.env.PATH },
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { readNativeStartupCleanup, readNativeStartupContext } from "../native-runtime/native-startup-cleanup.js";
|
||||
import { pendingFailedNativeRetryCancellationCondition } from "../native-runtime/native-cancellation-request.js";
|
||||
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
|
||||
import { registerLiveRecoveryWork } from "./live-recovery-work.js";
|
||||
@@ -1038,7 +1039,9 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
? "Native restart recovery is waiting for safe ownership evidence"
|
||||
: disposition.kind === "already_finalized"
|
||||
? "Native restart recovery found an already-finalized result"
|
||||
: "Native restart recovery blocked ambiguous or conflicting ownership",
|
||||
: disposition.reason === "native_startup_cleanup_pending"
|
||||
? "Native startup timed out; cancellation and verified cleanup are queued"
|
||||
: "Native restart recovery blocked ambiguous or conflicting ownership",
|
||||
payload: {
|
||||
restartKind: isClaim ? disposition.restartKind : restartKind,
|
||||
recoveryRequestId: isClaim
|
||||
@@ -2251,6 +2254,10 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
nativeControllerProcessStartedAt:
|
||||
nativeRunFinalizations.controllerProcessStartedAt,
|
||||
nativeControllerLeaseExpiresAt: nativeRunFinalizations.leaseExpiresAt,
|
||||
nativeLeaseOwner: nativeRunFinalizations.leaseOwner,
|
||||
nativeControllerGeneration: nativeRunFinalizations.controllerGeneration,
|
||||
nativeAttempt: nativeRunFinalizations.attempt,
|
||||
nativeResultId: nativeRunFinalizations.resultId,
|
||||
nativeWorkspaceFinalizationPending: pendingNativeWorkspaceFinalizationCondition(db),
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
@@ -2271,7 +2278,10 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
&& !isNativeRunnerOwnershipHeld(run)
|
||||
&& !activeRunExecutions.has(run.id) && !runningProcesses.has(run.id),
|
||||
).map(({ run }) => run.id);
|
||||
const remoteRecoveryRunIds = new Set(await listExpiredNativeRunnerAuthorities(db, now, expiredNativeRunIds));
|
||||
const earlyStartupRunIds = activeRuns.filter(row => expiredNativeRunIds.includes(row.run.id) &&
|
||||
readNativeStartupContext(row.run, { leaseOwner: row.nativeLeaseOwner, controllerGeneration: row.nativeControllerGeneration ?? 0,
|
||||
attempt: row.nativeAttempt ?? 0, resultId: row.nativeResultId })).map(row => row.run.id);
|
||||
const remoteRecoveryRunIds = new Set([...earlyStartupRunIds, ...await listExpiredNativeRunnerAuthorities(db, now, expiredNativeRunIds)]);
|
||||
if (remoteRecoveryRunIds.size) {
|
||||
const recovery = await recoverNativeRunsAfterRestart(now, [...remoteRecoveryRunIds]);
|
||||
for (const claim of recovery.claims) resumedRunIds.add(claim.runId);
|
||||
@@ -2321,6 +2331,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
nativeControllerLeaseExpiresAt,
|
||||
nativeWorkspaceFinalizationPending,
|
||||
} of activeRuns) {
|
||||
if (run.runtimeMode === "native" && readNativeStartupCleanup(run)) continue;
|
||||
if (parseObject(run.contextSnapshot).nativeQuestionCancellation) continue;
|
||||
// Native recovery owns these dispositions. Its writes happened after this
|
||||
// inventory was read; do not reap or overwrite them using the old row.
|
||||
|
||||
@@ -123,7 +123,8 @@ import {
|
||||
} from "../native-runtime/index.js";
|
||||
import { runningProcesses } from "../../adapters/index.js";
|
||||
import { parseObject } from "../../adapters/utils.js";
|
||||
import { isNativeRunnerOwnershipHeld } from "../native-runtime/native-runner-ownership.js";
|
||||
import { isNativeRunnerOwnershipHeld, nativeRunnerOwnershipNotHeldCondition } from "../native-runtime/native-runner-ownership.js";
|
||||
import { readNativeStartupCleanup, acknowledgeNativeStartupCleanup } from "../native-runtime/native-startup-cleanup.js";
|
||||
import {
|
||||
withCurrentBudgetEnforcement,
|
||||
type BudgetEnforcementScope,
|
||||
@@ -177,6 +178,7 @@ export interface HeartbeatRunControlDependencies extends Pick<ReturnType<typeof
|
||||
Pick<ReturnType<typeof createHeartbeatLifecycle>,
|
||||
"mergeRunStopMetadataForAgent" | "setRunStatusFromLive" | "setRunStatus" | "setWakeupStatus" | "appendRunEvent">,
|
||||
Pick<HeartbeatRetryDependencies, "finalizeAgentStatus"> {
|
||||
verifyStartupCleanup?: (run: typeof heartbeatRuns.$inferSelect) => Promise<void>;
|
||||
options: {
|
||||
closeWarmNativeSessionsForRun?: (input: {
|
||||
runId: string;
|
||||
@@ -1399,13 +1401,19 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
|
||||
}
|
||||
let terminationSettled = false;
|
||||
try {
|
||||
await cancelHeartbeatNativeRun({
|
||||
db,
|
||||
runId: run.id,
|
||||
reason,
|
||||
runtimeMode: run.runtimeMode,
|
||||
...(options.cancellationRequestId ? { cancellationRequestId: options.cancellationRequestId } : {}),
|
||||
});
|
||||
const startupCleanupRun = run.runtimeMode === "native" ? await getRun(run.id) : null;
|
||||
if (startupCleanupRun && readNativeStartupCleanup(startupCleanupRun) && !control && !running && !activeRunExecutions.has(run.id)) {
|
||||
if (!dependencies.verifyStartupCleanup) throw conflict("Native startup cleanup worker unavailable");
|
||||
await dependencies.verifyStartupCleanup(startupCleanupRun);
|
||||
} else {
|
||||
await cancelHeartbeatNativeRun({
|
||||
db,
|
||||
runId: run.id,
|
||||
reason,
|
||||
runtimeMode: run.runtimeMode,
|
||||
...(options.cancellationRequestId ? { cancellationRequestId: options.cancellationRequestId } : {}),
|
||||
});
|
||||
}
|
||||
if (running) {
|
||||
await terminateHeartbeatRunProcess({
|
||||
pid: running.child.pid,
|
||||
@@ -1466,12 +1474,13 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
|
||||
}
|
||||
|
||||
const finishedAt = new Date();
|
||||
const persistedCancellationResult =
|
||||
run.runtimeMode === "native"
|
||||
? await getRun(run.id).then((current) =>
|
||||
parseObject(current?.resultJson),
|
||||
)
|
||||
: {};
|
||||
let persistedCancellationRun = run.runtimeMode === "native" ? await getRun(run.id) : null;
|
||||
if (persistedCancellationRun && readNativeStartupCleanup(persistedCancellationRun)) {
|
||||
if (!dependencies.verifyStartupCleanup) throw conflict("Native startup cleanup worker unavailable");
|
||||
await dependencies.verifyStartupCleanup(persistedCancellationRun);
|
||||
persistedCancellationRun = await getRun(run.id);
|
||||
}
|
||||
const persistedCancellationResult = parseObject(persistedCancellationRun?.resultJson);
|
||||
return await setRunStatusFromLive(
|
||||
run.id,
|
||||
"cancelled",
|
||||
@@ -1520,12 +1529,18 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
|
||||
persistedCancellationResult.nativeCancellation,
|
||||
}
|
||||
: {}),
|
||||
...(Object.hasOwn(persistedCancellationResult, "nativeStartupCleanup")
|
||||
? { nativeStartupCleanup: persistedCancellationResult.nativeStartupCleanup } : {}),
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
},
|
||||
undefined,
|
||||
pendingNativeRetry ? nativeRetryCancellationCommitCondition(persistedCancellationResult) : undefined,
|
||||
and(pendingNativeRetry ? nativeRetryCancellationCommitCondition(persistedCancellationResult)
|
||||
: persistedCancellationRun && readNativeStartupCleanup(persistedCancellationRun) ? and(nativeRunnerOwnershipNotHeldCondition(),
|
||||
sql`not exists (select 1 from ${nativeRunFinalizations} where ${nativeRunFinalizations.runId} = ${run.id} and ${nativeRunFinalizations.resultId} is not null)`)
|
||||
: undefined,
|
||||
run.runtimeMode === "native" ? sql`coalesce(${heartbeatRuns.resultJson}->'nativeStartupCleanup'->>'state', '') <> 'pending'` : undefined),
|
||||
);
|
||||
} catch (error) {
|
||||
if (processCancellationSettlement) {
|
||||
@@ -1557,6 +1572,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
|
||||
const cancelled = cancellation.run;
|
||||
|
||||
if (cancellation.updated && cancelled) {
|
||||
if (readNativeStartupCleanup(cancelled)) await acknowledgeNativeStartupCleanup(db, { companyId: cancelled.companyId, runId: cancelled.id });
|
||||
await setWakeupStatus(run.wakeupRequestId, "cancelled", {
|
||||
finishedAt: cancelled.finishedAt ?? new Date(),
|
||||
error: reason,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { readNativeStartupCleanup } from "../native-runtime/native-startup-cleanup.js";
|
||||
import { notifyStrandedAgentStatus } from "../live-run-recovery-notifications.js";
|
||||
import { notifyRunDispatch } from "../run-dispatch-notifications.js";
|
||||
import { preserveAcceptedRunUsageProjectionSql } from "../usage-receipts.js";
|
||||
@@ -1159,6 +1160,9 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
|
||||
.where(eq(heartbeatRuns.id, runId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
const previousStatus = previous?.run ?? null;
|
||||
if (status === "failed" && patch?.errorCode === "process_lost" && previousStatus?.runtimeMode === "native" && readNativeStartupCleanup(previousStatus)) {
|
||||
return { updated: false, run: previousStatus };
|
||||
}
|
||||
|
||||
if (previousStatus && (status === "cancelled" || status === "interrupted")) {
|
||||
patch = { ...patch, resultJson: cancellationResultJson(previousStatus, status, patch?.resultJson, patch?.errorCode, patch?.error) };
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { NativeStartupCleanupPendingError } from "../native-runtime/native-startup-cleanup.js";
|
||||
import { NativeControllerRecoveryWaitingError } from "../native-runtime/native-controller-recovery.js";
|
||||
|
||||
import { usesNativeUsageDelivery } from "../native-runtime/native-usage-accounting.js";
|
||||
@@ -1072,15 +1073,21 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
|
||||
&& !(nativeExecution && "access" in nativeExecution.workspace && nativeExecution.workspace.access === "none")) {
|
||||
input.effects.onProviderResourceDisposition("stop_and_retain");
|
||||
}
|
||||
await writeExecutionControl(db, DELIVERY_QUEUES.executionFinalization, tx => tx
|
||||
.update(heartbeatRuns)
|
||||
.set({ executionControlDeadlineAt: new Date(Date.now() + 60_000) })
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRuns.id, run.id),
|
||||
eq(heartbeatRuns.status, "running"),
|
||||
),
|
||||
), { companyId: run.companyId });
|
||||
try {
|
||||
await writeExecutionControl(db, DELIVERY_QUEUES.executionFinalization, tx => tx
|
||||
.update(heartbeatRuns)
|
||||
.set({ executionControlDeadlineAt: new Date(Date.now() + 60_000) })
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRuns.id, run.id),
|
||||
eq(heartbeatRuns.status, "running"),
|
||||
),
|
||||
), { companyId: run.companyId });
|
||||
} catch (writeError) {
|
||||
if (adapterErr instanceof NativeStartupCleanupPendingError) throw adapterErr;
|
||||
throw writeError;
|
||||
}
|
||||
if (adapterErr instanceof NativeStartupCleanupPendingError) throw adapterErr;
|
||||
if (
|
||||
issueRef &&
|
||||
context.resumeSessionGoalHeartbeat === true &&
|
||||
|
||||
@@ -32,8 +32,8 @@ const invocations = [];
|
||||
const send = (id, result) => process.stdout.write(JSON.stringify({jsonrpc:"2.0",id,result})+"\\n");
|
||||
readline.createInterface({input:process.stdin,crlfDelay:Infinity}).on("line", line => {
|
||||
const message = JSON.parse(line);
|
||||
if (message.method === "initialize") send(message.id, {ok:true,supportedMethods:["environmentTask","configChanged","health"]});
|
||||
else if (message.method === "environmentTask" || message.method === "environmentDestroyLease") {
|
||||
if (message.method === "initialize") send(message.id, {ok:true,supportedMethods:["environmentTask","environmentAcquireLease","configChanged","health"]});
|
||||
else if (message.method === "environmentTask" || message.method === "environmentDestroyLease" || message.method === "environmentAcquireLease") {
|
||||
held.push(message.id);
|
||||
invocations.push(message.paperclipInvocation?.id);
|
||||
if (message.params.config.hostWrite) process.stdout.write(JSON.stringify({jsonrpc:"2.0",id:"host-"+message.id,method:"companies.get",params:{companyId:"company"},paperclipInvocationId:message.paperclipInvocation.id})+"\\n");
|
||||
@@ -43,7 +43,7 @@ readline.createInterface({input:process.stdin,crlfDelay:Infinity}).on("line", li
|
||||
else if (message.method === "configChanged") {
|
||||
if (message.params.config.exit) process.exit(1);
|
||||
if (message.params.config.hostAfter) process.stdout.write(JSON.stringify({jsonrpc:"2.0",id:"late-host",method:"companies.get",params:{companyId:"company"},paperclipInvocationId:invocations.at(-1)})+"\\n");
|
||||
if (message.params.config.release) for (const id of held.splice(0)) send(id, {kind:"description",taskId:"task"});
|
||||
if (message.params.config.release) for (const id of held.splice(0)) { send(id, {providerLeaseId:"late-task",kind:"description",taskId:"task"}); if (message.params.config.duplicate) send(id, {providerLeaseId:"late-task",kind:"description",taskId:"task"}); }
|
||||
send(message.id, {});
|
||||
} else if (message.method === "shutdown") {send(message.id, {});setImmediate(()=>process.exit(0));}
|
||||
});
|
||||
@@ -59,6 +59,70 @@ async function startWorker(hostHandlers: Record<string, unknown> = {}) {
|
||||
}
|
||||
|
||||
describe("Core startup responsibility across real worker IPC", () => {
|
||||
it("drops only a dead acquisition reply placeholder while retaining real writes until verified retirement", async () => {
|
||||
let release!: () => void, writing!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), entered = new Promise<void>(resolve => { writing = resolve; });
|
||||
const manager = await startWorker({ "companies.get": async () => { writing(); await gate; return {}; } });
|
||||
const settled = vi.fn(), observe = vi.fn(async () => {});
|
||||
try {
|
||||
const acquisition = manager.call(manifest.id, "environmentAcquireLease", { companyId: "company", environmentId: "environment",
|
||||
driverKey: "tasks", config: { hostWrite: true }, runId: "run", agentId: "agent", adapterType: "paperclip_runner" },
|
||||
10, undefined, settled, "dead-acquire", observe);
|
||||
const rejected = expect(acquisition).rejects.toThrow("timed out");
|
||||
await entered; await rejected;
|
||||
await manager.stopWorker(manifest.id);
|
||||
let drained = false;
|
||||
const draining = manager.drainStartup!("dead-acquire").then(() => { drained = true; });
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(drained).toBe(false);
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
release(); await draining;
|
||||
expect(observe).not.toHaveBeenCalled();
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
await manager.retireStartup!("dead-acquire");
|
||||
expect(settled).toHaveBeenCalledOnce();
|
||||
} finally { release(); await manager.stopAll(); }
|
||||
}, 25_000);
|
||||
|
||||
it.each([false, true])("retains a received acquisition reply through failed recording and retirement (worker exits=%s)", async workerExits => {
|
||||
const manager = await startWorker(), settled = vi.fn();
|
||||
let release!: () => void, recording!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), started = new Promise<void>(resolve => { recording = resolve; });
|
||||
let unavailable = true;
|
||||
const observe = vi.fn(async (result: unknown) => {
|
||||
expect(result).toMatchObject({ providerLeaseId: "late-task" });
|
||||
if (unavailable) throw new Error("Database unavailable");
|
||||
recording(); await gate;
|
||||
});
|
||||
try {
|
||||
const acquire = manager.call(manifest.id, "environmentAcquireLease", { companyId: "company", environmentId: "environment",
|
||||
driverKey: "tasks", config: {}, runId: "run", agentId: "agent", adapterType: "paperclip_runner" }, 10, undefined, settled, "late-acquire", observe);
|
||||
await expect(acquire).rejects.toThrow("timed out");
|
||||
await manager.call(manifest.id, "configChanged", { config: { release: true, duplicate: true } });
|
||||
await vi.waitFor(() => expect(observe).toHaveBeenCalledOnce());
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
if (workerExits) await manager.stopWorker(manifest.id);
|
||||
unavailable = false;
|
||||
vi.useFakeTimers();
|
||||
const failed = manager.retireStartup!("late-acquire").catch(error => error.message);
|
||||
await started;
|
||||
expect(observe).toHaveBeenCalledTimes(2);
|
||||
await vi.advanceTimersByTimeAsync(15_000);
|
||||
expect(await failed).toBe("Task cleanup is still pending");
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
vi.useRealTimers();
|
||||
let retired = false;
|
||||
const retiring = manager.retireStartup!("late-acquire").then(() => { retired = true; });
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(retired).toBe(false);
|
||||
release(); await retiring;
|
||||
expect(settled).toHaveBeenCalledOnce();
|
||||
await manager.retireStartup!("late-acquire");
|
||||
expect(observe).toHaveBeenCalledTimes(2);
|
||||
} finally { vi.useRealTimers(); release(); await manager.stopAll(); }
|
||||
});
|
||||
|
||||
it.each(["stop", "replace", "stopAll"])("keeps stopped startup work reachable through %s until verified retirement", async mode => {
|
||||
let release!: () => void, entered!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), started = new Promise<void>(resolve => { entered = resolve; });
|
||||
|
||||
@@ -38,6 +38,7 @@ import { isNewTerminalRunFailure, reportRunFailure } from "../run-failure-report
|
||||
import { isNativeRunnerOwnershipHeld } from "./native-runner-ownership.js";
|
||||
import { nativeRestartInterruptedTurnId, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js";
|
||||
import { retireSupersededNativeRetry } from "./native-superseded-retry.js";
|
||||
import { requestNativeStartupCleanup, readNativeStartupContext, promoteNativeStartupCleanup } from "./native-startup-cleanup.js";
|
||||
|
||||
export type NativeControllerIdentity = {
|
||||
bootId: string;
|
||||
@@ -623,22 +624,41 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
} as const;
|
||||
}
|
||||
|
||||
const startupContext = readNativeStartupContext(row.run, row.coordinator);
|
||||
const startupRetryAt = startupContext ? new Date(Math.max(now.getTime() + 30_000,
|
||||
startupContext.deadlineAtMs, row.coordinator.leaseExpiresAt?.getTime() ?? 0)) : undefined;
|
||||
if (input.liveRunExecutions?.has(row.run.id)) {
|
||||
return {
|
||||
kind: "awaiting_evidence",
|
||||
runId: row.run.id,
|
||||
reason: "current_controller_preparing",
|
||||
...(startupRetryAt ? { retryAt: startupRetryAt } : {}),
|
||||
} as const;
|
||||
}
|
||||
if (startupContext) {
|
||||
if (await promoteNativeStartupCleanup(tx as unknown as Db, row.run, row.coordinator, now)) {
|
||||
return { kind: "blocked", runId: row.run.id, reason: "native_startup_cleanup_pending" } as const;
|
||||
}
|
||||
return { kind: "awaiting_evidence", runId: row.run.id, reason: "native_startup_cleanup_pending", retryAt: startupRetryAt } as const;
|
||||
}
|
||||
|
||||
const profile = row.run.runnerProfileJson ?? {};
|
||||
const storedRemoteWorkspace = remoteWorkspaceRunReferenceSchema.safeParse(profile.remoteWorkspace);
|
||||
const remoteWorkspace = readNativeWorkspaceSyncReference(profile.nativeWorkspaceSync)
|
||||
?? (storedRemoteWorkspace.success ? storedRemoteWorkspace.data : null);
|
||||
const [pluginTask] = await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
|
||||
const [pluginTask] = await tx.select({ id: environmentLeases.id, metadata: environmentLeases.metadata }).from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, row.run.companyId), eq(environmentLeases.heartbeatRunId, row.run.id),
|
||||
sql`${environmentLeases.metadata}->>'runnerTaskLease' = 'true'`,
|
||||
)).orderBy(desc(environmentLeases.createdAt)).limit(1);
|
||||
const startup = pluginTask?.metadata?.runnerTaskStartup as { contextId?: unknown; deadlineAtMs?: unknown; runId?: unknown } | undefined;
|
||||
if (startup?.runId === row.run.id && typeof startup.contextId === "string" &&
|
||||
Number.isSafeInteger(startup.deadlineAtMs) && Number(startup.deadlineAtMs) <= now.getTime() &&
|
||||
!row.run.sessionIdAfter && row.coordinator.leaseOwner && row.coordinator.leaseExpiresAt && row.coordinator.leaseExpiresAt <= now) {
|
||||
await requestNativeStartupCleanup(tx as unknown as Db, { companyId: row.run.companyId, agentId: row.run.agentId,
|
||||
runId: row.run.id, contextId: startup.contextId, deadlineAtMs: Number(startup.deadlineAtMs),
|
||||
leaseOwner: row.coordinator.leaseOwner, requiresPreparationStop: true, leaseId: pluginTask!.id }, { publish: false });
|
||||
return { kind: "blocked", runId: row.run.id, reason: "native_startup_cleanup_pending" } as const;
|
||||
}
|
||||
const remoteSandbox = profile.nativeWorkspaceSync != null || profile.remoteWorkspace != null;
|
||||
const remoteProcess = remoteSandbox || !!pluginTask;
|
||||
const durableAuthority = (remoteWorkspace || pluginTask) && await hasNativeRunnerCheckpoint(tx, {
|
||||
|
||||
@@ -0,0 +1,208 @@
|
||||
import { and, eq, isNull, sql } from "drizzle-orm";
|
||||
import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, sandboxCreationOperations, type Db } from "@paperclipai/db";
|
||||
import { conflict } from "../../errors.js";
|
||||
import { persistActivity, publishActivity } from "../activity-log.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
|
||||
import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
|
||||
import { isNativeRunnerOwnershipHeld } from "./native-runner-ownership.js";
|
||||
|
||||
export const NATIVE_STARTUP_CLEANUP_PENDING = "native_startup_cleanup_pending";
|
||||
|
||||
export class NativeStartupCleanupPendingError extends Error {
|
||||
constructor(readonly contextId: string, readonly deadlineAtMs: number, readonly leaseOwner: string,
|
||||
readonly requiresPreparationStop: boolean, readonly leaseId?: string) {
|
||||
super(NATIVE_STARTUP_CLEANUP_PENDING);
|
||||
this.name = "NativeStartupCleanupPendingError";
|
||||
}
|
||||
}
|
||||
|
||||
type Run = typeof heartbeatRuns.$inferSelect;
|
||||
export interface NativeStartupCleanup {
|
||||
schema: "paperclip.native-startup-cleanup.v1";
|
||||
companyId: string; agentId: string; runId: string;
|
||||
contextId: string; deadlineAtMs: number; leaseOwner: string;
|
||||
controllerGeneration: number; attempt: number; leaseId: string; taskId: string;
|
||||
requiresPreparationStop: boolean; state: "pending" | "complete"; auditId: string;
|
||||
}
|
||||
|
||||
export function readNativeStartupCleanup(run: Pick<Run, "id" | "companyId" | "agentId" | "resultJson">): NativeStartupCleanup | null {
|
||||
const marker = run.resultJson?.nativeStartupCleanup as NativeStartupCleanup | undefined;
|
||||
return marker?.schema === "paperclip.native-startup-cleanup.v1" && marker.companyId === run.companyId &&
|
||||
marker.runId === run.id && marker.agentId === run.agentId && ["contextId", "leaseOwner", "leaseId", "taskId", "auditId"].every(key =>
|
||||
typeof marker[key as keyof NativeStartupCleanup] === "string" && String(marker[key as keyof NativeStartupCleanup]).length > 0) &&
|
||||
Number.isSafeInteger(marker.deadlineAtMs) && Number.isSafeInteger(marker.controllerGeneration) && marker.controllerGeneration > 0 &&
|
||||
Number.isSafeInteger(marker.attempt) && marker.attempt > 0 && typeof marker.requiresPreparationStop === "boolean" &&
|
||||
["pending", "complete"].includes(marker.state) ? marker : null;
|
||||
}
|
||||
|
||||
export function readNativeStartupContext(run: Run,
|
||||
owner: { leaseOwner: string | null; controllerGeneration: number; attempt: number; resultId: string | null }) {
|
||||
const value = run.runnerProfileJson?.nativeStartupContext as { schema?: unknown; companyId?: unknown; agentId?: unknown; runId?: unknown;
|
||||
contextId?: unknown; deadlineAtMs?: unknown; leaseId?: unknown; leaseOwner?: unknown; controllerGeneration?: unknown; attempt?: unknown } | undefined;
|
||||
if (run.runtimeMode !== "native" || run.status !== "running" || run.sessionIdAfter || owner.resultId || isNativeRunnerOwnershipHeld(run) ||
|
||||
!owner.leaseOwner || owner.controllerGeneration < 1 || owner.attempt < 1 || value?.schema !== "paperclip.native-startup-context.v1" ||
|
||||
value.companyId !== run.companyId || value.agentId !== run.agentId || value.runId !== run.id || value.leaseOwner !== owner.leaseOwner ||
|
||||
value.controllerGeneration !== owner.controllerGeneration || value.attempt !== owner.attempt ||
|
||||
typeof value.contextId !== "string" || !value.contextId || typeof value.leaseId !== "string" || !value.leaseId ||
|
||||
!Number.isSafeInteger(value.deadlineAtMs)) return null;
|
||||
return { contextId: value.contextId, leaseId: value.leaseId, deadlineAtMs: Number(value.deadlineAtMs) };
|
||||
}
|
||||
|
||||
export async function clearNativeStartupContext(db: Db, input: {
|
||||
companyId: string; runId: string; contextId: string; leaseOwner: string; controllerGeneration: number;
|
||||
}): Promise<void> {
|
||||
const cleared = await db.update(heartbeatRuns).set({
|
||||
runnerProfileJson: sql`${heartbeatRuns.runnerProfileJson} - 'nativeStartupContext'`,
|
||||
}).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId),
|
||||
eq(heartbeatRuns.status, "running"),
|
||||
sql`coalesce(${heartbeatRuns.resultJson}->'nativeStartupCleanup'->>'state', '') != 'pending'`,
|
||||
sql`${heartbeatRuns.runnerProfileJson}->'nativeStartupContext'->>'contextId' = ${input.contextId}`,
|
||||
sql`${heartbeatRuns.runnerProfileJson}->'nativeStartupContext'->>'leaseOwner' = ${input.leaseOwner}`,
|
||||
sql`${heartbeatRuns.runnerProfileJson}->'nativeStartupContext'->>'controllerGeneration' = ${String(input.controllerGeneration)}`,
|
||||
sql`exists (select 1 from ${nativeRunFinalizations} where ${nativeRunFinalizations.runId} = ${input.runId}
|
||||
and ${nativeRunFinalizations.leaseOwner} = ${input.leaseOwner}
|
||||
and ${nativeRunFinalizations.controllerGeneration} = ${input.controllerGeneration})`,
|
||||
)).returning({ id: heartbeatRuns.id });
|
||||
if (cleared.length !== 1) throw new Error("Native startup fallback lost authority");
|
||||
}
|
||||
|
||||
export async function promoteNativeStartupCleanup(db: Db, run: Run,
|
||||
owner: { leaseOwner: string | null; leaseExpiresAt: Date | null; controllerGeneration: number; attempt: number; resultId: string | null },
|
||||
now: Date): Promise<boolean> {
|
||||
const startup = readNativeStartupContext(run, owner);
|
||||
if (!startup || startup.deadlineAtMs > now.getTime() || !owner.leaseOwner || !owner.leaseExpiresAt || owner.leaseExpiresAt > now || isNativeRunnerOwnershipHeld(run)) return false;
|
||||
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.id, startup.leaseId),
|
||||
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
|
||||
sql`${environmentLeases.metadata}->>'runnerTaskLease' = 'true'`));
|
||||
if (!lease) return false;
|
||||
const prepared = lease.metadata?.runnerTaskStartup as { contextId?: unknown; runId?: unknown } | undefined;
|
||||
return requestNativeStartupCleanup(db, { companyId: run.companyId, agentId: run.agentId, runId: run.id,
|
||||
...startup, leaseOwner: owner.leaseOwner,
|
||||
requiresPreparationStop: prepared?.contextId === startup.contextId && prepared.runId === run.id }, { publish: false });
|
||||
}
|
||||
|
||||
export async function requestNativeStartupCleanup(db: Db, input: {
|
||||
companyId: string; agentId: string; runId: string; contextId: string; deadlineAtMs: number; leaseOwner: string;
|
||||
requiresPreparationStop: boolean;
|
||||
leaseId?: string;
|
||||
}, options: { publish?: boolean } = {}): Promise<boolean> {
|
||||
let publication: Parameters<typeof publishActivity>[0] | null = null;
|
||||
const [binding] = await db.select({ issueId: nativeRunFinalizations.issueId }).from(nativeRunFinalizations).where(and(
|
||||
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId)));
|
||||
if (!binding) return false;
|
||||
const requested = await db.transaction(async tx => {
|
||||
await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, binding.issueId))).for("update");
|
||||
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(
|
||||
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId))).for("update");
|
||||
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId),
|
||||
eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.agentId, input.agentId))).for("update");
|
||||
if (!run || !coordinator || run.runtimeMode !== "native" || run.status !== "running" || run.sessionIdAfter || coordinator.resultId ||
|
||||
isNativeRunnerOwnershipHeld(run)) return false;
|
||||
const existing = readNativeStartupCleanup(run);
|
||||
if (existing) return existing.contextId === input.contextId;
|
||||
if (coordinator.leaseOwner !== input.leaseOwner) return false;
|
||||
const leases = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
||||
eq(environmentLeases.heartbeatRunId, input.runId), input.leaseId ? eq(environmentLeases.id, input.leaseId) : undefined,
|
||||
sql`${environmentLeases.metadata}->>'runnerTaskLease' = 'true'`)).for("update");
|
||||
if (leases.length !== 1 || !leases[0]!.providerLeaseId) throw new Error("Native startup cleanup lease unavailable");
|
||||
const lease = leases[0]!;
|
||||
const activity = await persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system",
|
||||
actorId: "native-startup-cleanup", action: "native.startup_cleanup_requested", entityType: "heartbeat_run", entityId: run.id,
|
||||
agentId: run.agentId, runId: run.id, issueId: run.nativeIssueId ?? undefined,
|
||||
details: { contextId: input.contextId, deadlineAtMs: input.deadlineAtMs, leaseId: lease.id,
|
||||
controllerGeneration: coordinator.controllerGeneration, reason: "startup_worker_unsettled" } });
|
||||
if (!activity.activity?.id) throw new Error("Native startup cleanup audit unavailable");
|
||||
publication = activity.publication;
|
||||
const preparation = lease.metadata?.runnerTaskStartup as { contextId?: unknown; runId?: unknown } | undefined;
|
||||
const marker: NativeStartupCleanup = { schema: "paperclip.native-startup-cleanup.v1", companyId: run.companyId, agentId: run.agentId,
|
||||
runId: run.id, contextId: input.contextId, deadlineAtMs: input.deadlineAtMs, leaseOwner: input.leaseOwner,
|
||||
controllerGeneration: coordinator.controllerGeneration, attempt: coordinator.attempt, leaseId: lease.id,
|
||||
taskId: lease.providerLeaseId!, requiresPreparationStop: input.requiresPreparationStop ||
|
||||
(preparation?.contextId === input.contextId && preparation.runId === run.id), state: "pending", auditId: activity.activity.id };
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization);
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.environmentCleanup);
|
||||
await tx.update(heartbeatRuns).set({ errorCode: NATIVE_STARTUP_CLEANUP_PENDING,
|
||||
error: "Native startup timed out. Cancellation and verified cleanup are pending.",
|
||||
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ nativeStartupCleanup: marker,
|
||||
startupCancellation: { ...run.resultJson?.startupCancellation as object, requestedAt: new Date().toISOString() } })}::jsonb`,
|
||||
updatedAt: new Date() }).where(eq(heartbeatRuns.id, run.id));
|
||||
await tx.update(nativeRunFinalizations).set({ phase: "terminal_failure", failureCode: NATIVE_STARTUP_CLEANUP_PENDING,
|
||||
nextAttemptAt: null, controlDeadlineAt: new Date(), recoveryState: "blocked", updatedAt: new Date() })
|
||||
.where(eq(nativeRunFinalizations.runId, run.id));
|
||||
await tx.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({
|
||||
runnerTaskRetiring: true, runnerTaskStartupCleanup: { contextId: marker.contextId, requiresPreparationStop: marker.requiresPreparationStop },
|
||||
})}::jsonb`, updatedAt: new Date() }).where(eq(environmentLeases.id, lease.id));
|
||||
return true;
|
||||
});
|
||||
if (publication && options.publish !== false) publishActivity(publication);
|
||||
return requested;
|
||||
}
|
||||
|
||||
async function assertStartupCreationsStopped(db: Db, marker: NativeStartupCleanup): Promise<void> {
|
||||
const creations = await db.select({ creation: sandboxCreationOperations, lease: environmentLeases }).from(sandboxCreationOperations)
|
||||
.leftJoin(environmentLeases, eq(environmentLeases.id, sandboxCreationOperations.leaseId)).where(and(
|
||||
eq(sandboxCreationOperations.companyId, marker.companyId), eq(sandboxCreationOperations.runId, marker.runId),
|
||||
sql`${sandboxCreationOperations.metadata}->>'runnerTaskLease' = 'true'`));
|
||||
if (creations.some(({ creation, lease }) => creation.state !== "attached" &&
|
||||
(!["cleanup_pending", "resolved"].includes(creation.state) || !lease || lease.companyId !== marker.companyId || lease.heartbeatRunId !== marker.runId ||
|
||||
lease.providerLeaseId !== creation.metadata.creationProviderLeaseId || lease.cleanupStatus !== "success" || !hasRemoteTerminationReceipt(lease)))) {
|
||||
throw conflict("Task cleanup is still pending", { code: NATIVE_STARTUP_CLEANUP_PENDING });
|
||||
}
|
||||
}
|
||||
|
||||
export async function verifyNativeStartupCleanup(db: Db, run: Run, release: () => Promise<unknown>): Promise<void> {
|
||||
const marker = readNativeStartupCleanup(run);
|
||||
if (!marker || isNativeRunnerOwnershipHeld(run) || !["running", "cancelled"].includes(run.status)) throw conflict("Native startup cleanup intent unavailable");
|
||||
const read = async () => {
|
||||
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.id, marker.leaseId)));
|
||||
if (!lease || lease.heartbeatRunId !== run.id || lease.providerLeaseId !== marker.taskId) throw conflict("Native startup cleanup target changed");
|
||||
return lease;
|
||||
};
|
||||
const [owner] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
if (!owner || owner.controllerGeneration !== marker.controllerGeneration || owner.attempt !== marker.attempt ||
|
||||
owner.leaseOwner !== marker.leaseOwner || owner.resultId) throw conflict("Native startup cleanup ownership changed");
|
||||
let lease = await read();
|
||||
const stopped = (value: typeof lease) => hasRemoteTerminationReceipt(value) && value.cleanupStatus === "success" &&
|
||||
(!marker.requiresPreparationStop || value.metadata?.runnerTaskPreparationStopped === true);
|
||||
if (!stopped(lease)) { await release(); lease = await read(); }
|
||||
if (!stopped(lease)) throw conflict("Task cleanup is still pending", { code: NATIVE_STARTUP_CLEANUP_PENDING });
|
||||
await assertStartupCreationsStopped(db, marker);
|
||||
await db.transaction(async tx => {
|
||||
await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, run.companyId), eq(issues.id, run.nativeIssueId!))).for("update");
|
||||
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)).for("update");
|
||||
const [current] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)).for("update");
|
||||
const observed = current && readNativeStartupCleanup(current);
|
||||
const [receipt] = await tx.select().from(environmentLeases).where(eq(environmentLeases.id, marker.leaseId)).for("update");
|
||||
if (!coordinator || !current || isNativeRunnerOwnershipHeld(current) || coordinator.resultId || coordinator.controllerGeneration !== marker.controllerGeneration ||
|
||||
coordinator.attempt !== marker.attempt || coordinator.leaseOwner !== marker.leaseOwner || observed?.contextId !== marker.contextId ||
|
||||
!receipt || receipt.providerLeaseId !== marker.taskId || receipt.heartbeatRunId !== run.id || !stopped(receipt)) {
|
||||
throw conflict("Native startup cleanup ownership changed");
|
||||
}
|
||||
await assertStartupCreationsStopped(tx as unknown as Db, marker);
|
||||
await tx.update(heartbeatRuns).set({ resultJson: sql`jsonb_set(${heartbeatRuns.resultJson}, '{nativeStartupCleanup,state}', '"complete"'::jsonb)`, updatedAt: new Date() })
|
||||
.where(eq(heartbeatRuns.id, run.id));
|
||||
});
|
||||
}
|
||||
|
||||
export async function acknowledgeNativeStartupCleanup(db: Db, input: { companyId: string; runId: string }): Promise<void> {
|
||||
const [binding] = await db.select({ issueId: nativeRunFinalizations.issueId }).from(nativeRunFinalizations).where(and(
|
||||
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId)));
|
||||
if (!binding) return;
|
||||
await db.transaction(async tx => {
|
||||
await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, binding.issueId))).for("update");
|
||||
const [owner] = await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, input.runId)).for("update");
|
||||
const [run] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)).for("update");
|
||||
const marker = run && readNativeStartupCleanup(run);
|
||||
if (!owner || !run || run.status !== "cancelled" || !marker || marker.state !== "complete" || owner.resultId ||
|
||||
owner.failureCode !== NATIVE_STARTUP_CLEANUP_PENDING || owner.controllerGeneration !== marker.controllerGeneration ||
|
||||
owner.attempt !== marker.attempt || owner.leaseOwner !== marker.leaseOwner) return;
|
||||
const [lease] = await tx.select().from(environmentLeases).where(eq(environmentLeases.id, marker.leaseId)).for("update");
|
||||
if (!lease || lease.heartbeatRunId !== run.id || lease.providerLeaseId !== marker.taskId || !hasRemoteTerminationReceipt(lease) ||
|
||||
(marker.requiresPreparationStop && lease.metadata?.runnerTaskPreparationStopped !== true)) return;
|
||||
await assertStartupCreationsStopped(tx as unknown as Db, marker);
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle);
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.environmentCleanup);
|
||||
await tx.update(nativeRunFinalizations).set({ leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null, recoveryState: null,
|
||||
failureCode: "native_retry_cancelled", updatedAt: new Date() }).where(and(eq(nativeRunFinalizations.runId, input.runId), isNull(nativeRunFinalizations.resultId)));
|
||||
});
|
||||
}
|
||||
@@ -725,6 +725,7 @@ export interface PluginWorkerHandle {
|
||||
executeLogSink?: ExecuteLogSink,
|
||||
onWorkerSettled?: () => void,
|
||||
startupContextId?: string,
|
||||
observeResult?: (result: HostToWorkerMethods[M][1]) => Promise<void>,
|
||||
): Promise<HostToWorkerMethods[M][1]>;
|
||||
|
||||
/**
|
||||
@@ -865,6 +866,7 @@ export interface PluginWorkerManager {
|
||||
executeLogSink?: ExecuteLogSink,
|
||||
onWorkerSettled?: () => void,
|
||||
startupContextId?: string,
|
||||
observeResult?: (result: HostToWorkerMethods[M][1]) => Promise<void>,
|
||||
): Promise<HostToWorkerMethods[M][1]>;
|
||||
|
||||
/**
|
||||
@@ -914,6 +916,10 @@ export function createPluginWorkerHandle(
|
||||
// Unlike RPC promises, these survive timeouts. A late response is the
|
||||
// completion receipt. A worker crash cannot manufacture one.
|
||||
const unsettledCalls = new Map<string | number, () => void>();
|
||||
const observedResults = new Map<string | number, {
|
||||
contextId: string; response?: JsonRpcResponse; running?: Promise<void>;
|
||||
observe: (result: unknown) => Promise<void>; finish: () => void;
|
||||
}>();
|
||||
const startupCalls = new Map<string, Set<() => void>>();
|
||||
const startupRetirements = new Map<string, Promise<void>>();
|
||||
const startupHostHandlers = new Map<string, Set<Promise<void>>>();
|
||||
@@ -1156,24 +1162,50 @@ export function createPluginWorkerHandle(
|
||||
/**
|
||||
* Handle a JSON-RPC response from the worker (matching a pending request).
|
||||
*/
|
||||
function settleResponse(response: JsonRpcResponse): void {
|
||||
const id = response.id!;
|
||||
unsettledCalls.get(id)?.();
|
||||
unsettledCalls.delete(id);
|
||||
const pending = pendingRequests.get(id);
|
||||
if (!pending) { log.warn({ id }, "received response for unknown request id"); return; }
|
||||
clearTimeout(pending.timer);
|
||||
pendingRequests.delete(id);
|
||||
pending.resolve(response);
|
||||
}
|
||||
|
||||
function processObservedResult(id: string | number): Promise<void> {
|
||||
const observer = observedResults.get(id);
|
||||
if (!observer?.response) return Promise.resolve();
|
||||
if (observer.running) return observer.running;
|
||||
const response = observer.response;
|
||||
const processing = (async () => {
|
||||
if (isJsonRpcSuccessResponse(response)) await observer.observe(response.result);
|
||||
observedResults.delete(id);
|
||||
observer.finish();
|
||||
settleResponse(response);
|
||||
})();
|
||||
observer.running = processing;
|
||||
void processing.catch(() => {
|
||||
const pending = pendingRequests.get(id);
|
||||
pending?.resolve(createErrorResponse(id, JSONRPC_ERROR_CODES.INTERNAL_ERROR, "Plugin result could not be recorded"));
|
||||
}).finally(() => { if (observer.running === processing) observer.running = undefined; });
|
||||
return processing;
|
||||
}
|
||||
|
||||
function handleResponse(response: JsonRpcResponse): void {
|
||||
const id = response.id;
|
||||
if (id === null || id === undefined) {
|
||||
log.warn("received response with null/undefined id");
|
||||
return;
|
||||
}
|
||||
|
||||
unsettledCalls.get(id)?.();
|
||||
unsettledCalls.delete(id);
|
||||
const pending = pendingRequests.get(id);
|
||||
if (!pending) {
|
||||
log.warn({ id }, "received response for unknown request id");
|
||||
const observer = observedResults.get(id);
|
||||
if (observer) {
|
||||
if (observer.response) return;
|
||||
observer.response = response;
|
||||
void processObservedResult(id).catch(() => log.warn({ id }, "plugin result recording will retry during cleanup"));
|
||||
return;
|
||||
}
|
||||
|
||||
clearTimeout(pending.timer);
|
||||
pendingRequests.delete(id);
|
||||
pending.resolve(response);
|
||||
settleResponse(response);
|
||||
}
|
||||
|
||||
function readNonEmptyString(value: unknown): string | null {
|
||||
@@ -3087,6 +3119,11 @@ export function createPluginWorkerHandle(
|
||||
// Read-only requests cannot continue in an exited worker. Do not infer
|
||||
// completion for execute/cleanup calls, whose provider work may outlive it.
|
||||
for (const finish of [...unsettledInspections]) finish();
|
||||
for (const [id, observer] of observedResults) {
|
||||
if (observer.response) continue;
|
||||
observedResults.delete(id);
|
||||
observer.finish();
|
||||
}
|
||||
|
||||
// Close the one login pseudo-terminal route with a fixed non-secret exit and
|
||||
// clear the route one time. The pending pseudo-terminal calls
|
||||
@@ -3423,8 +3460,9 @@ export function createPluginWorkerHandle(
|
||||
executeLogSink?: ExecuteLogSink,
|
||||
onWorkerSettled?: () => void,
|
||||
startupContextId?: string,
|
||||
observeResult?: (result: HostToWorkerMethods[M][1]) => Promise<void>,
|
||||
): Promise<HostToWorkerMethods[M][1]> {
|
||||
const dispatched = measureRunPhase("execution.plugin_dispatch", () => callInternalRaw(method, params, timeoutMs, executeLogSink, onWorkerSettled, startupContextId));
|
||||
const dispatched = measureRunPhase("execution.plugin_dispatch", () => callInternalRaw(method, params, timeoutMs, executeLogSink, onWorkerSettled, startupContextId, observeResult));
|
||||
// Tracing wraps the RPC in a new promise. Guard the promise handed to the
|
||||
// caller as well, without swallowing its eventual rejection for that caller.
|
||||
void dispatched.catch(() => undefined);
|
||||
@@ -3438,6 +3476,7 @@ export function createPluginWorkerHandle(
|
||||
executeLogSink?: ExecuteLogSink,
|
||||
onWorkerSettled?: () => void,
|
||||
startupContextId?: string,
|
||||
observeResult?: (result: HostToWorkerMethods[M][1]) => Promise<void>,
|
||||
): Promise<HostToWorkerMethods[M][1]> {
|
||||
const rpcPromise = new Promise<HostToWorkerMethods[M][1]>((resolve, reject) => {
|
||||
if (startupContextId && startupRetirements.has(startupContextId)) {
|
||||
@@ -3484,6 +3523,19 @@ export function createPluginWorkerHandle(
|
||||
}
|
||||
if (method === "environmentInspectLease") unsettledInspections.add(finish);
|
||||
}
|
||||
if (observeResult && startupContextId) {
|
||||
let finish!: () => void;
|
||||
const processing = new Promise<void>(resolve => { finish = resolve; });
|
||||
let handlers = startupHostHandlers.get(startupContextId);
|
||||
if (!handlers) { handlers = new Set(); startupHostHandlers.set(startupContextId, handlers); }
|
||||
handlers.add(processing);
|
||||
observedResults.set(id, { contextId: startupContextId, observe: observeResult as (result: unknown) => Promise<void>, finish: () => {
|
||||
startupHostHandlers.get(startupContextId)?.delete(processing);
|
||||
if (startupHostHandlers.get(startupContextId)?.size === 0) startupHostHandlers.delete(startupContextId);
|
||||
finish();
|
||||
onStartupWorkChanged?.();
|
||||
} });
|
||||
}
|
||||
const timeout = resolveRpcCallTimeoutMs(timeoutMs, rpcTimeoutMs);
|
||||
const invocationScope = deriveInvocationScope(method, params);
|
||||
const invocation = invocationScope ? registerInvocation(invocationScope) : null;
|
||||
@@ -3550,6 +3602,8 @@ export function createPluginWorkerHandle(
|
||||
} catch (err) {
|
||||
clearTimeout(timer);
|
||||
pendingRequests.delete(id);
|
||||
observedResults.get(id)?.finish();
|
||||
observedResults.delete(id);
|
||||
unsettledCalls.get(id)?.();
|
||||
unsettledCalls.delete(id);
|
||||
clearInvocation(invocation);
|
||||
@@ -3637,6 +3691,7 @@ export function createPluginWorkerHandle(
|
||||
hasStartupWork() { return startupCalls.size > 0 || startupHostHandlers.size > 0 || startupRetirements.size > 0; },
|
||||
async drainStartup(contextId: string) {
|
||||
revokeStartupInvocations(contextId);
|
||||
for (const [id, observer] of observedResults) if (observer.contextId === contextId) void processObservedResult(id).catch(() => undefined);
|
||||
await waitForStartupHandlers(contextId);
|
||||
},
|
||||
async retireStartup(contextId: string) {
|
||||
@@ -3644,6 +3699,7 @@ export function createPluginWorkerHandle(
|
||||
if (current) return current;
|
||||
const retiring = (async () => {
|
||||
revokeStartupInvocations(contextId);
|
||||
for (const [id, observer] of observedResults) if (observer.contextId === contextId) void processObservedResult(id).catch(() => undefined);
|
||||
await waitForStartupHandlers(contextId);
|
||||
for (const finish of [...startupCalls.get(contextId) ?? []]) finish();
|
||||
})();
|
||||
@@ -3662,6 +3718,7 @@ export function createPluginWorkerHandle(
|
||||
executeLogSink?: ExecuteLogSink,
|
||||
onWorkerSettled?: () => void,
|
||||
startupContextId?: string,
|
||||
observeResult?: (result: HostToWorkerMethods[M][1]) => Promise<void>,
|
||||
): Promise<HostToWorkerMethods[M][1]> {
|
||||
if (status !== "running" && status !== "starting") {
|
||||
onWorkerSettled?.();
|
||||
@@ -3671,7 +3728,7 @@ export function createPluginWorkerHandle(
|
||||
),
|
||||
);
|
||||
}
|
||||
return callInternal(method, params, timeoutMs, executeLogSink, onWorkerSettled, startupContextId);
|
||||
return callInternal(method, params, timeoutMs, executeLogSink, onWorkerSettled, startupContextId, observeResult);
|
||||
},
|
||||
|
||||
openLoginPtySession(input: LoginPtyOpenInput) {
|
||||
@@ -4032,6 +4089,7 @@ export function createPluginWorkerManager(
|
||||
executeLogSink?: ExecuteLogSink,
|
||||
onWorkerSettled?: () => void,
|
||||
startupContextId?: string,
|
||||
observeResult?: (result: HostToWorkerMethods[M][1]) => Promise<void>,
|
||||
): Promise<HostToWorkerMethods[M][1]> {
|
||||
const handle = workers.get(pluginId);
|
||||
if (!startupContextId && ["environmentReleaseLease", "environmentDestroyLease"].includes(method)) {
|
||||
@@ -4057,7 +4115,7 @@ export function createPluginWorkerManager(
|
||||
// A caller timeout is not worker completion. Inspection capacity is
|
||||
// released only by its actual reply, send failure, or worker exit.
|
||||
const finish = onWorkerSettled ? () => { finishInspection?.(); onWorkerSettled(); } : finishInspection;
|
||||
const result = handle.call(method, params, timeoutMs, executeLogSink, finish, startupContextId);
|
||||
const result = handle.call(method, params, timeoutMs, executeLogSink, finish, startupContextId, observeResult);
|
||||
if (method === "configChanged") void result.then(() => {
|
||||
for (const listener of configApplied) {
|
||||
try { listener(); }
|
||||
|
||||
@@ -4,9 +4,9 @@ import { and, eq, lte, or, sql } from "drizzle-orm";
|
||||
import { companies, environments, sandboxCreationOperations, type Db } from "@paperclipai/db";
|
||||
import { readEnvironmentCreationCleanupError, type PluginEnvironmentLease, type PluginEnvironmentLeaseObservation } from "@paperclipai/plugin-sdk";
|
||||
import { assertEnvironmentCompanyBinding, environmentService } from "./environments.js";
|
||||
import { ENVIRONMENT_CLEANUP_RETRY_MS } from "./environment-cleanup-state.js";
|
||||
import { ENVIRONMENT_CLEANUP_RETRY_MS, recoverableSandboxCreationCondition } from "./environment-cleanup-state.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { notifyEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
|
||||
export type SandboxCreationOperation = typeof sandboxCreationOperations.$inferSelect;
|
||||
|
||||
@@ -78,26 +78,69 @@ export async function createOwnedSandbox(
|
||||
* This enriches the existing owner; it never transfers or reopens ownership. */
|
||||
export async function recordSandboxCreationResource(db: Db, input: {
|
||||
id: string; companyId: string; environmentId: string; providerLeaseId: string;
|
||||
providerMetadata?: Record<string, unknown>;
|
||||
}): Promise<void> {
|
||||
const [recorded] = await db.update(sandboxCreationOperations).set({
|
||||
metadata: sql`coalesce(${sandboxCreationOperations.metadata}, '{}'::jsonb) || ${JSON.stringify({ creationProviderLeaseId: input.providerLeaseId })}::jsonb`,
|
||||
updatedAt: new Date(),
|
||||
}).where(and(eq(sandboxCreationOperations.id, input.id), eq(sandboxCreationOperations.companyId, input.companyId),
|
||||
eq(sandboxCreationOperations.originalEnvironmentId, input.environmentId),
|
||||
or(sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' is null`,
|
||||
sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' = ${input.providerLeaseId}`),
|
||||
)).returning({ id: sandboxCreationOperations.id });
|
||||
if (!recorded) throw new Error("Sandbox creation resource identity changed");
|
||||
await writeEnvironmentCleanup(db, async tx => {
|
||||
const [recorded] = await tx.update(sandboxCreationOperations).set({
|
||||
nextAttemptAt: sql`case when ${sandboxCreationOperations.state} = 'needs_reconciliation' then clock_timestamp() else ${sandboxCreationOperations.nextAttemptAt} end`,
|
||||
metadata: sql`coalesce(${sandboxCreationOperations.metadata}, '{}'::jsonb) || ${JSON.stringify({ creationProviderLeaseId: input.providerLeaseId, ...(input.providerMetadata ? { creationProviderMetadata: input.providerMetadata } : {}) })}::jsonb`,
|
||||
updatedAt: new Date(),
|
||||
}).where(and(eq(sandboxCreationOperations.id, input.id), eq(sandboxCreationOperations.companyId, input.companyId),
|
||||
eq(sandboxCreationOperations.originalEnvironmentId, input.environmentId),
|
||||
or(sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' is null`,
|
||||
sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' = ${input.providerLeaseId}`),
|
||||
)).returning({ id: sandboxCreationOperations.id });
|
||||
if (!recorded) throw new Error("Sandbox creation resource identity changed");
|
||||
}, { companyId: input.companyId });
|
||||
}
|
||||
|
||||
export async function queueSandboxCreationCleanup(db: Db, input: {
|
||||
id: string; companyId: string; environmentId: string; providerLeaseId: string;
|
||||
providerMetadata?: Record<string, unknown>;
|
||||
}): Promise<void> {
|
||||
await db.transaction(async tx => {
|
||||
await tx.select({ id: environments.id }).from(environments).where(eq(environments.id, input.environmentId)).for("update");
|
||||
const [operation] = await tx.select().from(sandboxCreationOperations).where(and(eq(sandboxCreationOperations.id, input.id),
|
||||
eq(sandboxCreationOperations.companyId, input.companyId), eq(sandboxCreationOperations.originalEnvironmentId, input.environmentId))).for("update");
|
||||
if (!operation || (operation.metadata.creationProviderLeaseId && operation.metadata.creationProviderLeaseId !== input.providerLeaseId)) {
|
||||
throw new Error("Sandbox creation cleanup ownership changed");
|
||||
}
|
||||
if (["attached", "cleanup_pending", "resolved"].includes(operation.state)) return;
|
||||
const contextId = operation.metadata.runnerTaskLease === true && typeof operation.metadata.runnerTaskCreationContextId === "string"
|
||||
? operation.metadata.runnerTaskCreationContextId : null;
|
||||
if (operation.state !== "pending" && !(operation.state === "needs_reconciliation" && contextId &&
|
||||
operation.metadata.creationProviderLeaseId === input.providerLeaseId)) throw new Error("Sandbox creation cleanup ownership changed");
|
||||
await notifyEnvironmentCleanup(tx, { companyId: operation.companyId, runId: operation.runId });
|
||||
const lease = await environmentService(tx as unknown as Db).insertPendingCleanupLease({
|
||||
companyId: operation.companyId, environmentId: operation.originalEnvironmentId,
|
||||
heartbeatRunId: operation.runId, issueId: operation.issueId, executionWorkspaceId: operation.executionWorkspaceId,
|
||||
provider: operation.provider, providerLeaseId: input.providerLeaseId,
|
||||
metadata: { ...operation.metadata, creationOperationId: operation.id, environmentId: operation.originalEnvironmentId,
|
||||
providerMetadata: input.providerMetadata ?? operation.metadata.creationProviderMetadata ?? {}, runnerTaskRetiring: true,
|
||||
...(contextId ? { runnerTaskStartupCleanup: { contextId, requiresPreparationStop: false } } : {}) },
|
||||
failureReason: "interrupted_task_creation",
|
||||
});
|
||||
await tx.update(sandboxCreationOperations).set({ state: "cleanup_pending", leaseId: lease.id, updatedAt: new Date(),
|
||||
metadata: { ...operation.metadata, creationProviderLeaseId: input.providerLeaseId,
|
||||
creationProviderMetadata: input.providerMetadata ?? operation.metadata.creationProviderMetadata ?? {} },
|
||||
}).where(eq(sandboxCreationOperations.id, operation.id));
|
||||
});
|
||||
}
|
||||
|
||||
export async function recoverSandboxCreations(db: Db,
|
||||
inspect: (operation: SandboxCreationOperation) => Promise<PluginEnvironmentLeaseObservation | null>,
|
||||
): Promise<void> {
|
||||
const due = await db.select().from(sandboxCreationOperations).where(and(
|
||||
eq(sandboxCreationOperations.state, "pending"), lte(sandboxCreationOperations.nextAttemptAt, new Date()),
|
||||
recoverableSandboxCreationCondition(), lte(sandboxCreationOperations.nextAttemptAt, new Date()),
|
||||
)).limit(100);
|
||||
for (const candidate of due) {
|
||||
try {
|
||||
if (candidate.metadata.runnerTaskLease === true && typeof candidate.metadata.runnerTaskCreationContextId === "string" &&
|
||||
typeof candidate.metadata.creationProviderLeaseId === "string") {
|
||||
await queueSandboxCreationCleanup(db, { id: candidate.id, companyId: candidate.companyId,
|
||||
environmentId: candidate.originalEnvironmentId, providerLeaseId: candidate.metadata.creationProviderLeaseId });
|
||||
continue;
|
||||
}
|
||||
const [existingCompany] = await db.select({ id: companies.id }).from(companies).where(eq(companies.id, candidate.companyId)).limit(1);
|
||||
const observation = existingCompany ? await inspect(candidate) : null;
|
||||
const cleanup = observation?.creationCleanup ? readEnvironmentCreationCleanupError({ data: {
|
||||
@@ -140,7 +183,7 @@ export async function recoverSandboxCreations(db: Db,
|
||||
? { nextAttemptAt: new Date(Date.now() + ENVIRONMENT_CLEANUP_RETRY_MS), updatedAt: new Date() }
|
||||
: { state: "needs_reconciliation", updatedAt: new Date(),
|
||||
metadata: sql`coalesce(${sandboxCreationOperations.metadata}, '{}'::jsonb) || '{"creationReconciliationReason":"company_deleted"}'::jsonb` })
|
||||
.where(and(eq(sandboxCreationOperations.id, candidate.id), eq(sandboxCreationOperations.state, "pending")));
|
||||
.where(and(eq(sandboxCreationOperations.id, candidate.id), recoverableSandboxCreationCondition()));
|
||||
return Boolean(company);
|
||||
});
|
||||
logger.warn({ errorKind: retry ? "sandbox_creation_recovery_failed" : "sandbox_creation_needs_reconciliation", operationId: candidate.id, companyId: candidate.companyId },
|
||||
|
||||
Reference in new issue
Block a user