mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 11:01:16 +01:00
fix(runner): preserve cleanup ownership across startup failures (#18658)
## Thinking Path > - Paperclip controls which run owns an agent task. > - A plugin can continue accepted work after Core stops waiting. > - A late task-creation reply can therefore arrive after startup cancellation. > - Core must keep that work owned until its resource identity and cleanup outcome are known. > - This PR supplies the cleanup and recovery support before shared startup is enabled. > - The dependent activation PR can then reuse transport data within a bounded startup context. ## Linked Issues or Issue Description Refs #18405. This is the independently mergeable prerequisite for that stacked activation PR. **What existing behavior does this improve?** Cleanup of explicitly recorded startup work that outlives its caller. **Current behavior** Caller timeout, worker exit, or failed cleanup-intent persistence can leave work without a complete recovery path. **Proposed behavior** Retain the cleanup obligation, use existing Stop and recovery queues, and release ownership only after the exact task and accepted work have stopped. Ordinary calls keep their existing startup payloads and budgets. This PR does not create startup contexts or advertise transport reuse. **Reason and benefit** Separate the failure-handling contract from activation so each can be reviewed on its own. This support PR has **no standalone TTFT speedup**. ## What Changed - Add the startup cleanup marker, exact ownership checks, verified completion, and acknowledgement helpers. - Connect recorded cleanup obligations to existing Stop, deadline reconciliation, restart recovery, and retry scheduling. - Retain typed cleanup responsibility when a database write fails. Keep the nonterminal run from being treated as ordinary completed cleanup. - Observe late acquisition results and save resource identities before retiring them. Retain real host writes and received results across worker exit or persistence failure. - Queue abandoned replacement tasks through the existing creation journal and cleanup machinery. Require the replacement's own Stop proof. - Accept an explicit startup context in the internal bridge for lifetime tracking. Automatic context creation, marker production, capability advertisement, transport capture, and executor activation belong to #18405. - Add PostgreSQL and worker IPC regressions, including legacy payload assertions. ## Verification Initial split source `8fb482d8be9e70101b0cfdd0830f41cf5b8e3cf4` had tree `95508bd1859e991670cad8afd4de8d2167009ac5`. **190 distinct targeted tests pass**, along with server typechecking. The selected checks cover real PostgreSQL cleanup/ownership, worker IPC, the bridge and creation journal, ordinary/distributed Stop, restart and deadline recovery, idle-sleep behavior, and the unchanged legacy warm-task suite. Independent recovery and worker reviews found no blocking issue. The first correction, `b6342d42042601b376da38dc8e1dba9ca80e9539`, fixes the replacement cleanup finding with the existing startup-cleanup marker, scoped to contextual abandoned task creations. The replacement was never prepared, so preparation-stop proof is not required. No dispatch is fabricated and no normal-startup query is added. The actual queue/sweep/driver/receipt/cancellation regression fails before the fix and passes after it: void, running or foreign-task status retains ownership; exact-task stopped status permits completion even when Release returns void. **64 affected targeted cases and server typechecking pass**. Independent review found no blocker. Current source is `f53af72af76f263096c3c2e788506e6faef9474f`. It fixes both recovery findings from the prior head's Apex 3/5: orphan recovery now preserves pending or completed-but-not-cancelled cleanup through its existing final-write fence, and exact late identities can reach verified cleanup after reconciliation or restart. The regression selection went from four failing assertions to six passing cases. Affected selections cover 217 distinct cases; the exact final six-case regression and server typecheck pass. Independent reviewers found no blocker in either fix. Current-head standard review is 5/5 and all CI checks pass (54 successes, three expected skips, plus Snyk). Apex also completed at **5/5** on this exact head, with zero unresolved threads. The requested independent availability audit is complete on this head. It found an activation-dependent operability gap: an acquisition response that is never received can leave an unknown creation identity with no supported task inspection/reconciliation path, while cleanup keeps the run owner reserved. This does not occur in ordinary lower-only calls. The contract and recovery policy need follow-up before activation; the external review scores do not resolve that availability tradeoff. The split preserves the accepted implementation at `21db7b3851` after its clean merge with frozen master `946735d03a55c1fb4cc5434d6bf1d87742daa9df`. The initial dependent activation PR restored exact aggregate tree `f68c6895f87c7b93ae03e6324d955d2659a99dce`. Both subsequent cleanup corrections are integrated into upper head `27978f294a02da3992585c7f4daa73cd8fcf99f8`; its current aggregate tree is `a0b66c33f5e22745f034f45d24d213f5c210cf16`. Lower-only payload tests cover the absence of capability advertisement and transport projection. Only targeted local tests and typechecking are used. Full regression coverage runs in GitHub CI. No deployment or ROM qualification is part of this change. The lower audit covered connected worker, creation-journal, Stop, restart, deadline and orphan-recovery paths. A new isolated real-PostgreSQL characterization reproduced the unknown-ID hold through actual recovery, cleanup verification and run-control cancellation. It models a lost acquisition response; it is not a live provider or complete IPC test. No source edits were made by the audit. The upper/composition audit remains required after its review cycle. ## Risks - Recording a replacement result now commits with the existing cleanup notification. This conditional replacement path adds transaction/notification responses; common retained startup is unchanged. - This adds cleanup and recovery state handling. A missing receipt or permanently inaccessible old backend can keep ownership held indefinitely. Temporary outages retry; an ordinary retry is not evidence that a deleted or replaced backend stopped. - A caller timeout does not undo work already accepted by a plugin. Unknown resource creation remains unresolved until its identity/outcome is established. Current task creation has no supported inspect-by-operation or operator reconciliation API for a permanently lost identity; repeated Stop alone cannot resolve it. - The support code understands records produced by the dependent activation PR. It does not enable the shared startup policy by itself. - Existing legacy callers must retain their behavior; context-only automatic replacement cleanup remains explicitly scoped. ## Model Used OpenAI Codex, GPT-6 family. The exact model ID and context window are not exposed in this session. Tool-assisted implementation, targeted testing, and independent subagent review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, with available version and capability details - [x] I have checked ROADMAP.md and confirmed this is the requested split of existing work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue with the required fields - [x] I have not referenced internal issues or private links - [x] My branch name describes the change and contains no internal ticket ID - [x] Targeted local tests pass on this source - [x] I have added or updated relevant tests - [x] I have updated relevant documentation - [x] I have documented the risks - [x] All current-head GitHub CI gates are green - [x] Current-head standard Greptile and Apex are 5/5 with no unresolved findings - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed
1 parent
b4315943e8
commit
a9b18e1cbe
24 files changed
+1829
-106
No files matched your search
@@ -1,9 +1,19 @@
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { and, eq, or, sql } from "drizzle-orm";
|
||||
import { environmentLeases, sandboxCreationOperations, type Db } from "@paperclipai/db";
|
||||
import { hasDeferredEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
|
||||
export const ENVIRONMENT_CLEANUP_RETRY_MS = 5 * 60_000;
|
||||
|
||||
export function recoverableSandboxCreationCondition() {
|
||||
return or(eq(sandboxCreationOperations.state, "pending"), and(
|
||||
eq(sandboxCreationOperations.state, "needs_reconciliation"),
|
||||
sql`coalesce(${sandboxCreationOperations.metadata}->>'creationReconciliationReason', '') <> 'company_deleted'`,
|
||||
sql`${sandboxCreationOperations.metadata}->>'runnerTaskLease' = 'true'`,
|
||||
sql`jsonb_typeof(${sandboxCreationOperations.metadata}->'runnerTaskCreationContextId') = 'string' and ${sandboxCreationOperations.metadata}->>'runnerTaskCreationContextId' <> ''`,
|
||||
sql`jsonb_typeof(${sandboxCreationOperations.metadata}->'creationProviderLeaseId') = 'string' and ${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' <> ''`,
|
||||
));
|
||||
}
|
||||
|
||||
export function pendingCleanupDeadlineSql(now = Date.now()) {
|
||||
const deadline = sql`case
|
||||
when ${environmentLeases.metadata}->>'pendingCleanupInFlight' = 'true'
|
||||
@@ -21,7 +31,7 @@ export async function nextEnvironmentCleanupAt(db: Db, backoffMs = ENVIRONMENT_C
|
||||
extract(epoch from ${environmentLeases.updatedAt}) * 1000 + ${backoffMs},
|
||||
${pendingCleanupDeadlineSql()}))` }).from(environmentLeases).where(eq(environmentLeases.status, "pending_cleanup"));
|
||||
const [creation] = await db.select({ next: sql<string | null>`min(extract(epoch from ${sandboxCreationOperations.nextAttemptAt}) * 1000)` })
|
||||
.from(sandboxCreationOperations).where(eq(sandboxCreationOperations.state, "pending"));
|
||||
.from(sandboxCreationOperations).where(recoverableSandboxCreationCondition());
|
||||
const deadlines = [row?.next, creation?.next].filter((value): value is string => value != null).map(Number);
|
||||
const saved = deadlines.length ? Math.min(...deadlines) : null;
|
||||
const buffered = hasDeferredEnvironmentCleanup(db) ? Date.now() + backoffMs : null;
|
||||
|
||||
@@ -3382,7 +3382,7 @@ function createPluginEnvironmentDriver(
|
||||
async releaseRunLease(input) {
|
||||
await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"pluginLeaseCleanupMethod":"release"}'::jsonb` }).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.lease.companyId)));
|
||||
const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
|
||||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true);
|
||||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true && !input.lease.metadata?.runnerTaskStartup && !input.lease.metadata?.runnerTaskStartupCleanup);
|
||||
const { plugin, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
|
||||
await workerManager.call(plugin.id, "environmentReleaseLease", {
|
||||
driverKey,
|
||||
@@ -3425,7 +3425,7 @@ function createPluginEnvironmentDriver(
|
||||
async destroyRunLease(input) {
|
||||
await db.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"pluginLeaseCleanupMethod":"destroy"}'::jsonb` }).where(and(eq(environmentLeases.id, input.lease.id), eq(environmentLeases.companyId, input.lease.companyId)));
|
||||
const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
|
||||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true);
|
||||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true && !input.lease.metadata?.runnerTaskStartup && !input.lease.metadata?.runnerTaskStartupCleanup);
|
||||
const { pluginKey, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
|
||||
await destroyPluginEnvironmentLease({
|
||||
db,
|
||||
@@ -3462,7 +3462,7 @@ function createPluginEnvironmentDriver(
|
||||
|
||||
async retryPendingSandboxTeardown(input) {
|
||||
const taskStopped = await settleDispatchedEnvironmentTask(db, workerManager, input.lease) ||
|
||||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true);
|
||||
(input.lease.metadata?.runnerTaskLease === true && input.lease.metadata?.runnerTaskDispatched !== true && !input.lease.metadata?.runnerTaskStartup && !input.lease.metadata?.runnerTaskStartupCleanup);
|
||||
const { plugin, driverKey, driverConfig } = await resolvePluginDriverForRelease(input);
|
||||
const environmentId = input.environment?.id ?? readString(input.lease.metadata?.environmentId);
|
||||
if (!environmentId) throw new Error("Plugin lease cleanup has no saved environment identity");
|
||||
@@ -3778,7 +3778,7 @@ export function environmentRuntimeService(
|
||||
},
|
||||
|
||||
/** Dispatch plugin-provided Runner execution, including resource preparation and task lifecycle. */
|
||||
async task(input: { companyId: string; leaseId: string; operation: PluginEnvironmentTaskOperation }) {
|
||||
async task(input: { companyId: string; leaseId: string; operation: PluginEnvironmentTaskOperation; executionContext?: unknown }) {
|
||||
if (!options.pluginWorkerManager) throw new Error("Plugin-provided Runner execution worker manager unavailable");
|
||||
return executeEnvironmentTask(db, options.pluginWorkerManager, input);
|
||||
},
|
||||
|
||||
@@ -1,15 +1,17 @@
|
||||
import { attachSandboxCreation, environmentService } from "./environments.js";
|
||||
import { createOwnedSandbox, recordSandboxCreationResource } from "./sandbox-creation-operations.js";
|
||||
import { createOwnedSandbox, queueSandboxCreationCleanup, recordSandboxCreationResource } from "./sandbox-creation-operations.js";
|
||||
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { releaseProviderRecoveryTransfers, stageProviderFreshWorkspace } from "./provider-fresh-workspace.js";
|
||||
import { readWorkspaceProviderBinding } from "@paperclipai/shared";
|
||||
import { settlePluginTaskCredentialHandoff } from "./plugin-task-credential-writeback.js";
|
||||
import { and, eq, inArray, or, sql } from "drizzle-orm";
|
||||
import { environmentLeases, executionWorkspaces, environments, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db";
|
||||
import { freshWorkspaceSchema, environmentTaskOperationSchema, parseEnvironmentTaskResult, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
|
||||
import { freshWorkspaceSchema, environmentTaskOperationSchema, parseEnvironmentTaskResult,
|
||||
type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
|
||||
import { pluginRegistryService } from "./plugin-registry.js";
|
||||
import { hasRemoteTerminationReceipt, remoteExecutionHasStopped, remoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
|
||||
import { NativeExecutionContext } from "./native-runtime/native-execution-context.js";
|
||||
|
||||
export async function pluginEnvironmentSupportsTasks(db: Db, workers: PluginWorkerManager, lease: { companyId: string; metadata: Record<string, unknown> | null }) {
|
||||
const metadata = lease.metadata ?? {};
|
||||
@@ -31,6 +33,7 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
companyId: string;
|
||||
leaseId: string;
|
||||
operation: PluginEnvironmentTaskOperation;
|
||||
executionContext?: unknown;
|
||||
}) {
|
||||
const operation = environmentTaskOperationSchema.parse(input.operation);
|
||||
const [row] = await db.select({ lease: environmentLeases, environment: environments, run: heartbeatRuns })
|
||||
@@ -40,6 +43,13 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
.where(and(eq(environmentLeases.id, input.leaseId), eq(environmentLeases.companyId, input.companyId))).limit(1);
|
||||
if (!row || !row.lease.providerLeaseId) throw new Error("Plugin-provided Runner execution lease unavailable");
|
||||
const { lease, environment, run } = row;
|
||||
const context = input.executionContext;
|
||||
if (context !== undefined) {
|
||||
if (!(context instanceof NativeExecutionContext) || !run) {
|
||||
throw new Error("native_execution_context_unavailable");
|
||||
}
|
||||
context.assertScope({ companyId: input.companyId, agentId: run.agentId, runId: run.id });
|
||||
}
|
||||
const taskId = row.lease.providerLeaseId;
|
||||
const metadata = lease.metadata ?? {};
|
||||
const probe = metadata.runnerTaskProbe as { runId?: unknown; sessionId?: unknown; expiresAtMs?: unknown } | undefined;
|
||||
@@ -128,6 +138,19 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
!workers.getWorker(plugin.id)?.supportedMethods.includes("environmentTask")) {
|
||||
throw new Error("Plugin-provided Runner execution provider unavailable");
|
||||
}
|
||||
const callWorker = async <M extends HostToWorkerMethodName>(method: M, params: HostToWorkerMethods[M][0], timeout?: number): Promise<HostToWorkerMethods[M][1]> => {
|
||||
if (!context) {
|
||||
const cleanup = metadata.runnerTaskStartupCleanup as { contextId?: unknown; requiresPreparationStop?: unknown } | undefined;
|
||||
if (["stop", "status", "complete", "credentials"].includes(operation.kind) && typeof cleanup?.contextId === "string" && typeof cleanup.requiresPreparationStop === "boolean") {
|
||||
return workers.call(plugin.id, method, params, timeout, undefined, undefined, cleanup.contextId);
|
||||
}
|
||||
return timeout === undefined ? workers.call(plugin.id, method, params) : workers.call(plugin.id, method, params, timeout);
|
||||
}
|
||||
const settled = context.trackWorker();
|
||||
const result = await workers.call(plugin.id, method, params, Math.max(1, Math.min(timeout ?? 15_000, context.deadlineAtMs - Date.now())), undefined, settled, context.id);
|
||||
context.assertOpen();
|
||||
return result;
|
||||
};
|
||||
const issue = scopesProjects && lease.issueId ? await db.select({ id: issues.id }).from(issues)
|
||||
.where(and(eq(issues.id, lease.issueId), eq(issues.companyId, input.companyId))).limit(1).then(rows => rows[0]) : null;
|
||||
if (scopesProjects && lease.issueId && !issue) throw new Error("Plugin-provided Runner execution issue unavailable");
|
||||
@@ -219,7 +242,7 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
}
|
||||
try {
|
||||
if (operation.kind === "describe" && metadata.runnerTaskReused === true) {
|
||||
const state = parseEnvironmentTaskResult({kind:"status"}, taskId, await workers.call(plugin.id, "environmentTask", {
|
||||
const state = parseEnvironmentTaskResult({kind:"status"}, taskId, await callWorker("environmentTask", {
|
||||
driverKey: metadata.driverKey, companyId: input.companyId, environmentId: environment?.id ?? null,
|
||||
issueId: lease.issueId, config: driverConfig, taskId, runId: run?.id ?? null, agentId: run?.agentId ?? null,
|
||||
projectIds, lease: {providerLeaseId: taskId, metadata}, operation:{kind:"status"},
|
||||
@@ -233,37 +256,74 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
runnerTaskSessionRetained:state.sessionStateAvailable === true})}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, metadata.runnerTaskPreviousLeaseId), eq(environmentLeases.companyId, input.companyId)));
|
||||
const replacementDriverKey = metadata.driverKey;
|
||||
let abandoned = false;
|
||||
const fresh = await createOwnedSandbox(db, {
|
||||
companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId,
|
||||
issueId: lease.issueId, executionWorkspaceId: lease.executionWorkspaceId, provider: lease.provider!,
|
||||
pluginId: plugin.id, driverKey: metadata.driverKey, config: {}, requestKey: taskId,
|
||||
metadata: { driver: "plugin", pluginId: plugin.id, pluginKey: plugin.pluginKey, driverKey: metadata.driverKey, runnerTaskLease: true },
|
||||
}, identity => workers.call(plugin.id, "environmentAcquireLease", {driverKey:replacementDriverKey,
|
||||
companyId:input.companyId,environmentId:environment!.id,issueId:lease.issueId,config:driverConfig,
|
||||
runId:identity.runId,creationOperationId:identity.id,agentId:run!.agentId,adapterType:"paperclip_runner",
|
||||
...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? {requestedExpiresAt:metadata.runnerTaskRequestedExpiresAt} : {})}));
|
||||
metadata: { driver: "plugin", pluginId: plugin.id, pluginKey: plugin.pluginKey, driverKey: metadata.driverKey,
|
||||
runnerTaskLease: true, agentId: run!.agentId,
|
||||
...(context ? { runnerTaskCreationContextId: context.id } : {}) },
|
||||
}, async identity => {
|
||||
const params = { driverKey: replacementDriverKey, companyId: input.companyId, environmentId: environment!.id,
|
||||
issueId: lease.issueId, config: driverConfig, runId: identity.runId, creationOperationId: identity.id,
|
||||
agentId: run!.agentId, adapterType: "paperclip_runner",
|
||||
...(typeof metadata.runnerTaskRequestedExpiresAt === "string" ? { requestedExpiresAt: metadata.runnerTaskRequestedExpiresAt } : {}) };
|
||||
if (!context) return workers.call(plugin.id, "environmentAcquireLease", params, 30_000);
|
||||
const settled = context.trackWorker();
|
||||
let observedResource: Parameters<typeof queueSandboxCreationCleanup>[1] | undefined;
|
||||
let dispatched = false;
|
||||
try {
|
||||
return await context.wait(() => {
|
||||
dispatched = true;
|
||||
return workers.call(plugin.id, "environmentAcquireLease", params, 30_000, undefined, settled, context.id,
|
||||
async result => {
|
||||
if (!result.providerLeaseId || result.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable");
|
||||
const resource = { id: identity.id, companyId: input.companyId, environmentId: environment!.id,
|
||||
providerLeaseId: result.providerLeaseId, providerMetadata: result.metadata };
|
||||
await recordSandboxCreationResource(db, resource);
|
||||
observedResource = resource;
|
||||
if (abandoned || !context.active || context.signal.aborted || Date.now() >= context.deadlineAtMs) {
|
||||
await queueSandboxCreationCleanup(db, resource);
|
||||
}
|
||||
});
|
||||
});
|
||||
} catch (error) {
|
||||
if (!dispatched) settled();
|
||||
abandoned = true;
|
||||
if (observedResource) await queueSandboxCreationCleanup(db, observedResource);
|
||||
throw error;
|
||||
}
|
||||
});
|
||||
if (!fresh.providerLeaseId || fresh.providerLeaseId === taskId) throw new Error("Plugin replacement task identity unavailable");
|
||||
await recordSandboxCreationResource(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId,
|
||||
environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId });
|
||||
const {runnerTaskIdentity:_identity,runnerTaskPriorIdentity:_prior,runnerTaskCredentialHandoff:_handoff,
|
||||
runnerTaskResumeFrom:_resume,runnerTaskPreviousLeaseId:_previous,...rest}=metadata;
|
||||
const replacementMetadata={...rest,providerMetadata:fresh.metadata ?? {},runnerTaskReused:false,runnerTaskDispatched:false,
|
||||
runnerTaskExecutionStopped:false,runnerTaskSessionRetained:false};
|
||||
{
|
||||
try {
|
||||
context?.assertOpen();
|
||||
if (!context) await recordSandboxCreationResource(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId,
|
||||
environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId });
|
||||
const expiresAt=fresh.expiresAt ? new Date(fresh.expiresAt) : null;
|
||||
const deadline=typeof metadata.runnerTaskRequestedExpiresAt === "string" ? Date.parse(metadata.runnerTaskRequestedExpiresAt) : null;
|
||||
if ((expiresAt && (!Number.isFinite(expiresAt.getTime()) || expiresAt.getTime() <= Date.now())) ||
|
||||
(deadline !== null && (!Number.isFinite(deadline) || !expiresAt || expiresAt.getTime() > deadline)))
|
||||
throw new Error("Plugin replacement task expiry is not bounded");
|
||||
await attachSandboxCreation(db, String(fresh.metadata!.creationOperationId), { companyId: input.companyId, environmentId: environment!.id, heartbeatRunId: lease.heartbeatRunId }, async tx => {
|
||||
context?.assertOpen();
|
||||
const [updated] = await tx.update(environmentLeases).set({providerLeaseId:fresh.providerLeaseId,expiresAt,
|
||||
metadata:{...replacementMetadata,creationOperationId:fresh.metadata!.creationOperationId},
|
||||
}).where(and(eq(environmentLeases.id,lease.id),eq(environmentLeases.providerLeaseId,taskId),
|
||||
eq(environmentLeases.status,"active"))).returning({id:environmentLeases.id});
|
||||
eq(environmentLeases.status,"active"), sql`coalesce(${environmentLeases.metadata}->>'runnerTaskRetiring', 'false') != 'true'`)).returning({id:environmentLeases.id});
|
||||
if (!updated) throw new Error("Plugin task replacement ownership changed");
|
||||
return (await environmentService(tx).getLeaseById(updated.id))!;
|
||||
const attached = (await environmentService(tx).getLeaseById(updated.id))!;
|
||||
context?.assertOpen();
|
||||
return attached;
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
if (context) await queueSandboxCreationCleanup(db, { id: String(fresh.metadata!.creationOperationId), companyId: input.companyId,
|
||||
environmentId: environment!.id, providerLeaseId: fresh.providerLeaseId, providerMetadata: fresh.metadata });
|
||||
throw error;
|
||||
}
|
||||
return executeEnvironmentTask(db, workers, input);
|
||||
}
|
||||
@@ -275,14 +335,17 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
return snapshot.credential;
|
||||
});
|
||||
}
|
||||
const result = await workers.call(plugin.id, "environmentTask", {
|
||||
const params = {
|
||||
driverKey: metadata.driverKey, companyId: input.companyId, environmentId: environment?.id ?? null,
|
||||
issueId: lease.issueId, config: driverConfig,
|
||||
taskId, runId: run?.id ?? (activeProbe ? String(probe.runId) : null), agentId: run?.agentId ?? null, projectIds,
|
||||
lease: { providerLeaseId: lease.providerLeaseId, metadata: lease.metadata ?? undefined },
|
||||
...(workspaceBinding ? { workspace: workspaceBinding } : {}),
|
||||
...(freshWorkspace && scopesProjects ? { fresh: freshWorkspace } : {}), operation,
|
||||
}, operation.kind === "copilot_metadata" ? Math.max(1, Math.min(40_000, Number(probe?.expiresAtMs) - Date.now())) : 15_000);
|
||||
};
|
||||
const timeout = operation.kind === "copilot_metadata" ? Math.max(1, Math.min(40_000, Number(probe?.expiresAtMs) - Date.now()))
|
||||
: 15_000;
|
||||
const result = await callWorker("environmentTask", params, timeout);
|
||||
const parsed = parseEnvironmentTaskResult(operation, taskId, result);
|
||||
if (operation.kind === "read_file") {
|
||||
const [current] = await db.select({ lease: environmentLeases, run: heartbeatRuns }).from(environmentLeases)
|
||||
@@ -316,7 +379,9 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
}
|
||||
if ((parsed.kind === "status" || parsed.kind === "copilot_metadata") && parsed.executionStopped === true) {
|
||||
await db.update(environmentLeases).set({
|
||||
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ runnerTaskExecutionStopped: true, runnerTaskSessionRetained: parsed.kind === "status" && parsed.sessionStateAvailable === true })}::jsonb`,
|
||||
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ runnerTaskExecutionStopped: true,
|
||||
runnerTaskSessionRetained: parsed.kind === "status" && parsed.sessionStateAvailable === true,
|
||||
...(parsed.kind === "status" && parsed.preparationStopped === true ? { runnerTaskPreparationStopped: true } : {}) })}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, input.companyId)));
|
||||
}
|
||||
if (operation.kind === "complete") {
|
||||
@@ -353,16 +418,22 @@ export async function settleDispatchedEnvironmentTask(db: Db, workers: PluginWor
|
||||
metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || '{"runnerTaskRetiring":true}'::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId)))
|
||||
.returning({ metadata: environmentLeases.metadata, heartbeatRunId: environmentLeases.heartbeatRunId });
|
||||
if (current?.metadata?.runnerTaskDispatched !== true) {
|
||||
const startup = current?.metadata?.runnerTaskStartup;
|
||||
const startupCleanup = current?.metadata?.runnerTaskStartupCleanup as { requiresPreparationStop?: boolean } | undefined;
|
||||
if (current?.metadata?.runnerTaskDispatched !== true && !startup && !startupCleanup) {
|
||||
if (current?.metadata) await settlePluginTaskCredentialHandoff(db, { companyId: lease.companyId, leaseId: lease.id }, current.metadata,
|
||||
async () => { throw new Error("Undispatched task must not request credentials"); });
|
||||
return current?.metadata?.runnerTaskExecutionStopped === true;
|
||||
}
|
||||
const scope = { companyId: lease.companyId, leaseId: lease.id };
|
||||
try {
|
||||
const contextId = (startupCleanup as { contextId?: unknown } | undefined)?.contextId ?? (startup as { contextId?: unknown } | undefined)?.contextId;
|
||||
const draining = typeof contextId === "string" ? workers.drainStartup?.(contextId) : undefined;
|
||||
void draining?.catch(() => undefined);
|
||||
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "stop" } });
|
||||
const state = await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "status" } });
|
||||
if (state.kind !== "status" || state.executionStopped !== true) throw new Error("Plugin Runner termination is unconfirmed");
|
||||
if (state.kind !== "status" || state.executionStopped !== true || ((startup || startupCleanup?.requiresPreparationStop) && state.preparationStopped !== true)) throw new Error("Plugin Runner termination is unconfirmed");
|
||||
await draining;
|
||||
await executeEnvironmentTask(db, workers, { ...scope, operation: { kind: "complete" } });
|
||||
return true;
|
||||
} catch {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
||||
import { enableRunFailureReporting } from "../__tests__/helpers/run-failure-reporting.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { eq, inArray } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, completionContracts, createDb, environmentLeases, environments, heartbeatRuns, issues, issueRecoveryActions, issueThreadInteractions, nativeRunFinalizations, nativeRunResults, statusDecisions, workAssessments } from "@paperclipai/db";
|
||||
import {
|
||||
@@ -78,6 +78,55 @@ describeEmbeddedPostgres("reconcileAbandonedExecutionControl reports a genuine f
|
||||
return { companyId, agentId, issueId, runId };
|
||||
}
|
||||
|
||||
it("recognizes pending cleanup selected only by the run deadline batch", async () => {
|
||||
const f = await seedAbandonedRunFixture(), now = new Date();
|
||||
const cleanupIds = Array.from({ length: 51 }, () => randomUUID());
|
||||
const hidden = cleanupIds[50]!;
|
||||
await db.insert(heartbeatRuns).values(cleanupIds.map(id => ({ id, companyId: f.companyId, agentId: f.agentId,
|
||||
nativeIssueId: f.issueId, runtimeMode: "native", status: "running",
|
||||
executionControlDeadlineAt: id === hidden ? new Date(now.getTime() - 120_000) : null })));
|
||||
await db.insert(nativeRunFinalizations).values(cleanupIds.map(runId => ({ runId, companyId: f.companyId,
|
||||
issueId: f.issueId, phase: "terminal_failure", failureCode: "native_startup_cleanup_pending",
|
||||
controlDeadlineAt: new Date(now.getTime() - (runId === hidden ? 60_000 : 120_000)) })));
|
||||
const attempted: string[] = [];
|
||||
try {
|
||||
await reconcileAbandonedExecutionControl(db, now, { startupCleanup: async ({ runId }) => { attempted.push(runId); throw new Error("Provider unavailable"); } });
|
||||
expect(attempted).toContain(hidden);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, hidden)))[0].status).toBe("running");
|
||||
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, hidden)))[0].controlDeadlineAt! > now).toBe(true);
|
||||
} finally {
|
||||
await db.update(nativeRunFinalizations).set({ controlDeadlineAt: null }).where(inArray(nativeRunFinalizations.runId, cleanupIds));
|
||||
await db.update(heartbeatRuns).set({ executionControlDeadlineAt: null }).where(inArray(heartbeatRuns.id, cleanupIds));
|
||||
}
|
||||
});
|
||||
|
||||
it("moves failed startup cleanup retries behind later expired runs", async () => {
|
||||
const f = await seedAbandonedRunFixture();
|
||||
const now = new Date();
|
||||
const cleanupIds = Array.from({ length: 50 }, () => randomUUID());
|
||||
const firstDeadline = new Date(now.getTime() - 120_000);
|
||||
await db.insert(heartbeatRuns).values(cleanupIds.map(id => ({ id, companyId: f.companyId, agentId: f.agentId,
|
||||
nativeIssueId: f.issueId, runtimeMode: "native", status: "running", executionControlDeadlineAt: firstDeadline,
|
||||
contextSnapshot: { issueId: f.issueId } })));
|
||||
await db.insert(nativeRunFinalizations).values(cleanupIds.map(runId => ({ runId, companyId: f.companyId,
|
||||
issueId: f.issueId, phase: "terminal_failure", failureCode: "native_startup_cleanup_pending", controlDeadlineAt: firstDeadline })));
|
||||
const attempted: string[] = [];
|
||||
const startupCleanup = async ({ runId }: { runId: string }) => { attempted.push(runId); throw new Error("Provider unavailable"); };
|
||||
try {
|
||||
await reconcileAbandonedExecutionControl(db, now, { startupCleanup });
|
||||
await reconcileAbandonedExecutionControl(db, now, { startupCleanup });
|
||||
expect(new Set(attempted)).toEqual(new Set(cleanupIds));
|
||||
expect(attempted).toHaveLength(50);
|
||||
const [later] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
expect(later?.status).toBe("failed");
|
||||
const retries = await db.select().from(nativeRunFinalizations).where(inArray(nativeRunFinalizations.runId, cleanupIds));
|
||||
expect(retries.every(row => row.controlDeadlineAt && row.controlDeadlineAt > now)).toBe(true);
|
||||
} finally {
|
||||
await db.update(nativeRunFinalizations).set({ controlDeadlineAt: null }).where(inArray(nativeRunFinalizations.runId, cleanupIds));
|
||||
await db.update(heartbeatRuns).set({ executionControlDeadlineAt: null }).where(inArray(heartbeatRuns.id, cleanupIds));
|
||||
}
|
||||
});
|
||||
|
||||
it("reports exactly one Sentry event for a genuine finalization-deadline failure", async () => {
|
||||
const { runId } = await seedAbandonedRunFixture();
|
||||
const captureCallsBefore = mockCaptureRunFailure.mock.calls.length;
|
||||
|
||||
@@ -8,7 +8,7 @@ import { notifyInstructionCollection } from "./instruction-collection-notificati
|
||||
import { notifyExecutionRunWork } from "./execution-control-notifications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { and, eq, isNotNull, lte, sql } from "drizzle-orm";
|
||||
import { and, asc, eq, isNotNull, lte, sql } from "drizzle-orm";
|
||||
import {
|
||||
agents,
|
||||
heartbeatRuns,
|
||||
@@ -22,17 +22,25 @@ import { hasLiveLegacyController } from "./legacy-controller-lease.js";
|
||||
import { reportRunFailure } from "./run-failure-report.js";
|
||||
import { getNativeReviewAssignment, readNativeReviewAssignmentContext } from "./native-runtime/native-review-participant.js";
|
||||
import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js";
|
||||
import { EXECUTION_RECONCILIATION_INTERVAL_MS } from "./execution-control-deadline.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import { NATIVE_STARTUP_CLEANUP_PENDING, readNativeStartupContext, promoteNativeStartupCleanup } from "./native-runtime/native-startup-cleanup.js";
|
||||
|
||||
/** Only newly recorded control deadlines are eligible. Upgrades never replay ambiguous historical runs. */
|
||||
export async function reconcileAbandonedExecutionControl(
|
||||
db: Db,
|
||||
now = new Date(),
|
||||
options: { startupCleanup?: (scope: { companyId: string; runId: string }) => Promise<unknown> } = {},
|
||||
) {
|
||||
const nativeDue = await db
|
||||
.select({
|
||||
runId: nativeRunFinalizations.runId,
|
||||
issueId: nativeRunFinalizations.issueId,
|
||||
companyId: nativeRunFinalizations.companyId,
|
||||
failureCode: nativeRunFinalizations.failureCode,
|
||||
leaseOwner: nativeRunFinalizations.leaseOwner,
|
||||
controllerGeneration: nativeRunFinalizations.controllerGeneration,
|
||||
attempt: nativeRunFinalizations.attempt,
|
||||
})
|
||||
.from(nativeRunFinalizations)
|
||||
.where(
|
||||
@@ -41,20 +49,28 @@ export async function reconcileAbandonedExecutionControl(
|
||||
lte(nativeRunFinalizations.controlDeadlineAt, now),
|
||||
),
|
||||
)
|
||||
.orderBy(asc(nativeRunFinalizations.controlDeadlineAt), asc(nativeRunFinalizations.runId))
|
||||
.limit(50);
|
||||
const controlDue = await db
|
||||
.select({
|
||||
runId: heartbeatRuns.id,
|
||||
companyId: heartbeatRuns.companyId,
|
||||
context: heartbeatRuns.contextSnapshot,
|
||||
nativeIssueId: nativeRunFinalizations.issueId,
|
||||
failureCode: nativeRunFinalizations.failureCode,
|
||||
leaseOwner: nativeRunFinalizations.leaseOwner,
|
||||
controllerGeneration: nativeRunFinalizations.controllerGeneration,
|
||||
attempt: nativeRunFinalizations.attempt,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.leftJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
|
||||
.where(
|
||||
and(
|
||||
isNotNull(heartbeatRuns.executionControlDeadlineAt),
|
||||
lte(heartbeatRuns.executionControlDeadlineAt, now),
|
||||
),
|
||||
)
|
||||
.orderBy(asc(heartbeatRuns.executionControlDeadlineAt), asc(heartbeatRuns.id))
|
||||
.limit(50);
|
||||
const due = [
|
||||
...new Map(
|
||||
@@ -62,10 +78,10 @@ export async function reconcileAbandonedExecutionControl(
|
||||
...nativeDue,
|
||||
...controlDue.map((row) => ({
|
||||
...row,
|
||||
issueId:
|
||||
issueId: row.nativeIssueId ?? (
|
||||
typeof row.context?.issueId === "string"
|
||||
? row.context.issueId
|
||||
: null,
|
||||
: null),
|
||||
})),
|
||||
].map((row) => [row.runId, row]),
|
||||
).values(),
|
||||
@@ -78,6 +94,25 @@ export async function reconcileAbandonedExecutionControl(
|
||||
while (nextCandidate < due.length) {
|
||||
const candidate = due[nextCandidate++]!;
|
||||
try {
|
||||
if (candidate.failureCode === NATIVE_STARTUP_CLEANUP_PENDING) {
|
||||
if (!options.startupCleanup) continue;
|
||||
try {
|
||||
await options.startupCleanup({ companyId: candidate.companyId, runId: candidate.runId });
|
||||
} finally {
|
||||
await db.transaction(async tx => {
|
||||
if (candidate.issueId) await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, candidate.issueId)).for("update");
|
||||
const [owner] = await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, candidate.runId)).for("update");
|
||||
if (owner?.failureCode !== NATIVE_STARTUP_CLEANUP_PENDING || !owner.controlDeadlineAt || owner.controlDeadlineAt > now ||
|
||||
owner.leaseOwner !== candidate.leaseOwner || owner.controllerGeneration !== candidate.controllerGeneration || owner.attempt !== candidate.attempt) return;
|
||||
const cleanupRetryAt = new Date(Math.max(now.getTime(), Date.now()) + EXECUTION_RECONCILIATION_INTERVAL_MS);
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization);
|
||||
await tx.update(nativeRunFinalizations).set({ controlDeadlineAt: cleanupRetryAt }).where(eq(nativeRunFinalizations.runId, candidate.runId));
|
||||
await tx.update(heartbeatRuns).set({ executionControlDeadlineAt: cleanupRetryAt }).where(and(eq(heartbeatRuns.id, candidate.runId),
|
||||
isNotNull(heartbeatRuns.executionControlDeadlineAt), lte(heartbeatRuns.executionControlDeadlineAt, now)));
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// Set inside the transaction only when the write below genuinely
|
||||
// transitions the run into "failed". Read after the transaction
|
||||
// commits, so a rolled-back write never reports a false failure.
|
||||
@@ -128,6 +163,18 @@ export async function reconcileAbandonedExecutionControl(
|
||||
].some((deadline) => deadline && deadline <= now)
|
||||
)
|
||||
return false;
|
||||
if (coordinator?.failureCode === NATIVE_STARTUP_CLEANUP_PENDING) return false;
|
||||
const startupContext = coordinator && readNativeStartupContext(run, coordinator);
|
||||
if (startupContext) {
|
||||
if (!await promoteNativeStartupCleanup(tx as unknown as Db, run, coordinator!, now)) {
|
||||
const retryAt = new Date(Math.max(now.getTime() + EXECUTION_RECONCILIATION_INTERVAL_MS,
|
||||
startupContext.deadlineAtMs, coordinator!.leaseExpiresAt?.getTime() ?? 0));
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization);
|
||||
await tx.update(nativeRunFinalizations).set({ controlDeadlineAt: retryAt }).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
await tx.update(heartbeatRuns).set({ executionControlDeadlineAt: retryAt }).where(eq(heartbeatRuns.id, run.id));
|
||||
}
|
||||
return false;
|
||||
}
|
||||
await tx
|
||||
.update(heartbeatRuns)
|
||||
.set({ executionControlDeadlineAt: null })
|
||||
|
||||
@@ -398,6 +398,7 @@ import {
|
||||
isNativeRunnerOwnershipHeld,
|
||||
NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE,
|
||||
} from "./native-runtime/native-runner-ownership.js";
|
||||
import { NativeStartupCleanupPendingError, requestNativeStartupCleanup, verifyNativeStartupCleanup, readNativeStartupCleanup } from "./native-runtime/native-startup-cleanup.js";
|
||||
import {
|
||||
emitAgentTaskRun,
|
||||
emitAgentTaskRunById,
|
||||
@@ -1436,6 +1437,14 @@ export function heartbeatService(
|
||||
}
|
||||
|
||||
const runControl = createHeartbeatRunControl(db, {
|
||||
verifyStartupCleanup: async run => {
|
||||
await verifyNativeStartupCleanup(db, run, () => envOrchestrator.releaseForRun({
|
||||
heartbeatRunId: run.id, companyId: run.companyId, agentId: run.agentId, status: "expired",
|
||||
cancelActiveWork: true, providerResourceDisposition: "destroy",
|
||||
}));
|
||||
const marker = readNativeStartupCleanup(run)!;
|
||||
await options.pluginWorkerManager?.retireStartup?.(marker.contextId);
|
||||
},
|
||||
getRun,
|
||||
getRunCleanupState,
|
||||
getRunExecutionWait,
|
||||
@@ -1759,7 +1768,7 @@ export function heartbeatService(
|
||||
},
|
||||
});
|
||||
})().catch((error) => {
|
||||
if (!nativeRecoveryDeadlines.has(runId)) {
|
||||
if (!nativeRecoveryDeadlines.has(runId) || nativeRecoveryDeadlines.get(runId) === pending) {
|
||||
scheduleNativeSessionResumeDispatch(runId, new Date(Date.now() + 30_000), mode);
|
||||
}
|
||||
logger.error(
|
||||
@@ -1956,6 +1965,19 @@ export function heartbeatService(
|
||||
Parameters<typeof cleanupGitHubOperationLaunchers>[0] | null = null;
|
||||
let nativeSessionResumeScheduled = false;
|
||||
let nativeOwnershipHeld = false;
|
||||
let nativeStartupCleanupScheduled = false;
|
||||
let nativeStartupCleanupError: NativeStartupCleanupPendingError | undefined;
|
||||
const requestStartupCleanup = async (error: NativeStartupCleanupPendingError) => {
|
||||
nativeStartupCleanupError = error;
|
||||
try {
|
||||
nativeStartupCleanupScheduled = await requestNativeStartupCleanup(db, { companyId: run.companyId, agentId: run.agentId, runId: run.id,
|
||||
contextId: error.contextId, deadlineAtMs: error.deadlineAtMs, leaseOwner: error.leaseOwner,
|
||||
leaseId: error.leaseId, requiresPreparationStop: error.requiresPreparationStop });
|
||||
} catch (writeError) {
|
||||
scheduleNativeSessionResumeDispatch(run.id, new Date(Date.now() + 30_000), "restart");
|
||||
throw writeError;
|
||||
}
|
||||
};
|
||||
let instructionWorkingCopy: ReturnType<typeof createHeartbeatInstructionWorkingCopy> | null = null;
|
||||
let nativeDispatchStarted = false;
|
||||
let nativeWorkspaceFinalizeScheduled = false;
|
||||
@@ -3527,6 +3549,10 @@ export function heartbeatService(
|
||||
}));
|
||||
} catch (err) {
|
||||
await persistUsageCaptureFailure?.();
|
||||
if (err instanceof NativeStartupCleanupPendingError) {
|
||||
await requestStartupCleanup(err);
|
||||
return;
|
||||
}
|
||||
if (err instanceof NativeControllerDetachedForRestartError) {
|
||||
nativeSessionResumeScheduled = true;
|
||||
return;
|
||||
@@ -3659,7 +3685,12 @@ export function heartbeatService(
|
||||
});
|
||||
}
|
||||
} catch (outerErr) {
|
||||
if (
|
||||
if (nativeStartupCleanupScheduled || nativeStartupCleanupError || outerErr instanceof NativeStartupCleanupPendingError) {
|
||||
const pending = outerErr instanceof NativeStartupCleanupPendingError ? outerErr : nativeStartupCleanupError;
|
||||
if (!nativeStartupCleanupScheduled && pending) await requestStartupCleanup(pending).catch(error => {
|
||||
logger.warn({ err: error, runId: run.id }, "Native startup cleanup intent will retry through owned recovery");
|
||||
});
|
||||
} else if (
|
||||
nativeOwnershipHeld ||
|
||||
outerErr instanceof NativeRunnerOwnershipUnverifiedError
|
||||
) {
|
||||
@@ -3696,7 +3727,7 @@ export function heartbeatService(
|
||||
);
|
||||
} else {
|
||||
await preserveNativeWorkspaceOnFailure();
|
||||
if (!nativeOwnershipHeld) await runCompletion.failRunSetup({
|
||||
if (!nativeOwnershipHeld && !nativeStartupCleanupScheduled && !nativeStartupCleanupError) await runCompletion.failRunSetup({
|
||||
run, runId, outerErr, signal: executionControl.controller.signal,
|
||||
providerDispatchStarted: legacyAdapterEntered || nativeDispatchStarted,
|
||||
identityRedactor, readFailureReportSecrets,
|
||||
@@ -3746,7 +3777,7 @@ export function heartbeatService(
|
||||
latestRun &&
|
||||
!nativeSessionResumeScheduled &&
|
||||
!nativeWorkspaceFinalizeScheduled &&
|
||||
!nativeOwnershipHeld
|
||||
!nativeOwnershipHeld && !nativeStartupCleanupScheduled && !nativeStartupCleanupError
|
||||
) {
|
||||
latestRun = await terminalizeRunOnLeaseRelease(latestRun).catch(
|
||||
(terminalizeErr) => {
|
||||
@@ -3772,7 +3803,7 @@ export function heartbeatService(
|
||||
if (
|
||||
!nativeSessionResumeScheduled &&
|
||||
!nativeWorkspaceFinalizeScheduled &&
|
||||
!nativeOwnershipHeld
|
||||
!nativeOwnershipHeld && !nativeStartupCleanupScheduled && !nativeStartupCleanupError
|
||||
) {
|
||||
// Keep launchers during same-run recovery. At a terminal boundary all
|
||||
// operations have settled; clean before the remote lease can be stopped.
|
||||
@@ -3895,7 +3926,7 @@ export function heartbeatService(
|
||||
}).where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "cancelled")));
|
||||
}
|
||||
if (latestRun?.runtimeMode === "native" && isHeartbeatRunTerminalStatus(latestRun.status) &&
|
||||
!nativeSessionResumeScheduled && !nativeWorkspaceFinalizeScheduled && !nativeOwnershipHeld) {
|
||||
!nativeSessionResumeScheduled && !nativeWorkspaceFinalizeScheduled && !nativeOwnershipHeld && !nativeStartupCleanupScheduled && !nativeStartupCleanupError) {
|
||||
await settleNativeRunProcess(db, run.id).catch(error => {
|
||||
logger.warn({ err: error, runId: run.id }, "failed to release settled native process identity");
|
||||
});
|
||||
@@ -3932,7 +3963,7 @@ export function heartbeatService(
|
||||
caseHandoffCleanupSettled = hasStoppedLocalProcessTree(latestRun);
|
||||
}
|
||||
if (caseHandoffCleanupSettled && latestRun && !nativeSessionResumeScheduled && !nativeWorkspaceFinalizeScheduled &&
|
||||
!nativeOwnershipHeld && isHeartbeatRunTerminalStatus(latestRun.status)) {
|
||||
!nativeOwnershipHeld && !nativeStartupCleanupScheduled && !nativeStartupCleanupError && isHeartbeatRunTerminalStatus(latestRun.status)) {
|
||||
const handoff = await pendingCaseHandoffForRun(db, run.id);
|
||||
if (handoff) await recordCaseHandoffExecutionReleased(db, run.id, handoff.issueId);
|
||||
}
|
||||
|
||||
@@ -168,13 +168,13 @@ describe("heartbeat recovery module callbacks", () => {
|
||||
expect(database.access).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("persists an authentication ownership hold before its event without terminating or retrying", async () => {
|
||||
it.each(["adopted_runner_authentication_timeout"] as const)("persists a %s ownership hold before its event without terminating or retrying", async reason => {
|
||||
const database = guardedDatabase();
|
||||
const deps = callbacks(database.db);
|
||||
const run = { id: "run", runtimeMode: "native", status: "running" } as Run;
|
||||
const blocked = { ...run, nativePhase: "terminal_failure", errorCode: NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE };
|
||||
deps.setRunStatusFromLive.mockResolvedValue({ updated: true, run: blocked });
|
||||
expect(await createHeartbeatRecovery(database.db, deps).markNativeOwnershipUnverified(run, { reason: "adopted_runner_authentication_timeout" })).toBe(blocked);
|
||||
expect(await createHeartbeatRecovery(database.db, deps).markNativeOwnershipUnverified(run, { reason })).toBe(blocked);
|
||||
expect(deps.setRunStatusFromLive).toHaveBeenCalledWith("run", "running", ["running"], expect.objectContaining({ nativePhase: "terminal_failure" }));
|
||||
expect(deps.appendRunEvent).toHaveBeenCalledAfter(deps.setRunStatusFromLive);
|
||||
expect(deps.cancelHeartbeatNativeRun).not.toHaveBeenCalled();
|
||||
@@ -207,6 +207,42 @@ describePostgres("heartbeat recovery module ownership", () => {
|
||||
return (await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, invocationSource: "automation", status: "running", runtimeMode: "legacy" }).returning())[0];
|
||||
}
|
||||
|
||||
it("routes an expired pre-capability startup to cleanup before generic orphan ownership holds", async () => {
|
||||
const run = await seedRun(), leaseId = randomUUID(), environmentId = randomUUID(), leaseOwner = randomUUID();
|
||||
const deadlineAtMs = Date.now() - 1_000, contextId = randomUUID();
|
||||
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
|
||||
await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, run.issueId!));
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, nativePhase: "observed",
|
||||
runnerInstanceId: randomUUID(), nativeSessionId: randomUUID(), runnerProfileJson: { nativeStartupContext: {
|
||||
schema: "paperclip.native-startup-context.v1", companyId: run.companyId, agentId: run.agentId, runId: run.id,
|
||||
contextId, deadlineAtMs, leaseId, leaseOwner, controllerGeneration: 1, attempt: 1 } } }).where(eq(heartbeatRuns.id, run.id));
|
||||
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
|
||||
phase: "observed", attempt: 1, leaseOwner, controllerGeneration: 1, leaseExpiresAt: new Date(Date.now() - 1_000),
|
||||
controllerBootId: randomUUID(), controllerPid: process.pid, controllerProcessStartedAt: new Date() });
|
||||
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "plugin", config: {} });
|
||||
await db.insert(environmentLeases).values({ id: leaseId, companyId: run.companyId, environmentId, heartbeatRunId: run.id,
|
||||
issueId: run.issueId, provider: "plugin", providerLeaseId: "early-task", status: "active", metadata: { runnerTaskLease: true } });
|
||||
const deps = callbacks(db);
|
||||
deps.getRun.mockImplementation(async id => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0] ?? null);
|
||||
native.claims.mockImplementationOnce(claimNativeRestartRecoveries);
|
||||
try {
|
||||
await createHeartbeatRecovery(db, deps).recoverActiveExecutions({ runId: run.id });
|
||||
expect(native.claims).toHaveBeenCalledWith(expect.objectContaining({ runIds: [run.id] }));
|
||||
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
|
||||
expect(current).toMatchObject({ status: "running", errorCode: "native_startup_cleanup_pending",
|
||||
resultJson: { nativeStartupCleanup: { contextId, state: "pending" } } });
|
||||
expect(deps.executeRun).not.toHaveBeenCalled();
|
||||
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
await db.delete(environmentLeases).where(eq(environmentLeases.id, leaseId));
|
||||
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, run.id));
|
||||
await db.delete(activityLog).where(eq(activityLog.runId, run.id));
|
||||
await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, run.issueId!));
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
|
||||
}
|
||||
});
|
||||
|
||||
async function ownedProcess() {
|
||||
const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)", `paperclip-recovery-${randomUUID()}`], {
|
||||
stdio: "ignore", detached: true, env: { PATH: process.env.PATH },
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { readNativeStartupCleanup, readNativeStartupContext } from "../native-runtime/native-startup-cleanup.js";
|
||||
import { pendingFailedNativeRetryCancellationCondition } from "../native-runtime/native-cancellation-request.js";
|
||||
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
|
||||
import { registerLiveRecoveryWork } from "./live-recovery-work.js";
|
||||
@@ -1038,7 +1039,9 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
? "Native restart recovery is waiting for safe ownership evidence"
|
||||
: disposition.kind === "already_finalized"
|
||||
? "Native restart recovery found an already-finalized result"
|
||||
: "Native restart recovery blocked ambiguous or conflicting ownership",
|
||||
: disposition.reason === "native_startup_cleanup_pending"
|
||||
? "Native startup timed out; cancellation and verified cleanup are queued"
|
||||
: "Native restart recovery blocked ambiguous or conflicting ownership",
|
||||
payload: {
|
||||
restartKind: isClaim ? disposition.restartKind : restartKind,
|
||||
recoveryRequestId: isClaim
|
||||
@@ -2251,6 +2254,10 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
nativeControllerProcessStartedAt:
|
||||
nativeRunFinalizations.controllerProcessStartedAt,
|
||||
nativeControllerLeaseExpiresAt: nativeRunFinalizations.leaseExpiresAt,
|
||||
nativeLeaseOwner: nativeRunFinalizations.leaseOwner,
|
||||
nativeControllerGeneration: nativeRunFinalizations.controllerGeneration,
|
||||
nativeAttempt: nativeRunFinalizations.attempt,
|
||||
nativeResultId: nativeRunFinalizations.resultId,
|
||||
nativeWorkspaceFinalizationPending: pendingNativeWorkspaceFinalizationCondition(db),
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
@@ -2271,7 +2278,10 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
&& !isNativeRunnerOwnershipHeld(run)
|
||||
&& !activeRunExecutions.has(run.id) && !runningProcesses.has(run.id),
|
||||
).map(({ run }) => run.id);
|
||||
const remoteRecoveryRunIds = new Set(await listExpiredNativeRunnerAuthorities(db, now, expiredNativeRunIds));
|
||||
const earlyStartupRunIds = activeRuns.filter(row => expiredNativeRunIds.includes(row.run.id) &&
|
||||
readNativeStartupContext(row.run, { leaseOwner: row.nativeLeaseOwner, controllerGeneration: row.nativeControllerGeneration ?? 0,
|
||||
attempt: row.nativeAttempt ?? 0, resultId: row.nativeResultId })).map(row => row.run.id);
|
||||
const remoteRecoveryRunIds = new Set([...earlyStartupRunIds, ...await listExpiredNativeRunnerAuthorities(db, now, expiredNativeRunIds)]);
|
||||
if (remoteRecoveryRunIds.size) {
|
||||
const recovery = await recoverNativeRunsAfterRestart(now, [...remoteRecoveryRunIds]);
|
||||
for (const claim of recovery.claims) resumedRunIds.add(claim.runId);
|
||||
@@ -2321,6 +2331,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
nativeControllerLeaseExpiresAt,
|
||||
nativeWorkspaceFinalizationPending,
|
||||
} of activeRuns) {
|
||||
if (run.runtimeMode === "native" && readNativeStartupCleanup(run)) continue;
|
||||
if (parseObject(run.contextSnapshot).nativeQuestionCancellation) continue;
|
||||
// Native recovery owns these dispositions. Its writes happened after this
|
||||
// inventory was read; do not reap or overwrite them using the old row.
|
||||
|
||||
@@ -123,7 +123,8 @@ import {
|
||||
} from "../native-runtime/index.js";
|
||||
import { runningProcesses } from "../../adapters/index.js";
|
||||
import { parseObject } from "../../adapters/utils.js";
|
||||
import { isNativeRunnerOwnershipHeld } from "../native-runtime/native-runner-ownership.js";
|
||||
import { isNativeRunnerOwnershipHeld, nativeRunnerOwnershipNotHeldCondition } from "../native-runtime/native-runner-ownership.js";
|
||||
import { readNativeStartupCleanup, acknowledgeNativeStartupCleanup } from "../native-runtime/native-startup-cleanup.js";
|
||||
import {
|
||||
withCurrentBudgetEnforcement,
|
||||
type BudgetEnforcementScope,
|
||||
@@ -177,6 +178,7 @@ export interface HeartbeatRunControlDependencies extends Pick<ReturnType<typeof
|
||||
Pick<ReturnType<typeof createHeartbeatLifecycle>,
|
||||
"mergeRunStopMetadataForAgent" | "setRunStatusFromLive" | "setRunStatus" | "setWakeupStatus" | "appendRunEvent">,
|
||||
Pick<HeartbeatRetryDependencies, "finalizeAgentStatus"> {
|
||||
verifyStartupCleanup?: (run: typeof heartbeatRuns.$inferSelect) => Promise<void>;
|
||||
options: {
|
||||
closeWarmNativeSessionsForRun?: (input: {
|
||||
runId: string;
|
||||
@@ -1399,13 +1401,19 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
|
||||
}
|
||||
let terminationSettled = false;
|
||||
try {
|
||||
await cancelHeartbeatNativeRun({
|
||||
db,
|
||||
runId: run.id,
|
||||
reason,
|
||||
runtimeMode: run.runtimeMode,
|
||||
...(options.cancellationRequestId ? { cancellationRequestId: options.cancellationRequestId } : {}),
|
||||
});
|
||||
const startupCleanupRun = run.runtimeMode === "native" ? await getRun(run.id) : null;
|
||||
if (startupCleanupRun && readNativeStartupCleanup(startupCleanupRun) && !control && !running && !activeRunExecutions.has(run.id)) {
|
||||
if (!dependencies.verifyStartupCleanup) throw conflict("Native startup cleanup worker unavailable");
|
||||
await dependencies.verifyStartupCleanup(startupCleanupRun);
|
||||
} else {
|
||||
await cancelHeartbeatNativeRun({
|
||||
db,
|
||||
runId: run.id,
|
||||
reason,
|
||||
runtimeMode: run.runtimeMode,
|
||||
...(options.cancellationRequestId ? { cancellationRequestId: options.cancellationRequestId } : {}),
|
||||
});
|
||||
}
|
||||
if (running) {
|
||||
await terminateHeartbeatRunProcess({
|
||||
pid: running.child.pid,
|
||||
@@ -1466,12 +1474,13 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
|
||||
}
|
||||
|
||||
const finishedAt = new Date();
|
||||
const persistedCancellationResult =
|
||||
run.runtimeMode === "native"
|
||||
? await getRun(run.id).then((current) =>
|
||||
parseObject(current?.resultJson),
|
||||
)
|
||||
: {};
|
||||
let persistedCancellationRun = run.runtimeMode === "native" ? await getRun(run.id) : null;
|
||||
if (persistedCancellationRun && readNativeStartupCleanup(persistedCancellationRun)) {
|
||||
if (!dependencies.verifyStartupCleanup) throw conflict("Native startup cleanup worker unavailable");
|
||||
await dependencies.verifyStartupCleanup(persistedCancellationRun);
|
||||
persistedCancellationRun = await getRun(run.id);
|
||||
}
|
||||
const persistedCancellationResult = parseObject(persistedCancellationRun?.resultJson);
|
||||
return await setRunStatusFromLive(
|
||||
run.id,
|
||||
"cancelled",
|
||||
@@ -1520,12 +1529,18 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
|
||||
persistedCancellationResult.nativeCancellation,
|
||||
}
|
||||
: {}),
|
||||
...(Object.hasOwn(persistedCancellationResult, "nativeStartupCleanup")
|
||||
? { nativeStartupCleanup: persistedCancellationResult.nativeStartupCleanup } : {}),
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
},
|
||||
undefined,
|
||||
pendingNativeRetry ? nativeRetryCancellationCommitCondition(persistedCancellationResult) : undefined,
|
||||
and(pendingNativeRetry ? nativeRetryCancellationCommitCondition(persistedCancellationResult)
|
||||
: persistedCancellationRun && readNativeStartupCleanup(persistedCancellationRun) ? and(nativeRunnerOwnershipNotHeldCondition(),
|
||||
sql`not exists (select 1 from ${nativeRunFinalizations} where ${nativeRunFinalizations.runId} = ${run.id} and ${nativeRunFinalizations.resultId} is not null)`)
|
||||
: undefined,
|
||||
run.runtimeMode === "native" ? sql`coalesce(${heartbeatRuns.resultJson}->'nativeStartupCleanup'->>'state', '') <> 'pending'` : undefined),
|
||||
);
|
||||
} catch (error) {
|
||||
if (processCancellationSettlement) {
|
||||
@@ -1557,6 +1572,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
|
||||
const cancelled = cancellation.run;
|
||||
|
||||
if (cancellation.updated && cancelled) {
|
||||
if (readNativeStartupCleanup(cancelled)) await acknowledgeNativeStartupCleanup(db, { companyId: cancelled.companyId, runId: cancelled.id });
|
||||
await setWakeupStatus(run.wakeupRequestId, "cancelled", {
|
||||
finishedAt: cancelled.finishedAt ?? new Date(),
|
||||
error: reason,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { readNativeStartupCleanup } from "../native-runtime/native-startup-cleanup.js";
|
||||
import { notifyStrandedAgentStatus } from "../live-run-recovery-notifications.js";
|
||||
import { notifyRunDispatch } from "../run-dispatch-notifications.js";
|
||||
import { preserveAcceptedRunUsageProjectionSql } from "../usage-receipts.js";
|
||||
@@ -1159,6 +1160,9 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
|
||||
.where(eq(heartbeatRuns.id, runId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
const previousStatus = previous?.run ?? null;
|
||||
if (status === "failed" && patch?.errorCode === "process_lost" && previousStatus?.runtimeMode === "native" && readNativeStartupCleanup(previousStatus)) {
|
||||
return { updated: false, run: previousStatus };
|
||||
}
|
||||
|
||||
if (previousStatus && (status === "cancelled" || status === "interrupted")) {
|
||||
patch = { ...patch, resultJson: cancellationResultJson(previousStatus, status, patch?.resultJson, patch?.errorCode, patch?.error) };
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { NativeStartupCleanupPendingError } from "../native-runtime/native-startup-cleanup.js";
|
||||
import { NativeControllerRecoveryWaitingError } from "../native-runtime/native-controller-recovery.js";
|
||||
|
||||
import { usesNativeUsageDelivery } from "../native-runtime/native-usage-accounting.js";
|
||||
@@ -1072,15 +1073,21 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
|
||||
&& !(nativeExecution && "access" in nativeExecution.workspace && nativeExecution.workspace.access === "none")) {
|
||||
input.effects.onProviderResourceDisposition("stop_and_retain");
|
||||
}
|
||||
await writeExecutionControl(db, DELIVERY_QUEUES.executionFinalization, tx => tx
|
||||
.update(heartbeatRuns)
|
||||
.set({ executionControlDeadlineAt: new Date(Date.now() + 60_000) })
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRuns.id, run.id),
|
||||
eq(heartbeatRuns.status, "running"),
|
||||
),
|
||||
), { companyId: run.companyId });
|
||||
try {
|
||||
await writeExecutionControl(db, DELIVERY_QUEUES.executionFinalization, tx => tx
|
||||
.update(heartbeatRuns)
|
||||
.set({ executionControlDeadlineAt: new Date(Date.now() + 60_000) })
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRuns.id, run.id),
|
||||
eq(heartbeatRuns.status, "running"),
|
||||
),
|
||||
), { companyId: run.companyId });
|
||||
} catch (writeError) {
|
||||
if (adapterErr instanceof NativeStartupCleanupPendingError) throw adapterErr;
|
||||
throw writeError;
|
||||
}
|
||||
if (adapterErr instanceof NativeStartupCleanupPendingError) throw adapterErr;
|
||||
if (
|
||||
issueRef &&
|
||||
context.resumeSessionGoalHeartbeat === true &&
|
||||
|
||||
@@ -32,8 +32,8 @@ const invocations = [];
|
||||
const send = (id, result) => process.stdout.write(JSON.stringify({jsonrpc:"2.0",id,result})+"\\n");
|
||||
readline.createInterface({input:process.stdin,crlfDelay:Infinity}).on("line", line => {
|
||||
const message = JSON.parse(line);
|
||||
if (message.method === "initialize") send(message.id, {ok:true,supportedMethods:["environmentTask","configChanged","health"]});
|
||||
else if (message.method === "environmentTask" || message.method === "environmentDestroyLease") {
|
||||
if (message.method === "initialize") send(message.id, {ok:true,supportedMethods:["environmentTask","environmentAcquireLease","configChanged","health"]});
|
||||
else if (message.method === "environmentTask" || message.method === "environmentDestroyLease" || message.method === "environmentAcquireLease") {
|
||||
held.push(message.id);
|
||||
invocations.push(message.paperclipInvocation?.id);
|
||||
if (message.params.config.hostWrite) process.stdout.write(JSON.stringify({jsonrpc:"2.0",id:"host-"+message.id,method:"companies.get",params:{companyId:"company"},paperclipInvocationId:message.paperclipInvocation.id})+"\\n");
|
||||
@@ -43,7 +43,7 @@ readline.createInterface({input:process.stdin,crlfDelay:Infinity}).on("line", li
|
||||
else if (message.method === "configChanged") {
|
||||
if (message.params.config.exit) process.exit(1);
|
||||
if (message.params.config.hostAfter) process.stdout.write(JSON.stringify({jsonrpc:"2.0",id:"late-host",method:"companies.get",params:{companyId:"company"},paperclipInvocationId:invocations.at(-1)})+"\\n");
|
||||
if (message.params.config.release) for (const id of held.splice(0)) send(id, {kind:"description",taskId:"task"});
|
||||
if (message.params.config.release) for (const id of held.splice(0)) { send(id, {providerLeaseId:"late-task",kind:"description",taskId:"task"}); if (message.params.config.duplicate) send(id, {providerLeaseId:"late-task",kind:"description",taskId:"task"}); }
|
||||
send(message.id, {});
|
||||
} else if (message.method === "shutdown") {send(message.id, {});setImmediate(()=>process.exit(0));}
|
||||
});
|
||||
@@ -59,6 +59,70 @@ async function startWorker(hostHandlers: Record<string, unknown> = {}) {
|
||||
}
|
||||
|
||||
describe("Core startup responsibility across real worker IPC", () => {
|
||||
it("drops only a dead acquisition reply placeholder while retaining real writes until verified retirement", async () => {
|
||||
let release!: () => void, writing!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), entered = new Promise<void>(resolve => { writing = resolve; });
|
||||
const manager = await startWorker({ "companies.get": async () => { writing(); await gate; return {}; } });
|
||||
const settled = vi.fn(), observe = vi.fn(async () => {});
|
||||
try {
|
||||
const acquisition = manager.call(manifest.id, "environmentAcquireLease", { companyId: "company", environmentId: "environment",
|
||||
driverKey: "tasks", config: { hostWrite: true }, runId: "run", agentId: "agent", adapterType: "paperclip_runner" },
|
||||
10, undefined, settled, "dead-acquire", observe);
|
||||
const rejected = expect(acquisition).rejects.toThrow("timed out");
|
||||
await entered; await rejected;
|
||||
await manager.stopWorker(manifest.id);
|
||||
let drained = false;
|
||||
const draining = manager.drainStartup!("dead-acquire").then(() => { drained = true; });
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(drained).toBe(false);
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
release(); await draining;
|
||||
expect(observe).not.toHaveBeenCalled();
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
await manager.retireStartup!("dead-acquire");
|
||||
expect(settled).toHaveBeenCalledOnce();
|
||||
} finally { release(); await manager.stopAll(); }
|
||||
}, 25_000);
|
||||
|
||||
it.each([false, true])("retains a received acquisition reply through failed recording and retirement (worker exits=%s)", async workerExits => {
|
||||
const manager = await startWorker(), settled = vi.fn();
|
||||
let release!: () => void, recording!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), started = new Promise<void>(resolve => { recording = resolve; });
|
||||
let unavailable = true;
|
||||
const observe = vi.fn(async (result: unknown) => {
|
||||
expect(result).toMatchObject({ providerLeaseId: "late-task" });
|
||||
if (unavailable) throw new Error("Database unavailable");
|
||||
recording(); await gate;
|
||||
});
|
||||
try {
|
||||
const acquire = manager.call(manifest.id, "environmentAcquireLease", { companyId: "company", environmentId: "environment",
|
||||
driverKey: "tasks", config: {}, runId: "run", agentId: "agent", adapterType: "paperclip_runner" }, 10, undefined, settled, "late-acquire", observe);
|
||||
await expect(acquire).rejects.toThrow("timed out");
|
||||
await manager.call(manifest.id, "configChanged", { config: { release: true, duplicate: true } });
|
||||
await vi.waitFor(() => expect(observe).toHaveBeenCalledOnce());
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
if (workerExits) await manager.stopWorker(manifest.id);
|
||||
unavailable = false;
|
||||
vi.useFakeTimers();
|
||||
const failed = manager.retireStartup!("late-acquire").catch(error => error.message);
|
||||
await started;
|
||||
expect(observe).toHaveBeenCalledTimes(2);
|
||||
await vi.advanceTimersByTimeAsync(15_000);
|
||||
expect(await failed).toBe("Task cleanup is still pending");
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
vi.useRealTimers();
|
||||
let retired = false;
|
||||
const retiring = manager.retireStartup!("late-acquire").then(() => { retired = true; });
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(retired).toBe(false);
|
||||
release(); await retiring;
|
||||
expect(settled).toHaveBeenCalledOnce();
|
||||
await manager.retireStartup!("late-acquire");
|
||||
expect(observe).toHaveBeenCalledTimes(2);
|
||||
} finally { vi.useRealTimers(); release(); await manager.stopAll(); }
|
||||
});
|
||||
|
||||
it.each(["stop", "replace", "stopAll"])("keeps stopped startup work reachable through %s until verified retirement", async mode => {
|
||||
let release!: () => void, entered!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; }), started = new Promise<void>(resolve => { entered = resolve; });
|
||||
|
||||
@@ -38,6 +38,7 @@ import { isNewTerminalRunFailure, reportRunFailure } from "../run-failure-report
|
||||
import { isNativeRunnerOwnershipHeld } from "./native-runner-ownership.js";
|
||||
import { nativeRestartInterruptedTurnId, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js";
|
||||
import { retireSupersededNativeRetry } from "./native-superseded-retry.js";
|
||||
import { requestNativeStartupCleanup, readNativeStartupContext, promoteNativeStartupCleanup } from "./native-startup-cleanup.js";
|
||||
|
||||
export type NativeControllerIdentity = {
|
||||
bootId: string;
|
||||
@@ -623,22 +624,41 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
} as const;
|
||||
}
|
||||
|
||||
const startupContext = readNativeStartupContext(row.run, row.coordinator);
|
||||
const startupRetryAt = startupContext ? new Date(Math.max(now.getTime() + 30_000,
|
||||
startupContext.deadlineAtMs, row.coordinator.leaseExpiresAt?.getTime() ?? 0)) : undefined;
|
||||
if (input.liveRunExecutions?.has(row.run.id)) {
|
||||
return {
|
||||
kind: "awaiting_evidence",
|
||||
runId: row.run.id,
|
||||
reason: "current_controller_preparing",
|
||||
...(startupRetryAt ? { retryAt: startupRetryAt } : {}),
|
||||
} as const;
|
||||
}
|
||||
if (startupContext) {
|
||||
if (await promoteNativeStartupCleanup(tx as unknown as Db, row.run, row.coordinator, now)) {
|
||||
return { kind: "blocked", runId: row.run.id, reason: "native_startup_cleanup_pending" } as const;
|
||||
}
|
||||
return { kind: "awaiting_evidence", runId: row.run.id, reason: "native_startup_cleanup_pending", retryAt: startupRetryAt } as const;
|
||||
}
|
||||
|
||||
const profile = row.run.runnerProfileJson ?? {};
|
||||
const storedRemoteWorkspace = remoteWorkspaceRunReferenceSchema.safeParse(profile.remoteWorkspace);
|
||||
const remoteWorkspace = readNativeWorkspaceSyncReference(profile.nativeWorkspaceSync)
|
||||
?? (storedRemoteWorkspace.success ? storedRemoteWorkspace.data : null);
|
||||
const [pluginTask] = await tx.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
|
||||
const [pluginTask] = await tx.select({ id: environmentLeases.id, metadata: environmentLeases.metadata }).from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, row.run.companyId), eq(environmentLeases.heartbeatRunId, row.run.id),
|
||||
sql`${environmentLeases.metadata}->>'runnerTaskLease' = 'true'`,
|
||||
)).orderBy(desc(environmentLeases.createdAt)).limit(1);
|
||||
const startup = pluginTask?.metadata?.runnerTaskStartup as { contextId?: unknown; deadlineAtMs?: unknown; runId?: unknown } | undefined;
|
||||
if (startup?.runId === row.run.id && typeof startup.contextId === "string" &&
|
||||
Number.isSafeInteger(startup.deadlineAtMs) && Number(startup.deadlineAtMs) <= now.getTime() &&
|
||||
!row.run.sessionIdAfter && row.coordinator.leaseOwner && row.coordinator.leaseExpiresAt && row.coordinator.leaseExpiresAt <= now) {
|
||||
await requestNativeStartupCleanup(tx as unknown as Db, { companyId: row.run.companyId, agentId: row.run.agentId,
|
||||
runId: row.run.id, contextId: startup.contextId, deadlineAtMs: Number(startup.deadlineAtMs),
|
||||
leaseOwner: row.coordinator.leaseOwner, requiresPreparationStop: true, leaseId: pluginTask!.id }, { publish: false });
|
||||
return { kind: "blocked", runId: row.run.id, reason: "native_startup_cleanup_pending" } as const;
|
||||
}
|
||||
const remoteSandbox = profile.nativeWorkspaceSync != null || profile.remoteWorkspace != null;
|
||||
const remoteProcess = remoteSandbox || !!pluginTask;
|
||||
const durableAuthority = (remoteWorkspace || pluginTask) && await hasNativeRunnerCheckpoint(tx, {
|
||||
|
||||
@@ -0,0 +1,208 @@
|
||||
import { and, eq, isNull, sql } from "drizzle-orm";
|
||||
import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, sandboxCreationOperations, type Db } from "@paperclipai/db";
|
||||
import { conflict } from "../../errors.js";
|
||||
import { persistActivity, publishActivity } from "../activity-log.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
|
||||
import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
|
||||
import { isNativeRunnerOwnershipHeld } from "./native-runner-ownership.js";
|
||||
|
||||
export const NATIVE_STARTUP_CLEANUP_PENDING = "native_startup_cleanup_pending";
|
||||
|
||||
export class NativeStartupCleanupPendingError extends Error {
|
||||
constructor(readonly contextId: string, readonly deadlineAtMs: number, readonly leaseOwner: string,
|
||||
readonly requiresPreparationStop: boolean, readonly leaseId?: string) {
|
||||
super(NATIVE_STARTUP_CLEANUP_PENDING);
|
||||
this.name = "NativeStartupCleanupPendingError";
|
||||
}
|
||||
}
|
||||
|
||||
type Run = typeof heartbeatRuns.$inferSelect;
|
||||
export interface NativeStartupCleanup {
|
||||
schema: "paperclip.native-startup-cleanup.v1";
|
||||
companyId: string; agentId: string; runId: string;
|
||||
contextId: string; deadlineAtMs: number; leaseOwner: string;
|
||||
controllerGeneration: number; attempt: number; leaseId: string; taskId: string;
|
||||
requiresPreparationStop: boolean; state: "pending" | "complete"; auditId: string;
|
||||
}
|
||||
|
||||
export function readNativeStartupCleanup(run: Pick<Run, "id" | "companyId" | "agentId" | "resultJson">): NativeStartupCleanup | null {
|
||||
const marker = run.resultJson?.nativeStartupCleanup as NativeStartupCleanup | undefined;
|
||||
return marker?.schema === "paperclip.native-startup-cleanup.v1" && marker.companyId === run.companyId &&
|
||||
marker.runId === run.id && marker.agentId === run.agentId && ["contextId", "leaseOwner", "leaseId", "taskId", "auditId"].every(key =>
|
||||
typeof marker[key as keyof NativeStartupCleanup] === "string" && String(marker[key as keyof NativeStartupCleanup]).length > 0) &&
|
||||
Number.isSafeInteger(marker.deadlineAtMs) && Number.isSafeInteger(marker.controllerGeneration) && marker.controllerGeneration > 0 &&
|
||||
Number.isSafeInteger(marker.attempt) && marker.attempt > 0 && typeof marker.requiresPreparationStop === "boolean" &&
|
||||
["pending", "complete"].includes(marker.state) ? marker : null;
|
||||
}
|
||||
|
||||
export function readNativeStartupContext(run: Run,
|
||||
owner: { leaseOwner: string | null; controllerGeneration: number; attempt: number; resultId: string | null }) {
|
||||
const value = run.runnerProfileJson?.nativeStartupContext as { schema?: unknown; companyId?: unknown; agentId?: unknown; runId?: unknown;
|
||||
contextId?: unknown; deadlineAtMs?: unknown; leaseId?: unknown; leaseOwner?: unknown; controllerGeneration?: unknown; attempt?: unknown } | undefined;
|
||||
if (run.runtimeMode !== "native" || run.status !== "running" || run.sessionIdAfter || owner.resultId || isNativeRunnerOwnershipHeld(run) ||
|
||||
!owner.leaseOwner || owner.controllerGeneration < 1 || owner.attempt < 1 || value?.schema !== "paperclip.native-startup-context.v1" ||
|
||||
value.companyId !== run.companyId || value.agentId !== run.agentId || value.runId !== run.id || value.leaseOwner !== owner.leaseOwner ||
|
||||
value.controllerGeneration !== owner.controllerGeneration || value.attempt !== owner.attempt ||
|
||||
typeof value.contextId !== "string" || !value.contextId || typeof value.leaseId !== "string" || !value.leaseId ||
|
||||
!Number.isSafeInteger(value.deadlineAtMs)) return null;
|
||||
return { contextId: value.contextId, leaseId: value.leaseId, deadlineAtMs: Number(value.deadlineAtMs) };
|
||||
}
|
||||
|
||||
export async function clearNativeStartupContext(db: Db, input: {
|
||||
companyId: string; runId: string; contextId: string; leaseOwner: string; controllerGeneration: number;
|
||||
}): Promise<void> {
|
||||
const cleared = await db.update(heartbeatRuns).set({
|
||||
runnerProfileJson: sql`${heartbeatRuns.runnerProfileJson} - 'nativeStartupContext'`,
|
||||
}).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId),
|
||||
eq(heartbeatRuns.status, "running"),
|
||||
sql`coalesce(${heartbeatRuns.resultJson}->'nativeStartupCleanup'->>'state', '') != 'pending'`,
|
||||
sql`${heartbeatRuns.runnerProfileJson}->'nativeStartupContext'->>'contextId' = ${input.contextId}`,
|
||||
sql`${heartbeatRuns.runnerProfileJson}->'nativeStartupContext'->>'leaseOwner' = ${input.leaseOwner}`,
|
||||
sql`${heartbeatRuns.runnerProfileJson}->'nativeStartupContext'->>'controllerGeneration' = ${String(input.controllerGeneration)}`,
|
||||
sql`exists (select 1 from ${nativeRunFinalizations} where ${nativeRunFinalizations.runId} = ${input.runId}
|
||||
and ${nativeRunFinalizations.leaseOwner} = ${input.leaseOwner}
|
||||
and ${nativeRunFinalizations.controllerGeneration} = ${input.controllerGeneration})`,
|
||||
)).returning({ id: heartbeatRuns.id });
|
||||
if (cleared.length !== 1) throw new Error("Native startup fallback lost authority");
|
||||
}
|
||||
|
||||
export async function promoteNativeStartupCleanup(db: Db, run: Run,
|
||||
owner: { leaseOwner: string | null; leaseExpiresAt: Date | null; controllerGeneration: number; attempt: number; resultId: string | null },
|
||||
now: Date): Promise<boolean> {
|
||||
const startup = readNativeStartupContext(run, owner);
|
||||
if (!startup || startup.deadlineAtMs > now.getTime() || !owner.leaseOwner || !owner.leaseExpiresAt || owner.leaseExpiresAt > now || isNativeRunnerOwnershipHeld(run)) return false;
|
||||
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.id, startup.leaseId),
|
||||
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
|
||||
sql`${environmentLeases.metadata}->>'runnerTaskLease' = 'true'`));
|
||||
if (!lease) return false;
|
||||
const prepared = lease.metadata?.runnerTaskStartup as { contextId?: unknown; runId?: unknown } | undefined;
|
||||
return requestNativeStartupCleanup(db, { companyId: run.companyId, agentId: run.agentId, runId: run.id,
|
||||
...startup, leaseOwner: owner.leaseOwner,
|
||||
requiresPreparationStop: prepared?.contextId === startup.contextId && prepared.runId === run.id }, { publish: false });
|
||||
}
|
||||
|
||||
export async function requestNativeStartupCleanup(db: Db, input: {
|
||||
companyId: string; agentId: string; runId: string; contextId: string; deadlineAtMs: number; leaseOwner: string;
|
||||
requiresPreparationStop: boolean;
|
||||
leaseId?: string;
|
||||
}, options: { publish?: boolean } = {}): Promise<boolean> {
|
||||
let publication: Parameters<typeof publishActivity>[0] | null = null;
|
||||
const [binding] = await db.select({ issueId: nativeRunFinalizations.issueId }).from(nativeRunFinalizations).where(and(
|
||||
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId)));
|
||||
if (!binding) return false;
|
||||
const requested = await db.transaction(async tx => {
|
||||
await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, binding.issueId))).for("update");
|
||||
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(
|
||||
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId))).for("update");
|
||||
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId),
|
||||
eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.agentId, input.agentId))).for("update");
|
||||
if (!run || !coordinator || run.runtimeMode !== "native" || run.status !== "running" || run.sessionIdAfter || coordinator.resultId ||
|
||||
isNativeRunnerOwnershipHeld(run)) return false;
|
||||
const existing = readNativeStartupCleanup(run);
|
||||
if (existing) return existing.contextId === input.contextId;
|
||||
if (coordinator.leaseOwner !== input.leaseOwner) return false;
|
||||
const leases = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
||||
eq(environmentLeases.heartbeatRunId, input.runId), input.leaseId ? eq(environmentLeases.id, input.leaseId) : undefined,
|
||||
sql`${environmentLeases.metadata}->>'runnerTaskLease' = 'true'`)).for("update");
|
||||
if (leases.length !== 1 || !leases[0]!.providerLeaseId) throw new Error("Native startup cleanup lease unavailable");
|
||||
const lease = leases[0]!;
|
||||
const activity = await persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system",
|
||||
actorId: "native-startup-cleanup", action: "native.startup_cleanup_requested", entityType: "heartbeat_run", entityId: run.id,
|
||||
agentId: run.agentId, runId: run.id, issueId: run.nativeIssueId ?? undefined,
|
||||
details: { contextId: input.contextId, deadlineAtMs: input.deadlineAtMs, leaseId: lease.id,
|
||||
controllerGeneration: coordinator.controllerGeneration, reason: "startup_worker_unsettled" } });
|
||||
if (!activity.activity?.id) throw new Error("Native startup cleanup audit unavailable");
|
||||
publication = activity.publication;
|
||||
const preparation = lease.metadata?.runnerTaskStartup as { contextId?: unknown; runId?: unknown } | undefined;
|
||||
const marker: NativeStartupCleanup = { schema: "paperclip.native-startup-cleanup.v1", companyId: run.companyId, agentId: run.agentId,
|
||||
runId: run.id, contextId: input.contextId, deadlineAtMs: input.deadlineAtMs, leaseOwner: input.leaseOwner,
|
||||
controllerGeneration: coordinator.controllerGeneration, attempt: coordinator.attempt, leaseId: lease.id,
|
||||
taskId: lease.providerLeaseId!, requiresPreparationStop: input.requiresPreparationStop ||
|
||||
(preparation?.contextId === input.contextId && preparation.runId === run.id), state: "pending", auditId: activity.activity.id };
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization);
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.environmentCleanup);
|
||||
await tx.update(heartbeatRuns).set({ errorCode: NATIVE_STARTUP_CLEANUP_PENDING,
|
||||
error: "Native startup timed out. Cancellation and verified cleanup are pending.",
|
||||
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ nativeStartupCleanup: marker,
|
||||
startupCancellation: { ...run.resultJson?.startupCancellation as object, requestedAt: new Date().toISOString() } })}::jsonb`,
|
||||
updatedAt: new Date() }).where(eq(heartbeatRuns.id, run.id));
|
||||
await tx.update(nativeRunFinalizations).set({ phase: "terminal_failure", failureCode: NATIVE_STARTUP_CLEANUP_PENDING,
|
||||
nextAttemptAt: null, controlDeadlineAt: new Date(), recoveryState: "blocked", updatedAt: new Date() })
|
||||
.where(eq(nativeRunFinalizations.runId, run.id));
|
||||
await tx.update(environmentLeases).set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({
|
||||
runnerTaskRetiring: true, runnerTaskStartupCleanup: { contextId: marker.contextId, requiresPreparationStop: marker.requiresPreparationStop },
|
||||
})}::jsonb`, updatedAt: new Date() }).where(eq(environmentLeases.id, lease.id));
|
||||
return true;
|
||||
});
|
||||
if (publication && options.publish !== false) publishActivity(publication);
|
||||
return requested;
|
||||
}
|
||||
|
||||
async function assertStartupCreationsStopped(db: Db, marker: NativeStartupCleanup): Promise<void> {
|
||||
const creations = await db.select({ creation: sandboxCreationOperations, lease: environmentLeases }).from(sandboxCreationOperations)
|
||||
.leftJoin(environmentLeases, eq(environmentLeases.id, sandboxCreationOperations.leaseId)).where(and(
|
||||
eq(sandboxCreationOperations.companyId, marker.companyId), eq(sandboxCreationOperations.runId, marker.runId),
|
||||
sql`${sandboxCreationOperations.metadata}->>'runnerTaskLease' = 'true'`));
|
||||
if (creations.some(({ creation, lease }) => creation.state !== "attached" &&
|
||||
(!["cleanup_pending", "resolved"].includes(creation.state) || !lease || lease.companyId !== marker.companyId || lease.heartbeatRunId !== marker.runId ||
|
||||
lease.providerLeaseId !== creation.metadata.creationProviderLeaseId || lease.cleanupStatus !== "success" || !hasRemoteTerminationReceipt(lease)))) {
|
||||
throw conflict("Task cleanup is still pending", { code: NATIVE_STARTUP_CLEANUP_PENDING });
|
||||
}
|
||||
}
|
||||
|
||||
export async function verifyNativeStartupCleanup(db: Db, run: Run, release: () => Promise<unknown>): Promise<void> {
|
||||
const marker = readNativeStartupCleanup(run);
|
||||
if (!marker || isNativeRunnerOwnershipHeld(run) || !["running", "cancelled"].includes(run.status)) throw conflict("Native startup cleanup intent unavailable");
|
||||
const read = async () => {
|
||||
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.id, marker.leaseId)));
|
||||
if (!lease || lease.heartbeatRunId !== run.id || lease.providerLeaseId !== marker.taskId) throw conflict("Native startup cleanup target changed");
|
||||
return lease;
|
||||
};
|
||||
const [owner] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
if (!owner || owner.controllerGeneration !== marker.controllerGeneration || owner.attempt !== marker.attempt ||
|
||||
owner.leaseOwner !== marker.leaseOwner || owner.resultId) throw conflict("Native startup cleanup ownership changed");
|
||||
let lease = await read();
|
||||
const stopped = (value: typeof lease) => hasRemoteTerminationReceipt(value) && value.cleanupStatus === "success" &&
|
||||
(!marker.requiresPreparationStop || value.metadata?.runnerTaskPreparationStopped === true);
|
||||
if (!stopped(lease)) { await release(); lease = await read(); }
|
||||
if (!stopped(lease)) throw conflict("Task cleanup is still pending", { code: NATIVE_STARTUP_CLEANUP_PENDING });
|
||||
await assertStartupCreationsStopped(db, marker);
|
||||
await db.transaction(async tx => {
|
||||
await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, run.companyId), eq(issues.id, run.nativeIssueId!))).for("update");
|
||||
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)).for("update");
|
||||
const [current] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)).for("update");
|
||||
const observed = current && readNativeStartupCleanup(current);
|
||||
const [receipt] = await tx.select().from(environmentLeases).where(eq(environmentLeases.id, marker.leaseId)).for("update");
|
||||
if (!coordinator || !current || isNativeRunnerOwnershipHeld(current) || coordinator.resultId || coordinator.controllerGeneration !== marker.controllerGeneration ||
|
||||
coordinator.attempt !== marker.attempt || coordinator.leaseOwner !== marker.leaseOwner || observed?.contextId !== marker.contextId ||
|
||||
!receipt || receipt.providerLeaseId !== marker.taskId || receipt.heartbeatRunId !== run.id || !stopped(receipt)) {
|
||||
throw conflict("Native startup cleanup ownership changed");
|
||||
}
|
||||
await assertStartupCreationsStopped(tx as unknown as Db, marker);
|
||||
await tx.update(heartbeatRuns).set({ resultJson: sql`jsonb_set(${heartbeatRuns.resultJson}, '{nativeStartupCleanup,state}', '"complete"'::jsonb)`, updatedAt: new Date() })
|
||||
.where(eq(heartbeatRuns.id, run.id));
|
||||
});
|
||||
}
|
||||
|
||||
export async function acknowledgeNativeStartupCleanup(db: Db, input: { companyId: string; runId: string }): Promise<void> {
|
||||
const [binding] = await db.select({ issueId: nativeRunFinalizations.issueId }).from(nativeRunFinalizations).where(and(
|
||||
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId)));
|
||||
if (!binding) return;
|
||||
await db.transaction(async tx => {
|
||||
await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, binding.issueId))).for("update");
|
||||
const [owner] = await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, input.runId)).for("update");
|
||||
const [run] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)).for("update");
|
||||
const marker = run && readNativeStartupCleanup(run);
|
||||
if (!owner || !run || run.status !== "cancelled" || !marker || marker.state !== "complete" || owner.resultId ||
|
||||
owner.failureCode !== NATIVE_STARTUP_CLEANUP_PENDING || owner.controllerGeneration !== marker.controllerGeneration ||
|
||||
owner.attempt !== marker.attempt || owner.leaseOwner !== marker.leaseOwner) return;
|
||||
const [lease] = await tx.select().from(environmentLeases).where(eq(environmentLeases.id, marker.leaseId)).for("update");
|
||||
if (!lease || lease.heartbeatRunId !== run.id || lease.providerLeaseId !== marker.taskId || !hasRemoteTerminationReceipt(lease) ||
|
||||
(marker.requiresPreparationStop && lease.metadata?.runnerTaskPreparationStopped !== true)) return;
|
||||
await assertStartupCreationsStopped(tx as unknown as Db, marker);
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle);
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.environmentCleanup);
|
||||
await tx.update(nativeRunFinalizations).set({ leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null, recoveryState: null,
|
||||
failureCode: "native_retry_cancelled", updatedAt: new Date() }).where(and(eq(nativeRunFinalizations.runId, input.runId), isNull(nativeRunFinalizations.resultId)));
|
||||
});
|
||||
}
|
||||
@@ -725,6 +725,7 @@ export interface PluginWorkerHandle {
|
||||
executeLogSink?: ExecuteLogSink,
|
||||
onWorkerSettled?: () => void,
|
||||
startupContextId?: string,
|
||||
observeResult?: (result: HostToWorkerMethods[M][1]) => Promise<void>,
|
||||
): Promise<HostToWorkerMethods[M][1]>;
|
||||
|
||||
/**
|
||||
@@ -865,6 +866,7 @@ export interface PluginWorkerManager {
|
||||
executeLogSink?: ExecuteLogSink,
|
||||
onWorkerSettled?: () => void,
|
||||
startupContextId?: string,
|
||||
observeResult?: (result: HostToWorkerMethods[M][1]) => Promise<void>,
|
||||
): Promise<HostToWorkerMethods[M][1]>;
|
||||
|
||||
/**
|
||||
@@ -914,6 +916,10 @@ export function createPluginWorkerHandle(
|
||||
// Unlike RPC promises, these survive timeouts. A late response is the
|
||||
// completion receipt. A worker crash cannot manufacture one.
|
||||
const unsettledCalls = new Map<string | number, () => void>();
|
||||
const observedResults = new Map<string | number, {
|
||||
contextId: string; response?: JsonRpcResponse; running?: Promise<void>;
|
||||
observe: (result: unknown) => Promise<void>; finish: () => void;
|
||||
}>();
|
||||
const startupCalls = new Map<string, Set<() => void>>();
|
||||
const startupRetirements = new Map<string, Promise<void>>();
|
||||
const startupHostHandlers = new Map<string, Set<Promise<void>>>();
|
||||
@@ -1156,24 +1162,50 @@ export function createPluginWorkerHandle(
|
||||
/**
|
||||
* Handle a JSON-RPC response from the worker (matching a pending request).
|
||||
*/
|
||||
function settleResponse(response: JsonRpcResponse): void {
|
||||
const id = response.id!;
|
||||
unsettledCalls.get(id)?.();
|
||||
unsettledCalls.delete(id);
|
||||
const pending = pendingRequests.get(id);
|
||||
if (!pending) { log.warn({ id }, "received response for unknown request id"); return; }
|
||||
clearTimeout(pending.timer);
|
||||
pendingRequests.delete(id);
|
||||
pending.resolve(response);
|
||||
}
|
||||
|
||||
function processObservedResult(id: string | number): Promise<void> {
|
||||
const observer = observedResults.get(id);
|
||||
if (!observer?.response) return Promise.resolve();
|
||||
if (observer.running) return observer.running;
|
||||
const response = observer.response;
|
||||
const processing = (async () => {
|
||||
if (isJsonRpcSuccessResponse(response)) await observer.observe(response.result);
|
||||
observedResults.delete(id);
|
||||
observer.finish();
|
||||
settleResponse(response);
|
||||
})();
|
||||
observer.running = processing;
|
||||
void processing.catch(() => {
|
||||
const pending = pendingRequests.get(id);
|
||||
pending?.resolve(createErrorResponse(id, JSONRPC_ERROR_CODES.INTERNAL_ERROR, "Plugin result could not be recorded"));
|
||||
}).finally(() => { if (observer.running === processing) observer.running = undefined; });
|
||||
return processing;
|
||||
}
|
||||
|
||||
function handleResponse(response: JsonRpcResponse): void {
|
||||
const id = response.id;
|
||||
if (id === null || id === undefined) {
|
||||
log.warn("received response with null/undefined id");
|
||||
return;
|
||||
}
|
||||
|
||||
unsettledCalls.get(id)?.();
|
||||
unsettledCalls.delete(id);
|
||||
const pending = pendingRequests.get(id);
|
||||
if (!pending) {
|
||||
log.warn({ id }, "received response for unknown request id");
|
||||
const observer = observedResults.get(id);
|
||||
if (observer) {
|
||||
if (observer.response) return;
|
||||
observer.response = response;
|
||||
void processObservedResult(id).catch(() => log.warn({ id }, "plugin result recording will retry during cleanup"));
|
||||
return;
|
||||
}
|
||||
|
||||
clearTimeout(pending.timer);
|
||||
pendingRequests.delete(id);
|
||||
pending.resolve(response);
|
||||
settleResponse(response);
|
||||
}
|
||||
|
||||
function readNonEmptyString(value: unknown): string | null {
|
||||
@@ -3087,6 +3119,11 @@ export function createPluginWorkerHandle(
|
||||
// Read-only requests cannot continue in an exited worker. Do not infer
|
||||
// completion for execute/cleanup calls, whose provider work may outlive it.
|
||||
for (const finish of [...unsettledInspections]) finish();
|
||||
for (const [id, observer] of observedResults) {
|
||||
if (observer.response) continue;
|
||||
observedResults.delete(id);
|
||||
observer.finish();
|
||||
}
|
||||
|
||||
// Close the one login pseudo-terminal route with a fixed non-secret exit and
|
||||
// clear the route one time. The pending pseudo-terminal calls
|
||||
@@ -3423,8 +3460,9 @@ export function createPluginWorkerHandle(
|
||||
executeLogSink?: ExecuteLogSink,
|
||||
onWorkerSettled?: () => void,
|
||||
startupContextId?: string,
|
||||
observeResult?: (result: HostToWorkerMethods[M][1]) => Promise<void>,
|
||||
): Promise<HostToWorkerMethods[M][1]> {
|
||||
const dispatched = measureRunPhase("execution.plugin_dispatch", () => callInternalRaw(method, params, timeoutMs, executeLogSink, onWorkerSettled, startupContextId));
|
||||
const dispatched = measureRunPhase("execution.plugin_dispatch", () => callInternalRaw(method, params, timeoutMs, executeLogSink, onWorkerSettled, startupContextId, observeResult));
|
||||
// Tracing wraps the RPC in a new promise. Guard the promise handed to the
|
||||
// caller as well, without swallowing its eventual rejection for that caller.
|
||||
void dispatched.catch(() => undefined);
|
||||
@@ -3438,6 +3476,7 @@ export function createPluginWorkerHandle(
|
||||
executeLogSink?: ExecuteLogSink,
|
||||
onWorkerSettled?: () => void,
|
||||
startupContextId?: string,
|
||||
observeResult?: (result: HostToWorkerMethods[M][1]) => Promise<void>,
|
||||
): Promise<HostToWorkerMethods[M][1]> {
|
||||
const rpcPromise = new Promise<HostToWorkerMethods[M][1]>((resolve, reject) => {
|
||||
if (startupContextId && startupRetirements.has(startupContextId)) {
|
||||
@@ -3484,6 +3523,19 @@ export function createPluginWorkerHandle(
|
||||
}
|
||||
if (method === "environmentInspectLease") unsettledInspections.add(finish);
|
||||
}
|
||||
if (observeResult && startupContextId) {
|
||||
let finish!: () => void;
|
||||
const processing = new Promise<void>(resolve => { finish = resolve; });
|
||||
let handlers = startupHostHandlers.get(startupContextId);
|
||||
if (!handlers) { handlers = new Set(); startupHostHandlers.set(startupContextId, handlers); }
|
||||
handlers.add(processing);
|
||||
observedResults.set(id, { contextId: startupContextId, observe: observeResult as (result: unknown) => Promise<void>, finish: () => {
|
||||
startupHostHandlers.get(startupContextId)?.delete(processing);
|
||||
if (startupHostHandlers.get(startupContextId)?.size === 0) startupHostHandlers.delete(startupContextId);
|
||||
finish();
|
||||
onStartupWorkChanged?.();
|
||||
} });
|
||||
}
|
||||
const timeout = resolveRpcCallTimeoutMs(timeoutMs, rpcTimeoutMs);
|
||||
const invocationScope = deriveInvocationScope(method, params);
|
||||
const invocation = invocationScope ? registerInvocation(invocationScope) : null;
|
||||
@@ -3550,6 +3602,8 @@ export function createPluginWorkerHandle(
|
||||
} catch (err) {
|
||||
clearTimeout(timer);
|
||||
pendingRequests.delete(id);
|
||||
observedResults.get(id)?.finish();
|
||||
observedResults.delete(id);
|
||||
unsettledCalls.get(id)?.();
|
||||
unsettledCalls.delete(id);
|
||||
clearInvocation(invocation);
|
||||
@@ -3637,6 +3691,7 @@ export function createPluginWorkerHandle(
|
||||
hasStartupWork() { return startupCalls.size > 0 || startupHostHandlers.size > 0 || startupRetirements.size > 0; },
|
||||
async drainStartup(contextId: string) {
|
||||
revokeStartupInvocations(contextId);
|
||||
for (const [id, observer] of observedResults) if (observer.contextId === contextId) void processObservedResult(id).catch(() => undefined);
|
||||
await waitForStartupHandlers(contextId);
|
||||
},
|
||||
async retireStartup(contextId: string) {
|
||||
@@ -3644,6 +3699,7 @@ export function createPluginWorkerHandle(
|
||||
if (current) return current;
|
||||
const retiring = (async () => {
|
||||
revokeStartupInvocations(contextId);
|
||||
for (const [id, observer] of observedResults) if (observer.contextId === contextId) void processObservedResult(id).catch(() => undefined);
|
||||
await waitForStartupHandlers(contextId);
|
||||
for (const finish of [...startupCalls.get(contextId) ?? []]) finish();
|
||||
})();
|
||||
@@ -3662,6 +3718,7 @@ export function createPluginWorkerHandle(
|
||||
executeLogSink?: ExecuteLogSink,
|
||||
onWorkerSettled?: () => void,
|
||||
startupContextId?: string,
|
||||
observeResult?: (result: HostToWorkerMethods[M][1]) => Promise<void>,
|
||||
): Promise<HostToWorkerMethods[M][1]> {
|
||||
if (status !== "running" && status !== "starting") {
|
||||
onWorkerSettled?.();
|
||||
@@ -3671,7 +3728,7 @@ export function createPluginWorkerHandle(
|
||||
),
|
||||
);
|
||||
}
|
||||
return callInternal(method, params, timeoutMs, executeLogSink, onWorkerSettled, startupContextId);
|
||||
return callInternal(method, params, timeoutMs, executeLogSink, onWorkerSettled, startupContextId, observeResult);
|
||||
},
|
||||
|
||||
openLoginPtySession(input: LoginPtyOpenInput) {
|
||||
@@ -4032,6 +4089,7 @@ export function createPluginWorkerManager(
|
||||
executeLogSink?: ExecuteLogSink,
|
||||
onWorkerSettled?: () => void,
|
||||
startupContextId?: string,
|
||||
observeResult?: (result: HostToWorkerMethods[M][1]) => Promise<void>,
|
||||
): Promise<HostToWorkerMethods[M][1]> {
|
||||
const handle = workers.get(pluginId);
|
||||
if (!startupContextId && ["environmentReleaseLease", "environmentDestroyLease"].includes(method)) {
|
||||
@@ -4057,7 +4115,7 @@ export function createPluginWorkerManager(
|
||||
// A caller timeout is not worker completion. Inspection capacity is
|
||||
// released only by its actual reply, send failure, or worker exit.
|
||||
const finish = onWorkerSettled ? () => { finishInspection?.(); onWorkerSettled(); } : finishInspection;
|
||||
const result = handle.call(method, params, timeoutMs, executeLogSink, finish, startupContextId);
|
||||
const result = handle.call(method, params, timeoutMs, executeLogSink, finish, startupContextId, observeResult);
|
||||
if (method === "configChanged") void result.then(() => {
|
||||
for (const listener of configApplied) {
|
||||
try { listener(); }
|
||||
|
||||
@@ -4,9 +4,9 @@ import { and, eq, lte, or, sql } from "drizzle-orm";
|
||||
import { companies, environments, sandboxCreationOperations, type Db } from "@paperclipai/db";
|
||||
import { readEnvironmentCreationCleanupError, type PluginEnvironmentLease, type PluginEnvironmentLeaseObservation } from "@paperclipai/plugin-sdk";
|
||||
import { assertEnvironmentCompanyBinding, environmentService } from "./environments.js";
|
||||
import { ENVIRONMENT_CLEANUP_RETRY_MS } from "./environment-cleanup-state.js";
|
||||
import { ENVIRONMENT_CLEANUP_RETRY_MS, recoverableSandboxCreationCondition } from "./environment-cleanup-state.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { notifyEnvironmentCleanup, writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
|
||||
export type SandboxCreationOperation = typeof sandboxCreationOperations.$inferSelect;
|
||||
|
||||
@@ -78,26 +78,69 @@ export async function createOwnedSandbox(
|
||||
* This enriches the existing owner; it never transfers or reopens ownership. */
|
||||
export async function recordSandboxCreationResource(db: Db, input: {
|
||||
id: string; companyId: string; environmentId: string; providerLeaseId: string;
|
||||
providerMetadata?: Record<string, unknown>;
|
||||
}): Promise<void> {
|
||||
const [recorded] = await db.update(sandboxCreationOperations).set({
|
||||
metadata: sql`coalesce(${sandboxCreationOperations.metadata}, '{}'::jsonb) || ${JSON.stringify({ creationProviderLeaseId: input.providerLeaseId })}::jsonb`,
|
||||
updatedAt: new Date(),
|
||||
}).where(and(eq(sandboxCreationOperations.id, input.id), eq(sandboxCreationOperations.companyId, input.companyId),
|
||||
eq(sandboxCreationOperations.originalEnvironmentId, input.environmentId),
|
||||
or(sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' is null`,
|
||||
sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' = ${input.providerLeaseId}`),
|
||||
)).returning({ id: sandboxCreationOperations.id });
|
||||
if (!recorded) throw new Error("Sandbox creation resource identity changed");
|
||||
await writeEnvironmentCleanup(db, async tx => {
|
||||
const [recorded] = await tx.update(sandboxCreationOperations).set({
|
||||
nextAttemptAt: sql`case when ${sandboxCreationOperations.state} = 'needs_reconciliation' then clock_timestamp() else ${sandboxCreationOperations.nextAttemptAt} end`,
|
||||
metadata: sql`coalesce(${sandboxCreationOperations.metadata}, '{}'::jsonb) || ${JSON.stringify({ creationProviderLeaseId: input.providerLeaseId, ...(input.providerMetadata ? { creationProviderMetadata: input.providerMetadata } : {}) })}::jsonb`,
|
||||
updatedAt: new Date(),
|
||||
}).where(and(eq(sandboxCreationOperations.id, input.id), eq(sandboxCreationOperations.companyId, input.companyId),
|
||||
eq(sandboxCreationOperations.originalEnvironmentId, input.environmentId),
|
||||
or(sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' is null`,
|
||||
sql`${sandboxCreationOperations.metadata}->>'creationProviderLeaseId' = ${input.providerLeaseId}`),
|
||||
)).returning({ id: sandboxCreationOperations.id });
|
||||
if (!recorded) throw new Error("Sandbox creation resource identity changed");
|
||||
}, { companyId: input.companyId });
|
||||
}
|
||||
|
||||
export async function queueSandboxCreationCleanup(db: Db, input: {
|
||||
id: string; companyId: string; environmentId: string; providerLeaseId: string;
|
||||
providerMetadata?: Record<string, unknown>;
|
||||
}): Promise<void> {
|
||||
await db.transaction(async tx => {
|
||||
await tx.select({ id: environments.id }).from(environments).where(eq(environments.id, input.environmentId)).for("update");
|
||||
const [operation] = await tx.select().from(sandboxCreationOperations).where(and(eq(sandboxCreationOperations.id, input.id),
|
||||
eq(sandboxCreationOperations.companyId, input.companyId), eq(sandboxCreationOperations.originalEnvironmentId, input.environmentId))).for("update");
|
||||
if (!operation || (operation.metadata.creationProviderLeaseId && operation.metadata.creationProviderLeaseId !== input.providerLeaseId)) {
|
||||
throw new Error("Sandbox creation cleanup ownership changed");
|
||||
}
|
||||
if (["attached", "cleanup_pending", "resolved"].includes(operation.state)) return;
|
||||
const contextId = operation.metadata.runnerTaskLease === true && typeof operation.metadata.runnerTaskCreationContextId === "string"
|
||||
? operation.metadata.runnerTaskCreationContextId : null;
|
||||
if (operation.state !== "pending" && !(operation.state === "needs_reconciliation" && contextId &&
|
||||
operation.metadata.creationProviderLeaseId === input.providerLeaseId)) throw new Error("Sandbox creation cleanup ownership changed");
|
||||
await notifyEnvironmentCleanup(tx, { companyId: operation.companyId, runId: operation.runId });
|
||||
const lease = await environmentService(tx as unknown as Db).insertPendingCleanupLease({
|
||||
companyId: operation.companyId, environmentId: operation.originalEnvironmentId,
|
||||
heartbeatRunId: operation.runId, issueId: operation.issueId, executionWorkspaceId: operation.executionWorkspaceId,
|
||||
provider: operation.provider, providerLeaseId: input.providerLeaseId,
|
||||
metadata: { ...operation.metadata, creationOperationId: operation.id, environmentId: operation.originalEnvironmentId,
|
||||
providerMetadata: input.providerMetadata ?? operation.metadata.creationProviderMetadata ?? {}, runnerTaskRetiring: true,
|
||||
...(contextId ? { runnerTaskStartupCleanup: { contextId, requiresPreparationStop: false } } : {}) },
|
||||
failureReason: "interrupted_task_creation",
|
||||
});
|
||||
await tx.update(sandboxCreationOperations).set({ state: "cleanup_pending", leaseId: lease.id, updatedAt: new Date(),
|
||||
metadata: { ...operation.metadata, creationProviderLeaseId: input.providerLeaseId,
|
||||
creationProviderMetadata: input.providerMetadata ?? operation.metadata.creationProviderMetadata ?? {} },
|
||||
}).where(eq(sandboxCreationOperations.id, operation.id));
|
||||
});
|
||||
}
|
||||
|
||||
export async function recoverSandboxCreations(db: Db,
|
||||
inspect: (operation: SandboxCreationOperation) => Promise<PluginEnvironmentLeaseObservation | null>,
|
||||
): Promise<void> {
|
||||
const due = await db.select().from(sandboxCreationOperations).where(and(
|
||||
eq(sandboxCreationOperations.state, "pending"), lte(sandboxCreationOperations.nextAttemptAt, new Date()),
|
||||
recoverableSandboxCreationCondition(), lte(sandboxCreationOperations.nextAttemptAt, new Date()),
|
||||
)).limit(100);
|
||||
for (const candidate of due) {
|
||||
try {
|
||||
if (candidate.metadata.runnerTaskLease === true && typeof candidate.metadata.runnerTaskCreationContextId === "string" &&
|
||||
typeof candidate.metadata.creationProviderLeaseId === "string") {
|
||||
await queueSandboxCreationCleanup(db, { id: candidate.id, companyId: candidate.companyId,
|
||||
environmentId: candidate.originalEnvironmentId, providerLeaseId: candidate.metadata.creationProviderLeaseId });
|
||||
continue;
|
||||
}
|
||||
const [existingCompany] = await db.select({ id: companies.id }).from(companies).where(eq(companies.id, candidate.companyId)).limit(1);
|
||||
const observation = existingCompany ? await inspect(candidate) : null;
|
||||
const cleanup = observation?.creationCleanup ? readEnvironmentCreationCleanupError({ data: {
|
||||
@@ -140,7 +183,7 @@ export async function recoverSandboxCreations(db: Db,
|
||||
? { nextAttemptAt: new Date(Date.now() + ENVIRONMENT_CLEANUP_RETRY_MS), updatedAt: new Date() }
|
||||
: { state: "needs_reconciliation", updatedAt: new Date(),
|
||||
metadata: sql`coalesce(${sandboxCreationOperations.metadata}, '{}'::jsonb) || '{"creationReconciliationReason":"company_deleted"}'::jsonb` })
|
||||
.where(and(eq(sandboxCreationOperations.id, candidate.id), eq(sandboxCreationOperations.state, "pending")));
|
||||
.where(and(eq(sandboxCreationOperations.id, candidate.id), recoverableSandboxCreationCondition()));
|
||||
return Boolean(company);
|
||||
});
|
||||
logger.warn({ errorKind: retry ? "sandbox_creation_recovery_failed" : "sandbox_creation_needs_reconciliation", operationId: candidate.id, companyId: candidate.companyId },
|
||||
|
||||
Reference in new issue
Block a user