mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(native): let the outgoing server checkpoint its idle warm session while the incoming follow-up waits for it (#21791)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap. A warm agent session stays with the outgoing server that ran its last turn. > - Since #21774, a follow-up that the incoming server claims during the overlap waits (`native_controller_recovery_waiting_for_owner`) and checks again every 5 s. > - About 1 s after the traffic switch (#21627), the outgoing server tries to save and close its idle warm session. The checkpoint fence refuses that save with `native_runner_checkpoint_owner_lost`. The task's current run is the parked follow-up. That run has its own finalization row under another controller boot, and the fence does not accept that kind of successor. > - The close retry from #21782 fails the same way each time. So the follow-up waits until the outgoing server exits, or until the predecessor's finish time plus the idle window plus 2 min. In the CHA-5 case that was about 225 s, and the worst case is about 425 s. Speed is a requirement. > - This pull request records which owner a parked follow-up waits on, and lets the fence accept the outgoing owner's final save when the follow-up is parked on exactly that owner and has not started. > - The benefit is that the outgoing server saves and closes the session on its first attempt, and the follow-up continues at its next 5 s check. ## Linked Issues or Issue Description Refs #21774, #21782, #21697, #21627, #21279 **What happened?** During a release-swap overlap, a warm follow-up was claimed by the incoming server while the outgoing server still owned the idle warm session. The follow-up waited correctly. The outgoing server's idle close failed with `native_runner_checkpoint_owner_lost` on every attempt. The wait ended only when the outgoing server exited, about 225 s after the follow-up was created. **Expected behavior** The outgoing server saves and closes the session while the follow-up waits. The follow-up sees the saved session and continues within one 5 s check. **Steps to reproduce** 1. Run a warm native session on a remote sandbox and let a run finish on server A. 2. Start server B with an overlap and switch traffic to it. 3. Post a follow-up comment so that B claims the follow-up while A is still alive. 4. A's idle close fails with `native_runner_checkpoint_owner_lost`, and the follow-up waits until A exits. **Paperclip version or commit** master `d14a3b88b1`. ## What Changed - `NativeControllerRecoveryWaitingError` carries the owner that the follow-up waits on: the predecessor run and its controller boot. `waitForLiveWarmSessionOwner` and the live-owner wait in `inspectRemoteWarmSessionRecovery` set it. - `deferNativeControllerRecovery` stores that owner in the finalization `failureDetail.waitingForOwner`. The executor passes it through. - `validateNativeRunnerCheckpointOwner` accepts one more successor kind: a parked follow-up waiting on this exact owner. All of these must be true, or the fence still refuses: - The checkpoint holds an accepted result and has no new work (unchanged rule). - The follow-up finalization is `retryable_failure` with `native_controller_recovery_waiting_for_owner`, `attempt = 0`, no result, and no lease owner (no claim is running it). - `waitingForOwner` names this owner's run and controller boot. - The follow-up run is a running native run for the same company and task, on the same native session as the owner's run. - The follow-up has no process fields, no own process-start events, and no checkpoint rows. - The fence locks the follow-up's finalization row (`for share`), so a resumption claim and the save cannot interleave. - Tests in `native-controller-recovery.integration.test.ts` (real Postgres): - The outgoing server saves and closes its idle warm session while the incoming follow-up waits, and the follow-up resumes within one recheck. - Negative cases: the save is still refused when the follow-up has its own process, its own process-start event, its own checkpoint, or a live resumption claim, or when it waits on a different owner boot or run. ## Verification - Red on master: with the `native-runner-checkpoint.ts` change removed, the scenario test fails with `native_runner_checkpoint_owner_lost` at the outgoing server's first close attempt. The follow-up keeps waiting. Green with the change. - The negative cases pass with and without the change. - Measured in the test scenario (warm idle window 600 s): - Before: the follow-up waits until the outgoing server exits, or about 720 s (finish + 600 s idle window + 120 s margin). - After: the outgoing save and close succeed on the first attempt. The follow-up is claimable and its wait resolves at `retryAt`, 4998 ms after it parked. - `vitest run` of `native-controller-recovery.integration.test.ts`, `native-runner-restart-recovery.integration.test.ts` and `native-harness-backup-storage.integration.test.ts`: 349 passed, 14 skipped. - `vitest run` of `native-session-executor.test.ts`, `remote-warm-session-recovery.test.ts`, `native-runner-checkpoint-save.test.ts`, `native-runner-checkpoint-retirement.test.ts` and `native-runner-checkpoint-authority.integration.test.ts`: 1566 passed, 7 skipped. - `tsc --noEmit` for `server` passes. - `pnpm verify` exits 2 (opaque setup dependency). Required CI supplies the broad coverage. - Not run: a release-swap wave. The fix is not yet measured on a live overlap. ## Risks - The fence is wider. It accepts only a final save with an accepted result and no new work, and only while the current run is parked on this exact owner and has not started. A follow-up that has a claim, a process, process-start events or checkpoint rows still causes a refusal. - A follow-up that a server parked before this change has no `waitingForOwner`. The fence refuses its owner's save as before. At its next recheck, the follow-up parks again with the owner recorded. - While a resumption claim runs the follow-up's recheck (a short window every 5 s), the save is refused. The close retry from #21782 then tries again. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
cec8c2b3f4
commit
df9914dbd9
5 files changed
+175
-15
No files matched your search
@@ -8,14 +8,15 @@ import { createHash, randomUUID } from "node:crypto";
|
||||
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agentTaskSessions, agents, companies, controllerInstances, createDb, environmentLeases, environments, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, nativeRunnerCheckpoints } from "@paperclipai/db";
|
||||
import { DurablePrpControlPlane, parseNativeExecutionInput, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js";
|
||||
import { startRecoveryPostgres, recoveryPostgresSupport } from "../../__tests__/helpers/recovery-postgres.js";
|
||||
import { createIsolatedTestHome } from "../../__tests__/helpers/isolated-test-home.js";
|
||||
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
||||
import { nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js";
|
||||
import { assertNativeRunnerCheckpointOwner, loadNativeRunnerCheckpoint, nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js";
|
||||
import { assertNativeHarnessBackupOwner, nativeHarnessBackupOwner } from "./native-harness-backup-storage.js";
|
||||
import { assertNativeRetainedSandboxBinding, createNativeControllerRecovery, deferNativeControllerRecovery, deferNativeRunForBusyAgentDirectory, NATIVE_AGENT_DIRECTORY_BUSY_WAIT, importNativeControllerRecovery, NativeControllerRecoveryWaitingError, NATIVE_RETAINED_OWNER_WAIT, resolveNativeControllerRecovery } from "./native-controller-recovery.js";
|
||||
import { claimNativeSessionResumptions } from "./native-finalization-reconciler.js";
|
||||
import { claimNativeRestartRecoveries, currentNativeControllerIdentity } from "./native-restart-recovery.js";
|
||||
@@ -27,6 +28,7 @@ import { inspectRemoteWarmSessionRecovery, waitForLiveWarmSessionOwner } from ".
|
||||
import { currentProcessSpaceHash } from "../controller-process-identity.js";
|
||||
import { registerControllerInstanceDb, resetControllerInstanceLeaseForTests } from "../controller-instance-lease.js";
|
||||
import { readProcessStartedAt } from "../hot-restart.js";
|
||||
import { PROCESS_START_REQUESTED } from "../native-process-start-events.js";
|
||||
|
||||
describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable discovery and import", () => {
|
||||
let database: Awaited<ReturnType<typeof startRecoveryPostgres>>;
|
||||
@@ -610,17 +612,20 @@ describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable d
|
||||
await new Promise(resolve => oldServer.once("spawn", resolve));
|
||||
const startedAt = await readProcessStartedAt(oldServer.pid!);
|
||||
expect(startedAt).toBeTruthy();
|
||||
const { inspect, recordOwner, runId, current } = await seedWarmFollowUpOnOldServer();
|
||||
await recordOwner({ controllerBootId: randomUUID(), controllerPid: oldServer.pid!, controllerHostHash: currentProcessSpaceHash(),
|
||||
const { inspect, recordOwner, runId, current, fixture } = await seedWarmFollowUpOnOldServer();
|
||||
const outgoingBoot = randomUUID();
|
||||
await recordOwner({ controllerBootId: outgoingBoot, controllerPid: oldServer.pid!, controllerHostHash: currentProcessSpaceHash(),
|
||||
controllerProcessStartedAt: new Date(startedAt!), leaseOwner: null, leaseExpiresAt: null });
|
||||
const before = Date.now();
|
||||
const waiting = await inspect().catch(error => error);
|
||||
expect(waiting).toBeInstanceOf(NativeControllerRecoveryWaitingError);
|
||||
expect(waiting.owner).toEqual({ runId: fixture.scope.runId, controllerBootId: outgoingBoot });
|
||||
expect(waiting.retryAt.getTime()).toBeGreaterThan(before);
|
||||
expect(waiting.retryAt.getTime()).toBeLessThanOrEqual(Date.now() + 10_000);
|
||||
await deferNativeControllerRecovery(db, { ...current.binding, retryAt: waiting.retryAt });
|
||||
await deferNativeControllerRecovery(db, { ...current.binding, retryAt: waiting.retryAt, waitingForOwner: waiting.owner });
|
||||
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0])
|
||||
.toMatchObject({ phase: "retryable_failure", attempt: 0, failureCode: NATIVE_RETAINED_OWNER_WAIT, resultId: null });
|
||||
.toMatchObject({ phase: "retryable_failure", attempt: 0, failureCode: NATIVE_RETAINED_OWNER_WAIT, resultId: null,
|
||||
failureDetail: expect.objectContaining({ waitingForOwner: { runId: fixture.scope.runId, controllerBootId: outgoingBoot } }) });
|
||||
const [task] = await db.select().from(issues).where(eq(issues.executionRunId, runId));
|
||||
expect(task?.status).toBe("in_progress");
|
||||
|
||||
@@ -685,6 +690,129 @@ describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable d
|
||||
} finally { resetControllerInstanceLeaseForTests(); }
|
||||
});
|
||||
|
||||
async function parkFollowUpBehindOutgoingServer(waitingFor?: (owner: { runId: string; controllerBootId: string }) => { runId: string; controllerBootId: string }) {
|
||||
const seeded = await seedWarmFollowUpOnOldServer();
|
||||
const { fixture, current, identity, recordOwner, runId } = seeded;
|
||||
const outgoingBoot = fixture.owner.bootId;
|
||||
await recordOwner({ controllerBootId: outgoingBoot, controllerPid: 7, controllerHostHash: "outgoing-container",
|
||||
controllerProcessStartedAt: new Date(Date.now() - 600_000), leaseOwner: null, leaseExpiresAt: null });
|
||||
await db.insert(controllerInstances).values({ bootId: outgoingBoot, processSpace: "outgoing-container", pid: 7,
|
||||
leaseExpiresAt: new Date(Date.now() + 90_000) });
|
||||
await db.update(nativeRunFinalizations).set({ controllerBootId: randomUUID() }).where(eq(nativeRunFinalizations.runId, runId));
|
||||
const wait = (checkpointedAt: Date | null) => waitForLiveWarmSessionOwner({ db, execution: current,
|
||||
priorRunId: fixture.scope.runId, checkpointedAt: async () => checkpointedAt });
|
||||
const waiting = await wait(null).catch(error => error);
|
||||
expect(waiting).toBeInstanceOf(NativeControllerRecoveryWaitingError);
|
||||
expect(waiting.owner).toEqual({ runId: fixture.scope.runId, controllerBootId: outgoingBoot });
|
||||
const parkedAt = Date.now();
|
||||
await deferNativeControllerRecovery(db, { ...current.binding, retryAt: waiting.retryAt,
|
||||
waitingForOwner: waitingFor ? waitingFor(waiting.owner) : waiting.owner });
|
||||
const outgoingRoot = join(root, `${fixture.scope.runId}-outgoing`);
|
||||
(await loadNativeRunnerCheckpoint(db, { companyId, issueId: fixture.scope.issueId, runId: fixture.scope.runId }))!
|
||||
.restoreSessionDirectory(outgoingRoot);
|
||||
const outgoing = new DurablePrpControlPlane({ stateDirectory: join(outgoingRoot, "control-plane"), identity,
|
||||
expectedRunnerVersion: "discovery-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
|
||||
outgoing.setCheckpointPersistence(nativeRunnerCheckpointWriter(db, fixture.owner));
|
||||
const fenceHarnessBackup = async () => assertNativeHarnessBackupOwner(db, await nativeHarnessBackupOwner(db, {
|
||||
companyId, issueId: fixture.scope.issueId, runId: fixture.scope.runId, bootId: outgoingBoot }));
|
||||
const saveSuspend = async () => {
|
||||
outgoing.queueCommand("runner.suspend", {}, `idle-close-${randomUUID()}`);
|
||||
await outgoing.flushCheckpoint();
|
||||
};
|
||||
const closeIdleSession = async () => {
|
||||
await fenceHarnessBackup();
|
||||
await saveSuspend();
|
||||
return new Date();
|
||||
};
|
||||
return { ...seeded, wait, waiting: waiting as NativeControllerRecoveryWaitingError, parkedAt, outgoing,
|
||||
fenceHarnessBackup, saveSuspend, closeIdleSession };
|
||||
}
|
||||
|
||||
it("the outgoing server checkpoints and closes its idle warm session while the incoming server's follow-up waits for it, and the follow-up resumes within one recheck", async () => {
|
||||
registerControllerInstanceDb(db);
|
||||
const parked = await parkFollowUpBehindOutgoingServer();
|
||||
try {
|
||||
const { fixture, runId, wait, waiting, parkedAt, outgoing, closeIdleSession } = parked;
|
||||
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0])
|
||||
.toMatchObject({ phase: "retryable_failure", attempt: 0, leaseOwner: null, failureCode: NATIVE_RETAINED_OWNER_WAIT,
|
||||
failureDetail: expect.objectContaining({ waitingForOwner: waiting.owner }) });
|
||||
const [prior] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.scope.runId));
|
||||
const ownerExitDeadline = prior!.finishedAt!.getTime() + 600_000 + 120_000;
|
||||
|
||||
const checkpointedAt = await closeIdleSession();
|
||||
const saved = await loadNativeRunnerCheckpoint(db, { companyId, issueId: fixture.scope.issueId, runId: fixture.scope.runId });
|
||||
const restoredRoot = join(root, `${runId}-idle-close`);
|
||||
saved!.restoreSessionDirectory(restoredRoot);
|
||||
const restored = JSON.parse(await readFile(join(restoredRoot, "control-plane", "control-plane-state.json"), "utf8"));
|
||||
expect(restored.commands).toContainEqual(expect.objectContaining({ type: "runner.suspend" }));
|
||||
await outgoing.stop();
|
||||
|
||||
expect(await claimNativeSessionResumptions({ db, runnerInstanceId: "incoming", runIds: [runId] })).toEqual([]);
|
||||
const claims = await claimNativeSessionResumptions({ db, runnerInstanceId: "incoming", runIds: [runId], now: waiting.retryAt });
|
||||
expect(claims).toHaveLength(1);
|
||||
await expect(wait(checkpointedAt)).resolves.toBeUndefined();
|
||||
const waitedMs = Math.max(waiting.retryAt.getTime(), checkpointedAt.getTime()) - parkedAt;
|
||||
expect(waitedMs).toBeLessThanOrEqual(5_000);
|
||||
expect(ownerExitDeadline - parkedAt).toBeGreaterThan(600_000);
|
||||
} finally {
|
||||
await parked.outgoing.stop();
|
||||
resetControllerInstanceLeaseForTests();
|
||||
}
|
||||
});
|
||||
|
||||
it("holds a resumption claim of the parked follow-up until the outgoing server's save commits", async () => {
|
||||
registerControllerInstanceDb(db);
|
||||
const parked = await parkFollowUpBehindOutgoingServer();
|
||||
try {
|
||||
const { fixture, runId, waiting } = parked;
|
||||
let claim!: Promise<Awaited<ReturnType<typeof claimNativeSessionResumptions>>>;
|
||||
let claimSettled = false;
|
||||
await db.transaction(async tx => {
|
||||
await assertNativeRunnerCheckpointOwner(tx, fixture.owner, false);
|
||||
claim = claimNativeSessionResumptions({ db, runnerInstanceId: "incoming", runIds: [runId], now: waiting.retryAt });
|
||||
void claim.finally(() => { claimSettled = true; });
|
||||
await vi.waitFor(async () => {
|
||||
const [blocked] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity
|
||||
where datname = current_database() and wait_event_type = 'Lock' and query ilike '%native_run_finalizations%'`);
|
||||
expect(blocked.count).toBeGreaterThan(0);
|
||||
}, { timeout: 1_500, interval: 20 });
|
||||
expect(claimSettled).toBe(false);
|
||||
});
|
||||
expect(await claim).toHaveLength(1);
|
||||
await expect(parked.saveSuspend()).rejects.toThrow("native_runner_checkpoint_owner_lost");
|
||||
} finally {
|
||||
await parked.outgoing.stop();
|
||||
resetControllerInstanceLeaseForTests();
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["own-process", "own-process-event", "own-checkpoint", "claimed", "other-owner-boot", "other-owner-run"] as const)(
|
||||
"refuses the outgoing server's idle checkpoint when the parked follow-up is no longer waiting on it: %s", async scenario => {
|
||||
registerControllerInstanceDb(db);
|
||||
const parked = await parkFollowUpBehindOutgoingServer(owner => scenario === "other-owner-boot"
|
||||
? { ...owner, controllerBootId: randomUUID() } : scenario === "other-owner-run" ? { ...owner, runId: randomUUID() } : owner);
|
||||
try {
|
||||
const { fixture, runId, waiting, fenceHarnessBackup, saveSuspend } = parked;
|
||||
if (scenario === "own-process") {
|
||||
await db.update(heartbeatRuns).set({ processPid: 4242, processGroupId: 4242, processStartedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
|
||||
} else if (scenario === "own-process-event") {
|
||||
await db.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: 1, eventType: PROCESS_START_REQUESTED });
|
||||
} else if (scenario === "own-checkpoint") {
|
||||
await db.insert(nativeRunnerCheckpoints).values({ companyId, issueId: fixture.scope.issueId, runId, part: "header", material: {} });
|
||||
} else if (scenario === "claimed") {
|
||||
expect(await claimNativeSessionResumptions({ db, runnerInstanceId: "incoming", runIds: [runId], now: waiting.retryAt })).toHaveLength(1);
|
||||
}
|
||||
const before = await loadNativeRunnerCheckpoint(db, { companyId, issueId: fixture.scope.issueId, runId: fixture.scope.runId });
|
||||
await expect(fenceHarnessBackup()).rejects.toThrow("native_runner_checkpoint_owner_lost");
|
||||
await expect(saveSuspend()).rejects.toThrow("native_runner_checkpoint_owner_lost");
|
||||
expect((await loadNativeRunnerCheckpoint(db, { companyId, issueId: fixture.scope.issueId, runId: fixture.scope.runId }))?.records)
|
||||
.toEqual(before?.records);
|
||||
} finally {
|
||||
await parked.outgoing.stop();
|
||||
resetControllerInstanceLeaseForTests();
|
||||
}
|
||||
});
|
||||
|
||||
it("sends a re-wake after a crash restart to a fresh sandbox when the parked predecessor's warm Runner cannot be adopted", async () => {
|
||||
const fixture = await seed(), runId = randomUUID(), environmentId = randomUUID(), leaseId = randomUUID();
|
||||
const previous = parseNativeExecutionInput({
|
||||
|
||||
@@ -70,8 +70,10 @@ export type NativeControllerRecoveryOutcome =
|
||||
|
||||
export const NATIVE_RETAINED_OWNER_WAIT = "native_controller_recovery_waiting_for_owner";
|
||||
|
||||
export type NativeRetainedOwnerWait = { runId: string; controllerBootId: string };
|
||||
|
||||
export class NativeControllerRecoveryWaitingError extends Error {
|
||||
constructor(readonly retryAt: Date) {
|
||||
constructor(readonly retryAt: Date, readonly owner?: NativeRetainedOwnerWait) {
|
||||
super(NATIVE_RETAINED_OWNER_WAIT);
|
||||
this.name = "NativeControllerRecoveryWaitingError";
|
||||
}
|
||||
@@ -83,7 +85,7 @@ export class NativeControllerRecoveryWaitingError extends Error {
|
||||
* permissions and task ownership when this deadline arrives. */
|
||||
export async function deferNativeControllerRecovery(db: Db, input: {
|
||||
companyId: string; agentId: string; issueId: string; runId: string;
|
||||
leaseOwner?: string; retryAt: Date;
|
||||
leaseOwner?: string; retryAt: Date; waitingForOwner?: NativeRetainedOwnerWait;
|
||||
}): Promise<void> {
|
||||
await db.transaction(async tx => {
|
||||
const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues)
|
||||
@@ -128,7 +130,9 @@ export async function deferNativeControllerRecovery(db: Db, input: {
|
||||
failureDetail: { originalFailureCode: NATIVE_RETAINED_OWNER_WAIT,
|
||||
message: "Waiting for the retained native controller owner to release its lease.",
|
||||
recoveryMode: record(run.runnerProfileJson).sessionCheckpoint ? "same_session_resume" : "bootstrap_retry",
|
||||
providerSessionEstablished: false, providerEventsExist: false },
|
||||
providerSessionEstablished: false, providerEventsExist: false,
|
||||
...(input.waitingForOwner ? { waitingForOwner: { runId: input.waitingForOwner.runId,
|
||||
controllerBootId: input.waitingForOwner.controllerBootId } } : {}) },
|
||||
updatedAt: new Date(),
|
||||
}).where(eq(nativeRunFinalizations.runId, input.runId));
|
||||
});
|
||||
|
||||
@@ -3,12 +3,14 @@ import { NATIVE_PREPARATION_FENCING_VERSION } from "./native-active-runs.js";
|
||||
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { and, eq, exists, gt, isNull, lte, inArray, notExists, or, sql, type SQL } from "drizzle-orm";
|
||||
import { checkpointDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, primeDatabaseWorkTransactionId, type Db } from "@paperclipai/db";
|
||||
import { checkpointDb, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, primeDatabaseWorkTransactionId, type Db } from "@paperclipai/db";
|
||||
import { inspectDurablePrpCheckpointHeader, inspectUnauthenticatedBootstrapCheckpoint, inspectUnclaimedBootstrapCheckpoint, decodeDurablePrpCheckpoint, type DurablePrpCheckpoint, type DurablePrpControlPlane, type DurableRecoveryIdentity } from "../../vendor/paperclip-runner/index.js";
|
||||
import { createLocalEncryptedCipher, deriveLocalEncryptedInstanceSecret } from "../../secrets/local-encrypted-provider.js";
|
||||
import { legacyControllerBootId } from "../legacy-controller-lease.js";
|
||||
import { assertNativeSourceCheckpointConsumer, lockSourceRecoveryCoordinators } from "./native-source-recovery-ownership.js";
|
||||
import { runnerdStateHasNoProviderTurn, runnerdStateIsIncompleteBootstrap } from "./native-bootstrap-proof.js";
|
||||
import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js";
|
||||
import { OWN_PROCESS_START_EVENT_TYPES } from "../native-process-start-events.js";
|
||||
|
||||
type Binding = { companyId: string; issueId: string; runId: string };
|
||||
export const NATIVE_CHECKPOINT_HEADER_INSPECTION_MAX_BYTES = 1024 * 1024;
|
||||
@@ -116,13 +118,37 @@ export async function validateNativeRunnerCheckpointOwner(
|
||||
eq(heartbeatRuns.issueId, owner.issueId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"]),
|
||||
notExists(tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
|
||||
.where(eq(nativeRunFinalizations.runId, heartbeatRuns.id))))) : [];
|
||||
const [parkedSuccessor] = !successor && !preparingSuccessor && !waitingSuccessor && task?.executionRunId && task.executionRunId !== owner.runId
|
||||
? await parkedSuccessorWaitingForOwner(tx, owner, task.executionRunId) : [];
|
||||
if (!current.resultId || newWork || !task
|
||||
|| (task.executionRunId && task.executionRunId !== owner.runId && !successor && !preparingSuccessor && !waitingSuccessor)) {
|
||||
|| (task.executionRunId && task.executionRunId !== owner.runId && !successor && !preparingSuccessor && !waitingSuccessor && !parkedSuccessor)) {
|
||||
throw new Error("native_runner_checkpoint_owner_lost");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function parkedSuccessorWaitingForOwner(tx: CheckpointTransaction, owner: NativeRunnerCheckpointFenceOwner, successorRunId: string) {
|
||||
return tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
|
||||
.innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)).where(and(
|
||||
eq(nativeRunFinalizations.runId, successorRunId), eq(nativeRunFinalizations.companyId, owner.companyId),
|
||||
eq(nativeRunFinalizations.issueId, owner.issueId), eq(nativeRunFinalizations.phase, "retryable_failure"),
|
||||
eq(nativeRunFinalizations.failureCode, NATIVE_RETAINED_OWNER_WAIT), eq(nativeRunFinalizations.attempt, 0),
|
||||
isNull(nativeRunFinalizations.resultId), isNull(nativeRunFinalizations.leaseOwner),
|
||||
sql`${nativeRunFinalizations.failureDetail}->'waitingForOwner'->>'runId' = ${owner.runId}`,
|
||||
sql`${nativeRunFinalizations.failureDetail}->'waitingForOwner'->>'controllerBootId' = ${owner.bootId}`,
|
||||
eq(heartbeatRuns.companyId, owner.companyId), eq(heartbeatRuns.nativeIssueId, owner.issueId),
|
||||
eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "running"),
|
||||
isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), isNull(heartbeatRuns.processStartedAt),
|
||||
sql`${heartbeatRuns.nativeSessionId} = (select source.native_session_id from heartbeat_runs source
|
||||
where source.id = ${owner.runId} and source.company_id = ${owner.companyId})`,
|
||||
notExists(tx.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints).where(and(
|
||||
eq(nativeRunnerCheckpoints.companyId, owner.companyId), eq(nativeRunnerCheckpoints.runId, successorRunId)))),
|
||||
notExists(tx.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
|
||||
eq(heartbeatRunEvents.companyId, owner.companyId), eq(heartbeatRunEvents.runId, successorRunId),
|
||||
isNull(heartbeatRunEvents.sourceEventId), inArray(heartbeatRunEvents.eventType, [...OWN_PROCESS_START_EVENT_TYPES])))),
|
||||
)).for("share", { of: nativeRunFinalizations });
|
||||
}
|
||||
|
||||
export async function assertNativeRunnerCheckpointOwner(tx: CheckpointTransaction, owner: NativeRunnerCheckpointOwner, newWork: boolean) {
|
||||
const locked = await lockNativeRunnerCheckpointOwner(tx, owner);
|
||||
await validateNativeRunnerCheckpointOwner(tx, owner, locked, newWork);
|
||||
|
||||
@@ -10735,7 +10735,7 @@ export async function executePaperclipNativeSession(input: {
|
||||
executionFailure = error;
|
||||
if (error instanceof NativeControllerRecoveryWaitingError) {
|
||||
await deferNativeControllerRecovery(input.db, { ...input.execution.binding,
|
||||
leaseOwner: input.leaseOwner, retryAt: error.retryAt });
|
||||
leaseOwner: input.leaseOwner, retryAt: error.retryAt, waitingForOwner: error.owner });
|
||||
}
|
||||
if (!(error instanceof NativeRunnerOwnershipUnverifiedError) && !(error instanceof NativeStartupCleanupPendingError) && startup.stopRequested && startup.executionContext?.signal.reason instanceof Error &&
|
||||
startup.executionContext.signal.reason.message === "native_session_cancelled") {
|
||||
|
||||
@@ -53,10 +53,10 @@ export async function waitForLiveWarmSessionOwner(input: {
|
||||
const finishedAt = prior?.finishedAt ?? null;
|
||||
const retryAt = await liveWarmSessionOwnerRetryAt({ owner, finishedAt,
|
||||
idleTimeoutMs: input.execution.session.lifecyclePolicy.idleTimeoutMs ?? null });
|
||||
if (!retryAt) return;
|
||||
if (!retryAt || !owner.controllerBootId) return;
|
||||
const checkpointedAt = await input.checkpointedAt();
|
||||
if (checkpointedAt && finishedAt && checkpointedAt >= finishedAt) return;
|
||||
throw new NativeControllerRecoveryWaitingError(retryAt);
|
||||
throw new NativeControllerRecoveryWaitingError(retryAt, { runId: input.priorRunId, controllerBootId: owner.controllerBootId });
|
||||
}
|
||||
|
||||
export type RemoteWarmSessionRecovery = {
|
||||
@@ -228,7 +228,9 @@ export async function inspectRemoteWarmSessionRecovery(input: {
|
||||
if (prior.owner.phase !== "committed" || prior.owner.leaseOwner) return deny("owner_unsettled");
|
||||
const liveOwnerRetryAt = await liveWarmSessionOwnerRetryAt({ owner: prior.owner, finishedAt: prior.run.finishedAt,
|
||||
idleTimeoutMs: previous.session?.lifecyclePolicy?.idleTimeoutMs ?? null });
|
||||
if (liveOwnerRetryAt) throw new NativeControllerRecoveryWaitingError(liveOwnerRetryAt);
|
||||
if (liveOwnerRetryAt) {
|
||||
throw new NativeControllerRecoveryWaitingError(liveOwnerRetryAt, { runId: identity.runId, controllerBootId: prior.owner.controllerBootId });
|
||||
}
|
||||
if (sourceProviderLeaseId === providerLeaseId) settledPredecessor = { runId: identity.runId, providerLeaseId };
|
||||
const checkpoint = input.loadCheckpoint(previous);
|
||||
if (!checkpoint || checkpoint.identity.runId !== identity.runId ||
|
||||
|
||||
Reference in new issue
Block a user