mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
fix(native): a same-boot stall never leaves a run waiting on itself (#20721)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep an execution lease. Restart recovery takes over a run only when it can prove that the recorded controller stopped. > - A stall (SIGSTOP) longer than the lease, with no restart, makes the executor lose its own lease. The executor then detaches for "the next controller" and exits. > - The next controller is this same process. Recovery asks the PID probe if the recorded controller is alive. The recorded controller is this process, so the answer is always "alive". > - The task then waits forever: `in_progress`, no Retry, an active run, and an active environment lease. > - A second gap: after back-to-back stalls mid-tool, the executor returns and nothing in this process ever looks at the run again. No terminal state, no recovery and no Retry. > - This pull request lets recovery recognise its own process. When this process no longer drives the run and its own lease lapsed, the prior owner is gone. The existing evidence checks then decide, as for any stopped controller. > - It also makes heartbeat schedule that recovery as soon as such an executor exits. > - The benefit is that a same-boot stall never needs a manual Retry. When provider work may have started, the run fails with a working Retry instead of a wait with no end. ## Linked Issues or Issue Description Refs #20694 (the same class of fix for legacy `preparing` runs), #20529, #20691. **What happened?** Chaos wave `w-20261111T034105-s9330`, task t02 (Daytona, OpenCode, fault TM1). The server process was frozen with SIGSTOP for about 84 s while run `89a3922b` prepared. After SIGCONT, `native.process_start_requested` was recorded and `native.session.execute` failed after 118 ms: the lease renewal found the lease lapsed, and the executor detached and exited. From then on, recovery recorded `native.recovery.transition` "waiting for safe ownership evidence" with reason `controller_still_alive` every 30 s for more than 20 minutes. The run stayed `running`, the issue stayed `in_progress`, no Retry was shown, and the environment lease stayed `active`. Chaos wave `w-20261111T040439-s9331` (Daytona, codex/claude/opencode, T1, TM1). Three stalls of about 90 s came back to back, with less than 1 s between them. Each run was in a tool call. After the second thaw, `native.session.execute` failed for all three runs (lease lost, executor detached). After that, no run had any event: no terminal state, no `native.recovery.transition` and no Retry, for 20 minutes. The in-sandbox turns kept going. A single 90 s stall (wave s9340) stays green, because a later recovery signal happens to find the run. With stacked stalls, that signal can land while the executor is still tearing down, so it skips the run and sets no deadline. **Expected behavior** When this process no longer drives a run that it recorded as its own, recovery takes the run back. No turn reached the runner here, so the bootstrap restarts on the same run. When a turn or runner work may have started, the run fails and shows a working Retry. **Steps to reproduce** 1. Start a native remote (Daytona) run. 2. Send SIGSTOP to the server during preparation, and hold it longer than the execution lease. 3. Send SIGCONT. The executor loses its lease, detaches and exits. 4. See that every recovery scan waits with `controller_still_alive`, and the run never settles. The new tests in `native-runner-restart-recovery.integration.test.ts` ("same-boot controller whose own lease lapsed") reproduce this at the recovery layer. ## What Changed - `claimNativeRestartRecoveries`: when the recorded controller is this exact process (same boot ID and PID): - If this process still drives the run (heartbeat execution or native executor registered), recovery leaves the run to it (`current_controller_preparing`). - If neither drives it and the self lease lapsed, the takeover is allowed (`current_controller_released`). The existing evidence checks then choose the outcome: bootstrap again on the same run when a durable record proves no command reached the runner, or fail with a Retry (`blocked`) when runner or provider work may have started. - An unexpired self lease still waits. A claim that has not reached its executor holds it, and it lapses by itself. - A run that this process handed to the next server (graceful drain `restart_suspended` record, or the in-process hot-restart handoff marker) stays with the next server. - `heartbeat.ts`: when the executor ends with `NativeControllerDetachedForRestartError`, there was no shutdown or hot-restart handoff, and this process's own lease lapsed (`ownNativeExecutionLeaseLapsed`), heartbeat schedules restart recovery for the run when the executor exits. Recovery then reattaches the remote runner whose turn is still running, restarts an unstarted bootstrap, or fails the run with a working Retry. - `claimNativeRestartRecoveries`: a self-owned lease that is still live, with no driver here, now returns a `retryAt` at its expiry, so the wait is bounded. - `heartbeat/shutdown-handoff.ts`: add `isRunHandedOffForShutdown`. Heartbeat recovery passes it to the claim. - `suspendedByCleanDrain` now uses the shared `handedToNextControllerSinceClaim` helper. Behavior is the same. - Heartbeat test: an executor detaches after its own lease lapsed, with no restart. The same process must take the run back. Red before (timed out, the run stayed `running`), green after. - Tests: five cases for the same-boot lapsed lease: bootstrap again with no runner command; fail with a working Retry when a runner identity was recorded; leave the run to a live in-process executor; leave a handed-off run to the next server; still wait for a different live controller. ## Verification - Red before the fix: the three same-boot cases returned `awaiting_evidence` / `controller_still_alive`. - Green after the fix: - `pnpm --dir server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts` - Daytona wave `w-20261111T050504-s9331` from this branch (same seed and shape as s9331: 3 tasks, concurrency 3, three 90 s TM1 stalls with about 1 s between them). All 3 tasks reached `done` with no Retry. Run `632f0564` (claude) shows the exact s9331 sequence: `native.session.execute` failed after 216 s, then `reattach_remote_runner` 0.4 s later (before the third stall), then a second reattach after it, and the turn completed. Two tasks are graded FAIL only on the T1 crossing check: their own fault landed after their tool loop ended (for example loop ended 05:11:26, fault 05:11:44). Their status is `done`, with no Retry. - `tsc --noEmit -p server` is clean. ## Risks - A same-process recovery scan can now claim a run that this process recorded as its own. The guard needs all of these: the self lease lapsed, no in-process execution of the run, and no handoff to the next server. A claim that is still on its way to its executor holds a live lease, so it is not taken twice. - Upgrade safety: the decision uses only the coordinator row and in-process state. Old saved rows are judged by the same rules. A new process never matches an old boot ID, so deploys keep the existing proof path. - No new waits or file operations run before `turn.start`. - There is still no ceiling on `awaiting_evidence` for a genuinely different live controller (case c). This PR does not change that policy. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
1 parent
fcf4e39b41
commit
16024ffa2a
8 files changed
+202
-9
No files matched your search
@@ -384,6 +384,55 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut
|
||||
expect(reached).toHaveBeenCalled();
|
||||
}, 60_000);
|
||||
|
||||
// Daytona waves s9330 and s9331 (TM1): stalls of the same server outlived
|
||||
// the execution lease. Renewal found the lease lapsed and detached the
|
||||
// executor for "the next controller" with no shutdown handoff, and the
|
||||
// executor returned with the run still running. The next controller is this
|
||||
// same process; nothing may leave the run running with no recovery.
|
||||
it("recovers a run whose executor detached after its own lease lapsed, with no restart", async () => {
|
||||
const f = await seedRecoverableRun("SELF-DETACHED-LAPSED");
|
||||
let opened = 0;
|
||||
const reachedAgain = vi.fn();
|
||||
const backend = vi.fn((): NativeSessionBackend => {
|
||||
opened += 1;
|
||||
const first = opened === 1;
|
||||
const open = async () => {
|
||||
if (first) {
|
||||
// What the remote renewal's loss does: the lease lapsed during the
|
||||
// stall, and the executor is detached for the next controller.
|
||||
await f.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) })
|
||||
.where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
void detachNativeSessionsForRestart([f.runId]);
|
||||
throw new Error("runner stream lost while the controller was stopped");
|
||||
}
|
||||
reachedAgain();
|
||||
throw new Error("stop at provider dispatch");
|
||||
};
|
||||
return {
|
||||
async descriptor() {
|
||||
return { kind: "mock", name: "self-detached", version: "1", capabilities: {
|
||||
resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true },
|
||||
runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } };
|
||||
},
|
||||
openSession: open,
|
||||
recoverSession: open,
|
||||
} as unknown as NativeSessionBackend;
|
||||
});
|
||||
const heartbeat = heartbeatService(f.db, { nativeSessionBackendFactory: backend });
|
||||
try {
|
||||
const { claims } = await heartbeat.recoverNativeRunsAfterRestart(new Date(), [f.runId]);
|
||||
expect(claims).toEqual([expect.objectContaining({ runId: f.runId })]);
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
// The same process takes the run back on its own, without a restart.
|
||||
await waitForCondition("the same process to take the run back", () => reachedAgain.mock.calls.length > 0, 15_000);
|
||||
} finally { await heartbeat.drainActiveRunExecutions(); }
|
||||
expect(reachedAgain).toHaveBeenCalled();
|
||||
const transitions = await f.db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.runId),
|
||||
eq(heartbeatRunEvents.eventType, "native.recovery.transition")));
|
||||
expect(transitions.map(event => (event.payload as Record<string, unknown>).reason))
|
||||
.not.toContain("controller_still_alive");
|
||||
}, 60_000);
|
||||
|
||||
|
||||
async function shutdownDispositions(db: Awaited<ReturnType<typeof seedRecoverableRun>>["db"], runId: string) {
|
||||
const events = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, runId),
|
||||
@@ -832,7 +881,9 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut
|
||||
.where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
const onClaim = vi.fn();
|
||||
expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim }))
|
||||
.toEqual([{ kind: "awaiting_evidence", runId: f.runId, reason: "controller_still_alive" }]);
|
||||
.toEqual([{ kind: "awaiting_evidence", runId: f.runId, reason: "controller_still_alive",
|
||||
// This process's own live lease lapses by itself; recovery looks again then.
|
||||
retryAt: expect.any(Date) }]);
|
||||
expect(onClaim).not.toHaveBeenCalled();
|
||||
}, 60_000);
|
||||
|
||||
|
||||
@@ -55,7 +55,7 @@ import { prepareHeartbeatTaskContext } from "./heartbeat/task-context.js";
|
||||
import { selectHeartbeatEnvironment } from "./heartbeat/environment-selection.js";
|
||||
import { reconcileMuseNativeTerminalRun } from "./native-runtime/muse-terminal-reconciliation.js";
|
||||
import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js";
|
||||
import { nativeRecoveryClaimRelinquished, releaseLapsedNativeRestartClaim } from "./native-runtime/native-execution-lease.js";
|
||||
import { nativeRecoveryClaimRelinquished, ownNativeExecutionLeaseLapsed, releaseLapsedNativeRestartClaim } from "./native-runtime/native-execution-lease.js";
|
||||
import { executeHeartbeatRuntime, NativeSessionResumeScheduledError, NativeWorkspaceFinalizeScheduledError } from "./heartbeat/runtime-execution.js";
|
||||
import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js";
|
||||
import { nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
|
||||
@@ -2041,6 +2041,11 @@ export function heartbeatService(
|
||||
Parameters<typeof cleanupGitHubOperationLaunchers>[0] | null = null;
|
||||
let nativeSessionResumeScheduled = false;
|
||||
let restartClaimLapsed = false;
|
||||
// The executor detached for "the next controller" because its own lease
|
||||
// lapsed (for example after a stall), with no shutdown or hot-restart
|
||||
// handoff. The next controller is this process: recover the run here once
|
||||
// the executor has exited. A shutdown detach leaves it to the next server.
|
||||
let detachedWithoutHandoff = false;
|
||||
const shutdownHandedOff = watchShutdownHandoff(run.id);
|
||||
let nativeOwnershipHeld = false;
|
||||
let nativeStartupCleanupScheduled = false;
|
||||
@@ -3707,6 +3712,8 @@ export function heartbeatService(
|
||||
}
|
||||
if (err instanceof NativeControllerDetachedForRestartError) {
|
||||
nativeSessionResumeScheduled = true;
|
||||
detachedWithoutHandoff = !shutdownHandedOff() && await ownNativeExecutionLeaseLapsed(db, run)
|
||||
.catch(() => false);
|
||||
return;
|
||||
}
|
||||
if (err instanceof NativeRunnerOwnershipUnverifiedError) {
|
||||
@@ -4168,7 +4175,7 @@ export function heartbeatService(
|
||||
} finally {
|
||||
controllerLease.stop();
|
||||
activeRunExecutions.delete(run.id);
|
||||
if (restartClaimLapsed) scheduleNativeSessionResumeDispatch(run.id, new Date(), "restart");
|
||||
if (restartClaimLapsed || detachedWithoutHandoff) scheduleNativeSessionResumeDispatch(run.id, new Date(), "restart");
|
||||
// A failed owned Stop remains visible until this exact executor settles,
|
||||
// including a graceful exit result arriving after the cancellation error.
|
||||
// It is never retained beyond the active execution's cleanup.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { settleOwnableKind } from "../recovery/ownership-invariant-report.js";
|
||||
import { markRunHandedOffForShutdown } from "./shutdown-handoff.js";
|
||||
import { isRunHandedOffForShutdown, markRunHandedOffForShutdown } from "./shutdown-handoff.js";
|
||||
import { ORPHANED_RUN_LEASE_FAILURE_REASON } from "../environment-run-lease-settlement.js";
|
||||
import { ENVIRONMENT_LEASE_CONTROLLER_KEY } from "../environment-lease-controller.js";
|
||||
import { readNativeStartupCleanup, readNativeStartupContext } from "../native-runtime/native-startup-cleanup.js";
|
||||
@@ -1009,6 +1009,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
},
|
||||
db,
|
||||
liveRunExecutions: activeRunExecutions,
|
||||
handedOffForShutdown: isRunHandedOffForShutdown,
|
||||
runIds,
|
||||
restartKind,
|
||||
recoveryRequestId: intent?.recoveryRequestId ?? null,
|
||||
|
||||
@@ -11,3 +11,8 @@ export function watchShutdownHandoff(runId: string): () => boolean {
|
||||
return current !== undefined && current !== atStart;
|
||||
};
|
||||
}
|
||||
|
||||
/** This process handed the run to the next server (graceful drain or hot restart). */
|
||||
export function isRunHandedOffForShutdown(runId: string): boolean {
|
||||
return runsHandedOffForShutdown.has(runId);
|
||||
}
|
||||
@@ -220,6 +220,23 @@ export async function nativeRecoveryClaimRelinquished(input: {
|
||||
&& row.lastDisposition === "restart_suspended";
|
||||
}
|
||||
|
||||
/**
|
||||
* The run's controller is this process and its execution lease has lapsed
|
||||
* (or was released): the executor that held it detached because of that loss,
|
||||
* not because a shutdown or hot restart handed the run on.
|
||||
*/
|
||||
export async function ownNativeExecutionLeaseLapsed(db: Db, input: { companyId: string; id: string }): Promise<boolean> {
|
||||
const controller = await currentNativeControllerIdentity();
|
||||
const [row] = await db.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
|
||||
.innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId))
|
||||
.where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.id),
|
||||
eq(heartbeatRuns.status, "running"), isNull(nativeRunFinalizations.resultId),
|
||||
eq(nativeRunFinalizations.controllerBootId, controller.bootId), eq(nativeRunFinalizations.controllerPid, controller.pid),
|
||||
or(isNull(nativeRunFinalizations.leaseExpiresAt), sql`${nativeRunFinalizations.leaseExpiresAt} <= clock_timestamp()`)))
|
||||
.limit(1);
|
||||
return Boolean(row);
|
||||
}
|
||||
|
||||
export async function releaseLapsedNativeRestartClaim(input: {
|
||||
db: Db; companyId: string; runId: string; leaseOwner: string; controllerGeneration: number; providerAttempt: number;
|
||||
}): Promise<boolean> {
|
||||
|
||||
@@ -87,7 +87,9 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
|
||||
|
||||
expect(await claimNativeRestartRecoveries({ db, restartKind: "hard", runIds: [f.runId] })).toEqual([
|
||||
{ kind: kind === "owned" ? "awaiting_evidence" : "blocked", runId: f.runId,
|
||||
reason: kind === "owned" ? "controller_still_alive" : "provider_checkpoint_permanently_failed" },
|
||||
reason: kind === "owned" ? "controller_still_alive" : "provider_checkpoint_permanently_failed",
|
||||
// This process's own live lease lapses by itself; recovery looks again then.
|
||||
...(kind === "owned" ? { retryAt: expect.any(Date) } : {}) },
|
||||
]);
|
||||
const current = await readRun(f.runId);
|
||||
expect(current.status).toBe(kind === "owned" ? "running" : "failed");
|
||||
|
||||
@@ -164,9 +164,13 @@ export function suspendedByCleanDrain(coordinator: {
|
||||
}): boolean {
|
||||
// Attempt 0 means no execution has claimed the run yet. Its first provider
|
||||
// start must still be counted, whatever interrupted preparation.
|
||||
if (coordinator.attempt <= 0) return false;
|
||||
for (let index = coordinator.recoveryHistory.length - 1; index >= 0; index -= 1) {
|
||||
const disposition = coordinator.recoveryHistory[index]?.disposition;
|
||||
return coordinator.attempt > 0 && handedToNextControllerSinceClaim(coordinator.recoveryHistory);
|
||||
}
|
||||
|
||||
/** The latest record since the last claim hands the run to the next server. */
|
||||
function handedToNextControllerSinceClaim(history: ReadonlyArray<{ disposition?: unknown }>): boolean {
|
||||
for (let index = history.length - 1; index >= 0; index -= 1) {
|
||||
const disposition = history[index]?.disposition;
|
||||
if (disposition === "restart_suspended") return true;
|
||||
if (RECOVERY_CLAIM_DISPOSITIONS.has(disposition)) return false;
|
||||
}
|
||||
@@ -1029,6 +1033,8 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
recoveryRequestId?: string | null;
|
||||
runIds?: string[];
|
||||
liveRunExecutions?: Pick<ReadonlySet<string>, "has">;
|
||||
/** Runs this process handed to the next server (graceful drain or hot restart). */
|
||||
handedOffForShutdown?: (runId: string) => boolean;
|
||||
now?: Date;
|
||||
limit?: number;
|
||||
/** Called after each committed claim, before scanning the next candidate. */
|
||||
@@ -1306,8 +1312,27 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
&& (controllerHostHash === "unknown" || row.coordinator.controllerHostHash !== controllerHostHash);
|
||||
const nonExecutionOwner = row.coordinator.leaseOwner?.startsWith("native-finalizer:")
|
||||
|| row.coordinator.leaseOwner?.startsWith("native-cleanup:");
|
||||
// The recorded controller is this exact process (same boot and PID), so
|
||||
// a PID probe can only ever answer "alive" and the wait would never end.
|
||||
// This process knows whether it still drives the run: its executor is
|
||||
// registered for the whole execution. While it is, leave the run to it.
|
||||
// Once it has exited (for example after a stall outlived its lease and
|
||||
// renewal detached it for "the next controller"), the prior owner is
|
||||
// provably gone, and the evidence checks below decide as for any
|
||||
// stopped controller. An unexpired self lease still waits: a claim that
|
||||
// has not reached its executor yet holds it, and it lapses on its own.
|
||||
// A run this process handed to the next server (graceful drain or hot
|
||||
// restart) stays with that server: this process is about to exit.
|
||||
const ownControllerProcess = !nonExecutionOwner && isCurrentNativeControllerProcess(row.coordinator);
|
||||
if (ownControllerProcess && activeNativeExecutionRunIds.has(row.run.id)) {
|
||||
return { kind: "awaiting_evidence", runId: row.run.id, reason: "current_controller_preparing" } as const;
|
||||
}
|
||||
const ownLeaseLapsed = ownControllerProcess && row.coordinator.leaseExpiresAt !== null
|
||||
&& row.coordinator.leaseExpiresAt <= now && !input.handedOffForShutdown?.(row.run.id)
|
||||
&& !handedToNextControllerSinceClaim(row.coordinator.recoveryHistory);
|
||||
const evaluatedTakeover = nonExecutionOwner
|
||||
? { allowed: false, reason: "native_finalization_or_cleanup_owner" }
|
||||
: ownLeaseLapsed ? { allowed: true, reason: "current_controller_released" }
|
||||
: await evaluateNativeControllerTakeover({
|
||||
owner: row.coordinator,
|
||||
now,
|
||||
@@ -1379,6 +1404,10 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
runId: row.run.id,
|
||||
reason: takeover.reason,
|
||||
...(durableAuthority && row.coordinator.leaseExpiresAt ? { retryAt: row.coordinator.leaseExpiresAt }
|
||||
// This process's own unexpired lease with no driver here lapses by
|
||||
// itself; look again then instead of waiting for another signal.
|
||||
: ownControllerProcess && row.coordinator.leaseExpiresAt && row.coordinator.leaseExpiresAt > now
|
||||
&& !input.handedOffForShutdown?.(row.run.id) ? { retryAt: row.coordinator.leaseExpiresAt }
|
||||
: ["controller_lease_held", "controller_lease_grace"].includes(takeover.reason)
|
||||
? { retryAt: new Date(now.getTime() + CONTROLLER_LEASE_RENEW_MS) } : {}),
|
||||
} as const;
|
||||
|
||||
+82
-1
@@ -38,7 +38,7 @@ import {
|
||||
nativeRunnerCheckpoints,
|
||||
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
|
||||
} from "@paperclipai/db";
|
||||
import { beforeAll, expect, it, vi } from "vitest";
|
||||
import { beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { heartbeatService } from "../heartbeat.js";
|
||||
|
||||
import {
|
||||
@@ -79,6 +79,7 @@ import {
|
||||
restoreNativeRunnerCheckpoint,
|
||||
} from "./native-runner-checkpoint.js";
|
||||
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
|
||||
import { setNativeExecutionActive } from "./native-active-runs.js";
|
||||
import { RUNNER_BRIDGE_READ_OPERATIONS, RUNNER_BRIDGE_SCHEMAS } from "./runner-bridge-tools.js";
|
||||
import { HttpError } from "../../errors.js";
|
||||
import { legacyControllerBootId } from "../legacy-controller-lease.js";
|
||||
@@ -3297,6 +3298,86 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
|
||||
restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({ runId: fixture.runId, ...expected })]);
|
||||
});
|
||||
|
||||
// Daytona wave s9330 t02 (TM1): SIGSTOP held this same server past its
|
||||
// execution lease while the executor was starting the native process. On
|
||||
// SIGCONT the lease renewal found the lease lapsed, the executor detached for
|
||||
// "the next controller" and exited, and every recovery scan then waited on
|
||||
// the recorded controller: this same live PID and boot. Nothing else can hold
|
||||
// a lease that names this process, and nothing in it drives the run any more.
|
||||
describe("same-boot controller whose own lease lapsed", () => {
|
||||
it("restarts the bootstrap instead of waiting on itself when no turn reached the runner", async () => {
|
||||
const fixture = await seedUnlaunchedRemoteClaim("SELF-LAPSED-UNCOMMANDED",
|
||||
{ liveController: true, leaseExpired: true, event: { eventType: "native.process_start_requested" } });
|
||||
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: await currentNativeControllerIdentity(),
|
||||
restartKind: "hard", runIds: [fixture.runId], liveRunExecutions: new Set() });
|
||||
expect(claim).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId, controllerGeneration: 2 });
|
||||
const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
|
||||
.where(eq(nativeRunFinalizations.runId, fixture.runId));
|
||||
expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ reason: "remote_runner_never_received_commands" });
|
||||
});
|
||||
|
||||
it("fails with a Retry instead of waiting on itself when the runner may have started", async () => {
|
||||
const fixture = await seedUnlaunchedRemoteClaim("SELF-LAPSED-IDENTITY",
|
||||
{ liveController: true, leaseExpired: true, event: { eventType: "native.process_identity_recorded" } });
|
||||
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: await currentNativeControllerIdentity(),
|
||||
restartKind: "hard", runIds: [fixture.runId], liveRunExecutions: new Set() }))
|
||||
.toEqual([expect.objectContaining({ kind: "blocked", runId: fixture.runId, reason: "ambiguous_remote_recovery_evidence" })]);
|
||||
const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
|
||||
.where(eq(nativeRunFinalizations.runId, fixture.runId));
|
||||
expect(coordinator).toMatchObject({ phase: "terminal_failure", leaseOwner: null, failureCode: "native_restart_recovery_blocked" });
|
||||
const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
expect(run.status).toBe("failed");
|
||||
const [issue] = await fixture.db.select().from(issues).where(eq(issues.id, fixture.issueId));
|
||||
expect(await taskInterruptionPresentation(fixture.db, issue)).toMatchObject({
|
||||
state: "failed", canRetry: true, observedRunId: fixture.runId });
|
||||
const personId = randomUUID();
|
||||
await fixture.db.insert(authUsers).values({ id: personId, name: "Retry person", email: `${personId}@fixture.invalid`,
|
||||
emailVerified: true, createdAt: new Date(), updatedAt: new Date() });
|
||||
await fixture.db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: personId,
|
||||
status: "active", membershipRole: "member" });
|
||||
expect(await requestTaskRetry({ db: fixture.db, companyId, issueId: fixture.issueId, actorId: personId,
|
||||
actorSource: "session", idempotencyKey: `self-lapsed-${fixture.runId}`, observedRunId: fixture.runId }))
|
||||
.toMatchObject({ accepted: true });
|
||||
});
|
||||
|
||||
it("leaves the run to this process's executor while it still drives the run", async () => {
|
||||
const fixture = await seedUnlaunchedRemoteClaim("SELF-LAPSED-EXECUTING",
|
||||
{ liveController: true, leaseExpired: true, event: { eventType: "native.process_start_requested" } });
|
||||
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: await currentNativeControllerIdentity(),
|
||||
restartKind: "hard", runIds: [fixture.runId], liveRunExecutions: new Set([fixture.runId]) }))
|
||||
.toEqual([expect.objectContaining({ kind: "awaiting_evidence", reason: "current_controller_preparing" })]);
|
||||
setNativeExecutionActive(fixture.runId, true);
|
||||
try {
|
||||
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: await currentNativeControllerIdentity(),
|
||||
restartKind: "hard", runIds: [fixture.runId], liveRunExecutions: new Set() }))
|
||||
.toEqual([expect.objectContaining({ kind: "awaiting_evidence", reason: "current_controller_preparing" })]);
|
||||
} finally {
|
||||
setNativeExecutionActive(fixture.runId, false);
|
||||
}
|
||||
});
|
||||
|
||||
it("leaves a run this process handed to the next server for that server", async () => {
|
||||
const fixture = await seedUnlaunchedRemoteClaim("SELF-LAPSED-HANDED-OFF",
|
||||
{ liveController: true, leaseExpired: true, event: { eventType: "native.process_start_requested" } });
|
||||
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: await currentNativeControllerIdentity(),
|
||||
restartKind: "hard", runIds: [fixture.runId], liveRunExecutions: new Set(),
|
||||
handedOffForShutdown: runId => runId === fixture.runId }))
|
||||
.toEqual([expect.objectContaining({ kind: "awaiting_evidence", reason: "controller_still_alive" })]);
|
||||
});
|
||||
|
||||
it("keeps waiting for a different live controller in this process space", async () => {
|
||||
const fixture = await seedUnlaunchedRemoteClaim("OTHER-LIVE-LAPSED",
|
||||
{ leaseExpired: true, event: { eventType: "native.process_start_requested" } });
|
||||
// Another live process (the test runner's parent) recorded as controller.
|
||||
const parentStartedAt = new Date((await readProcessStartedAt(process.ppid))!);
|
||||
await fixture.db.update(nativeRunFinalizations).set({ controllerPid: process.ppid,
|
||||
controllerProcessStartedAt: parentStartedAt }).where(eq(nativeRunFinalizations.runId, fixture.runId));
|
||||
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: await currentNativeControllerIdentity(),
|
||||
restartKind: "hard", runIds: [fixture.runId], liveRunExecutions: new Set() }))
|
||||
.toEqual([expect.objectContaining({ kind: "awaiting_evidence", reason: "controller_still_alive" })]);
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ attempt: 2, driverKind: "codex_app_server" },
|
||||
{ attempt: 3, driverKind: "codex_app_server" },
|
||||
|
||||
Reference in new issue
Block a user