mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 00:56:52 +01:00
fix(native): re-bootstrap a claimed Daytona run killed before runner launch (#19616)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. Restart recovery decides what to do with each live run after
the control plane restarts.
> - #19583 lets recovery start the same-run bootstrap again when the
Postgres controller checkpoint proves that the runner received only
bootstrap commands.
> - A live replay found an earlier window. The control plane died after
the executor claimed the run and staged the sandbox workspace, but
before it asked for a runner process. No runner checkpoint existed, so
#19583 did not apply.
> - Recovery classified that run as `ambiguous_remote_recovery_evidence`
and blocked the task. The task also showed "Workspace repair required",
which did not describe the real cause.
> - This pull request proves the "nothing reached a runner" case from
durable evidence and starts the bootstrap again. It also names the
restart in the settled hold note.
> - The benefit is that a restart at any point of Daytona startup before
runner launch continues automatically.
## Linked Issues or Issue Description
**What happened?**
A chaos replay killed the control plane about 2 s after a claimed ACPX
Claude run started on Daytona. The kill came after `stage.workspace` and
environment acquisition, before `native.process_start_requested`. The
restarted server took over because the previous controller process was
dead. It then returned `ambiguous_remote_recovery_evidence` with
disposition `blocked`. The task rested blocked as "Workspace repair
required".
Evidence in the saved database at that point: coordinator `attempt: 1`,
`controllerGeneration: 1`, previous controller identity recorded and
dead. Native workspace sync was `prepared`. There were no runner
checkpoint rows, no session checkpoint, no process PID or start time, no
provider session, and no execution-start or runner-sourced event. The
local session root was an empty directory.
**Expected behavior**
When durable evidence proves that no runner process was requested and no
controller checkpoint exists, recovery continues the run with a fresh
bootstrap. Any evidence that a runner may have started keeps the hold.
**Steps to reproduce**
1. Start a native run on a Daytona environment.
2. Kill the control plane after workspace staging and before the
executor requests the runner process.
3. Restart the control plane.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.
Root cause:
`server/src/services/native-runtime/native-restart-recovery.ts`,
function `classifyNativeRunnerRecoveryEvidence`, remote branch. The only
no-checkpoint remote rule required an unclaimed coordinator (`attempt:
0`, no controller identity). A claimed run with no launch evidence fell
through to "ambiguous".
Refs #19583, #16349
## What Changed
- `native-restart-recovery.ts`: add the `remote_runner_never_launched`
proof, which leads to `bootstrap_incomplete`. It applies only to a
classic sandbox run with a prepared workspace and the preparation
fencing version. The run must have a real executor claim (controller
identity and lease owner) whose controller is gone (takeover allowed).
It needs no durable checkpoint, no session checkpoint, no process
fields, no provider session, and no provider evidence. The run must also
have no `native.process_start_requested`, process identity,
`adapter.invoke`, `tool.execution.started`, or runner-sourced event. The
same three-attempt budget applies. The shared event query moved into
`hasNativeExecutionStartEvidence`.
- `native-session-executor.ts`:
`restoreRemoteBootstrapCheckpointForRecovery` removes an empty session
root that preparation created. Without that step, the fresh bootstrap
read it as `durable_identity_unreadable`.
- `execution-recovery-resolution.ts`: the new
`executionRecoverySettlementNote` names a settled
`native_restart_recovery_blocked` hold as a stop after a server restart.
It no longer shows "Workspace repair required" because a later workspace
export was retried. Other causes keep their notes.
- `doc/run-log-events.md`: document the new reason.
## Verification
- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "unlaunched|never requested|Postgres checkpoint"`. Before the fix,
the proven case (lease live with a dead owner, and lease expired) and
the budget case returned `blocked` /
`ambiguous_remote_recovery_evidence`. The empty-root executor case
failed.
- Fail-closed cases: these keep the hold:
`native.process_start_requested`, a process identity event,
`adapter.invoke`, a runner-sourced event, a recorded PID, a recorded
provider session, and a spent budget. A live claiming controller returns
`awaiting_evidence` / `controller_still_alive`. The existing
"contradictory evidence" cases (`claimed` with no controller identity,
`launch`, `process`, and others) stay blocked.
- Wording: `server/src/services/execution-recovery-resolution.test.ts`.
- Regression set (10 files, including `workspace-recovery`,
`native-safe-replacement`, `heartbeat/recovery`,
`native-session-executor`, and `native-restart-recovery`): 2035 passed,
14 skipped.
- Live Daytona replay of the failing chaos task, 3 runs, sequential,
ACPX Claude, double SIGKILL, current master plus this change. Image:
content-matched `paperclip-daytona-runner@sha256:5a29a797…` built from
c6eecac1e0, with runner inputs pinned to it, because the image-only
build sat in the Actions queue. Result: **3/3 PASS** (R1 pass; R2, R4,
R5, R6 and T pass). Each run's faults landed after the runner launched.
Recovery waited on the live remote controller lease, then continued
without a Retry. Run 3 continued through `reattach_remote_runner`. The
saved logs for runs 1 and 2 do not name their claim kind. The earlier
pre-launch window (the subject of this PR) is not hit reliably by replay
timing. The integration tests cover that window with the same evidence
shape as the saved failing database.
## Risks
- The proof depends on the executor writing
`native.process_start_requested` before a backend may spawn a runner
(`onSessionAdmission` in `executeNativeSession`). A future change that
spawns a runner earlier would weaken this proof. The fail-closed tests
pin the event as the gate.
- A fresh bootstrap acquires a new sandbox. The staged sandbox from the
dead attempt keeps its lease until run settlement releases all run
leases. It never ran a runner.
- The note change affects only settled holds with the cause
`native_restart_recovery_blocked`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, GitHub CLI) and extended reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c9391720a1
commit
15ee6a25a6
6 files changed
+189
-29
No files matched your search
@@ -355,6 +355,12 @@ unfinished `session.open`, with no committed or acknowledged event. The
|
||||
executor restores that checkpoint and checks the same proof before it
|
||||
bootstraps again. After three provider attempts it records
|
||||
`execution_recovery_budget_exhausted` instead.
|
||||
A claimed remote run whose previous controller is gone, with no
|
||||
`native.process_start_requested`, process identity, adapter invocation,
|
||||
runner-sourced event or controller checkpoint, records
|
||||
`remote_runner_never_launched` with the same disposition and budget. The
|
||||
executor records `native.process_start_requested` before a backend may spawn a
|
||||
runner, so its absence (with no checkpoint) proves nothing reached a runner.
|
||||
|
||||
For a local Codex, OpenCode, or ACPX turn lost while runnerd was stopped, the canonical `turn.failed`
|
||||
event retains `error.code: "provider_turn_lost_on_restore"` and
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { executionRecoverySettlementNote } from "./execution-recovery-resolution.js";
|
||||
|
||||
describe("executionRecoverySettlementNote", () => {
|
||||
it("names a restart-recovery hold instead of asking for a workspace repair", () => {
|
||||
const note = executionRecoverySettlementNote({ current: true, cause: "native_restart_recovery_blocked",
|
||||
workspaceRestoreFailed: true });
|
||||
expect(note).not.toContain("Workspace repair required");
|
||||
expect(note).toContain("after a server restart");
|
||||
});
|
||||
|
||||
it("keeps the workspace repair note for other holds with a restore failure", () => {
|
||||
expect(executionRecoverySettlementNote({ current: true, cause: "execution_finalization_deadline_exceeded",
|
||||
workspaceRestoreFailed: true })).toContain("Workspace repair required");
|
||||
expect(executionRecoverySettlementNote({ current: true, cause: "execution_finalization_deadline_exceeded",
|
||||
workspaceRestoreFailed: false })).toContain("Automatic recovery stopped. Recorded work is preserved");
|
||||
expect(executionRecoverySettlementNote({ current: false, cause: "native_restart_recovery_blocked",
|
||||
workspaceRestoreFailed: true })).toContain("Recovery closed");
|
||||
});
|
||||
});
|
||||
@@ -36,6 +36,23 @@ import { readSystemRecovery } from "./system-recovery-state.js";
|
||||
import { alias } from "drizzle-orm/pg-core";
|
||||
|
||||
/** An operator records observed outcomes; this is not permission to blindly retry. */
|
||||
/**
|
||||
* The user-facing note for a settled recovery hold. A restart-recovery hold
|
||||
* names its own cause: a workspace export retried after that hold does not
|
||||
* mean the user must repair the workspace.
|
||||
*/
|
||||
export function executionRecoverySettlementNote(input: {
|
||||
current: boolean; cause: string; workspaceRestoreFailed: boolean;
|
||||
}): string {
|
||||
if (!input.current) return "Recovery closed because the task's owner, execution, or status changed. No work was replayed.";
|
||||
if (input.cause === "native_restart_recovery_blocked") {
|
||||
return "Automatic recovery stopped after a server restart because the previous execution could not be verified. Saved work is preserved; actions with unverified outcomes will not be repeated.";
|
||||
}
|
||||
return input.workspaceRestoreFailed
|
||||
? "Workspace repair required. Verify safe staging or repair before continuing. Saved work and approval decisions remain in force."
|
||||
: "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated.";
|
||||
}
|
||||
|
||||
export async function validateExecutionReconciliation(input: {
|
||||
db: Db;
|
||||
companyId: string;
|
||||
@@ -626,11 +643,8 @@ export async function settleUnrecoverableExecutions(
|
||||
!["done", "cancelled"].includes(task.status) &&
|
||||
(!task.executionRunId || task.executionRunId === run.id) &&
|
||||
(!task.checkoutRunId || task.checkoutRunId === run.id);
|
||||
const note = current
|
||||
? hasWorkspaceRestoreFailure(run.resultJson)
|
||||
? "Workspace repair required. Verify safe staging or repair before continuing. Saved work and approval decisions remain in force."
|
||||
: "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated."
|
||||
: "Recovery closed because the task's owner, execution, or status changed. No work was replayed.";
|
||||
const note = executionRecoverySettlementNote({ current, cause: action.cause,
|
||||
workspaceRestoreFailed: hasWorkspaceRestoreFailure(run.resultJson) });
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: task.companyId, issueId: task.id });
|
||||
let nativeFailureBlock = action.evidence.nativeFailureBlock;
|
||||
let nativeBootstrapFailureBlock = action.evidence.nativeBootstrapFailureBlock;
|
||||
|
||||
@@ -453,8 +453,9 @@ export function classifyNativeRunnerRecoveryEvidence(input: {
|
||||
runnerGroupAlive: boolean;
|
||||
remoteSandbox?: boolean;
|
||||
unclaimedBootstrap?: boolean;
|
||||
/** The durable controller checkpoint of a claimed remote bootstrap proves no provider work. */
|
||||
/** Durable evidence proves a claimed remote bootstrap reached no provider work. */
|
||||
remoteBootstrapProvenIncomplete?: boolean;
|
||||
remoteBootstrapIncompleteReason?: "remote_runner_never_launched";
|
||||
processStartMatches: boolean;
|
||||
knownProviderProcessAlive?: boolean;
|
||||
knownProviderProcessIdentityAmbiguous?: boolean;
|
||||
@@ -487,7 +488,8 @@ export function classifyNativeRunnerRecoveryEvidence(input: {
|
||||
// Each same-run bootstrap is another provider attempt; keep restarts bounded.
|
||||
if ((input.providerAttempt ?? 0) >= 3)
|
||||
return { claimKind: null, reason: "execution_recovery_budget_exhausted" };
|
||||
return { claimKind: "bootstrap_incomplete", reason: "remote_bootstrap_proven_incomplete" };
|
||||
return { claimKind: "bootstrap_incomplete",
|
||||
reason: input.remoteBootstrapIncompleteReason ?? "remote_bootstrap_proven_incomplete" };
|
||||
}
|
||||
return input.hasCheckpoint &&
|
||||
input.checkpointIdentityMatches === true &&
|
||||
@@ -547,6 +549,31 @@ export function classifyNativeRunnerRecoveryEvidence(input: {
|
||||
return { claimKind: null, reason: "ambiguous_provider_recovery_evidence" };
|
||||
}
|
||||
|
||||
/** Any record that execution reached a runner: a requested or identified
|
||||
* process, an adapter invocation, a tool start, or a runner-sourced event. */
|
||||
async function hasNativeExecutionStartEvidence(
|
||||
tx: Pick<Db, "select">, run: { companyId: string; id: string },
|
||||
): Promise<boolean> {
|
||||
const [executionEvent] = await tx
|
||||
.select({ id: heartbeatRunEvents.id })
|
||||
.from(heartbeatRunEvents)
|
||||
.where(and(
|
||||
eq(heartbeatRunEvents.companyId, run.companyId),
|
||||
eq(heartbeatRunEvents.runId, run.id),
|
||||
or(
|
||||
isNotNull(heartbeatRunEvents.sourceEventId),
|
||||
inArray(heartbeatRunEvents.eventType, [
|
||||
"adapter.invoke",
|
||||
PROCESS_START_REQUESTED,
|
||||
PROCESS_IDENTITY_RECORDED,
|
||||
"tool.execution.started",
|
||||
]),
|
||||
),
|
||||
))
|
||||
.limit(1);
|
||||
return executionEvent !== undefined;
|
||||
}
|
||||
|
||||
function historyEntry(input: {
|
||||
now: Date;
|
||||
restartKind: NativeRestartKind;
|
||||
@@ -1051,26 +1078,27 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
controller.bootId === (await currentNativeControllerIdentity()).bootId) {
|
||||
return { kind: "awaiting_evidence", runId: row.run.id, reason: "current_controller_preparing" } as const;
|
||||
}
|
||||
const [executionEvent] = await tx
|
||||
.select({ id: heartbeatRunEvents.id })
|
||||
.from(heartbeatRunEvents)
|
||||
.where(and(
|
||||
eq(heartbeatRunEvents.companyId, row.run.companyId),
|
||||
eq(heartbeatRunEvents.runId, row.run.id),
|
||||
or(
|
||||
isNotNull(heartbeatRunEvents.sourceEventId),
|
||||
inArray(heartbeatRunEvents.eventType, [
|
||||
"adapter.invoke",
|
||||
PROCESS_START_REQUESTED,
|
||||
PROCESS_IDENTITY_RECORDED,
|
||||
"tool.execution.started",
|
||||
]),
|
||||
),
|
||||
))
|
||||
.limit(1);
|
||||
unclaimedBootstrap = !executionEvent;
|
||||
unclaimedBootstrap = !(await hasNativeExecutionStartEvidence(tx, row.run));
|
||||
}
|
||||
|
||||
// A claimed remote run whose executor died while still preparing. The
|
||||
// executor durably records native.process_start_requested before any
|
||||
// backend may spawn a runner, and a remote runner exchanges commands only
|
||||
// through a Postgres controller checkpoint. With neither, nothing reached
|
||||
// a runner. The previous controller is gone (takeover was allowed above).
|
||||
const remoteRunnerNeverLaunched = !unclaimedBootstrap && remoteWorkspace?.state === "prepared" && !pluginTask
|
||||
&& !durableAuthority && stagedAttachment === null
|
||||
&& row.run.runnerProfileJson?.nativePreparationFencingVersion === NATIVE_PREPARATION_FENCING_VERSION
|
||||
&& row.coordinator.phase === "observed" && row.coordinator.attempt >= 1 && row.coordinator.controllerGeneration >= 1
|
||||
&& Boolean(row.coordinator.controllerBootId && row.coordinator.controllerPid
|
||||
&& row.coordinator.controllerProcessStartedAt && row.coordinator.leaseOwner)
|
||||
&& !row.coordinator.failureDetail?.successorRunId
|
||||
&& !row.run.processPid && !row.run.processGroupId && !row.run.processStartedAt
|
||||
&& !row.run.sessionIdBefore && !row.run.sessionIdAfter
|
||||
&& Boolean(row.run.nativeSessionId && row.run.runnerInstanceId)
|
||||
&& !hasCheckpoint && !hasProviderEvidence
|
||||
&& !(await hasNativeExecutionStartEvidence(tx, row.run));
|
||||
|
||||
// A claimed remote bootstrap (runner launched, perhaps connected) has no
|
||||
// provider evidence when its controller dies before session.open
|
||||
// completes. The owner's lease has expired above, and the controller
|
||||
@@ -1087,7 +1115,8 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
const classification = classifyNativeRunnerRecoveryEvidence({
|
||||
remoteSandbox: remoteSandbox || Boolean(pluginTask && durableAuthority),
|
||||
unclaimedBootstrap,
|
||||
remoteBootstrapProvenIncomplete,
|
||||
remoteBootstrapProvenIncomplete: remoteBootstrapProvenIncomplete || remoteRunnerNeverLaunched,
|
||||
remoteBootstrapIncompleteReason: remoteRunnerNeverLaunched ? "remote_runner_never_launched" : undefined,
|
||||
runnerPidAlive,
|
||||
runnerGroupAlive,
|
||||
processStartMatches: exactRunnerIdentity,
|
||||
|
||||
+77
-1
@@ -2310,7 +2310,7 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
|
||||
contract: { revision: "1", objective: "Restart bootstrap", criteria: [{ id: "result", requirement: "Complete work" }] } },
|
||||
});
|
||||
const root = resolve(runtimeRoot, `marker-${fixture.runId}`);
|
||||
await mkdir(resolve(root, "control-plane"), { recursive: true, mode: 0o700 });
|
||||
await mkdir(root, { recursive: true, mode: 0o700 });
|
||||
const writeMarker = (runId: string) => writeFile(resolve(root, "control-plane", "control-plane-state.json"), JSON.stringify({
|
||||
schema: "paperclip.runner.external-control-plane-state.v1",
|
||||
identity: { runnerInstanceId: fixture.native.runnerInstanceId, environmentLeaseId: fixture.runId, runId,
|
||||
@@ -2320,6 +2320,15 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
|
||||
controllerGeneration: 2, providerAttempt: 1, restartKind: "hard" as const, recoveryRequestId: null };
|
||||
const target = { kind: "remote", transport: "sandbox", remoteCwd: "/workspace" } as AdapterExecutionTarget;
|
||||
try {
|
||||
// RC-29b: preparation may leave an empty session root before any
|
||||
// controller state exists. It carries no evidence and must not block.
|
||||
await restoreRemoteBootstrapCheckpointForRecovery({ db: fixture.db, execution, root,
|
||||
restartRecovery: undefined, runnerExecutionTarget: target });
|
||||
expect(existsSync(root)).toBe(true);
|
||||
await restoreRemoteBootstrapCheckpointForRecovery({ db: fixture.db, execution, root,
|
||||
restartRecovery: claim, runnerExecutionTarget: target });
|
||||
expect(existsSync(root)).toBe(false);
|
||||
await mkdir(resolve(root, "control-plane"), { recursive: true, mode: 0o700 });
|
||||
await writeMarker(fixture.runId);
|
||||
expect(runnerdStateProvesIncompleteBootstrap(root)).toBe(false);
|
||||
// Only a bootstrap recovery on a classic sandbox restores the checkpoint.
|
||||
@@ -2357,6 +2366,73 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
|
||||
restartKind: "hard", runIds: [fixture.runId] })).toEqual([{ kind: "blocked", runId: fixture.runId, reason }]);
|
||||
});
|
||||
|
||||
// RC-29b: a restart while the executor was still preparing a claimed Daytona
|
||||
// run, before it asked for a runner process. The executor records
|
||||
// native.process_start_requested (and only then may a backend spawn or a
|
||||
// controller checkpoint exist), so its absence proves nothing reached a runner.
|
||||
async function seedUnlaunchedRemoteClaim(label: string, options: {
|
||||
attempt?: number; leaseExpired?: boolean; liveController?: boolean;
|
||||
event?: { eventType: string; sourceEventId?: string };
|
||||
run?: Partial<typeof heartbeatRuns.$inferInsert>;
|
||||
} = {}) {
|
||||
const fixture = await seedRun(`REMOTE-UNLAUNCHED-${label}`);
|
||||
await fixture.db.update(heartbeatRuns).set({
|
||||
runnerProfileJson: { ...fixture.preparedProfile, nativeWorkspaceSync: remotePreparedWorkspace() },
|
||||
...options.run,
|
||||
}).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
if (options.event) await fixture.db.insert(heartbeatRunEvents).values({ companyId, runId: fixture.runId, agentId,
|
||||
seq: 900, stream: "system", level: "info", ...options.event });
|
||||
const current = await currentNativeControllerIdentity();
|
||||
await fixture.db.update(nativeRunFinalizations).set({
|
||||
attempt: options.attempt ?? 1, controllerGeneration: 1,
|
||||
controllerBootId: options.liveController ? current.bootId : randomUUID(),
|
||||
controllerPid: options.liveController ? current.pid : 2_000_000_001,
|
||||
controllerProcessStartedAt: options.liveController ? current.processStartedAt : new Date(Date.now() - 60_000),
|
||||
leaseOwner: `previous:${randomUUID()}`,
|
||||
leaseExpiresAt: new Date(Date.now() + (options.leaseExpired ? -1_000 : 20 * 60_000)),
|
||||
}).where(eq(nativeRunFinalizations.runId, fixture.runId));
|
||||
return fixture;
|
||||
}
|
||||
|
||||
it.each([false, true])("restarts a claimed remote run that never requested a runner (lease expired=%s)", async (leaseExpired) => {
|
||||
const fixture = await seedUnlaunchedRemoteClaim(`PROVEN-${leaseExpired}`, { leaseExpired });
|
||||
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
|
||||
restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({
|
||||
kind: "bootstrap_incomplete", runId: fixture.runId, providerAttempt: 1, controllerGeneration: 2 })]);
|
||||
const [coordinator] = await fixture.db.select().from(nativeRunFinalizations)
|
||||
.where(eq(nativeRunFinalizations.runId, fixture.runId));
|
||||
expect(coordinator).toMatchObject({ phase: "observed", recoveryState: "bootstrap_incomplete", failureCode: null });
|
||||
expect(coordinator.recoveryHistory.at(-1)).toMatchObject({ disposition: "bootstrap_incomplete",
|
||||
reason: "remote_runner_never_launched" });
|
||||
const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
expect(run).toMatchObject({ status: "running", errorCode: null });
|
||||
expect(await fixture.db.select().from(issueRecoveryActions)
|
||||
.where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it.each<[string, Parameters<typeof seedUnlaunchedRemoteClaim>[1], { kind: string; reason: string }]>([
|
||||
["a runner process start was requested", { event: { eventType: "native.process_start_requested" } },
|
||||
{ kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }],
|
||||
["a runner process identity was recorded", { event: { eventType: "native.process_identity_recorded" } },
|
||||
{ kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }],
|
||||
["the adapter was invoked", { event: { eventType: "adapter.invoke" } },
|
||||
{ kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }],
|
||||
["a runner event was committed", { event: { eventType: "item.updated", sourceEventId: "runner-event-1" } },
|
||||
{ kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }],
|
||||
["a process pid is recorded", { run: { processPid: 443 } },
|
||||
{ kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }],
|
||||
["a provider session is recorded", { run: { sessionIdAfter: "provider-session" } },
|
||||
{ kind: "blocked", reason: "ambiguous_remote_recovery_evidence" }],
|
||||
["the bootstrap budget is spent", { attempt: 3 },
|
||||
{ kind: "blocked", reason: "execution_recovery_budget_exhausted" }],
|
||||
["the claiming controller is still alive", { liveController: true },
|
||||
{ kind: "awaiting_evidence", reason: "controller_still_alive" }],
|
||||
])("keeps a claimed unlaunched remote run held when %s", async (label, options, expected) => {
|
||||
const fixture = await seedUnlaunchedRemoteClaim(label.replace(/\W+/g, "-"), options);
|
||||
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
|
||||
restartKind: "hard", runIds: [fixture.runId] })).toEqual([expect.objectContaining({ runId: fixture.runId, ...expected })]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ attempt: 2, driverKind: "codex_app_server" },
|
||||
{ attempt: 3, driverKind: "codex_app_server" },
|
||||
|
||||
@@ -126,6 +126,7 @@ import {
|
||||
opendirSync,
|
||||
readFileSync,
|
||||
readdirSync,
|
||||
rmdirSync,
|
||||
readlinkSync,
|
||||
readSync,
|
||||
renameSync,
|
||||
@@ -6044,8 +6045,22 @@ export async function restoreRemoteBootstrapCheckpointForRecovery(input: {
|
||||
}): Promise<void> {
|
||||
const target = input.runnerExecutionTarget;
|
||||
if (input.restartRecovery?.kind !== "bootstrap_incomplete" || target?.kind !== "remote"
|
||||
|| target.transport !== "sandbox" || target.runnerTask || !existsSync(input.root)
|
||||
|| readRunnerdDurableIdentity(input.root) !== null) return;
|
||||
|| target.transport !== "sandbox" || target.runnerTask || !isSafeNativeStateDirectory(input.root)) return;
|
||||
// Preparation can create the session root before the executor opens any
|
||||
// controller state. An empty root holds no evidence; remove it so the fresh
|
||||
// bootstrap does not read it as an unreadable identity.
|
||||
if (readdirSync(input.root).length === 0) {
|
||||
try {
|
||||
rmdirSync(input.root);
|
||||
return;
|
||||
} catch (error) {
|
||||
// An entry that appeared after the read goes through the checks below.
|
||||
const code = (error as NodeJS.ErrnoException).code;
|
||||
if (code === "ENOENT") return;
|
||||
if (code !== "ENOTEMPTY" && code !== "EEXIST") throw error;
|
||||
}
|
||||
}
|
||||
if (readRunnerdDurableIdentity(input.root) !== null) return;
|
||||
const marker = readRunnerdDurableIdentity(input.root, true);
|
||||
if (!marker || marker.runId !== input.execution.binding.runId) return;
|
||||
await restoreNativeRunnerCheckpoint(input.db, input.execution.binding, input.root, nativeSessionKey(input.execution));
|
||||
|
||||
Reference in new issue
Block a user