Files
PaperClipAI/server/src/__tests__/heartbeat-native-predispatch-retry.test.ts
T
DottaandClaude Opus 5.5 890dcbdd66 fix(native): wait for a live warm session owner during a deploy overlap (#21627)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap: the outgoing server still
serves while the incoming server starts new runs
> - A warm agent session stays open in the server process that ran the
last turn, after that run finished
> - The incoming server took over a warm session while the outgoing
server was alive and still held it, because the finished run's
finalization lease ends at commit
> - The takeover then failed, the task went blocked with a Retry, and in
one case a retry loop ran 27 times on the same sandbox
> - This pull request makes the incoming server wait while the server
that holds the warm session is alive, makes the outgoing server close
idle warm sessions as soon as it leaves new runs to the replacement, and
stops the retry loop when the promised fresh sandbox does not come
> - The benefit is that a follow-up during a deploy finishes without a
person, and no Runner ever gets two controllers

## Linked Issues or Issue Description

**What happened?**
A zero-downtime release-swap test (Gate B, wave B1, master aaec67b891)
found two failures with one cause.
Task 1: run A finished on the outgoing server at 15:13:56. Its warm
session stayed idle in that server. The incoming server claimed the
follow-up at 15:13:57. `inspectRemoteWarmSessionRecovery` asked
`evaluateNativeControllerTakeover`, which allowed the takeover with
`remote_controller_lease_released_or_expired`. That lease covers the
finished run, not the warm session. The outgoing server was alive until
15:18:18. `runner.prp.authenticate` failed after 62.3 s, the run failed
with `remote_warm_session_unproven`, the failure was saved as ambiguous
and owned by the board, and the task stayed blocked with a Retry.
Task 2: the same shape on a later swap started 27 automatic runs in 13
minutes. Fourteen of them said "The next attempt starts on a fresh
sandbox", but each one resumed the same sandbox and got the same refusal
(`journal_unsettled`).

**Expected behavior**
A follow-up claimed during an overlap waits for the warm session owner
to let go, then continues with no Retry banner and no second controller.
If the outgoing server dies with the session open, the takeover
continues after its ownership ends. A message that promises a fresh
sandbox is true, or the retries stop after a bounded number with a true
state.

**Steps to reproduce**
1. Run the restart-chaos release-swap preset with a warm (opencode or
codex) staging mix.
2. Let a run finish on the outgoing server after the traffic switch,
with a follow-up comment already queued.
3. See the follow-up fail with `remote_warm_session_unproven` and the
task go blocked.

**Paperclip version or commit**
master aaec67b891 and 45531b719e.

**Deployment mode**
Hosted deploy overlap (release swap) with Daytona sandboxes.

## What Changed

- `remote-warm-session-recovery.ts`: before live adoption, the run
proves that the server process that ran the finished turn is gone. It
uses the shared owner proof (`proveOwnerGone`): a PID check in the same
process space, or the `controller_instances` lease in another process
space. While that server is alive, the run defers with the existing
`native_controller_recovery_waiting_for_owner` wait and checks again
after 5 s. The wait ends when the server exits or is fenced, or when the
finished turn's warm window (idle timeout plus two minutes) ends.
- `native-session-executor.ts` and `run-start-handover.ts`: a server
that leaves new run starts to a replacement closes its idle warm
sessions after 1 s, through the existing idle-timeout close. A session
that becomes idle after that closes after 1 s too. This is the same
close that shutdown already does, done earlier.
- `predecessor-retirement-fallback.ts`: when a retry meets an unretired
owner on the same sandbox that its source already asked to replace, it
does not ask again. The run fails without another automatic retry, and
the task shows a Retry.
- `predecessor-retirement.ts`: the error no longer says "The next
attempt starts on a fresh sandbox", because the replacement is only a
request.
- `doc/execution-semantics.md`: documents the owner wait, the hand-off
at the swap, and the retry stop.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts`
- New: "waits while the outgoing server that ran the finished turn still
holds its warm session, then takes it over once that server exits" (red
on master: master returned a takeover while the owner was alive).
- New: "waits for a live outgoing server in another process space until
its controller lease ends" (red on master).
- New: "stops waiting for a live outgoing server once the finished
turn's warm window has closed" (bound check).
- `cd server && npx vitest run
src/__tests__/heartbeat-native-predispatch-retry.test.ts`
- New: "stops retrying with a truthful failure when the next attempt met
the same unretired sandbox instead of a fresh one" (red on master:
master scheduled another retry on the same sandbox).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"replacement server starts"`
- New: "closes a warm session promptly when a replacement server starts
this server's runs (before/after the turn ends)" (both red on master).
- Focused suites: native-session-executor, remote-warm-session-recovery,
native-controller-recovery, native-recovered-owner-followup,
native-runner-restart-recovery, heartbeat-native-predispatch-retry,
environment-runtime-reusable-resume-replacement,
task-interruption-presentation, sandbox-runner-port-holders.
- `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify` exits
2 (a doc is a global CI input), so required CI supplies broad coverage.
- Not run here: the restart-chaos release-swap wave. The maintainer runs
it after merge.

## Risks

- A follow-up can now wait up to the outgoing server's remaining overlap
when that server cannot close its idle session. Before, it failed and
needed a Retry. The wait is bounded by the owner's exit or the warm
window.
- The outgoing server closes idle warm sessions earlier during an
overlap. The next turn then resumes from a suspended Runner instead of a
live one, as it already does after a shutdown.
- A retry chain that meets the same unretired sandbox twice now stops
one attempt earlier than the shared cap. The task shows a Retry instead
of looping.
- Rows without a recorded process space keep the existing one-release
local proof, so they behave as before outside the same host.
- Previous-release state: this change adds no persisted field.
Finalization rows saved by the previous release already record the
controller boot, PID, process space and start time, so they carry
forward. A follow-up already deferred by the previous release resumes
through the same wait code.
- Runtime cost: the owner proof adds one `ps` start-time read (same
process space) or one `controller_instances` row read (other process
space) per inspection. It runs once per attempt, at most every 5 s while
a live owner holds the session.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:59:17 -06:00

521 lines
36 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { existsSync, mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { and, eq, inArray } from "drizzle-orm";
import { activityLog, agents, authUsers, companyMemberships, createDb, environmentLeases, environments, heartbeatRuns, issueComments,
issueRecoveryActions, issueRelations, issues } from "@paperclipai/db";
import { NativePredecessorRetirementUnprovenError } from "../services/native-runtime/predecessor-retirement.js";
import { NativeProviderArtifactTransferTimeoutError } from "../services/native-runtime/native-provider-artifacts.js";
import { taskInterruptionPresentation } from "../services/task-interruption-presentation.js";
import type { NativeRunIdentity, NativeSession, NativeSessionBackend, PrpEvent } from "@paperclipai/paperclip-runner";
import type { ExecutionContinuationEnvelope } from "@paperclipai/shared";
import { renderPaperclipWakePrompt } from "@paperclipai/adapter-utils/server-utils";
import { describe, expect, it, vi } from "vitest";
import { heartbeatService, type HeartbeatServiceOptions } from "../services/heartbeat.js";
import { instanceSettingsService } from "../services/instance-settings.js";
import { closeIdleWarmNativeSessionsForRestart, scopedRunnerdStateRoot } from "../services/native-runtime/native-session-executor.js";
import { nativeRestartFixture, waitForCondition } from "./helpers/native-restart-fixture.js";
import { getRecoveryPostgresSupport } from "./helpers/recovery-postgres-fixture.js";
// Budget admission is the last database write before the dispatch gate enters
// the native executor. Exhausted serialization retries there fail the run with
// no provider work started.
const admissionFaults = vi.hoisted(() => ({ remaining: 0 }));
vi.mock("../services/budget-reservations.js", async importOriginal => {
const actual = await importOriginal<typeof import("../services/budget-reservations.js")>();
const { AccountingContentionError } = await import("../services/accounting-transaction.js");
return {
...actual,
reserveRunBudget: async (...args: Parameters<typeof actual.reserveRunBudget>) => {
if (admissionFaults.remaining > 0) {
admissionFaults.remaining -= 1;
throw new AccountingContentionError(Object.assign(new Error("could not serialize access due to concurrent update"),
{ name: "PostgresError", code: "40001" }), 4);
}
return actual.reserveRunBudget(...args);
},
};
});
const support = await getRecoveryPostgresSupport({});
type AdmissionFailure = { remaining: number; turnNeverIssued: boolean; environmentId: string; provider: string;
providerLeaseId: string; leasePolicy: "reuse_by_environment" | "reuse_by_execution_workspace"; failedRuns: string[];
sessionRoots: string[] };
function nativeBackend(db: ReturnType<typeof createDb>, options: { failDescriptorOnce?: { failed: boolean };
unretiredPredecessorOnce?: { failed: boolean; environmentId: string; taskBlocked?: boolean; repeat?: string[] }; turnMessages?: Map<string, string>;
admissionFailure?: AdmissionFailure } = {}):
NonNullable<HeartbeatServiceOptions["nativeSessionBackendFactory"]> {
return execution => {
let identity: NativeRunIdentity;
let sourceInstanceId = "";
let turn = Promise.withResolvers<string>();
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true };
const session: NativeSession = {
identity: () => identity,
capabilities: async () => capabilities,
attachRun: async input => { identity = input.identity; },
startTurn: async input => {
options.turnMessages?.set(identity.runId, JSON.stringify(input.message));
const turnId = randomUUID();
turn.resolve(turnId);
return { turnId };
},
async *events() {
const turnId = await turn.promise;
const event = (sourceSeq: number, eventType: PrpEvent["eventType"], payload: Record<string, unknown>): PrpEvent => ({
schema: "paperclip.prp.event.v1", sourceKind: "runner", sourceInstanceId,
sourceEventId: `${sourceInstanceId}:${identity.runId}:${sourceSeq}`, sourceSeq,
runId: identity.runId, normalizedSessionId: identity.sessionId, turnId,
schemaVersion: 1, priority: 0, emittedAt: new Date().toISOString(), eventType, payload,
});
yield event(1, "turn.started", {});
yield event(2, "item.completed", { kind: "agentMessage", channel: "final", text: "Done." });
turn = Promise.withResolvers<string>();
yield event(3, "turn.completed", { status: "completed" });
},
result: async () => null,
snapshot: async () => ({ backendKind: "mock", sessionId: "predispatch-session", identity,
providerSessionId: "predispatch-provider-session", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }),
close: async () => {},
};
const backend: NativeSessionBackend = {
descriptor: async () => {
if (options.failDescriptorOnce && !options.failDescriptorOnce.failed) {
options.failDescriptorOnce.failed = true;
throw Object.assign(new Error("could not serialize access due to concurrent update"), { name: "PostgresError", code: "40001" });
}
const unretired = options.unretiredPredecessorOnce;
if (unretired && (!unretired.failed || unretired.repeat)) {
unretired.repeat?.push(execution.binding.runId);
// Staging: this run resumed the task's reused sandbox, where
// the previous run's warm Runner could not be retired before any
// provider work. The executor reports that after its coordinator claim.
unretired.failed = true;
if (unretired.taskBlocked) await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, execution.binding.issueId));
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, execution.binding.runId));
await db.insert(environmentLeases).values({ companyId: run!.companyId, environmentId: unretired.environmentId,
issueId: execution.binding.issueId, heartbeatRunId: run!.id, provider: "daytona", providerLeaseId: "sandbox-reused",
leasePolicy: "reuse_by_environment", status: "active", metadata: { remoteCwd: "/workspace" } });
throw new NativePredecessorRetirementUnprovenError("previous-run", "sandbox-reused",
new Error("Controller journal drain timed out; preserve the session before replacing its writer."));
}
const [run] = await db.select({ runnerInstanceId: heartbeatRuns.runnerInstanceId }).from(heartbeatRuns)
.where(eq(heartbeatRuns.id, execution.binding.runId));
if (!run?.runnerInstanceId) throw new Error("Normal native admission did not persist a runner identity");
sourceInstanceId = run.runnerInstanceId;
return { kind: "mock", name: "native-predispatch-retry", version: "1", capabilities,
runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } };
},
openSession: async input => { identity = input.identity; return session; },
recoverSession: async snapshot => { identity = snapshot.identity; return { recovered: true, session }; },
};
const failure = options.admissionFailure;
if (!failure) return backend;
return Object.assign(backend, {
providerTurnNeverIssued: () => failure.turnNeverIssued,
providerAdmissionPersisted: async () => {
if (failure.remaining <= 0) return;
failure.remaining -= 1;
const runId = execution.binding.runId;
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
await db.insert(environmentLeases).values({ companyId: run!.companyId, environmentId: failure.environmentId,
issueId: execution.binding.issueId, heartbeatRunId: runId, provider: failure.provider,
providerLeaseId: `${failure.providerLeaseId}-${failure.failedRuns.length}`, leasePolicy: failure.leasePolicy,
status: "active", metadata: { remoteCwd: "/workspace" } });
failure.failedRuns.push(runId);
const root = scopedRunnerdStateRoot(execution);
mkdirSync(root, { recursive: true, mode: 0o700 });
writeFileSync(join(root, "failed-attempt"), runId);
failure.sessionRoots.push(root);
throw new NativeProviderArtifactTransferTimeoutError("upload", 120_000, 512 * 1024 * 1024);
},
});
};
}
describe.skipIf(!support.supported)("heartbeat native pre-dispatch failure retry", () => {
const fixture = nativeRestartFixture({});
async function startTask(db: ReturnType<typeof createDb>, heartbeat: ReturnType<typeof heartbeatService>,
status: "in_progress" | "blocked" = "in_progress") {
const { companyId, agentId } = fixture.current;
const userId = randomUUID();
await db.insert(authUsers).values({ id: userId, name: "Task owner", email: `${userId}@example.invalid`,
createdAt: new Date(), updatedAt: new Date() });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "owner" });
await db.update(agents).set({ status: "idle", adapterConfig: { provider: "codex" } }).where(eq(agents.id, agentId));
await instanceSettingsService(db).updateExperimental({ enableAgentChat: true, enableNativeRunner: true });
const [issue] = await db.insert(issues).values({ companyId, title: "Survive a database fault", status,
assigneeAgentId: agentId, responsibleUserId: userId, conversationAgentId: agentId, conversationUserId: userId,
conversationState: "active" }).returning();
if (status === "blocked") {
const [blocker] = await db.insert(issues).values({ companyId, title: "Open subtask", status: "todo" }).returning();
await db.insert(issueRelations).values({ companyId, issueId: blocker!.id, relatedIssueId: issue!.id, type: "blocks" });
}
const [message] = await db.insert(issueComments).values({ companyId, issueId: issue!.id,
authorType: "user", authorUserId: userId, body: "Start." }).returning();
const run = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual", reason: "issue_commented",
requestedByActorType: "user", requestedByActorId: userId, manualUserWake: true,
payload: { issueId: issue!.id, commentId: message!.id },
contextSnapshot: { issueId: issue!.id, taskKey: issue!.id, wakeCommentId: message!.id } });
expect(run).not.toBeNull();
await waitForCondition("the first run to settle", () => heartbeat.getTaskDrainStatus().quiescent);
return { issueId: issue!.id, runId: run!.id };
}
it("re-queues a native run that failed at budget admission and completes it without a Retry", async () => {
const { companyId } = fixture.current;
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db) });
try {
admissionFaults.remaining = 1;
const { issueId, runId } = await startTask(db, heartbeat);
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(failed).toMatchObject({ status: "failed", runtimeMode: "native", errorCode: "adapter_failed",
resultJson: expect.objectContaining({ failureCause: "40001",
nativePreDispatchFailure: { version: 1, providerWorkStarted: false } }) });
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.retryOfRunId, runId)));
expect(successors).toEqual([expect.objectContaining({ scheduledRetryReason: "interrupted_preparation",
scheduledRetryAttempt: 1 })]);
expect(successors[0]!.contextSnapshot?.executionRetryAccounting).toMatchObject({ failureRetries: 0 });
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]?.status).not.toBe("blocked");
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
// The first attempt is due at once; the scheduler promotes it and the
// queue starts it, exactly once.
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 1 });
await heartbeat.resumeQueuedRuns();
await waitForCondition("the retry to succeed", async () =>
(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successors[0]!.id)))[0]?.status === "succeeded");
await waitForCondition("the retry to settle", () => heartbeat.getTaskDrainStatus().quiescent);
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.retryOfRunId, runId)))).toHaveLength(1);
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
} finally {
admissionFaults.remaining = 0;
await heartbeat.drainActiveRunExecutions();
await closeIdleWarmNativeSessionsForRestart();
await db.$client.end();
}
}, 90_000);
it("re-queues a run whose predecessor could not be retired onto a fresh sandbox without a Retry", async () => {
const { companyId } = fixture.current;
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
const [environment] = await db.insert(environments).values({ name: `reused-${randomUUID()}`, driver: "sandbox" }).returning();
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db,
{ unretiredPredecessorOnce: { failed: false, environmentId: environment!.id } }) });
try {
const { issueId, runId } = await startTask(db, heartbeat);
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(failed).toMatchObject({ status: "failed", runtimeMode: "native", errorCode: "adapter_failed",
resultJson: expect.objectContaining({ failureCause: "native_predecessor_retirement_unproven",
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
predecessorRetirementFallback: expect.objectContaining({ version: 1, providerWorkStarted: false,
predecessorRunId: "previous-run", replacedProviderLeaseIds: ["sandbox-reused"] }) }) });
// The reused sandbox is replaced at the next acquisition, not resumed.
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.heartbeatRunId, runId),
eq(environmentLeases.providerLeaseId, "sandbox-reused")));
expect(lease?.metadata).toMatchObject({ replaceBeforeReuse: { schema: "paperclip.replace-before-reuse.v1", runId,
predecessorRunId: "previous-run" } });
expect(await db.select().from(activityLog).where(and(eq(activityLog.runId, runId),
eq(activityLog.action, "environment.lease_replacement_requested")))).toHaveLength(1);
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.retryOfRunId, runId)));
expect(successors).toEqual([expect.objectContaining({ scheduledRetryReason: "interrupted_preparation", scheduledRetryAttempt: 1 })]);
expect(successors[0]!.contextSnapshot?.executionRetryAccounting).toMatchObject({ failureRetries: 0 });
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]?.status).not.toBe("blocked");
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 1 });
await heartbeat.resumeQueuedRuns();
await waitForCondition("the retry to succeed", async () =>
(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successors[0]!.id)))[0]?.status === "succeeded");
await waitForCondition("the retry to settle", () => heartbeat.getTaskDrainStatus().quiescent);
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
} finally {
await heartbeat.drainActiveRunExecutions();
await closeIdleWarmNativeSessionsForRestart();
await db.$client.end();
}
}, 90_000);
it("stops retrying with a truthful failure when the next attempt met the same unretired sandbox instead of a fresh one", async () => {
const { companyId } = fixture.current;
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
const [environment] = await db.insert(environments).values({ name: `reused-${randomUUID()}`, driver: "sandbox" }).returning();
const attempts: string[] = [];
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db,
{ unretiredPredecessorOnce: { failed: false, environmentId: environment!.id, repeat: attempts } }) });
try {
const { issueId, runId } = await startTask(db, heartbeat);
const [first] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.retryOfRunId, runId)));
expect(first).toMatchObject({ scheduledRetryReason: "interrupted_preparation", scheduledRetryAttempt: 1 });
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 1 });
await heartbeat.resumeQueuedRuns();
await waitForCondition("the second attempt to end", async () =>
(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, first!.id)))[0]?.status === "failed");
await waitForCondition("the second attempt to settle", () => heartbeat.getTaskDrainStatus().quiescent);
expect(attempts).toEqual([runId, first!.id]);
const [second] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, first!.id));
expect(second?.resultJson ?? {}).not.toHaveProperty("predecessorRetirementFallback");
expect(second?.error ?? "").not.toMatch(/fresh sandbox/i);
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.retryOfRunId, first!.id)))).toHaveLength(0);
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 0 });
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
expect(await taskInterruptionPresentation(db, issue!)).toMatchObject({ canRetry: true });
} finally {
await heartbeat.drainActiveRunExecutions();
await closeIdleWarmNativeSessionsForRestart();
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"])));
await db.$client.end();
}
}, 90_000);
it.each([
{ title: "continues a follow-up on a blocked task whose predecessor could not be retired", blockedAfter: false },
{ title: "does not continue a task that was blocked only after its run failed", blockedAfter: true },
])("$title", async ({ blockedAfter }) => {
const { companyId } = fixture.current;
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
const [environment] = await db.insert(environments).values({ name: `reused-${randomUUID()}`, driver: "sandbox" }).returning();
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db,
{ unretiredPredecessorOnce: { failed: false, environmentId: environment!.id, taskBlocked: !blockedAfter } }) });
try {
const { issueId, runId } = await startTask(db, heartbeat, blockedAfter ? "in_progress" : "blocked");
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(failed).toMatchObject({ status: "failed", resultJson: expect.objectContaining({
failureCause: "native_predecessor_retirement_unproven",
predecessorRetirementFallback: expect.objectContaining({ issueStatusAtFailure: blockedAfter ? "in_progress" : "blocked" }) }) });
if (blockedAfter) await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, issueId));
const [task] = await db.select().from(issues).where(eq(issues.id, issueId));
expect(task?.status).toBe("blocked");
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.retryOfRunId, runId)));
expect(successors).toEqual([expect.objectContaining({ status: "scheduled_retry", scheduledRetryReason: "interrupted_preparation" })]);
expect(await taskInterruptionPresentation(db, task!)).toMatchObject({ state: "working", canRetry: false });
if (blockedAfter) {
await heartbeat.promoteDueScheduledRetries();
await heartbeat.resumeQueuedRuns();
await waitForCondition("the stale retry to settle", () => heartbeat.getTaskDrainStatus().quiescent);
const [stale] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successors[0]!.id));
expect(["cancelled", "scheduled_retry"]).toContain(stale?.status);
return;
}
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 1 });
await heartbeat.resumeQueuedRuns();
await waitForCondition("the retry to settle", () => heartbeat.getTaskDrainStatus().quiescent);
const [retried] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successors[0]!.id));
expect({ status: retried?.status, errorCode: retried?.errorCode, error: retried?.error }).toEqual({ status: "succeeded", errorCode: null, error: null });
} finally {
await heartbeat.drainActiveRunExecutions();
await closeIdleWarmNativeSessionsForRestart();
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"])));
await db.$client.end();
}
}, 90_000);
it("delivers the monitor check to the automatic retry of a monitor run that failed before provider work", async () => {
const { companyId, agentId } = fixture.current;
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
const [environment] = await db.insert(environments).values({ name: `reused-${randomUUID()}`, driver: "sandbox" }).returning();
const turnMessages = new Map<string, string>();
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db,
{ unretiredPredecessorOnce: { failed: false, environmentId: environment!.id }, turnMessages }) });
try {
await db.update(agents).set({ status: "idle", adapterConfig: { provider: "codex" } }).where(eq(agents.id, agentId));
await instanceSettingsService(db).updateExperimental({ enableNativeRunner: true });
const userId = randomUUID();
await db.insert(authUsers).values({ id: userId, name: "Task owner", email: `${userId}@example.invalid`,
createdAt: new Date(), updatedAt: new Date() });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "owner" });
const [issue] = await db.insert(issues).values({ companyId, title: "Phase 2 on the monitor check", status: "in_progress",
assigneeAgentId: agentId, responsibleUserId: userId, monitorNotes: "Phase 2 check" }).returning();
const run = await heartbeat.wakeup(agentId, { source: "automation", triggerDetail: "system", reason: "issue_monitor_due",
payload: { issueId: issue!.id }, contextSnapshot: { issueId: issue!.id, taskKey: issue!.id,
source: "issue.monitor", wakeReason: "issue_monitor_due", monitorNotes: "Phase 2 check" } });
expect(run).not.toBeNull();
await waitForCondition("the monitor run to settle", () => heartbeat.getTaskDrainStatus().quiescent);
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id));
expect(failed).toMatchObject({ status: "failed", resultJson: expect.objectContaining({
failureCause: "native_predecessor_retirement_unproven" }) });
const [successor] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.retryOfRunId, run!.id)));
expect(successor).toMatchObject({ scheduledRetryReason: "interrupted_preparation" });
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 1 });
await heartbeat.resumeQueuedRuns();
await waitForCondition("the retry to start its provider turn", () => turnMessages.has(successor!.id));
await waitForCondition("the retry to settle", () => heartbeat.getTaskDrainStatus().quiescent);
const [retried] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successor!.id));
const continuation = retried!.contextSnapshot?.executionContinuation as ExecutionContinuationEnvelope;
expect(continuation.trigger).toMatchObject({ sourceRunId: run!.id, retriedReason: "issue_monitor_due" });
expect(renderPaperclipWakePrompt(retried!.contextSnapshot?.paperclipWake)).toContain("- retried wake: issue_monitor_due");
expect(turnMessages.get(successor!.id)).toContain("- retried wake: issue_monitor_due");
expect(turnMessages.has(run!.id)).toBe(false);
} finally {
await heartbeat.drainActiveRunExecutions();
await closeIdleWarmNativeSessionsForRestart();
await db.$client.end();
}
}, 90_000);
it.each([
{ target: "remote sandbox", provider: "e2b", leasePolicy: "reuse_by_execution_workspace" as const },
{ target: "plugin-task sandbox", provider: "daytona", leasePolicy: "reuse_by_environment" as const },
])("continues on a fresh $target when the provider artifacts fail to save before any turn", async ({ provider, leasePolicy }) => {
const { companyId } = fixture.current;
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
const [environment] = await db.insert(environments).values({ name: `capture-${randomUUID()}`, driver: "sandbox" }).returning();
const turnMessages = new Map<string, string>();
const failure: AdmissionFailure = { remaining: 1, turnNeverIssued: true, environmentId: environment!.id, provider,
providerLeaseId: `sandbox-${provider}`, leasePolicy, failedRuns: [], sessionRoots: [] };
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, { admissionFailure: failure, turnMessages }) });
try {
const { issueId, runId } = await startTask(db, heartbeat);
expect(failure.failedRuns).toEqual([runId]);
expect(turnMessages.has(runId)).toBe(false);
expect(existsSync(failure.sessionRoots[0]!)).toBe(false);
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(failed).toMatchObject({ status: "failed", runtimeMode: "native",
resultJson: expect.objectContaining({ failureCause: "native_failed_before_provider_turn",
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
predecessorRetirementFallback: expect.objectContaining({ version: 1, providerWorkStarted: false,
predecessorRunId: null, replacedProviderLeaseIds: [`sandbox-${provider}-0`] }) }) });
expect(failed!.error).toBe("Setting up the computer took too long, so the task is continuing on a fresh one.");
expect(failed!.error).not.toMatch(/[a-z0-9]+_[a-z0-9_]+|\d|\(|\)/i);
expect(failed!.runnerProfileJson).toMatchObject({ failedBeforeProviderTurn: {
cause: "native_failed_before_provider_turn", technicalCause: "native_provider_artifacts_transfer_timeout" } });
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.heartbeatRunId, runId),
eq(environmentLeases.providerLeaseId, `sandbox-${provider}-0`)));
expect(lease?.metadata).toMatchObject({ replaceBeforeReuse: { schema: "paperclip.replace-before-reuse.v1", runId } });
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.retryOfRunId, runId)));
expect(successors).toEqual([expect.objectContaining({ scheduledRetryReason: "interrupted_preparation", scheduledRetryAttempt: 1 })]);
expect(successors[0]!.contextSnapshot?.executionRetryAccounting).toMatchObject({ failureRetries: 0 });
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
expect(issue!.status).not.toBe("blocked");
expect(await taskInterruptionPresentation(db, issue!)).toMatchObject({ state: "working", canRetry: false });
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 1 });
await heartbeat.resumeQueuedRuns();
await waitForCondition("the continuation to succeed", async () =>
(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successors[0]!.id)))[0]?.status === "succeeded");
await waitForCondition("the continuation to settle", () => heartbeat.getTaskDrainStatus().quiescent);
expect(turnMessages.has(successors[0]!.id)).toBe(true);
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.retryOfRunId, runId)))).toHaveLength(1);
const [settled] = await db.select().from(issues).where(eq(issues.id, issueId));
expect(await taskInterruptionPresentation(db, settled!)).toMatchObject({ state: "resolved", canRetry: false });
} finally {
await heartbeat.drainActiveRunExecutions();
await closeIdleWarmNativeSessionsForRestart();
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"])));
await db.$client.end();
}
}, 90_000);
it("keeps today's ambiguous hold when the Runner ledger cannot prove no turn was sent", async () => {
const { companyId } = fixture.current;
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
const [environment] = await db.insert(environments).values({ name: `capture-${randomUUID()}`, driver: "sandbox" }).returning();
const failure: AdmissionFailure = { remaining: 1, turnNeverIssued: false, environmentId: environment!.id, provider: "daytona",
providerLeaseId: "sandbox-turn-maybe-sent", leasePolicy: "reuse_by_environment", failedRuns: [], sessionRoots: [] };
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, { admissionFailure: failure }) });
try {
const { runId } = await startTask(db, heartbeat);
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(failed?.status).toBe("failed");
expect(failed?.resultJson ?? {}).not.toHaveProperty("predecessorRetirementFallback");
expect(failed?.resultJson?.failureCause).not.toBe("native_failed_before_provider_turn");
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.heartbeatRunId, runId),
eq(environmentLeases.providerLeaseId, "sandbox-turn-maybe-sent-0")));
expect(lease?.metadata ?? {}).not.toHaveProperty("replaceBeforeReuse");
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.retryOfRunId, runId), eq(heartbeatRuns.scheduledRetryReason, "interrupted_preparation")))).toHaveLength(0);
} finally {
await heartbeat.drainActiveRunExecutions();
await closeIdleWarmNativeSessionsForRestart();
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"])));
await db.$client.end();
}
}, 90_000);
it("backs off a lasting pre-turn failure and stops at the shared lane cap without hammering", async () => {
const { companyId } = fixture.current;
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
const [environment] = await db.insert(environments).values({ name: `capture-${randomUUID()}`, driver: "sandbox" }).returning();
const failure: AdmissionFailure = { remaining: Number.POSITIVE_INFINITY, turnNeverIssued: true, environmentId: environment!.id,
provider: "daytona", providerLeaseId: "sandbox-lasting", leasePolicy: "reuse_by_environment", failedRuns: [], sessionRoots: [] };
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, { admissionFailure: failure }) });
const successorOf = async (runId: string) => (await db.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.retryOfRunId, runId))));
const runToEnd = async (runId: string) => {
await heartbeat.resumeQueuedRuns();
await waitForCondition("the attempt to end", async () =>
(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]?.status === "failed");
await waitForCondition("the attempt to settle", () => heartbeat.getTaskDrainStatus().quiescent);
};
try {
const { issueId, runId } = await startTask(db, heartbeat);
const [first] = await successorOf(runId);
expect(first).toMatchObject({ scheduledRetryReason: "interrupted_preparation", scheduledRetryAttempt: 1 });
expect((await heartbeat.promoteDueScheduledRetries()).runIds).toEqual([first!.id]);
const secondStartedAt = Date.now();
await runToEnd(first!.id);
const [second] = await successorOf(first!.id);
expect(second).toMatchObject({ scheduledRetryReason: "interrupted_preparation", scheduledRetryAttempt: 2 });
expect(second!.scheduledRetryAt!.getTime() - secondStartedAt).toBeGreaterThanOrEqual(15_000);
expect((await heartbeat.promoteDueScheduledRetries()).runIds).not.toContain(second!.id);
expect(failure.failedRuns).toHaveLength(2);
const [waiting] = await db.select().from(issues).where(eq(issues.id, issueId));
expect(await taskInterruptionPresentation(db, waiting!)).toMatchObject({ state: "working", canRetry: false });
await db.update(heartbeatRuns).set({ scheduledRetryAttempt: 12, scheduledRetryAt: new Date(Date.now() - 1_000),
contextSnapshot: { ...second!.contextSnapshot, executionRetryAccounting: {
...(second!.contextSnapshot?.executionRetryAccounting as Record<string, unknown>), interruptedPreparations: 12 } } })
.where(eq(heartbeatRuns.id, second!.id));
expect((await heartbeat.promoteDueScheduledRetries()).runIds).toEqual([second!.id]);
await runToEnd(second!.id);
expect(await successorOf(second!.id)).toHaveLength(0);
expect(failure.failedRuns).toHaveLength(3);
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
expect(await taskInterruptionPresentation(db, issue!)).toMatchObject({ state: "failed", canRetry: true });
} finally {
await heartbeat.drainActiveRunExecutions();
await closeIdleWarmNativeSessionsForRestart();
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"])));
await db.$client.end();
}
}, 120_000);
it("keeps the hold when the failure follows entry into the native executor", async () => {
const { companyId } = fixture.current;
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, { failDescriptorOnce: { failed: false } }) });
try {
const { runId } = await startTask(db, heartbeat);
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(failed?.status).toBe("failed");
expect(failed?.resultJson ?? {}).not.toHaveProperty("nativePreDispatchFailure");
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.retryOfRunId, runId), eq(heartbeatRuns.scheduledRetryReason, "interrupted_preparation")))).toHaveLength(0);
} finally {
await heartbeat.drainActiveRunExecutions();
await closeIdleWarmNativeSessionsForRestart();
await db.$client.end();
}
}, 90_000);
});