mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap: the outgoing server still serves while the incoming server starts new runs > - A warm agent session stays open in the server process that ran the last turn, after that run finished > - The incoming server took over a warm session while the outgoing server was alive and still held it, because the finished run's finalization lease ends at commit > - The takeover then failed, the task went blocked with a Retry, and in one case a retry loop ran 27 times on the same sandbox > - This pull request makes the incoming server wait while the server that holds the warm session is alive, makes the outgoing server close idle warm sessions as soon as it leaves new runs to the replacement, and stops the retry loop when the promised fresh sandbox does not come > - The benefit is that a follow-up during a deploy finishes without a person, and no Runner ever gets two controllers ## Linked Issues or Issue Description **What happened?** A zero-downtime release-swap test (Gate B, wave B1, masteraaec67b891) found two failures with one cause. Task 1: run A finished on the outgoing server at 15:13:56. Its warm session stayed idle in that server. The incoming server claimed the follow-up at 15:13:57. `inspectRemoteWarmSessionRecovery` asked `evaluateNativeControllerTakeover`, which allowed the takeover with `remote_controller_lease_released_or_expired`. That lease covers the finished run, not the warm session. The outgoing server was alive until 15:18:18. `runner.prp.authenticate` failed after 62.3 s, the run failed with `remote_warm_session_unproven`, the failure was saved as ambiguous and owned by the board, and the task stayed blocked with a Retry. Task 2: the same shape on a later swap started 27 automatic runs in 13 minutes. Fourteen of them said "The next attempt starts on a fresh sandbox", but each one resumed the same sandbox and got the same refusal (`journal_unsettled`). **Expected behavior** A follow-up claimed during an overlap waits for the warm session owner to let go, then continues with no Retry banner and no second controller. If the outgoing server dies with the session open, the takeover continues after its ownership ends. A message that promises a fresh sandbox is true, or the retries stop after a bounded number with a true state. **Steps to reproduce** 1. Run the restart-chaos release-swap preset with a warm (opencode or codex) staging mix. 2. Let a run finish on the outgoing server after the traffic switch, with a follow-up comment already queued. 3. See the follow-up fail with `remote_warm_session_unproven` and the task go blocked. **Paperclip version or commit** masteraaec67b891and45531b719e. **Deployment mode** Hosted deploy overlap (release swap) with Daytona sandboxes. ## What Changed - `remote-warm-session-recovery.ts`: before live adoption, the run proves that the server process that ran the finished turn is gone. It uses the shared owner proof (`proveOwnerGone`): a PID check in the same process space, or the `controller_instances` lease in another process space. While that server is alive, the run defers with the existing `native_controller_recovery_waiting_for_owner` wait and checks again after 5 s. The wait ends when the server exits or is fenced, or when the finished turn's warm window (idle timeout plus two minutes) ends. - `native-session-executor.ts` and `run-start-handover.ts`: a server that leaves new run starts to a replacement closes its idle warm sessions after 1 s, through the existing idle-timeout close. A session that becomes idle after that closes after 1 s too. This is the same close that shutdown already does, done earlier. - `predecessor-retirement-fallback.ts`: when a retry meets an unretired owner on the same sandbox that its source already asked to replace, it does not ask again. The run fails without another automatic retry, and the task shows a Retry. - `predecessor-retirement.ts`: the error no longer says "The next attempt starts on a fresh sandbox", because the replacement is only a request. - `doc/execution-semantics.md`: documents the owner wait, the hand-off at the swap, and the retry stop. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts` - New: "waits while the outgoing server that ran the finished turn still holds its warm session, then takes it over once that server exits" (red on master: master returned a takeover while the owner was alive). - New: "waits for a live outgoing server in another process space until its controller lease ends" (red on master). - New: "stops waiting for a live outgoing server once the finished turn's warm window has closed" (bound check). - `cd server && npx vitest run src/__tests__/heartbeat-native-predispatch-retry.test.ts` - New: "stops retrying with a truthful failure when the next attempt met the same unretired sandbox instead of a fresh one" (red on master: master scheduled another retry on the same sandbox). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "replacement server starts"` - New: "closes a warm session promptly when a replacement server starts this server's runs (before/after the turn ends)" (both red on master). - Focused suites: native-session-executor, remote-warm-session-recovery, native-controller-recovery, native-recovered-owner-followup, native-runner-restart-recovery, heartbeat-native-predispatch-retry, environment-runtime-reusable-resume-replacement, task-interruption-presentation, sandbox-runner-port-holders. - `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify` exits 2 (a doc is a global CI input), so required CI supplies broad coverage. - Not run here: the restart-chaos release-swap wave. The maintainer runs it after merge. ## Risks - A follow-up can now wait up to the outgoing server's remaining overlap when that server cannot close its idle session. Before, it failed and needed a Retry. The wait is bounded by the owner's exit or the warm window. - The outgoing server closes idle warm sessions earlier during an overlap. The next turn then resumes from a suspended Runner instead of a live one, as it already does after a shutdown. - A retry chain that meets the same unretired sandbox twice now stops one attempt earlier than the shared cap. The task shows a Retry instead of looping. - Rows without a recorded process space keep the existing one-release local proof, so they behave as before outside the same host. - Previous-release state: this change adds no persisted field. Finalization rows saved by the previous release already record the controller boot, PID, process space and start time, so they carry forward. A follow-up already deferred by the previous release resumes through the same wait code. - Runtime cost: the owner proof adds one `ps` start-time read (same process space) or one `controller_instances` row read (other process space) per inspection. It runs once per attempt, at most every 5 s while a live owner holds the session. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
521 lines
36 KiB
TypeScript
521 lines
36 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { existsSync, mkdirSync, writeFileSync } from "node:fs";
|
|
import { join } from "node:path";
|
|
import { and, eq, inArray } from "drizzle-orm";
|
|
import { activityLog, agents, authUsers, companyMemberships, createDb, environmentLeases, environments, heartbeatRuns, issueComments,
|
|
issueRecoveryActions, issueRelations, issues } from "@paperclipai/db";
|
|
import { NativePredecessorRetirementUnprovenError } from "../services/native-runtime/predecessor-retirement.js";
|
|
import { NativeProviderArtifactTransferTimeoutError } from "../services/native-runtime/native-provider-artifacts.js";
|
|
import { taskInterruptionPresentation } from "../services/task-interruption-presentation.js";
|
|
import type { NativeRunIdentity, NativeSession, NativeSessionBackend, PrpEvent } from "@paperclipai/paperclip-runner";
|
|
import type { ExecutionContinuationEnvelope } from "@paperclipai/shared";
|
|
import { renderPaperclipWakePrompt } from "@paperclipai/adapter-utils/server-utils";
|
|
import { describe, expect, it, vi } from "vitest";
|
|
import { heartbeatService, type HeartbeatServiceOptions } from "../services/heartbeat.js";
|
|
import { instanceSettingsService } from "../services/instance-settings.js";
|
|
import { closeIdleWarmNativeSessionsForRestart, scopedRunnerdStateRoot } from "../services/native-runtime/native-session-executor.js";
|
|
import { nativeRestartFixture, waitForCondition } from "./helpers/native-restart-fixture.js";
|
|
import { getRecoveryPostgresSupport } from "./helpers/recovery-postgres-fixture.js";
|
|
|
|
// Budget admission is the last database write before the dispatch gate enters
|
|
// the native executor. Exhausted serialization retries there fail the run with
|
|
// no provider work started.
|
|
const admissionFaults = vi.hoisted(() => ({ remaining: 0 }));
|
|
vi.mock("../services/budget-reservations.js", async importOriginal => {
|
|
const actual = await importOriginal<typeof import("../services/budget-reservations.js")>();
|
|
const { AccountingContentionError } = await import("../services/accounting-transaction.js");
|
|
return {
|
|
...actual,
|
|
reserveRunBudget: async (...args: Parameters<typeof actual.reserveRunBudget>) => {
|
|
if (admissionFaults.remaining > 0) {
|
|
admissionFaults.remaining -= 1;
|
|
throw new AccountingContentionError(Object.assign(new Error("could not serialize access due to concurrent update"),
|
|
{ name: "PostgresError", code: "40001" }), 4);
|
|
}
|
|
return actual.reserveRunBudget(...args);
|
|
},
|
|
};
|
|
});
|
|
|
|
const support = await getRecoveryPostgresSupport({});
|
|
|
|
type AdmissionFailure = { remaining: number; turnNeverIssued: boolean; environmentId: string; provider: string;
|
|
providerLeaseId: string; leasePolicy: "reuse_by_environment" | "reuse_by_execution_workspace"; failedRuns: string[];
|
|
sessionRoots: string[] };
|
|
|
|
function nativeBackend(db: ReturnType<typeof createDb>, options: { failDescriptorOnce?: { failed: boolean };
|
|
unretiredPredecessorOnce?: { failed: boolean; environmentId: string; taskBlocked?: boolean; repeat?: string[] }; turnMessages?: Map<string, string>;
|
|
admissionFailure?: AdmissionFailure } = {}):
|
|
NonNullable<HeartbeatServiceOptions["nativeSessionBackendFactory"]> {
|
|
return execution => {
|
|
let identity: NativeRunIdentity;
|
|
let sourceInstanceId = "";
|
|
let turn = Promise.withResolvers<string>();
|
|
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true };
|
|
const session: NativeSession = {
|
|
identity: () => identity,
|
|
capabilities: async () => capabilities,
|
|
attachRun: async input => { identity = input.identity; },
|
|
startTurn: async input => {
|
|
options.turnMessages?.set(identity.runId, JSON.stringify(input.message));
|
|
const turnId = randomUUID();
|
|
turn.resolve(turnId);
|
|
return { turnId };
|
|
},
|
|
async *events() {
|
|
const turnId = await turn.promise;
|
|
const event = (sourceSeq: number, eventType: PrpEvent["eventType"], payload: Record<string, unknown>): PrpEvent => ({
|
|
schema: "paperclip.prp.event.v1", sourceKind: "runner", sourceInstanceId,
|
|
sourceEventId: `${sourceInstanceId}:${identity.runId}:${sourceSeq}`, sourceSeq,
|
|
runId: identity.runId, normalizedSessionId: identity.sessionId, turnId,
|
|
schemaVersion: 1, priority: 0, emittedAt: new Date().toISOString(), eventType, payload,
|
|
});
|
|
yield event(1, "turn.started", {});
|
|
yield event(2, "item.completed", { kind: "agentMessage", channel: "final", text: "Done." });
|
|
turn = Promise.withResolvers<string>();
|
|
yield event(3, "turn.completed", { status: "completed" });
|
|
},
|
|
result: async () => null,
|
|
snapshot: async () => ({ backendKind: "mock", sessionId: "predispatch-session", identity,
|
|
providerSessionId: "predispatch-provider-session", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }),
|
|
close: async () => {},
|
|
};
|
|
const backend: NativeSessionBackend = {
|
|
descriptor: async () => {
|
|
if (options.failDescriptorOnce && !options.failDescriptorOnce.failed) {
|
|
options.failDescriptorOnce.failed = true;
|
|
throw Object.assign(new Error("could not serialize access due to concurrent update"), { name: "PostgresError", code: "40001" });
|
|
}
|
|
const unretired = options.unretiredPredecessorOnce;
|
|
if (unretired && (!unretired.failed || unretired.repeat)) {
|
|
unretired.repeat?.push(execution.binding.runId);
|
|
// Staging: this run resumed the task's reused sandbox, where
|
|
// the previous run's warm Runner could not be retired before any
|
|
// provider work. The executor reports that after its coordinator claim.
|
|
unretired.failed = true;
|
|
if (unretired.taskBlocked) await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, execution.binding.issueId));
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, execution.binding.runId));
|
|
await db.insert(environmentLeases).values({ companyId: run!.companyId, environmentId: unretired.environmentId,
|
|
issueId: execution.binding.issueId, heartbeatRunId: run!.id, provider: "daytona", providerLeaseId: "sandbox-reused",
|
|
leasePolicy: "reuse_by_environment", status: "active", metadata: { remoteCwd: "/workspace" } });
|
|
throw new NativePredecessorRetirementUnprovenError("previous-run", "sandbox-reused",
|
|
new Error("Controller journal drain timed out; preserve the session before replacing its writer."));
|
|
}
|
|
const [run] = await db.select({ runnerInstanceId: heartbeatRuns.runnerInstanceId }).from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, execution.binding.runId));
|
|
if (!run?.runnerInstanceId) throw new Error("Normal native admission did not persist a runner identity");
|
|
sourceInstanceId = run.runnerInstanceId;
|
|
return { kind: "mock", name: "native-predispatch-retry", version: "1", capabilities,
|
|
runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } };
|
|
},
|
|
openSession: async input => { identity = input.identity; return session; },
|
|
recoverSession: async snapshot => { identity = snapshot.identity; return { recovered: true, session }; },
|
|
};
|
|
const failure = options.admissionFailure;
|
|
if (!failure) return backend;
|
|
return Object.assign(backend, {
|
|
providerTurnNeverIssued: () => failure.turnNeverIssued,
|
|
providerAdmissionPersisted: async () => {
|
|
if (failure.remaining <= 0) return;
|
|
failure.remaining -= 1;
|
|
const runId = execution.binding.runId;
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
await db.insert(environmentLeases).values({ companyId: run!.companyId, environmentId: failure.environmentId,
|
|
issueId: execution.binding.issueId, heartbeatRunId: runId, provider: failure.provider,
|
|
providerLeaseId: `${failure.providerLeaseId}-${failure.failedRuns.length}`, leasePolicy: failure.leasePolicy,
|
|
status: "active", metadata: { remoteCwd: "/workspace" } });
|
|
failure.failedRuns.push(runId);
|
|
const root = scopedRunnerdStateRoot(execution);
|
|
mkdirSync(root, { recursive: true, mode: 0o700 });
|
|
writeFileSync(join(root, "failed-attempt"), runId);
|
|
failure.sessionRoots.push(root);
|
|
throw new NativeProviderArtifactTransferTimeoutError("upload", 120_000, 512 * 1024 * 1024);
|
|
},
|
|
});
|
|
};
|
|
}
|
|
|
|
describe.skipIf(!support.supported)("heartbeat native pre-dispatch failure retry", () => {
|
|
const fixture = nativeRestartFixture({});
|
|
|
|
async function startTask(db: ReturnType<typeof createDb>, heartbeat: ReturnType<typeof heartbeatService>,
|
|
status: "in_progress" | "blocked" = "in_progress") {
|
|
const { companyId, agentId } = fixture.current;
|
|
const userId = randomUUID();
|
|
await db.insert(authUsers).values({ id: userId, name: "Task owner", email: `${userId}@example.invalid`,
|
|
createdAt: new Date(), updatedAt: new Date() });
|
|
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "owner" });
|
|
await db.update(agents).set({ status: "idle", adapterConfig: { provider: "codex" } }).where(eq(agents.id, agentId));
|
|
await instanceSettingsService(db).updateExperimental({ enableAgentChat: true, enableNativeRunner: true });
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Survive a database fault", status,
|
|
assigneeAgentId: agentId, responsibleUserId: userId, conversationAgentId: agentId, conversationUserId: userId,
|
|
conversationState: "active" }).returning();
|
|
if (status === "blocked") {
|
|
const [blocker] = await db.insert(issues).values({ companyId, title: "Open subtask", status: "todo" }).returning();
|
|
await db.insert(issueRelations).values({ companyId, issueId: blocker!.id, relatedIssueId: issue!.id, type: "blocks" });
|
|
}
|
|
const [message] = await db.insert(issueComments).values({ companyId, issueId: issue!.id,
|
|
authorType: "user", authorUserId: userId, body: "Start." }).returning();
|
|
const run = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual", reason: "issue_commented",
|
|
requestedByActorType: "user", requestedByActorId: userId, manualUserWake: true,
|
|
payload: { issueId: issue!.id, commentId: message!.id },
|
|
contextSnapshot: { issueId: issue!.id, taskKey: issue!.id, wakeCommentId: message!.id } });
|
|
expect(run).not.toBeNull();
|
|
await waitForCondition("the first run to settle", () => heartbeat.getTaskDrainStatus().quiescent);
|
|
return { issueId: issue!.id, runId: run!.id };
|
|
}
|
|
|
|
it("re-queues a native run that failed at budget admission and completes it without a Retry", async () => {
|
|
const { companyId } = fixture.current;
|
|
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
|
|
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db) });
|
|
try {
|
|
admissionFaults.remaining = 1;
|
|
const { issueId, runId } = await startTask(db, heartbeat);
|
|
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(failed).toMatchObject({ status: "failed", runtimeMode: "native", errorCode: "adapter_failed",
|
|
resultJson: expect.objectContaining({ failureCause: "40001",
|
|
nativePreDispatchFailure: { version: 1, providerWorkStarted: false } }) });
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.retryOfRunId, runId)));
|
|
expect(successors).toEqual([expect.objectContaining({ scheduledRetryReason: "interrupted_preparation",
|
|
scheduledRetryAttempt: 1 })]);
|
|
expect(successors[0]!.contextSnapshot?.executionRetryAccounting).toMatchObject({ failureRetries: 0 });
|
|
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]?.status).not.toBe("blocked");
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
|
|
|
|
// The first attempt is due at once; the scheduler promotes it and the
|
|
// queue starts it, exactly once.
|
|
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 1 });
|
|
await heartbeat.resumeQueuedRuns();
|
|
await waitForCondition("the retry to succeed", async () =>
|
|
(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successors[0]!.id)))[0]?.status === "succeeded");
|
|
await waitForCondition("the retry to settle", () => heartbeat.getTaskDrainStatus().quiescent);
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.retryOfRunId, runId)))).toHaveLength(1);
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
|
|
} finally {
|
|
admissionFaults.remaining = 0;
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await closeIdleWarmNativeSessionsForRestart();
|
|
await db.$client.end();
|
|
}
|
|
}, 90_000);
|
|
|
|
it("re-queues a run whose predecessor could not be retired onto a fresh sandbox without a Retry", async () => {
|
|
const { companyId } = fixture.current;
|
|
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
|
|
const [environment] = await db.insert(environments).values({ name: `reused-${randomUUID()}`, driver: "sandbox" }).returning();
|
|
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db,
|
|
{ unretiredPredecessorOnce: { failed: false, environmentId: environment!.id } }) });
|
|
try {
|
|
const { issueId, runId } = await startTask(db, heartbeat);
|
|
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(failed).toMatchObject({ status: "failed", runtimeMode: "native", errorCode: "adapter_failed",
|
|
resultJson: expect.objectContaining({ failureCause: "native_predecessor_retirement_unproven",
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
|
predecessorRetirementFallback: expect.objectContaining({ version: 1, providerWorkStarted: false,
|
|
predecessorRunId: "previous-run", replacedProviderLeaseIds: ["sandbox-reused"] }) }) });
|
|
// The reused sandbox is replaced at the next acquisition, not resumed.
|
|
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.heartbeatRunId, runId),
|
|
eq(environmentLeases.providerLeaseId, "sandbox-reused")));
|
|
expect(lease?.metadata).toMatchObject({ replaceBeforeReuse: { schema: "paperclip.replace-before-reuse.v1", runId,
|
|
predecessorRunId: "previous-run" } });
|
|
expect(await db.select().from(activityLog).where(and(eq(activityLog.runId, runId),
|
|
eq(activityLog.action, "environment.lease_replacement_requested")))).toHaveLength(1);
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.retryOfRunId, runId)));
|
|
expect(successors).toEqual([expect.objectContaining({ scheduledRetryReason: "interrupted_preparation", scheduledRetryAttempt: 1 })]);
|
|
expect(successors[0]!.contextSnapshot?.executionRetryAccounting).toMatchObject({ failureRetries: 0 });
|
|
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]?.status).not.toBe("blocked");
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
|
|
|
|
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 1 });
|
|
await heartbeat.resumeQueuedRuns();
|
|
await waitForCondition("the retry to succeed", async () =>
|
|
(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successors[0]!.id)))[0]?.status === "succeeded");
|
|
await waitForCondition("the retry to settle", () => heartbeat.getTaskDrainStatus().quiescent);
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
|
|
} finally {
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await closeIdleWarmNativeSessionsForRestart();
|
|
await db.$client.end();
|
|
}
|
|
}, 90_000);
|
|
|
|
it("stops retrying with a truthful failure when the next attempt met the same unretired sandbox instead of a fresh one", async () => {
|
|
const { companyId } = fixture.current;
|
|
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
|
|
const [environment] = await db.insert(environments).values({ name: `reused-${randomUUID()}`, driver: "sandbox" }).returning();
|
|
const attempts: string[] = [];
|
|
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db,
|
|
{ unretiredPredecessorOnce: { failed: false, environmentId: environment!.id, repeat: attempts } }) });
|
|
try {
|
|
const { issueId, runId } = await startTask(db, heartbeat);
|
|
const [first] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.retryOfRunId, runId)));
|
|
expect(first).toMatchObject({ scheduledRetryReason: "interrupted_preparation", scheduledRetryAttempt: 1 });
|
|
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 1 });
|
|
await heartbeat.resumeQueuedRuns();
|
|
await waitForCondition("the second attempt to end", async () =>
|
|
(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, first!.id)))[0]?.status === "failed");
|
|
await waitForCondition("the second attempt to settle", () => heartbeat.getTaskDrainStatus().quiescent);
|
|
expect(attempts).toEqual([runId, first!.id]);
|
|
const [second] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, first!.id));
|
|
expect(second?.resultJson ?? {}).not.toHaveProperty("predecessorRetirementFallback");
|
|
expect(second?.error ?? "").not.toMatch(/fresh sandbox/i);
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.retryOfRunId, first!.id)))).toHaveLength(0);
|
|
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 0 });
|
|
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
|
expect(await taskInterruptionPresentation(db, issue!)).toMatchObject({ canRetry: true });
|
|
} finally {
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await closeIdleWarmNativeSessionsForRestart();
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"])));
|
|
await db.$client.end();
|
|
}
|
|
}, 90_000);
|
|
|
|
it.each([
|
|
{ title: "continues a follow-up on a blocked task whose predecessor could not be retired", blockedAfter: false },
|
|
{ title: "does not continue a task that was blocked only after its run failed", blockedAfter: true },
|
|
])("$title", async ({ blockedAfter }) => {
|
|
const { companyId } = fixture.current;
|
|
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
|
|
const [environment] = await db.insert(environments).values({ name: `reused-${randomUUID()}`, driver: "sandbox" }).returning();
|
|
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db,
|
|
{ unretiredPredecessorOnce: { failed: false, environmentId: environment!.id, taskBlocked: !blockedAfter } }) });
|
|
try {
|
|
const { issueId, runId } = await startTask(db, heartbeat, blockedAfter ? "in_progress" : "blocked");
|
|
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(failed).toMatchObject({ status: "failed", resultJson: expect.objectContaining({
|
|
failureCause: "native_predecessor_retirement_unproven",
|
|
predecessorRetirementFallback: expect.objectContaining({ issueStatusAtFailure: blockedAfter ? "in_progress" : "blocked" }) }) });
|
|
if (blockedAfter) await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, issueId));
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, issueId));
|
|
expect(task?.status).toBe("blocked");
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.retryOfRunId, runId)));
|
|
expect(successors).toEqual([expect.objectContaining({ status: "scheduled_retry", scheduledRetryReason: "interrupted_preparation" })]);
|
|
expect(await taskInterruptionPresentation(db, task!)).toMatchObject({ state: "working", canRetry: false });
|
|
if (blockedAfter) {
|
|
await heartbeat.promoteDueScheduledRetries();
|
|
await heartbeat.resumeQueuedRuns();
|
|
await waitForCondition("the stale retry to settle", () => heartbeat.getTaskDrainStatus().quiescent);
|
|
const [stale] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successors[0]!.id));
|
|
expect(["cancelled", "scheduled_retry"]).toContain(stale?.status);
|
|
return;
|
|
}
|
|
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 1 });
|
|
await heartbeat.resumeQueuedRuns();
|
|
await waitForCondition("the retry to settle", () => heartbeat.getTaskDrainStatus().quiescent);
|
|
const [retried] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successors[0]!.id));
|
|
expect({ status: retried?.status, errorCode: retried?.errorCode, error: retried?.error }).toEqual({ status: "succeeded", errorCode: null, error: null });
|
|
} finally {
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await closeIdleWarmNativeSessionsForRestart();
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"])));
|
|
await db.$client.end();
|
|
}
|
|
}, 90_000);
|
|
|
|
it("delivers the monitor check to the automatic retry of a monitor run that failed before provider work", async () => {
|
|
const { companyId, agentId } = fixture.current;
|
|
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
|
|
const [environment] = await db.insert(environments).values({ name: `reused-${randomUUID()}`, driver: "sandbox" }).returning();
|
|
const turnMessages = new Map<string, string>();
|
|
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db,
|
|
{ unretiredPredecessorOnce: { failed: false, environmentId: environment!.id }, turnMessages }) });
|
|
try {
|
|
await db.update(agents).set({ status: "idle", adapterConfig: { provider: "codex" } }).where(eq(agents.id, agentId));
|
|
await instanceSettingsService(db).updateExperimental({ enableNativeRunner: true });
|
|
const userId = randomUUID();
|
|
await db.insert(authUsers).values({ id: userId, name: "Task owner", email: `${userId}@example.invalid`,
|
|
createdAt: new Date(), updatedAt: new Date() });
|
|
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "owner" });
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Phase 2 on the monitor check", status: "in_progress",
|
|
assigneeAgentId: agentId, responsibleUserId: userId, monitorNotes: "Phase 2 check" }).returning();
|
|
const run = await heartbeat.wakeup(agentId, { source: "automation", triggerDetail: "system", reason: "issue_monitor_due",
|
|
payload: { issueId: issue!.id }, contextSnapshot: { issueId: issue!.id, taskKey: issue!.id,
|
|
source: "issue.monitor", wakeReason: "issue_monitor_due", monitorNotes: "Phase 2 check" } });
|
|
expect(run).not.toBeNull();
|
|
await waitForCondition("the monitor run to settle", () => heartbeat.getTaskDrainStatus().quiescent);
|
|
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id));
|
|
expect(failed).toMatchObject({ status: "failed", resultJson: expect.objectContaining({
|
|
failureCause: "native_predecessor_retirement_unproven" }) });
|
|
const [successor] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.retryOfRunId, run!.id)));
|
|
expect(successor).toMatchObject({ scheduledRetryReason: "interrupted_preparation" });
|
|
|
|
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 1 });
|
|
await heartbeat.resumeQueuedRuns();
|
|
await waitForCondition("the retry to start its provider turn", () => turnMessages.has(successor!.id));
|
|
await waitForCondition("the retry to settle", () => heartbeat.getTaskDrainStatus().quiescent);
|
|
const [retried] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successor!.id));
|
|
const continuation = retried!.contextSnapshot?.executionContinuation as ExecutionContinuationEnvelope;
|
|
expect(continuation.trigger).toMatchObject({ sourceRunId: run!.id, retriedReason: "issue_monitor_due" });
|
|
expect(renderPaperclipWakePrompt(retried!.contextSnapshot?.paperclipWake)).toContain("- retried wake: issue_monitor_due");
|
|
expect(turnMessages.get(successor!.id)).toContain("- retried wake: issue_monitor_due");
|
|
expect(turnMessages.has(run!.id)).toBe(false);
|
|
} finally {
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await closeIdleWarmNativeSessionsForRestart();
|
|
await db.$client.end();
|
|
}
|
|
}, 90_000);
|
|
|
|
it.each([
|
|
{ target: "remote sandbox", provider: "e2b", leasePolicy: "reuse_by_execution_workspace" as const },
|
|
{ target: "plugin-task sandbox", provider: "daytona", leasePolicy: "reuse_by_environment" as const },
|
|
])("continues on a fresh $target when the provider artifacts fail to save before any turn", async ({ provider, leasePolicy }) => {
|
|
const { companyId } = fixture.current;
|
|
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
|
|
const [environment] = await db.insert(environments).values({ name: `capture-${randomUUID()}`, driver: "sandbox" }).returning();
|
|
const turnMessages = new Map<string, string>();
|
|
const failure: AdmissionFailure = { remaining: 1, turnNeverIssued: true, environmentId: environment!.id, provider,
|
|
providerLeaseId: `sandbox-${provider}`, leasePolicy, failedRuns: [], sessionRoots: [] };
|
|
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, { admissionFailure: failure, turnMessages }) });
|
|
try {
|
|
const { issueId, runId } = await startTask(db, heartbeat);
|
|
expect(failure.failedRuns).toEqual([runId]);
|
|
expect(turnMessages.has(runId)).toBe(false);
|
|
expect(existsSync(failure.sessionRoots[0]!)).toBe(false);
|
|
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(failed).toMatchObject({ status: "failed", runtimeMode: "native",
|
|
resultJson: expect.objectContaining({ failureCause: "native_failed_before_provider_turn",
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
|
predecessorRetirementFallback: expect.objectContaining({ version: 1, providerWorkStarted: false,
|
|
predecessorRunId: null, replacedProviderLeaseIds: [`sandbox-${provider}-0`] }) }) });
|
|
expect(failed!.error).toBe("Setting up the computer took too long, so the task is continuing on a fresh one.");
|
|
expect(failed!.error).not.toMatch(/[a-z0-9]+_[a-z0-9_]+|\d|\(|\)/i);
|
|
expect(failed!.runnerProfileJson).toMatchObject({ failedBeforeProviderTurn: {
|
|
cause: "native_failed_before_provider_turn", technicalCause: "native_provider_artifacts_transfer_timeout" } });
|
|
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.heartbeatRunId, runId),
|
|
eq(environmentLeases.providerLeaseId, `sandbox-${provider}-0`)));
|
|
expect(lease?.metadata).toMatchObject({ replaceBeforeReuse: { schema: "paperclip.replace-before-reuse.v1", runId } });
|
|
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.retryOfRunId, runId)));
|
|
expect(successors).toEqual([expect.objectContaining({ scheduledRetryReason: "interrupted_preparation", scheduledRetryAttempt: 1 })]);
|
|
expect(successors[0]!.contextSnapshot?.executionRetryAccounting).toMatchObject({ failureRetries: 0 });
|
|
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
|
expect(issue!.status).not.toBe("blocked");
|
|
expect(await taskInterruptionPresentation(db, issue!)).toMatchObject({ state: "working", canRetry: false });
|
|
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
|
|
|
|
expect(await heartbeat.promoteDueScheduledRetries()).toMatchObject({ promoted: 1 });
|
|
await heartbeat.resumeQueuedRuns();
|
|
await waitForCondition("the continuation to succeed", async () =>
|
|
(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successors[0]!.id)))[0]?.status === "succeeded");
|
|
await waitForCondition("the continuation to settle", () => heartbeat.getTaskDrainStatus().quiescent);
|
|
expect(turnMessages.has(successors[0]!.id)).toBe(true);
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.retryOfRunId, runId)))).toHaveLength(1);
|
|
const [settled] = await db.select().from(issues).where(eq(issues.id, issueId));
|
|
expect(await taskInterruptionPresentation(db, settled!)).toMatchObject({ state: "resolved", canRetry: false });
|
|
} finally {
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await closeIdleWarmNativeSessionsForRestart();
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"])));
|
|
await db.$client.end();
|
|
}
|
|
}, 90_000);
|
|
|
|
it("keeps today's ambiguous hold when the Runner ledger cannot prove no turn was sent", async () => {
|
|
const { companyId } = fixture.current;
|
|
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
|
|
const [environment] = await db.insert(environments).values({ name: `capture-${randomUUID()}`, driver: "sandbox" }).returning();
|
|
const failure: AdmissionFailure = { remaining: 1, turnNeverIssued: false, environmentId: environment!.id, provider: "daytona",
|
|
providerLeaseId: "sandbox-turn-maybe-sent", leasePolicy: "reuse_by_environment", failedRuns: [], sessionRoots: [] };
|
|
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, { admissionFailure: failure }) });
|
|
try {
|
|
const { runId } = await startTask(db, heartbeat);
|
|
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(failed?.status).toBe("failed");
|
|
expect(failed?.resultJson ?? {}).not.toHaveProperty("predecessorRetirementFallback");
|
|
expect(failed?.resultJson?.failureCause).not.toBe("native_failed_before_provider_turn");
|
|
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.heartbeatRunId, runId),
|
|
eq(environmentLeases.providerLeaseId, "sandbox-turn-maybe-sent-0")));
|
|
expect(lease?.metadata ?? {}).not.toHaveProperty("replaceBeforeReuse");
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.retryOfRunId, runId), eq(heartbeatRuns.scheduledRetryReason, "interrupted_preparation")))).toHaveLength(0);
|
|
} finally {
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await closeIdleWarmNativeSessionsForRestart();
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"])));
|
|
await db.$client.end();
|
|
}
|
|
}, 90_000);
|
|
|
|
it("backs off a lasting pre-turn failure and stops at the shared lane cap without hammering", async () => {
|
|
const { companyId } = fixture.current;
|
|
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
|
|
const [environment] = await db.insert(environments).values({ name: `capture-${randomUUID()}`, driver: "sandbox" }).returning();
|
|
const failure: AdmissionFailure = { remaining: Number.POSITIVE_INFINITY, turnNeverIssued: true, environmentId: environment!.id,
|
|
provider: "daytona", providerLeaseId: "sandbox-lasting", leasePolicy: "reuse_by_environment", failedRuns: [], sessionRoots: [] };
|
|
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, { admissionFailure: failure }) });
|
|
const successorOf = async (runId: string) => (await db.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.retryOfRunId, runId))));
|
|
const runToEnd = async (runId: string) => {
|
|
await heartbeat.resumeQueuedRuns();
|
|
await waitForCondition("the attempt to end", async () =>
|
|
(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]?.status === "failed");
|
|
await waitForCondition("the attempt to settle", () => heartbeat.getTaskDrainStatus().quiescent);
|
|
};
|
|
try {
|
|
const { issueId, runId } = await startTask(db, heartbeat);
|
|
const [first] = await successorOf(runId);
|
|
expect(first).toMatchObject({ scheduledRetryReason: "interrupted_preparation", scheduledRetryAttempt: 1 });
|
|
expect((await heartbeat.promoteDueScheduledRetries()).runIds).toEqual([first!.id]);
|
|
const secondStartedAt = Date.now();
|
|
await runToEnd(first!.id);
|
|
const [second] = await successorOf(first!.id);
|
|
expect(second).toMatchObject({ scheduledRetryReason: "interrupted_preparation", scheduledRetryAttempt: 2 });
|
|
expect(second!.scheduledRetryAt!.getTime() - secondStartedAt).toBeGreaterThanOrEqual(15_000);
|
|
expect((await heartbeat.promoteDueScheduledRetries()).runIds).not.toContain(second!.id);
|
|
expect(failure.failedRuns).toHaveLength(2);
|
|
const [waiting] = await db.select().from(issues).where(eq(issues.id, issueId));
|
|
expect(await taskInterruptionPresentation(db, waiting!)).toMatchObject({ state: "working", canRetry: false });
|
|
|
|
await db.update(heartbeatRuns).set({ scheduledRetryAttempt: 12, scheduledRetryAt: new Date(Date.now() - 1_000),
|
|
contextSnapshot: { ...second!.contextSnapshot, executionRetryAccounting: {
|
|
...(second!.contextSnapshot?.executionRetryAccounting as Record<string, unknown>), interruptedPreparations: 12 } } })
|
|
.where(eq(heartbeatRuns.id, second!.id));
|
|
expect((await heartbeat.promoteDueScheduledRetries()).runIds).toEqual([second!.id]);
|
|
await runToEnd(second!.id);
|
|
expect(await successorOf(second!.id)).toHaveLength(0);
|
|
expect(failure.failedRuns).toHaveLength(3);
|
|
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
|
expect(await taskInterruptionPresentation(db, issue!)).toMatchObject({ state: "failed", canRetry: true });
|
|
} finally {
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await closeIdleWarmNativeSessionsForRestart();
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, ["queued", "scheduled_retry"])));
|
|
await db.$client.end();
|
|
}
|
|
}, 120_000);
|
|
|
|
it("keeps the hold when the failure follows entry into the native executor", async () => {
|
|
const { companyId } = fixture.current;
|
|
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
|
|
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, { failDescriptorOnce: { failed: false } }) });
|
|
try {
|
|
const { runId } = await startTask(db, heartbeat);
|
|
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(failed?.status).toBe("failed");
|
|
expect(failed?.resultJson ?? {}).not.toHaveProperty("nativePreDispatchFailure");
|
|
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.retryOfRunId, runId), eq(heartbeatRuns.scheduledRetryReason, "interrupted_preparation")))).toHaveLength(0);
|
|
} finally {
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await closeIdleWarmNativeSessionsForRestart();
|
|
await db.$client.end();
|
|
}
|
|
}, 90_000);
|
|
});
|