Files
PaperClipAI/server/src/services/native-runtime/native-client-timeout-continuation.integration.test.ts
T
DottaandClaude Opus 5.5 68f641ba0b fix(recovery): continue a native run cut off by a sandbox client timeout (#21736)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime runs an agent's provider turn in a sandbox (for
example Daytona) and records each failure on the run and its native
coordinator
> - A Daytona SDK request can hit its client-side deadline during a
provider turn. The run then fails with `native_session_interrupted` in
an ambiguous state, because the turn started and its outcome is unknown
> - That failure blocks the task under a board-owned watchdog hold. No
recovery owner selected the hold, because its cause is not an
execution-reconciliation cause. The task never ran again, and only a
person's Retry could continue it
> - Recovery from an infrastructure fault must work automatically,
without a person
> - This pull request lets the execution-disposition pass hand that hold
to the existing fresh-continuation fallback
> - The benefit is that the task continues once, in an isolated fresh
run, and the cut-off turn is never replayed

## Linked Issues or Issue Description

**What happened?**
A native run on a Daytona execution environment (native runtime, an
OpenRouter model) started its provider turn. A Daytona request then
failed with `HTTP request timed out after 300000ms waiting for a
response. This is a client-side deadline
(DaytonaConfig.requestTimeoutMs, or the per-call operation/execution
timeout); any operation already started on the server may still be
running.` The run failed with `native_session_interrupted`. The task had
no further run: no automatic retry and no continuation.

**Expected behavior**
The task continues automatically, exactly once, in a fresh run from its
saved work. The cut-off provider turn is not replayed.

**Steps to reproduce**
1. Start a native run for an assigned task. Let its provider turn start
(a `turn.started` event is saved).
2. Make a sandbox request in that turn fail with the Daytona client-side
timeout error.
3. The executor saves the coordinator as `terminal_failure` with
`recoveryMode: ambiguous_state`, blocks the task, and writes an active
board hold with cause `native_session_interrupted`.
4. Run the execution-disposition worker
(`settleUnrecoverableExecutions`). Before this change, nothing happens
and the task stays blocked.

**Paperclip version or commit**
`master` at 9ebc276b42 (observed on a build from 5acf27bc3).

**Deployment mode**
Hosted cloud with a Daytona execution environment.

Causal path:
- The Daytona error is not a named class, so
`nativeSessionFailureSourceCode` returns the generic
`native_session_interrupted`.
- `nativeProviderRecoveryEvidence` returns `ambiguous_state`: provider
events exist, but there is no checkpoint with a provider identity. The
executor then makes the failure terminal on the first attempt (no
same-run resume) and writes a board hold whose cause is
`native_session_interrupted`.
- `unresolvedExecutionDispositions` and `heldExecutionWhere` select only
holds with `EXECUTION_RECONCILIATION_CAUSES` (or a narrow
`resume_native_run` shape). The safe replacement lane accepts
`native_session_interrupted` only for the historical
listener-unavailable case. The assigned-work sweep leaves a task blocked
by an active hold to the hold's owner. So no owner continued the task.

## What Changed

- `server/src/services/execution-recovery-resolution.ts`: the
disposition pass also selects an active watchdog hold with a
restart-class cause (`RESTART_CLASS_FAILURE_CODES`) for a terminal
native run with no retry pending, unless the safe replacement lane owns
that run. It hands the run to `continueFreshInTransaction` and records
the fallback's result on the hold, so a run that does not continue is
checked again only when due.
- `server/src/services/fresh-continuation-fallback.ts`: the fallback
treats an active watchdog hold with a restart-class cause as its own.
When it continues the task, it settles that hold and unblocks the task,
as it does for reconciliation holds.
- New red/green test
`server/src/services/native-runtime/native-client-timeout-continuation.integration.test.ts`.
It drives the real native executor with a provider turn that starts and
then fails with the Daytona client-timeout error.

The cut-off turn is not replayed. The fresh continuation isolates the
old run as it does for every continuation: the old run loses its write
authority, its remote workspace epoch is fenced (or the continuation
gets a new workspace), and the coordinator records `replacementDenied:
fresh_continuation`. The fallback keeps its existing checks:
person-owned causes, exact recovery still under way, and the per-task
hourly cap.

## Verification

- Red on `master`: `npx vitest run
src/services/native-runtime/native-client-timeout-continuation.integration.test.ts`
fails with no continuation recorded.
- Green with this change. The test asserts that the run fails with
`native_session_interrupted` and the Daytona error text in an ambiguous
state, then that two disposition passes record exactly one fresh
continuation, the task goes from `blocked` to `todo`, no active hold
remains, the old run is marked `replacementDenied`, and two
continuation-delivery passes wake the assignee exactly once.
- Focused suites (all tests that use the disposition pass, the
fresh-continuation fallback, or the execution-control schedule, 28
files): 27 files pass. `chat-channels.integration.test.ts` has 9
failures that are unrelated to this change (see the PR comment for the
`master` baseline).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify --explain` selects 2751 of 3578 files (a near-full run).
I did not run it locally. CI supplies that coverage.

## Risks

- Behavior change: a terminal native interruption hold that before
stayed with the board now continues automatically in a fresh run. This
is the intended product behavior. The fallback still refuses
person-owned causes and still respects the per-task cap.
- Holds with restart-class causes are now settled by the fallback when
it continues a run. Before, they could stay active next to the
continuation record.
- No migration. No API or UI contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) and code execution for tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:15:41 +00:00

155 lines
10 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import {
activityLog, agents, companies, completionContracts, createDb, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations,
} from "@paperclipai/db";
import type { ControlPlanePort, NativeExecutionInputV1, PrpEvent } from "@paperclipai/paperclip-runner";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()),
executeNativeSession: provider.execute,
}));
import { executePaperclipNativeSession } from "./native-session-executor.js";
import { buildNativeCompletionContract } from "./completion-contracts.js";
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "../execution-recovery-resolution.js";
import { executionControlSchedule } from "../execution-control-work.js";
import { ASSIGNED_WORK_CONTINUED_ACTION } from "../recovery/continuation-ledger.js";
import { FRESH_CONTINUATION_WINDOW_MS } from "../fresh-continuation-fallback.js";
const DAYTONA_CLIENT_TIMEOUT = "HTTP request timed out after 300000ms waiting for a response. This is a client-side deadline "
+ "(DaytonaConfig.requestTimeoutMs, or the per-call operation/execution timeout); any operation already started on the "
+ "server may still be running.";
describe("a native turn cut off by a sandbox client timeout", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
const companyId = randomUUID();
const agentId = randomUUID();
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("native-client-timeout-");
db = createDb(temporary.connectionString);
await db.insert(companies).values({ id: companyId, name: "Client timeout", issuePrefix: "NCT" });
await db.insert(agents).values({ id: agentId, companyId, name: "Builder", status: "active", adapterType: "paperclip_runner",
adapterConfig: { provider: "opencode", model: "openrouter/test-model" } });
}, 240_000);
afterAll(async () => {
await db?.$client.end({ timeout: 0 });
await temporary?.cleanup();
});
async function failTurnWithClientTimeout(issueId: string) {
const runId = randomUUID();
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native",
nativeIssueId: issueId, invocationSource: "assignment", triggerDetail: "system",
contextSnapshot: { issueId, taskId: issueId, wakeReason: "issue_assigned" } }).returning();
await db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, issueId));
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
const execution: NativeExecutionInputV1 = {
schema: "paperclip.native-execution-input.v1",
binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
provider: { kind: "opencode", model: "openrouter/test-model" },
task: { identifier: issueId, title: issue.title, description: null, prompt: "Build it.", workMode: "standard" },
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "opencode_server", protocolVersion: 1,
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
contract: buildNativeCompletionContract(issue, { revision: contract.revision }) },
interactionResponses: [], credentialBindings: [],
};
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
const turnStarted: PrpEvent = {
schema: "paperclip.prp.event.v1", runId, normalizedSessionId: native.normalizedSessionId,
turnId: native.turnId, sourceInstanceId: native.runnerInstanceId, sourceEventId: "turn-started", sourceSeq: 1,
sourceKind: "runner", eventType: "turn.started", schemaVersion: 1, priority: 0, emittedAt: new Date().toISOString(),
payload: {},
};
provider.execute.mockReset().mockImplementation(async (options: { controlPlane: ControlPlanePort }) => {
await options.controlPlane.openRun({ identity: { companyId, issueId, runId, agentId, sessionId: native.normalizedSessionId },
backendKind: "mock", sourceInstanceId: native.runnerInstanceId });
await options.controlPlane.appendEvent(turnStarted);
throw new Error(DAYTONA_CLIENT_TIMEOUT);
});
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: {
descriptor: async () => { throw new Error("Unexpected provider call"); },
openSession: async () => { throw new Error("Unexpected provider call"); },
} }).catch(() => undefined);
return runId;
}
const continuationsOf = (runId: string) => db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId),
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,sourceRunId}' = ${runId}`));
it("continues the task automatically exactly once in a fresh run, without replaying the cut-off turn", async () => {
const issueId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Build the feature", status: "in_progress",
workMode: "standard", assigneeAgentId: agentId });
const runId = await failTurnWithClientTimeout(issueId);
const [failed] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(failed).toMatchObject({ status: "failed", errorCode: "native_session_interrupted", error: DAYTONA_CLIENT_TIMEOUT });
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: "native_session_interrupted", nextAttemptAt: null,
failureDetail: expect.objectContaining({ recoveryMode: "ambiguous_state", providerEventsExist: true }) });
await settleUnrecoverableExecutions(db);
await settleUnrecoverableExecutions(db);
const continuations = await continuationsOf(runId);
expect(continuations).toHaveLength(1);
expect(continuations[0]!.evidence).toMatchObject({ continuationDelivery: "pending",
freshContinuation: { sourceRunId: runId, failureCode: "native_session_interrupted" },
freshContinuationSettlement: { replay: "none", decision: "continue_in_fresh_run" } });
const [task] = await db.select().from(issues).where(eq(issues.id, issueId));
expect(task).toMatchObject({ status: "todo", assigneeAgentId: agentId });
const open = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, issueId),
sql`${issueRecoveryActions.status} in ('active', 'escalated')`));
expect(open).toEqual([]);
const [settled] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
expect(settled.failureDetail).toMatchObject({ replacementDenied: "fresh_continuation" });
expect(provider.execute).toHaveBeenCalledOnce();
const wake = vi.fn(async () => ({ id: randomUUID() }) as never);
await deliverReconciledExecutions(db, wake);
await deliverReconciledExecutions(db, wake);
expect(wake).toHaveBeenCalledOnce();
expect(wake).toHaveBeenCalledWith(agentId, expect.objectContaining({
payload: expect.objectContaining({ issueId }) }));
}, 120_000);
it("registers the capped continuation's next check with the disposition worker and continues the task when it is due", async () => {
const issueId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Build the capped feature", status: "in_progress",
workMode: "standard", assigneeAgentId: agentId });
const runId = await failTurnWithClientTimeout(issueId);
const firstAt = new Date(Date.now() - 10 * 60_000);
const capped = await db.insert(activityLog).values(Array.from({ length: 6 }, (_, index) => ({ companyId, actorId: "test",
action: ASSIGNED_WORK_CONTINUED_ACTION, entityType: "issue", entityId: issueId, details: { outcome: "queued" },
createdAt: new Date(firstAt.getTime() + index * 1_000) }))).returning({ id: activityLog.id });
await settleUnrecoverableExecutions(db);
expect(await continuationsOf(runId)).toHaveLength(0);
const [hold] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, issueId),
eq(issueRecoveryActions.status, "active")));
const nextCheckAt = firstAt.getTime() + FRESH_CONTINUATION_WINDOW_MS;
expect(hold?.evidence).toMatchObject({ freshContinuationCheck: { kind: "backoff", nextCheckAt } });
expect(await executionControlSchedule(db).disposition()).toBe(nextCheckAt);
await db.delete(activityLog).where(sql`${activityLog.id} in ${capped.map(row => row.id)}`);
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
'{freshContinuationCheck,nextCheckAt}', to_jsonb(${Date.now() - 1}::bigint))` }).where(eq(issueRecoveryActions.id, hold!.id));
await settleUnrecoverableExecutions(db);
expect(await continuationsOf(runId)).toHaveLength(1);
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "todo" });
}, 120_000);
});