Files
PaperClipAI/server/src/services/execution-control-work.ts
T
DottaandClaude Opus 5.5 68f641ba0b fix(recovery): continue a native run cut off by a sandbox client timeout (#21736)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime runs an agent's provider turn in a sandbox (for
example Daytona) and records each failure on the run and its native
coordinator
> - A Daytona SDK request can hit its client-side deadline during a
provider turn. The run then fails with `native_session_interrupted` in
an ambiguous state, because the turn started and its outcome is unknown
> - That failure blocks the task under a board-owned watchdog hold. No
recovery owner selected the hold, because its cause is not an
execution-reconciliation cause. The task never ran again, and only a
person's Retry could continue it
> - Recovery from an infrastructure fault must work automatically,
without a person
> - This pull request lets the execution-disposition pass hand that hold
to the existing fresh-continuation fallback
> - The benefit is that the task continues once, in an isolated fresh
run, and the cut-off turn is never replayed

## Linked Issues or Issue Description

**What happened?**
A native run on a Daytona execution environment (native runtime, an
OpenRouter model) started its provider turn. A Daytona request then
failed with `HTTP request timed out after 300000ms waiting for a
response. This is a client-side deadline
(DaytonaConfig.requestTimeoutMs, or the per-call operation/execution
timeout); any operation already started on the server may still be
running.` The run failed with `native_session_interrupted`. The task had
no further run: no automatic retry and no continuation.

**Expected behavior**
The task continues automatically, exactly once, in a fresh run from its
saved work. The cut-off provider turn is not replayed.

**Steps to reproduce**
1. Start a native run for an assigned task. Let its provider turn start
(a `turn.started` event is saved).
2. Make a sandbox request in that turn fail with the Daytona client-side
timeout error.
3. The executor saves the coordinator as `terminal_failure` with
`recoveryMode: ambiguous_state`, blocks the task, and writes an active
board hold with cause `native_session_interrupted`.
4. Run the execution-disposition worker
(`settleUnrecoverableExecutions`). Before this change, nothing happens
and the task stays blocked.

**Paperclip version or commit**
`master` at 9ebc276b42 (observed on a build from 5acf27bc3).

**Deployment mode**
Hosted cloud with a Daytona execution environment.

Causal path:
- The Daytona error is not a named class, so
`nativeSessionFailureSourceCode` returns the generic
`native_session_interrupted`.
- `nativeProviderRecoveryEvidence` returns `ambiguous_state`: provider
events exist, but there is no checkpoint with a provider identity. The
executor then makes the failure terminal on the first attempt (no
same-run resume) and writes a board hold whose cause is
`native_session_interrupted`.
- `unresolvedExecutionDispositions` and `heldExecutionWhere` select only
holds with `EXECUTION_RECONCILIATION_CAUSES` (or a narrow
`resume_native_run` shape). The safe replacement lane accepts
`native_session_interrupted` only for the historical
listener-unavailable case. The assigned-work sweep leaves a task blocked
by an active hold to the hold's owner. So no owner continued the task.

## What Changed

- `server/src/services/execution-recovery-resolution.ts`: the
disposition pass also selects an active watchdog hold with a
restart-class cause (`RESTART_CLASS_FAILURE_CODES`) for a terminal
native run with no retry pending, unless the safe replacement lane owns
that run. It hands the run to `continueFreshInTransaction` and records
the fallback's result on the hold, so a run that does not continue is
checked again only when due.
- `server/src/services/fresh-continuation-fallback.ts`: the fallback
treats an active watchdog hold with a restart-class cause as its own.
When it continues the task, it settles that hold and unblocks the task,
as it does for reconciliation holds.
- New red/green test
`server/src/services/native-runtime/native-client-timeout-continuation.integration.test.ts`.
It drives the real native executor with a provider turn that starts and
then fails with the Daytona client-timeout error.

The cut-off turn is not replayed. The fresh continuation isolates the
old run as it does for every continuation: the old run loses its write
authority, its remote workspace epoch is fenced (or the continuation
gets a new workspace), and the coordinator records `replacementDenied:
fresh_continuation`. The fallback keeps its existing checks:
person-owned causes, exact recovery still under way, and the per-task
hourly cap.

## Verification

- Red on `master`: `npx vitest run
src/services/native-runtime/native-client-timeout-continuation.integration.test.ts`
fails with no continuation recorded.
- Green with this change. The test asserts that the run fails with
`native_session_interrupted` and the Daytona error text in an ambiguous
state, then that two disposition passes record exactly one fresh
continuation, the task goes from `blocked` to `todo`, no active hold
remains, the old run is marked `replacementDenied`, and two
continuation-delivery passes wake the assignee exactly once.
- Focused suites (all tests that use the disposition pass, the
fresh-continuation fallback, or the execution-control schedule, 28
files): 27 files pass. `chat-channels.integration.test.ts` has 9
failures that are unrelated to this change (see the PR comment for the
`master` baseline).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify --explain` selects 2751 of 3578 files (a near-full run).
I did not run it locally. CI supplies that coverage.

## Risks

- Behavior change: a terminal native interruption hold that before
stayed with the board now continues automatically in a fresh run. This
is the intended product behavior. The fallback still refuses
person-owned causes and still respects the per-task cap.
- Holds with restart-class causes are now settled by the fallback when
it continues a run. Before, they could stay active next to the
continuation record.
- No migration. No API or UI contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) and code execution for tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:15:41 +00:00

135 lines
6.8 KiB
TypeScript

import { nextWorkspaceRecoveryAt } from "./system-recovery.js";
import { and, asc, eq, isNotNull, sql } from "drizzle-orm";
import { adapterAuthSessions, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
import { type DeliveryQueue } from "./delivery-work-notifications.js";
import { EXECUTION_RECONCILIATION_INTERVAL_MS } from "./execution-control-deadline.js";
import { obsoleteExecutionConversationHold, unresolvedExecutionDispositions } from "./execution-recovery-resolution.js";
import { heldExecutionsAwaitingFreshContinuation, nextFreshContinuationCheckAt } from "./fresh-continuation-fallback.js";
import { hasPendingNativeReplacements, nextNativeOwnershipProofDeadlineAt } from "./native-runtime/native-safe-replacement.js";
import { nextAbandonedNativeOwnerReleaseAt } from "./native-runtime/native-failed-run-owner-release.js";
import { hasPendingPauseResume } from "./issue-tree-resume.js";
import { hasPendingBudgetResume } from "./budget-resume-continuation.js";
import { continuationDeliveryDue } from "./recovery/continuation-delivery-attempt.js";
import { nextNativeInterruptionCheckAt } from "./native-interruption-continuation.js";
export function registerExecutionControlWork(
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
duties: ReadonlyArray<{
queue: DeliveryQueue;
run: () => Promise<unknown>;
next: () => Promise<number | null>;
onSettled?: () => void;
settleEmptyWhilePaused?: boolean;
}>,
canRun: () => boolean = () => true,
) {
const workers = duties.map(duty => {
let requested = 1;
let reconciled = 0;
let next: number | null = null;
let earliestRunAt = 0;
let pausedGeneration = 0, pausedPending = true;
const worker = coordinator.register(duty.queue, {
retryMs: EXECUTION_RECONCILIATION_INTERVAL_MS,
pausePassiveMaintenance: true,
onSettled: () => { requested += 1; duty.onSettled?.(); },
run: async () => {
if (!canRun()) { requested += 1; next = null; return; }
const now = Date.now();
if (now < earliestRunAt) {
if (reconciled !== requested || (next !== null && next <= now)) next = earliestRunAt;
return;
}
const generation = ++requested;
earliestRunAt = now + EXECUTION_RECONCILIATION_INTERVAL_MS;
await duty.run();
const deadline = await duty.next();
next = deadline === null ? null : Math.max(deadline, earliestRunAt);
reconciled = generation;
},
hasPending: async (runAllowed = true) => {
if (reconciled === requested) return false;
if (runAllowed || !duty.settleEmptyWhilePaused || !canRun() || reconciled === 0) return true;
if (pausedGeneration !== requested) {
const generation = requested;
const deadline = await duty.next();
if (generation !== requested) return true;
pausedGeneration = generation; pausedPending = deadline !== null; next = deadline;
if (!pausedPending) reconciled = generation;
}
return pausedPending;
},
nextRunAt: () => next,
});
return { ...worker, wake: () => { requested += 1; worker.wake(); } };
});
return { ready: Promise.all(workers.map(worker => worker.ready)), wake: () => { for (const worker of workers) worker.wake(); } };
}
export function executionControlSchedule(db: Db) {
const retry = () => Date.now() + EXECUTION_RECONCILIATION_INTERVAL_MS;
const deadline = (dates: Array<Date | null | undefined>) => {
const at = dates.flatMap(date => date ? [date.getTime()] : []);
if (!at.length) return null;
const earliest = Math.min(...at);
return earliest > Date.now() ? earliest : retry();
};
return {
async finalization() {
const [native, legacy, abandonedOwner] = await Promise.all([
db.select({ at: nativeRunFinalizations.controlDeadlineAt }).from(nativeRunFinalizations)
.where(isNotNull(nativeRunFinalizations.controlDeadlineAt)).orderBy(asc(nativeRunFinalizations.controlDeadlineAt)).limit(1),
db.select({ at: heartbeatRuns.executionControlDeadlineAt }).from(heartbeatRuns)
.where(isNotNull(heartbeatRuns.executionControlDeadlineAt)).orderBy(asc(heartbeatRuns.executionControlDeadlineAt)).limit(1),
nextAbandonedNativeOwnerReleaseAt(db),
]);
return deadline([native[0]?.at, legacy[0]?.at, abandonedOwner]);
},
async workspaceRecovery() {
const at = await nextWorkspaceRecoveryAt(db);
return at === null ? null : deadline([new Date(at)]);
},
async replacement() {
if (await hasPendingNativeReplacements(db)) return retry();
const at = await nextNativeOwnershipProofDeadlineAt(db);
return at === null ? null : deadline([new Date(at)]);
},
async continuation() {
const pending = and(
eq(issueRecoveryActions.status, "resolved"),
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
);
const rows = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions)
.where(and(pending, continuationDeliveryDue(new Date()))).limit(1);
if (rows.length || await hasPendingPauseResume(db) || await hasPendingBudgetResume(db)) return retry();
const [waiting] = await db.select({ at: sql<string | null>`min((${issueRecoveryActions.evidence}->>'continuationDeliveryRetryAt')::timestamptz)` })
.from(issueRecoveryActions).where(pending);
return deadline([waiting?.at ? new Date(waiting.at) : null]);
},
async status() {
const rows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(isNotNull(heartbeatRuns.executionStatusDeliveryId)).limit(1);
return rows.length ? retry() : null;
},
async disposition() {
const now = new Date();
const [conversation, execution, held, nextCheck, nextInterruptionCheck] = await Promise.all([
db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(obsoleteExecutionConversationHold()).limit(1),
unresolvedExecutionDispositions(db, 1),
heldExecutionsAwaitingFreshContinuation(db, now, 1),
nextFreshContinuationCheckAt(db),
nextNativeInterruptionCheckAt(db),
]);
return conversation.length || execution.length || held.length ? retry()
: deadline([nextCheck ? new Date(nextCheck) : null, nextInterruptionCheck ? new Date(nextInterruptionCheck) : null]);
},
async localLogin() {
const rows = await db.select({ at: adapterAuthSessions.expiresAt }).from(adapterAuthSessions).where(and(
eq(adapterAuthSessions.connectionMethod, "local_subscription"), isNotNull(adapterAuthSessions.expiresAt),
)).orderBy(asc(adapterAuthSessions.expiresAt)).limit(1);
return deadline([rows[0]?.at]);
},
};
}