mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs an agent's provider turn in a sandbox (for example Daytona) and records each failure on the run and its native coordinator > - A Daytona SDK request can hit its client-side deadline during a provider turn. The run then fails with `native_session_interrupted` in an ambiguous state, because the turn started and its outcome is unknown > - That failure blocks the task under a board-owned watchdog hold. No recovery owner selected the hold, because its cause is not an execution-reconciliation cause. The task never ran again, and only a person's Retry could continue it > - Recovery from an infrastructure fault must work automatically, without a person > - This pull request lets the execution-disposition pass hand that hold to the existing fresh-continuation fallback > - The benefit is that the task continues once, in an isolated fresh run, and the cut-off turn is never replayed ## Linked Issues or Issue Description **What happened?** A native run on a Daytona execution environment (native runtime, an OpenRouter model) started its provider turn. A Daytona request then failed with `HTTP request timed out after 300000ms waiting for a response. This is a client-side deadline (DaytonaConfig.requestTimeoutMs, or the per-call operation/execution timeout); any operation already started on the server may still be running.` The run failed with `native_session_interrupted`. The task had no further run: no automatic retry and no continuation. **Expected behavior** The task continues automatically, exactly once, in a fresh run from its saved work. The cut-off provider turn is not replayed. **Steps to reproduce** 1. Start a native run for an assigned task. Let its provider turn start (a `turn.started` event is saved). 2. Make a sandbox request in that turn fail with the Daytona client-side timeout error. 3. The executor saves the coordinator as `terminal_failure` with `recoveryMode: ambiguous_state`, blocks the task, and writes an active board hold with cause `native_session_interrupted`. 4. Run the execution-disposition worker (`settleUnrecoverableExecutions`). Before this change, nothing happens and the task stays blocked. **Paperclip version or commit** `master` at9ebc276b42(observed on a build from5acf27bc3). **Deployment mode** Hosted cloud with a Daytona execution environment. Causal path: - The Daytona error is not a named class, so `nativeSessionFailureSourceCode` returns the generic `native_session_interrupted`. - `nativeProviderRecoveryEvidence` returns `ambiguous_state`: provider events exist, but there is no checkpoint with a provider identity. The executor then makes the failure terminal on the first attempt (no same-run resume) and writes a board hold whose cause is `native_session_interrupted`. - `unresolvedExecutionDispositions` and `heldExecutionWhere` select only holds with `EXECUTION_RECONCILIATION_CAUSES` (or a narrow `resume_native_run` shape). The safe replacement lane accepts `native_session_interrupted` only for the historical listener-unavailable case. The assigned-work sweep leaves a task blocked by an active hold to the hold's owner. So no owner continued the task. ## What Changed - `server/src/services/execution-recovery-resolution.ts`: the disposition pass also selects an active watchdog hold with a restart-class cause (`RESTART_CLASS_FAILURE_CODES`) for a terminal native run with no retry pending, unless the safe replacement lane owns that run. It hands the run to `continueFreshInTransaction` and records the fallback's result on the hold, so a run that does not continue is checked again only when due. - `server/src/services/fresh-continuation-fallback.ts`: the fallback treats an active watchdog hold with a restart-class cause as its own. When it continues the task, it settles that hold and unblocks the task, as it does for reconciliation holds. - New red/green test `server/src/services/native-runtime/native-client-timeout-continuation.integration.test.ts`. It drives the real native executor with a provider turn that starts and then fails with the Daytona client-timeout error. The cut-off turn is not replayed. The fresh continuation isolates the old run as it does for every continuation: the old run loses its write authority, its remote workspace epoch is fenced (or the continuation gets a new workspace), and the coordinator records `replacementDenied: fresh_continuation`. The fallback keeps its existing checks: person-owned causes, exact recovery still under way, and the per-task hourly cap. ## Verification - Red on `master`: `npx vitest run src/services/native-runtime/native-client-timeout-continuation.integration.test.ts` fails with no continuation recorded. - Green with this change. The test asserts that the run fails with `native_session_interrupted` and the Daytona error text in an ambiguous state, then that two disposition passes record exactly one fresh continuation, the task goes from `blocked` to `todo`, no active hold remains, the old run is marked `replacementDenied`, and two continuation-delivery passes wake the assignee exactly once. - Focused suites (all tests that use the disposition pass, the fresh-continuation fallback, or the execution-control schedule, 28 files): 27 files pass. `chat-channels.integration.test.ts` has 9 failures that are unrelated to this change (see the PR comment for the `master` baseline). - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify --explain` selects 2751 of 3578 files (a near-full run). I did not run it locally. CI supplies that coverage. ## Risks - Behavior change: a terminal native interruption hold that before stayed with the board now continues automatically in a fresh run. This is the intended product behavior. The fallback still refuses person-owned causes and still respects the per-task cap. - Holds with restart-class causes are now settled by the fallback when it continues a run. Before, they could stay active next to the continuation record. - No migration. No API or UI contract change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits) and code execution for tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
45 lines
2.8 KiB
TypeScript
45 lines
2.8 KiB
TypeScript
import { and, eq, inArray, isNull, not, sql } from "drizzle-orm";
|
|
import { heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
|
import { RESTART_CLASS_FAILURE_CODES, isRestartClassFailureCode } from "./continuation-cause.js";
|
|
import { canonicalExecutionUuid } from "./execution-identity.js";
|
|
import { continueFreshInTransaction, recordFreshContinuationCheck } from "./fresh-continuation-fallback.js";
|
|
import { nativeReplacementPending } from "./native-runtime/native-safe-replacement.js";
|
|
|
|
export function nativeInterruptionHold() {
|
|
return and(
|
|
inArray(issueRecoveryActions.cause, [...RESTART_CLASS_FAILURE_CODES]),
|
|
eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.nativePhase, "terminal_failure"),
|
|
eq(nativeRunFinalizations.phase, "terminal_failure"), isNull(nativeRunFinalizations.nextAttemptAt),
|
|
not(nativeReplacementPending()!),
|
|
)!;
|
|
}
|
|
|
|
export async function continueNativeInterruption(tx: Db, input: {
|
|
action: typeof issueRecoveryActions.$inferSelect;
|
|
run: typeof heartbeatRuns.$inferSelect;
|
|
coordinator: typeof nativeRunFinalizations.$inferSelect | undefined;
|
|
now: Date;
|
|
}): Promise<boolean> {
|
|
const { action, run, coordinator, now } = input;
|
|
if (!isRestartClassFailureCode(action.cause) || run.runtimeMode !== "native" || run.nativePhase !== "terminal_failure"
|
|
|| coordinator?.phase !== "terminal_failure" || coordinator.nextAttemptAt || coordinator.leaseOwner || coordinator.resultId)
|
|
return false;
|
|
const fresh = await continueFreshInTransaction(tx, { companyId: action.companyId, runId: run.id, trigger: "recovery_refused", now });
|
|
await recordFreshContinuationCheck(tx, { companyId: action.companyId, actionId: action.id }, fresh, now);
|
|
return true;
|
|
}
|
|
|
|
export async function nextNativeInterruptionCheckAt(db: Db): Promise<number | null> {
|
|
const [row] = await db.select({ at: sql<string | null>`min((${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}')::bigint)` })
|
|
.from(issueRecoveryActions)
|
|
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, issueRecoveryActions.companyId),
|
|
eq(heartbeatRuns.id, canonicalExecutionUuid(sql`${issueRecoveryActions.evidence}->>'runId'`))))
|
|
.innerJoin(nativeRunFinalizations, and(eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
|
|
eq(nativeRunFinalizations.runId, heartbeatRuns.id)))
|
|
.where(and(inArray(issueRecoveryActions.status, ["active", "escalated"]), eq(issueRecoveryActions.kind, "active_run_watchdog"),
|
|
sql`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}' is not null`,
|
|
inArray(heartbeatRuns.status, ["failed", "timed_out", "interrupted", "cancelled"]),
|
|
isNull(nativeRunFinalizations.leaseOwner), isNull(nativeRunFinalizations.resultId), nativeInterruptionHold()));
|
|
return row?.at ? Number(row.at) : null;
|
|
}
|