mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
fix(tasks): show a pending native same-run resume instead of a competing Retry (#19503)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native run cannot reattach to its provider after a control-plane restart, the native coordinator resumes the same run by itself after about 30 seconds, up to three attempts > - The run projection shows this correctly as "Retry scheduled" with only `inspect_run` permitted > - The task interruption shown on the issue read only system-recovery evidence and `scheduledRetryAt`. It never read the coordinator's `nextAttemptAt`, so it showed `state: failed, canRetry: true` and offered a manual Retry while the automatic resume was still pending > - The two surfaces disagree, and a user click can race the automatic resume. That is one way a Retry gets parked behind "execution checks" > - This pull request makes the task interruption read the coordinator's pending resume time > - The benefit is one consistent story: while the automatic resume is pending, the task shows it with its time and no competing Retry. If recovery ends, exactly one Retry returns ## Linked Issues or Issue Description Refs #19487, #19497, #19426. **What happened?** A native run failed with `native_session_interrupted` after a restart. Its finalization row had `phase: retryable_failure` and `nextAttemptAt` 30 seconds later. The run projection said "Retry scheduled". The issue interruption said `state: failed, canRetry: true, nextRetryAt: null`. **What did you expect?** While the automatic resume is pending, the task shows that it is resuming, with the time, and no Retry. When the resume fails for good or never runs, exactly one Retry that works on the first click. **Steps to reproduce** 1. Run a native task and restart the control plane so the native attach fails. 2. Read `GET /api/issues/:id` while the coordinator's resume is pending. 3. Observe `interruption.canRetry: true` while `execution.phase` is `retry_scheduled`. ## What Changed - `task-interruption-presentation.ts`: the batched run read selects the native coordinator's `next_attempt_at` when the native run failed, has no result, and its finalization phase is `retryable_failure`. - A pending resume in the future shows `retry_scheduled` with its time and `canRetry: false`. - A due resume shows `working` for up to two minutes while its dispatch runs. After that, or once recovery is terminal, the task falls through to the existing single Retry. ## Verification - `task-interruption-presentation.test.ts`: new case for a future resume, a due resume, a stale resume, and no resume. It fails before the change (the task offered Retry) and passes after. - `task-interruption-query.test.ts` (embedded Postgres): the real batched query shows the pending resume for both list and detail reads, and returns one Retry after the coordinator becomes `terminal_failure`. - All 40 presentation tests and 17 query tests pass. Server typecheck passes. ## Risks - Low risk. This changes only the read model for tasks whose native coordinator holds a pending same-run resume. - If a resume never dispatches, Retry is hidden for up to two minutes after its due time before it returns. - The `POST /issues/:id/retry` endpoint is unchanged. A client that ignores `canRetry` can still submit a Retry during the resume. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ff1d1dea5d
commit
543aa4d331
3 files changed
+61
-5
No files matched your search
@@ -80,6 +80,24 @@ describe("unadmitted chat Retry presentation", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// The native coordinator resumes the same run itself after an attach failure.
|
||||
// A manual Retry offered meanwhile would race that automatic resume.
|
||||
it("shows a pending native same-run resume instead of a competing Retry", () => {
|
||||
const now = Date.parse("2026-11-06T15:25:50.000Z");
|
||||
const failed = { ...run, runtimeMode: "native" as const, errorCode: "native_session_interrupted", resultJson: null };
|
||||
const resumeAt = new Date("2026-11-06T15:26:18.687Z");
|
||||
expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: resumeAt }, agent, paused: false, approval: false }, now))
|
||||
.toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() });
|
||||
// A due resume is being dispatched; it is still not the user's turn.
|
||||
expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 30_000) }, agent, paused: false, approval: false }, now))
|
||||
.toMatchObject({ state: "working", canRetry: false });
|
||||
// A resume that never ran, or no resume at all, leaves exactly one Retry.
|
||||
expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 10 * 60_000) }, agent, paused: false, approval: false }, now))
|
||||
.toMatchObject({ state: "failed", canRetry: true });
|
||||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false }, now))
|
||||
.toMatchObject({ state: "failed", canRetry: true });
|
||||
});
|
||||
|
||||
it("explains a persisted Retry wait without exposing the saved payload", () => {
|
||||
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
||||
const wait = { id: "retry", requestedAt: new Date(), sourceRunId: failed.id,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { and, desc, eq, inArray, isNull, sql, type SQL } from "drizzle-orm";
|
||||
import { agents, agentWakeupRequests, approvals, chatActions, companies, heartbeatRuns, issueApprovals, issueRecoveryActions,
|
||||
issues, issueTreeHoldMembers, issueTreeHolds, projects, type Db } from "@paperclipai/db";
|
||||
issues, issueTreeHoldMembers, issueTreeHolds, nativeRunFinalizations, projects, type Db } from "@paperclipai/db";
|
||||
import { unionAll } from "drizzle-orm/pg-core";
|
||||
import { HEARTBEAT_RUN_STATUSES, readProviderFailure, type TaskInterruptionPresentation } from "@paperclipai/shared";
|
||||
import { captureTaskInterruptionFailure } from "../sentry.js";
|
||||
@@ -26,11 +26,17 @@ import { AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON,
|
||||
const resourceWaitRetryReasons = new Set<string>([WORKSPACE_BUSY_RETRY_REASON, AI_CONNECTION_BUSY_RETRY_REASON,
|
||||
AI_CONNECTION_POOL_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON]);
|
||||
|
||||
// A due same-run resume dispatches within seconds; past this, offer Retry.
|
||||
const NATIVE_RESUME_DISPATCH_GRACE_MS = 2 * 60_000;
|
||||
|
||||
type ProjectionIssue = Pick<typeof issues.$inferSelect, "id" | "companyId" | "status" | "assigneeAgentId" | "projectId">;
|
||||
type ProjectionRun = Pick<typeof heartbeatRuns.$inferSelect,
|
||||
"id" | "status" | "scheduledRetryAt" | "errorCode" | "livenessState" | "resultJson"> &
|
||||
Partial<Pick<typeof heartbeatRuns.$inferSelect, "agentId" | "error" | "companyId" | "nativeIssueId" | "runtimeMode" | "startedAt"
|
||||
| "scheduledRetryReason">>;
|
||||
| "scheduledRetryReason">> & {
|
||||
/** Due time of the native coordinator's own same-run resume, if one is pending. */
|
||||
nativeResumeAt?: Date | null;
|
||||
};
|
||||
type ProjectionWait = { id: string; requestedAt: Date; sourceRunId?: string | null; executionWaitReason?: string | null;
|
||||
preparationStatus?: string | null; preparationRetryAt?: number | string | null };
|
||||
type ProjectionState = {
|
||||
@@ -148,10 +154,16 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio
|
||||
}
|
||||
if (run.status === "scheduled_retry" && resourceWaitRetryReasons.has(run.scheduledRetryReason ?? "") && !system)
|
||||
return { ...base, state: "working" };
|
||||
const retryAt = system?.nextAttemptAt ?? run.scheduledRetryAt?.getTime();
|
||||
// The native coordinator resumes this same run by itself. A manual Retry
|
||||
// would race it, so show the resume until it is due and briefly while its
|
||||
// dispatch runs. A resume that never ran falls through to one Retry.
|
||||
const nativeResumeAt = run.nativeResumeAt?.getTime();
|
||||
const retryAt = system?.nextAttemptAt ?? nativeResumeAt ?? run.scheduledRetryAt?.getTime();
|
||||
if (retryAt && retryAt > now) return { ...base, state: "retry_scheduled",
|
||||
nextRetryAt: new Date(retryAt).toISOString(), canRetry: !preparationPending,
|
||||
nextRetryAt: new Date(retryAt).toISOString(), canRetry: !preparationPending && !nativeResumeAt,
|
||||
...(system?.nextAttemptAt && recoveryStep ? { recoveryStep } : {}) };
|
||||
if (nativeResumeAt && !system && now - nativeResumeAt < NATIVE_RESUME_DISPATCH_GRACE_MS)
|
||||
return { ...base, state: "working" };
|
||||
if (recoveryRunning) return { ...base, state: "working", ...(recoveryStep ? { recoveryStep } : {}) };
|
||||
if (retryWait?.sourceRunId === run.id && retryWait.executionWaitReason) {
|
||||
const reason = Object.hasOwn(retryWaitMessages, retryWait.executionWaitReason) ? retryWait.executionWaitReason : "execution_recovery";
|
||||
@@ -203,6 +215,10 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
or ${heartbeatRuns.errorCode} = 'workspace_validation_failed' then ${heartbeatRuns.error} else null end`,
|
||||
errorCode: heartbeatRuns.errorCode, livenessState: heartbeatRuns.livenessState, scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
|
||||
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
|
||||
nativeResumeAt: sql<Date | null>`(select ${nativeRunFinalizations.nextAttemptAt} from ${nativeRunFinalizations}
|
||||
where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
|
||||
and ${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} = 'failed'
|
||||
and ${nativeRunFinalizations.resultId} is null and ${nativeRunFinalizations.phase} = 'retryable_failure')`.mapWith(value => value == null ? null : new Date(value)),
|
||||
// The run whose resultJson supplied providerFailure. A scheduled retry
|
||||
// shows its predecessor's reason, so redaction must use that run's registry.
|
||||
providerFailureRunId: sql<string>`case when ${heartbeatRuns.resultJson}->'providerFailure' is null
|
||||
|
||||
Reference in new issue
Block a user