mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
fix(recovery): keep a running run visible when a dropped continuation is newer (#21907)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task shows a Retry banner when its latest run failed and nothing
else continues the task
> - The banner reads "the latest run" by creation time only
> - The queue cancels a queued system continuation before it starts when
the task no longer needs it. That run is newer than the run that is
really working
> - So a task showed "failed" with Retry for about 79 seconds while its
run was working
> - The stranded-work sweep also queued one of those extra
continuations, because it did not see a native status wake that was
still waiting for dispatch
> - This pull request ignores never-started, superseded continuations in
the banner, and makes the sweep see a pending native status wake
> - The benefit is no false Retry offer while an agent works, and no
duplicate continuation after a slow finalization
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave (6 rolling deploys, 207 tasks), one task showed
state `failed` with Retry for 75–79 seconds while its run was running.
Two queued system continuations for that task were cancelled before they
started with `recovery_no_longer_needed`. That cancel is correct. But
the newest cancelled run became the "latest run" in the interruption
projection. The projection then fell through to `{ state: "failed",
canRetry: true }`.
The first of the two extra continuations came from a double queue. The
finishing run asked for its own continuation through a native status
wake (`agent_tool_call_loop`). Its finalization did not apply at once
("native result persisted but finalization did not apply; the
reconciliation loop will retry"), so the dispatch of that wake waited
about 17 seconds. In that window, the stranded-work sweep saw an
`in_progress` task with a succeeded run and no live path, and queued its
own continuation (`issue.productive_terminal_continuation_recovery`).
The sweep's live-path check counted only runs and
`deferred_issue_execution` wakes. It did not count a native status wake
that was still `queued` or `claimed` without a run. When the native
status wake was dispatched later, the queue correctly dropped it. The
dropped wake also carried the better instruction (the tool-loop
guidance).
**Expected behavior**
A task with a running run shows "working" with no Retry. One finished
run produces one continuation.
**Steps to reproduce**
1. Give a task a running run R1.
2. Add a newer run R2 for the task with status `cancelled`, no
`startedAt`, and error code `recovery_no_longer_needed`.
3. Read the task interruption. Before this change: `failed`, `canRetry:
true`, observed run R2.
For the sweep: give an `in_progress` task a succeeded native run and a
`native-status-committer` wake in `queued` or `claimed` with no run. Run
`reconcileStrandedAssignedIssues`. Before this change it queues a second
continuation.
**Paperclip version or commit**
master `c5218f5ec4` (wave), reproduced on `b4fa1c7fb1`.
## What Changed
- `task-interruption-presentation.ts`: the latest-run candidates skip a
run that is `cancelled`, never started (`startedAt` is null), and has
error code `recovery_no_longer_needed` or `issue_closed_before_start`.
The filter is in each per-arm `limit 1` index scan, so the query shape
and cost stay the same.
- `recovery/service.ts`: `hasActiveExecutionPath` also counts a
`native-status-committer` wake in `queued` or `claimed` with no run for
the issue. The native status wake dispatcher owns that wake, so the
sweep does not queue a second continuation and does not escalate the
task while it waits.
- Tests (red on master, green here):
- `task-interruption-query.test.ts`: a running R1 plus a newer
never-started R2 cancelled with each code shows `working`, `canRetry:
false`, observed R1. Negatives keep their behavior: a cancelled run that
had started, and a run a person cancelled, still show `failed` with
Retry.
- `heartbeat-process-recovery.test.ts`: a succeeded native run with its
own status wake still `queued` or `claimed` gets no second continuation
and no escalation.
## Verification
- `pnpm verify` exited 2 (opaque setup dependency), so I ran focused
suites:
- `server/src/__tests__/task-interruption-query.test.ts`,
`task-interruption-automatic-path.test.ts`,
`issue-confirm-run-stopped-routes.test.ts`: 83/83 pass. This includes
the query-plan test that bounds `heartbeat_runs` rows read.
- `server/src/__tests__/heartbeat-process-recovery.test.ts`,
`server/src/services/recovery/*.test.ts`, the interruption suites and
`task-interruption-presentation.test.ts` together: 18 files, 871/871
pass.
- Red/green: the new interruption tests fail on master with `state:
"failed", canRetry: true` and the dropped run as observed run. The new
sweep tests fail on master with `continuationRequeued: 1`.
- `tsc --noEmit` for `server` passes.
## Risks
- Low. The banner change only hides runs that never started and were
dropped as superseded. Person-cancelled runs, started runs, and other
cancel codes behave as before.
- The live-path change makes the sweep wait while a native status wake
is pending. The dispatcher always settles such a wake: it dispatches it,
or marks it skipped. A throttled wake stays queued only until its
throttle time, so the sweep waits at most that long.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
698c371b1b
commit
5561a4068c
4 files changed
+84
-2
No files matched your search
@@ -17312,6 +17312,40 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["queued", "claimed"] as const)(
|
||||
"does not queue a second continuation while the finished run's own status wake (%s) awaits dispatch (Gate C C1 t169)",
|
||||
async (intentStatus) => {
|
||||
const { companyId, agentId, issueId, runId } =
|
||||
await seedStrandedIssueFixture({
|
||||
status: "in_progress",
|
||||
runStatus: "succeeded",
|
||||
livenessState: "advanced",
|
||||
});
|
||||
await db.insert(agentWakeupRequests).values({
|
||||
companyId,
|
||||
agentId,
|
||||
source: "automation",
|
||||
triggerDetail: "system",
|
||||
reason: "issue_continuation_needed",
|
||||
status: intentStatus,
|
||||
claimedAt: intentStatus === "claimed" ? new Date() : null,
|
||||
requestedByActorType: "system",
|
||||
requestedByActorId: "native-status-committer",
|
||||
idempotencyKey: `agent_tool_call_loop:${runId}`,
|
||||
payload: { issueId, taskId: issueId, sourceRunId: runId, continuationCause: "agent_tool_call_loop" },
|
||||
});
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId }).where(eq(heartbeatRuns.id, runId));
|
||||
const heartbeat = heartbeatService(db);
|
||||
|
||||
const result = await heartbeat.reconcileStrandedAssignedIssues();
|
||||
expect(result.continuationRequeued).toBe(0);
|
||||
expect(result.escalated).toBe(0);
|
||||
expect(result.skipped).toBe(1);
|
||||
expect(await selectRecoveryRunsByAgent(agentId)).toHaveLength(1);
|
||||
expect((await selectRecoveryIssueById(issueId))?.status).toBe("in_progress");
|
||||
},
|
||||
);
|
||||
|
||||
it("does not replenish exhausted legacy repair from a recent comment", async () => {
|
||||
const { companyId, agentId, issueId, runId } =
|
||||
await seedStrandedIssueFixture({
|
||||
|
||||
@@ -747,6 +747,43 @@ describe.skipIf(!support.supported)("task interruption latest run query", () =>
|
||||
taskId: reassigned!.id, agentAdapter: "process",
|
||||
}));
|
||||
});
|
||||
describe("a newer continuation dropped before it started (Gate C C1 t169)", () => {
|
||||
const seed = async (newer: Partial<typeof heartbeatRuns.$inferInsert>) => {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), runningId = randomUUID(), droppedId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Continuing task", issuePrefix: `C${companyId.slice(0, 7)}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Continuing agent", adapterType: "paperclip_runner" });
|
||||
const [issue] = await db.insert(issues).values({ companyId, title: "Keep working",
|
||||
status: "in_progress", assigneeAgentId: agentId }).returning();
|
||||
const now = Date.now();
|
||||
await db.insert(heartbeatRuns).values({ id: runningId, companyId, agentId, nativeIssueId: issue!.id, scopeKind: "issue",
|
||||
status: "running", runtimeMode: "native", createdAt: new Date(now - 60_000), startedAt: new Date(now - 59_000) });
|
||||
await db.insert(heartbeatRuns).values({ id: droppedId, companyId, agentId, nativeIssueId: issue!.id, scopeKind: "issue",
|
||||
status: "cancelled", runtimeMode: "native", createdAt: new Date(now - 30_000), finishedAt: new Date(now - 30_000),
|
||||
contextSnapshot: { issueId: issue!.id, wakeReason: "issue_continuation_needed" }, ...newer });
|
||||
return { companyId, issue: issue!, runningId, droppedId };
|
||||
};
|
||||
|
||||
it.each(["recovery_no_longer_needed", "issue_closed_before_start"])(
|
||||
"shows the running run, not a failed attempt (%s)", async errorCode => {
|
||||
const f = await seed({ errorCode, error: "The task no longer needed this recovery run when its turn came." });
|
||||
for (const presentation of [(await taskInterruptionPresentations(db, f.companyId, [f.issue])).get(f.issue.id),
|
||||
await taskInterruptionPresentation(db, f.issue)]) {
|
||||
expect(presentation).toMatchObject({ state: "working", canRetry: false, observedRunId: f.runningId });
|
||||
}
|
||||
});
|
||||
|
||||
it("still reports a cancelled run that had started", async () => {
|
||||
const f = await seed({ errorCode: "recovery_no_longer_needed", startedAt: new Date(Date.now() - 29_000) });
|
||||
expect((await taskInterruptionPresentations(db, f.companyId, [f.issue])).get(f.issue.id))
|
||||
.toMatchObject({ state: "failed", canRetry: true, observedRunId: f.droppedId });
|
||||
});
|
||||
|
||||
it("still reports a run a person cancelled before it started", async () => {
|
||||
const f = await seed({ errorCode: "cancelled", resultJson: { cancelledByActorType: "user", cancelledByUserId: "board" } });
|
||||
expect((await taskInterruptionPresentations(db, f.companyId, [f.issue])).get(f.issue.id))
|
||||
.toMatchObject({ state: "failed", canRetry: true, observedRunId: f.droppedId });
|
||||
});
|
||||
});
|
||||
// N-03: ONB-1 on staging waited 16 minutes on an overdue workspace_busy
|
||||
// retry. Nothing failed, yet the banner offered a Retry that cannot help.
|
||||
it.each(["workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", "runner_revision_wait", "storage_unavailable_wait"])(
|
||||
|
||||
@@ -42,6 +42,7 @@ import {
|
||||
import { CONTROLLER_LEASE_LOST_FAILURE_CODE, hasLiveLegacyController, legacyControllerBootId, liveLegacyControllerExpiresAt } from "../legacy-controller-lease.js";
|
||||
import { isCompletedOnboardingHandoffWake } from "../chat-completion-delivery.js";
|
||||
import { instanceSettingsService } from "../instance-settings.js";
|
||||
import { NATIVE_STATUS_COMMITTER_ACTOR_ID } from "../issue-rewake-throttle.js";
|
||||
import { isWaitingConversation, settleConversationTurnForRun, deliverConversationComments } from "../agent-conversations.js";
|
||||
import {
|
||||
and,
|
||||
@@ -1259,7 +1260,15 @@ export function recoveryService(
|
||||
.where(
|
||||
and(
|
||||
eq(agentWakeupRequests.companyId, companyId),
|
||||
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
||||
or(
|
||||
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
||||
and(
|
||||
eq(agentWakeupRequests.requestedByActorType, "system"),
|
||||
eq(agentWakeupRequests.requestedByActorId, NATIVE_STATUS_COMMITTER_ACTOR_ID),
|
||||
inArray(agentWakeupRequests.status, ["queued", "claimed"]),
|
||||
isNull(agentWakeupRequests.runId),
|
||||
),
|
||||
),
|
||||
sql`${agentWakeupRequests.payload} ->> 'issueId' = ${issueId}`,
|
||||
agentId ? eq(agentWakeupRequests.agentId, agentId) : sql`true`,
|
||||
),
|
||||
|
||||
@@ -440,8 +440,10 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
const requestedUuid = sql<string>`case when ${requestedIssueId} ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
|
||||
then ${requestedIssueId}::uuid else null end`;
|
||||
const legacyIssue = sql<string>`${heartbeatRuns.contextSnapshot}->>'issueId'`;
|
||||
const attempt = sql`not (${heartbeatRuns.status} = 'cancelled' and ${heartbeatRuns.startedAt} is null
|
||||
and coalesce(${heartbeatRuns.errorCode}, '') in ('recovery_no_longer_needed', 'issue_closed_before_start'))`;
|
||||
const candidate = (condition: ReturnType<typeof and>) => db.select({ id: heartbeatRuns.id, createdAt: heartbeatRuns.createdAt })
|
||||
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), condition))
|
||||
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), condition, attempt))
|
||||
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
||||
const candidates = unionAll(
|
||||
candidate(eq(heartbeatRuns.nativeIssueId, requestedUuid)),
|
||||
|
||||
Reference in new issue
Block a user