mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
fix(presentation): a task with a continuation waiting to start shows as continuing, not as needing Retry (#21353)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run dies (for example during a server swap), the
fresh-continuation fallback records that Paperclip will continue the
task in a fresh run from saved work
> - The record says `continuationDelivery: "pending"` until a delivery
pass starts the new run. Delivery is paced, so a record can wait a long
time behind an older backlog
> - The task view showed such a task as "working" only for 15 minutes
after the record was made. After that it showed a failed attempt with a
Retry button, although the record still said Paperclip would continue
the task
> - Product rule: a task that the system has recorded it will continue
by itself never shows as failed with a Retry button
> - This pull request ties the "continuing" state to the record's own
delivery state instead of a timer
> - The benefit is that the task view stays truthful however long
delivery takes, and Retry comes back as soon as the system gives up
## Linked Issues or Issue Description
Refs #21264, #21324, #21297, #21320, #21200
**What happened?**
On hosted staging (build d1d71a7755, which contains #21264 and #21324),
five tasks had a run that died after a server swap. Each task had a
recorded fresh continuation that was not delivered yet. The primary
recovery record had cause `native_continuation_requires_reconciliation`,
status `resolved`, `continuationDelivery: "pending"`, isolation
`remote_epoch`, and no `continuationDeliveryAttemptedAt`. The other
holds of the task were coalesced into it. The records were 30 to 110
minutes old. The task view showed responsibility `system`, state
`failed`, and `canRetry: true`.
**Expected behavior**
While a continuation is recorded and its delivery is pending, the task
shows as continuing (`working`, `canRetry: false`, responsibility
`system`). This lasts until delivery starts the new run (then the new
run decides the state) or the system gives up (`invalidated`; then Retry
is offered).
**Steps to reproduce**
1. Record a fresh continuation for a failed run (`continuationDelivery:
"pending"`) and coalesce the task's other holds into it.
2. Let more than 15 minutes pass without delivery.
3. Read the task. It shows `failed` with Retry.
## What Changed
- `taskInterruptionPresentations` in
`server/src/services/task-interruption-presentation.ts`: a primary
fresh-continuation record with `continuationDelivery: "pending"` reads
as "continuing" with no time limit. Before, this applied only for 15
minutes (`RECOVERY_CONTINUATION_DELIVERY_BOUND_MS`). The state is the
existing `working` state that the code already uses for a recorded
continuation; no new state is added. The UI shows the agent's turn as
queued and shows no Retry button.
- A refused delivery attempt (`continuationDeliveryAttemptedAt`) does
not end the state. The record stays `pending` and the delivery pass
tries again on the next pass, so the system still owns the next step.
- The select no longer reads the hold time columns, because the
projection does not use them now.
- `recovery-wake-herd.test.ts`: the #21324 test that expected "not
working" past the bound now expects `working` with `canRetry: false`,
and then proves that the delivery pass still delivers that overdue
continuation.
- No change to the delivery pass, the fallback, or the Retry path.
## What a Retry click does in this state today
`requestTaskRetry` admits a person's Retry as a user wake. The queue's
"a pending fresh continuation absorbs this wake" check
(`hasPendingFreshContinuation` in `heartbeat/queue.ts`) applies only to
system wakes, so the Retry starts its own run from saved work. The
delivery pass then sees a run created after the record and marks the
continuation `delivered` (superseded). In the normal order this does not
start a second run. But the click only asks for what the system already
recorded it would do, and it races the delivery pass: if the pass has
already checked for a newer run and is waking the agent when the click
is admitted, the task can get two continuation runs, one after the
other. That is why the button must not show while the continuation is
pending.
## Verification
- Red (before the fix), `pnpm exec vitest run
src/__tests__/task-interruption-automatic-path.test.ts` in `server/`: 2
failed. The exact staging record shape at 30 minutes read `state:
"failed"`, `canRetry: true`, and a pending record after a refused
attempt read the same.
- Green (after the fix), in `server/`: `pnpm exec vitest run
src/__tests__/task-interruption-automatic-path.test.ts
src/__tests__/recovery-wake-herd.test.ts
src/__tests__/task-interruption-query.test.ts
src/services/task-interruption-presentation.test.ts
src/services/fresh-continuation-fallback.test.ts`: 5 files, 202 tests
pass.
- New tests in `task-interruption-automatic-path.test.ts`:
- The staging record shape at 1, 30 and 110 minutes shows `working`,
`canRetry: false`, responsibility `system`.
- A refused delivery attempt keeps `working`.
- Delivery given up (`invalidated`, with the coalesced hold) offers
Retry.
- Delivered: the task follows the new run (queued: `working` on the new
run; the new run fails: Retry on the new run).
- A resolved recovery record with no recorded continuation offers Retry
as before.
- A board-owned hold with no recorded continuation offers Retry as
before.
- The #21200, #21264 and #21320 guard tests in the same file pass
unchanged.
- `node scripts/typecheck-layers.mjs` in `server/` passes.
## Risks
- #21324 limited "working" to 15 minutes so that a continuation that
never gets delivered does not hide behind "working". With this change, a
continuation that stays pending forever shows "working" forever. #21324
also made delivery bounded (an overdue record is read on every pass and
gets one wake per agent per pass), and a separate change is making
delivery prompt. A record that cannot be delivered (task done,
cancelled, or reassigned) is marked `invalidated`, and then Retry shows.
- Person-owned waits still win: approval, reconnect, budget, pause and
project access are checked before this state.
- Presentation only. No schema, migration, or execution change.
- This change supersedes the expectation of the #21324 test "does not
show a task as working when its continuation is pending past the
delivery bound". That test hid the continuing state after 15 minutes.
The state must stay truthful: the system has recorded that it will
continue the task, so the task must not show a failed attempt or a Retry
that can start a second run.
- Follow-up: while a continuation waits, a person has no control. On
staging a continuation can wait a long time behind a deep queue. After
#21348 is on master (a person's Retry folds into the pending
continuation in the admitting transaction, and runs that a person starts
rank ahead of system recovery), the planned follow-up keeps the
continuing state and, after 15 minutes, also offers a Retry worded as
"move it forward now" that folds into the pending continuation. This
pull request does not add that Retry.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
386dbe76a1
commit
e23549cd9c
3 files changed
+84
-8
No files matched your search
@@ -535,13 +535,17 @@ suite("recovery wake herd", () => {
|
||||
expect(await deliveryOf(healthy.action.id)).toBe("delivered");
|
||||
}, 120_000);
|
||||
|
||||
it("does not show a task as working when its continuation is pending past the delivery bound and nothing runs", async () => {
|
||||
it("shows a task as continuing, never as needing Retry, while its recorded continuation is pending past the delivery bound", async () => {
|
||||
const agentId = await seedCompany(1);
|
||||
const record = await pendingContinuation(agentId, 240);
|
||||
await holdPastBound(record, RECOVERY_CONTINUATION_DELIVERY_BOUND_MS);
|
||||
const [issueRow] = await db.select().from(issues).where(eq(issues.id, record.issue.id));
|
||||
const presentation = await taskInterruptionPresentation(db, issueRow!);
|
||||
expect(presentation?.state).not.toBe("working");
|
||||
expect(await taskInterruptionPresentation(db, issueRow!)).toMatchObject({ state: "working", canRetry: false, responsibility: "system" });
|
||||
|
||||
const wake = fakeWake();
|
||||
await deliverReconciledExecutions(db, wake as never);
|
||||
expect(await deliveryOf(record.action.id)).toBe("delivered");
|
||||
expect(wake).toHaveBeenCalledTimes(1);
|
||||
|
||||
const fresh = await pendingContinuation(agentId, 30);
|
||||
const [freshRow] = await db.select().from(issues).where(eq(issues.id, fresh.issue.id));
|
||||
|
||||
@@ -82,6 +82,82 @@ describe.skipIf(!support.supported)("Retry is offered only when no automatic pat
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
|
||||
});
|
||||
|
||||
async function seedRecordedContinuation(minutesAgo: number) {
|
||||
const recordedAt = new Date(Date.now() - minutesAgo * 60_000);
|
||||
const task = await seedSwapEndedRun(new Date(recordedAt.getTime() - 30_000));
|
||||
const freshContinuation = { schema: "paperclip.fresh-continuation.v1", sourceRunId: task.runId, trigger: "held_recovery",
|
||||
failureCode: "native_restart_recovery_blocked", delivery: "recorded", recordedAt: recordedAt.toISOString(),
|
||||
isolation: "remote_epoch", stopProof: "unproven_epoch_abandoned", unverifiedActions: [], createdObjects: [] };
|
||||
const settlement = { schema: "paperclip.fresh-continuation.v1", actorType: "system", actorId: "fresh-continuation",
|
||||
decision: "continue_in_fresh_run", replay: "none", sourceRunId: task.runId, reason: "exact_resumption_unavailable",
|
||||
isolation: "remote_epoch", recordedAt: recordedAt.toISOString() };
|
||||
const reconciliation = { runId: task.runId, providerStopped: true, providerStopProof: "isolated", actionOutcome: "mixed",
|
||||
actorId: "fresh-continuation", recordedAt: recordedAt.toISOString() };
|
||||
const [primary] = await db.insert(issueRecoveryActions).values({ companyId: task.companyId, sourceIssueId: task.issue.id,
|
||||
kind: "active_run_watchdog", status: "resolved", outcome: "restored", ownerType: "system", returnOwnerAgentId: task.agentId,
|
||||
cause: "native_continuation_requires_reconciliation", fingerprint: `native-continuation:${task.runId}`,
|
||||
nextAction: "Paperclip continues the task in a fresh run from its saved work.", resolutionNote: "continued_fresh_from_last_saved",
|
||||
resolvedAt: recordedAt, createdAt: recordedAt, updatedAt: recordedAt,
|
||||
evidence: { runId: task.runId, executionReconciliation: reconciliation, freshContinuationSettlement: settlement,
|
||||
freshContinuation, continuationDelivery: "pending" } }).returning();
|
||||
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: recordedAt, updatedAt: recordedAt,
|
||||
evidence: { ...task.hold.evidence, executionReconciliation: reconciliation, freshContinuationSettlement: settlement,
|
||||
continuationDelivery: "coalesced", coalescedIntoRecoveryActionId: primary!.id } })
|
||||
.where(eq(issueRecoveryActions.id, task.hold.id));
|
||||
return { ...task, primary: primary! };
|
||||
}
|
||||
|
||||
it("shows a task whose recorded continuation is still waiting to start as continuing, however long delivery takes", async () => {
|
||||
for (const minutesAgo of [1, 30, 110]) {
|
||||
const { issue, runId } = await seedRecordedContinuation(minutesAgo);
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "working",
|
||||
canRetry: false, observedRunId: runId });
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps a recorded continuation as continuing after a refused delivery attempt, since delivery tries again", async () => {
|
||||
const { issue, primary } = await seedRecordedContinuation(45);
|
||||
await db.update(issueRecoveryActions).set({ evidence: { ...primary.evidence,
|
||||
continuationDeliveryAttemptedAt: new Date(Date.now() - 60_000).toISOString() } }).where(eq(issueRecoveryActions.id, primary.id));
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "working", canRetry: false });
|
||||
});
|
||||
|
||||
it("offers Retry once the system gives up on a recorded continuation", async () => {
|
||||
const { issue, primary, hold } = await seedRecordedContinuation(45);
|
||||
await db.update(issueRecoveryActions).set({ evidence: { ...primary.evidence, continuationDelivery: "invalidated" } })
|
||||
.where(eq(issueRecoveryActions.id, primary.id));
|
||||
await db.update(issueRecoveryActions).set({ evidence: { ...hold.evidence, continuationDelivery: "invalidated",
|
||||
coalescedIntoRecoveryActionId: primary.id } }).where(eq(issueRecoveryActions.id, hold.id));
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "failed", canRetry: true });
|
||||
});
|
||||
|
||||
it("follows the new run once a recorded continuation is delivered", async () => {
|
||||
const { companyId, agentId, issue, primary } = await seedRecordedContinuation(45);
|
||||
const nextRunId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
|
||||
status: "queued", runtimeMode: "native", retryOfRunId: primary.evidence.runId as string });
|
||||
await db.update(issueRecoveryActions).set({ evidence: { ...primary.evidence, continuationDelivery: "delivered",
|
||||
continuationRunId: nextRunId } }).where(eq(issueRecoveryActions.id, primary.id));
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "working", canRetry: false, observedRunId: nextRunId });
|
||||
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "adapter_failed", finishedAt: new Date(Date.now() - 10 * 60_000) })
|
||||
.where(eq(heartbeatRuns.id, nextRunId));
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true, observedRunId: nextRunId });
|
||||
});
|
||||
|
||||
it("still offers Retry for a resolved recovery record that recorded no continuation", async () => {
|
||||
const { issue, hold } = await seedSwapEndedRun(new Date(Date.now() - 45 * 60_000));
|
||||
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: new Date(Date.now() - 40 * 60_000),
|
||||
evidence: { ...hold.evidence, freshContinuationSettlement: { schema: "paperclip.fresh-continuation.v1", decision: "continue_in_fresh_run" } } })
|
||||
.where(eq(issueRecoveryActions.id, hold.id));
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "failed", canRetry: true });
|
||||
});
|
||||
|
||||
it("still offers Retry when the board owns the hold and no continuation is recorded", async () => {
|
||||
const { issue, hold } = await seedSwapEndedRun(new Date(Date.now() - 45 * 60_000));
|
||||
await db.update(issueRecoveryActions).set({ ownerType: "board" }).where(eq(issueRecoveryActions.id, hold.id));
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
|
||||
});
|
||||
|
||||
it("shows a healthy task as working while the message queued during its succeeded run is delivered", async () => {
|
||||
const { companyId, agentId, issue } = await seedTask();
|
||||
const runId = randomUUID(), finishedAt = new Date(Date.now() - 3_000);
|
||||
|
||||
@@ -11,7 +11,6 @@ import { EXECUTION_RECONCILIATION_CAUSES, HEARTBEAT_RUN_STATUSES, readProviderFa
|
||||
import { captureTaskInterruptionFailure } from "../sentry.js";
|
||||
import { isAiAuthenticationBlocked } from "./ai-auth-failure.js";
|
||||
import { readSystemRecovery } from "./system-recovery-state.js";
|
||||
import { RECOVERY_CONTINUATION_DELIVERY_BOUND_MS, recoveryContinuationHeldSince } from "./recovery-continuation-pacing.js";
|
||||
import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js";
|
||||
import { authorizationService } from "./authorization.js";
|
||||
import { taskWorkspaceProjectRequirements } from "./task-workspace-access.js";
|
||||
@@ -592,8 +591,6 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
runId: sql<string>`${issueRecoveryActions.evidence}->>'runId'`,
|
||||
delivery: sql<string | null>`${issueRecoveryActions.evidence}->>'continuationDelivery'`,
|
||||
recorded: sql<boolean>`${issueRecoveryActions.evidence} ? 'freshContinuation'`,
|
||||
evidence: sql<Record<string, unknown>>`jsonb_build_object('freshContinuation', ${issueRecoveryActions.evidence}->'freshContinuation')`,
|
||||
resolvedAt: issueRecoveryActions.resolvedAt, updatedAt: issueRecoveryActions.updatedAt,
|
||||
checkKind: sql<string | null>`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,kind}'`,
|
||||
nextCheckAt: sql<string | null>`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}'` })
|
||||
.from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, ids),
|
||||
@@ -603,8 +600,7 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
const rows = freshRows.filter(row => row.issueId === issueId && row.runId === runId);
|
||||
// Only the primary record (it carries `freshContinuation`) says whether a
|
||||
// continuation is still on its way; holds coalesced into it never do.
|
||||
if (rows.some(row => row.recorded && (row.delivery === "not_requested" || (row.delivery === "pending"
|
||||
&& Date.now() - recoveryContinuationHeldSince(row) < RECOVERY_CONTINUATION_DELIVERY_BOUND_MS)))) return { state: "continuing" };
|
||||
if (rows.some(row => row.recorded && ["not_requested", "pending"].includes(row.delivery ?? ""))) return { state: "continuing" };
|
||||
const scheduled = rows.filter(row => ["exact_recovery_pending", "backoff"].includes(row.checkKind ?? "") && row.nextCheckAt)
|
||||
.map(row => Number(row.nextCheckAt)).sort((a, b) => a - b)[0];
|
||||
return scheduled ? { state: "scheduled", nextCheckAt: scheduled } : undefined;
|
||||
|
||||
Reference in new issue
Block a user