## Thinking Path > - Paperclip manages agents and the tasks they work on. > - Recovery should continue unfinished work when it is eligible. > - An active unanswered question is a persisted reason to wait for a person. > - Treating a later Retry as an answer makes the recovery sweep repeatedly request continuations that admission refuses. > - Provider-limit holds also produce one cancellation toast per task, even though provider work never started. > - This PR aligns recovery with persisted interaction status and presents these waits accurately. ## Linked Issues or Issue Description Refs: #20466. This addresses the question blocker presentation and automatic recovery loop; it preserves the existing interaction-resolution policy. Related work: #19420 schedules provider quota retries, and #21570 handles notices and reply ownership in agent chats. This change concerns assigned-task reconciliation, task/inbox status, and per-run cancellation toasts. **What happened?** The assigned-work sweep ignored pending interactions whenever a newer run carried `explicitUserContinuation`. Continuation admission still refused the pending question, so repeated sweeps recorded unsuccessful continuation attempts. The task could meanwhile show an older provider error as its current blocker. Separately, each pre-dispatch provider-limit hold generated a cancellation toast, and the one-hour fallback was described as a known reset and promised start time. **Expected behavior** Wait quietly while the question remains active under the shared interaction eligibility rules, then react to the existing resolution notification. Dismissed or historical ordinary questions remain answerable without blocking work; governed requests keep their gates. Show the current question on task and inbox surfaces. Keep provider failure history, suppress cancellation toasts for expected provider holds, and describe scheduled availability checks without promising the provider will accept work. **Steps to reproduce** 1. Leave an `ask_user_questions` interaction pending with `supersedeOnUserComment: false`. 2. Create a later failed/interrupted explicit Retry and run assigned-work reconciliation repeatedly. 3. Observe that the question remains pending while the sweep attempts fresh continuations. 4. Separately, let an agent with recent provider rate-limit evidence receive automatic task wakes without a reported reset time. ## What Changed - Remove inferred question supersession from both assigned-work fact loading and candidate selection. Use the existing active-interaction predicate consistently in those reads, fresh-continuation admission, and the task projection. Cover dismissed/historical ordinary questions, retained governed gates, repeated checks, idle workers, and the existing resolution notification. - Add the `question` interruption action, load active pending question IDs alongside approvals in one company-scoped query, and show that blocker on tasks and inbox rows. Preserve a supported manual Retry after a failed attempt, suppress duplicate actions for saved input, and keep other current user gates authoritative. This preserves the existing issue-detail SQL statement budget. Historical run failures remain intact. Refresh the shared source SDK input pin. - Suppress cancellation toasts specifically for cancelled `provider_limit_wait` runs. Label provider retry deadlines as availability rechecks, retain relative-time formatting, and record successful scheduling as an informational lifecycle event. - Document the wait contract in the continuation architecture guide. ## Verification - Demonstrated failing regressions before the implementation: continuation loop, task projection, task notice, cancellation toast, provider deadline wording, and deadline formatting. - Initial focused Vitest qualification: 642 tests passed across 14 files, using: ```sh pnpm exec vitest run \ server/src/__tests__/{assigned-work-invariant,assigned-work-recheck-idle,heartbeat-provider-limit-wait,task-interruption-query}.test.ts \ server/src/services/{task-interruption-presentation,execution-projection,fresh-continuation-fallback,issue-thread-interactions}.test.ts \ server/src/services/recovery/assigned-work-invariant.test.ts \ ui/src/{components/ExecutionBlockerNotice.test.tsx,context/LiveUpdatesProvider.test.ts,context/LiveUpdatesProvider.hook.test.tsx,lib/provider-limit-message.test.ts,pages/Inbox.test.tsx} ``` - `pnpm --filter @paperclipai/shared typecheck` — passed. - Review follow-up: 157 tests passed in `assigned-work-invariant`, `task-interruption-query`, and `fresh-continuation-fallback`; 575 passed in `task-interruption-query`, `assigned-work-recheck-idle`, `task-interruption-presentation`, `issue-question-context`, `issue-thread-interactions`, and `explicit-native-continuation` (the query file overlaps). The strengthened final dismissed/historical/governed projection regression also passed separately. - Final query-consolidation verification: 157 tests passed across five files, including the unchanged 64-statement issue-detail census and mixed approval/question batches: ```sh pnpm exec vitest run \ server/src/__tests__/{issue-detail-statements,task-interruption-query,assigned-work-invariant,assigned-work-recheck-idle}.test.ts \ server/src/services/task-interruption-presentation.test.ts ``` - `pnpm --filter @paperclipai/server typecheck` — passed, including test layers. - Manual-recovery follow-up: 204 tests passed across seven projection/recovery/notice suites, and all 104 Inbox tests passed. The lost-runner integration still starts exactly one run from a manual Retry and leaves the question unanswered. The saved-decision census verifies one decision batch query for a 41-task list, with at most three statements referencing interactions; the overall 64-statement issue-detail limit remains unchanged. - `pnpm --filter @paperclipai/ui typecheck` — passed. - `pnpm --filter @paperclipai/ui build` — passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` — passed. - `pnpm verify` — exit 2: broader CI is required for changed global inputs. Focused tests and package checks supply local evidence; broad CI remains required. - `git diff --check` and a local secret/instance-data scan — passed. - Final CI on `1c16fe7c0d8`: [trusted CI workflow](https://github.com/paperclipai/paperclip/actions/runs/44650155842) passed, including all server shards, native-server verification, build, canary, e2e, release registry, and serialized suites. All PR checks are complete, with no failures; the branch is conflict-free. - Greptile reviewed the latest revision, `1c16fe7c0d8`, at 5/5 with no actionable findings or unresolved review threads. The focused tests cover genuine provider failures, live work and terminal tasks, unchanged task locks/retry delivery, preservation of historical failures, and question resolution waking the affected task while other active pending questions remain quiet. Review regressions also cover dismissed/historical ordinary questions retaining Retry and governed requests retaining their gates. No live task was retried or answered for validation. ## Risks - An active pending question now consistently blocks automatic assigned-work recovery. Shared interaction eligibility determines when ordinary conversation input becomes deferred or historical; governed requests retain their authority. Automatic recovery no longer infers that a newer Retry resolved an active question. - Manual Retry keeps its existing admission policy and recovery checks. The question notice preserves that supported action after a failed attempt without changing approval or credential authority; pending saved input prevents duplicate actions. - The shared interruption action gains `question`; the task and inbox consumers are updated together. The pending-question query returns only task IDs and uses the existing company/task/status index. - Provider cooldown policy, quotas, credentials, routing, cancellation history, and durable retry timing are unchanged. The actual provider can still return a rate limit at the recheck. - No migration is required. Existing large owner modules receive small localized changes; no code file crosses 1,000 lines. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, xhigh reasoning effort, with code execution and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
7.8 KiB
Durable continuation scheduling
Paperclip does not keep an agent process alive between turns. A heartbeat run is finite: it starts, performs work, records a terminal result, and exits. If work must continue later, Paperclip represents that intent in database state and creates another heartbeat run when the continuation becomes eligible.
“Durable continuation scheduler” is a useful umbrella term, but it is not the name of one class or queue. Two related mechanisms provide the behavior:
- Explicit continuation effects are written by native status arbitration.
- Stranded-issue reconciliation is a startup and periodic safety net for an assigned open issue that has no live execution or durable wait path.
The second mechanism produced the repeated issue_continuation_needed runs on
DOT-2.
When it runs
The heartbeat scheduler is enabled unless
HEARTBEAT_SCHEDULER_ENABLED=false. Its interval is configured with
HEARTBEAT_SCHEDULER_INTERVAL_MS and defaults to 30,000 ms. The configured
value is clamped to a minimum of 10,000 ms.
Continuation recovery runs:
- once during server startup, after orphaned runs are reaped, due retries are promoted, and already-queued work is resumed; and
- on every heartbeat scheduler tick, after the same orphan/retry/queue cleanup.
The periodic tick calls reconcileStrandedAssignedIssues(). Consequently, a
new recovery run normally appears within one scheduler interval. The interval
is polling cadence, not a promise that every continuation waits exactly that
long.
Explicit native continuation effects do not need to wait for this scan. The
native status-decision committer writes an idempotent agent_wakeup_requests
row directly. The normal queued-run machinery then claims it.
What is durable
The system reconstructs intent from persisted control-plane records rather than an in-memory timer owned by an agent:
- the issue status and assignee;
- the issue execution lock/run identity;
- heartbeat run status, context, retry ancestry, and terminal timestamps;
agent_wakeup_requestsrows;- native status decisions and their materialized effects;
- pending interactions, approvals, monitors, blockers, and execution stages;
- scheduled retry timestamps and recovery actions.
Because these records survive a process restart, startup reconciliation can resume queued work or repair an issue whose previous execution disappeared.
Explicit continuations
A native result can report yielded with a continuation containing:
- a kind:
same_agent,retry,delegated_issue,response_wake, ormonitor; - a summary; and
- an idempotency key.
The native status arbiter keeps the issue in_progress and emits an
enqueue_continuation effect. The status-decision committer materializes that
effect as an idempotent wake request. A monitor continuation uses
monitor_due; other continuation kinds use issue_status_changed at this
boundary.
This is intentional continuation: the result explicitly says more work is needed and leaves a persisted execution path.
Stranded-issue reconciliation
reconcileStrandedAssignedIssues() scans agent-owned issues in todo,
in_progress, and relevant in_review states. Before creating work, it checks
for reasons not to wake the agent, including:
- an active or queued execution path already exists;
- a pending interaction or durable wait path exists;
- the issue or its tree is paused;
- a provider-quota monitor is pending;
- the run was explicitly cancelled by an operator;
- the assigned agent is not invokable;
- invocation budget or retry/backoff policy prevents a run; or
- recovery has failed enough times that the issue should be escalated instead.
For an eligible in_progress issue with no live path, it creates an automated
wake/run with:
wakeReason: issue_continuation_needed
retryReason: issue_continuation_needed
source: issue.continuation_recovery
This is a liveness repair, not evidence that the previous model requested another turn. Its invariant is: an assigned open issue should have either a live execution path, a durable reason to wait, or a visible terminal/blocking disposition.
Active pending interactions remain a reason to wait under the shared interaction eligibility rules. A newer Retry alone does not answer a question. Dismissed or historical ordinary questions remain answerable conversation data, without blocking current work; governed requests retain their gates. Assigned-work reconciliation leaves active question waits quiet, and the existing interaction-resolution notification makes them eligible for another check. The task and inbox show an active question as the current blocker, while older provider failures remain in run history. After a failed attempt, the question notice preserves an available manual Retry; automatic recovery still waits. Existing saved input prevents a duplicate Retry, and other current user gates retain their precedence.
A provider-limit hold before provider work is scheduling, so it does not send a cancellation toast for every held task. Its durable retry rechecks provider availability. When the provider reports no reset time, the fallback deadline is a recheck time rather than a promised reset or work-start time.
Why DOT-2 looped
DOT-2's runner repeatedly produced a successful result that reported done.
The native evidence classifier did not accept the model's evidence references,
so status arbitration preserved in_progress and created another continuation
path. After each run exited, the periodic reconciler saw:
assigned + in_progress + successful terminal run + no live/wait path
It therefore queued issue_continuation_needed again on the next approximately
30-second tick. The new run received the original task title because the native
model envelope also omitted the wake-comment text, so it repeated the same
answer.
The fix has two parts:
- native model input now contains the redacted server-authored task prompt, including the current wake comment; and
- ordinary low-risk issue completion can accept a schema-valid completion claim, while tools, governed effects, interactions, and approvals keep their independent authorization gates.
Thus a completed one-shot task becomes done; it is no longer a candidate for
stranded-issue reconciliation.
How to diagnose a suspected loop
Inspect the latest runs for the issue and compare these fields:
invocationSource— recovery-created runs useautomation;contextSnapshot.wakeReason;contextSnapshot.retryReason;retryOfRunId;- terminal run status and
resultJson.authoritativeDecision; resultJson.issueStatusAfter;- issue
status,executionRunId, and pending interactions/monitors; - lifecycle events explaining enqueue, suppression, escalation, or cleanup.
Repeated successful runs with wakeReason: issue_continuation_needed, an
authoritative decision of in_progress, and no durable wait path usually mean
the result/disposition contract is failing to converge. Fix the status or
continuation decision; increasing the polling interval only hides the bug.
Primary implementation locations
server/src/index.ts— startup recovery and periodic heartbeat scheduler.server/src/config.ts— scheduler enable flag and interval.server/src/services/recovery/service.ts—reconcileStrandedAssignedIssues()and recovery wake creation.server/src/services/issue-rewake-throttle.ts— repeated state-wake throttling and progress detection.server/src/services/native-runtime/status-arbiter.ts— native terminal disposition and explicit continuation decisions.server/src/services/native-runtime/status-decision-committer.ts— durable, idempotent materialization of native continuation effects.server/src/services/heartbeat.ts— run lifecycle, immediate recovery, queue promotion, and wake execution.