Files
PaperClipAI/doc/architecture/durable-continuation-scheduler.md
T
Devin FoleyandPaperclip c41145e72a fix: wait on pending questions and quiet provider holds (#22363)
## Thinking Path

> - Paperclip manages agents and the tasks they work on.
> - Recovery should continue unfinished work when it is eligible.
> - An active unanswered question is a persisted reason to wait for a
person.
> - Treating a later Retry as an answer makes the recovery sweep
repeatedly request continuations that admission refuses.
> - Provider-limit holds also produce one cancellation toast per task,
even though provider work never started.
> - This PR aligns recovery with persisted interaction status and
presents these waits accurately.

## Linked Issues or Issue Description

Refs: #20466. This addresses the question blocker presentation and
automatic recovery loop; it preserves the existing
interaction-resolution policy.

Related work: #19420 schedules provider quota retries, and #21570
handles notices and reply ownership in agent chats. This change concerns
assigned-task reconciliation, task/inbox status, and per-run
cancellation toasts.

**What happened?**

The assigned-work sweep ignored pending interactions whenever a newer
run carried `explicitUserContinuation`. Continuation admission still
refused the pending question, so repeated sweeps recorded unsuccessful
continuation attempts. The task could meanwhile show an older provider
error as its current blocker. Separately, each pre-dispatch
provider-limit hold generated a cancellation toast, and the one-hour
fallback was described as a known reset and promised start time.

**Expected behavior**

Wait quietly while the question remains active under the shared
interaction eligibility rules, then react to the existing resolution
notification. Dismissed or historical ordinary questions remain
answerable without blocking work; governed requests keep their gates.
Show the current question on task and inbox surfaces. Keep provider
failure history, suppress cancellation toasts for expected provider
holds, and describe scheduled availability checks without promising the
provider will accept work.

**Steps to reproduce**

1. Leave an `ask_user_questions` interaction pending with
`supersedeOnUserComment: false`.
2. Create a later failed/interrupted explicit Retry and run
assigned-work reconciliation repeatedly.
3. Observe that the question remains pending while the sweep attempts
fresh continuations.
4. Separately, let an agent with recent provider rate-limit evidence
receive automatic task wakes without a reported reset time.

## What Changed

- Remove inferred question supersession from both assigned-work fact
loading and candidate selection. Use the existing active-interaction
predicate consistently in those reads, fresh-continuation admission, and
the task projection. Cover dismissed/historical ordinary questions,
retained governed gates, repeated checks, idle workers, and the existing
resolution notification.
- Add the `question` interruption action, load active pending question
IDs alongside approvals in one company-scoped query, and show that
blocker on tasks and inbox rows. Preserve a supported manual Retry after
a failed attempt, suppress duplicate actions for saved input, and keep
other current user gates authoritative. This preserves the existing
issue-detail SQL statement budget. Historical run failures remain
intact. Refresh the shared source SDK input pin.
- Suppress cancellation toasts specifically for cancelled
`provider_limit_wait` runs. Label provider retry deadlines as
availability rechecks, retain relative-time formatting, and record
successful scheduling as an informational lifecycle event.
- Document the wait contract in the continuation architecture guide.

## Verification

- Demonstrated failing regressions before the implementation:
continuation loop, task projection, task notice, cancellation toast,
provider deadline wording, and deadline formatting.
- Initial focused Vitest qualification: 642 tests passed across 14
files, using:

  ```sh
  pnpm exec vitest run \

server/src/__tests__/{assigned-work-invariant,assigned-work-recheck-idle,heartbeat-provider-limit-wait,task-interruption-query}.test.ts
\

server/src/services/{task-interruption-presentation,execution-projection,fresh-continuation-fallback,issue-thread-interactions}.test.ts
\
    server/src/services/recovery/assigned-work-invariant.test.ts \

ui/src/{components/ExecutionBlockerNotice.test.tsx,context/LiveUpdatesProvider.test.ts,context/LiveUpdatesProvider.hook.test.tsx,lib/provider-limit-message.test.ts,pages/Inbox.test.tsx}
  ```
- `pnpm --filter @paperclipai/shared typecheck` — passed.
- Review follow-up: 157 tests passed in `assigned-work-invariant`,
`task-interruption-query`, and `fresh-continuation-fallback`; 575 passed
in `task-interruption-query`, `assigned-work-recheck-idle`,
`task-interruption-presentation`, `issue-question-context`,
`issue-thread-interactions`, and `explicit-native-continuation` (the
query file overlaps). The strengthened final
dismissed/historical/governed projection regression also passed
separately.
- Final query-consolidation verification: 157 tests passed across five
files, including the unchanged 64-statement issue-detail census and
mixed approval/question batches:

  ```sh
  pnpm exec vitest run \

server/src/__tests__/{issue-detail-statements,task-interruption-query,assigned-work-invariant,assigned-work-recheck-idle}.test.ts
\
    server/src/services/task-interruption-presentation.test.ts
  ```
- `pnpm --filter @paperclipai/server typecheck` — passed, including test
layers.
- Manual-recovery follow-up: 204 tests passed across seven
projection/recovery/notice suites, and all 104 Inbox tests passed. The
lost-runner integration still starts exactly one run from a manual Retry
and leaves the question unanswered. The saved-decision census verifies
one decision batch query for a 41-task list, with at most three
statements referencing interactions; the overall 64-statement
issue-detail limit remains unchanged.
- `pnpm --filter @paperclipai/ui typecheck` — passed.
- `pnpm --filter @paperclipai/ui build` — passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse
HEAD) --check` — passed.
- `pnpm verify` — exit 2: broader CI is required for changed global
inputs. Focused tests and package checks supply local evidence; broad CI
remains required.
- `git diff --check` and a local secret/instance-data scan — passed.
- Final CI on `1c16fe7c0d8`: [trusted CI
workflow](https://github.com/paperclipai/paperclip/actions/runs/44650155842)
passed, including all server shards, native-server verification, build,
canary, e2e, release registry, and serialized suites. All PR checks are
complete, with no failures; the branch is conflict-free.
- Greptile reviewed the latest revision, `1c16fe7c0d8`, at 5/5 with no
actionable findings or unresolved review threads.

The focused tests cover genuine provider failures, live work and
terminal tasks, unchanged task locks/retry delivery, preservation of
historical failures, and question resolution waking the affected task
while other active pending questions remain quiet. Review regressions
also cover dismissed/historical ordinary questions retaining Retry and
governed requests retaining their gates. No live task was retried or
answered for validation.

## Risks

- An active pending question now consistently blocks automatic
assigned-work recovery. Shared interaction eligibility determines when
ordinary conversation input becomes deferred or historical; governed
requests retain their authority. Automatic recovery no longer infers
that a newer Retry resolved an active question.
- Manual Retry keeps its existing admission policy and recovery checks.
The question notice preserves that supported action after a failed
attempt without changing approval or credential authority; pending saved
input prevents duplicate actions.
- The shared interruption action gains `question`; the task and inbox
consumers are updated together. The pending-question query returns only
task IDs and uses the existing company/task/status index.
- Provider cooldown policy, quotas, credentials, routing, cancellation
history, and durable retry timing are unchanged. The actual provider can
still return a rate limit at the recheck.
- No migration is required. Existing large owner modules receive small
localized changes; no code file crosses 1,000 lines.

## Model Used

OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, xhigh
reasoning effort, with code execution and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-19 19:54:15 -08:00

7.8 KiB

Durable continuation scheduling

Paperclip does not keep an agent process alive between turns. A heartbeat run is finite: it starts, performs work, records a terminal result, and exits. If work must continue later, Paperclip represents that intent in database state and creates another heartbeat run when the continuation becomes eligible.

“Durable continuation scheduler” is a useful umbrella term, but it is not the name of one class or queue. Two related mechanisms provide the behavior:

  1. Explicit continuation effects are written by native status arbitration.
  2. Stranded-issue reconciliation is a startup and periodic safety net for an assigned open issue that has no live execution or durable wait path.

The second mechanism produced the repeated issue_continuation_needed runs on DOT-2.

When it runs

The heartbeat scheduler is enabled unless HEARTBEAT_SCHEDULER_ENABLED=false. Its interval is configured with HEARTBEAT_SCHEDULER_INTERVAL_MS and defaults to 30,000 ms. The configured value is clamped to a minimum of 10,000 ms.

Continuation recovery runs:

  • once during server startup, after orphaned runs are reaped, due retries are promoted, and already-queued work is resumed; and
  • on every heartbeat scheduler tick, after the same orphan/retry/queue cleanup.

The periodic tick calls reconcileStrandedAssignedIssues(). Consequently, a new recovery run normally appears within one scheduler interval. The interval is polling cadence, not a promise that every continuation waits exactly that long.

Explicit native continuation effects do not need to wait for this scan. The native status-decision committer writes an idempotent agent_wakeup_requests row directly. The normal queued-run machinery then claims it.

What is durable

The system reconstructs intent from persisted control-plane records rather than an in-memory timer owned by an agent:

  • the issue status and assignee;
  • the issue execution lock/run identity;
  • heartbeat run status, context, retry ancestry, and terminal timestamps;
  • agent_wakeup_requests rows;
  • native status decisions and their materialized effects;
  • pending interactions, approvals, monitors, blockers, and execution stages;
  • scheduled retry timestamps and recovery actions.

Because these records survive a process restart, startup reconciliation can resume queued work or repair an issue whose previous execution disappeared.

Explicit continuations

A native result can report yielded with a continuation containing:

  • a kind: same_agent, retry, delegated_issue, response_wake, or monitor;
  • a summary; and
  • an idempotency key.

The native status arbiter keeps the issue in_progress and emits an enqueue_continuation effect. The status-decision committer materializes that effect as an idempotent wake request. A monitor continuation uses monitor_due; other continuation kinds use issue_status_changed at this boundary.

This is intentional continuation: the result explicitly says more work is needed and leaves a persisted execution path.

Stranded-issue reconciliation

reconcileStrandedAssignedIssues() scans agent-owned issues in todo, in_progress, and relevant in_review states. Before creating work, it checks for reasons not to wake the agent, including:

  • an active or queued execution path already exists;
  • a pending interaction or durable wait path exists;
  • the issue or its tree is paused;
  • a provider-quota monitor is pending;
  • the run was explicitly cancelled by an operator;
  • the assigned agent is not invokable;
  • invocation budget or retry/backoff policy prevents a run; or
  • recovery has failed enough times that the issue should be escalated instead.

For an eligible in_progress issue with no live path, it creates an automated wake/run with:

wakeReason:  issue_continuation_needed
retryReason: issue_continuation_needed
source:      issue.continuation_recovery

This is a liveness repair, not evidence that the previous model requested another turn. Its invariant is: an assigned open issue should have either a live execution path, a durable reason to wait, or a visible terminal/blocking disposition.

Active pending interactions remain a reason to wait under the shared interaction eligibility rules. A newer Retry alone does not answer a question. Dismissed or historical ordinary questions remain answerable conversation data, without blocking current work; governed requests retain their gates. Assigned-work reconciliation leaves active question waits quiet, and the existing interaction-resolution notification makes them eligible for another check. The task and inbox show an active question as the current blocker, while older provider failures remain in run history. After a failed attempt, the question notice preserves an available manual Retry; automatic recovery still waits. Existing saved input prevents a duplicate Retry, and other current user gates retain their precedence.

A provider-limit hold before provider work is scheduling, so it does not send a cancellation toast for every held task. Its durable retry rechecks provider availability. When the provider reports no reset time, the fallback deadline is a recheck time rather than a promised reset or work-start time.

Why DOT-2 looped

DOT-2's runner repeatedly produced a successful result that reported done. The native evidence classifier did not accept the model's evidence references, so status arbitration preserved in_progress and created another continuation path. After each run exited, the periodic reconciler saw:

assigned + in_progress + successful terminal run + no live/wait path

It therefore queued issue_continuation_needed again on the next approximately 30-second tick. The new run received the original task title because the native model envelope also omitted the wake-comment text, so it repeated the same answer.

The fix has two parts:

  • native model input now contains the redacted server-authored task prompt, including the current wake comment; and
  • ordinary low-risk issue completion can accept a schema-valid completion claim, while tools, governed effects, interactions, and approvals keep their independent authorization gates.

Thus a completed one-shot task becomes done; it is no longer a candidate for stranded-issue reconciliation.

How to diagnose a suspected loop

Inspect the latest runs for the issue and compare these fields:

  • invocationSource — recovery-created runs use automation;
  • contextSnapshot.wakeReason;
  • contextSnapshot.retryReason;
  • retryOfRunId;
  • terminal run status and resultJson.authoritativeDecision;
  • resultJson.issueStatusAfter;
  • issue status, executionRunId, and pending interactions/monitors;
  • lifecycle events explaining enqueue, suppression, escalation, or cleanup.

Repeated successful runs with wakeReason: issue_continuation_needed, an authoritative decision of in_progress, and no durable wait path usually mean the result/disposition contract is failing to converge. Fix the status or continuation decision; increasing the polling interval only hides the bug.

Primary implementation locations

  • server/src/index.ts — startup recovery and periodic heartbeat scheduler.
  • server/src/config.ts — scheduler enable flag and interval.
  • server/src/services/recovery/service.ts — reconcileStrandedAssignedIssues() and recovery wake creation.
  • server/src/services/issue-rewake-throttle.ts — repeated state-wake throttling and progress detection.
  • server/src/services/native-runtime/status-arbiter.ts — native terminal disposition and explicit continuation decisions.
  • server/src/services/native-runtime/status-decision-committer.ts — durable, idempotent materialization of native continuation effects.
  • server/src/services/heartbeat.ts — run lifecycle, immediate recovery, queue promotion, and wake execution.