Commit Graph
53 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 e82c4f0a31 fix(runner): prove a warm Runner's suspension on a slow sandbox host so the next run does not fail (#21472)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**

This PR extends the failed-close retry that #21442 (merged) added. It is
based on `master`, which contains #21442.

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote sandbox is reused by the next run of the same task.
Before the next run starts, the previous run's Runner must be retired:
suspended, with durable proof.
> - An idle warm close had 10 s for everything. On a loaded sandbox host
one liveness probe or one read of the durable state used that up. The
suspend was never sent, or it completed but was never proven.
> - The next run then replayed that cached close failure, failed before
any provider work, and the task showed a failed run with Retry
> - For a follow-up on a task blocked on its subtasks, the automatic
continuation was refused and the Retry stayed until another wake
> - This pull request bounds each probe, gives the suspend proof its own
read window, lets a later close prove the suspension again, and lets
that follow-up continue by itself
> - The benefit is that a slow host no longer turns a completed
suspension into a failed task, and the person sees no Retry for it

## Linked Issues or Issue Description

**What happened?**

Local release-swap cells on Daytona (G5c and G6c, 2026-11-13) each
showed one Retry banner: "previous run's session could not be retired
(runner did not durably suspend before checkpoint)". G6c had no fault
injection, so this is a real class.

- In the G6c case, the Runner completed its suspend and exited. The
server's proof read (a sandbox exec under load) ran out of the remaining
0.6 s of the close grace, so the suspension was "not proven".
- In the G5c case (an adopted Runner), the liveness probe
`adoptedRunner.isAlive()` took about 220 s and used the whole close
deadline. The suspend was never queued. Five adopted sessions failed the
same way in the same second.
- The successor run's retirement called `transport.close()`, which
returned the cached rejection of the predecessor's failed close. It did
not send a new suspend and did not read the state again.
- The successor failed before provider work. Its fresh-sandbox
continuation (`interrupted_preparation`) was refused as "blocked after
recovery", because the parent task was blocked on its subtasks before
the follow-up started. The task showed a failed run with Retry for 27 s,
until a blocker-stall wake.

**Expected behavior**

A completed suspend is proven, even on a slow host. A follow-up that
fails before any provider work on a task that was already blocked
continues by itself, and the task shows as working while that
continuation is queued.

**Steps to reproduce**

1. Close an idle warm Runner whose durable state read takes longer than
the close grace, or whose liveness probe stalls.
2. On master the close fails and the next close replays the same
failure.
3. The new tests in `runnerd-codex-transport-close-proof.test.ts` and
`heartbeat-native-predispatch-retry.test.ts` do this.

**Paperclip version or commit**

`master` at c6b497d831 (contains #21442).

## What Changed

- `runnerd-codex-transport.ts` (controller-side TypeScript):
- Each liveness probe in the close is bounded. A probe that does not
answer counts as possibly alive, never as gone, and the suspend goes
over the live connection. A connected Runner's probe gets a 1 s slice,
so drain and suspend keep their budget.
- Once the suspend command's completion is observed, what the
durable-state read shows decides how long the proof may take:

    | Read after the completion | Proof window | Maximum close time |
    | --- | --- | --- |
    | Suspended, right Runner | proven at once | unchanged |
| Another run's or Runner's state | ends at the close deadline, as on
master | close grace (10 s; 45 s for Pi) |
| Right Runner, not yet `suspended` | settle window of 3 s from the
first such read (`RUNNER_SUSPENSION_PROOF_SETTLE_MS`, test option
`closeProofSettleMs`) | close grace + 3 s |
| Unreadable or slow read | full proof window of 30 s from the
completion (`RUNNER_SUSPENSION_PROOF_READ_MS`, `closeProofReadMs`) |
close grace + 30 s |

All rows add the existing bounded exit-probe slices. runner-core saves
the `suspended` lifecycle before it delivers the suspend result
(`persist_lifecycle_before_command_delivery` in `durable/runner.rs`), so
a readable non-suspended state after the completion can only be a stale
remote read; the settle window covers a few read attempts and is not a
write-ordering wait.
- The settlement diagnostic names the branch that ended the window
(`proofWindowEnd`: `proven`, `wrong_identity`,
`not_suspended_after_settle`, `unreadable_after_window`,
`suspend_not_completed`).
- A close that failed only because readable proof was missing is retried
by the next close: it reads the durable state again with a fresh bound
and finishes settled. A close whose proof was refuted stays memoized, as
on master. This extends the failed-close retry from #21442.
  - The close failure carries per-step timings.
- `native-runtime/predecessor-retirement.ts`,
`heartbeat/run-completion.ts`: the predecessor-retirement fallback
records the task's status when the run failed.
- `modules/run-dispatch/domain/policy.ts`,
`modules/run-dispatch/adapters/postgres.ts`: a continuation of a run
that did no provider work, on a task that was already blocked then, is
not treated as blocked after recovery. Its dependency check follows the
wake it continues. The continuation keeps a person's comment wake.
- `task-interruption-presentation.ts`: while that run's failure handling
queues the continuation (up to 30 s), the task shows as working with no
Retry. The window ends when the continuation is scheduled, when the run
records `continuationScheduling: "refused"` (written by
`run-completion.ts` when the fallback does not schedule it), or after 30
s. The fallback record alone no longer ends it.

## Verification

Red on master, green on this branch:

- `runnerd-codex-transport-close-proof.test.ts`
- `proves a completed suspend on its own read window when the durable
state read outlasts the close grace`
- `suspends five adopted idle Runners whose liveness probes stall
together, instead of spending the close on the probe`
- `heartbeat-native-predispatch-retry.test.ts`
- `continues a follow-up on a blocked task whose predecessor could not
be retired`
- `does not continue a task that was blocked only after its run failed`
- `task-interruption-presentation.test.ts`
- `shows the task as working while its fresh-sandbox continuation is
being queued`
- `keeps the task working after the fallback is recorded while its
continuation is still being scheduled` (review fix)
  - `offers Retry when the continuation was refused or never queued`
- `runnerd-codex-transport.test.ts` (review fixes)
- `gives a suspend that completes near the close deadline its whole
proof window from the completion`
- `bounds the proof window from the observed completion, so a proof read
that never answers still ends`
- `proves a suspension whose suspended state appears 1.5 s after the
command completed`
- `fails a suspension whose state stays ready after the settle window,
not after the whole proof window`
- `fails at the close deadline when the durable state belongs to another
run`
- The three existing tests in
`runnerd-codex-transport-suspension.test.ts` that expect a conclusive
negative to fail within the close grace pass unchanged.

Local runs on this branch: `tsc --noEmit` is clean for `server` and
`packages/paperclip-runner`. Passing on this branch:
`runnerd-codex-transport-suspension`, `-close-proof`, `-adoption`,
`-remote-retention`, `runnerd-detach-durability` (all files), the
proof-window cases of `runnerd-codex-transport.test.ts`,
`task-interruption-presentation.test.ts` (72), `run-dispatch` policy and
Postgres adapter tests and `heartbeat-native-predispatch-retry.test.ts`
(166 together).

Conflict resolved: `task-interruption-presentation.ts` gained
`WAITING_CONTINUATION_MOVE_FORWARD_MS` on master (#21385). Both
constants are kept.

Runner content ID: the Runner change is controller-side TypeScript under
`packages/paperclip-runner/src` only. There is no Rust, `protocol/` or
`generated/` change. Existing Runner images work with this controller.

Daytona: this fix has not run on Daytona at all. The two banners above
are the evidence of the class (G5c and G6c ran on 588c6ba267, without
this fix).

## Risks

- Not proven on Daytona.
- A close can now take up to 30 s longer when the proof read is
unreadable or slow, and up to 3 s longer when the read is stale. This is
on the close path only, not before `turn.start`.
- A probe that does not answer now counts as possibly alive, so the
close sends a suspend instead of treating the Runner as gone.
- Ownership: this PR changes
`server/src/modules/run-dispatch/domain/policy.ts`,
`server/src/modules/run-dispatch/adapters/postgres.ts` and
`server/src/services/heartbeat/run-completion.ts`. CODEOWNERS does not
list these files. Their recent changes come from the restart-recovery
and Retry work (#21348, #21256, #20863, #21302). That area should review
the dispatch-policy change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:32:22 -06:00
DottaandClaude Opus 5.5 3b03ac10e5 fix(retry): answer a refused Retry with its refusal instead of accepted (#21497)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person presses Retry on a stopped task, and the Retry API answers
before dispatch runs
> - The route answered 202 "accepted" when it saved the request.
Admission ran afterwards
> - When admission refused the request for a reason another click cannot
change, the person saw "accepted", no run started, and the task showed
no Retry
> - A Retry that is shown must work, or the answer must say clearly why
it did not start
> - This pull request admits a plain Retry before the route answers
(bounded wait) and answers a refusal as a refusal
> - The benefit is that the API never reports "accepted" for a Retry
that admission refused, and the UI shows the reason

## Linked Issues or Issue Description

Refs #21481 (a fix for one cause of the refusal).

**What happened?**
On a hosted staging instance, two tasks got HTTP 202 `accepted: true`
from `POST /api/issues/:id/retry`. No run was created. The tasks then
showed a failed state with no Retry button and the wait reason
`previous_execution_unverified`.

**Expected behavior**
The Retry answer either reports a Retry that admission took, or it names
the refusal so that the UI can show it. It never reports "accepted" for
a request that admission refused.

**Steps to reproduce**
1. Stop a task's run so that its outcome is unknown (for example, a
recorded provider event in a stopped preparation).
2. Press Retry.
3. See HTTP 202 `accepted: true`, then see that the saved request waits
on `previous_execution_unverified` and no run starts.

**Paperclip version or commit**
`master` at `a48eaaef57`.

## What Changed

- `POST /api/issues/:id/retry` runs admission of a plain Retry (not a
saved-result finalization, not a connected chat retry) before it
answers. It waits at most 5 seconds. If admission takes longer,
admission continues after the answer, and the answer stays "accepted" as
before.
- When admission refused the request for a reason another click cannot
change (`heldRetryOffersRetry` is false), the answer is HTTP 200 with
`accepted: false` and `refused: { reason, message }`. The message is the
same stable text that the task view shows.
- The saved request is kept. If its gate clears later (for example,
saved-work recovery verifies the files), it still starts by itself.
- New `taskRetryRefusal` and `taskRetryWaitMessage` in
`server/src/services/task-interruption-presentation.ts`.
- UI: `taskRetryNotStartedReason` shows `refused.message`; the API type
includes `refused`.
- Tests that asserted 202 for a held Retry now assert the refusal
answer.

## Verification

- Red without the route change, green with it: `cd server && npx vitest
run src/services/legacy-preparation-retry-hold.test.ts` (new test: a
stopped preparation with a recorded provider event; the route answers
200 with `refused.reason: previous_execution_unverified`, and the saved
request stays deferred).
- Red without the hook change, green with it: `cd ui && npx vitest run
src/hooks/useTaskRetry.test.tsx`.
- Updated and passing:
`src/services/workspace-recovery-preparation-gates.test.ts` (48),
`src/services/workspace-recovery-preparation-proof.test.ts` (53),
`src/services/workspace-recovery-preparation-dispatch.test.ts` (25).
- Passing unchanged:
`src/__tests__/issue-scheduled-retry-routes.test.ts` (34),
`src/services/unrecoverable-saved-work-retry.test.ts` (5),
`src/services/workspace-recovery-warm-handoff.test.ts` (44),
`src/__tests__/heartbeat-native-restart-suspension.test.ts` (115),
`src/__tests__/issue-agent-mutation-ownership-routes.test.ts` (229),
`src/__tests__/agent-cross-tenant-authz-routes.test.ts` (18).
- Not run locally: `src/__tests__/chat-channels.integration.test.ts`
(did not finish on the loaded machine). CI covers it.

## Risks

- The Retry answer can take up to 5 seconds longer when admission is
slow (for example, when the agent's start lock is held). Before, the
route answered at once and admitted in the background.
- A refused request can still start later by itself when its gate
clears. The answer then said "not started", but the task view updates
when the run starts.
- API change: a refused Retry is now HTTP 200 with `accepted: false` and
`refused`. The board UI handles it. Other clients that only check
`accepted` see a refusal, which is correct.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:45:03 -06:00
DottaandClaude Opus 5.5 5026093d0f feat(presentation): offer Move it forward now while a continuation is still waiting to start (#21385)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, Paperclip records a fresh continuation
and delivers it later as a new run. Since #21353 the task reads as
continuing (`working`, no Retry) while it waits
> - #21373 added "Move it forward now" (`retryAction: "move_forward"`).
It is offered only after the delivered continuation run waited 15
minutes in the agent's queue
> - Before delivery (`continuationDelivery: "pending"`) the task showed
no button. On hosted staging such continuations waited 30 to 110 minutes
behind a deep queue, and the person had no control
> - #21378 and #21379 made the click safe in that state: a
`move_forward` request with a recorded, undelivered continuation is
accepted and delivers that continuation as the one run, with its
recorded isolation
> - This pull request also offers "Move it forward now" while the
continuation waits for delivery, once it has waited 15 minutes since it
was recorded
> - The benefit is one action, with the same label and copy, for both
waiting states. The click never starts a second run

## Linked Issues or Issue Description

Refs #21353, #21373, #21378, #21379

**What happened?**

A task has a fresh continuation that is recorded but not delivered
(`continuationDelivery: "pending"`). It waits behind a deep agent queue
for 15 minutes or more. The task shows "continuing" with no action.

**Expected behavior**

After 15 minutes the task still reads as continuing (`working`,
responsibility `system`, never `failed`). It also offers "Move it
forward now". The click delivers the recorded continuation as the one
run.

**Steps to reproduce**

1. Record a fresh continuation for a task while its agent is busy. Do
not run the delivery pass.
2. Wait 15 minutes.
3. Read the task presentation. It shows `canRetry: false` and no
`retryAction`.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: the
fresh-continuation read also loads the record status, schema,
`recordedAt` and reconciled run. A `continuing` state can now carry
`pendingSince` when the primary record is `resolved`,
`continuationDelivery` is `pending`, the schema is the
fresh-continuation schema, and the reconciled run is the observed run.
These are the same facts that `undeliveredContinuation` checks at click
time.
- The projection offers `canRetry: true, retryAction: "move_forward"`
(state `working`) when `pendingSince` is 15 minutes or older, no person
Retry for that run already waits, and the run's agent is the current
assignee. Under 15 minutes, nothing changes.
- No button when the system gave up delivery (`invalidated`; the normal
Retry rules apply), when the delivery is `not_requested`, when a person
already asked, or for a held restore (the existing execution-blocker
post-filter removes the offer).
- New integration test
`server/src/services/fresh-continuation-move-forward-offer.test.ts`. It
uses the real click path for the three recorded kinds
(`fresh_isolated_workspace`, `writer_ended`, native `remote_epoch`):
presentation, `requestTaskRetry` with the shown `observedRunId` and
`retryAction`, `resumeTaskRetry`, then the delivery pass.
- Updated guards in `task-interruption-automatic-path.test.ts` and
`recovery-wake-herd.test.ts` (#21353, #21373). Their assertions that a
pending continuation shows no Retry at 30, 45, 110 and 240 minutes now
expect the move-forward offer. The `working` / `system` / never-failed
assertions stay. New cases cover 14 minutes, `not_requested`, a waiting
person Retry and reassignment.
- `doc/execution-semantics.md`: describes the new offer and its races.
- No shared type, validator, UI or API client change. The action and its
UI copy ("Waiting for the agent to finish other work. Paperclip
continues this task by itself." / "Move it forward now") already exist
and do not depend on the state. No runner-e2e SDK public-input pin
covers the changed files.

## Verification

- Red: with the presentation change reverted, 16 tests fail (`expected {
intentId: null, …(7) } to match object { responsibility: 'system', …
retryAction: 'move_forward' }`).
- Green: `npx vitest run
src/services/fresh-continuation-move-forward-offer.test.ts`: 14 of 14
pass. For each kind:
- The click gives exactly one run. It is the recorded continuation
(`recoveryActionId`, `previousRunId`, `retryOfRunId`,
`forceFreshSession`). The fresh isolated workspace is kept
(`freshContinuationIsolation`, `readFreshWorkspaceReadiness`). The
record reads `delivered` with that run. Two later delivery passes and a
repeated resume create no second run, and the offer then ends.
  - The click and two concurrent delivery passes give one run.
- Delivered between the offer and the click: the click is refused
(`accepted: false`, no wake) and the only run is the delivered
continuation.
- The system gave up the delivery between the offer and the click:
`alreadyMovedOn: true`, no run.
- The task moved to another run between the offer and the click:
refused, the only run is that other run.
  - Under 15 minutes: no button.
- Guards: `task-interruption-automatic-path.test.ts` (37/37),
`recovery-wake-herd.test.ts`, `task-interruption-presentation.test.ts`,
`task-interruption-query.test.ts`,
`fresh-continuation-retry-delivery.test.ts`,
`fresh-continuation-fallback.test.ts`, `task-retry.test.ts`: all pass.
Also the saved-work, workspace-recovery, native restart and
closed-session retry suites that read the presentation.
- `npx tsc --noEmit -p .` in `server/`: no errors.

## Risks

Low risk. The change is presentation only. It adds four selected JSON
fields to a read that already exists and one conditional module import.
The click path (`task-retry.ts`, `fresh-continuation-retry-fold.ts`) is
unchanged and already holds the task row lock. A record without
`recordedAt` or with an older schema gets no button, as before. People
now see a button on tasks that showed none after 15 minutes. The button
can only deliver the recorded continuation or do nothing.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:24:12 -06:00
DottaandClaude Opus 5.5 d9b42b50fd feat(presentation): let a person move a waiting continuation forward without starting a second run (#21373)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, Paperclip records a fresh continuation
and continues the task by itself. Since #21353 the task reads as
continuing (`working`, no Retry) for as long as that continuation waits
> - That is truthful, but a person has no control while it waits. On
hosted staging one agent had 21 running and 209 queued runs, and
continuations waited 30 to 110 minutes. Pacing (#21234, #21354) admits a
waiting continuation into the agent's queue within minutes, so most of
that wait is in the queue, behind other work
> - #21353 removed Retry from this state because a Retry started its own
run, which raced the continuation and could give the task two runs
> - A Retry that observes the queued continuation run cannot start a
run: admission sees a queued run and joins the Retry to it (`coalesced`,
`runId` = that run). Under #21234's ranking, a queued run that a person
joined starts ahead of system recovery and ordinary work
> - This pull request keeps the continuing state and, once the queued
continuation has waited 15 minutes, also offers Retry labelled "Move it
forward now"
> - The benefit is that a person can move a long-waiting task to the
front of its agent's queue, and the click can never create a second run

## Linked Issues or Issue Description

Refs #21353, #21348, #21234, #21354, #21264, #21320, #21200

**What happened?**

On hosted staging, tasks with a recorded continuation waited 30 to 110
minutes behind a deep agent queue. The task showed "continuing" with no
action. The person could only wait.

**Expected behavior**

The task still never shows "failed". After a long wait the person can
move the waiting continuation forward. The click must not start a second
run.

## Which states a Retry joins (established before building)

I read #21348's fold (`foldWaitingRetriesIntoContinuation`) and its
tests, and drove the real click path (`requestTaskRetry` followed by
`resumeTaskRetry`, as the route does) in a probe against each record
state:

| Continuation state | What a Retry does today | Offer Retry? |
|---|---|---|
| Recorded, waiting for delivery (`continuationDelivery: "pending"`, no
run yet) | The Retry is admitted at once as its own `task_retry` run,
for every isolation (`fresh_isolated_workspace`, `writer_ended`,
native). The delivery pass then marks the record delivered (superseded).
#21348's fold never runs: it folds only a Retry that is still
`deferred_issue_execution` when the continuation run is created, and
only for `fresh_isolated_workspace`. Nothing keeps the Retry deferred,
so the click races the delivery pass. | **No** (unchanged) |
| Delivered, continuation run `queued` | `requestTaskRetry` sees the
latest run is busy and records the Retry as `coalesced` into that run in
the same transaction. No run is created; `resumeTaskRetry` acts only on
deferred Retries. The run is now ranked as a person's (`joinedByUser` in
`queue.ts`). | **Yes, after 15 minutes** |
| Continuation run `running` | Same join, but the run already started,
so there is nothing to move forward. | No |
| Continuation ended | A Retry gets its own run (the existing failed-run
path). | Existing behaviour |

So the Retry offered here is joined by the existing busy-run admission
in `requestTaskRetry`, not by #21348's fold. Its source run is the
queued continuation itself (the run the presentation names as
`observedRunId`), not the run the continuation was created for.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: for a task
whose latest run is `queued`, created 15 minutes ago or more, and named
as `continuationRunId` by a delivered fresh-continuation record, the
projection returns `state: "working"`, `canRetry: true`, `retryAction:
"move_forward"`, responsibility `system`, `observedRunId` = that queued
run. It does not offer this when a person's wake already joined the run,
when the run's agent is not the assignee, or when a person-owned wait
applies (approval, reconnect, budget, pause and the others are checked
first). Two small reads run only for queued runs older than 15 minutes.
The existing restore-hold check that withdraws `canRetry` also withdraws
`retryAction`.
- `packages/shared/src/types/execution-projection.ts`: new optional
`retryAction?: "move_forward"` on `TaskInterruptionPresentation`. It
only changes the label; `canRetry` still decides whether Retry is
offered.
- `ui/src/components/ExecutionBlockerNotice.tsx`: a `working`
presentation with `retryAction: "move_forward"` shows "Waiting for the
agent to finish other work. Paperclip continues this task by itself."
and a "Move it forward now" button that calls the same task Retry.
Token-only classes; `pnpm check:token-gates` is clean.
- One join-only action, `move_forward`, carried from the offer to
admission (contracts kept in sync):
- `packages/shared/src/validators/issue.ts`: `taskRetrySchema` accepts
`retryAction: "move_forward"`. The route already passes the body to
`requestTaskRetry`.
- `server/src/services/task-retry.ts` (`admitTaskRetry`): right after
the stale check, still under the task row lock and before any other
Retry path, a `move_forward` Retry whose observed run is no longer
`queued` or `running` returns `{ accepted: false, alreadyMovedOn: true,
requestId: null }` and writes nothing. This block was cleared with the
owner of Retry admission. The owner will later extend it so that
"continuation recorded, not yet delivered" also counts as actionable,
after their pending-delivery change lands (about two lines on their
side). Until then this presentation never offers the button in that
state, so the early return cannot misfire from our UI.
- `ui/src/api/issues.ts` and `ui/src/hooks/useTaskRetry.ts`: the click
sends `retryAction: "move_forward"` when the offer has it.
`alreadyMovedOn` shows "The task already moved on, so nothing new was
started. It now shows its current state."
- What a click does at each timing: continuation `queued` → joined to
that run (moves it forward); `running` → joined (nothing more to do);
ended → "already moved on", nothing started.
- A joined click cannot become a run later. The busy path writes it as
`coalesced` with `runId` set to the continuation and `finishedAt` set
(`task-retry.ts:456-461`). Promotion after a run ends
(`heartbeat/run-control.ts:944` `releaseIssueExecutionAndPromote`;
`modules/wake-queue/adapters/postgres.ts:89` `DEFERRED_WAKE_STATUS` and
the selects near `:1358`), `resumeTaskRetry` (`run-control.ts:833-839`)
and `resumeExecutionWaitComments` (`:876`) pick up only
`deferred_issue_execution` wakes.
- `ui/src/pages/Inbox.tsx`: the failed-run row does not show its Retry
for this state. That row sends the failed run as the observed run, which
is no longer the latest run, so its Retry would be refused as stale.
- `doc/execution-semantics.md`: one paragraph on the offer, when it
ends, and why there is none before delivery.

## Verification

- Red (before the product change, new tests only), in `server/`: `pnpm
exec vitest run src/__tests__/task-interruption-automatic-path.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 2 failed, 58 passed. Both
failures are the missing offer (`canRetry: false`, no `retryAction`) for
a continuation queued 15+ minutes.
- Green after merging master 8e41247b65, in `server/`: `pnpm exec vitest
run src/__tests__/recovery-wake-herd.test.ts
src/__tests__/task-interruption-automatic-path.test.ts
src/services/task-retry.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/task-interruption-query.test.ts`: 5 files, 279 tests pass.
UI: `useTaskRetry.test.tsx` and `ExecutionBlockerNotice.test.tsx`: 35
tests pass.
- New tests:
- (b, integration, `recovery-wake-herd.test.ts`) A continuation
delivered by the real delivery pass waits 16 minutes behind older
ordinary work. The task offers "move forward". The click goes through
the real `requestTaskRetry` and `resumeTaskRetry`. Result: the task has
exactly one run after the source run, the continuation; the Retry wake
is `coalesced` with `runId` = the continuation; the offer disappears;
the next delivery pass adds nothing; the real dispatcher
(`resumeQueuedRuns`) starts the continuation ahead of the older work;
after it ends, still only that one run.
- (Greptile P1, red then green) "starts nothing when the continuation
ended between showing Move it forward now and the click": the
continuation ends after the offer is read. Before the admission block
the click was accepted (`accepted: true`, a new deferred Retry that
becomes a turn). Now it returns `alreadyMovedOn`, the task has one run,
and there is no `task_retry` wake.
- "never turns a joined move-forward click into a new run after the
continuation it joined ends": the click joins the queued continuation.
The run is then cancelled through the real `cancelRun` release. Two
rounds of `resumeExecutionWaitComments`, `resumeTaskRetry`, the delivery
pass and `resumeQueuedRuns` follow. The joined wake stays `coalesced` on
the ended run. No run comes from the click, and the task's later runs
equal those of an identical task with no click (the system's own
`issue_assignment_recovery`).
- UI hook (`useTaskRetry.test.tsx`): the click sends `retryAction:
"move_forward"`, and the `alreadyMovedOn` outcome shows the plain
message.
- (a) Under 15 minutes: no Retry. Control in the herd file: without a
click the older work starts first.
- (b, presentation) Queued 15, 45 and 110 minutes: `working`, `canRetry:
true`, `retryAction: "move_forward"`.
- (c) No Retry while the continuation waits for delivery (16, 45, 110
minutes), when it is running, when a person already joined it, for a
queued run that is not a recorded continuation, and a paused agent still
asks the person to resume.
- UI (`ExecutionBlockerNotice.test.tsx`): plain wording with one "Move
it forward now" button that calls Retry; no button for a read-only user;
nothing shown for a plain `working` state.
- (d) The #21200, #21264, #21320 and #21353 guard tests pass unchanged,
including "follows the new run once a recorded continuation is
delivered" (a just-queued continuation shows no Retry).
- `node scripts/typecheck-layers.mjs` in `server/`, `tsc --noEmit` in
`packages/shared`, `pnpm run typecheck` in `ui/`: clean.
- Not related to this change:
`src/services/fresh-continuation-fallback.test.ts` has one failure
("starts the continuation while the unreachable old lease is active", a
deadlock in the `afterEach` truncate) that also fails on master
18fb6ee571 without this change.

## Risks

- Moving a waiting continuation forward lets it start before other
queued work of the same agent. That is the purpose, and it is the
ranking #21234 already gives any run a person joined.
- No Retry is offered while a continuation is still waiting for
delivery. The delivery pacing admits a waiting continuation into the
queue within minutes, and a fresh interruption at once (#21354), so the
offer appears after the continuation reaches the queue. For the recovery
owner (not changed here): offering Retry before delivery would need the
Retry admission to hold or deliver the pending continuation itself under
the task lock, instead of starting its own run beside it.
- Admission changes only for a request that carries `retryAction:
"move_forward"`. Every other Retry is unchanged. No schema, migration or
dispatch change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:03:23 -06:00
DottaandClaude Opus 5.5 7b27fc453f fix(presentation): a provider failure the system is still recovering shows as working, not as needing Retry (#21369)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a provider ends a run's session with a terminal failure
(`native_provider_terminal_failed`), Paperclip recovers by itself: it
checks whether a safe replacement is possible, saves the remote files,
and then continues the task in a fresh run from saved work
> - These steps take 15 to 60 seconds. Until the continuation is
recorded, the task view did not know the system was still deciding
> - In a restart-chaos wave, two tasks showed a failed attempt with a
Retry button for 13 to 27 seconds, and one of them changed state in a
loop: failed, then working, then failed with Retry, then working
> - Product rule: a task that the system will continue by itself never
shows as failed with a Retry button, and the state does not change back
and forth during one recovery
> - This pull request reads the record that says the system is still
deciding (the native coordinator's terminal-failure row) and shows the
task as working during that time, within the existing 2-minute bound
> - The benefit is one steady "working" state from the failure to the
new run, and Retry comes back at once when the system hands the task to
a person

## Linked Issues or Issue Description

Refs #21264, #21353, #21200, #21320

**What happened?**

Restart-chaos wave `w-20261113T092657-s7302` on master d1d71a7755, tasks
t030 and t039 (Claude through ACPX). The provider returned a temporary
server error ("ACP agent reported a terminal service failure. API Error:
503 ..."). The run failed with `native_provider_terminal_failed`. The
system then continued the task by itself
(`issue_recovery_action_restored`), and a new run completed the task. In
between, the task view showed:

- t030: 10:20:18 working → 10:20:32 failed, Retry offered → 10:20:45
working (13.4 s with Retry).
- t039: 10:21:15 failed, no Retry → 10:21:29 working → 10:21:46 failed,
Retry offered → 10:22:12 working (26.8 s from the failure; it changed
state in a loop).

The wave database is not kept, so the record at each step comes from the
samples (`recoveryOwner`, `executionBlocker`, issue status), the server
log, and the code paths that write those records:

1. The run ends. The native executor saves the coordinator row (`phase:
terminal_failure`, `failureCode: native_provider_terminal_failed`, no
`replacementDenied`) and a board-owned hold with cause
`native_provider_terminal_failed`, and blocks the task. The projection
fell through to "failed". At t039 10:21:15, the execution-blocker check
removed Retry because the remote files were not saved yet.
2. Workspace recovery adopts the hold (`ownerType: system`,
`systemRecovery` running). While it runs, the task shows "working" (t039
10:21:29).
3. The files are saved (`systemRecovery` complete). No record now says
"running", the execution-blocker check no longer removes Retry, and the
task shows "failed" with Retry (t039 10:21:46, t030 10:20:32).
4. Safe replacement is denied (`replacementDenied` on the coordinator
row, and the hold is upserted as board-owned). The settle pass then
records a fresh continuation (log: "Unfinished run continues in a fresh
run from saved work", `trigger: recovery_refused`), sets the task back
to `todo`, and the task shows "working" (t039 10:22:10–12).

**Expected behavior**

From the failure until the continuation is recorded, the task shows
"working" with no Retry, without changing back and forth. When recovery
is handed to a person, or nothing is recorded within the bound, Retry is
offered.

**Steps to reproduce**

1. Let a native run fail with `native_provider_terminal_failed` on a
remote workspace.
2. Read the task before the fresh continuation is recorded.
3. The task shows "failed" with Retry.

## What Changed

- `server/src/services/task-interruption-presentation.ts`:
- The run read selects `providerRecoveryAt`: the coordinator row's
update time while the row says the system is still deciding (a failed
native run, `phase: terminal_failure`, `failureCode:
native_provider_terminal_failed`, no result, no lease owner, no
successor run). This matches the rule that `execution-projection.ts`
already uses to show "Checking recovery" for this failure.
- A denied replacement (`replacementDenied`) does not end this time.
After a denial, the settle pass continues the task in a fresh run unless
the failure code is a person's. `native_provider_terminal_failed` is not
a person's code.
- While the time is inside the existing 2-minute bound
(`AUTOMATIC_DISPOSITION_GRACE_MS`), the task shows `working`, `canRetry:
false`, responsibility `system`, with the saved-files step text when
there is one. The bound counts from the latest of: the run end, the
coordinator row update, and the recovery record update. So each recovery
step extends it, and the steps do not make the state change back and
forth.
- Retry is offered at once when recovery was handed to a person: a
fresh-continuation check that decided it cannot continue (any kind other
than `exact_recovery_pending` or `backoff`), `ownerGoneAttestation`,
`explicitUserContinuation`, or an exhausted saved-work recovery.
- No change to recovery decision code, the fallback, or the Retry path.
No new record or marker. The coordinator row already records "still
deciding".

## Verification

- Red (before the fix), in `server/`: `pnpm exec vitest run
src/__tests__/task-interruption-automatic-path.test.ts`: the new t039
sequence test failed at its first step (`state: "failed"`, `canRetry:
true` right after the failure).
- Green (after the fix), in `server/`: `pnpm exec vitest run
src/__tests__/task-interruption-automatic-path.test.ts
src/__tests__/task-interruption-query.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 4 files, 148 tests pass.
- New tests in `task-interruption-automatic-path.test.ts`:
- The t039 sequence (failure, files being saved, files saved,
replacement denied with the hold upserted to board) shows `working` with
`canRetry: false` at every step.
  - The bound passed with nothing recorded: Retry.
  - The continuation worker decided it cannot continue: Retry at once.
  - A person took over the hold: Retry at once.
- The #21200, #21264, #21320 and #21353 guard tests pass unchanged. This
includes "still offers Retry when recovery gave up and handed the hold
to the board" and the person-owned hold tests.
- Related suites: `workspace-recovery-preparation-dispatch`,
`explicit-native-continuation`, `workspace-recovery-retry-copyback`,
`workspace-recovery-admission`, `native-safe-replacement` and
`run-failure-report`: 1046 of 1046 tests pass. One run on a loaded
machine had a 30-second `afterAll` cleanup timeout in
`native-safe-replacement` after all its tests passed. An earlier
parallel run had one timing failure in
`workspace-recovery-preparation-dispatch` that did not repeat.
- `node scripts/typecheck-layers.mjs` in `server/`: passes.

## Risks

- A provider failure that the system will not continue (for example, the
task was reassigned) can show "working" for up to 2 minutes before Retry
appears. This is the same bound #21264 uses for a system-owned hold on
an ended run.
- The rule applies only to `native_provider_terminal_failed`, which is
the failure that the safe-replacement evaluation owns. Other terminal
failures are unchanged.
- One more small read per page, only when a listed task has a run in
this state.
- Note for the recovery owner, not changed here: the provider's 503
("credential validation failed. This is a server-side issue") is
classified as a permanent provider failure
(`native_provider_terminal_failed`). Whether a 5xx from the provider
should instead be a temporary retry on the same run is the recovery
owner's decision.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs, reading
restart-chaos wave evidence).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:43:34 -06:00
DottaandClaude Opus 5.5 57b787d3d2 fix(retry): continue from recorded work when saved files cannot be recovered (#21363)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run finishes but copying its files out of the sandbox fails,
Paperclip holds the task and tries to recover the saved files. When
recovery gives up, the task offers Retry.
> - Some saved files can never be recovered: the sandbox that held them
was destroyed, or Paperclip cannot prove which runs used a shared
sandbox. Each Retry ran the same check again, failed again, and parked
the Retry in a wait that never ends.
> - The task told the person to press Retry again or to wait for a
Paperclip update. That breaks the rule that recovery must work and that
a Retry that is shown must work.
> - This pull request makes Retry continue the task in a fresh session
and a new workspace from the work that is recorded, when the saved files
are provably unrecoverable or cannot be proven safe. The task says
clearly that the unsaved files were not recovered.
> - The benefit is that an offered Retry starts a run, and nobody waits
for a fix that another click cannot bring.

## Linked Issues or Issue Description

Refs #21315, #21295, #21267

**What happened?**

On hosted staging, two blocked tasks offered Retry (`canRetry: true`).
Each Retry was accepted, and within about 3 seconds it was parked in a
wait. No run started. Every later Retry did the same.

- Task A: the hold cause was
`native_workspace_sync_out_retry_exhausted`. Recovery ended with
`source_unavailable` (`use_alternative_strategy`). The sandbox that held
the files had a signed absence proof and a `destroyed` termination
receipt. Retry was parked with `saved_work_recovery_failed`: "Saving the
agent's files failed. Retry to try again".
- Task B: the same hold cause. Recovery ended with
`allocation_ownership_ambiguous` (`service_required`), because the
sandbox was a reusable sandbox that other runs also used. Each Retry
opened a new recovery generation, ran the same check, and was parked
with `saved_work_recovery_refused`: "Retry runs the same check again and
succeeds only after Paperclip is updated."
- In both tasks, the agent's turn had finished and its result was
accepted. Its work was recorded outside the sandbox: a plan document and
screenshot attachments for task A, and a pushed branch with an open pull
request and an implementation document for task B.

Root cause: admission had a branch for an accepted result whose files
were not yet saved. When the recovery budget was spent, the branch
always returned a `saved_work_recovery_*` wait. It did not check whether
the files could ever be recovered, so the fresh-workspace path that
admission already has was never reached. The automatic retained-source
fallback did not help either, because it handles only the remote
workspace v1 reference, and these runs use the native workspace sync v3
reference. Also, Retry opened a new recovery generation for an ownership
refusal, which only repeated the same check.

**Expected behavior**

When the saved files are provably unrecoverable or cannot be proven
safe, Retry continues the task in a fresh session and a new workspace
from the recorded work. The old files are never applied and the source
sandbox is not changed. No provider turn is replayed. The task says that
the unsaved files could not be recovered. The Retry offer and admission
agree.

**Steps to reproduce**

1. Let a native run finish with an accepted result, and let its
workspace copy-back fail (`native_workspace_sync_out_retry_exhausted`).
2. Let saved-work recovery end with `source_unavailable` (sandbox
destroyed) or with `allocation_ownership_ambiguous` (shared reusable
sandbox).
3. Press Retry.
4. Observe that the Retry receipt waits with
`saved_work_recovery_failed` or `saved_work_recovery_refused`, and no
run starts. Press Retry again and observe the same result.

**Paperclip version or commit**

Staging on `origin/master` after #21315 (`bbafa78fe2`).

**Deployment mode**

Hosted cloud (staging), remote Daytona sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime runs).

## What Changed

- `savedFilesUnrecoverable` and `unrecoverableSavedFilesFailure`
(`system-recovery-state.ts`) are the one rule for saved files that no
recovery can apply. The rule is true for a spent recovery whose failure
is a permanent source refusal: `source_unavailable`,
`allocation_ownership_ambiguous` or `historical_source_missing`. It is
false for `source_binding_invalid`, because a new recovery generation
can still verify that source. It is false for `archive_unsafe` and
`snapshot_verification_failed`, because a fresh attempt still imports
earlier saved snapshots and would fail before the agent starts. It is
also false after a fresh attempt is recorded.
- Admission (`explicit-native-continuation.ts`) no longer parks a Retry
of an accepted result in `saved_work_recovery_*` when every live
recovery of the run is unrecoverable. The Retry takes the existing
fresh-workspace path (`fresh_isolated_workspace`). That path keeps all
stop proofs, leaves the source sandbox and workspace untouched, and
starts a fresh session.
- When such a Retry starts, the task gets one system comment. The
comment says that Paperclip could not recover the unsaved files, why,
and that the task continues from its recorded work (comments, documents,
attachments and pushed commits). The settled hold records the same text.
- Retry (`task-retry.ts`) binds to an unrecoverable recovery generation.
It does not open a new generation that repeats the same check.
- The blocker text for unrecoverable files says that Retry continues the
task in a new workspace from its recorded work. It no longer says that
Retry succeeds only after an update.
- `canAdoptNativeSourceRetention` and the permanent source-refusal lists
moved into `system-recovery-state.ts`, so that admission, Retry and
messages share them. The old modules re-export them.

## Verification

- Red/green: `cd server && npx vitest run
src/services/unrecoverable-saved-work-retry.test.ts`. The tests fail on
`origin/master` sources (or on the earlier head for the snapshot case)
and pass with this change. They are built from the two staging shapes
(destroyed sandbox with `source_unavailable`, and a shared reusable
sandbox with `allocation_ownership_ambiguous`):
- offers a Retry that continues the task (blocker and task view agree,
and the text names the fresh workspace)
- starts a fresh run from the recorded work: the Retry receipt has no
wait and a run, the strategy is `fresh_isolated_workspace`, the hold is
settled with the fresh attempt, one system comment explains the lost
files with no raw IDs, the source lease is unchanged, and no new
recovery generation opens
- keeps a Retry out of a fresh attempt that would import a succeeded
snapshot whose archive is unsafe
- `workspace-recovery-warm-handoff.test.ts` › "wakes physical
preservation from HTTP Retry before passive history without replaying
the provider" failed on an earlier head of this PR, which also treated
`source_binding_invalid` as unrecoverable. It passes unchanged now.
- Updated tests that pinned the old repeat-the-check behavior:
`workspace-recovery-preparation-gates`, `task-retry`,
`system-recovery-state`, `workspace-recovery-retry-copyback`.
- Related suites pass: `system-recovery-state`,
`workspace-recovery-preparation-gates`,
`workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`,
`task-retry`, `saved-work-retry-eligibility`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
`workspace-recovery-admission`, `task-interruption-query`,
`native-workspace-sync-retention`.
- `cd server && node scripts/typecheck-layers.mjs` passes (source and
test layers).

## Risks

- Medium-low. A Retry of an unrecoverable hold now starts a new agent
turn in a new workspace instead of waiting. The turn starts only through
the existing fresh-workspace admission, which keeps every stop and
cleanup proof.
- Unsaved files of the old run are not applied. That was already true;
the change is that the task continues and says so.
- `source_binding_invalid`, `snapshot_verification_failed` and
`archive_unsafe` keep today's path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:23:23 -06:00
DottaandClaude Opus 5.5 e23549cd9c fix(presentation): a task with a continuation waiting to start shows as continuing, not as needing Retry (#21353)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run dies (for example during a server swap), the
fresh-continuation fallback records that Paperclip will continue the
task in a fresh run from saved work
> - The record says `continuationDelivery: "pending"` until a delivery
pass starts the new run. Delivery is paced, so a record can wait a long
time behind an older backlog
> - The task view showed such a task as "working" only for 15 minutes
after the record was made. After that it showed a failed attempt with a
Retry button, although the record still said Paperclip would continue
the task
> - Product rule: a task that the system has recorded it will continue
by itself never shows as failed with a Retry button
> - This pull request ties the "continuing" state to the record's own
delivery state instead of a timer
> - The benefit is that the task view stays truthful however long
delivery takes, and Retry comes back as soon as the system gives up

## Linked Issues or Issue Description

Refs #21264, #21324, #21297, #21320, #21200

**What happened?**

On hosted staging (build d1d71a7755, which contains #21264 and #21324),
five tasks had a run that died after a server swap. Each task had a
recorded fresh continuation that was not delivered yet. The primary
recovery record had cause `native_continuation_requires_reconciliation`,
status `resolved`, `continuationDelivery: "pending"`, isolation
`remote_epoch`, and no `continuationDeliveryAttemptedAt`. The other
holds of the task were coalesced into it. The records were 30 to 110
minutes old. The task view showed responsibility `system`, state
`failed`, and `canRetry: true`.

**Expected behavior**

While a continuation is recorded and its delivery is pending, the task
shows as continuing (`working`, `canRetry: false`, responsibility
`system`). This lasts until delivery starts the new run (then the new
run decides the state) or the system gives up (`invalidated`; then Retry
is offered).

**Steps to reproduce**

1. Record a fresh continuation for a failed run (`continuationDelivery:
"pending"`) and coalesce the task's other holds into it.
2. Let more than 15 minutes pass without delivery.
3. Read the task. It shows `failed` with Retry.

## What Changed

- `taskInterruptionPresentations` in
`server/src/services/task-interruption-presentation.ts`: a primary
fresh-continuation record with `continuationDelivery: "pending"` reads
as "continuing" with no time limit. Before, this applied only for 15
minutes (`RECOVERY_CONTINUATION_DELIVERY_BOUND_MS`). The state is the
existing `working` state that the code already uses for a recorded
continuation; no new state is added. The UI shows the agent's turn as
queued and shows no Retry button.
- A refused delivery attempt (`continuationDeliveryAttemptedAt`) does
not end the state. The record stays `pending` and the delivery pass
tries again on the next pass, so the system still owns the next step.
- The select no longer reads the hold time columns, because the
projection does not use them now.
- `recovery-wake-herd.test.ts`: the #21324 test that expected "not
working" past the bound now expects `working` with `canRetry: false`,
and then proves that the delivery pass still delivers that overdue
continuation.
- No change to the delivery pass, the fallback, or the Retry path.

## What a Retry click does in this state today

`requestTaskRetry` admits a person's Retry as a user wake. The queue's
"a pending fresh continuation absorbs this wake" check
(`hasPendingFreshContinuation` in `heartbeat/queue.ts`) applies only to
system wakes, so the Retry starts its own run from saved work. The
delivery pass then sees a run created after the record and marks the
continuation `delivered` (superseded). In the normal order this does not
start a second run. But the click only asks for what the system already
recorded it would do, and it races the delivery pass: if the pass has
already checked for a newer run and is waking the agent when the click
is admitted, the task can get two continuation runs, one after the
other. That is why the button must not show while the continuation is
pending.

## Verification

- Red (before the fix), `pnpm exec vitest run
src/__tests__/task-interruption-automatic-path.test.ts` in `server/`: 2
failed. The exact staging record shape at 30 minutes read `state:
"failed"`, `canRetry: true`, and a pending record after a refused
attempt read the same.
- Green (after the fix), in `server/`: `pnpm exec vitest run
src/__tests__/task-interruption-automatic-path.test.ts
src/__tests__/recovery-wake-herd.test.ts
src/__tests__/task-interruption-query.test.ts
src/services/task-interruption-presentation.test.ts
src/services/fresh-continuation-fallback.test.ts`: 5 files, 202 tests
pass.
- New tests in `task-interruption-automatic-path.test.ts`:
- The staging record shape at 1, 30 and 110 minutes shows `working`,
`canRetry: false`, responsibility `system`.
  - A refused delivery attempt keeps `working`.
- Delivery given up (`invalidated`, with the coalesced hold) offers
Retry.
- Delivered: the task follows the new run (queued: `working` on the new
run; the new run fails: Retry on the new run).
- A resolved recovery record with no recorded continuation offers Retry
as before.
- A board-owned hold with no recorded continuation offers Retry as
before.
- The #21200, #21264 and #21320 guard tests in the same file pass
unchanged.
- `node scripts/typecheck-layers.mjs` in `server/` passes.

## Risks

- #21324 limited "working" to 15 minutes so that a continuation that
never gets delivered does not hide behind "working". With this change, a
continuation that stays pending forever shows "working" forever. #21324
also made delivery bounded (an overdue record is read on every pass and
gets one wake per agent per pass), and a separate change is making
delivery prompt. A record that cannot be delivered (task done,
cancelled, or reassigned) is marked `invalidated`, and then Retry shows.
- Person-owned waits still win: approval, reconnect, budget, pause and
project access are checked before this state.
- Presentation only. No schema, migration, or execution change.
- This change supersedes the expectation of the #21324 test "does not
show a task as working when its continuation is pending past the
delivery bound". That test hid the continuing state after 15 minutes.
The state must stay truthful: the system has recorded that it will
continue the task, so the task must not show a failed attempt or a Retry
that can start a second run.
- Follow-up: while a continuation waits, a person has no control. On
staging a continuation can wait a long time behind a deep queue. After
#21348 is on master (a person's Retry folds into the pending
continuation in the admitting transaction, and runs that a person starts
rank ahead of system recovery), the planned follow-up keeps the
continuing state and, after 15 minutes, also offers a Retry worded as
"move it forward now" that folds into the pending continuation. This
pull request does not add that Retry.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 05:24:53 -06:00
DottaandClaude Opus 5.5 22c9300dac fix(recovery): a run that fails before the agent is asked anything continues by itself on a fresh computer (#21297)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox opens a provider session and then
waits at a launch gate before the first prompt goes to the provider
> - On remote OpenCode sandboxes the gate saves the provider launch
files; when that save fails, no prompt was ever sent
> - Today that failure ends as `native_session_interrupted` with
ambiguous recovery evidence, so the task goes blocked and the user must
press Retry
> - Nothing was sent to the provider, so nothing can be duplicated: the
system can safely continue by itself
> - This pull request types that failure, proves it from the Runner turn
ledger at the failure point, and continues the task on a fresh sandbox
in the existing interrupted-preparation lane
> - The benefit is that a setup failure before the first prompt no
longer needs a human, and the user sees plain words instead of internal
codes

## Linked Issues or Issue Description

Refs #21287
Refs #21304

**What happened?**
On a remote OpenCode sandbox, the provider launch-files gate failed
before the first prompt (for example
`native_provider_artifacts_transfer_timeout`). The run ended blocked
with a Retry banner. The run error showed internal codes and raw byte
counts to the user.

**Expected behavior**
A failure that happens before any prompt reaches the provider continues
automatically on a fresh sandbox. The task shows as working while the
continuation is pending. The visible text is plain and has no internal
codes or raw numbers.

**Steps to reproduce**
1. Use an OpenCode agent on a Daytona sandbox environment with
controller persistence.
2. Arm the crash point `native.provider_artifacts.before_capture` in
fail mode (or let the launch-file upload time out).
3. Start a task. The first run fails at the gate before any prompt.
4. On master the task goes blocked with Retry. With this change a
successor starts on a new sandbox and the task completes.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- New typed cause `native_failed_before_provider_turn`
(`server/src/services/native-runtime/failed-before-provider-turn.ts`).
The executor raises it only when the launch gate fails, the run was not
cancelled, and the Runner turn ledger proves that no turn command was
issued. The proof is taken at the failure point, before the session
closes. In all other cases the original error is rethrown and today's
path stays.
- The executor quarantines the failed run's session root, so the
successor starts in a new root.
- `heartbeat/predecessor-retirement-fallback.ts` accepts the new cause:
it records the proof on `runnerProfileJson.failedBeforeProviderTurn`,
marks the run's reusable sandbox leases for replacement, and schedules
the successor in the existing `interrupted_preparation` lane (same cap
of 12 and the same backoff).
- `remote-workspace-unchanged-proof.ts` accepts the runner process,
runner command and pre-turn events only when that proof record is
present. It still refuses any turn evidence, an unsettled save by
another run on the same workspace, and setup writes. When the proof
refuses, the run's epoch is abandoned unapplied: the files are kept and
recorded, and the next run restores the last committed save into a fresh
root.
- When the proof refuses and the epoch is abandoned, the replacement
mark stays. The next acquisition replaces the sandbox. #21304's
retention hold makes `retireReplacedReusableSandboxLease` park the old
lease as `retained` instead of `pending_cleanup`, so the task continues
on a fresh sandbox and the sandbox that holds the abandoned files is
kept. When the proof holds (only the Runner's own session folder was
written), the old sandbox is replaced and cleaned up as usual.
- `task-interruption-presentation.ts` shows a pending
`interrupted_preparation` retry as `working` with no Retry until it is
due (plus the dispatch grace).
- The visible run error is now plain: "Setting up the computer took too
long, so the task is continuing on a fresh one." (for a setup timeout)
or "The computer could not be set up before the agent started, so the
task is continuing on a fresh one." The technical cause code stays in
`runnerProfileJson.failedBeforeProviderTurn.technicalCause`, the server
log and the run log.
- New crash point `native.provider_artifacts.before_capture` (fail mode
throws the typed transfer timeout at the gate).
`tests/runner-e2e/source-sdk-public-inputs.json` is refreshed for the
changed registry file.
- `tests/restart-chaos/main.ts`: `RESTART_CHAOS_EXTERNAL_CRASH_CONTROL`
lets an outside driver arm fail-mode crash points without a restart
fault.

## Verification

Local (targeted):
- `server/src/__tests__/heartbeat-native-predispatch-retry.test.ts`: 8/8
pass. New cases: fresh remote sandbox and fresh plugin-task sandbox
continuation (now also asserts the plain visible text with no
identifier-looking tokens, and the technical cause in the proof record),
the ambiguous hold when the ledger cannot prove no turn, and backoff
plus the lane cap.
- `server/src/services/remote-workspace-unchanged-release.test.ts`:
27/27 pass. New cases: hand-on when the ledger proves no turn; refusal
on a recorded turn, an unsettled predecessor save, and setup writes. Two
more cases follow the marked lease through the next acquisition's
replacement decision (`readReplaceBeforeReuseMarker` +
`reusableSandboxReplacementPlan` + `retireReplacedReusableSandboxLease`)
and then through `sweepPendingCleanupLeases`. "replaces the marked
sandbox when the failed run provably left its files unchanged": the old
lease goes to `pending_cleanup` and the sweep destroys it. "continues on
a fresh sandbox and parks the one that holds abandoned files, which
cleanup leaves alone": the old lease is `retained` with the
abandoned-files reason and is held by the receipt, the sweep does not
destroy it, and the file is still there.
- Red without the product change (product files reset to the base, tests
kept): 8 failed, 25 passed of 33. All 8 failures are the new cases. The
visible-text assertion alone was red on the previous commit: the error
read "The computer could not be prepared before the agent started
(native_provider_artifacts_transfer_timeout: the upload of 536870912
bytes did not finish within 120 s). The task continues on a fresh
computer."
- `task-interruption-presentation.test.ts`,
`native-provider-artifact-gate.test.ts`,
`crash-point-call-sites.test.ts`: 79/79 pass.
`packages/shared/src/crash-points.test.ts`: 16/16.
`tests/restart-chaos/crash-points.test.ts`: 12/12.
- `tsc --noEmit` clean for `server`, `packages/shared` and
`tests/restart-chaos`.

Daytona (local servers, Daytona sandboxes, image `sha256:787c3e9c…`,
these commits rebased onto a298ca7fbc, gate failures forced with the
crash point):
- F1, wave `w-20261113T042929-s2301`: 4 OpenCode tasks, one forced
failure on each first run. 4/4 failed runs recorded cause
`native_failed_before_provider_turn`. Each successor (retry reason
`interrupted_preparation`) was created 0.0–0.1 s after the failure and
started 1–3 s later, each on a NEW sandbox. Successor first agent event
at 32.8–56.6 s. 4/4 tasks done hands-off. 0 Retry banners. 0 blocked. No
operator wording. No reconciliation codes.
- F3, wave `w-20261113T043934-s2303`: 1 task, 3 consecutive forced
failures. Retries started 7 s, 35 s and 57 s after creation (backoff).
The 4th run succeeded. 0 Retry banners while pending. Not proven: that
retries 2 and 3 in F3 got fresh sandboxes (proven in F1 only).
- F2 (what bootstrap writes before the first turn): the whole sandbox
workspace was sampled every 5 s on all 4 failed runs, right after the
workspace was ready and at the gate failure. Bootstrap wrote nothing to
project files. All 14 new paths are under
`<workspace>/.paperclip-runtime/paperclip-runner/sessions/<session-sha>/`
(runner state, identity and lineage, `runtime-context.json`, the
instructions copy `AGENTS.md`, OpenCode config, data and log). The only
deletion was the staged agent-files `AGENTS.md` (or a
`.paperclip-upload-<uuid>` temp file). The successor never reuses them:
new session root, new workspace generation, restored from the snapshot.
3 of the files are byte-identical when recreated; the rest are
run-specific and recreated by the successor's own bootstrap. The project
workspace is not a git checkout there. Raw manifests are kept with the
PR author.
- The Daytona runs used the error text before the plain-wording commit.
The wording change is covered by the unit test above.

## Risks

- Scope: this applies only where the gate exists: remote OpenCode
sandboxes with controller persistence. Local runs cannot hit it.
- The proof is the Runner's in-memory turn ledger, read at the failure
point. In unit tests it is mocked. If a turn may have been sent, today's
ambiguous hold stays (tested).
- At the lane cap (12) the run stops with a working Retry instead of
waiting longer. A later change can move this onto the shared hourly
limit.
- A direct "is the workspace dirty" check by the Runner is not added,
because it needs a new Runner command. The unchanged proof instead
relies on the recorded evidence and refuses on any turn evidence,
unsettled saves or setup writes.
- Whether a replacement mark gives a fresh sandbox depends on
`reusableSandboxReplacementPlan`. When it refuses, the sandbox is
resumed, but the session root and workspace root are still fresh.
- Refused case (abandoned files): the task continues on a fresh sandbox,
in a fresh session root and a fresh workspace root. The old files stay
on the parked sandbox, recorded in the abandoned-epoch receipt and
unapplied. That sandbox is kept under #21304's hold until the files are
recovered, discarded, or the retention period ends; this PR does not
change that hold.
- The changes in `heartbeat/` and `remote-workspace-unchanged-proof.ts`
were cleared with their owner under these conditions: evidence-scoped,
latest committed epoch only, files kept when unproven, negatives tested.
- `server/src/services/fresh-continuation-fallback.ts` is deliberately
untouched.
- Upgrade safety: runs saved by an older build have no proof record, so
the unchanged proof keeps its previous behavior for them.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use (shell, file edits, test runs). Earlier commits in this branch
came from Claude sessions in the same fix loop.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:48:51 -06:00
DottaandClaude Opus 5.5 4fe34df748 Do not show Retry on a parent with open subtasks while its queued message is settled (#21320)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The task view shows a "Retry" notice when a task needs a person to
start it again.
> - The product rule is that a task that continues by itself never shows
this notice, not even for a few seconds.
> - A parent task that waits for its subtasks showed the notice for 3.4
seconds after its run succeeded.
> - The cause is the queued-message grace in the task interruption
presentation: it does not apply to `blocked` tasks.
> - #21200 kept `blocked` out of that grace on purpose, so this pull
request adds only the exact observed shape: a parent with open subtasks
and a queued task message.
> - The benefit is that a parent that waits for its subtasks no longer
shows a false failure, and every other blocked task keeps its Retry.

## Linked Issues or Issue Description

Refs #21264, #21200 (added the queued-message grace and kept `blocked`
out of it)

**What happened?**
A Codex parent task ran across a two-server release swap. Its run waited
for two subtasks and then succeeded with the disposition `blocked`. The
task status became `blocked`. A user comment (a deferred wake, status
`deferred_issue_execution`) had been queued 13 minutes earlier, while
the run was active. For 3.4 seconds after the run finished, the task
presentation was `state: "failed"`, `canRetry: true`. Then the release
drain cancelled the wake as "Queued messages were already delivered to a
completed run", and the presentation changed to `resolved`. The subtasks
finished later. Their `issue_children_completed` wake continued the
parent, and it completed without help.

The cause is in `projectTaskInterruptionState` in
`server/src/services/task-interruption-presentation.ts`. The
`queuedDuringRun` check shows a succeeded run with user input queued
before it finished as `working` for up to 2 minutes. It applies only to
`todo`, `in_progress` and `in_review`. For a `blocked` task the
presentation fell through to the last line: `failed` with `canRetry:
true`.

The workspace finalization deferral
(`native_workspace_finalization_busy`) in the same window does not
change the presentation. The presentation does not read it.

**How this differs from what #21200 narrowed**
#21200 first applied the grace to every status. Its commit `a3fae9bb67`
removed `blocked` because input queued on a blocked task is not promoted
by itself, and because a recovery that gives up hands its hold to the
board and must end with Retry. This change does not add `blocked` back.
It adds one narrow case, and all of these must be true:
- The latest run **succeeded**. A failed, interrupted or given-up run is
not affected.
- The task is `blocked` **and has open subtasks of its own** (not `done`
or `cancelled`). Their completion wakes the parent again
(`issue_children_completed`).
- The pending item is an ordinary **task comment** wake (`commentId` or
`wakeCommentIds` in its payload), not a Retry request
(`taskRetry.sourceRunId`).
- The comment was queued before the run finished, the run finished less
than 2 minutes ago, and saved-work recovery is not exhausted. These are
the existing bounds.

A blocked task that waits for a person or a board decision has no open
subtasks, so it keeps Retry as before.

**Expected behavior**
After the run succeeds, the parent shows `working` with no Retry while
its queued message is settled. Then it shows `resolved` (message already
delivered) or `working` (message promoted to a new run). If nothing
settles the message in 2 minutes, Retry shows.

**Steps to reproduce**
1. Assign a parent task that creates subtasks and waits for them.
2. Post a comment on the parent while its run is active.
3. Let the run succeed with the disposition `blocked`.
4. Read the task presentation in the first second after the run
finishes. It shows `failed` with Retry.

**Paperclip version or commit**
master `bbafa78fe2`. The wave that found it ran on `62d673cc3f`.

## What Changed

- `server/src/services/task-interruption-presentation.ts`:
- The batched wait read also reads whether the wake carries a task
comment (`queuedComment`).
- One company-scoped read finds which `blocked` tasks with a succeeded
latest run and a queued comment have open subtasks
(`waitingOnOpenSubtasks`). It runs only for those tasks.
- The `queuedDuringRun` grace also applies to such a task. The `todo`,
`in_progress` and `in_review` behavior does not change.
- `server/src/__tests__/task-interruption-automatic-path.test.ts`, new
embedded-Postgres tests:
- The exact observed state: a `blocked` parent with two open subtasks, a
run that succeeded 1 second ago, a user comment wake queued 13 minutes
earlier. Shows `working`, no Retry.
- The same parent after the wake is cancelled as already delivered:
`resolved`.
- The same parent after the wake is promoted to a new run: `working` on
the new run.
- Guard: the same parent after the 2-minute grace: `failed` with Retry.
- Guard: a `blocked` task with a queued comment and no subtasks, or only
`done` or `cancelled` subtasks (waiting for a person or a board
decision): `failed` with Retry, unchanged.
- Guard: a parent with open subtasks whose queued wake carries no
comment: `failed` with Retry, unchanged.
- Guard: a parent with open subtasks and a queued comment whose run
failed and whose recovery hold went to the board: `failed` with Retry.
- Guard: a parent with open subtasks and a queued comment whose
saved-work recovery is exhausted: `failed` with Retry.
- A parent whose run succeeded with no queued message stays `resolved`
when its subtasks are open, blocked or cancelled.
- The existing #21200 assertion (a `blocked` task with queued input and
no subtasks shows Retry) is kept unchanged.

## Verification

- `cd server && npx vitest run
src/__tests__/task-interruption-automatic-path.test.ts`
- Red on master `bbafa78fe2`: "shows a parent whose run just succeeded
waiting on its subtasks as working while its queued message is settled"
returns `state: "failed"`, `canRetry: true`. This is the value that the
wave recorded. The other 15 tests pass on master.
  - Green with this change: 16 of 16.
- A plain `blocked` inclusion fails 3 of the guards: the #21200
assertion, "still offers Retry on a blocked task with a queued message
and no open subtasks" and "still offers Retry on a parent with open
subtasks when its queued wake carries no message". This shows that the
narrow condition is necessary.
- `task-interruption-presentation.test.ts`,
`task-interruption-query.test.ts`: green (116 tests with the file
above).
- `cd server && npx tsc --noEmit -p .`: 0 errors.
- Not verified: no new chaos wave. The flash was seen once in a loaded
release-swap wave. The server test reproduces its saved state.

## Risks

- Low risk. The change is presentation only. It does not change wake
promotion, dispatch or Retry.
- A parent with open subtasks and a stranded queued comment now shows
`working` for up to 2 minutes after its run succeeded, and then Retry.
Before, it showed Retry at once.
- One extra indexed read (`issues_company_parent_idx`) runs only for
blocked tasks with a succeeded latest run and a queued comment.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edits, test execution). Extended
reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:27:22 -06:00
DottaandClaude Opus 5.5 fde0328ff5 fix(recovery): deliver every pending continuation within a bounded time (#21324)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Execution recovery continues a task in a fresh run after a restart
or a lost run left it unfinished
> - #21234 paced these continuations: at most 1 per agent and 5 per
company per pass, and a continuation waits while its agent already has a
queued recovery run or a full queue
> - A continuation that waited 5 minutes must enter the queue anyway,
but that bound applied only to the full-queue check. The "agent already
has a queued recovery run" check was in the page query, and it had no
bound.
> - On staging, an agent had queued recovery runs that did not start for
hours. Its pending continuation was never read again, and its task
showed "working" with nothing running or queued
> - This pull request applies the 5-minute bound to both checks, stops
one undeliverable continuation from using its agent's or company's slot,
and stops the task view from showing "working" for a continuation that
is overdue
> - The benefit is that every pending continuation enters the queue
within a bounded time, also after a restart, and the pacing still
protects people from recovery herds

## Linked Issues or Issue Description

Refs #21234, #21135

**What happened?**

On staging, a task had a fresh-continuation hold with
`continuationDelivery: "pending"` since about 03:30Z. More than 3.5
hours later, no run had started for the task. The task view showed the
interruption state `working` with `canRetry: false`, so the user saw
nothing wrong. The task's agent had 20 concurrency slots and 11 running
runs. It also had queued `issue_continuation_needed` runs from 22:54Z
that had not started. The delivery pass reads pending continuations with
a query that leaves out every agent that has a queued recovery run
(`issue_recovery_action_restored` or `issue_continuation_needed`). That
condition had no time bound, so the pass never read this continuation
again. The 5-minute admission bound was never reached. Across the
company, 203 open tasks showed `working` while the API showed no queued
or running run for them. 162 queued runs were more than 1 hour old.

**Expected behavior**

Every pending continuation enters its agent's queue within a bounded
time, also when the agent's queue holds recovery runs that do not start,
and also after a server restart. Pacing still limits how many
continuations enter a queue at the same time. A task does not show
"working" when its continuation is overdue and nothing runs or is
queued.

**Steps to reproduce**

1. Give an agent a concurrency limit of 1.
2. Put one queued `issue_recovery_action_restored` run on that agent
that does not start.
3. Record a pending fresh continuation for another task of the same
agent.
4. Wait more than 5 minutes, and let the 15-second continuation pass
run.
5. See that the continuation stays `pending`, the task shows `working`,
and no run starts.

**Paperclip version or commit**

master at bbafa78fe2 (includes #21234, ade7e0643f)

**Deployment mode**

Hosted cloud (staging)

**Agent adapter(s) involved**

paperclip_runner (native runtime). The problem is in the shared
continuation delivery pass, so it applies to all adapters.

## What Changed

- `deliverReconciledExecutions` in
`server/src/services/execution-recovery-resolution.ts`: a continuation
held for 5 minutes or more is read even when its agent has a queued
recovery run. Before, only the full-queue check let it through.
- Each page of the pass now reads at most one candidate per agent
(`row_number() over (partition by agent)`), so one agent's backlog
cannot fill the page budget and hide other agents' continuations.
- A continuation counts against the company's limit only when its wake
puts a run in the queue. A continuation that was invalidated or replaced
by a newer run does not use its agent's turn.
- When an agent refuses a wake (no run, or an error), the pass skips
that agent for the rest of the pass and stamps the row with
`continuationDeliveryAttemptedAt`. Overdue rows always rank before rows
that are not overdue. Among overdue rows, the turn is the later of the
hold time and the last attempt, so a refused row goes behind rows held
before that attempt and ahead of rows held after it. One refused row
cannot pin its agent, and new arrivals cannot keep an older refused row
waiting.
- New module `server/src/services/recovery-continuation-pacing.ts` holds
the pacing constants and the "held since" rule. It adds
`RECOVERY_CONTINUATION_DELIVERY_BOUND_MS` (15 minutes). The old
constants are still exported from `execution-recovery-resolution.ts`.
- `taskInterruptionPresentations` in
`server/src/services/task-interruption-presentation.ts` shows a pending
continuation as "continuing" only within the delivery bound. After that,
the task shows its real state instead of `working`.
- No change to the schedule: the continuation duty runs only while
pending continuations exist, and it checks the database on start, so a
restart does not lose pending rows.

## Verification

- `pnpm exec vitest run src/__tests__/recovery-wake-herd.test.ts` (in
`server/`): 15 tests pass. New tests:
- A continuation held past the bound is delivered while the agent's
queue holds recovery runs that never start.
- After a restart with a continuation pending past the bound and a full
queue, a new schedule is due within one pass, the pass delivers the
continuation, and the schedule then goes idle.
- Continuations whose wake returns no run do not use the per-company
count. Another continuation of the same agent is delivered on the next
pass.
- An overdue continuation is refused once while the agent's queue is
full. On the next pass, it is delivered even though new continuations
keep arriving for the same agent.
- One agent refuses a backlog of 210 overdue continuations. Another
agent's continuation is delivered in the same pass, and the refusing
agent gets one wake attempt in that pass.
- A task whose continuation is pending past the delivery bound with
nothing running does not show `working`. A fresh pending continuation
still shows `working` with `canRetry: false`.
- Red/green: with the two source files reverted to master (and only the
new constant exported), all 4 first new tests fail. The backlog test
fails on the first PR head (200 refused wake attempts in one pass). The
temporary-refusal test fails on the second PR head (the overdue row
stays pending). With this change, all 15 pass.
- Related suites pass: fresh-continuation fallback, workspace recovery
fallback, native safe replacement, native workspace reconstruction, task
interruption presentation, queued-run dispatch order, execution recovery
resolution, assigned-work invariant, execution control work, task
interruption automatic path and query.
- `node scripts/typecheck-layers.mjs` (in `server/`) passes.

## Risks

- A continuation that waited 5 minutes now enters a queue that already
has recovery runs. The per-agent (1) and per-company (5) limits per pass
still apply, and the queue still starts person-initiated work first.
- An agent that refuses wakes gets one wake attempt per pass. Its
refused rows rotate through `continuationDeliveryAttemptedAt`, and they
do not block other agents.
- After 15 minutes without delivery, the task no longer shows `working`.
With this fix, delivery happens well within that bound, so this is a
safety net.
- No schema or migration changes. No new polling.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, and read-only staging
API requests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:12:13 -06:00
DottaandClaude Opus 5.5 bbafa78fe2 fix(retry): offer Retry only when admission can take it, and start held saved-work Retries (#21315)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run fails and its saved work is held for recovery, the task
shows a Retry button. Admission then decides if the Retry can start a
new turn.
> - The task view, the task's execution blocker, and admission each
computed their own answer. The task view said `canRetry: true`. The
blocker said `canRetry: false`, and its own `workspaceRecovery` part
said `canRetry: true`.
> - Admission refused these Retries with no reason. The receipt kept the
generic `execution_recovery` wait ("Retry is waiting for execution
checks"), and each later click joined the same held receipt. Nothing
ran.
> - The refusals had real causes. A preparation proof rejected
host-written workspace notices in the run log. A repair proof did not
accept a snapshot that the failed preparation only restored and never
changed.
> - This pull request fixes both proofs, names every admission refusal
of a held Retry, and makes the task view and the blocker read one shared
Retry answer.
> - The benefit is that Retry starts work in these cases, and Retry is
offered only when admission can take it.

## Linked Issues or Issue Description

**What happened?**

On hosted staging, several blocked tasks showed a Retry button. Pressing
Retry was accepted but nothing ran. The task said "Retry is waiting for
execution checks. Your message is saved." The task view had
`interruption.canRetry: true` and `retryWait.reason:
execution_recovery`. The task's `executionBlocker` had `canRetry:
false`.

- Saved-work recovery failed with `source_unavailable`. The legacy
preparation had failed before the provider started
(`executionRecovery.providerWorkStarted: false`), its sandbox epoch was
abandoned, and its sandbox was destroyed. The blocker contradicted
itself: `workspaceRecovery.canRetry: true`, top-level `canRetry: false`.
- Saved work showed "recovered", but `workspaceRepairRequired` stayed
`true`. The failed preparation had restored a snapshot that an earlier
run produced. The repair proof accepted only a snapshot that the run
itself saved.

**Expected behavior**

The task view, the execution blocker, and admission give the same Retry
answer. A Retry that admission can take starts work. When saved work
cannot be recovered, Retry continues in a fresh session from the
recorded work. Every admission refusal of a held Retry has a named
reason. No Retry is offered for a refusal that another click cannot
change.

**Steps to reproduce**

1. Let a legacy `paperclip_runner` preparation fail before the provider
starts, after the host writes its workspace notices ("Execution
workspace reuse freshness action ...", "Skipping saved session resume
...") to the run log.
2. Let saved-work recovery end with `source_unavailable` and
`use_alternative_strategy`.
3. Press Retry.
4. Observe that the Retry receipt is held with `executionWait.reason =
execution_recovery` and no run starts.

Or:

1. Let a legacy preparation restore the task's saved snapshot and fail
with `setup_failed` before the provider starts.
2. Let saved-work recovery complete.
3. Observe that the blocker keeps `workspaceRepairRequired: true`, and
that Retry is held with `execution_recovery`.

**Paperclip version or commit**

Staging on `origin/master` `62d673cc3f` (after #21267 and #21295).

**Deployment mode**

Hosted cloud (staging), remote Daytona sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (legacy runtime runs, native agent).

Refs #21267, #21295.

## What Changed

- `hasVerifiedPreparationLog` (`verified-preparation-log.ts`) now
accepts a log made only of host notices that the host writes before it
hands the turn to an adapter: the mentioned-skills line (first record
only, checked against the saved skill selection) and the workspace
freshness and session-reset notices. Every record must have the same
attempt, sequential `seq`, `stdout`, and the excerpt and watermark must
match. Any other line still keeps the run held.
- `verifiedUnchangedRestoredSnapshot`
(`remote-workspace-snapshot-reuse-receipt.ts`) proves that a failed
preparation's saved state is the snapshot it restored. It needs
`executionStage: preparing`, `providerWorkStarted: false`, exactly one
succeeded restore of that snapshot at that epoch and lease, and no
snapshot save by the run. `verifiedWorkspaceRepairReceipt` uses it next
to the existing reuse proof.
- `savedWorkOffersRetry` (`workspace-recovery-receipt.ts`) is the one
rule for Retry on a saved-work hold. Retry is offered when the saved
work is recovered and verified, or when recovery is exhausted in a state
that only a Retry restarts (a new recovery generation, or the
fresh-session fallback after `source_unavailable`). Retry is not offered
while recovery re-arms by itself (a stop recheck or a newer capture
policy), or while a copyback owner's stop is unconfirmed. The blocker
uses this rule for the top-level `canRetry` and for
`workspaceRecovery.canRetry`, so the two parts cannot contradict.
- `task-retry-eligibility.ts` lists the held-Retry refusals that another
click cannot change. The task view and the blocker both use it.
- Admission (`explicit-native-continuation.ts`) names each refusal of a
held request that was silent before: `previous_execution_unverified`,
`execution_settling`, `cleanup_pending`, and `remote_cleanup`. It names
a refusal only when an execution blocker holds the task, so a request
without a hold still takes the normal path.
- The task view now reads the blocker's answer for a task with a
saved-work hold. It shows no Retry when the blocker offers none, and it
names the recovery wait.

## Verification

- Red/green: `cd server && npx vitest run
src/services/saved-work-retry-eligibility.test.ts`. The tests fail on
`origin/master` sources and pass with this change. They are built from
the staging shapes:
- offers the same Retry in the task view and the blocker when saved work
is no longer available
- offers no Retry in either view while saved-work recovery waits for an
unconfirmed file copy to stop
- continues in a fresh session from the recorded work when saved work is
no longer available
- verifies the repair of a restored snapshot that the failed preparation
never changed
- keeps the repair unverified when the failed preparation may have
changed the restored snapshot
- names a refusal that another click cannot change and offers no Retry
for it in either view
- names the admission refusal of a held Retry and offers Retry only when
admission would take it
- `npx vitest run src/services/verified-preparation-log.test.ts`: new
cases accept the host notices and keep a provider line, a different
attempt, or a late skills line held.
- Related suites: deferred execution wait, execution blocker read,
explicit native continuation, governed wait runner loss, legacy
preparation retry hold, remote workspace snapshot reuse, task
interruption presentation, task retry, and all `workspace-recovery-*`
suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Medium-low. The preparation-log proof accepts more host lines. Each
accepted line has a fixed host format, and the run must still pass every
other preparation proof (events, leases, no provider evidence).
- The restored-snapshot proof applies only when the run recorded that
the provider never started and it never saved its own snapshot.
- Admission now records a named wait instead of the generic one for some
held requests. It changes no admission decision.
- The task view calls `getExecutionBlocker` for tasks that offer Retry
and have a saved-work hold. These tasks are few, so the extra reads are
small.
- Not covered here: a copyback owner on a replaced deployment still
needs its stop confirmed before saved work can be copied out of a full
sandbox. The task view now shows that wait instead of a Retry that
admission refuses.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:52:25 -06:00
DottaandClaude Opus 5.5 c6c7a61e2c fix(retry): release a Retry held on a resolved decision (#21267)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, the task shows a Retry button. Admission
can hold an accepted Retry while a person must answer a pending approval
or question (`decision_pending`).
> - The hold is saved on the deferred Retry receipt. Nothing cleared
that saved reason when the person answered, so the task kept saying
"Retry is waiting for a pending approval or question" when nothing was
left to answer.
> - The deferred-receipt worker drops a refused receipt from its
schedule. No producer hinted it when the decision resolved, so the held
Retry did not start until some unrelated event rescanned the company.
> - A Retry was also held by a confirmation that was already open when
the failed turn was admitted. That turn had already run past the
confirmation, so the Retry added a gate that the original turn did not
have.
> - A person's comment on the task started a continuation, but the held
Retry receipt stayed deferred next to it.
> - This pull request makes a resolved decision release the held Retry
at once, keeps the projection from showing a resolved decision as the
wait, stops holding a Retry on a decision that predates the failed turn,
and settles a held Retry when a comment or answer continues the same
failed run.
> - The benefit is that Retry always works: a person never waits on a
decision that is already resolved.

## Linked Issues or Issue Description

**What happened?**

A board user pressed Retry on blocked tasks. No run started. The task
said "Retry is waiting for a pending approval or question to be
resolved. Your message is saved." In one task, the only interaction was
a `request_confirmation` that the previous successful run created. The
user's later comment started the turn that failed. The confirmation
stayed pending, so every Retry attached to the same held receipt. After
the user rejected the confirmation and sent a new comment, the task
still showed the same wait. No run started until a server restart
rescanned the deferred receipts about three minutes later. The task
projection said `canRetry: true` with the `decision_pending` wait.

**Expected behavior**

An accepted Retry is never held on a decision that is already resolved.
Accepting or rejecting the decision releases a held Retry right away. A
person's comment on the blocked task starts work, and the held Retry is
settled. The task projection never shows a resolved decision as the
Retry wait.

**Steps to reproduce**

1. Let an agent run on a remote runner create a `request_confirmation`
for the task.
2. Let the runner be lost so the run fails with
`native_continuation_requires_reconciliation`, and let saved work
recovery complete.
3. Press Retry. The Retry receipt is saved with `executionWait.reason =
decision_pending`.
4. Reject or accept the confirmation.
5. Observe that the receipt keeps `decision_pending`, the task keeps
showing the wait, and the Retry does not start until an unrelated
rescan.

**Paperclip version or commit**

Staging Core `57aeda190b`. Fixed on top of `origin/master` `53608bd1d6`.

**Deployment mode**

Hosted cloud (staging), remote sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime).

## What Changed

- `hasPendingContinuationDecision` (`explicit-native-continuation.ts`)
is now the single check for a decision that holds a person's
continuation. Admission and the task projection both use it.
- For a Retry, a pending interaction that was created before the failed
source run was created does not hold the Retry. That turn was already
admitted while the decision was open. Decisions that the failed run
created (for example a plan review) still hold the Retry.
- `releaseDecisionHeldExecution` (`deferred-execution-wait.ts`) removes
the saved `decision_pending` wait from deferred receipts on the task and
hints each receipt to the deferred-receipt worker. It runs in the
resolution transaction of every interaction resolution path (through
`notifyChatConfirmationResolution`) and in question answers.
- The task projection reads the live decision state. A saved
`decision_pending` wait whose decision is resolved is not shown. This
also repairs receipts saved before this change.
- When a comment or an interaction answer continues a failed run,
`retireTaskRetriesContinuedBy` (`task-retry.ts`) settles held Retry
receipts for the same source run as `task_retry_superseded`.

## Verification

- Red/green: `cd server && npx vitest run
src/services/governed-wait-runner-loss.test.ts`. The five new tests fail
on `origin/master` sources and pass with this change (17/17 pass):
  - releases the held Retry as soon as the person chooses accept
  - releases the held Retry as soon as the person chooses reject
- never shows a resolved confirmation as the Retry's wait, and both
projections offer the same Retry
- a person's comment starts work and settles the held Retry once nothing
is left to answer
- starts a Retry at once when the confirmation was already open before
the failed turn started
- The existing tests that keep a Retry or a message on hold while a plan
review, governed tool action, secret proposal, or connection
authorization is pending still pass.
- Related suites: deferred execution wait, explicit native continuation,
interaction resolution and service, task interruption presentation and
query, task retry, deferred receipt work, interaction routes.

## Risks

- Medium-low. A Retry now starts when the only pending interaction is
older than the failed run. That turn already ran with the interaction
open, so the Retry gets no new authority. Governed tool actions still
need their own approval before the tool runs.
- Interaction resolution now also updates deferred receipts on the same
task. The update is limited to receipts with a saved `decision_pending`
wait. A receipt that is still blocked by another pending decision is
held again on its next admission.
- Issue-linked approvals (`approvals` table) do not yet hint the held
receipt when they resolve. The projection no longer shows them as the
wait after they resolve.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:58:48 -06:00
DottaandClaude Opus 5.5 d169dd5a90 fix(presentation): offer Retry only when no automatic continuation or scheduled retry is pending (#21264, includes #21200)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run fails with a transient error, Paperclip schedules the
next attempt by itself and shows "retrying automatically" with a
countdown.
> - The task presentation also offered the Retry button next to that
countdown.
> - The system owns the next attempt while the run is `scheduled_retry`,
so the button is misleading: it suggests that a person must act.
> - This pull request stops offering Retry while a scheduled retry is
pending. Retry returns when the scheduled retry is past due with nothing
pending, or when the retries are exhausted.
> - The benefit is one clear message: Paperclip retries by itself, and
Retry appears only when it is the way forward.

## Linked Issues or Issue Description

**What happened?**
A legacy run in status `scheduled_retry` (for example reason
`transient_failure`, which is not a shared-resource wait and has no
system recovery) projected `state: "retry_scheduled"` with `canRetry:
true`. The UI then showed the Retry button beside "retrying
automatically". The cause is in `projectTaskInterruptionState` in
`server/src/services/task-interruption-presentation.ts`: the `retryAt >
now` branch set `canRetry: !preparationPending && !nativeResumeAt`,
which ignores that the run itself is the scheduled attempt.

**Expected behavior**
While a run is `scheduled_retry` and its retry time is in the future,
the task shows the countdown and offers no Retry. When the scheduled
retry is past due with nothing pending, or the retries are exhausted,
Retry is offered.

**Steps to reproduce**
1. Let a task's run fail with a transient failure that schedules an
automatic retry.
2. Open the task before the retry time.
3. The task shows "retrying automatically" and the Retry button.

**What a Retry click does today in this state**
`requestTaskRetry` (`server/src/services/task-retry.ts`) treats a latest
run in `scheduled_retry` as busy. It accepts the click (202), records
the request as `coalesced` onto the scheduled run and returns
`scheduledRetryRunId`. It does not start a second attempt next to the
scheduled one. The button is therefore misleading but not unsafe, and
this pull request changes only what the task shows.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: in the
`retry_scheduled` branch, `canRetry` is now also false when `run.status
=== "scheduled_retry"`.
- `server/src/services/task-interruption-presentation.test.ts`: three
tests for a scheduled transient-failure retry.
- This branch is based on current `master` and includes #21200, which
has not merged yet and changes the same file. The diff of this pull
request against #21200 is one line and the tests.

## Verification

- `cd server && npx vitest run
src/services/task-interruption-presentation.test.ts -t "automatic
scheduled retry"`
- "shows a pending transient-failure retry as scheduled without a
competing Retry": red on `master` (`canRetry: true`), green with this
change.
- "offers Retry once the scheduled retry is past due with nothing else
pending": green before and after.
- "offers Retry once the scheduled retries are exhausted": green before
and after.
- Locally: `task-interruption-presentation.test.ts`,
`task-interruption-query.test.ts` and
`task-interruption-automatic-path.test.ts` pass (107 tests). Server
typecheck: 0 errors. The wider suites that read the presentation run in
CI; they were not repeated locally because the machine was under memory
pressure.

## Risks

- Low risk. The change affects presentation only. The Retry route still
accepts a click and coalesces it onto the scheduled run, as before.
- If a scheduled retry never dispatches, the task offers Retry again
once the retry time passes, as before this change.
- No schema, API or UI change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code with tool use (shell, file edits, test execution). Extended
reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:14:41 -06:00
DottaandClaude Opus 5.5 4363a3115a fix(recovery): continue any unfinished run in a fresh run from saved work (#21135)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs are interrupted when a server restarts, a deploy swaps
servers, a controller lease is lost, or a sandbox Runner cannot be
reached again
> - Exact recovery (resume the same session, replace, retry) refuses in
many ways, and each refusal leaves the task blocked or idle for a person
> - On staging, 577 of 1,000 runs in 11 hours failed or were
interrupted, across 25 error codes. About 900 earlier PRs each fixed one
refusal code, and the rate did not converge
> - This pull request adds one universal fallback: when an unfinished
run cannot be resumed exactly, for any reason that is not the person's,
the task continues in a fresh run from its last saved workspace and its
conversation
> - The benefit is that a new failure code next week falls back without
code changes, and tasks stop waiting on a person for infrastructure
faults

## Linked Issues or Issue Description

**What happened?**
After a server restart or deploy swap, many runs end with codes such as
`native_continuation_requires_reconciliation`
(`runner_remote_process_identity_unavailable`), `controller_lease_lost`,
`process_lost`, `server_shutdown_interrupted` and
`native_session_interrupted` (including `runner_harness_state_mismatch:
resumed_not_suspended` / `resumed_unavailable`). The disposition worker
then settles the recovery hold as "blocked, no replay". The task stays
Blocked until a person presses Retry or reconciles the run.

**Expected behavior**
A task waits on a person only for reasons that are the person's: an
approval, a question, a credential to share, a budget stop, a provider
usage limit, or a Stop they asked for. For every other failure, the
system starts a new run for the same task from the last verified saved
workspace, without a person.

**Steps to reproduce**
1. Start a task on a sandbox agent.
2. Restart or swap the server while the run is in progress, so the new
server cannot re-reach the Runner.
3. See that the run fails with
`native_continuation_requires_reconciliation` and the task stays Blocked
with no successor run.

**Paperclip version or commit**
`master` at 9e910ff9ae.

## What Changed

- New module `server/src/services/fresh-continuation-fallback.ts`. One
entry point, `ensureFreshContinuation` / `continueFreshInTransaction`,
takes a run that ended without finishing and guarantees one of:
something already follows it (a newer run, a linked replacement, an
operator decision), exact recovery is still under way (checked again
later), the cause is the person's, or one fresh continuation is
recorded.
- Classification, not an allowlist: `PERSON_OWNED_FAILURE_CODES` plus
task-level waits (pending approval, pending question, paused or
unavailable agent, paused company, review in progress, a Stop that a
person asked for, a Blocked status that another decision owns). Every
other code, including unknown future codes, falls back.
- Idempotent per source run. The task row lock serializes callers on
every server; the recorded `freshContinuation.sourceRunId` makes later
calls a no-op.
- Isolation, never waiting on proof. A local process still running on
this host is settled by its own owner first. If the old local writer is
proven gone, the task workspace is kept. Otherwise the continuation gets
a new isolated workspace (`fresh_isolated_workspace`), so two runs never
write one directory. The decision records the stop proof honestly
(`providerStopProof: isolated`).
- Claim admission no longer waits on the cleanup of a predecessor that a
fresh continuation replaced (`heartbeat/queue.ts`, predecessor loop).
Without this, the successor stayed queued behind the unreachable lease,
which is the staging failure itself.
- Only reconciliation holds that this decision owns are settled.
Workspace restore holds that workspace recovery still owns, and resolved
history, are untouched.
- `continuation_user_authorization_missing` is the system's failure (a
Retry's authorization was not carried forward), not the person's.
- Safety: the old run keeps its fencing. When it still owns its remote
workspace epoch, the epoch is abandoned
(`abandonUnverifiedRemoteWorkspaceEpoch`), so the old writer can never
publish and the next run restores the last committed save into a new
root on a new allocation. The unverified files stay in the abandoned
root, unapplied. The native coordinator records `replacementDenied:
fresh_continuation`, so no exact replay also starts.
- The replaced run loses its write authority. The fallback records
`freshContinuationSupersededAt` on the old run. The agent-JWT middleware
(`loadAgentRunAuthentication` reads one more field in its existing row
read) and `assertAgentRunWriteAllowed` refuse every write of the old
token with `agent_run_superseded`. Before this change, a failed or
interrupted run's token could still write. So an agent that is still
alive in an unreachable sandbox cannot act on the task next to its
continuation.
- The opening context (`recoveryOutcomes` in the execution continuation)
says that the previous attempt was cut off and lists the actions with
unverified outcomes, so the agent checks before it repeats a side
effect. No copy asks a person to inspect or repair a sandbox, workspace
or run. The hold text written at run end no longer says "Inspect the
original failure and reconcile".
- Delivery reuses the existing path: the settled hold gets an
`executionReconciliation` decision and `continuationDelivery: pending`,
and `deliverReconciledExecutions` starts the run (fresh session,
`retryOfRunId` = source).
- Budget per task, not per restart: at most 6 fresh continuations per
task per rolling hour (the server-restart lane's cap). At the cap the
hold stays with the system and is checked again when the window has
room. The failure allowance is never charged.
- Call sites:
- `settleUnrecoverableExecutions` (execution-recovery-resolution.ts):
every hold whose exact recovery is ruled out now goes through the
fallback before it would block the task. Interrupted preparations keep
their own replay lane first.
- `continueHeldExecutions` (same worker): picks up holds saved by older
releases (settled-but-blocked no-replay holds, finished or exhausted
saved-work recoveries) for the task's latest unfinished run.
- Wake admission (heartbeat/queue.ts): an ordinary system continuation
wake (sweep, monitor, timer) that a hold would defer now calls the
fallback in the same transaction and becomes the fresh continuation
itself. This is the path the idle-task sweep uses. A person's Retry,
message or queued send keeps its exact admission; the disposition worker
settles the hold, and the saved input then continues.
- `executionControlSchedule.disposition` wakes the worker for held tasks
and for the next due recheck.

### Human gate versus system uncertainty

One documented rule (`isHumanGate` plus `personOwnedTaskWait`) decides
what the fallback never touches. The rule goes by cause, never by owner
type:
- Human gates: a pending approval, a pending question, a user-owned
hold, an owner-gone attestation, a recorded explicit continuation, an
operator reconciliation, a person's Stop.
- System uncertainty: a no-replay record ("outcome unknown after an
interruption"), a hold that is `board`-owned only by default, an
unverifiable controller process space or host.

For uncertainty the fallback starts a fresh run, never a replay. The
interrupted turn is not re-sent. No tool call or write is re-executed.
Governed actions go through their approval gate again. The earlier
`automaticRecovery` decision is kept on the hold as
`automaticRecoveryHistory`, beside the attributed
`freshContinuationSettlement` and an activity entry.

### Restart recovery refused for an unverifiable controller

The case is `controller_process_space_unverified`, its remote variant,
or `controller_host_unrecorded` / `controller_host_unprovable`. Its
lease has lapsed; a live, renewing lease is still waited for (server
overlap). After `NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS` (2 minutes) the
run ends (failed, `native_restart_recovery_blocked`). That frees the
task's execution lock and the agent's slot, and a system-owned hold lets
the fallback continue the task. Recording the stops is optional. The "a
board operator must record both stops" text is replaced.

Board/operator attention sites in native restart recovery:
- `surfaceNativeControllerNamespaceAttention`: copy replaced; the run
now ends after the bounded wait.
- `requestNativeControllerStopConfirmation`
(native-controller-host-recovery.ts): copy says confirmation is
optional; the same bounded end applies.
- The blocked restart-recovery hold (`nextAction` "Inspect the preserved
checkpoint and reconcile…"): copy replaced; the fallback settles the
hold by cause.
- `owner-gone-attestation.ts` (generic resource records from older
releases): unchanged. It is not a run-recovery path, and it is listed as
a follow-up.

### Every wait and its bound

| Wait | Owner that rechecks | Worst-case total before the fresh
continuation |
| --- | --- | --- |
| `exact_recovery_pending` (execution settling, native finalizer,
controller lease, interrupted-preparation lane, saved-work recovery,
restore hold, retained-source fallback, live local process) |
Disposition worker, woken at each deadline; rechecks at 1, 2, 3 and 4
minutes, then every 4 | About 10 minutes from the first pending check.
Up to 15 while the owner shows progress (a new event, output, lease
renewal or coordinator update since the last check). |
| Saved-work (workspace) recovery that is `scheduled`,
`awaiting_evidence`, or `running` without progress | Held-run scan, now
selected and scheduled like the other holds | About 10 minutes; 15 at
most |
| Saved-work recovery that is `running` and advancing (revision,
attempt, state or hold update since the last check) | Held-run scan |
Its own 30-minute budget |
| Unverifiable lapsed controller in restart recovery | Restart recovery
(`retryAt`) | 2 minutes, then the run ends and the fallback continues it
within one disposition pass |
| Hourly cap (`backoff`) | Disposition worker, woken at the deadline |
Until the rolling hour frees a slot, at most 1 hour. This is the
deliberate restart-lane bound, not a stuck state. |
| Person-owned or dependency wait | The person's own action wakes the
task; rechecked hourly | Not bounded: it is the person's |

Restart-recovery classifications (`claimNativeRestartRecoveries`, which
`recoverNativeRunsAfterRestart` calls):

| Classification | Who rechecks, when | Bound |
| --- | --- | --- |
| Claims: `reattach_existing_runner`, `reattach_remote_runner`,
`resume_dead_runner`, `bootstrap_incomplete` | The claiming server
resumes exactly | Exact resume; no fallback |
| `already_finalized` | Nobody; it is terminal | None needed |
| `blocked` | The run is terminal and its hold goes to the fallback
(disposition worker) | Fallback bounds above |
| `awaiting_evidence` / `current_controller_preparing` | This process's
own executor | Ends with that executor |
| `awaiting_evidence` / `recovery_rows_busy` | Restart dispatch after 1
second | Self-resolving: the row lock clears |
| `awaiting_evidence` / a live lease (`controller_lease_held`,
`controller_lease_grace`, `controller_still_alive`) | Restart dispatch
at the lease-renew interval | Never ended while the old controller
renews its lease |
| `awaiting_evidence` / unverifiable lapsed controller (process space or
host) | Restart dispatch every 2 minutes | Ends 2 minutes after the
lease lapsed |
| `awaiting_evidence` / any other reason (startup cleanup, finalizer or
cleanup owner, recovery failure, routine busy, future reasons) | Restart
dispatch at the owner's `retryAt`, or within 2 minutes when it has none
| Ends 2 minutes after the lease lapsed, or after 10 minutes, whichever
comes first. It never ends while a lease is live. |

Ending a run sets it failed (`native_restart_recovery_blocked`),
releases the task lock and the agent's slot, and records a system-owned
hold. The fallback then continues the task once, isolated.

No wait needs longer. Saved-work recovery and retained-source checks
keep their own attempt budgets. After 10 to 15 minutes the fallback
continues anyway in a new root, without the unverified files, which stay
recorded and unapplied.

### When the fallback preempts saved-work recovery

The unverified files are not destroyed. They stay where recovery found
them: in the abandoned remote epoch's root, or in the old local
workspace that the task is no longer bound to. They are recorded and
never applied. Settling the hold records the decision on it, so the
retained-workspace worker stops scheduling it (its query requires no
`executionReconciliation` unless recovery already succeeded). If an
attempt already in flight finishes later, it cannot publish into the
fresh run's workspace. For a remote workspace, the epoch it targets is
abandoned and the writer is cleared. For a local one, the task now
points at a new workspace. Whenever a wait is preempted, the
continuation is isolated: a fenced remote epoch or a new local
workspace.

### What people see

The task's interruption presentation shows "working" while the primary
fresh-continuation record is pending and has not started (coalesced
holds never count, and they follow the primary's final state), and a
timed wait with no Retry while a check is scheduled. The Retry banner
appears only when the system has given up or the cause is the person's.
Holds settled together with the primary record are `coalesced`; they no
longer show "closed or reassigned". A pending continuation is never a
second run: if a person's Retry or saved input started a run after it
was recorded, delivery links to that run instead. A task that finished
before delivery gets no run.

### Readers of what the fallback writes

- Recheck deadline (`freshContinuationCheck`):
- `unresolvedExecutionDispositions` and
`heldExecutionsAwaitingFreshContinuation` read it to skip work that is
not due.
- `nextFreshContinuationCheckAt` and
`executionControlSchedule.disposition` read it to schedule the worker.
- Tests: the cap deadline is scheduled, and a cancelled task goes quiet.
- Isolation decision:
- The task's workspace binding is cleared. Every path that starts the
next run (system continuation, saved input, Retry, sweep wake) resolves
the workspace from that binding at run start (heartbeat.ts
`issueContext.executionWorkspaceId`).
- `freshContinuationIsolation` in the run context is read by
`readFreshWorkspaceReadiness` for the shared-directory case.
- `freshContinuationSupersededAt` on the old run is read by the auth
middleware, `assertAgentRunWriteAllowed` and the claim gate.

## Verification

- `pnpm vitest run src/services/fresh-continuation-fallback.test.ts`
(server): every test passes (it covers the cases below).
- The continuation starts (leaves the queue) while the unreachable old
lease is `active` or `pending_cleanup`. This was red before the
claim-admission fix.
- Isolation: a writer on another host gets a new isolated workspace; a
process alive on this host waits for its owner; a proven-gone writer
keeps the workspace.
  - Owned holds only: a restore hold and resolved history are untouched.
  - The recheck schedule goes quiet when the task is cancelled.
- `continuation_user_authorization_missing` continues under the task's
responsible user.
- One test per top-five staging code (native and legacy hold shapes):
the task gets exactly one fresh successor run from saved state.
- `runner_harness_state_mismatch` (`resumed_not_suspended`,
`resumed_unavailable`) falls back.
  - An unknown code falls back.
- Two concurrent callers plus the disposition worker create one
successor.
- Old-saved/new-resumed: a settled-but-blocked hold from the previous
release, with its coordinator in `awaiting_runner_reattach`, continues
on the first pass.
- An ordinary system continuation wake is admitted as the fresh
continuation.
- Fencing: after the continuation is recorded, the old run's token is
refused for writes (`agent_run_superseded`) and the continuation's token
is not. The old run's remote workspace epoch is abandoned: the writer is
cleared, the location and provider root are cleared, and the epoch
advances. So the next preparation restores the last save into a new root
(`prepareRemoteWorkspace`: `sameAllocation` is false when the location
is null, server/src/services/remote-workspaces.ts:437-470).
- The opening context names the cut-off run and an unverified tool
action.
- Person-owned causes (pending approval, credential not shared, budget
stop) are not continued.
  - The hourly cap backs off without blocking, then continues.
- Red/green: with the wiring reverted to `master`, 9 of these tests fail
(the task stays Blocked with no successor).
- Affected suites pass locally: `execution-recovery-dead-controller`,
`completed-turn-cleanup-hold-settle`, `execution-control-work`,
`heartbeat-process-recovery`, `issue-recovery-actions`,
`native-safe-replacement`, `native-failed-workspace-recovery`,
`task-retry`, `runner-lost-mid-turn-continuation`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
wake-queue `postgres`, `agent-conversations`, `native-startup-retry`,
`governed-wait-runner-loss`, `native-preclaim-continuation`.
- `pnpm verify --explain` selects the full suite (the wake queue is
shared), so CI supplies full coverage.

Existing expectations changed because they asserted the "wait for a
person" behavior this PR replaces:
- `issue-monitor-scheduler.test.ts`: "explains held monitor dispatch
without replaying uncertain execution" is now "continues a held task in
a fresh run when its monitor is due". A due monitor on a task held by a
`process_lost` hold now starts one fresh run instead of deferring.
- `agent-run-authentication.test.ts`: the auth row also reads
`freshContinuationSupersededAt` (still under 512 bytes), and a
superseded run is refused.
- `assigned-work-invariant.test.ts` (#21139): "with the real wake queue,
a hold that defers the continuation leaves a violation…" is now "…a hold
reached by the sweep's wake becomes one pending fresh continuation". The
sweep's wake now yields one pending fresh continuation, and the next
sweep adds no attempt.
- `heartbeat-native-restart-suspension.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: awaiting-evidence
dispositions now carry a `retryAt`, because every wait is rechecked. A
lapsed unverifiable controller no longer waits forever.
- `native-remote-retry-admission.test.ts`: the blocked-hold text is now
"This run stopped before it finished. Your saved work is kept." The old
text asked people to reconcile.

Holdless legacy runs: in the 1,000 staging runs, 0 tasks end on
`controller_lease_lost` or `process_lost` (every one has a later run).
12 end on `server_shutdown_interrupted` with no retry scheduled (attempt
0). On staging now, 9 of those 12 sit `in_progress` with no run and no
hold, 1 is done and 2 have a live run. Those 9 belong to the idle-task
sweep (#21139).

## Risks

- Speed: nothing is added to the normal path before `turn.start`. The
fallback runs only on the disposition worker after exact recovery is
ruled out, or at wake admission when a hold already defers the wake. The
auth middleware reads one more JSON field in the row read it already
does.

- Behavior shift: tasks that today stay Blocked for a person after an
infrastructure failure now continue automatically. A side effect with an
unknown outcome is not replayed; the new run is told about it and must
check first.
- After deploy, the disposition worker continues tasks already blocked
by these codes, at most 25 per pass and at most 6 per task per hour. No
migration. No manual Retry is needed.
- The old executor is not proven stopped before the successor starts.
Fencing is: its API writes are refused, its workspace epoch is
abandoned, the native `replacementDenied` marker is set, and the
successor has a separate run identity.

Known gaps (follow-ups):
- No quarantine attachment: unverified files stay unapplied in the
abandoned workspace root (recorded on the workspace in
`remoteWorkspaceAbandonedEpochs`). They are not read into a task
attachment, because the sandbox is usually unreachable in these
failures.
- Agent Chat conversations are excluded at wake admission. They keep
their owner-bound continuation.
- Under a `reuse_by_environment` lease policy the new allocation can be
on the same provider sandbox. It always uses a new root and a new epoch,
and the old run cannot publish or write.
- Tasks left `in_progress` with no run and no hold (for example after
`server_shutdown_interrupted` with no retry scheduled) belong to the
idle-task sweep. That sweep's wake reaches this fallback when a hold
exists.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:34:34 -06:00
DottaandClaude Opus 5.5 34f71f7d57 fix(server): defer a run that meets a busy agent directory instead of failing it (#20829)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run of an agent with managed agent files gets a private copy of
the agent directory. Remote copies serialize their prepare, checkpoint,
collect, and release operations through one Postgres advisory lock per
agent, taken with `pg_try_advisory_xact_lock`
> - When the lock is busy, the operation fails at once with "Agent
directory operation is already active"
> - On staging, a deploy restarted the controller while three runs of
the same agent were active. When one run resumed, another run of that
agent was checkpointing its directory. The resume failed, used its last
attempt, and the task became blocked with "Retry or send a message"
> - A run must not fail because another run of the same agent is briefly
using the agent directory
> - This pull request keeps the fail-fast lock and, when it is busy
during preparation, puts the same native run back on the existing
same-run retry schedule a few seconds later
> - The benefit is that lock contention delays a run by seconds instead
of ending it as failed

## Linked Issues or Issue Description

**What happened?**
A task's run failed with `native_continuation_requires_reconciliation`
and the error "Agent directory operation is already active". The task
was blocked and asked the user to retry. The same error ended at least
two other runs of the same agent within one day.

**Steps to reproduce**
1. Use an agent with managed agent files in a remote sandbox.
2. Let one run of the agent checkpoint or collect its agent directory.
3. At the same time, let another run of the same agent prepare (for
example, a same-run resume after a controller restart).
4. The second run fails immediately with "Agent directory operation is
already active".

**Expected behavior**
The second run waits until the first operation is done, then continues.
It does not end as failed.

## What Changed

- `server/src/services/agent-directory-working-copies.ts`: The busy lock
now throws `AgentDirectoryOperationBusyError`, still a 409 with the same
message. Lock behavior is unchanged from `master`: a fail-fast
`pg_try_advisory_xact_lock` held for the life of the holder's
transaction.
- `server/src/services/heartbeat/agent-directory-busy.ts`:
`prepareWithAgentDirectoryDeferral` wraps only instruction preparation
and turns the busy error into `AgentDirectoryPreparationDeferral`
("Another run of this agent is still using its files"). A busy lock met
while collecting files after a turn is not wrapped and keeps its
existing path.
- **Before native admission** (a brand-new run, such as the first run of
a task): `server/src/services/heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` cancels the run as an expected
control-plane wait (`providerWorkStarted: false`) and schedules an
`agent_directory_busy` retry 5–15 seconds later. The task keeps its
execution lock on the retry. The lane spends no failure retries
(`execution-recovery-attempt.ts`), its count is the retry run's
`scheduledRetryAttempt`, and it stops after 30 attempts.
`pre-dispatch-execution-evidence.ts` classifies this wait like the
existing workspace and AI-connection waits, so retry scheduling does not
ask for reconciliation.
- **A resumed native run**:
`server/src/services/native-runtime/native-controller-recovery.ts`
`deferNativeRunForBusyAgentDirectory` keeps the same run on the existing
same-run retry schedule without spending an attempt. It uses the same
guards as `deferNativeControllerRecovery`: it locks the task, requires
the task's execution run to be this run, requires the current lease
owner, and refuses after a Stop, a startup cancellation or a pending
native cancellation, so those settle normally. Its count is stored in
the run's `contextSnapshot`, which a resume claim does not clear, and it
stops after 30.
- `server/src/services/heartbeat.ts`: routes a preparation deferral to
the same-run path or, before admission, to the scheduled-retry path.
After the bound it follows the normal failure path with the plain
message.
- No schema or migration. Postgres releases a dead holder's advisory
lock with its session. The staging evidence showed a live holder
(another run's checkpoint), not a stale one.
- `doc/DEVELOPING.md`: Documented both paths and the bound.

Earlier revisions of this PR waited on the lock and added idle-session
expiry, keepalive, ownership fences, fenced remote removal and attempt
slots. This revision removes all of them.

### How the directory-wait retry keeps the original wake

The retry no longer replaces the wake. `scheduleBoundedRetryForRun`
takes `preserveWakeContext` for this wait: the retry keeps the original
`wakeReason`, `retryReason` and the whole wake context (interaction,
comment, continuation and admission fields), and its wake request keeps
the original reason. The wait is recorded separately as
`preparationWait: { kind: "agent_directory_busy", attempt, nextAttemptAt
}`, next to the carried `executionRetryAccounting`. Gates that key on
the wake reason therefore treat the retry exactly as they treated the
original wake.

| # | Site | Treatment for the directory wait | Why |
| --- | --- | --- | --- |
| 1 | `heartbeat/retries.ts` `scheduleBoundedRetryForRun` | Changed:
`preserveWakeContext` keeps the original wake context and reason and
adds `preparationWait` | Root cause: the retry overwrote the wake reason
|
| 2 | `run-dispatch/domain/wake-context.ts`
`isResolvedInteractionContinuationWakeContext` /
`PREPARATION_WAIT_RETRY_REASONS` | Special case removed | The original
wake context is kept, so the continuation is classified from it |
| 3 | `heartbeat/retries.ts` `interactionRunnerWait` | Special case
removed | The interaction fields are kept in the context |
| 4 | `run-dispatch/adapters/postgres.ts` scheduled-retry dependency
facts | Changed: a preparation-wait retry whose original wake was
admitted as a dependency-blocked interaction
(`dependencyBlockedInteraction`, recorded at wake time) is admitted the
same way | Without the wait the original wake would have run; this gate
is only met because of the wait |
| 5 | `run-dispatch/domain/wake-context.ts`
`isNonAssigneeWorkspaceBusyRetry` + `heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` | Kept:
`preparationWaitDeferredWhileAssignee`, derived only from the recorded
admission authority | The scheduled-retry ownership gate has no
wake-reason path; the workspace and AI waits use the same recorded flag
|
| 6 | `heartbeat/retries.ts` explicit user continuation admission |
Kept: admits the `agent_directory_busy` scheduling reason | The
successor needs its own authorization receipt |
| 7 | `execution-recovery-attempt.ts` uncharged lane and
`agentDirectoryWaits` | Kept | Cap and accounting |
| 8 | `pre-dispatch-execution-evidence.ts` (`isPreDispatchResourceWait`,
`isAgentDirectoryPreparationWait`), used by
`legacy-execution-recovery.ts`, `cancelled-native-startup.ts`,
`explicit-native-continuation.ts` | Kept | Describe the cancelled
pre-provider parent run, not the retry's wake |
| 9 | `explicit-native-continuation.ts`
`undeliveredLegacyUserCommentIds`, `run-dispatch/adapters/postgres.ts`
`unstartedUserTurnOrigin` | Kept | The cancelled parent did not consume
the user's comment |
| 10 | `execution-projection.ts` label and saved accounting,
`task-interruption-presentation.ts`, UI toast and chat suppression |
Kept | Presentation of the wait |
| 11 | `run-dispatch/domain/policy.ts` AI-wait lock requirement;
AI-subscription lock recheck; AI wait detection
(`execution-configuration.ts`, `provider-limit-wait.ts`, `heartbeat.ts`,
`ai-connections.ts`, `ai-connection-runtime.ts`, `local-ai-login.ts`) |
Not applicable | AI-subscription specific |
| 12 | `workspace-busy-wait.ts`, `environment-selection.ts`,
`workspace-preparation.ts`, UI workspace-holder banners | Not applicable
| Workspace holder parking; the directory wait is a timed retry |
| 13 | `cancelled-native-startup.ts` paused workspace-retry repair;
`failureRetriesBefore*Wait` | Not applicable | Historical workspace and
AI records only; every directory wait saves its ledger |
| 14 | `failedRunRetryBudgetSpent`; storage-wait exhaustion escalation;
`status-relevance.ts` `isAdmissionWait` | Not applicable | A directory
wait never fails the run, stops at its own bound, and its run has
started |

## Verification

- Original wake kinds after a directory wait
(`heartbeat-agent-directory-busy.test.ts`, real heartbeat): an
assignment runs; a plain user comment on a dependency-blocked task is
answered with `dependencyBlockedInteraction`; an accepted interaction
continuation keeps its parked-task and reassignment gates; a user
message continuation runs with its comment; an accepted non-assignee
mention runs; and a reassignment during preparation does not let the
previous assignee restart. Each retry keeps the original `wakeReason`
and records `preparationWait`. With `preserveWakeContext` off, five of
these fail. Restart recovery keeps its attempt credit
(`native-runner-restart-recovery.integration.test.ts`).

- Non-assignee wake: an accepted mention for an agent other than the
assignee meets a busy directory; its retry records
`preparationWaitDeferredWhileAssignee: false`, survives promotion, runs
and succeeds once the directory is free. Without the shared clause in
`isNonAssigneeWorkspaceBusyRetry`, promotion cancels it.

- Interaction continuations: `heartbeat-agent-directory-busy.test.ts`
runs an accepted interaction continuation into a busy directory. The
retry keeps `mutation`, `interactionId`, `interactionKind` and
`interactionStatus` and is still classified as a continuation. If the
task is moved to todo, or reassigned, while the retry waits, the retry
does not run and the task stays parked or with its new owner. With the
old explicit storage/Runner lists, both cases fail.

- Red/green, new run:
`server/src/__tests__/heartbeat-agent-directory-busy.test.ts` runs a
real heartbeat for a new assignment whose first instruction preparation
meets a busy lock. Without the preparation wrapper, the run ends failed
(red). With it, the first run is cancelled as a wait, an
`agent_directory_busy` retry is scheduled with the task still locked to
it, the retry runs and succeeds, and no run of the task is failed
(green).
- Red/green, same-run: with the deferral as a no-op, the same-run
deferral tests failed (`expected false to be true`); they now pass.
- `native-controller-recovery.integration.test.ts` covers repeated
claim-and-defer cycles of a resumed run: each cycle keeps the attempt
count, cannot be claimed before the delay, stores the count on the run,
and the fourth deferral past a bound of 3 is refused. It also covers
Stop after the busy error, a foreign lease owner and a moved task (each
refused, nothing changed), a run before admission, and that a raw busy
error (as from collection) is never deferred.
- Focused suites: 232/232 passed (`agent-directory-working-copies`,
`instruction-working-copy`, `heartbeat-agent-directory-busy`,
`native-controller-recovery.integration`, `heartbeat-retry-scheduling`).
`tsc --noEmit` for `server` reports no errors in the changed files.
- Staging evidence (read-only): the failed run had a graceful restart at
07:25Z, then same-run resumes. Its third attempt failed at 07:31:08Z.
Another run of the same agent recorded a warm checkpoint receipt at
07:31:37Z. A second run of the same agent failed the same way at
07:59:46Z.

## Risks

- A deferred run waits 5 to 15 seconds per deferral, up to 30 times. If
the holder is still busy after that, the run fails as before with a
plain message.
- Before admission, deferral cancels the run and creates a retry run,
like the existing workspace-busy wait.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:56:45 -06:00
DottaandClaude 805531cc71 fix(native): never leave a cleanup-gated task without Retry or an owner (#20639)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs close their provider session at the end of each attempt.
When close cannot prove that the runner stopped, the runner keeps a
cleanup gate (`native_session_cleanup_quarantined`) in server memory
> - #20572 hid Retry for every gated run and said "Paperclip finishes
the cleanup automatically". Only a remote environment release settles
the gate. A local gate that needs an operator never settles, and a
server restart loses it
> - A restart-chaos wave on master found two tasks that rested blocked
with no Retry, no scheduled retry and no recovery owner. A person could
not continue them
> - This pull request makes the run's own recovery retire the gate when
the host proves the old processes stopped. It also offers Retry whenever
nothing settles the gate, and it makes that Retry work
> - The benefit is that a cleanup-gated task either continues by itself
or shows a Retry that starts. It never rests without both

## Linked Issues or Issue Description

Refs #20572, #20578

**What happened?**
Restart-chaos wave `w-20261110T185144-s9107` (local, claude, fault TM2:
the server is paused with SIGSTOP for about 33 s). Tasks t04 and t05
completed their provider turn. The runner checkpoint owner lapsed while
the server was paused (`native_runner_checkpoint_owner_lost`). The
session close then failed with `native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`. The provider had
drained and no tool call was pending. The runner kept an
operator-required cleanup gate. Thirty seconds later the run's own
same-run recovery verified that the runner process and group had stopped
(`native.local_process_stopped`). The gate still refused the recovery,
and the run ended `failed` with `native_session_cleanup_quarantined`.
The task showed `canRetry: false` and a cleanup wait until the deadline.
Nothing owned that wait.

**Expected behavior**
When the old runner and provider processes are proven stopped, the run's
own recovery continues. When nothing settles the gate, the task offers a
Retry that starts.

**Steps to reproduce**
1. Run restart-chaos L7: `--env local --adapters codex,claude
--workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1
--concurrency 3 --tasks 6 --seed 9107`.
2. Observe a claude TM2 task rest `blocked` with `canRetry=false`,
`retryWait=true` and no scheduled retry.

**Paperclip version or commit**
master at ed3ffc3bc7.

## What Changed

- `native-session-executor.ts`: add `provenStoppedOwnNativeSession`. It
returns proof only for a local run with a pid-bound
`native.local_process_stopped` record (no later launch) and with every
recorded provider process gone. An ACPX session open names no provider
PID; its sidecar counts only when it shares the runner's process group,
which the stop record proves empty.
- `native-session-executor.ts`: add
`retireStoppedOwnSessionCleanupGate`. With that proof, it retires the
run's own operator-required gate through the existing
`completeTerminatedLocalNativeSessionCleanup` boundary and records
`native.session_cleanup_gate_retired`.
- `heartbeat.ts`: the native recovery preamble calls it right after it
records the local process stop, before admission. The recovery decision
itself does not change.
- `explicit-native-continuation.ts`: Retry or a new message accepts the
run's own stop proof when the retained-state inventory proof does not
apply. It still starts a fresh session, and it never resumes the old
turn.
- `task-interruption-presentation.ts`: hide Retry for a gated run only
while a live owner settles the gate: an unreleased remote environment,
or a scheduled close in this process. Otherwise offer Retry. The same
rule applies to a Retry held on `cleanup_quarantined`.
- `doc/execution-semantics.md`: describe the new rule.

## Verification

- Red before the fix, green after (5 tests failed on master sources):
- `server/src/services/task-interruption-presentation.test.ts`: "offers
Retry when nothing settles the gate by itself", "holds a Retry on a
cleanup gate only while an owner settles it", and an invariant test: a
failed presentation without Retry must name a wait, and a cleanup wait
needs a settling owner.
- `server/src/__tests__/task-interruption-query.test.ts` (real
Postgres): an unreleased remote lease hides Retry; the released lease
offers it.
- `server/src/services/quarantined-task-retry.test.ts`: "starts a task
Retry for a run's own quarantine proven stopped by its own process
record" (stopped, and provider alive stays held).
- New: "retires a run's own failed-close gate once its runner and
providers are proven stopped" for codex PIDs, an ACPX sidecar in the
runner's group, a sidecar with its own group, a live provider, and no
stop record.
- `npx vitest run` of `quarantined-task-retry`,
`task-interruption-presentation`, `task-interruption-query` and
`explicit-native-continuation`: 575 passed.
- `npx tsc --noEmit -p server` passes.
- Live: LIVE_PROOF_PLACEHOLDER

## Risks

- The gate retirement removes process ownership only. It runs only for a
local run whose exact runner and group and every recorded provider
process are proven gone. A provider without a recorded process identity
keeps the gate.
- Retry on a gated run still runs every existing stop check. If the stop
cannot be proven, the Retry waits and the task keeps a Retry, instead of
a wait that nobody settles.
- No persisted shape changes. Runs saved by earlier releases use the
same fields.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 15:20:42 -06:00
DottaandClaude c37eacbee3 fix(native): Retry re-finalizes a saved result that ran out of finalization retries (#20578)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save an accepted result. Then a finalizer applies that
result to the task.
> - Finalization has its own budget of 3 failed attempts (#20555). When
the budget runs out, the coordinator goes to `terminal_failure` with
`native_finalization_retry_exhausted`.
> - For a provider-succeeded run, the task then rests `in_review` with a
board recovery card. The task shows "resolved" and offers no Retry. The
user is stuck.
> - The ordinary Retry is also wrong here: it starts a new provider run,
which repeats work the agent already finished.
> - This pull request adds two things. First, contention failures on
such a result get two longer automatic delays before the board gets the
task. Second, when the state is still exhausted, the task offers Retry,
and that Retry applies the same saved result again.
> - The benefit: fewer tasks need a Retry, and when one does, Retry
works and does no duplicate provider work.

## Linked Issues or Issue Description

Refs #20555 (it names this gap under Risks).

**What happened?**
In restart-chaos wave `w-20261110T044338-s20385`, task t03 ended with a
succeeded run, an accepted result, and a coordinator in
`terminal_failure` / `native_finalization_retry_exhausted`. The issue
stayed `in_review` with a board-owned `active_run_watchdog` card and
`canRetry=false`. #20555 stopped the early exhaustion. A real exhaustion
still leaves the same stuck state. Coordinators that older builds
already left in this state are stuck the same way.

**Expected behavior**
Transient finalization failures recover by themselves when that is safe.
When they do not, the task offers Retry. That Retry re-finalizes the
saved result. It does not start a provider run and does not charge a
provider attempt.

**Steps to reproduce**
1. Let a native run save an accepted result and end `succeeded`.
2. Make each finalization of that result fail until the budget of 3 runs
out.
3. Before this fix, the task presentation is `resolved` with
`canRetry=false`, and the board card has no action that applies the
result. The new test
`server/src/__tests__/native-exhausted-finalization-retry.test.ts`
reproduces this.

**Paperclip version or commit**
master at c58356003f (after #20555).

## What Changed

- `native-run-finalizer.ts` (`recordRetryableFailure`): when the budget
runs out on an accepted result of a provider-succeeded run, and the
cause is contention (`status_cas_exhausted` or
`side_effect_planning_failed`), the coordinator stays
`retryable_failure`. It tries again after 2 minutes, then after 10
minutes, each with a fresh budget. Each delay is recorded once in
`recoveryHistory` (`automatic_finalization_backoff`). After both delays,
the old terminal path runs. Deterministic causes (for example
`native_finalization_invalid`) and workspace-scope failures are
unchanged. The existing deadline worker (`native-result-commit-work`)
runs the retry, so no new timer or poll is added.
- New `native-runtime/exhausted-finalization-retry.ts`:
- `exhaustedNativeFinalizationCondition()`: one SQL predicate for
"succeeded native run, accepted result, coordinator `terminal_failure`
with `native_finalization_retry_exhausted`, no lease, no successor".
- `rearmExhaustedNativeFinalization()`: under the task lock, reopens the
same coordinator (`arbitrating` or `workspace_finalizing`), records
`exhausted_finalization_retry` in `recoveryHistory`, resolves the board
card (`handed_back`), and resets `finalizationFailureCount` to 0. It
sets a 30 s fallback `nextAttemptAt`, so the deadline worker finishes
the job if the process stops first.
- `refinalizeSavedNativeResult()`: runs `finalizeNativeRun` for the same
run with `preserveProviderAttempt: true`. A failure goes through the
finalizer's normal bounded budget.
- `task-interruption-presentation.ts`: the run read adds
`nativeFinalizationRetry` from that predicate. A succeeded run with this
flag shows `state: "failed"`, `canRetry: true`. It also adds
`nativeFinalizationRetryPending` (`savedResultRetryPendingCondition()`:
coordinator not committed or terminal, with an
`exhausted_finalization_retry` entry). While it is true the task shows
`state: "working"` with `recoveryStep: "finishing_recovery"`, so the
user sees the Retry took effect. Done or cancelled tasks, user gates,
and ordinary runs are unchanged.
- `task-retry.ts` (`admitTaskRetry`): for a succeeded native run in the
exhausted state, Retry re-arms the coordinator instead of creating a
provider retry. It writes a `coalesced` `task_retry` receipt bound to
the same run with strategy `finalize_result` (an existing strategy
value), joins any unadmitted Retry intent to it, and logs
`issue.retry_requested` with `strategy: "finalize_result"`. The
idempotency key still returns the same receipt on replay. A later click
with a new key, after an earlier saved-result Retry of the same run
(still finalizing, or applied), joins that Retry with another coalesced
`finalize_result` receipt and starts no provider work.
- `routes/issues.ts` (`POST /issues/:id/retry`): after the receipt
commits, it starts `refinalizeSavedNativeResult` for that run instead of
the provider Retry dispatch.
- `doc/execution-semantics.md`: documents the automatic delays and the
re-finalizing Retry.

## Verification

- `cd server && npx vitest run
src/__tests__/native-exhausted-finalization-retry.test.ts` (4 tests):
- Exhausted state reports `state: "failed"`, `canRetry: true`. Retry
re-arms the coordinator, writes a `finalize_result` receipt and an
`issue.retry_requested` activity row, and replays idempotently. A second
click with a new key while pending, and a third after commit, join the
same Retry; no provider Retry wake exists. While pending, the task shows
`working` / `finishing_recovery`. Re-finalization commits, and the task
shows resolved. The run count for the task stays 1, the coordinator
`attempt` is unchanged, the run stays `succeeded`, and no board card
stays active.
- Old-shape row (no `finalizationFailureCount`, `attempt: 3`, already
terminal): the same Retry works with no migration.
- Re-finalization that fails again: the coordinator goes to
`retryable_failure` with a fresh count of 1, then to `terminal_failure`
after the budget, and the task again shows `canRetry: true`. Nothing is
lost silently.
- Contention failures (failpoint `status_projection`): two automatic
delays (2 min, 10 min) with the task status unchanged, then the board
Retry, which commits with no new run.
- Red/green: with only the new module and test applied (finalizer,
presentation, task Retry, and route reverted), all 4 tests fail
(`canRetry` false; no automatic delay).
- `npx vitest run src/services/task-interruption-presentation.test.ts`
adds unit cases: the flag offers Retry; ordinary succeeded runs stay
resolved; ordinary failed runs keep `canRetry: true`; done tasks stay
resolved.
- Also passing: `native-finalization-retry-budget`,
`native-finalization-recovery`, `native-run-finalizer-telemetry`,
`task-interruption-query`, `interrupted-preparation-restore-retry`,
`native-result-commit-work`, `workspace-recovery-admission`,
`workspace-recovery-preparation-gates`,
`deferred-execution-receipt-work`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Not run: the full local suite (`pnpm verify` selects about 2,450 files
for these shared modules; CI covers it), and no live restart-chaos wave.

## Risks

- Previous-release story: coordinators saved by older builds in
`terminal_failure` / `native_finalization_retry_exhausted` qualify by
phase and code. The count is optional. Nothing is migrated. Older rows
do not get the automatic delays; they get the working Retry.
- Automatic delays add at most two extra finalization rounds of the same
saved result, only for contention causes, only for provider-succeeded
accepted results. During the delays the task keeps its prior status, as
it does during the normal 30 s retries. Supersession by a newer run is
still checked first on every attempt.
- The re-finalizing Retry applies the agent's saved result after a
person may have looked at the task in review. It uses the normal status
arbiter and CAS. Done and cancelled tasks, other assignees, and newer
runs refuse it.
- No change before `turn.start`: all new work is on the finalization,
presentation, and Retry paths.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code
with tool use (shell, code editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 11:48:47 -06:00
DottaandClaude Opus 5.5 be0d100a8b fix(native): settle cleanup-gate refusals after a completed turn without a dead-end Retry (#20572)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute a provider turn, then close the provider session
and prove it stopped
> - When close cannot prove that the old runner, provider and tools
stopped, the runner holds a cleanup gate
(`native_session_cleanup_quarantined`) that refuses the next session on
that task
> - A run finished its provider turn and saved its work. A server
restart had left one tool callback without a saved result, so close
could not prove the drain, and the gate refused the run's own same-run
recovery 30 seconds later
> - The task then showed "Couldn't finish this attempt" with a Retry
button, but the retry route refuses this error code with 409. The saved
error text told an operator to act
> - This pull request makes the task projection, the execution blocker,
the executor and the retry route agree, and replaces the operator text
with plain wording
> - The benefit is that finished work no longer looks like a failed
attempt, and the UI never offers a Retry that the server refuses

## Linked Issues or Issue Description

**What happened?**
A remote native run completed its provider turn and saved its comments
and documents. A server restart during the turn left one semantic tool
callback without a saved result. Session close then reported
`native_session_settlement_incomplete` (runner suspended, provider drain
unproven), and the runner kept an operator-required cleanup gate. The
same-run disposition recovery was refused at admission, and the run
ended `failed` with `native_session_cleanup_quarantined`.

**Expected behavior**
The task should show its real state (here, waiting on a child task). It
should not show a failure banner or a Retry that the server refuses. No
text should ask an administrator to act.

**Steps to reproduce**
1. Start a remote native run and restart the server while a tool
callback runs.
2. Let the resumed turn complete without a semantic result, so a
same-run recovery is scheduled.
3. Observe the refusal at admission, the "Couldn't finish this attempt"
banner, and a Retry that returns 409.

**Paperclip version or commit**
master at d0057ee384 (remote native runner, Daytona sandbox).

## What Changed

- Add `server/src/services/native-runtime/completed-turn-cleanup.ts`. It
holds one predicate for "a cleanup-gate refusal after this run's own
provider turn completed" and the plain user wording. It reads only the
saved `errorCode` and `runnerProfileJson.sessionCheckpoint`.
- `task-interruption-presentation.ts`: the projection never sets
`canRetry: true` for a cleanup-quarantined run, which matches
`routes/agents.ts` and `canRetryStoppedRun`. A quarantine after a
completed turn projects as `resolved` only while the task waits on
unresolved blocker tasks, so the task shows that wait. Without such a
wait, the missing task disposition stays visible. Other quarantined runs
show a plain settling message without Retry. The list query computes the
completed-turn fact in SQL.
- `native-session-executor.ts`: the refusal keeps its permanent, named
code, its hold and its ownership, and does not replay provider work. The
next action uses plain wording, and names the saved work after a
completed turn.
- `execution-blocker.ts`: show plain wording for this code instead of
the stored error text, also for rows that earlier releases saved.
- Runner error message and retry-route conflict message use plain
wording.
- `doc/execution-semantics.md`: document how the cleanup gate settles
and how the task shows it.

## Verification

- Red before the fix, green after:
- `server/src/services/task-interruption-presentation.test.ts` "native
cleanup quarantine presentation" (2 failed before: `canRetry: true`,
`state: failed`).
- `server/src/__tests__/task-interruption-query.test.ts` "reads a
quarantine saved after a completed turn as settled work" (real Postgres,
row in the previous release's saved shape; before: `state: failed,
canRetry: true`).
- `server/src/services/native-runtime/native-session-executor.test.ts`
"settles a cleanup refusal after a completed turn" (before: operator
next action and board ownership).
- `server/src/services/explicit-native-continuation.test.ts` "shows
plain wording for a cleanup refusal saved with the earlier operator
text" (before: stored operator text).
- The four files pass in full: 1851 tests.
- `tsc --noEmit` passes for `server` and `packages/paperclip-runner`.
- `pnpm verify` returns exit 2 (a global doc input changed), so required
CI supplies the broad coverage.
- No UI files changed, so token gates and UI tests do not apply.

## Risks

- Previous-release story: no persisted shape changes. Runs that earlier
releases quarantined carry forward. The new rules read existing fields,
so these runs show the new projection without a migration.
- The code stays permanent and named. No automatic provider replay is
added.
- A quarantine whose turn did not complete keeps its board-owned
recovery. Only its text and its Retry visibility change.
- The runner still keeps the in-process cleanup gate until environment
release proves the stop. Existing lease termination receipts
(`acknowledgeRemoteStop`) clear it. This PR does not change the runner
drain rule for a callback that a restart orphaned. That is a separate
follow-up.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:09:59 -06:00
DottaandClaude 82abb3ea04 fix(native): classify a Codex API-key unknown model as a model rejection (#20488)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner agents run on a provider (here Codex) with a model the
user saves on the agent
> - When the provider rejects that model, Paperclip should classify the
run as `native_provider_model_rejected`, ask the user to choose a model,
and let Retry finish the task
> - Paperclip recognized only the ChatGPT-account rejection (a 400 JSON
envelope). With API-key auth, Codex returns a different terminal error
for an unknown model
> - So the run failed as a generic `adapter_failed` ("Native session
failed"), and recovery opened a reconciliation incident ("Recovery
needed") for a turn that never did any work
> - This pull request recognizes the real API-key shape, bound to the
configured model, and names the rejected model on the task
> - The benefit is that the task asks for a model choice instead of a
reconciliation, and Retry completes the task after the user picks a
valid model

## Linked Issues or Issue Description

**What happened?**
The restart-chaos config fault CF3 `model_id` saves a Codex agent with
the model `chaos-missing-model-<nonce>`. The run stays "running" for
about 60 seconds, then fails with `errorCode: adapter_failed`, "Native
session failed". The task shows "Recovery needed", cause
`native_continuation_requires_reconciliation`. The grader fails as
`CF3/model_id:actionable_message` (waves `w-20261110T061914-s206-cf` t01
and `w-20261110T053831-s205-cf` t02).

Root cause: the wave runs Codex with an OpenAI API key. I drove the
bundled `codex app-server` (0.160.0) directly with an unknown model:
- With API-key auth, Codex retries the Responses API 404 ten times
(about 59 s), then fails the turn with `{"message":"unexpected status
404 Not Found: The model \`<id>\` does not exist or you do not have
access to it., url: https://api.openai.com/v1/responses, cf-ray: …,
request id:
req_…","codexErrorInfo":{"httpConnectionFailed":{"httpStatusCode":404}}}`.
- With ChatGPT auth, the turn fails in about 3 s with `message` =
`{"type":"error","status":400,"error":{"type":"invalid_request_error","message":"The
'<id>' model is not supported when using Codex with a ChatGPT
account.","code":"model_not_found"}}`. Paperclip already classified this
one.

`createNativeProviderFailureObservation` parsed only the JSON envelope,
and the runner's provider-failure category for an HTTP 404 is `unknown`.
The terminal therefore stayed unclassified.

**Expected behavior**
The run is `native_provider_model_rejected`, the task shows `userAction:
"choose_model"` and names the model, no reconciliation incident opens,
and Retry completes the task after the user picks a valid model.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs
tests/restart-chaos/config-wave.ts --faults CF3 --adapters codex
--references model_id --tasks 1 --concurrency 1 --restarts K2 --seed
206` on `master` (with `OPENAI_API_KEY` set).

**Paperclip version or commit**
`master` at 5ab98b0.

## What Changed

- `server/src/services/native-runtime/native-provider-failure.ts`: the
Codex observation recognizes the API-key terminal shape
(`codexErrorInfo` `httpConnectionFailed` or `responseStreamDisconnected`
with `httpStatusCode: 404`, and the message prefix for the configured
model only). It classifies it as `native_provider_model_rejected` with a
new closed diagnostic `NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC` (`codex`,
`model_not_found`, `404`, `api_key`). No provider text or model name is
stored in the diagnostic. A recoverable turn, another model's 404, or
another status is not classified.
- `readNativeModelRejectionDiagnostic` accepts both closed diagnostics,
so run failure diagnostics show it.
- `TaskInterruptionPresentation.rejectedModel` (shared type): with
`userAction: "choose_model"`, the task query carries the model id from
the run's native execution input. Only a
`native_provider_model_rejected` run discloses it.
- `TaskChatThread` names the model in the "Model unavailable" marker:
`The provider rejected the model "<id>".`
- The restart-chaos config wave reads `interruption.rejectedModel` as
task-level text. For `choose_model`, its repair step uses the Retry that
the "Model unavailable" marker offers (`POST /agents/{id}/wakeup` with
`reason: "retry_failed_run"`, the same request as
`agentsApi.retryFailedRun`), because a model rejection does not offer
the task-level Retry.

## Verification

- Red before, green after (the classification line disabled makes both
fail):
- `native-provider-failure.test.ts` — "classifies the API-key 404 for
the configured model as a model rejection (codex 0.160.0 shape)", with
the exact terminal error captured from `codex app-server` 0.160.0, plus
negative cases.
- `native-provider-failure.integration.test.ts` — new `api-key` scenario
through `executePaperclipNativeSession`, the persisted evidence reader
and `finalizeNativeRun`: the run ends `native_provider_model_rejected`,
the issue is `blocked` with that decision, and no issue recovery action
is created.
- `task-interruption-query.test.ts` — the real task query carries
`rejectedModel` only for a model rejection. `TaskChatThread.test.tsx` —
the marker names the model.
- `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter
@paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui
typecheck` pass.
- Live proof: config wave `w-20261110T090801-s206-cf` (CF3 `model_id`,
codex with an OpenAI API key, K2, seed 206) on commit 05e8938: **1/1
pass**. The run ends `native_provider_model_rejected`; the task rests as
`waiting_for_user` / `choose_model` with `rejectedModel:
"chaos-missing-model-472e1851"` and execution cause
`native_provider_model_rejected` (no reconciliation incident), rests
again after the K2 restart, and finishes 27 s after the model is
restored and the marker's Retry is used. On `master` the same wave
failed as `actionable_message` with "Recovery needed" /
`native_continuation_requires_reconciliation`
(`w-20261110T061914-s206-cf`).

## Risks

- Low risk. The match is bound to the configured model, a 404 status and
a non-recoverable turn, so other provider failures keep their current
behavior.
- The match depends on the Codex terminal text. If a later Codex release
changes it, the run falls back to the old generic failure, not to a
wrong classification.
- Claude (ACPX) unknown models are not changed here. Claude Code reports
an unknown model as a synthetic assistant message ("There's an issue
with the selected model …") rather than a provider error; this PR does
not change or live-verify that path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, a direct `codex
app-server` probe).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 03:11:10 -06:00
DottaandClaude a14ae9b5c9 fix(workspaces): name a failed workspace setup command on the task page (#20478)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before an agent starts, Paperclip prepares the task workspace and
runs the project's configured setup commands (for example the workspace
provision command)
> - When that command fails, the run fails as `setup_failed`, and the
task stops with a Retry
> - The task page shows only "Failed" and a generic recovery notice. The
real error (the command, its exit status and output) is visible only on
the run page
> - The user cannot see from the task which setting is broken or what to
change
> - This pull request records a bounded, named cause for a failed setup
command and shows it on the task page
> - The benefit is that the task names the broken setting and what to
change, and classification, retry and recovery stay the same

## Linked Issues or Issue Description

**What happened?**
The restart-chaos config fault wave CF1 sets a project
`workspaceStrategy.provisionCommand` to `bash
./scripts/provision-worktree.sh`, a script that does not exist. The run
fails as `setup_failed` with `Execution workspace provision command
"bash ./scripts/provision-worktree.sh" failed: bash:
./scripts/provision-worktree.sh: No such file or directory`. The task
page shows "Failed" and "Paperclip automatically retried continuation …
still has no live execution path". The grader fails the task as
`CF1/provision_command:message_only_in_run_detail`.

**Expected behavior**
The task page names the failed setup step, the configured command, its
exit status and a short tail of its output, and says what to change.
Retry behavior stays the same.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs
tests/restart-chaos/config-wave.ts --faults CF1 --adapters codex --tasks
2 --concurrency 2 --restarts K1,K2 --seed 204` on `master`.

**Paperclip version or commit**
`master` at 5ab98b0 (wave `w-20261110T043929-s204-cf`, tasks t01 and
t03).

## What Changed

- `WorkspaceCommandExitError` now carries the setup step
(`workspace_provision`, `workspace_seed`,
`workspace_runtime_provision`), the configured command and the command
output.
- New `server/src/services/workspace-setup-command-failure.ts`: it
builds a bounded, redacted `setupCommandFailure` record (step, command
up to 200 characters, exit code, last 300 characters of output). It also
recognizes the older message format, so runs recorded before this change
still get a name.
- `run-completion.ts` adds `setupCommandFailure` to the setup failure
`resultJson`. The error code (`setup_failed`) and all retry and recovery
fields stay the same.
- `TaskInterruptionPresentation.setupFailure` (shared type) is set for
`setup_failed` runs. It goes through the same per-run secret redaction
and user-name redaction as `workspaceFailure`.
- `ExecutionBlockerNotice` shows "Couldn’t set up the task workspace.
<command> failed with exit code N: <output>. Fix the command in the
project’s workspace settings, then retry." A scheduled transient retry
keeps its countdown and only adds the name.
- The restart-chaos config wave reads `interruption.setupFailure` as a
task-level text for its actionable-message check.

## Verification

- Red before, green after (the tests fail with the source changes
reverted):
- `server/src/__tests__/workspace-runtime.test.ts` — "names a failed
project provision command with its exit status and output (CF1)" runs a
real git worktree with a missing provision script (exit 127).
- `server/src/services/task-interruption-presentation.test.ts` — "setup
command failure presentation (CF1)": names the command, keeps
`state`/`canRetry`/`userAction` identical, recognizes the older message,
keeps a scheduled retry, bounds output and strips credentials from the
command.
- `ui/src/components/ExecutionBlockerNotice.test.tsx` and
`tests/restart-chaos/config-faults.test.ts` cover the notice text and
the harness check.
- `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter
@paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui
typecheck` pass.
- Live proof: config wave `w-20261110T083227-s204-cf` (CF1, codex, 2
tasks, K1 and K2, seed 204) on commit ae9a4b7: **2/2 pass**. Both tasks
show `The project's workspace provision command "bash
./scripts/provision-worktree.sh" failed with exit code 127: bash:
./scripts/provision-worktree.sh: No such file or directory` on the task,
rest again after the restart, and finish after the repair and Retry. On
`master` the same wave failed both tasks as `message_only_in_run_detail`
(`w-20261110T043929-s204-cf`).

## Risks

- Low risk. The change adds a presentation field. It does not change the
error code, retry, recovery or blocking behavior.
- The command and output are user-configured text. They are bounded,
passed through `redactSensitiveText`, the run's secret registry and
user-name redaction. If redaction fails, the field is omitted, like
`workspaceFailure`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:35:18 -06:00
DottaandClaude Opus 5.5 4a2eb920cf fix(storage): wait out a store outage on a run's critical path instead of failing it (#20299)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A cloud instance keeps run logs and evidence in object storage, so a
run touches the store as soon as it starts
> - In storage chaos wave `w-20261109T145628-s9902` (ST1: 75–176 s
windows where every S3 call returns 503), only 1 of 8 tasks finished. A
run starting or resuming during an outage failed within a second as
`adapter_failed`, generic recovery burned its attempts seconds apart,
the task was escalated to the board, and the user saw the raw S3 error
> - This pull request treats a transient storage error as an
infrastructure wait: the run is retried on a minute-scale schedule,
rapid recovery is suppressed, the board shows it as working, and the run
error says storage was unavailable
> - The benefit is that a store outage of several minutes delays work
instead of failing and stranding it

## Linked Issues or Issue Description

Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (master
`610b54f551`, `--storage s3`, ST1).

**Where the run fails.** At run start or resume, `run-output` `begin`
opens the run log: `getRunLogStore().begin` → object run log store →
object byte journal `claim` → S3 conditional PUT. The journal retries
only 3 times within about 300 ms, so a 503 escapes and fails the run as
`adapter_failed`. The async stack ends in the SDK middleware with no app
frame. `log_store` stays null on the failed run, which pins the claim.

**What followed.**
- `failRun` released the issue into generic immediate recovery. Recovery
retried 1–2 s apart inside the outage (t01: 14:58:33, :35, :59,
14:59:01), then stopped and opened a board-owned
`stranded_assigned_issue` action. These are the open recovery actions on
CHA-1, CHA-4, CHA-7 and CHA-8; a later Retry hit the outage again and
escalated the same way.
- The run error was the raw S3 text ("Injected storage outage").

**Slow store (ST2, `w-20261109T151946-s9903`).** With about 9.7 s of
added latency per call, the same claim hit the journal's 15 s
per-operation deadline (`AbortSignal.timeout(15_000)`). The run failed
after about 36 s with `AbortError: Request aborted` and burned 3 runs
the same way; 4 of 8 tasks needed a Retry.

t02/t03 (Retry parked with `process_identity_missing`) are a separate
native-recovery class and are handled in a follow-up.


**Daytona storage chaos ST1 (`w-20261109T161202-s9912`, 1/8):** with S3
returning 503 for 72 to 154 s at a time:
- Workspace restore failed at once (`remote_workspace_restore_failed:
step=snapshot_download; storage=provider_error`), three times in 40 s,
then `setup_failed`. It left an active recovery action behind.
- Runs reattached after a restart died within 2 s on the run-log claim,
then became reconciliation-required.
- Setup failed on other storage reads.
- 6 of 7 failures needed a UI Retry.

So the rule here is one class rule: a transient store error on a run's
critical path waits, it never fails the run.

## What Changed

- New `server/src/storage/storage-unavailable.ts`:
- `isStorageUnavailableError` reads the recorded storage failure
diagnostics first, then the SDK error shape. Unavailable means
429/500/502/503/504, the S3 throttling and timeout names, network codes
(`ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, `EPIPE`, `EAI_AGAIN`,
`ENOTFOUND`), or an `AbortError` whose cause is a `TimeoutError` (a
per-operation deadline). 403, 404 and 412 are not outages, and neither
is an abort caused by the run's own Stop.
- It defines the `storage_unavailable` code, the
`storage_unavailable_wait` retry reason, the retry policy (60 s × 10
attempts, about 10 minutes) and the user-facing message.
- `server/src/services/heartbeat/run-completion.ts`:
- `failRun` classifies such a failure as `storage_unavailable` with the
plain message.
- Unless native recovery owns the run, it schedules the storage wait
with `scheduleBoundedRetryForRun`, skips the interaction-continuation
retry, and releases the issue with `suppressImmediateRecovery`.
  - `failRunSetup` uses the same policy for this code.
- `server/src/services/heartbeat/retries.ts`: `storage_unavailable`
joins the transient infrastructure codes. It gets its own retry budget
and policy, like `runner_revision_mismatch`.
- `server/src/services/task-interruption-presentation.ts`:
`storage_unavailable_wait` is a resource wait, so the board shows the
task as working with no Retry while it waits.

- **One shared outage wait for the run's critical path**
(`server/src/storage/storage-unavailable.ts`):
- `retryThroughStorageOutage` retries a transient store failure with
backoff (2 s doubling to 30 s) until that storage operation's outage has
lasted 5 minutes, then gives the error back. The outermost wait of an
operation records its first failure, and nested and parallel waits
inside it share that start; no other request's success extends it.
Shared probes and shared reads run under their own scope, so one run's
Stop never cancels work another run waits on. A stop signal ends the
wait, and non-outage errors are never retried.
- Inside `waitOutStorageOutages`, every replayable S3 request waits
instead of failing: get, head, delete, buffered and file puts, and
multipart create, file parts and complete. A consumed upload stream is
never resent, and requests outside the scope keep failing fast.
- The scope covers run execution (`heartbeat.ts` `executeRun`: setup,
workspace restore, run-log claim and the run itself), plus
`loadWorkspaceSnapshotArchive` and `uploadWorkspaceSnapshot`, which
copy-back and recovery workers use too.
- The run-log claim retries as a whole, because the journal bounds each
operation with a 15 s deadline.
- Past the budget, the run takes the bounded `storage_unavailable` wait
below.
- The classifier follows the error's `cause` chain, so a restore failure
wrapping an outage counts. `failRunSetup` checks storage before restore
failures, so that case takes the storage wait instead of two quick
retries and an escalation (the R6 leak).
- `object-byte-journal.ts`: when its deadline expires, the journal
rethrows the recorded storage error instead of a bare `TimeoutError`.

## Verification

Red/green (each new test fails against master and passes with this PR):
- `run-completion.test.ts` "waits a minute and suppresses rapid recovery
when a run fails because storage is unavailable (storage chaos ST1)". An
S3-shaped 503 records `storage_unavailable` with the plain message (no
raw S3 text), schedules `STORAGE_UNAVAILABLE_RETRY`, releases with
`suppressImmediateRecovery: true`, and skips the continuation retry.
"keeps an ordinary adapter failure on the normal recovery path" guards
the default.
- `task-interruption-query.test.ts` "does not present an overdue
storage_unavailable_wait wait as a failed attempt": working, `canRetry:
false`.
- `storage-unavailable.test.ts`: which errors count as an outage and
which do not, including recorded diagnostics, a deadline-aborted call
(ST2), and a Stop-aborted call that must not count.

Static checks:
- `tsc --noEmit` passes for `server`.

Live verification: Phase 3 will rerun ST1 (seed 9902) with the MinIO
fault proxy after merge.

- `server/src/__tests__/storage-outage-wait.test.ts` runs against a
local S3 that answers 503 for 3 s:
- Inside the scope, a snapshot `getObject` and a run-log conditional PUT
succeed once the store is back.
  - Outside it, a request fails fast.
  - A stop ends the wait.
- Unit cases cover the budget, non-outage errors, and a
`RemoteWorkspaceRestoreError` wrapping an outage.
- `run-completion.test.ts`: a setup failure from a restore that wrapped
an outage takes the storage wait (red before).
- `object-byte-journal.test.ts`: a real `claim` whose head PUT hits its
deadline is classified as a storage outage (red before).

## Risks

- A storage outage now holds a task for up to about 10 minutes of
automatic waits instead of escalating within seconds. A longer outage
still ends in the existing board recovery.
- A run whose failure is misclassified as storage-unavailable waits
instead of using fast recovery. The classifier accepts only server-side
or network failures from the storage SDK, never 4xx answers.
- No schema change.

- A run can now wait until a storage outage has lasted 5 minutes before
it fails over to the bounded wait. It shows as working throughout, and
Stop ends the wait.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:15:48 -06:00
DottaandPaperclip 3cc1032f6a fix: deliver saved answers to dependency-blocked tasks (#20363)
## Thinking Path

> - Paperclip lets people direct agent work through tasks and structured
answers.
> - Dependencies stop task execution until prerequisite work finishes.
> - A person must still be able to answer or redirect the assignee of a
blocked task.
> - Ordinary comments already get a bounded answer/triage turn. A saved
question response has no comment ID, so the same gates reject it.
> - Later dispatch also requires an active task and tries to check it
out. That conflicts with bounded interaction on a blocked task.
> - This pull request verifies the stored human response at each gate
and keeps the turn in bounded interaction mode.
> - The assignee can receive the answer without starting dependency work
or removing its blockers.

## Linked Issues or Issue Description

Refs #19923. That change covers dependency waits and comment retries.
This change covers stored interaction responses that have no comment ID.

**What happened?**

A parent task waits on a parked child. The assignee asked a setup
question. After a person answers, the response stays queued. Sending
that saved response is refused as `issue_dependencies_blocked`. Ordinary
new messages can reach the same assignee.

Passing admission alone is insufficient. The dispatch status check
rejects a blocked task, and automatic checkout then tries to start work
whose prerequisites are unfinished.

**Expected behavior**

Deliver the stored human answer or confirmation to its requesting
assignee as a bounded answer/triage turn. Keep the task blocked and keep
its dependency relationships. Preserve current-owner, pause, approval,
budget, recovery, and execution gates.

**Steps to reproduce**

1. Assign a parent task and make a backlog child block it.
2. Save a question on the parent and answer it as a person.
3. Dispatch its resolved-interaction wake, or send the saved answer from
the task queue.
4. Observe that the answer does not reach the provider while an ordinary
comment can run in answer/triage mode.

**Paperclip version or commit**

Reproduced on master at `653ab8b18f76`. This branch is rebased onto
master at `0e3fdebea8`.

## What Changed

- Verify the stored response by company, task, requesting agent, status,
kind, human resolver, and resolution time. A context flag alone is
insufficient.
- Allow a verified response through dependency admission and claim in
the existing bounded interaction mode.
- Recheck that response during scheduled retry promotion and final
provider dispatch.
- Keep bounded responses from checking out the blocked task or setting
it to `in_progress`, including when the last blocker finishes during
preparation.
- Preserve continuation classification through workspace and AI-account
resource waits, so the final response and owner checks still run.
- Correct the task-case API summary so the existing `case for task`
discovery regression passes on the first search page. This fixes a
pre-existing failure exposed by full CI, without changing search ranking
or assertions.
- Ignore obsolete Retry notices after newer conversation work, without
acknowledging saved input or changing execution authority.
- Document the response contract. Add real PostgreSQL and
assembled-heartbeat regressions.

## Verification

- Final commit `ba4b337cf1bb0236b7c0a742665b222ee894ff51`: [required CI
passed, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/42706363120/attempts/2),
including typecheck, production build, server/chat tests, Runner checks,
and all 8 browser shards. PR status: 59 passing checks and 3 normal
skips.
- Red: answered questions and accepted/rejected confirmations return no
admitted run before the fix. All three new positive cases fail at the
dependency gate. A resource-wait retry also reproduces as refused before
its gate is fixed.
- Green: at delivery revision `9375a603e4`, focused heartbeat
integration and classification passed 14 tests; retry scheduling and
dispatch safety passed 241 tests. These use real PostgreSQL and
controlled adapter and resource-availability boundaries. The wider
initial dependency, native-continuation, and queue suites also passed
before the final refinements.
- Negative cases keep pending, missing, wrong-task, and agent-resolved
responses blocked. Changing the response proof immediately before
dispatch cancels the run without calling the provider. Policy controls
retain the current assignee, task status, and pending approval checks.
- Positive cases execute once, leave the parent blocked, preserve the
dependency edge, and leave the child in backlog with no child execution.
The same saved response survives a resource wait; removing its human
proof refuses promotion.
- The resource-wait provider-boundary and blocker-completion regressions
also failed before their fixes. All 14 focused integration and
classifier cases passed at delivery revision `9375a603e4`, including a
changed response during retry preparation.
- Red: a retained Retry for an older run makes a newer successful
conversation display as failed. Green: all 78 database-backed query and
interruption-presentation tests pass, including a later real failure and
independent queued input controls.
- Server layered typecheck and module-boundary checks passed at
`11992aad0c7fbaf13e7548d71ee633560bd8bb6f`, before the one-line API
summary correction. That CI passed build, typecheck and every browser
shard, but exposed the pre-existing case-search assertion and a
report-assets test timeout. Final-head CI passed as linked above.
- Fresh Greptile review is 5/5 on final commit
`ba4b337cf1bb0236b7c0a742665b222ee894ff51`, with no unresolved threads.
- The existing API discovery suite reproduced its first-page assertion
before the summary correction; all 3 cases pass afterward. The timed-out
report-assets privacy test passed locally in isolation and the full
support job passed on the final-head rerun. Runner shutdowns interrupted
the first attempt; affected jobs were rerun once without code changes. A
separate onboarding timeout also passed on that rerun.
- `pnpm verify --explain` requests broad CI because the execution
contract is a global input. The final CI run supplied that coverage.
- Separately recovered the reported staging conversation through an
ordinary message and the existing steering control. The agent
acknowledged the message and the saved-answer queue drained. The fresh
Notebook action subsequently succeeded. This was on the previously
deployed revision; it does not claim this code is deployed.

## Risks

- This changes admission and dispatch. Each gate checks the persisted
response; no caller-supplied marker grants the exception.
- The bounded turn does not check out the task, resolve a dependency, or
approve another action. Existing pause, ownership, budget, approval, and
stopped-execution controls still apply.
- Each additional lookup is one primary-key read and runs only for a
response on a dependency-blocked path. There is no new polling worker.
- No migration is required. Older queued contexts get the exception only
when their stored human response still verifies. No prior run or action
is replayed by this change.
- This PR does not roll out the code to staging.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools, code execution, and
browser inspection. The runtime does not expose the exact model ID,
context-window limit, or reasoning-mode label; those values are not
asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 16:27:34 -06:00
DottaandPaperclip 0e325e8d52 fix(runner): settle saved handoffs after verified shutdown (#20336)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - An agent can save a question or approval handoff before its provider
stops.
> - A missing provider terminal notification does not prove that the
saved work failed.
> - Runner already verifies provider exit, event drain, tool settlement
and suspension.
> - This change uses that proof to settle the same saved handoff.
> - Unproven shutdown holds continuation and explains that the work is
saved.
> - Recovery must never repeat the completed work or allow overlapping
execution.

## Linked Issues or Issue Description

Refs #20330. That PR extends the Codex terminal drain window. This
change handles the separate case where no terminal arrives but the owned
shutdown can prove stopped execution.

**What happened?**

A native run saved a human handoff, accepted cancellation, and later
stopped its provider. The missing terminal notification still made the
attempt fail and show “Couldn’t finish this attempt.”

**Expected behavior**

Keep the saved handoff. Accept its yielded result after either an
authoritative provider terminal or verified shutdown of the exact
execution. If shutdown remains uncertain, retain the handoff and explain
the hold without offering a duplicate Retry.

**Steps to reproduce**

Save a governed wait, acknowledge interruption, and omit the provider
terminal. The new regression uses the real Runner binary and a
controlled Codex process. The old runtime rejects the handoff instead of
asking the existing shutdown owner to settle it.

**Deployment mode**

Native Runner with Codex. Local verification used macOS arm64 and Node
26.4.0. Hosted sandbox recovery has not been qualified.

## What Changed

- Add an optional driver shutdown-settlement operation. Runner returns
proof only after its existing stop, drain, tool settlement and
suspension barrier passes.
- Bind the receipt to the run, normalized session and turn. Persist it
with the saved wait and validate it during recovery.
- Complete the same yielded handoff after verified shutdown. Retire the
stopped session from warm reuse. Do not invent a provider terminal or
complete usage.
- Retain a named shutdown-pending condition under the existing bounded
same-run recovery policy. Preserve actual provider failures and
integrity failures.
- Show saved-work shutdown guidance in task and inbox presentation. Keep
genuine approvals authoritative and disable duplicate Retry.
- Document the completion and restart contract. The checkpoint field is
additive; older checkpoints retain the terminal-or-shutdown path.

## Verification

- Red: the new runtime regression failed against the original
implementation because verified shutdown was never requested. Server
presentation and error-classification regressions also failed before the
fix.
- Red: both inbox shutdown states showed generic text before the inbox
change.
- Green: Runner runtime, shutdown and harness-backend suites pass 301
tests. Tests cover missing terminals, failed stop proof, wrong identity,
cancellation, late proof, genuine provider failure, and restart after
receipt persistence. Recovery completes once without another model turn.
- Green: the native executor and task-presentation suites pass 1,371
tests. Execution projection, inbox and blocker presentation pass 125
tests. Two test-authoring mistakes were corrected: an integrity-error
text expectation and an assertion for a nonexistent projection field.
Their initial failures remain in local logs.
- Runner TypeScript and full server layered type checks pass. Token
gates and diff whitespace checks pass.
- Live provider acceptance: real Runner and Codex with gpt-6.1-sol, one
disposable local workspace, one tool call, and one deliberately dropped
terminal notification. The saved file, bound settlement receipt, one
accepted yielded result, one provider turn, confirmed provider exit and
suspended Runner all passed. The tool handler and persistence port were
controlled. This does not claim a full server or hosted journey.
- The full browser question-and-answer cell stopped before execution
because OPENAI_API_KEY was absent. No model work ran in that attempt.
Hosted sandbox recovery and the full browser continuation journey remain
unverified.
- `pnpm verify` returned exit 2 because the execution-spec change
requires broad CI. Required CI will supply that coverage; a full local
repository suite was not repeated.

- Review correction in `8553da244e`: a failed trailing-event write could
hide a provider failure. A new regression reproduced red, then passed
after all ordinary drain errors retained the named pending condition. No
settlement receipt or result can be accepted after that error. The live
missing-terminal check passed again on the corrected runtime.
- The real PostgreSQL control-plane regression and three existing
governed-wait controls pass. The new case persists one accepted yielded
result, a bound shutdown receipt, and the original pending human
question without a provider terminal.
- First CI revision `7c2cb9b1`: test-program typecheck found a missing
`PrpEvent` annotation in the new fixture. The annotation is fixed and
the Runner test-program typecheck passes. Runner support also timed out
in the unchanged `registered roots remain protected even when equal to
genuine public static bytes` case after 30 seconds. One unchanged local
diagnostic passed in 3.69 seconds overall. The hosted timeout cause
remains unknown; no timeout, fixture, or assertion was weakened.
[Original failed support
job](https://github.com/paperclipai/paperclip/actions/runs/42670851821/job/129346900133).

- Rebased onto the merged #20330 and current recovery changes. The
combined branch passes 1,402 executor/presentation tests. Both prompt
and 18-second provider terminals still pass. A warm session with no
terminal passes after the full 30-second drain and verified
stop/suspension. Its initial overly broad assertion rejected a
nonterminal turn event; the corrected assertion rejects every terminal
type. The rebased CI gate was cancelled before any runner or step
started; its cause is unconfirmed and it supplies no validation credit.
Final revision `b49ec5a144` starts fresh CI and review.

- Final-head CI attempt 1 failed before creating jobs. GitHub reports an
internal server error (correlation
`2c576b07-b67f-4179-a69f-74f7d99cd926`), followed by empty
runner-selection outputs for the aggregate jobs. One unchanged-commit
retry was requested; the initial admission failure remains recorded. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/42673159868).

- **Current blocker:** final head
`b49ec5a144e94e41e96e2508a79bd9aec91fe45a` has a fresh 5/5 Greptile
review and zero unresolved threads, but required CI is not complete.
GitHub reports an active [Actions and Checks API
incident](https://www.githubstatus.com/incidents/tp8cghg7tlc4). Attempt
2 ran only part of the full selected graph. Build, canary, Rust, sandbox
providers, admitted browser shards and other completed test jobs passed.
The E2E aggregate correctly failed because `policy` and `e2e_shards` had
`abandoned` results. Server shard 5/12 received a runner shutdown signal
at 18:36:55 UTC and its test step was cancelled; its job still reports
in progress. The shutdown cause is not established. The separate review
workflow had the same pre-execution internal-error class and passed one
bounded retry. No further retry was submitted during the incident.
Resume with the missing/cancelled coverage after GitHub recovers; do not
count the absent typecheck, Runner or support jobs as passes. [Failed
coverage
gate](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129354318236),
[interrupted server
shard](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129350854174).

## Risks

- A missing terminal can use one additional bounded shutdown interval.
Unknown shutdown still blocks continuation.
- Provider terminal and usage accounting keep their separate meaning.
The receipt proves stopped execution and settled tool outcomes, not
model success.
- Recovery rejects a foreign receipt, a later turn, an active turn with
a settled receipt, and an actual failed provider terminal.
- There is no database migration. Older checkpoint shapes remain
readable. Drivers without this optional proof retain a named pending
state.
- This PR does not deploy the fix or repair an existing hosted task.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools and local
code execution. The exact serving model ID and context window are not
exposed in this session. Live test inference used gpt-6.1-sol.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 13:46:16 -06:00
DottaandClaude Opus 5.5 8fe524f796 fix(issues): let the board confirm an unidentified run stopped instead of an endless Retry wait (#20310)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a new attempt can start on a task, Paperclip must prove the
previous run's agent stopped, so two agents never work the same task at
once
> - A local native run can be interrupted after it asked to start the
runner but before it recorded the runner's process. The runner may have
saved state and started a provider, and provider processes can outlive
it, so the existing check (#19694) rightly refuses to assume the run
stopped
> - Today that case has no way out: every Retry is accepted and then
waits forever as `process_identity_missing` (storage chaos ST1, t02/t03)
> - This pull request makes the wait explicit and actionable. The task
explains why it waits, and a board operator who has checked can confirm
the run stopped; the parked Retry then starts at the click
> - A second shape from the Daytona storage baseline (t06) parked a
Retry with no reason at all: a follow-up was interrupted by a graceful
shutdown while it was still restoring its workspace. No provider ran,
but the restore hold kept the same agent held, so the Retry never
started
> - The benefit is that no Retry stays accepted-and-waiting with nothing
anyone can do

## Linked Issues or Issue Description

Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (ST1), tasks
t02 and t03.

**What happened (CHA-3, run `3c3f6241`):**
- The run requested a runner launch (`native.process_start_requested`),
and a K2 restart hit before its process identity was recorded.
- After the restart, the resumed attempt failed on the storage outage.
The task was blocked.
- The board's Retry (202) was parked as `executionWait.reason =
process_identity_missing`: "The previous run has no verified stop
record."
- `recordUnidentifiedLocalRunnerStop` cannot prove the stop because this
run's runner saved control-plane state. Once that exists, a provider may
have started (#19694), and a provider is not guaranteed to carry
`--run-id` or `PAPERCLIP_RUN_ID` (the Codex provider env omits it on
purpose). The shared owner prover needs a PID, which the run never
recorded.
- Nothing could ever clear the wait.

**Daytona storage baseline `w-20261109T154810-s9911`, task t06:**
- The run succeeded. The follow-up was cancelled
(`continuation_task_ownership_changed`), and the next run was
interrupted by a graceful shutdown (`server_shutdown_interrupted`) 3 s
into `preparing`. It was restoring the saved remote workspace and had
launched no provider.
- Restore recovery saved the files (`workspaceRecovery.status:
recovered`, system recovery `complete`). But the
`legacy_execution_requires_reconciliation` restore hold releases only
for a different owner, or for a native Stop fenced before any provider
turn, so it kept the same agent held.
- The UI Retry returned 202 and parked with `executionWait.reason:
execution_recovery`, the generic wait, so the task showed a Retry that
never started.

Automatic proof needs the runner to record its identity before saving
state and to group provider processes under it. That is follow-up runner
work; this PR removes the endless wait now.

## What Changed

- `server/src/services/native-local-process-stop.ts`: new
`confirmUnidentifiedLocalRunnerStopped`. It accepts only a finished
native run of the task with no recorded process and whose latest launch
evidence is an unconfirmed launch on this host in a local environment.
It still refuses while a runner with `--run-id <run>` is running. It
records `native.local_process_stopped` with `basis: "operator_attested"`
and the attesting user. That is the receipt the existing continuation
gate already accepts.
- `server/src/routes/issues.ts`: `POST
/issues/:id/recovery-actions/confirm-run-stopped` (`runId`,
`stopEvidence` of 20 to 12,000 characters). It is board-only and
requires issue read plus `runtime:confirm_owner_stop`, like the other
stop attestations. It logs `issue.run_stop_confirmed` with the evidence,
promotes deferred intents for the issue, and resumes each parked task
Retry, so work starts at the click. Refusals return 409 with a code.
- `packages/shared`: `confirmRunStoppedSchema`, and the presentation
user action `confirm_run_stopped`.
- `server/src/services/task-interruption-presentation.ts`: a Retry
parked as `process_identity_missing` for a native run that recorded no
process now presents as waiting for the user (`canRetry: false`,
`userAction: "confirm_run_stopped"`), with a message explaining why and
who can act. A run with a recorded process keeps the existing
verification wait.
- UI:
- `ExecutionBlockerNotice` shows the explanation and, for users who can
confirm a stop, "Confirm it stopped" with a short evidence form.
- It reuses `PreviousServerStopConfirmation`, now with optional
`action`, explicit `runId` and `formId`, and a custom button label.
  - The Inbox names the action.

- `server/src/services/execution-blocker.ts`:
`interruptedPreparationBeforeProviderWorkCondition`. The restore hold
also releases for the same agent when the stopped run was a
system-interrupted preparation (legacy runtime never resolved, still
`preparing`) with no process identity, no runner or provider events, no
finalization or result, and every lease released with a confirmed stop.
It mirrors the existing interrupted-preparation proof and still requires
the completed workspace-recovery receipt. Auto-replay is unchanged; this
only lets the user's Retry or message start.
- Review fixes:
- `unidentifiedLocalLaunchRefusal` holds the endpoint's launch, host and
lease checks, and the presentation offers `confirm_run_stopped` only
when it passes.
  - The attested stop wakes cost accounting in the same transaction.
  - The confirmation form is keyed by task and run.
  - The route is registered in OpenAPI.

## Verification

Red/green (each new test fails against master and passes with this PR):
- `server/src/__tests__/issue-confirm-run-stopped-routes.test.ts`:
- The parked Retry presents `confirm_run_stopped` with the explanation.
- Confirming records the operator-attested stop and the activity entry,
satisfies the stop check, and resumes the parked Retry: its receipt is
coalesced into a newly started run.
  - It refuses with 409 while a runner for the run is running.
  - Agents and viewers get 403.
-
`ui/src/components/ExecutionBlockerNotice.confirm-run-stopped.test.tsx`:
the notice offers "Confirm it stopped" (and no dead Retry) and posts the
run and evidence. A viewer sees the explanation without the action.

- `server/src/__tests__/interrupted-preparation-restore-retry.test.ts`
(t06 shape):
  - The restore hold no longer blocks.
- A Retry is accepted and resumed into a started run, and the audit hold
keeps `replay: blocked`.
- The hold stays when a provider turn started, when a runner event was
recorded, when an operator stopped the run, or when the sandbox has no
stop receipt.
- `issue-confirm-run-stopped-routes.test.ts`:
- A case per refusal code checks the presentation keeps the generic wait
and the endpoint returns 409.
  - An `alreadyStopped` case.
  - An accounting-wake assertion.
- The UI test switches targets with stale evidence.

Regression and static checks:
- `ExecutionBlockerNotice`, `WorkspaceExportRecovery` and its control
tests: 74 passed.
- `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`.
`pnpm check:token-gates` is clean.

## Risks

- A board operator can now attest that an unidentified run stopped. The
attestation is permission-gated, requires written evidence, is
activity-logged, and is refused while a runner for the run is visibly
running. A wrong attestation could start a second agent beside an
orphaned provider, the same trust model as the existing previous-server
attestations.
- The confirmation component now accepts a run without a recovery
action. Existing callers pass the action as before.
- No schema migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 11:36:41 -06:00
DottaandClaude Opus 5.5 87e755ce8e fix(recovery): verify cross-task warm sandbox handoffs and say plainly when recovery refuses (#20282)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes keep a warm sandbox and hand it to
the next run through a chain of reusable leases
> - Acquisition binds that chain to the execution workspace, so a task
can inherit a warm sandbox from an earlier task in the same workspace
> - Saved-files recovery verifies the same chain, but it required every
earlier lease, run, and finalization to belong to the recovering task
> - A valid chain that started with another task therefore always failed
with `allocation_ownership_ambiguous`. That code is `service_required`,
so recovery stopped after one attempt, and each board Retry repeated the
same refusal
> - This pull request makes the chain check match the acquisition rule,
and it checks each predecessor against its own task
> - It also makes the board say plainly when recovery refuses the saved
state, because "failed after automatic retries" was not true and Retry
looked like it did nothing
> - The benefit is that a task blocked with "Saving the agent's files
failed after automatic retries" can recover its saved files, and a
refusal that Retry cannot change is shown with its reason

## Linked Issues or Issue Description

**What happened?**
A native run in a remote sandbox finished its turn, and its result was
accepted. The copy of its files back to the host failed and was
scheduled to retry. A graceful server restart then interrupted the
retry, so the run settled with
`native_workspace_sync_out_retry_exhausted`, and the task became
blocked. Every saved-files recovery generation then failed on its first
attempt with `allocation_ownership_ambiguous` (`service_required`). The
board showed "Saving the agent's files failed after automatic retries.
Retry to try again". Each Retry replaced the generation
(`issue.recovery_generation_replaced`), and the new generation failed in
the same way within one second. To the user, Retry did nothing. The
sandbox lease chain was valid: same agent, same execution workspace,
same sandbox, and matching handoff receipts on all eight leases. But its
first three leases belonged to an earlier task in the same workspace.

**Expected behavior**
Recovery must accept a warm sandbox chain that acquisition accepted.
Acquisition binds reuse to the execution workspace, and it requires the
same task only when the lease has no workspace. Recovery must check each
predecessor run and finalization against that predecessor's own task. It
must still refuse a chain whose records disagree or that has no shared
workspace.

**Steps to reproduce**
1. Run a native task in a remote sandbox with `reuse_by_environment`,
with an execution workspace that another task also uses.
2. Run the other task in the same workspace first, so the sandbox lease
is handed from that task to this one.
3. Make the last run of the second task fail to copy its files back (for
example, restart the server during the copy-back retry).
4. Wait for saved-files recovery, or click Retry. Recovery fails with
`allocation_ownership_ambiguous`, and Retry repeats the failure.

**Paperclip version or commit**
`master` at a3a25ce6c9 (the same code as the 2026.1106.0 staging build
for every file this change touches).

## What Changed

- `retiredReusableLeasePredecessors` (`environment-lease-handoff.ts`)
requires the same task only when the lease has no execution workspace.
This matches the binding that `acquireLease` uses when it replaces a
reusable lease.
- `inspectWorkspaceRecoverySource` (`workspace-recovery-source.ts`)
checks each predecessor run and its finalization against the predecessor
lease's own task, not the recovering task. A predecessor from another
task is accepted only when both leases have the same execution
workspace. All other checks stay: same agent, terminal run, no live
controller or lease owner, a settled finalization, and a saved snapshot
at an older epoch on the same allocation, workspace, and remote
directory.
- The other callers of the chain walk (remote termination resolution and
retired backup restore) get the same rule.
- The test fixture `warmHandoffSource` gets an option that starts the
chain with an earlier task in a shared workspace.
- Honest copy for exhausted recovery (`exhaustedSystemRecoveryAction`):
- A deterministic refusal (`service_required`) now says "Paperclip
refused to recover the saved files", names the reason in plain words and
the code, and says that Retry runs the same check again and succeeds
only after Paperclip is updated.
- Other exhausted recovery says "stopped after N attempts" only when
there were several attempts. One attempt says "failed".
- Continuation (`explicit-native-continuation.ts`) reports a new wait
reason, `saved_work_recovery_refused`, when the exhausted recovery is a
deterministic refusal. The task presentation shows its own message for
it. `saved_work_recovery_failed` no longer claims automatic retries.
- Retry stays available for a refusal. It is the action that runs
recovery again after an update. After each Retry, the board shows the
same refusal and its reason, not an unchanged generic notice.

### Recovery refusal classes

| Retry class | Failure codes | Automatic recovery | Board Retry |
| --- | --- | --- | --- |
| `retry_transient` | `provider_unavailable`, `export_interrupted` and
other transient codes | Retries with backoff, up to 5 retries in 30
minutes | Starts a new recovery round |
| `recheck_evidence` | `agent_stop_unconfirmed`,
`workspace_writer_active`, `effect_unresolved`,
`copyback_stop_unconfirmed` | Checks the evidence again later | Starts a
new round (the copy-back stop case has its own wait text) |
| `use_alternative_strategy` | `historical_source_missing`,
`source_unavailable` (no recorded source) | Stops. The retained-source
fallback can continue from the last save | Starts a fresh attempt |
| `service_required` (deterministic) | `allocation_ownership_ambiguous`,
`source_binding_invalid`, `snapshot_verification_failed`,
`archive_unsafe`, `source_unavailable` (recorded source gone) | Stops
after one attempt. The same saved state gives the same result | Runs the
same check again. The board now says this and names the reason |

## Verification

- Red before the fix: `recovers saved work after 1|2 warm handoffs that
began with another task in the same workspace` failed with `expected
'allocation_ownership_ambiguous' to be null`.
- Green after the fix: `npx vitest run
src/services/workspace-recovery-warm-handoff.test.ts
src/services/workspace-recovery-source.test.ts` passed (60 tests).
- Safety cases that stay fenced: a predecessor lease that names a
different task than its run (`lease_task`), and a cross-task predecessor
with no shared workspace (`unshared_workspace`). The existing `task`
fault (another task with no workspace) also stays fenced.
- Sibling and retry suites passed: environment runtime and orphan
sandbox, cancelled workspace preservation, workspace recovery fallback,
task retry, remote workspace lifecycle, explicit native continuation,
native cleanup admission, and remote retry admission.
- Copy, red before and green after: `system-recovery-state.test.ts`
(`states a deterministic refusal plainly and never claims retries that
did not happen`), and the end-to-end `names a deterministic saved-files
refusal behind Retry when no fallback can continue, and another Retry
shows it again` in `workspace-recovery-preparation-gates.test.ts`. On
the old code the wait reason was `saved_work_recovery_failed`. With the
change it is `saved_work_recovery_refused`, with the refusal reason, for
both Retry clicks.
- Updated expectation: the board handoff in
`workspace-recovery-retry-copyback.test.ts` now expects the refusal text
for its `source_binding_invalid` source (79 tests pass).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.
- I compared the real incident's lease chain with every predecessor
condition through read-only diagnostics. All eight leases match on
agent, sandbox, workspace, and remote directory, and each predecessor
run saved a snapshot at an older epoch.

## Risks

- Recovery now trusts a predecessor from another task in the same
execution workspace. Acquisition already allows that handoff, and
recovery still requires the predecessor's own run and finalization to be
settled and saved. A workspace is the unit that owns the files, so this
does not let one workspace read another workspace's files.
- I could not read one predecessor's finalization row in the real
incident. If that row is not in a settled phase, recovery still refuses
with `allocation_ownership_ambiguous`, which is the old behavior.
- `snapshot_verification_failed` is labeled `service_required` at its 19
call sites, so it now shows the refusal text. If one of those sites is
really transient, the text says "Retry runs the check again", which is
still true.
- A second cause kept the same task stuck: a remote snapshot save that a
restart interrupted stayed `running`, and that blocks the
retained-source fallback. That fix comes in a separate pull request.
- No schema or API change. The new wait reason is a new string in an
existing field. The UI shows the message as it receives it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only browser diagnostics).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 09:53:37 -06:00
DottaandClaude Opus 5.5 4c15eea770 fix(accounting): a run's unproven accounting never holds the company (#20254)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, each lost provider receipt leaves an accounting
debt that settles later
> - In restart-chaos G1 (seed 9802, task t02), six debts each waited the
full 10-minute closure bound, and the board's accepted Retry waited with
them
> - The last debt settled as an unpriced charge. Under the blocking
unpriced-usage policy, that charge hard-stopped the whole company, which
cancelled the Retry, and the task showed "budget"
> - Dotta's plan decision 1 says accounting is never a company-wide
gate: admission reads per-run exposure and real hard stops only
> - This pull request makes those debts settle at once when the stop is
proven, records accounting-settled charges without a company hold, and
labels the remaining accounting wait correctly
> - The benefit is that a restart no longer stalls a company's work for
10 minutes or stops it outright

## Linked Issues or Issue Description

Refs #20250 (Phase 2: a run that only inherited its predecessor's runner
identity settles at once; this PR builds on it), #19944, #19826, #19857

**What happened?**
Restart-chaos wave w-20261109T111954-s9802 (G1), task t02, master
c5da0b88ad. After a K-fault restart at 11:27, six `run_accounting_debt`
records were "unprovable". Each settled exactly 10 minutes after its run
finished: bf9ea6b4 and c2e222a2 at 11:30, 097d0036 at 11:35, f64f9f46
and aa2c9f03 at 11:36, and 0b4b404f at 11:37:23.
- Four runs were `cancelled` (issue_terminalized), with
`native.local_process_stopped` as their last process event and no
recorded pid. The receipt-closed rule accepted that stop proof only for
`failed` runs.
- 0b4b404f failed with `native_session_cleanup_quarantined` before it
started any process. It journaled no receipt, but it carried pid 11356
of the previous run's warm runner, so the never-started rule did not
apply.
- At 11:37:23, 0b4b404f settled as an unpriced unknown-billing charge
with zero tokens. With `unpricedUsagePolicy: block`, the company hit a
hard stop at 11:37:24 (24 of 3000 cents). The pause cancelled the
board's Retry (751751ea, accepted at 11:28:37), and the task showed
`userAction: "budget"`.

**Expected behavior**
Accounting debts whose provider stop is proven settle at once.
Accounting never holds or pauses the whole company. A wait caused by
unpriced usage is labeled as accounting, not budget.

**Steps to reproduce**
Restart the control plane while native runs are being cancelled, with
the company's unpriced-usage policy set to block. Before this change,
the debts wait 10 minutes and then hard-stop the company.

## What Changed

- `run-cost-accounting.ts`: a verified local process stop closes the
receipt source for `cancelled`, `timed_out` and `interrupted` runs as
well as `failed` ones. The inherited-pid run (0b4b404f) is covered by
#20250, so this PR does not change the never-started rule.
- `budget-exposure.ts`: a final charge that accounting settled without
its receipt (receipt unavailable, closure timed out, capture failed, or
lost on restore) never trips the blocking unpriced-usage policy.
Unpriced usage from a complete receipt still does. A capture failure
stays exempt even if a later report adds usage, because its receipt is
never certified complete. This removes the restart-derived special case,
because the rule now covers every accounting-settled charge.
- `task-interruption-presentation.ts`: a `budget.blocked` wait caused by
the unpriced-usage refusal is shown as `accounting.unpriced_usage`, with
accounting wording, for both a held Retry and a never-started task. Only
the fixed refusal phrase is matched; the stored message is never shown.
- `doc/SPEC-implementation.md`: the admission and accounting rules are
updated to match.

## Verification

- `closed-native-receipt-accounting.test.ts`: new tests for cancelled,
timed-out and interrupted runs with a verified stop; they were red
before the fix. When a timed-out closure settles under the blocking
policy, the board's saved Retry is promoted (due at once) and never
cancelled. That assertion was red on master, where the hard stop
cancelled it. Seven tests that asserted the old rule (an
accounting-settled charge keeps the blocking policy) now assert that the
charge is recorded and the company stays active and admits work. All 61
pass.
- `task-interruption-query.test.ts` (Postgres): a Retry held by the
unpriced refusal, and a skipped wake, are labeled
`accounting.unpriced_usage`. A budget hard stop keeps `budget.blocked`.
The test was red before the fix.
- Also passing: `run-cost-accounting`, `cost-data-quality`,
`idle-accounting-work`, `native-usage-delivery.integration`,
`task-interruption-presentation`, and the accounting and budget tests in
`heartbeat-process-recovery`. Server typecheck has no errors in the
changed files. The only local errors are in
`native-session-executor.ts`, from a stale local runner build.
- Not run: the G1 chaos rerun. Phase 3 runs it after merge.

## Risks

- Behavior change: a charge that accounting settled without a receipt no
longer hard-stops a company under the blocking policy. That includes a
remote run whose closure evidence never arrived, which may still have
spent. The charge stays in Costs with provenance and an activity entry,
and the spend still counts toward the budget limit. This follows plan
decision 1.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 07:02:47 -06:00
DottaandClaude Opus 5.5 4420a5adcf fix(issues): a Retry held by a blocked subtask names it instead of failing silently (#20109)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board's Retry is the one action a person takes when a task
stops; it must either start work or say why it can't yet
> - Restart chaos t05 left a parent task blocked on a blocked subtask.
The board clicked Retry twice; each was accepted and nothing happened
> - The Retry was saved and the queue deferred it behind the unresolved
blocker, which is correct. But the task still looked failed with Retry
offered, so the wait was invisible
> - This pull request presents a saved Retry that waits only on
unfinished blockers as waiting, names the blocked subtask and links it
> - The benefit is that the board sees what to act on (the subtask's own
Retry) instead of clicking a Retry that cannot run

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan). Restart chaos wave
`w-20261108T200620-s9701`, task t05 (claude, M1, K4).

**Problem:** parent CHA-5 was blocked by subtask CHA-10, which was
itself blocked on a board recovery action. The board pressed Retry on
CHA-5 at 20:27:46 and 20:40:46. Each `POST /issues/:id/retry` returned
202, and the saved Retry stayed `deferred_issue_execution` behind
CHA-10, so no run started. The task's interruption presentation kept
saying `failed` with `canRetry: true` and gave no reason.

## What Changed

- `server/src/services/task-interruption-presentation.ts`:
- For tasks with a saved (deferred, user-requested) Retry, the loader
reads the queue's own dependency readiness (`listDependencyReadiness`).
That counts cancelled blockers and done blockers whose workspace has not
finalized, like the queue does.
- When the saved Retry for the latest run is held only by unfinished
blockers (no execution wait, no preparation), the task is presented as
`working` with `canRetry: false` and `retryWait: { reason:
"dependency_blocked", message }`.
  - The message has three forms:
    - a blocked blocker: "Retry or unblock it";
- a cancelled blocker: "Remove or replace that dependency to continue",
because the queue keeps it unresolved;
    - otherwise: waiting for the blockers to finish.
- The presentation carries no blocker names or ids. Issue reads return
it to restricted agents too.
- The Retry response (`POST /issues/:id/retry`) already returns this
presentation, so the API says the Retry is waiting on dependencies as
soon as it is saved. The blockers themselves are in the issue's
`blockedBy`, which issue reads already filter per viewer.
- `ui/src/components/ExecutionBlockerNotice.tsx` takes `blockers` (the
issue's `blockedBy`, from `IssueDetail`). For a `dependency_blocked`
wait it lists each readable unfinished blocker with a link and what it
needs: "Blocked: open it to Retry" or "Cancelled: remove or replace it".
Tokens only; `pnpm check:token-gates` is clean.
- No shared type change, and no code comments added.

## Verification

Red/green:
- `server/src/__tests__/task-interruption-query.test.ts` "explains a
saved Retry held by unfinished blockers instead of offering Retry again
(restart chaos t05)". It sets up a parent with a failed run, a saved
Retry deferred, and a blocked subtask that blocks it.
  - On master: `{ state: "failed", canRetry: true }`, no reason.
- With this PR: `working`, `canRetry: false`, `dependency_blocked`. The
presentation has no blocker title or identifier.
- A cancelled blocker gets the remove-or-replace message. Once the
blocker is done, the dependency wait is gone.
- `ui/src/components/ExecutionBlockerNotice.test.tsx`: the notice links
the blocked and the cancelled blockers with their actions, skips the
done one, and offers no second Retry.

Regression: `task-interruption-query` (25 tests) and
`ExecutionBlockerNotice` (16 tests) pass, as do the presentation unit
tests, workspace-recovery admission, gates and copyback tests, and
`heartbeat-dependency-scheduling`. `tsc --noEmit` is clean for `server`
and `ui`.

## Risks

- One more batched read (dependency readiness plus blocker statuses),
only for tasks in the page that have a saved Retry.
- Presentation only. Admission is unchanged: the queue still runs the
saved Retry when the blockers are done.
- A Retry saved before a blocker was added shows the same wait, which is
accurate, because the queue holds it too.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (#20069)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (code
comments; no user docs affected)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending)
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 17:18:55 -06:00
DottaandClaude Opus 5.5 cbf811cd3b fix(recovery): prove a replaced deployment's copyback owner gone automatically (#19913)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save their workspace after the result is accepted, under
a copyback owner receipt
> - A deploy can restart the server while that save is in progress
> - The new server runs in a new container, so it cannot see the old
process and fails closed
> - The run then waits on a board action that asks a person to verify
the previous controller
> - Recoveries must continue without a human step, and copy must not ask
anyone to inspect a server
> - This pull request adds a deployment-level rule to the shared
owner-liveness primitive and uses it in workspace finalization
> - The benefit is that a save interrupted by a deploy resumes on its
own, with no second writer

## Linked Issues or Issue Description

**What happened?**
A deploy restarted the server while three runs were saving their
accepted results. Each run kept a copyback owner receipt that names the
old Railway deployment on another container. The restarted server
reported `native_workspace_finalization_owner_unverified`, owned by the
board, with the next action "Verify the previous controller and its
workspace-copyback processes have stopped...". The cloud harness stop
receipt did not arrive, so the runs stayed in "Recovery needed" for
about an hour, until a person confirmed the stop by hand.

**Expected behavior**
The restarted server proves that the previous controller is gone,
resumes the save, and settles the run with no human step. No user-facing
text asks a person to verify a server or controller.

**Steps to reproduce**
1. Start a native run on a Railway service with a volume.
2. Let the result be accepted and the workspace save start.
3. Deploy, so that the server restarts during the save.
4. See that the run waits for a board confirmation.

**Paperclip version or commit**
Observed on e2d490c61. Reproduced on master before this change. The
owner-proof work in #19848, #19851, #19853, #19868, #19871 and #19883
does not cover this shape: `proveOwnerGone` returns `unprovable`
(`different_host`) for an owner on another container.

## What Changed

- `owner-proof.ts`: `LocalOwnerStamp` can carry the platform `worker`
(deployment identity). `ProcessProbe` can carry this process's
deployment and whether the platform gives it an exclusive volume. New
verdict reason `deployment_replaced`.
- New rule in `proveOwnerGone`, after the current-boot check: the owner
is gone when this server runs with a Railway volume
(`RAILWAY_VOLUME_ID`, `RAILWAY_VOLUME_MOUNT_PATH`) and the stamp names
the same project, environment and service with a different deployment
ID. Railway stops the previous deployment before it mounts the volume
into the new one, and a service with a volume has no replicas. So every
process of the old deployment, including orphaned copyback children, has
exited.
- `native-workspace-finalization-ownership.ts`: evaluate that proof
before the cloud harness request. On `deployment_replaced`, claim the
exact owner receipt under the existing advisory lock and row-lock
revalidation, log `issue.workspace_owner_stop_verified` with
`deploymentProof`, mark the recovery `resuming`, and continue the save
with no provider turn. Other cases keep the harness request and the
existing fences.
- New owner receipts also record the mounted volume ID (`volumeId`).
When a receipt has it, the proof requires this server to mount the same
volume. Both `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH` must be
set. The finalizer reaches this check only while it holds the run's
copyback advisory lock, so the old owner's lock session has also ended.
- Copy: the hold's next action, the route conflict and the two Retry
wait messages now describe the wait. They no longer ask a person to
verify a server or controller. The hold's text still names the Resume
saved result action for platforms where neither this rule nor the
harness can confirm the stop.
- `doc/native-workspace-finalization-recovery.md`: describe the rule.

Previous-release story: receipts written by earlier releases already
record `worker.deploymentId`, so they carry forward and take this path
after the upgrade. No schema change.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-workspace-owner-restart-recovery.test.ts
src/services/owner-proof.test.ts`
- Before the fix: "resumes the interrupted save automatically, with no
board action" fails (`acquired: false`).
  - After the fix: all pass.
- The restart test seeds the staging receipt shape (old deployment,
other hostname, PID 3, existing `operator_required` hold). It checks
that the save resumes once, the hold becomes `resuming`, the task
presentation no longer waits for a user, the receipt clears, and no
agent wake is created.
- Negative cases stay fenced: same deployment, a service without a
volume, a half-configured volume, another service, another environment,
and a receipt that names a different volume. Unit tests cover the same
rule in `proveOwnerGone`, including that the current boot stays alive.
- Related suites pass: finalizer, queued-wake recovery, route ownership,
workspace recovery gates and dispatch, explicit continuation, recovery
services (1005 tests).
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- The rule depends on Railway volume exclusivity. It applies only when
this server has a volume mounted and the receipt names the same service
and environment. Without a volume, the existing harness and board paths
still apply.
- A receipt from a deployment that ran before the volume was attached
could, in theory, overlap during that one redeploy. Volumes are attached
at stack creation, so this is not a practical case.
- Copy changes are text only. Error codes are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 16:12:23 -06:00
12681d1611 fix(sentry): stop resending retained task interruption failures (#19041)
Keep task-state Sentry reports within the recent saved-change window while
retaining reports for newly failed recovery and delayed Retry preparation.
Include the safe run error code without changing event grouping.

Verified 402 relevant local tests, every server typecheck layer, all final-head
CI checks, and Greptile 5/5 with no unresolved review threads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-07 11:31:46 -08:00
DottaandClaude Opus 5.5 464e6e16e0 fix(native): resume copyback after a restart kills its same-host owner; one board action otherwise (#19717)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save the agent's workspace files after the provider turn
finishes (copyback)
> - One controller owns copyback at a time. The run records that owner
in `runnerProfileJson.nativeWorkspaceFinalizationOwner` (host, PID,
token)
> - A control-plane restart can kill the controller during copyback. The
dead controller can never confirm its own stop
> - Recovery then waited for a board confirmation. No surface offered
that confirmation, so the run stayed `running` and the task stayed
`in_progress` with no visible action
> - This pull request lets the next controller on the same host prove
the stop by itself. It also gives the board one clear action when that
proof is not possible
> - The benefit is that a restart during copyback resumes by itself, and
a task never waits silently for a person

## Linked Issues or Issue Description

**What happened?**
In a Daytona restart-chaos wave, a kill during workspace copyback left
the run `running` and its task `in_progress`. Recovery reported
`native_workspace_finalization_owner_unverified`. The run was still
`running` when the wave timed out.

**Steps to reproduce**
1. Start a native remote (sandbox) run.
2. Kill the control-plane process (SIGKILL) after the provider result is
accepted, during copyback.
3. Restart the control plane on the same host.

**Expected behavior**
If the previous controller and its copyback children are proven stopped
on this host, recovery resumes copyback without a new agent turn.
Otherwise, the task shows one board action to confirm that the previous
server stopped.

**Actual behavior**
The recorded owner was cleared only by
`resumeNativeWorkspaceAfterOwnerStop`. Same-host recovery also required
a completed-copyback barrier, which an interrupted copyback never has.
The task presentation showed `working`, and the UI had no control for
the confirmation.

Refs #19578 (boot-ID fencing for unstarted runs; same proof style).

## What Changed

- Root cause:
`server/src/services/native-runtime/native-workspace-finalization-ownership.ts`
sent a dead same-host owner to `requireStopVerification` when no
`workspace_finalize` operation had succeeded. That case is exactly a
restart during copyback.
- New `native-workspace-owner-stop-proof.ts`. The owner receipt now
records the controller's process group (`processGroupId`). Copyback
children (tar, Git) are not detached, so they keep that group. A
successor on the same host releases the receipt only when the receipt
names another controller boot, the PID is dead or recycled (process
start time), and the group is empty or has no member outside the
successor's own process tree that started after the old controller. Age
alone never proves a stop.
- A still-live group member returns `local_stop_pending`. The
finalization worker rechecks it on its existing ownership recheck
interval. The board action stays visible meanwhile.
- A successful local proof writes `issue.workspace_owner_stop_verified`
with the proof and marks the recovery `resuming`, like the existing
platform proof.
- A receipt from another host, a receipt without a process group, or a
host without `ps` keeps the fence. The task interruption is now
`waiting_for_user` with `userAction: "confirm_owner_stop"` (new shared
enum value), not `working`.
- UI: the task chat shows one **Confirm the previous server stopped**
action. It asks for stop evidence and calls the existing
`resume-workspace-finalization` API. It is shown only to roles that hold
`runtime:confirm_owner_stop` (company owner or administrator, local
board, instance administrator). The inbox names the same action.
- The copyback advisory lock, the coordinator finalization lease, and
remote save fencing are unchanged.
- Docs: `doc/native-workspace-finalization-recovery.md`.

## Verification

- Red then green on embedded Postgres
(`server/src/services/native-runtime/native-workspace-finalizer.test.ts`):
- same-host controller killed with SIGKILL, its process group empty:
before `owner_unverified`, after `acquired` with the proof logged and no
agent wake
- a non-detached orphan child of the dead controller still runs:
`local_stop_pending`, no copyback; after the child exits, `acquired`
  - a receipt without a process group: still `owner_unverified`
- a foreign-host owner: the task presentation was `working` (red), now
`waiting_for_user` / `confirm_owner_stop`; after the board confirmation
it is `working` again
- The existing orphan test (a detached child) and all other finalizer
tests pass: `npx vitest run
src/services/native-runtime/native-workspace-finalizer.test.ts
src/__tests__/native-workspace-finalization-work.test.ts
src/services/task-interruption-presentation.test.ts` (135 passed).
- UI: `npx vitest run src/components/WorkspaceExportRecovery` (34
passed), including the action call body and the non-administrator case.
- `pnpm check:token-gates` clean. Server and UI typecheck clean for the
changed files.
- Live Daytona restart-chaos wave: results are added in a comment below.

## Risks

- A process-table check that reports a member wrongly as live only keeps
the existing fence. The board action is still available.
- A subreaper that adopts orphans does not change their process group,
so the proof does not depend on the parent PID.
- A child that a future change spawns with `detached: true` would leave
the group. Copyback paths do not do that today. The existing
detached-orphan test still fails closed, because a receipt without a
process group is not released.
- Old receipts have no process group. They keep the board action.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 23:31:07 -06:00
DottaandClaude Opus 5.5 c45c7cb854 fix(workspaces): ignore killed-transfer scratch in sparse checkpoints (#19730)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run in a remote sandbox copies its workspace back to the
host after the turn. A sparse checkpoint copies only changed entries,
and the host validates each entry against the sandbox's own manifest
> - Every Daytona transfer stages a `.paperclip-upload-<uuid>` scratch
file at the sandbox workspace root and removes it only in the same
process
> - A control-plane kill during any transfer leaves that scratch behind.
The sparse checkpoint then recorded it as a workspace file, but
providers never export it. Every later copyback and every saved-work
recovery attempt failed with "Workspace checkpoint payload mismatch"
> - The failed recovery then left an accepted Retry showing "Paperclip
is finishing the saved result" with no run
> - This pull request stops recording transfer scratch, makes the
checkpoint export an exact copy, names the failing entry, and gives a
Retry behind a spent recovery an honest reason and a working action
> - The benefit is that a restart during a transfer no longer makes a
task's saved work permanently unrecoverable

## Linked Issues or Issue Description

Refs #19721 (RC-50: the killed copyback owner; this PR covers the
separate RC-50b cause)

**What happened?**

Restart-chaos waves s12/t01 and s14/t05 (Claude on Daytona). After a
SIGKILL of the control plane, the native result was accepted, and then
workspace copyback failed with "Workspace checkpoint payload mismatch".
In s14 all 6 saved-work recovery attempts failed the same way. A Retry
accepted during recovery kept `retryWait.reason =
finalizing_saved_result` after the recovery budget was spent. No run was
created.

**Steps to reproduce**

1. Start a native Claude run in a Daytona sandbox.
2. SIGKILL the control plane while any provider transfer is in flight.
3. Let the run finish its turn and copy its workspace back.

**Expected behavior**

Copyback succeeds. Provider scratch is not workspace content. A Retry
behind a spent recovery names the failure and can start a new recovery.

**Actual behavior**

The s14 sandbox (6f4a0b6a) still held
`/home/daytona/paperclip-workspace/.paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar`.
Its mtime was 03:29:23, the second of the K4 SIGKILL. The Daytona
provider stages transfer scratch at `remoteDir`, which is the lease
`remoteCwd` (the workspace root). The runtime workspace wipe
deliberately preserves `.paperclip-upload-*`. The checkpoint helper
(`workspace-checkpoint.ts`, `checkpointMain`) recorded the stale tar.
The ordinary export excludes `.paperclip-upload*`, so the host received
nothing for that entry. `validateCheckpointPayload` then threw "payload
mismatch" on every attempt. The file-mode hypothesis was disproved: the
sandbox umask is 022 and the scratch is mode 644.

## What Changed

- `packages/adapter-utils/src/exclude-patterns.ts`: new
`isSandboxTransferScratchPath`, which matches a root-level
`.paperclip-upload-*` entry. This agrees with the wipe's preserve rule
in `buildWorkspaceTarExtractCommand`.
- `packages/adapter-utils/src/workspace-checkpoint.ts`: the sandbox
checkpoint helper skips that scratch. A mismatch error now names the
entry and its recorded, baseline and received forms (kind, octal mode,
short hash; never contents).
- `packages/adapter-utils/src/sandbox-managed-runtime.ts`: the
checkpoint export requests `preserveSourceEntries: true`. The checkpoint
output is an exact copy that the host validates entry by entry. An
ordinary export applies the host umask and drops `._*` and
`.paperclip-upload*` names at any depth. That gives the same permanent
mismatch for a legitimate user file such as `._summary`,
`notes/.paperclip-upload-plan.md`, or a `0775` script.
- `server/src/services/explicit-native-continuation.ts` and
`task-interruption-presentation.ts`: when the run's saved-work recovery
is exhausted, a waiting Retry reports `saved_work_recovery_failed` with
Retry available, instead of `finalizing_saved_result`. The existing task
Retry path starts one new recovery generation.
- `doc/execution-semantics.md`: a short additive note.

## Verification

- `packages/adapter-utils/src/workspace-checkpoint-capture.test.ts` uses
the real checkpoint helper and an export that behaves like Daytona's
ordinary export:
- Before the fix: `Workspace checkpoint payload mismatch at
".paperclip-upload-d99fe36b-….tar": sandbox recorded file mode=644 …;
baseline absent; received absent`. That is the wave error.
  - After the fix: the scratch is not recorded, and validation passes.
- A second case shows that reserved-looking user names and `0775` modes
need the exact export.
- `packages/adapter-utils/src/workspace-checkpoint.test.ts` is an
end-to-end `prepareSandboxManagedRuntime` restore through a provider
model. In that model, ordinary exports drop reserved names and apply the
umask, and preserve exports copy exactly. Two red checks:
- Without `preserveSourceEntries`, it fails with the mismatch on
`._summary`.
- Without the scratch skip, the stale tar leaks into the host workspace.
- Green: the agent file, the `._summary` file and the `0775` mode
arrive, and no scratch arrives.
- `server/src/services/workspace-recovery-preparation-gates.test.ts`,
"names a spent saved-work recovery behind an accepted Retry, and the
next Retry finishes it" (s14 shape):
  - Red: `finalizing_saved_result` after 6 failed attempts.
- Green: `saved_work_recovery_failed` with `canRetry: true`. The next
Retry starts a new generation, recovery completes, and the Retry
continues in exactly one new run.
- Suites: adapter-utils checkpoint, capture, sandbox-managed-runtime,
seed-reuse and managed-seed (168 passed). Server preparation-gates (47),
explicit-native-continuation and task-interruption-presentation (455),
workspace-recovery admission and retry-copyback, and
native-workspace-finalizer. `tsc --noEmit` is clean for adapter-utils
and server.
- Live evidence: I started the stopped s14 sandbox read-only, listed its
workspace, and stopped it again. No Daytona replay was run. The runner
content ID of this checkout matches no built image.

## Risks

- A user file named `.paperclip-upload-*` directly at the workspace root
is no longer captured by a sparse checkpoint. Providers already drop
that name from every ordinary export, and the runtime already reserves
it at the root.
- The checkpoint export now uses the provider's exact-copy mode. A
workspace file with setuid, setgid or sticky bits is refused by that
mode, as it is for the existing exact-copy transfers. The ordinary
export would have stripped those bits and failed validation anyway.
- The stale scratch tar stays in the sandbox. It is ignored, not
deleted, because a concurrent transfer can own a file with the same
prefix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, test runs, embedded Postgres, a
read-only Daytona sandbox listing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 22:46:16 -06:00
DottaandClaude Opus 5.5 db41bfcb38 fix(recovery): let an accepted Retry finish after a killed copyback owner (#19721)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run saves the agent's workspace files after the provider
turn (copyback). One controller owns copyback at a time and records a
durable owner receipt on the run (`nativeWorkspaceFinalizationOwner`)
> - When copyback of an accepted result fails, retained saved-work
recovery exports the files from the kept sandbox. A task Retry waits for
that recovery before it continues
> - A control-plane restart can kill the controller while it owns
copyback. The dead controller can never confirm its own stop, and
nothing in the retained recovery path could release its receipt
> - The recovery worker then spent one attempt on each recheck of that
receipt until it was exhausted, and the accepted Retry kept saying
"Paperclip is finishing the saved result" with no run. The task waited
forever
> - This pull request stops spending attempts on that recheck, names the
real wait, and lets the board release the exact receipt so the same
recovery and the same Retry continue
> - The benefit is that an accepted Retry never waits silently: it
finishes, or it shows one honest action that makes it finish

## Linked Issues or Issue Description

Refs #19717 (same-host automatic proof for a killed copyback owner; this
PR covers the retained-recovery path and the case where no automatic
proof is possible)
Refs #19713, #19609, #19694 (related restart-recovery fixes; no overlap
in files except this doc)

**What happened?**

Two Daytona restart-chaos waves. In each, a native run's copyback failed
and the run failed with `native_workspace_sync_out_retry_exhausted`. The
agent's result was accepted. Retained recovery started. A SIGKILL of the
control plane landed while a recovery attempt owned copyback:

- Wave s13, task t06 (Claude, crash loop): Retry was accepted at
02:30:35. A SIGKILL landed 2 s later. All later attempts failed with
`copyback_stop_unconfirmed`. After 6 attempts the recovery was
exhausted. The Retry showed `retryWait.reason = finalizing_saved_result`
for more than 15 minutes. No run was created.
- Wave s12, task t01 (Claude): attempt 1 failed with "Workspace
checkpoint payload mismatch". The SIGKILL at 01:43:54 killed the owner
of attempt 2. Attempts 3 to 5 failed with `copyback_stop_unconfirmed`.
The Retry was accepted with HTTP 2xx and produced no run.

In both kept databases the run carries an owner receipt for the killed
process.

**Steps to reproduce**

1. Accept a native result and make its workspace copyback fail, so that
retained recovery owns the export.
2. Press Retry on the task.
3. Kill the control plane while a recovery attempt owns copyback, so
that its owner receipt stays on the run. Use another host, or a receipt
without automatic stop proof.
4. Let recovery run.

**Expected behavior**

Rechecking the receipt does not spend attempts. The task names the wait
and does not offer a Retry that cannot help. A board member can confirm
that the previous server stopped. The recovery then finishes, and the
waiting Retry continues by itself.

**Actual behavior**

`withNativeWorkspaceFinalizationOwnership` returned `owner_unverified`
for a retained repair without creating any board action
(`native-workspace-finalization-ownership.ts`,
`requireStopVerification`). `recoverRetainedWorkspace` reserved an
attempt for each recheck and failed it as `copyback_stop_unconfirmed`
(`workspace-recovery.ts`), until the budget was exhausted. The Retry
gate in `explicit-native-continuation.ts` returned
`finalizing_saved_result` for an accepted result with no verified
repair, with no check for this hold.
`resumeNativeWorkspaceAfterOwnerStop` accepted only an
`owner_unverified` action on a `running` run, so no route could release
the receipt of a finished run. The exhausted handoff told the board to
press Retry again, which repeats the same failure.

## What Changed

- `server/src/services/workspace-recovery.ts`: when retained recovery
finds an unverifiable copyback owner, it records the recheck without
reserving an attempt. The 30-minute recovery window still bounds it. The
retained action records the exact receipt as `workspaceOwnerRecovery: {
state: "operator_required", owner }`. A newer receipt replaces the
record and clears an older confirmation.
- `server/src/services/explicit-native-continuation.ts`: a Retry that
waits for an accepted saved result reports
`workspace_owner_stop_required` when the run's active retained action
has that hold. It no longer reports `finalizing_saved_result` for this
case.
- `server/src/services/task-interruption-presentation.ts`: the new wait
has a fixed message, and Retry is not offered for it, because another
click cannot release the receipt.
-
`server/src/services/native-runtime/native-workspace-owner-recovery.ts`:
the existing board route `POST
/issues/:id/recovery-actions/resume-workspace-finalization` (permission
`runtime:confirm_owner_stop`) also accepts a retained recovery action
with that hold. The confirmation must name the receipt that is still on
the run. A live owner on this host still rejects it. It clears only that
receipt, records the stop proof and an activity entry, and returns the
same recovery to the worker. A spent budget gets one new generation, as
a deliberate recovery request does. It accepts no result and starts no
agent turn. The owner check is shared with the existing path.
- `server/src/services/system-recovery.ts`: an exhausted
`copyback_stop_unconfirmed` handoff names the board confirmation instead
of another Retry.
- `ui/src/api/issues.ts`: the route receipt's `resultId` can be `null`
(a retained recovery has no live result binding).
- `doc/native-workspace-finalization-recovery.md`: an additive paragraph
describes this path.

## Verification

- Red then green on embedded Postgres:
`server/src/services/workspace-recovery-preparation-gates.test.ts`,
"lets an accepted Retry finish after a killed copyback owner instead of
waiting forever" with one case per wave (s13, s12). Each case uses a
real accepted remote export, a Retry through the HTTP route, a
killed-owner receipt, and 8 recovery passes.
- Before the fix (both cases): `attempts: 6, state: "exhausted"`. That
is the wave state.
- After the fix: no attempt is spent on the recheck. The hold is
recorded. The Retry wait is `workspace_owner_stop_required` with
`canRetry: false`. The board confirmation through the route returns 202
and clears the receipt. The same recovery completes. The same accepted
Retry then finishes the saved result (s13: the task is `done`) or
continues in exactly one new run (s12).
- `npx vitest run
src/services/workspace-recovery-preparation-gates.test.ts -t "killed
copyback owner"`: 2 passed.
- `pnpm verify` (affected selection): see the comment below. Server `tsc
--noEmit` is clean.
- No live Daytona replay. Both waves ran on old signed images. The local
proof uses the same accepted-export fixture as the existing Retry tests,
with the receipt state copied from the kept databases.

## Risks

- A copyback owner on another host that is still alive is now rechecked
within the recovery window without spending attempts. Before this change
it spent them. The window still bounds the wait.
- The board confirmation for a retained action is an operator statement,
the same as the existing owner-stop path. It is limited to roles with
`runtime:confirm_owner_stop`. It never clears a receipt other than the
one it names, and a live same-host owner still rejects it.
- #19717 changes the same ownership area and adds a UI action for the
running-run case. That UI does not show a retained action yet. After
both merge, a small follow-up can show its button for this hold. Until
then the API route and the task's wait message cover it.
- Not addressed here: the "Workspace checkpoint payload mismatch" that
started wave s12. The checkpoint outbound transfer extracts without `-p`
(`sandbox-managed-runtime.ts` checkpoint sync-out, `extractHostTarball`
in the Daytona provider), so the host umask can change file modes that
the remote manifest recorded. That is a separate signature.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, test runs, embedded Postgres).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 22:07:00 -06:00
DottaandClaude Opus 5.5 08f7ac03dc fix(recovery): keep a never-started task waiting through a temporary accounting hold (#19702)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Admission checks every wake. A short company accounting hold can
refuse new work after a control-plane restart.
> - When the hold refused a new task's assignment wake, stranded-work
recovery found the task with no run while the hold was still active. It
moved the task to `blocked` with a board recovery comment.
> - A `blocked` task leaves the stranded-assignment sweep, so nothing
woke it after the hold cleared. The task stayed `blocked` with no run,
no interruption and no Retry.
> - The hold was temporary. Tasks created a few seconds later (after the
sweep) started normally once accounting settled.
> - This pull request keeps a never-started task `todo` while a
temporary budget hold lasts. The existing assigned-todo liveness
dispatch then admits it once the hold clears. The task shows the hold it
waits for.
> - The benefit is that a restart no longer silently drops new tasks
created during the accounting settle window.

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos storm on master `06bbcea12f`, three new tasks were
created while the company accounting hold was active. Each assignment
wake got 409 "Company cannot start work while completed runs await
accounting." (or "...native run recovers its unfinished accounting.").
One task rested `blocked` with 0 runs, interruption `null` and
`canRetry=false`. The other two ran later.

**Root cause**
`reconcileStrandedAssignedIssues`
(`server/src/services/recovery/service.ts`, the
`readInvocationBudgetBlock` branch near line 4826) escalates any
assigned task to `blocked` when a budget block exists. It does this
before the assigned-todo no-run branch, which already skips a
budget-blocked task and dispatches it later. So a task that never
started was treated as failed recovery work. The refused wake itself is
stored as a `skipped` `budget.blocked` diagnostic, which is correct; the
escalation removed the only path that re-admits it.

**Expected behavior**
A wake refused by a temporary admission hold waits and is admitted once
the hold clears, with one run. A real hard stop is still a board
decision. Removed assignees and cancelled tasks still do not wake.

**Steps to reproduce**
Create a company with a hard-stop budget and a completed run that has
`cost_accounting_pending = true`. Assign a new `todo` task. The wake
gets 409. Run the stranded-assignment sweep. Before this change the task
becomes `blocked` and never runs.

Related: #19700 bounds the accounting hold itself. This PR is
independent: it handles what happens to a task whose first wake the hold
refused.

## What Changed

- Stranded-work recovery skips a `todo` task with no run while a
temporary budget hold blocks admission. It does not escalate it to
`blocked`. The existing assigned-todo liveness dispatch admits the task
once the hold clears.
- A hard stop (spend at the limit, or blocked unpriced usage) keeps the
existing board escalation. Recovery checks this with the read-only
`policyBlocks` for the policies of the blocking scope. It does not
change budget hold semantics.
- The task interruption projection now covers a `todo` task with no run
whose latest wake was refused with `budget.blocked`. It shows "Waiting
for the budget to allow new work. The task starts automatically when it
does." with no Retry. A budget pause shows the existing "budget" user
action.
- The no-run projection applies the same person-owned gates as the
run-backed projection: a budget pause shows the "budget" action, and an
ordinary agent, project, company or task pause shows "resume". Only an
accounting hold shows the automatic-start text.
- `ExecutionBlockerNotice` shows that wait text for a `working`
presentation that carries a wait reason.
- `doc/execution-semantics.md` records the rule.

## Verification

- New embedded-Postgres regression
`server/src/__tests__/heartbeat-refused-assignment-wake.test.ts`:
- hold active, new task assigned, wake refused (409, `skipped`
`budget.blocked`), two recovery sweeps: the task stays `todo` with no
run and shows the wait reason. Before the fix this failed with `expected
'blocked' to be 'todo'` (the CHA-9 shape).
- hold cleared, two sweeps: exactly one provider turn starts, from
`issue.assigned_todo_liveness_dispatch`, and the task finishes. A later
sweep does not start another turn.
- counter-tests: a terminated assignee and a cancelled task get no run
after the hold clears.
- Existing `creates a board recovery action for budget-blocked assigned
work` (real hard stop) still passes unchanged.
- `pnpm exec vitest run
src/__tests__/heartbeat-refused-assignment-wake.test.ts
src/__tests__/heartbeat-process-recovery.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/heartbeat-issue-liveness-escalation.test.ts` (server) and
`src/components/ExecutionBlockerNotice.test.tsx` (ui).
- Server typecheck and `pnpm check:token-gates`.
- Live restart-chaos storm (local, codex/claude/opencode, 6 in flight,
K2 every 3-4 minutes for 20 minutes, no RC-06 mask), revision
e38bb5c8f7: 17 tasks, 10 pass.
- Two new tasks (CHA-16, CHA-17) were created while the company
accounting hold was active. Their assignment wakes got 409 "Company
cannot start work while completed runs await accounting." Both stayed
`todo` (not `blocked`) and showed `state: working`, `canRetry: false`,
`retryWait.reason: budget.blocked`, "Waiting for the budget to allow new
work. The task starts automatically when it does." No task rested
`blocked` with 0 runs.
- The hold itself never cleared in this wave (`pendingRunCount: 1` at
wave end), so these two tasks did not start before their deadline. That
stuck hold is the separate accounting-closure defect that #19700 bounds.
The other 5 failures (`R3:retry_no_new_run:B5`) are Retries refused by
the same stuck hold. The re-admission after the hold clears is proved by
the embedded-Postgres regression above.

## Risks

- Low. The change narrows one escalation to tasks that never started,
under a budget block that is not a hard stop. Those tasks now stay
`todo` and are re-checked by the existing 30-second stranded-assignment
sweep, which already re-checks every admission gate.
- The hard-stop check reads the blocking scope's active policies once
per budget-blocked never-started task in a sweep. It runs only while a
hold is active.
- The interruption projection adds one indexed read (company + payload
issue id) for `todo` tasks in the batch.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 21:12:00 -06:00
DottaandClaude Opus 5.5 0c904e2fd5 fix(tasks): treat a queued-message Interrupt as a hand-off, not a failed attempt (#19633)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can send a message while an agent works. The message waits in
a queue strip above the composer. The strip has an **Interrupt** button
that stops the current turn so the message starts now.
> - In a browser walk of onboarding, Interrupt showed a red "Couldn't
finish this attempt. Your message is saved." banner with Retry. The
thread also showed two rows for the same stopped run ("Stopped for 22s"
and "Stopped after 21s"). After about 20 s it recovered by itself.
> - A stop that the user asked for is not a failure. The red banner and
Retry make a normal action look broken.
> - #19369 stopped resource waits from showing as failed attempts. This
is the same banner, reached through the Interrupt path.
> - This pull request shows the Interrupt hand-off as working, and
renders the stopped turn once.
> - The benefit is that steering a running agent looks calm and correct:
one stopped row, no error, and the queued message starts next.

## Linked Issues or Issue Description

Refs #19369 (same banner class, resource-wait path)

**What happened?**
While an agent ran, the user queued a message and clicked Interrupt on
the queued strip. A red banner "Couldn't finish this attempt. Your
message is saved." with Retry appeared at the top of the task. Two rows
showed the same stopped run: "Stopped for 22s" and "Stopped after 21s".
About 20 s later the banner went away and the message was delivered.

**Expected behavior**
A user Interrupt is not a failure. No red banner or Retry. One "Stopped"
row, and the queued message starts next.

**Steps to reproduce**
1. Start a task with a Paperclip Runner agent.
2. While it works, send a message so it waits in the queue strip.
3. Wait more than 10 s, then click Interrupt on the queued strip.
4. See the red banner, Retry, and two "Stopped" rows.

**Paperclip version or commit**
master e482f0f632

**Agent adapter(s) involved**
Not adapter-specific (core bug); seen with Claude API through Paperclip
Runner.

## What Changed

Cause 1 — banner
(`server/src/services/task-interruption-presentation.ts`):
- The Interrupt route cancels the run with
`queuedCommentInterruptQueueId` set to the queued wake, and stamps
`queuedCommentInterrupt.requestedAt` on that wake. The projection skips
its "intentional Stop" rule whenever a pending wake exists, and treats a
pending wake as "working" only for 10 s after the message was *queued*.
A message queued earlier than 10 s before Interrupt, or a cleanup that
takes longer than 10 s, fell through to `state: "failed", canRetry:
true`.
- The projection now reads `queuedCommentInterruptQueueId` from the run
and `queuedCommentInterrupt.requestedAt` from the wake (both bounded
strings in the existing batched queries). When the cancelled run was
stopped for exactly the pending wake, it returns `working` for up to 5
min after the click. Pause, approval, reconnect and other user actions
stay first. A pending Retry preparation and exhausted system recovery
keep their own presentation. After 5 min, a hand-off that never started
shows one Retry, as before.

Cause 2 — two rows (`ui/src/components/TaskChatThread.tsx`):
- When a native runner turn ends, the live tail stays until the settled
turn *and* its reply comment render, or a 15 s backstop expires. A
stopped turn posts no reply, so the tail stayed for 15 s next to the
settled turn. That gave "Stopped after 21s" (live tail) and "Stopped for
22s" (settled turn).
- A cancelled settling run now hands off as soon as its settled turn
renders.

Tests:
- `server/src/__tests__/task-interruption-query.test.ts`: new
embedded-Postgres test drives the real batched query with the exact
`resultJson` the Interrupt route writes. It checks `working`/no Retry/no
Sentry failure capture, a run stopped for a different queue still fails,
and the 5 min bound.
- `ui/src/components/TaskChatThread.test.tsx`: new test renders a live
native runner turn, then the cancelled run with no reply. It checks
there is no live tail and exactly one "Stopped" row.

## Verification

Red on master (tests added before the fix):

```
$ npx vitest run src/__tests__/task-interruption-query.test.ts -t "queued-message Interrupt"
 FAIL  task interruption latest run query > shows a queued-message Interrupt as handing off, not as a failed attempt
AssertionError: expected { …(8) } to match object { state: 'working', …(2) }
-   "canRetry": false,
+   "canRetry": true,
-   "state": "working",
+   "state": "failed",
 Tests  1 failed | 18 skipped (19)
```

```
$ npx vitest run src/components/TaskChatThread.test.tsx -t "interrupted native turn"
 ❯ src/components/TaskChatThread.test.tsx:5528:82
    expect(container.querySelector('[data-testid="task-chat-live-transcript"]')).toBeNull();
 Tests  1 failed | 368 skipped (369)
```

Green after the fix:

```
$ npx vitest run src/__tests__/task-interruption-query.test.ts src/services/task-interruption-presentation.test.ts
 Test Files  2 passed (2)
      Tests  60 passed (60)

$ npx vitest run src/components/TaskChatThread.test.tsx -t "interrupted native turn"
 Tests  1 passed | 368 skipped (369)
```

Local `pnpm verify` (affected Vitest selection) result is added in a
follow-up comment; required CI supplies typecheck, build and the full
suite.

## Risks

- Low risk. Both changes are presentation only. Admission, cleanup and
Stop gates do not change.
- A hand-off that is blocked for a long time now shows no banner for up
to 5 min. The queued strip still shows its own wait text ("Waiting for
the previous run to finish recovery…") during that time, so the wait is
visible. After 5 min, Retry returns.
- The thread change applies only to a settling run with status
`cancelled`. Failed and succeeded runs keep the existing reply-comment
hand-off.
- Not in scope: the task status icon changed to `todo` during the
hand-off. That is the issue status, not this projection.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 17:18:04 -06:00
DottaandClaude Opus 5.5 dc1adb7209 fix(tasks): advertise a native resume only while the scheduler would claim it (#19523)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native run cannot reattach to its provider after a
control-plane restart, the native coordinator resumes the same run by
itself after about 30 seconds, up to three attempts
> - The run projection shows this correctly as "Retry scheduled" with
only `inspect_run` permitted
> - The task interruption shown on the issue read only system-recovery
evidence and `scheduledRetryAt`. It never read the coordinator's
`nextAttemptAt`, so it showed `state: failed, canRetry: true` and
offered a manual Retry while the automatic resume was still pending
> - The two surfaces disagree, and a user click can race the automatic
resume. That is one way a Retry gets parked behind "execution checks"
> - #19503 made the task interruption read the coordinator's pending
resume time. Review found that the coordinator also pushes that time
forward while it defers an ineligible candidate, so a reassigned or
blocked task could hide Retry for good
> - This pull request advertises the resume only while the scheduler's
synchronous admission checks pass
> - The benefit is one consistent story: while the automatic resume is
pending, the task shows it with its time and no competing Retry. If
recovery ends, exactly one Retry returns

## Linked Issues or Issue Description

Follow-up to #19503. Refs #19487, #19497.

**What happened?**
A native run failed with `native_session_interrupted` after a restart.
Its finalization row had `phase: retryable_failure` and `nextAttemptAt`
30 seconds later. The run projection said "Retry scheduled". The issue
interruption said `state: failed, canRetry: true, nextRetryAt: null`.

**What did you expect?**
While the automatic resume is pending, the task shows that it is
resuming, with the time, and no Retry. When the resume fails for good or
never runs, exactly one Retry that works on the first click.

**Steps to reproduce**
1. Run a native task and restart the control plane so the native attach
fails.
2. Read `GET /api/issues/:id` while the coordinator's resume is pending.
3. Observe `interruption.canRetry: true` while `execution.phase` is
`retry_scheduled`.

## What Changed

- `task-interruption-presentation.ts`: the pending-resume read now also
requires the task to be in a resumable status, assigned to the run's
agent, not owned by another execution or checkout run, the run's
question not cancelled, and the agent invokable (not paused, terminated
or pending approval). These mirror `readNativeSessionResumptionAgent`.
- `native-session-resumption-statuses.ts`: one shared status set used by
both the scheduler admission and the read model.
- No time cap: provider cooldowns and remote backoff schedules stay
visible as a pending resume.

## Verification

- `task-interruption-query.test.ts` (embedded Postgres): a reassigned
task and a blocked task show one Retry; a paused agent asks to be
resumed; in each case `readNativeSessionResumptionAgent` also refuses
the candidate, so the read model and the scheduler agree. A resume an
hour out (provider cooldown) stays `retry_scheduled` with no Retry. Red
on master for the reassigned case.
- All 57 presentation and query tests pass. Server typecheck passes.

## Risks

- Low risk. Read model only.
- A review-assignment resume (reviewer is not the assignee) shows Retry
instead of the pending resume. That is the safe fallback.
- Deferrals for a pause hold, dependency readiness or a competing run
are not mirrored. Those gates also hold a manual Retry, so no competing
action appears.
- The `POST /issues/:id/retry` endpoint is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, embedded Postgres tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:38:29 -06:00
DottaandClaude Opus 5.5 4938a930d6 fix(budgets): stop pending or restart-lost accounting from hard-stopping companies (#19497)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Budget admission and run accounting decide whether any task in a
company may start
> - A control-plane restart that kills a provider turn mid-flight can
lose that turn's final usage receipt. ACPX Claude reports usage only at
turn end, so the receipt never becomes complete
> - Accounting kept such a run "awaiting receipt" forever. Every later
admission in the company then refused with "completed runs await
accounting", and the budget evaluator opened a hard-stop incident that
paused the company. No product action could clear it: raising the budget
refuses while runs await accounting, accounting retry finds no new data,
and allowing unpriced usage does not apply to pending runs
> - Restart chaos waves showed this as a dead task Retry, a new task
blocked as "over budget" at 22 of 500 cents, and a hard stop that
cancelled queued Retries and an unrelated running task
> - This pull request closes the debt from evidence: once native
recovery reaches terminal failure and the provider stop is verified, the
known part settles as an unpriced charge. Until then the run holds new
work as recovering accounting, without a hard stop
> - The benefit is that a restart no longer bricks a company with a hard
budget. The operator's unpriced-usage policy decides admission, and
unknown spend still never shows as zero cost

## Linked Issues or Issue Description

Refs #19399, #19426, #19431 (earlier restart Retry fixes; this is the
budget layer under the same journey).

**What happened?**
After a control-plane SIGKILL during an ACPX Claude turn, the killed run
ended `native_provider_terminal_failed` with
`usage_json.accountingReceiptReady = false` and `cost_accounting_pending
= true`. It stayed that way for good. The task Retry was accepted but
parked with `budget.blocked` ("Company cannot start work while completed
runs await accounting"), which the UI showed as "Retry is waiting for
execution checks". A new Codex task was escalated to `blocked` with "the
original recovery target is over budget" while the company had spent 22
of 500 cents. After the next restart the budget evaluator opened a
hard-stop incident (`amount_observed` 22.08 < 500), paused the company
and cancelled both queued Retries.

**What did you expect?**
A restart should not stop the company. Unknown usage from a stopped
provider should become an honest, operator-governed state, and Retry
should start a run.

**Steps to reproduce**
1. Give a company a hard-stop budget.
2. Start an ACPX Claude task and SIGKILL the control plane mid-turn, so
the resumed turn fails.
3. Observe the run stay `cost_accounting_pending` with an incomplete
receipt, and every new admission refuse.

## What Changed

- `budgets.ts`: completed runs that await accounting now hold new work
only. They no longer open a hard-stop incident, pause the company, place
an agent budget hold, or deliver Stop to running work. Only spend at the
limit, or unpriced usage under a blocking policy, does that. Chaos wave
A and a pre-restart lifecycle cell showed the wedge: a run that had just
succeeded stayed terminal for about 100 ms before its charge committed,
the evaluator paused the company in that gap, and the Stop it delivered
left a cancelled run whose receipt never closed.
- `run-cost-accounting.ts`: a failed native run is a closed receipt
source only with positive stop evidence: its finalization is
`terminal_failure` with no result (not a finalization-deadline failure),
it no longer records a process identity, and its latest server-authored
process event is `native.local_process_stopped`. One SQL predicate
serves the reconciler scan and settlement. Accounting now settles its
known part as an `unpriced` charge instead of waiting forever. The
background reconciler selects these runs. The reservation settles. Other
incomplete receipts stay pending as before. A cancelled run keeps its
existing accounting drain, so Stop recovery can still deliver its final
receipt.
- `native-local-process-stop.ts` and its transactional callers:
releasing a verified local or remote process stop wakes the accounting
worker, so the closed debt settles promptly.
- `usage-receipts.ts`: a charge settled without its receipt is marked
`accountingReceiptUnavailable`. A later complete receipt that carries
its own price (including a rate-card price) corrects it. A receipt with
no own price and unchanged usage does not reopen the hold.
- `budget-exposure.ts`: the blocking unpriced-usage policy no longer
counts a charge settled from a lost receipt with no observed usage. That
is unknown, not "usage with no reliable price". Complete receipts that
report no tokens (for example Cursor Cloud) still count. A live wave
under the default policy hard-stopped its company at 0 of 500 cents on
one such charge.
- `budget-exposure.ts`: a terminal-failure native run with an
incomplete, not capture-failed receipt counts as recovering accounting.
Admission holds new work ("New work must wait while a native run
recovers its unfinished accounting") without a hard-stop incident, a
company pause or cancellation of running work.
- `recovery/service.ts`: the stranded-work comment names the actual
budget gate instead of always saying "over budget".
- `task-interruption-presentation.ts`: a Retry parked by
`budget.blocked` says it waits for the budget, not for "execution
checks".
- `doc/SPEC-implementation.md`: documents the closed-receipt contract
next to the existing incomplete-receipt rule.

## Verification

- New embedded-Postgres suite
`server/src/__tests__/closed-native-receipt-accounting.test.ts`, built
from the persisted shape of the killed run:
- Red before the fix: the run stayed pending, `reconcileRunCosts`
accounted 0, and admission refused with "completed runs await
accounting".
- Debt stays pending with no verified stop event, with a
finalization-deadline failure, with a recorded process identity, and
while the coordinator can still produce a result.
- Under a blocking policy, a lost receipt with no usage does not
hard-stop; one with tokens still blocks. A run with a still-recorded
process identity paused the company with a hard-stop incident.
- Green after: the stopped run settles as one `unpriced` charge with the
reservation settled; admission is clear under an `allow` policy and
names "no reliable price" under `block`; a run with a recorded process
identity or a non-terminal coordinator stays pending; the closing window
holds admission without an incident or pause; a final receipt that
arrives later still enters late-receipt reconciliation.
- Pending accounting from a run that just finished holds admission with
no incident, no pause, no enforcement version bump, and an unrelated
running run stays running. Red before the `budgets.ts` change: the
company was paused.
- A later priced receipt corrects the unavailable charge to its price;
an unpriced one leaves it settled.
- `task-interruption-presentation.test.ts`: the `budget.blocked` Retry
wait message.
- `budget-enforcement-scoping.test.ts`: the accounting-hold release
cases now assert that pending accounting places no budget hold and that
admission clears after settlement.
- Existing suites pass locally: all budget suites (control publication,
enforcement scoping, enforcement worker, evaluation work, live attempts,
history, service, live usage, budget cancellation),
native-stop-accounting, explicit-native-continuation,
native-case-handoff, cost-data-quality, heartbeat-cost-accounting,
run-cost-accounting, native-usage-delivery. Server typecheck passes.
- Runner e2e cell
`lifecycle-baseline.runner-codex.local.lifecycle-dependency-restart-neutral`,
which failed on master with a false agent budget hold before any
restart, passes on this branch.
- Live restart-chaos wave (local runner, K1 SIGKILL, codex + 2× ACPX
Claude + OpenCode, 4 restarts, company policy `unpricedUsagePolicy:
allow`) with the accounting fix: every killed run settled (no
`cost_accounting_pending` left), the company stayed active, the OpenCode
Retry completed the task on the first click and the Codex Retry finished
its task. That wave also exposed a 1-second false hard stop in the
closing window (observed 0 of 500 cents) that cancelled the running
OpenCode turn; the `budget-exposure.ts` change fixes it and has its own
red/green test.
- Confirming live wave with both commits, same plan (2× ACPX Claude,
OpenCode, Codex; 4 K1 restarts): both Claude tasks passed R3 (one Retry
click started a run that completed the task), OpenCode passed R1
(continued by itself), and the Codex Retry completed its task. Zero
budget incidents, zero cancelled runs, zero runs left pending
accounting, company active; three killed runs recorded as `unpriced`
charges. The only open signature is an unrelated workspace-durability
check (`R2:pre_fault_files_lost` on the Codex long-bench task).

## Risks

- Behavior change: a stopped native run's lost receipt now becomes an
`unpriced` ledger charge with its known usage. If that usage has tokens,
the default `block` policy still stops new work with "recorded usage has
no reliable price". If it has none, the charge is visible in Costs but
does not stop the company. Other unpriced charges keep their existing
effect. Unknown spend is never recorded as a priced receipt. Budget
enforcement stays best effort, as the spec says; killed OpenCode and
Codex requests already lose their in-flight usage.
- Remote runs and runs without a verified local stop keep their debt
pending (an admission hold only, no hard stop).
- Contract change: pending accounting no longer pauses a scope or stops
running work. New work still waits with "completed runs await
accounting". A scope's real hard stop (spend at the limit, or blocked
unpriced usage) is unchanged.
- If a provider stop can never be verified, the run stays "recovering
accounting": new work waits without a hard-stop incident. Before, the
same state paused the company. The hold is visible in admission errors
and Costs.
- A receipt that arrives after settlement posts a late correction. The
original unpriced charge stays in the ledger.
- No schema or migration change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, embedded Postgres tests, live
restart-chaos harness).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:27:51 -06:00
DottaandClaude Opus 5.5 543aa4d331 fix(tasks): show a pending native same-run resume instead of a competing Retry (#19503)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native run cannot reattach to its provider after a
control-plane restart, the native coordinator resumes the same run by
itself after about 30 seconds, up to three attempts
> - The run projection shows this correctly as "Retry scheduled" with
only `inspect_run` permitted
> - The task interruption shown on the issue read only system-recovery
evidence and `scheduledRetryAt`. It never read the coordinator's
`nextAttemptAt`, so it showed `state: failed, canRetry: true` and
offered a manual Retry while the automatic resume was still pending
> - The two surfaces disagree, and a user click can race the automatic
resume. That is one way a Retry gets parked behind "execution checks"
> - This pull request makes the task interruption read the coordinator's
pending resume time
> - The benefit is one consistent story: while the automatic resume is
pending, the task shows it with its time and no competing Retry. If
recovery ends, exactly one Retry returns

## Linked Issues or Issue Description

Refs #19487, #19497, #19426.

**What happened?**
A native run failed with `native_session_interrupted` after a restart.
Its finalization row had `phase: retryable_failure` and `nextAttemptAt`
30 seconds later. The run projection said "Retry scheduled". The issue
interruption said `state: failed, canRetry: true, nextRetryAt: null`.

**What did you expect?**
While the automatic resume is pending, the task shows that it is
resuming, with the time, and no Retry. When the resume fails for good or
never runs, exactly one Retry that works on the first click.

**Steps to reproduce**
1. Run a native task and restart the control plane so the native attach
fails.
2. Read `GET /api/issues/:id` while the coordinator's resume is pending.
3. Observe `interruption.canRetry: true` while `execution.phase` is
`retry_scheduled`.

## What Changed

- `task-interruption-presentation.ts`: the batched run read selects the
native coordinator's `next_attempt_at` when the native run failed, has
no result, and its finalization phase is `retryable_failure`.
- A pending resume in the future shows `retry_scheduled` with its time
and `canRetry: false`.
- A due resume shows `working` for up to two minutes while its dispatch
runs. After that, or once recovery is terminal, the task falls through
to the existing single Retry.

## Verification

- `task-interruption-presentation.test.ts`: new case for a future
resume, a due resume, a stale resume, and no resume. It fails before the
change (the task offered Retry) and passes after.
- `task-interruption-query.test.ts` (embedded Postgres): the real
batched query shows the pending resume for both list and detail reads,
and returns one Retry after the coordinator becomes `terminal_failure`.
- All 40 presentation tests and 17 query tests pass. Server typecheck
passes.

## Risks

- Low risk. This changes only the read model for tasks whose native
coordinator holds a pending same-run resume.
- If a resume never dispatches, Retry is hidden for up to two minutes
after its due time before it returns.
- The `POST /issues/:id/retry` endpoint is unchanged. A client that
ignores `canRetry` can still submit a Retry during the resume.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, embedded Postgres tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 11:33:46 -06:00
DottaandClaude Opus 5.5 140fcf98ed fix(retry): name a declined task Retry and let a new click replace it (#19426)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task interrupted by a restart or deploy must resume automatically,
or offer a Retry that works on first use
> - A board Retry saves an intent first. Dispatch then rechecks the
requester's authority and saved source before it starts a run
> - When that recheck refused the intent, dispatch returned a bare
"deferred". The intent stayed saved with no reason, and the task still
showed Retry
> - Each later click with a new key joined that same intent, so Retry
returned 202 and never started a run
> - This pull request names the refusal on the intent and lets a new
click replace an intent that can never start
> - The benefit is that a Retry button that is shown starts one run, or
explains why it cannot

## Linked Issues or Issue Description

**What happened?**
A native task in a managed sandbox lost its runner twice. Saved-work
recovery recovered the files. The task showed "Saved work recovered.
Retry or send a message to continue." and `canRetry: true`. A Retry
click returned 202 and logged `issue.retry_requested`, but no run
started. A second click with a new idempotency key returned the same
request ID and `state: failed`. No run started.

**Expected behavior**
If Retry is offered, a click starts exactly one run. If the Retry cannot
start, the task names the reason, and a later click is a fresh attempt.
A Retry accepted during the pre-deploy drain starts after the restart.

**Steps to reproduce**
1. Recover the files of a failed native run so the hold offers Retry.
2. Save a Retry intent whose dispatch-time authority recheck fails (for
example, the requester lost access).
3. Dispatch it: no run starts, and the intent has no wait reason.
4. Click Retry again with a new key: the response returns the first
request ID, and no run starts.

**Paperclip version or commit**
Reproduced on `b400914345` (staging served `4d49a31286`; the affected
code is the same).

Related: #19399 (automatic Retry after a lost runner), #19422.

## What Changed

- `heartbeat/queue.ts`: when dispatch declines a task Retry because the
requester can no longer change the task, or its saved board message
changed, record a named wait (`retry_authority_denied`,
`retry_source_changed`) and log it. Before, it returned "deferred" with
no record.
- `task-retry.ts`: a new click with a new key replaces an unadmitted
intent that dispatch declined for its requester's access
(`retry_authority_denied`). The old intent becomes `skipped` /
`task_retry_replaced`, and activity records `replacedRequestId`. Waits
on shared state (cleanup, recovery, an active run, preservation) still
coalesce. The restart hook never replaces a user's click. Chat-owned
intents keep their own owner.
- `task-interruption-presentation.ts`: fixed public messages for the two
new wait reasons. `retry_authority_denied` keeps Retry, because a new
click is a fresh attempt. `retry_source_changed` is terminal: an edited
saved message cannot be replayed, so Retry is hidden and the user sends
a new message.
- `doc/execution-semantics.md`: document the rule.

## Verification

- New regressions in `server/src/services/workspace-recovery.test.ts`
use the real recovery, Retry and dispatch services on embedded Postgres.
The fixture runs the automatic Retry from a lost runner, then that
Retry's own run loses its runner, and recovery saves its files:
- `starts exactly one run when Retry is pressed after the automatic
Retry's own run lost its runner` (also checks that a replayed dispatch
and a second click start no second run).
- `does not let a Retry that could not start swallow a later Retry`: red
before the fix in two steps. With no fix, the declined intent has no
wait reason. With only the wait fix, the second click returned the first
request ID. Green after the fix: one run, bound to the second click.
- `dispatches a Retry accepted during the pre-deploy drain once the
restarted server resumes`: green before and after. The drain does not
lose a saved Retry.
- `npx vitest run src/services/workspace-recovery.test.ts
src/services/task-retry.test.ts
src/services/task-interruption-presentation.test.ts
src/services/quarantined-task-retry.test.ts
src/services/explicit-native-continuation.test.ts`: 892 passed.
- Not verified on staging. After deploy, an intent that dispatch refuses
shows a named `retryWait` reason.

## Risks

- Low risk. Only intents that dispatch declined for their requester's
access are replaced. All other coalescing stays.
- A replaced intent's late dispatch finds it `skipped` and starts
nothing, so a double click still starts at most one run.
- The exact gate that held the staging intent is not confirmed from
records. If it was a shared-state wait, the new code names it only if it
is one of the two reasons above.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 03:52:21 -06:00
DottaandClaude Opus 5.5 a4bb059f59 fix(heartbeat): retry transient sandbox provider lease refusals and explain the block (#19412)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task runs in an execution environment. A managed sandbox
environment gets a lease from a sandbox provider (for example Daytona)
before the agent starts.
> - When the provider answered sandbox create with a temporary error
(HTTP 502), the run failed as the generic `setup_failed`.
> - `setup_failed` is the fail-closed code for unknown setup errors. It
gets no automatic retry, so the task went to `blocked` at once. The task
showed only "Couldn't finish this attempt".
> - A user had to find and press Retry. The Retry then worked, which
shows that the failure was temporary.
> - This pull request classifies temporary provider refusals during
lease acquisition from structured fields. It retries them within the
existing bounded bootstrap budget, and it explains the block when the
retries are used up.
> - The benefit is that a short provider outage no longer stops a task.
When the outage continues, the task shows the cause and a Retry that
works.

## Linked Issues or Issue Description

**What happened?**
A task was assigned to an agent that uses a managed Daytona sandbox
environment. Sandbox creation returned `HTTP 502`, and the cleanup of
the provisional sandbox could not be confirmed. The run failed with
`setup_failed` and the error `Failed to acquire lease for environment
"Paperclip Sandbox" (sandbox): Daytona sandbox creation failed (HTTP
502); cleanup could not be confirmed for paperclip-create-…`. Three
seconds later the task was `blocked`. No automatic retry was scheduled.
The task showed no specific reason. A manual Retry started a new run
that completed normally.

**Expected behavior**
A temporary provider error (HTTP 5xx, 429, timeout) before the agent
starts must retry automatically with bounded backoff. The task must show
why it waits. If the retries are used up, the task must show the cause,
a next action, and a Retry that works on the first click.

**Steps to reproduce**
1. Configure an agent with a sandbox environment.
2. Make the provider return HTTP 502 on sandbox create.
3. Assign a task to the agent.
4. See that the run fails with `setup_failed` and the task goes to
`blocked` with no retry.

**Paperclip version or commit**
`master` at `612da1e8e0`.

## What Changed

- Plugin SDK: an `environmentAcquireLease` error now carries a closed
diagnostic envelope (HTTP status or a closed errno code) across the
worker RPC. It takes the status of the create call only, not of the
compensating cleanup lookup. The existing creation-cleanup ownership
envelope is unchanged. New module: `environment-acquire-error.ts`. The
resume diagnostic helpers are shared.
- Server: new `EnvironmentProviderUnavailableError` and code
`environment_provider_unavailable`. The environment run orchestrator
sets it as the cause of `lease_acquire_failed` only when structured
evidence shows a transient status (408, 425, 429, 500, 502, 503, 504, or
a closed connection errno). It never parses message text. Saved-sandbox
resume and stopped-worker refusals keep their own codes.
- Setup failure (`failRunSetup`): a fresh, pre-dispatch run records
`environment_provider_unavailable` with a `providerFailure` (phase
`startup`, retryability `transient`). The run uses the bounded bootstrap
retry (30 s, then 60 s). An accepted interaction continuation keeps its
own continuation retry, as for `plugin_worker_unavailable`.
- Recovery: the new code is registered with the other bootstrap codes.
It is not sent to the generic continuation retry. It does not create an
execution-reconciliation hold, because no provider work started. On
exhaustion the stranded notice has the title "Sandbox provider was
temporarily unavailable" and a specific next action.
- UI: `ProviderFailureNotice` labels the `sandbox` provider as "Sandbox
provider". The task notice shows the provider failure, "Paperclip will
retry automatically" while a retry is scheduled, and Retry after
exhaustion.
- Docs: `doc/execution-semantics.md` describes the policy and the
sandbox ownership rule.

## Verification

Red before the fix, green after the fix:

- `server/src/__tests__/heartbeat-process-recovery.test.ts` › "transient
sandbox provider refusal during lease acquisition":
- "retries automatically and runs exactly one successful successor
without user action": the real heartbeat. The lease runtime returns a
Daytona 502 once, then delegates. Before the fix: one `failed` run and
no successor. After the fix: one `scheduled_retry` successor with no
hold and a `retry_scheduled` presentation. After promotion, exactly one
successor succeeds.
- "explains the block with a working Retry once automatic retries are
used up": three refusals. The task is `blocked` with a "Sandbox provider
was temporarily unavailable" notice. The presentation is `{ state:
"failed", canRetry: true, providerFailure: { httpStatus: 502 } }`, and
no execution-reconciliation hold exists. One `requestTaskRetry` is
accepted, and its run succeeds.
- `server/src/services/heartbeat/run-completion.test.ts` › "routes a …
sandbox provider lease refusal through bounded bootstrap recovery": the
real orchestrator wrapping. Only the fresh, structured 502 is retried. A
message-only 502, a 400, a run after provider dispatch, and a retained
native run stay `setup_failed`.
- `packages/plugins/sdk/tests/environment-acquire-error.test.ts`: the
real worker RPC. The test checks the create status (not the 404 cleanup
lookup), the errno, no provider text, the unchanged cleanup envelope,
and that forged envelopes are rejected.

Commands:

```sh
pnpm --filter @paperclipai/plugin-sdk exec vitest run tests/environment-acquire-error.test.ts tests/environment-creation-cleanup.test.ts tests/environment-resume-error.test.ts
cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts src/services/recovery/ src/services/heartbeat/run-completion.test.ts src/services/heartbeat/retries.test.ts src/services/legacy-execution-recovery.test.ts src/services/task-interruption-presentation.test.ts src/services/task-retry.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts
cd packages/plugins/sandbox-providers/daytona && npx vitest run
pnpm --filter @paperclipai/plugin-sdk typecheck && pnpm --filter @paperclipai/server typecheck
```

I did not run a live Daytona 502 on staging. Re-check on staging after
deploy (see Risks).

## Risks

- Sandbox leaks or double creation: each retry is a new run with a new
durable creation record and a new provider name
(`paperclip-create-<operationId>`). The failed attempt stays owned by
its own record, and the existing creation reconciler settles it. A retry
never reuses or abandons it. While a provisional sandbox waits for
reconciliation, up to two more sandboxes can be created for the same
task. This is the same as a manual Retry today.
- Plugins built with an older SDK do not send the envelope. Their
refusals stay `setup_failed`, which is the current behavior.
- An in-process error with `status`/`statusCode` 500–504 or a connection
errno during lease acquisition is now retried twice. The budget is
bounded and shared with the other bootstrap codes.
- An RPC timeout on the host side is not classified by this change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code as an agent with tool use (shell, file edits, test runs).
Extended reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 00:08:44 -06:00
DottaandClaude Opus 5.5 9ceaec6f97 fix(issues): stop showing resource waits as failed attempts (#19369)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The task page shows one interruption banner. The server builds it
from the task's latest run.
> - An auto-retry that waits for a busy workspace, AI connection or
runner update is not a failure. The dispatcher re-checks the resource,
so the run can stay `scheduled_retry` after its retry time.
> - The projection did not know about these waits. After the retry time
passed, it fell through to "Couldn't finish this attempt" with a Retry
button.
> - Retry cannot free the resource. The banner cleared for the 10-second
grace and then came back. Users had no way to fix it.
> - This pull request projects a scheduled resource wait as `working`,
with no Retry.
> - The benefit is that the failed-attempt banner shows only for real
failures.

## Linked Issues or Issue Description

**What happened?**
During onboarding on Paperclip Cloud staging (Claude subscription and
Codex subscription paths), the onboarding task showed "Couldn't finish
this attempt. Your message is saved." with a Retry button. The banner
did not name an attempt. Retry cleared it briefly, then it came back.

**Steps to reproduce**
1. Start a task run while another run holds the same workspace.
2. The run is cancelled with `workspace_busy`, and a `scheduled_retry`
run is created with `scheduledRetryReason: "workspace_busy"`.
3. Keep the workspace busy after `scheduledRetryAt`. On the staging
company, one retry stayed scheduled for 16 minutes past its retry time.
4. Open the task. The failed-attempt banner shows. Click Retry. The
banner comes back.

**Expected behavior**
No failed-attempt banner while the system waits for a shared resource.
The run status already says "Waiting for workspace"
(`execution-projection.ts`).

## What Changed

- `server/src/services/task-interruption-presentation.ts`: a
`scheduled_retry` run whose `scheduledRetryReason` is `workspace_busy`,
`ai_connection_busy`, `ai_connection_pool_wait` or
`runner_revision_wait` (and that has no system recovery) projects as
`state: "working"`, `canRetry: false`. User gates and failed Retry
preparations stay authoritative because they are checked first.
- The batched query now selects `scheduledRetryReason`.
- Tests: a Postgres integration test (`task-interruption-query.test.ts`)
that seeds the staging run chain for each reason, and unit tests for the
guard cases (failure-lane retries, paused agent, failed Retry
preparation).

## Verification

Red on `master` (fix not applied), `cd server && npx vitest run
src/__tests__/task-interruption-query.test.ts -t "overdue"`:

```
     × does not present an overdue workspace_busy wait as a failed attempt 213ms
     × does not present an overdue ai_connection_busy wait as a failed attempt 25ms
     × does not present an overdue ai_connection_pool_wait wait as a failed attempt 13ms
     × does not present an overdue runner_revision_wait wait as a failed attempt 11ms
- Expected
+ Received
-   "canRetry": false,
+   "canRetry": true,
-   "state": "working",
+   "state": "failed",
      Tests  4 failed | 11 skipped (15)
```

Green with the fix, same tests plus the new unit tests:

```
 ✓ task-interruption-query.test.ts > does not present an overdue workspace_busy wait as a failed attempt
 ✓ task-interruption-query.test.ts > does not present an overdue ai_connection_busy wait as a failed attempt
 ✓ task-interruption-query.test.ts > does not present an overdue ai_connection_pool_wait wait as a failed attempt
 ✓ task-interruption-query.test.ts > does not present an overdue runner_revision_wait wait as a failed attempt
 ✓ task-interruption-presentation.test.ts > scheduled resource wait presentation > keeps an overdue workspace wait out of the failed-attempt UI
 ✓ task-interruption-presentation.test.ts > scheduled resource wait presentation > keeps failure retries, user gates and failed Retry preparation authoritative
```

Neighbouring suites: `task-interruption-presentation`,
`task-interruption-query`, `workspace-recovery`, `agent-project-access`,
`heartbeat-dependency-scheduling`: 5 files, 458 tests passed.

Server `tsc --noEmit` reports no errors in changed files. The only
errors locally come from the unbuilt `@paperclipai/paperclip-runner`
vendor package, which this change does not touch. No UI change, so `pnpm
check:token-gates` is not applicable.

Evidence: the staging onboarding task's run history shows
`workspace_busy` cancellations at 17:46, 17:52 and 18:10, each followed
by a `scheduled_retry`. One had `scheduledRetryAt` 17:53:57 and started
at 18:10:20. The next run succeeded at 18:20.

## Risks

- Low risk. Presentation only; dispatch and admission do not change.
- If a resource wait never clears, the task shows no failed-attempt
banner. The run status still shows "Waiting for workspace", and the
existing retry-attempt limits still end the wait.
- Not in scope: genuine failures without a provider or workspace
explanation still show the generic text, and an overdue failure-lane
retry still shows as failed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser for read-only staging evidence).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 19:08:27 -06:00
b929ad43de fix: keep dependency waits neutral and explain GitHub access failures (#19315)
## Thinking Path

> - Paperclip manages agent work and shows users what can move it
forward.
> - Task status and tool approval cards describe different parts of that
work.
> - A queued dependency wait has no provider attempt to retry.
> - GitHub can reject an approved action while other connection tools
still work.
> - These cases showed a false Retry banner and an unhelpful provider
error.
> - This pull request keeps dependency waits neutral and gives GitHub
access failures a direct repair path.

## Linked Issues or Issue Description

Related: #19268 preserves pending Retry preparation and current failure
causes.

**What happened?**

A queued wake cancelled by the dependency gate appeared as a failed
attempt with Retry. A GitHub MCP permission denial appeared as raw HTTP
error text after an approved repository action.

**Expected behavior**

Keep the task's dependency state. Show GitHub access failures with a
clear explanation and a link to the selected connection's Permissions
page. Preserve actual failures, provider details, and approval gates.

**Steps to reproduce**

1. Queue a wake, then leave its task dependent on unfinished work before
dispatch.
2. Open the task after the dependency gate cancels that wake.
3. Separately approve a GitHub action whose MCP result contains `403
Resource not accessible by integration`.
4. Reopen the saved tool approval.

## What Changed

- Recognize both queues' unstarted dependency-wait receipts in
cancellation provenance and task projections. No dispatch or recovery
gate changes.
- Classify GitHub's specific permission error after MCP result
normalization. Keep result validation, the raw diagnostic, and working
tools available.
- Apply the same display code to both historical GitHub error codes.
Identify the provider from company-scoped connection metadata, including
renamed apps, without rewriting stored decisions.
- Show a token-based inline access card with **Manage GitHub access**
and collapsed **Error details**.
- Give agents trusted recovery steps after an approved GitHub action
fails. Keep provider text in the untrusted result and prevent automatic
replay.
- Add regressions, a production-component Storybook case, and
execution/connector documentation.

## Verification

- Red before the fix: cancellation provenance, dependency-wait
projection, the real queue cancellation, approved MCP failure
classification, saved interaction projection, and the error card
assertions failed at the expected behavior.
- Green focused tests: queue cancellation, one ready-task executor,
approval-before-dispatch, one failed GitHub request, connection reuse,
retained provider diagnostics, saved interaction reads, task list/detail
projection and redaction.
- UI: 106 component tests passed. Token gates and Storybook catalog
checks passed.
- `node scripts/typecheck-low-memory.mjs`: passed.
- Local source instance: verified a saved dependency wait stays neutral
after reload; a saved GitHub error shows repair guidance, expands its
original error, and opens the correct connection Permissions page.
PostgreSQL and HTTP APIs were real. The GitHub denial was a controlled
MCP fixture; no account or provider permission changed.
- `pnpm build`: passed for all workspace packages, the Runner binary,
server, CLI and UI. The server build and all server typecheck layers
also passed after the review repair.
- Additional safety checks passed for realistic repository arguments,
active recovery, failed Retry preparation, and bounded provider
diagnostics.
- Review repair: 45 pure tests and 7 PostgreSQL cases passed. These
cover both historical error codes, renamed apps, and
company/provider/application identity boundaries. A second local browser
check verified the renamed app, retained error detail, and correct
Permissions destination.
- Final review regressions: the dispatcher and metadata cases failed
before the repair. Afterward, 25 focused cases and 59
projection/identity cases passed, including connection-key precedence.
The native wait and app-only GitHub source also passed a local browser
check.
- The historical read adds at most one connection query per interaction
read batch. It selects IDs and the provider key only and runs only for
candidate permission failures. It adds no provider requests.
- Full local `pnpm test:run` stopped at the chat suite's `afterAll`
cleanup timeout. All 268 assertions in that group and the three prior
chat groups passed. The failure is retained here; it is not counted as a
full-suite pass. CI passed on `e8a692e1e2` after one workspace-startup
rerun. Fresh CI is pending for the combined head. Its first workspace-a
attempt timed out in the CLI company import/export test's child-server
startup wait; 4,968 UI tests and 287 CLI tests passed in that job. The
same shard passed on the preceding commit. The workspace shard passed on
its one rerun. All other test, build and browser jobs passed. The [first
failure](https://github.com/paperclipai/paperclip/actions/runs/42093823928/job/127478869941)
is retained in the Actions history. Greptile reviewed the preceding head
at 5/5. Fresh review is pending for the combined head.

## Risks

The GitHub classifier matches the documented permission-denial phrase,
not every 403. Rate limits, missing resources and other providers retain
their existing error behavior. Historical normalization affects display
only. A dependency cancellation needs an unstarted run and a supported
queue receipt; current user gates, Retry preparation and recovery remain
authoritative. No schema migration or credential changes.

## Model Used

OpenAI GPT-6, through the Codex agent. The exact runtime model ID and
context-window size are not exposed in this session. Used reasoning,
repository tools, TypeScript, PostgreSQL, MCP fixtures and browser
verification. The parallel agent-recovery commit records Anthropic
Claude Opus 5.5 as co-author.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with version and capability
details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the bug template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change
- [x] I have run the focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open findings
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 20:55:34 +00:00
DottaandClaude Opus 5.5 629abc8509 fix(recovery): keep pending Retry unclickable and scope failure causes (#19268)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - #19260 shows the saved-work recovery step on a task, and records why
a recovery attempt failed.
> - The review of #19260 found three gaps after the PR merged.
> - A scheduled recovery offered Retry while a Retry preparation was
already pending.
> - Classified failures could get a misleading `provider_error` cause.
> - The new cause text in the board handoff had no test.
> - This pull request closes these gaps.
> - The benefit is a Retry button that matches what is allowed, and
causes that name the correct subsystem.

## Linked Issues or Issue Description

Refs #19260

**What happened?**
After #19260, a task with a scheduled recovery and a pending (`issued`
or `processing`) Retry preparation showed an active Retry button. The
server only coalesces the second request. Also, a failure that recovery
classified as `archive_unsafe`, `source_unavailable` or
`snapshot_verification_failed` still recorded a provider cause. The
handoff text could then read "last failure: archive_unsafe, cause:
provider_error".

**Expected behavior**
Retry is not offered while a Retry preparation is pending. Only the
generic transient fallback records a provider cause.

**Steps to reproduce**
1. Request a chat Retry on a task while its saved-work recovery is
scheduled.
2. The task projection returns `retry_scheduled` with `canRetry: true`.

**Paperclip version or commit**
master at b1d00eec.

**Deployment mode**
Authenticated cloud deployment (staging).

## What Changed

- `task-interruption-presentation.ts`: a scheduled recovery returns
`canRetry: false` when a Retry preparation for the same run is pending
with a valid deadline.
- `workspace-recovery.ts`: a cause is recorded only for the generic
transient fallback (`provider_unavailable` or `export_interrupted` from
an unclassified error).
- Tests: presentation `canRetry` with and without a pending preparation;
the board handoff text with and without a cause.

## Verification

- `task-interruption-presentation.test.ts`,
`system-recovery-state.test.ts` and `workspace-recovery.test.ts` pass
(419 tests). One workspace-recovery case failed once under full-suite
load. It passes when it runs alone.
- Server typecheck passes.

## Risks

- Low risk. This change affects presentation and diagnostic labels only.
Retry admission still checks all gates on the server.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 11:14:24 -06:00
DottaandClaude Opus 5.5 01dd6ab336 fix(recovery): show the saved-work recovery step and record why it failed (#19260)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Sometimes a failed run left files in a sandbox. Then Retry first
runs system recovery to save those files before the task continues.
> - Recovery has steps: confirm that the previous attempt stopped, save
its files, then finish.
> - The task page did not show these steps. People saw "Queued" or
"Trying again in…" with a retry time that kept moving.
> - Each unclassified failure of the stop-confirmation step was recorded
only as `provider_unavailable`, so a repeated deferral could not be
diagnosed.
> - This pull request shows the current recovery step on the task. It
also records why each failed attempt failed, in closed labels.
> - The benefit is that people can see what Retry waits for, and
operators can see why recovery deferred.

## Linked Issues or Issue Description

Refs #19249

**What happened?**
A person clicked Retry on a task whose failed run had a retained
sandbox. For about 7 minutes the task showed only "Queued", and the
retry time moved forward twice. Recovery deferred twice in the
stop-confirmation step. Each deferral was recorded only as
`provider_unavailable` with an opaque fingerprint, so the cause could
not be found.

**Expected behavior**
The task shows the recovery step that it waits for, such as "Checking
that the previous attempt stopped. Trying again in 30s…". Each failed
attempt keeps a closed-label cause, for example `sandbox_not_running` or
`transport_timeout`, with an optional errno-style code and HTTP status.

**Steps to reproduce**
1. Get a remote task run to fail after its sandbox was placed, so that
recovery owns saved work.
2. Click Retry while the sandbox provider does not respond to the stop
check.
3. The task shows "Queued" or "Trying again in…". The recovery evidence
records only `provider_unavailable`.

**Paperclip version or commit**
master at 499712c6.

**Deployment mode**
Authenticated cloud deployment (staging).

## What Changed

- `system-recovery-state.ts`: a failed attempt can keep an optional
`failure.cause` with a closed `label`, an errno-style `code` and an HTTP
`status`. The schema drops a value that does not match.
- `workspace-recovery.ts`: an unclassified failure records
`sandboxProviderFailureLabel(error)` as its cause, and the warning log
includes the label. The cause never contains provider text.
- `system-recovery.ts`: the board handoff message after exhaustion names
the last cause.
- `TaskInterruptionPresentation` has an optional `recoveryStep`:
`checking_previous_run`, `saving_previous_files` or
`finishing_recovery`. The server sets it from the recovery phase only
for a scheduled or actively running system recovery.
- `ExecutionBlockerNotice` shows the step. While recovery runs it shows
"Saving files from the previous attempt…". While a recovery retry is
scheduled it shows "Checking that the previous attempt stopped. Trying
again in 30s…". Without a recovery step, the output does not change.

## Verification

- Red, then green: the new tests in `system-recovery-state.test.ts`,
`task-interruption-presentation.test.ts` and
`ExecutionBlockerNotice.test.tsx` failed before the change and pass
after it.
- `workspace-recovery.test.ts` (380 tests) passes. Its exhausted
unknown-failure case now expects `cause: { label: "provider_error" }`.
- An ordinary scheduled retry, a working run without recovery and the
existing failed states render as before.
- Server typecheck passes. UI typecheck passes for `src`. The only UI
errors are in Storybook support files whose generated catalog is not
built locally.
- `pnpm check:token-gates` passes.

## Risks

- Low risk. All new fields are optional JSON fields. No database
migration is necessary, and old records parse as before.
- This change does not change recovery behavior, retry timing or
authority. It adds presentation and diagnostics only.
- The cause label uses the existing closed transport label set. An
unrecognized error is recorded as `provider_error`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:03:25 -06:00
DottaandPaperclip b6352578c5 fix(task-retry): retain chat Retry during identity contention (#18896)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task Retry must preserve a person's request and enforce the current
chat authority.
> - Chat retry preparation takes nonblocking source locks before it can
authorize the saved source.
> - Sustained lock contention could discard the Retry request before it
was saved.
> - This pull request saves one unadmitted request in the existing chat
retry queue.
> - The worker checks the source and current access before normal
admission. The task shows a scheduled retry while it waits.

## Linked Issues or Issue Description

**What happened?**

Task Retry returned HTTP 500 when another transaction held the saved
chat principal's identity lock or the endpoint authorization row. The
source preparation error rolled back the canonical Retry intent. The
same gap existed when Retry had to restore a certified input batch that
had never reached a provider. With the bounded synchronous retries in
#18895, sustained contention returns 409 without a saved intent.

**Expected behavior**

HTTP 202 retains one Retry request. Acceptance does not grant source
authority or mean that a provider started. The existing worker prepares
the request if the source locks become available before its saved
deadline. It must reject changed input, revoked access, Stop, or
superseding work before input adoption, publication, or provider work.

**Steps to reproduce**

1. Prepare a failed task with its original admitted GitHub chat source.
2. Hold that principal's `chat-identity` advisory lock or the endpoint
row on a second PostgreSQL connection.
3. Send the normal task Retry HTTP request. Check the response and
persisted intent.
4. Release the lock. Process the existing chat retry queue and check
that one authorized successor is queued.

**Paperclip version or commit**

Reproduced on master `8771606fd87f69ed4057779e3dce3a9bca8088e6`.

**Deployment mode**

Source checkout, real Express Retry route and embedded PostgreSQL.
External chat transport is a test fixture. The live hosted fix has not
been deployed or verified.

Related: #18895 handles brief admission contention with bounded
synchronous attempts and returns 409 if it remains busy. This change
retains a durable request through sustained identity contention. It uses
the existing background retry owner and adds real HTTP, restored-input,
Stop, publication, restart, and task-presentation coverage.

## What Changed

- Preserve a strict pending-source action only when the identity
try-lock reports contention or the endpoint authorization query reports
lock-unavailable through an explicit Retry callback. Bind it to the
canonical task Retry, company, task, agent, endpoint, conversation, and
principal. Other database and authorization errors keep their existing
behavior.
- Revalidate the canonical intent, current board permission, saved
source, and latest execution before the existing worker can prepare that
action. Ordinary authorization rejects an unprepared action.
- Keep certified input restoration inside source preparation. Recheck
the saved deadline before the prepared payload write. Expiry, a nested
savepoint, or outer rollback cannot commit partial restoration.
- Give source authorization contention one saved 60-second deadline and
a separate counter. Keep its waiting phase until current source
authorization succeeds or the action terminates. It does not spend the
eight delivery attempts. Duplicate clicks, worker restart, and
background sweeps do not renew either limit. Reuse the existing work
notifications and outbox deadlines.
- Persist completion of the authorization wait under the current claim
before delivery. A restart revalidates current authority. Generic
delivery failure preserves the stored phase even when a commit
acknowledgment is lost; it does not grant a wake or renew the saved
deadline, counters, or claim.
- Project pending preparation and prepared but unsettled actions as the
existing scheduled-retry state with the actual outbox deadline. Keep the
exact task, intent, agent, and source binding. Active execution keeps
its normal working state. No UI or API enum change is needed.
- Add regressions and document the admission boundary.

## Verification

- RED: the original real HTTP route returned 500 under a held PostgreSQL
identity lock. The restored-input case also returned 500 while its 60
sibling source and budget cases passed. The current endpoint-row case
returned 409 without a saved intent under the bounded synchronous
fallback.
- RED: the worker failed a valid Retry during an actual nine-second
PostgreSQL lock. Removing only the latest-run guard makes the new
queued-run negative fail. Source preparation could also commit restored
input after its deadline, and an already prepared action could process
after a late lock release. The prepared but unsettled task view lost its
scheduled state.
- GREEN: all 91 selected real PostgreSQL integration cases passed.
Coverage includes the 62-case saved-input matrix; coalescing; restart;
one real heartbeat successor; source, actor, Stop, newer-work and
publication refusals; a real nine-second lock; fixed-deadline
exhaustion; the separate eight delivery attempts; certified restoration
rollback on expiry; prepared-action early and late release; a real
conversation-row wait that crosses the deadline during current
authorization; endpoint-row contention; unknown source; an
already-revoked actor; unchanged unrelated ingress errors; durable-work
idle settlement; actual outbox presentation deadlines; and recovery
after authorization commits, with Stop and a lost commit acknowledgment.
- GREEN: all 75 cases in `task-retry.test.ts`,
`task-interruption-presentation.test.ts`,
`chat-failed-run-retry-boundary.test.ts`, and `chat-action-work.test.ts`
passed without a name filter.
- RED → GREEN: stopping after source authorization committed left a
stale waiting flag, so a fresh worker failed a previously authorized
Retry after the deadline. The fix saves that phase in the same
transaction. A real PostgreSQL commit followed by an injected lost
acknowledgment also reproduced generic settlement restoring the stale
flag. The corrected settlement preserves the stored phase. No wake is
sent on the ambiguous response. Recovery queues once after current
checks; Stop still prevents a successor. The deadline, contention count,
original input, failed run, and publications stay unchanged.
- Module boundary checks pass. Full workspace typecheck and the full
build passed at `d8a4a1a739aa6825714b04691d4166a00991808b`. At this
exact head, both required checks (`ci / verify` and `ci / e2e`) pass.
The inventory has 53 successful, three skipped, and one neutral check,
plus the automatic Paperclip Review in progress. The legacy Snyk status
passes. Greptile is 5/5 on this head, with no new comments or unresolved
threads. A current Paperclip assessment has not been published yet.
- The full local `pnpm test:run` did not pass: its second chat shard hit
a 15-second timeout in the HTTP/restart test after 348 and 293 passing
cases. One bounded diagnostic of the same 642-case shard pair passed,
with that case completing in 606 ms. The matching Linux CI pair passed
too. The original timeout remains unexplained and nonreproduced. No
timeout changed. The earlier stopped run was incomplete, and both
conflicting documentation additions are retained.
- CI chat 3/3 exposed a separate fixture assumption: restoration can
commit before a durable wake receipt exists. One known pre-receipt
delivery failure reproduces the same missing-run error. The corrected
fixture drives the existing worker until the exact canonical intent and
wake agree. It proves one real wake, one successor, exact source/actor
binding, and one restoration. The exact failed shard pair now passes
locally: 635 cases. Workspace typecheck also passes at test-only head
`94d691c9df61b05019b96900d5800a8f1c484e9a`. Production source is
unchanged from the previous verified build. That head completed all
registered checks: 53 successful, two neutral, three skipped, plus the
successful legacy status. Greptile scored 5/5. The subsequent Paperclip
4/5 persistence finding is addressed by the source repair above; its new
exact-head review is pending.
- Compatibility: 111 selected cases passed with #18895 head
`9b5d877161ffcbf16c0fb6920e8155a3620c85b0` in a disposable worktree. The
branch is now synchronized with master `4a83ba4050`, which includes that
change.
- Preserve the old CI failure: at head
`37954951fb66f3b50d0f2a3bec78d6993c479e87`, server shard 6/12 initially
failed the empty organization-audience replacement versus member-removal
assertion in `tool-access-service.test.ts`. The exact race family passed
on unchanged base and head; all 501 file tests and two 30-case controls
passed on base. The one Linux replay passed 119 files and 2,833 cases,
with three skipped. No guard or assertion changed. The initial failure
remains unexplained and nonreproduced; this PR does not claim to fix it.
- The prior external Superagent scan could not start because its
Security Graph returned HTTP 503. Its neutral check was not a completed
security assessment.
- These tests exercise the real Express route, PostgreSQL transactions,
and heartbeat queue. External chat transport is a fixture. They do not
run a live model or prove recovery of an original hosted task.

## Risks

- A still-pending source authorization wait fails after its fixed
60-second contention deadline. A completed wait is saved, but new
contention keeps the original deadline. A healthy lock can exceed that
limit. Other delivery errors retain the eight-attempt budget. Current
authority checks remain required before admission. Inspection does not
reset either limit.
- The new pending phase must never be treated as an authorized chat
source. Normal authorization rejects it until the worker prepares and
binds the exact source.
- The task projection reads the existing linked outbox deadline. It adds
a scoped join and does not create a timer or change a business deadline.
- No schema migration, provider identity change, or new delivery queue
is included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository editing, tool
use, and test execution. The host did not expose an exact deployment
snapshot ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] Required Paperclip CI gates are green (`ci / verify` and `ci /
e2e`; automatic review progress is recorded above)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 07:18:48 -06:00
DottaandPaperclip a80eec1164 fix(recovery): continue Retry after completed file preservation (#18845)
Reuse completed file preservation for task Retry and show bounded admission explanations, including newer saved messages and identifier routes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 16:01:27 -06:00
DottaandPaperclip f7ad1dc7ca fix(workspaces): make attached project repositories writable (#18770)
## Thinking Path

> - Paperclip lets people assign work to AI agents and inspect the
result.
> - A task can use repositories from more than one project.
> - Remote workspace preparation treated every additional project as a
read-only source.
> - This rejected valid Git repositories before the provider started,
even when no source was read-only.
> - The task page then hid the preparation reason behind a generic
failed-attempt notice.
> - This pull request prepares authorized repositories as writable task
copies and shows redacted workspace validation errors.
> - People can work across both projects and see the reason if folder
preparation fails.

## Linked Issues or Issue Description

**What happened?**

A remote task with a selected Git-backed project and another mentioned
Git-backed project failed before provider execution. It reported
`remote_workspace_readonly_sources_unsupported`. Neither project had a
read-only setting. The task notice showed only that the attempt failed.

**Expected behavior**

Each attached repository should be writable inside the task workspace.
This should preserve source access checks and saved work. A workspace
validation failure should explain its cause on the task page.

**Steps to reproduce**

1. Configure remote workspace storage and a native sandbox environment.
2. Select a Git-backed project for a task.
3. Mention another authorized Git-backed project in the task.
4. Start the task. Preparation rejects the additional source before it
starts the provider.

**Paperclip version or commit**

Reproduced against `dc9f387edc`, before this change.

**Deployment mode**

Managed remote workspace storage. Local regression checks use disposable
PostgreSQL and independent filesystem processes.

Related merged work: Refs #18403 and #18504. Those changes established
remote storage and metadata-only references. They still reject
configured cross-project repositories. No duplicate writable-repository
repair was found. Refs #18774 for the test-only retention-fixture repair
reused to unblock CI.

## What Changed

- Add every configured Git repository of an authorized referenced
project to `.paperclip-repositories/` as an editable task copy.
- Retain saved paths, refs, pins and files. Recheck retained project
access and source trust. Keep explicit read-only, local-source import
and remote feature gates.
- Keep committed snapshot repository coverage separate from later
requests. Honor configured default refs, resolve remote branch names,
and publish secondary checkouts only after checkout succeeds. Share
configured-ref resolution between root and attached-repository scripts.
- Include redacted workspace validation messages in the task
interruption projection and existing failure notice. Preserve Retry
permissions and render messages as plain text. Explain retained-folder
access and trust denials while preserving the dedicated low-trust
project-access path.
- Add source, authorization, redaction, UI and save/restore regressions.
Update the implementation contract and workspace maps.
- Reuse the reviewed test-only repair from #18774. Batch the
shared-workspace history fixture, verify its complete privacy-source
map, and drain owned test bodies before cleanup. Production retention
behavior is unchanged.

## Verification

- Red before repair: four source-resolution cases failed with the
reported read-only error. The task-error projection test also failed
because it omitted the cause.
- Green: 49 source and transport tests, 16 task projection and redaction
tests, and 10 UI tests. The assembled preparation check passes. All 48
remote workspace lifecycle tests pass. The full preparation suite passed
all 126 assertions but its cleanup hook exceeded 30 seconds. The
unchanged base reproduces the same cleanup timeout with all 126
assertions passing. The focused preparation check finishes cleanly.
- The real PostgreSQL/Git lifecycle checks cover both first-turn and
later project attachments. They verify the configured release branch,
saved edits, replacement allocations, unchanged source folders,
committed archive contents, and released writers.
- Review regressions were red before repair: the default ref became
null, later attachment failed during restore, non-default remote branch
checkout failed, and invalid secondary refs left partial checkouts. They
now pass.
- The local browser shows the workspace validation explanation and Retry
action. The fixture uses a saved failed run; this proves the error
display, not a successful live sandbox run.
- `pnpm -r typecheck` and `pnpm build` pass. Map, module-boundary and
token checks pass.
- A local two-turn acceptance probe repeated at `17ed67eaa7` used real
Codex CLI 0.165.0, `gpt-6-astra`, and `xhigh` effort. Codex wrote files
in both task repositories. The services saved them, removed the first
allocation and its Git sources, restored another allocation, and Codex
read the saved files and wrote a follow-up in both folders. This uses
the real source resolver and persistence services with a local
command-runner stand-in for sandbox transport. It does not qualify
Daytona, managed subscription forwarding, or the original task. The
first probe waited for stdin; closing the test harness pipe fixed that
setup error before provider work.
- `pnpm test:run` passed the first chat shard, then failed one chat
response concurrency assertion in the second shard. That case passes in
isolation on this head. The same isolated chat case passes on the
unchanged base. Both corresponding unchanged-base chat shards also pass
(351 and 324 tests). The cause of the earlier wider-suite chat failure
is not established. The preparation cleanup timeout and chat failure
remain part of this record.
- CI at `17ed67eaa7`: all other verification jobs passed. Browser shard
6 initially reported a request callback from the preceding test after
teardown; its single unchanged-code retry passed. Server shard 10 timed
out the 2,100-row retention confirmation fixture at 15 seconds on both
attempts. That exact case passes locally, but the repeated CI failure
required a repair. Its setup repeatedly rebuilt a shared-workspace
privacy map. The logs and failures remain part of this record.
- Reused the test-only repair from #18774 without changing its source
file. At `ab774e9418`, all 54 retention tests and all 25 ordinary
production privacy tests pass locally (79 total). Full CI
[41650626903](https://github.com/paperclipai/paperclip/actions/runs/41650626903)
passed on that commit, and its exact-head Greptile review was 5/5. The
fixture preserves its 2,100 rows, query-bound assertions, privacy-source
map, and completed-history test deadlines.
- The additional review identified retained-source access/trust denials
that still bypassed the workspace explanation. Real-resolver regressions
reproduced both plain-403 failures before the correction. Both now yield
a safe workspace validation explanation, while the dedicated low-trust
denial and saved workspace remain unchanged. The optional duplicate
ref-resolution block is also consolidated.
- Final review corrections: 65 source, transport and error-projection
tests passed, followed by all 48 lifecycle cases against an owned
disposable PostgreSQL database (113 total). The initial combined command
skipped the explicitly gated lifecycle suite; the subsequent dedicated
invocation enabled and passed all 48. Full workspace typecheck and build
pass. The first standalone lifecycle launcher lacked the repository
TypeScript loader; rerunning with that loader resolved its import setup
error. At previous head `7be02ea09c4ac76df560e8b517050cdf398f33f0`, full
CI [41654422329, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/41654422329/attempts/2)
passed, Greptile is 5/5 on that exact head, and all review threads are
resolved. The first attempt timed out the unchanged voice-session
report/history case at 15 seconds; 3,027 cases in its shard passed. That
exact case passed locally, then its single unchanged-code shard retry
passed. The wider-suite timeout cause remains unproven. The completed
Paperclip advisory check is neutral; its earlier findings are corrected.
No staging deployment or original-task recovery is claimed.

- Master advanced while checks ran, producing conflicts in two adjacent
test additions. Merged `27d35314be` into this branch as
`d5c8d552939c86056abf8fe10833881b2ade41a4`, preserving both
workspace-error tests and the audited Stop/receipt tests. Kept workspace
maps to minimal corrections under the updated map rule. All 82 source,
transport and interruption cases, all 48 real-database save/restore
lifecycle cases, full typecheck, build, and map checks pass locally on
the combined code. Current-head [CI
41660719214](https://github.com/paperclipai/paperclip/actions/runs/41660719214)
and Greptile are pending; the earlier verified head has not been merged.

## Risks

- Additional remote project repositories are now editable task copies.
Shared source folders and upstream push permissions do not change.
- Snapshot coverage uses existing workspace metadata. Older workspaces
capture their previous inventory at reservation; successful save
publication advances coverage and snapshot identity in one transaction.
No SQL migration is needed.
- A retained copy can contain files from a project that is no longer
mentioned. Preparation rechecks that project's current access and trust
policy before reuse.
- Explicit read-only descriptors still fail. Source-less projects remain
metadata-only. Local folders still require import. No migration or
provider-pack update is needed.
- Error display reads only workspace validation errors. The existing
run-secret registry and username censor run before display truncation. A
redaction failure omits the explanation.
- The retained fixture repair pauses its row trigger only within the
isolated seed transaction, rebuilds the exact source map, and restores
the trigger before commit. A rollback regression and 25 ordinary privacy
cases pass. Its paged-case test budget honors the existing bounded inner
waits; production deadlines are unchanged.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution.
The exact serving model ID and context-window size are not exposed by
this session. The test-only fixture changes are reused from #18774,
whose model record identifies OpenAI GPT-6.1 Sol. The optional live
acceptance probe uses `gpt-6-astra` with `xhigh` effort through Codex
CLI 0.165.0.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 15:33:44 +00:00
DottaandPaperclip 27d35314be feat: connect experimental Cursor Cloud to native Runner (#18069)
## Thinking Path

> - Paperclip helps people manage AI agents at work.
> - Native Runner owns execution and task authority.
> - Operators need an explicit native Cursor Cloud choice.
> - Tools/files need company, task, session and run scope.
> - This connects setup and lifecycle to Runner Core.
> - Legacy agents retain their selected runtime/history.

## Linked Issues or Issue Description

Depends on #18068. Managed-only API placement and API-owned Stop
containment are in #18310. Refs #5664, #11803, #17299, and #17471. No
duplicate native integration was found.

## What Changed

- Add explicit native setup, Test, saved settings and dispatch with
no-prompt account/catalog/callback checks.
- Scope MCP to company, agent, task, session and run through
authenticated HTTPS. Register verified artifacts/work products; preserve
unpriced accounting, completion and budgets.
- Connect Question/Plan continuation, cold handoff and admitted fresh
sessions; recheck waiting admission under the task lock.
- Present audited Stop intent while preserving physical containment
proof; bound typed Stop metadata. Preserve title edits through
preference/navigation readiness.
- Select one live/history representation per run to prevent duplicate
settled cards. Integrate upstream task/read-state/retry/schema contracts
and repair two test fixtures.

## Verification

Core #18068 merged normally as `3910352c923a`. Product head
`5a60f541c982` has passing repository CI/build/browser checks, including
required `ci / verify` and `ci / e2e`, plus a fresh exact-head Greptile
5/5 review with zero unresolved threads. Two non-required external app
checks remain pending: Paperclip Review is queued waiting for its
assigned agent; Superagent Security Scan is reviewing the diff. These
are not claimed completed. The normal merge endpoint enforces the
repository rules; no policy or status override is used. The dependent
Top PR is addressing its fresh deferred-Stop review finding.

Product `5a60f541c982` passed 270 UI and 20 selected
Cloud/native/disk-full tests, UI/server typechecks, tokens and maps.
Top’s clean normal merge passed tokens/maps/diff checks; its matrix is
pending. Earlier receipts retain Core’s 47 Rust, 4 compiled-native
retirement and 19 settlement controls, normal Runner build, and runtime
9af build/recursive typecheck. Reproduced loading/shutdown CI fixture
failures passed 7/7 browser and 121/121 unit tests, including a
broken-unsubscribe negative control. No timeout/oracle was relaxed.

The actual-parent restack adopts 75 mainline paths, including
schema/permission and generic runtime changes, beyond the prior two
fixture repairs and client projection fix. The full difference from
hosted `9af1a5de42a2` is therefore larger than four files; backend
source also differs. The merged config-read checks match Core391, and
the Cursor placement/runtime/relay/selection modules match frozen9af;
selected affected regressions pass. Paid proof keeps its actual 9af
source/image identity. The separately tested live/history card fix is
not claimed deployed on that host.

On 9af, original saved settings survived reopening and fresh no-prompt
Test authenticated catalog/model/binding; no new Save is claimed. Fresh
Linux admission passed. With `claude-sonnet-4-5`, Artifact downloaded
exact 27 bytes and matched 121 canonical identities in complete
S3-served history. First Stop’s strict quiet FAILED on a 10.373-second
gap despite independent cancellation/no late artifact. One targeted
unchanged-sampler retry retained all 24 samples: 35.076 seconds quiet /
maximum gap 4.584 seconds, 67 ordered canonical identities, and separate
CANCELLED/artifact-absent status after the original 180-second deadline.
The pre-reload duplicate settled card is disclosed. Exact-ID cleanup for
the three new provider agents is awaiting authorization, so full grades
and the fresh Artifact’s post-cleanup download remain blocked. A
historical deleted-provider artifact was newly downloaded on 9af;
original F226 Question/Plan/reconnect and cleanup proof keep their
identities.

Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook
failures; Mac 17 SQL failures/skips; earlier
title/startup/custody/Stop/deletion failures with unknown causes/bounded
recoveries; `dea023` Stop’s warning/50 duplicate identities; pre-repair
ACK and original Core CI failures. WSP’s 251-assertion, 7/8-suite
attempt failed physical PostgreSQL teardown; the unchanged wrapper rerun
passed 128. Retry alone establishes no cause or full local-suite pass.
All paid attempts remain recorded; vendor dollar cost is UNKNOWN.

## Risks

Experimental: Cursor independently acknowledges neither the effective
model nor instruction digest before the first prompt. Explicit values
are submitted and checked through behavior; unpriced provider dollar
cost stays UNKNOWN.

Callbacks need authenticated reachable HTTPS. Scope, revocation,
budgets, completion and cleanup remain authoritative. Operator Stop
intent is separate from physical containment. Saved legacy agents retain
their selected runtime. Prior shared title-save/deletion failures remain
disclosed.

## Model Used

OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools,
code execution, browser verification, and delegated code review. The
serving model’s exact API ID and context window are not exposed. No
value is inferred from the model catalog.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the limits of available model
metadata
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public issues or described the issue in this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch describes the change and contains no internal ticket ID
- [x] I have run the scoped local tests reported above; pending gates
are stated separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All required CI gates pass on the current head
- [x] Fresh Greptile is 5/5 with no unresolved findings on the current
head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 09:01:39 -06:00
DottaandPaperclip b22ef22d56 fix(runner): show provider failure details on tasks and runs (#18622)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner starts providers and records their run outcomes.
> - Provider failures can contain useful details below a generic
protocol error.
> - Those details were lost at the sidecar, command receipt, and run
projection boundaries.
> - A blocked task therefore showed a generic failure even when the
provider gave a clear reason.
> - This pull request carries a bounded, sanitized provider failure
through those boundaries.
> - Operators can read the actual cause on the task and run pages.
Existing recovery policy still controls retries.

## Linked Issues or Issue Description

Refs #17543 for the related closed JSON-RPC classification work. Refs
#17272 for provider recovery in the legacy OpenCode adapter. This change
adds display evidence for native Runner providers and keeps recovery
policy separate.

**What happened?**

A native Claude session returned an HTTP 429 message below an ACP
`Internal error`. The task and run showed only a generic startup
failure. Operators had to inspect the provider response to learn that
the account was rate limited.

**Expected behavior**

The task and run show the sanitized provider message, provider name,
startup or turn phase, and available HTTP status. Unknown categories
keep the message. Display evidence does not grant a retry or replace a
recovery decision.

**Steps to reproduce**

1. Start a native Claude run whose account is rate limited during model
selection, or return a structured provider failure from a Codex or
OpenCode turn.
2. Let the run fail and open its task page.
3. Before this change, the page shows a generic failure. With this
change, the page shows the provider explanation. The run page shows the
same explanation after reload.

**Paperclip version or commit**

Source checkout based on `f8a888a072`.

## What Changed

- Add one JSON Schema for `ProviderFailure` and generate matching Runner
and shared TypeScript readers. Keep the Runner package independent of
the app workspace at runtime.
- Normalize known ACPX, Codex, and OpenCode error fields. Preserve
unknown messages. Redact known credential values and credential patterns
before applying the 4,000-character limit. Do not copy response bodies,
headers, stacks, or prompts.
- Carry startup failures through the sidecar, Rust Runner, durable
command receipts, and private transport error details. Keep provider
text out of generic transport error strings and Sentry failure messages.
- Keep turn failures in authenticated run events. Recover the same
evidence from persisted events during server finalization. Store the
cause in the company-scoped run result.
- Show one shared notice on failed tasks and run details. Keep existing
retry permissions, scheduled retry state, and resolved-task behavior.
- Reapply registered run-secret and current-user redaction to task
display data. Batch secret lookups for only the visible provider causes.
- Document the contract and update the capability maps. Add regressions
for nested Claude 429 errors, OpenAI quota and rate limits, OpenCode
errors, redaction, durable receipt replay, restart finalization, and UI
rendering.

## Verification

- `pnpm -r typecheck` passed. `pnpm build` passed. All GitHub Actions
build, test, package, and validation jobs passed on `d49c79ab4a`. The
security checks passed. Greptile gave this head 5/5 with no open review
threads. Paperclip Review remains pending. Its review worker exhausted
six workspace recovery attempts with export_interrupted. A fresh review
request was accepted and is waiting for that saved workspace. The
ordinary Retry action rejected reuse of the stopped request. This is an
external review-worker blocker; no check was overridden. The duplicate
local `pnpm test:run` was stopped after the CI test shards passed.
- Focused server settlement, provider evidence, and restart tests: 93
passed. Runner admission and normalization tests: 103 passed. Focused
Runner wire and terminal tests: 29 passed. The real Rust/sidecar command
receipt replay test passed.
- The new projection test failed before implementation because the task
had no provider cause. The startup producer regression also failed
before extraction moved ahead of the generic error-code replacement.
Both pass with this change.
- Live check: a rebuilt local sidecar received an actual Claude HTTP 429
during startup. The new cause kept the full provider message. A
synthetic task in an isolated local app then reached the same provider
error through the native Runner path. The dedicated run result, task
API, task banner, and run detail all showed the same message after
reload. The later reporting fix keeps run.error generic; a regression
proves the display cause still survives restart and stays out of Sentry
capture.
- The first isolated app attempt failed earlier in
`admission_credential`. An ordinary manual Retry reached the provider
and produced the 429 above. This PR does not claim to fix that separate
admission failure. An initial parallel local database test setup also
failed; the serial restart suite passed.
- Security review found that copying display text into run.error could
forward it to Sentry. A regression failed with that behavior. The fix
keeps the cause in its display field. All 38 settlement/restart tests
and the server typecheck pass.
- All 29 protocol contract tests pass, including bounds and extra-field
rejection. An earlier full Rust core run passed. The later full run hit
one 5-second receipt-limit timing assertion; its isolated rerun passed.
The full Linux and Windows Rust CI jobs passed on this head.
- Review regressions pass for explicit Codex driver identity, native
Rust startup and resume receipt propagation, oversized OpenCode/ACPX
errors, task read-time registered-secret redaction, and the standalone
source closure. The focused Runner run passed 103 tests;
projection/restart tests passed 16.
- Read-time redaction edge cases passed: a missing decryption key omits
only the unsafe optional explanation; expanded redacted messages remain
visible within the Unicode character limit. All 14 task projection tests
and server typecheck pass.
- The package Build check found an old Pi assertion against
Error.message. The probe now checks the exact closed classification in
the authenticated receipt and requires the new provider cause. Both
polling transport cases pass. A local credential-free Pi probe stopped
at admission_installation because this checkout lacks the Pi
distribution; the installed Linux package probe passed in CI on this
head.
- UI coverage includes startup and turn text, unknown categories,
literal rendering of provider text, read-only retry behavior, and
resolved-task suppression. Token gates and map checks passed.
- The live check covers Claude startup on macOS. Codex, OpenCode,
terminal failures, and restart paths have deterministic regression
coverage. The change is not deployed.

## Risks

- Provider messages are untrusted text. The boundary selects specific
fields, redacts credential patterns and known environment secret values,
bounds the message, and renders plain text. New provider-specific secret
formats may need added redaction patterns.
- Optional fields preserve compatibility with older producers and stored
runs. Old runs without the field keep their current fallback text. No
database migration is required.
- The cause stays in company-scoped run data. This change does not send
provider text to first-party Telemetry or add it to OpenTelemetry span
attributes.
- Normalized categories and retryability are display evidence. Existing
recovery classifiers, retry limits, ownership checks, and approval gates
keep their authority.

## Model Used

- OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and browser verification. The host does not expose the exact
serving variant or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 06:13:26 -06:00
DottaandPaperclip 67948173f4 fix(sentry): report saved-message task attempt failures (#18541)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tasks retain user messages when an execution attempt cannot finish.
> - The task interruption projection drives the saved-message failure
notice.
> - That state can occur without a new terminal run transition, which
the existing Sentry reporter requires.
> - This pull request reports that failed task state through the
existing Sentry capture path.
> - Operators can find the failure through the same tag and saved run
links.

## Linked Issues or Issue Description

Refs #13446 and #16300. The related PR search found no open
implementation of this change.

**What happened?**

A task can show “Couldn’t finish this attempt. Your message is saved.”
without a matching run-failure Sentry event. A stale saved-message wait
can produce this state after the previous run succeeded.

**Expected behavior**

When run-failure reporting is enabled, this failed task state also
reports with `event_kind=run_failure`.

**Steps to reproduce**

1. Enable the backend Sentry DSN and
`PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` or `all`.
2. Leave a saved-message wait on an unfinished task after its last run
has ended.
3. Read the task after the wait grace period expires.
4. The task shows the failure notice. Before this change, that read does
not report it to Sentry.

**Paperclip version or commit**

Base commit: `e122478368`.

## What Changed

- Report the system-owned failed presentation from the existing batch
task reader. The pure projection remains unchanged.
- Reuse `captureRunFailure()` with `event_kind=run_failure`,
`failure_source=task_interruption`, and
`error_code=task_attempt_failed`.
- Preserve the actual saved run status and company, agent, task and run
references. Load the run agent's adapter through a company-scoped join
in the existing run query, including after reassignment. Add no database
query or background timer.
- Send fixed error text. Do not read chat bodies, prompts or raw run
diagnostics for this event.
- Suppress repeated reports for the same company/task/run/intent with a
bounded process-local cache. Failed or disabled captures do not consume
the report.
- Document the enabled modes, event fields, duplicate behavior and cache
limits.

## Verification

- Regression before implementation: the two new Sentry tests failed
because the capture entry point was absent; the existing 38 Sentry tests
passed.
- Focused verification: 332 tests passed across
`run-failure-sentry.test.ts`, `task-interruption-query.test.ts`,
`task-interruption-presentation.test.ts`, and
`run-failure-report.test.ts`. The reassignment regression first
reproduced the missing adapter, then passed after the query correction.
- Command: `PAPERCLIP_TEST_POSTGRES_ADMIN_URL=<local-test-admin-url>
pnpm exec vitest run server/src/__tests__/run-failure-sentry.test.ts
server/src/__tests__/task-interruption-query.test.ts
server/src/services/task-interruption-presentation.test.ts
server/src/services/__tests__/run-failure-report.test.ts`.
- PostgreSQL tests used isolated databases on a local PostgreSQL 17
server, with automatic cleanup. Initial embedded starts failed because
installation scripts had not restored library links; after repair, the
host semaphore limit prevented new clusters. The existing
external-database helper supplied the working test path.
- `pnpm -r typecheck` passed.
- `pnpm build` passed.
- `git diff --check` passed.
- `pnpm test:run` was attempted. It stopped in the unchanged chat
integration shard collection step with “No test found” for line filters.
The complete local suite has not passed.
- Sentry capture assertions use the SDK test double. Live Sentry
ingestion and a deployed browser journey were not tested.

## Risks

- An enabled instance can send more Sentry events when task reads
discover retained failures. It can emit a task-state event alongside the
separate terminal run event. The source tag distinguishes them.
- Deduplication is process-local and retains 1,000 reports. Restarts,
multiple replicas or cache eviction can report a retained failure again.
- No schema, permission, execution or UI changes. The existing opt-in
reporting gate stays in place.

## Model Used

OpenAI GPT-6 through Codex, with tool use, code editing and test
execution. The exact runtime model ID and context window are not exposed
in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 08:17:22 -06:00
DottaandPaperclip f69dfc0e23 perf: use indexed latest-run probes for task interruption lists (#17969)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task lists show the latest run's interruption state for each visible
task.
> - The old lookup applies a coalesced text expression to company run
history.
> - That lookup cannot use the separate binding indexes for a small task
page.
> - This pull request selects the newest run through those existing
indexes.
> - The benefit is less database work with the same interruption result.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The latest-run lookup used by task-list interruption enrichment.

**Current behavior**

The query scans company history for a coalesced native, first-class, or
legacy task ID. A local two-task fixture inspects 1,506 runs to select
six candidates. A recent hosted sample recorded about 110 ms per
latest-run lookup. This is one part of the list handler, not its whole
latency.

**Proposed behavior**

Probe each existing binding index for the newest candidate. Pick the
newest candidate across the three binding branches. Read the same
evidence fields in the same SQL statement. Preserve native precedence,
first-class precedence, legacy text matching, company scope, and
ordering ties.

**Reason and benefit**

The representative local 100-task page improves from 17.17 ms to 3.00 ms
(5.72×). A modern 100-task page with 40 runs per task improves from
26.78 ms to 3.82 ms (7.02×). All positional SQL fields, row order, and
full service results match.

**Breaking changes**

None. Stored history, API fields, visibility filtering, execution
authority, and external-conversation state are unchanged.

Related behavior: #16965, #17509, #17552. The duplicate search found no
open PR for this query rewrite. ROADMAP.md has no duplicate planned
feature.

## What Changed

- Use three disjoint indexed candidate branches with a lateral
newest-run selection.
- Guard UUID parameters to preserve the old text matching for
noncanonical input.
- Keep final evidence projection in the same database snapshot.
- Add real PostgreSQL work and contract regressions, a pinned-source
benchmark, and a verification note.
- Require the plan regression to find real run-table nodes. Record the
deep-native-history follow-up without adding indexes or DDL.

## Verification

- Baseline red: the real two-task lookup inspects 1,506 run rows and
fails the work bound. The unchanged semantic oracle passes.
- Treatment green: both PostgreSQL query regressions pass. They cover
conflicting bindings, malformed JSON, empty values, noncanonical IDs,
duplicates, foreign companies, missing runs, and timestamp ties across
branches.
- Seven paired PostgreSQL 17.11 cases pass complete positional SQL and
full service equality. Each arm has two warmups and five alternating
measured samples. Fixture payload sizes are random input bytes before
Base64 encoding: 24 KiB becomes 32 KiB of stored text per noise field.
- Representative 100 × 12 history: 17.17 ms (16.79–17.48) → 3.00 ms
(2.64–4.28). Shared buffer hits: 30,033 → 3,400.
- Modern 100 × 40 history: 26.78 ms (25.79–27.40) → 3.82 ms (3.47–4.78).
- The benchmark command, all cases, plans, fixture limits, and
feedback-loop timing are described in [the verification
note](doc/plans/2026-10-27-task-interruption-query-performance.md).
- Reproduce with `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-task-interruption.ts
--baseline-ref=09ad9adce2cf4742944353a3707a2f908e9bb35e`.
- `pnpm -r typecheck` and `pnpm build` passed at
`20d46e3a16d10cf3c5399fc1211b17db6a4afdc6`. The current head has the
identical production runtime; it adds one test assertion and
documentation.
- The focused query, real project-access repair/retry, and presentation
suites pass: 19 tests across three files. The earlier plugin-sdk loading
failure is resolved by normal build setup.
- `pnpm test:run` was attempted, then stopped after repeated native
embedded PostgreSQL bootstrap failures. The full local suite is
incomplete. All 16 affected real-DB executor cases pass when rerun with
disposable PostgreSQL 17 in Docker.
- Linux CI passes the full server suites on this head. One unchanged
Runner close fixture failed on the first CI attempt; its source tree is
identical to the passing upstream #17439 baseline. One targeted job
retry passed on the original head. Fresh checks run again for the
test/documentation review fix. A separate macOS diagnostic fails the
unchanged close variants too; that close issue is not diagnosed or
counted as a local pass here.
- The original head passed 55 checks with one neutral result and two
expected Storybook skips. Greptile gave 5/5. The current head adds one
nonempty-plan assertion and documents the deep-history follow-up. Its
focused PostgreSQL suite passes 2/2. Final head
`c1db106d7ab56b530af82d2bb0556feb0eda290b` is mergeable with all CI
checks complete and passing, Greptile 5/5, and no unresolved review
findings; the documented per-task history limitation remains an explicit
follow-up.

## Risks

- More index probes add about 0.43 ms to the one-run tiny control. The
maximum 1,000-task page with one modern run per task is 9.99 → 14.11 ms,
a 4.12 ms regression. Dense history pages save substantially more in the
local fixture.
- Each branch can still walk one task's own history. The first-class and
legacy probes can discard native-bound rows after index lookup. The plan
note specifies a 10,000-run follow-up, migration-free alternatives, and
a measured index-build budget if indexes prove necessary. This PR does
not make work independent of history depth.
- Local SQL results do not prove deployed HTTP gains. The affected query
averaged about 110 ms in the hosted sample; its critical-path
contribution can be smaller because other enrichment reads run
concurrently.
- The main issue-list SQL and external-conversation gates remain
separate work. This PR has no migration or startup job.

## Model Used

OpenAI Codex, GPT-6 family, with code execution and repository tool use.
The runtime does not expose the exact model ID or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 07:18:17 -05:00