Commit Graph
29 Commits
Author SHA1 Message Date
DottaandClaude Fable 5.2 39ef43a73e fix(retry): move a recorded continuation forward before its delivery (#21379)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run cannot resume, the server records a continuation for the
task and delivers it later as a new run
> - #21378 made a person's Retry deliver a recorded, undelivered
continuation as the one run, with its workspace isolation
> - #21373 added the `move_forward` Retry action, which does nothing
when the observed run is not queued or running
> - Together, a `move_forward` request before delivery returned "already
moved on" and started no work
> - That state can last a long time when the agent has a deep queue, and
it is when a person wants to move the task forward
> - This pull request accepts the request when the task has a recorded,
undelivered continuation
> - The benefit is one action that moves waiting work forward in each
state, with one run and the recorded isolation

## Linked Issues or Issue Description

**What happened?**

A task has a continuation that is recorded but not delivered. The latest
run is the failed source run. A Retry request with `retryAction:
"move_forward"` returns `accepted: false, alreadyMovedOn: true`. No run
starts.

**Expected behavior**

The request is accepted. Wake admission delivers the recorded
continuation as the one run for the task. The run keeps the recorded
isolation. A later delivery pass creates no second run.

**Steps to reproduce**

1. Record a fresh continuation for a task and do not run the delivery
pass.
2. Call `requestTaskRetry` with `observedRunId` set to the source run
and `retryAction: "move_forward"`.
3. The receipt says `alreadyMovedOn: true`.

Refs #21373, #21378, #21348.

## What Changed

- `admitTaskRetry` in `server/src/services/task-retry.ts` does not
return `alreadyMovedOn` when `undeliveredContinuation` finds a recorded,
undelivered continuation for the task. The check runs under the task row
lock.
- A new test in `fresh-continuation-retry-delivery.test.ts` covers the
three record kinds: `fresh_isolated_workspace`, `writer_ended` and
native.
- The presentation code is unchanged. The interface does not offer the
action in this state yet.

## Verification

- The new test failed before the change for the three kinds (`expected {
requestId: null, … } to match object { accepted: true }`).
- `npx vitest run
src/services/fresh-continuation-retry-delivery.test.ts`: 15 of 15
passed.
- `npx vitest run src/services/task-retry.test.ts`: 117 of 117 passed.
- `npx vitest run src/__tests__/recovery-wake-herd.test.ts -t move`: 2
of 2 passed. The case where the run ended and no continuation is pending
still returns `alreadyMovedOn`.
- `npx tsc --noEmit -p .` in `server/`: no errors.

## Risks

Low risk. The change adds one read under a lock that the function
already holds. The read runs only for a `move_forward` request when the
observed run is not queued or running. A request without a pending
continuation behaves as before.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 07:31:16 -06:00
DottaandClaude Opus 5.5 d9b42b50fd feat(presentation): let a person move a waiting continuation forward without starting a second run (#21373)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, Paperclip records a fresh continuation
and continues the task by itself. Since #21353 the task reads as
continuing (`working`, no Retry) for as long as that continuation waits
> - That is truthful, but a person has no control while it waits. On
hosted staging one agent had 21 running and 209 queued runs, and
continuations waited 30 to 110 minutes. Pacing (#21234, #21354) admits a
waiting continuation into the agent's queue within minutes, so most of
that wait is in the queue, behind other work
> - #21353 removed Retry from this state because a Retry started its own
run, which raced the continuation and could give the task two runs
> - A Retry that observes the queued continuation run cannot start a
run: admission sees a queued run and joins the Retry to it (`coalesced`,
`runId` = that run). Under #21234's ranking, a queued run that a person
joined starts ahead of system recovery and ordinary work
> - This pull request keeps the continuing state and, once the queued
continuation has waited 15 minutes, also offers Retry labelled "Move it
forward now"
> - The benefit is that a person can move a long-waiting task to the
front of its agent's queue, and the click can never create a second run

## Linked Issues or Issue Description

Refs #21353, #21348, #21234, #21354, #21264, #21320, #21200

**What happened?**

On hosted staging, tasks with a recorded continuation waited 30 to 110
minutes behind a deep agent queue. The task showed "continuing" with no
action. The person could only wait.

**Expected behavior**

The task still never shows "failed". After a long wait the person can
move the waiting continuation forward. The click must not start a second
run.

## Which states a Retry joins (established before building)

I read #21348's fold (`foldWaitingRetriesIntoContinuation`) and its
tests, and drove the real click path (`requestTaskRetry` followed by
`resumeTaskRetry`, as the route does) in a probe against each record
state:

| Continuation state | What a Retry does today | Offer Retry? |
|---|---|---|
| Recorded, waiting for delivery (`continuationDelivery: "pending"`, no
run yet) | The Retry is admitted at once as its own `task_retry` run,
for every isolation (`fresh_isolated_workspace`, `writer_ended`,
native). The delivery pass then marks the record delivered (superseded).
#21348's fold never runs: it folds only a Retry that is still
`deferred_issue_execution` when the continuation run is created, and
only for `fresh_isolated_workspace`. Nothing keeps the Retry deferred,
so the click races the delivery pass. | **No** (unchanged) |
| Delivered, continuation run `queued` | `requestTaskRetry` sees the
latest run is busy and records the Retry as `coalesced` into that run in
the same transaction. No run is created; `resumeTaskRetry` acts only on
deferred Retries. The run is now ranked as a person's (`joinedByUser` in
`queue.ts`). | **Yes, after 15 minutes** |
| Continuation run `running` | Same join, but the run already started,
so there is nothing to move forward. | No |
| Continuation ended | A Retry gets its own run (the existing failed-run
path). | Existing behaviour |

So the Retry offered here is joined by the existing busy-run admission
in `requestTaskRetry`, not by #21348's fold. Its source run is the
queued continuation itself (the run the presentation names as
`observedRunId`), not the run the continuation was created for.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: for a task
whose latest run is `queued`, created 15 minutes ago or more, and named
as `continuationRunId` by a delivered fresh-continuation record, the
projection returns `state: "working"`, `canRetry: true`, `retryAction:
"move_forward"`, responsibility `system`, `observedRunId` = that queued
run. It does not offer this when a person's wake already joined the run,
when the run's agent is not the assignee, or when a person-owned wait
applies (approval, reconnect, budget, pause and the others are checked
first). Two small reads run only for queued runs older than 15 minutes.
The existing restore-hold check that withdraws `canRetry` also withdraws
`retryAction`.
- `packages/shared/src/types/execution-projection.ts`: new optional
`retryAction?: "move_forward"` on `TaskInterruptionPresentation`. It
only changes the label; `canRetry` still decides whether Retry is
offered.
- `ui/src/components/ExecutionBlockerNotice.tsx`: a `working`
presentation with `retryAction: "move_forward"` shows "Waiting for the
agent to finish other work. Paperclip continues this task by itself."
and a "Move it forward now" button that calls the same task Retry.
Token-only classes; `pnpm check:token-gates` is clean.
- One join-only action, `move_forward`, carried from the offer to
admission (contracts kept in sync):
- `packages/shared/src/validators/issue.ts`: `taskRetrySchema` accepts
`retryAction: "move_forward"`. The route already passes the body to
`requestTaskRetry`.
- `server/src/services/task-retry.ts` (`admitTaskRetry`): right after
the stale check, still under the task row lock and before any other
Retry path, a `move_forward` Retry whose observed run is no longer
`queued` or `running` returns `{ accepted: false, alreadyMovedOn: true,
requestId: null }` and writes nothing. This block was cleared with the
owner of Retry admission. The owner will later extend it so that
"continuation recorded, not yet delivered" also counts as actionable,
after their pending-delivery change lands (about two lines on their
side). Until then this presentation never offers the button in that
state, so the early return cannot misfire from our UI.
- `ui/src/api/issues.ts` and `ui/src/hooks/useTaskRetry.ts`: the click
sends `retryAction: "move_forward"` when the offer has it.
`alreadyMovedOn` shows "The task already moved on, so nothing new was
started. It now shows its current state."
- What a click does at each timing: continuation `queued` → joined to
that run (moves it forward); `running` → joined (nothing more to do);
ended → "already moved on", nothing started.
- A joined click cannot become a run later. The busy path writes it as
`coalesced` with `runId` set to the continuation and `finishedAt` set
(`task-retry.ts:456-461`). Promotion after a run ends
(`heartbeat/run-control.ts:944` `releaseIssueExecutionAndPromote`;
`modules/wake-queue/adapters/postgres.ts:89` `DEFERRED_WAKE_STATUS` and
the selects near `:1358`), `resumeTaskRetry` (`run-control.ts:833-839`)
and `resumeExecutionWaitComments` (`:876`) pick up only
`deferred_issue_execution` wakes.
- `ui/src/pages/Inbox.tsx`: the failed-run row does not show its Retry
for this state. That row sends the failed run as the observed run, which
is no longer the latest run, so its Retry would be refused as stale.
- `doc/execution-semantics.md`: one paragraph on the offer, when it
ends, and why there is none before delivery.

## Verification

- Red (before the product change, new tests only), in `server/`: `pnpm
exec vitest run src/__tests__/task-interruption-automatic-path.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 2 failed, 58 passed. Both
failures are the missing offer (`canRetry: false`, no `retryAction`) for
a continuation queued 15+ minutes.
- Green after merging master 8e41247b65, in `server/`: `pnpm exec vitest
run src/__tests__/recovery-wake-herd.test.ts
src/__tests__/task-interruption-automatic-path.test.ts
src/services/task-retry.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/task-interruption-query.test.ts`: 5 files, 279 tests pass.
UI: `useTaskRetry.test.tsx` and `ExecutionBlockerNotice.test.tsx`: 35
tests pass.
- New tests:
- (b, integration, `recovery-wake-herd.test.ts`) A continuation
delivered by the real delivery pass waits 16 minutes behind older
ordinary work. The task offers "move forward". The click goes through
the real `requestTaskRetry` and `resumeTaskRetry`. Result: the task has
exactly one run after the source run, the continuation; the Retry wake
is `coalesced` with `runId` = the continuation; the offer disappears;
the next delivery pass adds nothing; the real dispatcher
(`resumeQueuedRuns`) starts the continuation ahead of the older work;
after it ends, still only that one run.
- (Greptile P1, red then green) "starts nothing when the continuation
ended between showing Move it forward now and the click": the
continuation ends after the offer is read. Before the admission block
the click was accepted (`accepted: true`, a new deferred Retry that
becomes a turn). Now it returns `alreadyMovedOn`, the task has one run,
and there is no `task_retry` wake.
- "never turns a joined move-forward click into a new run after the
continuation it joined ends": the click joins the queued continuation.
The run is then cancelled through the real `cancelRun` release. Two
rounds of `resumeExecutionWaitComments`, `resumeTaskRetry`, the delivery
pass and `resumeQueuedRuns` follow. The joined wake stays `coalesced` on
the ended run. No run comes from the click, and the task's later runs
equal those of an identical task with no click (the system's own
`issue_assignment_recovery`).
- UI hook (`useTaskRetry.test.tsx`): the click sends `retryAction:
"move_forward"`, and the `alreadyMovedOn` outcome shows the plain
message.
- (a) Under 15 minutes: no Retry. Control in the herd file: without a
click the older work starts first.
- (b, presentation) Queued 15, 45 and 110 minutes: `working`, `canRetry:
true`, `retryAction: "move_forward"`.
- (c) No Retry while the continuation waits for delivery (16, 45, 110
minutes), when it is running, when a person already joined it, for a
queued run that is not a recorded continuation, and a paused agent still
asks the person to resume.
- UI (`ExecutionBlockerNotice.test.tsx`): plain wording with one "Move
it forward now" button that calls Retry; no button for a read-only user;
nothing shown for a plain `working` state.
- (d) The #21200, #21264, #21320 and #21353 guard tests pass unchanged,
including "follows the new run once a recorded continuation is
delivered" (a just-queued continuation shows no Retry).
- `node scripts/typecheck-layers.mjs` in `server/`, `tsc --noEmit` in
`packages/shared`, `pnpm run typecheck` in `ui/`: clean.
- Not related to this change:
`src/services/fresh-continuation-fallback.test.ts` has one failure
("starts the continuation while the unreachable old lease is active", a
deadlock in the `afterEach` truncate) that also fails on master
18fb6ee571 without this change.

## Risks

- Moving a waiting continuation forward lets it start before other
queued work of the same agent. That is the purpose, and it is the
ranking #21234 already gives any run a person joined.
- No Retry is offered while a continuation is still waiting for
delivery. The delivery pacing admits a waiting continuation into the
queue within minutes, and a fresh interruption at once (#21354), so the
offer appears after the continuation reaches the queue. For the recovery
owner (not changed here): offering Retry before delivery would need the
Retry admission to hold or deliver the pending continuation itself under
the task lock, instead of starting its own run beside it.
- Admission changes only for a request that carries `retryAction:
"move_forward"`. Every other Retry is unchanged. No schema, migration or
dispatch change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:03:23 -06:00
DottaandClaude Opus 5.5 57b787d3d2 fix(retry): continue from recorded work when saved files cannot be recovered (#21363)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run finishes but copying its files out of the sandbox fails,
Paperclip holds the task and tries to recover the saved files. When
recovery gives up, the task offers Retry.
> - Some saved files can never be recovered: the sandbox that held them
was destroyed, or Paperclip cannot prove which runs used a shared
sandbox. Each Retry ran the same check again, failed again, and parked
the Retry in a wait that never ends.
> - The task told the person to press Retry again or to wait for a
Paperclip update. That breaks the rule that recovery must work and that
a Retry that is shown must work.
> - This pull request makes Retry continue the task in a fresh session
and a new workspace from the work that is recorded, when the saved files
are provably unrecoverable or cannot be proven safe. The task says
clearly that the unsaved files were not recovered.
> - The benefit is that an offered Retry starts a run, and nobody waits
for a fix that another click cannot bring.

## Linked Issues or Issue Description

Refs #21315, #21295, #21267

**What happened?**

On hosted staging, two blocked tasks offered Retry (`canRetry: true`).
Each Retry was accepted, and within about 3 seconds it was parked in a
wait. No run started. Every later Retry did the same.

- Task A: the hold cause was
`native_workspace_sync_out_retry_exhausted`. Recovery ended with
`source_unavailable` (`use_alternative_strategy`). The sandbox that held
the files had a signed absence proof and a `destroyed` termination
receipt. Retry was parked with `saved_work_recovery_failed`: "Saving the
agent's files failed. Retry to try again".
- Task B: the same hold cause. Recovery ended with
`allocation_ownership_ambiguous` (`service_required`), because the
sandbox was a reusable sandbox that other runs also used. Each Retry
opened a new recovery generation, ran the same check, and was parked
with `saved_work_recovery_refused`: "Retry runs the same check again and
succeeds only after Paperclip is updated."
- In both tasks, the agent's turn had finished and its result was
accepted. Its work was recorded outside the sandbox: a plan document and
screenshot attachments for task A, and a pushed branch with an open pull
request and an implementation document for task B.

Root cause: admission had a branch for an accepted result whose files
were not yet saved. When the recovery budget was spent, the branch
always returned a `saved_work_recovery_*` wait. It did not check whether
the files could ever be recovered, so the fresh-workspace path that
admission already has was never reached. The automatic retained-source
fallback did not help either, because it handles only the remote
workspace v1 reference, and these runs use the native workspace sync v3
reference. Also, Retry opened a new recovery generation for an ownership
refusal, which only repeated the same check.

**Expected behavior**

When the saved files are provably unrecoverable or cannot be proven
safe, Retry continues the task in a fresh session and a new workspace
from the recorded work. The old files are never applied and the source
sandbox is not changed. No provider turn is replayed. The task says that
the unsaved files could not be recovered. The Retry offer and admission
agree.

**Steps to reproduce**

1. Let a native run finish with an accepted result, and let its
workspace copy-back fail (`native_workspace_sync_out_retry_exhausted`).
2. Let saved-work recovery end with `source_unavailable` (sandbox
destroyed) or with `allocation_ownership_ambiguous` (shared reusable
sandbox).
3. Press Retry.
4. Observe that the Retry receipt waits with
`saved_work_recovery_failed` or `saved_work_recovery_refused`, and no
run starts. Press Retry again and observe the same result.

**Paperclip version or commit**

Staging on `origin/master` after #21315 (`bbafa78fe2`).

**Deployment mode**

Hosted cloud (staging), remote Daytona sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime runs).

## What Changed

- `savedFilesUnrecoverable` and `unrecoverableSavedFilesFailure`
(`system-recovery-state.ts`) are the one rule for saved files that no
recovery can apply. The rule is true for a spent recovery whose failure
is a permanent source refusal: `source_unavailable`,
`allocation_ownership_ambiguous` or `historical_source_missing`. It is
false for `source_binding_invalid`, because a new recovery generation
can still verify that source. It is false for `archive_unsafe` and
`snapshot_verification_failed`, because a fresh attempt still imports
earlier saved snapshots and would fail before the agent starts. It is
also false after a fresh attempt is recorded.
- Admission (`explicit-native-continuation.ts`) no longer parks a Retry
of an accepted result in `saved_work_recovery_*` when every live
recovery of the run is unrecoverable. The Retry takes the existing
fresh-workspace path (`fresh_isolated_workspace`). That path keeps all
stop proofs, leaves the source sandbox and workspace untouched, and
starts a fresh session.
- When such a Retry starts, the task gets one system comment. The
comment says that Paperclip could not recover the unsaved files, why,
and that the task continues from its recorded work (comments, documents,
attachments and pushed commits). The settled hold records the same text.
- Retry (`task-retry.ts`) binds to an unrecoverable recovery generation.
It does not open a new generation that repeats the same check.
- The blocker text for unrecoverable files says that Retry continues the
task in a new workspace from its recorded work. It no longer says that
Retry succeeds only after an update.
- `canAdoptNativeSourceRetention` and the permanent source-refusal lists
moved into `system-recovery-state.ts`, so that admission, Retry and
messages share them. The old modules re-export them.

## Verification

- Red/green: `cd server && npx vitest run
src/services/unrecoverable-saved-work-retry.test.ts`. The tests fail on
`origin/master` sources (or on the earlier head for the snapshot case)
and pass with this change. They are built from the two staging shapes
(destroyed sandbox with `source_unavailable`, and a shared reusable
sandbox with `allocation_ownership_ambiguous`):
- offers a Retry that continues the task (blocker and task view agree,
and the text names the fresh workspace)
- starts a fresh run from the recorded work: the Retry receipt has no
wait and a run, the strategy is `fresh_isolated_workspace`, the hold is
settled with the fresh attempt, one system comment explains the lost
files with no raw IDs, the source lease is unchanged, and no new
recovery generation opens
- keeps a Retry out of a fresh attempt that would import a succeeded
snapshot whose archive is unsafe
- `workspace-recovery-warm-handoff.test.ts` › "wakes physical
preservation from HTTP Retry before passive history without replaying
the provider" failed on an earlier head of this PR, which also treated
`source_binding_invalid` as unrecoverable. It passes unchanged now.
- Updated tests that pinned the old repeat-the-check behavior:
`workspace-recovery-preparation-gates`, `task-retry`,
`system-recovery-state`, `workspace-recovery-retry-copyback`.
- Related suites pass: `system-recovery-state`,
`workspace-recovery-preparation-gates`,
`workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`,
`task-retry`, `saved-work-retry-eligibility`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
`workspace-recovery-admission`, `task-interruption-query`,
`native-workspace-sync-retention`.
- `cd server && node scripts/typecheck-layers.mjs` passes (source and
test layers).

## Risks

- Medium-low. A Retry of an unrecoverable hold now starts a new agent
turn in a new workspace instead of waiting. The turn starts only through
the existing fresh-workspace admission, which keeps every stop and
cleanup proof.
- Unsaved files of the old run are not applied. That was already true;
the change is that the task continues and says so.
- `source_binding_invalid`, `snapshot_verification_failed` and
`archive_unsafe` keep today's path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:23:23 -06:00
DottaandClaude Opus 5.5 c6c7a61e2c fix(retry): release a Retry held on a resolved decision (#21267)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, the task shows a Retry button. Admission
can hold an accepted Retry while a person must answer a pending approval
or question (`decision_pending`).
> - The hold is saved on the deferred Retry receipt. Nothing cleared
that saved reason when the person answered, so the task kept saying
"Retry is waiting for a pending approval or question" when nothing was
left to answer.
> - The deferred-receipt worker drops a refused receipt from its
schedule. No producer hinted it when the decision resolved, so the held
Retry did not start until some unrelated event rescanned the company.
> - A Retry was also held by a confirmation that was already open when
the failed turn was admitted. That turn had already run past the
confirmation, so the Retry added a gate that the original turn did not
have.
> - A person's comment on the task started a continuation, but the held
Retry receipt stayed deferred next to it.
> - This pull request makes a resolved decision release the held Retry
at once, keeps the projection from showing a resolved decision as the
wait, stops holding a Retry on a decision that predates the failed turn,
and settles a held Retry when a comment or answer continues the same
failed run.
> - The benefit is that Retry always works: a person never waits on a
decision that is already resolved.

## Linked Issues or Issue Description

**What happened?**

A board user pressed Retry on blocked tasks. No run started. The task
said "Retry is waiting for a pending approval or question to be
resolved. Your message is saved." In one task, the only interaction was
a `request_confirmation` that the previous successful run created. The
user's later comment started the turn that failed. The confirmation
stayed pending, so every Retry attached to the same held receipt. After
the user rejected the confirmation and sent a new comment, the task
still showed the same wait. No run started until a server restart
rescanned the deferred receipts about three minutes later. The task
projection said `canRetry: true` with the `decision_pending` wait.

**Expected behavior**

An accepted Retry is never held on a decision that is already resolved.
Accepting or rejecting the decision releases a held Retry right away. A
person's comment on the blocked task starts work, and the held Retry is
settled. The task projection never shows a resolved decision as the
Retry wait.

**Steps to reproduce**

1. Let an agent run on a remote runner create a `request_confirmation`
for the task.
2. Let the runner be lost so the run fails with
`native_continuation_requires_reconciliation`, and let saved work
recovery complete.
3. Press Retry. The Retry receipt is saved with `executionWait.reason =
decision_pending`.
4. Reject or accept the confirmation.
5. Observe that the receipt keeps `decision_pending`, the task keeps
showing the wait, and the Retry does not start until an unrelated
rescan.

**Paperclip version or commit**

Staging Core `57aeda190b`. Fixed on top of `origin/master` `53608bd1d6`.

**Deployment mode**

Hosted cloud (staging), remote sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime).

## What Changed

- `hasPendingContinuationDecision` (`explicit-native-continuation.ts`)
is now the single check for a decision that holds a person's
continuation. Admission and the task projection both use it.
- For a Retry, a pending interaction that was created before the failed
source run was created does not hold the Retry. That turn was already
admitted while the decision was open. Decisions that the failed run
created (for example a plan review) still hold the Retry.
- `releaseDecisionHeldExecution` (`deferred-execution-wait.ts`) removes
the saved `decision_pending` wait from deferred receipts on the task and
hints each receipt to the deferred-receipt worker. It runs in the
resolution transaction of every interaction resolution path (through
`notifyChatConfirmationResolution`) and in question answers.
- The task projection reads the live decision state. A saved
`decision_pending` wait whose decision is resolved is not shown. This
also repairs receipts saved before this change.
- When a comment or an interaction answer continues a failed run,
`retireTaskRetriesContinuedBy` (`task-retry.ts`) settles held Retry
receipts for the same source run as `task_retry_superseded`.

## Verification

- Red/green: `cd server && npx vitest run
src/services/governed-wait-runner-loss.test.ts`. The five new tests fail
on `origin/master` sources and pass with this change (17/17 pass):
  - releases the held Retry as soon as the person chooses accept
  - releases the held Retry as soon as the person chooses reject
- never shows a resolved confirmation as the Retry's wait, and both
projections offer the same Retry
- a person's comment starts work and settles the held Retry once nothing
is left to answer
- starts a Retry at once when the confirmation was already open before
the failed turn started
- The existing tests that keep a Retry or a message on hold while a plan
review, governed tool action, secret proposal, or connection
authorization is pending still pass.
- Related suites: deferred execution wait, explicit native continuation,
interaction resolution and service, task interruption presentation and
query, task retry, deferred receipt work, interaction routes.

## Risks

- Medium-low. A Retry now starts when the only pending interaction is
older than the failed run. That turn already ran with the interaction
open, so the Retry gets no new authority. Governed tool actions still
need their own approval before the tool runs.
- Interaction resolution now also updates deferred receipts on the same
task. The update is limited to receipts with a saved `decision_pending`
wait. A receipt that is still blocked by another pending decision is
held again on its next admission.
- Issue-linked approvals (`approvals` table) do not yet hint the held
receipt when they resolve. The projection no longer shows them as the
wait after they resolve.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:58:48 -06:00
DottaandClaude Opus 5.5 f2e449f8f6 fix(retry): continue an accepted Retry past chat input held because it predates the stop (#21129)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's Retry on a connected task (for example a GitHub pull
request review) must start work. #21080 made a Retry whose chat replay
is refused after the click continue from the recovered saved work
> - After #21080 was deployed, accepted Retries on a busy GitHub review
agent still did not start
> - Those tasks had a saved chat request that arrived while the failed
run was still running. Admission holds such a request as
`message_predates_stop`, so it can never start alone
> - #21080 refused to continue while any saved chat input existed, so
this held input blocked the continuation. For a second group, the latest
run was an admission that the recovery hold cancelled before dispatch,
and the recovered saved work belongs to the earlier failed run
> - This pull request lets the continuation adopt input that was held
because it predates the stop, and lets the recovered run that held a
cancelled admission count as the saved work
> - The benefit is that an accepted Retry starts work on these tasks too

## Linked Issues or Issue Description

Follow-up to #21080. No public issue exists. This is the bug
description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

After #21080 was deployed, `POST /api/issues/:id/retry` returned 202 for
blocked tasks of a GitHub review agent (`paperclip_runner`), but no run
started. Each Retry receipt changed to `failed` about 5 seconds later
(on the 14:48 Retries, 5 seconds for one task and about 75 seconds for
two others), with no run and no `issue.retry_continued_from_saved_work`
activity. Every affected task had a saved chat request (a newer GitHub
review request) with an `issue.execution_waiting` record of reason
`message_predates_stop`. The request arrived while the failed run was
running, so admission holds it permanently. The interruption projection
kept pointing at that held request. In a second group of tasks, the
latest run was cancelled before dispatch with
`execution_reconciliation_required`, and the recovered saved work
belonged to the earlier failed run that held it.

Root cause: in `continueRefusedTaskChatRetry`
(`server/src/services/task-retry.ts`), any saved chat input from
`selectTaskRetryInput` stopped the continuation, so the receipt settled
`failed`. The held input can never start by itself, so nothing ran. The
recovered-work check also looked only at the latest run, so it missed
the recovered run that held a cancelled admission.

**Expected behavior**

An accepted Retry starts work. Saved input that admission holds because
it predates the stop joins the continuation, and the task continues from
the recovered saved work.

**Steps to reproduce**

1. Bind a task to a GitHub chat conversation. Let a native run fail with
`native_continuation_requires_reconciliation`, and recover its saved
work.
2. Save a chat request while that run is still running, so admission
holds it with `message_predates_stop`.
3. Click Retry while the chat identity lock is busy, so the source check
is deferred and then refused.
4. The route returns 202. The Retry receipt changes to `failed`, and no
run starts.

**Paperclip version or commit**

a65644d1e1 (the deployed #21080). Reproduced on ef2b53b675.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

`paperclip_runner` (native runs, and legacy admissions cancelled before
dispatch). The fix is in core Retry code and is not adapter-specific.

## What Changed

- `server/src/services/task-retry.ts` (`continueRefusedTaskChatRetry`):
- Saved chat input whose source comment was written no later than the
failed run's stop boundary (`userStopBoundaryAt`: the recorded user Stop
request, otherwise `finishedAt`) no longer stops the continuation. This
is the same boundary and timestamp that admission uses. Admission holds
that input as `message_predates_stop`. The continuation coalesces its
receipts into the Retry receipt (`coalescedIntoWakeupRequestId`), so the
task stops waiting on it. The receipt ids are recorded in the activity
details.
- Saved input created after the stop, and unadmitted chat input, still
settle the Retry as failed, as before.
- When the latest run is an admission cancelled before dispatch
(`execution_reconciliation_required`, never started), the recovered-work
check also accepts the run named by that run's
`executionWait.recoveryActionId`.
- `doc/execution-semantics.md`: both rules.
- Tests in `server/src/__tests__/chat-channels.integration.test.ts`: two
new variants of the #21080 regression test.
`person_with_input_before_stop` saves a chat request before the failed
run stops. `person_after_held_admission` adds a legacy admission that
the recovery hold cancelled. Both start one run that resumes the
recovered workspace (`resume_preserved_workspace`). The held-admission
fixture binds the task to the recovered workspace, as on the hosted
tasks. Refusal edges `input_after_stop` and `input_inside_user_stop`
settle the Retry failed with no run and leave the input deferred.

## Verification

- Red before the fix: both new variants fail. The Retry receipt is
`failed` (or stays held), and no run starts.
- Green after the fix: both variants pass, and the held input receipt is
coalesced into the Retry receipt.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts -t
"[Rr]etry"` with `src/services/task-retry.test.ts` passes 369/369 on the
final commit.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Staging was read with GET requests only.

## Risks

- Behavior change: input that arrived during a failed run no longer
starts its own turn through the chat retry. It is adopted into the
saved-work continuation, and the comment stays in the task thread.
Before this change, that input could never start at all.
- Input that arrived after the stop keeps its own admission path.
- No migration. Receipts that already failed stay failed. The next click
takes the new path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:34:36 -06:00
DottaandClaude Opus 5.5 a65644d1e1 fix(retry): continue an accepted connected Retry from recovered saved work when its chat replay is refused later (#21080)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks from a connected chat (for example GitHub pull request
reviews) show Retry when a run fails
> - A person's Retry on such a task first asks the chat retry owner to
repeat the exact failed chat request. #21038 made a refusal at the click
continue the task from saved work instead
> - The chat owner can also defer its source check (a busy identity,
endpoint, or agent row lock) and refuse it a few seconds later. Then the
route has already answered 202, and the task Retry receipt was marked
failed with no run
> - The task stayed blocked, and its execution blocker told the person
to send a message, while the interruption projection still offered Retry
> - This pull request makes that late refusal take the same board
continuation as a refusal at the click, and makes the blocker offer
Retry for recovered saved work
> - The benefit is that an accepted Retry always starts work, and the
two projections agree

## Linked Issues or Issue Description

No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a hosted deploy, about 20 blocked tasks of one busy GitHub review
agent (`paperclip_runner`, native remote runs) did not restart after
Retry. `POST /api/issues/:id/retry` returned 202 with `accepted: true`,
a `chatActionId`, and `presentation.state: retry_scheduled`. No run was
created. The task Retry receipt changed to `failed` 5 to 10 seconds
later, and the task stayed blocked. The activity record of each click
(`chat.failed_run_retry_requested`) has `sourceAuthorizationPending:
true`, so the chat owner deferred its source check at the click. No chat
wake row was written, so the chat replay ended before admission. The
last run failed with `native_continuation_requires_reconciliation` /
`runner_remote_process_identity_unavailable`. Its saved work was
recovered (`workspaceRecovery.status: recovered`, remote lease destroyed
with a termination receipt). The projections disagreed: `interruption`
had `canRetry: true`, but `executionBlocker` had `canRetry: false` and
`nextAction: "Saved work recovered. Send a message to continue."`.

Root cause: when the chat retry action settles `failed`,
`releaseUnadmittedTaskRetryInput` in
`server/src/services/chat-channels/task-retry-input.ts` marked the
linked task Retry receipt `failed`. #21038 added a board-continuation
fallback only for a refusal inside the click transaction. A refusal
after a deferred source check (for example 8 lock-busy attempts on the
agent row, about 7 seconds) had no fallback. Separately,
`getExecutionBlocker` hid Retry for every chat-bound task (`canRetry &&
!chatBinding`), and `workspaceRecoveryDisplay` always said "Send a
message to continue" for recovered work.

**Expected behavior**

A Retry that is accepted starts work. If the exact chat replay is
refused after the click, the person's Retry continues the task from its
recovered saved work. The execution blocker and the interruption both
offer Retry, and neither tells the person to send a message to unblock a
failed run.

**Steps to reproduce**

1. Bind a task to a GitHub chat conversation. Make its latest native run
fail with `native_continuation_requires_reconciliation` and recover its
saved work.
2. Hold the chat identity advisory lock while you click Retry, so the
source check is deferred (`sourceAuthorizationPending: true`).
3. Hold a row lock on the agent while the chat retry processes, so the
deferred check fails 8 times.
4. The route returned 202. The chat action settles `failed`, the task
Retry receipt changes to `failed`, and no run starts.

**Paperclip version or commit**

`master` at a8580160 (the deployed revision). Reproduced on f30005440b.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

`paperclip_runner` (native remote runs). The fix is in core Retry code
and is not adapter-specific.

## What Changed

- `server/src/services/task-retry.ts`: new `settleRefusedTaskChatRetry`.
When a chat retry action settles `failed`, the linked task Retry receipt
becomes an ordinary board continuation if all of these are true: it is a
person's Retry (or an automatic Retry that the same person's click has
joined), its source run is still the latest run and its saved work is
recovered (system recovery `complete`, workspace recovery `recovered`),
the clicking person still has task access, the chat wake has no run, and
no saved or unadmitted chat input remains. Any other refusal still
settles the receipt as failed, so the existing source-edited, revoked,
and identity-expiry outcomes do not change. A paused agent does not fail
the receipt: it stays deferred and starts when dispatch allows. The
receipt drops `chatActionId`, stays `deferred_issue_execution`, notifies
the deferred-receipt worker, and records
`issue.retry_continued_from_saved_work`. Otherwise the receipt is failed
as before. `reconcileTaskChatRetryReceipts` uses the same check, so a
crash before settlement is also covered. Its failed write matches the
chat binding it read, so a stale pass cannot fail a receipt that
settlement already converted.
- The task Retry intent now records `initiatedBy` for an automatic
Retry, so an automatic Retry keeps the exact-replay requirement after
the click too.
- `server/src/services/chat-channels/task-retry-input.ts`:
`releaseUnadmittedTaskRetryInput` calls the task owner's settlement
instead of failing the receipt directly.
- `server/src/services/execution-blocker.ts` and
`workspace-recovery-receipt.ts`: for a failed or timed-out run whose
saved work is recovered, the blocker offers Retry also on a chat-bound
task. The text follows actual Retry eligibility: "Saved work recovered.
Retry continues from it." when Retry is offered, "Saved work recovered."
when it is not (for example cleanup quarantine), and "Send a message to
continue" only for a stopped run. Only the task blocker decides Retry.
The run-level projection (`execution-projection.ts`) has no task
context, so it says "Saved work recovered." and never claims Retry.
`execution-projection.ts` uses the same text.
- `ui/src/lib/activity-format.ts`: labels for the new activity.
- `doc/execution-semantics.md`: the late-refusal rule.
- Tests: a new real-service case in
`server/src/__tests__/chat-channels.integration.test.ts` that reproduces
the hosted state (remote saved workspace, destroyed lease receipt,
recovered recovery action, deferred source check, refused replay), for a
person's Retry and for an automatic Retry. A gate case per continuation
condition (unrecovered workspace and revoked access settle failed; a
paused agent stays deferred and runs after resume), a stale-reconcile
race case, and a joined-click case. Two existing text expectations are
updated.

## Verification

- Red before the fix: with only the source changes removed, the new test
fails on the projection (`canRetry: false`, "Send a message to
continue."). With only the Retry settlement removed, it fails with the
receipt `status: "failed"` and no run.
- Green after the fix: `npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "continues an
accepted GitHub task Retry from recovered saved work"` passes 2/2. The
person's Retry starts one queued run that resumes the preserved
workspace, and the automatic Retry still settles as failed.
- Focused suites pass: `task-retry`, `execution-projection`,
`execution-projection-recovery`, `workspace-recovery-retry-copyback`,
`chat-failed-run-retry-boundary`, `workspace-recovery-admission`,
`remote-workspace-recovery-receipt-fallback` (372/372), and other
recovery, blocker and continuation suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Staging was read with GET requests only. No POST was sent.

## Risks

- Behavior change: a person's Retry on a connected task that the chat
owner refuses after the click now starts a board continuation. It is the
same continuation #21038 already uses for a refusal at the click, with
the clicking person's authority, and dispatch keeps every normal gate.
- Receipts that already failed on staging stay failed. The next click on
those tasks takes the fixed path. No migration.
- The blocker now reports `canRetry: true` for chat-bound tasks only
when the failed run's saved work is verifiably recovered.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:57:42 -06:00
DottaandClaude Opus 5.5 3c058c37e6 fix(retry): continue a connected task from saved work when its exact chat request cannot be repeated (#21038)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks that come from a connected chat (for example GitHub pull
request reviews) show a Retry button when a run fails
> - The task projection offers Retry, but the Retry route sends a
connected task to the chat retry owner, which only replays the exact
failed chat request
> - When that exact request cannot be replayed (the attempt already
saved a result, its output was published, the latest run did not start
from a chat message, or the requester's access changed), the route
returns 409 and tells the person to check the connection by hand
> - A Retry button that is shown must work, and recovery must not ask
the person to inspect anything
> - This pull request makes a person's Retry continue the task from its
saved work when the exact chat replay is refused, the same way a new
message on the task would
> - The benefit is that the projection and the route agree, and blocked
connected tasks become retryable after deploy with no migration

## Linked Issues or Issue Description

No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a hosted deploy, five blocked tasks that came from a connected GitHub
conversation showed Retry (`interruption.canRetry: true`). Each click on
Retry returned 409 `chat_failed_run_retry_not_authorized`. The message
was "The exact failed chat request can no longer be retried safely.
Check its current connection, access, and recovery state." or "This
attempt already recorded a result. Send a new message on the task to
continue from the saved work." The latest runs were legacy sandbox setup
failures, native runs that needed reconciliation, a run that started
from an issue monitor instead of a chat message, and a native run that
already saved its result. Root cause: `admitTaskRetry` in
`server/src/services/task-retry.ts` called the chat retry owner for
every connected task and passed its exact-replay refusal straight to the
person. The projection cannot predict that refusal.

**Expected behavior**

A shown Retry works. If the exact chat request cannot be repeated, Retry
continues the task from its saved work, the same way a new message on
the task would. Users do not get operator instructions.

**Steps to reproduce**

1. Bind a task to a restricted chat conversation.
2. Make its latest run fail so that the chat retry owner refuses the
exact replay. For example, use a run whose source is not a chat message,
or a native run with an accepted result.
3. Click Retry (`POST /api/issues/:id/retry`).
4. The route returns 409 `chat_failed_run_retry_not_authorized`.

**Paperclip version or commit**

`master` at 5a7378af5 (the deployed revision), reproduced on 51d77648bc.

**Deployment mode**

Hosted (cloud) instance with Daytona remote sandboxes and a GitHub chat
connection.

**Agent adapter(s) involved**

Not adapter-specific (core bug). Seen with `paperclip_runner` legacy and
native runs.

## What Changed

- `server/src/services/task-retry.ts`: a person's Retry runs the exact
chat replay inside a savepoint. On a
`chat_failed_run_retry_not_authorized` refusal, the savepoint rolls back
and the same Retry receipt stays an ordinary board continuation. The
clicking person is the responsible user, the run starts a fresh session
(`forceFreshSession`) from saved work, and no chat action, chat
authority, or output replay is created. Retry that Paperclip requests by
itself (`initiatedBy: restart_recovery | provider_limit_reset`) keeps
the exact-replay requirement. Other errors (403, lock contention,
database errors) still propagate.
- `server/src/services/chat-channels/failed-run-retry.ts`: plain refusal
wording. Runs saved by an earlier release with the old diagnostic are
still recognized by the pre-provider check.
- Tests: new red/green cases in `server/src/services/task-retry.test.ts`
(four refusal reasons on a chat task whose latest run came from an issue
monitor, plus automatic Retry still refused). Updated two real-service
route cases in `server/src/__tests__/chat-channels.integration.test.ts`
(accepted native result on an inherited GitHub task, and an unknown chat
source) from expecting 409 to expecting a board continuation without a
chat replay.
- A person's Retry on that person's own saved board turn (`source:
issue.comment.reopen`) that cannot be certified for replay also
continues from saved work, without replaying the saved message.
Automatic Retry still gets 409 `saved_board_message_not_replayable`.
- A person's Retry when no chat retry owner is wired also continues from
saved work. Automatic Retry still gets 409.
- The fallback is not used while the conversation holds saved or
unadmitted chat input for the task. That input keeps its own admission
path and the existing refusal.
- Docs: `doc/execution-semantics.md` and `doc/connections/GITHUB.md`.

### 409 paths in the task Retry route after this change

| Path | Who sees it | Result |
| --- | --- | --- |
| `chat_failed_run_retry_not_authorized` from the chat retry owner |
Person | Continues from saved work (board continuation) |
| `saved_board_message_not_replayable` | Person | Continues from saved
work |
| No chat retry owner wired | Person | Continues from saved work |
| Any of the three above while saved or unadmitted chat input is pending
| Person | 409. The pending input continues through its own admission
and delivery. |
| Any of the three above | Automatic Retry (restart recovery, provider
limit reset) | 409, recorded by the hook. No button. |
| `task_retry_busy` (chat authorization lock held) | Person | 409 with
`retryable: true` after 10 internal attempts. A new click works. |
| Idempotency key reused for a different task or user | Client error |
409. The UI makes a new key for each click. |

## Verification

- Red before the fix: `npx vitest run src/services/task-retry.test.ts -t
"connected task"` failed 4/4 with `Error: refused` (409). The board-turn
and missing-owner cases failed with "This saved message cannot be
retried safely..." and "Retry is unavailable for this connected
conversation." `src/__tests__/chat-channels.integration.test.ts -t
"continues an inherited GitHub task"` failed with `expected 202
"Accepted", got 409 "Conflict"`.
- Green after the fix: `src/services/task-retry.test.ts`,
`src/__tests__/durable-chat-wakeup.test.ts`,
`src/services/budget-resume-continuation.test.ts`: 160/160 passed.
- Full `src/__tests__/chat-channels.integration.test.ts`: 1949/1951
passed. The 2 local failures (`binds Slack tools...`, `materializes
direct external-chat finals...`) read `.claude/skills/slack/SKILL.md`
from the parent checkout of a nested local worktree (ENOENT). They do
not touch Retry and depend on the local checkout layout.
- `npx tsc --noEmit -p server/tsconfig.json`: passed.
- `pnpm verify` exited 2 (a doc file is a global CI input), so required
CI supplies broad coverage.
- Staging was inspected with GET requests only. No POST was sent to
staging.

## Risks

- Behavior change: a person's Retry on a connected task can now start a
board continuation where it was refused before. It uses the same
authority as a new message from that person, and dispatch keeps all
normal gates (task access, executor containment, recovery holds,
unadmitted chat input).
- The continuation does not publish the earlier result again and does
not reuse the chat requester's authority.
- No migration. Existing blocked tasks become retryable at the next
click.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 01:27:54 -06:00
DottaandClaude a46b9ac724 fix(native): wait out an ACP usage limit's reported reset, then Retry the task (#20692)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs (Paperclip Runner with an ACP harness, such as Claude)
can stop because the provider account runs out of its usage allowance
> - The ACP harness reports this as a "terminal limit failure".
Paperclip blocked the task behind a board hold that called the session
state ambiguous, and the only way forward was a manual Retry
> - The provider also says when the limit resets, for example "resets
Nov 12, 6pm (UTC)" for a weekly limit. Paperclip read only clock-only
resets ("resets 2am"), so the agent-level wait fell back to one hour.
Each later wake for the agent then hit the limit again and blocked
another task
> - This pull request reads calendar resets, says on the hold when the
limit resets, and requests the task's Retry itself at that time when the
reset is reported and bounded
> - The benefit is that a usage limit no longer needs a person to click
Retry, and the agent's other tasks wait instead of failing one by one

## Linked Issues or Issue Description

**What happened?**
A native ACP run failed with `native_provider_usage_limit` ("ACP agent
reported a terminal limit failure. You've hit your weekly limit · resets
Nov 12, 6pm (UTC)"). The task went to `blocked` with a board-owned hold.
The hold said "Inspect the original provider failure and explicitly
resolve the ambiguous session state". Nothing resumed the task after the
reset. The agent-level `provider_limit_wait` recorded "no reset time was
reported, so Paperclip waits one hour", so every hour another wake
dispatched, hit the limit, and blocked another task. In one 48-hour
window on a staging instance, this blocked 7 tasks for one agent.

**What did you expect?**
When the provider reports a bounded reset, the task waits and continues
after the reset, like the existing `provider_limit_wait` path. When the
reset is unknown, the task fails with a clear message and a working
Retry.

**Steps to reproduce**
1. Run a native ACP agent until its provider reports a weekly or session
limit with a reset time.
2. Look at the task: it is blocked with an "ambiguous session state"
hold.
3. Wait past the reported reset: the task stays blocked until a person
clicks Retry.

Refs #20666 (other recovery work in the same staging census; not changed
here)

## What Changed

- `provider-quota-reset.ts`: parse calendar resets such as "resets Nov
12, 6pm (UTC)", "resets Jul 17th at 4:30 PM (America/Chicago)" and
"Resets on August 2, 9am (UTC)". A past date rolls into next year. An
impossible date (Nov 31) is refused. Clock-only parsing is unchanged.
This also fixes the agent-level `provider_limit_wait` reset and the
legacy `provider_quota` reset for these messages.
- New `native-runtime/provider-limit-resume.ts`:
- `readNativeUsageLimitReset` reads the reset only from the run's own
durable events. The last turn event must be a provider-observed
`turn.failed` with a `rate_limit` or `quota_exhausted` provider failure,
and the reset must be at most eight days out.
- `requestProviderLimitResumeRetry` asks for the task's Retry
(`requestTaskRetry`, `initiatedBy: "provider_limit_reset"`) on behalf of
the user who requested the run, or the run's accountable user for
automatic wakes. It is idempotent per run and records its outcome on the
hold. At most three automatic Retries follow one another without a
successful run.
- `registerProviderLimitResumeWork` is a scoped deadline owner (shared
work scheduler, no polling). Each active usage-limit hold is due at its
reset. Startup rebuilds it from the holds, and the executor notifies it
in the hold's transaction.
- `native-session-executor.ts`: for a terminal usage limit, the hold and
run failure detail say when the limit resets and whether Paperclip
retries then, instead of "ambiguous session state". The evidence records
`providerLimitResetAt`.
- `task-retry.ts`: accept `initiatedBy: "provider_limit_reset"`. Its
activity is recorded as system actor `provider-limit-reset` on behalf of
the user, like `restart_recovery`.
- `doc/execution-semantics.md`: document the wait and its limits.

Previous-release story: holds saved by the previous release carry
forward. At startup the owner reads each active
`native_provider_usage_limit` hold. It rewrites the hold's message with
the reset when the reset is known and still ahead, and it requests the
Retry at the reset. Holds whose reset is unknown, more than a day in the
past, or that have no accountable user keep their Retry and are recorded
as not requested, so they are not read again.

## Verification

- `npx vitest run server/src/services/provider-quota-reset.test.ts` — 4
tests. Three fail on `origin/master` code (calendar resets return
`null`), all pass with this change.
- `npx vitest run
server/src/services/native-runtime/native-provider-capacity.integration.test.ts`
— 5 tests on embedded Postgres:
- the existing usage-terminal test (both replay modes) still blocks with
a Retry, and no automatic Retry is requested without a reported reset;
- "waits for a reported usage-limit reset and then requests the task's
Retry once": the hold says when the limit resets, an old-release hold is
rewritten, nothing starts before the reset, one Retry receipt is created
for the accountable user at the reset, and a second pass does nothing.
Fails on `origin/master` executor code;
- "starts the Retry from the hold's own deadline": a real delivery
coordinator rebuilds the deadline from the hold and calls Retry dispatch
once, after the reset;
- "leaves a visible Retry when no user is accountable for the run".
Fails on `origin/master` executor code.
- Focused regression suites pass (642 tests): `task-retry`,
`quarantined-task-retry`, `heartbeat-native-restart-suspension`,
`native-startup-retry.integration`, `provider-failure-classification`,
`heartbeat-provider-limit-wait`, `deferred-execution-receipt-work`,
`chat-run-publications`, `native-runner-restart-recovery.integration`,
`native-remote-retry-admission`.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.
- `pnpm verify` selected a broad run (exit 2, because the docs file is a
global CI input). CI supplies the broad coverage.

## Risks

- The automatic Retry uses the accountable user's authority for
automatic (system) wakes. It is the same Retry the hold already offered
that user, and dispatch still checks Retry authority, stop proof, saved
work and action reconciliation. It is never broader than a click.
- A provider that keeps reporting resets that do not restore capacity
gets at most three automatic Retries in a row; then the Retry stays
visible.
- The deadline owner reads all active usage-limit holds at startup
(limit 500 rows). The number of such holds is small.
- Not changed: the agent-level wait still holds new wakes for at most
one day, as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads). Extended
reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 20:57:03 -06:00
DottaandClaude c37eacbee3 fix(native): Retry re-finalizes a saved result that ran out of finalization retries (#20578)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save an accepted result. Then a finalizer applies that
result to the task.
> - Finalization has its own budget of 3 failed attempts (#20555). When
the budget runs out, the coordinator goes to `terminal_failure` with
`native_finalization_retry_exhausted`.
> - For a provider-succeeded run, the task then rests `in_review` with a
board recovery card. The task shows "resolved" and offers no Retry. The
user is stuck.
> - The ordinary Retry is also wrong here: it starts a new provider run,
which repeats work the agent already finished.
> - This pull request adds two things. First, contention failures on
such a result get two longer automatic delays before the board gets the
task. Second, when the state is still exhausted, the task offers Retry,
and that Retry applies the same saved result again.
> - The benefit: fewer tasks need a Retry, and when one does, Retry
works and does no duplicate provider work.

## Linked Issues or Issue Description

Refs #20555 (it names this gap under Risks).

**What happened?**
In restart-chaos wave `w-20261110T044338-s20385`, task t03 ended with a
succeeded run, an accepted result, and a coordinator in
`terminal_failure` / `native_finalization_retry_exhausted`. The issue
stayed `in_review` with a board-owned `active_run_watchdog` card and
`canRetry=false`. #20555 stopped the early exhaustion. A real exhaustion
still leaves the same stuck state. Coordinators that older builds
already left in this state are stuck the same way.

**Expected behavior**
Transient finalization failures recover by themselves when that is safe.
When they do not, the task offers Retry. That Retry re-finalizes the
saved result. It does not start a provider run and does not charge a
provider attempt.

**Steps to reproduce**
1. Let a native run save an accepted result and end `succeeded`.
2. Make each finalization of that result fail until the budget of 3 runs
out.
3. Before this fix, the task presentation is `resolved` with
`canRetry=false`, and the board card has no action that applies the
result. The new test
`server/src/__tests__/native-exhausted-finalization-retry.test.ts`
reproduces this.

**Paperclip version or commit**
master at c58356003f (after #20555).

## What Changed

- `native-run-finalizer.ts` (`recordRetryableFailure`): when the budget
runs out on an accepted result of a provider-succeeded run, and the
cause is contention (`status_cas_exhausted` or
`side_effect_planning_failed`), the coordinator stays
`retryable_failure`. It tries again after 2 minutes, then after 10
minutes, each with a fresh budget. Each delay is recorded once in
`recoveryHistory` (`automatic_finalization_backoff`). After both delays,
the old terminal path runs. Deterministic causes (for example
`native_finalization_invalid`) and workspace-scope failures are
unchanged. The existing deadline worker (`native-result-commit-work`)
runs the retry, so no new timer or poll is added.
- New `native-runtime/exhausted-finalization-retry.ts`:
- `exhaustedNativeFinalizationCondition()`: one SQL predicate for
"succeeded native run, accepted result, coordinator `terminal_failure`
with `native_finalization_retry_exhausted`, no lease, no successor".
- `rearmExhaustedNativeFinalization()`: under the task lock, reopens the
same coordinator (`arbitrating` or `workspace_finalizing`), records
`exhausted_finalization_retry` in `recoveryHistory`, resolves the board
card (`handed_back`), and resets `finalizationFailureCount` to 0. It
sets a 30 s fallback `nextAttemptAt`, so the deadline worker finishes
the job if the process stops first.
- `refinalizeSavedNativeResult()`: runs `finalizeNativeRun` for the same
run with `preserveProviderAttempt: true`. A failure goes through the
finalizer's normal bounded budget.
- `task-interruption-presentation.ts`: the run read adds
`nativeFinalizationRetry` from that predicate. A succeeded run with this
flag shows `state: "failed"`, `canRetry: true`. It also adds
`nativeFinalizationRetryPending` (`savedResultRetryPendingCondition()`:
coordinator not committed or terminal, with an
`exhausted_finalization_retry` entry). While it is true the task shows
`state: "working"` with `recoveryStep: "finishing_recovery"`, so the
user sees the Retry took effect. Done or cancelled tasks, user gates,
and ordinary runs are unchanged.
- `task-retry.ts` (`admitTaskRetry`): for a succeeded native run in the
exhausted state, Retry re-arms the coordinator instead of creating a
provider retry. It writes a `coalesced` `task_retry` receipt bound to
the same run with strategy `finalize_result` (an existing strategy
value), joins any unadmitted Retry intent to it, and logs
`issue.retry_requested` with `strategy: "finalize_result"`. The
idempotency key still returns the same receipt on replay. A later click
with a new key, after an earlier saved-result Retry of the same run
(still finalizing, or applied), joins that Retry with another coalesced
`finalize_result` receipt and starts no provider work.
- `routes/issues.ts` (`POST /issues/:id/retry`): after the receipt
commits, it starts `refinalizeSavedNativeResult` for that run instead of
the provider Retry dispatch.
- `doc/execution-semantics.md`: documents the automatic delays and the
re-finalizing Retry.

## Verification

- `cd server && npx vitest run
src/__tests__/native-exhausted-finalization-retry.test.ts` (4 tests):
- Exhausted state reports `state: "failed"`, `canRetry: true`. Retry
re-arms the coordinator, writes a `finalize_result` receipt and an
`issue.retry_requested` activity row, and replays idempotently. A second
click with a new key while pending, and a third after commit, join the
same Retry; no provider Retry wake exists. While pending, the task shows
`working` / `finishing_recovery`. Re-finalization commits, and the task
shows resolved. The run count for the task stays 1, the coordinator
`attempt` is unchanged, the run stays `succeeded`, and no board card
stays active.
- Old-shape row (no `finalizationFailureCount`, `attempt: 3`, already
terminal): the same Retry works with no migration.
- Re-finalization that fails again: the coordinator goes to
`retryable_failure` with a fresh count of 1, then to `terminal_failure`
after the budget, and the task again shows `canRetry: true`. Nothing is
lost silently.
- Contention failures (failpoint `status_projection`): two automatic
delays (2 min, 10 min) with the task status unchanged, then the board
Retry, which commits with no new run.
- Red/green: with only the new module and test applied (finalizer,
presentation, task Retry, and route reverted), all 4 tests fail
(`canRetry` false; no automatic delay).
- `npx vitest run src/services/task-interruption-presentation.test.ts`
adds unit cases: the flag offers Retry; ordinary succeeded runs stay
resolved; ordinary failed runs keep `canRetry: true`; done tasks stay
resolved.
- Also passing: `native-finalization-retry-budget`,
`native-finalization-recovery`, `native-run-finalizer-telemetry`,
`task-interruption-query`, `interrupted-preparation-restore-retry`,
`native-result-commit-work`, `workspace-recovery-admission`,
`workspace-recovery-preparation-gates`,
`deferred-execution-receipt-work`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- Not run: the full local suite (`pnpm verify` selects about 2,450 files
for these shared modules; CI covers it), and no live restart-chaos wave.

## Risks

- Previous-release story: coordinators saved by older builds in
`terminal_failure` / `native_finalization_retry_exhausted` qualify by
phase and code. The count is optional. Nothing is migrated. Older rows
do not get the automatic delays; they get the working Retry.
- Automatic delays add at most two extra finalization rounds of the same
saved result, only for contention causes, only for provider-succeeded
accepted results. During the delays the task keeps its prior status, as
it does during the normal 30 s retries. Supersession by a newer run is
still checked first on every attempt.
- The re-finalizing Retry applies the agent's saved result after a
person may have looked at the task in review. It uses the normal status
arbiter and CAS. Done and cancelled tasks, other assignees, and newer
runs refuse it.
- No change before `turn.start`: all new work is on the finalization,
presentation, and Retry paths.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code
with tool use (shell, code editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 11:48:47 -06:00
DottaandClaude 1e13c58a4e fix(heartbeat): name the cause of generic adapter failures and keep SQL text off the board (#20483)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Heartbeat run settlement records why a run failed, and the board
shows that record.
> - Runs that fail without a named code settle as `adapter_failed` (or
`setup_failed`) with no cause.
> - A user, an operator, and the restart chaos harness cannot tell what
went wrong from such a run.
> - Some run errors also show raw SQL statement text to users.
> - This pull request records a bounded `failureCause` code for generic
failures, shows it on the board, and keeps SQL text out of run errors.
> - The benefit is that a generic failure now names its cause, for
example "adapter failed (40001)", without leaking messages, SQL, or
secrets.

## Linked Issues or Issue Description

Refs #20385 (the nested-cause extraction this change reuses). Related:
#20421 (restart chaos universal checks U1-U4).

**What happened?**

The restart chaos audit of 79 saved waves found 30 failed runs with
`errorCode: "adapter_failed"` and a null or generic error. Every U3
("diagnosable") warning was `adapter_failed`. Nobody could tell what
failed. Some board run errors also showed raw SQL statement text from
wrapped database errors (Drizzle `Failed query: ... params: ...`).

**Expected behavior**

A run that fails with a generic code names a short, safe cause. The
board shows the cause next to the stop reason. Run error text never
shows SQL statements or their parameters.

**Steps to reproduce**

Make an adapter throw a wrapped database error (an `Error` whose nested
`cause` has `code: "40001"` and whose message has a Drizzle `Failed
query:` statement). Before this change, the run settles as
`adapter_failed` with no cause, and `heartbeat_runs.error` contains the
SQL statement and parameters. The new tests in
`server/src/services/heartbeat/run-completion.test.ts` reproduce this.

## What Changed

- Runner: move the #20385 event-commit cause extraction into
`safeErrorCauseCode`
(`packages/paperclip-runner/src/control-plane/safe-error-cause.ts`) and
export it. The Runner's own diagnostics use the same function, unchanged
in behavior.
- Shared: add the `failureCause` contract
(`packages/shared/src/run-failure-cause.ts`): one `[A-Za-z0-9_.:-]`
token of at most 80 characters, `normalizeRunFailureCause`,
`readRunFailureCause`, and `withoutSqlStatementText`.
- Server: add `server/src/services/heartbeat/run-failure-cause.ts`.
Settlement records `resultJson.failureCause` only when the code is
generic (`adapter_failed`, `setup_failed`). `errorCode` stays the same.
- Thrown errors (`failRun`, `failRunSetup`): `provider_http_<status>`,
else `provider_<category>`, else the first safe code in the error and up
to three nested causes (`40001`, `ECONNRESET`), else the error class
name, else `unknown_error`.
- Adapter results without a code (`completeRun`): the error family,
`signal_<name>`, `process_cancellation_failed`, `exit_code_<n>`,
`adapter_reported_error`, or `unknown_error`.
- The failure log lines (`heartbeat execution failed`, `heartbeat
execution setup failed`, and a new `heartbeat run failed` warning for
generic adapter results) include `failureCause` and `errorCode`.
- Server: replace SQL statement text in run error text with "Database
query failed (<cause>)" at all three settlement paths, including the
cancelled branch. The execution blocker's `runError`, the attention
inbox (failed-run excerpt and metadata), and continuation summaries
apply the same replacement on read, so older rows also stop showing SQL
there.
- Server: Retry admission and failed-preparation replay accept only a
closed set of `resultJson` keys. `failureCause` is added to each of
those lists (`task-retry.ts`, `chat-channels/failed-run-retry.ts`,
`cancelled-native-startup.ts`, `fresh-workspace-continuation.ts`). It is
a diagnostic code with no execution or replay authority, so a
failed-preparation run stays retryable.
- Server: `runsForIssue` returns a validated top-level `failureCause`.
It is null for older rows, non-string JSON values, and malformed values.
- UI: `RunForIssue.failureCause`; `IssueRunLedger` shows "adapter failed
(40001)". Older runs keep "adapter failed". The Storybook run-ledger
fixture has a failed run with a cause.
- Restart chaos harness: run samples keep `failureCause`. U3 treats a
generic code with a specific `failureCause` as diagnosable. A bare
`Error`, `unknown_error`, or the no-code fallback
`adapter_reported_error` still counts as vague. U3 stays a warning.
- Docs: `doc/run-log-events.md` and the harness README describe the
field and the SQL replacement.

## Verification

- Red before the fix: the four new `run-completion.test.ts` cases failed
against `origin/master`'s `run-completion.ts` (no `failureCause`, SQL
text in `error`).
- Green:
- `server/src/services/heartbeat/run-completion.test.ts`: all tests
passed (real embedded PostgreSQL). New cases cover a nested `40001`
three levels deep for execution and setup failures, SQL and a secret
parameter absent from `error` and `resultJson`, the bound (an
81-character code and a SQL "code" fall back to the class name),
`exit_code_137` for a code-less adapter result, a cancelled result with
SQL text, no cause for a named code, and the log line. On this loaded
host the suite's `afterAll` database teardown hit its 30 s hook limit
after every test passed (the known local teardown limit, also seen in
#20385). CI runs the suite.
- `server/src/services/heartbeat/run-failure-cause.test.ts` (4),
`server/src/__tests__/activity-service.test.ts` (recorded, legacy,
malformed, overlong, boolean, and number cause projections),
`server/src/__tests__/issue-continuation-summary.test.ts` (legacy SQL
row).
- `packages/shared/src/run-failure-cause.test.ts` (6),
`ui/src/components/IssueRunLedger.test.tsx` (21), restart chaos
`universal.test.ts` and `observer.test.ts`.
-
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`:
207 passed (the stall-diagnostic cases still read `40001` and
`ECONNRESET`).
- Typecheck: shared, runner, server (all layers), UI, restart chaos
harness. `pnpm check:token-gates`: all gates clean.
- `pnpm verify` returned exit 2 (global doc input changed); required CI
supplies broad coverage.
- CI on `6295f8d` found that the new key made failed-preparation runs
ineligible for Retry (`chat-channels.integration.test.ts` saved
board-message Retry got 409;
`workspace-recovery-preparation-dispatch.test.ts` dispatched 0 Retries).
Both reproduced locally and pass after the allowlist fix, with the other
459 Retry cases in `chat-channels.integration.test.ts` and all 25 in
`workspace-recovery-preparation-dispatch.test.ts`.
- No chaos waves were run. Storybook was not opened in a browser; the
fixture typechecks and the label is covered by the UI test.

## Risks

- Low risk. `errorCode` values do not change, so retry, recovery, and
comparison logic is unaffected.
- No schema change and no migration. `failureCause` is a new optional
`resultJson` key. Rows saved by the previous release have no key and
render as before; a new row read by an old server is ignored.
- The SQL replacement only acts on a Drizzle `Failed query:` marker, or
on `$n` placeholder text with a statement shape. Ordinary error text
does not change. When it acts, it also drops the text after the
statement.
- Code that checks `resultJson` against a closed key list must allow
`failureCause`. I found and updated the four lists on the Retry path; a
list I missed would refuse a Retry for a run with a generic failure.
- This is the run log / instance database path. It adds no Telemetry
event.
- The server failure log line still logs the raw error message
(unchanged behavior); only the board-facing text is changed.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file editing) and code execution for
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:14:53 -06:00
DottaandClaude Opus 5.5 ef2b8d5324 fix(issues): make every Retry start work or say why (#20283)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a task's run fails, the board recovers it with a Retry button,
so that button must either start work or say why it did not
> - An audit of every Retry-type button found three gaps. The task Retry
shown during a scheduled retry's countdown returned "accepted" but
changed nothing. A stale Retry looked like success in the Inbox. "Retry
now" skipped the task's read and change checks, so a member who cannot
see a private task could start its retry
> - This pull request makes the task Retry start a scheduled retry
immediately, makes every Retry that did not start work say why, and
gives "Retry now" the same authorization as the task Retry
> - The benefit is that no Retry click silently does nothing

## Linked Issues or Issue Description

Refs #19831. The board reported a dead Retry button. A read-only audit
of every Retry button confirmed:
- **Task Retry during a scheduled retry was a no-op.**
`TaskInterruptionPresentation.canRetry` is true in `retry_scheduled`, so
`ExecutionBlockerNotice` shows Retry. `requestTaskRetry` treats
`scheduled_retry` as busy and inserts a `coalesced` wake pointing at
that run. `resumeTaskRetry` reads only `deferred_issue_execution`, and
dispatch uses coalesced user wakes only to carry comments. Nothing
promoted the run: the API answered 202 `accepted: true` and the
countdown did not change.
- **A stale Retry looked like success.** `accepted: false, stale: true`
comes back as HTTP 200. The Inbox ignored `accepted` and opened the task
as if work had started. The task page showed no reason.
- **"Retry now" checked less than the task Retry.** It ran only
`assertBoard` and `getAccessibleResource`, skipping
`assertIssueReadAllowed` and `issue:mutate`. A company member who cannot
read a private task could promote that task's scheduled retry.
- **"Retry now" said nothing for some outcomes.** It toasted only
`promoted` and `gate_suppressed`.

## What Changed

- `server/src/services/task-retry.ts`: when the Retry coalesces into the
task's latest run and that run is `scheduled_retry` (with no other Retry
already active), the receipt returns `scheduledRetryRunId`.
- `server/src/routes/issues.ts`:
- `POST /issues/:id/retry` then promotes that run through the existing
`retryScheduledRetryNow`, the same path "Retry now" uses. It logs
`issue.scheduled_retry_retry_now` with `source: "task_retry"` and
returns `scheduledRetry: { outcome, message }`.
- `POST /issues/:id/scheduled-retry/retry-now` now runs
`assertIssueReadAllowed` and `decideIssueAccess("issue:mutate")`,
matching `/retry`. A private task the member cannot read now answers
404.
- `ui/src/hooks/useTaskRetry.ts`: new `taskRetryNotStartedReason`. When
a Retry did not start work (stale, or a scheduled retry that is
gate-suppressed, waiting, or gone), the hook returns that reason as its
error. `ExecutionBlockerNotice` and `StalledDependencyNotice` already
display it.
- `ui/src/pages/Inbox.tsx`: a task Retry that did not start work shows
the reason in the error toast and does not navigate.
- `ui/src/hooks/useRetryNowMutation.ts` and
`IssueScheduledRetryCard.tsx`: every "Retry now" outcome now shows its
headline and message, including `no_scheduled_retry` on the card.
- `ui/src/api/issues.ts`: `retryTask`'s response type includes
`scheduledRetry`.

## Verification

Red/green (each new test fails against master and passes with this PR):
- `server/src/__tests__/issue-scheduled-retry-routes.test.ts`:
- "starts a scheduled retry now when the task Retry is clicked during
its countdown": 202 with `scheduledRetry.outcome: "promoted"`, the same
run becomes `queued`, and no second retry run is created.
- "refuses Retry now on a private task the member cannot see, like the
task Retry": on master "Retry now" answers 200 `promoted`; now both
routes answer 404 and the run stays `scheduled_retry`.
- The seed now adds the board user's membership, which the stricter
route needs. All 33 tests pass.
- `ui/src/hooks/useTaskRetry.test.tsx`: a replaced attempt, a gate-held
scheduled retry and a waiting scheduled retry each show their reason; a
promoted one shows nothing extra.
- `ui/src/pages/Inbox.test.tsx`: an Inbox Retry that did not start work
(stale, or gate-held) toasts the reason and does not navigate.
- `ui/src/components/IssueScheduledRetryCard.test.tsx`: "explains when
there was no scheduled retry left to start".

Other Retry buttons: the chat failed-run Retry already reports stopped
and failed retries (`agents.test.ts`). Agent lifecycle "Retry now" and
cost "Retry accounting" start no task work and already surface their API
errors.

Regression and static checks:
- UI: 7 files, 168 passed. Server: task-retry and scheduled-retry
suites, 149 passed. `issue-agent-mutation-ownership-routes.test.ts`
passed.
- `maintenance-server-routes.test.ts` "assembled maintenance server
production wiring" fails identically on master on this Mac
(`ensure-build-deps`).
- `tsc --noEmit` passes for `server` and `ui/src`. `pnpm
check:token-gates` is clean.

## Risks

- A task Retry during a scheduled retry now starts the run immediately,
through the existing "Retry now" promotion gates (pause, budget, cleanup
waits). A gate that holds it is reported, not bypassed.
- "Retry now" is stricter: a member without read or change access to the
task gets 404 or 403, as on `/retry`.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 10:25:21 -06:00
DottaandClaude Opus 5.5 140fcf98ed fix(retry): name a declined task Retry and let a new click replace it (#19426)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task interrupted by a restart or deploy must resume automatically,
or offer a Retry that works on first use
> - A board Retry saves an intent first. Dispatch then rechecks the
requester's authority and saved source before it starts a run
> - When that recheck refused the intent, dispatch returned a bare
"deferred". The intent stayed saved with no reason, and the task still
showed Retry
> - Each later click with a new key joined that same intent, so Retry
returned 202 and never started a run
> - This pull request names the refusal on the intent and lets a new
click replace an intent that can never start
> - The benefit is that a Retry button that is shown starts one run, or
explains why it cannot

## Linked Issues or Issue Description

**What happened?**
A native task in a managed sandbox lost its runner twice. Saved-work
recovery recovered the files. The task showed "Saved work recovered.
Retry or send a message to continue." and `canRetry: true`. A Retry
click returned 202 and logged `issue.retry_requested`, but no run
started. A second click with a new idempotency key returned the same
request ID and `state: failed`. No run started.

**Expected behavior**
If Retry is offered, a click starts exactly one run. If the Retry cannot
start, the task names the reason, and a later click is a fresh attempt.
A Retry accepted during the pre-deploy drain starts after the restart.

**Steps to reproduce**
1. Recover the files of a failed native run so the hold offers Retry.
2. Save a Retry intent whose dispatch-time authority recheck fails (for
example, the requester lost access).
3. Dispatch it: no run starts, and the intent has no wait reason.
4. Click Retry again with a new key: the response returns the first
request ID, and no run starts.

**Paperclip version or commit**
Reproduced on `b400914345` (staging served `4d49a31286`; the affected
code is the same).

Related: #19399 (automatic Retry after a lost runner), #19422.

## What Changed

- `heartbeat/queue.ts`: when dispatch declines a task Retry because the
requester can no longer change the task, or its saved board message
changed, record a named wait (`retry_authority_denied`,
`retry_source_changed`) and log it. Before, it returned "deferred" with
no record.
- `task-retry.ts`: a new click with a new key replaces an unadmitted
intent that dispatch declined for its requester's access
(`retry_authority_denied`). The old intent becomes `skipped` /
`task_retry_replaced`, and activity records `replacedRequestId`. Waits
on shared state (cleanup, recovery, an active run, preservation) still
coalesce. The restart hook never replaces a user's click. Chat-owned
intents keep their own owner.
- `task-interruption-presentation.ts`: fixed public messages for the two
new wait reasons. `retry_authority_denied` keeps Retry, because a new
click is a fresh attempt. `retry_source_changed` is terminal: an edited
saved message cannot be replayed, so Retry is hidden and the user sends
a new message.
- `doc/execution-semantics.md`: document the rule.

## Verification

- New regressions in `server/src/services/workspace-recovery.test.ts`
use the real recovery, Retry and dispatch services on embedded Postgres.
The fixture runs the automatic Retry from a lost runner, then that
Retry's own run loses its runner, and recovery saves its files:
- `starts exactly one run when Retry is pressed after the automatic
Retry's own run lost its runner` (also checks that a replayed dispatch
and a second click start no second run).
- `does not let a Retry that could not start swallow a later Retry`: red
before the fix in two steps. With no fix, the declined intent has no
wait reason. With only the wait fix, the second click returned the first
request ID. Green after the fix: one run, bound to the second click.
- `dispatches a Retry accepted during the pre-deploy drain once the
restarted server resumes`: green before and after. The drain does not
lose a saved Retry.
- `npx vitest run src/services/workspace-recovery.test.ts
src/services/task-retry.test.ts
src/services/task-interruption-presentation.test.ts
src/services/quarantined-task-retry.test.ts
src/services/explicit-native-continuation.test.ts`: 892 passed.
- Not verified on staging. After deploy, an intent that dispatch refuses
shows a named `retryWait` reason.

## Risks

- Low risk. Only intents that dispatch declined for their requester's
access are replaced. All other coalescing stays.
- A replaced intent's late dispatch finds it `skipped` and starts
nothing, so a double click still starts at most one run.
- The exact gate that held the staging intent is not confirmed from
records. If it was a shared-state wait, the new code names it only if it
is one of the two reasons above.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 03:52:21 -06:00
DottaandClaude Opus 5.5 9668e73b4b fix(native): continue a turn automatically when reattachment loses its runner (#19399)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in managed sandboxes recover through reattachment: after
a server restart, and for a same-run retry when a turn ends without a
semantic result
> - Reattachment fails closed when the original runner or sandbox is
gone (`native_remote_recovery_lease_mismatch`,
`runner_remote_process_identity_unavailable`)
> - Saved-work recovery then saves the files, but the task stays blocked
with "Saved work recovered. Send a message to continue." The recovered
hold offers no Retry
> - A restart, deploy or maintenance window must not need a user message
to continue work
> - This pull request requests the task Retry automatically, once, when
it is safe, and offers Retry on the recovered hold when it is not
> - The benefit is that interrupted work continues after infrastructure
events, and a user who must act gets a Retry that works on first use

## Linked Issues or Issue Description

**What happened?**
A native task in a managed sandbox was interrupted. In one case a server
restart claimed a queued user-comment turn for reattachment. In another
case a turn completed without a semantic result, and the same-run retry
claimed the run for reattachment with no restart. In both cases the
original runner or sandbox lease was gone. The run failed with
`native_remote_recovery_lease_mismatch` or
`runner_remote_process_identity_unavailable`. Recovery saved the work.
The task stayed `blocked` with "Saved work recovered. Send a message to
continue." The execution blocker had `canRetry: false`. A manual `POST
/api/issues/:id/retry` started a run that succeeded.

**Expected behavior**
After an infrastructure interruption, the task continues automatically
when the interrupted turn's outcome is known. Otherwise the task offers
a Retry that works on first use.

**Steps to reproduce**
1. Start a native task in a managed sandbox from a user comment.
2. Lose the sandbox lease or the remote runner process (for example,
restart the server during a maintenance window).
3. Let reattachment fail and saved-work recovery complete.
4. See that the task waits for a new message and offers no Retry on the
execution blocker.

**Root cause**
Reattachment (`reattach_remote_runner`) correctly refuses to replace a
missing runner or allocation. Nothing then continued the user's request.
The automatic continuation rule only allows a new user turn after
reconciliation. `getExecutionBlocker` hid Retry whenever a workspace
restore hold existed, also after the files were verifiably recovered.
The same-run retry path also re-enters the restart-recovery owner, so
its event said "after server restart" with no restart.

## What Changed

- New `server/src/services/heartbeat/restart-interrupted-turn.ts`: after
a reattachment claim executes, if the run failed because its runner or
allocation was lost, request the same task Retry that `POST
/issues/:id/retry` creates. It runs once per run (idempotency key
`restart-interrupted-turn:<runId>`), under the authority of the user who
sent the message, assigned the task, or clicked Retry. It requires that
the run recorded no provider turn or that its last provider turn is
settled. System wakes, in-flight turns, and a Retry that this hook
already requested do not continue automatically.
- `heartbeat/recovery.ts`: call the hook after each reattachment claim.
Label same-run retry reattachment as "Reattaching to the existing remote
runner to resume this run", and add `trigger: "same_run_retry"` to the
event payload.
- `execution-blocker.ts`: a failed run whose saved work is verifiably
recovered now offers Retry (`canRetry: true`) with "Saved work
recovered. Retry or send a message to continue." An intentional Stop
keeps "send a message" only.
- `task-retry.ts`: optional `initiatedBy: "restart_recovery"` records
the activity as actor `system`/`restart-recovery` with
`onBehalfOfUserId`.
- `doc/execution-semantics.md`: document the rule.

## Verification

- New regressions in `server/src/services/workspace-recovery.test.ts`
use the real recovery, Retry and dispatch services on embedded Postgres:
- `continues a user turn once ... after $title and recovery saved the
work` (3 cases: restart lease loss before the turn, same-run retry with
the runner gone after a settled turn, same-run retry with the lease gone
after a settled turn). Each asserts one queued successor bound to the
Retry and the user's authority after recovery, idempotent hook calls,
system activity attribution, and no second automatic attempt for the
automatic Retry's own failure.
- `does not continue automatically when $title, and offers a Retry that
works on first use` (in-flight turn, system wake, unrelated error). Each
asserts no successor, `canRetry: true` on the blocker and the
interruption presentation, and one successor after one Retry click.
- Red before the fix: all 6 tests fail (no successor, blocker `canRetry:
false`). Green after the fix.
- `npx vitest run src/services/workspace-recovery.test.ts
src/services/task-retry.test.ts
src/services/task-interruption-presentation.test.ts
src/services/execution-blocker-read.test.ts
src/services/heartbeat/recovery.test.ts
src/services/explicit-native-continuation.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/workspace-restore-repair-route.test.ts
src/services/native-runtime/native-failed-workspace-recovery.test.ts`:
1098 passed.
- Not verified on staging. After the next deploy, check a task
interrupted by a restart: it should continue without a message, or show
Retry.

## Risks

- The automatic Retry uses the original user's authority. Admission
rechecks that user's current permission, stop proof, saved-work
recovery, approvals, pauses, budget and ownership. A refused request
leaves the visible Retry.
- A settled turn that ended without a result is continued with a fresh
Retry turn from saved work. A turn whose outcome is unknown (in flight)
is not.
- The hook runs after the claim's execution returns. If the server stops
between the failure and the request, the task shows Retry instead.
- Not in scope: same-run disposition recovery still cannot start a new
runner in the existing sandbox after the original remote runner exits.
This change continues the task with one fresh Retry turn instead. If
that Retry turn also ends without a result and loses its runner, the
task shows Retry.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #19355

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:29:46 -06:00
Devin FoleyandPaperclip 46fb933b01 fix(recovery): keep the stop receipt when a file-preserving restart fails (#19181)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs can execute in a remote sandbox. When a run fails,
Paperclip stops the sandbox and records the provider's stop receipt on
the environment lease.
> - System workspace recovery can restart that stopped sandbox to save
its files. Before the restart, it removed the stop receipt from the
lease.
> - If the restart failed and the provider could not stop the sandbox
after that, the lease had no receipt. Continuation admission requires a
stop receipt on every remote lease of the failed run. So every Retry and
every comment waited for an environment that never stopped.
> - This pull request keeps the original receipt until the provider
confirms the restart. It restores the receipt when the restart fails and
the sandbox cannot stop. A restart that fails because the sandbox disk
is full now ends recovery with a fresh-workspace outcome.
> - The benefit is that a failed file-preserving restart no longer
blocks the task permanently. Saved input and Retry continue in a fresh
isolated workspace, and the original sandbox stays retained.

## Linked Issues or Issue Description

No public issue exists. This is a bug report in the format of
`bug_report.yml`.

**What happened?**

1. A native run failed because its Daytona sandbox filled its 10 GB
disk.
2. Paperclip stopped the sandbox normally and recorded the stop receipt
on the environment lease.
3. Some minutes later, system workspace recovery tried to restart the
sandbox to preserve its files. Before the restart,
`prepareStoppedSandboxForRecovery` deleted the stop receipt.
4. The restart failed on the full disk. The sandbox went into the
provider's `error` state.
5. The stop-only teardown after the failed restart also failed, because
the provider cannot stop a sandbox in that state. The lease stayed in
`pending_cleanup` with no receipt. The cleanup sweep retried the same
failing stop with backoff.
6. `admitExplicitNativeContinuation` requires a provider stop receipt on
every remote lease of the failed run. Every Retry and every comment was
deferred with "Waiting for the previous environment to stop. Your
message will start automatically." The message never started.

Also, after recovery was exhausted, a Retry replaced the exhausted
generation with a new one. That generation repeated the same restart,
and the Retry still reported success.

**Expected behavior**

A lease must not lose its stop receipt while the real sandbox state is
unknown or not running. When the file-preserving restart cannot succeed,
recovery must end in a state that lets the task continue, for example in
a fresh isolated workspace.

**Steps to reproduce**

1. Use a native run with a retained remote sandbox lease that has a
`stopped` receipt and a pending workspace recovery.
2. Make the provider resume fail (for example with `no space left on
device`), and make the provider stop fail.
3. Run `recoverRetainedWorkspace` until recovery is exhausted.
4. Save a user comment on the task and run the deferred-comment
admission.
5. Before this change, the lease is `pending_cleanup` without a receipt.
The comment stays deferred with reason `remote_cleanup`, and no
successor run starts.

**Paperclip version or commit**

Observed on a hosted staging build. Reproduced on `master` at
`6452e964a36` with the new tests in this PR.

**Deployment mode**

Hosted, with the Daytona sandbox provider plugin. The defect is in core
server recovery code, so it is not specific to one adapter.

## What Changed

- `server/src/services/sandbox-stop-and-retain.ts`
- `prepareStoppedSandboxForRecovery` still writes the stop-only intent
and moves the lease to `pending_cleanup` before the provider call. It
now keeps the prior `remoteExecutionTermination` (and
`sandboxStopAndRetainReceipt`, if present) as `priorStop` inside the
`sandboxStopAndRetain` intent.
- New `confirmSandboxRecoveryRestart` removes `priorStop` when the
provider resume returns. From that point, the sandbox may be running,
and only a fresh stop can settle the lease.
- New `restoreSandboxStopAfterFailedRestart` restores the prior receipts
and the `released` / `success` lease state. It works only while
`priorStop` is present, so only when no restart was confirmed. It checks
the restored receipt with `hasRemoteTerminationReceipt` first. It uses
the same attempt fence as `settleStopOnlyCleanup`, and it writes through
`writeEnvironmentCleanup` so dispatch and cleanup wake up.
- `settleStopOnlyCleanup` now shares that attempt fence
(`stopOnlyAttempt`). Its behavior does not change.
- `server/src/services/workspace-recovery.ts`
- After `resumeWorkspaceRecoveryLease` returns, recovery confirms the
restart before any other check.
- If the restart is not confirmed, recovery still tries the stop-only
teardown first. A successful stop records a fresh receipt, as before. If
the stop also fails, recovery restores the prior receipt instead of
leaving the lease without one.
- A later attempt that finds a pending stop-only lease (for example
after a process restart during the reopen) uses the same rule when its
stop fails.
- A resume error that reports a full sandbox disk (`No space left on
device` or `Disk quota exceeded`) becomes `source_unavailable` /
`use_alternative_strategy`. The generation is exhausted at once, and the
existing fresh-workspace continuation applies. Other resume errors keep
the existing bounded retry policy (`provider_unavailable`, at most six
attempts in 30 minutes). Because the receipt stays, that policy also
ends in a state that lets the task continue.
- `server/src/services/task-retry.ts`: Retry binds to an exhausted
generation with a `use_alternative_strategy` failure and no fresh
attempt, the same way it binds to a complete or running generation. It
does not create a new generation that repeats a step the policy already
rejected. The Retry continues in a fresh isolated workspace. An explicit
workspace recovery request (`requestWorkspaceRecovery`) still starts a
new generation.
- `doc/task-recovery.md`: describes the receipt handling around the
reopen and the Retry binding.
- `server/src/services/workspace-recovery.test.ts`: four new tests (five
cases) with the real database and mocked provider calls.

### Design choice

I kept the existing write-ahead fence. The lease still leaves `released`
and shows no receipt while the provider reopen is in progress. So no
admission, reuse or destructive cleanup can act in that window, and a
crash still leaves a stop-only obligation. The original receipt is not
deleted. It stays in the stop-only intent and is discarded only when the
provider confirms the restart. This makes one simple rule: the suspended
receipt exists only while no restart has been confirmed.

I did not use "restore only when the provider confirms that the sandbox
is not running". The Daytona plugin reports a sandbox in the `error`
state as `active` in `inspectLease`, and its `stopLease` cannot stop
that state. So with the current plugin, no provider confirmation exists
for this exact failure, and the hold would stay. A separate change to
the Daytona plugin's `stopLease` can recover an errored sandbox before
it stops it. This PR does not depend on that change. If that change
lands and its recovery succeeds, the existing stop attempt records a
fresh receipt, and the restore path is not used.

I also did not change continuation admission to accept a lease in
`pending_cleanup`. Many gates treat pending cleanup as unfinished work
(`hasPendingRunCleanup`, the remote stop check and others). A change
there would be wider and more sensitive.

The restored receipt is the provider's original receipt, bound to the
same company, run, lease, provider, allocation and plugin. No process ID
or run status takes its place.

## Verification

All commands ran from `server/` in a clean worktree of `master` on macOS
(Apple Silicon), with the embedded PostgreSQL test database. Vitest ran
from the real path.

- Baseline on `master` (`6452e964a36`), before any change: `npx vitest
run src/services/workspace-recovery.test.ts` → exit 0, 331 passed.
- The new tests fail on `master`. I reverted only the three source files
and kept the new tests:
- `npx vitest run src/services/workspace-recovery.test.ts -t
"file-preserving restart fails|without repeating a restart|keeps the
stop obligation once|interrupted restart that the provider"` → exit 1, 4
failed, 1 passed.
- In the comment cases, no successor run starts (`expected [] to have a
length of 1`). A temporary log (not committed) showed the lease in
`pending_cleanup` in both cases. In the `unknown_failure` case, the
comment was deferred with `remote_cleanup` / "Waiting for the previous
environment to stop. Your message will start automatically." This is the
same hold as in the incident.
- The guard test "keeps the stop obligation once the provider has
confirmed a restart" passes on `master`, as expected. It makes sure that
the fix does not restore a receipt after a confirmed restart.
- With this change, the same command → exit 0, 5 passed.
- `npx vitest run src/services/workspace-recovery.test.ts` on the
committed head → exit 0, 336 passed (331 existing and 5 new).
- Neighboring suites (`task-retry`, `explicit-native-continuation`,
`heartbeat/recovery`, `environment-runtime`, and the other server test
files that import the changed modules: `live-recovery-work`,
`retained-native-session-cleanup-work`, `execution-control-work`,
`legacy-workspace-restore-recovery`, `deferred-execution-receipt-work`,
`issue-agent-mutation-ownership-routes`,
`native-workspace-export-retry`,
`native-cancelled-workspace-preservation`,
`native-workspace-warm-reuse`, `native-workspace-reconstruction`,
`native-failed-workspace-recovery`, `heartbeat/workspace-preparation`) →
exit 0, 16 files, 1456 passed.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts
src/__tests__/slack-conversation-lifecycle.test.ts
src/services/native-runtime/preserved-native-response-wait.integration.test.ts`
(other importers of `task-retry`) → exit 0, 3 files, 2074 passed.
- `pnpm --filter @paperclipai/server typecheck` → exit 0.
- Not run: the full `pnpm -r typecheck`, `pnpm test:run` and `pnpm
build` repo checks. CI runs them. No live Daytona probe ran.

New tests:

- `lets saved input continue after a file-preserving restart fails on a
sandbox that cannot stop` (`disk_full`, `unknown_failure`): resume fails
and stop fails. The lease keeps its original receipt and is `released` /
`success`. Recovery is exhausted (after one attempt for a full disk, and
after six attempts for an unknown failure). The saved comment starts one
successor run with `fresh_isolated_workspace` readiness. The source
files stay in the retained sandbox.
- `continues a Retry in a fresh workspace without repeating a restart
that cannot succeed`: after a full-disk exhaustion, Retry keeps the same
generation and binds its intent. The restart is not called again, and
the Retry starts one run in a fresh isolated workspace.
- `keeps the stop obligation once the provider has confirmed a restart`:
resume returns, a later check fails, and the stop fails. The lease stays
`pending_cleanup` without a receipt, and the intent has no `priorStop`.
- `restores the stop receipt after an interrupted restart that the
provider never confirmed`: the lease starts in `pending_cleanup` with a
suspended receipt, as after a process restart during the reopen. The
stop fails, the receipt is restored, and the attempt continues to the
normal reopen.

Assumption and limits (runtime dependency rule): the provider calls are
mocked, and the database is real. The tests do not exercise the real
Daytona error text. The full-disk check matches the patterns that
Paperclip already uses for sandbox disk-full errors. If the provider
error text does not match, recovery uses the bounded retry path, which
now also ends in a fresh-workspace continuation. The change adds one
database write after each confirmed reopen and one after a failed reopen
and stop. It adds no polling.

## Risks

- A resume can fail after the provider started the sandbox, for example
when the reply is lost. If the stop-only teardown then also fails,
recovery restores a `stopped` receipt for a sandbox that can be running.
The old run's processes do not survive the earlier stop, so the receipt
is still correct as proof that the old execution ended. But the retained
sandbox can run until the provider stops it or a later recovery or
cleanup stops it. Before this change, the same case caused a permanent
hold.
- A full-disk restart failure now exhausts recovery at once. The task
continues in a fresh workspace without the failed run's files. The files
stay in the retained sandbox. An operator can still request workspace
recovery explicitly after the disk is freed.
- Retry no longer renews an exhausted generation that has a
`use_alternative_strategy` failure and no fresh attempt. For destroyed
or missing sources, the result is the same as before but without the
extra attempt.
- No schema or migration change. A suspended receipt is one extra field
in the existing `sandboxStopAndRetain` metadata. Older servers ignore
the field. The intent equality checks compare the stored value, so a
mixed-version deploy keeps working.
- #18820 also changes `server/src/services/workspace-recovery.test.ts`.
The two PRs touch different tests.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (model ID `claude-opus-5-5`), running as a
Claude Code agent with tool use (shell, file reads and edits, local test
and typecheck runs) and extended reasoning. The runtime does not expose
the context window size to the agent.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (affected suites; full repo
runs are left to CI, as noted above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 21:34:14 -08:00
53735db7de fix(retry): resume verified preparation failures (#19004)
## Thinking Path

> - Paperclip must preserve saved agent work while proving earlier
execution stopped.
> - A deliberate task Retry can observe a newer failure while an older
preparation still holds execution.
> - That older preparation can also have a finalized empty object log
despite never reaching the provider.
> - Binding only the latest run and requiring every log field to be
absent keeps valid recovery deferred.
> - This change follows the exact persisted predecessor chain and binds
current Retry authority to eligible holds.
> - It recognizes only an exact finalized empty-log receipt within the
existing strict preparation proof.
> - One fresh user turn can then start with saved work and the existing
ownership, stop and accounting checks.

## Linked Issues or Issue Description

Refs #18989, #18860 and #18997. The first waits for predecessor cleanup
during automatic dispatch. The second selects the committed native
session owner. The third retains admitted Retry authority when no source
recovery action exists. None passes current board Retry intent to a held
failed predecessor. A search of open repairs found no duplicate.
ROADMAP.md was checked; this is a recovery bug fix.

**What happened?**

A failed native preparation retains a legacy run row and an inherited
session. Its remote workspace is later saved and verified. An automatic
successor also fails during preparation. Board Retry observes that newer
run, but the effective hold belongs to its predecessor. The Retry does
not bind the predecessor's recovery generation. The native
historical-source guard then keeps the request deferred.

**Expected behavior**

Carry the deliberate Retry through its verified same-company, same-task
and same-agent persisted predecessor chain. Start one fresh user turn
after every source passes current execution, stop and saved-work checks.
Keep incomplete or unrelated proof held. Preserve prior work and
recorded action outcomes.

**Steps to reproduce**

1. Fail native preparation with an inherited session and a remote
workspace.
2. Finish its real workspace recovery and stop receipts.
3. Save a newer failed automatic successor with `retryOfRunId` bound to
that source.
4. Submit ordinary task Retry for the newer run.
5. Before the change, no successor is admitted. The request remains
deferred behind the older hold.

**Paperclip version or commit**

The regression fails on master
`039e0dad2c51c80251e1302a19c5566e039c357a`, which includes #18989.

**Deployment mode**

Source checkout on macOS arm64 with real PostgreSQL, HTTP task Retry and
the assembled heartbeat service. The remote filesystem boundary uses
temporary files. Hosted rollout and an original-task Retry are not yet
verified.

This PR incorporates the reviewed empty-log follow-up #19021, preserving
its commits and regression controls. Both parts are required for hosted
preparation recovery, so the combined revision is qualified and deployed
together.

## What Changed

- Accept a finalized zero-byte `s3_segments` log only when its reference
binds the exact company, agent and run, its digest matches empty
content, compression is false, and no output progress or nonempty
excerpt exists. Add a real-recorder HTTP regression and 14 negative
controls. Unknown or nonempty logs stay held.

- Read a complete persisted Retry chain in one recursive query, limited
to 32 runs. Reject missing, cyclic, newer, unfinished, cross-company,
cross-task and cross-agent sources. Transcript hints grant no authority.
- Bind eligible historical recovery records to the current Retry intent.
Keep complete or running recovery budgets. Leave successful ancestors
and mismatched recovery-source records unchanged.
- Recheck that same chain when selecting an ancestor's fresh-user
strategy.
- Admit only a held ancestor through the shared native preparation
proof. Keep the closed preparation failure codes, no-provider state,
profile, input, result, output and accounting checks. Treat inherited
session input as prior context and accept the exact bounded host start
marker. Verify current ownership, saved work and stop receipts before
forcing a fresh session.
- Add HTTP and PostgreSQL regressions and document the rule.
- Give the HTTP Retry nudge and explicit test worker the same real
heartbeat facade and provider boundary. Establish the revoked catalog
receipt before either can dispatch.

## Verification

- Combined integration head `91f021f233151192f69dd7a035d083abcc67c47c`
requires fresh complete CI and review; qualification is in progress.

- Repair validation: all **six** recovered preparation provider-boundary
cases pass. The catalog cases still require the exact access refusal,
zero provider inputs and zero provider turns. The success case still
checks preserved file bytes and one native turn.
- [Previous exact-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785508136)
failed the catalog case with `remote_workspace_transport_unsupported`.
The route created a default heartbeat that could race the injected
worker and allocate outside the controlled transport. The repaired test
shares its real heartbeat across both dispatchers and holds the agent
start lock until the revoked catalog receipt is committed. It tracks and
drains both Retry nudges during teardown. It changes no production
access check or expected refusal.
- The first broad local reproduction hit a full local disk. It passed
201 cases and failed 86, including explicit manifest free-space errors.
Generated build caches were removed from these isolated checkouts. The
owning run passed 670/671 cases. Its one failure was the saved-Files
loader refusing insufficient temporary disk space. After cleanup, that
unchanged test passes in isolation. The final shared-facade and
start-lock version passes all six dispatch cases. Both legacy
continuation suites pass all 29 cases. The final fixture teardown also
retires deferred input before removing recovery proof, so global queue
sweeps cannot revisit earlier cases. All seven dispatch and saved-Files
controls pass together after that change. No disk reserve, assertion or
timeout changed. The initial local typecheck also stopped on `ENOSPC`
while materializing provider assets. Server typecheck now passes after
cleanup. No test timeout or reserve was lowered.
- Current-head CI is complete: 52 successful checks, three expected
skips, and a successful security status. Both required gates (`ci / e2e`
and `ci / verify`) passed on `0adc2e38103533dd45bcc56f957ac019642c57b1`.
[Full CI
evidence](https://github.com/paperclipai/paperclip/actions/runs/41898305521).
Greptile scored final head `0adc2e38103533dd45bcc56f957ac019642c57b1`
5/5 with no blocking findings or unresolved threads. The earlier
verification below describes the previous head.

- RED on unchanged master `039e0dad2c`: the certified-preparation
sibling passes, while the inherited-session case admits zero successors
instead of one. A second isolated RED on latest base `8f06bd4e17`,
including #18997, uses the exact final three-node test fixture. The HTTP
route returns 202, then the test observes zero successors instead of
one. All baseline production files remain unchanged. The initial
dependency/setup failures are separate from these behavioral failures.
- The positive cases cover the failed-failed-successful chain. They keep
the completed result unchanged, verify the original saved file and
validate fresh-session continuation authority. Repeated HTTP input and
concurrent resume calls keep one successor.
- PostgreSQL controls accept a complete 32-run chain. A 33-run chain and
a newer-created predecessor return no usable ancestry and do not bind
the older recovery record.
- Safety controls cover company, task and agent scope, conflicting
canonical bindings, missing and cyclic edges, transcript hints, live
execution and controllers, unknown stop, failed saving, unsafe writer
state, Stop, actor and generation changes, independent holds and
mismatched recovery records.
- Ancestor controls retain provider-event, native-input,
session-checkpoint, controller-recovery, result, process, runtime,
accounting and unknown-result guards. Host start and warning prefixes
use exact origins, sequence and payloads. A fifth extra event is
rejected.
- The first complete owning run found 22 introduced admission failures.
The initial historical-native allowance also applied to exact-source
Retry. It admitted contradictory preparation evidence. The fix now
requires a canonical ancestor and the shared strict startup proof. The
original direct-source assertions remain unchanged.
- The next focused run passed 82 cases, including both catalogue-warning
controls. One new result fixture lacked the database's required owner
tuple. Two existing controls reached their original 30-second limit. The
fixture now uses the real native task and completion-contract owner
tuple. A controlled nine-case run passes the result case, both unchanged
timed-out controls, three failure-code controls and all three positives.
The earlier timeout causes remain undetermined. All failed-run logs are
retained.
- Complete owning suites passed on previous head
`137a8eb3febf41052e112fc3c4ffe1abb92bcab7`: all 675 tests across
workspace recovery, task Retry, explicit continuation and legacy
continuation. The command exits successfully with normal teardown. This
includes every earlier introduced admission failure and the two
unchanged timeout controls. Its base includes the merged #18997
authority repair and #18982 rolling-copy packaging repair. The rebase
retained both test imports at their one adjacent-import conflict.
- Node 24.19.0 and pnpm 9.15.4. Module boundaries and `git diff --check`
pass. The initial PostgreSQL start lacked package library links after
install with scripts disabled. The documented hydration restored the
isolated dependency; that setup failure was not a behavioral red.
- Previous-head [full
typecheck](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785506215)
and [full
build](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785506094)
passed. Earlier attempts received synchronized runner shutdown signals
and a lost-runner annotation. The shutdown initiator remains unknown. A
CI owner used the documented hosted-runner failover for attempt three
and restored the original routing value after capture. That attempt
exposed the catalog failure above. Source workflows and check
requirements remain unchanged. The current head now has green CI,
Greptile 5/5, no unresolved review threads, and no merge conflicts.
Hosted rollout and the original task are not yet verified.


- Combined recovery proof: 24 of 24 cases passed on this exact head in
43.13 seconds, with normal teardown and unchanged assertions and limits.
They cover the real recorder, all 14 empty-output rejection controls,
ancestor and direct Retry, saved messages, the configured native
provider, and verified stop. The targeted TypeScript graph also passed;
the preceding integration step passed 95 safety/provider/message cases
and all 49 task Retry cases.
- The incorporated #19021 was independently red/green with the real
recorder: unchanged predecessor code returned HTTP 202 but admitted zero
successors. Its 18 focused cases, all 690 owning recovery cases, full
typecheck/build, 1,904 local chat cases, complete CI and Greptile 5/5
passed before integration. The unchanged CLI startup lane passed one
justified retry; no hosted acceptance is inferred from those results.
- Additional local qualification of the unchanged workspace fixture
alongside the native bootstrap fixture passed 288 of 289 cases. One
manual-admission HTTP case ended with `socket hang up` after 451 ms,
without an assertion or HTTP status. That unchanged case passed once in
isolation at the original 40-second limit, with normal teardown, and
passed in exact `0adc` CI in 17.2 seconds. The earlier disconnect cause
remains unknown; no assertion or timeout changed.
- An earlier CI attempt failed the newly merged native ingress fixture
because its shared Node command failed strict permission qualification.
The current base includes #19020's owned executable fixture repair; no
production permission check was relaxed.
- Integration with current master had one documentation conflict. An
automatic test merge also duplicated an app declaration; that syntax
failure was retained and fixed by reusing the configured real HTTP app.
Both new message-state assertions remain intact. Workflows and lockfile
retain master's contents.

## Risks

- A damaged or overlong chain remains held. The helper does not truncate
it and grant partial authority.
- The new query runs only for a deliberate Retry or its ancestor
strategy check. It adds no background polling, table, credential or
provider policy.
- A fresh intent is not stop or file-recovery proof. The existing owners
retain those checks. Prior effects remain fenced, and no old result is
replayed.
- Local tests do not prove hosted rollout. Live acceptance must use the
served fixed revision and the task's original saved work.

## Model Used

OpenAI GPT-6 through Codex for the original change. OpenAI GPT-6.1 Sol
(`gpt-6.1-sol`), xhigh reasoning, for the merge and CI repair. Both used
repository tools and PostgreSQL verification. Context-window limits were
not exposed by the harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-11-04 14:12:58 -06:00
Devin FoleyandClaude Opus 5.5 925965a09b fix(task-retry): wait out brief chat authorization contention (#18895)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A user can click Retry on a task that is connected to an external
chat conversation.
> - That Retry prepares the chat retry inside one admission transaction.
The chat step takes a per-principal authorization lock with
`pg_try_advisory_xact_lock`.
> - When another chat transaction holds that lock for a short time, the
chat step throws `NativeChatReviewPresentationContentionError`. This
error is not an `HttpError`. The route returns HTTP 500, and the error
handler reports a server crash.
> - The contention is routine and short. The admission transaction does
not contact a provider, and it rolls back completely. The owning service
can safely try it again.
> - This pull request retries the admission a bounded number of times.
If the lock stays busy, the service returns a retryable 409 instead of a
500.
> - The benefit: the user's click usually succeeds without an error, and
Paperclip stops reporting expected lock contention as a crash.

## Linked Issues or Issue Description

**What happened?**

A hosted staging server reported
`NativeChatReviewPresentationContentionError: Chat response
authorization is temporarily busy; no provider delivery was attempted`
from `POST /api/issues/:id/retry`. The stack is `requestTaskRetry` →
`writeExecutionControl` → `prepareFailedChatRunRetry` →
`failedChatRetrySource`. The error came from the
`chat-identity:<company>:<principal>` advisory try-lock. The route
returned HTTP 500 for each occurrence.

**Expected behavior**

Short lock contention must not fail a deliberate Retry click. The
service must wait it out. If the lock stays busy, the service must
return a clear, retryable client error and must not report a server
crash.

**Steps to reproduce**

1. Bind a task to a restricted chat endpoint and conversation. Make the
latest task run a failed run.
2. In a different database transaction, hold
`pg_try_advisory_xact_lock(hashtextextended('chat-identity:<company>:<principal>',
0))`.
3. Call `POST /api/issues/:id/retry`. Before this change, the chat retry
step throws `NativeChatReviewPresentationContentionError` and the route
returns 500.

**Paperclip version or commit**

Reproduced on master `8771606fd87f69ed4057779e3dce3a9bca8088e6`. The
server release that reported it was
`d59791828482f213713ca55000c40e6c9c6c03e5`.

**Deployment mode**

Hosted staging, Node 24, PostgreSQL.

Related: #18845 and #18621 changed nearby Retry admission. They do not
handle this contention.

## What Changed

- `server/src/services/task-retry.ts`: `requestTaskRetry` now calls the
unchanged admission transaction (`admitTaskRetry`) in a bounded loop: 10
attempts, 100 ms apart. It retries only
`NativeChatReviewPresentationContentionError` and lock-not-available
contention (`isExternalChatWaitAuthorizationContention`, which matches
`55P03`). Every other error, including 403/404/409 decisions, propagates
unchanged on the first attempt.
- If the lock stays busy for all attempts, the service throws
`conflict(..., { code: "task_retry_busy", retryable: true })`. The UI's
existing 4xx handling resets the Retry button, and the error handler
does not report a 4xx as a crash.
- `server/src/services/task-retry.test.ts`: three PostgreSQL cases:
- Real advisory-lock contention. A second transaction holds the same
`chat-identity` lock that production checks, then releases it. Retry
succeeds and persists exactly one wake request. A replay with the same
idempotency key returns the same request.
- Persistent contention returns 409 `task_retry_busy` after exactly 10
attempts and persists no wake request.
  - A 403 from the chat step does not retry.

The transaction body is unchanged. It was moved into `admitTaskRetry`
without edits.

## Verification

- RED: with the old `task-retry.ts` and the new tests, the contention
case fails with the exact production error
(`NativeChatReviewPresentationContentionError: Chat response
authorization is temporarily busy…`). The persistent case fails because
the error is not an `HttpError`.
- GREEN: `pnpm exec vitest run src/services/task-retry.test.ts` passes
all 39 cases against embedded PostgreSQL.
- `pnpm --filter @paperclipai/server typecheck` passes (all layers,
including tests).
- `src/services/heartbeat/workspace-preparation.test.ts` (another
`requestTaskRetry` caller): 165 of 167 cases pass. The 2 failures
(`remote_workspace_migration_source_changed` in "resumes an imported
path/file-url Git source…") also fail on unchanged master in the same
environment, so this change does not cause them.
- Runtime assumption (rule `runtime-dependency-boundaries`): the retry
is safe only if a failed attempt leaves no persisted state. The
contention test uses a real PostgreSQL advisory lock held by a separate
connection and checks that only one wake row exists after the failed
attempts. Each attempt is a separate transaction, so the loop holds no
row lock or advisory lock while it waits.
- The full repository `pnpm test:run` and `pnpm build` were not run
locally. I rely on CI for those.

## Risks

- Low risk. The worst case adds about 1 second of request time when the
lock stays busy, and then returns 409 instead of 500.
- An attempt can fail after an earlier step in the same transaction (for
example, workspace preservation registration). The whole transaction
rolls back, so the next attempt repeats those steps from a clean state.
Those steps run in the same transaction and do not contact a provider.
- If `requestTaskRetry` is called inside an outer transaction, each
attempt rolls back only to its savepoint. Locks that the outer
transaction holds stay held. In that case the loop ends with the same
409 after its bounded budget.
- No schema, API shape, or UI change.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), through Claude Code with
tool use (shell, file editing, test execution) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; see
Verification for the two pre-existing failures)
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
behavior or command documentation changes are needed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 18:05:40 -08:00
DottaandPaperclip a80eec1164 fix(recovery): continue Retry after completed file preservation (#18845)
Reuse completed file preservation for task Retry and show bounded admission explanations, including newer saved messages and identifier routes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 16:01:27 -06:00
DottaandPaperclip 367f29478d fix(recovery): preserve the bound remote writer before continuation (#18803)
## Thinking Path

> - Paperclip manages agent work and preserves task files.
> - A remote workspace records the exact run that owns its unsaved
files.
> - A later setup failure can become the latest task run without owning
those files.
> - Retry only inspected that latest run, so the original writer had no
file-recovery owner.
> - A released reusable source also failed the legacy retention rule.
> - This change registers the bound writer, saves its files, and then
admits the saved user request once.
> - Existing stop, ownership, approval and execution-evidence checks
remain effective.

## Linked Issues or Issue Description

**What happened?**

A native run prepared a remote workspace and failed before finalization.
Its sandbox stopped with a dedicated stop receipt. The workspace stayed
dirty and retained that run as its writer. A later setup failure became
the latest task run. Retry inspected the newer run and left the original
file obligation unregistered. New attempts failed at the writer fence.

Registration alone was insufficient. After file recovery, continuation
still required the Retry source to be one of the older recovery sources.
A newer pre-provider failure did not satisfy that check. Workspace
selection could also choose a fresh directory instead of the recovered
files.

**Expected behavior**

Find the writer through the task's current workspace binding. Register
only its exact prepared source. Preserve the released lease and its stop
receipts. Save the files before continuation. Independently prove that a
newer setup failure never dispatched work. Select the verified saved
workspace and admit the Retry once.

**Steps to reproduce**

1. Fail a native run with a prepared remote workspace and a released
reusable sandbox.
2. Keep matching dedicated stop receipts and the workspace's dirty
writer authority.
3. Record a newer pre-provider setup failure for the same task.
4. Request Retry. Before the fix, no file-maintenance record owns the
actual writer.

**Paperclip version or commit**

Reproduced before production edits on master `f7ad1dc7ca`. The branch
was rebased on master `adb49a44346849e751548b2fb640693dbfad4b6f`.

**Deployment mode**

Authenticated hosted execution with Paperclip Runner and remote
workspace storage.

Companion: #18798 (now merged upstream) fixes a pre-existing inbox test
synchronization race. Its test-only commit
`110aa88489cad926a451c972ad731ede469347c1` is cherry-picked here with
attribution so this PR can run the complete CI matrix.

Related: #17800 registers missing preservation for the selected source
run. #18678 holds unfinished remote exports. #18718 repairs a prior
warm-controller projection failure. #17916 addresses a separate legacy
warm-copyback policy case.

## What Changed

- Discover the failed writer through the company-scoped task/workspace
binding during Retry and authenticated message admission.
- Reuse the exact prepared-source validator for released remote
workspaces. Registration changes neither the lease policy nor its stop
receipts. Physical recovery retains its full allocation checks.
- Check a newer failed preparation separately before allowing an older,
repaired source to release continuation. Provider events, leases,
results and execution ownership remain disqualifying evidence.
- Select the bound saved workspace through the continuation's verified
repair receipt.
- Add PostgreSQL safety cases and an HTTP Retry workflow with real file
capture and saved-copy reads. Document the execution contract.
- Reuse #18798's completion barrier for the existing inbox cancellation
test; preserve its eight-slot limit, rejection and coalescing
assertions.

## Verification

- RED before production edits: the exact stopped writer has no
`workspaceRestoreRecovery` after Retry. Six unsafe-source controls pass
on the baseline.
- GREEN: the expanded Retry/message/preview selection passes 17 cases.
All 627 tests in the five affected recovery suites pass on recovery
commit `b572304f772d236b8a8bdf095e13e651d808f41c`, including Stop,
ownership, saved messages and duplicate Retry.
- The HTTP workflow registers maintenance, keeps execution held before
save, reads back the saved file bytes, and admits exactly one
continuation with `resume_preserved_workspace`. A registration-only
repair first failed after save because it did not reconcile the newer
Retry source. The incident-shaped pair passes with an independent active
stranded-task action present.
- Greptile identified the path where the newer setup failure already
owns a reconciliation hold. Its new regression failed before the
correction (saved files, zero queued continuations); the
contradictory-dispatch control passed. All four HTTP variants pass after
the correction, covering both hold shapes and both dispatch-evidence
controls.
- Repository typecheck and build passed. Server typecheck and build also
passed after the recovery changes. Module-boundary and whitespace checks
pass.
- The full local Vitest command failed in the GitHub fast-response
acknowledgement test (`delivery_unknown`: one response write instead of
zero). That isolated case passes on this branch and unchanged master;
the original full-run cause remains unestablished. A sibling run passed
199 test cases but timed out in workspace-suite database cleanup.
Unchanged master also passed all 126 workspace cases and timed out in
the same cleanup hook; all five acknowledgement outcomes passed in that
baseline comparison. These local failures remain recorded rather than
being reported as suite passes.
- The first hosted matrix failed the pre-existing inbox cancellation
assertion (expected nine computes, saw eight). The same assertion failed
in [master
CI](https://github.com/paperclipai/paperclip/actions/runs/41664339150/job/126015526840)
and reproduced locally before applying #18798's existing repair. All six
inbox HTTP/database cases pass after that repair. Final head
`7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9` adds only that attributed
test change to the verified recovery code.
- [Final hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/41677310377)
passes on head `7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9`, including `ci
/ verify`, `ci / e2e`, typecheck, build, every server/chat/workspace
group, Runner checks and all eight browser shards. There are 54
successful check results, two informational neutral results and three
skipped results, with no pending or failing results.
- Greptile and the Paperclip reviewer both report 5/5 for that final
head; no review threads remain open. The Paperclip review check is
informational (`neutral`) and records its local-testing limitations.
- The file workflow uses real PostgreSQL, subprocess file capture,
encrypted saved objects and production recovery code. Provider
resume/stop calls are simulated at the environment driver boundary. This
does not prove a live Daytona or Codex recovery. No deployment or
original-task retry is claimed.

## Risks

- File-maintenance discovery must not authorize another executor.
Existing writer, coordinator and physical allocation checks still own
that decision.
- A false bootstrap summary cannot prove that a newer run never
executed. The new path also requires its immutable native-adapter claim,
unresolved preparation state, absent
coordinator/process/session/lease/result evidence, and no dispatch
events.
- Additional database reads run only during explicit task admission or
Retry strategy selection. They are company/task scoped and add no
polling loop. The saved-writer query returns one matching snapshot and
omits prompt bodies.
- No schema, credential, provider profile or deployment setting changes.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection, code
execution and test tools. The runtime does not expose the exact serving
model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 11:25:24 -06:00
DottaandPaperclip 0710c56842 fix: retry saved chat input after a successful run (#18621)
## Thinking Path

> - Paperclip manages AI agents and the work they receive.
> - Connected conversations save incoming requests before they can run.
> - A request can remain queued when the previous turn finishes
successfully.
> - Task Retry sent that case through the failed-run retry guard, which
rejected the successful run.
> - This change resumes the saved request through its existing admission
owner.
> - The operator can continue saved work while preserving the prior
result and current permissions.

## Linked Issues or Issue Description

**What happened?**

Task Retry returned a conflict for a restricted connected conversation
when its latest run succeeded and newer input remained saved. A GitHub
review can reach this state when a new PR head arrives during the
previous review. The prior review cannot publish against the new head,
but its completed turn is not a failed run that can be replayed.

**Expected behavior**

Retry should resume the saved request with its original source identity
and current authorization. It must preserve the previous successful
result and must not replay that turn.

**Steps to reproduce**

1. Start a review through a restricted GitHub endpoint.
2. Send a PR synchronize event while that review runs and retain its
deferred input receipt.
3. Let the original run finish successfully with the task still open.
4. Click Retry. Before this change, the failed-run guard rejects the
request. With this change, the saved input creates one successor through
normal admission.

Related work: #17725, #16965, #15033.

## What Changed

- Select the newest saved input by its canonical chat action and wake
receipt.
- Store its receipt ID in the task retry intent and resume it through
the existing chat owner.
- Preserve caller permissions, source authorization, task state, agent
state, cleanup waits, and duplicate-click handling.
- Exclude saved-input intents from generic board-message replay. Keep
board messages in separate queued receipts and allow independently saved
input after successful board turns.
- Add signed webhook, HTTP endpoint, accepted-result preservation, and
refusal tests. Update execution semantics and capability maps. Stabilize
the existing Teams lifecycle test with explicit database result
ordering.

## Verification

- Current-head [CI
passes](https://github.com/paperclipai/paperclip/actions/runs/41482727909):
54 successful checks, one neutral review, three expected skips, and a
successful Snyk status. Greptile gives commit `53f02e9105` 5/5 with no
unresolved review threads.
- Reproduced the original HTTP 409 before the fix. Reproduced both
review findings with three failing cases before fixing them.
- Passed 25 focused and related GitHub retry cases, 52 ordinary queue
tests, and three native startup cases against disposable PostgreSQL.
Earlier related verification also passed 83 GitHub retry cases and 109
recovery/receipt cases.
- Passed `pnpm -r typecheck`, `pnpm build`, `node
scripts/check-maps.mjs`, and the module boundary check. The final server
typecheck passed after the review fixes.
- The full local `pnpm test:run` attempt stopped on an existing Slack
queue deadline assertion with a 1 ms database/host clock difference.
That case passed in isolation. A second attempt was stopped when review
changes superseded its source. Full local completion is not claimed; all
chat, server, browser, build, and runner CI jobs passed on the final
commit.
- The runner CI job needed one rerun after a two-second startup timing
assertion failed. That case and its two related cases passed locally;
the rerun passed without code changes.
- Live GitHub review publication with this patch is not verified. The
signed provider fixture uses mocked GitHub responses. This branch is not
deployed.

## Risks

- This changes how Retry dispatches deferred external input after a
successful run. It relies on the existing inbound owner to recheck
source authority before execution.
- Tests cover paused connections and agents, changed source bindings,
completed tasks, pending cleanup, duplicate clicks, and preservation of
the accepted result.
- No schema migration or public API shape change is required.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code execution, and browser
tools. The exact deployed model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 11:46:46 -06:00
DottaandPaperclip 9ca9af544a Drive remaining live run and resource recovery from durable work (#18419)
Reviewed and verified at 6a4c193906.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 16:00:35 -05:00
DottaandPaperclip 3ee4c5b613 Make saved execution receipt recovery react to durable work (#18409)
Reviewed and verified at 8f306e0a6e.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 10:03:59 -05:00
DottaandPaperclip baf651c8d0 perf(heartbeat): drive retry promotion from durable deadlines (#18387)
Reviewed and verified at f1c48c924d.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 09:07:52 -05:00
4d0847c4ca fix(server): settle external conversations from committed work (#17772)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/6/changes/paperclip/6
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages AI agents and the tasks that they execute.
> - External conversations must become idle after an authorized response
is published and all related work is complete.
> - Native replay and stranded-task recovery still discover this
settlement work through recurring scans.
> - A new deadline worker must preserve response ordering, monitor
handoffs, and unfinished work.
> - This pull request gives settlement one shared readiness predicate
and activates its worker from committed changes.
> - Existing rows provide recovery truth. No new table or per-feature
timer is added.
> - The benefit is a settlement worker that becomes quiet and resumes
when relevant work changes.

## Linked Issues or Issue Description

Refs #17758. This is slice 56. Slice 55 has merged. Related: #17606
(chat milestones), #17663 (bounded run reads), #17665 (monitor proof),
and #13809 (answered Slack conversation lifecycle). The provider
applicability remains Slack and Speko-owned conversations. This does not
extend Telegram settlement from #16559.

**What happened?**

Conversation settlement depended on periodic native replay and generic
task recovery. An exhausted monitor also cleared its schedule before
recovery admission. During that gap, an answered conversation could look
finished. Failed or deferred admission lost the monitor schedule.

**Expected behavior**

Relevant committed changes activate settlement. The worker uses the
owning service's evidence and becomes quiet when no settlement is owed.
Monitor recovery stays pending until admission, and restart does not
admit an already accepted recovery twice.

**Steps to reproduce**

1. Create an active external conversation with a successful run and an
authorized published response.
2. Keep execution, approval, a goal, or a monitor pending. Let the
settlement worker become quiet.
3. Resolve that prerequisite through its owning service. Verify that the
conversation settles without another periodic sweep.
4. Delay or fail exhausted-monitor recovery admission. Verify that the
monitor remains pending and the conversation cannot settle during the
handoff.

**Paperclip version or commit**

Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`.
Parent #17758 merged as `8ac5a128e66fb672e88106412f76cf278599bfc4`. The
publication adopts that actual squash through a normal merge with an
unchanged complete child tree.

## What Changed

- Preserve quiet after coarse company-only maintenance hints. The shared
inbound-notice and response-availability helpers keep settlement enabled
by default. Only the milestone dispatcher suppresses their duplicate
settlement fanout. Its scoped issue-review signal and the explicit
company-authority signal still wake real work. Four database cases prove
both quiet and real company-wide producer delivery.
- Preserve source-generation checks for chat handoff,
task-before-comment locking, issue-scoped decision invalidation, and
current conversation event columns from master.
- Track initial external conversation settlement through shutdown
without blocking HTTP readiness on endpoint or task locks.

- Reuse the real queue's coalesced recovery receipt after restart,
without admitting the same monitor recovery again.

- Share the existing conversation readiness predicate between bounded
discovery and locked settlement. Preserve current provider policy,
run/publication proof, company boundaries, and endpoint-to-issue lock
order.
- Register a scoped deadline worker through the delivery coordinator.
Preserve startup, restore/worktree suppression, drain/resume, failure
recovery, and joined shutdown.
- Extend existing review, milestone, wake-notice, and finalization
signals. Add notifications at independent issue, comment, goal, session,
and wake-receipt unblocks.
- Keep monitor owner recovery pending until guarded admission. Reuse an
accepted receipt for the same monitor schedule after restart, even when
its limit reason changes; retain compatibility with legacy receipt keys.
Commit board escalation and its monitor clear together.
- Delegate periodic native settlement while preserving resource
callbacks. Keep direct/manual defaults and preserve generic recovery
ordering with a read-only readiness guard.
- Add the producer and lifecycle audit in
`doc/plans/2026-10-26-external-conversation-settlement-sleep.md` and
update the stack ledger.

## Verification

Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`.
Publication head: `b256bb52647affd9f5279ea8c40f48bbe40f031f`.
The complete accepted child tree stays unchanged when adopting merged
parent #17758. This publication adds only the audit document.

- **3,129 distinct cases in 37 complete server files pass**, plus **18
selected chat-response cases** (3,147 selected cases). The other 1,796
chat cases were not run locally. Full settlement, queue, scheduling,
run-state, goals, interactions, task Retry, decision targets, status
reassessment, workspace, process recovery, and review consumers are
covered.
- **Preserved failure:** the first 35-file command passed 2,445 cases
and timed out one restored-work handoff at its existing 15-second limit.
That aggregate command failed. The unchanged full 89-case actions file
then passed on a fresh fixture; the affected case took 22 ms. Root cause
was not established. No timeout or assertion changed. The 34 other files
passed originally. Distinct counts exclude duplicate actions passes.
Fresh CI remains a separate gate.
- **Indirect wake regression:** the original unconditional baseline
failed two quiet checks. Guards alone left the chat case failing with
three candidate queries. The final controlled indirect baseline fails
one of four cases; the repair passes all four with the same test hash.
Positive controls preserve real company-only producers.
- **Source and fixture controls:** all 38 selected test files restore to
their exact committed hashes after fixture imports. Fresh PostgreSQL 17
fixtures use 413 migrations. All five manual merge resolutions preserve
current authority, locking, and event-column contracts.
- **Static checks pass:** database build; server typecheck and build;
module boundaries; migration order; Copilot profile identity. Runner
packages equal the accepted parent byte for byte. No new Runner/package
qualification is claimed.
- [Current integration and wake
audit](https://github.com/paperclipai/paperclip/blob/b256bb52647affd9f5279ea8c40f48bbe40f031f/doc/plans/2026-10-26-external-conversation-settlement-sleep.md#current-master-integration-and-indirect-wake-audit-2026-10-29).
Latest-head CI and exact-head Greptile 5/5 must pass before merge.

<details><summary>Earlier revision verification and preserved failure
history</summary>

Receiver failure fairness: a controlled 1,100 ms batch reproduced a
30-second delay for healthy due receivers. The worker now supplies its
existing per-item deadline through `nextRunAtOnError`. All 20 receiver
lifecycle tests pass on the owner and combined tip
`31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production
typechecking passes on the owner. The failed receiver retains its own
backoff, and settled receivers return to zero recurring SQL. The
five-second assertion budget is unchanged. See [the owning
audit](https://github.com/paperclipai/paperclip/blob/83204a67880d980cfbea3771cee19a3cda0022b7/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch).
Current-head CI and review remain required.

Approved delegated Test-tab calls now evaluate caller and target
policies in the existing claim transaction. Real HTTP regressions first
showed a missing caller charge and shared-limit self-rejection. All 78
connection-management cases and 92 gateway cases now pass on the owning
branch, along with server production typechecking. All 78
connection-management cases also pass at combined tip
`d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover
shared charges, separate charges, rollback on denial, duplicate
approval, and rejection of a later authorized call. Intermediate
counter/identity fixture failures and a reused-database Slack collision
are recorded in [the owning
audit](https://github.com/paperclipai/paperclip/blob/ef9ae5767430d0093efc736ae0b303ba5afaae65/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes).
The earlier combined checks below remain evidence for their stated
revisions. Current-head CI and review remain required.

Inherited voice-cleanup verification: the controlled slow-provider
regression and all 13 lifecycle tests passed on its owner and combined
tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only
the fixture clock advancement and its audit note.

Current head: `83204a67880d980cfbea3771cee19a3cda0022b7`. This remains a
single-duty PR stacked on #17758.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-24-recurring-server-sleep-stack.md),
[2026-10-26-external-conversation-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-26-external-conversation-settlement-sleep.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.


</details>

## Risks

- Signals cover supported in-process writers. They do not supply change
capture for independent SQL or other replicas.
- Real company-wide changes can inspect active candidates. Coarse
maintenance hints do not activate settlement. Configured idle and
retained history must remain quiet.
- Monitor receipt binding and admission order must preserve company,
agent, issue, and exact schedule identity. Current parent handoff locks
and source-generation checks remain intact.
- The local restored-work timeout is retained above. A fresh full-file
pass does not establish its cause; latest-head CI must also pass.
- Generic recovery and Muse/resource duties remain separate slices. This
does not make the whole server idle. Existing Cloud sleep blockers
remain. No deployed sleep, database suspension, external alarm coverage,
fleet cost, or live provider qualification claim is made.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, PostgreSQL
tests, and GitHub tools. The exact runtime model identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 06:02:43 -05:00
DottaandPaperclip 208a093d19 fix(chat): preserve resumed review waits and retry newer saved input (#17725)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/36/changes/paperclip/36
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub review agents publish findings and wait for an authorized
reply or head change.
> - Review tools could retain the requester after resume while
completion lost that same wait proof.
> - A yielded report could then queue its own continuation or repeat a
missing-wait error.
> - Ordinary Retry could also select an older failed turn while a newer
saved request had no admission.
> - This change shares the existing authority proofs and canonical
admission owners.
> - Current access, publication, execution and budget gates remain
required.

## Linked Issues or Issue Description

Refs #17318. Builds on the canonical saved-input retry owner in #17733.
Related #16550 covers deferred legacy comments after ownership waits.
The separate recovery repair in #17724 owns the cleanup-to-saved-input
handoff; this PR does not add a second admission owner.

A resumed review could publish an assessment, then turn its
`response_wake` into an immediate `issue_status_changed` wake. A
provider that omitted its continuation could instead repeat a completion
error. A newer linked-human message saved during a budget hold could
also remain unstarted because Retry selected the older failed source.

## What Changed

- Resolve passive review waits through the accepted requester ledger
used by GitHub tools. Require the latest reviewed head, exact company,
task, agent, repository and processed publication under current access.
- Fill omitted/null yielded continuation from that server proof. Permit
an inherited normal tree resume only through its accepted parent
identity. Permit an ordinary follow-up only after this run has a
processed useful reply linked to its exact inbound message. Reject
acknowledgements, partial assessments, pending or uncertain writes as
that reply proof.
- Recheck the same run/review/head/source proof during feedback,
finalization and final status commit. Preserve explicit malformed-input
rejection and Done policy. Add no monitor permission or publication
grant.
- Keep heartbeat and publication projections narrow. Preserve both
published Copilot v62 identities and generate source-bound v63 from the
pinned merged base. No v63 live qualification is claimed.
- Certify the structured isolated-directory binding failure only at its
pre-allocation boundary and only through canonical task Retry. Other
workspace validation reasons and unknown effects remain denied.
- Route Retry to eligible newer saved input through the existing
canonical owner. Preserve the old result and skipped receipt. Require
current principal, source, App, runtime fence, budget, native ownership
and exact cleanup proof.
- Recover absent historical App metadata only through an earlier trusted
processed response receipt and an immutable non-null endpoint claim.
Explicit malformed or conflicting metadata stays denied.
- Revalidate execution through the exact claimed wake, coalesced task
Retry and resolved user-continuation receipt after admission. A second
normal Retry can recover the exact never-started authorization failure
with unchanged saved input. Require empty execution evidence, exact
diagnostics and no allocated or active executor.
- Explain when a recorded result prevents exact replay. Ordinary board
Retry directs a new task message to continue from saved work. Inherited
requester identity alone never grants replay.
- Keep accepted board-message preparation failures with the canonical
task Retry owner. Certify the exact unchanged comment, original user
wake, accepted same-user identity and absence of provider execution. Pin
that source, recheck it at admission, and bind identity initialization
to the coalesced intent and exact claimed successor receipt.
Uncertifiable sources retain an actionable fresh-message refusal.
- Return safe named authority categories for board Retry refusals.
Update execution documentation and capability maps. Add no schema or
polling loop.

## Verification

- A fresh external review request exposed a separate isolated-directory
preparation failure before allocation. Its first normal Retry saved an
intent but admitted no successor. This follow-up certifies only that
exact structured preparation boundary through the existing canonical
Retry owner, with trusted failed settlement and no executor, lease,
result or output. A real HTTP/queue regression reproduces the missing
successor before the repair. Twenty-three current-source safety cases
and the final 98-case Retry/HTTP group pass, including a fresh click
after a denied intent, source revocation, accepted results and the older
direct API remaining denied. The warning case uses actual configuration
preparation and production event append; unknown warning shape or
provider provenance stays denied. Full workspace typecheck, build, maps
and diff checks pass. Renewed hosted normal Retry now succeeds on the
final combined source: it preserves the exact saved request and linked
requester, starts one native successor in a fresh attempt workspace,
publishes the current-head 2/5 assessment and commits In review with
passive response wake.

- A later hosted normal Retry published a low-score assessment but lost
its accepted source identity because the previous attempt failed before
identity initialization. It repeated continuation errors, then submitted
Done. This is failed acceptance, not a successful qualification. The
initializer now retains the exact source only after execution-time
failed-chat Retry authorization; generic Retry and Done policy are
unchanged. A real queue/initializer regression fails with a null source
on the previous code and passes with the newer saved source. Two real
database completion cases verify that source yields the authoritative
wait and commits In review. Renewed hosted normal Retry now retains the
accepted source and exposes the authoritative current-review wait. It
completed after two semantic feedback corrections (wrong contract
revision, then wrong criterion IDs); the correct contract was present in
emitted input. The third finish was accepted and committed In review,
with no missing-wait loop or Done escalation.
- Repeated board-message preparation failures now walk their canonical
intent, failed dispatch and accepted parent identity, rechecking each
contained predecessor. A production queue regression fails the second
ordinary HTTP Retry before the repair and passes afterward. Changed
source, contradictory execution evidence and cyclic receipts remain
denied; traversal stops after twenty links.

- New inherited omission cases failed before the repair. The final full
PostgreSQL 18 wait/status suite passes **252 tests** under Node 24.22.
It covers omitted/null input, malformed-first recovery, same-head
resume, exact-source replies, changed heads, revoked access,
pending/uncertain/mixed publications, final commit revalidation, no
duplicate review and no self-wake.
- **15 existing publication tests passed before the pinned merge** on a
separate fresh database. They retain default partial-delivery and
failure-fallback behavior. Context reads remain below the unchanged 16
KiB heartbeat and publication response limits with large saved output
present.
- Saved-input tests cover a real signed PR webhook, production
accounting-budget refusal, normal policy recovery and canonical Retry.
Historical producer-shaped tests prove the missing-App case red, then
check trusted prior receipts and malformed, late, revoked or mismatched
proof. A broader order-coupled test was reproduced on the prior source;
its global fault injector now targets its own authorization query. The
43-case sequence passes without changing production behavior or its
safety assertions.
- New signed-ingress and real heartbeat execution tests reproduced the
admitted-successor failure, then the next normal Retry failure. The
final pinned merged source passes 249 full GitHub review/publication and
Retry cases on a fresh database, including all 49 saved-input cases. A
separate 74-case Retry/HTTP run also passes. Exact dispatch negatives
cover source, company, agent, task, successor, receipt, pause,
assignment, accepted success, controller and cleanup proof. Bootstrap
contradictions remain denied. The canonical Retry suite passes 16 cases
through its supported external PostgreSQL test connection.
- The normal board HTTP route reproduced the unusable inherited-result
refusal. It now returns actionable guidance without starting a
successor. The actual UI Retry hook and banner tests pass 12 cases.
- A production-shaped accepted board comment followed by a pre-provider
qualification failure reproduced HTTP 409 before the repair. The final
real HTTP/queue test certifies one canonical successor and preserves the
accepted identity. It rejects changed source, actor, task, agent,
result, controller, process, profile, allocation and receipt evidence. A
post-acceptance source change keeps the same intent pending. Execution
revalidation rejects removed or mismatched proof. The probe stops
deliberately after production identity initialization; it does not claim
provider execution. The final source passes the 75-case Retry group, the
focused two-case board/inherited-policy group and 19 canonical Retry
cases. Full workspace typecheck, build and map checks pass.
- Full workspace `pnpm -r typecheck` and `pnpm build` pass. Map,
module-boundary and diff checks pass. Full local `pnpm test:run` remains
unverified. The 256-case explicit-native suite cannot initialize its
packaged PostgreSQL because libzstd is absent; an existing
command-scoped fallback did not resolve it. Current-head remote
full-suite checks are required. Merged-source full GitHub+Retry (249),
separate Retry/HTTP (74), full wait/status (252), profile/history and
completion (75), and visible Retry hook/banner (12) checks pass. The
fresh owned PostgreSQL 18 databases applied all 377 migrations.
- Actual Rust Runner, OpenCode 1.18.34 and exact DeepSeek used unchanged
production-generated synthetic contexts. Resume finished with an
explicit server wait. The author-reply trial omitted continuation twice:
its first report failed the existing empty-criteria check; the corrected
second report still omitted continuation and received the exact server
wait. It completed once with no missing-wait loop or Done escalation.
Controller acknowledgments were synthetic; these probes do not establish
hosted publication or board state.
- Hosted combined testing proved automatic admission of the exact saved
author reply, a useful GitHub answer and committed passive wait. It
required two malformed-continuation rejections before a nullish fallback
succeeded. A later normal Pause/Resume completed in one finish with
explicit response_wake and no new review/comment. The admitted normal
Retry then exposed the execution guard gap repaired here. Those remain
checkpoint results. On the final combined source, renewed normal Retry
completed actual native provider work and GitHub publication, committed
the passive wait, released ownership and cleaned up its environment. A
subsequent ordinary unmentioned author reply woke once, made no GitHub
tool calls or new assessment, and completed its first finish with an
explicit response wake. Both exact leases expired with successful
cleanup and the queue was empty. However, final destination inspection
found a separate automatic fallback overwrote the existing summary with
a generic no-final-response message. External summary preservation
therefore remains failed acceptance; the separate publication repair in
#17726 owns that producer. Full combined publication preservation has
not passed. These hosted runs include the pinned-base native additions;
they do not qualify Copilot v63.
- Current-head automated review is 5/5 with zero unresolved threads. All
current-head CI gates passed (54 successful checks/contexts, four
skipped, none pending). The earlier login-cleanup idle-work assertion
also reproduced locally while its controlled baseline passed; its cause
remains unknown, but the exact assertion passed in the fresh
current-head CI shard. No timeout, assertion or production workaround
was added. This PR is ready for review within its source, authority,
native finalization, state, wake and cleanup scope. The independent
automatic fallback publication regression is tracked by #17726; no
complete combined workflow pass is claimed.

## Risks

- The latest saved review defines the durable head boundary. Ingress
must record a new head before admission. A reply receipt proves delivery
only for its exact saved source; a shared GitHub comment ID alone cannot
authorize this wait.
- Historical App recovery cannot infer identity without direct proof or
an earlier valid server response receipt. It never replays that
response.
- This does not infer review-score policy from instructions or override
Done. No deployment, merge or repair of historical tasks is claimed by
the PR.

## Model Used

OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, code
editing and test execution. The context-window size is not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused database and
native suites; full local suite limitation described above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (verified on immutable latest
head)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(verified on the latest head)
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 22:22:47 -05:00
DottaandPaperclip 8373d78679 fix(recovery): register missing native preservation on Retry (#17800)
## Thinking Path

> - Paperclip lets people manage agents and their saved work.
> - Native runs can leave a prepared remote workspace after a control
deadline.
> - The deadline repair in #17788 records preservation for future
failures.
> - Older failed runs can still have an active lease without a
preservation owner.
> - Retry correctly refuses another executor, but nothing can finish the
missing preservation.
> - This pull request registers the exact prepared source through the
existing recovery services when the user retries.
> - The bounded maintenance worker can then stop the provider and save
its files before normal admission continues.

## Linked Issues or Issue Description

**What happened?**

A historical failed native run has a prepared workspace and an active
retained allocation. It has no workspace preservation record. Ordinary
Retry cannot prove that the old executor stopped and has no maintenance
owner to produce that proof.

**Expected behavior**

A deliberate Retry registers missing preservation for the exact unowned
source. Existing recovery proves termination and saves the files.
Existing admission then checks current user authority, effects, pause
and budget gates before one successor can enter the queue.

**Steps to reproduce**

1. Persist a failed native run with a prepared workspace, an active
lease and a terminal, unowned finalization.
2. Omit the workspace preservation record, as an older deadline handler
did.
3. Call the normal task Retry route. Before this fix, preservation
remains absent and Retry cannot continue.

**Paperclip version or commit**

The regression fails on `46de15ff4b99bbee59c38f32f4f30b80ad5bbd24`,
which already includes #17788.

Related: Refs #17788 (future deadline failures), #17741 (committed
recovery wakes), #17733 (connected Retry authority). #17773 adds source
diagnostics and does not repair this missing registration.

## What Changed

- Use native workspace preservation inside the existing issue-locked
Retry transaction. A live finalization owner remains held.
- Adopt a valid unowned preservation obligation through the existing
bounded maintenance service. Bind it to the deliberate Retry intent
without resetting an active generation.
- Add real PostgreSQL regressions for exact, mismatched and owned
sources. Verify that registration cannot create a termination receipt or
a successor.
- Exercise the HTTP route, worker, real temporary file merge and queue
admission with the stop reply held. Duplicate requests admit one queued
turn after preservation.
- Extend connected GitHub and Telegram Retry coverage for missing
historical preservation. Document the contract and access boundaries.

## Verification

- Red: the new historical-state regression fails on the base commit
because `workspaceRestoreRecovery` remains absent. The mismatched and
owned negative controls pass.
- Green: the three focused historical-state cases pass with the fix
against isolated local PostgreSQL 17.
- The local HTTP journey restores saved files and retains newer host
edits. It admits no successor while the provider stop reply is held. It
admits one successor after verified preservation. The real GitHub and
Telegram admission paths wait behind that recovery.
- `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs` and
`git diff --check` pass.
- Green: 152 tests across four relevant suites, including all connected
Retry safety cases. The six new cases pass again on final head
`b8900801a3181fa673ea45f1129f968bbb08d1fd`.
- Final-head GitHub checks: 55 successful, two intentionally skipped
Storybook checks, and one neutral Paperclip bot review. Greptile is 5/5
with no unresolved threads. The Paperclip bot stopped before finishing
and supplied no approval.
- The full local command imported an earlier fixture revision before its
assertions were corrected. It reported those two stale connected-fixture
assertions and later hit the host PostgreSQL semaphore limit again. It
was stopped after final-head CI completed. It is not counted as a full
local pass; the complete CI test lanes passed on the final head.
- The first local rerun hit the Mac PostgreSQL semaphore limit before
assertions. Verification moved to an isolated disposable PostgreSQL
container. Early assembled fixtures also exposed the existing
destruction-cleanup gate; the retained-workspace fixture now uses its
actual `keep_running` disposition. This fix does not weaken cleanup.

## Risks

- Retry now registers maintenance for historical native failures. Exact
descriptor, allocation and coordinator checks remain owned by the
existing preservation service. Physical recovery still requires its
existing exclusive owner and provider stop receipt. Registration alone
grants no execution authority.
- No database migration or provider protocol changes. No model, trust,
assignment, retry-budget or external identity changes.
- Local recovery uses real database, filesystem, HTTP, worker and
admission code. Only the provider lifecycle boundary is simulated. No
hosted allocation or live model turn has been verified for this new
change.
- The workspace recovery test file crosses 1,000 lines because this case
shares its real database, filesystem and assembled route fixture.
Keeping it there avoids duplicating that fixture or extracting unrelated
production behavior.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools and code
execution. The exact model identifier and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 16:22:36 -05:00
DottaandPaperclip 0373f9f261 fix(recovery): wake retained workspace preservation after Retry (#17741)
## Thinking Path

> - Paperclip lets people manage AI agents and their saved work.
> - A stopped run can owe workspace preservation before saved input can
continue.
> - Retry saved a recovery request but did not wake the worker that owns
preservation.
> - Older recovery records could also take the bounded scan slots before
a due request.
> - A second Retry could change the revision while the physical owner
was still saving files.
> - This pull request wakes a separate worker after commit, selects due
work first, and keeps the active owner's revision.
> - Saved work can progress without replaying the stopped provider turn
or blocking other finalization.

## Linked Issues or Issue Description

**What happened?**

An operator approved a saved design and selected Retry after native
workspace finalization failed. The API accepted one canonical request.
The source remained scheduled at zero preservation attempts for more
than 30 minutes. No new provider turn started. A local regression also
found that repeated Retry during export invalidated the physical owner's
completion revision.

**Expected behavior**

Committed Retry must wake bounded source preservation. Due recovery must
progress ahead of passive cleanup obligations and unrelated history.
Duplicate requests must keep one physical attempt. Saved input must
still pass normal admission after verified recovery.

**Steps to reproduce**

1. Retain a terminal run with an exhausted workspace recovery and saved
authorized input.
2. Let the delivery coordinator become quiet, then request Retry.
3. Observe the recovery state and physical preservation receipt. Repeat
Retry while the export is held.

**Paperclip version or commit**

Observed on served commit `4b89eec644d6e1a99d58926e77ae58927c3fa87b`.
Local baseline: `a1f77821b9cd5c9c846fdf7a0fa5cc620ad2b6bb`.

Related work: Refs #17053, #17267, #17094. This change repairs explicit
Retry delivery. It is separate from the run instruction projection in
#17724 and historical unsafe-export discovery in #17687.

## What Changed

- Notify a dedicated workspace recovery duty from the transaction that
saves Retry, recovery admission, adoption, or recovery progress.
- Reconstruct work and saved retry deadlines at startup. Fill each free
export slot independently, track physical work for idle drain, and join
the dedicated worker's exports on shutdown.
- Drain one bounded pass through completed cleanup and saved-input
obligations. Restart that pass after committed changes without
repeatedly rechecking paused input.
- Share source eligibility between deadline selection and candidate
consumption. Keep resolved files-only obligations and ignore unrelated
resolved or cancelled history.
- Select due unfinished preservation before passive historical records.
Rotate past source locks held by other workers and retain the indexed
source lookup.
- Bind the canonical Retry intent without changing an active physical
attempt's generation or revision. Preserve that binding across later
progress writes.
- Add real producer, physical preservation, startup, deadline, idle, and
independent finalization regressions. Document the delivery contract.

## Verification

- Red controls: Retry without a notification stayed scheduled; duplicate
Retry invalidated the active export; whole-batch waiting stranded a
third Retry; a scan without cursor rotation starved a source behind two
actual advisory locks. Removing unfinished-first selection left HTTP
Retry scheduled behind five real recovered sources still owing cleanup.
The fixture explicitly verifies their eligibility with the worker's
actual query filter.
- Further red controls: the previous worker visits only two of three
eligible completed sources. Disabling the committed-change cursor reset
misses a new earlier source arriving while two callbacks are held. Both
pass with the final worker.
- Green: the public Retry HTTP route, registered production worker, real
file copyback, duplicate canonical receipts, and public recovery receipt
reload passed on an owned local fixture. Its agent is paused to keep
this files-only check from starting a provider turn.
- Full workspace recovery suite passed 56 tests, including the
foreign-lock regression.
- Execution-control lifecycle suite passed 24 tests. It covers idle SQL,
stored deadlines, cancelled and resolved history, independent
finalization, and shutdown. One, three, and seven paused completed
sources are each visited in a bounded pass, then stay deferred without a
successor or SQL during 90 seconds of fake-clock idle time. A committed
earlier source is evaluated when it arrives during two held callbacks.
- Both GitHub and Telegram `automatic_running` continuation regressions
passed with the new canonical intent binding.
- `pnpm -r typecheck` passed.
- `pnpm build` passed.
- `node scripts/check-maps.mjs` and `node
scripts/check-module-boundaries.mjs` passed.
- Local `pnpm test:run` could not complete: embedded PostgreSQL
repeatedly failed to start. The owned run was stopped after this setup
failure repeated. Latest-head CI supplies broad coverage.
- Earlier setup failures included a localhost sandbox denial and loss of
the owned disposable Docker database. These were separate from the
product red controls above.
- Local database tests use an explicitly named disposable PostgreSQL
`_test` database. That opt-in database is reset before the suite.
Default embedded test behavior is unchanged.
- Exact-head CI is green at `ef7dcf1ba1060864c9d216c33b51f4d9f8f57fbd`;
Greptile and Ponytail both rate it 5/5, with no unresolved review
threads. Earlier CI caught the indexed discovery assertion and two exact
chat intent-binding regressions; all three were fixed and pass locally
and in CI. A separate chat shard lost contact with its self-hosted
runner without a product assertion result. A later shared runner
shutdown interrupted several jobs; one infrastructure rerun passed
before the final test-and-worker follow-up triggered a fresh green run.
- No staging deployment is part of this PR. The original approval
remains saved; deployed recovery still needs acceptance after rollout.

## Risks

- More recovery writes now wake a bounded maintenance duty. Completed
history must not keep it active. Idle and deadline tests cover this
boundary.
- Resolved files-only obligations must remain eligible without granting
replay. The startup regression checks that path.
- Provider stop proof, copyback ownership, source binding, retry
budgets, and normal execution admission remain required. There is no
schema or public API change.

## Model Used

- OpenAI Codex based on GPT-6, with code execution and browser tools.
The exact serving model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 10:28:12 -05:00
DottaandPaperclip 338bf75761 fix(chat): admit task Retry after GitHub reconnect (#17733)
## Thinking Path

> - Paperclip manages agents and their tasks.
> - Task Retry must preserve saved input and enforce current access.
> - A GitHub reconnect invalidates old runtime callbacks.
> - Retry reused that old callback authority and rejected valid new
intent.
> - Retry also replayed an older failed request when a follow-up was
already saved.
> - This change admits a deliberate Retry under current authority and
transfers saved input once.

## Linked Issues or Issue Description

Refs #16965, #17053 and #17317. These repairs preserve work and contain
old executors. This change fixes the subsequent chat admission boundary.
A search found no overlapping open reconnect/admission fix.

**What happened?**

Ordinary task Retry returned `chat_failed_run_retry_not_authorized`
after the same GitHub App reconnected. The endpoint was healthy, but the
saved request still carried its old runtime generation. A newer queued
follow-up also made Retry reject the older failed request instead of
continuing the follow-up. No successor was admitted.

HTTP qualification found two related defects. Concurrent clicks could
omit the canonical chat action from a response. Signed-in board dispatch
also passed an incomplete task resource to the permission resolver,
which rejected a permitted mutation.

**Expected behavior**

A deliberate Retry checks current authority for the same App, sender,
repository, conversation and task. It continues saved input once. A
later reconnect, changed App, revoked access or changed source prevents
dispatch. Old runtime callbacks remain invalid.

**Steps to reproduce**

1. Admit a GitHub request and record a failed native run.
2. Save a follow-up while execution is blocked.
3. Reconnect the same GitHub App.
4. Click ordinary task Retry.
5. Before this change, admission returns 409. Without reconnect, the
newer input alone also causes 409.

**Paperclip version or commit**

Baseline source: `41c42895fa862c586ff841f8c50200d93ffc83a4`.

**Deployment mode**

Source checkout on macOS. Regressions use real PostgreSQL transactions
and the assembled chat, HTTP route and heartbeat admission services.
Provider boundaries are simulated. No hosted recovery is claimed.

## What Changed

- Record the admitting App identity for new GitHub deliveries.
Historical deliveries can use their original working-comment binding.
Reauthorize deliberate Retry under a pinned current generation and
credential-reference fingerprint.
- Select the complete saved-input queue from the failed run's actual
conversation before replaying the older failed batch. Keep exact-run
Retry bounded while ordinary task Retry supports larger queues. Transfer
its receipts with the dispatch transaction. Recover the same ownership
after restart or a lost response. Release unadmitted input after a
definitive rejection.
- Preserve canonical chat action IDs across concurrent clicks. Supply
current task status and assignment fields to board mutation checks.
- Keep exact-run Retry, stale webhook, containment, source integrity,
publication, pause and budget checks. Update recovery contracts and
verification recipes.

## Verification

- Red: all three new PostgreSQL regressions failed on unchanged
production code. Reconnect failed at chat authorization. Newer input
failed at the obsolete-source guard. The combined case also failed.
- Green: 61 selected chat integration cases passed. These include 27 new
cases and 34 existing cases for containment, retained-source recovery,
exact batches and source revocation.
- A further red/green PostgreSQL regression reproduces the 51-message
overflow identified during review. All 51 messages reach one successor,
including a later Retry of that successor. A separate 52-message
consumed-batch regression also passes.
- A competing conversation retains its own deferred message. Distinct
automatic policy contexts are checked independently; revoking only the
later policy blocks the whole unadmitted batch. Batch validation still
rejects a later message with a changed App binding or deleted source. A
real PostgreSQL query capture reduced the 51-message preparation from
1,103 calls to fewer than 180.
- All 16 durable task Retry tests passed.
- The real HTTP route returns 202 and admits one queued successor for
concurrent deliveries. Fault injection covers rollback during input
handoff, response loss after dispatch commit, and permission loss when
resuming an existing durable wait. A wait only adopts input after
current admission succeeds; a one-shot settlement marker keeps status
projection read-only. A new deliberate Retry works after another
reconnect or a failed successor.
- Current-App, repository, sender, source edit/delete, reassignment,
endpoint pause and board-permission controls passed. Historical
deliveries remain unchanged.
- Final-head CI: 55 checks passed, three expected skips. The unchanged
minute-bucket rate-limit assertion passed locally against PostgreSQL and
on its one CI rerun. Greptile reviewed the final commit at 5/5 with no
unresolved threads.
- Workspace typecheck and full build passed. Map alignment, module
boundaries, UI token gates and `git diff --check` passed.
- Hosted task recovery and a live provider turn remain unverified. The
local tests hold agent capacity to inspect admission without starting a
model or publishing a GitHub response.

## Risks

- A deliberate GitHub Retry can admit old saved input under current
connection authority. It requires the same App binding and current
access. It pins that admission; it does not rewrite the historical
delivery or make stale callbacks valid.
- The durable chat queue has an optional admission callback. It
transfers input only when the transaction contains a valid queued
execution or saved wait. Other chat consumers have no callback and
retain their behavior.
- No migration, Cloud dependency, deployment, configuration change or
workspace cleanup is included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, source inspection, code
execution and GitHub tools. The exact serving variant and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 10:26:45 -05:00
DottaandPaperclip 3f1ad6b25b fix(chat): resume task Retry after retained-source containment (#17317)
## Thinking Path

> - Paperclip manages agent work through task and execution ownership.
> - A task Retry must retain the user's request while the system
verifies an old executor.
> - Connected conversations also retain their external principal and
accepted source batch.
> - Chat Retry admission required containment before it could save a
wait for that recovery.
> - A saved chat receipt could also remain deferred after containment
became available.
> - This pull request lets the exact recovery-bound request wait and
resume through its chat owner.
> - Normal admission still checks containment, current access and policy
before one successor starts.

## Linked Issues or Issue Description

**What happened?**

An ordinary task Retry on a connected conversation failed with
`failed_chat_retry_not_authorized` while its retained remote executor
still needed verification. The new recovery generation existed, but the
chat action failed before it could save a dispatch wait. This occurred
after the repairs in #17267 were confirmed live.

The local regression found a second gap. A processed chat action with a
deferred wake was not reconsidered after containment. Re-entering normal
admission also tried to insert its existing durable receipt again. A
historical hold for the same failed run could also hide the current
Retry's workspace strategy, so the request still waited after the old
executor was contained. Preparation could then read that sibling instead
of the recovery record that owned the fresh successor, losing the
fresh-workspace decision.

**Expected behavior**

Save the exact authorized Retry while its matching recovery generation
verifies the executor. Once containment is recorded, reconsider the same
chat receipt through normal admission. Start one successor with the
original source batch. Keep publication, principal, pause and budget
checks.

**Steps to reproduce**

1. Admit a GitHub or Telegram request and record a failed native run
with a retained remote lease.
2. Leave the executor's termination unconfirmed and exhaust its prior
recovery generation.
3. Use ordinary task Retry to create a new recovery generation.
4. Process the connected Retry. The old admission code fails instead of
retaining a wait.
5. With a deferred receipt, record exact containment and reconcile
again. The old dispatch path leaves it stranded.

**Paperclip version or commit**

Reproduced from `412914cfa78d5abefcc34d3eb8d4bb86e461caac`. The branch
is rebased onto current master.

**Deployment mode**

Hosted tenant for the reported failure. Isolated local PostgreSQL and
the assembled heartbeat/chat services for the regression.

Refs #17267 and #16965. This change does not add a Cloud dependency or
change the absence protocol.

## What Changed

- Permit a saved admission wait only for the exact current task Retry
and active source recovery generation.
- Keep promotion and execution dependent on independent containment
proof.
- Bind a Retry to active automatic recovery without replacing its
generation, attempt count or deadline.
- Share the current exact Retry strategy across historical holds for the
same run. Retain each hold's stop and effect checks.
- Select the canonical recovery record that proves the committed fresh
successor during workspace preparation.
- Let task receipt reconciliation return the same deferred action to its
chat owner.
- Recheck current board and external authority, then update the existing
wake atomically with its successor.
- Preserve the exact chat source batch. Keep generic board-comment
filtering unchanged.
- Add GitHub and Telegram cases for waiting, containment, one successor,
stale generations, source edits, endpoint pause and supersession.
- Update recovery documentation and capability maps.

## Verification

- Red: real PostgreSQL tests reproduced the admission rejection for
GitHub and Telegram. Six active-recovery cases then failed on the
generation-ID assumption. Two same-run historical-hold cases remained
deferred after containment on the old strategy lookup. A preparation
regression then returned no readiness when the historical hold was
stored first.
- Green: 73 selected chat integration tests and 394 continuation, task
Retry, workspace recovery, recovery-state and workspace-preparation
tests passed. The chat tests use the real chat admission and heartbeat
queue. Unrelated capacity stays occupied so no real provider starts.
They assert no executor before containment, one successor afterward,
unchanged source leases while waiting, preserved recovery budgets, and
denial after authority changes. A separate source hold still blocks
work. Both sibling insertion orders pass through the committed-readiness
lookup and real workspace preparation: the new directory has independent
Git storage and files, and unpublished old work remains intact.
- `pnpm -r typecheck`, `node scripts/check-maps.mjs` and `git diff
--check` passed after the review fixes. The source build passed on
`f6660efa48e37a80cd9857217a537606de85b5c8`.
- The local host reached its PostgreSQL semaphore limit. Successful runs
used a test-only loader to reduce disposable cluster connection and
worker limits. No repository database helper or application setting
changed.
- The full local `pnpm test:run` began before the final review edits,
failed in its first server phase and was stopped after those failures.
Later phases did not run. This is not a full local pass. The transient
Retry-notice case passes after fixture isolation; the four Discord
close-owned publication cases also pass in isolation. Other broad-run
failure causes are not established.
- Exact-head CI and security checks pass on
`f6660efa48e37a80cd9857217a537606de85b5c8`; Greptile is 5/5 with zero
unresolved review threads. Optional Paperclip Review completed neutral
with a 5/5 static assessment of all 11 changed files and no findings; it
did not run tests or hosted acceptance. All 57 current-head checks are
terminal: 54 success, two skipped and the optional neutral review, with
no failures or pending checks.
- The real hosted workflow confirmed that the deployed absence repair
recorded containment. A later ordinary UI Retry was accepted and saved a
durable request, but it remains behind the historical source hold. The
useful existing review remains unchanged. This follow-up is not
deployed; a live successful successor and final task disposition remain
unverified. No missing files are claimed recovered.

## Risks

- The existing task reconciliation sweep can restage a deferred chat
action. This reuses its receipt and generation; it does not grant new
execution or publication authority.
- Dispatch must keep the board Retry identity separate from the
connected request's principal. The regression checks these bindings and
preserves current chat authorization at each phase.
- Historical holds reuse a strategy only from the locked same-run
recovery that names the current Retry intent. A different run, malformed
recovery record or independent source hold remains fenced.
- Unknown executors without a matching active recovery still fail
admission. Promotion and execution require containment even when
admission saved a wait.
- No schema or API shape changes. No credential changes, manual
deployment or service restart.

## Model Used

OpenAI GPT-6 through Codex, with repository tools, code execution and
browser verification. The exact served model variant and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 07:51:34 -07:00
DottaandPaperclip d33b40a5c3 fix: recover saved work and provide reliable task Retry (#16965)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The control plane must retain work when an execution or workspace
export fails.
> - Retained-workspace holds can leave saved input waiting with no
usable Retry action.
> - Users cannot verify process ownership or repair provider
allocations.
> - This pull request gives system recovery a durable budget and gives
each deliberate Retry a durable intent.
> - Safe recovery can preserve files, settle an accepted result, or
prepare an independent workspace.
> - One task state keeps infrastructure diagnostics out of the ordinary
conversation while preserving real permission and approval gates.

## Linked Issues or Issue Description

**What happened?**

A task with a retained workspace can show recovery, failed-run and
saved-message warnings at the same time. Historical runs without native
workspace descriptors cannot use the native restore path. Old recovery
metadata can prevent a new attempt even when an independent workspace
would be safe.

**Expected behavior**

Paperclip owns bounded infrastructure retries and file preservation. A
user can request another attempt with ordinary Retry. Thinking appears
immediately. The service preserves completed effects and current user
controls. It does not claim that starting fresh recovered the old files.

**Steps to reproduce**

1. Retain a failed run and its workspace after export or process-stop
failure.
2. Save a follow-up while the recovery hold is active.
3. Open the task and try to continue without infrastructure credentials
or operator certification.

Refs #16906, #16835, #16883. This extends their recovery and receipt
contracts. Related work: #16962 and #16963.

## What Changed

- Add task-level Retry with company-scoped authorization, all-status
idempotency and durable admission before physical work. Keep exact-run
Retry compatible.
- Add typed recovery phases, finite persisted backoff, historical
adoption and durable local failure diagnostics.
- Preserve native workspaces or verified historical snapshots. Admit an
independent fresh workspace only after execution containment. Keep
old-source preservation and cleanup obligations separate.
- Carry completed publication and API receipts forward. Fence uncertain
mutation retries across replacement runs. Preserve connected-chat
identity and authorization.
- Re-enter normal dispatch for saved input. Reuse accepted results
through the existing finalizer.
- Use one task interruption projection across the thread, composer and
task list. Show optimistic Thinking, ordinary Retry and genuine user
gates.
- Add PostgreSQL, filesystem, API, browser-fixture and controlled
provider coverage. Document the behavior and update the feature/access
maps.

## Verification

- All GitHub Actions CI jobs pass on
`6d8c6d27d50586d8548b31d03b47f6d73a0ee11d`: typecheck, all
server/chat/workspace and serialized suites, Runner checks, eight
browser shards, canary and build. The required merge gate passes.
- Greptile reviewed the final commit at 5/5, with no actionable findings
or unresolved review threads. Its two earlier P1 findings are fixed.
- Full recursive typecheck and build pass locally on the final commit.
All 16 task Retry cases pass against isolated real PostgreSQL. The
unrelated-success case and both stale Retry cases fail on the previous
code and pass after the fix.
- The corrected gateway fixture passes its uncertain-mutation and
independent-read case. Four real HTTP API cases pass for definite
403/422 rejection, completed-effect reuse, and uncertain-effect fences
across deliberate and automatic retries. These focused gateway/API
checks used a temporary database lifecycle harness on an existing
healthy local PostgreSQL process. The standard repository helper is
unchanged.
- Earlier qualification covers 291 task UI cases, saved-file copyback
and ancestry, current Agent Chat ownership races, nine chat Retry API
browser cases, and six assembled browser recovery fixtures. A real
API/PostgreSQL/native-runner journey showed Thinking in 25 ms despite a
delayed response, admitted one successor in a separate workspace,
retained original files, and survived reload.
- The controlled Daytona test interrupts export, joins the controller,
restarts it in a separate process, and verifies one agent start, one
external sentinel operation, retained file contents and no premature
cleanup. It does not establish abrupt-kill or live LLM/GitHub
acceptance.
- The full local `pnpm test:run` was attempted and was not green: an
OAuth row-lock timing assertion failed, and a later suite exhausted
macOS PostgreSQL semaphores. The ten OAuth timing cases passed on rerun
against a separate database. Later embedded database initialization
failures remain recorded; the final CI matrix passes.
- Security scanning passed on production-code commit
`63953edf069b111c6becc3144dffa6deb52f68b5`. Its only successor changes
one test-fixture identity line. The final scan is neutral/inconclusive
because its pinned merge-base checkout failed or timed out. Both rerun
API requests returned 404, so no fresh successful scan is claimed.
Socket and Snyk checks pass on the final head.
- The optional Paperclip Review remains queued behind an earlier run in
workspace finalization. Its prior completed assessment was incomplete
because a comparison tool was blocked by the platform filter. It is not
counted as a completed code review. GitHub does not require this check
for merge.
- Hosted acceptance remains pending deployment. Local and CI results do
not establish recovery of a hosted task or its retained files.

## Risks

- Recovery changes span admission, physical ownership and dispatch.
Incorrect evidence could permit overlapping work. Existing physical
locks and authoritative stop requirements remain in place.
- The migration adds concurrent indexes. It requires the existing
top-level migration runner and retries incomplete index creation safely.
- A fresh attempt can complete while old-source preservation remains
unresolved. The service retains that obligation and does not report
those files as recovered.
- Unknown tool mutations remain blocked until their authoritative
receipt owner resolves them. Generic upstream tools have no universal
result-query protocol.
- Recovery and diagnostics stay inside the stack. No Cloud incident
endpoint, notification worker or new outbound reporting channel is
included.

## Model Used

OpenAI Codex, GPT-6 family. The runtime does not expose a more specific
model version or context-window size. The model used reasoning, shell
execution, source editing and browser tools. No model subagents were
used for this implementation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; the full
local run limitation is documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 19:58:53 -05:00