Commit Graph
70 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 57b787d3d2 fix(retry): continue from recorded work when saved files cannot be recovered (#21363)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run finishes but copying its files out of the sandbox fails,
Paperclip holds the task and tries to recover the saved files. When
recovery gives up, the task offers Retry.
> - Some saved files can never be recovered: the sandbox that held them
was destroyed, or Paperclip cannot prove which runs used a shared
sandbox. Each Retry ran the same check again, failed again, and parked
the Retry in a wait that never ends.
> - The task told the person to press Retry again or to wait for a
Paperclip update. That breaks the rule that recovery must work and that
a Retry that is shown must work.
> - This pull request makes Retry continue the task in a fresh session
and a new workspace from the work that is recorded, when the saved files
are provably unrecoverable or cannot be proven safe. The task says
clearly that the unsaved files were not recovered.
> - The benefit is that an offered Retry starts a run, and nobody waits
for a fix that another click cannot bring.

## Linked Issues or Issue Description

Refs #21315, #21295, #21267

**What happened?**

On hosted staging, two blocked tasks offered Retry (`canRetry: true`).
Each Retry was accepted, and within about 3 seconds it was parked in a
wait. No run started. Every later Retry did the same.

- Task A: the hold cause was
`native_workspace_sync_out_retry_exhausted`. Recovery ended with
`source_unavailable` (`use_alternative_strategy`). The sandbox that held
the files had a signed absence proof and a `destroyed` termination
receipt. Retry was parked with `saved_work_recovery_failed`: "Saving the
agent's files failed. Retry to try again".
- Task B: the same hold cause. Recovery ended with
`allocation_ownership_ambiguous` (`service_required`), because the
sandbox was a reusable sandbox that other runs also used. Each Retry
opened a new recovery generation, ran the same check, and was parked
with `saved_work_recovery_refused`: "Retry runs the same check again and
succeeds only after Paperclip is updated."
- In both tasks, the agent's turn had finished and its result was
accepted. Its work was recorded outside the sandbox: a plan document and
screenshot attachments for task A, and a pushed branch with an open pull
request and an implementation document for task B.

Root cause: admission had a branch for an accepted result whose files
were not yet saved. When the recovery budget was spent, the branch
always returned a `saved_work_recovery_*` wait. It did not check whether
the files could ever be recovered, so the fresh-workspace path that
admission already has was never reached. The automatic retained-source
fallback did not help either, because it handles only the remote
workspace v1 reference, and these runs use the native workspace sync v3
reference. Also, Retry opened a new recovery generation for an ownership
refusal, which only repeated the same check.

**Expected behavior**

When the saved files are provably unrecoverable or cannot be proven
safe, Retry continues the task in a fresh session and a new workspace
from the recorded work. The old files are never applied and the source
sandbox is not changed. No provider turn is replayed. The task says that
the unsaved files could not be recovered. The Retry offer and admission
agree.

**Steps to reproduce**

1. Let a native run finish with an accepted result, and let its
workspace copy-back fail (`native_workspace_sync_out_retry_exhausted`).
2. Let saved-work recovery end with `source_unavailable` (sandbox
destroyed) or with `allocation_ownership_ambiguous` (shared reusable
sandbox).
3. Press Retry.
4. Observe that the Retry receipt waits with
`saved_work_recovery_failed` or `saved_work_recovery_refused`, and no
run starts. Press Retry again and observe the same result.

**Paperclip version or commit**

Staging on `origin/master` after #21315 (`bbafa78fe2`).

**Deployment mode**

Hosted cloud (staging), remote Daytona sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime runs).

## What Changed

- `savedFilesUnrecoverable` and `unrecoverableSavedFilesFailure`
(`system-recovery-state.ts`) are the one rule for saved files that no
recovery can apply. The rule is true for a spent recovery whose failure
is a permanent source refusal: `source_unavailable`,
`allocation_ownership_ambiguous` or `historical_source_missing`. It is
false for `source_binding_invalid`, because a new recovery generation
can still verify that source. It is false for `archive_unsafe` and
`snapshot_verification_failed`, because a fresh attempt still imports
earlier saved snapshots and would fail before the agent starts. It is
also false after a fresh attempt is recorded.
- Admission (`explicit-native-continuation.ts`) no longer parks a Retry
of an accepted result in `saved_work_recovery_*` when every live
recovery of the run is unrecoverable. The Retry takes the existing
fresh-workspace path (`fresh_isolated_workspace`). That path keeps all
stop proofs, leaves the source sandbox and workspace untouched, and
starts a fresh session.
- When such a Retry starts, the task gets one system comment. The
comment says that Paperclip could not recover the unsaved files, why,
and that the task continues from its recorded work (comments, documents,
attachments and pushed commits). The settled hold records the same text.
- Retry (`task-retry.ts`) binds to an unrecoverable recovery generation.
It does not open a new generation that repeats the same check.
- The blocker text for unrecoverable files says that Retry continues the
task in a new workspace from its recorded work. It no longer says that
Retry succeeds only after an update.
- `canAdoptNativeSourceRetention` and the permanent source-refusal lists
moved into `system-recovery-state.ts`, so that admission, Retry and
messages share them. The old modules re-export them.

## Verification

- Red/green: `cd server && npx vitest run
src/services/unrecoverable-saved-work-retry.test.ts`. The tests fail on
`origin/master` sources (or on the earlier head for the snapshot case)
and pass with this change. They are built from the two staging shapes
(destroyed sandbox with `source_unavailable`, and a shared reusable
sandbox with `allocation_ownership_ambiguous`):
- offers a Retry that continues the task (blocker and task view agree,
and the text names the fresh workspace)
- starts a fresh run from the recorded work: the Retry receipt has no
wait and a run, the strategy is `fresh_isolated_workspace`, the hold is
settled with the fresh attempt, one system comment explains the lost
files with no raw IDs, the source lease is unchanged, and no new
recovery generation opens
- keeps a Retry out of a fresh attempt that would import a succeeded
snapshot whose archive is unsafe
- `workspace-recovery-warm-handoff.test.ts` › "wakes physical
preservation from HTTP Retry before passive history without replaying
the provider" failed on an earlier head of this PR, which also treated
`source_binding_invalid` as unrecoverable. It passes unchanged now.
- Updated tests that pinned the old repeat-the-check behavior:
`workspace-recovery-preparation-gates`, `task-retry`,
`system-recovery-state`, `workspace-recovery-retry-copyback`.
- Related suites pass: `system-recovery-state`,
`workspace-recovery-preparation-gates`,
`workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`,
`task-retry`, `saved-work-retry-eligibility`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
`workspace-recovery-admission`, `task-interruption-query`,
`native-workspace-sync-retention`.
- `cd server && node scripts/typecheck-layers.mjs` passes (source and
test layers).

## Risks

- Medium-low. A Retry of an unrecoverable hold now starts a new agent
turn in a new workspace instead of waiting. The turn starts only through
the existing fresh-workspace admission, which keeps every stop and
cleanup proof.
- Unsaved files of the old run are not applied. That was already true;
the change is that the task continues and says so.
- `source_binding_invalid`, `snapshot_verification_failed` and
`archive_unsafe` keep today's path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:23:23 -06:00
DottaandClaude Opus 5.5 566843820c fix(recovery): a queued continuation is not shown blocked, recent temporary failures continue, and notices say what a person can do (#21336)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, recovery continues the task, shows the
person a notice, and sweeps for tasks that nothing will continue.
> - On staging, a task stayed "Blocked / needs attention" for 24 minutes
while its Retry's run was queued.
> - Two tasks escalated to the board after a sandbox resume timeout, a
temporary failure, and nobody continued them.
> - Recovery notices asked a "Board operator" to inspect evidence and
repair sandboxes. Recovery must work without that.
> - A children-completed system wake could start on a parent that was
already closed.
> - This pull request fixes those four gaps.
> - The benefit is that interrupted tasks continue on their own, and
notices tell a person only what happened and the one thing they can do.

## Linked Issues or Issue Description

Refs #21250, #21257 (merged), #21258, #21234

**What happened?**
1. A task whose continuation was queued was shown as blocked (one
staging task). Execution recovery blocked the task at 23:51:28, and
restart recovery's Retry queued the successor at 23:51:50 through
`admitExplicitNativeContinuation`. That path resolves the interruption's
holds, but it never lifted the block those holds owned. The run started
at 00:15, so the task was queued for 24 minutes and shown blocked the
whole time. This was not a race: the issue's `updatedAt` equals
`blockedTransitionAt`, so nothing wrote to the issue after the Retry.
2. Board-owned stranded holds after a temporary failure had no continuer
(two staging tasks). A person's Retry failed while resuming the saved
sandbox (`reusable_sandbox_resume_failed`, cause `transport_timeout`,
`providerWorkStarted=false`), and stranded recovery escalated the task
to the board. The sweep leaves a blocked task that has any active hold
to the hold's owner. The fresh-continuation fallback takes only
execution-reconciliation holds. So nobody continued these tasks.
3. Recovery notices addressed a "Board operator" and asked people to
inspect evidence and repair runtimes, sandboxes or workspaces.
4. An `issue_children_completed` system wake could start on a parent
that was already done or cancelled.

**Expected behavior**
A task with a queued continuation is not shown blocked. A recent
temporary failure is continued automatically. A notice says what
happened and the one thing a person can do. A system wake does not start
on a closed task.

**Steps to reproduce**
1. Fail a run so that execution recovery blocks the task, then press
Retry while the agent's queue is full. The task stays Blocked until the
run starts.
2. Make a saved-sandbox resume time out on a person's Retry. The task
escalates to the board and stays blocked.

## What Changed

- **Queued continuation is not shown blocked**
(`explicit-native-continuation.ts`,
`recovery/assigned-work-invariant.ts`,
`recovery/assigned-work-sweep.ts`):
- `admitExplicitNativeContinuation` (a person's Retry, restart
recovery's Retry, or a saved message) moves the task from Blocked back
to To do when the replaced interruption set the block. That means the
hold's own block receipt, or a block written at or after the source run
ended with no later decision. A declared unblock action or an open
blocker keeps the block.
- New invariant violation `blocked_with_live_run`: the task is blocked,
has a queued or running run, and is not waiting on anyone else. Under
the row lock, the sweep lifts that block without starting a run or
spending the cap. Its suspect pre-filter now lets blocked tasks with
live runs through.
- **Recent temporary failures are continued** (new
`recovery/stranded-hold-continuation.ts`,
`recovery/assigned-work-sweep.ts`, `fresh-continuation-fallback.ts`,
`recovery/stranded-notice.ts`):
- The assigned-work sweep is the single owner of board-owned
`stranded_assigned_issue` holds whose failed run ended within the last
24 hours with a restart-class code, or with a temporary setup failure
before provider work (for example a sandbox resume refused for a
timeout).
- Under the row lock it resolves the hold (outcome `restored`, owner
system), restores the task's status from before the block, and queues
one continuation under the usual per-task cap and spacing.
- Older escalations and lasting causes (access denied, lost sandbox,
unknown) stay with their owner. The fallback explicitly never takes a
`stranded_assigned_issue` hold.
- **Plain recovery notices** (`recovery/service.ts`,
`recovery/stranded-notice.ts`, `recovery/successful-run-handoff.ts`):
see the wording table below.
- **Closed parent** (`heartbeat/queue.ts`): `issue_children_completed`
is added to the claim-time cancel for system continuations on a done or
cancelled task. A person's saved message still runs, and so does
onboarding's completion report (`onboardingCompletion: true`), which is
meant to run on a done parent.
- **Review fixes** (Greptile, first round):
- A person's block is never lifted by a Retry or saved message. The
block-time inference applies only when no hold carries a block receipt,
and a block whose activity entry was written by a person is kept. This
is the same signal the sweep uses (`attributeBlock`, so `blockedBy:
"user"` is a wait).
- The sweep's suspect query lets a blocked task with a live run through
the queued/claimed-wake filter, so a continuation queued by
`enqueueWakeup` (a queued wake bound to its run) reaches the
`blocked_with_live_run` repair.
- The saved-sandbox and workspace-setup timeout notices promise an
automatic retry only when the sweep will continue that hold. Both use
one decision (`strandedRunCauseIsTemporary`, inside
`strandedHoldContinuesAutomatically`). Otherwise the notice says to
press Retry.

### Old → new wording

| Where | Old | New |
|---|---|---|
| service.ts hold nextAction, successful run with missing state | Board
operator: inspect the run evidence, then explicitly choose a valid issue
disposition... | The agent finished without saying whether the task is
done. Mark the task done if it is, or press Retry to let the agent
continue from the saved work. |
| service.ts, process_lost / default | Board operator: inspect the retry
history / inspect the evidence, repair the runtime if appropriate... |
This attempt could not be finished. Press Retry to start it again from
the saved work. (`RETRY_FROM_SAVED_WORK_NEXT_ACTION`) |
| service.ts, provider_quota | Wait for provider quota recovery, then
retry the original assignee; do not wake a takeover owner. | The AI
provider's usage limit was reached. The same agent continues the task
once the limit resets. |
| service.ts, environment provider | No agent work started. Retry the
task once the environment provider is available. | The sandbox service
was unavailable, so no agent work started. Press Retry once it is
available again. |
| service.ts, inactivity | Board operator: inspect the inactivity
evidence... | The agent stopped responding, so this attempt was ended.
Press Retry to start it again from the saved work. |
| service.ts, worktree branch incoherence | Board operator: repair the
source task git worktree branch incoherence... | The task's workspace no
longer matches its branch, so the agent did not start. Press Retry to
start it again. |
| service.ts, base materialization | Board operator: repair the project
workspace repository URL or clone access... | Paperclip could not copy
the project's repository. Restore access to the repository in the
project settings, then press Retry. |
| service.ts, workspace link | Board operator: repair the source task
workspace link, project workspace cwd, or git checkout... | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| service.ts, base ref | Check the starting branch and repository
access, then repair the task's branch... | The starting branch for this
task's workspace was not found in the repository. Set a starting branch
that exists, then press Retry. |
| service.ts, sandbox plugin | Board operator: the sandbox provider
plugin named in the run failure is not ready; ... explicitly retry... |
The plugin that starts this agent's sandbox is not running. To fix it,
&lt;remedy&gt;, then press Retry. |
| service.ts, missing secret | Board operator: bind the missing
secret(s) named in the run failure... | A secret this agent needs is not
set. Add it in the agent's settings, then press Retry. |
| service.ts, review participant | Board operator: repair the failed
review participant path... | The reviewer did not finish the review.
Choose a reviewer, or press Retry to ask the same reviewer again. |
| service.ts failure summary | Latest retry failure details were
withheld from the issue thread; inspect the linked run for evidence. |
The failure details are kept with the run. |
| service.ts nested recovery comment | the assigned recovery owner or
board operator should fix the runtime/adapter problem... | press Retry
on the original task to start it again from the saved work, then close
this task. |
| service.ts recovery-issue stop comment | the current recovery owner
should inspect the failed run evidence, restore a live execution path...
| press Retry to start this task again from the saved work. |
| service.ts deliberate-wait exhaustion hold and comment | Inspect the
evidence and choose whether to repair, retry... / Next action: repair
the liveness disposition or request an explicit source-owner decision. |
The agent stopped without saying what happens next with this task. Mark
the task done if it is, or press Retry... / Next action: mark the task
done if it is, or press Retry... |
| stranded-notice REMOTE_WORKSPACE_RESTORE_NEXT_ACTION | Open the failed
run... inspect and move it aside, then retry | The saved work could not
be put back into the sandbox, so no agent work started. The saved
version was kept. Press Retry to start again from it. |
| stranded-notice scan notices (timeout, plugin worker, saturated,
output limit, failed, cancelled) | Check repository access and server
load / Check the worker status / Inspect the failed run... | "&lt;what
happened&gt;, so no agent work started. Press Retry ..." |
| stranded-notice workspace validation seed | ...Moving it to `blocked`
so the workspace link, cwd, or git checkout can be repaired | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| stranded-notice base ref seed | ...blocked until its workspace
configuration or repository access is repaired / Check that the
configured base ref exists... | starting branch not found / Set a
starting branch that exists in the repository, then press Retry. |
| stranded-notice stale shared holder | Check whether the other run is
still working and stop it if it is stuck... | Stop the other run on this
task, or choose a separate workspace for this task, then press Retry. |
| stranded-notice secret default | required secret/env bindings are
missing. Moving it to `blocked` so an operator can bind... | The agent
did not start because a secret it needs is not set. Add the secret in
the agent's settings, then press Retry. |
| stranded-notice review seeds (2) | ...so the board can inspect the
evidence, repair the reviewer runtime... | The reviewer ... did not
finish / cannot take work. Choose a reviewer, or press Retry ... |
| stranded-notice default body | could not restore a live execution
path... visible for intervention | This attempt could not be finished
automatically. Press Retry to start it again from the saved work. |
| stranded-notice metadata | Board decision required / "Inspect the
evidence, then retry ... repair the execution path ..." / "The recovery
owner should either restore a live execution path..." | Waiting for you
/ `RETRY_FROM_SAVED_WORK_NEXT_ACTION` / The assigned agent continues the
task from the saved work |
| successful-run-handoff.ts | Board decision required / inspect the
evidence, then retry the original owner... | Waiting for you / Mark the
task done if it is, or press Retry to let the agent continue from the
saved work |
| stranded-notice, saved sandbox resume timeout and workspace setup
timeout | asked to inspect the run or the provider | No agent work
started; Paperclip tries again automatically. |

Kept on purpose:
- `buildImmediateExecutionPathRecoveryNoticeSeed` ("No live execution
path" title, "retried continuation ... visible for intervention" body).
The UI recognizes this exact text
(`ui/src/components/task-chat/adjacent-errors.ts` regex and title,
`ui/src/lib/system-notice-humanizer.ts`) and humanizes it, so changing
it needs a UI change in the same PR.
- The UI's own "Board decision required"
(`IssueRecoveryActionCard.tsx`).
- Recovery-issue comments that still contain internal codes
(`stranded_issue_recovery`).
- The agent-facing disposition instruction in
`successful-run-handoff.ts`.

### Deliberately changed test expectations

Each of these asserted the old wording or the old behavior:
- `server/src/services/recovery/stranded-notice.test.ts`: stale shared
holder; base ref body and next action; scan timeout metadata; plugin
worker metadata; secret default body; owner row is now "Waiting for
you".
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: owner row
is now "Waiting for you" (3 places); failure-summary line;
board-escalation nextAction expectations (review participant,
process_lost, missing disposition, deliberate wait, default); sandbox
plugin; workspace link and validation comments; missing secret action
and comment; review participant comment.
- `server/src/__tests__/assigned-work-invariant.test.ts`: the old "left
to the hold owner" test now uses a 30-hour-old run. A recent temporary
failure is now continued, which is the new behavior.
- `server/src/services/heartbeat/queue.test.ts`:
`issue_children_completed` is now in the claim-time cancel list.
- `server/src/services/heartbeat/run-completion.test.ts`: the
restore-failure next action now says "The saved version was kept".
- `server/src/__tests__/issue-recovery-actions.test.ts` and
`server/src/__tests__/heartbeat-workspace-branch-containment.test.ts`:
the branch-incoherence next action now says the "workspace no longer
matches its branch".
- `server/src/services/recovery/stranded-notice.test.ts`: the timeout
notice is checked for both an automatic and a manual retry.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `task-retry.test.ts` "lifts the failure's block once the Retry's run
is queued..."; invariant unit test and sweep integration test for
`blocked_with_live_run`.
- sweep integration test "continues a recent board escalation of a
temporary setup failure...". A recent one is continued; a 30-hour-old
one and `provider_access_denied` are left with their owner. Plus a
`stranded-notice.test.ts` case.
- the wording expectations above.
- `heartbeat/queue.test.ts`.

Review-fix tests, each red before its fix:
- `task-retry.test.ts` "lifts the failure's block...": new `newer` case
(the hold's receipt names an older status version) and `person` case (a
person's activity entry set the block). Both stay blocked.
- `__tests__/assigned-work-invariant.test.ts` "a recovery block whose
task already has a queued run...": now seeds a queued wake bound to the
queued run, as `enqueueWakeup` does, and asserts `listSuspects` returns
the task.
- `heartbeat-dependency-scheduling.test.ts` "dispatches only a verified
completed onboarding report" (it failed in CI on the first push) passes
again.

Evidence:
- Item 1: the staging task's activity log. The issue's `updatedAt`
equals `blockedTransitionAt`; nothing wrote after the Retry.
- Item 2: the two staging runs ended at 12:28Z and 14:23Z on Nov 12,
inside the 24-hour window.
- Ruled out, so there is no change for it: a third staging task is
waiting on a real person's gate. A pending `request_confirmation` was
created before the fallback run failed.

Local runs, on master after #21257 (e063587edd) plus these commits
(server `tsc --noEmit` also passes):
- `stranded-notice.test.ts`, `recovery/assigned-work-invariant.test.ts`,
`task-retry.test.ts` and `__tests__/assigned-work-invariant.test.ts`: 4
files plus `heartbeat/queue.test.ts`, 329 tests passed.
- `heartbeat-process-recovery.test.ts` and `recovery-wake-herd.test.ts`:
475 tests passed.

Code comments that the original commits added were removed to follow
CONTRIBUTING; the explanations are in this description.

## Risks

- The sweep now resolves board-owned stranded holds for recent temporary
failures. It is bounded to failures within the last 24 hours with a
restart-class or temporary-setup cause, one continuation per task under
the usual cap and spacing. Lasting causes stay with their owner.
- Lifting a block on explicit continuation is limited to blocks the
replaced interruption set. A declared unblock action or an open blocker
keeps the block.
- Wording only changes server text. The UI-recognized "No live execution
path" notice is unchanged.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 04:04:20 -06:00
DottaandClaude Opus 5.5 bbafa78fe2 fix(retry): offer Retry only when admission can take it, and start held saved-work Retries (#21315)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run fails and its saved work is held for recovery, the task
shows a Retry button. Admission then decides if the Retry can start a
new turn.
> - The task view, the task's execution blocker, and admission each
computed their own answer. The task view said `canRetry: true`. The
blocker said `canRetry: false`, and its own `workspaceRecovery` part
said `canRetry: true`.
> - Admission refused these Retries with no reason. The receipt kept the
generic `execution_recovery` wait ("Retry is waiting for execution
checks"), and each later click joined the same held receipt. Nothing
ran.
> - The refusals had real causes. A preparation proof rejected
host-written workspace notices in the run log. A repair proof did not
accept a snapshot that the failed preparation only restored and never
changed.
> - This pull request fixes both proofs, names every admission refusal
of a held Retry, and makes the task view and the blocker read one shared
Retry answer.
> - The benefit is that Retry starts work in these cases, and Retry is
offered only when admission can take it.

## Linked Issues or Issue Description

**What happened?**

On hosted staging, several blocked tasks showed a Retry button. Pressing
Retry was accepted but nothing ran. The task said "Retry is waiting for
execution checks. Your message is saved." The task view had
`interruption.canRetry: true` and `retryWait.reason:
execution_recovery`. The task's `executionBlocker` had `canRetry:
false`.

- Saved-work recovery failed with `source_unavailable`. The legacy
preparation had failed before the provider started
(`executionRecovery.providerWorkStarted: false`), its sandbox epoch was
abandoned, and its sandbox was destroyed. The blocker contradicted
itself: `workspaceRecovery.canRetry: true`, top-level `canRetry: false`.
- Saved work showed "recovered", but `workspaceRepairRequired` stayed
`true`. The failed preparation had restored a snapshot that an earlier
run produced. The repair proof accepted only a snapshot that the run
itself saved.

**Expected behavior**

The task view, the execution blocker, and admission give the same Retry
answer. A Retry that admission can take starts work. When saved work
cannot be recovered, Retry continues in a fresh session from the
recorded work. Every admission refusal of a held Retry has a named
reason. No Retry is offered for a refusal that another click cannot
change.

**Steps to reproduce**

1. Let a legacy `paperclip_runner` preparation fail before the provider
starts, after the host writes its workspace notices ("Execution
workspace reuse freshness action ...", "Skipping saved session resume
...") to the run log.
2. Let saved-work recovery end with `source_unavailable` and
`use_alternative_strategy`.
3. Press Retry.
4. Observe that the Retry receipt is held with `executionWait.reason =
execution_recovery` and no run starts.

Or:

1. Let a legacy preparation restore the task's saved snapshot and fail
with `setup_failed` before the provider starts.
2. Let saved-work recovery complete.
3. Observe that the blocker keeps `workspaceRepairRequired: true`, and
that Retry is held with `execution_recovery`.

**Paperclip version or commit**

Staging on `origin/master` `62d673cc3f` (after #21267 and #21295).

**Deployment mode**

Hosted cloud (staging), remote Daytona sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (legacy runtime runs, native agent).

Refs #21267, #21295.

## What Changed

- `hasVerifiedPreparationLog` (`verified-preparation-log.ts`) now
accepts a log made only of host notices that the host writes before it
hands the turn to an adapter: the mentioned-skills line (first record
only, checked against the saved skill selection) and the workspace
freshness and session-reset notices. Every record must have the same
attempt, sequential `seq`, `stdout`, and the excerpt and watermark must
match. Any other line still keeps the run held.
- `verifiedUnchangedRestoredSnapshot`
(`remote-workspace-snapshot-reuse-receipt.ts`) proves that a failed
preparation's saved state is the snapshot it restored. It needs
`executionStage: preparing`, `providerWorkStarted: false`, exactly one
succeeded restore of that snapshot at that epoch and lease, and no
snapshot save by the run. `verifiedWorkspaceRepairReceipt` uses it next
to the existing reuse proof.
- `savedWorkOffersRetry` (`workspace-recovery-receipt.ts`) is the one
rule for Retry on a saved-work hold. Retry is offered when the saved
work is recovered and verified, or when recovery is exhausted in a state
that only a Retry restarts (a new recovery generation, or the
fresh-session fallback after `source_unavailable`). Retry is not offered
while recovery re-arms by itself (a stop recheck or a newer capture
policy), or while a copyback owner's stop is unconfirmed. The blocker
uses this rule for the top-level `canRetry` and for
`workspaceRecovery.canRetry`, so the two parts cannot contradict.
- `task-retry-eligibility.ts` lists the held-Retry refusals that another
click cannot change. The task view and the blocker both use it.
- Admission (`explicit-native-continuation.ts`) names each refusal of a
held request that was silent before: `previous_execution_unverified`,
`execution_settling`, `cleanup_pending`, and `remote_cleanup`. It names
a refusal only when an execution blocker holds the task, so a request
without a hold still takes the normal path.
- The task view now reads the blocker's answer for a task with a
saved-work hold. It shows no Retry when the blocker offers none, and it
names the recovery wait.

## Verification

- Red/green: `cd server && npx vitest run
src/services/saved-work-retry-eligibility.test.ts`. The tests fail on
`origin/master` sources and pass with this change. They are built from
the staging shapes:
- offers the same Retry in the task view and the blocker when saved work
is no longer available
- offers no Retry in either view while saved-work recovery waits for an
unconfirmed file copy to stop
- continues in a fresh session from the recorded work when saved work is
no longer available
- verifies the repair of a restored snapshot that the failed preparation
never changed
- keeps the repair unverified when the failed preparation may have
changed the restored snapshot
- names a refusal that another click cannot change and offers no Retry
for it in either view
- names the admission refusal of a held Retry and offers Retry only when
admission would take it
- `npx vitest run src/services/verified-preparation-log.test.ts`: new
cases accept the host notices and keep a provider line, a different
attempt, or a late skills line held.
- Related suites: deferred execution wait, execution blocker read,
explicit native continuation, governed wait runner loss, legacy
preparation retry hold, remote workspace snapshot reuse, task
interruption presentation, task retry, and all `workspace-recovery-*`
suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Medium-low. The preparation-log proof accepts more host lines. Each
accepted line has a fixed host format, and the run must still pass every
other preparation proof (events, leases, no provider evidence).
- The restored-snapshot proof applies only when the run recorded that
the provider never started and it never saved its own snapshot.
- Admission now records a named wait instead of the generic one for some
held requests. It changes no admission decision.
- The task view calls `getExecutionBlocker` for tasks that offer Retry
and have a saved-work hold. These tasks are few, so the extra reads are
small.
- Not covered here: a copyback owner on a replaced deployment still
needs its stop confirmed before saved work can be copied out of a full
sandbox. The task view now shows that wait instead of a Retry that
admission refuses.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:52:25 -06:00
DottaandClaude Opus 5.5 c6c7a61e2c fix(retry): release a Retry held on a resolved decision (#21267)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, the task shows a Retry button. Admission
can hold an accepted Retry while a person must answer a pending approval
or question (`decision_pending`).
> - The hold is saved on the deferred Retry receipt. Nothing cleared
that saved reason when the person answered, so the task kept saying
"Retry is waiting for a pending approval or question" when nothing was
left to answer.
> - The deferred-receipt worker drops a refused receipt from its
schedule. No producer hinted it when the decision resolved, so the held
Retry did not start until some unrelated event rescanned the company.
> - A Retry was also held by a confirmation that was already open when
the failed turn was admitted. That turn had already run past the
confirmation, so the Retry added a gate that the original turn did not
have.
> - A person's comment on the task started a continuation, but the held
Retry receipt stayed deferred next to it.
> - This pull request makes a resolved decision release the held Retry
at once, keeps the projection from showing a resolved decision as the
wait, stops holding a Retry on a decision that predates the failed turn,
and settles a held Retry when a comment or answer continues the same
failed run.
> - The benefit is that Retry always works: a person never waits on a
decision that is already resolved.

## Linked Issues or Issue Description

**What happened?**

A board user pressed Retry on blocked tasks. No run started. The task
said "Retry is waiting for a pending approval or question to be
resolved. Your message is saved." In one task, the only interaction was
a `request_confirmation` that the previous successful run created. The
user's later comment started the turn that failed. The confirmation
stayed pending, so every Retry attached to the same held receipt. After
the user rejected the confirmation and sent a new comment, the task
still showed the same wait. No run started until a server restart
rescanned the deferred receipts about three minutes later. The task
projection said `canRetry: true` with the `decision_pending` wait.

**Expected behavior**

An accepted Retry is never held on a decision that is already resolved.
Accepting or rejecting the decision releases a held Retry right away. A
person's comment on the blocked task starts work, and the held Retry is
settled. The task projection never shows a resolved decision as the
Retry wait.

**Steps to reproduce**

1. Let an agent run on a remote runner create a `request_confirmation`
for the task.
2. Let the runner be lost so the run fails with
`native_continuation_requires_reconciliation`, and let saved work
recovery complete.
3. Press Retry. The Retry receipt is saved with `executionWait.reason =
decision_pending`.
4. Reject or accept the confirmation.
5. Observe that the receipt keeps `decision_pending`, the task keeps
showing the wait, and the Retry does not start until an unrelated
rescan.

**Paperclip version or commit**

Staging Core `57aeda190b`. Fixed on top of `origin/master` `53608bd1d6`.

**Deployment mode**

Hosted cloud (staging), remote sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime).

## What Changed

- `hasPendingContinuationDecision` (`explicit-native-continuation.ts`)
is now the single check for a decision that holds a person's
continuation. Admission and the task projection both use it.
- For a Retry, a pending interaction that was created before the failed
source run was created does not hold the Retry. That turn was already
admitted while the decision was open. Decisions that the failed run
created (for example a plan review) still hold the Retry.
- `releaseDecisionHeldExecution` (`deferred-execution-wait.ts`) removes
the saved `decision_pending` wait from deferred receipts on the task and
hints each receipt to the deferred-receipt worker. It runs in the
resolution transaction of every interaction resolution path (through
`notifyChatConfirmationResolution`) and in question answers.
- The task projection reads the live decision state. A saved
`decision_pending` wait whose decision is resolved is not shown. This
also repairs receipts saved before this change.
- When a comment or an interaction answer continues a failed run,
`retireTaskRetriesContinuedBy` (`task-retry.ts`) settles held Retry
receipts for the same source run as `task_retry_superseded`.

## Verification

- Red/green: `cd server && npx vitest run
src/services/governed-wait-runner-loss.test.ts`. The five new tests fail
on `origin/master` sources and pass with this change (17/17 pass):
  - releases the held Retry as soon as the person chooses accept
  - releases the held Retry as soon as the person chooses reject
- never shows a resolved confirmation as the Retry's wait, and both
projections offer the same Retry
- a person's comment starts work and settles the held Retry once nothing
is left to answer
- starts a Retry at once when the confirmation was already open before
the failed turn started
- The existing tests that keep a Retry or a message on hold while a plan
review, governed tool action, secret proposal, or connection
authorization is pending still pass.
- Related suites: deferred execution wait, explicit native continuation,
interaction resolution and service, task interruption presentation and
query, task retry, deferred receipt work, interaction routes.

## Risks

- Medium-low. A Retry now starts when the only pending interaction is
older than the failed run. That turn already ran with the interaction
open, so the Retry gets no new authority. Governed tool actions still
need their own approval before the tool runs.
- Interaction resolution now also updates deferred receipts on the same
task. The update is limited to receipts with a saved `decision_pending`
wait. A receipt that is still blocked by another pending decision is
held again on its next admission.
- Issue-linked approvals (`approvals` table) do not yet hint the held
receipt when they resolve. The projection no longer shows them as the
wait after they resolve.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:58:48 -06:00
DottaandClaude Opus 5.5 6ac7211cb9 fix(heartbeat): stop re-delivering a remote-stop follow-up that a succeeded run already read (#21203)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's follow-up comment must reach the agent once
> - #21175 made a succeeded run's mid-turn read of a comment (through
`get_task_history`) count as delivery for the queued-comment paths
> - The remote-Stop resume path decides delivery with
`undeliveredLegacyUserCommentIds`, which ignores those reads
> - So a follow-up that a succeeded run already read was sent again, and
the agent posted a second DONE on a finished task
> - This pull request applies the same rule on that path and adds a read
helper for a later change
> - The benefit is one answer per follow-up on every resume path, with
no dropped messages

## Linked Issues or Issue Description

Refs #21175

**What happened?**
A saved follow-up comment was resumed after a remote Stop (wake reason
`issue_commented`, idempotency key `remote-stop-comment:…`). A succeeded
run had already read this comment mid-turn. The resumed wake started a
new turn, and the agent posted a second DONE on a finished task. The
cause: `undeliveredLegacyUserCommentIds`
(explicit-native-continuation.ts) checks only wake input and accepted
first turns. It does not check the `task_history.read` records that
#21175 added.

**Expected behavior**
A follow-up that a succeeded run read at its current revision is
delivered. The remote-Stop resume path skips it. A read by a run that
did not succeed does not count, so that follow-up is still delivered
once.

**Steps to reproduce**
1. A native run reads the task history after a person posts a follow-up.
2. The run succeeds. The follow-up stays saved as a deferred wake.
3. The remote-Stop resume path runs for that wake.
4. Before this change, the follow-up is delivered again.

**Paperclip version or commit**
`master` at 549816e6a0.

## What Changed

- `server/src/services/queued-comment-delivery.ts`:
`historyReadDeliveredCommentIds` exposes the #21175 rule for a list of
comment IDs. `taskHistoryReadsForRun({ companyId, runId })` returns each
comment one run read, with the read time and the comment revision that
the read returned. It returns these whatever the run's outcome. It reads
only that run's events through `heartbeat_run_events_company_run_idx`.
No delivery decision uses it.
- `server/src/services/explicit-native-continuation.ts`:
`undeliveredLegacyUserCommentIds` also removes comments that a succeeded
run read at their current revision. Its two callers are the remote-Stop
resume path and the queued explicit-continuation check, and both now use
the #21175 rule.
- Deliberate asymmetry: a read by a run that did not succeed never
counts. A dropped person's message is worse than a duplicate answer.
- `doc/run-log-events.md`: documents both points.
- New test file
`server/src/services/queued-comment-history-read.test.ts`.

## Verification

- RED on `549816e6a0` (`npx vitest run
src/services/queued-comment-history-read.test.ts`): 2 failed, 1 passed.
  - The follow-up that a succeeded run read was still returned as owed.
  - `taskHistoryReadsForRun` did not exist.
- The case where a failed run read the follow-up (still owed) passed, as
intended.
- GREEN: 3/3 pass.
- Neighbouring suites (`first-turn-queued-input`,
`explicit-native-continuation`, `heartbeat-agent-directory-busy`,
`heartbeat-held-comment-wake`, `execution-control-work`): 517/517 pass.
Server `tsc` is clean.

## Risks

- Low risk. The function and its callers keep their contract. It returns
fewer comment IDs only when a succeeded run of the assignee read that
exact revision.
- Comments read through the REST API by non-native adapters are not
recorded, so their behavior does not change.
- No migration. The added lookup uses the time-bounded, index-backed
query from #21175.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:23:34 -06:00
DottaandClaude Opus 5.5 cc6728e8de fix(recovery): continue upgrade-stopped Retry runs that the sweep refused or skipped (#21108)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Hosted servers are upgraded often. Each upgrade stops the old server
gracefully, and a run that is still preparing ends `interrupted`
(`server_shutdown_interrupted`) with bootstrap evidence that provider
work did not start.
> - #21078 let a run that a person started with Retry continue
automatically after such a stop. After it deployed, a read-only audit of
a staging server found four groups of Retry-started tasks that still did
not continue: some were refused on every sweep, some were never looked
at, and one Retry that a person pressed failed and was not retried.
> - Product rule: an upgrade never leaves a task that needs a manual
Retry, and a Retry a person presses must work. A refusal that cannot
succeed later must not repeat forever.
> - This pull request fixes each cause found, keeps every authorization
check of the continuation gate, and ends the refusal loop with one plain
escalation.
> - The benefit is that Retry-started tasks continue through upgrades,
including tasks earlier releases left waiting, with no manual step.

## Linked Issues or Issue Description

Refs #21078, #21046, #21019, #14987, #19384

**What happened?** On a hosted staging server after #21078 deployed
(read-only audit, run events and task activity):
1. A Retry-started task was refused every ~2 minutes ("The automatic
retry could not revalidate the original user continuation."). While
preparing, its Retry had switched to a fresh isolated workspace because
the saved files could not be restored (activity
`issue.fresh_workspace_selected` by the system, 40 minutes after the
Retry). That rewrote the Retry request's strategy from `continue` to
`fresh_isolated_workspace`, and `readTaskRetryRetryAuthority` accepts
only `continue`.
2. Two tasks whose Retry replaced a native session had no recovery event
for hours. The stranded-task sweep skipped them silently.
3. A task whose Retry selected a fresh workspace at admission
(`explicitUserContinuation.readiness`) was refused on purpose
("fresh-workspace Retries keep their own contracts"), yet that contract
had no path for a shutdown before provider work. The task stayed in
progress after every upgrade.
4. A person pressed Retry. The run failed before provider work with
`reusable_sandbox_resume_failed`, a transient failure that is already in
the bounded bootstrap retry lane. One second later the stranded-task
sweep escalated the task to `blocked`, which cleared the claim. Two
seconds after that, the run's own failure path scheduled its retry and
was refused because the claim no longer named the run.

**Expected behavior:** each of these continues automatically once, under
the person's original Retry. A refusal that cannot succeed later
escalates the task once with a working Retry.

**Root causes:**
- Cases 1 and 3: the continuation gate refused every fresh-workspace
Retry, whether the fresh workspace was chosen at admission or later in
preparation. That includes one that a shutdown stopped before it did
anything.
- Case 2: the sweep's generic skips run before the continuation. Two of
them can match only because of state that the replaced session left. An
unfinished session goal (`agent_task_sessions.goal_status`) is skipped
as "goal recovery owns it". A native finalization row is counted as a
live execution path by `hasActiveExecutionPath`. The person's Retry had
already started a fresh session, so neither owns the task. The audit
could not read the database, so which of the two applied on staging is
not proven; the tests cover both.
- Case 4: a race. The sweep does not wait while a terminal run's own
executor is still settling it, so it escalates before that executor
schedules the run's bounded retry.
- The refusal loop: after a refusal the sweep left the task
`in_progress` and tried again on the next pass.

**Paperclip version or commit:** master `482063aeca`.

## What Changed

- `explicit-native-continuation.ts`: `readTaskRetryRetryAuthority`
renews a fresh-workspace Retry **only** when the parent run was stopped
by a graceful shutdown before provider work
(`isShutdownInterruptedPreparation`). The admitted `readiness` must name
the same Retry and source. A strategy that preparation switched to fresh
is rebuilt exactly as preparation records it: this Retry, its source,
and the source's leases. An automatic retry of the fresh attempt
inherits the selection from its own receipt. The successor's receipt
records the selection under `automaticRetry.readiness`. The successor's
`explicitUserContinuation` keeps its plain shape.
- `fresh-workspace-continuation.ts`: `readFreshWorkspaceReadiness` reads
that receipt for an automatic continuation, so the successor prepares
the same fresh attempt. A remote fresh workspace is keyed by the Retry,
so it is reused rather than created again. `freshReadinessSchema` is
exported and reused.
- `recovery/service.ts`:
- **Shutdown-stopped Retry preparations go first:** at the start of the
stranded sweep, every `in_progress` task whose latest run is a legacy
`server_shutdown_interrupted` preparation (`providerWorkStarted: false`)
carrying `explicitUserContinuation` is offered to the scheduler. It is
skipped when the agent is not invokable, the task has a live execution
path, a board-owned recovery action or a pause hold, or the run is still
settling. The unfinished-goal skip does not apply here, because the
person's Retry started a fresh session.
- `hasActiveExecutionPath` ignores a native finalization that a person's
Retry already replaced (`failureDetail.replacementDenied =
explicit_user_continuation` on a terminal run).
- The sweep skips a task while its latest terminal run is still being
settled: a live execution in this process, or another server's unexpired
controller lease. It checks again when that lease expires.
- A refused continuation (`continuation_user_authorization_missing`)
escalates the task once to `blocked` with the plain notice "Retry needed
… Press Retry to continue." Before, the task stayed in progress and was
refused again on the next pass.
- `heartbeat.ts`: `scheduleRecoveryRetry` reports a refused continuation
to the sweep.
- `doc/execution-semantics.md`: describes these rules.

**Failure classes admitted for a Retry-started run (case 4):** no new
class. The run's own bounded retry lanes are unchanged.
`reusable_sandbox_resume_failed`, `remote_workspace_restore_failed`,
`plugin_worker_unavailable`, `environment_provider_unavailable`,
`repository_catalog_unavailable`, runner revision mismatch and the other
transient bootstrap failures were already retried before provider work.
The fix removes the race that took the claim away from that retry.

**Refusals that remain by design:**
- The task is reassigned, `done` or `cancelled`.
- Another run holds the claim, or newer or live work exists.
- The Retry actor, generation or run changed, or it is a chat or
queued-message Retry.
- The message was edited or deleted.
- The receipt is missing or revoked.
- The run had another interruption cause, or provider work started.
- A fresh-workspace Retry whose run *failed* (not a shutdown) keeps its
one-run contract.
- A readiness that names another Retry or another source.

Each of these now escalates the task once instead of refusing on every
sweep.

## Verification

- `server/src/services/task-retry.test.ts`, new tests. Red on
`origin/master` sources and green on this branch (checked by restoring
the master sources):
- "continues a fresh-workspace Retry that a shutdown stopped before
provider work":
- Runs twice: once with the fresh workspace selected at admission, once
with it selected at preparation.
- Two concurrent sweeps create one successor, which gets the claim, and
dispatch accepts it.
- `readFreshWorkspaceReadiness(successor)` returns the same fresh
attempt.
- A second upgrade stops the successor, and the next successor carries
the same attempt.
- "continues a shutdown-stopped Retry whose replaced session left state
behind": two cases, an unfinished session goal and a native finalization
that the Retry replaced.
- "escalates a refused shutdown continuation once instead of refusing it
on every sweep": two sweeps. The task ends `blocked` with one "Press
Retry to continue" notice and one refusal event.
- "leaves a failed Retry run to its own executor while that executor
still settles it":
- Production shape: `reusable_sandbox_resume_failed` and a live
controller lease from the server that ran the run.
    - The sweep leaves the task and its claim alone.
    - The run's bounded retry is then scheduled and dispatch accepts it.
- "keeps a fresh-workspace Retry's own contract when it does not match":
three refusal cases (other Retry, other source, failed rather than
stopped). Green on both.
- Existing suites pass:
- `task-retry`, `explicit-native-continuation`,
`heartbeat-retry-scheduling`, `heartbeat-process-recovery`,
`interrupted-preparation-hold-settle`,
`heartbeat-issue-liveness-escalation`, `issue-recovery-actions`,
`recovery-deadline-acceptance`, `legacy-continuation-authority`,
`workspace-recovery-admission`, `native-workspace-export-retry`,
`heartbeat-paused-assignee-new-task`, `issue-tree-pause-resume-run`,
`heartbeat-stale-queue-invalidation`. 1,619 tests passed, with one
failure that a change in this branch fixed; rerun of the affected
suites: 560/560.
- The one failure came from the first version of the settling check,
which also waited for this process's own lease after a shutdown drain.
The check now uses this process's live executions and other processes'
leases only.
- `tsc --noEmit` for `server`.
- Required CI covers the full suite.

**After deploy, the next audit should show:**
- Tasks stopped by a shutdown while their Retry was preparing get one
`scheduled_retry` successor on the first sweep. This includes tasks
whose Retry selected a fresh workspace and tasks whose Retry replaced a
native session.
- They show no further "could not revalidate" events.
- A Retry-started run that fails transiently before provider work gets
its bounded retry, and the task is not escalated in between.
- A task whose continuation is truly refused is `blocked` once with
"Retry needed".

## Risks

- **Behavior change:** the sweep now offers shutdown-stopped Retry or
message preparations to the scheduler before its generic skips. All
authorization checks still run at admission, under the task and run row
locks. The scheduler reuses an existing successor, so two servers create
one continuation. The tests run two sweeps at once.
- **Settling wait:** a task whose latest terminal run still has another
server's live controller lease waits until that lease expires (at most
90 s) before the sweep acts. Normal turn starts are unchanged, and
nothing is added before a turn starts.
- **Escalation on refusal:** a refused continuation now moves the task
to `blocked` with a Retry notice, where before it stayed `in_progress`
with no path. Pressing Retry is a new, separately admitted request.
- **Rows from older releases:** no schema change and no backfill. A
shutdown-stopped fresh-workspace Retry is continued once. An older
successor receipt without `automaticRetry.readiness` behaves as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, read-only API reads, Vitest on embedded PostgreSQL) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:41:25 -06:00
DottaandClaude Opus 5.5 c4927e1ec2 fix(recovery): continue a Retry-started run that a server shutdown stopped while preparing (#21078)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade stops the old server gracefully. A legacy run that is
still preparing ends `interrupted` with `server_shutdown_interrupted`.
The new server must continue it automatically (#21019, #21046).
> - A run that a person started with Retry (or a message) carries
`explicitUserContinuation`. Its automatic retries need their own receipt
from `admitExplicitContinuationRetry` (#14987, #19384).
> - That admission refused every shutdown-interrupted Retry run. The
task then stayed in progress with no run, and the stranded-task sweep
refused it again about every 2.5 minutes, forever. A person had to press
Retry.
> - An upgrade is routine. It must never leave a task that needs a
manual Retry.
> - This pull request lets the person's original authorization cover a
continuation after a shutdown that stopped the run before provider work
started. All other checks of the gate stay.
> - The benefit is that Retry-started tasks continue through an upgrade,
and tasks that earlier releases left waiting continue on their own after
deploy.

## Linked Issues or Issue Description

Refs #21019, #21046, #14987, #19384

**What happened?**
On a hosted staging server, each automatic upgrade stopped legacy runs
that were still preparing (`executionRecovery {kind: bootstrap,
providerWorkStarted: false}`). When the run came from a person's Retry
(wake reason `task_retry`), the continuation was refused with "The
automatic retry could not revalidate the original user continuation."
(`continuation_user_authorization_missing`). The task stayed
`in_progress` with no run and no next retry. Runs from older upgrades
were refused again by each sweep. A run from the same upgrade that did
not come from a Retry got its continuation in the
`interrupted_preparation` lane.

**Expected behavior**
A Retry-started run that a graceful shutdown stopped before provider
work continues automatically, once, under the person's original Retry.
Tasks that an earlier release already refused continue after deploy with
no manual step.

**Root cause**
Three conditions refused it (`server/src/services/heartbeat/retries.ts`,
`server/src/services/explicit-native-continuation.ts`):
1. The scheduler admits an explicit continuation only for the
`transient_failure`, `server_restart`, preparation-wait and
agent-directory lanes. A shutdown-interrupted preparation with unspent
failure budget uses `interrupted_preparation`, so it was refused before
admission ran.
2. `admitExplicitContinuationRetry` accepted only a `failed` or
`timed_out` parent. These parents are `interrupted`.
3. It required `task.executionRunId` to be the parent. The refusal
released the claim (`releaseIssueExecutionAndPromote`), so every later
sweep was refused by this check too.

**Paperclip version or commit**
master `f30005440b`.

## What Changed

- `retries.ts`: an explicit continuation may also use the
`interrupted_preparation` lane. Lane choice is unchanged (see below).
- `explicit-native-continuation.ts`: `admitExplicitContinuationRetry`
also accepts a parent that is `interrupted` by a graceful shutdown
(`isServerReplacementInterruption`, so `server_shutdown_interrupted`)
with bootstrap evidence `providerWorkStarted: false`
(`isShutdownInterruptedPreparation`).
- For such a parent only, a released claim (`executionRunId` null) is
accepted. Admission restores the claim to the parent under the task
lock, as the shutdown path leaves it, so the scheduler passes the claim
to the successor and dispatch accepts it. If the parent validation then
refuses, admission releases the claim again in the same transaction. A
claim that another run holds still refuses.
- `doc/execution-semantics.md`: describe the rule and the conditions
that still refuse. Also apply the sentence that #19384 left pending (a
plain task Retry is renewable).
- No schema change, no backfill. Nothing changes on the path that starts
a turn.

**Which lane a graceful-shutdown legacy preparation uses (after
#21046).** `scheduleBoundedRetryForRun` sets `interruptedPreparation`
for an `interrupted` run with the bootstrap evidence.
`usesServerRestartLane(run, true)` returns false while the failure
budget is unspent, so the run uses `interrupted_preparation`: it does
not spend the failure budget, the first continuation is due at once, and
it is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`. With a
spent failure budget it uses `server_restart`. This is the intended
design (#21046 comment on `usesServerRestartLane`), so no lane change.
The `server_restart` lane is for `process_lost` replaced preparations
and for runs past dispatch.

**Conditions that still refuse** (each has a test or an existing test):
- The task has another assignee, or is `done` or `cancelled`.
- Another run holds the task's claim.
- Newer work on the task (a run created after the parent, a retry child,
or a live run).
- The Retry request changed: other actor, not admitted, other
generation, bound to another run, a chat Retry, a fresh-workspace Retry.
A message continuation with an edited or deleted message.
- A missing or revoked receipt (dispatch validation of the parent
fails).
- An execution blocker on the task, or a live adapter controller for the
parent.
- An `interrupted` parent with another cause (for example an operator
interrupt), or with no bootstrap evidence, or with `providerWorkStarted:
true`. A parent that failed after provider work keeps the existing rules
(`failed`/`timed_out` only).
- Native runtime parents (unchanged, see below).

**Native runtime.** A native `paperclip_runner` run is not interrupted
by a graceful shutdown: `drainRunningRunsForShutdown` suspends it and
the next server reattaches it (`restart_suspended`). It does not reach
this gate. Native replacement paths (`native-safe-replacement`,
`native-provider-capacity-retry`) remove `explicitUserContinuation` from
the successor and use their own contract. So the native path does not
have this refusal, and this pull request does not change it.

## Verification

- `server/src/services/task-retry.test.ts`, new tests. Red on
`origin/master` (no successor), green on this branch:
- "continues a Retry-admitted run that a graceful shutdown stopped while
it was preparing": production shape. A legacy run admitted by a task
Retry, `preparing`, owned by this server, with bootstrap evidence.
`drainRunningRunsForShutdown("SIGTERM")` interrupts it with
`server_shutdown_interrupted` and schedules one
`interrupted_preparation` successor (`failureRetries: 0`,
`interruptedPreparations: 1`) with its own receipt. The claim passes to
the successor and dispatch accepts it. A concurrent stranded sweep and a
second schedule call reuse the same successor.
- "continues a task an earlier release left waiting after refusing its
shutdown continuation" (a direct Retry run, and an automatic retry of a
Retry run): the row and task as an older release left them (claim
released). Two concurrent stranded sweeps schedule exactly one
successor; the claim passes to it; dispatch accepts it.
- "does not continue a shutdown-interrupted Retry run without its
original authority" for: provider work started, no bootstrap proof,
another interruption cause, claim held by another run, reassigned,
superseded, other Retry actor, cancelled task. No successor, no receipt,
and the claim is left as it was.
- I confirmed that each source change is needed: without the lane change
both main tests fail; without the released-claim rule the older-release
test fails.
- Suites: `task-retry`, `explicit-native-continuation`,
`legacy-explicit-continuation-recovery`, `quarantined-task-retry`
(587/587); `heartbeat-process-recovery`, `heartbeat-retry-scheduling`
(520/520).
- `tsc --noEmit` for `server`.
- `pnpm verify` selects a broad run (exit 2) because `retries.ts` is
central; required CI covers that.

**After deploy, the next audit should show:** a Retry-started run that a
shutdown stops while preparing gets a `scheduled_retry` successor with
`scheduledRetryReason: interrupted_preparation` (or `server_restart`
with a spent failure budget) and no
`continuation_user_authorization_missing` event. The tasks that earlier
upgrades left waiting get one such successor on the first stranded sweep
of the new server, with the task claim on the successor.

## Risks

- Behavior change: a shutdown-interrupted preparation of a Retry or
message turn now continues automatically, within the existing uncharged
and bounded lanes. The run did no provider work, so there is nothing to
reconcile.
- Rows saved by older releases: a refused row whose claim was released
is continued once. If another run took the claim, or newer work exists,
it stays refused.
- Two servers at once: admission runs in the scheduler transaction under
the task and run row locks, and the scheduler reuses an existing
successor (`retryOfRunId`). The tests run two sweeps concurrently and
get one successor. An old server that still refuses cannot clear the
claim after the new server moved it to the successor (its release
matches only the stopped run).
- The claim restore writes `executionRunId` for a terminal run inside
the scheduling transaction. If admission refuses after the restore, it
releases the claim again before the transaction commits.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, Vitest on embedded PostgreSQL) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:02:56 -06:00
DottaandClaude Opus 5.5 34f71f7d57 fix(server): defer a run that meets a busy agent directory instead of failing it (#20829)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run of an agent with managed agent files gets a private copy of
the agent directory. Remote copies serialize their prepare, checkpoint,
collect, and release operations through one Postgres advisory lock per
agent, taken with `pg_try_advisory_xact_lock`
> - When the lock is busy, the operation fails at once with "Agent
directory operation is already active"
> - On staging, a deploy restarted the controller while three runs of
the same agent were active. When one run resumed, another run of that
agent was checkpointing its directory. The resume failed, used its last
attempt, and the task became blocked with "Retry or send a message"
> - A run must not fail because another run of the same agent is briefly
using the agent directory
> - This pull request keeps the fail-fast lock and, when it is busy
during preparation, puts the same native run back on the existing
same-run retry schedule a few seconds later
> - The benefit is that lock contention delays a run by seconds instead
of ending it as failed

## Linked Issues or Issue Description

**What happened?**
A task's run failed with `native_continuation_requires_reconciliation`
and the error "Agent directory operation is already active". The task
was blocked and asked the user to retry. The same error ended at least
two other runs of the same agent within one day.

**Steps to reproduce**
1. Use an agent with managed agent files in a remote sandbox.
2. Let one run of the agent checkpoint or collect its agent directory.
3. At the same time, let another run of the same agent prepare (for
example, a same-run resume after a controller restart).
4. The second run fails immediately with "Agent directory operation is
already active".

**Expected behavior**
The second run waits until the first operation is done, then continues.
It does not end as failed.

## What Changed

- `server/src/services/agent-directory-working-copies.ts`: The busy lock
now throws `AgentDirectoryOperationBusyError`, still a 409 with the same
message. Lock behavior is unchanged from `master`: a fail-fast
`pg_try_advisory_xact_lock` held for the life of the holder's
transaction.
- `server/src/services/heartbeat/agent-directory-busy.ts`:
`prepareWithAgentDirectoryDeferral` wraps only instruction preparation
and turns the busy error into `AgentDirectoryPreparationDeferral`
("Another run of this agent is still using its files"). A busy lock met
while collecting files after a turn is not wrapped and keeps its
existing path.
- **Before native admission** (a brand-new run, such as the first run of
a task): `server/src/services/heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` cancels the run as an expected
control-plane wait (`providerWorkStarted: false`) and schedules an
`agent_directory_busy` retry 5–15 seconds later. The task keeps its
execution lock on the retry. The lane spends no failure retries
(`execution-recovery-attempt.ts`), its count is the retry run's
`scheduledRetryAttempt`, and it stops after 30 attempts.
`pre-dispatch-execution-evidence.ts` classifies this wait like the
existing workspace and AI-connection waits, so retry scheduling does not
ask for reconciliation.
- **A resumed native run**:
`server/src/services/native-runtime/native-controller-recovery.ts`
`deferNativeRunForBusyAgentDirectory` keeps the same run on the existing
same-run retry schedule without spending an attempt. It uses the same
guards as `deferNativeControllerRecovery`: it locks the task, requires
the task's execution run to be this run, requires the current lease
owner, and refuses after a Stop, a startup cancellation or a pending
native cancellation, so those settle normally. Its count is stored in
the run's `contextSnapshot`, which a resume claim does not clear, and it
stops after 30.
- `server/src/services/heartbeat.ts`: routes a preparation deferral to
the same-run path or, before admission, to the scheduled-retry path.
After the bound it follows the normal failure path with the plain
message.
- No schema or migration. Postgres releases a dead holder's advisory
lock with its session. The staging evidence showed a live holder
(another run's checkpoint), not a stale one.
- `doc/DEVELOPING.md`: Documented both paths and the bound.

Earlier revisions of this PR waited on the lock and added idle-session
expiry, keepalive, ownership fences, fenced remote removal and attempt
slots. This revision removes all of them.

### How the directory-wait retry keeps the original wake

The retry no longer replaces the wake. `scheduleBoundedRetryForRun`
takes `preserveWakeContext` for this wait: the retry keeps the original
`wakeReason`, `retryReason` and the whole wake context (interaction,
comment, continuation and admission fields), and its wake request keeps
the original reason. The wait is recorded separately as
`preparationWait: { kind: "agent_directory_busy", attempt, nextAttemptAt
}`, next to the carried `executionRetryAccounting`. Gates that key on
the wake reason therefore treat the retry exactly as they treated the
original wake.

| # | Site | Treatment for the directory wait | Why |
| --- | --- | --- | --- |
| 1 | `heartbeat/retries.ts` `scheduleBoundedRetryForRun` | Changed:
`preserveWakeContext` keeps the original wake context and reason and
adds `preparationWait` | Root cause: the retry overwrote the wake reason
|
| 2 | `run-dispatch/domain/wake-context.ts`
`isResolvedInteractionContinuationWakeContext` /
`PREPARATION_WAIT_RETRY_REASONS` | Special case removed | The original
wake context is kept, so the continuation is classified from it |
| 3 | `heartbeat/retries.ts` `interactionRunnerWait` | Special case
removed | The interaction fields are kept in the context |
| 4 | `run-dispatch/adapters/postgres.ts` scheduled-retry dependency
facts | Changed: a preparation-wait retry whose original wake was
admitted as a dependency-blocked interaction
(`dependencyBlockedInteraction`, recorded at wake time) is admitted the
same way | Without the wait the original wake would have run; this gate
is only met because of the wait |
| 5 | `run-dispatch/domain/wake-context.ts`
`isNonAssigneeWorkspaceBusyRetry` + `heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` | Kept:
`preparationWaitDeferredWhileAssignee`, derived only from the recorded
admission authority | The scheduled-retry ownership gate has no
wake-reason path; the workspace and AI waits use the same recorded flag
|
| 6 | `heartbeat/retries.ts` explicit user continuation admission |
Kept: admits the `agent_directory_busy` scheduling reason | The
successor needs its own authorization receipt |
| 7 | `execution-recovery-attempt.ts` uncharged lane and
`agentDirectoryWaits` | Kept | Cap and accounting |
| 8 | `pre-dispatch-execution-evidence.ts` (`isPreDispatchResourceWait`,
`isAgentDirectoryPreparationWait`), used by
`legacy-execution-recovery.ts`, `cancelled-native-startup.ts`,
`explicit-native-continuation.ts` | Kept | Describe the cancelled
pre-provider parent run, not the retry's wake |
| 9 | `explicit-native-continuation.ts`
`undeliveredLegacyUserCommentIds`, `run-dispatch/adapters/postgres.ts`
`unstartedUserTurnOrigin` | Kept | The cancelled parent did not consume
the user's comment |
| 10 | `execution-projection.ts` label and saved accounting,
`task-interruption-presentation.ts`, UI toast and chat suppression |
Kept | Presentation of the wait |
| 11 | `run-dispatch/domain/policy.ts` AI-wait lock requirement;
AI-subscription lock recheck; AI wait detection
(`execution-configuration.ts`, `provider-limit-wait.ts`, `heartbeat.ts`,
`ai-connections.ts`, `ai-connection-runtime.ts`, `local-ai-login.ts`) |
Not applicable | AI-subscription specific |
| 12 | `workspace-busy-wait.ts`, `environment-selection.ts`,
`workspace-preparation.ts`, UI workspace-holder banners | Not applicable
| Workspace holder parking; the directory wait is a timed retry |
| 13 | `cancelled-native-startup.ts` paused workspace-retry repair;
`failureRetriesBefore*Wait` | Not applicable | Historical workspace and
AI records only; every directory wait saves its ledger |
| 14 | `failedRunRetryBudgetSpent`; storage-wait exhaustion escalation;
`status-relevance.ts` `isAdmissionWait` | Not applicable | A directory
wait never fails the run, stops at its own bound, and its run has
started |

## Verification

- Original wake kinds after a directory wait
(`heartbeat-agent-directory-busy.test.ts`, real heartbeat): an
assignment runs; a plain user comment on a dependency-blocked task is
answered with `dependencyBlockedInteraction`; an accepted interaction
continuation keeps its parked-task and reassignment gates; a user
message continuation runs with its comment; an accepted non-assignee
mention runs; and a reassignment during preparation does not let the
previous assignee restart. Each retry keeps the original `wakeReason`
and records `preparationWait`. With `preserveWakeContext` off, five of
these fail. Restart recovery keeps its attempt credit
(`native-runner-restart-recovery.integration.test.ts`).

- Non-assignee wake: an accepted mention for an agent other than the
assignee meets a busy directory; its retry records
`preparationWaitDeferredWhileAssignee: false`, survives promotion, runs
and succeeds once the directory is free. Without the shared clause in
`isNonAssigneeWorkspaceBusyRetry`, promotion cancels it.

- Interaction continuations: `heartbeat-agent-directory-busy.test.ts`
runs an accepted interaction continuation into a busy directory. The
retry keeps `mutation`, `interactionId`, `interactionKind` and
`interactionStatus` and is still classified as a continuation. If the
task is moved to todo, or reassigned, while the retry waits, the retry
does not run and the task stays parked or with its new owner. With the
old explicit storage/Runner lists, both cases fail.

- Red/green, new run:
`server/src/__tests__/heartbeat-agent-directory-busy.test.ts` runs a
real heartbeat for a new assignment whose first instruction preparation
meets a busy lock. Without the preparation wrapper, the run ends failed
(red). With it, the first run is cancelled as a wait, an
`agent_directory_busy` retry is scheduled with the task still locked to
it, the retry runs and succeeds, and no run of the task is failed
(green).
- Red/green, same-run: with the deferral as a no-op, the same-run
deferral tests failed (`expected false to be true`); they now pass.
- `native-controller-recovery.integration.test.ts` covers repeated
claim-and-defer cycles of a resumed run: each cycle keeps the attempt
count, cannot be claimed before the delay, stores the count on the run,
and the fourth deferral past a bound of 3 is refused. It also covers
Stop after the busy error, a foreign lease owner and a moved task (each
refused, nothing changed), a run before admission, and that a raw busy
error (as from collection) is never deferred.
- Focused suites: 232/232 passed (`agent-directory-working-copies`,
`instruction-working-copy`, `heartbeat-agent-directory-busy`,
`native-controller-recovery.integration`, `heartbeat-retry-scheduling`).
`tsc --noEmit` for `server` reports no errors in the changed files.
- Staging evidence (read-only): the failed run had a graceful restart at
07:25Z, then same-run resumes. Its third attempt failed at 07:31:08Z.
Another run of the same agent recorded a warm checkpoint receipt at
07:31:37Z. A second run of the same agent failed the same way at
07:59:46Z.

## Risks

- A deferred run waits 5 to 15 seconds per deferral, up to 30 times. If
the holder is still busy after that, the run fails as before with a
plain message.
- Before admission, deferral cancels the run and creates a retry run,
like the existing workspace-busy wait.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:56:45 -06:00
DottaandClaude Opus 5.5 7e8901aa60 fix(recovery): let Retry and the person's answer continue a held task with a pending decision (#20872)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can lose its Runner process.
Recovery then saves the sandbox files and puts an execution hold on the
task
> - The hold offers Retry. A person can also continue the task with a
new message
> - Many held tasks also have a pending decision: the question that a
parked turn asked, or an older connection request
> - Continuation admission refuses every continuation while a decision
is pending ("decision_pending"). The answer to that decision is not a
message, so admission defers it behind the hold ("execution_recovery")
> - So Retry waits for the answer and the answer waits for the hold. The
task shows a Retry that does nothing, and nothing continues it
> - This pull request lets a person's Retry pass an open question, and
lets the person's own answer continue the task through the hold
> - The benefit is that a task with a lost Runner and a pending decision
always has a control that works

## Linked Issues or Issue Description

**What happened?**
On staging, the same-run retry of six remote native runs found the
Runner process gone (`process_not_running`, lifecycle `ready`). Each run
failed with `runner_remote_process_identity_unavailable`. Saved-work
recovery then recovered the sandbox files, and the task showed "Saved
work recovered. Retry or send a message to continue." with `canRetry:
true`. Four of these tasks also had a pending decision: the question
that the parked turn asked, or an older pending connection request. For
those tasks, Retry is deferred with `decision_pending`, and an answer to
the decision is deferred with `execution_recovery`. No control continues
the task.

**What did you expect to happen?**
Retry starts one new run. The person's answer to the pending decision
also continues the task.

**Steps to reproduce**
1. A remote native run asks the user a question and parks the turn.
2. The Runner process exits. The same-run retry fails with
`runner_remote_process_identity_unavailable`.
3. Saved-work recovery recovers the files. The task offers Retry.
4. Click Retry: the Retry waits with "A pending approval or question
must be resolved before this message can start."
5. Answer the question: the answer waits with "Waiting for execution
recovery. Your message is saved."

Refs #20838, #20853

## What Changed

- `admitExplicitNativeContinuation`
(`server/src/services/explicit-native-continuation.ts`): a task Retry is
no longer held by a plain pending question (`ask_user_questions`) or
connection request (`connection_intent`). Any interaction with approval
semantics still holds Retry: a plan review (`issue_document` target), a
governed tool action, a secret proposal, a connection authorization, and
other confirmations. A pending approval still holds every continuation.
A new message supersedes a plain question, as on master, and waits for a
decision with approval semantics.
- `admitExplicitNativeContinuation`: the person's own resolution of an
interaction (`mutation: "interaction"`, answered/accepted/rejected,
resolved by the same user) is explicit user input. It continues the held
task like a new message: a fresh session, the hold resolved as
`explicitUserContinuation` with the `interactionId`, and
`issue.execution_recovery_settled` activity. It never replays the failed
run. Governed tool-action confirmations are excluded. The existing
`readQueuedInteractionResponse` reads the answer.
- `heartbeat/queue.ts`: both admission calls pass the wake payload to
the admission.
- `buildExecutionContinuation`
(`server/src/services/execution-continuation.ts`): dispatch rechecks the
user authorization. It accepted an interaction receipt only through a
saved interrupt queue, so the answered successor failed with
`continuation_user_authorization_missing`. It now also accepts the
person's own answer: the interaction named by the durable authorization
was resolved by that user and not by an agent, and the authorization
names this successor run. The check does not depend on the run's current
trigger, so a later answer that replaces the trigger cannot invalidate
it.
- New API-level test
`server/src/services/governed-wait-runner-loss.test.ts` with the two
staging shapes (a parked question from the failed run; an older
connection request on a monitor wake). It checks that Retry starts
exactly one run and leaves the decision pending, that the person's
answer starts exactly one run and settles the hold, and that an answer
resolved by a different user keeps the hold.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- `cd server && pnpm exec vitest run
src/services/governed-wait-runner-loss.test.ts`: 5 passed. The answer
tests now also build the successor's continuation as dispatch does and
check that the agent receives the answer.
- Red for dispatch (first head, ac17aebe): both answer tests fail with
`continuation_user_authorization_missing`.
- Ponytail follow-ups: a two-answer test (the second answer waits behind
the successor; dispatch also holds with a replaced trigger) and 'keeps
Retry on hold while %s is pending' for four approval-bearing kinds. On
head 60beb773, 5 of these fail. Two more tests show that a new message
supersedes a plain question and waits for a plan review (both unchanged
from master). All 12 tests in the file pass now. The PR adds no code
comments; the rationale is in `doc/execution-semantics.md` and here.
- Continuation suites (execution-continuation,
execution-continuation-provenance, legacy-continuation-authority,
explicit-native-continuation, task-retry, governed-wait-runner-loss):
634 passed.
- Red on master (product hunks removed, test kept): 4 of 5 fail. Retry
tests: `successors: 0`, deferred `decision_pending`. Answer tests:
`successors: 0`, deferred `execution_recovery`. Every case asserts
`runError: "runner_remote_process_identity_unavailable"`.
- Related suites: explicit-native-continuation, task-retry,
workspace-recovery-admission, retry-workspace-continuity,
workspace-recovery-retry-copyback, workspace-recovery-fallback,
native-workspace-reconstruction, native-recovery-preparation-failure,
issue-thread-interactions-service. 957 of 959 passed. The 2 failures are
30 s `afterEach` hook timeouts in native-workspace-reconstruction; all
51 of its tests pass alone, and the same file on unchanged master shows
5 hook timeouts on this machine.
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json`: passes.

## Risks

- Medium. A Retry can now start while a plain question or connection
request is pending; decisions with approval semantics still hold it. The
new run sees the open decision in its task context, and the decision's
answer still wakes the agent later. Approvals keep holding Retry, so
approval gates are unchanged.
- An answer now continues a held task in a fresh session. All other
admission gates still run: stop proof, saved-work recovery, workspace
repair, active execution, pause, budget and ownership. Only the user who
resolved the interaction can continue this way.
- This does not make recovery automatic. A task whose Runner was lost
still rests on the hold until a person clicks Retry or answers.
Automatic settlement of a parked governed wait is a separate change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:42:36 -06:00
Nicky LeachandPaperclip b2edea5754 fix(workspaces): verify existing worktrees before fresh retries (#20876)
## Thinking Path

> - Paperclip manages work for AI agents.
> - Retry checks whether the task can keep its existing workspace.
> - A provider can complete checkout before Core receives the result.
> - Recreating a completed worktree costs time and can change its base
commit.
> - Core uses the same ownership and writer checks for local and
provider workspaces.
> - The provider verifies its checkout before Core creates a fresh
workspace.

## Linked Issues or Issue Description

**What happened?**

A retry can allocate a new provider worktree even though the original
checkout completed.

**Steps to reproduce**

Complete workspace checkout through a storage provider. Fail Runner
startup or lose the prepare reply. Retry the task.

**Expected behavior**

Reuse the same worktree when the provider confirms checkout and the
prior run can no longer write to it.

Related: #20865 addresses timeouts while the original prepare is in
progress. This PR checks an earlier checkout when the user retries.

## What Changed

- Check the prior task binding and writer state before asking a storage
provider to verify a checkout.
- Extend the existing prepare operation with an optional verification
mode. Providers opt in through their manifest.
- Return through normal workspace reuse after verification. Keep the
fresh retry fallback when verification cannot establish checkout.

## Verification

- Core workspace preparation, plugin workspace ownership, retry
continuity, and SDK workspace-provider suites passed locally (213
tests).
- Focused cases pass for provider verification, retry through heartbeat
dispatch, and real local Git checkout verification with no plugin
worker. The heartbeat test confirms that ordinary preparation and
adapter dispatch keep the original workspace, branch, and base commit.
- Core server typecheck passed before the final audit and timing edits.
GitHub CI covers the final revision and broader regressions.

## Risks

Provider verification adds one bounded call before a fresh retry. Older
providers keep their current behavior. Verification must preserve the
workspace, branch, and recorded base commit. The normal execution
preparation checks still run before admission. No database migration is
needed.

## Model Used

OpenAI GPT-6 (Codex), with source inspection and local test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal issue
details
- [x] I have run targeted tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 18:04:18 +00:00
DottaandClaude 889b33da72 fix(native): apply a saved accepted result on Retry instead of a new turn (#20819)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can finish with an accepted result,
and then fail to save its workspace. Saved-work recovery saves the files
again from the retained sandbox, with no provider turn.
> - #20758 made a save that finds the sandbox disk full stop at once as
`sandbox_disk_full` and hand the task to the board with a Retry.
> - When the user frees space and presses Retry, the save now succeeds.
But the Retry then started a new agent turn (`strategy: continue`). That
turn repeats work that already finished and spends provider budget
again. The accepted result was not applied (`replacementDenied:
explicit_user_continuation`).
> - The decision for this case: when a durable record proves that the
run's result was already accepted, Retry applies that saved result and
starts no new turn.
> - This pull request makes Retry admission wait for the run's
saved-work holds and then applies the accepted result through the
existing preserved-result finalization.
> - The benefit is that a Retry after a failed save finishes the task
from the work that was already done, with no repeated provider work or
spend.

## Linked Issues or Issue Description

Refs #20758 (full-disk save rests as `sandbox_disk_full` with a Retry),
Refs #20810 (chaos fault DF1 for a full disk on the final workspace
save)

**What happened?**
A native run had an accepted, succeeded result. Its workspace save
failed because the sandbox disk was full, and the task rested at
`sandbox_disk_full` with a Retry. The user freed space and pressed
Retry. The save succeeded, and then Retry admission started a new run
with `strategy: continue`. The accepted result stayed unapplied (the
finalization coordinator stayed in `terminal_failure`).

Two paths caused this:
1. Retry admission let the Retry through as soon as the saved files were
verified. That can happen before preserved-result finalization runs: the
recovery worker dispatches the Retry right after one hold settles, while
the hold that Retry added for the same run is still scheduled. On master
this is deterministic in that order, and a race otherwise.
2. Preserved-result finalization reads the newest saved-work hold of the
run. While that hold was still scheduled, it declined, and nothing held
the Retry back.

**Expected behavior**
After the save succeeds, and a durable record proves that the run's
result was already accepted, Retry applies that result. The run and task
finish from it. No new `turn.start` and no new provider spend. If the
save fails again with a full disk, the task keeps the
`sandbox_disk_full` rest and a working Retry. If there is no proven
accepted result, Retry keeps the previous behaviour and starts one run.

**Steps to reproduce**
1. Give a native run an accepted, succeeded result whose copy-back
failed (`native_workspace_sync_out_retry_exhausted`).
2. Make the saved-work recovery save fail with
`remote_workspace_sandbox_disk_full`. The hold goes to the board.
3. Free space. Press Retry. Let recovery save the files, then dispatch
the Retry before preserved-result finalization runs.
4. On master, a new run starts with `previousRunId` set to the source
run, and the accepted result is not applied.

## What Changed

- `preserved-result-finalization.ts`: the eligibility checks move into
one read (`readPreservedResult`). They are unchanged: an accepted,
succeeded result of this run, a completed saved-work recovery with a
verified receipt, no Stop, no newer run, no current execution, and no
applied generation. A new read-only
`canResumePreservedResultFinalization` uses the same read without locks.
`resumePreservedResultFinalization` keeps the finalizer's lock order
(coordinator first) and takes the issue and run rows with NOWAIT, so it
cannot deadlock with the finalizer, owner-stop confirmation or Retry
dispatch; a busy row is retried briefly, and the last attempt waits for
its locks. It also writes an activity entry,
`issue.saved_result_finalization_resumed`, when it applies a result.
- `explicit-native-continuation.ts`: Retry admission holds an accepted
Retry (`evaluate`) of a run whose saved files are verified and whose
accepted result is still held, with the wait `finalizing_saved_result`.
It holds while a saved-work hold of that run is still `scheduled`,
`running` or `awaiting_evidence` and the result is otherwise applicable
(no Stop, newer run, other owner or current execution), or once
`canResumePreservedResultFinalization` is true. In every other case
admission is unchanged.
- `heartbeat/run-control.ts` (`resumeTaskRetry`): after dispatch, if
admission held the Retry with `finalizing_saved_result`, it calls
`resumePreservedResultFinalization`. That coalesces the Retry into the
source run with `strategy: finalize_result` and re-arms finalization.
This is one key read after admission; the resumption runs only for a
held Retry. The recovery worker already calls the same function after
each saved hold, so either path finishes the work.
- Tests: `workspace-recovery-disk-full.test.ts` now proves the new Retry
behaviour (red on master), a dispatch that lands while another hold of
the run is still saving (red on master), a Retry while the disk is still
full, and three negative cases that keep one new run.
`workspace-recovery-preparation-gates.test.ts` (s12 and RC-50b) and
`explicit-native-continuation.test.ts` (copy-back + `task_retry`)
encoded the old "new run" result for the same situation. They now expect
the saved result to be applied. A fixture helper `recoverRunHolds` runs
the recovery steps for every hold of a run, as the worker does.
- `ui/src/components/WorkspaceExportRecovery.tsx`: at the
`sandbox_disk_full` rest, the retained-file panel no longer shows its
own "Recover saved files" button and no longer hides the task's Retry.
Before, the API said `canRetry: true` and the note said "Retry saves the
files again", but the board showed only "Recover saved files" (seen in
the DF1 chaos wave). The panel now says to free space and use Retry. A
new UI test covers it. `pnpm check:token-gates` is clean.
- `doc/execution-semantics.md`: documents the Retry behaviour and its
conditions.

## Verification

Red/green (`cd server && npx vitest run
src/services/workspace-recovery-disk-full.test.ts`):

- On master source with the new tests: 3 failed, 3 passed (the 3
negative cases pass on master, as they should). Each failure shows a
second heartbeat run (the new agent turn): `expected [ …(2) ] to deeply
equal [ Array(1) ]`.
- On this branch: 7 passed. A fourth negative case (an operator Stop
while a hold is still settling keeps one new run) was added after
review; it also passes on master.
- Guard removal 1: with the `canResumePreservedResultFinalization` check
replaced by `true`, the negative test "an operator Stop recorded on the
run" fails (`expected [] to have a length of 1 but got +0`: the Retry
waits forever instead of starting its one run).
- Guard removal 2: with the "hold still saving" check removed, the test
"holds a Retry dispatched while another saved-work hold of the run is
still settling" fails (a second run starts).

Related suites pass on this branch: `workspace-recovery-disk-full`,
`workspace-recovery-preparation-gates`, `workspace-recovery-admission`,
`native-exhausted-finalization-retry`, `explicit-native-continuation`
(641 tests).

On the first revision, `task-retry`,
`workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`,
`preserved-native-response-wait.integration`, `deferred-execution-wait`
and `deferred-execution-receipt-work` passed (317 tests). Other retry
and recovery suites were not run locally; required CI `general_tests`
covers them. After the review fixes, the five related suites above
passed again (642 tests).

`npx tsc --noEmit -p server/tsconfig.json` is clean.

`pnpm verify --explain` selects a broad plan (exit 2) because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

## Risks

- Behaviour change: an accepted Retry of a terminal run with an
accepted, succeeded result and verified saved files now applies that
result and starts no run. Before, it started one new run. This is the
intended change. It also covers the s12 and RC-50b recovery-handoff
cases, which used to start a new run.
- A held Retry waits while any saved-work hold of the run is still
saving. The recovery worker drives those holds to `complete` or
`exhausted`. When a hold ends `exhausted` or the records do not prove
the result, the wait ends and the Retry starts one run, as before. A
hold that stays in `awaiting_evidence` keeps the Retry waiting with
"Paperclip is finishing the saved result."
- Upgrade-safe: no persisted shape changes. The checks read existing
records (`native_run_results`, `native_run_finalizations`, the recovery
action evidence, workspace receipts). A record written by an older
server that lacks any required fact keeps the old behaviour. An older
server that reads the new state sees a coalesced Retry and a re-armed
finalization, which it already handles.
- No new work before `turn.start` on the normal path. Admission adds the
eligibility read only when it has already proven an accepted result and
verified saved files for a Retry of that run. `resumeTaskRetry` adds one
key read after admission.
- Lock order: `resumePreservedResultFinalization` keeps the finalizer's
order (coordinator, then issue, then run). Because it can now run beside
Retry dispatch (which locks the issue first), it takes the issue and run
rows with NOWAIT. A busy row ends the attempt, and it retries up to 9
times at 100 ms. The 10th attempt waits for its locks, as this
resumption always did, so a recovered result is never left unapplied.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 10:03:53 -06:00
DottaandClaude 805531cc71 fix(native): never leave a cleanup-gated task without Retry or an owner (#20639)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs close their provider session at the end of each attempt.
When close cannot prove that the runner stopped, the runner keeps a
cleanup gate (`native_session_cleanup_quarantined`) in server memory
> - #20572 hid Retry for every gated run and said "Paperclip finishes
the cleanup automatically". Only a remote environment release settles
the gate. A local gate that needs an operator never settles, and a
server restart loses it
> - A restart-chaos wave on master found two tasks that rested blocked
with no Retry, no scheduled retry and no recovery owner. A person could
not continue them
> - This pull request makes the run's own recovery retire the gate when
the host proves the old processes stopped. It also offers Retry whenever
nothing settles the gate, and it makes that Retry work
> - The benefit is that a cleanup-gated task either continues by itself
or shows a Retry that starts. It never rests without both

## Linked Issues or Issue Description

Refs #20572, #20578

**What happened?**
Restart-chaos wave `w-20261110T185144-s9107` (local, claude, fault TM2:
the server is paused with SIGSTOP for about 33 s). Tasks t04 and t05
completed their provider turn. The runner checkpoint owner lapsed while
the server was paused (`native_runner_checkpoint_owner_lost`). The
session close then failed with `native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`. The provider had
drained and no tool call was pending. The runner kept an
operator-required cleanup gate. Thirty seconds later the run's own
same-run recovery verified that the runner process and group had stopped
(`native.local_process_stopped`). The gate still refused the recovery,
and the run ended `failed` with `native_session_cleanup_quarantined`.
The task showed `canRetry: false` and a cleanup wait until the deadline.
Nothing owned that wait.

**Expected behavior**
When the old runner and provider processes are proven stopped, the run's
own recovery continues. When nothing settles the gate, the task offers a
Retry that starts.

**Steps to reproduce**
1. Run restart-chaos L7: `--env local --adapters codex,claude
--workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1
--concurrency 3 --tasks 6 --seed 9107`.
2. Observe a claude TM2 task rest `blocked` with `canRetry=false`,
`retryWait=true` and no scheduled retry.

**Paperclip version or commit**
master at ed3ffc3bc7.

## What Changed

- `native-session-executor.ts`: add `provenStoppedOwnNativeSession`. It
returns proof only for a local run with a pid-bound
`native.local_process_stopped` record (no later launch) and with every
recorded provider process gone. An ACPX session open names no provider
PID; its sidecar counts only when it shares the runner's process group,
which the stop record proves empty.
- `native-session-executor.ts`: add
`retireStoppedOwnSessionCleanupGate`. With that proof, it retires the
run's own operator-required gate through the existing
`completeTerminatedLocalNativeSessionCleanup` boundary and records
`native.session_cleanup_gate_retired`.
- `heartbeat.ts`: the native recovery preamble calls it right after it
records the local process stop, before admission. The recovery decision
itself does not change.
- `explicit-native-continuation.ts`: Retry or a new message accepts the
run's own stop proof when the retained-state inventory proof does not
apply. It still starts a fresh session, and it never resumes the old
turn.
- `task-interruption-presentation.ts`: hide Retry for a gated run only
while a live owner settles the gate: an unreleased remote environment,
or a scheduled close in this process. Otherwise offer Retry. The same
rule applies to a Retry held on `cleanup_quarantined`.
- `doc/execution-semantics.md`: describe the new rule.

## Verification

- Red before the fix, green after (5 tests failed on master sources):
- `server/src/services/task-interruption-presentation.test.ts`: "offers
Retry when nothing settles the gate by itself", "holds a Retry on a
cleanup gate only while an owner settles it", and an invariant test: a
failed presentation without Retry must name a wait, and a cleanup wait
needs a settling owner.
- `server/src/__tests__/task-interruption-query.test.ts` (real
Postgres): an unreleased remote lease hides Retry; the released lease
offers it.
- `server/src/services/quarantined-task-retry.test.ts`: "starts a task
Retry for a run's own quarantine proven stopped by its own process
record" (stopped, and provider alive stays held).
- New: "retires a run's own failed-close gate once its runner and
providers are proven stopped" for codex PIDs, an ACPX sidecar in the
runner's group, a sidecar with its own group, a live provider, and no
stop record.
- `npx vitest run` of `quarantined-task-retry`,
`task-interruption-presentation`, `task-interruption-query` and
`explicit-native-continuation`: 575 passed.
- `npx tsc --noEmit -p server` passes.
- Live: LIVE_PROOF_PLACEHOLDER

## Risks

- The gate retirement removes process ownership only. It runs only for a
local run whose exact runner and group and every recorded provider
process are proven gone. A provider without a recorded process identity
keeps the gate.
- Retry on a gated run still runs every existing stop check. If the stop
cannot be proven, the Retry waits and the task keeps a Retry, instead of
a wait that nobody settles.
- No persisted shape changes. Runs saved by earlier releases use the
same fields.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 15:20:42 -06:00
DottaandClaude bf45700ca3 fix(native): make Retry work after a turn start cut off once Codex recorded the turn (#20516)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run sends `turn/start` through its runner. A
control-plane restart can kill the runner and Codex while that request
is in flight
> - #20507 lets the runner resume by itself when Codex's thread history
proves the turn was never created. When Codex did record the turn, the
turn may have done work, so the runner still fails closed and nothing
replays it
> - That branch must still end in a Retry that works. The next run on
the same conversation is refused at the runner-state fence before it
asks for a runner. #20507 let Retry continue after that run by matching
the refusal's error text
> - Driving that branch end to end with the real runner exposed a second
dead end. When the owner's runner dies, its provider-session close
cannot prove suspension, so this server process can keep an
operator-required cleanup quarantine for the task. Retry was then
refused outright ("Send a new message")
> - This pull request replaces the text match with a server-authored
run-log event, lets Retry take the same verified-stop path as a new
message, and adds a real-runner test of the whole branch: the cut-off
owner fails closed, the follow-up is refused with structured proof,
Retry is admitted, and the fresh Retry run completes
> - The benefit is that this restart window cannot strand a task, and
the Retry gate no longer depends on error wording

## Linked Issues or Issue Description

**What happened?**
Follow-up to #20507 (restart-chaos wave `w-20261110T033426-s108`, t03).
#20507 left one item open: prove that a person's Retry produces a
working run when the restart cut off `turn.start` after Codex recorded
the turn. It also matched the fence refusal by its error text.

**Expected behavior**
After that cut-off, the owner stays failed closed and the turn is never
started again by itself. A person's Retry starts a new run that
completes. It never fails with `quarantine_owner_unverified` and never
returns 202 with no run.

**Steps to reproduce**
1. Run `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-cut-turn-start-retry.integration.test.ts`.
2. Without the new event, the test fails: the refused follow-up has no
proof that it never launched, so Retry after it would wait on
`process_identity_missing`.

## What Changed

- `native-session-executor.ts`: when the runner-state fence refuses a
run before launch (`prior_owner_*` or `legacy_prior_owner_*`), the
executor records the server-authored event
`native.prior_owner_fence_refused` with the closed refusal `reason`,
then fails the run as before. A failed write leaves the run held (the
safe side).
- `native-local-process-stop.ts`: `nativeRunNeverLaunched` accepts a
failed run that has this event (and still no launch, identity,
`adapter.invoke` or runner-sourced event). It no longer reads the error
text.
- `explicit-native-continuation.ts`: when the task's in-process cleanup
quarantine is operator-required, a Retry no longer returns
`cleanup_quarantined` at once. Like a new message, it must first pass
`verifyStoppedNativeSessionForContinuation`, which releases the
quarantine and records `native.stopped_conversation_verified`. A native
Retry always starts a fresh conversation (`forceFreshSession`), so it
never resumes the interrupted turn. If the proof fails, Retry still
returns `cleanup_quarantined`.
- `verifyStoppedNativeSessionForContinuation`
(`native-session-executor.ts`), three narrow additions:
- a run with `native.prior_owner_fence_refused` may stand in for the
saved state's owner, like a run refused by the cleanup quarantine (both
were refused before launch);
- when the session root is gone, it reads the fence's single
`identity_indeterminate` quarantined copy of the owner's state (several
copies stay ambiguous);
- a provider startup attempt in phase `initialization_failed` whose
direct child exited and whose process and process group are gone no
longer blocks the proof. A same-run restore that fails closed leaves
exactly this record. Those PIDs are also re-checked before the
quarantine is released.
- New real-runnerd integration test
`native-cut-turn-start-retry.integration.test.ts` (real runnerd and fake
Codex with `--hang-turn-start after`):
1. The owner run's `turn/start` hangs after Codex recorded the turn; the
runner and Codex are killed. The owner's provider state keeps the
ambiguous start, and the thread history keeps the turn.
2. The same-run resume restores the runner and fails closed with "cannot
safely recover an ambiguous Codex turn start". Codex saw exactly one
`turn/start`.
3. A follow-up on the same conversation is refused with `prior_owner_*`,
never `quarantine_owner_unverified`. It records
`native.prior_owner_fence_refused`, launches nothing, and
`nativeRunNeverLaunched` is true for it.
4. The owner's operator-required in-process cleanup quarantine is pinned
(whether the real close lands there is timing-dependent; about half of
the local runs did).
5. Retry of the refused follow-up passes
`admitExplicitNativeContinuation` (not a dry run) with no block reason,
releases the quarantine and records
`native.stopped_conversation_verified`.
6. The Retry run, prepared with a fresh conversation as task Retry does
(`forceFreshSession`), uses a different runner-state root and completes
with exit code 0. Codex saw one new `thread/start` and one new
`turn/start`; the cut-off turn was never started again.
- `explicit-native-continuation.test.ts`: the fence rows now insert the
event. A new row shows that the refusal text alone keeps the run held.
The "worker loss (retry)" case now expects a verified Retry to be
admitted like a new message.
- `native-recovered-owner-followup.test.ts`: a fixture event moves to a
free sequence number, because the refused follow-up now writes its own
event.
- Docs: `doc/run-log-events.md` (new event) and
`packages/paperclip-runner/docs/durable-recovery.md`.

## Verification

- Red/green:
- Without the event write, the integration test fails (no fence event,
so the follow-up is not proven unlaunched).
- Without the gate and proof changes, it fails with `expected [
'cleanup_quarantined' ] to deeply equal []`: Retry is refused while the
owner's quarantine is held.
  - With all changes, it passes.
- `vitest run` of `native-recovered-owner-followup`,
`explicit-native-continuation` (427 tests), `quarantined-task-retry`,
`issue-confirm-run-stopped-routes`, `native-finalization-recovery`,
`task-interruption-query`, `execution-projection`,
`chat-run-publications`, `closed-native-receipt-accounting`,
`agent-live-run-routes` and the new integration test pass.
- Server `tsc --noEmit` passes.
- Runner code and the pack `contractDigest` do not change.

## Risks

- Low-medium risk. The new event is written only on the fence refusal
path, which already fails the run. It is local run-log data (no
Telemetry or OpenTelemetry).
- Retry after an operator-required cleanup quarantine is a behaviour
change: it was always refused, and now it is admitted only after the
same exact stop proof a new message needs. That proof checks the owner's
runner stop record, every recorded provider and startup PID and process
group, and the unchanged state fingerprint before it releases the
quarantine.
- A run refused by a build between #20507 and this change has only the
error text. It stays held on `process_identity_missing` until a person
sends a new message or the board confirms the stop. #20507 has not
shipped in a release yet.
- The test models the restart in-process: it kills the runner and Codex
and calls the executor again. It does not drive the HTTP Retry route;
the route's own behaviour (fresh conversation, admission gate) is
covered by existing tests and by `nativeRunNeverLaunched` here.
- Open decision, unchanged: a turn Codex recorded before the restart is
never replayed or adopted. Continuing is a person's decision.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20507

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 06:16:42 -06:00
DottaandClaude 52b906b112 fix(native): let Retry continue after a run that failed before native selection (#20482)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before Paperclip starts a new message or a Retry after a failed
native run, it proves that the run's local process stopped
> - A native run can fail before native selection, for example when a
budget reservation is refused or a preparation query fails. That run
never had a runner, a native session or a process ID
> - The stop gate still waited for a stop record of that process. Retry
returned 202, then the task stayed on
`retryWait.reason=process_identity_missing` and no run started
> - Native selection writes the run's native issue, native session,
runner instance and coordinator in one transaction. A runner launch
needs all of them, and the executor records
`native.process_start_requested` before any spawn
> - This pull request treats a settled run with none of that evidence as
having nothing to stop
> - The benefit is that Retry and new messages continue after a failure
that happened before the agent started

## Linked Issues or Issue Description

**What happened?**
In local restart-chaos database-load wave `w-20261110T050818-s20386`,
all 4 tasks failed before any agent process started. One example is t01,
run `f7e2ee55`. A preparation query failed, and the run ended
`adapter_failed` with no native session, no runner instance, no process
ID and no launch event. Retry returned 202. For 16 or more minutes,
`retryWait` then stayed on `process_identity_missing` and no run
started.

**Expected behavior**
If the previous run never had a process, there is nothing to verify, so
Retry and new input continue.

**Steps to reproduce**
1. Make a native run fail before native selection (for example, a
refused budget reservation or a failed query).
2. Click Retry, or send a message. Before this change, the input waits
on `process_identity_missing` and does not continue.

## What Changed

- `native-local-process-stop.ts`: `nativeRunRefusedBeforeLaunch`
(renamed `nativeRunNeverLaunched`) now also proves "never launched" for
a settled native run in this state:
  - no process ID and no process group;
- no native issue, no native session and no runner instance (native
selection writes them together). A coordinator row requires the native
issue by foreign key, so the native-issue check also rules out a
coordinator;
- no launch request, no identity record, no `adapter.invoke` and no
runner-authored event (any event with a source event ID).
The existing case, a run that the cleanup quarantine refused before
launch, does not change.
- `explicit-native-continuation.test.ts`: the guard "failed for another
reason without a process record" now separates real pre-selection
failures from the shapes that must stay held.
- Admitted: a run that failed before it requested a launch. This case is
red on `master`.
- Still held on `process_identity_missing`: a run that has a runner
event, a run with a prepared native session, and a run with a
coordinator, for `failed`, `timed_out` and `interrupted` runs. A
`timed_out` run that failed before launch is admitted. The chat suite's
`unreconciled` case (a native issue and phase, no session) also stays
held.

## Verification

- Red/green: on `master`, `routes input after a run failed before it
requested a launch` fails with `process_identity_missing`. It passes
with this change.
- `vitest run src/services/explicit-native-continuation.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/issue-confirm-run-stopped-routes.test.ts`: 3 files, 462
tests pass. This includes every existing "keeps the hold for
missing_process" and historical-run case.
- Server `tsc --noEmit` passes.

## Risks

- Low risk. The new proof needs structural evidence that no runner could
exist: no native selection and no launch or runner event. A run that
reached native selection, or that has any launch evidence, keeps every
existing stop proof.
- The proof does not depend on the build: a runner from any build needs
the native session and the runner instance. Saved runs from older builds
that were selected keep those fields, so they stay held exactly as
before.
- Relates to #20477, which fixes the restart window inside an answer
run's attachment. This pull request is independent of it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #19829

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:20:31 -06:00
DottaandClaude Opus 5.5 87e755ce8e fix(recovery): verify cross-task warm sandbox handoffs and say plainly when recovery refuses (#20282)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes keep a warm sandbox and hand it to
the next run through a chain of reusable leases
> - Acquisition binds that chain to the execution workspace, so a task
can inherit a warm sandbox from an earlier task in the same workspace
> - Saved-files recovery verifies the same chain, but it required every
earlier lease, run, and finalization to belong to the recovering task
> - A valid chain that started with another task therefore always failed
with `allocation_ownership_ambiguous`. That code is `service_required`,
so recovery stopped after one attempt, and each board Retry repeated the
same refusal
> - This pull request makes the chain check match the acquisition rule,
and it checks each predecessor against its own task
> - It also makes the board say plainly when recovery refuses the saved
state, because "failed after automatic retries" was not true and Retry
looked like it did nothing
> - The benefit is that a task blocked with "Saving the agent's files
failed after automatic retries" can recover its saved files, and a
refusal that Retry cannot change is shown with its reason

## Linked Issues or Issue Description

**What happened?**
A native run in a remote sandbox finished its turn, and its result was
accepted. The copy of its files back to the host failed and was
scheduled to retry. A graceful server restart then interrupted the
retry, so the run settled with
`native_workspace_sync_out_retry_exhausted`, and the task became
blocked. Every saved-files recovery generation then failed on its first
attempt with `allocation_ownership_ambiguous` (`service_required`). The
board showed "Saving the agent's files failed after automatic retries.
Retry to try again". Each Retry replaced the generation
(`issue.recovery_generation_replaced`), and the new generation failed in
the same way within one second. To the user, Retry did nothing. The
sandbox lease chain was valid: same agent, same execution workspace,
same sandbox, and matching handoff receipts on all eight leases. But its
first three leases belonged to an earlier task in the same workspace.

**Expected behavior**
Recovery must accept a warm sandbox chain that acquisition accepted.
Acquisition binds reuse to the execution workspace, and it requires the
same task only when the lease has no workspace. Recovery must check each
predecessor run and finalization against that predecessor's own task. It
must still refuse a chain whose records disagree or that has no shared
workspace.

**Steps to reproduce**
1. Run a native task in a remote sandbox with `reuse_by_environment`,
with an execution workspace that another task also uses.
2. Run the other task in the same workspace first, so the sandbox lease
is handed from that task to this one.
3. Make the last run of the second task fail to copy its files back (for
example, restart the server during the copy-back retry).
4. Wait for saved-files recovery, or click Retry. Recovery fails with
`allocation_ownership_ambiguous`, and Retry repeats the failure.

**Paperclip version or commit**
`master` at a3a25ce6c9 (the same code as the 2026.1106.0 staging build
for every file this change touches).

## What Changed

- `retiredReusableLeasePredecessors` (`environment-lease-handoff.ts`)
requires the same task only when the lease has no execution workspace.
This matches the binding that `acquireLease` uses when it replaces a
reusable lease.
- `inspectWorkspaceRecoverySource` (`workspace-recovery-source.ts`)
checks each predecessor run and its finalization against the predecessor
lease's own task, not the recovering task. A predecessor from another
task is accepted only when both leases have the same execution
workspace. All other checks stay: same agent, terminal run, no live
controller or lease owner, a settled finalization, and a saved snapshot
at an older epoch on the same allocation, workspace, and remote
directory.
- The other callers of the chain walk (remote termination resolution and
retired backup restore) get the same rule.
- The test fixture `warmHandoffSource` gets an option that starts the
chain with an earlier task in a shared workspace.
- Honest copy for exhausted recovery (`exhaustedSystemRecoveryAction`):
- A deterministic refusal (`service_required`) now says "Paperclip
refused to recover the saved files", names the reason in plain words and
the code, and says that Retry runs the same check again and succeeds
only after Paperclip is updated.
- Other exhausted recovery says "stopped after N attempts" only when
there were several attempts. One attempt says "failed".
- Continuation (`explicit-native-continuation.ts`) reports a new wait
reason, `saved_work_recovery_refused`, when the exhausted recovery is a
deterministic refusal. The task presentation shows its own message for
it. `saved_work_recovery_failed` no longer claims automatic retries.
- Retry stays available for a refusal. It is the action that runs
recovery again after an update. After each Retry, the board shows the
same refusal and its reason, not an unchanged generic notice.

### Recovery refusal classes

| Retry class | Failure codes | Automatic recovery | Board Retry |
| --- | --- | --- | --- |
| `retry_transient` | `provider_unavailable`, `export_interrupted` and
other transient codes | Retries with backoff, up to 5 retries in 30
minutes | Starts a new recovery round |
| `recheck_evidence` | `agent_stop_unconfirmed`,
`workspace_writer_active`, `effect_unresolved`,
`copyback_stop_unconfirmed` | Checks the evidence again later | Starts a
new round (the copy-back stop case has its own wait text) |
| `use_alternative_strategy` | `historical_source_missing`,
`source_unavailable` (no recorded source) | Stops. The retained-source
fallback can continue from the last save | Starts a fresh attempt |
| `service_required` (deterministic) | `allocation_ownership_ambiguous`,
`source_binding_invalid`, `snapshot_verification_failed`,
`archive_unsafe`, `source_unavailable` (recorded source gone) | Stops
after one attempt. The same saved state gives the same result | Runs the
same check again. The board now says this and names the reason |

## Verification

- Red before the fix: `recovers saved work after 1|2 warm handoffs that
began with another task in the same workspace` failed with `expected
'allocation_ownership_ambiguous' to be null`.
- Green after the fix: `npx vitest run
src/services/workspace-recovery-warm-handoff.test.ts
src/services/workspace-recovery-source.test.ts` passed (60 tests).
- Safety cases that stay fenced: a predecessor lease that names a
different task than its run (`lease_task`), and a cross-task predecessor
with no shared workspace (`unshared_workspace`). The existing `task`
fault (another task with no workspace) also stays fenced.
- Sibling and retry suites passed: environment runtime and orphan
sandbox, cancelled workspace preservation, workspace recovery fallback,
task retry, remote workspace lifecycle, explicit native continuation,
native cleanup admission, and remote retry admission.
- Copy, red before and green after: `system-recovery-state.test.ts`
(`states a deterministic refusal plainly and never claims retries that
did not happen`), and the end-to-end `names a deterministic saved-files
refusal behind Retry when no fallback can continue, and another Retry
shows it again` in `workspace-recovery-preparation-gates.test.ts`. On
the old code the wait reason was `saved_work_recovery_failed`. With the
change it is `saved_work_recovery_refused`, with the refusal reason, for
both Retry clicks.
- Updated expectation: the board handoff in
`workspace-recovery-retry-copyback.test.ts` now expects the refusal text
for its `source_binding_invalid` source (79 tests pass).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.
- I compared the real incident's lease chain with every predecessor
condition through read-only diagnostics. All eight leases match on
agent, sandbox, workspace, and remote directory, and each predecessor
run saved a snapshot at an older epoch.

## Risks

- Recovery now trusts a predecessor from another task in the same
execution workspace. Acquisition already allows that handoff, and
recovery still requires the predecessor's own run and finalization to be
settled and saved. A workspace is the unit that owns the files, so this
does not let one workspace read another workspace's files.
- I could not read one predecessor's finalization row in the real
incident. If that row is not in a settled phase, recovery still refuses
with `allocation_ownership_ambiguous`, which is the old behavior.
- `snapshot_verification_failed` is labeled `service_required` at its 19
call sites, so it now shows the refusal text. If one of those sites is
really transient, the text says "Retry runs the check again", which is
still true.
- A second cause kept the same task stuck: a remote snapshot save that a
restart interrupted stayed `running`, and that blocks the
retained-source fallback. That fix comes in a separate pull request.
- No schema or API change. The new wait reason is a new string in an
existing field. The UI shows the message as it receives it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only browser diagnostics).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 09:53:37 -06:00
DottaandClaude Opus 5.5 b54317d3fb fix(native): give a preparing run's first turn the user's queued messages instead of interrupting it (#20168)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Paperclip Runner (native) run prepares its workspace and
environment before it starts its first provider turn. This can take
about a minute.
> - A user's message that arrives during that time waits as a deferred
wake behind the run. The Interrupt control then cancelled the run before
any provider turn.
> - The cancel threw away the whole preparation. A new run had to
prepare again, so the user waited longer for the first token.
> - This pull request gives the run's first turn the queued messages.
The run seals its first-turn input immediately before `turn.start`, so
the messages are part of the prompt that the turn starts with.
> - The benefit is that the user sees the message handled in the same
run, with no cancel and no second preparation.

## Linked Issues or Issue Description

**What happened?**
A Paperclip Runner run took about 54 s to prepare (environment startup,
workspace, runtime selection). The user sent a second message 32 s into
that time and clicked Interrupt 21 s later. Interrupt cancelled the run
as `operator_interrupted` ("Interrupted to send queued messages") before
any provider turn. A new run then prepared again from the start. The
runtime-selection transaction that froze the run's input completed about
5 s before the click, and the coordinator claim came 4 s after it, so
the message could not be given to the first turn at any point in the old
code.

**Expected behavior**
When a user message arrives for a task whose current run has not started
its first provider turn, the run does not stop. The message goes into
that run's first turn. A run that is in a turn is interrupted as before.

**Steps to reproduce**
1. Assign a task to a Paperclip Runner agent with a sandbox environment,
so preparation takes a long time.
2. While the run prepares, post a second comment on the task.
3. Click Interrupt on the queued message before the run starts its
provider turn.
4. See that the run is cancelled and a new run prepares again.

**Paperclip version or commit**
Staging revision b43cccbb0.

## What Changed

- `server/src/services/native-runtime/first-turn-queued-input.ts` (new):
`claimFirstTurnQueuedInput` seals a run's first-turn input in one
transaction. It locks the task, the deferred wakes, and the run in the
same order as the queue routes. It takes the responsible user's plain
queued comments in queue order (not interaction answers, interrupt
intents, independent continuations, conversations, or other users'
messages). It renders them with source-trust sanitizing and secret
redaction, then writes a `firstTurnQueuedInput` receipt
(`paperclip.first-turn-queued-input.v1`) into the run context. A second
call for the same run returns the same messages.
`runAcceptsFirstTurnQueuedInput` is true for a running native (or
preparing native) run that has no receipt and no accepted provider turn.
- `packages/paperclip-runner/src/native-session-runtime.ts`: new
`getFirstTurnQueuedInput` option. The runner calls it immediately before
the first `session.startTurn` and appends the result to `task.prompt`
(or to `events` for a resumed-session continuation envelope). It is not
called for the disposition-only recovery turn.
- `native-session-executor.ts`: passes the claim to the runner. A failed
seal commits nothing, so the executor logs it and starts the turn
without the messages. The messages stay queued.
- `queued-comment-delivery.ts`: a message in a run's receipt counts as
delivered only after the provider accepted that run's turn
(`turn.accepted` or the provider's `turn.started`), and only if the
message was not edited after the seal. Release promotion, Send, and
later first turns use this rule. `explicit-native-continuation.ts` uses
the same rule for stopped native runs.
- `routes/issues.ts` Interrupt: when the target run has not sealed its
first turn and has no accepted turn, the route does not record interrupt
intent and does not cancel the run. The response has
`executionWait.reason = "first_turn_pending"`. Other runs are
interrupted as before.
- Queue views (`routes/issues.ts` and the wake-queue adapter) hide
messages that the active run took into its first turn. Queue edits keep
those ids on the wake, so a run that fails before acceptance still
leaves them queued.
- `run-state.ts`: the issue-control run projection includes the receipt,
so the queue poll can hide taken messages.
- `doc/execution-semantics.md`: new section "Queued messages before a
native run's first turn".

## Verification

- Red before the fix, green after:
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/issue-queued-comments-routes.test.ts -t "first
turn|already started"`: Interrupt on a preparing run and on a claimed
native run before `turn.start` does not stop the run, and the first turn
gets both messages in order. Before the fix, both cases cancelled the
run.
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/native-session-runtime.test.ts -t "first-turn queued input"`: the
claim runs immediately before `turn.start`, and the first turn's prompt
ends with the queued messages. Before the fix, the claim was never
called.
- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/first-turn-queued-input.test.ts`: a run that
fails before `turn.start` leaves the message queued. The next run
delivers it, and after that no later run or copied retry context takes
it again. Before the delivery rule, the "exactly once" and "accepted
then failed" cases failed.
- Safety cases that pass before and after: Interrupt still cancels a run
whose provider accepted a turn (including a run saved by the previous
release with no receipt) and a run whose first-turn input is already
sealed. The existing "explicit interrupt fallback" cases for preparing
and native runs without a responsible user still pass.
- Speed: the seal is one short transaction before `turn.start`. In the
embedded-Postgres test it took 5.5 ms. It removes a full second
preparation (54 s on staging) from the message-to-first-token time.
- Related suites: `src/modules/wake-queue`,
`explicit-native-continuation`, `issue-queued-comment-queue`,
`issue-queued-comments-routes`, `queued-comment-consumption`,
`heartbeat/run-state` (888 tests), `native-session-executor.test.ts`
(1306 tests), and the full `native-session-runtime.test.ts` (238 tests)
pass. `pnpm --filter @paperclipai/server typecheck` passes.
- Not done: no live run against a real provider or sandbox. Staging was
read only.

## Risks

- Previous-release state: a deferred wake saved by the previous release
has the same shape and is taken normally. A run saved by the previous
release has no receipt. If it is resumed before its first turn, it seals
on the new release. If its provider already accepted a turn, Interrupt
stops it as before. A run saved by the previous release while it was in
`turn.start` but before acceptance counts as unsealed, so in that short
window an Interrupt does not stop it and the message is delivered at the
turn boundary.
- Rollback: if the server rolls back after a run sealed and its turn was
accepted, the old release does not read the receipt and can deliver
those messages a second time.
- A first turn that does not use the fresh-turn path (restart
continuation, disposition-only recovery) does not call the claim. Its
messages are delivered at the turn boundary as before.
- A message that arrives after the seal waits for the turn boundary.
Interrupt stops the run as before.
- No schema migration. The receipt is a key in the run context.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution, read-only
browser).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:44:18 -06:00
DottaandClaude Opus 5.5 cbf811cd3b fix(recovery): prove a replaced deployment's copyback owner gone automatically (#19913)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save their workspace after the result is accepted, under
a copyback owner receipt
> - A deploy can restart the server while that save is in progress
> - The new server runs in a new container, so it cannot see the old
process and fails closed
> - The run then waits on a board action that asks a person to verify
the previous controller
> - Recoveries must continue without a human step, and copy must not ask
anyone to inspect a server
> - This pull request adds a deployment-level rule to the shared
owner-liveness primitive and uses it in workspace finalization
> - The benefit is that a save interrupted by a deploy resumes on its
own, with no second writer

## Linked Issues or Issue Description

**What happened?**
A deploy restarted the server while three runs were saving their
accepted results. Each run kept a copyback owner receipt that names the
old Railway deployment on another container. The restarted server
reported `native_workspace_finalization_owner_unverified`, owned by the
board, with the next action "Verify the previous controller and its
workspace-copyback processes have stopped...". The cloud harness stop
receipt did not arrive, so the runs stayed in "Recovery needed" for
about an hour, until a person confirmed the stop by hand.

**Expected behavior**
The restarted server proves that the previous controller is gone,
resumes the save, and settles the run with no human step. No user-facing
text asks a person to verify a server or controller.

**Steps to reproduce**
1. Start a native run on a Railway service with a volume.
2. Let the result be accepted and the workspace save start.
3. Deploy, so that the server restarts during the save.
4. See that the run waits for a board confirmation.

**Paperclip version or commit**
Observed on e2d490c61. Reproduced on master before this change. The
owner-proof work in #19848, #19851, #19853, #19868, #19871 and #19883
does not cover this shape: `proveOwnerGone` returns `unprovable`
(`different_host`) for an owner on another container.

## What Changed

- `owner-proof.ts`: `LocalOwnerStamp` can carry the platform `worker`
(deployment identity). `ProcessProbe` can carry this process's
deployment and whether the platform gives it an exclusive volume. New
verdict reason `deployment_replaced`.
- New rule in `proveOwnerGone`, after the current-boot check: the owner
is gone when this server runs with a Railway volume
(`RAILWAY_VOLUME_ID`, `RAILWAY_VOLUME_MOUNT_PATH`) and the stamp names
the same project, environment and service with a different deployment
ID. Railway stops the previous deployment before it mounts the volume
into the new one, and a service with a volume has no replicas. So every
process of the old deployment, including orphaned copyback children, has
exited.
- `native-workspace-finalization-ownership.ts`: evaluate that proof
before the cloud harness request. On `deployment_replaced`, claim the
exact owner receipt under the existing advisory lock and row-lock
revalidation, log `issue.workspace_owner_stop_verified` with
`deploymentProof`, mark the recovery `resuming`, and continue the save
with no provider turn. Other cases keep the harness request and the
existing fences.
- New owner receipts also record the mounted volume ID (`volumeId`).
When a receipt has it, the proof requires this server to mount the same
volume. Both `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH` must be
set. The finalizer reaches this check only while it holds the run's
copyback advisory lock, so the old owner's lock session has also ended.
- Copy: the hold's next action, the route conflict and the two Retry
wait messages now describe the wait. They no longer ask a person to
verify a server or controller. The hold's text still names the Resume
saved result action for platforms where neither this rule nor the
harness can confirm the stop.
- `doc/native-workspace-finalization-recovery.md`: describe the rule.

Previous-release story: receipts written by earlier releases already
record `worker.deploymentId`, so they carry forward and take this path
after the upgrade. No schema change.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-workspace-owner-restart-recovery.test.ts
src/services/owner-proof.test.ts`
- Before the fix: "resumes the interrupted save automatically, with no
board action" fails (`acquired: false`).
  - After the fix: all pass.
- The restart test seeds the staging receipt shape (old deployment,
other hostname, PID 3, existing `operator_required` hold). It checks
that the save resumes once, the hold becomes `resuming`, the task
presentation no longer waits for a user, the receipt clears, and no
agent wake is created.
- Negative cases stay fenced: same deployment, a service without a
volume, a half-configured volume, another service, another environment,
and a receipt that names a different volume. Unit tests cover the same
rule in `proveOwnerGone`, including that the current boot stays alive.
- Related suites pass: finalizer, queued-wake recovery, route ownership,
workspace recovery gates and dispatch, explicit continuation, recovery
services (1005 tests).
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- The rule depends on Railway volume exclusivity. It applies only when
this server has a volume mounted and the receipt names the same service
and environment. Without a volume, the existing harness and board paths
still apply.
- A receipt from a deployment that ran before the volume was attached
could, in theory, overlap during that one redeploy. Volumes are attached
at stack creation, so this is not a practical case.
- Copy changes are text only. Error codes are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 16:12:23 -06:00
DottaandClaude Opus 5.5 a94881d1a7 fix(heartbeat): save a comment refused by a temporary hold and route post-quarantine input to its own gate (#19829)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user's follow-up comment or Retry must be delivered once,
eventually, even when a temporary company admission hold (accounting
still settling) is active
> - #19741 kept claimed runs queued through such a hold and woke held
work when it cleared
> - A restart-chaos wave still lost a comment, because its wake was
refused before admission and written as a terminal skip
> - The same wave left a Retry waiting forever for a stop record of a
process that never existed
> - This pull request saves the refused comment as a deferred receipt,
and routes input after a pre-launch quarantine failure to the
quarantine's own gate
> - The benefit is that the comment is delivered after the hold, and the
task asks for the action that actually unblocks it instead of offering a
Retry that can never start

## Linked Issues or Issue Description

Refs #19741 and #19702.

**What happened?**
In a restart-chaos wave, a failed run's accounting held the company for
600 s.
1. A user commented while the task's run was live. The enqueue budget
gate refused the comment's wake before it reached the deferral behind
the live run. Because it was a user request with no saved receipt, it
was written as `skipped` (`budget.blocked`). The live run finished
without it and the task ended `done`, so the comment was lost.
2. A Retry created a run that the native cleanup-quarantine fence
refused before it requested any runner launch
(`native_session_cleanup_quarantined`, no process events). A second
Retry then waited with `process_identity_missing` ("The previous run has
no verified stop record") for a process that never existed. The task
kept offering Retry, and that Retry could never start.

**Expected behavior**
A comment refused by a temporary hold stays saved and is delivered once
later. Input after a run that was refused before launch is handled by
the quarantine gate: Retry is not offered (send a new message), and a
new message gets the verified-stop check of the previous provider
session.

## What Changed

- `heartbeat/queue.ts`: when the enqueue budget gate refuses a user's
comment wake and the block is temporary (not a reached hard stop, per
`isBoardOwnedBudgetBlock`), the wake is saved as a
`deferred_issue_execution` receipt. The receipt carries the comment ids
and an `executionWait` of `budget.blocked`, and the gate notifies the
deferred-receipt worker and throws `PersistedExecutionWaitRefusal`. It
is not written as `skipped`. Normal promotion (when the live run ends)
and #19741's hold-release wakeups deliver it.
- `native-local-process-stop.ts`: new `nativeRunRefusedBeforeLaunch`. It
returns true for a failed native run with
`native_session_cleanup_quarantined`, no recorded process, and no
launch-request or identity event.
- `explicit-native-continuation.ts`: such a run passes the
process-identity check, so the existing quarantine branch decides. A
Retry gets `cleanup_quarantined` (shown with `canRetry: false` and "send
a new message"). A new message gets
`verifyStoppedNativeSessionForContinuation`. Any launch request or
identity record keeps the original stop proofs.

- `native-session-executor.ts`
(`verifyStoppedNativeSessionForContinuation`): when the source has no
process record and no launch event, the verifier relies on the previous
owner's stop evidence. Before this change it required the source to
share the owner's process identity, so a new message stayed held even
after the owner stopped. A source that carries a process identity must
still match the owner exactly.

## Verification

Each new test fails on `master` and passes with this change, on embedded
Postgres:
- `server/src/__tests__/heartbeat-held-comment-wake.test.ts`: a comment
whose wake the hold refuses at enqueue is saved with a `budget.blocked`
wait and is not skipped. It is delivered exactly once after accounting
settles.
- `server/src/services/explicit-native-continuation.test.ts`:
- A run refused by the quarantine before launch routes a new message to
the quarantine's stop verification and a Retry to `cleanup_quarantined`.
These failed on `master` with `process_identity_missing`.
- Guards (green on both): a quarantined run that requested a launch, and
another failure without a process record, still require
`process_identity_missing`.
- `server/src/services/quarantined-task-retry.test.ts`: a never-launched
quarantined source is certified once the previous owner has stopped (red
on `master`). It stays held if the source requested a launch or the
owner is still running. All 31 tests pass, and the `stopped native
conversation physical cleanup` suite passes (46).
- Related suites pass locally (6 files, 620 tests): explicit native
continuation, held comment wake, queued comments routes, comment wake
batching, refused assignment wake, deferred receipt work. Two
`deferred-execution-receipt-work` tests failed once under parallel load
and passed when run alone.

## Risks

- The saved-comment path applies only to user comment wakes refused by a
temporary budget hold. Hard stops, system wakes, case handoffs, durable
chat requests, and existing saved receipts are unchanged.
- The pre-launch rule is limited to the quarantine failure code with no
launch evidence. Every other missing process identity still requires the
existing stop proof.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in
Claude Code with tool use (shell, file editing, local test execution,
kept restart-chaos instance inspection).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 08:54:14 -06:00
DottaandClaude Opus 5.5 c45c7cb854 fix(workspaces): ignore killed-transfer scratch in sparse checkpoints (#19730)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run in a remote sandbox copies its workspace back to the
host after the turn. A sparse checkpoint copies only changed entries,
and the host validates each entry against the sandbox's own manifest
> - Every Daytona transfer stages a `.paperclip-upload-<uuid>` scratch
file at the sandbox workspace root and removes it only in the same
process
> - A control-plane kill during any transfer leaves that scratch behind.
The sparse checkpoint then recorded it as a workspace file, but
providers never export it. Every later copyback and every saved-work
recovery attempt failed with "Workspace checkpoint payload mismatch"
> - The failed recovery then left an accepted Retry showing "Paperclip
is finishing the saved result" with no run
> - This pull request stops recording transfer scratch, makes the
checkpoint export an exact copy, names the failing entry, and gives a
Retry behind a spent recovery an honest reason and a working action
> - The benefit is that a restart during a transfer no longer makes a
task's saved work permanently unrecoverable

## Linked Issues or Issue Description

Refs #19721 (RC-50: the killed copyback owner; this PR covers the
separate RC-50b cause)

**What happened?**

Restart-chaos waves s12/t01 and s14/t05 (Claude on Daytona). After a
SIGKILL of the control plane, the native result was accepted, and then
workspace copyback failed with "Workspace checkpoint payload mismatch".
In s14 all 6 saved-work recovery attempts failed the same way. A Retry
accepted during recovery kept `retryWait.reason =
finalizing_saved_result` after the recovery budget was spent. No run was
created.

**Steps to reproduce**

1. Start a native Claude run in a Daytona sandbox.
2. SIGKILL the control plane while any provider transfer is in flight.
3. Let the run finish its turn and copy its workspace back.

**Expected behavior**

Copyback succeeds. Provider scratch is not workspace content. A Retry
behind a spent recovery names the failure and can start a new recovery.

**Actual behavior**

The s14 sandbox (6f4a0b6a) still held
`/home/daytona/paperclip-workspace/.paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar`.
Its mtime was 03:29:23, the second of the K4 SIGKILL. The Daytona
provider stages transfer scratch at `remoteDir`, which is the lease
`remoteCwd` (the workspace root). The runtime workspace wipe
deliberately preserves `.paperclip-upload-*`. The checkpoint helper
(`workspace-checkpoint.ts`, `checkpointMain`) recorded the stale tar.
The ordinary export excludes `.paperclip-upload*`, so the host received
nothing for that entry. `validateCheckpointPayload` then threw "payload
mismatch" on every attempt. The file-mode hypothesis was disproved: the
sandbox umask is 022 and the scratch is mode 644.

## What Changed

- `packages/adapter-utils/src/exclude-patterns.ts`: new
`isSandboxTransferScratchPath`, which matches a root-level
`.paperclip-upload-*` entry. This agrees with the wipe's preserve rule
in `buildWorkspaceTarExtractCommand`.
- `packages/adapter-utils/src/workspace-checkpoint.ts`: the sandbox
checkpoint helper skips that scratch. A mismatch error now names the
entry and its recorded, baseline and received forms (kind, octal mode,
short hash; never contents).
- `packages/adapter-utils/src/sandbox-managed-runtime.ts`: the
checkpoint export requests `preserveSourceEntries: true`. The checkpoint
output is an exact copy that the host validates entry by entry. An
ordinary export applies the host umask and drops `._*` and
`.paperclip-upload*` names at any depth. That gives the same permanent
mismatch for a legitimate user file such as `._summary`,
`notes/.paperclip-upload-plan.md`, or a `0775` script.
- `server/src/services/explicit-native-continuation.ts` and
`task-interruption-presentation.ts`: when the run's saved-work recovery
is exhausted, a waiting Retry reports `saved_work_recovery_failed` with
Retry available, instead of `finalizing_saved_result`. The existing task
Retry path starts one new recovery generation.
- `doc/execution-semantics.md`: a short additive note.

## Verification

- `packages/adapter-utils/src/workspace-checkpoint-capture.test.ts` uses
the real checkpoint helper and an export that behaves like Daytona's
ordinary export:
- Before the fix: `Workspace checkpoint payload mismatch at
".paperclip-upload-d99fe36b-….tar": sandbox recorded file mode=644 …;
baseline absent; received absent`. That is the wave error.
  - After the fix: the scratch is not recorded, and validation passes.
- A second case shows that reserved-looking user names and `0775` modes
need the exact export.
- `packages/adapter-utils/src/workspace-checkpoint.test.ts` is an
end-to-end `prepareSandboxManagedRuntime` restore through a provider
model. In that model, ordinary exports drop reserved names and apply the
umask, and preserve exports copy exactly. Two red checks:
- Without `preserveSourceEntries`, it fails with the mismatch on
`._summary`.
- Without the scratch skip, the stale tar leaks into the host workspace.
- Green: the agent file, the `._summary` file and the `0775` mode
arrive, and no scratch arrives.
- `server/src/services/workspace-recovery-preparation-gates.test.ts`,
"names a spent saved-work recovery behind an accepted Retry, and the
next Retry finishes it" (s14 shape):
  - Red: `finalizing_saved_result` after 6 failed attempts.
- Green: `saved_work_recovery_failed` with `canRetry: true`. The next
Retry starts a new generation, recovery completes, and the Retry
continues in exactly one new run.
- Suites: adapter-utils checkpoint, capture, sandbox-managed-runtime,
seed-reuse and managed-seed (168 passed). Server preparation-gates (47),
explicit-native-continuation and task-interruption-presentation (455),
workspace-recovery admission and retry-copyback, and
native-workspace-finalizer. `tsc --noEmit` is clean for adapter-utils
and server.
- Live evidence: I started the stopped s14 sandbox read-only, listed its
workspace, and stopped it again. No Daytona replay was run. The runner
content ID of this checkout matches no built image.

## Risks

- A user file named `.paperclip-upload-*` directly at the workspace root
is no longer captured by a sparse checkpoint. Providers already drop
that name from every ordinary export, and the runtime already reserves
it at the root.
- The checkpoint export now uses the provider's exact-copy mode. A
workspace file with setuid, setgid or sticky bits is refused by that
mode, as it is for the existing exact-copy transfers. The ordinary
export would have stripped those bits and failed validation anyway.
- The stale scratch tar stays in the sandbox. It is ignored, not
deleted, because a concurrent transfer can own a file with the same
prefix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, test runs, embedded Postgres, a
read-only Daytona sandbox listing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 22:46:16 -06:00
DottaandClaude Opus 5.5 db41bfcb38 fix(recovery): let an accepted Retry finish after a killed copyback owner (#19721)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run saves the agent's workspace files after the provider
turn (copyback). One controller owns copyback at a time and records a
durable owner receipt on the run (`nativeWorkspaceFinalizationOwner`)
> - When copyback of an accepted result fails, retained saved-work
recovery exports the files from the kept sandbox. A task Retry waits for
that recovery before it continues
> - A control-plane restart can kill the controller while it owns
copyback. The dead controller can never confirm its own stop, and
nothing in the retained recovery path could release its receipt
> - The recovery worker then spent one attempt on each recheck of that
receipt until it was exhausted, and the accepted Retry kept saying
"Paperclip is finishing the saved result" with no run. The task waited
forever
> - This pull request stops spending attempts on that recheck, names the
real wait, and lets the board release the exact receipt so the same
recovery and the same Retry continue
> - The benefit is that an accepted Retry never waits silently: it
finishes, or it shows one honest action that makes it finish

## Linked Issues or Issue Description

Refs #19717 (same-host automatic proof for a killed copyback owner; this
PR covers the retained-recovery path and the case where no automatic
proof is possible)
Refs #19713, #19609, #19694 (related restart-recovery fixes; no overlap
in files except this doc)

**What happened?**

Two Daytona restart-chaos waves. In each, a native run's copyback failed
and the run failed with `native_workspace_sync_out_retry_exhausted`. The
agent's result was accepted. Retained recovery started. A SIGKILL of the
control plane landed while a recovery attempt owned copyback:

- Wave s13, task t06 (Claude, crash loop): Retry was accepted at
02:30:35. A SIGKILL landed 2 s later. All later attempts failed with
`copyback_stop_unconfirmed`. After 6 attempts the recovery was
exhausted. The Retry showed `retryWait.reason = finalizing_saved_result`
for more than 15 minutes. No run was created.
- Wave s12, task t01 (Claude): attempt 1 failed with "Workspace
checkpoint payload mismatch". The SIGKILL at 01:43:54 killed the owner
of attempt 2. Attempts 3 to 5 failed with `copyback_stop_unconfirmed`.
The Retry was accepted with HTTP 2xx and produced no run.

In both kept databases the run carries an owner receipt for the killed
process.

**Steps to reproduce**

1. Accept a native result and make its workspace copyback fail, so that
retained recovery owns the export.
2. Press Retry on the task.
3. Kill the control plane while a recovery attempt owns copyback, so
that its owner receipt stays on the run. Use another host, or a receipt
without automatic stop proof.
4. Let recovery run.

**Expected behavior**

Rechecking the receipt does not spend attempts. The task names the wait
and does not offer a Retry that cannot help. A board member can confirm
that the previous server stopped. The recovery then finishes, and the
waiting Retry continues by itself.

**Actual behavior**

`withNativeWorkspaceFinalizationOwnership` returned `owner_unverified`
for a retained repair without creating any board action
(`native-workspace-finalization-ownership.ts`,
`requireStopVerification`). `recoverRetainedWorkspace` reserved an
attempt for each recheck and failed it as `copyback_stop_unconfirmed`
(`workspace-recovery.ts`), until the budget was exhausted. The Retry
gate in `explicit-native-continuation.ts` returned
`finalizing_saved_result` for an accepted result with no verified
repair, with no check for this hold.
`resumeNativeWorkspaceAfterOwnerStop` accepted only an
`owner_unverified` action on a `running` run, so no route could release
the receipt of a finished run. The exhausted handoff told the board to
press Retry again, which repeats the same failure.

## What Changed

- `server/src/services/workspace-recovery.ts`: when retained recovery
finds an unverifiable copyback owner, it records the recheck without
reserving an attempt. The 30-minute recovery window still bounds it. The
retained action records the exact receipt as `workspaceOwnerRecovery: {
state: "operator_required", owner }`. A newer receipt replaces the
record and clears an older confirmation.
- `server/src/services/explicit-native-continuation.ts`: a Retry that
waits for an accepted saved result reports
`workspace_owner_stop_required` when the run's active retained action
has that hold. It no longer reports `finalizing_saved_result` for this
case.
- `server/src/services/task-interruption-presentation.ts`: the new wait
has a fixed message, and Retry is not offered for it, because another
click cannot release the receipt.
-
`server/src/services/native-runtime/native-workspace-owner-recovery.ts`:
the existing board route `POST
/issues/:id/recovery-actions/resume-workspace-finalization` (permission
`runtime:confirm_owner_stop`) also accepts a retained recovery action
with that hold. The confirmation must name the receipt that is still on
the run. A live owner on this host still rejects it. It clears only that
receipt, records the stop proof and an activity entry, and returns the
same recovery to the worker. A spent budget gets one new generation, as
a deliberate recovery request does. It accepts no result and starts no
agent turn. The owner check is shared with the existing path.
- `server/src/services/system-recovery.ts`: an exhausted
`copyback_stop_unconfirmed` handoff names the board confirmation instead
of another Retry.
- `ui/src/api/issues.ts`: the route receipt's `resultId` can be `null`
(a retained recovery has no live result binding).
- `doc/native-workspace-finalization-recovery.md`: an additive paragraph
describes this path.

## Verification

- Red then green on embedded Postgres:
`server/src/services/workspace-recovery-preparation-gates.test.ts`,
"lets an accepted Retry finish after a killed copyback owner instead of
waiting forever" with one case per wave (s13, s12). Each case uses a
real accepted remote export, a Retry through the HTTP route, a
killed-owner receipt, and 8 recovery passes.
- Before the fix (both cases): `attempts: 6, state: "exhausted"`. That
is the wave state.
- After the fix: no attempt is spent on the recheck. The hold is
recorded. The Retry wait is `workspace_owner_stop_required` with
`canRetry: false`. The board confirmation through the route returns 202
and clears the receipt. The same recovery completes. The same accepted
Retry then finishes the saved result (s13: the task is `done`) or
continues in exactly one new run (s12).
- `npx vitest run
src/services/workspace-recovery-preparation-gates.test.ts -t "killed
copyback owner"`: 2 passed.
- `pnpm verify` (affected selection): see the comment below. Server `tsc
--noEmit` is clean.
- No live Daytona replay. Both waves ran on old signed images. The local
proof uses the same accepted-export fixture as the existing Retry tests,
with the receipt state copied from the kept databases.

## Risks

- A copyback owner on another host that is still alive is now rechecked
within the recovery window without spending attempts. Before this change
it spent them. The window still bounds the wait.
- The board confirmation for a retained action is an operator statement,
the same as the existing owner-stop path. It is limited to roles with
`runtime:confirm_owner_stop`. It never clears a receipt other than the
one it names, and a live same-host owner still rejects it.
- #19717 changes the same ownership area and adds a UI action for the
running-run case. That UI does not show a retained action yet. After
both merge, a small follow-up can show its button for this hold. Until
then the API route and the task's wait message cover it.
- Not addressed here: the "Workspace checkpoint payload mismatch" that
started wave s12. The checkpoint outbound transfer extracts without `-p`
(`sandbox-managed-runtime.ts` checkpoint sync-out, `extractHostTarball`
in the Daytona provider), so the host umask can change file modes that
the remote manifest recorded. That is a separate signature.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, test runs, embedded Postgres).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 22:07:00 -06:00
DottaandClaude Opus 5.5 f09b4704b6 fix(recovery): keep the agent's work when Retry continues after exhausted or blocked recovery (#19666)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Task recovery restarts agent work after a control-plane restart or a
failed run
> - When automatic recovery runs out or is blocked, the user clicks
Retry
> - Retry then started a fresh isolated attempt cloned from committed
HEAD, so the agent lost its uncommitted files, its task branch and the
worktree Git link
> - A restart must never lose the agent's work
> - This pull request makes Retry keep the task's own workspace when
that is safe, and tells the agent where the old files are when a fresh
attempt is still required
> - The benefit is that a Retry continues the real work instead of
starting over

## Linked Issues or Issue Description

**What happened?**
After `native_restart_recovery_blocked` or an exhausted recovery
generation, a Retry ran in
`isolated-workspace-attempts/<company>/<issue>/<run>/workspace`. That
directory was a detached clone of the base ref. It did not contain the
uncommitted edits from the issue's git worktree. It was not on the task
branch. The agent was not told where the old files were. A restart-chaos
wave graded this as an R2 failure (`pre_fault_files_lost`). A second
defect: when a restart hit the bootstrap of a fresh attempt, preparing
the same run again failed with
`isolated_task_directory_binding_mismatch`, so the Retry could not
finish.

**Expected behavior**
Retry continues in the task's own workspace with its edits and `.git`
intact when no other writer can hold it. When a fresh attempt is really
needed, the old directory stays as it was and the agent gets its path
and its changed files. Preparing the same attempt again after a restart
is idempotent.

**Paperclip version or commit**
`master` at `e482f0f632`, restart-chaos waves on 2026-11-06.

**Steps to reproduce**
1. Start a native task in a git worktree. Let the agent write an
uncommitted file.
2. Kill the control plane so recovery ends in
`native_restart_recovery_blocked` or an exhausted generation.
3. Click Retry.
4. The successor runs in a fresh clone without the uncommitted file or
the task branch.

Root cause:
- `server/src/services/fresh-workspace-continuation.ts`
`canSelectFreshWorkspace` returns a fresh intent for every task Retry
that has a system recovery record.
- `server/src/services/explicit-native-continuation.ts` (fresh selection
in the source loop) then always persisted `fresh_isolated_workspace`.
- `server/src/services/fresh-workspace-seed.ts`
`materializeFreshWorkspaceSeed` clones the project source at the base
ref with `--no-local`, checks out a detached commit and removes
`origin`. Uncommitted work and the branch are not carried over.
- RC-35: `server/src/services/heartbeat/workspace-preparation.ts`
rejected any reusable existing workspace while `fresh` was true,
including the attempt directory that the same run had already created
and bound.

## What Changed

- New continuation mode `reuse_issue_workspace`. At Retry admission,
after all stop proofs pass, `retainedIssueWorkspaceForRetry` checks that
the task's bound workspace is the single local directory the failed run
used, owned by the task, not a shared workspace, not under remote
storage authority, with no other lease, failed cleanup or running
workspace operation, a settled recovery generation, no workspace-export
failure, and a readable Git state rooted at that directory. Then the
Retry continues there with a fresh session. Otherwise the existing fresh
attempt is kept.
- The reuse decision consumes the generation's one alternative attempt
(`freshAttemptRunId`), sets the Retry strategy to `continue`, and
records `issue.retry_workspace_reused`.
- Environment selection pins the reused workspace binding and fails with
`retry_workspace_binding_mismatch` if the binding changed before
dispatch.
- Fresh fallback: the run's continuation context gets `priorWorkspace`
(path, branch, bounded list of uncommitted files). Git status is read
with optional locks disabled. The prompt tells the agent to inspect it
and carry over the work. The old directory is not changed.
- RC-35: preparing the same fresh attempt again after a restart reuses
the attempt directory that this run already bound.
- Reuse checks run Git through the shared workspace scan scheduler with
short deadlines, because admission can hold the issue lock.
- `doc/task-recovery.md` and `doc/execution-semantics.md` describe the
new contract.

## Verification

- `server/src/services/retry-workspace-continuity.test.ts` (embedded
Postgres, real git worktree with uncommitted edits):
- exhausted-generation Retry and blocked-recovery Retry continue in the
same worktree, `.git` file and branch intact. Both failed before the fix
(readiness was `fresh_isolated_workspace`).
- Blocked recovery without a generation and duplicate holds for one run
also reuse the worktree. A running operation orphaned by a restart of
the stopped source does not block reuse.
- The fresh attempt is kept when the workspace is unsafe: live lease,
running workspace operation, corrupt `.git`, missing directory, remote
authority, shared workspace, running export generation. `priorWorkspace`
is omitted for the unsafe writer and shared cases.
- `workspace-preparation.test.ts`:
- A `reuse_issue_workspace` Retry prepares in the original worktree with
its uncommitted file and branch.
- RC-35 red/green: re-preparing the same fresh attempt reuses its bound
directory and keeps files written before the restart. Another run still
gets `isolated_task_directory_binding_mismatch`.
- The fresh fallback sets `priorWorkspace` with the old path and
`unpublished.txt`.
- `environment-selection.test.ts`: the new mode pins the binding and
rejects a changed binding or cwd.
- `packages/adapter-utils/src/server-utils.test.ts`: the prompt names
the retained directory and its files.
- Regression files run locally: `workspace-preparation`,
`environment-selection`, `task-retry`, `quarantined-task-retry`,
`explicit-native-continuation`, `workspace-recovery`.
- Restart-chaos (`tests/restart-chaos`, local, B1+B5, codex+claude,
K1/K4/K6, concurrency 2, rc06-masked):
- Final head: wave `w-20261107T014633-s7206` on `c6776283f3`: 8/8 PASS,
all 8 restart-crossing. 5 tasks needed a Retry (t03-t07: codex B1 K6,
codex B1 K1, codex B5 K6, codex B5 K1, claude B5 K6). All five finished
in the original git worktree with `.git` and every pre-fault file
present (`workspaceMovedTo` empty, `lostFiles` empty); no "Retry keeps a
fresh attempt" log line.
- Wave `w-20261107T004829-s7205` on `29373a6b6a`: 8/8 PASS, all 8
restart-crossing. 3 tasks needed a Retry (t02, t06, t07). All three
finished in the original git worktree with `.git` and every pre-fault
file present; no "Retry keeps a fresh attempt" log line. (`cc178492ef`
only makes the operation filter stricter for run-less jobs.)
- Wave `w-20261107T000412-s7204` on `d197082107`: 8/8 PASS, 7
restart-crossing. 3 tasks needed a Retry (t02, t04, t06). All three
finished in the original git worktree (`workspaceMovedTo` is empty), R2
passed with `.git` and every pre-fault file present, and no "Retry keeps
a fresh attempt" log line appeared.
- Wave `w-20261106T231416-s7203` on the first commit: 6/6 PASS, but its
t01 Retry still moved to `isolated-workspace-attempts`. That instance
was not kept, so the exact rejection was not recorded. The likely cause
is a `running` workspace operation that a killed server left on the
stopped source (the source restarted three times). The second commit
treats such restart orphans as not live and logs the rejection reason. A
`--replay` of that t01 on `d197082107` finished in the original worktree
with its `.git` file and no attempt directory.
- Review follow-ups in `d197082107`, `29373a6b6a`, `cc178492ef` and
`c6776283f3` (run-less workspace jobs still block reuse; red/green
covered by the `running_operation` case): holds are grouped by source
run; the readiness fallback (blocked recovery without a generation) also
reuses the task workspace; Git probes go through the shared scan
scheduler with 10-15 s deadlines; `priorWorkspace` is only set for a
task-owned directory with no other writer; the reuse rejection reason is
logged; `priorWorkspace` uses the same task-owned, settled,
single-writer predicate as reuse; Git probes share one budget per
workspace; the RC-35 check is a pure path check; renames are parsed from
either porcelain column.

## Risks

- A Retry now writes into the old directory instead of a new one. The
checks above prevent a second writer. A stale binding fails the run with
`retry_workspace_binding_mismatch`; it does not write to another
directory.
- If the cause of the failure was inside the workspace itself, reuse
could repeat it. Workspace-export failures, restore failures and
required workspace recovery keep the fresh attempt.
- A workspace operation of the stopped source that started before the
source settled and before the current server process no longer blocks
reuse (a restart orphan). Operations with no run, of another run, or
started in the current process still block it.
- `priorWorkspace` is an optional new field in the continuation
envelope. Older adapters ignore it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 20:27:22 -06:00
DottaandClaude Opus 5.5 30857db55e fix(native): recover a run killed between runner launch and its identity record (#19694)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs start a local `paperclip-runnerd` process. The server
records the request to start it, then records its process identity after
it spawns.
> - If the server is killed between those two records, there is no
process ID to check after the restart
> - Restart recovery then restarted the bootstrap, but the executor
failed with `durable_identity_unreadable` because the runner never wrote
its state
> - The user's Retry then waited forever with "Retry is waiting for
Paperclip to verify that the previous execution stopped", because the
stop check needs a process ID
> - This pull request restarts that bootstrap and proves on the host
that no runner for the run exists
> - The benefit is that a restart in the runner launch window no longer
loses the task or leaves Retry permanently stuck

## Linked Issues or Issue Description

**What happened?**
In restart-chaos testing, an ACPX Claude run was created and the server
was SIGKILLed 1.3 s later, right after `native.process_start_requested`
and before the runner identity was recorded. On restart, recovery
claimed `bootstrap_incomplete` (`dead_runner_before_provider_identity`).
The executor then threw `runner_state_identity_mismatch:
durable_identity_unreadable`: the state root existed, but
`control-plane/control-plane-state.json` had never been written. The run
failed with `native_continuation_requires_reconciliation`. A Retry was
accepted (HTTP 200), but no run was ever created. The issue stayed
blocked with `retryWait.reason=process_identity_missing` for over 20
minutes and through another restart.

**Expected behavior**
When the runner never wrote state, no provider started, so the bootstrap
can start again. A Retry should be admitted once the host proves that no
runner for the run exists.

**Paperclip version or commit**
`master` at `fb3cf1a072`.

**Steps to reproduce**
Create a native local task and SIGKILL the server about 1 s after the
run starts, after the runner launch is requested and before its identity
is recorded. Restart the server, then click Retry.

## What Changed

- `migrateRunnerdStateRootForExecution` (`native-session-executor.ts`):
for a `bootstrap_incomplete` restart on a local target, a root is
quarantined (kept for inspection) and the bootstrap starts again only
when both of these hold:
  - the root has no `control-plane-state.json`;
  - no live runner for the run exists on this host.

A state file that exists but cannot be read, or a runner that started
but has not connected yet, still fails closed with
`durable_identity_unreadable`.
- `unidentifiedLocalRunnerProvenAbsent` (`native-session-executor.ts`)
proves that a runner is absent when both of these hold:
- no host process carries the run's `--run-id <runId>` argument
(`paperclip-runnerd` is always started with it, detached and
unreferenced);
- no saved control-plane state for this run exists, in the canonical
root or in quarantine.

A provider starts only after its runner connects and writes that state.
- `native.process_start_requested` now records the launching `hostname`.
- `recordUnidentifiedLocalRunnerStop` (`native-local-process-stop.ts`)
applies the proof only when all of these hold:
  - the run is native;
  - the run has no process ID;
- its latest launch event is `native.process_start_requested`, from this
host;
  - all of its leases are local.

  A launch recorded without a host stays blocked.
- The process listing runs asynchronously with a 5 s deadline. A failed
or slow listing proves nothing.

It then writes `native.local_process_stopped` with null process IDs and
`basis: "unidentified_runner_absent"`. A later launch request
invalidates it as usual.
- The continuation and Retry gate (`explicit-native-continuation.ts`)
uses this proof before returning `process_identity_missing`. A dry run
checks the proof but records nothing.
- `doc/run-log-events.md` documents the new receipt basis.

## Verification

- `native-session-executor.test.ts`:
- The bootstrap restart with absent state was red before the fix
(`durable_identity_unreadable`) and is green after it (root quarantined,
backend admitted).
- Unreadable state, and a live runner that has not connected, still
reject.
- Proof cases: absent → true; a live process with `--run-id <runId>` →
false; this run's saved state → false; another run ID → false.
- `explicit-native-continuation.test.ts` (embedded Postgres, real
admission):
- The `absent` and `dry_run` cases were red before the fix (blocked) and
are green after it (admitted). Only the real admission records the
receipt.
- The `live_runner`, `saved_state`, `foreign_host` and `unbound_launch`
cases stay blocked.
- Broader local run after the review fixes:
`explicit-native-continuation`, `native-session-executor`,
`native-recovered-owner-followup`,
`native-runner-restart-recovery.integration` and `task-retry` gave 1836
passed, 0 failed.
- A chaos replay of the original task (`--replay
w-20261106T232854-s7103/t26`) passed (R1 PASS). Its kill landed after
the runner identity was recorded (`resume_dead_runner`), not in the
launch window, so the tests above are what reproduce that window.
- `tsc --noEmit` for `server` passes.

## Risks

- The proof depends on `ps -axww` listing every process on the launching
host. If `ps` fails or times out, the proof fails closed (still
blocked). Remote runners and other hosts are excluded.
- A runner that was spawned but has not yet written its state could be
mid-start when admission runs. It would still carry `--run-id` in its
arguments, so the proof refuses.
- Only the bootstrap path with no state file changes. Ownership,
checkpoint and reattach checks are unchanged.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 20:06:36 -06:00
DottaandClaude Opus 5.5 785ead5c7a fix(tasks): hold a Retry while the task's own native session close is retained (#19609)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native provider session close fails, the runner keeps a
cleanup owner or quarantine in process memory. Its admission gate
refuses new sessions for that task (scoped to the task since #19532)
> - The server could not see that state. A task's Retry was accepted
(202), its run was dispatched, and the runner refused it at once with
`native_session_cleanup_quarantined: prior session cleanup remains
incomplete`
> - The user saw a Retry that failed instantly, and nothing said what to
do
> - This pull request lets the server read the runner's per-task cleanup
gate and hold the Retry with a clear reason. The Retry starts
automatically when the close is released
> - The benefit is that a Retry is never dispatched only to fail on a
block that is already known

## Linked Issues or Issue Description

**What happened?**
After two hard restarts, a Codex run's close stayed incomplete. The
board Retry for that task was accepted, and its run failed in about 1 s
with `native_session_cleanup_quarantined: prior session cleanup remains
incomplete`. The next Retry did the same.

**Steps to reproduce**
1. Let a native session close fail or stay pending for a task.
2. Press Retry on that task.

**Expected behavior**
The Retry waits with a clear reason, and starts when the cleanup
finishes. If the cleanup needs an operator, Retry is not offered and the
one working action is shown.

**Actual behavior**
The Retry is accepted and its run fails at once.

## What Changed

- `native-session-runtime.ts`:
- New `nativeSessionCleanupBlock({ companyId, issueId })`. It is a
read-only view of the in-process admission gate for one task: `pending`
(a retained close or a bounded automatic close recovery still owns the
task's provider resources), `operator_required`, or `null`. It grants no
authority.
- New `onNativeSessionCleanupSettled(listener)`. It is called whenever a
cleanup owner or quarantine entry is released.
- `explicit-native-continuation.ts` (the continuation gate for a Retry
or a new message):
- `pending`: the input is held with `local_cleanup` ("Retry is waiting
for the previous execution to finish cleanup. Your message is saved.").
It starts automatically when the close is released.
- `operator_required`: handled exactly like a recorded
`native_session_cleanup_quarantined` run. Retry is not offered. A new
message runs the existing stopped-process proof
(`verifyStoppedNativeSessionForContinuation`), which retires the runner
quarantine when the old processes are proven stopped.
- `native-cleanup-release-wake.ts` (registered in
`server/src/index.ts`): each cleanup release wakes deferred execution
for the company in a transaction, so a held Retry re-evaluates.
- `server/src/vendor/paperclip-runner/index.ts`: exports the two runtime
functions.

## Verification

- Red before the gate (server change stashed): `pnpm --filter
@paperclipai/server exec vitest run
src/services/quarantined-task-retry.test.ts -t "holds a Retry"`
dispatched a successor run while the task's close was retained
(`expected [ { … } ] to have a length of +0 but got 1`).
- Green after the gate:
- The Retry is held (`deferred_issue_execution`, `executionWait.reason =
local_cleanup`), and no successor is dispatched.
  - A different task is not blocked.
- When the close settles, the release is announced, and the next pass
dispatches exactly one successor.
- The whole `quarantined-task-retry.test.ts` file passes (28 tests).
`src/native-session-runtime.test.ts` passes. Server `tsc --noEmit`
passes after the runner package build.
- Live chaos replays: results are added in a comment.

## Risks

- The gate reads process memory, so it applies only where the server and
the runner session share a process (local runner). Other deployments
keep today's behavior.
- A held Retry re-evaluates on the release wake (sent in its own
transaction) or on the existing 30 s deferred-execution pass. When the
bounded automatic close budget is spent with nothing else scheduled, the
view reports `operator_required`. Retry is then no longer offered, and a
new message reaches the stopped-process recovery instead of waiting
forever.
- No schema or API shape changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #19532.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 18:52:46 -06:00
DottaandPaperclip c9391720a1 fix(heartbeat): send saved follow-ups after task completion (#19600)
## Thinking Path

> - Paperclip manages agents and their work.
> - Task messages remain in a durable queue while an earlier run saves
its result.
> - That result can mark the task done before the user sends the saved
messages.
> - Send records current user intent, but its wake guard allowed only
active task states.
> - The old queue therefore remained pending with an obsolete
finalization message.
> - This change admits that exact Send receipt and reopens the task
through the issue service in the successor transaction.
> - The agent can receive the saved input once while the completed
result remains unchanged.

## Linked Issues or Issue Description

**What happened?**

Send on a saved message can leave the message queued after its native
predecessor succeeds and commits. The old wait message remains visible.
Allowing admission alone also fails: continuation construction rejects a
task that remains done.

**Expected behavior**

Explicit Send must deliver the saved input once through normal
admission. It must retain the previous run and its result.

**Steps to reproduce**

1. Save a follow-up while a native run finishes its result.
2. Let that run succeed, commit its accepted result, and mark the task
done.
3. Send the saved follow-up. Before this change, the receipt stays
deferred.

Related public work: Refs #19542. That change selects recovered
workspace files. This change handles the later saved-message admission.
Refs #19602 for the independent workflow fixture repair required by CI.

## What Changed

- Allow the exact explicit Send wake guard to accept a done task. Keep
automatic recovery and cancelled task guards unchanged.
- Reopen the task through the company-scoped issue service after
admission gates pass. Commit reopening, queue consumption and the
successor together. Publish the audit after commit.
- Add a native succeeded/committed predecessor regression with unchanged
saved work and one successor after two resume calls.
- Verify real heartbeat startup under the clicking operator. Keep
automatic input, cancelled tasks and paused agents held.
- Check workspace preflight for the prospective reopened task. Use the
normal company-scoped transition if preflight blocks a done task, so its
completion metadata is cleared.
- Check a successful native predecessor for its committed accepted
result, current ownership and pending cleanup. Keep approvals and
questions effective even without a recovery action.
- Document the saved Send contract and update the queue test dependency.

## Verification

- Final merge qualification for
`675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`: all 56 applicable status
checks pass, with no pending or failed checks. [Final
CI](https://github.com/paperclipai/paperclip/actions/runs/42307623559)
includes typecheck, build, all applicable test and E2E shards, and
aggregate verification. Current-head Greptile is 5/5; all five review
threads are resolved. Any paused or queued application review is not
claimed as a new completed review.

- Final base qualification at
`675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`: official preflight passes
(1.5s); all six integrated model-normalization cases pass (19.07s); both
affected test typecheck layers pass. The integration changes no
production server/shared/Runner/adapter inputs, so the preceding 12-case
Send proof, two ordinary status guards, and unchanged production
typecheck layers remain applicable. Fresh final-head CI supplies broad
coverage.

- New review regression: both failed and cancelled native predecessors
on a task manually marked Done fail before the repair with
`continuation_task_ownership_changed` through real
`resumeQueuedCommentInterrupt` admission. At
`bfb0eff7941090ac3ac43ce4366ba2e3d46612e9`, all 12 affected
Send/source/cleanup/workspace/decision cases pass (246.80s), ordinary
Done/cancelled rejection cases pass (2; 100.31s), and all seven server
typecheck layers pass. History validation uses the prospective reopened
status only for an already-verified explicit native Send. No status is
changed during a dry run.
- Prior `e64e4bb1` CI passed the affected serialized server suites,
including the complete saved-message route suite, typecheck and build.
General server shard 4 still failed two unrelated stale default-model
expectations (`gpt-5.6-sol` versus the new `gpt-6.1-sol` default). The
equivalent repair has already merged in #19610 as
`8621bcf53ee2e4bbefb7d193add42dd544ca06e8`; the separately prepared
#19645 is closed as superseded. This source fix does not weaken or edit
those assertions.

- Original RED: the committed-source and completed-task startup cases
left the saved receipt deferred.
- Review RED: Done plus an unrunnable isolated worktree created one
successor instead of zero. A successful committed predecessor with
pending lease cleanup also created one instead of zero. The new
Done-to-blocked path retained stale completion time. Pending approvals
and questions each allowed a successor. All failures were captured
before their respective repairs.
- Integrated production revision
`99018264dd1ef002ff3500378184bcd167d9206b`: 417 affected cases pass
across the complete native continuation, responsible-user startup, and
workspace preflight suites (386 + 18 + 13). These include the original
successful startup case and six cleanup/finalization faults that hold
the exact saved receipt until settlement, then deliver it once.
- `pnpm --filter @paperclipai/server typecheck` passes all seven layers
on that integrated head, including the local Runner/SDK dependency
build. Module-boundary, whitespace, and added-code-comment checks pass.
- Normal master integrations preserve branch history. No force push or
assertion weakening was used. Fresh CI and automated review are running
on `675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`, which normally integrates
the already-merged #19610 fixture repair after the reproduced inherited
CI failure.
- The earlier CI attempt failed in five unchanged workflow fixture cases
because IMAGE_ONLY was unset. The merged #19602 repairs that fixture. A
later inherited duplicate default in that independent fixture has a
separate repair; this PR does not edit it.
- Follow-up route-fixture RED: three existing native queue delivery
cases failed because their fixture marked a run successful without an
accepted result or committed finalization. The same three cases pass
after the fixture records the complete native success contract (42.08s);
delivery assertions and production gates are unchanged. The affected
test-tree TypeScript layer also passes. The full route run passed all
119 assertions but failed its 30-second database teardown hook, so it is
not recorded as a suite pass. That result is retained. One unchanged
bounded rerun was stopped after more than 15 minutes without a final
verdict; it is inconclusive, not a pass.
- Final integrated head `e64e4bb1`: all 13 focused route and
completed-task admission cases pass across two real-PostgreSQL suites
(188.95s), including the three repaired route fixtures and every new
cleanup/finalization/decision hold. Fresh preflight passes; all seven
server typecheck layers pass. Required final CI supplies the broader
route and server coverage.
- These tests use real PostgreSQL and normal admission, cleanup, queue
consumption, and startup services. The startup suite controls the
provider boundary. Hosted provider acceptance remains pending normal
deployment.

## Risks

- Explicit Send can reopen a done task. This is limited to its validated
saved receipt and current assignee. A cancelled task remains held.
- Reopening uses the existing company-scoped issue transition and shares
the successor transaction. A failed admission does not reopen the task.
Provider Stop, workspace, approval, pause and budget checks retain their
existing owners.
- Cleanup and pending decisions retain the done task and saved receipt.
An unrunnable workspace uses the existing blocked-task diagnostic. No
schema or runtime package changes.

## Model Used

- OpenAI Codex, GPT-6 family; exact serving snapshot/context window not
exposed; repository analysis, code editing, tool execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 17:15:07 -06:00
DottaandPaperclip 06c0566a08 fix: make task Retry work after a refused automatic attempt (#19466)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task Retry stores a durable request before it starts a new run.
> - An automatic attempt can stop at the execution gate before a
provider starts.
> - That cancelled attempt becomes the source of the next Retry click.
> - Admission rejected that source, or chose an older run that did not
match the stored Retry authority.
> - This pull request verifies the refused attempt and keeps it as the
exact authority source.
> - The user can start one new turn after the older execution holds pass
their existing checks.

## Linked Issues or Issue Description

**What happened?**

Retry can remain deferred when the latest run was cancelled with
`execution_reconciliation_required` before dispatch. An older failed
preparation has an execution recovery hold. The Retry receipt names the
cancelled child, but admission only accepts a failed preparation as an
extra source. It can also choose the older run as `previousRunId`, which
fails the exact source check at dispatch.

**Expected behavior**

Retry starts one new turn when the latest attempt has a verified
pre-dispatch refusal and every older execution hold is ready. Provider
activity, retained execution resources, missing receipts, or a live
earlier process keep the hold.

**Steps to reproduce**

1. Seed a failed native preparation and its recovery hold.
2. Let a later automatic attempt be cancelled by the stale queued
execution gate before dispatch.
3. Click Retry against that latest cancelled attempt.
4. Before this change, continuation admission returns no receipt. After
this change, one successor starts with authority bound to the clicked
attempt.

**Paperclip version or commit**

Reproduced on `2f35eb1080`. Rebased onto `6cafcaf89b` for submission.

**Deployment mode**

Local source checkout with embedded PostgreSQL and the native Runner on
macOS arm64.

Refs #15383 and #16939. Those changes cover legacy conversation
cancellation and queue promotion. This change covers the exact source of
a task Retry receipt after a gate refused its automatic child.

## What Changed

- Reuse the existing database-backed pre-dispatch refusal proof when
admitting the exact Retry source.
- Keep the clicked source as `previousRunId`. Continue to check every
older execution hold.
- Add database regressions for safe refusal, unsafe evidence, concurrent
dispatch, and duplicate clicks.
- Add an opt-in acceptance test that drives the authenticated Retry
route through a real Codex turn and task finalization.
- Document the admission contract.

## Verification

- TDD: the new database regression returned `null` on the original code.
It passes with the repair.
- Final head `c43a84b1bf`: 487 tests pass across task Retry, explicit
native continuation, and execution continuation. Server TypeScript and
module boundary checks pass.
- Live acceptance: `PAPERCLIP_LIVE_REFUSED_RETRY=true
PAPERCLIP_RUNNER_BINARY=<current-source-runnerd> pnpm --dir server exec
vitest run --config vitest.config.ts
src/services/native-runtime/refused-task-retry.live.test.ts`.
- The real HTTP request returned 202 on the final rebased head
`c43a84b1bf`. Codex `gpt-6.1-sol` with high effort produced the
requested file. The task became done, both task locks cleared, and
finalization committed. A repeated click reused the same receipt and did
not create another run. The source file stayed unchanged.
- Server TypeScript check and module boundary check.
- `pnpm verify --explain` returned 2 because the execution contract doc
requires broad CI coverage. The required CI checks supply that coverage.

## Risks

This changes execution admission. The existing proof requires the stored
refusal receipt and no provider work or retained execution resources.
Earlier recovery holds, process stop checks, decisions, actor authority,
and task ownership still apply. No schema change or migration.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. The host does not expose an exact author model ID or context
window. The live acceptance provider was `gpt-6.1-sol` with high
reasoning effort.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green — all GitHub Actions checks pass
on `c43a84b1bf`; Paperclip Review completed at 5/5 with no findings
(neutral check conclusion).
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
reviewed `c43a84b1bf` with no actionable findings
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 07:03:54 -08:00
DottaandPaperclip 70d23612dd fix: offer Interrupt when queued-message steering is unavailable (#19450)
## Thinking Path

> - Paperclip lets people manage work performed by AI agents.
> - Task chat saves follow-up messages while an agent runs.
> - A native runner can accept Steer only after an active provider turn
exists.
> - During preparation, or with an unsupported provider, the queue
showed a disabled Steer button and no delivery action.
> - The server also rejected Interrupt for ordinary native queues.
> - This change offers an explicit Interrupt fallback and uses the
existing cancellation and admission path.
> - People can send their saved messages while Stop confirmation,
cleanup, and execution gates remain enforced.

## Linked Issues or Issue Description

Related: #15865 covers mixed legacy queues. #15030 covers steering
acknowledgements. This change covers native queues without available
steering.

**What happened?**

A queued message had a disabled Steer button while the native runner was
still preparing. Clicking it did nothing. The server returned HTTP 409
for an explicit Interrupt request against the same native queue.

**Expected behavior**

Offer Steer when the provider can accept same-turn input. Otherwise
offer Interrupt, labeled with its actual effect. Preserve saved messages
until the previous execution stops and normal admission accepts the
successor.

**Steps to reproduce**

1. Start a task with the Paperclip Runner adapter.
2. Submit a follow-up while the run is preparing, before its provider
turn opens.
3. Observe the disabled Steer button above the composer.
4. With this change, click Interrupt. The existing Stop path cancels the
previous execution and submits the saved queue through normal admission.

**Paperclip version or commit**

Reproduced locally on master at 523818ea9b. The branch is rebased on
2f9d3d43c3.

## What Changed

- Offer Interrupt for unavailable or unsupported native steering when
the parent supplies an interrupt callback.
- Keep Steer for an available provider turn.
- Accept explicit Interrupt for a deferred native queue and record its
run target.
- Bind native continuation to the exact Interrupt queue, run, actor, and
cancellation receipt. This lets one click send the messages after Stop
completes.
- Add component, task-thread, real PostgreSQL route, and continuation
authorization regressions.
- Close the route fixture database pool before disposing its PostgreSQL
instance.
- Document the fallback and its Stop and admission requirements.

## Verification

- Red: two component cases failed because Interrupt was absent. Two
route cases returned HTTP 409, `This queue does not support
interruption`, before the repair.
- Red: stronger delivery assertions also caught `send_predates_stop`
after the first repair. The admission fix covers both the freshness
exception and later eligibility check.
- Green: both route regressions now wait for preparation settlement,
then deliver both saved messages exactly once without a second click.
The complete route suite passes 119 tests. The complete continuation
suite passes 376 tests, including wrong actor, queue, target, and
missing-receipt cases.
- UI: task-thread and queued-message suites pass 394 tests, with one
existing skip. The issue-page suite also passes (the earlier three-suite
run passed 644 tests before the final three UI cases).
- Token gates and `git diff --check` pass.
- Full workspace build and low-memory typecheck pass. The initial
typecheck caught a missing `membershipRole` in an unrelated fixture; the
rebase picked up the correction already on master, and the complete
typecheck then passed.
- The full local `pnpm test:run` passed all six chat shards (1,931
tests). It was stopped during general-server validation after the final
commit passed all PR CI gates. The complete local suite is not claimed
as a pass.
- Earlier route-suite teardown timed out with an open database pool.
Closing that pool fixed teardown; the complete route suite then passed.
- All final-head CI gates pass: 52 successful checks and two intentional
skips. Greptile is 5/5 on the same head with no unresolved review
threads.

## Risks

Interrupt stops the previous execution and starts a successor; it is a
different action from same-turn steering. The UI labels that action
explicitly. Cleanup or execution gates can keep the messages queued.
Existing company access, board permissions, queue revision, run target,
cancellation proof, and admission checks remain in place. No schema
migration is needed.

## Model Used

OpenAI Codex (GPT-6), with reasoning, tool use, and code execution. The
exact model ID and context window are not exposed by this client.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 07:11:38 -06:00
Devin FoleyandPaperclip bf53060904 fix(recovery): continue saved input after verified workspace copy-back (#19422)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution preserves accepted results and task files after a
failure.
> - A later user message can request a new turn after the old execution
stops.
> - File recovery can finish by copying files back or by retaining a
verified archive.
> - The retained-result gate accepted the archive proof but missed a
verified copy-back receipt.
> - This change accepts that existing receipt for terminal failures and
preserves execution authority and prior results.

## Linked Issues or Issue Description

**What happened?**
A task keeps a user message queued with `finalizing_saved_result` after
its old native coordinator reached `terminal_failure`. The accepted
result and a successful, exact `workspace_recovery` copy-back receipt
are saved. The gate only checks the alternate
`workspace_recovery_snapshot` receipt. An older failed result can also
keep a newer failed turn's saved input blocked.

**Expected behavior**
A new user message or deliberate task Retry can continue from verified
recovered files after the old execution stops. Preserve the old result
and unknown action outcomes. Keep an active finalizer or missing stop
evidence blocked.

**Steps to reproduce**
1. Retain an accepted result when a native run reaches terminal failure.
2. Complete host copy-back with the exact run, allocation, descriptor
and final file digest.
3. Save a new user message or request task Retry.
4. Observe that admission still reports `finalizing_saved_result`.

**Paperclip version or commit**
Reproduced against `4d49a3128613b282fb344272ed7ca15f98dea2ac`.

**Deployment mode**
Authenticated native execution with retained workspace recovery. Local
regressions use disposable PostgreSQL and the real heartbeat queue. They
start no model provider or sandbox.

Related work: #19355 covers saved input after Stop. #18267 concerns
manual reconciliation of duplicate legacy holds; it excludes workspace
and system recovery and does not fix this receipt gate. I searched open
PRs and issues before starting this change.

## What Changed

- Reuse the existing host-verified repair receipt when an accepted
result belongs to a coordinator in `terminal_failure`.
- Retain the separate checks for active controllers, environment
termination, live executions, questions, approvals and current user
authority.
- Add message, saved-message and task Retry regressions, plus concurrent
queue delivery and invalid-receipt cases.
- Document this continuation path. Add no schema, API or background
worker.

## Verification

- All 483 tests pass across explicit continuation, task Retry,
quarantined Retry and finalized recovery migration suites.
- The final focused fixture has 17 cases. All pass with the change. With
the original production gate restored, all four positive cases fail and
all 13 safety cases pass.
- Negative cases cover missing, unfinished and mismatched receipts,
another task or source, unrecovered files, a live finalizer, an active
controller, missing remote stop and active execution.
- The real heartbeat path delivers one saved message under concurrent
recovery calls. Later recovery calls create no second turn. The old
accepted result and recovery budget stay unchanged; unknown outcomes
stay unknown.
- The first delivery fixture omitted the newer failed source needed by
the recovery scanner. I corrected the fixture to preserve that history
and repeated the unchanged-code comparison. Earlier failure logs remain
in the verification record.
- `pnpm -r typecheck`: passed.
- `node scripts/check-module-boundaries.mjs` and `git diff --check`:
passed.
- `pnpm build`: passed.
- Full local `pnpm test:run`: running with a 20-minute bound; no
full-suite pass is claimed. All hosted CI checks passed. Greptile scored
the exact head `d04fb98892fd77853ad03e6b44cd612e805abc13` 5/5 with no
findings or review threads.
- Hosted delivery of the already saved input still requires a qualified
staging build containing this change.

## Risks

The new path requires the existing exact repair receipt and a terminal
failure. A coordinator still accepting or committing a result keeps the
previous hold. Stop, owner and live-run checks still run before
admission. The old result is neither applied nor replayed. Receipt
verification adds bounded indexed reads only on this retained-result
failure path. No migration or provider configuration changes are
required.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex in T3 Code, with xhigh
reasoning. The context-window size is not exposed. Used code editing,
tool calls and local test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used, including the available runtime
details
- [x] I have checked ROADMAP.md and confirmed this is a scoped bug fix
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in this PR using the bug-report fields
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal task
id
- [x] I have run the affected tests locally and they pass
- [x] I have added regression tests and updated relevant documentation
- [x] I have considered and documented the risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open findings
- [x] I will address all review findings before requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 00:15:30 -08:00
Devin FoleyandClaude Opus 5.5 4033f7f17f fix(recovery): let a task Retry run keep its bounded transient retry (#19384)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can click Retry on a blocked task. Paperclip then admits one
new run that continues in the saved workspace
> - That run can fail before the provider starts, for example when the
download of the saved workspace snapshot gets a transient `ECONNRESET`
> - The scheduler classifies this failure as transient and tries to
schedule the normal bounded retry
> - The automatic-retry admission accepts only continuations that a user
comment started. It refuses a Retry continuation (`commentId: null`,
`taskRetryRequestId` set) with `continuation_user_authorization_missing`
> - So the task does not retry. Paperclip escalates the restore failure,
and the task becomes blocked again after one network reset
> - This pull request lets a plain task Retry renew its authority for
the bounded transient retry, with its own receipt for each new run
> - The benefit is that a transient setup failure after Retry uses the
normal retry budget and does not block the task

## Linked Issues or Issue Description

**What happened?**
A run started by the task Retry button failed during setup:
`remote_workspace_restore_failed: step=snapshot_download;
storage=transport_closed code=ECONNRESET`. The run event said "The
automatic retry could not revalidate the original user continuation."
(retry reason `transient_failure`, attempt 1 of 2). No retry was
scheduled, and the task became blocked.

**Expected behavior**
A transient setup failure of a Retry-admitted run gets the same bounded
automatic retry as a run that a user comment started.

**Root cause**
`admitExplicitContinuationRetry` in
`server/src/services/explicit-native-continuation.ts` required
`explicitUserContinuation.commentId` to be a GUID. A Retry continuation
has `commentId: null`. Also, dispatch in
`server/src/services/execution-continuation.ts` bound a Retry receipt to
exactly the run that it admitted, so a successor run could not prove its
authority.

Refs #19207 (it named remote restore failures and left a board recovery
action)
Refs #19340 (it retries quarantined tasks in their saved workspace)
Refs #19326 (same bounded retry path for an out-of-date Runner)

## What Changed

- `admitExplicitContinuationRetry` now accepts two kinds of authority.
The comment path is unchanged and moved into
`readCommentRetryAuthority`. The new `readTaskRetryRetryAuthority`
accepts a plain task Retry only. The Retry request must be coalesced,
from a user, with the same generation and actor, strategy `continue`,
and no chat, queued-message, canonical-intent or fresh-workspace
readiness. Interrupt and failed-run retries stay one-run.
- The successor run gets its own `explicit_user_continuation_retry`
receipt with the same `taskRetryRequestId` and actor and
`automaticRetry.sourceRunId` set to the parent. The Retry request stays
bound to the run that it admitted.
- Dispatch (`buildExecutionContinuation`) accepts a Retry successor only
from its own receipt, its exact `retryOfRunId` parent, the current
`executionRunId` and the unchanged Retry intent. The direct path for the
admitted run is unchanged.
- Tests in `server/src/services/task-retry.test.ts`: one regression for
the full retry chain, and eight refusal cases.

## Verification

- Red: before the fix, the new test failed with `{ outcome:
"not_scheduled", errorCode: "continuation_user_authorization_missing"
}`. This is the same refusal as the incident.
- Green: `pnpm --filter @paperclipai/server exec vitest run
src/services/task-retry.test.ts` gives 58/58.
- The regression test proves these points: the first retry is scheduled
with `{ previousRunId: parent, commentId: null, taskRetryRequestId }`;
the Retry request stays bound to the admitted run; dispatch accepts the
successor; a repeated schedule call adopts the same successor without a
second receipt; a second transient failure retries from its immediate
parent; the budget then stops with `retry_exhausted`; dispatch refuses
the successor when its own receipt is removed.
- The refusal cases prove that no successor and no receipt are created
after: wrong actor, unadmitted request, wrong generation, request bound
to another run, chat Retry, fresh-workspace readiness, reassignment,
superseding work.
- Nearby suites pass (601/601): `execution-continuation`,
`legacy-explicit-continuation-recovery`, `explicit-native-continuation`,
`execution-continuation-provenance`, `quarantined-task-retry`,
`issue-queued-comments-routes`, `task-retry`.
- I did not run the layered server typecheck locally because of a disk
limit in the sandbox. CI must confirm it.

## Risks

- Behavior change: a plain task Retry run that fails transiently now
retries automatically within the existing bounded budget (2 attempts).
The scheduler still decides which failures are transient. The same rule
already applies to runs that a user comment started.
- Chat, queued-message and fresh-workspace Retries, failed-run retries
and Interrupt keep their one-run contracts.
- No schema or migration change.
- Documentation gap: `doc/execution-semantics.md` (line 1210) says
"One-run Retry and Interrupt intents retain their separate admission
contracts; they are not renewable user-message authority." That sentence
must change to describe this rule. The author's publication path cannot
carry the full 320 KB file, so that one-sentence edit is still pending.
The proposed text: "A plain task Retry that continues in the saved
workspace renews the same way: the successor's receipt names the same
Retry request and actor, while the Retry request itself stays bound to
the run it admitted. Dispatch accepts that successor only from its own
receipt, its exact retry parent, and the unchanged admitted Retry
request, so a transient failure before or during that run, such as a
reset saved-workspace download, uses the normal bounded retry budget
instead of blocking the task. Chat, queued-message and fresh-workspace
Retries, failed-run retries, and Interrupt intents retain their separate
one-run admission contracts; they are not renewable authority."

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code as a Paperclip agent, with tool use and code execution (shell,
Vitest on embedded PostgreSQL).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (the
`doc/execution-semantics.md` sentence above is pending)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 17:54:00 -08:00
DottaandClaude Opus 5.5 a20dfbc695 fix(recovery): continue a message sent during Stop settlement after recovery (#19355)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can stop or interrupt a running native task and then send a
message to give new direction
> - When the stopped run left a workspace hold, the dispatch gate
withholds the follow-up run until recovery saves the work
> - The gate must save the user's message on its receipt so that the
deferred-receipt worker continues it after recovery
> - Two conditions made the gate skip the message instead of saving it,
so the task stayed in progress with "Saved work recovered. Send a
message to continue." although the user already sent one
> - This pull request saves the message in both conditions
> - The benefit is that the message the user already sent continues
once, automatically, after recovery

## Linked Issues or Issue Description

**What happened?**
After a user stopped or interrupted a native task with an isolated
sandbox workspace and sent a message, the follow-up run was cancelled
with `execution_reconciliation_required`. Its error said "Saved work
recovered. Send a message to continue." Nothing re-woke the task. In the
interrupt case, a periodic sweep started a run about four minutes later.
In the case where the follow-up first waited for the stopped run's
workspace save (`workspace_busy`), each later retry parked on the same
resolved recovery action, and the task did not continue.

**Expected behavior**
When the recovery that the follow-up waited on resolves, the message
that the user already sent continues once, automatically. The product
does not ask for a second message.

**Steps to reproduce**
1. Run a native task in a managed sandbox with an isolated workspace.
2. Stop the run (or send a comment with interrupt), then send a message
while the stop and workspace save are still settling.
3. Wait for the workspace recovery to resolve.
4. See that the follow-up run is cancelled with
`execution_reconciliation_required` and that no run continues the
message.

**Paperclip version or commit**
`master` before this change.

**Root cause**
`deferUnstartedUserComment` (run dispatch) refused to save the withheld
message in two cases:
1. It compared the message time with the source run's `finishedAt`. A
native run finishes only after its provider stop and workspace save
settle, which takes seconds. A message written after the Stop request
but before that settlement was treated as input written before the Stop.
The receipt was marked `skipped`.
2. It required the run's own receipt to be user-owned. A workspace-busy
retry or an admission retry carries the user's message under a
system-owned receipt, so the message was dropped with the retry.

## What Changed

- Add `userStopBoundaryAt(run)` in `run-cancellation.ts`. For a run
cancelled by a user Stop or interrupt, it returns the recorded Stop
request time (`cancellation.recordedAt`) when that is earlier than
`finishedAt`. For all other runs it returns `finishedAt`.
- Use this boundary in the dispatch gate's message deferral, in native
explicit-continuation admission, and in the remote-Stop continuation
check, so the three places agree on "message written after the Stop".
- When the withheld run is a system-owned retry, walk a bounded chain
(max 8) of retries that did not reach the provider (`workspace_busy`
with `providerWorkStarted: false`, or a verified pre-dispatch
`execution_reconciliation_required`) to the originating user receipt.
Save the message on that receipt with the user's original wake context.
Mark the retry's system receipt `skipped`.
- Update the "input written before the Stop" test so that the Stop
request, and not only the settlement, comes after the message.

## Verification

- New regressions in `server/src/services/workspace-recovery.test.ts`
(`continues a message ... without a second message, after recovery
already resolved`): a direct follow-up and a workspace-busy retry. Each
runs recovery to completion first, then lets the gate withhold the
follow-up, then runs only the deferred-receipt worker. Each asserts that
exactly one successor run is queued with the user's message and the
user's authority, and that no provider work starts.
- Red before the fix: both tests fail because the receipt is `skipped`
instead of `deferred_issue_execution`. Green after the fix.
- `npx vitest run src/services/workspace-recovery.test.ts` (384 passed).
- `npx vitest run src/services/explicit-native-continuation.test.ts
src/modules/run-dispatch src/services/execution-continuation.test.ts
src/services/run-cancellation.test.ts` (581 passed).
- 19 more heartbeat, retry, queued-comment, and native recovery suites
(1272 passed). One failure, `heartbeat-process-recovery > redacts opaque
environment-bound credentials from Sentry diagnostics: result`, also
fails on unchanged `master` and is not related.

## Risks

- A message written after the Stop request but before the stopped run
settled now continues. The stopping run cannot consume new input after
its Stop request, and admission still removes comments that a run
already delivered (`undeliveredLegacyUserCommentIds`). All admission
gates (holds, stop proofs, approvals, budget) still apply.
- Only user Stops and interrupts move the boundary. Other cancellations
keep `finishedAt`.
- The retry walk is bounded, stays in one company, agent, and issue, and
accepts only retries without provider work. It revives a user receipt
only from `cancelled` or `skipped`, so a second retry cannot save the
same receipt twice.
- Not in scope: a run that fails during native preparation with no user
message (for example `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=unexpected_root_entry` on a system wake). After its
recovery resolves, the task still waits for a user message. That is a
separate root cause in the native session executor's prior-owner
retirement proof.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 18:08:24 -06:00
DottaandPaperclip 789a71bc17 fix(runner): retry quarantined tasks in their saved workspace (#19340)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native sessions retain execution ownership and action history across
turns.
> - Task Retry must prove that old execution has stopped before it
starts another turn.
> - An attempt can inherit its predecessor's identity and fail before it
launches a provider.
> - The cleanup proof searched under that failed attempt and compared
unrelated event sequences.
> - This pull request verifies the retained owner and starts one fresh
authorized turn.
> - Saved results and uncertain action outcomes remain intact.

## Linked Issues or Issue Description

**What happened?** Task Retry accepted repeated clicks but kept the
canonical request deferred with `executionWait.reason: local_cleanup`.
It created no successor, although the retained local worker and provider
had stopped.

**Expected behavior:** After exact cleanup proof, admit one fresh turn.
Preserve the old journal and result. Keep unresolved action outcomes
unknown.

**Steps to reproduce:**
1. Finish a local native turn with retained cleanup state and a saved
result.
2. Let the next attempt inherit its identity and fail with
`native_session_cleanup_quarantined` before any launch.
3. Upgrade across a tool-contract change, then click task Retry.
4. The old code requires the failed attempt's stop receipt and journal
identity. It also requires the current tool fingerprint and equal
sequence numbers across raw and normalized events. Those requirements
cannot match the retained owner.

**Paperclip version or commit:** Reproduced on `31bf6e4482`.
**Deployment mode / installation method:** Authenticated, self-hosted
source checkout.
**Agent adapter(s) involved:** Native `paperclip_runner`, local Codex
app-server. The shared physical-stop proof also covers ACPX.
**Database mode:** Embedded PostgreSQL 18.1.
**Access context:** Board task Retry.
**Node.js version / operating system:** Incident: Node 24.20.0, Linux
ARM64. Local verification: macOS ARM64.
**Relevant logs or output:** Accepted Retry stays in
`deferred_issue_execution`; no new run or provider launch occurs.
**Relevant config:** Per-turn local native execution. The old tool
contract differs from the current contract.
**Additional context:** Related: #19311 prevents the oversized tool
result that can lead to retained cleanup. #18217 covers broader cleanup
isolation and retained-result maintenance. #19134 admits stopped Runner
sources under the current run lease. This patch fixes explicit
continuation admission on current master.

## What Changed

- Resolve the retained journal's owner under the same company, task,
agent, workspace, native session, and runner instance.
- Require terminal ownership, settled controllers, released local
leases, and exact inherited process birth. Refuse any launch evidence on
the failed attempt.
- Accept a current host's verified process absence from its exact launch
receipt when an older server lacks a stop receipt. A later launch
invalidates that observation.
- Validate normalized receipts against their own IDs, sequences, and
digests. Match ordered provider identities to the independent raw
journal.
- Keep tool compatibility at the session reuse boundary. Explicit
continuation always uses a fresh session.
- Bind the stop proof to its exact successor and canonical Retry
receipt. Reuse the proven local workspace through normal preparation.
Refuse changed bindings, closed or remote workspaces, pending
operations, and other live leases. Projectless run-ID placeholders keep
the existing fresh-workspace fallback.
- Retain journals, results, files, and uncertain action outcomes. Add
regression and opt-in live acceptance coverage.
- Make the split-handoff browser test wait for both child-completion
notices and all run settlement before teardown. Assert the final task
set stays unchanged.

## Verification

- Red: canonical Retry on the original code accepts one saved intent but
creates zero successors and records `local_cleanup`.
- Red: distinct raw and normalized event sequences reproduce that wait
after owner resolution alone. After admission was repaired, live
acceptance then exposed the empty-attempt workspace problem. A
PostgreSQL regression also reproduced that wrong workspace selection.
- Green at `6684df5e82`: **479 targeted cases**, including 26 new
PostgreSQL cases for duplicate clicks, exact process and workspace
ownership, controller/lease fences, changed state, and independent
receipt integrity. The complete 1,150-test native executor suite passed
earlier.
- `pnpm -r typecheck` and `pnpm build` pass again at final head
`432a52bea8`, including its build stamp. Token gates pass. Master now
contains the unrelated directory-fixture type correction found during
the first typecheck.
- Real authenticated HTTP Retry with Codex `gpt-6.1-sol`, high effort,
and per-turn execution passes again at final head `432a52bea8` (93.3
seconds). It preserves the first file line, appends the second once,
starts a fresh session in the existing workspace, keeps the retained
journal unchanged, commits finalization, and clears task locks.
- The strengthened split-handoff browser scenario passes **three
consecutive local runs**. Its original CI teardown and one same-head
rerun failed when cleanup cancelled an active legacy
completion-notification run and hit a PostgreSQL row-lock timeout. The
scenario now waits for the complete handoff before cleanup. [Initial
failure](https://github.com/paperclipai/paperclip/actions/runs/42106466518/job/127523693634),
[same-head
rerun](https://github.com/paperclipai/paperclip/actions/runs/42106466518/job/127531698321).
These failures remain part of the verification record.
- The local full `pnpm test:run` passed all six chat integration shards.
Its obsolete pre-rebase attempt was stopped during the general server
phase; no complete local-suite pass is claimed. Full CI, including both
required gates, passed at `6684df5e82`. At final head `432a52bea8`, all
initial CI groups passed except the unchanged status-card deadline test,
whose post-wakeup clock assertion missed its bound by 10 ms. That exact
case passed three unchanged local repetitions (16.8/27.0/18.3 seconds);
the single isolated shard rerun passed all **2,218 tests** (123 files),
with no source changes. [Initial timing
failure](https://github.com/paperclipai/paperclip/actions/runs/42114473652/job/127550722187).
- Greptile reviewed `6684df5e82` at **5/5**, with no actionable
findings. A second reviewer identified the supported projectless
placeholder case. A new real PostgreSQL regression reproduced its
workspace-binding failure before the one-line fallback correction; all
**45 focused cases** now pass, including default environment selection
and changed binding/cwd refusal. Greptile reviewed final head
`432a52bea8` at **5/5**, with no blocking findings. The second
reviewer’s thread is resolved.
- The patched Linux build verified the actual retained worker/provider
ownership and process absence in a transaction that was rolled back. No
retained state was retired or provider started during that
pre-deployment check.

## Risks

- This changes an execution admission proof. Incorrect ownership could
permit a second executor. The proof retains process, controller, lease,
scope, journal, and receipt fences, and duplicate admission is tested
through the real queue.
- Missing or conflicting evidence remains deferred. Windows, remote
execution, automatic replacement, and session reuse retain their
existing policies.
- No database migration or UI change. The original affected task still
needs deployment of the fix before recovery can be claimed.

## Model Used

- OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, SSH diagnostics, and live provider testing. This session does
not expose the exact backend model build or context-window size. The
live test agent uses OpenAI `gpt-6.1-sol` at high effort.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 16:04:38 -06:00
DottaandClaude Opus 5.5 b8e8ab4641 fix(recovery): continue saved input after a recovered sandbox stops, not after its destroy (#19224)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A board user can stop or interrupt a native run and send a message
for the next turn
> - When the stop leaves sandbox files unsaved, automatic recovery saves
them, re-stops the sandbox, and the saved message continues
> - For agents whose sandbox is destroyed after each turn (an ephemeral
lease), recovery then queues the destroy. That step removed the lease's
stop receipt
> - The continuation gate waits for a stop receipt, so the message
waited for the destroy and its cleanup retry, about 5 to 8 minutes
> - Agents with retained sandboxes continued within seconds, so the
delay looked agent specific
> - This pull request keeps the verified stop on the cleanup lease and
lets the continuation and the next remote writer accept it
> - The benefit is that a redirect, Retry or interrupt continues right
after its files are saved, for every lease policy

## Linked Issues or Issue Description

Refs #19206

**What happened?**
On hosted staging, a Stop or `interrupt: true` comment on a native run
left a follow-up turn that started 5 to 8 minutes later for some agents,
and 25 to 40 seconds later for others. The slow agents use ephemeral
sandbox leases. In each slow case, the stopped run's lease was released
by `pending_cleanup_retry` about 5 minutes after recovery, and the saved
message started one second later. The fast agents use retained leases,
which recovery releases at once.

**Expected behavior**
The saved message continues right after recovery saves the files and the
sandbox is stopped. The destroy of a stopped sandbox is resource cleanup
and must not hold the next turn.

**Steps to reproduce**
1. Use an agent whose sandbox lease is ephemeral (destroy after each
turn).
2. Interrupt or stop its native run so that recovery must save the
sandbox files.
3. Post a message. See the follow-up start only after the sandbox
destroy finishes.

**Paperclip version or commit**
Hosted, at `56517dda`.

## What Changed

- `server/src/services/workspace-recovery.ts`:
`scheduleRecoveredWorkspaceCleanup` keeps the verified stop receipt as
`workspaceRecoveryCleanup.stopReceipt` when it queues the destroy.
- `server/src/services/remote-execution-termination.ts`: new
`hasRecoveredCleanupStopReceipt(lease)`. It is true only when the lease
has recovery cleanup for its own run, was released, and the carried
receipt matches the lease, run, provider and provider lease id.
- `server/src/services/explicit-native-continuation.ts`: the
`remote_cleanup` gate accepts that receipt.
- `server/src/services/execution-blocker.ts`: `hasPendingRunCleanup`
ignores a lease with that valid receipt, so a saved native interrupt
send (and Retry or fresh-workspace checks) does not wait for the
destroy. Other cleanup holds are unchanged.
- `server/src/services/remote-workspaces.ts`: the next remote writer
accepts that receipt for the ended writer's lease.
- `server/src/services/execution-blocker.ts` (from #19206 review, not in
the merged head): pre-turn evidence uses the exact startup lifecycle
event list, so turn-scoped events such as `workspace.diff.recorded`
count as provider work.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- Red before the fix, green after: `cd server && npx vitest run
src/services/workspace-recovery.test.ts -t "admits the interrupt
redirect"`. An interrupted native run with recovered files and a queued
destroy blocked the user message with `remote_cleanup` before the fix.
After the fix the message is admitted. A lease without the stop receipt,
or with a receipt for another run, still blocks with `remote_cleanup`.
The test runs the same input as a plain message and as a saved native
interrupt send (`queuedCommentInterruptId`); `hasPendingRunCleanup` was
true before the fix and blocked the interrupt send with
`cleanup_pending`.
- `cd server && npx vitest run
src/services/remote-workspace-reservation.test.ts -t "ended writer"`:
the next writer reserves the workspace from an ended writer whose
recovered lease awaits destroy; a missing or foreign receipt still
blocks. Removing the new branch fails the positive case.
- Updated `saves the authoritative remote workspace after native failed
...`: Retry now continues before the destroy, and the cleanup lease
carries the stop receipt for its run and lease.
- `workspace-recovery.test.ts -t "provider work cannot be excluded"`
includes `workspace.diff.recorded` as provider work.
- Related suites pass: `workspace-recovery.test.ts`,
`explicit-native-continuation.test.ts`,
`remote-workspace-reservation.test.ts`,
`remote-workspaces.integration.test.ts`,
`remote-workspace-lifecycle.test.ts`, `task-retry*.test.ts`,
`native-cancelled-workspace-preservation.test.ts`,
`workspace-recovery-source.test.ts`. `cd server && npx tsc --noEmit`
passes.
- Hosted evidence (read only): slow cases have ephemeral leases released
by `pending_cleanup_retry` about 5 minutes after
`issue.workspace_recovered`, with settlement one second later. Fast
cases have retained leases released by recovery.

## Risks

- Low to medium. The next turn can start while the old stopped sandbox
is still being destroyed. The receipt proves that the sandbox was
stopped after the files were saved, and an ephemeral successor uses a
new sandbox. A destroy failure stays with the existing cleanup retry.
- Leases without the carried receipt behave as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 05:08:03 -06:00
DottaandClaude Opus 5.5 46a50a3bdd fix(recovery): release a recovered workspace hold for the new assignee after reassignment (#19184)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a board user reassigns a task in the middle of a run, Paperclip
stops the old run and saves its sandbox files (copy-back).
> - If copy-back fails at the stop, Paperclip records a
file-preservation hold for the stopped run. The hold blocks all
execution on the task until the files are recovered.
> - The new assignee's wakes were already queued. The final dispatch
gate saw the hold and cancelled them.
> - Automatic recovery then saved the files. But the hold stayed
effective for the new assignee: explicit continuation accepts only the
stopped run's own agent, so no message, Retry or stranded-work sweep
could start the new assignee.
> - The task stayed `in_progress` with no run, no wake and no active
recovery.
> - This pull request releases a recovered file-preservation hold for a
different current owner.
> - The benefit is that a reassigned task continues with its new owner
after its files are safe, with no manual action.

## Linked Issues or Issue Description

**What happened?**
A board user reassigned a running native task from agent A to agent B
and posted a comment. A's run ended `cancelled` / `issue_reassigned`.
Copy-back was still in progress, so B's first run was cancelled
`workspace_busy`. Paperclip then recorded a resolved no-replay hold
(`native_continuation_requires_reconciliation`, fingerprint
`native-workspace:<A run>`). The final dispatch gate cancelled B's two
queued runs with `execution_reconciliation_required` ("Recovering saved
work…"). Automatic recovery then saved the files
(`workspaceRecovery.status = recovered`, `workspaceRepairRequired:
false`). The blocker still said "Saved work recovered. Send a message to
continue." with `canContinue: false`. The task stayed `in_progress` with
no run for hours. Two hosted reproductions had the same shape. A sibling
reassignment without a copy-back failure recovered normally.

**Expected behavior**
After the saved-file receipt verifies, the new assignee continues. The
hold must still block every executor while the files are not recovered.
The stopped run's own agent keeps its no-replay hold.

**Steps to reproduce**
1. Start a native run on a task with a retained sandbox.
2. Reassign the task to another agent while the run executes, so that
copy-back records a `workspaceRestoreRecovery` hold.
3. Let automatic workspace recovery finish.
4. Before this change, `getExecutionBlocker` still returns the hold for
the new assignee, and the stranded-work sweep queues nothing.

**Paperclip version or commit**
`b9fbbf8d30` (also seen on `818d2bb6`).

**Deployment mode**
Hosted, PostgreSQL. The test uses embedded PostgreSQL.

Related open pull requests (checked; none covers this path): #19177
(board reconciliation of sibling holds), #19169 (disposition repairs
behind a cancelled blocker), #19143 (park a workspace-busy wait), #19149
(interrupt stop identity).

## What Changed

- New `restoreHoldReleasedForCurrentOwnerPredicate`
(`server/src/services/execution-blocker.ts`) is a SQL predicate. It
matches a file-preservation hold (`workspaceRestoreRecovery`, not
`restore_unsafe_archive`) when all of these are true:
  - the stopped run is terminal and finished;
- the task has an agent assignee, and that agent is not the stopped
run's agent;
  - the stopped run's agent is not the current review participant;
- `completedWorkspaceRecoveryCondition` is true for the hold: completed
system recovery, `workspaceRecovery.status = recovered`, succeeded
preservation operations bound to this hold, and the `workspace_finalize`
receipt;
- every saved operation is a copyback into the task workspace
(`workspace_recovery`). A snapshot-only save keeps the hold, because the
new owner's ordinary turn does not attach snapshots.
- The predicate is wrapped in `coalesce(..., false)`. Its negation
therefore cannot hide an unrelated hold through NULL JSON reads.
- The proof is the durable completed-recovery receipt. It does not read
live workspace authority (epoch, writer, snapshot), so reuse of the
workspace by the new owner cannot re-arm the hold.
- `getExecutionBlocker` excludes matching holds from both the
file-preservation query and the general blocker query. There is no scan
limit and no extra query per hold.
- `admitExplicitNativeContinuation` excludes the same holds from its
source checks. A released hold therefore cannot reject a new owner's
message with "owner changed".
- `completedWorkspaceRecoveryCondition` also accepts a SQL expression
for the run id. Its behavior is unchanged.
- The hold row, its `automaticRecovery.replay = "blocked"` evidence and
its status are not changed. If the task returns to the stopped run's
agent, the hold applies again.
- `doc/execution-semantics.md` describes the rule next to the existing
"Reassignment does not discard the stopped run's files" sentence.

## Verification

New test in `server/src/services/workspace-recovery.test.ts`: `hands a
reassigned task to its new assignee once the previous owner's files are
recovered`. It uses the real retained-workspace recovery, embedded
PostgreSQL and the real stranded-work sweep. It builds the incident
shape: a cancelled `issue_reassigned` source run, a resolved no-replay
hold, a new assignee, and that assignee's gate-cancelled run.

- Before recovery, the blocker holds the new owner
(`workspaceRepairRequired: true`).
- After recovery, the blocker is `null`. The sweep creates exactly one
pending turn for the new owner. That turn is not a retry of the stopped
run. A second sweep adds no turn.
- The hold row keeps `status: resolved`, `outcome: blocked` and `replay:
blocked`.
- Live repair evidence on the stopped run is then changed, so
`hasVerifiedWorkspaceRepair` returns false (as after a remote reuse).
The blocker stays `null`.
- After a reassignment back to the stopped run's agent, the hold applies
again.
- An unrelated reconciliation hold for the same stopped run, without
file evidence, still blocks the new owner.
- New test `keeps a snapshot-only recovery held for a new assignee`: the
snapshot-only save does not release the hold. This test fails against
the first revision of this pull request.

Red before the fix (test kept, `execution-blocker.ts` restored from
`master`):

```
cd server && npx vitest run src/services/workspace-recovery.test.ts -t "hands a reassigned task"
AssertionError: expected { …(13) } to be null   (blocker: "Saved work recovered. Send a message to continue.", canContinue: false)
```

With only the blocker assertion removed, the sweep also failed:
`expected [] to have a length of 1 but got +0`.

Green after the fix: 1 passed.

Related suites, all pass (991 tests): `workspace-recovery`,
`execution-blocker-read`, `legacy-explicit-continuation-recovery`,
`workspace-restore-reconciliation`, `native-failed-workspace-recovery`,
`workspace-restore-repair-route`, `execution-projection-recovery`,
`heartbeat-undispatched-execution-wait`,
`issue-update-comment-wakeup-routes`,
`heartbeat-observed-workspace-preservation`,
`native-workspace-reconstruction`, `explicit-native-continuation`.

Not run: a live run on a deployed instance. The hosted incident tasks
are not recovered by this pull request until it is deployed.

## Risks

- A new owner can start after the files are recovered, while the stopped
run's action outcomes stay unknown. This matches the existing rule for
holds without a workspace: Paperclip does not create those holds for a
reassigned task. The new owner gets a fresh turn, not a replay.
- The hold still blocks every executor until the exact receipt verifies,
so there is no second executor on the workspace during copy-back.
- Other execution holds (non-workspace holds recorded before a
reassignment) keep their current effect. This change does not alter
them.
- A reassignment after a snapshot-only save still needs a board action,
as before. That path is not changed here.
- The release is one correlated `exists` in the existing blocker
queries. Its subqueries use primary keys and the run and issue indexes.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, with tool use
and code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 23:23:21 -06:00
7d1e9b3477 fix(recovery): retain exact warm native export sources (#17916)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native task recovery must save unfinished workspace exports before
continuation.
> - A prepared export can still refer to a sandbox with a warm reuse
policy.
> - The strict recovery inspector rejects that policy and exhausts
source verification.
> - Ordinary reuse must not give the same unfinished allocation to
another run.
> - An existing user Retry can authorize bounded file maintenance after
exact ownership checks.
> - This change retains that exact source and preserves the original
failure record.
> - Recovery completes only when files, stop evidence and policy
settlement agree.

## Linked Issues or Issue Description

**What happened?**

A failed native run has a prepared `keep_running` workspace export. Its
exact sandbox lease still has `reuse_by_environment` policy. Native
export cleanup can preserve that policy while it stops the sandbox.
Generic workspace recovery then rejects the source with
`lease_policy_not_retained`. A saved user Retry remains blocked before
file export.

**Expected behavior**

Keep unfinished exports out of ordinary warm reuse. With current user
authority and an unchanged saved Retry, retain the exact source for one
bounded maintenance generation. Preserve files and return continuation
to normal admission. Keep all ambiguous sources held.

**Steps to reproduce**

1. Prepare a native workspace export with a warm sandbox and
`keep_running` disposition.
2. Save a normal board Retry for the failed source run.
3. Persist its first exhausted source-verification failure before file
export.
4. Run the normal maintenance worker. Before this change, source policy
rejection leaves the saved files unrestored.

Refs #17773 for source diagnostics. Related #17839 records a missing
preservation obligation during blocked restart. This change handles an
existing prepared warm export whose verification already failed.

## What Changed

- Exclude unfinished physical allocations from reuse and obsolete-lease
cleanup across every historical lease row. Reuse a clean candidate or
acquire a fresh sandbox through normal selection. Preserve exact
same-run reattachment, workspace-less execution and ordinary task
admission checks.
- Preserve the newer prepared remote-workspace writer proof for native
and legacy runs. Share its binding check between single-source recovery
and one batched allocation read.
- Keep persisted recovery-action IDs when constructing continuation
source checks. Synthetic sources cannot grant snapshot receipt
authority.
- Permit one audited maintenance adoption for the first exhausted
source-verification attempt. Require an unadmitted canonical saved board
Retry, current user authority, unchanged assignment and latest source
run.
- Keep all company, source, allocation, workspace writer, controller and
physical finalization ownership checks. The public read-only inspector
keeps its strict policy check.
- Pin the exact source for file maintenance. Preserve the previous
generation and reuse the existing bounded recovery worker. Create no
provider turn or new user request.
- Restore the warm policy in the same transaction as verified native
copyback completion. A failed write or changed settlement keeps a
recovery obligation.
- When local merge metadata is missing, settle the exact verified
snapshot while keeping its unfinished allocation retained. Deliver the
saved Retry through normal fresh isolation and attach the verified
files. Keep the accepted result as history.
- Require native copyback proof before reopening an accepted result's
original workspace finalizer.
- Validate snapshot receipts through one shared boundary for warm,
retained and ephemeral sources. Bind the receipt to the current
generation's operation IDs, company, issue, run, recovery action, lease
and provider allocation.
- Retire fixture input before removing recovery proof, and use the same
configured heartbeat for HTTP and worker dispatch. Keep the exact
catalog refusal and no-provider-work assertions.
- Document the recovery contract in technical documentation. Leave
capability maps unchanged for this bug fix.

## Verification

- Candidate-selection RED: all six clean-candidate and fresh-allocation
controls fail on unchanged `f54e7baf538` because a pending export aborts
the whole acquisition. GREEN: the focused suite passes **15/15 cases**
after candidate filtering. Both provider paths preserve the protected
lease and never resume or destroy its allocation. Obsolete fingerprints
cannot send that allocation to cleanup. Historical-row fencing, modern
native/legacy writers, exact same-run reuse, workspace-less execution
and the single-read allocation-history check remain covered.
- The existing reusable-runtime selection cases passed **40/40**.
Driver/capability contract suites passed **5/5**. Reusable service
controls passed **2/2**. After merging current master, the combined
lease/provider suites pass 61 cases and the recovery integration
selection passes 36 cases, including the upstream inherited-session and
resolved-interruption assertions. This gives 97 passing targeted cases
on the final source. A duplicate test-app declaration from the automatic
merge was removed while retaining the upstream assertions.
- The source-history repair has meaningful red/green evidence: three
valid retired-handoff cases could not complete under the old blanket
history-row rejection. Settlement now uses the same full source
inspector as adoption. It accepts exact retired handoffs and still
rejects a predecessor that becomes active after export. The 26 focused
history, rollback, snapshot and ambiguity cases passed before this last
merge.
- Server typecheck passes on the final merged source. Module boundaries
and `git diff --check` pass. Full local repository tests and a
successful current-head full build are not claimed. Current-head CI is
complete on `0ef974d76483e11fb0b0304bcb05bbd3b56e2ef9`: both required
gates pass, with 54 successful checks, three expected skips, one neutral
Paperclip review check and a successful security status. [Full CI
evidence](https://github.com/paperclipai/paperclip/actions/runs/41904339370).
Greptile reviewed that exact head at **5/5** with no actionable
findings. Its check completed successfully. [Ponytail also reviewed the
exact head at
5/5](https://github.com/paperclipai/paperclip/pull/17916#issuecomment-6640083408).
No review threads remain unresolved, and the branch is mergeable.
- Earlier local broad runs exposed a dropped recovery-action ID in the
merged source projection and deferred test input being revisited after
its records were removed. Both were repaired. Later local disk
exhaustion caused manifest and saved-file reserve failures; generated
caches were removed only from these checkouts. The affected saved-file
control passed unchanged after cleanup. The original logs are retained.
No disk reserve or timeout was lowered.
- On the prior head, the first [full CI
attempt](https://github.com/paperclipai/paperclip/actions/runs/41899038530)
received simultaneous runner shutdowns. Four jobs were cancelled, and
two late jobs have lost-runner annotations with unavailable logs. The
Runner job had five earlier ACPX failures whose full assertion report
was lost on interruption. Their cause remains unproven. Identical Runner
source passed the [#19004
control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429725),
including all seven enabled tests in that file.
- That attempt's [browser
shard](https://github.com/paperclipai/paperclip/actions/runs/41899038530/job/126817404595)
returned to the exact saved task URL, but its conversation was still
loading at the unchanged heading deadline. The
[artifact](https://github.com/paperclipai/paperclip/actions/runs/41899038530/artifacts/13358840931)
is retained. All five Cloud-auth cases passed locally on the unchanged
head and in the [same-source CI
control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429966).
The delay remains unexplained. One rerun of failed/interrupted jobs was
submitted after the workflow completed. This new source revision
requires fresh CI.
- The earlier [original-head
workflow](https://github.com/paperclipai/paperclip/actions/runs/40552228175)
passed all 48 jobs, and prior head `f54e7baf538` received Greptile 5/5.
These results do not establish readiness for the new head. The earlier
Superagent scan was neutral after a pinned-base checkout failure; its
external analysis remains unverified.
- These tests use real PostgreSQL and controlled provider boundaries.
Hosted rollout and original-task acceptance remain unverified.


## Risks

- Maintenance can resume and stop the exact sandbox for file export. It
cannot admit a provider turn, choose another allocation or bypass normal
continuation gates.
- A saved Retry alone is insufficient. A changed source, newer run,
active writer, competing lease, lost user authority or previous adoption
keeps the hold.
- Policy restoration can fail. Completion and restoration now commit
together, so normal recovery remains responsible for a retry.
- Snapshot preservation cannot repair missing local merge metadata. Its
old allocation remains explicitly retained. A snapshot receipt grants no
permission to reuse or destroy that allocation, or to replay the
accepted result.
- The allocation check uses one projected join per candidate over the
existing provider lease index. It adds no polling timer or recurring
scan. A fresh fallback can create another sandbox while the unfinished
source remains retained; it grants no authority to execute a provider
turn.
- This change does not establish the cause of a prior provider failure
or prove that any specific hosted sandbox still contains its original
files.

## Model Used

OpenAI Codex, GPT-6 family, for the original change. OpenAI GPT-6.1 Sol
(`gpt-6.1-sol`), xhigh reasoning, for the merge and integration repairs.
Both used repository tools and PostgreSQL verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-11-04 14:47:42 -08:00
53735db7de fix(retry): resume verified preparation failures (#19004)
## Thinking Path

> - Paperclip must preserve saved agent work while proving earlier
execution stopped.
> - A deliberate task Retry can observe a newer failure while an older
preparation still holds execution.
> - That older preparation can also have a finalized empty object log
despite never reaching the provider.
> - Binding only the latest run and requiring every log field to be
absent keeps valid recovery deferred.
> - This change follows the exact persisted predecessor chain and binds
current Retry authority to eligible holds.
> - It recognizes only an exact finalized empty-log receipt within the
existing strict preparation proof.
> - One fresh user turn can then start with saved work and the existing
ownership, stop and accounting checks.

## Linked Issues or Issue Description

Refs #18989, #18860 and #18997. The first waits for predecessor cleanup
during automatic dispatch. The second selects the committed native
session owner. The third retains admitted Retry authority when no source
recovery action exists. None passes current board Retry intent to a held
failed predecessor. A search of open repairs found no duplicate.
ROADMAP.md was checked; this is a recovery bug fix.

**What happened?**

A failed native preparation retains a legacy run row and an inherited
session. Its remote workspace is later saved and verified. An automatic
successor also fails during preparation. Board Retry observes that newer
run, but the effective hold belongs to its predecessor. The Retry does
not bind the predecessor's recovery generation. The native
historical-source guard then keeps the request deferred.

**Expected behavior**

Carry the deliberate Retry through its verified same-company, same-task
and same-agent persisted predecessor chain. Start one fresh user turn
after every source passes current execution, stop and saved-work checks.
Keep incomplete or unrelated proof held. Preserve prior work and
recorded action outcomes.

**Steps to reproduce**

1. Fail native preparation with an inherited session and a remote
workspace.
2. Finish its real workspace recovery and stop receipts.
3. Save a newer failed automatic successor with `retryOfRunId` bound to
that source.
4. Submit ordinary task Retry for the newer run.
5. Before the change, no successor is admitted. The request remains
deferred behind the older hold.

**Paperclip version or commit**

The regression fails on master
`039e0dad2c51c80251e1302a19c5566e039c357a`, which includes #18989.

**Deployment mode**

Source checkout on macOS arm64 with real PostgreSQL, HTTP task Retry and
the assembled heartbeat service. The remote filesystem boundary uses
temporary files. Hosted rollout and an original-task Retry are not yet
verified.

This PR incorporates the reviewed empty-log follow-up #19021, preserving
its commits and regression controls. Both parts are required for hosted
preparation recovery, so the combined revision is qualified and deployed
together.

## What Changed

- Accept a finalized zero-byte `s3_segments` log only when its reference
binds the exact company, agent and run, its digest matches empty
content, compression is false, and no output progress or nonempty
excerpt exists. Add a real-recorder HTTP regression and 14 negative
controls. Unknown or nonempty logs stay held.

- Read a complete persisted Retry chain in one recursive query, limited
to 32 runs. Reject missing, cyclic, newer, unfinished, cross-company,
cross-task and cross-agent sources. Transcript hints grant no authority.
- Bind eligible historical recovery records to the current Retry intent.
Keep complete or running recovery budgets. Leave successful ancestors
and mismatched recovery-source records unchanged.
- Recheck that same chain when selecting an ancestor's fresh-user
strategy.
- Admit only a held ancestor through the shared native preparation
proof. Keep the closed preparation failure codes, no-provider state,
profile, input, result, output and accounting checks. Treat inherited
session input as prior context and accept the exact bounded host start
marker. Verify current ownership, saved work and stop receipts before
forcing a fresh session.
- Add HTTP and PostgreSQL regressions and document the rule.
- Give the HTTP Retry nudge and explicit test worker the same real
heartbeat facade and provider boundary. Establish the revoked catalog
receipt before either can dispatch.

## Verification

- Combined integration head `91f021f233151192f69dd7a035d083abcc67c47c`
requires fresh complete CI and review; qualification is in progress.

- Repair validation: all **six** recovered preparation provider-boundary
cases pass. The catalog cases still require the exact access refusal,
zero provider inputs and zero provider turns. The success case still
checks preserved file bytes and one native turn.
- [Previous exact-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785508136)
failed the catalog case with `remote_workspace_transport_unsupported`.
The route created a default heartbeat that could race the injected
worker and allocate outside the controlled transport. The repaired test
shares its real heartbeat across both dispatchers and holds the agent
start lock until the revoked catalog receipt is committed. It tracks and
drains both Retry nudges during teardown. It changes no production
access check or expected refusal.
- The first broad local reproduction hit a full local disk. It passed
201 cases and failed 86, including explicit manifest free-space errors.
Generated build caches were removed from these isolated checkouts. The
owning run passed 670/671 cases. Its one failure was the saved-Files
loader refusing insufficient temporary disk space. After cleanup, that
unchanged test passes in isolation. The final shared-facade and
start-lock version passes all six dispatch cases. Both legacy
continuation suites pass all 29 cases. The final fixture teardown also
retires deferred input before removing recovery proof, so global queue
sweeps cannot revisit earlier cases. All seven dispatch and saved-Files
controls pass together after that change. No disk reserve, assertion or
timeout changed. The initial local typecheck also stopped on `ENOSPC`
while materializing provider assets. Server typecheck now passes after
cleanup. No test timeout or reserve was lowered.
- Current-head CI is complete: 52 successful checks, three expected
skips, and a successful security status. Both required gates (`ci / e2e`
and `ci / verify`) passed on `0adc2e38103533dd45bcc56f957ac019642c57b1`.
[Full CI
evidence](https://github.com/paperclipai/paperclip/actions/runs/41898305521).
Greptile scored final head `0adc2e38103533dd45bcc56f957ac019642c57b1`
5/5 with no blocking findings or unresolved threads. The earlier
verification below describes the previous head.

- RED on unchanged master `039e0dad2c`: the certified-preparation
sibling passes, while the inherited-session case admits zero successors
instead of one. A second isolated RED on latest base `8f06bd4e17`,
including #18997, uses the exact final three-node test fixture. The HTTP
route returns 202, then the test observes zero successors instead of
one. All baseline production files remain unchanged. The initial
dependency/setup failures are separate from these behavioral failures.
- The positive cases cover the failed-failed-successful chain. They keep
the completed result unchanged, verify the original saved file and
validate fresh-session continuation authority. Repeated HTTP input and
concurrent resume calls keep one successor.
- PostgreSQL controls accept a complete 32-run chain. A 33-run chain and
a newer-created predecessor return no usable ancestry and do not bind
the older recovery record.
- Safety controls cover company, task and agent scope, conflicting
canonical bindings, missing and cyclic edges, transcript hints, live
execution and controllers, unknown stop, failed saving, unsafe writer
state, Stop, actor and generation changes, independent holds and
mismatched recovery records.
- Ancestor controls retain provider-event, native-input,
session-checkpoint, controller-recovery, result, process, runtime,
accounting and unknown-result guards. Host start and warning prefixes
use exact origins, sequence and payloads. A fifth extra event is
rejected.
- The first complete owning run found 22 introduced admission failures.
The initial historical-native allowance also applied to exact-source
Retry. It admitted contradictory preparation evidence. The fix now
requires a canonical ancestor and the shared strict startup proof. The
original direct-source assertions remain unchanged.
- The next focused run passed 82 cases, including both catalogue-warning
controls. One new result fixture lacked the database's required owner
tuple. Two existing controls reached their original 30-second limit. The
fixture now uses the real native task and completion-contract owner
tuple. A controlled nine-case run passes the result case, both unchanged
timed-out controls, three failure-code controls and all three positives.
The earlier timeout causes remain undetermined. All failed-run logs are
retained.
- Complete owning suites passed on previous head
`137a8eb3febf41052e112fc3c4ffe1abb92bcab7`: all 675 tests across
workspace recovery, task Retry, explicit continuation and legacy
continuation. The command exits successfully with normal teardown. This
includes every earlier introduced admission failure and the two
unchanged timeout controls. Its base includes the merged #18997
authority repair and #18982 rolling-copy packaging repair. The rebase
retained both test imports at their one adjacent-import conflict.
- Node 24.19.0 and pnpm 9.15.4. Module boundaries and `git diff --check`
pass. The initial PostgreSQL start lacked package library links after
install with scripts disabled. The documented hydration restored the
isolated dependency; that setup failure was not a behavioral red.
- Previous-head [full
typecheck](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785506215)
and [full
build](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785506094)
passed. Earlier attempts received synchronized runner shutdown signals
and a lost-runner annotation. The shutdown initiator remains unknown. A
CI owner used the documented hosted-runner failover for attempt three
and restored the original routing value after capture. That attempt
exposed the catalog failure above. Source workflows and check
requirements remain unchanged. The current head now has green CI,
Greptile 5/5, no unresolved review threads, and no merge conflicts.
Hosted rollout and the original task are not yet verified.


- Combined recovery proof: 24 of 24 cases passed on this exact head in
43.13 seconds, with normal teardown and unchanged assertions and limits.
They cover the real recorder, all 14 empty-output rejection controls,
ancestor and direct Retry, saved messages, the configured native
provider, and verified stop. The targeted TypeScript graph also passed;
the preceding integration step passed 95 safety/provider/message cases
and all 49 task Retry cases.
- The incorporated #19021 was independently red/green with the real
recorder: unchanged predecessor code returned HTTP 202 but admitted zero
successors. Its 18 focused cases, all 690 owning recovery cases, full
typecheck/build, 1,904 local chat cases, complete CI and Greptile 5/5
passed before integration. The unchanged CLI startup lane passed one
justified retry; no hosted acceptance is inferred from those results.
- Additional local qualification of the unchanged workspace fixture
alongside the native bootstrap fixture passed 288 of 289 cases. One
manual-admission HTTP case ended with `socket hang up` after 451 ms,
without an assertion or HTTP status. That unchanged case passed once in
isolation at the original 40-second limit, with normal teardown, and
passed in exact `0adc` CI in 17.2 seconds. The earlier disconnect cause
remains unknown; no assertion or timeout changed.
- An earlier CI attempt failed the newly merged native ingress fixture
because its shared Node command failed strict permission qualification.
The current base includes #19020's owned executable fixture repair; no
production permission check was relaxed.
- Integration with current master had one documentation conflict. An
automatic test merge also duplicated an app declaration; that syntax
failure was retained and fixed by reusing the configured real HTTP app.
Both new message-state assertions remain intact. Workflows and lockfile
retain master's contents.

## Risks

- A damaged or overlong chain remains held. The helper does not truncate
it and grant partial authority.
- The new query runs only for a deliberate Retry or its ancestor
strategy check. It adds no background polling, table, credential or
provider policy.
- A fresh intent is not stop or file-recovery proof. The existing owners
retain those checks. Prior effects remain fenced, and no old result is
replayed.
- Local tests do not prove hosted rollout. Live acceptance must use the
served fixed revision and the task's original saved work.

## Model Used

OpenAI GPT-6 through Codex for the original change. OpenAI GPT-6.1 Sol
(`gpt-6.1-sol`), xhigh reasoning, for the merge and CI repair. Both used
repository tools and PostgreSQL verification. Context-window limits were
not exposed by the harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-11-04 14:12:58 -06:00
Devin FoleyandPaperclip edb9d29942 fix(recovery): preserve Retry authority and reopen cross-task cases (#18997)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tasks keep execution identity and saved workspace files separate
from control-plane updates.
> - A scanner can have permission to reopen a case while its run cannot
restore that case's files.
> - Retry can also follow a preparation failure that has no
recovery-action record.
> - Both paths can accept valid work but leave the target unable to run.
> - This change restores files under the target executor and saves
admitted Retry authority on the canonical receipt.
> - Existing workspace, user, source and successor checks still apply.

## Linked Issues or Issue Description

**What happened?**

An authorized agent updates another closed task. The route tries to
restore the target workspace under the caller's run identity and returns
`403 Task workspace run identity is unavailable`.

A separate failure occurs when a normal board Retry follows a
pre-provider preparation failure. Admission has no recovery-action row
on which to save continuation authority. Dispatch then fails with
`continuation_user_authorization_missing`.

**Expected behavior**

Accept the authorized task update. Restore its terminal workspace when
the target's admitted executor starts. An admitted Retry must retain its
exact dispatch authority even when the predecessor has no
recovery-action record.

**Steps to reproduce**

1. Complete a task with an isolated workspace. From a different running
task, use an authorized comment or PATCH with `resume: true` on that
closed task.
2. Separately, fail native preparation before provider work starts, with
no recovery-action row. Submit board Retry, admit its successor, and
build the execution continuation.
3. Before this change, the first path returns 403 and the second rejects
its admitted authority.

**Paperclip version or commit**

Reproduced against `e968131efa7e63d321f3bcca8d47a62513c2efb4`. Rebased
onto `afbc5a885b5`.

**Deployment mode**

Server-managed native task execution. Tests use real PostgreSQL and
temporary workspace files.

Related PRs: #16927 covers terminal workspace races. #17916 retains
exact native export sources. Neither covers these route and Retry
receipt failures. A GitHub search found no duplicate. ROADMAP.md was
checked; this is a recovery bug fix.

## What Changed

- Defer workspace restoration only for authorized cross-task comments
and updates from a live run. Checkout keeps its existing workspace
identity checks.
- Restore the target's selected terminal workspace under its own
admitted run. Preserve its saved files and existing source checks. Leave
manually archived workspaces unchanged.
- Store server-created continuation authority on the canonical admitted
Retry receipt when no source recovery-action record exists. Existing
action records remain authoritative. Read the fallback only after
matching the company, task, agent, user, source, generation and
successor.
- Add route and PostgreSQL regression tests. Document both paths.

## Verification

- Red: both cross-task route tests returned the observed 403 before the
route fix.
- Red: the target executor could not reuse its archived terminal
workspace before the restoration fix.
- Red: an admitted preparation Retry with no recovery-action row failed
with `continuation_user_authorization_missing` before the receipt fix.
- Green: 419 recovery and continuation tests pass, plus 183 focused
route, environment and Retry tests. The assembled heartbeat test starts
the agent in the same terminal worktree and verifies both its unpushed
commit and uncommitted file bytes. It fails before the restoration fix.
- Negative cases reject wrong Retry runs, users, source runs,
generations, unadmitted receipts and missing authority. The original
failed run stays unchanged. Workspace tests verify saved file bytes.
- Full `pnpm build`, `pnpm -r typecheck`, `pnpm check:module-boundaries`
and `git diff --check` pass.
- The full suite found an existing assembled chat authority regression:
clearing a recovery-action record could be bypassed by the new fallback.
The fix now writes the fallback only when that source has no action
record. That assembled regression passes with the original test
unchanged.
- Full `pnpm test:run` is running again. Its first attempt also hit an
unrelated ancestor skill-root lookup for a missing Slack skill. The
rerun uses a temporary untracked link to the repository's own bundled
skills. This link is excluded from the PR. Full-suite completion and
hosted CI are pending.
- Hosted CI has multiple runner interruptions. E2e shard 1/8, server
shards 3/12, 8/12 and 12/12, and workspaces-a shard 1/2 show runner
shutdown signals. E2e shard 5/8 and server shards 1/12 and 2/12 have
GitHub annotations reporting lost runner communication; their logs
return 404. The `ci / verify` assertion failed because the general-test
matrix was cancelled. GitHub rejected job and failed-workflow reruns
while this workflow is still running. Rerun the interrupted checks after
it settles.
- Greptile reviewed exact head
`a1e3929aedb1ee3bfcd001f4ebb1510106bf4070` at 2026-11-04 17:37 UTC: 5/5,
with no new blocking findings and no unresolved review threads.

## Risks

- A deferred workspace restore can still fail at target admission. The
run keeps the existing workspace failure path; this change does not
bypass file recovery or provider stop checks.
- Retry receipts now store server-created authority. Existing action
receipts remain valid. No schema migration is required.
- Local regression tests establish the software behavior. The new
revision has not been deployed to a hosted instance.

## Model Used

OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh
reasoning, repository inspection, tool use and code execution. The
context-window size is not exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:55:16 -06:00
DottaandPaperclip 2518ac85ab fix(recovery): continue verified failed native preparation (#18908)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task Retry keeps a durable request while the system stops old work
and saves its files.
> - Native preparation can fail after it allocates a workspace, but
before it starts a provider.
> - Completed preservation can resolve the action while the original
Retry still waits.
> - The old classifier rejected every allocated failed preparation. The
source check also returned before its canonical Retry fallback.
> - This pull request accepts only complete server proof and moves the
existing exact-source fallback before that return.
> - The same Retry can then start one normal continuation. Current
access and unknown execution still hold work.
> - Saving files and admitting that continuation must use the same
parent-row lock order; the composed head includes the supporting
preservation fix.

## Linked Issues or Issue Description

**What happened?**

Task Retry accepted a failed native preparation and saved its request.
The workspace had complete preservation and stop receipts. The request
still waited for execution recovery and created no continuation.

**Expected behavior**

The same deliberate Retry must continue after the system verifies all
saved files and source leases. It must preserve current task, actor,
repository, Stop and execution checks.

**Steps to reproduce**

1. Record a native-adapter claim that fails during unresolved legacy
preparation, before provider execution.
2. Finish real workspace preservation and exact lease cleanup.
3. Submit normal HTTP task Retry and reconcile its canonical request.
4. On the base commit, no run is linked to that request. With this
change, one continuation is linked to it.

**Paperclip version or commit**

Reproduced on master `d7283e875772ca8c684dc86e471e6b48d543b7b1`.

**Deployment mode**

Source checkout with real local PostgreSQL and filesystem archives.
Object storage, sandbox transport and the native provider backend are
controlled in the final execution test. This is not a live model or
hosted object-storage claim.

**Prerequisite: #18918** at `4c1d7769e6cfa9536a67c4e3b536cef155f19966`.
This head includes that parent through normal history-preserving merges
while retaining `master` as its target. The parent owns remote workspace
save, prepare and provision lock order, legacy import reservation and
publication, their concurrency regressions, and the database lock-order
documentation. Merge the parent before this PR; no deployment is
implied.

Related: Refs #18892 for interrupted preparation. Refs #18882 for the
separate connected-chat setup path. Refs #18898 for already-admitted
cleanup waits. This change does not depend on #18882 or #18898.

## What Changed

- Accept allocated failed preparation only with the server error and
durable settlement records, no
execution/session/checkpoint/output/result/coordinator evidence, all
exact remote source leases stopped, and verified saved workspace proof.
- Recognize only the host's closed accounting scope and typed
pre-provider unsupported-harness warning. Unknown accounting, provider
metrics and arbitrary diagnostics remain holds.
- Check the canonical failed-preparation source before an empty
resolved-action list can return. Reuse the normal locked continuation
checks.
- Add HTTP, preservation-worker and native execution coverage, including
source failures produced by real heartbeat workspace preparation. Keep
one canonical request and one provider turn. Preserve the original
failure and real archive files.
- Restore storage, environment and experimental settings even if native
test setup or owned async cleanup fails.
- Cover changed Stop, source, actor, controller, provider evidence,
snapshot generation, writer and lease receipts. Keep current
unavailable-catalog denial before provider startup.
- Share one bounded host-settlement receipt reader between failed-native
preparation and existing-directory readiness. Keep each caller's
accounting, profile and additional warning restrictions.
- Include parent #18918's company/task/run-before-workspace ordering for
save, prepare and provision. Keep legacy import publication out of
reverse native-owner waits with exact NOWAIT refusal. Retain foreign
keys, ownership checks, uploaded work and transaction-aware
notifications.
- Document the failed-preparation continuation and parent lock-order
contracts.

## Verification

- Causal base comparison: the normal HTTP/preservation regression fails
on the canonical request's missing successor. Both existing
interrupted-preparation controls pass; the same cases pass with the
classifier/fallback repair.
- Actual heartbeat-produced catalog failures reproduce RED before the
host-accounting fix, both with and without the real typed
unsupported-harness warning. No source failure records are patched in
those cases.
- The native execution case keeps `paperclip_runner`, native runtime
selection and the requested Codex model. A controlled native session
reads real restored `saved.txt`, writes `continued.txt`, emits an
accepted result and finishes through the real finalizer once. Only the
external native backend is controlled.
- Current unavailable-catalog admission and suspended authenticated
membership remain denials before provider work. Extra
provider/session/accounting material, Stop, newer source,
writer/generation and lease-receipt mismatches remain held.
- Both settlement consumers pass 53 focused real
PostgreSQL/filesystem/native-boundary cases on Node24.22.2.
Existing-directory warning presentation/provenance and
durable-measurement restrictions are preserved.
- Parent #18918 proves real PostgreSQL lock cycles through actual save,
prepare, provision, reservation and import publication owners. Its
native status peer test proves a busy refusal without a deadlock.
Immutable task binding, writer/epoch fences, same-journal retry and
unexpected-error rollback controls retain uploaded objects and original
files. The original spontaneous peer identity is not claimed.
- Historical head `6204f565` passed all 54 CI checks, but its local
affected run passed only 572/573 with a real PostgreSQL preservation
deadlock and its Paperclip review was 3/5. Those results remain
historical evidence; this composed head includes the explicit parent
repair.
- Production-composed head `93a02d4cf74f4d8d2cc6c1b62884bc58280500bf`,
Node24.22.2 / pnpm9.15.4: full `pnpm build` and recursive `pnpm -r
typecheck` passed. The six affected recovery, continuation, Retry,
migration and concurrency files passed 724 tests. Opt-in remote
workspace integration passed 65/65 and ownership lifecycle passed 16/16,
with no skips.
- Final head `65dcd6c6a601448b5a436b7a7f2c84b8cfde761b` adds only the
parent's unsupported-host test guard. Its 18 concurrency cases and
generated server test-layer typecheck passed on the exact head.
Production bytes match the qualified head above.
- Exact-head CI passed on `65dcd6c6`: 54 success, 3 skipped, 1 neutral;
no failed or pending jobs. Greptile reviewed this exact commit at 5/5
with no actionable issue and zero open threads.
- The current-head Paperclip assessment is registered and queued with no
run at the latest observation. It has not delivered a current-head
score. The earlier 3/5 assessment and its resolved findings remain
history. No extra review request or hosted task replay was used to force
delivery.

## Risks

- The new allocated-source proof is conservative. Extra or missing
execution material keeps work held and needs inspection.
- This changes admission after completed preservation. It does not
certify unknown provider effects or restore repository access.
- Administrative import can now return a named busy hold while an
execution row is owned. It retains the archive and journal for a later
administrative retry.
- There is no schema migration or new API. Stop, task ownership, current
actor access, budget and approval checks retain their normal owners.

## Model Used

OpenAI Codex, GPT-6 family. The exact runtime model ID and context
window are not exposed in this agent session. The agent used reasoning,
tool calls, code edits, real local PostgreSQL and filesystem tests. A
separate Codex agent reviewed the admission and execution boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 07:17:49 -06:00
DottaandPaperclip 0ab3ce7a03 fix(workspaces): preserve legacy recovery across storage migration (#18841)
## Thinking Path

> - Paperclip manages durable tasks and agent work through the control
plane.
> - An ephemeral web server needs existing workspace files in shared
storage.
> - Failed historical runs can still own recoverable sandbox files or
unfinished coordinator state.
> - Moving local files must preserve those obligations without replaying
the old agent turn.
> - Migration journals the exact source and fences old copyback; the
recovery owner saves verified sandbox files separately.
> - New user input can continue only after matching preservation
receipts and independent execution-stop checks.
> - Accepted results retain their finalization owner and must have its
durable workspace receipt before storage moves.

## Linked Issues or Issue Description

Builds on merged #18826 and #18835.

**What happened?**

Existing workspaces with failed native copyback could not migrate to
remote storage. Simply changing their storage authority would let later
recovery target a retired local directory, lose child-task attachments,
or leave a recovered task unable to accept a new turn.

**Expected behavior**

Preserve local files, exact sandbox provenance, task bindings,
checkpoints, and unresolved history. Recover surviving sandbox files
into immutable task-scoped attachments after migration, and let a new
authorized user turn use those files without replaying the old
execution.

**Steps to reproduce**

Create a terminal native run with a prepared copyback descriptor and a
retained sandbox. Migrate its local workspace, remove the old host
filesystem, recover the sandbox, and submit a new user turn. Also
exercise a child task reusing its ancestor's workspace and an expired
observed coordinator.

## What Changed

- Journal exact legacy run, task, descriptor, and unfinished coordinator
bindings during offline migration. Descriptor-free runs retain the
complete declared lease/allocation/directory identities; publication
locks and revalidates them.
- Publish a run-scoped fence that prevents automatic restart or copyback
into the retired local authority.
- Release only the exact journaled expired coordinator lease under the
offline deployment fence and locked revalidation, recording the prior
owner in recovery history. Preserve phase, checkpoint, result, and run
status.
- Save recovered sandbox files through the existing immutable snapshot
owner, retaining allocation and provider-stop checks.
- Attach recovered files using the current task's identity when a child
reuses an ancestor workspace.
- Accept matching migration and snapshot receipts for workspace-repair
admission and explicit new-user continuation. Changed coordinator
bindings and invalid receipts remain blocked.
- Require the finalizer's successful workspace_finalize operation before
moving a workspace whose accepted result has finalized copyback; the
existing owner can then finish assessment/status effects.
- Document unresolved workspace-policy, task-ownership, and
missing-provenance cases as separate owner actions that block
detachment.

## Verification

- 569 tests across migration, retained recovery, and explicit
continuation passed on ffcde13f57.
- The final saved-result follow-up passes all 75 migration tests. Four
receipt cases cover prepared copyback, a successful finalization
receipt, a missing receipt, and a failed receipt. The successful case
invokes the actual finalizer after migration and completes the existing
saved operation.
- Five real-file journeys cover a retained host, removed host,
child-task reuse, an expired observed coordinator, and a descriptor-free
run with two separate sandbox allocations. They recover sandbox files,
clear the repair blocker, admit an explicit /new user comment, reserve a
new remote workspace, and verify recovered attachment contents.
- Negative cases reject changed coordinator phase/owner, invalid
migration fences, changed source sets and allocations, mismatched
task/allocation/action receipts, malformed archives, and unfinished
snapshot operations. Descriptor-free continuation requires every
journaled source; one missing archive keeps the hold.
- Twelve PostgreSQL/S3 and maintenance-service tests pass on the exact
final rebuilt runtime, including a plain-Node bundled process,
authenticated original-key export, overlapping task-folder restore, and
complete predecessor filesystem removal. The final server and runtime
bundle builds also pass.
- Head 9ab6a38fa4 passed all 573
migration/recovery/continuation tests, server typecheck, server build,
runtime bundle, and 12 PostgreSQL/S3/bundled-maintenance tests.
- Verification caught an expired-owner continuation rejection (160
recovery cases passed, one failed); the admission fix and subsequent
569-test run passed. Earlier fixture failures and the parent PR's
unrelated full-suite failure remain documented in their verification
history.
- Greptile and Paperclip Review both scored final head
fa58d4a5ce 5/5, with no remaining
findings.
- CI exposed 15 blocker-read projection regressions: descriptor-free
checks performed an extra full run read for ordinary unmigrated tasks.
The follow-up gates the lookup on the migration marker, includes that
marker in the bounded blocker projection, and selects only recovery
fields. All 31 blocker-projection and real-file journey cases plus
server typecheck pass. Final-head CI passed all 52 successful checks,
including repository typecheck, build, server tests, and browser suites.
- Preserved CI failure: the unchanged Runner package had one Cursor
Cloud Stop/checkpoint timing failure (4,903 cases passed). All four
focused local cases passed. Retrying only the failed job and its
verification gate passed without source changes.
- This PR does not claim tenant enrollment or a hosted rolling-deploy
result.

## Risks

Cloud must stop every predecessor deployment before enabling the offline
migration contract. Migration preserves uncertain work; it does not
certify provider termination, invent a completed result, or authorize
sandbox deletion. Missing or ambiguous sources remain held. Preserved
attachments are separate files rather than an automatic merge into newer
task edits. Parallel shared-write policy, stale task ownership, missing
task provenance, and accepted results without a successful finalization
receipt remain enrollment blockers. Original folders and the original
volume remain retained.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. This session does not expose a more specific serving model ID
or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with available version and
capability details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the bug template
- [x] I have not referenced internal or instance-local issues or links
- [x] My branch name describes the change and contains no internal task
ID
- [x] I have run the focused tests locally and they pass
- [x] I have added tests for the changed behavior
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address reviewer findings before merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 16:17:13 -06:00
DottaandPaperclip 367f29478d fix(recovery): preserve the bound remote writer before continuation (#18803)
## Thinking Path

> - Paperclip manages agent work and preserves task files.
> - A remote workspace records the exact run that owns its unsaved
files.
> - A later setup failure can become the latest task run without owning
those files.
> - Retry only inspected that latest run, so the original writer had no
file-recovery owner.
> - A released reusable source also failed the legacy retention rule.
> - This change registers the bound writer, saves its files, and then
admits the saved user request once.
> - Existing stop, ownership, approval and execution-evidence checks
remain effective.

## Linked Issues or Issue Description

**What happened?**

A native run prepared a remote workspace and failed before finalization.
Its sandbox stopped with a dedicated stop receipt. The workspace stayed
dirty and retained that run as its writer. A later setup failure became
the latest task run. Retry inspected the newer run and left the original
file obligation unregistered. New attempts failed at the writer fence.

Registration alone was insufficient. After file recovery, continuation
still required the Retry source to be one of the older recovery sources.
A newer pre-provider failure did not satisfy that check. Workspace
selection could also choose a fresh directory instead of the recovered
files.

**Expected behavior**

Find the writer through the task's current workspace binding. Register
only its exact prepared source. Preserve the released lease and its stop
receipts. Save the files before continuation. Independently prove that a
newer setup failure never dispatched work. Select the verified saved
workspace and admit the Retry once.

**Steps to reproduce**

1. Fail a native run with a prepared remote workspace and a released
reusable sandbox.
2. Keep matching dedicated stop receipts and the workspace's dirty
writer authority.
3. Record a newer pre-provider setup failure for the same task.
4. Request Retry. Before the fix, no file-maintenance record owns the
actual writer.

**Paperclip version or commit**

Reproduced before production edits on master `f7ad1dc7ca`. The branch
was rebased on master `adb49a44346849e751548b2fb640693dbfad4b6f`.

**Deployment mode**

Authenticated hosted execution with Paperclip Runner and remote
workspace storage.

Companion: #18798 (now merged upstream) fixes a pre-existing inbox test
synchronization race. Its test-only commit
`110aa88489cad926a451c972ad731ede469347c1` is cherry-picked here with
attribution so this PR can run the complete CI matrix.

Related: #17800 registers missing preservation for the selected source
run. #18678 holds unfinished remote exports. #18718 repairs a prior
warm-controller projection failure. #17916 addresses a separate legacy
warm-copyback policy case.

## What Changed

- Discover the failed writer through the company-scoped task/workspace
binding during Retry and authenticated message admission.
- Reuse the exact prepared-source validator for released remote
workspaces. Registration changes neither the lease policy nor its stop
receipts. Physical recovery retains its full allocation checks.
- Check a newer failed preparation separately before allowing an older,
repaired source to release continuation. Provider events, leases,
results and execution ownership remain disqualifying evidence.
- Select the bound saved workspace through the continuation's verified
repair receipt.
- Add PostgreSQL safety cases and an HTTP Retry workflow with real file
capture and saved-copy reads. Document the execution contract.
- Reuse #18798's completion barrier for the existing inbox cancellation
test; preserve its eight-slot limit, rejection and coalescing
assertions.

## Verification

- RED before production edits: the exact stopped writer has no
`workspaceRestoreRecovery` after Retry. Six unsafe-source controls pass
on the baseline.
- GREEN: the expanded Retry/message/preview selection passes 17 cases.
All 627 tests in the five affected recovery suites pass on recovery
commit `b572304f772d236b8a8bdf095e13e651d808f41c`, including Stop,
ownership, saved messages and duplicate Retry.
- The HTTP workflow registers maintenance, keeps execution held before
save, reads back the saved file bytes, and admits exactly one
continuation with `resume_preserved_workspace`. A registration-only
repair first failed after save because it did not reconcile the newer
Retry source. The incident-shaped pair passes with an independent active
stranded-task action present.
- Greptile identified the path where the newer setup failure already
owns a reconciliation hold. Its new regression failed before the
correction (saved files, zero queued continuations); the
contradictory-dispatch control passed. All four HTTP variants pass after
the correction, covering both hold shapes and both dispatch-evidence
controls.
- Repository typecheck and build passed. Server typecheck and build also
passed after the recovery changes. Module-boundary and whitespace checks
pass.
- The full local Vitest command failed in the GitHub fast-response
acknowledgement test (`delivery_unknown`: one response write instead of
zero). That isolated case passes on this branch and unchanged master;
the original full-run cause remains unestablished. A sibling run passed
199 test cases but timed out in workspace-suite database cleanup.
Unchanged master also passed all 126 workspace cases and timed out in
the same cleanup hook; all five acknowledgement outcomes passed in that
baseline comparison. These local failures remain recorded rather than
being reported as suite passes.
- The first hosted matrix failed the pre-existing inbox cancellation
assertion (expected nine computes, saw eight). The same assertion failed
in [master
CI](https://github.com/paperclipai/paperclip/actions/runs/41664339150/job/126015526840)
and reproduced locally before applying #18798's existing repair. All six
inbox HTTP/database cases pass after that repair. Final head
`7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9` adds only that attributed
test change to the verified recovery code.
- [Final hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/41677310377)
passes on head `7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9`, including `ci
/ verify`, `ci / e2e`, typecheck, build, every server/chat/workspace
group, Runner checks and all eight browser shards. There are 54
successful check results, two informational neutral results and three
skipped results, with no pending or failing results.
- Greptile and the Paperclip reviewer both report 5/5 for that final
head; no review threads remain open. The Paperclip review check is
informational (`neutral`) and records its local-testing limitations.
- The file workflow uses real PostgreSQL, subprocess file capture,
encrypted saved objects and production recovery code. Provider
resume/stop calls are simulated at the environment driver boundary. This
does not prove a live Daytona or Codex recovery. No deployment or
original-task retry is claimed.

## Risks

- File-maintenance discovery must not authorize another executor.
Existing writer, coordinator and physical allocation checks still own
that decision.
- A false bootstrap summary cannot prove that a newer run never
executed. The new path also requires its immutable native-adapter claim,
unresolved preparation state, absent
coordinator/process/session/lease/result evidence, and no dispatch
events.
- Additional database reads run only during explicit task admission or
Retry strategy selection. They are company/task scoped and add no
polling loop. The saved-writer query returns one matching snapshot and
omits prompt bodies.
- No schema, credential, provider profile or deployment setting changes.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection, code
execution and test tools. The runtime does not expose the exact serving
model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 11:25:24 -06:00
DottaandPaperclip 2b374caba0 fix(recovery): preserve unfinished remote exports before Retry (#18678)
## Thinking Path

> - Paperclip manages agent work and preserves each task's saved files.
> - A completed native result can still need workspace export.
> - Remote workspaces now use a different saved descriptor from legacy
copyback.
> - The execution hold checked only the legacy descriptor.
> - Task Retry could admit another run before the original export
finished.
> - A later failure could then supersede the original recovery record.
> - This change keeps unfinished remote exports fenced and permits
explicit historical file recovery.
> - Recovery keeps human review and newer execution authority intact.

## Linked Issues or Issue Description

**What happened?**

A native run completed successfully, but its remote workspace export
failed. The recovery action used
`native_workspace_sync_out_retry_exhausted`. Task Retry created a second
task run while the original files still needed preservation. The newer
failure then superseded the original action. A board file-recovery
request could queue that cancelled record, but the worker excluded it
from discovery.

**Expected behavior**

Keep Retry behind the unfinished export. Save the exact source and
finalize the existing accepted result once. Preserve pending human
review. An explicit request for a superseded source must save its files
without applying an old result over newer work.

**Steps to reproduce**

1. Save an accepted successful native result with a prepared
`remoteWorkspace` descriptor.
2. Fail its final export and retain its exact sandbox lease.
3. Select Task Retry before preservation completes.
4. Observe another task run before the original workspace is saved.
5. Supersede the original action and request file recovery for its exact
source. Observe queued work that the worker does not select.

**Paperclip version**

Observed at `1f8f2e195f4897cd1c237bdc068de95780ab0c3c`. The regression
is present on current master before this patch.

**Deployment mode**

Authenticated hosted instance with native Paperclip Runner and remote
sandbox workspaces.

Related: #18403 introduced the remote workspace path. #18644 reduces
export scratch disk use. #17916 covers a separate legacy warm-copyback
policy case. This change repairs remote export admission and explicit
historical preservation.

## What Changed

- Include prepared remote descriptors in the existing native export
hold.
- Retain remote source identity in the blocker and continuation read
projections.
- Let the existing worker select a cancelled export-repair record only
after an explicit board file-recovery request. Keep the current action
and workspace binding unchanged.
- Add real PostgreSQL, HTTP, filesystem and Git regressions. Update
execution semantics and recovery maps.

## Verification

- Red before the admission fix: HTTP Retry creates two task runs instead
of one.
- Red before the historical-source fix: the explicit recovery stays
scheduled and saves no snapshot.
- All 588 focused tests pass across the workspace recovery, recovery
source, explicit continuation and native export retry suites. The three
new integration cases verify duplicate request identity, original result
finalization, restored bytes and Git HEAD, pending human review,
current-writer exclusion, and preservation without older-result
commitment over newer work.
- `pnpm -r typecheck`, `pnpm build`, map validation and `git diff
--check` pass.
- Full hosted CI passes on `c598718379fa939ec539e8e91a15a94d90321b59`,
including all server, workspace, serialized, Runner and browser lanes
([run](https://github.com/paperclipai/paperclip/actions/runs/41529739604)).
Greptile reviewed that commit at 5/5 with no findings; there are no
unresolved review threads. Paperclip Review also published a complete
5/5 assessment for that exact commit with zero findings. Its check
conclusion is advisory/neutral, titled `5/5`. The first attempt lacked a
structured assessment; one normal completion request recovered the saved
analysis and published the validated assessment.
- The local `pnpm test:run` passed all six chat partitions, then was
stopped during the general server group after equivalent full CI
completed. Its partial log is retained; it is not reported as a
completed local full-suite pass.
- The test provider boundary uses real temporary files and the
production archive path. It simulates provider resume/stop calls. The
deployed fix and original hosted task recovery remain unverified until
merge and rollout.
- The older local Node 24.11.0 runtime failed to load the installed tsx
worker. Node 26.4.0 passed the isolated worker check. Required
repository checks use an isolated Node 24.22.2 runtime. The earlier
failure logs remain recorded.

## Risks

- Recognizing another unfinished export can keep previously admitted
work waiting. This is intentional until persistence is verified.
- Explicit historical recovery can resume the exact stopped sandbox to
read files. It retains source, allocation, ownership, writer and
bounded-attempt checks. It starts no provider turn.
- A newer run still prevents commitment of an older accepted result.
This change does not infer that a cancelled run had no external effects.
- No schema, credential, deployment or fleet configuration changes.

## Model Used

OpenAI Codex, GPT-6 family. The runtime does not expose the exact model
ID or context-window size. Reasoning, repository inspection, tool use
and code execution assisted this change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 16:03:09 -06:00
DottaandPaperclip baf651c8d0 perf(heartbeat): drive retry promotion from durable deadlines (#18387)
Reviewed and verified at f1c48c924d.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 09:07:52 -05:00
DottaandPaperclip 4e715eb7ea fix(server): schedule historical unsafe export recovery (#17687)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs can retain accepted results after an unsafe workspace
export fails.
> - Historical recovery can retire an old notice or resume the accepted
result without another provider turn.
> - The periodic reaper repeatedly scans failures that cannot currently
recover.
> - Existing rows and committed changes can identify the actual recovery
obligation.
> - This pull request wakes recovery from those changes and retries only
owed work.
> - Dormant history becomes quiet while ownership and explicit operator
decisions remain protected.

## Linked Issues or Issue Description

**What happened?**

The native finalization reaper repeatedly visits historical unsafe
workspace-export failures. Cancelled tasks, newer runs, changed
contracts, and explicit operator decisions can make those rows
permanently ineligible. They still produce periodic database reads.

**What did you expect?**

Retire stale unsafe-export notices and recover eligible accepted
results. Stop reading dormant history until a relevant committed change
occurs. Keep a retry while a real obligation is blocked by ownership or
a transient failure.

**Steps to reproduce**

Retain an unsafe-export terminal failure whose task is cancelled or
superseded. Leave the instance idle after startup. The periodic reaper
continues reading that history.

Prerequisite #17680 is merged into master. Related: #14402 established
safe export finalization; #14329 was an earlier historical recovery
proposal. #17679 owns separate policy-review cleanup. This PR changes
only historical unsafe-export scheduling and its producers.

## What Changed

- Register a scoped worker through the existing coordinator and
scheduler. Remove this duty from periodic reaping. Preserve the
manual/startup recovery entry point.
- Distinguish stale notices, eligible accepted results, dormant history,
and concrete ownership contention. Reuse the existing mutation owner and
lock order. Preserve explicit settlement vetoes and exact
result/contract identity.
- Signal relevant task, run, action, and deletion writes before they
commit. Existing transaction boundaries handle nested commits and
uncertain outcomes. Notifications contain only scope IDs.
- Retain a retry only for owed work. Reconstruct work at startup,
preserve drain/resume, and stop through the shared lifecycle.
- Add concurrent, idempotent partial index migration 0410 for unsafe
terminal failures. Keep the five newest generated schema snapshots.
- Document producer coverage, query plans, supported writer boundaries,
and test failures in the unsafe-export audit.

## Verification

- Source `2a13dd08a6cc52f0adab09e3df9ce31e97b746b9` passes all 2188
cases across 24 complete files, without skips. The main selection passes
2129 cases in 192.58 seconds; safe-replacement passes 59 in 26.04
seconds. Coverage includes all 968 native-session cases, real
export/action producers, explicit continuation, workspace
reconstruction, interaction routing, Stop, queue and Slack consumers.
The audit gives per-file counts.
- Twenty separate fresh PostgreSQL 17 databases migrate through 0410
before setup. Four embedded-only fixture imports temporarily use their
own external database on this IPC-exhausted host, then are restored byte
for byte. Assertions and timing limits stay unchanged. This is qualified
external-database acceptance, not an unmodified embedded CI run.
- Database build, server typecheck/build, module boundaries and
exact-head migration order pass. Normal integration preserves the
complete original runtime/schema patch and unrelated upstream blobs.
Original concurrent index SQL, all prior executable history, journal
entries and retained snapshot bytes are unchanged; five snapshots 0406
through 0410 remain. Added comment removal preserves the full
non-comment TypeScript token streams.
- A populated 0409-to0410 upgrade preserves all 408 prior migration
receipts and 38 companies. It appends exactly one receipt; replay adds
none. The index is valid and ready on fresh and upgraded databases.
- The owning audit retains earlier fixture failures, skipped checks and
notice-retirement regression history. Older combined stack checks retain
their original scope in their owning audits. Actual parent
`73d24aa5acfcf17b3f676b4da7cd05eb70b60b99` is reconciled through a
normal merge. Its tree equals the reviewed ancestor and the complete
accepted child tree is unchanged. Fresh published-head CI and Greptile
5/5 remain required. No deployed sleep, fleet-cost or live provider
qualification claim is made.

- Inherited prefix correction: current integration
`8f43c9f81fac23b50997e7f0e1d0d9cf383a8109` changes only two values in a
previously unselected corpus fixture. All production code, the complete
package tree and all 24 accepted test files remain byte-identical to the
2188-case source above. The additional complete 72-case corpus passes in
14.00 seconds and again on the same database in 13.08 seconds. Retained
PSC/OUT and sentinel rows remain unchanged. The repeat is not
double-counted. The parent audit retains the original collision and
rejected cleanup attempt.

- Latest parent context: integration
`4d51231abe8b68c21ce896d0d2ac16865978c8ab` carries only the parent's
one-argument GitHub observation correction and an independent master
cleanup assertion. All production code, packages, 24 accepted suite
files and the additional corpus are unchanged. The two focused
native-presentation cases pass in 15.14 seconds and the complete
cleanup-removal suite passes all six cases in 8.31 seconds on two fresh
0410 PostgreSQL fixtures. This does not claim the full parent workflow
was rerun here.

Current head `b28399babac72ba9db1c4a4ef882c5e87aded38c` has passing CI
and Greptile 5/5 with no actionable findings. The first CI attempt lost
runners: fourteen jobs explicitly received simultaneous shutdown
signals, two reported lost runner communication, and two aggregate gates
failed downstream. One failed-job retry on unchanged source passes. The
underlying host shutdown cause remains unconfirmed; no code fix or extra
retry is claimed. [CI run and
attempts](https://github.com/paperclipai/paperclip/actions/runs/40819408528).
No Ponytail verdict is present.

## Risks

- A missing future producer can delay dormant-history discovery until
restart. The audit names current producers. Independent SQL writers and
other server processes remain outside the process-local signal contract.
- Physical ownership and uncertain operations retain retries. Time alone
does not authorize taking workspace ownership. No provider turn or
unsafe archive retry is added.
- The recovery owner remains authoritative if state changes after
discovery. Terminal UI status does not authorize resource destruction or
accepted-result replay.
- The partial index adds matching-row write/storage cost. It builds
concurrently through the existing restart-safe migration runner.
- This proves the changed worker becomes quiet. It does not prove host
suspension, external wake coverage, or fleet cost savings. Other sleep
migrations remain. This slice still requires fresh current-head CI and
reviews.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning and code-execution tools.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 00:13:22 -05:00
Devin FoleyandPaperclip c9213d1be4 fix: send saved queued messages after Stop (#17959)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People can save follow-up messages while an agent runs.
> - Stop preserves those messages and stops the active execution.
> - The queue removed its delivery controls when that execution ended.
> - Native admission also refused an explicit resend of older queued
input.
> - This change adds Send for a stopped queue and records fresh board
intent.
> - The saved message can continue through normal cleanup and admission
checks.

## Linked Issues or Issue Description

**What happened?**

A person queued a message with a screenshot while an agent was starting.
Steering was not yet available. They stopped the run. The message
remained visible, but no control could send it. Native queue delivery
also rejected this transition.

**Expected behavior**

A stopped queue offers Send. The click preserves message IDs, content,
attachments and original authors. It starts at most one eligible
continuation. Stop remains effective until the person gives new
direction.

**Steps to reproduce**

1. Start a native task and queue a second message.
2. Stop the active run before it consumes that message.
3. Open the task after cleanup. Observe the saved message with no send
control.

Refs #16965 for the queue-control change and #15015 for stopped
conversation recovery. No open repair for this exact transition was
found.

## What Changed

- Show Send for a saved deferred queue with no active run.
- Accept the existing board queue action for a stopped native target.
Keep live native interruption restricted.
- Use the server-recorded click time to authorize older native messages.
Reject stale, malformed and wrong-actor intents.
- Reuse normal queue dispatch, revision checks, cleanup proofs and
execution admission, including a failed native predecessor after a
verified Send.
- Ignore a fully committed, successful recovered export when admitting a
fresh message, while retaining the wait for results still committing.
- Honor an applied `/new` conversation boundary while retaining native
execution holds until the existing stop and cleanup checks verify them.
Workspace holds also survive reset. Check the newest failed or cancelled
native execution even when older recovery actions exist.
- Keep native automatic delivery separate from the explicit Send action.
Update execution and access documentation.

## Verification

- Meaningful regressions failed before the fixes: stopped queues lacked
Send, native admission refused fresh intent, cleanup and dispatch
receipts were missing, deferred rows disappeared, live plan Interrupt
rejected its own receipt, and Agent Chat `/new` inherited an old
non-workspace execution hold.
- Six assembled-wakeup cases fail before the reset cleanup fix. All 11
reset/history cases pass after it: live processes, missing identity,
quarantined cleanup, remote leases and live controllers remain held;
verified cleanup proceeds and retires the hold. Two more cases fail
before the fallback fix and pass after it: Send verifies a stopped or
live native predecessor despite a later reset receipt and no recovery
record.
- **525 server tests pass** across seven queue admission, route,
dispatch, envelope, provenance, blocker and conversation files. **343
queue/thread UI tests** also pass. Server/UI TypeScript, token and map
guards pass. The final scoped server typecheck also builds native Runner
successfully.
- The latest regressions prove that completed recovered exports permit
fresh input, and clearing conversation history still requires old native
stop and cleanup verification; an unsafe workspace still blocks after
`/new`; failed native Send reaches actual continuation dispatch with the
original queued comment. A result still being committed remains
deferred. History plus a newer Stop cannot hide a missing
process/controller/lease receipt.
- The real Chromium Stop/Send journey passes on current head
`75fa5e7222c59f43ff7ac326e81093bc9da28330`: queue a follow-up, Stop,
reload, explicitly Send, then verify exactly one useful response and no
live run. Earlier browser journeys cover legacy Interrupt, interruption
during a write and Stop followed by a new direction. These use an
isolated app/database and controlled ACP fixture, not an external AI
account or a staging deployment.
- Negative cases retain actor, company, issue, assignee, queue and
cancellation-receipt bindings; cleanup/process/workspace/controller
holds; approvals; pause and budget gates. Typed responses keep their
original resolver and immutable result.
- Full local recursive typecheck stopped in unchanged Rust Runner
compilation with ENOSPC, so the chained full tests/build did not run.
All hosted CI gates pass on final commit
`75fa5e7222c59f43ff7ac326e81093bc9da28330`, including typecheck, tests,
build and E2E. Apex review on that commit is 5/5, with no unresolved
findings. Earlier unrelated CI failures involved PostgreSQL
fixture-cleanup deadlock and an unchanged decision-expiry deadline
assertion; the latter's 20 cases pass locally without source changes. No
assertions or limits were weakened. Local resource cleanup was limited
to this task's verified completed test processes and rebuildable npm
caches.

## Risks

- Send can authorize a new turn for a saved message after Stop. A Send
click must be newer than the stopped execution; a live plan Interrupt is
bound to the cancellation receipt produced by that same click. It does
not imply that cleanup finished or that an external action is safe to
replay.
- The existing endpoint and intent format are reused. No schema or
credential change is required. Live native steering and legacy
interruption keep their existing paths.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection, tool
use and code execution. The session does not expose the exact serving
model ID or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:34:14 -07:00
DottaandPaperclip f6ba9afab0 perf(server): narrow explicit continuation run reads (#17896)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Recovery must prove that old execution stopped before it admits new
user input.
> - Explicit continuation reads the source run before and after it takes
the run lock.
> - Both reads transfer the saved prompt and the full Runner profile.
> - The preflight needs only identity, comment, adapter, and workspace
receipt fields.
> - This pull request narrows those reads and keeps the complete locked
profile for session proofs.
> - The benefit is less database transfer with the same admission and
recovery rules.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Explicit native continuation admission after a failed or stopped run.

**Current behavior**

The preflight and locked reread select complete heartbeat run rows.
Large saved contexts and execution profiles cross the database
connection twice.

**Proposed behavior**

Read only required scalar fields and typed context keys. Read only
historical adapter identity and workspace sync receipts before the lock.
Keep the complete Runner profile in the locked reread. Keep lock order,
company scope, text lookup of historical run IDs, stop proofs, queued
input, and replay rules.

**Reason and benefit**

A synthetic PostgreSQL 17 fixture removes 74.9% of the two queries'
DataRow bytes. This result is local query evidence. It is not a claim
about deployed latency or monthly transfer.

**Breaking changes**

None intended. No schema, API, migration, workflow, or dispatch contract
changes.

**Additional context**

Related: #17491 narrows the separate execution blocker read path. This
PR covers explicit continuation admission. The PR has no dependency on
the other byte reduction PRs.

## What Changed

- Select required continuation fields and typed context keys instead of
complete run rows.
- Read a small historical adapter/workspace profile during preflight.
Preserve the full locked profile for exact suspended-session and
quarantined-cleanup proofs.
- Narrow four helper input types to their actual field requirements.
Their runtime behavior is unchanged.
- Add real PostgreSQL admission comparison coverage and a reproducible
baseline/treatment benchmark.
- Document the query boundary and benchmark command.

## Verification

- Baseline: `663fe667d66d3863d0919a8724f1d99500feef0b`.
- Disposable PostgreSQL 17.11. One source run has a 192 KiB
incompressible context and two 96 KiB profile payloads.
- Command: `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-continuation-run-reads.ts
--baseline-ref=663fe667d66d3863d0919a8724f1d99500feef0b`. Set
`PAPERCLIP_CONTINUATION_BENCHMARK_DATABASE_URL` to a disposable database
ending in `_test` to use an external server.
- The harness captures SQL generated by both real service versions.
Baseline imports use current helpers; only their TypeScript input types
changed in this PR. Their runtime behavior stays fixed.
- Preflight DataRow bytes: 394,067 to 466 (99.88% less). Locked reread:
394,067 to 197,168 (49.97% less). Combined: 788,134 to 197,634 (74.92%
less).
- Byte counts cover exact PostgreSQL text-format DataRow frames. They
exclude RowDescription, command metadata, TLS, TCP, and other queries in
the service.
- Seven measured samples per query follow two warmups. The order
alternates. Final-head median preflight time: 6.53 to 2.28 ms (ranges
5.34–7.41 and 1.84–4.46 ms). Median locked read: 8.06 to 5.26 ms (ranges
5.94–13.84 and 4.61–6.81 ms). These are warm local SQL measurements
under shared machine load. The deterministic byte counts are the
acceptance criterion; these noisy timings do not establish a deployed
latency gain.
- The harness compares the admitted continuation, blocked reasons, every
retained scalar column, kept JSON, and the full locked profile. It fails
unless the fixture removes more than half of the two reads' DataRow
bytes. It records SQL, parameters, revision identity, samples, and
EXPLAIN plans.
- All 291 continuation tests pass together on the final head with no
skips in a clean isolated PostgreSQL 17 database. This includes all
seven new projection comparisons and six malformed/missing JSON cases.
- Edited server TypeScript check passes: `node
node_modules/typescript/bin/tsc --noEmit -p server/tsconfig.json`.
- `pnpm -r typecheck` and `pnpm build` pass on the final head. The
normal server build verifies the complete pinned Copilot closure. Local
full `pnpm test:run` is not claimed: host embedded PostgreSQL startup
limits prevent that gate. The isolated PostgreSQL regressions pass; the
full exact-head test matrix runs in CI. The targeted exact-head test
retry passes all 4,086 tests in 98 files. All CI verification checks and
the dependent aggregate are complete and successful on head
`47c1b5f7d17a58310f7a3ebd494fdcaa814777c2`. Greptile completed cleanly
on head `47c1b5f7d17a58310f7a3ebd494fdcaa814777c2`: 5/5, no findings,
and no unresolved review threads.
- Local setup evidence: embedded initdb could not load its ICU data
library. Tests therefore use a dedicated PostgreSQL 17 container. A run
against reused test fixtures produced stale sweep failures; a new clean
database passed all 285 cases. The first comparison mock replaced an
unrelated aliased query; it was narrowed to the two admission reads
before the clean pass.

- The first [server test shard
6/12](https://github.com/paperclipai/paperclip/actions/runs/40529210562/job/122277218565)
failed one existing workspace-recovery case: a third export stayed
`scheduled` at its 20-second poll deadline; 4,085 other tests passed.
The exact case passes on the pinned baseline and final head with the
same PostgreSQL 17 fixture. Temporary spies recorded zero calls to both
changed admission functions in either replay. Other touched helpers have
type-only changes. This supports one timing-flake retry; the exact
scheduler delay remains unproven. The single [targeted shard
retry](https://github.com/paperclipai/paperclip/actions/runs/40529210562/job/122285744135)
passes all 4,086 tests in 98 files. The exact case completes in 17.25
seconds. The dependent aggregate is complete and successful. No timeout
or unrelated runtime code was changed.

- The separate Paperclip Review check ended `NEUTRAL` without a final
report. The completed source review is the exact-head Greptile 5/5
result above.

## Risks

- A future admission check may need an additional saved context key.
Keep the projection and helper contracts in sync with that consumer.
- Exact suspended-session and quarantined-cleanup proofs still require
the complete Runner profile. This PR retains it under the lock rather
than weakening those checks.
- JSON values retain their types and missing keys stay absent.
Malformed-context comparisons preserve the existing controller hold.
- These measurements use synthetic local data. Deployed transfer savings
require a fresh capture after rollout. Nothing was deployed.

## Model Used

OpenAI GPT-6 through Codex. The exact runtime model ID and context
window are not exposed by this session. The work used repository tools,
code execution, and real PostgreSQL validation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 12:05:51 -05:00
DottaandPaperclip ac22b1b1f9 fix: preserve recovered input and task directory continuity (#17878)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/60/changes/paperclip/60
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip lets people manage AI agents and their work.
> - Recovery must preserve files and the authority of each human
request.
> - Some failed native runs preserve a snapshot without completing
native copyback.
> - The saved input could remain blocked after recovery settled.
> - A later turn could also treat the new task directory as a Git base
because it inherited the agent default.
> - This change continues the same authorized input once and keeps its
saved directory usable on later turns and Retry after a pre-provider
preparation failure.

## Linked Issues or Issue Description

Refs #14905. Related: #17724 covers cleanup handoff after native
copyback. This change covers snapshot-only preservation and later turns
in its directory. #17814 covers a separate same-run checkpoint restore
defect. The open legacy-adapter change in #13291 does not cover this
native saved-input path.

## What Changed

- Forward the trusted deferred inbound request to native continuation
admission. Verify the exact unconsumed receipt and its source, actor,
company, agent, and task.
- Let historical holds for the same failed run use its settled recovery
strategy. Keep each controller and termination check.
- Keep the old native sync proof unchanged. Attach recovered files
through the verified snapshot path in a fresh attempt.
- Recognize only the current same-task, active local directory in the
canonical task or attempt namespace. Keep it on later turns instead of
applying inherited Git defaults.
- Keep the current healthy directory when canonical Retry follows the
exact pre-allocation Git-base failure. Require the committed source
link, strict server-produced failure events, no execution evidence or
unfinished owner, and the current canonical binding. Recognize the known
configuration warning without accepting arbitrary or provider-origin
warnings.
- Keep owned-directory readiness separate from repository-free strategy
selection. A repaired real Git checkout with explicit
task/project/assignee Git policy uses normal Git validation and
preparation. Preserve explicit fresh, new sources, immutable native
bindings, and locked path/binding checks.
- Add real PostgreSQL and filesystem regressions for admission, file
preservation, and the complete later-turn heartbeat policy chain.

## Verification

- Real PostgreSQL callback/queue regressions are RED on unchanged
production and GREEN with this repair: one successor from the same
authenticated saved request, with filesystem snapshot preservation. The
24-case source/Retry matrix passes and retains termination, allocation,
generation and revocation negatives.
- Actual browser use exposed both later-turn Git-base failure and
canonical Retry selecting an empty sibling. The real heartbeat
selection/configuration/preparation/Retry regressions reproduce each.
All 81 workspace-preparation cases passed before the final repaired-Git
edge; all 17 final affected phase-chain/safety cases pass, including
that new case. Its finalized oracle checks effective Git policy, an
actual Git checkout and saved-file bytes. Known preparation warnings
pass; unknown/provider warnings and execution contradictions do not
certify reuse.
- Full workspace typecheck, full build with matching `5342bd52da` stamp,
maps and diff checks pass. Runtime commit `5342bd52da` passed CI with 53
successful checks and 2 conditional Storybook skips, and Greptile 5/5
with zero unresolved threads. The follow-up `9096fe8f62` removes only
the feature/access map additions as requested; runtime and test files
are byte-identical. Map structural and diff checks pass. Greptile
reviewed this exact final commit at 5/5 with no unresolved threads. All
test and build jobs passed. The final required `ci / verify` and `ci /
e2e` aggregate gates are queued without runners; one bounded gate-only
requeue retained every successful job. GitHub auto-merge is enabled and
waits for those gates.
- Normal UI acceptance on exact final source with actual OpenCode,
configured DeepSeek model and Daytona passes both turns: snapshot
continuation writes a useful review/report; reload and a second ordinary
message retain the same workspace and logical native/provider session.
All six files remain byte-identical, the report's computed SHA and
unpredictable marker are confirmed, the concrete result is actual 250
versus correct 350, native finalization/copyback commit, and normal
lease cleanup succeeds. A separate normal UI canonical Retry on
preceding `9ac1d29147` preserves the current directory/report/marker and
writes a continuity note; the final Git-policy split leaves that
exercised non-Git path unchanged.
- Boundaries: local app/server source with real remote native execution;
seeded historical failure/stop receipts and actual stored snapshot, not
a replay of the original staging allocation or external review
publication. Prior local verification rewrote installed dependency
metadata; its generated lock diff was retained and reverted. Final
launch/end source and dependency metadata hashes match; no installed
dependency byte-identity claim beyond that observed freeze.
- Required full local suite was attempted but remains incomplete:
embedded PostgreSQL bootstrap failures prevented assertions in several
suites, so only its owned test tree was stopped. A prior
workspace-recovery test timed out and the following seed collided with
an active hold; both unchanged tests passed in isolation. Runtime-head
`5342bd52da` hosted CI passed the full gate; the documentation-only
follow-up waits for the aggregate jobs noted above. Its [first Rust
attempt](https://github.com/paperclipai/paperclip/actions/runs/40528918372/job/122276159874)
failed an unchanged ACPX shutdown cleanup assertion; the preceding
same-Rust head passed. [One bounded Rust-only
retry](https://github.com/paperclipai/paperclip/actions/runs/40528918372/job/122281990961)
passed. No test assertion or production guard was weakened.

## Risks

- Incorrect input authority could repeat work. The existing owner
reauthorizes the same durable source; admission verifies its receipt
under the task lock.
- Directory recognition could select unrelated files. It requires the
current scoped binding and canonical namespace, then preparation
rechecks the stored binding and path under the source lock. Missing or
symlinked retained roots are not recreated.
- Snapshot preservation does not certify native copyback or old external
outcomes. The old run and its unknown outcomes stay unchanged.
Unconfirmed controllers or termination remain blocked.
- There is no schema migration or new public API.

## Model Used

OpenAI `gpt-6.1-sol` through Codex, with reasoning, code execution, and
tool use. The host does not expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — targeted suites; the full
local attempt limitation and final CI proof are documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 16:02:43 +00:00
DottaandPaperclip 663fe667d6 fix(recovery): reconstruct missing native workspace obligations (#17877)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A failed native run can still own prepared files in a remote
sandbox.
> - Cleanup must preserve those files before another executor can start.
> - Older failure producers can omit the preservation record.
> - A saved message then waits for cleanup without a recovery owner.
> - This change reconstructs that record during user continuation and
orphan cleanup.
> - The existing worker saves the files, verifies stop, and returns
input to normal admission.

## Linked Issues or Issue Description

**What happened?**

A failed native run retained a prepared workspace and an active sandbox
lease. It had no file-preservation obligation. Cleanup protected the
files. The next user message waited for a stop receipt that no worker
owned.

**Expected behavior**

A new message registers the missing maintenance work at once. Existing
orphan cleanup recovers historical saved input. Ordinary Retry remains
available. Saved input can start once, after verified preservation and
containment.

**Steps to reproduce**

1. Persist a failed native run with an exact prepared remote workspace
and its active lease.
2. Leave its finalizer terminal and unowned. Omit the workspace recovery
record.
3. Send a user comment or run orphan cleanup.
4. Observe that no preservation worker starts and the comment remains
deferred.

**Paperclip version or commit**

Reproduced on `bd382eee536f4ada680072b1cc19100163c3d04a`.

**Deployment mode**

Native Runner with a remote sandbox. Local verification uses disposable
PostgreSQL 18 and real temporary workspace files.

Refs #17788 and #17800. Those fixes preserve future deadline failures
and register historical preservation on Retry. This change adds
automatic recovery for new and saved messages. Refs #17839 for the
separate restart-failure producer repair.

## What Changed

- Register missing preservation before an authenticated continuation
waits for remote cleanup.
- Reconstruct the same exact-source obligation during the existing
orphan lease sweep.
- Notify the existing preservation worker in the registration
transaction. Do not add a timer or history scan.
- Test ephemeral and reusable sources, live owners, active executors,
mismatched sources, rollback, lock contention, Retry and saved-input
delivery.
- Document the recovery contract and feature-map behavior.

## Verification

- Red: both new orphan-recovery cases fail on unchanged production code.
Four ownership and source negative controls pass.
- Red: a new comment starts no preservation work without the admission
repair.
- All 461 test cases across five affected suites pass across the local
runs. One existing 30-second timing case timed out under concurrent
verification; its isolated rerun passed with the same deadline and
assertions.
- The three final-head entry journeys pass again: HTTP Retry, HTTP
comment creation, and historical saved-input recovery.
- `pnpm -r typecheck` and `pnpm build` pass on final head
`8fab8b2ba9049f640fd884094a2007a69465e6da`. The affected server
typecheck and build also pass.
- The integration journey holds the provider stop reply, checks that no
successor starts, preserves real files and newer host edits, and checks
one saved-input successor.
- Module-boundary checks, map checks and `git diff --check` pass.
- Native macOS PostgreSQL cannot start because host semaphore capacity
is exhausted. Database verification uses separate disposable PostgreSQL
databases for each suite. An earlier shared-database attempt caused
fixture interference and is not counted as a pass.
- Greptile gives 5/5 with no findings or unresolved threads on
`8fab8b2ba9049f640fd884094a2007a69465e6da`. All final-head GitHub checks
are settled: 55 passed, two Storybook checks skipped, and one neutral
Paperclip review. Full CI test, build, typecheck, Runner and browser
lanes pass. No full local `pnpm test:run` pass is claimed. The fix is
not deployed. This PR makes no claim of a live hosted model turn or
recovery of the original incident.

## Risks

- Registration grants maintenance authority only. Existing coordinator,
descriptor, allocation, stop, file-receipt and execution gates remain
effective.
- A locked source remains held. Its one-second lock timeout does not
prevent an unrelated source from registering recovery.
- New preservation writes wake existing work. Settled sources produce no
new recurring work. Historical discovery uses the existing orphan
cleanup lifecycle and its backoff.
- Provider lifecycle is simulated in local integration. PostgreSQL, HTTP
routing, file export/merge, the worker and queue admission are real.
- No schema, credential, model, trust, retry-budget or assignment
change.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection, tool
use and code execution. The runtime does not expose the exact model ID
or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 09:50:50 -05:00
Devin FoleyandPaperclip 832c44f26a fix: resume user follow-ups that reassign a task (#17821)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The execution control plane saves user messages while an earlier run
needs recovery.
> - A user comment can also assign a task back to its agent.
> - That update creates an assignment wake, but continuation accepted
only comment wake reasons.
> - The saved question could stay blocked after workspace recovery and
confirmed process cleanup.
> - This pull request admits assignment wakes that carry a valid,
undelivered user comment through the existing continuation checks.
> - The same rule applies to new messages and saved requests already
waiting for recovery.

## Linked Issues or Issue Description

**What happened?**

A human follow-up that also assigns a task back to its agent produces
`issue_assigned`. After a native run stops during handoff, the
continuation policy rejects that wake reason. The question remains saved
behind the execution recovery hold even after the workspace is recovered
and the old sandbox is destroyed.

**Expected behavior**

Deliver the saved human question once after the existing stop,
ownership, authority, and execution gates pass. Assignment alone must
not grant permission to continue.

**Steps to reproduce**

1. Finish task work by assigning it to a human for review. Let the
native run stop with `issue_reassigned` and retain its recovery hold.
2. Recover the workspace and confirm destruction of the old sandbox.
3. Post a new human question while assigning the task back to the agent.
4. Observe that the assignment wake stays deferred instead of delivering
the question.

Refs: #17288, #14156. These related changes cover legacy reassignment
and assignments without comments. This change covers a human comment
carried by an assignment wake behind a native or legacy continuation
hold.

## What Changed

- Share the comment-bearing wake classification between continuation
admission and saved-request recovery.
- Accept `issue_assigned` only through the existing persisted
user-comment validation and execution gates.
- Add database regressions for new and saved assignment wakes,
concurrent recovery, single delivery, and unchanged prior run results.
- Check that bare assignments, invalid authors and comments, changed
ownership, active execution, and pending approvals or questions retain
their holds. Cover native and legacy cleanup paths.
- Update execution semantics and both recovery maps.

## Verification

- Red: both new and saved assignment-with-comment cases stayed deferred
without a successor on the original code.
- Green: the final continuation suite passes all 284 tests. The four
companion recovery, retry, and queue suites pass another 208 tests.
- `pnpm -r typecheck` and `pnpm build` pass.
- `git diff --check`, `node scripts/check-maps.mjs`, and `node
scripts/check-module-boundaries.mjs` pass.
- Apex review is 5/5 on `56c9c2d90dcda55b3f760946c9aba6aee448c361`, with
no unresolved review threads. The only earlier finding concerned the
contribution rule against new source comments; those comments were
removed.
- Full `pnpm test:run` is still running locally. CI is waiting for the
`paperclip-public-pr-x64` runner fleet; the superseded workflow was
cancelled so the current head could proceed. No passing CI claim is made
yet.
- Live recovery: corrected one existing saved wake to the supported
comment reason. The normal server admitted one successor with the exact
saved question and prior-run context, then started real tool calls. This
verifies admission on the existing deployment; the new source change is
not deployed yet. The resumed turn later failed with a separate provider
terminal event whose error was null, followed by a workspace restore
failure. The task is blocked again on that failure. Its useful answer
has not been delivered, so end-to-end completion remains unverified.

## Risks

- Assignment wakes now reach continuation validation when they carry a
comment. The persisted comment must still belong to the task and
requesting human, follow the stopped run, and remain undelivered.
- The existing stop proof, cleanup, current assignment, approval, pause,
budget, and competing-execution checks remain effective. Regression
cases cover both native and legacy paths.
- No schema migration or data rewrite is required for saved requests.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, source inspection, tool use,
and code execution. The runtime does not expose the exact model ID or
context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 20:19:16 -07:00
DottaandPaperclip 3f1ad6b25b fix(chat): resume task Retry after retained-source containment (#17317)
## Thinking Path

> - Paperclip manages agent work through task and execution ownership.
> - A task Retry must retain the user's request while the system
verifies an old executor.
> - Connected conversations also retain their external principal and
accepted source batch.
> - Chat Retry admission required containment before it could save a
wait for that recovery.
> - A saved chat receipt could also remain deferred after containment
became available.
> - This pull request lets the exact recovery-bound request wait and
resume through its chat owner.
> - Normal admission still checks containment, current access and policy
before one successor starts.

## Linked Issues or Issue Description

**What happened?**

An ordinary task Retry on a connected conversation failed with
`failed_chat_retry_not_authorized` while its retained remote executor
still needed verification. The new recovery generation existed, but the
chat action failed before it could save a dispatch wait. This occurred
after the repairs in #17267 were confirmed live.

The local regression found a second gap. A processed chat action with a
deferred wake was not reconsidered after containment. Re-entering normal
admission also tried to insert its existing durable receipt again. A
historical hold for the same failed run could also hide the current
Retry's workspace strategy, so the request still waited after the old
executor was contained. Preparation could then read that sibling instead
of the recovery record that owned the fresh successor, losing the
fresh-workspace decision.

**Expected behavior**

Save the exact authorized Retry while its matching recovery generation
verifies the executor. Once containment is recorded, reconsider the same
chat receipt through normal admission. Start one successor with the
original source batch. Keep publication, principal, pause and budget
checks.

**Steps to reproduce**

1. Admit a GitHub or Telegram request and record a failed native run
with a retained remote lease.
2. Leave the executor's termination unconfirmed and exhaust its prior
recovery generation.
3. Use ordinary task Retry to create a new recovery generation.
4. Process the connected Retry. The old admission code fails instead of
retaining a wait.
5. With a deferred receipt, record exact containment and reconcile
again. The old dispatch path leaves it stranded.

**Paperclip version or commit**

Reproduced from `412914cfa78d5abefcc34d3eb8d4bb86e461caac`. The branch
is rebased onto current master.

**Deployment mode**

Hosted tenant for the reported failure. Isolated local PostgreSQL and
the assembled heartbeat/chat services for the regression.

Refs #17267 and #16965. This change does not add a Cloud dependency or
change the absence protocol.

## What Changed

- Permit a saved admission wait only for the exact current task Retry
and active source recovery generation.
- Keep promotion and execution dependent on independent containment
proof.
- Bind a Retry to active automatic recovery without replacing its
generation, attempt count or deadline.
- Share the current exact Retry strategy across historical holds for the
same run. Retain each hold's stop and effect checks.
- Select the canonical recovery record that proves the committed fresh
successor during workspace preparation.
- Let task receipt reconciliation return the same deferred action to its
chat owner.
- Recheck current board and external authority, then update the existing
wake atomically with its successor.
- Preserve the exact chat source batch. Keep generic board-comment
filtering unchanged.
- Add GitHub and Telegram cases for waiting, containment, one successor,
stale generations, source edits, endpoint pause and supersession.
- Update recovery documentation and capability maps.

## Verification

- Red: real PostgreSQL tests reproduced the admission rejection for
GitHub and Telegram. Six active-recovery cases then failed on the
generation-ID assumption. Two same-run historical-hold cases remained
deferred after containment on the old strategy lookup. A preparation
regression then returned no readiness when the historical hold was
stored first.
- Green: 73 selected chat integration tests and 394 continuation, task
Retry, workspace recovery, recovery-state and workspace-preparation
tests passed. The chat tests use the real chat admission and heartbeat
queue. Unrelated capacity stays occupied so no real provider starts.
They assert no executor before containment, one successor afterward,
unchanged source leases while waiting, preserved recovery budgets, and
denial after authority changes. A separate source hold still blocks
work. Both sibling insertion orders pass through the committed-readiness
lookup and real workspace preparation: the new directory has independent
Git storage and files, and unpublished old work remains intact.
- `pnpm -r typecheck`, `node scripts/check-maps.mjs` and `git diff
--check` passed after the review fixes. The source build passed on
`f6660efa48e37a80cd9857217a537606de85b5c8`.
- The local host reached its PostgreSQL semaphore limit. Successful runs
used a test-only loader to reduce disposable cluster connection and
worker limits. No repository database helper or application setting
changed.
- The full local `pnpm test:run` began before the final review edits,
failed in its first server phase and was stopped after those failures.
Later phases did not run. This is not a full local pass. The transient
Retry-notice case passes after fixture isolation; the four Discord
close-owned publication cases also pass in isolation. Other broad-run
failure causes are not established.
- Exact-head CI and security checks pass on
`f6660efa48e37a80cd9857217a537606de85b5c8`; Greptile is 5/5 with zero
unresolved review threads. Optional Paperclip Review completed neutral
with a 5/5 static assessment of all 11 changed files and no findings; it
did not run tests or hosted acceptance. All 57 current-head checks are
terminal: 54 success, two skipped and the optional neutral review, with
no failures or pending checks.
- The real hosted workflow confirmed that the deployed absence repair
recorded containment. A later ordinary UI Retry was accepted and saved a
durable request, but it remains behind the historical source hold. The
useful existing review remains unchanged. This follow-up is not
deployed; a live successful successor and final task disposition remain
unverified. No missing files are claimed recovered.

## Risks

- The existing task reconciliation sweep can restage a deferred chat
action. This reuses its receipt and generation; it does not grant new
execution or publication authority.
- Dispatch must keep the board Retry identity separate from the
connected request's principal. The regression checks these bindings and
preserves current chat authorization at each phase.
- Historical holds reuse a strategy only from the locked same-run
recovery that names the current Retry intent. A different run, malformed
recovery record or independent source hold remains fenced.
- Unknown executors without a matching active recovery still fail
admission. Promotion and execution require containment even when
admission saved a wait.
- No schema or API shape changes. No credential changes, manual
deployment or service restart.

## Model Used

OpenAI GPT-6 through Codex, with repository tools, code execution and
browser verification. The exact served model variant and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 07:51:34 -07:00
DottaandPaperclip 898144f4b3 fix: stop idle execution-control recovery polling (#17094)
Replace idle execution-control polling with workers driven by committed writes and persisted deadlines. Preserve startup recovery, bounded retries, and shutdown safety. Stabilize the Slack setup browser regression by waiting for the restored setup screen before simulating its webhook.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-21 09:54:28 -05:00
DottaandPaperclip d33b40a5c3 fix: recover saved work and provide reliable task Retry (#16965)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The control plane must retain work when an execution or workspace
export fails.
> - Retained-workspace holds can leave saved input waiting with no
usable Retry action.
> - Users cannot verify process ownership or repair provider
allocations.
> - This pull request gives system recovery a durable budget and gives
each deliberate Retry a durable intent.
> - Safe recovery can preserve files, settle an accepted result, or
prepare an independent workspace.
> - One task state keeps infrastructure diagnostics out of the ordinary
conversation while preserving real permission and approval gates.

## Linked Issues or Issue Description

**What happened?**

A task with a retained workspace can show recovery, failed-run and
saved-message warnings at the same time. Historical runs without native
workspace descriptors cannot use the native restore path. Old recovery
metadata can prevent a new attempt even when an independent workspace
would be safe.

**Expected behavior**

Paperclip owns bounded infrastructure retries and file preservation. A
user can request another attempt with ordinary Retry. Thinking appears
immediately. The service preserves completed effects and current user
controls. It does not claim that starting fresh recovered the old files.

**Steps to reproduce**

1. Retain a failed run and its workspace after export or process-stop
failure.
2. Save a follow-up while the recovery hold is active.
3. Open the task and try to continue without infrastructure credentials
or operator certification.

Refs #16906, #16835, #16883. This extends their recovery and receipt
contracts. Related work: #16962 and #16963.

## What Changed

- Add task-level Retry with company-scoped authorization, all-status
idempotency and durable admission before physical work. Keep exact-run
Retry compatible.
- Add typed recovery phases, finite persisted backoff, historical
adoption and durable local failure diagnostics.
- Preserve native workspaces or verified historical snapshots. Admit an
independent fresh workspace only after execution containment. Keep
old-source preservation and cleanup obligations separate.
- Carry completed publication and API receipts forward. Fence uncertain
mutation retries across replacement runs. Preserve connected-chat
identity and authorization.
- Re-enter normal dispatch for saved input. Reuse accepted results
through the existing finalizer.
- Use one task interruption projection across the thread, composer and
task list. Show optimistic Thinking, ordinary Retry and genuine user
gates.
- Add PostgreSQL, filesystem, API, browser-fixture and controlled
provider coverage. Document the behavior and update the feature/access
maps.

## Verification

- All GitHub Actions CI jobs pass on
`6d8c6d27d50586d8548b31d03b47f6d73a0ee11d`: typecheck, all
server/chat/workspace and serialized suites, Runner checks, eight
browser shards, canary and build. The required merge gate passes.
- Greptile reviewed the final commit at 5/5, with no actionable findings
or unresolved review threads. Its two earlier P1 findings are fixed.
- Full recursive typecheck and build pass locally on the final commit.
All 16 task Retry cases pass against isolated real PostgreSQL. The
unrelated-success case and both stale Retry cases fail on the previous
code and pass after the fix.
- The corrected gateway fixture passes its uncertain-mutation and
independent-read case. Four real HTTP API cases pass for definite
403/422 rejection, completed-effect reuse, and uncertain-effect fences
across deliberate and automatic retries. These focused gateway/API
checks used a temporary database lifecycle harness on an existing
healthy local PostgreSQL process. The standard repository helper is
unchanged.
- Earlier qualification covers 291 task UI cases, saved-file copyback
and ancestry, current Agent Chat ownership races, nine chat Retry API
browser cases, and six assembled browser recovery fixtures. A real
API/PostgreSQL/native-runner journey showed Thinking in 25 ms despite a
delayed response, admitted one successor in a separate workspace,
retained original files, and survived reload.
- The controlled Daytona test interrupts export, joins the controller,
restarts it in a separate process, and verifies one agent start, one
external sentinel operation, retained file contents and no premature
cleanup. It does not establish abrupt-kill or live LLM/GitHub
acceptance.
- The full local `pnpm test:run` was attempted and was not green: an
OAuth row-lock timing assertion failed, and a later suite exhausted
macOS PostgreSQL semaphores. The ten OAuth timing cases passed on rerun
against a separate database. Later embedded database initialization
failures remain recorded; the final CI matrix passes.
- Security scanning passed on production-code commit
`63953edf069b111c6becc3144dffa6deb52f68b5`. Its only successor changes
one test-fixture identity line. The final scan is neutral/inconclusive
because its pinned merge-base checkout failed or timed out. Both rerun
API requests returned 404, so no fresh successful scan is claimed.
Socket and Snyk checks pass on the final head.
- The optional Paperclip Review remains queued behind an earlier run in
workspace finalization. Its prior completed assessment was incomplete
because a comparison tool was blocked by the platform filter. It is not
counted as a completed code review. GitHub does not require this check
for merge.
- Hosted acceptance remains pending deployment. Local and CI results do
not establish recovery of a hosted task or its retained files.

## Risks

- Recovery changes span admission, physical ownership and dispatch.
Incorrect evidence could permit overlapping work. Existing physical
locks and authoritative stop requirements remain in place.
- The migration adds concurrent indexes. It requires the existing
top-level migration runner and retries incomplete index creation safely.
- A fresh attempt can complete while old-source preservation remains
unresolved. The service retains that obligation and does not report
those files as recovered.
- Unknown tool mutations remain blocked until their authoritative
receipt owner resolves them. Generic upstream tools have no universal
result-query protocol.
- Recovery and diagnostics stay inside the stack. No Cloud incident
endpoint, notification worker or new outbound reporting channel is
included.

## Model Used

OpenAI Codex, GPT-6 family. The runtime does not expose a more specific
model version or context-window size. The model used reasoning, shell
execution, source editing and browser tools. No model subagents were
used for this implementation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; the full
local run limitation is documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 19:58:53 -05:00
DottaandPaperclip bc091f0a21 fix(recovery): recover retained workspaces before saved messages (#16906)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A stopped run can leave useful files in its original sandbox.
> - The server keeps that sandbox when it cannot confirm file copyback.
> - The task can then save new user input but offers no safe way to
recover the files.
> - A retry of the old agent turn can repeat work that already finished.
> - This pull request recovers files through the existing workspace
owner and delivers only saved human input.
> - The user can recover a task without certifying infrastructure facts
or losing completed work.

## Linked Issues or Issue Description

Refs #16883 and #16835 for publication receipts and verified process
termination.
Related: #16478 clears stale holds after copyback already succeeded.
This change performs the missing export and records its proof. #16901
manages the main GitHub response comment and does not perform workspace
repair.

**What happened?**

A failed run retained its workspace. A later user message stayed saved
behind a file-repair hold. The task asked the user to verify process
termination and file recovery but offered no repair control. Sending a
message could show an optimistic Working state and move the task to Todo
before admission succeeded.

**Expected behavior**

The harness should recover the original files when it can prove
ownership and termination. It should preserve newer edits and completed
operations. Only waiting user input may start another turn. Missing
evidence must keep the source held with a specific explanation.

**Steps to reproduce**

1. Interrupt a sandbox run before workspace finalization succeeds.
2. Retain its exact allocation and save a user follow-up.
3. Observe the file-repair hold and missing end-user repair path.

## What Changed

- Add source-bound background recovery for native and legacy runs. An
accepted native result is not required.
- Share the physical copyback lock and durable owner. Resume only the
exact stopped allocation, export files, verify persistence and obtain a
fresh stop receipt.
- Persist retry limits and idempotent requests on the existing recovery
record. Limit each server to two background exports. Add an indexed
candidate scan and a separate cleanup obligation.
- Add a task-editor recovery endpoint. Deny agents and viewers. Reject
caller-supplied stop or repair claims.
- Deliver waiting human input once through ordinary admission. Preserve
pause, approval, budget and assignment gates. Require historical
conversational-adapter evidence for legacy-to-Runner continuation.
- Include exact GitHub publication receipts in continuation context.
Preserve unknown effects and the original failed run.
- Show observed recovery progress, a transient-failure retry action and
specific service blockers. Keep saved messages visible. Suppress false
startup feedback and implicit Todo promotion behind a file hold. Hide
unavailable workspace links.
- Update execution semantics, capability maps and production-component
stories.

## Verification

- Red: the original read model failed the new recovery-progress
assertion; the other 23 checks in that file passed.
- Passed: recovery regressions with 28 cases with real PostgreSQL,
files, tar and Git operations. Lifecycle provider calls are controlled
test boundaries. Coverage includes native/legacy export, concurrent
owners, rollback after export, interrupted archive export with a new
database connection, lost replies, historical discovery, durable retry
limits, separate cleanup and saved-input delivery. The actual candidate
query uses its partial index with 3,000 unrelated history rows. A first
automatic repair can bind verified old-controller stop proof without an
accepted native result. Review also reproduced and fixed export-only
continuation and cleanup lost after a committed repair. Cleanup remains
discoverable without waiting input and after task reconciliation.
- Passed: two GitHub integration cases verify that continuation context
carries the existing publication receipt and does not write another
response. Company and task boundaries are checked.
- Passed: 293 continuation, workspace-sync, platform-receipt and OpenAPI
checks. After file repair, actual dispatch retains pause, cancellation,
approval, real budget-policy and assignment gates.
- Passed: 174 route checks, including editor admission, agent/viewer
denial and rejection of fabricated repair claims.
- Passed: 351 task detail, workspace properties and comment-reopen
checks after updating the new read boundaries and asynchronous
workspace-link expectations. Two new route cases keep held tasks Blocked
when a comment is saved.
- Passed: 265 task-thread and recovery-notice checks. The broader
continuation suite also passed; its new legacy migration case first
exposed a gap that the focused recovery suite now covers.
- Passed after rebase: `pnpm -r typecheck` and `pnpm build`. The later
master merge adds an unrelated permission migration; recovery migration
generation and safety checks passed after the merge.
- Passed: token gates, module boundaries, map structure and migration
checks. The recovery index is migration 0360 after current master.
- Browser: inspected the production recovery component and its
retry-to-queued transition with fixture data. The first dev dependency
load failed; a reload after optimization resolved it. This is UI
evidence, not a live provider recovery claim.
- Every test shard passed on final commit `bbbc09efb4`. The older local
`pnpm test:run` was stopped after the complete CI test matrix passed.
Local full typecheck and production build passed. The first CI build
compiled and booted, then lost its runner during the clean public
npm-install check. An isolated retry passed the build, production boot
and clean public npm-install verification.
- Greptile reviewed final commit `bbbc09efb4` at 5/5. All review threads
are resolved.
- The change is not deployed. Hosted recovery remains unverified until
the served revision includes this change.

## Risks

- Recovery crosses process, provider and persistence boundaries. Missing
or mismatched ownership, stop, descriptor or file evidence keeps the
hold. It never selects a replacement allocation or silently starts over.
- File recovery does not certify external action outcomes or authorize
replay. New input still uses normal execution admission.
- Historical sources without sufficient durable evidence can remain
blocked. Destructive reconstruction stays separate.
- The index migration is idempotent. It changes no task data. Snapshot
pruning follows repository policy.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, with code execution and browser tools.
`gpt-6-astra` assisted earlier investigation. The runtime does not
expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-19 21:51:46 -05:00