Commit Graph
5 Commits
Author SHA1 Message Date
DottaandPaperclip e3fb4bc73a fix(native): recover saved listener startup failures (#19479)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native recovery must preserve saved work and respect current task
authority.
> - A closed local API listener can stop startup before a provider
starts.
> - #19462 gives new failures a bounded bootstrap retry.
> - Older saved failures still have a generic terminal code and a
recovery hold.
> - This pull request verifies the old producer outcome and uses the
existing safe replacement path.
> - Eligible tasks continue once. Unknown provider outcomes and user
holds remain blocked.

## Linked Issues or Issue Description

Refs: #19462

**What happened?**

A restart can leave an old listener startup failure in
`terminal_failure` with `native_session_interrupted`. Startup recovery
does not claim it after #19462. A completed file recovery can also
retain its no-replay hold.

**Expected behavior**

Continue automatically when durable records prove that no provider
started, the source stopped, and the current files were saved. Keep the
existing attempt budget and normal admission checks.

**Steps to reproduce**

1. Close the actual API listener after a fresh native coordinator claim.
2. Persist the old executor's generic failure. Save the workspace and
confirm source termination.
3. Restart recovery. The baseline schedules no successor. The repair
schedules one successor after all receipts pass.

**Paperclip version or commit**

Baseline: `0252421d5c`. Old failure producer: `2f35eb1080`.

**Deployment mode**

Built from source. Local macOS arm64, Node 26.4.0, real embedded
PostgreSQL, independent filesystem execution. No external AI service is
used by the controlled acceptance.

## What Changed

- Recognize the exact saved local-listener guard through one proof
predicate shared by discovery and the locked transaction.
- Require a fresh first attempt, matching execution bindings, no prior
provider lineage, no provider events or tool calls, no checkpoint or
admission receipt, and no cancellation.
- Reuse current snapshot, exact stop, source block, attempt budget, and
successor idempotency checks.
- Wait for completed source file-recovery receipts. Retire only that
same source's verified preservation hold in the successor transaction.
- Bound discovery to 100 candidates per pass, keep completed history
quiet, and wake through existing transactional producer notifications.
- Retain retries after transient database and final ownership-proof
errors; advance the bounded cursor and force a rescan so a failing page
cannot hide later work.
- Let only the replacement duty discharge a proved-empty notification
during drain through its existing read-only readiness check, once per
notification generation. Keep real work and concurrent changes
outstanding without running replacement during drain.
- Wake replacement at canonical local-stop proof writes and the existing
dispatch eligibility boundary; these hints grant no new execution
authority.
- Add database regressions, scheduler quietness checks, real controller
handoff coverage, and an opt-in controlled Runner continuation.

## Verification

Final production head: `0df41b2fef2644195d1224d36984af3827da3e3b`.

Local environment: macOS arm64, Node 26.4.0, TypeScript 7.0.2, Vitest
5.0.3, frozen-lockfile dependencies. Tests execute freshly;
`--fsModuleCache` reuses only content-validated module transforms.

- Native safe replacement: 62/62 PASS on isolated PostgreSQL 17 (63.18
seconds), including clean per-suite database disposal. All 62 assertions
also passed on embedded PostgreSQL 18 beta, but its Darwin cleanup hook
timed out; that invocation is not counted as a pass. Docker's disk was
full, so the disposable PostgreSQL 17 data used a size-limited RAM
mount, with no host mounts. The owned container was removed afterward.
- Actual stopped-process producer suite: 31/31 PASS. A real exited child
produces the stop receipt, and its replacement notification settles only
after the outer transaction commits.
- Workspace and separate-process drain regressions: 33/33 PASS (75.90
seconds).
- Shared coordinator and execution-control lifecycle checks: 39/39 PASS
(16.43 seconds). Pending work, concurrent changes, cold startup,
shutdown, and passive drain boundaries remain covered.
- Actual suppressed Cloud routine/drain workflow: PASS (34.64 seconds),
using the unchanged test and timeouts. Its matching local RED reproduced
CI's 462-query, zero-pending-query, one-idle-hold timeout after an
admitted write committed during drain (69.46 seconds).
- Actual historical controller handoff: PASS (48.40 seconds). A separate
controller loads the original executor from `2f35eb1080`, writes its
generic listener failure, and exits. Recovery at the final head admits
exactly one successor. The real Rust Runner and controlled Codex process
complete governed `paperclip_finish`; attempt two commits and the saved
file remains intact. Only the child loads the old artifact, and both
isolated artifacts are restored afterward. Rust source subtree
`77961b70f177ec3c38d11abcaec92fd3d0e96ec0` matches the built binary. No
external AI service is used.
- Seven corrected existing fixtures pass: transactional stop-proof calls
(5 cases) and notification inventory (2 cases). Their earlier CI
failures exposed fixture calls outside the documented transaction and an
outdated expected queue set.
- Server typecheck: PASS.

Red/green proofs also cover the original saved failure on baseline
`0252421d5c` (zero successors instead of one), held-history quietness,
bounded forward scanning through 100 parked rows and concurrent wakes,
real snapshot/cleanup producer notifications, actual issue-row lock
timeout, and task/history eligibility notifications. An obsolete
prior-run question can be skipped to request another proof check; actual
skip while the task is paused or the source owns provider tool activity
schedules no successor.

The failed-page regression starts with 25 persistent verifier errors and
a healthy candidate 26. Baseline RED repeats the first page. Repair
GREEN advances the healthy candidate, revisits the original failures
after their fault clears, creates all 26 successors exactly once, and
becomes quiet.

Greptile is SUCCESS, 5/5 on the exact final head (2026-11-06 17:09:21
UTC). Final-head CI is GREEN: 56 successful checks, three intentional
skips, one neutral result, and no pending or failing checks. [Attempt
two and
results](https://github.com/paperclipai/paperclip/actions/runs/42258352876/attempts/2)
include passing build, typecheck, Rust, native server, package, browser
E2E, serialized server, general server/chat tests, and final
verification. Five jobs in the first attempt were interrupted together
at 17:13:12 UTC; the native-server log records an explicit runner
shutdown signal. The single infrastructure retry reran only those five
jobs and dependent verification, preserving the 49 successful checks on
the unchanged head. No timeout was weakened, and setup failures or
interrupted invocations are not counted as passes.

Focused commands from the repository root:

```sh
./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/services/remote-workspaces.integration.test.ts server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts -t 'native safe-replacement decision|reconciles the drained remote backend' --fsModuleCache --reporter=dot
./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/__tests__/execution-control-work.test.ts server/src/__tests__/delivery-work-coordinator.test.ts -t 'execution-control lifecycle|work coordinator' --fsModuleCache --reporter=dot
node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit --pretty false
```

The historical acceptance sets `PAPERCLIP_TEST_HISTORICAL_LISTENER=1`
and uses an original-executor child artifact with a `try/finally`
restore. Normal regressions require no Rust binary.

## Risks

- Compatibility is deliberately limited to positively proved fresh
remote startup failures. Other old ambiguous failures remain held.
- Existing provider work, a changed task block, incomplete preservation,
or revoked authority must prevent execution. Normal retry admission
remains in control.
- This PR does not deploy the change or mutate a hosted task. Hosted
recovery still requires the repaired revision and all current proof
gates.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The exact backend model identifier and context window are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 11:49:28 -06:00
Nicky LeachandClaude Opus 5.5 026dc6d135 fix(native): release a session-resume owner when the resumed run fails before its provider claim (#19371)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a finalization row. Its lease owner tells the
control plane that an execution may still be live.
> - Pausing an agent waits for `agentExecutionsHaveStopped`. That check
refuses to finish while any run of the agent has a lease owner, unless
the run was cancelled and has stop proof.
> - A same-run session resume claims that lease before it dispatches the
run. If the run then fails before the native executor takes its own
provider claim, nothing releases the resume owner.
> - The leaked owner blocks pause and resume for good, and the task
keeps looking like it has a live recovery path.
> - This pull request settles those rows at the source, and it adds a
stop-proof-gated release for rows that already leaked.
> - The benefit is that agents can pause and resume after an ordinary
resume failure, without database edits, and the cancelled-run stop-proof
rule stays as it is.

## Linked Issues or Issue Description

**What happened?**

An agent stayed in `pausing` with `hostComplete=false` and climbing
attempts. Its runs had `native_run_finalizations` rows with `phase:
observed`, `recovery_state: resuming_session`, a `lease_owner` like
`default:resume:<uuid>`, and no `result_id`. Each heartbeat run had
ended `failed`: either `setup_failed` (`Failed to acquire lease for
environment … Sandbox creation has failed`,
`native_remote_recovery_lease_mismatch`) or `adapter_failed`
(`runner_state_identity_mismatch: …`). The lease expired about 20
minutes after the run finished. Nothing settled the row. The only fix
was to set `lease_owner` to null by hand.

**Expected behavior**

When a resumed run fails before any provider turn starts, its
finalization row goes to `terminal_failure` and its lease owner is
released, as the native executor does for failures it sees. A row that
leaked before this fix is released in the background once its lease
expires and the run has stop proof. Pause then completes.

**Steps to reproduce**

1. Leave a native run in `retryable_failure` with a persisted
checkpoint, on a task that already has an active recovery incident.
2. Let session resume claim it (`claimNativeSessionResumptions` sets
`phase: observed` and a `…:resume:…` owner).
3. Make the resumed run fail before the executor claims it, for example
when the environment driver cannot create the sandbox.
4. The heartbeat failure path marks the run `failed`. The issue release
path does not settle the row, because it settles only when it opens a
new incident. The owner stays set, and `agentExecutionsHaveStopped`
returns false indefinitely.

**Paperclip version or commit**

`master` at 5ac05871b5.

**Additional context**

Root cause: two writers can end a resumed run without the executor.
`failRun` and the setup-failure path in `heartbeat/run-completion.ts`
never touch the coordinator. `recordNativeTerminalRecoveryIfNeeded` in
the wake-queue adapter settles the coordinator only when no recovery
incident exists yet. Agents that need their computer connection paired
again usually already have an incident, so the owner leaks.

Related, but different: #18905, #18866, #18072, and #17692 change how
the stop check judges runs. This pull request does not change the stop
check.

## What Changed

-
`server/src/services/native-runtime/native-failed-run-owner-release.ts`
(new): one settlement for a result-less `observed` owner on a run that
ended `failed`, `timed_out`, or `interrupted`. It writes
`terminal_failure`, null lease fields, `recoveryState: "blocked"`, the
run's failure code, and a `releasedOwner` record. It sets `nativePhase`,
sends the same finalization and execution work notifications as the
executor's terminal path, and logs `native.failed_run_owner_released`
activity. Locks follow the issue, coordinator, run order.
- `settleUnstartedNativeSessionResume`: runs after a dispatched session
resume returns. It matches only the exact claim owner and the attempt
the claim saw. The executor increments the attempt when it takes
provider ownership, so an unchanged attempt proves that no provider turn
started under this owner. The executor's own failures no longer match.
- `releaseAbandonedNativeOwners`: the background release for rows that
already leaked. It requires an expired lease and all of the stop proof
that `agentExecutionsHaveStopped` checks: settled environment leases, no
live run process or process group, no live controller lease, and no live
controller process. This process answers from its in-memory execution
registries. It skips cancelled runs, pending native Stops, and pinned
workspace restores, because those have their own stop-proof owners.
- `native-finalization-reconciler.ts`: `NativeSessionResumeClaim` now
carries `attempt`.
- `heartbeat.ts` and `heartbeat/recovery.ts`: both session-resume
dispatch sites settle the claim after `executeRun` returns. This is one
`.finally` per site.
- `execution-control-reconciliation.ts` and `execution-control-work.ts`:
the execution-finalization queue runs the background release. Its
deadline includes the earliest lease expiry among eligible rows, so the
release does not poll an idle instance. Rows with unreleased sandboxes
are excluded from the deadline.
- `agent-execution-stop.ts` is unchanged. The cancelled-only release
keeps its stop-proof rule.

## Verification

- `npx vitest run
src/__tests__/heartbeat-native-resume-lease-failure.test.ts` drives the
real heartbeat resume dispatch (`reapOrphanedRuns`). A sandbox driver's
`acquireRunLease` throws `Sandbox creation has failed`, and the task has
a recovery incident. The run fails with `Failed to acquire lease for
environment "Managed computer" (sandbox): Sandbox creation has failed`,
and the coordinator ends `terminal_failure` with no lease owner. I
removed the `.finally` wiring and ran the test again. It then failed,
and the row stayed `observed` with a `default:resume:` owner and a
20-minute expiry, which is the production state.
- `npx vitest run
src/services/native-runtime/native-failed-run-owner-release.test.ts` (10
tests):
- A claimed resume that fails at lease acquisition is settled.
`agentExecutionsHaveStopped` changes from false to true. A repeated
settlement does nothing.
- An owner the executor claimed (incremented attempt) and a run that is
still running are not changed.
- A leaked `runner_state_identity_mismatch` row is released by
`reconcileAbandonedExecutionControl`, and then
`agentExecutionsHaveStopped` returns true.
- An unexpired owner still blocks the stop check. Its expiry becomes the
finalization queue deadline.
- An active environment lease and a live controller process block
release. After both stop, the row is released.
  - Cancelled runs and pending native Stops are not changed.
  - Added during review:
    - Releasing an owner wakes the cost-accounting worker.
- Releasing a run's environment lease wakes the finalization worker, and
the owner is then released.
- A recorded PID and group that now belong to an unrelated process do
not hold the owner.
- Bounded passes rotate past a held row (`[0, 1, 1]` with page size 1),
then wrap back to it.
- The immediate settle keeps the owner while the run's recorded process
is alive.
- Suites that cover the changed files pass together: the two new files,
`native-session-resumption.test.ts` (with the claim shape updated),
`execution-control-reconciliation.test.ts`,
`execution-control-work.test.ts`, and
`agent-lifecycle/__tests__/lifecycle.test.ts`. With
`environment-cleanup-work.test.ts` added, that is 7 files and 156 tests.
I did not run the full `heartbeat-process-recovery.test.ts` suite
locally. CI runs it.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Behaviour change: a resumed run that fails during setup now ends with
a terminal coordinator in every case. Before, this happened only when
the issue release path opened a new incident. After a restart, restart
recovery no longer sees such a row as `observed`. This matches what the
release path already does when no incident exists.
- The background release trusts an expired lease on a run that is
already terminal, together with the same process and lease evidence that
the stop check uses. A controller on another host cannot be probed by
PID. The stop check has the same limit today. A live owner still renews
its lease, and lease-gated writes from an expired owner are already
refused.
- If an expired row's process or controller still looks live, the
release retries at the shared 15-second finalization cadence until that
process exits. Each pass handles one bounded page, and a rotating cursor
moves to the next page on each pass.
- A run-scoped environment cleanup notification now also wakes the
execution-finalization queue. Passes are coalesced, and a pass is a
cheap deadline query when nothing is eligible.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
(shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 07:19:09 -08:00
DottaandClaude Opus 5.5 5c185db77c fix(tree-control): resume work a subtree pause interrupted (#19394)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Task tree control lets the board pause a task tree and then resume
it.
> - A pause stops the runs that are active in the tree. The release must
then continue the work that was pending.
> - A child task whose run was active at pause time did not start again
after release. The child became blocked and asked for a user message.
> - A pause is an operator hold, not a failure. It must not strand work,
and it must not require reconciliation for a run that never started.
> - This pull request re-dispatches interrupted work one time on
release, and it settles a legacy Stop that never entered the adapter.
> - The benefit is that pause and resume keep a task tree moving without
manual recovery, and real safety gates stay in force.

## Linked Issues or Issue Description

**What happened?**

A parent task delegated two children. One child's run was active (in
host workspace preparation, before the adapter started) when the board
paused the parent's tree. The pause cancelled that run. After the board
released the hold, no new run started for that child. The child became
`blocked` with the execution blocker cause
`legacy_execution_requires_reconciliation`. Later the next action became
"Send a message to continue." A sibling child that had no active run at
pause time resumed normally.

**Expected behavior**

A run that a subtree pause cancels is an operator hold. On release, the
pending work starts again one time, automatically. It does not go
through execution reconciliation when the run never entered its adapter.
It does not wait for a user message.

**Steps to reproduce**

1. Create a parent task and a child task that is assigned to an agent.
2. Wake the child. While its run is in host preparation (or queued),
call `POST /api/issues/:parentId/tree-holds` with `{ "mode": "pause" }`.
3. Call `POST /api/issues/:parentId/tree-holds/:holdId/release` without
`wakeAgents`.
4. Before this change, no new run starts, and `getExecutionBlocker`
reports `legacy_execution_requires_reconciliation` for the child.

**Root cause**

- (a) The pause records each interrupted run in the hold member snapshot
(`activeRunId`). Nothing used that record on release.
- (b) Release woke tasks only when `metadata.wakeAgents === true`.
Without it, release relied on stranded-issue recovery. That recovery
stands down when the latest run was cancelled by an operator
(`isOperatorCancelledRun`). So the interrupted child had no owner.
- (c) The pause Stop of a legacy run in preparation is written directly
as `cancelled`. Legacy terminalization classifies a cancelled run with
no bootstrap evidence as an unknown outcome. It records
`legacy_execution_requires_reconciliation`, although the gated dispatch
never entered the adapter.
- Native runs: the verified native Stop does not create the
reconciliation hold, but gap (b) applies to native runs too. The release
change is runtime-independent.

## What Changed

- `issueTreeControlService.findOwedPauseResume` decides if a task is
owed one resume. The task's latest run must be an operator Stop that a
pause hold member snapshot names, recorded while that hold was active.
The task must still be executable by the same agent, and no pause may
still cover it. The obligation belongs to the run, so overlapping pauses
keep it until the last covering release.
- New `services/issue-tree-resume.ts` re-dispatches owed work with wake
reason `issue_tree_resumed` (`holdId` and `interruptedRunId` in the
context). A compare-and-set claim (`treeHoldResume`) on the interrupted
run makes the resume happen at most once. A failed wake returns the
claim.
- Pause release and `mode: "resume"` call it for every member of the
released hold, also without `wakeAgents`. When `wakeAgents` is true, the
existing loop wakes every executable member.
- A task that still has an execution blocker is not replayed. The
release writes `issue.tree_resume_wake_skipped`, the blocker stays
visible, and the obligation stays unclaimed.
- `releaseHold` updates only an `active` hold. A repeated release gets
`409`.
- `settleCancelledLegacyPreparation`: when the executor that owned a
cancelled legacy run proves that dispatch never entered the adapter, it
records bootstrap evidence and resolves that run's
`legacy-execution:<runId>` hold as `false_positive`. It also corrects an
automatic no-replay disposition and restores a `blocked` status only
when that disposition made the change. Then it re-dispatches owed work,
for a release that happened while the Stop was unwinding. It requires a
requested Stop with no existing recovery receipt, and no process,
session, native finalization, unreleased lease, or execution-start
event.
- `settleUnrecoverableExecutions` waits while a legacy executor still
holds its controller lease, and records a `legacyCancellationBlock`
receipt when it blocks a task for a cancelled legacy run.
- Docs: `doc/execution-semantics.md` and `doc/composer-stop.md` describe
the resume rule.

## Verification

- New regression
`server/src/__tests__/issue-tree-pause-resume-run.test.ts` uses embedded
Postgres, the real heartbeat, and the real tree-control routes:
- A run paused during preparation: nothing runs during the hold, there
is no blocker, and release creates exactly one `issue_tree_resumed` run
that succeeds. A second release returns `409` and creates no run.
- Release before the stopped preparation finishes unwinding (with the
periodic disposition run in between): exactly one resumed run after the
executor settles.
- An automatic no-replay disposition that ran first: the blocker clears,
the status returns to `in_progress`, and release resumes once.
- Overlapping child and parent pauses: no resume while the parent pause
remains, then exactly one resume after it is released.
- A queued run cancelled by the pause: release creates exactly one
resumed run.
  - A task reassigned to a user during the hold: no resume.
- A mid-turn legacy Stop with unknown outcome: no replay, the
reconciliation blocker remains, and `issue.tree_resume_wake_skipped` is
logged.
- Red before each fix: the first 4 cases failed on master; the 3 review
cases failed on the first PR commit. All 7 pass now.
- Also passed locally: `issue-tree-control-routes`,
`issue-tree-control-service`, `heartbeat-workspace-busy`,
`heartbeat-undispatched-execution-wait`, `execution-control-work`,
`legacy-workspace-restore-recovery`, `legacy-continuation-authority`,
`task-retry`, `legacy-execution-recovery`, `agent-conversations`,
`heartbeat-retry-scheduling`, `satisfied-connection-intents`.
- Not verified here: a live staging pause/resume. Re-check on staging
after deploy.

## Risks

- Behavior change: release now starts the work that the pause
interrupted, also without `wakeAgents`. Before, that work stayed idle.
Other idle tasks still wake only with `wakeAgents`.
- The settlement trusts the executor's adapter-entry flag. The flag is
set only inside the atomic dispatch gate, which refuses a cancelled run.
The settlement also requires no process, session, lease, or execution
event.
- A native run whose Stop has not finalized at release time is not a
target. Stranded recovery and explicit wake still apply.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:08:01 -06:00
DottaandPaperclip 0373f9f261 fix(recovery): wake retained workspace preservation after Retry (#17741)
## Thinking Path

> - Paperclip lets people manage AI agents and their saved work.
> - A stopped run can owe workspace preservation before saved input can
continue.
> - Retry saved a recovery request but did not wake the worker that owns
preservation.
> - Older recovery records could also take the bounded scan slots before
a due request.
> - A second Retry could change the revision while the physical owner
was still saving files.
> - This pull request wakes a separate worker after commit, selects due
work first, and keeps the active owner's revision.
> - Saved work can progress without replaying the stopped provider turn
or blocking other finalization.

## Linked Issues or Issue Description

**What happened?**

An operator approved a saved design and selected Retry after native
workspace finalization failed. The API accepted one canonical request.
The source remained scheduled at zero preservation attempts for more
than 30 minutes. No new provider turn started. A local regression also
found that repeated Retry during export invalidated the physical owner's
completion revision.

**Expected behavior**

Committed Retry must wake bounded source preservation. Due recovery must
progress ahead of passive cleanup obligations and unrelated history.
Duplicate requests must keep one physical attempt. Saved input must
still pass normal admission after verified recovery.

**Steps to reproduce**

1. Retain a terminal run with an exhausted workspace recovery and saved
authorized input.
2. Let the delivery coordinator become quiet, then request Retry.
3. Observe the recovery state and physical preservation receipt. Repeat
Retry while the export is held.

**Paperclip version or commit**

Observed on served commit `4b89eec644d6e1a99d58926e77ae58927c3fa87b`.
Local baseline: `a1f77821b9cd5c9c846fdf7a0fa5cc620ad2b6bb`.

Related work: Refs #17053, #17267, #17094. This change repairs explicit
Retry delivery. It is separate from the run instruction projection in
#17724 and historical unsafe-export discovery in #17687.

## What Changed

- Notify a dedicated workspace recovery duty from the transaction that
saves Retry, recovery admission, adoption, or recovery progress.
- Reconstruct work and saved retry deadlines at startup. Fill each free
export slot independently, track physical work for idle drain, and join
the dedicated worker's exports on shutdown.
- Drain one bounded pass through completed cleanup and saved-input
obligations. Restart that pass after committed changes without
repeatedly rechecking paused input.
- Share source eligibility between deadline selection and candidate
consumption. Keep resolved files-only obligations and ignore unrelated
resolved or cancelled history.
- Select due unfinished preservation before passive historical records.
Rotate past source locks held by other workers and retain the indexed
source lookup.
- Bind the canonical Retry intent without changing an active physical
attempt's generation or revision. Preserve that binding across later
progress writes.
- Add real producer, physical preservation, startup, deadline, idle, and
independent finalization regressions. Document the delivery contract.

## Verification

- Red controls: Retry without a notification stayed scheduled; duplicate
Retry invalidated the active export; whole-batch waiting stranded a
third Retry; a scan without cursor rotation starved a source behind two
actual advisory locks. Removing unfinished-first selection left HTTP
Retry scheduled behind five real recovered sources still owing cleanup.
The fixture explicitly verifies their eligibility with the worker's
actual query filter.
- Further red controls: the previous worker visits only two of three
eligible completed sources. Disabling the committed-change cursor reset
misses a new earlier source arriving while two callbacks are held. Both
pass with the final worker.
- Green: the public Retry HTTP route, registered production worker, real
file copyback, duplicate canonical receipts, and public recovery receipt
reload passed on an owned local fixture. Its agent is paused to keep
this files-only check from starting a provider turn.
- Full workspace recovery suite passed 56 tests, including the
foreign-lock regression.
- Execution-control lifecycle suite passed 24 tests. It covers idle SQL,
stored deadlines, cancelled and resolved history, independent
finalization, and shutdown. One, three, and seven paused completed
sources are each visited in a bounded pass, then stay deferred without a
successor or SQL during 90 seconds of fake-clock idle time. A committed
earlier source is evaluated when it arrives during two held callbacks.
- Both GitHub and Telegram `automatic_running` continuation regressions
passed with the new canonical intent binding.
- `pnpm -r typecheck` passed.
- `pnpm build` passed.
- `node scripts/check-maps.mjs` and `node
scripts/check-module-boundaries.mjs` passed.
- Local `pnpm test:run` could not complete: embedded PostgreSQL
repeatedly failed to start. The owned run was stopped after this setup
failure repeated. Latest-head CI supplies broad coverage.
- Earlier setup failures included a localhost sandbox denial and loss of
the owned disposable Docker database. These were separate from the
product red controls above.
- Local database tests use an explicitly named disposable PostgreSQL
`_test` database. That opt-in database is reset before the suite.
Default embedded test behavior is unchanged.
- Exact-head CI is green at `ef7dcf1ba1060864c9d216c33b51f4d9f8f57fbd`;
Greptile and Ponytail both rate it 5/5, with no unresolved review
threads. Earlier CI caught the indexed discovery assertion and two exact
chat intent-binding regressions; all three were fixed and pass locally
and in CI. A separate chat shard lost contact with its self-hosted
runner without a product assertion result. A later shared runner
shutdown interrupted several jobs; one infrastructure rerun passed
before the final test-and-worker follow-up triggered a fresh green run.
- No staging deployment is part of this PR. The original approval
remains saved; deployed recovery still needs acceptance after rollout.

## Risks

- More recovery writes now wake a bounded maintenance duty. Completed
history must not keep it active. Idle and deadline tests cover this
boundary.
- Resolved files-only obligations must remain eligible without granting
replay. The startup regression checks that path.
- Provider stop proof, copyback ownership, source binding, retry
budgets, and normal execution admission remain required. There is no
schema or public API change.

## Model Used

- OpenAI Codex based on GPT-6, with code execution and browser tools.
The exact serving model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 10:28:12 -05:00
DottaandPaperclip 898144f4b3 fix: stop idle execution-control recovery polling (#17094)
Replace idle execution-control polling with workers driven by committed writes and persisted deadlines. Preserve startup recovery, bounded retries, and shutdown safety. Stabilize the Slack setup browser regression by waiting for the restored setup screen before simulating its webhook.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-21 09:54:28 -05:00