0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-01 15:35:12 +01:00

Files

T

DottaandPaperclip 8a3d2cb04f fix(recovery): avoid recursive holds for undispatched waits (#16894 )

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Queue admission must stop work while a previous execution needs
recovery.
> - The final dispatch gate can cancel a queued run before it starts.
> - Legacy recovery treats this known wait as an unknown provider
outcome and creates another hold.
> - The extra hold can survive repair of the original execution and
leave saved messages stuck.
> - This pull request uses one database proof to prevent the extra
incident and recognize old false holds.
> - Saved input can then pass through normal admission with the original
work and safety gates intact.

## Linked Issues or Issue Description

Refs #15483.

**What happened?**

The dispatcher writes an `execution_reconciliation_required`
cancellation when an existing recovery hold denies a queued run. No
runtime is selected and no provider starts. Legacy recovery later
creates a `legacy_execution_requires_reconciliation` action for that
cancelled admission. Its settled `replay: blocked` record remains an
effective hold after the real source recovery finishes. The task has
saved messages but no eligible Retry or Continue action.

**Expected behavior**

A verified admission denial stays a wait. It must not create another
provider-outcome incident. Existing false incidents must stop blocking
after the server verifies their recorded source. Real execution,
uncertain outcomes and file recovery stay held. The harness determines
these infrastructure facts; the user does not certify them in a form.

**Steps to reproduce**

1. Persist a queued-run gate cancellation with its task-bound
`executionWait` receipt and no execution authority or resource evidence.
2. Run legacy reconciliation. On the old code, it creates a second
incident.
3. Persist the old incident as resolved with replay blocked, then clear
the original source hold.
4. Restart queue recovery with saved user messages. On the old code, no
successor is admitted.

Related work: #15965 covers other pre-start dispatch cancellations but
explicitly excludes this existing-hold code and leaves historical holds
unchanged. #16634 covers stale-cancel classification and queue drain,
without re-evaluating historical effective holds. #15383 covers an
operator Stop during conversation startup. This change addresses the
existing-hold receipt and its persisted false incident; it does not copy
those broader changes.

## What Changed

- Add one database proof for a legacy admission denied by an existing
execution hold. Require the exact dispatcher receipt, matching task
identity, no runtime or process claim, no lease or finalization
coordinator, and no execution events.
- Use the proof before legacy recovery creates an incident and when it
computes effective historical holds. Keep the old audit rows.
- Preserve source file obligations and the original recovery action.
Keep the normal queue, pause, ownership, company and budget checks.
- Add real PostgreSQL regressions for incident prevention, old state,
concurrent queue recovery, message order, conflicting evidence and scope
boundaries.
- Update the execution contract and both recovery maps.

## Verification

- Red: the regression on unfixed source fails on incident
classification, historical hold projection and actual periodic
saved-message admission. The other 24 safety cases pass.
- Green: all 27 new database cases pass. Two safety assertions were
aligned with the existing contract: daily limits stop dispatch after
queue promotion; a human comment can intentionally reopen a terminal
task. The final regression checks the real daily dispatch gate and agent
pause.
- Existing `explicit-native-continuation.test.ts`: 255 tests pass.
Existing legacy recovery and PostgreSQL dispatcher suites: 61 tests
pass.
- `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs` and
`git diff --check` pass.
- GitHub PR CI run `39313204660` passed on
`ef6009e656b7c9b4c788480dce322874565960ac`. All 54 successful check-runs
and the Snyk status are green. Two optional Storybook checks were
skipped. The independent Paperclip Review completed with a neutral check
and a 5/5 assessment with no findings. Greptile also reports 5/5 on the
same commit. There are no unresolved review threads.
- `pnpm test:run` is still running locally. Its overall result is
pending; no full local suite pass is claimed. The exact-head GitHub CI
suite passed.
- A read-only probe of an affected hosted task verifies the same
database proof and projects no false blocker with the working source. No
live rows were changed. Deployment and a real resumed provider run
remain unverified.

## Risks

- An incorrect exclusion could admit work with uncertain effects. The
proof fails closed on missing receipts, source mismatches,
runtime/process ownership, any retained lease, coordinator, execution
event or file obligation. Real upstream file recovery remains blocking
in the regression.
- The effective-hold query adds indexed existence checks scoped to the
recorded run and company. It makes no provider calls and performs no
database repair writes.
- Historical audit records keep their recorded outcome. Only their
effective blocking status changes when the server can prove the
admission never ran.
- No schema, permission, credential or UI form changes are included.
Live recovery still needs verification after rollout.

## Model Used

OpenAI Codex, model ID `gpt-6.1-sol`, with reasoning, tool use and code
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-19 15:26:39 -05:00

16 KiB

Raw Blame History

Recovery — agent access

Product map: Recovery.

Review date: 2026-10-12. Source commit: 58e16dbddfc9c8d0cf60867f9d9847f0feda055b. Native REST transport reconciled in this change; see the shared transport contract.

This is a source-reviewed baseline of representative operations and known gaps. Each capability needs operation-level review before its entire scope can be qualified. No live agent execution is claimed. Follow the map contract for format, permission checks, evidence and maintenance.

`diagnosis`

Review: baseline.

API: present — representative operation IDs:

GET /api/issues/{id}/diagnostics/blockers.
GET /api/issues/{id}/recovery-actions.

Agent access: available — REST / native call_api; CLI issue diagnostics.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board. Graceful legacy shutdown retains the exact sandbox until its owned adapter settles, including after the first shutdown wait. A renewed settlement fence protects cleanup by other controllers. Loss of the owner retains the source for repair after expiry. This automatic protection grants no agent operation, execution permission, or bypass of workspace repair gates. The server can recover a prepared remote bootstrap only with a durable never-claimed executor receipt and no contradictory execution evidence. This automatic classification grants no new agent action or permission; normal execution admission and durable runner-state verification still apply. The current server's active preparation remains with its existing execution. A preparation that loses the coordinator lease cannot fail the shared run or revoke the new owner's gateway tokens, even after lease expiry. Only normal recovery can establish a takeover claim; no agent permission is added. Older unmarked preparations do not qualify for this automatic path during an upgrade. The fencing marker comes from fresh server-owned identity creation; caller profile data cannot upgrade an existing preparation.

Remote process-health reads retry within a bounded budget. An unavailable probe cannot authorize replacement or erase the original ownership marker. This is internal supervision; it grants no new agent recovery action. Existing task, company, process-identity and workspace-recovery checks remain required.

An execution-hold denial before legacy runtime selection is not a provider attempt. The server verifies the dispatch receipt and task identity against persisted execution, lease and coordinator evidence. The same proof excludes historical false incidents from effective holds while retaining audit history. This internal classification grants no new agent API or resolution authority. The original hold, file recovery, company scope and normal admission gates stay effective. Local database regressions cover automatic saved-message admission and denied cases; live provider recovery remains a separate verification step.

Evidence: source-only — implementation at the reviewed commit above.

Gaps: Diagnostics expose recorded cause and boundaries; source-backed reading does not prove successful recovery.

`bounded-continuation`

Review: baseline.

API: present — representative operation IDs:

POST /api/issues/{id}/recovery-actions/resolve.

Agent access: partial — Eligible agent REST / native call_api / CLI under source-bound recovery authority.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Gaps: Eligible resolutions use native call_api. The route retains owner and management authority, active-run/source-trust checks, board-only outcomes and reconciliation controls.

Scoped source review: 2026-10-13, recovery correction against ea6c8eb5e16d61111968f633a66748b397b64a3e. The recovery service reuses the existing completed-onboarding handoff validator. It checks the source company, onboarding origin, Done parent, assignee and Done child, with no unfinished siblings, before exempting the report from the issue-terminal backstop. Other terminal work and process-death/controller ownership checks remain enforced. This internal wake handling adds no recovery-resolution or cross-company agent authority. Real PostgreSQL regressions cover allowed and rejected bindings; revised live agent authorization and callback delivery remain not-run.

`source-action`

The recovery-actions read API includes settled execution reconciliation holds with an automatic replay: blocked receipt, as well as retained workspace holds. The existing company and task-read checks remain in force. The board notice reads this history to explain its current hold; this adds no agent mutation authority. Route and notice regressions cover the source contract, without claiming live agent or provider qualification.

Review: baseline.

API: present — representative operation IDs:

GET /api/issues/{id}/recovery-actions.
POST /api/issues/{id}/recovery-actions/resolve.

Agent access: partial — Eligible agent REST / native call_api / CLI; source-bound recovery state.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board. Completion or cancellation of a task does not clear its exact unresolved native workspace owner hold. Physical owner release still requires the operator stop-verification procedure.

For a resolved legacy or native action that retains a file hold, the same resolution API supports source-only file repair. Board runtime management and recovery-resolution authority are required. The request identifies the exact source action/run and records recovered-file evidence; the server verifies the original stop receipt. It preserves the current task status and does not wake or replay the agent. The task notice does not expose a file-certification form. The API still fails closed without permission or source bindings. Agent Chat retains the file diagnosis while this hold exists and suppresses the warning only for a verified durable recovery path. The repair route regressions and task notice cover the local contract; they do not independently recover files or qualify a live provider.

Evidence: source-only — implementation at the reviewed commit above. Exact owner hold retention in the recovery state helper was reviewed at 89770ba707fb1ebd4cabd184eec2e7fee226e0c3.

Verification: not-run — exercise a listed permitted operation with actual agent credentials/run context; inspect its result and applicable persistence/attribution. Assert the relevant denied actor, owner, company or mode from Policy. Review the remaining operations before qualifying the whole capability. Local recovery route regressions verify one unchanged hold across reads and sweeps on done/cancelled tasks, no physical copyback, and cleanup after the receipt is cleared. Exercise this sequence in a disposable instance; automatic cleanup must use the system actor and must not create reader participation. No live recovery proof is claimed.

Gaps: Permitted resolutions use native call_api under existing eligibility and ownership checks. Source/action policy determines whether continuation is allowed.

A recorded operator Stop before legacy runtime selection releases only that run's task locks and preserves deferred input for normal later admission. This does not grant an agent a new retry or provider-stop authority. Local database regressions cover the release boundary and a competing active routine cycle; they are not live provider-stop proof.

`exhaustion`

Review: baseline.

API: present — representative operation IDs:

GET /api/issues/{id}/recovery-actions.
GET /api/heartbeat-runs/{runId}.

Agent access: partial — REST / native call_api reads; operator escalation flow.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Gaps: Reading exhaustion is supported. Continuing past the recorded gate requires the specified owner/action.

`workspace`

Review: baseline.

API: present — representative operation IDs:

POST /api/issues/{id}/recovery-actions/retry-workspace-export.
POST /api/issues/{id}/recovery-actions/reconstruct-workspace.
POST /api/issues/{id}/recovery-actions/resume-workspace-finalization.
POST /api/issues/{id}/recovery-actions/resolve for retained-source file repair.
POST /api/execution-workspaces/{id}/reconcile-branch.

Agent access: restricted — Board/runtime management APIs; agent reads diagnosis and performs permitted workspace work.

Policy: Export retry and branch reconciliation require board runtime management. Reconstruction also requires board runtime management. It verifies exact source destruction and same-task attachment hashes, records lost work and external action outcomes, and admits one fresh continuation through normal execution gates. The old failed export remains failed. A retained or uncertain source cannot use it. An intervening native run-only cancellation requires its own acknowledged Stop, confirmed stopped leases, board file repair and not_performed action outcome. The reconstruction records and rechecks that exact proof; other later runs stay denied. Retained legacy file-only repair requires the board, exact stopped-source receipts, and recovered-file evidence. It can record repair for a successful legacy turn without changing its result or authorizing replay. Queued and running sources remain denied. Source-only reconciliation regressions cover this boundary. Physical-owner stop confirmation uses the separate runtime:confirm_owner_stop action. It requires active company owner/admin membership, trusted local board access, or instance administration. Agents and ordinary runtime managers are denied. The exact source task, running native run, accepted result and owner receipt must match; active copyback locks, live local writers, and newer runs prevent admission. An authorized administrator must first verify through the deployment platform that the exact controller and all copyback children stopped, then attest that evidence. Elapsed time, an expired coordinator lease or a lost advisory connection is not proof. Only the exact owner receipt is released; normal reconciliation performs export and commitment without another provider turn or bypassing existing task waits.

Evidence: source-only — route, administrator authorization, and owner admission reviewed for this change. Local authorization, route, UI and finalization regressions cover member/agent denial, exact-owner admission and the real reconciler preserving monitor and approval waits. This is not live provider-stop proof.

Verification: not-run for live agent/deployment execution — assert the documented agent denial with an owned test fixture; verify any listed agent request/result path separately. A delegated operation needs its own policy and access review.

Gaps: Export retry and branch reconciliation require board runtime:manage at their HTTP routes. The exact-owner operator fallback additionally requires runtime:confirm_owner_stop. Native call_api preserves these agent denials. Automatic managed-cloud recovery uses independently verified, signed platform proof through the finalization reconciler; agents cannot attest owner shutdown.

`unknown-outcome`

Review: baseline.

API: present — representative operation IDs:

POST /api/chat-endpoints/{endpointId}/publications/{publicationId}/resolve.

Agent access: restricted — Board provider-outcome resolution; agent may inspect assigned provider tools.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Verification: not-run — assert the documented agent denial with an owned test fixture; verify any listed agent request/result path separately. A delegated operation needs its own policy and access review.

Gaps: Uncertain-side-effect resolution is operator/source-specific; do not replay to obtain apparent parity.

`user-continuation`

Review: baseline.

API: present — representative operation IDs:

POST /api/issues/{id}/comments.

Agent access: partial — User comment API and server-owned wake reconsideration; native agent consumes bound wake.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Gaps: Receiving the continuation is supported; impersonating a saved user message or retry control is not.

`reconnect`

Review: baseline.

API: present — representative operation IDs:

GET /api/companies/{companyId}/events/ws.
GET /api/issues/{id}.

Agent access: partial — Existing event protocol; REST refresh of persisted task state.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Gaps: Streaming reconnect needs its protocol client. Polling/read APIs remain available; replay is not proof of a fresh successful action.

16 KiB Raw Blame History

Recovery — agent access

diagnosis

bounded-continuation

source-action

exhaustion

workspace

unknown-outcome

user-continuation

reconnect

16 KiB

Raw Blame History

`diagnosis`

`bounded-continuation`

`source-action`

`exhaustion`

`workspace`

`unknown-outcome`

`user-continuation`

`reconnect`