Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-23 09:07:46 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
572bef4ba7679451ac1543db25cb8ebd533bf919
PaperClipAI/maps/feature-map/recovery.md
T
DottaandPaperclip 4145b5048a docs: deepen task, approval, secret, and recovery maps (#16994) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its feature map describes user journeys. Its access map describes
the matching agent paths.
> - The recent index and capability additions improved discovery and
breadth.
> - Broad baseline records still combined operations with different
actors, permissions, and retry behavior.
> - This pull request reviews the existing task, approval, secret, and
recovery capabilities at operation level.
> - Each operation now has its own access boundary, source evidence, and
verification recipe. Source review remains separate from live proof.

## Linked Issues or Issue Description

Refs: #16987, #16990, #16025. These related pull requests are merged.
The search found no open duplicate or matching issue.

**Issue type**

Missing documentation.

**Where is the issue?**

The paired task, questions-and-approvals, secrets, and recovery files
under `maps/feature-map/` and `maps/agent-access-map/`.

**What's wrong?**

Baseline records do not show enough detail to verify who can perform
each action, which state must persist, or what happens after a denial,
race, or retry. A declared scenario tool can also be mistaken for an
admitted production tool.

**Suggested fix**

Split the four areas into source-reviewed operation records. Add compact
checkpoints to their user recipes. Preserve existing capability IDs and
the map schema.

## What Changed

- Promote 27 existing capability records to `operation-reviewed`, with
125 operation records: tasks 32, questions and approvals 39, secrets 28,
recovery 26.
- Give each operation its own API, agent surface, policy, source
evidence, verification recipe, and remaining gap.
- Separate human, legacy agent, run JWT, native binding, mode, and
resource authority. Split operations where those rules differ.
- Correct production completion paths, plan acceptance and handoff,
question dismissal and withdrawal, secret proposal authority, runtime
recovery grants, and explicit duplicate-risk decisions.
- Preserve access gaps found in source, including the binding-proposal
dependency cascade and narrower native/CLI fields.
- Add one checkpoint table per product recipe for saved results, denial,
races, and retries.
- Keep all 191 existing capability IDs. Change only eight Markdown
files. Keep every operation's evidence `source-only` and verification
`not-run`; the subset below does not prove an entire operation across
its actors and surfaces.

## Verification

Source review uses revision `3bbc2104b65caf9f7763938a01260a5673e12f7a`.
Read-only area reviews and a second source review covered the changed
pairs.

Passed:

- `node scripts/check-maps.mjs`: 35 paired areas, 191 capability IDs,
valid references and fields. This is structural verification.
- `node --test scripts/check-maps.test.mjs
scripts/storybook-feature-index.test.mjs`: 17 passed.
- `node scripts/storybook-catalog.mjs`: 308 files, 2,899 stories.
Generated outputs remain ignored.
- Capability comparison with the base: all 191 IDs unchanged; the four
reviewed areas contain 27 capabilities and 125 source-only operation
records.
- `pnpm -r typecheck` and `pnpm build`.
- `git diff --check`.
- Focused existing interaction and proposal route tests: 80 passed. The
two selected DB-backed suites skipped 119 tests because local embedded
PostgreSQL could not start. These skipped tests are not passes.

Live subset on a fresh local instance with a disposable PostgreSQL
container:

- Created the company and an unassigned task through the browser. Added
a comment and completed the task. The saved status and comment survived
reload.
- Created a managed secret with inert dummy data through the browser.
Its metadata and version survived reload. The metadata response did not
contain the dummy value.
- Created a question fixture through board REST. Invalid input returned
422 and left it pending. Browser dismissal kept it pending and preserved
the draft. Reopening restored the choice. Submission survived reload. A
duplicate response returned 409 and left the saved answer unchanged.
- Used a real issued legacy agent API key. Task read and safe
company-secret catalog returned 200. Full secret listing and run-only
secret list/value routes returned 403. Revoked the key after the checks.
- Read an empty recovery projection. Resolving a nonexistent action
returned 404 and left the task unchanged. This did not execute recovery.
- Closed the owned browser tab and stopped only the verified test
runtime and labelled database container. Evidence survived cleanup. No
provider or model ran.

Limits and retained failures:

- Native embedded PostgreSQL launch failed because the host could not
allocate semaphores. The container-backed app was the successful
alternate environment.
- Full `pnpm test:run` was attempted. Fifteen DB-dependent
native-session cases failed at database startup, and several DB suites
skipped. The run was stopped after repeated failures of that
prerequisite. The full local suite did not pass.
- The sandboxed build attempt failed at the local tsx IPC socket. The
required build passed outside the sandbox.
- The first CI build job passed compilation and production boot, then
its npm-install check was interrupted when the runner received a
shutdown signal. The corrected head passed the full CI run, including
the build and public-install checks.
- Initial Greptile review found the recovery note field and native
search/history limits were inaccurate. Corrected `resolutionNote` and
the production cap of 100, then reran the map checks. Greptile reviewed
the corrected head at 5/5; both earlier threads are resolved.
- An initial harness used a nonexistent generic secret-value URL and
received 404. The actual mapped agent routes were then checked
separately. An initial recovery assertion expected 409; the actual
missing-action contract returned 404. These were harness corrections,
not product fixes.
- Native run bindings, two-person ownership, secret delivery/rotation,
external vaults, provider-stop proof, workspace repair, and actual
recovery execution remain unexecuted. The map does not claim those
journeys passed.

The separate external **Paperclip Review** check is still in progress on
`904509df80cc6de16d36906a410a66786f619b20`. The reviewer reports that
its full changed-file request is pending. CI is green and Greptile is
5/5, but this external review has not published a final assessment. Its
review service must finish that request and assessment.

## Risks

Documentation can become stale as policy and runtime bindings change.
Each operation cites source and retains its verification gaps. The
records describe observed enforcement; they do not authorize broader
access or certify live behavior. This pull request changes no product
code, schema, or permission enforcement.

## Model Used

OpenAI Codex, `gpt-6.1-sol`; 475,000-token context window; xhigh
reasoning; tool use and code execution. Read-only source agents assisted
the area reviews.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused map and route
checks; full-suite host blocker recorded above)
- [x] I have added or updated tests where applicable (existing map
regression tests cover this documentation-only change)
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 13:59:25 -05:00

355 lines
25 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
navigation: {"section":"Projects and execution","sectionOrder":40,"order":50}
---
# Recovery
People can see why a task stopped, who owns the next action, and whether a safe
continuation is available. Recovery retains task context and ownership, honors
pause and budget gates, and distinguishes confirmed failure from an uncertain
external outcome. A retry acknowledgement alone is not proof that work resumed.
Feature ID: `recovery`.
Agent access: [companion](../agent-access-map/recovery.md).
## Sub-features
- `diagnosis`: `recovery/diagnosis` — a task/run exposes the recorded cause, evidence, owner, attempt
progress, and next action without requiring a raw-log interpretation.
- `bounded-continuation`: `recovery/bounded-continuation` — supported interrupted work continues with preserved
conversation context and bounded attempts, without blindly replaying tools.
- `source-action`: `recovery/source-action` — source-scoped recovery actions remain attached to the task;
independent repair work may have its own issue and return owner.
- `exhaustion`: `recovery/exhaustion` — exhausted or overdue recovery exposes the current gate and
permitted operator action; it does not claim an automatic path is still live.
- `workspace`: `recovery/workspace` — workspace divergence/export failures show the applicable repair,
reconcile, or isolated reissue action with evidence and authority checks.
Retained-file recovery runs in the harness with observed task progress and a
Retry recovery action only after transient failures. It preserves newer files,
keeps uncertain sources held, and delivers saved human input through ordinary
admission. The task does not expose infrastructure-certification forms or a
workspace link when the destination is disabled. Local PostgreSQL and real
filesystem regressions cover native and legacy recovery and one-time delivery;
hosted provider recovery remains separate acceptance.
- `unknown-outcome`: `recovery/unknown-outcome` — uncertain external side effects require inspection and
an explicit decision, not automatic replay.
- `user-continuation`: `recovery/user-continuation` — saved user messages after a stop are reconsidered once
cleanup proves the old execution is stopped; duplicate wakes are avoided.
- `reconnect`: `recovery/reconnect` — browser stream recovery refreshes persisted state without claiming
the agent itself restarted or that a failed run succeeded.
## How to get to it (user POV)
### `source-task`
Open the affected task from Tasks, Inbox/Blocked, search, or a direct task link.
Inspect the recovery notice/card in its history, expand details if needed, and
use the currently offered action. Follow any linked repair task and return owner.
An authorized board user can expand Recovery details on a stopped-run notice to
read the decision that still blocks replay, including a settled recovery record.
The notice matches the current action and run; it does not show another run's
historical decision or authorize a retry. Source-reviewed behavior; the
[notice regressions](../../ui/src/components/ExecutionBlockerNotice.test.tsx)
cover empty active recovery, a mismatched source, and failed history reads.
### `run-detail`
Open the agent's run (`/agents/:agentId/runs/:runId`) from activity or the task's
run link. The run page hosts workspace recovery controls only for a failed run
with error code `workspace_validation_failed` and a linked source task that still
has a live `workspace_validation` recovery action. Use the source task for other
stopped-run recovery checks.
### `operator-retry`
On an exhausted disposition or workspace recovery card, inspect the current
reason and use the permitted retry/repair/reconcile action. Some repairs require
confirmation; unavailable actions must explain the gate or remain absent.
### `post-stop-message`
After a stop, send a new instruction on the same task or inspect one already
saved while cleanup was pending. Observe the waiting message, continuation, and
eventual result rather than manually changing the task status to look complete.
A continuation denied before startup by an existing execution hold remains a
wait. Once the original hold clears, the server verifies the dispatch receipt
and absence of execution evidence, ignores any historical false incident for
that unused admission, and reconsiders saved input through normal admission.
No infrastructure-certification form is needed. The
[database regressions](../../server/src/__tests__/heartbeat-undispatched-execution-wait.test.ts)
cover incident prevention, retained audit history, concurrent restart admission,
ordered messages, contradictory execution evidence, and unchanged pause and
task gates. These tests do not qualify a live provider or prove task completion.
### `browser-reconnect`
While viewing an active or recovering task, interrupt the browser connection
and restore it, or suspend and return to the tab. Reload is a separate check.
The current task and run state should reconcile with the server.
## Driving it
Preconditions: follow the [baseline](./README.md#before-driving-a-journey).
Use a disposable instance and reproducible failure. Record source task/run,
adapter/runtime mode, cause, attempt count, current owner, and verified stop
evidence. Do not kill an unrelated server or induce provider side effects just
to get a failure screen. Use the existing deterministic recovery suite first.
**Operation checkpoints**
Use these checks alongside the entry-point recipes below and the operation records in the [access companion](../agent-access-map/recovery.md). Record the actual actor, surface, saved result and denial separately. Source review and linked tests do not establish a live pass.
| Capability ID | Action and persisted result | Denial, race and retry checks |
| --- | --- | --- |
| `recovery/diagnosis` | Open the source task and exact run; compare cause, source action, original owner, recovery owner, attempt bounds and next action after reload. | Test an unrelated/private task and a stale run notice. Read-side retirement of a stale action must not create a new attempt, participant, or permission to replay. |
| `recovery/bounded-continuation` | Use a deterministic recoverable interruption and inspect the original run/session, attempt receipts, saved result and eventual useful output. | Repeat restart/receipt delivery, race Stop or a new owner, and retain unknown external outcomes. Accepted completion/finalization recovery must not start a new provider turn or repeat its side effect. |
| `recovery/source-action` | Inspect active and retained actions, follow the repair owner, and resolve one eligible source-bound action with its exact ID. | Retry the settled resolution and reject a stale/foreign action. Recovery ownership alone cannot mutate the source task; inspect source status, audit and wake independently. |
| `recovery/exhaustion` | Exhaust or age a bounded fixture; use the currently offered retry once, then inspect its durable acknowledgement and actual continuation. | Reassign, pause, stop, close, add an unresolved dependency, or impose a budget hold before retry. The old notice must not reopen terminal work or run the wrong owner. |
| `recovery/workspace` | Exercise retained-file recovery, export retry, reconstruction, branch repair/reconciliation and exact-owner finalization as distinct authorized operator paths. | Use exact source/action/run/receipt bindings and preserve existing files. Reject stale source, live/uncertain writers, unsafe archives and insufficient authority. Verify actual files and accepted result separately from queued acknowledgement. |
| `recovery/unknown-outcome` | Inspect the provider and saved intent before recording a source-specific chat, email or governed-provider outcome. | Use performed, not-performed and unresolved cases. Wrong company/request, stale state and unauthorized resolution must fail. Unknown effects must not replay automatically. An explicit retry must record the operator’s duplicate-risk acceptance and dispatch only the authorized attempt. |
| `recovery/user-continuation` | Save a distinguishable instruction while Stop cleanup is pending; after verified stop, inspect exactly one eligible continuation and its useful response. | Keep the message waiting under task/ancestor pause, budget or retained-file holds. Do not impersonate a human message or infer delivery from a queued wake. |
| `recovery/reconnect` | Disconnect the browser, let the server change persisted task/run state, reconnect, then reload and compare. | Inspect duplicate comments/decisions and private/company boundaries after access changes. Browser transport recovery and agent execution recovery require separate results. |
### `source-task`
Automated: [recovery actions](../../server/src/__tests__/issue-recovery-actions.test.ts)
covers source scoping, bounded attempts, operator stops, quota monitoring,
ownership, stale action retirement, and continuation deduplication.
[Recovery cards](../../ui/src/components/IssueRecoveryActionCard.test.tsx) cover
diagnosis, retry progress, exhaustion, and action availability. The fixture-backed
[execution-recovery journeys](../../tests/e2e/execution-recovery/recovery.spec.ts)
start their own test-drive instances and exercise safe, uncertain, restart,
manager-lineage, and legacy-unknown paths:
```sh
pnpm exec vitest run server/src/__tests__/issue-recovery-actions.test.ts ui/src/components/IssueRecoveryActionCard.test.tsx
pnpm exec playwright test -c tests/e2e/execution-recovery/playwright.config.ts recovery.spec.ts
```
Manual: open a stopped task through each affected navigation path. Confirm its
explanation, original owner, recovery owner if different, and next action. Follow
the supported recovery until the source task produces a useful result or a
specific escalation. Reload and verify that resolved notices retire and attempt
history remains. A fixture pass does not qualify every live harness/provider.
The [native restart integration suite](../../server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts)
also covers a prepared remote workspace interrupted before the executor's first
claim. Real database tests assert one same-run bootstrap claim under concurrent
controllers and reject prior claims, launch/provider evidence, and malformed
workspace references. These cases prove classification, not a live remote
provider restart. For that journey, interrupt a disposable deployment after
workspace preparation and before its first executor claim, then confirm one
provider turn starts on the original run through the normal admission gates.
Active remote process supervision distinguishes a failed health read from a
confirmed exit. The [executor tests](../../server/src/services/native-runtime/native-session-executor.test.ts)
exercise timeout recovery with a real listening process, unchanged ownership,
and rejected identity evidence. The Runner restart tests require positive exit
evidence before replacement. This coverage does not prove recovery of a
historical retained sandbox or behavior during a live provider outage.
For completion recovery, restart after a native completion proposal is appended
but before its session checkpoint. Verify that the original result is finalized
after provider termination, with no new provider turn or repeated external action.
The [runtime regressions](../../packages/paperclip-runner/src/native-session-runtime.test.ts)
exercise the real Codex driver with a simulated transport, including a still-active
provider and invalid, conflicting, duplicate, or stale proposals. This does not
qualify a hosted sandbox restart.
The same database suite pauses an original preparation while recovery claims
the remote run. It then verifies that the old preparation yields without opening
a provider or changing the recovery owner, for both live and expired leases.
[Heartbeat boundary tests](../../server/src/services/heartbeat/runtime-execution.test.ts)
verify that the detachment signal transfers recovery ownership before cleanup.
The suite also rejects an older unmarked preparation and verifies that calling
preparation again cannot add the fencing marker from supplied profile data.
For a native workspace owner hold, repeat issue reads and recovery checks after
the task becomes done or cancelled. Verify one unchanged action ID, attempt
count, and timestamps; task completion must not authorize copyback. Clear only
the exact receipt after operator stop verification, then confirm stale action
cleanup. The [route regressions](../../server/src/__tests__/issue-recovery-actions.test.ts)
and [heartbeat regression](../../server/src/__tests__/heartbeat-process-recovery.test.ts)
cover this sequence without a live operator recovery claim.
An intentionally Done onboarding parent can still need its validated
child-completion report. Recovery must not mark that callback successful before
execution solely because the parent is Done. The existing
[stale-lock sweep suite](../../server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts)
covers delayed native/Legacy startup, invalid parent/child/company/assignee
bindings and a dead owner. Process-death and ownership backstops remain active.
Repeat the real onboarding journey and correlate callback turn events with its
reply and downloadable child file; status alone does not prove delivery. The
literal `ea6c8eb5e16d61111968f633a66748b397b64a3e` live failure is retained,
and the revised behavior still needs live qualification.
### `run-detail`
Automated: [run workspace recovery](../../ui/src/components/RunWorkspaceRecoverySurface.test.tsx)
tests this host; it is component coverage, not an end-to-end repair.
Manual: prepare a failed run with error code `workspace_validation_failed`, a
linked source task, and a live `workspace_validation` recovery action on that
task. Open that exact run and compare its recovery diagnosis to the source task.
Use an eligible action from the run surface, return to the task,
and verify the same action state, persisted outcome, and restored execution.
Repeat with an actor who lacks authority. Verify that the run-page controls are
absent for other error codes or after the source action retires; continue other
recovery checks on the source task. Full run-detail navigation remains a manual
gap.
### `operator-retry`
Automated: [disposition notices](../../ui/src/components/DispositionRecoveryNotice.test.tsx)
tests pending/error/gated states and waiting for the real retry acknowledgement.
The recovery-card suite covers workspace divergence, reissue, reconcile, and
confirmation. [Workspace export](../../ui/src/components/WorkspaceExportRecovery.test.tsx)
covers quiet restoration, legacy/operator assistance and resolved states.
For a retained legacy or native file hold, open the source task after its recovery
action has resolved but its files still need recovery. The task shows a compact
operator-attention notice with inspection and collapsed diagnostics, without a
file-certification form. A source-bound live recovery uses the ordinary working
state; a missed or exhausted retry and an unknown action outcome remain visible.
An authorized board operator can use the resolution API to record verified files
and a backup link without changing task status or starting another run. The
server independently validates the original stopped-execution receipt. Agent
Chat retains the hold rather than suggesting another message. The
[notice](../../ui/src/components/ExecutionBlockerNotice.test.tsx) and
[repair route](../../server/src/__tests__/workspace-restore-repair-route.test.ts)
cover quiet live recovery, read-only access, stale or missing stop proof,
source-only repair, and preserved task status. These tests use fixtures; separately
verify original files and their attached artifact before recording a real repair.
Repeat source-only repair with a `succeeded` legacy run that still has a retained
file obligation. Its result must remain successful, its current task owner and
status must stay unchanged, and later saved input must become eligible for normal
admission. A running or queued source and a normal retry of a successful turn
remain denied. The [reconciliation regressions](../../server/src/services/workspace-restore-reconciliation.test.ts)
cover these cases and stop-receipt validation without claiming live provider proof.
For an abandoned native copyback owner, the harness verifies the exact stopped
deployment and the reconciler restores saved work automatically. Task users see
no process details or shutdown form. The documented operator API handles older
receipts after independent platform verification, through a separate administrator
gate. The [authorization regressions](../../server/src/__tests__/authorization-service.test.ts)
and [route regressions](../../server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts)
cover administrator admission, ordinary-member and agent denial, suspended and
foreign membership, missing role data, and stale cloud admin rows.
The [native workspace finalizer suite](../../server/src/services/native-runtime/native-workspace-finalizer.test.ts)
runs the real reconciler with a valid saved yield from `result_accepted`,
`workspace_finalizing`, and `retryable_failure`. It verifies physical fencing
before verified stop, automatic and operator recovery, successful export and commitment, hold resolution,
unchanged saved results, retained approval/monitor waits, and no new provider wake.
These local regressions do not prove a deployment platform stopped a foreign writer.
Manual: reproduce an exhausted disposable action. Click once, verify pending
state prevents repeated clicks, and inspect a successful continuation or an
explicit rejection. Reload to confirm the durable outcome. For a workspace
repair, record the compared branches/revisions and preserve uncommitted work;
confirm the repair's actual result before resuming. Check the denied and stale
action paths. A changed card label is insufficient proof of workspace integrity.
For graceful legacy shutdown, interrupt a disposable sandbox run while file
export is in flight. Delay settlement past the first 30-second wait. Confirm
the lease stays available, no repair hold or successor appears before export,
and the finalizer drain waits for the same owner. After export, confirm saved
files, successful lease cleanup, and one bounded continuation. Reload before
checking the successor. Repeat with controller loss: the expired fence must
retain the exact source and require repair, never claim restored files or start
a second executor. The database-backed
[shutdown regression](../../server/src/__tests__/heartbeat-process-recovery.test.ts)
checks finalizer joining, release order, durable retry uniqueness, and reload.
The [settlement regressions](../../server/src/services/heartbeat/recovery.test.ts)
cover other-controller fencing and owner loss. Provider boundaries are controlled
in these tests; they do not qualify a live sandbox export.
For an abandoned native controller, follow the [stop-verification procedure](../../doc/native-workspace-finalization-recovery.md)
with a disposable deployment. A new platform-bound owner recovers automatically
only after the harness signs proof that the exact deployment fully stopped.
Legacy owners require independently verified stop evidence through the operator
API; the task has no shutdown form. Confirm
the original accepted result commits, the recovery hold resolves, saved files
remain intact, and existing approval waits or monitor checks remain. No new agent
turn should start. Repeat with a retry-phase owner and an ordinary runtime manager;
the first can resume through automatic proof or authorized operator confirmation,
while the second cannot use the operator API.
### `post-stop-message`
Automated: [legacy continuation](../../server/src/services/recovery/legacy-continuation.test.ts),
[queue routes](../../server/src/__tests__/issue-queued-comments-routes.test.ts), and
[native restart recovery](../../server/src/services/native-runtime/native-restart-recovery.test.ts)
cover their respective recovery paths. [Legacy failure continuation](../../tests/e2e/legacy-failure-continuation.spec.ts)
adds a browser journey; it is not native runtime proof.
[Queued Stop database regressions](../../server/src/modules/wake-queue/adapters/queued-operator-stop-postgres.test.ts)
verify that an operator cancellation before runtime selection releases its own
locks, preserves deferred input without launching it, and leaves another active
cycle of the same routine unchanged. Provider cancellations and already selected
or dispatched runs retain their existing release behavior.
Manual: stop a disposable run, submit a distinguishable follow-up, and observe
any cleanup wait. After confirmed stop, verify the follow-up reaches one eligible
continuation and produces the requested answer. Repeat with a task/ancestor
pause and a budget hold: recovery must not bypass the gate. For unknown external
outcomes, inspect the provider before authorizing another operation.
### `browser-reconnect`
Automated: [live-update recovery](../../ui/src/context/LiveUpdatesProvider.recovery.test.tsx)
covers client transport recovery with mocks. It does not prove runner restart.
Manual: go offline during a disposable run, let the server reach a new state,
then reconnect. Confirm task/run/attention data refresh without duplicate
comments or decisions; reload and compare. A healthy websocket proves transport
only. Separately inspect the underlying execution if it is still stuck.
## Gotchas
- Legacy and native runtimes have different ownership and restart paths. Record
which one was exercised instead of inferring coverage from the adapter name.
- A process disappearance, elapsed timeout, or successful API retry does not
prove an external tool operation never happened. Unknown outcomes stay unknown
until corroborated; automatic replay can duplicate a side effect.
- Active-run watchdogs, task watchdogs, reviewers, and recovery owners have
different authority. No recovery card grants board permission by itself.
- Idle Agent Chat is not stranded work. An intentional operator stop must not
silently trigger recovery that fights the user's instruction.
- Source inspection and mocked component tests do not qualify real server
restart continuity. Use the appropriate runtime suite for that claim.
- See [execution semantics](../../doc/execution-semantics.md) for the governing
recovery contract and [steering](./steering.md) for saved follow-ups.
### Storybook verification
#### `exhaustion`
Capability ID: `recovery/exhaustion`.
Scope: ui
Storybook: Exhausted/overdue attempts, operator retry pending/error, blocked authority/gate and specific escalation.
Production: Inspect durable attempt bounds and actual eligible retry/repair; acknowledgment alone does not prove continued execution.
Sources: Source: [IssueRecoveryActionCard.tsx](../../ui/src/components/IssueRecoveryActionCard.tsx), [DispositionRecoveryNotice.tsx](../../ui/src/components/DispositionRecoveryNotice.tsx), [issue-recovery-actions.ts](../../server/src/services/issue-recovery-actions.ts). Tests: [IssueRecoveryActionCard.test.tsx](../../ui/src/components/IssueRecoveryActionCard.test.tsx), [DispositionRecoveryNotice.test.tsx](../../ui/src/components/DispositionRecoveryNotice.test.tsx), [issue-recovery-actions.test.ts](../../server/src/__tests__/issue-recovery-actions.test.ts), [recovery.spec.ts](../../tests/e2e/execution-recovery/recovery.spec.ts)
#### `unknown-outcome`
Capability ID: `recovery/unknown-outcome`.
Scope: ui
Storybook: Uncertain external outcome, required inspection/evidence and explicit permitted operator decision.
Production: Corroborate provider outcome before authorization; enforce no automatic external side-effect replay and reject stale/unauthorized decisions.
Sources: Source: [IssueRecoveryActionCard.tsx](../../ui/src/components/IssueRecoveryActionCard.tsx), [issue-recovery-actions.ts](../../server/src/services/issue-recovery-actions.ts). Tests: [IssueRecoveryActionCard.test.tsx](../../ui/src/components/IssueRecoveryActionCard.test.tsx), [issue-recovery-actions.test.ts](../../server/src/__tests__/issue-recovery-actions.test.ts), [recovery.spec.ts](../../tests/e2e/execution-recovery/recovery.spec.ts)
#### `user-continuation`
Capability ID: `recovery/user-continuation`.
Scope: ui
Storybook: Saved follow-up waiting on stop cleanup, pause/budget gate, one eligible continuation and result/failure.
Production: Confirm previous execution stopped before delivery, preserve message, avoid duplicate wakes and honor actual task/ancestor/budget gate in separate legacy/native paths.
Sources: Source: [TaskChatQueuedMessages.tsx](../../ui/src/components/task-chat/TaskChatQueuedMessages.tsx), [legacy-continuation.ts](../../server/src/services/recovery/legacy-continuation.ts), [native-restart-recovery.ts](../../server/src/services/native-runtime/native-restart-recovery.ts). Tests: [legacy-continuation.test.ts](../../server/src/services/recovery/legacy-continuation.test.ts), [native-restart-recovery.test.ts](../../server/src/services/native-runtime/native-restart-recovery.test.ts), [issue-queued-comments-routes.test.ts](../../server/src/__tests__/issue-queued-comments-routes.test.ts), [legacy-failure-continuation.spec.ts](../../tests/e2e/legacy-failure-continuation.spec.ts)
Reference in new issue View Git Blame Copy Permalink