Commit Graph
9 Commits
Author SHA1 Message Date
Dotta a3e7e63330 Guard Native recovery with process-space proof and explicit containment (#19813)
## Thinking Path

> - Paperclip manages agents and their work.
> - Recovery must preserve execution ownership and accepted results.
> - A saved PID proves nothing outside its recorded process namespace.
> - Uncertain unfinished work must stay held until its original owners
stop.
> - Operators need a source-bound containment action and a separate
ordinary Retry.
> - Accepted results must use the existing finalizer without another
provider turn.
> - This PR keeps those paths safe, recoverable, and quiet between real
obligations.

## Linked Issues or Issue Description

Refs: #19713

Refs: #19591

Recovery could use a local PID to release or replace a controller from
another process namespace. Connected-chat Retry also rejected successful
containment on a task in `todo`. The new containment action used a
broader permission than the existing owner-stop action. A later review
found that result-less guards also rejected the existing accepted-result
confirmation action.

## What Changed

- Record and verify the controller's kernel/PID namespace. Keep unknown
and foreign unfinished ownership held. Preserve independent durable stop
evidence and the existing remote lease fences.
- Use the shared accepted-result ownership reconciler. Bind settlement
to the observed lease token, generation, boot, PID, namespace and
process birth. Retain finalizer, cleanup, current-controller, live-owner
and remote exclusions.
- Share the remote-owner reader across recovery consumers. Scope runner
leases to the same company, task and run. Do not make local PID probes
for remote accepted-result ownership.
- Gate original-host containment on `runtime:confirm_owner_stop`. Bind
it to the current task, source, assignee, original expiry, lease and
Stop state. Audit separate controller/provider containment and reviewed
action outcomes.
- Keep containment separate from ordinary Retry. Use one canonical
committed receipt and the normal admission, budget, workspace and
lineage checks. Preserve the recorded external-action outcomes.
- Keep the controller-only confirmation actionable for both result-less
work and accepted results. Result-less work gets its next source-bound
attention in the same transaction. Accepted results keep their original
result and use normal shared settlement/finalization.
- Keep identity inspection quiet. Preserve the original foreign-owner
expiry and use the existing scoped finalizer in accepted-result recovery
controls.
- Update the recovery card, validators, execution projection, docs and
access maps. Reuse existing canonical columns; add no migration or
ownership backfill.

## Verification

Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`.

**The current fix has a causal red/green check.** On the test-only
revision, both foreign-local stopped/fenced controller cases failed
because recovery probed saved runner, process-group and provider PIDs in
this host's namespace. This measured the wrong probes; it did not run
duplicate providers. After the fix, the same two cases passed every
no-probe, no-claim, no-controller-stamp, no-stop-receipt and
unchanged-authority assertion. Three existing remote preparation, grace
and fenced-controller progress cases also passed. Total: **5 passed, 0
failed;170 filtered rows excluded**. All three normal affected App
TypeScript programs passed on this head (660,681 and428 roots).

The fix separates a controller wait from execution stop proof.
Independent live/grace evidence still schedules recovery. A
stopped/fenced foreign controller can pass this namespace guard only for
remote execution, which keeps its later checkpoint and never-launched
checks. Foreign local execution stays held before runner or provider PID
inspection.

Earlier source-scoped results remain evidence, not fresh executions on
the current head:

- On `c86dd741`, the complete failed-owner file passed **25/25**, with
no skips or filters. Empty scans wrote no controller stamps. Foreign
ownership kept its original expiry. Normal returned-claim settlement and
attention at the exact deadline passed.
- On `c86dd741`, **15 selected accepted-result/restart controls
passed**. The81 filtered rows are excluded. Both completion cases used
the existing scoped finalizer and kept their accepted result without
another provider turn.
- Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334`
plus `c86dd741`) passed the complete **17-case maintenance-server
file**. This includes the original90-real-second quiet observation and
two qualified hourly windows. The quiet case took118,918ms. The four
jobs that failed on the earlier publication all passed on that synthetic
merge.
- On `6243b32c`, the complete ownership-report file passed **29/29**.
Native NULL namespace stayed unprovable and generic compatibility
remained. Three selected recovery controls passed;170 filtered rows are
excluded.
- On `f81f2c3c`,356 business cases passed: whole ownership-route221,
recoveryUI70, owner-proof38 and controller-confirmation8; selected
accepted-result authority18; and one real connected-chat containment,
separate authorized Retry, useful Native result and downstream reply
case. The2,024 filtered rows are excluded. A separate earlier
accepted-result causal case kept the original result and completed after
restart without replay.

Normal Shared exports, App programs, UI typecheck, token gates and map
checks passed on their recorded revisions. Unchanged DB, adapter, Eval
and Runner inputs/outputs have explicit carry proofs. The clean normal
merge with fetched master `8fe524f7` preserves all31 exclusive owned
paths and14,827 unrelated master entries. The nine overlapping paths
were reviewed, including the upstream unidentified-run stop action.
Fresh CI must validate that actual merge composition.

PG cases used existing external PG17, fresh owned databases, pools at
most4, one worker and no SQL observer during cases. Registered clients
and recorded process groups closed. Independent final SQL audits found
zero campaign sessions before normal database cleanup. Original failing
fixtures remain retained.

Original failures are preserved. One new test first failed compilation
due to a missing getter import; an import-only correction passed its
full owning programs before the causal red/green check. A local quiet
attempt stopped in the normal prerequisite hook before any business case
and its collector rejected two normal build markers. The later17-case CI
pass establishes quiet recovery; it does not turn that failed local
collector into a pass. An earlier useful case passed before its private
cleanup collector failed on an owned read-only directory; normal cleanup
completed after all owners closed. Earlier runner-shutdown cancellation
and a separate private whole-suite failure remain unexplained.

Current-head checks are complete:57 success,1 neutral and3 skipped,
including the successful aggregate CI gate. The legacy status passed.
Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this
head. All seven review threads are resolved. Full recursive typecheck,
the full test suite and release build were not repeated locally. No
result here proves deployed suspension, a live provider or termination
on another host.

## Risks

- Unknown physical ownership can require an owner/admin action. A false
containment statement can be unsafe.
- New work, reassignment, Stop, renewed authority, accepted results or
pending cleanup can invalidate an action. The operator must refresh.
- Controller confirmation does not prove provider/tool containment.
Completed external actions must not be replayed.
- Generic legacy and child-process compatibility remains separate from
the strict Native controller policy.
- Existing saved state and canonical schema are preserved. Permanent
safety refusals remain held with their normal named recovery path.

## Model Used

OpenAI `gpt-6.1-sol` agents implemented and tested the changes with
reasoning, repository tools and code execution. A GPT-6-based Codex
managing agent reviewed and integrated them. The host does not expose
the manager's exact build or the context-window sizes.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-09 12:06:11 -06:00
DottaandClaude Opus 5.5 4f7eee10c2 feat(recovery): let the board confirm an unrecorded owner stopped for every settling process kind (#20084)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Recovery releases in-progress records (leases, claims, intents) only
when it can prove their owner is gone
> - A record saved before its server process was fully recorded is
`unprovable`, and it is never released on its own
> - #20067 added a board-only action to confirm that such an owner
stopped, but only the environment run lease used it
> - Other settling kinds (for example the legacy controller lease, which
shows `missing_identity` rows in the Phase 2 reconciler data) stay stuck
with no way forward
> - This pull request opens the same confirmation to every settling kind
whose record names a partly recorded server process
> - The benefit is that an operator can release these stuck records
safely, and recovery settles them at once

## Linked Issues or Issue Description

Refs #20067, #20054

**What happened?**
The Phase 2 reconciler data shows `legacy_controller_lease` records that
stay `unprovable` with `missing_identity`: they were saved before
Paperclip stored the controller's PID space. Recovery never releases an
unprovable owner, and only the environment run lease offered the board's
"confirm the previous server stopped" action. These records, and records
of other settling kinds saved the same way, had no path forward.

Also, a board confirmation did not wake the reconciler. Unprovable
records are not watched, so the release waited for some unrelated pass.

**Expected behavior**
Recovery asks the board on the record's task. After the board confirms,
recovery releases the record through the kind's own compare-and-swap
settle, at once. A confirmation never releases a record that changed
since the board was asked.

**Steps to reproduce**
1. Have a legacy run whose `controller_process` was recorded without a
host hash (an older server).
2. Run the ownership reconciliation pass. The record is `unprovable`
(`missing_identity`) and no board action appears.

**Paperclip version or commit**
master at c55af26821.

## What Changed

- `ownership-registry.ts`: a new `ownerGoneAttestable` helper opts a
settling kind in. The kind asks the board (cause
`ownership_owner_unprovable`) only for a process-owned record that is
unprovable for a legacy reason and has a version. Opted in:
`legacy_controller_lease`, `environment_teardown_claim`,
`sandbox_creation_intent`, `workspace_runtime_control_operation`,
`native_workspace_finalize_attempt`, `native_finalizer_lease`,
`native_cleanup_lease`.
- `owner-gone-attestation.ts`:
- The request records the record version (`recordUpdatedAt`), and each
pass refreshes an open request to the current version.
- The board submits the version it was shown (`recordVersion`, a new
optional field in `confirmOwnerGoneSchema`). The confirmation is refused
with 409 unless that equals the request's current version and the locked
record's version.
- `recordVersionAttestations` reads back every resolved confirmation.
Any one that names the record's current version applies.
- `confirmOwnerGone` now wakes the ownership reconciler in its
transaction.
- Each opted-in kind has a version lock. For an `operator_attested`
proof, its settle runs in a transaction that locks the record row and
re-checks the confirmed version before the kind's own compare-and-swap.
- `ownership-invariant-report.ts`: in the periodic pass and in one-shot
settles, a matching confirmation turns a legacy `unprovable` verdict
into `gone` (`operator_attested`). A live or provable owner always wins.
The extra read runs only when a kind has such a record.
- `WorkspaceExportRecovery.tsx`: the board's confirmation names the
stuck record by kind (for example "This task's earlier run"), with a
generic fallback, and sends the version it showed.
- `doc/execution-semantics.md`:
- It describes the action, the version binding and which kinds are
offered.
- It updates the controller-lease section to match #20054: the holdings
list, and that the lease is off only on Railway deployments with an
exclusive volume.

## Verification

- `pnpm exec vitest run src/__tests__/owner-gone-attestation.test.ts`
(server): 10/10. The new legacy-controller cases fail without the
change:
  - the board is asked, and nothing is released before the confirmation;
- the confirmation wakes the reconciler, and the next pass settles the
record as `operator_attested`;
- a record that changed after the confirmation is not released, and a
fresh request is raised;
- a record of these kinds with no owner stamp is not offered, because
its settle could never release it;
- a confirmation for a version the record no longer has is refused, and
the refreshed request then applies;
- a write between the reconciler's listing and its settle stops the
release inside the settle's transaction.
- The 10 server ownership suites (registry, report, owner proof,
controller lease, legacy controller, teardown, finalize, token leases):
332/332.
- `pnpm exec vitest run
src/components/WorkspaceExportRecovery.control.test.tsx` (ui): 48/48.
- `pnpm check:token-gates`: all gates clean.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` and
`ui`: passed.

## Risks

- A wrong release would be serious. The safeguards:
- The confirmation applies only while the record's last-write time
equals the one the board was asked about, and only while the prover
still finds the recorded process unprovable for a legacy reason.
- Each kind's settle still compare-and-swaps on the exact recorded
process.
- A record of these kinds with no owner stamp at all is not offered.
Those settles cannot release it, so it keeps its existing recovery path.
- Kinds that are unowned by design (queued pending cleanup, accounting
debt) are not offered.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 12:16:07 -06:00
DottaandClaude Opus 5.5 0746d1203d feat(recovery): let the board confirm that an unrecorded owner stopped (#20067)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, recovery releases an in-progress record only once
its owner is proven gone, never by age
> - A run lease saved before owners were stamped has no owner Paperclip
can judge on an ephemeral (multi-host) server, so it stays stuck and
holds its run's sandbox forever
> - Only a person can tell that the previous server stopped (for
example, its container was replaced)
> - This pull request asks the board on the stuck task and lets an owner
confirm the stop, with an activity-log entry; recovery then releases the
record on its normal path
> - The benefit is that legacy stuck records have a way out that never
guesses

## Linked Issues or Issue Description

Follow-up to the Phase 2 durable-ownership work: #20054 (controller
instance lease) and #19883 (process-space recording). It generalizes the
existing "confirm the previous server stopped" action for native
controllers (`native_controller_host_unverified`). No duplicate PR was
found.

**What happened?**

On an ephemeral server, an active run lease of a finished run that was
saved before holders were stamped has no owner stamp. The ownership
reconciler reports it `unprovable (no_owner_stamp)` and never settles
it. No operator action exists, so the lease and its sandbox stay
reserved.

**Expected behavior**

The task shows the stuck resource, and a board owner can confirm that
the previous server stopped. Recovery then releases the lease. Agents
and operators without the owner-stop permission cannot.

**Steps to reproduce**

1. Run with `PAPERCLIP_EPHEMERAL_SERVER=true`.
2. Have an active environment lease with no `leaseController` stamp,
whose run is terminal.
3. Run an ownership reconciliation pass. The lease stays `unprovable`
with no action on the task.

**Paperclip version or commit**

`master` before this PR.

## What Changed

- `packages/shared`: `confirmOwnerGoneSchema` (`actionId`, `recordId`,
`stopEvidence` of 20 to 12000 characters).
- `recovery/owner-gone-attestation.ts`:
- `requestOwnerGoneConfirmation(kind)` is an `onUnprovable` hook. For a
legacy unprovable owner (`no_owner_stamp`, `process_space_unrecorded`,
`missing_identity`) on a record that has a task, it opens one idempotent
board recovery action on that task (cause `ownership_owner_unprovable`).
- `confirmOwnerGone(...)` verifies the exact action and record. It asks
the kind to write the attestation (compare-and-swap), resolves the
action, and writes an `issue.ownership_owner_gone_confirmed`
activity-log entry with the actor and the stated evidence.
- Ownership registry:
  - `OwnableKind.attestOwnerGone` is a new optional hook.
- `environment_run_lease_active` implements it: only an unstamped active
lease of a terminal run, with the notification for the cleanup path.
- The lease's owner then reads `operator_attested`, which proves gone,
so the existing settle releases it. A holder stamped since the board was
asked owns the lease, and its stamp decides.
- Route: `POST /api/issues/<built-in function
id>/recovery-actions/confirm-owner-gone`. It uses `assertBoard` plus
`runtime:confirm_owner_stop`, the same board-only permission as
confirm-controller-stopped. It is registered in OpenAPI.
- UI: `issuesApi.confirmOwnerGone`. The task's recovery card shows the
existing `PreviousServerStopConfirmation` for this cause, with
resource-specific wording; owners confirm and operators see the wait.
`isPreviousServerStopCause` includes the cause, so the generic card does
not offer other resolutions.

## Verification

- `owner-gone-attestation.test.ts`:
  - the stuck lease opens one board action on the task;
- confirmation resolves the action, writes the activity log, and the
next pass settles the lease (`operator_attested`) exactly once;
  - a mismatched record is refused with 409;
- a holder stamped after the ask is refused and the lease stays active;
  - no action for a judgeable owner or for a record with no task.
- Red, with the `onUnprovable` hook removed: 2 of 3 fail.
- UI: `WorkspaceExportRecovery.control.test.tsx` checks that the owner
confirms with the right payload and that an operator sees the wait
without the control (47/47).
- Server suites: openapi-routes, ownership and recovery suites, plus the
existing native-controller-host recovery. Low-memory server typecheck is
clean; `check:token-gates` is clean.

## Risks

- The board can attest a server stopped that is in fact running. The
action and the UI copy ask for the evidence, and the activity log
records who attested what. This is the same trust model as
confirm-controller-stopped.
- Only run leases opt in so far. Records without a task (login/probe
leases) keep the legacy idle sweep.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 09:46:36 -06:00
DottaandClaude Opus 5.5 f439150cb3 fix(recovery): limit the null-host owner fallback to single-host servers and record each use (#19883)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #19868 and #19871 made restart recovery judge a recorded pid only in
the PID space that recorded it
> - Rows saved before those host columns existed have no recorded host.
They kept the previous local dead-pid proof, so in-flight native runs
would not be stranded at upgrade
> - On an ephemeral (multi-host) server that fallback is unsafe: the row
may have been written on another host
> - This pull request limits the fallback to single-host servers,
records every use, and gives ephemeral servers a working board action
> - The benefit is that single-host upgrades still recover on their own,
multi-host servers never accept a cross-host proof, and the next release
can remove the fallback with evidence

## Linked Issues or Issue Description

**Bug description**
After #19868 and #19871, a `native_run_finalizations` row with
`controller_host_hash = null`, or a run with `process_host_hash = null`,
still accepted "the pid is dead here" as proof on every server. That
includes an ephemeral server, where the row may come from another host.
Nothing recorded how often the fallback was used, so it had no safe
removal path.

**Expected behavior**
- Single-host server: the fallback applies for one release, and each use
writes a `legacy_host_unknown_dead_pid_proof` activity-log entry.
- Ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`): the row is
unprovable, and the board has an action to confirm the previous server
stopped.
- A mismatched host is always unprovable.

Refs #19868, #19871, #19846

## What Changed

- `native-restart-recovery.ts` `evaluateNativeControllerTakeover`:
- A coordinator row with `controllerHostHash: null` returns
`controller_host_unrecorded` when `ephemeralServerEnabled()` is true.
- Otherwise a pid proof from that row carries `legacyHostFallback:
true`.
- `operator_attested` is accepted as proof
(`operator_attested_controller_stopped`).
  - Receipt-built owners, which have no host key, are unchanged.
- New `recordLegacyHostFallbackProof` writes the activity-log entry. It
is not a run-log event: run events are progress evidence for restart
recovery's crash-loop credit, and the record must not change any
recovery decision. It is called from restart recovery, the
dead-controller result-lease release, native retirement, and remote
warm-session recovery.
- New `native-controller-host-recovery.ts`:
- `requestNativeControllerStopConfirmation` (board-owned
`native_controller_host_unverified` recovery action, idempotent, never
displaces another active action).
- `confirmNativeControllerStopped`. It verifies the exact recorded
controller and stamps `operator_attested` with compare-and-set. It then
resolves the action, wakes live-run recovery, and logs
`issue.native_controller_stop_confirmed`.
- Route `POST /issues/:id/recovery-actions/confirm-controller-stopped`.
It is board-only and requires `runtime:confirm_owner_stop`. Generic
`restored`/`false_positive` resolution of this cause is refused. A
shared `confirmNativeControllerStoppedSchema` validates the body.
- UI: `WorkspaceExportRecovery` reuses the "Confirm the previous server
stopped" control for this cause, and the generic recovery card hides it.
Adds the `issuesApi.confirmControllerStopped` client call.
- `recovery/service.ts` orphan reaper: a null `process_host_hash` is
judged only when not ephemeral. A successful use records the entry
best-effort, after the terminal write. On an ephemeral server the run is
left to its existing Stop control.
- `ephemeralServerEnabled` moves to a leaf `ephemeral-server-mode.ts`
(re-exported by `ephemeral-server-contract.ts`). Importing the contract
module from recovery code created an import cycle in the runner child
process.
- The new route is registered in OpenAPI.
- `doc/execution-semantics.md`: documents the host rule and the
one-release sunset.

## Verification

- `native-restart-recovery.test.ts`: null host on a single-host server
is released with the flag. Null host on an ephemeral server is
unprovable. A mismatched host is unprovable in both modes. An attested
controller is accepted. Receipt owners are unchanged.
- New `native-controller-host-recovery.test.ts` (embedded Postgres):
  - The fallback writes the activity-log entry and no run event.
  - Ephemeral mode creates exactly one board action.
  - A wrong run id is refused.
- Confirmation stamps the attestation, after which takeover is allowed.
The action resolves and the activity is logged. A second confirmation is
refused.
- `recovery-stale-issue-lock-sweep.test.ts`: on a single-host server a
null host is terminalized and records one event. On an ephemeral server
it is left alone with no event. The existing audit-failure case still
clears the lock.
- `WorkspaceExportRecovery.control.test.tsx`: the controller
confirmation posts to `confirmControllerStopped` and never to the
copyback endpoint.
- Locally: these suites plus `remote-warm-session-recovery`,
`native-controller-recovery.integration` and
`native-startup-cleanup-recovery` (136 passed), and after the CI fixes
`native-runner-restart-preparation` and `native-runner-restart-recovery`
integration, `openapi-routes` and the suites above (256 passed); the UI
recovery suites (52 passed); `node scripts/typecheck-layers.mjs` with no
errors in the touched files; `pnpm check:token-gates` clean.

## Risks

- Previous-release story: no schema change. Null-host rows saved by
earlier releases recover as before on single-host servers, now with an
event, and wait for the board on ephemeral servers. The next release
removes the fallback; the event count shows whether any rows remain.
- On an ephemeral server, a null-host orphaned run process is not reaped
automatically. The operator uses Stop. That is the intended "unprovable"
outcome.
- A board confirmation is an attestation. It applies only to the exact
recorded controller (boot id + pid) and only while the host is still
unrecorded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and
code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:25:47 -06:00
DottaandClaude Opus 5.5 824b25654c fix(ui): keep a Retry for retained-file holds whose source cannot be verified (#19840)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a native run finishes but its files cannot be saved, the task
stays blocked behind a retained-file recovery hold.
> - The server records that hold as blocked and retryable, and the task
notice tells the user to "Retry to try again".
> - The task page hides its own Retry for these holds and relies on the
retained-file recovery panel instead.
> - That panel only binds the recovery record when a read-only source
inspection returns `verified`.
> - Inspection can return `blocked` for normal reasons: a sandbox that
predates ownership receipts (state saved by an older release), or a
workspace that moved on after its files were recovered.
> - Then neither surface offers an action, and "Check file recovery
status" always answers "File recovery changed. Refresh the task", also
after a reload.
> - This pull request binds the record by its own identity, uses
inspection only to gate the physical file-recovery request, and keeps
the task Retry visible whenever the panel has no control of its own.

## Linked Issues or Issue Description

Refs #18848 (the change that added the retained-file recovery panel and
hid the generic Retry for these holds).

**What happened?**

A blocked task with an active
`native_workspace_sync_out_retry_exhausted` hold showed "The saved files
need recovery before work can continue", "The recorded source is
unavailable or has changed", and "Saving the agent's files failed after
automatic retries. Retry to try again". It showed no Retry and no
recover button. "Check file recovery status" always answered "File
recovery changed. Refresh the task to inspect the current recovery", and
a reload did not change it. The hold was saved by an earlier release:
`systemRecovery` exhausted with `allocation_ownership_ambiguous` /
`service_required`, and `workspaceRecovery` blocked with `retryable:
true`. The source inspection endpoint returned `status: "blocked"` for
it. A second task whose files were already recovered showed the same
"need recovery" and "has changed" text, because its inspection reported
a reference mismatch after recovery.

**Expected behavior**

A retryable hold always shows a working Retry or recover control. A
notice that says "Retry" has a Retry button. Status checks never loop on
"changed, refresh". Recovered files show as recovered. When the source
cannot be verified, the page names the reason and who must inspect it.

**Steps to reproduce**

1. Use an active retained-export hold whose `workspaceRecovery` is
blocked and retryable, and whose `systemRecovery` is exhausted.
2. Make the source inspection (`GET
/api/issues/:id/recovery-actions?inspectSourceRunId=…&inspectSourceActionId=…`)
return `status: "blocked"`.
3. Open the task as a board operator.
4. Before this change, the page shows no Retry and no recover button.
"Check file recovery status" reports "File recovery changed. Refresh the
task".

**Paperclip version or commit**

Observed on build `8812c19`. The regressions fail on master `0f5ce5050e`
before the production edits.

## What Changed

- `WorkspaceExportRecovery`: bind the recovery record by action, run and
schema identity, not by inspection status. Inspection status now gates
only the physical "Recover saved files" request (and the request
re-checks it before POST).
- `WorkspaceExportRecovery`: report through `onRecoveryControlChange`
whether the panel offers its own control. It reports `true` only while
the source is loading or verified and the files are not yet recovered,
or while it shows the previous-server stop confirmation.
- `IssueDetail`: hide the task Retry for retained-export holds only
while the panel reports that it owns the control. Otherwise the existing
task Retry (`POST /issues/:id/retry`) shows. The server renews a bounded
recovery generation and re-verifies the source.
- Show recovered files as recovered even when inspection no longer
matches after recovery.
- Name an unverifiable source in plain words (with the code for other
reasons). State that Paperclip does not apply files it can't verify.
- Recovery text never asks a person to inspect a sandbox or environment,
and never names an owner or administrator as the one who must act. This
covers the retained-file panel, the run source inspector, the copyback
wait, and the native cleanup quarantine message. The execution blocker
and the recovery action read model describe a spent recovery budget from
its current state, so text saved by older releases ("If Retry keeps
failing, inspect the run and its environment") is not shown.
- Task Retry uses the existing `createUuid()` helper, so it works on
plain-HTTP previews without `crypto.randomUUID`. An accepted Retry also
refreshes the inspected file-recovery state. After a failed accepted
request, the unverified-source reason stays visible; it hides only while
recovery runs.
- When the recovery record was replaced, "Check file recovery status"
reloads the task detail instead of telling the user to refresh.
- Tests: component regressions for an old-release hold, a recovered hold
with a mismatched source, verified ownership, and a replaced record. A
page regression proves Retry is shown and calls `retryTask` for
unverified and stale sources. A server safety test proves that Retry for
the old-release hold shape is accepted, opens one new bounded
generation, queues file recovery again, keeps the hold active, and is
idempotent.

## Verification

- RED (master `0f5ce5050e` production files, new tests): 5 component
tests fail. The failure output shows the exact staging text: "Check file
recovery status", "File recovery changed. Refresh the task to inspect
the current recovery.", "The recorded source is unavailable or has
changed." 2 page tests (unverified and stale) fail with `expected [
'Check file recovery status', … ] to include 'Retry'`.
- GREEN: `cd ui && npx vitest run
src/components/WorkspaceExportRecovery.control.test.tsx
src/components/WorkspaceExportRecovery.test.tsx
src/components/RunWorkspaceRecoverySurface.test.tsx
src/components/ExecutionBlockerNotice.test.tsx
src/pages/IssueDetail.test.tsx` passes 320 tests.
- Server: `cd server && npx vitest run src/services/task-retry.test.ts`
passes 81 tests, including the new old-release safety case. This is
safety evidence, not a red: the server path was already correct, and the
defect was in the UI.
- Full UI suite: 11564 pass. 4 tests in `IssuesList.test.tsx` and
`CompanyEnvironments.test.tsx` failed under full-suite load and pass in
isolation (106/106). They are unrelated to this change.
- `pnpm check:token-gates` is clean. `node scripts/run-tsc.mjs -p
ui/tsconfig.json --noEmit` reports no errors in the touched files. The
remaining errors are local artifacts that were not built (plugin SDK UI
entry, generated Storybook catalog).
- Staging was read only (GET requests). The inspect request did not loop
in a fresh tab: two calls in 30 seconds. Most of the tab's request
volume is periodic company issue-list refetches, which this change does
not touch.

## Risks

- Low risk. This is a UI-only behavior change. Server authorization,
source verification, stop receipts, and the bounded recovery policy are
unchanged and remain authoritative.
- The task Retry is now visible for retained-export holds whose source
does not verify. Retry renews one recovery generation. The worker
re-runs the same source-identity checks, so Retry cannot bypass an
ownership refusal. If the refusal is permanent (for example an ambiguous
sandbox allocation), the hold returns to exhausted and the page names
the reason. An automatic fallback for such sources is follow-up work.
- Wording: a parametrized UI test renders every recovery state and
asserts that no administrator, owner, or "inspect the environment" copy
appears. A server test proves that a stored old-release message is
replaced by state-derived text. Both failed before the change.
- Previous-release story: the old-release hold shape (board-owned,
`systemRecovery` revision 5 exhausted with `service_required`,
`workspaceRecovery` blocked and retryable) is covered by component,
page, and server tests. No persisted state shape changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (repository search, file edits, shell, test
execution, and a read-only browser).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 10:24:03 -06:00
DottaandClaude Opus 5.5 4dc5281f51 fix(ui): offer the owner-stop confirmation for retained saved-work recovery (#19742)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run saves the agent's files after its turn (copyback). One
controller owns copyback and records an owner receipt on the run
> - #19721 made retained saved-work recovery stop spending attempts when
a restart killed that owner. It records the receipt as
`workspaceOwnerRecovery.state = "operator_required"`, and a waiting
Retry reports `workspace_owner_stop_required`
> - #19717 added one task control, **Confirm the previous server
stopped**, for the direct finalization hold. That control did not cover
the retained hold, so the task told the board to confirm but offered
nothing to click
> - This pull request shows the same control for the retained hold, with
the recorded receipt token
> - The benefit is that every copyback-owner hold has exactly one
visible action, and the waiting Retry then continues by itself

## Linked Issues or Issue Description

Refs #19721 (retained hold, server side), #19717 (the confirmation
control).

**What happened?**
The retained recovery action named the stopped copyback owner, and the
task's Retry said "Retry is waiting for a board member to confirm that
the previous server stopped copying files". No control called the
confirmation route for this state.

**Steps to reproduce**
1. Retained saved-work recovery owns copyback for a finished run.
2. A restart kills the controller while it owns copyback, on another
host or without automatic stop proof.
3. Open the task.

**Expected behavior**
The task shows one **Confirm the previous server stopped** action to an
owner or administrator. It sends the recorded receipt token to
`resume-workspace-finalization`.

**Actual behavior**
The task shows only the retained file-recovery controls. The token is in
`evidence.workspaceOwnerRecovery.owner`, which the #19717 control did
not read.

## What Changed

- `WorkspaceExportRecovery`: when a retained export action has
`workspaceOwnerRecovery.state = "operator_required"` and no recorded
confirmation, render the existing confirmation control with that
receipt's token, in place of the retained file-recovery controls. After
the confirmation (state `resuming`), the ordinary retained controls
return.
- The confirmation control now takes the owner token as a prop, so the
direct hold (`evidence.owner`) and the retained hold
(`evidence.workspaceOwnerRecovery.owner`) share it.
- Docs: one sentence in `doc/native-workspace-finalization-recovery.md`.
- There are no server changes. The route already dispatches the retained
path (#19721).

## Verification

- Red then green: the new test "offers the same action for retained
saved-work recovery held by a stopped copyback owner" fails against
master's component and passes with this change. It checks that the
request body carries the retained receipt token, that the host name is
not shown, and that the retained controls return after `resuming`.
- `cd ui && npx vitest run src/components/WorkspaceExportRecovery` (35
passed).
- `pnpm check:token-gates` clean. UI typecheck clean.

## Risks

- Low. The change is UI only. The server rechecks the receipt, the hold
state and `runtime:confirm_owner_stop`.
- While the hold is active, the "Recover saved files" control is hidden.
It cannot release the receipt, so hiding it removes a futile action.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 23:48:43 -06:00
DottaandClaude Opus 5.5 464e6e16e0 fix(native): resume copyback after a restart kills its same-host owner; one board action otherwise (#19717)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save the agent's workspace files after the provider turn
finishes (copyback)
> - One controller owns copyback at a time. The run records that owner
in `runnerProfileJson.nativeWorkspaceFinalizationOwner` (host, PID,
token)
> - A control-plane restart can kill the controller during copyback. The
dead controller can never confirm its own stop
> - Recovery then waited for a board confirmation. No surface offered
that confirmation, so the run stayed `running` and the task stayed
`in_progress` with no visible action
> - This pull request lets the next controller on the same host prove
the stop by itself. It also gives the board one clear action when that
proof is not possible
> - The benefit is that a restart during copyback resumes by itself, and
a task never waits silently for a person

## Linked Issues or Issue Description

**What happened?**
In a Daytona restart-chaos wave, a kill during workspace copyback left
the run `running` and its task `in_progress`. Recovery reported
`native_workspace_finalization_owner_unverified`. The run was still
`running` when the wave timed out.

**Steps to reproduce**
1. Start a native remote (sandbox) run.
2. Kill the control-plane process (SIGKILL) after the provider result is
accepted, during copyback.
3. Restart the control plane on the same host.

**Expected behavior**
If the previous controller and its copyback children are proven stopped
on this host, recovery resumes copyback without a new agent turn.
Otherwise, the task shows one board action to confirm that the previous
server stopped.

**Actual behavior**
The recorded owner was cleared only by
`resumeNativeWorkspaceAfterOwnerStop`. Same-host recovery also required
a completed-copyback barrier, which an interrupted copyback never has.
The task presentation showed `working`, and the UI had no control for
the confirmation.

Refs #19578 (boot-ID fencing for unstarted runs; same proof style).

## What Changed

- Root cause:
`server/src/services/native-runtime/native-workspace-finalization-ownership.ts`
sent a dead same-host owner to `requireStopVerification` when no
`workspace_finalize` operation had succeeded. That case is exactly a
restart during copyback.
- New `native-workspace-owner-stop-proof.ts`. The owner receipt now
records the controller's process group (`processGroupId`). Copyback
children (tar, Git) are not detached, so they keep that group. A
successor on the same host releases the receipt only when the receipt
names another controller boot, the PID is dead or recycled (process
start time), and the group is empty or has no member outside the
successor's own process tree that started after the old controller. Age
alone never proves a stop.
- A still-live group member returns `local_stop_pending`. The
finalization worker rechecks it on its existing ownership recheck
interval. The board action stays visible meanwhile.
- A successful local proof writes `issue.workspace_owner_stop_verified`
with the proof and marks the recovery `resuming`, like the existing
platform proof.
- A receipt from another host, a receipt without a process group, or a
host without `ps` keeps the fence. The task interruption is now
`waiting_for_user` with `userAction: "confirm_owner_stop"` (new shared
enum value), not `working`.
- UI: the task chat shows one **Confirm the previous server stopped**
action. It asks for stop evidence and calls the existing
`resume-workspace-finalization` API. It is shown only to roles that hold
`runtime:confirm_owner_stop` (company owner or administrator, local
board, instance administrator). The inbox names the same action.
- The copyback advisory lock, the coordinator finalization lease, and
remote save fencing are unchanged.
- Docs: `doc/native-workspace-finalization-recovery.md`.

## Verification

- Red then green on embedded Postgres
(`server/src/services/native-runtime/native-workspace-finalizer.test.ts`):
- same-host controller killed with SIGKILL, its process group empty:
before `owner_unverified`, after `acquired` with the proof logged and no
agent wake
- a non-detached orphan child of the dead controller still runs:
`local_stop_pending`, no copyback; after the child exits, `acquired`
  - a receipt without a process group: still `owner_unverified`
- a foreign-host owner: the task presentation was `working` (red), now
`waiting_for_user` / `confirm_owner_stop`; after the board confirmation
it is `working` again
- The existing orphan test (a detached child) and all other finalizer
tests pass: `npx vitest run
src/services/native-runtime/native-workspace-finalizer.test.ts
src/__tests__/native-workspace-finalization-work.test.ts
src/services/task-interruption-presentation.test.ts` (135 passed).
- UI: `npx vitest run src/components/WorkspaceExportRecovery` (34
passed), including the action call body and the non-administrator case.
- `pnpm check:token-gates` clean. Server and UI typecheck clean for the
changed files.
- Live Daytona restart-chaos wave: results are added in a comment below.

## Risks

- A process-table check that reports a member wrongly as live only keeps
the existing fence. The board action is still available.
- A subreaper that adopts orphans does not change their process group,
so the proof does not depend on the parent PID.
- A child that a future change spawns with `detached: true` would leave
the group. Copyback paths do not do that today. The existing
detached-orphan test still fails closed, because a receipt without a
process group is not released.
- Old receipts have no process group. They keep the board action.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 23:31:07 -06:00
a3313f6944 fix(ui): complete read recovery and enforce query error rendering (#18822)
## Thinking Path

> - Paperclip lets people manage agent work through the board.
> - The board reads task, agent, settings, plugin, and run data from the
server.
> - A deploy can interrupt those reads after content is already visible.
> - Local error branches can replace that content with an error or show
raw codes.
> - This pull request moves the remaining reads to the shared recovery
behavior.
> - The query-error guard prevents new local error patterns from
returning.

## Linked Issues or Issue Description

**What happened?**
Loaded lists, settings, task panels, and manual reads could clear their
content or show errors during a server restart. Some reads also disabled
the shared retry policy.

**Expected behavior**
Keep loaded content during a temporary failure. Show the shared
connection banner. Retry reads after recovery. Replace content after a
real access denial or missing resource, with readable copy.

**Steps to reproduce**
Open a task, settings page, or run log. Return 503 with
`tenant_app_unavailable` from the API. Restore the API and check that
content refreshes. Repeat with 403 and 404 to check the failure state.

Related merged work: Refs #15754, Refs #15772, Refs #17960. This change
covers remaining reads and enforcement. Browser persistence and the
outage browser suite follow separately.

## What Changed

- Repair an existing PostgreSQL deadline fixture at UTC boundaries. A
daily cap defers a heartbeat until both its original interval and the
next UTC day allow it. Keep the counted run in its seeded day and cover
noon, just before midnight and the first second after midnight with
explicit clocks and real PostgreSQL. Production scheduling is unchanged.

- Integrate the saved list, settings, shell, task, and manual-read
recovery changes against current master. Preserve the current task,
Inbox, and run-log structure.
- Add `useResilientFetch` for component-local reads. Preserve loaded
data, cancel stale requests, and retry through the shared connectivity
policy. Keep browser and pairing capabilities in memory.
- Retain run log chunks across stream changes and catch up after
recovery without duplicate chunks or stalled requests.
- Keep genuine authorization and missing-resource failures visible.
Translate known single-word codes such as `forbidden`. Show known skills
and disconnected-app failures before sibling loading states. Keep Retry
bound to its failed request.
- Make the query-error guard enforce by default and run it in the CI
policy job. Document the narrow exceptions and the read recovery
contract.
- Invalidate pending OAuth discovery when the operator leaves setup or
its context changes. Keep reserved-popup cleanup bound to the current
attempt.

## Verification

- Current read head `09d1db29dea` is merge-ready: all 59 check runs are
completed (54 successful, three skipped, two neutral), the legacy Snyk
status is successful, and both required `ci / verify` and `ci / e2e`
gates pass. Greptile Apex reviewed this exact head at 5/5 with no
unresolved threads; the branch has no merge conflicts. The current CI
run is `41748339602`.
- Current read head `09d1db29dea` includes master `683fa4725e9`. The
latest change touches only the existing server deadline test fixture.
Application source remains the previously verified `86376fa751e`
revision: all 60 affected skills/disconnected-app tests and both
application UI typechecks passed, along with token gates and the
enforcing query-error guard. Those UI suites were not repeated for this
test-only repair.
- The fixture repair passes all seven deadline integration cases and the
complete server typecheck, including the server test layer. The old
assertion deterministically fails at 23:59:06.888 UTC; its
one-second-old run also fails the first-second-of-day probe. The
corrected fixture passes noon and both midnight probes. The permanent
three-case coverage uses seeded 2099 clocks.
- Twelve new regressions fail against the previous implementation. Both
skills variants and disconnected-app pages show either request's 403 or
429 while the sibling pauses or remains fetching. Permission failures
stay visible. Retry stays usable for its failed request and recovers
independently. An initial test-fixture query-key typo was corrected
before the passing run. The finding is replied to and resolved. Greptile
reviewed prior application revision `86376fa751e` at 5/5 with no new
actionable findings or unresolved threads. The current test-only head
has now received its own Apex 5/5 review with no unresolved threads, and
its CI is successful as recorded above.
- The preceding OAuth fix passed all 237 setup tests on both application
branches. Its ten regressions prove that Back, Cancel, unmount, or
context changes retire initial discovery without creating a connection,
caching old identities, navigating an abandoned popup, or closing a
newer attempt.
- Earlier master integration `e992f770339` passed all 309 tests across
eight affected Slack, recovery, retry, submission and task files, plus
UI typecheck, token gates, enforcing query-error scan and seven guard
tests. The integration preserves Slack workspace reconnect and shared
recovery, readable execution errors, source-inspection company scoping,
and current submission behavior. Newly inherited retained-file recovery
controls keep cached data through transient failures and hide it after
definitive denial.
- The preceding cache head `97c0bc3966f` passed 1,038 focused
cache/write/editor/account/task/API tests. All 22 built-UI
outage/write/loading cases passed in 4.7 minutes on exact browser head
`b7deaaee5da`, with strict checking of all three specs. The latest
skills/app rendering fix is covered by its affected component suites;
those broader cases were not repeated for this separate read path.
- Historical broader UI, CLI and build results and unresolved aggregate
server/DB/adapter failures remain in the PR discussion. Interrupted CI
runs were audited and recorded separately; they are not counted as
passes. Controlled local outages do not prove hosted authentication or a
real Cloud restart/deploy. Real hosted acceptance remains required after
merge and deployment.

Commands for the latest fix:

```sh
pnpm --filter @paperclipai/ui exec vitest run src/pages/CompanySkills.test.tsx src/pages/apps/AppNotConnected.test.tsx
pnpm --filter @paperclipai/ui typecheck
pnpm check:token-gates
node scripts/check-query-error-rendering.mjs --enforce --summary
```

## Risks

This migration touches many read surfaces. The main risks are initial
loading, request cancellation, retained authorization state, and run-log
paging. The tests cover those states. There are no server, schema,
dependency, or API contract changes. The follow-up browser suite and
hosted deploy acceptance are required before release acceptance.

Recovered commits retain their original authors. If squash-merging,
preserve `Co-Authored-By: Bender (Fable) <bender@agents.paperclip.ing>`
and the Paperclip trailer.

## Model Used

OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning effort, tool
use, and code execution. The context window size is not exposed. Saved
source commits keep their original authorship.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the bug report template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] The affected local tests pass; broader limitations are recorded
above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Bender (Fable) <bender@agents.paperclip.ing>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 14:07:18 -08:00
DottaandPaperclip 519b345d69 fix(ui): expose governed retained-file recovery (#18848)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The control plane must preserve saved work before it admits another
attempt.
> - Retained-file recovery already has an authorized API and a bounded
worker.
> - The task UI hid that operation when workspace repair blocked
execution.
> - The run inspector could check source identity but could not request
preservation.
> - This pull request exposes that existing operation with its exact
source binding.
> - The worker still owns file verification, stop receipts, and
execution release.

## Linked Issues or Issue Description

Refs #18678 and #18803 for the existing retained-source and continuation
contracts.

**What happened?**

A failed native run could leave an active, system-owned export recovery
action. The task showed a generic Retry control. Its workspace repair
gate correctly refused another provider turn. The run inspector offered
only read-only source inspection. The authorized file-preservation API
had no production UI caller.

**Expected behavior**

An authorized board operator can request preservation of the exact
retained files. The UI keeps the repair hold until the server records
verified preservation. A lost response must keep the same request
identity.

**Steps to reproduce**

1. Use an active retained-export action whose workspace recovery is
blocked and retryable.
2. Open its source task while `workspaceRepairRequired` is true.
3. Open the exact failed run and inspect recovery.
4. Before this change, neither surface can request the existing
file-preservation operation.

**Paperclip version or commit**

The focused regression fails on master
`1f55073f96b2485f49f98ee8c261aa14ec2f79ac` before the production UI
edits.

I checked the roadmap and searched open public PRs for this UI repair.
No duplicate was found. This is a repair to the existing recovery
capability.

## What Changed

- Add Recover saved files to the existing task recovery area and
exact-run inspector. Reuse the current board task-mutation access check
and the existing `recoverWorkspace` API.
- Read the exact action, run, company, task, and source observation
again before sending. Refuse changed, retired, foreign, unverified, or
non-retryable sources.
- Save one account-scoped request before POST. Keep its key through a
lost response and reload. Reconcile with GET before an explicit resend.
Use a new key only after an explicit click and a fresh, confirmed
exhausted attempt; do not resend automatically.
- Show queued, recovering, blocked, and verified receipt states
separately. A request acknowledgement does not mark files saved or
resolve the task hold. Hide generic Retry for native export holds. Keep
the existing Retry path for legacy restoration.
- Cover both mounted surfaces and negative access, source, account,
persistence, duplicate-click, and lost-response cases. Update the
existing design guidance and correct the affected run-detail recipe
only.

## Verification

- RED: the new component regression fails with `missing Recover saved
files` before production changes. The original failure is retained.
- GREEN: 265 tests pass across
`WorkspaceExportRecovery.control.test.tsx`,
`WorkspaceExportRecovery.test.tsx`,
`RunWorkspaceRecoverySurface.test.tsx`, and the full
`IssueDetail.test.tsx` suite.
- Review RED/GREEN: the prior UI reused the failed key after a confirmed
exhausted attempt. A focused regression now proves a different key for
an explicit new attempt, while the lost-response controls still prove
key reuse. Mounted task controls prove that verified native preservation
stays available, unverified or stale native sources never offer generic
Retry, and viewers cannot mutate. A separate RED against the exact prior
UI proves the lost legacy control. Its GREEN clicks the existing legacy
Retry API while the workspace hold remains active.
- Full UI TypeScript check passes: `node scripts/run-tsc.mjs -p
ui/tsconfig.json --noEmit`.
- `node scripts/check-token-gates.mjs`, `node scripts/check-maps.mjs`,
and `git diff --check` pass. The map check proves structure only.
- One early legacy test probe used an unsigned fixture and a readonly
array; those setup failures are retained. The corrected same-user
fixture proves the source RED and GREEN. The first test command used an
unnamed project filter and stopped before loading tests. The corrected
package configuration produced the recorded RED. One initial storage
fault probe did not intercept jsdom's storage method; the corrected
prototype fault proves that unavailable storage sends no POST.
- Dependencies use Node 24.22.2 and a frozen pnpm 9.15.4 install with
scripts disabled. The maintained shared and SDK prerequisite builds pass
on this base. No dependency or lockfile changes are included.
- A later Paperclip Review follow-up found the plain-HTTP UUID gap. Its
focused regression fails before POST with the prior direct call, then
passes with the existing UUID helper. The same storage, lost-response,
retry and source controls remain green.
- Original full CI `41715060582` fails two Cloud-auth task-entry browser
checks at their existing heading assertion. The retained original
artifacts show a late task pane (one already contains the heading; one
still shows Loading conversation). The cause remains unproven. No auth,
fixture, timeout or assertion change is included. Fresh Greptile is 5/5
with zero open threads and Paperclip Review passes on the UUID fix head.
Current full CI `41720068249` attempt 1 failed. Three workers were
terminated by AWS Spot capacity loss. The CLI startup and dense-ZIP
checks also timed out at their unchanged deadlines. The CLI/shared input
trees match a previous passing CI job, and the dense-ZIP case passes one
focused local check. The CLI timeout cause remains unknown. The
coordinator requested one bounded recheck. Attempt 2 succeeds on the
same source, with all current checks green. No source, fixture, timeout
or assertion changed for that recheck. The worker issued no retry. This
pass does not establish the original CLI timeout cause.
- Full recursive typecheck, repository tests, and application build were
not run locally under the bounded UI verification scope. Full CI is
requested.
- No live preservation request, provider turn, deployment, or sandbox
cleanup was performed for this repair. Original saved-file recovery
remains unverified until rollout and real acceptance.

## Risks

The client access check is a display guard. The existing server
authorization and physical source, ownership, stop, and snapshot checks
remain authoritative. Unavailable browser storage blocks the request
before POST. A failed acknowledgement retains the request for its owner.
The UI never calls task Retry, recovery resolution, reconstruction,
provider execution, or destructive cleanup. The existing worker retains
its bounded recovery policy and may finish the saved result after real
preservation.

## Model Used

OpenAI Codex, GPT-6 family, with repository inspection, code editing,
and test execution. The exact runtime model ID and context-window size
are not exposed to this agent.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 16:00:03 -06:00