Commit Graph
14 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 08f7ac03dc fix(recovery): keep a never-started task waiting through a temporary accounting hold (#19702)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Admission checks every wake. A short company accounting hold can
refuse new work after a control-plane restart.
> - When the hold refused a new task's assignment wake, stranded-work
recovery found the task with no run while the hold was still active. It
moved the task to `blocked` with a board recovery comment.
> - A `blocked` task leaves the stranded-assignment sweep, so nothing
woke it after the hold cleared. The task stayed `blocked` with no run,
no interruption and no Retry.
> - The hold was temporary. Tasks created a few seconds later (after the
sweep) started normally once accounting settled.
> - This pull request keeps a never-started task `todo` while a
temporary budget hold lasts. The existing assigned-todo liveness
dispatch then admits it once the hold clears. The task shows the hold it
waits for.
> - The benefit is that a restart no longer silently drops new tasks
created during the accounting settle window.

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos storm on master `06bbcea12f`, three new tasks were
created while the company accounting hold was active. Each assignment
wake got 409 "Company cannot start work while completed runs await
accounting." (or "...native run recovers its unfinished accounting.").
One task rested `blocked` with 0 runs, interruption `null` and
`canRetry=false`. The other two ran later.

**Root cause**
`reconcileStrandedAssignedIssues`
(`server/src/services/recovery/service.ts`, the
`readInvocationBudgetBlock` branch near line 4826) escalates any
assigned task to `blocked` when a budget block exists. It does this
before the assigned-todo no-run branch, which already skips a
budget-blocked task and dispatches it later. So a task that never
started was treated as failed recovery work. The refused wake itself is
stored as a `skipped` `budget.blocked` diagnostic, which is correct; the
escalation removed the only path that re-admits it.

**Expected behavior**
A wake refused by a temporary admission hold waits and is admitted once
the hold clears, with one run. A real hard stop is still a board
decision. Removed assignees and cancelled tasks still do not wake.

**Steps to reproduce**
Create a company with a hard-stop budget and a completed run that has
`cost_accounting_pending = true`. Assign a new `todo` task. The wake
gets 409. Run the stranded-assignment sweep. Before this change the task
becomes `blocked` and never runs.

Related: #19700 bounds the accounting hold itself. This PR is
independent: it handles what happens to a task whose first wake the hold
refused.

## What Changed

- Stranded-work recovery skips a `todo` task with no run while a
temporary budget hold blocks admission. It does not escalate it to
`blocked`. The existing assigned-todo liveness dispatch admits the task
once the hold clears.
- A hard stop (spend at the limit, or blocked unpriced usage) keeps the
existing board escalation. Recovery checks this with the read-only
`policyBlocks` for the policies of the blocking scope. It does not
change budget hold semantics.
- The task interruption projection now covers a `todo` task with no run
whose latest wake was refused with `budget.blocked`. It shows "Waiting
for the budget to allow new work. The task starts automatically when it
does." with no Retry. A budget pause shows the existing "budget" user
action.
- The no-run projection applies the same person-owned gates as the
run-backed projection: a budget pause shows the "budget" action, and an
ordinary agent, project, company or task pause shows "resume". Only an
accounting hold shows the automatic-start text.
- `ExecutionBlockerNotice` shows that wait text for a `working`
presentation that carries a wait reason.
- `doc/execution-semantics.md` records the rule.

## Verification

- New embedded-Postgres regression
`server/src/__tests__/heartbeat-refused-assignment-wake.test.ts`:
- hold active, new task assigned, wake refused (409, `skipped`
`budget.blocked`), two recovery sweeps: the task stays `todo` with no
run and shows the wait reason. Before the fix this failed with `expected
'blocked' to be 'todo'` (the CHA-9 shape).
- hold cleared, two sweeps: exactly one provider turn starts, from
`issue.assigned_todo_liveness_dispatch`, and the task finishes. A later
sweep does not start another turn.
- counter-tests: a terminated assignee and a cancelled task get no run
after the hold clears.
- Existing `creates a board recovery action for budget-blocked assigned
work` (real hard stop) still passes unchanged.
- `pnpm exec vitest run
src/__tests__/heartbeat-refused-assignment-wake.test.ts
src/__tests__/heartbeat-process-recovery.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/heartbeat-issue-liveness-escalation.test.ts` (server) and
`src/components/ExecutionBlockerNotice.test.tsx` (ui).
- Server typecheck and `pnpm check:token-gates`.
- Live restart-chaos storm (local, codex/claude/opencode, 6 in flight,
K2 every 3-4 minutes for 20 minutes, no RC-06 mask), revision
e38bb5c8f7: 17 tasks, 10 pass.
- Two new tasks (CHA-16, CHA-17) were created while the company
accounting hold was active. Their assignment wakes got 409 "Company
cannot start work while completed runs await accounting." Both stayed
`todo` (not `blocked`) and showed `state: working`, `canRetry: false`,
`retryWait.reason: budget.blocked`, "Waiting for the budget to allow new
work. The task starts automatically when it does." No task rested
`blocked` with 0 runs.
- The hold itself never cleared in this wave (`pendingRunCount: 1` at
wave end), so these two tasks did not start before their deadline. That
stuck hold is the separate accounting-closure defect that #19700 bounds.
The other 5 failures (`R3:retry_no_new_run:B5`) are Retries refused by
the same stuck hold. The re-admission after the hold clears is proved by
the embedded-Postgres regression above.

## Risks

- Low. The change narrows one escalation to tasks that never started,
under a budget block that is not a hard stop. Those tasks now stay
`todo` and are re-checked by the existing 30-second stranded-assignment
sweep, which already re-checks every admission gate.
- The hard-stop check reads the blocking scope's active policies once
per budget-blocked never-started task in a sweep. It runs only while a
hold is active.
- The interruption projection adds one indexed read (company + payload
issue id) for `todo` tasks in the batch.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 21:12:00 -06:00
DottaandClaude Opus 5.5 4938a930d6 fix(budgets): stop pending or restart-lost accounting from hard-stopping companies (#19497)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Budget admission and run accounting decide whether any task in a
company may start
> - A control-plane restart that kills a provider turn mid-flight can
lose that turn's final usage receipt. ACPX Claude reports usage only at
turn end, so the receipt never becomes complete
> - Accounting kept such a run "awaiting receipt" forever. Every later
admission in the company then refused with "completed runs await
accounting", and the budget evaluator opened a hard-stop incident that
paused the company. No product action could clear it: raising the budget
refuses while runs await accounting, accounting retry finds no new data,
and allowing unpriced usage does not apply to pending runs
> - Restart chaos waves showed this as a dead task Retry, a new task
blocked as "over budget" at 22 of 500 cents, and a hard stop that
cancelled queued Retries and an unrelated running task
> - This pull request closes the debt from evidence: once native
recovery reaches terminal failure and the provider stop is verified, the
known part settles as an unpriced charge. Until then the run holds new
work as recovering accounting, without a hard stop
> - The benefit is that a restart no longer bricks a company with a hard
budget. The operator's unpriced-usage policy decides admission, and
unknown spend still never shows as zero cost

## Linked Issues or Issue Description

Refs #19399, #19426, #19431 (earlier restart Retry fixes; this is the
budget layer under the same journey).

**What happened?**
After a control-plane SIGKILL during an ACPX Claude turn, the killed run
ended `native_provider_terminal_failed` with
`usage_json.accountingReceiptReady = false` and `cost_accounting_pending
= true`. It stayed that way for good. The task Retry was accepted but
parked with `budget.blocked` ("Company cannot start work while completed
runs await accounting"), which the UI showed as "Retry is waiting for
execution checks". A new Codex task was escalated to `blocked` with "the
original recovery target is over budget" while the company had spent 22
of 500 cents. After the next restart the budget evaluator opened a
hard-stop incident (`amount_observed` 22.08 < 500), paused the company
and cancelled both queued Retries.

**What did you expect?**
A restart should not stop the company. Unknown usage from a stopped
provider should become an honest, operator-governed state, and Retry
should start a run.

**Steps to reproduce**
1. Give a company a hard-stop budget.
2. Start an ACPX Claude task and SIGKILL the control plane mid-turn, so
the resumed turn fails.
3. Observe the run stay `cost_accounting_pending` with an incomplete
receipt, and every new admission refuse.

## What Changed

- `budgets.ts`: completed runs that await accounting now hold new work
only. They no longer open a hard-stop incident, pause the company, place
an agent budget hold, or deliver Stop to running work. Only spend at the
limit, or unpriced usage under a blocking policy, does that. Chaos wave
A and a pre-restart lifecycle cell showed the wedge: a run that had just
succeeded stayed terminal for about 100 ms before its charge committed,
the evaluator paused the company in that gap, and the Stop it delivered
left a cancelled run whose receipt never closed.
- `run-cost-accounting.ts`: a failed native run is a closed receipt
source only with positive stop evidence: its finalization is
`terminal_failure` with no result (not a finalization-deadline failure),
it no longer records a process identity, and its latest server-authored
process event is `native.local_process_stopped`. One SQL predicate
serves the reconciler scan and settlement. Accounting now settles its
known part as an `unpriced` charge instead of waiting forever. The
background reconciler selects these runs. The reservation settles. Other
incomplete receipts stay pending as before. A cancelled run keeps its
existing accounting drain, so Stop recovery can still deliver its final
receipt.
- `native-local-process-stop.ts` and its transactional callers:
releasing a verified local or remote process stop wakes the accounting
worker, so the closed debt settles promptly.
- `usage-receipts.ts`: a charge settled without its receipt is marked
`accountingReceiptUnavailable`. A later complete receipt that carries
its own price (including a rate-card price) corrects it. A receipt with
no own price and unchanged usage does not reopen the hold.
- `budget-exposure.ts`: the blocking unpriced-usage policy no longer
counts a charge settled from a lost receipt with no observed usage. That
is unknown, not "usage with no reliable price". Complete receipts that
report no tokens (for example Cursor Cloud) still count. A live wave
under the default policy hard-stopped its company at 0 of 500 cents on
one such charge.
- `budget-exposure.ts`: a terminal-failure native run with an
incomplete, not capture-failed receipt counts as recovering accounting.
Admission holds new work ("New work must wait while a native run
recovers its unfinished accounting") without a hard-stop incident, a
company pause or cancellation of running work.
- `recovery/service.ts`: the stranded-work comment names the actual
budget gate instead of always saying "over budget".
- `task-interruption-presentation.ts`: a Retry parked by
`budget.blocked` says it waits for the budget, not for "execution
checks".
- `doc/SPEC-implementation.md`: documents the closed-receipt contract
next to the existing incomplete-receipt rule.

## Verification

- New embedded-Postgres suite
`server/src/__tests__/closed-native-receipt-accounting.test.ts`, built
from the persisted shape of the killed run:
- Red before the fix: the run stayed pending, `reconcileRunCosts`
accounted 0, and admission refused with "completed runs await
accounting".
- Debt stays pending with no verified stop event, with a
finalization-deadline failure, with a recorded process identity, and
while the coordinator can still produce a result.
- Under a blocking policy, a lost receipt with no usage does not
hard-stop; one with tokens still blocks. A run with a still-recorded
process identity paused the company with a hard-stop incident.
- Green after: the stopped run settles as one `unpriced` charge with the
reservation settled; admission is clear under an `allow` policy and
names "no reliable price" under `block`; a run with a recorded process
identity or a non-terminal coordinator stays pending; the closing window
holds admission without an incident or pause; a final receipt that
arrives later still enters late-receipt reconciliation.
- Pending accounting from a run that just finished holds admission with
no incident, no pause, no enforcement version bump, and an unrelated
running run stays running. Red before the `budgets.ts` change: the
company was paused.
- A later priced receipt corrects the unavailable charge to its price;
an unpriced one leaves it settled.
- `task-interruption-presentation.test.ts`: the `budget.blocked` Retry
wait message.
- `budget-enforcement-scoping.test.ts`: the accounting-hold release
cases now assert that pending accounting places no budget hold and that
admission clears after settlement.
- Existing suites pass locally: all budget suites (control publication,
enforcement scoping, enforcement worker, evaluation work, live attempts,
history, service, live usage, budget cancellation),
native-stop-accounting, explicit-native-continuation,
native-case-handoff, cost-data-quality, heartbeat-cost-accounting,
run-cost-accounting, native-usage-delivery. Server typecheck passes.
- Runner e2e cell
`lifecycle-baseline.runner-codex.local.lifecycle-dependency-restart-neutral`,
which failed on master with a false agent budget hold before any
restart, passes on this branch.
- Live restart-chaos wave (local runner, K1 SIGKILL, codex + 2× ACPX
Claude + OpenCode, 4 restarts, company policy `unpricedUsagePolicy:
allow`) with the accounting fix: every killed run settled (no
`cost_accounting_pending` left), the company stayed active, the OpenCode
Retry completed the task on the first click and the Codex Retry finished
its task. That wave also exposed a 1-second false hard stop in the
closing window (observed 0 of 500 cents) that cancelled the running
OpenCode turn; the `budget-exposure.ts` change fixes it and has its own
red/green test.
- Confirming live wave with both commits, same plan (2× ACPX Claude,
OpenCode, Codex; 4 K1 restarts): both Claude tasks passed R3 (one Retry
click started a run that completed the task), OpenCode passed R1
(continued by itself), and the Codex Retry completed its task. Zero
budget incidents, zero cancelled runs, zero runs left pending
accounting, company active; three killed runs recorded as `unpriced`
charges. The only open signature is an unrelated workspace-durability
check (`R2:pre_fault_files_lost` on the Codex long-bench task).

## Risks

- Behavior change: a stopped native run's lost receipt now becomes an
`unpriced` ledger charge with its known usage. If that usage has tokens,
the default `block` policy still stops new work with "recorded usage has
no reliable price". If it has none, the charge is visible in Costs but
does not stop the company. Other unpriced charges keep their existing
effect. Unknown spend is never recorded as a priced receipt. Budget
enforcement stays best effort, as the spec says; killed OpenCode and
Codex requests already lose their in-flight usage.
- Remote runs and runs without a verified local stop keep their debt
pending (an admission hold only, no hard stop).
- Contract change: pending accounting no longer pauses a scope or stops
running work. New work still waits with "completed runs await
accounting". A scope's real hard stop (spend at the limit, or blocked
unpriced usage) is unchanged.
- If a provider stop can never be verified, the run stays "recovering
accounting": new work waits without a hard-stop incident. Before, the
same state paused the company. The hold is visible in admission errors
and Costs.
- A receipt that arrives after settlement posts a late correction. The
original unpriced charge stays in the ledger.
- No schema or migration change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, embedded Postgres tests, live
restart-chaos harness).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:27:51 -06:00
DottaandClaude Opus 5.5 543aa4d331 fix(tasks): show a pending native same-run resume instead of a competing Retry (#19503)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native run cannot reattach to its provider after a
control-plane restart, the native coordinator resumes the same run by
itself after about 30 seconds, up to three attempts
> - The run projection shows this correctly as "Retry scheduled" with
only `inspect_run` permitted
> - The task interruption shown on the issue read only system-recovery
evidence and `scheduledRetryAt`. It never read the coordinator's
`nextAttemptAt`, so it showed `state: failed, canRetry: true` and
offered a manual Retry while the automatic resume was still pending
> - The two surfaces disagree, and a user click can race the automatic
resume. That is one way a Retry gets parked behind "execution checks"
> - This pull request makes the task interruption read the coordinator's
pending resume time
> - The benefit is one consistent story: while the automatic resume is
pending, the task shows it with its time and no competing Retry. If
recovery ends, exactly one Retry returns

## Linked Issues or Issue Description

Refs #19487, #19497, #19426.

**What happened?**
A native run failed with `native_session_interrupted` after a restart.
Its finalization row had `phase: retryable_failure` and `nextAttemptAt`
30 seconds later. The run projection said "Retry scheduled". The issue
interruption said `state: failed, canRetry: true, nextRetryAt: null`.

**What did you expect?**
While the automatic resume is pending, the task shows that it is
resuming, with the time, and no Retry. When the resume fails for good or
never runs, exactly one Retry that works on the first click.

**Steps to reproduce**
1. Run a native task and restart the control plane so the native attach
fails.
2. Read `GET /api/issues/:id` while the coordinator's resume is pending.
3. Observe `interruption.canRetry: true` while `execution.phase` is
`retry_scheduled`.

## What Changed

- `task-interruption-presentation.ts`: the batched run read selects the
native coordinator's `next_attempt_at` when the native run failed, has
no result, and its finalization phase is `retryable_failure`.
- A pending resume in the future shows `retry_scheduled` with its time
and `canRetry: false`.
- A due resume shows `working` for up to two minutes while its dispatch
runs. After that, or once recovery is terminal, the task falls through
to the existing single Retry.

## Verification

- `task-interruption-presentation.test.ts`: new case for a future
resume, a due resume, a stale resume, and no resume. It fails before the
change (the task offered Retry) and passes after.
- `task-interruption-query.test.ts` (embedded Postgres): the real
batched query shows the pending resume for both list and detail reads,
and returns one Retry after the coordinator becomes `terminal_failure`.
- All 40 presentation tests and 17 query tests pass. Server typecheck
passes.

## Risks

- Low risk. This changes only the read model for tasks whose native
coordinator holds a pending same-run resume.
- If a resume never dispatches, Retry is hidden for up to two minutes
after its due time before it returns.
- The `POST /issues/:id/retry` endpoint is unchanged. A client that
ignores `canRetry` can still submit a Retry during the resume.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, embedded Postgres tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 11:33:46 -06:00
DottaandClaude Opus 5.5 140fcf98ed fix(retry): name a declined task Retry and let a new click replace it (#19426)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task interrupted by a restart or deploy must resume automatically,
or offer a Retry that works on first use
> - A board Retry saves an intent first. Dispatch then rechecks the
requester's authority and saved source before it starts a run
> - When that recheck refused the intent, dispatch returned a bare
"deferred". The intent stayed saved with no reason, and the task still
showed Retry
> - Each later click with a new key joined that same intent, so Retry
returned 202 and never started a run
> - This pull request names the refusal on the intent and lets a new
click replace an intent that can never start
> - The benefit is that a Retry button that is shown starts one run, or
explains why it cannot

## Linked Issues or Issue Description

**What happened?**
A native task in a managed sandbox lost its runner twice. Saved-work
recovery recovered the files. The task showed "Saved work recovered.
Retry or send a message to continue." and `canRetry: true`. A Retry
click returned 202 and logged `issue.retry_requested`, but no run
started. A second click with a new idempotency key returned the same
request ID and `state: failed`. No run started.

**Expected behavior**
If Retry is offered, a click starts exactly one run. If the Retry cannot
start, the task names the reason, and a later click is a fresh attempt.
A Retry accepted during the pre-deploy drain starts after the restart.

**Steps to reproduce**
1. Recover the files of a failed native run so the hold offers Retry.
2. Save a Retry intent whose dispatch-time authority recheck fails (for
example, the requester lost access).
3. Dispatch it: no run starts, and the intent has no wait reason.
4. Click Retry again with a new key: the response returns the first
request ID, and no run starts.

**Paperclip version or commit**
Reproduced on `b400914345` (staging served `4d49a31286`; the affected
code is the same).

Related: #19399 (automatic Retry after a lost runner), #19422.

## What Changed

- `heartbeat/queue.ts`: when dispatch declines a task Retry because the
requester can no longer change the task, or its saved board message
changed, record a named wait (`retry_authority_denied`,
`retry_source_changed`) and log it. Before, it returned "deferred" with
no record.
- `task-retry.ts`: a new click with a new key replaces an unadmitted
intent that dispatch declined for its requester's access
(`retry_authority_denied`). The old intent becomes `skipped` /
`task_retry_replaced`, and activity records `replacedRequestId`. Waits
on shared state (cleanup, recovery, an active run, preservation) still
coalesce. The restart hook never replaces a user's click. Chat-owned
intents keep their own owner.
- `task-interruption-presentation.ts`: fixed public messages for the two
new wait reasons. `retry_authority_denied` keeps Retry, because a new
click is a fresh attempt. `retry_source_changed` is terminal: an edited
saved message cannot be replayed, so Retry is hidden and the user sends
a new message.
- `doc/execution-semantics.md`: document the rule.

## Verification

- New regressions in `server/src/services/workspace-recovery.test.ts`
use the real recovery, Retry and dispatch services on embedded Postgres.
The fixture runs the automatic Retry from a lost runner, then that
Retry's own run loses its runner, and recovery saves its files:
- `starts exactly one run when Retry is pressed after the automatic
Retry's own run lost its runner` (also checks that a replayed dispatch
and a second click start no second run).
- `does not let a Retry that could not start swallow a later Retry`: red
before the fix in two steps. With no fix, the declined intent has no
wait reason. With only the wait fix, the second click returned the first
request ID. Green after the fix: one run, bound to the second click.
- `dispatches a Retry accepted during the pre-deploy drain once the
restarted server resumes`: green before and after. The drain does not
lose a saved Retry.
- `npx vitest run src/services/workspace-recovery.test.ts
src/services/task-retry.test.ts
src/services/task-interruption-presentation.test.ts
src/services/quarantined-task-retry.test.ts
src/services/explicit-native-continuation.test.ts`: 892 passed.
- Not verified on staging. After deploy, an intent that dispatch refuses
shows a named `retryWait` reason.

## Risks

- Low risk. Only intents that dispatch declined for their requester's
access are replaced. All other coalescing stays.
- A replaced intent's late dispatch finds it `skipped` and starts
nothing, so a double click still starts at most one run.
- The exact gate that held the staging intent is not confirmed from
records. If it was a shared-state wait, the new code names it only if it
is one of the two reasons above.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 03:52:21 -06:00
DottaandClaude Opus 5.5 9ceaec6f97 fix(issues): stop showing resource waits as failed attempts (#19369)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The task page shows one interruption banner. The server builds it
from the task's latest run.
> - An auto-retry that waits for a busy workspace, AI connection or
runner update is not a failure. The dispatcher re-checks the resource,
so the run can stay `scheduled_retry` after its retry time.
> - The projection did not know about these waits. After the retry time
passed, it fell through to "Couldn't finish this attempt" with a Retry
button.
> - Retry cannot free the resource. The banner cleared for the 10-second
grace and then came back. Users had no way to fix it.
> - This pull request projects a scheduled resource wait as `working`,
with no Retry.
> - The benefit is that the failed-attempt banner shows only for real
failures.

## Linked Issues or Issue Description

**What happened?**
During onboarding on Paperclip Cloud staging (Claude subscription and
Codex subscription paths), the onboarding task showed "Couldn't finish
this attempt. Your message is saved." with a Retry button. The banner
did not name an attempt. Retry cleared it briefly, then it came back.

**Steps to reproduce**
1. Start a task run while another run holds the same workspace.
2. The run is cancelled with `workspace_busy`, and a `scheduled_retry`
run is created with `scheduledRetryReason: "workspace_busy"`.
3. Keep the workspace busy after `scheduledRetryAt`. On the staging
company, one retry stayed scheduled for 16 minutes past its retry time.
4. Open the task. The failed-attempt banner shows. Click Retry. The
banner comes back.

**Expected behavior**
No failed-attempt banner while the system waits for a shared resource.
The run status already says "Waiting for workspace"
(`execution-projection.ts`).

## What Changed

- `server/src/services/task-interruption-presentation.ts`: a
`scheduled_retry` run whose `scheduledRetryReason` is `workspace_busy`,
`ai_connection_busy`, `ai_connection_pool_wait` or
`runner_revision_wait` (and that has no system recovery) projects as
`state: "working"`, `canRetry: false`. User gates and failed Retry
preparations stay authoritative because they are checked first.
- The batched query now selects `scheduledRetryReason`.
- Tests: a Postgres integration test (`task-interruption-query.test.ts`)
that seeds the staging run chain for each reason, and unit tests for the
guard cases (failure-lane retries, paused agent, failed Retry
preparation).

## Verification

Red on `master` (fix not applied), `cd server && npx vitest run
src/__tests__/task-interruption-query.test.ts -t "overdue"`:

```
     × does not present an overdue workspace_busy wait as a failed attempt 213ms
     × does not present an overdue ai_connection_busy wait as a failed attempt 25ms
     × does not present an overdue ai_connection_pool_wait wait as a failed attempt 13ms
     × does not present an overdue runner_revision_wait wait as a failed attempt 11ms
- Expected
+ Received
-   "canRetry": false,
+   "canRetry": true,
-   "state": "working",
+   "state": "failed",
      Tests  4 failed | 11 skipped (15)
```

Green with the fix, same tests plus the new unit tests:

```
 ✓ task-interruption-query.test.ts > does not present an overdue workspace_busy wait as a failed attempt
 ✓ task-interruption-query.test.ts > does not present an overdue ai_connection_busy wait as a failed attempt
 ✓ task-interruption-query.test.ts > does not present an overdue ai_connection_pool_wait wait as a failed attempt
 ✓ task-interruption-query.test.ts > does not present an overdue runner_revision_wait wait as a failed attempt
 ✓ task-interruption-presentation.test.ts > scheduled resource wait presentation > keeps an overdue workspace wait out of the failed-attempt UI
 ✓ task-interruption-presentation.test.ts > scheduled resource wait presentation > keeps failure retries, user gates and failed Retry preparation authoritative
```

Neighbouring suites: `task-interruption-presentation`,
`task-interruption-query`, `workspace-recovery`, `agent-project-access`,
`heartbeat-dependency-scheduling`: 5 files, 458 tests passed.

Server `tsc --noEmit` reports no errors in changed files. The only
errors locally come from the unbuilt `@paperclipai/paperclip-runner`
vendor package, which this change does not touch. No UI change, so `pnpm
check:token-gates` is not applicable.

Evidence: the staging onboarding task's run history shows
`workspace_busy` cancellations at 17:46, 17:52 and 18:10, each followed
by a `scheduled_retry`. One had `scheduledRetryAt` 17:53:57 and started
at 18:10:20. The next run succeeded at 18:20.

## Risks

- Low risk. Presentation only; dispatch and admission do not change.
- If a resource wait never clears, the task shows no failed-attempt
banner. The run status still shows "Waiting for workspace", and the
existing retry-attempt limits still end the wait.
- Not in scope: genuine failures without a provider or workspace
explanation still show the generic text, and an overdue failure-lane
retry still shows as failed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser for read-only staging evidence).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 19:08:27 -06:00
b929ad43de fix: keep dependency waits neutral and explain GitHub access failures (#19315)
## Thinking Path

> - Paperclip manages agent work and shows users what can move it
forward.
> - Task status and tool approval cards describe different parts of that
work.
> - A queued dependency wait has no provider attempt to retry.
> - GitHub can reject an approved action while other connection tools
still work.
> - These cases showed a false Retry banner and an unhelpful provider
error.
> - This pull request keeps dependency waits neutral and gives GitHub
access failures a direct repair path.

## Linked Issues or Issue Description

Related: #19268 preserves pending Retry preparation and current failure
causes.

**What happened?**

A queued wake cancelled by the dependency gate appeared as a failed
attempt with Retry. A GitHub MCP permission denial appeared as raw HTTP
error text after an approved repository action.

**Expected behavior**

Keep the task's dependency state. Show GitHub access failures with a
clear explanation and a link to the selected connection's Permissions
page. Preserve actual failures, provider details, and approval gates.

**Steps to reproduce**

1. Queue a wake, then leave its task dependent on unfinished work before
dispatch.
2. Open the task after the dependency gate cancels that wake.
3. Separately approve a GitHub action whose MCP result contains `403
Resource not accessible by integration`.
4. Reopen the saved tool approval.

## What Changed

- Recognize both queues' unstarted dependency-wait receipts in
cancellation provenance and task projections. No dispatch or recovery
gate changes.
- Classify GitHub's specific permission error after MCP result
normalization. Keep result validation, the raw diagnostic, and working
tools available.
- Apply the same display code to both historical GitHub error codes.
Identify the provider from company-scoped connection metadata, including
renamed apps, without rewriting stored decisions.
- Show a token-based inline access card with **Manage GitHub access**
and collapsed **Error details**.
- Give agents trusted recovery steps after an approved GitHub action
fails. Keep provider text in the untrusted result and prevent automatic
replay.
- Add regressions, a production-component Storybook case, and
execution/connector documentation.

## Verification

- Red before the fix: cancellation provenance, dependency-wait
projection, the real queue cancellation, approved MCP failure
classification, saved interaction projection, and the error card
assertions failed at the expected behavior.
- Green focused tests: queue cancellation, one ready-task executor,
approval-before-dispatch, one failed GitHub request, connection reuse,
retained provider diagnostics, saved interaction reads, task list/detail
projection and redaction.
- UI: 106 component tests passed. Token gates and Storybook catalog
checks passed.
- `node scripts/typecheck-low-memory.mjs`: passed.
- Local source instance: verified a saved dependency wait stays neutral
after reload; a saved GitHub error shows repair guidance, expands its
original error, and opens the correct connection Permissions page.
PostgreSQL and HTTP APIs were real. The GitHub denial was a controlled
MCP fixture; no account or provider permission changed.
- `pnpm build`: passed for all workspace packages, the Runner binary,
server, CLI and UI. The server build and all server typecheck layers
also passed after the review repair.
- Additional safety checks passed for realistic repository arguments,
active recovery, failed Retry preparation, and bounded provider
diagnostics.
- Review repair: 45 pure tests and 7 PostgreSQL cases passed. These
cover both historical error codes, renamed apps, and
company/provider/application identity boundaries. A second local browser
check verified the renamed app, retained error detail, and correct
Permissions destination.
- Final review regressions: the dispatcher and metadata cases failed
before the repair. Afterward, 25 focused cases and 59
projection/identity cases passed, including connection-key precedence.
The native wait and app-only GitHub source also passed a local browser
check.
- The historical read adds at most one connection query per interaction
read batch. It selects IDs and the provider key only and runs only for
candidate permission failures. It adds no provider requests.
- Full local `pnpm test:run` stopped at the chat suite's `afterAll`
cleanup timeout. All 268 assertions in that group and the three prior
chat groups passed. The failure is retained here; it is not counted as a
full-suite pass. CI passed on `e8a692e1e2` after one workspace-startup
rerun. Fresh CI is pending for the combined head. Its first workspace-a
attempt timed out in the CLI company import/export test's child-server
startup wait; 4,968 UI tests and 287 CLI tests passed in that job. The
same shard passed on the preceding commit. The workspace shard passed on
its one rerun. All other test, build and browser jobs passed. The [first
failure](https://github.com/paperclipai/paperclip/actions/runs/42093823928/job/127478869941)
is retained in the Actions history. Greptile reviewed the preceding head
at 5/5. Fresh review is pending for the combined head.

## Risks

The GitHub classifier matches the documented permission-denial phrase,
not every 403. Rate limits, missing resources and other providers retain
their existing error behavior. Historical normalization affects display
only. A dependency cancellation needs an unstarted run and a supported
queue receipt; current user gates, Retry preparation and recovery remain
authoritative. No schema migration or credential changes.

## Model Used

OpenAI GPT-6, through the Codex agent. The exact runtime model ID and
context-window size are not exposed in this session. Used reasoning,
repository tools, TypeScript, PostgreSQL, MCP fixtures and browser
verification. The parallel agent-recovery commit records Anthropic
Claude Opus 5.5 as co-author.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with version and capability
details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the bug template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change
- [x] I have run the focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open findings
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 20:55:34 +00:00
DottaandClaude Opus 5.5 629abc8509 fix(recovery): keep pending Retry unclickable and scope failure causes (#19268)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - #19260 shows the saved-work recovery step on a task, and records why
a recovery attempt failed.
> - The review of #19260 found three gaps after the PR merged.
> - A scheduled recovery offered Retry while a Retry preparation was
already pending.
> - Classified failures could get a misleading `provider_error` cause.
> - The new cause text in the board handoff had no test.
> - This pull request closes these gaps.
> - The benefit is a Retry button that matches what is allowed, and
causes that name the correct subsystem.

## Linked Issues or Issue Description

Refs #19260

**What happened?**
After #19260, a task with a scheduled recovery and a pending (`issued`
or `processing`) Retry preparation showed an active Retry button. The
server only coalesces the second request. Also, a failure that recovery
classified as `archive_unsafe`, `source_unavailable` or
`snapshot_verification_failed` still recorded a provider cause. The
handoff text could then read "last failure: archive_unsafe, cause:
provider_error".

**Expected behavior**
Retry is not offered while a Retry preparation is pending. Only the
generic transient fallback records a provider cause.

**Steps to reproduce**
1. Request a chat Retry on a task while its saved-work recovery is
scheduled.
2. The task projection returns `retry_scheduled` with `canRetry: true`.

**Paperclip version or commit**
master at b1d00eec.

**Deployment mode**
Authenticated cloud deployment (staging).

## What Changed

- `task-interruption-presentation.ts`: a scheduled recovery returns
`canRetry: false` when a Retry preparation for the same run is pending
with a valid deadline.
- `workspace-recovery.ts`: a cause is recorded only for the generic
transient fallback (`provider_unavailable` or `export_interrupted` from
an unclassified error).
- Tests: presentation `canRetry` with and without a pending preparation;
the board handoff text with and without a cause.

## Verification

- `task-interruption-presentation.test.ts`,
`system-recovery-state.test.ts` and `workspace-recovery.test.ts` pass
(419 tests). One workspace-recovery case failed once under full-suite
load. It passes when it runs alone.
- Server typecheck passes.

## Risks

- Low risk. This change affects presentation and diagnostic labels only.
Retry admission still checks all gates on the server.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 11:14:24 -06:00
DottaandClaude Opus 5.5 01dd6ab336 fix(recovery): show the saved-work recovery step and record why it failed (#19260)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Sometimes a failed run left files in a sandbox. Then Retry first
runs system recovery to save those files before the task continues.
> - Recovery has steps: confirm that the previous attempt stopped, save
its files, then finish.
> - The task page did not show these steps. People saw "Queued" or
"Trying again in…" with a retry time that kept moving.
> - Each unclassified failure of the stop-confirmation step was recorded
only as `provider_unavailable`, so a repeated deferral could not be
diagnosed.
> - This pull request shows the current recovery step on the task. It
also records why each failed attempt failed, in closed labels.
> - The benefit is that people can see what Retry waits for, and
operators can see why recovery deferred.

## Linked Issues or Issue Description

Refs #19249

**What happened?**
A person clicked Retry on a task whose failed run had a retained
sandbox. For about 7 minutes the task showed only "Queued", and the
retry time moved forward twice. Recovery deferred twice in the
stop-confirmation step. Each deferral was recorded only as
`provider_unavailable` with an opaque fingerprint, so the cause could
not be found.

**Expected behavior**
The task shows the recovery step that it waits for, such as "Checking
that the previous attempt stopped. Trying again in 30s…". Each failed
attempt keeps a closed-label cause, for example `sandbox_not_running` or
`transport_timeout`, with an optional errno-style code and HTTP status.

**Steps to reproduce**
1. Get a remote task run to fail after its sandbox was placed, so that
recovery owns saved work.
2. Click Retry while the sandbox provider does not respond to the stop
check.
3. The task shows "Queued" or "Trying again in…". The recovery evidence
records only `provider_unavailable`.

**Paperclip version or commit**
master at 499712c6.

**Deployment mode**
Authenticated cloud deployment (staging).

## What Changed

- `system-recovery-state.ts`: a failed attempt can keep an optional
`failure.cause` with a closed `label`, an errno-style `code` and an HTTP
`status`. The schema drops a value that does not match.
- `workspace-recovery.ts`: an unclassified failure records
`sandboxProviderFailureLabel(error)` as its cause, and the warning log
includes the label. The cause never contains provider text.
- `system-recovery.ts`: the board handoff message after exhaustion names
the last cause.
- `TaskInterruptionPresentation` has an optional `recoveryStep`:
`checking_previous_run`, `saving_previous_files` or
`finishing_recovery`. The server sets it from the recovery phase only
for a scheduled or actively running system recovery.
- `ExecutionBlockerNotice` shows the step. While recovery runs it shows
"Saving files from the previous attempt…". While a recovery retry is
scheduled it shows "Checking that the previous attempt stopped. Trying
again in 30s…". Without a recovery step, the output does not change.

## Verification

- Red, then green: the new tests in `system-recovery-state.test.ts`,
`task-interruption-presentation.test.ts` and
`ExecutionBlockerNotice.test.tsx` failed before the change and pass
after it.
- `workspace-recovery.test.ts` (380 tests) passes. Its exhausted
unknown-failure case now expects `cause: { label: "provider_error" }`.
- An ordinary scheduled retry, a working run without recovery and the
existing failed states render as before.
- Server typecheck passes. UI typecheck passes for `src`. The only UI
errors are in Storybook support files whose generated catalog is not
built locally.
- `pnpm check:token-gates` passes.

## Risks

- Low risk. All new fields are optional JSON fields. No database
migration is necessary, and old records parse as before.
- This change does not change recovery behavior, retry timing or
authority. It adds presentation and diagnostics only.
- The cause label uses the existing closed transport label set. An
unrecognized error is recorded as `provider_error`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:03:25 -06:00
DottaandPaperclip b6352578c5 fix(task-retry): retain chat Retry during identity contention (#18896)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task Retry must preserve a person's request and enforce the current
chat authority.
> - Chat retry preparation takes nonblocking source locks before it can
authorize the saved source.
> - Sustained lock contention could discard the Retry request before it
was saved.
> - This pull request saves one unadmitted request in the existing chat
retry queue.
> - The worker checks the source and current access before normal
admission. The task shows a scheduled retry while it waits.

## Linked Issues or Issue Description

**What happened?**

Task Retry returned HTTP 500 when another transaction held the saved
chat principal's identity lock or the endpoint authorization row. The
source preparation error rolled back the canonical Retry intent. The
same gap existed when Retry had to restore a certified input batch that
had never reached a provider. With the bounded synchronous retries in
#18895, sustained contention returns 409 without a saved intent.

**Expected behavior**

HTTP 202 retains one Retry request. Acceptance does not grant source
authority or mean that a provider started. The existing worker prepares
the request if the source locks become available before its saved
deadline. It must reject changed input, revoked access, Stop, or
superseding work before input adoption, publication, or provider work.

**Steps to reproduce**

1. Prepare a failed task with its original admitted GitHub chat source.
2. Hold that principal's `chat-identity` advisory lock or the endpoint
row on a second PostgreSQL connection.
3. Send the normal task Retry HTTP request. Check the response and
persisted intent.
4. Release the lock. Process the existing chat retry queue and check
that one authorized successor is queued.

**Paperclip version or commit**

Reproduced on master `8771606fd87f69ed4057779e3dce3a9bca8088e6`.

**Deployment mode**

Source checkout, real Express Retry route and embedded PostgreSQL.
External chat transport is a test fixture. The live hosted fix has not
been deployed or verified.

Related: #18895 handles brief admission contention with bounded
synchronous attempts and returns 409 if it remains busy. This change
retains a durable request through sustained identity contention. It uses
the existing background retry owner and adds real HTTP, restored-input,
Stop, publication, restart, and task-presentation coverage.

## What Changed

- Preserve a strict pending-source action only when the identity
try-lock reports contention or the endpoint authorization query reports
lock-unavailable through an explicit Retry callback. Bind it to the
canonical task Retry, company, task, agent, endpoint, conversation, and
principal. Other database and authorization errors keep their existing
behavior.
- Revalidate the canonical intent, current board permission, saved
source, and latest execution before the existing worker can prepare that
action. Ordinary authorization rejects an unprepared action.
- Keep certified input restoration inside source preparation. Recheck
the saved deadline before the prepared payload write. Expiry, a nested
savepoint, or outer rollback cannot commit partial restoration.
- Give source authorization contention one saved 60-second deadline and
a separate counter. Keep its waiting phase until current source
authorization succeeds or the action terminates. It does not spend the
eight delivery attempts. Duplicate clicks, worker restart, and
background sweeps do not renew either limit. Reuse the existing work
notifications and outbox deadlines.
- Persist completion of the authorization wait under the current claim
before delivery. A restart revalidates current authority. Generic
delivery failure preserves the stored phase even when a commit
acknowledgment is lost; it does not grant a wake or renew the saved
deadline, counters, or claim.
- Project pending preparation and prepared but unsettled actions as the
existing scheduled-retry state with the actual outbox deadline. Keep the
exact task, intent, agent, and source binding. Active execution keeps
its normal working state. No UI or API enum change is needed.
- Add regressions and document the admission boundary.

## Verification

- RED: the original real HTTP route returned 500 under a held PostgreSQL
identity lock. The restored-input case also returned 500 while its 60
sibling source and budget cases passed. The current endpoint-row case
returned 409 without a saved intent under the bounded synchronous
fallback.
- RED: the worker failed a valid Retry during an actual nine-second
PostgreSQL lock. Removing only the latest-run guard makes the new
queued-run negative fail. Source preparation could also commit restored
input after its deadline, and an already prepared action could process
after a late lock release. The prepared but unsettled task view lost its
scheduled state.
- GREEN: all 91 selected real PostgreSQL integration cases passed.
Coverage includes the 62-case saved-input matrix; coalescing; restart;
one real heartbeat successor; source, actor, Stop, newer-work and
publication refusals; a real nine-second lock; fixed-deadline
exhaustion; the separate eight delivery attempts; certified restoration
rollback on expiry; prepared-action early and late release; a real
conversation-row wait that crosses the deadline during current
authorization; endpoint-row contention; unknown source; an
already-revoked actor; unchanged unrelated ingress errors; durable-work
idle settlement; actual outbox presentation deadlines; and recovery
after authorization commits, with Stop and a lost commit acknowledgment.
- GREEN: all 75 cases in `task-retry.test.ts`,
`task-interruption-presentation.test.ts`,
`chat-failed-run-retry-boundary.test.ts`, and `chat-action-work.test.ts`
passed without a name filter.
- RED → GREEN: stopping after source authorization committed left a
stale waiting flag, so a fresh worker failed a previously authorized
Retry after the deadline. The fix saves that phase in the same
transaction. A real PostgreSQL commit followed by an injected lost
acknowledgment also reproduced generic settlement restoring the stale
flag. The corrected settlement preserves the stored phase. No wake is
sent on the ambiguous response. Recovery queues once after current
checks; Stop still prevents a successor. The deadline, contention count,
original input, failed run, and publications stay unchanged.
- Module boundary checks pass. Full workspace typecheck and the full
build passed at `d8a4a1a739aa6825714b04691d4166a00991808b`. At this
exact head, both required checks (`ci / verify` and `ci / e2e`) pass.
The inventory has 53 successful, three skipped, and one neutral check,
plus the automatic Paperclip Review in progress. The legacy Snyk status
passes. Greptile is 5/5 on this head, with no new comments or unresolved
threads. A current Paperclip assessment has not been published yet.
- The full local `pnpm test:run` did not pass: its second chat shard hit
a 15-second timeout in the HTTP/restart test after 348 and 293 passing
cases. One bounded diagnostic of the same 642-case shard pair passed,
with that case completing in 606 ms. The matching Linux CI pair passed
too. The original timeout remains unexplained and nonreproduced. No
timeout changed. The earlier stopped run was incomplete, and both
conflicting documentation additions are retained.
- CI chat 3/3 exposed a separate fixture assumption: restoration can
commit before a durable wake receipt exists. One known pre-receipt
delivery failure reproduces the same missing-run error. The corrected
fixture drives the existing worker until the exact canonical intent and
wake agree. It proves one real wake, one successor, exact source/actor
binding, and one restoration. The exact failed shard pair now passes
locally: 635 cases. Workspace typecheck also passes at test-only head
`94d691c9df61b05019b96900d5800a8f1c484e9a`. Production source is
unchanged from the previous verified build. That head completed all
registered checks: 53 successful, two neutral, three skipped, plus the
successful legacy status. Greptile scored 5/5. The subsequent Paperclip
4/5 persistence finding is addressed by the source repair above; its new
exact-head review is pending.
- Compatibility: 111 selected cases passed with #18895 head
`9b5d877161ffcbf16c0fb6920e8155a3620c85b0` in a disposable worktree. The
branch is now synchronized with master `4a83ba4050`, which includes that
change.
- Preserve the old CI failure: at head
`37954951fb66f3b50d0f2a3bec78d6993c479e87`, server shard 6/12 initially
failed the empty organization-audience replacement versus member-removal
assertion in `tool-access-service.test.ts`. The exact race family passed
on unchanged base and head; all 501 file tests and two 30-case controls
passed on base. The one Linux replay passed 119 files and 2,833 cases,
with three skipped. No guard or assertion changed. The initial failure
remains unexplained and nonreproduced; this PR does not claim to fix it.
- The prior external Superagent scan could not start because its
Security Graph returned HTTP 503. Its neutral check was not a completed
security assessment.
- These tests exercise the real Express route, PostgreSQL transactions,
and heartbeat queue. External chat transport is a fixture. They do not
run a live model or prove recovery of an original hosted task.

## Risks

- A still-pending source authorization wait fails after its fixed
60-second contention deadline. A completed wait is saved, but new
contention keeps the original deadline. A healthy lock can exceed that
limit. Other delivery errors retain the eight-attempt budget. Current
authority checks remain required before admission. Inspection does not
reset either limit.
- The new pending phase must never be treated as an authorized chat
source. Normal authorization rejects it until the worker prepares and
binds the exact source.
- The task projection reads the existing linked outbox deadline. It adds
a scoped join and does not create a timer or change a business deadline.
- No schema migration, provider identity change, or new delivery queue
is included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository editing, tool
use, and test execution. The host did not expose an exact deployment
snapshot ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] Required Paperclip CI gates are green (`ci / verify` and `ci /
e2e`; automatic review progress is recorded above)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 07:18:48 -06:00
DottaandPaperclip a80eec1164 fix(recovery): continue Retry after completed file preservation (#18845)
Reuse completed file preservation for task Retry and show bounded admission explanations, including newer saved messages and identifier routes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 16:01:27 -06:00
DottaandPaperclip f7ad1dc7ca fix(workspaces): make attached project repositories writable (#18770)
## Thinking Path

> - Paperclip lets people assign work to AI agents and inspect the
result.
> - A task can use repositories from more than one project.
> - Remote workspace preparation treated every additional project as a
read-only source.
> - This rejected valid Git repositories before the provider started,
even when no source was read-only.
> - The task page then hid the preparation reason behind a generic
failed-attempt notice.
> - This pull request prepares authorized repositories as writable task
copies and shows redacted workspace validation errors.
> - People can work across both projects and see the reason if folder
preparation fails.

## Linked Issues or Issue Description

**What happened?**

A remote task with a selected Git-backed project and another mentioned
Git-backed project failed before provider execution. It reported
`remote_workspace_readonly_sources_unsupported`. Neither project had a
read-only setting. The task notice showed only that the attempt failed.

**Expected behavior**

Each attached repository should be writable inside the task workspace.
This should preserve source access checks and saved work. A workspace
validation failure should explain its cause on the task page.

**Steps to reproduce**

1. Configure remote workspace storage and a native sandbox environment.
2. Select a Git-backed project for a task.
3. Mention another authorized Git-backed project in the task.
4. Start the task. Preparation rejects the additional source before it
starts the provider.

**Paperclip version or commit**

Reproduced against `dc9f387edc`, before this change.

**Deployment mode**

Managed remote workspace storage. Local regression checks use disposable
PostgreSQL and independent filesystem processes.

Related merged work: Refs #18403 and #18504. Those changes established
remote storage and metadata-only references. They still reject
configured cross-project repositories. No duplicate writable-repository
repair was found. Refs #18774 for the test-only retention-fixture repair
reused to unblock CI.

## What Changed

- Add every configured Git repository of an authorized referenced
project to `.paperclip-repositories/` as an editable task copy.
- Retain saved paths, refs, pins and files. Recheck retained project
access and source trust. Keep explicit read-only, local-source import
and remote feature gates.
- Keep committed snapshot repository coverage separate from later
requests. Honor configured default refs, resolve remote branch names,
and publish secondary checkouts only after checkout succeeds. Share
configured-ref resolution between root and attached-repository scripts.
- Include redacted workspace validation messages in the task
interruption projection and existing failure notice. Preserve Retry
permissions and render messages as plain text. Explain retained-folder
access and trust denials while preserving the dedicated low-trust
project-access path.
- Add source, authorization, redaction, UI and save/restore regressions.
Update the implementation contract and workspace maps.
- Reuse the reviewed test-only repair from #18774. Batch the
shared-workspace history fixture, verify its complete privacy-source
map, and drain owned test bodies before cleanup. Production retention
behavior is unchanged.

## Verification

- Red before repair: four source-resolution cases failed with the
reported read-only error. The task-error projection test also failed
because it omitted the cause.
- Green: 49 source and transport tests, 16 task projection and redaction
tests, and 10 UI tests. The assembled preparation check passes. All 48
remote workspace lifecycle tests pass. The full preparation suite passed
all 126 assertions but its cleanup hook exceeded 30 seconds. The
unchanged base reproduces the same cleanup timeout with all 126
assertions passing. The focused preparation check finishes cleanly.
- The real PostgreSQL/Git lifecycle checks cover both first-turn and
later project attachments. They verify the configured release branch,
saved edits, replacement allocations, unchanged source folders,
committed archive contents, and released writers.
- Review regressions were red before repair: the default ref became
null, later attachment failed during restore, non-default remote branch
checkout failed, and invalid secondary refs left partial checkouts. They
now pass.
- The local browser shows the workspace validation explanation and Retry
action. The fixture uses a saved failed run; this proves the error
display, not a successful live sandbox run.
- `pnpm -r typecheck` and `pnpm build` pass. Map, module-boundary and
token checks pass.
- A local two-turn acceptance probe repeated at `17ed67eaa7` used real
Codex CLI 0.165.0, `gpt-6-astra`, and `xhigh` effort. Codex wrote files
in both task repositories. The services saved them, removed the first
allocation and its Git sources, restored another allocation, and Codex
read the saved files and wrote a follow-up in both folders. This uses
the real source resolver and persistence services with a local
command-runner stand-in for sandbox transport. It does not qualify
Daytona, managed subscription forwarding, or the original task. The
first probe waited for stdin; closing the test harness pipe fixed that
setup error before provider work.
- `pnpm test:run` passed the first chat shard, then failed one chat
response concurrency assertion in the second shard. That case passes in
isolation on this head. The same isolated chat case passes on the
unchanged base. Both corresponding unchanged-base chat shards also pass
(351 and 324 tests). The cause of the earlier wider-suite chat failure
is not established. The preparation cleanup timeout and chat failure
remain part of this record.
- CI at `17ed67eaa7`: all other verification jobs passed. Browser shard
6 initially reported a request callback from the preceding test after
teardown; its single unchanged-code retry passed. Server shard 10 timed
out the 2,100-row retention confirmation fixture at 15 seconds on both
attempts. That exact case passes locally, but the repeated CI failure
required a repair. Its setup repeatedly rebuilt a shared-workspace
privacy map. The logs and failures remain part of this record.
- Reused the test-only repair from #18774 without changing its source
file. At `ab774e9418`, all 54 retention tests and all 25 ordinary
production privacy tests pass locally (79 total). Full CI
[41650626903](https://github.com/paperclipai/paperclip/actions/runs/41650626903)
passed on that commit, and its exact-head Greptile review was 5/5. The
fixture preserves its 2,100 rows, query-bound assertions, privacy-source
map, and completed-history test deadlines.
- The additional review identified retained-source access/trust denials
that still bypassed the workspace explanation. Real-resolver regressions
reproduced both plain-403 failures before the correction. Both now yield
a safe workspace validation explanation, while the dedicated low-trust
denial and saved workspace remain unchanged. The optional duplicate
ref-resolution block is also consolidated.
- Final review corrections: 65 source, transport and error-projection
tests passed, followed by all 48 lifecycle cases against an owned
disposable PostgreSQL database (113 total). The initial combined command
skipped the explicitly gated lifecycle suite; the subsequent dedicated
invocation enabled and passed all 48. Full workspace typecheck and build
pass. The first standalone lifecycle launcher lacked the repository
TypeScript loader; rerunning with that loader resolved its import setup
error. At previous head `7be02ea09c4ac76df560e8b517050cdf398f33f0`, full
CI [41654422329, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/41654422329/attempts/2)
passed, Greptile is 5/5 on that exact head, and all review threads are
resolved. The first attempt timed out the unchanged voice-session
report/history case at 15 seconds; 3,027 cases in its shard passed. That
exact case passed locally, then its single unchanged-code shard retry
passed. The wider-suite timeout cause remains unproven. The completed
Paperclip advisory check is neutral; its earlier findings are corrected.
No staging deployment or original-task recovery is claimed.

- Master advanced while checks ran, producing conflicts in two adjacent
test additions. Merged `27d35314be` into this branch as
`d5c8d552939c86056abf8fe10833881b2ade41a4`, preserving both
workspace-error tests and the audited Stop/receipt tests. Kept workspace
maps to minimal corrections under the updated map rule. All 82 source,
transport and interruption cases, all 48 real-database save/restore
lifecycle cases, full typecheck, build, and map checks pass locally on
the combined code. Current-head [CI
41660719214](https://github.com/paperclipai/paperclip/actions/runs/41660719214)
and Greptile are pending; the earlier verified head has not been merged.

## Risks

- Additional remote project repositories are now editable task copies.
Shared source folders and upstream push permissions do not change.
- Snapshot coverage uses existing workspace metadata. Older workspaces
capture their previous inventory at reservation; successful save
publication advances coverage and snapshot identity in one transaction.
No SQL migration is needed.
- A retained copy can contain files from a project that is no longer
mentioned. Preparation rechecks that project's current access and trust
policy before reuse.
- Explicit read-only descriptors still fail. Source-less projects remain
metadata-only. Local folders still require import. No migration or
provider-pack update is needed.
- Error display reads only workspace validation errors. The existing
run-secret registry and username censor run before display truncation. A
redaction failure omits the explanation.
- The retained fixture repair pauses its row trigger only within the
isolated seed transaction, rebuilds the exact source map, and restores
the trigger before commit. A rollback regression and 25 ordinary privacy
cases pass. Its paged-case test budget honors the existing bounded inner
waits; production deadlines are unchanged.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution.
The exact serving model ID and context-window size are not exposed by
this session. The test-only fixture changes are reused from #18774,
whose model record identifies OpenAI GPT-6.1 Sol. The optional live
acceptance probe uses `gpt-6-astra` with `xhigh` effort through Codex
CLI 0.165.0.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 15:33:44 +00:00
DottaandPaperclip 27d35314be feat: connect experimental Cursor Cloud to native Runner (#18069)
## Thinking Path

> - Paperclip helps people manage AI agents at work.
> - Native Runner owns execution and task authority.
> - Operators need an explicit native Cursor Cloud choice.
> - Tools/files need company, task, session and run scope.
> - This connects setup and lifecycle to Runner Core.
> - Legacy agents retain their selected runtime/history.

## Linked Issues or Issue Description

Depends on #18068. Managed-only API placement and API-owned Stop
containment are in #18310. Refs #5664, #11803, #17299, and #17471. No
duplicate native integration was found.

## What Changed

- Add explicit native setup, Test, saved settings and dispatch with
no-prompt account/catalog/callback checks.
- Scope MCP to company, agent, task, session and run through
authenticated HTTPS. Register verified artifacts/work products; preserve
unpriced accounting, completion and budgets.
- Connect Question/Plan continuation, cold handoff and admitted fresh
sessions; recheck waiting admission under the task lock.
- Present audited Stop intent while preserving physical containment
proof; bound typed Stop metadata. Preserve title edits through
preference/navigation readiness.
- Select one live/history representation per run to prevent duplicate
settled cards. Integrate upstream task/read-state/retry/schema contracts
and repair two test fixtures.

## Verification

Core #18068 merged normally as `3910352c923a`. Product head
`5a60f541c982` has passing repository CI/build/browser checks, including
required `ci / verify` and `ci / e2e`, plus a fresh exact-head Greptile
5/5 review with zero unresolved threads. Two non-required external app
checks remain pending: Paperclip Review is queued waiting for its
assigned agent; Superagent Security Scan is reviewing the diff. These
are not claimed completed. The normal merge endpoint enforces the
repository rules; no policy or status override is used. The dependent
Top PR is addressing its fresh deferred-Stop review finding.

Product `5a60f541c982` passed 270 UI and 20 selected
Cloud/native/disk-full tests, UI/server typechecks, tokens and maps.
Top’s clean normal merge passed tokens/maps/diff checks; its matrix is
pending. Earlier receipts retain Core’s 47 Rust, 4 compiled-native
retirement and 19 settlement controls, normal Runner build, and runtime
9af build/recursive typecheck. Reproduced loading/shutdown CI fixture
failures passed 7/7 browser and 121/121 unit tests, including a
broken-unsubscribe negative control. No timeout/oracle was relaxed.

The actual-parent restack adopts 75 mainline paths, including
schema/permission and generic runtime changes, beyond the prior two
fixture repairs and client projection fix. The full difference from
hosted `9af1a5de42a2` is therefore larger than four files; backend
source also differs. The merged config-read checks match Core391, and
the Cursor placement/runtime/relay/selection modules match frozen9af;
selected affected regressions pass. Paid proof keeps its actual 9af
source/image identity. The separately tested live/history card fix is
not claimed deployed on that host.

On 9af, original saved settings survived reopening and fresh no-prompt
Test authenticated catalog/model/binding; no new Save is claimed. Fresh
Linux admission passed. With `claude-sonnet-4-5`, Artifact downloaded
exact 27 bytes and matched 121 canonical identities in complete
S3-served history. First Stop’s strict quiet FAILED on a 10.373-second
gap despite independent cancellation/no late artifact. One targeted
unchanged-sampler retry retained all 24 samples: 35.076 seconds quiet /
maximum gap 4.584 seconds, 67 ordered canonical identities, and separate
CANCELLED/artifact-absent status after the original 180-second deadline.
The pre-reload duplicate settled card is disclosed. Exact-ID cleanup for
the three new provider agents is awaiting authorization, so full grades
and the fresh Artifact’s post-cleanup download remain blocked. A
historical deleted-provider artifact was newly downloaded on 9af;
original F226 Question/Plan/reconnect and cleanup proof keep their
identities.

Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook
failures; Mac 17 SQL failures/skips; earlier
title/startup/custody/Stop/deletion failures with unknown causes/bounded
recoveries; `dea023` Stop’s warning/50 duplicate identities; pre-repair
ACK and original Core CI failures. WSP’s 251-assertion, 7/8-suite
attempt failed physical PostgreSQL teardown; the unchanged wrapper rerun
passed 128. Retry alone establishes no cause or full local-suite pass.
All paid attempts remain recorded; vendor dollar cost is UNKNOWN.

## Risks

Experimental: Cursor independently acknowledges neither the effective
model nor instruction digest before the first prompt. Explicit values
are submitted and checked through behavior; unpriced provider dollar
cost stays UNKNOWN.

Callbacks need authenticated reachable HTTPS. Scope, revocation,
budgets, completion and cleanup remain authoritative. Operator Stop
intent is separate from physical containment. Saved legacy agents retain
their selected runtime. Prior shared title-save/deletion failures remain
disclosed.

## Model Used

OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools,
code execution, browser verification, and delegated code review. The
serving model’s exact API ID and context window are not exposed. No
value is inferred from the model catalog.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the limits of available model
metadata
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public issues or described the issue in this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch describes the change and contains no internal ticket ID
- [x] I have run the scoped local tests reported above; pending gates
are stated separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All required CI gates pass on the current head
- [x] Fresh Greptile is 5/5 with no unresolved findings on the current
head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 09:01:39 -06:00
DottaandPaperclip b22ef22d56 fix(runner): show provider failure details on tasks and runs (#18622)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner starts providers and records their run outcomes.
> - Provider failures can contain useful details below a generic
protocol error.
> - Those details were lost at the sidecar, command receipt, and run
projection boundaries.
> - A blocked task therefore showed a generic failure even when the
provider gave a clear reason.
> - This pull request carries a bounded, sanitized provider failure
through those boundaries.
> - Operators can read the actual cause on the task and run pages.
Existing recovery policy still controls retries.

## Linked Issues or Issue Description

Refs #17543 for the related closed JSON-RPC classification work. Refs
#17272 for provider recovery in the legacy OpenCode adapter. This change
adds display evidence for native Runner providers and keeps recovery
policy separate.

**What happened?**

A native Claude session returned an HTTP 429 message below an ACP
`Internal error`. The task and run showed only a generic startup
failure. Operators had to inspect the provider response to learn that
the account was rate limited.

**Expected behavior**

The task and run show the sanitized provider message, provider name,
startup or turn phase, and available HTTP status. Unknown categories
keep the message. Display evidence does not grant a retry or replace a
recovery decision.

**Steps to reproduce**

1. Start a native Claude run whose account is rate limited during model
selection, or return a structured provider failure from a Codex or
OpenCode turn.
2. Let the run fail and open its task page.
3. Before this change, the page shows a generic failure. With this
change, the page shows the provider explanation. The run page shows the
same explanation after reload.

**Paperclip version or commit**

Source checkout based on `f8a888a072`.

## What Changed

- Add one JSON Schema for `ProviderFailure` and generate matching Runner
and shared TypeScript readers. Keep the Runner package independent of
the app workspace at runtime.
- Normalize known ACPX, Codex, and OpenCode error fields. Preserve
unknown messages. Redact known credential values and credential patterns
before applying the 4,000-character limit. Do not copy response bodies,
headers, stacks, or prompts.
- Carry startup failures through the sidecar, Rust Runner, durable
command receipts, and private transport error details. Keep provider
text out of generic transport error strings and Sentry failure messages.
- Keep turn failures in authenticated run events. Recover the same
evidence from persisted events during server finalization. Store the
cause in the company-scoped run result.
- Show one shared notice on failed tasks and run details. Keep existing
retry permissions, scheduled retry state, and resolved-task behavior.
- Reapply registered run-secret and current-user redaction to task
display data. Batch secret lookups for only the visible provider causes.
- Document the contract and update the capability maps. Add regressions
for nested Claude 429 errors, OpenAI quota and rate limits, OpenCode
errors, redaction, durable receipt replay, restart finalization, and UI
rendering.

## Verification

- `pnpm -r typecheck` passed. `pnpm build` passed. All GitHub Actions
build, test, package, and validation jobs passed on `d49c79ab4a`. The
security checks passed. Greptile gave this head 5/5 with no open review
threads. Paperclip Review remains pending. Its review worker exhausted
six workspace recovery attempts with export_interrupted. A fresh review
request was accepted and is waiting for that saved workspace. The
ordinary Retry action rejected reuse of the stopped request. This is an
external review-worker blocker; no check was overridden. The duplicate
local `pnpm test:run` was stopped after the CI test shards passed.
- Focused server settlement, provider evidence, and restart tests: 93
passed. Runner admission and normalization tests: 103 passed. Focused
Runner wire and terminal tests: 29 passed. The real Rust/sidecar command
receipt replay test passed.
- The new projection test failed before implementation because the task
had no provider cause. The startup producer regression also failed
before extraction moved ahead of the generic error-code replacement.
Both pass with this change.
- Live check: a rebuilt local sidecar received an actual Claude HTTP 429
during startup. The new cause kept the full provider message. A
synthetic task in an isolated local app then reached the same provider
error through the native Runner path. The dedicated run result, task
API, task banner, and run detail all showed the same message after
reload. The later reporting fix keeps run.error generic; a regression
proves the display cause still survives restart and stays out of Sentry
capture.
- The first isolated app attempt failed earlier in
`admission_credential`. An ordinary manual Retry reached the provider
and produced the 429 above. This PR does not claim to fix that separate
admission failure. An initial parallel local database test setup also
failed; the serial restart suite passed.
- Security review found that copying display text into run.error could
forward it to Sentry. A regression failed with that behavior. The fix
keeps the cause in its display field. All 38 settlement/restart tests
and the server typecheck pass.
- All 29 protocol contract tests pass, including bounds and extra-field
rejection. An earlier full Rust core run passed. The later full run hit
one 5-second receipt-limit timing assertion; its isolated rerun passed.
The full Linux and Windows Rust CI jobs passed on this head.
- Review regressions pass for explicit Codex driver identity, native
Rust startup and resume receipt propagation, oversized OpenCode/ACPX
errors, task read-time registered-secret redaction, and the standalone
source closure. The focused Runner run passed 103 tests;
projection/restart tests passed 16.
- Read-time redaction edge cases passed: a missing decryption key omits
only the unsafe optional explanation; expanded redacted messages remain
visible within the Unicode character limit. All 14 task projection tests
and server typecheck pass.
- The package Build check found an old Pi assertion against
Error.message. The probe now checks the exact closed classification in
the authenticated receipt and requires the new provider cause. Both
polling transport cases pass. A local credential-free Pi probe stopped
at admission_installation because this checkout lacks the Pi
distribution; the installed Linux package probe passed in CI on this
head.
- UI coverage includes startup and turn text, unknown categories,
literal rendering of provider text, read-only retry behavior, and
resolved-task suppression. Token gates and map checks passed.
- The live check covers Claude startup on macOS. Codex, OpenCode,
terminal failures, and restart paths have deterministic regression
coverage. The change is not deployed.

## Risks

- Provider messages are untrusted text. The boundary selects specific
fields, redacts credential patterns and known environment secret values,
bounds the message, and renders plain text. New provider-specific secret
formats may need added redaction patterns.
- Optional fields preserve compatibility with older producers and stored
runs. Old runs without the field keep their current fallback text. No
database migration is required.
- The cause stays in company-scoped run data. This change does not send
provider text to first-party Telemetry or add it to OpenTelemetry span
attributes.
- Normalized categories and retryability are display evidence. Existing
recovery classifiers, retry limits, ownership checks, and approval gates
keep their authority.

## Model Used

- OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and browser verification. The host does not expose the exact
serving variant or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 06:13:26 -06:00
DottaandPaperclip d0c8f0b090 fix: explain project access denials and add an authorized repair path (#17509)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Low-trust agents may read only the projects in their allowed scope.
> - Workspace setup correctly stops a task outside that scope.
> - The task previously showed a generic failure, and agent settings did
not offer an additive repair.
> - This pull request explains the denial and lets authorized operators
add projects while preserving other restrictions.
> - The saved task then offers an explicit retry. Saving access does not
start work.

## Linked Issues or Issue Description

**What happened?**
A project access denial appeared as an ordinary failed attempt.
Operators could not add a second allowed project from agent settings.

**Expected behavior**
The task should explain the denied access and link to a working
permission editor. The editor should preserve existing scope and
restrictions, enforce operator authority, and retain drafts after a
failed save.

**Steps to reproduce**
1. Assign a low-trust agent to a task whose project is outside its
allowed projects.
2. Let workspace setup deny project access.
3. Open the task, follow Manage project access, add the project, save,
reload, and return to the task.
4. Confirm that the saved message remains and Retry task is a separate
action.

## What Changed

- Record a typed, non-retryable project denial before provider dispatch.
Project denials do not enter automatic continuation or provider comment
policy.
- Project current and historical setup denials into a safe task notice.
Keep unrelated failures, prior assignees, and approval/pause/budget
gates separate. Recheck current project authority, including standard
trust and root-issue grants. Accepted retries keep their working state.
- Add board-only project-access reads and additive saves under the agent
row lock. Enforce company scope, agent-management authority,
pending-approval restrictions, and an audit record for new additions.
- Add the production notice and searchable multi-project editor,
including read-only, saving, cancellation, and failed-save states.
Preserve the return-to-task query through canonical agent redirects.
- Register board-only project-access routes in OpenAPI with a strict
request schema.
- Add production Storybook examples and API/persistence/concurrency
regressions. Retain the approved design snapshot and update the behavior
maps and implementation contract.

## Verification

- Passed: `pnpm -r typecheck`, `pnpm build`, `pnpm build-storybook`,
`pnpm check:token-gates`, `node scripts/check-maps.mjs`, and `node
scripts/storybook-catalog.mjs`.
- Passed: 11 permission/projection tests and 30 run-completion tests
against an isolated PostgreSQL database. These cover historical
failures, concurrent additions, idempotent audit, preserved
restrictions, foreign projects, read-only users, agent self-grants, and
no automatic wakeup.
- Passed: all 27 OpenAPI route tests, including route coverage and the
new board-only request/response contract.
- Browser: the real app displayed a historical denial; Manage project
access opened the editor; saving retained the original project and
survived reload; Return to task showed the ready notice. Saving created
one audit event and zero runs. Explicit Retry returned 202 and reached
the separate sandbox requirement, which remained enforced. No external
provider was used.
- All five production Storybook interaction checks pass: blocked task,
save, cancellation, failed-save recovery, and read-only settings. These
use fixture callbacks; durable authorization is covered separately
above.
- `pnpm test:run` was attempted locally but stopped after repeated
embedded PostgreSQL startup failures caused by macOS semaphore
exhaustion. Full-suite CI is required before merge. The separate UI
suite passed all 9,095 tests. The 41 focused server regressions also
passed after rebasing onto the latest master. A further 26
recovery-classification tests pass, including a red-to-green regression
for automatic continuation after a project denial.

## Risks

- No schema migration. The new endpoint only adds same-company project
IDs and preserves the rest of the stored policy.
- The historical notice recognizes one exact setup denial message. It
does not display arbitrary run errors or disclose a different source
project's identity.
- The retry still passes all current admission and isolation checks. The
notice does not promise that other execution requirements are satisfied.
- Related open work: #13309 covers inherited secret restrictions; #11234
covers the broader repository catalog. This change does not replace
either policy surface.

## Model Used

OpenAI Codex (GPT-6), with code execution and browser tools. The precise
runtime model ID and context-window size were not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused server and full UI
suites; full-suite infrastructure limit above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-24 16:16:29 -07:00