Commit Graph
8420 Commits
Author SHA1 Message Date
scotttong 856f14e507 fix(dashboard): scope areas and responsibilities to each user (#22513) 2026-11-21 13:59:00 -08:00
DottaandClaude Opus 5.5 558dbb0132 test(restart-chaos): grade hosted runs with R1 and a strict release-swap gate (#22490)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos`) proves that agent
work survives a control-plane release swap
> - Local waves grade each task with rules R1–R6. R1 is the time budget,
with credit for a provably fair queue wait (#22446, #22452)
> - Hosted mode (`hosted.ts`) runs the same proof on a real test staging
instance that takes automatic releases
> - Hosted mode passed a run when every task finished hands-off. It did
not apply R1, and banners, failed health probes, duplicates and the swap
shape were only reported
> - The next acceptance gate needs 10 consecutive hosted releases with
100% hands-off, 0 Retry banners and 0 failed health probes, so the exit
code must enforce these rules
> - This pull request adds hosted R1 and a strict "Gate D" verdict to
the hosted grade
> - The benefit is that one hosted run gives a single pass or fail with
the same R1 rule as the local waves

## Linked Issues or Issue Description

Refs #22446
Refs #22452
Refs #21153

**What happened?**

A hosted run (`hosted.ts --hosted-releases N`) exited 0 when every
started task finished without a person. The run did not grade R1. A
Retry banner that cleared by itself, a failed `/api/health` probe, a
duplicated agent comment, a stop-then-start release or a release with no
live run did not fail the run. These were only lines in `SUMMARY.md`.

**Expected behavior**

A hosted run passes only when every task finishes hands-off and inside
the R1 budget (with the same fair queue-wait credit as the local waves),
no Retry banner shows, no health probe fails, no work is duplicated, no
release is stop-then-start, and each release carries at least one live
run.

**Steps to reproduce**

1. Run `hosted.ts --hosted-releases 1` against a test staging instance
whose deploy is stop-then-start.
2. Let all tasks finish by themselves after the outage.
3. See exit 0, although health probes failed during the release.

**Paperclip version or commit**

`95ba0a7de6` (master)

## What Changed

- `hosted-gate.ts` (new): `hostedR1` builds the R1 input from API rows
and calls `gradeR1` from `grader.ts`. It uses the staging-mix budget
(`stagingMixBudgetMs`), outage time only for a release whose health
probes failed before the new build answered, and the fair queue-wait
proof (`queueWaitFairnessFrom`) from the company run list and each
agent's `maxConcurrentRuns`. A follow-up run's wait is not credited,
because the API has no wake evidence. `hostedGate` lists seven checks
and passes only when all hold.
- `hosted-readers.ts`: run rows keep `invocationSource`, `retryOfRunId`
and the wake comment ids.
- `hosted-grade.ts`: grades every started task with R1, adds the `Gate
D` block to `SUMMARY.md`, adds `gate` and `r1` to
`hosted-release-swap.json`, and returns the gate as the pass.
- `hosted-main.ts`: passes the task records to the grade.
- `README.md`: documents the gate and the API proxies for per-release
hand-off and reach.

## Verification

- `npx vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/hosted` passes (8 files, 56 tests). The new
`hosted-gate.test.ts` covers R1 pass and fail, a fair and an unfair
queue wait, outage credit, Retry and open tasks, and each gate check.
- `npx tsc -p tests/restart-chaos/tsconfig.json --noEmit` is clean.
- No hosted run was started for this change. It changes only how a
finished run is graded.

## Risks

- Low risk. The change is in the test harness only.
- Hosted runs that passed before can now fail on R1, a banner, a failed
probe, a duplicate, a stop-then-start release or an unloaded release.
This is intended.
- A follow-up run's queue wait gets no credit on hosted runs, so R1 can
be stricter than in local waves when follow-ups queue for a long time.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1121.0-canary.841900
2026-11-21 07:33:25 -06:00
f434f0e7bd fix(native): let a fresh replacement Runner publish its verified pack admission (#21498)
A new claim's own fresh Runner selection now supersedes the earlier claim's admission receipt while no provider artifacts are retained under it, so a refused reattachment no longer fails before the first provider turn as native_provider_artifacts_admission_unverified.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1121.0-canary.841800
2026-11-21 04:28:57 -08:00
21f4427b78 fix(server): pass a Cloud portfolio rate limit through as 429 (#22488)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A hosted stack reads its tenant portfolio from Paperclip Cloud
through `GET /api/cloud/stacks`
> - Paperclip Cloud can answer that read with 429 when it throttles a
stack
> - The proxy changed every non-OK upstream status into a 502, and the
error handler reports every 5xx to Sentry
> - So Cloud backpressure showed up as a server crash on a staging stack
(`upstreamStatus=429`, `phase=http_response`)
> - This pull request passes an upstream 429 through as a 429 with a
bounded `Retry-After` hint
> - The benefit is a correct status for clients and no false crash
reports for throttling

## Linked Issues or Issue Description

Refs #15340 (bounded Cloud portfolio diagnostics), Refs #19033 (retry
one Cloud portfolio connection reset).

**What happened?**
A hosted staging stack reported "Paperclip Cloud portfolio request
failed" to Sentry. The diagnostics show `upstreamStatus=429` after 44
ms. Cloud throttled the read, and the proxy answered 502 and reported a
crash.

**What did you expect to happen?**
The proxy tells the client that Cloud is rate limiting the read. It does
not report a server fault.

**Steps to reproduce**
Make the Cloud tenant portfolio endpoint answer 429, then call `GET
/api/cloud/stacks`. Before this change the route answers 502
`cloud_portfolio_upstream_error` and reports to Sentry.

## What Changed

- `server/src/routes/cloud.ts`: an upstream 429 now gives 429
`cloud_portfolio_rate_limited`.
- The route copies Cloud's `Retry-After` as whole seconds. It accepts
delta seconds or an HTTP-date and clamps the value to 1–300 seconds. It
omits the header when the value is missing or malformed.
- The route logs the bounded diagnostics at warn level. It does not
report the 429 to Sentry and does not cache the throttled answer.
- 401, 403, 503 and the other failure phases keep the 502 contract and
the crash report.
- `server/src/__tests__/cloud-routes.test.ts`: new cases for the 429
status, the `Retry-After` forms, no Sentry report, and no cache entry.
- `docs/api/cloud.md` (new, in the API Reference nav): documents `GET
/api/cloud/stacks`, its error codes, and the 429 with its bounded
`Retry-After`.
- `server/src/routes/openapi.ts`: the `/api/cloud/stacks` entry now
lists the 429 response.

## Verification

- Red: with the old `cloud.ts`, `cloud-routes.test.ts` gave 5 failed
(the new 429 cases) and 23 passed.
- Green: `cd server && npx vitest run src/__tests__/cloud-routes.test.ts
src/__tests__/sentry.test.ts src/__tests__/cloud-portfolio-error.test.ts
src/__tests__/run-failure-sentry-real-sdk.test.ts` gave 68 passed and 18
skipped.
- `npx tsc --noEmit -p server/tsconfig.json` shows no errors in the
changed files.
- After review fixes (6a8f146c55): `cd server && npx vitest run
src/__tests__/cloud-routes.test.ts src/__tests__/openapi-routes.test.ts`
gave 80 passed.

## Risks

- Low risk. Only the 429 path changes.
- The two UI consumers (`useCloudInviteUrl`, `CompanyAccess`) treat any
error the same way, so the UI needs no change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
canary/v2026.1121.0-canary.841700
2026-11-21 04:22:41 -08:00
95ba0a7de6 fix(ui): keep translated pages from crashing on React DOM updates (#22463)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI is a React app, and the browser error monitor reports
render failures from its route error boundary
> - Some people read the board with browser page translation (Chrome or
Google Translate) turned on
> - Translation replaces React-owned text nodes, or moves them into
`<font>` wrappers. React does not support this (facebook/react#11538)
> - The next React commit that removes such a node, or inserts before
it, gets `NotFoundError` from `removeChild` or `insertBefore`. The route
error boundary then replaces the page
> - Three production error reports show this exact shape. Each has the
existing `translation_marker: true` diagnostic, so the cause is
confirmed and not a guess
> - This pull request adds a narrow DOM guard that does what React asked
for only on a translated page
> - The benefit is that translated pages stay usable, and real DOM
errors still reach the error monitor

## Linked Issues or Issue Description

Refs facebook/react#11538

**What happened?**
A person used the board with browser page translation turned on (locale
`es-419`). After the page changed, the route error boundary showed an
error. The error monitor recorded `NotFoundError: Failed to execute
'removeChild' on 'Node': The node to be removed is not a child of this
node.` A second report from the same kind of session recorded the
`insertBefore` form of the same error.

**Steps to reproduce**
1. Open a board page that updates its text, for example a task whose
status changes.
2. Use the browser to translate the page into another language.
3. Let the page update so that React removes or moves a translated text
node.
4. The route error boundary replaces the page.

**Expected behavior**
The page keeps working while it is translated.

**Environment**
Hosted board, production build, Chrome page translation active
(`translated-ltr` class on `<html>`).

## What Changed

- Add `ui/src/lib/translation-dom-guard.ts`. It wraps
`Node.prototype.removeChild` and `Node.prototype.insertBefore`. The
wrapper acts only when both conditions are true: the page has the
translation marker class, and the node is detached or is still inside
the parent. In that case it removes the node from its real parent, or
inserts before the translation wrapper. A `MutationObserver` records
which `<font>` replaced each detached text node. For a `replaceChild`,
that is the added `<font>`. For an insert followed by a remove, it is
the `<font>` inserted next to that text node. The guard uses it only if
it is still next to the removed text, so it never takes a neighbor's
translation. When React removes a detached text node, the guard also
removes that translated replacement, so the old text does not stay on
the page. When React inserts before a detached node, the guard inserts
before its replacement, or appends if there is none. All other calls go
to the native method without change.
- Install the guard in `ui/src/main.tsx` before the first render. Only
the first call installs it, and its stop function restores the native
methods.
- Move the translation-marker check in `browser-error-context.ts` into
an exported `hasPageTranslationMarker()` helper. The guard and the error
diagnostic now read the same signal. The diagnostic output does not
change.
- Add `translation-dom-guard.test.tsx`. It renders real React in jsdom
and does what translation does to a text node. This includes a replace
in one step, an insert followed by a remove, and two adjacent text nodes
translated right to left and left to right.

## Verification

- Red: with a no-op guard, the new tests fail with the same
`NotFoundError` that production recorded. Against the first version of
this guard, the three replacement tests fail (the translated text stayed
on the page). Against the second version, the right-to-left two-node
test fails (the neighbor's translation was deleted).
- Green: `cd ui && npx vitest run src/lib/translation-dom-guard.test.tsx
src/lib/browser-error-context.test.ts src/lib/sentry.test.ts
src/lib/sentry-package-version.test.ts` passes (4 files, 66 tests).
- The tests also prove that the native errors stay on a page that is not
translated, and for a node outside the parent on a translated page.
- `tsc --noEmit -p ui/tsconfig.json` shows no errors in the changed
files. The local run had only the UI dependencies installed, so
unrelated packages that were not built (plugin SDK, runner protocol)
showed errors that were already there. Required CI does the full
typecheck, test, and build.
- I did not run a real Chrome translation session. The jsdom test makes
the same DOM change that translation makes.

## Risks

- The guard changes two global DOM methods. Untranslated pages pay one
extra property comparison per call. The translation check runs only when
the parent does not match.
- The `MutationObserver` watches child-list changes on the whole
document. Its callback only checks records that remove one text node,
and it keeps replacements in a `WeakMap`, so detached nodes can still be
garbage collected.
- On a translated page, a real bug that removes a detached node, or that
inserts before a node inside a nested element, is no longer reported.
This is limited to translated pages, which already showed this crash.
- If translation detaches a node without a `<font>` replacement that the
observer saw, removal leaves the page as it is, and an insert goes at
the end of the parent. The page does not crash.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent workspace.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1121.0-canary.841600
2026-11-21 03:57:16 -08:00
683d7ae120 fix(ui): name the plugin whose UI module raised a browser error (#22487)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The browser UI loads plugin UI modules. The host fetches each
module, rewrites its imports, and imports it from a random `blob:` URL
> - Browser errors go to Sentry. A stack frame from a plugin module
names only that random blob URL
> - So a Sentry report cannot tell which plugin raised an error. For
example, a plugin can abort its own request during an unmount, and the
report shows only `blob:https://<host>/<uuid>`
> - This pull request records each blob URL with its plugin key, and
tags the Sentry event with that plugin key
> - The benefit is that a plugin error can go to the correct plugin
owner, and the blob URL is never sent

## Linked Issues or Issue Description

Refs #22380, which marks the announcement card's own aborts as
intentional. Some events in the same Sentry group come from a plugin
module instead, and #22380 cannot cover them.

**What happened?**
Hosted browsers report `AbortError: signal is aborted without reason` as
an unhandled rejection. The innermost frame is a React effect cleanup in
a `blob:` module at line 5590. Only plugin UI modules run from `blob:`
URLs in the main thread. The event has no data that names the plugin.

**Expected behavior**
A Sentry event from a plugin module names the plugin, so the plugin
owner can fix the plugin.

**Steps to reproduce**
1. Install a plugin with a UI slot whose effect cleanup calls
`controller.abort()` on a pending `fetch` and does not catch the
rejection.
2. Open a page with that slot. Then go to a page without it.
3. Sentry records an unhandled `AbortError` whose frame is a `blob:` URL
and no plugin key.

## What Changed

- `ui/src/lib/plugin-module-origins.ts`: a new, bounded registry (100
recent modules). `rememberPluginModuleUrl` records a blob URL with its
plugin key. `readPluginModuleOrigin` finds the innermost `blob:` frame
of an event and returns its plugin key, `unmapped`, or nothing. It never
throws.
- `ui/src/plugins/slots.tsx`: `importPluginModule` records the blob URL
with the plugin key before the import.
- `ui/src/lib/sentry.ts`: `beforeSend` sets the `browser_plugin_module`
tag when an event has a `blob:` frame. The tag holds the plugin key (at
most 200 characters) or `unmapped`. It never holds the URL.
- Tests for the registry, the loader, and the Sentry hook.

## Verification

- `cd ui && npx vitest run src/lib/plugin-module-origins.test.ts
src/lib/sentry.test.ts src/plugins/slots-loading.test.tsx
src/plugins/slots.test.ts src/hooks/useAnnouncement.test.tsx`: 69/69
pass.
- Red first: with the source changes removed and the new tests kept, the
Sentry test and the loader test fail (`expected 'unmapped' to be
'fixture.navigation'`).
- `cd ui && npx vitest run src/lib src/plugins`: 279 files, 3231 tests
pass.
- `npx tsc --noEmit -p ui/tsconfig.json`: no errors in the changed
files.

## Risks

- Low risk. The change adds one tag. It does not drop, group, or change
any event.
- The tag holds a plugin key, which is a manifest identifier. It holds
no URL, path, or user data.
- The registry keeps at most 100 entries, so memory stays bounded when
modules reload.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution,
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1121.0-canary.841500
2026-11-21 03:26:38 -08:00
4631d3b41a fix(workspaces): attach a recovered saved version once, not once per failed run (#22484)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a run starts in a new sandbox, Paperclip prepares the task
workspace. It also attaches the saved files of earlier failed runs under
`.paperclip-recovered/<operation>` so the agent can continue their work
> - A run that restores a saved workspace and then fails can save the
same version again. Workspace recovery keeps a receipt for that run. One
saved version can then have a receipt from each of several failed runs
> - Preparation attached the files once for each receipt. It removed
duplicates only by receipt operation ID
> - On staging, six receipts named the same 1 GB saved version. Each new
sandbox received six copies (about 6 GB and 171k entries) before its
first save. The first save then failed with
`remote_workspace_sandbox_disk_full`. The read-only fallback also
refused with `budget_exceeded`, because its inventory limit is 100,000
entries
> - The next sandbox received the same six copies. No run could succeed,
so nothing retired the copies, and the task failed in the same way about
every 30 minutes
> - This pull request attaches each saved version once
> - The benefit is that a task with several recovered failed runs gets
one copy of their saved work, and its sandbox has space for its first
save

## Linked Issues or Issue Description

Refs #22466 (names this failure `sandbox_disk_full`). Refs #22383,
#22239, #22068 (other full-sandbox-disk handling). Those changes name
the failure and stop repeats. This change removes the cause of this
case.

**What happened?**
On staging, three tasks failed during setup again and again with
`remote_workspace_sandbox_disk_full: the sandbox disk is full; free
space in the sandbox before saving the workspace`. Each run used a new
sandbox. One workspace failed its initial save 18 times in about 8
hours. Its metadata recorded six recovered attachments. All six came
from one earlier workspace, one snapshot and one sha256 (1,040,502,418
bytes, 28,576 entries). Only the source run was different. Each of those
six source runs had restored that snapshot, saved it again unchanged
(`reusedSnapshotId` equals the parent), and failed for another reason.

**Expected behavior**
The new workspace gets one copy of the saved version. Its first save has
space.

**Steps to reproduce**
1. Save a workspace (version A).
2. Start a writer that restores A, saves A again unchanged, and fails.
Record its recovery receipt.
3. Do step 2 again with a second writer.
4. Prepare a new workspace with both failed writers as recovery
predecessors.
5. See two full copies of A under `.paperclip-recovered/`.

## What Changed

- `remote-workspace-recovery-attachments.ts`:
`attachPreservedRemoteWorkspaceSnapshots` first collects the verified
receipts. It then attaches each saved version (source workspace,
snapshot ID, sha256) once, under the receipt of the earliest source run.
If any receipt of a version is retired, the version is retired. Receipt
verification, access checks, and the provenance of an attached copy do
not change.
- `remote-workspace-snapshot-reuse.test.ts`: a new database-backed test
uses the existing reused-save recovery fixture with two failed writers.
The `same` case expects one copy. The `changed` case (the second writer
saved different files) expects two copies. Both cases check that a
retired receipt retires its whole version. The recovery stamping moved
into a helper (`recoverFailedWriter`) for both writers.

## Verification

- Red: on unmodified `master`, `pnpm vitest run
src/services/remote-workspace-snapshot-reuse.test.ts -t "attaches each
saved version once"` fails for the `same` case: `expected [ {…}, {…} ]
to deeply equal [ ObjectContaining{…} ]`, which is two copies of one
version. The `changed` case passes.
- Green: with the change, `pnpm vitest run
src/services/remote-workspace-snapshot-reuse.test.ts
src/services/workspace-recovery-preparation.test.ts
src/services/remote-workspace-recovery-receipt-fallback.test.ts` passes:
3 files, 125 tests. These include the existing same-run duplicate check,
retired copies, unverifiable receipts, and storage backend selection.
- `node scripts/typecheck-layers.mjs` in `server/`: no errors in
`server/src`. My sandbox installed only the server dependencies, so
`cli/`, `ui/` and `packages/kv-demo-mcp-server` reported missing-module
errors there. CI runs the full typecheck.
- I did not run a live sandbox preparation. The staging workspace
records above give the sizes and the six duplicate receipts.

## Risks

- Low. Only duplicate copies of one saved version are skipped. Different
versions are all attached, as before.
- The agent sees one recovered folder for that version instead of
several identical folders. The recovered-files list is context for the
agent only. No settlement step needs one copy per source run.
- Existing workspaces keep their old records. In a new sandbox, the
records of copies that are not attached again are dropped, as for any
copy that is absent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1121.0-canary.841400
2026-11-21 03:07:06 -08:00
Devin FoleyandPaperclip 1b1861c29d Fix skill upload conflicts blocking first-agent onboarding (#22477)
## Thinking Path

> - Paperclip manages agents and the skills they need to work.
> - First-agent setup resolves the bundled skills before onboarding can
finish.
> - Skill packages use durable upload claims to protect their
object-storage receipts.
> - A failed attempt kept its claim active for ten minutes. A concurrent
caller received an immediate conflict.
> - This pull request expires a failed owner's unpublished claim and
waits for an active upload to settle.
> - Onboarding can retry skill setup without a false upload-in-progress
error.

## Linked Issues or Issue Description

**What happened?**

First-agent onboarding could show "This skill package is being saved.
Retry after its upload settles." during the subscription connection
step. The agent's bundled skill setup called the package store. A failed
upload retained its active claim, and calls from another database client
also returned a conflict while an upload was in progress.

**Expected behavior**

A failed attempt permits an immediate retry with new object keys.
Concurrent callers reuse the completed receipt or claim the failed
attempt. They preserve another owner's active lease and every published
receipt.

**Steps to reproduce**

1. Fail the first object upload while resolving the first agent's
bundled skills.
2. Retry the same skill selection before the ten-minute lease expires.
3. Observe the upload-in-progress conflict. The new regression
reproduces this through
`companySkillService.resolveRequestedSkillEntries`.

**Paperclip version or commit**

Reproduced on `6bf312270c3e`.

**Deployment mode**

Hosted onboarding; the package-store fix also applies to self-hosted
instances.

Related implementation: #18082. Searched open and closed pull requests
and open issues for related skill-upload and lease failures. No
duplicate fix was found.

## What Changed

- Expire only the failed attempt's token-matched, unpublished upload
claim. Keep its exact keys for the existing deletion outbox. Cleanup has
a one-second client deadline and a transaction-local statement timeout.
Queued cleanup skips its write after that deadline; failed cleanup
retains the original crash lease.
- Wait for a competing package preparation within the existing
three-minute deadline. Re-read the receipt every 250 ms while this
concrete request remains active. Release the database connection between
reads and stop on cancellation. This adds no idle worker or permanent
polling.
- Preserve cancellation reasons, published receipts after lost commit
replies, and claims acquired by a newer owner.
- Add real PostgreSQL regression coverage for first-agent bundled skill
setup, cross-client contention, failed-owner takeover, crash expiry,
cancellation, ambiguous publication, and ownership changes.
- Document upload recovery behavior. No schema or public API change is
required.

## Verification

- Red: four new or updated checks failed against the original
implementation. They reproduced the exact conflict and the retained
ten-minute lease.
- Green: 189 targeted tests passed across
`skill-package-upload.test.ts`, `skill-package-store.test.ts`,
`skill-package-work.test.ts`, both package-storage migration suites, and
`agent-skills-routes.test.ts`. The combined run passed 188 tests. The
final SQL-lock regression passed separately.
- Command: `pnpm exec vitest run
server/src/__tests__/skill-package-upload.test.ts
server/src/__tests__/skill-package-store.test.ts
server/src/services/skill-package-work.test.ts
server/src/services/skill-package-storage-migration.test.ts
server/src/services/skill-package-storage-migration-postgres.test.ts
server/src/__tests__/agent-skills-routes.test.ts`.
- `pnpm --filter @paperclipai/server typecheck` passed, including source
and test layers.
- The cancellation test first exposed Node's timer wrapping the caller's
abort reason. The final implementation preserves that reason, and the
full targeted run passed.
- Greptile identified an unbounded cleanup write. Two new
real-PostgreSQL tests reproduced delayed errors and cancellations behind
an occupied one-connection pool. They now pass and prove that late
cleanup skips its write. A further row-lock test proves that the
PostgreSQL statement timeout frees the pool while another transaction
still holds the package row.
- Local setup initially lacked macOS embedded-Postgres library links.
Repaired ignored dependency links before the red and green database
tests. The initial server typecheck also lacked SDK links for excluded
plugin packages; ran the maintained `node
scripts/link-plugin-dev-sdk.mjs` helper before repeating it.
- `pnpm verify --explain` returned exit 2 and requested broader CI
coverage. No full local suite was substituted for that CI requirement.
- `git diff --check` passed. Reviewed the full diff for secrets, private
URLs, personal data, generated files, and run artifacts.
- Final revision: `4b4814ac01522599901a3dc29f650a1b33c0b4e2`. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/44880245673)
passed all applicable checks, including typecheck, build, all general
and serialized server test shards, all e2e shards, Runner verification,
and the final verify gate. [Greptile
review](https://github.com/paperclipai/paperclip/pull/22477#issuecomment-7059199437)
scored this revision 5/5 with no unresolved threads. GitHub reports the
branch as mergeable and clean.

## Risks

Contended preparations now wait within their existing request deadline.
A crashed process or an unavailable cleanup database still uses the
ten-minute lease expiry. Genuine storage failures remain errors; this
change does not hide an unavailable backend. The existing object
deletion grace remains intact.

Live provider authorization and a deployed Cloud onboarding journey were
not run. Tenant release and deployment are separate from this source
fix. No production migration is needed.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh
reasoning effort and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1121.0-canary.841300 nightly/v2026.1121.0-nightly.0
2026-11-21 00:25:10 -08:00
Devin FoleyandPaperclip 4697a50508 fix: retry verified legacy adapter preparation failures (#22464)
Accept the existing full preparation proof for the exact legacy adapter_failed source named by a saved Retry. Keep older recovery-hold ancestry checks and refusal gates intact. Add one-successor and missing-proof integration coverage, and synchronize the independent watchdog test fixture with reporter readiness.

Qualified on 36463a770a with full green CI, Apex 5/5, no unresolved review findings and no conflicts. Focused Retry/bootstrap checks and the affected server typecheck pass after the rebase; broader local tests apply to the identical source tree.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-20 23:33:16 -08:00
Devin FoleyandPaperclip 6bf312270c fix(ci): stabilize database-backed readiness checks (#22476)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - CI checks its recovery workers, attention routes, and external chat
admission against real PostgreSQL.
> - Large fixtures start from an empty database without statistics for
their new rows.
> - Slow query plans can exhaust test deadlines before the behavior
checks finish.
> - Route responses can also leave wakeups writing rows while teardown
deletes their agents.
> - Setup admission takes an endpoint lock that can deadlock with
resource foreign-key checks and agent invalidation.
> - This pull request prepares fixture statistics, drains pending work,
and uses a compatible endpoint lock.
> - The benefit is reliable readiness checks with the same fixtures,
assertions, and deadlines.

## Linked Issues or Issue Description

Refs: #21941, #21831, #21132. The compiler-memory mitigation in #19383
is already merged and is not repeated here.

**What happened?**

The recovery census and review-attention equivalence tests repeatedly
timed out on unrelated PRs. Recovery-action teardown also failed on an
agent-wakeup foreign key, and concurrent first GitHub mentions could
deadlock during resource admission.

Examples: [census
timeout](https://github.com/paperclipai/paperclip/actions/runs/44868742060/job/136738633187),
[review
timeout](https://github.com/paperclipai/paperclip/actions/runs/44856500775/job/136732143246),
[cleanup
failure](https://github.com/paperclipai/paperclip/actions/runs/44866635624/job/136731817497),
and [chat
deadlock](https://github.com/paperclipai/paperclip/actions/runs/44845946341/job/136667344125).

**Expected behavior**

The complete fixtures retain their regression coverage, pending wakeups
finish before deletion, and concurrent chat admission completes with
exactly one setup destination enabled.

**Steps to reproduce**

1. Run the two timeout suites against a freshly seeded database before
its statistics refresh.
2. Complete an issue-update request that schedules an assignee wake,
then delete its fixture immediately.
3. Run the new resource-admission regression on the base revision. It
forces the endpoint/agent lock order and fails with PostgreSQL SQLSTATE
`40P01`.

**Paperclip version or commit**

Base: `33ffaa5b9a2e0954c2afc063572346cb1be534a7`.

## What Changed

- Run `ANALYZE` on the populated fixture tables after seeding the
existing census and review-attention populations. No fixture reduction,
assertion removal, retry increase, or deadline increase.
- Await `drainActiveRunExecutions()` before recovery-action fixture
cleanup. This includes pending wake admission after HTTP responses.
- Use `NO KEY UPDATE` for setup endpoint admission. It still serializes
competing admission transactions and endpoint mutations while allowing
resource foreign-key `KEY SHARE` locks.
- Add a real PostgreSQL regression using the real `ensureResource`
write, with barriers that force concurrent endpoint admission and an
agent-locked resource insert. Both transactions must finish and exactly
one destination must become enabled.
- Document fixture statistics, teardown ownership, and admission locking
in `doc/test-infrastructure.md`.

## Verification

- The new concurrency regression fails on the original endpoint lock
with SQLSTATE `40P01`, then passes with `NO KEY UPDATE`.
- Both timeout suites pass with their full populations and original
deadlines: 8 tests.
- The full recovery-action suite passes: 120 tests.
- Existing GitHub first-repository and disabled-repository integration
checks pass: 7 tests.
- The new PostgreSQL concurrency regression passes: 1 test.
- The local baseline timeout suites passed slowly; the review
equivalence case took 35.8 seconds before statistics and 0.63 seconds
afterward. These are local observations, not a CI speed guarantee.
- The first broad typecheck caught an unsupported
`Promise.withResolvers` type in the new test. The barriers now use
ordinary promises compatible with the repository target. The final `pnpm
-r typecheck` passes.
- `git diff --check` passes.
- `pnpm -r typecheck` and `pnpm build` pass. The full local `pnpm
test:run` attempt stopped after all six chat shards: 1,969 tests passed
and two Slack tests failed on skill-root discovery. The remaining local
groups did not run. Both failures reproduce with the production change
reverted to `origin/master`: the resolver selects an unrelated ancestor
`skills/` directory without `slack/SKILL.md`, which also prevents the
mocked Slack run from starting. This is a confirmed baseline
local-layout issue, not a passing full-suite result. The [complete CI
run](https://github.com/paperclipai/paperclip/actions/runs/44873350026)
passes on `66c146b177b50e052a21921644dfd626c96c010d` on attempt 1. The
PR check rollup has 58 successes, three expected skips, and no pending
or failing checks. [Greptile reviewed the current head
`66c146b`](https://github.com/paperclipai/paperclip/pull/22476#issuecomment-7058185817)
at 5/5 with no actionable findings.

Review follow-up verification:

- Replaced the resource-read stub with the real
`createChatInboundAuthority(...).ensureResource` persistence path. The
revised regression fails with `FOR UPDATE` and PostgreSQL `40P01`, then
passes with `NO KEY UPDATE`.
- Scoped `ANALYZE` to the fixture tables and moved the behavior note
from `DEVELOPING.md` to `test-infrastructure.md`.
- All 9 affected tests pass after these edits; the complete server
typecheck also passes. The production lock change is unchanged from the
earlier full workspace typecheck and build passes.
- The previous CI attempt had two interrupted jobs: [Runner static
checks](https://github.com/paperclipai/paperclip/actions/runs/44872465981/job/136749952142)
and [serialized server
9/9](https://github.com/paperclipai/paperclip/actions/runs/44872465981/job/136749952782).
Both logs record a runner shutdown. The static lane reported 2,371
passing tests before cancellation; the interruption is not counted as a
pass. The final head passes the complete fresh CI run linked above,
including both lanes.

## Risks

Low scope. Production code changes only the endpoint row-lock strength.
The regression uses real foreign keys and connector invalidation
triggers; the existing integration checks retain first-destination
serialization and disabled-destination behavior. No schema, API, or
authorization changes. `ANALYZE` updates only test-database statistics.
Drain failures still fail teardown.

All current-head remote checks now pass, including typecheck, build,
tests, and e2e. The baseline local full-suite failure and unrun
remaining local groups are recorded above; focused local checks pass.
The already merged compiler-input fix does not establish the cause of
every past runner shutdown.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh
reasoning and shell/code execution. The session does not expose its
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1121.0-canary.841100
2026-11-20 22:52:05 -08:00
33ffaa5b9a fix(heartbeat): name a full sandbox disk that stops a workspace save during setup (#22466)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a run starts, Paperclip prepares the task's remote workspace
and saves it. The save builds an archive inside the sandbox
> - When the sandbox disk is full, that save fails with
`remote_workspace_sandbox_disk_full: the sandbox disk is full; ...`
> - #22383 taught the setup failure path to name a full sandbox disk
`sandbox_disk_full`. That code tells the user what to do, and automatic
retry and generic continuation recovery do not repeat it
> - The setup matcher accepts the runner's form, `sandbox_disk_full:
...`, at the start of the text or after `: `. In the save failure,
`sandbox_disk_full:` follows `remote_workspace_`, so it does not match
> - These runs therefore fail as the generic `setup_failed`. They share
one Sentry group with unrelated uncoded setup failures, and they are
started again on the same full disk
> - This pull request recognizes the exact save failure in the setup
failure path
> - The benefit is that these runs fail as `sandbox_disk_full`, get
their own Sentry group, show the existing plain-language action and are
not repeated automatically

## Linked Issues or Issue Description

Refs #22383 (names a full sandbox disk that fails a run during setup).
This PR covers the workspace save during setup, which that change does
not match.

**What happened?**
On staging, several runs failed while preparing their workspace with
`remote_workspace_sandbox_disk_full: the sandbox disk is full; free
space in the sandbox before saving the workspace`. Each run stayed in
preparation for about 19 minutes. The run had `errorCode: setup_failed`
and `resultJson.failureCause: "Error"`. The Sentry fingerprint
(`setup_failed`, adapter, `setup`, `exception:Error`) put these runs in
the same group as other uncoded setup failures. Each task was then
started again on the same sandbox and failed the same way.

**Expected behavior**
The run fails as `sandbox_disk_full`, like the other full-sandbox-disk
setup failures. The task shows the existing action (free disk space,
then press Retry). Bounded retry and generic continuation recovery do
not repeat it.

**Steps to reproduce**
1. Prepare a remote workspace in a sandbox whose disk is full.
2. Let the preparation save fail with
`REMOTE_WORKSPACE_SANDBOX_DISK_FULL`, either directly or after the
read-only fallback also fails (the fallback rethrows the same message
with the fallback failure as its cause).
3. See the run fail as `setup_failed`.

## What Changed

- `remote-workspace-transport.ts`: add
`isRemoteWorkspaceSandboxDiskFullError`. It is true when the error, or
one of its first causes, has the exact server-built
`REMOTE_WORKSPACE_SANDBOX_DISK_FULL` message. It does not match other
text.
- `run-completion.ts`: the setup failure ladder uses `sandbox_disk_full`
for that error too. The rung's position does not change, so named setup
codes (restore failure, timeout, storage and others) keep their
precedence.
- Tests in `run-completion.test.ts` with a real database: the direct
save failure, the save failure that keeps the read-only fallback failure
as its cause, and another failed save
(`remote_workspace_archive_changed`) that stays `setup_failed`. The
existing host-disk test still passes.

The message, the saved workspace preservation path and the
workspace-recovery handling are unchanged. Retry and continuation
behavior for `sandbox_disk_full` already exists
(`scheduleBoundedRetryForRun` returns `not_scheduled`; recovery lists
the code as non-retryable).

## Verification

- Red: on unmodified master, both new disk-full cases fail with
`errorCode: "setup_failed"`, the same code as the staging runs.
- Green: with the change, `pnpm vitest run
src/services/heartbeat/run-completion.test.ts
src/services/native-runtime/native-sandbox-disk-full.test.ts
src/services/remote-workspace-transport.test.ts`: 352 of 355 pass. The 3
failures are in `remote-workspace-transport.test.ts` (restore lock
`directory_shared`, ambient checkout filters, trusted SSH rewrites).
They fail the same way on unmodified master in my sandbox, so they come
from its Git setup.
- Retry and recovery behavior for the code: `pnpm vitest run
src/__tests__/failed-continuation-continues-again.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/services/workspace-recovery-disk-full.test.ts
src/services/recovery/provider-failure-classification.test.ts`: 4 files,
141 tests pass.
- The full server typecheck did not run locally because the sandbox has
too little memory and disk. CI runs it.

## Risks

- Low. The change only renames a setup failure that already cannot be
repaired by starting again on the same sandbox. It reuses an existing
code with existing retry, recovery and UI handling.
- Runs that hit this failure are no longer started again automatically.
A person frees disk space and presses Retry, as for the other
`sandbox_disk_full` failures.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 20:32:44 -08:00
6b9952525d build(server): omit tests from production compilation (#19383)
Use a separate production TypeScript configuration that excludes colocated tests while retaining every runtime source input and keeping tests in source typechecking. Verify the real compiler input sets and bound compiler-list capture for the larger source graph.

Validation: focused compiler-input regression passes; fresh CI has 61 passing checks and three optional skips on 181ad8d; Greptile Apex is 5/5 with no unresolved comments.

Co-Authored-By: Scott Goodson <scottgoodson@gmail.com>
Co-Authored-By: Devin Foley <devin@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-20 20:02:21 -08:00
Devin FoleyandClaude Opus 5.5 fec961b580 fix(workspaces): record why a full-disk read-only capture failed (#19619)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A remote agent workspace is saved to durable storage after each run.
The save builds a portable archive inside the sandbox.
> - When the sandbox disk is full, #19465 starts one read-only fallback.
The fallback copies a verified source to the controller and seals the
archive there.
> - If that fallback also fails, the code keeps the original disk-full
message and puts the fallback failure in `error.cause`.
> - `remoteWorkspaceFailureCause` reads only the top message. The
recorded copy-back cause and the run error therefore say only "the
sandbox disk is full". The fallback stage and reason are lost.
> - Also, the sandbox inventory script reports every failure as
`unsafe`, and `command()` drops that token. The fallback then says
`reason=operation_failed` for a linked worktree, a budget limit, or an
unreadable file.
> - #20678 later made a transient fallback failure (a dropped
connection) surface as its own cause. Every other fallback failure is
still wrapped in the disk-full error, and that wrapped cause is still
dropped. A staging save on a release with #20678 again recorded only the
bare disk-full message.
> - This pull request keeps the disk-full code and message unchanged. It
appends the fallback's closed-grammar cause to the recorded cause, and
it names the inventory refusal.
> - The benefit is that an operator can see why a full-disk save still
failed (for example `stage=inspect reason=unsafe_source`), so the next
fix targets the real step.

## Linked Issues or Issue Description

Refs #19465
Refs #19543
Refs #20678

**What happened?**

On a release that contains #19465 and #19543, a hosted run finished its
turn, and all three workspace save attempts failed. Each recorded cause
was only `remote_workspace_sandbox_disk_full: the sandbox disk is full;
free space in the sandbox before saving the workspace`. The read-only
fallback ran on each attempt, but the record did not say at which stage
it stopped or why. Later, the existing provider-cache reclaim freed
space and the save completed. The fallback failure itself is still
unexplained, because the record does not contain it.

**Expected behavior**

The recorded cause keeps the disk-full code, so retries and cache
reclaim work as before. It also names the fallback stage, a closed
reason, and a transport label. It never includes paths, file names, or
provider text.

**Steps to reproduce**

1. Create a workspace with a `.git` file (a linked worktree pointer) or
with one unreadable file.
2. Make the normal capture report a full disk, so the read-only fallback
runs.
3. Call `remoteWorkspaceFailureCause` on the capture error.
4. On `master`, the result is only the disk-full message. With this
change, the result ends with `;
remote_workspace_readonly_capture_failed: stage=inspect
reason=unsafe_source transport=provider_error` (or `reason=fs_eacces`).

## What Changed

- `remoteWorkspaceFailureCause`: when the error message is the exact
disk-full message and `error.cause` matches the closed
`remote_workspace_readonly_capture_failed: stage=… reason=… transport=…[
code=…]` grammar (the existing `READONLY_CAPTURE_FAILURE_CAUSE` from
#20678, now also allowing digits in the reason for errno codes), append
that cause after `; `. Other causes are not attached. The thrown
message, the derived code (`remote_workspace_sandbox_disk_full`), and
the disk-full equality checks used by retry and cache reclaim do not
change.
- The sandbox inventory script now writes
`remote_workspace_readonly_capture_budget` for a budget limit and
`remote_workspace_readonly_capture_fs_<errno>` for a filesystem error
code. Other refusals still write
`remote_workspace_readonly_capture_unsafe`. It never writes error text
or paths.
- `command()` accepts only these exact tokens from a failed capture
command, so the fallback can map them.
- The fallback maps `fs_<errno>`, `disk_full` and command timeouts
(`timed_out`) to their own reasons, in addition to the existing
`unsafe_source`, `source_changed` and `budget_exceeded`.
- New test file `remote-workspace-readonly-capture-cause.test.ts` with
four cases: source changed during transfer, linked worktree refusal
before transfer, unreadable file by errno only (no private file name in
the cause), and a cause outside the grammar that is not attached.

## Verification

Rebased onto `master` at `b4cea5b5a4` (2026-11-21). The only conflict
was in `remoteWorkspaceFailureCause`, where #20678 added a direct match
for a thrown read-only cause. Both behaviours are kept, and the PR now
reuses #20678's grammar instead of adding a second regex.

- Red on `master` (`b4cea5b5a4`, with the two source files swapped back
to `master`): three of the four new tests fail, each getting only the
bare disk-full message. The grammar guard test passes on both.
- Green with this change: all four new tests pass.
- From `server/`: `npx vitest run
src/services/remote-workspace-readonly-capture-cause.test.ts
src/services/remote-workspace-readonly-inventory.test.ts
src/services/remote-workspace-transport.test.ts
src/services/native-runtime/native-run-finalizer-telemetry.test.ts`: 213
passed, 6 failed. All 6 are in `remote-workspace-transport.test.ts` and
are environmental in this sandbox:
- `does not run ambient checkout filters…`, `uses trusted SSH rewrites
only for clone…` and `rejects an unsafe restore lock…: directory_shared`
fail the same way with the unchanged `master` source (the sandbox's Git
wrapper blocks clones).
- The other three hit the 15 s test timeout under load. Run alone,
`restores offline partial Git state…` passed twice with this change, and
passed once and failed once with the unchanged `master` source.
- The finalizer telemetry suite (#20678's tests, including "hands a full
sandbox disk to saved-work recovery on the first copy-back failure")
passes. The appended cause keeps the
`remote_workspace_sandbox_disk_full:` prefix, and the saved failure code
still comes from the text before the first `:`.
- A full typecheck was not run locally: `tsc` ran out of memory in this
sandbox. CI typecheck and build must pass.
- Environment: Linux, Node v25.9.0, filtered `pnpm install` of the
server package.

## Risks

- Low risk. The thrown error message and the save failure code do not
change, so retry policy, recovery classification and full-disk cache
reclaim behave as before.
- The run error text ("Last cause: …") becomes longer for this case. It
stays below the 1024-character bound and matches the existing
`readWorkspaceFailureCause` grammar (a test checks this).
- An unreadable or vanished file now reports `fs_<errno>` instead of
`unsafe_source`. That is more exact. Both reasons keep the source
retained and do not produce a receipt.
- No documentation text changed. `doc/execution-semantics.md` says that
transfer failures "retain the original disk-full error", and that stays
true.

## Model Used

- Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), used
through Claude Code with tool use (shell, file editing, test runner,
GitHub connector). The exact context window is not shown in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (the new tests pass; the
two clone-dependent failures also fail on unchanged `master` here)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
change needed: the documented contract is unchanged)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 19:32:44 -08:00
Nicky Leach 467fd32223 Publish tool profiles and their selectors atomically (#22455) 2026-11-20 19:02:54 -08:00
Nicky Leach 7f4cfae5e1 Narrow exact connected tool reads before loading schemas (#22457) 2026-11-20 19:02:39 -08:00
dependabot[bot] 2b8d92c769 build(deps-dev): bump @cloudflare/workers-types from 5.20261114.1 to 5.20261117.1 in /scripts/typecheck-sandbox-providers (#19570)
Bumps [@cloudflare/workers-types](https://github.com/cloudflare/workerd)
from 5.20261114.1 to 5.20261117.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/cloudflare/workerd/releases">@​cloudflare/workers-types's
releases</a>.</em></p>
<blockquote>
<h2>PGO profiles</h2>
<p>The profiles that optimize the Linux release builds
(build/pgo/README.md). Don't delete them: build/pgo/profile.json names
one of them, as may older commits that are built again.</p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/cloudflare/workerd/commits">compare
view</a></li>
</ul>
</details>
<br />

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-20 18:10:34 -08:00
Devin FoleyandPaperclip 18978bdc1d fix(native): preserve named human unblock owners (#22438)
## Thinking Path

> - Paperclip manages work carried out by agents and people.
> - Native finalization converts a runner result into a durable task
status and follow-up effects.
> - A blocked result can name the person who must decide what happens
next.
> - The classifier discarded that identity, and the arbiter saved a
generic board owner.
> - This could replace an explicit user owner saved through the task API
and leave the decision outside that person's inbox.
> - This pull request resolves human owners against active company
membership and preserves them through commit.
> - The named person receives the existing blocker card, and the blocked
worker does not get another wake.

## Linked Issues or Issue Description

**What happened?**

Native finalization replaced a named human unblock owner with `board`.
It also overwrote an explicit user owner saved through the task API
during the turn.

**Expected behavior**

Keep a valid human unblock owner and route the blocker card to that
person's inbox. Use the board when the reference is unknown, inactive,
ambiguous, or outside the company.

**Steps to reproduce**

1. Create an active company member with a distinct display name.
2. Have a native worker report a human blocker using that name.
3. Finalize the run and read the task's unblock owner and the member's
inbox.

Related: #21265 covers named agent handoffs and retains board ownership
for people. This change covers human member routing and does not add
agent handoffs.

## What Changed

- Resolve a human owner from an explicit user ID, an API-bound user, or
one matching company member by ID, full name, or email.
- Preserve the user owner for both blocker scopes and durable dependency
waits.
- Recheck the accepted result and active membership under the status
commit lock, including board fallbacks. Rebuild the assessment on every
status retry so a newly saved owner or revoked membership cannot be
overwritten by stale evidence.
- Enforce `canAssignTasksToHumans` before binding a human owner,
including dependency waits that do not emit a `bind_blocker` effect.
- Validate the result before retiring automatic completion reviews.
Invalid results leave existing reviews pending; valid results retain the
normal cleanup path.
- Keep human blocker routing out of the agent wake queue.
- Add embedded Postgres regressions for finalization, replay, personal
attention, invalid owners, late API ownership changes, membership
revocation, permission denial, and answer continuation for human-owned
blockers.
- Make the answer-continuation fixture persist the same result it
classifies, so commit-time evidence validation sees realistic saved
evidence. Assert the resolved user owner in both blocker scopes.
- Let the recovery census clock advance at real-time pace during
delivery I/O waits. This handles callbacks and nested timers without
jumping to maintenance deadlines. Preserve the existing real-clock
timeout and recovery assertions, and name pending queues in failures.
- Document human blocker resolution and compatibility.

## Verification

- Red-green reproduction: three ownership regressions failed on the
original implementation and passed with the fix.
- Red-green race reproduction: late API ownership changes and membership
revocation failed before the commit recheck and per-retry reassessment
fixes. Each finalization invocation now has to commit; a later replay
cannot hide an exhausted retry.
- Red-green review-retirement reproduction: a new invalid result
cancelled a proven automatic review before throwing
`native_result_invalid_disposition`. The new invalid/valid regression
pair verifies that invalid results preserve the pending review and task
decision, while valid completion still retires the card.
- Red-green answer-continuation reproduction: both human-owner cases
failed with `native_status_race` while saved evidence was empty.
Persisting the classified result and asserting `{ userId }` restored all
13 answer-continuation cases, including worker resumption.
- Red-green permission reproduction: an agent with
`canAssignTasksToHumans: false` could bind a human blocker before the
permission check. Direct and dependency cases now refuse finalization
without creating a personal blocker card.
- On refreshed head `a6d37916546`, `pnpm --filter @paperclipai/server
exec node scripts/typecheck-layers.mjs` passed on macOS with Node 24.
- On refreshed head `a6d37916546`, from the repository root, `CI=true
node node_modules/vitest/vitest.mjs run
server/src/services/native-runtime/native-agent-blocker.integration.test.ts
server/src/services/native-runtime/status-arbiter.test.ts
server/src/services/native-runtime/evidence-classifier.test.ts
server/src/__tests__/native-status-reassessment-work.test.ts
server/src/__tests__/routable-blocked.test.ts
server/src/__tests__/native-status-arbiter-corpus.test.ts
server/src/services/native-runtime/native-environment-blocker.integration.test.ts
server/src/__tests__/heartbeat-blocked-question-answer.test.ts
server/src/__tests__/issue-recovery-live-run-agent-census.test.ts`
passed: 292 tests in nine files on macOS with Node 24.
- Red-green census reproduction: both post-I/O callbacks and nested
timers must settle without jumping to a future maintenance deadline. The
nested case timed out under a zero-only clock pump; the shared
automatically advancing census clock handles both.
- Final CI at `a6d37916546`: [run 44848053016, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/44848053016/attempts/2)
completed successfully. Build, Canary Dry Run, policy, every server and
workspace test shard, Runner checks, serialized suites, e2e, and verify
are green. Greptile review 7 is 5/5 on the same head, Superagent and
other security checks pass, and no review threads remain unresolved. A
local merge-tree check against master `621c5577cdf` also passed with no
conflicts.
- Current CI at `a6d37916546`, [run
44848053016](https://github.com/paperclipai/paperclip/actions/runs/44848053016):
attempt 1 finished with 13 interrupted nonaggregate jobs, each with an
explicit runner shutdown signal in its log. Nine signals occurred within
three seconds on separate m6a.xlarge hosts in us-east-1a; Build and
Canary also shut down later. Aggregate e2e and verify failed on those
dependency results. The underlying termination cause remains unknown.
Shared CI settings are unchanged.
- The [automatic recovery
run](https://github.com/paperclipai/paperclip/actions/runs/44849704877)
skipped retry because Build had only the annotation `Process completed
with exit code 143.` The classifier reads logs for cancellation
annotations, but did not read this job's log. That log explicitly
records a runner shutdown at 01:10:49 UTC. The other 12 failed jobs were
recognized as runner loss. After verifying all 13 logs, manually
requested failed-job attempt 2; it completed successfully without source
or configuration changes between attempts. The shared recovery
classifier is unchanged; this false negative remains a documented
follow-up.
- Historical CI at `cf601ba`: all server shards (including the census),
Runner checks, typecheck/release registry, serialized suites, e2e, and
canary dry run passed. Build received runner shutdown signals on
attempts 1–7; aggregate verify failed because Build failed. Attempt 5
diagnostics show shutdown while the job lease was valid. Attempt 7
stopped after server dependency preparation, without a compiler error.
The termination cause remains unknown. Three recent PRs completed Build
on the same AWS fleet; this is not evidence of a fleet-wide outage.
Shared CI settings are unchanged.
- Diagnostic comparison at `cf601ba`: the server compiler passed locally
on macOS with four checkers, full typechecking, and
declaration/JavaScript emission (`node ../scripts/run-tsc.mjs --checkers
4 --extendedDiagnostics` under Node 24). The monitored process group
peaked at about 10.5 GiB RSS and finished in 25.8 seconds. This does not
establish the cause of the Linux runner shutdowns.
- Refreshed the branch onto master `611dc68581c` without conflicts; the
PR source diff is unchanged. Greptile review 7 scored head `a6d37916546`
5/5 with no new actionable findings and no unresolved review threads.
All applicable CI gates passed on attempt 2.
- Full local workspace build at `cf601ba` passed on macOS using Node 24
and the documented single-thread compiler mode:
`PAPERCLIP_TYPECHECK_LOW_MEMORY=1 npx --yes --package=node@24
--package=pnpm@9.15.4 pnpm build`. This includes typechecking and
artifact emission. The tracked working tree stayed clean.
- `pnpm verify --explain` returned exit 2 because this documentation
path requires broad CI coverage. Fresh CI will supply that coverage; the
full local test suite was not run.
- Earlier CI build and canary jobs received runner shutdown signals. The
server shard also exposed an answer-continuation fixture that saved an
empty result while classifying a named human. That mismatch was
reproduced locally, and the fixture now persists the classified result
and checks the user owner. Fresh CI is required for this revision.
- `git diff --check` and the staged gitleaks scan passed. The diff was
also checked for private issue references, identifiers, URLs, and
personal data.

## Risks

- Name and email matching ignore case. Several matches fall back to the
board. Invalid explicit IDs never fall through to another person.
- Resolution adds bounded reads only when a native result reports a
human blocker. It adds no worker, timer, or idle scan.
- No database or result-schema migration is required. Existing board and
agent decisions remain readable. Committed historical decisions are not
rewritten; new finalizations use current membership.
- This changes human routing only. It grants no access and changes no
merge, approval, or dependency policy.

## Model Used

- OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex in T3 Code, with
xhigh reasoning effort, code editing, shell tools, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-20 17:45:45 -08:00
621c5577cd fix(tasks): hide More messages when no hidden history exists (#22099)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The task page shows the instruction and the recent exchange first,
and shows a **More messages** button for older history
> - The button is visible on almost every task. A click often shows no
new message, and the button then goes away
> - The compact chat shows only the latest run, so the page treats any
task with two or more runs as a task with hidden history
> - The recent comments read also reports more comments when the omitted
comment is already pinned into the window
> - This pull request shows the button only when a message is really
hidden
> - The benefit is a clean task page and a button that always reveals
something

## Linked Issues or Issue Description

Refs #17751

**What happened?**

The task page shows **More messages** on almost every task. A click
often adds no message, and the button then goes away.

**Expected behavior**

The page shows **More messages** only when it hides an older comment or
an older run that has no visible answer.

**Steps to reproduce**

1. Create a task and let an agent answer it in two runs.
2. Open the task with a fresh application cache.
3. See **More messages** above the conversation. Click it. No new
message appears.

## What Changed

- The recent run window keeps the newest omitted run (`earlierRun`), and
it keeps that run as later turns arrive.
- The task page ignores the omitted run when the run succeeded and its
agent answer is already visible. A failed run, or a run with no visible
answer, still shows the button. The answer check is the same one the
chat thread uses (moved to `ui/src/lib/run-answer-visibility.ts`), so a
legacy withheld-run placeholder or a progress comment does not count as
an answer.
- `GET /issues/:id/recent-comments` reads seven rows instead of five. It
sets `hasMore` only when the window leaves out a comment. Older comments
that are pinned into the window (the latest human message and the first
instruction) do not count.
- Added regression tests for the page (including the placeholder case),
the run window cache and the server route.

## Verification

- `pnpm exec vitest run src/pages/IssueDetail.test.tsx
src/components/TaskChatThread.test.tsx src/lib` in `ui`: 3,637 passed, 1
skipped. The new page test fails without the fix.
- After the review fixes: `pnpm exec vitest run
src/pages/IssueDetail.test.tsx src/components/TaskChatThread.test.tsx
src/lib/issueChatBootstrapCache.test.ts
src/lib/issueDetailCache.test.ts` in `ui`: 738 passed, 1 skipped.
- `pnpm exec vitest run src/__tests__/task-cold-open.test.ts
src/__tests__/issue-chat-bootstrap-routes.test.ts` in `server`: 33
passed.
- `tsc --noEmit` in `ui` and `server`: no errors in the changed files.
- I did not do a check in the browser.

## Risks

Low risk. No schema change. The API response shape does not change. The
run window cache gets one optional field, and older saved caches without
the field keep the old behavior.

Limitation: the recent runs read returns only the two latest runs, so
the page sees one omitted run. If that run has a visible answer but an
older run failed, the page does not show the button. **More task actions
→ Show task activity** still shows that run.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 17:41:17 -08:00
Devin FoleyandPaperclip b4cea5b5a4 fix(runner): show disk exhaustion before recovery errors (#22448)
Show the confirmed sandbox disk failure before a later lost-turn or restore error in saved run errors and both run detail views. Preserve the original diagnostics and leave scarce inodes available when preparing the existing runner relaunch reserve.

Verified focused regressions, scoped typechecks, and full CI on the reviewed revision, including Runner checks and all eight end-to-end shards. Greptile is 5/5 with all review findings resolved.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-20 17:08:31 -08:00
611dc68581 fix(runner): continue a lost turn on a resumed warm session (#22447)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a managed sandbox run under the Paperclip runner
(`runnerd`). When `runnerd` exits mid-run, the transport relaunches it,
and the runner restores the same conversation
> - #22235 lets the execution that started a turn send one continuation
turn when the restored runner reports that turn as
`provider_turn_lost_on_restore`
> - That rule applies only to a session that the execution opened fresh
(`!recovered`). A run that resumes the warm conversation of its previous
run is a recovered session. A chat follow-up does this on every new
message
> - So a resumed session that loses its own task turn gets no
continuation. The run fails, and the user must press Retry
> - This pull request makes the task turn that this execution started
its own turn, whether the session was opened fresh or resumed
> - The benefit is that a chat follow-up keeps its conversation and
workspace after a mid-run `runnerd` restore, as #22235 intended

## Linked Issues or Issue Description

Refs #22235 (continue a turn lost to a mid-run runnerd restore). Refs
#22068 (disk reserve). Refs #22239 (stop retrying a run that filled its
sandbox disk).

**What happened?**
On staging, an agent in a chat task filled its 10 GiB managed sandbox
disk. It fetched the full history of a large Git repository and searched
it. `runnerd` exited with `failed to create private durable state
temporary file: No space left on device (os error 28)`. The disk reserve
let the transport relaunch `runnerd`, and the runner restored the same
conversation. Then the runner reported `turn.failed {code:
provider_turn_lost_on_restore, recoverable: true}` for the turn that
this run had started. The runtime did not send the continuation turn.
The run failed as `sandbox_disk_full` with the message "The runner
restored the conversation after process loss, but the previous turn is
no longer active." A manual Retry on a fresh provider session finished
the work.

The run started from the warm checkpoint of the previous run in the same
conversation (`startup reuse: provider_session (reused)`, then
`session.resumed` with the same provider session id).
`loadWarmNativeCheckpoint` rebinds that checkpoint to the new run and
clears its turns. `executeNativeSession` recovers it, so `recovered` is
true. The run then starts its own task turn. `restoredInThisExecution`
required `!recovered`, so that turn could not be continued.

**Expected behavior**
When a mid-run `runnerd` restore loses the task turn that this execution
started, the execution sends one continuation turn in the restored
conversation. This applies to a fresh session and to a resumed warm
session.

**Steps to reproduce**
1. Run a native agent through `paperclip_runner` in a managed sandbox,
as a follow-up on an existing conversation (a warm resumed session).
2. During the turn, make `runnerd` exit, for example by filling the
sandbox disk.
3. The transport relaunches `runnerd`, and the runner reports the turn
as lost on restore.
4. The run fails instead of continuing the turn.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`restoredInThisExecution` no longer requires a fresh session. The turn
id that this execution records when it starts its task turn is now
enough.
- The execution records that turn id only for a turn that carries the
task work. A disposition-only recovery turn (the turn that asks only for
`paperclip_finish` after a completed turn) is still not continued, as
before.
- All other guards stay: one continuation per run, the provider
allowlist, no compaction or goal runs, no pending runtime requests, no
saved semantic result, and the snapshot must show that same turn as
restart-interrupted. The control-plane restart claim path
(`restoredAtStartup`) does not change.
- `native-session-runtime.test.ts`: three new cases for a recovered warm
session:
  - it continues its own task turn once after a mid-run restore;
  - it does not continue a lost turn that it never started;
  - it does not continue a lost disposition-only recovery turn.

## Verification

- Red before the fix: `npx vitest run
src/native-session-runtime.test.ts` in `packages/paperclip-runner`
failed `continues its own turn on a resumed warm session once when
runnerd is restored mid-execution` with `NativeProviderTerminalFailure`
(`code: native_provider_terminal_failed`, `providerCode:
provider_turn_lost_on_restore`). This is the same failure as on staging.
- Green after the fix: the same file has 303 passed, 1 failed. The one
failure is `settles a governed wait through real Runner shutdown after a
missing Codex terminal`. It needs the Rust `paperclip-runnerd` binary
(`ENOENT runner/target/debug/paperclip-runnerd`). It fails the same way
before and after the change.
- The existing `requires the control plane restart claim` and `does not
continue a lost turn this execution never started` cases still pass.
- To prove that the disposition-only guard is necessary, I removed it
locally. The new disposition-only case then failed (the run continued).
I put the guard back.
- `native-session-runtime.provider-turn.test.ts`: 36/36 passed. The
Codex driver provider-turn, recovery, stale-generation and
terminal-replay suites and `terminal-call-freshness.test.ts` passed.
- Not run locally: `src/live/runnerd-restart-continuation.test.ts`. It
needs the real Rust `runnerd` binary, and this sandbox has no Rust
toolchain. CI builds the runner and runs these tests.
- Live acceptance: after the automatic staging deploy, a chat follow-up
whose `runnerd` restarts mid-turn must show one continuation turn in the
same sandbox instead of a failed run.

## Risks

- **Second executor.** The continuation starts only for the turn id that
this execution received from its own `startTurn` call. A turn from an
earlier process, or a turn that the execution adopted, is not this
execution's turn. That path still needs the control-plane restart claim.
The continuation runs at most once per run
(`restartContinuationStarted`).
- **Disk still full.** If the continuation fills the disk again, the
second lost turn finds `restartContinuationStarted` set. The run then
fails as `sandbox_disk_full`, and automatic retries stay off (#22239).
The worst case is the current behavior.
- **Disposition-only turns.** A recovered session that only asks for the
disposition does not change. This change does not add a continuation for
that turn.
- Low risk otherwise. The change is 1 condition and 1 assignment in the
runner runtime.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, Paperclip API and Sentry connectors).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (except one test that needs
the Rust runner binary; see Verification)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
doc change is necessary: the runtime comment states the rule)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 16:52:11 -08:00
Nicky LeachandPaperclip b6a904554d Load summaries for the Access Profiles index (#22456)
## Thinking Path

> - Paperclip helps people manage AI agents for work.
> - The Access Profiles page shows profile names, counts, and agent
assignments.
> - The index loaded every selector and started detail queries for each
connection.
> - The server can compute the displayed fields with the existing
visibility and review rules.
> - A summary route lets the index fetch just the data it renders.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Opening the Access Profiles index for a company with many assigned
tools.

**Current behavior**

The index downloads full profiles and requests other resources used by
detail views.

**Proposed behavior**

Fetch summaries and agents at mount. Keep the existing full-profile
route for detail callers.

**Reason and benefit**

In a generated fixture with 166 profiles and 176,257 entries, the
uncompressed response fell from 86,879,901 bytes to 58,095 bytes. The
mounted query count fell from 28 to 2 with 22 connections.

## What Changed

- Add the typed company summary route and its OpenAPI entry.
- Compute counts with the existing projection and use the same profile
visibility rules. Group equal selectors in PostgreSQL and send their
profile memberships as ordinals.
- Read summaries from the index. Keep profile mutations invalidating the
index.

## Verification

- Local UI typecheck and 27 UI tests passed. PostgreSQL projection and
route tests checked count parity, delegated visibility, and rejected
access.
- Layered server typecheck passed. PostgreSQL tests also covered all
selector types, null and unknown legacy selectors, duplicate
memberships, and overlapping include/exclude rules.
- On a disposable database with fresh PostgreSQL statistics, 166
profiles and 176,257 memberships, four interleaved trials per summary
implementation returned identical 58,095-byte JSON responses. PostgreSQL
transferred 36,338,231 bytes for flat summaries and 1,819,600 bytes for
grouped summaries (95% less). Both used five statements. Median service
and serialization time fell from 589 to 227 ms (564–627 ms versus
216–238 ms).

## Risks

The summary still enumerates selector memberships in PostgreSQL.
Grouping raised warm PostgreSQL executor time for the selector query
from about 17 ms to 136–137 ms and used about 15 MiB for the temporary
sort in the fixture. The smaller transfer reduced total service time in
the same synthetic workload. Exact timing will vary with database
resources and profile shape.

## Model Used

OpenAI Codex (GPT-6), with GPT-6.2 Sol agents for implementation and
local verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-20 16:45:34 -08:00
DottaandClaude Opus 5.5 e64872fcdb test(restart-chaos): credit a provably fair follow-up queue wait in R1 (#22452)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos`) grades each task
of a wave against rules R1–R6
> - #22446 lets R1 credit the first run's queue wait when the harness
can prove that wait was fair
> - The staging-mix shape gives each agent about 21 assignments on 4
slots, all within about 2.5 minutes. A follow-up comment that arrives
later also waits 20–26 minutes in a fair FIFO queue
> - The R1 budget (`1.5 × turn + 20 min`) cannot absorb that wait.
Recent grade summaries show a follow-up queued-wait p50 of about
450–1,100 s, so the problem is structural
> - This pull request extends the same fairness proof to the queue wait
of a run that a board follow-up comment started
> - The benefit is that a fair capacity queue no longer fails R1, while
deploy-caused waits and real dispatch-fairness bugs still fail

## Linked Issues or Issue Description

Refs #22446

**What happened?**

In wave `w-20261120T224738-s7106`, task t007 failed R1 as
`no_rest_within_budget (last status in_progress)`. Its first assignment
run ran from 22:49:15Z to 23:00:21Z and succeeded. The harness then
posted a follow-up comment, which created run `dde4d929` (invocation
source `automation`) at 23:00:21Z. That run sat queued for 1,558 s,
until 23:26:19Z, behind its agent's older assignments. It succeeded at
23:27:45Z, 17 s after the observer's deadline, so the observer stopped
watching. All 32 tasks finished hands-off, with 0 Retry banners, 0
duplicates and clean swaps.

**Expected behavior**

A task whose follow-up run waited fairly for agent capacity is not
marked late because of that wait. A follow-up run that a newer fresh
assignment overtook, or that waited while a slot was idle, still fails
R1. Waits of retries and recovery runs still count against the budget.

**Steps to reproduce**

1. Run a staging-mix wave with 32 arrivals and 4 slots per agent (for
example the plan in `w-20261120T224738-s7106`).
2. Find a task whose follow-up comment is posted while its first run
works.
3. See that the follow-up run queues for more than 20 minutes after the
first run ends, and that R1 fails although every slot was busy.

**Paperclip version or commit**

`104f323bee` (master, after #22446)

## What Changed

- `grader.ts`: `runSelection` chooses the runs whose queue wait R1 may
credit. The rules, in order:
1. If any run on the task has no parseable `createdAt`, nothing is
credited (`no createdAt for <ids>, so runs cannot be ordered`).
2. The first run (earliest `createdAt`) is a candidate unless it is a
retry.
3. For each harness follow-up comment (`followUp`,
`stagingMix.followUp`, `stagingMix.afterEndFollowUps`): without a send
time, nothing is credited for it (`no send time for follow-up comment
<id>`). Otherwise the candidate is the earliest run that carries the
comment, was created at or after the send time (`followUp.sentAt ?? at`;
the staging-mix `at` is taken before the request), and was not cancelled
before it executed (cancelled with no start, or a pre-dispatch
cancellation such as `continuation_task_ownership_changed`).
4. That run counts only if it is not a retry, the previous executed run
succeeded and finished at or before its `createdAt`, and the wake
evidence shows a comment wake reason (`issue_commented`,
`issue_comment_mentioned`, `issue_reopened_via_comment`) and no
continued or interrupted run. Without wake evidence nothing is credited
(`no wake evidence for follow-up run <id>`).
5. Each candidate's wait (`createdAt` to `startedAt`, clipped to the
task's `createdAt..restAt`, minus outage time) is credited only when
`queueWaitFairnessFrom` proves it fair. Any provably unfair wait fails
as `unfair_queue_wait` and names the run. An unproven wait is not
credited, and the detail says why.
- `grader.ts`: `gradeR1` adds the credited waits to the allowed time. A
candidate that never started gives `<kind> <id> never started`.
- `wave-grade-inputs.ts`, `types.ts`: `prepare` now adds `runWakes`
(each run's wake reason and continued run ids, from
`wake-evidence/<task>.json`) to the record before grading.
- `observer.ts`: the observer holds the timeout while any credited run
is queued (up to one more budget). It extends the deadline once per
credited run by that run's wait when the run starts.
- `README.md`: documents the follow-up credit.
- No product code changes.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
71 files, 884 tests pass.
- `pnpm exec tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean.
- New tests in `grader.test.ts` (`R1 follow-up queue wait credit`):
- credits a fair 1,558 s follow-up queue wait behind the agent's older
assignments
- fails as unfair when a newer fresh assignment started ahead of the
follow-up
  - fails as unfair when a slot sat idle while the follow-up was queued
  - does not credit the wait of a retry or a recovery run
- does not credit a follow-up run created while the previous run was
still running
- credits a fair wait of a run woken by an after-end follow-up, but not
after a recovery
  - refuses follow-up credit when an earlier run has no createdAt
- does not credit a recovery run that carries the comment after a
success
- reads each run's wake reason and continued runs from the wake evidence
- does not credit a run created before its follow-up comment was posted
- credits a run created after the comment request was sent but before it
returned
  - credits nothing when the earliest run has no createdAt
- credits the first executed carrier, never a run cancelled before it
executed
  - refuses follow-up credit without wake evidence
  - credits only the part of a wait inside the task's own lifetime
  - does not credit outage time inside the follow-up's wait
  - sums the fair waits of the first run and a follow-up
- New tests in `observer.test.ts`:
- waits while a follow-up run is queued and extends the deadline by its
queue wait once it starts (fails before the change)
  - does not hold the deadline for a queued run that a recovery created
- Offline re-grade of the saved wave `w-20261120T224738-s7106` with a
scratch script (not committed). The old `gradeR1` reproduces all 32
saved R1 details first. Records go through the new `prepare`, so
`runWakes` come from the saved wake evidence (t007: `dde4d929` has wake
reason `issue_commented` and no continued run).
  - The other 31 tasks: no R1 verdict changes (31 pass).
- t007 as saved: still `no_rest_within_budget`, because the old observer
timed out 17 s before the task finished.
- t007 as the new observer records it (deadline extended by the 1,558 s
wait to 23:53:26Z, task done at 23:27:45Z): `pass — done after 2310s
(allowed 3851s incl. 1558s fair queue wait)`. The wave's
`agent-slots.json` has `createdAt` for every run, so the proof is
complete.
- The same t007 record without slot evidence: `done_too_late 2310s >
2293s (queue wait not credited: no agent slot evidence …)`.

## Risks

- Low risk. The change is in the test harness only.
- Tasks inside their base budget grade exactly as before.
- A follow-up wait is credited only with wake evidence. Waves without
`wake-evidence/<task>.json` get no follow-up credit, and the R1 detail
says so.
- The observer cannot see wake reasons while it runs, so it may watch a
little longer than the grader later credits. The grader decides the
verdict.
- An observer can watch a task for longer, up to each credited run's
queue wait. This can make a wave a little longer when queues are long.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 18:41:34 -06:00
dependabot[bot] 7d8c4537f7 build(deps-dev): bump wrangler from 4.158.0 to 4.164.0 in /scripts/typecheck-sandbox-providers (#19571)
Bumps
[wrangler](https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler)
from 4.158.0 to 4.164.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/cloudflare/workers-sdk/releases">wrangler's
releases</a>.</em></p>
<blockquote>
<h2>wrangler@4.164.0</h2>
<h3>Minor Changes</h3>
<ul>
<li>
<p><a
href="https://redirect.github.com/cloudflare/workers-sdk/pull/16450">#16450</a>
<a
href="https://github.com/cloudflare/workers-sdk/commit/5f05841d4d4a4a67fee5ddc0150309b5ed782069"><code>5f05841</code></a>
Thanks <a href="https://github.com/xortive"><code>@​xortive</code></a>!
- Add custom CA and client certificate references to Workers VPC service
TLS settings</p>
<p>Workers VPC (open beta) service create and update commands accept
<code>--certificate-authority</code> and
<code>--client-certificate</code> UUIDs. Use either flag alone or both
together. Whenever you supply either reference, explicitly choose
<code>--cert-verification-mode</code>:</p>
<ul>
<li><code>verify_full</code>: verify the origin certificate chain and
hostname (the platform default).</li>
<li><code>verify_ca</code>: verify the chain but skip hostname
verification, weakening origin identity checks.</li>
<li><code>disabled</code>: do not verify the origin certificate; this
allows connections to impersonated origins and is unsafe for
production.</li>
</ul>
<p>For HTTP services, these settings apply to HTTPS connections, not
plaintext HTTP. TCP services can use them for upstream TLS, including
PostgreSQL.</p>
<p>TLS settings are per-service only: host and resolver networks have no
TLS settings or inheritance. Omitting TLS settings uses platform
defaults (certificate chain and hostname verification). Explicit
mode-only configuration, including <code>verify_full</code>, is
accepted. A custom CA cannot be combined with <code>disabled</code>; a
client certificate without a custom CA can.</p>
<p>Upload certificates using the existing commands, then use the
returned IDs below. Replace <code>&lt;ca-uuid&gt;</code>,
<code>&lt;client-uuid&gt;</code>, <code>&lt;tunnel-uuid&gt;</code>, and
service IDs with your own values; certificate references must be valid
UUIDs. Upload replacement certificates the same way before rotating
their references.</p>
<pre lang="sh"><code>pnpm exec wrangler cert upload
certificate-authority --ca-cert ./ca.pem --name private-ca
pnpm exec wrangler cert upload mtls-certificate --cert ./client.pem
--key ./client-key.pem --name private-client
</code></pre>
<p>Create an HTTPS service with both a custom trust chain and an mTLS
client certificate, or a PostgreSQL TCP service with only a custom
CA:</p>
<pre lang="sh"><code>pnpm exec wrangler vpc service create private-web \
--type http --https-port 443 --ipv4 10.0.0.10 --tunnel-id
&lt;tunnel-uuid&gt; \
--certificate-authority &lt;ca-uuid&gt; --client-certificate
&lt;client-uuid&gt; \
  --cert-verification-mode verify_full
<p>pnpm exec wrangler vpc service create private-db <br />
--type tcp --tcp-port 5432 --app-protocol postgresql <br />
--hostname db.internal.example --resolver-ips 10.0.0.53 --tunnel-id
&lt;tunnel-uuid&gt; <br />
--certificate-authority &lt;ca-uuid&gt; --cert-verification-mode
verify_full
</code></pre></p>
<p>For client-certificate-only configuration, use
<code>--client-certificate &lt;client-uuid&gt; --cert-verification-mode
verify_full</code> instead of the certificate flags in either create
command.</p>
<p>Updates replace the complete service configuration, including TLS
settings, in a single request, the same as <code>cf workers-vpc services
update</code>. Pass every TLS option the service should keep:
certificate references and the verification mode that you omit are
cleared, and an update without TLS options returns the service to the
platform defaults. Removing a reference releases its backend certificate
binding but does not delete the uploaded account certificate. For
example, rotate the web service's client certificate while keeping its
CA:</p>
<pre lang="sh"><code>pnpm exec wrangler vpc service update
&lt;web-service-uuid&gt; \
  --name private-web --type http --https-port 443 \
  --ipv4 10.0.0.10 --tunnel-id &lt;tunnel-uuid&gt; \
--certificate-authority &lt;ca-uuid&gt; --client-certificate
&lt;replacement-client-uuid&gt; \
  --cert-verification-mode verify_full
</code></pre>
<p>To remove the CA while keeping the client certificate, repeat the
update with only <code>--client-certificate &lt;client-uuid&gt;</code>
and a verification mode.</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/02a61779a14fe40a5626ec9d26153b93929282d7"><code>02a6177</code></a>
Version Packages (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16950">#16950</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/5f05841d4d4a4a67fee5ddc0150309b5ed782069"><code>5f05841</code></a>
[wrangler] Add custom CA and mTLS configuration to VPC services (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16450">#16450</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/06acbd4bd236ec272ea76c4e77a9ef2e0f395654"><code>06acbd4</code></a>
Version Packages (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16915">#16915</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/d36b39d440a8f408f41dcf4925a81f2f8a58f73b"><code>d36b39d</code></a>
<code>@cloudflare/build-output-utils</code> v1 RC (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16917">#16917</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/0bf81ac16041a1f47f5474f7d6f5b0da6160f76b"><code>0bf81ac</code></a>
Move <code>@cloudflare/config</code> package to 1.0 RC (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16907">#16907</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/db3c071619a64e0cf267fbadd4be12708c02df2d"><code>db3c071</code></a>
Stop wrangler dev from losing config changes after a reload (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16944">#16944</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/81e56ddf3e1ba28ea2320388fac86011a68f7be4"><code>81e56dd</code></a>
Use effective max_instances for container rollout defaults (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16920">#16920</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/349fc4e7443aa744cbe129f84f28571bf8625155"><code>349fc4e</code></a>
[miniflare] Start local Workflows from Artifacts events (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16246">#16246</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/79b890ab89a62823198a0c551c44ff647c80de4d"><code>79b890a</code></a>
Version Packages (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16863">#16863</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/f005593c1ed0d1f429dafc6919c4335f4d5070c7"><code>f005593</code></a>
[wrangler] Use the local Artifacts simulator when remote is omitted (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16735">#16735</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/cloudflare/workers-sdk/commits/wrangler@4.164.0/packages/wrangler">compare
view</a></li>
</ul>
</details>
<br />

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-20 16:39:54 -08:00
dependabot[bot] 8a2b2f57a3 build(deps): bump @boxd-sh/sdk from 0.2.22 to 0.3.2 in /scripts/typecheck-sandbox-providers (#19572)
Bumps
[@boxd-sh/sdk](https://github.com/boxd-sh/sdk/tree/HEAD/typescript) from
0.2.22 to 0.3.2.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/boxd-sh/sdk/commit/7bf5bf389a49323013fa65d222c15418b75732b6"><code>7bf5bf3</code></a>
CLI 0.3.5, SDKs 0.3.2</li>
<li><a
href="https://github.com/boxd-sh/sdk/commit/f784e0440a26bc0c514f97cdb1ad4d1f44b2a1bf"><code>f784e04</code></a>
Windows guests on self-hosted boxd: UEFI firmware boot, --os windows,
license...</li>
<li><a
href="https://github.com/boxd-sh/sdk/commit/d037b1f6e9c58cde0cdaf31d04a9baf4a7bb0021"><code>d037b1f</code></a>
Remove ListPlans, the retired monthly plan catalog</li>
<li><a
href="https://github.com/boxd-sh/sdk/commit/f2843553695415fcc47f89a300939d411ac9c8dc"><code>f284355</code></a>
api: SetOrgBotProtectionDefault changes existing machines too</li>
<li><a
href="https://github.com/boxd-sh/sdk/commit/9736596397bdc10f6415348d4fd619846a51653a"><code>9736596</code></a>
SDKs 0.3.1: computer use</li>
<li><a
href="https://github.com/boxd-sh/sdk/commit/5cdba5bb69cbfe0885d691440c9ad76c792f02cf"><code>5cdba5b</code></a>
Computer use API and a settable screen size</li>
<li><a
href="https://github.com/boxd-sh/sdk/commit/0ee9437a651c746efd6a8f25d165c01a5b52dc51"><code>0ee9437</code></a>
Show no balance for an org on a custom plan (comped), and cut long org
names ...</li>
<li><a
href="https://github.com/boxd-sh/sdk/commit/9fc94e185819f2de28054b199e91a45efddf5f7f"><code>9fc94e1</code></a>
boxd v2: storage engine, routed network and live migration</li>
<li><a
href="https://github.com/boxd-sh/sdk/commit/c4a3ee3d591977f4ffa36e6f573c9853839205b1"><code>c4a3ee3</code></a>
billing: vCPU bills CPU time used, not the machine's size</li>
<li><a
href="https://github.com/boxd-sh/sdk/commit/d912ff453fdf5a13226d7c4e4f5c49bf490b3e61"><code>d912ff4</code></a>
Raw TCP forwards stop refusing connections, and ExposePort answers once
the p...</li>
<li>See full diff in <a
href="https://github.com/boxd-sh/sdk/commits/v0.3.2/typescript">compare
view</a></li>
</ul>
</details>
<br />

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-20 16:23:41 -08:00
Nicky LeachandClaude Opus 5.5 5371c83152 fix(runner): bundle the Copilot metadata probe as a standalone entrypoint (#22436)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner ships a provider pack with pinned provider
runtimes and small entrypoints
> - Most provider entrypoints, such as the Kimi metadata probe and the
ACPX sidecar, are esbuild bundles that run as one file
> - The Copilot metadata probe is plain `tsc` output that imports
sibling driver modules
> - So an environment that installs only selected provider-pack files
cannot run the Copilot probe
> - This pull request makes the Copilot metadata probe a standalone
verified entrypoint, like the Kimi probe
> - The benefit is that every metadata probe has the same self-contained
shape, and a consumer can ship the probe as one file

## Linked Issues or Issue Description

No public issue exists. This is an enhancement.

**What existing behavior does this improve?**
The Copilot metadata probe entrypoint
(`dist/cli/copilot-metadata-probe.js`, the
`paperclip-runner-copilot-metadata-probe` bin and provider-pack shim).

**Subsystem affected**
Paperclip Runner provider pack build (`packages/paperclip-runner`).

**Current behavior**
The probe is plain `tsc` output. It imports
`../drivers/copilot-metadata-probe.js` and more modules. It fails when a
consumer installs only selected provider-pack artifacts. The Kimi probe
does not have this problem because
`scripts/build-verified-provider-entrypoints.mjs` bundles it.

**Proposed behavior**
The build bundles the Copilot probe into one self-contained file, like
the Kimi probe. The bundle still finds the pinned Copilot assets when it
runs from `dist/cli/`.

**Reason and benefit**
Every metadata probe has the same standalone shape. A consumer can ship
the probe as one file. An alternative was to ship the full
`dist/drivers` tree with the probe, but that keeps the multi-file shape.

**Breaking changes**
None. The bin name, the output path and the probe output do not change.

## What Changed

- `scripts/build-verified-provider-entrypoints.mjs`: add
`copilot-metadata-probe` (source `src/cli/copilot-metadata-probe.ts`,
outputs `dist/cli/copilot-metadata-probe.js` and `.cjs`) with the same
format and flags as `kimi-metadata-probe`. The bundler asserts one
output and only `node:` imports.
- `src/cli/copilot-metadata-probe.ts`: move the top-level `await` into
`main()`. esbuild cannot emit top-level `await` in the CommonJS bundle.
Output and exit codes do not change.
- `src/drivers/acpx/provider-assets-root.ts`: when no package root is
bound, accept the bundled `dist/cli/copilot-metadata-probe.(js|cjs)`
module URL for the `copilot` provider only. This is the same rule the
bundled ACPX sidecar uses. Before this change, the bundle's
`import.meta.url` is the `dist/cli` file, not
`dist/drivers/acpx/copilot-installation.js`, so the module-URL lookup
failed. The bound `PAPERCLIP_ACPX_PROVIDER_PACKAGE_ROOT` path does not
change.
- Tests: a bundler test that runs both bundles alone in a temporary pack
layout, and an asset-root test for the bundled probe path that also
proves other providers do not get the new path.

## Verification

- `node --test scripts/build-verified-provider-entrypoints.test.mjs` (44
tests, includes the new standalone-run test)
- `pnpm exec vitest run src/drivers/acpx/provider-assets-root.test.ts
src/drivers/copilot-metadata-probe.test.ts
src/drivers/acpx/copilot-installation.test.ts`
- `node --test scripts/kimi-metadata-probe.test.mjs
scripts/provider-pack-executable-shims.test.mjs
test/acpx-codex-package-contract.test.mjs`
- `pnpm exec vitest run
server/src/__tests__/copilot-connection-probe.test.ts`
- `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript`
- Manual check on darwin-arm64 with the pinned Copilot closure
installed, child process spawn denied by a preload:
- Bundled `.js` and `.cjs`, no bound root: installation verification
passes and the probe reaches the spawn step.
- Same run without the `provider-assets-root.ts` change:
`COPILOT_INSTALLATION_INVALID`.
- Bundled `.js` with `PAPERCLIP_ACPX_PROVIDER_PACKAGE_ROOT` bound:
installation verification passes.
- `pnpm verify` requires broader coverage (the bundler is a broad
input), so CI supplies it.

## Risks

- Low risk. The bin name, output path and shim do not change. The probe
output and exit codes do not change.
- The new module-URL rule accepts only
`/dist/cli/copilot-metadata-probe.(js|cjs)` and only for `copilot`. It
resolves to the package that holds `dist/cli`, the same as the sidecar
rule.
- The build now also writes `dist/cli/copilot-metadata-probe.cjs`.
Nothing reads it yet.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Provider: Anthropic Claude
- Model: Claude Opus 5.5 (`claude-opus-5-5`)
- Mode: agentic coding with tool use (shell, file edits, test runs)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 16:09:31 -08:00
Devin FoleyandPaperclip ae2afd525b Make harness choices deployment-configurable
Managed deployments need to control harness availability without changing Core policy. Add a generic environment contract and apply it across harness and model selectors, preserving saved supported profiles and validating import, team, and invite selections. Document deployment ownership in agent guidance.

Validation: focused local tests and typechecks passed; all applicable CI checks passed on the reviewed head. Greptile scored 5/5 with no unresolved comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-20 15:45:36 -08:00
DottaandClaude Opus 5.5 104f323bee test(restart-chaos): credit a provably fair first-run queue wait in R1 (#22446)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos`) grades each task
of a wave against rules R1–R6
> - R1 allows the workload budget plus outage time, but it ignores the
time the first run waits for a free agent slot
> - In a staging-mix wave, 32 tasks arrive in about 2.5 minutes and each
agent has 4 slots, so the last arrivals wait 21–25 minutes even with
perfect FIFO dispatch
> - One task (t031) is the tightest in every wave. Its margin went from
+165 s and +142 s to −87 s, and the wave failed R1 for a fair wait
> - This pull request credits the first run's initial queue wait in R1
only when the harness can prove the wait was fair
> - The benefit is that a fair queue no longer fails R1, and a real
dispatch-fairness bug fails with its own reason

## Linked Issues or Issue Description

**What happened?**

In wave `w-20261120T190100-s7106`, task t031 failed
`R1:no_rest_within_budget` by 87 s. Its first assignment run sat queued
from 19:04:31Z to 19:32:43Z. Its opencode agent was at its 4-run limit
for the whole wait, and the run started in the same second that a slot
freed. The R1 budget (`1.5 × turn + 20 min`) does not include queue
wait.

**Expected behavior**

A task whose first run waited fairly for agent capacity is not marked
late because of that wait. A task whose run was overtaken by a newer
run, or that waited while a slot was idle, still fails R1.

**Steps to reproduce**

1. Run a staging-mix wave with 32 arrivals and 4 slots per agent (for
example the plan in `w-20261120T190100-s7106`).
2. Look at the last task assigned to the busiest agent.
3. See that its first run queues for more than 20 minutes and that R1
fails although every slot was busy.

**Paperclip version or commit**

`f7db7312b4` (master)

## What Changed

- `agent-slots.ts`: `agentSlotsFromRows` records each run's `createdAt`,
`invocationSource` and `retryOfRunId`. New `queueWaitFairnessFrom`
decides if a run's queue wait was fair. It returns `unfair` when a newer
fresh assignment run of the same agent started during the wait, or when
the agent had an idle slot for more than 2 s
(`QUEUE_HANDOFF_TOLERANCE_MS`) outside a server outage. Newer follow-ups
and retries rank higher in the dispatcher by design, so they do not
count as overtaking. It returns `unproven` when slot evidence,
`createdAt` or the invocation source is missing, or when the only idle
slot overlapped an outage.
- `grader.ts`: new `firstRun`, `firstRunQueuedMs` and
`firstRunQueueWait`. `gradeR1` always checks fairness (it defaults to
"no slot evidence"). When the task is over its base budget, R1 adds the
first run's initial wait minus any fault overlap if the wait is fair. A
provably unfair wait fails as `unfair_queue_wait`. An unprovable wait
(including a first run that never started) is not credited and the
detail says why. Waits of retries and later runs are never credited.
Tasks inside the base budget grade exactly as before.
- `observer.ts`: the observer does not time a task out while its first
run is still queued (up to one more budget), and it extends the deadline
once by that run's queue wait when it starts, so it watches long enough
for the grader to apply the credit.
- `wave-grade-inputs.ts`: passes the fairness check to `gradeTask`.
- `README.md`: documents the R1 queue-wait credit.
- No product code changes.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
71 files, 865 tests pass.
- `pnpm exec tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean.
- New tests in `grader.test.ts` (`R1 fair queue wait credit`):
- credits a fair 1,509 s FIFO queue wait that left the task 87 s over
the old budget
- fails as an unfair queue wait when a newer run started ahead of the
queued run
- fails as an unfair queue wait when the agent had an idle slot during
the wait
- credits the wait when a newer follow-up or retry that the dispatcher
ranks higher started ahead
  - does not call an idle slot during a server outage unfair
  - says so when the first run never started
  - allows a same-second slot hand-off within the tolerance
  - does not credit a wait it cannot prove fair
  - leaves a short queue wait unchanged
- credits only the first run's initial wait, not outage time inside it
or later retries
- New tests in `observer.test.ts`: extends the deadline once by the
first run's queue wait when it starts; does not time out while the first
run is still queued, up to one more budget.
- Offline re-grade of the saved wave `w-20261120T190100-s7106` with the
new `gradeR1` (all 32 saved R1 verdicts reproduce first):
  - No verdict changes for the other 31 tasks.
- t031 with the wave's saved evidence: `done_too_late 2291s > 2183s
(queue wait not credited: no createdAt for 4 runs …)`. This wave's
`agent-slots.json` predates `createdAt`, and 4 of the 10 runs that
started during the wait are subtask runs that the per-task wake evidence
does not cover. The grader cannot prove FIFO from the saved files, so it
does not credit the wait.
- t031 with those 4 runs taken as created earlier (as the earlier DB
diagnosis found): `pass — done after 2291s (allowed 3701s incl. 1518s
fair queue wait)`.
- New waves record `createdAt` for every run, so the check is complete
from the next wave.
- `--replay` re-executes a task on a live server. It is not a re-grade,
so I did not use it.

## Risks

- Low risk. The change is in the test harness only.
- Tasks inside their base budget grade exactly as before. Only a task
that is over budget can change verdict. An over-budget R1 detail now
also says why a queue wait was not credited; the failure signature is
unchanged.
- The fairness rule models the dispatcher coarsely: only a newer fresh
assignment run counts as overtaking. Readiness and priority are not
compared, and all harness tasks use the same priority.
- An observer can watch a task for longer, up to the first run's queue
wait. This can make a wave a little longer when queues are long.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 16:44:42 -06:00
Devin FoleyandPaperclip 9bc90cb189 Show useful task notification content (#22441)
Task notifications repeated identifiers and omitted useful activity details.

Use task titles, actor and comment previews, and concrete field changes. Share bounded secret redaction per event after viewer access checks. Suppress updates without useful details.

Verified with focused regression tests, package typechecks and server build, and the full green CI run. Greptile scored the final head 5/5 with no unresolved review threads.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-20 14:40:39 -08:00
dependabot[bot] 1ced68c636 build(deps-dev): bump typescript from 5.9.3 to 7.0.2 in /scripts/typecheck-sandbox-providers (#19574)
Bumps [typescript](https://github.com/microsoft/TypeScript) from 5.9.3
to 7.0.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/microsoft/TypeScript/releases">typescript's
releases</a>.</em></p>
<blockquote>
<h2>TypeScript 7.0.2</h2>
<p><a
href="https://devblogs.microsoft.com/typescript/announcing-typescript-7-0/">https://devblogs.microsoft.com/typescript/announcing-typescript-7-0/</a></p>
<p>This tag was originally released at: <a
href="https://github.com/microsoft/typescript-go/releases/tag/typescript%2Fv7.0.2">https://github.com/microsoft/typescript-go/releases/tag/typescript%2Fv7.0.2</a></p>
<h2>TypeScript 6.0.3</h2>
<p>For release notes, check out the <a
href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0/">release
announcement blog post</a>.</p>
<ul>
<li><a
href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&amp;q=milestone%3A%22TypeScript+6.0.0%22">fixed
issues query for TypeScript 6.0.0 (Beta)</a>.</li>
<li><a
href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&amp;q=milestone%3A%22TypeScript+6.0.1%22">fixed
issues query for TypeScript 6.0.1 (RC)</a>.</li>
<li><a
href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&amp;q=milestone%3A%22TypeScript+6.0.2%22">fixed
issues query for TypeScript 6.0.2 (Stable)</a>.</li>
<li><a
href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&amp;q=milestone%3A%22TypeScript+6.0.3%22">fixed
issues query for TypeScript 6.0.3 (Stable)</a>.</li>
</ul>
<p>Downloads are available on:</p>
<ul>
<li><a href="https://www.npmjs.com/package/typescript">npm</a></li>
</ul>
<h2>TypeScript 6.0</h2>
<p>For release notes, check out the <a
href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0/">release
announcement blog post</a>.</p>
<ul>
<li><a
href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&amp;q=milestone%3A%22TypeScript+6.0.0%22">fixed
issues query for TypeScript 6.0.0 (Beta)</a>.</li>
<li><a
href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&amp;q=milestone%3A%22TypeScript+6.0.1%22">fixed
issues query for TypeScript 6.0.1 (RC)</a>.</li>
<li><a
href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&amp;q=milestone%3A%22TypeScript+6.0.2%22">fixed
issues query for TypeScript 6.0.2 (Stable)</a>.</li>
</ul>
<p>Downloads are available on:</p>
<ul>
<li><a href="https://www.npmjs.com/package/typescript">npm</a></li>
</ul>
<h2>TypeScript 6.0.1 RC</h2>
<p>For release notes, check out the <a
href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0-rc/">release
announcement blog post</a>.</p>
<ul>
<li><a
href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&amp;q=milestone%3A%22TypeScript+6.0.0%22">fixed
issues query for TypeScript 6.0.0 (Beta)</a>.</li>
<li><a
href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&amp;q=milestone%3A%22TypeScript+6.0.1%22">fixed
issues query for TypeScript 6.0.1 (RC)</a>.</li>
</ul>
<p>Downloads are available on:</p>
<ul>
<li><a href="https://www.npmjs.com/package/typescript">npm</a></li>
</ul>
<h2>TypeScript 6.0 Beta</h2>
<p>For release notes, check out the <a
href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0-beta/">release
announcement</a>.</p>
<ul>
<li><a
href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&amp;q=milestone%3A%22TypeScript+6.0.0%22+is%3Aclosed+">fixed
issues query for Typescript 6.0.0 (Beta)</a>.</li>
</ul>
<p>Downloads are available on:</p>
<ul>
<li><a href="https://www.npmjs.com/package/typescript">npm</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/microsoft/TypeScript/commit/1e4744d68260a7cb91b62b12edc3f6a2187faaf1"><code>1e4744d</code></a>
Merge branch 'main' into ts7-release</li>
<li><a
href="https://github.com/microsoft/TypeScript/commit/a5a219c3b5da0db4fa0ecf6c0b1f588c9af9c669"><code>a5a219c</code></a><code>microsoft/typescript-go#4558</code></li>
<li><a
href="https://github.com/microsoft/TypeScript/commit/ecfe30dce91368d52c9a49b6095bb0b673a238f8"><code>ecfe30d</code></a>
Update status localization</li>
<li><a
href="https://github.com/microsoft/TypeScript/commit/5de25b5f8fec2ca35eadaed041f1f06d2e214895"><code>5de25b5</code></a>
Hide executable name in TypeScript status</li>
<li><a
href="https://github.com/microsoft/TypeScript/commit/d7ce74a75da2b80e8201506a1599c06549432b93"><code>d7ce74a</code></a>
Show bundled TypeScript version for packaged servers</li>
<li><a
href="https://github.com/microsoft/TypeScript/commit/29be66a607707f90d7a53103a4469bb3015a4d54"><code>29be66a</code></a>
Correct TS 7 release version to 7.0.2</li>
<li><a
href="https://github.com/microsoft/TypeScript/commit/ed2bd1bfa4aac5211ce4bc58fcd1313c7eddc8ff"><code>ed2bd1b</code></a>
Merge branch 'main' into ts7-release</li>
<li><a
href="https://github.com/microsoft/TypeScript/commit/887307575c58ea640dbeba3b4e8fdb6347cd3044"><code>8873075</code></a>
Bump the github-actions group across 1 directory with 3 updates
(microsoft/ty...</li>
<li><a
href="https://github.com/microsoft/TypeScript/commit/9427131ae2d4e230a90ee8a09daac4e75da3e311"><code>9427131</code></a>
Set up stable / nightly extension split, other prep
(microsoft/typescript-go#...</li>
<li><a
href="https://github.com/microsoft/TypeScript/commit/d4eaca5460a1f5f02a829e62706794b0a6fb903e"><code>d4eaca5</code></a><code>microsoft/typescript-go#4549</code></li>
<li>Additional commits viewable in <a
href="https://github.com/microsoft/TypeScript/compare/v5.9.3...v7.0.2">compare
view</a></li>
</ul>
</details>
<details>
<summary>Maintainer changes</summary>
<p>This version was pushed to npm by <a
href="https://www.npmjs.com/~microsoft1es">microsoft1es</a>, a new
releaser for typescript since your current version.</p>
</details>
<br />

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-20 21:53:47 +00:00
dependabot[bot] f7db7312b4 build(deps): bump e2b from 2.62.1 to 2.64.0 in /scripts/typecheck-sandbox-providers (#20197)
Bumps [e2b](https://github.com/e2b-dev/e2b/tree/HEAD/packages/js-sdk)
from 2.62.1 to 2.64.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/e2b-dev/E2B/blob/main/packages/js-sdk/CHANGELOG.md">e2b's
changelog</a>.</em></p>
<blockquote>
<h2>2.64.0</h2>
<h3>Minor Changes</h3>
<ul>
<li>
<p>6a976f3: <code>e2b template list</code> and <code>e2b template
delete</code> now use the SDK's <code>Template.list()</code> and
<code>Template.delete()</code>. <code>TemplateInfo</code> gains
<code>maxCpuCount</code> (<code>max_cpu_count</code> in Python) for
resizable templates.</p>
<p>Breaking: <code>e2b template list -f json</code> now prints the SDK's
<code>TemplateInfo</code> objects (<code>templateId</code>,
<code>buildId</code>, <code>names</code>, …) instead of the raw API
response (<code>templateID</code>, <code>buildID</code>,
<code>aliases</code>, <code>createdBy</code>, …), and the pretty table
drops the <code>Created by</code> column (the API deprecated the
creator's email).</p>
</li>
<li>
<p>1b3b2a4: Add <code>shell</code> option to <code>commands.run</code>
to run the command in a custom shell, e.g. <code>shell: ['/bin/sh',
'-c']</code> runs <code>/bin/sh -c &lt;cmd&gt;</code> (default stays
<code>['/bin/bash', '-l', '-c']</code>)</p>
</li>
<li>
<p>6ee9a4b: Add resizable templates: build a template with
<code>maxCpuCount</code> and pick each sandbox's <code>cpuCount</code>
on create. <code>e2b template list</code> shows a <code>Max vCPUs</code>
column.</p>
</li>
<li>
<p>106efea: The API client accepts an <code>Authorization</code> header
in place of an API key. Requests can now authenticate with an OAuth
access token passed through <code>apiHeaders</code> (JS) or
<code>api_headers</code> (Python), together with the
<code>X-Team-ID</code> header that selects the team:</p>
<pre lang="ts"><code>await Sandbox.create({
apiHeaders: { Authorization: `Bearer ${accessToken}`, 'X-Team-ID':
teamId },
})
</code></pre>
<pre
lang="python"><code>Sandbox.create(api_headers={&quot;Authorization&quot;:
f&quot;Bearer {access_token}&quot;, &quot;X-Team-ID&quot;: team_id})
</code></pre>
</li>
<li>
<p>c513ae1: Add <code>Template.delete(nameOrId)</code> (JS) and
<code>Template.delete(name_or_id)</code> /
<code>AsyncTemplate.delete(name_or_id)</code> (Python) to delete a
template or snapshot; returns <code>false</code> when it isn't found.
<code>Sandbox.deleteSnapshot</code> /
<code>Sandbox.delete_snapshot</code> now share its implementation and
raise <code>TemplateError</code> / <code>TemplateException</code>
(subclasses of <code>SandboxError</code> /
<code>SandboxException</code>) on failure</p>
</li>
<li>
<p>e50db00: Add <code>Template.list()</code> returning a
<code>TemplatePaginator</code> (<code>AsyncTemplatePaginator</code> for
<code>AsyncTemplate</code>) that pages through the project's templates
as <code>TemplateInfo</code>, like <code>Sandbox.list()</code> and
<code>Secret.list()</code>.</p>
</li>
</ul>
<h3>Patch Changes</h3>
<ul>
<li>cf7208a: Bump <code>@connectrpc/connect</code> and
<code>@connectrpc/connect-web</code> to ^2.3.0 (HTTP/2 stream cleanup
and cancellation fixes) and <code>glob</code> to ^14.0.3</li>
<li>Updated dependencies [aec4543]
<ul>
<li><code>@​e2b/dockerfile-utils</code><a
href="https://github.com/0"><code>@​0</code></a>.1.2</li>
</ul>
</li>
</ul>
<h2>2.63.0</h2>
<h3>Minor Changes</h3>
<ul>
<li>a8155c4: Add <code>strict</code> to <code>pause()</code> (default
<code>false</code>). With <code>strict: true</code> /
<code>strict=True</code> a pause is refused with
<code>ServiceBusyError</code> / <code>ServiceBusyException</code> while
the sandbox's guest agent does not respond, and the sandbox keeps
running. Without it the API's default applies: the memory snapshot is
taken anyway and marked. Sandbox info exposes the mark as
<code>envdUnresponsiveAtPause</code> /
<code>envd_unresponsive_at_pause</code>, and resuming such a sandbox
without <code>onResume: 'reboot'</code> /
<code>on_resume=&quot;reboot&quot;</code> now throws
<code>GuestUnresponsiveError</code> /
<code>GuestUnresponsiveException</code>.</li>
</ul>
<h2>2.62.2</h2>
<h3>Patch Changes</h3>
<ul>
<li>08f05bb: Fix <code>TypeError: fetch failed</code> /
<code>NGHTTP2_ENHANCE_YOUR_CALM</code> on concurrent
<code>files.write()</code> calls. Node's HTTP/2 client queues each
request body in full against a 10 MB per-session memory budget and
resets the stream whose response arrives while it is exhausted; since
undici 8 every envd request of a process shares one session, so parallel
or large uploads reset each other and the reads next to them.
<code>files.write()</code> uploads now go over HTTP/1.1 on a pool of
their own (one request per connection, no shared budget;
<code>E2B_ENVD_UPLOAD_CONNECTIONS</code>, default 50), and an envd
dispatcher whose HTTP/2 session Node reports as out of memory is retired
and replaced instead of failing every later request. Nothing is retried:
a failed write may already have been applied.</li>
<li>1ad3f76: Template builds hash <code>copy()</code> files in 1 MB
chunks instead of reading each file fully into memory, cutting peak
memory for large files. Hashes are unchanged, so existing build caches
stay valid.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/e2b-dev/E2B/commit/607e4a6d6fca19ad49c090b40a6db0853165af2a"><code>607e4a6</code></a>
[skip ci] Release new versions</li>
<li><a
href="https://github.com/e2b-dev/E2B/commit/1b3b2a4b078ffb4c3f6197e03019686b7da644dd"><code>1b3b2a4</code></a>
feat(sdk): add shell option to commands.run (<a
href="https://github.com/e2b-dev/e2b/tree/HEAD/packages/js-sdk/issues/2138">#2138</a>)</li>
<li><a
href="https://github.com/e2b-dev/E2B/commit/02176c1a00013ddd8f128a3b5766f0f4ae1b989b"><code>02176c1</code></a>
test: stabilize auto-resume pause and CLI template pagination tests (<a
href="https://github.com/e2b-dev/e2b/tree/HEAD/packages/js-sdk/issues/2129">#2129</a>)</li>
<li><a
href="https://github.com/e2b-dev/E2B/commit/6a976f3d3d1c12a7baac348b3255abefe10b874a"><code>6a976f3</code></a>
feat(cli): use SDK Template.list and Template.delete in template
commands (<a
href="https://github.com/e2b-dev/e2b/tree/HEAD/packages/js-sdk/issues/2">#2</a>...</li>
<li><a
href="https://github.com/e2b-dev/E2B/commit/77343ca2d36081fb47ca78f7addf1c2b2dc69241"><code>77343ca</code></a>
refactor(cli): type the workspace-api client with openapi-typescript +
openap...</li>
<li><a
href="https://github.com/e2b-dev/E2B/commit/e50db000621409e5c4021ac6b2b80156441a0f69"><code>e50db00</code></a>
feat(sdk): add Template.list() returning a TemplatePaginator (<a
href="https://github.com/e2b-dev/e2b/tree/HEAD/packages/js-sdk/issues/2099">#2099</a>)</li>
<li><a
href="https://github.com/e2b-dev/E2B/commit/c513ae17f64b2894d4722d5b2215f52a47f331a2"><code>c513ae1</code></a>
feat(sdk): add Template.delete (<a
href="https://github.com/e2b-dev/e2b/tree/HEAD/packages/js-sdk/issues/2098">#2098</a>)</li>
<li><a
href="https://github.com/e2b-dev/E2B/commit/6ee9a4bd3c6a6f7ed69ca27138d28acda970dc39"><code>6ee9a4b</code></a>
feat(sdk,cli): resizable templates (<a
href="https://github.com/e2b-dev/e2b/tree/HEAD/packages/js-sdk/issues/2117">#2117</a>)</li>
<li><a
href="https://github.com/e2b-dev/E2B/commit/106efea7e260f752b478709e5fbc5ce0ce6a35ce"><code>106efea</code></a>
feat(sdk): accept an Authorization header in place of an API key (<a
href="https://github.com/e2b-dev/e2b/tree/HEAD/packages/js-sdk/issues/2102">#2102</a>)</li>
<li><a
href="https://github.com/e2b-dev/E2B/commit/cf7208ad85d5a747ba50ddf4e3bbdd364e4851bf"><code>cf7208a</code></a>
chore(deps): scheduled patch/minor dependency updates for js-sdk / cli /
pyth...</li>
<li>Additional commits viewable in <a
href="https://github.com/e2b-dev/e2b/commits/e2b@2.64.0/packages/js-sdk">compare
view</a></li>
</ul>
</details>
<br />

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-20 12:03:33 -08:00
Devin FoleyandPaperclip 2898acee2d fix: exclude responsibility repairs from Mine participation (#22434)
Exclude responsibility-repair audit entries from personal task
participation. Maintenance attributed to an operator no longer pulls
unrelated tasks into Mine, unread badges, or task notifications.

Preserve audit history and genuine participation through creation,
assignment, comments, deliberate edits, and pending requests. Add database
regressions and document the rule.

Verified with 174 targeted tests, server typecheck, fully green CI,
and Greptile 5/5 with no unresolved review comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-20 11:28:11 -08:00
dependabot[bot] 0e5e7a9ea3 build(deps-dev): bump @cloudflare/workers-types from 4.20260702.1 to 5.20261114.1 in /scripts/typecheck-sandbox-providers (#20198)
Bumps [@cloudflare/workers-types](https://github.com/cloudflare/workerd)
from 4.20260702.1 to 5.20261114.1.
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/cloudflare/workerd/commits">compare
view</a></li>
</ul>
</details>
<br />

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-20 11:01:07 -08:00
ef18d71672 fix(workspaces): reopen an expired workspace after its move to remote storage (#22429)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a run starts, Paperclip can reopen a task's archived
execution workspace
> - Git cache expiration removes the local files of an idle workspace
and saves a local restore recipe
> - Remote workspace storage migration can later move that expired
workspace into a saved snapshot. The migration changes the workspace
`cwd` to `/workspaces/<id>`, but it keeps the local restore recipe. That
recipe still names the removed worktree path
> - Run setup reads the recipe through `readGitCacheExpiration` before
it reopens the workspace. The recipe no longer matches the workspace
`cwd`, so validation fails and every run on that task fails in setup
> - This pull request makes run setup read only the current lifecycle's
expiration marker for a workspace in remote storage
> - The benefit is that the task reopens from its saved snapshot. The
reopen path already does this for remote-storage workspaces

## Linked Issues or Issue Description

Refs #22419 (gave this failure its own error code,
`git_cache_restoration_failed`). Refs #19736 (restore diagnostics). I
found no open PR for this cause.

**What happened?**
On staging, runs on some tasks fail during setup with "The saved
workspace commit could not be restored. Check repository access and
retry." The stack is `selectHeartbeatEnvironment` →
`readGitCacheExpiration` → `readGitCacheRestoration`. The task's
workspace is archived with `cleanupReason:
reproducible_git_cache_expired`. It also has a `remoteWorkspace`
authority from the storage migration, and `cwd` is `/workspaces/<id>`.
Its `gitCacheRestoration.worktreePath` is still the old local worktree
path. `readGitCacheRestoration` requires `recipe.worktreePath ===
workspace.cwd`, so it throws. Automatic recovery repeats the run and the
run fails again.

**Expected behavior**
Run setup sees that git cache expiration removed the workspace, and it
reopens the workspace. `reopenClosedIsolatedExecutionWorkspaceForIssue`
restores a workspace with a remote authority from its snapshot and does
not read the local recipe.

**Steps to reproduce**
1. Archive an isolated workspace with a completed `gitCacheExpiration`
marker and a `gitCacheRestoration` recipe.
2. Apply what the remote storage migration does: set `remoteWorkspace`
authority and set `cwd` to `/workspaces/<id>`.
3. Start a run on the bound task. On current master,
`selectHeartbeatEnvironment` throws `GitCacheRestorationFailure`.

## What Changed

- `server/src/services/heartbeat/environment-selection.ts`: add
`gitCacheExpirationRemovedWorkspace`. For a workspace without a remote
authority, it uses `readGitCacheExpiration` as before, with the full
recipe check. For a workspace with a remote authority, it reads only the
`gitCacheExpiration` marker. The marker must have the expected schema,
`workspaceRemoved: true`, and a `generation` that equals the current
`lifecycleGeneration`. A marker from an earlier lifecycle does not start
a reopen.
- `server/src/services/heartbeat/environment-selection.test.ts`: add two
tests that use the real `readGitCacheExpiration`, a migrated workspace
row and a committed snapshot. One test runs the real reopen operation
through `selectHeartbeatEnvironment`. It checks that the same workspace
becomes active, keeps its `/workspaces/<id>` cwd, stays bound to the
issue, and that `lifecycleGeneration` advances from 3 to 4. One test
checks that a marker from an earlier lifecycle does not start a reopen.

## Verification

- Red/green: on unmodified master, both new tests fail (rechecked after
the review change) with "The saved workspace commit could not be
restored" and the same stack as the staging events. With the fix, they
pass.
- `pnpm exec vitest run
src/services/heartbeat/environment-selection.test.ts`: 27 passed.
- `pnpm exec vitest run src/__tests__/execution-workspace-reopen.test.ts
src/__tests__/git-workspace-restoration-diagnostic.test.ts`: all pass.
(When I ran the two files together, one reopen test hit its 25 s
timeout. It passes alone, and that file does not import the changed
module.)
- `pnpm --filter @paperclipai/server typecheck` (all layers): passes.

## Risks

- Low risk. The change only affects archived workspaces that have a
remote authority. Local workspaces keep the full recipe check.
- A remote-storage workspace now reopens through the existing remote
path. That path checks the saved snapshot first
(`inspectRemoteWorkspacePersistence`) and returns `rebuild_failed` if
the snapshot is not ready. So a missing snapshot still fails the run
with a clear reason.
- Not in scope: the migration keeps the stale local recipe in metadata.
Other readers of that recipe do not run for remote-storage workspaces in
this path.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, as the Paperclip
Sentry triage agent, with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documentation describes this check)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 10:47:43 -08:00
github-actions[bot]andlockfile-bot f4a3f85b89 chore(lockfile): refresh pnpm lockfiles (#22435)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.
- Resync the SDK public source proof's lock pins from the refreshed
lock.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when it verifies
every added version on npm and all versions outside the harness
allowlist are at least 3 days old. Otherwise the PR waits for a
maintainer, and the daily refresh tries again. The bot merges only the
exact commit this workflow generated from manifests already reviewed on
master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-20 18:33:27 +00:00
Devin FoleyandClaude Opus 5.5 203c10d212 fix(api): tell agents that issue count requires attention=blocked (#17596)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents find REST routes through the runner API catalog
(`search_api`) and call them with `call_api`
> - The catalog shows `GET /api/companies/{companyId}/issues/count` as
"Count issues in a company". It gives no description and no query
parameters
> - The route only counts blocked issues that need attention. It returns
400 when the request does not include `attention=blocked`
> - Agents read the summary as a generic counter. They call the route
without the filter and get a 400. A hosted instance reported these 400
responses as failed HTTP operations
> - This pull request adds the route contract to the Paperclip skill API
reference. It also regenerates the catalog enrichment, so `search_api`
returns the contract with the operation
> - The benefit is that agents get the required filter and the correct
alternative before they call the route. Runtime validation does not
change

## Linked Issues or Issue Description

No public issue exists. I searched open pull requests for `issues/count`
and found no duplicate.

**What happened?**

An agent called `GET /api/companies/{companyId}/issues/count` with only
a `labelId` filter. The server returned `400 issues/count currently
requires attention=blocked`. The catalog entry that the agent read
showed only the summary "Count issues in a company", so the agent had no
way to know the filter was required.

**Expected behavior**

The catalog tells the agent that `attention=blocked` is required. It
also lists the accepted filters, says that `limit` and `offset` are
rejected, and tells the agent how to count anything else by paging
through the issue list.

**Steps to reproduce**

1. As an agent, call `search_api` with the query `count issues`.
2. Read the `GET /api/companies/{companyId}/issues/count` result. It has
no description and no `skillReference`.
3. Call the route through `call_api` with `query: { labelId:
"<label-id>" }`.
4. The server returns 400.

**Paperclip version or commit**

`master` at 2e47c42fad.

**Deployment mode**

Hosted. The behaviour is the same in every deployment mode, because the
route validation and the catalog are core code.

## What Changed

- `skills/paperclip/references/api-reference.md`:
- Add a row for `GET /api/companies/:companyId/issues/count` to the
Issues table. The row says that `?attention=blocked` is required and
that other requests return 400. It names the accepted issue-list filters
and says that `?limit=` and `?offset=` are rejected. For any other
count, it says to page through `GET /api/companies/:companyId/issues`
with `?limit=` (default 500, maximum 1000) and increasing `?offset=`
until a page is empty. A short page is not the end, because the list
route filters rows for restricted actors after the database applies the
limit.
- Merge the two `GET /api/issues/:issueId/heartbeat-context` rows into
one. The generator keeps only the last row for a route, so the first
description never reached the catalog. Merging also keeps every later
heading on its current line. The generated capability contract and
inventory record skill headings by line number, so they stay unchanged.
- `server/src/services/native-runtime/runner-api-reference.ts`:
regenerate with `node scripts/generate-runner-api-reference.mjs`. The
count route gets a new entry, which the catalog attaches to the
operation as `skillReference`. The heartbeat-context entry gets the
merged description.
- `server/src/services/native-runtime/runner-api-reference.test.ts`: new
regression test. It searches the catalog for `count issues` and checks
that the count operation's `skillReference` describes the contract and
the paging rule.

## Verification

- `node scripts/generate-runner-api-reference.mjs --check` passes. The
generated file matches the skill reference.
- `node
packages/paperclip-runner/scripts/generate-capability-contract.mjs
--check` and `node
packages/paperclip-runner/scripts/check-capability-inventory.mjs` pass.
On the first head they failed with "Generated contract drift:
generated/capability/capabilities.yaml", the cause of the Typecheck,
Build, Canary and Docker context failures there.
- The new test failed on `master` (no `skillReference` on the count
operation) and passed on the first head. That run covered the
runner-api-reference, runner-api, hiring-operational-examples and
openapi-routes suites (2019 + 1 tests) and `paperclip-skill-utils` (21
tests).
- The sandbox used for this revision did not have room to install
workspace dependencies, so I did not rerun vitest locally. Instead I
checked the regenerated catalog entry for each string the test asserts.
CI runs the full suites.
- I did not run `pnpm -r typecheck` or the full suite locally. CI runs
both.

## Risks

- Low risk. The change is documentation plus generated catalog entries.
- Route behaviour, authorization and the OpenAPI document do not change.
- The OpenAPI summary for the route stays "Count issues in a company". A
later change can also register the route in
`server/src/routes/openapi.ts` with its query schema and error
responses.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used as an agent with
tool use (shell, test execution and the GitHub connector).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 10:32:11 -08:00
Devin FoleyandPaperclip 42c8fee59b fix(run-log): give each object-journal read attempt its own deadline (#21508)
Give each object-journal read attempt its own deadline.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
2026-11-20 10:31:58 -08:00
1a50bd550e fix(sentry): report a saved-sandbox resume provider outage only after its retry decision (#21510)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A run that uses a saved (reusable) sandbox resumes that sandbox
during setup, before the provider work starts.
> - When the sandbox provider has an outage, the resume gets an HTTP 5xx
answer. Setup fails with `reusable_sandbox_resume_failed`. Paperclip
keeps the lease and schedules the bounded bootstrap retry, which resumes
the same lease when the provider recovers (#20626).
> - The setup path writes the failed status before it decides the retry.
That write sends the Sentry run-failure report. Thus each attempt during
an outage creates an "unexpected failure" event, also when the retry
recovers the run.
> - The reviewed chat setup wait had the same defect. #20205 fixed it
with a `pending` report handshake.
> - This pull request uses the same handshake for a sandbox resume that
the provider answered with an HTTP 5xx.
> - The benefit: Sentry shows a sandbox resume failure only when the
retry cannot recover it.

## Linked Issues or Issue Description

**What happened?**

Staging Sentry recorded `EnvironmentRunError: Failed to acquire lease
for environment "… Sandbox" (sandbox): Reusable sandbox lease "…" could
not be resumed (cause=provider_unavailable status=502); the lease was
preserved and no replacement was created.` The events came in bursts: 5
different tasks in one minute (HTTP 503/504), and 2 tasks in three
minutes on a different day (HTTP 502). All events had
`failure_phase=setup` and no provider work. In the source run, the
scheduled bootstrap retry got past the resume step.

**Expected behavior**

A provider outage that the bounded retry owns does not create a Sentry
event. A retry that is exhausted, not scheduled, or that fails to
schedule still creates one event. A refusal of the sandbox (an HTTP 4xx,
or no status) still reports as before.

**Steps to reproduce**

1. Use an agent with a reusable sandbox environment that has a saved
lease.
2. Make the sandbox provider answer the resume with HTTP 502.
3. Start a run. Setup fails with `reusable_sandbox_resume_failed` and
schedules the bootstrap retry.
4. See a Sentry run-failure event for each attempt.

**Paperclip version or commit**

`2756be2f32` (staging releases `d0057ee384`, `1d153f62b6`, `e23549cd9c`
also show the events).

I searched for duplicate PRs. #20626 added the retry and safe
replacement for a sandbox that will not resume. It does not change when
the failure is reported. #20205 is the precedent for the reviewed chat
wait.

Fixes PAPERCLIP-BACKEND-77

## What Changed

- `server/src/services/reusable-sandbox-resume-error.ts`:
`ReusableSandboxResumeError.providerOutage` is true only when the closed
HTTP status of the provider answer is 5xx. The text of a gateway page
does not decide.
- `server/src/services/heartbeat/run-completion.ts`: For a provider
outage, the failed-status write sets `sandboxResumeRetry: "pending"`.
After the retry decision, the setup path reports the run once with
`scheduled` or `not_scheduled`. The deferred-report code is shared with
the reviewed chat wait.
- `server/src/services/run-failure-report.ts` and
`run-failure-diagnostics.ts`: A `pending` sandbox resume write does not
report. In `unexpected` mode, a `scheduled` retry of a failed
`reusable_sandbox_resume_failed` run does not report. `all` mode still
reports it.
- Tests: `run-failure-report-sandbox-resume.test.ts` (new) covers the
report filter. `run-completion-reviewed-chat-report.test.ts` now also
covers the setup path for 502 and 504 with a scheduled retry, an
exhausted retry, no retry, a scheduler failure, a 4xx refusal and a
refusal without a status.

## Verification

- `vitest run
src/services/__tests__/run-failure-report-sandbox-resume.test.ts
src/services/heartbeat/run-completion-reviewed-chat-report.test.ts
src/services/__tests__/run-failure-report.test.ts`: 308 tests pass
(embedded Postgres).
- Red check: before the fix, the 6 new outage cases fail, because the
failed-status write reports immediately.
- Earlier, on the base commit, the related run-completion, retry
scheduling, recovery and run-failure-report suites passed (407 tests).
- `tsc --noEmit -p server/tsconfig.json`: no errors in server sources.
- The full workspace install did not fit on the agent sandbox disk, so
CI runs the complete affected-test selection.

## Risks

- Low risk. Run status, error code, the retry and lease handling do not
change. Only the time of the Sentry report changes.
- If the process stops between the failed-status write and the retry
decision, no report is sent for that attempt. The reviewed chat wait has
the same accepted behavior. A later terminal failure of the retry still
reports.
- 4xx and unclassified resume failures keep the report on the
failed-status write.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code
agent harness with tool use (shell, file edits, Sentry and GitHub
connectors).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 09:49:17 -08:00
88469bd2bc fix(retry): wait for the failed run's sandbox to stop before repeating its chat request (#22423)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task from a connected chat (for example a GitHub pull request
review) can fail. Retry then repeats the exact failed chat request in a
new run
> - Before that run starts the agent, it checks again that the failed
run's executor is stopped. For a remote sandbox, the proof is a stop
receipt on the sandbox lease
> - Workspace recovery can restart the failed run's sandbox at the same
time, to save its files. While it does this, the lease has no stop
receipt
> - The check then refused the repeated request. The run failed for
good, although recovery stopped the sandbox again a few seconds later
> - This pull request makes the repeated request wait while a stop of
that sandbox is in progress, and then ask the retry authority again
> - The benefit is that a correct Retry is not lost because of a short
overlap with workspace recovery

## Linked Issues or Issue Description

Refs #22414. That PR fixes the next step of the same incident: the
system replacement run that copied the consumed retry selector. This PR
fixes the first refusal. The two PRs change different files and are
independent.

**What happened?**

On a hosted staging deploy, a GitHub review reply task failed
(`native_continuation_requires_reconciliation`). Its Daytona sandbox
went into workspace recovery. A Retry started a new run that repeats the
exact chat request, in a fresh isolated workspace. The run passed
admission and its first execution check. After about 14 seconds of
workspace preparation, the `dispatching` recheck refused it with 409
`chat_failed_run_retry_not_authorized` ("This chat request can't be
repeated exactly as it was sent."). No provider work started. The
recovery reported "Saved work recovered" 12 seconds later. The GitHub
reply request was not answered.

**Expected behavior**

The repeated request waits while the failed run's sandbox stops again,
and then continues. If the stop does not settle, the request is refused
as before.

**Steps to reproduce**

1. Make a native chat task fail on a remote sandbox with a recorded stop
receipt.
2. Start workspace recovery for that run.
`prepareStoppedSandboxForRecovery` moves the lease to `pending_cleanup`
and removes `remoteExecutionTermination` while the sandbox restarts.
3. Press Retry during that interval.
`taskChatRetryExecutionStopped(source)` is false, so
`assertFailedNativeRetryState` refuses the run at the `dispatching`
recheck.

The new test does steps 1 and 2 with the real lease transitions on an
embedded Postgres database.

**Paperclip version or commit**

`201abacd6ee5` (the staging release in the incident).

## What Changed

- Add
`server/src/services/chat-channels/failed-chat-retry-source-stop-wait.ts`:
- `failedChatRetrySourceStopPending()` is true when a non-local lease of
the failed run is `pending_cleanup` with a valid stop-only cleanup
intent (`readStopOnlyCleanup`). Workspace recovery holds such an intent
while it restarts and stops the sandbox.
- `awaitFailedChatRetrySourceStop()` runs the authorization again every
5 seconds, for up to 3 minutes, while that stop is pending. It reads the
pending stop before and after each refused attempt. If a stop was
pending before the attempt but is gone after it, the stop settled while
the refused transaction held no locks, so it reads the authorization
once more at once. If no stop is pending before or after, it returns the
refusal after one attempt, exactly as before. It waits only for the
reason-less containment refusal. It does not wait for a reasoned
refusal, a mismatched retry selector
(`FailedChatRunRetryAuthorizationError`), or any other error. It stops
when the run is stopped, and it returns the last refusal at the
deadline.
- In `heartbeat.ts`, both execution-phase checks of an exact replay (the
first check and the `dispatching` recheck) use this wait. The run log
records one line when the wait starts.

The wait grants no authority. Each attempt still runs the unchanged
`authorizeFailedChatRunRetryWake`. The controller lease is checked on
each attempt, as before.

## Verification

- `cd server && npx vitest run
src/services/chat-channels/failed-chat-retry-source-stop-wait.test.ts`:
10 tests pass. They use real lease rows and the real
`prepareStoppedSandboxForRecovery`,
`restoreSandboxStopAfterFailedRestart` and
`taskChatRetryExecutionStopped`.
- Red check: with the helper changed to make one attempt only (the old
behavior), the wait test fails with the incident's refusal: `promise
rejected "Error: This chat request can't be repeated…" { status: 409,
details: { code: 'chat_failed_run_retry_not_authorized' } }`. The abort
test also fails. Then I restored the helper.
- Red check for the review finding: with the old "no pending stop, so
refuse" line, the new test that commits the stop receipt between the
refusal and the pending-stop read fails with the old refusal. It passes
with the final read.
- A first version of that fix read the authorization again after every
refusal. CI showed this was wrong: `durable-chat-wakeup.test.ts` counts
authorization calls, and a budget-hold case in
`chat-channels.integration.test.ts` changed outcome. I reproduced both
failures locally with that version. The current version rechecks only
when a stop was pending, and those tests pass:
`durable-chat-wakeup.test.ts` (51), the CI-failing chat integration
cases by name (73), the retry-filtered chat integration run (261), and
the focused suites (300).
- Safety cases in the same file: refusal at the deadline when the
sandbox stays held; immediate refusal when no stop is in progress (an
active lease whose executor can still run); no wait for a reasoned
refusal, a selector mismatch, or another error; no wait for a run that
is not an exact replay; the wait stops when the run is stopped.
- Focused existing suites pass: `authorization-run-read.test.ts`,
`wake-queue/application/use-cases.test.ts`,
`__tests__/durable-chat-wakeup.test.ts`, `task-retry.test.ts`,
`budget-resume-continuation.test.ts` (299 tests in total, including the
new file).
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify` selects almost the full suite because `heartbeat.ts` is
a shared module. I did not run that locally. Required CI supplies that
coverage.
- I did not do a live staging reproduction. The incident timing cannot
be produced on demand. The test reproduces the lease state that the
refusal saw.

## Risks

- A repeated request can now hold its own prepared sandbox for up to 3
more minutes before it fails. This happens only while a stop of the
failed run's sandbox is in progress.
- The wait can not let a second executor run. It changes only when the
check runs, and every attempt uses the unchanged authority.
- If workspace recovery takes longer than 3 minutes, the run is refused
as before.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
needed: the code and tests describe the behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 08:47:32 -08:00
Nicky LeachandClaude Opus 5.5 dce4d47f60 feat(plugins): carry Grok, Copilot and OpenCode credentials to plugin Runners (#22359)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Environment-driver plugins can run `paperclip_runner` tasks on hosts
that the plugin owns
> - Core transfers the selected managed AI credential to the plugin as a
typed task credential
> - The credential contract has only two types: a Codex auth file and a
direct Claude credential
> - Grok, GitHub Copilot and OpenCode agents in a task-capable plugin
environment therefore fail closed, although the Runner already reads
their credentials from its environment
> - This pull request adds one generic, allowlisted environment
credential for these three harnesses and accepts the Runner's Grok
launch settings
> - The benefit is that plugin-provided Runner execution can run Grok,
Copilot and OpenCode agents with their selected connection

## Linked Issues or Issue Description

**Subsystem affected**

Plugin environment tasks (`packages/plugins/sdk` task contract) and
plugin-provided Runner execution in `server`.

**Problem or motivation**

An agent with the Grok, Copilot or OpenCode harness cannot run in a
plugin environment that runs Runner tasks. Core stops at credential
preparation with "Plugin Runner does not support the selected credential
provider". A Grok agent fails before that point too. The Runner emits
`grokEffort` and `grokDisableWebSearch` in its launch requirements (Core
turns off Grok web search by default), but the strict `settings` schema
of `environmentRunnerLaunchRequirementsSchema` rejects both keys.

**Proposed solution**

Core transfers the selected connection's credential variables for these
harnesses through one typed `provider-env` credential. Each provider has
a fixed allowlist of complete variable sets. The plugin sets the
variables in the Runner process environment.

**Alternatives considered**

One credential type per harness (`grok-env`, `copilot-env`,
`opencode-env`). This adds a type for each new harness. One generic type
with an exported allowlist keeps the same strictness with less surface.

**Roadmap alignment**

The roadmap lists Grok and OpenCode as supported harnesses. This change
lets plugin environments run them. It adds no new product concept.

**Additional context**

Grok subscription login (an auth file with refresh and writeback) is out
of scope. It still fails closed, with an error that names the xAI API
key as the supported option.

## What Changed

- `packages/plugins/sdk`: Added the `provider-env` task credential: `{
type: "provider-env", provider: "grok" | "copilot" | "opencode",
environment }`. The environment must contain exactly one complete
variable set for the provider and no other variable. Values are
non-empty, NUL-free and at most 65,536 characters, as for the other
credential types.
  - `grok`: `XAI_API_KEY`
  - `copilot`: `COPILOT_GITHUB_TOKEN`
- `opencode`: `OPENROUTER_API_KEY`, or `PAPERCLIP_AI_PROVIDER_URL` with
`PAPERCLIP_AI_PROVIDER_KEY`
- `packages/plugins/sdk`: Exported the allowlist as
`ENVIRONMENT_TASK_PROVIDER_CREDENTIAL_KEY_SETS` and the
`PluginEnvironmentTaskCredentialProvider` type. `claude-env` and
`codex-auth-json` do not change.
- `packages/plugins/sdk`: Added `grokEffort` (`minimal` to `xhigh`) and
`grokDisableWebSearch` (boolean) to the launch requirements `settings`,
with the types that the Runner uses.
- `server`: `preparePluginRunnerCredentials` now transfers the selected
connection's variables for the `grok`, `copilot` and `opencode`
harnesses, with the same account identity semantics as the Claude
branch. It refuses a Grok subscription, a Grok base URL override, and an
incomplete or ambiguous set. The error never contains a credential
value.
- `server`: The submit check now matches a `provider-env` credential to
the task harness. Before, it treated every non-Codex credential as
Anthropic.
- Drift test:
`server/src/__tests__/plugin-runner-credential-names.test.ts` checks the
SDK key sets against the Runner. The SDK list stays written by hand, and
the Runner keeps its own lists of forwarded names. For `grok` and
`copilot`, each name must be in `ACPX_CREDENTIAL_NAMES`, survive sidecar
admission with the session binding that the Runner mints, and reach the
agent process. The binding requirement must match the Rust sidecar's
candidate agents. For `opencode`, each name must be in the OpenCode
proxy environment set. Each name must also be in the Rust sidecar
credential keys or the Rust OpenCode provider keys. The test reads those
from the Runner source, as the Codex security test does. A provider
without a Runner route fails the test.
`OPEN_CODE_RUNNER_ENVIRONMENT_KEYS` is now exported for this test.
- Docs: Added a "Typed credentials" section to
`doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`. It also documents the ACPX
credential binding that the Runner requires with a Copilot token.
Updated `doc/plugins/PLUGIN_RUNNER_EXECUTION.md`.

## Verification

- `npx vitest run tests/environment-task-release-resume.test.ts
tests/environment-tasks.test.ts` in `packages/plugins/sdk`: 96 passed.
The tests accept each valid shape. They reject mixed sets, two sets,
extra variables, another provider's variable, empty values, NUL bytes,
oversized values, unknown providers and extra fields, on `submit` and on
`resume`. They also check that every published allowlist set parses.
- `npx vitest run
src/services/native-runtime/plugin-runner-credentials.test.ts
src/__tests__/environment-task-runtime.test.ts` in `server`: 79 passed.
The new tests cover each provider's transfer, Grok subscription refusal,
Grok routing refusal, the fail-closed cases without value leaks, harness
matching at submit, and Runner-built Grok requirements
(`runnerProviderLaunchRequirements`) that parse through the task
contract and produce a Grok launch.
- `npx vitest run src/__tests__/plugin-runner-credential-names.test.ts`
in `server`: 4 passed. I broke each list locally to confirm that the
test fails: a renamed SDK key, an SDK provider without a Runner route,
and a dropped name in the Rust sidecar keys, the Rust candidate agents,
the Rust OpenCode keys and the TypeScript OpenCode keys. Each change
failed the test.
- `node scripts/typecheck-layers.mjs` in `server` (includes
`src/__tests__`): no errors.
- `npx vitest run
server/src/services/native-runtime/plugin-runner-probe.test.ts` (from
the root): 15 passed.
`server/src/services/native-runtime/plugin-runner-task.test.ts`: passed.
- `tsc --noEmit` for `packages/plugins/sdk` and `server`: no errors.
- I did not run the full suite locally. CI supplies that coverage.

## Risks

- Low risk for existing plugins. The change only adds a credential type
and two optional settings. A plugin that does not handle `provider-env`
gets a typed value that it does not recognize. Before this change, Core
failed these runs before it called the plugin.
- A plugin must set `PAPERCLIP_ACPX_CREDENTIAL_BINDING` for a Copilot
task. Otherwise the Runner refuses the token. The new doc section shows
the value.
- An OpenCode gateway connection without a key (`auth: none`) still
fails closed, because every value must be non-empty.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, through Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 08:47:18 -08:00
DottaandClaude Opus 5.5 a0fe90f24b fix(retry): never offer Retry while a live run owns the task (#22428)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each task shows an interruption banner. The banner offers Retry when
an attempt failed and nothing will move the task forward
> - The banner reads the task's newest run. A continuation that the
system cancels before it starts can be the newest run, while a different
run owns the task and does the work
> - In a restart-chaos wave, the banner offered Retry for about 140
seconds while the owning run worked. The task then finished with no
click
> - A Retry that the user does not need breaks the rule that the
needed-Retry rate is 0%
> - This pull request makes the banner and the Retry endpoint read the
same observed run. That run is the live run that owns the task, or else
the newest real attempt
> - The benefit is that Retry shows only when nothing will advance the
task, and a shown Retry is accepted

## Linked Issues or Issue Description

**What happened?**

A restart-chaos wave (three release swaps) showed a Retry banner on one
task for 141 to 144 seconds. The reason code was
`issue_execution_lock_changed`. Nobody clicked Retry, and the task
completed.

Timeline (UTC):

- 14:32:09. Run A starts.
- 14:51:12. A user follow-up wakes run B. Run B is queued behind run A.
- 14:52:46. Run A succeeds. The system schedules continuation run C
(`retryOfRunId` = A, reason `native_agent_blocker_continuation`). Run B
starts and takes the task execution lock.
- 14:53:16. The dispatch gate cancels C before it starts with
`issue_execution_lock_changed`, because run B owns the lock. This is
correct de-duplication.
- 14:53:17. The banner shows `failed` with Retry. Run C was created
after run B, so C is the newest run. The projection reads only the
newest run and ignores that run B is running and owns the lock.
- 14:55:41. Run B succeeds and the task is done. The banner goes away.

Retry was never needed. Run B owned the task for the full banner window.

**Expected behavior**

The banner does not offer Retry while a run owns the task and is queued
or running. A run that the system cancelled before it started because
another run took the lock is not a failed attempt. When Retry is shown,
pressing it is accepted for the run that the banner shows.

**Steps to reproduce**

1. Create a task. Run A succeeds and leaves the task in progress.
2. Queue run B for the same task (for example, a user comment). Create
continuation run C after run B.
3. Start run B and set it as the task execution lock owner.
4. Cancel run C before it starts with error code
`issue_execution_lock_changed`.
5. Read the task interruption. Before this change: `state: failed`,
`canRetry: true`, observed run C. After this change: `state: working`,
`canRetry: false`, observed run B.

**Paperclip version or commit**

`fe0c4aa95d` (master at the time of the wave).

## What Changed

- `server/src/services/task-retry-eligibility.ts`: add two shared
conditions.
- `taskAttemptCondition()`: a run cancelled before it started with
`recovery_no_longer_needed`, `issue_closed_before_start` or
`issue_execution_lock_changed` is not an attempt. The first two codes
were already excluded inline in the projection.
`issue_execution_lock_changed` is new.
- `liveExecutionOwnerCondition()`: the run is the task's execution lock
owner and it is queued or running.
- `server/src/services/task-interruption-presentation.ts`: the observed
run is the live execution owner when there is one. Otherwise it is the
newest attempt, with the same native, first-class and legacy precedence
as before. The owner lookup is a primary-key read, so the per-task run
scan does not grow.
- `server/src/services/task-retry.ts`: Retry admission, the refused chat
Retry continuation and the superseded-Retry check use the same attempt
filter. Admission also accepts the live execution owner when the request
names it. The newest attempt stays acceptable, so the automatic Retry
hooks (restart, provider limit, budget resume) keep their behavior.
- `heldTaskRetryRefusals` uses the same attempt filter.
- New test
`server/src/__tests__/task-retry-banner-execution-owner.test.ts`: one
state table. Each row asserts the list and detail projection, and when
Retry is shown, presses it through `requestTaskRetry`.

### State table (tested as one table)

| Row | Runs and lock | Banner | Observed run | Retry press |
| --- | --- | --- | --- | --- |
| 1 | Continuation scheduled; follow-up running and owns the lock |
working, no Retry | follow-up | n/a |
| 2 | Continuation cancelled unstarted (`issue_execution_lock_changed`);
follow-up running and owns the lock (t032) | working, no Retry |
follow-up | n/a |
| 3 | Same as row 2, lock pointer cleared mid-transfer | working, no
Retry | follow-up | n/a |
| 4 | Scheduled continuation overdue past the 2-minute dispatch grace;
follow-up running and owns the lock | working, no Retry | follow-up |
n/a |
| 5 | Newer attempt failed; older queued run owns the lock | working, no
Retry | queued owner | automatic Retry of the failed run is still
accepted |
| 6 | Continuation superseded; follow-up then succeeded | resolved, no
Retry | follow-up | n/a |
| 7 | Continuation superseded; follow-up then failed | failed, Retry |
follow-up | accepted |
| 8 | Continuation superseded; predecessor failed; nothing else runs |
failed, Retry | predecessor | accepted |
| 9 | Continuation started, then lost the lock | failed, Retry |
continuation | accepted |
| 10 | Lock pointer left on a finished failed run | failed, Retry | that
run | accepted |
| 11 | Row 2 with native runs (`nativeIssueId`) | working, no Retry |
native follow-up | Retry naming the owner is accepted |
| 12 | Queued owner past its 15-minute move-forward wait; newer failed
attempt | working, Retry (`move_forward`) | queued owner | accepted |

Invariants that the table checks:

- Retry is never shown while something will advance the task (rows 1 to
6, 11). Row 12 offers only `move_forward` on the owner.
- A Retry press that names the live owner is accepted (rows 1, 2, 3, 5,
11, 12).
- Retry is always shown when nothing will advance the task (rows 7 to
10).
- The shared attempt filter is tested in `heldTaskRetryRefusals`,
`retireSupersededTaskRetry` and the refused chat Retry continuation
(`person_with_superseded_continuation` in
`chat-channels.integration.test.ts`).
- When Retry is shown, pressing it is accepted for the observed run
(rows 7 to 10).

## Verification

- Red: with the service changes removed, rows 1 to 8 fail (8 of 10).
Rows 9 and 10 pass on both sides.
- Green: `cd server && npx vitest run
src/__tests__/task-retry-banner-execution-owner.test.ts` passes 14 of
14.
- Related suites pass: `task-interruption-query.test.ts` (includes the
query-plan bound test), `task-interruption-automatic-path.test.ts`,
`legacy-continuation-authority.test.ts`,
`native-closed-session-retry.integration.test.ts`,
`interrupted-preparation-restore-retry.test.ts`, and the other server
tests that call `taskInterruptionPresentation`, `requestTaskRetry` or
`heldTaskRetryRefusals`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change is read-side only. It does not change scheduling,
dispatch, the lock, or the run-dispatch gate.
- A run cancelled before start with `issue_execution_lock_changed` no
longer shows as a failed attempt. If the lock was released and nothing
else ran, the banner shows the previous attempt. A failed previous
attempt still offers Retry (row 8).
- The projection query has one more union branch. It is a primary-key
lookup through `issues.execution_run_id`. The existing query-plan bound
test passes.
- No runner, migration or API contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Refs #22306 (the earlier fix for a due scheduled retry shown as failed).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 10:32:41 -06:00
0e38c80323 fix(recovery): do not auto-replace a chat Retry under its consumed exact-request authority (#22414)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs that fail before they do provider work can continue
automatically. The native safe-replacement sweep schedules a system
successor run for them.
> - The successor copies the failed run's context. The sweep removes
consumed wake fields, but it keeps the `chatFailedRunRetry` selector and
`taskRetryRequestId`.
> - That selector is the authority of an exact chat Retry. It is bound
to the failed run's own chat action and to the user's Retry intent. The
successor gets a new system wakeup request, so admission can never
authorize it.
> - Thus every automatic successor of a chat Retry fails at setup with
`FailedChatRunRetryAuthorizationError`. The sweep has already retired
the task's recovery hold at that time, so the task can be left with no
owner of the next step.
> - This pull request stops the sweep from replacing a run that carries
`chatFailedRunRetry`.
> - The benefit is that the task keeps its hold. The user's next Retry
or message continues it through the normal authorized path, and no run
fails for a known reason.

## Linked Issues or Issue Description

**What happened?**
On staging, a user pressed Retry on a failed GitHub chat run. The Retry
run failed before its first claim, after a workspace restore failure.
Its own exact-request authority refused it
(`chat_failed_run_retry_not_authorized`). The pre-claim continuation
path then scheduled a `native_safe_replacement` successor. The successor
kept the copied `chatFailedRunRetry` selector under a new wakeup request
ID, and `authorizeFailedChatRunRetryWake` refused it at setup.

**Expected behavior**
An automatic system successor must not be scheduled under chat Retry
authority that it cannot hold. The task must keep its recovery hold for
the user's next Retry or message.

**Steps to reproduce**
1. Make a native run whose context has `chatFailedRunRetry` (an exact
chat Retry).
2. Fail it before its first coordinator claim, or with a provider
terminal failure, so it qualifies for safe replacement.
3. Run `reconcileSafeNativeReplacements`.
4. See that a successor run is scheduled with the copied selector. At
admission it always fails with `FailedChatRunRetryAuthorizationError`.

Related: #22413 changes the same file for a different defect (the
reviewed chat message batch on fresh-session retries). It does not touch
the eligibility predicate or `chatFailedRunRetry`. The two changes are
independent.

## What Changed

- `nativeReplacementPending()` now excludes a failed run whose context
carries `chatFailedRunRetry`. Because the shared eligibility predicate
has the guard, the pre-claim, historical listener, and provider
terminal-failure replacement paths all get it.
- New regression in `native-preclaim-continuation.test.ts`: this is the
staging shape (a pre-claim failure of an exact chat Retry after its
saved work is recovered). It schedules no successor, the hold stays
effective, and the task stays blocked.
- New paired regression in `native-safe-replacement.test.ts` for the
provider terminal-failure path. An ordinary run still gets exactly one
successor. An exact chat Retry gets none.

## Verification

- Red before the fix: the pre-claim chat Retry case failed with
`expected length 0, got 1`. The terminal-failure chat Retry case also
failed. Its ordinary-run control passed.
- Green after the fix, rebased on `fe0c4aa95d`:
- `npx vitest run src/services/native-preclaim-continuation.test.ts
src/services/native-runtime/native-safe-replacement.test.ts`: 89 passed.
- `npx vitest run
src/services/native-runtime/native-safe-replacement.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/execution-control-work.test.ts
src/__tests__/recovery-settled-records-census.test.ts
src/__tests__/durable-chat-wakeup.test.ts`: 206 passed.
  - `pnpm --filter @paperclipai/server typecheck`: passed.
- Second-order check: `nativeInterruptionHold()` uses
`not(nativeReplacementPending())`. Thus an excluded run can now reach
the existing fresh-continuation fallback, the same as other pre-claim
runs that do not qualify. That fallback records a continuation decision
and lets normal admission build a fresh context. It does not copy the
predecessor's selector.
- `pnpm verify` selects 2,950 of 3,817 files for this widely imported
module. I ran the focused suites above and rely on required CI for the
broad coverage.

## Risks

- Low risk. The change only removes candidates from automatic
replacement. Before this change, a removed candidate always produced a
successor that failed at setup.
- Behavior shift: after a chat Retry run fails, the user (or a new chat
message) continues the task. A system successor that cannot be
authorized does not start. Ordinary chat-message runs and non-chat runs
are not changed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1120.0-canary.838000
2026-11-20 07:53:08 -08:00
875738a8d4 fix(tool-access): keep provider-side OAuth registration failures out of Sentry (#22416)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People connect remote MCP apps through OAuth. Paperclip registers
its own OAuth client with the provider through dynamic client
registration (RFC 7591).
> - When the provider's registration endpoint answers with an error,
Paperclip returns a 502 to the person. The error handler also sends
every 5xx `HttpError` to Sentry.
> - A provider that refuses or fails registration (401, 403, 404, 429 or
5xx) is a provider-side failure. It is not a Paperclip server fault, so
it adds noise to Sentry.
> - MCP discovery already marks the same provider statuses as expected
remote failures, and the error handler does not report them.
> - This pull request applies the same marking to the registration
response. It also adds the provider status and the allowlisted error
label to registration failures that Paperclip still reports.
> - The benefit is that Sentry shows only registration failures that can
be Paperclip faults, and each one tells which status the provider sent.

## Linked Issues or Issue Description

**What happened?**
A hosted server reported `Error: OAuth dynamic client registration
failed` three times in two minutes. The stack is `startOAuth` ->
`registerOAuthClient` -> `requestOAuthClientRegistration`. The message
is the generic fallback, so the provider sent no recognized RFC 7591
`error` code. The event did not include the provider's HTTP status.
Thus, the event did not show if the provider refused the request or if
Paperclip sent bad metadata.

**Expected behavior**
The person gets the same 502 and code
`oauth_dynamic_client_registration_failed`. Sentry does not get a crash
report when the provider refuses or fails the request on its side (401,
403, 404, 429, 5xx). Sentry continues to get a report for other
statuses, such as a 400 that rejects Paperclip's metadata. That report
includes the status, so the cause is visible.

**Steps to reproduce**
1. Point an OAuth connection at an authorization server whose
registration endpoint returns 403 (closed or token-gated registration),
404, 429 or 503.
2. Start the OAuth connection.
3. See the 502 response and one Sentry crash report.

**Paperclip version or commit**
`master` at fe0c4aa95d.

No related open PR covers this reporting path. #20433 changes which
client method Paperclip accepts after a successful registration. It does
not change failure reporting.

## What Changed

- `server/src/services/tool-access/oauth/client.ts`: A non-OK
registration response now goes through `markMcpHttpResponseFailure`.
This is the same classifier that MCP discovery uses. 401, 403, 404, 429
and 5xx become expected remote failures, so `errorHandler` does not
report them. The HTTP response does not change.
-
`server/src/services/tool-access/oauth/provider-failure-diagnostics.ts`
(new): A server-only `WeakMap` record keeps the operation, the HTTP
status and the already-normalized provider error label. It does not keep
the provider URL, body, description or headers.
- `server/src/sentry.ts`: `captureException` adds `oauth_operation`,
`oauth_provider_status` and the optional `oauth_provider_error` tags and
an `oauth_provider_failure` context for an error that has this record.
Grouping does not change (`{{ default }}`).
- Tests: `oauth-dcr-failure-reporting.test.ts` sends each status through
the real `errorHandler`. `sentry-oauth-provider-failure.test.ts` checks
the exact Sentry tags and context.

## Verification

- `cd server && npx vitest run
src/__tests__/oauth-dcr-failure-reporting.test.ts
src/__tests__/sentry-oauth-provider-failure.test.ts
src/__tests__/sentry.test.ts src/__tests__/remote-http-rebinding.test.ts
src/__tests__/openrouter-catalog-error-handler.test.ts`: all pass.
- `cd server && npx vitest run
src/__tests__/generic-mcp-connection.test.ts
src/__tests__/tool-access-client-credentials.test.ts
src/__tests__/error-handler.test.ts`: 154 tests pass. This includes the
existing hostile-provider DCR redaction test.
- Red check: without the `client.ts` change, the six provider-side
status cases and the 400 diagnostics case fail. Without the `sentry.ts`
change, the Sentry tag test fails.
- Local `tsc` was stopped by the sandbox (SIGKILL), also for a scoped
config. CI typecheck must cover this.

## Risks

- Low risk. The HTTP status, body and code for the person do not change.
- Sentry stops reporting registration failures with 401, 403, 404, 429
and 5xx status. A provider outage during registration is then visible
only in the response and logs. This is the same policy that MCP
discovery already uses.
- The new tags contain only an integer status and an allowlisted label.
No provider text goes to Sentry.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edit and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
2026-11-20 07:52:23 -08:00
paperclipbugbotandClaude Opus 5.5 425a30139f fix(ui): attribute a stackless unhandled fetch rejection to the code that started it (#22396)
Chromium creates `TypeError: Failed to fetch` with no stack frames, and the
browser error monitor removes the SDK Breadcrumbs integration, so nothing
wraps fetch to add a call site. An unhandled network rejection reached
Sentry with no caller.

Once monitoring starts, wrap window.fetch once. Only on rejection, classify
the call-site stack as app, extension, other or unknown, keyed by the
rejected value in a WeakMap. Tag unhandled object rejections with
browser_fetch_initiator and browser_fetch_wrapper. No URL, path or stack
text is sent. The returned promise rethrows the identical value, so a
dropped promise is still reported.

Diagnostics only; this does not fix the underlying rejection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1120.0-canary.837800
2026-11-20 07:47:44 -08:00
54972096ba fix(native): wait out a provider rate limit that refuses the session start (#22386)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts each agent run in a provider session on a
remote sandbox
> - A provider can refuse `session.open` because the account reached its
rate limit or quota
> - After one same-run retry, a repeat of that refusal ended the run as
`native_provider_startup_rejected`
> - That code blocks the task for a board Retry and sends the failure to
Sentry, but the limit passes by itself
> - This pull request records that refusal as a provider usage-limit
hold, which already exists for limits on a turn
> - The benefit is that Paperclip retries the task at the reset by
itself, and Sentry gets no error for an expected wait

## Linked Issues or Issue Description

Refs #20808, #19246, #18854. No public issue exists for this case.

**What happened?**
On staging, an account-wide Anthropic 429 refused `session.open` for all
runs of one agent. Each run waited the one-hour default cooldown and
retried on the same run. The retry met the same 429. The repeat check
then ended each run as `native_provider_startup_rejected`. Sentry
recorded 25 events in two bursts. Each task became blocked and waited
for a person to click Retry.

**What did you expect to happen?**
A rate limit or quota at session start must wait for the reset, the same
as a limit that the provider reports on a turn. The usage-limit hold
retries the task at the reset by itself and does not report to Sentry.

**Steps to reproduce**
1. Use a remote native runner with an ACPX provider whose account is
rate limited.
2. Start a task run. The provider refuses `session.open` with
`classification=provider_rate_limit` and a typed `providerFailure`
(`phase: "startup"`, `category: "rate_limit"`).
3. Let the same-run retry run after the cooldown while the limit
continues.
4. See that the run fails as `native_provider_startup_rejected`, the
task becomes blocked for a board Retry, and Sentry gets a `run_failure`
event.

## What Changed

- `nativeRemoteStartupRejectionCode` returns
`native_provider_usage_limit` instead of
`native_provider_startup_rejected` when the typed startup failure is
`rate_limit` or `quota_exhausted`. The first refusal with a proven
runner exit still retries on the same run after the cooldown, as before.
Other startup failures do not change.
- `readNativeUsageLimitReset` also accepts a startup refusal when the
run has no turn event. It reads the current failure (from the executor)
or the run's saved `providerFailure`, and it requires `phase:
"startup"`. A run with a turn event keeps the old rules.
- `planProviderLimitResume` accepts the current startup failure, so the
hold gets its reset before the run row records the failure.
- The hold saves the startup refusal that created it
(`providerLimitStartup`, with its observation time). When the hold
resumes, it reads that refusal first. A same-run retry keeps the run
row's first saved failure, so the run row can name an older reset.
- Tests: a classifier unit test, an integration test for the hold and
its automatic Retry, an executor-level test for two startup attempts
with different resets, and a Sentry-report test for the startup shape.

## Verification

- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"session.open the provider itself rejected"`: 9 tests pass. The new test
fails on `master` with `Received: "native_provider_startup_rejected"`.
- `npx vitest run
src/services/native-runtime/native-provider-capacity.integration.test.ts
src/services/__tests__/run-failure-report-native-usage-limit.test.ts`:
26 tests pass. The new integration test fails on `master` (the plan is
not automatic without a turn event).
- `npx vitest run` on `native-startup-rejected-retry`,
`native-provider-capacity`, `native-startup-retry` and
`native-provider-limit-cooldown` (integration), and
`run-failure-report-native-usage-limit`: 76 tests pass. The new
two-attempt test fails on the first head (`c3ee40f166`) with `reason:
"reset_unknown"`.
- The full `native-session-executor.test.ts`,
`native-provider-limit-cooldown.integration.test.ts` and
`run-failure-report*.test.ts` files: 1865 tests pass and 1 fails. The
failure (`compares export process compatibility without changing
conversation identity (changed: false)`) also fails on an unchanged
`master` in the same sandbox.
- I could not run the full server typecheck locally, because the sandbox
has an 8 GiB memory limit and `tsc` stopped with an out-of-memory error.
CI runs the typecheck.

## Risks

- Low risk. The change applies only to a remote startup refusal with a
typed `rate_limit` or `quota_exhausted` category that would otherwise
end the run.
- The task still shows as blocked during the hold, as for a limit on a
turn. The hold now retries by itself at the reset, at most three times
in a row without a successful run.
- A quota without a reported reset gives the same "restore usage
capacity" action as before, so a person still decides.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1120.0-canary.837700
2026-11-20 07:22:34 -08:00
c669cfca17 fix(native): name a full sandbox disk that fails a run during setup (#22383)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs execute in a managed sandbox with a fixed disk
size
> - After a server restart, Paperclip reconnects to the running agent.
The reconnect opens a new process session in the sandbox through the
sandbox provider plugin
> - When the agent filled the sandbox disk, the provider refuses that
call with "no space left on device"
> - The setup failure path did not recognize this refusal. The run
failed as the generic `setup_failed`, and the task showed no useful
explanation
> - #22239 names `sandbox_disk_full` when the runner reports a full disk
during a turn. It does not cover a refusal that reaches the setup
failure path
> - This pull request names that refusal `sandbox_disk_full` in the
setup failure path, and keeps continuation recovery from repeating such
a run
> - The benefit is that the task tells the user what to do in plain
language, and Paperclip does not repeat work that fills the same disk
again

## Linked Issues or Issue Description

Refs #22239 (names a full sandbox disk during a turn). This PR covers
the setup failure path.

**What happened?**
A native agent ran a large dependency installation in its Daytona
sandbox. A server restart then happened. The reconnect could not open a
process session. The Daytona plugin returned `JsonRpcCallError`
(-32002): `internal server error: failed to create session config
directory: mkdir /home/daytona/.daytona/sessions/...: no space left on
device`. The run failed with `setup_failed` in the `setup` phase. The
Sentry report grouped it with unrelated `setup_failed` plugin errors.

**Expected behavior**
The run failure names `sandbox_disk_full`. The task shows the existing
plain-language action (free disk space through the agent, then press
Retry). Automatic retry and generic continuation recovery do not repeat
the work on a sandbox of the same size. A full disk on the Paperclip
host keeps its own code.

**Steps to reproduce**
1. Start a native run in a managed sandbox.
2. Let the agent fill the sandbox disk.
3. Restart the server while the run is active.
4. See the reconnect fail with "no space left on device" and the run
fail as `setup_failed`.

## What Changed

- `native-sandbox-disk-full.ts`: add `isSandboxProviderDiskFullError`.
It is true for the runner's named `sandbox_disk_full` failure, or for a
plugin worker error (`JsonRpcCallError`, code `WORKER_ERROR`) whose
message reports a full disk. It reads the error, its causes and the
first entries of an `AggregateError`.
- `run-completion.ts`: the setup failure ladder uses `sandbox_disk_full`
for that error before the generic `setup_failed` fallback. Named setup
codes and `storage_unavailable` keep their precedence.
- `recovery/service.ts`: add `sandbox_disk_full` to the codes that
generic continuation recovery does not repeat. `setup_failed` was
already in this set, so the new code does not open a new automatic path.
This also applies to the turn failures that #22239 names.
- Tests for the helper, the setup ladder (with a real database) and the
continuation classification.

## Verification

- In `server/`: `pnpm vitest run
src/services/heartbeat/run-completion.test.ts
src/services/native-runtime/native-sandbox-disk-full.test.ts
src/services/recovery/provider-failure-classification.test.ts
src/__tests__/failed-continuation-continues-again.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts`: 5 files, 312 tests
pass.
- Red check: without the `run-completion.ts` change, the new setup test
fails with `errorCode: "setup_failed"`, the same code as the observed
failure. Without the helper and recovery change, the helper and
continuation tests fail.
- The local layered typecheck could not finish: the sandbox memory limit
stopped its largest layer. CI runs the typecheck.

## Risks

- Low to medium. The change only renames a setup failure that was
already terminal and not retried. It uses an existing code that already
has retry and UI handling.
- A plugin worker error that reports a full disk on the Paperclip host
(not in the sandbox) gets `sandbox_disk_full`. This is unlikely: host
storage failures use `storage_unavailable`, which keeps precedence.
- Generic continuation recovery no longer repeats a turn failure named
`sandbox_disk_full`. This matches the intent of #22239.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 07:22:28 -08:00
92271da899 fix(native): keep a chat turn's message batch when a fresh-session retry continues it (#22413)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - External chat turns (GitHub, Slack, Discord) wake an agent on a task
with an admitted message batch.
> - When a task is in review, dispatch checks that the run is bound to
that reviewed chat batch. A run without a batch fails with
`reviewed_chat_execution_binding_not_authorized` (`context_mismatch`).
> - Automatic fresh-session retries (tool-receipt limit, provider
capacity, environment blocker, agent blocker, and safe replacement)
remove all consumed wake input from the failed run's context. This also
removes the chat message IDs, but the context keeps `source:
chat:<provider>`.
> - Thus every automatic retry of a reviewed chat turn fails at setup.
The one-shot review-path recovery that follows copies the same empty
context and fails the same way.
> - This pull request keeps the admitted chat message batch on those
retries and removes the failed run's dispatch proofs, so dispatch checks
ownership and current access again.
> - The benefit is that a reviewed chat turn can continue after a
transient failure, and access checks still apply.

## Linked Issues or Issue Description

Refs #13818 (the same approach for review-path recovery). Related:
#20689, #20697, #20842 (other causes of the same denial).

**What happened?**
A GitHub PR review turn on a task in review stopped on a transient
failure (tool-receipt limit or sandbox environment blocker). Paperclip
scheduled an automatic fresh-session retry. The retry failed at setup in
less than one second with
`reviewed_chat_execution_binding_not_authorized` and denial reason
`context_mismatch`. The review-path recovery wake then failed with the
same error. On staging this occurred 5 times on 3 tasks after #20689,
#20697 and #20842 were deployed.

**Expected behavior**
An automatic retry of a chat turn keeps the admitted message batch.
Dispatch then checks the retry against current task ownership and
current endpoint, conversation and principal access. If access is still
valid, the retry runs. If access was revoked, dispatch refuses the
retry.

**Steps to reproduce**
1. Put a task in review with a run-bound external chat turn (for example
`source: chat:github` with `wakeCommentIds`).
2. Fail that native run with a fresh-session retry lane (for example
`native_environment_blocker`).
3. Let dispatch claim the scheduled retry. The retry context has
`source: chat:github` but no `wakeCommentIds`.
4. Dispatch refuses the retry with `context_mismatch`.

**Paperclip version or commit**
`master` at 414abc2dbf.

## What Changed

- Add
`server/src/services/native-runtime/fresh-session-retry-context.ts`. It
holds the one list of consumed wake input keys and builds the retry
context. For a run-bound external chat turn, it keeps the admitted
message IDs (`wakeCommentIds`, `wakeCommentId`, `commentId`). It removes
the dispatch proofs that the failed run got
(`paperclipHarnessCheckedOut`, `paperclipExternalChatExecutionBound`).
Other runs keep the existing behavior.
- Use the helper in `materializeNativeFreshSessionRetry` (all its lanes)
and in `reconcileSafeNativeReplacements`. Before, each producer had its
own copy of the key list.
- Add a unit test for the helper.
- Add a regression test in
`heartbeat-reviewed-chat-binding.integration.test.ts` for Slack and
Discord. It checks that the retry passes the reviewed chat check, that
revoked access still refuses the retry, and that review-path recovery
keeps the batch.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts
src/services/native-runtime/fresh-session-retry-context.test.ts
src/services/native-runtime/native-safe-replacement.test.ts
src/services/native-runtime/native-agent-blocker.integration.test.ts
src/services/native-runtime/native-tool-receipt-limit.integration.test.ts`
passes: 5 files, 127 tests.
- Red check: with the `master` versions of
`native-provider-capacity-retry.ts` and `native-safe-replacement.ts`,
the new regression test fails (2 failed, 34 passed). The retry is
refused with `context_mismatch`.
- CI runs the full typecheck. The local full server typecheck stopped
because the sandbox ran out of memory.

## Risks

- Low risk. The change applies only to automatic fresh-session retries
whose context has a run-bound external chat source and admitted message
IDs.
- The retry does not get the failed run's authority. Dispatch must check
ownership and access again, and the regression test shows that a revoked
principal is still refused.
- Other retries keep the same context as before, because both producers
now use the same key list.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1120.0-canary.837500
2026-11-20 06:47:34 -08:00
5db2ccdae6 fix(sentry): keep a continued native tool-receipt-limit stop out of Sentry (#22399)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends unexpected run failures to Sentry as run-failure
events. It keeps known, recovered failures local
> - The runner stops a native turn when the turn holds too many tool
results (`semantic_tool_turn_receipt_limit`). The finalizer then
continues the task in a fresh session
> - The runner's stop has no recorded cancellation request. So the
cancelled run keeps `unknown`, unexpected cancellation provenance, and
`reportRunFailure` sends it to Sentry as "Native session cancelled"
> - The continuation owns recovery, so this event is noise. The native
capacity retry had the same problem, and #19051 and #19320 fixed it with
the successor run as the proof
> - This pull request uses the same successor-run proof for the
tool-receipt-limit continuation
> - The benefit is that a continued receipt-limit stop does not make
Sentry noise. Exhausted continuations still report

## Linked Issues or Issue Description

Refs #19051, #19320

**What happened?**
On staging, a native Codex run stopped at its tool-result receipt limit.
The finalizer committed the `semantic_tool_turn_receipt_limit` decision
and scheduled a fresh-session continuation run (`retryOfRunId` = the
stopped run, `scheduledRetryReason` =
`semantic_tool_turn_receipt_limit`). The continuation succeeded and the
task finished. Sentry still got a run-failure event: `Error: Native
session cancelled`, `run_status=cancelled`, cancellation `source:
unknown`, `expected: false`.

**Expected behavior**
A receipt-limit stop that has a scheduled continuation stays local. The
continuation reports itself if it fails. A stop with no continuation
(exhausted, or the user stopped it) stays reportable as before.

**Steps to reproduce**
1. Run a native turn that reaches the semantic-tool receipt limit.
2. Let the finalizer schedule the fresh-session continuation.
3. With `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected`, see that the
cancelled run is sent to Sentry.

## What Changed

- `server/src/services/run-failure-report.ts`:
`isScheduledNativeCapacityRetry` becomes `isScheduledNativeRetry`. It
reads the successor run for the lane that can own the run:
`native_provider_overloaded` for a failed capacity run (unchanged), and
`semantic_tool_turn_receipt_limit` for any other failed or cancelled
native run. The query, task binding, setup-phase, signal, and
read-failure behavior do not change.
- `server/src/services/__tests__/run-failure-report.test.ts`: regression
tests with the persisted shape of the staging run, plus negative cases
(no successor, other retry lane, other task, no native task, legacy,
signal, timed out, setup phase, `all` mode, failed read).
- The shared `buildRun` fixture in `run-failure-report.test.ts` now has
`createdAt`, like a persisted run row. Without it, the wider
scheduled-retry read failed on in-memory test runs and logged a spurious
warning. Real report inputs are full database rows, so production code
does not change for this.
-
`server/src/services/native-runtime/native-tool-receipt-limit.integration.test.ts`:
the real native executor and finalizer path now also asserts the Sentry
outcome. A continued stop sends no report. A cancelled turn without a
continuation still sends one. The fixture run now has `startedAt`, as a
real started run does. Without it, no cancelled-run report can occur,
and the new assertion would prove nothing.

## Verification

- Red before the fix: the two new unit cases failed (`captureRunFailure`
was called), and the integration assertion failed (`expected [...] to
have a length of +0 but got 1`).
- Green after the fix:
- `npx vitest run src/services/__tests__/run-failure-report.test.ts
src/__tests__/run-failure-sentry.test.ts
src/__tests__/run-failure-sentry-real-sdk.test.ts
src/services/native-runtime/native-provider-overload.integration.test.ts
src/services/native-runtime/native-tool-receipt-limit.integration.test.ts
src/services/native-runtime/status-arbiter.test.ts` → 383 passed, 13
skipped.
  - `node scripts/typecheck-layers.mjs` in `server` → no errors.
- `pnpm verify` did not finish in the local sandbox in the time limit.
The only failures were 3 cases in
`server/src/__tests__/archived-nested-repository.test.ts`. They also
fail on unchanged `master` in the same sandbox, and they do not use this
code. CI runs the full suite.

## Risks

- Low risk. The change only removes Sentry events. It does not change
run status, task status, retries, or the user's view.
- A failed or cancelled native run that is not a capacity failure now
does one more indexed read before its report. That read is bounded by
company, task, and agent, and it happens only when a report is about to
be sent.
- A successor with this retry reason exists only when the status arbiter
proved the receipt-limit stop from a persisted runner diagnostic. A user
Stop refuses the continuation, and an exhausted lane creates none, so
these runs still report.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, code search, and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1120.0-canary.837400
2026-11-20 06:37:30 -08:00
a097949864 fix(server): stop issue reference syncs from deadlocking on mutual mentions (#22389)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server keeps a table of issue reference mentions. It rebuilds
the rows for an issue when its title or description changes.
> - `syncIssue` locks the source issue row `FOR UPDATE` before it
rebuilds the rows (added in #13898).
> - Each mention insert does a foreign-key check. That check takes a
`FOR KEY SHARE` lock on the target issue. `FOR KEY SHARE` conflicts with
`FOR UPDATE`.
> - When issue A mentions B and issue B mentions A, two concurrent syncs
each hold one `FOR UPDATE` lock and wait for the other. Postgres reports
`deadlock detected`, and one update fails. We saw this on staging.
> - This pull request changes the lock to `FOR NO KEY UPDATE`.
> - The benefit is that syncs of one issue stay serialized, and
foreign-key checks from other issues do not wait on that lock.

## Linked Issues or Issue Description

Refs #13898

**What happened?**
A staging server reported `Failed query: insert into
"issue_reference_mentions" ...` caused by `PostgresError: deadlock
detected`. The insert came from `replaceSourceMentions` inside the
`syncIssue` transaction. The source issue description mentioned the
target issue.

**Steps to reproduce**
1. Create issue A with a description that mentions issue B.
2. Create issue B with a description that mentions issue A.
3. Run `syncIssue(A)` and `syncIssue(B)` at the same time, several
times.
4. One sync fails with `deadlock detected`.

**Expected behavior**
Both syncs complete, and each issue has its mention rows.

## What Changed

- `server/src/services/issue-references.ts`: `syncIssue` now locks the
source issue with `FOR NO KEY UPDATE`, not `FOR UPDATE`. `FOR NO KEY
UPDATE` still conflicts with itself, so concurrent syncs of one issue
stay serialized (the #13898 behavior). It does not conflict with the
`FOR KEY SHARE` lock of a foreign-key check, so the lock cycle cannot
form.
- `server/src/__tests__/issue-references-service.test.ts`: two
regression tests.
- One test holds a sync lock in an open transaction. It then syncs a
second issue that mentions the locked issue, and expects that sync to
complete without a wait.
- One test runs 20 concurrent syncs of two issues that mention each
other, and expects all four mention rows.

## Verification

- `cd server && npx vitest run
src/__tests__/issue-references-service.test.ts`
- Before the fix (old lock): the two new tests fail. One test shows
`expected 'blocked' to be 'completed'`. The other test fails with
`PostgresError: deadlock detected` on `select ... from "issues" ... for
update`. This is the same error as the staging event.
- After the fix: 5/5 tests pass. I ran the file three times and all runs
passed. The existing test "serializes concurrent syncs for a description
with repeated references" still passes.

## Risks

- Low risk. There is no schema change or migration. The lock mode
changes only for this read. `UPDATE` statements that do not change key
columns already take `FOR NO KEY UPDATE`, so this lock blocks the same
issue writers as before. Only foreign-key `KEY SHARE` checks can now
continue at the same time.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in a Claude Code agent session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 06:17:00 -08:00
97737c4f7f fix(workspaces): give a failed saved-workspace restore a stable error code (#22419)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a run starts, Paperclip can reopen a task's archived
execution workspace and restore its saved Git commit
> - When the saved restoration record no longer matches its workspace,
or a restore step fails, the code throws `GitCacheRestorationFailure`
> - That error records which step failed and what kind of failure it
was, but it has no `code`
> - Run-failure reporting groups failures by error code, never by
message. A setup failure without a code falls back to its class name,
and this class reports its name as plain `Error`. So these failures
share one Sentry group with every other uncoded setup failure
> - On staging this one failure makes up almost all of that catch-all
group: hundreds of events from automatic restarts of tasks whose saved
workspace record can no longer be restored
> - This pull request gives the error a stable code
> - The benefit is a separate, correctly named Sentry group for restore
failures, and run records whose `failureCause` names the restore failure
instead of `Error`

## Linked Issues or Issue Description

**What happened?**
On staging, runs keep failing during setup with "The saved workspace
commit could not be restored. Check repository access and retry." The
stack is `selectHeartbeatEnvironment` → `readGitCacheExpiration` →
`readGitCacheRestoration`. The stored restoration record of an archived
workspace fails validation, and the run fails as `setup_failed`. The
error has no code, so the Sentry fingerprint uses `exception:Error` and
puts it in the same group as unrelated setup failures.
`resultJson.failureCause` for these runs is the generic `Error`.

**Expected behavior**
A failed saved-workspace restore has a stable error code. Sentry groups
it on its own, and the run's failure cause names it.

**Related**
Refs #19736, which added the safe `step` and `kind` restore diagnostics
this change keeps. #22093 took the same approach for the
execution-target Git context probe. I found no open PR for this error.

**Steps to reproduce**
Construct `new GitCacheRestorationFailure()` (or call
`readGitCacheRestoration` with a record that does not match its
workspace). On current master the error has no `code`,
`failureCauseFromError` returns `Error`, and the run-failure diagnostics
record no root code.

## What Changed

- `server/src/services/git-workspace-restoration.ts`: add
`GIT_CACHE_RESTORATION_FAILED_CODE` (`git_cache_restoration_failed`) and
set it as `code` on every `GitCacheRestorationFailure`.
- The message, `step` and `kind` do not change.
`classifyWorkspaceRestorationError` still checks the class first, so
restore classification does not change.
- `server/src/__tests__/git-workspace-restoration-diagnostic.test.ts`:
add a test that the error carries the code, with and without a step
diagnostic, that `failureCauseFromError` returns it (also when the error
is a `cause`), and that `collectRunFailureDiagnostics` records it as the
root exception code. That root code is what the Sentry fingerprint uses.

## Verification

- Red/green: the new test fails on unmodified master and passes with the
fix.
- `vitest run` on `git-workspace-restoration-diagnostic`,
`git-expiration-auth-fallback` and `run-failure-sentry`: 63 tests pass.
- I also ran the other suites that use this error or its readers locally
(archived nested repository, fetch head, reproducible managed checkouts,
base storage, repository object cache, remote workspace migration, Git
workspace expiration, heartbeat environment selection, closed-workspace
routes). In this sandbox many `remote-workspace-migration` tests stopped
at about 10 seconds each. The migration code doesn't read this error's
`code` (it only compares a database lock code and a fixed supersession
code), so I expect the slow sandbox to be the cause. CI's full run is
the authoritative result.
- I couldn't run the full server typecheck locally because the sandbox
ran out of memory. The change only adds a new `code` field, which
`Error` does not define. CI's Typecheck job covers it.

## Risks

- Low risk. Only an added `code` property on one error class. No control
flow changes.
- `resultJson.failureCause` for these runs becomes
`git_cache_restoration_failed` instead of `Error`. The only code that
compares `failureCause` checks a different prefix
(`remote_workspace_sandbox_disk_full:`).
- Every setup-failure classifier in `run-completion` matches a specific
error class or HTTP error, not any coded error, so the run's `errorCode`
stays `setup_failed` and retry behavior does not change.
- Not in scope: automatic fresh continuations still retry this
deterministic failure. Stopping that needs a recovery-policy change in
`fresh-continuation-fallback`, which decides on `run.errorCode`. That's
a separate change for the recovery owners.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, as the Paperclip
Sentry triage agent.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted suites; see
Verification for the slow migration suite)
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documentation describes this error's fields)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 06:13:08 -08:00
a792409d07 fix(tools): retry a named MCP gateway create that loses a deadlock (#22408)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs prepare a run-scoped MCP gateway for their agent before
the provider starts. `createNamedGateway` writes the `tool_mcp_gateways`
row and its profile binding in one transaction.
> - That transaction locks the agent row first (`FOR NO KEY UPDATE`).
Then its gateway insert takes a key-share lock on the company and
profile rows for the foreign keys.
> - Another writer can lock the company row first and then the agent
row. The two transactions then wait for each other. Postgres detects a
deadlock and aborts the gateway create (`40P01`).
> - The run then fails before the provider starts. The task stops on
`native_continuation_requires_reconciliation`, so a person must
reconcile a run that did no work.
> - This pull request runs the gateway create transaction through the
shared bounded retry that #22329 and #20971 use for the same type of
lock cycle.
> - The benefit is that a short lock conflict no longer fails the run
preparation.

## Linked Issues or Issue Description

**What happened?**
A native run failed while it prepared its MCP gateway. The error was
`Failed query: insert into "tool_mcp_gateways" (...)`. The Postgres
cause was `deadlock detected` (`40P01`). The task then stopped and
needed reconciliation.

**Expected behavior**
A transient lock conflict does not fail the run preparation. The gateway
create writes the gateway and its profile binding, and the run
continues.

**Steps to reproduce**
1. Transaction A: lock the company row `FOR UPDATE`.
2. Call `createNamedGateway` for an agent in that company. It locks the
agent row and then waits for the company key-share lock.
3. Transaction A: lock the agent row `FOR UPDATE`.
4. Postgres reports a deadlock. Before this change the gateway create is
the victim and `createNamedGateway` rejects with the database error.

**Additional context**
The new test reproduces this with two real transactions on embedded
Postgres. Transaction A gets a longer `deadlock_timeout`, so the gateway
create is always the victim, as in the observed failure. The failing
statement in the test is the same insert as in the observed error.

Refs #22329
Refs #20971

## What Changed

- `server/src/services/tool-gateway.ts`: `createNamedGateway` runs its
transaction through `retryTransientLockContention`. The helper retries
only lock timeout, deadlock and serialization failure (`55P03`, `40P01`,
`40001`): 5 attempts, with 100 ms × attempt backoff. Other errors fail
as before.
- `server/src/__tests__/tool-gateway-named-gateway-deadlock.test.ts`:
new test with a real two-transaction deadlock between a
company-then-agent locker and a real `createNamedGateway` call. The test
checks that exactly one gateway row and one profile binding exist after
the retry.

## Verification

- Failing before the fix: `cd server && npx vitest run
src/__tests__/tool-gateway-named-gateway-deadlock.test.ts` failed.
`createNamedGateway` rejected with `Failed query: insert into
"tool_mcp_gateways" ...`, cause `deadlock detected`, code `40P01`.
- Passing after the fix: the same test passes. I ran it 3 times.
- `heartbeat-runtime-mcp-servers.test.ts`,
`agent-runtime-assignments.test.ts`,
`agent-interaction-permissions.test.ts` and
`tool-gateway-service.test.ts` pass (4 files, 238 tests).
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. Only the gateway create transaction retries, and only on the
three contention codes.
- Postgres rolls back the complete aborted transaction. A retry
therefore cannot write a duplicate gateway or profile binding.
- The interaction-permission check runs again in each attempt. A retry
cannot skip it.
- If a concurrent create with the same slug commits first, the retry
gets a unique violation. That error is not retried. The existing native
caller then reads the committed gateway by slug, as before.
- The worst added delay is about 1 s of backoff across 5 attempts, and
only when the lock conflict continues.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution on embedded
Postgres).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 05:47:47 -08:00