Commit Graph
3203 Commits
Author SHA1 Message Date
7671e0cf0d fix(identity): read a provisioned agent key without an exclusive agent row lock (#21506)
Run setup reads an existing agent identity key under FOR KEY SHARE on the agent row instead of FOR UPDATE, so it no longer joins queued-run claim deadlocks. The lock still waits for an in-progress agent deletion. First provisioning uses the same lock with ON CONFLICT DO NOTHING.

Co-Authored-By: paperclipbugbot <paperclipbugbot@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 01:17:29 -08:00
Devin FoleyandPaperclip 28ed2f1db0 fix(runner): support native OpenCode on macOS SSH hosts (#22534)
## Thinking Path
> - Paperclip coordinates agents across local, SSH, and sandbox
environments.
> - Native Runner must launch artifacts for the execution host.
> - A Linux controller currently selects Linux OpenCode assets for a
macOS SSH host.
> - Remote launch-file validation also accepts only ELF executables.
> - The standard Core image must carry a verified Mac runtime for this
host pair.
> - This pull request adds that runtime and selects it for Apple Silicon
SSH hosts.
> - Self-hosted and hosted controllers can use the same generic
implementation.

## Linked Issues or Issue Description
**What happened?**
A native OpenCode run on an Apple Silicon SSH host receives Linux
controller artifacts. Its private launch-file verifier also rejects
Mach-O files. A working legacy CLI agent does not exercise this path.

**Expected behavior**
The controller must select and verify the target's native daemon, Node,
OpenCode executable, and proxy. Existing Linux targets and legacy agents
must keep their current behavior.

**Steps to reproduce**
1. Run a standard Linux Core image.
2. Select an SSH environment on an Apple Silicon Mac.
3. Start a native OpenCode agent with a compatible model connection.
4. Observe the incompatible runtime selection before useful agent work.

**Paperclip version or commit**
Base: `324ec1f537a9ba1e84239faa81bcef326a5dd1b7`.

**Deployment mode**
Docker or another Linux controller, with macOS ARM64 execution over SSH.

Related work: #21662 covers local OpenCode confinement. #22236 covers
SSH process cleanup. This change does not depend on either PR. No
duplicate macOS OpenCode runtime PR was found.

## What Changed
- Select a verified `darwin-arm64` runtime for OpenCode SSH targets.
Preserve the Linux image default and explicit operator overrides.
- Verify Mach-O launch files against the admitted architecture, file
digests, and remote account. Keep existing Linux receipt and recovery
behavior.
- Build the Mac runtime on an Apple Silicon CI host with standalone Node
24.21.0. Keep OpenCode pinned to 1.18.34.
- Package the sealed OpenCode runtime artifacts and required package
metadata. Omit other harness distributions and their optional
qualification inventories.
- Verify archive provenance and safe extraction on Linux. Include the
runtime in standard and merge-queue Core images. Avoid storing the
transport archive in an image layer.
- Reuse artifacts only from successful protected master image builds
with identical runtime inputs. Test cache trust boundaries and register
the script tests in CI.
- Document runtime delivery, manual image inputs, and the initial Apple
Silicon scope. No Cloud source, schema, or agent configuration changes.

## Verification
- `pnpm -r typecheck` — passed.
- `pnpm build` — passed.
- Focused OpenCode artifact selection and private staging tests — 21
passed.
- Focused native executor tests for OpenCode, SSH, and provider packs —
60 passed.
- Real Rust daemon transport tests with the OpenCode protocol fixture —
4 passed.
- Installed OpenCode 1.18.34 server qualification with a local provider
fixture — 3 passed.
- Runtime packaging, cache trust, workflow, image-contract, and
merge-queue tests — 26 passed.
- The CI release-workflow test group, including canary reuse and
provenance — 218 passed.
- Docker workflow and build-stamp guards — 29 passed after adding the
Mac digest expectation, updating the excluded-source COPY expectation,
and removing the redundant production commit ARG.
- The source SDK registry, public-type, and evidence checks — 59 passed
after updating only the root package metadata hash. Lock, compiler,
parser, and model pins did not change.
- Fresh-host provider-pack upload harness — 7 passed. The new regression
first failed with `package_version:acpx`, then passed after the required
metadata was preserved.
- Node version policy — passed against a clean tracked-file snapshot.
Generated local provider assets make the direct workspace scan fail;
they are absent from the CI checkout.
- Built, packaged, installed, and verified the actual Mac runtime. Both
the controller verifier and the remote startup verifier accept its pins,
target, sealed files, manifest, and package metadata. The pre-change
private launch verifier rejects the real Mach-O files; the candidate
accepts those same files. The dedicated archive is about 101 MB.
- Live native smoke: real Mac daemon, private staged Mach-O files,
OpenCode 1.18.34, and a local GLM-5.3 backend. The model read two files,
edited the implementation, ran three tests, completed the semantic
finish call, and summarized the three passes. An independent test run
passed. The test file stayed unchanged. This used the existing `allow`
mode and a scratch account-home fixture on a Mac. It does not prove
hosted SSH dispatch.
- The full CI workflow passed on
`ae74808631f1933edec89aa1e80db7f2b756aca3`: server test shards, Runner
Rust and native server tests, type checking, builds, browser suites,
image checks, the native Mac producer, and Linux runtime installation.
Greptile is 5/5 with no unresolved threads.
- Local `pnpm test:run` used a disposable PostgreSQL 18 fixture after
embedded Postgres failed to start because its macOS package lacks
dynamic libraries. All six chat shards passed (1,971 tests). The
redundant unsharded local run was interrupted during the general-server
group after all current-head CI test jobs passed. The later local
workspace and serialized groups did not run. The owned PostgreSQL
fixture is stopped. This is not a local full-suite pass.

Retained review failures: the first PR revision omitted package metadata
and did not ignore downloaded image inputs. The fresh-host upload
regression reproduced the metadata failure before its fix. Generated
inputs are now ignored. CI also exposed a missing Node policy exception
for the pinned Mac producer; the exception is limited to that producer
job. A later policy attempt exposed an old exact-dependency assertion in
canary reuse tests. The assertion now covers the added runtime and
evaluates the publication gate for success, failure, cancellation, and
skip outcomes. The Runner support suite then found that the new test
registration changed a pinned root package input; the standard source
registry generator updated that one hash, and all 59 affected tests
passed. The server matrix then found four outdated image guard
expectations and a duplicate production commit ARG. The focused image
guards now pass after updating those expectations and keeping one commit
ARG before Mac verification.

Retained failures: the first producer attempt used Homebrew Node, which
failed the relocation check. The successful producer used standalone
Node 24.21.0. Live smoke attempts also encountered a stopped tunnel, an
upstream model failure, and the existing Rust restriction on OpenCode
permission requests in `ask` mode. The successful tool trace used the
existing `allow` mode. Its initial reporter looked for tool names in
generic item events; verification of the retained canonical tool events
and an independent rerun confirmed the result. The upstream model
failure is not explained by a later pass.

Not run locally: a full Docker image build, because the existing Docker
daemon does not respond. The PR adds native producer qualification and
Linux installation verification to CI. Hosted deployment, remote
service-account dispatch, and hosted continuation remain acceptance
steps after deployment.

## Risks
- A cold Mac runtime build adds CI time when Runner or producer inputs
change. Reuse is bounded and accepts only a successful protected master
producer. The artifact digest and runtime input identity remain
verified.
- Standard images gain a dedicated Mac runtime payload. Manual image
builds can omit it; Mac OpenCode SSH runs then fail with a named
missing-runtime error. Linux behavior remains available.
- This release covers Apple Silicon SSH distribution. It does not
qualify Intel Mac distribution or expand other harness support.
- The OpenCode permission bridge keeps its existing behavior. This PR
does not add `ask` approval support.
- No saved run/session schema or provider pin changes. Existing Linux
receipts retain their interpretation. A surviving admitted Runner
bypasses new artifact discovery. Legacy agents are not converted.
Existing account-change and old-receipt restoration tests remain in
coverage.
- No migration, cluster change, new sandbox image requirement, or Cloud
control-plane change. A staging application rebuild and deployment are
separate from this PR.

## Model Used
OpenAI GPT-6, through Codex, with repository tools, code execution,
tests, and SSH inspection. The session does not expose a more specific
model ID or context-window size.

## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-22 00:07:34 -08:00
4ec1447a67 fix(heartbeat): deliver a saved answer on a reviewed task with open blockers (#22525)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can ask a person a question on a task. The saved answer
wakes the agent.
> - Dependencies stop task work until the blocker tasks finish. #20363
lets a verified human answer reach the assignee in a bounded turn while
blockers stay open. That turn does not check out the task.
> - Run preparation used the bounded path only when the task status was
`blocked`. A task can wait in review (`in_review`) and still have open
blockers.
> - For such a task, dispatch admitted the answer turn. Then preparation
tried the normal resolved-interaction checkout. The checkout refused the
run with "Issue is blocked by unresolved blockers" (HTTP 422).
> - The run failed `setup_failed`. The answer did not reach the agent,
and Sentry received the refusal as an unexpected run failure.
> - This pull request keeps a verified answer in the bounded turn
whenever the task still has open blockers.
> - The benefit is that the person's answer reaches the agent, and the
task and its blockers do not change.

## Linked Issues or Issue Description

Refs #20363 (deliver saved answers to dependency-blocked tasks). This
change covers the same answer on a task that is not in `blocked` status.

**What happened?**

A parent task waited in review. Some of its child blocker tasks were
still blocked. The assignee had asked a question on the parent. A person
answered it. Dispatch admitted the answer turn with
`dependencyBlockedInteraction: true`. Run preparation then called the
resolved-interaction checkout, because the task status was `in_review`,
not `blocked`. The checkout dependency check threw "Issue is blocked by
unresolved blockers". The run failed `setup_failed` about one second
after it started. The adapter did not run. Sentry received the error.

**Expected behavior**

The saved answer reaches the assignee as a bounded answer turn. The task
stays `in_review`. Its blockers and dependency edges do not change. When
the last blocker has already finished, the task checks out normally, as
before.

**Steps to reproduce**

1. Assign a task to an agent and set its status to `in_review`.
2. Add a `backlog` task that blocks it.
3. Save an agent question on the task and answer it as a person.
4. Dispatch the resolved-interaction wake.
5. Observe that the run fails `setup_failed` with "Issue is blocked by
unresolved blockers".

**Paperclip version or commit**

Seen on a hosted staging server. The new regression case fails on
`master` at a722935dbc and passes with this change.

## What Changed

- `server/src/services/heartbeat.ts`: run preparation treats a verified
dependency-blocked answer as a bounded turn without checkout when the
task is `blocked` (unchanged) or when its dependencies are still not
ready. Before this change, only the `blocked` status qualified.
- The answer still needs the same proof as before: the stored
interaction is human-resolved, belongs to this task and agent, and
matches the wake's status and kind.
- `server/src/__tests__/heartbeat-dependency-scheduling.test.ts`: the
existing saved-response matrix gets two cases:
- `in_review_answered`: a reviewed task with an open blocker receives
the answer once and stays `in_review`.
- `in_review_blocker_finishes`: when the blocker finishes before
preparation, the reviewed task checks out normally and becomes
`in_progress`. The answer is delivered once.

## Verification

- Red before the fix, green after: `delivers only a saved human response
while dependencies remain blocked: in_review_answered`. Before the fix
the run ended `{ status: "failed", errorCode: "setup_failed", error:
"Issue is blocked by unresolved blockers" }`, which is the staging
failure. After the fix it succeeds.
- Safety case `in_review_blocker_finishes` passes before and after the
fix. A reviewed task with no open blockers keeps its normal checkout.
- The existing cases still pass. They cover a pending, missing,
wrong-task, or agent-resolved response that stays blocked, a response
that changes before dispatch, workspace waits, and a blocked task whose
blocker finishes.
- Commands (local, Node 25, embedded Postgres, branch at `master`
a722935dbc plus this change):
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-dependency-scheduling.test.ts
src/__tests__/heartbeat-agent-directory-busy.test.ts
src/modules/run-dispatch/domain/wake-context.test.ts
src/modules/run-dispatch/domain/policy.test.ts
src/__tests__/deferred-execution-receipt-work.test.ts
src/services/explicit-native-continuation.test.ts` (613 passed)
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-clean-continuation-dispatch.test.ts
src/__tests__/heartbeat-isolated-continuation-hold.test.ts
src/__tests__/issue-thread-interactions-service.test.ts
src/__tests__/agent-interaction-acceptance.test.ts` (148 passed)
  - `pnpm --filter @paperclipai/server typecheck` (passed)

## Risks

- Low risk. The change applies only to a wake that dispatch already
admitted as a verified dependency-blocked answer, and only while the
task still has open blockers. Before this change, that run always failed
at checkout.
- The bounded turn does not check out the task, change its status, or
resolve a blocker. Pause, ownership, approval, budget, and execution
gates are unchanged.
- No migration. No change to the API or to stored data.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
(shell, file edit, and test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
doc change: the code comment and tests record the contract)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:47:47 -08:00
DottaandClaude Opus 5.5 7a4a5bba8a test(server): advance the maintenance fake clock only after database connections finish startup (#22523)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The maintenance server tests boot the real server in a child process
with a fake clock. The test moves the clock forward (up to 30 s per
step) to reach deadlines fast.
> - The fake clock also replaces the timers of the PostgreSQL driver.
The driver starts a 30 s connect timer when it opens a connection, and
stops it at the first ReadyForQuery.
> - If the test moves the clock 30 s while a new database connection is
still in startup, the connect timer fires at once. The query fails with
`CONNECT_TIMEOUT`, the queue goes into retry, and the server does not
settle.
> - This race stopped the Release for 856f14e507 in `verify / Serialized
tests (2/5)`.
> - This pull request makes the child wait until every database
connection has finished startup before it moves the clock.
> - The benefit is that clock steps never fire a real-network timeout,
so these tests no longer depend on runner speed.

## Linked Issues or Issue Description

**What happened?**
The Cloud readiness run 44954319430 (Release for 856f14e507) failed in
`verify / Serialized tests (2/5)`. The failed test is
`maintenance-server-routes.test.ts > assembled maintenance server
production wiring > reacts to the first ordinary status-card source
write and accepts authorized controlled summarizer output`. The error is
"Maintenance did not settle". The child log shows `write CONNECT_TIMEOUT
127.0.0.1:<proxy>` from `connectTimedOut` in postgres.js, called from
`MockTimers.tick` at `server-child.ts:86`. That line is the `advance`
command. The same log shows a false 28.6 s "event loop stall" from the
same 30 s clock step.

**Expected behavior**
A fake clock step moves only the server's own deadlines. It must not
fire the driver's connect timeout for a connection that is still in
startup on the real network.

**Steps to reproduce**
1. Run `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/maintenance-clock-database-startup.test.ts`.
2. The first test starts a PostgreSQL connection against a server that
holds its startup reply, moves fake timers by 30 s, and then lets the
reply through. The query fails with `CONNECT_TIMEOUT`. This is the CI
failure in a deterministic form.
3. The second test waits with the new startup tracker before the same 30
s step. The query succeeds.

**Paperclip version or commit**
master at 856f14e507. The race exists since the fake-clock `advance`
command came in; the parent-side guard (`proxy.starting`) did not cover
a connection that opens after the guard check.

Classification: timing flake in the test harness (fake clock against a
real-network driver timer). It is not a product regression. The test
passed in every earlier Cloud readiness run on master.

## What Changed

- New helper
`server/src/__tests__/helpers/maintenance/database-startup.ts`. It uses
the `net.client.socket` diagnostics channel to track each client socket
from `connect()` until the first ReadyForQuery from the database (or
until it closes or is not a database socket). It reads only backend
frames with the existing protocol observer.
- `server-child.ts` waits for that tracker before each `advance`, and
then moves the clock in the same synchronous step. The driver starts its
connect timer in the same synchronous call that publishes
`net.client.socket`, so no connection can start its timer between the
check and the step.
- New test
`server/src/__tests__/maintenance-clock-database-startup.test.ts` with
the red case (unguarded step gives `CONNECT_TIMEOUT`) and the green case
(guarded step succeeds).
- No product code changes. No timeout changes.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/maintenance-clock-database-startup.test.ts`: 2 of 2 pass.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/maintenance-server-routes.test.ts -t "first ordinary
status-card source write"`: passes locally.
- Full `maintenance-server-routes.test.ts`,
`maintenance-postgres-proxy.test.ts` and
`maintenance-harness-resources.test.ts`: pass locally.
- `node scripts/typecheck-test-projects.mjs
server/src/__tests__/helpers/maintenance`: passes.

## Risks

- Low risk. Test-harness change only. If a database connection never
finishes startup, `advance` now fails after 30 s with a clear message
instead of a later "did not settle".

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, GitHub CLI) for investigation, the
fix and local verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 21:25:19 -06:00
sentry[bot] ac02de7732 fix(server): validate agentId query parameter on company activity endpoint (#22516)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - This change is in the activity log API, `GET
/companies/:companyId/activity`.
> - The route sends the `agentId` query value directly to the database.
The `activity_log.agent_id` column is a UUID column.
> - A malformed value makes PostgreSQL fail, and the client gets a 500
response for a client mistake.
> - This pull request validates the `agentId` query value as a UUID and
returns HTTP 400 for a malformed value.
> - The benefit is that clients get a clear 400 error, and the server
does not report a false server error.

## Linked Issues or Issue Description

No public GitHub issue exists. This is the bug description.

**What happened?**

A client sent `GET /companies/:companyId/activity?agentId=dc04285a-`.
The route sent the truncated value to an `eq(activityLog.agentId, ...)`
condition. PostgreSQL failed with `invalid input syntax for type uuid:
"dc04285a-"`, and the server returned 500.

**Expected behavior**

A malformed `agentId` returns HTTP 400 with validation issues. A valid
UUID returns HTTP 200. An empty `agentId` applies no filter.

**Steps to reproduce**

1. Start the server.
2. Send `GET /api/companies/<company-id>/activity?agentId=dc04285a-` as
a board user.
3. Look at the response status. Before this change it is 500.

**Paperclip version or commit**

`master` at `68f641ba0bb7e8e8740bf62cca1cbfab45da8264` and later.

**Deployment mode**

Not specific to one deployment mode. The route is in the core server.

## What Changed

- Validate the `agentId` query on `GET /companies/:companyId/activity`
with `z.string().guid().optional()`.
- Convert an empty `agentId` value to `undefined`. This matches the
service behavior for an omitted filter.
- Throw a standard `badRequest` (HTTP 400) with validation issues for a
malformed `agentId`.
- Update `docs/api/activity.md` with the UUID requirement, the
empty-value behavior, and the 400 response.
- Add regression tests: a malformed `agentId` returns 400 and does not
call the service, and a valid UUID returns 200 and passes the filter.

## Verification

- `pnpm vitest run server/src/__tests__/activity-routes.test.ts` runs
the new validation tests.
- CI on head `ed3078339894bdd68f2093ddaf921fdb297623aa` is green,
including `ci / verify`, `ci / e2e`, and all server test shards.

## Risks

- Low risk. The change is one route and one query parameter.
- A client that sends a malformed `agentId` now gets HTTP 400 instead of
HTTP 500.
- A client that sends an empty `agentId` now gets the same result as an
omitted filter.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Code: Sentry Seer Autofix (Claude 3.5 Sonnet, as reported by Seer),
with tool use.
- PR description update: Claude Opus 5.5 (`claude-opus-5-5`), with tool
use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

<!-- SEER_FIXES_SENTRY_ISSUE -->
Fixes
[PAPERCLIP-BACKEND-4E](https://paperclip-e0.sentry.io/issues/7805680817/?seerDrawer=true)
<!-- /SEER_FIXES_SENTRY_ISSUE -->

<sub>`@sentry <feedback>`: Autofix iterates on these changes</sub>
<sub>`@sentry stop iterating`: Autofix stops iterating on this run</sub>

<sub>This PR was automatically generated by Sentry at no cost. You can
[adjust this
setting](https://paperclip-e0.sentry.io/settings/projects/paperclip-backend/seer/)
at any time.</sub>

---------

Co-authored-by: sentry[bot] <39604003+sentry[bot]@users.noreply.github.com>
2026-11-21 19:20:02 -08:00
scotttong 856f14e507 fix(dashboard): scope areas and responsibilities to each user (#22513) 2026-11-21 13:59:00 -08:00
21f4427b78 fix(server): pass a Cloud portfolio rate limit through as 429 (#22488)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A hosted stack reads its tenant portfolio from Paperclip Cloud
through `GET /api/cloud/stacks`
> - Paperclip Cloud can answer that read with 429 when it throttles a
stack
> - The proxy changed every non-OK upstream status into a 502, and the
error handler reports every 5xx to Sentry
> - So Cloud backpressure showed up as a server crash on a staging stack
(`upstreamStatus=429`, `phase=http_response`)
> - This pull request passes an upstream 429 through as a 429 with a
bounded `Retry-After` hint
> - The benefit is a correct status for clients and no false crash
reports for throttling

## Linked Issues or Issue Description

Refs #15340 (bounded Cloud portfolio diagnostics), Refs #19033 (retry
one Cloud portfolio connection reset).

**What happened?**
A hosted staging stack reported "Paperclip Cloud portfolio request
failed" to Sentry. The diagnostics show `upstreamStatus=429` after 44
ms. Cloud throttled the read, and the proxy answered 502 and reported a
crash.

**What did you expect to happen?**
The proxy tells the client that Cloud is rate limiting the read. It does
not report a server fault.

**Steps to reproduce**
Make the Cloud tenant portfolio endpoint answer 429, then call `GET
/api/cloud/stacks`. Before this change the route answers 502
`cloud_portfolio_upstream_error` and reports to Sentry.

## What Changed

- `server/src/routes/cloud.ts`: an upstream 429 now gives 429
`cloud_portfolio_rate_limited`.
- The route copies Cloud's `Retry-After` as whole seconds. It accepts
delta seconds or an HTTP-date and clamps the value to 1–300 seconds. It
omits the header when the value is missing or malformed.
- The route logs the bounded diagnostics at warn level. It does not
report the 429 to Sentry and does not cache the throttled answer.
- 401, 403, 503 and the other failure phases keep the 502 contract and
the crash report.
- `server/src/__tests__/cloud-routes.test.ts`: new cases for the 429
status, the `Retry-After` forms, no Sentry report, and no cache entry.
- `docs/api/cloud.md` (new, in the API Reference nav): documents `GET
/api/cloud/stacks`, its error codes, and the 429 with its bounded
`Retry-After`.
- `server/src/routes/openapi.ts`: the `/api/cloud/stacks` entry now
lists the 429 response.

## Verification

- Red: with the old `cloud.ts`, `cloud-routes.test.ts` gave 5 failed
(the new 429 cases) and 23 passed.
- Green: `cd server && npx vitest run src/__tests__/cloud-routes.test.ts
src/__tests__/sentry.test.ts src/__tests__/cloud-portfolio-error.test.ts
src/__tests__/run-failure-sentry-real-sdk.test.ts` gave 68 passed and 18
skipped.
- `npx tsc --noEmit -p server/tsconfig.json` shows no errors in the
changed files.
- After review fixes (6a8f146c55): `cd server && npx vitest run
src/__tests__/cloud-routes.test.ts src/__tests__/openapi-routes.test.ts`
gave 80 passed.

## Risks

- Low risk. Only the 429 path changes.
- The two UI consumers (`useCloudInviteUrl`, `CompanyAccess`) treat any
error the same way, so the UI needs no change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
2026-11-21 04:22:41 -08:00
Devin FoleyandPaperclip 1b1861c29d Fix skill upload conflicts blocking first-agent onboarding (#22477)
## Thinking Path

> - Paperclip manages agents and the skills they need to work.
> - First-agent setup resolves the bundled skills before onboarding can
finish.
> - Skill packages use durable upload claims to protect their
object-storage receipts.
> - A failed attempt kept its claim active for ten minutes. A concurrent
caller received an immediate conflict.
> - This pull request expires a failed owner's unpublished claim and
waits for an active upload to settle.
> - Onboarding can retry skill setup without a false upload-in-progress
error.

## Linked Issues or Issue Description

**What happened?**

First-agent onboarding could show "This skill package is being saved.
Retry after its upload settles." during the subscription connection
step. The agent's bundled skill setup called the package store. A failed
upload retained its active claim, and calls from another database client
also returned a conflict while an upload was in progress.

**Expected behavior**

A failed attempt permits an immediate retry with new object keys.
Concurrent callers reuse the completed receipt or claim the failed
attempt. They preserve another owner's active lease and every published
receipt.

**Steps to reproduce**

1. Fail the first object upload while resolving the first agent's
bundled skills.
2. Retry the same skill selection before the ten-minute lease expires.
3. Observe the upload-in-progress conflict. The new regression
reproduces this through
`companySkillService.resolveRequestedSkillEntries`.

**Paperclip version or commit**

Reproduced on `6bf312270c3e`.

**Deployment mode**

Hosted onboarding; the package-store fix also applies to self-hosted
instances.

Related implementation: #18082. Searched open and closed pull requests
and open issues for related skill-upload and lease failures. No
duplicate fix was found.

## What Changed

- Expire only the failed attempt's token-matched, unpublished upload
claim. Keep its exact keys for the existing deletion outbox. Cleanup has
a one-second client deadline and a transaction-local statement timeout.
Queued cleanup skips its write after that deadline; failed cleanup
retains the original crash lease.
- Wait for a competing package preparation within the existing
three-minute deadline. Re-read the receipt every 250 ms while this
concrete request remains active. Release the database connection between
reads and stop on cancellation. This adds no idle worker or permanent
polling.
- Preserve cancellation reasons, published receipts after lost commit
replies, and claims acquired by a newer owner.
- Add real PostgreSQL regression coverage for first-agent bundled skill
setup, cross-client contention, failed-owner takeover, crash expiry,
cancellation, ambiguous publication, and ownership changes.
- Document upload recovery behavior. No schema or public API change is
required.

## Verification

- Red: four new or updated checks failed against the original
implementation. They reproduced the exact conflict and the retained
ten-minute lease.
- Green: 189 targeted tests passed across
`skill-package-upload.test.ts`, `skill-package-store.test.ts`,
`skill-package-work.test.ts`, both package-storage migration suites, and
`agent-skills-routes.test.ts`. The combined run passed 188 tests. The
final SQL-lock regression passed separately.
- Command: `pnpm exec vitest run
server/src/__tests__/skill-package-upload.test.ts
server/src/__tests__/skill-package-store.test.ts
server/src/services/skill-package-work.test.ts
server/src/services/skill-package-storage-migration.test.ts
server/src/services/skill-package-storage-migration-postgres.test.ts
server/src/__tests__/agent-skills-routes.test.ts`.
- `pnpm --filter @paperclipai/server typecheck` passed, including source
and test layers.
- The cancellation test first exposed Node's timer wrapping the caller's
abort reason. The final implementation preserves that reason, and the
full targeted run passed.
- Greptile identified an unbounded cleanup write. Two new
real-PostgreSQL tests reproduced delayed errors and cancellations behind
an occupied one-connection pool. They now pass and prove that late
cleanup skips its write. A further row-lock test proves that the
PostgreSQL statement timeout frees the pool while another transaction
still holds the package row.
- Local setup initially lacked macOS embedded-Postgres library links.
Repaired ignored dependency links before the red and green database
tests. The initial server typecheck also lacked SDK links for excluded
plugin packages; ran the maintained `node
scripts/link-plugin-dev-sdk.mjs` helper before repeating it.
- `pnpm verify --explain` returned exit 2 and requested broader CI
coverage. No full local suite was substituted for that CI requirement.
- `git diff --check` passed. Reviewed the full diff for secrets, private
URLs, personal data, generated files, and run artifacts.
- Final revision: `4b4814ac01522599901a3dc29f650a1b33c0b4e2`. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/44880245673)
passed all applicable checks, including typecheck, build, all general
and serialized server test shards, all e2e shards, Runner verification,
and the final verify gate. [Greptile
review](https://github.com/paperclipai/paperclip/pull/22477#issuecomment-7059199437)
scored this revision 5/5 with no unresolved threads. GitHub reports the
branch as mergeable and clean.

## Risks

Contended preparations now wait within their existing request deadline.
A crashed process or an unavailable cleanup database still uses the
ten-minute lease expiry. Genuine storage failures remain errors; this
change does not hide an unavailable backend. The existing object
deletion grace remains intact.

Live provider authorization and a deployed Cloud onboarding journey were
not run. Tenant release and deployment are separate from this source
fix. No production migration is needed.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh
reasoning effort and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-21 00:25:10 -08:00
Devin FoleyandPaperclip 6bf312270c fix(ci): stabilize database-backed readiness checks (#22476)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - CI checks its recovery workers, attention routes, and external chat
admission against real PostgreSQL.
> - Large fixtures start from an empty database without statistics for
their new rows.
> - Slow query plans can exhaust test deadlines before the behavior
checks finish.
> - Route responses can also leave wakeups writing rows while teardown
deletes their agents.
> - Setup admission takes an endpoint lock that can deadlock with
resource foreign-key checks and agent invalidation.
> - This pull request prepares fixture statistics, drains pending work,
and uses a compatible endpoint lock.
> - The benefit is reliable readiness checks with the same fixtures,
assertions, and deadlines.

## Linked Issues or Issue Description

Refs: #21941, #21831, #21132. The compiler-memory mitigation in #19383
is already merged and is not repeated here.

**What happened?**

The recovery census and review-attention equivalence tests repeatedly
timed out on unrelated PRs. Recovery-action teardown also failed on an
agent-wakeup foreign key, and concurrent first GitHub mentions could
deadlock during resource admission.

Examples: [census
timeout](https://github.com/paperclipai/paperclip/actions/runs/44868742060/job/136738633187),
[review
timeout](https://github.com/paperclipai/paperclip/actions/runs/44856500775/job/136732143246),
[cleanup
failure](https://github.com/paperclipai/paperclip/actions/runs/44866635624/job/136731817497),
and [chat
deadlock](https://github.com/paperclipai/paperclip/actions/runs/44845946341/job/136667344125).

**Expected behavior**

The complete fixtures retain their regression coverage, pending wakeups
finish before deletion, and concurrent chat admission completes with
exactly one setup destination enabled.

**Steps to reproduce**

1. Run the two timeout suites against a freshly seeded database before
its statistics refresh.
2. Complete an issue-update request that schedules an assignee wake,
then delete its fixture immediately.
3. Run the new resource-admission regression on the base revision. It
forces the endpoint/agent lock order and fails with PostgreSQL SQLSTATE
`40P01`.

**Paperclip version or commit**

Base: `33ffaa5b9a2e0954c2afc063572346cb1be534a7`.

## What Changed

- Run `ANALYZE` on the populated fixture tables after seeding the
existing census and review-attention populations. No fixture reduction,
assertion removal, retry increase, or deadline increase.
- Await `drainActiveRunExecutions()` before recovery-action fixture
cleanup. This includes pending wake admission after HTTP responses.
- Use `NO KEY UPDATE` for setup endpoint admission. It still serializes
competing admission transactions and endpoint mutations while allowing
resource foreign-key `KEY SHARE` locks.
- Add a real PostgreSQL regression using the real `ensureResource`
write, with barriers that force concurrent endpoint admission and an
agent-locked resource insert. Both transactions must finish and exactly
one destination must become enabled.
- Document fixture statistics, teardown ownership, and admission locking
in `doc/test-infrastructure.md`.

## Verification

- The new concurrency regression fails on the original endpoint lock
with SQLSTATE `40P01`, then passes with `NO KEY UPDATE`.
- Both timeout suites pass with their full populations and original
deadlines: 8 tests.
- The full recovery-action suite passes: 120 tests.
- Existing GitHub first-repository and disabled-repository integration
checks pass: 7 tests.
- The new PostgreSQL concurrency regression passes: 1 test.
- The local baseline timeout suites passed slowly; the review
equivalence case took 35.8 seconds before statistics and 0.63 seconds
afterward. These are local observations, not a CI speed guarantee.
- The first broad typecheck caught an unsupported
`Promise.withResolvers` type in the new test. The barriers now use
ordinary promises compatible with the repository target. The final `pnpm
-r typecheck` passes.
- `git diff --check` passes.
- `pnpm -r typecheck` and `pnpm build` pass. The full local `pnpm
test:run` attempt stopped after all six chat shards: 1,969 tests passed
and two Slack tests failed on skill-root discovery. The remaining local
groups did not run. Both failures reproduce with the production change
reverted to `origin/master`: the resolver selects an unrelated ancestor
`skills/` directory without `slack/SKILL.md`, which also prevents the
mocked Slack run from starting. This is a confirmed baseline
local-layout issue, not a passing full-suite result. The [complete CI
run](https://github.com/paperclipai/paperclip/actions/runs/44873350026)
passes on `66c146b177b50e052a21921644dfd626c96c010d` on attempt 1. The
PR check rollup has 58 successes, three expected skips, and no pending
or failing checks. [Greptile reviewed the current head
`66c146b`](https://github.com/paperclipai/paperclip/pull/22476#issuecomment-7058185817)
at 5/5 with no actionable findings.

Review follow-up verification:

- Replaced the resource-read stub with the real
`createChatInboundAuthority(...).ensureResource` persistence path. The
revised regression fails with `FOR UPDATE` and PostgreSQL `40P01`, then
passes with `NO KEY UPDATE`.
- Scoped `ANALYZE` to the fixture tables and moved the behavior note
from `DEVELOPING.md` to `test-infrastructure.md`.
- All 9 affected tests pass after these edits; the complete server
typecheck also passes. The production lock change is unchanged from the
earlier full workspace typecheck and build passes.
- The previous CI attempt had two interrupted jobs: [Runner static
checks](https://github.com/paperclipai/paperclip/actions/runs/44872465981/job/136749952142)
and [serialized server
9/9](https://github.com/paperclipai/paperclip/actions/runs/44872465981/job/136749952782).
Both logs record a runner shutdown. The static lane reported 2,371
passing tests before cancellation; the interruption is not counted as a
pass. The final head passes the complete fresh CI run linked above,
including both lanes.

## Risks

Low scope. Production code changes only the endpoint row-lock strength.
The regression uses real foreign keys and connector invalidation
triggers; the existing integration checks retain first-destination
serialization and disabled-destination behavior. No schema, API, or
authorization changes. `ANALYZE` updates only test-database statistics.
Drain failures still fail teardown.

All current-head remote checks now pass, including typecheck, build,
tests, and e2e. The baseline local full-suite failure and unrun
remaining local groups are recorded above; focused local checks pass.
The already merged compiler-input fix does not establish the cause of
every past runner shutdown.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh
reasoning and shell/code execution. The session does not expose its
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-20 22:52:05 -08:00
Nicky Leach 467fd32223 Publish tool profiles and their selectors atomically (#22455) 2026-11-20 19:02:54 -08:00
Nicky Leach 7f4cfae5e1 Narrow exact connected tool reads before loading schemas (#22457) 2026-11-20 19:02:39 -08:00
Devin FoleyandPaperclip 18978bdc1d fix(native): preserve named human unblock owners (#22438)
## Thinking Path

> - Paperclip manages work carried out by agents and people.
> - Native finalization converts a runner result into a durable task
status and follow-up effects.
> - A blocked result can name the person who must decide what happens
next.
> - The classifier discarded that identity, and the arbiter saved a
generic board owner.
> - This could replace an explicit user owner saved through the task API
and leave the decision outside that person's inbox.
> - This pull request resolves human owners against active company
membership and preserves them through commit.
> - The named person receives the existing blocker card, and the blocked
worker does not get another wake.

## Linked Issues or Issue Description

**What happened?**

Native finalization replaced a named human unblock owner with `board`.
It also overwrote an explicit user owner saved through the task API
during the turn.

**Expected behavior**

Keep a valid human unblock owner and route the blocker card to that
person's inbox. Use the board when the reference is unknown, inactive,
ambiguous, or outside the company.

**Steps to reproduce**

1. Create an active company member with a distinct display name.
2. Have a native worker report a human blocker using that name.
3. Finalize the run and read the task's unblock owner and the member's
inbox.

Related: #21265 covers named agent handoffs and retains board ownership
for people. This change covers human member routing and does not add
agent handoffs.

## What Changed

- Resolve a human owner from an explicit user ID, an API-bound user, or
one matching company member by ID, full name, or email.
- Preserve the user owner for both blocker scopes and durable dependency
waits.
- Recheck the accepted result and active membership under the status
commit lock, including board fallbacks. Rebuild the assessment on every
status retry so a newly saved owner or revoked membership cannot be
overwritten by stale evidence.
- Enforce `canAssignTasksToHumans` before binding a human owner,
including dependency waits that do not emit a `bind_blocker` effect.
- Validate the result before retiring automatic completion reviews.
Invalid results leave existing reviews pending; valid results retain the
normal cleanup path.
- Keep human blocker routing out of the agent wake queue.
- Add embedded Postgres regressions for finalization, replay, personal
attention, invalid owners, late API ownership changes, membership
revocation, permission denial, and answer continuation for human-owned
blockers.
- Make the answer-continuation fixture persist the same result it
classifies, so commit-time evidence validation sees realistic saved
evidence. Assert the resolved user owner in both blocker scopes.
- Let the recovery census clock advance at real-time pace during
delivery I/O waits. This handles callbacks and nested timers without
jumping to maintenance deadlines. Preserve the existing real-clock
timeout and recovery assertions, and name pending queues in failures.
- Document human blocker resolution and compatibility.

## Verification

- Red-green reproduction: three ownership regressions failed on the
original implementation and passed with the fix.
- Red-green race reproduction: late API ownership changes and membership
revocation failed before the commit recheck and per-retry reassessment
fixes. Each finalization invocation now has to commit; a later replay
cannot hide an exhausted retry.
- Red-green review-retirement reproduction: a new invalid result
cancelled a proven automatic review before throwing
`native_result_invalid_disposition`. The new invalid/valid regression
pair verifies that invalid results preserve the pending review and task
decision, while valid completion still retires the card.
- Red-green answer-continuation reproduction: both human-owner cases
failed with `native_status_race` while saved evidence was empty.
Persisting the classified result and asserting `{ userId }` restored all
13 answer-continuation cases, including worker resumption.
- Red-green permission reproduction: an agent with
`canAssignTasksToHumans: false` could bind a human blocker before the
permission check. Direct and dependency cases now refuse finalization
without creating a personal blocker card.
- On refreshed head `a6d37916546`, `pnpm --filter @paperclipai/server
exec node scripts/typecheck-layers.mjs` passed on macOS with Node 24.
- On refreshed head `a6d37916546`, from the repository root, `CI=true
node node_modules/vitest/vitest.mjs run
server/src/services/native-runtime/native-agent-blocker.integration.test.ts
server/src/services/native-runtime/status-arbiter.test.ts
server/src/services/native-runtime/evidence-classifier.test.ts
server/src/__tests__/native-status-reassessment-work.test.ts
server/src/__tests__/routable-blocked.test.ts
server/src/__tests__/native-status-arbiter-corpus.test.ts
server/src/services/native-runtime/native-environment-blocker.integration.test.ts
server/src/__tests__/heartbeat-blocked-question-answer.test.ts
server/src/__tests__/issue-recovery-live-run-agent-census.test.ts`
passed: 292 tests in nine files on macOS with Node 24.
- Red-green census reproduction: both post-I/O callbacks and nested
timers must settle without jumping to a future maintenance deadline. The
nested case timed out under a zero-only clock pump; the shared
automatically advancing census clock handles both.
- Final CI at `a6d37916546`: [run 44848053016, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/44848053016/attempts/2)
completed successfully. Build, Canary Dry Run, policy, every server and
workspace test shard, Runner checks, serialized suites, e2e, and verify
are green. Greptile review 7 is 5/5 on the same head, Superagent and
other security checks pass, and no review threads remain unresolved. A
local merge-tree check against master `621c5577cdf` also passed with no
conflicts.
- Current CI at `a6d37916546`, [run
44848053016](https://github.com/paperclipai/paperclip/actions/runs/44848053016):
attempt 1 finished with 13 interrupted nonaggregate jobs, each with an
explicit runner shutdown signal in its log. Nine signals occurred within
three seconds on separate m6a.xlarge hosts in us-east-1a; Build and
Canary also shut down later. Aggregate e2e and verify failed on those
dependency results. The underlying termination cause remains unknown.
Shared CI settings are unchanged.
- The [automatic recovery
run](https://github.com/paperclipai/paperclip/actions/runs/44849704877)
skipped retry because Build had only the annotation `Process completed
with exit code 143.` The classifier reads logs for cancellation
annotations, but did not read this job's log. That log explicitly
records a runner shutdown at 01:10:49 UTC. The other 12 failed jobs were
recognized as runner loss. After verifying all 13 logs, manually
requested failed-job attempt 2; it completed successfully without source
or configuration changes between attempts. The shared recovery
classifier is unchanged; this false negative remains a documented
follow-up.
- Historical CI at `cf601ba`: all server shards (including the census),
Runner checks, typecheck/release registry, serialized suites, e2e, and
canary dry run passed. Build received runner shutdown signals on
attempts 1–7; aggregate verify failed because Build failed. Attempt 5
diagnostics show shutdown while the job lease was valid. Attempt 7
stopped after server dependency preparation, without a compiler error.
The termination cause remains unknown. Three recent PRs completed Build
on the same AWS fleet; this is not evidence of a fleet-wide outage.
Shared CI settings are unchanged.
- Diagnostic comparison at `cf601ba`: the server compiler passed locally
on macOS with four checkers, full typechecking, and
declaration/JavaScript emission (`node ../scripts/run-tsc.mjs --checkers
4 --extendedDiagnostics` under Node 24). The monitored process group
peaked at about 10.5 GiB RSS and finished in 25.8 seconds. This does not
establish the cause of the Linux runner shutdowns.
- Refreshed the branch onto master `611dc68581c` without conflicts; the
PR source diff is unchanged. Greptile review 7 scored head `a6d37916546`
5/5 with no new actionable findings and no unresolved review threads.
All applicable CI gates passed on attempt 2.
- Full local workspace build at `cf601ba` passed on macOS using Node 24
and the documented single-thread compiler mode:
`PAPERCLIP_TYPECHECK_LOW_MEMORY=1 npx --yes --package=node@24
--package=pnpm@9.15.4 pnpm build`. This includes typechecking and
artifact emission. The tracked working tree stayed clean.
- `pnpm verify --explain` returned exit 2 because this documentation
path requires broad CI coverage. Fresh CI will supply that coverage; the
full local test suite was not run.
- Earlier CI build and canary jobs received runner shutdown signals. The
server shard also exposed an answer-continuation fixture that saved an
empty result while classifying a named human. That mismatch was
reproduced locally, and the fixture now persists the classified result
and checks the user owner. Fresh CI is required for this revision.
- `git diff --check` and the staged gitleaks scan passed. The diff was
also checked for private issue references, identifiers, URLs, and
personal data.

## Risks

- Name and email matching ignore case. Several matches fall back to the
board. Invalid explicit IDs never fall through to another person.
- Resolution adds bounded reads only when a native result reports a
human blocker. It adds no worker, timer, or idle scan.
- No database or result-schema migration is required. Existing board and
agent decisions remain readable. Committed historical decisions are not
rewritten; new finalizations use current membership.
- This changes human routing only. It grants no access and changes no
merge, approval, or dependency policy.

## Model Used

- OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex in T3 Code, with
xhigh reasoning effort, code editing, shell tools, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-20 17:45:45 -08:00
621c5577cd fix(tasks): hide More messages when no hidden history exists (#22099)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The task page shows the instruction and the recent exchange first,
and shows a **More messages** button for older history
> - The button is visible on almost every task. A click often shows no
new message, and the button then goes away
> - The compact chat shows only the latest run, so the page treats any
task with two or more runs as a task with hidden history
> - The recent comments read also reports more comments when the omitted
comment is already pinned into the window
> - This pull request shows the button only when a message is really
hidden
> - The benefit is a clean task page and a button that always reveals
something

## Linked Issues or Issue Description

Refs #17751

**What happened?**

The task page shows **More messages** on almost every task. A click
often adds no message, and the button then goes away.

**Expected behavior**

The page shows **More messages** only when it hides an older comment or
an older run that has no visible answer.

**Steps to reproduce**

1. Create a task and let an agent answer it in two runs.
2. Open the task with a fresh application cache.
3. See **More messages** above the conversation. Click it. No new
message appears.

## What Changed

- The recent run window keeps the newest omitted run (`earlierRun`), and
it keeps that run as later turns arrive.
- The task page ignores the omitted run when the run succeeded and its
agent answer is already visible. A failed run, or a run with no visible
answer, still shows the button. The answer check is the same one the
chat thread uses (moved to `ui/src/lib/run-answer-visibility.ts`), so a
legacy withheld-run placeholder or a progress comment does not count as
an answer.
- `GET /issues/:id/recent-comments` reads seven rows instead of five. It
sets `hasMore` only when the window leaves out a comment. Older comments
that are pinned into the window (the latest human message and the first
instruction) do not count.
- Added regression tests for the page (including the placeholder case),
the run window cache and the server route.

## Verification

- `pnpm exec vitest run src/pages/IssueDetail.test.tsx
src/components/TaskChatThread.test.tsx src/lib` in `ui`: 3,637 passed, 1
skipped. The new page test fails without the fix.
- After the review fixes: `pnpm exec vitest run
src/pages/IssueDetail.test.tsx src/components/TaskChatThread.test.tsx
src/lib/issueChatBootstrapCache.test.ts
src/lib/issueDetailCache.test.ts` in `ui`: 738 passed, 1 skipped.
- `pnpm exec vitest run src/__tests__/task-cold-open.test.ts
src/__tests__/issue-chat-bootstrap-routes.test.ts` in `server`: 33
passed.
- `tsc --noEmit` in `ui` and `server`: no errors in the changed files.
- I did not do a check in the browser.

## Risks

Low risk. No schema change. The API response shape does not change. The
run window cache gets one optional field, and older saved caches without
the field keep the old behavior.

Limitation: the recent runs read returns only the two latest runs, so
the page sees one omitted run. If that run has a visible answer but an
older run failed, the page does not show the button. **More task actions
→ Show task activity** still shows that run.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 17:41:17 -08:00
Nicky LeachandPaperclip b6a904554d Load summaries for the Access Profiles index (#22456)
## Thinking Path

> - Paperclip helps people manage AI agents for work.
> - The Access Profiles page shows profile names, counts, and agent
assignments.
> - The index loaded every selector and started detail queries for each
connection.
> - The server can compute the displayed fields with the existing
visibility and review rules.
> - A summary route lets the index fetch just the data it renders.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Opening the Access Profiles index for a company with many assigned
tools.

**Current behavior**

The index downloads full profiles and requests other resources used by
detail views.

**Proposed behavior**

Fetch summaries and agents at mount. Keep the existing full-profile
route for detail callers.

**Reason and benefit**

In a generated fixture with 166 profiles and 176,257 entries, the
uncompressed response fell from 86,879,901 bytes to 58,095 bytes. The
mounted query count fell from 28 to 2 with 22 connections.

## What Changed

- Add the typed company summary route and its OpenAPI entry.
- Compute counts with the existing projection and use the same profile
visibility rules. Group equal selectors in PostgreSQL and send their
profile memberships as ordinals.
- Read summaries from the index. Keep profile mutations invalidating the
index.

## Verification

- Local UI typecheck and 27 UI tests passed. PostgreSQL projection and
route tests checked count parity, delegated visibility, and rejected
access.
- Layered server typecheck passed. PostgreSQL tests also covered all
selector types, null and unknown legacy selectors, duplicate
memberships, and overlapping include/exclude rules.
- On a disposable database with fresh PostgreSQL statistics, 166
profiles and 176,257 memberships, four interleaved trials per summary
implementation returned identical 58,095-byte JSON responses. PostgreSQL
transferred 36,338,231 bytes for flat summaries and 1,819,600 bytes for
grouped summaries (95% less). Both used five statements. Median service
and serialization time fell from 589 to 227 ms (564–627 ms versus
216–238 ms).

## Risks

The summary still enumerates selector memberships in PostgreSQL.
Grouping raised warm PostgreSQL executor time for the selector query
from about 17 ms to 136–137 ms and used about 15 MiB for the temporary
sort in the fixture. The smaller transfer reduced total service time in
the same synthetic workload. Exact timing will vary with database
resources and profile shape.

## Model Used

OpenAI Codex (GPT-6), with GPT-6.2 Sol agents for implementation and
local verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-20 16:45:34 -08:00
Devin FoleyandPaperclip ae2afd525b Make harness choices deployment-configurable
Managed deployments need to control harness availability without changing Core policy. Add a generic environment contract and apply it across harness and model selectors, preserving saved supported profiles and validating import, team, and invite selections. Document deployment ownership in agent guidance.

Validation: focused local tests and typechecks passed; all applicable CI checks passed on the reviewed head. Greptile scored 5/5 with no unresolved comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-20 15:45:36 -08:00
Devin FoleyandPaperclip 9bc90cb189 Show useful task notification content (#22441)
Task notifications repeated identifiers and omitted useful activity details.

Use task titles, actor and comment previews, and concrete field changes. Share bounded secret redaction per event after viewer access checks. Suppress updates without useful details.

Verified with focused regression tests, package typechecks and server build, and the full green CI run. Greptile scored the final head 5/5 with no unresolved review threads.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-20 14:40:39 -08:00
Devin FoleyandPaperclip 2898acee2d fix: exclude responsibility repairs from Mine participation (#22434)
Exclude responsibility-repair audit entries from personal task
participation. Maintenance attributed to an operator no longer pulls
unrelated tasks into Mine, unread badges, or task notifications.

Preserve audit history and genuine participation through creation,
assignment, comments, deliberate edits, and pending requests. Add database
regressions and document the rule.

Verified with 174 targeted tests, server typecheck, fully green CI,
and Greptile 5/5 with no unresolved review comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-20 11:28:11 -08:00
Nicky LeachandClaude Opus 5.5 dce4d47f60 feat(plugins): carry Grok, Copilot and OpenCode credentials to plugin Runners (#22359)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Environment-driver plugins can run `paperclip_runner` tasks on hosts
that the plugin owns
> - Core transfers the selected managed AI credential to the plugin as a
typed task credential
> - The credential contract has only two types: a Codex auth file and a
direct Claude credential
> - Grok, GitHub Copilot and OpenCode agents in a task-capable plugin
environment therefore fail closed, although the Runner already reads
their credentials from its environment
> - This pull request adds one generic, allowlisted environment
credential for these three harnesses and accepts the Runner's Grok
launch settings
> - The benefit is that plugin-provided Runner execution can run Grok,
Copilot and OpenCode agents with their selected connection

## Linked Issues or Issue Description

**Subsystem affected**

Plugin environment tasks (`packages/plugins/sdk` task contract) and
plugin-provided Runner execution in `server`.

**Problem or motivation**

An agent with the Grok, Copilot or OpenCode harness cannot run in a
plugin environment that runs Runner tasks. Core stops at credential
preparation with "Plugin Runner does not support the selected credential
provider". A Grok agent fails before that point too. The Runner emits
`grokEffort` and `grokDisableWebSearch` in its launch requirements (Core
turns off Grok web search by default), but the strict `settings` schema
of `environmentRunnerLaunchRequirementsSchema` rejects both keys.

**Proposed solution**

Core transfers the selected connection's credential variables for these
harnesses through one typed `provider-env` credential. Each provider has
a fixed allowlist of complete variable sets. The plugin sets the
variables in the Runner process environment.

**Alternatives considered**

One credential type per harness (`grok-env`, `copilot-env`,
`opencode-env`). This adds a type for each new harness. One generic type
with an exported allowlist keeps the same strictness with less surface.

**Roadmap alignment**

The roadmap lists Grok and OpenCode as supported harnesses. This change
lets plugin environments run them. It adds no new product concept.

**Additional context**

Grok subscription login (an auth file with refresh and writeback) is out
of scope. It still fails closed, with an error that names the xAI API
key as the supported option.

## What Changed

- `packages/plugins/sdk`: Added the `provider-env` task credential: `{
type: "provider-env", provider: "grok" | "copilot" | "opencode",
environment }`. The environment must contain exactly one complete
variable set for the provider and no other variable. Values are
non-empty, NUL-free and at most 65,536 characters, as for the other
credential types.
  - `grok`: `XAI_API_KEY`
  - `copilot`: `COPILOT_GITHUB_TOKEN`
- `opencode`: `OPENROUTER_API_KEY`, or `PAPERCLIP_AI_PROVIDER_URL` with
`PAPERCLIP_AI_PROVIDER_KEY`
- `packages/plugins/sdk`: Exported the allowlist as
`ENVIRONMENT_TASK_PROVIDER_CREDENTIAL_KEY_SETS` and the
`PluginEnvironmentTaskCredentialProvider` type. `claude-env` and
`codex-auth-json` do not change.
- `packages/plugins/sdk`: Added `grokEffort` (`minimal` to `xhigh`) and
`grokDisableWebSearch` (boolean) to the launch requirements `settings`,
with the types that the Runner uses.
- `server`: `preparePluginRunnerCredentials` now transfers the selected
connection's variables for the `grok`, `copilot` and `opencode`
harnesses, with the same account identity semantics as the Claude
branch. It refuses a Grok subscription, a Grok base URL override, and an
incomplete or ambiguous set. The error never contains a credential
value.
- `server`: The submit check now matches a `provider-env` credential to
the task harness. Before, it treated every non-Codex credential as
Anthropic.
- Drift test:
`server/src/__tests__/plugin-runner-credential-names.test.ts` checks the
SDK key sets against the Runner. The SDK list stays written by hand, and
the Runner keeps its own lists of forwarded names. For `grok` and
`copilot`, each name must be in `ACPX_CREDENTIAL_NAMES`, survive sidecar
admission with the session binding that the Runner mints, and reach the
agent process. The binding requirement must match the Rust sidecar's
candidate agents. For `opencode`, each name must be in the OpenCode
proxy environment set. Each name must also be in the Rust sidecar
credential keys or the Rust OpenCode provider keys. The test reads those
from the Runner source, as the Codex security test does. A provider
without a Runner route fails the test.
`OPEN_CODE_RUNNER_ENVIRONMENT_KEYS` is now exported for this test.
- Docs: Added a "Typed credentials" section to
`doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`. It also documents the ACPX
credential binding that the Runner requires with a Copilot token.
Updated `doc/plugins/PLUGIN_RUNNER_EXECUTION.md`.

## Verification

- `npx vitest run tests/environment-task-release-resume.test.ts
tests/environment-tasks.test.ts` in `packages/plugins/sdk`: 96 passed.
The tests accept each valid shape. They reject mixed sets, two sets,
extra variables, another provider's variable, empty values, NUL bytes,
oversized values, unknown providers and extra fields, on `submit` and on
`resume`. They also check that every published allowlist set parses.
- `npx vitest run
src/services/native-runtime/plugin-runner-credentials.test.ts
src/__tests__/environment-task-runtime.test.ts` in `server`: 79 passed.
The new tests cover each provider's transfer, Grok subscription refusal,
Grok routing refusal, the fail-closed cases without value leaks, harness
matching at submit, and Runner-built Grok requirements
(`runnerProviderLaunchRequirements`) that parse through the task
contract and produce a Grok launch.
- `npx vitest run src/__tests__/plugin-runner-credential-names.test.ts`
in `server`: 4 passed. I broke each list locally to confirm that the
test fails: a renamed SDK key, an SDK provider without a Runner route,
and a dropped name in the Rust sidecar keys, the Rust candidate agents,
the Rust OpenCode keys and the TypeScript OpenCode keys. Each change
failed the test.
- `node scripts/typecheck-layers.mjs` in `server` (includes
`src/__tests__`): no errors.
- `npx vitest run
server/src/services/native-runtime/plugin-runner-probe.test.ts` (from
the root): 15 passed.
`server/src/services/native-runtime/plugin-runner-task.test.ts`: passed.
- `tsc --noEmit` for `packages/plugins/sdk` and `server`: no errors.
- I did not run the full suite locally. CI supplies that coverage.

## Risks

- Low risk for existing plugins. The change only adds a credential type
and two optional settings. A plugin that does not handle `provider-env`
gets a typed value that it does not recognize. Before this change, Core
failed these runs before it called the plugin.
- A plugin must set `PAPERCLIP_ACPX_CREDENTIAL_BINDING` for a Copilot
task. Otherwise the Runner refuses the token. The new doc section shows
the value.
- An OpenCode gateway connection without a key (`auth: none`) still
fails closed, because every value must be non-empty.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, through Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 08:47:18 -08:00
DottaandClaude Opus 5.5 a0fe90f24b fix(retry): never offer Retry while a live run owns the task (#22428)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each task shows an interruption banner. The banner offers Retry when
an attempt failed and nothing will move the task forward
> - The banner reads the task's newest run. A continuation that the
system cancels before it starts can be the newest run, while a different
run owns the task and does the work
> - In a restart-chaos wave, the banner offered Retry for about 140
seconds while the owning run worked. The task then finished with no
click
> - A Retry that the user does not need breaks the rule that the
needed-Retry rate is 0%
> - This pull request makes the banner and the Retry endpoint read the
same observed run. That run is the live run that owns the task, or else
the newest real attempt
> - The benefit is that Retry shows only when nothing will advance the
task, and a shown Retry is accepted

## Linked Issues or Issue Description

**What happened?**

A restart-chaos wave (three release swaps) showed a Retry banner on one
task for 141 to 144 seconds. The reason code was
`issue_execution_lock_changed`. Nobody clicked Retry, and the task
completed.

Timeline (UTC):

- 14:32:09. Run A starts.
- 14:51:12. A user follow-up wakes run B. Run B is queued behind run A.
- 14:52:46. Run A succeeds. The system schedules continuation run C
(`retryOfRunId` = A, reason `native_agent_blocker_continuation`). Run B
starts and takes the task execution lock.
- 14:53:16. The dispatch gate cancels C before it starts with
`issue_execution_lock_changed`, because run B owns the lock. This is
correct de-duplication.
- 14:53:17. The banner shows `failed` with Retry. Run C was created
after run B, so C is the newest run. The projection reads only the
newest run and ignores that run B is running and owns the lock.
- 14:55:41. Run B succeeds and the task is done. The banner goes away.

Retry was never needed. Run B owned the task for the full banner window.

**Expected behavior**

The banner does not offer Retry while a run owns the task and is queued
or running. A run that the system cancelled before it started because
another run took the lock is not a failed attempt. When Retry is shown,
pressing it is accepted for the run that the banner shows.

**Steps to reproduce**

1. Create a task. Run A succeeds and leaves the task in progress.
2. Queue run B for the same task (for example, a user comment). Create
continuation run C after run B.
3. Start run B and set it as the task execution lock owner.
4. Cancel run C before it starts with error code
`issue_execution_lock_changed`.
5. Read the task interruption. Before this change: `state: failed`,
`canRetry: true`, observed run C. After this change: `state: working`,
`canRetry: false`, observed run B.

**Paperclip version or commit**

`fe0c4aa95d` (master at the time of the wave).

## What Changed

- `server/src/services/task-retry-eligibility.ts`: add two shared
conditions.
- `taskAttemptCondition()`: a run cancelled before it started with
`recovery_no_longer_needed`, `issue_closed_before_start` or
`issue_execution_lock_changed` is not an attempt. The first two codes
were already excluded inline in the projection.
`issue_execution_lock_changed` is new.
- `liveExecutionOwnerCondition()`: the run is the task's execution lock
owner and it is queued or running.
- `server/src/services/task-interruption-presentation.ts`: the observed
run is the live execution owner when there is one. Otherwise it is the
newest attempt, with the same native, first-class and legacy precedence
as before. The owner lookup is a primary-key read, so the per-task run
scan does not grow.
- `server/src/services/task-retry.ts`: Retry admission, the refused chat
Retry continuation and the superseded-Retry check use the same attempt
filter. Admission also accepts the live execution owner when the request
names it. The newest attempt stays acceptable, so the automatic Retry
hooks (restart, provider limit, budget resume) keep their behavior.
- `heldTaskRetryRefusals` uses the same attempt filter.
- New test
`server/src/__tests__/task-retry-banner-execution-owner.test.ts`: one
state table. Each row asserts the list and detail projection, and when
Retry is shown, presses it through `requestTaskRetry`.

### State table (tested as one table)

| Row | Runs and lock | Banner | Observed run | Retry press |
| --- | --- | --- | --- | --- |
| 1 | Continuation scheduled; follow-up running and owns the lock |
working, no Retry | follow-up | n/a |
| 2 | Continuation cancelled unstarted (`issue_execution_lock_changed`);
follow-up running and owns the lock (t032) | working, no Retry |
follow-up | n/a |
| 3 | Same as row 2, lock pointer cleared mid-transfer | working, no
Retry | follow-up | n/a |
| 4 | Scheduled continuation overdue past the 2-minute dispatch grace;
follow-up running and owns the lock | working, no Retry | follow-up |
n/a |
| 5 | Newer attempt failed; older queued run owns the lock | working, no
Retry | queued owner | automatic Retry of the failed run is still
accepted |
| 6 | Continuation superseded; follow-up then succeeded | resolved, no
Retry | follow-up | n/a |
| 7 | Continuation superseded; follow-up then failed | failed, Retry |
follow-up | accepted |
| 8 | Continuation superseded; predecessor failed; nothing else runs |
failed, Retry | predecessor | accepted |
| 9 | Continuation started, then lost the lock | failed, Retry |
continuation | accepted |
| 10 | Lock pointer left on a finished failed run | failed, Retry | that
run | accepted |
| 11 | Row 2 with native runs (`nativeIssueId`) | working, no Retry |
native follow-up | Retry naming the owner is accepted |
| 12 | Queued owner past its 15-minute move-forward wait; newer failed
attempt | working, Retry (`move_forward`) | queued owner | accepted |

Invariants that the table checks:

- Retry is never shown while something will advance the task (rows 1 to
6, 11). Row 12 offers only `move_forward` on the owner.
- A Retry press that names the live owner is accepted (rows 1, 2, 3, 5,
11, 12).
- Retry is always shown when nothing will advance the task (rows 7 to
10).
- The shared attempt filter is tested in `heldTaskRetryRefusals`,
`retireSupersededTaskRetry` and the refused chat Retry continuation
(`person_with_superseded_continuation` in
`chat-channels.integration.test.ts`).
- When Retry is shown, pressing it is accepted for the observed run
(rows 7 to 10).

## Verification

- Red: with the service changes removed, rows 1 to 8 fail (8 of 10).
Rows 9 and 10 pass on both sides.
- Green: `cd server && npx vitest run
src/__tests__/task-retry-banner-execution-owner.test.ts` passes 14 of
14.
- Related suites pass: `task-interruption-query.test.ts` (includes the
query-plan bound test), `task-interruption-automatic-path.test.ts`,
`legacy-continuation-authority.test.ts`,
`native-closed-session-retry.integration.test.ts`,
`interrupted-preparation-restore-retry.test.ts`, and the other server
tests that call `taskInterruptionPresentation`, `requestTaskRetry` or
`heldTaskRetryRefusals`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The change is read-side only. It does not change scheduling,
dispatch, the lock, or the run-dispatch gate.
- A run cancelled before start with `issue_execution_lock_changed` no
longer shows as a failed attempt. If the lock was released and nothing
else ran, the banner shows the previous attempt. A failed previous
attempt still offers Retry (row 8).
- The projection query has one more union branch. It is a primary-key
lookup through `issues.execution_run_id`. The existing query-plan bound
test passes.
- No runner, migration or API contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Refs #22306 (the earlier fix for a due scheduled retry shown as failed).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 10:32:41 -06:00
875738a8d4 fix(tool-access): keep provider-side OAuth registration failures out of Sentry (#22416)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People connect remote MCP apps through OAuth. Paperclip registers
its own OAuth client with the provider through dynamic client
registration (RFC 7591).
> - When the provider's registration endpoint answers with an error,
Paperclip returns a 502 to the person. The error handler also sends
every 5xx `HttpError` to Sentry.
> - A provider that refuses or fails registration (401, 403, 404, 429 or
5xx) is a provider-side failure. It is not a Paperclip server fault, so
it adds noise to Sentry.
> - MCP discovery already marks the same provider statuses as expected
remote failures, and the error handler does not report them.
> - This pull request applies the same marking to the registration
response. It also adds the provider status and the allowlisted error
label to registration failures that Paperclip still reports.
> - The benefit is that Sentry shows only registration failures that can
be Paperclip faults, and each one tells which status the provider sent.

## Linked Issues or Issue Description

**What happened?**
A hosted server reported `Error: OAuth dynamic client registration
failed` three times in two minutes. The stack is `startOAuth` ->
`registerOAuthClient` -> `requestOAuthClientRegistration`. The message
is the generic fallback, so the provider sent no recognized RFC 7591
`error` code. The event did not include the provider's HTTP status.
Thus, the event did not show if the provider refused the request or if
Paperclip sent bad metadata.

**Expected behavior**
The person gets the same 502 and code
`oauth_dynamic_client_registration_failed`. Sentry does not get a crash
report when the provider refuses or fails the request on its side (401,
403, 404, 429, 5xx). Sentry continues to get a report for other
statuses, such as a 400 that rejects Paperclip's metadata. That report
includes the status, so the cause is visible.

**Steps to reproduce**
1. Point an OAuth connection at an authorization server whose
registration endpoint returns 403 (closed or token-gated registration),
404, 429 or 503.
2. Start the OAuth connection.
3. See the 502 response and one Sentry crash report.

**Paperclip version or commit**
`master` at fe0c4aa95d.

No related open PR covers this reporting path. #20433 changes which
client method Paperclip accepts after a successful registration. It does
not change failure reporting.

## What Changed

- `server/src/services/tool-access/oauth/client.ts`: A non-OK
registration response now goes through `markMcpHttpResponseFailure`.
This is the same classifier that MCP discovery uses. 401, 403, 404, 429
and 5xx become expected remote failures, so `errorHandler` does not
report them. The HTTP response does not change.
-
`server/src/services/tool-access/oauth/provider-failure-diagnostics.ts`
(new): A server-only `WeakMap` record keeps the operation, the HTTP
status and the already-normalized provider error label. It does not keep
the provider URL, body, description or headers.
- `server/src/sentry.ts`: `captureException` adds `oauth_operation`,
`oauth_provider_status` and the optional `oauth_provider_error` tags and
an `oauth_provider_failure` context for an error that has this record.
Grouping does not change (`{{ default }}`).
- Tests: `oauth-dcr-failure-reporting.test.ts` sends each status through
the real `errorHandler`. `sentry-oauth-provider-failure.test.ts` checks
the exact Sentry tags and context.

## Verification

- `cd server && npx vitest run
src/__tests__/oauth-dcr-failure-reporting.test.ts
src/__tests__/sentry-oauth-provider-failure.test.ts
src/__tests__/sentry.test.ts src/__tests__/remote-http-rebinding.test.ts
src/__tests__/openrouter-catalog-error-handler.test.ts`: all pass.
- `cd server && npx vitest run
src/__tests__/generic-mcp-connection.test.ts
src/__tests__/tool-access-client-credentials.test.ts
src/__tests__/error-handler.test.ts`: 154 tests pass. This includes the
existing hostile-provider DCR redaction test.
- Red check: without the `client.ts` change, the six provider-side
status cases and the 400 diagnostics case fail. Without the `sentry.ts`
change, the Sentry tag test fails.
- Local `tsc` was stopped by the sandbox (SIGKILL), also for a scoped
config. CI typecheck must cover this.

## Risks

- Low risk. The HTTP status, body and code for the person do not change.
- Sentry stops reporting registration failures with 401, 403, 404, 429
and 5xx status. A provider outage during registration is then visible
only in the response and logs. This is the same policy that MCP
discovery already uses.
- The new tags contain only an integer status and an allowlisted label.
No provider text goes to Sentry.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edit and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
2026-11-20 07:52:23 -08:00
92271da899 fix(native): keep a chat turn's message batch when a fresh-session retry continues it (#22413)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - External chat turns (GitHub, Slack, Discord) wake an agent on a task
with an admitted message batch.
> - When a task is in review, dispatch checks that the run is bound to
that reviewed chat batch. A run without a batch fails with
`reviewed_chat_execution_binding_not_authorized` (`context_mismatch`).
> - Automatic fresh-session retries (tool-receipt limit, provider
capacity, environment blocker, agent blocker, and safe replacement)
remove all consumed wake input from the failed run's context. This also
removes the chat message IDs, but the context keeps `source:
chat:<provider>`.
> - Thus every automatic retry of a reviewed chat turn fails at setup.
The one-shot review-path recovery that follows copies the same empty
context and fails the same way.
> - This pull request keeps the admitted chat message batch on those
retries and removes the failed run's dispatch proofs, so dispatch checks
ownership and current access again.
> - The benefit is that a reviewed chat turn can continue after a
transient failure, and access checks still apply.

## Linked Issues or Issue Description

Refs #13818 (the same approach for review-path recovery). Related:
#20689, #20697, #20842 (other causes of the same denial).

**What happened?**
A GitHub PR review turn on a task in review stopped on a transient
failure (tool-receipt limit or sandbox environment blocker). Paperclip
scheduled an automatic fresh-session retry. The retry failed at setup in
less than one second with
`reviewed_chat_execution_binding_not_authorized` and denial reason
`context_mismatch`. The review-path recovery wake then failed with the
same error. On staging this occurred 5 times on 3 tasks after #20689,
#20697 and #20842 were deployed.

**Expected behavior**
An automatic retry of a chat turn keeps the admitted message batch.
Dispatch then checks the retry against current task ownership and
current endpoint, conversation and principal access. If access is still
valid, the retry runs. If access was revoked, dispatch refuses the
retry.

**Steps to reproduce**
1. Put a task in review with a run-bound external chat turn (for example
`source: chat:github` with `wakeCommentIds`).
2. Fail that native run with a fresh-session retry lane (for example
`native_environment_blocker`).
3. Let dispatch claim the scheduled retry. The retry context has
`source: chat:github` but no `wakeCommentIds`.
4. Dispatch refuses the retry with `context_mismatch`.

**Paperclip version or commit**
`master` at 414abc2dbf.

## What Changed

- Add
`server/src/services/native-runtime/fresh-session-retry-context.ts`. It
holds the one list of consumed wake input keys and builds the retry
context. For a run-bound external chat turn, it keeps the admitted
message IDs (`wakeCommentIds`, `wakeCommentId`, `commentId`). It removes
the dispatch proofs that the failed run got
(`paperclipHarnessCheckedOut`, `paperclipExternalChatExecutionBound`).
Other runs keep the existing behavior.
- Use the helper in `materializeNativeFreshSessionRetry` (all its lanes)
and in `reconcileSafeNativeReplacements`. Before, each producer had its
own copy of the key list.
- Add a unit test for the helper.
- Add a regression test in
`heartbeat-reviewed-chat-binding.integration.test.ts` for Slack and
Discord. It checks that the retry passes the reviewed chat check, that
revoked access still refuses the retry, and that review-path recovery
keeps the batch.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts
src/services/native-runtime/fresh-session-retry-context.test.ts
src/services/native-runtime/native-safe-replacement.test.ts
src/services/native-runtime/native-agent-blocker.integration.test.ts
src/services/native-runtime/native-tool-receipt-limit.integration.test.ts`
passes: 5 files, 127 tests.
- Red check: with the `master` versions of
`native-provider-capacity-retry.ts` and `native-safe-replacement.ts`,
the new regression test fails (2 failed, 34 passed). The retry is
refused with `context_mismatch`.
- CI runs the full typecheck. The local full server typecheck stopped
because the sandbox ran out of memory.

## Risks

- Low risk. The change applies only to automatic fresh-session retries
whose context has a run-bound external chat source and admitted message
IDs.
- The retry does not get the failed run's authority. Dispatch must check
ownership and access again, and the regression test shows that a revoked
principal is still refused.
- Other retries keep the same context as before, because both producers
now use the same key list.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 06:47:34 -08:00
a097949864 fix(server): stop issue reference syncs from deadlocking on mutual mentions (#22389)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server keeps a table of issue reference mentions. It rebuilds
the rows for an issue when its title or description changes.
> - `syncIssue` locks the source issue row `FOR UPDATE` before it
rebuilds the rows (added in #13898).
> - Each mention insert does a foreign-key check. That check takes a
`FOR KEY SHARE` lock on the target issue. `FOR KEY SHARE` conflicts with
`FOR UPDATE`.
> - When issue A mentions B and issue B mentions A, two concurrent syncs
each hold one `FOR UPDATE` lock and wait for the other. Postgres reports
`deadlock detected`, and one update fails. We saw this on staging.
> - This pull request changes the lock to `FOR NO KEY UPDATE`.
> - The benefit is that syncs of one issue stay serialized, and
foreign-key checks from other issues do not wait on that lock.

## Linked Issues or Issue Description

Refs #13898

**What happened?**
A staging server reported `Failed query: insert into
"issue_reference_mentions" ...` caused by `PostgresError: deadlock
detected`. The insert came from `replaceSourceMentions` inside the
`syncIssue` transaction. The source issue description mentioned the
target issue.

**Steps to reproduce**
1. Create issue A with a description that mentions issue B.
2. Create issue B with a description that mentions issue A.
3. Run `syncIssue(A)` and `syncIssue(B)` at the same time, several
times.
4. One sync fails with `deadlock detected`.

**Expected behavior**
Both syncs complete, and each issue has its mention rows.

## What Changed

- `server/src/services/issue-references.ts`: `syncIssue` now locks the
source issue with `FOR NO KEY UPDATE`, not `FOR UPDATE`. `FOR NO KEY
UPDATE` still conflicts with itself, so concurrent syncs of one issue
stay serialized (the #13898 behavior). It does not conflict with the
`FOR KEY SHARE` lock of a foreign-key check, so the lock cycle cannot
form.
- `server/src/__tests__/issue-references-service.test.ts`: two
regression tests.
- One test holds a sync lock in an open transaction. It then syncs a
second issue that mentions the locked issue, and expects that sync to
complete without a wait.
- One test runs 20 concurrent syncs of two issues that mention each
other, and expects all four mention rows.

## Verification

- `cd server && npx vitest run
src/__tests__/issue-references-service.test.ts`
- Before the fix (old lock): the two new tests fail. One test shows
`expected 'blocked' to be 'completed'`. The other test fails with
`PostgresError: deadlock detected` on `select ... from "issues" ... for
update`. This is the same error as the staging event.
- After the fix: 5/5 tests pass. I ran the file three times and all runs
passed. The existing test "serializes concurrent syncs for a description
with repeated references" still passes.

## Risks

- Low risk. There is no schema change or migration. The lock mode
changes only for this read. `UPDATE` statements that do not change key
columns already take `FOR NO KEY UPDATE`, so this lock blocks the same
issue writers as before. Only foreign-key `KEY SHARE` checks can now
continue at the same time.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in a Claude Code agent session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 06:17:00 -08:00
97737c4f7f fix(workspaces): give a failed saved-workspace restore a stable error code (#22419)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a run starts, Paperclip can reopen a task's archived
execution workspace and restore its saved Git commit
> - When the saved restoration record no longer matches its workspace,
or a restore step fails, the code throws `GitCacheRestorationFailure`
> - That error records which step failed and what kind of failure it
was, but it has no `code`
> - Run-failure reporting groups failures by error code, never by
message. A setup failure without a code falls back to its class name,
and this class reports its name as plain `Error`. So these failures
share one Sentry group with every other uncoded setup failure
> - On staging this one failure makes up almost all of that catch-all
group: hundreds of events from automatic restarts of tasks whose saved
workspace record can no longer be restored
> - This pull request gives the error a stable code
> - The benefit is a separate, correctly named Sentry group for restore
failures, and run records whose `failureCause` names the restore failure
instead of `Error`

## Linked Issues or Issue Description

**What happened?**
On staging, runs keep failing during setup with "The saved workspace
commit could not be restored. Check repository access and retry." The
stack is `selectHeartbeatEnvironment` → `readGitCacheExpiration` →
`readGitCacheRestoration`. The stored restoration record of an archived
workspace fails validation, and the run fails as `setup_failed`. The
error has no code, so the Sentry fingerprint uses `exception:Error` and
puts it in the same group as unrelated setup failures.
`resultJson.failureCause` for these runs is the generic `Error`.

**Expected behavior**
A failed saved-workspace restore has a stable error code. Sentry groups
it on its own, and the run's failure cause names it.

**Related**
Refs #19736, which added the safe `step` and `kind` restore diagnostics
this change keeps. #22093 took the same approach for the
execution-target Git context probe. I found no open PR for this error.

**Steps to reproduce**
Construct `new GitCacheRestorationFailure()` (or call
`readGitCacheRestoration` with a record that does not match its
workspace). On current master the error has no `code`,
`failureCauseFromError` returns `Error`, and the run-failure diagnostics
record no root code.

## What Changed

- `server/src/services/git-workspace-restoration.ts`: add
`GIT_CACHE_RESTORATION_FAILED_CODE` (`git_cache_restoration_failed`) and
set it as `code` on every `GitCacheRestorationFailure`.
- The message, `step` and `kind` do not change.
`classifyWorkspaceRestorationError` still checks the class first, so
restore classification does not change.
- `server/src/__tests__/git-workspace-restoration-diagnostic.test.ts`:
add a test that the error carries the code, with and without a step
diagnostic, that `failureCauseFromError` returns it (also when the error
is a `cause`), and that `collectRunFailureDiagnostics` records it as the
root exception code. That root code is what the Sentry fingerprint uses.

## Verification

- Red/green: the new test fails on unmodified master and passes with the
fix.
- `vitest run` on `git-workspace-restoration-diagnostic`,
`git-expiration-auth-fallback` and `run-failure-sentry`: 63 tests pass.
- I also ran the other suites that use this error or its readers locally
(archived nested repository, fetch head, reproducible managed checkouts,
base storage, repository object cache, remote workspace migration, Git
workspace expiration, heartbeat environment selection, closed-workspace
routes). In this sandbox many `remote-workspace-migration` tests stopped
at about 10 seconds each. The migration code doesn't read this error's
`code` (it only compares a database lock code and a fixed supersession
code), so I expect the slow sandbox to be the cause. CI's full run is
the authoritative result.
- I couldn't run the full server typecheck locally because the sandbox
ran out of memory. The change only adds a new `code` field, which
`Error` does not define. CI's Typecheck job covers it.

## Risks

- Low risk. Only an added `code` property on one error class. No control
flow changes.
- `resultJson.failureCause` for these runs becomes
`git_cache_restoration_failed` instead of `Error`. The only code that
compares `failureCause` checks a different prefix
(`remote_workspace_sandbox_disk_full:`).
- Every setup-failure classifier in `run-completion` matches a specific
error class or HTTP error, not any coded error, so the run's `errorCode`
stays `setup_failed` and retry behavior does not change.
- Not in scope: automatic fresh continuations still retry this
deterministic failure. Stopping that needs a recovery-policy change in
`fresh-continuation-fallback`, which decides on `run.errorCode`. That's
a separate change for the recovery owners.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, as the Paperclip
Sentry triage agent.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted suites; see
Verification for the slow migration suite)
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documentation describes this error's fields)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 06:13:08 -08:00
a792409d07 fix(tools): retry a named MCP gateway create that loses a deadlock (#22408)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs prepare a run-scoped MCP gateway for their agent before
the provider starts. `createNamedGateway` writes the `tool_mcp_gateways`
row and its profile binding in one transaction.
> - That transaction locks the agent row first (`FOR NO KEY UPDATE`).
Then its gateway insert takes a key-share lock on the company and
profile rows for the foreign keys.
> - Another writer can lock the company row first and then the agent
row. The two transactions then wait for each other. Postgres detects a
deadlock and aborts the gateway create (`40P01`).
> - The run then fails before the provider starts. The task stops on
`native_continuation_requires_reconciliation`, so a person must
reconcile a run that did no work.
> - This pull request runs the gateway create transaction through the
shared bounded retry that #22329 and #20971 use for the same type of
lock cycle.
> - The benefit is that a short lock conflict no longer fails the run
preparation.

## Linked Issues or Issue Description

**What happened?**
A native run failed while it prepared its MCP gateway. The error was
`Failed query: insert into "tool_mcp_gateways" (...)`. The Postgres
cause was `deadlock detected` (`40P01`). The task then stopped and
needed reconciliation.

**Expected behavior**
A transient lock conflict does not fail the run preparation. The gateway
create writes the gateway and its profile binding, and the run
continues.

**Steps to reproduce**
1. Transaction A: lock the company row `FOR UPDATE`.
2. Call `createNamedGateway` for an agent in that company. It locks the
agent row and then waits for the company key-share lock.
3. Transaction A: lock the agent row `FOR UPDATE`.
4. Postgres reports a deadlock. Before this change the gateway create is
the victim and `createNamedGateway` rejects with the database error.

**Additional context**
The new test reproduces this with two real transactions on embedded
Postgres. Transaction A gets a longer `deadlock_timeout`, so the gateway
create is always the victim, as in the observed failure. The failing
statement in the test is the same insert as in the observed error.

Refs #22329
Refs #20971

## What Changed

- `server/src/services/tool-gateway.ts`: `createNamedGateway` runs its
transaction through `retryTransientLockContention`. The helper retries
only lock timeout, deadlock and serialization failure (`55P03`, `40P01`,
`40001`): 5 attempts, with 100 ms × attempt backoff. Other errors fail
as before.
- `server/src/__tests__/tool-gateway-named-gateway-deadlock.test.ts`:
new test with a real two-transaction deadlock between a
company-then-agent locker and a real `createNamedGateway` call. The test
checks that exactly one gateway row and one profile binding exist after
the retry.

## Verification

- Failing before the fix: `cd server && npx vitest run
src/__tests__/tool-gateway-named-gateway-deadlock.test.ts` failed.
`createNamedGateway` rejected with `Failed query: insert into
"tool_mcp_gateways" ...`, cause `deadlock detected`, code `40P01`.
- Passing after the fix: the same test passes. I ran it 3 times.
- `heartbeat-runtime-mcp-servers.test.ts`,
`agent-runtime-assignments.test.ts`,
`agent-interaction-permissions.test.ts` and
`tool-gateway-service.test.ts` pass (4 files, 238 tests).
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. Only the gateway create transaction retries, and only on the
three contention codes.
- Postgres rolls back the complete aborted transaction. A retry
therefore cannot write a duplicate gateway or profile binding.
- The interaction-permission check runs again in each attempt. A retry
cannot skip it.
- If a concurrent create with the same slug commits first, the retry
gets a unique violation. That error is not retried. The existing native
caller then reads the committed gateway by slug, as before.
- The worst added delay is about 1 s of backoff across 5 attempts, and
only when the lock conflict continues.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution on embedded
Postgres).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 05:47:47 -08:00
fa96716bb5 fix(storage): back off a throttled small upload that has no source file (#22404)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server saves many objects to S3-compatible storage: skill
packages, attachments, recovery archives and manifests
> - Object storage answers a burst with `503 SlowDown`. Most large and
file-backed uploads already back off (#20165, #20313, #20772, #21010)
> - A small upload with no source file still had almost no retry. The S3
SDK never resends a stream body, so a stream got one attempt. A Buffer
got the SDK's three closely spaced attempts
> - Hosted production and staging reported `SlowDown` on `put_object`
from request handlers, for example right after a skill import fetched
its files. The error continues on the current production release
> - This pull request holds a small stream's bytes within the existing
buffered-payload budget, and backs off a throttled small upload like a
file upload
> - The benefit is that a short request-rate limit no longer fails a
small save

## Linked Issues or Issue Description

Refs #20165, #20313, #20772, #21010 (the same backoff for other upload
paths).

**What happened?**

A request handler saved a small object with `storage.putFile` and a
stream body (for example a skill package archive), or with a small
Buffer. Object storage answered `503 SlowDown` ("Please reduce your
request rate."). The stream was not sent again. The Buffer was sent
three times within about a second. The request failed with a 500.

**Expected behavior**

A request-rate limit slows the upload down. It does not fail the save.

**Steps to reproduce**

1. Point the S3 provider at an endpoint that answers the first `PUT`
with `503 SlowDown`.
2. Call `putFile` with a small `Readable` body and no source file.
3. On `master` it throws `SlowDown: Please reduce your request rate.`
after one `PUT`.

**Paperclip version or commit**

`master` at `201abacd6e`.

## What Changed

- `server/src/storage/s3-multipart.ts`: `withBufferedUploadBody` holds a
small stream body within the shared buffered-payload budget that
multipart already uses. It checks the declared length.
`sendThrottledRequestWithRetry` uses the same jittered exponential
backoff as file uploads. It resends only request-rate limits (SlowDown,
503, 429).
- `server/src/storage/s3-provider.ts`: the small `putObject` path
without a source file sends Buffer bodies and buffered stream bodies
through that retry. When the budget is full, the stream is sent once, as
before.
- `server/src/__tests__/storage-s3-small-upload-throttle.test.ts` (new):
a real S3 client against a local endpoint.

## Verification

- `cd server && npx vitest run
src/__tests__/storage-s3-small-upload-throttle.test.ts`: with the
`master` source files, 3 of 4 cases fail (the stream and Buffer cases
reject with `SlowDown`). With this change, 4 of 4 pass.
- `cd server && npx vitest run src/__tests__/storage-* src/storage`: 17
files, 135 tests pass on top of `201abacd6e`, including `does not repeat
a write that timed out`.
- I did not run the full server typecheck locally. The sandbox ran out
of memory. CI must cover it.

## Risks

- Low. A small stream is now held in memory, at most 16 MiB, inside the
existing 32 MiB process budget. If the budget is full, behavior is
unchanged.
- Only request-rate limits are resent. The store did not apply those
writes. Other write failures keep their SDK handling, so a timed-out
write is still not repeated.
- A stream that ends early or runs long now fails before any request is
sent.
- Conditional publication puts (`putObjectConditional`) are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with
tool use (shell, file edit, test execution) and read-only Sentry reads.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
doc change is necessary: this is internal storage behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
2026-11-20 05:37:17 -08:00
16dfa200ac fix(environments): clean up an unconfirmed setup sandbox and answer a provider outage with 503 (#22377)
A provider outage during an interactive custom-image setup start left a
provisional sandbox with no cleanup owner, and the raw RPC error reached
the HTTP error handler as a 500 crash report.

- Keep the creation-cleanup envelope and create status for
  environmentStartInteractiveSetup errors at the worker RPC boundary.
- Record an in-scope unconfirmed setup sandbox as a pending-cleanup lease
  that the existing sweep destroys.
- Pin failed-creation cleanup (probe and teardown) to the plugin that
  created the sandbox.
- Answer a transient provider refusal with 503
  environment_provider_unavailable instead of a crash report.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 04:57:36 -08:00
7485e01161 fix(heartbeat): keep a credential denial an owner action when the lease lapses during AI selection (#22379)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat prepares each run before the agent starts. One
preparation step selects the AI account for the run.
> - AI selection can reject an account for a reason that only its owner
can fix. An example is a credential that is not shared with the
responsible user.
> - Run setup gives a lost controller lease priority over every other
failure code. When the lease lapsed during the same setup, the run got
`controller_lease_lost` instead of `configuration_incomplete`.
> - `controller_lease_lost` is retryable. So the run retried and failed
again with the same rejection. The code also bypassed the Sentry filter
for owned AI selection blockers, so the server sent a spurious Sentry
event.
> - This pull request keeps `configuration_incomplete` when AI selection
itself rejected the account, even when the lease is lost at the same
time.
> - The benefit is that the owner gets the correct action at once, with
no wasted retry and no Sentry noise.

## Linked Issues or Issue Description

Refs #15714 (this PR adds the Sentry filter for owned AI selection
blockers, and this change reuses that check).

**What happened?**
A legacy run was in setup. AI selection correctly rejected the account
(`ai_connection_credential_not_shared`). At the same time, a lease
renewal stalled for about 29 seconds and the controller lease lapsed.
Setup recorded the run as `controller_lease_lost` with the configuration
message. The run was retried. The retry failed again as
`configuration_incomplete`. The first failure also went to Sentry,
because the filter only matches `configuration_incomplete`.

**Expected behavior**
When AI selection rejects the account for an owned reason, the run fails
as `configuration_incomplete`. The run is not retried. The board gets
the configuration action. No Sentry event is sent. The run result keeps
the lease-loss evidence.

**Steps to reproduce**
1. Start a legacy run whose AI account selection fails with an owned
rejection (for example, a credential that is not shared with the
responsible user).
2. Make the controller lease lapse while setup selects the AI account.
3. Look at the run `errorCode`, the retry schedule, and the Sentry
report.

**Paperclip version or commit**
`master` at c41145e72a.

**Agent adapter(s) involved**
Not adapter-specific (core bug).

## What Changed

- Added `isOwnedAiConnectionSelectionRejection` in
`server/src/lib/ai-connection-configuration-failure.ts`. It matches a
`configurationIncomplete` result with reason `ai_connection_unavailable`
and a known `selectionFailure`.
- `run-completion.ts` setup failure: a lost lease no longer replaces the
error code when AI selection itself rejected the account. Any other AI
preparation error (the broad wrapper) still records a retryable
`controller_lease_lost`.
- `run-failure-report.ts`: the Sentry filter now uses the same shared
check. Its behavior does not change.
- Added tests:
- A unit test in `run-completion.test.ts` for both cases (owned
rejection keeps its code; unclassified error keeps
`controller_lease_lost`).
- A full heartbeat test in `heartbeat-process-recovery.test.ts`. It
checks the error code, no retry, the board configuration action, and no
Sentry report.
  - A test in `run-failure-report.test.ts` for the shared check.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/services/__tests__/run-failure-report.test.ts`
- `cd server && npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "credential sharing
denial"`
- Without the `run-completion.ts` change, both new regression tests fail
with `controller_lease_lost`. With the change, they pass.
- In the author's sandbox, 4 sandbox-provider lease tests in
`run-completion.test.ts` and 4 provider-outage tests in
`heartbeat-process-recovery.test.ts` fail. They fail the same way on
untouched `master` in that sandbox, so the cause is the environment. CI
runs the full suite.

## Risks

- Low risk. The change applies only when AI selection recorded an owned
rejection. A lost lease with any other setup error keeps the same
retryable code as before.
- The run is no longer retried in this case. That is correct, because
every host repeats the same rejection until the owner acts.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 04:29:05 -08:00
d60cecf625 fix(server): show a retry message when managed app sign-in is rate-limited (#22309)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People connect apps such as GitHub through the Paperclip Cloud
connector
> - The connector `session` call can return 429 with the reason
`RATE_LIMITED`
> - The OAuth start path does not catch this error, so the server
answers 500
> - A 500 tells the user that Paperclip is broken. It does not tell the
user to try again later. The error also goes to error reporting as a
server crash
> - This pull request changes a connector rate limit into a 429 with a
clear retry message
> - The benefit is that users see "try again in a moment" and the server
does not report a false crash

## Linked Issues or Issue Description

**What happened?**
A user on a Cloud beta workspace started GitHub connect two times. The
Paperclip Cloud connector rate-limited the `session` call (status 429,
reason `RATE_LIMITED`). Both times, the UI showed an internal server
error.

**Steps to reproduce**
1. Use a Cloud connector where the `session` call returns 429
`RATE_LIMITED`.
2. POST `/api/companies/:companyId/tools/apps/connect` with `galleryKey:
"github"` and `connectionMethodKey: "managed"`.
3. Look at the response status.

**Expected behavior**
The server answers 429 with a readable message and retry guidance.

**Actual behavior**
The server answers 500 with "Internal server error".

No public GitHub issue exists for this.

## What Changed

- `server/src/services/tool-access/oauth/authorization-start.ts`: the
managed connector branch catches errors from `startAuthorization`. A
`PaperclipCloudConnectorError` with status 429 or broker reason
`RATE_LIMITED` becomes an `HttpError` 429. The message is "`<App>`
sign-in is busy right now. Try again in a moment.", the code is
`paperclip_cloud_connector_rate_limited`, and
`details.retryAfterSeconds` comes from the broker `Retry-After` value
when the broker sends one.
- All other errors pass through with no change.
- `server/src/__tests__/tool-access-service.test.ts`: a new route test
for a rate-limited managed GitHub connect. It checks the 429 status, the
message, the code, `retryAfterSeconds`, and that the raw broker reason
is not in the response.

## Verification

- Reproduced first: before the fix, the new test failed with `expected
500 to be 429`.
- `pnpm exec vitest run src/__tests__/tool-access-service.test.ts` in
`server/`: 514 tests pass.
- `pnpm typecheck` in `server/`: no errors in `server/src`. The local
run reported errors only in sibling packages that the limited local
install did not include. CI runs the full check.
- The UI uses the `error` string of the response body as the message
(`ui/src/api/errors.ts` `apiErrorMessage`), so the user sees the new
text.

## Risks

- Low risk. The change applies only to the managed connector sign-in
start, and only when the connector reports a rate limit.
- The response does not set a `Retry-After` header. The retry delay is
in `details.retryAfterSeconds`, the same as other 429 responses that the
service layer makes.
- This change does not change the broker rate limit. A separate
investigation must find out why the broker rate-limits the `session`
call.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
and shell code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: paperclipbugbot <bots+paperclipbugbot@paperclip.ing>
2026-11-20 03:57:19 -08:00
DottaandClaude Opus 5.5 fe0c4aa95d fix(recovery): settle a late workspace hold when a remote-epoch continuation already took over (#22387)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Restart recovery must continue a task after a server is killed. A
person must never need to press Retry.
> - When Paperclip cannot resume a cut-off run exactly, it records a
"fresh continuation". For a remote workspace, the continuation abandons
the old run's workspace epoch and restores the last save into a new
root. It does not apply the old run's unsaved files.
> - Workspace recovery can try to save the old run's files after that.
The epoch is already abandoned, so the source is "unavailable". Recovery
then wrote an active hold that asked a person to press Retry to continue
in a new workspace, which the continuation had already arranged.
> - That hold made the dispatch gate withhold every queued run of the
task, and the task rested on a Retry banner.
> - This pull request settles that hold into the recorded continuation
at the moment workspace recovery would leave it exhausted. Dispatch and
continuation delivery are unchanged.
> - The benefit is that the queued continuation starts automatically
after a hard kill, with no Retry banner.

## Linked Issues or Issue Description

**What happened?**

A restart-chaos release-swap wave hard-killed the old server. One task
(of 6) then rested on a Retry banner for about 5 minutes. The other 5
tasks passed. The sequence for the failed task:

1. A run started on the incoming server. The wave's injected database
delay held its process-identity write for 480 s, so its runner attach
timed out. The run saved a same-run resume.
2. Restart recovery found a newer queued run for the task (a
child-completion wake). It treated the same-run resume as superseded,
and the run ended `failed` with `native_restart_recovery_blocked`.
3. The dispatch gate withheld the queued run
(`execution_reconciliation_required`). In the same transaction, it
recorded a fresh continuation with `remote_epoch` isolation, which
abandoned the old run's remote workspace epoch.
4. 4 s later, workspace preservation for the old run found the abandoned
epoch (`source_unavailable`, `use_alternative_strategy`). It left an
active hold: "Paperclip cannot recover the unsaved files from the
previous run ... Retry continues the task in a new workspace". The
retained-source fallback also refused it later, with
`workspace_authority_changed`, because the continuation owned the epoch.
5. The gate withheld both bounded retries because of that hold. The
retries were spent, and the task showed a Retry banner.
6. A person pressed Retry. Retry started a run in a new workspace, which
the recorded continuation had already decided to do.

**Expected behavior**

The queued continuation starts automatically. No Retry banner shows.

**Steps to reproduce**

1. Run the restart-chaos K1 kill variant: 2 release swaps, and the old
server is hard-killed 120 s after SIGTERM.
2. Use an OpenCode task on a remote sandbox whose run starts just before
the second swap and that gets a child-completion wake while it runs.
3. Watch the task after the kill. Workspace recovery writes the
unsaved-files hold, the gate cancels each queued run with
`execution_reconciliation_required`, and the task rests on a Retry
banner.

`server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts`
reproduces the database state at step 5.

**Paperclip version or commit**

`master` at 25b5cad7f9 (wave) and c41145e72a (base of this branch).

## What Changed

- `server/src/services/fresh-continuation-fallback.ts`: new
`settleHoldIntoAbandonedEpochContinuation()`. It settles one
workspace-recovery hold with the existing `settleLateHolds()` when all
of these are true:
- the hold is active or escalated, and its saved-work recovery is
exhausted;
- no person owns it, it has no human-gate evidence, and it is not an
unsafe-archive hold;
- a fresh-continuation record for the same run has `remote_epoch`
isolation and delivery `pending`, `delivered`, `delegated` or
`not_requested`;
- durable proof shows the run's prepared epoch was abandoned
(`isAbandonedPreparedRemoteWorkspace`: an abandoned-epoch receipt for
this run, epoch and location, and an authority that moved past it).
The hold becomes `resolved` / `restored`
(`continued_fresh_from_last_saved`) and is coalesced into the
continuation record. Its files are not applied.
- `server/src/services/workspace-recovery.ts` calls it at two points:
when an attempt records an exhausted failure, and when a recovery pass
starts on an already exhausted hold (holds written before this change).
- `server/src/services/recovery/human-gate-evidence.ts`: one shared list
of human-gate evidence keys (`executionReconciliation`,
`OWNER_GONE_ATTESTATION_KEY`, `explicitUserContinuation`), with a SQL
helper and a TypeScript helper. Both treat a present key as a person's
decision, whatever its value. `isHumanGate`,
`completed-turn-cleanup-hold-settlement.ts` and
`interrupted-preparation-hold-settlement.ts` use them.
- New tests:
`server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts`.

The dispatch gate, the execution-blocker predicate, the wake-queue lock
release, and continuation delivery are unchanged from `master`. An
earlier revision of this pull request changed them. That approach was
removed because each review round found another dispatch case.

`fresh_isolated_workspace` continuations are out of scope on purpose.
They do not abandon a remote epoch, so they do not cause this failure.
Only a run that carries the isolation marker is protected from the old
local directory, so a late hold for such a run keeps blocking.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-isolated-continuation-hold.test.ts` gives 20 of
20 passing.
- Recovery settles the late hold for each `remote_epoch` delivery state
(`pending`, `delivered`, `delegated`, `not_requested`). The blocker is
then empty, and the hold is coalesced into the continuation.
- Assembled heartbeat test: after recovery settles the hold, the queued
run is the only run that starts. The old run stays failed. With a hold
that a person owns, the gate still withholds the queued run.
- The hold stays for each of these cases: no continuation, a
`remote_epoch` continuation while the run still owns its epoch, a run
with no remote workspace, `fresh_isolated_workspace`, `writer_ended`,
`invalidated`, `coalesced`, a hold that a person owns, an unsafe
archive, each human-gate key, and a null-valued human-gate key.
- Red/green: with the two `workspace-recovery.ts` calls disabled, 5 of
20 fail; without the abandoned-epoch check, the 2 epoch cases fail: the
4 settle cases and the assembled start test.
- A focused regression run of the server test files for the blocker,
fresh continuations, workspace recovery, both hold settlements,
owner-gone attestation, and the run-dispatch and wake-queue adapters.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Only exhausted, system-owned workspace-recovery holds of a run whose
remote epoch a recorded continuation replaced are settled. All other
holds keep blocking.
- After settlement, workspace recovery stops for that hold. Its files
were already unreachable, because the epoch was abandoned.
- A human-gate key with a null or false value now counts as a person's
decision in TypeScript too. Before, TypeScript ignored it and SQL
counted it. This makes the two sides agree and is more conservative.
- No schema change, no migration, and no Runner change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code with tool use (shell, file edit, code search) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 04:34:53 -06:00
Devin FoleyandPaperclip c41145e72a fix: wait on pending questions and quiet provider holds (#22363)
## Thinking Path

> - Paperclip manages agents and the tasks they work on.
> - Recovery should continue unfinished work when it is eligible.
> - An active unanswered question is a persisted reason to wait for a
person.
> - Treating a later Retry as an answer makes the recovery sweep
repeatedly request continuations that admission refuses.
> - Provider-limit holds also produce one cancellation toast per task,
even though provider work never started.
> - This PR aligns recovery with persisted interaction status and
presents these waits accurately.

## Linked Issues or Issue Description

Refs: #20466. This addresses the question blocker presentation and
automatic recovery loop; it preserves the existing
interaction-resolution policy.

Related work: #19420 schedules provider quota retries, and #21570
handles notices and reply ownership in agent chats. This change concerns
assigned-task reconciliation, task/inbox status, and per-run
cancellation toasts.

**What happened?**

The assigned-work sweep ignored pending interactions whenever a newer
run carried `explicitUserContinuation`. Continuation admission still
refused the pending question, so repeated sweeps recorded unsuccessful
continuation attempts. The task could meanwhile show an older provider
error as its current blocker. Separately, each pre-dispatch
provider-limit hold generated a cancellation toast, and the one-hour
fallback was described as a known reset and promised start time.

**Expected behavior**

Wait quietly while the question remains active under the shared
interaction eligibility rules, then react to the existing resolution
notification. Dismissed or historical ordinary questions remain
answerable without blocking work; governed requests keep their gates.
Show the current question on task and inbox surfaces. Keep provider
failure history, suppress cancellation toasts for expected provider
holds, and describe scheduled availability checks without promising the
provider will accept work.

**Steps to reproduce**

1. Leave an `ask_user_questions` interaction pending with
`supersedeOnUserComment: false`.
2. Create a later failed/interrupted explicit Retry and run
assigned-work reconciliation repeatedly.
3. Observe that the question remains pending while the sweep attempts
fresh continuations.
4. Separately, let an agent with recent provider rate-limit evidence
receive automatic task wakes without a reported reset time.

## What Changed

- Remove inferred question supersession from both assigned-work fact
loading and candidate selection. Use the existing active-interaction
predicate consistently in those reads, fresh-continuation admission, and
the task projection. Cover dismissed/historical ordinary questions,
retained governed gates, repeated checks, idle workers, and the existing
resolution notification.
- Add the `question` interruption action, load active pending question
IDs alongside approvals in one company-scoped query, and show that
blocker on tasks and inbox rows. Preserve a supported manual Retry after
a failed attempt, suppress duplicate actions for saved input, and keep
other current user gates authoritative. This preserves the existing
issue-detail SQL statement budget. Historical run failures remain
intact. Refresh the shared source SDK input pin.
- Suppress cancellation toasts specifically for cancelled
`provider_limit_wait` runs. Label provider retry deadlines as
availability rechecks, retain relative-time formatting, and record
successful scheduling as an informational lifecycle event.
- Document the wait contract in the continuation architecture guide.

## Verification

- Demonstrated failing regressions before the implementation:
continuation loop, task projection, task notice, cancellation toast,
provider deadline wording, and deadline formatting.
- Initial focused Vitest qualification: 642 tests passed across 14
files, using:

  ```sh
  pnpm exec vitest run \

server/src/__tests__/{assigned-work-invariant,assigned-work-recheck-idle,heartbeat-provider-limit-wait,task-interruption-query}.test.ts
\

server/src/services/{task-interruption-presentation,execution-projection,fresh-continuation-fallback,issue-thread-interactions}.test.ts
\
    server/src/services/recovery/assigned-work-invariant.test.ts \

ui/src/{components/ExecutionBlockerNotice.test.tsx,context/LiveUpdatesProvider.test.ts,context/LiveUpdatesProvider.hook.test.tsx,lib/provider-limit-message.test.ts,pages/Inbox.test.tsx}
  ```
- `pnpm --filter @paperclipai/shared typecheck` — passed.
- Review follow-up: 157 tests passed in `assigned-work-invariant`,
`task-interruption-query`, and `fresh-continuation-fallback`; 575 passed
in `task-interruption-query`, `assigned-work-recheck-idle`,
`task-interruption-presentation`, `issue-question-context`,
`issue-thread-interactions`, and `explicit-native-continuation` (the
query file overlaps). The strengthened final
dismissed/historical/governed projection regression also passed
separately.
- Final query-consolidation verification: 157 tests passed across five
files, including the unchanged 64-statement issue-detail census and
mixed approval/question batches:

  ```sh
  pnpm exec vitest run \

server/src/__tests__/{issue-detail-statements,task-interruption-query,assigned-work-invariant,assigned-work-recheck-idle}.test.ts
\
    server/src/services/task-interruption-presentation.test.ts
  ```
- `pnpm --filter @paperclipai/server typecheck` — passed, including test
layers.
- Manual-recovery follow-up: 204 tests passed across seven
projection/recovery/notice suites, and all 104 Inbox tests passed. The
lost-runner integration still starts exactly one run from a manual Retry
and leaves the question unanswered. The saved-decision census verifies
one decision batch query for a 41-task list, with at most three
statements referencing interactions; the overall 64-statement
issue-detail limit remains unchanged.
- `pnpm --filter @paperclipai/ui typecheck` — passed.
- `pnpm --filter @paperclipai/ui build` — passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse
HEAD) --check` — passed.
- `pnpm verify` — exit 2: broader CI is required for changed global
inputs. Focused tests and package checks supply local evidence; broad CI
remains required.
- `git diff --check` and a local secret/instance-data scan — passed.
- Final CI on `1c16fe7c0d8`: [trusted CI
workflow](https://github.com/paperclipai/paperclip/actions/runs/44650155842)
passed, including all server shards, native-server verification, build,
canary, e2e, release registry, and serialized suites. All PR checks are
complete, with no failures; the branch is conflict-free.
- Greptile reviewed the latest revision, `1c16fe7c0d8`, at 5/5 with no
actionable findings or unresolved review threads.

The focused tests cover genuine provider failures, live work and
terminal tasks, unchanged task locks/retry delivery, preservation of
historical failures, and question resolution waking the affected task
while other active pending questions remain quiet. Review regressions
also cover dismissed/historical ordinary questions retaining Retry and
governed requests retaining their gates. No live task was retried or
answered for validation.

## Risks

- An active pending question now consistently blocks automatic
assigned-work recovery. Shared interaction eligibility determines when
ordinary conversation input becomes deferred or historical; governed
requests retain their authority. Automatic recovery no longer infers
that a newer Retry resolved an active question.
- Manual Retry keeps its existing admission policy and recovery checks.
The question notice preserves that supported action after a failed
attempt without changing approval or credential authority; pending saved
input prevents duplicate actions.
- The shared interruption action gains `question`; the task and inbox
consumers are updated together. The pending-question query returns only
task IDs and uses the existing company/task/status index.
- Provider cooldown policy, quotas, credentials, routing, cancellation
history, and durable retry timing are unchanged. The actual provider can
still return a rate limit at the recheck.
- No migration is required. Existing large owner modules receive small
localized changes; no code file crosses 1,000 lines.

## Model Used

OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, xhigh
reasoning effort, with code execution and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-19 19:54:15 -08:00
DottaandClaude Opus 5.5 25b5cad7f9 fix(native-runtime): start first OpenCode turns from any saved launch-file copy instead of waiting on a stalled one (#22367)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents in remote sandboxes must have a saved, verified copy
of their launch files before their first turn (#19622, #21287)
> - The first runs of a company share one copy of these files per
server. Other runs wait for it
> - In a restart chaos wave, the shared copy out of one sandbox stalled
for 330 s. Seven other runs waited silently for it. Two runs then failed
before their first turn with the cause `unknown`
> - The copy speed out of a sandbox differs from one sandbox to the
next. When the first copy failed, the other sandboxes copied the same
314 MB in 15 to 24 s
> - This pull request lets every waiting run use the first copy that any
run saves, starts a second copy when the first one is slow, and names
the cause of a stalled copy
> - The benefit is a shorter cold start for the first runs of a company,
no failed run when another run already saved the files, and a named
cause instead of `unknown`

## Linked Issues or Issue Description

Refs #19622
Refs #21287

**What happened?**
A restart chaos wave started 15 OpenCode runs on new Daytona sandboxes
(1 vCPU) in a new company. The first run (A) started the shared copy of
the 314 MB launch files out of its sandbox. The copy did not finish. The
plugin call timed out after 330 s, and run A failed before its first
turn. The other runs waited for that copy without any run event, so the
task looked dead for 284 to 343 s. When run A failed, every waiting run
started its own copy at the same time. Five copies finished in 15 to 24
s. One copy (run B) stalled again and failed after 330 s, although a
saved copy existed after 15 s. One copy (run C) took 5 min and started
its turn 10 min after the run started, although it could have used the
saved copy 5 min earlier. Both failures had the technical cause
`unknown`. Each failure also logged "native result could not be marked
workspace_failed; the reconciliation loop will retry persisted results",
but the run had no result, so there was nothing to retry.

**Expected behavior**
A run uses the first saved copy as soon as one exists. A slow copy does
not hold all first runs. A stalled copy out of the sandbox has a named
cause. A run with no result does not log a result-finalization failure.

**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage and no saved launch-file copy.
2. Start 8 or more runs together on new sandboxes.
3. Make the copy out of the first sandbox stall (for example, a slow
sandbox link).
4. See that all other runs wait for the full plugin timeout, and see the
failures with the cause `unknown`.

**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).

## What Changed

- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`):
- The first capture that saves a ready copy ends the wait for every run.
A run whose own capture is still copying uses the saved copy and aborts
its own copy.
- A run whose own capture fails uses a saved copy when one exists, and
fails only when none exists.
- When the running capture is slower than 90 s, one waiting run starts
one more capture from its own sandbox. At most two captures run at once.
A capture keeps its slot until its copy out of the sandbox ends.
- The start decision is owned by one `SharedArtifactCapture` coordinator
(`tryStart()` returns a release lease).
- When the shared capture fails, one waiting run starts the next
capture. The other runs keep waiting for it. Before, every waiting run
started its own copy at the same time.
- The capture callback now gets its own abort signal, so a superseded
copy stops before upload.
- A failed copy out of the sandbox is now
`NativeProviderArtifactExportError` with the code
`native_provider_artifacts_export_timeout` (plugin call timed out) or
`native_provider_artifacts_export_failed`. The run's technical cause
shows this code instead of `unknown`. A timeout shows "Setting up the
computer took too long ...".
- `runtime-execution.ts`: do not try to mark the workspace as failed on
a native result when the run has no result. Before, this always threw
`native_finalization_missing` and logged a false "will retry" warning.
- `doc/opencode-cold-recovery.md`: describe the new sharing, the second
capture and the new cause codes.

## Verification

- Red/green: with the old `native-provider-artifact-capture.ts`, these 5
new tests fail. With this change, they pass:
- `native-provider-artifact-gate.test.ts`: "starts a run's turn from
another run's saved artifact while its own capture is still copying",
"uses a saved artifact when the run's own capture fails after another
run saved one", "starts one more capture when the shared capture runs
past the hedge delay, not one per waiting run", "does not start a
capture per waiting run when the shared capture fails", "names a stalled
copy out of the sandbox instead of an unknown cause".
- Red/green: `heartbeat-native-predispatch-retry.test.ts` "continues on
a fresh ... sandbox when the provider artifacts fail to save before any
turn" now also asserts that no "could not be marked workspace_failed"
warning is logged. It fails before the `runtime-execution.ts` change and
passes after it.
- `cd server && npx vitest run
src/services/native-runtime/native-provider-artifact-gate.test.ts
src/services/native-runtime/native-provider-artifact-capture-confinement.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts
src/services/native-runtime/native-provider-artifact-capture.test.ts
src/services/heartbeat/runtime-execution.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/heartbeat-native-predispatch-retry.test.ts`: all pass.
- `npx tsc --noEmit -p server`: pass.

## Risks

- A slow first capture can now cause one more 314 MB copy after 90 s. At
most two copies run at once for one company and manifest. A run whose
saved copy cannot be reused (for example, an ownership fence mismatch)
starts its own copy at once. Before, a failed burst started one copy per
waiting run at once, which was more.
- A superseded copy never uploads, but the provider call that copies
files out of the sandbox continues until it ends or times out. Its local
temporary directory is removed when it ends.
- Sharing is still per server. Two servers can still copy the same
files.
- No change to the rule from #19622: a verified saved copy still exists
before the first turn.
- No runner or image change. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Default context
window; extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 21:11:30 -06:00
19a7381398 feat: restore personal archiving in task views (#22358)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People use Tasks to see shared work and manage the work that needs
their attention.
> - The old inbox already let each person archive a task and undo that
action.
> - The combined Tasks surface needs the same controls in Unread and in
task table rows.
> - The existing inbox archive rules determine when new activity brings
a task back.
> - This pull request adds an Archived view so people can find and
restore their archived tasks.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Personal task archiving in Tasks. Related work includes the combined
Inbox and Tasks surface (#15100), the Coordinator workspace (#17983),
moving Mine into Tasks (#19363), and the existing inbox archive rules
(#9931).

**Subsystem affected**

Cross-cutting. The UI and task list API reuse the existing inbox archive
state.

**Current behavior**

Mine has an archive action. Unread and the task table have no equivalent
action. The Tasks menu has no dedicated view for restoring archived
tasks.

**Proposed behavior**

Add Archive and Undo to the task views. Let people restore tasks from
Archived. Keep archived tasks available in All tasks.

**Reason and benefit**

People can clear their personal lists and still find the original task.

**Breaking changes**

The task list API adds an optional archive filter and viewer archive
field. Existing callers use the same response and filters unless they
opt in.

## What Changed

- Reuse the current inbox archive and restore actions, activity rules,
and authorization.
- Add Archive and Restore to task table rows, Archive and Undo to
Unread, and a personal Archived view.
- Give each archived task its own Undo control, including when people
archive tasks quickly.
- Add tests for the archive list, account isolation, new activity, row
actions, and Undo.
- Document the archive flow and API filter. Refresh the generated SDK
source input registry.

## Verification

- Passed 287 focused tests in 12 Vitest files. These cover the task
views, row actions, archive cache, and PostgreSQL routes.
- Passed local server and UI typechecks. The source input registry check
also passed.
- Reproduced a missing Undo action when archiving two tasks in a row.
After the fix, the 158 task and inbox tests and the 2 toast tests
passed.
- On macOS with Chrome, archived a task from Mine and restored it using
Undo. Archived another task from All tasks, reloaded Archived, restored
it, and verified its Todo status and return to Mine.
- A human tester also verified Archive, Undo, and Restore in the local
browser.

- On commit `c5526fd9b1`, GitHub's
[verification](https://github.com/paperclipai/paperclip/actions/runs/44630156078/job/135950130662)
and [browser
tests](https://github.com/paperclipai/paperclip/actions/runs/44630156078/job/135946228919)
passed. A targeted retry passed after the first attempt hit a watchdog
self-test failure and two cancelled jobs.
- [Greptile scored
5/5](https://github.com/paperclipai/paperclip/pull/22358#issuecomment-7024083610)
on the same commit. Both review threads are resolved.

## Risks

- An incorrect visibility rule could hide a task from a personal list.
The archived filter uses the same activity predicate as Mine. The
PostgreSQL test covers isolation between people and resurfacing after a
reply.
- The archive state is already stored per person and company. No
database migration is required.

## Model Used

- OpenAI Codex, GPT-6, with shell, code editing, and browser
verification tools. The serving model ID and context size were
unavailable in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <1504818+scotttong@users.noreply.github.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-19 16:22:07 -08:00
Devin FoleyandPaperclip 24132dac16 fix(tasks): verify PR merges with the managed run identity (#22357)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native task completion independently verifies that declared PR
deliverables are merged.
> - Runs can already use an authorized managed GitHub connection to
merge private PRs.
> - Completion used the background credential resolver, which cannot
select that run's managed identity.
> - This pull request reuses operation-time identity and grant checks
for completion verification.
> - Agents can finish delivered work while unavailable grants and
unverified merges remain blocked.

## Linked Issues or Issue Description

Related: #22244 introduced the native PR merge gate. No duplicate fix
was found in the open PR search. The roadmap's personal GitHub identity
capability already covers the intended identity model.

**What happened?**

A native agent with a managed GitHub connection merges a private PR,
then completion rejects it as not verified merged. The independent
server-side check only tries legacy company token names or an instance
token; it does not use the agent run's managed connection.

**Expected behavior**

Verify the merge with the run's current accepted GitHub identity,
retaining current grant, membership, trust and company boundaries.
GitHub must confirm the merge regardless of saved work-product metadata.

**Steps to reproduce**

1. Configure a managed GitHub identity with access to a private
repository and no legacy company GitHub token.
2. Use a native task run to merge and register a PR deliverable.
3. Finish the task. Before this change, the merge verifier cannot
authenticate with the managed identity and rejects completion.

## What Changed

- Bind completion credential lookup to the persisted company, native
task and run.
- Add a server-side completion credential path that shares managed
grant, membership and trust checks with command credentials. Both active
and succeeded runs capture identity under the existing task/run locks
and reconcile pending steering acknowledgements before credential
lookup. The command credential API still requires a running run.
- Resolve credentials once for all PRs in a verification attempt; do no
credential work for tasks without PR deliverables.
- Preserve legacy fallback only for runs without a managed identity, and
preserve the public-only restriction on server-environment credentials.
- Add 23 PostgreSQL-backed authorization regression cases and document
completion's identity behavior.

## Verification

- Local checks cover revision
`da2a3608b0375ead646bbbad35af853191488f07`. [Greptile
review](https://github.com/paperclipai/paperclip/pull/22357#issuecomment-7023243798)
is 5/5 on that revision with both prior threads resolved. [Required
CI](https://github.com/paperclipai/paperclip/actions/runs/44625662403)
passed on attempt 2 for the same revision, including typecheck, build,
canary, server/runner suites and all eight e2e shards.
- Red/green: the private-PR and identity-selection cases failed before
the initial fix. Review exposed a succeeded-run reassessment failure and
then an unsettled-steering gap in that path; each has a regression that
failed on the reviewed commit and passes after its correction.
Unresolved steering performs no token lookup, and durable
acknowledgement selects the new person. The full worker test completes
the task after automatic-review withdrawal while command credentials
remain unavailable to the finished run.
- `pnpm exec vitest run
server/src/services/native-runtime/native-pull-request-completion.test.ts
server/src/services/native-runtime/native-completion-feedback.test.ts
server/src/__tests__/github-operation-credentials.test.ts
server/src/__tests__/native-status-reassessment-work.test.ts
server/src/__tests__/run-identity.test.ts
server/src/__tests__/run-identity-projections.test.ts` — 222 tests pass
across six files, on macOS arm64 with Node 25.6.1 and embedded
PostgreSQL.
- `pnpm --filter @paperclipai/server typecheck` — passes.
- `pnpm verify --explain` — exits 2 because the subsystem documentation
selects broad CI coverage. Required CI supplies the broader test/build
coverage; this exit is not a full-suite pass.
- The credential selection, steering, grant and company checks use real
PostgreSQL state. Only the secret vault and GitHub HTTP transport are
mocked; test requests never reach GitHub.
- `git diff --check` and the staged secret/PII scan pass. No runtime
artifacts are committed.
- Earlier CI revision `4e42f64`: the [Build
job](https://github.com/paperclipai/paperclip/actions/runs/44624163145/job/135914203042)
ended after the runner received a shutdown signal and the workflow was
cancelled. Its log does not show a compiler error before interruption;
this interrupted attempt is not build verification. The current revision
subsequently passed its own CI build on attempt 2.
- Current revision canary: the [Canary Dry Run
job](https://github.com/paperclipai/paperclip/actions/runs/44625662403/job/135919504493)
was interrupted by runner shutdown before completion. The repository
runner-loss classifier returns `lost: true` for its actual annotation
and log. A targeted retry was refused while the workflow remained
active. The same revision subsequently passed Canary Dry Run and the
full workflow on attempt 2; the earlier interrupted attempt remains
recorded here separately.
- Live acceptance remains pending deployment; the change has not been
deployed.

## Risks

Completion now reads managed credentials through a server-side path
shared with the existing authorization checks. Successful finished runs
can be reassessed without granting them command execution credentials.
Denied or unavailable identities must remain blocked without falling
through to a different identity; regression tests cover revocation and
company/task isolation. Credential lookup is shared within one bounded
PR verification attempt, not cached across attempts. The existing HTTP
deadline and merge proof requirements remain in place. No migrations or
API contract changes.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, xhigh reasoning, with repository
inspection, code editing and test execution. Context-window size is not
exposed by the harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-19 14:44:03 -08:00
Nicky LeachandPaperclip d8ae55198d Scope runtime catalog reads to standing-instruction sources (#22356)
## Thinking Path

> - Paperclip uses standing instructions from authorized app connections
when it prepares a run.
> - The gateway already knows which connections have instructions
enabled.
> - The catalog read still loads tool schemas for every eligible
connection in the company.
> - This work delays prompt preparation and consumes database and
application resources.
> - The gateway can restrict the existing catalog query while preserving
the names used for authorization.

## Linked Issues or Issue Description

**Subsystem affected** — Tool gateway and run preparation.

**Problem or motivation** — The instruction resolver filters tool
descriptors after the database has returned unrelated catalog entries.

**Proposed solution** — Scope the persisted catalog read to possible
instruction sources. Include peers that share a candidate’s short
connection ID when assigning tool names.

**Alternatives considered** — An exact connection ID filter could change
a tool name and its policy decision. A separate instruction
authorization query would duplicate gateway policy.

**Roadmap alignment** — This improves the existing task-start path.
Related work includes #22347 and #14864.

**Additional context** — The change uses the existing read-only catalog
snapshot. It adds no schema or cache.

## What Changed

- Pass the instruction candidates to the shared gateway catalog reader.
- Read eligible naming peers in the same snapshot. Return only candidate
descriptors after assigning their names.
- Retain the existing run identity, guest restrictions, model
visibility, provider tools, policy decisions and grants.

## Verification

- Ran the connection instruction and gateway listing-memory integration
suites against embedded PostgreSQL. All 23 tests passed.
- Confirmed the bounded-read assertion fails on the baseline and passes
on the change. A full listing reads 20 catalog entries from two
connections. The instruction lookup reads 10 entries from one.
- Covered exact-name approval and block policies, peer quarantine,
unchanged instruction text and digest, and visibility of unrelated tools
in normal discovery.
- Ran integration tests for gateway discovery, gateway service behavior,
and assigned MCP tools. All 256 tests passed.
- Compared complete instruction resolution with embedded PostgreSQL on
macOS, using 16 alternating measured pairs after five warm-ups. The
one-tool fixture took 10.05 ms median on the baseline and 10.32 ms on
the change. With 1,994 synthetic entries across 21 connections and four
candidate entries, the medians were 50.44 ms and 24.47 ms. The median
paired saving was 26.03 ms. Final instruction text and digest matched
for every pair.
- The comparison measures the local instruction resolver. Remote
submission-to-turn latency was outside this verification.

## Risks

The candidate read uses the first UUID component because it participates
in gateway naming. Matching peers contribute to name collisions even
when an agent cannot use the peer. The regression checks that a blocked
candidate cannot gain access when the catalog is narrowed. Companies
that enable instructions on most connections can expect smaller savings.

## Model Used

OpenAI Codex directed and qualified the change. GPT-6.2 Sol implemented
the change and focused regression tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-19 14:07:48 -08:00
Nicky Leach 28043f8b93 Measure database and projection costs in runtime tool discovery (#22347)
## Thinking Path

> - Paperclip starts agent turns with instructions for the connections
the agent can use.
> - The gateway reads the connected-tool catalog during prompt
preparation.
> - A single discovery span currently covers both database waits and
descriptor construction.
> - Maintainers need to distinguish these costs before changing catalog
reads.
> - This change records the transaction, both queries, availability
checks, and descriptor construction.

## Linked Issues or Issue Description

**Subsystem affected** — Runtime tool discovery and run preparation.

**Problem or motivation** — The existing catalog span cannot explain
time spent before a remote turn starts.

**Proposed solution** — Measure the existing read in place. Record the
number of entries and connections returned.

**Alternatives considered** — A local microbenchmark cannot reproduce a
hosted database or a deployed catalog's sizes. A change to the query
would confound the measurement.

**Roadmap alignment** — This adds diagnostic detail to an existing
operation. Related work includes #14864, which bounded catalog memory
use.

**Additional context** — The PostgreSQL reads share one read-only
snapshot.

## What Changed

- Add a transaction span with children for the entry read and the
matched-connection read.
- Attach the two returned row counts to that transaction.
- Time availability checks and descriptor construction under the
existing catalog discovery span.
- Document which preparation timings can also reach the native run event
table.

## Verification

- Ran the gateway listing-memory, discovery HTTP, and execution timing
context suites with local PostgreSQL. All 30 tests passed.
- Checked the parentage and row counts with a real OpenTelemetry
exporter in memory.
- Ran the standing-instruction integration suite and measured the
catalog from two staging application containers with read-only database
sessions.

## Risks

Each recorded catalog operation adds up to five spans. The existing
per-phase bound applies. Only the OTel transaction span includes the
aggregate row counts. With or without OTel, the existing native
preparation timing path can also persist a run event for each new phase
that takes at least 50 ms, subject to its normal caps. The executor
defers those writes. The database query, tool eligibility, and
authorization contracts stay the same.

## Model Used

OpenAI Codex directed and qualified the work. GPT-6.2 Sol implemented
the instrumentation and regression test.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-19 12:41:34 -08:00
DottaandClaude Opus 5.5 6be53f0831 fix(issues): keep a held task blocked when its agent tries to resume it (#22245)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops with an unknown outcome, recovery can record an
execution hold for the task. Wake admission refuses every wake for the
task until the system continues it from saved work or a person uses
Retry. Recovery sets the task to "Blocked", so a person can see it.
> - An agent can move that task out of "Blocked". It can change the
status, send `reopen` or `resume`, or check the task out. None of these
routes checks the execution hold, so the hold stays.
> - The task then looks workable, but its agent cannot start. Its
scheduled follow-up check (monitor) comes due, and the scheduler tries
it on every tick. Wake admission refuses each try. Agents re-arm the
check, and the check stays overdue for days.
> - #22215 makes these tasks show as "Blocked" in lists. It does not
stop agents from making the state.
> - This pull request refuses an agent request that moves a held task
out of "Blocked". The response is 409 and gives the next action of the
hold.
> - The benefit is that a held task stays "Blocked" until its hold is
released. Its follow-up checks wait and do not retry on every tick, and
the agent gets the correct reason.

## Linked Issues or Issue Description

(B) No issue exists. Bug report:

**What happened?**
Scheduled follow-up checks on agent tasks stayed overdue for many days.
Each due check was skipped with `execution_reconciliation_required`. One
skip record merged more than 25,000 attempts. The tasks were in
`in_progress` or `in_review`, but an execution hold from a failed run
refused every wake. The agent re-armed the checks, and the checks never
started. An agent can move a task out of `blocked` while such a hold
exists. The resume guard and checkout do not check the hold.

**What did you expect to happen?**
A task that wake admission holds stays `blocked` until the hold is
released. An agent that tries to move it gets an error that tells it
what must happen first.

**Steps to reproduce**
1. Make a `blocked` task that is assigned to an agent.
2. Add a recovery record for the task with status `resolved`, cause
`provider_failure_meaning_unverified` and evidence
`automaticRecovery.replay = "blocked"`. `getExecutionBlocker` returns
this hold.
3. As the assigned agent, send `PATCH /api/issues/:id` with `{ "status":
"in_progress" }`. You can also send `reopen` or `resume`, or `POST
/api/issues/:id/checkout` with `expectedStatuses: ["blocked"]`.
4. See that the request succeeds and the task leaves `blocked`. The hold
stays, so wake admission still refuses the task.
5. Give the task an execution policy with a review stage. As the agent,
send `PATCH /api/issues/:id` with `{ "status": "done" }`. The execution
policy changes the status to `in_review` and assigns the reviewer. The
task leaves `blocked`, and the hold stays.

**Paperclip version or commit**
`master` at deacaf727c.

Refs #22215 (display status for the same holds). Related open PRs:
#14590 parks monitor wakes on held tasks and retires a no-replay hold
when a person restores the status. #14156 changes how a settled
no-replay hold is released. #19786 keeps the rule that only a board user
retires an automatic no-replay hold. This pull request adds the matching
rule for agents and task status.

## What Changed

- `server/src/routes/issues.ts`: new `assertAgentMayLeaveBlocked`. It
gets the status that the request will save. When an agent moves a
`blocked` task to a status other than `blocked`, `done` or `cancelled`,
it reads `getExecutionBlocker`. If a recovery hold applies, it returns
409 with `recoveryActionId`, `cause` and `nextAction`. The task does not
change.
- `PATCH /issues/:id` calls the check one time, after the execution
policy transition and before any write. It uses the final status. This
covers a status change, `reopen`, `resume`, and a `done` request that a
review or approval stage changes to `in_review`.
- `POST /issues/:id/comments` calls the check when the comment moves the
task to `todo` (`reopen` or `resume`). The native `comment_on_task` tool
uses the comments route. Agent comments without `reopen` or `resume` do
not move a `blocked` task.
- `POST /issues/:id/checkout` calls the check with `in_progress` when
`expectedStatuses` includes `blocked`.
- The check applies only to recovery holds (a blocker with a
`recoveryActionId`). A failed legacy run whose process or lease is still
alive also blocks wakes (`execution_owner_active`), but that lock clears
without a person. If the check refused it, the task would stay `blocked`
after the lock clears, and nothing would release it.
- `server/src/services/execution-blocker.ts`: `getExecutionBlocker` gets
a `recoveryHoldOnly` option. With this option, it does not return the
live-run lock first, so it finds a recovery hold that exists at the same
time. The check uses this option. Other callers do not change.
- Behavior for people does not change. A board user can still change the
status, send a message or use Retry. An agent can still close a held
task.
- `server/src/__tests__/issue-recovery-actions.test.ts`: new tests with
a real database. They seed a `blocked` task with a resolved no-replay
hold and try each agent path, including `done` with a review stage. A
second test shows that an agent can still cancel the task, and can move
it to `todo` after the hold stops blocking replay. A third test shows
that a live previous run without a recovery hold does not hold the
agent. A case with both a live previous run and a recovery hold shows
that the agent is refused.

## Verification

- Failing before the fix (first commit): `pnpm --dir server exec vitest
run src/__tests__/issue-recovery-actions.test.ts -t "no-replay hold when
its agent"`. The tests "keeps a task blocked behind a no-replay hold
when its agent tries a status change / reopen / resume comment /
checkout" failed with `expected 200 to be 409` (`201` for the comment).
- Failing before the fix (second commit): the new case "... tries a
completion that its review stage hands to a reviewer" failed with
`expected 200 to be 409`. The route accepted `done` and started to wake
the reviewer.
- Failing before the fix (third commit): the new case "... tries a
status change while a live previous run also owns the task" failed with
`expected 200 to be 409`.
- Passing after the fix: `vitest run
src/__tests__/issue-recovery-actions.test.ts -t "no-replay hold|live
previous run|close a held task"`, 8 of 8 pass.
- Affected suites on the current head: `vitest run` on
`issue-recovery-actions`, `issue-comment-reopen-routes`,
`issue-update-comment-wakeup-routes`,
`issue-stale-execution-lock-routes`, `issue-closed-workspace-routes`,
`low-trust-red-team-routes`, `issue-agent-mutation-ownership-routes`,
`issue-execution-policy-routes`, `issue-status-version-route-service`,
`heartbeat-held-comment-wake`, `legacy-continuation-authority` and
`issue-display-status`. 12 files, 703 tests pass (with
`--hookTimeout=240000`, because a setup hook is slow in the local
sandbox).
- Typecheck: after `pnpm --filter @paperclipai/paperclip-runner run
build:typescript`, `node server/scripts/typecheck-layers.mjs` checks all
server layers and test trees, with no errors in server files. The local
install has only the server dependencies, so files in `cli/`, `ui/` and
other packages could not resolve their modules. CI runs the full
typecheck.
- Not run locally: the full `pnpm verify` selection is too large for the
local sandbox. CI runs it.

## Risks

- Behavior change for agents: an agent that tries to move a held task
out of `blocked` now gets 409. Wake admission already refused that task,
so the agent could not run it.
- The system still continues a hold that it can settle (fresh
continuation from saved work and the assigned-work sweep). That path
does not need an agent to unblock the task.
- An agent `reopen` or `resume` comment on a held task is refused, and
the comment is not saved. The agent can post the comment without
`reopen` or `resume`.
- Cost: one `getExecutionBlocker` read, only when an agent tries to move
a `blocked` task.
- Not covered: tasks that are already in this state, and recovery paths
that record a hold without setting the task to `blocked`. These still
show as "Blocked" through #22215. #14590 changes monitor retries for
them.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edit,
test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 14:37:57 -06:00
Devin FoleyandPaperclip 46dde54f52 fix(cloud): migrate run evidence during online preparation (#22337)
Drive bounded run-log and provider trace migration pages during online durable-data preparation. Share the page owner and settled-native archive rules with background migration, preserve active-source and storage-integrity guards, and report fresh page results.

Keep canceled transfers owned and tracked until the underlying operation settles. Retain page cursors across the preparation budget.

Verified 52 focused tests, including embedded PostgreSQL and object storage, server typechecking, and all 58 successful PR checks. Greptile 5/5 with no unresolved review comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-19 10:28:05 -08:00
DottaandClaude Opus 5.5 264beefeff fix(tools): retry a tool audit write that loses a deadlock instead of failing the call (#22329)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents call connector tools through the tool gateway. The gateway
writes a `tool_call_events` audit row for each step of a call.
> - The audit insert has foreign keys to `companies`, `agents`,
`heartbeat_runs` and `issues`. Postgres takes a key-share lock on each
parent row, in trigger order: company, agent, run, issue.
> - Other writers lock the issue row and then the run row. When such a
transaction holds the issue and then asks for the run, while the audit
insert holds the run and waits for the issue, Postgres detects a
deadlock and aborts the audit insert (`40P01`).
> - The gateway then returns the raw `Failed query: insert into
"tool_call_events" ...` error to the agent. The agent does not get the
tool result. The same call works when the agent sends it again.
> - This pull request runs the audit insert through the shared bounded
retry that #20971 added for the same lock cycle on
`workspace_operations`.
> - The benefit is that a short lock conflict no longer fails a
connector call or loses its audit row.

## Linked Issues or Issue Description

**What happened?**
Some connector calls fail with `Failed query: insert into
"tool_call_events" ...`. This happens most often when an agent sends
several tool calls in parallel in one run. The failed call waits for
some seconds, then fails. The other parallel calls succeed. The same
call succeeds when the agent sends it again. The Postgres cause is
`deadlock detected` (`40P01`).

**Expected behavior**
A transient lock conflict does not fail the tool call. The gateway
writes the audit row and returns the tool result.

**Steps to reproduce**
1. Transaction A: lock the issue row `FOR UPDATE`.
2. Call a connector tool for a run on that issue. Its `tool_call_events`
insert takes key-share on the company, agent and run rows, then waits
for the issue.
3. Transaction A: lock the run row `FOR UPDATE`.
4. Postgres reports a deadlock. Before this change the audit insert is
the victim, and the tool call fails with the database error.

**Additional context**
The new test reproduces this with two real transactions on embedded
Postgres. Transaction A gets a longer `deadlock_timeout`, so the audit
insert is always the victim, as in production. The failing statement in
the test has the same column list and the same placeholder shape as the
production error.

Refs #20971
Refs #21703

Open PRs #15830 and #16596 make the audit insert non-fatal. That
approach drops the audit row. This change keeps the row.

## What Changed

- `server/src/services/tool-gateway.ts`: `writeToolCallEvent` runs its
`tool_call_events` insert through `retryTransientLockContention`. The
helper retries only lock timeout, deadlock and serialization failure
(`55P03`, `40P01`, `40001`): 5 attempts, with 100 ms × attempt backoff.
Other errors fail as before.
- `server/src/__tests__/tool-gateway-connector-telemetry.test.ts`: new
test with a real two-transaction deadlock between an issue-then-run
locker and the audit insert of a real `executeTool` call. The fixture
now also returns the issue.

## Verification

- Failing before the fix: `cd server && npx vitest run
src/__tests__/tool-gateway-connector-telemetry.test.ts -t "loses a
deadlock"` failed. `executeTool` rejected with `Failed query: insert
into "tool_call_events" ...`, cause `deadlock detected`, code `40P01`.
- Passing after the fix: the same test passes. I ran it 3 times.
- The full `tool-gateway-connector-telemetry.test.ts` file passes (9
tests).
- `tool-gateway-service.test.ts` passes (194 tests).
- Not run: `pnpm verify` and the server typecheck. The local sandbox has
not enough memory and disk for them. CI runs both.

## Risks

- Low risk. Only the `tool_call_events` insert retries, and only on the
three contention codes.
- Each attempt is a new single-statement write. A deadlock victim's
statement rolls back completely, so a retry cannot write a duplicate
row.
- The worst added delay is about 1 s of backoff across 5 attempts, and
only when the lock conflict continues.
- Authorization, screening and approval checks do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution on embedded
Postgres).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:23:21 -06:00
Devin FoleyandPaperclip 417802ca87 fix(plugins): refresh unchanged-version bundle descriptions (#22295)
Refresh known display descriptions in same-version bundled plugin registrations only when every behavioral field matches. Compare manifests in their persisted JSON form and await bundle reconciliation before final volume-free package qualification.

Verified 77 focused tests, server typechecking, and all 59 successful PR checks. Greptile 5/5 with no unresolved review comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-19 09:26:06 -08:00
DottaandClaude Opus 5.5 b7177db6aa fix(telemetry): report a Paperclip runner agent's underlying harness as adapter_type (#22324)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends product telemetry. Two events describe agent work:
`agent.task_run` and `agent.task_completed`.
> - Each event has an `adapter_type` dimension. The generated telemetry
contract gives this dimension a fixed enum.
> - A Paperclip runner agent stores `paperclip_runner` as its adapter
type. The enum does not have this value.
> - The collector and Cloud direct delivery change a value that is not
in the enum to `other`.
> - Thus, as more agents use the runner, the adapter breakdown in
analytics becomes mostly `other`. The data is not lost, but it is in one
bucket.
> - This pull request makes a runner agent report the harness that it
runs, for example `claude_local` or `codex_local`.
> - The benefit is that the adapter breakdown shows the real harness
again.

## Linked Issues or Issue Description

Refs #8818 (this PR added the generated telemetry contract and its
`adapter_type` enum).

I did not find an existing issue. This is the bug description:

**What happened?**

`agent.task_run` and `agent.task_completed` send the stored
`agents.adapter_type`. For a Paperclip runner agent, this value is
`paperclip_runner`. The `adapter_type` enum in
`packages/shared/src/telemetry/generated/paperclip-telemetry.ts` does
not include `paperclip_runner`. The telemetry normalizer changes the
value to `other`. In analytics, almost all runner agent events show
`adapter_type = other`.

**Expected behavior**

A runner agent reports the harness that it runs. The server already
resolves this harness with `agentHarnessType()` from
`@paperclipai/shared`. For example, a runner agent with the Claude ACP
configuration must report `claude_local`.

**Steps to reproduce**

1. Create a Paperclip runner agent (`adapterType: "paperclip_runner"`).
2. Let the agent complete a heartbeat run, or let it set a task to
`done`.
3. Look at the `agent.task_run` or `agent.task_completed` event.
`adapter_type` is `paperclip_runner` before delivery, and `other` after
normalization.

**Paperclip version or commit**

`master` at 1355f93546.

**Agent adapter(s) involved**

Paperclip runner (all harnesses).

## What Changed

- `server/src/services/agent-task-run-telemetry.ts`: new
`telemetryAdapterType(agent)` helper. It returns
`agentHarnessType(agent.adapterType, adapterConfig)`. `agent.task_run`
sends this value as `adapterType`.
- `server/src/routes/issues.ts`: `agent.task_completed` calls the same
helper. Thus, the two events cannot report different adapter types for
the same agent.
- A legacy agent (for example `claude_local`) sends the same value as
before. `agentHarnessType()` returns a legacy adapter type without a
change.
- `server/src/__tests__/agent-task-run-telemetry.test.ts`:
- New tests for the default, Claude, and OpenCode runner configurations.
They expect `codex_local`, `claude_local`, and `opencode_local`.
- New contract test. It resolves every runner provider and every ACPX
agent through `telemetryAdapterType()`, and compares the result with the
generated `adapter_type` enum of both events. The test fails when a
runner harness is missing from the contract and is not in the known-gap
list.
- `server/src/__tests__/issue-telemetry-routes.test.ts`: new test that a
runner agent with the Claude ACP configuration sends `claude_local` on
`agent.task_completed`.
- `packages/shared/src/telemetry/README.md`:
- The `agent.task_run` section tells what `adapter_type` means for a
runner agent, and lists the runner harnesses that the contract does not
have yet.
- The `Emission Boundary` section tells why runner harness resolution is
not an alias map. The receiving layer does not get the agent
configuration, so it cannot derive the harness from `paperclip_runner`.
The emitter sends the resolved value raw and does not fall back to
`other`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pin for
the changed README. I ran `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`.

## Verification

- Red-green: before the fix, the new emitter tests failed with
`paperclip_runner`. After the fix, they pass.
- `cd server && npx vitest run
src/__tests__/agent-task-run-telemetry.test.ts
src/__tests__/issue-telemetry-routes.test.ts
src/__tests__/posthog-direct-event-shape.test.ts` and the three native
tests that mock the telemetry service: 182 tests pass.
- The contract test fails when I remove one known gap (`muse`) from the
list. Then it passes again when I restore the list.
- `cd server && pnpm typecheck`: passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
`changed: false`.
- Before the rebase, I sent the old and new values through the real
PostHog normalizer. The default, Claude, and OpenCode runner
configurations changed from `other` to `codex_local`, `claude_local`,
and `opencode_local`.
- I did not run the full local suite. `pnpm verify` asks for broader
coverage because the pin registry is a global CI input. CI supplies that
coverage.

## Risks

- Low risk. The change affects only the telemetry dimension value. It
does not change agent behavior, the database, or an API.
- Eight of the fifteen runner harnesses map to enum values:
`codex_local`, `opencode_local`, `claude_local`, `cursor`,
`cursor_cloud`, `grok_local`, `pi_local`, and `kimi_local`.
- Seven runner harnesses continue to show `other`: `acpx:copilot`,
`antigravity_runtime`, `aws_agentcore`, `claude_managed`, `grok_bot`,
`muse`, and `openai_dot`. The fix for these is a change to the contract
in the telemetry backend.
- After this change, `adapter_type` does not show the difference between
a runner agent and a legacy agent with the same harness. A separate
dimension in the telemetry contract can show this difference later.
- Events from before this change keep `other`.

## Privacy Review Request

`rules/data-paths.mdc` requires a privacy review for a telemetry change.
I request a privacy review from a maintainer. This is my analysis:

- The change does not add an event, a dimension, or an enum value. Thus,
the generated contract does not change. The contract comes from the
telemetry backend, and this repository must not edit it.
- The new value comes from `agentHarnessType()`. On each validated write
path (create, hire, PATCH, config rollback, and company import), the
server accepts only a closed set of runner `provider` and `acpxAgent`
values. Thus, the emitted value is a fixed harness identifier, not user
text.
- An agent row that does not pass these checks (for example, a
plugin-managed hire or a direct database write) can hold a different
`provider` string. Before this change, the emitter already sent the raw
`agents.adapter_type` string, which has the same exposure. The receiving
layer changes any value that is not in the enum to `other` before
analytics.
- The value never contains a prompt, a credential, a path, a model
output, or a person's name.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) from Anthropic, in Claude Code. It
used tool use (shell, file read and edit, `gh`) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:20:01 -06:00
deacaf727c feat(connections): guided AWS role setup with a per-connection CloudFormation template (#22233)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The AWS connector (#22195) lets agents use AWS with credentials
minted per connection. Its role method stores no AWS keys
> - That method needs the customer to create an IAM role that trusts
exactly one Paperclip connection, but setup only offered the generic
form and a doc
> - The connect API also refused the draft step this method needs, so
the role method could not be set up through the UI at all
> - This pull request adds a guided setup screen. Paperclip saves a
draft, hands the user a prompt for their own agent that creates the
role, takes back the role ARN the agent reports, and tests the same
draft. CloudFormation and IAM steps are there under "Or do it by hand"
> - The benefit is that a person can connect AWS without stored keys and
without typing account details, using a template AWS has validated

## Linked Issues or Issue Description

Refs #22195.

**Subsystem affected**
Connections (AWS connector setup), UI, tool-access API.

**Problem or motivation**
The AWS role method must save a connection before AWS can trust it. The
API rejected `saveDraft` for every method except custom MCP servers, so
the UI could not complete this flow. There was also no screen that
showed the issuer, the subject, the trust policy or a template.

**Proposed solution**
- The connect API accepts `saveDraft` for methods that declare
`credentialMinter`.
- `GET /tool-connections/:id/aws-setup` returns the setup values and a
CloudFormation template generated for that connection.
- A new `AwsRoleSetup` step runs between saving the draft and testing
it. The role is named after the connection and CloudFormation fills in
the account, so the first form no longer asks for a role ARN; the user
pastes it back once the role exists.

**Alternatives considered**
- A one-click CloudFormation quick-create link. Rejected: AWS accepts
quick-create templates only from Amazon S3, so Paperclip would need to
host a public template bucket. The screen offers a download and a
console link instead.
- Instructions in a doc only. Rejected: too many hand-copied values.
- Asking for the role ARN up front. Rejected: the role does not exist
yet, so users had to invent an ARN and then match it exactly.
- Letting the user's agent report the role ARN to Paperclip itself,
through a short-lived link. Not done here: it needs a new
unauthenticated endpoint (and a Paperclip Cloud front-door change). The
user pastes the ARN instead.

## What Changed

- `packages/shared`:
  - The `saveDraft` validation allows methods with minted credentials.
  - New `AwsConnectionSetup` type.
- `server/src/services/connectors/aws/setup.ts` builds the setup values:
  - The issuer, audience and connection subject.
  - The provider host and the trust policy.
- A CloudFormation template with an optional identity provider
(`CreateIdentityProvider`), a role named `paperclip-<connection id>`,
`AccessPolicyArn` (`ReadOnlyAccess` by default for read-only
connections) and `AWSMCPSignInOAuthAccessPolicy`. The account comes from
CloudFormation, so the template needs no edits.
- The service method `getAwsSetup` and the route `GET
/api/tool-connections/{connectionId}/aws-setup`:
  - The route requires `connections:configure`.
  - It returns 422 for a connection that is not an AWS role connection.
- It reports whether the issuer key is published and any role ARN
already saved.
- It returns `available: false` (`address_changed`) when the instance
address changed after the draft was saved, since a role set up for the
old address could never pass the test.
  - It is registered in OpenAPI.
- UI:
- The AWS form no longer shows a Role ARN field (it is hidden with an
empty default).
- `AwsRoleSetup.tsx` shows, by default, only "Set it up with your agent"
(a copyable prompt that checks the AWS account, deploys the template
with `aws cloudformation deploy`, and reports the role ARN) and a field
to paste that ARN. "Or do it by hand" reveals the CloudFormation steps
(download, console link, parameter notes, IAM acknowledgement) and the
IAM console / AWS CLI steps with copyable values; the CLI commands read
the account ID themselves.
- It shows an access-key message when AWS cannot reach the instance, and
asks the user to save again when the instance address changed.
- Failed tests appear in place. AWS error codes show the server's own
plain message, because the shared error copy replaces every 502 with
"temporarily unavailable".
- `useAwsRoleSetupStep` owns the AWS sequence; `ConnectionSetupFlow`
only asks it for request options, hands back results and renders its
step:
- The first Connect for `aws-role` saves a draft and opens the AWS step.
  - "Test connection" resumes the same draft with the pasted role ARN.
- Back keeps the draft, so a later Connect updates it and does not
create a second one. Returning to an unfinished draft reopens the step.
- Docs: the `doc/connections/AWS.md` setup section and the plan's phase
table. The SDK public input pins are refreshed.

## Verification

- `server/src/__tests__/aws-mcp-connector.test.ts` covers a new
API-level case through the real route validation:
- Saving an `aws-role` draft over HTTP works without a role ARN, and
stores the keys the screen uses to reopen it.
- `aws-setup` returns the issuer, subject and the connection's role
name.
- The template's trust condition keys equal `<issuer host>:aud` and
`<issuer host>:sub`.
  - The template provider URL equals the issuer.
- After an address change it returns `address_changed`, and saving the
draft again records the new address.
  - An access-key connection returns 422.
  - No response field looks like a secret.
- `ui/src/features/connections/AwsRoleSetup.test.tsx` covers:
- Only the agent setup shows by default; Test connection stays off until
a valid role ARN is pasted, then sends it.
  - A saved role ARN is filled in.
- The by-hand steps and CLI commands; the agent prompt for read-only and
full access.
  - The template download (filename and object URL cleanup).
  - The unreachable-instance message.
  - The in-place error and the address-changed message.
  - The AWS message mapping.
- `ui/src/pages/apps/AppsConnect.test.tsx` covers the full two-step
flow: the form has no Role ARN field, the first Connect sends
`saveDraft: true` with no role ARN and does not finish, the AWS step
loads the setup, and "Test connection" resumes `aws-draft` with the
pasted role ARN and finishes. A second test reopens the step for a
returning draft.
- Red/green:
  - Removing the API change fails the API test.
  - Removing the draft-first wiring fails the flow test.
- Totals: server AWS, OpenAPI and connector tests (75), shared
`app-definitions` (42), UI connections and apps pages (1108). Typecheck
is clean for shared, server, UI (except Storybook's generated catalog,
which this worktree has not built) and all four server test-tree layers.
- Browser, on a real dev server with an HTTPS public address:
  - AWS appears in Connectors with its logo.
- The form has only region and access level. Connect saves the draft
(reusing an earlier one) and shows only the agent prompt and the role
ARN field; "Or do it by hand" reveals the CloudFormation and IAM steps.
- The copied prompt names the connection's role and stack, and the
template.
- Test connection with a pasted role ARN showed AWS's refusal in place
(no such role, placeholder issuer), as expected. An earlier run found
the generic 502 copy problem, which is fixed and re-checked.
- AWS CloudFormation `validate-template` (dev account, read-only)
accepts the current generated template (role named after the connection)
and reports `CAPABILITY_NAMED_IAM`. The screen tells the user to
acknowledge it.
- Not run: an agent actually running the prompt and creating a stack in
AWS, and Paperclip Cloud staging.

## Risks

- The feature is still behind `enableAwsConnector` (off by default). No
migration.
- The template creates IAM resources in the customer's account, and the
customer must acknowledge that. The role trusts only this connection's
subject.
- If the customer already has an identity provider for the issuer and
leaves `CreateIdentityProvider` at true, AWS fails the stack with a
clear error. The screen explains the parameter.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, the in-app browser against a local dev server, AWS
CLI `validate-template` in the dev account, an exploration subagent).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-19 10:53:53 -06:00
DottaandClaude Opus 5.5 3b978692ee fix(github): end a superseded automatic review with a clear outcome (#22218)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for work
> - The GitHub bot can do an automatic pull request review. An agent run does the review with review tools.
> - Each automatic run has authority for one commit only: the head commit of its pull request event.
> - When a new push moves the head during the review, the review tools refuse the run. This is correct.
> - But the refusal told the agent to "continue with the newest authorized PR event". The agent cannot do that. Agents call the refused tools again, then post the stale review as an ordinary comment.
> - This pull request makes the refusal a clear, terminal "superseded" outcome. It says if a newer review is already queued and tells the agent to stop.
> - The benefit is fewer wasted tool calls and no stale review comments on the pull request. Commit-specific authority does not change.

## Linked Issues or Issue Description

**What happened?**
An automatic pull request review starts for commit A. During the review, the author pushes commit B. Each review tool call (for example `read_pull_request`) then fails with "This automatic review was requested for an older pull request head. Continue with the newest authorized PR event or an authorized manual request." Agents call the tools again several times. Then they post their review as an ordinary pull request comment, because the comment tool does not check the head.

**What did you expect?**
The refusal tells the agent that the review is superseded. It names the old and new commits. It says if a newer automatic review for the new head is already queued on the task. It tells the agent to stop, not to call review tools again, and not to post the review as a comment. The error has a machine-readable reason code and is not retryable.

**Steps to reproduce**
1. Configure the GitHub bot for automatic reviews.
2. Open a pull request. An automatic review run starts.
3. Push a new commit while the run reviews.
4. The run calls a review tool. The tool refuses with the old, unclear message.

## What Changed

- `server/src/services/chat-github-reviews.ts`: the stale-head check now builds the error in a new `supersededAutomaticReview` helper. The helper looks for a newer automatic delivery for the current head of the same pull request. It matches the repository id and pull number that the event saves, because a new delivery has no conversation id until it is processed. It counts only deliveries that are newer than the run's own event, in the states `received`, `processing`, `processed`, or `retry`. The query stays scoped to the company and endpoint. The 409 error message names both commits, says if a newer review is queued, and tells the agent to stop. The error details carry `reasonCode: "github_review_superseded"`, `retryable: false`, `requestedCommit`, `currentCommit`, and `newerReviewQueued`.
- The first sentence of the message does not change, so existing logs and searches still match.
- The run still cannot read or review the newer head. Authorization does not change.
- `server/src/__tests__/chat-channels.integration.test.ts`: the existing stale-head assertion now checks three outcomes: a newer review is queued and processed, a newer review is still waiting with no conversation id, and no newer review is queued (the head moved with no event).

## Verification

- Failing before: `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t "admits signed PR events into an ordinary assigned task"` with only the test change. The test failed: `expected Error: This automatic review was requeste… to match object { status: 409, details: { reasonCode: "github_review_superseded", … } }`, and the received error had `details: undefined`.
- Passing after: the same command with the fix: `Tests 1 passed`.
- Failing before (second case): the test sends a second `synchronize` event and does not process it, so its delivery has no conversation id. Before the identity match, the test failed: expected `newerReviewQueued: true`, received `newerReviewQueued: false`. After the fix, the test passes.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts -t "GitHub agent review workflow"`: 300 passed.
- `pnpm --filter @paperclipai/server typecheck` (layered typecheck): passed.
- `pnpm verify` selected 2,925 of 3,792 test files because many modules import this service. I stopped it before the end. It reported no failures before I stopped it. Required CI gives the full coverage.

## Risks

- Low risk. The change applies only to the error path when an automatic review's head is stale.
- The error path does one more indexed-scope query on `chat_deliveries` (endpoint and conversation). This query runs only when a refusal occurs.
- The message text changes after its first sentence. Code that matches the full old text would need an update. No code in this repository does.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), by Anthropic, in Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before requesting merge

Related: #21290 (completes the check run of a superseded head; a different code path).

🤖 Generated with [Claude Code](https://claude.com/claude-code)



Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 09:57:39 -06:00
575ef12575 feat(responsibilities): file company skills under responsibilities (#22275)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A responsibility shows the skills of the agents doing its work.
People cannot say which skills a responsibility needs.
> - The maintainer chose to file skills under responsibilities like
projects, tasks and routines.
> - Agents also need to set a skill's catalog details without rewriting
its SKILL.md.
> - This pull request lets people and agents file, move and remove
company skills under a responsibility, and adds `update_skill_details`.
> - The benefit is that a responsibility lists the skills it needs,
beside the skills its agents already hold.

## Linked Issues or Issue Description

Refs #22253, #22261 and #22274; this pull request is stacked on them.

**Subsystem affected**

Database (`responsibility_links`), shared types and validators, server
responsibility service and routes, Dashboard versions, native runner
tools, and the responsibility page UI.

**Problem or motivation**

Skills under a responsibility are only computed from the agents doing
its work. A person cannot record that "Hiring needs interviewing", and
an agent cannot file a skill when it maps work for a person.

**Proposed solution**

Add a nullable `skill_id` to `responsibility_links` and allow it as the
one target of a link. Show filed and computed skills together. Let the
file, unfile, lookup and link routes and tools take skills. Keep filed
skills in Dashboard versions so undo covers them. Add a small live tool
for a skill's tagline and categories.

**Alternatives considered**

Keeping skills computed only, and giving the skill to the agent that
runs the responsibility, was the other option. It was not chosen.
Extending `update_skill` would change its scenario contract, whose mock
needs `markdown`, so the details tool is separate.

## What Changed

- Review fixes: the default responsibility view's Skills "+" adds an
existing skill, removes a filed one, or opens an agent's skills page. A
newest Dashboard version without a skill list is upgraded in place, so
the first skill filed after it is undone. Undo files again, or removes,
only skills the restoring person can see. A read scope without a skill
condition sees no filed skill. `update_skill_details` with only
`skillId` is a no-op, as its schema allows.

- Migration `0499`: add `responsibility_links.skill_id` (FK to
`company_skills`, on delete cascade). Widen the single-target check to
four targets. Add a unique `(responsibility_id, skill_id)` index and a
`skill_id` index. The migration is idempotent and online-compatible (NOT
VALID then VALIDATE, and the indexes are built CONCURRENTLY). It is
listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`.
- Shared: `ResponsibilityLink.skillId`;
`ResponsibilitySkillSummary.linkId` and `computed`; `skillId` in the
link and lookup schemas; `skill` in `RESPONSIBILITY_FILE_ITEM_KINDS`;
optional `skillIds` in Dashboard snapshot links.
- Server: filing checks the reader's skill visibility
(`skillAccessService`). The detail merges filed and computed skills into
one card each. Lookup accepts `skillId`. Dashboard capture, restore and
summary include filed skills. A version without `skillIds` leaves filed
skills as they are when it is restored.
- Tools: `file_under_responsibility`, `remove_from_responsibility` and
`find_responsibilities_for` accept `kind: "skill"`. `get_responsibility`
returns `linkId` and `computed` for each skill. Add
`update_skill_details` (tagline and categories).
- UI: the responsibility page has "Add existing skill". A filed skill
shows "Filed here" and a remove button. Computed skills keep their
holders.

## Verification

- Server: the `responsibilit*` tests (new: file, find, move and unfile a
skill; one card for a skill that is both filed and computed; undo of a
filed skill from the Dashboard history), `skill-tools.test.ts` (new
`update_skill_details` cases), `responsibility-tools`, `routine-tools`,
`routines-routes`, `paperclip-runner-tool-authority`, `runner-api` and
`openapi-routes`: 24 files, 2723 tests passed.
- UI: responsibility page and component tests (16 files, 158 tests,
including a new filed-skill test) pass. `pnpm --filter @paperclipai/ui
typecheck` passed.
- `@paperclipai/db` `check:migrations` passes and reports the override.
The snapshot drift test passes. `tsc` passes for `db`, `shared` and the
server layers.
- Runner: catalog and semantic-tool tests (317) pass. All generator
checks pass.

## Risks

- The migration changes a check constraint on `responsibility_links`.
The replacement accepts every row that the old check accepted. It is
swapped as NOT VALID and then validated. The previous release never sets
`skill_id`.
- Deleting a company skill removes its links (cascade).
- Filed skills follow the reader's skill visibility. A skill the reader
may not see is not listed.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-19 09:30:49 -06:00
b7319eb581 feat(routines): let agents list and change their person's routines in plain terms (#22274)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Routines are recurring work that an agent runs on a schedule,
webhook or request. People file them under their responsibilities.
> - #22253 and #22261 let agents read and change areas and
responsibilities. Agents still have no dedicated routine tools, and they
can change only routines assigned to themselves.
> - When a person says "move my weekly digest to Monday mornings" or
"map my routines to my responsibilities", the agent must use the generic
API, write cron, and it is refused for the person's routines.
> - This pull request adds `list_routines` and `update_routine`. It also
lets an agent change routines that the person its run works for is
responsible for.
> - The benefit is that an agent can find routines under no
responsibility, file them, and change their schedule in plain terms.
Each change is a routine revision the person can restore.

## Linked Issues or Issue Description

Refs #22261 (phase 2; this pull request is stacked on it), #22253 (phase
1).

**Subsystem affected**

Server (routine routes, native runner tool authority), the native runner
tool catalog, and the `paperclip` skill.

**Problem or motivation**

An agent can list and change only its own routines through the generic
API. It cannot change the schedule of a routine its person is
responsible for. It must write cron expressions, and it cannot find the
routines that are filed under no responsibility.

**Proposed solution**

Make the existing scenario operation `list_routines` live, with filters
for words, status, agent, project, responsible person, a responsibility
and "unfiled". Add `update_routine` for the title, instructions, agent,
project, status, priority, a plain `schedule`, and filing under a
responsibility. Let an agent manage a routine when the person its run
works for is responsible for it. Handing the routine to another agent
needs that person to have `tasks:assign`.

**Alternatives considered**

A new `routines` permission grant would add a setting with no current
need. Routine revisions already give an undo path, so this change adds
no new history.

## What Changed

- `list_routines` is now live (it was a scenario-only mock) and keeps
its scenario mock.
- Add `update_routine`. A `schedule` such as `{ frequency: "weekly",
dayOfWeek: "monday", time: "09:00", timezone }` maps to the routine's
schedule trigger through the existing routine-offer cadence helpers.
`schedule: null` turns the schedule off. `fileUnder` files the routine
under a responsibility.
- Add `server/src/services/routine-tools.ts`. It calls the routine
routes with the run token. Fields and schedule go in one `PATCH
/routines/:id` (one transaction and revision, guarded by
`baseRevisionId`); filing follows, and if it fails after the routine
changed the tool reports what was saved. Each part reports its own undo:
the routine revision, or the Dashboard history for filing.
- `PATCH /routines/:id` takes `schedule` (`{ cronExpression, timezone?
}` or `null`) and replaces the schedule under the routine row lock, so
racing changes leave one schedule. Without a zone it keeps the current
schedule's, else uses the person's saved time zone, else answers 422.
- `GET /companies/:companyId/routines` filters `q`, `status`,
`assigneeAgentId`, `responsibleUserId`, `responsibilityId` and
`unfiled=true` in SQL.
- In `routes/routines.ts`, an agent may manage routines assigned to
itself or routines its person is responsible for. On a routine the agent
does not run, taking it from another agent (or handing it on), turning
it on, changing its schedule (also through the trigger routes), starting
a run, or restoring a revision requires that person's `tasks:assign`, as
for a person. The person must have directed the run (see #22261).
`routine.updated` activity records `onBehalfOfUserId`.
- The routine and responsibility routes share one rule,
`agentPersonInCompany` in `routes/agent-person-authz.ts`.
- The catalog keeps each live action's `requiredClaims` (`list_routines`
is `routines:read`).
- The routine tools stay visible when Responsibilities is off. Only
`responsibilityId`, `unfiled` and `fileUnder` need it.
- Update `search_api` tool pointers, the routines reference, the skill
section, the chat directive, the operation-groups spec and the pinned
counts.

## Verification

- Server: `routine-tools.test.ts` (new), `routines-routes.test.ts` (new:
an agent changes its person's routine, hands it off only with
`tasks:assign`, and is refused for another person's routine), the
`responsibilit*` tests, `paperclip-runner-tool-authority` and
`runner-api`: all pass. The first test in `routines-routes.test.ts`
(Fireflies) times out at 15 s in my sandbox on a cold import, with or
without this change. It passes with `--testTimeout=60000`.
- Server `node scripts/typecheck-layers.mjs`: passed.
- Runner: catalog, semantic-tool, eval, scenario and permission-policy
tests (316) pass. All generator checks pass.

## Risks

- Medium-low. Agents get write access to their person's routines. The
route still checks the company, the routine revision guard and
`tasks:assign` for a hand-off. The person can restore any earlier
routine revision.
- `list_routines` changes from a scenario-only mock to a live tool, so
the reconciled catalog counts change.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-19 09:30:03 -06:00
DottaandClaude Opus 5.5 c0aa7751f8 feat(responsibilities): let agents change areas and responsibilities for their person, with undo (#22261)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People keep their areas and responsibilities in Paperclip
(experimental, `enableResponsibilities`). #22253 lets agents read them.
> - Agents can still only propose. They cannot rename or archive an
area, change an active responsibility, let one go, or move work between
responsibilities in one step.
> - A person who says "file my weekly routines under Sales" must then do
it by hand.
> - The maintainer chose this rule: an agent may make each change that
its person could make, the person can undo it from the Dashboard
history, and a permanent delete still needs the person's yes on a card.
> - This pull request adds eight change tools and the server rules
behind them.
> - The benefit is that an agent maps work to responsibilities for the
person, and the person keeps control through undo and the delete card.

## Linked Issues or Issue Description

Refs #22253 (phase 1; this pull request is stacked on it).

**Subsystem affected**

Server (responsibility and area routes, Dashboard versions, native
runner tool authority), shared validators and Dashboard types, the
native runner tool catalog, the `paperclip` skill, and the Dashboard
history UI.

**Problem or motivation**

An agent that works for a person can only propose areas and
responsibilities. It cannot change active ones, archive or delete them,
or move filed work in one step. Unfiling needs link IDs. There is no way
for the person to undo an agent's change.

**Proposed solution**

When an agent acts for the person its run works for (`onBehalfOfUserId`)
and that person directed the run (the run's own task is one they made,
such as their chat, or their own message woke it; parent tasks do not
count), apply that person's own rights: their own responsibilities, or
anyone's for a company owner or admin; areas for any active member. Save
each such change as a version of the person's Dashboard, so the existing
history and Undo line show it under the agent's name. Make a permanent
delete wait for an accepted confirmation card that the agent posted. Add
`file` and `unfile` routes that take items by kind and ID, with an
optional move.

**Alternatives considered**

One card for every change, or propose-only. The maintainer chose "act
for me, with undo". A separate undo journal would need a new table and a
new UI; Dashboard versions already have both. A new version cause needs
a check-constraint migration; this change uses `change_set` with no
`interactionId` instead.

## What Changed

- Add tools `create_area`, `update_area`, `delete_area`,
`create_responsibility`, `update_responsibility`,
`delete_responsibility`, `file_under_responsibility` and
`remove_from_responsibility`. They are standard-mode only and hidden
while Responsibilities is off.
- Add the writer rule for agents in `routes/agent-person-authz.ts`
(`agentPersonInCompany`, `agentActsForResponsibilityOwner`), used by
`routes/responsibilities.ts` and `routes/responsibility-areas.ts`. The
person must have directed the run: a person can name anyone as a task's
responsible person, and a run can also work for a routine's person or
the company default, so those runs do not borrow that person's rights.
An agent with no such person keeps the old propose-only rules.
- The response names only the version of the responsibility's owner (for
area changes, the acting person's), and `X-Paperclip-Dashboard-Owner`
says whose history holds it; tools return `dashboardOwnerUserId`. A
change Undo cannot reverse (the name, description, order or un-archiving
of an area other people's proposed or active responsibilities share)
saves no version and is not reported as undoable.
- `/file` without `moveFrom` follows the `/links` rule, so
`create_responsibility` with `fileItems` works for a proposal.
- A title-only create is allowed for an acting agent. It is active by
default and records the agent in `proposedByAgentId`.
- Add `responsibility-agent-changes.ts`. In one transaction that holds
each affected owner's Dashboard lock, it saves the dashboard before the
change (when a person changed it by hand), runs the change on services
bound to that transaction, and saves the result as a `change_set`
version by that agent. The response names the version in
`X-Paperclip-Dashboard-Version`. The tools say Undo only when a version
was saved.
- Dashboard snapshots now also hold each responsibility's filed items,
description, "run by" agent, evidence, health override and note, and
each area's description (optional fields). Restore puts them back. Older
versions still restore as before. A matching older version is upgraded
in place, so the first change after it compares and undoes in full.
- Create and update accept `areaTitle`. The server finds or makes that
area in the same transaction, so a refused change leaves no area. Agent
creates reuse a live responsibility with the same title under a lock, so
retries and parallel calls make no copies.
- Add `POST /api/responsibilities/:id/file` (`items`, optional
`moveFrom`: an ID or `all`) and `POST /api/responsibilities/:id/unfile`
(`items`), each in one transaction.
- Agent deletes need `confirmationId`: an accepted
`request_confirmation` card that this agent posted, with target key
`responsibility-delete:<id>` or `responsibility-area-delete:<id>`.
Someone who may manage the owner must accept it, on the card or by a
verified chat reply. The tools post the card (named after the item),
return `awaiting_confirmation`, and delete on the next call after
acceptance.
- `DELETE /api/responsibility-areas/:id` accepts
`moveResponsibilitiesTo`. Area changes and deletes save versions for
everyone whose Dashboard shows the area.
- Agent activity details now include `onBehalfOfUserId`. The Dashboard
history shows "<agent>, for you" for these versions.
`ResponsibilityDashboardVersionSummary` now has `interactionId`.
- Update the OpenAPI registry and metadata, the `search_api` tool
pointers, the skill, the API reference, the chat directive, the
operation-groups spec, and the pinned counts and SDK source pins.

## Verification

- Server: `vitest run` of the `responsibilit*` tests,
`responsibility-tools.test.ts`, `openapi-routes`, `runner-api*`,
`paperclip-runner-tool-authority` and `task-markdown` all pass. New
tests cover these cases: acting-agent create, update and undo; file,
move and undo; the rule for a person who may not manage the owner;
delete only after an accepted card, including a verified chat reply;
area delete with move and a version for the other owner; undo of a
health override, an area description and an un-archived empty area;
upgrade of an older version; an area by title that a refused change
rolls back; parallel creates.
- UI: `DashboardEditModule.test.tsx` and
`responsibility-dashboard.test.ts` passed. `pnpm --filter
@paperclipai/ui typecheck` passed.
- Server `node scripts/typecheck-layers.mjs` passed.
- Runner: catalog, semantic-tool, eval, protocol-action and
permission-policy tests (284) passed. All generator checks pass.
- I did not run the browser e2e suite locally.

## Risks

- Medium. Agents that work for a person can now change and archive that
person's responsibilities and areas without a card. Every change is a
Dashboard version that the person can restore, and a delete still needs
the card.
- Undo restores titles, sentences, areas, order, status, filed items,
description and the "run by" agent. It does not restore a bar or metric
change; those stay in the metric history.
- A restore of a version from before this change does not touch filed
items, which matches the earlier behavior.
- Undo of a rename or archive of an area that other people also use
keeps its name, as the existing restore rule does for shared areas.
- Manual edits do not take the Dashboard lock. A person's edit that
commits while an agent's change is open can appear in that agent's
version.
- Changes to proposed or let-go responsibilities are not on a Dashboard,
so they cannot be undone there. The tools say so.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 09:29:11 -06:00
Nicky Leach 503268b806 perf(wake-queue): preflight task admission without loading execution context (#22289)
## Thinking Path

> - Paperclip uses task admission to turn assignments and comments into
queued work.
> - Wake-queue owns the issue-admission transaction. Heartbeat
previously loaded full execution context to decide whether a
conversation wake could proceed.
> - After the conversation preflight, task state can change before
admission obtains its lock.
> - Wake-queue now exposes the preflight decision and obtains only the
fields it needs. It provides current task attribution from the locked
row.
> - Heartbeat uses the existing responsible-user and routine rules when
it creates a run.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Admitting board assignments and saved task comments.

**Subsystem affected**

Server task admission.

**Current behavior**

Conversation preflight and fresh-run attribution load full task
execution context from heartbeat, including chat joins.

**Proposed behavior**

Give wake-queue ownership of conversation preflight and its PostgreSQL
read. Use the issue row already locked by wake-queue for fresh-run
attribution.

**Additional context**

Builds on #22280 and #22282. #17396 covers a separate workspace
inspection on task context reads.

## What Changed

- Give heartbeat a preflight decision. Keep storage and the five-field
query inside wake-queue.
- Reuse the conversation and retry predicates for all callers. Keep the
locked recheck after preflight, including durable chat authorization.
- Supply the task's responsible user, parent and routine origin from the
current locked row. A company-only admission supplies no issue.

## Verification

On `b89bed954a`, all 307 tests passed across the wake-queue preflight
and release application suites, heartbeat queue and run-preparation
suites, and Slack conversation lifecycle suite. The integration suites
used embedded PostgreSQL. Coverage includes the owner 403, a
conversation settling before admission locks the issue, saved-comment
attribution after reassignment, and routine ownership written inside the
admission transaction. The layered server typecheck and module
dependency check passed on the same revision.

In a repeated local comparison of base `90d985b8b9` and PR head
`b89bed954a`, we measured 34 samples per revision, scenario, and
recovery-history state, alternating the two revisions in ABBA order.
Server-side admission medians were:

| Recovery history | Assignment | One comment | Two serial comments |
| --- | ---: | ---: | ---: |
| Empty | 23.45 → 21.80 ms | 23.95 → 23.00 ms | 47.30 → 45.40 ms |
| Resolved | 23.35 → 21.30 ms | 23.70 → 22.05 ms | 46.70 → 45.40 ms |

Typical SQL statements fell from 36/37/74 to 35/36/73 for the three
scenarios. Lock statement counts were 2/2/6 for both revisions.
Throughput varied under local contention. These are local measurements.

Standard Greptile scored the exact head 5/5 with no review threads.
[Greptile
review](https://github.com/paperclipai/paperclip/pull/22289#issuecomment-6999578088)

GitHub CI passed on the exact head after one targeted retry of the
serialized server shard. The initial status-card maintenance case timed
out waiting for idle, passed when run locally on the same revision, and
passed in the retried shard. [CI workflow attempt
2](https://github.com/paperclipai/paperclip/actions/runs/44458613437/attempts/2)

## Risks

An early decision can go stale, and the caller must preserve
authenticated actor identity. The locked path retains its own
eligibility checks. A regression exercises the gap between the two
decisions. An attribution test inserts its evidence inside the
transaction to detect a read through a different connection.

## Model Used

OpenAI GPT-6.2 Sol (gpt-6.2-sol) implemented the change. OpenAI Codex
reviewed and qualified it.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-19 07:23:02 -08:00
DottaandClaude Opus 5.5 ffcd49f6aa feat(responsibilities): let agents read a person's areas and responsibilities (#22253)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People keep their areas and responsibilities in Paperclip
(experimental, `enableResponsibilities`). Agents chat with them about
this work.
> - Agents have no dedicated tool to read a person's responsibilities.
The only path is the generic `search_api` and `call_api` fallback.
> - That path needs the person's user ID, and nothing in a chat tells
the agent that responsibilities exist. When the feature is off, every
route returns a 404 that looks like "nothing there".
> - So a chief-of-staff agent asked the person "what are your
responsibilities, or where are they written down?" when Paperclip
already had them.
> - This pull request adds three read tools, makes the owner default to
the person the run works for, and tells agents in chats how to use the
tools or that the feature is off.
> - The benefit is that an agent reads and maps a person's
responsibilities itself. This is phase 1 of 4. Later pull requests add
the change tools for areas, responsibilities, routines and skills.

## Linked Issues or Issue Description

**Subsystem affected**

Server (responsibility routes, native runner tool authority, chat
prompt), native runner tool catalog, and the `paperclip` skill.

**Problem or motivation**

An agent in a chat cannot find a person's responsibilities. It has no
dedicated tool. The list route requires `ownerUserId` from an agent, and
the chat does not give the agent that ID. The chat prompt does not
mention responsibilities. When the experiment is off, the 404 does not
tell the agent that the feature is off. The agent then asks the person
to list their responsibilities again.

**Proposed solution**

Add three optional native tools: `list_responsibilities`,
`get_responsibility` and `find_responsibilities_for`. They call the
existing routes with the run's agent token, so all route checks and the
activity log stay the same. The owner defaults to the person the run
works for. The routes return a `responsibilities_disabled` code while
the feature is off. The tools are hidden while the feature is off, and
the chat prompt then says that the feature is off.

**Alternatives considered**

Better `search_api` guidance alone still needs the user ID and many
calls. A new read service in the tool authority would copy the route
permission rules. The tools use the routes instead.

## What Changed

- Add `list_responsibilities` (areas with their responsibilities,
responsibilities with no area, totals, and company work filed under no
responsibility), `get_responsibility` (one responsibility with its filed
items, link IDs, skills and latest status report), and
`find_responsibilities_for` (where a project, task or routine is filed;
a task can be a `PAP-12` identifier).
- Add `server/src/services/responsibility-tools.ts`. It calls the routes
over HTTP with the run token and returns compact results with page
links.
- When an agent sends no `ownerUserId`, the responsibility list, create,
summary, automation and dashboard routes use the person the run works
for (`onBehalfOfUserId`). A run that works for no person still gets a
422 with a clear message.
- The nine "not enabled" checks now throw one shared error. The 404 and
its message do not change. The body now also has `code:
"responsibilities_disabled"` and a remediation line. One
`RESPONSIBILITIES_OFF_GUIDANCE` text is the remediation, the tool error
and the chat prompt note.
- `list_responsibilities` with `areaId` counts only that area. Archived
areas are listed only on request. When only proposals are hidden, the
tool says so.
- `services/paperclip-api-loopback.ts` holds the tools' authenticated
API call; the project tools use it too.
- Show the tools only while the instance has Responsibilities on. A
hidden tool that is called by name returns "Areas and responsibilities
are turned off".
- In standing chats, add an "Areas and responsibilities" prompt block:
use the tools before you ask the person, or say that the feature is off.
- `search_api` results for these routes now name the dedicated tools.
- Add an "Areas and responsibilities" section to
`skills/paperclip/SKILL.md` and to `references/api-reference.md`. Add
one line to the chief-of-staff persona.
- Register the tools in the runner catalogs. Regenerate the semantic
contracts, the action catalog, protocol coverage, the operation-groups
spec and the capability inventory. Update the pinned counts.

## Verification

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/catalog src/semantic-tools src/tools src/eval/workflow-evals.test.ts
src/protocol-actions src/evals/native-execution.test.ts
src/drivers/acpx/permission-policy.test.ts`: 276 passed.
- All runner generator checks pass: protocol manifest, capability
contract, capability inventory, protocol coverage, semantic contracts,
semantic action catalog, operation groups, replay goldens, workflow
traceability.
- Server: `vitest run` of `responsibilit*` route tests,
`responsibility-tools.test.ts` (new), `task-markdown.test.ts`,
`task-context*`, `paperclip-runner-tool-authority.test.ts`,
`runner-api.test.ts`, onboarding first-task tests and
`user-facing-wording-guidance.test.ts`: 20 files, 2620 tests passed.
- `node scripts/typecheck-layers.mjs` for the server: passed.
- New tests cover these items: the owner default for an agent and the
422 without a person, the `responsibilities_disabled` code on every
route, the tool projections, an identifier lookup, the "turned off"
mapping, the tool visibility switch, and the chat prompt block.

## Risks

- Low risk. The routes keep their permission checks. Agents can still
only read and propose in this phase.
- An agent that sends no `ownerUserId` now gets the responsible person's
list instead of a 422. Agents that name an owner see no change.
- The chat prompt gets one more short block in standing chats.
- The flag is pinned when a turn opens. If an admin changes it during a
turn, the tool list changes on the next turn. The routes always check
the current setting.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 08:52:42 -06:00
DottaandClaude Opus 5.5 53fedd653b fix(connections): split GitHub Copilot into its own connector (#22238)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Connectors catalog lets people give agents tools and AI
accounts.
> - GitHub showed repository access and GitHub Copilot authentication on
one card.
> - The card had an extra "Connect Copilot" button. Saved Copilot
accounts showed in the GitHub account list.
> - These are two different products. People must find each one by its
own name.
> - This pull request makes GitHub Copilot a separate connector. GitHub
keeps repository and MCP access only.
> - The benefit is a clear catalog. Each card opens its own setup, and
saved accounts show on the correct card.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Discovery and setup of the GitHub and GitHub Copilot connectors.

**Current behavior**

The GitHub card says "Connect GitHub repository access or authenticate
GitHub Copilot agents." It shows a "Connect Copilot" button next to its
own Connect button. Saved Copilot accounts show in the GitHub account
list. Agent connection search returns Copilot as a method of GitHub. The
Copilot setup link that search gives to agents opens the GitHub tool
setup.

**Proposed behavior**

GitHub and GitHub Copilot are two cards. GitHub says "Give agents access
to GitHub repositories, issues, and pull requests." GitHub Copilot has
its own logo and opens Copilot token setup. Saved Copilot accounts,
including accounts saved before this change, show on the GitHub Copilot
card.

**Reason and benefit**

People do not have to know that Copilot is behind the GitHub card. The
catalog names the two products. The Copilot runtime keeps the same
provider identity, token storage, grants, and model discovery.

**Additional context**

This follows the same pattern as the GitHub Code Review Bot split in
#14750. Copilot setup was added in #15705.

## What Changed

- Add a `github-copilot` app definition with the existing
`copilot-token` method. Remove that method from `github`. Restore the
GitHub description and the `developer` category.
- Add the GitHub Octicons Copilot mark (MIT license, from
`primer/octicons`) as light and dark brand assets.
- Map the `github` AI provider to the `github-copilot` catalog entry in
`aiConnectionCatalogSlug`. New Copilot accounts save under a "GitHub
Copilot" application.
- Remove the "Connect Copilot" button from the GitHub card.
- Show saved AI accounts on the card for their provider. This includes
Copilot accounts that were saved under the GitHub application. The
connection page and its reconnect link use the same rule.
- Redirect old `/apps/connect?source=github&method=copilot-token` links
to `source=github-copilot`. Keep the other query values.
- Keep the "My GitHub account" identity wording on the Copilot access
step.
- In connection search and connection requests, file AI accounts under
their catalog entry. Accept the `github` provider for the
`github-copilot` service. Copilot authentication recovery cards now open
GitHub Copilot setup, not GitHub tool setup.
- Keep older Copilot accounts and requests working. An older account
saved under the GitHub application reconnects through GitHub Copilot,
and an older Copilot draft resumes through GitHub Copilot. A recovery
request saved with the `github` service name can complete. A Copilot
request declined before this change stays declined in its continuation.
Adoption accepts the `github` provider binding for a `github-copilot`
request.
- Move the rule that finds an AI account's catalog entry
(`aiConnectionSourceSlug`) into `@paperclipai/shared`, so the UI and
server use one copy.
- Use the Copilot logo for Copilot in the AI account picker and account
details.
- Update the GitHub Copilot connection doc and the GitHub connections
Storybook story. The GitHub and GitHub Copilot setup stories open
**Change** before checking "Connect GitHub as", because that heading
sits in the collapsed access section.

## Verification

- New and updated tests fail on `master` and pass with this change:
  - `pnpm exec vitest run packages/shared/src/app-definitions.test.ts`
- `pnpm exec vitest run ui/src/pages/apps/Browse.test.tsx
ui/src/pages/apps/AppsConnect.test.tsx
ui/src/pages/apps/AppDetail.test.tsx
ui/src/pages/apps/app-connect-policy.test.ts
ui/src/pages/apps/Connections.test.tsx`
- `pnpm exec vitest run
server/src/__tests__/connection-intents-service.test.ts
server/src/__tests__/ai-connections.test.ts`
- `pnpm exec vitest run
ui/src/features/connections/ConnectionSetupFlow.oauth.test.ts`
- Older-account tests cover reconnect, draft resume, completion of a
request saved with `github`, a request declined before the split, and
Copilot adoption. Each test fails without its fix.
- The play functions of all 8 stories in
`github-connections.stories.tsx` pass when rendered with
`composeStories` in jsdom.
- Brand checks: `node scripts/check-app-brand-assets.mjs`,
`ui/src/lib/app-brand-assets.test.ts`, and
`ui/src/pages/apps/AppLogo.brand-assets.test.tsx`.
- Typecheck: `@paperclipai/shared`, `@paperclipai/ui`, and
`@paperclipai/server`.
- Manual check in `pnpm dev:once` with a local company:
1. Open Connectors and search for "GitHub". Two cards show: "GitHub" and
"GitHub Copilot". GitHub has no "Connect Copilot" button.
2. Seed one GitHub tool account and one Copilot account under the GitHub
application, as older installs have them. The GitHub tool account shows
on GitHub. The Copilot account shows on GitHub Copilot.
3. Click Connect on GitHub Copilot. The "Connect GitHub Copilot" page
opens with the Copilot logo and the token field.
4. Open `/apps/connect?source=github&method=copilot-token`. The page
redirects to `source=github-copilot`.
  5. Open GitHub setup. It offers repository access only.
  6. Check the catalog in light and dark themes.

## Risks

- Low risk. There is no database migration. Existing Copilot accounts
keep their application and stored source. The UI and connection search
group them by their AI provider.
- New Copilot accounts save under a new "GitHub Copilot" application and
record `github-copilot` as their source. Code that finds GitHub
repository identities by `sourceTemplateKey === "github"` no longer sees
new Copilot accounts. These are AI accounts, so this is the intended
separation.
- A pending Copilot recovery card from before this change keeps the
`github` service slug. The server still resolves it to GitHub Copilot,
and the card can complete.
- Search for "github" now returns two results: GitHub and GitHub
Copilot.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
(shell, file edits, test runs, and a headless browser for the manual
check).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 08:51:36 -06:00
DottaandClaude Opus 5.5 1355f93546 fix(dispatch): schedule a clean loop-limit continuation with its person's work (#22312)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue decides which queued run an agent starts next.
It puts system recovery in the lowest tier.
> - When a run hits the tool-call loop limit, the server queues a
continuation with the wake reason `issue_continuation_needed`. The
source run succeeded. It did not fail.
> - The queue classed that continuation as system recovery. While any
person-requested run was queued, it got no share. It started only when
the 60-minute aging bound expired.
> - In a restart-chaos wave, 4 of 7 loop continuations waited 60:00 to
60:28. One task broke the "no rest within budget" rule. One run then
resumed an hour-old sandbox, timed out and showed a Retry banner.
> - This pull request schedules a clean loop-limit continuation in the
tier of its task, so a person's task that continues its own work stays
with that person's work.
> - Real failure recovery keeps the recovery tier and the aggregate
share, so a failure loop cannot become high-priority churn.
> - The benefit is that a person's task continues within seconds, not
one hour, and the stale-sandbox Retry path is not reached.

## Linked Issues or Issue Description

Refs #21349

**What happened?**
A continuation queued after the tool-call loop limit waited for the
60-minute aging bound (`QUEUED_RUN_AGING_MS`) on an agent that had
person-requested runs queued. 4 of 7 loop continuations in one
restart-chaos full wave waited 60:00 to 60:28.

**Expected behavior**
A continuation of a person's task whose source run succeeded starts in
the same tier as that person's work. Failure retries stay in the
recovery tier.

**Steps to reproduce**
1. Queue a person's task on an agent with a concurrency limit of 1.
2. Let the run hit the tool-call loop limit, so the server queues an
`issue_continuation_needed` continuation with
`livenessContinuationReason: agent_tool_call_loop`.
3. Keep person-requested runs queued on the same agent.
4. The continuation waits until it is 60 minutes old.

**Paperclip version or commit**
`master` at 70a4351343.

## What Changed

- `queued-run-dispatch-order.ts`: add a `cleanContinuation` dispatch
fact. A clean continuation is not system recovery, so it takes the tier
of its task (person or work) and the normal readiness, priority and age
order. The module stays domain-agnostic: the wake reason is defined once
(`ISSUE_CONTINUATION_NEEDED_WAKE_REASON`, used in
`RECOVERY_WAKE_REASONS`).
- `queued-run-dispatch-order.ts`: one core predicate
(`countsAsRecoveryStart`) drives `isSystemRecoveryRun`, the in-pass
recovery counter, the place-in-line counter and the start-history
counter. Every start with a recovery wake reason that a person did not
ask for still uses up the recovery share. A clean continuation therefore
does not add recovery-reason starts.
- New `queued-run-provenance.ts`: the clean-continuation causes, the
source-run check and `QUEUED_RUN_ORDER_CONTEXT_KEYS`.
- `queue.ts` (`listQueuedRunCandidates`, the real dispatch query) and
`queued-run-line.ts` (the place-in-line query) both read
`QUEUED_RUN_ORDER_CONTEXT_KEYS`, so the dispatcher sees `retryReason`,
`livenessContinuationReason` and `livenessContinuationSourceRunId`.
- `queued-run-line.ts`: set `cleanContinuation` only when all of these
are true:
- the wake reason is `issue_continuation_needed` and
`livenessContinuationReason` is `agent_tool_call_loop`;
- the run has no `retryOfRunId`, `retryReason` or `previousRunId` (it is
not a failure retry);
- the source run (`livenessContinuationSourceRunId`, same company) has
status `succeeded`;
- the source run is not itself a loop continuation, so only the first
hop of a loop chain gets the faster tier.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/services/heartbeat/queued-run-line.test.ts
src/services/execution-projection-agent-queue.test.ts` passes (39
tests), plus `src/services/heartbeat/`,
`queued-run-dispatch-stacking.test.ts` and `recovery-wake-herd.test.ts`.
- Red before the fix, green after it:
- `starts a person's task continuing after the tool-call loop limit with
that person's work, not after the aging bound` (unit; failed with
`expected 'recovery' to be 'person'`)
- `counts a clean continuation's start against system recovery's share,
as the start history does` (unit)
- `starts a person's task continuing after the tool-call loop limit with
that person's work, and keeps failure recovery behind it` (embedded
Postgres; a failure retry, a chained loop continuation and a
continuation of a failed run all stay behind the person's work)
- Assembled heartbeat service
(`src/__tests__/heartbeat-clean-continuation-dispatch.test.ts`, embedded
Postgres, `resumeQueuedRunsForAgent` with one free slot), checks which
queued run actually starts:
- `starts a clean loop-limit continuation of a person's task before a
newer person assignment` (red without the `queue.ts` change, green with
it)
- `starts a newer person assignment before a failure retry of a person's
task` (guard)
- Guard that passes before and after: `keeps a failure retry of a
person's task in the recovery tier while people wait`.
- The `queuedRunDispatchLine` fuzz test now also generates clean
continuations and still matches the dispatcher order for 400 seeds.
- `pnpm verify` ran locally.

## Risks

- Low risk. The change is limited to dispatch order. No schema or API
change.
- A clean continuation can now start ahead of agent work. It is bounded:
one continuation per source run (existing idempotency key), only the
first hop of a loop chain, and only after a succeeded source run. A loop
chain's later hops and all failure retries stay in the recovery tier
with the existing share.
- Recovery-reason starts per hour do not increase. A clean continuation
start uses up recovery's share like before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code as an agent with tool use (shell, file edit, test runs). Extended
reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 05:24:53 -06:00
DottaandClaude Opus 5.5 440f96247b fix(cloud): resolve or name the remaining online move blockers (#22294)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A hosted server can move its durable data to shared storage while it
serves work, and then run without a volume
> - A bounded online pass moves the data, and a readiness check names
what still blocks the move
> - Some servers stay blocked after many passes, and the pass reports
only `skills:blocked` or `native_harness:source_not_successful`, or
reports nothing for run logs
> - Operators cannot see why, so they cannot tell recoverable data from
data that needs a maintenance step
> - This pull request recovers the cases that are safe to recover online
and gives every other case a precise code
> - The benefit is fewer blocked moves and a clear next step for each
one that stays blocked

## Linked Issues or Issue Description

Refs #19719

**What happened?**
A serving server that runs the online durable-data pass many times stays
ineligible. The pass reports `skills:blocked` without a reason,
`native_harness:source_not_successful` for several different causes, and
no reason at all for run logs that stay on the local disk.

**Expected behavior**
The pass moves every item that it can move safely while the server
serves work. For each item that stays, the pass reports a code that
names the cause.

**Steps to reproduce**
1. Install a catalog skill on a volume-backed server. Remove its
installed files and do not keep a saved version.
2. Keep a settled native run whose local log file has no final size or
digest and whose events do not prove full coverage.
3. Run the online durable-data pass.
4. See that the skill is reported only as `skills:blocked` and the run
log stays local with no reason in the pass result.

**Additional context**
The settled native log archive path from #19719 had no online caller.

## What Changed

- Skill installation conversion returns a `code` for each blocked
installation (for example `source_files_unavailable`,
`saved_version_incomplete`, `catalog_origin_unavailable`,
`legacy_source_unsafe`, `source_changed`). The pass reports
`skills:<code>`. An unknown cause stays `skills:blocked`.
- A spent budget during conversion now stops the page. Before, it was
reported as a blocked skill and the page moved past it.
- A catalog skill whose files are gone converts from the bundled catalog
package only when the catalog content hash equals the recorded origin
hash and the last audit recorded the installed bytes as equal to that
origin with no local changes.
- A modified catalog skill whose reset origin is gone gets its origin
version from the bundled catalog package only when the catalog content
hash equals the recorded origin hash.
- Native harness pre-copy replaces `source_not_successful` with
`source_run_active`, `source_run_unfinished`,
`source_coordinator_missing`, `source_coordinator_unsettled`,
`source_session_not_selected` and `source_lease_missing`. The
settled-source rules do not change.
- The run evidence worker archives a settled native log by run id when
the canonical import cannot prove event coverage. The existing
settled-source guard still decides. The worker records the reasons from
its last completed sweep, and the pass reports them as `run_logs:<code>`
and `provider_traces:<code>`.
- `migrateRunEvidence` accepts a `runId` filter.

## Verification

- `cd server && npx vitest run
src/__tests__/managed-skill-content.test.ts` (embedded PostgreSQL): new
catalog recovery, hash mismatch, unproven install and budget abort tests
fail before the change and pass after it.
- `npx vitest run
src/services/native-runtime/native-harness-online-preservation.integration.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
the new reason tests fail before the change and pass after it.
- `npx vitest run src/__tests__/run-evidence-migration-work.test.ts
src/services/native-log-archive.integration.test.ts`: the worker archive
test fails before the change and passes after it. The integration test
shows a real settled source that is blocked as
`native_log_source_coverage_unproven` and then archived by run id.
- `npx vitest run src/services/online-durable-data-migration.test.ts
src/services/online-durable-data-migration-postgres.test.ts`
- `pnpm --filter @paperclipai/server typecheck`

## Risks

- A settled native log with unproven coverage now moves to the read-only
historical archive instead of staying local. The archive keeps the exact
bytes. It does not claim canonical event coverage. This is the same
result that the offline path gives.
- The catalog fallback reads bundled or pinned catalog bytes. Each file
is checked against its pinned SHA-256, and the package must match the
recorded origin hash.
- Pass deferral codes change. Readers that match
`native_harness:source_not_successful` must use the new codes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:54:10 -06:00