Commit Graph
2618 Commits
Author SHA1 Message Date
Nicky Leach 323504e41e perf(runner): enable TCP_NODELAY for loopback PRP (#22091)
## Thinking Path

> - Paperclip manages AI agents for work.
> - Native Runner exchanges commands and events over WebSocket PRP.
> - Both Runner connection modes use TCP sockets.
> - Several small frames can follow each other during a start.
> - Loopback peers can receive those frames promptly with TCP_NODELAY.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Delivery of PRP command results and events on local TCP connections.

**Subsystem affected**

Native Runner PRP transport.

**Current behavior**

Runner uses the default TCP setting for acknowledged small writes on its
accepted and outbound PRP sockets. In an A/B test with a loopback model
fixture, enabling TCP_NODELAY on the accepted Runner socket reduced
median time to observe turn.started by 29 to 52 ms, depending on the
provider and session state.

**Proposed behavior**

Enable TCP_NODELAY on accepted and outbound PRP sockets before the
WebSocket handshake when the connected peer address is loopback.

**Reason and benefit**

The controller receives turn-start events sooner. The outbound path
shares the same setup rule and has socket-option coverage.

**Breaking changes**

The change is a per-connection socket option. Existing launch arguments,
durable checkpoints, and PRP frames carry forward.

## What Changed

- Share initial stream configuration between the listener and dialer.
- Check the actual peer address before enabling TCP_NODELAY.
- Complete a real WebSocket handshake in one test for each connection
mode and read back the option.

## Verification

- `cargo fmt --manifest-path packages/paperclip-runner/runner/Cargo.toml
--all --check`
- `cargo test --offline --locked -p paperclip-runner-core --lib
durable::transport::tests` passed locally on macOS (27 passed).
- Both new tests failed on the original socket setup and passed with the
change.
- `cargo test --offline --locked -p paperclip-runner-core --lib --bins
--test listener_accept` passed locally on macOS (676 passed, 4 ignored).

## Risks

- TCP can send more small packets during high-volume local output. The
check limits the option to loopback peers. The earlier A/B measured
turn-start timing on the listener path.

## Model Used

- OpenAI GPT-6 (Codex), with source inspection and local Rust tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g., `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-17 23:14:48 +00:00
DottaandClaude Opus 5.5 f295e496db fix(daytona): retry lease acquisition after a provider connection failure (#22087)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs on Daytona start with a lease acquisition in the Daytona
plugin
> - When the host cannot reach Daytona, the Daytona SDK raises a
connection error that has no HTTP status and no errno
> - The host only retries a lease failure that carries structured
evidence, so this failure became `setup_failed` and the task waited for
a person to press Retry
> - This pull request makes the plugin report the SDK connection error
as a closed transport code that the host already retries
> - The benefit is that a short network blip during sandbox startup no
longer needs a Retry click

## Linked Issues or Issue Description

**What happened?**

In a restart-chaos wave (201 tasks, 6 graceful release swaps), the host
could not resolve or reach Daytona for about 12 minutes. One task needed
a Retry click. Its only run failed with `setup_failed`
(`lease_acquire_failed`) and the message `Failed to acquire lease for
environment "Daytona warm ..." (sandbox): read EADDRNOTAVAIL`. Automatic
recovery did not continue the task because `setup_failed` is not a
transient setup code. The task stayed failed for about 100 minutes until
the grader pressed Retry.

**Expected behavior**

A lease acquisition that fails because the provider could not be reached
is retried automatically with the bounded bootstrap budget, as
`environment_provider_unavailable`. A person never needs to press Retry
for a network blip.

**Steps to reproduce**

1. Use the Daytona provider.
2. Make a Daytona API call during lease setup fail at the transport
level (no HTTP response), for example with `read EADDRNOTAVAIL` or
`getaddrinfo ENOTFOUND`.
3. See that the run fails with `setup_failed` and the task shows Retry.

**Paperclip version or commit**

master `ca34aca777` (wave) and `02ee538be9`.

Root cause: the pinned `@daytonaio/sdk` converts an Axios transport
error into `DaytonaConnectionError` (or
`DaytonaConnectionTimeoutError`). It keeps neither the errno nor the
cause. The worker RPC therefore sent no provider diagnostic, and
`readTransientLeaseAcquisitionFailure` returned `null`.

## What Changed

- The Daytona plugin wraps a `DaytonaConnectionError` from lease
acquisition in an error with code `CONNECTION_FAILED`, and a
`DaytonaConnectionTimeoutError` with code `ETIMEDOUT`. The message is
unchanged. It reads the SDK class name only, never message text.
- The plugin SDK closed diagnostic set accepts `CONNECTION_FAILED`, so
the code crosses the worker RPC.
- The host treats `CONNECTION_FAILED` as a transient connection failure.
The run becomes `environment_provider_unavailable` and uses the existing
bounded bootstrap retry (30 s, 60 s) and the automatic-run brake.
- Refreshed `tests/runner-e2e/source-sdk-public-inputs.json` for the
changed SDK file.

## Verification

- Red/green: with the source change removed, the new plugin tests
(`reports a DaytonaConnectionError during setup as a transient transport
failure`, timeout variant) and the new host tests (`treats errno
CONNECTION_FAILED as transient`, `retries a provider connection failure
reported across the worker RPC`) fail. With the change, they pass.
- `cd packages/plugins/sandbox-providers/daytona && npx vitest run` (637
passed).
- `npx vitest run
server/src/services/environment-provider-unavailable-error.test.ts
packages/plugins/sdk/tests/environment-acquire-error.test.ts` (66
passed).
- `pnpm typecheck` in the Daytona plugin passes.
- `pnpm verify` asks for broad CI coverage because the SDK input
registry changed. CI supplies that coverage.

## Risks

- Low risk. A wrong Daytona API URL also produces
`DaytonaConnectionError`. It now gets two bounded retries (about 90 s)
before the same Retry state as before.
- The retries count toward the automatic-run brake, so a long outage
cannot multiply run starts across the company.
- A create that timed out on the client keeps its existing
creation-cleanup envelope. The retry is a new creation attempt with its
own identity.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:55:47 -06:00
Devin FoleyandPaperclip eaf6461cee fix(server): give a slow remote more time to exclude agent files from Git (#22039)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote run starts, Paperclip runs a short Node program in
the sandbox that keeps the reserved runtime directory out of task Git
staging
> - That program had one 15 second attempt. A freshly created sandbox is
sometimes slow, so the attempt timed out
> - The run then failed before any provider work started, with the
generic `adapter_failed` code. Nothing retried it, and the wake that
started the run was lost
> - This pull request gives each attempt 30 seconds and retries a
timed-out attempt, up to three attempts
> - The benefit is that a slow sandbox no longer stops a task, while a
real failure still fails at once and names its reason

## Linked Issues or Issue Description

Builds on #21499 (merged), which adds the closed `reason=` code to this
error.

**What happened?**
A run failed during preparation with "Could not exclude agent files from
task Git staging". The sandbox lease was acquired about 50 seconds
before the agent-file copy was registered (3 to 7 seconds on the same
stack's earlier runs). The copy failed 20 seconds after it was
registered, which matches the 15 second limit. Earlier failures with
this message all took at least 16 seconds before failing, so they also
match a timeout rather than an error from the program itself.

**Steps to reproduce**
1. Start a remote run on a sandbox that takes more than 15 seconds to
run a short command.
2. The run fails in the preparing stage and is not retried.

**Expected behavior**
A slow sandbox gets more time, and a timed-out attempt is retried.

**Actual behavior**
One 15 second attempt, then the run fails.

## What Changed

- `agent-instruction-files.ts`: add `excludeRuntimeFromRemoteGit`. Each
attempt gets 30 seconds. A timed-out attempt is retried, up to three
attempts. Any other failure throws at once. The error keeps the closed
`reason=` code from #21499.
- `agent-directory-working-copies.ts` and
`agent-instruction-working-copies.ts`: both remote call sites use the
helper, so the private instruction copy also names a reason.
- `execution-target.ts`: `runSshCommand` rejects a command its timeout
killed with `code: null` and `killed: true`, but only `ETIMEDOUT` was
treated as a timeout, so an SSH timeout was thrown instead of returned.
A killed command now returns `timedOut: true`, as the sandbox path
already does.
- Tests: a timeout followed by success stages the agent files; three
timeouts fail with `reason=timed_out`; a non-timeout failure is
attempted once. An adapter-utils test covers the real SSH timeout
rejection.

## Verification

- `npx vitest run` on the agent-directory, instruction working-copy, Git
exclusion, copy retry backoff, instruction-working-copy and remote
checkpoint lock suites: 217/217 pass.
- adapter-utils `execution-target` and sandbox suites: 173/173 pass.
adapter-utils SSH suites: 65 pass, 2 skipped. Server adapter registry
suite: 67/67 pass.
- The new cases fail without this change.
- `npx tsc --noEmit` passes for server and adapter-utils.

## Risks

- Low. A sandbox that never answers now takes up to 90 seconds to fail
instead of 15 seconds.
- An SSH shell command that times out is now returned as `timedOut`
instead of thrown. Every caller that checks the result already treats
`timedOut` as a failure. The home-directory probe and the empty-file
helper ignore the result, so for them an SSH timeout now behaves as a
sandbox timeout already did.
- The exclusion program is idempotent: it writes its file through a
temporary name and a rename, so a retry after a late first attempt
leaves the same result.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 14:28:27 -08:00
Nicky LeachandPaperclip f05b90508a refactor(core): remove retired compatibility paths (#22055)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core manages chat authorization, AI connection defaults, navigation,
and the plugin host.
> - Core still writes old default rows and checks an Agent Chat flag
that always reads true.
> - It also carries a desktop rail that cannot open and private aliases
with canonical equivalents.
> - This pull request removes those paths and moves their tests onto the
paths Core uses.
> - The result is fewer Core database operations and less code to
maintain.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Core AI default writes, Agent Chat execution and delivery, private
plugin host and Runner setup, and board navigation.

**Subsystem affected**

Cross-cutting. Core server, board UI, private Runner helpers, and a
repository guard script.

**Current behavior**

The server writes provider-scoped and per-method AI defaults, then reads
only the provider-scoped rows. It reads settings to decide whether to
block Agent Chat, although every settings view enables it. The desktop
sidebar renders expanded while still tracking requests to open and close
a retired icon rail. The agent client retries conflicts that the server
already resolves when there is exactly one active match.

**Proposed behavior**

Write provider-scoped defaults directly. Use the existing Agent Chat
authorization and retry checks without the redundant settings reads.
Render the current sidebar directly and let the API report true agent
reference conflicts.

**Reason and benefit**

A personal non-custom connection save or explicit default selection
sends one fewer SQL statement. It also avoids the provider-default write
that the legacy trigger performed when the old statement inserted or
updated a row. A successful task-to-chat send or receipt replay avoids
two transaction settings SELECTs when the singleton row already exists.
Conversation comment, wake release, and retry paths also avoid a
transaction settings read when they previously entered the gate. Savings
in other settings paths depend on cache hits. Core-to-plugin RPC and
plugin-to-Launcher HTTP traffic stay the same.

**Breaking changes**

The public plugin SDK, host navigation, and organization-switcher fields
retain their existing interfaces and behavior. The settings API still
accepts and reports the Agent Chat compatibility key. This change needs
no schema migration or rollout ordering.

**Additional context**

Refs #10610 for prior reporting on the retired sidebar rail. Refs #21756
for adjacent Coordinator plugin slots, which remain a separate change.

## What Changed

- Stop writing per-method AI defaults. Preserve the provider-scope
upsert and first-account behavior. Keep the existing tables and
migration trigger.
- Remove Agent Chat checks that read a constant true value. Keep
conversation ownership, task scope, session, and request-retry checks.
- Remove unused lock-owner fields and private plugin lifecycle, secret
extraction, Gmail connector, PRP setup, heartbeat resolver, and ACPX
aliases.
- Use a single API request for agent references. Remove retired sidebar
rail state, hooks, and branches across the active and production
snapshot layouts. Preserve desktop resizing, mobile drawer controls,
contextual navigation, and public plugin slot inputs.
- Remove unused FeedCard and AgentDetail inputs and the test-only Git
push matcher alias.
- Keep the cleanup that erases historical plaintext chat retries from
browser storage.

Production source in the touched files is 106,800 → 105,633 nonblank
lines, down 1,167. The production Git diff adds 251 physical lines and
removes 1,480, down 1,229. Those figures exclude tests, fixtures, docs,
and Storybook. The entire Git diff adds 482 physical lines and removes
2,325, down 1,843.

## Verification

- Merged master at `b56f240f45` into the PR branch as `f73fea4009`. The
five sidebar files retain the expanded navigation cleanup and master’s
explicit session-replay masking contract. The merge also preserves
master’s account-repair recovery guard and GitHub star-prompt behavior.
- Focused UI Vitest run passed 361 tests across 11 files. This covers
analytics privacy and replay masking, sidebar and layout, GitHub star
prompt, and agent detail.
- Focused server Vitest run passed 45 tests across three files. This
covers restored configuration handling, conversation recovery, and Agent
Chat authority and receipt replay.
- `pnpm --filter @paperclipai/ui typecheck`, the Git push guard and its
14 tests, and `git diff --check origin/master...HEAD` passed.
- `pnpm verify --base origin/master` classified the Git-push guard as a
global CI input and returned exit 2. GitHub CI supplied the broader
coverage.
- GitHub CI and Runner image checks passed on `f73fea4009`. Both
required checks, `ci / verify` and `ci / e2e`, passed. [CI
workflow](https://github.com/paperclipai/paperclip/actions/runs/44181261752)
· [Runner image
workflow](https://github.com/paperclipai/paperclip/actions/runs/44181261413)
- Standard Greptile reported 5/5 with no actionable findings or open
review threads. [Greptile
review](https://github.com/paperclipai/paperclip/pull/22055#issuecomment-6959078925)
- The optional Superagent Security Scan finished neutral with an
inconclusive workflow result and zero annotations. The final rollup had
61 successes, five skips, and one neutral, with no failed or pending
checks. [PR
checks](https://github.com/paperclipai/paperclip/pull/22055/checks)

## Risks

Agent Chat authorization and retry are sensitive to small query changes.
The final diff keeps the conversation agent field and all existing
authorization and idempotency checks. The tests cover revoked defaults,
legacy migration reads, bound-listener authority, and mobile and desktop
navigation. Installations that still run software old enough to read the
per-method AI table will no longer see new defaults in that table.
Current Core reads the provider-scoped table.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with source inspection and
shell-based tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 14:00:14 -08:00
DottaandClaude Opus 5.5 23ce68a8dc feat(connections): end Google access when a Paperclip-managed Google connection is removed (#22065)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Connectors let agents use Google Workspace through a Google sign-in
that Paperclip Cloud manages for the user
> - When a user removes a Paperclip-managed Google connection, Paperclip
deletes the tokens locally, but the grant stays live at Google
> - Google's OAuth review expects that a disconnect ends the app's
access, and users expect it too
> - Google's revoke ends the user's whole grant to the shared client, so
only the Paperclip Cloud broker can decide when it is safe to revoke
> - This pull request asks the broker to revoke each removed grant,
keeps local removal unconditional, and tells the user the outcome in
plain language
> - The benefit is that removal ends Google access when no other
Paperclip connection needs it, and the user knows what to do when it did
not

## Linked Issues or Issue Description

**Subsystem affected**
Connectors: Paperclip-managed Google Workspace and the legacy
per-service Google connections (server removal, shared contracts,
connection UI).

**Problem or motivation**
Removing a Paperclip-managed Google connection or grant was local only.
The refresh token was deleted in Paperclip, but Google still listed
Paperclip as having access to the account. The docs said so ("Removal is
local for managed grants").

**Proposed solution**
On removal, send each removed grant's refresh token to the broker's
signed `POST /v1/connector/revoke`. The broker revokes at Google only
when no other live binding could share the grant. Paperclip records the
broker's outcome code and shows one of three plain messages. Local
removal never waits on, or fails because of, that call.

**Alternatives considered**
Call Google's revoke endpoint from the app. This ends the user's whole
grant to the shared client and can break their other Google connections,
so the broker owns the decision.

**Additional context**
Older brokers answer 409 `REVOCATION_NOT_AVAILABLE`. The app treats
that, a timeout, and a network error as "could not end access at Google"
and still removes the connection.

## What Changed

- `paperclip-cloud-connector.ts`: `revoke()` takes an optional
`timeoutMs`, returns the broker's `outcome` from an allowlist, and keeps
`REVOCATION_NOT_AVAILABLE` as an allowed rejection reason.
- New `tool-access/connections/google-access-removal.ts`. For a managed
Google connection, `prepare()` reads each removed grant's refresh token
and returns a `send()` step. `send()` calls the broker for each distinct
token, one after another, under one 5-second deadline. Every result and
every error (including a Paperclip Cloud client that cannot be created)
becomes an outcome code. It never rejects and never logs a token. An
unexpected error is logged with the error name only and reported as
`unexpected_error`.
- Order for removing a whole connection: read the tokens, run the whole
local teardown (grants, installs, app profile, gateway tokens, runtime
slots, catalog, OAuth state, archive the connection), call the broker,
delete the sent refresh-token secrets whatever the outcome, then delete
the remaining secrets. A failure anywhere in the teardown makes no
broker call. Order for removing one grant: read the token, commit the
revocation, call the broker, delete the grant's refresh-token secret.
`send()` owns the call and the deletion through an injected
secret-removal function.
- Owed revocations: a revoked managed Google grant whose refresh-token
secret still exists (and that nothing live uses) is sent by every later
removal, then the secret is deleted. This covers a process stop after
the teardown and before the call, with no new column. Once the secret is
deleted, nothing is sent again.
- Shared tokens: the shared (organization) sign-in writes the same
secret refs to the organization grant and the connection. So revoking an
organization grant alone neither calls the broker nor deletes the
secret, because the connection still uses that token; removing the
connection does both. The same rule applies to any refresh token still
used by another live grant or connection.
- Audit history: once the broker call settles, a
`connection.google_access_removal` audit event records `reasonCode` =
result and `providerAccess` (plus `grantId` for a single grant). If that
write fails, the removal still succeeds and the failure is logged.
- Revoke signs with the subject that consented to the grant
(`providerTenant.externalId`, saved by the managed callback from the
claim). Grants saved without it fall back to the refresh rule. Refresh
is unchanged; see Risks.
- `removal.ts` and `grants.ts`: `providerAccess: { provider, result,
outcomes }` goes to the responses and to the `tool_connection.archived`
and `tool_connection.grant_revoked` activity entries. The tool access
audit field `providerRevocation` keeps its Vercel meaning only.
- `packages/shared`: `ManagedGoogleAccessRemoval`,
`ManagedGoogleRevocationOutcome`, `ConnectionGrantRevocationResult`.
- UI: one helper, `connectionRemovalToast`, builds the removal toast
(base sentence plus the Google notice, tone and action) for Connectors,
Connections and the app page. No codes are shown.
- API: `docs/api/tool-connections.md` documents both DELETE responses
and `providerAccess`. The OpenAPI registration describes the optional
field and its four results.
- Docs: `GOOGLE-WORKSPACE.md` (order, owed revocations, decision table),
`GMAIL.md`, the Google connector memory note, and the help page on
deleting connection data.

Decision table, judged over the distinct tokens sent in this removal:

| Outcomes | `result` | User message |
| --- | --- | --- |
| Every token `revoked` | `ended` | "Paperclip's access to this Google
account was also ended at Google." |
| Any failure-type code: `provider_revocation_failed`,
`binding_not_known` (the broker had no record, so Google was not
called), `provider_revocation_disabled`, `revocation_in_progress`,
`revocation_not_available`, `broker_unavailable`, `broker_error`,
`credential_unavailable`, `unexpected_error` | `not_ended` | "Paperclip
removed the connection but couldn't end its access at Google. You can
remove Paperclip from your Google account's third-party connections." +
"Open Google account connections" |
| Only `other_connections_remain` | `other_connections_remain` | "Other
Paperclip connections still use this Google account, so Google keeps
Paperclip's access until you remove those too." |
| At least one `revoked`, the rest `other_connections_remain` |
`ended_unconfirmed` | "Paperclip's access was ended at Google. If more
than one Google account was connected here, check your Google account's
third-party connections to be sure." + "Open Google account connections"
|

Grants record no Google account identity: the managed callback saves
`externalId` (the Paperclip subject), `name` (the provider name), and
OAuth timing and scopes, and none of the 14 scopes returns an account
identifier. So the last row is always `ended_unconfirmed`. This includes
two tokens of one user and several agent grants on one Google account.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/google-managed-connection-removal.test.ts
src/services/tool-access
src/services/paperclip-cloud-connector-revoke.test.ts
src/services/paperclip-cloud-connector.test.ts
src/__tests__/tool-connection-removal.test.ts
src/__tests__/tool-access-service.test.ts
src/__tests__/openapi-routes.test.ts` (714 passed)
- `pnpm --filter @paperclipai/ui exec vitest run
src/features/connections/googleAccessRemoval.test.ts src/pages/apps`
(passed)
- Server test typecheck (`node server/scripts/typecheck-layers.mjs`)
passes.
- Red then green: the server tests failed on `master` sources (21), then
each review round's new tests failed on the previous head and pass now.
They cover: the order on both paths, local failure before the grant
commit making no call and the retry sending, cleanup failure after the
call making no second call, a process stop between commit and call being
sent on retry with the secret deleted after, a Paperclip Cloud client
that cannot be created, the consenting subject for an organization
grant, sequential sends under one deadline, each row of the decision
table (per-agent and same-user cases give `ended_unconfirmed`, a mixed
failure gives `not_ended`), the unexpected-error log without a token, no
token in any log call, the delete-stored-data option, and a toast test
per surface.
- Added time, measured locally with a stubbed broker that answers at
once: median 22.1 ms against 20.3 ms without the broker step (12 runs
each), about 2 ms. With a real broker, one round trip per removed token,
capped at 5 seconds in total.
- `pnpm verify` exits 2 (a global CI input changed), so required CI
supplies the broad coverage.
- Not verified against a live broker or a real Google account.

## Risks

- A removal of a managed Google connection or grant can take up to 5
seconds longer when the broker is slow.
- Refresh subject is not changed in this PR. In the broker
(`recordProviderGrant`), a ledger row's subject is fixed when the row is
first written, and a refresh under another subject leaves that row
untouched. So moving refresh to a new subject could stop updating the
row's last-seen time. After 200 days the row would then count as not
live, and a later removal elsewhere could revoke at Google while this
connection still works. That risk cannot be ruled out from code for rows
written before this release, so both refresh paths keep their old rules.
The two refresh paths already use different subject orders today: the
gateway prefers `connectorSubjectAgentId` and then
`connectorSubjectUserId`, while background refresh prefers the grant's
own subject. That is pre-existing, and this PR leaves it unchanged.
Revoke uses the consenting subject, which matches the claim-time ledger
row. When the row was first written by a refresh under a different
subject, the broker answers `binding_not_known` and the user gets the
warning.
- Repeated removals can send an already-removed token again; the broker
returns its recorded outcome and changes nothing at Google.
- Against an older broker, users see the "could not end access at
Google" message, which is accurate.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:52:20 -06:00
dependabot[bot] 24728076d6 build(deps-dev): bump wrangler from 4.156.2 to 4.158.0 in /scripts/typecheck-sandbox-providers (#20200)
Bumps
[wrangler](https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler)
from 4.156.2 to 4.158.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/cloudflare/workers-sdk/releases">wrangler's
releases</a>.</em></p>
<blockquote>
<h2>wrangler@4.158.0</h2>
<h3>Minor Changes</h3>
<ul>
<li>
<p><a
href="https://redirect.github.com/cloudflare/workers-sdk/pull/16186">#16186</a>
<a
href="https://github.com/cloudflare/workers-sdk/commit/f1d81a20c8e535ec7cb870143283f4a683fec70d"><code>f1d81a2</code></a>
Thanks <a
href="https://github.com/leostera"><code>@​leostera</code></a>! -
Support local Artifacts across Wrangler, Vite, and Workers Vitest</p>
<p>Set <code>remote: false</code> on an Artifacts binding to opt into
the Miniflare simulator and persistent local repositories during
development. Omitted <code>remote</code> continues to use the deployed
resource, with a warning that changes affect real repositories;
<code>remote: true</code> explicitly selects it. When remote bindings
are disabled (<code>wrangler dev --local</code>, or Vite's
<code>remoteBindings: false</code>), Artifacts bindings use the local
simulator. Workers Vitest uses the local simulator when
<code>remote</code> is omitted, so tests do not change real repositories
by accident; set <code>remote: true</code> to test against the deployed
resource. Local development does not require Cloudflare credentials, but
does require Git 2.32 or newer on the host.</p>
<p>In Miniflare's own options, Artifacts bindings route like every other
binding: only <code>dev.remote: true</code> with a remote proxy
connection uses the deployed resource. Wrangler and Vite resolve the
deployed-by-default rule, and Workers Vitest its local default, before
passing options to Miniflare.</p>
</li>
</ul>
<h3>Patch Changes</h3>
<ul>
<li>
<p><a
href="https://redirect.github.com/cloudflare/workers-sdk/pull/16700">#16700</a>
<a
href="https://github.com/cloudflare/workers-sdk/commit/dc35efbd3cdfe67b3ef972c75b8faae65dfed195"><code>dc35efb</code></a>
Thanks <a href="https://github.com/cjol"><code>@​cjol</code></a>! - Bump
bundled <code>brace-expansion</code> to pick up security fixes</p>
<p>These packages bundle <code>brace-expansion</code> (a dependency of
<code>minimatch</code>), and the bundled versions were affected by
published advisories. The bundles now include
<code>brace-expansion</code> 2.1.7 (from 2.0.1) and 5.0.12 (from 5.0.5).
Normal glob patterns expand as before; only the pathological patterns
targeted by the advisories are handled differently.</p>
</li>
<li>
<p><a
href="https://redirect.github.com/cloudflare/workers-sdk/pull/16700">#16700</a>
<a
href="https://github.com/cloudflare/workers-sdk/commit/dc35efbd3cdfe67b3ef972c75b8faae65dfed195"><code>dc35efb</code></a>
Thanks <a href="https://github.com/cjol"><code>@​cjol</code></a>! - Bump
bundled <code>fast-xml-parser</code> from 4.4.1 to 4.5.5 to pick up
security fixes</p>
<p>Miniflare bundles <code>fast-xml-parser</code> into its local R2 S3
API, and Wrangler bundles it through the AWS SDK used by <code>wrangler
pipelines</code>. Version 4.4.1 is affected by several published
advisories, and 4.5.5 includes their fixes. XML handling does not
change.</p>
</li>
<li>
<p><a
href="https://redirect.github.com/cloudflare/workers-sdk/pull/16700">#16700</a>
<a
href="https://github.com/cloudflare/workers-sdk/commit/dc35efbd3cdfe67b3ef972c75b8faae65dfed195"><code>dc35efb</code></a>
Thanks <a href="https://github.com/cjol"><code>@​cjol</code></a>! - Bump
bundled <code>minimatch</code> from 5.1.6 to 5.1.9 to pick up security
fixes</p>
<p>These packages bundle <code>minimatch</code>, and 5.1.6 is affected
by published ReDoS advisories. Version 5.1.9 includes the fixes. 5.1.8
is skipped because it broke recursive <code>**</code> matching, which
D1's <code>migrations_pattern</code> relies on.</p>
</li>
<li>
<p><a
href="https://redirect.github.com/cloudflare/workers-sdk/pull/16700">#16700</a>
<a
href="https://github.com/cloudflare/workers-sdk/commit/dc35efbd3cdfe67b3ef972c75b8faae65dfed195"><code>dc35efb</code></a>
Thanks <a href="https://github.com/cjol"><code>@​cjol</code></a>! - Bump
bundled <code>shell-quote</code> from 1.10.0 to 1.11.0 to pick up a
security fix</p>
<p><code>shell-quote</code> is bundled into Wrangler, which uses it to
parse and quote shell commands such as custom build and Pages dev
commands. Version 1.10.0 is affected by a published advisory, and 1.11.0
includes the fix. Command parsing does not change.</p>
</li>
<li>
<p><a
href="https://redirect.github.com/cloudflare/workers-sdk/pull/16700">#16700</a>
<a
href="https://github.com/cloudflare/workers-sdk/commit/dc35efbd3cdfe67b3ef972c75b8faae65dfed195"><code>dc35efb</code></a>
Thanks <a href="https://github.com/cjol"><code>@​cjol</code></a>! - Bump
bundled <code>yaml</code> from 2.8.1 to 2.9.0 to pick up a security
fix</p>
<p><code>yaml</code> is bundled into Wrangler, which uses it to show
container application details. Version 2.8.1 is affected by a published
advisory, and versions from 2.8.3 onwards include the fix. Output does
not change.</p>
</li>
<li>
<p><a
href="https://redirect.github.com/cloudflare/workers-sdk/pull/16688">#16688</a>
<a
href="https://github.com/cloudflare/workers-sdk/commit/fa3edb325694670084bc38af02232957315ccdb5"><code>fa3edb3</code></a>
Thanks <a
href="https://github.com/FlorentCollin"><code>@​FlorentCollin</code></a>!
- Use the <code>account_id</code> from the configuration file in
<code>wrangler d1 migrations list --remote</code></p>
<p>Before this change, the command ignored the <code>account_id</code>
in the configuration file. When your login had access to more than one
account, the command failed in non-interactive mode. In interactive
mode, it asked you to select an account. The command now uses the
configured <code>account_id</code>, as <code>wrangler d1 migrations
apply</code> does.</p>
</li>
<li>
<p><a
href="https://redirect.github.com/cloudflare/workers-sdk/pull/16695">#16695</a>
<a
href="https://github.com/cloudflare/workers-sdk/commit/b2061f40f812d8888b461f68506552b58f8e468b"><code>b2061f4</code></a>
Thanks <a
href="https://github.com/apps/dependabot"><code>@​dependabot</code></a>!
- Update <code>DEFAULT_COMPAT_DATE</code> to 2026-11-04</p>
<p>Use the bundled workerd release's compatibility date by default when
scaffolding new projects or when no compatibility date is
configured.</p>
</li>
<li>
<p><a
href="https://redirect.github.com/cloudflare/workers-sdk/pull/16695">#16695</a>
<a
href="https://github.com/cloudflare/workers-sdk/commit/b2061f40f812d8888b461f68506552b58f8e468b"><code>b2061f4</code></a>
Thanks <a
href="https://github.com/apps/dependabot"><code>@​dependabot</code></a>!
- Update dependencies of &quot;miniflare&quot;, &quot;wrangler&quot;</p>
<p>The following dependency versions have been updated:</p>
<table>
<thead>
<tr>
<th>Dependency</th>
<th>From</th>
<th>To</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>@​cloudflare/workers-types</code></td>
<td>^5.20261103.1</td>
<td>^5.20261104.1</td>
</tr>
<tr>
<td>workerd</td>
<td>1.20261103.1</td>
<td>1.20261104.1</td>
</tr>
</tbody>
</table>
</li>
<li>
<p><a
href="https://redirect.github.com/cloudflare/workers-sdk/pull/16306">#16306</a>
<a
href="https://github.com/cloudflare/workers-sdk/commit/36e64f20025cf0bd5e99a585ca3427c8d75fd18b"><code>36e64f2</code></a>
Thanks <a href="https://github.com/efalcao"><code>@​efalcao</code></a>!
- Report Cloudflare Access login redirects on remote bindings instead of
looping until &quot;Too many redirects&quot;</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/c579d500d0c5732393f2e85f0cdd085167aee32e"><code>c579d50</code></a>
Version Packages (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16681">#16681</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/f1d81a20c8e535ec7cb870143283f4a683fec70d"><code>f1d81a2</code></a>
[miniflare] Enable local Artifacts in development tools [3/3] (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16186">#16186</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/dc35efbd3cdfe67b3ef972c75b8faae65dfed195"><code>dc35efb</code></a>
Bump dependencies to clear 101 Dependabot alerts (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16700">#16700</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/4d88a45a952542c55ec63bbca671fb1f003a09c7"><code>4d88a45</code></a>
[miniflare] Stabilize Artifacts integration tests on Windows (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16693">#16693</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/0083f807de557d771f9d2d4f08d6406f5062c295"><code>0083f80</code></a>
[wrangler] Harden the local KV pagination regression test (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16697">#16697</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/fa3edb325694670084bc38af02232957315ccdb5"><code>fa3edb3</code></a>
[wrangler] Use config account_id in d1 migrations list --remote (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16688">#16688</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/9a0732c3f4f6cbbc016a7134fc1d20edb98b454e"><code>9a0732c</code></a>
Version Packages (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16654">#16654</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/44d1ee439c0a077f2fed4d86e35a4edf46b51f74"><code>44d1ee4</code></a>
fix(wrangler): stop <code>wrangler delete</code> orphaning Container
applications (<a
href="https://github.com/cloudflare/workers-sdk/tree/HEAD/packages/wrangler/issues/16011">#16011</a>)</li>
<li><a
href="https://github.com/cloudflare/workers-sdk/commit/8ac43df32667a77cd7e8e267da86e96645e3031b"><code>8ac43df</code></a>
Require explicit route metadata flags when downloading Worker
configuration (...</li>
<li>See full diff in <a
href="https://github.com/cloudflare/workers-sdk/commits/wrangler@4.158.0/packages/wrangler">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=wrangler&package-manager=npm_and_yarn&previous-version=4.156.2&new-version=4.158.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-17 13:51:44 -08:00
DottaandClaude Opus 5.5 73dc33c620 fix(runner): keep a disk reserve so a full sandbox disk does not strand the runner (#22068)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a Paperclip-managed sandbox run under the Paperclip runner
(`runnerd`), which keeps durable state on the sandbox disk
> - An agent can fill that disk with its own work. In one case the agent
installed a Rust toolchain and a second checkout with dependencies on a
10 GiB sandbox
> - `runnerd` then could not write its state and exited. Each relaunch
tried to upload its launch files to the full disk and failed, 83 times
in 7 minutes
> - The run ended as a generic provider failure that asked a person to
inspect it, and Retry repeated the same failure
> - This pull request keeps a small disk reserve that belongs to
Paperclip. Recovery releases it to relaunch the runner, and stops at
once with a named code when nothing is left to free
> - The benefit is that the agent's turn can continue and see the full
disk in its own tool output, instead of the run dying with it

## Linked Issues or Issue Description

**What happened?**
An agent in a managed Daytona sandbox filled the 10 GiB disk. `runnerd`
exited with `failed to persist bounded diagnostic: No space left on
device (os error 28)`. Runner process recovery then retried for the full
420000 ms reconnect grace. Every attempt failed with
`.paperclip-upload-<id>.tar.gz: create: ... no space left on device`.
The run failed as `native_provider_terminal_failed`, and the recovery
asked a board user to verify that the provider stopped.

**Steps to reproduce**
1. Run an agent in a managed sandbox and have it fill the disk (for
example a Rust toolchain plus a full dependency install).
2. `runnerd` exits when it next writes its state.
3. Watch recovery retry the relaunch upload until the grace expires.

**Expected behavior**
Paperclip relaunches its runner on a disk the agent filled, so the turn
continues and the agent sees the full disk in its tools. When the runner
truly cannot start, recovery says so at once with a named cause instead
of retrying for 7 minutes.

**Environment**
Managed Daytona sandbox (fleet snapshot, 4 CPU, 8 GiB memory, 10 GiB
disk), `paperclip_runner` native runtime.

## What Changed

- New `server/src/services/native-runtime/remote-runner-disk-reserve.ts`
owns the reserve: hold/release scripts, the full-disk classifier, and
`createRemoteRunnerDiskReserve`. A release waits for an in-flight hold,
and no hold starts after a release.
- Both remote runner launchers (agent runs and Dot) get a reserve path
in managed sandboxes only:
`<workspace>/.paperclip-runtime/paperclip-runner/disk-reserve`.
Workspace saves and Git already exclude `.paperclip-runtime`. SSH hosts
get no reserve.
- After the first launch's identity is confirmed, the launcher holds 256
MiB in the background, only while 2 GiB more is free. It is off the
startup path.
- A relaunch releases the reserve before it stages. A launch that hits a
full disk while staging releases the reserve and stages once more.
- Release tries twice and reports `released`, `not_held` or `failed`.
Only `not_held` ends the launch with `sandbox_disk_full`. A release that
cannot run rethrows the original error, so recovery still retries.
- Command-based staging (no `syncIn`) keeps a full disk recognizable
with an `ENOSPC` code instead of a bare
`runner_remote_*_staging_failed`.
- `SANDBOX_DISK_FULL_FAILURE_CODE` and `isSandboxDiskFullFailure` live
in the runner package. Runner process recovery checks the error code and
stops at once instead of retrying for the whole grace. Other staging
failures still retry.

## Verification

- Red before the fix, green after:
-
`server/src/services/native-runtime/remote-runner-disk-reserve.test.ts`:
release-and-restage, release before relaunch, `sandbox_disk_full` when
release frees too little, no retry for a failure that is not a full
disk, hold/release scripts in a real shell, no hold below the free-space
floor. Before the fix, the launch rejected with the exact upload error
from the incident.
- `packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts`
"fails at once when a replacement cannot be staged because the sandbox
disk is full": before the fix it was still retrying after 4 s; after, it
fails with `sandbox_disk_full` after one replacement.
- Kept: "retries a replacement whose staging failed before any process
launched" still passes, so a raw transient staging failure still
retries.
- Updated: "confirms no replacement process exists when staging fails
before the launch RPC" now expects `sandbox_disk_full` with the upload
error as cause. Exit is still confirmed and no launch RPC runs.
- Real sandbox from the same fleet snapshot (Daytona, 10 GiB): hold
created a 256 MiB reserve. The disk was filled to 100%. A 5 MB write
then failed. Release returned `true`, and a 50 MB write then succeeded.
A second release returned `false`. Hold on the now 98% full disk did not
create a reserve.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. `pnpm
verify` result is in a comment below.
- Review round: tests for a lost release RPC, hold/release ordering,
command-based staging on a full disk, the hold after a dispatched first
launch (not on relaunch), and no reserve on SSH. Each launcher-level
test failed with its fix removed.
- Focused suites on the review revision: transport 335/335, launcher
130/130 (filtered), launch assets 35/35, reserve 14/14. Server and
runner typecheck pass.
- Not run: a full product run in which an agent fills the disk and the
relaunched `runnerd` resumes the turn end to end.

## Risks

- Each managed sandbox holds 256 MiB more disk while it has more than 2
GiB free. That is 2.5% of a 10 GiB sandbox.
- A relaunch for any reason releases the reserve for the rest of that
run. The next run's launch takes it again.
- If the agent keeps writing after the relaunch, the disk can fill
again. The second time, nothing is left to release, and the run fails at
once with `sandbox_disk_full` instead of after 7 minutes.
- The run's error code is still `native_provider_terminal_failed`. The
message now names `sandbox_disk_full`. A dedicated failure code and
recovery text are a follow-up.
- `runnerd` loses its own error message when it cannot persist its
diagnostic. This PR does not change the Rust binary.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:45:20 -06:00
Nicky Leach 3a41e54c51 perf(runner): start ACPX once on parked checkpoint relaunch (#22064)
## Thinking Path

> - Paperclip manages work for AI agents.
> - Runner uses ACPX to resume provider sessions from durable
checkpoints.
> - A parked Runner can be gone after a reboot or retention eviction.
The host can then verify the saved park checkpoint and start a
replacement process.
> - The relaunch currently opens a sidecar for source-run readiness and
opens another for the incoming run.
> - The replacement Runner keeps the source-run command fence until
authenticated attachment.
> - Runner can open the sidecar once with the incoming run's bindings
and verify dormancy before it accepts the attachment.

## Linked Issues or Issue Description

Refs #19912 for ACPX run and tool authority design.

**What existing behavior does this improve?**

ACPX follow-up when the parked Runner process is gone and the host
starts a replacement from its verified park checkpoint.

**Subsystem affected**

Native Runner hosting and the ACPX provider executor.

**Current behavior**

The readiness probe and the idle event poll both enter generic recovery.
Recovery opens the source-run sidecar. Attachment then closes it and
opens a new sidecar for the incoming run.

**Proposed behavior**

A host can signal this path through private Runner control after
verifying an earlier ACPX park checkpoint and confirming that the old
execution has stopped. The replacement Runner checks provider
eligibility and checkpoint settlement, serves readiness from saved
state, and fences commands until authenticated `run.attach` starts the
provider with the incoming run’s bindings. A live frozen Runner follows
the existing thaw and claim flow.

**Reason and benefit**

This removes a sidecar startup and shutdown from the time to the next
ACPX turn while preserving the conversation and applying the incoming
run’s tool, instruction, and workspace bindings.

**Breaking changes**

The launch argument is optional and requires private host control in
listener mode. The feature reads checkpoints from the previous release.
No checkpoint or controller message format changes.

## What Changed

- Keep a relaunched park within the existing host claim command fence.
- Load a settled provider checkpoint that qualifies for host parking
without opening the provider during readiness or polling.
- Open the provider with the incoming run's settings. Verify the session
identity and that it has no active work before publishing the new
provider checkpoint.
- Verify one sidecar start, preservation of the saved checkpoint on
failed admission, and rejection of early turn execution.

## Verification

- `cargo fmt --manifest-path packages/paperclip-runner/runner/Cargo.toml
--all --check`
- `cargo test --offline --locked -p paperclip-runner-core --lib --bins`
passed locally (673 passed, 4 ignored).
- The confined sidecar regression uses previous-release checkpoints for
two ACPX providers. It verifies a single start on attachment for both
and preserves the saved checkpoint when the live provider reports an
active goal.
- The authenticated loopback test restarts with a saved incoming
provider checkpoint and a prepared authority receipt. It verifies
receipt replay, rejection of the prior authority after activation, and
command execution under the incoming run.
- `cargo test --offline --locked -p paperclip-runner-core --test
acpx_warm_attachment --test acpx_park_readiness --test durable_recovery`
passed locally (19 passed).
- Hosted CI passed. Greptile scored the current commit 5/5 with no
unresolved review threads.

## Risks

- An incorrect checkpoint could cross a run boundary. The private host
signal requires settled Runner and ACPX provider checkpoints. Runner
verifies live session identity and dormancy before transferring
authority.
- The existing transition receipt handles reconnects after attachment.
The optional launch signal has no effect on older Runner binaries.

## Model Used

- OpenAI GPT-6 (Codex), with source analysis and local Rust process and
protocol tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have documented the private launch contract and compatibility
above
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-17 13:32:28 -08:00
Devin FoleyandPaperclip 84b75502c0 fix(ssh): support ACP with durable Cloud workspaces (#22063)
## Thinking Path

> - Paperclip coordinates agents that run on local and remote computers.
> - SSH environments use the harness installed on a computer the
operator controls.
> - Cloud SSH tasks now use durable remote workspaces.
> - Claude and Codex default to ACP, but ACP rejects those in-place SSH
workspaces.
> - The shared ACP engine already has SSH process bridges and managed
account delivery.
> - This pull request lets that engine use the remote task directory and
stage runtime assets separately.
> - The benefit is ACP execution with the selected Paperclip account and
the existing SSH environment.

## Linked Issues or Issue Description

**What happened?**

A Claude or Codex agent using the default ACP engine cannot run in a
volume-free Cloud SSH environment. The adapter rejects its in-place
workspace before execution. Setting the engine to CLI works around that
restriction.

**Expected behavior**

ACP runs directly in the authoritative remote task directory. Paperclip
keeps using the selected account, permission settings, process bridges,
and durable workspace owner.

**Steps to reproduce**

1. Create an SSH environment with agent-managed authentication on a
volume-free server.
2. Select a Paperclip AI account for a Claude or Codex agent and leave
its engine at the default.
3. Assign a task to the agent.
4. Observe the ACP in-place workspace rejection.

Related: #21957 adds durable Cloud SSH workspaces. #21248 adds
agent-selected SSH accounts. A search for SSH ACP work found #21248 and
no open duplicate. ROADMAP.md has no overlapping item.

## What Changed

- Admit managed SSH ACP with an in-place workspace in both adapters.
- Keep ACP's session directory and fingerprint bound to the remote task
directory.
- Stage private runtime assets with workspace copying disabled. The
remote workspace owner still saves task files.
- Retain the copy-mode SSH path and existing account, confinement, and
command availability checks. No engine fallback or permission change is
added.
- Extend the existing lifecycle tests for in-place staging, session
reuse, current credentials, account changes, and cleanup after bridge or
turn failures.
- Extend the real OpenSSH/Postgres heartbeat test to the default ACP
engine. Its protocol fixture creates a file and verifies conversation
history on a follow-up.
- Update the SSH setup documentation.

Workspace placement belongs to the shared ACP engine. The adapter
changes remove their explicit SSH admission refusals. The server test
exercises the existing heartbeat path; no server execution logic
changes. Existing large test files retain their shared harnesses instead
of adding a second lifecycle mock graph.

## Verification

- 418 focused tests pass across 7 suites: 411 adapter/ACP lifecycle
tests, all 4 real SSH workspace tests, and 3 real ACP host-spawn smoke
tests.
- Typechecks pass for adapter-utils, Claude, Codex, and the server.
- `pnpm verify` returns 2 because this touches shared runtime code and
deployment documentation. Required CI supplies the broader build,
browser, and release coverage.
- Tested head: `33b17a9a21c0a5929523702ebed9c8faa3b22955`.
- `git diff --check` and the local secret/PII scan pass. Greptile
reviewed the tested head at 5/5 with no actionable findings. Final-head
[CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/44167483231/attempts/2)
passes on `33b17a9a21c0a5929523702ebed9c8faa3b22955`: 58 checks pass and
6 are skipped, with no failures or pending checks. Both required gates
pass. Four first-attempt jobs were cancelled after their runners
received shutdown signals, confirmed in each job log; automatic recovery
reran those jobs successfully. No source change was needed for that
recovery. Greptile is 5/5 with no unresolved findings, and the branch
has no merge conflicts. GitHub still requires code-owner review before
merge.

```sh
pnpm exec vitest run --maxWorkers=2 \
  packages/adapter-utils/src/acpx-engine/execute.test.ts \
  packages/adapters/claude-local/src/server/execute.remote.test.ts \
  packages/adapters/codex-local/src/server/execute.remote.test.ts \
  packages/adapters/claude-local/src/server/acp.test.ts \
  packages/adapters/codex-local/src/server/acp.test.ts
PAPERCLIP_ENABLE_DARWIN_SSH_ENV_LAB=1 pnpm exec vitest run --maxWorkers=2 \
  server/src/services/ssh-remote-workspace.integration.test.ts
pnpm exec vitest run packages/adapter-utils/src/acpx-engine/remote-spawn-smoke.test.ts
pnpm --filter @paperclipai/adapter-utils typecheck
pnpm --filter @paperclipai/adapter-claude-local typecheck
pnpm --filter @paperclipai/adapter-codex-local typecheck
pnpm --filter @paperclipai/server typecheck
```

The new admission and staging cases failed before the source fix. The
first whole-heartbeat follow-up test also changed an agent setting
between turns. That correctly reset the session. The fixture now keeps
those settings unchanged and verifies the resumed conversation. Running
both engine cases together also exposed duplicate company prefixes and
environment names in the fixture; each case now uses its own names.

The SSH qualification uses a real isolated OpenSSH server, real
Postgres, real ACPX runtime and bridges, and a synthetic ACP provider
with a selected test account. It does not call a live provider or run an
agent on a user's computer.

## Risks

The change affects the shared ACP staging path, restricted to SSH
targets that declare in-place realization. Sandbox staging retains its
prior behavior, including the existing Codex in-place refusal. Copy-mode
SSH sessions retain their prior directory layout. Previously saved
sessions keep their format; this adds no migration or session reset.

ACP still requires its server command on the remote login PATH. Host
authentication and local confinement remain CLI-only. Cold
controller/provider reconnection and live Claude/Codex provider
acceptance are not qualified by this test. No staging deployment or
existing agent configuration is changed by this pull request.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh
reasoning effort and code tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 13:18:26 -08:00
DottaandPaperclip 9911b7052f Expose continuation retry keys and preserve account-repair resumes (#22046)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner gives agents a tool to report work and wait for a later
check.
> - That tool requires a stable retry key for each explicit wait.
> - Codex can hide the wait fields when it renders our nested schema.
> - Hidden fields make a correct report harder to construct.
> - This pull request changes the provider schema structure and
preserves its rules.
> - Agents can see the required fields before they submit a report.

## Linked Issues or Issue Description

**What happened?**

The completion tool appears in Codex code mode with `continuation?:
unknown`. An agent can omit the required retry key and receive a
rejected report. The top-level schema was fixed in #21398. The nested
continuation still uses the same problematic structure.

**Expected behavior**

Codex must show the required `kind`, `summary`, and `idempotencyKey`
fields. Invalid reports must still fail validation.

**Steps to reproduce**

Render `PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA` with the public
Codex code-mode renderer at `b271854568d6c9282e999d6f36e5e85f447d027d`.
Its `render_map` function processes `allOf` before sibling properties.
The conditional-only branch renders as `unknown` and hides those
properties.

Source:
https://github.com/openai/codex/blob/b271854568d6c9282e999d6f36e5e85f447d027d/codex-rs/code-mode-protocol/src/json_schema_types.rs#L127

**Paperclip version or commit**

Baseline: `01451fae5b25f784956e9932f12031dc194ac247`.

The first complete CI browser attempt also exposed an intermittent
account-repair race. Repair restores the task and resolves its recovery
action, then a sweep can reclassify the same old failure and block it
again before the queued reply starts. The repair in this PR honors that
exact persisted resolution while holding the existing task lock. A later
failed run still escalates.

## What Changed

- Express the single provider continuation condition as direct `if` and
`then` fields. This is equivalent to its previous `allOf` wrapper.
- Keep the canonical output schema unchanged. Keep required retry keys
and case handoff IDs.
- Add a discovery guard, a 60-case provider/canonical validation
comparison, and authenticated HTTP bridge checks.

### Hypothesis and thesis for a fix

The nested `allOf` hides fields in the tested Codex renderer. The
equivalent direct condition lets that renderer reach the object fields.
This belongs in the existing provider compatibility schema. It changes
no controller policy, company access, saved state, or retry identity.
The experiment proves field visibility. It does not prove which renderer
served a historical request or measure model omission rates.

- Honor a resolved/restored configuration recovery for the same company,
task and failed run before re-escalating. Dispatch checks and recovery
for new failures remain intact.
- Add a PostgreSQL regression covering repeat sweeps, a new failure
identity, an unresolved recovery, and another task's repair. This
belongs in the existing locked escalation function; no new state or
subsystem is introduced. `service.ts` was already over 1,000 lines.

- Define the handoff condition once and explicitly compose both
canonical and provider schemas from it, removing the implicit
one-element `allOf` assumption raised in review.

## Verification

- Built and ran the exact pinned public Codex renderer in a disposable
Rust crate. The complete baseline schema renders `continuation?:
unknown`. The changed schema renders a nullable object with required
`kind`, `summary`, and `idempotencyKey` fields.
- The new discovery regression fails on the baseline. The same test
passes with the fix.
- Runner: `pnpm exec vitest run src/contracts/completion-result.test.ts
src/contracts/tool-input-variants.test.ts
src/contracts/provider-tool-schema-shape.test.ts
src/drivers/runner-tool-bridge.schemas.test.ts
src/drivers/runner-tool-bridge.continuation.test.ts` — 48 tests pass.
- Runner: `pnpm exec tsc -p tsconfig.json` and `node
scripts/generate-semantic-contracts.mjs --check` pass.
- Server: the selected `task-monitor.integration.test.ts` test for
unfinished monitor waits passes with embedded PostgreSQL. It rejects an
unscheduled wait and accepts the saved monitor.
- Server: the selected `issue-monitor-scheduler.test.ts` recovery test
passes with embedded PostgreSQL. It checks persistence, same-key replay,
and one scheduled wake. The fixture's successor run fails its separate
responsible-user setup. This is not proof of successful successor
execution.
- HTTP bridge handler is simulated. It verifies rejection before
dispatch, valid monitor/response-wake/handoff reports, retry identity,
and authentication.
- Recovery regression: baseline 1 failed / 3 passed (the restored task
was blocked again); unchanged test after the fix 4 passed. Together with
recovery state-read tests: 7 passed. Existing configuration-recovery
selection: 4 passed, 108 deselected.
- Final completion schema and authenticated HTTP bridge selection: 33
passed.
- Local real browser journey with disposable embedded PostgreSQL and
ACP/provider fixtures: baseline repeat5 produced 2 failures / 3 passes,
matching CI. Fixed repeat5: 5 passed in 1.8 minutes. The test
independently checks exactly one successful successor, stored connection
binding and persisted reply after reload.
- No full local suite, deployed recovery, or probabilistic model result
is claimed. Fresh CI and review are required for the updated head.

- Review follow-up: 48 focused schema/discovery/HTTP tests pass; `node
scripts/generate-semantic-contracts.mjs --check` passes. Fresh CI/review
required after the refactor.

## Risks

Low runtime risk. The accepted input rules are unchanged. A provider
that handles direct conditionals differently could expose a
compatibility issue. The canonical validator still enforces the output
contract. No migration or saved-session reset is needed. Previously
saved state keeps the same format. Recovery now honors existing
resolved/restored actions only for the exact failed source run, so a new
configuration failure remains eligible for escalation. Company, task,
ownership and dispatch checks remain in force.

## Model Used

OpenAI GPT-6 Codex agent, with reasoning, tool use, and code execution.
The runtime did not expose a more specific model ID.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 14:23:19 -06:00
DottaandClaude Opus 5.5 f4cb69205e Let plugin workers serve connector tools (plugin transport) (#22019)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins can now add connector types to the Connectors catalog
(#22004), but only for transports the host already runs (remote MCP,
local stdio)
> - A plugin cannot yet serve a connector's tools itself, and legacy
plugin tools (top-level manifest `tools`) still reach every agent
without a connection or install
> - A plugin should be able to ship a full connector: settings,
credentials, tools, grants, policies and audit, like a built-in one
> - This pull request adds the `plugin` transport, the SDK API to serve
it, and moves legacy plugin tools under the same activation rules
without changing today's access
> - The benefit is that any connector can live in a plugin, and agents
get plugin tools only when an operator enables them

## Linked Issues or Issue Description

Refs #22004

**Subsystem affected**
Plugins (SDK, worker protocol, manifest), Connectors (tool access,
catalog discovery), tool gateway, database.

**Problem or motivation**
A plugin connector can only describe an existing transport. There is no
way for a plugin worker to serve a connector's tools with per-connection
settings and credentials. Legacy plugin tools are visible to every agent
as soon as the plugin is enabled, which is the opposite of the connector
activation rules.

**Proposed solution**
Add a `plugin` transport. The gateway keeps every decision (grant,
policy, approval, rate limit, audit) and calls the plugin worker with
the connection, its settings and a per-call credential. Legacy top-level
tools become one implicit connector per plugin. Existing installations
keep exactly their current access through a one-time migration.

**Alternatives considered**
Move legacy tools to connection tool names (`app.<connection>:<tool>`).
Rejected: names change, so existing tool policies and agent instructions
that name `plugin:tool` would stop matching (a possible widening or
loss). Legacy tools keep their names and handlers; only their visibility
is now gated by an install of the implicit connector.

## What Changed

- `packages/db`: migration `0491` widens
`tool_connections_transport_check` with `plugin` (`NOT VALID`, annotated
`online_compatible`), and adds `plugin_connector_legacy_access`, seeded
with one row per installed legacy-tools plugin and company.
- `packages/shared`: method schema gains `settingsSchema` and `tools`
(`{ kind: "static", list: [{ name, displayName?, description?, risk,
parametersSchema }] } | { kind: "dynamic" }`), only for `transport:
"plugin"`; tool names must match `[a-z0-9_]`; plugin-served methods use
`none` or `api_key`. The connect API accepts `settings`.
- `packages/plugins/sdk`: `ToolRunContext.connection` (`id`, `appSlug`,
`methodKey`, `grantId`, `settings`, `credential?`);
`ctx.connectors.register(appSlug, toolName, fn)` and
`ctx.connectors.listTools(appSlug, fn)`; new `listConnectorTools` RPC;
`executeTool` routes connector calls by `appSlug`. A worker refuses to
start when a declared static tool (or dynamic lister) has no handler.
The test harness gains `executeConnectorTool` and `listConnectorTools`.
- `server/src/services/plugin-connector-runtime.ts`: routes one call to
the plugin's worker; a stopped worker is a
`PluginWorkerUnavailableError`.
- Connect flow (`tool-access.ts`): a `plugin` method creates a
`paperclip_plugin` application and a `plugin` connection with `config =
{ sourceTemplateKey, connectionMethodKey, settings }`, validated against
`settingsSchema`. Credentials, grants, credential policy, installs and
profiles use the existing code.
- Catalog: static tools come from the definition, dynamic tools from
`listConnectorTools`, with the existing version hashing and retirement.
Tool risk comes from the declaration.
- Gateway: plugin connections join connected-tool discovery and both
execution paths (direct and approved actions). For each call the gateway
resolves the grant and, for `api_key`, the credential for that call
only. The run context comes only from the session. A stopped worker
returns `503 plugin_connector_unavailable` with `retryable: true`. The
token broker refuses to export plugin connector credentials.
- Legacy tools: a plugin with top-level `tools` and no `apps` gets an
implicit connector (slug `default`, named after the plugin, no
authentication). Its tools keep their names, handlers and today's access
decision, and an agent sees them only when a connection of that
connector is installed for it or for the company. The connector's saved
action rules (Off, Ask first) can only restrict them.
`completeLegacyPluginToolAccess` turns each migration row into such a
connection with a company install, at boot and when a disabled plugin is
enabled again; it is single-flight across servers, resumes a draft left
by an interrupted attempt, and never overrides an operator's later
changes.
- Catalog: tools a dynamic connector stops offering are retired from
discovery; `listConnectorTools` carries the connection's `companyId`,
and uses the acting person's grant for personal connections.
- Board: the setup flow renders `settingsSchema` with the shared JSON
Schema form, sends `settings`, and restores saved settings on resume and
reconnect.
- `create-paperclip-plugin --template connector` now scaffolds a plugin
connector (`apps` + `ctx.connectors.register` + a harness test). SDK
README and `PLUGIN_SPEC.md` describe connectors; top-level `tools` stay
documented as legacy.
- `inferToolRisk` moves to `tool-risk-inference.ts` so the catalog can
reuse it.

Design notes where the spec left room:
- OAuth for `plugin`-transport methods is not supported in this PR
(`none` and `api_key` only); OAuth plugin connectors can use
`mcp_remote` today. Noted as follow-up.
- Board "Test" calls on a plugin connection pass an empty
`agentId`/`runId`, because there is no agent run.
- Companies created after the upgrade get no automatic legacy
connection; they follow the normal activation rules.

## Verification

- `server`: `npx vitest run
src/__tests__/plugin-connectors-plugin-transport.test.ts` (new):
- rules 1-3: a ready plugin and configured accounts give no tools; an
install for agent A gives only A the tools; agent B is denied on
invocation;
- the worker receives `runContext.connection` with the right id, method,
grant, settings and the call's credential; spoofed arguments do not
change it; each account calls with its own key; results do not contain
the key;
- invalid settings are rejected (422); a stopped worker is `503
plugin_connector_unavailable` (retryable); a disabled plugin hides the
tools;
- dynamic tools are listed through the worker and a tool it stops
offering is retired;
- rule 5: a new installation of a legacy-tools plugin gives no tools; an
installation recorded by the migration keeps the same tool names for
every agent, through one company-wide connection (two concurrent
carry-overs make one; an interrupted draft is finished; a retry keeps an
operator's restriction), and is not also exposed as connection tools;
Ask first and Off on the implicit connector restrict the legacy tools;
  - the token broker refuses to export a plugin connection's credential.
- Existing gateway plugin-tool tests (`tool-gateway`,
`tool-gateway-service`, `prompt-injection-gateway`) now set up the
legacy grant an existing installation has, and pass unchanged otherwise.
- Regression: `plugin-connectors-circleback-parity`, `tool-gateway`,
`tool-gateway-service`, `prompt-injection-gateway`,
`tool-access-service` (846 passed), plus `connector-tool-policy`,
`plugin-lifecycle-restart`, `remote-mcp-connectors`,
`ai-connection-router` in an earlier run.
- UI: `AppsConnect.test.tsx` covers sending plugin settings from the
settings schema.
- SDK: `tests/worker-rpc-host.test.ts` (start refusal without handlers,
connector routing, tool listing). Scaffold: `entrypoints.test.ts`.
- `packages/db`: `pnpm typecheck` including `check:migrations`; server
layered typecheck (sources and tests); `tsc --noEmit` for shared, SDK,
scaffold and UI.

## Risks

- Behavior change for new installations: legacy plugin tools no longer
reach agents until an operator connects the plugin's connector.
Installations that exist at upgrade time keep their tools through the
migration rows.
- The migration widens a check constraint (`NOT VALID`) and copies
plugin × company rows into a new table. Both are fast; the previous
release never writes `plugin`.
- Calls depend on the plugin worker; when it is down the agent gets a
retryable error.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:09:42 +00:00
DottaandClaude Opus 5.5 904c01d1a3 feat(mcp): graduate Assistant connections (MCP) out of experimental (#22015)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Assistant connections (MCP) let an outside assistant, or another
Paperclip instance, sign in as a person and use the `/mcp/paperclip`
endpoint
> - The feature was behind the experimental `enablePublicMcp` setting.
The setting was off by default and in practice off on Cloud stacks
> - Instance-to-instance work (one person's agent files a task on a
colleague's instance) fails when the receiving instance has the setting
off
> - This pull request moves the feature out of Experimental. It is now
always on, and a stored or Cloud-managed "off" value is ignored
> - The benefit is that every authenticated instance with a public URL,
including Cloud workspaces, accepts assistant connections with no setup
step

## Linked Issues or Issue Description

**Subsystem affected**
Public MCP (`server/src/services/public-mcp/`), instance settings,
Connectors → Assistant Connection (MCP) setup page, Experimental
settings page.

**Problem or motivation**
A remote MCP connector on one instance cannot use another instance's
`/mcp/paperclip` unless an admin on the other instance finds and turns
on an experimental toggle. On Cloud, the toggle is off and is managed by
the fleet. Instance-to-instance delegation therefore does not work by
default.

**Proposed solution**
Remove the gate. Keep `enablePublicMcp` only as a compatibility key, the
same way `enableApps` was graduated: old stored rows, PATCH bodies and
Cloud managed configs still parse, but the effective value is always
`true`.

**Alternatives considered**
An instance-admin kill switch that defaults ON. I did not keep one. Each
person already controls access: a connection exists only after that
person approves OAuth consent for one company, and the person or an
admin can revoke it. Deployment policy (authenticated mode and a valid
public URL) still decides whether the endpoint exists. A second switch
would add a state that can silently break instance-to-instance work
again. A schema migration was also not needed: the value lives in the
`experimental` JSON column, and the stored key becomes inert.

**Roadmap alignment**
Requested by the maintainer to enable instance-to-instance
communication.

## What Changed

- `packages/shared`: `enablePublicMcp` is a deprecated compatibility
key. The schema default is `true`. The feature catalog entry is
"Assistant connections (compatibility)" with both defaults `true`.
- `server` instance settings: normalization always returns
`enablePublicMcp: true`. The Cloud managed overlay skips the key, so a
fleet value of `false` cannot disable it. Removed the key from the Cloud
staging experimental-key list. Removed the client-mirror policy step
that ran only when the toggle changed.
- `server` public MCP: personal discovery, OAuth, tools, setup and event
delivery no longer read instance settings (`isEnabled()` returns `true`
for the personal endpoint without a DB read). Dot, Grok and Muse agent
connections keep their own experimental settings.
- `server` Cloud paths: the client-mirror sync, the hosted event
authority, the ingress-handoff expiry worker and the idle sleep proof
treat the front door as enabled. The warm-standby `503
workspace_unclaimed` behavior is unchanged.
- `server` Dot/Grok gates: they no longer also require
`enablePublicMcp`. Error messages no longer tell people to enable
Assistant connections.
- `ui`: removed the Experimental toggle, the "Enable assistant
connections / Open Experimental settings" block on the Assistant
Connection page, and the "Enable Assistant connections (MCP)" hint in
Dot setup. Updated the Dot and Grok copy.
- Storybook: deleted the Experimental-setting story and removed the
"enable the experiment" step from the journey and presenter.
- Docs: `doc/public-mcp.md` ("Enable an instance", "Start from
Connections", `GET /api/mcp/setup`), `doc/openai-dot-runner.md`,
`doc/idle-grok-dispatch.md`, and the agent-access map.
- Tests: new red/green tests for "enabled with no setting stored" and "a
stored or managed off value is ignored". Removed the tests for the
"disabled" behavior, which no longer exists.

## Verification

- Red/green: with the old source and the new tests,
`instance-settings-readonly.test.ts`,
`instance-settings-service.test.ts` and `public-mcp.test.ts` fail (4
failures, and the public-MCP suite fails in `beforeAll` because
`isEnabled()` was `false` with no setting stored). With the new source
they pass.
- `server`: `public-mcp.test.ts`, instance-settings (readonly, service,
managed overlay), `managed-config`, `cloud-event-authority`,
`events-work`, `dot-runner`, `dot-oauth-onboarding` and
`agent-runtime-assignments`. 354 passed.
- Cloud client-mirror lifecycle
(`client-mirror-lifecycle.postgres.test.ts`, skipped in CI) passed
against a local Postgres 17 with
`PAPERCLIP_MCP_SLEEP_TEST_DATABASE_URL`. 6/6, plus `events-work` 28/28.
- Idle and sleep: `idle-incoming-wake` (service and route),
`idle-deadline-proof`, `idle-incoming-wake-capability`,
`idle-sleep-proof`, `client-mirror-ack`, `recovery-deadline-acceptance`
and `idle-completed-history` pass. The warm-standby test now also
asserts that discovery after the claim does no SQL read.
- `ui`: `InstanceExperimentalSettings`, `AssistantConnection`,
`NewAgentDialog`, `NewAgent` and `ExternalAgentInviteDialog.muse` pass.
267 passed.
- Typecheck: `packages/shared`, `server` and `ui` pass. `node
scripts/check-maps.mjs` passes.
- `pnpm verify`: see the CI checks on this PR.
- After the deploy: confirm
`/.well-known/oauth-protected-resource/mcp/paperclip` and `/mcp/setup`
return 200 on paperclip.staging without changing any setting.

## Risks

- Behavior change: instances that had the setting off now serve
discovery, OAuth and `/mcp/paperclip`. Security invariants are
unchanged: the endpoint is still mounted only in authenticated mode with
a valid public URL, OAuth still needs a real signed-in user's consent,
local implicit board authority cannot approve, and revocation works the
same way.
- Cloud sleep: on a managed stack, the MCP sleep proof blocks sleep
until the client mirror publishes the tenant front door to Cloud. The
client-mirror sync runs at boot and does this once. Stacks that already
had the flag on use this path today. A stack whose Cloud connector
cannot publish the mirror would stay awake. Watch fleet sleep after the
deploy.
- Personal public-MCP requests now read instance settings one time less.
- Fleet admin: the Cloud fleet catalog can still show and set
`enablePublicMcp`. The value is ignored. Removing it from the fleet
catalog is a separate change in the Cloud repo.
- No migration. The `0354` trigger still compares the stored key. That
causes at most one extra connector-runtime invalidation if an old client
writes the key.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 11:42:36 -06:00
DottaandClaude Opus 5.5 b1efe32c60 feat(connections): name and guide connections to another Paperclip instance (#22018)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each Paperclip instance serves an assistant connection (MCP) at
`/mcp/paperclip`, and any instance can connect a remote MCP server by
its address
> - So one instance's agents can file and follow tasks on another
instance, for example a colleague's
> - But the other instance's tools look like the agent's own Paperclip
tools: same product, similar names, and the connection prefix was a
generic internal link key
> - An agent can then file local work on someone else's instance, or the
reverse
> - This pull request recognizes a Paperclip address, names the
connection and tool prefix after that instance, adds agent instructions,
and refuses a connection to the same instance
> - The benefit is that agents can tell their own Paperclip from a
connected one

## Linked Issues or Issue Description

**Subsystem affected**

Connections: "Connect your own MCP server" (generic remote MCP) and the
tool gateway names that agents see.

**Problem or motivation**

When instance A connects instance B through `/mcp/paperclip`, A's agents
get B's `paperclip_create_task` and other tools. The tool prefix came
from an internal key (`app-gallery🔗<uuid>`), so native runners saw
`app_mcp_app_gallery_link_paperclip_create_task_…`. Nothing told the
agent that the tools act on a different instance. An agent could send
internal work to another organization by mistake.

**Proposed solution**

Recognize an address whose path is `/mcp/paperclip`. Use only the typed
address, never data from the remote server. Name the connection after
the host (`acme Paperclip`). Use that name as the tool prefix for these
connections only. Add default agent instructions. Refuse the instance's
own address with `paperclip_self_connection` (422).

**Alternatives considered**

Read the company name from the remote `paperclip_connection` tool. This
needs an extra authenticated call during setup, and it puts
remote-controlled text in names and instructions. The host is enough to
identify the stack. Require approval for each write. We rejected this:
actions start on, the same as other connections.

**Roadmap alignment**

Supports instance-to-instance work through existing MCP connections. It
adds no new transport.

## What Changed

- `packages/shared/src/paperclip-peer.ts`: detect a Paperclip MCP
address, derive a label and name, compare with own origins (loopback
hosts on the same port are equal), and build default agent instructions.
- `server/src/services/tool-access.ts`: default name and default agent
instructions for a new Paperclip connection. Instructions that the
person supplies are kept.
- `server/src/routes/tool-access.ts`: refuse the instance's own address
before any network request.
- `server/src/services/tool-gateway.ts`: use the prefix recorded on a
Paperclip connection (`paperclipPeerToolNamespace`, derived from its
address at creation). Renames never change it. Other connections, and
Paperclip connections made before this change, keep their current tool
names.
- `ui/src/pages/apps/generic-mcp-connect.ts`: same default name in the
setup flow.
- `doc/connections/GENERIC-REMOTE-MCP.md`: new section "Connecting
another Paperclip instance".

## Verification

- `packages/shared`: `npx vitest run src/paperclip-peer.test.ts`, 20
passed.
- `server`: `npx vitest run
src/__tests__/generic-mcp-connection.test.ts`, 98 passed. The naming,
instructions, self-connection and tool-prefix tests fail on the base
revision and pass with this change (red/green).
- `ui`: `npx vitest run src/pages/apps/generic-mcp-connect.test.ts`, 29
passed.
- `tsc --noEmit` passes for `packages/shared` and `server`.
- Live check with two local instances. Instance B was in authenticated
mode on port 3201 with assistant connections on. Instance A was on port
3202.
- A connected to `http://localhost:3201/mcp/paperclip`. The name was
`localhost-3201 Paperclip`. A registered with B by dynamic client
registration and asked for read, write, configure and offline access.
- B's consent page showed "Connect Paperclip localhost-3202 to
Paperclip". After approval, A had all 37 tools active, writes included.
- From A, `paperclip_connection` returned B's company.
`paperclip_create_task` created `ACM-1` on B.
- An agent session on A listed
`mcp.localhost-3201-paperclip-…:paperclip-create-task` (runner name
`app_mcp_localhost_3201_p_paperclip_create_task_…`). Before the gateway
change, the name was `app_mcp_app_gallery_link_paperclip_connection_…`.
The run received the connection instructions.
- Connecting A to `http://localhost:3202/mcp/paperclip` returned 422
`paperclip_self_connection`.

## Risks

- Low risk. Only connections whose address path is exactly
`/mcp/paperclip` change. Other connections keep their names,
instructions and tool names.
- A Paperclip connection created before this change keeps its original
tool names, because only connections that record
`paperclipPeerToolNamespace` at creation get the new prefix. Saved exact
tool-name rules and pending approvals for those connections keep
matching. They also get no default instructions; people can add them.
- Native runners keep only 20 characters of the namespace (`mcp_` uses
4), so long host labels are cut. The tool part stays complete.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, browser pane).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 11:41:13 -06:00
Devin FoleyandPaperclip 1d2b669e7a fix(ssh): allow CLI agents on volume-free servers (#21957)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An SSH environment runs an installed harness on a computer the
operator controls.
> - Volume-free Cloud servers require durable task files outside the
controller.
> - The current admission path accepts native sandbox workspaces and
rejects existing SSH CLI agents before execution.
> - The remote workspace owner already provides saved versions and a
single-writer fence.
> - This pull request gives that owner an SSH archive transport and
admits the existing Claude and Codex CLI adapters under its authority.
> - The benefit is SSH task execution without a controller volume or a
new environment type.

## Linked Issues or Issue Description

**What happened?**

A task assigned to an existing SSH Claude Code agent fails on a
volume-free Cloud server. It reports that execution requires native
Paperclip Runner and remote workspace storage. The failure occurs before
provider work starts, even when SSH and the configured AI account are
valid.

**Expected behavior**

An SSH CLI agent can use the configured Paperclip AI account and an
installed harness. Task files remain durable when the Cloud controller
is replaced. The same SSH behavior applies to direct and Tailscale
connections.

**Steps to reproduce**

1. Configure an SSH environment on a volume-free server with external
Postgres, stable encryption, and object storage.
2. Assign a Claude Code or Codex CLI agent to that environment.
3. Give it a task without a project, such as creating `hello_world.txt`.
4. Observe the workspace admission failure before execution.

Related work: #21248 adds agent-selected SSH authentication. #20617 adds
Tailscale network selection. #15120 and #17291 cover SSH process and
lease cleanup. A search of open and closed PRs found no duplicate of
this workspace admission fix.

## What Changed

- Reuse remote workspace preparation, archives, saved versions, and
single-writer admission for SSH targets.
- Add an explicit file-sync capability to the existing SSH command
runner. It streams archive files with bounded deadlines and private
temporary files. Ordinary SSH runners keep their previous capabilities.
- Run the existing CLI adapter in the remote workspace. Bind both the
target and its SSH command specification to that directory.
- Admit a direct SSH adapter on a volume-free server only when its
durable workspace, environment, lease, and path agree. Local filesystem
authority remains rejected.
- Preserve the server-owned remote workspace receipt when selecting the
legacy runtime.
- Reuse the workspace finalization lock for a legacy run only when its
company, workspace, writer, epoch, and lease receipt match. A claimed
token can finish and clear itself after another task reserves a
successfully saved shared workspace. Storage writes still require the
current workspace authority.
- Transfer recovered copies to SSH through the verified workspace
archive format. Sandbox recovery retains its existing directory
transfer.
- Document Node.js, Git, explicit `engine=cli`, and uncertain-process
limitations. No UI, Cloud control-plane configuration, schema, or
deployment change is included.

The remote workspace service owns this change. The shared SSH utility
adds file transport because its existing authentication and stream
cleanup must remain the only owners of SSH credentials. The heartbeat
changes pass the storage authority through existing preparation,
selection, and finalization boundaries. The finalization owner extension
avoids a separate copy-back lock for SSH. Existing native callers keep
their prior admission contract.

The existing SSH utility and heartbeat test files are already large. The
added code stays beside their existing stream lifecycle and shared
fixtures. A larger extraction would widen this compatibility fix.

## Verification

- Red-green tests reproduce SSH rejection and verify admission only with
remote storage authority.
- A real isolated OpenSSH fixture exercises archive capture, a
replacement database client, a saved-workspace restore to another
allocation, and a failed SSH transfer. Failed saves retain the files and
block a competing writer.
- A whole-heartbeat test uses the real SSH transport and Claude CLI
adapter, a synthetic CLI, a selected Paperclip AI account, and fixture
object storage. A follow-up resumes the same provider session and its
retained history. It makes no provider API calls. The synthetic provider
also supplies its task disposition update.
- Existing Claude and Codex SSH adapter tests, workspace preparation,
and native workspace owner recovery are included in focused checks.
- Before the review fixes, clean-checkout verification passed 610 tests
in 11 files. This includes the full workspace preparation, runtime
selection, runtime execution, environment orchestrator,
ephemeral-server, workspace transport, native owner recovery, SSH
integration, SSH command, and Claude/Codex remote execution suites. The
macOS SSH fixture was enabled with
`PAPERCLIP_ENABLE_DARWIN_SSH_ENV_LAB=1`; Vitest used `--maxWorkers=2`.
- An initial run in the main worktree hit a skill-root lookup outside
that checkout. The full selected set passes in the clean checkout. The
final company-binding type correction changes no executable behavior;
all 8 focused selection regressions pass after that correction.
- At the initial head `dd77a4d3cf0`, server and adapter-utils typechecks
passed, and the clean verification checkout matched its tree
`3f90b0681b1004556d1a720ed45fc8d439b04e96`.
- Review regressions first reproduced both a lost ownership token after
shared-workspace handoff and a rejected recovered-directory transfer.
With the fixes, 223 tests in 5 affected suites pass, including real SSH
recovery and native owner-restart, save-concurrency, archive transport,
and recovery-receipt checks. The handoff test also confirms that a stale
owner cannot save again or claim ownership after a new writer takes
over. Server typecheck passes after these fixes. The tested review-fix
head is `c5182f5d9c56e738e683d9f26b226e9a820bba83`.
- [The initial CI server
shard](https://github.com/paperclipai/paperclip/actions/runs/44009944657/job/133794523181)
failed on an assertion still expecting the old native-sandbox-only
error. Its unsupported sandbox adapter remains rejected; the assertion
now matches the supported-remote-environment error. After updating that
assertion, all 115 tests in the workspace lifecycle suite pass. Together
with the 5 affected suites, this gives 338 passing tests for the review
fixes.
- `pnpm verify` returns 2 and requests broad CI coverage for the
shared/runtime change. The full local suite was not substituted for CI.
Final-head
[CI](https://github.com/paperclipai/paperclip/actions/runs/44012991355)
is green on `c5182f5d9c56e738e683d9f26b226e9a820bba83`: 58 checks pass
and 3 are skipped, with no failures or pending checks. Typecheck, build,
end-to-end, and the aggregate verification gate all pass. Greptile is
5/5 on this head, security scan passes, and there are no unresolved
review threads or merge conflicts.

Review-fix checks:

```sh
PAPERCLIP_ENABLE_DARWIN_SSH_ENV_LAB=1 pnpm exec vitest run --maxWorkers=2 \
  server/src/services/ssh-remote-workspace.integration.test.ts \
  server/src/services/native-runtime/native-workspace-owner-restart-recovery.test.ts \
  server/src/services/remote-workspace-transport.test.ts \
  server/src/services/remote-workspace-recovery-receipt-fallback.test.ts \
  server/src/services/remote-workspace-save-concurrency.test.ts
pnpm exec vitest run --maxWorkers=2 server/src/services/remote-workspaces.integration.test.ts
pnpm --filter @paperclipai/server typecheck
```

## Risks

This extends the remote workspace owner to another transport and allows
qualified SSH CLI execution on volume-free servers. It changes remote
file placement to a task directory under `.paperclip-workspaces`. It
does not import an existing controller-local workspace.

The installed harness uses its existing adapter. Set `engine=cli`
explicitly; ACP archive staging does not support these in-place
workspaces. This does not qualify native macOS provider packs or change
an agent's engine or permission settings automatically.

An unknown remote process after SSH loss remains held until it can be
reconciled. This change does not add automatic reconciliation of that
process. SSH computers are never stopped or deleted. Real provider
acceptance and a hosted staging journey have not been run for this
revision.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh
reasoning effort and code tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 09:33:11 -08:00
DottaandPaperclip 7f9867895c fix(ui): keep automated work out of personal recents (#22024)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The sidebar helps each person return to their own work.
> - A task visit previously added any readable task to personal recents.
> - Automated updates also moved old tasks above newer personal work.
> - The Mine inbox already defines personal participation.
> - This pull request applies that rule to recent task candidates and
preserves their order during background updates.
> - Users can see current task status without a stream of unrelated
automated work.

## Linked Issues or Issue Description

**What happened?**

Routine tasks and imported GitHub review tasks appeared in personal
recents after a read. Later agent updates moved these tasks up the list.
A connected account or stored creator ID could make these tasks look
like personal work.

**Expected behavior**

A task must pass the viewer's existing Mine participation rule before it
appears in personal recents. Routine ownership, connected-account
provenance and reads alone do not qualify. A viewer can qualify through
manual creation, assignment, a direct comment, an allowed user action or
a live request addressed to them. Inbox archive state does not remove
personal history. Passive reads and machine updates must not advance its
order.

**Steps to reproduce**

1. Open an automated routine task or imported review task without
participating in it.
2. Check the sidebar recents.
3. Let an agent update an older task while newer personal work is in the
list.
4. Before this fix, the automated task appears and background updates
can move older tasks up.

## What Changed

- Add a viewer participation field to bounded navigation summaries.
Reuse the canonical Mine query and existing access checks. The field is
a read projection; no database migration is needed.
- Show only confirmed personal tasks. Remove old nonparticipants after a
successful read. Do not trust participation from local storage or full
task caches.
- Keep status and title snapshots current without changing personal
order. Explicit local edits and comments can advance recency.
- Migrate browser history once to a new storage version. Older tabs
cannot restore machine-driven ordering.
- Keep confirmed rows visible if a new batch request fails. Keep these
facts within the current account and company. Hidden candidates do not
bypass the one-second reorder delay.
- Record sidebar renames as local activity. Place a first passive visit
below existing history.
- Use the saved comment time for comments sent with reassignment or
custom run settings. Pending sends do not advance recency.
- Keep readable subtasks as context beneath qualifying parents. Keep
inbox archive independent of recents.
- Update the design rules, UI behavior notes, regression tests and
Runner source contract pin.

## Verification

- Current head: `92ece0e61859a6c54b714b3611f062ff9b5633bb`. The PR
merges cleanly with current master.
- Latest red/green: both configured-comment cases failed before the fix
and now pass. Reassignment and custom run settings advance recency only
after the comment saves, using its saved time.
- Latest local checks: 339 tests passed across IssueDetail and recent
storage, hooks and sidebar. UI typecheck and diff checks passed.
- Earlier local coverage: 592 targeted API/UI tests passed. After the
master conflict resolution, all 29 overlapping Gmail tests and the
548-input source registry check passed.
- Both Greptile and Paperclip Review report 5/5 on the current head. All
four review threads are resolved.
- Current-head [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/44132164473)
passed, including the final verify gate, build, typecheck, browser
shards, serialized tests, Runner checks and all general-test shards. The
workspace-a (1/2) retry passed after the first attempt hit an
asynchronous jsdom/Radix event error in NewAgentDialog.
- `pnpm verify` requires broader CI because DESIGN.md changed. No full
local suite or deployed-browser acceptance was claimed. This change is
not deployed.

## Risks

- Unknown eligibility stays hidden until the summary succeeds. Failed
requests preserve saved candidates and previously confirmed state within
the current account and company, including after batch changes.
- This intentionally removes automated tasks from old personal history
when the viewer has no qualifying participation.
- Readable subtasks under a personal parent remain context. They do not
become independent recent entries from machine activity.
- No task ownership, routine definition, inbox membership or permission
rule changes.

## Model Used

OpenAI GPT-6, running as Codex, with reasoning, repository tools and
focused test execution. No additional model workers were used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 11:25:58 -06:00
DottaandClaude Opus 5.5 8172d27089 feat(cloud): browser product analytics through a same-origin PostHog proxy (#22012)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud runs the same app for hosted customers, and the same
UI ships in the npm package.
> - The PostHog project for Cloud gets only server events today. It has
no page views, sessions, replays or people.
> - So the Product and Web analytics pages are empty, and we cannot see
how Cloud users move through the board.
> - This pull request adds browser analytics for Cloud only. It is off
until an operator sets two switches on a Cloud stack.
> - Self-hosted installs do not change: they never download posthog-js
and never serve the PostHog proxy.
> - The benefit is real page views, sessions, heatmaps, Web Vitals and
masked session replays for Cloud, linked to the user and the workspace.

## Linked Issues or Issue Description

**Subsystem affected**
Board UI (analytics bootstrap and privacy markers), server session
payload and a new same-origin proxy route, shared validators.

**Problem or motivation**
Cloud sends only server events to PostHog. There is no browser SDK, so
PostHog has no page views, sessions, replays or identified people for
Cloud. We cannot see where Cloud users get stuck.

**Proposed solution**
The signed-in session gives the UI a PostHog config only on a
Cloud-managed stack with `PAPERCLIP_POSTHOG_PROJECT_TOKEN` and
`PAPERCLIP_POSTHOG_BROWSER_ENABLED=1`. The UI loads posthog-js after the
page settles. Events go through a strict same-origin proxy at `/_ph`, so
ad blockers do not drop them. Replay masks all text and all inputs by
default.

**Alternatives considered**
Bake a key into the UI build: refused, because the same build ships to
self-hosted users. A generic reverse proxy: refused, because it would
forward any path. Mask only content areas: refused, because a missed
surface would leak; masking is deny by default.

**Additional context**
This is the browser part of the Cloud direct PostHog plan. The server
direct sender is already on master (#22006). Fleet configuration is a
separate change. PostHog person deletion on account deletion is a
follow-up.

## What Changed

- Uses the shared Cloud PostHog config (`resolveCloudPostHogConfig` in
`server/src/services/cloud-posthog-config.ts`, already on master from
the server direct-sending change). This PR does not change it.
- `server/src/services/cloud-posthog-browser.ts`:
`resolveBrowserAnalyticsConfig` returns `{ token, apiHost: "/_ph",
uiHost, environment, stackId, replayEnabled }` only when
`PAPERCLIP_POSTHOG_BROWSER_ENABLED=1`. `environment` is always set: the
explicit variable, else `staging` for a `*.staging.paperclip.app` host,
else `production`.
- `GET /api/auth/get-session` carries the new optional
`browserAnalytics` field (the existing signed-in bootstrap request, so
no new request on the cold path). `packages/shared` has
`browserAnalyticsConfigSchema` and the session schema field.
- `server/src/routes/posthog-proxy.ts`: a strict proxy, mounted before
the body parsers and only when browser analytics is on. It forwards only
these requests and answers 404 to all others without contacting PostHog:
- `POST /e/`, `/i/v0/e/`, `/batch/`, `/s/`, `/flags/`, `/decide/` to the
ingest host.
- `GET /array/<token>/config(.js)` and `GET
/static/[<version>/]<name>.js` to the assets host (derived from the
ingest host).
- `stream.pipeline` joins the streams, so an error on either side
(PostHog closes the connection partway through a response, or the
browser goes away) closes both and never goes unhandled. It builds a new
header set from an allowlist plus `X-Forwarded-For`. It never forwards
the inbound Host, Cookie or Authorization headers. It drops `Set-Cookie`
from responses. Bodies stream; nothing is buffered. It refuses bodies
over 64 MiB (413, also for chunked bodies that grow past the limit) and
limits requests in flight to 16 per client address and 256 in total
(429).
- `ui/src/lib/analytics/product-analytics.ts` (the only module that
talks to posthog-js) and `ui/src/components/ProductAnalyticsGate.tsx`:
- posthog-js loads after the page settles (or at a 15 s deadline), only
when the session has a config. It loads as PostHog's browser script
`/_ph/static/array.js` through the proxy, the same way posthog-js itself
loads its extensions (recorder, web vitals). It is not an npm dependency
and is in no bundle chunk.
- Init: page views on history change, page leave, autocapture, Web
Vitals, heatmaps, `person_profiles: "identified_only"`,
`tracing_headers` for our own host only, replay only when
`replayEnabled`. Surveys, product tours, web experiments and
conversations are off.
- Super properties on every event: `deployment_source: "cloud"` and
`environment`.
- After sign-in: `identify(session.userId, { email, name })` (person
properties only). `session.userId` is the authenticated actor's user id
verbatim (for example `paperclip-id:...`): `GET /api/auth/get-session`
copies `req.actor.userId` into it, and server events use that same actor
id as `distinct_id`, so browser and server events land on one person.
Tests: `auth-routes.test.ts` (the session returns the actor id verbatim)
and `ProductAnalyticsGate.test.tsx` (identify uses it) and
`group("workspace", stackId)`. On sign-out: `reset()`, then the super
properties again. `useSignOut` calls it before it leaves the page,
because Cloud sign-out navigates away while the session query is still
populated.
- Privacy (`ui/src/lib/analytics/privacy.ts`), deny by default:
- Replay masks every text node (`maskTextSelector: "*"` with
`maskTextFn`) and every input value (`maskAllInputs`).
- Only elements with `ANALYTICS_UNMASK_PROPS` show their text: sidebar
items whose label is a fixed string (`analyticsChrome`, added at the 99
call sites with literal labels), and a shared `Button`, `Label`,
`TabsTrigger` or `EmptyState` only when its caller passes
`analyticsChrome`. None of them opts in by default, because they also
render customer text (agent names, document titles). Sidebar items with
dynamic labels (recent tasks, agents, projects) stay masked.
- Replay masks attribute values too (`maskAttributeFn`, deny by
default). Only layout attributes (`class`, `style`, `role`,
`data-state`, SVG geometry), rrweb metadata and stylesheet links keep
their values. `title`, `aria-label`, `alt`, `placeholder`, `src`,
`href`, `data-*` and element references (`id`, `for`, `aria-labelledby`,
`aria-describedby`, `aria-controls`, which often embed names or slugs)
are always masked, even on unmasked chrome. So the organization
switcher's `aria-label`, task titles in `title` attributes, image
sources and download links with file paths never reach a replay.
- `before_send` (`scrubCapturedEvent`) drops the `title` property from
every event, because posthog-js copies `document.title` (task and
organization names) into page views. Every URL string in every event,
including replay meta events, keeps only its origin and path: query and
hash values never leave (they can hold tokens, `code`/`state`, file
paths or search text), and the token segment of `/board-claim/<token>`
and `/invite/<token>` is redacted. It recurses only into plain objects
and arrays, so the `Date` timestamp stays intact.
- `ANALYTICS_MASK_CLASS` keeps customer content masked even inside
unmasked chrome: `MarkdownBody` (task and issue bodies, comments, chat
messages, documents and artifact markdown), `MarkdownEditor`, the task
chat thread, the issue chat thread, the comment timeline, the file
viewer (source, markdown and CSV views) and the organization name in the
organization switcher. Text in any `contenteditable` editor is always
masked.
- `ANALYTICS_NO_CAPTURE_CLASS` (`ph-no-capture`, a blank box in replay)
on every password-type `Input` (about 40 secret, token and API key
fields), environment variable values and new secret values.
- Autocapture never records element text or attribute values
(`mask_all_text`, `mask_all_element_attributes`).
- Connected-account content (Google user data and Slack data: email
bodies, calendar events, Slack messages in tasks, chat, tool results,
previews) never appears unmasked in a replay. Replay masks all text by
default; nothing that renders connected-account content is in the unmask
allowlist; autocapture records no element text anywhere
(`mask_all_text`). `ui/src/lib/analytics/unmask-allowlist.test.ts` pins
the allowlist: only 7 files may spread the unmask attribute, sidebar
opt-ins need a literal label, and any `analyticsChrome` opt-in on
`Button`/`Label`/`TabsTrigger`/`EmptyState` fails until it is added to a
reviewed list (empty today). The same test renders Gmail, Calendar and
Slack-style tool results next to unmasked navigation and checks they are
masked.
- Guard test: UI code must never call `capture()` with a
`PAPERCLIP_EVENTS` name, so each event has one sender.
- Docs: `doc/cloud-browser-analytics.md` (switches, per-stack opt-out,
proxy allowlist, identity, privacy) and a link from
`doc/observability.md`.
- No new npm dependency and no lockfile change: posthog-js comes from
PostHog through the proxy at run time. (An earlier revision added
`posthog-js` to `ui/package.json`; the Runner support suite requires
every `ui` dependency in the reviewed lockfile, which a pull request
cannot update, so the script approach replaced it.)

## Verification

- `pnpm verify`: see the result below.
- Focused tests:
- `server/src/__tests__/cloud-posthog-browser.test.ts`: exposure matrix
(self-hosted with every switch set, Cloud without token, without stack
ID, without the browser switch, switch not `1`, enabled, replay off),
environment fallback, proxy mount only when enabled, allowed routes, 404
for disallowed methods and paths (including `//host` and `..` paths)
with no upstream contact, Host/Cookie/Authorization stripped,
`Set-Cookie` dropped, 502 when PostHog is down, and no unhandled error
when PostHog closes a response partway (red on the old `pipe()` code,
green now).
- `server/src/__tests__/auth-routes.test.ts`: the session carries
`browserAnalytics` only on Cloud with the token and the switch.
- `ui/src/lib/analytics/*.test.ts`,
`ui/src/components/ProductAnalyticsGate.test.tsx`: no import of
posthog-js without a config or when signed out; load after the page
settles, once; identify, group, reset with a mocked posthog; email and
name never become event properties; deny-by-default replay masking;
guard test.
- Real run (local, headless Chromium, production UI build, never the
real PostHog project):
- Self-hosted server, no env: 0 requests for PostHog's script and 0
requests to `/_ph` in 9 cold loads and in a 40 s session with
navigation. `browserAnalytics` is `null`. `/_ph/e/` falls through to the
UI.
- Cloud config: a full Cloud boot needs S3, a managed database and the
control plane. So a small local stand-in served the same production UI
build, mounted the real proxy from `server/src/routes/posthog-proxy.ts`,
added the real `resolveBrowserAnalyticsConfig()` output to the session,
and forwarded other API calls to the local server.
`PAPERCLIP_POSTHOG_HOST` pointed at a local capture server.
- Captured through `/_ph` only (0 requests to any posthog.com host):
`$pageview` on each client navigation (5), `$identify` with `$set: {
email, name }`, `$groupidentify`, `$groups: { workspace: <stack id> }`
on events, `environment` and `deployment_source` on every event,
`$web_vitals`, `$autocapture` with no `$el_text`, `$$heatmap`, and 10
`$snapshot` replay events. A search of every decoded `/e/` and `/s/`
payload (events, replay text and replay attributes) found the email only
in `$identify`'s `$set`, and found none of: the task body, the typed
comment, the task title, the organization name. No Cookie header reached
the capture server. 96 of 113 API requests carried
`X-POSTHOG-SESSION-ID` (the others ran before posthog-js loaded).
- Client-side visits to `/invite/INVITESECRET123` and
`/cli-auth/abc?token=CLISECRET456` reached PostHog as `/invite/REDACTED`
and `/cli-auth/abc` (no query); neither secret was in any payload.
- Disallowed proxy requests (`PUT /_ph/e/`, `GET /_ph/api/projects/`,
`POST /_ph/static/x.js`) answered 404.
- Sample replay payload (decoded rrweb text nodes from a captured `/s/`
request). The task body, the typed comment, the task title, the
organization name and the email were not in any replay payload:

```json
{"text_nodes": [
  {"parentTag": "title", "textContent": "****** ** * *********"},
  {"parentTag": "a",     "textContent": "**** ** **** *******"},
  {"parentTag": "span",  "textContent": "****** **"},
  {"parentTag": "span",  "textContent": "*** ****"},
  {"parentTag": "span",  "textContent": "Tasks"},
  {"parentTag": "span",  "textContent": "Inbox"},
  {"parentTag": "h2",    "textContent": "******* *******"},
  {"parentTag": "p",     "textContent": "****** *** * ******* ** ******"}
]}
```

- Cold load, 9 fresh-profile loads each, median, same stand-in with
analytics off and on:

| | DOMContentLoaded | Board content visible |
| --- | --- | --- |
| Analytics off | 128 ms | 336 ms |
| Analytics on | 126 ms | 334 ms |

Final head. The difference is inside run-to-run noise (earlier pairs:
332/335 ms and 335/348 ms). PostHog's script loads only after the
content is visible.

## Risks

- Inert by default. Without a Cloud-managed stack, a project token, a
stack ID and `PAPERCLIP_POSTHOG_BROWSER_ENABLED=1`, the session field is
`null`, the proxy is not mounted and no analytics code runs. The OSS
package does not change behavior.
- Per-stack opt-out: `PAPERCLIP_POSTHOG_REPLAY_ENABLED=0` turns replay
off; unsetting `PAPERCLIP_POSTHOG_BROWSER_ENABLED` turns all browser
analytics off. Both apply on the next restart.
- Replay text shown in the clear comes only from fixed sidebar labels
and from shared components whose caller opts in. Today no `Button`,
`Label`, `TabsTrigger` or `EmptyState` call site opts in, so replays
show layout, clicks and navigation with most copy masked. Opting in
fixed-label call sites is a follow-up. Review sample replays on staging
before production.
- Image `src` values are masked, so images record as empty boxes.
Sandboxed iframes (HTML artifacts, MCP apps) also record as empty boxes.
Replays therefore show layout and product copy, not customer media.
- PostHog's script is not version-pinned: the assets host serves its
current browser build, as for any PostHog snippet install, and
posthog-js already loads its extensions from that host at run time. It
runs only on Cloud stacks with the browser switch on.
- The proxy needs no sign-in, like PostHog's documented proxies:
posthog-js sends some events while a page unloads, and the proxy can
only reach the two PostHog hosts, on the listed paths, with the public
project token. Body size and in-flight limits bound the load one client
can add.
- posthog-js rejects bots (including headless browsers) on its own,
which is the wanted production behavior.
- `SidebarNavItem.production.tsx`, `Sidebar*.tsx` and the shared `ui/*`
components changed by one attribute each; no visual change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, headless browser runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 11:23:03 -06:00
DottaandClaude Opus 5.5 7f1f0eef67 fix(google-workspace): request free/busy with read & write Calendar, and update connector docs (#22026)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The built-in Google Workspace connector gives agents Gmail, Drive
and Calendar through one connection, behind an experimental setting
> - The user picks each service as read only or read & write. Paperclip
then requests only scopes from a fixed list of 14
> - Read & write Calendar requested `calendar.calendarlist.readonly` and
`calendar.events`. Our code assumed `calendar.events` also covers
free/busy lookups
> - Google's REST `freeBusy.query` does not accept `calendar.events`. So
the REST `suggest_time` tool cannot read free/busy on a read & write
grant, and falls back to reading each attendee's events
> - This pull request adds `calendar.events.freebusy` (already one of
the 14) to the read & write Calendar selection of the unified
connection, and stops treating `calendar.events` as implying free/busy
> - The benefit is that `suggest_time` on REST can see coworkers who
share only free/busy, and the stored grant no longer claims a permission
that Google did not give. It also brings three Google connector planning
docs up to date

## Linked Issues or Issue Description

Refs #22016, #22017, #22010, #22014

**What happened?**

The read & write Calendar selection of the unified Google Workspace
connection requests `calendar.calendarlist.readonly` and
`calendar.events`. `toolScopes.calendar.suggest_time` lists
`calendar.events` as sufficient, and `googleWorkspaceEffectiveScopes`
treats `calendar.events` as implying `calendar.events.freebusy`.

Google's two references differ:

- REST
[`freeBusy.query`](https://developers.google.com/workspace/calendar/api/v3/reference/freebusy/query),
checked 2026-11-17: "This request allows authorization with at least one
of the following scopes:" `calendar.readonly`, `calendar`,
`calendar.events.freebusy`, `calendar.freebusy`. `calendar.events` is
not in the list.
- Google's hosted Calendar MCP
[`suggest_time`](https://developers.google.com/workspace/calendar/api/v3/reference/mcp/tools_list/suggest_time),
checked 2026-11-17: "Requires one of the following OAuth scopes:"
`calendar`, `calendar.readonly`, `calendar.events`,
`calendar.events.freebusy`, `calendar.freebusy`. Here `calendar.events`
is accepted.

So `calendar.events` is enough for Google's MCP tool but not for the
REST endpoint that our REST `suggest_time` uses.

**Expected behavior**

A user who picks read & write Calendar gets a grant that allows
`freeBusy.query`. A grant that does not have the free/busy permission is
not recorded as having it. Grants made before this change keep working
as read & write.

**Steps to reproduce**

1. Call `googleWorkspaceSelectedScopes({ calendar: "write" })`. It
returns two scopes, without `calendar.events.freebusy`.
2. Call `googleWorkspaceEffectiveScopes(readOnlyCalendarScopes,
[calendar.events])`. It returns `calendar.events.freebusy`, which Google
did not grant.
3. With Calendar on the REST transport, `suggest_time` on a read & write
grant gets an insufficient-scope answer from `freeBusy.query` and uses
the events fallback.

## What Changed

- `packages/shared/src/google-workspace-connectors.ts`:
- `googleWorkspaceSelectedScopes` adds selection-level scopes from a new
`SELECTION_ADDED_SCOPES` map. Read & write Calendar adds
`calendar.events.freebusy`. The per-service `calendar.write` profile
(`SERVICE_PROFILES["calendar.write"]`) is unchanged, because the Cloud
broker matches legacy profiles by exact scope set.
- `googleWorkspaceEffectiveScopes` no longer expands `calendar.events`
to `calendar.events.freebusy`. `calendar.events` still implies
`calendar.events.readonly`. `calendar.readonly` still implies free/busy,
which Google's reference allows.
- `googleWorkspaceSelectionForScopes` is unchanged. It needs the legacy
write profile's scopes, so a two-scope grant and a three-scope grant
both read back as read & write.
- `toolScopes.calendar.suggest_time` is unchanged. It still accepts
`calendar.events`: Google's MCP tool accepts it, and on REST the tool
has an events-based fallback that names attendees it cannot read.
- Counts: default all-read-and-write selection is now 10 scopes (was 9);
all read-only stays 8; the union of all 27 selections stays exactly the
14.
- Tests (written first and seen failing on `master`):
- `packages/shared/src/google-workspace-connectors.test.ts`: scopes for
read and for read & write Calendar; the legacy `calendar.write` profile
keeps two scopes; default count 10 and read-only count 8; old two-scope
grant, new three-scope grant and partial consent (user declines
free/busy) all round-trip as read & write and keep `list_calendars`,
`list_events`, `suggest_time`, `create_event` and `delete_event`; no
recorded free/busy that Google did not grant. The existing exhaustive
test still checks that no selection requests a scope outside the 14.
- `server/src/__tests__/tool-access-service.test.ts`: customer-owned
Workspace OAuth with read & write Calendar asks Google for all three
scopes. With and without free/busy in Google's answer, the connection is
active, the Calendar tools are listed, the grant stores exactly what
Google granted, and it reads back as read & write.
- Stale tests on `master` fixed: `#22014` made Gmail 9 of 9 in the REST
tool set while `#22016` still expected 3 Gmail REST tools.
`google-workspace-rest-dispatch.test.ts` and
`google-workspace-rest-transport.test.ts` now derive the Gmail list from
the allowlist and use Chat as the service with no REST tools.
- Docs: `doc/connections/GOOGLE-WORKSPACE.md`,
`doc/connections/GOOGLE-WORKSPACE-SERVICE-CONSENT.md` and
`doc/connections/tool-method-permission-reviews.json` say ten scopes and
explain the Calendar change.
- Docs: `doc/memory/google-connector-memory.md` (shipped PRs #22010,
#22014, #22016, #22017; three Nov 17 decisions; "Where Google content is
kept" replaced with #22010's inventory, what deletion keeps and its
limits; Scopes section; REST tools status table; still-open list),
`doc/plans/2026-11-17-google-workspace-production-checklist.md` (help
page and retention audit ticked with #22010; REST tool progress),
`doc/plans/2026-11-17-google-workspace-code-fixes.md` (fix 6 done with
limits; section 9 status table; fix 10 note).
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the changed shared files.

Not changed:

- `server/src/google-workspace-rest-tools/**`. Another change owns that
folder. Its README open question about this scope is now resolved and
can be removed there.
- The picker disclosure (`googleWorkspaceDisclosure.ts`). Read & write
Calendar already says agents "can see your calendars, events and
free/busy times". With this change that line is accurate on REST too.

Behavior for existing grants: a saved read & write grant without
free/busy still reads back as read & write, keeps every Calendar tool,
and refreshes without change. Nothing prompts the user to reconnect. A
user who chooses read & write Calendar in the picker is asked for the
new scope. An account-page reconnect without a new choice still restores
the exact saved request, as before.

Cloud broker and Google Console: no change needed.
`calendar.events.freebusy` is one of the 14 in `workspace.all`, the
broker accepts any nonempty subset of `workspace.all`
(`doc/connections/GOOGLE-WORKSPACE-SERVICE-CONSENT.md`), and the legacy
`calendar.write` profile's exact scope set did not change.

## Verification

- `cd packages/shared && npx vitest run
src/google-workspace-connectors.test.ts src/app-definitions.test.ts`: 55
passed. On `master` the new shared tests fail (3 failed).
- `cd server && npx vitest run src/__tests__/tool-access-service.test.ts
-t "free/busy"`: 2 passed. With `master`'s
`google-workspace-connectors.ts`: 2 failed.
- `cd server && npx vitest run
src/__tests__/google-workspace-rest-transport.test.ts
src/services/google-workspace-rest-dispatch.test.ts`: 29 passed (4
failed on `master` before this change).
- `cd ui && npx vitest run
src/features/connections/GoogleWorkspaceServices.test.tsx
src/pages/apps/AppDetail.test.tsx`: 115 passed.
- `cd packages/shared && npx tsc --noEmit -p .`: clean.
- `pnpm verify` exits 2 (a global CI input changed), so required CI
supplies the broad coverage.
- Docs facts checked against `gh pr view` for #22010, #22014, #22016,
#22017, the code, and paperclipai/paperclip-website PR 197 (still a
draft; does not mention OpenRouter).

## Risks

- Low risk. The submitted set of 14 does not change, and the broker
contract does not change.
- New read & write Calendar grants record `calendar.events.freebusy`
next to `calendar.events`. Code that reads stored scopes sees one more
scope. All consumers (callback, refresh, discovery, picker pre-fill) are
covered by the tests above.
- A read-only Calendar request on an account that Google reports only
`calendar.events` for no longer records free/busy. That matches what
Google granted; `suggest_time` is then not offered on that grant.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use, web fetch and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:40:58 +00:00
DottaandClaude Opus 5.5 01451fae5b Let plugins add connector types to the Connectors catalog (#22004)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use external services through connectors: a catalog of
connector types, connections (accounts) per type, and per-agent
installs, policies and audit in the tool gateway
> - Today only core can add a connector type. Every type is a JSON
definition compiled into `@paperclipai/shared`
> - Plugins can add agent tools, but those tools skip connections,
grants, per-agent installs and the Connectors page
> - This pull request lets a plugin add connector types with the same
definition schema, and makes them behave exactly like built-in
connectors
> - The benefit is that a connector can ship as a plugin, and an
operator controls it with the same Connectors page and the same access
rules

## Linked Issues or Issue Description

**Subsystem affected**
Plugins, Connectors (tool access), tool gateway

**Problem or motivation**
A plugin cannot add a connector type to the Connectors catalog. Plugin
tools (`tools` in the manifest) do not use connections, credentials per
account, agent installs, policies or the Connectors page.

**Proposed solution**
Add a manifest field `apps` (gated by a new `connectors.provide`
capability). Each entry uses the built-in app-definition schema. The
host namespaces the slug by plugin key and runs setup, OAuth, discovery,
grants, policies and audit through the existing code. This is the first
of two pull requests. The second adds a `plugin` transport so a plugin
worker can serve the tools itself.

**Activation rules (product requirements)**
1. Installing or enabling a plugin grants no tools. It only adds
connector types to the catalog.
2. Connecting an account grants no tools.
3. An agent gets the tools only from an agent install or a company
install of an active connection. Discovery and invocation both enforce
this.
4. A disabled or uninstalled plugin (or an upgrade that removes the app)
hides its connector types and connections from the Connectors APIs and
they reach no agent. Saved setup survives disable and comes back on
re-enable. Uninstall keeps rows archived for audit and revokes
credentials.


**Alternatives considered**
Keep plugin tools (`tools`) as the only plugin path. Rejected: they
bypass connections, per-account credentials, per-agent installs and the
Connectors page.

## What Changed

- `packages/shared`: new capability `connectors.provide`; manifest
`apps` validated with `appDefinitionSchema`. The validator rejects apps
without the capability, duplicate slugs, host-derived fields (`plugin`,
`aiConnectionRouter`, `experimentalFeatureKey`, `availability`), and
methods that need provider code in the host (REST adapters, chat
channels, AI runtimes).
- `packages/shared/src/plugin-apps.ts`: reversible namespaced slugs
(`plugin-<hex plugin key>-<slug>`), the catalog entry builder, and
`reviewedToolRisk`.
- `packages/shared/src/app-definitions.ts`: catalog lookups
(`getConnectableAppDefinition`, `getAppStoreDefinition`, ...) also
resolve the current plugin apps. The set is replaced on each read of
plugin state, so a built-in app is never shadowed.
- App definitions can declare `reviewedTools` (reviewed
read/write/destructive names and the risk for unlisted tools).
Circleback's reviewed risk moves from
`server/src/services/connectors/circleback/risk.ts` into
`circleback.json`, with the same classification. A copy of the
definition now classifies tools the same way.
- `validators/json-schema.ts`: `jsonSchemaSchema` moves to its own
module to break an import cycle between the plugin and app-definition
validators (still re-exported from `validators/plugin.ts`).
- `server/src/services/plugin-app-catalog.ts`: reads ready plugins from
the `plugins` table and answers "is this connection's connector
available". It does not trust in-memory lifecycle events, because
several lifecycle managers and processes can share one database.
- Gateway: connected-tool discovery and `resolveConnectionGrant` drop
connections from an unavailable plugin connector. Built-in connections
do not read the plugins table.
- Connectors APIs: the gallery lists ready plugin apps (with the same
Cloud connector profile filter as core); `listConnections` and
`listApplications` hide unavailable plugin connections; setup entry
points refresh plugin apps first.
- Plugin uninstall archives the plugin's connections through the
standard connection removal (grants revoked, owned credentials deleted,
rows kept).
- UI: the gallery response registers plugin apps for the setup screens;
`/apps/connect?source=plugin-…` loads the gallery first and shows an
unavailable state when the plugin is disabled.
- Docs: `doc/plugins/PLUGIN_SPEC.md` section 11A and the capability
list.

Design choices where the spec left room (picked to match "a plugin
connector behaves like a built-in connector"):
- Only `ready` plugins provide connectors. `error` and `upgrade_pending`
hide them, the same as the AI connection router plugins.
- `rest_api` is rejected for plugin apps in this PR. Core REST
connectors need reviewed adapter code in the host, so a definition alone
cannot work. The `plugin` transport in the next PR covers that case.
- Disabled plugin connections are hidden from list endpoints. Reading
one connection by id still works, so audit links keep resolving.

## Verification

- `packages/shared`: `npx vitest run src/plugin-apps.test.ts
src/validators/ src/app-definitions.test.ts` (416 passed). New tests:
manifest validation, slug namespacing, no shadowing of built-in apps,
reviewed risk.
- `server`: `npx vitest run
src/__tests__/plugin-connectors-circleback-parity.test.ts` (new). A
test-only plugin
(`server/src/__tests__/fixtures/plugin-connectors/circleback-parity/`)
ships an unmodified copy of `circleback.json`. The same journey runs
against built-in Circleback and the plugin copy with a recorded DCR
OAuth + MCP fixture, and must give identical catalog risk/status,
suggested defaults and per-agent tool sets. It covers each rule:
- Rule 1: the gallery adds only the namespaced type; no connections
exist; no agent tool changes.
  - Rule 2: two signed-in accounts, no agent sees a tool.
- Rule 3: an install for agent A gives only A the tools of that
connection; agent B is denied on invocation; each account calls the
provider with its own credential.
- Rule 4: disable hides the type, both connections and applications,
removes tools from discovery and invocation (404) and blocks new setup,
while built-in Circleback is unchanged; installs and grants are
byte-identical; re-enable restores tools with the same names; an upgrade
that removes the app hides it; uninstall archives both connections,
revokes grants and deletes owned secrets.
- Red/green: removing the gateway gate or the declared review makes the
test fail.
- An env-gated live check (`PAPERCLIP_CIRCLEBACK_LIVE=1`) starts the
real Circleback sign-in for both connectors and compares the requests.
It never runs in CI. I did not run it for this PR.
- Regression: `ai-connection-router`, `connector-tool-policy`,
`mcp-apps`, `plugin-lifecycle-restart`, `tool-access-service`,
`braintrust-connection`, `remote-mcp-connectors`,
`connection-intents-service`, `tool-gateway` (796 passed).
- `ui`: `app-connect-policy.test.ts`, `AppsConnect.test.tsx` (new plugin
cases) pass.
- `tsc --noEmit` for shared, server, ui (only pre-existing ungenerated
Storybook catalog errors), plugin SDK and CLI.
- `pnpm verify --explain` selects a broad plan because a doc under
`doc/plugins` changed; CI supplies that coverage.

## Risks

- No migration. Built-in connectors only gain one string check per
discovery; the plugins table is read only when a plugin connection is
present.
- Circleback's reviewed risk is now data. The behavior is the same, and
the existing policy tests and new tests cover it.
- Connection intents (agent connection requests in chat) still search
only built-in apps. Plugin connectors can be set up from the Connectors
page. This is a follow-up.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:51:39 +00:00
DottaandClaude Opus 5.5 c652c813bc feat(google-workspace): per-service transport switch for REST-backed tools (off by default) (#22016)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use Google Workspace through one built-in `google-workspace`
connection and the tool gateway
> - Today every Google tool call goes to Google's hosted Workspace MCP
servers, which are a Developer Preview and cannot be offered to the
public
> - #22000 added Paperclip's own Gmail tools over Google's REST APIs in
`server/src/google-workspace-rest-tools/`, but nothing calls them yet
> - This pull request adds one per-service transport switch and connects
the gateway to the REST tools behind it. Every service stays on Google's
MCP servers, so production behavior does not change
> - The benefit is that a service moves to the REST tools with a
one-line, reviewed change once all its tools exist, and moves back the
same way

## Linked Issues or Issue Description

Refs #22000

**Subsystem affected**
Tool gateway and catalog discovery for the built-in Google Workspace
connection.

**Problem or motivation**
Google's Workspace MCP servers are a Developer Preview. Paperclip
launches on its own tools over Google's REST APIs. The REST tools exist
(#22000), but the gateway cannot call them, and there is no way to
choose them per service.

**Proposed solution**
Add a per-service transport setting (`google_mcp` or `paperclip_rest`)
in the shared Google Workspace module. Add one server adapter that is
the only importer of the REST folder. When a service is set to
`paperclip_rest`, the gateway call path and discovery use the adapter in
process. All gateway controls stay the same.

**Alternatives considered**
A loopback or hosted MCP endpoint for the REST tools. It adds a
connection, JSON-RPC framing and the protocol probe to every call, and
authenticated public deployments refuse loopback addresses. An
environment variable or instance setting. A plain constant keeps every
flip a reviewed code change.

## What Changed

- `packages/shared/src/google-workspace-connectors.ts`: adds
`GOOGLE_WORKSPACE_TOOL_TRANSPORT` (every service is `google_mcp`),
`transport` on the result of `googleWorkspaceToolTarget`, and
`googleWorkspaceServiceToolNames`.
- New `server/src/services/google-workspace-rest-dispatch.ts`: the only
importer of the REST folder. It keeps only the bearer token from the
gateway's credential headers, passes the gateway's egress-guarded fetch
and cancellation signal, and returns either the MCP result or a token
problem. `googleWorkspaceRestTransportGaps()` lists allowlisted tools
that a `paperclip_rest` service does not implement. Today Calendar is
eligible (9 of 9) and Gmail is not (3 of 9); nothing is flipped.
- `server/src/services/tool-gateway.ts` (`executeRemoteHttpTool`): for a
`paperclip_rest` tool, the gateway rechecks the current grant, calls the
adapter, and returns through the same result handling as the MCP path.
Google's insufficient-scope answer maps to `oauth_insufficient_scope`. A
rejected token forces a refresh and returns
`oauth_refreshed_retry_required` without a replay. A passed deadline
returns `tool_timeout`. The audit record names the protocol `Google
Workspace REST`. The grant recheck is now one shared helper for both
paths. A timed-out Google Workspace write, on either transport, now
returns `tool_timeout` with a warning that the change may or may not
have been made; a timed-out read keeps the plain timeout message.
- `server/src/services/tool-access/catalog/discovery.ts`: for a
`paperclip_rest` service, discovery and health run one REST probe with
the caller's token, then list the REST tools under the same names and
scope filter. A 401 refreshes and probes again. A second 401 on a
managed grant marks it `needs_reauthorization`. Any other probe failure
makes only that service unavailable.
- `server/src/google-workspace-rest-tools/`: adds `probe.ts` and its
export. Each service has an ordered list of cheap reads with the scopes
that allow each (Gmail: `users.getProfile`; Calendar: one calendar list
entry, one primary-calendar event, or one minute of free/busy). The
probe uses the first read the grant allows. A grant that allows none is
listed without a probe, and an insufficient-scope answer never drops a
service. It also and lets `boundary.test.ts` allow exactly the adapter
as the importer. The README now describes the implemented seam, how to
flip a service, and the removal procedure.
- `doc/connections/GOOGLE-WORKSPACE.md`: describes the transport
setting.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the changed shared files.

## Verification

- `cd server && npx vitest run
src/__tests__/google-workspace-rest-transport.test.ts`: 15 tests. They
were red before the gateway and discovery changes (10 failed), and green
after. With Gmail set to `paperclip_rest` through a test-only change of
the map, the tests use the real managed setup, discovery, gateway,
policy and screening with a fake Google fetch. They prove: discovery
lists the same tool names filtered by scopes after one probe; a read
runs with the caller's token and is audited; a draft waits for approval
and then runs; a tool outside the grant and a narrowed grant are refused
before any Google request; Google's insufficient scope maps to
`oauth_insufficient_scope`; a 401 refreshes the token, is not replayed,
and the retry uses the new token; an injected email is blocked; a passed
deadline aborts the Google request and returns `tool_timeout`; the
experimental-feature gate stops calls and discovery; a failed Gmail
probe leaves Docs usable and Gmail comes back; a lasting 401 asks for
reconnection. A timed-out draft carries the uncertain-write warning in
the error, the invocation record and the audit entry on both transports,
and an approved REST draft whose request was lost returns the warning.
With the default map, a call goes to `gmailmcp.googleapis.com` and the
adapter is never called.
- `cd server && npx vitest run
src/services/google-workspace-rest-dispatch.test.ts
src/google-workspace-rest-tools`: eligibility (no partial flip),
bearer-only forwarding, signal and fetch pass-through, token outcome
mapping, probe, boundary.
- `cd packages/shared && npx vitest run
src/google-workspace-connectors.test.ts`: every service defaults to
`google_mcp`.
- Existing Google tests are unchanged and pass:
`google-workspace-injection-screening.test.ts`, the Google cases in
`tool-gateway-service.test.ts` and `tool-access-service.test.ts` (142
Workspace/Google/Gmail cases).
- `npx tsc --noEmit` for `packages/shared` and `server`: clean.
- `pnpm verify`: exit 2 (a doc under `doc/` is a global CI input), so
required CI supplies broad coverage.
- Speed, stubbed fetch, 300 calls after warm-up: the adapter adds about
15 µs per call. A full `gmail__get_message` through `executeTool` took a
median of 18.2 ms on REST and 18.3 ms on MCP (p90 21.6 ms and 22.3 ms).
The REST path adds no waits, no extra round trips and no dynamic
imports.

## Risks

- Low risk for production: every service stays on `google_mcp`, and the
default-map test proves the adapter is not called.
- The server now loads the REST folder at start. It uses only modules
the server already loads (MCP SDK, zod).
- A later flip must add a probe and the API host for each new REST
service. The `Record` type of the probe table makes typecheck fail when
a REST service has no probe.
- On `paperclip_rest`, MCP resources, MCP Tasks and elicitation do not
exist for that service.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 09:49:49 -06:00
DottaandClaude Opus 5.5 d1b5cfe038 feat(connections): let users delete data read through a connection (#22010)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call connected apps, such as Google Workspace, through
Paperclip's tool gateway
> - Paperclip saves part of each tool result, approval previews, app
views and the full run transcripts that contain tool output
> - No user action deleted this data. Removing the connection left it in
place
> - Google's restricted-scope review requires that users can delete the
data an app obtained, and that the privacy policy says how
> - This pull request adds "Delete data read through this connection"
for any connection, on the connection's page and as an option when you
remove it
> - The benefit is that the picker text ("They stay there until you
delete them") is now true, and the privacy policy can link to a help
page

## Linked Issues or Issue Description

**Subsystem affected**

Connections (tool gateway audit tables, run logs, run events) and the
Connectors UI.

**Problem or motivation**

Paperclip stores content from a connection's tool calls in several
places and gives users no way to delete it:

- `tool_invocations.arguments_summary/result_summary` and
`tool_call_events.arguments_summary/request_summary/result_summary` keep
the first ~4,000 characters of each redacted value
(`summarizeToolValue`, `tool-access-policy.ts` `redactArguments`). No
retention limit.
- `tool_action_requests` keeps `canonical_arguments_summary`,
`preview_markdown` and `signed_arguments`. The linked approval card
(`issue_thread_interactions`) keeps copies in `payload.detailsMarkdown`,
`payload.toolAction.previewMarkdown`,
`payload.toolAction.argumentsSummaryJson` and
`result.toolAction.resultSummary`.
- `tool_mcp_app_views` keeps `tool_input` and `tool_result` for MCP app
views.
- Run transcripts hold full tool output: the run log (`local_file` with
an optional S3 mirror, or `s3_segments`/`s3_archive` object journals),
`heartbeat_run_events.message/payload`, and
`heartbeat_runs.stdout_excerpt/stderr_excerpt`. Neither run-log store
had a delete path.
- Connection removal (`removal.ts`) archives the connection and revokes
credentials. It deletes no content.

**Proposed solution**

A durable, resumable, company-scoped deletion request per connection.
See "Design" below.

**Alternatives considered**

- Delete synchronously in the request. Rejected: an interrupted request
leaves partial state that needs a person to retry it.
- Delete only the run events that mention the connection. Rejected: tool
output also reaches the model's later messages, so the safe unit is the
whole run transcript.
- Delete the audit rows. Rejected: counts, outcomes and hashes feed the
activity history and usage counts.

## Design

**Rule.** The job never drops a reference to content it has not
confirmed deleted. It is marked complete, and writes the
`tool_connection.content_deleted` activity entry, only after every step
succeeded. A step that cannot confirm its work throws. The request then
stays pending, the UI says "Paperclip keeps trying until it finishes",
and the worker retries it.

**Store inventory.** Every place that stores tool arguments, results or
previews. I found them from `summarizeToolValue`, `redactArguments`,
`writeAudit`, and every column or JSON field named like summary,
preview, arguments, result, toolInput or toolResult.

| Store | Content | Cleared by this PR | Proof |
| --- | --- | --- | --- |
| `tool_invocations.arguments_summary`, `result_summary`, `progress` |
Redacted request/result preview, provider progress | Yes: `summary`
becomes `""` with `contentDeletedAt`; `progress` null; size and sha256
kept | content-deletion test 1 |
| `tool_call_events.arguments_summary`, `request_summary`,
`result_summary` | Same previews | Yes, same | test 1 |
| `tool_access_audit_events.details.argumentsSummary` (`writeAudit`) |
Argument preview | Yes, same; decision, ids, size, hash kept | test 1 |
| `tool_action_requests.canonical_arguments_summary`,
`preview_markdown`, `signed_arguments` | Approval preview and arguments
| Yes, for settled requests | test 1, card retry test |
| Approval card in `issue_thread_interactions`:
`payload.detailsMarkdown`, `payload.toolAction.previewMarkdown`,
`payload.toolAction.argumentsSummaryJson`,
`result.toolAction.resultSummary` | Approval preview, arguments,
executed result | Yes. Cards are found from every settled request of a
batch, in the same transaction | test 1, card retry test |
| `heartbeat_runs.context_snapshot` and `agent_wakeup_requests.payload`
(`toolAction(s)[].resultSummary`,
`paperclipAgentMessage.untrustedToolResults[].resultSummary`) | Approved
action result delivered to the agent | Yes, for settled runs and settled
wakes | test 1 |
| `tool_mcp_app_views.tool_input`, `tool_result`, `model_context` | App
view input, result, shared context | Yes: rows deleted | test 1 |
| Run log: object journal, archive journal, original `local_file` log
and its legacy S3 mirror | Full transcript | Yes, every copy, for each
settled run that called the connection or received an approved result |
run-log-store tests, test 1 |
| `heartbeat_run_events.message`, `payload` | Transcript events | Yes,
except structural events (lifecycle, terminal, presentation, usage,
compaction, session/turn start, `native.*`) | test 1 |
| `heartbeat_runs.stdout_excerpt`, `stderr_excerpt`, and every string
`summary`, `result`, `message`, `text`, `output`, `content` or `body` at
any depth of `result_json` (including `nativeResult`, `acceptedResult`,
`semanticResult` and their `continuation`) | Output excerpts and final
answer | Yes; schema and control fields kept | test 1, canary |
| `approvals.payload` (formal board approval) | Tool name, risk and
hashes only | Not needed: no content | — |
| `tool_mcp_tasks` | Request hash and sealed provider task id; no result
| Not needed: no content | — |
| `tool_mcp_elicitations.questions` and their question cards | Questions
the provider asked a person | No. These are provider prompts, not tool
results, and the answered question stays in the task. Follow-up if
wanted | — |
| `tool_mcp_skills`, `tool_mcp_skill_files`, `tool_catalog_entries` |
Skills and tool descriptions the provider publishes | No. These are
provider definitions, not data read through tool calls | — |
| Transcripts of later runs that read an app view's `model_context` |
Shared app context | No. Nothing records which runs read it. The help
page says so. Follow-up | — |
| `issue_comments.body`, issue documents (including continuation
summaries), files, chat messages | Task content written by agents or
Paperclip | No, by design. The user removes it by editing or deleting it
or the task. The confirmation says in bold that anything agents wrote
into tasks is not removed, and so does the help page. The canary
excludes `issue_comments.body` by name for this reason | canary asserts
it is kept |
| Raw provider traces | Full provider traffic | No. Admin opt-in,
encrypted, expire within 24 hours | — |
| Feedback exports already sent | Run logs the user chose to share | No.
Already outside Paperclip | — |
| Content of runs of a deleted task or company | Same stores | No.
Follow-up | — |

**Durable request (schema change).** I added one table,
`tool_connection_content_deletions`, and migration `0490`. I followed
`doc/DATABASE.md`: edited the schema, exported it from
`packages/db/src/schema/index.ts`, ran `pnpm db:generate`, and ran the
migration drift and compatibility tests. The classifier marks it
`online_compatible` (a new table and its indexes only). No existing
table can hold a durable per-connection request with a cutoff time,
cursor, counts and retry state. Without it, an interrupted deletion
could resume only when a person pressed the button again.

**Resume after restart.** The worker uses the shared
`registerDeadlineWork` lifecycle on a new
`tool-connection-content-deletion` queue. The request insert signals the
queue in the same transaction. At server start the worker runs once,
`next()` finds any pending row, and processing continues from the saved
phase and cursor. Nobody presses anything. `next()` returns a deadline
only while a pending row exists, so an idle instance sends no recurring
query. A failure records `attempts` and `last_error` and retries after
30 seconds.

**One run selection.** `runsWithStoredContentSql` selects the runs whose
transcripts are deleted: runs that made a tool call or tool-call event
through the connection, and runs whose context lists one of the
connection's approved action requests, limited to runs that still hold a
log, an excerpt, result text or a content event. The dialog count, its
in-progress warning and the job's `runs` phase all use this one
function.

**Phases.** `invocations`, `events`, `auditEvents`, `appViews`,
`deliveredResults` (50 action requests per page; matching runs and wakes
are locked with `FOR UPDATE` before their documents are rewritten),
`runs` (the shared selection, 50 run ids per page, then one run at a
time), `done`.

**Canary test.** `tool-connection-content-deletion-canary.test.ts` puts
a unique marker in a tool call's arguments and result, an approved
action with a preview, a result delivered to a later run and its wake, a
native run that quotes the marker in its answer and events, a run log
migrated from a local file with an S3 mirror to an object journal, an
MCP App view and the audit tables. It checks that the marker is first
present in the database, the object store, the legacy mirror and the log
files. After the deletion it scans every `text`, `varchar`, `json` and
`jsonb` column of every table found through `information_schema` (no
hand-written table list), every object in both stores and every file
under the log directory. Any hit fails with the table and column or the
key. It runs three times: a complete deletion, a deletion interrupted
twice by storage failures and resumed, and two deletions for different
connections on one task at the same time. The only exclusion is
`issue_comments.body`, named with its reason.

**Batching and time budget.** Tool rows are read in pages of 200 with
keyset pagination on the existing `(company_id, created_at)` indexes,
bounded by the request time. Each page's writes and the saved progress
commit in one transaction. Counts are therefore exact, and a crash
repeats at most one page, idempotently. Runs advance one at a time from
the shared selection. Each 1,000-event page is its own step. Log removal
is its own step and runs before its progress transaction, so object
storage calls never run inside a database transaction. The 10-second
budget is checked after every step. Nothing changes on the agent boot or
tool-call path.

**In-flight work.** A run counts as settled only when its status is
terminal, `finished_at` is set, cost accounting is done, and no native
finalization is open. Calls, approvals, events, app views, delivered
results and transcripts tied to other runs or wakes are skipped and
counted. The confirmation says "N runs are still working and will be
skipped. Delete again after they finish."

**Permissions.** Board users with configure access (connection manager
or creator) can delete, like `DELETE /tool-connections/:id`. Agents,
viewers and other companies are refused. The activity entries
`tool_connection.content_deletion_requested` and
`tool_connection.content_deleted` hold counts only.

**UI reads.** The dialog loads counts once when it opens. While a
deletion runs, it polls `GET
/tool-connections/:id/stored-content/deletion`, a single indexed row
read, every 2 seconds. Polling stops when the job completes or the
dialog closes. The Delete button is not gated on counts.

## What Changed

- `packages/db`: new `tool_connection_content_deletions` table and
migration `0490` (online compatible: new table and its indexes only).
- `packages/shared`: `ToolConnectionStoredContent`,
`ToolConnectionContentDeletion` and counts types; optional
`contentDeletedAt` on `ToolRedactedValueSummary`.
- `server`: `RunLogStore.remove`, which deletes every copy of a run's
log (`composeRunLogStore`); `content-deletion.ts` service; deadline
worker wired into startup and pre-drain; `GET
/tool-connections/:id/stored-content`, `POST` and `GET
/tool-connections/:id/stored-content/deletion`, and `DELETE
/tool-connections/:id?deleteStoredContent=true`; OpenAPI entries.
- `ui`: "Delete data" row on the connection's Permissions tab with a
confirmation that shows counts, skipped runs, and "This cannot be
undone"; "Also delete data read through this connection" checkbox in the
Connectors removal dialog.
- Docs: user help
`docs/guides/board-operator/deleting-connection-data.md` (for the
privacy policy link) and an API note in
`doc/connections/CONNECTOR-PLAYBOOK.md`.
- `tests/runner-e2e/source-sdk-public-inputs.json` refreshed for the
shared type change.

`server/src/routes/tool-access.ts` (3,010 lines) and
`ui/src/pages/apps/Browse.tsx` (1,722 lines) grow by about 40 and 20
lines. The routes reuse that file's connection access checks; the dialog
reuses the existing removal confirmation. The new logic lives in its own
files.

## Verification

- `server/src/__tests__/tool-connection-content-deletion.test.ts`
(embedded PostgreSQL): content is gone from every "Yes" store in the
inventory; rows, hashes and sizes stay; another connection and another
company are untouched; a running run is skipped and is deleted after it
settles; a storage failure leaves the request pending with `retrying`,
and the next pass completes it; an approval-card update that fails once
is cleared on the retry; an unrecognised log backend keeps the pointer,
keeps the job pending and writes no completion entry; a zero time budget
yields after every step and still finishes; a repeat request returns the
pending one; agents and outsiders are refused; removal with
`deleteStoredContent=true` queues the deletion.
- `run-log-store.test.ts` and `object-run-log-store.test.ts`: `remove`
deletes local and mirrored logs and object journals, removes a migrated
run's original local file and mirror with its journal, refuses an object
log when object storage is not configured, and is idempotent.
- `ui/.../StoredContentDeletion.test.tsx`: confirmation text, counts,
skipped-run warning, and the request.
- Red: these tests fail on `master` because `remove`, the service and
the routes do not exist. Green on this branch.
- Ran: the tests above, `openapi-routes`,
`server-startup-feedback-export`, `tool-connection-removal`,
`packages/db` migration drift and compatibility tests, all
`ui/src/pages/apps` tests (950), `pnpm check:token-gates`, server and UI
`tsc --noEmit`. `pnpm verify` exits 2 (global CI input changed), so
required CI supplies broad coverage.

## Risks

- Deleting is permanent by design. The confirmation says so.
- A `local_file` log written on another replica's local disk stays on
that pod's disk until the pod is replaced; its S3 mirror is deleted. New
logs on S3-configured instances use object journals, which are deleted.
- On versioned buckets, earlier object versions remain until the
bucket's lifecycle policy removes them (the existing journal delete path
does not remove versions).
- Not in this PR (follow-ups): the "No" rows of the inventory (except
agent-written task content, which is out of scope by design), mainly the
same deletion when a task or a company is deleted, and later runs that
read an app view's shared context.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 08:55:35 -06:00
DottaandClaude Opus 5.5 55b692d532 feat(telemetry): send Cloud server telemetry directly to PostHog (#22006)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends first-party telemetry events through a collector.
The collector anonymizes the events and copies them into PostHog.
> - Paperclip Cloud is our own service. It needs real IDs (workspace,
user, agent, task) so that a person can go from a PostHog event to the
exact workspace and task.
> - The collector path hashes these IDs. It also cannot link a server
event to the browser session that caused it.
> - This pull request adds a server-owned direct sender for Cloud only.
Self-hosted installs keep the collector path with no change.
> - The benefit is real IDs and session links on Cloud, and no change to
self-hosted privacy.

## Linked Issues or Issue Description

**Subsystem affected**

Server telemetry (`packages/shared/src/telemetry`,
`server/src/telemetry.ts`).

**Problem or motivation**

All Cloud telemetry goes through the collector. The collector hashes the
install ID, agent IDs and task IDs. So, on Cloud, you cannot find the
workspace, user or task of an event. Server events also cannot show in a
browser session replay.

**Proposed solution**

On a Cloud-managed server that has `PAPERCLIP_POSTHOG_PROJECT_TOKEN`,
`PAPERCLIP_CLOUD_STACK_ID` and `PAPERCLIP_POSTHOG_ENVIRONMENT`, send
each telemetry batch straight to PostHog with `posthog-node`. Use the
same event shape that the telemetry backend loader makes today, and add
the real IDs. Keep every other install on the collector.

**Alternatives considered**

Send real IDs through the collector. We did not do this, because the
collector exists to anonymize self-hosted data.

**Roadmap alignment**

This is Cloud analytics infrastructure. It does not change a core
product feature.

## What Changed

- Add `server/src/services/cloud-posthog-config.ts`.
`resolveCloudPostHogConfig()` returns `null` unless the instance is
Cloud-managed and has a token and a stack ID.
`isPostHogDirectDeliveryEnabled()` also requires an environment and the
kill switch on (`PAPERCLIP_POSTHOG_DIRECT` is not `0`).
- Add a narrow `TelemetryTransport` seam to `TelemetryClient` in
`packages/shared`. With a transport, each flushed batch goes to the
transport and never to the collector. Without one, the client does what
it did before. A transport can capture per-event context at track time.
The context does not go into the collector envelope.
- Typed helpers hash `task_id` unless the transport sets
`hashesPrivateRefs: false`. Private `source_ref` values (for example a
private repository name) are customer content, so they stay hashed on
every path, Cloud included, with `source_ref_hashed: true`.
- Add `server/src/services/posthog-direct/`:
- `event-shape.ts` builds PostHog messages with the loader's event
names, metadata keys and key order, the deterministic event uuid
formula, and the unknown-enum fallback. It drops events that the backend
quarantines (unregistered names).
- `enum-fallbacks.generated.ts` holds the backend fallback rules. Enum
values come from the shared generated contract.
- `transport.ts` loads `posthog-node` with dynamic `import()` only on
the first Cloud send.
- `request-context.ts` is Cloud-only Express middleware. It reads only
`X-POSTHOG-SESSION-ID` and `X-POSTHOG-WINDOW-ID`. It ignores
`X-POSTHOG-DISTINCT-ID`. The acting user comes only from the
authenticated request actor. Context stops when the response finishes,
so later background work does not use that user.
- `server/src/telemetry.ts` selects exactly one sender per process. On
start, the Cloud sender sends `$groupidentify` (group `workspace`, key =
stack ID) and one `cloud.workspace_started` event with `install_hash`
(sha256 of the raw install ID). Both go straight to `posthog-node` and
are not in `PAPERCLIP_EVENTS`.
- `distinct_id` is the acting user ID, else `workspace:<stackId>`.
Events without an acting user keep `$process_person_profile: false`.
GeoIP stays off on all server events.
- Shutdown flushes the client and then shuts down `posthog-node`.
- Add `scripts/generate-posthog-direct-fixtures.mjs`. It runs the
telemetry backend's own ingest and loader on one envelope and writes the
golden fixture and the fallback table. The script header tells how to
regenerate them.
- Update `packages/shared/src/telemetry/README.md` with a "Cloud Direct
Delivery" section about the privacy boundary.
- Add `posthog-node@5.38.2` to `server/package.json`. This PR does not
change `pnpm-lock.yaml`, because the lockfile refresh workflow owns it.

Intended differences from the loader output on Cloud: `distinct_id`
(user or workspace, not the HMAC install ID), raw `agent_id`, raw
`task_id` (private `source_ref` stays hashed), and these added
properties: `stack_id`, `company_id` and `user_id` when known,
`environment`, `$groups`, `$session_id` and `$window_id`. The uuid
formula is the same. The uuid is different only when the dimensions are
different (raw `task_id`).

## Verification

- `pnpm verify` returned exit 2 (a global CI input changed: the new
script). I ran focused checks. Required CI supplies the broad coverage.
- Focused tests pass: `packages/shared` `src/telemetry` (6 files, 42
tests). Server: `posthog-direct-telemetry`,
`posthog-direct-event-shape`, `cloud-posthog-config`,
`shared-telemetry-events`, `heartbeat-status-preserving-telemetry`,
`issue-thread-interactions-telemetry`, `native-run-finalizer-telemetry`,
`agent-task-run-telemetry`, `error-handler` (9 files, 115 tests), and
`server-startup-feedback-export` (133 tests).
- `tsc --noEmit` passes for `packages/shared`, `server` and `cli`.
- New tests:
- Config matrix: not Cloud, no token, no stack ID, no environment, kill
switch, browser and replay flags.
- Self-hosted: `posthog-node` is never loaded, the collector gets the
batch, and the envelope bytes are the same as before (with `task_id`
hashed). Tracing headers are ignored.
- Cloud: a private `source_ref` stays hashed and the raw repository name
never reaches the sender.
- Cloud: one batch goes only to PostHog. Real IDs, `$groups`,
`environment` on every event. `cloud.workspace_started` reaches the
PostHog sender with `install_hash`. No other event has `install_hash`.
- Golden: our builder output equals the backend loader output, except
the intended differences above. Key order is the same.
- `distinct_id`: user, agent and anonymous requests. A spoofed
`X-POSTHOG-DISTINCT-ID` never changes `distinct_id` or `user_id`. A bad
session header is dropped. Work after the response ends does not keep
the user (red without the response-end guard, green with it).
- Kill switch and a missing environment both fall back to the collector.
- Local run with a real server and a local capture server (no real
PostHog project):
- Cloud mode (authenticated, local Postgres, `PAPERCLIP_POSTHOG_HOST`
set to the capture server): the capture server got `$groupidentify`,
`cloud.workspace_started`, `server.started` and `goal.created` on
`/batch/`. `goal.created` from a Cloud tenant user had `distinct_id` =
that user, `user_id`, `company_id`, `$session_id`, `$window_id`,
`$groups` and no `$process_person_profile`. The spoofed distinct ID
header was not used. Nothing went to `/ingest`. `SIGTERM` ran the
`telemetry_flush` step and sent the queued events.
- Self-hosted mode (same token variables but no Cloud signal): the
capture server got only `/ingest` with the normal envelope. Nothing went
to `/batch/`.
- Runtime dependency note: `posthog-node` loads with dynamic `import()`
only on Cloud with direct delivery on. The load happens after startup,
outside the boot path. The unit test mocks the package to count loads.
The local run used the real package.

## Risks

- Medium. This is a telemetry change, so a privacy review is requested.
Self-hosted output does not change. The direct path is off unless all
three Cloud variables are set, so a merge does nothing until fleet
config sets them.
- Cloud events with an acting user create PostHog person profiles keyed
by the Paperclip user ID. This is intended under the Cloud terms.
- The fallback table can drift if the backend adds a new normalization
rule. A test fails when a rule does not match the shared enum contract.
Enum values come from the shared contract, so new enum values do not
drift.
- `posthog-node` keeps network failures in its own queue and drops
events that PostHog rejects. The client does not resend these, so events
are not sent twice.
- `pnpm-lock.yaml` is not in this PR. The lockfile refresh workflow adds
`posthog-node` after merge.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local server runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 08:54:32 -06:00
Nicky Leach bc0a236b95 feat(runner): accept a resume-compatible build for a relaunched released Runner (#21976) 2026-11-17 06:15:01 -08:00
DottaandClaude Opus 5.5 66ac5d7aa0 fix(google): keep Paperclip's Google client inside the 14 submitted scopes (#22001)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The built-in Google Workspace connector uses Paperclip's own Google
OAuth client through the Paperclip Cloud broker, or a customer-owned
client
> - Google is reviewing that client for exactly 14 scopes (Gmail
read/compose, Drive read/file, Docs, Sheets, Slides, Calendar)
> - The code still accepted the old 21-scope union for the unified
connection, and other Paperclip-managed Google methods (Gmail archive,
Chat, People, Workspace Search) could ask the same client for scopes
outside the submission
> - A consent request for an undeclared scope through the verified
client breaks the review
> - This pull request makes the 14 an explicit constant, enforces it on
every unified-connection request, callback and refresh, and stops new
consent for managed Google profiles outside it
> - The benefit is that Paperclip's Google client cannot request a scope
that Google has not reviewed, while saved connections keep working

## Linked Issues or Issue Description

Refs the plan in `doc/plans/2026-11-17-google-workspace-code-fixes.md`
(items 2, 3 and part of 10).

**What happened?**
The unified `workspace.all` profile was the 21-scope legacy union, and
`isGoogleWorkspaceScopeGrant` accepted any subset of it. The Gmail
`paperclip-archive` method asked for `gmail.modify`. The Chat, People
and Workspace Search cards had Paperclip-managed methods with Chat,
Contacts, Directory, profile and `calendar.readonly` scopes. None of
these are in the Google submission.

**Expected behavior**
Paperclip's own Google client requests, stores and refreshes only the 14
submitted scopes. New managed setup is not offered for anything else.

**Steps to reproduce**
1. Advertise `gmail.archive` (or `chat.read`) from the broker.
2. Open Apps. The managed archive/Chat method is offered, and starting
it signs a broker session for scopes outside the 14.
3. Refresh a unified grant: the signed binding lists all 21 legacy
scopes.

## What Changed

- `packages/shared/src/google-workspace-connectors.ts`: add
`GOOGLE_WORKSPACE_SUBMITTED_SCOPES` (the 14, written out by hand).
`workspace.all` now uses it. `isGoogleWorkspaceScopeGrant` checks
against it. Add `googleWorkspaceProfileWithinSubmittedScopes`.
- `server/src/services/paperclip-cloud-connector.ts`: add
`paperclipCloudConnectorConsentAllowed`. `startAuthorization` refuses a
Google profile outside the 14 before any broker call.
`appWithPaperclipCloudConnectorAvailability` does not offer those
methods for new setup, even when the broker advertises them.
- `authorization-start.ts`: a saved connection on such a profile cannot
restart consent (`paperclip_cloud_connector_scope_not_offered`, 422).
- Capping contract, the same on managed and customer-owned OAuth
(`paperclip-cloud-connector.ts` `openCredentials`, `cloud-callback.ts`,
`direct-callback.ts`, `grant-refresh.ts`): Paperclip caps Google's
reported scopes to the 14 (`googleWorkspaceSubmittedScopesIn`) and to
the saved request, then continues. A scope outside the 14 is ignored and
never enables an action. Only missing scope evidence or an empty result
marks the grant for reconnecting. Reconnecting with a new service choice
always works. What Paperclip *requests* stays fail-closed: an outgoing
unified authorization request with any scope outside the 14 is rejected.
- Docs: `doc/connections/GOOGLE-WORKSPACE.md`,
`doc/connections/GOOGLE-WORKSPACE-SERVICE-CONSENT.md`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the changed shared file.

### Investigation

**Scope flow, unified connection.** Both paths start in
`authorization-start.ts`. The requested set comes from the picker
(`googleWorkspaceSelectedScopes`), a saved draft, or the saved grant,
and must be inside the method's `scopesHint` (the 14 since the connector
was added in #15545). Managed: `cloudConnector.startAuthorization` signs
`scp` = the request; the broker validates it as a subset of its
registry; the claim is unsealed with the request as AAD;
`cloud-callback.ts` caps Google's grant to the request. Customer-owned:
`direct-callback.ts` checks Google's raw `scope`, then caps it. Refresh:
managed refresh signs `scp` = the profile definition (previously 21) and
unseals with that set as AAD; customer refresh checks the raw `scope`.

**What depends on the 21-scope set.** Only the managed refresh binding
(`scp` and envelope AAD) and tests. The broker (`paperclip-cloud`
origin/master, `src/connections/providers.ts`, `broker.ts`) accepts any
nonempty, duplicate-free subset of its `workspace.all` union for
session, claim and refresh, and seals with the signed `scp`. So
narrowing the app's `workspace.all` to 14 keeps the contract. I narrowed
the definition itself instead of adding a parallel check. A saved
unified grant can still report one of the 7 extra scopes: older app
versions signed the 21-scope `workspace.all` binding on refresh, and
Google can return scopes the same client was granted before. This PR
therefore caps those scopes instead of failing the grant.

**Managed Google methods.** See the table below. The broker's
origin/master registry has no `gmail.archive` profile, so
`paperclip-archive` could not reach the managed client today.

| App / method | Profile | Scopes | Before | After |
| --- | --- | --- | --- | --- |
| google-workspace / paperclip-workspace | workspace.all | subset of the
14 | 21 accepted | 14 only |
| gmail / paperclip-read, paperclip-draft | gmail.read, gmail.draft |
gmail.readonly, gmail.compose | offered | offered |
| drive, docs, sheets, slides, calendar / paperclip-read,
paperclip-write | per service | inside the 14 | offered | offered |
| gmail / paperclip-archive | gmail.archive | + gmail.modify | offered
if advertised | saved only |
| google-chat / paperclip-read, paperclip-write | chat.read, chat.write
| chat.* | offered if advertised | saved only |
| google-people / paperclip-read | people.read | directory, contacts,
userinfo.profile | offered if advertised | saved only |
| google-workspace-search / paperclip-read | workspace-search.read | +
calendar.readonly, chat.messages.readonly | offered if advertised |
saved only |

"Saved only": the method stays in the app definition so saved
connections resolve their method, refresh and run. New setup and new
consent are refused. Customer-owned methods are unchanged.

### Broker changes needed in paperclip-cloud (not in this PR)

1. Narrow the `workspace.all` request allowlist for `session` (and the
provider URL) to the 14. On refresh and claim, cap Google's reported
scopes to the allowlist (or ignore extras) instead of rejecting with
`provider_rejected`, and keep accepting a signed `scp` binding of the
old 21 from older app versions until they are retired.
2. Drop or disable `chat.read`, `chat.write`, `people.read`,
`workspace-search.read` for new sessions on the verified client. Keep
refresh for saved grants until the owner retires them.
3. Do not merge the `gmail-archive-oauth-profile` branch for the
Paperclip-managed client.

## Verification

- Red first, then green: `pnpm exec vitest run
packages/shared/src/google-workspace-connectors.test.ts
server/src/services/paperclip-cloud-connector.test.ts` (13 new or
changed tests failed before the change).
- Red first, then green: `pnpm exec vitest run
server/src/__tests__/tool-access-service.test.ts -t "refuses new managed
setup|saved Paperclip-managed Gmail archive|fails closed on invalid
refreshed|pilot"` (8 failed before the change).
- Red first, then green: `pnpm exec vitest run
server/src/__tests__/tool-access-service.test.ts -t "caps a|keeps a .*
grant on its submitted|reconnects a"` (4 capping tests failed on the
earlier fail-closed revision; both reconnect tests pass).
- `pnpm exec vitest run server/src/__tests__/tool-access-service.test.ts
-t "combined Workspace|Workspace|Gmail|managed"`: 161 passed. Full
`tool-access-service` and `connection-management.integration` files: 599
passed on the earlier revision.
- `pnpm exec vitest run packages/shared/src/app-definitions.test.ts
server/src/services/gmail-tool-governance.test.ts
server/src/services/gmail-archive-tool-policy.test.ts`: pass.
- `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter
@paperclipai/server typecheck`: pass.
- `pnpm verify` returned exit 2 (a global CI input changed). Required CI
supplies the broad coverage.

## Risks

- Rollout order is app first, then broker. Refresh now signs the 14. If
the broker narrows `workspace.all` first, refresh from app versions that
still sign 21 (including self-hosted instances on the broker) fails
until they upgrade.
- A saved unified grant whose provider-reported scopes include a scope
outside the 14 keeps working on its in-set scopes; the extra scopes no
longer enable any action. A grant with only out-of-set scopes needs
reconnecting, and a reconnect with a new service choice succeeds.
- When the broker narrows `workspace.all` to the 14, it must cap or
ignore extra scopes on refresh and claim (it re-validates them today),
not reject saved grants.
- Saved Chat, People, Workspace Search and Gmail archive managed
connections keep working, but their owners cannot reconnect them through
Paperclip's Google client. They see a 422 with a plain message. The
owner must decide whether to retire them.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 07:41:31 -06:00
DottaandClaude Opus 5.5 41b0f76d75 fix(runner): tell a provider its accepted result is final when it reports again (#21984)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run reports its result with `paperclip_finish` or
`paperclip_block`, and a turn's first accepted report is final
> - When the model reports again with a different result, the Runner
refuses the second report
> - After a committed blocker, the refusal tells the model to stop and
end the turn (#21959)
> - After a committed done, needs_review or yielded result, the refusal
was only "A different semantic result was already committed", so the
model did not know what to do and tried again
> - Each extra call is one more model request, and the turn holds the
agent's slot until it ends; the task's status commits only when the turn
ends
> - This pull request gives every refused later report a clear, final
message: the result is already accepted, stop calling `paperclip_finish`
and `paperclip_block`, put extra notes in the final response, and end
the turn
> - The benefit is that the model ends the turn after one refusal, so
the slot is free sooner and the task completes sooner

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave, an OpenCode agent (DeepSeek through OpenRouter)
finished a task, and the Runner accepted its `done` report. The model
then called `paperclip_finish` again with a different report. The Runner
replied only "A different semantic result was already committed". The
model repeated the call about 79 times over 277 seconds. The agent's
slot stayed busy, and the task's completion was late, because the task
status commits when the turn ends.

**Expected behavior**
After a turn commits any result (done, needs_review, yielded or
blocked), each later report with a different result gets a clear and
final instruction: the result is already accepted, so do not call
`paperclip_finish` or `paperclip_block` again, put any extra notes in
the final response, and end the turn. An identical repeat of the
accepted report stays an idempotent success.

**Steps to reproduce**
1. Start a native OpenCode turn that calls `paperclip_finish` with a
done result.
2. In the same turn, call `paperclip_finish` again with a different
summary, or call `paperclip_block`.
3. On master, the tool error is only "A different semantic result was
already committed", with no instruction to stop.

**Paperclip version or commit**
master 54f8068df4

## What Changed

- `packages/paperclip-runner/src/contracts/completion-result.ts`:
replace the bare `PRP_SEMANTIC_RESULT_CONFLICT_MESSAGE` with
`PRP_RESULT_ALREADY_ACCEPTED_MESSAGE`.
`prpSemanticResultConflictMessage` returns it for every committed result
that is not a blocker. The blocker messages share the same stop
instruction, which now names both `paperclip_finish` and
`paperclip_block`, and ask for extra notes in the final response.
- The OpenCode, Codex and Codex ACPX drivers already return
`prpSemanticResultConflictMessage(...)`, so all three get the new text
with no driver change.
- `packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs`:
add a `finish-after-finish` scenario. After the accepted
`paperclip_finish`, the fake provider sends an identical repeat, a
changed `paperclip_finish` and a late `paperclip_block`.
-
`packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts`:
a new test checks that the identical repeat succeeds, that both
conflicting reports get the final message, that only one result is
proposed, and that the turn completes.
- `packages/paperclip-runner/src/contracts/completion-result.test.ts`:
unit cases for done, needs_review, yielded and a missing committed
result, and a check that both blocker messages name both tools and say
to end the turn.
- Not in this change: a hard bound that ends the turn after N refused
reports. In the OpenCode driver the only way to stop a provider turn is
`/abort`, which the driver reports as `turn.cancelled`, not
`turn.completed`. A safe bound needs a new terminal path in each driver,
so it is not "small". This change fixes the cause (the message) first.

## Verification

- Red: with the old message, the 4 new `completion-result.test.ts` cases
and the new OpenCode driver test fail (the refusal text does not contain
the stop instruction).
- Green: `npx vitest run src/contracts/completion-result.test.ts
src/drivers/opencode/opencode-server-driver.test.ts` in
`packages/paperclip-runner`: 2 files, 123 tests pass.
- `npx vitest run src/drivers/acpx/codex-acpx-driver.test.ts
src/drivers/codex`: 28 files, 563 tests pass.
- `pnpm --filter @paperclipai/paperclip-runner typecheck` passes.
- `pnpm verify` selected broad coverage (exit 2) because the shared
OpenCode fixture changed. Required CI supplies that coverage.

## Risks

- Low risk. Only the text of a tool error changes. Result admission,
fingerprints and idempotent repeats do not change.
- Mixed versions during a deploy: the Runner creates this text and sends
it straight to the model. The server does not read or match it (no
server, Rust or test code matches the old string). An old server with a
new Runner gets the new text. A new server with an old Runner keeps the
old text. Neither case changes a stored record or a protocol shape.
- ACPX: no ACPX profile, pin, schema or `protocol/` file changes, so no
provider `commandDigest` or pack contract changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 03:53:12 -06:00
DottaandClaude Opus 5.5 22d3f9a03c fix(native): finish a failed warm session's cleanup by itself and continue the task (#21977)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. A follow-up run
can reuse the warm session of the run before it, in a remote sandbox.
> - When a warm follow-up fails, the previous session is closed. The
close saves a backup of the provider session first.
> - The close looked up the backup owner through the task's current tool
authority. The failed follow-up had already released that authority.
Every close attempt failed with `native_harness_storage_owner_missing`,
so the close could never finish.
> - After nine failed attempts, the task's admission gate became
`operator_required`. The gate retires only for a stopped local process
or for the exact run that owns the session. A remote sandbox session
owned by an earlier run never matched. A queued continuation and a Retry
stayed queued forever, and the task showed a Retry button that could not
work.
> - Separately, a remote transport failure could never produce the local
stop proof that the safe-replacement lane waits for. Nothing continued
the task until an unrelated wake arrived 15 minutes later.
> - This pull request makes the close save its backup under its own
owner, retires a cleanup gate when the task's sandbox is confirmed
terminated, and refuses an unprovable remote transport failure at once
so the task continues in a fresh isolated run.
> - The benefit is that recovery finishes by itself, and a Retry that is
shown starts a run.

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave, a warm follow-up run on a remote sandbox failed
with `provider_transport_failed` (`PRP command run.attach timed out`).
The previous run's session close then failed on every attempt with
`native_harness_storage_owner_missing`. The cleanup gate became
`operator_required`. The continuation run was created 15 minutes later
and then stayed queued until the wave ended ("queued run kept queued
while the task's previous session finishes cleanup"). The task showed a
Retry banner, but a Retry would wait behind the same gate.

**Expected behavior**
A failed session close settles or is retired by itself with evidence.
The next run of the task starts in seconds. A Retry that is shown starts
a run.

**Steps to reproduce**
1. Run a native task on a remote sandbox with a warm session lifecycle.
2. Make the warm follow-up run fail during startup, after it has reused
the previous session.
3. Watch the previous session's close: every attempt fails with
`native_harness_storage_owner_missing`, and the queued next run of the
task never starts.

**Paperclip version or commit**
master at b2b2c1e60b (wave) and 7685b7a4fc (this branch base).

**Deployment mode**
Hosted server with remote Daytona sandboxes (restart-chaos harness).

## What Changed

- `native-session-executor.ts`: the remote checkpoint resolves its
backup owner from the current scope authority, and falls back to the
backend's own authority epoch when the scope has none. The
controller-persistence `authorityRunId` closure uses the same fallback.
A newer run that took over the session still wins.
- `paperclip-runner` `native-session-runtime.ts`: new
`nativeSessionCleanupRemoteScopes` lists the remote resources whose
quarantined cleanup holds a task. New
`completeTerminatedRemoteNativeSessionCleanupForTask` retires settled
quarantines of the task in one terminated remote resource, whichever
run's session they hold. An in-flight close keeps its entry.
- `remote-execution-termination.ts`: new `terminatedRemoteCleanupScope`
confirms that the company's newest lease on the remote resource is
released with its termination receipt and that no lease on it is still
held.
- `native-cleanup-admission-hold.ts`: queue admission now retires a held
cleanup when its sandbox is confirmed terminated. It logs the retirement
and records a `native.session_cleanup_gate_retired` run event with the
termination receipt. The local stopped-process path is unchanged.
- `native-safe-replacement.ts`: a transport failure on a remote or
non-Codex session can never produce the local stop proof. It is now
refused at once with `provider_failure_meaning_unverified`. The existing
disposition pass then continues the task in a fresh isolated run from
saved work. A local Codex run keeps the existing wait for stop proof.

## Verification

- Red/green, each verified red without its fix:
- `server/src/services/native-runtime/native-session-executor.test.ts` —
"saves a closing session's backup under its own owner after a failed
follow-up released the scope authority". Red:
`native_harness_storage_owner_missing` (the incident error).
- `server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`
(embedded Postgres) — "starts a queued run once the previous session's
sandbox is confirmed terminated, whichever run released it", plus "keeps
the run queued while a different sandbox of the task is terminated".
Red: the queued run stays queued.
- `server/src/services/native-runtime/native-safe-replacement.test.ts`
(embedded Postgres) — "continues a transport failure whose stop can
never be proven locally without waiting for a person" for a remote
sandbox (refused and continued fresh) and a local Codex run (unchanged).
Red: no denial and no hold.
- `pnpm verify` and server and runner typechecks pass locally.

## Risks

- The admission gate now trusts a provider termination receipt on the
newest lease of the same remote resource, not only one bound to the
session's own run. A reused sandbox with a newer held lease is not
retired.
- A remote transport failure no longer waits for a stop proof that
cannot arrive. The fresh continuation is isolated (write authority
revoked, new root) and stays under the existing per-task and per-minute
continuation brakes. Nothing is added before `turn.start`.
- The run.attach timeout itself is not fixed here. The controller held
the attach frame behind a checkpoint flush to the follow-up's writer.
The run's own diagnostics were lost because they were written to the
predecessor's run log. This PR makes the failure recover by itself; the
attach stall needs a separate fix with better diagnostics.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use and code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 02:25:02 -06:00
DottaandClaude Opus 5.5 7685b7a4fc fix(native): continue a run that reports a system environment blocker (#21959)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run ends with one semantic report: `paperclip_finish` or
`paperclip_block`
> - An agent can report a system environment blocker, for example when
it thinks its tool results are corrupted
> - Today that blocker binds to the same agent, its wake is dropped, and
the task stays blocked with no Retry
> - In a restart-chaos rerun, an OpenCode agent reported such a blocker,
then recovered in the same turn and did all the work, but its later
`paperclip_finish` was refused with an opaque conflict error
> - This pull request makes Paperclip continue a system environment
blocker automatically in a fresh session, and tells the agent clearly
that a turn's accepted blocker is final
> - The benefit is that a temporary environment problem no longer needs
a person to unstick the task

## Linked Issues or Issue Description

**What happened?**

A native OpenCode run reported `paperclip_block` with owner `{kind:
system, name: execution-environment}`, `reasonCode:
environment_unavailable` and the unblock action "Resume/retry the run".
In the same turn the agent recovered: it saved the task document and
posted the requested comment. Its later `paperclip_finish` failed with
"A different semantic result was already committed". The run succeeded
with disposition `blocked`. The status decision bound a task-wide
blocker to the same agent. Both owner wakes were skipped
(`chat_control_recovery_proof_unresolved`). The task stayed blocked with
no Retry.

**Expected behavior**

A system environment or tool blocker is a harness problem. Paperclip
must continue the task automatically, with a bound, in a fresh session.
A provider that tries to replace an accepted blocker must get clear
guidance, not an opaque conflict.

**Steps to reproduce**

1. Run a native task with an agent that calls `paperclip_block` with
owner kind `system` and reason code `environment_unavailable`.
2. In the same turn, call `paperclip_finish` with a done report.
3. See that the finish is refused with "A different semantic result was
already committed" and that the task stays blocked with no Retry.

**Paperclip version or commit**

`master` at b2b2c1e60b.

## What Changed

- `server/src/services/native-runtime/native-environment-blocker.ts`:
new lane constants and `isNativeEnvironmentBlocker`. A blocker qualifies
when its owner kind is `system` and its reason code is
`environment_unavailable` or `tool_unavailable`.
- `status-arbiter.ts`: for the current owner's environment blocker (not
a chat, not paused), schedule a fresh-session retry
(`native_environment_blocker`). After 2 retries on the shared failure
counter, bind a board-owned blocker
(`native_environment_blocker_exhausted`) with a clear next step. Other
blockers keep their current behavior.
- `native-run-finalizer.ts`: compute the environment-blocker fact from
the accepted result, only for a succeeded run that the task's owner
still holds and that nobody stopped. Both new reason codes require the
provider-failure owner recheck under the issue lock.
- `status-decision-committer.ts`, `native-provider-capacity-retry.ts`:
materialize the retry through the existing fresh-session retry lane
(`forceFreshSession`, `retryOfRunId`, 30 s then 60 s delay, refuses a
stopped run). The retry run carries a liveness instruction that tells
the agent to check saved files, documents and comments before it repeats
a step.
- Runner `completion-result.ts` and the OpenCode, Codex and ACPX
drivers: when a turn already committed a blocker, a later different
report gets `PRP_BLOCKER_ALREADY_REPORTED_MESSAGE`. It says the first
accepted report of a turn is final, tells the agent to list the
completed steps in its final response, and says that Paperclip continues
a system environment blocker automatically. Other conflicts keep the old
text.
- I kept the one-result-per-turn rule. runnerd
(`admit_terminal_tool_authority`, `paperclip/runResult`), the native
runtime and checkpoint replay all enforce it, so a superseding second
result would need a protocol change across the Rust runner, three
drivers and replay. The automatic fresh-session retry closes the gap
without that change.
- OpenCode permission denial after reattach: not a product bug. The
`external_directory` rules come only from the run's working directory
and its instruction root, which stay the same across a hand-off. The
event log shows the same workspace and session paths after both resumes.
Only one call was denied, during the model's burst of garbled tool
calls, which matches the intended workspace confinement.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-environment-blocker.integration.test.ts`
(new). It runs the real executor, finalizer and committer on embedded
Postgres. Red before the arbiter change (2 of 3 fail), green after.
- An environment blocker schedules one fresh-session retry. The task
stays `in_progress` and shows `retry_scheduled` with no Retry button.
- With the retries spent, the task is blocked with a board-owned unblock
action.
  - A user-owned `credential_missing` blocker still binds as before.
- `cd server && npx vitest run
src/services/native-runtime/status-arbiter.test.ts`: new case for the
bound, pause hold, chat and non-environment exclusions.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/opencode/opencode-server-driver.test.ts -t "accepted blocker
is final"` (new). The fake OpenCode server blocks, then tries to finish
in the same turn. Red before the driver change, green after.
- Focused suites pass: OpenCode, Codex semantic-result and ACPX driver
tests plus runner `src/cli` (26 files, 554 tests); arbiter corpus,
tool-receipt-limit and provider-capacity integration tests (5 files, 198
tests).
- `tsc --noEmit` passes for `packages/paperclip-runner` and `server`.
- `pnpm verify` asks for broad CI coverage because the change touches a
shared runner fixture. Required CI supplies that coverage.

## Risks

- More automatic runs: each environment blocker can add at most 2 runs
in a chain. The retries share the failure counter with the other failure
lanes. The company-wide automatic-run brake admits them like any other
recovery run. Worst case during an outage is 2 extra starts for each
blocked task, and the brake holds them if they keep failing.
- A real, persistent environment problem now takes up to about 90
seconds of retries before it reaches the board. This is intended.
- Previous release: no persisted shape changes. The work assessment JSON
is unchanged, because the fact is computed from the saved result at
finalization. A run finalized by the previous release keeps its saved
decision.
- Agents see a new message when they try to replace an accepted blocker.
The tool descriptions are unchanged, so the prompt size is the same.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (code search, editing, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 22:40:37 -06:00
DottaandClaude Opus 5.5 3afb2c215b fix(server): name the blocking database sessions whenever a run's event writes stall (#21956)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server stores the events that a runner sends for each run, and
it records the identity of the runner process
> - Sometimes the server stops storing a run's events for minutes, and
the server log stays quiet
> - In two restart-chaos cases, the identity write of the run started
and then did not commit for about 197 s and 480 s
> - The existing database report only fires when the reattach phase ends
at its cap, so it did not fire for the cold-start case
> - This pull request logs the pending writes and the blocking database
sessions whenever any run write stays pending for 30 s
> - The benefit is that the next stall names the session and the query
that hold the run, so the fix can target one proven holder

## Linked Issues or Issue Description

Refs #21946

**What happened?**
Restart-chaos waves on master showed two stalls in event storage.
1. Cold start of an OpenCode run, with no restart. The runner emitted
`session.started` at 22:13:23.7Z. The server stored it at 22:16:40.97Z,
about 197 s later. The second `native.process_identity_recorded` write
of the run started at 22:13:23.706Z, and `runner.turn.submit` took
197270 ms. A second wave showed the same shape (about 191 s). The server
log had nothing in that window.
2. After a hard kill, every API GET timed out from 22:35:24Z to
22:43:23Z, server CPU was 0%, and no run events were stored. The process
identity transaction of the run began at 22:35:22.6Z and committed at
about 22:43:23Z.

**Expected behavior**
When the server cannot store a run's events, the server log names the
cause: which run writes are pending, and which database sessions hold or
wait for locks.

**Steps to reproduce**
1. Start a transaction that locks a `heartbeat_runs` row (`for update`)
and stays idle.
2. Record a process identity for that run
(`persistHeartbeatRunProcessMetadata`).
3. Before this change, nothing is logged while the write waits. After
this change, a warning names the idle holder and the waiting statement
after 30 s. The new database test does this with a 300 ms threshold.

**Relevant logs or output**
Example warning from the new test:
`runner event intake stalled; pending run writes, database sessions and
lock waits` with `operations: [{stage: "process_identity", pendingMs:
301}]` and `waits: [{state: "idle in transaction", query: "select id
from heartbeat_runs where id = ? ... for update"}, {waitEventType:
"Lock", blockedBy: [<holder pid>]}]`.

## What Changed

- New `server/src/services/run-intake-stall-watch.ts`. It keeps an
in-memory list of pending run operations: event receipts
(`PaperclipControlPlanePort.appendEvent`), event writes
(`appendHeartbeatRunEvent`, `appendHeartbeatRunEventBatch`), and process
identity writes (`persistHeartbeatRunProcessMetadata`).
- When an operation stays pending for 30 s, the watch logs one warning
per stalled run. The warning has: the pending operations and their age,
the pending count and oldest age per run, the pool size, the database
sessions per application name and state, the lock waits and holders
(`pg_blocking_pids`, query without literals, cut to 160 characters), and
the event-loop lag. Reports are at most one per 30 s. A second line logs
when a reported operation finishes.
- The watch arms a timer only while an operation is pending. It reads
the database only when it reports, through the private liveness pool
with a 5 s statement timeout.
- The server names its main pool `paperclip:<boot id>` unless
`DATABASE_APPLICATION_NAME` is set. This shows which server process owns
each session after a restart.
- `readDatabaseSessionWaits` moves from the reattach phase to
`server/src/services/database-session-waits.ts`. The reattach phase
re-exports it.
- `@paperclipai/db` exports `databaseClientOptionsFromEnv` and
`DatabaseClientOptions`.
- `doc/DATABASE.md` describes the new log lines and the pool name.
- I did not add `idle_in_transaction_session_timeout` to the main pool.
Some code holds a transaction open on purpose, and a blind timeout could
abort valid work. The new report names idle-in-transaction holders
instead.
- These are server log lines only. This change does not touch Telemetry
or OpenTelemetry.

## Verification

- `cd server && npx vitest run
src/services/run-intake-stall-watch.test.ts` (5 tests). The database
test holds a run row in an idle transaction, starts a process identity
write, and checks that the warning names the holder pid, its query, the
waiting statement, and the pool name. It is red without the tracking in
`run-lifecycle.ts` and `heartbeat-run-events.ts` (`expected [] to have a
length of 1`), and green with it.
- Unit tests with fake timers check one report per stalled run, the 30 s
throttle, the clear line, and that no timer stays armed when nothing is
pending.
- `cd server && npx vitest run
src/services/controller-process-identity.test.ts` checks the pool name
and the `DATABASE_APPLICATION_NAME` override.
- `npx tsc --noEmit` passes for `server` and `packages/db`.
- `pnpm verify` ran locally on the affected tests.

## Risks

- Low risk. Each run event adds one map insert and one map delete. No
database work happens while writes finish within 30 s.
- The main pool `application_name` changes from `paperclip` to
`paperclip:<boot id>`. A dashboard that filters on exactly `paperclip`
must use a prefix match, or the operator can set
`DATABASE_APPLICATION_NAME`.
- A report reads `pg_stat_activity` on the liveness pool. A report
during a full database stall can fail; the warning then has
`databaseReadError` and still lists the pending operations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 22:15:40 -06:00
Michael NguyenandClaude Fable 5.2 7b8df907d3 test(runner): copy a standalone sleep binary where /bin/sleep is a multicall binary (#21726)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Each pull request runs the Paperclip CI gates. A pull request from a
fork runs them on GitHub-hosted `ubuntu-latest` runners.
> - GitHub now routes some `ubuntu-latest` jobs to the `ubuntu-26.04`
image. On that image `ci / Verify Paperclip Runner (static checks)`
fails for each pull request, with no relation to the change under test.
> - The cause is a test fixture. Two tests copy `/bin/sleep` as a
stand-in for the daemon. On that image `/bin/sleep` is a multicall
binary that does not run from a copy with another name.
> - #21787 starts the copy with the process name `sleep`. The copy still
exits with status 1 on that image, so the two tests still fail there.
> - This pull request makes the two tests copy a standalone `sleep`
binary on such an image.
> - The benefit is that `ci / verify` shows the state of the change
again on each runner image.

## Linked Issues or Issue Description

Refs #21787 and #21792. #21792 fixed the sandbox test that failed on the
same image, so this pull request no longer changes that test. #21787
changed the runner staging tests, but two of them still fail on the
`ubuntu-26.04` image. Refs #21688 and #21692, which show the failure.

**What happened?**

`ci / Verify Paperclip Runner (static checks)` fails when the job runs
on the `ubuntu-26.04` runner image. Two tests in
`packages/paperclip-runner/scripts/stage-runner-binary.test.mjs` fail:
"atomically publishes while the previous executable is running" and "a
source rebuild selects fresh dev bytes while preserving release binaries
and their manifest". The same job passes on the `ubuntu-24.04` image and
on the self-hosted fleet.

**Expected behavior**

The check gives the same result on each runner image.

**Steps to reproduce**

1. Start an `ubuntu:26.04` container.
2. Run `cmp /bin/sleep /bin/echo`. The files are the same. Each is the
Rust coreutils multicall binary.
3. Run `cp /bin/sleep /tmp/paperclip-runnerd && /tmp/paperclip-runnerd
30`. The output is `coreutils: unknown program 'paperclip-runnerd'` and
the status is 1.
4. Run the test file from `master` in that container with Node 24. The
two tests fail.

**Operating system**

The `ubuntu-26.04` GitHub-hosted runner image, version 20261111.196.1.

## What Changed

- `packages/paperclip-runner/scripts/stage-runner-binary.test.mjs`: add
`standaloneSleep`. On Linux, when `/bin/sleep` and `/bin/echo` have the
same bytes, the two tests copy `/usr/bin/gnusleep` as the stand-in for
the daemon. On each other system the tests use `/bin/sleep` as before.

No product code changes. No workflow file changes. The branch no longer
changes `runner-bridge-tools.test.ts`.

## Verification

Tested revision: the head commit of this branch.

- `node --test scripts/stage-runner-binary.test.mjs` in
`packages/paperclip-runner`, in an `ubuntu:26.04` container with Node
24: the version from `master` fails the two tests. This version passes
them.
- The same command in an `ubuntu:24.04` container with Node 24: the two
tests pass.
- The same command on macOS: 9 of 9 tests passed.
- In the two containers one other test, "a build without release daemons
still compiles the Runner binary", fails because `pnpm` is not installed
there. It passes on macOS.
- An earlier revision of this branch, before the merge with `master`,
passed `ci / Verify Paperclip Runner (static checks)` on the
`ubuntu-26.04` image.

## Risks

- Low risk. The change is in one test file.
- The tests use `/usr/bin/gnusleep` only when the standard file is a
multicall binary. If an image has a multicall binary and does not have
the GNU file, the two tests fail with a missing file error, which names
the cause.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), through Claude Code, with tool
use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-16 18:21:00 -08:00
5e53fec10c feat(adapter-claude): add Claude Opus 4.5 to the static model fallback (#12379)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Each adapter supplies model choices for agent configuration.
> - The Claude adapter combines live discovery with a static fallback.
> - The fallback lacks Claude Opus 4.5.
> - This PR adds that model and a regression assertion.
> - Users can select it when live discovery is unavailable.

## Linked Issues or Issue Description

**What happened?**
The Claude adapter model selector omits Opus 4.5 when live discovery is
unavailable.

**Expected behavior**
The static fallback should include claude-opus-4-5.

**Steps to reproduce**
1. Use the claude_local adapter without working live model discovery.
2. Open the agent configuration model selector.
3. Look for Claude Opus 4.5.

Related model-list work: #10280 and #10327.
Searched open and closed PRs for Opus 4.5; no competing title match was
found.

## What Changed

- Add claude-opus-4-5 after Opus 4.6 and before Sonnet 5 in the static
fallback.
- Assert that the fallback returned by the server includes this model.
- Preserve the existing first model, claude-fable-5-1.

## Verification

- At head 13b46cadcf, CI build, typecheck,
general tests, Runner checks, and serialized server suites passed.
- The review workflow failed because this description lacked the
duplicate-search checkbox. It needs a rerun after this edit.
- Greptile: 5/5.

## Risks

Low risk. This adds a static model choice without changing execution
logic.

## Model Used

Original author reports Anthropic Claude Fable 5 (claude-fable-5) via
Claude Code.
OpenAI Codex assisted with adoption review and this description update.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-16 17:26:45 -06:00
c938bdfee4 perf(ci): stop printing Postgres TRUNCATE notices in server test shards (#21903)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server test suites run on 12 CI shards. Each suite gets a
disposable Postgres database from `packages/db`.
> - Many suites reset their state with `TRUNCATE ... CASCADE`. Postgres
sends one `NOTICE` for each cascaded table.
> - postgres.js prints each notice with `console.log` when a client sets
no `onnotice`. Vitest then carries each line from the worker to the
reporter.
> - On the slowest shard of a recent run, about 600k of the 760k log
lines were these notices. The noise makes the shard slower and the log
hard to read.
> - This pull request sets `client_min_messages = warning` on each
disposable test database. Every client that a suite opens then gets no
notices.
> - The benefit is a shorter slowest server shard and a much smaller
log.

## Linked Issues or Issue Description

Refs #21879 (a separate cause of the same slow shard: missing
shard-duration weights).

**What existing behavior does this improve?**
Disposable test databases send every Postgres `NOTICE` to the test
clients. The test logs print each notice.

**Subsystem affected**
Test database fixtures in `packages/db` (`test-embedded-postgres.ts` and
`test-postgres-admin.ts`). This change touches only test infrastructure.

**Current behavior**
On PR #21872 (run
[43858567594](https://github.com/paperclipai/paperclip/actions/runs/43858567594)),
the longest check was `ci / General tests (server (10/12))` at 768s. The
Vitest step took 722s of that. The other server shards took 597–729s.
The shard log had 760k lines. About 600k lines were `truncate cascades
to table "…"` notices (about 77k notice objects):

| suite | notices |
|---|---|
| heartbeat/workspace-preparation.test.ts | 19,089 |
| agent-timer-deadlines.test.ts | 16,445 |
| idle-incoming-wake.test.ts | 13,846 |
| public-mcp/events-work.test.ts | 13,156 |
| execution-control-work.test.ts | 4,485 |
| 6 more suites | about 10k |

About 20 suites already stop the notices one session at a time (`SET
client_min_messages = warning` or `onnotice = () => {}`).

**Proposed behavior**
The fixture sets `client_min_messages = warning` on the database itself.
It does this for the embedded-cluster path and for the
`PAPERCLIP_TEST_POSTGRES_ADMIN_URL` path. Warnings and errors still
show. A suite that needs notices can lower the level for its own
session.

**Reason and benefit**
The notice-heavy suites ran 15–50% over their manifest weights in this
run (for example, workspace-preparation took 193s against a weight of
173s, and idle-incoming-wake took 40s against 26s). Other suites were
near their weights. Estimate: about 0.5–1 min less on the slowest server
shard. The log on that shard goes from about 760k lines to about 160k
lines. If #21879 also merges, the server lane's critical path should go
from about 12.8 min to about 10–11 min.

**Breaking changes**
None. Only disposable test databases change.

## What Changed

- `packages/db/src/test-postgres-admin.ts`: add
`QUIET_TEST_DATABASE_SQL`. Run it on each new database on the admin-URL
path.
- `packages/db/src/test-embedded-postgres.ts`: run the same statement on
the `paperclip` database at each embedded start. It runs on each start,
not only when the template is built. Thus a template that was cached
before this change also gives a quiet database, and the template key
stays the same.
- `packages/db/src/test-postgres-admin.test.ts`: new test for the
admin-URL path.
- `packages/db/src/test-embedded-postgres.test.ts`: new tests for the
embedded template path and the fresh path.
- Each new test opens a new client, checks `SHOW client_min_messages`,
and runs a `TRUNCATE ... CASCADE`. The test fails if the client gets a
notice.

## Verification

- `cd packages/db && npx vitest run src/test-postgres-admin.test.ts
src/test-embedded-postgres.test.ts`: 2 files, 25 tests pass.
- I changed the statement to `RESET client_min_messages` and ran the new
tests again. All 3 new tests failed. Then I restored the statement.
- `cd packages/db && npx tsc --noEmit -p .`: no errors.
- To measure the time savings, compare the server shard timings of this
PR's CI run with run 43858567594.

## Risks

- Low risk. The change affects only disposable test databases.
Production databases do not use these fixtures.
- A test that expects to receive a `NOTICE` on a new client would now
not get it. Such a test can set `client_min_messages = notice` for its
session. No current suite depends on notices: the full CI run on this PR
passed.
- The time savings is an estimate. The shard planner and other suites
can change the slowest shard.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, used through Claude
Code as a Paperclip agent with tool use (shell, file edits, GitHub CLI,
CI log inspection). Standard reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 13:07:15 -08:00
f33c4cf6f7 feat(links): cache rich link status in the database and load it on idle (#21746)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task text contains links to pull requests and other work.
> - A link has useful status that should be visible when a task opens.
> - The current cache lives in the server process and the browser.
> - A new browser or server restart has no stored status to show.
> - This pull request stores link status in the database and reads it in
batches.
> - Idle page work updates stale status without a hover or a delay to
task data.

## Linked Issues or Issue Description

**Problem or motivation**

PR status is unknown in a new browser until the user points to each
link. The process cache cannot supply saved status after a server
restart. Related work: #20593 and #20654. A search found no other PR for
database link caching.

**Proposed solution**

Store status by company, viewer and canonical resource URL. Read saved
status in one indexed batch query. After document load and core request
settlement, use browser idle time to update missing or stale status. Use
the same query for hover and idle work.

**Alternatives considered**

Adding provider calls to task reads would delay task loading.
Company-wide reuse of personal GitHub results would cross viewer access
boundaries. A periodic server worker would create work when no page
needs the result.

**Roadmap alignment**

This extends the shipped rich-link feature. It does not duplicate a
roadmap project or add a new provider integration.

## What Changed

- Add an additive, repeat-safe `link_previews` migration. Store channel,
resource kind, free-form status, JSON metadata and freshness. The table
has no task relationship or GitHub status enum.
- Bound each company/viewer cache to 1,000 rows across concurrent server
instances. Remove rows expired for over 30 days during writes. Limit
provider lookups to 600 requests per minute per scope/process. Keep
cached batch reads available during backoff.
- Add a database store owned by the link service. Reuse canonical
results across tasks. Keep company and viewer isolation. Save hover and
idle results. Preserve last known status during an outage and cache
unavailable results.
- Add a board-only batch GET endpoint, including viewer-role access. It
reads at most 50 URLs with one indexed select and makes no provider
calls. Both endpoints use private, no-store HTTP responses.
- Add a page coordinator for mounted rich links. Reuse browser data,
share pending reads, and defer provider work until document load, core
request settlement and browser idle time. Limit automatic work to 200
unique links and three requests in flight. Remove queued work when links
leave the page. Keep encoded batch URLs below 6 KB.
- Keep exact navigation URLs, status styling, anchored cards and
keyboard access. Add a page-load Storybook case and browser regression.

The link service owns cache policy and provider calls. The browser
coordinator owns page timing. The shared contract adds only the batch
limit and response shape. The database adds storage because process and
browser caches cannot serve a new browser session. Plugin resolver
registration is outside this change; storage accepts future provider
channels and status values.

## Verification

- **All 93 affected tests pass:** 40 link service, real PostgreSQL and
HTTP cases, plus 53 OpenAPI cases. Two SQL regressions and one HTTP case
failed before the storage repair. Concurrent writers now retain at most
1,000 rows per company/viewer; cleanup removes long-expired rows and
keeps recent stale status. Request backoff preserves cached reads and
isolates companies/viewers. The earlier eight concurrent hydration
regressions also pass. Focused store/service/unit types pass. Commands:
`pnpm exec vitest run server/src/__tests__/link-previews.test.ts
server/src/__tests__/link-preview-store.integration.test.ts
server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1` and
`pnpm exec vitest run server/src/__tests__/openapi-routes.test.ts
--maxWorkers=1`.
- **All 50 affected server tests passed before the later server-only
repairs** on the master-integrated source: 20 recovery/deadline cases,
one authenticated review-counts HTTP case, 10 link-service cases, 13
real PostgreSQL cache cases and six link-route cases. Command: `pnpm
exec vitest run server/src/__tests__/issue-recovery-deadlines.test.ts
server/src/__tests__/tool-connection-review-counts-http.test.ts
server/src/__tests__/link-previews.test.ts
server/src/__tests__/link-preview-store.integration.test.ts
server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1`.
- Reproduced the CI watchdog assertion locally twice before the test
repair. Ordinary assignment recovery can win `wakeReason` while the
watchdog wake coalesces into the same successful run with its full
mandate. The test now verifies the persisted watchdog identity, watched
issue and stopped fingerprint, plus exactly one review run. All 21
recovery/HTTP cases passed before integrating master; all 50 affected
cases pass after integration. No runtime behavior was changed for this
repair.
- **Fresh static Storybook build and all 22 browser cases pass**, with
two workers and retries disabled. They cover states, anchored hover,
keyboard access, exact navigation, duplicate requests, reload
persistence, expiry and saved-status hydration followed by one idle
refresh. The first browser attempt could not launch because this session
lacked Chromium; installing the expected browser resolved setup. That
attempt had no behavioral verdict.
- The cache repair remains covered by four PostgreSQL red/green
regressions for expired memory and newer durable status during provider
failure/timeout, plus check-time precedence and failed-read recovery.
Earlier UI/database/focused server type and migration
safety/compatibility/replay checks remain unchanged-input evidence.
- Master `77ddf260bf9a3de3dcb8456a60f4f809d67401fd` is integrated
cleanly. Link migration 0489 and its snapshot chain are unchanged. SDK
source validation checks 545 inputs with no stale source identities or
changed runtime/lock pins. Whitespace validation passes.
- `pnpm verify --explain` returns exit 2 and requires broad CI. No full
local suite or deployed acceptance is claimed. Earlier local full-server
type checks exceeded memory; current-head CI supplies broad
type/build/test coverage.
- [Previous CI
43900336895](https://github.com/paperclipai/paperclip/actions/runs/43900336895)
passed full types/build but failed on embedded PostgreSQL startup
(`ECONNREFUSED`), the watchdog assertion above, and unrelated plan-task
browser cleanup (activity-log foreign key on company deletion). One
failed-jobs retry was requested. Its browser shard 7 now passes without
code changes; the exact deletion-race cause remains unproven. The
watchdog assertion is repaired in this revision. The previous failures
remain recorded, not counted as an overall pass.
- [Prior CI
43905482848](https://github.com/paperclipai/paperclip/actions/runs/43905482848)
passed full types, build and all eight browser shards. Several test jobs
were cancelled; the run has no overall pass. Its concurrent update
finding is repaired. [Next CI
43909197530](https://github.com/paperclipai/paperclip/actions/runs/43909197530)
also passed types, build and all browser shards, but several server jobs
were cancelled. Greptile and Paperclip Review gave that revision 5/5.
Its security scan found unbounded retained cache rows. The quota,
retention and request-limit repair above addresses that finding. All ten
review threads are resolved.
- [Current CI
43912947278](https://github.com/paperclipai/paperclip/actions/runs/43912947278),
fresh code reviews and a fresh security scan are pending on
`335d89ef1668dcdeb66bb2059512a7c2da29e273`. Auto-merge is disabled until
the current revision clears its gates.

Published head: `335d89ef1668dcdeb66bb2059512a7c2da29e273`. Local head:
`b37b87c704a2ba8a65233c689a6a743f4011ce14`. Both have tree
`643ebd19b9729b1e6a6bb695442269c3db32be3a`; GitHub tree creation
verified equality.

## Risks

- Each company/viewer retains at most 1,000 rows. Writes remove excess
rows and entries whose freshness expired over 30 days ago. Inactive
scopes remain bounded and are cleaned on the next write. Company
deletion removes all rows. No periodic worker runs. Scope writes use a
transaction lock; the existing deadline and concurrency budget cover
that work.
- Last known status can be stale until idle or hover work completes.
Provider failures keep that status through the negative-cache window.
- Automatic enrichment adds bounded provider traffic. It runs only for
mounted supported links, once per mount, with the existing timeout and
server concurrency limit.
- Cache storage is viewer-scoped because provider results can be
private. New viewers load their own results. This prevents reuse across
users with different repository access.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools, code
execution and browser automation. The runtime does not expose the exact
serving model ID or context-window size. No subagents were used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
2026-11-16 14:33:39 -06:00
Nicky LeachandClaude Opus 5.5 fb288f5666 fix(acpx): only treat an error result as a Claude sign-in failure (#21921)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Claude agents run through the Claude ACP bridge (`claude-agent-acp`
0.73.0), which Paperclip patches in `patches/`
> - The bridge reads Claude Code's final result for each turn. If the
result text contains "Please run /login", the bridge fails the turn as a
sign-in failure
> - That check does not look at `is_error`. A successful reply that
quotes the CLI's sign-in hint also fails the turn
> - Agents quote that hint when they report the output of a `claude`
command that they ran. The work is done, but the run fails with "ACP
agent reported a terminal access failure."
> - Upstream fixed this in
[agentclientprotocol/claude-agent-acp#1103](https://github.com/agentclientprotocol/claude-agent-acp/pull/1103)
(commit
[`6e97e07`](https://github.com/agentclientprotocol/claude-agent-acp/commit/6e97e071c581002b43d9556dbd1d1376ebf9516f),
released in 0.81.1), "fix: gate login result text on error state"
> - This pull request ports that upstream change into the 0.73.0 bridge
patch
> - The benefit is that a correct, finished run is not reported as a
failed sign-in

## Linked Issues or Issue Description

Upstream fix:
[agentclientprotocol/claude-agent-acp#1103](https://github.com/agentclientprotocol/claude-agent-acp/pull/1103)
(commit
[`6e97e07`](https://github.com/agentclientprotocol/claude-agent-acp/commit/6e97e071c581002b43d9556dbd1d1376ebf9516f),
released in 0.81.1). This pull request ports it.

Refs #14089. That pull request fixes the same class of problem in the
Claude CLI lane (`execute.ts`). This pull request fixes the ACP lane.

Related: #14093 shows the same error text, but it has a different cause.
This pull request does not fix #14093.

**What happened?**

An agent ran a script that calls the `claude` CLI without credentials.
The CLI printed "Not logged in · Please run /login". The agent finished
its task and quoted that text in its final reply. Claude Code ended the
turn with a successful result (`is_error: false`). The bridge saw
"Please run /login" in the result text and failed the turn with
`auth_required`. Paperclip then failed the run with "ACP agent reported
a terminal access failure."

**Expected behavior**

A successful result completes the turn. Only an error result with the
sign-in hint is a sign-in failure.

**Steps to reproduce**

1. Run a Claude agent on the ACP lane.
2. Make the model's final reply contain the text "Please run /login"
(for example, quote the output of `claude -p hi` without credentials).
3. See that the run fails with "ACP agent reported a terminal access
failure." The new test in this pull request does these steps against a
local API stand-in.

**Agent adapter(s) involved**

`claude_local` on the ACP engine, and the Runner's ACPX Claude profile.
Both use the same patched bridge.

## What Changed

- `patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch`: port
[agentclientprotocol/claude-agent-acp#1103](https://github.com/agentclientprotocol/claude-agent-acp/pull/1103)
(commit
[`6e97e07`](https://github.com/agentclientprotocol/claude-agent-acp/commit/6e97e071c581002b43d9556dbd1d1376ebf9516f),
released in 0.81.1). It has two parts, the same as upstream:
- The `success` result check is now `message.is_error &&
message.result.includes("Please run /login")`. A successful result that
quotes the sign-in hint no longer fails the turn.
- The goal-command check also requires `message.is_error`. A `/goal`
command whose successful result quotes the sign-in hint no longer rolls
the session goal back.
- Both patch comments name the upstream change. All other patch hunks
are unchanged.
-
`packages/paperclip-runner/src/drivers/acpx/acpx-runtime-adapter.claude-login-text.test.ts`:
a new test. It runs the real bridge and the bundled Claude Code against
a local API stand-in. The model's reply quotes the sign-in hint, and the
test expects the turn to complete.

## Verification

- Red: on `master`'s patch, the new test fails with
`{"status":"failed","error":{"message":"ACP agent reported a terminal
access failure."}}`.
- Green: with this patch, the new test passes (`status: "completed"`,
`stopReason: "end_turn"`).
- I ran Claude Code 2.1.286 (the bundled version) with no credentials.
Its real sign-in failure is a result with `is_error: true` and the text
"Not logged in · Please run /login". The guard still catches it. The
separate synthetic sign-in message check is unchanged.
- The goal-command part has no new test here. The upstream change adds a
test for it. The runner's `session-goals.test.ts` passes with the new
patch.
- `node --test scripts/acpx-patch-packaging.test.mjs`: 26 passed.
- `node --test
packages/paperclip-runner/test/acpx-codex-package-contract.test.mjs`: 9
passed.
- Runner real-bridge Claude tests
(`acpx-runtime-adapter.claude-*.test.ts`,
`acpx-runtime-adapter.test.ts`,
`claude-scheduling-tools.claude-acp.test.ts`,
`runner-tool-bridge.claude-*.test.ts`): 139 passed.
- `claude-local` ACP tests (`acp.test.ts`, `acp.auth.test.ts`,
`acp.quota.test.ts`): 95 passed.
- `pnpm --filter @paperclipai/paperclip-runner run
typecheck:typescript`: passed.
- I did not commit `pnpm-lock.yaml`. The patch hash changes, and CI
resolves the lockfile for this run.

## Risks

- Low risk. The change adds one condition to each of two checks in the
bridge patch. Both match the upstream change.
- The bridge's `commandDigest` does not change, because the patch does
not touch `dist/index.js`. Saved sessions are not reset.
- A sign-in failure that Claude Code reports as a successful, non-error
result is no longer caught by this check. Claude Code 2.1.286 reports
sign-in failures as error results, and upstream uses the same rule.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, subagents) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:14:57 -08:00
Devin FoleyandPaperclip cc9ff75cb8 fix(runtime): keep managed GitHub access across a controller restart (#21928)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs give agents managed GitHub access through `gh` and `git`
launchers on the provider's `PATH`
> - A provider process can keep running while the server restarts, and
the next controller adopts it
> - The detaching controller deleted the launcher directory, and the
adopting controller staged launchers in a new random directory
> - The running provider still names the old directory, so every later
`gh` or `git` call ran without credentials
> - This pull request keeps the launcher directory stable for one
provider session and task, and re-arms it after the restart
> - The benefit is that a long run keeps its GitHub identity across a
server deploy or restart

## Linked Issues or Issue Description

No public issue exists. This describes the bug in-PR, with the bug
template labels.

**What happened?**
An agent run on a Daytona sandbox used `gh` with managed GitHub access.
The server restarted during the run (a rolling deploy), and the
controller re-attached to the live provider process. After that, every
`gh` call failed with `You are not logged into any GitHub hosts`. `git
credential fill` returned no token. The agent reported that its token
had "expired" and used the unauthenticated public API for the rest of
the run.

**Steps to reproduce**
1. Start a native run with managed GitHub access on a remote sandbox.
2. Run a `gh` command in the run. It succeeds.
3. Restart the server while the provider process is still running. The
new controller re-attaches to the process.
4. Run a `gh` command again in the same run. It fails with "not logged
in", and no `Paperclip: GitHub ...` diagnostic appears.

**Expected behavior**
After the new controller adopts the provider, `gh` and `git` get
credentials from the new controller, as they did before the restart.

**Root cause**
- `createNativeGitHubAccess` staged the launchers in
`native-session-<random UUID>` and put that path in the provider's
`PATH`, `BASH_ENV`, and `GH_CONFIG_DIR`.
- On a restart detach, `githubAccess.stop()` ran `rm -rf` on that
directory (`detachReleasedWarmNativeSession`, and the run's final
`github_stop` while `nativeRunsDetachingForRestart` is set).
- The adopting controller made a new random directory. The provider
environment cannot change after spawn, so the provider never saw it.
- The shims were gone. `gh` resolved to the real binary with an empty
`GH_CONFIG_DIR`, so no Paperclip diagnostic appeared.

Refs #15354. That PR keeps ACPX sessions across launcher rotation in the
legacy adapter path. This PR covers the native runtime path.

## What Changed

- `createNativeGitHubAccess` takes an optional `launcherKey`. The
launcher directory is `native-github-<sha256(key)>`, so the same
provider session and task always get the same directory.
- The native session executor passes a key built from
`nativeSessionScopeKey` and the issue id.
- `prepareGitHubOperationLaunchers` takes an optional `broker`. It then
stages broker-file launchers and an owner-only `broker.json`. The
launchers already read `broker.json` on each operation in this mode
(`githubLauncherSource({ brokerFile: true })`). A restaged directory
therefore gives the running provider the new endpoint.
- Native access passes `broker` only for remote sandbox targets. A
sandbox provider is the one that outlives a controller restart. Local
launchers live in the shared host temp directory, so they keep the
capability in the provider environment only, as before.
- `stop({ preserveLaunchers: true })` keeps the launcher directory. A
restart detach of an active run and of a released warm session use this
option. A normal stop still removes the directory.
- A final stop does not remove a directory that a newer access in the
same process has already restaged.
- The callback bridge runtime stays in its own per-access directory. A
new bridge never shares a queue directory with an old bridge.
- `syncRemoteTextFiles` takes a `private` flag. It sets `umask 077` for
the whole staging script, so the upload copy, the partial file, and the
final file are never readable by group or other users. Local launcher
files are written to a temporary file and renamed, so a running launcher
never reads a partly written file.

## Verification

- New `native-github-access` test on a sandbox target: a live parent
process gets credentials, the access stops with `preserveLaunchers`, and
a new access with the same key re-arms the directory. The same parent
PID then gets the new controller's credentials, with empty stderr.
`broker.json` has mode `600`. The final stop removes the directory.
- New previous-release cutover test: a provider launched with the old
random-directory layout keeps working without managed GitHub after its
controller removes the directory. The new controller does not resolve
credentials for it. A provider that the new controller launches gets
credentials.
- New local regression: local launcher files do not contain the broker
capability, and there is no `broker.json`.
- New local restart test: a local provider adopted after a restart gets
a Paperclip diagnostic and no credentials. The new controller resolves
nothing for the old endpoint.
- New `sandbox-callback-bridge` test: staging a private file runs with
umask `077` and leaves the file at mode `600`.
- New `native-github-access` test: when a superseded access stops, its
successor's launchers stay usable. A different key gives a different
directory.
- New `native-session-executor` test: a run detached for restart while
active stops its GitHub access with `{ preserveLaunchers: true }`, and
creates it with a `launcherKey`. The existing overlapping-failure test
now asserts `{ preserveLaunchers: false }`.
- The new tests fail on the old behavior. I checked this by forcing
`brokerFile: false`, forcing `preserveLaunchers = false`, removing the
`umask 077` line, and writing `broker.json` for local targets.
- Passed: `native-github-access` (26), `native-session-executor` +
`github-operation-credentials` + `heartbeat-github-launchers` +
`native-stop-accounting` + `heartbeat-project-env` (1640 passed, 7
skipped), `native-runner-restart-recovery.integration` and
`heartbeat-native-restart-suspension`.
- Passed: adapter-utils `execution-target`, `github-launcher`,
`github-launcher-environment`, `sandbox-callback-bridge`; runner
`acpx/github-launcher`, `acpx/environment`,
`codex/codex-security-config`.
- `pnpm --filter @paperclipai/adapter-utils typecheck` and the server
`pnpm typecheck` pass.
- Not run locally: `native-warm-checkpoint-ownership.integration` needs
the compiled `paperclip-runnerd` binary. CI builds it.

## Previous-release story

- No persisted run, session, checkpoint, or database state changes. Only
the in-sandbox launcher layout changes.
- A provider launched by the previous release has a random
`native-session-*` launcher directory in its environment. The previous
release's controller removes that directory at the cutover restart. The
new controller cannot find it, so it does not re-arm it.
- That provider carries forward without managed GitHub until its next
spawn, which is the current behavior. The run is not refused and is not
retried. `gh` and `git` run without managed credentials, as they do
today after a restart. The cutover test covers this.
- A provider that the new release launches gets the stable directory and
survives later restarts.

## Risks

- On sandbox targets, the provider can now read the bridge capability in
`broker.json`, as well as in its environment. The file holds the same
per-access transport token that the environment already holds, not a run
token or a GitHub credential. The file and its staging copies are mode
`600`, and the broker still checks the active run and grants on each
request. Local targets write no `broker.json`.
- Local providers that outlive a restart still lose managed GitHub. This
is not a regression: master deletes the launcher directory at the
detach, so the old local provider also runs without credentials there.
With this PR, the local launchers stay in place and print a `Paperclip:
GitHub ...` diagnostic instead of a silent "not logged in". Re-arming
local providers needs a broker channel that other local runs under the
same user cannot read. That is out of scope here, because `broker.json`
in the shared temp directory was a security finding on this PR.
- On sandbox targets, `git` and `gh` children no longer inherit
`PAPERCLIP_API_URL`, `PAPERCLIP_API_KEY`, or
`PAPERCLIP_GITHUB_BRIDGE_TOKEN`. Broker-file mode removes them, as it
already did in its existing tests.
- After a restart detach, a launcher directory stays in the workspace
runtime directory. If no controller adopts the provider, that directory
stays until a later access with the same key stops, or until the
workspace is removed. It contains no GitHub credential.
- Between the detach and the adoption, `gh` gets a transport error from
the old endpoint. It then prints `broker_transport_unavailable` and
continues without credentials, as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs). Standard context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-16 12:13:04 -08:00
Devin FoleyandPaperclip 0308108531 fix(native): pipeline tool progress events with text deltas (#21533)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs store every relayed runner event in
`heartbeat_run_events`. Each stored event also locks and updates the
run's `heartbeat_runs` row.
> - Text deltas already commit in batches. Every other event, including
each `tool.execution.progressed` event, commits in its own transaction.
> - Tool receipts and usage accounting lock the same run row. Under
parallel tool calls, accounting failed with "other updates to the same
run kept conflicting", and a provider-pack receipt save hit a lock
timeout.
> - #21520 made events that arrive while a batch commits join the next
batch. This pull request adds tool progress events to the pipelined
types, so they use those batches too.
> - The benefit is fewer run-row transactions in tool-heavy turns,
without changing event order.

## Linked Issues or Issue Description

**What happened?**
A routine run started several subagents that made about 190 tool calls
in about 10 minutes. Three tool calls failed with "could not save this
run's budget or usage records because other updates to the same run kept
conflicting", and a provider-pack admission receipt failed with a lock
timeout (55P03). Every tool progress event of the run was a separate
transaction that updated the same run row.

**Expected behavior**
Observational events such as tool progress commit in batches, so they
hold the run row lock fewer times.

**Steps to reproduce**
1. Run a turn that makes many tool calls in parallel.
2. Each `tool.execution.progressed` event commits in its own transaction
and updates `heartbeat_runs`.

**Relevant logs or output**
```
could not save this run's budget or usage records because other updates to the same run kept conflicting
native provider-pack admission receipt could not be saved (cause=55P03)
```

Related: #21531 tried letting the runner relay events faster, which
would make this contention matter more. It is closed.

## What Changed

- `paperclip-control-plane-port.ts`:
- `pipelinedEventTypes` names `item.delta` and
`tool.execution.progressed`. Both go through the batched append path and
its group commit from #21520.
- The text-delta boundary is now tracked when an event is appended: a
pipelined event that is not a delta ends it, and a delta with text
starts it. The first text delta after any other event still commits at
once.
- `contracts/control-plane-port.ts`: optional `pipelinedEventTypes`, and
the `streaming` option documents that it applies to those types.
- `native-session-runtime.ts`: the runtime pipelines the port's
`pipelinedEventTypes`, and only `item.delta` when the port does not name
them.
- Tests:
- `paperclip-control-plane-port.test.ts`: three progress events appended
while a batch commits share one transaction, and a following event
commits after them.
- `native-session-runtime.test.ts`: progress appends start without
waiting only when the port pipelines them.

## Verification

- After rebasing onto current master: `pnpm exec vitest run
src/services/native-runtime/paperclip-control-plane-port.test.ts` (in
`server/`): 108 passed.
- `npx vitest run src/native-session-runtime.test.ts` (in
`packages/paperclip-runner`): 298 passed, 1 failed. `settles a governed
wait through real Runner shutdown after a missing Codex terminal` needs
the built Rust test binaries (`fake-codex-app-server`), which my
checkout does not have. CI builds them.
- `pnpm exec tsc --noEmit -p .` in `server/` (after building its
workspace dependencies) and in `packages/paperclip-runner` pass.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts
src/__tests__/heartbeat-process-recovery.test.ts
src/services/native-runtime/native-plan-wait.test.ts
src/services/native-runtime/paperclip-control-plane-port.test.ts` (in
`server/`): 2,586 passed, 2 failed. `binds Slack tools to the admitted
linked request...` reads a skills checkout that is not next to my
worktree. `materializes direct external-chat finals once...` failed in
the full file run but passes alone, with and without this change.

## Risks

- Progress events now commit up to one transaction later than before.
Their receipts and live UI updates follow their batch.
- A batch that contains progress events still writes rows one by one
inside its transaction, because progress events register chat-progress
wakes. The gain is one transaction and one lock wait per batch, not one
row update per batch.
- If a batch fails, every event in it fails, and the runner resends
them. That is the same as for text deltas today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-16 11:25:59 -08:00
DottaandClaude Opus 5.5 3452639a81 fix(runner): reject only the overflow call when parallel tool calls exceed the result reserve (#21913)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools through the runner's provider
tool bridge
> - The bridge reserves 768 KiB for each pending call against an 8 MiB
budget per turn
> - When an agent sends many tool calls in parallel, the reserve for the
pending calls can go over the budget, although the bytes the turn really
holds are small
> - The bridge then reports the durable receipt limit, and the backend
stops the whole turn. The run ends as cancelled with `needs_review`, and
the task stays blocked with a Retry button
> - This pull request rejects only the overflow call as retryable, and
keeps the turn stop for real held bytes. It also adds a server backstop:
a turn stopped at the receipt limit continues by itself in a fresh
session
> - The benefit is that a parallel burst of tool calls no longer leaves
a task waiting for a person

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave, an opencode agent sent 10 parallel `get_task`
calls. The turn already held about 0.7 MB of results. 10 × 768 KiB plus
0.7 MB is more than 8 MiB, so `begin_call` returned
`active_turn_receipt_limit()` for the 10th call. This branch does not
set `durable_run_receipt_limit_reached`, so it is a temporary reserve
overflow. `handle_tool_call` sends every receipt-limit error to
`stop_turn_at_tool_receipt_limit`, which interrupts the turn. The run
ended as cancelled with `needs_review` and a human review request. The
release then blocked the task
(`native_continuation_requires_reconciliation`) and showed Retry. No
continuation was scheduled. The task moved again only because a person
posted a comment ten minutes later.

**Expected behavior**
The overflow call gets a normal, retryable rejection, and the turn
continues. The agent can send the call again after a pending result
arrives. If a turn does stop at the receipt limit, Paperclip continues
the task in a fresh session, as it does for `agent_tool_call_loop`
(#21734). The task does not show Retry.

**Steps to reproduce**
1. Start a native turn that holds about 600 KiB of settled tool results.
2. Send 10 tool calls in parallel.
3. On master, the 10th call stops the turn, and the task becomes blocked
with Retry.

**Paperclip version or commit**
`master` at c5218f5ec4 (wave), rebased on 991f773d7f.

## What Changed

- `provider_bridge.rs`: the two `begin_call` branches that fail only
because of the per-call result reserve now return a new retryable error
(`tool_calls_in_flight`). The provider sees "Too many tool calls are in
flight. This call was not executed. Wait for the pending tool results,
then retry it." The branches that set
`durable_run_receipt_limit_reached` (real held bytes, exact identity
capacity, saturated replay history) still stop the turn.
- `provider_backend.rs`: the invalid tool call result sets `retryable`
from the error, so this rejection is `retryable: true`. Other rejections
stay `false`.
- `native-tool-receipt-limit.ts` (new): reads the persisted,
source-bound `harness.diagnostic` event with code
`semantic_tool_turn_receipt_limit` for the run, and holds the lane
constants and the hint for the next session.
- `status-arbiter.ts`: a run that did not succeed and whose runner
stopped it at the receipt limit preserves the task status and schedules
a retry with cause `semantic_tool_turn_receipt_limit` (and releases run
resources when it was cancelled). This applies only when the task owner
holds the decision, with no review, governance gate, open blocker, pause
hold or prior `blocked` status, and at most 2 times per chain. Other
cases keep their current path.
- `native-provider-capacity-retry.ts`: the capacity retry is now one
lane of a shared fresh-session retry helper. The new receipt-limit lane
creates a `scheduled_retry` run due in 5 seconds with
`forceFreshSession` and a liveness instruction that tells the agent to
make fewer parallel calls.
- `status-decision-committer.ts` and `native-run-finalizer.ts`: wire the
new lane, and require the task owner under the issue lock, as for the
capacity retry.

- `acpx_provider_session.rs` (Claude, Cursor and other ACPX agents)
handles the same retryable error per call: it drops the pending sidecar
call and resolves only that call with the retryable error. Other
`begin_call` errors still fail closed.
- The external provider path (Dot/Muse) is out of scope and unchanged:
it still fails closed on this overflow, as on master. Making a
same-requestId retry safe there needs retry generations across the
executor, the driver command ids and the broker receipts. That is a
follow-up.
- The receipt-limit backstop reads only the diagnostic of the final turn
in the run's session.
- A person's Stop always wins: the finalizer re-reads the saved Stop on
each arbitration attempt, the retry helper refuses under the run lock,
and the #21734 tool-call loop continuation is not queued, or is
cancelled, for a stopped run.

The runner change is runner-image content. The Daytona runner image must
be rebuilt before sandboxes get the runner fix. The server backstop
works with older images.

## Verification

- Red then green, `provider_bridge.rs` unit test
`parallel_calls_over_the_result_reserve_reject_only_the_overflow_call`:
with about 600 KiB settled, 9 calls are admitted. On master the 10th
returns `is_active_turn_receipt_limit()`. Now it is a retryable
rejection, the durable flag stays false, and the same call is admitted
after one pending result settles.
- Negative, `held_result_bytes_over_the_limit_still_stop_the_turn`: real
held bytes over the limit still return the receipt limit with the
durable flag set.
- `cargo test -p paperclip-runner-core -- --skip task_memory::`: all
pass (640 lib tests plus the integration suites). The `task_memory`
monitor tests are timing-based kernel OOM monitor tests that do not run
correctly on macOS (one hangs); this change does not touch them, and CI
runs them on Linux.
- Red then green, `native-tool-receipt-limit.integration.test.ts` (real
Postgres): a run stopped at the receipt limit gets one `scheduled_retry`
successor with `forceFreshSession` and the hint. The release does not
block the task and creates no recovery action. The task stays
`in_progress`. The presentation shows `retry_scheduled` with no Retry,
then `working` after promotion. With the master arbiter, no successor is
created. A negative case (another diagnostic code) keeps
`cancellation_run_only`.
- Red then green: ACPX
`receiver_rejects_only_the_tool_call_that_overflows_the_result_reserve`
(11-call burst through the fake sidecar).
- Red then green, server: an earlier turn's diagnostic gives no
continuation; a Stop saved before or right after the finalizer's first
read gives no continuation and settles the cancellation with no recovery
action; the retry helper throws `NativeStatusRaceError` for a stopped
run; the loop continuation is not queued, and a queued one is cancelled,
for a stopped run.
- `status-arbiter.test.ts`: the new branch, the 2-continuation bound,
and the `blocked` exclusion.
- Focused server tests pass: the new test,
`native-provider-overload.integration.test.ts`,
`status-arbiter.test.ts`, `native-status-arbiter-corpus.test.ts`,
`heartbeat-run-dispatch-work.test.ts`,
`native-run-finalizer-telemetry.test.ts`,
`native-workspace-finalizer.test.ts` (306 tests). `tsc --noEmit` passes
for `server`.
- `pnpm verify` exits 2 (a runner-core change needs broad coverage), so
required CI supplies it.

## Risks

- Dot/Muse agents still lose the turn on this overflow until the
follow-up lands (no change from master).
- An agent that keeps more than about ten large calls in flight now gets
rejections for the extra calls instead of a stopped turn. It can send
them again after results arrive. Real held bytes still stop the turn.
- The receipt-limit continuation starts a fresh session. The in-turn
conversation is lost; the task, comments, documents and saved workspace
stay. It runs at most 2 times per chain, then the current path applies.
- The retry lane shares the failure-retry count with the capacity lane.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21734

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:53:55 -06:00
github-actions[bot]andlockfile-bot ac22f7f42d chore(lockfile): refresh pnpm lockfiles (#21915)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.
- Resync the SDK public source proof's lock pins from the refreshed
lock.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-16 15:44:28 +00:00
DottaandClaude Opus 5.5 b4fa1c7fb1 perf(recovery): load changed rows, not every task, for company hints, boots and reconnects (#21872)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The task recovery queues (stranded assignments, assigned-work
invariant, resolved dependencies, stale issue locks) check tasks to find
work that has no live run and no queued wake
> - A plain company hint, or an unscoped `"*"` reload, made each queue
load and check every eligible task of the company, at about 22 read-only
statements per task
> - Many writers send that hint (issue creation, documents, budgets,
work intents, agent status changes), every server boot reloads every
task, and the notification connection publishes a `"*"` reload each time
it opens or reconnects
> - On staging this shows as bursts of about 1,400 to 1,550 checks for
one company roughly every 9 minutes, with almost no real work
> - #21861 removed one writer. This pull request fixes the class: a
company hint, a boot and a reconnect now load only changed rows since
the queue last looked, and the hourly full pass is saved so a replaced
server keeps its schedule
> - The benefit is that settled tasks are checked by the hourly full
pass and by real changes, not by every write, deploy or reconnect

## Linked Issues or Issue Description

Refs #21861, #21831, #21867

**What happened?**
A company hint `{ companyId }` on a task queue loaded every eligible
task of the company with no changed-rows filter. The same happened for a
`"*"` reload. Three sources send these all the time:
1. Each server boot starts every task queue with a `"*"` reload.
2. `reconcileDatabaseWorkNotifications` publishes a `"*"` reload to
every queue each time the deployment notification connection opens or
reconnects. The connection opens at boot and again whenever a native
execution starts after an idle period.
3. Many writers send a bare company hint: issue creation, document lock,
unlock, restore and delete, budget pause and resume, work intent
promotion, agent status changes, watchdog writes, and others.

In a staging-shaped fixture each server replacement caused two full
reloads (boot and connection open).

**Expected behavior**
A company hint, a boot or a reconnect re-checks the tasks whose rows
changed. A change that makes tasks eligible without writing any of their
rows (a company or project resume, a tree hold release) still re-checks
every task of the company. An agent status change re-checks that agent's
tasks. The hourly full pass stays the safety net, also across server
replacements.

**Steps to reproduce**
1. Seed one agent with settled in-review tasks and a live native run.
2. Start the recovery workers with
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:1`.
3. Restart the workers, call `reconcileDatabaseWorkNotifications(db)`,
or create an issue.
4. See one `check` event per settled task with `wake: "reload"` or
`wake: "company"`.

**Paperclip version or commit**
`master` at 278660149c.

## What Changed

- `recovery/work.ts`: the full-pass bookkeeping now decides a row filter
per load.
- A plain company hint and the changed-rows backstop load the rows
changed since the queue last looked at the company
(`changedTaskCondition`, minus the existing 5-minute margin). They load
every task only when no full pass is recent.
- A boot and a reconnect use a new reload scope. One statement loads,
per company, every task when that company has no saved full pass within
the hour, and otherwise the rows changed since that saved pass (or since
this server's last reload).
- An agent scope resolves the tasks the agent is assigned, currently
reviews, or holds a run lock on, and checks only those.
- The graph fallback for a graph too large to list now sends the
explicit `everyTask` hint, so it keeps its old meaning.
- When scheduling is suppressed, the queue forgets its passes and
ignores older saved ones, so the first pass after suppression checks
every task (entries were dropped while suppressed).
- New table `issue_recovery_deferrals` (queue, issue, company, due_at),
in the same migration. A row exists while a task's check has deferred
itself (grace period, back-off, live owner lease). A boot or reconnect
reload also loads these tasks and checks them again, which rebuilds each
deadline. Deferrals cannot be rebuilt from changed rows: several defer
with no limit while none of the task's rows change (see the table
below). The store is built so that any race or lost write costs at most
one extra check, never a missed one:
- A deferring check upserts the row with its deadline when the deadline
differs from the one this server last read or wrote (one statement; none
when unchanged). A failed write keeps the full pass unsaved, so a
successor's full pass rebuilds the deadline, and drops the server's
memory of the row.
- A check that no longer defers deletes the row only if `due_at` still
equals the deadline this server read or wrote (one statement, only when
it knew of a row). A row another server rewrote since stays.
- A reload deletes the row of a task it did not load (the task left the
queue, for any reason) when, in the same statement, `due_at` still
equals the value the reload read and either the task is closed or that
deadline passed more than 10 minutes ago. This bounds the table without
listing each queue's eligibility rules.
  - A check that does not defer and knows of no row adds no statement.
- Two overlapping servers both keep the row current; a crash leaves it
in place.
- New table `issue_recovery_full_passes` (company, queue, started_at),
migration `0488`. A full pass writes it only after every task it loaded
has been checked (a check that fails keeps it unsaved), so a server
replaced mid-pass leaves no record and its replacement runs the full
pass again; a server with no memory of its own reads it. This keeps the
hourly full pass on one schedule for replaced and overlapping servers.
Reads and writes are one statement each and only happen when a full pass
is due or done. A failed read means a full pass; a failed write means
one extra full pass later.
- `issue-recovery-notifications.ts`: new scope fields `everyTask`,
`agentId` and `reload`, `notifyAgentRecovery`, `resolveAgentIssues`, and
the reload hint.
- `run-dispatch-notifications.ts`: `RunDispatchScope.everyTask` and
`notifyAgentStatusDispatch`. The agent lifecycle status-change effect
uses it instead of a company hint.
- `work-signals.ts`: `setDatabaseWorkReconnectHint` lets a topic declare
the hint it gets on reconnect instead of an unscoped reload. The task
queues declare the reload hint. Other topics keep the unscoped reload.
- `scoped-deadline-work.ts`: optional `initialHint` for the first pass
after registration. The task queues start with the reload hint.
- Writers that make tasks eligible without changing their rows now send
`everyTask`: company resume (`companies.ts`), project and company budget
resume (`budgets.ts`), tree hold resume and hold creation
(`issue-tree-control.ts`), conversation reset that releases a tree hold
(`agent-conversations.ts`), work intent promotion for `pause_lifted` and
`tree_hold_released`, task reparenting and task deletion (`issues.ts`),
company import relations (`company-portability.ts`), and company-wide
chat milestones.
- Writers with a known task now send a task or graph hint instead of a
company hint: document lock, unlock, restore and delete
(`documents.ts`), blocker edits and attachment delete (`issues.ts`),
watchdog writes (`task-watchdogs.ts`).
- Native run writers that sent a company hint now send a graph hint for
the run and its task: Runner file handoff
(`native-runner-file-handoff.ts`) and chat attachment reuse
(`chat-attachment-reuse.ts`).
- Agent-scoped hint: only the agent lifecycle status-change effect sends
it, for transitions that go through the lifecycle (pause, resume, hold
and release, terminate, error recovery, environment pause, and budget
holds through `applyAgentBudgetHoldInTransaction`). A run starting or
finishing writes the agent row directly (execution start,
`finalizeAgentStatus`) and sends only run-scoped hints, which re-check
the run's task and its graph. Queued-run dispatch after a run finishes
is unchanged (its own dispatch queues).
- `assigned-work-sweep.ts`: `listSuspects` takes the row filter as SQL.
- Tests: new census `issue-recovery-company-reload-census.test.ts`, new
`issue-recovery-saved-deferrals.test.ts` (real test database, recorded
checks), a `restart` helper in the recovery fixture, and a work-signals
test for the reconnect hint.

## Verification

Census (`issue-recovery-company-reload-census.test.ts`, real workers,
fake clock, 28 simulated minutes, 140 + 20 settled reviews, an agent
with a live native run, no fixture agent can start a run). Steps: server
replacement plus connection open, issue creation, document lock, project
budget hard stop, agent pause, agent resume, second server replacement
plus connection open.

| | Settled-task checks | Stranded checks by wake | Statements |
| --- | --- | --- | --- |
| Before (master code) | 1,440 (320 per replacement, 160 per other step)
| company 830, reload 662, hint 163 | 33,485 |
| After | 0 | hint 168, reconnect 12, backstop 7, agent 5, company 5 |
4,293 |

An earlier run of the same census at staging size (700 + 80 settled
reviews, 35 minutes) gave 8,028 settled checks and 165,882 statements
before, and 240 settled checks (only the paused agent's own tasks) and
9,714 statements after.

Guard tests in the same file:
- A task stranded by a write that sends no hint, right before each step,
is checked within 1 minute (replacements, issue creation, budget stop,
pause, resume) or within the 5-minute backstop plus 2 minutes (document
lock, which now re-checks only its own task). Passes before and after.
- A server replaced after its first full pass loaded every task but
before it checked them all: the replacement checks every unchecked task
within 2 minutes. Fails before the deferred save (112 tasks never
checked), passes after.
- Run start and finish (same census file): an agent with a settled
backlog runs start, finish, start, finish through the run row, the
execution-start agent write, the execution-run hints and
`finalizeAgentStatus` (no process runs). 0 settled-task checks on this
branch and on master; only the run's task is checked. A run that fails
has its task checked within 1 minute of the run ending.
- `issue-recovery-saved-deferrals.test.ts`: a task's check defers 20
minutes, the full pass is saved, the server is replaced 5 minutes later
without shutdown work. The replacement checks the task within 1 minute
of the original deadline. Fails at the previous head (never checked
before the next full pass), passes after.
- Same file: server B saves a deadline; server A boots and reads it; B
clears it and defers again to a new deadline; A's check stops deferring
and clears the row it read; B is replaced. The successor checks at B's
deadline. Fails at the previous head (A deleted B's row by id), passes
after.
- Same file: a saved deferral is removed at the next reload once its
task closed.
- Same file: an open task that leaves the queue (taken over by a user)
while deferred has its row removed by the first reload after its
deadline is 10 minutes past. Fails at the previous head (row kept
forever), passes after.
- Same file: after scheduling suppression, a plain company hint checks
every task again even though a recent full pass is saved.
- A change that no changed-rows clause sees is not checked by either
server replacement, and is checked by the next full pass on the saved
schedule. Fails before (each replacement did a full pass), passes after.

The file takes about 30 s locally. Every wait has a 20 s real-time limit
and each test a 2-minute limit, as in #21867.

Focused suites (all `issue-recovery-*` suites and censuses,
assigned-work, scoped deadline work, deployment notifications, budgets,
companies, run dispatch notifications, and related idle tests) on the
final head (34 files with the new deferral test): 288 passed, 1 skipped.
The #21867 agent-row census still reports 0 settled checks.
`packages/db` `work-signals.test.ts`: 34 passed. Server `tsc --noEmit`:
passed. `pnpm --filter @paperclipai/db check:migrations`: passed. `pnpm
verify` returned exit 2 (broad input: the new migration), so required CI
supplies broad coverage.

## Risks

Time bound that changes:
- A task stranded by a change that no changed-rows rule sees was often
found sooner by an unrelated company hint, boot or reconnect. It is now
found by the next full pass: at the next backstop (needs activity in the
company), boot or reconnect after one hour from the last saved full
pass. This was already the documented worst case.

Row changes the changed-rows rules see (`recovery/changed-tasks.ts`),
for auditing completeness. A task is loaded when, since the window
start:
- its own `issues.updated_at` moved;
- a `heartbeat_runs` row whose `context_snapshot.issueId` is the task
has `updated_at` in the window;
- an `issue_recovery_actions` row with the task as `source_issue_id` has
`updated_at` in the window;
- an `agent_wakeup_requests` row whose `payload.issueId` is the task has
`updated_at` in the window;
- an `issue_thread_interactions` row of the task has `updated_at` in the
window;
- an `issue_comments` row of the task was created in the window;
- its assignee agent row has `updated_at` in the window, is not
`running`, and has no heartbeat since that update (pause, resume, hold,
termination, configuration).
Not seen: company, project, tree-hold and budget rows (their resumes
send `everyTask`), the activity log, other agents' rows, lease renewals
that do not move `updated_at`, and time passing. Deferrals are covered
separately by the saved deferral rows.

Rebuilt across a replacement (no change): every pending deferral, by the
saved deferral rows. Every way the four checks defer, from tracing each
`onDeferredDeadline` call:

| Queue | Deferral | Deadline | Bounded after a seen row change? |
| --- | --- | --- | --- |
| stranded | terminal run still settling | run lease, or now + 30 s
while the run executes in memory | no (repeats while it executes) |
| stranded | chat agent daily limit | next 00:00 UTC | < 24 h |
| stranded | accepted interaction, no run since | interaction resolved +
2 min | yes |
| stranded | transient continuation back-off | run finished + up to 16
min | yes |
| dependencies | refused or failed stall wake | last attempt + up to 20
min | yes |
| dependencies | waiter's workspace hold | now + 1 h | no (repeats while
held) |
| dependencies | waiter's agent takes no wakes | now + 10 min | no
(another agent's row) |
| locks | live legacy controller lease | lease expiry | no (renewal does
not move `updated_at`) |
| locks | lapsed lease, owner proven alive | lease + up to 5 min | yes |
| locks | run updated less than 5 min ago | run updated + 5 min | yes |
| locks | adopted or lost child process still alive | now + 30 s | no
(repeats while it lives) |
| locks | prior-boot setup | about run updated + 5 min | yes |
| assigned work | settling, saved-work due, reattach | row time + up to
10 min, or now + 2 s during reattach | yes |
| assigned work | repair budget, company run brake, stalled queued run,
saved work busy | now + 60 s to 1 h | no (repeats while the condition
lasts) |
| assigned work | `waiting_unchanged` (same refusal state) | last
refusal + up to 6 h (activity log) | no |
| assigned work | continuation cap, spacing, fallback back-off | first
attempt + up to 1 h | no (activity log) |

`unchanged` (assigned work only) changes queue priority, not whether a
deferral is pending.
- Unchanged: the 5-minute changed-rows backstop, the 1-minute bound for
paused, resumed, held and terminated agents (covered by the
#21861/#21867 guards and the new agent scope), and whole-company
re-checks for company and project resumes and tree hold releases.

Other risks:
- During the deploy that ships this change, the old build still sends a
bare company hint for a company or budget resume. The new build reads
that as changed rows only. The old build re-checks every task for its
own hint while its workers run; otherwise the next full pass does. After
this deploy both builds send `everyTask`.
- If another server's `everyTask` hint is lost while this server's
notification connection is closed, the reconnect no longer re-checks
every task. The writing server checks its own hint; otherwise the next
full pass does.
- The first boot after this deploy has no saved full passes, so it runs
one full pass per company (as every boot did before).
- Nothing is added before a run's first turn or on the run-start path.
- Schema change: two new company-scoped tables with cascade foreign
keys, no change to existing tables. The old build ignores them. Neither
is a timer: no deadline is stored or scheduled from them, they only tell
a boot which tasks to check again.
- Extra statements: per deferring check, one upsert when its deadline
changed (a deferral to `now + 30 s` changes each time, so about two
writes a minute per such task) and none otherwise; one delete when a
check stops deferring; one read of the queue's deferral rows and at most
one prune per reload; one write per completed full pass.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 05:52:57 -06:00
8299f7c389 perf(runner): check a listening Runner's backlog every 10 ms, not every reconnect delay (#21821)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote `paperclip_runner` agent runs `paperclip-runnerd` in a
sandbox or VM. A provider ingress proxies Core's WebSocket to a Runner
that listens (`--listen-address`).
> - In listen mode, the Runner checks for an incoming connection only
once per `--reconnect-delay-ms` (default 250 ms). It waits that delay
again after each disconnect, and it backs off after a failed handshake.
> - That delay exists to pace redials in dial mode. A listener never
dials. Core's connection waits in the kernel backlog until the next
`accept`.
> - Every listen-mode connect pays part of that wait. A warm run
attachment pays it in full: the Runner disconnects at the authority
rotation, and Core's new-authority connect then waits up to 250 ms or
more.
> - This pull request makes a listening Runner check for a connection
every 10 ms. Dial mode keeps the configured reconnect delay.
> - The benefit is about 0.4 s less on each warm run attachment and
about 0.1 s less on each cold start, for every listen-mode Runner.

## Linked Issues or Issue Description

**What happened?**
A warm follow-up turn on a listen-mode remote Runner reached its first
turn later than a cold turn on the same agent. In traces, a cold start
makes one Core→Runner connect of 134–178 ms. Each warm attachment makes
an old-authority reconnect (72–233 ms) and then a new-authority connect
of 411–504 ms. This was 6 of 6 warm runs.

**Expected behavior**
A connect to a listening Runner completes in about one network round
trip plus the WebSocket handshake. It does not wait for a reconnect
timer.

**Steps to reproduce**
1. Start `paperclip-runnerd` in listen mode with the default
`--reconnect-delay-ms`.
2. Let it reach the disconnected wait (startup, a park claim, or a
`run.attach` authority rotation).
3. Connect to the listener. The upgrade is not answered until the next
250 ms check.

**Root cause**
`durable/runner.rs` schedules the next `accept` at `now +
config.reconnect_delay` when no connection is pending, after a
disconnect, and as the base of the failure backoff. The Launcher-style
hosts do not pass `--reconnect-delay-ms`, so the 250 ms default applies.
The tests use a 1 ms delay, so they did not show the wait.

## What Changed

- `durable/transport.rs`: add `RunnerTransportEndpoint::retry_delay()`.
A listener returns `min(configured, 10 ms)`. A dial endpoint returns the
configured delay without change.
- `durable/runner.rs`: the four places that waited
`config.reconnect_delay` (no pending connection, failure backoff base,
two disconnect paths) now use
`endpoint.retry_delay(config.reconnect_delay)`.
- `DURABLE_TRANSPORT.md`: document the listen-mode connection check
interval.
- Add a unit test for the listener and dial delays.
- Add `tests/listener_accept.rs`. It starts `paperclip-runnerd` in
listen mode with the default reconnect delay. One connection drops
before authentication, as a superseded route's redial does. The test
then checks that the next upgrade is answered in under 120 ms. Before
this change, that upgrade waited 188–313 ms.

## Verification

- `cargo test -p paperclip-runner-core --lib` passes locally.
- `cargo clippy -p paperclip-runner-core --all-targets` reports no new
findings in the changed files.
- `cargo fmt --all -- --check` passes.
- New tests:
`durable::transport::tests::listener_checks_its_backlog_sooner_than_a_dial_reconnects`
and `tests/listener_accept.rs`.
- `tests/listener_accept.rs` fails on the previous `runner.rs` and
passes with this change (3 runs each).
- Existing listener and warm-attachment tests pass (`listener_*`,
`warm_run_attachment_*`, `warm_external_attachment_*`).
- I did not measure the change on a live host. The effect is visible in
the next deployment as a shorter `runner.transport.registration` on warm
attachments.

## Risks

- Low risk. Dial mode does not change.
- A disconnected listener now wakes every 10 ms instead of every 250 ms.
Each wake is one non-blocking `accept` and a non-blocking host-control
read. The loop already slept in 25 ms steps.
- Failed handshakes on a listener now back off from 10 ms (10, 20, 40 …
320 ms) instead of from 250 ms. Core's own reconnect loop still paces
its redials, and the ingress requires credentials, so this does not
create a hot loop.
- The change needs a Runner release before hosts pick it up.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, trace queries).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 22:17:52 +00:00
DottaandClaude Opus 5.5 46e5e4450f fix(runner): leave an undispatched tool call to the next controller on restart detach (#21786)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner keeps an agent session alive while its
controller restarts, for example during a deploy, and the next
controller adopts it
> - A Paperclip tool call can be between its committed input and its
handler when the controller detaches
> - In that window the old controller journaled "The turn stopped before
this operation was dispatched" as the final, non-retryable result, but
the turn did not stop
> - The agent then got a false failure for a call that never ran, and
the next controller could not recover the call. A restart must never
fail work
> - This pull request leaves that call unsettled for the next
controller, the same as a call that was already in its handler
> - The benefit is that a controller restart no longer turns an agent's
pending tool call into a permanent error, and a CI test that failed on
four pull requests now passes deterministically

## Linked Issues or Issue Description

**What happened?**
`src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts` ›
`settles an earlier generation's in-flight tool call at attach even when
that attach never finishes startup` failed in "ci / Verify Paperclip
Runner (vitest package (3/3))" on 2026-11-15. In each job the first try
and the CI retry both failed:
- PR #21757, head `d04825d812`, job 132824209377
- PR #21713, head `7758d56719`, job 132681892394
- PR #21729, head `c63a05a013`, job 132732839971

Every failure is the same, at line 74, right after
`detachControllerForRestart()`:

```
AssertionError: expected [ { …(8) } ] to deeply equal []
+   "type": "semantic_tool.result", "status": "pending",
+   "result": { "error": { "code": "semantic_tool_not_dispatched",
+     "message": "The turn stopped before this operation was dispatched.", "retryable": false } }
```

**Expected behavior**
When a controller detaches for restart, a tool call that it has not
dispatched stays unsettled. The next controller recovers it at attach,
the same as a call that is in its handler.

**Steps to reproduce**
1. Hold the semantic tool dispatch at a gate after the Runner commits
the tool input.
2. Call `detachControllerForRestart()`, then release the gate.
3. On `master`, the control-plane state has the
`semantic_tool_not_dispatched` result above. It is byte-identical to CI
(same command id and payload). The new test case `(still before dispatch
at detach)` does this.

**Root cause**
This is a Runner defect, not a test timing problem.
`#detachControllerOnce` sets `#closed`, resolves the closed signal and
resolves the turn-start admission with `false`. A call in
`#handleSemanticToolInput` that has not reached the handler then throws
`SemanticToolNotDispatchedError`. The durable control plane turns that
error into a journaled `semantic_tool_not_dispatched` result. That
result is correct when the turn stops. It is false when the controller
only detaches for restart, because the Runner and the turn continue. The
old test waited only for the committed input, so on a loaded CI runner
the detach sometimes came before the handler call.

**Production impact**
Yes. During a controller restart or deploy, an agent tool call in this
window got a non-retryable "turn stopped" error after the Runner was
adopted, and the next controller did not recover the call.

## What Changed

- `durable-prp-control-plane.ts`: new `SemanticToolHandedOffError`. When
the dispatcher rejects with it, the control plane journals no result.
The input stays unsettled for the next controller.
- `runnerd-codex-transport.ts`: the semantic tool callback goes through
`#dispatchSemanticToolInput`. If the call ends as not dispatched after
the transport detached for restart, it throws
`SemanticToolHandedOffError`. A normal close still journals
`semantic_tool_not_dispatched`.
- `runnerd-codex-transport-orphaned-tool-at-attach.test.ts`: the test
waits for the exact stage instead of the committed input. It runs two
cases: the call is in its handler at detach, and the call is held before
dispatch across the detach. Both then check the existing attach recovery
and the final `completed` result.

## Verification

- Red on `master`: the `(still before dispatch at detach)` case fails
every run with the CI assertion and payload.
- Green on this branch: `npx vitest run
src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts` passes
2/2.
- Loop on macOS with 4 extra busy-loop processes (load average about 13
before them), 50 runs each: this branch 50/50 (100 test cases). The old
`master` test was 50/50 locally too, so the local loop does not show the
CI interleaving. The gate does.
- All 36 runner test files that use `detachControllerForRestart` or
`SemanticToolNotDispatchedError`: 1180/1182 pass with 3 workers under
load. The 2 failures (`runnerd-codex-transport-settlement.test.ts` and
`runnerd-acpx-source-reconciliation.test.ts`, `kill EPERM`) pass 36/36
when run alone.
- `npx tsc -p tsconfig.json --noEmit` and `npx tsc -p tsconfig.test.json
--noEmit` pass in `packages/paperclip-runner`.
- The other two static-check flakes from the same day are test-only and
are fixed in a separate pull request.

## Risks

- Medium-low risk. This changes Runner restart behavior for one window.
A call that was not dispatched now goes to the next controller's
recovery (`recoverSemanticToolInput`) instead of getting a final error.
That is the same path that already handles a call in its handler, so no
new recovery code is involved.
- If no successor ever attaches, the input stays unsettled. That is the
same state as for a call that was in its handler at detach.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:27:04 -06:00
DottaandClaude Opus 5.5 7a9177af0f test(runner): remove coreutils and pid-file timing assumptions from static checks (#21787)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner package has a "static checks" CI job that runs
`node --test` script tests
> - Two of these tests failed on unrelated pull requests on 2026-11-15
and passed on re-run
> - Each failure costs a re-run of the failed jobs and delays merges,
and it teaches people to re-run instead of read
> - One test assumed that a copied `/bin/sleep` acts as `sleep` under
any name. The other read a pid file before the child finished writing it
> - This pull request removes both assumptions, and one more instance of
the second pattern in the same test family
> - The benefit is that these tests no longer depend on the runner image
or on process scheduling

## Linked Issues or Issue Description

**What happened?**

1. `scripts/stage-runner-binary.test.mjs` failed in "ci / Verify
Paperclip Runner (static checks)" on PR #21765, head `98197c87e9`, job
132850068174:
- `atomically publishes while the previous executable is running`:
`AssertionError: 1 !== null` at line 124 (`old.child.exitCode`), after
77 ms.
- `a source rebuild selects fresh dev bytes ...`: `Expected "actual" not
to be strictly deep-equal` (line 192).
- That job ran on a GitHub-hosted `ubuntu-26.04` image (20261111.82.6).
The same tests pass on the RunsOn `ubuntu24` image, for example job
132822472729.
2. `scripts/materialize-kimi-distribution.test.mjs` › `setup
cancellation physically stops an in-flight owned child` failed in the
same job on PR #21758, head `40e38fc28a`, job 132822472729:
`AssertionError: The real child did not start` after 149 ms.

**Expected behavior**
The tests pass on every supported runner image and on a loaded runner.

**Steps to reproduce**
1. stage-runner-binary: in a Linux container, replace `/bin/sleep` and
`/bin/echo` with one multicall binary that picks the utility from
`argv[0]` and exits 1 for an unknown name (this is how uutils coreutils
works on Ubuntu 26.04). Run `node --test
scripts/stage-runner-binary.test.mjs`. On `master` both tests fail with
the CI messages, 20 of 20 runs.
2. kimi: make the child create the pid file, wait 300 ms, then write the
pid. On `master` the test fails with `The real child did not start` in
about 125 ms, the same as CI.

**Root cause**
1. On Ubuntu 26.04, `/bin/sleep` and `/bin/echo` are the same uutils
multicall binary. The test copies it to `paperclip-runnerd` and runs it.
The binary sees `argv[0] = paperclip-runnerd`, finds no utility of that
name, and exits 1. The fresh-build fixture (a copy of `echo`) also has
the same bytes as the old artifact (a copy of `sleep`), so the "fresh
bytes" assertion fails too.
2. The child wrote the pid with `writeFileSync`, which creates the file
and then writes it. The test loop broke on the first successful read. An
empty read gave `Number("") === 0`, so the loop stopped and the
assertion failed. The loop also had a fixed budget of 100 polls of 10 ms
for a Node child to start.

**Production impact**
None. Only tests change. The staging code works with any executable. The
Kimi setup cancellation code is correct: the deterministic repro shows
that the child is stopped once the test reads a real pid.

## What Changed

- `stage-runner-binary.test.mjs`: the test helper passes `argv0` to
`spawn`. The copied `sleep` runs with `argv0: "sleep"`, so a multicall
binary still acts as `sleep`. The fresh-build fixture is now a copy of
`/bin/sh` and runs `-c "exit 0"`, so its bytes always differ from the
old `sleep` artifact.
- `materialize-kimi-distribution.test.mjs`: the child writes `pid.tmp`
and renames it to `pid`, so a read never sees a partial file. The test
waits until the pid is valid or the command settles. The 3 s command
timeout and 5 s test timeout are unchanged.
- `provision-kimi-package.test.mjs`: the wait for the `download-ready`
marker continues until the child exits, instead of a fixed 1 s budget.
The existing 5 s kill timer still bounds it.
- No retries, skips or timeout changes.

## Verification

- stage-runner-binary, Linux container (`node:24-bookworm-slim`) with a
multicall `sleep`/`echo` built from a 15-line Go program that dispatches
on `argv[0]`:
  - `master`: 0/20 pass, same two assertions as CI.
  - this branch: 50/50 pass.
  - this branch with stock coreutils, and on macOS: all 9 tests pass.
- kimi setup cancellation:
  - delayed-write repro: `master` fails, this branch passes.
- loop on macOS, `master` test without extra load: 59/60 (one natural
failure). This branch: 60/60.
- loop with 4 extra busy-loop processes (load average about 13 before
them): `master` 100/100, this branch 100/100.
- `node --test scripts/stage-runner-binary.test.mjs
scripts/materialize-kimi-distribution.test.mjs
scripts/provision-kimi-package.test.mjs`: 19 pass, 1 skipped (public
download test, opt-in).
- The other CI flake from the same day,
`runnerd-codex-transport-orphaned-tool-at-attach.test.ts`, is a Runner
defect. It is fixed in a separate pull request.

## Risks

- Low risk. Only tests change.
- The rebuild test now uses `/bin/sh` as the fixture for the fresh
build. This file exists on every supported Linux and macOS runner.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, GitHub CLI, Docker).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:13:19 -06:00
DottaandClaude Opus 5.5 171e70c11d refactor: drop span and run-log event catalogs from the observability docs (#21779)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `doc/observability.md` and `doc/run-log-events.md` described every
OpenTelemetry span and every run-log event
> - `rules/data-paths.mdc` told authors to update
`doc/run-log-events.md` for each event, so both files only grew: 2,313
and 1,263 lines
> - Each span and event is already defined next to the code that writes
it, and its name is a literal string that a search finds in one step
> - This pull request removes the per-span and per-event catalogs and
keeps only what code does not say
> - The benefit is two short docs that do not drift from the code, and
no rule that makes them grow again

## Linked Issues or Issue Description

Refs #21670, #21700, and #21743 (the same cleanup for other docs).

**Issue type**

Outdated or confusing documentation.

**Where is the issue?**

`doc/observability.md`, `doc/run-log-events.md`, and
`rules/data-paths.mdc`

**What's wrong?**

The two docs repeated span names, span attributes, and event payloads
that the code defines. Operator setup started at line 138 of
`doc/observability.md`. The rule required a doc edit for every new
run-log event.

**Suggested fix**

Keep operator setup and the Sentry privacy contract. Point to the code
for span and event definitions. Remove the rule line that requires a doc
entry per event.

## What Changed

- `doc/observability.md` is now 619 lines. It keeps: how to enable
tracing, `service.version` resolution, scope, how to enable Sentry, DSN
delivery, environment attribution, privacy settings, server request
data, the core of "Run failure context", browser data, fail-open
behavior, the default capture set, and operator responsibilities. Kept
text is unchanged except the intro and two sentences noted below.
- Deleted from `doc/observability.md`: budget, Slack receipt, task
detail, and browser tracing notes; controller preparation phases; native
Runner spans; sandbox startup spans; sandbox duplex transport; shared
skill preparation; invocation attribution; and about 500 lines of
per-failure detail in "Run failure context".
- Replaced that per-failure detail with one paragraph. It names the
failures that `unexpected` mode keeps out of Sentry, and says the
remaining diagnostic contexts are defined in `server/src/sentry.ts`.
"Default capture set" now says the same.
- `doc/run-log-events.md` is now 23 lines. It says what the run log is,
where events are defined and appended, and keeps the payload storage
note.
- `rules/data-paths.mdc`: the run-log line now says to define an event
where it is written, not in a doc. The span-allowlist line now says the
allowlist is in the code that opens the span. The review levels and
globs do not change.
- Repointed links in two feature maps, two agent-access map files,
`tests/runner-e2e/SUITES.md`, `doc/runner-api-tools.md`, and the
Telemetry README. Updated the `honeycomb-tracing` skill and one code
comment in `packages/adapter-utils/src/http2-bridge-server.ts`.

## Verification

- A script built the new `doc/observability.md` from line ranges of the
old file and asserts the first line of each range.
- A link check on all changed Markdown files found 0 missing files and 0
missing anchors. The remaining inbound anchors (`#enabling-tracing`,
`#sentry-error-monitoring`) still exist.
- No test reads either doc.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail.
- `pnpm verify` was not run. This worktree has no installed
dependencies. The only code change is a comment.

## Risks

- The privacy contract no longer lists each run-failure diagnostic
context field. `server/src/sentry.ts` is now the list. A reviewer of a
Sentry change must read that file.
- No browsable list of span names or run-log event names remains. A
search for the quoted name finds the producer.
- `.greptile/files.json` still scopes `rules/data-paths.mdc` to both
docs. Both files still exist, so it does not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code desktop, with tool
use. Earlier analysis in this session used Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:06:05 -06:00
DottaandClaude Opus 5.5 531db3c10b perf(db): keep dedicated connections for the next burst (#21769)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process runs every agent on one Node main thread, and
each new database connection costs a TLS handshake, about 0.36 ms of
SCRAM key derivation on that thread, and a type-fetch query
> - `withDedicatedDbConnection` gives lock holders a private session;
since #21693 it keeps at most 4 released sessions and closes the rest
> - When the same burst of more than 4 dedicated holders comes back,
every holder above 4 opens a new connection again
> - A harness on embedded PostgreSQL measured 504 new connections per
1,000 uses for bursts of 8 and 760 for bursts of 16
> - This pull request keeps released dedicated clients for reuse while
idle plus in-use clients stay within
`DATABASE_DEDICATED_CONNECTION_MAX`; the driver idle timeout still
closes their sessions when they are not used
> - The benefit is 8 and 16 new connections per 1,000 uses for the same
bursts, with no change to any pool maximum or to
`databaseConnectionBudget()`

## Linked Issues or Issue Description

**What is the problem?**
A staging measurement showed `SET
application_name='paperclip-runner-checkpoint'` at 3.17/s and `SET
application_name='paperclip'` at 3.86/s with 7 live runs. This looked
like connection churn. The investigation found two separate facts:

1. The `SET` statements are not new client connections. postgres.js
sends `application_name` in the startup packet, never as a `SET`
statement (`StartupMessage` in postgres.js `src/connection.js`).
PgBouncer in transaction mode sends `SET application_name` itself when
it gives a client transaction a server connection that a client with
another `application_name` used last. With one shared server pool, every
checkpoint transaction that lands after a main-pool transaction costs
one `SET`, and the reverse. The rate follows checkpoint transactions,
not connects. The checkpoint pool opens at most 3 connections per 1,000
saves (measured below).
2. Dedicated connections did churn by construction for recurring bursts
of more than 4 holders (measured below). This pull request fixes that.

**Steps to reproduce**
Run `packages/db/src/connection-reuse-census.test.ts`. It counts
`pg_stat_database.sessions` around 1,000 operations.

**Expected behavior**
A recurring burst reuses the sessions that the previous burst opened.

## What Changed

- `withDedicatedDbConnection` keeps a released client idle when fewer
than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle, as before, or when
idle plus in-use clients stay within the dedicated limit
(`DATABASE_DEDICATED_CONNECTION_MAX`, default 24). A client whose action
failed, an ended client and clients after shutdown are still closed.
- Each retained client keeps the main pool's `idle_timeout` (default 60
s), so its session closes after that idle time, as the main pool's do.
The next use reconnects.
- The worst case stays at `2 x DATABASE_DEDICATED_CONNECTION_MAX + 4`: a
new client opens only when no client is idle, and a client is kept above
4 only while idle plus in-use is within the limit.
`databaseConnectionBudget()` and every pool maximum are unchanged.
- New `connection-reuse-census.test.ts` measures connections opened per
1,000 checkpoint saves and per 1,000 dedicated uses.
- `dedicated-connection-pool.test.ts`: the burst test now checks that
the next burst reuses all sessions and that they close after the idle
timeout (1 s in the test). Before, it checked that at most 4 stayed
open.
- `doc/DATABASE.md` describes the new retention rule.

## Verification

Connections opened per 1,000 operations, embedded PostgreSQL,
`pg_stat_database.sessions` (load average about 10.5 on a shared
machine):

| Operation | Before | After |
| --- | --- | --- |
| Checkpoint saves, concurrency 1 / 7 / 24 | 1 / 3 / 3 | 1 / 3 / 3 |
| Dedicated uses, steady concurrency 1 / 4 / 8 / 16 | 1 / 4 / 8 / 16 | 1
/ 4 / 8 / 16 |
| Dedicated uses, recurring bursts of 8 | 504 | 8 |
| Dedicated uses, recurring bursts of 16 | 760 | 16 |

- `pnpm --filter @paperclipai/db exec vitest run
src/connection-reuse-census.test.ts
src/dedicated-connection-pool.test.ts
src/dedicated-connection-admission.test.ts src/client-options.test.ts`:
38 passed. The burst cases fail on `master` (504 and 760 against a bound
of 8 and 16).

## Risks

- After a burst, up to `DATABASE_DEDICATED_CONNECTION_MAX` dedicated
sessions stay open for up to `DATABASE_IDLE_TIMEOUT_SECONDS` (60 s)
instead of closing at once above 4. This stays inside the connection
budget. With `DATABASE_IDLE_TIMEOUT_SECONDS=0` they stay open until
shutdown, like the main pool.
- Retained sessions keep the existing rule: actions keep session state
transaction-scoped.
- The PgBouncer `SET application_name` statements stay. Removing them
needs one shared `application_name` for all pools, which removes the
per-pool names that operators and tests use. That is a separate
decision.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:51:55 -06:00
DottaandPaperclip 5b750fdf25 feat(ui): add foldable sidebar subtasks with bounded refreshes (#21649)
## Thinking Path

> - Paperclip helps people manage AI agents for work.
> - The sidebar helps people return to recent tasks.
> - Recent parents need foldable child tasks.
> - Busy companies must not refresh every child group for unrelated
work.
> - Exact unread counts must preserve personal reads and task access.
> - This pull request adds child groups and bounds their refresh work.
> - People can scan subtasks without repeated background requests.

## Linked Issues or Issue Description

**Subsystem affected**

Task navigation in the standard and Classic sidebars.

**Problem or motivation**

Recent parent tasks did not show their subtasks. The first
implementation refreshed every previously viewed group. One unrelated
update caused 52 child-page requests and one summary request after
viewing 50 groups. Exact child counts also repeated privacy and unread
checks per child.

**Proposed solution**

Show direct subtasks with independent folding and ten-row pages. Refresh
affected groups and summary batches. Stop offscreen reads. Bound
child-request concurrency. Aggregate viewer facts once and preserve the
existing inbox unread rule.

**Roadmap alignment**

This extends existing task navigation. The existing feature PR is reused
for its performance repair. No new PR duplicates it.

## What Changed

- Add shared parent groups to both sidebar layouts. Preserve Classic row
actions and active-child visibility.
- Keep unread state personal to each task. Folding and paging do not
mark tasks read. Counts include readable children beyond ten.
- Target live refreshes with authorized current and previous parent IDs.
Reuse parent metadata from the narrow run-binding read. Resolve fallback
lookups once per delivery batch. Keep the previous summary fallback for
older servers.
- Limit child requests to three at a time. Cancel active and queued
reads when groups leave the viewport.
- Discard extra cached pages on Show less and collapse.
- Compute exact navigation counts in one SQL statement. Materialize
readable candidate IDs once. Use indexed participation sources and
conservative unread candidates before the shared inbox rule.
- Skip recursive privacy walks for standalone open tasks without a
project. Keep the recursive rule for all restricted or inherited shapes
and all delegated principals.
- Add request-count, cancellation, reparenting, read-receipt,
inbox-parity and privacy regressions. Include a disposable PostgreSQL
benchmark.
- Update product and API documentation. Retain the production Storybook
states.

## Verification

- Published head: `4148861437c1b6b635479ad92509dbc71155a232`. Its tree
matches local `04be9d66d481d9bcd1c4d6158acf063a1e65ac7e`:
`a23533e93d3d155918f0254d296a7fd2f0c7c12f`.
- Rebased on master `ca23330c69b1f2b25bdadfcbda68317db97deaef`. Remote
publication preserves prior feature commits.
- The request regression failed before the fix and passes after it. An
unrelated event now sends zero child requests. Current servers also
avoid the unrelated summary request. Old-server events keep one summary
fallback.
- A relevant update after Show less reads one page. Reparenting
refreshes two affected groups. Offscreen reads stay at zero and resume
once on return. Active and queued cancellation tests pass.
- The same synthetic authenticated HTTP workload with 50,000 read
children improved from 767 ms to 92.84 ms median. Five warm samples per
case. The all-unread case improved from 463.85 to 351.65 ms. Untouched
children improved from 208.01 to 131.09 ms. The old-receipt case
improved from 799.70 to 245.80 ms. These local measurements are not a
production capacity guarantee.
- Focused PostgreSQL routes, inbox-parity, privacy and WebSocket tests
passed: 38 tests.
- All 325 affected UI tests passed. All 38 focused PostgreSQL, privacy
and WebSocket tests passed. UI typecheck and low-memory server
declaration checks passed.
- All 41 Chromium Storybook checks passed. They cover 20 states in both
themes and keyboard, folding, paging and unread behavior. They use
production components with HTTP fixtures.
- Both new review findings failed as regressions before their fixes and
pass afterward. Explicit child reads include waiting external
conversations. Deleting a child publishes a separate parent-authorized
update without deleted-child content. Both review threads are resolved.
- `pnpm verify --explain` returned exit 2 because product documentation
requires broader CI. [The prior CI
run](https://github.com/paperclipai/paperclip/actions/runs/43711489306)
found type errors in the new test fixtures. The same errors failed
locally. The fixture fix passes `node
server/scripts/typecheck-layers.mjs --layer test-tree-0` and all 21
issue summary route tests. Production code is unchanged. That CI run
then found a live-delivery read regression. The existing test failed
locally with eight statements against its six-statement budget. A new
PostgreSQL regression also found three parent lookups for two same-batch
calls. Run delivery now reuses parent metadata from its narrow
run-binding query. Fallback lookups resolve to one promise. Both
regressions pass without increasing the budgets. All 74 targeted route,
WebSocket and privacy tests pass. The affected below, above and
test-tree-0 typecheck layers pass. [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/43715541480/attempts/2)
passes on the published head. All 62 check runs are terminal and clean:
57 succeeded, two were neutral and three were skipped. Typecheck, build,
policy, all general and serialized server suites, E2E and the [final
verify
gate](https://github.com/paperclipai/paperclip/actions/runs/43715541480/job/132829783231)
pass. The separate security status passes. Attempt 1's server shard 7/12
reached its 20-minute job limit without a test verdict. One retry of
that job passed without a source change. [Greptile completed
successfully on this
head](https://github.com/paperclipai/paperclip/runs/132816163906) with a
[5/5
score](https://github.com/paperclipai/paperclip/pull/21649#issuecomment-6883916606).
All five review threads are resolved. The final CI gate is complete.
- A full local Storybook build exceeded available memory while server
typechecking ran. Browser verification uses the current-source
development server. Rebuilding stale Runner and SDK declarations
resolved the initial server check errors. The first browser attempt
lacked Chromium; all checks passed after installation.
- Reproduce the database measurements with `pnpm --filter
@paperclipai/server exec tsx scripts/benchmark-sidebar-subtasks.ts`.
- Extra local shard diagnosis ran 164 suites. Under this shell's `0077`
creation mask, 157 suites passed and three had fixture permission or
timing failures. Rerunning those three with `0022` resolved the
permission failures. The remaining ten cases passed with one worker: two
workspace cases and eight archive cases. The eight archive cases also
passed on the pre-performance-fix revision, whose archive source is
identical. The broad local shard is not claimed as one passing
execution. The same-head CI retry supplies complete final CI coverage.
- Nothing is merged or deployed.

## Risks

- Exact counts still grow with the readable hierarchy and comment
history. The change reduces repeated work; it does not cap or
approximate counts.
- A relevant summary batch can contain several recent parents. Older
servers retain a broader summary fallback until upgraded.
- Live delivery adds bounded, permission-checked parent lookups. Short
delivery-window reuse limits repeated reads.
- The authorization fast path applies only to open tasks with no privacy
parent and no project. Restricted cases retain the shared predicate.
- No migration is required.

## Model Used

OpenAI Codex, based on GPT-6. Used code execution, native GitHub tools,
PostgreSQL and Chromium verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked a public issue or described the issue in this
PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal issue
id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open findings
- [x] I will address all review comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-15 08:51:04 -06:00
DottaandClaude Opus 5.5 348ed0ae10 fix(runner): keep closing tool-call tags and model control tokens out of OpenCode chat (#21761)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner OpenCode driver streams model text into the
chat as progress and selects the final reply
> - DeepSeek v4.1 flash (through OpenRouter) wrote tool-call scaffolding
and tokenizer control tokens as ordinary text
> - #21361 hides tool-call markup, but its detector matched only opening
DSML tags, and nothing recognised control tokens
> - A person saw raw `</|DSML| invoke>` lines in a progress message, and
in another conversation got a degenerate reply instead of an answer
> - This pull request extends the same detector to closing DSML tags and
to model control tokens, on the streaming path and at completion
> - The benefit is that scaffolding never reaches the chat, and a
degenerate reply goes to the existing correction instead of being posted

## Linked Issues or Issue Description

Refs #21361

**What happened?**
Chat-eval cohort 16 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master 819f61f1fd)
showed two defects.
1. In `task-coordination.block-on-design`, after a real
`paperclip_finish` call, the model streamed a text part that held only
closing tags (`\r\n</|DSML| invoke>\n</|DSML|
calls>…</invoke>\r\n</parameter>`). The driver emitted it as an
`agentMessage` progress delta, so the chat showed raw markup. The cause:
`SENTINEL_FORMS` matched `<|DSML|` but not `</|DSML|`.
2. In `cases.case-history`, the reply to the second follow-up was
`<|begin▁of▁sentence|><?xml version="1.0" …?>\n<output>\nSearch output
truncated. Let me search more specifically…\n</output>`. The driver took
it as the final text part and posted it. The question was never
answered.

**Expected behavior**
Tool-call scaffolding and model control tokens never appear in progress
or in the reply. A reply made only of them goes to the empty-response
correction, which asks the model for the real answer. Real answers,
including answers about XML, chat templates or tool tags, stay as
written.

**Steps to reproduce**
Feed the recorded text parts above to the OpenCode driver (the new
`closing-markup` and `control-token-reply` cases in
`empty-response.test.ts` do this).

## What Changed

- `tool-call-markup.ts`: closing DSML tags (`</|DSML|`, `</|DSML|`)
count as tool-call markup.
- `tool-call-markup.ts`: new `controlTokenStart` finds DeepSeek, ChatML
and Llama control tokens (for example `<|begin▁of▁sentence|>`,
`<|im_start|>`, `<|eot_id|>`) outside code. A new
`visibleBeforeModelScaffolding` holds back both kinds while text
streams.
- Sentinels inside inline code spans are ignored, as fenced code already
was.
- A per-line fast path skips the sentinel scan when a line has no `<|`,
`<|`, `</|` or `</|`.
- `opencode-server-driver.ts`: prose before a control token stays the
answer, with the token removed (for example a trailing end-of-sentence
token). A part with no prose before the token is discarded with the
`opencode_control_token_reply_discarded` diagnostic. The existing
empty-response correction then asks for the answer, with an opening that
names the control tokens. Unlike tool-call markup, this does not make
earlier prose ineligible.

## Verification

- `pnpm vitest run packages/paperclip-runner/src/drivers/opencode/` (20
files, 296 tests pass).
- Red/green: with master's detector and driver, 9 new detector tests and
the `closing-markup` and `control-token-reply` driver cases fail. With
this change they pass.
- The tests use the recorded cohort 16 text. Negative controls: an XML
answer, an answer that names an `<output>` tag, control tokens in inline
code and in a fence, HTML, and comparison operators are left alone. A
driver case checks that `answer.<|end▁of▁sentence|>` keeps the answer
with no correction.
- Per-message cost (local tsx micro-benchmark, 20k calls, one prose line
with `<`): 500 chars 0.3 → 0.8 µs; 2 KB 2.2 → 2.6 µs; 8 KB about 8–16 µs
before and 7–9 µs after (noise range). The detector runs per text-part
update, as before. Nothing runs before the turn starts.
- Paid narrow eval run: results are added in a PR comment.

## Risks

- A real answer that writes a literal control token outside code is cut
at the token. Prose before it stays. That text is never meaningful
prose.
- Closing DSML tags in plain prose (outside code) now hide the rest of
the part. DSML tags are model-internal, so prose does not contain them.

## Model Used

- Claude, `claude-opus-5-5` (Claude Code agent with tool use and code
execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:44:00 -06:00
DottaandClaude Opus 5.5 e7f31705da perf(release): stage the verified host release daemon instead of compiling it in publish_canary (#21555)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every master push runs a canary, and only one canary runs at a time,
so its duration sets how often merged work ships
> - `publish_canary` takes about 30 min. 8 min of it is `pnpm build`,
and 4 min 40 s of that is an uncached `cargo build --release` of
`paperclip-runnerd`
> - The release then replaces that binary with the exact-source daemons
that `release_runner_binaries` already built and verified for all four
targets
> - This pull request stages the verified daemon for the publisher's own
target during a release, instead of compiling it
> - The benefit is about 4.5 min less per canary, with the same verified
bytes published

## Linked Issues or Issue Description

**What happened?**
In canary run 43487884481 (2026-11-13), the `Publish canary` step took
28.6 min. Its `pnpm build` ran 19:42-19:50. The
`packages/paperclip-runner build` took 5.3 min, and its `cargo build`
reported `Finished release profile ... in 4m 40s`. Every other package
waited for it. `release.sh` then runs `assemble-daemons`, which
overwrites `dist/bin` with the four exact-source release daemons.

**Expected behavior**
A release does not compile a binary that it discards. The Runner build
stamp stays valid, so the server build still reuses the Runner build.

Refs #21554 (the other canary speed-up)

## What Changed

- `packages/paperclip-runner/scripts/build-runner-binary.mjs` (new):
when `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR` is set, it checks the
release daemons with the existing `releaseDaemonManifest` (source
revision equal to `HEAD`, digest, architecture, build metadata) and
stages the host target's daemon at `dist/bin` with the existing
`stageRunnerBinary`. Otherwise it runs `pnpm run build:binary` as
before.
- The Runner `build` chain calls this script in place of `pnpm run
build:binary`. `build:binary` itself and the image build are unchanged.
- `stage-release-runner-binaries.mjs` exports `metadataContract`.
- Tests: `stage-runner-binary.test.mjs` (staging per target, wrong
source, bad digest, missing target, non-release fallback);
`runner-image-identity.test.ts` step list.
- `doc/RELEASING.md` describes the behavior.

## Verification

- `node --test
packages/paperclip-runner/scripts/stage-runner-binary.test.mjs`: 9 of 9
pass.
- `server`: `vitest run src/__tests__/runner-image-identity.test.ts`: 42
of 42 pass.
- With the real daemon artifacts of canary run 43487884481 (receipts
pointed at this commit): `PAPERCLIP_RELEASE_RUNNER_ARTIFACTS_DIR=...
pnpm run build` in `packages/paperclip-runner` finished in 15 s with no
`cargo` line. The build stamp was recorded, and
`server/scripts/prepare-runner-vendor.mjs` then reported
`action:reuse,reason:inputs_unchanged`. The staged daemon runs.
- After merge: the next canary's `packages/paperclip-runner build` log
shows "Staged the verified exact-source linux-x64 release daemon" and no
`Compiling` lines.

## Risks

- Low. A release still fails closed: a daemon with the wrong source,
digest, architecture or metadata stops the build, and `assemble-daemons`
checks all four again afterwards.
- Local, Docker and CI builds without the variable are unchanged.
- On macOS, staging re-signs the binary, as it does for a compiled one.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:40:21 -06:00
DottaandClaude Opus 5.5 689d62661d fix(server): wake a delivery task unscoped when it cannot apply settled hints (#21686)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The delivery work coordinator wakes background tasks when a database
work topic settles, and passes the settled scope hints to the task's
`onSettled`
> - The coordinator calls `onSettled(hints)` before `wake()`
> - If `onSettled` throws on a hint that it does not recognise, the task
is not woken. Only a generic listener warning is logged
> - Hints now cross server processes (#21681), so a hint shape from
another server version can arrive during a deploy overlap
> - This pull request makes the coordinator log a warning, apply the
settlement unscoped once, and always wake the task
> - The benefit is that a hint that a task cannot read never makes the
task miss committed work

## Linked Issues or Issue Description

Follow-up to #21681 (merged). This branch is based on master.

**What existing behavior does this improve?**
How the delivery work coordinator handles an `onSettled` callback that
throws.

**Subsystem affected**
`server/src/services/delivery-work-coordinator.ts`.

**Current behavior**
`subscribeDatabaseWork` calls the coordinator listener, which calls
`task.onSettled?.(hints)` and then `wake()`. If `onSettled` throws,
`wake()` does not run. The work-signal publisher catches the error and
logs "Database work listener failed". The task stays asleep until a
different event or deadline wakes it.

**Proposed behavior**
If `onSettled(hints)` throws, the coordinator logs a warning and calls
`onSettled(undefined)` once, which is the unscoped settlement. If that
also throws, it logs a second warning. In all cases it then wakes the
task.

**Reason and benefit**
A settled event means committed work can be ready. A task must never
miss it because a hint could not be applied.

**Breaking changes**
None.

## What Changed

- `delivery-work-coordinator.ts`: a settled event goes through
`applySettled(hints)`. It tries `onSettled(hints)`. If that throws and
hints were given, it tries `onSettled(undefined)` one time. Each failure
logs `Delivery settlement could not be applied for <queue>` with
`process.emitWarning`. Then `wake()` runs.
- Unscoped means `onSettled(undefined)`. Each registered task already
defines what that means. For example, scoped deadline queues, dispatch
work, muse maintenance and the revocation index mark `*` dirty.
Counter-style tasks set their request flag. `chat-action-work` does
nothing. The fallback is bounded: it calls `onSettled` at most twice per
event, it does not recurse, and it does not schedule another settlement.
So it cannot loop.
- `work-signals.test.ts`: a new test pins one rule of #21681 for the
local path and the remote path. An unhinted `signalDatabaseWork` in the
same transaction as a hinted one gives a settlement that is scoped to
the hinted set. The owner of `work-signals.ts` confirmed that this is
deliberate. The repository does not allow code comments in pull
requests, so this sentence and the test document it.

## Verification

- `cd server && npx vitest run
src/__tests__/delivery-work-coordinator.test.ts`: 31 pass. New tests:
- An `onSettled` that throws on an unknown hint shape: the task gets
`[hint]` and then the unscoped settlement, it is woken once, and one
warning is logged. `onError` is not called.
- An `onSettled` that always throws: the task gets 2 calls and is woken
once, with 2 warnings. A later unscoped event gives 1 call, 1 wake and 1
warning. No more runs happen after 60 seconds, so there is no loop.
- A normal `onSettled`: it gets the hints once, the task is woken once,
and no warning is logged.
- Red/green: with the coordinator from master, the 2 throwing tests
fail. `onSettled` is called only with the hints, and there is no
unscoped settlement and no wake. With this branch, all tests pass.
- `cd packages/db && npx vitest run src/work-signals.test.ts -t
alongside`: passes on real PostgreSQL. The local listener and the
receiving process both get `["company-1"]`.

## Risks

- Low risk. The change applies only when `onSettled` throws. Before,
that path skipped the wake. Now it wakes with an unscoped settlement.
For scoped deadline queues an unscoped settlement is a `*` reload. That
is the existing safe fallback, and the reconstruction interval of the
queue still throttles it.
- A task whose `onSettled` throws on every event now runs on every
settled event and logs a warning each time. Before, it did not run. The
warning makes such a defect visible.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:02:54 -06:00
DottaandClaude Opus 5.5 07d5a6ee17 perf(db): carry scope hints in cross-process work notifications (#21681)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Server processes share durable queues and wake each other with a
commit-bound PostgreSQL NOTIFY
> - The NOTIFY payload carries only `{version, origin, topic}`, so the
receiver gets an unscoped "settled" event
> - In `scoped-deadline-work.ts` an unscoped settle becomes `*`, which
reloads every candidate of that queue on the whole instance
> - On a large instance one cross-process notification can re-check
about 850 open tasks, at about 22 statements each
> - This blocks more than one server process per stack, and it costs
work during every deploy overlap
> - This pull request puts the topic's hints in the NOTIFY payload, so
the receiver reloads only the hinted scopes
> - The benefit is that a scoped commit in one process costs the same
scoped reload in every other process

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Cross-process work notifications between server processes on one
database.

**Subsystem affected**
`packages/db` work signals (`work-signals.ts`) and their consumers, such
as `server/src/services/scoped-deadline-work.ts`.

**Current behavior**
`receiveDatabaseWorkNotification` publishes "settled" with no hints. A
scoped deadline queue then reloads all of its candidates, throttled only
by its reconstruction interval.

**Proposed behavior**
The NOTIFY payload carries the deduplicated hints of its topic. The
receiving process publishes "settled" with the same hints that a local
commit publishes. Overflow and malformed input fall back to the old
unscoped behavior.

**Reason and benefit**
A scoped commit no longer causes an instance-wide reload in other
processes. This is necessary before a stack runs more than one server
process.

**Breaking changes**
None. Old and new processes interoperate in both directions (see Risks).

## What Changed

- `packages/db/src/work-signals.ts`: the per-level pending notifications
are now a map of topic to `{ hints, unscoped }`. Savepoint merge and
rollback keep the same rules as before.
- The hints follow the local "settled" semantics: the union of the hints
from `signalDatabaseWork` and `publishDatabaseWorkSettled`. An unhinted
`publishDatabaseWorkSettled`, `notifyDatabaseWorkOnCommit`, or a topic
with no hints sends the unscoped payload.
- A hinted payload is `{version: 1, origin, topic, hints: [...]}`. The
field is additive and the version stays 1.
- Each payload is at most 1,024 bytes
(`DATABASE_WORK_NOTIFICATION_MAX_BYTES`), because current receivers drop
any payload longer than 1,024 characters before they parse it. Many
hints split into several payloads. The payloads of one topic are at most
7,000 bytes in total (`DATABASE_WORK_TOPIC_NOTIFICATIONS_MAX_BYTES`). If
a topic exceeds this cap, or one hint alone does not fit, the topic
sends one unscoped payload.
- All payloads still go in the one batched `select pg_notify(...) from
jsonb_array_elements_text(...)` statement at commit. There are no added
statements or awaits on the commit path.
- `receiveDatabaseWorkNotification` accepts payloads up to 8,000
characters (the PostgreSQL limit). It keeps the drop of own-origin
payloads and of payloads without a usable origin or topic. A missing,
malformed, or empty hint list, or a version other than 1, gives an
unscoped wake. It never throws.
- The `remote_notification` debug event now logs the received hints
(`hints`: a JSON array, or `null` when unscoped).
- Tests: new payload, compatibility and fallback tests in
`packages/db/src/work-signals.test.ts`, and a new two-process test on
real PostgreSQL in
`server/src/__tests__/scoped-deadline-work-notifications.test.ts`.

## Verification

- `cd server && npx vitest run
src/__tests__/scoped-deadline-work-notifications.test.ts`: two pools on
one embedded PostgreSQL database. Process A commits work. Process B runs
a real delivery coordinator with `registerScopedDeadlineWork` over 800
candidate rows in 20 companies.
- Master (red): a commit hinted with one company makes B load `*`: 1
load, 801 rows.
- This branch (green): B loads only `company-7`: 1 load, 41 rows. A
commit with two hints loads only those two companies (80 rows). An
unscoped commit still loads all 801 rows.
- `cd packages/db && npx vitest run src/work-signals.test.ts` (32 tests,
real PostgreSQL). New tests:
- deduplicated hints per topic, including a released savepoint and a
rolled-back savepoint
- split into payloads of at most 1,024 bytes, at most 7,000 bytes in
total, and the union of hints is complete
  - overflow and one oversized hint fall back to one unscoped payload
- malformed hint lists and unknown versions wake unscoped; unusable
payloads do not wake; own-origin payloads are dropped; nothing throws
- an old-format payload `{version, origin, topic}` gives an unscoped
wake
- every new payload passes a copy of the receiver from master, which
reads it as an unscoped wake for the correct topic
  - the debug event logs the received hints
- With master's `work-signals.ts`, 5 of these tests fail (red). With
this branch, all pass.
- Also run: `scoped-deadline-work.test.ts`,
`delivery-work-coordinator.test.ts`, `deployment-notifications.test.ts`,
`run-start-handover.test.ts`, `heartbeat-run-start-handover.test.ts`,
`heartbeat-run-event-atomic-append.test.ts` (111 tests pass),
`work-signal-debug.test.ts` (12 pass), and `tsc --noEmit` for
`packages/db`.

## Risks

- Mixed deploys, new sender to old receiver: every payload is version 1
and at most 1,024 bytes. The old receiver ignores the `hints` field and
wakes unscoped, as today. A commit with many hints sends several
payloads, so an old receiver can get up to about 7 unscoped wakes for
one commit. The coordinator merges them into one `*` reload.
- The sender bounds bytes, and old receivers compare `.length` in UTF-16
code units. A string never has more UTF-16 code units than UTF-8 bytes,
so a payload of at most 1,024 bytes always passes the old check.
- PostgreSQL collapses identical NOTIFY payloads in one transaction. The
chunks of one topic always carry different hints, so PostgreSQL drops
none of them.
- Mixed deploys, old sender to new receiver: the old payload has no
`hints`, so the new receiver wakes unscoped, as today.
- I did not use one large payload of about 7 KB: current receivers drop
any payload over 1,024 characters before they parse it, so they would
miss the wake during a deploy overlap.
- Consumers that read hints (for example `chat-action-work`,
`approval-queue-seeding-work`) now get the same hints from another
process that they already get from a local commit. Before, they got no
hints from another process.
- The receiver accepts hint strings from the database channel. Consumers
decode them in the same way as local hints. `scoped-deadline-work`
already turns a decode failure into a `*` retry.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 05:38:06 -06:00
DottaandClaude Opus 5.5 01c1e708e9 fix(runner): stop a turn that keeps resending a refused tool call and continue fresh (#21734)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools, and
`RepeatedToolFailureGuard` refuses an exact repeat of a call that
already failed validation twice
> - The refusals are not counted and never end the turn, so a model
whose host drops an argument can resend the same refused call until the
run's wall clock
> - In a recorded restart-chaos wave, an opencode/DeepSeek agent sent
the same `PATCH /api/issues/{id}` through `call_api` 431 times in 40
minutes without `body`; 427 were refused. The run held an agent slot and
delayed other tasks
> - The system knows the next step (stop the turn and continue on a
fresh session), so the harness must own it, with no Retry banner
> - This pull request counts refusals, stops the provider turn after a
bounded number with the classified cause `agent_tool_call_loop`, and
queues an automatic fresh-session continuation that tells the model to
use a dedicated tool
> - It also records the declared argument names that each opencode tool
call delivered, so we can tell whether the bridge or the model drops
`body`
> - The benefit is that a looping agent frees its slot in seconds
instead of 40 minutes, and the task continues by itself

## Linked Issues or Issue Description

**What happened?**
An opencode agent on DeepSeek created two subtasks and then tried to set
their descriptions with `call_api` `PATCH /api/issues/{id}`. Every call
arrived with `operationId`, `pathParams`, `query` and `responseText`,
but without `body`. The guard refused each repeat. The model said "I
need to include body" more than 20 times and kept sending the same call:
431 calls, 427 refused, over 40 minutes. The run stayed active and its
agent slot delayed other queued work.

**Expected behavior**
After a small, bounded number of refusals, Paperclip stops the provider
turn and continues the task automatically on a fresh session, with a
hint to use a dedicated Paperclip tool. The board shows no Retry banner
because the system owns this recovery.

**Steps to reproduce**
1. Run a native agent whose model resends one tool call with a missing
required argument.
2. The guard refuses the third and later identical calls.
3. On master, the refusals never end the turn.

**Paperclip version or commit**
`master` at 6400698a8e.

## What Changed

- `repeated-tool-failure-guard.ts`: counts refusals per exact call and
per run. At 6 refusals of one call, or 20 refusals in the run, it throws
`ToolCallLoopError` (`code: "agent_tool_call_loop"`, tool, counts and
argument names, never values). The message tells the model the turn
stops and gives a short hint.
- `paperclip-runner-tool-authority.ts`:
`scheduleToolCallLoopContinuation` writes one idempotent native
status-wake outbox row (key `agent_tool_call_loop:<runId>`) with
`forceFreshSession: true` and the hint as the wake instruction. The
outbox dispatcher admits it with `allowRunCoalescing: false`, so it
never merges into a deferred comment wake or the active run. An assignee
run uses `issue_continuation_needed`. A completion-review run uses
`native_completion_review` with its review assignment, which the
dispatcher revalidates, so the fresh reviewer session keeps the review.
- `native-session-executor.ts`: the session authority epoch queues that
continuation when the guard raises the loop error. The governed-wait
lookup finds the wake on the next tool event and ends the provider turn
as `yielded` with a `same_agent` continuation, the same way the
connection-tools refresh ends a turn. If the wake cannot be queued, the
refusal still reaches the model and the error is logged.
- `status-decision-committer.ts`: a yielded continuation with that key
binds to the queued wake instead of creating a second one.
- `status-arbiter.ts` and `native-run-finalizer.ts`: a pending review
run that ended through this continuation skips the
`native_review_unresolved` resume recovery, because the queued review
wake owns the next reviewer run.
- `mcp-bridge.ts` and `opencode-proxy-finish-diagnostics.ts`: the
opencode tool-delivery diagnostic records `argumentNames` (only names
declared in the tool's input schema, sorted, at most 1,024 characters of
JSON), `argumentNamesTruncated` and `undeclaredArgumentCount`. Values
and undeclared names are never recorded. The forwarder's closed key
allowlist accepts the new fields and validates them, so a record stays
under its 2,048-character limit.

### Threshold choice

- **6 refusals of one exact call.** The guard already runs a call twice
and then refuses with an explicit correction. Six more refusals (eight
identical attempts in total) give a model that reads errors many chances
to change its arguments. In the recorded run the model never changed
them in 427 refusals, so six refusals end that loop within seconds.
- **20 refusals in the run.** A model can alternate between a few
refused variants and keep each one under 6. Refusals happen only for
exact repeats of a certain validation failure, so a healthy run almost
never reaches 20.

### Previous-release story

The guard counters live in memory for one run's tool authority, as
before. A deploy or restart during a run starts the counters again at
zero on the new process; that only delays the stop. The continuation is
an ordinary wake row. If a server on the previous release finalizes a
run that a new server parked, it does not know the key and queues its
own `same_agent` continuation too; the issue execution lock lets one run
and the other coalesces.

## Verification

- Red then green, `repeated-tool-failure-guard.test.ts`: "after N
refusals the guard raises a turn-stopping loop error" and "stops the
turn once refusals across different calls reach the run limit". Both
fail on master.
- Red then green, `native-session-executor.test.ts`: "a turn whose model
repeats a refused call N times ends with agent_tool_call_loop and
schedules a continuation". With the epoch handling disabled it fails (13
calls instead of 8, nothing parked). Also "keeps the refused call's
error when the loop continuation cannot be queued".
- `paperclip-runner-tool-authority.test.ts`: "queues one fresh-session
continuation outbox for the assignee / a completion reviewer stopped by
a tool-call loop" (real Postgres; idempotent).
- `native-tool-call-loop-continuation.test.ts`: with a comment wake
already deferred behind the looping run, the continuation keeps its own
keyed deferred wake, the comment wake is unchanged, and after release a
run starts with `forceFreshSession` and the loop cause.
- `status-arbiter.test.ts`: a pending reviewer stopped by a loop
releases its checkout without a resume recovery.
- `native-status-arbiter-corpus.test.ts`: the existing wake-binding test
now also covers the `agent_tool_call_loop` key.
- `mcp-bridge-delivery.test.ts` and
`opencode-proxy-finish-diagnostics.test.ts`: argument names are
recorded, values and undeclared names are not, invalid records are
dropped, and a boundary test with 32 names of 64 characters stays
forwardable with `argumentNamesTruncated: true`.
- Server: the six touched test files pass (1755 tests). Runner:
`src/drivers/opencode` and `src/cli` pass (569 tests). `tsc --noEmit`
passes for `server` and `packages/paperclip-runner`.
- `pnpm verify` exits 2 (runner package changes need broad coverage), so
required CI supplies the broad run.

## Risks

- A run that hits the limit ends its turn early. The limits are far
above what a model that reads its errors needs, and the task continues
automatically on a fresh session.
- The fresh session loses the in-turn conversation. The task, its
comments, documents and workspace stay, and the continuation names the
cause and the hint.
- The diagnostic now carries declared argument names. These are schema
names, not values, and the forwarder validates the shape before it
forwards a record.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21360

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 01:52:50 -06:00
Nicky LeachandClaude Opus 5.5 a78be8b601 feat(native): send launch preparation and run digests to plugin task prepare (#21717)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Environment plugins can run a native Runner as a plugin task, and a
warm session keeps that Runner between turns
> - On each warm turn Core sends the plugin a `prepare` operation with
the full launch requirements for the new run
> - The requirements' `requirementsDigest` covers the run identity, and
the run identity changes on every turn
> - So a plugin cannot tell that only the run moved, and it must redo
its whole preparation every warm turn
> - This pull request gives the plugin two digests: one that stays the
same while its preparation stays valid, and one that names the run
> - The benefit is that a plugin can transfer only the run identity when
nothing it reconciles changed

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The `prepare` operation that Core sends to an environment plugin for a
plugin Runner task, on cold starts and on warm attaches.

**Subsystem affected**
Native runtime, plugin Runner tasks, the Runner launch contract, and the
plugin SDK environment task contract.

**Current behavior**
A warm attach sends the full requirements. The only digest available,
`requirementsDigest`, includes `runId`, `turnId` and `itemId`, so it
changes on every turn even when the launch is otherwise the same.

**Proposed behavior**
Core computes two digests for every `prepare`:
- `preparationDigest`: the launch requirements an environment reconciles
when it prepares a Runner:
  - the runner, lease and session identity;
  - provider, model and settings;
  - the prompt and instruction text;
- the runtime context's instruction and skill bundles, working copy and
connection instructions.

It leaves out the `runId`, `turnId` and `itemId`, and the runtime
context's MCP assignment and binding (with `aggregateDigest`, which
hashes the assignment). MCP reaches the Runner through Core's tool
authority on every run attachment, never through files the environment
publishes. Core already continues a warm session across an MCP-only
change (`nativeRuntimeContextsMatch` with tool refresh). The binding is
also named per run (`native-mcp:<runId>`). So the digest stays the same
across warm turns of one Runner, lease and session, including when the
agent's MCP access changes.
- `runDigest`: `preparationDigest` plus everything it leaves out:
`runId`, `turnId`, `itemId`, and the MCP assignment and binding. It is
unique to one run's launch.

A plugin that advertises `launchDigestVersion: 1` in its task
description gets both fields on `prepare`.

**Reason and benefit**
A plugin can compare `preparationDigest` with its saved preparation and
transfer only the run identity when they match, instead of a full
preparation.

**Breaking changes**
None. Plugins without `launchDigestVersion` get no new fields.

## What Changed

- `packages/paperclip-runner/src/live/runner-provider-launch.ts`: add
`runnerLaunchPreparationDigest` and `runnerLaunchRunDigest`. The
preparation digest drops the MCP assignment and binding; the run digest
takes the full requirements and includes them. Both use the same
canonical JSON as `runnerLaunchRequirementsDigest`, with their own
schema tags so the three digests never collide. Export them from the
package and the server's vendored runner shim.
- `packages/plugins/sdk/src/environment-tasks.ts`: add optional
`preparationDigest` and `runDigest` to the `prepare` operation, and
optional `launchDigestVersion: 1` to the task description. The
capability gates both digests.
- `server/src/services/native-runtime/plugin-runner-task.ts` and
`native-session-executor.ts`: send both digests on `prepare` when the
description advertises `launchDigestVersion: 1`.
- `server/src/services/environment-task-runtime.ts`: recompute both
digests from the requirements, and refuse a `prepare` whose digests are
missing in part or do not match.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the SDK
public source pin.
- Tests:
- the preparation digest is stable across runs and changes when the
runner, lease, session, instructions, model, approval policy or settings
change;
- with a runtime context, an MCP-only change (the run's binding, no
binding, a changed access set) keeps the preparation digest and changes
the run digest;
- a change to the instruction bundle, instruction entry, a skill bundle,
the skill set, the working copy or the connection instructions changes
the preparation digest;
  - the run digest changes per run;
  - Core sends the digests only when advertised;
- the runtime refuses forged or partial digests before it calls the
plugin.

## Verification

- `npx vitest run src/live/runner-launch-preparation.test.ts` in
`packages/paperclip-runner`: 26 passed.
- `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts
src/__tests__/environment-task-runtime.test.ts` in `server`: 160 passed.
- `npx vitest run source-sdk-public-inputs.test.mjs
source-sdk-evidence.test.ts source-sdk-public-type.test.ts` in
`tests/runner-e2e`: 59 passed.
- `pnpm typecheck` in `server` and in `packages/paperclip-runner`:
clean.
- Not run locally: the full suite. CI covers it.

## Risks

Low risk. The digests are new optional fields behind a plugin capability
version. Core computes them and checks them again before it calls the
plugin, so a plugin can rely on them. They prove only that the launch
requirements are unchanged. A plugin that skips work on a match must
still check its own state, such as files it published into the sandbox.
If the runtime context carries values that change per run, the
preparation digest changes too, so a plugin does its full preparation.
That is safe, but the plugin saves nothing. The MCP binding is the one
such value Core sets per run today, and MCP is excluded entirely. A
plugin that does publish MCP configuration into its sandbox must not
rely on this digest for it. Values outside the requirements are not
covered at all: the account identity and credentials, the GitHub broker,
project IDs, and the plugin's own runtime revision. A plugin must keep
checking those itself. A match lets a plugin skip publishing its launch
configuration, not computing the descriptor: `prepare` must still return
a launch whose `requirementsDigest` matches the current requirements,
which carry the current MCP assignment and binding.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:07:54 -08:00
DottaandClaude Opus 5.5 b1e70ab6dd fix(db): recover prepared statements that a pooler or schema change invalidates (#21718)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server runs every database statement from one Node main thread.
With `prepare: true`, postgres.js sends a statement in one round trip
instead of two, and the CPU cost of each statement is about half
(benchmark from #21707).
> - `DATABASE_PREPARED_STATEMENTS=true` turns on preparation for ORM
queries (#17673). Hosted instances connect through a transaction-mode
pooler (PgBouncer 1.22 or later, Neon `-pooler` hosts), which supports
protocol-level prepared statements.
> - I tested the flag locally through PgBouncer 1.26.1 in transaction
mode. Evictions, server connection churn, `RECONNECT`, transactions,
pipelining, `SET LOCAL`, advisory transaction locks and savepoints all
work. One failure is not acceptable: after a migration changes a column
type, every prepared statement that reads that column fails, for every
client and for every new process.
> - PgBouncer shares one server-side prepared statement by SQL text.
PostgreSQL never repairs a named statement whose result type changed, so
it fails with `cached plan must not change result type` until PgBouncer
drops it. The postgres.js retry prepares the same text again, and
PgBouncer gives it the same broken statement. postgres.js also caches
the failed statement without columns, so the connection throws
`TypeError: Cannot read properties of undefined (reading 'length')` for
that text even after the schema is correct again.
> - This pull request makes the client recover. When a prepared
statement fails with a prepared-statement error, the client stops
preparing that text, retries once without preparation where that is
safe, and counts it. It also caps the number of prepared texts.
> - The benefit is that the flag can be turned on behind a pooler
without a deploy migration that breaks reads. Behavior with the flag
unset or `false` does not change.

## Linked Issues or Issue Description

Refs #17673 (the opt-in), #10795 (the flag), #21707 (the CPU benchmark).

**What happened?**

With `DATABASE_PREPARED_STATEMENTS=true` behind PgBouncer 1.26.1 in
transaction mode (`max_prepared_statements=5`, local PostgreSQL 17), a
change of a column type made every later execution of an affected
statement fail. A new client process failed in the same way. Measured
with the local probe, 20 executions after the change and 10 from a new
process:

| Change | Before: failures | After: failures |
| --- | --- | --- |
| `int` to `bigint`, ORM read outside a transaction | 20/20, new process
10/10 (0A000) | 0/20, 0/10 |
| `varchar` length change, ORM read | 20/20, 10/10 | 0/20, 0/10 |
| Add column, `select *` | 20/20, 10/10 | 0/20, 0/10 |
| `int` to `bigint`, first statement of a transaction | 20/20, 10/10 |
0/20, 0/10 |
| `int` to `bigint`, second statement of a transaction | 20/20, 10/10 |
1/20, 1/10 |
| Table recreated with its original shape, same client | 10/10
(TypeError in `DataRow`) | 0/10 |
| Add column, explicit-column ORM read | 0/20 | 0/20 |

**Expected behavior**

A statement whose pooled prepared statement is stale or lost runs
anyway, and the failure does not reach the caller.

**Steps to reproduce**

1. Run PostgreSQL behind PgBouncer 1.22 or later with `pool_mode =
transaction` and `max_prepared_statements` above 0.
2. Start Paperclip with `DATABASE_PREPARED_STATEMENTS=true` and run a
read of a table a few times.
3. Change the type of a column that the read returns, for example `alter
table ... alter column n type bigint`.
4. Run the read again from the same process or from a new process: it
fails with `cached plan must not change result type`.

## What Changed

- `packages/db/src/prepared-queries.ts`:
- The prepare decision for each `unsafe` query is made when the driver
builds the query, not when the caller creates it, so a query that waits
on a busy connection sees a text that was demoted after it was queued.
- A failure with 26000 (`prepared statement does not exist`), 42P05
(duplicate statement name), or 0A000 `cached plan must not change result
type` demotes the text: the client does not prepare it again in that
process.
- A pooled statement that fails in that way is retried once with
`prepare: false`. Statements in a reserved connection, cursors,
`forEach`, `describe` and queries with `onexecute` are not retried.
- When the failed statement is the first statement that the transaction
ran (only `SET TRANSACTION` before it), the client restarts the
transaction on the same connection: `ROLLBACK`, `BEGIN` with the same
options, the same `SET TRANSACTION` statements, then the statement
unprepared. The callback runs once and receives the result. Tagged
statements and statements in savepoints count toward the execution
order. The client does not restart when another statement of the
transaction is queued, and it never restarts inside a savepoint.
- At most 1,000 distinct texts are prepared for each client and its
private pools. Texts longer than 8 KB are not prepared. Explicit
per-query `prepare` options, simple queries and the flag-off paths are
unchanged.
- `packages/db/src/client.ts`, `packages/db/src/index.ts`: a client and
its dedicated, liveness, checkpoint, file-storage and listener pools
share one state. `preparedStatementStats(db)` returns the prepared,
unprepared, demoted and retried counts.
- `packages/db/src/prepared-queries-postgres.test.ts`: nine tests
against embedded PostgreSQL: restart after a stale plan and after a lost
statement (the callback runs once and its later work commits),
`repeatable read read only` kept after a restart for driver and ORM
transactions, no restart after a later statement, after a tagged first
statement, or for an out-of-order await, a savepoint whose first
statement fails while the outer transaction continues once, the pooled
retry with `.values()`, and the cap.
- `doc/DATABASE.md`, `docs/deploy/database.md`: the cap and the
recovery.

## Verification

- Red/green, `pnpm --filter @paperclipai/db exec vitest run
src/prepared-queries-postgres.test.ts`. With the old
`prepared-queries.ts` (stub stats exports only), the five new tests
fail: two transactions fail with the stale-plan and the
missing-statement errors, the ORM read fails with 42P05, the invalidated
text is prepared again, and 5 texts are prepared where the cap is 2.
With this change, 23/23 pass.
- `pnpm --filter @paperclipai/db exec vitest run` on
`prepared-queries-postgres`, `client-options`, `client`,
`dedicated-connection-pool`, `dedicated-connection-admission`,
`response-capture-postgres`, `lazy-transaction-queries`,
`client-disconnect-replay`: 91/91 pass on the final revision. `tsc
--noEmit -p packages/db` passes.
- 31 server test files with `DATABASE_PREPARED_STATEMENTS=true`
(services, routes, inbox, recovery, heartbeat events, agent auth,
prepared-statement helpers): 829/830 pass. The one failure
(`execution-workspaces-service`, a git-scan timing test) passes alone
with the flag on and off. The machine load average was 9 to 20.
- Local PgBouncer probe (not in CI, because CI has no PgBouncer):
PgBouncer 1.26.1, PostgreSQL 17, postgres.js 3.4.9 (patched), Node 26.4.
The results in the table above, plus 0 failures for 4,800 statements
over 60 distinct texts with `max_prepared_statements=5`, 0 failures with
`pg_terminate_backend` and `RECONNECT` every 25 statements, and 0
failures for transactions with `SET LOCAL`, pipelines of 20 statements,
`pg_advisory_xact_lock` and savepoint rollback.

- Through PgBouncer 1.26.1 (transaction mode,
`max_prepared_statements=5`, 4 server connections) and PostgreSQL 17, at
head 84732c8f67, compared with the master `prepared-queries.ts`. Each
row ran 10 times on its own client, warmed before the migration. "Type"
is `int` to `bigint` under an ORM read; "star" is an added column under
`select *`. Failures before → after:

| Row | Type change | Added column, `select *` |
| --- | --- | --- |
| (a) outside a transaction, same process | 10 → 0 | 10 → 0 |
| (b) first statement of an ORM transaction (callback runs once) | 10 →
0 | 10 → 0 |
| (b) first statement, `begin isolation level repeatable read read only`
| 0 → 0 (the statement was evicted and prepared again) | 10 → 0 |
| (b) first statement, ORM `repeatable read`, `read only` (`SET
TRANSACTION` repeated) | 10 → 0 | 10 → 0 |
| (c) later statement of a transaction, same process | 10 → 1 | 10 → 1 |
| (d) new process after the migration: (a) / (b) / (b) driver / (b) ORM
/ (c) | 10/10/0/0/0 → 0/0/0/0/1 | 10/10/10/10/10 → 0/0/0/0/1 |
| (e) same client after the table is restored: (a) / (b) / (b) driver /
(b) ORM / (c) | 10/10/0/10/10 → 0 | 10 TypeError each → 0 |

Every restarted transaction checked `transaction_isolation = repeatable
read` and `transaction_read_only = on` after the restart. The PostgreSQL
log shows each restart as `ROLLBACK`, `BEGIN` with the same options,
then the statement as an unnamed statement. With 12 clients restarting
at once over 2 server connections and 6 clients of background load,
12/12 transactions succeeded. In the 15 restarts I paired by
`application_name`, the second `BEGIN` ran on the same server connection
as the `ROLLBACK`, because both are sent together. A different server
connection is also safe: only `SET TRANSACTION` ran before the restart
point, and the client repeats it after the new `BEGIN`.
- Silent rollback check through PgBouncer with
`max_prepared_statements=2`: 3 clients ran 600 transactions, each with
an insert and 6 statements over 40 distinct texts, and more distinct
statements between transactions. PgBouncer parsed 6,409 statements on
the server for 1,089 client parses, so it evicted and prepared again the
shared `commit` (`PGBOUNCER_15`) many times. All 600 `COMMIT`s
succeeded, 600/600 rows were present, and no 26000 error was logged. The
silent rollback needs a `DEALLOCATE` on the session, which PgBouncer
does not allow (it closes the client) and Paperclip does not send.

## Risks

- A transaction restart happens below the callback. Code in the callback
before its first statement runs once. The restarted transaction has a
new transaction id.
- A postgres.js behavior that this PR does not change: with driver-level
preparation (the flag `true`, or unset for tagged SQL), the tagged
`commit` of `sql.begin` is a prepared statement. If something
deallocates it on the session, the driver retries `COMMIT` in the
aborted transaction, which PostgreSQL turns into a rollback without an
error. Through PgBouncer this did not happen in 600 transactions with
the `commit` statement evicted repeatedly (see Verification). It needs a
manual `DEALLOCATE` on a direct session.
- A later statement in a transaction still fails once for each affected
text in each process after a migration that changes a column type. The
text then runs unprepared.
- A demoted text stays unprepared until the process restarts. The cap
admits texts in first-use order.
- A client that sends `DEALLOCATE ALL` through PgBouncer is disconnected
by PgBouncer. Paperclip does not send it.
- With the flag unset or `false`, the code path is the same as before.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, local PostgreSQL and PgBouncer in a throwaway
container, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:57:04 -06:00