Commit Graph
7750 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 b76111e45f fix(recovery): repair a continuation whose delivery is stuck; an edited follow-up is not shown as read
- One module (recovery-continuation-bounds.ts) owns the hold bound (5 min)
  and the pending bound (hold bound + 2 min); the invariant, wake admission
  and delivery read both from it.
- Delivery: a continuation held past the hold bound is no longer skipped
  because another recovery run of its agent is queued.
- Admission: hasPendingFreshContinuation counts a record only within the
  pending bound. Past it a repair wake becomes the task's run; delivery's
  newer-run check settles the record as delivered by it (one successor).
- Opening note: only the comment revision the cut-off run read counts as
  read; a follow-up edited afterwards is listed as changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:50:08 -06:00
DottaandClaude Opus 5.5 cfd42b0516 fix(recovery): a held continuation counts as pending only within the pacing's admission bound
Paced delivery admits a held continuation within 5 minutes even when the
agent's queue is full. One still pending 7 minutes after it was recorded will
not be delivered, so the assigned-work sweep no longer counts it as the task's
pending wake, and the invariant can repair the task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:38:22 -06:00
DottaandClaude Opus 5.5 5565476f52 refactor(recovery): read the cut-off run's follow-ups through taskHistoryReadsForRun (#21203)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:36:33 -06:00
DottaandClaude Opus 5.5 d2be6d624d fix(recovery): tell the fresh run which follow-ups the cut-off run had already read
The opening note now lists the person's messages the interrupted run had read
(task_history.read run-log events, with the read time) before what that run then
created, including its own final comment, so the agent answers a re-delivered
follow-up once and does not post DONE again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:36:33 -06:00
DottaandClaude Opus 5.5 8d83b88410 fix(recovery): tell the fresh run what the cut-off run already created
A fresh continuation's opening note listed only actions with unknown outcomes.
Subtasks, comments and document saves the cut-off run had completed were not
mentioned, and continuations created the subtasks and posted DONE again. The
note now names them as existing (from issues.origin_run_id, comments and
document revisions created by that run) and asks the agent not to create them
again; the record keeps the list as createdObjects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:36:33 -06:00
DottaandClaude Opus 5.5 68c9502905 fix(recovery): a wake claimed long ago with no run is not a pending path
A claimed wake becomes a run within seconds. The invariant counted one that
stayed claimed with run_id null (wake 49c5540e, claimed 22:20:16, no run at
22:35) as pending forever; after 5 minutes it is now a violation the sweep
repairs, in both the suspect listing and the per-task facts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:36:33 -06:00
DottaandClaude Opus 5.5 9dd252a920 fix(recovery): let the periodic sweep find tasks behind a stale deferred wake; tie the working bound to the fallback's
The suspect listing excluded every task with a deferred wake, so the 5-minute
bound only applied to direct checks. 'System recovery working' is now believed
for the fallback's progress bound (15 min) plus one re-check (17 min).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:36:32 -06:00
DottaandClaude Opus 5.5 b7eb506eac fix(recovery): bound the invariant's 'pending' clauses for stale recovery state
The assigned-work invariant counted a task as pending forever when its
projection said system recovery was working, or when a wake sat deferred
behind a run that had already ended. Both were seen in the release-swap wave:
an undelivered fresh continuation and follow-ups never promoted after restart
recovery ended their run were reported OK and never repaired. 'Working' now
counts only within 20 minutes of the last run's end, and a deferred wake with
no live run only within 5 minutes of its creation; past that the task is a
violation and gets the sweep's one continuation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:36:32 -06:00
DottaandClaude Opus 5.5 ade7e0643f fix(recovery): pace recovery continuations and start person-initiated work first (#21234)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Execution recovery continues tasks that a restart or a lost run left
unfinished
> - Recent recovery changes restored many held tasks at the same time,
and each restored task got a queued run on its agent
> - The agent's queue starts runs oldest first, so a person's Retry or
chat message waited behind every older recovery run
> - On staging, one agent had 131 queued runs, and person Retries waited
more than 50 minutes
> - This pull request paces how many recovery continuations enter an
agent's queue, and it starts person-initiated work before system
recovery work
> - The benefit is that a burst of recovery cannot starve people, and
every recovery run still runs

## Linked Issues or Issue Description

Refs #21222, #21139, #21135

**What happened?**

Staging deployed #21139 and #21135 at 22:53Z. Within one hour, the
company had 201 queued runs and 78 running runs. 122 queued runs had the
recovery wake reason `issue_recovery_action_restored`. Each one was on a
different task. They all came from the reconciled-execution delivery
pass (`source: execution.reconciled`), at a rate of 5 to 15 per minute
from 22:57Z to 23:51Z. One agent had 131 queued runs. The oldest queued
run had waited since 22:53Z. A person's Retry at 23:14Z was still queued
4 minutes later. Six person Retries on that agent, queued from 23:08Z to
23:14Z, were still queued at 00:04Z, more than 50 minutes behind
recovery runs. At 00:04Z, that agent's newest start was a recovery run
created at 22:57Z. In the last 400 runs, 25 succeeded.

**Expected behavior**

A person's Retry, chat message, or assignment starts before system
recovery work on the same agent. Recovery work still drains at a steady
rate. All recovery runs run eventually, and none is dropped. A task
whose continuation waits shows that the task continues automatically. It
does not show that the task is idle, and it does not ask for Retry.

**Steps to reproduce**

1. Give an agent a concurrency limit of 1.
2. Queue many recovery continuations for the agent
(`issue_recovery_action_restored`), for example after a server restart
with many held tasks.
3. Click Retry on another task of the same agent.
4. See that the Retry stays queued until all older recovery runs start.

**Paperclip version or commit**

Staging build 2026.1106.0+2187.git.57aeda190

**Deployment mode**

Hosted cloud (staging)

**Agent adapter(s) involved**

paperclip_runner (native and legacy runtimes). The problem is in the
shared run queue, so it applies to all adapters.

## What Changed

- New module
`server/src/services/heartbeat/queued-run-dispatch-order.ts` decides the
order in which an agent's queued runs start. Person-initiated runs start
first: `task_retry`, external chat messages, and any wake that a user
requested. Other work comes next, together with recovery runs for
interruptions that ended in the last 30 minutes. Old-backlog recovery
runs come last. Inside each class, the existing order applies:
readiness, then priority, then age. Runs whose dependencies are not
ready stay last.
- Recovery runs get a bounded share. When the agent started 3 other runs
after the oldest queued recovery run arrived and none of them was
recovery work, the next slot goes to a recovery run. The share applies
across the whole batch when one dispatch fills many slots. Thus people
cannot starve recovery.
- `startNextQueuedRunForAgent` in
`server/src/services/heartbeat/queue.ts` uses the new order. It changes
only the ranking. It does not change `claimQueuedRun`, the admission
preflight, or the promoter.
- `deliverReconciledExecutions` in
`server/src/services/execution-recovery-resolution.ts` paces delivery.
It leaves a continuation pending when the agent already has a queued
recovery run, or when the agent's queue is at or above its concurrency
limit. One pass delivers at most 5 continuations per company and at most
1 per agent. The pass reads pending continuations in pages and leaves
out agents and companies that it already handled, so continuations for
busy agents cannot hide continuations for other agents. A continuation
that has waited 5 minutes or more is admitted even when ordinary work
keeps the queue full, and it goes ahead of fresher continuations,
longest wait first. Otherwise the most recent interruption goes first.
The existing 15-second continuation schedule delivers the remaining
items. This schedule runs only while pending continuations exist, so an
idle server does not poll.
- The assigned-work sweep
(`server/src/services/recovery/assigned-work-sweep.ts`) counts a pending
continuation as a pending wake for its task. Thus the invariant does not
see a held-back task as stranded, and it does not enqueue a second wake.

## Verification

- `pnpm exec vitest run
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/recovery-wake-herd.test.ts` (in `server/`): 17 tests pass.
- Red/green: with the source changes reverted to master, all tests in
`recovery-wake-herd.test.ts` fail. Each review fix also has a test that
fails on the head before that fix. With this change, they pass.
- `recovery-wake-herd.test.ts` covers these cases with embedded Postgres
and the real heartbeat queue:
- Six recovery wakes are queued, then a person's request. The first
dispatch starts the person's run. All six recovery runs start later.
- Four pending continuations for one agent: one pass delivers one. The
others stay pending and are delivered on later passes. The continuation
schedule stays armed while items are pending and returns null after all
items are delivered.
- A continuation stays pending while the agent's queue is at its
concurrency limit.
- The per-company cap applies, and the most recent interruption is
delivered first.
- A continuation that has waited past the bound is admitted while
ordinary work keeps the queue full.
- New interruptions arrive for the same agent on every 15-second pass,
and the older continuation is still admitted within the bound.
- Continuations for four busy agents (32 rows, more than one page) do
not hide an idle agent's continuation.
- A held-back continuation counts as a `queued_wake` pending intent for
the assigned-work invariant, and the task shows `working` with
`canRetry: false`.
- `queued-run-dispatch-order.test.ts` simulates dispatch: a Retry starts
ahead of 40 recovery wakes, and all 40 still run. With a person Retry
arriving at every step, every window of 4 starts includes 1 recovery
run. A fresh interruption's continuation is not put behind old-backlog
recovery runs. When one dispatch fills 8 slots, the order is PPPRPPPR (P
is a person run, R is a recovery run).
- Related suites pass: fresh-continuation fallback, issue recovery
actions, completed-turn cleanup hold, issue monitor scheduler, workspace
recovery fallback, assigned-work invariant, execution control work,
execution recovery resolution, and the heartbeat queue.
- `node scripts/typecheck-layers.mjs` (in `server/`) passes.

## Risks

- Behavior change: when a person's run waits behind recovery runs, it
now starts first. Recovery runs start a little later, but they still get
at least 1 of every 4 starts while people send work.
- Delivery of continuations is slower during a burst: at most 1 per
agent and 5 per company every 15 seconds. This is intentional. A
continuation that waits shows as continuing automatically, and the
invariant does not repair it twice.
- Ranking adds up to three small indexed reads when an agent has more
than one queued run. The reads run only when the queue has recovery runs
or wake requests to classify.
- No schema or migration changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, and read-only staging
API requests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:30:56 -06:00
Nicky LeachandClaude Fable 5.2 5d3fa712aa feat(docker): build and check the Runner alone as the runner-artifacts target (#21204)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Today the application image build compiles the Runner. Other images
that need the Runner build it again, or pull the whole application image
to copy it out
> - Each of those builds can produce a slightly different Runner, and
each one costs time
> - The Runner should be built one time for each change to its inputs,
and every image should copy the same bytes
> - This pull request adds the build target `runner-artifacts`, a
contract for its layout, and a workflow that builds and checks it. It
publishes nothing
> - Nothing consumes the target in this pull request. The application
image build does not change
> - The benefit is one checked Runner build that later changes can
publish and copy by digest

## Linked Issues or Issue Description

Refs #21096 (the merge-queue image build) and #21167 (provider pack
layers). Those are earlier work on the same build. No issue exists for
this change.

**Problem or motivation**

The Runner is compiled inside the application image build. The daemon
comes from the `runner-build` stage. The provider pack is built on top
of the full server and UI build. Downstream sandbox and VM images need
the same Runner. They rebuild it from source, or they pull the
application image and copy paths out of it. There is no contract for
those paths and no single identity for a Runner build.

**Proposed solution**

Build the Runner alone, from its own inputs, as a plain file tree. Give
each build an identity: a digest of its inputs. Document the layout as a
contract, and check every built tree against it. Keep every check that
proves the Runner is good in this one build.

**Alternatives considered**

- Keep the current builds and only document the paths in the application
image. This does not remove the duplicate builds, and consumers still
need the full application image.
- Build the Runner in `docker.yml`. That workflow has no pull-request
trigger, so a change to the Runner build would have no check before it
merges.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4. Each part is on top of the one before it:

1. `runner/pack-inputs-digest`: an optional field in the provider pack
manifest
2. `runner/runner-image-build` (#21204): build and check the Runner
image, with no publication (this pull request)
3. `runner/runner-image-layers`: layers by change rate, and the slim
variant
4. `runner/runner-image-publish`: publish both variants from `master`

This pull request had one larger form before. Publication, the image
layers and the second variant are now parts 3 and 4.

## What Changed

- `Dockerfile`: eight new stages that end in the target
`runner-artifacts` (`FROM scratch`). They install with a filtered,
frozen lockfile from a context that holds only the Runner's inputs. The
base is a Node image on Debian bookworm, pinned by digest. No existing
stage changed. `cloud` is still the default target.
- The tree holds: the stripped daemon, the Runner package's complete
`dist` with the build stamp that its own build records, the provider
pack with its transport archives, the Grok provisioning inputs, the
transport contract source and a bundled transport module, the lockfile,
and `runner-image.json`.
- `scripts/runner-image-identity.mjs`: computes the image identity. It
hashes every file under the paths that the build copies from the build
context (path, executable bit, content; symbolic links by target), the
whole `Dockerfile`, and the two platforms. The build runs it on the
files that the build context supplies after `.dockerignore`.
- `scripts/runner-image-layout.mjs`: the layout contract. `write`
assembles the tree in the build. `check` verifies a built tree: every
documented path, the build stamp, every pack artifact and tree digest in
`provider-pack.json`, and every transport archive. It also extracts the
transport archives the way a sandbox does and compares the result with
the pack. No checked path may resolve outside the tree.
- Publication form. The build argument
`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1` is off by default. Only with it
does the pack manifest carry `payload.runnerInputsDigest` (part 1), and
`runner-image.json` records `publication: true`. Another image build
that runs the stage inline builds the same Runner, but its pack names no
image. `check` requires the field in a publication build and refuses it
in any other. `--expect-inputs-digest` passes only for a publication
build.
- `docker/runner-image/Dockerfile.smoke`: runs `paperclip-runnerd
--build-metadata` on each base image that a consumer uses. Each base is
pinned by digest.
- `.github/workflows/runner-image.yml`: two jobs, on pull requests that
touch an input and on manual dispatch. `identity` resolves the lockfile
and computes the digest in the build context. `build` builds each
architecture on a native runner, checks the layout, and runs the smoke
matrix. The workflow has `contents: read` only. It holds no credential,
reads no registry, and publishes nothing. It builds the publication
form, so the check can bind the tree to the digest that the identity job
resolved. The tree stays on the runner.
- `packages/paperclip-runner/scripts/provider-pack-runtime-tree.mjs`:
the tree digest function moves here from `build-provider-pack.mjs`
without a change in behavior, so the layout check can use it. A test
pins its value.
- `.github/workflows/pr-merge-review.yml`: lists the new workflow, as
`doc/PR-MERGE-REVIEW-WEBHOOK.md` requires.
- `doc/runner-image-contract.md` (127 lines): the contract for
consumers. `doc/DOCKER.md` links to it.

Changes from the earlier form of this pull request:

- The identity hashes the whole `Dockerfile`. The script no longer
parses stages, and it no longer refuses recipes.
`scripts/runner-image-identity.mjs` went from 397 lines to 166. A short
test keeps the three rules that make the hash enough: the Runner stages
start only from pinned images, only `runner-artifacts-context` reads the
build context, and it copies only the paths that the identity hashes.
- The contract document went from 384 lines to 127.
- The workflow tests keep only the security properties: job permissions,
no credential and no registry access, triggers, actions pinned by
commit, and no expression inside a shell script. They no longer pin step
names, order, or wording.

## Verification

Verified on `linux/arm64` only, on a local Docker host:

- `docker buildx build --target runner-artifacts --platform linux/arm64
--build-arg PAPERCLIP_RUNNER_SOURCE_REVISION=<sha> --build-arg
PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1 --output type=local,dest=<tree> .`
builds in 166 seconds without a cache. The image is one layer of
850,121,093 bytes. The tree is 1,755,345,443 bytes in 9,969 files.
- `node scripts/runner-image-layout.mjs check <tree>
--expect-inputs-digest <digest> --expect-source-revision <sha>
--expect-architecture arm64` passes on the exported tree.
- `docker buildx build --file docker/runner-image/Dockerfile.smoke
--build-context runner-artifacts=<tree> docker/runner-image` passes on
all five base images.
- The digest from `--target runner-artifacts-identity` equals the digest
in the built `runner-image.json`:
`d05f51f075cb8fec687a1d27f574ba7cb41cd8ebbe9927c5cbb2cfc13b147880`.
- The same target with `PAPERCLIP_BUILD_COMMIT=<sha>` and no publication
argument builds. Its pack has `runnerSourceRevision` and no
`runnerInputsDigest`. `check` passes on it, and `check
--expect-inputs-digest` fails with "this is not a publication build".
- An edit to `ui/package.json` and to `server/src` leaves the digest
unchanged. A comment added to an application stage of the `Dockerfile`
changes it.
- The two points above ran one commit state earlier. The only difference
to this commit is one type annotation in a comment of
`scripts/runner-image-layout.mjs`.
- A named build context replaces the `runner-artifacts` stage, and no
Runner stage runs. A build with the application build's view of the
context accepts the tree's `dist` and build stamp as current
(`assessRunnerBuild`).
- The daemon has the same bytes in the five cold builds of this stack
that were compared.
- Tests: `cd server && node node_modules/vitest/vitest.mjs run` on
`runner-image-identity.test.ts`, `runner-image-layout.test.ts` and
`runner-image-workflow.test.ts` (132 tests), with
`remote-provider-pack-verification.test.ts` and the existing Dockerfile
tests: 186 tests pass. `tests/runner-e2e` workflow tests: 39 pass. `node
--test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the
image contract scripts and `provider-pack-runtime-tree.test.mjs`: 33
pass. `check-docker-deps-stage.mjs`, `check-node-version-policy.mjs` and
`docker buildx build --check` report nothing.

Never run:

- The `linux/amd64` build. It is the leg that downloads the Cursor and
Copilot assets for the pack.
- The workflow. No job has run on GitHub.
- The transport archive round trip with GNU tar outside the build. The
build runs it with GNU tar. The local host ran it with bsdtar.
- `actionlint` and `hadolint`. They are not installed on the local host.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- Low risk to the application image. The change adds stages and files.
No existing target reaches the new stages.
- The first run of the workflow is the first test of the amd64 build.
This pull request triggers it.
- The `Dockerfile` is one file for all images. Any edit to it, also to
an application stage, changes the Runner image identity. That is the
price of not parsing it.
- The Dockerfile frontend (`# syntax=docker/dockerfile:1.20`) is a tag,
not a digest. A new frontend release under that tag does not change the
identity.
- The image does not run the drift checks of the Runner's `build`
script. They need files outside the Runner's inputs, and repository CI
runs them. A test fails when the `build` script gains a step.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 17:30:51 -08:00
DottaandClaude Opus 5.5 2d1c2db64a feat(mcp): host the MCP Skills extension for connected apps (#21219)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected apps through MCP. Paperclip already hosts MCP
Apps views and speaks stateless MCP 2026-07-28.
> - The MCP Skills extension (SEP-2640, Final) lets an app publish
skills (instructions and files) next to its tools. Hugging Face already
serves 25.
> - Paperclip did not read them, so agents missed each app's own
workflow guidance.
> - Skills are a prompt-injection surface. The spec makes manifest
binding, verification, origin tagging and approval revocation host
requirements.
> - This pull request makes Paperclip a Skills host: discover, review,
verify, and deliver app skills to the agents that use the app.
> - The benefit is that connected apps can teach agents their workflows,
under board control.

## Linked Issues or Issue Description

**Problem**
Connected MCP servers can publish skills with the
`io.modelcontextprotocol/skills` extension. Paperclip ignored them. A
board could not see, review or approve them, and agents never received
them.

**Proposed solution**
Implement the host side of SEP-2640. List skills during catalog refresh,
record them per connection and company, bind board approval to the full
file manifest, and load approved skills into agent runs with verified,
lazily fetched content.

**Why it matters**
Apps ship workflow guidance with their tools. Without the extension,
each team must copy that guidance into company skills by hand, and it
goes stale.

Refs #21119, Refs #21140 (the MCP Apps and stateless MCP client work
this builds on).

## What Changed

- **Client capability.** Every stateless request declares
`io.modelcontextprotocol/skills`. Legacy handshake sessions do not. The
capabilities that `server/discover` advertises are remembered beside the
protocol era.
- **Discovery.** Catalog refresh calls `skills/list` (paged) only when
the server declared the extension. A listing inside its `ttlMs` is
reused. A skills failure never fails the tool refresh. Nothing runs
while a run starts.
- **Schema.** New `tool_mcp_skills` (company, connection, skill URI;
frontmatter, sorted manifest, manifest digest, review state) and
`tool_mcp_skill_files` (verified bytes keyed by connection, URI and
digest). Both cascade with the connection.
- **Governance.** New skills wait for review. Approval binds to the
order-free digest of every `{uri, digest, size}`; a request for any
other digest gets 409. A changed manifest or frontmatter revokes
approval and logs `tool_mcp_skill.approval_revoked`. Dynamic, oversized
and invalid skills are declined with a reason. Approve and decline use
the company skill policy (`skills.install`, `external_package`,
`mcp-connection:<id>`).
- **Run time.** Approved skills of connections the agent can use load in
parallel with other skill preparation, inside a 3 s budget. `SKILL.md`
is read with the agent's grant and checked for size, SHA-256 and
frontmatter. A failure calls `skills/get` and takes the skill out of
use. Each skill becomes a per-run bundle on the connector skill path,
named in its app's namespace, tagged with its origin. Paperclip writes
the frontmatter: `allowed-tools` only if the board approved it.
- **Supporting files.** `GET
/api/agents/me/mcp-skills/:skillId/files?path=` serves manifest-listed
files to an agent in an active run, through the skill's own connection.
The sandbox bridge allows the route.
- **Board API and UI.** `GET/POST /api/companies/:companyId/mcp-skills…`
(list, read instructions, approve, decline). The Skills page shows
"Skills from connected apps" with app, state, size, requested tools and
plain-language notes. It renders nothing when no app publishes skills.
- **Docs.** `doc/connections/MCP-SKILLS.md`, linked from
`doc/connections/README.md`.
- **Shared frontmatter parser.** It now decodes YAML single-quoted
scalars. Before, 5 of Hugging Face's 25 skills kept their quotes and
would fail the frontmatter check; now all 25 match. The generated runner
copy is regenerated.

File size notes: `server/src/services/tool-gateway.ts` grows by about
200 lines. The run-time reads must use its grant resolution and
`callAssignedConnectionProtocol`, like MCP Apps view reads. Splitting
the gateway is a separate refactor. `CompanySkills.tsx` and
`CompanySkills.production.tsx` gain 2–4 lines each; the section lives in
its own component.

## Verification

- `server/src/__tests__/mcp-skills.test.ts` (15 tests) uses a fake
stateless skills server and embedded Postgres. It covers: discovery only
after the server declares the extension; the capability in `_meta`;
`ttlMs` reuse; lazy fetch (no reads at listing or approval; `SKILL.md`
at load; supporting files on read); a stale-digest approval refused; a
reorder that keeps approval; a changed or added file that revokes it;
dynamic skills declined; a same-size tampered `SKILL.md` rejected,
refreshed with `skills/get`, and taken out of use; mismatched
frontmatter rejected; reads outside the manifest refused; the same URI
and name from two servers kept apart; company scoping; a slow app
skipped within the budget and cached later; HTTP authorization (board
only, agent needs an active run, run header must match) and activity
logging.
- Red/green: with the digest check disabled, the tamper test fails; with
the manifest comparison disabled, the revocation test fails. Both pass
with the code.
- `server/src/__tests__/mcp-protocol-versions.test.ts`,
`mcp-apps.test.ts`, `connection-instructions.test.ts`,
`tool-gateway-discovery-http.test.ts`: 47 pass (two capability
assertions updated for the new extension).
- `packages/shared/src/frontmatter.test.ts` adds a single-quoted scalar
case; the runner contract test passes (23 tests).
- `ui/src/components/skills/ConnectedAppSkills.test.tsx` (3) and
`ui/src/pages/CompanySkills.test.tsx` (44) pass.
`packages/adapter-utils/src/sandbox-callback-bridge.test.ts` (89)
passes.
- `pnpm --filter @paperclipai/server typecheck` passes. UI `tsc` passes
except the known missing generated Storybook catalog files.
- `pnpm verify` returned exit 2 (doc change is a global CI input);
required CI supplies broad coverage.
- Live check against `https://huggingface.co/mcp` without credentials,
through Paperclip's own services and the real egress guard: 25 skills
discovered as pending; three approved; cold load 135 ms, cached load 13
ms; a supporting file read through the manifest; the bundle tagged with
its origin.
- Storybook: `Agents and reusable capabilities/Skills and Skill
Studio/Connected app skills` (needs review, all states, tool request,
name collision, reading instructions).

## Risks

- **Migration:** two new tables only. No changes to existing tables.
Safe with the previous release running.
- **Run start:** companies without an approved app skill pay one indexed
read. With approved skills the wait is capped at 3 s; late or failing
skills are left out with a `skills.mcp_unavailable` run warning and
never fail the run.
- **Agent harness enforcement:** Paperclip cannot gate each
code-execution tool call an agent makes while following an app's skill.
Approval is explicit and per skill, the approval note says agents may
run commands the skill describes, and `allowed-tools` is never granted
without a separate tick.
- **Shared-home adapters** receive app skills like company skills. A
revoked skill leaves on the next skill sync.
- Not in this PR: per-connection opt-in for dynamic skills; serving
Paperclip's own skills from its public MCP endpoint;
`resources/directory/read`; app skills on an agent's skills tab.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs, live HTTP checks).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:18:06 -06:00
DottaandClaude Fable 5.2 826d41e646 feat(ui): redesign Settings → Secrets around what people manage (#21225)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents need API keys and tokens. People manage them in Settings →
Secrets.
> - The page showed every credential in one table. Tokens that Paperclip
saved for connections, chat channels and model sign-ins sat beside the
few secrets a person added, many with record IDs as names.
> - Each row repeated an always-true status, a version and a key. New
secret opened on two stacked mode switches and two storage pickers.
> - A person could not see which agents use a secret without opening it,
and had to open it again after creation to give an agent access.
> - This pull request redesigns the page, the New secret dialog, the
details panel, the Vaults tab, the import dialog, the Proposals tab and
My secrets. It keeps the existing data and API calls.
> - The benefit is a list that shows what a person manages, a create
flow with two fields, and a secret that an agent can use as soon as it
is saved.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The Settings → Secrets page, its New secret dialog, the secret details
panel, the Vaults tab, the import-from-vault dialog, the Proposals tab
and the My secrets tab.

**Subsystem affected**

UI: `ui/src/pages/Secrets.tsx` and `ui/src/pages/secrets/*`.

**Current behavior**

- One table lists all secrets. On a busy organization most rows are
credentials that Paperclip saved itself, and many have record IDs as
names.
- Each row shows Active, a version, a chip and the key. The row does not
show which agents use the secret.
- A three-line help banner is always visible.
- New secret starts with two mode switches (Organization / Each user,
Managed value / External reference), then a provider picker and a vault
picker. The Create button is disabled without a reason.
- The details panel has Details, Usage and Access events tabs.
- The Vaults tab has one section per provider, with empty placeholder
boxes and a status picker in the form.

**Proposed behavior**

- The list shows the secrets that the team added. Each row shows who
provides the value (Organization, or Each user with a count) and which
agents use it.
- Credentials that Paperclip saved collapse into "Managed by Paperclip".
They are grouped by origin and have readable names.
- One Filter menu covers provided-by, storage and status. Active filters
show as chips. Archived secrets stay hidden until a person asks for
them.
- New secret has a name, one explicit Organization / Each user choice, a
masked value, an optional list of agents, and "Stored in" only when more
than one storage place exists. Create stays enabled and says what is
missing. A `NAME=value` pasted into Name fills both fields.
- The details panel is one scrolling view: agents, other places the
secret is set, details, and recent activity.
- The Vaults tab is one flat list with health and secret counts. The
connect dialog has two required fields. Paperclip checks a new vault
immediately after it is saved.

**Reason and benefit**

A person can find their own secrets, see if a secret is used, and create
a secret that an agent can use in one step.

**Breaking changes**

None. The API gains one optional query parameter (`order` on the board
proposals list). The default order is unchanged. See Risks for the UI
behaviors that changed.

## What Changed

- **List.** Rewrote the Secrets tab in `Secrets.tsx`. Added
`secret-list-parts.tsx` for the Provided by and Used by cells, the help
popover and the managed groups. The Folders / Flat toggle is removed.
Folders are always on and search covers all folders.
- **Managed grouping.** Added `secret-origin.ts`. It infers the origin
of a secret from the name and key conventions of the server modules that
create secrets (`tool_app.*`, `tool_oauth.*`, `slack_manager.*`,
`chat.*`, `ai_*`, `routine-*`, `agent-observability.*`). It is used for
presentation only.
- **Agent usage.** Added `agent-secret-access.ts`. It holds the grant
and revoke logic that was inside the details panel, and builds one usage
index from the agents list for all rows.
- **New secret.** Rewrote the dialog. Added `buildCreateVaultOptions`
and `parseEnvAssignment`. Agents selected in the dialog get an
environment variable binding after the secret is saved. A failed grant
shows an error and keeps the secret.
- **Details.** Replaced the tabs with sections. "Also set in" lists the
bindings that are not agents. Activity shows agent names and the last
five entries, with "Show all activity".
- **Vaults.** Flat list with a built-in Paperclip row, a Default badge
that follows `getSelectableProviderConfig`, Check again, Enable, and an
Add vault menu. The dialog has Name and AWS region, with naming and tags
in a disclosure. The Status picker is removed.
- **Import, Proposals, My secrets.** Restyled to the same design.
Proposals adds a Recently decided list. The board list endpoint `GET
/api/companies/{companyId}/secret-proposals` takes a new optional
`order=resolved` to sort by decision time.
- **Copy.** "Provider vault" is "vault". "Local encrypted" is
"Paperclip" or "Paperclip-encrypted". Toasts and block reasons use plain
words.
- **Stories.** `secrets.stories.tsx` has new stories for the managed
section, a folder, New secret, the details panel and Vaults. The
Storybook usage fixture now returns the `target` field that the API
returns.
- **Tests.** Updated the page, import, proposals and My secrets tests.
Added `secret-origin.test.ts`. Updated the runner E2E vault-reference
journey and the Storybook visual render-state copy for the new labels.
- **Docs.** `Secrets → Provider vaults` is now `Secrets → Vaults` in
three documents.

`Secrets.tsx` was 4,855 lines and is 4,951 lines. The redesign edits the
page in place to keep its queries, mutations and deep links. Splitting
the file is a separate change.

## Verification

- `cd ui && node ../scripts/run-tsc.mjs -b`: no errors.
- `cd ui && pnpm vitest run src/pages/secrets src/pages/Secrets.test.ts
src/pages/Secrets.render.test.tsx
src/components/AgentSecretAccessEditor.test.tsx
src/components/AgentConfigForm src/components/IssueThreadInteractionCard
storybook/map-tooling.test.ts`: 14 files, 467 tests passed.
- `cd server && pnpm vitest run
src/__tests__/secret-proposals-routes.test.ts -t "decision time"`:
passed.
- `pnpm vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog-domains/external-vault-reference
chat-catalog-domains/external-personal-secret-menu
chat-catalog-domains/external-personal-secret-entry-dialog
chat-catalog-domains/external-personal-secret.test`: 32 passed, 2
skipped.
- `node scripts/check-token-gates.mjs`, `node scripts/check-maps.mjs`,
`node scripts/check-module-boundaries.mjs`, `node
scripts/storybook-catalog.mjs`: clean.
- `pnpm verify`: exit 2. It asks for broader coverage because a runner
E2E source file changed. Required CI supplies that coverage.
- All 22 Secrets stories render and their play functions pass in
headless Chromium against a local Storybook.
- End to end, on an isolated local instance (`paperclipai worktree init
--empty`, one company, two agents), with a headless Chromium script. 16
of 16 steps passed, with no page errors:
  - Empty state. Create explains missing fields.
- Create an organization secret and give an agent access. The agent's
saved config has the `secret_ref` binding.
- Paste `NAME=value`. Create an each-user secret. Add a value in My
secrets. The list shows "1 of 1 added".
  - Create a folder, create a secret in it, search across folders.
- In details: add an agent, replace the value, remove the agent. The
agent's saved config changes each time.
  - Disable, archive, and the Archived filter.
  - Managed secrets leave the team list and show under their groups.
- Add a Paperclip-encrypted vault and an AWS vault. New secret then
shows "Stored in".
- The import dialog explains a denied vault. The Proposals tab shows its
empty state. The list at 390 px width.
- Not run: the paid runner E2E journeys. I updated
`external-vault-reference.ts` for the new labels and could not execute
it. A live AWS import, and a proposal from a real agent run, were not
exercised.

## Risks

- **Origin grouping is a heuristic.** It reads naming conventions that
live in server modules. If a module changes its naming, its secrets show
in the team list again. Nothing is hidden or blocked by it. A recorded
origin on each secret is the durable fix.
- **Removed UI.** The Folders / Flat toggle, the "View references"
dialog, the vault Status picker, and creating draft GCP or HashiCorp
vault entries from the Add vault menu. Existing draft vaults still show
and can be edited. A disabled vault now has an Enable action.
- **Default status filter.** The list now shows active and disabled
secrets. Before, it showed active secrets only.
- **Built-in storage in "Stored in".** A provider's built-in storage is
not offered once that provider has a usable vault. The form already
replaced that choice with the vault before this change.
- **Agent access at creation.** Access is a second request for each
agent after the secret is saved. If one fails, the secret exists without
that access and an error says so.
- **Automation contracts.** Labels that agents and the runner E2E
journeys depend on are kept: the key in each My secrets row, "Set
value", "Set my value", "Update my value", "Save value", and the import
dialog test IDs. The import entry point moved into the "More secret
actions" menu and the journey is updated for it.
- **Value masking** uses `-webkit-text-security`. A browser without it
shows the value as typed.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) through Claude Code, with tool
use, code execution and browser automation. Three delegated agents on
the same model restyled the import dialog, the Proposals tab and My
secrets.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 19:17:41 -06:00
DottaandClaude Fable 5.2 2de5d97958 fix(startup): report ready without waiting for backlog first passes (#21247)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A hosted instance upgrades by starting a new server beside the old
server. The deployer sends traffic to the new server when `GET
/api/ready` returns 200.
> - `/api/ready` returns 200 only after the boot sequence in
`server/src/index.ts` sets the phase to "ready".
> - The boot sequence awaited the first pass of many delivery queues.
Each pass grows with saved backlog or waits on outside services.
> - On a busy instance, these passes took more than the deploy's
health-check window. The deployer killed two new servers in a row before
they became ready, including a rollback to an image that had started in
86 s one hour before.
> - This pull request moves those first passes off the readiness path.
Readiness now waits only on named, bounded preconditions, and each step
reports its duration.
> - The benefit: an upgrade or rollback becomes ready in boot time, not
in backlog time. A slow start also names its slow step in one log line.

## Linked Issues or Issue Description

**What happened?**
On the hosted staging instance, two replacement servers did not report
`/api/ready` and were killed after about 9 minutes. The first was a new
image and the second was a rollback to the old image. At that time the
instance had about 200 queued runs, about 80 running runs, a slow
database, and many continuation runs. The failed server logged "managed
environments ensured from managed config" (`index.ts` line 1284 on
master). It never logged "worktree run-execution cutoff state" (line
1726), which the server always logs when the heartbeat service exists.
Between those two lines, boot awaited only these steps: `tool_reviews`
(four tool-review queue first passes), `gateway_sessions`,
`delivery_workers` (budget enforcement, cost accounting, fast responses,
chat completions, connection continuations, tool action receipts and
question responses), the GitHub connection event and continuity workers,
and `terminalWorkspaceWorker.ready`. Each step is a delivery-coordinator
`register(...).ready`, and that promise resolves only after the queue's
whole first pass. No queue except `mcp-client-mirror` (also failing on
the healthy server) logged an error, so the cause was slow passes and
not failures. The log had no per-step markers, so it does not show which
of these awaits was still running at the kill.

**Expected behavior**
Readiness means "this server can serve requests and take over". It must
not wait on work that grows with backlog or that depends on an outside
service. Queues own that work and retry it on their own schedule after
readiness.

**Steps to reproduce**
1. Start the server while the first pass of any of the queues above does
not finish. The new test in `server-startup-feedback-export.test.ts`
does this for every queue.
2. On master, `startServer()` does not resolve and the phase stays
"recovering".

## What Changed

- `server/src/index.ts`: These first passes now start in the background
as tracked scheduler work and do not hold readiness: tool reviews,
gateway sessions, delivery workers, GitHub connection workers, the
terminal workspace sweep, environment lease cleanup (scheduler-disabled
path) and scheduled database backups. Shutdown and idle sleep still wait
for them through `trackHeartbeatSchedulerWork`, and each queue keeps its
retry schedule.
- `server/src/index.ts`: The restart backup after an idle wake (a
`pg_dump` whose time grows with the database) holds idle eligibility
(`markIdleStartupComplete`), as before. It no longer holds readiness.
- `server/src/index.ts`: Every remaining await on the boot path is a
named `startupTimeline.precondition(...)`: `listen`,
`runtime_control_operations`, `execution_policy`,
`managed_environments`, `worktree_run_execution`,
`scheduling_suppression`, `native_recovery`, `external_adapters` and
`cloud_configuration`. These preconditions do not change.
- `server/src/startup-timeline.ts`: Adds `precondition(step, work)` and
`background(step, work)`. A precondition that is still running after 30
s is named in the warning "startup step is still holding readiness",
which repeats every 30 s. A background step logs a warning once at 30 s
and logs its duration when it finishes late. A background step that
fails is logged with its step name. The line "startup timeline at
readiness" now includes `readinessStepsMs`, `backgroundStepsMs` and
`backgroundStepsOutstanding`. All timers are single unref'd timeouts and
the module adds no queries.
- `server/src/__tests__/startup-readiness-boundary.test.ts`: This guard
test parses `index.ts` with `parseAst` from `vite`. The server already
depends on `vite`, so this PR adds no dependency and no lockfile change.
The test fails when an await on the boot path (between the phases
"recovering" and "ready") is not a named precondition, or when the
precondition names do not match a reviewed list that gives the reason
for each name.
- `server/src/services/terminal-workspace-work.ts`: Updates a comment.
The first sweep no longer holds readiness, and it still skips
compaction.

## Verification

- Red/green regression test: `server-startup-feedback-export.test.ts`,
"reports ready while backlog queues' first passes never finish". This
test makes every first pass above never finish. With master `index.ts`
it fails with `expected 'still starting' to be 'ready'` for both
scheduler settings. With this change it passes.
- Guard test: with master `index.ts`,
`startup-readiness-boundary.test.ts` fails and lists 18 unnamed awaits,
including `timeStartupRecovery("delivery_workers")` and
`terminalWorkspaceWorker.ready`. With this change it passes.
- `cd server && npx vitest run
src/__tests__/server-startup-feedback-export.test.ts
src/startup-timeline.test.ts
src/__tests__/startup-readiness-boundary.test.ts
src/startup-recovery-timing.test.ts
src/__tests__/terminal-workspace-work.test.ts`: all tests pass.
- `cd server && pnpm typecheck`: passes.
- `pnpm verify` exited with code 2 on the first revision (a lockfile
change that this PR no longer makes). Required CI supplies the broad
coverage.
- After deploy, run `grep "startup timeline at readiness"` on a server
start. The line shows each precondition's duration and the background
passes that are still running.

## Risks

- The listed first passes now run at the same time as native run
recovery and the requests that the server serves, not before them.
Requests were already served during recovery. Commit-woken passes of the
same queues, and the fresh queued-run dispatch, already ran at the same
time as native recovery. The coordinator still serializes the passes of
each queue.
- `/api/ready` can now return 200 while a queue still has backlog. That
is the intended meaning, and it matches the queued-run and recovery
passes that already run after readiness (#21014).
- Native run recovery still holds readiness, which is correct because
adoption must happen before takeover. If that recovery is slow, the
server now logs its step name every 30 s.
- Low risk otherwise: no schema or API change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, model ID `claude-opus-5-5` (commit
trailer: Claude Fable 5.2). Tool use and code execution were enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 19:16:23 -06:00
DottaandClaude Opus 5.5 b0233b5b33 fix(native): count a parent run's read of a finished child as delivery of its result (#21227)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task gets a wake when a child task completes. If the parent
is running, that wake waits behind the run.
> - At finalization, a parent's done claim stays open
(`native_child_completion_pending`) while such a wake is still owed, so
the parent never finishes without the child result.
> - The server had no record of what a running parent actually read. A
parent that read its finished children mid-turn and then finished the
task still kept the wake owed.
> - The task stayed `in_progress`, the wake started a second parent
turn, and the agent wrote a second completion summary.
> - This pull request records a run's read of a finished child and
counts that read as delivery of the child result.
> - The benefit is one completion turn when the parent already read its
children, while unread child results still get a fresh turn.

## Linked Issues or Issue Description

Refs #21189 (child results saved before the run started). Refs #21175
(the same pattern for comments: `task_history.read`). Refs #15372 (the
pending child-completion guard).

**What happened?**

Restart-chaos wave `w-20261112T223803-s7101`, on a master build that
includes #21189, task t006 (OpenCode, native runner, staging-mix parent
with two subtasks):

- Parent run `bfe9941c` started at 22:55:56.
- Its children CHA-11 and CHA-12 finished at 23:07:53 and 23:11:38.
- The child-completion intent and its deferred dispatch row were saved
at 23:11:38, after the run started. So the #21189 rule counted them as
new input.
- At 23:26:41 the run's reasoning says "Both subtask ledgers exist …
Both subtasks are done". The run then saved the CHILD hashes and posted
DONE at 23:33:39.
- At finalization, the done claim was committed as
`native_child_completion_pending`. The deferred wake then started run
`40928116`, which wrote a second completion summary.

Root cause: `hasPendingNativeChildCompletion()`
(`server/src/services/native-runtime/native-child-completion-delivery.ts`)
proved delivery only by wake binding and timestamps. No durable record
showed that the finishing run had read the completed child.

**Expected behavior**

A finishing run's recorded read of a completed child counts as that
child result being delivered. A read while the child was open, a read by
another run, or a failed event write does not count. An unread child
result still keeps the task open and gets one fresh parent turn.

**Steps to reproduce**

1. Start a native parent run.
2. Complete a child while the run is active. The child-completion wake
waits behind the run.
3. Let the run read the child (`get_task_context`, or `GET
/api/issues/:childId`), then claim `done`.
4. Finalize and release the run. On `master`, the decision is
`native_child_completion_pending` and the wake becomes a second run.

The same wave's t001 grade ("duplicate completion") is not this bug.
That run only said it *will* post the DONE marker later, and the grader
counts any mention. Its `subtasks_once` failure comes from the
restart-recovery continuation. This PR does not change either one.

## What Changed

- `native-child-completion-delivery.ts`:
- New `recordChildTaskRead()` writes a `child_task.read` run-log event.
The payload is `{ parentIssueId, children: { id: { status, updatedAt } }
}`, and it lists only children in `done`.
- `hasPendingNativeChildCompletion()` no longer counts a
child-completion wake as pending when the finishing run recorded a read
of that wake's completed child in `done` at or after the child's
completion decision (`status_decisions.created_at`, or the wake's
`createdAt` when the decision row is missing).
  - The #21189 start-time rule is unchanged.
- `paperclip-runner-tool-authority.ts`: `get_task_context` records the
finished child tasks it returns. A failed write is logged and leaves the
result owed.
- `routes/issues.ts`: `GET /issues/:id`, `GET /issues/:id/documents` and
`GET /issues/:id/documents/:key` record the read when the request has
all of these:
  - an agent actor with a run token
  - a child issue in `done`

This runs after the existing access and company checks. So a board user,
another company's agent, or a read of an open task writes nothing. A
qualifying read adds one statement: the existing
`appendHeartbeatRunEvent` fast path, a single `update … returning` /
`insert` CTE bound to the run's id, company and agent. Every other read
adds no query.
- `doc/run-log-events.md`: documents `child_task.read` and how delivery
uses it.
- Tests: a finalization-to-release regression in
`heartbeat-process-recovery.test.ts` and a new
`native-child-task-read-routes.test.ts`.

## Verification

- RED on `master`: in `heartbeat-process-recovery.test.ts` "counts a
child result as read by the finishing parent run (saved after the run
started, read: after the child finished)", the run reads the child
through the real `get_task_context`, then finalization and the
wake-queue release run.
- On master: `native_child_completion_pending`, task `in_progress`, and
the wake is promoted into a second run, the same as t006.
- With this change: `done`, the wake is cancelled, and there is one run.
- The other cases in the same test behave the same before and after this
change:
  - saved before the run started: done
  - not read: pending, fresh turn with the child results
  - read while the child was open: pending
- `native-child-task-read-routes.test.ts` (5 tests):
- An agent read of a done child through each of the three routes records
the read and clears the pending result.
  - An open child records nothing.
  - A board read records nothing.
  - Another company's agent gets no 200 and records nothing.
- Neighbouring suites after merging `origin/master`:
`heartbeat-process-recovery`, `muse-control`,
`heartbeat-held-comment-wake`, `native-status-arbiter-corpus`,
`heartbeat-dependency-scheduling`, `paperclip-runner-tool-authority`,
`agent-task-hierarchy-depth-routes`, and the two new test files: 8
files, 982/982 tests pass.
- `pnpm --filter @paperclipai/server typecheck` passes.

**Not verified**

- No chaos wave has run on this change yet.
- In t006, the parent's tool calls fell in the 4,220 run events that the
evidence omitted. Its reasoning text proves the read, but the exact tool
(`get_task_context`, `get_task`, `read_task_document` or `call_api`) is
not visible.
- A read through `search_tasks`, or through a shell command that calls
the API without the run token, is not recorded. Such a result stays owed
and gets one fresh turn, which is the old behavior.

## Risks

- A recorded read of a done child now lets the parent finish without
another turn. If the agent read the child but ignored it, no second turn
reminds it. A read while the child was still open never counts.
- Each qualifying agent read of a done child appends one run-log row.
Board reads and reads of open tasks write nothing.
- No migration. Wakes saved by older builds work unchanged. Without a
recorded read, the old behavior applies.
- This is a run-log change only (instance database). It adds no
Telemetry or OpenTelemetry export.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:07:31 -06:00
Devin FoleyandPaperclip ad320aeb71 fix(native): wait for a dated provider limit reset and keep it out of Sentry (#19775)
Keep native quota cooldowns tied to valid dated reset messages. Parse explicit years and strict month names while preserving already-expired dates. Keep typed native quota and rate-limit failures on their tasks in unexpected Sentry reporting mode. Preserve setup, process, malformed and all-mode reporting.

Verified current-head CI, Greptile 5/5 at Medium impact, local parser/report red-green tests and PostgreSQL integration coverage.

Co-Authored-By: Devin Foley <devin@devinfoley.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-12 17:06:22 -08:00
Nicky LeachandClaude Fable 5.2 2dcfdabc79 feat(runner): let a provider pack record the Runner image it was built in (#21228)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - A provider pack has a manifest, `provider-pack.json`. It records the
commit the pack was built from as `payload.runnerSourceRevision`
> - The next pull requests build the Runner alone as an image that has
an identity of its own: a digest of the Runner's inputs
> - A pack from that image must be able to say which image it came from.
The commit does not say that, because many commits have the same inputs
> - This pull request adds one optional field to the pack manifest for
that digest
> - The benefit is that a reader of a pack can find the exact image that
built it. Packs that are built today do not change

## Linked Issues or Issue Description

Refs #21204 (the Runner image, part 2 of this stack). No issue exists
for this change.

**Problem or motivation**

The provider pack manifest records a source revision. It has no place
for the identity of a Runner image. The Runner image in #21204 is named
by a digest of its inputs, and one image serves every commit that has
those inputs. Without a field in the manifest, a pack cannot name the
image that built it.

**Proposed solution**

Add the optional field `payload.runnerInputsDigest`. The pack builder
writes it only when the environment variable
`PAPERCLIP_RUNNER_INPUTS_DIGEST` is set. The value must be 64 lowercase
hexadecimal characters. The pack builder fails for another value.
Nothing sets the variable in this pull request.

**Alternatives considered**

- Put the digest in a file beside the pack. Then the pack digest does
not cover it, and a copy of the pack loses it.
- Reuse `runnerSourceRevision`. That field is a commit, and code
compares it with other commits.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 1 of 4. Each part is on top of the one before it:

1. `runner/pack-inputs-digest`: an optional field in the provider pack
manifest (this pull request)
2. `runner/runner-image-build` (#21204): build and check the Runner
image, with no publication
3. `runner/runner-image-layers`: layers by change rate, and the slim
variant
4. `runner/runner-image-publish`: publish both variants from `master`

## What Changed

- `packages/paperclip-runner/scripts/build-provider-pack.mjs`: reads
`PAPERCLIP_RUNNER_INPUTS_DIGEST`. When it is set, the script checks its
form and writes `payload.runnerInputsDigest` after
`runnerSourceRevision`. When it is not set, the manifest is the same as
before, byte for byte.
-
`server/src/services/native-runtime/remote-provider-pack-verification.ts`:
the manifest type has the optional field `runnerInputsDigest`. No check
reads it.
-
`server/src/services/native-runtime/remote-provider-pack-verification.test.ts`:
one test. A sealed manifest with the field passes verification. A
manifest whose field was changed after it was sealed fails, because the
pack digest covers the field.

The change is 20 added lines in 3 files.

## Verification

- `cd server && node node_modules/vitest/vitest.mjs run
src/services/native-runtime/remote-provider-pack-verification.test.ts`:
19 tests pass.
- Part 2 builds a pack with the variable set and a pack without it, on
`linux/arm64`. The first has the field and the second does not.

Never run:

- The full test suite and the full typecheck. CI must supply them.

## Risks

- Low risk. The field is optional and additive, and nothing sets the
variable yet.
- The pack digest covers the payload. A pack that has the field
therefore has another digest than the same pack without it. No pack has
the field until part 2 builds one.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
canary/v2026.1113.0-canary.0
2026-11-12 16:58:14 -08:00
DottaandClaude Opus 5.5 19f8ed1ea2 fix(recovery): continue only recent interruptions from the held backlog, newest first, paced (#21222)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a server restarts or a deploy swaps servers, agent runs are
interrupted. The fresh-continuation fallback (#21135) continues each
interrupted task in a new run from its saved work.
> - The fallback also has a backlog pass. It continues tasks that older
releases left held by a recovery record.
> - The backlog pass had no age limit and no pacing. It took the oldest
records first, 25 per pass.
> - On the first hosted release with the fallback, it continued 176
tasks in 30 minutes. 74 of them had a source run that ended more than 7
days ago. The oldest ended four weeks ago. More than 100 runs were
queued, and one agent had about 130 runs.
> - The user-facing goal of the fallback is "an interruption is
invisible". It is not "run last week's failures again".
> - This pull request limits the backlog pass to recent interruptions,
newest first, at a paced rate.
> - The benefit is that a release no longer starts a herd of old,
unwanted runs, and recent interrupted work still continues by itself.

## Linked Issues or Issue Description

Refs #21135

**What happened?**
A hosted instance took its first release with the fresh-continuation
fallback. Within 30 minutes, the backlog pass continued 176 held tasks.
By source run age: 26 under 1 day, 76 from 1 to 7 days, 74 over 7 days.
The source end codes included `adapter_failed` (27),
`runner_remote_provider_artifact_incompatible` (14),
`native_provider_terminal_failed` (14) and `setup_failed` (8). These are
configuration or provider failures, not interruptions. The pass kept
firing at 12 to 36 continuations per 5 minutes, and the queued backlog
grew past 100 runs.

**Expected behavior**
The backlog pass continues a task only when the task was recently
interrupted by a restart, a deploy, a lost lease, a lost process or a
refused recovery. Old records and configuration or provider failures
stay as they are, for a person's Retry. The pass starts new runs at a
controlled rate, newest first.

**Steps to reproduce**
1. Have held recovery records that older releases saved, with source
runs that ended days or weeks ago, some with `adapter_failed`.
2. Deploy a release that contains #21135.
3. See that the disposition worker continues all of them, oldest first,
25 per pass, and queues a run for each.

## What Changed

- `continueHeldExecutions` and `heldExecutionsAwaitingFreshContinuation`
(server/src/services/fresh-continuation-fallback.ts):
- Continue a held record only if its source run ended within
`BACKLOG_CONTINUATION_MAX_AGE_MS` (24 hours) AND its error code is in
`BACKLOG_CONTINUATION_SOURCE_CODES` (restart, deploy, lease,
process-loss and recovery-refusal codes).
- Do not touch any other held record. Do not settle it and do not modify
it. A log line counts these records, at most once every 10 minutes.
  - Order by the source run's end time, newest first.
- Continue at most 5 records per pass and 10 per rolling minute per
server.
- The rule I chose, and why: 24 hours and interruption codes only. A
person who is waiting for a task notices within a day. A run that failed
for configuration or provider reasons fails again the same way, so a
person must change something and press Retry. The 7-day rule of the
assigned-work sweep is too wide for this pass: 76 of the 176 continued
tasks were from 1 to 7 days old.
- The live path does not change. Fresh interruptions come through
restart recovery and the disposition pass (`recovery_refused`), and
through continuation wakes. They are not paced and have no age rule.
- Upgrade safety: records that a release already continued carry
`freshContinuation` and are excluded as before. Queued continuation runs
are not changed.
- The 10-per-minute limit is per server. During a deploy overlap, two
servers can each continue 10 per minute. This is acceptable for a short
overlap, and it keeps this change free of shared state.

## Verification

- `cd server && npx vitest run
src/services/fresh-continuation-fallback.test.ts`: 53 passed.
- New test "continues only recent interruptions from the held backlog,
newest first, and leaves the rest untouched (staging herd)". It uses
mixed-age and mixed-cause records: 1 h and 6 h restart codes, 3-day and
28-day restart codes, and 2 h
`runner_remote_provider_artifact_incompatible` and `adapter_failed`.
Only the two recent restart-caused records continue, the 1 h record
first. The four other records keep the same status, evidence and
`updated_at` across repeated passes.
- New test "paces the held backlog to ten continuations a minute per
server": 12 eligible records give 10 continuations in the first minute
and the remaining 2 in the next minute.
- RED on origin/master: both new tests fail (`expected [] to have a
length of 1`; `expected 12 to be 10`).
- The existing tests for the live path (for example "continues tasks
that older releases left blocked by these codes", with a source run that
ended one minute ago) still pass.
- `pnpm --filter @paperclipai/server run typecheck`: clean.

## Risks

- A held record from 1 to 7 days ago, caused by an interruption, no
longer continues by itself. It stays as it was before #21135: the task
shows its recovery state and a person can press Retry.
- A record whose cause is not in the list (a future interruption code)
is not continued by the backlog pass. The live path still continues
fresh interruptions for any code that is not a person's.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 18:36:22 -06:00
DottaandClaude Opus 5.5 b4ca19f3a8 test(e2e): keep the stalled-font task test's font requests on the page route (#21171)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every PR runs the Playwright e2e suite in 8 shards
> - `task-loading-frame.spec.ts` "a stalled web font does not hide an
otherwise prepared conversation indefinitely" failed 17 times in 48
hours, on 17 different PR branches, most with no UI changes (for example
#21129)
> - The test stalls `**/*.woff2` with `page.route`. The app's service
worker claims the page and fetches the fonts itself. `page.route` does
not see service-worker requests
> - When the worker claimed the page before the font request, the font
loaded. The loading frame then showed for only a few frames, and
`toBeVisible()` missed it
> - This pull request blocks service workers for this one test, so the
font stall the test depends on always happens
> - The benefit is that e2e shard 1/8 stops failing at random on
unrelated PRs

## Linked Issues or Issue Description

**What happened?**
`tests/e2e/task-loading-frame.spec.ts:131` failed with
`expect(locator).toBeVisible() failed`, `Locator:
getByTestId('task-chat-loading-frame')`, `element(s) not found`, on
unrelated PRs. The failure snapshot shows a fully loaded conversation.

**Expected behavior**
The test stalls the web fonts, sees the loading frame, and then sees the
conversation within 5 seconds.

**Steps to reproduce**
`PAPERCLIP_E2E_PORT=3291 npx playwright test --config
tests/e2e/playwright.config.ts tests/e2e/task-loading-frame.spec.ts -g
"stalled web font" --repeat-each=40` on master: 6 of 40 runs fail with
the CI error.

**Paperclip version or commit**
master at 4e932fd178 and later.

**Deployment mode**
CI e2e (local_trusted, built UI).

**Agent adapter(s) involved**
N/A, CI test.

## What Changed

- `tests/e2e/task-loading-frame.spec.ts`: the stalled-font test runs in
a `test.describe` with `test.use({ serviceWorkers: "block" })`. The test
body is unchanged.

## Verification

- Before (master): `--repeat-each=40`: 34 passed, 6 failed. All 6
failures are the CI signature (`task-chat-loading-frame` not found).
- After: `--repeat-each=50`: 50 passed, 0 failed.
- `ui/public/sw.js` claims clients on activate and answers every
same-origin non-API GET with `fetch()`, including `/fonts/*.woff2`.
Playwright documents that `page.route` does not intercept requests that
a service worker handles.

## Risks

- Low risk. Test-only change. Only this test runs without a service
worker. The product path for a stalled font is still tested.
Service-worker behavior has its own unit tests.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, GitHub CLI, Playwright).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 18:19:39 -06:00
DottaandClaude Opus 5.5 8b4b6aec87 test(chat): keep the Retry route's background nudge out of the board-message retry test (#21174)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-channel integration suite runs in 3 "General tests (chat)"
CI shards on every PR
> - "retries the saved board-message turn in a connected task after
environment qualification failed before provider startup" failed on 4
unrelated PR branches in 48 hours
> - Every failure is a foreign-key error when the test deletes its saved
Retry: `heartbeat_runs_wakeup_request_id_agent_wakeup_requests_id_fk`
> - `POST /issues/:id/retry` answers 202 and then calls
`heartbeat.resumeTaskRetry` without awaiting it. When that nudge created
a run before the test's cleanup, the delete failed
> - The test drives dispatch itself (`resumeExecutionWaitComments`), so
the background nudge only adds a race
> - This pull request gives the test's route a heartbeat service whose
nudge does nothing
> - The benefit is that the chat shard stops failing at random on
unrelated PRs

## Linked Issues or Issue Description

**What happened?**
`server/src/__tests__/chat-channels.integration.test.ts` "retries the
saved board-message turn in a connected task after environment
qualification failed before provider startup" failed with `update or
delete on table "agent_wakeup_requests" violates foreign key constraint
"heartbeat_runs_wakeup_request_id_agent_wakeup_requests_id_fk"` on PRs
that did not touch chat or Retry code (for example
codex/github-pat-without-cloud, fix/21017-workspace-source-lock-wait,
fix/codex-internal-interrupt-recovery).

**Expected behavior**
The test passes every time. Its Retry dispatch happens only where the
test starts it.

**Steps to reproduce**
Add a 300 ms wait at the start of the `denySource` cleanup (before the
`agent_wakeup_requests` delete) and run the test. On master it fails 3
of 3 times with the CI error, because the route's background nudge
creates a run first.

**Paperclip version or commit**
master at 4e932fd178 and later.

**Deployment mode**
CI server tests (embedded PostgreSQL).

**Agent adapter(s) involved**
N/A, CI test.

## What Changed

- `server/src/__tests__/chat-channels.integration.test.ts`: while the
test builds `issueRoutes`, `heartbeatService` returns the real service
with a no-op `resumeTaskRetry`. The spy is restored right after the
routes are built. The same pattern is used in
`heartbeat-native-restart-suspension.test.ts`.
- No product code changes. The nudge is only a latency optimization.
Maintenance redelivers saved Retries (see the comment in
`routes/issues.ts`).

## Verification

- Forced race (300 ms wait before cleanup): master 0 of 3 pass with the
CI error; this branch 3 of 3 pass.
- Loop on this branch, no injection, 2 parallel loops on a loaded
machine: 25 of 25 pass.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts -t
"retries the saved board-message turn in a connected task after
environment qualification failed"`.

## Risks

- Low risk. Test-only change. The test still exercises the real Retry
route and the real dispatch path that it drives explicitly.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, GitHub CLI, Vitest).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 18:19:32 -06:00
DottaandClaude Opus 5.5 334b2ae066 fix(recovery): continue a task blocked behind its workspace's ended writer once the hold releases (#21221)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote task workspaces have one writer run at a time, and an ended
writer keeps the workspace until its files are saved or set aside
> - When a task's next runs fail setup on that hold ("The workspace
still has an unfinished writer or save operation."), stranded-work
recovery blocks the task with a board escalation
> - The retained-file recovery from #21136 later released such a hold on
staging (it continued from the last save), but nothing settled the board
escalation, so the task stayed Blocked until a person pressed Retry
> - Recovery must work without a person: the block belonged to the hold,
and the hold is gone
> - This pull request adds a pass to the active recovery-action sweep
that settles that escalation once the workspace's writer hold is free
> - The benefit is that such tasks continue by themselves with one plain
notice and one run, with no Retry

## Linked Issues or Issue Description

Refs #21136
Refs #21143

**What happened?**
On staging, an ended run held its task's own remote workspace after its
superseded workspace export failed. The task's queued runs failed setup
with the unfinished-writer error, and stranded-work recovery blocked the
task with a board-owned escalation. After #21136 was deployed, its
retained-file recovery recorded the hold 77 seconds after server start
and, 19 minutes later, continued the task from its last save and
released the workspace. The task still stayed Blocked behind the board
escalation, and no run was queued. A board user had to press Retry.

**What did you expect to happen?**
Once the hold that caused the setup failures is released, the task
returns to `todo` and its assignee continues it, with no Retry.

**Steps to reproduce**
1. Give a task a remote workspace whose writer is an ended run with
unsaved files.
2. Let the task's next run fail setup with the unfinished-writer error,
and let stranded-work recovery block the task.
3. Release the hold (save the files, or continue from the last save).
4. Run the active recovery-action sweep. Before this change the task
stays Blocked with an active board escalation.

**Why the merged fix did not unblock the task**
The retained-file fallback only releases a Blocked status that its own
hold set (`holdOwnsBlockedStatus`). Here stranded-work recovery set the
block with a different, board-owned escalation, and a late status
decision 1.4 seconds later kept that block. So the fallback correctly
did not take the status, and no code settled the board escalation after
the hold was gone.

## What Changed

- Add `continueTasksAfterWorkspaceWriterReleased`
(`server/src/services/recovery/released-writer-escalation.ts`). It
selects board-owned `stranded_assigned_issue` escalations whose latest
run failed setup with the unfinished-writer error before any provider
work, by the current assignee. In one transaction it checks that the
block belongs to the escalation, that no newer run exists for the task,
that dependencies allow work, and that the workspace's writer hold is
now free. Then it resolves the escalation with a named outcome, returns
the task to `todo`, records one plain activity notice, and records a
pending wake.
- A later status decision that only kept the Blocked status it found
(from `blocked` to `blocked`) does not own the block. A decision that
chose the block still keeps the task blocked.
- Run the pass in `reconcileActiveRecoveryActions`, next to the legacy
rehome pass, and include its next wake retry time in the pass deadline.
- Generalize the legacy rehome wake delivery to
`deliverPendingRecoveryWakes`, keyed by the evidence key, so both passes
share one crash-safe, backed-off wake delivery. The rehome wake keeps
its fresh session. The new wake resumes normally.
- Keep the company selected by the scheduled sweep while a
released-writer wake is owed (`recoveryActionSweepCondition`).
- Document the rule in `doc/execution-semantics.md`.

## Verification

- `cd server && npx vitest run
src/__tests__/released-writer-escalation.test.ts
src/__tests__/legacy-workspace-rehome.test.ts
src/__tests__/issue-blocker-stall-wakeups.test.ts
src/__tests__/issue-blocker-stall-waiter-statement.test.ts
--maxWorkers=2`: all pass.
- Red/green: with the sweep hook removed, "continues the task once the
hold is released" fails (the task stays `blocked`). With the hook, it
passes.
- The new test seeds the staging row shape: a native writer run ended
with `native_finalization_superseded` still writing the task's own dirty
workspace, three legacy setup failures with the unfinished-writer error
and `{ kind: "bootstrap", providerWorkStarted: false }`, a board
escalation with `latestRunErrorCode: "setup_failed"`, the block 4.4
seconds after the last failure, and a later `blocked` to `blocked`
status decision.
- The tests also cover: no change while the writer still holds the
workspace, one wake only, a pending wake after a crash, a refused wake
that backs off, a later decision that chose the block, a dependency that
forbids work, a newer run, and another setup error.
- `cd server && npx tsc --noEmit -p tsconfig.json`: passes.

## Risks

- Low risk. The pass only acts on board escalations whose evidence is
exactly the unfinished-writer setup failure, and only when workspace
reservation would now admit a new writer. It never changes the
workspace, never applies a result, and never releases a writer fence.
- The task gets one normal wake through the issue-state guard (`todo`,
same assignee), so it cannot start twice.
- The pass adds one bounded query per sweep of a company with active
recovery actions.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only browser API reads of staging) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:58:06 -06:00
DottaandClaude Opus 5.5 b1ef1f3022 fix(recovery): settle a run's late exhaustion hold into its recorded fresh continuation (#21217)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run cannot be resumed, #21135 continues the task in one fresh
run from its saved work, with no person action
> - In Runner Full-Stack E2E 43321773136, two tasks whose run lost its
Runner three times recorded that continuation ("Unfinished run continues
in a fresh run from saved work", `delivery: recorded`), but no
continuation run started in the next 5 minutes
> - The native executor saves its exhaustion hold
(`native_session_retry_exhausted`, board-owned) in the same failure
path. When that hold lands after the decision, the decision is
idempotent per run and returns `already_continued`, so nothing settles
the new hold. The hold then blocks the continuation's delivery.
> - This pull request settles such later reconciliation holds of the
same run into the recorded continuation
> - The benefit is that the recorded continuation starts, and the task
does not wait on a person for an infrastructure fault

## Linked Issues or Issue Description

Refs #21135

**What happened?**
Runner Full-Stack E2E 43321773136, cases `recovery.explain-stall` and
`recovery.source-action` (OpenCode, native Runner): the server logged
the fresh-continuation decision at 22:51:30 and 22:51:55. Then the same
run logged "native result could not be marked workspace_failed" and
"heartbeat execution failed" (`native_session_retry_exhausted`). No
continuation run was created in the following 5 minutes. The ownership
pass kept reporting one `execution_reconciliation_hold` in progress.

**Expected behavior**
One fresh continuation starts from saved work after an infrastructure
loss, whatever order the failure path saves its records in.

**Steps to reproduce**
1. Record a fresh continuation for a failed native run.
2. Save an active `native_session_retry_exhausted` hold for the same run
afterwards.
3. Run the disposition and continuation delivery workers. Before this
change, no successor run is created.

## What Changed

- `server/src/services/fresh-continuation-fallback.ts`: when a
continuation is already recorded for the run, the decision settles the
run's other owned reconciliation holds (blocking, not a human gate) into
it: resolved, `restored`, system-owned, the person-facing text,
`coalescedIntoRecoveryActionId` and the primary's delivery state. A task
those holds blocked returns to `todo`. The continuation delivery worker
is notified.
- `server/src/services/fresh-continuation-fallback.test.ts`: regression
test for a late exhaustion hold.

## Verification

- RED before the change: `npx vitest run
src/services/fresh-continuation-fallback.test.ts -t "exhaustion hold
lands after"` failed with `expected [] to have a length of 1 but got
+0`.
- GREEN after: the full file passes (52/52).
`issue-recovery-actions.test.ts` and `execution-control-work.test.ts`
also pass (191 tests in the three files). Server `tsc --noEmit` is
clean.
- The E2E recovery cases in #21210 check the continuation starts and
finishes on the next cohort.

## Risks

- Low. Only holds of the same run that the fallback already owns
(reconciliation causes, blocking, no person gate) are settled, and only
after a continuation exists for that run. Human gates and workspace
restore holds are untouched.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:46:21 -06:00
DottaandClaude Opus 5.5 3e869c6f08 test(restart-chaos): quiet period before the end-of-wave grade, explicit swap moments, request records and release attribution (#21215)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness grades every task at the end of a
release-swap wave, locally (RS1) and on a hosted staging instance
(hosted mode)
> - The first hosted 3-release run graded 79 / 80, and the one failure
was a harness artefact: the drain declared quiet 20 s after a parent's
run ended, before the product's children-completed wake created its
continuation, so the task read as "nothing pending" and cleanup then
cancelled the continuation
> - The same run showed three more reporting gaps: per-turn product
notices counted as duplicated comments, failed requests could not be
broken down because request records stayed in memory (and the API count
included the health probes), and the per-swap table charged every failed
run to the last swap before it
> - Separately, the local swap timeline marked the start of a swap with
`fault_started` carrying `"signal":"SIGTERM"`, while the outgoing
server's real SIGTERM was `fault_server_down`; that has led to wrong
conclusions twice
> - This pull request fixes the drain rule, the comment count, the
request records, the release attribution and the swap timeline
> - The benefit is that the one number and the per-swap table mean what
they say

## Linked Issues or Issue Description

Refs #21153
Refs #21126

**Describe the bug**
1. A task whose continuation is created after the drain declares quiet
is graded `nothing pending`, and cleanup cancels the continuation.
2. "Prepared <document> for this response." notices, one per turn, count
as duplicated agent comments.
3. Request failures cannot be broken down after the run, and the API
continuity count includes health probes.
4. A failed run is attributed to the last swap before it, however long
after.
5. `fault_started` carries `"signal":"SIGTERM"` for an overlap swap,
though it marks the swap request; the outgoing SIGTERM is a different
event.

**Steps to reproduce**
Run a hosted or local release-swap wave with parent tasks whose subtasks
finish near the end of the wave.

**Expected behavior**
The grade waits for continuations, notices are informational, request
records are saved, failed runs are attributed to a release only when
live at it or soon after, and the timeline names each swap moment.

## What Changed

- `release-swap-grade.ts`: shared `drainState` (quiet only when nothing
is progressing and no run was created or finished for 90 s,
`QUIET_PERIOD_MS`); `failedAtSwap` / `swapForFailedRun` /
`failedOutsideSwaps` (a failed run counts at a swap if it was live at
the outgoing SIGTERM or failed within 15 minutes of it,
`RELEASE_FAILURE_WINDOW_MS`, and at most one swap); `isAutoNotice`; the
summary lists failed runs outside any release window and the notice
repeats as information.
- `release-swap-finish.ts` (local RS1): uses `drainState` and the new
attribution; the duplicate-comment query separates the notice.
- `hosted-grade.ts`: uses `drainState` (`--hosted-quiet-seconds`,
default 90); reads runs, tasks and task views again immediately before
grading; breaks request failures down by status and by inside/outside
the release windows.
- `hosted-api.ts`, `hosted-main.ts`: every request record carries kind
(health or API) and a path class with ids replaced; `requests.jsonl` is
written to the results directory; API continuity no longer includes
health probes.
- `hosted-readers.ts`: duplicated agent comments and notice repeats are
returned separately.
- `release-swap-driver.ts`, `release-swap-moments.ts` (new): the
timeline records `swap_incoming_spawned`, `swap_incoming_ready`,
`swap_traffic_switched`, `swap_outgoing_sigterm` and
`swap_outgoing_exited`; the summary prints the five times per swap and,
for every run started during the swap, the two moments it started
between and which server started it; `release-swap.json` carries both.
- `supervisor.ts` (same line count, still 1,000 lines): `fault_started`
for an overlap swap carries `marks: "swap_requested"` instead of a
signal. No test reads that signal; `fault_server_down` stays.
- `README.md`: documents the quiet period, attribution window, notice
rule, request records and swap moments.

## Verification

- `node server/node_modules/vitest/vitest.mjs run --config
tests/restart-chaos/vitest.config.ts` on the hosted, release-swap and
triggers test files: 105 tests pass across 12 files.
- Red/green for the drain race (`hosted-drain.test.ts`): a parent's run
ends, a children-completed continuation is created 20 s later and
succeeds; with the old rule (quiet period 0) the task grades
`still_working`/not by the continuation, with the new rule it grades
`pass` after the continuation ends.
- New tests for the notice filter, `pathClass` and the request
breakdown, the 15-minute attribution, and the five swap moments (both
the timeline events from the driver and the summary placement of runs
between moments, with the starting server).
- `node server/node_modules/typescript/bin/tsc -p
tests/restart-chaos/tsconfig.json`: clean.

## Risks

- Low risk: harness-only. The end-of-wave grade waits up to 90 s longer
after the last run change, within the existing drain cap.
- Per-swap failed-run counts change meaning: runs that failed more than
15 minutes after a stop and were not live at it move to the "failed
outside a release window" line.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits). Default context window; no
extended thinking mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:45:08 -06:00
DottaandClaude Opus 5.5 2eda54162a fix(evals): grade recovery on fresh continuation, accept in-turn queued-comment reads, verify the saved service model (#21210)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks that agents in chat do the right thing,
graded on saved product state and rendered links
> - Full cohort 8 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master 7747487ee7)
failed six cases after two deliberate product changes and one harness
race
> - #21135 continues an unfinished run in a fresh run from saved work
instead of leaving an exhausted, board-owned recovery action. Three
recovery cases still waited for that action.
> - #21175 counts a `get_task_history` read of a queued comment's
current revision as delivery. Two steering cases still required a
continuation run.
> - `companies.prompt-injection-guard` read the settings back before its
own save finished
> - This pull request grades the recovery and steering cases on the new
contracts and makes the settings harness verify the saved model before
it continues
> - The benefit is that the evals accept the intended product behavior
and still fail a broken product

## Linked Issues or Issue Description

Refs #21135
Refs #21175

**What happened?**
Cohort 8 failed `recovery.explain-stall`, `recovery.retry-exhausted` and
`recovery.source-action`: the harness waited 30 s for an "exhausted
source recovery action with no automatic wake". #21135 now logs
"Unfinished run continues in a fresh run from saved work" (`trigger:
recovery_refused`, `delivery: recorded`) instead.
`steering.edit-queued-message` and `agent-chat.multiple-handoffs` failed
their run counts: the running worker turn read the queued comment
(current revision) and applied it, so no continuation run was queued.
`companies.prompt-injection-guard` read `openai/gpt-oss-safeguard-20b`:
the server log shows the harness GET finished before the settings PUT.

**Expected behavior**
Infrastructure loss continues automatically in one fresh run, the chat
explains it, and the continuation finishes the task. A queued comment is
consumed once, in the running turn or in one continuation. The settings
harness tests only the model it saved.

**Steps to reproduce**
Run the six `chat-features.runner-opencode.local.*` cases on master
after #21135 and #21175.

## What Changed

- `runtime-infrastructure.ts` (`recovery.explain-stall`,
`recovery.source-action`, oracle `infrastructure-recovery.v6`): after
three real Runner losses, the setup waits for one system-owned
fresh-continuation record of the failed run (`?inspectSourceRunId`),
resolved as `restored`, `replay: "none"`, the failed run marked
superseded, and one run with `retryOfRunId` = the failed run that waits
for its brief in its own workspace. The chat question is asked while it
runs. The harness then releases the brief, and the grade requires the
continuation to succeed, save the brief's reference as the latest
revision and complete the original task. No person-owned hold and no
blocked task at the reply. Three runs: failed 1, succeeded 2. The reply
must still come from a run that read the recovery record (now also the
continuation run) and link the original task. Setup timing is saved to
`runtime-fresh-continuation.json`.
- `runtime-infrastructure-retry.ts` (`recovery.retry-exhausted`, oracle
`retry-exhausted.v6`): an infrastructure loss no longer offers Retry.
The case uses a failure only the person can resolve
(`native_provider_model_rejected`, a `PERSON_OWNED_FAILURE_CODES`
member) and keeps the Retry-under-budget-hold journey unchanged. The
setup proves the task waits blocked with no fresh continuation and no
successor run for 45 s.
- `runtime-busy-worker.ts`: the Retry listener helpers move here; a
listener counts only when its `.waiting` file is newer than the run's
start.
- `runtime-queue.ts`: `historyReadRevisions`, `readCurrentRevision`,
`runHistoryReads` read `task_history.read` events; `settledQueue` needs
a 30 s quiet settled state.
- `runtime-queue-controls.ts` (`steering.edit-queued-message`, oracle
`structural.v5`) and `runtime-multiple-handoffs.ts` (oracle `v5`):
accept in-turn consumption only with a worker `task_history.read` of
every queued comment's current revision, the change saved as the latest
revision (blue; "Launch sooner"), an empty queue, a done task and zero
continuations. The continuation path stays: exactly one continuation
with the unread comments in order, when the turn did not read the
current revision. Run bounds 2–3 and 4–5. The worker brief no longer
tells the agent to leave follow-ups for a continuation.
- `identity-company-models.ts` (`prompt-injection-guard` oracle
`prompt-injection-guard.v2`, `fast-responses` `v4`): before saving,
check the model field holds the requested model; wait for the PUT;
require 200, the sent model, the saved model, then a matching read-back.
- `recovery.yaml`: context and expected outcome for the three recovery
cases. Recorded in `chat-full-catalog-testops-manifest.json`
`changedFields` and pinned in its test. Descriptors and the inventory
digest are updated. `maps/feature-map/recovery.md` exhaustion line
corrected.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains`:
283 files, 3895 tests pass. New calibration tests cover valid,
reworded-valid and broken cases: no continuation, a person-owned hold,
two continuation records or runs, a replayed turn, an unfinished
continuation, a missing saved result; an old-revision-only read, a
missing outcome, a double application, two continuations; a model field
that lost the model, a different sent model, a wrong saved model.
- `pnpm test:e2e:runner:typecheck`: pass.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <head sha>
--check`: `changed: false`.
- `node scripts/check-maps.mjs`: aligned.
- Paid run: Runner Full-Stack E2E 43321773136 at 9de2bd5a6f, six cases:
prompt-injection-guard passed. The two fresh-continuation cases found a
product failure (the continuation never started), fixed in #21217.
retry-exhausted showed that a model-rejected failure offers no Retry
control. multiple-handoffs failed correctly: the turn read the
instruction and did not apply it. edit-queued-message did not reach the
new grade (6-minute CEO turn). Details are in a PR comment.

## Risks

- The recovery cases depend on the fresh continuation starting within 5
minutes. The paid run showed it did not start; #21217 fixes that. Until
#21217 merges, those cases fail on master.
- `recovery.retry-exhausted` expects a Retry control that the product
does not show for a model-rejected failure. It fails until a product
decision is made.
- `recovery.retry-exhausted` no longer covers infrastructure exhaustion;
the two other recovery cases cover the new automatic path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:26:50 -06:00
DottaandClaude Opus 5.5 79334f58ab test(runtime): stop one slow plugin Runner release/resume case from failing the cases after it (#21213)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The plugin Runner release/resume integration test proves that a
released Runner can be resumed by the next run
> - The test failed on unrelated PR #21171 in the "Verify Paperclip
Runner (vitest native server)" lane
> - One slow case caused 5 more cases to fail, and the PR retry could
not run
> - #21199 fixed the same class of failure in the plugin Runner eviction
test
> - This pull request applies the same isolation to the release/resume
test, and also isolates its shared spies
> - The benefit is that one slow case no longer fails the cases after it

## Linked Issues or Issue Description

Refs #21199

**What happened?**

`server/src/services/native-runtime/plugin-runner-release-resume.integration.test.ts`
failed on unrelated PR #21171. "completes codex runs through a retained
cached process" timed out at 60s. Then 5 more cases failed, some at 60s
and some in under 3s with `RangeError: Maximum call stack size
exceeded`. Every retry failed on `duplicate key value violates unique
constraint "issues_identifier_idx"`.

Root cause: every case changed process-wide state and restored it in its
own `finally` block. That state was the default TLS trust,
`https.globalAgent`, a `vi.spyOn` on
`PaperclipRunnerToolAuthority.prototype.definitions`, and a `vi.spyOn`
on `inspectRemoteWarmSessionRecovery`. Vitest continues to the next case
when a case times out, but the timed-out function continues to run.
While it ran, the next case read the timed-out case's spy as the
"original" `definitions`. When the timed-out case restored its spies,
the two spies called each other, and the stack overflowed. Its late
`finally` also removed the next case's TLS trust, so that case stayed in
`connecting` with no wire frames until its own 60s timeout. It also
closed the next case's warm sessions with
`closeIdleWarmNativeSessionsForRestart()`. The retry could not help,
because the issue identifier came only from the case name.

**Expected behavior**
A case that times out fails alone. Later cases and the PR retry run
normally.

**Steps to reproduce**
1. Make the `codex cached` case sleep 62s once, after its first run.
2. From `server/`, run `npx vitest run
src/services/native-runtime/plugin-runner-release-resume.integration.test.ts
-t "codex runs through a retained (cached|adopted|adopted restart)
process"`.
3. Before this change, 3 of 3 cases fail, with `Maximum call stack size
exceeded`. With `--retry=1`, 3 of 3 still fail.
4. After this change, only the slow case fails. With `--retry=1`, 3 of 3
pass.

**Paperclip version or commit**
origin/master at the base of this branch (after #21199)

**Deployment mode**
CI and local test only (embedded Postgres, real `paperclip-runnerd`)

**Agent adapter(s) involved**
paperclip_runner (codex and claude ACPX harnesses through the plugin
task driver)

## What Changed

- Create one fixture certificate in `beforeAll`. Set the TLS trust and
`https.globalAgent` once for the file, and restore them in `afterAll`.
- Read the original `PaperclipRunnerToolAuthority.prototype.definitions`
once at module load, before any case spies on it.
- Move the per-case teardown into an `afterEach` that is registered
before the database fixture. The teardown closes idle warm sessions, the
host control, and the TLS proxy, and it restores both spies. Each case
binds its own teardown holder before its first await.
- Add a random suffix to the issue identifier of each attempt, so the PR
retry can run the case again.
- Increase the idle timeout of the adopted cases from 5s to 10s. The
product limits a release to `min(idleTimeoutMs, 30s)`, and #21199
measured a loaded release that took more than 4.8s.
- Probe the Runner port on `0.0.0.0`, the address that the Runner binds.

No product code changed.

## Verification

- Injection test (one case sleeps 62s): see the steps above. Before: 3
of 3 failed, also with `--retry=1`. After: 1 of 3 failed without retry,
and 3 of 3 passed with `--retry=1`.
- Before/after loops of the full file (14 cases), run in parallel on the
same loaded macOS host: see the loop result comment on this PR.
- `tsc --noEmit -p server/tsconfig.json` reports no errors in the
changed file.

## Risks

Low risk. Only the test changed. The adopted (non-restart) cases take
about 5s more each. The file creates one certificate instead of 14. A
case that hangs inside Core execution can still leave a busy warm
session in module state.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with
tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1112.0-canary.18
2026-11-12 17:19:06 -06:00
DottaandClaude Fable 5.2 9a9cd20d91 feat(ci): send one in ten CODEOWNER PR runs and their outside checks to Honeycomb (#21192)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Contributors need fast, predictable CI feedback.
> - PR #16378 added a Honeycomb trace of the main PR workflow, but only
for PRs with the `profile-ci` label.
> - Ordinary CI time was therefore never recorded. All of the last 300
`CI profile` runs were skipped.
> - Reviews and scans outside the `PR` workflow, such as Greptile, were
not in the trace.
> - This pull request exports one CODEOWNER PR run in ten without a
label, and adds outside checks to every trace.
> - The benefit is a continuous record of total CI time and of which job
or check was slowest. The Honeycomb key stays out of reach of outside
contributors.

## Linked Issues or Issue Description

Refs #16378

**What existing behavior does this improve?**

The Honeycomb CI trace that the `CI profile` workflow exports.

**Current behavior**

- The exporter runs only for a PR with the `profile-ci` label. The label
also forces the full suite.
- The trace has the jobs of the `PR` workflow only. Greptile, security
scans, commit statuses and other workflows are absent.
- The only total is the `PR` workflow time.

**Proposed behavior**

- One `PR` run in ten is exported without a label. The run number must
end in 0. The trace has every job and step. Each span has a sample rate
of 10.
- Each trace has one span for each outside check and each other workflow
on the head commit.
- The root span has `ci.workflow_ms`, `ci.all_checks_ms`,
`ci.last_to_finish` and `ci.slowest_check`.

**Reason and benefit**

Maintainers can see how long CI takes and what is slow, for example a
long Greptile review, without a label on each PR. Sampling keeps the
volume near 1.7 million events each month. Export of every run is near
17 million.

## What Changed

- `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or
one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without
the label).
- `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for
outside checks. Add the sample rate and the new root fields. Sampled
runs do not warn about absent test artifacts.
- `scripts/ci-profile/export.mjs`: select the mode. Read check runs,
commit statuses and workflow runs for the head commit. Wait a maximum of
15 minutes for outside apps that are not complete. Sampled runs read no
artifacts.
- `.github/workflows/ci-profile.yml`: no new job. The job condition now
requires a branch of this repository, an actor and rerun actor on the
`AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add
`checks: read` and `statuses: read`. Increase the timeout to 30 minutes
for the bounded wait.
- `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs,
outside checks, the totals and the job condition.

## Verification

- `node --test .github/scripts/tests/ci-profile.test.mjs
.github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail.
- `actionlint .github/workflows/ci-profile.yml`: no findings.
- Dry runs against real runs with the real CODEOWNERS check (`node
scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`):
- A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate
10, 5 outside checks and 1 other workflow.
- A sampled run from an outside contributor (run number 64800): "No
CODEOWNERS-authorized PR at this revision; no data exported."
- A CODEOWNER run that is not in the sample: "This attempt was not
selected for profiling; no data exported."
- New tests cover: the sample rule, fork and outsider PRs, a bot rerun
actor, a stale head, outside-check spans and totals, the bounded wait,
and the job condition.
- Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs
installed dependencies, and this change does not touch the reporters. CI
runs it.
- No data was sent to Honeycomb before merge. The workflow runs from
`master`, so the first real export occurs after merge.

## Risks

- The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If
that variable is empty, nothing is exported. All four CI CODEOWNERS are
on the roster today.
- The roster is only a first filter. The exporter still checks the PR
author and both actors against CODEOWNERS on `master` before it reads or
sends data. The key stays in the `ci-profiling` environment, which
permits only `master`.
- The export job can now run for a maximum of 30 minutes on a
GitHub-hosted runner while it waits for an outside check.
- GitHub gives no completion time for a workflow run. The span for
another workflow ends at the last update of that run.
- Honeycomb volume increases by about 1.7 million events each month.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with
tool use, code execution and repository inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:57:23 -06:00
DottaandClaude Opus 5.5 0904a65d82 fix(server): local board boundary identifies agent processes on Linux it cannot inspect (#21209)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In `local_trusted` mode, the operator's own local processes get
implicit board access without credentials
> - #21181 added a local board boundary. It refuses implicit board
access to requests that come from an agent run's process tree
> - On Linux CI hosts, the boundary sometimes did not decide a loopback
request. It logged a warning and kept the earlier permissive behavior
for that request
> - The cause was not IPv6. IPv4 and IPv6 loopback lookups worked. The
gap was a connection that belonged to a same-user process that the
server cannot inspect
> - This pull request makes the Linux boundary decide those connections
from the process ancestry that the server can read, and refuse them when
that ancestry is not clearly the operator's
> - The benefit is that the local board boundary now holds on Linux for
IPv4 and IPv6 loopback, and the operator's browser and CLI keep working

## Linked Issues or Issue Description

Refs #21181

**What happened?**
On Linux CI hosts, the local board boundary sometimes could not
attribute a loopback request to a process. It then logged that it could
not identify the local process and kept the earlier permissive behavior
for that request. In one chat eval cohort, 37 of 191 server logs
contained this warning. The warning is logged once per server process.

**Expected behavior**
The boundary decides every uncredentialed loopback request on Linux. A
request from an agent run gets no implicit board access. Requests from
the operator's browser and CLI keep implicit board access.

**Actual behavior**
Some requests on Linux were not decided by the boundary, and they got
implicit board access.

## What Changed

- The Linux inspector now records which same-user processes it cannot
inspect. Before, it only recorded that one existed.
- The Linux TCP table lookup now also returns the socket's owner UID. It
covers `/proc/net/tcp` and `/proc/net/tcp6`, including IPv4-mapped
addresses.
- If no inspectable process owns the connection, the server reads the
TCP tables again. If the connection is gone, has changed, or belongs to
a different user, the request is refused.
- If the connection is still open and same-user, the server checks every
process it cannot inspect. Implicit board access stays only if each one
has a readable parent chain that does not belong to an agent run.
Otherwise the request is refused.
- The "cannot identify" fallback now applies only when the platform has
no inspector, or when Linux cannot read the TCP tables. The server keeps
logging a warning in that case.
- `doc/DEPLOYMENT-MODES.md` describes the new Linux behavior.

**How the operator is recognized**
- Browser and CLI processes are normal same-user processes. The server
reads their open sockets and environment, so their requests are
classified as before.
- A browser helper process that the server cannot inspect is still the
operator when its readable parent chain leads to the browser and has no
agent-run markers.
- The server's own requests and requests from the server's ancestors
stay the operator, as before.
- Connections owned by other users were refused before, and they are
still refused.

## Verification

- `pnpm --dir server exec vitest run
src/__tests__/local-agent-process-boundary.test.ts` passes on macOS (15
passed, 10 Linux-only skipped).
- The same file ran on Linux (Docker, `node:24-bookworm`, non-root user,
IPv6 loopback on). Result: 25 passed.
- Red check on Linux with the previous boundary and the new tests: the
agent-run cases for processes the server cannot inspect failed for both
`127.0.0.1` and `::1`. The operator cases passed.
- New Linux-only tests run real processes over `127.0.0.1` and `::1`:
  - an operator process keeps board access
  - an agent-run process is refused
  - an operator process the server cannot inspect keeps board access
  - an agent-run process the server cannot inspect is refused
- New platform-independent tests cover the classification rules and the
`/proc/net/tcp6` fixture parsing.
- A paid chat-feature run on Linux CI, with temporary diagnostic logging
on a separate branch, checks the server logs for the warning and for
refused operator requests. The results will be in a PR comment.

## Risks

- A rare operator tool that the server cannot inspect, and whose parent
chain the server cannot read, now loses implicit board access. Such a
tool can use a board API key.
- The extra TCP table read happens only when a connection has no
inspectable owner. Normal requests do no extra work.
- Platforms without process inspection keep the previous behavior.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, Docker on Linux) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:52:06 -06:00
DottaandClaude Opus 5.5 b51f1cb55f test(chat): size the long Board publication tests' timeout to their measured work (#21205)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-channel integration suite is one 86,000-line file. Its
cases share one database, and the chat services scan that whole database
> - The "Lossless long Board publications" cases publish
100,000-character messages in many durable parts. They use the file's
default 15 s test timeout
> - Run alone, a case takes about 0.6 s. Late in a full run of the file,
the same cases take 6–10 s locally and 1.4–15 s in CI, because earlier
cases leave rows that the services still scan
> - On #21041, "keeps an ambiguous slack text part blocked across
restart…" hit the 15 s limit twice (30 041 ms with its retry), and the
GitHub variant passed at 14 960 ms
> - This pull request gives this describe block a 45 s timeout, sized to
the measured work
> - The benefit is that the chat shard stops failing on unrelated PRs
when these cases run late in a shard

## Linked Issues or Issue Description

**What happened?**
`server/src/__tests__/chat-channels.integration.test.ts` > "Lossless
long Board publications" > "keeps an ambiguous slack text part blocked
across restart and retries only that part and its tail" failed with
`Test timed out in 15000ms` on PR #21041 (General tests chat 2/3), a PR
that did not change chat code. It passed on rerun.

**Expected behavior**
The case passes wherever the shard puts it.

**Steps to reproduce**
Run the whole file: `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts --reporter=json`. In my
local full run, "preserves all 100000 characters for slack new-comment
publication" took 9 505 ms and "keeps an ambiguous slack text part…"
took 5 856 ms. Run alone, they take 817 ms and 577 ms. In CI logs from
the last 48 hours, the ambiguous-part cases took between 1 408 ms and
over 15 000 ms.

**Paperclip version or commit**
master at 4e932fd178 and later.

**Deployment mode**
CI server tests (embedded PostgreSQL).

**Agent adapter(s) involved**
N/A, CI test.

## What Changed

- `server/src/__tests__/chat-channels.integration.test.ts`:
`describe("Lossless long Board publications", { timeout: 45_000 }, …)`.
Other heavy publication cases in this file already use 40 s.

## Verification

- With an injected 18 s wait in the ambiguous slack case, the case
passes on this branch. On master the same wait fails at 15 s.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts -t
"Lossless long Board publications"`: 16 of 16 pass.

## Risks

- Low risk. Test-only change. A real hang in these cases now fails after
45 s instead of 15 s.
- This does not fix the underlying slowdown: cases get about 10 times
slower late in this file because services scan rows that earlier cases
leave in the shared database. Splitting the file or giving groups their
own database is the structural fix. That is larger than this PR.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, GitHub CLI, Vitest).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:35:30 -06:00
DottaandClaude Fable 5.2 161ef8121e perf(recovery): prepare the blocker-stall waiter read once instead of writing its SQL for every stalled task (#21201)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a task stalls on a recovery action that needs the board,
recovery wakes the agents of the tasks that wait on it: its parent and
the tasks it blocks (`wakeWaitersOfStalledIssues` in
`server/src/services/recovery/blocker-stall-wakeups.ts`).
> - Each pass reads the waiters of every stalled task with one statement
for each stall. The statement has 18 parameters and two correlated
subqueries. Its SQL text is 0.8 KB, and it is the same for every stall
of the same shape.
> - The server wrote that text with the query builder for every stall on
every pass. On a hosted instance the server main thread was near one
full core and garbage collection used a quarter of it. A statistics
sample showed this statement at about 7.8 executions per second.
> - I measured it on a local PostgreSQL through the real client. The
query builder allocates 103 KB to write and run this statement. The same
statement from a prepared query allocates 15 KB.
> - This pull request writes the SQL text of the waiter read one time
for each client and shape. Each execution then only binds values.
> - The benefit is 76% less allocation and 5 times fewer garbage
collections for a pass over 20 stalled tasks. The statement, its
parameters and its rows do not change.

## Linked Issues or Issue Description

Related: #21195 (the same change for the native chat progress page read;
it added the `preparedStatements` helper that this pull request uses),
#21131 and #18965. No issue exists for this read.

**What existing behavior does this improve?**

The cost, in the server process, of the waiter read that the
blocker-stall wake pass runs for each stalled task.

**Subsystem affected**

Recovery: blocker-stall wakes
(`server/src/services/recovery/blocker-stall-wakeups.ts`).

**Current behavior**

`wakeWaitersOfStalledIssues` builds the waiter query with the Drizzle
query builder inside its loop, and Drizzle then writes the SQL text from
it, one time for each stall and page. Measured on the merge base, Node
26, local PostgreSQL 18, median of 5 passes of 300 calls. The fixture
has 20 stalled tasks, each with a parent and one dependent task that are
already woken, so a pass writes nothing:

| Call | Statements | Waiter SQL texts written | Allocated | Process CPU
| Collections per 1,000 calls (pause time) |
| --- | --- | --- | --- | --- | --- |
| Company pass over 20 stalls | 21 | 20 | 2,298.3 KB | 3,855 µs | 100
(163.7 ms) |
| Pass for one stalled task | 4 | 1 | 355.0 KB | 547 µs | 13.3 (14.3 ms)
|

At the sampled rate of 7.8 executions per second the read allocates
about 0.8 MB per second. This rate is one sample of one instance, taken
soon after a long outage, when recovery had a backlog. It is not a
steady-state figure.

**Proposed behavior**

The query moves into one builder function, `stallWaiterPageQuery`. The
module prepares it one time for each client and shape, with placeholders
for the values. The shapes are: the stalled task has a parent or not;
all waiters or one waiter; first page or later page. The loop executes
the prepared statement.

**Reason and benefit**

Same benchmark, this branch:

| Call | Statements | Waiter SQL texts written | Allocated | Process CPU
| Collections per 1,000 calls (pause time) |
| --- | --- | --- | --- | --- | --- |
| Company pass over 20 stalls | 21 | 0 | 547.3 KB | 1,474 µs | 20 (24.8
ms) |
| Pass for one stalled task | 4 | 0 | 266.9 KB | 496 µs | 10 (9.9 ms) |

One waiter read costs about 88 KB less. At the sampled rate this removes
about 0.7 MB per second of allocation, which is about 0.9% of the 77.7
MB per second that the instance allocated. It is a small part of the
total. The CPU numbers come from a machine with other load and are less
stable than the allocation numbers.

**Breaking changes**

None.

## What Changed

- `server/src/services/recovery/blocker-stall-wakeups.ts`: the waiter
query moves out of the loop into `stallWaiterPageQuery(db, binding)`.
`stallWaiterPageStatement` returns the prepared statement for the shape
and the values to bind. The loop executes it. The lines inside the two
SQL fragments keep their indentation, so the SQL text is byte-identical.
The pull request adds no comment. The one existing comment of this read
stays at the call site, where the code still makes that choice.
- `server/src/__tests__/issue-blocker-stall-waiter-statement.test.ts`
(new): statement equivalence, row equivalence on PostgreSQL, and the
red/green mechanism test.
- `server/src/__tests__/helpers/blocker-stall-fixture.ts` (new): the
fixture that the test and the benchmark share.
- `server/src/__tests__/prepared-statements-driver.test.ts` (new):
proves at the driver that a prepared read and the query builder ask
postgres.js for the same thing, with the driver's prepared-statement
setting unset, false and true. It covers this read and the native chat
progress page read of #21195.
- `server/scripts/benchmark-blocker-stall-waiters.ts` (new, no
comments): the benchmark above. It loads the module at `--baseline-ref`
next to the current module in one process. `sweep` is one company pass
over `STALLS` stalled tasks whose waiters are already woken: one stall
read, then one waiter read for each stall. `task` is the pass for one
stalled task. Before it measures, the script also runs a company with a
paused agent and 101 waiters on a stall with a parent and on a stall
without one, and one pass for a single waiter of each. This covers six
shapes: the first page, the later page and one waiter, each for a stall
with a parent and for a stall without one. The script fails unless both
versions send the same statements, byte for byte, with the same
parameters and return the same result. Passes alternate between the
versions with a full collection before each pass. CPU is the whole
process. Allocated bytes are V8's `total_allocated_bytes`. Optional
environment variables: `STALLS` (default 20), `ITERATIONS` (default 300
calls for each pass), `PASSES` (default 5 for each version).

Not changed: which statements run, their order, the other reads of the
pass (they are small and vary by waiter), the wake itself, the driver's
prepared-statement setting, and the schema. The read takes no lock and
runs on the root client outside a transaction. Its two lists (waiter
statuses and wake statuses) are module constants, so the statement has
no variable-length list parameter.

## Verification

- Statement equivalence (`sends the SQL text and the parameters that the
query builder writes, for every shape of the read`): for all eight
shapes the prepared SQL text equals the text that the builder writes
with the values, and the bound parameters are equal. The test also
proves that a stall without a parent binds `null` for the parent.
- Against the merge base at the driver: the benchmark compares every
statement that the merge base module and this branch send for six of the
eight waiter-read shapes. The texts are byte-identical and the
parameters are equal. The two shapes that it does not reach (one waiter
on a later page) cannot occur: a one-waiter read returns at most one
row, so it has no later page.
- Row equivalence on PostgreSQL (`returns the rows of the query builder:
same values, types and order, and no other company's rows`): two
companies. The fixture has a stall with a parent and six dependents, and
a stall without a parent. One dependent is already woken, one is done,
one is hidden and one has no assignee. One has a failed wake only, and
one has a wake row with the same key in the other company; both stay
waiters. The prepared statement and the builder return strictly equal
rows for all waiters, one waiter, a later page, and a stall read under
the other company's id (no rows). Null columns stay null.
- Mechanism, red then green (`writes the SQL text of the waiter read one
time and then only binds values`): a pass over three stalls writes no
waiter SQL text and sends one identical text three times. On the merge
base module the same test fails: the pass writes three texts.
- Driver behavior (`asks the driver for the same thing as the query
builder with DATABASE_PREPARED_STATEMENTS=...`, three cases): Drizzle's
`.prepare(name)` on postgres.js does not send the name to the database.
`PostgresJsSession.prepareQuery` drops it
(`drizzle-orm/postgres-js/session.js`, lines 87 to 100), and
`PostgresJsPreparedQuery.execute` calls `client.unsafe(text, params)`
(lines 33 and 43). An ordinary query reaches the same call through
`_prepare().execute()` (`pg-core/query-builders/select.js`, lines 795 to
799). The test runs both reads through the builder and through the
prepared statement on one connection for each setting. The arguments and
the options of the driver call are strictly equal. With the setting
unset or false, the server has no named statement after either path, so
a transaction-mode pooler sees no change. With the setting true, the
server has one named statement. Its name is the one that postgres.js
generates for the connection, and both paths use that same statement.
- Existing tests: `issue-blocker-stall-wakeups.test.ts` and
`legacy-workspace-rehome.test.ts` passed (30 tests) with the new file (3
tests). `prepared-statements-driver.test.ts` passed (3 tests).
- `pnpm --filter @paperclipai/server typecheck` passed. `pnpm
check:module-boundaries` passed.
- `pnpm verify` returns exit 2 (broader verification required, because
the change adds a file under `server/scripts`). I did not run the full
local suite. CI supplies that coverage.

Reproduce:

```sh
pnpm --filter @paperclipai/server exec vitest run src/__tests__/issue-blocker-stall-waiter-statement.test.ts \
  src/__tests__/issue-blocker-stall-wakeups.test.ts src/__tests__/legacy-workspace-rehome.test.ts \
  src/__tests__/prepared-statements-driver.test.ts
TSX_DISABLE_CACHE=1 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-blocker-stall-waiters.ts \
  --baseline-ref=549816e6a0
```

Not verified: the effect on a hosted instance. The rate above is one
sample. The local numbers are from Node 26; the hosted instance runs
Node 24.

## Risks

- Low. The statement text and parameters are equal by test, the read
takes no lock, and it runs on the root client outside a transaction.
- A prepared statement keeps the text of the first build. The lists
inside the statement are module constants, so the text cannot go stale
in a running process.
- A test that mocks `db.select` to observe this read no longer sees it.
I found none for this read.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model `claude-fable-5-2` (Fable 5.2), in Claude
Code with tool use and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:32:29 -06:00
DottaandClaude Opus 5.5 6ac7211cb9 fix(heartbeat): stop re-delivering a remote-stop follow-up that a succeeded run already read (#21203)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person's follow-up comment must reach the agent once
> - #21175 made a succeeded run's mid-turn read of a comment (through
`get_task_history`) count as delivery for the queued-comment paths
> - The remote-Stop resume path decides delivery with
`undeliveredLegacyUserCommentIds`, which ignores those reads
> - So a follow-up that a succeeded run already read was sent again, and
the agent posted a second DONE on a finished task
> - This pull request applies the same rule on that path and adds a read
helper for a later change
> - The benefit is one answer per follow-up on every resume path, with
no dropped messages

## Linked Issues or Issue Description

Refs #21175

**What happened?**
A saved follow-up comment was resumed after a remote Stop (wake reason
`issue_commented`, idempotency key `remote-stop-comment:…`). A succeeded
run had already read this comment mid-turn. The resumed wake started a
new turn, and the agent posted a second DONE on a finished task. The
cause: `undeliveredLegacyUserCommentIds`
(explicit-native-continuation.ts) checks only wake input and accepted
first turns. It does not check the `task_history.read` records that
#21175 added.

**Expected behavior**
A follow-up that a succeeded run read at its current revision is
delivered. The remote-Stop resume path skips it. A read by a run that
did not succeed does not count, so that follow-up is still delivered
once.

**Steps to reproduce**
1. A native run reads the task history after a person posts a follow-up.
2. The run succeeds. The follow-up stays saved as a deferred wake.
3. The remote-Stop resume path runs for that wake.
4. Before this change, the follow-up is delivered again.

**Paperclip version or commit**
`master` at 549816e6a0.

## What Changed

- `server/src/services/queued-comment-delivery.ts`:
`historyReadDeliveredCommentIds` exposes the #21175 rule for a list of
comment IDs. `taskHistoryReadsForRun({ companyId, runId })` returns each
comment one run read, with the read time and the comment revision that
the read returned. It returns these whatever the run's outcome. It reads
only that run's events through `heartbeat_run_events_company_run_idx`.
No delivery decision uses it.
- `server/src/services/explicit-native-continuation.ts`:
`undeliveredLegacyUserCommentIds` also removes comments that a succeeded
run read at their current revision. Its two callers are the remote-Stop
resume path and the queued explicit-continuation check, and both now use
the #21175 rule.
- Deliberate asymmetry: a read by a run that did not succeed never
counts. A dropped person's message is worse than a duplicate answer.
- `doc/run-log-events.md`: documents both points.
- New test file
`server/src/services/queued-comment-history-read.test.ts`.

## Verification

- RED on `549816e6a0` (`npx vitest run
src/services/queued-comment-history-read.test.ts`): 2 failed, 1 passed.
  - The follow-up that a succeeded run read was still returned as owed.
  - `taskHistoryReadsForRun` did not exist.
- The case where a failed run read the follow-up (still owed) passed, as
intended.
- GREEN: 3/3 pass.
- Neighbouring suites (`first-turn-queued-input`,
`explicit-native-continuation`, `heartbeat-agent-directory-busy`,
`heartbeat-held-comment-wake`, `execution-control-work`): 517/517 pass.
Server `tsc` is clean.

## Risks

- Low risk. The function and its callers keep their contract. It returns
fewer comment IDs only when a succeeded run of the assignee read that
exact revision.
- Comments read through the REST API by non-native adapters are not
recorded, so their behavior does not change.
- No migration. The added lookup uses the time-bounded, index-backed
query from #21175.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:23:34 -06:00
DottaandClaude Opus 5.5 57aeda190b test(runtime): stop one slow plugin Runner eviction case from failing the cases after it (#21199)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The plugin Runner eviction integration test proves that a retained
Runner continues after the host evicts it
> - The test failed on many unrelated pull requests in the "Verify
Paperclip Runner (vitest native server)" lane
> - One slow case caused other cases to fail, and the PR retry could not
run
> - This pull request isolates each case from a case that timed out, and
gives the retry a fresh issue
> - The benefit is that one slow case no longer fails the cases after it

## Linked Issues or Issue Description

**What happened?**

`server/src/services/native-runtime/plugin-runner-eviction.integration.test.ts`
failed in CI on at least 5 unrelated branches in 48 hours, most recently
on #21149. The errors were `Test timed out in 60000ms` and `Timed out
waiting for automatic release A`. Each retry then failed at once with
`duplicate key value violates unique constraint
"issues_identifier_idx"`. The failed cases logged a Runner in
`connecting` with no wire frames.

Root cause: every case changed process-wide state. It set the default
TLS trust and `https.globalAgent` for its own certificate, and it
restored them in its own `finally` block. Vitest continues to the next
case when a case times out, but the timed-out function continues to run.
Its late `finally` then removed the TLS trust of the next case and
closed the warm sessions of the next case with
`closeIdleWarmNativeSessionsForRestart()`. The next case could not open
its Runner connection, waited for the 60s startup deadline, and also
timed out. That case then did the same to the case after it. In CI run
43266976387, one timeout caused 3 more failures. Locally, one timeout
caused 5 more failures.

Two more problems made this worse:
- The issue identifier came only from the case name. A retry of the same
case therefore failed on the unique identifier, so the PR retry never
ran the test again.
- Detached cases used a 5s idle timeout. The product limits a release to
`min(idleTimeoutMs, 30s)`, so these cases had 5s for the drain and the
park. Under load, one park took more than 4.8s, and the release expired.

**Expected behavior**
A case that times out fails alone. Later cases and the PR retry run
normally.

**Steps to reproduce**
1. Make one case slower than 60s. For example, add a 62s sleep after
release A in `codex cached retired`.
2. Run `npx vitest run
src/services/native-runtime/plugin-runner-eviction.integration.test.ts
-t "codex with a (cached|detached) Core owner"` from `server/`.
3. Before this change, 3 of 4 cases fail: the slow case, and 2 cases
that it disturbs. With `--retry=1`, the slow case also fails on the
duplicate identifier.
4. After this change, only the slow case fails. With `--retry=1`, all 4
cases pass.

**Paperclip version or commit**
master at ef6d347f79, rebased on the current origin/master

**Deployment mode**
CI and local test only (embedded Postgres, real `paperclip-runnerd`)

**Agent adapter(s) involved**
paperclip_runner (codex and claude ACPX harnesses through the plugin
task driver)

## What Changed

- Create one fixture certificate in `beforeAll`. Set the TLS trust and
`https.globalAgent` once for the file, and restore them in `afterAll`. A
case no longer changes process-wide trust.
- Move the per-case teardown (close idle warm sessions, close the host
controls, close the TLS proxy) into an `afterEach`. It is registered
before the database fixture, so it runs before the database clients
close, also when a case times out.
- Add a random suffix to the issue identifier of each attempt, so the PR
retry can run the case again.
- Increase the detached idle timeout from 5s to 10s, and wait 200ms past
it as before. This gives the release the time that it needs under load.
- Probe the Runner port on `0.0.0.0`, the address that the Runner binds.

No product code changed.

## Verification

All loops ran on a shared, loaded macOS host (load average 15 to 20 on
18 cores), plus 2 CPU busy loops. Each iteration ran the 12 cases that
match `with a (after inspection|missing state|reused lease) Core owner`.

- Before: 20 iterations, 2 failed (10%). In iteration 10, `claude reused
lease retired` timed out at 60s, and 5 later cases failed on
`readNativeSessionIdleWork() === "none"` because of the case that timed
out. In iteration 17, `claude after inspection retired` failed with
`Timed out waiting for automatic release A`.
- After: see the loop result comment on this PR.
- Injection test (one case sleeps 62s): before, 3 of 4 cases failed, and
with `--retry=1` the retry failed on the duplicate identifier. After,
only the slow case failed, and with `--retry=1` all 4 passed.
- Full file after the change: 40 of 40 passed in 320s locally (335s
before, because the file now creates one certificate instead of 40).
- `tsc --noEmit -p server/tsconfig.json` reports no errors in the
changed file.

## Risks

Low risk. Only the test changed. The detached cases take about 5s more
each (12 cases, about 60s in all), which is partly offset by creating
one certificate per file. A case that hangs inside Core execution can
still leave a busy warm session in module state. This change does not
prevent that. It removes the causes of the cascade that this PR found.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with
tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1112.0-canary.17
2026-11-12 16:18:01 -06:00
DottaandClaude Opus 5.5 7f1cdb842d fix(evals): admit the owner-routed routine follow-up run and brief only the requested change (#21198)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat eval catalog checks that agents in chat do the right thing
on saved product state.
> - In `routines.manage-other-owner`, the CEO cannot change Morgan's
routine. The valid outcome is to route the change to Morgan in one
linked task.
> - That valid path uses three runs: the CEO's turn, Morgan's run on the
task, and the CEO's automatic completion follow-up on the chat. The case
allowed only two runs, so the harness stopped valid attempts with
"exceeded its declared run bound".
> - In one of those attempts, the CEO also told Morgan to use UTC.
Nobody asked for UTC. The routine was on America/Chicago. The oracle
correctly failed that change.
> - This pull request admits the third run only in its declared role,
and it tells agents to brief only the change the person asked for.
> - The benefit is that the eval accepts the valid delegation path and
still fails extra work and unrequested setting changes.

## Linked Issues or Issue Description

**What happened?**

Runner Full-Stack E2E runs 43298807862 and 43300040122 stopped
`routines.manage-other-owner` with "Scenario exceeded its declared run
bound". Both had three runs: the CEO turn, Morgan's run on the routed
task, and the CEO's `chat_task_completed` follow-up. In 43298807862 the
CEO's brief also told Morgan to switch the routine to UTC, and
`same-owned-definition-and-triggers` failed.

**Expected behavior**

The delegation path with exactly one routed owner task passes with three
runs. The refusal path stays at two runs. An agent that relays a change
keeps every setting the person did not ask to change.

**Steps to reproduce**

Run `chat-features.runner-opencode.local.routines.manage-other-owner`
and let the CEO route the change to Morgan.

**Paperclip version or commit**

master at 7747487ee7.

## What Changed

- `tests/runner-e2e/chat-catalog-domains/integrator-routines.ts`: the
case bound is now 3 runs (minimum 1). A new `declared-run-roles` check
allows at most 2 runs, plus at most one run that is the CEO's completion
follow-up (`chatCompletionFollowUp` from `run-roles.ts`) on the source
chat for the single owner task with a bound Morgan run. The observation
now records the source chat id. Oracle version is now
`manage-other-owner.v7`.
- `tests/runner-e2e/chat-catalog-descriptors.json`: the static
descriptor matches the new bound and oracle version.
- `tests/runner-e2e/chat-catalog-domains/integrator-routines.test.ts`:
calibration tests. The delegation path with 3 runs passes. A 4th run
fails. A second follow-up fails. 3 runs without a delegated task fail. A
follow-up for a different task fails. The 2-run refusal path still
passes.
- `server/src/services/agent-conversations.ts`: one sentence in the chat
directive task guidance: "When you relay or delegate a change, state
only what the person asked to change, and keep every other setting, such
as a time zone, owner or schedule detail, as it is."
- `server/src/__tests__/agent-conversations.test.ts`: red/green test
that the directive and the built chat task prompt carry that sentence.
- No scenario yaml change, so the TestOps manifest does not change.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog
tests/runner-e2e/chat-catalog-domains/integrator`: 283 files, 3850 tests
pass.
- `pnpm test:e2e:runner:typecheck`: pass.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
pass.
- `server`: `agent-conversations.test.ts` and
`board-access-denial.test.ts` pass (109 tests);
`codex-local-execute.test.ts` passes. The new directive test failed
before the directive change and passes after it.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs
792f8fb308 --check`: `changed: false`.
- `pnpm verify` exited 2 (descriptor JSON is a global CI input), so
required CI supplies broad coverage.
- Paid single-case run: Runner Full-Stack E2E 43313272302 on a throwaway
branch at this commit. Result is reported in a comment.

## Risks

- Low risk. The eval change only admits a third run in one declared
role. Extra or unbound runs still fail.
- The directive gains one sentence. It can make an agent more
conservative when it relays a change, which is the intent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:10:47 -06:00
DottaandClaude Fable 5.2 549816e6a0 perf(chat): prepare the native progress page read once instead of writing its SQL on every wake (#21195)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run that came from a chat reports safe progress to that
chat. Each progress event wakes a queue, and each wake reads one page of
candidates (`nativeProgressPages` in
`server/src/services/chat-native-progress.ts`).
> - The page read joins six tables and has 44 parameters. Its SQL text
is 3.6 KB. The server wrote that text with the query builder on every
wake, and the text is the same for every wake of the same filter.
> - On a hosted instance the server main thread was near one full core
and garbage collection used a quarter of it. An allocation sample put
46% of all allocation under the query builder's SQL writing. The sample
does not name statements.
> - I measured single statements on a local PostgreSQL through the real
client. The query builder allocates about 20 KB for the smallest
statement, plus about 2 KB for each selected column and about 3 KB for
each predicate. This page read costs 296 KB and 0.34 ms of CPU to write.
The same statement from a prepared query costs 21 KB.
> - This pull request writes the SQL text of the page read one time for
each client and filter shape. Each execution then only binds values.
> - The benefit is 91% less allocation and 14 times fewer garbage
collections for one page read. The statement, its parameters and its
rows do not change.

## Linked Issues or Issue Description

Related: #21131 (fixed SQL text for the native run event append) and
#18965 (an earlier prepared read, in the GitHub review wait). No issue
exists for this read.

**What existing behavior does this improve?**

The cost, in the server process, of the candidate page read that every
native chat progress wake runs.

**Subsystem affected**

Chat channels: native run progress publication
(`server/src/services/chat-native-progress.ts`).

**Current behavior**

`nativeProgressPages` builds the complete query with the Drizzle query
builder for each page, and Drizzle then writes the SQL text from it.
Measured on the merge base, Node 26, local PostgreSQL 18, median of 5
passes of 2,000 calls:

| Call | SQL texts written | Allocated | Process CPU | Collections per
1,000 calls (pause time) |
| --- | --- | --- | --- | --- |
| One page read | 1 | 297.3 KB | 851 µs | 14.5 (30.7 ms) |
| One wake for a task (page read + one transaction) | 7 | 681.3 KB |
2,238 µs | 34 (76.6 ms) |

A statistics sample of one hosted instance on one morning showed this
statement at about 3.9 executions per second. At that rate the read
allocates about 1.1 MB per second for SQL text. This rate is one sample,
taken soon after a long outage. It is not a steady-state figure.

**Proposed behavior**

The query stays in one builder function, `nativeProgressPageQuery`. The
module prepares it one time for each client and each of the six filter
shapes (all tasks, one company, one task; first page or later page),
with placeholders for the values. Each page read executes the prepared
statement.

**Reason and benefit**

Same benchmark, this branch:

| Call | SQL texts written | Allocated | Process CPU | Collections per
1,000 calls (pause time) |
| --- | --- | --- | --- | --- |
| One page read | 0 | 26.8 KB | 219 µs | 1 (1.4 ms) |
| One wake for a task | 6 | 412.1 KB | 1,699 µs | 21 (55.5 ms) |

At the sampled rate this removes about 1.05 MB per second of allocation,
which is about 1.4% of the 77.7 MB per second that the instance
allocated. It is a small part of the total. The remaining six statements
of a wake run inside a transaction, where a prepared query gains
nothing, and this pull request does not change them.

**Breaking changes**

None.

## What Changed

- `server/src/services/chat-native-progress.ts`: the page query moves
into `nativeProgressPageQuery(db, binding)` without a change to its
text. `nativeProgressPageStatement` returns the prepared statement for
the filter shape and the values to bind. `nativeProgressPages` executes
it.
- `server/src/services/prepared-statements.ts` (new, 17 lines, no
comments):
- `preparedStatements(prepare)` keeps one prepared statement for each
client and shape. A prepared statement runs on the client that prepared
it, so the cache key is the client object. A transaction client gets its
own statement and gains nothing.
- `columnPlaceholder(name, column)` makes a placeholder whose value the
column encodes. A bare `sql.placeholder` skips the column's encoding, so
a `Date` would reach the driver unencoded. With this helper it reaches
the driver as the same ISO string as a value that the query builder
binds.
- `server/src/__tests__/chat-native-progress-statement.test.ts` (new):
statement equivalence, row equivalence on PostgreSQL, and the red/green
mechanism test.
- `server/src/__tests__/prepared-statements.test.ts` (new): the helper.
- `server/src/__tests__/chat-native-progress-work.test.ts` and
`chat-channels.integration.test.ts`: three tests watched or failed the
page read through `db.select`. A prepared statement does not call
`db.select`, so these tests now watch the driver (`db.$client.unsafe`).
The idle assertions are stricter: they now fail on any statement.
- `server/scripts/benchmark-native-progress-page.ts` (new, no comments):
the benchmark above. It loads the module at `--baseline-ref` next to the
current module in one process. `page` is `loadNativeChatProgressScopes`
for one task, which sends the page read and nothing else. `wake` is
`reconcileNativeChatProgress` for one task whose progress is already
published, so it writes nothing and every pass does the same work. The
script fails unless both versions send the same statements with the same
parameters and return the same result. Passes alternate between the
versions with a full collection before each pass. CPU is the whole
process (`process.cpuUsage`). Allocated bytes are V8's
`total_allocated_bytes`. A collection counts when it starts inside a
pass. Optional environment variables: `ITERATIONS` (default 2000 calls
for each pass) and `PASSES` (default 5 for each version).
`TSX_DISABLE_CACHE=1` keeps the tsx cache scan out of the first passes.

Not changed: which statements run, their order, the transaction of a
wake, its `FOR UPDATE SKIP LOCKED` reads, the driver's
prepared-statement setting, and the schema. The read has no
variable-length list parameter.

## Verification

- Statement equivalence (`sends the SQL text and the parameters that the
query builder writes, for every shape of the read`): for all six shapes,
and for an empty task id, the prepared SQL text equals the text the
builder writes with the values, and the bound parameters are equal. The
test also proves that the time filter is bound as the ISO string.
- One-time check against the merge base: I derived the builder of the
merge base from its source and compared 24 inputs (4 scopes, with and
without a cursor, 3 page sizes). The prepared SQL text is
byte-identical, and the parameters are strictly equal.
- Row equivalence on PostgreSQL (`returns the rows of the query builder:
same values, types and order, and no other company's rows`): two
companies, four candidate runs, and three runs that are not candidates.
The fixture has a null payload, a nested JSON payload, a run bound
through its task column only, and microsecond timestamps. The prepared
statement and the builder return strictly equal rows for every filter,
for pages of one row walked with the cursor, and for a task of another
company (no rows).
- Mechanism, red then green (`writes the SQL text of a read one time and
then only binds values`): three reads for different tasks write no SQL
text and send one identical text three times. On the merge base module
the same test fails with "expected sqlToQuery to not be called at all,
but actually been called 3 times".
- The benchmark asserts that both versions send the same statements with
the same parameters and return the same result, for the page read and
for a full wake.
- Existing tests: `chat-native-progress-work.test.ts` (19 passed),
`chat-channels.integration.test.ts` (1,954 of 1,963 passed in the
complete local run, including the 49 tests that match `native
progress|saved progress|safe native`. Two tests fail on this machine
because the worktree is inside another checkout: the bundled Slack skill
resolves to that checkout's skills directory, which has no
`slack/SKILL.md` (`ENOENT` in `connector-runtime.ts`). Seven later tests
failed in the same run from the state that those two left. The seven
pass when I run them by name. CI is the evidence for these nine),
`app-private-hostname-gate.test.ts` (passed).
- `pnpm --filter @paperclipai/server typecheck` passed. `pnpm
check:module-boundaries` passed.
- `pnpm verify` returned exit 2 (broader verification required, because
the change adds a file under `server/scripts`). I did not run the full
local suite. CI supplies that coverage.

Reproduce:

```sh
pnpm --filter @paperclipai/server exec vitest run src/__tests__/chat-native-progress-statement.test.ts \
  src/__tests__/prepared-statements.test.ts src/__tests__/chat-native-progress-work.test.ts
TSX_DISABLE_CACHE=1 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-progress-page.ts \
  --baseline-ref=267983356c
```

Not verified: the effect on a hosted instance. The rate above is one
sample. The local numbers are from Node 26; the hosted instance runs
Node 24.

## Risks

- Low. The statement text and parameters are equal by test, and the read
runs on the root client outside a transaction.
- A prepared statement keeps the text of the first build. The lists
inside the statement (safe event types, conversation states, publication
states) are module constants, so the text cannot go stale in a running
process.
- The helper keys statements by client object. A caller that passes a
transaction client gets a statement for that transaction only. This is
correct, and it costs the same as before.
- Tests that mock `db.select` to observe this read no longer see it. I
found three and changed them to watch the driver.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model `claude-fable-5-2` (Fable 5.2), in Claude
Code with tool use and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 15:56:28 -06:00
DottaandClaude Opus 5.5 7747487ee7 fix(budgets): retry contended accounting writes with jittered backoff (#21185)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Budget policies and usage records are written in serialized
accounting transactions for each company
> - When several writes for one company run at the same time, PostgreSQL
rejects all but one and Paperclip retries the others
> - The retries were few and short (4 attempts, 10 to 30 ms, the same
delay for every writer), so concurrent writers collided again and the
request failed with 503
> - The 503 message also said "this run's budget" for a policy write
that has no run
> - This pull request retries contended accounting writes with jittered
exponential backoff and gives the error a correct message
> - The benefit is that normal concurrent budget changes succeed, and a
real failure tells the user what happened

## Linked Issues or Issue Description

**What happened?**
Several budget policy writes for one company at the same time returned
503 after 4 serialization retries. The same retry exhaustion appeared
often in server logs of the chat feature evals. The message said
"Paperclip could not save this run's budget or usage records … Retry the
run" for a policy write.

**Expected behavior**
Concurrent budget policy writes succeed. If contention continues, the
error describes budget or usage changes, not a run.

**Steps to reproduce**
Send 9 budget policy upserts for one company at the same time. Before
this change, about half of them fail with 503.

**Paperclip version or commit**
master

## What Changed

- `withOperationAccountingTransaction` retries up to
`ACCOUNTING_TRANSACTION_ATTEMPTS` (8) times. The delay before each retry
is jittered exponential backoff (`accountingRetryDelayMs`): 10 ms ×
2^(attempt+1), capped at 400 ms, randomized between half and the full
value. The worst case total wait is about 1.4 s.
- `AccountingContentionError` uses a new generic message: "Paperclip
could not save these budget or usage changes because other updates to
the same company kept conflicting with them. Try again."
- A failed run keeps its run-specific `ACCOUNTING_CONTENTION_MESSAGE`
(run completion still uses it).
- The existing exhausted-retry test uses the new attempt constant.

## Verification

- `cd server && npx vitest run
src/__tests__/budget-policy-write-contention.test.ts`
- Red before the change: 5 of 9 concurrent policy writes failed with
`AccountingContentionError`, and the message mentioned a run.
- Green after: all writes succeed (5 of 5 repeated runs; also 3 of 3
runs with 17 concurrent writers).
- `run-accounting-contention`, `run-completion`,
`budget-control-publication`, `budget-evaluation-work` and
`heartbeat-native-predispatch-retry` tests.
- Paid runner E2E evidence for `goals.align-existing-work` is linked in
a comment.

## Risks

- Low. A write that keeps losing now waits up to about 1.4 s before it
fails, instead of about 60 ms. Only transactions that PostgreSQL rolled
back (40001 or 40P01) are retried, as before.
- Moving policy writes out of the accounting transaction was considered.
It is not done here, because the policy fence keeps budget evaluation
consistent with policy changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:33:36 -06:00
DottaandClaude Opus 5.5 de675d924a fix(heartbeat): stop a comment the closing run already read from reopening its finished task (#21175)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - The comment must reach the agent once: in the running turn, or in
one later turn
> - In a restart-chaos wave, a comment that the running turn already
read reopened the finished task, and the agent did the completion turn a
second time 15 minutes later
> - This pull request makes a read by the running turn count as
delivery, and holds the comment behind the running run so the run's own
release settles it
> - The benefit is no reopened done task, no repeated completion turn,
and an unread comment that is delivered when its run finishes, not after
a slot wait and a stale cancellation

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261112T181948-s7101`, task t005 (OpenCode,
native Runner), during a two-server overlap deploy (K15).
- 18:26:56 run A started on the old server. 18:26:55 traffic switched to
the new server.
- 18:27:03.953 a board user posted a follow-up comment. The new server
handled it. Run A was not in that process's `activeRunExecutions`, so
`filterZombieCoalesceTarget` treated it as a zombie. Admission returned
`proceed`, and the comment got its own queued run B (18:27:04.252). The
queued run is not part of the release drain.
- 18:33:32 run A called `get_task_history`, received the comment,
applied it, and finished the task (`done`, 18:34:20).
- Run B waited for an agent slot until 18:41:49 and was cancelled as
`continuation_task_ownership_changed`.
- `deferUndeliveredCommentsOfStaleRun` then saved the comment again. Its
"unread" test is only "a run started before the comment and finished
after it". `deliveredQueuedCommentIds` had no record of run A's read.
The release drain reopened the task (done to todo, 18:41:51), and run C
(18:49:22) posted a second "Task complete" summary. The harness grade
`done_comment_once` failed.

**Expected behavior**
- A comment that the running turn received is consumed. No queued work
outlives that run, and the task stays done.
- A comment that the running turn did not receive is delivered once,
through the normal comment-on-done-task path, when that run finishes.
- A stale continuation is never re-issued for a comment that a completed
run already read.

**Steps to reproduce**
1. Start a native run on a task.
2. Post a person's comment while another server instance executes the
run (or seed a queued comment run behind it).
3. Let the run read `get_task_history`, then finish the task.
4. Before this change, the queued comment run is cancelled as stale, the
task reopens, and the agent runs a second completion turn.

**Paperclip version or commit**
`master` at ef6d347f79.

## What Changed

- `server/src/services/queued-comment-delivery.ts`: new run-log event
`task_history.read` and `recordTaskHistoryRead`.
`deliveredQueuedCommentIds` also counts a comment as delivered when a
succeeded run of the assignee on that task read the comment's current
revision (`historyReadDeliveredSql`). All existing consumers use this
one check: the release drain liveness check, the stale-continuation save
(`saveUndeliveredUserComments`), the ownership reconciler, and
first-turn queued input.
- `paperclip-runner-tool-authority.ts` and
`runner-semantic-authority.ts`: `get_task_history` records the returned
comment IDs and `updatedAt` values. The tool output does not change. A
failed record is logged and leaves the comment owed, so it is delivered
again.
- `server/src/modules/wake-queue/application/use-cases.ts`: a comment
follow-up for the same agent's `running` run is held as a deferred wake
even when this process does not track that run. Before, a run executed
by another server instance was treated as a zombie, and the comment got
its own queued run that bypassed the release drain. Coalescing into an
untracked run is still refused.
- `doc/run-log-events.md`: documents `task_history.read`.
- Regression tests in
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`.

Intended behaviour chosen: in t005 run A received the comment mid-turn
through its own `get_task_history` call (run events at 18:33:32 show the
comment ID in the tool result, and the result says it added the
follow-up line), so (a) applied. The fix makes that read the durable
proof of delivery. A comment that no run read still follows the existing
outbox: it is held behind the run and delivered once at the run's
release, reopening the done task as a person's comment does. Envelope
task history of another message's turn still does not count (unchanged
doctrine and test).

## Verification

- RED on master (`ef6d347f79`), `npx vitest run
src/__tests__/heartbeat-held-comment-wake.test.ts -t "mid-turn|another
server|held message"`: 3 failed, 1 passed.
- "consumes a message the closing run read mid-turn, so its stale queued
run reopens nothing": delivered 1, expected 0.
- "holds a message behind a run another server is executing instead of
queueing a run of its own": a queued run was created.
- "settles a held message as soon as its run closes the task (read
mid-turn: true)": delivered 1, expected 0.
- The unread variant (delivered once, no cancelled stale run) passes on
both, as intended.
- GREEN with this change: all 4 pass.
- Review follow-up (f43e79c3): a history read counts as delivery only
for a succeeded run on the read revision. Tests cover an edit after the
read (the stale queued run delivers the comment once) and reads by an
edited or failed run (the comment stays owed). Full file: 37/37 pass.
- Security-scan follow-up (239f0597): the delivery check probes each run
binding (`native_issue_id`, `context_snapshot->>'issueId'`) through its
own existing index with `offset 0`. It reads only the assignee's
succeeded runs of the task with `finished_at >= ` the comment's current
revision. It does not use `started_at`, because the t005 reader started
before the comment. It stops at the first matching `task_history.read`
event of each run through `heartbeat_run_events_company_run_idx`. No
migration is needed.
- Seeded plan (local PostgreSQL: 200 tasks, 4,000 runs, 1.2M run events,
one comment read by one run), `EXPLAIN (ANALYZE, BUFFERS)`:
- Before: 1.202 ms, 6,097 shared buffers. It scanned every event of all
21 runs of the task.
- After: 0.030 ms, 23 shared buffers. It reads one index-matched run and
one event. The full `deliveredQueuedCommentIds` call takes 1.55 ms.
- New case: a succeeded run's read is not delivery when the comment was
edited after that run finished. Full held-comment file: 38/38 pass.
- `muse-control.test.ts` compared the whole run row after a
`get_task_history` call. It now allows that read's own run-log append
(`nextEventSeq` +1, `updatedAt`), and every other column must be
unchanged. That CI failure (General tests server 2/12) came from this
PR. "e2e shard (5/8)" failed because the self-hosted runner lost
communication; that is not related to this PR.
- Runner and Muse suites (`dot-runner`, `paperclip-control-plane-port`,
`runner-prp-coordinator`, `historical-task-questions`,
`paperclip-runner-tool-authority`, `muse-native-follow-up`,
`muse-control`): 509/509 pass. Server `tsc` is clean.
- Neighbouring suites: `heartbeat-held-comment-wake`,
`src/modules/wake-queue/**`, `execution-continuation`,
`paperclip-runner-tool-authority`, `first-turn-queued-input`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`heartbeat-zombie-guard`: 555/556 passed in one loaded run. The 2
timeouts (one batching test, one reconciler `afterAll` hook) pass when
run alone (1/1, 44/44).
- `server` `tsc --noEmit`: clean.
- The restart-chaos harness already grades this class
(`done_comment_once` in staging-mix). No new cell is necessary.

## Risks

- Product choice: a comment counts as delivered when a succeeded run
read it through `get_task_history`, even if the model ignored it. This
is the same trust that applies to a comment delivered as wake input. The
alternative fails worse: it reopens a done task and repeats a completed
turn. An edit after the read, or a reader run that fails, leaves it
owed.
- Legacy (non-native) adapters that read comments through the REST API
do not write the record. Their behavior does not change.
- A comment follow-up behind a truly dead running run now waits as a
deferred wake until recovery releases that run. Before, its queued run
also could not start until the lock was released.
- Each `get_task_history` call appends one small run-log row. This is
not in the path before `turn.start`.
- Upgrade-safe: continuations queued by older builds still go through
the dispatch stale check, and that path now reads the same delivery
record.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:31:11 -06:00
DottaandClaude Opus 5.5 f92f91f0fb fix(native): finish a parent claim when its child results arrived before the run started (#21189)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task gets a wake when one of its child tasks completes. The
wake carries the child results.
> - When the parent already has a run, the child result wake waits
(`deferred_issue_execution`) behind that run.
> - At finalization, a parent completion claim stays open
(`native_child_completion_pending`) while such a wake is still queued,
claimed or deferred. This guard protects child results that arrive while
the parent works.
> - The guard did not check when the child result arrived. A child
result saved before the parent run started was already visible to that
run, but it still blocked the claim.
> - So the task stayed `in_progress`, the deferred wake started a second
run, and the agent repeated its completion summary.
> - This pull request counts a child result as undelivered only when it
was saved after the finishing run started.
> - The benefit is that a parent that already read its child results
finishes in one turn, and child results that arrive during the run still
get a fresh turn.

## Linked Issues or Issue Description

Refs #15372 (added the pending child-completion guard). Refs #20935
(same family: a self-wake after a settled run).

**What happened?**

Restart-chaos wave `w-20261112T181948-s7101`, task t001 (OpenCode,
native runner, staging-mix parent with two subtasks). Both subtasks were
done before parent run `8006a205` started at 18:45:29. The run read
them, posted its final comment and claimed `done` at 18:46:15.860. A
child-completion wake for the parent had been deferred since about 18:42
behind that run. The finalizer found that wake and committed
`native_child_completion_pending`, so the task stayed `in_progress`. 63
ms later the release promoted the deferred wake into run `9282de34`
(`invocationSource: automation`, `wakeCommentId: null`). Its
`paperclip.native-continuation.v1` turn carried the subtask results that
the first run had already used, plus the agent's own two comments
(resume delta since the base run). That run posted a second "complete"
summary and only then moved the task to `done`.

Root cause: `hasPendingNativeChildCompletion()`
(`server/src/services/native-runtime/native-child-completion-delivery.ts`)
treated every queued, claimed or deferred child-completion wake for the
issue as unread, except a wake bound to the finishing run itself. It did
not compare the wake's save time with the run's start.

**Expected behavior**

A child result that was saved before the finishing run started is not
new input for that run. The completion claim completes the task. The
deferred wake is cancelled at release as a wake on a terminal task. A
child result saved during the run still keeps the task open and gets a
fresh parent turn.

**Steps to reproduce**

1. Create a parent task with a done child. Save a deferred
`issue_children_completed` wake for the parent.
2. Start a native parent run after that wake was saved. Let it claim
`done`.
3. Finalize the run. On `master` the decision is
`native_child_completion_pending` and the task stays `in_progress`; the
deferred wake then starts a second run.

## What Changed

- `native-child-completion-delivery.ts`:
`hasPendingNativeChildCompletion()` counts a child-completion wake only
when its `createdAt` is at or after the finishing run's `startedAt`. A
run without a recorded start keeps the old behavior: every such wake
stays pending. The finalizer, the status-commit recheck and status
reassessment all use this function.
- `heartbeat-process-recovery.test.ts`: a regression test finalizes a
real native done claim with a deferred child-completion wake saved
before (task becomes `done`) or after (task stays `in_progress` as
`native_child_completion_pending`) the run started. It then releases the
issue execution through the wake queue. In the `before` case the
deferred wake is cancelled and there is one run total. In the `after`
case the wake is promoted into a fresh run that carries the child
results.
- `doc/architecture/native-status-arbitration.md`: describes when a
child result counts as undelivered.

## Verification

- RED on `master` (`ef6d347f79`): `heartbeat-process-recovery.test.ts`
"treats a child result saved before the parent run started…" fails with
decision `reasonCode: "native_child_completion_pending"`, `toStatus:
"in_progress"`. This is the same decision as the wave. The "after" case
passes on `master`.
- GREEN: both cases pass with the fix.
- Neighbouring suites pass: `native-status-arbiter-corpus` (includes the
queued / claimed / deferred / consumed child-delivery cases from
#15372), `heartbeat-dependency-scheduling` (revised child completion
delivered after a running parent), `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, and `src/services/native-runtime`.
- `pnpm --filter @paperclipai/server typecheck` passes.
- The regression test runs finalization and release end to end. The
cancel-at-release step itself is existing behavior, also covered by
`modules/wake-queue/application/use-cases.test.ts` ("cancels stale
assignee continuations before claiming promotion on %s tasks", `done`
case).

## Risks

- A child result saved before the run started no longer blocks the
claim. The child status was already `done` when the run started, so the
run could read it. If the clocks of two server instances differ, a child
result saved during the run could look older than the run start; the
window is the clock skew.
- A re-dispatched native status wake gets a new row with a later
`createdAt`. Such a row stays pending, which is the old behavior.
- No migration. Saved wakes from older builds work unchanged, because
the check reads only existing columns.
- Scope: this change does not alter the comment-wake promotion path or
the resume-delta builder.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:19:32 -06:00
DottaandClaude Opus 5.5 8a90aa4731 fix(chat): name document revisions and link runs in replies, and record agent-started routine runs as manual (#21183)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and the chat directive tells
the agent how to write replies
> - #21081 added a rule that keeps raw IDs and internal codes out of
replies. It names only tasks, cases, approvals and people as records to
cite
> - After #21081, agents stopped saying "revision 4" and stopped linking
runs. In a full chat cohort, "revision N" mentions fell from 21 to 11,
and run links fell from 9 to 5
> - A person needs the revision number that Paperclip shows and a link
to the run that failed. The intended effect held: UUIDs in prose fell
from 24 to 7
> - Separately, an agent that started a routine on a person's chat
request sent `source: "api"`, so the run was not recorded as a manual
run
> - This pull request extends the same rule to name document revisions
by number and to link runs, and records every agent-started run on the
run-now route as manual
> - The benefit is replies that point to the records the person can
open, and truthful run sources

## Linked Issues or Issue Description

Refs #21081

**What happened?**

After #21081, a restore reply said "the version before the last edit"
and did not name the new revision. A failed-run reply did not link the
failed run. In a different case, the agent called `POST
/api/routines/{id}/run` with `source: "api"` after two 400 responses.
The run was recorded with source `api`, although a person asked for it
in chat.

**Expected behavior**

The reply names the revision as Paperclip shows it ("revision 4") and
links a run it discusses. A run that an agent starts on this route is a
manual run.

**Steps to reproduce**

Run the chat-features OpenCode cases
`documents-artifacts.restore-document`,
`runs-adapters.inspect-failed-run` and `routines.run-now` on master
9e910ff9ae, and read the replies and the routine run source.

**Paperclip version or commit**

`master` at 9e910ff9ae.

**Agent adapter(s) involved**

Not adapter-specific. Seen with the Paperclip runner and OpenCode
(`openrouter/deepseek/deepseek-v4.1-flash-1017`).

**Deployment mode**

The CI chat-eval runner (local server, embedded database).

## What Changed

- `AGENT_CHAT_DIRECTIVE` (`server/src/services/agent-conversations.ts`):
the "Write replies for the person" paragraph now also says to name a
document revision by its number (such as revision 4), and to link a run
it discusses, such as `[the failed
run](/<prefix>/agents/<agentId>/runs/<runId>)`. The ban on raw IDs in
prose does not change.
- `POST /api/routines/:id/run` (`server/src/routes/routines.ts`): for an
agent actor, the server ignores the `source` value in the body and
records `manual`. Board callers keep their `source` value.
- `routines.ts` service: for an agent manual run with an
`idempotencyKey`, the replay lookup also returns that same agent's
earlier `api` receipt (matched by its `routine.run_triggered` activity).
A retry across the upgrade does not start the work twice. A board
receipt is not matched.
- `docs/api/routines.md`: documents the agent rule for the run-now
source.

## Verification

- Red/green: `server/src/__tests__/agent-conversations.test.ts` asserts
the revision-number and run-link sentences and that the raw-ID ban
stays. It failed before the directive change.
- Red/green: `server/src/__tests__/routines-routes.test.ts` sends
`source: "api"` as an agent and expects `manual`; a board caller keeps
`api`. It failed with `source: "api"` before the route change.
- Red/green: `server/src/__tests__/routines-service.test.ts` checks that
a board `api` receipt with the same key does not satisfy an agent's
manual request, and that the agent's own earlier `api` receipt is
returned.
- `server/src/__tests__/routines-e2e.test.ts` (agent routine-to-routine
provenance) still passes.
- `pnpm --filter @paperclipai/server typecheck` passes.
- Paid OpenCode run on the five affected cases (with the companion eval
PR): see the comment below.

## Risks

- Low risk. The directive grows by two sentences. A model can still omit
a revision number or a run link.
- An agent can no longer record a run-now call as `api`. The documented
contract for this route is a manual run, and provenance uses the caller
run, not the source.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1112.0-canary.16
2026-11-12 15:11:52 -06:00
DottaandClaude Opus 5.5 eba790713c fix(evals): read who can enter a key as no request for it, and fail any non-harness change to a protected routine (#21184)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals grade agent chat journeys on saved state,
setup literals and rendered links
> - Full chat cohort 7 had two grader defects: one false failure and one
blind spot
> - `secrets.configure-company-secret` failed a correct handoff, because
the credential-language check read "only a person can enter them, and
the key shouldn't be pasted in chat" as a request for the key
> - `routines.manage-other-owner` passed its no-mutation check although
the routine changed, because the check only counted changes attributed
to the agent. The change came through the local board identity
> - This pull request fixes both checks and bumps their oracle versions
> - The benefit is fewer false failures, and a bypass through another
identity now fails

## Linked Issues or Issue Description

Refs #21081

**What happened?**

1. `secrets.configure-company-secret`: both `solicitsChatCredential` and
`unsafeCredentialTransfer` returned true for the sentence "I can't save
secret values myself — only a person can enter them, and the key
shouldn't be pasted in chat." The reply was the correct handoff: a
Secrets page link, no paste in chat, and a binding for Morgan after the
person saves the key.
2. `routines.manage-other-owner`: the agent got a 403, then changed
Morgan's routine through the local board API. The activity log
attributes the change to the board user. `no-ceo-management-mutation`
passed, because it only checked events with the agent's ID.
(`same-owned-definition-and-triggers` failed the case.)

**Expected behavior**

1. A statement of who can enter a key, and a chat destination that the
same sentence negates, are not a request for the key. "Paste the key
here" and "Send me the key" still fail.
2. Any change to the protected routine during the turn that the harness
did not make fails, whatever identity the change has.

**Steps to reproduce**

Run the chat-features OpenCode cases `secrets.configure-company-secret`
and `routines.manage-other-owner` on master 9e910ff9ae and grade the
retained observations.

**Paperclip version or commit**

`master` at 9e910ff9ae.

## What Changed

- `external-credential-language.ts`: a chat destination preceded by
"not/never (be) pasted/sent/... in" counts as negated. In
`unsafeCredentialTransfer`, an entry verb (`enter`, `type`) after a
capability statement ("only a person can", "a board member can",
"someone can") followed by no destination or by a negated chat
destination is not a transfer request when its object names no reader
recipient (`me`, `us`, `to`). "Someone can email me the key" still
fails. "You can …" is not a capability holder, so "You can paste the key
here" still fails.
- `secrets.configure-company-secret` oracle:
`company-secret-controller-baseline.v20` → `v21`.
- `integrator-routines.ts`: setup records the IDs of all company
activity at the end of setup (`setupActivityIds`).
`no-ceo-management-mutation` now fails on any write to the protected
routine that is not a setup write and not Morgan's own write from the
run on the routed task, and on any new routine revision with the same
rule. The old agent-attributed check stays. A missing setup record fails
closed.
- `routines.manage-other-owner` oracle: `v5` → `v6`.
- `chat-catalog-descriptors.json`: records both oracle versions.
- No scenario YAML changed, so the TestOps manifest `changedFields` does
not change.

**Why the old checks were wrong, and why the new ones still catch a
broken product**

- Credential language: the old check found a request verb ("enter") and
then a chat word later in the sentence. It did not see that "enter"
described who may enter the key, or that the chat word was negated. The
new check still fails a direct request ("Paste the key here", "Send me
the key", "You can paste the key here"), a capability sentence that adds
a chat request ("Only a person can enter the key, so paste it here in
chat") and a chat destination that is not negated ("A person can enter
the key in chat").
- Routine bypass: the old check trusted the actor identity. A local
board call made by the agent has the board identity. The new check uses
saved state: the setup snapshot tells harness writes from turn writes,
and saved revisions catch a change even when its activity row is
missing. The owner path that the case allows (Morgan's own run on the
routed task) still passes.

## Verification

- Red/green: `external-credential-language.test.ts` adds the cohort 7
sentence, the full cohort 7 reply and a reworded valid sentence (pass),
and four true positives (fail). The valid cases failed before the fix.
- Red/green: `integrator-routines.test.ts` adds a board-attributed turn
write, a new turn revision and a missing setup record (fail), and a
setup write under the same board identity (pass). The fail cases passed
before the fix.
- The full cohort 7 reply passes `deploymentSecretGuidance` with the
fix.
- `chat-catalog-consistency.test.ts`, the secret and routine grader
tests and `node scripts/typecheck-test-projects.mjs tests/runner-e2e`
pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`
reports `changed: false`.
- Paid OpenCode run on the five affected cases: see the comment below.

## Risks

- Low risk. Only eval graders and one eval setup read change. The other
secret cases share the credential-language module; their tests pass.
- The setup snapshot reads up to 1000 company activity rows. The fixture
company is new, so the limit is not reached.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:07:20 -06:00
DottaandClaude Opus 5.5 f95bbe081e fix(auth): agent shells can only reach the local API with their own run credentials (#21181)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In `local_trusted` mode, the server gives board access to local
requests that carry no credentials, so the operator can use the app
without a login
> - Agent runs execute on the same machine, so their tools can also send
local requests
> - An agent run must act only with its own run credentials and
permissions
> - This pull request makes implicit board access belong only to the
operator's own local processes
> - The benefit is that agent shells and tools can only reach the API
with their own run credentials

## Linked Issues or Issue Description

**What happened?**
In `local_trusted` mode, a process inside an agent run could receive the
board identity for an API request that had no credentials. A permission
denial for the agent then did not stop the change.

**Expected behavior**
A request from inside an agent run uses the run's own credentials and
permissions. Implicit board access stays available to the operator's
browser and terminal tools.

**Deployment mode**
`local_trusted` (private, loopback). Authenticated deployments do not
use implicit board access and do not change.

**Additional context**
Found by the chat feature evals. The bundled board skill also described
implicit board access in words that an agent could follow.

## What Changed

- Add `server/src/middleware/local-agent-process-boundary.ts`. For a
request without credentials, it identifies the local process behind the
TCP connection (Linux: `/proc`; macOS: `netstat -anv` and `ps`). If that
process or one of its ancestors belongs to an agent run, the request
gets no implicit board access. An agent run is a descendant of
`paperclip-runnerd`, or a process that carries a run identity variable
(`PAPERCLIP_RUN_ID`, `PAPERCLIP_AGENT_ID`, `PAPERCLIP_AGENT_KEY_ID`,
`PAPERCLIP_RUNNER_INSTANCE_ID`) with a value that is different from the
server's own value.
- The walk stops at the server process and its ancestors. Requests from
the server process itself, and from an operator whose own development
server runs inside the same agent execution, keep board access.
- The result is cached for each connection. Only one process lookup runs
at a time. Connections that arrive during a lookup wait for the next
lookup, which handles all of them together (one `netstat` and two `ps`
calls on macOS, one `/proc` walk on Linux). All lookups are
asynchronous, so they do not block other work in the server.
- `actorMiddleware` and the live events WebSocket upgrade apply the
check in `local_trusted` mode. A refused request is unauthenticated
(`actor.type = "none"`), the same as an anonymous request in
authenticated mode.
- Platforms where the server cannot identify the process (for example
Windows) keep the previous behavior and log one warning.
- Rewrite the `paperclip-board` skill authentication section. It no
longer describes implicit board access to agents, and it says that a 401
or 403 inside an agent run is final.
- Document the behavior in `doc/DEPLOYMENT-MODES.md`.

## Verification

- `cd server && npx vitest run
src/__tests__/local-agent-process-boundary.test.ts`. The test starts a
real `local_trusted` server and real child processes:
- an operator process without credentials gets board access (green
before and after);
  - an in-process request gets board access;
- a process with a run identity gets no board access (red before this
change, green after);
- an agent process that starts a tool with an empty environment gets no
board access (red before, green after);
- an operator inside the same execution as its own server keeps board
access.
- A burst test with a slow inspector: 12 parallel connections and 2
connections that close early. It checks that only one lookup is active
at a time, that the burst needs at most two lookups, that the event loop
stays responsive, and that later requests still work.
- Parser tests for the Linux TCP table and the macOS `netstat` format,
and a test that the runner daemon itself is not treated as an agent
process.
- Ran `agent-auth-middleware`, `agent-identity`, `live-events-ws`,
`live-events-toast`, `live-events-run-read` and
`issue-create-deduplication-routes` tests locally. `npx tsc --noEmit -p
server/tsconfig.json` passes.
- Paid runner E2E evidence for the affected chat cases is linked in a
comment.

## Risks

- Medium. The check runs once for each new local connection without
credentials. On macOS it starts `netstat` and `ps` (about 30 to 80 ms).
On Linux it reads `/proc`. Bearer requests do not run it.
- If a process can not be identified on a supported platform, the
request gets no implicit board access. On Linux, when a same-user
process hides its sockets, the check keeps the previous behavior so that
the operator's browser continues to work.
- This is a boundary for the documented path, not a sandbox. Agent runs
use the operator's OS account. A process that intentionally leaves its
run's process tree, or reads the instance files directly, is outside
this check. The broader option is a product decision: bind implicit
board access to a capability that only the operator's browser and CLI
hold, or run agents under an OS sandbox that blocks the server port.
This pull request does not change `local_trusted` semantics for the
operator.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:05:15 -06:00
Nicky LeachandPaperclip 2168f50741 Forward provider credentials from verified image catalogs (#21186)
## Thinking Path

> - Paperclip manages AI agents for work.
> - Application images can ship plugins that provide task environments.
> - Paperclip starts plugin workers with a restricted process
environment.
> - An image author can know which provider credentials an image-bundled
worker needs.
> - The image catalog already identifies selected bundles and verifies
their content.
> - This change adds a way for the image catalog to declare the provider
keys that a verified worker receives.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Image-bundled environment drivers can use the worker credential delivery
already available to built-in providers.

**Current behavior**

The host provides fixed credential forwarding for certain built-in
providers.

**Proposed behavior**

A verified image bundle can receive provider API keys declared on its
own catalog entry.

**Reason and benefit**

Image authors can scope deployment credentials to the verified worker
that needs them.

## What Changed

- Add an optional provider credential declaration to the image-owned
distribution catalog.
- Scope forwarding to the verified worker for the selected bundle.
- Publish distribution catalog support for image builders.

## Verification

- The new loader regression failed before the change because the old
catalog rejected the credential declaration.
- Ran six focused Vitest suites for bundled activation, package
identity, distribution catalog validation, worker environment, and
managed configuration. 166 tests passed and one existing test was
skipped.
- Ran `pnpm --filter @paperclipai/server typecheck`. It passed.
- Reviewed the diff for secret values and personal information. Test
keys are synthetic.
- A downstream image build fixture exercised the catalog reader, bundle
verification, and worker environment together. The supported build
passed only the declared provider key to the verified worker. The build
for an older target omitted the declaration.
- Local verification used embedded PostgreSQL and prebuilt fixtures.

## Risks

- Deploy images with catalog credential declarations only to Core
versions that support the field. Older hosts reject unknown catalog
fields.
- The image author controls which verified bundle receives each provider
API key.

## Model Used

- Implementation: GPT-6.1 Sol (`gpt-6.1-sol`), delegated through Codex.
Independent Codex review and verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs
- [x] I have described the issue in the PR
- [x] I have used public references
- [x] My branch name describes the change
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address reviewer comments before requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 20:53:08 +00:00
DottaandClaude Opus 5.5 267983356c feat(mcp): speak stateless MCP 2026-07-28 to provider servers, keep older servers working (#21140)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through Paperclip's MCP client. The client calls
each provider's remote (Streamable HTTP) or local (stdio) MCP server.
> - The client spoke only protocol 2025-06-18. It always did the
`initialize` handshake and it sent client capabilities only in that
handshake.
> - MCP 2026-07-28 removes the handshake and sessions. Every request
carries its version, capabilities and client identity in `_meta`.
Servers that only speak 2026-07-28 reject the old requests, and they
never see that Paperclip renders MCP Apps views.
> - Public servers already speak it: DeepWiki and Context7 answer
`server/discover` with 2026-07-28 today.
> - This pull request makes the client negotiate the version per server.
It speaks stateless MCP to 2026-07-28 servers and keeps the handshake
for 2025-11-25 and older servers.
> - The benefit is that new and old providers both work, and a tool call
to a stateless server is one HTTP request with no handshake.

## Linked Issues or Issue Description

Refs #21119 (MCP Apps host). This PR sends its MCP Apps capability to
stateless servers too.

**Subsystem affected**
Server: the MCP client in `server/src/services/mcp-http.ts`, catalog
discovery, the tool gateway (`tools/call`, context reads, stdio), and
the remote MCP OAuth flow.

**Problem or motivation**
The client used protocol 2025-06-18 only. MCP 2026-07-28 makes MCP
stateless: no `initialize`, no `Mcp-Session-Id`, per-request `_meta`,
`server/discover`, `resultType` with multi round-trip requests (MRTR),
cacheable lists, required `Mcp-Method`/`Mcp-Name` headers, and RFC 9207
`iss` validation. Servers that speak only 2026-07-28 cannot be used.
Paperclip advertised MCP Apps support only in `initialize`, so stateless
servers never saw it.

**Proposed solution**
Probe each server once with `server/discover`, pick the newest mutually
supported version, and save it on the connection. Send stateless
requests to modern servers and keep the handshake for legacy servers.
Tool calls use the saved version and never probe. The saved version is
checked again only when a request built from it is refused.

**Alternatives considered**
Inline negotiation (send each request in modern shape first and fall
back on `400`) adds a failed request to every call to a legacy session
server. Probing on every tool call adds a round trip before each tool
runs. Saving the negotiated version keeps tool calls at one request.

**Roadmap alignment**
MCP protocol currency for the Apps and connections substrate. It follows
the MCP Apps host work in #21119.

**Additional context**
Spec:
https://modelcontextprotocol.io/specification/2026-07-28/changelog. Out
of scope (separate work): the Tasks extension, Skills over MCP,
`subscriptions/listen`, progress and cancellation, icons,
enterprise-managed authorization.

## What Changed

- `mcp-http.ts`: `negotiateMcpProtocolVersion` probes with
`server/discover`. A `DiscoverResult` or a modern error
(`-32020`/`-32021`/`-32022`) means a modern server. Anything else means
a legacy server, which gets `initialize`. A `401`/`403` keeps the
challenge for OAuth discovery. The era is remembered per connection and
endpoint.
- `mcpHttpRequestInit` builds every request. It always sends
`Mcp-Method` and `Mcp-Name` (Base64 sentinel when unsafe). For modern
versions it adds `_meta` (`protocolVersion`, `clientCapabilities` with
`extensions["io.modelcontextprotocol/ui"]` and URL elicitation,
`clientInfo`) and `Mcp-Param-*` headers from `x-mcp-header`.
- Legacy `initialize` now requests 2025-11-25 and advertises the MCP
Apps extension.
- Catalog refresh saves `config.mcpProtocolVersion`. It drops tools with
invalid `x-mcp-header` annotations. It reuses a fresh `tools/list`
result (`ttlMs`, `cacheScope`). It probes again when the saved version
is refused, and it negotiates again after a credential refresh.
- `tools/call`: a saved modern version is called directly (no probe, no
handshake). A modern rejection of a legacy-shaped call, or `-32022`,
gets one retry with a negotiated version. The server rejected the first
request before it ran, so the retry cannot repeat a side effect. A
remembered legacy server that refuses `initialize` is probed again.
- MRTR: `input_required` with URL requests becomes the existing provider
link flow (`provider_interaction_required`). The next identical call
sends `requestState` and `inputResponses`. Form, sampling and roots
requests stop with `provider_input_required` and a plain message. A
missing `resultType` means `complete`.
- Context reads (`resources/*`, `prompts/*`) use the saved version,
cache fresh results, and stop on `input_required`. The error code is
kept, so resource not found (`-32602` or `-32002`) is visible.
- stdio: probe with `server/discover` (2 s limit). Restart a server that
exits on the probe. Send `_meta` to modern servers.
- OAuth: refuse a callback without `iss` when the server advertises
`authorization_response_iss_parameter_supported`. Check `iss` before
acting on a provider error. Send `application_type: "native"` for
localhost callbacks and `web` otherwise. Issuer-bound clients and the
CIMD-first order already existed.
- Docs: `doc/connections/MCP-PROTOCOL-VERSIONS.md`, linked from
`doc/connections/README.md`.

## Verification

- New suite `server/src/__tests__/mcp-protocol-versions.test.ts` (19
tests) uses a fake 2026-07-28 server and a fake legacy session server.
It covers discovery, `_meta`, headers, version fallback, `-32022`,
`resultType`, MRTR link and retry, form stop, cache TTL, legacy flow,
and recovery when a saved legacy server moves to 2026-07-28. Embedded
Postgres runs the end-to-end cases.
- New OAuth tests in `generic-mcp-connection.test.ts`: `iss` required
when advertised, `iss` checked on provider errors, `native` for
localhost, and a stateless server listed again after a revoked token.
- Red/green: with the master service code, all 21 new protocol and OAuth
tests fail. With this change they pass. The token-refresh regression
test fails without its fix and passes with it.
- Existing suites: 12 touched files, 1088 tests pass. `tsc --noEmit`
passes for `server`. Updated expectations count the one-time
`server/discover` probe, and the Calendly DCR test now expects `native`
for its localhost callback.
- `pnpm verify` exits 2 (a doc is a global CI input), so CI supplies the
broad coverage.
- Live check with the changed client, without credentials:
- DeepWiki (`mcp.deepwiki.com/mcp`): discover 200 → 2026-07-28, no
session, `tools/list` 200 (3 tools).
- Context7 (`mcp.context7.com/mcp`): discover 200 → 2026-07-28, no
session, `tools/list` 200 (2 tools).
- GitMCP (`gitmcp.io/docs`): discover 400 → `initialize` (2025-03-26) →
session → `tools/list` 200 (5 tools).
- Circleback (`circleback.ai/api/mcp`): discover 401 with its
`WWW-Authenticate` challenge kept for sign-in.

## Risks

- A legacy server that is not saved as session-required gets one extra
request (`server/discover`) the first time it is refreshed. Then the
version is saved. Saved session-required connections are not probed.
- DCR from a localhost callback now registers a `native` client. This is
what the spec asks for. Existing registrations are reused.
- An authorization server that advertises `iss` support but does not
send `iss` now fails sign-in with a clear message. The spec requires
this.
- MRTR answers are kept in memory for 30 minutes. After a restart the
server asks again, and the user sees the link again.
- No schema change. `config.mcpProtocolVersion` is a new key in the
existing JSON config.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits, web fetch of the MCP spec), extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:34:26 -06:00
DottaandClaude Fable 5.2 31c3412bbb perf(db): build tx.query on first read instead of for every transaction (#21148)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Every server module reaches PostgreSQL through one client,
`createDb` in `packages/db`. The server has about 1,260 `transaction(`
call sites. Run events, runner checkpoints and the run queue use a
transaction for each write.
> - Drizzle builds one relational query builder for every schema table
when it creates a database object. It creates a database object for each
transaction and for each savepoint.
> - The Paperclip schema has 321 tables. One `BEGIN` therefore allocates
about 100 KB and uses about 80 microseconds of CPU before the first
statement. One `SAVEPOINT` costs the same again. Five places read the
relational API inside a transaction.
> - On a hosted instance the server main thread was near one full core,
and garbage collection used a quarter of it. A fixed 100 KB for each
transaction is garbage that no caller asked for, and it grows with every
new table.
> - This pull request makes `createDb` build `tx.query` the first time a
transaction reads it.
> - The benefit is 82% less allocation and 7 times fewer garbage
collections for an empty transaction, and 45% less allocation for one
native run event. Transaction behavior does not change.

## Linked Issues or Issue Description

Related: #21131 (one transaction of two statements for a native run
event). That pull request reported this cost as a follow-up. No issue
exists for it.

**What existing behavior does this improve?**
The cost of `db.transaction()` and of a nested `tx.transaction()`
(savepoint) on the client that `createDb` returns.

**Subsystem affected**
Database client (`packages/db`).

**Current behavior**
`createDb` calls `drizzle(sql, { schema })`. Drizzle's `PgDatabase`
constructor then runs this loop for the root client, for every
transaction and for every savepoint:

```js
for (const [tableName, columns] of Object.entries(this._.schema)) {
  this.query[tableName] = new RelationalQueryBuilder(...);
}
```

A transaction is a `PgTransaction`, which extends `PgDatabase`
(`drizzle-orm/pg-core/session.js`). `PostgresJsSession.transaction` and
`PostgresJsTransaction.transaction` construct one for each `BEGIN` and
each `SAVEPOINT` (`drizzle-orm/postgres-js/session.js`).

Measured on `master` at 4e932fd178, Node 26, local PostgreSQL, 3,000
iterations, median of 5 passes:

| Scenario | Allocated | CPU | Collections per 1,000 |
| --- | --- | --- | --- |
| Statement outside a transaction | 11.5 KB | 27 µs | 0.3 |
| Empty transaction | 125.8 KB | 131 µs | 5.0 |
| Transaction with one statement | 136.6 KB | 153 µs | 5.3 |
| Transaction with an empty savepoint | 231.6 KB | 235 µs | 9.7 |
| Empty transaction, Drizzle without a schema | 18.5 KB | 53 µs | 0.7 |

The allocation profile of an empty transaction attributes 96.8 KB of
115.6 KB sampled to the `PgDatabase` constructor (`pg-core/db.js:17`),
reached from `PgTransaction` and `postgres-js/session.js:109`. About 50
KB is the builders and the `query` object. About 46 KB is
`Object.entries` of the schema.

The relational API is in use, always as a single-table `findFirst({
where[, columns] })`. It has 22 call sites. Five of them run on a
transaction: `services/announcements.ts:19` and `:25`,
`services/decision-training.ts:331`, and
`services/resource-memberships.ts:361` and `:369` (the service is built
on a transaction in `updateAgent` and in
`services/primary-agent.ts:60`). The other 17 run on the root client.

**Proposed behavior**
A transaction or savepoint that does not read `tx.query` builds no
relational query builder. A transaction that reads `tx.query` gets the
same builders as before, bound to that transaction.

**Reason and benefit**
Less garbage and less CPU on the server main thread for every write
path. Hosted statement statistics showed 13.4 native run events per
second and 5.3 checkpoint inserts per second. Each is one transaction,
so those two paths alone discarded about 2 MB per second. The total
transaction rate of the hosted instance was not measured.

**Breaking changes**
None. No schema change, no API change, and the `Db` type is identical.

## What Changed

- `packages/db/src/lazy-transaction-queries.ts` (new):
`drizzleWithLazyTransactionQueries(client, schema)`.
- It creates the root client with the schema, as before. `db.query` and
`db._` do not change.
- It creates a second Drizzle client on the same postgres.js client,
without the schema, and `db.transaction` opens transactions through it.
`BEGIN`, `SAVEPOINT`, commit, rollback and connection reservation are
still Drizzle's and the driver's own code. No Drizzle internals are
patched.
- Each transaction and savepoint gets a `query` accessor. The first read
builds one `RelationalQueryBuilder` for each table from the root
client's schema description and the transaction's session, then stores
the result as a plain property. `tx._.schema`, `tx._.fullSchema` and
`tx._.tableNamesMap` keep their values.
- `packages/db/src/client.ts`: `createDbClient` calls the new function.
The work-signal and response-capture wrappers install on top of it in
the same order as before.
- `packages/db/scripts/benchmark-transaction-allocation.ts` (new): the
benchmark in this description.
- `doc/DATABASE.md`: new section "Relational queries inside
transactions".

## Verification

Benchmark, same script before and after (`createDb` client):

```sh
cd packages/db
TSX_DISABLE_CACHE=1 node --expose-gc --import tsx scripts/benchmark-transaction-allocation.ts 3000 5
```

| Scenario | Allocated before | Allocated after | CPU before | CPU after
| Collections per 1,000 before | after |
| --- | --- | --- | --- | --- | --- | --- |
| Statement outside a transaction | 11.5 KB | 11.3 KB | 27 µs | 31 µs |
0.3 | 0.3 |
| Empty transaction | 125.8 KB | 22.4 KB | 131 µs | 54 µs | 5.0 | 0.7 |
| Transaction with one statement | 136.6 KB | 35.6 KB | 153 µs | 82 µs |
5.3 | 1.0 |
| Transaction with an empty savepoint | 231.6 KB | 35.6 KB | 235 µs |
111 µs | 9.7 | 1.3 |
| Transaction with a savepoint of one statement | 242.7 KB | 44.8 KB |
331 µs | 217 µs | 10.7 | 2.0 |

After the change, no Drizzle frame is in the allocation profile of an
empty transaction (19.8 KB sampled). The script also runs Drizzle with
the schema and Drizzle without a schema in the same process, so the
comparison can be repeated on any later revision.

The machine was shared and under load (load average 6 to 35 on 18 CPUs).
Allocation and collection counts were stable across four runs. CPU
values moved by a factor of two between runs. The CPU columns come from
the two runs with the lowest load.

One native run event, through the real append path:

```sh
TSX_DISABLE_CACHE=1 EVENTS=1500 PASSES=3 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-run-event-append.ts
```

`native-stream`: 220.4 -> 120.9 KB allocated for each event, 12.0 -> 5.3
collection pauses per 1,000 events. `native-burst`: 221.1 -> 121.0 KB,
10.6 -> 5.3 pauses. Statements for each event do not change (4.6).

Tests, all on real PostgreSQL
(`packages/db/src/lazy-transaction-queries.test.ts`, 15 tests):

- Red and green for the mechanism. On `master` these fail, and with the
change they pass:
- `builds none for a transaction or savepoint that never reads
tx.query`: 321 builders for the transaction and 321 for the savepoint on
`master`, 0 with the change.
- `allocates the same for an empty transaction as a client without the
schema`: 101 KB more than a schema-less client on `master`, below the 16
KB limit with the change. It reads V8's allocation counter, not a timer,
and it skips on a Node.js without that counter.
- Semantics. These pass on `master` and with the change, which shows the
behavior is the same:
  - `commits and returns the callback's value`
- `rolls back and rejects with the thrown error` (thrown error,
`tx.rollback()`, SQL error)
- `keeps the outer transaction usable after a savepoint rolls back`
(three levels)
- `discards a released savepoint when the outer transaction rolls back`
  - `applies the isolation level and access mode`
- `runs concurrent transactions on separate reserved connections and
releases them` (12 transactions on a pool of 3)
- `returns the only connection to the pool after a commit, a rollback
and a failed savepoint` (pool of 1)
  - `behaves the same with prepared statements enabled`
- Instrumentation. These pass on `master` and with the change:
- `registers each transaction and savepoint with the client's work
signals`: the transaction and the savepoint have the client's
work-signal scope, the registration statement runs on the transaction's
own connection, listeners get `intent` at once and `settled` only after
the outer commit, and a rollback reconciles. When transactions bypass
the work-signal wrapper, this test and five tests in
`work-signals.test.ts` fail.
- `response capture counts every statement of a transaction, a savepoint
and tx.query`: six statements, counted for each operation and table,
including two relational reads.
- Every `tx.query` pattern in the codebase:
- `reads tx.query through the transaction, as the announcement and
decision-training services do`
  - `reads tx.query inside a savepoint and follows its rollback`
- `gives tx.query one builder per schema table, the same object on every
read`

Commands run locally on the final revision:

- `pnpm exec vitest run src/lazy-transaction-queries.test.ts
src/work-signals.test.ts src/response-capture-postgres.test.ts` in
`packages/db` on the final revision: 43 passed.
`prepared-queries-postgres.test.ts` passed on the first commit.
- Before the rebase onto current `master`, also
`response-capture.test.ts`, `response-capture-bundle.test.ts`,
`client-options.test.ts`, `client-teardown-registry.test.ts`,
`checkpoint-pool.test.ts`, `client-disconnect-replay.test.ts`: 111
passed in 10 files.
- On the first commit, server callers of `tx.query` and `db.query`, and
hot transaction paths: `announcements-routes`, `decision-training`,
`heartbeat-run-event-atomic-append`, `heartbeat-run-event-sequencing`,
`primary-agent`, `resource-memberships-routes`, `workspace-preferences`,
`sidebar-preferences-routes`, `paperclip-control-plane-port`,
`native-runner-checkpoint-authority.integration`, `dispatch-work`,
`decision-target-work`, `issue-thread-interactions-service`: all passed.
`workspace-preferences` timed out once in its database start hook while
eight suites ran in parallel, and passed alone.
- `pnpm exec tsc --noEmit` in `packages/db`: passed. A type probe showed
that `Db` is the same type as before.
- `pnpm check:module-boundaries`: passed.
- `pnpm verify` exits 2: `packages/db` is a broad input. CI supplies the
full suites. I did not run the full local suite, repo-wide typecheck or
build.

Not verified: the hosted transaction rate, and the effect on a hosted
instance. CI uses Node 24, which has no V8 allocation counter. On CI the
allocation test skips itself, and the builder-count test covers the
mechanism. The benchmark needs Node.js 25 or later and stops with that
message on Node 24. I did not run that error path, because this machine
has Node 26 only.

## Risks

- Every module uses this client. The mitigation is that the transaction
path is still Drizzle's own `transaction()` on the same postgres.js
client. Only the schema argument is absent. The semantics tests pass
unchanged on `master` and on this branch.
- The change relies on Drizzle surface that is not documented API. The
lazy `tx.query` calls the `RelationalQueryBuilder` constructor from
`drizzle-orm/pg-core/query-builders/query`. That subpath is in Drizzle's
export map and the constructor is typed, but `drizzle-orm/pg-core` does
not re-export it. The change also writes the three schema fields of
`tx._`, which the types mark `readonly`. In `drizzle-orm@1.0.0-beta.22`
the same path exports a different class. An upgrade that changes the
constructor fails the typecheck and the `tx.query` tests. An upgrade
that renames `db._.schema` makes the function return the plain eager
client: behavior stays correct, the saving is lost, and the
builder-count test fails. The repository owner decides whether this
dependency on Drizzle is acceptable. A version that uses only documented
API removes the 22 relational reads from the server and creates the
client without the schema.
- `tx.query` is now an accessor until its first read. Code that reads
the property descriptor of `query` sees the difference. No code in the
repository does that.
- The second client and the builders use Drizzle's default dialect
options, as the root client does. If `createDb` ever passes `casing`,
`logger` or `cache` to Drizzle, the same option must go to all three.
The source comment says so.
- A transaction that reads `tx.query` pays the old cost once, on first
read.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code, with tool use
and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 14:31:21 -06:00
DottaandClaude Opus 5.5 82c6810107 fix(runtime): post the agent's blocker reply when a retried finalization blocks the task (#21169)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime finalizes an agent's run, applies the agent's
status decision to the task, and posts the agent's reply as a task
comment
> - When the first workspace copy-back fails, or the run reconnects, a
background retry commits the result instead of the live heartbeat
> - That retry path applies the "blocked" status but never posts the
agent's reply, so the task turns blocked with no visible reason
> - The retry path already shares one recovery step for replies, but
that step accepted only "done", "needs review" and conversation replies
> - This pull request lets the shared recovery step present a blocked
task's reply, so the retry path and the normal path converge on one
presentation step
> - The benefit is that a blocked task always tells the user why,
exactly once

## Linked Issues or Issue Description

**What happened?**

An agent reports that it is blocked. The task moves to "blocked". No
comment from the agent explains why. This happens when the run finishes
through a retried workspace copy-back or a reconnect loop instead of the
live heartbeat. The run succeeds and its result has the agent's blocker
summary, but the run has no presentation decision and the task has no
reply. On a staging company, 58 of the 150 most recent agent-blocked
tasks for one agent had no agent comment.

**Expected behavior**

Every finalization path that applies an agent's status decision also
presents the agent's reply. A blocked task always shows the agent's
explanation once. The status decision does not post a second copy.

**Steps to reproduce**

1. Run a native agent on a task. The agent reports `blocked` with a
summary and a task-wide blocker.
2. Make the first workspace copy-back fail
(`native_workspace_sync_out_failed`), so the heartbeat leaves
finalization to the retry.
3. Let the retry finish the copy-back and commit the result.
4. The task is blocked and the run succeeded, but the task has no agent
comment.

**Paperclip version or commit**

`master` at b8207c4ea9

**Deployment mode**

Hosted (staging cloud). The defect is in server finalization, so it
applies to all deployment modes that use the native runtime.

**Agent adapter(s) involved**

Native runtime (`paperclip_runner`), seen with the Codex-based agent.
The defect is adapter-independent.

## What Changed

- The response presentation recovery step
(`recoverCommittedNativeChatResponse`) now presents a non-conversation
task's reply when the agent reported `blocked` and the task is still
blocked.
- For a blocked reply, recovery uses the agent's final message when one
exists. Otherwise it uses the accepted result summary, the same as the
live completion path.
- A blocked reply on a channel task (an external chat run) uses the same
rule as live run completion. Recovery resolves the chat publication
authorization, prefers the final response, and posts with that
authorization. The worker does not refuse these runs, so it does not
record a false publication failure.
- The presentation work query now finds these runs, so the background
worker also presents them and the bounded publication retry applies to
them.
- A blocker reply is not presented when the task is no longer blocked or
when a later status decision was applied. This keeps old blockers from
posting stale comments.
- Presentation stays idempotent: the step records
`presentationDecision`, reuses a reply that the run already posted, and
never posts twice.
- Existing affected tasks need no separate backfill. The presentation
worker loads all pending candidates at startup, so each task that is
still blocked gets its saved reply once after deploy.

## Verification

- Red/green test:
`server/src/services/native-runtime/native-blocked-reply-recovery.test.ts`.
It completes a blocked native run, fails the first copy-back with
`native_workspace_sync_out_failed`, then commits the retry with
`finalizeNativeRun`.
- Red on `master`: the retry-path tests fail (no comment is posted, and
the worker finds no candidate). The channel task test also fails on the
first PR revision (5eb8cdab95).
- Green with this change: 4/4 pass. The tests check that the blocker
summary is posted once, that the final reply is preferred, that a
repeated retry and a repeated worker check do not double-post, and that
a stale blocker is not posted.
- `cd server && npx vitest run
src/services/native-runtime/native-blocked-reply-recovery.test.ts
src/services/native-runtime/native-response-publication-retry.test.ts
src/services/native-runtime/agent-chat-response-recovery.test.ts
src/__tests__/idle-native-finalization.test.ts
src/services/presentation-ownership-read.test.ts
src/services/chat-monitor-continuation.test.ts` passes (172 tests).
- `cd server && npx tsc --noEmit -p .` passes.

## Risks

- After deploy, tasks that are still blocked from earlier affected runs
get the agent's saved reply once. This is the intended backfill. Tasks
that moved on are skipped.
- The candidate query has one more branch with a `NOT EXISTS` on
`status_decisions`, keyed by the existing `(company_id, issue_id,
decision_version)` unique index. The cost is low.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, browser reads of staging API).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:30:32 -06:00
DottaandClaude Opus 5.5 f1d25aca14 test(restart-chaos): wait for full agent termination in hosted cleanup; count only real outages in continuity (#21176)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness has a hosted mode (#21153) that runs the
release-swap proof on a real staging instance
> - Its first 2-task smoke from master graded 6 / 6 tasks completed
hands-off, but cleanup ended with an error and the API continuity count
showed one failure that was not an outage
> - Cleanup deleted the agent as soon as its `status` read `terminated`,
while its `lifecycleState` was still `terminating`; the server refused
with 409 "Complete termination before deleting the agent"
> - The continuity count treated a deliberate 404 (an older build
without the assigned-work invariant endpoint) as a failed request
> - This pull request waits for full termination and counts only real
outages
> - The benefit is a clean exit code after a clean run and a continuity
number that means "the instance did not answer"

## Linked Issues or Issue Description

Refs #21153

**Describe the bug**
Hosted cleanup reports `remove agent ...: 409 Complete termination
before deleting the agent` and exits non-zero after a clean run. The API
continuity line counts a deliberate 404 as a failed request.

**Steps to reproduce**
Run the hosted smoke (`--hosted-releases 0 --hosted-tasks 2`) against an
instance that does not have
`/api/companies/:id/assigned-work-invariant`.

**Expected behavior**
Cleanup waits until the agent is fully terminated. Continuity counts
only no answer, server errors, and 401, 403, 408 or 429.

## What Changed

- `hosted-setup.ts`: agent removal waits until both `status` and
`lifecycleState` are `terminated` before the delete.
- `hosted-api.ts`: a 400, 404, 409 or 422 answer is a deliberate
refusal, not a continuity failure. No answer, 5xx, 401, 403, 408 and 429
still count.
- `hosted-main.ts`: `finalizeHosted` passes the poll interval through,
for the test.
- `README.md`: states the continuity rule.

## Verification

- `node server/node_modules/vitest/vitest.mjs run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/hosted-*.test.ts`: 34 tests pass. The setup-failure
cleanup test now returns `terminated` with `lifecycleState: terminating`
first and checks that the harness reads again before it deletes. The
continuity test checks that a 404 is not a failure and a 401 is.
- `node server/node_modules/typescript/bin/tsc -p
tests/restart-chaos/tsconfig.json`: clean.
- Live: the smoke `rc-hosted-smoke2` on a hosted staging instance hit
the 409. A second `--hosted-cleanup-only` found nothing left: the agent
is terminated and the product keeps its record (cost history), its tasks
are hidden, its secret is deleted and its project is archived.

## Risks

- Low risk: harness-only. Cleanup can wait up to 120 s per agent for
termination, as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits). Default context window; no
extended thinking mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:20:29 -06:00
DottaandClaude Opus 5.5 5412162716 test(restart-chaos): grade staging-mix hands-off and click Retry once after the settle (#21168)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness grades agent tasks across server swaps,
and the staging-mix workload (#21125) is its staging-shaped load
> - On a fix-PR comparison the harness clicked Retry as soon as the
banner appeared (once after 11 s), which pre-empted the product's own
continuation
> - So the wave could not show whether a task would have continued by
itself, and the banner itself went uncounted
> - This pull request makes staging-mix grading hands-off by default: no
Retry click during the wave or the settle, one click only after the
settle, and every banner recorded
> - The benefit is three honest numbers per wave: completed hands-off /
started, Retry banners shown, and Retry worked / clicked

## Linked Issues or Issue Description

Refs #21125, #21126, #21139.

**What existing behavior does this improve?**
Staging-mix Retry handling and the `done_comment_once` check in the
restart chaos harness.

**Subsystem affected**
Test harness only: `tests/restart-chaos/`. No product code changes.

**Current behavior**
The observer clicks Retry as soon as a task rests with the banner, so a
product continuation that would have come is never seen. A banner that
heals by itself is not recorded. `done_comment_once` counts every agent
comment that mentions `DONE-<nonce>`. A closing summary that quotes the
marker therefore failed all 6 tasks of wave `w-20261112T174255-s7101`.
The release-swap grader can click a second Retry after the observer's.

**Proposed behavior**
Hands-off by default. Every banner is recorded with its first-shown
time, its duration and its error code. One Retry click happens only
after the settle, and only for a task that is still not done and that
nobody clicked. The DONE check counts only the comment whose whole body
is the marker. A completion summary repeated by a different run becomes
its own `completion_once` finding.

**Reason and benefit**
The product goal is that the Retry banner never appears and that
interrupted work continues by itself. The wave now measures exactly
that.

## What Changed

- `staging-mix.ts`:
- The observer step records the Retry banner per task (`retryBanner`:
first shown, total shown time, error code). It clicks during an overlap
only with the new flag `--mix-retry-during-wave`.
  - `isDoneComment` counts only the bare `DONE-<nonce>` comment.
- `completion_once` fails when more than one run posted a comment that
mentions the marker.
  - Comments are now collected with their run ids.
- `observer.ts`:
- A staging-mix task with a banner is kept under watch instead of being
clicked or rested (hands-off).
  - `retryAfterSettle` reopens a task for the one post-settle click.
- New `staging-mix-retry.ts`:
- `shouldClickRetryAfterSettle` is the single click rule. It never
clicks a done task, a task with a scheduled retry, or a task already
clicked.
- `retryAfterSettleStage` waits out the settle (new flag
`--mix-settle-minutes`, default 15 after the last fault slot ends),
clicks once per stuck task, and follows each click to rest.
- The results module writes `staging-mix-results.json` and
`staging-mix-summary.md`. Their first three lines are `completed
hands-off / started`, `Retry banner shown: N tasks`, `Retry worked /
Retry clicked`. Then come the banner list and one end-of-wave read of
`GET /api/companies/:id/assigned-work-invariant` (`violationCount`,
`violations`, `agentsWithStalledSlots`, `openTasks`).
- `release-swap-finish.ts` uses the same click rule, so the swap grader
never clicks a task the observer already clicked. When the release-swap
cell drives the wave, the staging-mix settle stage does not run; the
swap grader makes the one click. The hosted mode can import the same
function.
- `main.ts`: the staging-mix end-of-wave report moved into
`StagingMixRun.report`. The file shrinks from 1,000 to 998 lines.
- README: the Retry and results sections, the two new checks, and the
local short-form command (without `--ephemeral-server`, which runs only
on Daytona).

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
59 files, 714 tests pass.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- New tests cover:
  - the default and opt-in flags;
  - the click rule;
- a settle where one task heals by itself (no click), one task gets
exactly one click, and one already-clicked task gets none;
  - the three headline numbers, including the swap grader's clicks;
  - the invariant read;
- the DONE count against the real comments from
`w-20261112T174255-s7101` t001 (summary plus bare marker passes; a
summary repeated by another run fails `completion_once`).
- Not run: a live wave with this change. The Daytona image for current
master is still queued.

## Risks

- Low risk. Test harness only.
- A hands-off task that keeps its banner now waits to its deadline
before the post-settle click, so a wave with many banners runs longer.
- RS1 waves on other workloads no longer get a second end-of-wave click
on a task the observer already clicked. That is intended.
- Known master CI failures, not touched here: the storage S3 multipart
throttle test (being fixed) and the Runner native-distribution-integrity
test.
- `ci / Runner support suite` failed on head 59b1b37870 with no failing
test: its log ends with "The runner has received a shutdown signal" and
"The operation was canceled" (runner infrastructure). This diff touches
only `tests/restart-chaos/`, which that suite does not run.
- On head 9907c4cdcc, six server jobs in run 43289297870 (General tests
server 2, 4, 8, 11 of 12; serialized server suites 6 and 7 of 9) were
cancelled the same way at 19:26Z (runner shutdown signal, then operation
canceled, no test failure). They were re-run once with `gh run rerun
43289297870 --failed`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits). Extended reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:20:03 -06:00
Nicky LeachandPaperclip 7f69ef8b44 Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path

> - Paperclip lets plugins own managed Git workspaces.
> - A portable Runner task can move to a different physical allocation.
> - The checkout must be present on the allocation that launches the
Runner.
> - Workspace preparation and finalization need exact placement when
they retry.
> - This change checks the placement at Runner admission and records it
before finalization.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Workspace preparation and finalization for portable Runner tasks.

**Current behavior**
A task replacement can change the allocation after Git preparation. A
lost finalization reply can leave retry without the original placement.

**Proposed behavior**
Prepare the checkout on the actual allocation before admission. Record
finalization placement so an exact retry can use it.

Portable ownership and controller recovery are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066).

## What Changed

- Reserve the managed workspace before acquiring the portable task
lease.
- Pass the task placement to the workspace provider. Reuse the described
task while placement stays the same and prepare again before admitting a
replacement.
- Record the initial preparation's run, agent, generation, and optional
task placement before dispatch. Inspection and retries on the same
Launcher replay the same request; VM and sandbox preparation use the
same recovery path.
- Keep provider request identity separate from Git credential authority:
reservation reconciliation uses the existing project-consent path in a
host-only scope bound to the plugin and workspace. Foreground
preparation still requires an active run; consent, repository, and
revocation checks are unchanged.
- Acquire the current Launcher before retrying preparation. If placement
changes, use ordinary preparation on the replacement so the provider can
settle the prior writer and recover the retained checkout. No additional
provider discovery call is added.
- Select finalization placement before publishing the operation record,
then dispatch and recover using that saved choice. Retain the Launcher
during bounded finalization retries.
- Preserve interrupted pre-upgrade finalization requests without
placement, including their original operation IDs and provider routing.
No migration or new durable mode flag is required.

## Verification

- Final revision `74ea3e6712`: server typecheck and exact SDK source-pin
verification (538 inputs) passed.
- All 185 tests passed across the full plugin worker manager, project
Git authorization, and plugin workspace integration suites (including 56
workspace tests). Independent verification reran the three lost-reply
cases: inspection, same-Launcher retry, and replacement after deleting
the original Launcher. The tests assert the correct request identity,
durable intent, and clearing that intent after success.
- Before the final retry-accounting fix, all 323 tests passed across
workspace admission, runtime execution, native workspace finalization,
plugin workspace integration, and terminal cleanup retry. Regressions
exercise crashes after operation insertion but before dispatch in both
finalization paths, exact replay after lease release, and retry after
ambiguous placement clears without publishing an incomplete operation.
Pre-upgrade interrupted requests retain their operation IDs despite an
available portable lease.
- Final retry-accounting fix: all 84 native workspace finalizer,
failure, and supersession tests passed. A real missing-plugin regression
first failed on the prior commit, then verified healthy-run progress,
three-attempt exhaustion, terminal cleanup, and absence of a misleading
workspace operation. Independent verification also passed the legacy and
crash-before-dispatch cases.
- Worker-process tests cover both preparation RPC methods: the provider
still receives the original run and agent, maintenance receives
project-scoped credential authority, and foreground or mismatched-scope
requests remain denied. Database tests confirm a stopped run cannot
authorize foreground Git access while valid project consent can
authorize maintenance.
- Existing metadata protection coverage proves callers cannot overwrite
the private preparation intent or read it through workspace responses.
- Two existing Launcher Git tests passed: retry preserves dirty files
and the original base commit without fetching, and retry resumes the
saved checkout intent while rejecting a changed pending branch.
- Implementation was delegated to GPT-6.1 Sol and independently
reviewed. The GitHub stack's automatic rebase preserved the original
workspace-placement patch.

- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706)
passed on `74ea3e6712`, including server and workspace tests, Runner,
end-to-end tests, typecheck, build, and canary dry run. Ordinary
Greptile and
[Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388)
both scored `74ea3e6712` 5/5, with no unresolved findings.

## Risks

- Live Daytona end-to-end qualification is pending. The checkout record
persists on the mounted volume; replacement preparation relies on the
provider confirming the previous writer is settled before using that
retained checkout. Local tests cover Core request routing and Launcher's
durable checkout recovery separately.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 12:12:41 -08:00
github-actions[bot]andlockfile-bot ea400ecb07 chore(lockfile): refresh pnpm lockfiles (#21178)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.
- Resync the SDK public source proof's lock pins from the refreshed
lock.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-12 19:50:19 +00:00
Huanyi Xie 8bff629465 fix(environments): skip blank lease ID queries (#15200)
## Thinking Path

> - Paperclip manages work for AI agents.
> - Runs can use environment leases.
> - A lease lookup sends its ID to a PostgreSQL UUID column.
> - An empty ID makes that query fail on every poll.
> - This pull request returns null before querying for blank IDs.
> - Valid lease IDs keep the existing lookup behavior.

## Linked Issues or Issue Description

Fixes: #15143

## What Changed

- Skip database access for empty or whitespace-only lease IDs.
- Cover empty, space, and newline IDs with a database-query spy.
- Keep an unknown valid UUID returning null. Existing lease tests cover
valid lookups.

## Verification

- The branch was rebased onto current master `252d9b50` to resolve the
test-file conflict.
- All 53 environment service tests passed on Node 24.19.0. The run used
cached dependencies and current database and shared-package source
aliases.
- The blank-ID cases assert that no database query occurs. The unknown
valid UUID still returns null. The existing suite covers stored leases.
- `git diff --check`: passed.
- A clean frozen install is blocked because upstream master adds two
OpenTelemetry packages without updating the lock file.
- Full typecheck and build were attempted on both unmodified master
`252d9b50` and this head in the same environment. Both fail with the
exact same runner `SessionRecord.cursor_prompt_usage` error (TS2339 at
`codex-runtime-adapter.ts:1572`). Full validation is pending CI on the
rebased commit.
- The earlier full test run had a Slack callback ordering failure that
also occurred on unmodified master.
- Greptile reviewed the previous head at 5/5 with no inline findings.

## Risks

Low risk. Only blank IDs return early. Non-blank IDs still use the
database. There is no schema or migration change.

## Model Used

OpenAI Codex, based on GPT-6, assisted with source review,
implementation and local test execution. The exact deployment model ID
and context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted suite; full suite
pending above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
user-facing configuration or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (new head checks are pending)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-12 13:49:09 -06:00
DottaandClaude Fable 5.2 a37f3056b6 perf(docker): keep dependencies and provider binaries in layers that survive a commit (#21167)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each merge to master builds the production Docker image for amd64
and arm64 and pushes it to GHCR
> - Each build pushes three layers again, about 3.5 GB compressed of a
4.7 GB image
> - A file-by-file comparison of two consecutive master images shows
that about 95% of those bytes do not change between commits
> - The unchanged bytes are in the same layers as small files that
change on every commit, so the registry sees new layers
> - This pull request moves the unchanged bytes into their own layers
> - The benefit is a smaller push for each master build and a smaller
pull for each deploy

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The production image build. It pushes about 3.5 GB of layers for each
architecture on each master commit. Each deploy pulls the same bytes
again.

**Subsystem affected**

Docker image build (`Dockerfile`, `production` and
`runner-provider-pack` stages).

**Current behavior**

I compared the amd64 images of two consecutive master commits (`2174f2d`
and `dfc510b`, 15 changed server source files). I hashed each file in
the three layers that changed.

| Layer | Compressed | Content that is identical between the two builds
|
| --- | --- | --- |
| `COPY --from=build /app /app` | 1,404 MB | 97.5%. The root
`node_modules` is 2.86 GB of 4.13 GB and has 0 changed files. |
| `COPY /provider-pack` | 1,008 MB | 100% of the 112 files of 1 MiB or
more (2.54 GB). Only 32 small files change (manifest, `.modules.yaml`,
`.bin` shims). |
| `COPY /provider-pack-transport/` | 1,059 MB | The archives differ only
in tar timestamps. This pull request does not change this layer. |

**Proposed behavior**

- Copy the root `node_modules` from the `deps` stage as its own layer.
Copy the remainder of `/app` from the `build` stage with
`--exclude=node_modules`.
- In the `runner-provider-pack` stage, move pack files of 1 MiB or more
to `/provider-pack-large` after the transport archive is made. Set their
mtimes to a fixed value.
- Copy `/provider-pack-large` with `COPY --link`, then copy the small
remainder of the pack to the same directory.

**Reason and benefit**

The dependency layer changes only when the lockfile changes. The large
provider-pack layer changes only when a provider version changes. The
expected push for a commit that changes source only goes from about 3.5
GB to about 1.5 GB for each architecture. The master build runs about
140 times a day on AWS runners, and this traffic is about 570 GB a day
of billed data transfer. Deploy pulls become smaller by the same amount.

**Breaking changes**

None. The file tree of the image is the same. Provider-pack files of 1
MiB or more now have an mtime of 1970-01-01.

## What Changed

- `Dockerfile`: the `production` stage copies `/app/node_modules` from
`deps` first, then `/app` from `build` without the root `node_modules`.
- `Dockerfile`: the `runner-provider-pack` stage splits large pack files
into `/provider-pack-large` with fixed mtimes. The `production` stage
copies that tree with `--link` before the remainder of the pack.
- `server/src/__tests__/docker-image-layer-reuse.test.ts`: a drift guard
for the copy order, the exclude, the `--link`, and the position of the
split after the archive step.

## Verification

- `cd server && npx vitest run
src/__tests__/docker-image-layer-reuse.test.ts
src/__tests__/docker-build-stamp.test.ts`: 9 tests pass. The 2 new tests
fail against the `Dockerfile` on master.
- `node scripts/check-docker-deps-stage.mjs`: PASS.
- `pnpm verify`: exit 2 (the `Dockerfile` is a global CI input).
Required CI supplies the broad coverage.
- A small Dockerfile with the same syntax version
(`docker/dockerfile:1.20`) and the same four `COPY` instructions, built
two times with a changed stamp file:
- `--exclude=node_modules` leaves out only the root `node_modules`. A
nested `node_modules` stays.
  - The dependency layer has the same digest in the two builds.
- The large-file layer has the same digest in the two builds only with
`--link` and fixed mtimes. Without one of them the digest changes.
- Hard links between large files stay hard links. The merged directory
contains each file one time.
- Evidence that the `build` stage writes nothing to the root
`node_modules`: in the published image, all 107,413 files there have an
mtime in the minute of the `deps` install. The app build ran 15 hours
later.

**Not verified:** a full build of the production image with this
`Dockerfile`. The local Docker disk was full, and no pre-merge workflow
builds this image. The first full build is the master build after merge.
The check after merge is to compare the layer digests of two consecutive
master images.

## Risks

- Medium until the first master build. If the image build fails, master
image builds stop until a revert. No broken image can be published,
because the build fails before the push.
- If a future build step writes to the root `node_modules` in the
`build` stage, the image will not contain that change. The comment in
the `Dockerfile` states this rule.
- A file that crosses the 1 MiB boundary moves between the two pack
layers. This only changes which layer is pushed.
- The first build after merge pushes all the new layers one time.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with
tool use (shell, AWS CLI, Docker, registry API).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 13:48:38 -06:00
DottaandClaude Fable 5.2 eaef46b60b feat(ui): browser page-load tracing to Honeycomb for hosted instances (#21156)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI is the main surface. Its load speed is a product
requirement.
> - The team wants to track page-load times over time and give each
metric an owner and an SLO in Honeycomb.
> - Server traces show only the server part of a page load. They do not
show JavaScript start-up, lazy route code, rendering, or client-side
navigation.
> - Paperclip is also self-hosted by others. Browser tracing must stay
off there, and it must not slow any page.
> - This pull request adds real-user browser tracing for hosted
Paperclip Cloud instances only. It loads lazily and sends a closed set
of attributes.
> - The benefit is one `page.load` measurement per page (named by route
pattern), plus Web Vitals and API spans that join the server trace,
ready for Honeycomb queries and SLOs.

## Linked Issues or Issue Description

No public issue exists. Feature description:

**Subsystem affected**

Board UI (`ui/`), the authenticated session route
(`server/src/routes/auth.ts`), and the shared session schema.

**Problem or motivation**

We cannot measure how long the all-tasks page or a task page takes to
load for real users. Server traces stop at the API response. Most
navigation is client side, and the standard OpenTelemetry document-load
instrumentation does not cover it. Without a per-page browser
measurement, no agent can own a page-load SLO.

**Proposed solution**

The server sends a `browserTracing` config in the authenticated session
only when the instance is cloud managed and
`PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set. The browser then measures
every hard load and client navigation in one shared recorder tied to the
router's own route tree. After the page settles, it lazily loads an
OpenTelemetry chunk that exports `page.load`, `web_vital.*`, and
same-origin `/api/` fetch spans to Honeycomb. An allowlist removes every
other attribute before export.

**Alternatives considered**

`@honeycombio/opentelemetry-web` was the first choice. Its defaults add
the full page URL, query string, hash, referrer, and user agent to every
span and to the resource, and it bundles log and metric pipelines. We
would have to replace its exporter and remove most of its defaults.
Plain OpenTelemetry web packages plus `web-vitals` are smaller and send
only what we choose. Honeycomb needs only OTLP/HTTP and its ingest-key
header.

**Roadmap alignment**

This is internal hosted-instance observability. It changes no product
behavior and no self-hosted behavior.

## What Changed

- `server/src/browser-tracing-config.ts`: resolves the browser tracing
config. It returns `null` unless `isCloudManagedInstance()` is true and
`PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set. It accepts only a Honeycomb
ingest key (`hcaik_` prefix), requires `https` (except loopback), and
fails closed with one value-free warning on any invalid setting.
Optional: `PAPERCLIP_BROWSER_TRACES_INGEST_KEY`, `_SAMPLE_RATE` (default
1), `_SERVICE_NAME` (default `paperclip-web`), `_ENVIRONMENT` (default
`SENTRY_ENVIRONMENT`).
- `GET /api/auth/get-session` adds `browserTracing` (config or `null`),
the same delivery path as the front-end Sentry DSN. The shared
`authSessionSchema` has the new optional field
`browserTracingConfigSchema`.
- `ui/src/lib/browser-tracing/page-load-recorder.ts` and `controller.ts`
(main bundle, no OpenTelemetry imports): one `page.load` measurement per
hard load (from navigation start) and per client navigation (from the
history change). A measurement ends at the first painted frame where no
full-page loading state is mounted and either every page-declared
content signal is ready or no query without data is fetching. Redirects
continue the measurement. A new navigation interrupts it. 30 s times
out. The sample decision is made once per page session before any
download.
- `ui/src/components/PageLoadTracker.tsx`: `App` passes the same
`<Route>` tree it renders, so every page is reported by route pattern
(for example `/:companyPrefix/issues/:issueId`) with no per-page code.
- `usePageContentReady` hook and blocking gates: `PaperclipLoading`
(also the route `Suspense` fallback) blocks readiness. The task page
uses the existing `issue-detail:navigate→content-paint` point. The plain
all-tasks list signals when its first rows render.
- `ui/src/lib/browser-tracing/sdk.ts` (lazy chunk, loaded after the page
settles): OpenTelemetry web tracer, OTLP/HTTP exporter, fetch
instrumentation for this origin's `/api/` only with `traceparent` sent
only to this origin, `web-vitals` LCP/INP/CLS, and a context manager
that makes the open `page.load` span the parent of its API calls.
`span-sanitizer.ts` applies the closed attribute allowlist (no URLs,
ids, query strings, status messages, or event attributes). Uncaught
errors are not sent because Sentry already reports them.
- New `ui` dev dependencies, pinned exactly. Each is needed, and the
OpenTelemetry ones are the same versions the server already resolves:
- `@opentelemetry/api@1.9.1` (already in the workspace): tracer and
context API.
- `@opentelemetry/sdk-trace-web@2.10.0`: the browser tracer provider,
batch processor, and stack context manager.
- `@opentelemetry/instrumentation-fetch@0.221.0`: fetch spans and
`traceparent` injection.
- `@opentelemetry/resources@2.10.0` (already a server dependency): the
resource.
- `@opentelemetry/otlp-transformer@0.221.0` (already a server
dependency): OTLP JSON serialization for a 60-line `OtlpFetchExporter`.
This replaces `@opentelemetry/exporter-trace-otlp-http`, which is a
server optional peer: installing it in the workspace made the server's
exact-version gate test see it as present.
  - `web-vitals@6.2.3`: LCP, INP, and CLS.
All of them are in the lazy chunk only. The lockfile is not committed:
`pr-trusted` rejects lockfile edits in pull requests, CI installs with
`--no-frozen-lockfile` after the frozen attempt reports
`ERR_PNPM_OUTDATED_LOCKFILE`, and the lockfile bot refreshes it after
merge.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the three shared files (the session schema is part of the pinned public
SDK surface; the contract must live in `packages/shared`).
- Docs: new `doc/browser-performance-tracing.md` (enablement, config,
key handling, span and attribute list, page-load definitions, example
Honeycomb queries and an SLO SLI). Pointers in `doc/observability.md`
and the Honeycomb tracing skill.
- Two App routing tests mock `PageLoadTracker` because they render `App`
without a `QueryClientProvider`.

## Verification

- `pnpm verify` returns exit 2 (broader verification required) because
docs are global CI inputs. Required CI supplies the broad coverage.
- Focused local tests, all pass:
- `ui`: `vitest run src/lib/browser-tracing
src/components/PageLoadTracker.test.tsx
src/components/SentryGate.test.tsx src/api/auth.test.ts
src/lib/issue-detail-performance src/pages/IssueDetail src/pages/Issues
src/App src/components/CloudAccessGate.test.tsx
src/components/IssuesList`: 17 files, 484 tests.
- `server`: `browser-tracing-config.test.ts`, `auth-routes.test.ts`,
`sentry.test.ts`: 63 passed, 5 skipped.
  - `packages/shared`: `validators/access.test.ts`: 19 passed.
- New tests prove: the SDK module is never imported when the session
config is `null`, absent, or signed out, or when the session is not
sampled; route templates contain no ids, slugs, or tokens; a client
navigation produces a `client` measurement under
`/:companyPrefix/issues/:issueId`; `traceparent` goes only to this
origin's `/api/` and not to a third-party or non-API URL; exported spans
carry only allowlisted attributes. A mutation check (tracker forced on)
turned the "never imports" tests red.
- Typecheck: `ui` (`pnpm typecheck`), `server` (`pnpm --filter
@paperclipai/server typecheck`), and `packages/shared` pass.
- Real-browser run: a local dev server with a local OTLP collector and
headless Chromium. The self-hosted session returned `browserTracing:
null` with the endpoint set, and the page made 0 SDK and 0 collector
requests. With a cloud session config injected into the session
response, the collector received a hard `page.load` (route
`/:companyPrefix/issues`, `ready_signal=idle`), a client `page.load` to
`/:companyPrefix/issues/:issueId` (`ready_signal=content`), a back
navigation, LCP/INP/CLS spans, and `/api/` fetch spans parented to the
open `page.load` span. No task identifier, company id, task title, host,
or URL appeared in the exported payload. No third-party request carried
`traceparent`.
- Bundle size (`ui` production build, initial JavaScript closure of
`index.html`): before 3,813,986 B raw / 1,153,991 B gzip; after
3,820,743 B / 1,156,590 B (+6.8 KB raw, +2.6 KB gzip, +0.2%). The lazy
SDK chunk is 84 KB raw / 27 KB gzip and is not fetched unless tracing is
on and sampled.
- Review fixes (each red on the old code, green on the new): the SDK now
loads by a 35 s deadline even if the page never settles, so a timed-out
hard load is exported (`controller.test.ts`); an agent chat with no
saved task uses the idle rule instead of a content gate that never fires
(`usePageContentReady.test.tsx`); the sample rate is rounded to exactly
1/N so Honeycomb's `SampleRate` weight matches
(`browser-tracing-config.test.ts`). The real-browser run was repeated
after the exporter change with the same result.

## Risks

- Low risk for self-hosted installs: the server never sends a config, so
no tracing code beyond the small recorder runs, and the SDK chunk is
never fetched.
- On hosted instances the feature is off until an operator sets
`PAPERCLIP_BROWSER_TRACES_ENDPOINT` and an ingest key. When on, the
controller wraps `history.pushState`/`replaceState` to time navigation
starts and restores them on sign-out.
- Every signed-in browser can read the ingest key, like the Sentry DSN.
Only Honeycomb ingest keys (send-only) are accepted. A collector
endpoint can keep the key server-side.
- Hard-load API calls happen before the SDK loads, so they have no fetch
spans. The hard-load `page.load` span is still recorded.
- The browser-to-server trace join needs server tracing. The hosted
image does not install the OpenTelemetry server peers today
(`CLOUD_BUNDLED_SERVER_DEPS` is `@sentry/node` only). Browser spans work
without it.
- `ui/src/pages/IssueDetail.tsx` was already over 1,000 lines. This
change adds 5 lines there for the task page content signal, next to the
existing content-paint measure.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use
(shell, file edits, local browser automation). Reasoning effort: low.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 13:48:35 -06:00
DottaandClaude Fable 5.2 8990e94005 fix(ci): keep community PR runs from delaying CODEOWNERS jobs in the hosted queue (#21166)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Its pull request CI runs on two pools: an AWS fleet for CODEOWNERS,
and GitHub-hosted runners for every other contributor
> - Some short jobs for CODEOWNERS pull requests also run on
GitHub-hosted runners: the commitperclip review and the merge review
notifier
> - GitHub starts about 550 hosted jobs each hour for this repository,
and all hosted jobs wait in one first-come queue
> - A community pull request run puts 52 jobs in that queue at one time,
so the short CODEOWNERS jobs wait behind them
> - This pull request limits how many jobs one hosted run can queue, and
lets the short CODEOWNERS jobs use the reserved AWS metadata fleet
> - The benefit is that community pull requests do not delay CODEOWNERS
work

## Linked Issues or Issue Description

**What happened?**

On 2026-11-12 the repository had about 348 queued workflow runs. Jobs on
the AWS fleet started in a median of 29 seconds. Jobs on `ubuntu-latest`
did not:

| Job | Median wait | Job length |
| --- | --- | --- |
| PR merge review notification | 42 min | 9 s |
| commitperclip PR Review | 36 min | 15 s |
| Community PR test jobs | 57 min (p90 76 min) | 7 min |

Between 12:00Z and 18:00Z, hosted job starts stayed between 492 and 567
each hour while the number of running hosted jobs changed from 63 to
134. The start rate is the limit. Community PR runs supplied 3,462 of
the 3,744 hosted jobs in that window (92%).

**Expected behavior**

A short job for a CODEOWNERS pull request starts in about one minute,
whatever the community pull request load is.

**Steps to reproduce**

1. Open several pull requests from forks in a short time.
2. Open a pull request as a CODEOWNER.
3. Look at the start time of `commitperclip PR Review` and `PR merge
review notification` for the CODEOWNER pull request.

## What Changed

- `pr-trusted.yml`: each of the four test matrices has `max-parallel: 4`
when the run uses `ubuntu-latest`. The run has the same jobs. The other
jobs wait in the run, not in the shared queue. AWS runs have no limit
(256).
- `commitperclip-review.yml` and `pr-merge-review.yml`: the job uses the
reserved `paperclip-ci-routing-x64` fleet when
`AWS_CI_SHORT_JOBS_ENABLED` is `true`, the pull request branch is in
this repository, and the author and the sender (or the run actor and the
triggering actor) are in the AWS admission list. The admission variables
are the same as in `ci-routing.yml`. All other events use
`ubuntu-latest`.
- Check-run and commit-status notifications stay on `ubuntu-latest`.
Those events do not identify the author.
- `ci-routing.test.mjs`: 24 placement cases for the two jobs, and a
contract test for the matrix limit.
- `doc/DEVELOPING.md`: the two settings and the runner group step.

## Verification

- `node --test .github/scripts/tests/ci-routing.test.mjs
.github/scripts/tests/notify-pr-merge-review.test.mjs
.github/scripts/tests/ci-plan.test.mjs
.github/scripts/tests/ci-profile.test.mjs
.github/scripts/tests/run-quality-gates.test.mjs`: 200 tests pass.
- `actionlint` accepts the three workflows, including the `max-parallel`
expression.
- Not verified before merge: `pr.yml` calls `pr-trusted.yml@master`, and
the two short workflows run from master. The effect on queue time can
only be measured after merge.

## Risks

- A community pull request run takes longer when the hosted queue is
empty: about 45 minutes for the server matrix, compared with about 13
minutes. Today these runs take 150 to 200 minutes because of the queue.
- The AWS routing does nothing until a maintainer sets
`AWS_CI_SHORT_JOBS_ENABLED`. Before that, the `paperclip-ci-routing`
runner group must allow the two workflow references in
`doc/DEVELOPING.md`. If the variable is set first, the two jobs for
CODEOWNERS pull requests stay queued.
- The two jobs run only scripts from master and no pull request code.
The secrets they read do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 13:48:25 -06:00