Commit Graph
4114 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 81d06cfc97 feat(mcp-tasks): add a demo MCP server that uses Tasks, test against it, and keep audit records out of agent errors (#21464)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call tools in connected apps through the tool gateway, which
speaks MCP to each app
> - MCP Tasks (#21263) lets an app answer a long tool call with a
durable task. It shipped behind the experimental setting
`enableMcpTasks`, off by default
> - No public app returns tasks without credentials, so nobody had seen
the feature work against a real server, and it was hard to understand
> - This pull request adds a small demo app that uses Tasks, tests the
gateway against it over HTTP, and records a live end-to-end run
> - The live run found one bug: every task stop sent the call's audit
record, with a server stack trace, to the agent. This pull request fixes
it
> - The benefit is a repeatable way to try and test Tasks,
plain-language docs, and evidence for the decision to turn the setting
on

## Linked Issues or Issue Description

Refs #21389
Refs #21263

## What Changed

- Add an MCP Tasks demo server to `packages/kv-demo-mcp-server` (new bin
`paperclip-tasks-demo-mcp-server`; run `node
packages/kv-demo-mcp-server/dist/tasks/main.js` after a build). It is
stateless MCP 2026-07-28 with `server/discover`, and it declares
`io.modelcontextprotocol/tasks`. It has six tools: `slow_report`,
`needs_approval` (form question), `will_fail`, `cancellable`, `expires`,
and `quick_echo`. It saves task state to a file, so it can restart
mid-task. It needs no credentials. It has a status page at `/` and JSON
at `/api/state`.
- I put it in the existing demo package, so there is no new workspace
package and no `pnpm-lock.yaml` change.
- Add `server/src/__tests__/mcp-tasks-demo-server.test.ts`. It runs the
real gateway against the real demo server over HTTP, with one test per
tool plus a restart.
- Fix: `server/src/routes/tool-gateway.ts` no longer sends
`details.execution` to callers. That record has request IDs and the
server-side error stack. The audit still keeps it. Add
`server/src/__tests__/tool-gateway-agent-errors.test.ts` (red before the
fix, green after).
- Docs: `doc/connections/MCP-TASKS.md` now has "In plain words", "What a
person sees", "Try it yourself", the live-run results, and a
recommendation. The package README documents the demo server.
- `enableMcpTasks` stays off by default.

## Verification

- `packages/kv-demo-mcp-server`: `npx vitest run` gives 25/25 pass (13
new demo-server tests). `tsc -p tsconfig.test.json` is clean.
- `server`: `mcp-tasks-demo-server.test.ts` gives 8/8 pass.
`tool-gateway-agent-errors.test.ts` gives 2/2 pass, and it failed 2/2
before the fix.
- Existing suites that cover the changed route and Tasks:
`mcp-tasks.test.ts` and `tool-gateway-discovery-http.test.ts` (results
in a comment below).
- `pnpm verify` exits 2, because the doc is a global CI input. Required
CI gives the broad coverage.
- Live end-to-end run on a local `test-drive` instance with
`enableMcpTasks` on. The agent was Codex `gpt-6-luna` at low effort. The
demo was connected through "Connect your own MCP server":

| Behaviour | Result | `tools/call` count at the app |
| --- | --- | --- |
| Polling (`slow_report`, 45 s) | Delivered after 22 polls | 1 |
| Question (`needs_approval`) | The card was answered in the UI.
`tasks/update` was sent, and the task finished | 1 |
| Failure (`will_fail`) | `provider_task_failed` with the app's message
| 1 |
| Cancel (`cancellable`) | Stop on the run sent `tasks/cancel` | 1 |
| Expiry (`expires`, 15 s) | `provider_task_expired`, no hang | 1 |
| Restart (`slow_report`, 120 s) | The server was down for 2 min 23 s.
The result was delivered once, and the agent was woken once | 1 |

After the run, all 6 `tool_mcp_tasks` rows were settled and none had a
next poll.

## Risks

- Low risk. The demo server is new code in a demo package, and the bin
is additive.
- The route change removes `execution` from tool-gateway error
responses. No UI code or test reads that field from these responses. The
audit and tool-call events still record it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:47:58 -06:00
DottaandClaude Opus 5.5 1f388ed385 Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm sandbox keeps an agent's Runner alive between runs, so the
next run on the task starts fast
> - When a run ends, the server closes the idle warm Runner in-process.
A separate SQL worker stops the sandbox when its warm window ends
(#21410, merged today)
> - Both steps start at the same moment. The sandbox stop wins, so the
idle close fails. The failed owner stays registered, and the next run
throws `native_instruction_materialization_handoff_unavailable` before
any provider work
> - Recovery must just work: a follow-up run must never fail because a
predecessor's warm copy is gone
> - This pull request lets the idle close finish before the sandbox
stops, prepares instructions fresh when the warm copy cannot be handed
off, and releases an owner whose sandbox the host already gave up
> - The benefit is that automatic follow-up runs start normally after a
warm window ends

## Linked Issues or Issue Description

Refs #21472 (base of this stacked PR), #21410 (introduced the
warm-window stop).

**What happened?**
In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three
automatic follow-up runs failed before any provider work with
`native_instruction_materialization_handoff_unavailable`: run
`5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a
release swap. In each case, the predecessor's idle close had failed
first with `provider_state_unreadable` / "Daytona sandbox lease … is no
longer active". The cause: the warm-window stop from #21410 stopped the
sandbox at the same moment that the idle close started. For the first
run's predecessor, the close was requested at 20:30:34 and the sandbox
got SIGTERM at 20:30:37.8.

**What did you expect?**
The follow-up run prepares its instructions fresh and continues. The
warm-window stop does not interrupt the host's own idle close.

**Steps to reproduce**
1. Run a warm Daytona task until it ends, so its warm window starts.
2. Let the warm window end while the idle timer closes the Runner.
3. Start a follow-up run on the same task. It fails with
`native_instruction_materialization_handoff_unavailable`.

Two of the three tasks then ended stranded. That downstream part (the
claim-gate cancellation with `execution_reconciliation_required` and the
stranded notice) is owned and fixed by another session. It is **not** in
this PR.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
`claimWarmNativeInstructionCopy`: after it retires a stale warm owner
whose copy cannot be adopted, it returns `null`. The caller then
prepares instructions fresh. Before, it threw
`native_instruction_materialization_handoff_unavailable`.
- `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now
waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window.
The in-process idle close (close grace plus the 30 s proof read)
finishes first.
- `native-session-executor.ts` idle timer: when an idle close fails, it
reads the lease rows for that sandbox. If every row is no longer
`active` or `retained` (the host gave the sandbox up), it releases the
in-process owner. The next run then prepares fresh.
- `doc/execution-semantics.md`: one paragraph about the stop margin and
fresh preparation.

## Verification

Red → green (each test fails on the base `35ba76d0ef` and passes on this
branch):

- `native-session-executor.test.ts` › "hands off collection before
probing or retiring a warm owner: unavailable" — red: throws
`handoff_unavailable`; green: returns `null` (fresh preparation).
- `native-session-executor.test.ts` › "forgets a warm owner whose idle
close failed only once the host gave up its sandbox: pending_cleanup /
released / retained" — red on all three; green: the owner is released
for `pending_cleanup` and `released`, and kept (retired by the
successor) for `retained`.
- `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the
warm runner time to finish before stopping its sandbox" — red: the
sandbox stops 3 s after the window; green: it stops only after the
margin. Two existing tests ("keeps a sandbox still inside its warm
window, and wakes when the window ends", "reconciles a warm sandbox left
running across a restart") now expect the deadline with the margin.

Local runs on this branch:

- `server`: `tsc --noEmit` clean.
- `vitest run src/__tests__/warm-sandbox-expiry.test.ts
src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed.
- `vitest run
src/services/native-runtime/native-session-executor.test.ts -t "hands
off collection before probing or retiring a warm owner|forgets a warm
owner whose idle close failed"`: 10/10 passed.

Not run: a Daytona wave with this change.

## Risks

- Warm sandboxes now stop up to 2 minutes later than the configured
window. This is a small cost increase per warm sandbox. I chose a fixed
margin over an in-process "is a close running?" check because the stop
is pure SQL. The margin works when two servers run at the same time (the
server that runs the stop may not be the server that owns the Runner),
and it needs no busy-loop or cross-server signal.
- The margin does not cover a re-armed idle timer or a close that takes
longer than 2 minutes. Fresh preparation (fix 1) and the owner release
(fix 3) cover those cases.
- Not run on Daytona. The evidence is from the wave above and the unit
tests.
- In the wave, a `restore_failed` copy-back onto the just-restarted
sandbox was also seen. I did not find its root cause. It is not changed
here.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test runs) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:47:42 -06:00
DottaandClaude Opus 5.5 a4c570aede feat(mcp): ask an app's form questions in the task chat and send the answer back (MCP elicitation) (#21466)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected MCP apps through the tool gateway.
> - An MCP app can ask the person a question in the middle of a tool
call (MCP elicitation). There are two kinds: a form, or a link to open.
> - Today a form question is a dead end:
> - A 2026-07-28 server's form `input_required` stops with "needs
information Paperclip can't collect yet".
> - An older server's `elicitation/create` makes a card, but the answer
is never sent back, and the server waits on an open stream.
> - Paperclip does not declare form support, so compliant servers never
ask.
> - This pull request shows the question as a card in the task chat. The
card names the app, and only a person can answer it. When the person
answers, the agent wakes. The agent's repeated call sends the answer to
the app exactly once, for both protocol eras.
> - The benefit: apps that need a choice or a detail from the person now
finish their work. Restarts lose nothing, and no call is left waiting.

## Linked Issues or Issue Description

Refs #21140 (stateless MCP client; this PR closes its "form elicitation"
follow-up). Refs #21263 (MCP Tasks; this PR shares its card code). Refs
#21389 (MCP Tasks follow-ups: the card answer validation item).

**Is your feature request related to a problem?**
When a connected MCP app asks the person a form question during a tool
call, the call stops and cannot continue. A 2026-07-28 server gets a
"needs input" error. An older server gets a card whose answer never
reaches it.

**Describe the solution you'd like**
Show the question in the task chat. Let a person answer, skip, or leave
it. Wake the agent when the question is resolved. Send the answer on the
agent's repeated call, once, with the server's exact state.

**Describe alternatives you've considered**
Holding the tool call open until the person answers. We rejected this:
it holds a stream and a run slot, and it does not survive deploys or
restarts. The plan (`doc/plans/2026-11-13-mcp-elicitation.md`) records
nine design decisions with their alternatives.

## What Changed

- **Plan:** `doc/plans/2026-11-13-mcp-elicitation.md` covers the spec,
the inventory, the decisions, the journey, tests, and risks.
- **New module `server/src/services/mcp-elicitation.ts`.** It is the
single mapping for direct calls and MCP Tasks:
- It turns each spec form field into a real card field: text with
defaults, number, yes/no, single choice (`enum` and `oneOf`), and
multiple choice (`items.enum` and `items.anyOf`).
  - It turns card answers back into typed `content`.
- Answered sends `accept`. Skip sends `decline`. Withdrawn, closed, or
expired sends `cancel`.
- It refuses forms that ask for passwords, keys, tokens, card numbers,
or one-time codes.
  - It keeps app text plain, so links in it are not clickable.
- It seals `requestState` with AES-256-GCM, bound to the company and
connection.
- **New table `tool_mcp_elicitations`** (migration 0479). It is
company-scoped, with one row per question round. It holds a keyed call
identity, the sealed state, the questions and their cards, a claim time,
and a 24-hour expiry.
- **Gateway, stateless servers:**
  - Paperclip declares `elicitation: {form, url}` on task tool calls.
- A form-only `input_required` saves the round, makes the cards, and
stops with a wait message.
- A repeated call before the person answers stops again without
contacting the app.
- After the answer, the repeated call claims the round and sends
`inputResponses` and the exact `requestState`, once.
- A failed send releases the claim. A crash mid-send lets the claim
lapse.
- **Gateway, older servers:**
  - Paperclip declares form support at `initialize`.
  - The first question gets `cancel` at once, so the server never waits.
- When the same question (same words and schema) arrives on the repeated
call, Paperclip answers it from the card.
- **Gateway, mixed rounds and retries:**
  - A form beside a URL step shows the link first.
- A repeated call that stopped for a question runs again in the same
run. It is no longer replayed by the per-run idempotency key.
- **Cards:** the title is "*App* needs your input". The resolver policy
is `human_only`. "Send to *App*" sends the answer. A user comment does
not supersede the card.
- **Activity log:** new events `tool_gateway.elicitation_sent`,
`tool_gateway.elicitation_refused`, and
`tool_gateway.elicitation_expired`. They record the action only, never
the values.
- **MCP Tasks:** task cards use the same mapping and gain the app name,
real fields, and `human_only`. A secret or an unsupported form stays
unsupported.
- **Storybook:** `Tasks and conversations/Questions and approvals/MCP
App Questions` has five states: Asking, Answered, Skipped, Expired, and
TwoQuestions. The fixtures are generated from the server mapper's
output.
- **Docs:** `doc/connections/MCP-ELICITATION.md` (linked from the
README) is new. `MCP-PROTOCOL-VERSIONS.md` and `MCP-TASKS.md` are
updated.

## Verification

- `server/src/__tests__/mcp-elicitation.test.ts` has 16 tests. They pass
on the branch.
- **Red/green:** 9 of the 9 gateway tests fail with master's
`tool-gateway.ts` and `mcp-http.ts`. The 7 pure unit tests cover the new
module.
- The tests use a fake 2026-07-28 server and a fake 2025-06-18 server
that asks over the open stream and waits for the reply.
- Covered: answer, skip (`decline`), withdraw (`cancel`), two questions
at once, a restart while waiting, a crash mid-send, another company with
the same app and arguments, expiry, a refused password field, an agent
that tries to answer (rejected), and capabilities.
- Existing suites: `mcp-tasks`, `mcp-protocol-versions`,
`tool-gateway-service`, `tool-gateway`, and `remote-mcp-connectors`. See
the results comment below.
- Storybook: the five stories render in headless Chromium, and the play
tests pass. The only console errors are the 404s from the backend-less
API calls that every story makes.
- `tsc --noEmit` is clean for `server` and `ui`. `node
scripts/storybook-catalog.mjs` passes.

## Risks

- **Older servers may start asking questions.** Declaring form support
at `initialize` can make older servers ask where they did not before.
Mitigation: the first question always gets `cancel` at once, so no call
hangs, and the agent gets a clear next step.
- **Per-call cost.** Each 2026-07-28 HTTP tool call in a task makes one
indexed lookup for a waiting round. Nothing is added before the first
token or to agent boot.
- **Migration.** One additive table with no backfill, so it is safe to
apply online.
- **Changed Tasks behavior.** A task card that is withdrawn or closed
now sends `cancel` instead of `decline`, as the spec defines. A skipped
card still sends `decline`.
- **Stale cards.** Expiry is checked only when the agent calls again. A
card can sit past 24 hours until the next call. Then it is withdrawn and
the app asks again.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use
(shell, file edits, headless browser), with extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:46:56 -06:00
DottaandClaude Opus 5.5 4879a146fb fix(secrets): stop credential reads from aborting concurrent budget admissions (#21452)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paid model calls (company decisions, fast responses, browser use,
prompt-injection checks) pass a budget admission before they start
> - Admission runs in a REPEATABLE READ transaction and share-locks the
credential's secret row, so a key rotation cannot slip between admission
and the call
> - Every credential read also wrote "Last used" (`last_resolved_at`,
`updated_at`) to that same row
> - When two calls ran at once, one call's credential read aborted the
other's admission with a serialization failure (40001). The retry then
saw the first call's new budget hold and denied the work as
`budget_blocked`
> - This is the flaky `decision-models.test.ts` failure that held Cloud
readiness, and so releases, on 2026-11-13
> - This pull request stops credential reads from writing the secret
row. "Last used" now comes from the access event that every read already
records
> - The benefit is that concurrent paid calls are no longer wrongly
denied or retried, and the release gate no longer fails on this test

## Linked Issues or Issue Description

No public issue exists. The bug is described here.

**What happened?**
Cloud readiness `verify / General tests (server (4/10))` failed on
master commits 6917d0568d and 5ede64f247. The test "admits concurrent
work from the same budget observation and records both obligations"
reported `expected "vi.fn()" to be called 2 times, but got 1 times`.
#21387 replaced the 1-second poll with an unbounded wait. That does not
fix it: one decision really is denied, so the test now waits until the
test timeout.

**Root cause**
I logged the accounting retry loop under load. Each batch of 50 runs had
6 to 10 retries with `could not serialize access due to concurrent
update` on `select … from company_secrets … for share`. The other
decision's credential read had updated the row. In 4 of 5 batches, the
retried admission saw the other decision's committed reservation and
returned `budget_blocked`.

**Expected behavior**
A credential read does not change admission. Two decisions admitted from
the same budget observation both run and both record their obligation.

**Steps to reproduce**
Run the updated test on the old `secrets.ts`. The test makes the second
credential read land after the first admission's snapshot. It then
commits an unrelated hold after both observations. The old code fails
every run with `first settled before its provider call:
{"status":"unavailable","reason":"budget_blocked"}`.

## What Changed

- `secretService` no longer writes `last_resolved_at` / `updated_at`
when it resolves a value. It still records the `secret_access_events`
row, as before.
- `secretService.list` reports `lastResolvedAt` as the later of the
stored column (legacy values) and the newest successful access event. It
reads one row per secret, backwards on the existing `(secret_id,
created_at)` index.
- A read without a consumer context (for example a GitHub API credential
read) now records a `system` access event, so it still counts as "Last
used".
- The removed `UPDATE` also made a credential read wait for a concurrent
rotation to commit. `credential()` in `ai-connections.ts` depends on
that wait to re-read the rotated version instead of handing a run a
consumed single-use token. Its existing version re-read now takes `FOR
SHARE`, so it keeps that wait. `prepareManagedAiRuntime` has its own
before/after freshness read around `runtimeCredential()`; that read
takes `FOR SHARE` too. A share lock creates no new row version and
cannot abort an admission. It adds no statement.
- The first version of this PR throttled the write to once a minute.
Review found that a read crossing the minute could still abort an
admission, so the write is removed instead.
- The concurrent-admission test is now deterministic. It starts decision
A and waits until A is blocked, then starts decision B and waits again.
It commits an unrelated 1-cent hold after both observations, then checks
that both decisions are admitted. A decision that settles before its
provider call fails the test immediately with its result, instead of
hanging.

## Verification

- Red: the new test against the old `secrets.ts` fails on every run with
the `budget_blocked` result above.
- Green: the same test with `repeats: 49` (50 runs) passes 50/50 locally
on a machine at load average 20 to 60.
- Before the change, 3 parallel batches of 50 runs of the old test each
produced a false denial.
- `npx tsc --noEmit -p server`: no errors.
- New `secrets-service.test.ts` test: a resolution leaves the secret row
unchanged, and `list()` still shows "Last used".
- `ai-connections.test.ts` quota test now checks that a read leaves
`updated_at` unchanged and records a success access event.

## Risks

- `updated_at` on a secret no longer moves when the secret is read. It
still moves on every real change.
- "Last used" on the Secrets page costs one indexed lookup per listed
secret.
- Only `list()` derives "Last used". Other endpoints that return a
secret row show the stored column, which is now set only by older
releases.
- No migration. No API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:46:50 -06:00
DottaandClaude Opus 5.5 e67f3bd63a test(environments): add an opt-in live Daytona proof for warm sandbox expiry (#21463)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes, such as Daytona, through reusable
environment leases
> - #21410 made Paperclip stop a warm sandbox when its warm window ends
> - Its tests use a real database but a mocked provider stop, and
staging had no warm reusable sandboxes to observe
> - This pull request adds an opt-in live test that proves the behavior
against real Daytona
> - The benefit is a repeatable live proof of warm expiry, resume, and
recovery after a restart

## Linked Issues or Issue Description

Refs #21410

**What happened?**
#21410 has no live provider proof. Staging creates no warm reusable
sandboxes today, so the deployed change cannot be observed there.

**Expected behavior**
A repeatable live check shows that a warm sandbox stays started inside
its window and stops after it. It also shows that the next turn resumes
the same sandbox, and that a restarted cleanup worker still stops the
sandbox on time.

**Steps to reproduce**
1. Set `PAPERCLIP_LIVE_WARM_EXPIRY=1` and `DAYTONA_API_KEY`.
2. Run `pnpm exec vitest run
src/__tests__/warm-sandbox-expiry.live.test.ts` in `server`.

**Additional context**
The test makes no model turn. It drives the production environment
runtime and the cleanup worker against real Daytona sandboxes.

## What Changed

- New `server/src/__tests__/warm-sandbox-expiry.live.test.ts`. The test
is skipped unless `PAPERCLIP_LIVE_WARM_EXPIRY=1` is set. It uses:
  - the real Daytona plugin and real sandboxes;
  - `environmentRuntime.acquireRunLease` with a 2 minute warm window;
  - `releaseRunLeases(..., "keep_running")`;
- the production `registerEnvironmentCleanupWork` on a real
`createDeliveryWorkCoordinator`, which must wake itself at the deadline.
- The test checks the following:
  - The sandbox is started 20 s before the window ends.
- After the window, the worker stops the sandbox. The sandbox is
stopped, not deleted, and the lease is released as stopped and kept.
  - The second turn resumes the same sandbox (`acquisition: resumed`).
- A new database client, a freshly loaded plugin module, and a new
cleanup worker start in the middle of the window. That worker stops the
sandbox after the window ends, not before.
- At the end, the test deletes only the sandboxes whose labels match its
own company and environment.

## Verification

- Live run on 2026-11-13 against the Daytona org, window 120 s, sandbox
`170bb215…`:
  - 18:39:32.624Z turn 1 lease retained; window ends 18:41:32.624Z
  - 18:41:13Z sandbox `started` (20 s before the window ends)
- 18:41:36Z sandbox `stopped`, auto-delete -1: the worker stopped it 4 s
after the window ended
- 18:41:39Z turn 2 resumed the same sandbox (`resumed`, `started`);
window ends 18:43:39.549Z
  - 18:42:10Z restart: a new worker booted; sandbox still `started`
  - 18:43:45Z sandbox `stopped`, 5 s after the window ended
  - 18:43:45Z fixture sandbox deleted; Daytona returns 404
- Provider calls: one acquire, one resume, two stops, no destroy before
cleanup
- Without the env flag the test is skipped (1 skipped).

## Risks

- Low risk. The change adds a test file only, and the test is skipped by
default. It makes real provider calls only when someone opts in.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, Daytona API).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:42:08 -06:00
DottaandClaude Opus 5.5 ce5f10d534 fix(wake-queue): start a person's saved message after a legacy run is continued from saved work (#21518)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails and cannot resume exactly, Paperclip
continues the task in a fresh run from its last saved work
> - If a person sent a message while the task was blocked, the
fresh-continuation fallback lets that saved message start the next run
(`delivery: saved_input`) and does not queue its own run
> - The release step that promotes the saved message skips the "needs
reconciliation" hold for a native run that the fallback superseded, but
not for a legacy run
> - So for a legacy run the task moves to To do, the person's message
stays saved, and no run starts until a watchdog finds the task idle
> - This pull request applies the same superseded-run rule to legacy
runs in the release step
> - The benefit is that a person's message after a failed run starts the
next run at once, for legacy runs as for native runs

## Linked Issues or Issue Description

**What happened?**
A person sent a message on a task whose last legacy run failed.
Paperclip continued the task from saved work and moved it to To do. The
message stayed saved. No run started until the assigned-work watchdog
queued one later.

**Steps to reproduce**
1. Let a legacy run fail with an active recovery hold, so the task is
Blocked.
2. Post a comment as a person. The wake is saved as
`deferred_issue_execution`.
3. Let the fresh-continuation fallback continue the task. It records
`delivery: saved_input`.
4. Run the deferred receipt and stranded-queue passes.

**Expected behavior**
One run starts and carries the person's message.

**Actual behavior**
No run starts. The release step returns `released` because
`legacyExecutionNeedsReconciliation` is true for the superseded legacy
run.

## What Changed

- `server/src/modules/wake-queue/adapters/postgres.ts`: the release
pre-drain check does not ask for legacy reconciliation when the run is
superseded by a fresh continuation. This matches the existing native
rule in the same function.
- `server/src/services/fresh-continuation-fallback.test.ts`: a red/green
test for native and legacy runs. A person's comment goes through the
real wake path while the task is blocked, the task continues from saved
work, and exactly one run must start with the comment.

## Verification

- Red: before the fix, the legacy case fails. The saved wake stays
`deferred_issue_execution` and no run starts. The native case passes.
- Green: after the fix, both cases pass.
- `pnpm verify` for the affected selection.

## Risks

- Low risk. The change applies only when the fresh-continuation fallback
has already marked the run superseded. That decision already records
that the old run is not replayed, and the successor runs as a fresh
turn. Other callers of `legacyExecutionNeedsReconciliationWithEvidence`
(retry scheduling, recovery) are not changed, so a superseded run still
cannot be retried automatically.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:40:42 -06:00
DottaandClaude Opus 5.5 f610e6a880 fix(recovery): replace a queued run the dispatcher keeps passing over instead of counting it as a pending path (#21502)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The assigned-work sweep is the backstop that makes sure every open
agent task has a live run, a pending path, or a named wait on a person
> - It counts a task with a queued run as healthy. A queued run that the
dispatcher declines on every pass is never started, so the task waits
forever and the sweep never looks again
> - On a hosted instance one agent had more than 30 queued continuations
that the claim declined on every pass. The sweep saw each task as
healthy
> - This pull request makes the sweep recognize a queued run that the
dispatcher keeps passing over, report it, and replace it once with a new
continuation when it is automatic work
> - The benefit is that an automatic task cannot stay stuck behind a run
that never starts, while a person's work is never cancelled

## Linked Issues or Issue Description

Refs #21391, #21349, #21433, #21394, #21443, #21450

**What happened?**

An agent had more than 30 queued automatic continuations
(`issue_continuation_needed`) that never started. The claim declined
each one on every dispatch pass. #21391 fixed that specific cause in the
chat-control gate. But the sweep still trusted any queued run as a
pending path. A run that the claim declines for any persistent reason
keeps the task "healthy" forever.

**Expected behavior**

A queued run that the dispatcher keeps passing over is not a healthy
pending path. The sweep reports it. For automatic work it cancels the
run through the run lifecycle and queues one new continuation in its
place. For a person's work it reports the run and leaves it.

**Steps to reproduce**

1. Queue an automatic run for a task and make the claim decline it
persistently (for example, a chat-control admission marker that does not
verify).
2. Let the run wait for more than one hour while its agent starts other
runs.
3. Run the assigned-work sweep for the task. Before this change, it
reports the task as `live_run` and does nothing.

## What Changed

- New violation `queued_run_stalled` in
`recovery/assigned-work-invariant.ts`. A queued run is stalled when all
of these are true (read from the database, on the database clock):
- it has been queued for longer than `QUEUED_RUN_AGING_MS` (60 minutes,
from `queued-run-dispatch-order.ts`);
  - it is the oldest queued run of its agent;
- its agent started more than `OLDEST_DISPATCH_SHARE` other runs after
the run became its oldest queued run: after the later of the run's aging
time and the moment the last older run of the agent started or finished.
- A stalled run is not a live run, and it does not let the task's
interruption projection ("working") count as a pending path. Its own
wake is not counted as a pending wake. A stalled run on a blocked task
still counts as live, so the block decides first.
- `listSuspects` lets a task through when its only active run is a
stalled queued run.
- `act()` handles the violation after all existing early returns:
- person work (the #21349 source-chain rule through
`listPersonOriginSourceRunIds`, which this change exports, or an
explicit user continuation): reported, never cancelled;
- a replacement that stalls again (`replacesStalledRunId` in its
context): reported, not cancelled again;
- both log one warning per run, with the run, agent and how long it
waited, and look again after `CONTINUATION_CAP_WINDOW_MS` (one hour);
- automatic work: the existing claim under the task row lock re-checks
the facts and the shared hourly cap and spacing. Inside that claim,
before the person check, it reads the run's row version (`xmin`) and the
number of wakes attached to it. Then the run is cancelled through the
run lifecycle (`setRunStatusFromLive` from `queued`, with that version
and wake count as its condition; wake cancelled, run event,
`releaseIssueExecutionAndPromote` with `suppressImmediateRecovery`). A
person's wake that joins the run after the check, or a claim that starts
it, changes one of them, so the cancel does nothing. Then one
continuation wake is queued with `previousRunId` and
`replacesStalledRunId` set to the cancelled run. If the cancel does
nothing, the unpublished claim entry is removed and nothing is queued.
- New `heartbeat/stalled-queued-run.ts` builds that cancel from the run
lifecycle functions. `heartbeat.ts` and `recovery/service.ts` pass it to
the sweep.
- Plain activity text: "Continued automatically: this task had waited
over an hour to start while its agent started other work, so Paperclip
queued it again."

### Why this signal

- #21433 keeps refused claims and claim back-off in memory only
(`declinedAgedClaimAt`, `claimBackoffs` in `heartbeat/queue.ts`).
Another server cannot read them. Run rows can be read from any server,
on the database clock.
- After #21349, an aged run ranks no lower than ordinary work, and the
oldest-run share starts the oldest ready queued run at least once in
every `OLDEST_DISPATCH_SHARE` starts. So when a run has been its agent's
oldest queued run since after the aging bound, and its agent started
more runs in that time than the share allows, the dispatcher offered the
run a slot and the claim declined it. Starts that happened while older
runs were still waiting do not count: those older runs were getting
their turns.
- A run behind a full agent sees no such starts and is left alone. Under
a slow backlog only the oldest queued run can match. So the younger
queued runs keep their place in line.
- Starts are counted only after the run reached 60 minutes of age, not
after the current server's boot. 60 minutes is longer than a release
swap overlap. So a run that only waits for a new server to reattach to
its Runners does not match. #21450 will add a reattach-phase gate before
this path.

### Why this does not set `retryOfRunId`

An earlier plan added `retryOfRunId` to the sweep's continuation so the
chat-control gate could follow it. An audit of its readers found three
changes in behavior: the admission-time chat-control check would start
to refuse sweep wakes; the claim would wait on the source run's cleanup
(a possible wait forever); and the agent prompt would call the run "a
Retry" that "re-delivers the wake". #21391 already makes these runs
claimable. So this change leaves `retryOfRunId` unset, as today. The
replacement links its source with `previousRunId`, as the sweep already
does.

### Questions from review, answered from the code

- **How often is the invariant pass triggered, and can two passes
overlap?** The `assigned-work-invariant` delivery queue
(`recovery/work.ts`, `scoped-deadline-work.ts`) runs on database-work
notifications, on each task's own deadline (`retryAt`), and on a 30 s
reconstruction of the suspect list. Passes are at least 1 s apart
(`minimumPassIntervalMs` default). In one process, passes of the queue
never overlap (`running` guard in `delivery-work-coordinator.ts`). Two
servers can sweep at the same time. Each repair is serialized by the
claim under the task row lock (`FOR UPDATE` in `claimContinuation`),
which re-reads the facts and the spacing. This change does not add a new
loop.
- **Can two runs or no run exist for one task?** Two runs: no. The
claim's spacing makes a second server back off, and the cancel happens
before the new wake (a queued run on the task would otherwise absorb the
wake). No run: there is a short gap between the cancel and the new wake,
because wake admission locks the task row in its own transaction. If the
server stops in that gap, the task has nothing pending, and the next
pass repairs it as `no_pending_path` under the same cap.

## Verification

- `server/src/__tests__/assigned-work-stalled-queued-run.test.ts` (new,
embedded Postgres, real `createHeartbeatQueue` admission and
`claimQueuedRun`):
- an automatic run that the real claim declines persistently, whose
agent started runs after it aged: reported, cancelled, exactly one
replacement, and the real claim starts the replacement;
  - a run behind a full agent: left alone, not a suspect;
  - a person's declined Retry: reported, not cancelled, one warning;
- a replacement that stalls again: not cancelled on two checks, one
warning, next look one hour later;
- two sweep instances on the same task at the same time: one
replacement;
- three aged runs queued in order, the first two get their turns slowly:
the third is not stalled when it becomes oldest, and is stalled only
after more than four further starts pass it over;
- a person's message joins the run (real wake admission) between the
claim and the cancel: the run is not cancelled, there is no replacement,
and the person's wake stays attached to the run;
- the run starts between the claim and the cancel: no cancel, no
replacement.
- First round, red before the fix: 4 failed, 1 passed (the full-agent
case passes before and after). Green: 5 passed.
- Review round: the three new tests on the previous head: 2 failed (the
backlog case and the person-join case), 1 passed (the started-run case,
which the `queued` check already covered). Green: 8 passed.
- Existing files run one at a time: `assigned-work-invariant.test.ts` 31
passed, `assigned-work-sweep-round-trips.test.ts` 2 passed,
`failed-continuation-continues-again.test.ts` 28 passed,
`recovery-wake-herd.test.ts` 35 passed,
`saved-work-recovery-overdue.test.ts` 4 passed.
- `pnpm --filter @paperclipai/server exec tsc --noEmit`: clean.
- `pnpm verify`: not completed locally (the machine is heavily loaded
and the run was stopped when the code changed). Required CI supplies the
broad coverage.

## Risks

- A task's oldest queued automatic run is cancelled and queued again at
most once per run. Its replacement starts at the back of the line. Only
a run that the dispatcher already passed over can match.
- A replacement that stalls again is only reported, so a persistent
claim refusal cannot cause repeated cancellations.
- The new subqueries run only for queued runs older than one hour. They
use the existing `(company_id, agent_id, started_at)` index. Healthy
runs and the path before `turn.start` do not change.
- #21443 (automatic-run brake) and #21450 (reattach first) touch
`assigned-work-sweep.ts`. The new path comes after every early return
that `act()` has today, so their gates can go before it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). A subagent on the same model
audited the `retryOfRunId` readers.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:40:38 -06:00
Devin FoleyandPaperclip 0a96f972e1 Connect Tailscale networks through device sign-in (#21526)
## Thinking Path

> - Paperclip manages AI agents and the environments where they run.
> - SSH environments can reach machines through a company Tailscale
connection.
> - The initial connector requires a managed OAuth app that Tailscale
limits to its own tailnet.
> - Customers need to connect their own tailnets without registering an
app or supplying a key.
> - This pull request uses Tailscale's normal device sign-in for new
connections.
> - Customers approve the Paperclip device and then select their
machines in SSH environments.

## Linked Issues or Issue Description

**Problem or motivation**

The current Tailscale enrollment path cannot serve independent customer
tailnets through the documented self-service OAuth app model. It also
requires Cloud broker configuration and pilot admission before a
customer can start.

**Proposed solution**

Start a private Tailscale device in the tenant and open its normal
sign-in link. Save its encrypted device state before reporting a
successful connection. Retain the existing SSH machine selection and
agent account behavior.

**Alternatives considered**

Auth-key entry would require customers to copy a credential. A public
OAuth integration still needs a confirmed registration path. Device
sign-in uses a documented Tailscale flow without either requirement. See
[device
authentication](https://tailscale.com/docs/features/tsnet#device-creation-and-authentication)
and [OAuth app
limitations](https://tailscale.com/docs/features/oauth-apps).

**Related PR search**

Searched GitHub PRs for `tailscale` and `device sign-in`, and issues for
`tailscale`. This follows the merged network connector in
[#20617](https://github.com/paperclipai/paperclip/pull/20617) and
preserves the SSH agent-account support in
[#21248](https://github.com/paperclipai/paperclip/pull/21248). The open
[#15648](https://github.com/paperclipai/paperclip/pull/15648) proposes
storing an OAuth client and checking its health; this change instead
enrolls a device through interactive sign-in so customers do not need an
OAuth app.

## What Changed

- New connections use device sign-in. Existing connected devices and
pending legacy OAuth callbacks remain supported.
- Pending sign-ins are bound to their company and initiating user. They
have a fixed ten-minute deadline. Device state uses server-owned vault
secrets with trusted version and digest checks.
- The active Tailscale CLI waits for consent and device approval.
Cancellation, expiry, secret deletion, and stale completion cannot
restore a cancelled connection.
- The connector stays open during sign-in. It provides a link when
popups are blocked and shows device approval separately. Progress checks
stop on completion, failure, expiry, or unmount.
- Shared response types, OpenAPI contracts, SDK source pins, and the
API/setup documentation describe the new flow. No database migration or
Cloud provider configuration is required.

## Verification

- 109 focused tests cover network processes, the real database and
encrypted vault, routes, UI, and legacy connection behavior.
- Server and UI typechecks pass.
- Follow-up `29cf296a597`: reproduced the missing OpenAPI route failure
locally. All 70 OpenAPI and Tailscale route tests and server typecheck
pass after synchronizing the API contract.
- Follow-up `79de05f38b9`: the transient-recovery regression failed
before the fix; all 20 connector UI tests and UI typecheck pass after
it. The query-error policy check reports no findings.
- The public SDK input registry check passes.
- [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/43535594257/attempts/2)
passed on the final head, `29cf296a597e470d432b84767050bc539caa280e`: 57
successful checks, no failures or pending checks. The interrupted
canary, E2E shard, and serialized-suite jobs passed on rerun. Greptile
is 5/5 on this head with no unresolved review threads.
- `pnpm verify --explain` exits 2 because the documentation change
selects broad CI coverage. Focused local checks passed; broad CI passed
on `29cf296a597e470d432b84767050bc539caa280e`.
- Live local qualification on
`29cf296a597e470d432b84767050bc539caa280e`: native device sign-in
completed with interactive user approval. An isolated Ubuntu 24.04 ARM64
EC2 machine appeared in discovery and passed SSH pairing and workspace
probes. The assigned Claude Code agent used a Paperclip-managed
subscription, created `hello_world.txt` containing exactly `Hello,
world!` plus a newline, posted a completion comment, and marked the task
done. An independent read through the private connector verified the
bytes, remote user, and hostname. The remote machine had no stored
Claude login and no leftover temporary credential directories.
- Restarted the local Paperclip server and verified the same Tailscale
device identity, online peer discovery, successful SSH probe, and
unchanged file without new consent. Temporary public SSH ingress was
removed.
- Local qualification covers one user tailnet and the final Core
revision. Browser UI acceptance, two unrelated tailnets, a hosted
restart, and an agent task on the final staging revision are not yet
verified.

## Risks

- The enrollment method changes. It must pass the live qualification
above before broad rollout claims.
- Existing encrypted device state and the vault master key must survive
a restart or redeploy. Device expiry or removal still requires
reconnecting.
- The existing one-serving-process-per-instance constraint remains.
Concurrent replicas need durable network ownership.
- Pending setup uses bounded progress requests and an event-driven CLI
with a deadline. It adds no idle database scan.

## Model Used

OpenAI GPT-6 Astra (`gpt-6-astra`) through Codex in T3 Code, with xhigh
reasoning. Context window size was not exposed by the runtime. No
subagents were used.

## Checklist

- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] My branch name describes the change and contains no internal
ticket or instance details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 17:38:55 -08:00
DottaandClaude Opus 5.5 cefef8f11d perf(workspaces): inspect only hinted workspaces after a scoped commit (#21519)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server cleans up execution workspaces whose task tree is done. A
background sweep (`terminal-workspaces`) finds these workspaces.
> - Every workspace write and every task status change sends a hint to
this sweep. Each hint starts a full rotation over every open candidate
workspace, and each candidate costs a recursive read of its task tree.
> - Runs write workspaces all the time, so with live runs the sweep
never stops, and its cost grows with the number of open workspaces.
> - On a hosted instance with 5 live runs, the task tree read ran 11.6
times per second. On a local copy with run churn it ran 81 to 101 times
per second.
> - This pull request inspects only the workspaces that a hint names or
reaches. Full rotations stay for startup, saved deadlines, failures and
hints without a scope.
> - The benefit is that a workspace change costs a few statements, not
one tree read per open workspace.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The terminal workspace sweep (`registerTerminalWorkspaceWork` in
`server/src/services/terminal-workspace-work.ts` and
`sweepTerminalWorkspaces` in
`server/src/services/execution-workspaces.ts`).

**Current behavior**

Producers send hints such as `workspace:<id>` and `issue:<id>`. The
worker ignores the scope: each settled hint wakes the queue, and the run
starts or continues a full keyset rotation over all candidate
workspaces, 50 per page with 1 second between pages. A hint that arrives
during a rotation also requests a rescan, so a new rotation starts after
the current one. Every candidate with an open task tree costs one
recursive tree read and is skipped.

**Proposed behavior**

The worker keeps the workspace ids and task ids from scoped hints. When
the queue runs only because of such hints, it calls
`sweepTerminalWorkspaces` with `targets`. The sweep then reads only
these candidates:

- the workspaces named by `workspace:` hints;
- the workspaces whose source task is the hinted task or one of its
ancestors (the tree of a workspace is its source task and its
descendants);
- the execution workspace of each hinted task.

A targeted inspection does not move the rotation cursor or its boundary
and does not clear saved deadlines. The per-workspace checks are the
same code as in a rotation.

A full rotation still runs at startup, when a saved deadline is due,
when a hint has no scope or an unknown scope, while a rotation is in
progress, and after a failed targeted inspection.

**Reason and benefit**

Background work must not grow with the backlog of open workspaces. With
4 candidate workspaces, a hint for one task tree now reads 1 tree
instead of 4. A hint during a rotation no longer starts another full
rotation.

## What Changed

- `server/src/services/terminal-workspace-work.ts`: parse scoped hints
into targets. Run a targeted inspection when only scoped hints are
pending. Keep the full rotation for startup, due deadlines, unscoped
hints, continuation pages and failures.
- `server/src/services/terminal-workspace-work.ts` also falls back to a
full rotation when more than 50 workspaces wait for inspection. It keeps
the 1-second pause between rotation pages and runs targeted inspections
between them.
- `server/src/services/issues.ts`: moving a task also hints its former
parent. Removing a task also hints its parent and its execution
workspace. Without this, a former parent's workspace could never be
reached.
- `server/src/services/execution-workspaces.ts`:
`sweepTerminalWorkspaces` accepts `targets`. A targeted page that fills
its limit requests a full rescan. and filters the candidates to the
targeted workspaces. A targeted inspection leaves the rotation state
alone. `terminalWorkspaceWorkSettled` takes `{ targeted: true }` from
the worker, and then a scoped hint does not request a full rescan.
Direct callers keep the old rescan behavior.
- `server/src/index.ts`: pass the new option through the existing
wrapper.
- Tests in `server/src/__tests__/terminal-workspace-work.test.ts` and
`server/src/__tests__/execution-workspaces-service.test.ts`.

## Verification

- New test `inspects only the workspaces a hint reaches, however many
trees stay open`: 3 workspaces with open trees and 1 done tree whose
child task is hinted. Before the change the targeted call checks 4
workspaces. After the change it checks 1, reads 1 tree, and archives the
done workspace.
- New test `inspects only hinted workspaces after a commit, and keeps
unscoped commits and deadlines on the full rotation`: before the change
the worker never passes targets. After the change a scoped commit runs a
targeted inspection, and an unscoped commit and a due deadline still run
a full rotation.
- `npx vitest run src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/git-workspace-expiration.test.ts` (in `server/`) pass.
- New tests `falls back to the full rotation when more workspaces are
hinted than one inspection reads`, `keeps the pause between rotation
pages while hinted workspaces wait` and `hints the former parent and
workspace when a task moves or is removed`. Each fails on the first
commit and passes on the head.
- `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all`. The fixture had 1 company, 60 agents, and open
tasks scaled from 150 to 5,000. Recovery re-dispatched runs to agents
that fail at once, and every run creates or updates an execution
workspace, so the hints keep coming. Each window is 180 seconds and
starts 150 seconds after the server is ready. The machine was shared.

The table gives the rate of the task tree read (`select id, status,
completed_at, cancelled_at, updated_at from issues ... WITH RECURSIVE
issue_tree`):

| Open tasks | Before (per s) | After (per s) |
| --- | --- | --- |
| 150 | 17.56 | 0.93 |
| 1,500 | 109.83 | 5.16 |
| 5,000 | 113.82 | 4.40 |

Total statements and transactions per second depend mostly on how fast
the failing runs cycle in the fixture, so they are noisy. Before →
after: 343 → 287 at 150, 1,647 → 1,562 at 1,500, 1,530 → 1,678 at 5,000
statements/s. In that window the 5,000-task run after the change also
had 10% more run events. With no activity, the server issued 0
statements before and after the change.

## Risks

- A workspace that waits on something without a hint is no longer
inspected as a side effect of unrelated activity. The known cases keep
their path: a workspace held by an active run keeps its past-due
deadline, so the 5-minute deadline rotation inspects it again; cooldown,
compaction, Git idle and retry deadlines still start full rotations. A
delivery that changes outside Paperclip without a work product change
was already only found by a later rotation.
- A failed targeted inspection requests a full rotation, so a lost
target is recovered by the rotation.
- Low migration risk. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:04:35 -06:00
DottaandClaude Opus 5.5 3cd671575d perf(attention): scope native finalization retention hints to the run's task graph (#21521)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server archives old attention items (reviews, blocked tasks,
decisions, failed runs) in a background queue, `attention-retention`.
> - A hint for a whole company makes this queue reload every attention
candidate of the company, 100 per page. Each page also reads review
attention, blocker graphs and source evidence.
> - Each native run finalization sent such a company hint, although a
run changes only the attention items of its own task graph.
> - On a hosted instance with about 1,400 open tasks and live native
runs, the decision retention read ran 26.7 times per second. Native run
finalization is one of the producers that sent a company hint on every
run.
> - The same finalization already sends the issue-scoped hint through
`notifyNativeStatusReassessment` when the run has a task or a run id.
This pull request sends the company hint only when the run has neither.
> - The benefit is that a finished run reloads only the candidates of
its task, its ancestors and the tasks it blocks, not every candidate of
the company.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The attention retention hint that native run finalization sends
(`notifyNativeFinalizationWork` in
`server/src/services/native-finalization-notifications.ts`).

**Current behavior**

`notifyNativeFinalizationWork` calls `notifyAttentionRetentionWork(tx, {
companyId })`. The `attention-retention` queue has no reconstruction
interval, so each finalization reloads all eight candidate kinds of the
company, page by page, with the batch eligibility reads for each page.

**Proposed behavior**

`notifyNativeFinalizationWork` sends the company hint only when the
scope has no `issueId` and no `runId`. Otherwise
`notifyNativeStatusReassessment`, which the same function already calls,
sends `notifyIssueAttentionRetentionWork(tx, { companyId, issueId, runId
})`. That helper resolves the task of the run, adds its parents and the
tasks it blocks (at most 100), and sends a hint with these task ids.
When the run has no task, or the graph has more than 100 tasks, the
helper sends the company hint itself.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. The
attention items that a run can change are the failed run, the review,
the recovery actions, the decisions and the blocker items of its task
graph. The issue-scoped hint covers all of them, and the ordinary task
write path uses the same helper.

## What Changed

- `server/src/services/native-finalization-notifications.ts`: send the
company attention retention hint only for a finalization without a task
or run.
- `server/src/__tests__/attention-retention-work.test.ts`: new test
`scopes native run finalization to the task graph of the run`.

## Verification

- The new test sends the finalization hint once with a run id and once
with a task id. Before the change both hints are `{ companyId }`. After
the change both hints name the task and its parent.
- `npx vitest run src/__tests__/attention-retention-work.test.ts` and
the native work tests (`native-result-commit-work`,
`native-environment-settlement-work`, `native-run-projection-work`,
`native-recovery-notice-work`, `retained-native-session-cleanup-work`,
`native-status-reassessment-work`, `muse-terminal-settlement`,
`ownership-reconciliation-report`) pass (in `server/`).
- `pnpm check:module-boundaries` passes.

## Risks

- No new reads. The issue-scoped hint was already computed in the same
transaction by `notifyNativeStatusReassessment`.
- An attention item outside the task graph of the run is not reloaded by
this hint. Agent status, budget and company changes keep their own
hints.
- Low risk. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:04:30 -06:00
DottaandClaude Opus 5.5 a82fa96540 perf(run-events): collect deltas that arrive while a batch commits into one next batch (#21520)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane port commits streamed text deltas of a run in
batches (#21460): the first text at once, later deltas after at most 250
ms
> - Each batch is cut when its timer fires, even when the previous batch
of the same run has not committed yet
> - When the database is slow, every run then queues one small batch per
250 ms behind its slow commit. The number of statements stays the same
exactly when the database has the least room for them, and the queue
grows
> - This pull request cuts a batch only when the previous batch of the
run has committed. Deltas that arrive while a batch commits all join the
next batch
> - The benefit is fewer, larger statements under load, with no change
when the database keeps up, and no longer wait for any delta

## Linked Issues or Issue Description

Refs #21460 (batched deltas). Refs #21511 (one statement per delta
batch). Refs #21418 (agent scaling plan and load model).

**What existing behavior does this improve?**
How `PaperclipControlPlanePort` groups queued text deltas into batches
when a batch commit is slower than the batch time.

**Subsystem affected**
Server run event ingest:
`server/src/services/native-runtime/paperclip-control-plane-port.ts`.

**Current behavior**
`#flushStreamingEvents` takes the queued deltas when it runs (on the 250
ms timer, at 100 deltas, or before any other event) and chains their
commit behind the previous batch. If one commit takes two seconds, the
run queues about eight batches, each one statement, behind it.

**Proposed behavior**
`#flushStreamingEvents` chains one cut behind the previous batch and
takes the queued deltas when that cut runs. While a cut waits, later
flush requests join it. A run has at most one batch committing and one
cut waiting.

**Reason and benefit**
Under load, the number of delta statements per run falls to about one
per commit time instead of one per 250 ms. When commits are faster than
250 ms, nothing changes.

**Breaking changes**
None. Rows, receipts, order and the first-text rule are the same.

## What Changed

- `#flushStreamingEvents` queues one cut on the run's commit chain
(`#streamingFlushQueued`). The cut takes every delta queued at that
time, sets the first-text flag, commits the batch, and then releases the
pending count of that batch.
- A flush while a cut waits returns the same chain, so a caller that
waits for it (every other event, `checkpointSession`, `completeRun`,
`replayEvents`, `provesNoRunProviderWork`) still waits for every delta
queued before it.
- A batch can now hold more than 100 deltas when commits are slow. The
100-delta rule still asks for a flush; the batch is one statement with a
fixed number of bind parameters (#21460, #21511).

## Verification

- Red/green: new port test "collects the deltas that arrive while a
batch commits into one next batch, before the next other event"
(`paperclip-control-plane-port.test.ts`). A transaction holds the run
row, so the first batch waits. Six more deltas arrive 15 ms apart with a
5 ms batch time, then a non-streaming event. Before: 8 inserts (one per
timer). After: 3 inserts (the waiting batch, one batch of six, the other
event). Rows are stored 1–9 in order and every receipt resolves.
- The whole port suite with its conformance suite passes: 101 tests.
Other suites, one file at a time: `native-observer-replay` (1),
`paperclip-semantic-conformance` (1), `native-interaction-bridge` (7),
`chat-native-progress-work` (35). I ran the four streaming port tests
two more times; they pass each time.
- `pnpm --filter @paperclipai/server run typecheck` passes. `pnpm
check:module-boundaries` passes.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). The model applies the port's batching rules itself, so for
these runs I changed my local copy only (not in this pull request): with
`DELTA_COALESCE=1`, `flushDeltas` queues one cut on the agent's append
chain and takes the queued deltas when the cut runs, as the port now
does. All rows run with #21511. Default pool of 10 connections; embedded
PostgreSQL on the same, shared machine; runs one after the other with a
pause between them. No run had a failed operation, a failed checkpoint
save or a lost lease.

| Agents | Batching | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | this change | 60 s | 9.7 | 0.51 | 3,832 | 151 / 3,087 | 12,000 /
12,000 | 3 / 3,994 |
| 400 | cut on timer (before) | 60 s | 12.7 | 0.74 | 5,465 | 1,129 /
6,040 | 18,719 / 24,000 | 5,721 / 15,238 |
| 400 | this change | 60 s | 12.5 | 0.78 | 7,010 | 892 / 3,131 | 23,993
/ 24,000 | 1,896 / 4,193 |
| 600 | cut on timer (before) | 60 s | 10.0 | 0.69 | 6,158 | 2,357 /
9,483 | 18,771 / 36,000 | 2,280 / 30,985 |
| 600 | this change | 120 s | 11.4 | 0.72 | 6,730 | 1,147 / 6,335 |
61,158 / 72,000 | 11,678 / 21,212 |
| 600 | this change (repeat) | 120 s | 13.6 | 0.74 | 6,498 | 1,225 /
6,057 | 58,984 / 72,000 | 11,465 / 25,361 |
| 800 | this change | 60 s | 11.5 | 0.80 | 6,228 | 2,054 / 7,186 |
24,877 / 48,000 | 24,544 / 31,331 |

The process does more work per second with this change (more other
events finish), so statements per second rise; delta lateness falls. 600
agents are still not carried reliably: other events fall behind. #21528
removes two statements from each such event and one from each checkpoint
save; its description has the stacked results.

## Risks

- A delta never waits longer than before: it already waited for the
previous batch of its run. When the database keeps up, batches are the
same as before.
- Under load, one batch can be large. The live view then gets one larger
step instead of several small steps behind the same wait.
- A failed batch now rejects more deltas at once (every delta that
joined it). The runner resends them, as before; each resend is
idempotent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:00:08 -06:00
DottaandClaude Opus 5.5 0bba04675c perf(run-events): append in-order native events in one statement (#21511)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - After #21460, text deltas commit in batches, but each batch still
takes five statements: BEGIN, the run lock, a replay probe, one insert,
and COMMIT. A single event without a wake takes four
> - In the agent scaling load model, the process issues about 36
statements per agent per second and saturates its one thread between 200
and 400 agents
> - This pull request writes a batch of in-order native events, or one
such event, with one statement that commits on its own
> - The benefit is about half the statements per agent, less CPU per
agent on the main thread, and less time on a pool connection, with the
same rows, receipts, cursor and locks

## Linked Issues or Issue Description

Refs #21460 (batched deltas). Refs #21435 (source cursor). Refs #21418
(agent scaling plan and load model).

**What existing behavior does this improve?**
The append of native source events that continue their source's stored
cursor: batched text deltas, the first delta after another event, and
every other native event that registers no wake.

**Subsystem affected**
Server run event ingest: `server/src/services/heartbeat-run-events.ts`,
and the port comment in `paperclip-control-plane-port.ts`.

**Current behavior**
A delta batch takes BEGIN, a run lock read (`for no key update`), a
replay probe, one insert, and COMMIT: five statements. One delta that
commits at once takes four. In the load model at 200 agents the process
sends about 6,700 statements per second at event-loop utilization 0.72.

**Proposed behavior**
The same append is one statement. Its update of the run row takes the
same row lock. It requires the stored cursor to end just before the
batch, requires the `next_event_seq` that its own snapshot read, and
finds no replayed event. Then it moves the sequence and the cursor and
inserts every row. If any check fails, it writes nothing and the append
takes the existing locked transaction.

**Reason and benefit**
Fewer statements free the main thread and the shared pool for other
agents. In the load model at 200 agents, statements per second fall from
about 6,700 to about 3,300 and event-loop utilization falls from 0.72 to
0.56.

**Breaking changes**
None. No schema change. Rows, receipts and the cursor are the same.
Servers from before this change keep working on the same runs.

## What Changed

- `appendNewNativeSourceEvents` (`heartbeat-run-events.ts`) is one
statement: a CTE reads `next_event_seq` from the statement snapshot, an
`update heartbeat_runs ... returning` locks the run row and moves
`next_event_seq` and the source cursor, and an `insert ... select from
jsonb_array_elements($9)` writes every row. The text is fixed, with nine
bind parameters for any batch size.
- **Why the `next_event_seq` check.** A statement that waits for the run
row lock keeps the snapshot it took before the wait. Its replay probe
cannot see events that the lock holder committed. Every event append
(native, ordinary, the run coordinator store, the retry queue) moves
`next_event_seq` in its own transaction. So the update requires the
value that its snapshot read. PostgreSQL re-checks that condition on the
newest row version after the wait, and any append committed meanwhile
makes the statement write nothing. The unique indexes on `(run_id,
source_event_id)` and `(run_id, source_instance_id, source_seq)` are a
second guard: a unique violation from this statement also falls back.
- A source without a cursor starts at 0. The probe still finds any
stored event of that source, so a run that an older server wrote falls
back and recounts.
- `appendHeartbeatRunEventBatch` and `appendHeartbeatRunEvent` try the
one statement first for native events without a wake, outside a caller's
transaction. Every other case (a replay, a gap, an event with a wake, a
cursor that lags, a run changed meanwhile, a binding mismatch, a run
deleted meanwhile, a caller's transaction) takes the existing
transaction, unchanged.
- Inside a caller's transaction the append keeps its SAVEPOINT path. A
failed statement there would abort the caller's transaction, and the
fallback could not run.
- `assertWritable` runs before the statement is sent and after it
returns (see Risks).

## Verification

- Red/green on embedded PostgreSQL. I ran the new tests against the
previous `heartbeat-run-events.ts` and saw 9 of them fail; all 25 tests
in the file pass with the change
(`server/src/__tests__/heartbeat-run-event-atomic-append.test.ts`):
- One statement for an in-order event and for a batch (before: 4 and 5).
- A writer revoked before the send sends nothing; a statement that fails
on its third row stores no row and no sequence; the resend commits.
- A writer revoked while the statement runs: the rows stay, the receipt
is rejected, and the resend is a replay with the same rows.
- A batch waits for Stop on the run row and appends after it; a batch
and a single event that wait for a run deletion fail with
`heartbeat_run_event_binding_mismatch` and store nothing.
- An older server stores events 2 and 3 while it holds the run row,
without the cursor: the waiting batch 2–4 falls back, returns duplicate,
duplicate, committed, and the cursor ends at 4.
- Two connections append the same 200-event stream in batches of 10:
each event is stored once, in source order, sequences 1–200 without a
gap, one `committed` and one `duplicate` receipt per event.
- A 7,000-event batch (more than any bind parameter limit at 10
parameters per row) is one statement. Before: it fell back to one
statement per event and timed out at 15 s.
- A run without a cursor starts at its first event; a run with events
from an older server replays and then continues.
- Equivalence: 12 random streams of 30 steps (in-order batches, gaps,
replays of earlier ranges, fills of gaps, ordinary events, conflicting
replays). Each step goes to two runs, one through the new path and one
forced through the one-by-one locked path. Receipts, errors, rows,
`next_event_seq` and cursors are equal at every step.
- Mutation check: with the `next_event_seq` check and the
unique-violation fallback removed, the older-server race test fails.
- The port suite (`paperclip-control-plane-port.test.ts`, with the
package conformance suite) passes: 125 tests. Two tests changed: the
streamed-event test now expects one statement and the new revocation
behavior, and the batch test checks that the five deltas are one
statement sent before the boundary event.
- Other focused suites pass, one file at a time:
`heartbeat-run-event-sequencing` (5), `native-event-boundary-golden`
(1), `native-observer-replay` (1), `paperclip-semantic-conformance` (1),
`native-interaction-bridge` (7), `chat-native-progress-work` (19),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51).
In 2 of 3 runs with this change, all 51 tests passed but the file's
`afterAll` (closing the side pools and the embedded database) hit its 30
s timeout on this busy machine; the third run passed cleanly, and one
run on `origin/master` passed. The native events in that file all
register a wake, so they do not take the new path.
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model: `scripts/benchmark-agent-scaling-load.ts` as merged in
#21418 (per-source counts stop when the window closes). I did not change
its batching rules for this pull request. Model rates: 20 deltas/s, 1
other event/s, 0.3 API requests/s per agent; default pool of 10
connections; embedded PostgreSQL on the same, shared machine. I ran
"before" (`origin/master`) and "after" one after the other, with a pause
between runs, and I give the load average at the start of each run. The
machine was busy with other work (load 10–15 on 18 cores), so compare
rows that ran close together. No run had a failed operation, a failed
checkpoint save or a lost lease.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | after | 60 s | 12.7 | 0.74 | 5,465 | 1,129 / 6,040 | 18,719 /
24,000 | 5,721 / 15,238 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | after | 60 s | 10.0 | 0.69 | 6,158 | 2,357 / 9,483 | 18,771 /
36,000 | 2,280 / 30,985 |
| 600 | after, `DELTA_BATCH_MS=0` | 60 s | 10.8 | 0.68 | 8,014 | 36,135
/ 52,222 | 36,000 / 36,000 | 698 / 3,422 |

At 200 agents an earlier pair gave 0.61 → 0.43 event-loop utilization
and 7,051 → 3,210 statements/s. One process with this change alone still
does not carry 600 agents: at 400 and above the 10-connection pool and
the shared database are the limit (event-loop utilization stays below
0.75 while other events fall behind). #21520 and a third pull request
for the remaining statements of other events and checkpoint saves
continue from here; their description has the stacked results.
- Statements: a delta batch 5 → 1; one delta or one native event without
a wake 4 → 1. Unchanged: an event with a wake (6) and a checkpoint save
(5).

## Risks

- **`assertWritable` window (read first).** Before, the writer check ran
inside the transaction, before the first statement and before COMMIT. A
revocation while the statements ran rolled the append back. Now a
revocation while the one statement runs does not roll it back: the rows
commit, and the caller gets the error instead of the receipt, so no
observer runs and the runner gets no ack. This check guards a controller
that hands its run to a successor at restart
(`nativeRunsDetachingForRestart`). I think this is acceptable: (1) the
earlier check was not atomic with COMMIT either, so a revocation during
the COMMIT round trip already committed; the window grows from one
COMMIT round trip to one statement round trip plus its wait for a pool
connection; (2) the append is idempotent by source event id and payload
hash, so the successor sees the committed row as a replay of the event
that the runner resends; (3) the old controller never acts on the
rejected receipt. A new test covers exactly this.
- A crash loses nothing new: the statement commits or not as a whole,
and a resend after an unknown outcome is a replay.
- The check on `next_event_seq` makes the statement write nothing
whenever any other event of the run committed after its snapshot. Under
heavy contention on one run, more appends take the fallback. The
fallback is the previous code path.
- The statement text is fixed SQL sent through the driver
(`$client.unsafe`), like the rest of the event stream statements.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:59:01 -06:00
DottaandClaude Opus 5.5 2178c3df88 fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a recovery read to a remote Runner goes unanswered, the next
attempt of the same run carries `replaceUnansweredRunner`
> - That attempt verified the Runner was alive and then refused to adopt
it anyway. On a resumed sandbox lease the reuse check then failed with
`resumed_not_suspended` and left the Runner running. Otherwise a second
Runner was launched, and its launch fence killed the busy Runner, which
lost the provider turn.
> - After a late reconnect a Runner is often slow, not dead: it is still
replaying the events it saved with no controller attached
> - This pull request makes the attempt check progress first. It adopts
a live Runner that is making progress and replaces only a Runner that
made no progress.
> - The benefit is that a slow but working Runner keeps its turn, and
the retry no longer causes the failure it was meant to fix.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13. The retry after a
first-attempt timeout failed with these codes (counts from the traced
sample of 136 failures):

- `runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_not_suspended` (part of 40
`ingress` failures)
- `runner_remote_process_identity_unavailable` (25): the Runner was gone
by the time the retry checked, because the launch fence or a previous
retry had killed it
- "restored the conversation after process loss, but…" /
`provider_turn_lost_on_restore` (10): the replacement Runner restored
from the old state and the provider turn was gone

Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9
failed. Over 420 s: 14 of 15 failed. The process ID of the Runner
changed inside the same sandbox in the failed runs, which shows the
replacement.

Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt
with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit`
verifies the Runner is alive and then returns `undefined` because of
that flag, so `adoptedProcess` is `undefined`.

**Expected behavior**

A live Runner that is making progress is adopted. A Runner that made no
progress is stopped with proof and then restored from its own journal.

**Steps to reproduce**

1. Reattach a remote run whose previous attempt left a recovery read
unanswered (`replaceUnansweredRunner: true`), with the Runner alive and
its saved event sequence past the controller's last acknowledgement.
2. On master the attempt launches a second Runner, or fails with
`resumed_not_suspended` on a resumed sandbox lease.
3. The new tests in `native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts` do this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): with
`replaceUnansweredRunner`, the attempt adopts the live Runner, launching
nothing and signalling nothing, when:
- the Runner's saved event sequence is past the controller's last
acknowledgement, or
- its saved state (event and command sequences, generation, lifecycle)
changes within a bounded wait of about 10 s.
- The baseline is the first successful state read; every successful read
is also compared against the controller's acknowledgement. A Runner
whose state cannot be read at all during the wait is adopted, not
replaced: missing evidence is not proof of a wedge. A Runner whose
identity changes during the wait fails closed.
- A settled exit during the wait keeps the attempt allowance that
admitted the replacement, so it also restores on the extra fourth
attempt.
- Only a Runner that made no progress is replaced. The relaunch's
existing launch fence stops it and its process lineage with proof, and
the new Runner restores its own journal. A resumed sandbox lease no
longer refuses that relaunch with `resumed_not_suspended`.
- A Runner that exits during the wait follows the existing exited-Runner
rules: a settled suspension restores, and a mid-turn exit stays
fail-closed.
- The wait runs only on this recovery path, never on a normal run start.
- `doc/execution-semantics.md`: describes adopt versus replace.
- The claim flag `replaceUnansweredRunner` is set in claim logic that
another session owns. This PR does not change it; it changes only what
the executor does with it.

## Verification

Red on master, green on this branch:

- `native-session-executor.test.ts`
- `adopts a live Runner that emitted events past the controller's last
acknowledgement`
- `adopts a live Runner whose saved state advances within the bounded
wait`
- `hands a live Runner that made no progress to the relaunch, whose
launch fence stops it, without signalling it here`
- `restores from the journal of a Runner that suspended and exited
during the wait`
- `stays fail-closed for a Runner that exited mid-turn during the wait`
- `native-runner-restart-recovery.integration.test.ts`: `reattaches a
live %s Runner whose recovery read went unanswered on a resumed sandbox
lease` (progressing, wedged)
- Review fixes, `native-runner-restart-recovery.integration.test.ts`
(the first four were red before the fix):
- `adopts a live Runner whose first state read failed and whose later
reads advance`
- `adopts a live Runner whose first readable state is already past the
controller's acknowledgement`
- `does not replace a live Runner whose state could not be read at all
during the wait`
- `restores from the journal of a Runner that suspended and exited
during the wait on the admitted fourth attempt`
- `fails closed when the Runner's identity changes during the wait`
(pins existing behavior)
- `native-remote-retry-admission.test.ts`: unchanged, still passes.

Local runs on this branch: `tsc --noEmit` is clean for `server` on this
branch (master c6b497d831, which contains #21442).
`native-runner-restart-recovery.integration.test.ts`, the
adopt-or-replace block: 12 of 12 pass. The full
`native-session-executor.test.ts` and
`native-remote-retry-admission.test.ts` runs did not finish on this
shared machine (load 20 to 96); CI ran them on this head and they
passed.

Conflict resolved: master added a relaunch branch for a Runner that
exited for a fresh bootstrap (#21374) at the same place. Both branches
are kept; the cheaper check for an unanswered Runner that made no
progress comes first.

Daytona results that exist. They ran on a combined branch with this PR,
#21450, #21442 (merged) and #21362 (merged), so they cannot be credited
to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed in 6
s with `runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. The chain did not recur. All runs that crossed swap 2 were
adopted on the same Runner process ID.
- Caveat: the adopt-or-replace retry path that this PR changes was never
exercised in a cell. In G5c no first attempt failed, because all
identity writes started together 4 to 5 s after the claim. G5c also ran
before the phase timing fix in #21450.

## Risks

- Not proven on Daytona: the adopt-or-replace path itself (see the
caveat above).
- Without #21442 (merged) an adopted Runner can still time out on its
first command while it replays a backlog. This PR stops the replacement;
#21442 (merged) makes the adopted Runner answer.
- A Runner whose state cannot be read during the whole wait is adopted.
If it is in fact wedged, the adopted attempt times out again and the
attempt limit bounds it.
- A wedged Runner that still emits a few events counts as progressing
and is adopted. The existing attempt limit bounds this.
- The progress check adds a wait of up to about 10 s on this recovery
path only. A normal run start has no added wait.
- The claim construction (`native-restart-recovery.ts`) is owned by
another session and is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:56:11 -06:00
DottaandClaude Opus 5.5 47a2c82d35 fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server stops and the incoming
server takes over the live runs on remote Runners
> - The incoming server reached those runs late: it waited up to 60 s
for the outgoing server's controller lease to lapse, re-verified each
run's whole log from object storage, and deferred a reattach when the
agent directory was busy. Queued starts competed with the reattaches for
the same resources.
> - On hosted staging, the later a live run was reached, the more likely
it was to fail. Runs reached in under 150 s never failed; runs reached
after 420 s almost always failed.
> - This pull request hands live runs over at the start of shutdown,
makes a reattach skip work that the same run already did, and holds
queued starts while handed-over runs are being reached, with a 180 s
cap.
> - The benefit is that live runs are reached within seconds of a swap
instead of minutes, so they continue instead of failing.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

In one traced run the incoming server waited on
`remote_controller_lease_live` three times (09:14:21, 09:15:18,
09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of
that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the
run's whole log from object storage. A control run spent 23 s in
`task.prepare`. Queued starts used the same preparation path, the same 8
object-I/O slots and the same database pool, and nothing gave a reattach
priority. The first-attempt failures were `PRP command session.snapshot
timed out` and `codex_history_read_failed`. The retry then failed with
`runner_harness_state_mismatch: resumed_not_suspended`,
`runner_remote_process_identity_unavailable` or
`provider_turn_lost_on_restore`.

**Expected behavior**

Every live run is reached within about 30 s of the swap. Queued work
starts after that, or after a bounded wait.

**Steps to reproduce**

1. Run a two-server release swap with live remote runs and queued runs.
2. Observe `native.recovery.transition` "waiting" events with
`remote_controller_lease_live`, then the claim, then a long
`heartbeat.prepare.run_output` span.
3. The new tests in `reattach-phase.test.ts` and
`heartbeat-native-early-hand-off.test.ts` model this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- Early hand-off at shutdown (`heartbeat/recovery.ts`,
`server/src/index.ts`): in deploy-overlap mode, the outgoing server
releases the remote controller lease of its live runs at SIGTERM
(`recoveryState` `awaiting_runner_reattach`). A run another server
already took over is left alone. The incoming server no longer waits for
the lease to lapse.
- Reattach fast path:
- `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`:
a reattach starts without waiting for the existing log to be
re-verified.
- `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that
meets a busy agent directory waits in process (at most 30 s) instead of
being deferred to a full re-preparation. An ordinary start is still
deferred at once, with no added wait.
- Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is
a worker on the shared delivery coordinator, queue `reattach-phase`.
- Trigger: one startup listing of a previous server's live runs; then
the hand-over notification (`notifyReattachHandOver`, sent in the
hand-over transaction); the earliest lease deadline it watches (a killed
owner); and a 2 s re-check only while handed-over runs are still
pending, which ends with the phase (cap 180 s). The bounded re-check
exists because a run can also end on another path that sends no
notification to this queue.
- Source of truth: `native_run_finalizations` and `heartbeat_runs` rows.
- Quiet condition: nothing pending and no watched lease, so no deadline
and no SQL. It stops with the coordinator at shutdown.
- Reached hook for every driver (`native-session-executor.ts`): a
reattached run counts as reached when the native session runtime hands
the executor its recovered or opened session (`onSession`). Every driver
reaches that point only after the Runner answered this server's attach
and recovery commands, including an adopted `RunnerdExternalDriver`
Runner, which records no new process identity. The identity hook stays
as an earlier signal.
- Claim wake: when the reattach-phase worker finds runs that just became
claimable (hand-over notification or a killed owner's lease deadline),
it starts this server's restart-recovery claim for them at once. A
validation cell showed the claim otherwise waited about 21.5 s after the
hand-over.
- `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase`
return now sits before every action path of `act()`, except starting an
overdue saved-work recovery (it resumes an existing run).
- The in-transaction admission re-check of a queued claim
(`confirmRunAdmissionStillOpen`) also honours the reattach phase. A
phase that begins during a claim returns the run to the queue through
the existing rollback.
- A reattach whose log re-verification runs in the background awaits it
before the log is finalized (`settleDeferredLogRecovery`, both
finalization sites in `run-completion.ts`).
- Reattach phase (`heartbeat/reattach-phase.ts`,
`heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while
runs that a previous controller handed over (lease released or observed
expired, or claimed by this server's restart recovery) are claimable but
not yet reached. A run is reached when its process identity is recorded.
Authentication is logged as a mark. The cap (180 s,
`PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs
became claimable. Later batches join or re-arm the phase. Runs reached
after 150 s are counted and logged.
- Gates on the phase:
- `claimQueuedRun`, start-next, resume-queued and dispatch in
`heartbeat/queue.ts`, `run-lifecycle.ts` and
`native-runner-execution-driver.ts` leave queued runs queued.
- `execution-recovery-resolution.ts`: one line, the first line of
`deliverReconciledExecutions`.
- `recovery/assigned-work-sweep.ts`: one line, outcome
`deferred_reattach_phase`.
- `heartbeat/recovery.ts`: a one-line claim hook
(`noteReattachRunsClaimed`) after restart recovery claims runs.
  - The owner of those three files agreed to these one-line gates.
- `tests/restart-chaos/daytona-image.ts`:
`PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a
controller-only branch against an existing Daytona image. It accepts a
content-ID mismatch only for the exact content ID of the named image,
and it prints a TEST OVERRIDE warning. This is harness-only.

## Verification

Red on master, green on this branch:

- `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six
two-server scenarios were red on the earlier phase timing:
- `does not hold queued starts while the previous owner is alive and not
handing over`
- `closes the gate when the previous owner hands its runs over, and
counts reach latency from the hand-over`
- `runs the cap from the hand-over, so a wedged run ends the phase
without the overlap eating it`
- `begins when a killed owner's lease is observed expired, without any
hand-over`
- `re-arms for a later batch, including runs the previous owner started
after discovery`
- `closes the gate at once when this server claims a run between two
lease reads, dated from the hand-over`
- plus `claimQueuedRun leaves a queued run queued and unstarted, in the
admission gate's shape`, `is the same gate the assigned-work sweep and
reconciled continuation delivery read`, `ends at the cap, counted from
when the runs became claimable, so one wedged Runner cannot hold run
starts`
- `heartbeat-native-early-hand-off.test.ts`: `releases the remote
controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56
ms), `leaves a run another server already took over alone`, `holds the
incoming server's queued starts only from the hand-over, not while the
outgoing server still drives its runs`
- `heartbeat/run-output.test.ts`: `lets a reattach start before its
existing log is re-verified`
- `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at
once, with no wait added to the run-start path`, `lets a reattach wait
its turn in process instead of a deferred full re-preparation`, `still
defers a reattach once its bounded wait runs out`
- `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only
for the exact image content ID named by the test override`
- Review fixes (red before the fix, green after):
`reattach-phase.test.ts` `starts the claim of handed-over runs within a
second of the hand-over notification`; `assigned-work-invariant.test.ts`
`takes no action while handed-over live runs are being reached, and acts
once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never
lets the outgoing server claim back a run it handed off, while the
incoming server may take it over`; `reattach-phase.test.ts` `retries a
failed first listing instead of going quiet, and still finds the
previous owner's runs`, `keeps a hand-over notification whose listing
failed until a listing succeeds`, `moves the cap earlier when a claimed
run turns out to have been handed over before its claim`;
`queue.test.ts` `returns the claim to the queue when a reattach phase
begins while the claim is being checked`; `run-completion.test.ts`
`finalizes the run log in completeRun|failRun only after a reattach's
deferred log recovery settles`. `native-session-executor.test.ts` `ends
the reattach phase once a reattached %s session is driven by this server
again, without the cap or manual settling` over codex_app_server,
opencode_server, acpx_runtime, muse_external, grok_bot_mcp and
openai_dot_mcp (6 of 6 red without the hook; the test drives the
executor with a mocked session runtime, not each real driver). Lifecycle
tests for the new worker: `does no database work and keeps no deadline
on an instance with nothing to reattach`, `waits for the hand-over
notification or the lease deadline instead of polling while the previous
owner drives its runs`, `begins at a killed owner's lease deadline,
without any hand-over or notification`, `stops with the delivery
coordinator at shutdown and ignores a later hand-over`.

Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing
on this branch: `reattach-phase.test.ts`,
`heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53
tests together), `agent-directory-busy.test.ts`,
`heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the
new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30
to 96 on this shared machine, the full `queue.test.ts` and
`recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres
setup; CI supplies them.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR, #21462,
#21442 and #21362 (merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755): SIGTERM to PRP authentication 21.5 to
38 s at swap 1.
- G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7
s at swap 1. All runs that crossed swap 2 were adopted on the same
Runner process. The staging chain (`provider recovery timed out after
900000ms`, then `resumed_not_suspended`) did not recur.
- Caveat: G5c ran before the phase timing fix in this PR. In G5c the
phase opened at process start, its 180 s cap ran out during the overlap,
and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease
claims. The timing fix (the phase begins when the runs become claimable)
has not run on Daytona.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection,
combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11
survivors at both.
- Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after
hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after
SIGTERM. The 30 s target was met.
- Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at
19 s, so the outgoing server drained gracefully. The incoming server did
not claim on the hand-over; it claimed only when the outgoing server
exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity
47 to 57 s after SIGTERM. The target was missed. Why the claims waited
for the exit is not explained.
- The host load was 49 to 70 from other work. 7 of 32 tasks timed out in
the queue or budget after the swaps; we suspect the overload.

## Risks

- Product trade-off: while handed-over live runs are being reached,
queued starts wait, up to the 180 s cap. On an instance with no
handed-over runs, nothing waits.
- Not proven: two-server behavior with the phase timing fix and the
notification-driven watch (no cell has run them); warm standby becoming
active; runs that the outgoing server starts after discovery.
- Early hand-off at SIGTERM orphans a server-side exec tool that is in
flight. With #21442 the next controller settles it as interrupted with
an unknown outcome; it is never run again.
- Swap 2 in G6c: the incoming server claimed only at the outgoing
server's exit. That is unexplained.
- Cross-session ownership: the gates in
`execution-recovery-resolution.ts` and
`recovery/assigned-work-sweep.ts`, and the claim hook in
`heartbeat/recovery.ts`, are in files that the restart-recovery work
owns. They are one-line gates agreed with that owner. That owner said
the gate did not need to be inside the claim transaction; that was
before the phase could begin during a claim. After review, the
in-transaction re-check in `heartbeat/queue.ts` also reads the phase,
through the existing rollback. #21427 (open) also edits
`execution-recovery-resolution.ts`, `heartbeat/queue.ts` and
`recovery/assigned-work-sweep.ts`; whichever merges second needs a
rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned
by the restart-recovery work): a 3-line guard in
`claimNativeRestartRecoveries` returns a run this process handed off at
shutdown as `blocked` / `handed_off_to_next_controller` before takeover
is evaluated, so the outgoing server never claims back a run it handed
off. #21478 (merged) added a sweep path that starts an overdue
saved-work recovery; it is deliberately not gated by the reattach phase,
because it resumes an existing run (reattach-like work), which the phase
makes room for rather than competes with.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:55:31 -06:00
Devin FoleyandPaperclip 089748fbf9 fix(recovery): settle uncertain runner API receipts instead of fencing them forever (#21522)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs change Paperclip through the runner `call_api`
tool. Each mutation reserves a receipt before dispatch, so a crash
cannot replay a mutation that may have committed.
> - When the response of a mutation is lost, the receipt records an
unknown outcome. `priorRecoveryApiReceipt` then fences every later
mutation with the same operation id on that task, in every later run.
> - Nothing can settle that fence. The agent gets "Reconcile the
existing operation", but it has no tool to do that. The task stays stuck
for agents.
> - This pull request records the server's own final status for each run
mutation, and settles an unknown receipt from that record. A later
change by a board user on the task also lifts the fence.
> - The benefit is that a lost response no longer blocks a task's
mutations forever, and the fence still holds when the outcome is really
unknown.

## Linked Issues or Issue Description

**What happened?**
A routine run sent `PATCH /api/issues/{id}` to hand the task to a board
user. The server committed the change, but the server was slow and the
response took longer than the 30 second client timeout. The receipt
recorded `api_transport_failure` with `outcome: "unknown"`. Hours later,
the board user commented and reassigned the task to the agent. The new
run did the work, but both of its hand-off PATCH calls returned
`api_outcome_unknown` within 2 seconds and never reached the route.
Every later run on that task would get the same result.

**Expected behavior**
When the server has a record of the earlier call's result, the uncertain
receipt settles from that record. When a person has changed the task
after the uncertain run, the agent can act on the current state.

**Steps to reproduce**
1. In a native run, send a `call_api` mutation that commits on the
server, but let the client time out before the response.
2. Start a later run in the same retry or continuation ancestry.
3. Send any mutation with the same operation id. It returns
`api_outcome_unknown` and is not dispatched.

**Relevant logs or output**
```
call_api PATCH /api/issues/{id} -> { ok: false, status: null, error: "api_outcome_unknown", outcome: "unknown",
  guidance: "Reconcile the existing operation before repeating it. ..." }
```

No related public PR exists. I searched open and closed PRs for this
fence.

## What Changed

- `runner-api-client.ts`: `executeRunnerApi` sends the receipt key in
`X-Paperclip-Runner-Call` with mutations. Reads do not send it.
- `paperclip-runner-tool-authority.ts`: `#callApi` passes the receipt
key of the reserved mutation to the client.
- `middleware/runner-call-outcome.ts` (new): for an agent JWT request
with a run id and a valid call key, it records the final HTTP status as
a `runner.api_completed` activity row of the run. It records when the
handler ends the response, so it also records after the client has gone
away. It records one time per request.
- `recovery-effect-receipts.ts`: `priorRecoveryApiReceipt` settles an
unknown receipt:
- A recorded 2xx is an applied effect. An exact retry from the ancestry
gets an applied result and is not dispatched again. Other calls
continue.
- A recorded definite 4xx rejection (not 408) no longer fences, the same
as a completed 4xx receipt today.
  - A recorded 5xx, or no record, keeps the fence.
- A board user's `issue.updated` or `issue.comment_added` on the task
after the uncertain run finished lifts the fence of that run. Agent,
system, read and inbox activity do not.
- The fenced result now tells the agent to read the current state and
not to repeat a change that is already there.

## Verification

- `pnpm --filter @paperclipai/server run typecheck` passes.
- `npx vitest run src/middleware/runner-call-outcome.test.ts
src/services/task-retry.test.ts
src/services/native-runtime/runner-api.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-api-artifacts.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/__tests__/teams-catalog-service.test.ts
src/__tests__/primary-agent-guidance.test.ts
src/__tests__/skill-import-discovery.test.ts` (in `server/`): 2418
passed, 2 skipped.
- `npx vitest run src/__tests__/runner-api-recovery.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/confirmation-comment-api.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/runner-api-log-redaction.test.ts
src/services/native-runtime/runner-api-artifacts.integration.test.ts
src/services/native-runtime/native-case-tools.integration.test.ts
src/__tests__/connection-management.integration.test.ts`: 404 passed, 1
failed. The failure is `runner-api-artifacts.integration.test.ts >
preserves exact HTTP bytes in the immutable asset and downloaded helper
input` ("Artifact installation is unavailable to this run or provider").
It fails the same way with the unchanged `master` files in my
environment, so it is not from this change. CI covers it.
- Red before the fix: with the `master` version of
`recovery-effect-receipts.ts`, the new settlement and supersession cases
in `task-retry.test.ts` fail.
- New tests:
- `task-retry.test.ts`: settlement by recorded status 200, 422, 500, and
a record of a different call only; supersession by a later user, agent
or system change, including a change before the run finished and a read
marker.
- `runner-call-outcome.test.ts`: records the final status one time;
ignores reads, a missing or malformed key, agent API keys, board actors
and non-UUID run ids.
- `runner-api.test.ts`: the call key header is sent with mutations only.

## Risks

- Each run mutation that carries a call key adds one activity row
(`runner.api_completed`, entity `heartbeat_run`). The row has no request
body and no response body. Only the status and the receipt key hash.
- The settlement trusts the server's own final status. Routes commit
before they respond, so a 2xx means the effect is applied. A route that
answers 2xx before it commits its effect is still treated as applied.
- A run can write its own outcome record with its own token. This can
only settle that run's own receipts, so it does not give an agent new
access.
- Receipts made before this change have no record. They keep the fence
until a board user changes the task.
- No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 16:45:51 -08:00
DottaandClaude Opus 5.5 e82c4f0a31 fix(runner): prove a warm Runner's suspension on a slow sandbox host so the next run does not fail (#21472)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**

This PR extends the failed-close retry that #21442 (merged) added. It is
based on `master`, which contains #21442.

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote sandbox is reused by the next run of the same task.
Before the next run starts, the previous run's Runner must be retired:
suspended, with durable proof.
> - An idle warm close had 10 s for everything. On a loaded sandbox host
one liveness probe or one read of the durable state used that up. The
suspend was never sent, or it completed but was never proven.
> - The next run then replayed that cached close failure, failed before
any provider work, and the task showed a failed run with Retry
> - For a follow-up on a task blocked on its subtasks, the automatic
continuation was refused and the Retry stayed until another wake
> - This pull request bounds each probe, gives the suspend proof its own
read window, lets a later close prove the suspension again, and lets
that follow-up continue by itself
> - The benefit is that a slow host no longer turns a completed
suspension into a failed task, and the person sees no Retry for it

## Linked Issues or Issue Description

**What happened?**

Local release-swap cells on Daytona (G5c and G6c, 2026-11-13) each
showed one Retry banner: "previous run's session could not be retired
(runner did not durably suspend before checkpoint)". G6c had no fault
injection, so this is a real class.

- In the G6c case, the Runner completed its suspend and exited. The
server's proof read (a sandbox exec under load) ran out of the remaining
0.6 s of the close grace, so the suspension was "not proven".
- In the G5c case (an adopted Runner), the liveness probe
`adoptedRunner.isAlive()` took about 220 s and used the whole close
deadline. The suspend was never queued. Five adopted sessions failed the
same way in the same second.
- The successor run's retirement called `transport.close()`, which
returned the cached rejection of the predecessor's failed close. It did
not send a new suspend and did not read the state again.
- The successor failed before provider work. Its fresh-sandbox
continuation (`interrupted_preparation`) was refused as "blocked after
recovery", because the parent task was blocked on its subtasks before
the follow-up started. The task showed a failed run with Retry for 27 s,
until a blocker-stall wake.

**Expected behavior**

A completed suspend is proven, even on a slow host. A follow-up that
fails before any provider work on a task that was already blocked
continues by itself, and the task shows as working while that
continuation is queued.

**Steps to reproduce**

1. Close an idle warm Runner whose durable state read takes longer than
the close grace, or whose liveness probe stalls.
2. On master the close fails and the next close replays the same
failure.
3. The new tests in `runnerd-codex-transport-close-proof.test.ts` and
`heartbeat-native-predispatch-retry.test.ts` do this.

**Paperclip version or commit**

`master` at c6b497d831 (contains #21442).

## What Changed

- `runnerd-codex-transport.ts` (controller-side TypeScript):
- Each liveness probe in the close is bounded. A probe that does not
answer counts as possibly alive, never as gone, and the suspend goes
over the live connection. A connected Runner's probe gets a 1 s slice,
so drain and suspend keep their budget.
- Once the suspend command's completion is observed, what the
durable-state read shows decides how long the proof may take:

    | Read after the completion | Proof window | Maximum close time |
    | --- | --- | --- |
    | Suspended, right Runner | proven at once | unchanged |
| Another run's or Runner's state | ends at the close deadline, as on
master | close grace (10 s; 45 s for Pi) |
| Right Runner, not yet `suspended` | settle window of 3 s from the
first such read (`RUNNER_SUSPENSION_PROOF_SETTLE_MS`, test option
`closeProofSettleMs`) | close grace + 3 s |
| Unreadable or slow read | full proof window of 30 s from the
completion (`RUNNER_SUSPENSION_PROOF_READ_MS`, `closeProofReadMs`) |
close grace + 30 s |

All rows add the existing bounded exit-probe slices. runner-core saves
the `suspended` lifecycle before it delivers the suspend result
(`persist_lifecycle_before_command_delivery` in `durable/runner.rs`), so
a readable non-suspended state after the completion can only be a stale
remote read; the settle window covers a few read attempts and is not a
write-ordering wait.
- The settlement diagnostic names the branch that ended the window
(`proofWindowEnd`: `proven`, `wrong_identity`,
`not_suspended_after_settle`, `unreadable_after_window`,
`suspend_not_completed`).
- A close that failed only because readable proof was missing is retried
by the next close: it reads the durable state again with a fresh bound
and finishes settled. A close whose proof was refuted stays memoized, as
on master. This extends the failed-close retry from #21442.
  - The close failure carries per-step timings.
- `native-runtime/predecessor-retirement.ts`,
`heartbeat/run-completion.ts`: the predecessor-retirement fallback
records the task's status when the run failed.
- `modules/run-dispatch/domain/policy.ts`,
`modules/run-dispatch/adapters/postgres.ts`: a continuation of a run
that did no provider work, on a task that was already blocked then, is
not treated as blocked after recovery. Its dependency check follows the
wake it continues. The continuation keeps a person's comment wake.
- `task-interruption-presentation.ts`: while that run's failure handling
queues the continuation (up to 30 s), the task shows as working with no
Retry. The window ends when the continuation is scheduled, when the run
records `continuationScheduling: "refused"` (written by
`run-completion.ts` when the fallback does not schedule it), or after 30
s. The fallback record alone no longer ends it.

## Verification

Red on master, green on this branch:

- `runnerd-codex-transport-close-proof.test.ts`
- `proves a completed suspend on its own read window when the durable
state read outlasts the close grace`
- `suspends five adopted idle Runners whose liveness probes stall
together, instead of spending the close on the probe`
- `heartbeat-native-predispatch-retry.test.ts`
- `continues a follow-up on a blocked task whose predecessor could not
be retired`
- `does not continue a task that was blocked only after its run failed`
- `task-interruption-presentation.test.ts`
- `shows the task as working while its fresh-sandbox continuation is
being queued`
- `keeps the task working after the fallback is recorded while its
continuation is still being scheduled` (review fix)
  - `offers Retry when the continuation was refused or never queued`
- `runnerd-codex-transport.test.ts` (review fixes)
- `gives a suspend that completes near the close deadline its whole
proof window from the completion`
- `bounds the proof window from the observed completion, so a proof read
that never answers still ends`
- `proves a suspension whose suspended state appears 1.5 s after the
command completed`
- `fails a suspension whose state stays ready after the settle window,
not after the whole proof window`
- `fails at the close deadline when the durable state belongs to another
run`
- The three existing tests in
`runnerd-codex-transport-suspension.test.ts` that expect a conclusive
negative to fail within the close grace pass unchanged.

Local runs on this branch: `tsc --noEmit` is clean for `server` and
`packages/paperclip-runner`. Passing on this branch:
`runnerd-codex-transport-suspension`, `-close-proof`, `-adoption`,
`-remote-retention`, `runnerd-detach-durability` (all files), the
proof-window cases of `runnerd-codex-transport.test.ts`,
`task-interruption-presentation.test.ts` (72), `run-dispatch` policy and
Postgres adapter tests and `heartbeat-native-predispatch-retry.test.ts`
(166 together).

Conflict resolved: `task-interruption-presentation.ts` gained
`WAITING_CONTINUATION_MOVE_FORWARD_MS` on master (#21385). Both
constants are kept.

Runner content ID: the Runner change is controller-side TypeScript under
`packages/paperclip-runner/src` only. There is no Rust, `protocol/` or
`generated/` change. Existing Runner images work with this controller.

Daytona: this fix has not run on Daytona at all. The two banners above
are the evidence of the class (G5c and G6c ran on 588c6ba267, without
this fix).

## Risks

- Not proven on Daytona.
- A close can now take up to 30 s longer when the proof read is
unreadable or slow, and up to 3 s longer when the read is stale. This is
on the close path only, not before `turn.start`.
- A probe that does not answer now counts as possibly alive, so the
close sends a suspend instead of treating the Runner as gone.
- Ownership: this PR changes
`server/src/modules/run-dispatch/domain/policy.ts`,
`server/src/modules/run-dispatch/adapters/postgres.ts` and
`server/src/services/heartbeat/run-completion.ts`. CODEOWNERS does not
list these files. Their recent changes come from the restart-recovery
and Retry work (#21348, #21256, #20863, #21302). That area should review
the dispatch-policy change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:32:22 -06:00
Nicky LeachandClaude Opus 5.5 6d8ebc4a85 fix(secrets): mask secret-bound environment values in run output (#21341)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can bind a company secret to an environment variable for an
agent, an environment, a project, or a routine
> - Core resolves these values for each run and gives them to the agent
process or to the Runner
> - Core stores the run output: the run log, the run excerpts, adapter
events, and native (Runner protocol) transcript events
> - The run-output redactor did not know the resolved secret values.
Only failure diagnostics and the `meta.env` keys of `adapter.invoke`
were masked
> - So when an agent ran `env` or `echo $STRIPE_KEY`, Core stored the
secret in plaintext, and every person who can view the run could read it
> - This pull request gives the resolved values of secret-bound keys to
the run-output redactor and to the native control-plane port
> - The benefit is that an accidental print of a secret does not put the
secret into stored run output, for local and native runs

## Linked Issues or Issue Description

**What happened?**
An agent printed a secret-bound environment variable during a run (for
example with `env` or `echo $STRIPE_KEY`). Core stored the value in
plaintext in the run log, in `stdoutExcerpt` and `stderrExcerpt`, and in
the transcript events. Anyone who can view the run can read the value.

**Expected behavior**
Core masks the value of each secret-bound environment variable before it
stores run output. Core does not mask plain (non-secret) configured
values.

**Steps to reproduce**
1. Create a company secret and bind it to an agent environment variable
with `secret_ref`.
2. Configure the agent so that its command prints the variable.
3. Run the agent and open the run log. The value shows in plaintext.

**Related**
Refs #21330. That PR keeps configured secrets out of OpenCode's own
storage and model requests inside the Runner. This PR is the Core side:
it masks secrets in the run output that Core stores, for every run type.
The two changes are independent.

## What Changed

- `secretEnvironmentValues(env, secretKeys)` in
`agent-identity-redaction.ts` selects the raw values of keys in
`secretKeys` (the keys that came from `secret_ref` or `user_secret_ref`
bindings). `secretOutputForms` derives the forms to mask: the full
value, the value without outer whitespace, the JSON-escaped form, and
each line of a multi-line value (PEM armor lines are not added). It
skips a form that has fewer than 8 characters after trimming. This is
the same 8-character limit that diagnostic redaction uses for unknown
values.
- The output redactor keeps these secrets in a separate set, beside the
existing agent private-key values. The private-key values are masked
everywhere, as before. Environment secrets are masked only in output
text. They are not masked in protocol fields: `schema`, `type`, `kind`,
`status`, `state`, `outcome`, `stream`, `channel`, `role`, `level`, ids,
refs, codes, revisions, versions, digests, timestamps, and names that
end in these words (for example `eventType`, `turnTerminalState`,
`reportedWorkDisposition`). So a secret that equals a protocol word (for
example `completed`) cannot change an event type or a terminal state.
- The streaming match now scans left to right. At each position it tries
the longest value first. If a longer value could still start at that
position, it holds the text until the next chunk, also when a shorter
value already matches there. So the tail of a longer secret cannot leak
when a shorter secret is its prefix.
- `prepareHeartbeatExecutionConfiguration` adds the resolved secrets to
the run's output redactor immediately after it resolves them. This one
redactor already masks the local adapter log stream, the excerpts, live
log events, adapter events, `adapter.invoke` metadata, the adapter
result, and the canonical native log export.
- For a native run, `executeHeartbeatRuntime` stores the delivered
secrets (only values that have a form to mask) encrypted under a new
context key, `paperclipOutputSecretRedactions`, in one transaction. It
then adds all stored values of the run to the redactor. A run that
reconnects to a live Runner after a `"latest"` secret rotation then
masks the value that the Runner holds and the new value. This store
reuses the encryption and fingerprint logic of the run redaction
registry, but it is separate: API read redaction does not apply its
values, so a secret cannot change ids, event names or terminal states in
API responses. API reads hide the new key like the existing registry
key.
- `executeHeartbeatRuntime` gives the native executor the run's secrets
and the observability credentials (`observabilitySecretValues`).
- The native executor gives the secrets to its failure-diagnostic
redactor and to `PaperclipControlPlanePort` (`secrets`). The port masks
them in each PRP event before it persists the event, with the existing
delta-aware handling for values that are split across `item.delta`
events. This covers native runs on the local Runner and on plugin-hosted
agent computers.
- After a controller restart, a new control-plane port cannot know the
text that the old port held back, because that text is never stored. On
the first delta of each stream it has not seen, the port masks leading
text that matches the tail of a secret (4 or more characters). It skips
text that can start a secret, and it keeps checking the stream while a
delta falls entirely inside such a tail.
- The native executor classifies failures from the original error. It
masks only the agent private key in place, as before. It masks
environment secrets in a separate redactor that it uses only for text
that it writes or logs (`failureDetail`, failure messages, stderr
diagnostics). Core masks the rethrown error before it stores it.
- `coalesceIntoActiveExecutionRun` (wake queue) writes the merged
context with `withRunRedactionHistory`, which takes
`paperclipSecretRedactions` and `paperclipOutputSecretRedactions` from
the locked current row. Before, a coalesced wake built from an older
context could delete either key. This also protects the existing
registry. This is a change outside the secrets module, in the one place
that replaces a running run's context concurrently.
- When an adapter or native run fails, the heartbeat keeps the original
error for retry decisions and rethrows it. The copies that it saves in
`workspace_finalize` metadata (`errorMessage`) and in a failed session
goal action go through the run's output redactor first.
- `docs/deploy/secrets.md` describes the mask and its limits.

## Verification

- `server/src/__tests__/run-output-secret-env-redaction.test.ts` (new,
embedded Postgres): a real `process` adapter run with a `secret_ref`
bound to `"latest"`. The agent prints the secret split across two
writes, in JSON, and on stderr. The stored log, run row, and run events
do not contain the value. The plain value and a 5-character secret are
not masked. The test then rotates the secret and runs again. The second
run masks the rotated value, and a SHA-256 digest in the output proves
the run received the rotated value. Red-green: it fails without the new
line in `execution-configuration.ts`.
-
`server/src/services/native-runtime/paperclip-control-plane-port.test.ts`:
the port masks a secret in a persisted tool event and in a value split
across two `item.delta` events; plain and short values stay (red-green:
fails without the port change). A new test binds secrets equal to
`interrupted`, `completed`, `succeeded` and `satisfied`, then appends a
`turn.interrupted` event, checkpoints, and completes the run. The event
type, payload status, terminal states and criterion status stay
unchanged and the result is saved (red-green: fails when protocol fields
are masked).
- `server/src/services/heartbeat/runtime-execution.test.ts`: a value
stored by an earlier attempt, the current value, and an observability
header value all reach native dispatch and the heartbeat redactor. Short
values are not stored. The shared run redaction registry stays empty,
and an API read hides the new key (red-green: fails without the store
read).
- `server/src/services/native-runtime/native-session-executor.test.ts`:
a native failure that contains the secret is masked in failure writes,
logs and the rethrown error, and the executor gives the secrets to the
control-plane port.
- `server/src/services/agent-identity-redaction.test.ts` (new): value
selection, the minimum length, multi-line and JSON-escaped forms, nested
values, every split point of a streamed value, overlapping secrets
(`abcdefgh` and `abcdefgh1234567890`) split at the shorter value,
protocol fields in events, terminal states and deltas, and secrets added
by a later attempt.
- Restart, retry and coalescing (each fails without its fix): a port
test restarts between two deltas of a secret and stores `echo ` and
`***REDACTED*** done`; an executor test binds `timed out` and keeps
`provider_session_open_timeout`; a wake-queue adapter test keeps both
redaction keys when a coalesced wake replaces the context.
- `server/src/services/heartbeat/run-output.test.ts`: the local output
path masks the log, excerpt, live event, adapter event, and adapter
metadata.
- `pnpm --filter @paperclipai/server typecheck` passes. The changed test
files and the other users of the redactor (`agent-identity`, `run-log`,
`heartbeat-case-handoff`, `native-run-log-export.integration`) pass in
full locally: 1986 passed, 7 skipped; the 6 run-redaction-registry test
files also pass (231 tests) (macOS, embedded Postgres). `pnpm verify`
exits 2 (broader verification required) because `docs/deploy/secrets.md`
is a global CI input, so required CI supplies broad coverage.

## Risks

- This is a guard against accidental leaks. It is not a security
boundary. An agent can change a value before it prints it (for example,
encode it in base64 or print one character at a time), and Core cannot
detect that.
- Core does not mask a secret value or line that has fewer than 8
characters. This prevents a short value from erasing ordinary output. A
secret that short is not masked in run output.
- Core does not mask environment secrets in protocol fields (see What
Changed). If a secret appears in such a field, it stays. These fields
hold protocol words and generated identifiers, not agent output.
- If a value is common text of 8 or more characters, Core masks that
text wherever it occurs in that run's output text. This only affects
runs that bind that secret.
- If a key is secret-bound in one source and a later source sets the
same key to a plain value, `secretKeys` still includes the key, so Core
masks the plain value. This is the existing meaning of `secretKeys`.
- For native runs, Core now stores the delivered secrets encrypted in
`contextSnapshot` under `paperclipOutputSecretRedactions`, with the same
local encryption as the existing run redaction registry. It adds one
transaction per native run that has secrets. API reads do not apply
these values and hide the key.
- After a controller restart in the middle of a secret, a tail shorter
than 4 characters can still be stored. A new stream that starts with the
tail of a secret (4 or more characters) is masked at its start even
without a restart. This is a small amount of extra masking.
- Core masks values at write time. Run output that Core stored before
this change is not changed.
- No schema change and no migration. The new fields are optional and
in-process only.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:14:41 -08:00
DottaandClaude Opus 5.5 de80299d2c perf(recovery): check only blocked tasks whose blockers are done (#21517)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. One queue,
`resolved-dependencies`, wakes a blocked task when all of its blockers
are done.
> - The queue reloads every blocked task of a company when the company
gets a recovery hint. Every run dispatch and run state change sends that
hint.
> - Each reloaded task gets a dependency check of about five statements,
also when its blockers are still open and nothing can happen.
> - On a hosted instance with 577 blocked tasks and 5 live runs, this
loop ran all the time: five statements at 7.3 per second each.
> - This pull request loads only the blocked tasks whose every blocker
is done. These are the only tasks the check can wake.
> - The benefit is that the cost of this queue no longer grows with the
number of blocked tasks that wait on open work.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The background check that wakes blocked tasks when their blockers are
done (`resolved-dependencies` in
`server/src/services/recovery/work.ts`).

**Current behavior**

A company recovery hint reloads every blocked, agent-assigned task of
the company. Run dispatch and every run state change send this hint, so
with live runs it comes all the time. Each task then gets
`reconcileResolvedDependencyWakeBackstop` with its issue id. That reads
the task, reads the dependency readiness, and then reads the stall wakes
of the task (children, blockers, recovery actions). For a task with open
blockers this finds nothing to do. The queue reloads at most every 30
seconds per hint and works 25 tasks per pass with 1 second between
passes, so with hundreds of blocked tasks it never becomes idle.

**Proposed behavior**

The queue loads only blocked tasks that have at least one blocker and no
blocker that is not done. The per-task check is unchanged, so the
workspace-finalize barrier and the existing-wake dedup still apply. A
blocked task enters the queue when its last blocker becomes done,
because that change sends the company hint and the next reload now finds
it.

Stall wakes for tasks with open blockers come from the company pass of
the `active-recovery-actions` queue (`reconcileActiveRecoveryActions`).
That pass reads every stall of the company with the same predicate, and
it gets the same company hint.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. With 300
blocked tasks behind one open blocker, a company hint now checks 0 of
them instead of 300.

## What Changed

- `server/src/services/recovery/work.ts`: the shared `register` helper
takes an optional extra SQL condition. The `resolved-dependencies` queue
gives a condition that keeps only blocked tasks with at least one
`blocks` relation and only `done` blockers.
- `server/src/__tests__/issue-recovery-deadlines.test.ts`: a new test
with 300 blocked tasks behind an open blocker and 1 blocked task behind
a done blocker. It runs the real coordinator, notification and SQL.

## Verification

- `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts -t
"blockers are done"` (in `server/`). Before the change it fails: the
first pass checks all 301 tasks instead of the 1 ready task. After the
change it passes. The test also shows that the 300 tasks are checked as
soon as their blocker becomes done.
- `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts
src/__tests__/issue-recovery-work-lifecycle.test.ts
src/__tests__/issue-blocker-stall-wakeups.test.ts` pass. The stall test
file covers the stall wakes through the company pass.
- `pnpm check:module-boundaries` passes.
- Local benchmark: the real server against PostgreSQL with statement
logging, one company, 60 agents, about 1,440 open tasks (780 in review,
580 blocked with blocker relations, 50 in progress, 30 to do), with run
churn from the recovery queues. Results are in the table below.

The benchmark table follows in an update of this description.

## Risks

- Low risk. The check itself is unchanged. The change only removes tasks
from the queue that the check skipped as not ready.
- A blocked task whose only blocker is cancelled stays out of the queue.
The check also treated it as not ready, because only a done blocker
resolves a dependent.
- Stall wakes for blocked tasks with open blockers now come only from
the company recovery-actions pass. That pass uses a superset of the
stall predicate.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 17:52:03 -06:00
Nicky LeachandClaude Opus 5.5 1140984b82 feat(plugins): tell an opted-in workspace provider the run's environment (#21512)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin can be a project workspace provider and also an environment
driver for the agent runs that use those workspaces
> - Core asks the workspace provider to prepare the project workspace
before the run acquires its environment lease
> - The provider knows the run, agent, project, and execution workspace,
but not the environment that the run selected
> - A provider whose compute exists per run (for example, a sandbox that
it creates from the environment's driver config) cannot start or find
the run's compute for the workspace work
> - This pull request lets a provider opt in to receive the run's
selected environment, when that environment is one of the provider's own
drivers
> - The benefit is that such a provider can prepare the workspace on the
run's own compute, with no change to the order of operations in Core

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Plugin workspace preparation for runs whose environment is a driver of
the same plugin.

**Subsystem affected**
Plugin SDK workspace provider params, the plugin manifest workspace
provider declaration, and `callWorkspaceProvider` /
`preparePluginWorkspace` in the server.

**Current behavior**
`prepare` params carry `runId`, `agentId`, `projectId`, and
`executionWorkspaceId`. They carry `taskLease` only when Core acquired
the lease first. They never name the run's environment.

**Proposed behavior**
A provider that declares `workspaceProvider.workspaceEnvironmentVersion:
1` receives `environment: { id, driverKey, config }` when the run's
selected environment is a plugin environment of that same provider.
`config` is the environment's `driverConfig`, the value that the
driver's acquire receives.

## What Changed

- SDK: optional strict `environment` field on
`pluginWorkspaceParamsSchema`.
- Manifest: optional `workspaceProvider.workspaceEnvironmentVersion: 1`,
next to the sibling version fields. Core sends the new field only to
providers that declare it, because providers parse the params with the
strict schema.
- Server: `callWorkspaceProvider` adds `environment` only when the
provider opted in, the environment driver is `plugin`, its `pluginKey`
is this provider, and it names a `driverKey`. The field is not part of
the operation ID.
- Server: `preparePluginWorkspace` passes the run's environment to its
prepare call. This covers normal, verify-existing, fresh, and
replacement prepares.
- Calls that do not have the run's environment do not send it: finalize,
inspect, remove, reservation reconciliation, and the retry-candidate
verification in `fresh-workspace-continuation.ts`.
- Lease acquisition, the `sessionHandoffVersion` gate, and the order of
operations do not change.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the
capability and its exceptions.

## Verification

- `packages/plugins/sdk/tests/workspace-provider.test.ts`: 11 passed.
The schema accepts the field and rejects malformed values.
- `packages/shared/src/validators/plugin.test.ts`: 65 passed. The
manifest accepts only version 1.
- `server`: `plugin-workspaces.integration`,
`plugin-fresh-workspaces.integration`, `project-environments`, and
`heartbeat/workspace-preparation`: 226 passed, against embedded
Postgres. The new integration test checks the exact shape, an unchanged
operation ID, verification prepares, and omission for an undeclared
provider, another plugin's environment, non-plugin drivers, a missing
`driverKey`, and no environment. It fails when the server stops passing
the environment.
- `pnpm typecheck` passes in `packages/shared`, `packages/plugins/sdk`,
and `server`.
- The full suite was not run locally. CI runs it.

## Risks

- Low risk. Providers that do not declare the capability get the same
params as before.
- `config` is the stored `driverConfig`, in the same form that the
driver's acquire gets. Secret references in it reach the provider in
that same form.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:43:10 -08:00
Devin FoleyandPaperclip 21bf7ebcee feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path

> - Paperclip manages work for AI agents.
> - SSH environments run tasks on computers that users own.
> - A Paperclip server needs network access before it can reach a
private computer.
> - A reusable Tailscale connection belongs in Connectors.
> - Connected machines should become choices in the existing SSH
environment form.
> - This change connects a tailnet and resolves selected machines
through a private network client.
> - Agents retain their choice of harness and AI account.

## Linked Issues or Issue Description

**Subsystem affected**

Connectors, environment settings, SSH execution, company secret storage,
and runtime packaging.

**Problem or motivation**

Entering a Tailscale address in the SSH form does not connect the
Paperclip server to that tailnet. Requiring a separate device key for
every environment also duplicates network setup.

**Proposed solution**

Add a Tailscale Connector with **Connect Tailscale Network**. Use
brokered OAuth consent to enroll one private network client per
connection. Let SSH environments select a connected network and stable
machine identity. Preserve the enrolled node state through the company
vault so a restart does not reuse the one-time token.

**Alternatives considered**

A separate Tailscale environment type would duplicate SSH behavior and
mix network access with harness selection. This change uses the existing
`ssh` driver. Related #15648 adds OAuth client credentials and a health
check; this change uses user consent and supplies outbound task
execution. Related #11743 concerns HTTPS brokering.

**Roadmap alignment**

This extends the existing Connected Apps, environment, and Personal &
Shared AI Accounts capabilities. This PR is stacked on #21248, which
provides SSH account selection independently of Tailscale. The hosted
OAuth broker is a companion control-plane change.

## What Changed

- Preserve Cloud reauthentication in the current browser tab. Explain
canceled or failed consent in the connector and offer a retry of the
same authorized saved connection.
- Record Cloud enrollment starts initiated through the Tailscale
connector in the company activity log.
- Check install-preparation descendant liveness through the kernel
process table in the Runner support test. Keep the direct child’s strict
reaping assertion and reject any non-zombie descendant.
- Align the application image stage contract with the Tailscale client
stage after the upstream Runner-image refactor. Keep the exact stage
list and check every application stage, including `tailscale-client`,
for prohibited Runner compiler commands.
- Group Tailscale consent, discovery, daemon lifecycle, pairing, and
deletion under `server/src/services/connectors/tailscale/`, with their
service tests. Update existing route, runtime, environment, vault, and
company-deletion imports. Keep the existing contracts, authorization,
audit, and transaction owners.
- Handle early stdin closure in the Kubernetes real-shell test helper.
Stop the input stream on `EPIPE`, preserve nonzero child exits, and
reject a zero exit when input was lost. Add two real-process regression
cases. Production transfer code and confinement checks are unchanged.
- Read configured Tailscale executable paths when a client starts, after
supported `.env` loading. Keep explicit command overrides available.
- Discard pending pairing results after switching to Direct SSH or
closing the form. Retain the existing company, connection, machine, and
port checks.
- Store network credentials with an immutable server-owned purpose from
their first vault row. Exclude them from generic catalogs, bindings,
environment projection, and value reads. Use a dedicated audited server
consumer for pinned connector reads. Block unclassified legacy
serialized node state at the same value boundary. Keep SSH private-key
binding separate from connector credentials.
- Gate network queries and setup on verified local-implicit or
instance-admin access. Hide Tailscale catalog rows for other members,
show an admin requirement on direct setup navigation, and avoid device
discovery before management access is known.
- Keep failed-client cleanup in the network manager. A failed request
cannot disconnect a newer retry for the same identity; explicit
connector disconnect and enrollment retirement still stop their
networks.
- Check for readable Ed25519, ECDSA, or RSA host public keys before
pairing edits `authorized_keys`. Print a known-hosts entry for every
available key, or fail before changing authorized keys if none are
available.
- Return self-hosted Cloud enrollment to the Tailscale connector. Save
the unprefixed setup path and let the shared callback add the verified
company prefix. Keep external and unrelated destinations rejected.
- Hide pairing instructions after an SSH port change. Ignore late
key-selection results for the old port. Operators prepare again for
matching known-hosts output.
- Retire connector references before deleting a company or its vault
credential. Stop company daemons after the database commit. Keep failed
secret retirement retryable and prevent late node-state writes from
restoring deleted connections. Discard retired state without blocking
behind the vault deletion lock.
- Reload a mismatched credential before replacing a ready daemon.
Recheck the active entry and manager generation after the read. An older
request cannot stop a newer healthy socket or cross shutdown.
- Pin connector credentials to the exact vault version and SHA-256
digest recorded by trusted enrollment and node-state writes. Ignore
generic secret rotations. Advance both fields with a compare-and-set
guard. Reject unpinned or tampered credentials before network startup.
- Save an explicit Direct SSH selection and clear old Tailscale fields
after config merging. Preserve the operator's entered direct host.
- Route all Tailscale SSH runtime access through the connection service
after SSH-key loading. Pin access to the environment's saved connector
identity and recheck after device discovery. Remove standalone key-based
daemon startup and reject raw enrollment keys or credential-only network
configurations. Align the shared config type and SDK pins.
- Correct two E2E assumptions exposed during connector qualification.
Keep the two-window chat fixture stale until its own mutation. Validate
the original failed run and its permitted write-revocation timestamp in
the legacy recovery browser test.
- Add the Tailscale Connector page, catalog entry, OAuth return flow,
connected-network management, device discovery, and disconnect. Document
all six routes in OpenAPI and the API reference, including their
instance-admin requirement.
- Add company-scoped connection storage in `0480_lazy_lethal_legion`,
trusted credential version/digest fields in `0481_round_pet_avengers`,
and server-owned secret purpose/backfill in
`0482_aberrant_rumiko_fujikawa`. Generate all three migrations through
the maintained command and retain the five newest snapshots.
- Add machine pairing that generates a fresh SSH key per request, stores
its private half in the company vault, and uses the generated public
half in the install command. Do not trust editable saved metadata.
Preserve existing authorized keys, including a file without a final
newline. Keep SSH username, workspace, and trusted host keys explicit.
- Use the existing SSH driver. Resolve the machine's current address
from its stable device ID. Reject disconnected, removed, offline, and
wrong-company selections.
- Hide Network when Direct is the only option. Select a sole connected
tailnet automatically and hide its selector. Remove the requested setup
helper copy. Do not restrict the harness to Claude.
- Run a private userspace Tailscale daemon with a separate socket.
Persist its node state before execution. Restore it after restart, bound
startup, serialize reconnect cleanup, and close it during server
shutdown.
- Guard OAuth completion with a durable attempt marker. Disconnect and
newer authorization invalidate older callbacks. Close replaced daemons
and remove cancelled enrollment secrets without changing newer
connection state.
- Preserve SSH host verification and tailnet-specific session identity.
Honor the configured Tailscale executable path, including spaces and
OpenSSH percent escaping.
- Package checksum-pinned Tailscale clients and their license in the
standard runtime image. Exercise the client build stage in PR CI and
include it in image smoke checks.

## Verification

- Merge status: all CI and image checks pass, and current-head Apex is
5/5 with no unresolved comments. GitHub rejected a direct merge because
the active master ruleset requires code-owner approval for
`.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow
pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3)
applies; eligible maintainers already have review requests. Auto-merge
is enabled with the squash method for the verified head. The PR remains
open pending that required approval; no branch policy was bypassed.
- Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including
master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI
verification
passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320).
All required checks pass, all applicable test/build gates are green, and
the branch is mergeable against master. SSH prerequisite #21248 and
Cloud broker paperclipai/paperclip-cloud#1314 are already merged.
- [Apex reviewed this exact revision at
5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437).
There are no unresolved review threads. The PR is classified as Critical
impact.
- Final image qualification passes for the [application Runner
path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267),
[amd64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198),
and [arm64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156).
Typechecks, Runner support, native/Rust tests, provider tests,
serialized-server suites, browser shards, and the remaining applicable
CI checks pass. This is source/image CI qualification, not a production
rollout.
- The last master conflict affected only Docker stage declarations and
their boundary test. Both the checksum-pinned Tailscale stage and
upstream `runner-lockfile` guard are retained, including the upstream
`deps` inheritance. The Tailscale client stage is byte-identical. All
180 cases across five Docker/image contract suites pass locally, along
with server and UI typechecks. The maintained 543-input SDK check has no
drift. Runtime and lock pins are unchanged. Secret/PII and whitespace
scans pass.
- Seven OAuth regression cases failed before the fixes. The current-tab
test uses the real handoff helper, checks that the session is stored
before reauthentication navigation, mounts the shared return page,
resumes consent, and verifies handoff cleanup. Callback tests check
trusted saved-connection context, fixed safe outcomes, failure audit
metadata, denied membership and expired-state rejection, JSON API
errors, and no provider text in browser returns. UI retries require a
connection in the current company's catalog. All 80 OAuth
route/service/UI/shared-handoff cases pass; server and UI source/test
typechecks pass.
- [Server shard 4/12 on the prior
refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436)
passed 2,784 cases and skipped 21, then failed a reassignment fixture's
`TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its
automatic retry found the surviving `REA` prefix. The unchanged test
passes all three cases locally. A controlled reader/cleanup overlap
reproduces the same deadlock, but the exact competing CI query was not
logged. Each case now has a distinct company prefix and identifier; the
owned database is removed at suite end. The three test bodies and
assertions are byte-identical. Eight local reassignment/lock-release
cases pass, and the [final server shard 4/12
passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392).
Production reassignment logic, deadlines, and retries are unchanged.
- The initial master integration passed 330 cases across 17 suites.
Coverage includes connector state, device discovery, network lifecycle,
pairing, routes, SSH binding and exports, environment persistence, and
connector UI. All three unpublished migrations were regenerated after
master's `0479_short_darkstar`: `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL
body is byte-identical to its earlier unpublished migration. Generated
snapshots preserve existing master tables and add only the intended
Tailscale table and secret-purpose column. Exactly five snapshots
remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift
checks pass. The later Docker refresh did not alter this migration
sequence.
- Earlier attempts remain part of the verification record. On `1c020a0`,
Rust cache restoration and server shard 5/12 were interrupted by
shutdown signals, Runner shard 2/3 lost communication with no
retrievable log, and an amd64 provider download failed with npm
`ECONNRESET`. Same-head retries recovered, without establishing the
cause of the communication loss. Later superseded runs also had
cancellations with incomplete logs. These interrupted attempts are not
counted as passes. The final revision has fresh successful CI.
- Local platform limits remain recorded. The Kubernetes suite on macOS
passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and
skipped four Linux-only cases. Unchanged source reproduced the same 12
failures. An initial run also lacked provider dependencies until the
maintained preparation command installed them. Those local attempts are
not green qualification. Final Linux provider CI passes. Local Docker
qualification was unavailable; final image checks passed in CI.
- Before PR separation, a live tailnet/SSH Claude Code task completed on
Ubuntu ARM64 without provider login for the remote user. The selected
Paperclip subscription ran it, wrote the expected file, posted
completion, and marked the task done. File contents and private runtime
cleanup were checked. This is prior live evidence; the final revision
was not live-retested. Other providers have automated coverage only.
Hosted registration for unrelated tailnets and production concurrency
remain qualification limits below.
- The Superagent security and supply-chain reports remain neutral and
inconclusive. Earlier attempts to restart the failed security scan
returned HTTP 404. Supply-chain intelligence lacked coverage for
extracted pinned dependencies and the shared workflow reference. These
reports do not establish successful security qualification. The required
CI and review gates pass; this external coverage gap remains disclosed.

## Risks

- The Superagent security and supply-chain reports are inconclusive. The
security scan re-run request returned HTTP 404, and the supply-chain
scanner lacks coverage for the extracted dependencies and shared
workflow reference. These neutral results are not successful security
qualification.
- The server-owned secret purpose is internal database metadata, not an
HTTP-writable field. The generic secret service owns enforcement so
direct API reads and runtime projection share the same restriction.
Previously copied node state remains valid outside Paperclip until the
enrolled device is revoked in Tailscale; a local disconnect cannot stop
an external copy.
- Tailscale network configurations now require a connected Connector and
selected machine. The unreleased key-only path is removed; any saved
configuration from that path needs Connector setup.
- The connector remains a pilot. The local test used an OAuth app
registered in the consenting user's tailnet. Qualify hosted app
registration for unrelated customer tailnets before general
availability.
- This implementation supports one serving process per instance.
Concurrent replicas need exclusive ownership of the persisted network
identity before production rollout. Preserve the database and company
vault master key across restarts.
- Each machine-pairing preparation creates a new vault secret. Unused
generated keys follow the existing secret deletion lifecycle. A saved
SSH key remains available as an explicit choice.
- Apply additive migrations `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before
application rollout. The new credential version and digest columns are
nullable. Existing unpinned connections require fresh OAuth consent; the
migration deliberately does not trust or backfill a generic secret
value. Regenerate or renumber migrations if master advances the sequence
before merge.
- The target needs an SSH server, its chosen harness, and allowed
tailnet access. A reachable network does not grant SSH or provider
authorization. Pairing reads standard `/etc/ssh` host-public-key files;
custom host-key locations require manual known-hosts configuration.
- Roll back the application revision to remove new entry points.
Existing connection secrets and registered Tailscale nodes retain their
separate lifecycle. Do not drop the table or delete node state as part
of an application rollback.
- Production qualification still needs denied-policy, revocation,
cancellation, and deployment-overlap checks. Local Docker and a
production deployment were not performed.

## Model Used

OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution
and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and this extends existing connector and
environment features
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the feature request fields
- [x] I have not referenced internal issues, local URLs, or private
machine identifiers
- [x] My branch name describes the change
- [x] I have run focused tests locally and they pass
- [x] I have added tests for new behavior and failure paths
- [x] I have updated documentation
- [x] I have documented migration and runtime risks
- [x] All Paperclip CI gates are green
- [ ] Required code-owner approval received for the workflow change
- [x] Greptile Apex is 5/5 on the final head with no unresolved comments
- [x] I will address CI and review findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:15:43 -08:00
Nicky LeachandClaude Fable 5.2 647641a99b feat(ci): let docker.yml copy the published Runner image, behind a repository variable (#21340)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as an attested image, once for each set of its inputs
> - Since #21332 the application image takes its Runner only from the
`runner-artifacts` stage, and a build can replace that stage with the
published image
> - No workflow does that yet, so every application image still builds
its own Runner, and nothing on an application image says which Runner it
has
> - A build that copies the published Runner must also build the
lockfile that Runner was built from. The workflow refreshes the lockfile
before every build, and the server build then refuses the published
Runner
> - This pull request lets `docker.yml` copy the published Runner image
and label the application image with it, behind a repository variable
that is off, and makes that build use the committed lockfile
> - The benefit is one Runner for all images of the same inputs, named
on the image, with a switch that goes back to today's builds without a
revert

## Linked Issues or Issue Description

Refs #21204, #21229, #21230 (the Runner image), #21306 (the resolver),
#21307 (the report-only job), #21308 (the server reads a pack from its
Runner image), and #21331, #21332, #21333 (the application image takes
its Runner from `runner-artifacts`). No issue exists for this change.

**Problem or motivation**

`docker.yml` builds the application image on `master` and on tags. Each
build makes its own Runner. The report-only job that runs the resolver
on `master` feeds no build. Downstream image composers cannot tell from
an application image which Runner it has.

There is a second problem, which only shows when a build copies the
published Runner. The Runner's build stamp covers the whole
`pnpm-lock.yaml`. The step that vendors the Runner into the server fails
when the stamp does not match the tree. `docker.yml` refreshes
`pnpm-lock.yaml` on the runner before every build. The refresh rewrites
the file whenever the manifest of any package is ahead of it, also a
package the Runner does not read. On the tree of this pull request it
rewrites the file even for a pristine checkout: the registry changed the
deprecation text of one package. The published Runner was built from the
committed file, so the image build would fail.

**Proposed solution**

One job resolves the Runner once for the run. Both architecture builds
need it and add its answer to their build: a build context that replaces
the `runner-artifacts` stage, two labels, and the build argument with
which the build checks the Runner. A build that copies the published
Runner does not refresh the lockfile: it builds the committed tree, the
tree the Runner was published for. The job runs only when the repository
variable `RUNNER_IMAGE_CONSUME_ENABLED` is exactly `true`. This pull
request merges with the variable unset, so no image changes when it
merges.

**Alternatives considered**

- Resolve in each architecture build. The two builds could then
disagree, and one image index would hold two Runners.
- Write the labels in the `Dockerfile`. A `LABEL` cannot be left out for
an inline build.
- For the lockfile, copy the Runner image's `/pnpm-lock.yaml` over
`/app/pnpm-lock.yaml` in the `build` stage. Nothing after that step
reads the file: the two vendor checks are its only readers in `build`,
`production` copies it and takes `node_modules` from `deps`, both
`cloud-*` stages install with `--ignore-workspace --no-lockfile`, and
the server reads only a plugin's own lockfile and the pack's own
lockfile at run time. But the image would then ship a lockfile that its
`node_modules` did not come from, the copy changes that file's layer for
every inline build, and each `Dockerfile` edit makes a new Runner image.
- For the lockfile, bind-mount the Runner image's lockfile over the file
for the two steps that run the vendor check. This also edits the
`Dockerfile`, and it makes the lockfile part of the check always true
for a published Runner.
- For the lockfile, let the resolver refresh and compare. The answer
would depend on the registry at two different moments.
- Skipping the refresh needs no `Dockerfile` edit, keeps the stamp check
as strict as it is, and builds exactly the tree the inputs digest was
computed from.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This pull request is built on `runner/app-image-consume`. Merge in this
order:

1. #21331, #21332, #21333: the application image takes its Runner from
the `runner-artifacts` stage, checks the inputs digest a build names,
and stops early on a stale lockfile.
2. `runner/app-image-consume`: the wait and the `unsupported` answer of
the resolver.
3. This pull request.

The same change for the preview cloud image build in `release.yml` is
deferred. It has no pull request yet.

## What Changed

- `.github/workflows/docker.yml`: the job `runner_image` replaces
`runner_image_report`.
- It runs only when `vars.RUNNER_IMAGE_CONSUME_ENABLED == 'true'`, in
this repository. A fork never runs it.
- Steps: `inputs` with no credential; `resolve`; then `gh attestation
verify` of the full index against the signer `runner-image.yml` on
`refs/heads/master`.
- For a `master` push, `resolve` runs with `--wait` and the job token.
For a tag or a manual run it does not wait and receives no token.
- Only the verification step writes the job's outputs. An image that
does not verify fails that step with a warning and writes no output. The
job is `continue-on-error`: the job is red, the run is not, and the
builds are inline.
- Permissions: `contents: read`, `actions: read`, `attestations: read`.
No registry login, no write.
- Limits: 5 minutes for `inputs`, 55 for the lookup, 5 for the
verification, 70 for the job. The wait of 40 minutes ends inside them.
- `build-and-push` needs `runner_image` and adds its outputs to the
build: `build-contexts`, one more line of `build-args`
(`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`), two more lines of `labels`.
With empty outputs these are the values the build had before.
- The step "Refresh lockfile for Docker build context" runs only when
the answer is not `published`.
- The step that looks for a merge-queue image of the commit runs only
when the answer is not `published`. The two steps after it already
depend on its result.
- Labels on an image that copied the published Runner:
`io.paperclip.runner.image=ghcr.io/paperclipai/paperclip-runner@sha256:<full
index digest>` and `io.paperclip.runner.inputs-digest=<digest>`. An
image without them made its own Runner.
- The `Dockerfile` does not change. The Runner image digest does not
move.
- Tests that pin lines this changes follow:
`scripts/merge-group-image.test.mjs` (the condition of the merge-queue
lookup), `scripts/reuse-canary-image.test.mjs` (the `needs` of the build
job), and `docker-build-stamp.test.ts` (no workflow names a build
context; `docker.yml` now names exactly one).
- `server/src/__tests__/docker-runner-workflows.test.ts`: the
report-only tests give way to tests of the new job.
- `server/scripts/prepare-runner-vendor.test.mjs`: one fixture test of
the lockfile case.
- `doc/runner-image-contract.md` and `doc/DOCKER.md`: how an application
image gets its Runner, the labels and what their absence means, the
variable, the wait, the verification, the lockfile rule, the merge-queue
rule and the rollback.

## Verification

### Real builds

`docker buildx build --target build --platform linux/arm64 --output
type=cacheonly`, with a `docker-container` builder made for this test.
The Runner image is the publication form
(`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1`, `--target runner-artifacts`)
built from a clean export of this branch, exported as an OCI layout:
inputs `c4f9d089…34f0`, image `sha256:bef7e9cb…ffef`. The application
builds receive `--build-context
runner-artifacts=oci-layout://…@sha256:bef7e9cb…` and
`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST=c4f9d089…`. The refresh command
is the one in the workflow, run with pnpm 9.15.4 in the export: `pnpm
install --resolution-only --ignore-scripts --no-frozen-lockfile`.

| Build context | Result |
| --- | --- |
| The committed tree | Passes in 78 s. Both vendor checks: `reusing the
current Runner build (inputs unchanged …; 1333 files and 1860 outputs
checked)` |
| `ui/package.json`: `clsx` `^2.1.1` → `^2.1.0`, then the refresh
command (what the workflow did before this change) | Fails at `RUN node
server/scripts/prepare-runner-vendor.mjs`:
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1 forbids the Runner rebuild
this step needs (inputs changed since the recorded build; 1333 files
checked)` |
| The committed tree, then only the refresh command | Fails the same
way. The refresh changed only the `deprecated` text of one package |
| The same `ui/package.json` change with no refresh (what the workflow
does now for a `published` answer) | Passes in 66 s. `deps` resolves the
manifest inside the image; both vendor checks reuse the Runner |
| One line added to `packages/paperclip-runner/src/index.ts` | Fails at
the vendor step, as before: a Runner of other inputs is still refused |

Lockfile behind the Runner's own manifests
(`packages/paperclip-runner/package.json`: `smol-toml` `^1.9.1` →
`^1.9.0`), no build context:

| Step | Result |
| --- | --- |
| `runner-image-resolve.mjs inputs`, then `resolve` | `lock: behind`, no
digest; `inline (lock-behind)` |
| Build without the refresh | Fails in 4 s at the `runner-lockfile`
stage: `ERROR: pnpm-lock.yaml is behind the manifests the Runner is
installed from …` |
| Build after the refresh command (what the workflow does for an
`inline` answer) | Passes in 193 s: the `runner-lockfile` stage passes,
the build makes its own Runner with its pack, and both vendor checks
reuse it |

### Tests

macOS (Node 26.12.0) and Linux (`node:24.22.3-bookworm`, linux/arm64,
with `jq` and `zstd` added), on this branch:

- `pnpm exec vitest run` on `docker-runner-workflows`,
`runner-image-resolve`, `runner-image-workflow`, `docker-build-stamp`,
`container-init-reaping`, `cloud-image-bundled-plugins`,
`docker-image-layer-reuse`, `cloud-image-sentry`,
`runner-lockfile-guards` and `runner-image-identity` in
`server/src/__tests__`, and
`server/scripts/prepare-runner-vendor.test.mjs`: 225 tests in 11 files
pass on Linux; on macOS 224 pass and 1 is skipped (it needs bash 4.4).
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/preview-artifacts.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/release-dry-run-notes.test.mjs`: 991 of 991 pass on
both.
- The changed test files typecheck with the options of
`server/src/__tests__/tsconfig.json`.

What the workflow tests hold:

- Variable off: the job's condition is the variable and the repository,
with no alternative. With empty outputs, the build step has exactly the
3 build arguments, 0 build contexts and 1 label entry it had before, and
the refresh step runs.
- Published: both architecture builds are one job that reads only the
outputs of `runner_image`, with no architecture in those lines.
- The verification step runs as real shell against a stub `gh`. When
`gh` succeeds it writes the context, the labels and the build argument.
When `gh` fails the step exits 1, writes no output and prints the
warning. The step has no `continue-on-error`; the job has. The condition
of the build job does not name the resolver job and has no `success()`.
- The lookup step runs as real shell against a stub `node`. It passes
`--wait` only when the event is a `master` push.
- Credentials: only the lookup and the verification steps have a token.
The steps that build have no `env`.
- The refresh step has the condition `needs.runner_image.outputs.mode !=
'published'`, and it is the only `pnpm install` of the build job.
- The fixture test gives the vendor step a Runner that another tree
built for the committed lockfile. The step reuses it. After a change of
one specifier of another package in the lockfile the step fails and
names the lockfile. With the committed lockfile back it reuses the
Runner again.
- The merge-queue workflow and the pull-request check do not name the
resolver, the variable, a build context or a Runner label.

Read-only checks against the real services:

- `gh attestation verify
oci://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…1bf7 --repo
paperclipai/paperclip --signer-workflow
paperclipai/paperclip/.github/workflows/runner-image.yml --source-ref
refs/heads/master`: exit 0 for the full index, and exit 0 for the slim
index `sha256:69ab2001…69b3`.
- The same command with `--signer-workflow …/docker.yml`: exit 1 for
both. With `--source-ref refs/heads/not-master`: exit 1. With `--repo
cli/cli`: exit 1. For a digest that does not exist: exit 1. With no
token: exit 4.
- The parsed YAML of `docker.yml` differs from its base only in the
resolver job, the `needs` of the build job, the conditions of the
refresh step and of the merge-queue lookup, and the three build inputs.

Never run:

- `docker.yml`. Only GitHub can run it, and it has no pull-request
trigger.
- A build with the variable on. The first one is the first run after a
maintainer sets the variable.
- A build that pulls the published Runner from the registry as
`docker-image://`. The real builds used the same image form from a local
OCI layout.
- An `amd64` build.
- The `production` and `cloud` targets. The real builds stopped at
`build`, the last stage that checks the Runner.

## Risks

- With the variable on, an application build on `master` starts after
the resolver job. That is 2 to 3 minutes when the image exists, and up
to 40 minutes from the push when the commit's own Runner publication is
in progress. `docker.yml` holds one concurrency slot for each ref, so a
long wait can make more `master` commits lose their image build to a
newer push.
- A build that copies the published Runner ships the committed
`pnpm-lock.yaml` in `/app`, while `deps` resolves newer manifests of
other packages inside the image. A local `docker build` has always done
this; a CI image did not. Nothing in the image reads that file.
- Any other build that gives the `Dockerfile` a published Runner and a
refreshed lockfile fails at the vendor step. That is the intended answer
to a tree the Runner was not built for, and the message names the
lockfile.
- The pack of an image that copied the published Runner names the commit
that published the Runner (`runnerSourceRevision`), which can be older
than the commit of the image. A consumer that compares the two must
accept that before the variable is set.
- A failed verification makes the resolver job red on a green run. That
is on purpose: it means a published tag whose attestation does not
verify.
- `scripts/merge-group-image.mjs` refuses to promote a merge-queue image
for a push that changes `docker.yml`. The push that lands this change is
such a push. The merge queue is not enabled.
- With the variable off there is no report of the resolver on `master`
any more: the job that replaced the report-only job does not run.
- Rollback: set the variable to another value or delete it. The next run
builds inline, with the refresh. Images that exist do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds, read-only
calls to the GitHub API and the container registry).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:47:20 -08:00
Nicky LeachandClaude Fable 5.2 e824c5cc16 feat(ci): let the Runner image resolver wait for the publication of a master push (#21339)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, once for
each set of its inputs
> - The application image will copy that Runner instead of making its
own. `scripts/runner-image-resolve.mjs` tells a build whether the Runner
of its inputs is published
> - On a `master` push, the commit's own run of `runner-image.yml` is
often still publishing when the application build asks. The resolver
answers once, so that build is told `absent` for an image that exists
some minutes later
> - The resolver also cannot say that a tree is not safe to give a
published Runner to. A `Dockerfile` that does not check the Runner it
receives builds with any Runner
> - This pull request adds an opt-in wait for the commit's own
publication run, and an `unsupported` answer for a `Dockerfile` that
cannot check its Runner
> - The benefit is that the next pull request can let `docker.yml` copy
the published Runner without a blind timeout and without a label that is
not true

## Linked Issues or Issue Description

Refs #21204, #21229, #21230 (the Runner image), #21306 (the resolver),
#21307 (the report-only job), and #21331, #21332, #21333 (the
application image takes its Runner from `runner-artifacts`). No issue
exists for this change.

**Problem or motivation**

A `master` push starts `runner-image.yml` and `docker.yml` at the same
time. Publication takes about 30 minutes from the push, and several of
those minutes are runner queue time. An application build that asks the
resolver at once gets `absent` for every commit that changes the
Runner's inputs, and for commits that follow one closely. A fixed poll
of the tag has the opposite fault: when no image will ever come (the
lockfile is behind, or the run failed), every build waits the full
timeout.

**Proposed solution**

`resolve --wait` waits for a run, not for a time. It finds the runs of
`runner-image.yml` for the push of the commit to `master`, and it looks
up the image again when they end. No caller uses the flag in this pull
request, so no workflow changes behavior.

**Alternatives considered**

- Poll the tag until a timeout. Every build with no image to come waits
the full timeout.
- Call `runner-image.yml` from `docker.yml` as a reusable workflow.
Publication would then share the one concurrency slot of `docker.yml`.
- Wait until the run completes and only then look. The run goes on after
publication (it writes the build cache), so the wait would last longer
than needed. The command looks up the image while a run is in progress.
- Verify the attestation in the resolver. The workflow that uses the
answer does it, in a step of its own, in the next pull request.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This pull request is built on #21333. Merge in this order:

1. #21331, #21332, #21333: the application image takes its Runner from
the `runner-artifacts` stage, checks the inputs digest a build names,
and stops early on a stale lockfile.
2. This pull request: the wait and the `unsupported` answer of the
resolver.
3. `runner/app-image-consume-docker`: `docker.yml` copies the published
Runner, behind a repository variable.

The same change for the preview cloud image build in `release.yml` is
deferred. It has no pull request yet.

## What Changed

- `scripts/runner-image-resolve.mjs`
- `resolve --dir <directory> --wait`. When the first answer is `absent`
or `incomplete`, the command lists the runs of `runner-image.yml` for
the commit (Actions API, `GITHUB_SHA`, `GITHUB_TOKEN`) and looks again
when they end.
- Only a run of the `push` event on `master` in this repository counts.
A pull-request, scheduled or manual run of the same commit is never
waited for. The request filters by event and branch, and the script
checks each run again.
- A run that fails, is cancelled or publishes nothing ends the wait when
it ends. With several runs for one commit, the command waits for all of
them.
- While a run is in progress, the command looks up the image every 30
seconds, so the wait ends at publication.
  - A run that is not listed yet is looked for during 2 minutes.
- The limit is 40 minutes from the push: `RUNNER_IMAGE_WAIT_MINUTES`,
the one constant. The push time is the creation time of the oldest run
of the commit, so a build that starts late waits less.
- Three failed API requests in a row end the wait. No token or no full
commit SHA means no wait.
- After every ending the command looks up the image one last time. No
ending is an error. Without an image the answer is `inline`, and the
command prints a notice with the reason.
- `inputs` reads the `Dockerfile` first. Without a line `ARG
PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST` it runs no build, and `resolve`
answers `inline` with the reason `unsupported`.
- `resolve` reports `build_args`: the value of that argument for a
`published` answer.
- The wait and the list of runs take the clock, the sleep and `fetch` as
arguments, so the tests use no network and do not sleep.
- `server/src/__tests__/runner-image-resolve.test.ts`: every outcome of
the wait and the answer that follows it, the run filter, the limit
measured from the push, the API failures, and the command line.
- `doc/runner-image-contract.md`: a "Waiting for a publication" section,
`build_args`, and the `unsupported` answer.

## Verification

macOS (Node 26.12.0) and Linux (`node:24.22.3-bookworm`, linux/arm64,
with `jq` and `zstd` added), on this branch on top of #21333:

- `pnpm exec vitest run` on `runner-image-resolve`,
`docker-runner-workflows`, `runner-image-workflow`,
`docker-build-stamp`, `container-init-reaping`,
`cloud-image-bundled-plugins`, `docker-image-layer-reuse`,
`cloud-image-sentry`, `runner-lockfile-guards` and
`runner-image-identity` in `server/src/__tests__`, and
`server/scripts/prepare-runner-vendor.test.mjs`: 218 tests in 11 files
pass on Linux; on macOS 217 pass and 1 is skipped (it needs bash 4.4).
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/preview-artifacts.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/release-dry-run-notes.test.mjs`: 991 of 991 pass on
both.
- The changed test file typechecks with the options of
`server/src/__tests__/tsconfig.json`.

Read-only checks against the real services:

- The lookup, with no credential, answered `published` for the inputs
`d4df566f…af00` and `0e6662ad…d1ca`, with the index digests that
`master` published, and `absent` for a digest that does not exist.
- The real answer of the Actions API for the `runner-image.yml` runs of
four `master` commits went through the wait, with a clock that does not
sleep. Three completed runs ended it at once. A queued run ended it at
the limit of 40 minutes from its creation time. A commit with no run
ended it after the 2 minutes for a run that is not listed.
- Of the 60 newest runs of `runner-image.yml`, the filter accepted the
19 `push` runs on `master` and none of the 41 pull-request runs.
- `runner-image-resolve.mjs inputs` ran on a clean export of the tree of
the next pull request, with a `docker-container` builder. The answer was
`lock: current` and the inputs digest `c4f9d089…34f0`. The `Dockerfile`
of #21332 declares the argument, so the answer was not `unsupported`.
The same command on a tree with one changed range in
`packages/paperclip-runner/package.json` answered `lock: behind`, and
`resolve` then answered `inline (lock-behind)`.

Never run:

- `resolve --wait` in a workflow. No workflow calls it in this pull
request.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- `resolve` now prints a notice for every `inline` answer. The
report-only job in `docker.yml` therefore shows one notice on each
`master` push whose Runner image is not published yet, until the next
pull request replaces the job. Nothing reads that job.
- `inputs` answers `unsupported` for a tree whose `Dockerfile` does not
declare `ARG PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST` at the start of a
line. #21332 adds that line; a later edit that moves or renames it turns
every answer to `inline`.
- The wait is bounded by time from the push, not by its own duration. A
build that starts more than 40 minutes after the push looks once and
does not wait.
- A registry that does not answer adds at most about 3 minutes to one
lookup, because the registry client retries. A caller must give the step
more than 40 minutes. The next pull request gives it 55.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, read-only calls to the GitHub API
and the container registry).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:33:40 -08:00
4c5e7bf867 fix(heartbeat): retry a legacy Stop that loses the task-row lock race (#21492)
Legacy Stop retries its terminal write up to 3 times when another writer
holds the task row past the 1s lock_timeout, and returns a retryable
409 run_stop_busy instead of a 500 when contention persists. Board Stop
also bounds its earlier applySupersession task-row lock to 1s so it can
no longer wait without limit.

Co-Authored-By: Paperclip <paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:29:44 -08:00
Nicky LeachandClaude Fable 5.2 999709b0a6 fix(docker): stop a stale-lockfile build early and keep beta and stable releases buildable (#21333)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The application image now takes its Runner from the
`runner-artifacts` stages, which install with a frozen lockfile
> - A contributor cannot commit `pnpm-lock.yaml`. After a change to a
manifest of the Runner, the lockfile on `master` is behind until the
refresh merges
> - In that time a plain `docker build` fails late, inside a long step,
with a message that is easy to miss. And a beta or a stable release can
be cut from such a commit, because the release workflows refresh the
lockfile for their own build
> - This pull request adds an early check with a clear instruction to
the image build, and the same check to the beta and the stable release
> - The benefit is a failure in seconds that says what to run, and beta
and stable tags that always build from source

## Linked Issues or Issue Description

This is the third part of a chain. The first part is #21331. The second
part takes the Runner of the application image from the
`runner-artifacts` stage. Refs #21204 and #21306 (the Runner image and
its lockfile rule). No issue exists for this change.

**What happened?**

A tree with a lockfile behind the manifests of the Runner fails `docker
build` after the daemon compile (169 s in the test). The error is
`ERR_PNPM_OUTDATED_LOCKFILE` in the middle of the largest `RUN` of the
`Dockerfile`.

**Expected behavior**

The build fails within seconds of the check running. The message names
the files and the command to run. The application's dependency install
starts from the check, so it cannot start before the check passes.
Stages that do not depend on it (the Runner's toolchain download and
daemon compile, and the tool layer of `production`) can already be
running, and the failed check cancels them. A beta or a stable release
from such a commit does not start.

How often this happens: of 72 nightly tags since 2026-08-24, 9 are on a
commit whose lockfile is behind the manifests of the Runner (2026-09-01,
09-18 through 09-23, 09-28 and 10-26). Of 12 beta tags, none is. A beta
is the promotion of a nightly, so a beta can be cut from such a commit.

**Steps to reproduce**

1. Change a dependency range in
`packages/paperclip-runner/package.json`. Do not refresh
`pnpm-lock.yaml`.
2. Run `docker build .`.

**Paperclip version or commit**

`master` with the previous pull request of this chain.

**Deployment mode**

Not applicable. This is the image build and the release tooling.

## What Changed

- `Dockerfile`: a `runner-lockfile` stage copies only the manifests the
Runner build installs from, the lockfile and the patches. It runs the
frozen, filtered, offline lockfile check, with the arguments of
`runnerLockState`. `deps` starts from it. For a stale lockfile it prints
the files and `pnpm install --resolution-only --ignore-scripts
--no-frozen-lockfile`. For any other failure it prints the output of
pnpm and says so.
- `scripts/check-release-runner-lockfile.mjs`: exports the Runner inputs
of a commit with `git archive` and gives them to `runnerLockState`
(`scripts/runner-image-publish.mjs`). Exit 0 for current, 1 for behind,
2 for an error.
- `scripts/release.sh` and `scripts/release-lib.sh`:
`require_runner_lockfile_current` runs for the `beta` and the `stable`
channel, after `--print-version` exits and before any build. It runs in
a dry run too. It does not run for `canary` or `nightly`.
- Both beta paths end in the `publish_beta` job: the promotion of a
nightly, and a `candidate/beta-*` branch. That job checks out the commit
that `select_beta` chose and runs `release.sh beta` there. So the check
reads the committed files of the commit the beta is built from. A beta
is refused before its soak starts. The stable that promotes it is
checked again.
- Tests: `server/src/__tests__/runner-lockfile-guards.test.ts` (18
tests). Three of them pin the checkout of `publish_beta`,
`preview_stable` and `publish_stable` to the selected commit. Two tests
plus one stub in `scripts/__tests__/release-dry-run-notes.test.mjs`
cover the beta, the candidate beta and the stable path.
- Docs: `doc/DOCKER.md`, `doc/RELEASING.md`,
`doc/runner-image-contract.md`.

## Verification

Builds: `linux/arm64`, local. They ran before the rebase onto `master`,
except the last one. `master` changed neither the `Dockerfile` nor a
manifest in between.

- A tree with `ajv` changed from `^8.20.0` to `^8.19.0` in the manifest
of the Runner: `docker buildx build --target build` fails after 6 s at
`runner-lockfile`, with the message.
- The same tree after `pnpm install --resolution-only --ignore-scripts
--no-frozen-lockfile` on the host, which is what the image workflows
run: `--target build` builds. The check passes, the Runner stages
install, and the vendor step logs `reusing the current Runner build` in
both of its runs.
- After the rebase: `build` and `runner-provider-pack` built from this
branch without a build commit. The check passed.
- The release check with the real pnpm: `behind` for the first tree,
`current` for the second, exit 0 for this checkout.
- `vitest run server/src/__tests__/runner-lockfile-guards.test.ts` with
the other changed server tests, on macOS and in `node:24.22.3-bookworm`
(249 pass in 11 files on Linux). The test file typechecks.
- `node --test scripts/__tests__/release-dry-run-notes.test.mjs`: 14
pass. `scripts/release-lib.test.mjs`: 15 pass on Linux; on macOS 4 of
its tests fail on `master` too. `release-verify-workflow.test.mjs` and
`acpx-patch-packaging.test.mjs`, which read the release scripts: pass.
- `docker buildx build --check`: no warning.
- The Docker builds above ran before the second part of the chain got
its expectation step and before the beta guard. The expectation step was
built on the second part alone. This branch as it is now was not built
again; its tests ran.
- Not run: the release workflow, any `linux/amd64` build, the full test
suite. Only CI and a release dry run can prove the release path.

## Risks

- **A beta can be refused.** When the newest nightly is on a commit
whose lockfile is behind, `release.sh beta` stops, in the dry run and in
the real run. The operator must promote a nightly whose commit includes
the lockfile refresh, or cut a candidate branch that includes it. `node
scripts/check-release-runner-lockfile.mjs --commit <ref>` gives the
answer before the dispatch.
- **A stable can be refused** only when its beta was cut before this
change, or with release tooling that does not have the check. Promotions
run the release tooling of the source commit.
- The check runs in `release.sh`. In the beta workflow it runs in
`publish_beta`, after the approval of the `npm-beta` environment and
after the candidate verification. In the stable workflow it runs after
verification, not in the `preflight_stable` job. The selection jobs have
no pnpm.
- `release.sh stable` now needs `git`, `tar` and `pnpm` on `PATH` before
its first build step. The workflow has them.
- `scripts/__tests__/release-dry-run-notes.test.mjs` is not in a CI
command today. The Vitest file carries the enforced tests.
- The image build has one more small stage. It reads only manifests, so
a source change does not run it again.
- The digest of the Runner image changes again, because its identity
covers the whole `Dockerfile`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude
Code. The model used tools: it edited files, ran the tests and ran the
Docker builds that this description reports.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:17:15 -08:00
DottaandClaude Fable 5.2 05fef9f27b perf(server): add a load model of per-agent server cost and the plan for 1,000 concurrent agents (#21418)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every agent run of a workspace
> - A hosted workspace slows down with about 50 agents running at the
same time
> - The target is 1,000 agents at the same time, served by several
server processes
> - This pull request adds the plan and a local load model that measures
what each running agent costs the server
> - The benefit is a measured ranking of the causes, and a repeatable
before/after number for each fix

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The server controls agent runs. With about 50 runs active, requests wait
and run events fall behind.

**Current behavior**

Each text delta from an agent is one database transaction. In the load
model, 50 agents at 20 deltas per second issue about 3,900 statements
per second, and the process completes about half of the offered deltas.
Agent authentication slows from 7 ms to 464 ms because it shares the
connection pool.

**Proposed behavior**

The plan gives three phases: measure, decrease the cost per agent, then
let several server processes own runs. Each fix is one pull request with
a test and a before/after number from the load model.

**Reason and benefit**

The ranking was an estimate from code. The load model replaces the
estimate with a measurement and gives each fix a number to improve.

## What Changed

- Added `doc/plans/2026-11-13-thousand-concurrent-agents.md`: finish
line, evidence, load model result, phases, and owner decisions. It now
also has a "Load model after Phase 1" section. That section has a table
at 50, 100, 200 and 400 agents and a row at 200 agents with deltas and
leases as before Phase 1. The 400-agent delta count was measured before
the count fix and is marked as not reliable.
- Added `scripts/benchmark-agent-scaling-load.ts`: N simulated agents
call the real event append, controller checkpoint save, lease renewal,
and agent authentication code against an embedded PostgreSQL. It reports
statements for each database pool, event-loop utilization, and lateness
for each source.
- Deltas: by default the model commits deltas as the server port does
after #21460. The first text after another event commits at once. Later
deltas commit in one transaction (`appendHeartbeatRunEventBatch`) per
250 ms or 100 deltas, and the stream does not wait for the receipts.
`DELTA_BATCH_MS=0` keeps one transaction per delta, as before batching.
- Leases: by default the model renews every lease through the shared
schedule from #21444 (`startNativeSessionExecutionLeaseRenewal`,
`renewNativeExecutionLeasesNow`). `LEASE_MODE=single` keeps one
transaction per run, as before.
- The commit counter is read before the checkpoint drain, so commits
that come after the window do not count in it.
- Per-source counts freeze when the window closes. An operation that
settles later, while queued work drains, is not done.
- Batch lease mode needs `LEASE_S` of at most 20. The shared schedule
renews a remote lease on its own 20 s after its last renewal, and the
model does not count that work, so the script rejects a longer interval.
- Added `server/src/__tests__/agent-scaling-load-model.test.ts`. It runs
the model end to end in both modes: the default (batched deltas and
shared lease renewal) and `DELTA_BATCH_MS=0 LEASE_MODE=single`. It
checks that every source finishes without failures and that each pool is
counted.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/agent-scaling-load-model.test.ts` on a tree that contains
#21460 and #21444: 2 of 2 pass (both modes). The test also checks that
no source reports more done than offered.
- `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-agent-scaling-load.ts` with `AGENTS=50 SECONDS=20`
and higher agent counts. The results are in the plan.
- `pnpm check:module-boundaries` passes.
- No product code changed.

## Risks

Low risk. The pull request adds a document and a script that no product
code imports. The load model runs on loopback PostgreSQL, so its
absolute numbers are not a prediction for a hosted database. The plan
states this limit.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and
code execution. Subagents on the same model read the code paths.
- The two-mode load model test and the merge with master: Claude Opus
5.5 (`claude-opus-5-5`), Claude Code, with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 16:06:40 -06:00
defed64af9 fix(native-runtime): retry a native failure write after a brief lock timeout (#21494)
An interrupted native turn records its same-run resume in one transaction
under a 1 s lock_timeout. When another transaction briefly held a row that
the write needs (for example the agent row that the new recovery action
references), the insert failed with 55P03 and the whole write rolled back.
The run kept its dead lease, and the task needed reconciliation instead of
the recorded automatic resume.

Retry the whole failure write on transient lock contention with the
existing bounded helper. A rolled-back attempt has no effect, so a fresh
attempt is safe. The regression test holds the agent row until one attempt
is cancelled by its lock timeout, so only a retry can complete the write.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:02:04 -08:00
DottaandPaperclip 0ead74b4f3 feat(slack): show useful progress without extra messages (#20827)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - Chat connectors show agent work in the source conversation.
> - Slack progress uses a small set of general status messages.
> - Native runtimes already mark some completed assistant notes as
public progress.
> - The server can select those notes and control their size and rate.
> - This pull request keeps one progress message and sends the final
answer as a new reply.
> - Users can follow useful changes without reading each tool event.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Automatic progress updates in Slack task threads.

**Current behavior**

Users see general messages such as “Maya is making progress”. The first
answer can replace that message. A source update that arrives during the
progress delay can be lost if no later event arrives.

**Proposed behavior**

Show the latest short public note in one message. Wait 10 seconds before
the first update and at least 30 seconds between updates. Recover
pending work after a restart. Send the final answer as a fresh reply.

**Reason and benefit**

Users can see what the agent checks and what it found. The thread stays
short. The model needs no new tool call or summary model.

Related change: [resolved-card chat
bindings](https://github.com/paperclipai/paperclip/pull/20825). This
branch includes that master revision. The progress change uses the
existing source authority checks.

## What Changed

- Select completed `agentMessage` items from the normalized PRP
`progress` channel. Keep unknown channels, reasoning, tool output, and
raw provider fields out of progress text.
- Sanitize each note. Keep at most two sentences and 400 characters. Add
a task link when a public board URL is available.
- Prefer public notes over general tool status. Remove repeated text and
combine bursts into the latest pending note.
- Use run events and the publication outbox as saved state. Use the
shared deadline worker for the grace period and update delay. A delivery
receipt wakes the worker when an earlier update is still in flight.
Settled history does not poll.
- Apply the run's registered-secret filter before text selection or
storage. Reuse the latest usable note within a sweep. Invalidate the
cache when the source event or secret registry changes.
- Reuse the exact source-bound Slack fast-response receipt when it
replaced the working placeholder. Check its current outbound message
link before editing it.
- Send Slack answers as new replies. After delivery, queue a separate
cleanup edit for a generic working message. Keep its current outbound
link as the edit authority. Leave useful public notes in place. Cancel
pending public progress when an answer, question, or terminal run takes
over. Keep the existing delivery policy for other connectors.
- Add one short Slack prompt instruction. Update product documentation
and regression tests.

The Slack module owns public-note selection, cadence, lane eligibility,
and cleanup. The shared progress service holds its existing issue and
run locks and calls a narrow provider policy. The secret registry
supplies redaction and a version token for the per-sweep note cache. The
existing outbox owns delivery, retries, and authority checks. The Slack
module owns receipt reuse. Shared delivery changes supply the receipt
wake and select the Slack reply policy. No schema or configuration
change is required.

## Verification

- The initial content-selection regression failed before the
implementation and passed after it.
- The follow-up reproduced two review defects before the fixes. A
registered synthetic secret appeared in saved progress. A generic
working message stayed active after the answer. Both regressions now
pass.
- The final content, database-worker, query, protocol-boundary, and
redaction selection passes 78 tests across four files.
- The earlier affected transport selection passes all 65 cases. It
covers public progress, native cadence, separate answers, Slack and
Telegram retry isolation, FIFO restart, and cleanup ownership. Its first
run passed 64 cases and found one old assertion that rejected the
intended cleanup edit. The corrected assertion checks the exact
successor message ID.
- Full server typecheck passes: `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm
--filter @paperclipai/server typecheck`. An earlier attempt found two
transaction type errors. The helper contracts now accept their actual
database operations. The final run passes all source and test layers.
- The broad Slack selection stopped after 189 passing cases when the
sandbox killed its worker with SIGKILL. That attempt is not a pass. The
focused transport run above supplies affected local coverage. Required
CI must supply broad coverage on the final head.
- `pnpm verify --explain` returns exit 2 because the product
documentation change requires broad CI coverage. No full local suite was
started.
- Preflight installed dependencies and built the SDK. Cargo is absent,
so local Rust checks could not run. No Rust source changed.
- [CI on
`02552bb584`](https://github.com/paperclipai/paperclip/actions/runs/43493597242)
passed build, typecheck, policy, all browser shards, Runner/Rust checks,
and the other test lanes. One chat shard failed in three Slack cases.
Each expected one edit after an answer was already sent. The intended
cleanup adds a “Response sent.” edit before the later failure notice.
All three failures reproduced locally. The test assertions now check
both exact edit texts and the original run's message ID. They also check
that another sweep adds no edits.
- The latest local run passes all 85 selected transport cases. It
includes all 18 Slack/Telegram answer and failure orders, plus progress,
deferred FIFO, retry isolation, and cleanup coverage. Command: `pnpm
exec vitest run server/src/__tests__/chat-channels.integration.test.ts
-t 'preserves selected answer and failure lanes|public progress|native
progress|keeps intermediate.*agent comments|keeps
a.*retry.*queue|FIFO'`.
- [Final
CI](https://github.com/paperclipai/paperclip/actions/runs/43499797545)
passed on `55ec4c053fe5263d9033d807777dbf2e679407e3`. All 52 executed
jobs passed. This includes build, typecheck, general tests, serialized
tests, all eight browser shards, Runner/Rust checks, and the required
aggregates. The affected chat shard now passes. The test-file selection
job was skipped because this run used full coverage.
- [Greptile](https://github.com/paperclipai/paperclip/runs/132142301124)
completed on the same head with 5/5 and no new findings. All four inline
review threads are resolved. The review job and Socket checks passed. Of
61 check runs, 57 succeeded, three were skipped, and one was neutral. No
checks remain pending or failed. The PR is mergeable with a clean state.
- [Superagent Security
Scan](https://github.com/paperclipai/paperclip/runs/132142345827)
returned a neutral result with “PR scan workflow failed”. It supplied no
security verdict. Do not count it as a successful scan.
- Live Slack notification and unread behavior has not been tested.
Service tests use real PostgreSQL and a fake provider runtime.

## Risks

- Only runtimes that map commentary to the explicit public `progress`
channel supply contextual prose. Other runtimes retain general status
messages.
- Slack final answers now create a fresh reply. This is an intended
delivery change.
- Public commentary must suit the current conversation audience. Text
sanitization adds protection; it does not authorize sharing unrelated
private material.
- Source events and outbox rows supply recovery state. No new timer
table or recurring idle scan is added.
- The existing large integration suite retains its provider fixtures and
lifecycle assertions. No source file crosses the 1,000-line threshold in
this change.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code tools. The exact
deployment ID was not supplied.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:48:05 -06:00
Devin FoleyandClaude Opus 5.5 06a4a07781 fix(native): settle a Stop seen at budget admission as that cancellation (#21483)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs prepare a sandbox, then reserve budget, then start
provider work
> - An operator can press Stop while a native run still prepares
> - Budget admission reads the recorded Stop on the locked run row and
throws a generic 409 "Run cancellation already requested"
> - Heartbeat treats that throw as a run failure, so an expected Stop
becomes a failed run and a `run_failure` error event
> - This pull request makes admission report the Stop through the
existing native startup Stop contract
> - The benefit is that a Stop during preparation ends as a
cancellation, with no false failure, no retry, and no stranded Stop
intent

## Linked Issues or Issue Description

**What happened?**
An operator stopped a native run while it prepared. `reserveRunBudget`
found `resultJson.cancellation` / `startupCancellation` on the locked
run row and threw `conflict("Run cancellation already requested")`.
Heartbeat sent this error to `failRun`. The run ended `failed` with
`adapter_failed` or `native_continuation_requires_reconciliation`, and
Sentry got a `run_failure` event with `cancellationExpected: true`. The
pending native Stop intent stayed `pending`. Restart recovery then
logged `native_stop_requires_existing_remote_session` for a session that
never existed. Two staging Sentry groups show this: 20 events since
2026-11-10.

**Expected behavior**
A Stop that the server records before provider work starts ends the run
as `cancelled`. It does not report a failure, it does not schedule a
retry, and it acknowledges the Stop intent.

**Steps to reproduce**
1. Start a native run on a task.
2. Press Stop while the run prepares, before budget admission.
3. See that the run ends `failed` with the error "Run cancellation
already requested".

No public issue or pull request covers this. A GitHub search for
"cancellation already requested" found no related fix.

## What Changed

- `reserveRunBudget` now throws the existing
`NativeCancellationPendingRecoveryError` for a native run when it finds
a recorded Stop. Heartbeat already settles this error through
`cancelRunInternal("Recovered durable native run cancellation")`. This
is the same path as a Stop that native startup sees when the session
opens.
- Legacy runs keep the 409 conflict. The incident evidence is for native
runs only.
- The final dispatch gate does not change. It already treats a recorded
Stop as a lost race, not as a failure.
- New regression file
`server/src/__tests__/heartbeat-native-stop-before-admission.test.ts`.

## Verification

- `npx vitest run
src/__tests__/heartbeat-native-stop-before-admission.test.ts` (in
`server/`, embedded Postgres):
- Before the fix: the test "settles an operator Stop recorded before
budget admission" fails. The run ends `status: "failed"`, `error: "Run
cancellation already requested"`. This is the staging result.
- After the fix: 2/2 pass. The run ends `cancelled`, the provider
session never opens, no retry run exists, any task hold records
`originalFailureCode: "cancelled"`, and the native Stop intent becomes
`acknowledged`.
- The second test is a safety case. A real `heartbeat.cancelRun` that
settles during admission still ends `cancelled`, and the Stop request
resolves.
- Focused suites pass, 13 files and 304 tests: native pre-dispatch
retry, admission and cleanup, budget enforcement scoping, closed native
receipt accounting, native recovery lease reacquire, native cleanup
admission, native startup cleanup recovery, native cancellation request,
native Stop pending, run-dispatch Postgres adapter, companies service,
execution preparation timing.
- `pnpm typecheck` in `server/` passes.
- `pnpm verify` selects a broad set because the change is under
heartbeat. CI supplies that coverage.
- Not verified: a live staging Stop. Staging gets this fix after merge
through the normal automatic deploy.

## Risks

- Low risk. The change applies only when admission finds a recorded Stop
on a native run. Before this change that run always failed.
- `budget-reservations.ts` now imports an error class from
`native-session-executor.ts`. The executor does not import
`budget-reservations`, so this adds no import cycle.
- A user Stop with `dispatched: false` can still put an
`active_run_watchdog` hold on the task. That is existing behavior for
Stops that reach no provider (#18820 keeps the strict dispatched
receipt). This PR does not change it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution,
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:47:52 -08:00
Nicky LeachandClaude Opus 5.5 5df1ada9c1 feat(plugins): send configured environment on plugin task resume (#21352)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21335 lets a plugin task provider that restarts the Runner on
`resume` opt in (`resumeCredentialVersion: 1`) to receive the claimant
run's harness credential
> - A provider restarting the Runner also needs the agent's configured
environment, which Core sends on `resume` only during a session handoff
> - Without it, a Runner a provider moves to a new computer starts
without the agent's configured environment variables
> - This pull request widens that opt-in to both launch inputs, under a
name that says so, before any provider adopts the old name
> - The benefit is that a provider can restart a parked Runner on resume
with the same credential and environment a `submit` would carry

## Linked Issues or Issue Description

Refs #21289
Follows: #21335

**What existing behavior does this improve?**
The opt-in resume launch inputs added in #21335.

**Subsystem affected**
Plugin SDK environment task schema (`describe` capability), the plugin
Runner task owner's `resume`, and the plugin environment task docs.

**Current behavior**
With `resumeCredentialVersion: 1`, `resume` carries `credential`.
`configuredEnvironment` is sent on `resume` only for a session handoff.

**Proposed behavior**
The capability is renamed `resumeLaunchInputsVersion: 1`. With it,
`resume` carries `credential` as before and also `configuredEnvironment`
(same builder as `submit`) when the provider advertises
`configuredEnvironmentVersion: 1`. Session handoff is unchanged.

**Reason and benefit**
A provider that restarts the Runner on resume gets both launch inputs it
needs.

**Breaking changes**
The capability name changes from `resumeCredentialVersion` to
`resumeLaunchInputsVersion`. No provider advertises the old name yet (it
merged in #21335 and has no adopter), so nothing changes behavior.

## What Changed

- Rename `resumeCredentialVersion` → `resumeLaunchInputsVersion` in the
SDK `describe` result schema, executor plumbing, owner input, docs and
tests.
- With the capability, `resume` also adds `configuredEnvironment` when
the provider advertises `configuredEnvironmentVersion: 1`. A normal
resume does not mark the variables as delivered, since a live Runner
keeps the environment it started with.
- Docs: the "Release and resume" section describes both fields; the
"Configured environment" section points to it.
- The runner-e2e SDK public-input pin is updated.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts`: 68 passed.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts`: 153 passed, including a normal
resume with the capability (credential sent, variables added only with
`configuredEnvironmentVersion: 1`, delivered flag unset) and redaction
of both fields from metadata writes and sanitized errors.
- `native-session-executor.test.ts -t "uses a task-only"`: 5 passed.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts`: 46 passed.
- `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- A provider restarting the Runner gets the claimant run's current
variables, which can differ from the ones the parked Runner started
with. That is intended.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:46:37 -08:00
DottaandClaude Opus 5.5 3b03ac10e5 fix(retry): answer a refused Retry with its refusal instead of accepted (#21497)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person presses Retry on a stopped task, and the Retry API answers
before dispatch runs
> - The route answered 202 "accepted" when it saved the request.
Admission ran afterwards
> - When admission refused the request for a reason another click cannot
change, the person saw "accepted", no run started, and the task showed
no Retry
> - A Retry that is shown must work, or the answer must say clearly why
it did not start
> - This pull request admits a plain Retry before the route answers
(bounded wait) and answers a refusal as a refusal
> - The benefit is that the API never reports "accepted" for a Retry
that admission refused, and the UI shows the reason

## Linked Issues or Issue Description

Refs #21481 (a fix for one cause of the refusal).

**What happened?**
On a hosted staging instance, two tasks got HTTP 202 `accepted: true`
from `POST /api/issues/:id/retry`. No run was created. The tasks then
showed a failed state with no Retry button and the wait reason
`previous_execution_unverified`.

**Expected behavior**
The Retry answer either reports a Retry that admission took, or it names
the refusal so that the UI can show it. It never reports "accepted" for
a request that admission refused.

**Steps to reproduce**
1. Stop a task's run so that its outcome is unknown (for example, a
recorded provider event in a stopped preparation).
2. Press Retry.
3. See HTTP 202 `accepted: true`, then see that the saved request waits
on `previous_execution_unverified` and no run starts.

**Paperclip version or commit**
`master` at `a48eaaef57`.

## What Changed

- `POST /api/issues/:id/retry` runs admission of a plain Retry (not a
saved-result finalization, not a connected chat retry) before it
answers. It waits at most 5 seconds. If admission takes longer,
admission continues after the answer, and the answer stays "accepted" as
before.
- When admission refused the request for a reason another click cannot
change (`heldRetryOffersRetry` is false), the answer is HTTP 200 with
`accepted: false` and `refused: { reason, message }`. The message is the
same stable text that the task view shows.
- The saved request is kept. If its gate clears later (for example,
saved-work recovery verifies the files), it still starts by itself.
- New `taskRetryRefusal` and `taskRetryWaitMessage` in
`server/src/services/task-interruption-presentation.ts`.
- UI: `taskRetryNotStartedReason` shows `refused.message`; the API type
includes `refused`.
- Tests that asserted 202 for a held Retry now assert the refusal
answer.

## Verification

- Red without the route change, green with it: `cd server && npx vitest
run src/services/legacy-preparation-retry-hold.test.ts` (new test: a
stopped preparation with a recorded provider event; the route answers
200 with `refused.reason: previous_execution_unverified`, and the saved
request stays deferred).
- Red without the hook change, green with it: `cd ui && npx vitest run
src/hooks/useTaskRetry.test.tsx`.
- Updated and passing:
`src/services/workspace-recovery-preparation-gates.test.ts` (48),
`src/services/workspace-recovery-preparation-proof.test.ts` (53),
`src/services/workspace-recovery-preparation-dispatch.test.ts` (25).
- Passing unchanged:
`src/__tests__/issue-scheduled-retry-routes.test.ts` (34),
`src/services/unrecoverable-saved-work-retry.test.ts` (5),
`src/services/workspace-recovery-warm-handoff.test.ts` (44),
`src/__tests__/heartbeat-native-restart-suspension.test.ts` (115),
`src/__tests__/issue-agent-mutation-ownership-routes.test.ts` (229),
`src/__tests__/agent-cross-tenant-authz-routes.test.ts` (18).
- Not run locally: `src/__tests__/chat-channels.integration.test.ts`
(did not finish on the loaded machine). CI covers it.

## Risks

- The Retry answer can take up to 5 seconds longer when admission is
slow (for example, when the agent's start lock is held). Before, the
route answered at once and admitted in the background.
- A refused request can still start later by itself when its gate
clears. The answer then said "not started", but the task view updates
when the run starts.
- API change: a refused Retry is now HTTP 200 with `accepted: false` and
`refused`. The board UI handles it. Other clients that only check
`accepted` see a refusal, which is correct.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:45:03 -06:00
4cf04d64cb fix(native): retry a transient endpoint failure after a same-run bootstrap restart (#21488)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts a Runner in a remote sandbox for each
agent run. A bounded same-run retry (at most 3 startup attempts) handles
transient startup failures.
> - A server restart can stop a run during its first Runner bootstrap.
Restart recovery then claims `bootstrap_incomplete`. It stops and
quarantines the unfinished bootstrap and launches a fresh bootstrap on
the same run.
> - The executor builds its "unclaimed bootstrap" proof only when
`restartRecovery` is not set. Thus the fresh bootstrap of a
`bootstrap_incomplete` restart can never prove that a transient failure
is safe to retry.
> - A transient HTTP 502 from the Runner endpoint request in that fresh
bootstrap was then classified as `ambiguous_state`. The run stopped and
needed manual board recovery.
> - This pull request gives the unclaimed-bootstrap proof to a
`bootstrap_incomplete` restart as well as to a first attempt.
> - The benefit is that a transient endpoint failure after a restart
uses the bounded automatic retry and does not need a person to recover
the run.

## Linked Issues or Issue Description

Refs #20414 (related: it fixes executor admission of a
`bootstrap_incomplete` restart at state-root migration. It does not
change the retry evidence that this PR changes.)

**What happened?**
A server restart interrupted the first Runner bootstrap of a remote
native run. Restart recovery claimed `bootstrap_incomplete` and launched
a fresh bootstrap on the same run. The fresh bootstrap registered its
controller checkpoint. Then `runner.ingress.acquire` failed with HTTP
502, before any Runner connection or provider event. The run ended with
`nativePhase: terminal_failure`, `recoveryOwner: board`, attempt 1 of 3,
and the next action "resolve the ambiguous session state". The error was
`Runner connection endpoint could not be acquired (HTTP 502).`
(`runner_ingress_acquisition_failed`).

**What did you expect to happen?**
The transient 502 occurred before the Runner connected and before any
provider work. It should use the bounded same-run startup retry, as it
does on a first attempt.

**Steps to reproduce**
1. Start a native run on a remote sandbox target.
2. Restart the server during the first Runner bootstrap, so that restart
recovery claims `bootstrap_incomplete`.
3. Make the next `runner.ingress.acquire` call return HTTP 502.
4. See that the run stops as ambiguous state and does not schedule a
startup retry.

## What Changed

- Add `attemptLaunchesFreshBootstrap()` to `native-bootstrap-proof.ts`.
It returns true for a first attempt and for a `bootstrap_incomplete`
restart. It returns false for `reattach_existing_runner`,
`reattach_remote_runner` and `resume_dead_runner`, because these
continue an existing Runner or provider session.
- In `native-session-executor.ts`, use this helper in place of
`!input.restartRecovery` when the executor builds the
unclaimed-bootstrap evidence for a startup failure.
- Add `native-bootstrap-proof.test.ts` for the helper.
- Add a case to `native-startup-retry.integration.test.ts`. It runs the
real executor under a `bootstrap_incomplete` restart claim, and the
fresh bootstrap gets a typed HTTP 502 from the Runner endpoint before
the Runner connects.

## Verification

- `native-bootstrap-proof.test.ts`: the test "accepts a same-run restart
of an incomplete bootstrap" failed before the fix and passes after the
fix (5/5 tests pass, vitest 5.1.0).
- `native-startup-retry.integration.test.ts`, case "transient 502 after
a bootstrap_incomplete restart": the run must settle as
`retryable_failure` with `recoveryMode: "bootstrap_retry"`, attempt 1
and a set `nextAttemptAt`. Without the executor change, the executor
builds no bootstrap evidence and the run settles as `ambiguous_state`.
CI runs this suite; I could not run it locally.
- The other guards for a retry did not change. The checkpoint must be
unclaimed, there must be no provider events and no session checkpoint,
and the 3-attempt startup budget applies. If the earlier attempt opened
a provider session, provider events exist and the run still fails closed
as ambiguous.
- I could not run the full typecheck and the real-Runner integration
suites locally (no Rust toolchain in the sandbox). CI must run them.

## Risks

- Low risk. The change only affects a `bootstrap_incomplete` restart
whose fresh bootstrap fails during startup. Such a failure now gets the
same retry decision as a first attempt. All the other evidence checks
still apply, so a run with provider activity still stops for manual
recovery.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with
tool use (shell, file edit, and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:37:24 -08:00
DottaandClaude Opus 5.5 532e502791 fix(heartbeat): drain native resume dispatches whose timer already fired (#21479)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat service runs agent work and resumes interrupted native
sessions on timers
> - `drainActiveRunExecutions()` lets a caller that tears down shared
state (a test cleanup) wait until no background run work is in flight
> - The drain cleared resume timers that had not fired yet, but it could
not see a timer that had already fired and was still reading and
claiming run state
> - That untracked work raced the `fresh-continuation-fallback.test.ts`
cleanup TRUNCATE and deadlocked it, which failed Cloud readiness and
held releases on 2026-11-13
> - This pull request makes the drain wait for fired resume dispatches
too, and removes the deadlock retry from that suite's cleanup
> - The benefit is a deterministic cleanup: nothing still writes run
tables when the drain returns, so the release gate no longer fails on
this test

## Linked Issues or Issue Description

No public issue exists. The bug is described here.

**What happened?**
Cloud readiness `verify / General tests (server (5/10))` failed on
master commits 18fb6ee571 and 5ede64f247. The `afterEach` TRUNCATE in
`fresh-continuation-fallback.test.ts` failed with `deadlock detected`
(40P01). Another backend held a lock on one run table and was waiting on
a table that the TRUNCATE had already locked. #21375 added a drain and
wrapped the TRUNCATE in `truncateTablesWithDeadlockRetry`. The retry
hides the race but does not remove it.

**Root cause**
`scheduleNativeSessionResumeDispatch` arms a timer. When it fires, its
async body calls `getSchedulingSuppression`, then
`recoverNativeRunsAfterRestart` or `dispatchNativeSessionResumptions`.
All of these read and claim run rows before any execution is registered.
Recovery arms an immediate timer (`new Date()`).
`drainActiveRunExecutions()` only cleared timers that had not fired. It
did not track the fired body, so it returned while that work still ran.
A failed dispatch could also re-arm a 30-second timer after the drain
had cleared the timers.

**Expected behavior**
When `drainActiveRunExecutions()` resolves, no heartbeat background work
writes to the database.

**Steps to reproduce**
The new test `heartbeat-drain-native-resume-dispatch.test.ts` holds an
ACCESS EXCLUSIVE lock on `instance_settings` and `heartbeat_runs`. It
arms an immediate resume dispatch and waits until a backend is blocked
behind that lock, which proves the timer fired. Then it starts the
drain. On the old code the drain resolves at once. The leaked query then
also outlives the test and fails at teardown with `CONNECTION_ENDED`.

## What Changed

- `heartbeat.ts`: the fired resume-dispatch body is tracked in a
module-level set, `activeNativeResumeDispatches`, shared by all service
instances like the other drain sets. `drainActiveRunExecutions()` awaits
that set. It still clears only the timers armed before it started. A
timer armed during the drain that is already due (for example the
immediate dispatch a run arms when its executor detaches) counts as
pending work: the drain yields so it fires, then drains what it starts.
Timers due later keep their deadline. A first version cleared timers on
every pass, and CI showed that this cancelled the self-detach takeover
in `heartbeat-native-restart-suspension.test.ts`.
- `heartbeatService` exposes `scheduleNativeSessionResumeDispatch` so
the regression test can drive it. Recovery already uses it.
- `fresh-continuation-fallback.test.ts` uses a plain TRUNCATE again. A
future leak fails the test instead of being retried away.
- New test `heartbeat-drain-native-resume-dispatch.test.ts`, described
above.

## Verification

- Red: the new drain test fails on the old `heartbeat.ts` (`expected
true to be false`: the drain returned while the dispatch was blocked).
- Green: the new drain test passes with the fix.
- Repeated runs with the fix and the plain TRUNCATE (no retry): 50/50
for "runs a person's Retry pressed before delivery…", 50/50 for "…old
lease is pending_cleanup", and 80 runs of "…old lease is active". No
deadlock occurred.
- In the loaded local runs, 11 of the 80 "active" repeats hit the 30 s
hook timeout. I logged `pg_stat_activity` every 5 s during cleanup. In
each case the drain had finished within 5.6 s. The TRUNCATE was `active`
with no lock wait and no other open backend: it was slow cascade I/O on
a host at load average 30 to 60. These were not leaks.
- `heartbeat-native-restart-suspension.test.ts`, the drain test and
`fresh-continuation-fallback.test.ts` together: 188/188 locally.
- `npx tsc --noEmit -p server`: no errors.

## Risks

- The drain now also waits for fired resume dispatches. These dispatches
are short. A slow one makes a test cleanup wait longer instead of racing
it.
- Production behavior does not change. Only callers of
`drainActiveRunExecutions()` see the difference.
- Follow-up, not in this PR: four other heartbeat suites still use
`truncateTablesWithDeadlockRetry`. They may no longer need it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:36:03 -06:00
DottaandClaude Opus 5.5 342fb308bd perf(run-events): commit streaming text deltas in batches (#21460)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - Text deltas (`item.delta`) are about 85% of that process's SQL
statements: each delta is one transaction of four statements on the
shared 10-connection pool
> - At 50 agents and 20 deltas per second, the process falls behind and
unrelated agent requests wait for the pool
> - This pull request commits text deltas in batches: one transaction
for all deltas that arrive within 250 ms, with one multi-row insert
> - The benefit is about 7 times fewer transactions and 5–6 times fewer
statements per delta, with no delay for the first text of an item

## Linked Issues or Issue Description

Refs #21435 (this pull request is stacked on it). Refs #21131. Item 1a
of the agent scaling plan in #21418.

**What existing behavior does this improve?**
The persistence of native text deltas. `native-session-runtime` awaited
`PaperclipControlPlanePort.appendEvent` for each event, and the port ran
one transaction per event.

**Subsystem affected**
Server run event ingest (`paperclip-control-plane-port.ts`,
`heartbeat-run-events.ts`) and the runner package's turn consumer
(`native-session-runtime.ts`).

**Current behavior**
Each delta takes one transaction: BEGIN, run lock, append, COMMIT. In
the load model with 50 agents (deltas only), that is about 4.0
statements and 1.0 transaction per delta.

**Proposed behavior**
The turn consumer starts the next append without waiting for a delta's
receipt. The port collects deltas and commits them in one transaction
with one insert. In the same load model, that is about 0.7 statements
and 0.14 transactions per delta.

**Reason and benefit**
Fewer transactions free the shared pool and the main thread for other
agents' requests. This is the largest per-agent cost in the scaling
plan.

**Breaking changes**
None. The row shape does not change. A port without the new capability
keeps the earlier serial behavior.

## What Changed

- `appendHeartbeatRunEventBatch` (`heartbeat-run-events.ts`) appends
native source events of one run and one source in one transaction. When
the events continue the stored source cursor (from #21435), it sends
four statements for the whole batch: the run lock, one replay probe, one
`insert ... select from jsonb_array_elements(...)` of every row, and the
commit. Any other batch (a replay, a gap, an event with a wake, or a run
without a cursor) appends its events one by one in the same transaction,
with the single-append checks.
- **Storage choice: one row per delta, written by one statement per
batch.** I did not store one row per batch. The transcript, live events,
run log export, chat progress, replay (`replayEvents`), recovery cursors
and the UI all read one row per delta. One statement per batch keeps
every reader unchanged, and rows written before this change read the
same.
- `PaperclipControlPlanePort` sets `pipelinesStreamingEvents`. An
`item.delta` appended with the new `streaming` option goes into a queue:
- It commits at once until a delta with text commits after the latest
other event. So the first text of each item, and the time to first
token, never wait for the timer.
- Later deltas commit after at most 250 ms, or at once when the batch
has 100 deltas.
- Every other event, `checkpointSession`, `completeRun`, `replayEvents`
and `provesNoRunProviderWork` first commit the queue. A non-streaming
event is never committed before the deltas that preceded it.
- A delta appended without `streaming` commits at once, so callers that
wait for each receipt are not slowed.
- The receipt of a delta settles only after its batch commits and its
observers (live event, executor callback) run, in order.
- If a batch fails, every delta in it rejects and nothing from it is
stored.
- `native-session-runtime` `consumeTurn` appends deltas with `streaming`
and does not wait for their receipts (at most 256 pending). Before any
other event, any result, and the governed-wait settlement, it calls
`flushStreamingEvents()` and then waits for all pending receipts, so a
boundary never waits for the timer. A failed delta stops the turn before
a later event is appended. The consumer joins every pending delta append
before it settles, also when the provider stream fails.
- If one observer of a committed batch fails, only that delta's receipt
rejects. The other rows are still published and observed.
- `ControlPlanePort` contract: optional `pipelinesStreamingEvents`,
`flushStreamingEvents()` and `AppendControlPlaneEventOptions.streaming`.

## Verification

- Red/green. The new tests failed before the change and pass after it:
- Runner: "starts text delta appends without waiting and flushes them
before any other event" (`native-session-runtime.test.ts`). The port in
the test has no timer, so the turn finishes only if the boundary
flushes. Before: the appends were serial, and later the boundary waited
for the timer.
- Runner: "joins pending delta appends before a failed turn settles".
Before: the turn settled while a delta commit still ran.
- Port: "observes every committed delta of a batch when one observer
fails".
- Port: "commits the first text delta at once and later deltas in one
batch before the next other event". It checks one transaction and one
insert for five deltas, flush before a non-streaming event, receipts
only after commit, and immediate commit of the first delta with text.
Before: each delta committed alone.
- Port: "flushes queued deltas on the timer and before a replay read,
and rejects every delta of a failed batch". It checks the
crash-before-commit case: a revoked writer fails the batch, nothing is
stored, and the resent deltas commit idempotently (the stored one is a
duplicate).
- Store: four new tests in `heartbeat-run-event-atomic-append.test.ts`:
one lock, one probe and one insert per batch; replay idempotency and
payload conflict; a batch that fails before commit stores nothing and
its resend commits; the one-by-one path for no cursor and for gaps.
- Focused suites pass: `heartbeat-run-event-atomic-append`,
`heartbeat-run-event-sequencing`, `paperclip-control-plane-port`
(including the package conformance suite),
`paperclip-semantic-conformance`, `native-observer-replay`,
`native-event-boundary-golden`, `live-recovery-work`,
`native-interaction-bridge`, `chat-native-progress-work`, and
`native-session-runtime.test.ts` in the runner package.
- `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript`
and `pnpm --filter @paperclipai/server run typecheck` pass. `pnpm
check:module-boundaries` passes.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418,
`AGENTS=50 SECONDS=20`, local embedded PostgreSQL). The load model calls
the store directly, so for this run I added a local `BATCH_MS` mode that
applies the port's batching rules (first delta at once, 250 ms timer,
100 cap, flush before other events) and does not wait for delta
receipts. That mode is not in this pull request. The machine was very
busy (load average 40–60), so absolute figures are noisy; per-delta
ratios are the stable result:

| Deltas only | Statements/s | Transactions/s | Deltas done of offered |
Statements per delta | Transactions per delta | Lateness p50 / p99 |
  | --- | --- | --- | --- | --- | --- | --- |
| One transaction per delta | 2,920 | 730 | 14,668 of 20,000 | 3.98 |
1.00 | 3,074 / 5,384 ms |
| Batched (250 ms) | 704 | 141 | 19,940 of 20,000 | 0.71 | 0.14 | 668 /
2,858 ms |
| One transaction per delta (repeat) | 1,712 | 428 | 8,575 of 20,000 |
3.99 | 1.00 | 6,558 / 11,464 ms |
| Batched (250 ms, repeat) | 856 | 172 | 20,000 of 20,000 | 0.86 | 0.17
| 187 / 556 ms |

With all sources on, the machine was saturated (event-loop utilization
0.96–0.98) in both modes and the results were not usable (764 and 12,800
of 20,000 deltas done).

## Risks

- **Ack and crash (read first).** In the live runnerd path the runner's
ack does not wait for the `heartbeat_run_events` row today.
`DurablePrpControlPlane.#event` acks after the in-memory journal commit.
Streaming events are not in the controller checkpoint. So a delta that
is acked but not yet appended is lost if the server crashes, and the
runner does not resend it. This is the earlier behavior. This pull
request makes that window up to 250 ms longer. The final text still
survives through `item.completed`, which commits immediately and in
order. This pull request does not change `durable-prp-control-plane.ts`.
- Live text in the UI arrives in steps of up to 250 ms after the first
text of an item.
- A delta failure now surfaces at the next non-delta event or at the end
of the turn, not at the delta. The turn stops before it appends a later
event.
- The governed-wait drain and the control-event replay still append one
event at a time. They do not carry deltas.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:34:01 -06:00
Devin FoleyandPaperclip 12aa08721c feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.

Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on d634944df1.
Greptile Apex reviewed that revision at 5/5 with no unresolved threads.
External neutral security reports remain documented in the pull request.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 13:20:13 -08:00
Devin FoleyandClaude Opus 5.5 2756be2f32 fix(models): answer an unreachable OpenRouter catalog with 503 instead of a crash report (#21468)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent setup lists the models that an OpenRouter key can select. The
server gets this list from the public OpenRouter catalog.
> - The catalog request has its own 10 second deadline. When the
deadline expired, fetch rejected with a raw `TimeoutError` (a
`DOMException`).
> - The error handler answered 500 "Internal server error" and sent a
crash report with no first-party stack frames. The user did not see the
existing retry guidance.
> - A slow or unavailable third-party catalog is not a fault in the
Paperclip server. It must not create a crash report.
> - This pull request maps catalog outages to a 503 response with the
existing retry message and a stable code. Only those errors skip crash
reporting.
> - The benefit: users see "Retry or enter a model ID manually", and
Sentry does not get noise from a provider outage.

## Linked Issues or Issue Description

**What happened?**

A staging server answered `GET
/api/companies/:companyId/adapters/opencode_local/models` with `500 —
The operation was aborted due to timeout`. At the same time, other
outbound requests from the host were slow. Sentry recorded
`TimeoutError: The operation was aborted due to timeout`. The event had
only Node internal frames (`AbortSignal.timeout`).

**Expected behavior**

The route returns a clear "catalog unavailable" response. The UI shows
the retry guidance. A third-party outage does not create a crash report.

**Steps to reproduce**

1. Make `https://openrouter.ai/api/v1/models` slow (no answer for more
than 10 seconds), or make it unreachable.
2. Request the OpenCode model list with `provider=openrouter`.
3. See a 500 response and a Sentry `TimeoutError` event.

**Paperclip version or commit**

`e23549cd9c631e4595a3213e03d0c960360452c6`

I searched for duplicate PRs. #17083, #17961 and #10290 change the
OpenRouter catalog contents or model validation. They do not change how
the server handles a catalog timeout.

## What Changed

- `server/src/services/openrouter-models.ts`: The catalog now maps three
failure types to `HttpError(503)` with code
`openrouter_catalog_unavailable` and the existing retry message: its own
deadline, a classified transport failure (uses
`classifyRemoteConnectionError`), and a 408, 429 or 5xx status. A
module-local `WeakSet` records only the errors that it creates for these
outages. `isOpenRouterCatalogUnavailable()` reads that record.
- Other non-OK statuses become a 502 with code
`openrouter_catalog_error`. These are still reported. Malformed JSON and
an invalid catalog shape are also still reported.
- `server/src/middleware/error-handler.ts`: The handler does not send a
crash report for an error from `isOpenRouterCatalogUnavailable()`. It
still answers 503 with the message and code. The new check is next to
the existing MCP connection-failure exclusion.
- Tests: unit tests for the deadline, DNS, 429 and 5xx cases, and for
the failures that stay reportable. A route-level test sends a stalled
catalog request through the real error handler. It checks for a 503
response and no crash report. It also checks that a different 503 is
still reported.

## Verification

- `vitest run server/src/services/openrouter-models.test.ts`: 7 tests
pass. I ran this file alone because the full workspace install did not
fit in the agent sandbox disk.
- Red check: with the original `openrouter-models.ts`, 5 of the 7 tests
fail. The deadline test receives `DOMException TimeoutError`, which is
the same error as the production event.
- `server/src/__tests__/openrouter-catalog-error-handler.test.ts` needs
the full workspace dependencies. CI runs it.

## Risks

- Low risk. Successful responses do not change. The cache and in-flight
deduplication do not change.
- The failure status changes from 500 to 503 (or 502) for the three
routes that read this catalog: adapter models, fast-response models and
prompt-injection-prevention models. The UI already shows
`error.message`, so users now see the retry guidance in place of
"Internal server error".
- The crash-report exclusion is narrow. It applies only to error objects
that this module created after an actual outage. A copied message or
code does not match.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code
agent harness with tool use (shell, file edits, Sentry, Railway and
GitHub connectors).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:00:24 -08:00
DottaandClaude Fable 5.2 1a96247480 fix(recovery): continue a failed continuation again instead of escalating it to the board (#21394)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Automatic task recovery continues a task when its run ends for a
reason that is not a person's to fix
> - A continuation or recovery run can itself fail before it does any
work: a sandbox create timeout, a full sandbox disk, a lease mismatch, a
provider pack check, or a database deadlock
> - Today the stranded reconciler treats such a failure as non-retryable
and blocks the task for the board. The assigned-work sweep then leaves
the board hold alone, because its list of temporary causes is a short
allow-list. The held backlog pass uses a similar allow-list. Nothing
continues the task, and it shows "failed, Retry"
> - This pull request makes the cause test "not owned by a person" on
these paths, and routes a failed continuation run back through the
fresh-continuation fallback
> - The benefit is that such a task continues from its saved work, under
the existing hourly cap, and nobody needs to press Retry

## Linked Issues or Issue Description

**What happened?**
A task was continued automatically after an interruption. The
continuation run failed during setup, before any provider turn (for
example `setup_failed` with a sandbox HTTP timeout). The stranded
reconciler then set the task to `blocked` with a board-owned recovery
hold. The assigned-work sweep left the task to that hold, and the
fresh-continuation fallback never owns that hold kind. The task stayed
blocked and showed a Retry button. Settled holds whose run was a
recovery run that failed with `adapter_failed`, `setup_failed` or a
provider pack error stayed in the same state, because the held backlog
pass accepted only restart-class codes.

**Expected behavior**
A run that ends for a reason a person does not own is continued again
from saved work. The continuation goes through the fresh-continuation
fallback, so the old workspace epoch is fenced or a new isolated
workspace is used. The per-task limit of 6 automatic continuations per
hour still applies. At the limit the task waits for the next window and
is not blocked.

**Steps to reproduce**
1. Record a fresh continuation for an interrupted run of an
agent-assigned task.
2. Let the continuation run fail with `setup_failed` before provider
work.
3. Run the stranded reconciler. The task becomes `blocked` with a board
hold.
4. Run the assigned-work sweep. It returns `left_to_hold_owner`. No
continuation follows.

**Paperclip version or commit**
master at 6917d0568d

**Deployment mode**
Hosted (two servers can run at once during a deploy)

## What Changed

- New `continuation-cause.ts` holds the cause classification:
`PERSON_OWNED_FAILURE_CODES` (moved, unchanged), a new short list
`PERMANENT_FAILURE_CODES` (low-trust boundary codes,
`adapter_engine_unavailable`, `agent_not_invokable`, `agent_not_found`,
`agent_interaction_delivery_denied`), `isSystemOwnedFailureCode`, and
`isAutomaticContinuationRun` (a retry, scheduled retry, fresh
continuation, sweep or recovery run; never a person's Retry).
`fresh-continuation-fallback.ts` re-exports what callers use.
- `stranded-hold-continuation.ts`: the sweep continues a recent (24
hours) board hold when the failure cause is system-owned, not only for a
short allow-list. A reusable sandbox resume failure still needs a
transient cause.
- `recovery/service.ts`: the stranded reconciler does not escalate an
agent-assigned `todo` or `in_progress` task to the board when its latest
run was an automatic continuation or recovery run that ended within 24
hours with a system-owned cause. The assigned-work sweep owns that task.
A person's Retry, a person-owned cause, a permanent cause, or an older
failure escalates as before.
- `assigned-work-sweep.ts`: for a failed automatic continuation run, the
sweep asks the fresh-continuation fallback first. A recorded
continuation is the repair (fenced epoch or new isolated workspace). On
the fallback's hourly backoff the sweep records "not started" with the
retry time and keeps the task unblocked. While exact recovery still
waits, it records the retry time. A person-owned reason from the
fallback is written on a refused attempt.
- `task-automatic-attempts.ts`: the shared 6-per-hour ledger also counts
`server_restart` and `interrupted_preparation` scheduled retries, so the
sweep cannot add attempts on top of those lanes in the same hour.
- `fresh-continuation-fallback.ts`: the held backlog pass also accepts a
source run that was an automatic continuation or recovery run with a
system-owned code (24 hours, 5 per pass, 10 per minute unchanged). With
no hold to carry a pending check, a wait is measured from the source
run's end, so it is bounded.
- `fresh-continuation-fallback.ts`: rechecks the cause with the shared
classifier and applies the shared hourly cap inside the task lock. A
continuation keeps the old writer's workspace unless a wait was actually
cut short (bound measured from a recorded check, or a wait skipped).
- New test file `failed-continuation-continues-again.test.ts` (15
tests).

### Changed test expectations

- `interrupted-preparation-hold-settle.test.ts`, "bounds repeated lost
leases before provider work" (both cases): the lane cap still holds (2
and 12 successors). The end state is no longer `blocked`: the task keeps
its status, no further run exists, and the sweep returns a deferral with
a future retry time (`deferred_cap` at 12 ledger attempts,
`deferred_rate_limit` spacing at 2). The fixture now settles the
scheduled wakes and retry times the loop leaves behind, so the sweep
reads the real state. A person's Retry is still accepted.
- Same file, "blocks the task when a replaced preparation's lane is
spent" is renamed "leaves the task to the assigned-work sweep, not
blocked, when a replaced preparation's lane is spent". It still expects
no successor run, and now expects `todo` with no active hold.
- `heartbeat-process-recovery.test.ts` and `task-retry.test.ts`: no
change. Their runs either ended months before (outside the 24-hour
window) or were a person's Retry, so they escalate as before. The bound,
no-duplicate, and redaction tests keep their assertions.

## The one classifier

`continuation-cause.ts` `continuesAutomatically(run)` (and its SQL twin
`continuesAutomaticallySql`) decides whether a failed run continues on
its own:

- the cause is system-owned: not in `PERSON_OWNED_FAILURE_CODES`
(unchanged), not in the new `PERMANENT_FAILURE_CODES`, not an external
execution whose stop cannot be confirmed (Dot and Muse, the same rule
the retry scheduler uses to decline them permanently), and a reusable
sandbox resume failure only with a transient cause; and
- the run was an automatic continuation or recovery run (a scheduled
retry, a recovery wake, a fresh continuation; never a person's Retry),
or a first run that ended with a restart-class code or a transient setup
failure before provider work (the pre-existing allow-list for the
backlog and the board-hold takeover).

A person's failed Retry keeps master's outcome at every entry point: the
classifier never treats it as an automatic run. Whether it should
continue on its own is a separate product decision.

## Every entry point and the cap it uses

Shared cap = `sharedContinuationAttempts` in
`recovery/continuation-ledger.ts`: 6 per task per rolling hour, counting
fresh continuation records, settled sweep wakes, and transient, restart
and preparation retries. Only attempts that started or created a run
count.

- Disposition pass (`settleUnrecoverableExecutions` to
`continueFreshInTransaction`): cause test of the classifier and the
shared cap, inside the fallback under the task lock. Run kind is not
required on this path: it handles fresh failures of any run.
- Held backlog (`continueHeldExecutions`): `continuesAutomaticallySql`
in the listing; then the fallback's cause test and shared cap. Pacing 5
per pass, 10 per minute, 24 hours.
- System continuation wake in the queue (`heartbeat/queue.ts`,
unchanged) to the fallback: cause test and shared cap.
- Dispatch refusal (`run-dispatch` Postgres adapter, unchanged) to the
fallback: cause test and shared cap.
- Sweep board-hold takeover (`strandedHoldContinuesAutomatically`):
`continuesAutomatically`, 24 hours.
- Sweep route to the fallback (trigger `assigned_work`): recovery run
with a system-owned cause, then the fallback's cause test and shared
cap; the sweep claim also applies its ledger.
- Old reconciler deferral (`assignedWorkSweepOwnsFailedRun`): automatic
run with a system-owned cause (`canContinueRecoveryRunAutomatically`),
24 hours. First runs keep their existing escalation and budgets.
- Sweep plain wake (unchanged path, for first runs): the sweep claim
ledger (`taskAutomaticAttempts`: started sweep attempts, scheduler
retries, and fresh records from other paths), spacing from every
attempt.

## Self-review: places where the rule still differs, on purpose

- The fallback's fresh path continues a first run with any system-owned
cause, while the backlog limits first runs to restart-class and
transient setup codes. The backlog limit is the existing herd rule for
records older releases left behind.
- The fallback treats a pending question as a person's wait; the sweep
treats a question asked before a person's Retry as superseded. In that
one shape the sweep sends its ordinary wake, as before.
- The scheduler lanes (transient, restart, preparation) keep their own
bounds and do not read the shared cap before scheduling; their runs
count in it, so every other path stops once they fill the hour. Changing
the scheduler is outside these files.
- The sweep's plain wake path does not test the cause; it relies on the
invariant's waiting reasons, as before.

## Waits and their bounds

- Every exact-recovery wait in the fallback (settling execution, native
finalizer, live controller lease, preparation lane, saved-work recovery,
workspace restore owner, retained-source check, previous local process):
it starts at the first recorded pending check on the run's hold. When no
check is recorded (no hold, a hold nobody checks, or a replaced record),
it starts at the source run's end (`finished_at`, a database row that
every server reads the same). Bound: 10 minutes idle, 15 minutes while
the owner shows progress, 30 minutes while saved-work recovery advances.
Then the fallback continues from saved work.
- A source run that ended more than 24 hours ago with no recorded check
does not wait: the fallback answers `retry_needed` (the existing named
state that leaves an old hold for a person's Retry), so nothing waits
silently and the 24-hour rule holds.
- The sweep on a fallback backoff: waits until the oldest continuation
leaves the hour (6 per task per hour), task stays unblocked.
- The sweep while exact recovery waits: rechecks at the fallback's retry
time (at most 4 minutes), until the bound above.

## Decisions on permanent causes

- `PERSON_OWNED_FAILURE_CODES` is not changed. A separate
`PERMANENT_FAILURE_CODES` list keeps causes that cannot heal by
themselves out of the new automatic paths, so they still block and tell
a person. `repository_catalog_unavailable` was transient in the old
allow-list too; in the failing Retry test it was a person's Retry, which
the reconciler still escalates.

## Verification

- `failed-continuation-continues-again.test.ts`: 28 tests, including a
12-case table (person's Retry, automatic continuation, first assignment;
a person's Retry gets master's outcome everywhere) x (setup failure,
restart code, permanent code, person-owned code) through the backlog,
the sweep takeover and route, and the reconciler, and a test that
waiting checks spend no attempt (red before the fix for the first-run
case and for the cap); earlier: 15 tests (the latest three added
red-first: permanent codes and a person's Retry stay blocked on the
backlog; no entry point continues a person-owned or permanent code; six
mixed attempts stop the backlog until the next window; plus a pin that
the ledger uses the restart lane's window, cap and reason). Earlier:
Before the fix, the 10 behavior tests failed for the expected reasons
(board escalation, `left_to_hold_owner`, no backlog continuation,
endless wait with no hold, endless wait on an unchecked hold at 20
minutes and 30 hours, checked alternately by two sweep instances). The
person-owned guard passed before and after. After: 11 of 11 pass. After
the last change, alone: `fresh-continuation-fallback.test.ts` 72/72,
`fresh-continuation-move-forward-offer.test.ts` 14/14,
`assigned-work-invariant.test.ts` 31/31,
`interrupted-preparation-hold-settle.test.ts` 29/29.
- Locally after the change, one file at a time:
`assigned-work-invariant.test.ts` 31/31,
`recovery/assigned-work-invariant.test.ts` 50/50,
`interrupted-preparation-hold-settle.test.ts` 29/29,
`task-retry.test.ts` 117/117, the 10 previously failing
`heartbeat-process-recovery.test.ts` tests 12/12 (filtered run),
`fresh-continuation-fallback.test.ts` 66/72 in a loaded run with 6
timeouts at 30 to 90 s that pass alone (8/8 filtered).
- `tsc --noEmit` for `server`: clean.
- CI supplies the full suite.

## Risks

- Behavior change: a recent failure with a cause outside the
person-owned list is no longer escalated to the board by the stranded
reconciler. The sweep continues it instead, at most 6 times per hour per
task, then once per window. Some codes outside that list may be
configuration a person must change (for example `low_trust_*`,
`adapter_engine_unavailable`, `agent_not_found`). Such tasks now retry
at the capped pace instead of blocking. The person-owned list is not
changed in this pull request.
- Upgrade: tasks blocked by an older release with a board hold for such
a cause, within 24 hours, are continued by the sweep on its first
passes. The sweep's 10 repairs per minute per process, the delivery
pacing, and the hourly cap bound the burst.
- Two servers: the sweep's claim and the fallback each run under the
task row lock, and the fallback is idempotent per source run.

## Model Used

- Claude Fable 5.2 (Anthropic), via Claude Code with tool use and code
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:49:16 -06:00
DottaandClaude Opus 5.5 b5adf35119 test(runner): guard that a relaunch after a lost launch acknowledgement leaves one Runner (#21477)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) that the
server launches in a sandbox.
> - A deploy can stop the server after the launch command reached the
sandbox but before the server read back the Runner's process identity (a
lost acknowledgement).
> - The database then has no PID for the run, so the successor server
launches the Runner again on the same state folder.
> - Two Runners must never serve the same run's state. #20852 proposed a
large launch-intent design for this window. Since then, #21079 added a
fence that stops a leftover Runner on the same state folder before each
launch.
> - This pull request adds a Linux regression test that drives the real
launcher through the lost-acknowledgement window.
> - The benefit is that a future change to the launch path cannot bring
back a second Runner without failing this test.

## Linked Issues or Issue Description

Refs #20852 (superseded: current master already prevents the duplicate
Runner through #21079). Refs #21079.

**What happened?**
#20852 claimed that a relaunch after a lost launch acknowledgement could
start a second Runner for the same run. Nothing proved whether current
master still has this problem.

**Expected behavior**
After a lost acknowledgement, the successor's relaunch stops the
orphaned Runner. Exactly one Runner serves the run's state folder, and
the process marker names it.

**Steps to reproduce**
1. Launch a remote Runner, and lose the launch RPC's acknowledgement
before the identity read (the old server stops).
2. Relaunch the same run on the same sandbox from a new launcher (the
successor).
3. Count the live `paperclip-runnerd` processes that serve the run's
`--state-dir`.

## What Changed

- New
`server/src/services/native-runtime/remote-runner-lost-launch-ack.test.ts`.
It is a test only and changes no product code.
- It runs only where `/proc` exists (Linux CI), like the existing #21079
fence tests.
- It drives the exported `createRemoteRunnerProcessLauncher` with a
local command runner. The old launcher's runner runs the launch command,
then throws `remote_launch_ack_lost`. After that, every command fails,
because the old server is gone.
- A small bash stand-in for `paperclip-runnerd` keeps the Runner's
argv[0] and `--state-dir`. Its `reap-lineage` reports a verified, empty
lineage.
- The test checks these results: the orphan was alive and had no PID on
the handle; the successor's Runner has a new PID; the orphan stops;
exactly one Runner serves the state folder; the marker names the
successor's PID.

## Verification

- Linux reproduction (Docker `node:24-bookworm-slim`). I used the exact
outer launch command and `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT`, which I
extracted from this head:
- master scripts: `runners after successor relaunch [61], orphan alive
false`. Result: one Runner (PASS).
- the same scripts without the #21079 stale-state fence
(counterfactual): `runners after successor relaunch [24, 56], orphan
alive true`. Result: duplicate Runner (FAIL).
- macOS: `npx vitest run
src/services/native-runtime/remote-runner-lost-launch-ack.test.ts`
reports the file as skipped (1 skipped), as intended, because there is
no `/proc`.
- The test runs for real only on Linux CI. CI on this head is the first
real run of the vitest file.

## Risks

- Low risk. The change is a test only.
- The test starts real local processes. Its teardown kills only the PIDs
it found on its own temporary state folder, and it never signals PID 0.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, Docker, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:46:24 -06:00
DottaandClaude Opus 5.5 4e2c149f2f test(decisions): let the concurrent admission test survive a legitimate serialization retry (#21416)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Decision models let features ask a company AI connection short
questions. Each decision reserves budget before it calls the provider.
> - The server test "admits concurrent work from the same budget
observation" fails or hangs intermittently in CI. This blocks Core image
publication from master.
> - The test assumed that both admission snapshots always survive. They
do not. A legitimate serialization retry can make one admission see the
other's committed hold.
> - This pull request changes the test so that it proves what the
product guarantees, with the same strength, and without the timing
assumption.
> - The benefit is a deterministic test and unblocked image publication.

## Linked Issues or Issue Description

Refs #21387

**What happened?**
The test "company decision service > admits concurrent work from the
same budget observation and records both obligations" in
`server/src/__tests__/decision-models.test.ts` hangs intermittently. CI
run 43451649374 (attempt 2, job "verify / General tests (server
(4/10))") failed with `Test timed out in 15000ms` at line 371. Before
#21387 the same race failed with `expected "vi.fn()" to be called 2
times, but got 1 times`.

**Root cause**
- `decide()` resolves the credential before admission.
`secrets.resolveSecretValue` writes `company_secrets.last_resolved_at`
and `updated_at` in its own statement
(`server/src/services/secrets.ts:1417-1421`).
- Admission runs in a REPEATABLE READ
`withOperationAccountingTransaction`
(`server/src/services/accounting-transaction.ts:31-53`). Its snapshot
starts before it waits on the test's lock.
- When decision B's `last_resolved_at` write commits after decision A's
snapshot, A's `SELECT ... FOR SHARE` on the same secret row
(`server/src/services/decision-models.ts:263-264`) raises 40001.
- The retry loop takes a fresh snapshot. If B has committed its 1c hold
by then, A sees 1c held against a 1c hard stop and correctly returns
`budget_blocked`. A never reaches the provider, so the test waits
forever.

**Expected behavior**
The test passes every time. It still fails if admission becomes a
company-wide writer mutex or if the hard stop stops blocking.

## What Changed

- Set the test budget to 2c with 1c reservations. Both decisions are now
admitted whether or not one admission retries.
- Kept the proof that both admissions are in flight at once (both wait
on the same row lock at the same time). This is the no-mutex guarantee.
- Kept the checks that both providers run concurrently, both holds are
`held` at 1c, both decisions succeed, and two cost events exist.
- Added a check that a third decision returns `budget_blocked` while
both holds are held.
- Renamed the test to "admits concurrent work without a company writer
mutex and records both obligations".
- No product code changed. The retry and the `budget_blocked` result
after the retry are correct hard-stop behavior.

## Verification

- Root cause reproduced with temporary instrumentation (not committed).
Starting decision B after decision A was parked gave 40001 on the
`company_secrets` FOR SHARE in 4/4 runs. With the retry delay forced to
300 ms, the old test failed with the exact CI message `expected
"vi.fn()" to be called 2 times, but got 1 times`, and the log showed
`admitted` then `budget_blocked`.
- The new test passes in that forced worst case (staggered starts plus a
300 ms retry delay): both admitted, the third `budget_blocked`.
- 50 repetitions (vitest `repeats: 49`, temporary) with 16 extra CPU
busy loops on an 18-core machine: 50/50 pass.
- Red check: changing the company read in
`withOperationAccountingTransaction` to `FOR NO KEY UPDATE` (a company
writer mutex) makes the test fail with `expected 1 to be 2`.
- `npx vitest run src/__tests__/decision-models.test.ts`: 37/37 in 2 of
3 runs. One run had a failure in a different test ("returns answers and
a committed charge before slow budget stop delivery"). That test passes
alone and is not changed here.

## Risks

- Low risk. Only one test changes.
- The test no longer proves that two admissions from the same snapshot
both pass a 1c hard stop. That overshoot is not a product guarantee: any
concurrent committed write to a share-locked row restarts the snapshot.
- Each concurrent decision on the same credential can force a sibling
admission to retry because of the `last_resolved_at` write. This is
extra latency, not a correctness problem. A follow-up can remove that
write from the decision hot path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:46:14 -06:00
DottaandClaude Opus 5.5 c1ea9304f7 fix(recovery): continue Runner preparations whose remote workspace was handed on unchanged (#21481)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery must continue a task automatically after a deploy, and
a shown Retry must start a run
> - A Runner (paperclip_runner) run can be stopped while it still
prepares its remote sandbox workspace, by a deploy or by a setup failure
> - When the run did not change the workspace, workspace settlement
hands the workspace on at its saved version and turns the run's
workspace reference to `saved`
> - The "no provider work" proof still asks for a workspace-repair
receipt for any run with a remote workspace reference. Only a copy-back
or an abandoned prepared epoch made that receipt. A run released
unchanged never got one
> - So the reconciliation hold never settled, every automatic
continuation was refused, and a person's Retry waited on
`previous_execution_unverified`
> - This pull request accepts the host's released-unchanged receipt for
the exact run, epoch and snapshot as a verified workspace repair
> - The benefit is that these preparations continue by themselves, and
Retry on them starts immediately

## Linked Issues or Issue Description

**What happened?**
On a hosted staging instance, a release swap stopped a Runner run during
preparation. No process started and no provider action happened. The
task showed "Automatic recovery stopped". The automatic continuation
reported "could not start yet" five times. A person then pressed Retry.
The API accepted the Retry, but no run started, and the task showed no
Retry button. A second task failed in setup ("RPC call
environmentExecute timed out") during preparation and had the same
result after Retry.

**Expected behavior**
A run that stopped before any process or provider action continues
automatically after a deploy. Retry on such a task starts a run
immediately.

**Steps to reproduce**
1. Use a `paperclip_runner` agent with a remote sandbox workspace.
2. Stop a run with a graceful shutdown (or a setup failure) while it is
in the `preparing` stage, after it restored the saved workspace version.
3. Let workspace settlement release the workspace unchanged
(`writerRunId: null`, `dirty: false`, an unchanged-baseline receipt for
the run, run reference state `saved`).
4. See that the ownership pass keeps the
`legacy_execution_requires_reconciliation` hold
(`provider_work_not_disproved`), and that a Retry stays deferred with
`previous_execution_unverified`.

**Paperclip version or commit**
Observed on staging at `b0e99dfcfa`; still present on `master` at
`37393f5dc5`.

**Root cause**
`isInterruptedNativePreparation` and `failedNativePreparation` in
`server/src/services/cancelled-native-startup.ts` require
`hasVerifiedWorkspaceRepair` for any run with
`runnerProfileJson.remoteWorkspace`. `hasVerifiedWorkspaceRepair` in
`server/src/services/workspace-recovery-receipt.ts` did not accept the
released-unchanged receipt that `releaseUnchangedRemoteWorkspaceWriter`
writes.

## What Changed

- Add `isReleasedUnchangedRemoteWorkspace` in
`server/src/services/remote-workspace-abandoned-preparation.ts`. It is
true only when the run's reference is `saved`, and the workspace's
unchanged-baseline receipt names this run, the same epoch and the same
snapshot, with a release time.
- `hasVerifiedWorkspaceRepair` accepts that receipt, next to the
copy-back receipt and the abandoned prepared epoch.
- Add regression tests for the deploy-interrupted shape (held
preparation settles and continues; Retry starts) and for the
setup-failure shape (Retry starts). Add a negative test: a preparation
that still holds its workspace unsaved keeps its hold.

## Verification

- Red without the fix, green with it:
- `cd server && npx vitest run
src/__tests__/interrupted-preparation-hold-settle.test.ts` (32 passed;
the 2 new positive tests fail without the fix)
- `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts` (6 passed; the new
test fails without the fix)
- Related files pass unchanged:
`src/services/saved-work-retry-eligibility.test.ts` (8),
`src/__tests__/interrupted-preparation-restore-retry.test.ts` (7).
- Focused test files only (the machine was loaded). CI supplies
typecheck, build and the full suite.

## Risks

- Low risk. The new proof applies only when the host itself released the
run as the unchanged writer of that exact epoch and snapshot. A run that
still holds unsaved files, a later writer that replaced the receipt, or
any provider execution evidence still keeps the hold.
- Not changed: the retry API still answers "accepted" for a saved Retry
intent that admission later refuses for another reason. That is a
separate follow-up.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:46:05 -06:00
DottaandClaude Opus 5.5 a48eaaef57 perf(server): skip the agent row lock for unchanged agent-file checkpoints and upload before the lock (#21457)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent keeps its own files (AGENT_HOME). A warm native session
saves these files at the end of each turn (an agent-file checkpoint)
> - Each checkpoint took the agent row lock (`agents ... FOR UPDATE`) on
the small file-storage pool, also when the run changed no file
> - A checkpoint that changed files also uploaded the new bytes to
object storage while it held that lock. Instruction edits and other runs
of the same agent waited for the upload
> - With many agents, these long transactions use the 2-connection
file-storage pool and slow every agent
> - This pull request removes the lock from an unchanged checkpoint and
moves the uploads of a changed checkpoint before the lock
> - The benefit is that an unchanged checkpoint takes no row lock, and
the row lock is no longer held across object-storage I/O

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The warm agent-file checkpoint: `checkpoint()` in
`server/src/services/agent-directory-working-copies.ts`,
`agentFileStore.locked` and `apply` in `agent-file-store.ts`, and
`persistDurableAgentFiles` in `agent-file-durability.ts`.

**Current behavior**
- When the run changed no file, the checkpoint still ran
`store.locked(..., write = true)`. That transaction locks the agent row
(`FOR UPDATE` and `FOR NO KEY UPDATE`), restores and invalidates the
local projection, walks the canonical tree, and compares the durable
head. Its only result is the authorization check.
- When the run changed files, `persistDurableAgentFiles` uploaded each
new file to object storage inside the transaction that holds the agent
row lock.

**Proposed behavior**
- An unchanged checkpoint (`saveUnchanged`) first reads, without a lock,
whether the locked file-tree step would change anything
(`agentFileProjectionCurrent`): the durable head exists and the local
projection is marked at its generation, or durable files are off, no
head exists and the local adoption is recorded. If so, it runs only the
write authorization (`authorizeSave`): the agent interaction policy and
`authorizeInstructionCommit`, in a short transaction without row locks
that always rolls back, so it writes nothing. If not, it runs the locked
save as on `origin/master`, which adopts, restores or heals the durable
head. A de-authorized agent still gets the same error (the existing test
"rechecks authorization even for an unchanged warm checkpoint" still
passes).
- A changed checkpoint runs the same read-only authorization (it
consumes no accepted change confirmation), then uploads the run's
changed files before it takes the lock (`uploadAgentFilesBeforeCommit`).
A run that lost its save permission uploads nothing. The locked commit
runs the authorization again, and only that commit consumes a
confirmation. The locked commit uses an uploaded object only for a file
with the same SHA-256 and size. It uploads any other file itself, as
before.

**Reason and benefit**
The target is 1,000 agents in one controller process. Most warm turns
change no agent file. These turns now take no row lock and send fewer
statements. A changed turn holds the row lock for less time, because
object-storage latency is outside the lock.

## What Changed

- `agent-file-store.ts`: add `authorizeSave(companyId, agentId, actor)`
and `saveUnchanged(companyId, agentId, actor)`. `apply` calls it for a
checkpoint, then uploads the run's changed files before `locked`, and
passes the uploaded objects to `persistDurableAgentFiles`. `locked`
takes an optional map of uploaded objects.
- `agent-file-durability.ts`: move the verified upload stream into
`uploadAgentFile`. Add `uploadAgentFilesBeforeCommit`. It reads the
durable head without a lock, skips files the head already has at the
same path and digest, and uploads the others. If an upload fails, it
stops and the locked commit uploads the rest, so errors are the same as
before. `persistDurableAgentFiles` uses an uploaded object when the
canonical file has the same digest and size.
- `agent-directory-working-copies.ts`: an unchanged checkpoint calls
`saveUnchanged` instead of `store.locked(..., true)`.
- `agent-file-durability.ts`: add `agentFileProjectionCurrent` (one head
read and two file checks, no lock).
- The commit is still inside the agent row lock. The head cannot change
between the read and the write of the commit, so it needs no separate
compare-and-set. The upload before the lock does not need one either: an
object is immutable and verified by digest, and the commit records it
only for a file with that digest.

## Verification

- New `server/src/__tests__/agent-file-checkpoint-locking.test.ts`
(embedded PostgreSQL, object-storage fixture). All tests fail on the
code before the change and pass with it:
- An unchanged checkpoint completes while another transaction holds the
agent row lock. It sends no `FOR UPDATE`, `FOR NO KEY UPDATE` or `FOR
SHARE` statement and only one unlocked read of `agent_file_roots`.
- Durable files on, the agent has local files and no head: an unchanged
checkpoint creates the head, as on `origin/master` (added after review;
failed without the fallback).
- The local projection is not marked at the head: an unchanged
checkpoint takes the locked path and marks it again (added after review;
failed without the fallback).
- A changed checkpoint uploads its file while the agent row is not
locked (the fixture checks the lock with `FOR UPDATE NOWAIT` from
another connection during each upload). The head then records that
object.
- A run loses its save permission before a changed checkpoint. The
checkpoint is refused and uploads nothing (added after review; I saw it
fail without the authorization before the upload).
- An agent with `agents:suggest-changes` and one accepted change
confirmation saves a changed checkpoint. An unchanged checkpoint first
leaves the confirmation unconsumed. The changed save succeeds once and
consumes the confirmation once; the next change is refused before it
uploads (added after review; I saw it fail when the preview committed).
- Two warm runs of the same agent save at the same time. Both files are
in the canonical tree and in the head, and every head object has the
correct digest.
- An instruction edit commits while a checkpoint waits in its upload,
without a wait for the checkpoint. Then the checkpoint commits. Both
changes are in the head.
- A save stops between upload and commit (the instruction entry changes
while the bytes upload, so the commit refuses). The head does not change
and stays restorable. The next save commits the same file. The unused
object is never authoritative; the existing object collector removes it
after its grace period.
- Existing `agent-directory-working-copies.test.ts`,
`agent-file-durability.test.ts` and `agent-file-checkpoints.test.ts`
pass.
- Measurement (opt-in, `PAPERCLIP_AGENT_FILE_CHECKPOINT_BENCHMARK=1`,
test "measures checkpoint statements and agent row lock time"): one warm
copy with 51 files; 10 unchanged checkpoints; then 5 checkpoints that
each change one file, with 250 ms object-store latency. Wall times are
noisy (load average about 40). Statement counts are exact.

| | Before (`origin/master`) | After (final head) |
| --- | --- | --- |
| Unchanged checkpoint: statements | 26 | 25 (one unlocked head read,
and the read-only authorization) |
| Unchanged checkpoint: row-lock statements | 2 | 0 |
| Unchanged checkpoint: median time | 214 ms | 26–126 ms |
| Changed checkpoint: statements | 29 | 47 (the read-only authorization
before the upload and one head read) |
| Changed checkpoint: median time the agent row is locked | 924 ms |
51–227 ms |
| Changed checkpoint: uploads while the agent row is locked | 1 | 0 |

The "after" ranges come from two runs at different machine load (load
average between about 18 and 40).

## Risks

- Master side effects of the locked unchanged save, and what this change
does with each:
- Kept: the write authorization (`assertAgentInteraction`,
`authorizeInstructionCommit`), so a run that lost its permission still
gets the same error.
- Kept through the fallback: one-time adoption (no head yet), restore of
a projection that is not marked at the head, re-mark after a failed
persist (a write path removes the marker before it changes files), and
heal of a head that lags the root after such a failure.
`agentFileProjectionCurrent` returns false in each of these cases, so
the locked save runs.
- Dropped when the projection is current: the re-mark at the same
generation (no change), and the compare of the local tree with the head.
With the marker at the head generation, every writer that changed the
tree went through the locked save and moved the marker, so the compare
finds no difference. An edit made outside Paperclip directly in the
canonical directory is not picked up by an unchanged checkpoint any
more; the next save that goes through the locked path still records it.
- Dropped when the projection is current: the storage-limit check of the
canonical tree in `persistDurableAgentFiles`. An over-quota agent whose
run changed nothing no longer gets an error from that check; any save
that changes files still enforces the limits.
- Dropped: the `FOR NO KEY UPDATE` locks that `assertAgentInteraction`
takes on the actor and target agents (`lock: "update"`). Those locks
hold permission changes off a write until it commits. The read-only
authorization writes nothing, so there is nothing to hold them off. The
locked commit of a changed save still takes them. An unchanged save has
no mention texts, so master locked no other agents for it.
- Effects of `authorizeSave` outside the rolled-back transaction: none.
I read `assertAgentInteraction` (`agentInteractionDenial`: reads only),
`authorizeInstructionCommit`, `resolveInstructionActor` (reads only),
`authorizationService.decide` (reads only; its memo maps live on the
bound actor and on the service instance, both created for this call and
dropped after it), and `changeConsentGateService.assertConsented` (one
`UPDATE` of the confirmation row, rolled back). None of them publishes
live events, queues notifications or wakes, writes activity through
another connection, makes external calls, or changes an in-process cache
that outlives the call.
- Behavior change: on `origin/master`, an unchanged checkpoint of a
consent-gated agent (`agents:suggest-changes`) consumed an accepted
change confirmation, because the locked transaction committed. Now the
fast path does not; only the commit of a changed save consumes it. When
the fast path cannot be used (adoption, restore or heal), the locked
save runs as on master and can still consume a confirmation.

- Not in this change: the manifest is still stored twice in the receipt
(`baseline` and `checkpointManifest`). Servers of the previous version
read `receipt.baseline` during a deploy overlap, and the next capture
needs the stamps in `checkpointManifest`, so neither can be removed
safely now.
- Not in this change: on the volume-free server, `serial()` still opens
a dedicated connection with an advisory-lock transaction for each
checkpoint. That lock serializes controllers and is not a row lock;
changing it is a separate design decision.
- A changed checkpoint can now leave an unused object in storage when
its commit is refused after the upload (for example, the instruction
entry changes during the upload) or the process stops before the commit.
This was already possible for a failed commit. Such an object is never
in the durable head, so no reader can reach it, and the existing
`collectAgentFileObjects` maintenance removes it after
`AGENT_FILE_OBJECT_GRACE_MS`. A permission that was removed before the
save stops the save before any upload.
- Old and new servers can run at the same time: the head format and the
receipt format do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:41:46 -06:00
DottaandPaperclip 8f105e743a fix(accounting): recover closed legacy runs without inventing usage (#21493)
Preserve exact physical process closure independently from incomplete cost reporting. Recover a missing receipt through the existing authorized accounting Retry without replaying provider work.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 14:37:13 -06:00
DottaandClaude Opus 5.5 11f4a76743 perf(native-runtime): share one frozen assigned tool catalog across sessions (#21482)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every native agent session. Each session
gets the app tools (MCP tools) that are assigned to its agent.
> - Each live or warm session kept two private copies of its assigned
tool catalog: the gateway descriptors and a canonical copy of every
input schema.
> - With 400 tools of about 5.7 KB each, that was about 7.6 MB of heap
for each session. With 800 tools, it was about 15 MB. Warm sessions stay
for 5 minutes after the last turn, so idle agents also hold this memory.
> - At 1,000 concurrent agents, this is 7 to 15 GB of heap, and the
server cannot get there.
> - This pull request keeps one deeply frozen catalog for all sessions
of the same company that get the same admitted tools and schemas.
> - The benefit is that each additional session on the same catalog adds
about 40 to 80 KB of heap instead of 7.6 to 15 MB, and session start is
faster.

## Linked Issues or Issue Description

Refs #21415 (releases a finished session's tool authority). Refs #21440
(adds the catalog memory benchmark that this change extends).

**What existing behavior does this improve?**
`createAssignedMcpTools` in
`server/src/services/native-runtime/assigned-mcp-tools.ts`. Each call
kept every gateway descriptor (with its provider metadata) and a
separate canonical clone of each input schema. Each `definitions()` call
also made a new `structuredClone` of every schema.

**Subsystem affected**
Native runtime: assigned app tools (MCP) for native agent sessions.

**Current behavior**
Retained heap grows by one full catalog for each session. Measured with
the real `createAssignedMcpTools` (5.7 KB schemas): 400 tools is 7.58 MB
per session, and 800 tools is 15.1 MB per session, for 1, 10, and 100
sessions. Each `definitions()` call takes 14 to 41 ms because it clones
every schema.

**Proposed behavior**
Sessions of the same company that get identical admitted tools share one
frozen catalog. The total for 100 sessions is about one catalog: 4.2 MB
at 400 tools and 8.1 MB at 800 tools. `definitions()` takes 0.01 ms.

**Reason and benefit**
The server must hold 1,000 concurrent agents in one process. The catalog
was the largest per-session heap cost. Revocation and authorization do
not change: every call is still authorized again by the gateway, and
every new session still reads a fresh listing.

**Breaking changes**
None in the API or data. The direct definitions that `definitions()`
returns are now frozen. A consumer that writes to a returned definition
or schema gets a `TypeError` and does not change other sessions. No
production consumer writes to them (see Risks).

## What Changed

- `server/src/services/native-runtime/assigned-mcp-tools.ts`:
- A session keeps only a small frozen record for each tool (gateway
name, display name, description, risk), plus the frozen schema and the
frozen definition. It does not keep the gateway descriptors.
- New per-process index of shared catalogs. The key is the SHA-256 of an
identity string: `[companyId, [[assignedName, gatewayName, displayName,
description, risk, sha256(JSON of the input schema)] for each listed
tool, in listing order]]`. On a hit, the code also compares the full
identity string, so a hash collision cannot give a wrong catalog.
- The identity is computed from the fresh listing that the gateway
returns for this session. The gateway applies all per-agent, per-run,
per-profile, and policy filtering, and the approval description suffix,
before it returns the listing. Thus two sessions share only if they were
given exactly the same tools, names, descriptions, risks, and schemas in
the same company. Without `companyId`, nothing is shared.
- Schemas, definitions, tool records, and the definition arrays are
deeply frozen. Search results still return cloned copies.
- Sessions that start at the same time with the same catalog wait for
one build and share it.
- The index holds each catalog through a `WeakRef`, so it never keeps a
catalog alive. The index has at most 512 entries and 256 MiB (accounted
schema and description bytes). The oldest entries are removed first. A
catalog that is larger than the byte limit is not indexed.
- New `close()` on the assigned tools. When the last session that holds
a catalog closes, the index entry is removed. A `FinalizationRegistry`
also removes entries whose catalog was collected.
- `definitions()` returns the shared frozen definitions for the
session's work mode. It no longer clones every schema on each call.
-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
`close()` also closes the assigned tools. The session executor already
closes the authority when it revokes a session's tool authority.
- `server/src/services/native-runtime/native-session-executor.ts`: one
line. It passes the run binding's `companyId` to
`createAssignedMcpTools`.
- `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: passes
`companyId`, adds `--companies`, measures session start and
`definitions()` time, and returns fresh copies of fixed rows (like
database reads).
- Tests in `assigned-mcp-tools.test.ts` and
`paperclip-runner-tool-authority.test.ts`.

## Verification

Red/green tests in
`server/src/services/native-runtime/assigned-mcp-tools.test.ts` ("shared
assigned tool catalogs"):

- Retained heap: 20 sessions of one company on a 60-tool, 4 KB-schema
catalog. The heap for the other 19 sessions must be less than two
catalogs. Red on master: `expected 14025976 to be less than 3788112`.
Green with this change.
- Shared deeply frozen schemas, also for sessions that start together.
Red on master (`Object.is` equality fails).
- Mutation isolation: writes to a returned definition, schema property,
and `required` array, and a delete, throw `TypeError`. Changes to the
gateway's source rows and to a search result do not change any session.
Red on master (no throw).
- Release and bounds: the entry stays while one session holds it and
goes when the last session closes. `close()` is idempotent. A new
session after release builds a new catalog. 517 companies keep the index
at 512 entries or fewer. Red on master (no `close` or stats).
- Guard tests: two companies with identical catalogs do not share.
Sessions without a company do not share. Fewer tools, an edited schema,
an edited description, or a different risk do not share. A revoked or
edited tool is served from fresh discovery. On master these pass,
because master never shares. To prove they test the key, I removed
`companyId` and the schema digest from the identity: three guard tests
failed, and they pass with the full key.
- An existing test changed: "does not project gateway secrets..." now
expects a write to a returned schema to throw `TypeError` (before, the
write changed only a private clone). The isolation check after the write
is the same.
- `paperclip-runner-tool-authority.test.ts`: `authority.close()` closes
the assigned tools.

Heap (post-GC `heapUsed` delta, `node --expose-gc`, 5.7 KB schemas, one
company). Command: `cd server && node --expose-gc --import tsx
scripts/benchmark-assigned-mcp-tools-memory.ts --tools <N>
--schema-bytes 5700 --sessions <S> --turns 5`.

| Tools | Sessions | Total before | Per session before | Total after |
Per session after |
| --- | --- | --- | --- | --- | --- |
| 400 | 1 | 7.60 MB | 7.60 MB | 3.96 MB | 3.96 MB |
| 400 | 10 | 75.9 MB | 7.59 MB | 4.04 MB | 0.40 MB |
| 400 | 100 | 757.9 MB | 7.58 MB | 4.18 MB | 0.042 MB |
| 800 | 1 | 15.2 MB | 15.2 MB | 7.91 MB | 7.91 MB |
| 800 | 10 | 151.4 MB | 15.1 MB | 7.98 MB | 0.80 MB |
| 800 | 100 | 1,513 MB | 15.1 MB | 8.13 MB | 0.081 MB |

The first session also uses about half the heap of before, because the
session no longer keeps the gateway descriptors. When all sessions are
dropped without `close()`, the catalog is collected (0 bytes held after
release).

Session start, interleaved A/B in one process (old and new
`createAssignedMcpTools` alternate, 40 starts each, median of the last
35, includes a fresh copy of the listing; the machine had a load average
of 25 to 65):

| Tools | Listing copy only | Before | After |
| --- | --- | --- | --- |
| 400 | 22.6 ms | 68.0 ms | 37.6 ms |
| 800 | 30.5 ms | 72.5 ms | 41.2 ms |

A start that reuses a catalog does no schema clone and no admission. It
still serializes and hashes each schema to compute the key. The first
start of a new catalog does the same admission work as before (the cold
first start was about 600 to 750 ms in both versions on this busy
machine).

`definitions()` median: 14 to 41 ms before, 0.01 ms after.

Other checks:

- `cd server && npx vitest run
src/services/native-runtime/assigned-mcp-tools.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-tool-results-prp.integration.test.ts
src/__tests__/deployment-connections.test.ts`: 204 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"assigned|tool authority|released|MCP"`: 31 passed.
- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "assigned"`: 10
passed.
- `cd server && npx tsc --noEmit`: pass.
- `pnpm check:module-boundaries`: pass.
- `pnpm verify` selects a broad plan (the benchmark script is a broad
input), so CI supplies the full suite.

## Risks

- Frozen definitions: the direct definitions now go to the runner
drivers as frozen objects. I checked the consumers.
`PaperclipRunnerToolAuthority.definitions()` filters and spreads them.
The Codex, Cursor Cloud, OpenCode, ACPX, and runnerd drivers read them,
spread them, clone them (`structuredClone` in ACPX and the OpenCode MCP
bridge), or serialize them. Their types are `readonly
Readonly<Record<string, unknown>>[]`. The first-party semantic tool
definitions are already shared module constants that go through the same
path without clones. If a consumer that I did not find writes to a
definition, it gets a `TypeError` instead of a silent change. Catalogs
over 256 tools or over the listing budget use the two on-demand tools
and are not affected.
- Authorization: the shared data is only the result of a listing that
the gateway already authorized for that session, keyed on the full
content plus the company. Execution still goes through the gateway with
the session's own token on every call. Search still intersects with a
fresh listing. A new session still reads a fresh listing, so a revoked
or edited tool takes effect for new sessions and for existing sessions
at the same points as before.
- Several server processes: the index is a per-process memo of the
listing content. Each process builds its own catalog from its own
listing. Nothing is stored or shared outside the process.
- The index is bounded (512 entries, 256 MiB) and holds catalogs only
weakly. Index eviction only stops future sharing. It never frees memory
that a session uses.
- No schema change. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs, heap and timing benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:30 -06:00
DottaandClaude Opus 5.5 533d02ff1a perf(recovery): match scoped deadline entries only against scopes of their own company (#21467)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Live-run recovery uses the shared scoped deadline worker
(`scoped-deadline-work.ts`). Each queue keeps its entries (one for each
running run) and the scopes that writes marked dirty (for example one
company and one run)
> - To schedule and to pick due work, the worker tested every entry
against every dirty scope. It also decoded each scope hint with
`JSON.parse` again inside those loops, and each reconstructed scope
walked all entries to drop vanished ones
> - With many runs and many dirty run scopes, this work grows with
entries × scopes on the event loop of the one server process that
controls all runs
> - This pull request lets an owner declare a partition key for its
scopes. Live-run recovery uses the company. The worker then tests an
entry only against the scopes of its own company, and decodes each hint
once
> - The benefit is that matching cost grows with entries + scopes, with
the same results

## Linked Issues or Issue Description

Refs #21444 (the lease renewal change from the same load model).

**What existing behavior does this improve?**
The scoped deadline worker in
`server/src/services/scoped-deadline-work.ts`, which live-run recovery
(`server/src/services/heartbeat/live-recovery-work.ts`) uses for its
seven queues.

**Subsystem affected**
Server: background work scheduling for live-run and resource recovery.

**Current behavior**
For 400 entries in 40 companies and 40 dirty run scopes, one schedule
computation and one pass call `matches` 27,830 times and decode each
hint many times.

**Proposed behavior**
The same case calls `matches` fewer than 3,200 times and decodes each
hint at most once. Owners that do not declare a key behave exactly as
before.

**Reason and benefit**
The plan for 1,000 concurrent agents needs per-run background work that
does not grow with the product of runs and changes. The work runs on the
main thread of the controlling process.

**Breaking changes**
None. `scopeKey` is optional.

## What Changed

- `scoped-deadline-work.ts`: new optional `scopeKey` option
(`entry(entry)` and `scope(scope)`). The contract: a scope with a
defined key matches only entries with the same key. A scope without a
key (for example `*`) is tested against every entry, as before.
- The worker indexes the dirty scopes by key, so `next()` and the due
filter test each entry only against the scopes of its key and the
unkeyed scopes.
- When a reconstructed keyed scope drops vanished entries, it visits
only the entries of its key (an index built once for each pass).
- Scope hints are decoded once and the decoded value is reused while the
hint is dirty or scanning.
- `live-recovery-work.ts`: declares the company as the key. Its
`matches` already requires the same company for every defined scope.

## Verification

- New test in `scoped-deadline-work.test.ts`: 400 entries in 40
companies, then 40 run-scoped invalidations. It requires fewer than
3,200 `matches` calls and at most 40 decodes, and the next deadline is
unchanged.
- Red first: with `origin/master` `scoped-deadline-work.ts`, the test
failed with 27,830 `matches` calls.
- `pnpm exec vitest run src/__tests__/scoped-deadline-work.test.ts`: 22
passed.
- The live-run recovery matrix (`live-recovery-work.test.ts`) is covered
by CI.

## Risks

- If an owner declares a key that its `matches` does not respect, the
worker can miss a match. Only live-run recovery declares a key, and its
`matches` returns false for any defined scope of another company.
- Low risk otherwise: owners without a key keep the old code path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:21 -06:00
DottaandClaude Opus 5.5 754b1f36d3 perf(native): renew all of a process's execution leases in one transaction, without a recovery wake (#21444)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every running native agent. Each run
holds a native execution lease that the process must renew before it
expires (60 seconds for a remote run, 20 minutes for a local run)
> - Each run renewed its lease on its own timer, in its own transaction,
on a private pool of two connections. At 1,000 remote runs that is about
50 transactions each second through two connections
> - When renewals queue behind each other, a renewal can miss its 60
second deadline, and the run is lost although the server is healthy
> - Each renewal also woke six live-run recovery queues for its run. The
recovery owner then re-checked a healthy run about every 20 seconds,
although a renewal only moves the deadline later
> - This pull request renews all leases of one process with one shared
timer. All leases that are due go into one transaction of fixed size,
the same model that the legacy controller lease already uses. A renewal
no longer wakes recovery, as #20924 did for the legacy lease
> - The benefit is that the renewal cost no longer grows with the number
of runs, and all safety properties of a single renewal stay the same

## Linked Issues or Issue Description

Refs #20924 (the same change for the legacy controller lease), #21242.

**What existing behavior does this improve?**
Native execution lease renewal in
`server/src/services/native-runtime/native-execution-lease.ts`. Each run
had one `setInterval` and one transaction per renewal: statement
timeout, row lock, recovery notification, update, commit.

**Subsystem affected**
Server: native runtime, controller lease renewal on the liveness
connection pool.

**Current behavior**
The renewal cost grows with the number of runs. A local load model with
1,000 remote runs sends 35.7 lease transactions and 212.8 lease
statements each second, and the event-loop utilization is 0.34. All of
them use a pool of two connections.

**Proposed behavior**
One timer for each database client renews all due leases in one
transaction for each 200 runs. With 1,000 remote runs the same model
sends 0.29 lease transactions and 1.7 lease statements each second, and
the event-loop utilization is 0.07.

**Reason and benefit**
The plan for 1,000 concurrent agents needs renewals that do not scale
with the run count (item 1d). A renewal that misses its deadline loses
the run.

**Breaking changes**
None. The `NativeExecutionLeaseRenewal` contract (`stop()`,
`remainingMs()`) and `renewNativeSessionExecutionLease` do not change.

## What Changed

- `native-execution-lease.ts`: a renewal (batched or single) no longer
calls `notifyLiveRunRecovery`. A renewal only moves the lease deadline
later. Recovery already holds a deadline at the earlier expiry. At that
deadline it re-reads the row and moves to the new expiry. A lapsed lease
is still found at its expiry. This change does not touch the signals of
the writers that release, revoke or take over a lease, or that change
the run status. This also removes six notifications for each renewal,
which was 1,200 notifications for a batch of 200 runs during a deploy
overlap.
- `native-execution-lease.ts`: one shared schedule for each database
client replaces the timer of each run. When the first lease is due, all
leases that are due within a quarter of their own interval renew with
it. Thus the leases of a process converge on one batch for each
interval.
- A batch is one transaction on the liveness pool for each 200 runs:
`statement_timeout`, one `SELECT … FOR UPDATE SKIP LOCKED` of the
coordinator rows, and one `UPDATE … FROM (VALUES …)`.
- The batch keeps the guarantees of a single renewal:
- It locks each coordinator row before it reads the expiry, as before.
Thus an owner whose lease expired while another writer held the row is
never revived.
- It extends a row only for the exact owner: run, company, task, lease
owner, attempt, controller boot id, pid, process start, and a live
lease.
- The confirmed deadline is sampled after the locks and before the
update, as before.
- A run whose row was locked and not extended fails with
`native_session_lease_lost`. The other runs in the batch succeed.
- These rows do not renew in the batch. They renew alone through the
unchanged single-run path:
- A row that another writer holds (`SKIP LOCKED`). Thus one busy row
cannot hold the batch until the statement timeout.
- A row that the lock did not find (another owner, or no row). The
single path gives the same answer as before.
- A local lease that lapsed (host sleep). Only the single path may take
it back, under the run row lock.
- A batch that is cancelled (statement timeout `57014` or deadlock
`40P01`) extends nothing. Its runs retry after 1 second, and each run
keeps its own deadline timer. Before this change, a statement timeout
waited for the next 20 second tick.
- Any other batch failure, also an error thrown while the statement is
built, may come from one bad row. Each run of that batch then renews
alone at once through the single path, so a bad row can cost only its
own run. A run that fails alone while others in its batch renew keeps
renewing alone (out of later batches) until a single renewal succeeds.
During a database outage the single renewals also fail; those runs
return to the batch at their next interval, not after a 1 second retry,
so an outage causes at most one queued single transaction for each run
and interval.
- Runs that become due while a batch is in flight wait for that batch. A
batch that runs for more than 15 seconds no longer holds back the next
batch.
- A tick that passed while a renewal was in flight is skipped, as a
fixed interval skips it. Thus a slow database gets no extra load.
- The shared timer replaces the interval timer of each run. It exists
only while this process holds a lease, and it fires at the next renewal
deadline. With no held lease there is no timer and no SQL.
- New export `renewNativeExecutionLeasesNow(db)` renews all leases of
the process at once. Tests use it.

## Verification

- New PostgreSQL tests in `native-execution-lease-batch.test.ts`:
- 3 runs and 60 runs each renew in 1 transaction with the same number of
statements.
  - 450 runs renew in 3 transactions.
- In one batch, the runs whose lease was taken over, whose controller
boot changed, or whose lease expired fail with
`native_session_lease_lost`. Their rows do not change. The healthy runs
renew. A newer attempt under the same owner is not lost, as before.
- Interleaving: another transaction holds a coordinator row from before
its lease expires until after it expired. The healthy run in the same
batch renews at once. After the lock is released, the expired owner is
lost and its row stays expired.
- A takeover that commits before the batch: the batch extends nothing
and the run is lost.
- A local and a remote lease renew in one transaction, each for its own
lease length.
- A poison binding (an attempt outside the integer range) makes the
shared UPDATE fail with `22003`. The other 4 runs still renew in the
same tick, only the poison run is lost, and the next tick renews the 4
runs in 1 transaction again.
- An invalid date throws while the statement is built. The other runs
renew, the broken run is lost alone, and no lease stays busy (the next
call renews again in 1 transaction).
- New PostgreSQL test in `live-recovery-work.test.ts` with the real
registered recovery owners: another server's native run renews its
lease. No recovery pass runs after the renewal. At the old expiry
recovery re-reads the row, keeps the run running and moves its deadline
to the new expiry. When renewals stop, recovery acts on the run within
10 seconds after the end of the last lease.
- Red first: with a shim on `origin/master` that renews each lease
alone, the transaction test failed (`expected 3 to be 1`) and the
450-run test failed.
- The unit tests in `native-execution-lease.test.ts` now drive the
batch. New tests show that a non-cancelled batch failure renews the run
alone at once, and that an outage (batch and single renewal both fail)
does not retry every second and loses no run.
- `native-execution-lease-reacquire.test.ts`: the host-sleep tests now
go through the batch, which hands the lapsed local lease to the single
path.
- Load model (`startNativeSessionExecutionLeaseRenewal` for N remote
runs that start at random times over 20 seconds, 70 second window, local
PostgreSQL, machine load average about 50):

| Runs | Lease transactions/s before → after | Lease statements/s before
→ after | Event-loop utilization before → after | Smallest margin to the
60 s deadline | Lost |
| --- | --- | --- | --- | --- | --- |
| 50 | 1.81 → 0.11 | 10.9 → 0.69 | 0.015 → 0.095 (noise) | 40.0 s → 40.0
s | 0 → 0 |
| 1,000 | 35.67 → 0.29 | 212.8 → 1.71 | 0.341 → 0.065 | 39.0 s → 40.0 s
| 0 → 0 |

The window includes the 20 second ramp, so the steady rate before is
higher (one transaction each 20 seconds for each run, 50/s at 1,000
runs). After the change, one batch carries up to 200 runs.

## Risks

- **Behavior change: batch-wide error handling.** One shared transaction
now carries up to 200 runs, so one database error reaches all of them. A
cancelled batch only skips the extension and retries after 1 second. Any
other batch error sends each run to the single path at once, so one bad
row (a bad value, a cast error, an error while the statement is built)
costs only its own run. In a database outage, up to one single
transaction for each run waits on the two-connection pool at each
interval. Each run's own deadline is unchanged.
- **Behavior change: no recovery signal on renewal.** See below.
- During a deploy overlap the old server renews its runs one by one and
the new server renews in batches. Both use the same row lock and the
same conditions, so they do not conflict. Rows written by the old
version have the same columns.
- `SKIP LOCKED` sends a busy row to the single path, which waits for the
lock as before.
- **No recovery signal on renewal:** recovery of a healthy run of
another server now runs once for each lease period at the old expiry,
not after each renewal. A dead run is found at the same time as before:
at the expiry of its last lease.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:16 -06:00
DottaandClaude Opus 5.5 969cf246ae perf(run-events): keep each native source's cursor on the run row (#21435)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - Each native source event append counted all retained events of its
source (`count(*)`, `max(source_seq)`) to compute the acknowledgement
cursor
> - So the cost of each append grew with the length of the run, and long
runs made every text delta slower
> - This pull request keeps the gap-free cursor of each source on the
run row and moves it in the same UPDATE that reserves the event sequence
> - The benefit is that an in-order append stops reading other events,
so its cost does not grow with run length

## Linked Issues or Issue Description

Refs #21131 (earlier reduction of statements per event). Item 1b of the
agent scaling plan in #21418.

**What existing behavior does this improve?**
The append of a native source event (`appendNativeSourceEvent` in
`server/src/services/heartbeat-run-events.ts`). It ran a `retained` CTE
that counted every event of the source on every append.

**Subsystem affected**
Server run event ingest (`heartbeat_run_events`, `heartbeat_runs`).

**Current behavior**
On a run that already holds 50,000 events, one in-order append takes
about 10–13 ms (p50) on a local PostgreSQL. On a run with 400 events it
takes about 3.3 ms.

**Proposed behavior**
The append reads the stored cursor in the run lock statement that it
already sends. An in-order event moves the cursor and does not count
events. On the same 50,000-event run an append takes about 1.7–3.3 ms
(p50).

**Reason and benefit**
Text deltas are about 85% of the server's statements in the load model.
Their cost must not grow with run length.

**Breaking changes**
None. The new column is nullable. The previous release ignores it.

## What Changed

- Add the nullable column `heartbeat_runs.native_source_cursors`
(`jsonb`, `{"<source instance id>": <seq>}`). Migration `0479` is `ADD
COLUMN` only and classifies as `online_compatible`.
- The run lock read of a native append also reads the cursor of its
source.
- When the event sequence is at most cursor + 1, the append statement
has no `retained` CTE. It moves the cursor in the UPDATE that reserves
the sequence, and the receipt uses the cursor.
- Every other case (no cursor, a gap, or a cursor behind the stored
events) runs the earlier recount. In the same UPDATE it stores the
cursor when the source has no gap, and removes it when the source has a
gap. So the event that fills a gap recounts.
- The cursor is a lower bound. A writer that does not keep it (the
previous release during a deploy overlap, or
`NativeRunCoordinatorStore`) can only make it lag. The next out-of-order
event then recounts and corrects it.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-event-atomic-append.test.ts
src/__tests__/heartbeat-run-event-sequencing.test.ts` passes (19 tests).
- Red/green: before the change, 3 tests failed: the two new tests and
the updated lock-read statement test. After the change, all pass.
- New: "keeps the source cursor on the run so an in-order append does
not count the run's events" checks that the append statement has no
`count(` and that a replay returns the cursor.
- New: "recounts a source whose run has no cursor or a cursor behind its
stored events" covers rows saved before the change, rows from a writer
that does not keep the cursor, a gap, and the event that fills the gap.
- Focused related tests: `paperclip-control-plane-port.test.ts`,
`native-observer-replay`, `native-event-boundary-golden`,
`chat-native-progress-work`, `live-recovery-work`,
`native-interaction-bridge`.
- `pnpm check:module-boundaries` passes. `pnpm verify` returned exit 2
(broader CI coverage required) because of local benchmark scripts that
are not in this pull request. Required CI supplies the broad coverage.
- Append cost on a run with many events (local embedded PostgreSQL,
1,000 in-order appends, busy machine, load average about 20–40):

  | Events already in run | Before p50 / p90 | After p50 / p90 |
  | --- | --- | --- |
  | 50,000 (run 1) | 10.4 / 15.8 ms | 1.7 / 3.7 ms |
  | 50,000 (run 2) | 12.9 / 21.6 ms | 3.3 / 5.9 ms |
  | 400 | 3.3 / 6.2 ms | 3.7 / 7.7 ms (noise) |

- Statements per event do not change: BEGIN, run lock read, one append
statement, COMMIT (4). The append statement no longer scans the source's
events.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418,
`AGENTS=50 SECONDS=20`): the machine was too busy for a stable result.
Deltas only: before 20,000 of 20,000 done at 3,999 statements/s. After:
13,796 and 14,817 of 20,000 in two runs. All sources: before 6,346 of
20,000 deltas done. After: 20,000 and 2,083 in two runs. A 20-second
window gives only 400 events per run, so it cannot show this change. The
long-run figures above are the evidence.

## Risks

- Low. The column is nullable and the previous release does not read it.
Runs that are in flight during a deploy have no cursor. Their first
append recounts once and stores the cursor.
- A stale cursor can only make the acknowledgement lag, never pass the
stored events. Event rows are deleted only with their company or agent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:08 -06:00
DottaandClaude Opus 5.5 0d832a02d3 perf(ci): shorten the general server and chat test lanes (#21393)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the `PR` CI workflow before it can merge
> - The `General tests (server (N/12))` and `General tests (chat (N/3))`
jobs finish last, so they set the wall-clock time of CI
> - Before this change, server shards took 9.5 to 11.7 minutes and chat
shards took 6.4 to 9.6 minutes; the target is 7 minutes or less
> - Measurement shows that the chat lane was unbalanced and that the
server lane is limited by CPU on the 4-vCPU runners
> - This pull request balances the chat lane by recorded test time, runs
three chat partitions per job, and takes general server weights from PR
runs
> - The benefit is a chat lane under 7 minutes and measured evidence for
the shard count that the server lane needs

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The PR CI general server and chat test jobs. They are the slowest jobs
of every pull request.

**Subsystem affected**

CI test scheduling: `scripts/run-vitest-stable.mjs`,
`scripts/test-line-shard.mjs` and
`scripts/ci/refresh-test-durations.mjs`.

**Current behavior**

Baseline: green `PR` runs
[43437558012](https://github.com/paperclipai/paperclip/actions/runs/43437558012)
and
[43441184316](https://github.com/paperclipai/paperclip/actions/runs/43441184316),
all on m7i-flex.xlarge (4 vCPU, 16 GB).

- Server shards: 9.5 to 11.7 job minutes, 502 to 654 s test step,
effective file concurrency 2.5 to 2.9 of 3 workers. Vitest reports
`tests 61-68%, import 25-31%, transform 6-7%, setup 1%`. Each shard sums
1,410 to 1,765 s of file time. The longest file is 303 s
(`deferred-execution-receipt-work.test.ts`), so the floor is total work
divided by workers, not one file.
- Chat shards: 9.3 to 9.6, 7.6 to 7.8 and 6.4 to 6.5 job minutes. Each
job ran two partitions of `chat-channels.integration.test.ts`. The
partitions were balanced by collected-case count, but one `it.each` line
can run many cases. One of six partitions recorded 532 s of test time
against a 377 s mean.

**Proposed behavior**

- Chat partitions keep each source line whole and balance by recorded
time per collected entry.
- Each chat job runs three partitions in place of two.
- General server weights come from the server jobs of merged PR runs,
which run files with the same concurrency as the lane they place.

**Reason and benefit**

The slowest job sets the CI wall-clock time for every pull request. The
chat lane goes from a 9.3-9.6 minute maximum to 6.5-6.8 minutes.

## What Changed

- `scripts/test-line-shard.mjs`: `partitionTestLines` takes optional
recorded durations. With durations, it balances source lines by recorded
time; entries without a record take the median. Without durations, the
behavior is unchanged.
- `scripts/chat-shard-durations.json` (new): recorded time per collected
chat entry, from the CI profile records of PR run
[43444854171](https://github.com/paperclipai/paperclip/actions/runs/43444854171).
926 of 928 collected entries are measured; the other two are the bogus
collection entries that the launcher already drops.
- `scripts/ci/refresh-chat-shard-durations.mjs` (new): builds that
manifest from downloaded `ci-profile-*` artifacts. Each executed case
maps to the collected `it`/`it.each` entry that registers it. A case
without an entry fails the refresh.
- `scripts/run-vitest-stable.mjs`: a sharded chat job runs three
partitions in place of two, and the chat launcher passes the recorded
durations. A chat job collects the suite once with `vitest list` in
place of once per partition, and the per-partition filter checks run at
the same time. Before, each partition ran two blocking collections of
the 86,000-line file one after another. The server worker count stays at
three.
- `scripts/ci/refresh-test-durations.mjs` and
`scripts/ci/pr-run-durations.mjs`: the weekly refresh reads general
server weights from the two newest merged-PR runs. Each run must have
the full server shard count from `.github/ci-task-matrix.json` and cover
at least 90% of the lane. Serialized weights still come from master
Cloud readiness runs.
- `scripts/general-server-shard-durations.json`: refreshed with the new
source (PR runs 43439145752 and 43439279537).
- Tests: `scripts/__tests__/run-vitest-stable-shard.test.mjs`,
`scripts/ci/test-selection-pr-timings.test.mjs` and
`server/src/__tests__/vitest-chat-shards.test.ts` cover the weighted
partition, the entry mapping, the manifest format, the PR-run collection
rules and the three-partition chat launcher.
- `doc/DEVELOPING.md`: describes the three chat partitions, the chat
weights and how to refresh them, and the new source of the server
weights.

## Verification

Experiments ran in this PR's own CI with a temporary resource sampler
(removed before review).

| Setting (run) | Server step vs. same shard's baseline | CPU busy | Max
memory used | Result |
|---|---|---|---|---|
| 3 workers (43444854171, shards 1-4) | +8% (profiling overhead) |
62-73% | 6.8 GB | 1 flaky retry |
| 4 workers (43444854171, shards 5-8) | -4% | 82-86% | 8.2 GB | clean |
| 5 workers (43444854171, shards 9-12) | -10% | 70-93% | 9.5 GB | 1 hard
failure: `decision-models.test.ts` timed out at 15 s twice |
| 4 workers (43450469416) | mixed m5a/m5n/m6i runners | 78-89% | 7.7 GB
| 1 hard failure: `workspace-recovery-warm-handoff.test.ts` timed out at
30 s on m5a |

The server lane is CPU bound: at four workers the CPU is 82-89% busy and
wall time improves by only about 10%. Five workers fail tests. Memory is
not the limit (no swap, no memory pressure, at least 6 GB free). Node's
compile cache and Postgres `fsync=off` showed no attributable gain
(import share stayed 29-34%; `fsync=off` cut iowait from 2-4% to
0.2-0.7% of CPU time), so neither is in this PR.

Final configuration (server: 3 workers with refreshed weights; chat: 3
weighted partitions per job), job minutes / test-step seconds /
effective concurrency:

| Shard | Baseline 43437558012 | Baseline 43441184316 | A: 43465156503
attempt 1 | B: attempt 2 | C: attempt 3 |
|---|---|---|---|---|---|
| server 1/12 | 11.0 / 609 / 2.81 | 10.2 / 562 / 2.82 | 10.1 / 535 /
2.88 | 11.6 / 615 / 2.86 | 11.3 / 600 / 2.86 |
| server 2/12 | 10.2 / 567 / 2.71 | 10.9 / 607 / 2.79 | 9.7 / 510 / 2.85
| 8.9 / 466 / 2.86 | 11.9 / 635 / 2.83 |
| server 3/12 | 10.9 / 609 / 2.88 | 10.9 / 601 / 2.86 | 9.4 / 498 / 2.81
| 11.7 / 623 / 2.80 | 20.3 / 1109 / 0.00 |
| server 4/12 | 11.0 / 613 / 2.77 | 11.2 / 622 / 2.84 | 11.3 / 602 /
2.84 | 9.3 / 487 / 2.83 | 12.5 / 676 / 2.84 |
| server 5/12 | 10.1 / 555 / 2.83 | 9.9 / 544 / 2.84 | 10.2 / 542 / 2.85
| 11.6 / 618 / 2.82 | 11.9 / 640 / 2.84 |
| server 6/12 | 9.9 / 551 / 2.85 | 10.4 / 579 / 2.85 | 9.8 / 516 / 2.69
| 9.1 / 480 / 2.68 | 9.2 / 480 / 2.67 |
| server 7/12 | 10.8 / 596 / 2.84 | 10.1 / 558 / 2.86 | 10.8 / 575 /
2.80 | 11.8 / 631 / 2.80 | 20.8 / 1129 / 2.69 |
| server 8/12 | 11.7 / 654 / 2.62 | 11.2 / 624 / 2.56 | 9.8 / 513 / 2.83
| 11.2 / 589 / 2.82 | 19.4 / 1059 / 2.40 |
| server 9/12 | 9.7 / 533 / 2.85 | 9.7 / 533 / 2.84 | 11.5 / 615 / 2.68
| 9.8 / 515 / 2.66 | 13.0 / 703 / 2.68 |
| server 10/12 | 9.8 / 540 / 2.79 | 9.5 / 502 / 2.81 | 10.6 / 561 / 2.84
| 11.8 / 625 / 2.83 | 9.3 / 488 / 2.83 |
| server 11/12 | 10.9 / 605 / 2.83 | 11.3 / 627 / 2.75 | 10.3 / 545 /
2.75 | 9.5 / 498 / 2.72 | 20.3 / 1107 / 0.00 |
| server 12/12 | 10.6 / 592 / 2.50 | 11.7 / 645 / 2.53 | 9.6 / 503 /
2.81 | 8.9 / 468 / 2.82 | 19.7 / 1065 / 2.70 |
| chat 1/3 | 9.6 / 528 / 1.56 | 9.3 / 511 / 1.58 | 6.6 / 310 / 2.69 |
5.7 / 272 / 2.65 | 6.8 / 331 / 2.69 |
| chat 2/3 | 7.8 / 418 / 1.74 | 7.6 / 408 / 1.75 | 6.3 / 305 / 2.57 |
6.5 / 311 / 2.58 | 5.5 / 260 / 2.58 |
| chat 3/3 | 6.5 / 337 / 1.74 | 6.4 / 333 / 1.73 | 6.0 / 285 / 2.65 |
5.4 / 258 / 2.68 | 5.8 / 262 / 2.69 |

Summary: chat maximum 9.3-9.6 min before, 6.6 / 6.5 / 6.8 min in A / B /
C. Server maximum 11.7 min before, 11.5 / 11.8 min in A / B; the server
worker setting is unchanged, so its run-to-run spread is mostly runner
type (m8i.xlarge shards took 8.9-9.8 min, m6a.xlarge 11.9-13.0 min).

Runners: A all m7i.xlarge; B m8i/m6i; C m6a/m6in/m8i/m5a. In C, the five
server shards on m5a.xlarge summed 2,545-3,037 s of file time against
about 1,700 s elsewhere; three reached the 20-minute job limit and two
failed tests on 15 s timeouts. The server lane settings in this PR are
the same as master's, so this is a fleet finding: m5a.xlarge cannot run
a server shard inside the job limit (follow-up for the fleet
configuration, outside this PR).

Flakiness of the final setting: the nine chat jobs in A, B and C had no
failure and no retried test. Server shards in A and B had no failure and
no retried test. Before the last commit (single chat collection), three
runs of 43456289623 had chat at 6.7-7.5 min with no chat retries; server
retries there (2 in one run, 1 in another) were on the unchanged
three-worker server setting.

Local checks (correctness only):

- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/ci/test-selection-pr-timings.test.mjs
scripts/ci/test-selection-timings.test.mjs`: 68 passed.
- `pnpm exec vitest run --project @paperclipai/server
server/src/__tests__/vitest-chat-shards.test.ts`: passed.
- `pnpm verify` exits 2 (global CI input changed), so the required CI
checks supply the broad coverage.

## Risks

- Three chat partitions per job use more memory: the peak was 11.4 GB of
16 GB, with at least 4.2 GB free and no memory pressure. CPU was 55-63%
busy.
- Stale chat weights only change balance, never which tests run. Shard
membership is still checked with `vitest list` before each run.
- The weekly refresh now downloads general server logs from merged-PR
runs. Runs that selected a subset of files are skipped before download
because their shard count is smaller.
- The server lane is still above 7 minutes. See the remaining floor
below.

**Remaining floor and shard-count recommendation (for a follow-up under
`.github/`)**

A server shard sums about 1,600 s of file time, and 3 workers on 4 vCPUs
give at most about 2.8 effective concurrency, so the test step is about
560-640 s. Job overhead is about 50 s. A job of 7 minutes needs a test
step of about 360 s, so about 1,000 s of file time per shard. The
current 19,200 s of general server file time therefore needs about 19-20
shards of 4 vCPUs at 3 workers. More workers on the same runners do not
help, because the CPU is already the limit.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
GitHub CLI) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:03 -06:00
DottaandClaude Fable 5.2 0199c963e2 feat(apps): ship the redesigned sign-in step for a pasted MCP server (#21469)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Apps lets an operator connect a remote MCP server by pasting its
address ("Connect your own MCP server")
> - Step 2 of that flow put every sign-in choice in an "Advanced
authentication" box with wrapped pill buttons and a hint paragraph
> - The box hid the choice behind a vague label, and its shape did not
match the access line ("Connects for everyone… Change") right below it
> - The owner approved a redesign: a stated sign-in default with a
Change link, and five options with their fields directly below each
option
> - This pull request wires that redesign into the real screen, removes
the old step, and updates copy that named the removed box
> - The benefit is a screen that reads as two stated answers and one
action, with the same default path (paste the address, press Check link)

## Linked Issues or Issue Description

No public issue exists. Enhancement:

**What existing behavior does this improve?**
The sign-in section of the "Connect your own MCP server" step in Apps.

**Subsystem affected**
UI: Apps connection setup (`ui/src/features/connections`). Small copy
changes in the server OAuth client registration message and docs.

**Current behavior**
Step 2 shows an "Advanced authentication" collapsible with five wrapped
pill buttons and a hint paragraph below them. The fields for the chosen
mode appear further down. Guidance banners tell the operator to look
"under Advanced authentication". The page also shows the "Unverified
server" badge twice (header and body). The stepper reads "Pick app · Add
your key", even when the server needs no key.

**Proposed behavior**
Step 2 states the sign-in choice in one line ("Paperclip asks the server
how to sign in.") with a Change link. Change shows five radio cards: Let
Paperclip check, Key or token, Browser sign-in, Custom headers, No
sign-in needed. The fields for the selected option are directly below
it. Done collapses the list to a one-line summary (for example "Signs in
with 1 custom header."). The list stays open while the chosen option
still needs a value. Client ID and secret are behind "Use your own
client ID" unless they have values or the server asked for them. The
stepper reads "Pick app · Connect".

**Reason and benefit**
The choice is visible as a stated default in the same shape as the
access line. Each option keeps its fields next to it. No copy points at
a box that does not exist.

**Breaking changes**
None. The five auth modes and their request payloads do not change.

**Additional context**
Related open PRs change server behavior for the same flow. They are not
duplicates. Refs #15192, Refs #15450.

## What Changed

- `ui/src/features/connections/GenericMcpConnectStep.tsx`: new step
component (same props as the old `LinkConnectStep`). It uses stable
field ids `generic-mcp-key`, `generic-mcp-client-id`,
`generic-mcp-client-secret` and a `data-testid` on the sign-in
Change/Done link.
- `ui/src/features/connections/ConnectionSetupFlow.tsx`: render
`GenericMcpConnectStep`; delete `LinkConnectStep`,
`GENERIC_AUTH_MODE_OPTIONS`, `SegmentedOption` and unused imports;
change the pasted-link stepper label from "Add your key" to "Connect"
(`STEP_LABELS` is used only by the pasted-link flow; curated apps keep
their own `["Add your key"]`).
- `ui/src/pages/apps/generic-mcp-connect.ts`: `oauth_challenge` and
`oauth_manual_client_required` guidance now say "below" instead of
"under Advanced authentication".
- `server/src/services/tool-access/oauth/client-registration.ts`: the
manual-client message no longer names the removed box.
- `doc/connections/GENERIC-REMOTE-MCP.md`: describe the sign-in line,
Change, and the five options.
- Comments in `packages/shared` and `server/src/services/tool-access.ts`
reworded. `tests/runner-e2e/source-sdk-public-inputs.json` regenerated
for the two shared files (blob hashes only).
- `ui/storybook/stories/connection-setup/byo-mcp-sign-in.stories.tsx`:
14 stories for the shipped component; annotation is now `current` /
`production`.
- Tests: `AppsConnect.test.tsx` generic MCP block uses the sign-in
toggle and radio options; new tests for the collapsed summary, the
open-while-empty-key rule, and a single "Unverified server" label. New
`GenericMcpConnectStep.test.ts` unit-tests `genericMcpAuthSummary`.

Before: "Advanced authentication" box with pill buttons and a hint;
fields below the pills; two "Unverified server" badges; stepper "Pick
app · Add your key".
After: one sign-in line with Change; five radio cards with fields under
the selected card; one "Unverified server" badge in the header; stepper
"Pick app · Connect". Storybook: `byo-mcp-sign-in--choosing`,
`byo-mcp-sign-in--server-asked-for-key`.

## Verification

- `cd ui && pnpm exec vitest run src/pages/apps/AppsConnect.test.tsx
src/pages/apps/generic-mcp-connect.test.ts
src/features/connections/GenericMcpConnectStep.test.ts
src/features/connections/ConnectionSetupFlow.architecture.test.ts
src/features/connections/ConnectionSetupFlow.oauth.test.ts`: 5 files,
300 tests pass.
- `cd ui && node ../scripts/run-tsc.mjs -b`: pass.
- `pnpm -s typecheck` in `packages/shared` and `server`: pass.
- `pnpm check:token-gates`: all gates clean.
- `node scripts/storybook-catalog.mjs`: pass (generated indexes not
committed).
- `pnpm verify`: exit 2 (broader verification required, because
`doc/connections/GENERIC-REMOTE-MCP.md` is a global CI input). Required
CI supplies that coverage. The focused files above were rerun on the
final revision after the rebase.
- Headless Playwright against the Storybook dev server: the Default,
Choosing and Server-asked-for-key stories render the new step, the new
guidance copy, and the "Pick app · Connect" stepper.

## Risks

- Low risk. UI change on one screen; request payloads
(`genericConnectPayload`) do not change.
- Tests that click the first collapsed "Change" button now skip the
sign-in toggle, because the access line also has a "Change" button.
- The server message for `oauth_manual_client_required` changes text. No
code matches on the text; the client uses the error code.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless Playwright).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:35:49 -06:00
DottaandClaude Opus 5.5 e9d7895648 fix(recovery): start an overdue saved-work recovery instead of looping on no pending path (#21478)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run stops before its sandbox files are saved, Paperclip holds
the task and recovers the files first. A recovery queue does this work.
> - The queue runs two attempts at a time for the whole instance. A
remote attempt can take many minutes.
> - On staging, 29 recoveries waited with 0 attempts. The oldest waited
since 15:01Z. One task waited 3 h 33 min for its single attempt.
> - During that time the task said "Recovering saved work…", but the
assigned-work invariant saw no pending path. The sweep sent a
continuation wake every few minutes. The hold refused each wake, and the
activity log showed "could not start yet".
> - This pull request makes the queued recovery the task's pending path
for a bound. After the bound, the sweep starts the recovery itself
instead of sending a wake that the hold refuses.
> - The benefit: a held task gets its files recovered and continues
without a person, and the sweep stops its refused-wake loop.

## Linked Issues or Issue Description

**What happened?**
A scheduled task ran a legacy retry on a Daytona sandbox. Setup failed
with `setup_failed` ("remote_workspace_operation_timed_out: capture
(configured timeout 300000ms)"). Paperclip held the task for saved-work
recovery (`legacy_execution_requires_reconciliation`, resolved, replay
blocked, system recovery `scheduled`, 0 attempts). After 45 minutes the
recovery had not started. The task showed "Recovering saved work…" with
no Retry. The assigned-work sweep logged `issue.continued_automatically`
with outcome `not_started` and violation `no_pending_path` at 17:58,
18:02, 18:10 and 18:25 UTC.

**Expected behavior**
The recovery proceeds by itself and the task continues. If the recovery
cannot finish, the task shows a Retry that works. "Recovering…" never
shows with nothing behind it.

**Steps to reproduce**
1. A legacy run with a retained remote workspace fails during setup, and
its saved-work hold is adopted by system recovery.
2. The recovery queue does not reach the hold (it has two slots, and
they are busy).
3. The assigned-work sweep checks the task 10 or more minutes later. It
finds `no_pending_path` and sends a wake. The hold refuses the wake.
This repeats until the continuation cap.

The regression test
`server/src/services/saved-work-recovery-overdue.test.ts` reproduces
this state on real rows.

**Paperclip version or commit**
`master` at `c6301c92ff` (staging, 2026-11-13).

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with a Daytona sandbox on the
legacy runtime.

## What Changed

- `assigned-work-invariant.ts`: new pending intent
`saved_work_recovery`. A saved-work recovery that the queue still owes
counts as the task's pending path while an attempt runs, while its next
attempt is scheduled, or for `SAVED_WORK_RECOVERY_START_BOUND_MS` (10
minutes) after it became due.
- `assigned-work-invariant.ts`: new violation
`saved_work_recovery_overdue`, used when that bound passes. It comes
before the `no_pending_path`, `retry_offered` and blocked checks, so the
sweep never sends a wake that the hold refuses.
- `assigned-work-sweep.ts`: the sweep loads the fact with the same
eligibility predicate the recovery worker uses. For the overdue
violation it calls `startSavedWorkRecovery` and records no continuation
activity.
- `workspace-recovery.ts`: `startOverdueWorkspaceRecovery` starts one
attempt outside the queue's two slots, with at most two such attempts
per process. It skips runs that are already in flight and uses the same
ownership lock and attempt bookkeeping as the queue. The queue skips
runs that this path started. `recoverRunAction` moved to module scope.
Its "Retained workspace remains pending" warning now logs the error.
- `heartbeat.ts` and `recovery/service.ts`: wire the dependency.
- The sweep returns the end of the bound as the task's re-check
deadline, so the overdue start does not depend on another notification.

## Verification

- New `server/src/services/saved-work-recovery-overdue.test.ts` (real
Postgres, recovery fixture with a mocked sandbox):
- Overdue hold: the sweep starts the recovery and does not call
`enqueueWakeup`. No continuation activity is written. The system
recovery reaches `complete`. The next sweep records a fresh
continuation, and `deliverReconciledExecutions` starts a successor run.
- Hold within the bound: verdict `ok / pending_intent /
saved_work_recovery`. No wake is sent and no attempt starts.
- A second start for the same run returns `in_flight`. Exactly one
attempt is recorded, and a completed hold returns `not_eligible`.
- Red on the old invariant and sweep, with `no_pending_path` and
`enqueue_refused`, the same as staging. Green with this change.
- `server/src/services/recovery/assigned-work-invariant.test.ts`: 54
pass, with new cases for the bound, a running or scheduled attempt, a
stale attempt, and a pending question.
- `server/src/__tests__/assigned-work-invariant.test.ts`: 31 pass.
- `server/src/services/workspace-recovery-passive-fairness.test.ts`:
passes.
- `tsc --noEmit -p server/tsconfig.json`: clean.

## Risks

- A sweep-started attempt can run beside the queue's two attempts. That
adds up to two more concurrent recovery attempts per process, each with
its own database connection while it holds the ownership lock. Each
attempt for a run still takes that run's advisory lock, so two servers
cannot run the same recovery at once.
- The 10-minute bound is new. A queue that is slow but working now gets
help after 10 minutes. It does not get the hours it took on staging.
- Out of scope, reported for follow-up: why the queue was so slow on
staging (two slots for the whole instance, attempts that take minutes),
and why the setup capture hit its 300 s timeout. Server logs were not
available from this session.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, browser reads of the staging API, local test
runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:26:26 -06:00
d018a9d1cd fix(queue): retain chat bindings on resolved card continuations (#20825)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task can receive a card response while its agent runs.
> - The queue saves this response for the next turn.
> - Chat authorization requires the saved inbound message binding.
> - Queue promotion removed this binding when a prior turn received the
message.
> - This pull request retains live bindings for resolved card responses.
> - The next turn can check current chat access and continue the
approved work.

## Linked Issues or Issue Description

**What happened?**

An accepted plan card queued behind a chat run could start with a chat
source and no message IDs. Reviewed chat authorization then failed with
context_mismatch before provider work started.

**Expected behavior**

A resolved card must retain its live chat binding through queue
promotion. The new run must check current access. A consumed
message-only wake must not start another turn.

**Steps to reproduce**

1. Start a chat run with an inbound message.
2. Queue a card response with that message binding.
3. Complete the first run and release its queue.
4. Inspect the promoted context and check chat authorization.

Related work: #20689 cancels consumed message-only wakes. #19156
protects a successor checkout. This change retains the binding for an
independent card response.

## What Changed

- Retain delivered message bindings for resolved chat card responses
during queue promotion.
- Keep deleted, foreign-task, and self-authored message filters and
current chat authorization.
- Add a persisted queue regression and Slack/Discord
promotion-to-authorization checks, including access revocation.
- Document the queue behavior.

## Verification

- Before the fix, the persisted queue test failed because the promoted
context had no wakeCommentIds. Three existing tests passed.
- The first focused run passed 108 tests. The added Slack and Discord
checks passed after promotion and refused revoked access.
- Final head a38bcb9bc5: 110/110 focused
tests pass. Low-memory server typecheck passes. pnpm verify exits 2 and
requires broad hosted CI. CI typecheck, build, all
E2E/general/serialized tests and Runner checks pass except the Runner
support suite and its aggregate gate. The support suite fails on
inherited stale SDK lock pins (source-sdk-public-type,
source-sdk-evidence and source-sdk-public-inputs); existing PR #20623
owns that repair and has passing CI. No retry is appropriate for this
deterministic failure. Greptile reviewed this exact head at 5/5 with no
actionable findings; there are no inline or unresolved review threads.
- The hosted failed task independently resumed. No source task replay or
deployment was performed. Fixed-release hosted provider acceptance
remains unverified.
- Preflight installed dependencies but could not build Rust binaries
because Cargo is absent. The first server typecheck ended without a
diagnostic; the low-memory retry passed.

## Risks

An overly broad exception could deliver consumed input twice. The
exception applies only to resolved card responses with a chat source.
Message-only wakes still cancel. Message existence, task scope, author
filters, execution ownership, and current chat access still apply.

## Model Used

OpenAI GPT-6 Codex agent with reasoning, repository tools, and code
execution. The runtime did not expose a more specific model build ID.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:12:02 -06:00
Devin Foley b4861d5e37 fix(diagnostics): report Daytona acquisition state for runs retried as provider outages (#21476)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before an agent starts on a sandbox environment, the host gets an
environment lease. On Daytona, this creates a sandbox.
> - When the create does not finish in time, the Daytona plugin records
the failed phase, the elapsed time, the budget, and the Daytona sandbox
state. #15736 and #15946 added these fields to the run failure report,
so that operators can see where the acquisition stopped.
> - #20857 changed the error code of these failures from `setup_failed`
to `environment_provider_unavailable`, so that the host retries them.
> - The run failure diagnostics read the acquisition fields only when
the error code is `setup_failed`. Thus every Daytona create timeout now
reaches the error report without its phase or provider state.
> - On staging, more than 270 Daytona lease timeouts occurred in three
days, and none of them shows the provider state.
> - This pull request also reads the acquisition fields when the error
code is `environment_provider_unavailable`.
> - The benefit is that the error report shows again whether the sandbox
was stuck in `creating`, in a snapshot state, or absent. This evidence
is necessary to find the cause of the timeouts.

## Linked Issues or Issue Description

Refs #20857, Refs #20828, Refs #15946, Refs #15736

**What happened?**
A Daytona lease acquisition timeout is now classified as
`environment_provider_unavailable` (#20857). The run failure report for
this failure does not contain `environmentAcquisitionPhase`,
`environmentAcquisitionElapsedMs`, `environmentAcquisitionBudgetMs` or
`environmentAcquisitionProviderState`.

**What did you expect?**
The report contains the acquisition phase and the provider state, as it
did for `setup_failed` runs before #20857.

**Steps to reproduce**
1. Let a Daytona sandbox create pass its acquisition deadline.
2. The plugin returns "Daytona lease acquisition timed out; allocation
cleanup is pending" with an acquisition diagnostic.
3. The host fails the run with `environment_provider_unavailable`.
4. Look at the `run_execution` context of the error report. The
acquisition fields are not there.

## What Changed

- `collectRunFailureDiagnostics` reads the private acquisition
diagnostic for a setup failure with the error code `setup_failed` or
`environment_provider_unavailable`. The other conditions do not change:
the phase must be `setup`, and the diagnostic must belong to the same
run and company.
- A new test builds the real wrapper chain (lease failure, then provider
failure, then the RPC error that holds the diagnostic). It checks that
the fields are reported for the failed run, and that they are not
reported for another run or for the execute phase.

## Verification

- `cd server && npx vitest run
src/services/__tests__/run-failure-diagnostics-provider-unavailable.test.ts`
fails on master (the report has only `failurePhase`). It passes with
this change.
- `cd server && npx vitest run
src/services/__tests__/run-failure-diagnostics-provider-unavailable.test.ts
src/services/__tests__/run-failure-diagnostics.test.ts
src/__tests__/environment-acquisition-diagnostics.test.ts`: 3 files, 139
tests pass.

## Risks

Low risk. The change adds four bounded scalar fields to an error report.
These fields already exist for `setup_failed` runs. The change does not
alter retries, the error grouping, or the run result.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in
an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-11-13 12:04:25 -08:00