Commit Graph
3015 Commits
Author SHA1 Message Date
Nicky LeachandClaude Opus 5.5 c603f591dc fix(runner): bound ACPX instructions only where the sandbox delivers them (#21646)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner prepares a private sandbox for each ACPX agent before it
opens the provider session
> - `validateProviderPolicy` in that sandbox stage refuses system
instructions over 32 KiB for every ACPX agent
> - Only Pi gets the text from the sandbox stage. Claude, Codex, Grok
and Antigravity do not use it there. The host accepts 256 KiB
> - An agent with a large instruction bundle fails every run at
`session.open`. The error says only `classification=admission_sandbox`,
so the operator cannot see the cause
> - This pull request applies the 32 KiB sandbox bound only to Pi, and
gives every instructions rejection its own closed admission code
> - The benefit is that Claude and Codex agents with large instructions
can start, and an over-bound rejection names its real cause

## Linked Issues or Issue Description

Fixes: #20814

Refs #14430 (this pull request added the 32 KiB check, with no stated
reason for the other providers)

Related: #21644 (also fixes #20814, but only for Claude, and does not
change the classification)

**What happened?**
Every run of a Claude ACPX agent failed with
`provider_session_open_rejected`. The message was `ACPX sidecar command
session.open was rejected (retryable=false,
classification=admission_sandbox)`. A reproduction in a sandbox from the
same image showed that all the filesystem steps of
`prepareAcpxRuntimeSandbox` succeed. The real cause: the agent's
instructions entry file was 54,734 bytes. That is more than the 32 KiB
check in `validateProviderPolicy`. The host bound (`boundedInstructions`
in `runtime-host.ts`) is 256 KiB. Agents with smaller instructions (12.7
KB) work.

**Expected behavior**
The sandbox stage must refuse only the text that it delivers. #14430
added the check when only Pi consumed `systemInstructions` in the
sandbox (as the `PAPERCLIP_PI_SYSTEM_INSTRUCTIONS` env var). The check
runs before any per-provider branch, so it also refuses Claude, Codex,
Grok and Antigravity. Those providers get the text in the session
request. When a bound refuses the text, the error must name the
instructions, not the sandbox.

**Steps to reproduce**
1. Create a Claude ACPX agent whose composed instructions are more than
32 KiB (for example, an `AGENTS.md` of 54 KB).
2. Start a run of the agent on the Paperclip Runner.
3. See that `session.open` fails with
`classification=admission_sandbox`.

**Paperclip version/commit**
`master` at 163d9f323c.

**Deployment mode**
Paperclip Runner with ACPX providers (any deployment).

Where each provider consumes `systemInstructions` today:

| Provider | Delivery | Bound |
| --- | --- | --- |
| Pi | `PAPERCLIP_PI_SYSTEM_INSTRUCTIONS` env var (sandbox) | 32 KiB in
`validateProviderPolicy` (kept) |
| Cursor | `PAPERCLIP_CURSOR_INSTRUCTIONS` env var (sandbox) | 32 KiB in
`cursorInstructionBinding` |
| Kimi | `PAPERCLIP_KIMI_INSTRUCTIONS` env var (sandbox) | 32 KiB in
`kimiInstructionBinding` |
| Copilot | `copilot-instructions.md` (host, after the sandbox stage) |
32 KiB in `composeCopilotSystemInstructions` and
`refreshCopilotSystemInstructions` |
| Claude, Codex, Grok, Antigravity | session request
(`systemPrompt.append` or the provider's own session admission) | host
bound, 256 KiB |

## What Changed

- `runtime-sandbox.ts`: `validateProviderPolicy` gets the agent. It
applies the 32 KiB bound only to Pi. The type and NUL checks stay for
all agents. Cursor, Kimi and Copilot keep their own 32 KiB checks.
- `admission-diagnostics.ts`: new closed admission code
`ACPX_ADMISSION_INSTRUCTIONS`, and `acpxInstructionsAdmissionError`,
which tags the error with that code. The tag wins over the stage code
that the host records later.
- The instructions bound and NUL rejections in `runtime-sandbox.ts`,
`runtime-host.ts` (`boundedInstructions`), `cursor-instructions.ts`,
`kimi-instructions.ts` and `copilot-profile.ts` use the new tagged
error. The messages contain no instruction text and no provider output.
- `acpx_sidecar_transport.rs`: `ACPX_ADMISSION_INSTRUCTIONS` maps to
`classification=admission_instructions`. The classification set stays
closed.
- `antigravity-connection-probe.ts`: the closed classification pattern
accepts `admission_instructions`.
- `copilot-installation.test.ts`: the v70 identity test compared two
historical source hashes with the live files. Since #18325, Runner
source changes do not advance profile identities. The test now pins the
hashes that the v70 record holds.

## Verification

- Red-green: the new tests failed on `master` (11 failed) and pass with
the fix.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/acpx/runtime-sandbox.test.ts
src/drivers/acpx/runtime-host.test.ts
src/drivers/acpx/kimi-instructions.test.ts
src/drivers/acpx/cursor-instructions.test.ts
src/drivers/acpx/copilot-profile.test.ts
src/drivers/acpx/copilot-runtime-host.test.ts
src/drivers/acpx/pi-acp-runtime.test.ts src/cli`: 29 files, 489 tests
pass.
- New tests:
- The sandbox admits 54,734-byte instructions for Claude, Codex, Grok
and Antigravity, and does not put them in the launch environment.
- The host opens a Claude runtime with 54,734-byte instructions and
passes the full text to the provider.
- Pi and Cursor sandboxes refuse more than 32 KiB with
`ACPX_ADMISSION_INSTRUCTIONS`. NUL also gives that code.
- Through the host and the sidecar error mapping
(`acpxSidecarErrorCode`), oversized Pi (32 KiB + 1), Copilot (32 KiB +
1) and Claude (256 KiB + 1) instructions give
`ACPX_ADMISSION_INSTRUCTIONS` before the provider launches.
- The Rust classification test maps `ACPX_ADMISSION_INSTRUCTIONS` to
`admission_instructions`.
- The Antigravity probe logs `admission_instructions` as a closed
classification.
- `cargo test -p paperclip-runner-core --lib
classifies_only_allowlisted_internal_sidecar_failures` passes. `cargo
fmt --check` passes.
- `tsc -p packages/paperclip-runner/tsconfig.json --noEmit` passes.
- `cd server && npx vitest run
src/__tests__/antigravity-connection-probe.test.ts`: 11 tests pass.
- I did not do a live provider run. I did not run the full local suite.
CI supplies that coverage.

## Risks

- Low risk. The bound gets less strict only for providers that do not
get the text from the sandbox stage. Those providers stay under the host
bound (256 KiB). Pi, Cursor, Kimi and Copilot keep 32 KiB.
- An over-bound rejection now reports `admission_instructions` and not
`admission_sandbox` (or `admission_configuration` for Copilot). A
dashboard or alert that matches the old classification for this case
must also match the new one. Server code that checks a specific
classification does not check these values, except the Antigravity
probe, which this pull request updates.
- The NUL rejection message in `validateProviderPolicy` changes from
"exceed their bounded size" to "are invalid".

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:40:20 -08:00
DottaandClaude Opus 5.5 6ba46fd080 fix(github): read only the webhook secrets each delivery step uses (#21651)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents receive GitHub work through a GitHub App chat channel. GitHub
sends each event as a signed webhook delivery.
> - Each delivery passes three steps: durable staging, the durable
replay worker, and the replayed request's preflight under the credential
lease.
> - Each step resolved every stored GitHub App secret. A GitHub App
connection stores seven secrets, so each delivery decrypted and audited
about 21 secrets. The steps use only the signing secret, the bound
installation id and the App id.
> - On our staging instance, these reads are now the largest source of
secret reads and `secret_access_events` rows.
> - This pull request makes each webhook step read only its own keys,
and only when it needs them.
> - The benefit is fewer vault reads and audit inserts per delivery, and
the App private key and OAuth client secret stay sealed on the webhook
path.

## Linked Issues or Issue Description

Refs #21641 (fixed the other source: review-check polls now reuse cached
installation tokens).

**What happened?**
On our staging instance (serving 163d9f323c), from 2026-11-14 20:23:52Z
to 20:32Z, 32 `github_webhook_ingress` chat actions arrived. In the same
window, `secret_access_events` recorded 97 resolutions of
`credentials.webhookSecret`. It recorded the same count for
`privateKey`, `appId`, `installationId`, `clientId`, `clientSecret` and
`slug`. That is about 3 full credential resolutions per delivery, and
about 21 audit rows per delivery (roughly 70–80 rows per minute). In the
same window, 84 review-check polls caused no secret reads, because
#21641 fixed that source.

**What did you expect?**
A delivery reads only the secrets it uses. An unsigned or forged
delivery reads only the signing secret.

**Steps to reproduce**
1. Configure a GitHub App chat endpoint.
2. Send one signed `issue_comment` delivery for the bound installation.
3. Count the new `secret_access_events` rows for the connection. Before
this change, each stored key has 3 rows. A bad signature still reads
every key once.

The three call sites per delivery:
1. `server/src/services/chat-channels/github/webhook-ingress.ts`
`stageGitHubWebhookIngress` (`resolveCredentialRefs` with all refs).
2. `server/src/services/chat-channels/github/webhook-ingress.ts`
`processGitHubWebhookIngress` (`resolveCredentialRefs` with all refs, to
re-sign the retained body). This was the unattributed third resolution.
3. `server/src/services/chat-channels/github/webhook-authentication.ts`
preflight and ping (`resolveCredentials` with no key filter).

## What Changed

- Added `githubCredentialRefsFor(refs, keys)` in the GitHub webhook
ingress module. It selects the secret refs for named keys.
- Staging reads `webhookSecret` first. After the signature holds, it
reads `installationId` (only when the event names an installation and is
not an `installation` event) and `appId` (it scopes the recovery-receipt
denial query).
- The replay worker reads `webhookSecret` to re-sign the body. It reads
`appId`, `privateKey` and `installationId` only for a recovered delivery
that needs a source recheck.
- The preflight and ping read `webhookSecret` through the existing
`resolveCredentials` key filter. The preflight reads `installationId`
only after the signature holds and only for an event that names an
installation.
- The failed-delivery recovery scan reads only `appId`, `privateKey` and
`installationId`.
- Unchanged: the timing-safe compare, the credential-mutation lease, the
re-read of current endpoint state under the lease, the status and fence
checks, and the superseded-credential retry. No cache of decrypted
values.
- Per valid delivery, secret reads go from 7 keys × 3 (21 on staging) to
6: `webhookSecret` × 3, `installationId` × 2, `appId` × 1. A bad
signature goes from all keys to 1 read.
- `webhook-ingress.ts` was already above 1,000 lines. It grows by about
30 lines because the key selection belongs with the steps that use it.
`chat-channels.integration.test.ts` holds the GitHub webhook fixtures,
so the new test goes beside the existing signature tests.

## Verification

- Red on master, green with this change: `pnpm exec vitest run
server/src/__tests__/chat-channels.integration.test.ts -t "resolves only
the secrets a"`.
- `current`: before, `{webhookSecret: 3, installationId: 3, appId: 3,
privateKey: 3}` (the fixture stores 4 keys). After, `{webhookSecret: 3,
installationId: 2, appId: 1}`.
- `invalid_signature`: before, every key once. After, `{webhookSecret:
1}`.
- Regression: GitHub webhook, signature, rotation, ping, recovery,
installation, paused and revoked tests in
`chat-channels.integration.test.ts` (filter `-t
"[Ww]ebhook|signature|rotat|[Pp]ing|recovery|installation|GitHub
callback|revoked|paused"`): 304 passed, 1 failed. The failure is a Slack
test (`materializes direct external-chat finals once…`, matched by the
`ping` filter in "bookkeeping"). It passes when run alone, so it is a
load flake and not related to this change.
- `pnpm exec vitest run
server/src/services/chat-channels/github/webhook-ingress.test.ts
server/src/services/chat-github-provider-stress.test.ts
server/src/services/chat-github-webhook-config.test.ts`: 105 passed.
- `tsc --noEmit -p server/tsconfig.json`: passed.
- `pnpm verify` selects 2,726 of 3,554 Vitest files for this change (the
chat integration file pulls in a wide dependency set). I did not run
that broad set locally. Required CI supplies that coverage for the final
revision.

## Risks

- Low risk. Each step reads a subset of the same refs from the same
fenced record, so the values it uses do not change.
- A missing `webhookSecret` ref still fails the signature check as
before. A missing `installationId` ref still makes an
installation-bearing event "ignored" as before.
- The preflight now resolves `webhookSecret` and `installationId` in two
reads under the same credential lease. The lease blocks credential
mutation between them.
- Installation lifecycle events (`installation-lifecycle.ts`) still
resolve the full set, because they re-verify and re-persist all App
credentials. They are rare.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run in Claude Code
with tool use (shell, file edits, test execution) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:06:29 -06:00
DottaandClaude Opus 5.5 9b581d689b fix(chat): report an outcome once and create no run when an earlier reply gave it (#21489)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In agent chat, a finished delegated task or a declined tool action
wakes the chat agent so it tells the person
> - #21401 stopped a second completion report when an earlier reply
already gave it
> - But the suppressed completion still created a zero-second heartbeat
run, and declined tool actions still got a second report
> - Run accounting must count only real turns, and the person must get
each outcome once
> - This pull request settles an already-reported outcome before it
becomes a run, and applies the same rule to declined tool actions
> - The benefit is one report per outcome, no phantom run, and no extra
model call

## Linked Issues or Issue Description

Refs #21401

**What happened?**
Chat eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`)
showed three problems after #21401.
1. `questions-and-approvals.duplicate-answer`: the completion wake was
suppressed, but it still created a 0-second run (`runtimeMode: legacy`,
no usage). The bounded-native-runs check rejected that run.
2. `skills.studio-test`: the CEO's single reply reported the Studio
result after it read the test task done (0 passed, 1 failed, links
COM-2). The completion was then suppressed, but the eval waited for a
separate completion run and timed out after 240 s.
3. `questions-and-approvals.decline-tool`: the person declined the tool
review one second after the turn for their message started. That turn
read the declined action and said so. The decline continuation, which
waited behind that turn, then posted the same declined outcome again.

**Expected behavior**
Each finished task or declined tool action is reported once in the
conversation. An outcome that an earlier reply already reported creates
no heartbeat run.

**Steps to reproduce**
1. In chat, delegate a short task, or start a governed tool read that
needs approval.
2. While it waits, send a message that wakes the chat agent.
3. Let the task finish, or decline the tool review, while that turn
runs. The turn reads the outcome and reports it.
4. See the completion wake become a 0-second run, or see a second
decline report from the decline continuation.

**Paperclip version or commit**
master `df1b12ad8e`

## What Changed

- `server/src/services/chat-completion-delivery.ts`:
`settleReportedChatCompletions` marks each undispatched completion that
an earlier reply reported as delivered, with that reply as its response.
It returns true only when nothing else is owed. The outbox `deliver`
also checks this before it requests a wake, so no wake and no run exist
for an already-reported completion.
- `server/src/modules/wake-queue`: a new transaction port,
`skipAlreadyReportedChatWake`, runs in the release drain before a
deferred wake is promoted. When the earlier reply already reported every
outcome of the wake, the wake ends `skipped` ("Already reported by an
earlier reply in this conversation") and no run is created. Wakes that
carry queued comments or a failed-chat retry are never skipped.
- `server/src/services/chat-interaction-outcome-report.ts` (new): the
same structural rule for declined tool actions. A conversation agent run
that lists its interactions records `chat_interaction.declined_read` for
each declined tool action. The decline continuation is settled when the
same agent replied in the conversation after the decline, from a run
with that receipt, posted after the read. The rule covers only declined
tool-action confirmations. Answers, plan reviews and accepted actions
keep their own turn.
- `server/src/routes/issues.ts`: `GET /issues/:id/interactions` calls
the recorder for agent runs.
- The run-start check from #21401 stays as a fallback for a completion
turn that was admitted before the earlier reply landed.
- `tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.ts`:
the Studio oracle (v5) accepts either the completion turn's reply or an
earlier CEO reply as the delivery. The earlier reply counts only from
saved state: no completion run exists, the replying run has a
`chat_handoff.completed_read` receipt for the test task, the reply was
posted after that read, it is the only such reply, and it links the
saved result. No wording check is added.
- `doc/run-log-events.md`: documents both read receipts and the skipped
wake outcome.

## Verification

- `cd server && npx vitest run
src/__tests__/chat-completion-delivery.test.ts
src/modules/wake-queue/adapters/chat-declined-outcome-reported.test.ts
src/modules/wake-queue/application/use-cases.test.ts`: all pass.
- Red: with the drain check and the outbox check disabled, 5 tests fail:
"starts no completion wake or run ...", "settles a deferred completion
wake without a run ... (true)", "settles the decline continuation with
no run ...", and both "settles an already-reported chat wake without a
run" cases.
- Negative controls stay green: a linking reply without a done read, a
decline the earlier turn never read, a non-tool-action card, and an
accepted card all still run their continuation.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.test.ts`:
34 pass. Red on the old oracle: "accepts the earlier reply the server
took as the completion report" fails. New negative cases reject no done
read, a read of another task, a reply before the read, a reply from
another run, two replies, an extra completion run, and a missing task
link.
- A narrow paid eval run of `duplicate-answer`, `studio-test` and
`decline-tool` on this branch follows in a comment.

## Risks

- A completion turn admitted before the earlier reply landed (a narrow
race) still ends as a short succeeded run with no model call, as in
#21401.
- If an earlier reply read the outcome but did not mention it, the
person does not get a second message. The read receipt, the
reply-after-read order and, for completions, the task link make this
unlikely.
- The decline rule covers only declined tool-action confirmations. Other
interaction outcomes are unchanged.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:16 -06:00
DottaandClaude Opus 5.5 8a97037bd2 fix(runs): say who started a retry run in the issue run read (#21486)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat agent answers questions about failed runs by reading the
task's run list (`GET /api/issues/{id}/runs`)
> - For a retry run, that read returned only raw invocation fields
(`on_demand`, `manual`) and an owner id. It did not say who started the
retry
> - In chat-eval cohort 13, the agent told the person "Paperclip retried
automatically" about a run that the board started with Retry
> - This pull request adds a plain-words `retriedBy` field to each retry
run, and tells the chat agent to use it
> - The benefit is that people get a true account of who retried their
work, with a name and no raw ids

## Linked Issues or Issue Description

**What happened?**
Chat-eval cohort 13 ran OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`.
In `runs-adapters.inspect-failed-run`, the reply said "Paperclip retried
automatically on a supported model". The successful run was a person's
manual Retry: `invocationSource: on_demand`, `triggerDetail: manual`,
wake reason `retry_failed_run`, and the responsible user was the board.
The agent's own reasoning hedged ("probably automatic"), because the run
read did not say who started the retry.

**Expected behavior**
The run reads that the agent uses say who started each retry: a person,
by display name, or Paperclip's automatic recovery, in plain words and
with no raw ids.

**Steps to reproduce**
1. Let a run fail on a task.
2. Press Retry on the failed run as a board user.
3. Read `GET /api/issues/{id}/runs`. Before this change, nothing in the
retry row says that a person pressed Retry.

## What Changed

- `runsForIssue` (`server/src/services/activity.ts`) reads the wake
request of each retry run. It adds `retriedBy` to each row:
- `{ kind: "person", personName, summary }` when a user wake with reason
`retry_failed_run` or `task_retry` started the run. The name comes from
the user record, or "Board" for the local board, with the same fallback
as activity actor names.
  - `{ kind: "automatic", summary }` for an `automation` retry run.
- `null` for a run that is not a retry, or a retry whose starter is not
recorded. The read does not guess. It does not use `responsibleUserId`,
because automatic retries keep the original responsible user.
- The OpenAPI description of `GET /api/issues/{id}/runs` and the chat
agent directive tell the agent to use `retriedBy` and not to call a
retry automatic unless `retriedBy` says so.
- The UI `RunForIssue` type has the new optional field.

## Verification

- `cd server && npx vitest run src/__tests__/activity-service.test.ts
src/__tests__/agent-conversations.test.ts
src/__tests__/activity-routes.test.ts
src/__tests__/openapi-routes.test.ts` passes.
- Red/green: the new test "says who started each retry run" fails
without the service change (`expected undefined to be null`) and passes
with it. It covers a board Retry, a named member's Retry, an automatic
retry that keeps the board as responsible user, a retry with no recorded
wake, and no raw ids in the output.
- `npx tsc --noEmit -p server` passes.
- A narrow paid run of `runs-adapters.inspect-failed-run` (with the two
sibling cohort-13 PRs) ran on a combined branch. The results are in a PR
comment.

## Risks

- Low risk. The change adds one read-only field and one bounded query by
primary key on the wake requests of retry runs in the page.
- No persisted state changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:04 -06:00
DottaandClaude Opus 5.5 95a6ab31e6 feat(db): attribute work signals by caller behind a debug flag (#21640)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Background queues wake from work signals. A transaction calls
`signalDatabaseWork(topic, hint)`, and the delivery coordinator wakes
the queue for that topic with that hint.
> - On a hosted instance with zero runs, the stranded-assignment
recovery queue re-checks tasks about 7 times a second. About 2.6 hinting
commits per second drive it. HTTP traffic, chat review polls and local
fixtures with all heartbeat workers do not explain the writer.
> - Statement statistics show what runs, but not who signalled it.
Nothing records the caller of a work signal or why a queue woke.
> - This pull request adds an opt-in attribution log for work signals
and recovery-queue wakes, off by default.
> - The benefit is that one short run on an affected instance names the
writer of any idle signal loop, without code changes.

## Linked Issues or Issue Description

No issue exists. This describes the gap with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

Observability of `signalDatabaseWork`
(`packages/db/src/work-signals.ts`) and of the issue recovery queues
(`server/src/services/recovery/work.ts`).

**Current behavior**

A queue that wakes several times a second at idle shows up only as
repeated statements. Nothing tells you which caller published the hint,
whether the wake came from another process, a reconnect, a direct
settled publish or a stored deadline, or which load and check it caused.

**Proposed behavior**

An environment variable, `PAPERCLIP_DEBUG_WORK_SIGNALS`, read once when
the module loads. It holds a comma-separated topic allowlist, and each
topic can carry `:N` for its stack sampling rate. When it is unset or
empty, the signal path only checks one boolean and does nothing else.

**Reason and benefit**

Several idle-load loops this week took hours of statement analysis
before the writer was found. Attribution by caller turns that into one
log query.

## What Changed

- `packages/db/src/work-signal-debug.ts` (new):
- `parseWorkSignalDebugSpec` reads `topic[:N],…`. An optional numeric
suffix is parsed from the right, so topics that contain colons (for
example `live:<companyId>`) work with or without `:N`. N defaults to 20;
malformed entries are ignored.
- `createWorkSignalDebug` returns `enabled`, `allows(topic)`,
`signal(topic, hint, entry)`, `event(topic, name, fields, stack, entry)`
and `flush()`. A disabled instance constructs no `Error`, captures no
stack, builds no line and starts no timer.
- Stack capture (`captureWorkSignalStack(entry)`) raises
`Error.stackTraceLimit` to 40 and restores it in a `finally`. It calls
`Error.captureStackTrace(holder, entry)`, so V8 cuts every frame above
the public entry (`signalDatabaseWork`, `publishDatabaseWorkSettled`, or
the recovery `wake()`).
- Frames are dropped by file (`node_modules`, `node:internal`, the
signal wrapper files) and by function name (`signalDatabaseWork`,
`notifyDeliveryWork`, …), so trimming works on the bundled `index.mjs`.
Positions are kept raw (`index.mjs:L:C`) for offline source-map lookup.
At most 10 frames are kept.
- Each (topic, event) pair has its own budget of 10 lines per second,
checked BEFORE any stack is captured. A `suppressed` line names the
topic, the dropped event and the count. Frequent `check` lines therefore
never drop `signal` or `direct_wake` lines.
- Stack sampling (1 in N) counts per (topic, event), so `check` and
`load` do not shift the phase of `signal` sampling. `direct_wake` always
captures its caller's stack.
- Lines go to an in-memory buffer that an unref'd 1-second timer flushes
to stderr. The signal path does no I/O and no await.
- `workSignalDebug` is the process instance, built from
`process.env.PAPERCLIP_DEBUG_WORK_SIGNALS` once at module load.
- `packages/db/src/work-signals.ts`: when enabled, it records:
  - `signal` in `signalDatabaseWork`;
  - `remote_notification` and `notification_reconnect`;
- `settled_publish` (with a sampled stack) for
`publishDatabaseWorkSettled`;
  - `uncertain_settled` when an uncertain transaction resolves.
- `server/src/services/recovery/work.ts`: under the same flag:
  - each task-queue `load` logs its scope key;
- each `check` logs the issue and company id, `dueAt`, and `wake:
"deadline"` or `"load"`. The marker rides on the queued entry itself,
and only while the flag is on, so nothing is tracked otherwise and
nothing outlives the entry;
- `wake()` logs ONE `direct_wake` line listing the queues it actually
invalidates (assigned-work only when registered), under the first
allowlisted one, with its caller's stack.
- **No request route is logged.** The server has no request async
context to read one synchronously, and this change does not add one.
- `docs/deploy/environment-variables.md`: documents the flag, internal
ids only, raw positions, and that enabling or changing it requires a
restart.
- Tests:
  - `packages/db/src/work-signal-debug.test.ts`, 12 tests:
- strict no-op when unset or empty (no `Error`, no `captureStackTrace`,
no stack capture, no sink);
    - the process instance is disabled without the env;
    - allowlist filtering, and spec parsing including colon topics;
    - per-(topic, event) sampling, and always-on direct-wake stacks;
- source-tree trimming, and a synthetic single-file bundle stack with
raw positions kept and wrappers dropped by name;
    - real capture below the entry with the limit raised and restored;
- checks at about 8/s alongside signals at about 3/s for 5 seconds, with
every signal line kept;
    - suppressed records never capture a stack;
    - buffered, timer-driven output.
- `packages/db/src/work-signals.test.ts`: on real PostgreSQL, the signal
and settled-publish paths never call the debug log while the flag is
off.
- `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`,
on the recovery fixture with real PostgreSQL:
    - no recovery log calls while the flag is off;
- with the flag on, a run graph hint logs its signal, the graph-scoped
load and the check;
- `wake()` logs exactly one `direct_wake` that lists the queues and
carries the caller's stack.

## Verification

- `packages/db`: `vitest run src/work-signal-debug.test.ts
src/work-signals.test.ts`: 39 passed. `tsc --noEmit` passes.
- `server`: `vitest run
src/__tests__/issue-recovery-work-debug-attribution.test.ts
src/__tests__/issue-recovery-deadlines.test.ts`: 17 passed. `tsc
--noEmit` passes.
- Staging values:
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:10`, then restart.
- Sample output from the synthetic single-file bundle test:

```
{"msg":"paperclip work signal debug","at":1000000,"event":"signal","topic":"stranded-assignments","hint":"{\"companyId\":\"c\",\"graphRunId\":\"r\"}","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","at async notifyRunDispatch (file:///app/server/dist-runtime/index.mjs:120501:3)","at async PostgresJsDatabase.connection.transaction [as transaction] (file:///app/server/dist-runtime/index.mjs:4021:22)","at async Object.settleRunProjection (file:///app/server/dist-runtime/index.mjs:300777:5)","at async file:///app/server/dist-runtime/index.mjs:310002:17"]}
{"msg":"paperclip work signal debug","at":1000000,"event":"check","topic":"stranded-assignments","issueId":"i","companyId":"c","wake":"load","dueAt":1}
{"msg":"paperclip work signal debug","at":1000000,"event":"direct_wake","topic":"stranded-assignments","queues":"stranded-assignments,orphan-blockers","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","…"]}
{"msg":"paperclip work signal debug","at":1001000,"event":"suppressed","topic":"stranded-assignments","suppressedEvent":"check","count":17}
```

## Risks

- Low risk. It is off by default. While off, each signal costs one
property read and one boolean check.
- While on, the log can show internal ids from hints (company, issue,
run). It holds no secrets and no user identifiers, and the topic
allowlist limits what is logged.
- Stack capture costs CPU. Sampling (default 1 in 20) and the 10 lines
per second cap bound it.
- The file owner of `packages/db/src/work-signals.ts` reviews this
before merge.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21621.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:54:00 -06:00
DottaandClaude Opus 5.5 163d9f323c perf(github): reuse repository installation tokens until shortly before expiry (#21641)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel lets an agent review pull requests and
answer comments through a GitHub App
> - Each GitHub call gets a repository-scoped installation token from
`githubBotRepositoryToken`
> - That function minted a new token on every call. To mint, it resolved
every App secret through the secret service, and each resolution wrote
one audit row
> - The review-check head poll calls it about every 5 minutes for each
waiting check. On staging this made the secret audit insert one of the
busiest statements in the database
> - This pull request keeps a minted token in memory until shortly
before it expires, and checks the connection's current state on every
call
> - The benefit is far fewer database writes and reads, fewer GitHub
token mints, and an audit log that records real secret use

## Linked Issues or Issue Description

**What happened?**
On paperclip.staging (commit a18479bbba, measured at 19:05Z on
2026-11-14), `secret_access_events` got about 285 rows in 60 s. These
were 44 resolutions each of the GitHub App connection's
`credentials.appId`, `privateKey`, `installationId`, `webhookSecret`,
`clientId` and `clientSecret` (+21 `slug`), with `consumer_type`
`tool_connection` and actor `system`. That is about 4.7 audit inserts/s.
The matching `secret_versions` and binding reads ran at about 3.6/s each
in Neon. These were among the busiest statements on the database.

**Expected behavior**
A GitHub installation token is valid for one hour. Paperclip should
reuse it and resolve the App secrets only when it mints a new token.

**Steps to reproduce**
1. Connect a GitHub App chat endpoint and get pull request reviews into
the `received` review-check state (staging had 84 `github_review_check`
actions).
2. Let the review-check worker poll the pull request head (23 polls/min
measured on staging).
3. Count `secret_access_events` rows per minute. Each poll adds one row
per App secret and mints a new token at GitHub.

**Paperclip version or commit**
master at a18479bbba (the commit serving paperclip.staging at the time
of measurement).

**Where in the code**
`githubBotRepositoryToken` in
`server/src/services/chat-github-client.ts`. Callers:
`chat-github-checks.ts` (review-check head poll),
`chat-github-response-comments.ts` and `chat-github-reviews.ts`.

## What Changed

- `githubBotRepositoryToken` keeps a minted token in a per-process,
in-memory cache (at most 512 entries, least recently used first out). It
never persists or logs tokens.
- The cache uses the token until 5 minutes before GitHub's `expires_at`
(capped at one hour). If GitHub does not give a usable expiry, the token
is not kept.
- The cache key holds the company, endpoint, repository id, the exact
permission set, and an authority revision. The revision holds the
connection id, the endpoint's runtime generation (pause, revoke and
availability changes advance it), and for each mint credential its
secret ref, version selector, secret status, current version,
`updatedAt`, and binding. The installation id is a secret, so its secret
ref and version identify it in the key.
- Each call reads the current authority from the database in two narrow
statements: one row from the endpoint and connection join (status,
enabled, refs, runtime generation), and the three mint secrets with
their bindings (status and version columns only, no secret values). The
endpoint and connection status checks run on every call, as before.
- If a status check fails, all cached tokens for that endpoint are
dropped. A token cannot survive a revoke, pause, disable or credential
change.
- Secrets are resolved only when a token is minted, and only the three
that minting uses (`appId`, `privateKey`, `installationId`). So the
audit log gets three rows per mint instead of one row per App secret per
call.
- New `githubBotRepositoryClient` sends repository requests. If GitHub
returns 401 for a cached token, it drops that token, mints a new one,
and sends the request once more. A 401 for a newly minted token is not
retried. The review-check, response-comment and review paths use this
client.
- The token policy stays in the GitHub chat client module
(`rules/module-boundaries.mdc`). The callers only change how they get a
request function.

## Verification

- New test file
`server/src/__tests__/github-repository-token-cache.test.ts` (embedded
Postgres, real secret service, mocked GitHub fetch):
- `mints once and resolves the App's secrets only when it mints`: red on
master (second call minted `installation-token-2`), green now. It also
checks that only the three mint secrets are audited.
- `refreshes a token shortly before GitHub's expiry`: reused at +54 min,
minted again at +56 min.
  - `never keeps a token GitHub issues for less than the safety margin`
- `mints again after a credential rotation and uses the new credential`:
rotates the installation id secret and checks that the new mint uses it.
  - `mints again after the connection's credential refs change`
- `never serves a cached token across a pause, revoke or disabled
connection`: includes a pause and resume with no call in between.
  - `mints a separate token for each repository`
- `replaces a cached token GitHub rejects and retries the request once`:
also checks that a 401 for a new token is not retried.
- On master, 5 of the 8 tests fail. All 8 pass with this change.
- Focused suites pass locally: every server test file that matches
`github`, plus `chat-sdk-runtime.adapters`, `chat-provider-inventory`,
`idle-completed-history` and `connection-management.integration` (41 of
42 files, 955 of 956 tests). The one failure,
`github-connection-enrollment-work.test.ts`, also fails on unchanged
master (`expected +0 to be 1`), so this change did not cause it.
- `pnpm --filter @paperclipai/server typecheck` passes.
- CI supplies the full typecheck, build and test coverage for the final
revision.

## Risks

- Low risk. A cached token is used only while the endpoint, connection
and mint secrets are unchanged, and the status checks still read current
database state on every call.
- A GitHub-side revoke of a token (for example, an App uninstall) makes
GitHub return 401. The client then mints again once. If the installation
is gone, the mint fails as before.
- The cache is per process. Each server process mints its own token, at
most once per hour per repository and permission set.
- Two concurrent first calls for the same key can both mint. This is the
same as before and is rare.
- No schema, migration or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:42:06 -06:00
DottaandClaude Opus 5.5 a18479bbba perf(workspaces): start a full terminal workspace pass only at startup and on the hourly backstop (#21630)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep walks every candidate workspace in a
full pass. On staging that is about 4,000 finished remote workspaces,
and each one costs a persistence probe and a work product read.
> - #21617 limited full passes to startup and an hourly backstop that
only activity arms. But three other paths in the worker still started a
full pass at once:
>   - a wake whose hints do not parse to workspace or task targets;
>   - more than 50 pending targets;
>   - any run that finds no targets.
> - Across one staging deploy the probe count rose by about 2.8 full
passes in 26 minutes. At 1,000 concurrent agents, more than 50
run-finish hints between passes would be normal, so the overflow path
alone would bring back passes proportional to history.
> - This pull request works through targets in batches of 50 and turns
hintless wakes into the backstop. It gives the one caller that needs a
broad look, a workspace provider plugin becoming ready, its own hint
that inspects only plugin workspaces.
> - The benefit is that a full pass happens only at startup and at most
once per backstop hour, whatever the activity.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On staging, the hourly steady state holds (one full pass per hour, no
probes between). But across a deploy the probe count rose by 11,351 in
under 26 minutes, about 2.8 full passes over the roughly 4,000 finished
remote workspaces.

**Steps to reproduce**

The new test file
`server/src/__tests__/terminal-workspace-rotation-limits.test.ts` runs
the real coordinator and worker against real PostgreSQL on a fake clock,
with finished remote workspaces that cannot be archived (the staging
state). These are the counts on master. The numbers are workspaces
checked by full passes; a pass over N workspaces checks N:

| Scenario | Full-pass checks on master | Expected |
| --- | --- | --- |
| A burst of 120 run-finish `issue:` hints, 300 workspaces | 600 (2
extra passes) | 0 |
| One hintless wake and one unparseable hint, 120 workspaces, 30 minutes
| 1,490 (about 12 passes) | 0, then 1 at the backstop |
| A workspace provider plugin becomes ready (hintless notify from
`plugin-registry`), 123 workspaces | 123 (1 pass) | 0; only the 3 plugin
workspaces |
| Restart: old instance drains while hints arrive, new instance starts
and gets the plugin hint and 120 run-finish hints, 202 workspaces, 30
minutes | 2,424 (12 passes) | 202 (the startup pass) |

**Cause of the passes across a deploy:**
- The new instance runs its startup pass.
- Each workspace provider plugin that turns `ready` sends a hintless
notification (`plugin-registry.ts`), which started a full pass.
- Run recovery at startup finishes many runs, and their `issue:` hints
overflow 50, which started another full pass.
- Any later wake without targets also started a full pass, including the
coordinator's 5-minute retry timer. In the fixture this produced two
back-to-back passes every 5 minutes.
- The draining old instance starts nothing, because the worker's
`canRun` is false during drain. The test confirms this.

**Expected behavior**

A full pass runs only at startup and when the activity-armed backstop is
due, at most once per hour. Targets are worked through in targeted
batches.

## What Changed

- `server/src/services/terminal-workspace-work.ts`:
- **Pending targets.** Workspace and task targets go into pending sets
with no cap. Each run inspects at most 50 of them, and the rest stay
queued for the next run, which is scheduled at once. A failed targeted
run puts its targets back.
  - **Hintless or unparseable wakes** only arm the hourly backstop.
  - **No targets, nothing due:** the run does nothing.
- **Due deadlines:** the due workspaces are queued as targets, including
isolated re-home and nested clone deadlines keyed by a workspace. A due
non-workspace deadline (the isolated source orphan sweep, task directory
work) runs a targeted sweep for its prelude and arms the backstop.
- **A failed plugin workspace listing** restores the plugin refresh
flag, so the coordinator's retry lists again.
- **Full passes** happen only at startup, when the backstop is due, and
on continuation pages of a pass already under way.
- **New hint `plugin-workspaces`** (`TERMINAL_WORKSPACE_PLUGIN_HINT`):
it queues the ids of plugin-provider candidate workspaces from
`listPluginTerminalWorkspaceIds`.
- The worker now always tells the service `targeted: true`, so a hint
during a pass no longer requests a rescan lap.
- `server/src/services/execution-workspaces.ts`:
- New `listPluginTerminalWorkspaceIds` reads the plugin-provider
candidates.
- A targeted sweep reads all matches of its at most 50 targets, with no
page limit and no rescan request.
- A failed targeted sweep no longer requests a rescan lap, because the
worker re-queues its targets.
- `server/src/services/plugin-registry.ts`: the two workspace-provider
notifications send `plugin-workspaces` instead of no hint.
- `server/src/__tests__/terminal-workspace-work.test.ts`: a new test,
`retries the plugin workspace listing after it fails instead of waiting
for the backstop`, fails without the retry fix. Three existing tests now
describe the new behaviour:
  - unscoped commits go to the backstop;
  - more than 50 hinted workspaces are inspected in batches of 50 and 1;
- a due deadline on a service without targeting still runs a full pass.
- `server/src/__tests__/terminal-workspace-rotation-limits.test.ts`:
four new tests, the scenarios above.

The startup pass stays. The 1,000-workspace test in
`terminal-workspace-deadlines.test.ts` shows that durable deadlines keep
an idle instance quiet after startup. But only the startup pass
reconciles workspaces whose hints were lost while no instance was
running, so removing it would need a stored "last full pass" marker
first.

## Verification

- The four new tests fail on master with the counts in the table and
pass after the change:
- 0 full-pass checks for the burst, worked as targeted batches of at
most 50 (120 checks in total);
- 0 full-pass checks for 30 minutes after hintless and unparseable
wakes, then 1 pass at the backstop;
  - only the 3 plugin workspaces are inspected after the plugin hint;
- exactly 1 full pass across the restart: none on the draining instance,
only the startup pass on the new one.
- Prompt archiving and the missed-hint backstop keep their existing
tests in `terminal-workspace-deadlines.test.ts`:
  - the release of a holding run inspects its workspace at once;
- a run made terminal outside `writeRunStatus` is archived within 1 hour
1 minute;
  - 1,000 finished remote workspaces stay unread at idle.
- `npx vitest run` in `server/` on `terminal-workspace-rotation-limits`,
`terminal-workspace-work`, `terminal-workspace-deadlines`,
`terminal-workspace-idle`, `terminal-workspace-page-batch`,
`execution-workspaces-service`, `plugin-workspaces.integration`,
`workspace-operations-reconciliation`, `execution-workspace-reopen`,
`isolated-task-directory-retention`, `git-workspace-expiration` and
`workspace-delivery-cleanup.integration`.
- `pnpm check:module-boundaries` passes.

## Risks

- **A notification that relied on an immediate broad pass now waits up
to an hour.** The only caller I found that sent such a notification on
purpose is the workspace provider readiness in `plugin-registry.ts`, and
it now has its own targeted hint. Other hintless notifications arm the
backstop.
- **The isolated source orphan sweep** runs only at the start of a full
pass. Its due deadline now arms the backstop instead of starting a pass
at once. This only matters when the isolated source re-home feature is
enabled.
- **A targeted sweep has no page limit.** Its at most 50 targets bound
the matched workspaces to those tasks, their ancestors and their
execution workspaces.
- **A burst of N hints costs N/50 targeted runs, back to back.** The
cost scales with activity, not with history.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 12:19:43 -06:00
DottaandClaude Opus 5.5 890dcbdd66 fix(native): wait for a live warm session owner during a deploy overlap (#21627)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap: the outgoing server still
serves while the incoming server starts new runs
> - A warm agent session stays open in the server process that ran the
last turn, after that run finished
> - The incoming server took over a warm session while the outgoing
server was alive and still held it, because the finished run's
finalization lease ends at commit
> - The takeover then failed, the task went blocked with a Retry, and in
one case a retry loop ran 27 times on the same sandbox
> - This pull request makes the incoming server wait while the server
that holds the warm session is alive, makes the outgoing server close
idle warm sessions as soon as it leaves new runs to the replacement, and
stops the retry loop when the promised fresh sandbox does not come
> - The benefit is that a follow-up during a deploy finishes without a
person, and no Runner ever gets two controllers

## Linked Issues or Issue Description

**What happened?**
A zero-downtime release-swap test (Gate B, wave B1, master aaec67b891)
found two failures with one cause.
Task 1: run A finished on the outgoing server at 15:13:56. Its warm
session stayed idle in that server. The incoming server claimed the
follow-up at 15:13:57. `inspectRemoteWarmSessionRecovery` asked
`evaluateNativeControllerTakeover`, which allowed the takeover with
`remote_controller_lease_released_or_expired`. That lease covers the
finished run, not the warm session. The outgoing server was alive until
15:18:18. `runner.prp.authenticate` failed after 62.3 s, the run failed
with `remote_warm_session_unproven`, the failure was saved as ambiguous
and owned by the board, and the task stayed blocked with a Retry.
Task 2: the same shape on a later swap started 27 automatic runs in 13
minutes. Fourteen of them said "The next attempt starts on a fresh
sandbox", but each one resumed the same sandbox and got the same refusal
(`journal_unsettled`).

**Expected behavior**
A follow-up claimed during an overlap waits for the warm session owner
to let go, then continues with no Retry banner and no second controller.
If the outgoing server dies with the session open, the takeover
continues after its ownership ends. A message that promises a fresh
sandbox is true, or the retries stop after a bounded number with a true
state.

**Steps to reproduce**
1. Run the restart-chaos release-swap preset with a warm (opencode or
codex) staging mix.
2. Let a run finish on the outgoing server after the traffic switch,
with a follow-up comment already queued.
3. See the follow-up fail with `remote_warm_session_unproven` and the
task go blocked.

**Paperclip version or commit**
master aaec67b891 and 45531b719e.

**Deployment mode**
Hosted deploy overlap (release swap) with Daytona sandboxes.

## What Changed

- `remote-warm-session-recovery.ts`: before live adoption, the run
proves that the server process that ran the finished turn is gone. It
uses the shared owner proof (`proveOwnerGone`): a PID check in the same
process space, or the `controller_instances` lease in another process
space. While that server is alive, the run defers with the existing
`native_controller_recovery_waiting_for_owner` wait and checks again
after 5 s. The wait ends when the server exits or is fenced, or when the
finished turn's warm window (idle timeout plus two minutes) ends.
- `native-session-executor.ts` and `run-start-handover.ts`: a server
that leaves new run starts to a replacement closes its idle warm
sessions after 1 s, through the existing idle-timeout close. A session
that becomes idle after that closes after 1 s too. This is the same
close that shutdown already does, done earlier.
- `predecessor-retirement-fallback.ts`: when a retry meets an unretired
owner on the same sandbox that its source already asked to replace, it
does not ask again. The run fails without another automatic retry, and
the task shows a Retry.
- `predecessor-retirement.ts`: the error no longer says "The next
attempt starts on a fresh sandbox", because the replacement is only a
request.
- `doc/execution-semantics.md`: documents the owner wait, the hand-off
at the swap, and the retry stop.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts`
- New: "waits while the outgoing server that ran the finished turn still
holds its warm session, then takes it over once that server exits" (red
on master: master returned a takeover while the owner was alive).
- New: "waits for a live outgoing server in another process space until
its controller lease ends" (red on master).
- New: "stops waiting for a live outgoing server once the finished
turn's warm window has closed" (bound check).
- `cd server && npx vitest run
src/__tests__/heartbeat-native-predispatch-retry.test.ts`
- New: "stops retrying with a truthful failure when the next attempt met
the same unretired sandbox instead of a fresh one" (red on master:
master scheduled another retry on the same sandbox).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"replacement server starts"`
- New: "closes a warm session promptly when a replacement server starts
this server's runs (before/after the turn ends)" (both red on master).
- Focused suites: native-session-executor, remote-warm-session-recovery,
native-controller-recovery, native-recovered-owner-followup,
native-runner-restart-recovery, heartbeat-native-predispatch-retry,
environment-runtime-reusable-resume-replacement,
task-interruption-presentation, sandbox-runner-port-holders.
- `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify` exits
2 (a doc is a global CI input), so required CI supplies broad coverage.
- Not run here: the restart-chaos release-swap wave. The maintainer runs
it after merge.

## Risks

- A follow-up can now wait up to the outgoing server's remaining overlap
when that server cannot close its idle session. Before, it failed and
needed a Retry. The wait is bounded by the owner's exit or the warm
window.
- The outgoing server closes idle warm sessions earlier during an
overlap. The next turn then resumes from a suspended Runner instead of a
live one, as it already does after a shutdown.
- A retry chain that meets the same unretired sandbox twice now stops
one attempt earlier than the shared cap. The task shows a Retry instead
of looping.
- Rows without a recorded process space keep the existing one-release
local proof, so they behave as before outside the same host.
- Previous-release state: this change adds no persisted field.
Finalization rows saved by the previous release already record the
controller boot, PID, process space and start time, so they carry
forward. A follow-up already deferred by the previous release resumes
through the same wait code.
- Runtime cost: the owner proof adds one `ps` start-time read (same
process space) or one `controller_instances` row read (other process
space) per inspection. It runs once per attempt, at most every 5 s while
a live owner holds the session.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:59:17 -06:00
DottaandClaude Opus 5.5 45531b719e perf(recovery): stop re-asserting a failed legacy run whose decision is recorded (#21621)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Stranded-work recovery checks tasks that are assigned to an agent
but have no live run. It runs as a scoped-deadline queue that a change
notification or a stored deadline wakes.
> - For a failed legacy run, the check calls
`terminalizeLegacyExecution` with the run's current status. When the
run's recovery decision is already recorded (for example "continue in a
fresh run from saved work"), that call changes nothing. But it still
updated the run row and notified about 18 delivery queues, including the
run's own recovery graph.
> - The graph notification re-checked the same task, which re-asserted
the same run again. The loop ran about once a second per waiting task,
forever, with zero runs active. On a hosted instance this shows up as
constant idle transactions.
> - This pull request makes a re-assertion that changes nothing return
before it writes or notifies.
> - The benefit is that an idle instance does no stranded-recovery work
for tasks that wait on a recorded decision. A real change (another run
fails, a comment, a status change) still notifies and recovers at once.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

With zero runs active, the server runs many transactions per second. A
CPU profile groups most samples under `reconcileStrandedAssignedIssues`
(`server/src/services/recovery/service.ts`). A local fixture on real
PostgreSQL reproduces it: a task in `in_progress`, assigned to an agent,
whose latest legacy run failed and whose recovery decision is recorded.

Each check reaches the branch that terminalizes a failed legacy run that
needs reconciliation (`service.ts`, the
`legacyExecutionNeedsReconciliationWithEvidence` branch, and the same
call in `enqueueStrandedIssueRecovery`). `terminalizeLegacyExecution`
then:
- updates the run with the same status (only `updated_at` changes);
- calls `notifyExecutionRunWork`, `notifyInstructionCollection` and
`notifyUnsafeWorkspaceExportRecovery`, which hint the run graph of the
task and about 18 queues;
- finds the recorded decision and returns.

The graph hint re-checks the same task after the queue's 1-second pass
interval, so the loop never ends.

**Expected behavior**

A check that finds nothing to change stores nothing and wakes nothing.
Only a real change re-checks the task.

**Steps to reproduce**

1. Create a task in `in_progress` assigned to an agent, with a failed
legacy run (`adapter_failed`).
2. Start the issue-state recovery work. The boot pass records a
fresh-continuation decision for the run.
3. Leave the instance idle. On master: 591,045 statements in 30 minutes
for 10 such tasks (fake clock, real PostgreSQL), and one no-op `update
"heartbeat_runs"` per task per second.

**Paperclip version or commit**

master at 7db92ea319.

## What Changed

- `server/src/services/legacy-execution-recovery.ts`:
- `terminalizeLegacyExecution` returns the unchanged run, without a
write or a notification, when all of these are true: the call re-asserts
the current status with no patch; the run has no workspace restore
failure or required workspace recovery; the task's execution and
checkout locks do not point at the run; and either the run needs no
reconciliation (`reconcileIfNeeded`), no assignee or reviewer owns a
decision, or the decision is already recorded.
- Why the short-circuit is safe: stranded-work and quota checks
re-assert the run's unchanged terminal status each time they revisit its
task. Once the decision is recorded, that re-assertion changes nothing.
If it wrote or notified anyway, the notifications would re-check the
same task, which would re-assert the run again, about once a second for
as long as the task waits.
- `findRecordedLegacyDecision` returns the resolved decision already
recorded for the run (a reconciliation for this run, an explicit user
continuation with a real successor, or a no-replay hold).
`decisionOwner` says whether the task's assignee or current reviewer
owns a decision about the run.
- The recorded-decision query moves into `findRecordedLegacyDecision`,
and the owner condition into `decisionOwner`. The full path uses both
helpers, so its behavior does not change.
- `server/src/__tests__/issue-recovery-recorded-decision-idle.test.ts`:
two new tests on real PostgreSQL. They use the real coordinator, the
real issue-state recovery work and the production 5-minute backstop on a
fake clock.

## Verification

- `stays quiet for half an hour once each failed run has its decision`:
10 waiting tasks, zero runs. The test first checks that each failed run
has a recorded decision, then counts statements over 30 idle minutes.
  - master: 591,045 statements (red).
- this branch: 0 statements (green). No run is re-written or started,
and no task moves.
- `still recovers promptly when another run of a waiting task fails`:
after 10 idle minutes, a new legacy run of one task fails and notifies
like a finished run.
- this branch: within 5 seconds, recovery creates the run's
`legacy-execution:<run>` hold. Within the next 6 minutes, the
fresh-continuation fallback resolves that hold with
`executionReconciliation.runId` set to the new run. Over the next 30
minutes there is no `update "heartbeat_runs"` and fewer than 5,000
statements (the assigned-work invariant keeps its own spaced re-checks).
- master: 18,000 no-op `update "heartbeat_runs"` statements in the same
window (red).
- Neighbouring suites pass locally: `legacy-workspace-restore-recovery`,
`legacy-workspace-restore-lock-contention`,
`legacy-continuation-authority`, `heartbeat-process-recovery`,
`issue-recovery-assignments`, `recovery-deadline-ordinary-bootstrap`,
`issue-recovery-deadlines` (600 of 601 in one parallel run; `resolves a
run's task graph through index lookups` failed once under machine load
on a planner row count, and passes alone; it does not use the changed
code).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The short-circuit applies only when the full path would also
write nothing except `updated_at`. Every other case (status change,
patch, workspace restore evidence, an owned lock, no recorded decision)
takes the unchanged full path.
- A re-assertion no longer re-sends notifications. Queues that relied on
that repeated hint as a side effect lose it. The first terminalization
of a run still sends all notifications, and run, comment and status
writes still send their own.
- No schema or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21552, #21616, #21617.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:30:03 -06:00
DottaandClaude Opus 5.5 7db92ea319 perf(workspaces): read a sweep page's task trees and active runs in two statements (#21618)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep inspects candidate workspaces 50 per
page. For each workspace it read the task tree with its own recursive
query. For many of them it also ran one or two queries to check for an
active run.
> - The cost of a page therefore grew with the page: about 50 to 110
statements. On the hosted instance, building the SQL text of these
statements costs about as much main-thread time as parsing their
results.
> - The archive statement re-checks the terminal tree and the absence of
a live run in the same statement, and the remote and plugin cleanups
re-check active runs under the lifecycle lock. So these early reads are
only pre-filters.
> - This pull request reads the task trees and the active-run workspaces
of a whole page in two fixed-text statements.
> - The benefit is that a page costs a constant number of these reads,
whatever its size.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The per-page reads of the terminal workspace sweep
(`sweepTerminalWorkspaces` in
`server/src/services/execution-workspaces.ts`).

**Current behavior**

Each candidate calls `listWorkspaceIssueTree`, one recursive query. A
candidate that reaches the active-run checks calls
`workspaceHasActiveRun`: one query for the linked tasks' run ids, then
one for live runs.

**Proposed behavior**

- `readPageIssueTrees` reads the tree rows of every root on the page in
one statement. Its SQL text is fixed: the roots are one `jsonb`
parameter. The trees are then built in memory and keep the same row
shape.
- `readPageActiveRunWorkspaces` returns, in one fixed-text statement,
the page workspaces that have a `queued` or `running` run on a linked
task.
- The two pre-filter checks in the loop use these results. A workspace
whose source task changed earlier in the same pass (rehome) still reads
its own tree.
- The authoritative checks are unchanged: the archive `UPDATE` with its
terminal-tree and live-run predicates, the active-run check inside the
remote and plugin cleanup under the lifecycle lock, and the
per-workspace check before a stale reopen flag is cleared.

**Reason and benefit**

One rotation pass should issue a bounded number of statements per page.
For a page of 50 workspaces (40 with open trees, 10 held by a run), the
tree reads fall from 50 to 1 and the active-run reads from 20 to 1.

## What Changed

- `server/src/services/execution-workspaces.ts`: add
`readPageIssueTrees` and `readPageActiveRunWorkspaces`, and use them for
the per-page pre-filter.
- `server/src/__tests__/terminal-workspace-page-batch.test.ts`: two new
tests on real PostgreSQL.

## Verification

- `reads the task trees and active runs of a page in a fixed number of
statements`: 2 companies, 50 workspaces, each with a child task, 10 held
by a `running` run.
  - Before the change it fails: 50 tree reads.
- After the change: one tree read, one active-run read, fewer than 10
statements for the page. The results are the same (`checked: 50`,
`skippedNonTerminalTree: 40`, `skippedActiveRun: 10`).
- `does not archive a workspace whose run starts after the page was
read`:
- The page is read first. Then, during that workspace's close-readiness
check, a `running` run is inserted and set as its task's checkout run.
- The sweep counts the workspace `eligible: 1`, `archived: 0`,
`skippedRace: 1`, and it stays `active`.
  - After the run succeeds, the next sweep archives it (`archived: 1`).
- `npx vitest run src/__tests__/terminal-workspace-page-batch.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/nested-clone-migration.test.ts
src/__tests__/execution-workspace-compaction.test.ts
src/__tests__/git-restore-point-expiration.test.ts
src/__tests__/plugin-workspaces.integration.test.ts` (in `server/`).
- `pnpm check:module-boundaries` passes.

## Risks

- **The pre-filter reads happen once, at the start of the page.** A page
can take long, because cleanups do remote and Git I/O. A run that starts
mid-page on a workspace the pre-filter saw as idle is still caught by a
check made at the destructive step itself, in the same statement or
under a lock. Line numbers are on this head:
- **Archive** of an active workspace: the archive `UPDATE` in
`sweepTerminalWorkspaces` carries `NOT EXISTS (… live_run.status IN
('queued', 'running'))`
(`server/src/services/execution-workspaces.ts:4291`). It changes 0 rows
and the workspace counts as `skippedRace` (`:4340`).
- **Remote cleanup** (destroying the sandbox):
`runTerminalWorkspaceCleanup` re-runs `workspaceHasActiveRun` under the
lifecycle lock before `cleanupRemoteWorkspaceAllocation` (`:2702`).
- **Plugin workspace removal**: the same check under the lifecycle lock
before `removePluginWorkspace` (`:2706`).
- **Git cache expiration** and **isolated task directory expiry**:
`isGitWorkspaceExpirationEligible` re-reads the tree and refuses any run
that is not finished, inside the locked transaction
(`server/src/services/git-workspace-expiration.ts:311` and `:416`,
`server/src/services/isolated-task-directory-retention.ts:244` and
`:290`).
- A run that ends mid-page leaves its workspace for the run-release hint
or the next inspection.
- **Tree rows are now deduplicated** (`UNION` instead of `UNION ALL`).
Parent cycles therefore cannot repeat rows.
- Low risk otherwise. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 08:51:28 -06:00
DottaandClaude Opus 5.5 081dacba1d perf(workspaces): inspect only due workspaces and wait for a holding run's release (#21617)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep stores one deadline per workspace:
cooldown end, compaction, cleanup retry, Git idle.
> - When any deadline was due, the worker ran a full rotation over every
open candidate workspace. That costs a task tree read and more for each
one.
> - A workspace held by a `running` run kept a past deadline, so the
5-minute deadline rotation came back for it forever. A run that never
finishes would keep the instance awake indefinitely.
> - This pull request makes a due deadline inspect only the due
workspaces. A held workspace waits for its run instead of for a timer.
> - The benefit is that an idle instance does work only for workspaces
whose deadline really arrived. The cost no longer grows with the number
of open workspaces.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

How the terminal workspace worker
(`server/src/services/terminal-workspace-work.ts`) acts on due deadlines
and on workspaces held by an active run.

**Current behavior**

`hasPending` reads `nextTerminalWorkspaceSweepAt`. When the earliest
deadline is past, the next run starts a full keyset rotation, 50
workspaces per page with 1 second between pages, over every candidate. A
workspace whose issue still has a `queued` or `running` run is skipped,
but the archive deadline set earlier in the same pass stays. It is
already past, so the rotation repeats every 5 minutes.

**Proposed behavior**

- When the run fires for a due deadline, the worker takes the due
workspace ids from the service (`takeDueTerminalWorkspaceTargets`) and
runs a targeted inspection of only those workspaces. The service first
clears the deadlines of the targeted workspaces; the inspection sets
fresh ones.
- If more than 50 workspaces are due, or a due deadline belongs to the
isolated source, nested clone or task directory work, the worker still
runs a full rotation. Startup still runs a full rotation.
- A workspace skipped because a run holds it drops its deadline.
- `writeRunStatus` (`server/src/services/heartbeat/run-lifecycle.ts`)
publishes `issue:<runIssueId>` to the terminal workspace queue when a
run reaches a terminal status. That targeted inspection reaches the
workspace the run held.
- **Backstop.** About 200 code paths write `heartbeat_runs` directly
(reapers, recovery terminalization, bulk cancels), and they do not pass
through `writeRunStatus`. So the worker keeps a full rotation as a
backstop, armed only by activity, like the recovery backstop from
#21552:
  - Any terminal-workspace hint arms it.
- Once armed, a full rotation runs at most once per hour
(`TERMINAL_WORKSPACE_BACKSTOP_MS`) after the previous full rotation.
  - An idle instance with no hint schedules nothing.
- A missed release hint therefore delays archiving by at most one hour
after activity, without a restart.
- The hour is a compromise. On master today, the 5-minute rotation
re-reads every candidate. On the hosted instance that is about 1,000
finished remote workspaces, roughly 3,000 statements per lap. The hourly
backstop re-reads them at most once per hour while there is activity,
about 1 statement per second averaged.

**Reason and benefit**

Background work must be driven by deadlines and change notifications,
and its cost must not grow with the backlog. Every guarantee keeps a
named trigger:
- cooldown end, compaction, retry backoff and Git idle each have a
stored deadline, and a due deadline inspects exactly that workspace;
- the release of a holding run publishes a hint;
- task status changes and workspace writes still publish hints;
- startup still reconciles everything.

## What Changed

- `server/src/services/execution-workspaces.ts`:
  - New `takeDueTerminalWorkspaceTargets()`.
- A targeted inspection clears the deadlines of its targets before it
inspects them.
  - A workspace skipped for an active run drops its deadline.
- `server/src/services/terminal-workspace-work.ts`: a due run uses the
due targets instead of a full rotation, unless they overflow or a
non-workspace deadline is due. A due run that finds nothing due does
nothing. Any hint arms the hourly backstop rotation.
- `server/src/services/heartbeat/run-lifecycle.ts`: a terminal run
status hints the run's task to the terminal workspace queue.
- `server/src/__tests__/terminal-workspace-deadlines.test.ts`: four new
tests on real PostgreSQL. They use the real coordinator and worker on a
fake clock.
- `server/src/__tests__/terminal-workspace-idle.test.ts` (from #21616):
its idle window now bounds the number of sweeps instead of requiring
zero targeted sweeps, because a due deadline is now a targeted
inspection.

## Verification

- `inspects only the workspace whose cooldown ends, however many trees
stay open`: 60 workspaces with open task trees and 1 done workspace
whose 1-day cooldown ends 1 hour later.
- Before the change: after startup, every due deadline starts a full
rotation (27 full sweeps in 2 hours).
- After the change: no full rotation; the done workspace is inspected
alone (`checked: 1`) once its cooldown ends; fewer than 200 statements
in 2 hours.
- `waits for the release of the run that holds a workspace instead of
rechecking it`: a done workspace held by a `running` run.
- Before the change: a full rotation every 5 minutes (22 sweeps in 2
hours).
- After the change: 0 statements in 2 idle hours. When the run is set to
`succeeded` through the lifecycle, the workspace is inspected (`checked:
1`, `skippedActiveRun: 0`).
- `leaves a thousand finished remote workspaces unread at idle`. This is
the hosted state: 1,000 `active` remote workspaces with a location, a
committed snapshot and a task done 5 days ago, and zero runs.
- After startup and 2 idle hours: no full rotation. No remote workspace
is read again; the only inspections are of the one due local workspace.
Statements stay within 10 per inspection.
  - On master, every due deadline re-reads all 1,000.
- `archives a workspace whose run ended outside the run lifecycle within
the activity-armed backstop`:
- A workspace held by a `running` run. The run is set to `failed` by a
direct `UPDATE`, so no release hint is published, and one unrelated hint
arrives.
- The workspace is archived within one hour and one minute, without a
restart.
  - With the backstop scheduling disabled, it stays `active`.
- `npx vitest run src/__tests__/terminal-workspace-deadlines.test.ts
src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/nested-clone-migration.test.ts
src/__tests__/execution-workspace-compaction.test.ts
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 333
tests pass.
- `pnpm check:module-boundaries` passes.

Local fixture with #21616 applied as the base, 30 idle minutes on a fake
clock, real PostgreSQL. Each fixture holds open trees, a stranded
`running` holder, cooldown, done local workspaces with missing files,
closed remote workspaces with and without a committed snapshot:

| Workspaces | Before: sweeps | Before: statements | After: sweeps |
After: statements |
| --- | --- | --- | --- | --- |
| 140 | 20 (full rotations) | 1,371 | 21 (targeted only) | 866 |
| 420 | 84 (full rotations) | 6,597 | 33 (targeted only) | 3,272 |

After the change, no sweep reads the task tree of a workspace whose tree
is still open (`skippedNonTerminalTree` falls from 1,440 to 0 at 420
workspaces). What remains comes only from workspaces with a due retry or
inspection backoff, so it grows with the number of stuck workspaces, not
open ones. Per inspected workspace a pass issues the same 3–6 statements
as before; batching them per page is a separate pull request.

## Risks

- **A run that leaves `running` without `writeRunStatus`** does not
hint. The activity-armed backstop catches it within one hour of
activity, and startup reconciles. This pull request does not decide that
a stranded run is gone; it waits for a status change.
- **A workspace whose deadline is reset to "now"** (a reopen-pending
flag within its grace period, task directory budget overflow) is
inspected alone on each due run. The inspection backoff in
`nextTerminalWorkspaceSweepAt` still spaces these runs.
- **Workspaces without a stored deadline** (for example an undelivered
workspace whose deadline was deleted) are inspected again only when a
hint or the startup rotation reaches them. Before this change they were
also reached by the rotation that a different workspace's deadline
started.

## Open decision

**Should a terminal workspace whose local files are gone be archived, or
should it wait?** This pull request does not change the behaviour; it
leaves the decision to the owner.

Today such a workspace fails its close-readiness inspection and is
reported as `skippedUndelivered`. A non-remote workspace keeps its past
deadline, so the 5-minute inspection backoff re-inspects it again and
again. That costs about 3–4 statements per stuck workspace every 5
minutes (from the local fixture: 255 inspections, 866 statements), and
it never ends while the files stay missing. The cause is typically a
replaced container.

The two options:
- **Archive it.** Nothing remains on disk to keep.
- **Store a "files missing" state.** It is then inspected again only
when a write announces a change.

Either option removes this cost. Archiving also removes the row from
later rotations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 08:23:49 -06:00
DottaandClaude Opus 5.5 20ccf03676 fix(workspaces): settle closed remote workspaces after one confirmed destroy (#21616)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server cleans up closed execution workspaces in a background
sweep (`terminal-workspaces`). For a remote workspace, cleanup destroys
the sandbox lease after it confirms that the saved snapshot is
committed.
> - A closed remote workspace stays a sweep candidate while
`metadata.remoteWorkspace.location` is an object. A confirmed destroy
did not clear that location.
> - The cleanup write goes through the lifecycle gateway, and the
gateway published a `workspace:<id>` hint for the same workspace. That
hint started a new inspection, which cleaned the workspace again and
published again.
> - On a hosted instance with no runs, this loop kept three statements
at about 15–24 per second each and a dedicated lock connection per
inspection. Every pass also asked the sandbox provider to destroy a
sandbox that was already destroyed.
> - This pull request clears the location after a confirmed destroy,
skips the provider call when the stored lease already has a destroy
receipt, and stops the sweep from hinting itself for its own writes.
> - The benefit is that an idle instance with closed remote workspaces
settles: each workspace is cleaned once, and the provider is called at
most once per lease.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

With no runs, the terminal workspace sweep never became idle. On the
hosted instance these statements ran all the time:
`workspaceHasActiveRun` (22.8/s), the task tree read (22.6/s) and the
snapshot persistence read in `inspectRemoteWorkspacePersistence`
(14.8/s). `SET application_name` for the
`paperclip-workspace-finalization-lock` connections ran at 9.6/s.

**Steps to reproduce**

1. Create closed (`archived`) remote workspaces that have a committed
snapshot, a clean authority and a `location`.
2. Start the terminal workspace worker with no other activity.
3. Watch the sweeps. Each cleanup reports `archived`, the gateway
publishes `workspace:<id>`, and the next targeted inspection cleans the
same workspace again.

Locally, a fixture with 140 workspaces (40 of them in this state) ran
568 sweeps, 136,888 statements and 23,880 lease destroy calls in 30 idle
minutes. The destroys always hit the same 40 workspaces.

**Expected behavior**

A closed remote workspace is cleaned once. After the destroy is
confirmed it leaves the candidate set, and an idle instance issues no
statements between real deadlines.

## What Changed

- `server/src/services/remote-workspace-lifecycle.ts`
(`cleanupRemoteWorkspaceAllocation`):
- Read the stored lease first. When it already carries a destroy receipt
for the same environment, lease and provider lease, do not call
`destroyLease` again.
- After a confirmed destroy, set `metadata.remoteWorkspace.location` to
`null`. One fixed-text `jsonb_set` update does this, and only when the
stored location is still the location that was destroyed.
- `server/src/services/execution-workspaces.ts`: the gateway takes
`sweepOwned`. The sweep's own cleanup, cleanup-failure and reopen-flag
writes set it, and then the gateway does not publish a terminal
workspace hint for that workspace. All other gateway callers still
publish.
- `server/src/__tests__/terminal-workspace-idle.test.ts`: two new tests
on real PostgreSQL.

## Verification

- `settles after one cleanup of each closed remote workspace and
inspects nothing on its own writes`:
- Fixture: 40 workspaces (open trees, a stranded `running` holder,
cooldown, done, closed remote with a committed snapshot, closed remote
without a snapshot).
- Runs the real coordinator and worker on a fake clock for 2 minutes,
then 30 idle minutes.
- Asserts: each closed remote workspace is destroyed exactly once; no
targeted inspection in the idle window; fewer than 5,000 statements in
30 minutes; every cleaned workspace has `location: null`.
- Before the change it fails: 850 destroy calls instead of 10 in the
warm-up alone. After the change it passes.
- `clears the location of an already destroyed lease without calling the
provider again`: the stored lease has a destroy receipt. Before the
change, the sweep calls `destroyLease`. After the change the provider
mock is never called, the location is cleared, and the next inspection
finds no candidate.
- `npx vitest run src/__tests__/terminal-workspace-idle.test.ts
src/services/remote-workspace-lifecycle.test.ts
src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts` (in
`server/`).

Local fixture, 30 idle minutes, 140 workspaces (40 closed remote with a
committed snapshot):

| | Before | After |
| --- | --- | --- |
| Sweeps | 568 (all targeted, continuous) | 20 (one deadline rotation
every 5 minutes) |
| Statements | 136,888 | 1,371 |
| Lease destroy calls | 23,880 | 40 (once per workspace) |
| `workspaceHasActiveRun` reads | 45,440 | 260 |
| Snapshot persistence reads | 22,720 | 40 |

The remaining rotation every 5 minutes comes from deadlines of
workspaces that cannot make progress (a stranded active-run holder,
cleanup backoff). A separate pull request handles it.

## Risks

- **Who reads `remoteWorkspace.location`.** For a closed, clean
workspace with a committed snapshot and a destroyed lease, none of these
readers needs the location:
- the live run binding checks (`remote-workspace-lifecycle`,
`fresh-continuation-fallback`, `workspace-recovery-fallback`,
`workspace-recovery-source`) need a current writer or a dirty run
reference;
- the abandonment and hold paths (`remote-workspace-epoch-abandonment`,
`remote-workspace-abandoned-files-hold`,
`remote-workspace-unchanged-proof`) need a dirty or held allocation;
- `reusable-sandbox-resume-replacement` treats `null` as no conflicting
sandbox;
- `remote-workspace-files` reads the saved snapshot when the location is
`null`. With a destroyed location it would fail as unavailable.
- The lease-release path
(`clearRemoteWorkspaceLocationForReleasedLease`) already writes the same
`null` for the same state. This change makes the sweep stop depending on
that side effect.
- **Provider calls.** Before this change every loop pass re-claimed the
lease and called the driver's `destroyRunLease` again, through
`destroyClaimedSandboxLease` in
`server/src/services/environment-runtime.ts`. After it, a lease with a
stored destroy receipt causes no provider call.
- **Existing backlog after deploy.** The startup rotation and the next
deadline rotation visit each closed remote workspace that still has a
location once, 50 per page with 1 second between pages. A lease that
already has a destroy receipt costs no provider call, and its location
is cleared. Only leases without a receipt get one destroy call. The
workspaces then leave the candidate set.
- **No self-hint.** The sweep already handles the workspace it writes,
and an inspection's deadline bookkeeping is unchanged. Other writers
still publish hints. The git restore point and task directory paths
depend on task or explicit hints, not on these writes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 07:23:35 -06:00
DottaandClaude Opus 5.5 c1451814c1 perf(chat): scope GitHub review check run hints to the issue and throttle company-wide rebuilds (#21610)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel keeps each unresolved "Paperclip Review"
check in step with its task and run. A background queue
(`chat-github-review-checks`) rebuilds its checks when a hint arrives
> - Each run milestone (`notifyChatRunMilestones`) sends this queue a
company-wide hint, also when the milestone knows the issue. Each
company-wide hint rebuilds every unfinished check of the company and
reads its full state again. Passes are limited to one a second
> - So in a company with a steady stream of run milestones, each
unresolved check reads its whole state about once a second, although
nothing about it changed
> - On a hosted test database the five state statements of a review
check ran 10.3 times a second each. #21608 removes the endless 5-second
head poll. This is the second cause
> - This pull request sends the issue with the hint, rebuilds only the
checks of that issue, and rebuilds for a company-wide hint at most once
every 30 seconds
> - The benefit is that run activity no longer makes every unresolved
check of the company read its state each second. A run change of the
check's own task still publishes at once

## Linked Issues or Issue Description

Refs #21608 (backs off the head poll of an unresolved check; the other
cause of the same hosted load). Refs #21580 (the same 30-second rule for
the GitHub response sweeps). Refs #18384 (the same rule for external
objects).

**What happened?**

On a hosted test database with zero active runs, the five statements of
`githubReviewCheckState` (`chat_endpoints`, `chat_github_reviews`,
`chat_message_links`, the `heartbeat_runs` `wakeCommentId` lookup and
`chat_deliveries`) ran 10.3 times a second each. One source is the
rebuild of the review check queue for company-wide hints.

**Current behavior**

`notifyChatRunMilestones` sends `{"companyId": ...}` to
`chat-github-review-checks` for every milestone, also when its scope has
an issue. The queue scope has no issue field. So each milestone rebuilds
every received and retryable check of the company, and calls
`githubReviewCheckState` for each received check.

**Proposed behavior**

A milestone with an issue sends `{"companyId", "issueId"}`. The queue
matches a check to an issue through the issue of its latest review for
the pull request head. This is the same review that
`githubReviewCheckState` reads. A company-wide hint (company ID only)
rebuilds at most once every 30 seconds. All narrower hints, the startup
scope (`*`) and saved deadlines are not changed.

**Steps to reproduce**

1. Admit a GitHub review check that stays unresolved (a review with no
run that the check can find).
2. Send a run milestone for the company once a second (for example,
status changes of other runs).
3. Count statements. Each hint rebuilds the check and reads its full
state.

## What Changed

- `review-check-work.ts`: the scope has an optional `issueId`. `page()`
reads the issue of the latest review for the check head
(`reviewIssueId`, through the existing `chat_github_reviews_pull_idx`
index) and filters on it for an issue scope. `matches()` uses the same
value.
- `review-check-work.ts`: set `reconstructionIntervalMs` to
`githubReviewCheckReconstructionIntervalMs`: 30 seconds for a
company-only scope, 0 for all other scopes.
- `chat-run-milestone-notifications.ts`: send the issue with the review
check hint when the milestone has one.
- `chat-channels.integration.test.ts`: two regression tests on the real
queue, the real `dispatch` and embedded PostgreSQL with a controlled
clock. The test helper now counts only the state read of
`chat_github_reviews`, not the new subquery in `page()`.

## Verification

Red/green regression tests:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "company-wide hints
at most once per interval|at once from its issue milestone"`
- Red (without the fix): `expected [ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ] to
deeply equal [ 1, +0, +0, +0, +0, +0, +0, +0, …(2) ]`. Each of 10
company-wide hints read the check state. The milestone hint was
`{"companyId"}` only.
- Green: 2 of 2 pass. The first company-wide hint reads the state once.
The next 9 read nothing. After 30 seconds the waiting hint reads it
once. A hint for the check's issue reads it at once each time. A hint
for another issue or another company reads nothing. A real
`notifyChatRunMilestones` with the check's issue publishes `in_progress`
on the next pass and resets the head poll interval.

Statement counts for one unresolved check with one company-wide hint
each second (120 simulated seconds, real queue and `dispatch`, embedded
PostgreSQL):

| | Statements a second | Check state reads a second |
| --- | --- | --- |
| Before | 6.0 | 1.0 |
| After | 0.2 | 0.033 |

In the real-coordinator reproduction of #21608, one hint each second
raised one stuck check from 0.70 to 1.67 state reads a second and from
8.4 to 15.2 statements a second.

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: passed,
including all review check queue tests.
- `cd server && npx vitest run
src/__tests__/native-status-reassessment-work.test.ts
src/__tests__/chat-run-milestone-work.test.ts
src/services/native-runtime/external-chat-wait.integration.test.ts
src/services/chat-github-review-repair.test.ts
src/__tests__/idle-completed-history.test.ts
src/__tests__/chat-admission-condition.test.ts
src/__tests__/app-private-hostname-gate.test.ts`: 518 passed.
- `cd server && node ../scripts/run-tsc.mjs` and `node
scripts/typecheck-layers.mjs` (with test layers): passed.
- `pnpm verify --explain` retains 2,718 test files conservatively, so I
did not run it locally. Required CI supplies that coverage.

## Risks

- A change that only a company-wide hint reports can take up to 30
seconds to reach a check. Company-wide hints now come from milestones
without an issue (a run with no task, a comment merged into an active
run) and from company authority changes. A run with no task cannot be
the run of a review check. For a merged comment, the run's next status
change sends an issue hint. The saved head poll deadline still applies.
Each dispatch reads the full state again before it publishes.
- An issue hint finds a check through the issue of its latest review for
the head. If a run of another issue were the check's run, its change
would wait for the next company-wide rebuild (30 seconds at most) or the
next head poll. Run status and run identity milestones use the run's
task issue, which is the review's issue for a GitHub review task.
- The deadline reconciliation snapshot reports the queue as not
reconciled while a company-wide hint waits (up to 30 seconds). This is
the same as the GitHub response sweeps and the external-object refresh
sweep.
- `page()` reads one more indexed subquery for each row. It uses
`chat_github_reviews_pull_idx`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local embedded PostgreSQL
reproductions).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:46:48 -06:00
DottaandClaude Opus 5.5 a7453eba60 perf(chat): back off GitHub review check head polls while the check cannot change (#21608)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel publishes a "Paperclip Review" check for
each admitted pull request head. A background queue
(`chat-github-review-checks`) keeps each unresolved check in step with
its task and run
> - While a check is queued or running, the queue polls GitHub for the
pull request head every 5 seconds. This finds a push whose webhook was
lost
> - The poll never stops while the check is unresolved. A check whose
run never comes, or whose run the check cannot find, stays queued
forever. It re-reads its whole state and calls GitHub every 5 seconds,
also when nothing changes
> - On a hosted test database with zero active runs, the five state
statements of this check ran 10.3 times a second each
> - This pull request doubles the head poll interval for each poll that
finds the same state, up to 5 minutes. A state change resets it to 5
seconds
> - The benefit is that an idle unresolved check costs about one poll
each 5 minutes, not one each 5 seconds. Local changes still publish at
once

## Linked Issues or Issue Description

Refs #17662 (added the 5-second head reconciliation for unresolved
checks). Refs #21580 (the same kind of fix for the GitHub response
sweeps).

**What happened?**

On a hosted test database (Neon, 488-second window, zero running or
queued runs, 2 runs created in 30 minutes), these five statements ran at
exactly 10.3 a second each: the `chat_endpoints` source read, the
`chat_github_reviews` read, the `chat_message_links` comment read, the
`heartbeat_runs` `wakeCommentId` lookup and the `chat_deliveries` state
read. All five come from `githubReviewCheckState`
(`server/src/services/chat-channels/github/review-check-state.ts`).

**Expected behavior**

A check whose state cannot change makes no recurring database reads or
GitHub calls until something that can change it happens: a local hint, a
lease expiry, or a saved deadline. The missed-webhook head check stays,
at a bounded cost.

**Steps to reproduce**

1. Admit a GitHub review check. Bind a task review to it, but give it no
run that the check can find (for example, the run was never created, or
its wake comment is not the delivery's comment).
2. Start the review check queue and leave the instance idle.
3. Count statements. The check is due again 5 seconds after each poll,
for ever. Each poll evaluates the check state about 3 to 4 times (before
dispatch, in dispatch, after dispatch, and in the rebuild that the
publication lease release starts). Each poll also asks GitHub for an
installation token and the pull request.

**Cause**

`dispatch` (`server/src/services/chat-github-checks.ts`) saved `retryAt
= now + 5s` after each poll. `entry()` in `review-check-work.ts` uses
that deadline when the state is unchanged. So an unresolved check was
due every 5 seconds without end.

## What Changed

- `chat-github-checks.ts`: save `unchangedPolls` in the check result. A
poll that finds the same state as the saved state adds one (up to 6). A
changed state, a first publication and a recovered failure set it to 0.
The next head poll is due after `min(5 minutes, 5 seconds ×
2^unchangedPolls)`.
- Export `GITHUB_REVIEW_CHECK_HEAD_POLL_MS` (5 seconds) and
`GITHUB_REVIEW_CHECK_MAX_HEAD_POLL_MS` (5 minutes).
- `chat-channels.integration.test.ts`: two regression tests. They use
the real review check queue, the real `dispatch`, embedded PostgreSQL, a
controlled clock and one pass each second. The fixture has the hosted
state: a review with no run, a delivery with a linked comment, and no
run for that comment.

## Verification

Reproduction with the real delivery coordinator, the real queue and
embedded PostgreSQL. One queued check with no findable run, 30 seconds
of real time, before this change:

| Company-wide hints | Statements a second | Check state reads a second
| GitHub pull reads a second |
| --- | --- | --- | --- |
| None | 8.4 | 0.70 | 0.17 |
| One each second (`notifyChatRunMilestones`) | 15.2 | 1.67 | 0.17 |

So one stuck check costs about 0.7 state reads a second from the head
poll alone. The hosted rate (10.3 a second) is consistent with about 15
such checks, or fewer checks with frequent company-wide hints.
Company-wide hints are a separate cause and a separate pull request.

Red/green regression test, 600 simulated seconds with one pass each
second:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "unresolved GitHub
review check cannot change|publishes a local change at once"`
- Red (without the fix): `expected 120 to be less than or equal to 7`.
Over 600 seconds: 5,160 statements, 360 check state reads, 120 GitHub
pull reads.
- Green: 258 statements, 18 check state reads, 6 GitHub pull reads (8.6
→ 0.43 statements a second, a 95% drop). The saved interval stops at 5
minutes.
- Correctness (green): after the back-off, a restarted queue makes no
early GitHub call. A run bound to the review publishes `in_progress` at
the next pass and resets the interval. A pushed head is still found
within 5 minutes, the check is cancelled and the review is superseded.

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed,
including all 12 review check queue tests.
- `cd server && npx vitest run
src/services/chat-github-review-repair.test.ts
src/__tests__/idle-completed-history.test.ts
src/__tests__/chat-admission-condition.test.ts
src/__tests__/app-private-hostname-gate.test.ts`: 109 passed.
- `cd server && node ../scripts/run-tsc.mjs` and `node
scripts/typecheck-layers.mjs` (with test layers): passed.
- `pnpm verify --explain` retains 2,718 test files conservatively, so I
did not run it locally. Required CI supplies that coverage.

## Risks

- A push whose webhook is lost is found later: after up to 5 minutes,
not 5 seconds, when the check has not changed for a while. The new
head's own webhook still creates its new check at once. Only the old
head's check stays queued or running on GitHub a little longer before it
is cancelled.
- Local changes are not delayed. `entry()` still makes a check due at
once when its state differs from the saved state, and run, review,
enqueue and lease hints still rebuild it.
- Saved actions from an older release have no `unchangedPolls`. They
start at 0. An invalid saved value also starts at 0. No migration.
- Restart recovery is unchanged: the deadline is saved on the action,
and startup rebuilds all unfinished checks.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local embedded PostgreSQL
reproductions).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:22:35 -06:00
DottaandPaperclip 7c0c3c4f31 fix(mcp): preserve Tasks sleep authority and re-enable recovery (#21603)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected apps can retain unfinished provider work through MCP
Tasks.
> - Managed defaults can enable Tasks without a stored experimental
flag.
> - The sleep check used the stored flag and could miss an active
provider task.
> - Enabling Tasks also failed to notify an owner that had become idle.
> - This pull request uses the effective setting and wakes the existing
owner after the setting commits.
> - Pending work keeps its recovery path. Completed history can become
quiet.

## Linked Issues or Issue Description

Refs #21263, which added durable provider Tasks. The current roadmap
covers the governed MCP Tool Gateway. This change repairs two existing
paths.

**What happened?**

The gateway could enable MCP Tasks through managed defaults while the
idle check reported no pending work. After an operator disabled and
enabled Tasks, an existing task could stay idle until a separate task
write woke its owner.

**Expected behavior**

The gateway and sleep check must use the same effective setting.
Enabling Tasks must notify the existing owner. It must retain the task's
original deadline and backoff.

**Steps to reproduce**

1. Retain a task with `next_poll_at`. Use managed Tasks defaults without
a stored true flag. Read the normal sleep projection.
2. Disable Tasks, retain the same task, then enable Tasks through the
settings service without writing the task again.

## What Changed

- Read the effective Tasks flag inside the same read-only sleep
transaction. Bind that value to the existing pending-task query.
- Register the existing Tasks queue notification before the settings
transaction writes. The normal commit protocol owns the wake.
- Add two real-PostgreSQL regressions to the existing nine-case suite.
Extend eight mock transactions with the read projection they now need.
Keep their assertions unchanged.
- Publish the Tailscale enrollment fixture state by rename so its CLI
watcher cannot parse a truncated JSON file. Existing enrollment
assertions and time limits remain unchanged.

## Verification

The two regressions fail against the preceding current-master products:
the pending-work projection reports `none`, and re-enable does not rearm
the owner within the original two-second oracle. After the fix, all nine
PostgreSQL cases pass at the recorded source, including completed
history, disabled Tasks, credential debt and fail-closed query errors.

At current `72bc32f`, the normal DB build and migrations, all nine
PostgreSQL cases, 155 affected mock cases, and the complete normal
Server typecheck/planner pass. The mock run includes three selected
safety families and six complete files; it explicitly excludes 196
existing durable-PG cases. All original case deadlines and assertions
remain. The startup file's earlier 133-case pass is limited to its
unchanged mock contract; a new hot-restart implementation is not
relabeled as tested. Root independently verified all current outputs and
process closure. The MCP product and regression files remain
byte-identical at `321c4b2ae2f3`; its additional change is the Tailscale
test fixture repair below. Exact current-head CI passes: 55 successful
checks, four explicit skips and one neutral result; Snyk succeeds. Fresh
Greptile is 5/5 for this exact head. The complete latest edited summary
and all paginated threads were inspected, with no open findings.

An earlier additional whole-eight-file run reached its 180-second
aggregate watchdog while the existing mixed PostgreSQL suite was still
running. It produced no individual failing assertion. Its startup file
passed 133 cases. The original failure remains recorded. The follow-up
selects the three affected mock families and runs the other affected
mock files whole; it does not claim that the interrupted PostgreSQL file
passed. No test assertion or case deadline changed. Full required CI
remains a merge gate.

The first remote run at `72bc32f` passed 51 checks but failed one
Tailscale enrollment case and an interrupted browser shard; two
aggregate gates then failed. The browser log records a runner shutdown
signal; its underlying cause is unknown, and one specific-job recovery
was requested. The Tailscale case failed its status CLI on both original
CI attempts. A local control then held publication until the original
file watcher observed an empty state: the CLI threw `Unexpected end of
JSON input`, and the existing enrollment assertion failed. Publishing
the state atomically passed the same control. All nine original
enrollment cases now pass in 4.393 seconds with the fixture repair. The
recorded remote interleaving remains unknown; the original logs and the
initial local non-reproduction are retained. No retry count, assertion,
timeout or product behavior was weakened. The later exact-head run
passes all required checks. A subsequent server shard was interrupted by
a runner shutdown and later recovered at the same source; the
interruption and unknown underlying cause remain recorded.

Use Node24/pnpm9, one worker, an owned external PostgreSQL test database
and the existing fixture/admin URL options. The two new regressions are
in `server/src/__tests__/idle-sleep-diagnostics-postgres.test.ts`.

These checks do not establish deployed suspension or a live provider's
availability.

## Risks

- The read adds the normal effective-settings projection to the existing
bounded snapshot. Query errors must still block sleep.
- A settings update now notifies the existing task owner. Transaction
rollback must not publish that notification.
- This adds no schema, provider protocol, timer or replacement task
framework. Existing active task polling remains required until its work
settles.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI `gpt-6.1-sol` implemented the regressions and advised the fix.
GPT-6-based Codex also reviewed and integrated it. The host does not
expose every supervising model ID or context-window size. The agents
used source editing, Git, code execution and regression tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (the focused scope above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
existing pending-task contract remains accurate)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-14 05:00:50 -06:00
DottaandClaude Opus 5.5 aaec67b891 fix(native): close idle warm sessions in parallel across agents at shutdown (#21597)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - At shutdown, the server closes its idle warm Runner sessions before
it prepares the hot-restart snapshot
> - It closed them one at a time, and each full close (checkpoint, then
two stopped-copy collections) takes 3-5 s
> - In a zero-downtime swap test, five closes used the whole 20 s
pre-drain budget. Hot-restart preparation timed out, a checkpoint was
cut mid-way, and shutdown logged "hot-restart shutdown preparation
failed"
> - The same test logged 34 "Delivery reconciliation failed" errors, all
from directory cleanup retries that found the agent directory lock busy
> - This pull request closes idle warm sessions in parallel across
agents, one at a time within an agent, and treats a busy directory lock
as a quiet retry
> - The benefit is that shutdown preparation fits its budget with
several warm sessions, and expected lock contention no longer looks like
a failure

## Linked Issues or Issue Description

**What happened?**
`closeIdleWarmNativeSessions` awaited each warm session's full close in
turn. In swap 2 of a release-swap test, five idle sessions closed at
22:28:42, :46, :51, :56 and :59 (one "transport_close_requested"
diagnostic each). The `hot_restart_prepare` step then timed out at 19.9
s. Separately, `recoverDirectoryRelease` threw
`AgentDirectoryOperationBusyError` when another close held the agent
directory lock. The delivery coordinator logged each one at ERROR level.

I first checked if these sessions should have taken the cheap
released-session detach. They could not: `entry.released` is set only
after a successful `session.release()` through a release/resume-capable
plugin Runner task. The Runner state files from all sandboxes in the
test recorded only `run.prepare`, `session.open`, `turn.start`,
`runner.drain` and `runner.suspend` commands, and no `runner.release`.
The Daytona daemon logs show only `/process/execute` and
`/process/session` calls, and no Runner-task API. So no session in that
test was ever released, and a full close was the only correct shutdown
for them.

**Expected behavior**
Idle warm sessions of different agents close at the same time. Two
sessions of the same agent never close together, because they collide on
the agent directory advisory lock. A cleanup retry that finds that lock
busy stays pending until its claimed retry time and logs no error.

**Steps to reproduce**
1. Keep five or more idle warm sessions on remote sandboxes.
2. Send SIGTERM.
3. The closes run one after another and `hot_restart_prepare` times out.

**Paperclip version or commit**
`master` at bbed32450c.

## What Changed

- `closeIdleWarmNativeSessions` groups sessions by company and agent. Up
to four agents close at once, and each agent's sessions close one at a
time. The per-session close logic is unchanged.
- Idle-sleep retirement (`prepareIdleWarmNativeSessionsForSleep`) keeps
a concurrency of one, so it still checks drain authority before each
retirement.
- `recoverDirectoryRelease` catches `AgentDirectoryOperationBusyError`,
keeps the deferred retry time, and returns the pending entry instead of
throwing.

## Verification

- Red, then green (each failed against `master` source and passes with
the fix):
- `closes idle warm sessions of different agents in parallel and one at
a time within an agent` in
`server/src/services/native-runtime/native-session-executor.test.ts`.
Before the fix, the second agent's close was never started while the
first agent's close was pending.
- `keeps a cleanup pending without an error while another server holds
the agent directory lock` in
`server/src/__tests__/agent-directory-working-copies.test.ts`. It holds
the real Postgres advisory lock from a second connection. Before the
fix, the coordinator received an `AggregateError` with
`AgentDirectoryOperationBusyError`.
- `native-session-executor.test.ts` (1502 passed) and
`agent-directory-working-copies.test.ts` pass in full.
- `pnpm typecheck` in `server` passes.

## Risks

- Up to four warm closes now run at once, so shutdown makes more Runner
and database calls at the same time. The cap keeps this small.
- The order of entries in `failures` can change when several closes
fail.
- A busy release now waits for its deferred retry silently. A lock that
stays busy for a long time shows as a stuck pending cleanup, not as
repeated errors.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 04:15:19 -06:00
DottaandClaude Opus 5.5 30fc64e212 fix(instructions): collect a waiting working copy as soon as its stop proof is written (#21596)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run gets a private working copy of the agent's instructions or
agent files. A queue collects the copy after the provider is proven to
have stopped.
> - #21586 stopped this queue from re-reading copies forever: a copy
that waits for proof of the stop is now checked again after half of its
wait, capped at 6 hours.
> - With only that change, a copy whose stop is recorded T after its run
finished is collected up to T/2 later (at most 6 h). Before #21586 this
took at most 30 seconds. A sandbox that takes minutes to stop therefore
delays the save of an agent's instruction changes, and copies of
stranded runs wait up to 6 h after their stop is finally recorded.
> - This pull request makes the write that records the proof bring the
run's waiting copy forward, so the queue checks it on its next pass.
> - The benefit: collection is prompt again when the proof arrives,
without polling copies that are still waiting.

## Linked Issues or Issue Description

Follow-up to #21586. No public issue exists. Bug description:

**What happened?**
After #21586, a working copy waiting for proof of the provider stop
keeps its saved backoff deadline when the proof arrives. Proof is a
`native.local_process_stopped` event or a remote lease termination
receipt. A copy whose run finished 30 days ago is collected up to 6
hours after its stop is recorded.

**Expected behavior**
When a stop proof for a run is written, that run's waiting copy is
checked on the next queue pass. A write for a run without a waiting copy
costs one indexed no-op UPDATE and sends no work signal.

**Steps to reproduce**
1. Create a working copy for a native run that finished 30 days ago and
has no stop event, then let the collection queue check it once. Its next
check is about 6 h later.
2. Record the local stop (`recordNativeLocalProcessStopInTransaction`),
or release the run's remote lease with a termination receipt.
3. On master the copy stays in `pending_collection` until the old
deadline.

**Paperclip version or commit**
`origin/master` at 633dec7e43.

## What Changed

- New `notifyInstructionCopyStopEvidence(tx, { companyId, runId })` in
`instruction-collection-notifications.ts`.
- It reads the run's copy by its primary key (`run_id`, scoped by
`company_id`), and only if that copy waits in the collection queue or
the agent-files cleanup queue.
  - If there is no such copy, it stops: one SELECT, no write, no signal.
- If there is one, it first registers the run-scoped hint for both the
collection and the cleanup queue, and then runs one UPDATE by the
primary key. The UPDATE sets `next_attempt_at` to `now()` (or keeps an
earlier value) and sets a new `updated_at`. All of this happens in the
producer's transaction.
- The collection claim now compares `updated_at` with the value the
worker read (millisecond precision). If a stop proof is written between
the worker's evidence read and its backoff write, the claim fails and
the worker returns the copy as due, so the next pass collects it.
Without this fence, the worker could save a 6 h deadline over the proof.
- The writes that record stop proof call it in their own transaction:
- `native.local_process_stopped` receipts: `appendLocalProcessStop`
(used by `recordNativeLocalProcessStop` and its transaction form),
`recordUnidentifiedLocalRunnerStop`, and the operator confirmation in
`confirmUnidentifiedLocalRunnerStopped`.
- Remote lease termination receipts: `notifyEnvironmentCleanup` when it
has a run scope. Every lease release (`releaseLease`) and every
`writeEnvironmentCleanup` write calls it, which covers the receipt
writes in `environment-runtime`, `environment-task-runtime`,
`native-workspace-export-resume`, `sandbox-stop-and-retain` and
pending-cleanup recovery. The one remaining receipt write (sandbox
absence in `sandbox-stop-and-retain`) calls it directly.
- Orchestration's own stop path already calls `collectStopped` directly
after it joins the provider, so it collects without the queue and needs
no change.

## Verification

- `server/src/__tests__/agent-instruction-copy-retry-backoff.test.ts`, 3
new tests (embedded PostgreSQL, real queue registration):
- A local copy whose run finished 30 days ago (next check about 6 h
away) is collected within 10 seconds after
`recordNativeLocalProcessStopInTransaction` records the stop.
- A remote copy with the same backoff is collected within 10 seconds
after `environmentService.releaseLease` writes a termination receipt.
- For a run without a waiting copy, the transaction contains one SELECT
and nothing else, and no hint is sent. For a run with a waiting copy,
the only hint is `{"companyId":…,"runId":…}` (run-scoped, not
company-wide).
- A stop recorded between the worker's evidence read and its claim
(injected in the claim's transaction hook) is still collected on the
next pass.
- Red first: with the producer changes reverted, the two collection
tests fail (the copies stay in `pending_collection` until the old
deadline). Without the `updated_at` fence, the race test fails.
- Existing suites, one at a time: see the PR comment for the list and
results.
- `pnpm check:module-boundaries` passes.

## Risks

- Every lease cleanup write with a run scope now runs one extra
primary-key SELECT on `agent_instruction_working_copies`, and an UPDATE
only when a waiting copy exists.
- For agent-files cleanup (scratch files only, the saved data is not at
risk), the same race is not fenced. A missed advance there waits for the
cleanup backoff.
- A lease cleanup write that does not carry a termination receipt can
also bring a waiting copy forward. The queue then checks the stop
evidence once more (3–5 statements) and moves the copy back to its
backoff if the stop is still not proven.
- A release written with only a lease scope (no run) does not bring a
copy forward. That copy is still collected at its saved deadline.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:57:36 -06:00
DottaandClaude Opus 5.5 009c8884a4 perf(attention): read blocker retention state once per page (#21595)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items. One
kind is `blocker_attention`: a blocked task whose terminal blocker needs
a person.
> - When the queue loads a page of blocked tasks, it reads `keep` and
`archived_at` from `decision_retention` for each of them, one full-row
`select` per task.
> - A company hint walks every page. On a hosted instance with 2,112
blocker attention items and one active run, these per-task reads ran at
about 16 per second.
> - This pull request reads the retention state for the whole page in
one narrow statement. It also scopes three remaining run-path attention
hints to the task graph.
> - The benefit is one retention read per page instead of one per
blocked task, with the same decisions.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The `blocker_attention` eligibility read in
`server/src/services/attention-retention.ts` (`eligible()`), and three
run-path producers of company-wide attention hints.

**Subsystem affected**

Server background retention (`attention-retention.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-review-participant.ts`,
`native-runtime/native-workspace-reconstruction.ts`).

**Current behavior**

- For each blocked task whose blocker attention is terminal (and for the
owner variant), `eligible()` ran `select <all columns> from
decision_retention where company_id = $1 and source_kind = $2 and
source_id = $3`. It did this during `load`, once per task, on every
company hint.
- The hosted statement is identified by that column prefix: `id,
company_id, source_kind, source_id, source_activity_at, …`.
- I checked every other full-row reader of `decision_retention`. None
fires per run event:
- `syncItems`: attention summary and list requests, which a board calls
once per load; retention `check` and the sweep, which run for due
entries only; Keep and Archive actions.
  - `getState`: user actions only.
  - `archiveReviewedManifest`: proposal acceptance only.
  - `decisions.list`: only when archive proposals exist.
- `native-provider-capacity-retry`, `native-review-participant` and
`native-workspace-reconstruction` sent `{ companyId }` attention hints,
which make the queue walk every page.

**Proposed behavior**

- `pageEvidence` reads `source_id`, `keep` and `archived_at` for every
blocker root and every terminal blocker on the page in one statement
(ids passed as one JSON parameter). `eligible()` uses that map when it
has page evidence.
- A missing row means "not kept, not archived", exactly as the per-task
read returning no row did.
- The per-entry `check` path still reads one row, now only `keep` and
`archived_at`.
- The three producers send `notifyIssueAttentionRetentionWork` with
their task (and run). That is the task, its parents and the tasks it
blocks, with a company fallback past 100.
  - A capacity retry changes the failed-run items of its task.
  - A review participant claim changes the review of its task.
- A workspace reconstruction resolves the recovery action of its task,
which can also change the blocker attention of the tasks it blocks.
- The configuration, lifecycle, import and fallback producers keep their
company hints.

**Reason and benefit**

Background work must not grow with the backlog. A page now costs one
retention read instead of up to 100.

## What Changed

- `server/src/services/attention-retention.ts`: batched and narrowed
blocker retention read; `PageEvidence.blockerRetention`.
-
`server/src/services/native-runtime/native-provider-capacity-retry.ts`,
`native-review-participant.ts`, `native-workspace-reconstruction.ts`:
scoped attention hints.
- Tests in `server/src/__tests__/attention-retention-work.test.ts` and
`server/src/services/native-runtime/native-workspace-reconstruction.test.ts`.

## Verification

- `reads blocker retention state once per page, not once per blocked
task` (I saw it fail first). 160 blocked tasks, each behind its own
unassigned blocker; the blockers' retention rows are kept (40), archived
(40), plain (40) or missing (40). The full-size response capture and the
statement log measure the load of both pages:
- before: one `decision_retention` read per task (160), 26,240 result
bytes;
- after: 2 reads (one per page), 5,480 bytes, none of them selecting
`source_activity_at`.
  - The same tasks are listed: only plain and missing.
  - `check` still drops the kept and archived ones.
- `scopes the attention retention hint of a reconstruction to the task
graph` (I saw it fail first): before, the reconstruction sent `{
companyId }`; after, it sends a hint that names the task.
- `batches 100 blocker graphs with exact terminal timestamps and fresh
terminal visibility` pinned 200 retention reads per page (one per root
and variant). It now expects 1; the rest of that test is unchanged and
passes.
- Not separately tested: the capacity-retry and review-participant hint
changes. Each is one line that passes the task the function already
holds, the same pattern as the reconstruction test.
- `attention-retention-work` (58), `attention-retention-candidates`,
`native-workspace-reconstruction`, `native-review-participant` and
`heartbeat-run-dispatch-work` pass, each file run alone (in `server/`,
against dedicated local `_test` databases).
- `pnpm check:module-boundaries` passes.

## Risks

- The page read happens once, before the per-entry decisions of that
page. Before, each decision read its row just in time. A Keep or Archive
that lands between the page read and an entry's decision is seen by the
next load or by `check`, which re-reads under a row lock, as before.
- A run-path hint that changes more than its task graph would now miss
the wider effect. The 90-day archive delay makes a missed reload cheap.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:05:59 -06:00
DottaandClaude Opus 5.5 633dec7e43 fix(instructions): back off working copies that cannot make progress (#21586)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run gets a private working copy of the agent's instructions or
agent files. Background queues collect the copy after the provider stops
and clean up its scratch directory.
> - When the queue cannot prove that the provider stopped, or cannot
finish a cleanup, it re-arms the same copy every 30 seconds with no
limit. The attempt count does not increase on the "stop not proven"
branch.
> - Every re-arm is a transaction that sends work signals to its own
queue, to the stranded-agent-status queue and to the skill-retention
queue. A self-signal also reloads the copy's scope. On a hosted instance
with only a few live runs, these copies were one of the largest sources
of SQL calls and kept the main thread busy.
> - This pull request makes a copy that cannot progress wait on a
growing, capped backoff. It also removes the work signals from a write
that only moves a deadline later, and reads the copy once per check.
> - The benefit: a copy that waits for proof of the provider stop costs
about 1/350 of the old steady-state load, and a copy that becomes
collectable is still collected automatically.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**
A hosted instance had only a handful of live agent runs, but its server
main thread was about 95% busy. An analysis of its most-called SQL found
the instruction working-copy queues re-reading the same unresolved
copies forever, at about 25–30 statements/s. A copy whose run is
terminal but whose provider stop is not proven was re-armed every 30 s.
A completed agent-files copy whose remote scratch cleanup cannot finish
(its lease is released but not destroyed, or the lease row is gone) was
also re-armed every 30 s. Each re-arm was a transaction with work
signals that woke three queues.

**Expected behavior**
A copy that cannot progress is not re-read until a growing, capped
backoff elapses. A write that only moves a deadline later does not wake
other queues. A copy that becomes collectable is still collected
automatically, and no data is lost.

**Steps to reproduce**
1. Create working copies of terminal native runs that have no
`native.local_process_stopped` event, and completed agent-files copies
with `cleanupPending: true` whose remote lease does not exist.
2. Register the instruction-collection and directory-cleanup queues.
3. Count statements while time advances. With 300 + 300 such copies,
master sends 252 statements/s, 38.7 transactions/s and 38.7 work
signals/s.

**Paperclip version or commit**
`origin/master` at 3c3e6c2d75.

## What Changed

- `instructionCopyRetryAt(waitingSince)` sets the next check to half of
the time the copy has waited. The minimum is `INSTRUCTION_COPY_RETRY_MS`
(30 s) and the maximum is `INSTRUCTION_COPY_MAX_RETRY_MS` (6 h). An
unproven stop measures the wait from the run's `finished_at`. A cleanup
measures it from `process_stopped_at`. The deadline stays in the
existing `next_attempt_at` column, so a restart keeps it. There is no
schema change.
- The collection check now reads the copy once, checks the stop
evidence, and then claims. For an unproven stop, the claim writes the
backoff deadline (and the move from `prepared` to `pending_collection`)
in the same UPDATE. Before, the check was two reads plus two signalling
transactions.
- `tryPatch` skips all work signals when a write only moves deadlines
later and keeps the copy in the same queues (collection, cleanup, commit
and the stranded-agent predicate). A write that changes queue membership
or moves a deadline earlier still signals as before. The write stays in
its own transaction, so the existing claim and race tests keep their
hook.
- A cleanup that cannot finish gets the same capped backoff after
`release` re-arms it. The saved copy and its cleanup receipt are kept.
The claim does not change `attempts`, because a concurrent foreground
collection uses `attempts` in its compare-and-set.
- A remote agent-files copy that stopped but was not destroyed, and an
unavailable agent-files copy that waits for a destroy receipt, also get
the capped backoff instead of a 30 s re-arm.
- No copy is parked or dropped. A copy with an unproven stop stays in
the queue with the 6 h cap. When proof of the stop arrives later (a
`native.local_process_stopped` event, a lease termination receipt, or a
recovery that records that the host is gone), the next backoff check
collects it automatically. No manual call is necessary.

## Verification

- New suite
`server/src/__tests__/agent-instruction-copy-retry-backoff.test.ts`
(embedded PostgreSQL, real queue registration, statement meter):
- The backoff constants and their values at 0 s, 30 min, 2 h, 12 h and
30 days.
- A copy that has waited 2 days: two checks in 12 h, 5 statements and 1
transaction per check, and zero work signals.
- A copy owned by a live older server (run finished 5 minutes ago): it
is checked every few minutes. When the older server records the stop,
the copy is collected within 10 minutes, without a manual call.
- A copy that has waited 30 days: it is collected within one 6 h cap
after the stop is recorded.
- A backlog of 300 copies that have waited 30 days: one pass drains it
with zero work signals, and the hour after it is silent.
- A write that moves the deadline earlier, and a write that changes
queue membership, both still send the signal.
- A cleanup that cannot finish, with release returning and with release
throwing: at most 32 deadline writes in the first 6 h, at most 4 in the
next 6 h, zero work signals, and the copy is still saved with its
cleanup pending. Before the fix, the throwing case made 74 writes in 6
h.
- Red first: on master this suite fails 5 of 7 tests (the 30 s re-arm,
150 statements in the drain window, extra signals, a cleanup checked
every 30 s). The two "collects after stop" tests pass on master and
still pass: the positive behavior is kept.
- Steady state with 300 unproven stops plus 300 stuck cleanups, each
waiting for 1 day (10-minute window after warm-up):

| | statements/s | transactions/s | work signals/s | stop checks/s |
cleanup checks/s |
  | --- | --- | --- | --- | --- | --- |
  | master | 251.6 | 38.7 | 38.7 | 9.7 | 9.6 |
  | this PR | 0.45 | 0.10 | 0 | 0 | 0.03 |

- Per check: an unproven stop goes from 12 statements, 2 transactions
and 2 work signals to 5 statements, 1 transaction and 0 work signals. A
stuck cleanup goes from about 14 statements, 2 transactions and 2 work
signals to about 15 statements, 3 transactions and 0 work signals.
Cleanup checks are exponentially less frequent.
- Backlog drain: queue batches stay at 20 per pass, with one pass per
second at most. An overdue copy moves to its long backoff on its first
check. The first drain of N stuck copies therefore takes about N/20
seconds. For example, 17,000 copies drain in about 15 minutes with no
work signals.
- Existing suites, one at a time: `agent-instruction-working-copies`
(50/50), `agent-directory-working-copies` (120/120),
`heartbeat-instruction-working-copy` (4/4),
`heartbeat-agent-directory-busy` (8/8), `agent-file-checkpoint-locking`
(9 passed, 1 skipped), `agent-file-durability` (17/17),
`idle-sleep-safety` (226/226), `heartbeat-process-recovery` (459/459),
`agent-instructions-routes` (26/26), `services/heartbeat/agent-status`
(42/42).
- `pnpm check:module-boundaries` passes.

## Risks

- Product-visible choice: the 6 h cap. A copy that has waited for a long
time is collected at most 6 h after its stop proof arrives (half of the
wait while the wait is shorter than 12 h). During that time, the
uncollected candidate shows in the agent's instruction candidates
(`pending_collection`, `INSTRUCTION_STOP_UNCONFIRMED`) for legacy
copies, and in the run's agent-files diagnostics for agent-files copies.
Its bytes are kept on disk. An explicit `collectStopped` from
orchestration still collects immediately.
- There is no seven-day park. A park at `attempts = 3` would leave the
queue predicate, and a later stop proof would not reach the copy without
changes in files that another fix owns. The copy therefore stays in the
queue with the 6 h cap. The steady cost is about 5 statements per copy
every 6 h.
- During the 40-minute deploy overlap, a copy owned by a live older
server waited only a short time, so its backoff is still 30 s to a few
minutes. The test covers this case.
- A write that only moves a deadline later no longer wakes the
stranded-agent-status and skill-retention queues. The stranded-agent
predicate is compared before and after the write, and any change still
signals.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:47:06 -06:00
DottaandClaude Opus 5.5 c3f9a0c040 fix(retention): evaluate native retention blockers for many sessions in one statement (#21588)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep runner checkpoints and harness backups so that a
session can resume. Retention expires them after the session can no
longer resume.
> - A retention blocker decides whether a session can still resume: a
live or recent run, an open or recently closed task, an unsettled
finalization, or a live lease.
> - The startup checkpoint pass evaluated that blocker in its own
transaction, once for each retained checkpoint, with three reads. A
session that stays blocked (for example, a long-lived open task) was
evaluated again for each of its runs on every server start. On a hosted
instance with frequent deploys and a 40-minute overlap of two servers,
this pass ran almost continuously at about 7 checkpoints/s, about 21
statements/s.
> - Backup reconciliation also used four reads and a transaction for
each run hint that reached a backup.
> - This pull request evaluates the blocker for many sessions in one
fixed-text statement. Only candidates that are not blocked enter the
existing locked transaction.
> - The benefit: a restart costs 2 statements for each page of 200
blocked checkpoints instead of 5 statements and 1 transaction for each
checkpoint, and a backup hint costs 2 statements with no transaction.

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**
A hosted instance had only a handful of live agent runs, but its server
main thread was about 95% busy. An analysis of its most-called SQL found
three retention-blocker reads (`company_id, agent_id from
heartbeat_runs`; the session's runs; the session's issues) at exactly
7.0/s each, each in its own transaction. The startup checkpoint pass
sent them once for each retained checkpoint older than 7 days whose
session was still blocked, and it repeated the full pass after every
start.

**Expected behavior**
A checkpoint or backup that is blocked from expiring costs no
transaction and no statements of its own, because one statement
evaluates its session together with the rest of its page. It expires as
soon as its blocker clears: checkpoints on the next startup pass (the
current design), and backups at their exact deadline.

**Steps to reproduce**
1. Create 100 native sessions with 3 finished runs each (finished 10
days ago), settled finalizations, a checkpoint header for each run, and
an open task.
2. Register the startup checkpoint expiration and count statements. Then
register it again, as a restart.
3. On master each start costs 1,502 statements and 300 transactions, and
nothing expires.

**Paperclip version or commit**
`origin/master` at 3c3e6c2d75.

## What Changed

- `SESSION_RETENTION_BLOCKERS_SQL` (new module
`native-runtime/native-session-retention-blockers.ts`) is one fixed-text
statement, sent through `$client.unsafe`. It takes a JSON list of
sessions and returns, for each one, the same blocker reasons as
`sessionRetentionBlocker`, in the same order: missing run, live run,
incomplete native recovery, recent run (checkpoint mode), unknown, open,
executing or recently closed task, pending finalization, live lease. It
also returns the session's resume pointer, run, task and lease
dependencies, and the latest close time.
- `expireCheckpoints` and `expireDurableBackups` evaluate each page of
candidates with this statement. They skip blocked candidates and
candidates whose session still has a resume pointer. A candidate that is
not blocked still enters the existing transaction. That transaction
checks the blocker again under row locks before it retires anything.
- `reconcileBackup` uses the statement for its dependencies and its
deadline. It replaces three dependency reads plus a blocker transaction.
The lease references use one index-backed arm for each reference kind
(run, backup stamp, task, identity lease).
- The unused `lock: false` path of `sessionRetentionBlocker` and the
`backupDependencies` helper are removed.
- The existing index test now checks the plan of the new statement.
Every arm that reads `environment_leases` uses an index in both the
custom and the generic plan.

## Verification

- New suite
`server/src/__tests__/native-retention-blocker-batch.test.ts` (embedded
PostgreSQL, real coordinator registration, statement meter):
- 300 blocked checkpoints: a start costs 4 statements and 0
transactions, and a restart costs the same. On master it is 1,502
statements and 300 transactions for each start. The checkpoints are
kept.
- When a session's task is closed long enough ago, its checkpoints
expire on the next startup pass. The other sessions are not touched.
- A session with a resume pointer keeps its checkpoints for 2 statements
in total.
- Two servers that run the startup pass at the same time expire every
checkpoint exactly once with no failures, because the locked transaction
checks again and the retirement header is compared before deletion.
- 50 run hints that reach 50 blocked backups: at most 2 statements for
each hint (the scope lookup and one blocker statement) and no
transaction. On master it is 9 statements and 1 transaction for each
hint. The backups are kept.
- Existing suites, one at a time: `native-harness-backup-expiration`
(109/109, including the backup expiry flows for task close, reopen and
lease release), `native-harness-backup-expiration-config` (7/7),
`native-harness-backup-storage.integration` (73/73),
`native-harness-backup-publication` (4/4),
`native-retired-backup-source.integration` (38/38).
- `pnpm check:module-boundaries` passes.

## Risks

- The startup pass still runs on every server start. It also runs on
both servers during the deploy overlap. Its new cost is 2 statements for
each page of 200 candidates (the candidate page and one blocker
statement). A page of candidates that are not blocked still uses one
locked transaction for each candidate, as before.
- Two servers can run the pass at the same time. They cannot retire the
same session twice or retire a blocked one: the unlocked statement only
filters, and the existing transaction locks the finalizations, checks
the blocker again, and compares the retirement header before it deletes.
The concurrency test covers this.
- If the SQL blocker and the TypeScript blocker ever disagree in the
"not blocked" direction, the locked transaction still refuses. If they
disagree in the "blocked" direction, retention is only delayed. When
more than one task blocks a session, the reported reason can differ from
the TypeScript loop, which reads the tasks in no fixed order.
- Checkpoints keep their current design: they are considered again on
the next startup, not at a deadline. Backups keep their exact deadline
after a task closes.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:47:01 -06:00
DottaandClaude Opus 5.5 f452b449fd fix(recovery): stop re-checking settled native runs that cannot be released (#21590)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.

**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.

**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.

`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.

**Paperclip version or commit**
`master` at 3c3e6c2d75.

**Deployment mode**
Any. The cost grows with every deploy of a hosted instance.

## What Changed

- `server/src/services/native-local-process-stop.ts`
- New `settledNativeProcessCandidates`: one fixed-text statement that
finds releasable runs. It reads only runs recorded in this server's PID
space (`process_host_hash`), sandbox runs (`remote`, with a non-local
lease), and rows saved before the hash (their latest identity event must
name this host). It excludes runs with an unsettled coordinator phase
and runs with no checkpoint and no provider evidence. Each PID-space
branch reads its own range of the new partial index.
- `releaseSettledNativeRunProcess` now reads the run, phase, evidence,
lease facts and identity match in one `for update` statement (it was 6
statements). It reads the lease rows again only for a sandbox run. The
new `inspectSettledNativeRunProcess` also returns whether a remote stop
acknowledgement can apply.
- Startup discovery uses the same candidate statement, so it opens no
transaction for an excluded run.
- `server/src/services/heartbeat/live-recovery-work.ts` (settled kind
only)
- The load runs the candidate statement and a separate indexed statement
for cancelled `cursor_cloud_api` runs whose Stop is still `requested`.
- The check uses the loaded entry. It no longer runs the load again for
each entry.
- A "pending" run backs off 30 s, 60 s, 120 s, up to 15 min. A
run-scoped hint (identity event, lease cleanup, stop receipt) resets the
backoff.
- A "skipped" run that the predicate cannot exclude (for example a birth
time mismatch on this host) waits for a run-scoped hint or a restart.
- `server/src/services/heartbeat/recovery.ts` (the settled handler
only): returns the outcome. It calls `acknowledgeRemoteStop` only when
the run has a non-local lease or is a cancelled `cursor_cloud_api` run.
For any other run that function reads two rows and writes nothing.
- `packages/db`: migration 0487 builds
`heartbeat_runs_settled_process_identity_idx` and
`heartbeat_runs_pending_api_stop_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. The migration is `online_compatible` and is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. `doc/DATABASE.md` describes both
indexes.

End state for each reason that a check cannot finish:

| Reason | Before | After | What re-admits the run |
| --- | --- | --- | --- |
| Recorded by another PID space | checked on every hint | not read |
only a server in that PID space reads it (a live older server in a
deploy overlap still releases its own runs) |
| Coordinator phase not settled | checked on every hint | not read | the
phase write; the finalizer sends a company hint in the same transaction
|
| No checkpoint and no provider evidence | checked on every hint | not
read | the next hint after a provider or turn event |
| Live local process, unreleased lease, missing sandbox receipt | every
hint and every 30 s | backoff 30 s to 15 min | a run-scoped hint resets
the backoff |
| Other same-host skip | checked on every hint | parked | a run-scoped
hint (an identity event sends one) or a restart |

## Verification

- `pnpm exec vitest run
src/__tests__/settled-native-process-rechecks.test.ts` (new, 12 tests,
real PostgreSQL, real queue and coordinator, fake clock):
- each exclusion: replaced container, unsettled phase, no evidence: 0
checks over 4 hints (red on `master`: 3 to 5 checks);
- a live older server and a stopped older server (different hash): the
new server never reads the run and writes no receipt; this server's own
run is released at start;
- re-admission: phase settles, late provider evidence, legacy rows with
no hash (this host released, another host not read), sandbox run
released from any server with its receipt;
- live process: at most 4 checks over 8 hints (red: 17), released on the
run-scoped hint after the process exits;
- one check is BEGIN, one `for update` statement and COMMIT (3
statements); the next hints do not check it; a run-scoped hint checks it
once more (red: 5 statements and a check on every hint);
- a restart costs one check for each pending run; startup discovery with
50 excluded runs sends 1 statement (red: 302).
  - On `master` 11 of the 12 tests fail.
- Measurement on embedded PostgreSQL, 300 stuck runs, one company hint
per second for 300 s (fake clock), same harness before and after:
- before: 9.0 checks/s, 84.3 statements/s, 9.4 statements per check, 5.9
s CPU (2.2 ms per check);
- after: 0 checks/s, 3.2 statements/s for all seven residual queues
together, 0.16 s CPU.
- Load plan with 12,000 excluded finished runs (10% without a hash) in a
31,000-row `heartbeat_runs`: the candidate statement reads only index
entries for this server's hash, `remote` and `unknown`, and a bitmap
scan of the 1,200 legacy rows: 2.7 ms, 4,416 buffers. Without the index
the same exclusion read all 31,000 rows (31,127 buffers).
- Existing suites, one file at a time: all pass.
`live-recovery-work.test.ts` 51/51 (an earlier run under machine load
hit the `afterAll` hook timeout after all tests passed; the rerun passed
clean, and `native-safe-replacement.test.ts` hits the same hook timeout
on the base commit), `native-settled-process-release.test.ts` 31/31,
`native-local-process-stop.test.ts` 5/5, `heartbeat/recovery.test.ts`
107/107, `native-runner-restart-recovery.integration.test.ts` 231 passed
/ 14 skipped, `native-restart-recovery.test.ts` 49/49,
`native-controller-recovery.integration.test.ts` 30/30,
`retained-native-session-cleanup-work.test.ts` 29/29,
`native-provider-failure.integration.test.ts` 16/16,
`native-safe-replacement.test.ts` 70/70 (with the `afterAll` hook
timeout that also occurs on the base commit),
`native-remote-retry-admission.test.ts` 93/93.
- `pnpm check:module-boundaries`, `pnpm check:migrations` (packages/db),
server typecheck: pass.
- Not run: the full suite and browser suites.

## Risks

- Unresolved by design: a finished run whose server was replaced keeps
its process fields indefinitely and is never released, because only the
host that recorded a process can prove that it stopped, and the shared
prover refuses a controller-lease proof for a run with a process group.
This pull request only stops paying for it. These runs keep no stop
receipt (`native.local_process_stopped`), so what waits for that receipt
still waits: accounting close of the run's usage source, retained native
session cleanup, and instruction or directory working copies that wait
for proof that the run stopped. A follow-up must decide the ownership
rule for a replaced container.
- Rows saved before `process_host_hash` exist are read through the
legacy branch, which checks the latest identity event for each of them
on each load. On an instance with many such rows the load costs one
index lookup per row. The count can be measured with `select count(*)
from heartbeat_runs where process_host_hash is null and process_pid is
not null and status in
('failed','cancelled','timed_out','interrupted')`.
- A run that this server recorded but whose `process_host_hash` changed
(a new PID namespace on the same host, for example a container restart)
is no longer read here. Before, it was read and skipped, so no release
is lost.
- The backoff and parked state live in memory. A restart checks each
pending run once.
- The migration builds two partial indexes concurrently on
`heartbeat_runs`. It holds no long lock.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:29:23 -06:00
DottaandClaude Opus 5.5 e8c79235ca perf(attention): scope run lifecycle retention hints and read only used columns (#21579)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items (failed
runs, reviews, questions, recovery actions, blocked tasks). A company
hint makes it reload every candidate of the company, 100 per page, with
eligibility reads for each page.
> - Several run lifecycle producers sent that company hint: every agent
status change after a run, the execution-control writes and legacy run
terminalization when they name only a run, scheduled retries, and the
Runner's task monitor tool.
> - Each page also read whole rows it did not use: full `heartbeat_runs`
rows (recent ones carry tens of KB of `context_snapshot` and
`runner_profile_json`), full `issues` rows (descriptions), and the whole
agent roster with its configuration.
> - This pull request names the task graph or the source kind in those
hints, and reads only the columns each eligibility decision uses.
> - The benefit is fewer company reloads per run event and far fewer
result bytes per reconstruction, with the same decisions.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`. Stacked on #21552 (it builds
on that pull request's change to `execution-control-notifications.ts`);
review that one first.

**What existing behavior does this improve?**

The attention retention queue
(`server/src/services/attention-retention.ts`) and the producers that
wake it on run lifecycle paths.

**Subsystem affected**

Server background retention (`attention-retention.ts`,
`retention-work-notifications.ts`, `execution-control-notifications.ts`,
`heartbeat/agent-status.ts`, `heartbeat/retries.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`).

**Current behavior**

- `heartbeat/agent-status.ts` sends `{ companyId }` on every agent
status change after a run (two places). `heartbeat/retries.ts` sends it
for each scheduled retry. `writeExecutionControl` and
`notifyExecutionRunWork` send it when the scope names only a run. The
Runner's `set_task_monitor` tool sends it for one task.
- On a local fixture with 1,500 open tasks and run churn, the queue
loaded 47 company candidate rows per second from these hints.
- `pageEvidence` read `select *` from `heartbeat_runs`,
`issue_recovery_actions` and (through `getTableColumns`) `issues`;
`reviewEligibility` read `select *` from `issues`;
`interactionEligibility` read the full agent roster; the per-entry
`check` repeated full-row reads.

**Proposed behavior**

- Agent status changes send `{ companyId, sourceKind:
"agent_error_alert" }`. The status moves between `running`, `idle` and
`error`. All three are invokable for blocker attention
(`BLOCKER_ATTENTION_INVOKABLE_AGENT_STATUSES`) and for questions
(`agent-eligibility.ts`). Failed-run attention only excludes terminated
agents, and this path never sets that. So only agent error alerts can
change.
- Run-scoped execution-control writes, legacy run terminalization and
scheduled retries use `notifyIssueAttentionRetentionWork` with the run
id (the run's task, its parents and the tasks it blocks; company
fallback past 100). This is the same helper as #21521. The task monitor
tool scopes to its task.
- `notifyIssueAttentionRetentionWork` matched the run's saved task id
with `issue.id::text = …`. It now validates and casts the id, so the
primary key is used, as #21552 does for recovery.
- Narrow reads, same decisions:
- runs: `id`, `agent_id`, `created_at` and the saved task id computed in
SQL with the same rules as `readRunIssueId` (`issueId` unless it is
absent or JSON null, then `taskId`; only non-empty strings).
  - recovery actions: `id`, `source_issue_id`.
- issues for evidence and `visibleIssue`: the blocker-attention input
columns plus `unblock_descriptor` and `blocked_transition_at`.
- review issues: `id`, `company_id`, `status`, `execution_state`,
`assignee_user_id`.
- agent roster for question addressees: `id`, `company_id`, `name`,
`reports_to`, `status` (the `AgentOrgRow` that
`evaluateAgentInvokability` takes).

**Reason and benefit**

A run event no longer reloads every attention candidate of the company.
A reconstruction no longer transfers whole run, task and agent rows.

## What Changed

- `server/src/services/attention-retention.ts`: column lists for runs,
recovery actions, issues and the roster; the run probe type replaces
whole rows.
- `server/src/services/heartbeat/agent-status.ts`,
`heartbeat/retries.ts`, `execution-control-notifications.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`: scoped attention
hints.
- `server/src/services/retention-work-notifications.ts`: indexed lookup
of a run's saved task.
- `server/src/__tests__/attention-retention-work.test.ts`: two tests.

## Verification

- `reads only the fields its decisions use from large runs, reviews and
agents` (I saw it fail first): 50 failed runs, 50 tasks in review with
large descriptions, 10 questions to a paused agent, about 40 KB of
payload in each run's snapshot, profile and stdout and in the agents'
configuration. It loads the three kinds with the full-size response
capture (`full-database-response-capture.ts`). Before: `heartbeat_runs`
6,317,700 bytes, `agents` 84,336 bytes, `issues` 4,235,810 bytes. After:
under the test bounds of 20,000, 5,000 and 60,000 bytes. The decisions
are the same before and after: 50 failed runs, 40 reviews and 10
questions.
- `scopes run lifecycle attention hints to the run's task graph and
agent status to agent alerts` (I saw it fail first):
`writeExecutionControl` and `notifyExecutionRunWork` with a run id send
the hint for the task and its parent. `finalizeAgentStatus` sends the
agent-alert hint. Before, all three sent `{ companyId }`.
- `attention-retention-work` (57 tests),
`attention-retention-candidates`, `attention-service`,
`execution-control-work` and `heartbeat-agent-status-recovery` pass,
each file run alone (in `server/`, against a dedicated local `_test`
database).
- `pnpm check:module-boundaries` passes.

Not done in this pull request, and why:

- I did not reproduce `syncItems` re-syncing unchanged rows from the
retention `check`. On my fixture (300 stalled blocked tasks, 4,500 old
recovery-action and blocker retention rows, run churn), `syncItems` ran
0 times in a 180-second window and `decision_retention` reads were 0 per
second. `check` calls `syncItems` only for entries that are already due
(older than the 90-day archive delay), and a due entry is archived and
leaves the candidates. So I did not batch `syncItems` or skip checks of
unchanged rows.
- `readIssues` still selects the `issues` columns that blocker attention
needs. I did not narrow `issueService.listBlockerAttentionBatch`.

## Risks

- A producer that changes more than the run's task graph would now miss
the wider effect. Company-level changes (agent configuration, company
status, budgets) keep their company hints. The 90-day archive delay
makes a missed reload cheap: the item is reloaded on the next hint that
covers it.
- The saved task id now comes from SQL. The SQL mirrors `readRunIssueId`
for strings, absent keys and JSON null. A non-string `issueId` value now
yields no task, as before.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:17:21 -06:00
DottaandClaude Opus 5.5 340157d1d6 perf(recovery): scope run lifecycle recovery hints to the run's task graph (#21552)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. Two of
them, `stranded-assignments` and `assigned-work-invariant`, find
assigned tasks that have no live run and restart them.
> - Every run dispatch and every run state change sent a recovery hint
for the whole company. On each hint these queues reload every eligible
open task of the company and check each one again, at most every 30
seconds per queue.
> - With live runs the hint comes all the time, so the queues never
stop. Their cost is (run activity) x (open-task backlog). On a hosted
instance with about 1,400 open tasks and 5 live runs, the server main
thread stayed at 87-92% of a core.
> - A run event can only change the tasks of its own task graph. This
pull request makes run and task producers name that graph, and keeps one
slow full pass as a backstop.
> - The benefit is that a run event costs a few checks, whatever the
size of the backlog, and a missed hint still delays recovery by at most
five minutes.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The issue recovery queues in `server/src/services/recovery/work.ts`
(`stranded-assignments`, `assigned-work-invariant`,
`resolved-dependencies`, `stale-issue-locks`) and the producers that
wake them on run dispatch, run start, run state change, run finalization
and accounting settlement.

**Subsystem affected**

Server background recovery (`server/src/services/recovery/`,
`server/src/services/scoped-deadline-work.ts`, the notification helpers
in `server/src/services/*-notifications.ts`).

**Current behavior**

`notifyRunDispatch`, `notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` sent `notifyIssueRecovery(tx, {
companyId })`. Most run lifecycle paths call one of them, often two or
three in the same transaction (for example `notifyIssueReviewState`).
Some other run paths sent the company hint directly: the wake status
update, the execution-control deadline writes, legacy run
terminalization, scheduled retries, accounting settlement, task checkout
and task document writes. Each company hint makes each task queue reload
all eligible open tasks of the company, 100 per page, and check each
task again. A company hint is reconstructed at most every 30 seconds per
queue, so one run event per 30 seconds is enough to keep a full pass
running all the time. `issue-recovery-deadlines.test.ts` pinned this: a
company hint re-checks every eligible task.

**Proposed behavior**

Design. For each task queue, a run event of run R on task T can change:

| Queue | What the check reads | What one run event can change |
| --- | --- | --- |
| `stranded-assignments` | T's own runs, wakes, handoffs, questions,
recovery actions, goal, assignee eligibility | T (its runs). The parent
of T and the tasks T blocks when the event also writes T's status. |
| `assigned-work-invariant` | the same plus open blockers and open
children | T, the parent of T (open children), the tasks T blocks (open
blockers) |
| `resolved-dependencies` | blocked tasks whose blockers are all done |
the tasks T blocks |
| `stale-issue-locks` | tasks whose checkout or execution run is R | T |

The assignee's agent status (`running`, `idle`, `error`) does not change
whether the agent is invokable
(`packages/shared/src/agent-eligibility.ts`), and a freed concurrency
slot starts the agent's next queued run through the
`queued-run-dispatch` queue, which keeps its company hint. So the
agent's other tasks are not in the set.

- `notifyIssueGraphRecovery(tx, { companyId, runId?, issueId? })` sends
`{ companyId, graphRunId?, graphIssueId? }` to the four task queues. It
runs no query in the producer transaction.
- When a task queue loads that hint, it resolves the graph with one
bounded, indexed query (`resolveIssueGraph`): the task of the run
(`issue_id`, `native_issue_id`, or the saved `issueId`/`taskId`,
validated and cast so the primary key is used), its parent, and the
tasks it blocks. The four queues share the result for 2 seconds. With no
task or more than 100 tasks, the queue turns the hint into the plain
company hint, so task-less runs coalesce under its 30-second gate
instead of each starting its own full pass. The lookup runs after
commit, so it sees the written state. Graph hints do not wait for the
30-second reconstruction gate (they load at most 100 tasks); hints for
the same run coalesce while dirty, and the lookup is shared across the
four queues, so it runs about once per pass instead of once per producer
call. An earlier round ran the lookup in every producer transaction: 34
lookups/s at 1,500 open tasks and 70/s at 5,000.
- The three company-level queues (`orphan-blockers`,
`active-recovery-actions`, `task-watchdogs`) keep the company hint. They
hold one entry per company or per watchdog, not per open task.
- `notifyRunDispatch`, `notifyScheduledRetryDispatch`,
`notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` use it when the scope names a run
or a task. A scope with only a company keeps the company hint.
- Run paths that knew the run or the task but sent only the company now
pass it. This includes three `writeExecutionControl` deadline writes,
legacy run terminalization (`notifyExecutionRunWork`) and native safe
replacement. Those helpers also forward the scope to
`notifyLiveRunRecovery` and `notifyLiveResourceRecovery`, so their hints
to the live-run queues now carry the run id.
- Recovery action changes (`issue-recovery-actions.ts`: create,
supersede, update, exhaust, resolve) name their source task. A recovery
action is a fact of its source task for these queues. The company-level
`active-recovery-actions` queue still gets the company hint. In the
fixture, failing runs create and resolve recovery actions every few
seconds, and that alone kept a full pass running.
- `writeExecutionControl` and `notifyExecutionRunWork` send the
attention retention hint for a task through
`notifyIssueAttentionRetentionWork` (the task, its parents and the tasks
it blocks), a superset of the single-task hint they sent before. A
recovery action of a blocker can change the blocker attention of its
dependents, so the narrower recovery action scope needs this.
- `promoteDeferredWorkIntents` scopes `lock_released` with a task to
that task, and `accounting_settled` to the run when the company has no
active hard-stop budget policy. Only a reached hard stop can refuse work
because of accounting (`evaluateAdmissionInTransaction`), so without
such a policy the settlement of one run cannot admit another task. The
other hold releases keep the company hint.

Backstop. Each scoped hint also sends `{ companyId, backstop: true }` to
the same four queues. Nothing else arms it. The queue runs it as a full
company pass, but not before 5 minutes after the last full pass of that
queue for that company or of the startup pass. It is a scheduler
deadline (`nextWakeAt` reports it), but it holds no idle intent and does
not mark the queue unreconciled. So:

- Startup still reconstructs every company.
- A failed scoped load retries with the existing back-off (30 seconds to
5 minutes). The backstop is pending in any case.
- A graph with no task or more than 100 tasks loads the whole company at
once.
- A task that a scoped hint missed is checked at most
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes) after the event. While runs
continue, each queue runs at most one full pass per company per 5
minutes. With no activity, nothing is scheduled.

Why arming on activity only cannot leave a stranded task undetected when
the instance then goes quiet. A task becomes stranded only through a
write or through a run that stops without a write:

1. The write names the task's graph (run and task producers). The scoped
pass checks the task at once.
2. The write sends a company hint (pause, budget, agent and company
configuration, tree holds, the producers listed below). The company pass
checks it within the 30-second gate, as before.
3. The write is a run event whose graph does not contain the task. That
same event armed the backstop. The backstop deadline stays scheduled
while the instance is quiet and checks the task at most 5 minutes later.
4. A run stops without any write (process lost, controller gone). The
existing deadline paths find it: `stale-issue-locks` waits for the
stored controller lease, and `active-run-silence` and
`orphan-active-leases` act on their own deadlines. Each of them
terminalizes the run, which is a run write and falls under 1 or 3. The
new test `restarts a task whose run vanished without a final event once
its lease lapses` covers this with the production backstop.
5. The process restarts or a quiet instance is woken by a restart. The
startup pass reconstructs every company.
6. A check that found a task waiting (grace, back-off, monitor) returns
its own deadline. That entry keeps its deadline whatever hints arrive.

Five minutes is the existing upper bound of recovery back-off in this
area: the maximum retry delay of a failed scoped-deadline check
(`scoped-deadline-work.ts`), and `DEFERRED_WAKE_PENDING_BOUND_MS`, the
time `assigned-work-sweep.ts` already trusts a deferred wake before it
calls a task stranded.

Producers that keep the company hint, and why. They can change
eligibility of tasks outside one task graph, or I could not show that
their effect stays in the graph. None of them fires once per run event
in the fixture.

- Eligibility for many tasks: company status (`companies.ts`), budgets
(`budgets.ts`), agent lifecycle and configuration (`agent-lifecycle.ts`,
`agent-configuration.ts`), instance settings (`instance-settings.ts`,
`*`), tree holds (`issue-tree-control.ts`), company deletion and import
(`company-deletion.ts`, `company-portability.ts`), and the hold releases
`pause_lifted`, `tree_hold_released`, `scheduling_resumed` and
`reconciler_backstop`. `accounting_settled` also keeps it when the
company has an active hard-stop budget policy.
- Task writes that can move a task between graphs or change eligibility:
task creation and a task update that changes the parent in `issues.ts`
(other task updates and checkout releases now name the task), document
delete and restore paths in `documents.ts`, task watchdogs
(`task-watchdogs.ts`), system and workspace recovery
(`system-recovery.ts`, `task-retry.ts`,
`workspace-recovery-capture-policy.ts`,
`workspace-recovery-stop-proof.ts`).
- Conversation and chat paths: `agent-conversations.ts`, the daily-cap
release in `heartbeat/queue.ts`, the Telegram attachment failure,
`chat-attachment-reuse.ts`, `native-runner-file-handoff.ts`,
`notifyChatRunMilestones` without a task or run (company authority
change), and `notifyNativeStatusReassessment` from `access.ts`.
- `writeExecutionControl` and `notifyExecutionRunWork` callers that pass
only a company.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. The checks
a run event causes now depend on its task graph, not on the company.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery`, `resolveIssueGraph`, the scope matcher and
decoder, `ISSUE_RECOVERY_MAX_ISSUE_IDS` (100) and
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes).
- `server/src/services/recovery/work.ts`: the task queues resolve a
graph hint when they load it (shared for 2 seconds), remember the last
full pass per company, and gate the backstop hint on it.
`registerIssueRecoveryWork` takes an optional `backstopMs`.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `issueIds`.
- `server/src/services/scoped-deadline-work.ts`: an optional
`backstop(scope)` that returns the earliest time of a backstop hint. A
pending backstop hint keeps its deadline but does not count as pending
work or as unreconciled.
- Producers: `issue-recovery-actions.ts`,
`execution-control-notifications.ts`, `run-dispatch-notifications.ts`,
`native-status-reassessment-notifications.ts`,
`external-conversation-settlement-notifications.ts`,
`work-intent-outbox.ts`, `run-cost-accounting.ts`,
`heartbeat/run-lifecycle.ts`, `heartbeat/runtime-execution.ts`,
`heartbeat/retries.ts`, `heartbeat.ts`, `legacy-execution-recovery.ts`,
`issues.ts` (checkout), `documents.ts` (task document writes),
`modules/wake-queue/adapters/postgres.ts` (recovery runs for a task),
`heartbeat/queue.ts` (wake enqueue), `recovery/service.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-safe-replacement.ts`,
`modules/run-dispatch/adapters/workspace-busy-wait.ts` (scheduled
retries).
- `bounds subsidiary scans for100 assignment candidates`: a resolved
recovery action now adds one scoped load per task queue (bound 8 to 11),
and the coalescing check sends one company hint before the burst of 100,
because the first full pass after the resolve is now a backstop pass
with its own key.
- `checks only blocked tasks whose blockers are done`: uses the shared
recording harness and runs out the backstop before its quiet check,
because the blocker's status change is now a graph hint (its 300
dependents overflow to a company load).
- Tests. `issue-recovery.test-fixture.ts` and
`live-recovery-work.test.ts` register the queues with `backstopMs: 0`,
so their quiet checks keep the earlier timing. The new tests use the
production value.

## Verification

New tests. I saw each red test fail before the change.

- `issue-recovery-deadlines.test.ts` `re-checks only the tasks a run
lifecycle event can affect, whatever the backlog` (red before): a run of
the agent on a task with a parent and a dependent, 1,000 backlog tasks
of another agent, and a second open task of the same agent. Seven real
producers (`notifyNativeStatusReassessment`, `notifyExecutionRunWork`,
`notifyRunDispatch`, `promoteDeferredWorkIntents` lock release and
accounting settlement, recovery action create and resolve) check exactly
the 3 tasks of the graph. The recovery action steps were red before the
third commit. With 4,000 more backlog tasks the event issues the same
number of statements. Before the change it checked 1,004 tasks.
- `resolves a run's task graph through index lookups, not the backlog`
(red before the second commit, from a Greptile finding): with 2,000
backlog tasks the graph lookup read 2,005 issue rows
(`pg_stat_xact_user_tables`) because it compared `id::text`. It now
validates and casts the saved id, reads fewer than 50 rows, and returns
exactly the run's graph.
- `finds a stranded task outside the affected set within the backstop
bound` (red before): a task inserted without a hint is not checked by
the run event, and is checked at most 5 minutes after the last full
pass. After 20 quiet minutes, a run event checks a new unrelated task
within 5 seconds, because the last full pass is older than 5 minutes.
Then the queues are quiet.
- `keeps the company pass for hold releases that can admit any task`
(guard, passes before and after): accounting settlement with a hard-stop
policy and a lifted pause still check every task.
- `falls back to the company pass when a run affects more tasks than a
hint names` (guard): a run task that blocks 100 more tasks checks the
whole company.
- `limits an ordinary comment to its issue among one thousand open
assignments`: the old final assertion (a company hint checks all 1,000
tasks) stays, and the test now also runs out the backstop and ends
quiet.
- `issue-recovery-assignments.test.ts`, real heartbeat, real producers,
production backstop (guards; they also pass with the old company hint):
a task whose run failed, a task whose run the system cancelled, a run
that vanished without a final event and is found when its controller
lease lapses, and an agent's queued task that starts when its previous
run ends. Each one recovers within 15 seconds, well before the backstop.
- `scoped-deadline-work.test.ts` `keeps a backstop hint's deadline
without holding the worker busy` (red before).

Suites, each file alone, in `server/` (issue recovery, live recovery,
recovery action and attention suites against dedicated local `_test`
databases): `issue-recovery-deadlines` (13/13),
`issue-recovery-assignments` (7/7), `issue-recovery-work-lifecycle`
(14/14), `scoped-deadline-work` (24/24), `live-recovery-work` (51/51),
`assigned-work-invariant` (32/32), `assigned-work-sweep-round-trips`
(2/2), `saved-work-recovery-overdue` (4/4),
`issue-recovery-dependencies` (5/5), `issue-recovery-conversations`
(3/3), `issue-recovery-actions` (112/112), `attention-retention-work`
(55/55). `recovery-deadline-acceptance` skips without its external
database. `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all` and instrumented counters (not committed). Fixture:
1 company, 60 agents, open tasks at 150 / 1,500 / 5,000 (52% in review,
39% blocked behind open blockers, 3% in progress, 2% to do), with
process agents that fail at once so recovery re-dispatches runs all the
time. The tables are analyzed after loading. Each window is 180 s,
starting 150 s after the server is ready, so each "after" window
contains one backstop pass. Base is `master` at 54c3616ad9; after is
this branch at dcd170e29f. The last commit (graph hints without the
30-second gate, task updates and checkout releases scoped) is covered by
the tests below but not re-measured. The machine was shared (load
average 9-16, one base window at 45). Run churn differs between runs, so
the table gives counts per run event (runs created in the window).

| Open tasks | Runs created (/s) base → after | Stranded checks per run
base → after | Assigned-work checks per run base → after |
`agent_task_sessions` reads (/s, 4 shapes, stranded check) | Statements
per run event base → after |
| --- | --- | --- | --- | --- | --- |
| 150 | 0.08 → 0.14 | 45.1 → 6.2 | 2.00 → 0.96 | 3.75 → 1.27 | 2,260 →
1,257 |
| 1,500 | 1.09 → 1.00 | 22.0 → 3.9 | 1.24 → 1.53 | 26.5 → 7.2 | 1,313 →
1,060 |
| 5,000 | 1.12 → 1.99 | 20.4 → 1.6 | 4.10 → 0.91 | 29.3 → 8.1 | 1,400 →
829 |

New statements: the task graph lookup (`with seed(id) ...`) at 0.8 / 9.3
/ 18.5 per second, and scoped loads of the task queues (`select "id",
"company_id" from "issues" ...` grew from 0.7 / 6.6 / 7.8 to 2.9 / 20.6
/ 34.4 per second, each a lookup of at most 100 ids). Stale-lock checks
grew from 0 / 50 / 52 to 22 / 190 / 445 per window: each run event now
checks the locks of its own task at once instead of waiting for the next
full pass. In an earlier round the graph lookup ran in every producer
transaction: 34/s at 1,500 and 70/s at 5,000.

With no activity the server issues no statements before and after this
change (quiet checks in `issue-recovery-deadlines`,
`issue-recovery-assignments`, `live-recovery-work`).

## Risks

- A task outside the graph of the run whose state the event changes is
now found by the backstop, at most 5 minutes later, instead of within 30
seconds. One known case: from #21427, `assigned-work-invariant` counts a
held recovery wake as pending while its agent has queued recovery runs.
When the last one starts, the continuation queue delivers the held wake;
the invariant check of that task waits for the backstop.
- A producer that names a task but changes something wider would now
miss the wider effect. Producers that change eligibility for many tasks
(pause, budget, agent and company configuration, tree holds) still send
the company hint.
- The graph is resolved after commit, from the written state. A parent
change is not a run event and keeps its company hint.
- The live-recovery and issue-recovery suites opt out of the backstop
deadline with `backstopMs: 0`.
- The execution-control deadline writes and legacy run terminalization
now pass the run id. `writeExecutionControl` also forwards that scope to
the live-run queues (`active-run-silence`, `native-process-sessions` and
others), so those hints now name the run instead of the company. Those
queues already handle run-scoped hints from native finalization.
- Test-only option: `registerIssueRecoveryWork(..., { backstopMs })`.
The shared recovery fixtures pass `0` so their quiet checks keep the
earlier timing; the new tests use the production value.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:29:22 -06:00
DottaandClaude Opus 5.5 c1d3f90b37 fix(recovery): keep the resolved-dependency probes per task whatever the statistics (#21574)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `resolved-dependencies` recovery queue loads the blocked tasks
whose blockers are all done (#21517). It runs this load on recovery
hints.
> - The load has two correlated subqueries on `issue_relations`.
PostgreSQL turns them into a semi join and an anti join, and lets the
statistics choose how.
> - After a bulk change in a company that the statistics have not seen
(a large import, a restore, a big cleanup), the planner estimates one
row for the company. It then picks nested loops whose inner side scans
every relation of the company and filters by task id. On a local copy
with 5,000 open tasks the query ran for more than 20 seconds (once 19
minutes) until the table was analyzed; after `ANALYZE` it takes 3.7 ms.
> - This pull request keeps both subqueries as per-task subplans, so
each one is an index probe by (company, task id) whatever the statistics
say.
> - The benefit is that the load stays proportional to the blocked tasks
it reads, also right after bulk changes. This is prevention: the hosted
instance does not show this query as slow today.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a local database that had been analyzed with a smaller dataset and
then loaded with 5,000 open tasks in a new company, the
`resolved-dependencies` load query did not finish within 20 seconds; in
a benchmark run it stayed active for 19 minutes after its server
stopped.

**Steps to reproduce**

1. Load a company with about 1,500 tasks and run `ANALYZE issues,
issue_relations`.
2. With autovacuum off for these tables, load another company with 5,000
tasks, 1,900 of them blocked with `blocks` relations.
3. Run the load query of `resolved-dependencies` for the new company.

Plan with stale statistics (abridged):

```
Limit
  ->  Nested Loop Semi Join  (rows=1)
        Join Filter: (issues.id = r.related_issue_id)
        ->  Nested Loop Anti Join  (rows=1)
              Join Filter: (r_1.related_issue_id = issues.id)
              ->  Index Scan using issues_company_id_uq on issues  Index Cond: (company_id = ...)
              ->  Nested Loop
                    ->  Index Scan using issue_relations_company_related_issue_idx on issue_relations r_1
                          Index Cond: (company_id = ...)
                    ->  Index Scan using issues_company_id_uq on issues blocker  Index Cond: (company_id = ...)
        ->  Index Scan using issue_relations_company_related_issue_idx on issue_relations r  Index Cond: (company_id = ...)
```

The index condition holds only `company_id`. The task id is a join
filter, so each blocked task rescans every relation (and every task, for
the blocker) of the company.

**Expected behavior**

Each blocked task probes its own relations through
`issue_relations_company_related_issue_idx` (company_id,
related_issue_id) and its blockers through `issues_company_id_uq`
(company_id, id).

## What Changed

- `server/src/services/recovery/work.ts`: the candidate predicate is a
named export, `resolvedDependencyCandidate`, and both subqueries end
with `offset 0`. PostgreSQL does not pull a subquery with `OFFSET` up
into a join, so it stays a correlated subplan. Its index condition then
includes the outer task id. The predicate and its results are otherwise
unchanged.
- `server/src/__tests__/resolved-dependency-candidates.test.ts`: two
tests.

Indexes that the subplans use (all existing):
`issue_relations_company_related_issue_idx` (company_id,
related_issue_id) for both relation probes, and `issues_company_id_uq`
(company_id, id) for the blocker status.
`issue_relations_company_issue_idx` (company_id, issue_id) is not needed
here.

## Verification

- `selects the same tasks as the earlier predicate`: 40 blocked tasks
with a done blocker, an open blocker, a done and a cancelled blocker, or
no blocker. The new predicate returns the same 10 tasks as the earlier
one.
- `probes the relations of each task, whatever the planner statistics
say`: analyzes a small company, then loads 3,000 blocked tasks in a new
company. The plan (`EXPLAIN (FORMAT JSON)`) has two subplans and no semi
or anti join, and the load returns 100 rows within a 5-second statement
timeout. I saw it fail first with the earlier predicate: the plan had a
semi join and an anti join.
- Local plan on the stale database: EXPLAIN ANALYZE shows `SubPlan 1`
and `SubPlan 2` with `Index Cond: ((company_id = issues.company_id) AND
(related_issue_id = issues.id))`, 1,933 probes each, 4.9 ms.
- `issue-recovery-deadlines.test.ts` and
`issue-recovery-dependencies.test.ts` pass (14 tests). `pnpm
check:module-boundaries` passes.

## Risks

- With fresh statistics the planner could use a merge or hash join for a
large page; the subplan form gives up that choice. Each blocked task now
costs two index probes, which is the same order as the fresh-statistics
plan (3.7 ms before, 4.9 ms after on 1,933 blocked tasks, measured once
each on a loaded machine).
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:28:55 -06:00
DottaandClaude Opus 5.5 3ae7aba643 perf(db): reuse dedicated connections instead of opening one per call (#21567)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server uses `withDedicatedDbConnection` (packages/db) when an
action must hold one database session outside the main pool: advisory
transaction locks, lock keepalive, and `pg_settings` reads for workspace
lifecycle, finalization, source locks, skill caches and status cards
> - Each call built a new postgres.js client, opened a new physical
connection (TCP, TLS and startup through the pooler), built a new
drizzle client with the full schema, and then ended all of it
> - On staging the dedicated application name shows about 21 connection
setups per second, and the main thread is pinned (event-loop utilization
0.965)
> - This pull request keeps a small set of idle single-connection
clients per owner and reuses them
> - The benefit is that a dedicated action costs about 0.2 ms of
main-thread time instead of 4 to 8 ms, with the same exclusive-session
guarantee

## Linked Issues or Issue Description

**What existing behavior does this improve?**

`withDedicatedDbConnection` opens and closes one new database client and
one new physical connection for every call.

**Subsystem affected**

Database client (`packages/db`) and the server services that take
dedicated advisory-lock sessions.

**Current behavior**

Each call builds a new postgres.js client, opens a new connection (TCP,
TLS and startup through the pooler), and builds a new drizzle client
with the full schema. On staging (Neon statement statistics, 326 s
window) `SET application_name='paperclip-workspace-finalization-lock'`
ran 21.4 times per second. Only this factory uses that application name.
In a 30 s staging CPU profile, drizzle `relations.ts` was 2.2%
inclusive, `n:net` 5.8% and postgres.js construction 0.3%. Part of that
is this per-call setup.

**Proposed behavior**

A caller still gets exclusive use of one session for the whole action.
The session and its drizzle client are reused by later actions, so the
per-call cost is the action's own queries, not a full connection setup.

**Reason and benefit**

The server main thread is saturated on staging (event-loop utilization
0.965). Reuse removes about 5 ms of main-thread work per call.

**Breaking changes**

None. The exported function keeps its signature.
`closeDedicatedDbConnections` and `DEDICATED_CONNECTION_IDLE_MAX` are
new exports.

## What Changed

- `packages/db/src/client.ts`: each `createDb` owner now has one set of
dedicated single-connection clients, shared with every private client
derived from that owner (so nested dedicated actions use the same set).
- An action takes an idle client (last in, first out) or opens a new
one. It never waits. Callers hold advisory locks and nest dedicated
actions, so a waiting queue could deadlock.
- On success the client returns to the idle set if fewer than
`DEDICATED_CONNECTION_IDLE_MAX` (4) are idle. Otherwise it is closed.
- When the action throws, the client is closed and never reused. This
covers a failed or cancelled transaction, a broken socket, and an
interrupted lock wait.
- When an action ends the client itself (the source-lock cancel path
calls `$client.end`), the client is never reused.
- Idle clients keep the owner's idle timeout and max lifetime, so
postgres.js still closes idle physical connections after
`DATABASE_IDLE_TIMEOUT_SECONDS`.
- New `closeDedicatedDbConnections(db, timeoutSeconds)` closes the idle
set and stops retention. Ending the owner `$client` does the same. A
later action still works with a connection that closes on release.
- `server/src/index.ts`: shutdown closes the dedicated set next to the
other private pools.
- `doc/DATABASE.md`: documents the idle cap, the "not an admission
limit" rule, and the transaction-scoped session state contract.
- Session state audit: every current caller uses `pg_advisory_xact_lock`
/ `pg_try_advisory_xact_lock` and `set_config(..., true)`. No caller
uses session advisory locks, `SET` without `LOCAL`, `LISTEN`, or
temporary tables. A reset round trip per call is therefore not added.

## Verification

- New `packages/db/src/dedicated-connection-pool.test.ts` (embedded
PostgreSQL, 10 tests).
- Red on `origin/master`: 200 sequential calls used 200 distinct
sessions (`expected 200 to be 1`). All 10 tests failed.
- Green with this change, 3 runs in a row: 200 sequential calls use 1
session; 7 concurrent callers each hold a different session for the
whole transaction, and afterwards at most 4 stay open and later calls
reuse them; a failed transaction's connection is closed; a terminated
socket during the action is not reused; an idle connection terminated by
the server reconnects; a client the action ended is not reused;
transaction-local settings do not leak; shutdown and owner `$client.end`
close every retained connection; nested actions do not block.
- Focused suites pass: packages/db `work-signals`,
`prepared-queries-postgres`, `response-capture-postgres`,
`checkpoint-pool`, `client`, `client-options`,
`client-teardown-registry`, `postgres-connection-recovery`,
`client-disconnect-replay` (132 tests). Server suites that use dedicated
connections: `skill-package-store`, `status-card-evaluation-work`
(updated: it now proves advisory locks are released, since sessions are
reused), `agent-instruction-working-copies`,
`workspace-recovery-retry-copyback`, `native-execution-lease-batch`,
`company-skills-empty-runtime`, `decisions-service`,
`legacy-controller-lease`, `server-startup-feedback-export` (467 tests).
- `pnpm verify --explain` selects 2779 files for a
`packages/db/src/client.ts` change. Per AGENTS.md I ran the focused
checks above and let CI give the broad coverage.
- Benchmark: 200 calls of `withDedicatedDbConnection(db, d =>
d.transaction(tx => tx.execute(select pg_try_advisory_xact_lock(42))))`
against local embedded PostgreSQL 18, 10 warm-up calls, 5 alternating
before/after process pairs per mode. Main-thread time is
`eventLoopUtilization().active`.

| Mode | Variant | Wall ms/call (median, range) | Main-thread ms/call
(median, range) |
| --- | --- | --- | --- |
| TLS | before | 35.55 (32.93–37.52) | 5.18 (4.68–7.95) |
| TLS | after | 0.19 (0.17–0.21) | 0.18 (0.17–0.20) |
| Plain | before | 18.39 (17.91–20.53) | 3.65 (3.58–4.30) |
| Plain | after | 0.17 (0.13–0.18) | 0.13 (0.12–0.14) |

At the staging rate of about 21 calls per second, this removes roughly
100 ms of main-thread work per second (TLS figures; staging CPUs may be
slower).

## Risks

- A retained connection that the server or network drops while idle:
postgres.js reconnects on the next query when it has seen the close
(tested). If the drop and the reuse race, the first query of that action
fails like a main-pool query would. Before this change, a dedicated
action always started on a fresh connection.
- Through a transaction-mode pooler, `SET application_name=...` is a
server-side parameter switch, so its count may not drop as much as the
client-side connection count. Measure the deployed effect with pooler
client connections and main-thread CPU, not only that statement.
- Concurrency above 4 opens extra connections exactly as before; they
close on release. Total retained connections per owner grow by at most 4
idle clients, which still close after the idle timeout.
- An action that sets session-level state would now leak it to the next
action. The contract is documented on `withDedicatedDbConnection` and in
`doc/DATABASE.md`; no current caller does this.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:37:04 -06:00
DottaandClaude Opus 5.5 c36b96ae00 fix(recovery): hold the start of queued automatic runs while a company's automatic-run brake is on (#21561)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The recovery system starts automatic runs: continuations, retries,
and sweep replacements
> - #21443 added a per-company automatic-run brake. It stops recovery
from creating new automatic runs while a company's runs keep failing
before they start
> - The brake did not hold automatic runs that were already queued. The
dispatcher still started them, so a burst of queued recovery runs kept
failing in the same way
> - This pull request adds the brake's start gate to the queued-run
dispatch pass, and wakes a held company at its next try without a
polling loop
> - The benefit is that a failing company stops spending runs on
automatic work, people's work still starts at once, and held work starts
again on its own when the brake releases

## Linked Issues or Issue Description

Refs #21443
Refs #21433
Refs #21427
Refs #21505

**What happened?**
While the automatic-run brake was open for a company, its already-queued
automatic runs (continuations and retries) still started. Each one
failed before it started, for the same cause that opened the brake.

**Expected behavior**
A queued automatic run must stay queued while the brake is open. Only
the brake's single probe, and then the half-open release rate (2, 4, 8
per window), can start one. A person's work (a Retry, or a continuation
of a person's request) must never be held. Held runs must start again on
their own, with nobody acting.

**Steps to reproduce**
1. Make 20 automatic runs in one company fail before they start, so the
brake opens.
2. Queue automatic continuations for that company.
3. Run a dispatch pass. Before this change, the continuations start.

## What Changed

- `server/src/services/heartbeat/queue.ts`: the dispatch pass asks
`reserveAutomaticRunStart` after the tried/backoff skip and the
claim-attempt budget check, and before the budget increment and the full
row read. A held run spends no claim attempt, does no row read, and does
not enter the declined-claim backoff. If the brake grants the start (its
probe or a half-open release start) but the claim then returns null,
throws, or finds the row gone, the pass gives the start back, so another
automatic run can use it in the same pass. The dispatcher's
person-origin answer is passed as is, and it is `undefined` only when
the dispatcher did not check origin (an agent with one queued run). Then
the brake's own cached person check decides.
- `server/src/services/heartbeat/queued-run-line.ts`:
`queuedRunDispatchFacts` also returns `personOriginChecked`, so the
queue knows when the dispatcher's negative answer is real.
- `server/src/services/heartbeat/dispatch-work.ts`: the queued-run
dispatch deadline worker adds companies whose next try is due to its
scopes, and includes the next try time in its next wake. A held company
wakes for its next try without a polling loop. If the brake's bounded
wake list had to drop a company, the worker wakes all companies at the
earliest dropped time.
- `server/src/services/recovery/automatic-run-brake.ts`: when the brake
holds a run, it records the company's next try time (probe time or the
next release window) in a bounded map. A probe or half-open release
grant returns a release that gives the start back with a compare-and-set
on its own run or window. It exports `reserveAutomaticRunStart`,
`takeDueAutomaticRunWakes`, and `nextAutomaticRunWakeAt`. The brake now
uses `isRecoveryRun`, `isPersonOwnedFailureCode`, and
`isRestartClassFailureCode` from `continuation-cause.ts` directly. The
rules are the same as before; the lazy loader keeps only the imports
that would make an import cycle.
- `server/src/__tests__/automatic-run-start-gate.test.ts` (new, 11 tests
through the real heartbeat queue): bounded queries for 250 held runs
over 100 passes, and for 1,500 held runs (past the 1,024-run cache) over
3 passes; a declined probe claim gives the probe back and another
automatic run starts in the same pass, and the probe stays with that run
on a later pass; an unused half-open start goes back to its window; with
276 held companies (past the 256-company wake list) every company is
still woken; a work run behind 250 held runs starts in the same pass; a
person's Retry is never held; the only queued run is not held when it
continues a person's Retry; exactly one probe starts across two server
instances; the release gives 2, 6, 14 cumulative starts over three
windows; held runs are not in the declined-claim backoff after the
pause; the dispatch worker wakes the company at the next try; and an
end-to-end test that a held automatic run stays queued while the brake
is open, then starts and succeeds on its own at the next try.
- Two small test updates in `automatic-run-brake.test.ts` and
`automatic-run-brake-import.test.ts` for the new cache size field and
the moved rule.

- `server/src/__tests__/automatic-run-brake-recovery-passes.test.ts`:
one test for the sweep path from #21505 (continue a blocked task behind
a run refused at dispatch). The sweep only records the continuation (no
wake, no new run), and `deliverReconciledExecutions`, which the brake
already gates, creates the run only after the pause. No second gate is
needed on that path.

## Verification

- `npx vitest run src/__tests__/automatic-run-start-gate.test.ts`: 11
passed.
- Red before for the review fixes: on the first PR head, the 1,500-run
cost test made 3,848 queries (bound 130), the declined-probe test
started 0 runs (expected 1), and the 276-company wake test missed 20
companies.
- Red before: with the `queue.ts` gate line removed, 5 of the 8 tests
fail (bounded pass, single probe, release rate, backoff, end-to-end).
- Red before for the person case: with the dispatcher's `false` passed
through as final, "never holds the only queued run when it continues a
person's Retry" fails (0 started). It passes after the change.
- `automatic-run-brake.test.ts` 19 passed,
`automatic-run-brake-import.test.ts` 1 passed,
`automatic-run-brake-recovery-passes.test.ts` 28 passed,
`failed-continuation-continues-again.test.ts` 32 passed (after merging
master with #21505).
- Nearby queue tests (on the final head): `heartbeat-run-dispatch-work`
15, `queued-run-dispatch-stacking` 6,
`heartbeat-queued-run-claim-isolation` 10,
`assigned-work-stalled-queued-run` 12, all passed.
- `server` `tsc --noEmit` is clean.

## Risks

- Behavior change only while a company's brake is open or half-open.
When the brake is closed, the gate returns from its cache with no query.
- The first time the gate sees a non-person automatic run while the
brake is on, the brake does one person-origin query for that run. The
result is cached per run (bounded to 1,024 runs).
- No migration. No API or UI change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:33:53 -06:00
DottaandClaude Opus 5.5 84cb28f9b7 fix(connections): keep machine sign-in secrets and settings in one locked commit (#21566)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21473 added machine sign-in (MCP OAuth client credentials) for
organization connections
> - Auto-merge squashed #21473 from an earlier head, so its last review
fixes did not reach master
> - Those fixes close a P1 race and two smaller setup gaps that Greptile
found on #21473
> - This pull request applies that last commit to current master, with
its tests
> - The benefit is that a client ID and its secret always stay together,
and key switching and app selection work as intended

## Linked Issues or Issue Description

Refs #21473

**What existing behavior does this improve?**
Machine sign-in setup and rotation, added in #21473.

**Problem**
- A connection test wrote a replacement client secret before checking
whether the connection changed. If the save was then refused, the secret
stayed and could pair with another client ID, so the next token request
failed.
- Switching a client-secret connection to a Paperclip key always created
an RS256 key, even when the sign-in service accepts only ES256.
- Machine sign-in dropped the setup's app ID and created a separate app.

**Proposed improvement**
Write the secret and the settings in one locked transaction after the
change check. Record the accepted signing algorithms and use them. Pass
the app ID through.

## What Changed

- `client-credentials-service.ts`: `saveConnection` locks the row,
checks that the sign-in method and credential references are unchanged
and that the connection is not removed, then writes any new credential
and the settings in the same transaction. Test, secret rotation and key
staging all use it.
- Setup records `tokenEndpointAuthSigningAlgValuesSupported`. Switching
from a secret to a key uses `chooseAssertionAlgorithm` on it.
- UI: `MachineSignInSetup` takes `applicationId`, and the custom-server
flow passes `prefill.applicationId`.
- Tests: two new server tests and one UI expectation.

## Verification

- `server/src/__tests__/tool-access-client-credentials.test.ts`: 25/25
pass. The 2 new tests fail on master's service code ("never leaves a
refused test's secret behind with another client ID", "creates the kind
of key the sign-in service accepts when switching from a secret").
- `MachineSignInSetup.test.tsx` and
`GenericMcpConnectStep.machine.test.tsx`: 8/8. The app ID expectation
fails on master's component.
- Server `tsc --noEmit` clean. UI `tsc` clean except the pre-existing
generated Storybook catalog imports.

## Risks

Low. The change is limited to machine sign-in. A concurrent change now
returns 409 `client_credentials_changed` instead of saving partly. No
migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:30:14 +00:00
DottaandClaude Opus 5.5 bbed32450c perf(load-model): model recorded native run rates, the recorded event mix and coalesced delta batches (#21549)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `scripts/benchmark-agent-scaling-load.ts` (#21418) estimates how
many concurrent agents one server process can carry
> - It assumed 20 text deltas/s and 1 other event/s per agent, and every
other event was an `item.completed`
> - Recorded hosted native runs show about 0.2 deltas/s and 0.5 other
events/s, with a mix in which 39% of the other events are server trace
spans that take a cheaper path
> - This pull request makes the recorded rates and mix the model's
default, keeps the old rates as an explicit stress profile, adds a burst
profile, and mirrors the delta batching of #21520
> - The benefit is a model that measures what production does, so the
next optimizations target the real per-agent cost

## Linked Issues or Issue Description

Refs #21418 (load model and plan). Refs #21520 (delta batching that the
model now mirrors).

**What existing behavior does this improve?**
The load model's default rates, its event mix and its delta batching
rule, and the plan document's measurements.

**Subsystem affected**
`scripts/benchmark-agent-scaling-load.ts`, its test
`server/src/__tests__/agent-scaling-load-model.test.ts`, and
`doc/plans/2026-11-13-thousand-concurrent-agents.md`.

**Current behavior**
The model defaults to 20 deltas/s and 1 other event/s per agent. Every
other event is a native `item.completed` with a chat-progress wake and a
checkpoint save. It cuts a delta batch on the timer even while the
previous batch commits.

**Proposed behavior**
- `PROFILE=recorded` (default): 0.2 deltas/s and 0.5 other events/s per
agent. `PROFILE=burst`: 2.5 and 3, a run's busiest minute.
`PROFILE=stress`: the earlier 20 and 1. API requests stay at 0.3/s. Each
rate can still be set on its own.
- `EVENT_MIX=recorded` (default) sends other events in the recorded
proportions. A `run.performance.span` is an ordinary append with no wake
and no checkpoint, as the server's run trace writes it.
`tool.execution.*` and `item.completed` are native appends with a
chat-progress wake, then a checkpoint save. `EVENT_MIX=<type>` sends one
type only.
- Delta batching follows #21520: deltas that arrive while a batch
commits all join one next batch.
- The report adds `profile` and `eventsIssuedByType`.

**Reason and benefit**
At the recorded rates the per-agent cost is the non-delta runner events
and their checkpoint saves, not deltas. The plan and the next
optimizations need that picture.

**Breaking changes**
None for the product. Anyone who runs the model with no options now gets
the recorded profile; `PROFILE=stress` gives the earlier load.

## What Changed

- Profiles, the recorded event mix (a 100-slot cycle per agent, offset
by agent so the fleet sends the mix at every moment), the span append
path, and the coalesced delta batches.
- The model test now checks the profile in the report and that both
server trace spans and runner events with a progress wake are in the
mix, for the recorded profile and for the stress profile with one
transaction per delta.
- The plan has a new section "Load model at recorded rates": the
recorded source (40 recent native runs on one hosted workspace, as
aggregates), the profiles, the mix and its server paths, fresh runs at
500, 1,000 and 2,000 agents plus burst and stress, and what they show.
It records the 3-connection checkpoint pool as a computed per-process
limit on a hosted database, and that SQL building in Drizzle costs about
as much main-thread time as result parsing on the hosted instance. A new
section "Considered and not done" records why `run.performance.span`
rows stay (about 3% of the statements per agent, with readers) and why
checkpoint saves per critical event only are not done now.

## Verification

- `server/src/__tests__/agent-scaling-load-model.test.ts` passes (2
tests). The new assertions fail against the previous script: it reports
no `profile` and no `eventsIssuedByType`.
- `server/src/__tests__/agent-scaling-load-model.test.ts` also checks
each profile's rates with no overrides, and that a rate set on its own
wins.
- Load model runs, recorded in the plan (60 s, main pool of 10, embedded
PostgreSQL on the same machine, each run started below a 1-minute load
average of 15): recorded rates at 1,000 agents (event-loop utilization
0.61) and 2,000 (0.85) finish every operation with no failure; 500
agents all in their busiest minute run at 0.95 with p99 lateness under
0.4 s. About 4.4 statements per agent per second at recorded rates, 35%
of them on the checkpoint pool.
- Earlier runs at load averages of 20–55 had shown one process falling
behind at 500 burst agents and at 2,000 recorded; the quieter runs after
the model fixes (event order, spans off the runner stream) do not, and
the plan says so. The checkpoint pool as a per-process limit is now
stated as a computed limit for a hosted round trip (about 400 agents in
a busy minute at 1 ms), not as measured.
- Greptile findings fixed: a waiting delta batch no longer takes deltas
sent after the next runner event, takes every delta queued when it
starts, and spans no longer join the runner event stream.

## Risks

- Low: a benchmark script, its test and a plan document. No product code
changes.
- The machine that measured the plan's table was shared; the plan marks
the rows that the machine distorted.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:13:07 -06:00
DottaandClaude Opus 5.5 67652580ee perf(tool-access): index enabled policies so each decision checks only candidates (#21563)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool access policies decide whether an agent can call each tool, and
an MCP tools/list response decides access once for each listed tool
> - Each decision matched the raw selector of every enabled policy two
times, so one listing cost tools x policies selector checks on the
server's main thread
> - On staging, a company with 363 enabled policies kept the main thread
at 95-98% of a core: policy matching was 22% of CPU time, with single 80
ms runs inside the selector match
> - This pull request compiles the enabled policies once for each
decision cache and indexes them by the selector key that rejects most
(catalog entry, then tool name)
> - The benefit is that each decision checks only the policies that can
match its tool, with the same decisions, so a listing no longer blocks
the event loop

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Tool access decisions for MCP tool listings
(`toolAccessPolicyService.decide`, called by `decideToolsForListing` in
the tool gateway).

**Subsystem affected**
Server: tool access policy evaluation
(`server/src/services/tool-access-policy.ts`).

**Current behavior**
Each decision loops over all enabled policies two times. The first loop
matches the selector and schema-checks the conditions of every
conditioned policy. The second loop evaluates conditions and matches the
selector of every policy. The selector match reads and allocates every
selector key in a fixed order, so a catalog entry mismatch is found only
at the end. A listing of 800 tools with 365 policies does 305,600
selector evaluations. On staging this was 22% of main-thread time
(event-loop utilization 0.965, delay p99 144 ms, max 710 ms).

**Proposed behavior**
The decision cache holds the enabled policies compiled once: parsed
selectors, an index by catalog entry and tool name, and a memoized
conditions schema check. Each decision evaluates only the candidate
policies, in the original priority and creation order.

**Reason and benefit**
Listing cost becomes proportional to the policies that can match each
tool. In the benchmark, one listing goes from about 164 ms to 12 ms of
main-thread time, and the longest event-loop block goes from 84 ms to 4
ms.

**Breaking changes**
None. The policy data model, API, and listing contract do not change.

## What Changed

- Add `server/src/services/tool-access/policy-selector-index.ts`. It
compiles a policy selector into a matcher with the same semantics as the
raw `selectorMatches`, and it indexes policies by `catalogEntryId`, then
by `toolName` (matched against the tool name and the upstream tool name)
for policies without a catalog entry selector. Policies with neither key
are candidates for every call. Candidates keep their original order.
- `decideInternal` caches the compiled index under the same
`enabled-policies:${companyId}` key. The new `matchEnabledToolPolicies`
keeps the two-step semantics: the first selected policy with unsupported
conditions denies the call before any match applies, then the selected
policies with matching conditions go to the unchanged decision loop
(block, rate limit, trust rule, require approval, allow, explanations).
- Decisions without a cache, and decisions that consume a rate limit
(which read fresh rows), compile the rows they read in the same way.
- The conditions schema check runs once per policy and cache, not once
per decision.
- Conditions are now evaluated only for policies whose selector matches.
Before, a malformed legacy condition on an unrelated policy could throw
in every decision. A selected policy with such conditions still gets the
unsupported-conditions deny first.
- Add `server/scripts/benchmark-tool-policy-matching.ts` and a section
in `doc/mcp-discovery-performance.md`.

## Verification

- Red/green regression:
`server/src/__tests__/tool-access-policy-index.test.ts` "reads policy
selectors once per decision batch, however many tools it decides". It
decides 40 tools with one cache against a staging-shaped set of 365
policies and counts reads of the raw policy selectors. On
`origin/master` it fails: 308,110 reads for 40 tools against 7,711 for
the first tool. With this change the count stays at the first-tool
count. The test also checks that cached and uncached decisions are
equal.
- Equivalence: the same file compares the compiled selector with the raw
`selectorMatches` for 404 selectors x 300 contexts (including empty,
whitespace, non-string, string-plural, duplicate and contradictory
values). It compares the indexed policy selection with the old unindexed
two-step selection (copied into the test) for 5 random sets of 400
policies x 400 contexts, and for the staging-shaped set over 800 tools.
The sets include catalog-entry-only, catalog-entry+connection,
multi-key, tool-name, unindexed, conditioned, unsupported-condition,
block, rate_limit, trust_rule, allow and legacy policy types with mixed
priorities.
- Index edge cases: an earlier tool-name policy (matched through the
upstream name), an unindexed policy and an empty catalog-entry selector
apply before a catalog-entry block; an unsupported-condition policy in
another bucket still denies. A DB-backed test checks the same through
`decide`, with and without a cache.
- Focused suites pass locally: `tool-access-policy-index`,
`tool-access-policy-service`, `tool-gateway-listing-memory`,
`connector-tool-policy`, `tool-gateway`, `tool-gateway-service` (6
files, 406 tests). `node server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a broad input), so
CI supplies the broad coverage.
- Benchmark, local embedded PostgreSQL, macOS arm64, Node 24. One
listing of 800 tools, 365 enabled policies (staging shape plus the 2
fixture policies), 16 concurrent decisions with one cache as in the
gateway. Three runs of 9 alternating rounds after 2 warm-up rounds. Run
3 uses the event-loop delay histogram that also covers the end of the
listing. Both versions return identical decisions.

| Per listing | `origin/master` median (min-max) | This PR median
(min-max) |
  | --- | --- | --- |
  | Elapsed, run 1 | 157.5 ms (146.1-192.8) | 11.4 ms (10.3-15.3) |
  | Elapsed, run 2 | 165.3 ms (161.7-185.4) | 11.9 ms (10.1-14.0) |
  | Process CPU, run 1 | 160.5 ms (149.7-188.9) | 13.7 ms (11.1-21.0) |
  | Process CPU, run 2 | 162.8 ms (157.2-192.7) | 12.3 ms (9.6-20.5) |
  | Elapsed, run 3 | 163.9 ms (151.6-185.3) | 12.4 ms (10.4-18.5) |
  | Process CPU, run 3 | 165.4 ms (155.7-214.9) | 14.5 ms (10.6-20.2) |
| Longest event-loop block, run 3 | 84.1 ms (76.9-95.1) | 3.7 ms
(3.0-5.0) |
  | Selector evaluations | 305,600 | 1,164 |
  | Raw selector property reads | 6,155,830 | 9,490 (compile only) |

  ```sh
cd server && pnpm exec tsx scripts/benchmark-tool-policy-matching.ts
--baseline-ref origin/master --tools 800 --rounds 9
  ```

- Not verified: the hosted staging profile after deploy.

## Risks

- Low risk. The decision loop is unchanged; only the set of policies
that reach it is narrowed, and every left-out policy has a selector that
rejects the call. The equivalence tests compare against the old
selection.
- The index lives only in the per-listing decision cache (or for one
uncached decision). There is no cross-request cache, so policy edits
apply on the next listing as before.
- The cached index memoizes the conditions schema check on shared
entries. The check is pure, so concurrent decisions in one listing get
the same result.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test and benchmark runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:51:43 -06:00
DottaandClaude Opus 5.5 9824277f65 perf(run-events): append events whose only wake is chat progress in one statement (#21559)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Most runner events that are not text deltas
(`tool.execution.started/progressed/completed`, `item.completed`) wake
the chat progress projection
> - That wake is registered inside the append's transaction, so each
such event takes 4 statements (BEGIN, the run lock read, the append,
COMMIT), where an event without a wake takes 1 (#21511)
> - The chat progress projection rebuilds itself from the stored events;
the wake only tells it when to look
> - This pull request writes an event whose only wake is chat progress
with the one-statement append, and sends the wake after that statement
commits
> - The benefit is 1 statement instead of 4 for most non-delta runner
events, with response, reassessment, retention and recovery wakes
unchanged

## Linked Issues or Issue Description

Refs #21511 (one-statement append). Refs #21528 (lock read returns the
transaction id). Refs #21549 (load model at recorded rates).

**What existing behavior does this improve?**
`appendHeartbeatRunEvent` for native source events whose only wake is
chat progress.

**Subsystem affected**
Server run event ingest (`heartbeat-run-events.ts`), the chat progress
notification helper, and two additive exports in
`packages/db/src/work-signals.ts`.

**Current behavior**
A progress event opens a transaction, reads the run lock with
`runtime_mode`, the bound task and the transaction id, probes for a
replay, registers the chat-progress wake as a tracked intent, appends,
and commits.

**Proposed behavior**
When chat progress is the event's only wake, the append takes the
one-statement path of #21511. That statement also returns the run's
`runtime_mode` and bound task id from the row it locks, and, where
commit notifications are on, calls `pg_notify` inside the same statement
for runs that are native and bound to a task, so other server processes
get the wake at commit. After the statement returns, this process
publishes the wake as settled to its own listeners. A replay, a gap, a
changed run or any other fallback takes the existing transaction, which
registers the wake as before.

**Reason and benefit**
At recorded rates these events are about 61% of the non-delta events
(#21549), so this cuts about 3 statements per such event.

**Breaking changes**
None. No schema change.

## What Changed

- `appendHeartbeatRunEvent`: `progressOnly` (progress set, every other
wake clear) also qualifies for the one-statement path.
- `APPEND_NEW_NATIVE_SOURCE_EVENTS_WITH_PROGRESS_SQL`: the one-statement
append, plus `runtime_mode` and the bound task id (`boundRunIssueIdSql`,
rendered once) from the locked row, plus a conditional `pg_notify`.
- `chat-native-progress-notifications.ts`:
`chatNativeProgressCommitNotification` and
`publishChatNativeProgressSettled`.
- `work-signals.ts` (additive): `databaseWorkCommitNotification(owner,
topic)` returns the cross-process payload when notifications are on;
`publishDatabaseWorkSettled(owner, topic, hint)` publishes a settled
wake to this process's listeners.

## Verification

- Red/green on embedded PostgreSQL. Each new test failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "appends an event whose
only wake is chat progress in one statement and wakes progress after it
commits": 1 statement (before: a transaction), the wake is published
once as settled with no intent, and a replay publishes nothing.
- "keeps the transaction for the last events of a turn and a run, and
one statement for progress-only events":
`tool.execution.started/progressed/completed` and a non-final
`item.completed` take one statement; a final `item.completed`,
`run.result.accepted`, `turn.completed`, `turn.failed`, `turn.cancelled`
and `turn.interrupted` keep the transaction.
- "sends the progress wake to other server processes with the one
statement's commit": with commit notifications on, another connection
that listens on the work channel receives exactly one chat-progress
notification, and a replay sends none.
- `chat-native-progress-work.test.ts` "wakes the projection from a
one-statement progress event after the statement commits": the real chat
progress worker publishes the progress for an event appended in one
statement.
- Mutation check: without the settled publish after the statement, that
test fails.
- Lost-wake repair (condition b): an event row stored without any wake
models a lost hint.
- "repairs a lost progress wake with the run's next progress event":
nothing is published for the stored event; the run's next progress event
publishes the progress.
- "repairs a lost progress wake with the startup reconcile": a worker
that starts after the stored event publishes it.
- The existing response-wake test now uses a final reply and still shows
the intent before the write, and the race test of #21528 uses a final
reply so it still covers the transaction path.
- Other focused suites, one file at a time: the port suite with
conformance (105), `native-observer-replay` (1),
`paperclip-semantic-conformance` (1), `native-interaction-bridge` (7),
`heartbeat-run-event-sequencing` (5), `native-event-boundary-golden`
(1), `heartbeat-native-restart-suspension` (115), `live-recovery-work`
(51), `work-signals` (25), atomic-append (31),
`chat-native-progress-work` (38). `pnpm --filter @paperclipai/server run
typecheck` and `pnpm check:module-boundaries` pass.

- CPU per operation (a local script, not committed: 800
`tool.execution.progressed` appends to one native task run on embedded
PostgreSQL, one connection, three rounds, `origin/master` code against
this branch): main-thread CPU per event 0.28–0.50 ms before and 0.10 ms
after in the warm rounds; wall time per event 0.84–1.04 ms before and
0.52–0.53 ms after.
- Inside a caller's transaction: new test "keeps a progress-only event
inside a caller's transaction on the locked path, and settles its wake
only when that transaction ends". `publishDatabaseWorkSettled` publishes
at once only outside a transaction; inside one it waits for the
outermost commit and drops the wake on rollback (new
`work-signals.test.ts` test, which failed with the unguarded publish).
The one-statement path is only taken when the owner is not a transaction
(`!is(db, PgTransaction)`, #21511), so the two guards agree.

## Risks

- **Lost wake.** If the statement's outcome is unknown (the connection
drops after the server ran it), this process sends no wake for that
event. In a chat channel (Slack, Telegram) the progress line for that
task can then stay one step behind (for example it does not move from
"working" to the latest tool step) until the run's next runner event, a
run milestone, or a server restart. A turn normally ends with an event
that keeps the transaction
(`turn.completed/failed/cancelled/interrupted`), and the run's
completion notifies chat run milestones, which include chat progress
(`notifyChatRunMilestones` in `run-completion.ts` and
`native-run-finalizer.ts`). So progress cannot stay stale after a run
ends; in the worst case it lags one step for the rest of a turn. The
projection rebuilds from stored events, so nothing is lost or
duplicated.
- The chat-progress wake no longer publishes an intent before the write.
Intents let the coordinator fence a pass against an in-flight write; the
settled wake after the commit makes the projection run again after the
row is visible.
- `packages/db/src/work-signals.ts` gets two additive exports and no
change to existing behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:37:35 -06:00
DottaandClaude Opus 5.5 6b9e45787f fix(recovery): continue a task whose system retry was refused at claim behind a saved-work hold (#21505)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops before it finishes, automatic recovery continues
the task, through the fresh-continuation fallback, in a new run that
starts from the saved work
> - A follow-up run (for example one started by a new comment or by
finished subtasks) can fail before any provider work and leave a
saved-work hold. The system then queues its own retry, and the dispatch
gate cancels that retry at claim with
`execution_reconciliation_required` because the hold is still in place
> - The fallback takes over the hold when that happens, but it continued
only "recent interruptions". That test required the failed run to be a
recovery run itself or to fail with a restart code. A first follow-up
run that failed with `adapter_failed` passed neither test, even though
the system had already decided to retry it. So the fallback answered "a
person must press Retry". The backlog pass used the same test, and the
assigned-work sweep never listed the task. The task stayed blocked with
no run, no pending path and no Retry button
> - This pull request treats a system retry refused at dispatch as proof
that the system continues the task. It also stops cancelled-at-dispatch
attempts from counting against the hourly budget, lets the sweep
continue a blocked task whose latest run was refused at dispatch, and
keeps recovery notices from asking for a Retry that the task does not
offer
> - The benefit is that such a task continues by itself in exactly one
fresh run from the saved work, whether the saved work was recovered
before or after the refusal, and a person's own failed Retry keeps
today's behaviour

## Linked Issues or Issue Description

Refs #21394, #21348, #21363, #21478

**What happened?**
A task's follow-up run, a legacy run that was not a recovery run, failed
before any provider work with `adapter_failed`. Its files could not be
restored, so it left a saved-work hold
(`legacy_execution_requires_reconciliation`, `workspaceRestoreFailure:
restore_failed`). The system queued a retry of that run. The dispatch
gate cancelled the retry at claim with
`execution_reconciliation_required`, and the "older run hold" path
handed the hold to the fresh-continuation fallback
(`run-dispatch/adapters/postgres.ts` `settleOlderRunHold`). The fallback
found the refused run and asked `interruptionEligibility` whether the
interruption was recent. That predicate required
`continuesAutomaticallySql(source)` (a recovery run, or a restart,
deploy or bootstrap code), so it returned `person_owned` /
`retry_needed`. The stranded reconciler then blocked the task with
"Press Retry", but the saved-work hold reports that Retry is not
available. This happened twice in a local restart-chaos wave. In one
case the saved work was recovered two minutes before the claim. In the
other it was recovered thirty seconds after the claim.

**Expected behavior**
A task never rests blocked with no live run, no pending path and no
Retry. While the saved-work recovery runs, that recovery is the pending
path. When it finishes, one continuation starts from the saved work
through the fallback (isolation, the shared hourly cap). A run that was
cancelled at dispatch before it started does not use up the automatic
budget. A person's own failed Retry is still left for that person.

**Steps to reproduce**
1. Fail a follow-up run (wake reason `issue_children_completed`) before
provider work with `adapter_failed` and `workspaceRestoreFailure:
restore_failed`.
2. Queue the system's retry of that run (`retryOfRunId`, wake reason
`issue_continuation_needed`) and claim it. The dispatch gate cancels it
with `execution_reconciliation_required`.
3. Before this change, no continuation is recorded, the stranded
reconciler blocks the task, and nothing ever continues it.

**Paperclip version or commit**
master at 2756be2f32

## What Changed

- `fresh-continuation-fallback.ts`, `interruptionEligibility`: the
interruption counts as one the system continues when the failed run's
cause is system-owned and a later run that the dispatch gate refused
before it started is a system recovery run (`recoveryRunSql`, which
excludes a person's Retry). A failed run that was itself a person's
Retry (`task_retry` or `explicitUserContinuation`) never qualifies this
way. The 24-hour recency bound is unchanged. The claim-gate hand-off,
the backlog pass and its deadline query all use this one predicate.
- `recovery/continuation-ledger.ts`,
`recovery/task-automatic-attempts.ts`: one shared rule,
`startedOrPendingAutomaticRetry`. An automatic retry that the dispatch
gate cancelled before it started (`cancelled`, no `startedAt`,
`execution_reconciliation_required`) does not count toward the hourly
cap in either ledger.
- `recovery/assigned-work-sweep.ts`:
- When the latest run was refused at dispatch, the sweep continues from
the run it was refused for through the fallback. Before, it gave up.
- A blocked task with an active hold and a recent refused-at-dispatch
latest run is now listed as a suspect. The check hands it to the
fallback before it answers `left_to_hold_owner`. A pending recovery
becomes `waiting_on_recovery` with its recheck time.
- Holds that wait for a person's configuration or confirmation are still
excluded.
- Review follow-up (`recovery/task-automatic-attempts.ts`,
`recovery/assigned-work-sweep.ts`):
- The task ledger counts a sweep's fresh-continuation record (`trigger:
assigned_work`) itself. It no longer depends on a companion activity
entry, and an activity entry marked `continuedFromSavedWork` is not
counted a second time. Every sweep continuation counts once, whatever
path recorded it.
- When the sweep claims a continuation, it marks its claim entry
`continuedFromSavedWork` inside the transaction that records the fresh
continuation. A restart between the two writes can no longer leave one
attempt counted twice.
- The sweep's continuation behind a refused run spends the per-minute
repair budget (`ASSIGNED_WORK_REPAIRS_PER_MINUTE`). The excess is
deferred with a `retryAt`.
- `recovery/stranded-notice.ts`, `recovery/service.ts`: when the task's
execution blocker offers no Retry, the escalation notice and the
recovery action's next action leave out every Retry instruction. They
say that Paperclip is holding the task until its saved work is settled.
- No change to the dispatch gate (`heartbeat/queue.ts`, the restore-hold
predicate in `run-dispatch/adapters/postgres.ts`, or
`execution-blocker.ts`). The recorded continuation settles the hold, so
the delivered run is not refused again.

## Verification

Red before the change, green after (embedded Postgres, real services:
`terminalizeLegacyExecution`, the dispatch adapter's
`cancelStaleQueuedRun`, `reconcileStrandedAssignedIssues`,
`deliverReconciledExecutions`, `continueHeldExecutions`,
`recoveryService().assignedWork`):
- `server/src/services/fresh-continuation-fallback.test.ts`
- "continues a follow-up … when the saved work was recovered before the
system retry was refused at claim": red (`continuationDelivery:
"invalidated"`, no continuation). Green: the continuation is recorded at
the claim, and the stranded reconciler does not block the task. The
delivered run is queued and passes the claim gate (`not_stale`).
- "… recovered after the system retry was refused at claim": red (no
pending check). Green: the claim records `exact_recovery_pending` /
`saved_work_recovery_running`. When the recovery completes, the backlog
pass continues the task, and the delivered run passes the claim gate.
- "continues a follow-up … with its files in place, after its system
retry was refused at claim": red, then green. This is the control
without a restore failure.
- "still leaves a person's failed Retry of a follow-up for that person
…": stays `person_owned` / `retry_needed`, before and after.
- "counts only automatic retries that started, or still may, against the
task's hourly budget": new.
- `server/src/__tests__/failed-continuation-continues-again.test.ts`,
"the sweep continues a blocked task whose system retry was refused at
claim behind a recovered saved-work hold": red (the task was not a
suspect). Green: `repaired`, a pending continuation in the hold, the
task back in `todo`, and no wake enqueued. The next check reports a
pending intent.
- Same file, "a continuation the sweep records behind a refused run
counts once toward the task's hourly cap": red (5 counted after 5
attempts plus the new continuation), green (6, and the transient-retry
cap check refuses the next attempt).
- Same file, "continuations behind refused runs spend the sweep's
per-minute repair budget": 20 tasks. Red (20 continued in one pass),
green (10 continue, 10 are deferred with a `retryAt`, and the 10
continue in the next pass).
- Same file, "a sweep continuation interrupted right after it was
recorded still counts once": the continuation commits, then the
post-commit activity write fails (a simulated server stop). Red: 6
counted for 5 real attempts. Green: 5 counted, and the next automatic
retry is still allowed.
- `server/src/services/recovery/stranded-notice.test.ts`, "never tells a
person to press Retry when the task offers no Retry": new.

Local runs on this branch, one file at a time:
- `fresh-continuation-fallback.test.ts`: 77 passed. The suite's
`afterAll` teardown hit the 30 s hook timeout on a heavily loaded
machine.
- `failed-continuation-continues-again.test.ts`: 32 passed
- `assigned-work-invariant.test.ts`: 31 passed
- `stranded-notice.test.ts`: 33 passed
- CI on the first head (ce4431410b): every required check passed
- `server` `tsc --noEmit`: clean
- `pnpm verify --explain` selects most of the repository (2666 files),
so required CI supplies the broad coverage.

## Risks

- More tasks continue automatically. Each continuation goes through the
existing fallback. The 24-hour recency bound, the per-task hourly cap
(6), isolation of the next run, and every person-owned gate
(person-owned failure codes, a person's Stop, review, a person-owned
hold, a person's failed Retry) are unchanged.
- The hourly cap now ignores automatic retries that never started. A
task whose retries are refused at dispatch again and again is still
bounded: the refusal hands the hold to the fallback, and the fallback
records one continuation per source run.
- Known separate gap, not changed here: after an interrupted
preparation's holds are resolved (`settleInterruptedNativeBootstrap`),
`scheduleBoundedRetry` can report `exhausted` or `blocked`, and nothing
follows that up. The assigned-work sweep may now pick such a task up
through the refused-at-dispatch path, but this pull request does not
prove that.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:01:39 -06:00
DottaandClaude Fable 5.2 689f3b190e fix(recovery): pause automatic recovery runs while a company's runs fail before starting (#21443)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Automatic recovery continues interrupted and failed tasks without a
person, with a per-task limit of 6 automatic attempts per hour
> - There is no company-level limit. When the sandbox provider refuses
to create sandboxes, every task fails the same way, and recovery keeps
creating new runs for each task
> - On a hosted instance, one hour of a provider outage made 510
automatic runs, 587 failures and 0 successes across 289 tasks. No task
had more than 4 runs, so the per-task limit did not engage. Each start
asked a refusing provider for a sandbox and used server memory, which
made the outage longer
> - Recovery must still continue every task by itself. The answer is to
wait while the infrastructure fails, not to give the task to a person
> - This pull request adds a company-scoped pause for automatic work: it
opens when recent automatic runs fail before the agent starts, the
recovery passes stop creating new automatic runs while it is open, and
it closes by itself when runs start normally again
> - The benefit is that an infrastructure outage no longer multiplies
itself through recovery, and recovery resumes with no person acting

## Linked Issues or Issue Description

**What happened?**
During a sandbox provider outage, automatic recovery created hundreds of
new runs per hour across hundreds of tasks. All of them failed before
the agent did any work. The per-task limit did not engage because no
single task reached it.

**Expected behavior**
When most recent automatic runs of a company fail before they start,
recovery waits instead of creating more runs. It tries again at a small
rate. It resumes at full rate when runs start normally again. A person's
work is never held.

**Steps to reproduce**
1. Make the sandbox provider refuse lease creation (HTTP 503) for a
company with many assigned tasks.
2. Let recovery continue the failed tasks.
3. Count automatic runs created per hour: it grows with the number of
tasks, and every run fails at sandbox creation.

Refs #21394 (continuation classification this change will reuse when it
lands), #21433 (start-pass rewrite that the start hook will build on).

## What Changed

- New table `automatic_run_brakes` (one row per company, cascade delete
with the company; migration `0480`). Migration `0481` builds
`heartbeat_runs_company_finished_idx (company_id, finished_at desc)
where finished_at is not null` concurrently, so the signal reads the
runs that finished in the window directly. It stores the state (closed,
open, half_open), the reason with the window numbers, and the
compare-and-set fields that two servers share. A company with no row is
closed.
- New module `server/src/services/recovery/automatic-run-brake.ts`:
- Signal: on the database clock, the automatic runs of the company that
finished in the last 10 minutes (and since the last state change),
newest 500 by finish time, plus up to 200 running runs. A run counts as
a failure only when it ended in its own start phase: no provider work
(`executionRecovery.providerWorkStarted = false` or the native
`nativePreDispatchFailure` record), a legacy run still preparing, or a
transient setup code. A run interrupted by a server stop, restart or
lease loss is neutral, so a release swap does not open the pause. A run
that succeeded, or ended for a person-owned or task reason, counts as
healthy.
- Opens at 12 or more counted runs with 80% or more failures, or on 5
sandbox refusals (`environment_provider_unavailable`) in a row.
- Each outcome counts once, at its own time: a finished run at its
finish time, a running run that got past setup at its start time. Read
time never orders evidence.
- Closes in two steps: open moves to half_open after at least 2 minutes
when the newest 2 counted runs are healthy or the newest one succeeded;
half_open releases at a doubling rate per minute (2, 4, 8, 16, 32, 64)
and then closes. It reopens from half_open if the signal trips again. It
closes by itself after 15 minutes with no automatic activity.
- Reads are one indexed query per company, cached for 5 seconds in a
bounded in-process map (256 companies). Each state change writes one
structured log line
(`automatic_run_brake.opened|half_open|closed|reopened`) with the
numbers.
- It also contains the start gate `holdAutomaticRunStart` (one probe at
a time, at most one new probe per 60 s, release budget per window by
compare-and-set, no I/O while the cached "held until" time has not
passed, person origin taken as an argument with a cached fallback). This
pull request does not call it from the start pass; see Risks.
- Recovery passes defer creating runs while the pause is open. The gate
is only where a run is created:
- `deliverReconciledExecutions`: after the agent-room check and right
before the wake, the delivery asks for a creation slot; when the pause
holds it, the continuation stays pending. The held backlog
(`continueHeldExecutions`) is not gated: recording a pending
continuation creates no run and costs nothing, and its delivery passes
this gate once.
- Assigned-work sweep `act()`: after the process rate limit and before
it claims a continuation, it asks for a creation slot; when held, it
returns `deferred_rate_limit` with the next try time, so nothing is
written to the per-task attempt ledger. New assignments with no earlier
run are not deferred.
- A reserved slot is given back (compare-and-set on its own timestamp)
when no run is created: the wake is refused or throws, the task is
capped, the claim changed, or the task went back to review. The
give-back is conditional in SQL on no run for that task having been
created since the reservation, so a wake that commits its run and then
throws keeps the slot.
- The sweep's overdue saved-work recovery path returns before the slot
is asked for: it starts a recovery attempt, not a run, and is never
held.
- Person work is never deferred. The gate is called only at the delivery
(`agentHasRoom`) and the sweep (`act()`), and each call passes the task
and the run it continues. While the pause is open, the gate does not
hold the record when any of these is true; it holds only when none is.
With no task or no source run it does not hold (fails open).

| Shape | How it is recognised |
| --- | --- |
| A Retry or "move forward" waiting for the task
(`deferred_issue_execution`, `queued` or `claimed`, no run yet) | an
unstarted wake request from a user for the task |
| A Retry folded (`coalesced`) into the continuation, or a Retry while
the delivery is pending | a wake request from a user for the task,
requested after the source run finished (any status) |
| A person's message or a new assignment by a person after the last run
| the same: a wake request from a user for the task after the source run
finished |
| A held task selected because a person pressed Retry (the refused Retry
run) | a run for the task created after the source run finished whose
wake is a person reason (`task_retry`, chat message), or that has
`taskRetryRequestId` or `explicitUserContinuation` |
| A continuation of a person's Retry (any depth) | the source run's
chain reaches a person's run (`listPersonOriginSourceRunIds`, the
dispatch rule) |
| A Retry during pending delivery that becomes the continuation itself
(`deliverContinuationAsRun`) | not gated: that run is created by the
person's own wake, never by these passes |
| The held backlog's saved-input path | not gated: recording or
releasing a held continuation creates no run here |
- To prevent a deadlock, a pass may still create one automatic run per
minute per company when no automatic run of the company started in the
last 2 minutes, however many runs wait (compare-and-set, shared by
servers).
- `GET /api/companies/:id/assigned-work-invariant` now returns
`automaticWork` with the state, the time it paused, the window numbers
and a plain sentence, for example "Automatic work is paused for this
company since T because 47 of 50 recent automatic runs failed before
starting."
- `listPersonOriginSourceRunIds` in `heartbeat/queued-run-line.ts` is
now exported, unchanged, so the gate uses the same person-origin rule as
dispatch.
- The outcome rules (person-owned codes, restart codes, transient setup
codes, native pre-dispatch record) come from their owning modules
through one memoised dynamic import. A static import of
`heartbeat/retries.ts` made a module cycle at server start.

### What the pause consumes, and how waste is bounded

| Token | Taken | Given back or bounded |
| --- | --- | --- |
| Creation slot (one per minute per company while open) | right before
the run is created, after every other check | released only when no run
for the task was committed after the reservation |
| Probe slot (start gate) | when the start gate lets one automatic run
through | another run may take it when the reserved run has not started
within 10 seconds (claim declined, agent paused or full) or was
cancelled, deleted or finished |
| Gradual-release start (start gate) | one per run the start gate lets
through | a declined claim spends one start of that window; the window
ends within 60 seconds and the next one doubles |
| Probe success count | once per probe run, by compare-and-set on its id
| a probe run counts once; cancelled or deleted probes count as neither
success nor failure |

### When the pause opens, stays open and closes

| State | Leaves when | Bound |
| --- | --- | --- |
| closed | opens when, in the last 10 minutes since the last change, 12
or more counted automatic runs finished and 80% or more failed before
starting, or the newest 5 counted runs were sandbox refusals | none
needed |
| open | moves to half_open when it has been open 2 minutes or more and
the newest 2 counted runs are healthy, or the newest one succeeded |
evidence arrives at least once a minute: when no automatic run started
in the last 2 minutes, a pass may create one per minute |
| open | closes when no automatic run started or finished for 15 minutes
(queued runs do not count) | 15 minutes after the last activity |
| open | stays open only while new automatic runs keep failing before
starting | each new try is spaced by the existing passes and by one
creation per minute when nothing starts |
| half_open | closes after 6 release minutes (2, 4, 8, 16, 32, 64) | 6
minutes |
| half_open | reopens when the signal trips again, counted only since
half_open began | none needed |

Restart-class endings (server stop, restart, lease loss, process loss,
interrupted native session) are neutral in every state, so a release
swap does not open the pause.

## Verification

- `npx vitest run src/__tests__/automatic-run-brake.test.ts` (14 tests,
pass): classification; opens on 20 failures before start; 50 queued
automatic runs give exactly one probe and no failure or attempt; a
person's Retry, a new assignment, a run a person joined and the
continuation of a person's Retry are not held; mixed and person-owned
outcomes stay closed; old data is ignored; 40 working runs interrupted
by a restart stay closed, then the same continuations failing at sandbox
acquisition open it; 5 refusals in a row open it; failed probe waits for
the interval, two probes past setup move to half_open, the release gives
exactly 2, 4, 8, 16, 32, 64 starts per window and then closes; reopen
from half_open; probe slot frees after cancel, delete or no start; quiet
close after 15 minutes; two instances on one database agree and start
one probe; one creation per minute while open across servers; 250 held
runs over 100 passes use at most 5 queries (at most 255 with the
person-origin fallback, one per run, once); caches stay at 256 companies
and 1024 runs.
- `npx vitest run
src/__tests__/automatic-run-brake-recovery-passes.test.ts` (pass; one
test per person shape above at the delivery and the sweep, with the
pause open and no creation slot, plus a control that still holds a
continuation no person asked for): includes "no pause record: every pass
behaves as before", "the waiting run cannot start: a pass continues a
task on another agent, then the pause closes", "one held task spends one
creation slot", and "a person's Retry or message is never paused at the
delivery, the sweep or the held backlog".: sweep, delivery and held
backlog create nothing while open, write no attempts, and continue by
themselves after recovery; the sweep still continues one task a minute
when nothing automatic waits. With the three deferral lines removed, all
4 fail (red), and they pass with them (green).
- The pause reads as closed and logs once when its run rules cannot load
(test in `automatic-run-brake.test.ts`).
- `npx vitest run src/__tests__/automatic-run-brake-import.test.ts`: the
module loads after the heartbeat service, in the server's import order.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify`: see the comment below for the result. CI supplies the
full suite.

## Risks

- This pull request does not hold starts. Already queued runs and
`transient_failure_retry` runs still start while the pause is open; they
act as the natural tries. The start gate goes in as a follow-up on top
of #21433, at a point before that pass counts a claim attempt and reads
the full run row, so a held run costs no attempt and no read.
- The sweep reuses the `deferred_rate_limit` outcome for a deferral, to
keep its outcome type unchanged while #21394 edits that file.
- The automatic-run predicate is built from master exports
(`RECOVERY_WAKE_REASONS`, retry markers). It will switch to
`isAutomaticContinuationRun` when #21394 lands.
- Migration `0480` adds one small table.
- Migration `0481` (`0481_heartbeat_runs_company_finished_index.sql`)
runs `CREATE INDEX CONCURRENTLY IF NOT EXISTS` as a top-level statement:
it is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` in
`packages/db/src/client.ts`, like 0470 and 0476, and the runner drops
and rebuilds an invalid index after an interrupted build. The index is
declared in the Drizzle schema, so `pnpm db:generate` reports no
changes, and `pnpm --filter @paperclipai/db check:migrations` passes. On
a large `heartbeat_runs` table the build is online (no write lock) but
takes time proportional to the table. After it exists, it is one more
index on a hot table: each run that finishes writes one more index
entry.
- No user-visible text changes outside the operator report.

## Model Used

- Claude Fable 5.2 (Anthropic), via Claude Code with tool use (shell,
file edits, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 21:20:09 -06:00
DottaandClaude Opus 5.5 f6c5986b61 feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service

## Linked Issues or Issue Description

**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.

**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.

**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.

**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.

**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.

## What Changed

- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.

## Verification

- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.

## Risks

- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:10:32 +00:00
DottaandClaude Opus 5.5 6b631dd477 fix(recovery): check held retained orphans only for candidate leases (#21542)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The ownership reconciliation pass lists in-progress records of each
kind every 30 seconds, and settles the records whose owner is gone
> - One kind, the retained orphan sandbox, lists leases with a large SQL
filter that has correlated subqueries on `heartbeat_runs`, `issues`,
`issue_recovery_actions` and `environment_leases`
> - The planner ran those subqueries for every released, failed and
expired lease in history, before the cheap orphan predicates. The cost
grew with lease history, not with the number of orphans
> - On a busy hosted database one page of this listing ran for 17 to 138
seconds as one statement. It held an AccessShareLock on `heartbeat_runs`
for all of that time, so an online `ALTER TABLE heartbeat_runs ADD
COLUMN` with a 5 second lock timeout failed on every try
> - This pull request runs a cheap candidate filter first and evaluates
the held checks only for candidate rows
> - The benefit is that the listing cost now follows the number of
candidates. Schema changes on `heartbeat_runs` no longer wait behind it,
and the pass reads far fewer pages

## Linked Issues or Issue Description

No public issue exists. This describes the bug in the format of the bug
report template.

**What happened?**
A hosted instance could not apply a pending migration that adds a column
to `heartbeat_runs`. The deployer sets `lock_timeout = 5s` and tries
three times. All three tries timed out. The database showed one session
that ran `select "id", "company_id", "heartbeat_run_id", "issue_id",
"metadata", "updated_at" from "environment_leases" ...` in state
`active` for up to 138 seconds. Its transaction start and query start
differed by less than 0.1 seconds, so it was one statement. It held an
AccessShareLock on `heartbeat_runs`. That instance had about 18,600
lease rows: 13,240 expired, 4,695 released, 610 failed, 69 retained, 17
active and 9 pending cleanup. Each failed try also stalled all
statements on `heartbeat_runs` for up to 5 seconds.

**Expected behavior**
The ownership listing must finish in a short time that does not grow
with lease history. It must not keep a lock on `heartbeat_runs` long
enough to block an online schema change.

**Steps to reproduce**
1. Use a database with many terminal environment leases (expired,
released or failed) and few or no retained orphan sandboxes.
2. Run the `environment_retained_orphan_sandbox` listing
(`OWNABLE_KINDS`, `listInProgress`), or wait for the ownership
reconciliation pass.
3. Run `EXPLAIN (ANALYZE, BUFFERS)` on the listing. The `heartbeat_runs`
primary key subplans run once for each terminal lease, not only for
candidates.
4. While the listing runs, run `ALTER TABLE heartbeat_runs ADD COLUMN
probe jsonb` with `lock_timeout = '5s'`. The ALTER waits until the
listing ends.

**Paperclip version or commit**
`master` at 0a96f972e1.

**Database mode**
Hosted PostgreSQL. Reproduced on embedded PostgreSQL 18.

**Additional context**
The 138 second session was not a transaction held open across remote
work. The call path is `registerOwnershipReconciliationWork` →
`runOwnershipReconciliationPass(db)` → `kind.listInProgress(db, 200,
offset)` on the pool, with no transaction. Only the `leaseKind` and
`operationKind` listings send this exact unqualified column list. Of
these, only this kind's filter reads `heartbeat_runs`.

## What Changed

- `releasableRetainedOrphanCondition` and `nextRetainedOrphanDueAt`
first apply `retainedOrphanCandidateCondition`. This is a filter with no
subqueries and no NULL results: cleanup status, run, provider, lease
status, lease policy, and the metadata keys of the three retained orphan
shapes. Expired, active and pending-cleanup leases fail it at once.
- The held checks (always held and softly held) now run inside `CASE
WHEN <retained orphan condition> THEN ... ELSE false END`. PostgreSQL
evaluates a CASE in order, so the correlated subqueries run only for
candidates. The planner can no longer split the `NOT (a OR b OR ...)`
into separate top-level subplans and run them before the candidate
predicates.
- The two `heartbeat_runs` checks of the always-held part are now one
scalar subquery with a primary key lookup. It reads
`runner_profile_json->'nativeWorkspaceSync'` once through a fenced
lateral, instead of about 12 `#>>` path reads. A scalar subquery cannot
become a hashed subplan, so the planner cannot replace it with a full
scan of `heartbeat_runs`.
- The listed rows, the order and the paging do not change.
`softlyHeldSql` is now exported so that the test can rebuild the
previous condition.
- The paging stays OFFSET-based. The `listInProgress(db, limit, offset)`
contract is shared by all ownership kinds and by the I2 rule, which
compares consecutive passes. Keyset paging is a change to that shared
contract, so this pull request does not make it.

## Verification

- `cd server && npx vitest run
src/__tests__/environment-retained-orphan-listing-cost.test.ts
src/__tests__/environment-retained-orphan-sandbox.test.ts
src/__tests__/abandoned-workspace-files-hold.test.ts`: 3 files, 55 tests
pass.
- New test `environment-retained-orphan-listing-cost.test.ts`:
- Equivalence: four seeded random fixtures of 600 leases each, across
two companies. The fixtures cover every lease status and policy, runs of
each status and runtime mode, valid and invalid native workspace
descriptors, every always-held metadata key, sibling leases on the same
sandbox, open and resolved recovery actions, open runs on the issue, and
past, future and absent expiry. Every page (limit 7 and 200) equals the
previous SQL, which is kept only in the test. `nextRetainedOrphanDueAt`
also equals the previous query.
- Red/green: over 2,000 history leases with the hosted status mix (71%
expired, 25% released, 4% failed), the test counts the `heartbeat_runs`
rows that the listing reads, from `EXPLAIN (ANALYZE, FORMAT JSON)`. On
`master` it reads 1,760 rows for 1 candidate, so the test fails. With
this change the count is within the bound of 10 per candidate.
- Incident regression: while listings run in a loop over 5,000 history
leases, `ALTER TABLE heartbeat_runs ADD COLUMN ...` and `DROP COLUMN`
with `lock_timeout = '5s'` complete in less than 5 seconds.
- Local measurement (embedded PostgreSQL 18, Apple M-series, warm
cache), not in CI:
- 18,640 leases with the hosted status mix and a 19 KB runner profile
per run. `master`: 422 ms, 296,771 buffers (12,357 read). This change:
26 ms, 36,004 buffers. The `heartbeat_runs` subplans run 13 times (once
for each candidate), not once for each lease.
- 300,000 history leases. `master`: one listing page takes 2,822 ms, and
the ALTER under load waits 2,448 ms. This change: 212 ms, and the ALTER
takes 96 ms.
- `pnpm check:module-boundaries` passes.

## Risks

- Low risk. The listing and the due-time query return the same rows, and
the equivalence test compares them. The settle is unchanged and still
re-checks every hold under the row lock.
- The candidate filter must stay a superset of
`retainedOrphanSandboxCondition`. If a later change adds a retained
orphan shape and does not update the candidate filter, the listing skips
that shape. The equivalence test catches this only for the shapes in the
fixture.
- The listing still reads each lease row once with cheap predicates. A
partial index on the candidate filter could make it scale only with
candidates. This pull request does not add a migration.
- The ownership pass still lists all kinds every 30 seconds when its
fingerprint changes. This pull request does not change that cadence.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edit, embedded PostgreSQL runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:48:21 -06:00
DottaandClaude Opus 5.5 1dee321844 perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every native event that announces work to another part of the server
(chat progress, a final response, an evidence outcome, recovery)
registers a wake in its transaction before it writes the event
> - Such an event takes six statements: BEGIN, the run lock read, a
replay probe, a read of the transaction id for the wake, the append, and
COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock,
the transaction id, the write, COMMIT
> - In the agent scaling load model these two paths are now the largest
per-agent cost after #21511 and #21520, and other events fall behind
first at 600 agents
> - This pull request returns the transaction id from the lock read that
each path already sends, and moves the replay probe into the run lock
read when it is safe
> - The benefit is four statements for such an event and four for a
checkpoint save, with the same locks, the same wake order and the same
replay rule

## Linked Issues or Issue Description

Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent
scaling plan and load model).

**What existing behavior does this improve?**
The statement count of native events that register a wake
(`appendNativeSourceEvent`) and of controller checkpoint saves
(`nativeRunnerCheckpointWriter`).

**Subsystem affected**
`packages/db/src/work-signals.ts` (one new export),
`server/src/services/heartbeat-run-events.ts`,
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Current behavior**
`signalDatabaseWork` sends `select pg_current_xact_id()` for the first
wake of each transaction. A native event with a wake reads the run lock,
then probes for its replay in a separate statement, because a statement
that waits for the lock keeps its earlier snapshot.

**Proposed behavior**
- The run lock read and the checkpoint owner lock read also return
`pg_current_xact_id()::text`. Both already lock a row, so the
transaction has an id at that point. `primeDatabaseWorkTransactionId`
gives it to the transaction's wake state, and the wake sends no
statement for it.
- For an event that may register a progress, response, reassessment or
retention wake, the run lock read also reads `next_event_seq` from its
own snapshot and checks for a replay in that snapshot. Every event
append advances `next_event_seq` in its transaction. So when the locked
row has the same `next_event_seq` as the snapshot, no append committed
in between, and the probe result is current. Otherwise, or when the
probe finds a replay, the separate probe runs as before.

**Reason and benefit**
Fewer round trips while a pool connection is held in a transaction. In
the load model these paths hold the main pool and the three-connection
checkpoint pool longest per event.

**Breaking changes**
None. No schema change.

## What Changed

- `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`,
exported from `@paperclipai/db`): inside a transaction whose id is not
yet known, it records the id. Outside a transaction it does nothing. An
uncertain commit still resolves with `pg_xact_status` on the same id.
- `appendNativeSourceEvent`: the lock read returns the transaction id
when the event has any wake. For a probing wake it is one statement: a
CTE that reads `next_event_seq` from the snapshot, a CTE that locks the
run row (`for no key update`, unchanged), and `exists (...)` for the
replay. The wake is still registered after the replay check and before
the append.
- `lockNativeRunnerCheckpointOwner` returns the transaction id with its
owner lock and primes the wake state. Its return value and every caller
are unchanged.

## Verification

- Red/green on embedded PostgreSQL; each new assertion failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "registers a native
event's wakes before its write and only when the event is new": four
statements (before: six), the wake's intent and settlement are
published, and a replay publishes no wake.
- New: "finds a replay that another writer committed while the lock read
waited, and registers no wake for it". Another connection holds the run
row (the test waits until PostgreSQL reports the append waiting for that
lock), stores the same event and moves `next_event_seq`; the waiting
append returns `duplicate`, runs the separate probe, and publishes no
wake. With the `next_event_seq` comparison removed, this test fails.
- New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the
owner lock with the transaction id, the write, COMMIT (before: five
statements), and its recovery wake is published; a save after the lease
expired writes nothing and fails with
`native_runner_checkpoint_owner_lost`.
- `work-signals.test.ts`: a primed transaction sends no id statement for
its wakes; a primed transaction whose connection is killed after its
write stays pending and then resolves through `reconcileDatabaseWork`,
with no row stored.
- Added after merging `origin/master` (with #21511, #21520 and #21436)
into this branch:
- "gives the wake the transaction id of the transaction that took the
run lock": inside a caller's transaction, an event with a progress wake
sends no separate `pg_current_xact_id()` statement; the transaction's
connection is then killed, and `reconcileDatabaseWork` checks
`pg_xact_status` with exactly the id that `pg_current_xact_id()` returns
on that same transaction. Priming a wrong id makes it fail.
- "primes nothing when the lock read fails, so the transaction is not
left uncertain": a lock read that fails on `lock_timeout` leaves no
pending work; after the same failure inside a savepoint, the next signal
on that transaction still sends its own `pg_current_xact_id()`. Priming
before the lock read makes it fail.
- Both call sites pass only the value returned by the statement they
just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)`
then `primeDatabaseWorkTransactionId(tx, run.xid)`) and
`lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then
`primeDatabaseWorkTransactionId(tx, xid)`).
- Retest on the combined tree, one file at a time: atomic-append 28
(including the equivalence test and both race tests), the port suite
with conformance 105, checkpoint-save 2, `live-recovery-work` 51,
`work-signals` 25, runner `native-session-runtime` 297; server typecheck
passes. No code change was needed for the combination.
- Other focused suites pass, one file at a time: the port suite with its
conformance suite (100), `chat-native-progress-work` (35),
`native-observer-replay` (1), `heartbeat-run-event-sequencing` (5),
`native-event-boundary-golden` (1), `native-recovered-owner-followup`
(97), `native-restart-recovery-isolation` (1),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51),
`work-signals` (25).
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). Stacked: this pull request merged locally with #21511, and the
model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model
applies the port's batching rules itself; that mode is not in any pull
request). Default pool of 10 connections; embedded PostgreSQL on the
same, shared machine; runs one after the other with a pause between
them. No run had a failed operation, a failed checkpoint save or a lost
lease. "Before" rows are `origin/master` from the same session.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 /
12,000 | 1 / 70 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 /
24,000 | 2 / 1,274 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 /
6,335 | 61,158 / 72,000 | 11,678 / 21,212 |
| 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 /
72,000 | 1,136 / 4,980 |
| 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 |
71,958 / 72,000 | 1,319 / 5,712 |
| 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 /
48,000 | 1,974 / 3,835 |

With the three changes, one process finishes 99.9% of the offered work
at 600 agents, but its event-loop utilization is 0.90–0.93, above the
0.75 target, and p99 lateness is about 5 s. It saturates between 600 and
800 agents.
- Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5
where commit-time notifications are on).

## Risks

- The transaction id is read in a statement that already assigns one (a
row lock), so no transaction gets an id it did not have before.
- The probe in the lock read is used only when `next_event_seq` did not
change between the snapshot and the lock. Every writer of
`heartbeat_run_events` moves `next_event_seq` in the same transaction
(native and ordinary appends, the run coordinator store, the retry
queue). A future writer that inserts events without it would make this
probe miss a concurrent replay; the append statement then still finds
the replay and stores nothing, but the wake would already be registered
(a spurious wake, not a lost or duplicate event).
- This pull request and #21511 both edit `heartbeat-run-events.ts`, in
different functions.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:48:17 -06:00
Devin FoleyandPaperclip dc018f9bc0 Keep agent-owned connections healthy during catalog refresh (#21539)
Use dedicated agent grants for catalog refresh, GitHub health checks, and reconnect status. Preserve shared connection health when a different agent lacks authorization.

Add PostgreSQL and HTTP regressions for credential selection, stale health recovery, caller isolation, and actual provider rejection. Correct the MCP demo approval fixture to submit a valid saved-card option through normal answer validation.

Validation: 629 focused tests, server typecheck, all CI checks, and Greptile 5/5 with no unresolved review comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 18:45:20 -08:00
DottaandClaude Opus 5.5 a82e235828 fix(heartbeat): keep recovery behind people's work, guarantee a start to tasks people wait on, and admit recovery a few runs at a time (#21427)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts an agent's queued runs when a slot is
free. #21234, #21277, #21349 and #21391 order them: person work first, a
recovery share, aging, and a share for the longest-waiting run
> - On hosted staging, one agent with 20 slots had 295 queued runs. 139
of them were system recovery wakes that entered the queue within one
hour. Runs that people asked for waited 7 hours
> - Two rules let this happen. Recovery took every fourth start even
when a person's run waited, and an aged recovery run moved into the
agent-work tier ahead of todo tasks. The continuation producer also
admitted any continuation held past 5 minutes, however deep the queue
was
> - This pull request keeps recovery behind people's work, gives a
person's long-waiting run a bounded start, re-checks a recovery run
before it starts, and admits recovery to an agent's queue a few runs at
a time
> - The benefit is that a task a person asked for starts within a
bounded number of dispatches, and a recovery burst cannot flood an
agent's queue

## Linked Issues or Issue Description

Refs #21234, #21277, #21349, #21391, #21405, #21433, #21502

**What happened?**

On hosted staging (2026-11-13), one agent had `maxConcurrentRuns` 20 and
all 20 slots held hour-long runs. 295 runs were queued. 139 had wake
reason `issue_recovery_action_restored` (source `execution.reconciled`),
enqueued between about 10:46Z and 11:47Z. Two tasks that people created
were stalled by a restart at 07:25Z. Their next runs were queued at
07:29Z and were still queued 7 hours later. Both were system wakes: one
`issue_monitor_due`, one `issue_continuation_needed` from the
assigned-work sweep. Recent starts had waited from 1 to 290 minutes.

**Why did it happen?**

1. The recovery share gave every fourth start to system recovery, also
while a person's run waited.
2. A recovery run that waited one hour aged into the agent-work tier.
Recovery tasks are usually in progress, so they sorted ahead of todo
tasks in that tier.
3. The oldest-run share served the oldest aged run of any class. Older
agent work and older recovery came before a person's run.
4. Only a wake that a person requested counted as person work. A system
wake on a task that a person created and waits on (a monitor check, a
continuation after a restart) competed as ordinary work or recovery.
5. `deliverReconciledExecutions` admitted a continuation held past
`RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) whatever the queue
depth. At one per agent per 15-second pass, one agent got about 139
continuations in an hour.

**Expected behavior**

A run that a person created or asked for never waits behind system
recovery. Waiting never fails or cancels it. A long wait moves it
forward, so it starts within a bounded number of dispatches. Recovery
enters the queue a few runs at a time. A recovery run that is no longer
needed is dropped quietly.

**Steps to reproduce**

1. Give an agent 20 slots, all busy.
2. Queue 60 agent assignments, then two system wakes on tasks people
created (a monitor check and a continuation), then 30 later person runs
on in-progress tasks, then a burst of 139 recovery wakes.
3. Free one slot at a time while people and agents keep adding work.
4. See that the monitor check starts only at dispatch 244.

## What Changed

- `server/src/services/heartbeat/queued-run-dispatch-order.ts`:
- System recovery has its own tiers below agent work (fresh, then
backlog). Aging moves old backlog only into the fresh recovery tier,
never ahead of agent work.
- The recovery share applies only when no ready person run waits. It
comes out of agent work, never out of a person's turn. Within the share,
the longest-waiting aged recovery run goes first.
- The oldest-run share serves the longest-waiting aged run that a person
asked for or whose task a person created (`issues.created_by_user_id`),
first. This includes a system continuation of such a task. Then it
serves the longest-waiting aged agent-work run. It never serves other
recovery.
- New dispatch fact `personTask`, read with the queued tasks' status and
priority (same query).
- `queuedRunDispatchLine` (the O(n log n) place-in-line order from
#21405) follows the same rules. `queued-run-line.test.ts` still proves
it equals the dispatcher's order over 400 random queues, now with
`personTask` in them.
- `server/src/services/heartbeat/queued-run-line.ts`: the start-history
counters use `dispatchHistoryStart` and `dispatchCountersFromStarts`, so
dispatch, the place-in-line view and the tests share one rule.
- New pure helpers `oldestShareCandidate`, `dispatchHistoryStart` and
`dispatchCountersFromStarts`. The queue and the tests use the same
counter rule (which includes the #21391 rule that a person's recovery
restart does not use up the recovery share).
- `server/src/services/heartbeat/queue.ts`:
- At claim, a system recovery run (`issue_recovery_action_restored`,
`issue_continuation_needed`, not requested by a person) is re-checked.
If the task is reassigned, or another run continues it (started after it
was queued, running, a queued non-recovery run, or a newer queued
recovery run), the run is cancelled before it starts with
`recovery_no_longer_needed` and its wake is skipped. This is the same
quiet path as the existing closed-task check: no failure, no provider
work. A person's undelivered message on the run still keeps it.
- `server/src/services/recovery-queue-admission.ts` (new):
`QUEUED_RECOVERY_RUNS_PER_AGENT` = 3 and the queue-depth check.
- `server/src/services/execution-recovery-resolution.ts`: a continuation
is admitted only while its agent has fewer than 3 queued recovery runs,
fresh or backlog. The 5-minute hold bound still orders held
continuations, but no longer overrides the cap. The rest stays pending
in its recovery action. The existing continuation pass (every 15 s while
any continuation is pending) admits the next one after a queued recovery
run starts or is dropped. No new polling.
- `server/src/services/recovery/legacy-workspace-rehome.ts`: the
pending-wake delivery for rehomed and released-writer tasks uses the
same cap. A held wake waits 60 s for its next turn and does not count as
a refusal.
- The oldest-run share counts as served only by a start that people wait
on while its candidate is one, so older agent work never uses up a
person's turn (`servesOldestShare`, used by the dispatcher, the
place-in-line order and the start history, which now records
`personWaiting`).
- `server/src/services/recovery/assigned-work-sweep.ts`: a rehome or
released-writer wake held by the cap (`HELD_RECOVERY_WAKE_KEYS`) counts
as the task's pending path while its assignee has queued recovery runs.
- `server/src/services/recovery/assigned-work-sweep.ts` (with #21502's
stalled-queued-run replacement): a start of work people wait on does not
count as passing over a queued run that people do not wait on. Waiting
behind people's work is intended, so the sweep does not cancel such a
run and replace it with a recovery continuation. Test:
`assigned-work-stalled-queued-run.test.ts`, "leaves an automatic run
that waits behind work on tasks people created". It fails without the
rule.
- `server/src/services/recovery/assigned-work-sweep.ts`: a pending
continuation still counts as the task's wake path past
`PENDING_CONTINUATION_BOUND_MS` while its agent has queued recovery
runs, so the sweep does not treat a held task as stranded.

**Bounds**

- A run a person asked for, or any run on a task a person created: once
it has waited past `QUEUED_RUN_AGING_MS` (1 hour), it starts within
`OLDEST_DISPATCH_SHARE` × (such aged runs queued before it + 1)
dispatches. Before that, runs a person requested already go first. There
is no time limit, and waiting never cancels it.
- Recovery run, proposed bounded behavior: at most 3 queued per agent.
When no person run waits, an aged recovery run starts within
`RECOVERY_DISPATCH_SHARE` × (aged recovery runs before it + 1)
dispatches. At start it is re-validated against the current task, and it
is dropped quietly if stale. While people keep the agent busy, recovery
waits. It is held durably in its recovery action, not lost and not
failed.

## Verification

Regression test that reproduces today's shape
(`server/src/services/heartbeat/queued-run-dispatch-order.test.ts`,
"deep per-agent queue: a recovery burst behind 20 busy slots"). It uses
20 busy slots, one dispatch per freed slot, 60 older agent assignments,
the two 07:29 system wakes on tasks people created (monitor check and
continuation), 30 later person runs, a 139-run recovery burst, and new
person and assignment arrivals on every step. The same model, run
against master's dispatcher and this one over 600 dispatches:

| | Before (master) | After |
| --- | --- | --- |
| Dispatch at which the 07:29 monitor check starts | 244 (≈12 h at ~20
starts/h) | 4 |
| Dispatch at which the 07:29 continuation starts | 5 | 8 |
| Recovery starts while a person-requested run waited | 89 | 0 |

With person-requested runs in the 07:29 slots instead (the first version
of the test), master starts them at dispatch 244 and 248, and this
change at 4 and 8.

- Red on master: the new dispatch tests fail (`expected -1 to be greater
than or equal to 0`, and recovery started ahead of agent assignments).
Green with this change.
- Admission cap, embedded Postgres (`recovery-wake-herd.test.ts`,
"queues at most a few recovery runs per agent..."): with the cap
removed, 12 of 12 continuations enter one agent's queue in 12 passes.
With the cap, 3 enter, the rest stay pending, and all 12 are delivered
as queued runs leave. The queue never holds more than 3 recovery runs.
- Older agent work (`queued-run-dispatch-order.test.ts`): with 60 older,
higher-priority assignments and no person arrivals, an aged
`issue_monitor_due` run on a person-created task starts within
`OLDEST_DISPATCH_SHARE` in dispatch and in the place-in-line order. With
the old reset rule it does not start in 8 dispatches.
- Held restored-task wake (`legacy-workspace-rehome.test.ts`): with the
agent's recovery queue full, the wake stays pending with no failure
counted, the sweep reads the task as `pending_intent`, and the wake is
delivered after a queued recovery run leaves.
- Start-time re-check, embedded Postgres, real heartbeat queue
(`heartbeat-stale-queue-invalidation.test.ts`): a recovery run whose
task was reassigned, or that a newer run already continued, is cancelled
with `recovery_no_longer_needed`, its wake is `skipped`, and the adapter
is never called. A recovery run that the task still needs starts.
- Updated tests that encoded the old rules: recovery's share and aging
now come out of agent work, not out of a person's turn. Sweep liveness
while held is covered.
- `pnpm --filter @paperclipai/server exec tsc --noEmit`: passes.

## Risks

- Behavior change: a system run on a task a person created now takes the
oldest-run share ahead of older agent work. On an agent whose queue is
mostly such tasks, agent-made subtasks get that share less often. They
still start in normal order.
- Behavior change: while people keep an agent fully busy, system
recovery for that agent waits. This is intended. Recovery is held
durably and starts once no person run waits.
- Behavior change: a queued system recovery run can be cancelled at
start as no longer needed. It is cancelled before any provider work,
with the same quiet outcome as a continuation whose task closed.
- A continuation held behind the cap stays pending longer than 7
minutes. The sweep now treats it as live while its agent has queued
recovery runs. If such queued runs can never start (for example
dependency-blocked), the next continuations wait behind them.
- One extra indexed read per recovery claim, on the existing
`(company_id, context_snapshot->>'issueId', created_at)` index. No
schema change. Mixed versions are safe: an old server keeps its old
order until it is replaced.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:21:35 -06:00
DottaandClaude Opus 5.5 a726dce4f0 feat(mcp): show live progress for long connected-app tool calls in the task chat (#21449)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call tools from connected apps through the Paperclip MCP
gateway
> - Some tool calls take a long time: exports, builds, large searches
> - The MCP progress utility lets an app report how far a call is, but
Paperclip ignored it
> - So the task chat showed only a spinner until the call finished, with
no sign of life
> - This pull request reads the app's progress and shows it on the
tool's row while the call runs
> - The benefit is that people can see that a long call is working, how
far it is, and how long it has run

## Linked Issues or Issue Description

**Subsystem affected**
Connected apps (MCP gateway) and the task chat.

**Problem or motivation**
A long tool call from a connected MCP app shows only a spinner in the
task chat. Paperclip does not send a `progressToken`, and it drops
`notifications/progress` from the app.

**Proposed solution**
Send a `progressToken` on each `tools/call` that an agent run makes.
Read the app's progress notifications from the call's response stream.
Show a progress bar (or an indeterminate sweep), the latest message, and
elapsed time on the running tool row. Keep the latest state so that a
reload mid-call shows it.

**Alternatives considered**
A new transport for progress. The existing company live-events socket
already carries per-run events with run read checks, so this change uses
it.

**Additional context**
Spec:
https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress

## What Changed

- `mcp-http.ts`: `mcpHttpRequestInit` can add `_meta.progressToken`.
`readMcpHttpResponse` gives each notification on the response stream to
an observer before the result. The observer cannot fail or delay the
call. New `mcpProgressNotificationOf` matches a notification to its
token.
- `tool-gateway.ts`: remote HTTP and local stdio `tools/call` from an
agent run send a per-call token and forward progress to a reporter.
Calls without a run send no token.
- New `server/src/services/tool-call-progress.ts`: checks each update
(finite, increasing, plain single-line text of at most 200 characters).
It publishes a `heartbeat.run.tool_progress` live event at most every
250 ms (latest wins) and saves the latest update on the invocation at
most every 1 s. Nothing is awaited by the call. A final `done` event
goes out when the call ends.
- DB: nullable `tool_invocations.progress` jsonb column (migration
`0479`, `ADD COLUMN` only). A save never writes onto a finished call.
- Live delivery: the new event follows its run like transcript output. A
client gets it only for runs it follows, after the run read check, in
the run's company only.
- New route `GET /api/issues/:issueId/tool-call-progress` (task read
check) returns the latest progress of the task's running calls, for a
reload mid-call. Documented in `docs/api/issues.md`.
- UI: `ToolCallProgressProvider` (one per chat) keeps the latest update
per call. A running tool row subscribes and shows a bar or sweep, the
message, the percent and elapsed time (design tokens, reduced motion
honored). Finished rows drop it and never fetch or re-render for
progress. `LiveUpdatesProvider` ignores the event for query
invalidation.
- `packages/kv-demo-mcp-server`: new `slow_with_progress` demo tool
(`steps`, `stepMs`, `withTotal`) in its own file.
- Storybook: `Tasks/Tool progress` stories (running with total, running
without total, done).
- Docs: new `doc/connections/MCP-PROGRESS.md`;
`MCP-PROTOCOL-VERSIONS.md` no longer lists progress as unsupported.

## Verification

- Red/green: on the original source, 5 of the new server tests fail
(token in `_meta`, SSE with interleaved progress and result,
notification matching, run-interest delivery, gateway end-to-end). With
this change they pass.
- `server/src/__tests__/mcp-http.test.ts` (13),
`tool-call-progress.test.ts` (9: throttling, latest wins, bounded text,
out-of-order drop, finish, failures never throw, company/run/tool
scoping), `tool-call-progress-store.test.ts` (2, embedded Postgres:
company and task scoping, no write onto a finished call),
`realtime/live-event-delivery.test.ts` (run interest),
`tool-gateway.test.ts` end-to-end against the real SDK kv demo server
over Streamable HTTP: 6 provider updates over about 1.2 s give 3 to 7
throttled events, all for the right company, run, task and tool, the
last one `done`; no event reaches another company; the invocation keeps
the progress; the result is unchanged.
- UI: `tool-call-progress.test.tsx` (10: name matching, row states,
plain-text rendering, drop on complete, restore after reload, no fetch
for finished rows). All 67 task-chat test files pass.
- `packages/kv-demo-mcp-server`: 13 tests, including progress over HTTP
with and without a total.
- Storybook: checked the three stories in a browser. The bar fills to
40%, the sweep animates, the elapsed time ticks, and the done row shows
no progress.
- `pnpm verify`: see the CI checks for the final revision.

## Risks

- Low. The migration adds one nullable column.
- An SSE response is now bounded per message, not per stream, so handled
notifications (such as progress) do not use up the result's size limit.
Each message is still bounded.
- Progress code runs only while a run's tool call is in flight. It adds
a few bytes to the request and throttled, fire-and-forget work. Agent
boot and time to first token do not change.
- A row finds its call by run and tool name. If one run calls the same
tool twice at the same time, both rows show the newer call's progress.
- Progress from a call that became an MCP Task is not shown.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:14:22 -06:00
DottaandClaude Fable 5.2 5d0b27acca fix(run-events): watch both late appends for rejection as soon as they start (#21536)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server stores each run event through `appendHeartbeatRunEvent`
and `appendHeartbeatRunEventBatch`
> - Pull request #21511 added a test that deletes a run while two
appends wait for its row lock
> - Both appends must fail, but the test attached the rejection check of
the second append only after it awaited the first
> - When the second append failed first, its rejection had no handler,
and Vitest failed the whole shard with one unhandled error
> - This pull request attaches both rejection checks when the appends
start
> - The benefit is that server test shards on master and on unrelated
pull requests pass again

## Linked Issues or Issue Description

Refs #21511

**What happened?**

Server test shards failed on master and on unrelated pull requests.
Every test passed. Vitest reported one unhandled error: `Error:
heartbeat_run_event_binding_mismatch`, from
`server/src/__tests__/heartbeat-run-event-atomic-append.test.ts`.

**Expected behavior**

The test awaits both expected failures. No rejection stays without a
handler.

**Steps to reproduce**

Run the general server test shard that contains
`heartbeat-run-event-atomic-append.test.ts` on master at `cefef8f11d`.
Examples: master "verify / General tests (server (1/10))", pull request
#21443 "ci / General tests (server (9/12))", pull request #21505 "ci /
General tests (server (1/12))".

## What Changed

- In the test "waits for Stop on the run row and appends after it, and
fails without rows after a run deletion", the two `expect(...).rejects`
checks are now created immediately after the two appends start. The test
awaits them after it releases the deletion.
- No product code changed. The product code rejects correctly; the test
left one rejection without a handler for a short time.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-run-event-atomic-append.test.ts`: 25 of 25 tests
pass, three runs in a row, no unhandled error.
- The failure before this change is the master shard named above. It
depends on which append rejects first, so I did not get a local
reproduction on demand.

## Risks

Low risk. The change is in one test. The assertions are the same.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 20:07:31 -06:00
DottaandClaude Opus 5.5 81d06cfc97 feat(mcp-tasks): add a demo MCP server that uses Tasks, test against it, and keep audit records out of agent errors (#21464)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call tools in connected apps through the tool gateway, which
speaks MCP to each app
> - MCP Tasks (#21263) lets an app answer a long tool call with a
durable task. It shipped behind the experimental setting
`enableMcpTasks`, off by default
> - No public app returns tasks without credentials, so nobody had seen
the feature work against a real server, and it was hard to understand
> - This pull request adds a small demo app that uses Tasks, tests the
gateway against it over HTTP, and records a live end-to-end run
> - The live run found one bug: every task stop sent the call's audit
record, with a server stack trace, to the agent. This pull request fixes
it
> - The benefit is a repeatable way to try and test Tasks,
plain-language docs, and evidence for the decision to turn the setting
on

## Linked Issues or Issue Description

Refs #21389
Refs #21263

## What Changed

- Add an MCP Tasks demo server to `packages/kv-demo-mcp-server` (new bin
`paperclip-tasks-demo-mcp-server`; run `node
packages/kv-demo-mcp-server/dist/tasks/main.js` after a build). It is
stateless MCP 2026-07-28 with `server/discover`, and it declares
`io.modelcontextprotocol/tasks`. It has six tools: `slow_report`,
`needs_approval` (form question), `will_fail`, `cancellable`, `expires`,
and `quick_echo`. It saves task state to a file, so it can restart
mid-task. It needs no credentials. It has a status page at `/` and JSON
at `/api/state`.
- I put it in the existing demo package, so there is no new workspace
package and no `pnpm-lock.yaml` change.
- Add `server/src/__tests__/mcp-tasks-demo-server.test.ts`. It runs the
real gateway against the real demo server over HTTP, with one test per
tool plus a restart.
- Fix: `server/src/routes/tool-gateway.ts` no longer sends
`details.execution` to callers. That record has request IDs and the
server-side error stack. The audit still keeps it. Add
`server/src/__tests__/tool-gateway-agent-errors.test.ts` (red before the
fix, green after).
- Docs: `doc/connections/MCP-TASKS.md` now has "In plain words", "What a
person sees", "Try it yourself", the live-run results, and a
recommendation. The package README documents the demo server.
- `enableMcpTasks` stays off by default.

## Verification

- `packages/kv-demo-mcp-server`: `npx vitest run` gives 25/25 pass (13
new demo-server tests). `tsc -p tsconfig.test.json` is clean.
- `server`: `mcp-tasks-demo-server.test.ts` gives 8/8 pass.
`tool-gateway-agent-errors.test.ts` gives 2/2 pass, and it failed 2/2
before the fix.
- Existing suites that cover the changed route and Tasks:
`mcp-tasks.test.ts` and `tool-gateway-discovery-http.test.ts` (results
in a comment below).
- `pnpm verify` exits 2, because the doc is a global CI input. Required
CI gives the broad coverage.
- Live end-to-end run on a local `test-drive` instance with
`enableMcpTasks` on. The agent was Codex `gpt-6-luna` at low effort. The
demo was connected through "Connect your own MCP server":

| Behaviour | Result | `tools/call` count at the app |
| --- | --- | --- |
| Polling (`slow_report`, 45 s) | Delivered after 22 polls | 1 |
| Question (`needs_approval`) | The card was answered in the UI.
`tasks/update` was sent, and the task finished | 1 |
| Failure (`will_fail`) | `provider_task_failed` with the app's message
| 1 |
| Cancel (`cancellable`) | Stop on the run sent `tasks/cancel` | 1 |
| Expiry (`expires`, 15 s) | `provider_task_expired`, no hang | 1 |
| Restart (`slow_report`, 120 s) | The server was down for 2 min 23 s.
The result was delivered once, and the agent was woken once | 1 |

After the run, all 6 `tool_mcp_tasks` rows were settled and none had a
next poll.

## Risks

- Low risk. The demo server is new code in a demo package, and the bin
is additive.
- The route change removes `execution` from tool-gateway error
responses. No UI code or test reads that field from these responses. The
audit and tool-call events still record it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:47:58 -06:00
DottaandClaude Opus 5.5 1f388ed385 Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm sandbox keeps an agent's Runner alive between runs, so the
next run on the task starts fast
> - When a run ends, the server closes the idle warm Runner in-process.
A separate SQL worker stops the sandbox when its warm window ends
(#21410, merged today)
> - Both steps start at the same moment. The sandbox stop wins, so the
idle close fails. The failed owner stays registered, and the next run
throws `native_instruction_materialization_handoff_unavailable` before
any provider work
> - Recovery must just work: a follow-up run must never fail because a
predecessor's warm copy is gone
> - This pull request lets the idle close finish before the sandbox
stops, prepares instructions fresh when the warm copy cannot be handed
off, and releases an owner whose sandbox the host already gave up
> - The benefit is that automatic follow-up runs start normally after a
warm window ends

## Linked Issues or Issue Description

Refs #21472 (base of this stacked PR), #21410 (introduced the
warm-window stop).

**What happened?**
In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three
automatic follow-up runs failed before any provider work with
`native_instruction_materialization_handoff_unavailable`: run
`5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a
release swap. In each case, the predecessor's idle close had failed
first with `provider_state_unreadable` / "Daytona sandbox lease … is no
longer active". The cause: the warm-window stop from #21410 stopped the
sandbox at the same moment that the idle close started. For the first
run's predecessor, the close was requested at 20:30:34 and the sandbox
got SIGTERM at 20:30:37.8.

**What did you expect?**
The follow-up run prepares its instructions fresh and continues. The
warm-window stop does not interrupt the host's own idle close.

**Steps to reproduce**
1. Run a warm Daytona task until it ends, so its warm window starts.
2. Let the warm window end while the idle timer closes the Runner.
3. Start a follow-up run on the same task. It fails with
`native_instruction_materialization_handoff_unavailable`.

Two of the three tasks then ended stranded. That downstream part (the
claim-gate cancellation with `execution_reconciliation_required` and the
stranded notice) is owned and fixed by another session. It is **not** in
this PR.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
`claimWarmNativeInstructionCopy`: after it retires a stale warm owner
whose copy cannot be adopted, it returns `null`. The caller then
prepares instructions fresh. Before, it threw
`native_instruction_materialization_handoff_unavailable`.
- `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now
waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window.
The in-process idle close (close grace plus the 30 s proof read)
finishes first.
- `native-session-executor.ts` idle timer: when an idle close fails, it
reads the lease rows for that sandbox. If every row is no longer
`active` or `retained` (the host gave the sandbox up), it releases the
in-process owner. The next run then prepares fresh.
- `doc/execution-semantics.md`: one paragraph about the stop margin and
fresh preparation.

## Verification

Red → green (each test fails on the base `35ba76d0ef` and passes on this
branch):

- `native-session-executor.test.ts` › "hands off collection before
probing or retiring a warm owner: unavailable" — red: throws
`handoff_unavailable`; green: returns `null` (fresh preparation).
- `native-session-executor.test.ts` › "forgets a warm owner whose idle
close failed only once the host gave up its sandbox: pending_cleanup /
released / retained" — red on all three; green: the owner is released
for `pending_cleanup` and `released`, and kept (retired by the
successor) for `retained`.
- `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the
warm runner time to finish before stopping its sandbox" — red: the
sandbox stops 3 s after the window; green: it stops only after the
margin. Two existing tests ("keeps a sandbox still inside its warm
window, and wakes when the window ends", "reconciles a warm sandbox left
running across a restart") now expect the deadline with the margin.

Local runs on this branch:

- `server`: `tsc --noEmit` clean.
- `vitest run src/__tests__/warm-sandbox-expiry.test.ts
src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed.
- `vitest run
src/services/native-runtime/native-session-executor.test.ts -t "hands
off collection before probing or retiring a warm owner|forgets a warm
owner whose idle close failed"`: 10/10 passed.

Not run: a Daytona wave with this change.

## Risks

- Warm sandboxes now stop up to 2 minutes later than the configured
window. This is a small cost increase per warm sandbox. I chose a fixed
margin over an in-process "is a close running?" check because the stop
is pure SQL. The margin works when two servers run at the same time (the
server that runs the stop may not be the server that owns the Runner),
and it needs no busy-loop or cross-server signal.
- The margin does not cover a re-armed idle timer or a close that takes
longer than 2 minutes. Fresh preparation (fix 1) and the owner release
(fix 3) cover those cases.
- Not run on Daytona. The evidence is from the wave above and the unit
tests.
- In the wave, a `restore_failed` copy-back onto the just-restarted
sandbox was also seen. I did not find its root cause. It is not changed
here.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test runs) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:47:42 -06:00
DottaandClaude Opus 5.5 a4c570aede feat(mcp): ask an app's form questions in the task chat and send the answer back (MCP elicitation) (#21466)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected MCP apps through the tool gateway.
> - An MCP app can ask the person a question in the middle of a tool
call (MCP elicitation). There are two kinds: a form, or a link to open.
> - Today a form question is a dead end:
> - A 2026-07-28 server's form `input_required` stops with "needs
information Paperclip can't collect yet".
> - An older server's `elicitation/create` makes a card, but the answer
is never sent back, and the server waits on an open stream.
> - Paperclip does not declare form support, so compliant servers never
ask.
> - This pull request shows the question as a card in the task chat. The
card names the app, and only a person can answer it. When the person
answers, the agent wakes. The agent's repeated call sends the answer to
the app exactly once, for both protocol eras.
> - The benefit: apps that need a choice or a detail from the person now
finish their work. Restarts lose nothing, and no call is left waiting.

## Linked Issues or Issue Description

Refs #21140 (stateless MCP client; this PR closes its "form elicitation"
follow-up). Refs #21263 (MCP Tasks; this PR shares its card code). Refs
#21389 (MCP Tasks follow-ups: the card answer validation item).

**Is your feature request related to a problem?**
When a connected MCP app asks the person a form question during a tool
call, the call stops and cannot continue. A 2026-07-28 server gets a
"needs input" error. An older server gets a card whose answer never
reaches it.

**Describe the solution you'd like**
Show the question in the task chat. Let a person answer, skip, or leave
it. Wake the agent when the question is resolved. Send the answer on the
agent's repeated call, once, with the server's exact state.

**Describe alternatives you've considered**
Holding the tool call open until the person answers. We rejected this:
it holds a stream and a run slot, and it does not survive deploys or
restarts. The plan (`doc/plans/2026-11-13-mcp-elicitation.md`) records
nine design decisions with their alternatives.

## What Changed

- **Plan:** `doc/plans/2026-11-13-mcp-elicitation.md` covers the spec,
the inventory, the decisions, the journey, tests, and risks.
- **New module `server/src/services/mcp-elicitation.ts`.** It is the
single mapping for direct calls and MCP Tasks:
- It turns each spec form field into a real card field: text with
defaults, number, yes/no, single choice (`enum` and `oneOf`), and
multiple choice (`items.enum` and `items.anyOf`).
  - It turns card answers back into typed `content`.
- Answered sends `accept`. Skip sends `decline`. Withdrawn, closed, or
expired sends `cancel`.
- It refuses forms that ask for passwords, keys, tokens, card numbers,
or one-time codes.
  - It keeps app text plain, so links in it are not clickable.
- It seals `requestState` with AES-256-GCM, bound to the company and
connection.
- **New table `tool_mcp_elicitations`** (migration 0479). It is
company-scoped, with one row per question round. It holds a keyed call
identity, the sealed state, the questions and their cards, a claim time,
and a 24-hour expiry.
- **Gateway, stateless servers:**
  - Paperclip declares `elicitation: {form, url}` on task tool calls.
- A form-only `input_required` saves the round, makes the cards, and
stops with a wait message.
- A repeated call before the person answers stops again without
contacting the app.
- After the answer, the repeated call claims the round and sends
`inputResponses` and the exact `requestState`, once.
- A failed send releases the claim. A crash mid-send lets the claim
lapse.
- **Gateway, older servers:**
  - Paperclip declares form support at `initialize`.
  - The first question gets `cancel` at once, so the server never waits.
- When the same question (same words and schema) arrives on the repeated
call, Paperclip answers it from the card.
- **Gateway, mixed rounds and retries:**
  - A form beside a URL step shows the link first.
- A repeated call that stopped for a question runs again in the same
run. It is no longer replayed by the per-run idempotency key.
- **Cards:** the title is "*App* needs your input". The resolver policy
is `human_only`. "Send to *App*" sends the answer. A user comment does
not supersede the card.
- **Activity log:** new events `tool_gateway.elicitation_sent`,
`tool_gateway.elicitation_refused`, and
`tool_gateway.elicitation_expired`. They record the action only, never
the values.
- **MCP Tasks:** task cards use the same mapping and gain the app name,
real fields, and `human_only`. A secret or an unsupported form stays
unsupported.
- **Storybook:** `Tasks and conversations/Questions and approvals/MCP
App Questions` has five states: Asking, Answered, Skipped, Expired, and
TwoQuestions. The fixtures are generated from the server mapper's
output.
- **Docs:** `doc/connections/MCP-ELICITATION.md` (linked from the
README) is new. `MCP-PROTOCOL-VERSIONS.md` and `MCP-TASKS.md` are
updated.

## Verification

- `server/src/__tests__/mcp-elicitation.test.ts` has 16 tests. They pass
on the branch.
- **Red/green:** 9 of the 9 gateway tests fail with master's
`tool-gateway.ts` and `mcp-http.ts`. The 7 pure unit tests cover the new
module.
- The tests use a fake 2026-07-28 server and a fake 2025-06-18 server
that asks over the open stream and waits for the reply.
- Covered: answer, skip (`decline`), withdraw (`cancel`), two questions
at once, a restart while waiting, a crash mid-send, another company with
the same app and arguments, expiry, a refused password field, an agent
that tries to answer (rejected), and capabilities.
- Existing suites: `mcp-tasks`, `mcp-protocol-versions`,
`tool-gateway-service`, `tool-gateway`, and `remote-mcp-connectors`. See
the results comment below.
- Storybook: the five stories render in headless Chromium, and the play
tests pass. The only console errors are the 404s from the backend-less
API calls that every story makes.
- `tsc --noEmit` is clean for `server` and `ui`. `node
scripts/storybook-catalog.mjs` passes.

## Risks

- **Older servers may start asking questions.** Declaring form support
at `initialize` can make older servers ask where they did not before.
Mitigation: the first question always gets `cancel` at once, so no call
hangs, and the agent gets a clear next step.
- **Per-call cost.** Each 2026-07-28 HTTP tool call in a task makes one
indexed lookup for a waiting round. Nothing is added before the first
token or to agent boot.
- **Migration.** One additive table with no backfill, so it is safe to
apply online.
- **Changed Tasks behavior.** A task card that is withdrawn or closed
now sends `cancel` instead of `decline`, as the spec defines. A skipped
card still sends `decline`.
- **Stale cards.** Expiry is checked only when the agent calls again. A
card can sit past 24 hours until the next call. Then it is withdrawn and
the app asks again.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use
(shell, file edits, headless browser), with extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:46:56 -06:00
DottaandClaude Opus 5.5 4879a146fb fix(secrets): stop credential reads from aborting concurrent budget admissions (#21452)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paid model calls (company decisions, fast responses, browser use,
prompt-injection checks) pass a budget admission before they start
> - Admission runs in a REPEATABLE READ transaction and share-locks the
credential's secret row, so a key rotation cannot slip between admission
and the call
> - Every credential read also wrote "Last used" (`last_resolved_at`,
`updated_at`) to that same row
> - When two calls ran at once, one call's credential read aborted the
other's admission with a serialization failure (40001). The retry then
saw the first call's new budget hold and denied the work as
`budget_blocked`
> - This is the flaky `decision-models.test.ts` failure that held Cloud
readiness, and so releases, on 2026-11-13
> - This pull request stops credential reads from writing the secret
row. "Last used" now comes from the access event that every read already
records
> - The benefit is that concurrent paid calls are no longer wrongly
denied or retried, and the release gate no longer fails on this test

## Linked Issues or Issue Description

No public issue exists. The bug is described here.

**What happened?**
Cloud readiness `verify / General tests (server (4/10))` failed on
master commits 6917d0568d and 5ede64f247. The test "admits concurrent
work from the same budget observation and records both obligations"
reported `expected "vi.fn()" to be called 2 times, but got 1 times`.
#21387 replaced the 1-second poll with an unbounded wait. That does not
fix it: one decision really is denied, so the test now waits until the
test timeout.

**Root cause**
I logged the accounting retry loop under load. Each batch of 50 runs had
6 to 10 retries with `could not serialize access due to concurrent
update` on `select … from company_secrets … for share`. The other
decision's credential read had updated the row. In 4 of 5 batches, the
retried admission saw the other decision's committed reservation and
returned `budget_blocked`.

**Expected behavior**
A credential read does not change admission. Two decisions admitted from
the same budget observation both run and both record their obligation.

**Steps to reproduce**
Run the updated test on the old `secrets.ts`. The test makes the second
credential read land after the first admission's snapshot. It then
commits an unrelated hold after both observations. The old code fails
every run with `first settled before its provider call:
{"status":"unavailable","reason":"budget_blocked"}`.

## What Changed

- `secretService` no longer writes `last_resolved_at` / `updated_at`
when it resolves a value. It still records the `secret_access_events`
row, as before.
- `secretService.list` reports `lastResolvedAt` as the later of the
stored column (legacy values) and the newest successful access event. It
reads one row per secret, backwards on the existing `(secret_id,
created_at)` index.
- A read without a consumer context (for example a GitHub API credential
read) now records a `system` access event, so it still counts as "Last
used".
- The removed `UPDATE` also made a credential read wait for a concurrent
rotation to commit. `credential()` in `ai-connections.ts` depends on
that wait to re-read the rotated version instead of handing a run a
consumed single-use token. Its existing version re-read now takes `FOR
SHARE`, so it keeps that wait. `prepareManagedAiRuntime` has its own
before/after freshness read around `runtimeCredential()`; that read
takes `FOR SHARE` too. A share lock creates no new row version and
cannot abort an admission. It adds no statement.
- The first version of this PR throttled the write to once a minute.
Review found that a read crossing the minute could still abort an
admission, so the write is removed instead.
- The concurrent-admission test is now deterministic. It starts decision
A and waits until A is blocked, then starts decision B and waits again.
It commits an unrelated 1-cent hold after both observations, then checks
that both decisions are admitted. A decision that settles before its
provider call fails the test immediately with its result, instead of
hanging.

## Verification

- Red: the new test against the old `secrets.ts` fails on every run with
the `budget_blocked` result above.
- Green: the same test with `repeats: 49` (50 runs) passes 50/50 locally
on a machine at load average 20 to 60.
- Before the change, 3 parallel batches of 50 runs of the old test each
produced a false denial.
- `npx tsc --noEmit -p server`: no errors.
- New `secrets-service.test.ts` test: a resolution leaves the secret row
unchanged, and `list()` still shows "Last used".
- `ai-connections.test.ts` quota test now checks that a read leaves
`updated_at` unchanged and records a success access event.

## Risks

- `updated_at` on a secret no longer moves when the secret is read. It
still moves on every real change.
- "Last used" on the Secrets page costs one indexed lookup per listed
secret.
- Only `list()` derives "Last used". Other endpoints that return a
secret row show the stored column, which is now set only by older
releases.
- No migration. No API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:46:50 -06:00
DottaandClaude Opus 5.5 e67f3bd63a test(environments): add an opt-in live Daytona proof for warm sandbox expiry (#21463)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes, such as Daytona, through reusable
environment leases
> - #21410 made Paperclip stop a warm sandbox when its warm window ends
> - Its tests use a real database but a mocked provider stop, and
staging had no warm reusable sandboxes to observe
> - This pull request adds an opt-in live test that proves the behavior
against real Daytona
> - The benefit is a repeatable live proof of warm expiry, resume, and
recovery after a restart

## Linked Issues or Issue Description

Refs #21410

**What happened?**
#21410 has no live provider proof. Staging creates no warm reusable
sandboxes today, so the deployed change cannot be observed there.

**Expected behavior**
A repeatable live check shows that a warm sandbox stays started inside
its window and stops after it. It also shows that the next turn resumes
the same sandbox, and that a restarted cleanup worker still stops the
sandbox on time.

**Steps to reproduce**
1. Set `PAPERCLIP_LIVE_WARM_EXPIRY=1` and `DAYTONA_API_KEY`.
2. Run `pnpm exec vitest run
src/__tests__/warm-sandbox-expiry.live.test.ts` in `server`.

**Additional context**
The test makes no model turn. It drives the production environment
runtime and the cleanup worker against real Daytona sandboxes.

## What Changed

- New `server/src/__tests__/warm-sandbox-expiry.live.test.ts`. The test
is skipped unless `PAPERCLIP_LIVE_WARM_EXPIRY=1` is set. It uses:
  - the real Daytona plugin and real sandboxes;
  - `environmentRuntime.acquireRunLease` with a 2 minute warm window;
  - `releaseRunLeases(..., "keep_running")`;
- the production `registerEnvironmentCleanupWork` on a real
`createDeliveryWorkCoordinator`, which must wake itself at the deadline.
- The test checks the following:
  - The sandbox is started 20 s before the window ends.
- After the window, the worker stops the sandbox. The sandbox is
stopped, not deleted, and the lease is released as stopped and kept.
  - The second turn resumes the same sandbox (`acquisition: resumed`).
- A new database client, a freshly loaded plugin module, and a new
cleanup worker start in the middle of the window. That worker stops the
sandbox after the window ends, not before.
- At the end, the test deletes only the sandboxes whose labels match its
own company and environment.

## Verification

- Live run on 2026-11-13 against the Daytona org, window 120 s, sandbox
`170bb215…`:
  - 18:39:32.624Z turn 1 lease retained; window ends 18:41:32.624Z
  - 18:41:13Z sandbox `started` (20 s before the window ends)
- 18:41:36Z sandbox `stopped`, auto-delete -1: the worker stopped it 4 s
after the window ended
- 18:41:39Z turn 2 resumed the same sandbox (`resumed`, `started`);
window ends 18:43:39.549Z
  - 18:42:10Z restart: a new worker booted; sandbox still `started`
  - 18:43:45Z sandbox `stopped`, 5 s after the window ended
  - 18:43:45Z fixture sandbox deleted; Daytona returns 404
- Provider calls: one acquire, one resume, two stops, no destroy before
cleanup
- Without the env flag the test is skipped (1 skipped).

## Risks

- Low risk. The change adds a test file only, and the test is skipped by
default. It makes real provider calls only when someone opts in.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, Daytona API).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:42:08 -06:00
DottaandClaude Opus 5.5 f610e6a880 fix(recovery): replace a queued run the dispatcher keeps passing over instead of counting it as a pending path (#21502)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The assigned-work sweep is the backstop that makes sure every open
agent task has a live run, a pending path, or a named wait on a person
> - It counts a task with a queued run as healthy. A queued run that the
dispatcher declines on every pass is never started, so the task waits
forever and the sweep never looks again
> - On a hosted instance one agent had more than 30 queued continuations
that the claim declined on every pass. The sweep saw each task as
healthy
> - This pull request makes the sweep recognize a queued run that the
dispatcher keeps passing over, report it, and replace it once with a new
continuation when it is automatic work
> - The benefit is that an automatic task cannot stay stuck behind a run
that never starts, while a person's work is never cancelled

## Linked Issues or Issue Description

Refs #21391, #21349, #21433, #21394, #21443, #21450

**What happened?**

An agent had more than 30 queued automatic continuations
(`issue_continuation_needed`) that never started. The claim declined
each one on every dispatch pass. #21391 fixed that specific cause in the
chat-control gate. But the sweep still trusted any queued run as a
pending path. A run that the claim declines for any persistent reason
keeps the task "healthy" forever.

**Expected behavior**

A queued run that the dispatcher keeps passing over is not a healthy
pending path. The sweep reports it. For automatic work it cancels the
run through the run lifecycle and queues one new continuation in its
place. For a person's work it reports the run and leaves it.

**Steps to reproduce**

1. Queue an automatic run for a task and make the claim decline it
persistently (for example, a chat-control admission marker that does not
verify).
2. Let the run wait for more than one hour while its agent starts other
runs.
3. Run the assigned-work sweep for the task. Before this change, it
reports the task as `live_run` and does nothing.

## What Changed

- New violation `queued_run_stalled` in
`recovery/assigned-work-invariant.ts`. A queued run is stalled when all
of these are true (read from the database, on the database clock):
- it has been queued for longer than `QUEUED_RUN_AGING_MS` (60 minutes,
from `queued-run-dispatch-order.ts`);
  - it is the oldest queued run of its agent;
- its agent started more than `OLDEST_DISPATCH_SHARE` other runs after
the run became its oldest queued run: after the later of the run's aging
time and the moment the last older run of the agent started or finished.
- A stalled run is not a live run, and it does not let the task's
interruption projection ("working") count as a pending path. Its own
wake is not counted as a pending wake. A stalled run on a blocked task
still counts as live, so the block decides first.
- `listSuspects` lets a task through when its only active run is a
stalled queued run.
- `act()` handles the violation after all existing early returns:
- person work (the #21349 source-chain rule through
`listPersonOriginSourceRunIds`, which this change exports, or an
explicit user continuation): reported, never cancelled;
- a replacement that stalls again (`replacesStalledRunId` in its
context): reported, not cancelled again;
- both log one warning per run, with the run, agent and how long it
waited, and look again after `CONTINUATION_CAP_WINDOW_MS` (one hour);
- automatic work: the existing claim under the task row lock re-checks
the facts and the shared hourly cap and spacing. Inside that claim,
before the person check, it reads the run's row version (`xmin`) and the
number of wakes attached to it. Then the run is cancelled through the
run lifecycle (`setRunStatusFromLive` from `queued`, with that version
and wake count as its condition; wake cancelled, run event,
`releaseIssueExecutionAndPromote` with `suppressImmediateRecovery`). A
person's wake that joins the run after the check, or a claim that starts
it, changes one of them, so the cancel does nothing. Then one
continuation wake is queued with `previousRunId` and
`replacesStalledRunId` set to the cancelled run. If the cancel does
nothing, the unpublished claim entry is removed and nothing is queued.
- New `heartbeat/stalled-queued-run.ts` builds that cancel from the run
lifecycle functions. `heartbeat.ts` and `recovery/service.ts` pass it to
the sweep.
- Plain activity text: "Continued automatically: this task had waited
over an hour to start while its agent started other work, so Paperclip
queued it again."

### Why this signal

- #21433 keeps refused claims and claim back-off in memory only
(`declinedAgedClaimAt`, `claimBackoffs` in `heartbeat/queue.ts`).
Another server cannot read them. Run rows can be read from any server,
on the database clock.
- After #21349, an aged run ranks no lower than ordinary work, and the
oldest-run share starts the oldest ready queued run at least once in
every `OLDEST_DISPATCH_SHARE` starts. So when a run has been its agent's
oldest queued run since after the aging bound, and its agent started
more runs in that time than the share allows, the dispatcher offered the
run a slot and the claim declined it. Starts that happened while older
runs were still waiting do not count: those older runs were getting
their turns.
- A run behind a full agent sees no such starts and is left alone. Under
a slow backlog only the oldest queued run can match. So the younger
queued runs keep their place in line.
- Starts are counted only after the run reached 60 minutes of age, not
after the current server's boot. 60 minutes is longer than a release
swap overlap. So a run that only waits for a new server to reattach to
its Runners does not match. #21450 will add a reattach-phase gate before
this path.

### Why this does not set `retryOfRunId`

An earlier plan added `retryOfRunId` to the sweep's continuation so the
chat-control gate could follow it. An audit of its readers found three
changes in behavior: the admission-time chat-control check would start
to refuse sweep wakes; the claim would wait on the source run's cleanup
(a possible wait forever); and the agent prompt would call the run "a
Retry" that "re-delivers the wake". #21391 already makes these runs
claimable. So this change leaves `retryOfRunId` unset, as today. The
replacement links its source with `previousRunId`, as the sweep already
does.

### Questions from review, answered from the code

- **How often is the invariant pass triggered, and can two passes
overlap?** The `assigned-work-invariant` delivery queue
(`recovery/work.ts`, `scoped-deadline-work.ts`) runs on database-work
notifications, on each task's own deadline (`retryAt`), and on a 30 s
reconstruction of the suspect list. Passes are at least 1 s apart
(`minimumPassIntervalMs` default). In one process, passes of the queue
never overlap (`running` guard in `delivery-work-coordinator.ts`). Two
servers can sweep at the same time. Each repair is serialized by the
claim under the task row lock (`FOR UPDATE` in `claimContinuation`),
which re-reads the facts and the spacing. This change does not add a new
loop.
- **Can two runs or no run exist for one task?** Two runs: no. The
claim's spacing makes a second server back off, and the cancel happens
before the new wake (a queued run on the task would otherwise absorb the
wake). No run: there is a short gap between the cancel and the new wake,
because wake admission locks the task row in its own transaction. If the
server stops in that gap, the task has nothing pending, and the next
pass repairs it as `no_pending_path` under the same cap.

## Verification

- `server/src/__tests__/assigned-work-stalled-queued-run.test.ts` (new,
embedded Postgres, real `createHeartbeatQueue` admission and
`claimQueuedRun`):
- an automatic run that the real claim declines persistently, whose
agent started runs after it aged: reported, cancelled, exactly one
replacement, and the real claim starts the replacement;
  - a run behind a full agent: left alone, not a suspect;
  - a person's declined Retry: reported, not cancelled, one warning;
- a replacement that stalls again: not cancelled on two checks, one
warning, next look one hour later;
- two sweep instances on the same task at the same time: one
replacement;
- three aged runs queued in order, the first two get their turns slowly:
the third is not stalled when it becomes oldest, and is stalled only
after more than four further starts pass it over;
- a person's message joins the run (real wake admission) between the
claim and the cancel: the run is not cancelled, there is no replacement,
and the person's wake stays attached to the run;
- the run starts between the claim and the cancel: no cancel, no
replacement.
- First round, red before the fix: 4 failed, 1 passed (the full-agent
case passes before and after). Green: 5 passed.
- Review round: the three new tests on the previous head: 2 failed (the
backlog case and the person-join case), 1 passed (the started-run case,
which the `queued` check already covered). Green: 8 passed.
- Existing files run one at a time: `assigned-work-invariant.test.ts` 31
passed, `assigned-work-sweep-round-trips.test.ts` 2 passed,
`failed-continuation-continues-again.test.ts` 28 passed,
`recovery-wake-herd.test.ts` 35 passed,
`saved-work-recovery-overdue.test.ts` 4 passed.
- `pnpm --filter @paperclipai/server exec tsc --noEmit`: clean.
- `pnpm verify`: not completed locally (the machine is heavily loaded
and the run was stopped when the code changed). Required CI supplies the
broad coverage.

## Risks

- A task's oldest queued automatic run is cancelled and queued again at
most once per run. Its replacement starts at the back of the line. Only
a run that the dispatcher already passed over can match.
- A replacement that stalls again is only reported, so a persistent
claim refusal cannot cause repeated cancellations.
- The new subqueries run only for queued runs older than one hour. They
use the existing `(company_id, agent_id, started_at)` index. Healthy
runs and the path before `turn.start` do not change.
- #21443 (automatic-run brake) and #21450 (reattach first) touch
`assigned-work-sweep.ts`. The new path comes after every early return
that `act()` has today, so their gates can go before it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). A subagent on the same model
audited the `retryOfRunId` readers.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:40:38 -06:00
Devin FoleyandPaperclip 0a96f972e1 Connect Tailscale networks through device sign-in (#21526)
## Thinking Path

> - Paperclip manages AI agents and the environments where they run.
> - SSH environments can reach machines through a company Tailscale
connection.
> - The initial connector requires a managed OAuth app that Tailscale
limits to its own tailnet.
> - Customers need to connect their own tailnets without registering an
app or supplying a key.
> - This pull request uses Tailscale's normal device sign-in for new
connections.
> - Customers approve the Paperclip device and then select their
machines in SSH environments.

## Linked Issues or Issue Description

**Problem or motivation**

The current Tailscale enrollment path cannot serve independent customer
tailnets through the documented self-service OAuth app model. It also
requires Cloud broker configuration and pilot admission before a
customer can start.

**Proposed solution**

Start a private Tailscale device in the tenant and open its normal
sign-in link. Save its encrypted device state before reporting a
successful connection. Retain the existing SSH machine selection and
agent account behavior.

**Alternatives considered**

Auth-key entry would require customers to copy a credential. A public
OAuth integration still needs a confirmed registration path. Device
sign-in uses a documented Tailscale flow without either requirement. See
[device
authentication](https://tailscale.com/docs/features/tsnet#device-creation-and-authentication)
and [OAuth app
limitations](https://tailscale.com/docs/features/oauth-apps).

**Related PR search**

Searched GitHub PRs for `tailscale` and `device sign-in`, and issues for
`tailscale`. This follows the merged network connector in
[#20617](https://github.com/paperclipai/paperclip/pull/20617) and
preserves the SSH agent-account support in
[#21248](https://github.com/paperclipai/paperclip/pull/21248). The open
[#15648](https://github.com/paperclipai/paperclip/pull/15648) proposes
storing an OAuth client and checking its health; this change instead
enrolls a device through interactive sign-in so customers do not need an
OAuth app.

## What Changed

- New connections use device sign-in. Existing connected devices and
pending legacy OAuth callbacks remain supported.
- Pending sign-ins are bound to their company and initiating user. They
have a fixed ten-minute deadline. Device state uses server-owned vault
secrets with trusted version and digest checks.
- The active Tailscale CLI waits for consent and device approval.
Cancellation, expiry, secret deletion, and stale completion cannot
restore a cancelled connection.
- The connector stays open during sign-in. It provides a link when
popups are blocked and shows device approval separately. Progress checks
stop on completion, failure, expiry, or unmount.
- Shared response types, OpenAPI contracts, SDK source pins, and the
API/setup documentation describe the new flow. No database migration or
Cloud provider configuration is required.

## Verification

- 109 focused tests cover network processes, the real database and
encrypted vault, routes, UI, and legacy connection behavior.
- Server and UI typechecks pass.
- Follow-up `29cf296a597`: reproduced the missing OpenAPI route failure
locally. All 70 OpenAPI and Tailscale route tests and server typecheck
pass after synchronizing the API contract.
- Follow-up `79de05f38b9`: the transient-recovery regression failed
before the fix; all 20 connector UI tests and UI typecheck pass after
it. The query-error policy check reports no findings.
- The public SDK input registry check passes.
- [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/43535594257/attempts/2)
passed on the final head, `29cf296a597e470d432b84767050bc539caa280e`: 57
successful checks, no failures or pending checks. The interrupted
canary, E2E shard, and serialized-suite jobs passed on rerun. Greptile
is 5/5 on this head with no unresolved review threads.
- `pnpm verify --explain` exits 2 because the documentation change
selects broad CI coverage. Focused local checks passed; broad CI passed
on `29cf296a597e470d432b84767050bc539caa280e`.
- Live local qualification on
`29cf296a597e470d432b84767050bc539caa280e`: native device sign-in
completed with interactive user approval. An isolated Ubuntu 24.04 ARM64
EC2 machine appeared in discovery and passed SSH pairing and workspace
probes. The assigned Claude Code agent used a Paperclip-managed
subscription, created `hello_world.txt` containing exactly `Hello,
world!` plus a newline, posted a completion comment, and marked the task
done. An independent read through the private connector verified the
bytes, remote user, and hostname. The remote machine had no stored
Claude login and no leftover temporary credential directories.
- Restarted the local Paperclip server and verified the same Tailscale
device identity, online peer discovery, successful SSH probe, and
unchanged file without new consent. Temporary public SSH ingress was
removed.
- Local qualification covers one user tailnet and the final Core
revision. Browser UI acceptance, two unrelated tailnets, a hosted
restart, and an agent task on the final staging revision are not yet
verified.

## Risks

- The enrollment method changes. It must pass the live qualification
above before broad rollout claims.
- Existing encrypted device state and the vault master key must survive
a restart or redeploy. Device expiry or removal still requires
reconnecting.
- The existing one-serving-process-per-instance constraint remains.
Concurrent replicas need durable network ownership.
- Pending setup uses bounded progress requests and an event-driven CLI
with a deadline. It adds no idle database scan.

## Model Used

OpenAI GPT-6 Astra (`gpt-6-astra`) through Codex in T3 Code, with xhigh
reasoning. Context window size was not exposed by the runtime. No
subagents were used.

## Checklist

- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] My branch name describes the change and contains no internal
ticket or instance details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 17:38:55 -08:00
DottaandClaude Opus 5.5 cefef8f11d perf(workspaces): inspect only hinted workspaces after a scoped commit (#21519)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server cleans up execution workspaces whose task tree is done. A
background sweep (`terminal-workspaces`) finds these workspaces.
> - Every workspace write and every task status change sends a hint to
this sweep. Each hint starts a full rotation over every open candidate
workspace, and each candidate costs a recursive read of its task tree.
> - Runs write workspaces all the time, so with live runs the sweep
never stops, and its cost grows with the number of open workspaces.
> - On a hosted instance with 5 live runs, the task tree read ran 11.6
times per second. On a local copy with run churn it ran 81 to 101 times
per second.
> - This pull request inspects only the workspaces that a hint names or
reaches. Full rotations stay for startup, saved deadlines, failures and
hints without a scope.
> - The benefit is that a workspace change costs a few statements, not
one tree read per open workspace.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The terminal workspace sweep (`registerTerminalWorkspaceWork` in
`server/src/services/terminal-workspace-work.ts` and
`sweepTerminalWorkspaces` in
`server/src/services/execution-workspaces.ts`).

**Current behavior**

Producers send hints such as `workspace:<id>` and `issue:<id>`. The
worker ignores the scope: each settled hint wakes the queue, and the run
starts or continues a full keyset rotation over all candidate
workspaces, 50 per page with 1 second between pages. A hint that arrives
during a rotation also requests a rescan, so a new rotation starts after
the current one. Every candidate with an open task tree costs one
recursive tree read and is skipped.

**Proposed behavior**

The worker keeps the workspace ids and task ids from scoped hints. When
the queue runs only because of such hints, it calls
`sweepTerminalWorkspaces` with `targets`. The sweep then reads only
these candidates:

- the workspaces named by `workspace:` hints;
- the workspaces whose source task is the hinted task or one of its
ancestors (the tree of a workspace is its source task and its
descendants);
- the execution workspace of each hinted task.

A targeted inspection does not move the rotation cursor or its boundary
and does not clear saved deadlines. The per-workspace checks are the
same code as in a rotation.

A full rotation still runs at startup, when a saved deadline is due,
when a hint has no scope or an unknown scope, while a rotation is in
progress, and after a failed targeted inspection.

**Reason and benefit**

Background work must not grow with the backlog of open workspaces. With
4 candidate workspaces, a hint for one task tree now reads 1 tree
instead of 4. A hint during a rotation no longer starts another full
rotation.

## What Changed

- `server/src/services/terminal-workspace-work.ts`: parse scoped hints
into targets. Run a targeted inspection when only scoped hints are
pending. Keep the full rotation for startup, due deadlines, unscoped
hints, continuation pages and failures.
- `server/src/services/terminal-workspace-work.ts` also falls back to a
full rotation when more than 50 workspaces wait for inspection. It keeps
the 1-second pause between rotation pages and runs targeted inspections
between them.
- `server/src/services/issues.ts`: moving a task also hints its former
parent. Removing a task also hints its parent and its execution
workspace. Without this, a former parent's workspace could never be
reached.
- `server/src/services/execution-workspaces.ts`:
`sweepTerminalWorkspaces` accepts `targets`. A targeted page that fills
its limit requests a full rescan. and filters the candidates to the
targeted workspaces. A targeted inspection leaves the rotation state
alone. `terminalWorkspaceWorkSettled` takes `{ targeted: true }` from
the worker, and then a scoped hint does not request a full rescan.
Direct callers keep the old rescan behavior.
- `server/src/index.ts`: pass the new option through the existing
wrapper.
- Tests in `server/src/__tests__/terminal-workspace-work.test.ts` and
`server/src/__tests__/execution-workspaces-service.test.ts`.

## Verification

- New test `inspects only the workspaces a hint reaches, however many
trees stay open`: 3 workspaces with open trees and 1 done tree whose
child task is hinted. Before the change the targeted call checks 4
workspaces. After the change it checks 1, reads 1 tree, and archives the
done workspace.
- New test `inspects only hinted workspaces after a commit, and keeps
unscoped commits and deadlines on the full rotation`: before the change
the worker never passes targets. After the change a scoped commit runs a
targeted inspection, and an unscoped commit and a due deadline still run
a full rotation.
- `npx vitest run src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/git-workspace-expiration.test.ts` (in `server/`) pass.
- New tests `falls back to the full rotation when more workspaces are
hinted than one inspection reads`, `keeps the pause between rotation
pages while hinted workspaces wait` and `hints the former parent and
workspace when a task moves or is removed`. Each fails on the first
commit and passes on the head.
- `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all`. The fixture had 1 company, 60 agents, and open
tasks scaled from 150 to 5,000. Recovery re-dispatched runs to agents
that fail at once, and every run creates or updates an execution
workspace, so the hints keep coming. Each window is 180 seconds and
starts 150 seconds after the server is ready. The machine was shared.

The table gives the rate of the task tree read (`select id, status,
completed_at, cancelled_at, updated_at from issues ... WITH RECURSIVE
issue_tree`):

| Open tasks | Before (per s) | After (per s) |
| --- | --- | --- |
| 150 | 17.56 | 0.93 |
| 1,500 | 109.83 | 5.16 |
| 5,000 | 113.82 | 4.40 |

Total statements and transactions per second depend mostly on how fast
the failing runs cycle in the fixture, so they are noisy. Before →
after: 343 → 287 at 150, 1,647 → 1,562 at 1,500, 1,530 → 1,678 at 5,000
statements/s. In that window the 5,000-task run after the change also
had 10% more run events. With no activity, the server issued 0
statements before and after the change.

## Risks

- A workspace that waits on something without a hint is no longer
inspected as a side effect of unrelated activity. The known cases keep
their path: a workspace held by an active run keeps its past-due
deadline, so the 5-minute deadline rotation inspects it again; cooldown,
compaction, Git idle and retry deadlines still start full rotations. A
delivery that changes outside Paperclip without a work product change
was already only found by a later rotation.
- A failed targeted inspection requests a full rotation, so a lost
target is recovered by the rotation.
- Low migration risk. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:04:35 -06:00
DottaandClaude Opus 5.5 3cd671575d perf(attention): scope native finalization retention hints to the run's task graph (#21521)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server archives old attention items (reviews, blocked tasks,
decisions, failed runs) in a background queue, `attention-retention`.
> - A hint for a whole company makes this queue reload every attention
candidate of the company, 100 per page. Each page also reads review
attention, blocker graphs and source evidence.
> - Each native run finalization sent such a company hint, although a
run changes only the attention items of its own task graph.
> - On a hosted instance with about 1,400 open tasks and live native
runs, the decision retention read ran 26.7 times per second. Native run
finalization is one of the producers that sent a company hint on every
run.
> - The same finalization already sends the issue-scoped hint through
`notifyNativeStatusReassessment` when the run has a task or a run id.
This pull request sends the company hint only when the run has neither.
> - The benefit is that a finished run reloads only the candidates of
its task, its ancestors and the tasks it blocks, not every candidate of
the company.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The attention retention hint that native run finalization sends
(`notifyNativeFinalizationWork` in
`server/src/services/native-finalization-notifications.ts`).

**Current behavior**

`notifyNativeFinalizationWork` calls `notifyAttentionRetentionWork(tx, {
companyId })`. The `attention-retention` queue has no reconstruction
interval, so each finalization reloads all eight candidate kinds of the
company, page by page, with the batch eligibility reads for each page.

**Proposed behavior**

`notifyNativeFinalizationWork` sends the company hint only when the
scope has no `issueId` and no `runId`. Otherwise
`notifyNativeStatusReassessment`, which the same function already calls,
sends `notifyIssueAttentionRetentionWork(tx, { companyId, issueId, runId
})`. That helper resolves the task of the run, adds its parents and the
tasks it blocks (at most 100), and sends a hint with these task ids.
When the run has no task, or the graph has more than 100 tasks, the
helper sends the company hint itself.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. The
attention items that a run can change are the failed run, the review,
the recovery actions, the decisions and the blocker items of its task
graph. The issue-scoped hint covers all of them, and the ordinary task
write path uses the same helper.

## What Changed

- `server/src/services/native-finalization-notifications.ts`: send the
company attention retention hint only for a finalization without a task
or run.
- `server/src/__tests__/attention-retention-work.test.ts`: new test
`scopes native run finalization to the task graph of the run`.

## Verification

- The new test sends the finalization hint once with a run id and once
with a task id. Before the change both hints are `{ companyId }`. After
the change both hints name the task and its parent.
- `npx vitest run src/__tests__/attention-retention-work.test.ts` and
the native work tests (`native-result-commit-work`,
`native-environment-settlement-work`, `native-run-projection-work`,
`native-recovery-notice-work`, `retained-native-session-cleanup-work`,
`native-status-reassessment-work`, `muse-terminal-settlement`,
`ownership-reconciliation-report`) pass (in `server/`).
- `pnpm check:module-boundaries` passes.

## Risks

- No new reads. The issue-scoped hint was already computed in the same
transaction by `notifyNativeStatusReassessment`.
- An attention item outside the task graph of the run is not reloaded by
this hint. Agent status, budget and company changes keep their own
hints.
- Low risk. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:04:30 -06:00
DottaandClaude Opus 5.5 0bba04675c perf(run-events): append in-order native events in one statement (#21511)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - After #21460, text deltas commit in batches, but each batch still
takes five statements: BEGIN, the run lock, a replay probe, one insert,
and COMMIT. A single event without a wake takes four
> - In the agent scaling load model, the process issues about 36
statements per agent per second and saturates its one thread between 200
and 400 agents
> - This pull request writes a batch of in-order native events, or one
such event, with one statement that commits on its own
> - The benefit is about half the statements per agent, less CPU per
agent on the main thread, and less time on a pool connection, with the
same rows, receipts, cursor and locks

## Linked Issues or Issue Description

Refs #21460 (batched deltas). Refs #21435 (source cursor). Refs #21418
(agent scaling plan and load model).

**What existing behavior does this improve?**
The append of native source events that continue their source's stored
cursor: batched text deltas, the first delta after another event, and
every other native event that registers no wake.

**Subsystem affected**
Server run event ingest: `server/src/services/heartbeat-run-events.ts`,
and the port comment in `paperclip-control-plane-port.ts`.

**Current behavior**
A delta batch takes BEGIN, a run lock read (`for no key update`), a
replay probe, one insert, and COMMIT: five statements. One delta that
commits at once takes four. In the load model at 200 agents the process
sends about 6,700 statements per second at event-loop utilization 0.72.

**Proposed behavior**
The same append is one statement. Its update of the run row takes the
same row lock. It requires the stored cursor to end just before the
batch, requires the `next_event_seq` that its own snapshot read, and
finds no replayed event. Then it moves the sequence and the cursor and
inserts every row. If any check fails, it writes nothing and the append
takes the existing locked transaction.

**Reason and benefit**
Fewer statements free the main thread and the shared pool for other
agents. In the load model at 200 agents, statements per second fall from
about 6,700 to about 3,300 and event-loop utilization falls from 0.72 to
0.56.

**Breaking changes**
None. No schema change. Rows, receipts and the cursor are the same.
Servers from before this change keep working on the same runs.

## What Changed

- `appendNewNativeSourceEvents` (`heartbeat-run-events.ts`) is one
statement: a CTE reads `next_event_seq` from the statement snapshot, an
`update heartbeat_runs ... returning` locks the run row and moves
`next_event_seq` and the source cursor, and an `insert ... select from
jsonb_array_elements($9)` writes every row. The text is fixed, with nine
bind parameters for any batch size.
- **Why the `next_event_seq` check.** A statement that waits for the run
row lock keeps the snapshot it took before the wait. Its replay probe
cannot see events that the lock holder committed. Every event append
(native, ordinary, the run coordinator store, the retry queue) moves
`next_event_seq` in its own transaction. So the update requires the
value that its snapshot read. PostgreSQL re-checks that condition on the
newest row version after the wait, and any append committed meanwhile
makes the statement write nothing. The unique indexes on `(run_id,
source_event_id)` and `(run_id, source_instance_id, source_seq)` are a
second guard: a unique violation from this statement also falls back.
- A source without a cursor starts at 0. The probe still finds any
stored event of that source, so a run that an older server wrote falls
back and recounts.
- `appendHeartbeatRunEventBatch` and `appendHeartbeatRunEvent` try the
one statement first for native events without a wake, outside a caller's
transaction. Every other case (a replay, a gap, an event with a wake, a
cursor that lags, a run changed meanwhile, a binding mismatch, a run
deleted meanwhile, a caller's transaction) takes the existing
transaction, unchanged.
- Inside a caller's transaction the append keeps its SAVEPOINT path. A
failed statement there would abort the caller's transaction, and the
fallback could not run.
- `assertWritable` runs before the statement is sent and after it
returns (see Risks).

## Verification

- Red/green on embedded PostgreSQL. I ran the new tests against the
previous `heartbeat-run-events.ts` and saw 9 of them fail; all 25 tests
in the file pass with the change
(`server/src/__tests__/heartbeat-run-event-atomic-append.test.ts`):
- One statement for an in-order event and for a batch (before: 4 and 5).
- A writer revoked before the send sends nothing; a statement that fails
on its third row stores no row and no sequence; the resend commits.
- A writer revoked while the statement runs: the rows stay, the receipt
is rejected, and the resend is a replay with the same rows.
- A batch waits for Stop on the run row and appends after it; a batch
and a single event that wait for a run deletion fail with
`heartbeat_run_event_binding_mismatch` and store nothing.
- An older server stores events 2 and 3 while it holds the run row,
without the cursor: the waiting batch 2–4 falls back, returns duplicate,
duplicate, committed, and the cursor ends at 4.
- Two connections append the same 200-event stream in batches of 10:
each event is stored once, in source order, sequences 1–200 without a
gap, one `committed` and one `duplicate` receipt per event.
- A 7,000-event batch (more than any bind parameter limit at 10
parameters per row) is one statement. Before: it fell back to one
statement per event and timed out at 15 s.
- A run without a cursor starts at its first event; a run with events
from an older server replays and then continues.
- Equivalence: 12 random streams of 30 steps (in-order batches, gaps,
replays of earlier ranges, fills of gaps, ordinary events, conflicting
replays). Each step goes to two runs, one through the new path and one
forced through the one-by-one locked path. Receipts, errors, rows,
`next_event_seq` and cursors are equal at every step.
- Mutation check: with the `next_event_seq` check and the
unique-violation fallback removed, the older-server race test fails.
- The port suite (`paperclip-control-plane-port.test.ts`, with the
package conformance suite) passes: 125 tests. Two tests changed: the
streamed-event test now expects one statement and the new revocation
behavior, and the batch test checks that the five deltas are one
statement sent before the boundary event.
- Other focused suites pass, one file at a time:
`heartbeat-run-event-sequencing` (5), `native-event-boundary-golden`
(1), `native-observer-replay` (1), `paperclip-semantic-conformance` (1),
`native-interaction-bridge` (7), `chat-native-progress-work` (19),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51).
In 2 of 3 runs with this change, all 51 tests passed but the file's
`afterAll` (closing the side pools and the embedded database) hit its 30
s timeout on this busy machine; the third run passed cleanly, and one
run on `origin/master` passed. The native events in that file all
register a wake, so they do not take the new path.
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model: `scripts/benchmark-agent-scaling-load.ts` as merged in
#21418 (per-source counts stop when the window closes). I did not change
its batching rules for this pull request. Model rates: 20 deltas/s, 1
other event/s, 0.3 API requests/s per agent; default pool of 10
connections; embedded PostgreSQL on the same, shared machine. I ran
"before" (`origin/master`) and "after" one after the other, with a pause
between runs, and I give the load average at the start of each run. The
machine was busy with other work (load 10–15 on 18 cores), so compare
rows that ran close together. No run had a failed operation, a failed
checkpoint save or a lost lease.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | after | 60 s | 12.7 | 0.74 | 5,465 | 1,129 / 6,040 | 18,719 /
24,000 | 5,721 / 15,238 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | after | 60 s | 10.0 | 0.69 | 6,158 | 2,357 / 9,483 | 18,771 /
36,000 | 2,280 / 30,985 |
| 600 | after, `DELTA_BATCH_MS=0` | 60 s | 10.8 | 0.68 | 8,014 | 36,135
/ 52,222 | 36,000 / 36,000 | 698 / 3,422 |

At 200 agents an earlier pair gave 0.61 → 0.43 event-loop utilization
and 7,051 → 3,210 statements/s. One process with this change alone still
does not carry 600 agents: at 400 and above the 10-connection pool and
the shared database are the limit (event-loop utilization stays below
0.75 while other events fall behind). #21520 and a third pull request
for the remaining statements of other events and checkpoint saves
continue from here; their description has the stacked results.
- Statements: a delta batch 5 → 1; one delta or one native event without
a wake 4 → 1. Unchanged: an event with a wake (6) and a checkpoint save
(5).

## Risks

- **`assertWritable` window (read first).** Before, the writer check ran
inside the transaction, before the first statement and before COMMIT. A
revocation while the statements ran rolled the append back. Now a
revocation while the one statement runs does not roll it back: the rows
commit, and the caller gets the error instead of the receipt, so no
observer runs and the runner gets no ack. This check guards a controller
that hands its run to a successor at restart
(`nativeRunsDetachingForRestart`). I think this is acceptable: (1) the
earlier check was not atomic with COMMIT either, so a revocation during
the COMMIT round trip already committed; the window grows from one
COMMIT round trip to one statement round trip plus its wait for a pool
connection; (2) the append is idempotent by source event id and payload
hash, so the successor sees the committed row as a replay of the event
that the runner resends; (3) the old controller never acts on the
rejected receipt. A new test covers exactly this.
- A crash loses nothing new: the statement commits or not as a whole,
and a resend after an unknown outcome is a replay.
- The check on `next_event_seq` makes the statement write nothing
whenever any other event of the run committed after its snapshot. Under
heavy contention on one run, more appends take the fallback. The
fallback is the previous code path.
- The statement text is fixed SQL sent through the driver
(`$client.unsafe`), like the rest of the event stream statements.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:59:01 -06:00
DottaandClaude Opus 5.5 47a2c82d35 fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server stops and the incoming
server takes over the live runs on remote Runners
> - The incoming server reached those runs late: it waited up to 60 s
for the outgoing server's controller lease to lapse, re-verified each
run's whole log from object storage, and deferred a reattach when the
agent directory was busy. Queued starts competed with the reattaches for
the same resources.
> - On hosted staging, the later a live run was reached, the more likely
it was to fail. Runs reached in under 150 s never failed; runs reached
after 420 s almost always failed.
> - This pull request hands live runs over at the start of shutdown,
makes a reattach skip work that the same run already did, and holds
queued starts while handed-over runs are being reached, with a 180 s
cap.
> - The benefit is that live runs are reached within seconds of a swap
instead of minutes, so they continue instead of failing.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

In one traced run the incoming server waited on
`remote_controller_lease_live` three times (09:14:21, 09:15:18,
09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of
that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the
run's whole log from object storage. A control run spent 23 s in
`task.prepare`. Queued starts used the same preparation path, the same 8
object-I/O slots and the same database pool, and nothing gave a reattach
priority. The first-attempt failures were `PRP command session.snapshot
timed out` and `codex_history_read_failed`. The retry then failed with
`runner_harness_state_mismatch: resumed_not_suspended`,
`runner_remote_process_identity_unavailable` or
`provider_turn_lost_on_restore`.

**Expected behavior**

Every live run is reached within about 30 s of the swap. Queued work
starts after that, or after a bounded wait.

**Steps to reproduce**

1. Run a two-server release swap with live remote runs and queued runs.
2. Observe `native.recovery.transition` "waiting" events with
`remote_controller_lease_live`, then the claim, then a long
`heartbeat.prepare.run_output` span.
3. The new tests in `reattach-phase.test.ts` and
`heartbeat-native-early-hand-off.test.ts` model this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- Early hand-off at shutdown (`heartbeat/recovery.ts`,
`server/src/index.ts`): in deploy-overlap mode, the outgoing server
releases the remote controller lease of its live runs at SIGTERM
(`recoveryState` `awaiting_runner_reattach`). A run another server
already took over is left alone. The incoming server no longer waits for
the lease to lapse.
- Reattach fast path:
- `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`:
a reattach starts without waiting for the existing log to be
re-verified.
- `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that
meets a busy agent directory waits in process (at most 30 s) instead of
being deferred to a full re-preparation. An ordinary start is still
deferred at once, with no added wait.
- Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is
a worker on the shared delivery coordinator, queue `reattach-phase`.
- Trigger: one startup listing of a previous server's live runs; then
the hand-over notification (`notifyReattachHandOver`, sent in the
hand-over transaction); the earliest lease deadline it watches (a killed
owner); and a 2 s re-check only while handed-over runs are still
pending, which ends with the phase (cap 180 s). The bounded re-check
exists because a run can also end on another path that sends no
notification to this queue.
- Source of truth: `native_run_finalizations` and `heartbeat_runs` rows.
- Quiet condition: nothing pending and no watched lease, so no deadline
and no SQL. It stops with the coordinator at shutdown.
- Reached hook for every driver (`native-session-executor.ts`): a
reattached run counts as reached when the native session runtime hands
the executor its recovered or opened session (`onSession`). Every driver
reaches that point only after the Runner answered this server's attach
and recovery commands, including an adopted `RunnerdExternalDriver`
Runner, which records no new process identity. The identity hook stays
as an earlier signal.
- Claim wake: when the reattach-phase worker finds runs that just became
claimable (hand-over notification or a killed owner's lease deadline),
it starts this server's restart-recovery claim for them at once. A
validation cell showed the claim otherwise waited about 21.5 s after the
hand-over.
- `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase`
return now sits before every action path of `act()`, except starting an
overdue saved-work recovery (it resumes an existing run).
- The in-transaction admission re-check of a queued claim
(`confirmRunAdmissionStillOpen`) also honours the reattach phase. A
phase that begins during a claim returns the run to the queue through
the existing rollback.
- A reattach whose log re-verification runs in the background awaits it
before the log is finalized (`settleDeferredLogRecovery`, both
finalization sites in `run-completion.ts`).
- Reattach phase (`heartbeat/reattach-phase.ts`,
`heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while
runs that a previous controller handed over (lease released or observed
expired, or claimed by this server's restart recovery) are claimable but
not yet reached. A run is reached when its process identity is recorded.
Authentication is logged as a mark. The cap (180 s,
`PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs
became claimable. Later batches join or re-arm the phase. Runs reached
after 150 s are counted and logged.
- Gates on the phase:
- `claimQueuedRun`, start-next, resume-queued and dispatch in
`heartbeat/queue.ts`, `run-lifecycle.ts` and
`native-runner-execution-driver.ts` leave queued runs queued.
- `execution-recovery-resolution.ts`: one line, the first line of
`deliverReconciledExecutions`.
- `recovery/assigned-work-sweep.ts`: one line, outcome
`deferred_reattach_phase`.
- `heartbeat/recovery.ts`: a one-line claim hook
(`noteReattachRunsClaimed`) after restart recovery claims runs.
  - The owner of those three files agreed to these one-line gates.
- `tests/restart-chaos/daytona-image.ts`:
`PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a
controller-only branch against an existing Daytona image. It accepts a
content-ID mismatch only for the exact content ID of the named image,
and it prints a TEST OVERRIDE warning. This is harness-only.

## Verification

Red on master, green on this branch:

- `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six
two-server scenarios were red on the earlier phase timing:
- `does not hold queued starts while the previous owner is alive and not
handing over`
- `closes the gate when the previous owner hands its runs over, and
counts reach latency from the hand-over`
- `runs the cap from the hand-over, so a wedged run ends the phase
without the overlap eating it`
- `begins when a killed owner's lease is observed expired, without any
hand-over`
- `re-arms for a later batch, including runs the previous owner started
after discovery`
- `closes the gate at once when this server claims a run between two
lease reads, dated from the hand-over`
- plus `claimQueuedRun leaves a queued run queued and unstarted, in the
admission gate's shape`, `is the same gate the assigned-work sweep and
reconciled continuation delivery read`, `ends at the cap, counted from
when the runs became claimable, so one wedged Runner cannot hold run
starts`
- `heartbeat-native-early-hand-off.test.ts`: `releases the remote
controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56
ms), `leaves a run another server already took over alone`, `holds the
incoming server's queued starts only from the hand-over, not while the
outgoing server still drives its runs`
- `heartbeat/run-output.test.ts`: `lets a reattach start before its
existing log is re-verified`
- `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at
once, with no wait added to the run-start path`, `lets a reattach wait
its turn in process instead of a deferred full re-preparation`, `still
defers a reattach once its bounded wait runs out`
- `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only
for the exact image content ID named by the test override`
- Review fixes (red before the fix, green after):
`reattach-phase.test.ts` `starts the claim of handed-over runs within a
second of the hand-over notification`; `assigned-work-invariant.test.ts`
`takes no action while handed-over live runs are being reached, and acts
once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never
lets the outgoing server claim back a run it handed off, while the
incoming server may take it over`; `reattach-phase.test.ts` `retries a
failed first listing instead of going quiet, and still finds the
previous owner's runs`, `keeps a hand-over notification whose listing
failed until a listing succeeds`, `moves the cap earlier when a claimed
run turns out to have been handed over before its claim`;
`queue.test.ts` `returns the claim to the queue when a reattach phase
begins while the claim is being checked`; `run-completion.test.ts`
`finalizes the run log in completeRun|failRun only after a reattach's
deferred log recovery settles`. `native-session-executor.test.ts` `ends
the reattach phase once a reattached %s session is driven by this server
again, without the cap or manual settling` over codex_app_server,
opencode_server, acpx_runtime, muse_external, grok_bot_mcp and
openai_dot_mcp (6 of 6 red without the hook; the test drives the
executor with a mocked session runtime, not each real driver). Lifecycle
tests for the new worker: `does no database work and keeps no deadline
on an instance with nothing to reattach`, `waits for the hand-over
notification or the lease deadline instead of polling while the previous
owner drives its runs`, `begins at a killed owner's lease deadline,
without any hand-over or notification`, `stops with the delivery
coordinator at shutdown and ignores a later hand-over`.

Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing
on this branch: `reattach-phase.test.ts`,
`heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53
tests together), `agent-directory-busy.test.ts`,
`heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the
new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30
to 96 on this shared machine, the full `queue.test.ts` and
`recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres
setup; CI supplies them.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR, #21462,
#21442 and #21362 (merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755): SIGTERM to PRP authentication 21.5 to
38 s at swap 1.
- G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7
s at swap 1. All runs that crossed swap 2 were adopted on the same
Runner process. The staging chain (`provider recovery timed out after
900000ms`, then `resumed_not_suspended`) did not recur.
- Caveat: G5c ran before the phase timing fix in this PR. In G5c the
phase opened at process start, its 180 s cap ran out during the overlap,
and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease
claims. The timing fix (the phase begins when the runs become claimable)
has not run on Daytona.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection,
combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11
survivors at both.
- Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after
hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after
SIGTERM. The 30 s target was met.
- Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at
19 s, so the outgoing server drained gracefully. The incoming server did
not claim on the hand-over; it claimed only when the outgoing server
exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity
47 to 57 s after SIGTERM. The target was missed. Why the claims waited
for the exit is not explained.
- The host load was 49 to 70 from other work. 7 of 32 tasks timed out in
the queue or budget after the swaps; we suspect the overload.

## Risks

- Product trade-off: while handed-over live runs are being reached,
queued starts wait, up to the 180 s cap. On an instance with no
handed-over runs, nothing waits.
- Not proven: two-server behavior with the phase timing fix and the
notification-driven watch (no cell has run them); warm standby becoming
active; runs that the outgoing server starts after discovery.
- Early hand-off at SIGTERM orphans a server-side exec tool that is in
flight. With #21442 the next controller settles it as interrupted with
an unknown outcome; it is never run again.
- Swap 2 in G6c: the incoming server claimed only at the outgoing
server's exit. That is unexplained.
- Cross-session ownership: the gates in
`execution-recovery-resolution.ts` and
`recovery/assigned-work-sweep.ts`, and the claim hook in
`heartbeat/recovery.ts`, are in files that the restart-recovery work
owns. They are one-line gates agreed with that owner. That owner said
the gate did not need to be inside the claim transaction; that was
before the phase could begin during a claim. After review, the
in-transaction re-check in `heartbeat/queue.ts` also reads the phase,
through the existing rollback. #21427 (open) also edits
`execution-recovery-resolution.ts`, `heartbeat/queue.ts` and
`recovery/assigned-work-sweep.ts`; whichever merges second needs a
rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned
by the restart-recovery work): a 3-line guard in
`claimNativeRestartRecoveries` returns a run this process handed off at
shutdown as `blocked` / `handed_off_to_next_controller` before takeover
is evaluated, so the outgoing server never claims back a run it handed
off. #21478 (merged) added a sweep path that starts an overdue
saved-work recovery; it is deliberately not gated by the reattach phase,
because it resumes an existing run (reattach-like work), which the phase
makes room for rather than competes with.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:55:31 -06:00